Compare commits
56 Commits
codex/aipr
...
v0.31.2
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
003a46ac30 | ||
|
|
4b0be4cb76 | ||
|
|
b7647379de | ||
|
|
0f89372d71 | ||
|
|
2b0d4cfc49 | ||
|
|
e6d0332fba | ||
|
|
fe45a99cbd | ||
|
|
ae77b06c3c | ||
|
|
b5dd4f12f8 | ||
|
|
75cb214f23 | ||
|
|
4c21973197 | ||
|
|
51ae5e6ec9 | ||
|
|
1cf1f32ddd | ||
|
|
8f3ab88743 | ||
|
|
f8b43a995b | ||
|
|
d9adaf4134 | ||
|
|
40e51d5b20 | ||
|
|
93c1c1e550 | ||
|
|
48eb13b993 | ||
|
|
11179e7e67 | ||
|
|
07e26d6d5a | ||
|
|
7cd29cf9c0 | ||
|
|
2ee4773f4f | ||
|
|
b1d0624061 | ||
|
|
812c825dc6 | ||
|
|
a359d94127 | ||
|
|
c92be9c054 | ||
|
|
10e2bae8c2 | ||
|
|
60ed88b609 | ||
|
|
e85a9fc614 | ||
|
|
a2210f0f78 | ||
|
|
62ad09e816 | ||
|
|
89a71e6f29 | ||
|
|
60f5ff9bab | ||
|
|
fbb6adfbf5 | ||
|
|
749e6e76b6 | ||
|
|
83839b8b11 | ||
|
|
ed898aef9c | ||
|
|
abe0b5c11b | ||
|
|
306ba7f850 | ||
|
|
39f90bd575 | ||
|
|
c4ea918fac | ||
|
|
34d94a6b6b | ||
|
|
ef65acd49c | ||
|
|
5639546990 | ||
|
|
c8fe8cad59 | ||
|
|
d395769df6 | ||
|
|
8bd9d34376 | ||
|
|
2d43263b9e | ||
|
|
2da6ed166b | ||
|
|
da587398d9 | ||
|
|
f5308340af | ||
|
|
981617ee80 | ||
|
|
7abf391c74 | ||
|
|
f12719914d | ||
|
|
bc90e00e25 |
120
.claude/commands/cleanup.md
Normal file
120
.claude/commands/cleanup.md
Normal file
@@ -0,0 +1,120 @@
|
||||
---
|
||||
description: 审查当前工作区未提交代码中的垃圾代码,并在不影响逻辑的前提下自动清理
|
||||
argument-hint: 可选:指定要检查的文件或目录(默认检查所有未提交修改)
|
||||
allowed-tools: ["Read", "Edit", "Bash", "Grep", "Glob"]
|
||||
---
|
||||
|
||||
# /cleanup — 垃圾代码审查与清理
|
||||
|
||||
分析当前工作区(git diff)中的未提交代码,找出并修复常见垃圾代码,**不得改变任何运行逻辑**。
|
||||
|
||||
## 检查范围
|
||||
|
||||
若 `$ARGUMENTS` 非空,则只检查指定文件/目录;否则检查所有未提交修改(`git diff HEAD`)。
|
||||
|
||||
## 审查清单
|
||||
|
||||
按优先级检查以下问题(只报告在本次 diff 中**新增或修改**的代码里存在的问题):
|
||||
|
||||
### 1. 重复逻辑 (Duplicate Logic)
|
||||
- 完全相同或高度相似的代码块在多处出现
|
||||
- 同一函数/方法被多个地方各自实现,已有公共版本未被复用
|
||||
- 相同的 DOM 查询、正则、模板字符串在同一文件重复
|
||||
|
||||
### 2. Magic Numbers / Magic Strings
|
||||
- 裸数字直接参与计算(如偏移量、时间、尺寸、阈值),没有命名常量
|
||||
- 硬编码字符串(如 id 名、状态值、URL 片段)散落在逻辑中
|
||||
- 例外:`0`, `1`, `-1`, `100`, `""` 等语义明确的惯用值不算
|
||||
|
||||
### 3. 命名问题
|
||||
- 含义不明的缩写变量(如 `or_`, `tmp2`, `x2`)
|
||||
- 命名与实际用途不符
|
||||
- 同一概念在不同地方用不同名字表达
|
||||
|
||||
### 4. 死代码 / 无效代码
|
||||
- 注释掉的旧代码块(3行以上)
|
||||
- 声明后从未使用的变量/参数/导入
|
||||
- 永远不会执行的条件分支
|
||||
|
||||
### 5. 代码风格问题
|
||||
- 尾部空白字符(trailing whitespace)
|
||||
- 同一文件内风格不一致(如混用单双引号、缩进不统一)
|
||||
- 空行使用不一致(连续多个空行等)
|
||||
|
||||
### 6. 其他常见问题
|
||||
- 私有辅助函数应被 export 但没有,导致调用方重复实现
|
||||
- 类型/接口重复定义
|
||||
- 过于冗长的条件表达式可以简化(不改逻辑)
|
||||
|
||||
## 执行步骤
|
||||
|
||||
### Step 1 — 获取待检查文件列表
|
||||
|
||||
```bash
|
||||
# 无参数时:获取所有未提交修改
|
||||
git diff HEAD --name-only
|
||||
|
||||
# 有参数时:用 $ARGUMENTS 过滤
|
||||
```
|
||||
|
||||
### Step 2 — 逐文件阅读并分析
|
||||
|
||||
- 用 Read 工具读取完整文件(不只读 diff)
|
||||
- 对照审查清单,记录每个问题:文件名、行号、问题类型、建议修复方式
|
||||
|
||||
### Step 3 — 报告问题清单
|
||||
|
||||
在修改前,先以列表形式输出所有发现的问题:
|
||||
|
||||
```
|
||||
发现 N 个问题:
|
||||
|
||||
[文件] js/foo.js
|
||||
· L34, L78: 重复逻辑 — 两处都实现了相同的 DOM 查询,可提取到 getPanel()
|
||||
· L91: Magic number — 硬编码 14 作为偏移量,应命名为 TOOLTIP_OFFSET
|
||||
|
||||
[文件] js/bar.js
|
||||
· L12: 命名问题 — 变量 `or_` 语义不明,应命名为 outerR/outerG/outerB
|
||||
...
|
||||
```
|
||||
|
||||
如果没有发现问题,直接输出"未发现垃圾代码,当前代码质量良好。"并停止。
|
||||
|
||||
### Step 4 — 执行修复
|
||||
|
||||
对每个问题,使用 Edit 工具进行**最小化修改**:
|
||||
|
||||
- **重复逻辑**:提取为共享常量/函数,更新所有调用点
|
||||
- **Magic number**:在文件顶部或逻辑附近声明 `const NAME = value`,替换所有引用
|
||||
- **命名问题**:重命名变量,更新所有使用处
|
||||
- **死代码**:直接删除
|
||||
- **尾部空白/风格**:修正
|
||||
- **未 export 的函数**:添加 `export`,在调用方改为导入(不重复实现)
|
||||
|
||||
**修复原则:**
|
||||
- 只改在审查清单中发现的问题,不做额外优化
|
||||
- 每次 Edit 只修改确实有问题的行,保持 diff 最小
|
||||
- 改完后用 `grep` 验证旧的坏代码已消失
|
||||
|
||||
### Step 5 — 输出总结
|
||||
|
||||
```
|
||||
清理完成:
|
||||
|
||||
修复了 N 个问题:
|
||||
✓ earth.js — 提取重复 vertexShader 为 ATMOS_VERTEX_SHADER 常量
|
||||
✓ main.js — 提取 TOOLTIP_CURSOR_OFFSET = 14(4处引用)
|
||||
✓ controls.js — export updateLayerButtonState,移除 main.js 中的重复实现
|
||||
...
|
||||
|
||||
未修改的问题(需人工确认):
|
||||
! foo.js L45 — 注释代码块较长,建议手动确认是否可删除
|
||||
```
|
||||
|
||||
## 约束
|
||||
|
||||
- **禁止**改变函数签名、接口定义、导出 API(除非问题正是私有函数应被 export)
|
||||
- **禁止**添加新功能、新抽象、新参数
|
||||
- **禁止**修改注释内容(只删除注释掉的死代码)
|
||||
- **禁止**修改测试文件逻辑
|
||||
- 如果一个 Magic number 的语义不完全确定,**跳过**,在总结中标记为"需人工确认"
|
||||
146
.claude/commands/release.md
Normal file
146
.claude/commands/release.md
Normal file
@@ -0,0 +1,146 @@
|
||||
---
|
||||
description: 发版工作流:根据变更类型决定版本号,更新所有版本文件和 changelog,运行验证,commit 并 push
|
||||
argument-hint: 可选:feature | bugfix | 或直接描述本次发布内容
|
||||
allowed-tools: ["Read", "Edit", "Bash", "Glob", "Grep"]
|
||||
---
|
||||
|
||||
# /release — Planet 发版工作流
|
||||
|
||||
## 版本号规则
|
||||
|
||||
| 变更类型 | 版本跳动 | 适用场景 |
|
||||
|---------|---------|---------|
|
||||
| `feature` | `+0.1.0` | 纯新功能,无 bugfix |
|
||||
| `improvement` | `+0.0.1` | UI 调整、小功能增强、bugfix 混合,或以 UI/体验改进为主的迭代 |
|
||||
| `bugfix` | `+0.0.1` | 纯 bug 修复,无新功能 |
|
||||
| `docs` / `maintenance` / `refactor` | 默认不发版,除非用户明确要求 |
|
||||
|
||||
意图混合时以用户明确描述为准;bugfix + 小 feature 混合默认判定为 `improvement`(`+0.0.1`)。
|
||||
|
||||
## 必须同步更新的文件
|
||||
|
||||
使用 `git rev-parse --show-toplevel` 获取仓库根目录,以下路径均相对于根目录:
|
||||
|
||||
- `VERSION`
|
||||
- `frontend/package.json`(`"version"` 字段)
|
||||
- `pyproject.toml`(`version =` 字段)
|
||||
- `uv.lock`(**不要手动编辑**,通过 `uv lock` 重新生成)
|
||||
- `docs/CHANGELOG.md`
|
||||
- `docs/version-history.md`
|
||||
|
||||
## 执行步骤
|
||||
|
||||
### Step 1 — 环境检查
|
||||
|
||||
```bash
|
||||
git branch --show-current # 确认在 dev 分支
|
||||
git status --short # 检查是否有无关的未暂存修改
|
||||
cat VERSION # 读取当前版本
|
||||
```
|
||||
|
||||
若当前**不在 `dev` 分支**,停下来告知用户,不要继续。
|
||||
|
||||
若存在无关的未暂存修改,列出并询问用户是否一并提交,或先 stash。
|
||||
|
||||
### Step 2 — 确定发版类型与新版本号
|
||||
|
||||
- 若 `$ARGUMENTS` 提供了明确类型(`feature` / `bugfix`),直接使用
|
||||
- 否则根据当前 `git diff HEAD` 和 `git log` 推断
|
||||
- 计算新版本号(例:`0.26.2` → bugfix → `0.26.3`)
|
||||
- **先输出发版计划供用户确认**:
|
||||
|
||||
```
|
||||
发版计划:
|
||||
类型:bugfix
|
||||
版本:0.26.2 → 0.26.3
|
||||
分支:dev
|
||||
将更新:VERSION, frontend/package.json, pyproject.toml, uv.lock, CHANGELOG.md, version-history.md
|
||||
```
|
||||
|
||||
### Step 3 — 更新版本号文件
|
||||
|
||||
按顺序更新(每步用 Edit 工具,精确替换,不要重写整个文件):
|
||||
|
||||
1. `VERSION` — 直接替换全部内容为新版本号
|
||||
2. `frontend/package.json` — 替换 `"version": "x.x.x"` 行
|
||||
3. `pyproject.toml` — 替换 `version = "x.x.x"` 行
|
||||
4. 运行 `uv lock` 重新生成 `uv.lock`(在仓库根目录下执行)
|
||||
|
||||
### Step 4 — 更新 CHANGELOG.md
|
||||
|
||||
在文件顶部插入新条目,格式:
|
||||
|
||||
```markdown
|
||||
## [x.x.x] — YYYY-MM-DD
|
||||
|
||||
### ✨ Features / 🐛 Fixes / 🔧 Improvements
|
||||
- ...(只列高信号条目,最多 5 条)
|
||||
- ...
|
||||
|
||||
---
|
||||
```
|
||||
|
||||
日期使用 `date +%Y-%m-%d` 获取今天的日期。
|
||||
|
||||
### Step 5 — 更新 docs/version-history.md
|
||||
|
||||
- 更新文件头部的"当前开发版本"字段
|
||||
- 在时间线表格顶部插入新行:`| vx.x.x | YYYY-MM-DD | 一句话摘要 |`
|
||||
|
||||
### Step 6 — 验证
|
||||
|
||||
针对本次变更范围做最小验证:
|
||||
|
||||
- Python 文件有修改:`python3 -m py_compile <changed_files>`
|
||||
- Frontend 文件有修改:运行项目标准检查(若无则跳过并说明)
|
||||
- 版本号一致性检查:用 grep 确认 VERSION、package.json、pyproject.toml 中的版本号完全一致
|
||||
|
||||
```bash
|
||||
grep -h "version" VERSION frontend/package.json pyproject.toml
|
||||
```
|
||||
|
||||
### Step 7 — 提交前预览
|
||||
|
||||
展示将要提交的文件列表:
|
||||
|
||||
```bash
|
||||
git diff --stat HEAD
|
||||
```
|
||||
|
||||
再次确认所有必须文件都在变更列表中,**不包含**非预期文件(如调试文件、.env 等)。
|
||||
|
||||
### Step 8 — Commit & Push(用户确认后)
|
||||
|
||||
```bash
|
||||
git add VERSION frontend/package.json pyproject.toml uv.lock docs/CHANGELOG.md docs/version-history.md
|
||||
# 若有代码变更也一并 stage
|
||||
git add <code_files>
|
||||
|
||||
git commit -m "release: bump version to x.x.x"
|
||||
git tag vx.x.x
|
||||
git push origin dev
|
||||
git push origin vx.x.x
|
||||
```
|
||||
|
||||
commit message 固定格式:`release: bump version to x.x.x`
|
||||
|
||||
### Step 9 — 完成确认
|
||||
|
||||
输出摘要:
|
||||
|
||||
```
|
||||
✓ 版本号已更新:0.26.2 → 0.26.3
|
||||
✓ CHANGELOG 已更新
|
||||
✓ version-history 已更新
|
||||
✓ uv.lock 已重新生成
|
||||
✓ 验证通过
|
||||
✓ commit: release: bump version to 0.26.3
|
||||
✓ tag: v0.26.3
|
||||
✓ 已 push 到 origin/dev
|
||||
```
|
||||
|
||||
## 注意事项
|
||||
|
||||
- `uv.lock` 只能通过 `uv lock` 生成,绝不手动编辑
|
||||
- 发版 commit 只包含版本文件 + 本次功能代码,不混入无关改动
|
||||
- 若环境中 `uv` 不可用,说明原因并跳过 lockfile 更新,提醒用户手动运行
|
||||
124
.codex/skills/cleanup/SKILL.md
Normal file
124
.codex/skills/cleanup/SKILL.md
Normal file
@@ -0,0 +1,124 @@
|
||||
---
|
||||
name: cleanup
|
||||
description: Use when the user asks to clean up, lint, or review uncommitted code for common code smells — duplicate logic, magic numbers, unclear naming, dead code, style inconsistencies. Fixes issues without changing any runtime behavior.
|
||||
---
|
||||
|
||||
# Cleanup
|
||||
|
||||
Review and fix code quality issues in the current working tree without altering any logic or behavior.
|
||||
|
||||
## When To Use
|
||||
|
||||
- The user asks to clean up, tidy, or lint uncommitted changes
|
||||
- The user wants a code smell review before releasing or committing
|
||||
- The user mentions magic numbers, duplicate logic, dead code, or naming issues
|
||||
|
||||
Do not refactor architecture, add features, or change behavior.
|
||||
|
||||
## Scope
|
||||
|
||||
If the user specifies a file or directory, check only that. Otherwise check all uncommitted changes (`git diff HEAD`).
|
||||
|
||||
Only report issues present in **newly added or modified** lines of this diff — do not audit unchanged code.
|
||||
|
||||
## Checklist
|
||||
|
||||
### 1. Duplicate Logic
|
||||
- Identical or near-identical code blocks appearing in multiple places
|
||||
- A function/helper that already exists but is re-implemented elsewhere instead of being reused
|
||||
- Repeated DOM queries, regex literals, or template strings within the same file
|
||||
|
||||
### 2. Magic Numbers / Magic Strings
|
||||
- Bare numeric literals used in calculations (offsets, timeouts, sizes, thresholds) without a named constant
|
||||
- Hardcoded strings (IDs, status values, URL fragments) scattered through logic
|
||||
- Exceptions: `0`, `1`, `-1`, `100`, `""` and other idiomatically clear values are fine
|
||||
|
||||
### 3. Naming Issues
|
||||
- Cryptic abbreviations (`or_`, `tmp2`, `x2`)
|
||||
- Names that do not match actual behavior
|
||||
- The same concept referred to by different names in different places
|
||||
|
||||
### 4. Dead Code
|
||||
- Commented-out code blocks (3+ lines)
|
||||
- Variables, parameters, or imports declared but never used
|
||||
- Branches that can never execute
|
||||
|
||||
### 5. Style Inconsistencies
|
||||
- Trailing whitespace
|
||||
- Mixed quote styles or indentation within the same file
|
||||
- Inconsistent blank-line usage (multiple consecutive blank lines, etc.)
|
||||
|
||||
### 6. Other
|
||||
- Private helper functions that should be exported but are not, causing callers to duplicate the implementation
|
||||
- Overly verbose conditions that can be simplified without changing logic
|
||||
|
||||
## Steps
|
||||
|
||||
### Step 1 — Get the file list
|
||||
|
||||
```bash
|
||||
git diff HEAD --name-only
|
||||
```
|
||||
|
||||
Filter to the user-specified path if one was provided.
|
||||
|
||||
### Step 2 — Read and analyze each file
|
||||
|
||||
Read the full file (not just the diff) with the Read tool. For each file, record every issue found: filename, line number, category, and suggested fix.
|
||||
|
||||
### Step 3 — Report findings before touching anything
|
||||
|
||||
Print a structured list:
|
||||
|
||||
```
|
||||
Found N issues:
|
||||
|
||||
[file] js/foo.js
|
||||
· L34, L78: Duplicate logic — same DOM query implemented twice; extract to getPanel()
|
||||
· L91: Magic number — bare 14 used as pixel offset; name it TOOLTIP_OFFSET
|
||||
|
||||
[file] js/bar.js
|
||||
· L12: Naming — variable `or_` is unclear; rename to outerR, outerG, outerB
|
||||
...
|
||||
```
|
||||
|
||||
If no issues are found, output "No code smells detected. Code quality looks good." and stop.
|
||||
|
||||
### Step 4 — Fix each issue
|
||||
|
||||
Use the Edit tool for **minimal, targeted changes**:
|
||||
|
||||
- **Duplicate logic**: extract to a shared constant or function; update all call sites
|
||||
- **Magic number/string**: declare `const NAME = value` near the top of the relevant scope; replace all usages
|
||||
- **Naming**: rename the variable/function; update all references
|
||||
- **Dead code**: delete it
|
||||
- **Trailing whitespace / style**: fix in place
|
||||
- **Unexported helper**: add `export`; update callers to import instead of re-implementing
|
||||
|
||||
Principles:
|
||||
- Only fix issues identified in the checklist — no extra improvements
|
||||
- Keep each Edit as small as possible
|
||||
- After fixing, verify the old bad pattern is gone with grep
|
||||
|
||||
### Step 5 — Summary
|
||||
|
||||
```
|
||||
Cleanup complete:
|
||||
|
||||
Fixed N issues:
|
||||
✓ earth.js — extracted duplicate vertexShader into ATMOS_VERTEX_SHADER constant
|
||||
✓ main.js — extracted TOOLTIP_CURSOR_OFFSET = 14 (4 references updated)
|
||||
✓ controls.js — exported updateLayerButtonState; removed duplicate implementation in main.js
|
||||
...
|
||||
|
||||
Skipped (needs manual review):
|
||||
! foo.js L45 — large commented-out block; confirm it is safe to delete
|
||||
```
|
||||
|
||||
## Constraints
|
||||
|
||||
- **Do not** change function signatures, exported interfaces, or public APIs (unless the issue is a missing export)
|
||||
- **Do not** add new features, abstractions, or parameters
|
||||
- **Do not** rewrite comments (only delete commented-out dead code)
|
||||
- **Do not** touch test file logic
|
||||
- If a magic number's intent is uncertain, skip it and flag it in the summary
|
||||
157
.codex/skills/release/SKILL.md
Normal file
157
.codex/skills/release/SKILL.md
Normal file
@@ -0,0 +1,157 @@
|
||||
---
|
||||
name: release
|
||||
description: Use when the user asks to release, bump version, update changelog/version files, or commit/push a repository release for the Planet repo. Determines version bump type from changes, updates all required version-bearing files, updates changelog and version-history, runs minimal validation, then commits, tags, and pushes.
|
||||
---
|
||||
|
||||
# Release Workflow
|
||||
|
||||
Use this skill for release-oriented work in this repository.
|
||||
|
||||
## When To Use
|
||||
|
||||
- The user asks to `发版`
|
||||
- The user asks to bump a version
|
||||
- The user asks to update `CHANGELOG`, `version-history`, or version files as part of a release
|
||||
- The user asks to commit/push a release or a publishable bugfix/feature bundle
|
||||
|
||||
Do not use this skill for ordinary commits that are not being released.
|
||||
|
||||
## Versioning Rules
|
||||
|
||||
- `feature` -> bump `+0.1.0`
|
||||
- `bugfix` -> bump `+0.0.1`
|
||||
- `docs`, `maintenance`, and `refactor` do not bump by default unless the user explicitly wants a release
|
||||
|
||||
When intent is mixed, prefer the user's stated release intent.
|
||||
|
||||
## Required Files
|
||||
|
||||
Use `git rev-parse --show-toplevel` to get the repo root. All paths are relative to it:
|
||||
|
||||
- `VERSION`
|
||||
- `frontend/package.json` (`"version"` field)
|
||||
- `pyproject.toml` (`version =` field)
|
||||
- `uv.lock` (**never edit manually** — regenerate by running `uv lock`)
|
||||
- `docs/CHANGELOG.md`
|
||||
- `docs/version-history.md`
|
||||
|
||||
## Workflow
|
||||
|
||||
### Step 1 — Environment check
|
||||
|
||||
```bash
|
||||
git branch --show-current # must be on dev
|
||||
git status --short # check for unrelated uncommitted changes
|
||||
cat VERSION # read current version
|
||||
```
|
||||
|
||||
If not on `dev`, stop and tell the user. Do not proceed.
|
||||
|
||||
If unrelated uncommitted changes exist, list them and ask the user whether to include them or stash first.
|
||||
|
||||
### Step 2 — Determine release type and next version
|
||||
|
||||
- If the user provided an explicit type (`feature` / `bugfix`), use it
|
||||
- Otherwise infer from `git diff HEAD` and recent `git log`
|
||||
- Compute the next version (e.g. `0.26.2` → bugfix → `0.26.3`)
|
||||
- **Show the release plan before making any changes:**
|
||||
|
||||
```
|
||||
Release plan:
|
||||
Type: bugfix
|
||||
Version: 0.26.2 → 0.26.3
|
||||
Branch: dev
|
||||
Will update: VERSION, frontend/package.json, pyproject.toml, uv.lock, CHANGELOG.md, version-history.md
|
||||
```
|
||||
|
||||
### Step 3 — Update version files
|
||||
|
||||
Update in order (use Edit for precise replacement, never rewrite whole files):
|
||||
|
||||
1. `VERSION` — replace entire content with new version string
|
||||
2. `frontend/package.json` — replace `"version": "x.x.x"` line
|
||||
3. `pyproject.toml` — replace `version = "x.x.x"` line
|
||||
4. Run `uv lock` at repo root to regenerate `uv.lock`
|
||||
|
||||
### Step 4 — Update CHANGELOG.md
|
||||
|
||||
Insert a new entry at the top of the file:
|
||||
|
||||
```markdown
|
||||
## x.x.x
|
||||
|
||||
Released: YYYY-MM-DD
|
||||
|
||||
### Highlights
|
||||
|
||||
- ...
|
||||
|
||||
### Added / Fixed / Improved
|
||||
|
||||
- ... (high-signal items only, max 5)
|
||||
|
||||
---
|
||||
```
|
||||
|
||||
Get today's date with `date +%Y-%m-%d`.
|
||||
|
||||
### Step 5 — Update docs/version-history.md
|
||||
|
||||
- Update the "current dev version" field in the file header
|
||||
- Insert a new row at the top of the timeline table: `| vx.x.x | YYYY-MM-DD | one-line summary |`
|
||||
|
||||
### Step 6 — Validate
|
||||
|
||||
Run the smallest relevant validation for the changes in scope:
|
||||
|
||||
- Python files changed: `python3 -m py_compile <changed_files>`
|
||||
- Frontend files changed: run the project-standard check if available; otherwise skip and say so
|
||||
- Version consistency: confirm VERSION, package.json, pyproject.toml, and uv.lock all show the same version
|
||||
|
||||
```bash
|
||||
grep -h "version" VERSION frontend/package.json pyproject.toml
|
||||
```
|
||||
|
||||
### Step 7 — Pre-commit preview
|
||||
|
||||
Show what will be committed:
|
||||
|
||||
```bash
|
||||
git diff --stat HEAD
|
||||
```
|
||||
|
||||
Confirm all required files are present and no unexpected files (debug files, `.env`, etc.) are included.
|
||||
|
||||
### Step 8 — Commit, tag, and push
|
||||
|
||||
```bash
|
||||
git add VERSION frontend/package.json pyproject.toml uv.lock docs/CHANGELOG.md docs/version-history.md
|
||||
# also stage any code changes included in this release
|
||||
git add <code_files>
|
||||
|
||||
git commit -m "release: bump version to x.x.x"
|
||||
git tag vx.x.x
|
||||
git push origin dev
|
||||
git push origin vx.x.x
|
||||
```
|
||||
|
||||
Commit message format is fixed: `release: bump version to x.x.x`
|
||||
|
||||
### Step 9 — Completion summary
|
||||
|
||||
```
|
||||
✓ Version bumped: 0.26.2 → 0.26.3
|
||||
✓ CHANGELOG updated
|
||||
✓ version-history updated
|
||||
✓ uv.lock regenerated
|
||||
✓ Validation passed
|
||||
✓ commit: release: bump version to 0.26.3
|
||||
✓ tag: v0.26.3
|
||||
✓ Pushed to origin/dev
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- `uv.lock` must only be updated by running `uv lock`, never manually
|
||||
- The release commit should include only version files + the code for this release — no unrelated changes
|
||||
- If `uv` is unavailable in the environment, say so explicitly and remind the user to run it manually
|
||||
5
.gitignore
vendored
5
.gitignore
vendored
@@ -145,3 +145,8 @@ docs/.venv/
|
||||
*.temp
|
||||
tmp/
|
||||
temp/
|
||||
|
||||
# ----------------------
|
||||
# Runtime Data
|
||||
# ----------------------
|
||||
data/ai/bgp-briefs/
|
||||
|
||||
63
README.md
63
README.md
@@ -102,6 +102,13 @@
|
||||
| Axios | HTTP 客户端 |
|
||||
| Socket.io-client | WebSocket 客户端 |
|
||||
| ECharts | 统计图表 |
|
||||
| Bun | 前端包管理与脚本运行 |
|
||||
|
||||
前端工程统一使用 Bun:
|
||||
|
||||
- 安装依赖使用 `bun install`
|
||||
- 运行脚本使用 `bun run <script>`
|
||||
- 不使用 `npm`、`pnpm`、`yarn`
|
||||
|
||||
### 虚幻引擎客户端
|
||||
|
||||
@@ -205,10 +212,47 @@
|
||||
./planet.sh health
|
||||
```
|
||||
|
||||
前端命令约定:
|
||||
|
||||
```bash
|
||||
cd frontend
|
||||
bun install
|
||||
bun run dev
|
||||
bun run build
|
||||
```
|
||||
|
||||
不要使用 `npm run ...`,避免在 WSL/Windows 混合环境里触发 `cmd.exe` 路径兼容问题。
|
||||
|
||||
## API 文档
|
||||
|
||||
启动服务后访问: `http://localhost:8000/docs`
|
||||
|
||||
## 启动容错参数
|
||||
|
||||
`planet.sh` 现在为依赖安装、数据库、AI Provider 启动加入了有限次重试,并会在数据库与 `aiprovider` 启动后额外等待 Docker healthcheck。
|
||||
|
||||
可通过环境变量临时调整:
|
||||
|
||||
```bash
|
||||
# 例: 放宽 AI Provider 与数据库在网络抖动下的自愈次数
|
||||
AI_PROVIDER_START_MAX_RETRIES=5 \
|
||||
AI_PROVIDER_RETRY_INTERVAL=10 \
|
||||
DATABASE_START_MAX_RETRIES=5 \
|
||||
DATABASE_RETRY_INTERVAL=10 \
|
||||
./planet.sh restart
|
||||
```
|
||||
|
||||
常用参数:
|
||||
|
||||
- `DEPENDENCY_INSTALL_MAX_RETRIES` / `DEPENDENCY_INSTALL_RETRY_INTERVAL`: 控制 `uv sync`、`bun install` 的重试次数与间隔,默认 `3` 次、`5` 秒
|
||||
- `DATABASE_START_MAX_RETRIES` / `DATABASE_RETRY_INTERVAL`: 控制 `postgres`、`redis` 的启动/重启与健康检查自愈,默认 `3` 次、`5` 秒
|
||||
- `AI_PROVIDER_START_MAX_RETRIES` / `AI_PROVIDER_RETRY_INTERVAL`: 控制 `aiprovider` 的构建/启动与容器重启自愈,默认 `3` 次、`5` 秒
|
||||
- `BACKEND_MAX_RETRIES`: 控制后端进程启动重试次数,默认 `3`
|
||||
- `FRONTEND_MAX_RETRIES`: 控制前端 dev server 启动重试次数,默认 `3`
|
||||
- `BACKEND_HEALTH_CHECK_ATTEMPTS` / `BACKEND_HEALTH_CHECK_INTERVAL`: 控制后端 HTTP 健康检查等待次数与间隔,默认 `10` 次、`2` 秒
|
||||
- `FRONTEND_HEALTH_CHECK_ATTEMPTS` / `FRONTEND_HEALTH_CHECK_INTERVAL`: 控制前端 HTTP 可访问检查等待次数与间隔,默认 `10` 次、`2` 秒
|
||||
- `AI_PROVIDER_HEALTH_CHECK_ATTEMPTS` / `AI_PROVIDER_HEALTH_CHECK_INTERVAL`: 控制 `aiprovider` HTTP 健康检查等待次数与间隔,默认 `10` 次、`2` 秒
|
||||
|
||||
## AI 接口预留
|
||||
|
||||
项目现在采用“两层”设计:
|
||||
@@ -284,8 +328,25 @@ AI_PROVIDER_SERVICE_TOKEN=change_me
|
||||
|
||||
详细文档:
|
||||
|
||||
- [docs/aiprovider.md](/home/ray/dev/linkong/planet/docs/aiprovider.md)
|
||||
- [docs/technical/agents-aiprovider.md](/home/ray/dev/linkong/planet/docs/technical/agents-aiprovider.md)
|
||||
- [aiprovider/README.md](/home/ray/dev/linkong/planet/aiprovider/README.md)
|
||||
- [docs/technical/frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md)
|
||||
- [docs/plans/frontend-ai-playground-development-plan.md](/home/ray/dev/linkong/planet/docs/plans/frontend-ai-playground-development-plan.md)
|
||||
- [docs/plans/agents-situational-awareness-foundation-plan.md](/home/ray/dev/linkong/planet/docs/plans/agents-situational-awareness-foundation-plan.md)
|
||||
|
||||
## 前端页面布局规范
|
||||
|
||||
管理后台页面默认遵循“单屏工作区”原则:
|
||||
|
||||
- 页头、摘要区、主工作区应在一屏内形成稳定结构
|
||||
- 主表格 / 主图表 / 主分析区应占据页面主要可视空间
|
||||
- 模块内容超出时优先在卡片、表格、标签页内部滚动
|
||||
- 不依赖整页纵向撑开来容纳主要工作区
|
||||
|
||||
当前推荐参考实现:
|
||||
|
||||
- [frontend/src/pages/BGP/BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx)
|
||||
- [docs/technical/frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md)
|
||||
|
||||
## License
|
||||
|
||||
|
||||
3
TODO.md
3
TODO.md
@@ -13,6 +13,9 @@
|
||||
- [x] 接入 `IPtoASN / IPtoCountry` 作为 prefix-centric geography 的主数据源
|
||||
- [x] 接入 `OpenGeoFeed` 作为 prefix geography 的高质量覆盖/override 数据源
|
||||
- [x] 把 RIR delegated 设计成 prefix geography 的 fallback,而不是主来源
|
||||
- [ ] 为 `aiprovider` 建立 `provider -> api adapter -> compat policy` 的配置中心,优先落成 `json` 或 `yaml` 文件,运行时按 `provider/model` 读取兼容设置,而不是把专项兼容继续散落在 Python 分支里
|
||||
- [ ] 为市面上主流 AI 服务补专项兼容配置并固化到配置文件中,至少覆盖 `OpenAI / Anthropic / MiniMax / Ollama / Moonshot / DeepSeek / Qwen / GLM / Gemini / OpenRouter / vLLM / LM Studio / One API`
|
||||
- [ ] 在兼容配置中补齐可声明项:`api adapter`、`base_url pattern`、`auth header`、`thinking default`、`reasoning block mapping`、`stream path`、`tool-call capability`、`multimodal capability`、`provider-specific request patch`
|
||||
- [ ] 接入 `inetnum` / `inet6num` whois 作为比 RIR 更细粒度的后备层
|
||||
- [x] 在 activity layer 之后继续补 `route leak` 和 `path instability / flap` detector
|
||||
- [ ] 对 [frontend/public/earth/js/bgp.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp.js) 做按职责拆分的小重构,拆成 data / markers / overlays / animation,降低后续维护复杂度
|
||||
|
||||
@@ -6,29 +6,51 @@ AI_TIMEOUT_SECONDS=60
|
||||
AI_HTTP_RETRY_ATTEMPTS=2
|
||||
AI_ANALYSIS_SYSTEM_PROMPT=你是态势感知分析助手。请基于输入的上下文、观测与约束,输出结构化、克制、可执行的分析。
|
||||
|
||||
# Select one provider mode:
|
||||
# - openai_compatible
|
||||
# - claude_compatible
|
||||
# Provider identity. Recommended values:
|
||||
# - minimax
|
||||
# - openai
|
||||
# - ollama
|
||||
AI_PROVIDER=ollama
|
||||
# Compatibility aliases still accepted:
|
||||
# - openai_compatible
|
||||
# - anthropic_compatible
|
||||
# - claude_compatible
|
||||
AI_PROVIDER=minimax
|
||||
|
||||
# Request adapter style, following OpenClaw's API-seam pattern:
|
||||
# - auto
|
||||
# - openai-completions
|
||||
# - anthropic-messages
|
||||
# - ollama-generate
|
||||
AI_PROVIDER_API=anthropic-messages
|
||||
|
||||
# Common model selection
|
||||
AI_MODEL=qwen2.5:7b
|
||||
AI_MODEL=MiniMax-M2.7
|
||||
|
||||
# MiniMax CN Anthropic-compatible example
|
||||
AI_BASE_URL=https://api.minimaxi.com/anthropic
|
||||
AI_API_KEY=sk-cp-change-me
|
||||
AI_MAX_TOKENS=1200
|
||||
AI_ANTHROPIC_VERSION=2023-06-01
|
||||
|
||||
# OpenAI-compatible example (vLLM / LM Studio / One API / local gateway)
|
||||
# AI_PROVIDER=openai_compatible
|
||||
# AI_PROVIDER=openai
|
||||
# AI_PROVIDER_API=openai-completions
|
||||
# AI_BASE_URL=http://127.0.0.1:8001/v1
|
||||
# AI_API_KEY=local-key
|
||||
# AI_MODEL=your-local-model
|
||||
|
||||
# Claude-compatible example (Anthropic / MiniMax / Claude-compatible gateway)
|
||||
# AI_PROVIDER=claude_compatible
|
||||
# AI_BASE_URL=http://127.0.0.1:8002
|
||||
# Anthropic-compatible example (Claude-compatible gateway)
|
||||
# AI_PROVIDER=anthropic
|
||||
# AI_PROVIDER_API=anthropic-messages
|
||||
# AI_BASE_URL=http://127.0.0.1:8002/anthropic
|
||||
# AI_API_KEY=local-key
|
||||
# AI_MODEL=your-model
|
||||
# AI_MAX_TOKENS=1200
|
||||
# AI_ANTHROPIC_VERSION=2023-06-01
|
||||
|
||||
# Ollama native example
|
||||
AI_BASE_URL=http://127.0.0.1:11434
|
||||
AI_API_KEY=
|
||||
AI_MAX_TOKENS=1200
|
||||
AI_ANTHROPIC_VERSION=2023-06-01
|
||||
# AI_PROVIDER=ollama
|
||||
# AI_PROVIDER_API=ollama-generate
|
||||
# AI_BASE_URL=http://127.0.0.1:11434
|
||||
# AI_API_KEY=
|
||||
# AI_MODEL=qwen2.5:7b
|
||||
|
||||
@@ -4,21 +4,31 @@
|
||||
|
||||
完整使用说明见:
|
||||
|
||||
- [docs/aiprovider.md](/home/ray/dev/linkong/planet/docs/aiprovider.md)
|
||||
- [docs/agents/aiprovider.md](/home/ray/dev/linkong/planet/docs/agents/aiprovider.md)
|
||||
|
||||
当前支持:
|
||||
|
||||
- `AI_PROVIDER=openai`
|
||||
- `AI_PROVIDER=openai_compatible`
|
||||
- `AI_PROVIDER=anthropic`
|
||||
- `AI_PROVIDER=anthropic_compatible`
|
||||
- `AI_PROVIDER=claude_compatible`
|
||||
- `AI_PROVIDER=ollama`
|
||||
- provider identity:
|
||||
- `AI_PROVIDER=openai`
|
||||
- `AI_PROVIDER=anthropic`
|
||||
- `AI_PROVIDER=minimax`
|
||||
- `AI_PROVIDER=ollama`
|
||||
- request adapter:
|
||||
- `AI_PROVIDER_API=openai-completions`
|
||||
- `AI_PROVIDER_API=anthropic-messages`
|
||||
- `AI_PROVIDER_API=ollama-generate`
|
||||
|
||||
兼容别名仍然保留:
|
||||
|
||||
- `openai_compatible`
|
||||
- `anthropic_compatible`
|
||||
- `claude_compatible`
|
||||
|
||||
典型配置:
|
||||
|
||||
```env
|
||||
AI_PROVIDER=openai_compatible
|
||||
AI_PROVIDER=openai
|
||||
AI_PROVIDER_API=openai-completions
|
||||
AI_BASE_URL=https://api.openai.com/v1
|
||||
AI_API_KEY=your_api_key
|
||||
AI_MODEL=gpt-4o-mini
|
||||
@@ -26,13 +36,14 @@ AI_TIMEOUT_SECONDS=60
|
||||
AI_PROVIDER_SERVICE_TOKEN=change_me
|
||||
```
|
||||
|
||||
Claude 兼容供应商示例:
|
||||
MiniMax 中国大陆节点示例:
|
||||
|
||||
```env
|
||||
AI_PROVIDER=claude_compatible
|
||||
AI_BASE_URL=https://your-claude-compatible-endpoint.example.com
|
||||
AI_API_KEY=your_api_key
|
||||
AI_MODEL=your-claude-compatible-model
|
||||
AI_PROVIDER=minimax
|
||||
AI_PROVIDER_API=anthropic-messages
|
||||
AI_BASE_URL=https://api.minimaxi.com/anthropic
|
||||
AI_API_KEY=sk-cp-xxxxx
|
||||
AI_MODEL=MiniMax-M2.7
|
||||
AI_TIMEOUT_SECONDS=60
|
||||
AI_MAX_TOKENS=1200
|
||||
AI_ANTHROPIC_VERSION=2023-06-01
|
||||
@@ -43,12 +54,15 @@ AI_PROVIDER_SERVICE_TOKEN=change_me
|
||||
|
||||
- Anthropic 官方 Claude API
|
||||
- Claude 兼容网关
|
||||
- MiniMax 等提供 Claude/Anthropic 风格消息接口的服务
|
||||
- MiniMax 等提供 Anthropic Messages 风格接口的服务
|
||||
|
||||
这套命名方式参考了 OpenClaw 的接入模式: provider 负责标识供应商, `AI_PROVIDER_API` 负责标识协议适配层, 避免把“供应商”和“协议”绑死在一起。
|
||||
|
||||
Ollama 原生示例:
|
||||
|
||||
```env
|
||||
AI_PROVIDER=ollama
|
||||
AI_PROVIDER_API=ollama-generate
|
||||
AI_BASE_URL=http://127.0.0.1:11434
|
||||
AI_API_KEY=
|
||||
AI_MODEL=qwen2.5:7b
|
||||
@@ -58,8 +72,8 @@ AI_PROVIDER_SERVICE_TOKEN=change_me
|
||||
|
||||
本地模型接入建议:
|
||||
|
||||
- `vLLM`、`LM Studio`、`One API`:优先使用 `openai_compatible`
|
||||
- `MiniMax`、Claude 兼容网关:使用 `claude_compatible`
|
||||
- `vLLM`、`LM Studio`、`One API`:`AI_PROVIDER=openai` + `AI_PROVIDER_API=openai-completions`
|
||||
- `MiniMax`、Claude 兼容网关:`AI_PROVIDER=minimax|anthropic` + `AI_PROVIDER_API=anthropic-messages`
|
||||
- `Ollama`:可直接使用 `ollama`
|
||||
|
||||
启动模板:
|
||||
|
||||
@@ -9,6 +9,7 @@ class Settings(BaseSettings):
|
||||
SERVICE_VERSION: str = "0.1.0"
|
||||
|
||||
AI_PROVIDER: str = "disabled"
|
||||
AI_PROVIDER_API: str = "auto"
|
||||
AI_BASE_URL: str = "https://api.openai.com/v1"
|
||||
AI_API_KEY: str = ""
|
||||
AI_MODEL: str = ""
|
||||
|
||||
@@ -8,6 +8,7 @@ from fastapi import HTTPException, status
|
||||
|
||||
from aiprovider.config import settings
|
||||
from aiprovider.schemas import (
|
||||
AIContentBlock,
|
||||
AIProviderStatusResponse,
|
||||
SituationalAnalysisRequest,
|
||||
SituationalAnalysisResponse,
|
||||
@@ -18,9 +19,39 @@ def _normalize_provider(value: str) -> str:
|
||||
return (value or "disabled").strip().lower()
|
||||
|
||||
|
||||
def _normalize_provider_api(value: str) -> str:
|
||||
return (value or "auto").strip().lower().replace("_", "-")
|
||||
|
||||
|
||||
def _resolve_provider_api(provider: str, configured_api: str) -> str:
|
||||
if configured_api and configured_api != "auto":
|
||||
return configured_api
|
||||
|
||||
if provider in {"openai", "openai-compatible", "openai_compatible"}:
|
||||
return "openai-completions"
|
||||
if provider in {
|
||||
"anthropic",
|
||||
"anthropic-compatible",
|
||||
"anthropic_compatible",
|
||||
"claude-compatible",
|
||||
"claude_compatible",
|
||||
"minimax",
|
||||
"kimi-coding",
|
||||
"moonshot-anthropic",
|
||||
}:
|
||||
return "anthropic-messages"
|
||||
if provider == "ollama":
|
||||
return "ollama-generate"
|
||||
return "disabled"
|
||||
|
||||
|
||||
class ProviderService:
|
||||
def __init__(self) -> None:
|
||||
self.provider = _normalize_provider(settings.AI_PROVIDER)
|
||||
self.provider_api = _resolve_provider_api(
|
||||
self.provider,
|
||||
_normalize_provider_api(settings.AI_PROVIDER_API),
|
||||
)
|
||||
self.base_url = settings.AI_BASE_URL.rstrip("/")
|
||||
self.api_key = settings.AI_API_KEY
|
||||
self.default_model = settings.AI_MODEL
|
||||
@@ -32,9 +63,11 @@ class ProviderService:
|
||||
|
||||
def get_status(self) -> AIProviderStatusResponse:
|
||||
enabled = self.provider != "disabled"
|
||||
configured = enabled and bool(self.base_url and self.api_key and self.default_model)
|
||||
has_credentials = bool(self.api_key) if self._requires_api_key() else True
|
||||
configured = enabled and bool(self.base_url and has_credentials and self.default_model)
|
||||
return AIProviderStatusResponse(
|
||||
provider=self.provider,
|
||||
api=self.provider_api if enabled else None,
|
||||
enabled=enabled,
|
||||
configured=configured,
|
||||
model=self.default_model or None,
|
||||
@@ -49,7 +82,8 @@ class ProviderService:
|
||||
)
|
||||
|
||||
model = payload.preferred_model or self.default_model
|
||||
if not self.base_url or not self.api_key or not model:
|
||||
has_credentials = bool(self.api_key) if self._requires_api_key() else True
|
||||
if not self.base_url or not has_credentials or not model:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
|
||||
detail="AI provider is not fully configured. Check AI_BASE_URL, AI_API_KEY, and AI_MODEL.",
|
||||
@@ -57,28 +91,40 @@ class ProviderService:
|
||||
|
||||
prompt = self._build_prompt(payload)
|
||||
|
||||
if self.provider in {"openai", "openai_compatible"}:
|
||||
if self.provider_api == "openai-completions":
|
||||
data = await self._request_openai_compatible(model, prompt)
|
||||
content = self._extract_openai_content(data)
|
||||
elif self.provider in {"anthropic", "anthropic_compatible", "claude_compatible"}:
|
||||
data = await self._request_anthropic_compatible(model, prompt)
|
||||
content_blocks = self._extract_openai_blocks(data)
|
||||
elif self.provider_api == "anthropic-messages":
|
||||
data = await self._request_anthropic_messages(model, prompt, payload.thinking)
|
||||
content = self._extract_anthropic_content(data)
|
||||
elif self.provider == "ollama":
|
||||
content_blocks = self._extract_anthropic_blocks(data)
|
||||
elif self.provider_api == "ollama-generate":
|
||||
data = await self._request_ollama(model, prompt)
|
||||
content = self._extract_ollama_content(data)
|
||||
content_blocks = self._extract_ollama_blocks(data)
|
||||
else:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
detail=f"Unsupported AI provider: {self.provider}",
|
||||
detail=f"Unsupported AI provider API: {self.provider_api}",
|
||||
)
|
||||
|
||||
text_blocks = [block.text for block in content_blocks if block.text]
|
||||
thinking_blocks = [block.thinking for block in content_blocks if block.thinking]
|
||||
|
||||
return SituationalAnalysisResponse(
|
||||
provider=self.provider,
|
||||
model=model,
|
||||
content=content,
|
||||
content_blocks=content_blocks,
|
||||
text_blocks=text_blocks,
|
||||
thinking_blocks=thinking_blocks,
|
||||
raw_response=data,
|
||||
)
|
||||
|
||||
def _requires_api_key(self) -> bool:
|
||||
return self.provider_api != "ollama-generate"
|
||||
|
||||
def _build_prompt(self, payload: SituationalAnalysisRequest) -> str:
|
||||
sections = [
|
||||
f"任务标题:\n{payload.title}",
|
||||
@@ -113,7 +159,12 @@ class ProviderService:
|
||||
request_body=request_body,
|
||||
)
|
||||
|
||||
async def _request_anthropic_compatible(self, model: str, prompt: str) -> dict[str, Any]:
|
||||
async def _request_anthropic_messages(
|
||||
self,
|
||||
model: str,
|
||||
prompt: str,
|
||||
thinking: dict[str, Any] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
request_body = {
|
||||
"model": model,
|
||||
"system": self.system_prompt,
|
||||
@@ -131,8 +182,15 @@ class ProviderService:
|
||||
"max_tokens": self.max_tokens,
|
||||
"temperature": 0.2,
|
||||
}
|
||||
resolved_thinking = self._resolve_anthropic_thinking(thinking)
|
||||
if resolved_thinking:
|
||||
request_body["thinking"] = resolved_thinking
|
||||
if self.provider == "minimax" and self.base_url.endswith("/anthropic"):
|
||||
path = "/v1/messages"
|
||||
else:
|
||||
path = "/messages"
|
||||
return await self._post(
|
||||
path="/messages",
|
||||
path=path,
|
||||
headers={
|
||||
"x-api-key": self.api_key,
|
||||
"anthropic-version": self.anthropic_version,
|
||||
@@ -141,6 +199,25 @@ class ProviderService:
|
||||
request_body=request_body,
|
||||
)
|
||||
|
||||
def _resolve_anthropic_thinking(self, thinking: dict[str, Any] | None) -> dict[str, Any] | None:
|
||||
if thinking:
|
||||
return thinking
|
||||
|
||||
# OpenClaw treats MiniMax's Anthropic-compatible path specially:
|
||||
# disable thinking by default unless the caller explicitly opts in.
|
||||
if self.provider == "minimax":
|
||||
return {"type": "disabled"}
|
||||
|
||||
return None
|
||||
|
||||
async def _request_anthropic_compatible(
|
||||
self,
|
||||
model: str,
|
||||
prompt: str,
|
||||
thinking: dict[str, Any] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
return await self._request_anthropic_messages(model, prompt, thinking)
|
||||
|
||||
async def _request_ollama(self, model: str, prompt: str) -> dict[str, Any]:
|
||||
request_body = {
|
||||
"model": model,
|
||||
@@ -218,6 +295,30 @@ class ProviderService:
|
||||
)
|
||||
return ""
|
||||
|
||||
def _extract_openai_blocks(self, payload: dict[str, Any]) -> list[AIContentBlock]:
|
||||
choices = payload.get("choices") or []
|
||||
if not choices:
|
||||
return []
|
||||
|
||||
message = choices[0].get("message") or {}
|
||||
content = message.get("content")
|
||||
if isinstance(content, str):
|
||||
return [AIContentBlock(type="text", text=content)]
|
||||
if not isinstance(content, list):
|
||||
return []
|
||||
|
||||
blocks: list[AIContentBlock] = []
|
||||
for item in content:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
blocks.append(
|
||||
AIContentBlock(
|
||||
type=str(item.get("type", "text")),
|
||||
text=item.get("text") if isinstance(item.get("text"), str) else None,
|
||||
metadata={k: v for k, v in item.items() if k not in {"type", "text"}},
|
||||
)
|
||||
)
|
||||
return blocks
|
||||
def _extract_anthropic_content(self, payload: dict[str, Any]) -> str:
|
||||
content = payload.get("content")
|
||||
if isinstance(content, str):
|
||||
@@ -233,8 +334,39 @@ class ProviderService:
|
||||
fragments.append(item["text"])
|
||||
return "".join(fragments)
|
||||
|
||||
def _extract_anthropic_blocks(self, payload: dict[str, Any]) -> list[AIContentBlock]:
|
||||
content = payload.get("content")
|
||||
if not isinstance(content, list):
|
||||
return []
|
||||
|
||||
blocks: list[AIContentBlock] = []
|
||||
for item in content:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
blocks.append(
|
||||
AIContentBlock(
|
||||
type=str(item.get("type", "unknown")),
|
||||
text=item.get("text") if isinstance(item.get("text"), str) else None,
|
||||
thinking=item.get("thinking") if isinstance(item.get("thinking"), str) else None,
|
||||
signature=item.get("signature") if isinstance(item.get("signature"), str) else None,
|
||||
metadata={
|
||||
k: v
|
||||
for k, v in item.items()
|
||||
if k not in {"type", "text", "thinking", "signature"}
|
||||
},
|
||||
)
|
||||
)
|
||||
return blocks
|
||||
|
||||
|
||||
def _extract_ollama_content(self, payload: dict[str, Any]) -> str:
|
||||
response = payload.get("response")
|
||||
if isinstance(response, str):
|
||||
return response
|
||||
return ""
|
||||
|
||||
def _extract_ollama_blocks(self, payload: dict[str, Any]) -> list[AIContentBlock]:
|
||||
response = payload.get("response")
|
||||
if isinstance(response, str) and response:
|
||||
return [AIContentBlock(type="text", text=response)]
|
||||
return []
|
||||
|
||||
@@ -3,6 +3,14 @@ from typing import Any
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class AIContentBlock(BaseModel):
|
||||
type: str
|
||||
text: str | None = None
|
||||
thinking: str | None = None
|
||||
signature: str | None = None
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class SituationalAnalysisRequest(BaseModel):
|
||||
title: str = Field(..., min_length=1, max_length=200)
|
||||
objective: str = Field(..., min_length=1, max_length=1000)
|
||||
@@ -10,17 +18,22 @@ class SituationalAnalysisRequest(BaseModel):
|
||||
observations: list[str] = Field(default_factory=list)
|
||||
constraints: list[str] = Field(default_factory=list)
|
||||
preferred_model: str | None = Field(default=None, max_length=200)
|
||||
thinking: dict[str, Any] | None = None
|
||||
|
||||
|
||||
class SituationalAnalysisResponse(BaseModel):
|
||||
provider: str
|
||||
model: str
|
||||
content: str
|
||||
content_blocks: list[AIContentBlock] = Field(default_factory=list)
|
||||
text_blocks: list[str] = Field(default_factory=list)
|
||||
thinking_blocks: list[str] = Field(default_factory=list)
|
||||
raw_response: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class AIProviderStatusResponse(BaseModel):
|
||||
provider: str
|
||||
api: str | None = None
|
||||
enabled: bool
|
||||
configured: bool
|
||||
model: str | None = None
|
||||
|
||||
@@ -13,7 +13,9 @@ from app.api.v1 import (
|
||||
collected_data,
|
||||
visualization,
|
||||
bgp,
|
||||
news,
|
||||
system_control,
|
||||
tv,
|
||||
)
|
||||
|
||||
api_router = APIRouter()
|
||||
@@ -33,3 +35,5 @@ api_router.include_router(settings.router, prefix="/settings", tags=["settings"]
|
||||
api_router.include_router(system_control.router, prefix="/system", tags=["system"])
|
||||
api_router.include_router(visualization.router, prefix="/visualization", tags=["visualization"])
|
||||
api_router.include_router(bgp.router, prefix="/bgp", tags=["bgp"])
|
||||
api_router.include_router(tv.router, prefix="/tv", tags=["tv"])
|
||||
api_router.include_router(news.router, prefix="/news", tags=["news"])
|
||||
|
||||
@@ -1,15 +1,52 @@
|
||||
from uuid import uuid4
|
||||
|
||||
from fastapi import APIRouter, Depends, Request, Response
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request, Response
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.security import get_current_user
|
||||
from app.db.session import get_db
|
||||
from app.models.user import User
|
||||
from app.schemas.ai import (
|
||||
AIProviderStatusResponse,
|
||||
AlertBriefRequest,
|
||||
AlertBriefResponse,
|
||||
BGPBriefRequest,
|
||||
BGPBriefRecordResponse,
|
||||
BGPBriefRecordSummary,
|
||||
PlaygroundMessageActionResponse,
|
||||
PlaygroundMessageCreateRequest,
|
||||
PlaygroundMessageEditRequest,
|
||||
PlaygroundMessageResendRequest,
|
||||
PlaygroundMessageStopRequest,
|
||||
PlaygroundSessionResponse,
|
||||
PlaygroundSessionUpsertRequest,
|
||||
PlaygroundThreadResponse,
|
||||
SituationalAlertBriefRequest,
|
||||
SituationalAlertBriefResponse,
|
||||
SituationalAnalysisRequest,
|
||||
SituationalAnalysisResponse,
|
||||
)
|
||||
from app.services.alert_ai_brief import build_alert_brief_request
|
||||
from app.services.ai_client import AIProviderClient, get_ai_provider_client
|
||||
from app.services.bgp_ai_brief import build_bgp_brief_request
|
||||
from app.services.bgp_ai_brief_store import (
|
||||
get_bgp_brief_record,
|
||||
get_latest_bgp_brief_record,
|
||||
list_bgp_brief_records,
|
||||
save_bgp_brief_record,
|
||||
)
|
||||
from app.services.playground_session_store import (
|
||||
get_playground_session,
|
||||
upsert_playground_session,
|
||||
)
|
||||
from app.services.playground_chat_service import (
|
||||
create_turn,
|
||||
edit_user_message,
|
||||
get_thread,
|
||||
resend_turn,
|
||||
stop_message,
|
||||
)
|
||||
from app.services.situational_alert_ai_brief import build_situational_alert_brief_request
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
@@ -37,3 +74,208 @@ async def analyze_situational_awareness(
|
||||
request_id = request.headers.get("X-Request-ID") or str(uuid4())
|
||||
response.headers["X-Request-ID"] = request_id
|
||||
return await provider_client.analyze(payload, request_id=request_id)
|
||||
|
||||
|
||||
@router.get("/playground/thread", response_model=PlaygroundThreadResponse | None)
|
||||
async def get_playground_thread(
|
||||
session_key: str = "default",
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await get_thread(
|
||||
db,
|
||||
user_id=current_user.id,
|
||||
session_key=session_key,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/playground/session", response_model=PlaygroundSessionResponse | None)
|
||||
async def get_saved_playground_session(
|
||||
session_key: str = "default",
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await get_playground_session(
|
||||
db,
|
||||
user_id=current_user.id,
|
||||
session_key=session_key,
|
||||
)
|
||||
|
||||
|
||||
@router.put("/playground/session", response_model=PlaygroundSessionResponse)
|
||||
async def save_playground_session(
|
||||
payload: PlaygroundSessionUpsertRequest,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await upsert_playground_session(
|
||||
db,
|
||||
user_id=current_user.id,
|
||||
payload=payload,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/playground/messages", response_model=PlaygroundMessageActionResponse)
|
||||
async def create_playground_message(
|
||||
payload: PlaygroundMessageCreateRequest,
|
||||
current_user: User = Depends(get_current_user),
|
||||
provider_client: AIProviderClient = Depends(get_ai_provider_client),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await create_turn(
|
||||
db,
|
||||
user_id=current_user.id,
|
||||
payload=payload,
|
||||
provider_client=provider_client,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/playground/messages/stop", response_model=PlaygroundMessageActionResponse)
|
||||
async def stop_playground_message(
|
||||
payload: PlaygroundMessageStopRequest,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await stop_message(
|
||||
db,
|
||||
user_id=current_user.id,
|
||||
payload=payload,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/playground/messages/resend", response_model=PlaygroundMessageActionResponse)
|
||||
async def resend_playground_message(
|
||||
payload: PlaygroundMessageResendRequest,
|
||||
current_user: User = Depends(get_current_user),
|
||||
provider_client: AIProviderClient = Depends(get_ai_provider_client),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await resend_turn(
|
||||
db,
|
||||
user_id=current_user.id,
|
||||
payload=payload,
|
||||
provider_client=provider_client,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/playground/messages/edit", response_model=PlaygroundMessageActionResponse)
|
||||
async def edit_playground_message(
|
||||
payload: PlaygroundMessageEditRequest,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await edit_user_message(
|
||||
db,
|
||||
user_id=current_user.id,
|
||||
payload=payload,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/bgp/briefs", response_model=list[BGPBriefRecordSummary])
|
||||
async def list_saved_bgp_briefs(
|
||||
current_user: User = Depends(get_current_user),
|
||||
):
|
||||
return list_bgp_brief_records()
|
||||
|
||||
|
||||
@router.get("/bgp/briefs/latest", response_model=BGPBriefRecordResponse | None)
|
||||
async def get_latest_saved_bgp_brief(
|
||||
current_user: User = Depends(get_current_user),
|
||||
):
|
||||
return get_latest_bgp_brief_record()
|
||||
|
||||
|
||||
@router.get("/bgp/briefs/{brief_id}", response_model=BGPBriefRecordResponse)
|
||||
async def get_saved_bgp_brief(
|
||||
brief_id: str,
|
||||
current_user: User = Depends(get_current_user),
|
||||
):
|
||||
record = get_bgp_brief_record(brief_id)
|
||||
if record is None:
|
||||
raise HTTPException(status_code=404, detail="BGP brief not found")
|
||||
return record
|
||||
|
||||
|
||||
@router.post("/bgp/brief", response_model=BGPBriefRecordResponse)
|
||||
async def analyze_bgp_brief(
|
||||
payload: BGPBriefRequest,
|
||||
request: Request,
|
||||
response: Response,
|
||||
current_user: User = Depends(get_current_user),
|
||||
provider_client: AIProviderClient = Depends(get_ai_provider_client),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
request_id = request.headers.get("X-Request-ID") or str(uuid4())
|
||||
response.headers["X-Request-ID"] = request_id
|
||||
|
||||
brief_request, facts, context = await build_bgp_brief_request(
|
||||
db,
|
||||
incident_limit=payload.incident_limit,
|
||||
anomaly_limit=payload.anomaly_limit,
|
||||
collector_limit=payload.collector_limit,
|
||||
)
|
||||
brief_request.preferred_model = payload.preferred_model
|
||||
brief_request.thinking = payload.thinking
|
||||
|
||||
analysis = await provider_client.analyze(brief_request, request_id=request_id)
|
||||
return save_bgp_brief_record(
|
||||
analysis,
|
||||
request_id=request_id,
|
||||
facts=facts,
|
||||
context=context,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/alerts/brief", response_model=AlertBriefResponse)
|
||||
async def analyze_alert_brief(
|
||||
payload: AlertBriefRequest,
|
||||
request: Request,
|
||||
response: Response,
|
||||
current_user: User = Depends(get_current_user),
|
||||
provider_client: AIProviderClient = Depends(get_ai_provider_client),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
request_id = request.headers.get("X-Request-ID") or str(uuid4())
|
||||
response.headers["X-Request-ID"] = request_id
|
||||
|
||||
brief_request, facts, context = await build_alert_brief_request(
|
||||
db,
|
||||
alert_limit=payload.alert_limit,
|
||||
)
|
||||
brief_request.preferred_model = payload.preferred_model
|
||||
brief_request.thinking = payload.thinking
|
||||
|
||||
analysis = await provider_client.analyze(brief_request, request_id=request_id)
|
||||
return AlertBriefResponse(
|
||||
**analysis.model_dump(),
|
||||
title=brief_request.title,
|
||||
objective=brief_request.objective,
|
||||
facts=facts,
|
||||
context=context,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/situational-alerts/brief", response_model=SituationalAlertBriefResponse)
|
||||
async def analyze_situational_alert_brief(
|
||||
payload: SituationalAlertBriefRequest,
|
||||
request: Request,
|
||||
response: Response,
|
||||
current_user: User = Depends(get_current_user),
|
||||
provider_client: AIProviderClient = Depends(get_ai_provider_client),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
request_id = request.headers.get("X-Request-ID") or str(uuid4())
|
||||
response.headers["X-Request-ID"] = request_id
|
||||
|
||||
brief_request, facts, context = await build_situational_alert_brief_request(db)
|
||||
brief_request.preferred_model = payload.preferred_model
|
||||
brief_request.thinking = payload.thinking
|
||||
|
||||
analysis = await provider_client.analyze(brief_request, request_id=request_id)
|
||||
return SituationalAlertBriefResponse(
|
||||
**analysis.model_dump(),
|
||||
title=brief_request.title,
|
||||
objective=brief_request.objective,
|
||||
facts=facts,
|
||||
context=context,
|
||||
)
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
from datetime import UTC, datetime
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, Depends
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from sqlalchemy import select, func, case
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
@@ -9,6 +9,7 @@ from app.db.session import get_db
|
||||
from app.models.user import User
|
||||
from app.core.security import get_current_user
|
||||
from app.models.alert import Alert, AlertSeverity, AlertStatus
|
||||
from app.schemas.alert import AlertResolutionRequest
|
||||
|
||||
|
||||
router = APIRouter()
|
||||
@@ -77,7 +78,7 @@ async def acknowledge_alert(
|
||||
@router.post("/{alert_id}/resolve")
|
||||
async def resolve_alert(
|
||||
alert_id: int,
|
||||
resolution: str,
|
||||
payload: AlertResolutionRequest,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
@@ -85,12 +86,12 @@ async def resolve_alert(
|
||||
alert = result.scalar_one_or_none()
|
||||
|
||||
if not alert:
|
||||
return {"error": "Alert not found"}
|
||||
raise HTTPException(status_code=404, detail="Alert not found")
|
||||
|
||||
alert.status = AlertStatus.RESOLVED
|
||||
alert.resolved_by = current_user.id
|
||||
alert.resolved_at = datetime.now(UTC)
|
||||
alert.resolution_notes = resolution
|
||||
alert.resolution_notes = payload.resolution
|
||||
await db.commit()
|
||||
|
||||
return {"message": "Alert resolved", "alert": alert.to_dict()}
|
||||
@@ -101,25 +102,44 @@ async def get_alert_stats(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
critical_query = select(func.count(Alert.id)).where(
|
||||
Alert.severity == AlertSeverity.CRITICAL,
|
||||
Alert.status == AlertStatus.ACTIVE,
|
||||
result = await db.execute(
|
||||
select(
|
||||
func.sum(
|
||||
case(
|
||||
(
|
||||
(Alert.severity == AlertSeverity.CRITICAL)
|
||||
& (Alert.status == AlertStatus.ACTIVE),
|
||||
1,
|
||||
),
|
||||
else_=0,
|
||||
)
|
||||
).label("critical"),
|
||||
func.sum(
|
||||
case(
|
||||
(
|
||||
(Alert.severity == AlertSeverity.WARNING)
|
||||
& (Alert.status == AlertStatus.ACTIVE),
|
||||
1,
|
||||
),
|
||||
else_=0,
|
||||
)
|
||||
).label("warning"),
|
||||
func.sum(
|
||||
case(
|
||||
(
|
||||
(Alert.severity == AlertSeverity.INFO)
|
||||
& (Alert.status == AlertStatus.ACTIVE),
|
||||
1,
|
||||
),
|
||||
else_=0,
|
||||
)
|
||||
).label("info"),
|
||||
)
|
||||
)
|
||||
warning_query = select(func.count(Alert.id)).where(
|
||||
Alert.severity == AlertSeverity.WARNING,
|
||||
Alert.status == AlertStatus.ACTIVE,
|
||||
)
|
||||
info_query = select(func.count(Alert.id)).where(
|
||||
Alert.severity == AlertSeverity.INFO,
|
||||
Alert.status == AlertStatus.ACTIVE,
|
||||
)
|
||||
|
||||
critical_result = await db.execute(critical_query)
|
||||
warning_result = await db.execute(warning_query)
|
||||
info_result = await db.execute(info_query)
|
||||
row = result.one()
|
||||
|
||||
return {
|
||||
"critical": critical_result.scalar() or 0,
|
||||
"warning": warning_result.scalar() or 0,
|
||||
"info": info_result.scalar() or 0,
|
||||
"critical": row.critical or 0,
|
||||
"warning": row.warning or 0,
|
||||
"info": row.info or 0,
|
||||
}
|
||||
|
||||
@@ -22,16 +22,161 @@ def _parse_dt(value: Optional[str]) -> Optional[datetime]:
|
||||
if not value:
|
||||
return None
|
||||
return datetime.fromisoformat(value.replace("Z", "+00:00"))
|
||||
def _event_filters(
|
||||
*,
|
||||
prefix: Optional[str],
|
||||
origin_asn: Optional[int],
|
||||
peer_asn: Optional[int],
|
||||
collector: Optional[str],
|
||||
event_type: Optional[str],
|
||||
source: Optional[str],
|
||||
time_from: Optional[datetime],
|
||||
time_to: Optional[datetime],
|
||||
):
|
||||
filters = [BGPObservation.source.in_(BGP_SOURCES)]
|
||||
if source:
|
||||
filters.append(BGPObservation.source == source)
|
||||
if prefix:
|
||||
filters.append(BGPObservation.prefix == prefix)
|
||||
if origin_asn is not None:
|
||||
filters.append(BGPObservation.origin_asn == origin_asn)
|
||||
if peer_asn is not None:
|
||||
filters.append(BGPObservation.peer_asn == peer_asn)
|
||||
if collector:
|
||||
filters.append(BGPObservation.collector == collector)
|
||||
if event_type:
|
||||
filters.append(BGPObservation.event_type == event_type)
|
||||
if time_from:
|
||||
filters.append(BGPObservation.observed_at >= time_from)
|
||||
if time_to:
|
||||
filters.append(BGPObservation.observed_at <= time_to)
|
||||
return filters
|
||||
|
||||
|
||||
def _matches_time(value: Optional[datetime], time_from: Optional[datetime], time_to: Optional[datetime]) -> bool:
|
||||
if value is None:
|
||||
return False
|
||||
if time_from and value < time_from:
|
||||
return False
|
||||
if time_to and value > time_to:
|
||||
return False
|
||||
return True
|
||||
def _anomaly_filters(
|
||||
*,
|
||||
severity: Optional[str],
|
||||
anomaly_type: Optional[str],
|
||||
status: Optional[str],
|
||||
prefix: Optional[str],
|
||||
origin_asn: Optional[int],
|
||||
time_from: Optional[datetime],
|
||||
time_to: Optional[datetime],
|
||||
):
|
||||
filters = []
|
||||
if severity:
|
||||
filters.append(BGPAnomaly.severity == severity)
|
||||
if anomaly_type:
|
||||
filters.append(BGPAnomaly.anomaly_type == anomaly_type)
|
||||
if status:
|
||||
filters.append(BGPAnomaly.status == status)
|
||||
if prefix:
|
||||
filters.append(BGPAnomaly.prefix == prefix)
|
||||
if origin_asn is not None:
|
||||
filters.append(BGPAnomaly.origin_asn == origin_asn)
|
||||
if time_from:
|
||||
filters.append(BGPAnomaly.created_at >= time_from)
|
||||
if time_to:
|
||||
filters.append(BGPAnomaly.created_at <= time_to)
|
||||
return filters
|
||||
|
||||
|
||||
def _incident_filters(
|
||||
*,
|
||||
severity: Optional[str],
|
||||
incident_type: Optional[str],
|
||||
status: Optional[str],
|
||||
):
|
||||
filters = []
|
||||
if severity:
|
||||
filters.append(BGPIncident.severity == severity)
|
||||
if incident_type:
|
||||
filters.append(BGPIncident.incident_type == incident_type)
|
||||
if status:
|
||||
filters.append(BGPIncident.status == status)
|
||||
return filters
|
||||
|
||||
|
||||
async def _build_event_summary_payload(db: AsyncSession) -> dict:
|
||||
base_filters = [BGPObservation.source.in_(BGP_SOURCES)]
|
||||
|
||||
total_result = await db.execute(
|
||||
select(func.count(BGPObservation.id)).where(*base_filters)
|
||||
)
|
||||
collectors_result = await db.execute(
|
||||
select(func.count(func.distinct(BGPObservation.collector))).where(
|
||||
*base_filters, BGPObservation.collector.isnot(None)
|
||||
)
|
||||
)
|
||||
prefixes_result = await db.execute(
|
||||
select(func.count(func.distinct(BGPObservation.prefix))).where(
|
||||
*base_filters, BGPObservation.prefix.isnot(None)
|
||||
)
|
||||
)
|
||||
type_result = await db.execute(
|
||||
select(BGPObservation.event_type, func.count(BGPObservation.id))
|
||||
.where(*base_filters)
|
||||
.group_by(BGPObservation.event_type)
|
||||
)
|
||||
|
||||
return {
|
||||
"total": total_result.scalar() or 0,
|
||||
"collector_count": collectors_result.scalar() or 0,
|
||||
"prefix_count": prefixes_result.scalar() or 0,
|
||||
"by_type": {row[0]: row[1] for row in type_result.fetchall()},
|
||||
}
|
||||
|
||||
|
||||
async def _build_anomaly_summary_payload(db: AsyncSession) -> dict:
|
||||
total_result = await db.execute(select(func.count(BGPAnomaly.id)))
|
||||
type_result = await db.execute(
|
||||
select(BGPAnomaly.anomaly_type, func.count(BGPAnomaly.id))
|
||||
.group_by(BGPAnomaly.anomaly_type)
|
||||
.order_by(func.count(BGPAnomaly.id).desc())
|
||||
)
|
||||
severity_result = await db.execute(
|
||||
select(BGPAnomaly.severity, func.count(BGPAnomaly.id))
|
||||
.group_by(BGPAnomaly.severity)
|
||||
.order_by(func.count(BGPAnomaly.id).desc())
|
||||
)
|
||||
status_result = await db.execute(
|
||||
select(BGPAnomaly.status, func.count(BGPAnomaly.id))
|
||||
.group_by(BGPAnomaly.status)
|
||||
.order_by(func.count(BGPAnomaly.id).desc())
|
||||
)
|
||||
|
||||
return {
|
||||
"total": total_result.scalar() or 0,
|
||||
"by_type": {row[0]: row[1] for row in type_result.fetchall()},
|
||||
"by_severity": {row[0]: row[1] for row in severity_result.fetchall()},
|
||||
"by_status": {row[0]: row[1] for row in status_result.fetchall()},
|
||||
}
|
||||
|
||||
|
||||
async def _build_incident_summary_payload(db: AsyncSession) -> dict:
|
||||
total_result = await db.execute(select(func.count(BGPIncident.id)))
|
||||
type_result = await db.execute(
|
||||
select(BGPIncident.incident_type, func.count(BGPIncident.id))
|
||||
.group_by(BGPIncident.incident_type)
|
||||
.order_by(func.count(BGPIncident.id).desc())
|
||||
)
|
||||
severity_result = await db.execute(
|
||||
select(BGPIncident.severity, func.count(BGPIncident.id))
|
||||
.group_by(BGPIncident.severity)
|
||||
.order_by(func.count(BGPIncident.id).desc())
|
||||
)
|
||||
status_result = await db.execute(
|
||||
select(BGPIncident.status, func.count(BGPIncident.id))
|
||||
.group_by(BGPIncident.status)
|
||||
.order_by(func.count(BGPIncident.id).desc())
|
||||
)
|
||||
|
||||
return {
|
||||
"total": total_result.scalar() or 0,
|
||||
"by_type": {row[0]: row[1] for row in type_result.fetchall()},
|
||||
"by_severity": {row[0]: row[1] for row in severity_result.fetchall()},
|
||||
"by_status": {row[0]: row[1] for row in status_result.fetchall()},
|
||||
}
|
||||
|
||||
|
||||
@router.get("/events")
|
||||
@@ -49,41 +194,36 @@ async def list_bgp_events(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
stmt = (
|
||||
select(BGPObservation)
|
||||
.where(BGPObservation.source.in_(BGP_SOURCES))
|
||||
.order_by(BGPObservation.observed_at.desc(), BGPObservation.id.desc())
|
||||
)
|
||||
if source:
|
||||
stmt = stmt.where(BGPObservation.source == source)
|
||||
|
||||
result = await db.execute(stmt)
|
||||
records = result.scalars().all()
|
||||
dt_from = _parse_dt(time_from)
|
||||
dt_to = _parse_dt(time_to)
|
||||
|
||||
filtered = []
|
||||
for record in records:
|
||||
if prefix and record.prefix != prefix:
|
||||
continue
|
||||
if origin_asn is not None and record.origin_asn != origin_asn:
|
||||
continue
|
||||
if peer_asn is not None and record.peer_asn != peer_asn:
|
||||
continue
|
||||
if collector and record.collector != collector:
|
||||
continue
|
||||
if event_type and record.event_type != event_type:
|
||||
continue
|
||||
if (dt_from or dt_to) and not _matches_time(record.observed_at, dt_from, dt_to):
|
||||
continue
|
||||
filtered.append(record)
|
||||
|
||||
filters = _event_filters(
|
||||
prefix=prefix,
|
||||
origin_asn=origin_asn,
|
||||
peer_asn=peer_asn,
|
||||
collector=collector,
|
||||
event_type=event_type,
|
||||
source=source,
|
||||
time_from=dt_from,
|
||||
time_to=dt_to,
|
||||
)
|
||||
offset = (page - 1) * page_size
|
||||
count_result = await db.execute(
|
||||
select(func.count(BGPObservation.id)).where(*filters)
|
||||
)
|
||||
data_result = await db.execute(
|
||||
select(BGPObservation)
|
||||
.where(*filters)
|
||||
.order_by(BGPObservation.observed_at.desc(), BGPObservation.id.desc())
|
||||
.offset(offset)
|
||||
.limit(page_size)
|
||||
)
|
||||
records = data_result.scalars().all()
|
||||
|
||||
return {
|
||||
"total": len(filtered),
|
||||
"total": count_result.scalar() or 0,
|
||||
"page": page,
|
||||
"page_size": page_size,
|
||||
"data": [record.to_dict() for record in filtered[offset : offset + page_size]],
|
||||
"data": [record.to_dict() for record in records],
|
||||
}
|
||||
|
||||
|
||||
@@ -92,21 +232,7 @@ async def get_bgp_event_summary(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
result = await db.execute(select(BGPObservation).where(BGPObservation.source.in_(BGP_SOURCES)))
|
||||
records = result.scalars().all()
|
||||
|
||||
collectors = sorted({record.collector for record in records if record.collector})
|
||||
prefixes = sorted({record.prefix for record in records if record.prefix})
|
||||
by_type: dict[str, int] = {}
|
||||
for record in records:
|
||||
by_type[record.event_type] = by_type.get(record.event_type, 0) + 1
|
||||
|
||||
return {
|
||||
"total": len(records),
|
||||
"collector_count": len(collectors),
|
||||
"prefix_count": len(prefixes),
|
||||
"by_type": by_type,
|
||||
}
|
||||
return await _build_event_summary_payload(db)
|
||||
|
||||
|
||||
@router.get("/collectors")
|
||||
@@ -138,6 +264,32 @@ async def get_bgp_collector_summary(
|
||||
}
|
||||
|
||||
|
||||
@router.get("/overview/summary")
|
||||
async def get_bgp_overview_summary(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
event_summary = await _build_event_summary_payload(db)
|
||||
anomaly_summary = await _build_anomaly_summary_payload(db)
|
||||
incident_summary = await _build_incident_summary_payload(db)
|
||||
collectors = await build_bgp_collector_coverage(db, source_filter=BGP_SOURCES)
|
||||
active_collectors = [item for item in collectors if item["observation_count"] > 0]
|
||||
|
||||
return {
|
||||
"incidentSummary": incident_summary,
|
||||
"anomalySummary": anomaly_summary,
|
||||
"eventSummary": event_summary,
|
||||
"collectorSummary": {
|
||||
"total": len(collectors),
|
||||
"active_collectors": len(active_collectors),
|
||||
"observed_prefixes": sum(item["prefix_count"] for item in active_collectors),
|
||||
"observed_origins": sum(item["origin_asn_count"] for item in active_collectors),
|
||||
"recent_24h_events": sum(item["recent_24h_observation_count"] for item in active_collectors),
|
||||
"recent_7d_events": sum(item["recent_7d_observation_count"] for item in active_collectors),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@router.get("/events/{event_id}")
|
||||
async def get_bgp_event(
|
||||
event_id: int,
|
||||
@@ -164,31 +316,35 @@ async def list_bgp_anomalies(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
stmt = select(BGPAnomaly).order_by(BGPAnomaly.created_at.desc(), BGPAnomaly.id.desc())
|
||||
if severity:
|
||||
stmt = stmt.where(BGPAnomaly.severity == severity)
|
||||
if anomaly_type:
|
||||
stmt = stmt.where(BGPAnomaly.anomaly_type == anomaly_type)
|
||||
if status:
|
||||
stmt = stmt.where(BGPAnomaly.status == status)
|
||||
if prefix:
|
||||
stmt = stmt.where(BGPAnomaly.prefix == prefix)
|
||||
if origin_asn is not None:
|
||||
stmt = stmt.where(BGPAnomaly.origin_asn == origin_asn)
|
||||
|
||||
result = await db.execute(stmt)
|
||||
records = result.scalars().all()
|
||||
dt_from = _parse_dt(time_from)
|
||||
dt_to = _parse_dt(time_to)
|
||||
if dt_from or dt_to:
|
||||
records = [record for record in records if _matches_time(record.created_at, dt_from, dt_to)]
|
||||
|
||||
filters = _anomaly_filters(
|
||||
severity=severity,
|
||||
anomaly_type=anomaly_type,
|
||||
status=status,
|
||||
prefix=prefix,
|
||||
origin_asn=origin_asn,
|
||||
time_from=dt_from,
|
||||
time_to=dt_to,
|
||||
)
|
||||
offset = (page - 1) * page_size
|
||||
total_result = await db.execute(
|
||||
select(func.count(BGPAnomaly.id)).where(*filters)
|
||||
)
|
||||
data_result = await db.execute(
|
||||
select(BGPAnomaly)
|
||||
.where(*filters)
|
||||
.order_by(BGPAnomaly.created_at.desc(), BGPAnomaly.id.desc())
|
||||
.offset(offset)
|
||||
.limit(page_size)
|
||||
)
|
||||
records = data_result.scalars().all()
|
||||
|
||||
return {
|
||||
"total": len(records),
|
||||
"total": total_result.scalar() or 0,
|
||||
"page": page,
|
||||
"page_size": page_size,
|
||||
"data": [record.to_dict() for record in records[offset : offset + page_size]],
|
||||
"data": [record.to_dict() for record in records],
|
||||
}
|
||||
|
||||
|
||||
@@ -197,29 +353,7 @@ async def get_bgp_anomaly_summary(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
total_result = await db.execute(select(func.count(BGPAnomaly.id)))
|
||||
type_result = await db.execute(
|
||||
select(BGPAnomaly.anomaly_type, func.count(BGPAnomaly.id))
|
||||
.group_by(BGPAnomaly.anomaly_type)
|
||||
.order_by(func.count(BGPAnomaly.id).desc())
|
||||
)
|
||||
severity_result = await db.execute(
|
||||
select(BGPAnomaly.severity, func.count(BGPAnomaly.id))
|
||||
.group_by(BGPAnomaly.severity)
|
||||
.order_by(func.count(BGPAnomaly.id).desc())
|
||||
)
|
||||
status_result = await db.execute(
|
||||
select(BGPAnomaly.status, func.count(BGPAnomaly.id))
|
||||
.group_by(BGPAnomaly.status)
|
||||
.order_by(func.count(BGPAnomaly.id).desc())
|
||||
)
|
||||
|
||||
return {
|
||||
"total": total_result.scalar() or 0,
|
||||
"by_type": {row[0]: row[1] for row in type_result.fetchall()},
|
||||
"by_severity": {row[0]: row[1] for row in severity_result.fetchall()},
|
||||
"by_status": {row[0]: row[1] for row in status_result.fetchall()},
|
||||
}
|
||||
return await _build_anomaly_summary_payload(db)
|
||||
|
||||
|
||||
@router.get("/anomalies/{anomaly_id}")
|
||||
@@ -244,22 +378,29 @@ async def list_bgp_incidents(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
stmt = select(BGPIncident).order_by(BGPIncident.created_at.desc(), BGPIncident.id.desc())
|
||||
if severity:
|
||||
stmt = stmt.where(BGPIncident.severity == severity)
|
||||
if incident_type:
|
||||
stmt = stmt.where(BGPIncident.incident_type == incident_type)
|
||||
if status:
|
||||
stmt = stmt.where(BGPIncident.status == status)
|
||||
|
||||
result = await db.execute(stmt)
|
||||
records = result.scalars().all()
|
||||
filters = _incident_filters(
|
||||
severity=severity,
|
||||
incident_type=incident_type,
|
||||
status=status,
|
||||
)
|
||||
offset = (page - 1) * page_size
|
||||
total_result = await db.execute(
|
||||
select(func.count(BGPIncident.id)).where(*filters)
|
||||
)
|
||||
data_result = await db.execute(
|
||||
select(BGPIncident)
|
||||
.where(*filters)
|
||||
.order_by(BGPIncident.created_at.desc(), BGPIncident.id.desc())
|
||||
.offset(offset)
|
||||
.limit(page_size)
|
||||
)
|
||||
records = data_result.scalars().all()
|
||||
|
||||
return {
|
||||
"total": len(records),
|
||||
"total": total_result.scalar() or 0,
|
||||
"page": page,
|
||||
"page_size": page_size,
|
||||
"data": [record.to_dict() for record in records[offset : offset + page_size]],
|
||||
"data": [record.to_dict() for record in records],
|
||||
}
|
||||
|
||||
|
||||
@@ -268,29 +409,7 @@ async def get_bgp_incident_summary(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
total_result = await db.execute(select(func.count(BGPIncident.id)))
|
||||
type_result = await db.execute(
|
||||
select(BGPIncident.incident_type, func.count(BGPIncident.id))
|
||||
.group_by(BGPIncident.incident_type)
|
||||
.order_by(func.count(BGPIncident.id).desc())
|
||||
)
|
||||
severity_result = await db.execute(
|
||||
select(BGPIncident.severity, func.count(BGPIncident.id))
|
||||
.group_by(BGPIncident.severity)
|
||||
.order_by(func.count(BGPIncident.id).desc())
|
||||
)
|
||||
status_result = await db.execute(
|
||||
select(BGPIncident.status, func.count(BGPIncident.id))
|
||||
.group_by(BGPIncident.status)
|
||||
.order_by(func.count(BGPIncident.id).desc())
|
||||
)
|
||||
|
||||
return {
|
||||
"total": total_result.scalar() or 0,
|
||||
"by_type": {row[0]: row[1] for row in type_result.fetchall()},
|
||||
"by_severity": {row[0]: row[1] for row in severity_result.fetchall()},
|
||||
"by_status": {row[0]: row[1] for row in status_result.fetchall()},
|
||||
}
|
||||
return await _build_incident_summary_payload(db)
|
||||
|
||||
|
||||
@router.get("/incidents/{incident_id}")
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from fastapi import APIRouter, Depends
|
||||
from sqlalchemy import select, func, text
|
||||
from sqlalchemy import case, select, func, text
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.db.session import get_db
|
||||
@@ -118,58 +118,77 @@ async def get_stats(
|
||||
built_in_count = len(COLLECTOR_INFO)
|
||||
built_in_active = built_in_count # Built-in are always "active" for counting purposes
|
||||
|
||||
# Count custom configs from database
|
||||
result = await db.execute(select(func.count(DataSourceConfig.id)))
|
||||
custom_count = result.scalar() or 0
|
||||
|
||||
result = await db.execute(
|
||||
select(func.count(DataSourceConfig.id)).where(DataSourceConfig.is_active == True)
|
||||
select(
|
||||
func.count(DataSourceConfig.id).label("custom_count"),
|
||||
func.sum(
|
||||
case((DataSourceConfig.is_active == True, 1), else_=0)
|
||||
).label("custom_active"),
|
||||
)
|
||||
)
|
||||
custom_active = result.scalar() or 0
|
||||
datasource_stats = result.one()
|
||||
custom_count = datasource_stats.custom_count or 0
|
||||
custom_active = datasource_stats.custom_active or 0
|
||||
|
||||
# Total datasources
|
||||
total_datasources = built_in_count + custom_count
|
||||
active_datasources = built_in_active + custom_active
|
||||
|
||||
# Tasks today (from database)
|
||||
result = await db.execute(
|
||||
select(func.count(CollectionTask.id)).where(CollectionTask.started_at >= today_start)
|
||||
)
|
||||
tasks_today = result.scalar() or 0
|
||||
|
||||
result = await db.execute(
|
||||
select(func.count(CollectionTask.id)).where(
|
||||
CollectionTask.status == "success",
|
||||
CollectionTask.started_at >= today_start,
|
||||
select(
|
||||
func.count(CollectionTask.id).label("tasks_today"),
|
||||
func.sum(
|
||||
case(
|
||||
(CollectionTask.status == "success", 1),
|
||||
else_=0,
|
||||
)
|
||||
).label("success_tasks"),
|
||||
)
|
||||
.where(CollectionTask.started_at >= today_start)
|
||||
)
|
||||
success_tasks = result.scalar() or 0
|
||||
task_stats = result.one()
|
||||
tasks_today = task_stats.tasks_today or 0
|
||||
success_tasks = task_stats.success_tasks or 0
|
||||
success_rate = (success_tasks / tasks_today * 100) if tasks_today > 0 else 0
|
||||
|
||||
# Alerts
|
||||
result = await db.execute(
|
||||
select(func.count(Alert.id)).where(
|
||||
Alert.severity == AlertSeverity.CRITICAL,
|
||||
Alert.status == "active",
|
||||
select(
|
||||
func.sum(
|
||||
case(
|
||||
(
|
||||
(Alert.severity == AlertSeverity.CRITICAL)
|
||||
& (Alert.status == "active"),
|
||||
1,
|
||||
),
|
||||
else_=0,
|
||||
)
|
||||
).label("critical_alerts"),
|
||||
func.sum(
|
||||
case(
|
||||
(
|
||||
(Alert.severity == AlertSeverity.WARNING)
|
||||
& (Alert.status == "active"),
|
||||
1,
|
||||
),
|
||||
else_=0,
|
||||
)
|
||||
).label("warning_alerts"),
|
||||
func.sum(
|
||||
case(
|
||||
(
|
||||
(Alert.severity == AlertSeverity.INFO)
|
||||
& (Alert.status == "active"),
|
||||
1,
|
||||
),
|
||||
else_=0,
|
||||
)
|
||||
).label("info_alerts"),
|
||||
)
|
||||
)
|
||||
critical_alerts = result.scalar() or 0
|
||||
|
||||
result = await db.execute(
|
||||
select(func.count(Alert.id)).where(
|
||||
Alert.severity == AlertSeverity.WARNING,
|
||||
Alert.status == "active",
|
||||
)
|
||||
)
|
||||
warning_alerts = result.scalar() or 0
|
||||
|
||||
result = await db.execute(
|
||||
select(func.count(Alert.id)).where(
|
||||
Alert.severity == AlertSeverity.INFO,
|
||||
Alert.status == "active",
|
||||
)
|
||||
)
|
||||
info_alerts = result.scalar() or 0
|
||||
alert_stats = result.one()
|
||||
critical_alerts = alert_stats.critical_alerts or 0
|
||||
warning_alerts = alert_stats.warning_alerts or 0
|
||||
info_alerts = alert_stats.info_alerts or 0
|
||||
|
||||
response = {
|
||||
"total_datasources": total_datasources,
|
||||
|
||||
@@ -3,7 +3,7 @@ from datetime import datetime, timedelta, timezone
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy import func, select, text
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.time import to_iso8601_utc
|
||||
@@ -11,10 +11,17 @@ from app.core.security import get_current_user
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
from app.db.session import get_db
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.models.data_snapshot import DataSnapshot
|
||||
from app.models.datasource import DataSource
|
||||
from app.models.datasource_config import DataSourceConfig
|
||||
from app.models.task import CollectionTask
|
||||
from app.models.user import User
|
||||
from app.services.scheduler import get_latest_task_id_for_datasource, run_collector_now, sync_datasource_job
|
||||
from app.services.scheduler import (
|
||||
cancel_running_collector_now,
|
||||
get_latest_task_id_for_datasource,
|
||||
run_collector_now,
|
||||
sync_datasource_job,
|
||||
)
|
||||
|
||||
router = APIRouter()
|
||||
STALE_RUNNING_TASK_TIMEOUT_MINUTES = 90
|
||||
@@ -34,6 +41,156 @@ def is_due_for_collection(datasource: DataSource, now: datetime) -> bool:
|
||||
return datasource.last_run_at + timedelta(minutes=datasource.frequency_minutes) <= now
|
||||
|
||||
|
||||
def _task_rank_column(order_column):
|
||||
return func.row_number().over(
|
||||
partition_by=CollectionTask.datasource_id,
|
||||
order_by=(order_column.desc().nullslast(), CollectionTask.id.desc()),
|
||||
).label("row_num")
|
||||
|
||||
|
||||
async def _load_latest_running_tasks(
|
||||
db: AsyncSession,
|
||||
datasource_ids: list[int],
|
||||
) -> dict[int, CollectionTask]:
|
||||
if not datasource_ids:
|
||||
return {}
|
||||
|
||||
ranked_tasks = (
|
||||
select(
|
||||
CollectionTask.id.label("task_id"),
|
||||
_task_rank_column(CollectionTask.started_at),
|
||||
)
|
||||
.where(CollectionTask.datasource_id.in_(datasource_ids))
|
||||
.where(CollectionTask.status == "running")
|
||||
.subquery()
|
||||
)
|
||||
result = await db.execute(
|
||||
select(CollectionTask)
|
||||
.join(ranked_tasks, CollectionTask.id == ranked_tasks.c.task_id)
|
||||
.where(ranked_tasks.c.row_num == 1)
|
||||
)
|
||||
return {task.datasource_id: task for task in result.scalars().all()}
|
||||
|
||||
|
||||
async def _load_latest_completed_tasks(
|
||||
db: AsyncSession,
|
||||
datasource_ids: list[int],
|
||||
) -> dict[int, CollectionTask]:
|
||||
if not datasource_ids:
|
||||
return {}
|
||||
|
||||
ranked_tasks = (
|
||||
select(
|
||||
CollectionTask.id.label("task_id"),
|
||||
_task_rank_column(CollectionTask.completed_at),
|
||||
)
|
||||
.where(CollectionTask.datasource_id.in_(datasource_ids))
|
||||
.where(CollectionTask.completed_at.isnot(None))
|
||||
.where(CollectionTask.status.in_(("success", "failed", "cancelled")))
|
||||
.subquery()
|
||||
)
|
||||
result = await db.execute(
|
||||
select(CollectionTask)
|
||||
.join(ranked_tasks, CollectionTask.id == ranked_tasks.c.task_id)
|
||||
.where(ranked_tasks.c.row_num == 1)
|
||||
)
|
||||
return {task.datasource_id: task for task in result.scalars().all()}
|
||||
|
||||
|
||||
async def _load_latest_task_ids(
|
||||
db: AsyncSession,
|
||||
datasource_ids: list[int],
|
||||
) -> dict[int, int]:
|
||||
if not datasource_ids:
|
||||
return {}
|
||||
|
||||
ranked_tasks = (
|
||||
select(
|
||||
CollectionTask.id.label("task_id"),
|
||||
CollectionTask.datasource_id.label("datasource_id"),
|
||||
func.row_number().over(
|
||||
partition_by=CollectionTask.datasource_id,
|
||||
order_by=CollectionTask.id.desc(),
|
||||
).label("row_num"),
|
||||
)
|
||||
.where(CollectionTask.datasource_id.in_(datasource_ids))
|
||||
.subquery()
|
||||
)
|
||||
result = await db.execute(
|
||||
select(ranked_tasks.c.datasource_id, ranked_tasks.c.task_id)
|
||||
.where(ranked_tasks.c.row_num == 1)
|
||||
)
|
||||
return {datasource_id: task_id for datasource_id, task_id in result.all()}
|
||||
|
||||
|
||||
async def _load_datasource_data_counts(
|
||||
db: AsyncSession,
|
||||
sources: list[str],
|
||||
) -> dict[str, int]:
|
||||
if not sources:
|
||||
return {}
|
||||
|
||||
result = await db.execute(
|
||||
select(CollectedData.source, func.count(CollectedData.id))
|
||||
.where(CollectedData.source.in_(sources))
|
||||
.group_by(CollectedData.source)
|
||||
)
|
||||
return {source: count for source, count in result.all()}
|
||||
|
||||
|
||||
async def _load_datasource_endpoint_overrides(
|
||||
db: AsyncSession,
|
||||
sources: list[str],
|
||||
) -> dict[str, str]:
|
||||
if not sources:
|
||||
return {}
|
||||
|
||||
result = await db.execute(
|
||||
select(DataSourceConfig.name, DataSourceConfig.endpoint)
|
||||
.where(DataSourceConfig.name.in_(sources))
|
||||
.where(DataSourceConfig.is_active.is_(True))
|
||||
.where(DataSourceConfig.endpoint.isnot(None))
|
||||
)
|
||||
return {
|
||||
name: endpoint
|
||||
for name, endpoint in result.all()
|
||||
if endpoint
|
||||
}
|
||||
|
||||
|
||||
async def _load_datasource_list_context(
|
||||
db: AsyncSession,
|
||||
datasources: list[DataSource],
|
||||
) -> tuple[dict[int, CollectionTask], dict[int, CollectionTask], dict[str, int], dict[str, str]]:
|
||||
datasource_ids = [datasource.id for datasource in datasources]
|
||||
sources = [datasource.source for datasource in datasources]
|
||||
|
||||
running_tasks = await _load_latest_running_tasks(db, datasource_ids)
|
||||
datasource_by_id = {datasource.id: datasource for datasource in datasources}
|
||||
now = datetime.now(timezone.utc)
|
||||
|
||||
stale_datasource_ids: list[int] = []
|
||||
for datasource_id, task in running_tasks.items():
|
||||
started_at = task.started_at
|
||||
if started_at is None:
|
||||
continue
|
||||
if started_at.tzinfo is None:
|
||||
started_at = started_at.replace(tzinfo=timezone.utc)
|
||||
if now - started_at > timedelta(minutes=STALE_RUNNING_TASK_TIMEOUT_MINUTES):
|
||||
datasource = datasource_by_id.get(datasource_id)
|
||||
if datasource is not None:
|
||||
await fail_and_rollback_stale_running_task(db, datasource, task)
|
||||
stale_datasource_ids.append(datasource_id)
|
||||
|
||||
if stale_datasource_ids:
|
||||
running_tasks = await _load_latest_running_tasks(db, datasource_ids)
|
||||
|
||||
completed_tasks = await _load_latest_completed_tasks(db, datasource_ids)
|
||||
data_counts = await _load_datasource_data_counts(db, sources)
|
||||
endpoint_overrides = await _load_datasource_endpoint_overrides(db, sources)
|
||||
return running_tasks, completed_tasks, data_counts, endpoint_overrides
|
||||
|
||||
|
||||
async def get_datasource_record(db: AsyncSession, source_id: str) -> Optional[DataSource]:
|
||||
datasource = None
|
||||
try:
|
||||
@@ -52,18 +209,6 @@ async def get_datasource_record(db: AsyncSession, source_id: str) -> Optional[Da
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
|
||||
async def get_last_completed_task(db: AsyncSession, datasource_id: int) -> Optional[CollectionTask]:
|
||||
result = await db.execute(
|
||||
select(CollectionTask)
|
||||
.where(CollectionTask.datasource_id == datasource_id)
|
||||
.where(CollectionTask.completed_at.isnot(None))
|
||||
.where(CollectionTask.status.in_(("success", "failed", "cancelled")))
|
||||
.order_by(CollectionTask.completed_at.desc())
|
||||
.limit(1)
|
||||
)
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
|
||||
async def get_running_task(db: AsyncSession, datasource_id: int) -> Optional[CollectionTask]:
|
||||
result = await db.execute(
|
||||
select(CollectionTask)
|
||||
@@ -87,17 +232,152 @@ async def get_running_task(db: AsyncSession, datasource_id: int) -> Optional[Col
|
||||
if now - started_at <= timedelta(minutes=STALE_RUNNING_TASK_TIMEOUT_MINUTES):
|
||||
return task
|
||||
|
||||
existing_error = (task.error_message or "").strip()
|
||||
datasource = await db.get(DataSource, datasource_id)
|
||||
if datasource is not None:
|
||||
await fail_and_rollback_stale_running_task(db, datasource, task)
|
||||
else:
|
||||
existing_error = (task.error_message or "").strip()
|
||||
stale_reason = (
|
||||
f"Marked failed automatically after stale running timeout "
|
||||
f"({STALE_RUNNING_TASK_TIMEOUT_MINUTES}m)"
|
||||
)
|
||||
task.status = "failed"
|
||||
task.phase = "failed"
|
||||
task.completed_at = now
|
||||
task.error_message = f"{existing_error}\n{stale_reason}".strip() if existing_error else stale_reason
|
||||
await db.commit()
|
||||
return None
|
||||
|
||||
|
||||
async def rollback_orphaned_running_task(
|
||||
db: AsyncSession,
|
||||
datasource: DataSource,
|
||||
running_task: CollectionTask,
|
||||
) -> None:
|
||||
snapshot_result = await db.execute(
|
||||
select(DataSnapshot)
|
||||
.where(
|
||||
DataSnapshot.datasource_id == datasource.id,
|
||||
DataSnapshot.task_id == running_task.id,
|
||||
)
|
||||
.order_by(DataSnapshot.id.desc())
|
||||
.limit(1)
|
||||
)
|
||||
snapshot = snapshot_result.scalar_one_or_none()
|
||||
|
||||
await db.execute(CollectedData.__table__.delete().where(CollectedData.task_id == running_task.id))
|
||||
|
||||
await db.execute(
|
||||
text(
|
||||
"""
|
||||
UPDATE collected_data
|
||||
SET is_current = FALSE
|
||||
WHERE source = :source
|
||||
"""
|
||||
),
|
||||
{"source": datasource.source},
|
||||
)
|
||||
|
||||
if snapshot is not None:
|
||||
snapshot.status = "cancelled"
|
||||
snapshot.is_current = False
|
||||
snapshot.completed_at = datetime.now(timezone.utc)
|
||||
summary = dict(snapshot.summary or {})
|
||||
summary["rollback"] = True
|
||||
summary["rollback_reason"] = "orphaned_running_task_after_backend_restart"
|
||||
snapshot.summary = summary
|
||||
|
||||
if snapshot.parent_snapshot_id is not None:
|
||||
parent_snapshot = await db.get(DataSnapshot, snapshot.parent_snapshot_id)
|
||||
if parent_snapshot:
|
||||
parent_snapshot.is_current = True
|
||||
await db.execute(
|
||||
text(
|
||||
"""
|
||||
UPDATE collected_data
|
||||
SET is_current = TRUE
|
||||
WHERE snapshot_id = :snapshot_id
|
||||
"""
|
||||
),
|
||||
{"snapshot_id": snapshot.parent_snapshot_id},
|
||||
)
|
||||
|
||||
running_task.status = "cancelled"
|
||||
running_task.phase = "cancelled"
|
||||
running_task.completed_at = datetime.now(timezone.utc)
|
||||
existing_error = (running_task.error_message or "").strip()
|
||||
cancel_reason = "Cancelled after backend restart because the running task handle was lost; incomplete writes rolled back"
|
||||
running_task.error_message = f"{existing_error}\n{cancel_reason}".strip() if existing_error else cancel_reason
|
||||
datasource.last_status = "cancelled"
|
||||
datasource.last_run_at = datetime.now(timezone.utc)
|
||||
await db.commit()
|
||||
|
||||
|
||||
async def fail_and_rollback_stale_running_task(
|
||||
db: AsyncSession,
|
||||
datasource: DataSource,
|
||||
running_task: CollectionTask,
|
||||
) -> None:
|
||||
snapshot_result = await db.execute(
|
||||
select(DataSnapshot)
|
||||
.where(
|
||||
DataSnapshot.datasource_id == datasource.id,
|
||||
DataSnapshot.task_id == running_task.id,
|
||||
)
|
||||
.order_by(DataSnapshot.id.desc())
|
||||
.limit(1)
|
||||
)
|
||||
snapshot = snapshot_result.scalar_one_or_none()
|
||||
|
||||
await db.execute(CollectedData.__table__.delete().where(CollectedData.task_id == running_task.id))
|
||||
|
||||
await db.execute(
|
||||
text(
|
||||
"""
|
||||
UPDATE collected_data
|
||||
SET is_current = FALSE
|
||||
WHERE source = :source
|
||||
"""
|
||||
),
|
||||
{"source": datasource.source},
|
||||
)
|
||||
|
||||
if snapshot is not None:
|
||||
snapshot.status = "failed"
|
||||
snapshot.is_current = False
|
||||
snapshot.completed_at = datetime.now(timezone.utc)
|
||||
summary = dict(snapshot.summary or {})
|
||||
summary["rollback"] = True
|
||||
summary["rollback_reason"] = "stale_running_task_timeout"
|
||||
snapshot.summary = summary
|
||||
|
||||
if snapshot.parent_snapshot_id is not None:
|
||||
parent_snapshot = await db.get(DataSnapshot, snapshot.parent_snapshot_id)
|
||||
if parent_snapshot:
|
||||
parent_snapshot.is_current = True
|
||||
await db.execute(
|
||||
text(
|
||||
"""
|
||||
UPDATE collected_data
|
||||
SET is_current = TRUE
|
||||
WHERE snapshot_id = :snapshot_id
|
||||
"""
|
||||
),
|
||||
{"snapshot_id": snapshot.parent_snapshot_id},
|
||||
)
|
||||
|
||||
existing_error = (running_task.error_message or "").strip()
|
||||
stale_reason = (
|
||||
f"Marked failed automatically after stale running timeout "
|
||||
f"({STALE_RUNNING_TASK_TIMEOUT_MINUTES}m)"
|
||||
f"({STALE_RUNNING_TASK_TIMEOUT_MINUTES}m); incomplete writes rolled back"
|
||||
)
|
||||
task.status = "failed"
|
||||
task.phase = "failed"
|
||||
task.completed_at = now
|
||||
task.error_message = f"{existing_error}\n{stale_reason}".strip() if existing_error else stale_reason
|
||||
running_task.status = "failed"
|
||||
running_task.phase = "failed"
|
||||
running_task.completed_at = datetime.now(timezone.utc)
|
||||
running_task.error_message = f"{existing_error}\n{stale_reason}".strip() if existing_error else stale_reason
|
||||
datasource.last_status = "failed"
|
||||
datasource.last_run_at = datetime.now(timezone.utc)
|
||||
await db.commit()
|
||||
return None
|
||||
|
||||
|
||||
@router.get("")
|
||||
@@ -121,14 +401,17 @@ async def list_datasources(
|
||||
|
||||
collector_list = []
|
||||
config = get_data_sources_config()
|
||||
running_tasks, completed_tasks, data_counts, endpoint_overrides = await _load_datasource_list_context(
|
||||
db,
|
||||
datasources,
|
||||
)
|
||||
for datasource in datasources:
|
||||
running_task = await get_running_task(db, datasource.id)
|
||||
last_task = await get_last_completed_task(db, datasource.id)
|
||||
endpoint = await config.get_url(datasource.source, db)
|
||||
data_count_result = await db.execute(
|
||||
select(func.count(CollectedData.id)).where(CollectedData.source == datasource.source)
|
||||
running_task = running_tasks.get(datasource.id)
|
||||
last_task = completed_tasks.get(datasource.id)
|
||||
endpoint = endpoint_overrides.get(datasource.source) or config.get_yaml_url(
|
||||
datasource.source,
|
||||
)
|
||||
data_count = data_count_result.scalar() or 0
|
||||
data_count = data_counts.get(datasource.source, 0)
|
||||
|
||||
last_run_at = datasource.last_run_at or (last_task.completed_at if last_task else None)
|
||||
last_run = to_iso8601_utc(last_run_at)
|
||||
@@ -189,9 +472,13 @@ async def trigger_all_datasources(
|
||||
skipped_sources: list[dict] = []
|
||||
failed_sources: list[dict] = []
|
||||
now = datetime.now(timezone.utc)
|
||||
running_tasks = await _load_latest_running_tasks(
|
||||
db,
|
||||
[datasource.id for datasource in datasources],
|
||||
)
|
||||
|
||||
for datasource in datasources:
|
||||
running_task = await get_running_task(db, datasource.id)
|
||||
running_task = running_tasks.get(datasource.id)
|
||||
if running_task is not None:
|
||||
skipped_sources.append(
|
||||
{
|
||||
@@ -219,7 +506,7 @@ async def trigger_all_datasources(
|
||||
)
|
||||
continue
|
||||
|
||||
previous_task_ids[datasource.id] = await get_latest_task_id_for_datasource(datasource.id)
|
||||
previous_task_ids[datasource.id] = None
|
||||
success = run_collector_now(datasource.source)
|
||||
if not success:
|
||||
failed_sources.append(
|
||||
@@ -241,13 +528,24 @@ async def trigger_all_datasources(
|
||||
}
|
||||
)
|
||||
|
||||
latest_task_ids = await _load_latest_task_ids(
|
||||
db,
|
||||
[datasource.id for datasource in datasources],
|
||||
)
|
||||
for datasource_id in previous_task_ids:
|
||||
previous_task_ids[datasource_id] = latest_task_ids.get(datasource_id)
|
||||
|
||||
for _ in range(20):
|
||||
await asyncio.sleep(0.1)
|
||||
pending = [item for item in triggered_sources if item["task_id"] is None]
|
||||
if not pending:
|
||||
break
|
||||
latest_task_ids = await _load_latest_task_ids(
|
||||
db,
|
||||
[item["id"] for item in pending],
|
||||
)
|
||||
for item in pending:
|
||||
task_id = await get_latest_task_id_for_datasource(item["id"])
|
||||
task_id = latest_task_ids.get(item["id"])
|
||||
if task_id is not None and task_id != previous_task_ids.get(item["id"]):
|
||||
item["task_id"] = task_id
|
||||
|
||||
@@ -346,6 +644,7 @@ async def get_datasource_stats(
|
||||
@router.post("/{source_id}/trigger")
|
||||
async def trigger_datasource(
|
||||
source_id: str,
|
||||
force: bool = Query(False),
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
@@ -356,6 +655,26 @@ async def trigger_datasource(
|
||||
if not datasource.is_active:
|
||||
raise HTTPException(status_code=400, detail="Data source is disabled")
|
||||
|
||||
running_task = await get_running_task(db, datasource.id)
|
||||
if running_task is not None and not force:
|
||||
raise HTTPException(
|
||||
status_code=409,
|
||||
detail={
|
||||
"reason": "running_task_in_progress",
|
||||
"message": "当前采集任务尚未完成,重新触发会丢失本次未完成进度。是否强制重新采集?",
|
||||
"task_id": running_task.id,
|
||||
"phase": running_task.phase,
|
||||
"progress": running_task.progress,
|
||||
"records_processed": running_task.records_processed,
|
||||
"total_records": running_task.total_records,
|
||||
},
|
||||
)
|
||||
|
||||
if running_task is not None and force:
|
||||
cancelled = await cancel_running_collector_now(datasource.source)
|
||||
if not cancelled:
|
||||
await rollback_orphaned_running_task(db, datasource, running_task)
|
||||
|
||||
previous_task_id = await get_latest_task_id_for_datasource(datasource.id)
|
||||
success = run_collector_now(datasource.source)
|
||||
if not success:
|
||||
@@ -375,6 +694,7 @@ async def trigger_datasource(
|
||||
"source_id": datasource.id,
|
||||
"task_id": task_id,
|
||||
"collector_name": datasource.source,
|
||||
"force": force,
|
||||
"message": f"Collector '{datasource.source}' has been triggered",
|
||||
}
|
||||
|
||||
@@ -412,6 +732,7 @@ async def clear_datasource_data(
|
||||
async def get_task_status(
|
||||
source_id: str,
|
||||
task_id: Optional[int] = None,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
datasource = await get_datasource_record(db, source_id)
|
||||
|
||||
13
backend/app/api/v1/news.py
Normal file
13
backend/app/api/v1/news.py
Normal file
@@ -0,0 +1,13 @@
|
||||
from fastapi import APIRouter, Query
|
||||
|
||||
from app.services.earth_news import get_earth_news_payload
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
@router.get("/earth-feed")
|
||||
async def get_earth_feed(
|
||||
lat: float | None = Query(None, description="Current Earth view center latitude"),
|
||||
lon: float | None = Query(None, description="Current Earth view center longitude"),
|
||||
):
|
||||
return await get_earth_news_payload(lat=lat, lon=lon)
|
||||
@@ -1,3 +1,4 @@
|
||||
from copy import deepcopy
|
||||
from datetime import UTC, datetime
|
||||
from typing import Optional
|
||||
|
||||
@@ -13,6 +14,7 @@ from app.models.datasource import DataSource
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.models.user import User
|
||||
from app.services.scheduler import sync_datasource_job
|
||||
from app.services.tv_streams import DEFAULT_TV_SETTINGS, get_tv_settings_payload, normalize_tv_settings
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
@@ -36,6 +38,7 @@ DEFAULT_SETTINGS = {
|
||||
"max_login_attempts": 5,
|
||||
"password_policy": "medium",
|
||||
},
|
||||
"tv": DEFAULT_TV_SETTINGS,
|
||||
}
|
||||
|
||||
|
||||
@@ -67,8 +70,34 @@ class CollectorSettingsUpdate(BaseModel):
|
||||
frequency_minutes: int = Field(default=60, ge=1, le=10080)
|
||||
|
||||
|
||||
class TVStreamSourceUpdate(BaseModel):
|
||||
id: str = Field(min_length=1, max_length=100)
|
||||
name: str = Field(min_length=1, max_length=200)
|
||||
provider: str = Field(default="Unknown", max_length=100)
|
||||
region: str = Field(default="Global", max_length=100)
|
||||
language: str = Field(default="und", max_length=32)
|
||||
source_type: str = Field(default="iframe", pattern="^(iframe|hls|video|external|youtube)$")
|
||||
embed_url: str = ""
|
||||
stream_url: str = ""
|
||||
homepage_url: str = ""
|
||||
poster_url: str = ""
|
||||
youtube_video_id: str = ""
|
||||
youtube_channel: str = ""
|
||||
is_enabled: bool = True
|
||||
is_fallback: bool = False
|
||||
sort_order: int = Field(default=10, ge=0, le=9999)
|
||||
collector_source: Optional[str] = None
|
||||
notes: str = ""
|
||||
|
||||
|
||||
class TVSettingsUpdate(BaseModel):
|
||||
default_source_id: str = Field(default=DEFAULT_TV_SETTINGS["default_source_id"], min_length=1)
|
||||
auto_fallback: bool = True
|
||||
sources: list[TVStreamSourceUpdate] = Field(default_factory=list)
|
||||
|
||||
|
||||
def merge_with_defaults(category: str, payload: Optional[dict]) -> dict:
|
||||
merged = DEFAULT_SETTINGS[category].copy()
|
||||
merged = deepcopy(DEFAULT_SETTINGS[category])
|
||||
if payload:
|
||||
merged.update(payload)
|
||||
return merged
|
||||
@@ -79,6 +108,26 @@ async def get_setting_record(db: AsyncSession, category: str) -> Optional[System
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
|
||||
async def get_setting_payloads(db: AsyncSession, categories: list[str]) -> dict[str, dict]:
|
||||
if not categories:
|
||||
return {}
|
||||
|
||||
result = await db.execute(
|
||||
select(SystemSetting).where(SystemSetting.category.in_(categories))
|
||||
)
|
||||
records_by_category = {
|
||||
record.category: record
|
||||
for record in result.scalars().all()
|
||||
}
|
||||
return {
|
||||
category: merge_with_defaults(
|
||||
category,
|
||||
records_by_category.get(category).payload if records_by_category.get(category) else None,
|
||||
)
|
||||
for category in categories
|
||||
}
|
||||
|
||||
|
||||
async def get_setting_payload(db: AsyncSession, category: str) -> dict:
|
||||
record = await get_setting_record(db, category)
|
||||
return merge_with_defaults(category, record.payload if record else None)
|
||||
@@ -175,6 +224,25 @@ async def update_security_settings(
|
||||
return {"status": "updated", "security": payload}
|
||||
|
||||
|
||||
@router.get("/tv")
|
||||
async def get_tv_settings(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return {"tv": await get_tv_settings_payload(db)}
|
||||
|
||||
|
||||
@router.put("/tv")
|
||||
async def update_tv_settings(
|
||||
settings: TVSettingsUpdate,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
payload = normalize_tv_settings(settings.model_dump())
|
||||
saved = await save_setting_payload(db, "tv", payload)
|
||||
return {"status": "updated", "tv": normalize_tv_settings(saved)}
|
||||
|
||||
|
||||
@router.get("/collectors")
|
||||
async def get_collector_settings(
|
||||
current_user: User = Depends(get_current_user),
|
||||
@@ -212,10 +280,15 @@ async def get_all_settings(
|
||||
):
|
||||
result = await db.execute(select(DataSource).order_by(DataSource.module, DataSource.id))
|
||||
datasources = result.scalars().all()
|
||||
setting_payloads = await get_setting_payloads(
|
||||
db,
|
||||
["system", "notifications", "security"],
|
||||
)
|
||||
return {
|
||||
"system": await get_setting_payload(db, "system"),
|
||||
"notifications": await get_setting_payload(db, "notifications"),
|
||||
"security": await get_setting_payload(db, "security"),
|
||||
"system": setting_payloads["system"],
|
||||
"notifications": setting_payloads["notifications"],
|
||||
"security": setting_payloads["security"],
|
||||
"tv": await get_tv_settings_payload(db),
|
||||
"collectors": [serialize_collector(datasource) for datasource in datasources],
|
||||
"generated_at": to_iso8601_utc(datetime.now(UTC)),
|
||||
}
|
||||
|
||||
70
backend/app/api/v1/tv.py
Normal file
70
backend/app/api/v1/tv.py
Normal file
@@ -0,0 +1,70 @@
|
||||
from urllib.parse import quote, urljoin
|
||||
|
||||
import httpx
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query
|
||||
from fastapi.responses import Response
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.db.session import get_db
|
||||
from app.services.tv_streams import get_public_tv_payload, is_allowed_tv_proxy_url
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
@router.get("/streams")
|
||||
async def list_public_tv_streams(
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await get_public_tv_payload(db)
|
||||
|
||||
|
||||
@router.get("/proxy")
|
||||
async def proxy_tv_stream(
|
||||
url: str = Query(..., description="Upstream TV stream or manifest URL"),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
payload = await get_public_tv_payload(db)
|
||||
if not is_allowed_tv_proxy_url(url, payload.get("sources", [])):
|
||||
raise HTTPException(status_code=403, detail="TV proxy target is not allowed")
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(follow_redirects=True, timeout=20.0) as client:
|
||||
upstream = await client.get(
|
||||
url,
|
||||
headers={
|
||||
"User-Agent": "Mozilla/5.0",
|
||||
"Referer": "https://tv.cctv.com/live/cctv4/",
|
||||
},
|
||||
)
|
||||
upstream.raise_for_status()
|
||||
except httpx.HTTPError as exc:
|
||||
raise HTTPException(status_code=502, detail=f"Failed to fetch TV stream: {exc}") from exc
|
||||
|
||||
content_type = upstream.headers.get("content-type", "application/octet-stream")
|
||||
raw_content = upstream.content
|
||||
response_url = str(upstream.url)
|
||||
is_manifest = (
|
||||
response_url.endswith(".m3u8")
|
||||
or "mpegurl" in content_type.lower()
|
||||
or raw_content.lstrip().startswith(b"#EXTM3U")
|
||||
)
|
||||
|
||||
headers = {"Cache-Control": "no-store"}
|
||||
|
||||
if is_manifest:
|
||||
manifest_text = upstream.text
|
||||
rewritten_lines: list[str] = []
|
||||
for line in manifest_text.splitlines():
|
||||
stripped = line.strip()
|
||||
if not stripped or stripped.startswith("#"):
|
||||
rewritten_lines.append(line)
|
||||
continue
|
||||
absolute_url = urljoin(response_url, stripped)
|
||||
rewritten_lines.append(f"/api/v1/tv/proxy?url={quote(absolute_url, safe='')}")
|
||||
return Response(
|
||||
content="\n".join(rewritten_lines),
|
||||
media_type="application/vnd.apple.mpegurl",
|
||||
headers=headers,
|
||||
)
|
||||
|
||||
return Response(content=raw_content, media_type=content_type, headers=headers)
|
||||
@@ -6,7 +6,8 @@ Returns GeoJSON format compatible with Three.js, CesiumJS, and Unreal Cesium.
|
||||
|
||||
from datetime import UTC, datetime
|
||||
import math
|
||||
from fastapi import APIRouter, HTTPException, Depends, Query
|
||||
import httpx
|
||||
from fastapi import APIRouter, HTTPException, Depends, Query, Response
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from sqlalchemy import select, func
|
||||
from typing import List, Dict, Any, Optional
|
||||
@@ -23,6 +24,9 @@ from app.services.cable_graph import build_graph_from_data, CableGraph, haversin
|
||||
from app.services.collectors.bgp_common import RIPE_RIS_COLLECTOR_COORDS
|
||||
|
||||
router = APIRouter()
|
||||
TERRAIN_TILE_URL_TEMPLATE = (
|
||||
"https://s3.amazonaws.com/elevation-tiles-prod/terrarium/{z}/{x}/{y}.png"
|
||||
)
|
||||
|
||||
|
||||
# ============== Converter Functions ==============
|
||||
@@ -205,42 +209,84 @@ def convert_satellite_to_geojson(records: List[CollectedData]) -> Dict[str, Any]
|
||||
return {"type": "FeatureCollection", "features": features}
|
||||
|
||||
|
||||
def dedupe_satellite_records(records: List[CollectedData]) -> List[CollectedData]:
|
||||
"""Keep only the newest record for each satellite identity."""
|
||||
latest_by_key: Dict[str, CollectedData] = {}
|
||||
|
||||
for record in records:
|
||||
metadata = record.extra_data or {}
|
||||
norad_id = metadata.get("norad_cat_id")
|
||||
dedupe_key = (
|
||||
str(norad_id)
|
||||
if norad_id not in (None, "")
|
||||
else str(record.source_id or record.entity_key or record.name or record.id)
|
||||
)
|
||||
|
||||
existing = latest_by_key.get(dedupe_key)
|
||||
if existing is None or (record.id or 0) > (existing.id or 0):
|
||||
latest_by_key[dedupe_key] = record
|
||||
|
||||
return sorted(latest_by_key.values(), key=lambda item: item.id or 0, reverse=True)
|
||||
def _current_collected_data_stmt(source: str):
|
||||
return (
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source == source)
|
||||
.where(CollectedData.is_current.is_(True))
|
||||
.order_by(CollectedData.id.desc())
|
||||
)
|
||||
|
||||
|
||||
def dedupe_collected_records(records: List[CollectedData]) -> List[CollectedData]:
|
||||
"""Keep only the newest record for each collected entity."""
|
||||
latest_by_key: Dict[str, CollectedData] = {}
|
||||
async def _load_current_collected_data(
|
||||
db: AsyncSession,
|
||||
source: str,
|
||||
*,
|
||||
exclude_unknown_name: bool = False,
|
||||
limit: Optional[int] = None,
|
||||
) -> List[CollectedData]:
|
||||
stmt = _current_collected_data_stmt(source)
|
||||
if exclude_unknown_name:
|
||||
stmt = stmt.where(CollectedData.name != "Unknown")
|
||||
if limit is not None:
|
||||
stmt = stmt.limit(limit)
|
||||
|
||||
for record in records:
|
||||
dedupe_key = str(
|
||||
record.source_id
|
||||
or record.entity_key
|
||||
or record.name
|
||||
or record.id
|
||||
)
|
||||
existing = latest_by_key.get(dedupe_key)
|
||||
if existing is None or (record.id or 0) > (existing.id or 0):
|
||||
latest_by_key[dedupe_key] = record
|
||||
result = await db.execute(stmt)
|
||||
return list(result.scalars().all())
|
||||
|
||||
return sorted(latest_by_key.values(), key=lambda item: item.id or 0, reverse=True)
|
||||
|
||||
async def _load_current_collected_data_by_sources(
|
||||
db: AsyncSession,
|
||||
sources: List[str],
|
||||
) -> Dict[str, List[CollectedData]]:
|
||||
if not sources:
|
||||
return {}
|
||||
|
||||
stmt = (
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source.in_(sources))
|
||||
.where(CollectedData.is_current.is_(True))
|
||||
.order_by(CollectedData.source.asc(), CollectedData.id.desc())
|
||||
)
|
||||
result = await db.execute(stmt)
|
||||
|
||||
grouped_records: Dict[str, List[CollectedData]] = {source: [] for source in sources}
|
||||
for record in result.scalars().all():
|
||||
grouped_records.setdefault(record.source, []).append(record)
|
||||
|
||||
return grouped_records
|
||||
|
||||
|
||||
def _build_landing_point_cable_maps(
|
||||
relation_records: List[CollectedData],
|
||||
cable_records: List[CollectedData],
|
||||
) -> tuple[Dict[int, List[int]], Dict[int, str]]:
|
||||
city_to_cable_ids_map: Dict[int, List[int]] = {}
|
||||
for relation_record in relation_records:
|
||||
if not relation_record.extra_data:
|
||||
continue
|
||||
city_id = relation_record.extra_data.get("city_id")
|
||||
cable_id = relation_record.extra_data.get("cable_id")
|
||||
if city_id is None or cable_id is None:
|
||||
continue
|
||||
city_to_cable_ids_map.setdefault(city_id, [])
|
||||
if cable_id not in city_to_cable_ids_map[city_id]:
|
||||
city_to_cable_ids_map[city_id].append(cable_id)
|
||||
|
||||
cable_id_to_name_map: Dict[int, str] = {}
|
||||
for cable_record in cable_records:
|
||||
if not cable_record.extra_data:
|
||||
continue
|
||||
cable_id = cable_record.extra_data.get("cable_id")
|
||||
cable_name = cable_record.name
|
||||
if cable_id and cable_name:
|
||||
cable_id_to_name_map[cable_id] = cable_name
|
||||
|
||||
return city_to_cable_ids_map, cable_id_to_name_map
|
||||
|
||||
|
||||
def _filter_known_records(records: List[CollectedData]) -> List[CollectedData]:
|
||||
return [record for record in records if record.name != "Unknown"]
|
||||
|
||||
|
||||
def convert_supercomputer_to_geojson(records: List[CollectedData]) -> Dict[str, Any]:
|
||||
@@ -722,9 +768,7 @@ def convert_bgp_incidents_to_geojson(
|
||||
async def get_cables_geojson(db: AsyncSession = Depends(get_db)):
|
||||
"""获取海底电缆 GeoJSON 数据 (LineString)"""
|
||||
try:
|
||||
stmt = select(CollectedData).where(CollectedData.source == "arcgis_cables")
|
||||
result = await db.execute(stmt)
|
||||
records = dedupe_collected_records(list(result.scalars().all()))
|
||||
records = await _load_current_collected_data(db, "arcgis_cables")
|
||||
|
||||
if not records:
|
||||
raise HTTPException(
|
||||
@@ -742,36 +786,25 @@ async def get_cables_geojson(db: AsyncSession = Depends(get_db)):
|
||||
@router.get("/geo/landing-points")
|
||||
async def get_landing_points_geojson(db: AsyncSession = Depends(get_db)):
|
||||
try:
|
||||
landing_stmt = select(CollectedData).where(CollectedData.source == "arcgis_landing_points")
|
||||
landing_result = await db.execute(landing_stmt)
|
||||
records = dedupe_collected_records(list(landing_result.scalars().all()))
|
||||
|
||||
relation_stmt = select(CollectedData).where(CollectedData.source == "arcgis_cable_landing_relation")
|
||||
relation_result = await db.execute(relation_stmt)
|
||||
relation_records = dedupe_collected_records(list(relation_result.scalars().all()))
|
||||
|
||||
cable_stmt = select(CollectedData).where(CollectedData.source == "arcgis_cables")
|
||||
cable_result = await db.execute(cable_stmt)
|
||||
cable_records = dedupe_collected_records(list(cable_result.scalars().all()))
|
||||
|
||||
city_to_cable_ids_map = {}
|
||||
for rel in relation_records:
|
||||
if rel.extra_data:
|
||||
city_id = rel.extra_data.get("city_id")
|
||||
cable_id = rel.extra_data.get("cable_id")
|
||||
if city_id is not None and cable_id is not None:
|
||||
if city_id not in city_to_cable_ids_map:
|
||||
city_to_cable_ids_map[city_id] = []
|
||||
if cable_id not in city_to_cable_ids_map[city_id]:
|
||||
city_to_cable_ids_map[city_id].append(cable_id)
|
||||
|
||||
cable_id_to_name_map = {}
|
||||
for cable in cable_records:
|
||||
if cable.extra_data:
|
||||
cable_id = cable.extra_data.get("cable_id")
|
||||
cable_name = cable.name
|
||||
if cable_id and cable_name:
|
||||
cable_id_to_name_map[cable_id] = cable_name
|
||||
records_by_source = await _load_current_collected_data_by_sources(
|
||||
db,
|
||||
[
|
||||
"arcgis_landing_points",
|
||||
"arcgis_cable_landing_relation",
|
||||
"arcgis_cables",
|
||||
],
|
||||
)
|
||||
records = records_by_source.get("arcgis_landing_points", [])
|
||||
relation_records = records_by_source.get(
|
||||
"arcgis_cable_landing_relation",
|
||||
[],
|
||||
)
|
||||
cable_records = records_by_source.get("arcgis_cables", [])
|
||||
|
||||
city_to_cable_ids_map, cable_id_to_name_map = _build_landing_point_cable_maps(
|
||||
relation_records,
|
||||
cable_records,
|
||||
)
|
||||
|
||||
if not records:
|
||||
raise HTTPException(
|
||||
@@ -786,38 +819,67 @@ async def get_landing_points_geojson(db: AsyncSession = Depends(get_db)):
|
||||
raise HTTPException(status_code=500, detail=f"Internal error: {str(e)}")
|
||||
|
||||
|
||||
@router.get("/terrain/terrarium/{z}/{x}/{y}.png")
|
||||
async def get_terrarium_tile(z: int, x: int, y: int):
|
||||
"""Proxy Terrarium elevation tiles through the backend to avoid browser CORS issues."""
|
||||
if z < 0 or x < 0 or y < 0:
|
||||
raise HTTPException(status_code=400, detail="Invalid terrain tile coordinates")
|
||||
|
||||
url = TERRAIN_TILE_URL_TEMPLATE.format(z=z, x=x, y=y)
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(
|
||||
timeout=20.0,
|
||||
follow_redirects=True,
|
||||
) as client:
|
||||
upstream = await client.get(url)
|
||||
upstream.raise_for_status()
|
||||
except httpx.HTTPStatusError as exc:
|
||||
raise HTTPException(
|
||||
status_code=exc.response.status_code,
|
||||
detail=f"Terrain tile upstream error: {exc.response.status_code}",
|
||||
) from exc
|
||||
except httpx.HTTPError as exc:
|
||||
raise HTTPException(
|
||||
status_code=502,
|
||||
detail=f"Terrain tile fetch failed: {exc}",
|
||||
) from exc
|
||||
|
||||
cache_control = upstream.headers.get("cache-control") or "public, max-age=86400"
|
||||
etag = upstream.headers.get("etag")
|
||||
last_modified = upstream.headers.get("last-modified")
|
||||
headers = {
|
||||
"Cache-Control": cache_control,
|
||||
}
|
||||
if etag:
|
||||
headers["ETag"] = etag
|
||||
if last_modified:
|
||||
headers["Last-Modified"] = last_modified
|
||||
|
||||
return Response(
|
||||
content=upstream.content,
|
||||
media_type=upstream.headers.get("content-type", "image/png"),
|
||||
headers=headers,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/geo/all")
|
||||
async def get_all_geojson(db: AsyncSession = Depends(get_db)):
|
||||
cables_stmt = select(CollectedData).where(CollectedData.source == "arcgis_cables")
|
||||
cables_result = await db.execute(cables_stmt)
|
||||
cables_records = dedupe_collected_records(list(cables_result.scalars().all()))
|
||||
|
||||
points_stmt = select(CollectedData).where(CollectedData.source == "arcgis_landing_points")
|
||||
points_result = await db.execute(points_stmt)
|
||||
points_records = dedupe_collected_records(list(points_result.scalars().all()))
|
||||
|
||||
relation_stmt = select(CollectedData).where(CollectedData.source == "arcgis_cable_landing_relation")
|
||||
relation_result = await db.execute(relation_stmt)
|
||||
relation_records = dedupe_collected_records(list(relation_result.scalars().all()))
|
||||
|
||||
city_to_cable_ids_map = {}
|
||||
for rel in relation_records:
|
||||
if rel.extra_data:
|
||||
city_id = rel.extra_data.get("city_id")
|
||||
cable_id = rel.extra_data.get("cable_id")
|
||||
if city_id is not None and cable_id is not None:
|
||||
if city_id not in city_to_cable_ids_map:
|
||||
city_to_cable_ids_map[city_id] = []
|
||||
if cable_id not in city_to_cable_ids_map[city_id]:
|
||||
city_to_cable_ids_map[city_id].append(cable_id)
|
||||
|
||||
cable_id_to_name_map = {}
|
||||
for cable in cables_records:
|
||||
if cable.extra_data:
|
||||
cable_id = cable.extra_data.get("cable_id")
|
||||
cable_name = cable.name
|
||||
if cable_id and cable_name:
|
||||
cable_id_to_name_map[cable_id] = cable_name
|
||||
records_by_source = await _load_current_collected_data_by_sources(
|
||||
db,
|
||||
[
|
||||
"arcgis_cables",
|
||||
"arcgis_landing_points",
|
||||
"arcgis_cable_landing_relation",
|
||||
],
|
||||
)
|
||||
cables_records = records_by_source.get("arcgis_cables", [])
|
||||
points_records = records_by_source.get("arcgis_landing_points", [])
|
||||
relation_records = records_by_source.get("arcgis_cable_landing_relation", [])
|
||||
city_to_cable_ids_map, cable_id_to_name_map = _build_landing_point_cable_maps(
|
||||
relation_records,
|
||||
cables_records,
|
||||
)
|
||||
|
||||
cables = (
|
||||
convert_cable_to_geojson(cables_records)
|
||||
@@ -850,17 +912,12 @@ async def get_satellites_geojson(
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""获取卫星 TLE GeoJSON 数据"""
|
||||
stmt = (
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source == "celestrak_tle")
|
||||
.where(CollectedData.name != "Unknown")
|
||||
.order_by(CollectedData.id.desc())
|
||||
records = await _load_current_collected_data(
|
||||
db,
|
||||
"celestrak_tle",
|
||||
exclude_unknown_name=True,
|
||||
limit=limit,
|
||||
)
|
||||
result = await db.execute(stmt)
|
||||
records = dedupe_satellite_records(list(result.scalars().all()))
|
||||
|
||||
if limit is not None:
|
||||
records = records[:limit]
|
||||
|
||||
if not records:
|
||||
return {"type": "FeatureCollection", "features": [], "count": 0}
|
||||
@@ -878,15 +935,12 @@ async def get_supercomputers_geojson(
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""获取 TOP500 超算中心 GeoJSON 数据"""
|
||||
stmt = (
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source == "top500")
|
||||
.where(CollectedData.name != "Unknown")
|
||||
.order_by(CollectedData.id.desc())
|
||||
records = await _load_current_collected_data(
|
||||
db,
|
||||
"top500",
|
||||
exclude_unknown_name=True,
|
||||
limit=limit,
|
||||
)
|
||||
result = await db.execute(stmt)
|
||||
records = dedupe_collected_records(list(result.scalars().all()))
|
||||
records = records[:limit]
|
||||
|
||||
if not records:
|
||||
return {"type": "FeatureCollection", "features": [], "count": 0}
|
||||
@@ -904,15 +958,12 @@ async def get_gpu_clusters_geojson(
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""获取 GPU 集群 GeoJSON 数据"""
|
||||
stmt = (
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source == "epoch_ai_gpu")
|
||||
.where(CollectedData.name != "Unknown")
|
||||
.order_by(CollectedData.id.desc())
|
||||
records = await _load_current_collected_data(
|
||||
db,
|
||||
"epoch_ai_gpu",
|
||||
exclude_unknown_name=True,
|
||||
limit=limit,
|
||||
)
|
||||
result = await db.execute(stmt)
|
||||
records = dedupe_collected_records(list(result.scalars().all()))
|
||||
records = records[:limit]
|
||||
|
||||
if not records:
|
||||
return {"type": "FeatureCollection", "features": [], "count": 0}
|
||||
@@ -990,37 +1041,27 @@ async def get_all_visualization_data(db: AsyncSession = Depends(get_db)):
|
||||
- supercomputers: TOP500 超算
|
||||
- gpu_clusters: GPU 集群
|
||||
"""
|
||||
cables_stmt = select(CollectedData).where(CollectedData.source == "arcgis_cables")
|
||||
cables_result = await db.execute(cables_stmt)
|
||||
cables_records = dedupe_collected_records(list(cables_result.scalars().all()))
|
||||
|
||||
points_stmt = select(CollectedData).where(CollectedData.source == "arcgis_landing_points")
|
||||
points_result = await db.execute(points_stmt)
|
||||
points_records = dedupe_collected_records(list(points_result.scalars().all()))
|
||||
|
||||
satellites_stmt = (
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source == "celestrak_tle")
|
||||
.where(CollectedData.name != "Unknown")
|
||||
records_by_source = await _load_current_collected_data_by_sources(
|
||||
db,
|
||||
[
|
||||
"arcgis_cables",
|
||||
"arcgis_landing_points",
|
||||
"celestrak_tle",
|
||||
"top500",
|
||||
"epoch_ai_gpu",
|
||||
],
|
||||
)
|
||||
satellites_result = await db.execute(satellites_stmt)
|
||||
satellites_records = dedupe_satellite_records(list(satellites_result.scalars().all()))
|
||||
|
||||
supercomputers_stmt = (
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source == "top500")
|
||||
.where(CollectedData.name != "Unknown")
|
||||
cables_records = records_by_source.get("arcgis_cables", [])
|
||||
points_records = records_by_source.get("arcgis_landing_points", [])
|
||||
satellites_records = _filter_known_records(
|
||||
records_by_source.get("celestrak_tle", []),
|
||||
)
|
||||
supercomputers_result = await db.execute(supercomputers_stmt)
|
||||
supercomputers_records = dedupe_collected_records(list(supercomputers_result.scalars().all()))
|
||||
|
||||
gpu_stmt = (
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source == "epoch_ai_gpu")
|
||||
.where(CollectedData.name != "Unknown")
|
||||
supercomputers_records = _filter_known_records(
|
||||
records_by_source.get("top500", []),
|
||||
)
|
||||
gpu_records = _filter_known_records(
|
||||
records_by_source.get("epoch_ai_gpu", []),
|
||||
)
|
||||
gpu_result = await db.execute(gpu_stmt)
|
||||
gpu_records = dedupe_collected_records(list(gpu_result.scalars().all()))
|
||||
|
||||
cables = (
|
||||
convert_cable_to_geojson(cables_records)
|
||||
@@ -1084,13 +1125,8 @@ async def get_cable_graph(db: AsyncSession) -> CableGraph:
|
||||
global _cable_graph
|
||||
|
||||
if _cable_graph is None:
|
||||
cables_stmt = select(CollectedData).where(CollectedData.source == "arcgis_cables")
|
||||
cables_result = await db.execute(cables_stmt)
|
||||
cables_records = list(cables_result.scalars().all())
|
||||
|
||||
points_stmt = select(CollectedData).where(CollectedData.source == "arcgis_landing_points")
|
||||
points_result = await db.execute(points_stmt)
|
||||
points_records = list(points_result.scalars().all())
|
||||
cables_records = await _load_current_collected_data(db, "arcgis_cables")
|
||||
points_records = await _load_current_collected_data(db, "arcgis_landing_points")
|
||||
|
||||
cables_data = convert_cable_to_geojson(cables_records)
|
||||
points_data = convert_landing_point_to_geojson(points_records)
|
||||
|
||||
@@ -11,6 +11,7 @@ COLLECTOR_URL_KEYS = {
|
||||
"fao_landing_points": "fao.landing_point_url",
|
||||
"telegeography_cables": "telegeography.cable_url",
|
||||
"telegeography_landing": "telegeography.landing_point_url",
|
||||
"telegeography_systems": "telegeography.cable_url",
|
||||
"huggingface_models": "huggingface.models_url",
|
||||
"huggingface_datasets": "huggingface.datasets_url",
|
||||
"huggingface_spaces": "huggingface.spaces_url",
|
||||
@@ -23,6 +24,7 @@ COLLECTOR_URL_KEYS = {
|
||||
"top500": "top500.url",
|
||||
"epoch_ai_gpu": "epoch_ai.gpu_clusters_url",
|
||||
"spacetrack_tle": "spacetrack.tle_query_url",
|
||||
"celestrak_tle": "celestrak.base_url",
|
||||
"ris_live_bgp": "ris_live.url",
|
||||
"bgpstream_bgp": "bgpstream.url",
|
||||
"iptoasn_prefix_geo": "iptoasn.combined_url",
|
||||
@@ -41,18 +43,22 @@ class DataSourcesConfig:
|
||||
with open(config_path, "r") as f:
|
||||
self._yaml_config = yaml.safe_load(f) or {}
|
||||
|
||||
def get_yaml_url(self, collector_name: str) -> str:
|
||||
key = COLLECTOR_URL_KEYS.get(collector_name, "")
|
||||
def get_yaml_value(self, key: str):
|
||||
if not key:
|
||||
return ""
|
||||
return None
|
||||
|
||||
parts = key.split(".")
|
||||
value = self._yaml_config
|
||||
for part in parts:
|
||||
if isinstance(value, dict):
|
||||
value = value.get(part, "")
|
||||
value = value.get(part)
|
||||
else:
|
||||
return ""
|
||||
return None
|
||||
return value
|
||||
|
||||
def get_yaml_url(self, collector_name: str) -> str:
|
||||
key = COLLECTOR_URL_KEYS.get(collector_name, "")
|
||||
value = self.get_yaml_value(key)
|
||||
return value if isinstance(value, str) else ""
|
||||
|
||||
async def get_url(self, collector_name: str, db) -> str:
|
||||
|
||||
@@ -2,53 +2,87 @@
|
||||
# All external data source URLs should be configured here
|
||||
|
||||
arcgis:
|
||||
# ArcGIS 海缆 GeoJSON 查询接口
|
||||
cable_url: "https://services.arcgis.com/6DIQcwlPy8knb6sg/ArcGIS/rest/services/SubmarineCables/FeatureServer/2/query"
|
||||
# ArcGIS 登陆点 GeoJSON 查询接口
|
||||
landing_point_url: "https://services.arcgis.com/6DIQcwlPy8knb6sg/ArcGIS/rest/services/SubmarineCables/FeatureServer/1/query"
|
||||
# ArcGIS 海缆与登陆点关联关系查询接口
|
||||
cable_landing_relation_url: "https://services.arcgis.com/6DIQcwlPy8knb6sg/ArcGIS/rest/services/SubmarineCables/FeatureServer/3/query"
|
||||
|
||||
fao:
|
||||
# FAO 登陆点 CSV 下载地址
|
||||
landing_point_url: "https://data.apps.fao.org/catalog/dataset/1b75ff21-92f2-4b96-9b7b-98e8aa65ad5d/resource/b6071077-d1d4-4e97-aa00-42e902847c87/download/landing-point-geo.csv"
|
||||
|
||||
telegeography:
|
||||
# TeleGeography 海缆/系统主数据源,当前使用 GitHub 镜像 JSON
|
||||
cable_url: "https://raw.githubusercontent.com/lintaojlu/submarine_cable_information/main/cable.json"
|
||||
# TeleGeography 登陆点主数据源,当前使用 GitHub 镜像 JSON
|
||||
landing_point_url: "https://raw.githubusercontent.com/lintaojlu/submarine_cable_information/main/landing_point.json"
|
||||
# TeleGeography 历史 API 存档,用于 cable collector 的 fallback
|
||||
archived_cable_url: "https://web.archive.org/web/2024/https://www.submarinecablemap.com/api/v3/cable"
|
||||
# TeleGeography 官网页面,用于 cable collector 的最终 HTML 抓取 fallback
|
||||
live_map_url: "https://www.submarinecablemap.com"
|
||||
|
||||
huggingface:
|
||||
# Hugging Face 模型目录 API
|
||||
models_url: "https://huggingface.co/api/models"
|
||||
# Hugging Face 数据集目录 API
|
||||
datasets_url: "https://huggingface.co/api/datasets"
|
||||
# Hugging Face Spaces 目录 API
|
||||
spaces_url: "https://huggingface.co/api/spaces"
|
||||
|
||||
cloudflare:
|
||||
# Cloudflare Radar 设备类型摘要接口
|
||||
radar_device_url: "https://api.cloudflare.com/client/v4/radar/http/summary/device_type"
|
||||
# Cloudflare Radar 请求量时间序列接口
|
||||
radar_traffic_url: "https://api.cloudflare.com/client/v4/radar/http/timeseries/requests"
|
||||
# Cloudflare Radar 热点地理位置接口
|
||||
radar_top_locations_url: "https://api.cloudflare.com/client/v4/radar/http/top/locations"
|
||||
|
||||
peeringdb:
|
||||
# PeeringDB IXP API
|
||||
ixp_url: "https://www.peeringdb.com/api/ix"
|
||||
# PeeringDB Network API
|
||||
network_url: "https://www.peeringdb.com/api/net"
|
||||
# PeeringDB Facility API
|
||||
facility_url: "https://www.peeringdb.com/api/fac"
|
||||
|
||||
top500:
|
||||
# TOP500 榜单页面,用于主表抓取
|
||||
url: "https://top500.org/lists/top500/list/2025/11/"
|
||||
# TOP500 站点根地址,用于拼详情页链接
|
||||
base_url: "https://top500.org"
|
||||
|
||||
epoch_ai:
|
||||
# Epoch AI GPU Cluster 页面
|
||||
gpu_clusters_url: "https://epoch.ai/data/gpu-clusters"
|
||||
|
||||
spacetrack:
|
||||
# Space-Track 站点根地址,用于首页访问和登录地址推导
|
||||
base_url: "https://www.space-track.org"
|
||||
# Space-Track TLE 主查询接口
|
||||
tle_query_url: "https://www.space-track.org/basicspacedata/query/class/gp/orderby/EPOCH%20desc/limit/1000/format/json"
|
||||
|
||||
celestrak:
|
||||
# CelesTrak TLE 基础接口,collector 会在其后拼接 GROUP / FORMAT 参数
|
||||
base_url: "https://celestrak.org/NORAD/elements/gp.php"
|
||||
|
||||
ris_live:
|
||||
# RIPE RIS Live 流式订阅地址
|
||||
url: "https://ris-live.ripe.net/v1/stream/?format=json&client=planet-ris-live"
|
||||
|
||||
bgpstream:
|
||||
# CAIDA BGPStream Broker API
|
||||
url: "https://broker.bgpstream.caida.org/v2"
|
||||
|
||||
iptoasn:
|
||||
# IPtoASN prefix geography 合并数据下载地址
|
||||
combined_url: "https://iptoasn.com/data/ip2asn-combined.tsv.gz"
|
||||
|
||||
opengeofeed:
|
||||
# OpenGeoFeed 公共 geofeed CSV
|
||||
public_csv_url: "https://opengeofeed.org/feed/public.csv"
|
||||
|
||||
nro:
|
||||
# NRO delegated stats 下载地址
|
||||
delegated_stats_url: "https://ftp.ripe.net/pub/stats/ripencc/nro-stats/latest/nro-delegated-stats"
|
||||
|
||||
@@ -155,6 +155,13 @@ DEFAULT_DATASOURCES = {
|
||||
"priority": "P1",
|
||||
"frequency_minutes": 1440,
|
||||
},
|
||||
"news_live_streams": {
|
||||
"id": 26,
|
||||
"name": "News Live Streams",
|
||||
"module": "L4",
|
||||
"priority": "P2",
|
||||
"frequency_minutes": 720,
|
||||
},
|
||||
}
|
||||
|
||||
ID_TO_COLLECTOR = {info["id"]: name for name, info in DEFAULT_DATASOURCES.items()}
|
||||
|
||||
@@ -95,6 +95,8 @@ async def init_db():
|
||||
import app.models.bgp_observation # noqa: F401
|
||||
import app.models.collected_data # noqa: F401
|
||||
import app.models.system_setting # noqa: F401
|
||||
import app.models.playground_session # noqa: F401
|
||||
import app.models.playground_message # noqa: F401
|
||||
|
||||
async with engine.begin() as conn:
|
||||
await conn.run_sync(Base.metadata.create_all)
|
||||
|
||||
@@ -9,6 +9,8 @@ from app.models.bgp_anomaly import BGPAnomaly
|
||||
from app.models.bgp_incident import BGPIncident
|
||||
from app.models.bgp_observation import BGPObservation
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.models.playground_session import PlaygroundSession
|
||||
from app.models.playground_message import PlaygroundMessage
|
||||
|
||||
__all__ = [
|
||||
"User",
|
||||
|
||||
40
backend/app/models/playground_message.py
Normal file
40
backend/app/models/playground_message.py
Normal file
@@ -0,0 +1,40 @@
|
||||
from sqlalchemy import JSON, Boolean, Column, DateTime, ForeignKey, Integer, String, Text
|
||||
from sqlalchemy.sql import func
|
||||
|
||||
from app.db.session import Base
|
||||
|
||||
|
||||
class PlaygroundMessage(Base):
|
||||
__tablename__ = "playground_messages"
|
||||
|
||||
id = Column(Integer, primary_key=True, autoincrement=True)
|
||||
public_id = Column(String(64), unique=True, index=True, nullable=False)
|
||||
session_id = Column(Integer, ForeignKey("playground_sessions.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
user_id = Column(Integer, ForeignKey("users.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
parent_message_id = Column(Integer, ForeignKey("playground_messages.id", ondelete="SET NULL"), nullable=True)
|
||||
role = Column(String(20), nullable=False)
|
||||
kind = Column(String(20), nullable=False, default="message")
|
||||
status = Column(String(20), nullable=False, default="done")
|
||||
title = Column(String(255), nullable=True)
|
||||
content = Column(Text, nullable=False, default="")
|
||||
thinking_content = Column(Text, nullable=False, default="")
|
||||
meta = Column(JSON, nullable=False, default=list)
|
||||
provider = Column(String(100), nullable=True)
|
||||
model = Column(String(200), nullable=True)
|
||||
request_id = Column(String(100), nullable=True)
|
||||
raw_response = Column(JSON, nullable=False, default=dict)
|
||||
content_blocks = Column(JSON, nullable=False, default=list)
|
||||
text_blocks = Column(JSON, nullable=False, default=list)
|
||||
thinking_blocks = Column(JSON, nullable=False, default=list)
|
||||
sort_order = Column(Integer, nullable=False, default=0, index=True)
|
||||
is_visible = Column(Boolean, nullable=False, default=True)
|
||||
created_at = Column(DateTime(timezone=True), server_default=func.now(), nullable=False)
|
||||
updated_at = Column(
|
||||
DateTime(timezone=True),
|
||||
server_default=func.now(),
|
||||
onupdate=func.now(),
|
||||
nullable=False,
|
||||
)
|
||||
|
||||
def __repr__(self):
|
||||
return f"<PlaygroundMessage public_id={self.public_id} role={self.role} status={self.status}>"
|
||||
27
backend/app/models/playground_session.py
Normal file
27
backend/app/models/playground_session.py
Normal file
@@ -0,0 +1,27 @@
|
||||
from sqlalchemy import JSON, Column, DateTime, ForeignKey, Integer, String, UniqueConstraint
|
||||
from sqlalchemy.sql import func
|
||||
|
||||
from app.db.session import Base
|
||||
|
||||
|
||||
class PlaygroundSession(Base):
|
||||
__tablename__ = "playground_sessions"
|
||||
__table_args__ = (
|
||||
UniqueConstraint("user_id", "session_key", name="uq_playground_sessions_user_session_key"),
|
||||
)
|
||||
|
||||
id = Column(Integer, primary_key=True, autoincrement=True)
|
||||
user_id = Column(Integer, ForeignKey("users.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
session_key = Column(String(100), nullable=False, default="default")
|
||||
title = Column(String(200), nullable=False, default="Playground 会话")
|
||||
state = Column(JSON, nullable=False, default={})
|
||||
created_at = Column(DateTime(timezone=True), server_default=func.now(), nullable=False)
|
||||
updated_at = Column(
|
||||
DateTime(timezone=True),
|
||||
server_default=func.now(),
|
||||
onupdate=func.now(),
|
||||
nullable=False,
|
||||
)
|
||||
|
||||
def __repr__(self):
|
||||
return f"<PlaygroundSession user_id={self.user_id} session_key={self.session_key}>"
|
||||
@@ -3,6 +3,14 @@ from typing import Any
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class AIContentBlock(BaseModel):
|
||||
type: str
|
||||
text: str | None = None
|
||||
thinking: str | None = None
|
||||
signature: str | None = None
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class SituationalAnalysisRequest(BaseModel):
|
||||
title: str = Field(..., min_length=1, max_length=200)
|
||||
objective: str = Field(..., min_length=1, max_length=1000)
|
||||
@@ -10,18 +18,158 @@ class SituationalAnalysisRequest(BaseModel):
|
||||
observations: list[str] = Field(default_factory=list)
|
||||
constraints: list[str] = Field(default_factory=list)
|
||||
preferred_model: str | None = Field(default=None, max_length=200)
|
||||
thinking: dict[str, Any] | None = None
|
||||
|
||||
|
||||
class BGPBriefRequest(BaseModel):
|
||||
incident_limit: int = Field(default=5, ge=1, le=10)
|
||||
anomaly_limit: int = Field(default=6, ge=1, le=12)
|
||||
collector_limit: int = Field(default=5, ge=1, le=10)
|
||||
preferred_model: str | None = Field(default=None, max_length=200)
|
||||
thinking: dict[str, Any] | None = None
|
||||
|
||||
|
||||
class AlertBriefRequest(BaseModel):
|
||||
alert_limit: int = Field(default=8, ge=1, le=20)
|
||||
preferred_model: str | None = Field(default=None, max_length=200)
|
||||
thinking: dict[str, Any] | None = None
|
||||
|
||||
|
||||
class SituationalAlertBriefRequest(BaseModel):
|
||||
preferred_model: str | None = Field(default=None, max_length=200)
|
||||
thinking: dict[str, Any] | None = None
|
||||
|
||||
|
||||
class SituationalAnalysisResponse(BaseModel):
|
||||
provider: str
|
||||
model: str
|
||||
content: str
|
||||
content_blocks: list[AIContentBlock] = Field(default_factory=list)
|
||||
text_blocks: list[str] = Field(default_factory=list)
|
||||
thinking_blocks: list[str] = Field(default_factory=list)
|
||||
raw_response: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class BGPBriefRecordSummary(BaseModel):
|
||||
id: str
|
||||
title: str
|
||||
provider: str
|
||||
model: str
|
||||
request_id: str | None = None
|
||||
generated_at: str
|
||||
|
||||
|
||||
class BGPBriefRecordResponse(BGPBriefRecordSummary):
|
||||
content_markdown: str
|
||||
facts: list[str] = Field(default_factory=list)
|
||||
context: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class AlertBriefResponse(SituationalAnalysisResponse):
|
||||
title: str
|
||||
objective: str
|
||||
facts: list[str] = Field(default_factory=list)
|
||||
context: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class SituationalAlertBriefResponse(SituationalAnalysisResponse):
|
||||
title: str
|
||||
objective: str
|
||||
facts: list[str] = Field(default_factory=list)
|
||||
context: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class AIProviderStatusResponse(BaseModel):
|
||||
provider: str
|
||||
api: str | None = None
|
||||
enabled: bool
|
||||
configured: bool
|
||||
model: str | None = None
|
||||
base_url: str | None = None
|
||||
|
||||
|
||||
class PlaygroundSessionState(BaseModel):
|
||||
messages: list[dict[str, Any]] = Field(default_factory=list)
|
||||
selectedPresetKey: str = Field(default="bgp-brief", max_length=100)
|
||||
title: str = Field(default="", max_length=200)
|
||||
objective: str = Field(default="", max_length=1000)
|
||||
constraints: str = Field(default="")
|
||||
inputValue: str = Field(default="")
|
||||
analysis: dict[str, Any] | None = None
|
||||
latestAnalysisMessageId: str | None = Field(default=None, max_length=200)
|
||||
analysisMeta: dict[str, Any] = Field(default_factory=dict)
|
||||
helpExpanded: bool = True
|
||||
|
||||
|
||||
class PlaygroundSessionUpsertRequest(BaseModel):
|
||||
session_key: str = Field(default="default", min_length=1, max_length=100)
|
||||
title: str | None = Field(default=None, max_length=200)
|
||||
state: PlaygroundSessionState
|
||||
|
||||
|
||||
class PlaygroundMessageRecord(BaseModel):
|
||||
id: str
|
||||
role: str
|
||||
kind: str = "message"
|
||||
status: str = "done"
|
||||
title: str | None = None
|
||||
content: str = ""
|
||||
thinking_content: str = ""
|
||||
meta: list[str] = Field(default_factory=list)
|
||||
markdown: bool = True
|
||||
provider: str | None = None
|
||||
model: str | None = None
|
||||
request_id: str | None = None
|
||||
raw_response: dict[str, Any] = Field(default_factory=dict)
|
||||
content_blocks: list[dict[str, Any]] = Field(default_factory=list)
|
||||
text_blocks: list[str] = Field(default_factory=list)
|
||||
thinking_blocks: list[str] = Field(default_factory=list)
|
||||
parent_message_id: str | None = None
|
||||
created_at: str
|
||||
updated_at: str
|
||||
|
||||
|
||||
class PlaygroundSessionResponse(BaseModel):
|
||||
id: str
|
||||
session_key: str
|
||||
title: str
|
||||
state: PlaygroundSessionState
|
||||
created_at: str
|
||||
updated_at: str
|
||||
|
||||
|
||||
class PlaygroundThreadResponse(BaseModel):
|
||||
session: PlaygroundSessionResponse
|
||||
messages: list[PlaygroundMessageRecord] = Field(default_factory=list)
|
||||
|
||||
|
||||
class PlaygroundMessageCreateRequest(BaseModel):
|
||||
session_key: str = Field(default="default", min_length=1, max_length=100)
|
||||
title: str = Field(..., min_length=1, max_length=200)
|
||||
objective: str = Field(..., min_length=1, max_length=1000)
|
||||
constraints: str = Field(default="")
|
||||
input: str = Field(..., min_length=1)
|
||||
selected_preset_key: str = Field(default="bgp-brief", max_length=100)
|
||||
help_expanded: bool = True
|
||||
|
||||
|
||||
class PlaygroundMessageActionResponse(BaseModel):
|
||||
session: PlaygroundSessionResponse
|
||||
messages: list[PlaygroundMessageRecord] = Field(default_factory=list)
|
||||
active_message_id: str | None = None
|
||||
|
||||
|
||||
class PlaygroundMessageStopRequest(BaseModel):
|
||||
session_key: str = Field(default="default", min_length=1, max_length=100)
|
||||
message_id: str = Field(..., min_length=1, max_length=64)
|
||||
|
||||
|
||||
class PlaygroundMessageResendRequest(BaseModel):
|
||||
session_key: str = Field(default="default", min_length=1, max_length=100)
|
||||
user_message_id: str = Field(..., min_length=1, max_length=64)
|
||||
|
||||
|
||||
class PlaygroundMessageEditRequest(BaseModel):
|
||||
session_key: str = Field(default="default", min_length=1, max_length=100)
|
||||
user_message_id: str = Field(..., min_length=1, max_length=64)
|
||||
content: str = Field(..., min_length=1)
|
||||
|
||||
5
backend/app/schemas/alert.py
Normal file
5
backend/app/schemas/alert.py
Normal file
@@ -0,0 +1,5 @@
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class AlertResolutionRequest(BaseModel):
|
||||
resolution: str = Field(..., min_length=1, max_length=1000)
|
||||
103
backend/app/services/alert_ai_brief.py
Normal file
103
backend/app/services/alert_ai_brief.py
Normal file
@@ -0,0 +1,103 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections import Counter
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.models.alert import Alert, AlertSeverity, AlertStatus
|
||||
from app.schemas.ai import AlertBriefRequest, SituationalAnalysisRequest
|
||||
|
||||
|
||||
def _format_counter(counter: Counter[str], empty_text: str = "无") -> str:
|
||||
if not counter:
|
||||
return empty_text
|
||||
return ",".join(f"{key} {value}" for key, value in counter.items())
|
||||
|
||||
|
||||
async def build_alert_brief_request(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
alert_limit: int = 8,
|
||||
) -> tuple[SituationalAnalysisRequest, list[str], dict[str, Any]]:
|
||||
recent_alerts_result = await db.execute(
|
||||
select(Alert)
|
||||
.order_by(Alert.created_at.desc(), Alert.id.desc())
|
||||
.limit(max(alert_limit, 1))
|
||||
)
|
||||
total_result = await db.execute(select(func.count(Alert.id)))
|
||||
active_result = await db.execute(select(func.count(Alert.id)).where(Alert.status == AlertStatus.ACTIVE))
|
||||
acknowledged_result = await db.execute(
|
||||
select(func.count(Alert.id)).where(Alert.status == AlertStatus.ACKNOWLEDGED)
|
||||
)
|
||||
resolved_result = await db.execute(select(func.count(Alert.id)).where(Alert.status == AlertStatus.RESOLVED))
|
||||
|
||||
recent_alerts = recent_alerts_result.scalars().all()
|
||||
total_alerts = total_result.scalar() or 0
|
||||
active_alerts = active_result.scalar() or 0
|
||||
acknowledged_alerts = acknowledged_result.scalar() or 0
|
||||
resolved_alerts = resolved_result.scalar() or 0
|
||||
|
||||
severity_counts = Counter((item.severity.value if item.severity else "unknown") for item in recent_alerts)
|
||||
status_counts = Counter((item.status.value if item.status else "unknown") for item in recent_alerts)
|
||||
datasource_counts = Counter((item.datasource_name or "未命名数据源") for item in recent_alerts)
|
||||
active_datasource_counts = Counter(
|
||||
(item.datasource_name or "未命名数据源")
|
||||
for item in recent_alerts
|
||||
if item.status == AlertStatus.ACTIVE
|
||||
)
|
||||
|
||||
facts = [
|
||||
f"告警总量 {total_alerts} 条,其中 active {active_alerts} 条、acknowledged {acknowledged_alerts} 条、resolved {resolved_alerts} 条。",
|
||||
f"最近告警严重度分布:{_format_counter(severity_counts)}。",
|
||||
f"最近告警状态分布:{_format_counter(status_counts)}。",
|
||||
f"最近告警数据源分布:{_format_counter(Counter(dict(datasource_counts.most_common(6))))}。",
|
||||
]
|
||||
|
||||
if active_datasource_counts:
|
||||
facts.append(
|
||||
"当前待处理告警主要集中在:"
|
||||
+ _format_counter(Counter(dict(active_datasource_counts.most_common(5))))
|
||||
+ "。"
|
||||
)
|
||||
|
||||
if recent_alerts:
|
||||
facts.append(
|
||||
"最近告警摘录:"
|
||||
+ ";".join(
|
||||
[
|
||||
f"{item.datasource_name or '未命名数据源'} / {item.severity.value if item.severity else '-'} / {item.status.value if item.status else '-'} / {item.message or '-'}"
|
||||
for item in recent_alerts[:6]
|
||||
]
|
||||
)
|
||||
)
|
||||
|
||||
context = {
|
||||
"source": "alerts",
|
||||
"total_alerts": total_alerts,
|
||||
"active_alerts": active_alerts,
|
||||
"acknowledged_alerts": acknowledged_alerts,
|
||||
"resolved_alerts": resolved_alerts,
|
||||
"severity_distribution": dict(severity_counts),
|
||||
"status_distribution": dict(status_counts),
|
||||
"top_datasources": dict(datasource_counts.most_common(6)),
|
||||
"top_active_datasources": dict(active_datasource_counts.most_common(5)),
|
||||
}
|
||||
|
||||
return (
|
||||
SituationalAnalysisRequest(
|
||||
title="告警态势 AI 简报",
|
||||
objective="基于当前告警总量、严重度、状态、数据源分布与最近告警摘录,生成一份面向值班人员的简明告警态势简报,突出待处理风险、告警集中点和优先动作。",
|
||||
observations=facts,
|
||||
constraints=[
|
||||
"明确区分事实、推断与建议。",
|
||||
"优先指出仍处于 active 状态且高严重度的告警簇。",
|
||||
"不要把 acknowledged 或 resolved 告警误判成当前仍在扩大。",
|
||||
"如果证据不足,请明确指出缺失的上下文。",
|
||||
],
|
||||
context=context,
|
||||
),
|
||||
facts,
|
||||
context,
|
||||
)
|
||||
259
backend/app/services/bgp_ai_brief.py
Normal file
259
backend/app/services/bgp_ai_brief.py
Normal file
@@ -0,0 +1,259 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections import Counter
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.api.v1.bgp import BGP_SOURCES
|
||||
from app.models.bgp_anomaly import BGPAnomaly
|
||||
from app.models.bgp_incident import BGPIncident
|
||||
from app.models.bgp_observation import BGPObservation
|
||||
from app.schemas.ai import SituationalAnalysisRequest
|
||||
from app.services.bgp_collectors import build_bgp_collector_coverage
|
||||
from app.services.bgp_enrichment import lookup_prefix_geography
|
||||
|
||||
|
||||
def _format_counter(counter: dict[str, int], empty_text: str = "无") -> str:
|
||||
if not counter:
|
||||
return empty_text
|
||||
return ",".join(f"{key} {value}" for key, value in counter.items())
|
||||
|
||||
|
||||
def _severity_rank(value: str | None) -> int:
|
||||
order = {
|
||||
"critical": 0,
|
||||
"high": 1,
|
||||
"medium": 2,
|
||||
"low": 3,
|
||||
"info": 4,
|
||||
}
|
||||
return order.get((value or "").lower(), 99)
|
||||
|
||||
|
||||
def _normalize_geo_key(country: str | None, city: str | None) -> str:
|
||||
if city and country:
|
||||
return f"{city}, {country}"
|
||||
return city or country or "未知区域"
|
||||
|
||||
|
||||
def _top_counter_items(counter: Counter[str], limit: int = 5) -> dict[str, int]:
|
||||
return {name: count for name, count in counter.most_common(limit) if name}
|
||||
|
||||
|
||||
def _collect_incident_regions(incidents: list[BGPIncident]) -> Counter[str]:
|
||||
counter: Counter[str] = Counter()
|
||||
for item in incidents:
|
||||
for region in item.affected_regions or []:
|
||||
if not isinstance(region, dict):
|
||||
continue
|
||||
counter[_normalize_geo_key(region.get("country"), region.get("city"))] += 1
|
||||
return counter
|
||||
|
||||
|
||||
def _collect_collector_regions(collectors: list[dict[str, Any]]) -> Counter[str]:
|
||||
counter: Counter[str] = Counter()
|
||||
for item in collectors:
|
||||
counter[_normalize_geo_key(item.get("country"), item.get("city"))] += int(item.get("recent_24h_observation_count") or 0)
|
||||
return counter
|
||||
|
||||
|
||||
def _format_geo_evidence(prefix_geographies: dict[str, dict[str, Any]], limit: int = 6) -> str:
|
||||
if not prefix_geographies:
|
||||
return "没有命中 prefix geography 证据。"
|
||||
|
||||
rows = []
|
||||
for prefix, item in list(prefix_geographies.items())[:limit]:
|
||||
region = _normalize_geo_key(item.get("country"), item.get("city"))
|
||||
source = item.get("source") or item.get("geography_mode") or "unknown"
|
||||
as_hint = item.get("asn")
|
||||
as_name = item.get("as_name")
|
||||
as_text = ""
|
||||
if as_hint:
|
||||
as_text = f" / ASN AS{as_hint}"
|
||||
if as_name:
|
||||
as_text += f" ({as_name})"
|
||||
rows.append(f"{prefix} -> {region} / 来源 {source}{as_text}")
|
||||
return ";".join(rows)
|
||||
|
||||
|
||||
async def build_bgp_brief_request(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
incident_limit: int = 5,
|
||||
anomaly_limit: int = 6,
|
||||
collector_limit: int = 5,
|
||||
) -> tuple[SituationalAnalysisRequest, list[str], dict[str, int | str | dict[str, int]]]:
|
||||
incidents_result = await db.execute(
|
||||
select(BGPIncident)
|
||||
.order_by(BGPIncident.created_at.desc(), BGPIncident.id.desc())
|
||||
.limit(max(incident_limit, 1))
|
||||
)
|
||||
anomalies_result = await db.execute(
|
||||
select(BGPAnomaly)
|
||||
.order_by(BGPAnomaly.created_at.desc(), BGPAnomaly.id.desc())
|
||||
.limit(max(anomaly_limit, 1))
|
||||
)
|
||||
observations_result = await db.execute(
|
||||
select(BGPObservation).where(BGPObservation.source.in_(BGP_SOURCES))
|
||||
)
|
||||
incident_count_result = await db.execute(select(func.count(BGPIncident.id)))
|
||||
anomaly_count_result = await db.execute(select(func.count(BGPAnomaly.id)))
|
||||
|
||||
incidents = incidents_result.scalars().all()
|
||||
anomalies = anomalies_result.scalars().all()
|
||||
observations = observations_result.scalars().all()
|
||||
collectors = await build_bgp_collector_coverage(db, source_filter=BGP_SOURCES)
|
||||
|
||||
total_incidents = incident_count_result.scalar() or 0
|
||||
total_anomalies = anomaly_count_result.scalar() or 0
|
||||
total_observations = len(observations)
|
||||
active_collectors = [item for item in collectors if item["observation_count"] > 0]
|
||||
|
||||
incident_status_counts = Counter((item.status or "unknown") for item in incidents)
|
||||
incident_severity_counts = Counter((item.severity or "unknown") for item in incidents)
|
||||
incident_type_counts = Counter((item.incident_type or "unknown") for item in incidents)
|
||||
anomaly_type_counts = Counter((item.anomaly_type or "unknown") for item in anomalies)
|
||||
event_type_counts = Counter((item.event_type or "unknown") for item in observations)
|
||||
incident_region_counts = _collect_incident_regions(incidents)
|
||||
|
||||
top_collectors = sorted(
|
||||
active_collectors,
|
||||
key=lambda item: (
|
||||
-int(item["recent_24h_observation_count"]),
|
||||
-int(item["observation_count"]),
|
||||
str(item["collector"]),
|
||||
),
|
||||
)[: max(collector_limit, 1)]
|
||||
collector_region_counts = _collect_collector_regions(top_collectors)
|
||||
|
||||
prefix_candidates = sorted(
|
||||
{
|
||||
prefix
|
||||
for item in incidents
|
||||
for prefix in (item.affected_prefixes or [])
|
||||
if prefix
|
||||
}
|
||||
| {item.prefix for item in anomalies if item.prefix}
|
||||
)
|
||||
prefix_geographies = await lookup_prefix_geography(db, prefix_candidates) if prefix_candidates else {}
|
||||
geography_region_counts = Counter(
|
||||
_normalize_geo_key(item.get("country"), item.get("city"))
|
||||
for item in prefix_geographies.values()
|
||||
if item.get("country") or item.get("city")
|
||||
)
|
||||
hotspot_region_counts = geography_region_counts + incident_region_counts
|
||||
collector_bias_regions = [
|
||||
region
|
||||
for region, count in collector_region_counts.most_common(3)
|
||||
if count > hotspot_region_counts.get(region, 0)
|
||||
]
|
||||
|
||||
observations_lines: list[str] = [
|
||||
f"当前共有 {total_incidents} 起 BGP incidents、{total_anomalies} 条 anomalies、{total_observations} 条原始观测事件。",
|
||||
f"活跃观测站 {len(active_collectors)} 个;近 24 小时事件数合计 {sum(int(item['recent_24h_observation_count']) for item in active_collectors)}。",
|
||||
f"最近 incidents 严重度分布:{_format_counter(dict(sorted(incident_severity_counts.items(), key=lambda item: _severity_rank(item[0]))))}。",
|
||||
f"最近 incidents 状态分布:{_format_counter(dict(incident_status_counts))}。",
|
||||
f"最近 incidents 类型分布:{_format_counter(dict(incident_type_counts.most_common(5)))}。",
|
||||
f"最近 anomalies 类型分布:{_format_counter(dict(anomaly_type_counts.most_common(6)))}。",
|
||||
f"观测事件类型分布:{_format_counter(dict(event_type_counts.most_common(6)))}。",
|
||||
]
|
||||
|
||||
if hotspot_region_counts:
|
||||
observations_lines.append(
|
||||
"区域热点事实层:"
|
||||
+ _format_counter(_top_counter_items(hotspot_region_counts, limit=5), empty_text="无明显区域聚集")
|
||||
+ "。"
|
||||
)
|
||||
|
||||
if prefix_geographies:
|
||||
observations_lines.append("Prefix geography 证据:" + _format_geo_evidence(prefix_geographies))
|
||||
|
||||
if collector_bias_regions:
|
||||
observations_lines.append(
|
||||
"观测偏差提示:重点观测站最近 24h 活跃度更集中在 "
|
||||
+ "、".join(collector_bias_regions)
|
||||
+ ",这些区域的事件升温结论需要结合 prefix geography 与 affected regions 交叉验证。"
|
||||
)
|
||||
elif top_collectors:
|
||||
observations_lines.append(
|
||||
"观测偏差提示:当前未发现明显高于区域热点事实层的单一观测站集中区域,但仍需区分 collector coverage 与真实区域风险。"
|
||||
)
|
||||
|
||||
if incidents:
|
||||
observations_lines.append(
|
||||
"最近 incident 摘要:" + ";".join(
|
||||
[
|
||||
f"{item.incident_type} / {item.severity} / {item.status}"
|
||||
f" / 前缀 {', '.join(item.affected_prefixes[:2]) if item.affected_prefixes else '-'}"
|
||||
f" / 观测站 {len(item.affected_collectors or [])} 个"
|
||||
for item in incidents
|
||||
]
|
||||
)
|
||||
)
|
||||
|
||||
if anomalies:
|
||||
observations_lines.append(
|
||||
"最近 anomaly 摘要:" + ";".join(
|
||||
[
|
||||
f"{item.anomaly_type} / {item.severity}"
|
||||
f" / 前缀 {item.prefix or '-'}"
|
||||
f" / ASN {item.new_origin_asn or item.origin_asn or '-'}"
|
||||
for item in anomalies
|
||||
]
|
||||
)
|
||||
)
|
||||
|
||||
if top_collectors:
|
||||
observations_lines.append(
|
||||
"重点观测站:" + ";".join(
|
||||
[
|
||||
f"{item['collector']} ({', '.join([part for part in [item.get('city'), item.get('country')] if part]) or '未知位置'})"
|
||||
f" / 近24h {item['recent_24h_observation_count']} 条"
|
||||
f" / 前缀 {item['prefix_count']} 个"
|
||||
for item in top_collectors
|
||||
]
|
||||
)
|
||||
)
|
||||
|
||||
context = {
|
||||
"source": "bgp-overview",
|
||||
"incident_total": total_incidents,
|
||||
"anomaly_total": total_anomalies,
|
||||
"observation_total": total_observations,
|
||||
"active_collectors": len(active_collectors),
|
||||
"top_incident_types": dict(incident_type_counts.most_common(5)),
|
||||
"top_anomaly_types": dict(anomaly_type_counts.most_common(6)),
|
||||
"top_event_types": dict(event_type_counts.most_common(6)),
|
||||
"region_hotspots": _top_counter_items(hotspot_region_counts, limit=6),
|
||||
"incident_regions": _top_counter_items(incident_region_counts, limit=6),
|
||||
"collector_bias_regions": collector_bias_regions,
|
||||
"prefix_geography_sources": dict(
|
||||
Counter(str(item.get("source") or "unknown") for item in prefix_geographies.values()).most_common(5)
|
||||
),
|
||||
"prefix_geography_sample": {
|
||||
prefix: {
|
||||
"country": item.get("country"),
|
||||
"city": item.get("city"),
|
||||
"source": item.get("source"),
|
||||
"asn": item.get("asn"),
|
||||
"as_name": item.get("as_name"),
|
||||
}
|
||||
for prefix, item in list(prefix_geographies.items())[:8]
|
||||
},
|
||||
}
|
||||
|
||||
return SituationalAnalysisRequest(
|
||||
title="BGP 态势 AI 简报",
|
||||
objective="基于当前 BGP incidents、anomalies、原始观测事件、观测站覆盖与 prefix geography 证据,生成一份面向操作员的简明态势简报,突出区域热点、观测偏差、当前风险、证据和优先动作。",
|
||||
observations=observations_lines,
|
||||
constraints=[
|
||||
"明确区分事实、推断与建议。",
|
||||
"优先指出需要立即关注的高严重度 incident 或异常模式。",
|
||||
"需要单独指出哪些区域结论来自 prefix geography / affected regions,哪些可能受 collector coverage 偏差影响。",
|
||||
"结论应服务值班排障,不要写成泛泛的模型演示文案。",
|
||||
"如果证据不足,要明确指出缺失数据。",
|
||||
],
|
||||
context=context,
|
||||
), observations_lines, context
|
||||
160
backend/app/services/bgp_ai_brief_store.py
Normal file
160
backend/app/services/bgp_ai_brief_store.py
Normal file
@@ -0,0 +1,160 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from uuid import uuid4
|
||||
|
||||
from app.core.config import ROOT_DIR
|
||||
from app.schemas.ai import BGPBriefRecordResponse, BGPBriefRecordSummary, SituationalAnalysisResponse
|
||||
|
||||
|
||||
_BRIEF_STORAGE_DIR = ROOT_DIR / "data" / "ai" / "bgp-briefs"
|
||||
_METADATA_PREFIX = "<!-- planet-bgp-brief-meta "
|
||||
_METADATA_SUFFIX = " -->"
|
||||
_BRIEF_TITLE = "BGP AI 简报"
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class _StoredBrief:
|
||||
id: str
|
||||
title: str
|
||||
provider: str
|
||||
model: str
|
||||
request_id: str | None
|
||||
generated_at: str
|
||||
content_markdown: str
|
||||
facts: list[str]
|
||||
context: dict[str, Any]
|
||||
path: Path
|
||||
|
||||
|
||||
def _ensure_storage_dir() -> Path:
|
||||
_BRIEF_STORAGE_DIR.mkdir(parents=True, exist_ok=True)
|
||||
return _BRIEF_STORAGE_DIR
|
||||
|
||||
|
||||
def _build_metadata_line(metadata: dict[str, Any]) -> str:
|
||||
return f"{_METADATA_PREFIX}{json.dumps(metadata, ensure_ascii=False)}{_METADATA_SUFFIX}"
|
||||
|
||||
|
||||
def _parse_brief_file(path: Path) -> _StoredBrief | None:
|
||||
try:
|
||||
raw_text = path.read_text(encoding="utf-8")
|
||||
except OSError:
|
||||
return None
|
||||
|
||||
first_line, separator, remainder = raw_text.partition("\n")
|
||||
if not separator or not first_line.startswith(_METADATA_PREFIX) or not first_line.endswith(_METADATA_SUFFIX):
|
||||
return None
|
||||
|
||||
metadata_payload = first_line[len(_METADATA_PREFIX) : -len(_METADATA_SUFFIX)]
|
||||
|
||||
try:
|
||||
metadata = json.loads(metadata_payload)
|
||||
except json.JSONDecodeError:
|
||||
return None
|
||||
|
||||
return _StoredBrief(
|
||||
id=str(metadata.get("id") or path.stem),
|
||||
title=str(metadata.get("title") or _BRIEF_TITLE),
|
||||
provider=str(metadata.get("provider") or "-"),
|
||||
model=str(metadata.get("model") or "-"),
|
||||
request_id=metadata.get("request_id"),
|
||||
generated_at=str(metadata.get("generated_at") or datetime.fromtimestamp(path.stat().st_mtime, UTC).isoformat()),
|
||||
content_markdown=remainder.lstrip("\n"),
|
||||
facts=list(metadata.get("facts") or []),
|
||||
context=dict(metadata.get("context") or {}),
|
||||
path=path,
|
||||
)
|
||||
|
||||
|
||||
def list_bgp_brief_records(limit: int = 50) -> list[BGPBriefRecordSummary]:
|
||||
storage_dir = _ensure_storage_dir()
|
||||
records: list[_StoredBrief] = []
|
||||
|
||||
for path in storage_dir.glob("*.md"):
|
||||
parsed = _parse_brief_file(path)
|
||||
if parsed is not None:
|
||||
records.append(parsed)
|
||||
|
||||
records.sort(key=lambda item: item.generated_at, reverse=True)
|
||||
|
||||
return [
|
||||
BGPBriefRecordSummary(
|
||||
id=item.id,
|
||||
title=item.title,
|
||||
provider=item.provider,
|
||||
model=item.model,
|
||||
request_id=item.request_id,
|
||||
generated_at=item.generated_at,
|
||||
)
|
||||
for item in records[: max(limit, 1)]
|
||||
]
|
||||
|
||||
|
||||
def get_bgp_brief_record(brief_id: str) -> BGPBriefRecordResponse | None:
|
||||
path = _ensure_storage_dir() / f"{brief_id}.md"
|
||||
parsed = _parse_brief_file(path)
|
||||
if parsed is None:
|
||||
return None
|
||||
|
||||
return BGPBriefRecordResponse(
|
||||
id=parsed.id,
|
||||
title=parsed.title,
|
||||
provider=parsed.provider,
|
||||
model=parsed.model,
|
||||
request_id=parsed.request_id,
|
||||
generated_at=parsed.generated_at,
|
||||
content_markdown=parsed.content_markdown,
|
||||
facts=parsed.facts,
|
||||
context=parsed.context,
|
||||
)
|
||||
|
||||
|
||||
def get_latest_bgp_brief_record() -> BGPBriefRecordResponse | None:
|
||||
summaries = list_bgp_brief_records(limit=1)
|
||||
if not summaries:
|
||||
return None
|
||||
return get_bgp_brief_record(summaries[0].id)
|
||||
|
||||
|
||||
def save_bgp_brief_record(
|
||||
analysis: SituationalAnalysisResponse,
|
||||
*,
|
||||
request_id: str | None,
|
||||
facts: list[str] | None = None,
|
||||
context: dict[str, Any] | None = None,
|
||||
generated_at: datetime | None = None,
|
||||
) -> BGPBriefRecordResponse:
|
||||
created_at = generated_at or datetime.now(UTC)
|
||||
brief_id = f"{created_at.strftime('%Y%m%dT%H%M%SZ')}-{uuid4().hex[:8]}"
|
||||
path = _ensure_storage_dir() / f"{brief_id}.md"
|
||||
|
||||
metadata = {
|
||||
"id": brief_id,
|
||||
"title": _BRIEF_TITLE,
|
||||
"provider": analysis.provider,
|
||||
"model": analysis.model,
|
||||
"request_id": request_id,
|
||||
"generated_at": created_at.isoformat(),
|
||||
"facts": facts or [],
|
||||
"context": context or {},
|
||||
}
|
||||
|
||||
markdown_text = f"{_build_metadata_line(metadata)}\n\n{analysis.content.rstrip()}\n"
|
||||
path.write_text(markdown_text, encoding="utf-8")
|
||||
|
||||
return BGPBriefRecordResponse(
|
||||
id=brief_id,
|
||||
title=_BRIEF_TITLE,
|
||||
provider=analysis.provider,
|
||||
model=analysis.model,
|
||||
request_id=request_id,
|
||||
generated_at=created_at.isoformat(),
|
||||
content_markdown=analysis.content,
|
||||
facts=facts or [],
|
||||
context=context or {},
|
||||
)
|
||||
@@ -6,7 +6,7 @@ from collections import defaultdict
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy import case, distinct, func, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.time import to_iso8601_utc
|
||||
@@ -14,6 +14,16 @@ from app.models.bgp_observation import BGPObservation
|
||||
from app.services.collectors.bgp_common import RIPE_RIS_COLLECTOR_COORDS
|
||||
|
||||
|
||||
def _collector_base_filters(source_filter: tuple[str, ...] | None) -> list[Any]:
|
||||
filters: list[Any] = [
|
||||
BGPObservation.collector.isnot(None),
|
||||
func.length(func.btrim(BGPObservation.collector)) > 0,
|
||||
]
|
||||
if source_filter:
|
||||
filters.append(BGPObservation.source.in_(source_filter))
|
||||
return filters
|
||||
|
||||
|
||||
async def build_bgp_collector_coverage(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
@@ -24,88 +34,148 @@ async def build_bgp_collector_coverage(
|
||||
recent_24h_threshold = now - timedelta(hours=24)
|
||||
recent_7d_threshold = now - timedelta(days=7)
|
||||
|
||||
stmt = select(BGPObservation).order_by(BGPObservation.observed_at.desc(), BGPObservation.id.desc())
|
||||
if source_filter:
|
||||
stmt = stmt.where(BGPObservation.source.in_(source_filter))
|
||||
filters = _collector_base_filters(source_filter)
|
||||
country_expr = func.nullif(BGPObservation.collector_geo["country"].as_string(), "")
|
||||
city_expr = func.nullif(BGPObservation.collector_geo["city"].as_string(), "")
|
||||
|
||||
result = await db.execute(stmt)
|
||||
records = list(result.scalars().all())
|
||||
aggregate_stmt = (
|
||||
select(
|
||||
BGPObservation.collector.label("collector"),
|
||||
func.count(BGPObservation.id).label("observation_count"),
|
||||
func.count(distinct(BGPObservation.prefix)).label("prefix_count"),
|
||||
func.count(distinct(BGPObservation.origin_asn)).label("origin_asn_count"),
|
||||
func.count(distinct(BGPObservation.peer_asn)).label("peer_asn_count"),
|
||||
func.sum(case((BGPObservation.observed_at >= recent_15m_threshold, 1), else_=0)).label("recent_15m_observation_count"),
|
||||
func.sum(case((BGPObservation.observed_at >= recent_24h_threshold, 1), else_=0)).label("recent_24h_observation_count"),
|
||||
func.sum(case((BGPObservation.observed_at >= recent_7d_threshold, 1), else_=0)).label("recent_7d_observation_count"),
|
||||
func.count(distinct(case((BGPObservation.observed_at >= recent_15m_threshold, BGPObservation.prefix), else_=None))).label("recent_15m_prefix_count"),
|
||||
func.count(distinct(case((BGPObservation.observed_at >= recent_24h_threshold, BGPObservation.prefix), else_=None))).label("recent_24h_prefix_count"),
|
||||
func.count(distinct(case((BGPObservation.observed_at >= recent_7d_threshold, BGPObservation.prefix), else_=None))).label("recent_7d_prefix_count"),
|
||||
func.max(BGPObservation.observed_at).label("latest_observed_at"),
|
||||
)
|
||||
.where(*filters)
|
||||
.group_by(BGPObservation.collector)
|
||||
)
|
||||
aggregate_rows = (await db.execute(aggregate_stmt)).all()
|
||||
|
||||
latest_subquery = (
|
||||
select(
|
||||
BGPObservation.collector.label("collector"),
|
||||
BGPObservation.event_type.label("latest_event_type"),
|
||||
country_expr.label("country"),
|
||||
city_expr.label("city"),
|
||||
func.row_number()
|
||||
.over(
|
||||
partition_by=BGPObservation.collector,
|
||||
order_by=(BGPObservation.observed_at.desc(), BGPObservation.id.desc()),
|
||||
)
|
||||
.label("rn"),
|
||||
)
|
||||
.where(*filters)
|
||||
.subquery()
|
||||
)
|
||||
latest_rows = (
|
||||
await db.execute(
|
||||
select(
|
||||
latest_subquery.c.collector,
|
||||
latest_subquery.c.latest_event_type,
|
||||
latest_subquery.c.country,
|
||||
latest_subquery.c.city,
|
||||
).where(latest_subquery.c.rn == 1)
|
||||
)
|
||||
).all()
|
||||
|
||||
event_counts_subquery = (
|
||||
select(
|
||||
BGPObservation.collector.label("collector"),
|
||||
BGPObservation.event_type.label("event_type"),
|
||||
func.count(BGPObservation.id).label("count"),
|
||||
func.row_number()
|
||||
.over(
|
||||
partition_by=BGPObservation.collector,
|
||||
order_by=(func.count(BGPObservation.id).desc(), BGPObservation.event_type.asc()),
|
||||
)
|
||||
.label("rn"),
|
||||
)
|
||||
.where(*filters)
|
||||
.group_by(BGPObservation.collector, BGPObservation.event_type)
|
||||
.subquery()
|
||||
)
|
||||
top_event_rows = (
|
||||
await db.execute(
|
||||
select(
|
||||
event_counts_subquery.c.collector,
|
||||
event_counts_subquery.c.event_type,
|
||||
event_counts_subquery.c.count,
|
||||
).where(event_counts_subquery.c.rn <= 3)
|
||||
)
|
||||
).all()
|
||||
|
||||
scope_rows = (
|
||||
await db.execute(
|
||||
select(
|
||||
BGPObservation.collector.label("collector"),
|
||||
country_expr.label("country"),
|
||||
city_expr.label("city"),
|
||||
)
|
||||
.where(*filters)
|
||||
.distinct()
|
||||
)
|
||||
).all()
|
||||
|
||||
latest_by_collector = {
|
||||
row.collector: {
|
||||
"latest_event_type": row.latest_event_type,
|
||||
"country": row.country,
|
||||
"city": row.city,
|
||||
}
|
||||
for row in latest_rows
|
||||
}
|
||||
|
||||
scope_by_collector: dict[str, dict[str, set[str]]] = defaultdict(lambda: {"countries": set(), "cities": set()})
|
||||
for row in scope_rows:
|
||||
if row.country:
|
||||
scope_by_collector[row.collector]["countries"].add(row.country)
|
||||
if row.city:
|
||||
scope_by_collector[row.collector]["cities"].add(row.city)
|
||||
|
||||
top_events_by_collector: dict[str, list[dict[str, Any]]] = defaultdict(list)
|
||||
for row in top_event_rows:
|
||||
top_events_by_collector[row.collector].append(
|
||||
{"event_type": row.event_type, "count": row.count}
|
||||
)
|
||||
|
||||
by_collector: dict[str, dict[str, Any]] = {}
|
||||
for record in records:
|
||||
collector = str(record.collector or "").strip()
|
||||
if not collector:
|
||||
continue
|
||||
for row in aggregate_rows:
|
||||
collector = row.collector
|
||||
latest = latest_by_collector.get(collector, {})
|
||||
fallback_location = RIPE_RIS_COLLECTOR_COORDS.get(collector, {})
|
||||
scope = scope_by_collector.get(collector, {"countries": set(), "cities": set()})
|
||||
|
||||
coverage = by_collector.get(collector)
|
||||
if coverage is None:
|
||||
location = record.collector_geo or RIPE_RIS_COLLECTOR_COORDS.get(collector, {})
|
||||
coverage = {
|
||||
"collector": collector,
|
||||
"city": location.get("city"),
|
||||
"country": location.get("country"),
|
||||
"latitude": location.get("latitude"),
|
||||
"longitude": location.get("longitude"),
|
||||
"observation_count": 0,
|
||||
"prefixes": set(),
|
||||
"origin_asns": set(),
|
||||
"peer_asns": set(),
|
||||
"event_types": defaultdict(int),
|
||||
"countries": set(),
|
||||
"cities": set(),
|
||||
"recent_15m_observation_count": 0,
|
||||
"recent_24h_observation_count": 0,
|
||||
"recent_7d_observation_count": 0,
|
||||
"recent_15m_prefixes": set(),
|
||||
"recent_24h_prefixes": set(),
|
||||
"recent_7d_prefixes": set(),
|
||||
"latest_observed_at": None,
|
||||
"latest_event_type": None,
|
||||
}
|
||||
by_collector[collector] = coverage
|
||||
|
||||
coverage["observation_count"] += 1
|
||||
if record.prefix:
|
||||
coverage["prefixes"].add(record.prefix)
|
||||
if record.origin_asn is not None:
|
||||
coverage["origin_asns"].add(record.origin_asn)
|
||||
if record.peer_asn is not None:
|
||||
coverage["peer_asns"].add(record.peer_asn)
|
||||
if record.event_type:
|
||||
coverage["event_types"][record.event_type] += 1
|
||||
|
||||
observed_at = record.observed_at
|
||||
if observed_at is not None:
|
||||
aware_observed_at = (
|
||||
observed_at.astimezone(UTC)
|
||||
if observed_at.tzinfo
|
||||
else observed_at.replace(tzinfo=UTC)
|
||||
)
|
||||
if aware_observed_at >= recent_15m_threshold:
|
||||
coverage["recent_15m_observation_count"] += 1
|
||||
if record.prefix:
|
||||
coverage["recent_15m_prefixes"].add(record.prefix)
|
||||
if aware_observed_at >= recent_24h_threshold:
|
||||
coverage["recent_24h_observation_count"] += 1
|
||||
if record.prefix:
|
||||
coverage["recent_24h_prefixes"].add(record.prefix)
|
||||
if aware_observed_at >= recent_7d_threshold:
|
||||
coverage["recent_7d_observation_count"] += 1
|
||||
if record.prefix:
|
||||
coverage["recent_7d_prefixes"].add(record.prefix)
|
||||
|
||||
geo = record.collector_geo or {}
|
||||
if geo.get("country"):
|
||||
coverage["countries"].add(geo["country"])
|
||||
if geo.get("city"):
|
||||
coverage["cities"].add(geo["city"])
|
||||
|
||||
current_latest = coverage["latest_observed_at"]
|
||||
if current_latest is None or (
|
||||
record.observed_at is not None and record.observed_at > current_latest
|
||||
):
|
||||
coverage["latest_observed_at"] = record.observed_at
|
||||
coverage["latest_event_type"] = record.event_type
|
||||
by_collector[collector] = {
|
||||
"collector": collector,
|
||||
"city": latest.get("city") or fallback_location.get("city"),
|
||||
"country": latest.get("country") or fallback_location.get("country"),
|
||||
"latitude": fallback_location.get("latitude"),
|
||||
"longitude": fallback_location.get("longitude"),
|
||||
"observation_count": row.observation_count or 0,
|
||||
"prefix_count": row.prefix_count or 0,
|
||||
"origin_asn_count": row.origin_asn_count or 0,
|
||||
"peer_asn_count": row.peer_asn_count or 0,
|
||||
"recent_15m_observation_count": row.recent_15m_observation_count or 0,
|
||||
"recent_24h_observation_count": row.recent_24h_observation_count or 0,
|
||||
"recent_7d_observation_count": row.recent_7d_observation_count or 0,
|
||||
"recent_15m_prefix_count": row.recent_15m_prefix_count or 0,
|
||||
"recent_24h_prefix_count": row.recent_24h_prefix_count or 0,
|
||||
"recent_7d_prefix_count": row.recent_7d_prefix_count or 0,
|
||||
"top_event_types": top_events_by_collector.get(collector, []),
|
||||
"latest_observed_at": to_iso8601_utc(row.latest_observed_at),
|
||||
"latest_event_type": latest.get("latest_event_type"),
|
||||
"baseline_scope": {
|
||||
"countries": sorted(scope["countries"]),
|
||||
"cities": sorted(scope["cities"]),
|
||||
},
|
||||
}
|
||||
|
||||
for collector, location in RIPE_RIS_COLLECTOR_COORDS.items():
|
||||
if collector in by_collector:
|
||||
@@ -117,57 +187,22 @@ async def build_bgp_collector_coverage(
|
||||
"latitude": location.get("latitude"),
|
||||
"longitude": location.get("longitude"),
|
||||
"observation_count": 0,
|
||||
"prefixes": set(),
|
||||
"origin_asns": set(),
|
||||
"peer_asns": set(),
|
||||
"event_types": defaultdict(int),
|
||||
"countries": {location.get("country")} if location.get("country") else set(),
|
||||
"cities": {location.get("city")} if location.get("city") else set(),
|
||||
"prefix_count": 0,
|
||||
"origin_asn_count": 0,
|
||||
"peer_asn_count": 0,
|
||||
"recent_15m_observation_count": 0,
|
||||
"recent_24h_observation_count": 0,
|
||||
"recent_7d_observation_count": 0,
|
||||
"recent_15m_prefixes": set(),
|
||||
"recent_24h_prefixes": set(),
|
||||
"recent_7d_prefixes": set(),
|
||||
"recent_15m_prefix_count": 0,
|
||||
"recent_24h_prefix_count": 0,
|
||||
"recent_7d_prefix_count": 0,
|
||||
"top_event_types": [],
|
||||
"latest_observed_at": None,
|
||||
"latest_event_type": None,
|
||||
"baseline_scope": {
|
||||
"countries": [location["country"]] if location.get("country") else [],
|
||||
"cities": [location["city"]] if location.get("city") else [],
|
||||
},
|
||||
}
|
||||
|
||||
results: list[dict[str, Any]] = []
|
||||
for collector in sorted(by_collector.keys()):
|
||||
item = by_collector[collector]
|
||||
top_event_types = sorted(
|
||||
item["event_types"].items(),
|
||||
key=lambda pair: (-pair[1], pair[0]),
|
||||
)
|
||||
results.append(
|
||||
{
|
||||
"collector": item["collector"],
|
||||
"city": item["city"],
|
||||
"country": item["country"],
|
||||
"latitude": item["latitude"],
|
||||
"longitude": item["longitude"],
|
||||
"observation_count": item["observation_count"],
|
||||
"prefix_count": len(item["prefixes"]),
|
||||
"origin_asn_count": len(item["origin_asns"]),
|
||||
"peer_asn_count": len(item["peer_asns"]),
|
||||
"recent_15m_observation_count": item["recent_15m_observation_count"],
|
||||
"recent_24h_observation_count": item["recent_24h_observation_count"],
|
||||
"recent_7d_observation_count": item["recent_7d_observation_count"],
|
||||
"recent_15m_prefix_count": len(item["recent_15m_prefixes"]),
|
||||
"recent_24h_prefix_count": len(item["recent_24h_prefixes"]),
|
||||
"recent_7d_prefix_count": len(item["recent_7d_prefixes"]),
|
||||
"top_event_types": [
|
||||
{"event_type": event_type, "count": count}
|
||||
for event_type, count in top_event_types[:3]
|
||||
],
|
||||
"latest_observed_at": to_iso8601_utc(item["latest_observed_at"]),
|
||||
"latest_event_type": item["latest_event_type"],
|
||||
"baseline_scope": {
|
||||
"countries": sorted(country for country in item["countries"] if country),
|
||||
"cities": sorted(city for city in item["cities"] if city),
|
||||
},
|
||||
}
|
||||
)
|
||||
|
||||
return results
|
||||
return [by_collector[collector] for collector in sorted(by_collector.keys())]
|
||||
|
||||
@@ -7,7 +7,7 @@ from collections import defaultdict
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import select, text
|
||||
from sqlalchemy import Integer, cast, select, text
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.countries import get_country_centroid, normalize_country
|
||||
@@ -231,6 +231,13 @@ async def _lookup_prefix_geography(
|
||||
return results
|
||||
|
||||
|
||||
async def lookup_prefix_geography(
|
||||
db: AsyncSession,
|
||||
prefix_values: list[str],
|
||||
) -> dict[str, dict[str, Any]]:
|
||||
return await _lookup_prefix_geography(db, prefix_values)
|
||||
|
||||
|
||||
async def enrich_bgp_events_for_batch(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
@@ -261,29 +268,40 @@ async def enrich_bgp_events_for_batch(
|
||||
historical_prefix_baseline: dict[str, dict[str, Any]] = {}
|
||||
if prefix_values:
|
||||
previous_result = await db.execute(
|
||||
select(BGPObservation).where(
|
||||
select(
|
||||
BGPObservation.prefix,
|
||||
BGPObservation.origin_asn,
|
||||
BGPObservation.collector,
|
||||
BGPObservation.collector_geo,
|
||||
).where(
|
||||
BGPObservation.source == source,
|
||||
BGPObservation.prefix.in_(prefix_values),
|
||||
)
|
||||
)
|
||||
by_prefix: defaultdict[str, list[BGPObservation]] = defaultdict(list)
|
||||
for observation in previous_result.scalars().all():
|
||||
if observation.prefix:
|
||||
by_prefix[observation.prefix].append(observation)
|
||||
by_prefix: defaultdict[str, list[dict[str, Any]]] = defaultdict(list)
|
||||
for prefix, origin_asn, collector, collector_geo in previous_result.all():
|
||||
if prefix:
|
||||
by_prefix[str(prefix)].append(
|
||||
{
|
||||
"origin_asn": origin_asn,
|
||||
"collector": collector,
|
||||
"collector_geo": collector_geo or {},
|
||||
}
|
||||
)
|
||||
|
||||
for prefix, observations in by_prefix.items():
|
||||
unique_origins = sorted(
|
||||
{
|
||||
observation.origin_asn
|
||||
observation["origin_asn"]
|
||||
for observation in observations
|
||||
if observation.origin_asn is not None
|
||||
if observation["origin_asn"] is not None
|
||||
}
|
||||
)
|
||||
unique_collectors = sorted(
|
||||
{
|
||||
observation.collector
|
||||
observation["collector"]
|
||||
for observation in observations
|
||||
if observation.collector
|
||||
if observation["collector"]
|
||||
}
|
||||
)
|
||||
historical_prefix_baseline[prefix] = {
|
||||
@@ -292,9 +310,9 @@ async def enrich_bgp_events_for_batch(
|
||||
"historical_observation_count": len(observations),
|
||||
"historical_regions": _compact_locations(
|
||||
[
|
||||
observation.collector_geo or {}
|
||||
observation["collector_geo"] or {}
|
||||
for observation in observations
|
||||
if observation.collector_geo
|
||||
if observation["collector_geo"]
|
||||
]
|
||||
),
|
||||
}
|
||||
@@ -303,7 +321,13 @@ async def enrich_bgp_events_for_batch(
|
||||
prefix_geographies = await _lookup_prefix_geography(db, prefix_values) if prefix_values else {}
|
||||
if origin_asns:
|
||||
peeringdb_result = await db.execute(
|
||||
select(CollectedData).where(CollectedData.source == "peeringdb_network")
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source == "peeringdb_network")
|
||||
.where(CollectedData.is_current.is_(True))
|
||||
.where(
|
||||
cast(CollectedData.extra_data["asn"].as_string(), Integer).in_(origin_asns),
|
||||
)
|
||||
.order_by(CollectedData.id.desc())
|
||||
)
|
||||
for record in peeringdb_result.scalars().all():
|
||||
metadata = record.extra_data or {}
|
||||
|
||||
@@ -48,14 +48,36 @@ def _collector_regions_from_anomaly(anomaly: BGPAnomaly) -> list[dict]:
|
||||
return collected
|
||||
|
||||
|
||||
def _dedupe_collected_records(records: list[CollectedData]) -> list[CollectedData]:
|
||||
latest_by_key: dict[str, CollectedData] = {}
|
||||
for record in records:
|
||||
dedupe_key = str(record.source_id or record.entity_key or record.name or record.id)
|
||||
existing = latest_by_key.get(dedupe_key)
|
||||
if existing is None or (record.id or 0) > (existing.id or 0):
|
||||
latest_by_key[dedupe_key] = record
|
||||
return list(latest_by_key.values())
|
||||
async def _load_current_infrastructure_records(
|
||||
db: AsyncSession,
|
||||
) -> tuple[list[CollectedData], list[CollectedData], list[CollectedData]]:
|
||||
result = await db.execute(
|
||||
select(CollectedData)
|
||||
.where(
|
||||
CollectedData.source.in_(
|
||||
(
|
||||
"arcgis_landing_points",
|
||||
"arcgis_cable_landing_relation",
|
||||
"arcgis_cables",
|
||||
)
|
||||
)
|
||||
)
|
||||
.where(CollectedData.is_current.is_(True))
|
||||
.order_by(CollectedData.source.asc(), CollectedData.id.desc())
|
||||
)
|
||||
grouped_records = {
|
||||
"arcgis_landing_points": [],
|
||||
"arcgis_cable_landing_relation": [],
|
||||
"arcgis_cables": [],
|
||||
}
|
||||
for record in result.scalars().all():
|
||||
grouped_records.setdefault(record.source, []).append(record)
|
||||
|
||||
return (
|
||||
grouped_records["arcgis_landing_points"],
|
||||
grouped_records["arcgis_cable_landing_relation"],
|
||||
grouped_records["arcgis_cables"],
|
||||
)
|
||||
|
||||
|
||||
async def infer_related_infrastructure(
|
||||
@@ -75,19 +97,9 @@ async def infer_related_infrastructure(
|
||||
if not valid_regions:
|
||||
return {"related_cables": [], "related_ixps": []}
|
||||
|
||||
landing_result = await db.execute(
|
||||
select(CollectedData).where(CollectedData.source == "arcgis_landing_points")
|
||||
landing_records, relation_records, cable_records = await _load_current_infrastructure_records(
|
||||
db,
|
||||
)
|
||||
relation_result = await db.execute(
|
||||
select(CollectedData).where(CollectedData.source == "arcgis_cable_landing_relation")
|
||||
)
|
||||
cable_result = await db.execute(
|
||||
select(CollectedData).where(CollectedData.source == "arcgis_cables")
|
||||
)
|
||||
|
||||
landing_records = _dedupe_collected_records(list(landing_result.scalars().all()))
|
||||
relation_records = _dedupe_collected_records(list(relation_result.scalars().all()))
|
||||
cable_records = _dedupe_collected_records(list(cable_result.scalars().all()))
|
||||
|
||||
city_to_cable_ids: dict[int, list[int]] = {}
|
||||
for relation in relation_records:
|
||||
|
||||
@@ -35,6 +35,7 @@ from app.services.collectors.bgpstream import BGPStreamBackfillCollector
|
||||
from app.services.collectors.iptoasn import IPtoASNPrefixGeoCollector
|
||||
from app.services.collectors.opengeofeed import OpenGeoFeedPrefixGeoCollector
|
||||
from app.services.collectors.nro_delegated import NRODelegatedPrefixGeoCollector
|
||||
from app.services.collectors.news_live_streams import NewsLiveStreamsCollector
|
||||
|
||||
collector_registry.register(TOP500Collector())
|
||||
collector_registry.register(EpochAIGPUCollector())
|
||||
@@ -61,3 +62,4 @@ collector_registry.register(BGPStreamBackfillCollector())
|
||||
collector_registry.register(IPtoASNPrefixGeoCollector())
|
||||
collector_registry.register(OpenGeoFeedPrefixGeoCollector())
|
||||
collector_registry.register(NRODelegatedPrefixGeoCollector())
|
||||
collector_registry.register(NewsLiveStreamsCollector())
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
"""Base collector class for all data sources"""
|
||||
|
||||
import asyncio
|
||||
from abc import ABC, abstractmethod
|
||||
from typing import Dict, List, Any, Optional
|
||||
from datetime import UTC, datetime
|
||||
@@ -166,6 +167,59 @@ class BaseCollector(ABC):
|
||||
await db.commit()
|
||||
return snapshot.id
|
||||
|
||||
async def _rollback_incomplete_run(
|
||||
self,
|
||||
db: AsyncSession,
|
||||
*,
|
||||
task_id: int,
|
||||
snapshot_id: Optional[int],
|
||||
reason: str,
|
||||
) -> None:
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.models.data_snapshot import DataSnapshot
|
||||
|
||||
await db.execute(CollectedData.__table__.delete().where(CollectedData.task_id == task_id))
|
||||
|
||||
parent_snapshot_id: Optional[int] = None
|
||||
if snapshot_id is not None:
|
||||
snapshot = await db.get(DataSnapshot, snapshot_id)
|
||||
if snapshot:
|
||||
parent_snapshot_id = snapshot.parent_snapshot_id
|
||||
snapshot.status = "cancelled"
|
||||
snapshot.is_current = False
|
||||
snapshot.completed_at = datetime.now(UTC)
|
||||
summary = dict(snapshot.summary or {})
|
||||
summary["rollback"] = True
|
||||
summary["rollback_reason"] = reason
|
||||
snapshot.summary = summary
|
||||
|
||||
await db.execute(
|
||||
text(
|
||||
"""
|
||||
UPDATE collected_data
|
||||
SET is_current = FALSE
|
||||
WHERE source = :source
|
||||
"""
|
||||
),
|
||||
{"source": self.name},
|
||||
)
|
||||
|
||||
if parent_snapshot_id is not None:
|
||||
parent_snapshot = await db.get(DataSnapshot, parent_snapshot_id)
|
||||
if parent_snapshot:
|
||||
parent_snapshot.is_current = True
|
||||
|
||||
await db.execute(
|
||||
text(
|
||||
"""
|
||||
UPDATE collected_data
|
||||
SET is_current = TRUE
|
||||
WHERE snapshot_id = :snapshot_id
|
||||
"""
|
||||
),
|
||||
{"snapshot_id": parent_snapshot_id},
|
||||
)
|
||||
|
||||
async def run(self, db: AsyncSession) -> Dict[str, Any]:
|
||||
"""Full pipeline: fetch -> transform -> save"""
|
||||
from app.services.collectors.registry import collector_registry
|
||||
@@ -227,7 +281,24 @@ class BaseCollector(ABC):
|
||||
"records_processed": records_count,
|
||||
"execution_time_seconds": (datetime.now(UTC) - start_time).total_seconds(),
|
||||
}
|
||||
except asyncio.CancelledError:
|
||||
await db.rollback()
|
||||
task.status = "cancelled"
|
||||
task.phase = "cancelled"
|
||||
task.error_message = "Collection cancelled by operator and rolled back"
|
||||
task.completed_at = datetime.now(UTC)
|
||||
if snapshot_id is not None:
|
||||
await self._rollback_incomplete_run(
|
||||
db,
|
||||
task_id=task_id,
|
||||
snapshot_id=snapshot_id,
|
||||
reason="cancelled_by_operator",
|
||||
)
|
||||
await db.commit()
|
||||
await self._publish_task_update(force=True)
|
||||
raise
|
||||
except Exception as e:
|
||||
await db.rollback()
|
||||
task.status = "failed"
|
||||
task.phase = "failed"
|
||||
task.error_message = str(e)
|
||||
@@ -276,20 +347,34 @@ class BaseCollector(ABC):
|
||||
updated_count = 0
|
||||
unchanged_count = 0
|
||||
seen_entity_keys: set[str] = set()
|
||||
previous_current_keys: set[str] = set()
|
||||
progress_commit_interval = 1000
|
||||
|
||||
previous_current_result = await db.execute(
|
||||
select(CollectedData.entity_key).where(
|
||||
select(CollectedData)
|
||||
.where(
|
||||
CollectedData.source == self.name,
|
||||
CollectedData.is_current == True,
|
||||
)
|
||||
.order_by(CollectedData.entity_key.asc(), CollectedData.collected_at.desc().nullslast(), CollectedData.id.desc())
|
||||
)
|
||||
previous_current_keys = {row[0] for row in previous_current_result.fetchall() if row[0]}
|
||||
previous_current_records = previous_current_result.scalars().all()
|
||||
previous_current_keys = {record.entity_key for record in previous_current_records if record.entity_key}
|
||||
previous_current_map: dict[str, CollectedData] = {}
|
||||
stale_previous_records: list[CollectedData] = []
|
||||
|
||||
for existing_record in previous_current_records:
|
||||
entity_key = existing_record.entity_key
|
||||
if not entity_key:
|
||||
continue
|
||||
if entity_key not in previous_current_map:
|
||||
previous_current_map[entity_key] = existing_record
|
||||
continue
|
||||
stale_previous_records.append(existing_record)
|
||||
|
||||
for stale_record in stale_previous_records:
|
||||
stale_record.is_current = False
|
||||
|
||||
for i, item in enumerate(data):
|
||||
print(
|
||||
f"DEBUG: Saving item {i}: name={item.get('name')}, metadata={item.get('metadata', 'NOT FOUND')}"
|
||||
)
|
||||
raw_metadata = item.get("metadata", {})
|
||||
extra_data = build_dynamic_metadata(
|
||||
raw_metadata,
|
||||
@@ -318,20 +403,9 @@ class BaseCollector(ABC):
|
||||
previous_record = None
|
||||
|
||||
if entity_key and entity_key not in seen_entity_keys:
|
||||
result = await db.execute(
|
||||
select(CollectedData)
|
||||
.where(
|
||||
CollectedData.source == self.name,
|
||||
CollectedData.entity_key == entity_key,
|
||||
CollectedData.is_current == True,
|
||||
)
|
||||
.order_by(CollectedData.collected_at.desc().nullslast(), CollectedData.id.desc())
|
||||
)
|
||||
previous_records = result.scalars().all()
|
||||
if previous_records:
|
||||
previous_record = previous_records[0]
|
||||
for old_record in previous_records:
|
||||
old_record.is_current = False
|
||||
previous_record = previous_current_map.get(entity_key)
|
||||
if previous_record is not None:
|
||||
previous_record.is_current = False
|
||||
|
||||
record = CollectedData(
|
||||
snapshot_id=snapshot_id,
|
||||
@@ -375,7 +449,7 @@ class BaseCollector(ABC):
|
||||
seen_entity_keys.add(entity_key)
|
||||
records_added += 1
|
||||
|
||||
if i % 100 == 0:
|
||||
if (i + 1) % progress_commit_interval == 0:
|
||||
await self.update_progress(i + 1, commit=True)
|
||||
|
||||
if snapshot_id is not None:
|
||||
|
||||
@@ -21,7 +21,7 @@ class CelesTrakTLECollector(BaseCollector):
|
||||
|
||||
@property
|
||||
def base_url(self) -> str:
|
||||
return "https://celestrak.org/NORAD/elements/gp.php"
|
||||
return self._resolved_url or ""
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
satellite_groups = [
|
||||
@@ -40,7 +40,7 @@ class CelesTrakTLECollector(BaseCollector):
|
||||
async with httpx.AsyncClient(timeout=120.0) as client:
|
||||
for group in satellite_groups:
|
||||
try:
|
||||
url = f"https://celestrak.org/NORAD/elements/gp.php?GROUP={group}&FORMAT=json"
|
||||
url = f"{self.base_url}?GROUP={group}&FORMAT=json"
|
||||
response = await client.get(url)
|
||||
|
||||
if response.status_code == 200:
|
||||
|
||||
@@ -39,6 +39,16 @@ class CloudflareRadarDeviceCollector(HTTPCollector):
|
||||
if CLOUDFLARE_API_TOKEN:
|
||||
self.headers["Authorization"] = f"Bearer {CLOUDFLARE_API_TOKEN}"
|
||||
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
return self._resolved_url or self.base_url
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(self.request_url, headers=self.headers)
|
||||
response.raise_for_status()
|
||||
return self.parse_response(response.json())
|
||||
|
||||
def parse_response(self, response: Dict[str, Any]) -> List[Dict[str, Any]]:
|
||||
"""Parse Cloudflare Radar device type response"""
|
||||
data = []
|
||||
@@ -87,6 +97,16 @@ class CloudflareRadarTrafficCollector(HTTPCollector):
|
||||
if CLOUDFLARE_API_TOKEN:
|
||||
self.headers["Authorization"] = f"Bearer {CLOUDFLARE_API_TOKEN}"
|
||||
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
return self._resolved_url or self.base_url
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(self.request_url, headers=self.headers)
|
||||
response.raise_for_status()
|
||||
return self.parse_response(response.json())
|
||||
|
||||
def parse_response(self, response: Dict[str, Any]) -> List[Dict[str, Any]]:
|
||||
"""Parse Cloudflare Radar traffic timeseries response"""
|
||||
data = []
|
||||
@@ -135,6 +155,16 @@ class CloudflareRadarTopASCollector(HTTPCollector):
|
||||
if CLOUDFLARE_API_TOKEN:
|
||||
self.headers["Authorization"] = f"Bearer {CLOUDFLARE_API_TOKEN}"
|
||||
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
return self._resolved_url or self.base_url
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(self.request_url, headers=self.headers)
|
||||
response.raise_for_status()
|
||||
return self.parse_response(response.json())
|
||||
|
||||
def parse_response(self, response: Dict[str, Any]) -> List[Dict[str, Any]]:
|
||||
"""Parse Cloudflare Radar top locations response"""
|
||||
data = []
|
||||
|
||||
@@ -23,7 +23,7 @@ class EpochAIGPUCollector(BaseCollector):
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
"""Fetch Epoch AI GPU clusters data from webpage"""
|
||||
url = "https://epoch.ai/data/gpu-clusters"
|
||||
url = self._resolved_url or ""
|
||||
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(url)
|
||||
|
||||
@@ -18,11 +18,9 @@ class FAOLandingPointCollector(BaseCollector):
|
||||
frequency_hours = 168
|
||||
data_type = "landing_point"
|
||||
|
||||
csv_url = "https://data.apps.fao.org/catalog/dataset/1b75ff21-92f2-4b96-9b7b-98e8aa65ad5d/resource/b6071077-d1d4-4e97-aa00-42e902847c87/download/landing-point-geo.csv"
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(self.csv_url)
|
||||
response = await client.get(self._resolved_url or "")
|
||||
response.raise_for_status()
|
||||
return self.parse_csv(response.text)
|
||||
|
||||
|
||||
@@ -21,6 +21,18 @@ class HuggingFaceModelCollector(HTTPCollector):
|
||||
data_type = "model"
|
||||
base_url = "https://huggingface.co/api/models"
|
||||
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
return self._resolved_url or self.base_url
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
from httpx import AsyncClient
|
||||
|
||||
async with AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(self.request_url, headers=self.headers)
|
||||
response.raise_for_status()
|
||||
return self.parse_response(response.json())
|
||||
|
||||
def parse_response(self, response: Dict[str, Any]) -> List[Dict[str, Any]]:
|
||||
"""Parse Hugging Face models API response"""
|
||||
data = []
|
||||
@@ -63,6 +75,18 @@ class HuggingFaceDatasetCollector(HTTPCollector):
|
||||
data_type = "dataset"
|
||||
base_url = "https://huggingface.co/api/datasets"
|
||||
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
return self._resolved_url or self.base_url
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
from httpx import AsyncClient
|
||||
|
||||
async with AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(self.request_url, headers=self.headers)
|
||||
response.raise_for_status()
|
||||
return self.parse_response(response.json())
|
||||
|
||||
def parse_response(self, response: Dict[str, Any]) -> List[Dict[str, Any]]:
|
||||
"""Parse Hugging Face datasets API response"""
|
||||
data = []
|
||||
@@ -104,6 +128,18 @@ class HuggingFaceSpacesCollector(HTTPCollector):
|
||||
data_type = "space"
|
||||
base_url = "https://huggingface.co/api/spaces"
|
||||
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
return self._resolved_url or self.base_url
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
from httpx import AsyncClient
|
||||
|
||||
async with AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(self.request_url, headers=self.headers)
|
||||
response.raise_for_status()
|
||||
return self.parse_response(response.json())
|
||||
|
||||
def parse_response(self, response: Dict[str, Any]) -> List[Dict[str, Any]]:
|
||||
"""Parse Hugging Face Spaces API response"""
|
||||
data = []
|
||||
|
||||
79
backend/app/services/collectors/news_live_streams.py
Normal file
79
backend/app/services/collectors/news_live_streams.py
Normal file
@@ -0,0 +1,79 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
|
||||
from app.services.collectors.base import BaseCollector
|
||||
|
||||
|
||||
class NewsLiveStreamsCollector(BaseCollector):
|
||||
"""Collect normalized news live-stream sources from a JSON endpoint."""
|
||||
|
||||
name = "news_live_streams"
|
||||
priority = "P2"
|
||||
module = "L4"
|
||||
frequency_hours = 12
|
||||
data_type = "news_live_stream"
|
||||
fail_on_empty = False
|
||||
|
||||
async def fetch(self) -> list[dict[str, Any]]:
|
||||
request_url = (self._resolved_url or "").strip()
|
||||
if not request_url:
|
||||
return []
|
||||
|
||||
async with httpx.AsyncClient(timeout=45.0, follow_redirects=True) as client:
|
||||
response = await client.get(
|
||||
request_url,
|
||||
headers={
|
||||
"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
|
||||
"Accept": "application/json",
|
||||
},
|
||||
)
|
||||
response.raise_for_status()
|
||||
return self.parse_response(response.json())
|
||||
|
||||
def parse_response(self, response: Any) -> list[dict[str, Any]]:
|
||||
if isinstance(response, dict):
|
||||
candidates = response.get("sources") or response.get("streams") or response.get("data") or []
|
||||
elif isinstance(response, list):
|
||||
candidates = response
|
||||
else:
|
||||
candidates = []
|
||||
|
||||
normalized: list[dict[str, Any]] = []
|
||||
for index, item in enumerate(candidates):
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
|
||||
stream_id = item.get("id") or item.get("source_id") or item.get("slug") or f"news-live-{index + 1}"
|
||||
name = str(item.get("name") or item.get("title") or f"News Live {index + 1}").strip()
|
||||
if not name:
|
||||
continue
|
||||
|
||||
metadata = {
|
||||
"provider": item.get("provider") or item.get("publisher") or "Collector",
|
||||
"region": item.get("region") or item.get("country") or "Global",
|
||||
"language": item.get("language") or "und",
|
||||
"source_type": item.get("source_type") or "iframe",
|
||||
"embed_url": item.get("embed_url") or item.get("url") or "",
|
||||
"stream_url": item.get("stream_url") or "",
|
||||
"homepage_url": item.get("homepage_url") or item.get("source_url") or "",
|
||||
"poster_url": item.get("poster_url") or "",
|
||||
"sort_order": item.get("sort_order", 200 + index),
|
||||
"notes": item.get("notes") or item.get("description") or "",
|
||||
"is_enabled": item.get("is_enabled", True),
|
||||
}
|
||||
|
||||
normalized.append(
|
||||
{
|
||||
"source_id": str(stream_id),
|
||||
"name": name,
|
||||
"description": metadata["notes"],
|
||||
"metadata": metadata,
|
||||
"reference_date": item.get("reference_date", datetime.now(UTC).isoformat()),
|
||||
}
|
||||
)
|
||||
|
||||
return normalized
|
||||
@@ -16,6 +16,7 @@ from typing import Dict, Any, List
|
||||
from datetime import UTC, datetime
|
||||
|
||||
import httpx
|
||||
from urllib.parse import urlencode
|
||||
from app.services.collectors.base import HTTPCollector
|
||||
|
||||
|
||||
@@ -38,9 +39,13 @@ class PeeringDBIXPCollector(HTTPCollector):
|
||||
"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
|
||||
"Accept": "application/json",
|
||||
}
|
||||
# API key is added to URL as query parameter
|
||||
if PEERINGDB_API_KEY:
|
||||
self.base_url = f"{self.base_url}?key={PEERINGDB_API_KEY}"
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
base = self._resolved_url or self.base_url
|
||||
if not PEERINGDB_API_KEY:
|
||||
return base
|
||||
separator = "&" if "?" in base else "?"
|
||||
return f"{base}{separator}{urlencode({'key': PEERINGDB_API_KEY})}"
|
||||
|
||||
async def fetch_with_retry(
|
||||
self, max_retries: int = 3, base_delay: float = 2.0
|
||||
@@ -51,7 +56,7 @@ class PeeringDBIXPCollector(HTTPCollector):
|
||||
for attempt in range(max_retries):
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(self.base_url, headers=self.headers)
|
||||
response = await client.get(self.request_url, headers=self.headers)
|
||||
|
||||
if response.status_code == 429:
|
||||
# Rate limited - wait and retry with exponential backoff
|
||||
@@ -141,8 +146,13 @@ class PeeringDBNetworkCollector(HTTPCollector):
|
||||
"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
|
||||
"Accept": "application/json",
|
||||
}
|
||||
if PEERINGDB_API_KEY:
|
||||
self.base_url = f"{self.base_url}?key={PEERINGDB_API_KEY}"
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
base = self._resolved_url or self.base_url
|
||||
if not PEERINGDB_API_KEY:
|
||||
return base
|
||||
separator = "&" if "?" in base else "?"
|
||||
return f"{base}{separator}{urlencode({'key': PEERINGDB_API_KEY})}"
|
||||
|
||||
async def fetch_with_retry(
|
||||
self, max_retries: int = 3, base_delay: float = 2.0
|
||||
@@ -153,7 +163,7 @@ class PeeringDBNetworkCollector(HTTPCollector):
|
||||
for attempt in range(max_retries):
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(self.base_url, headers=self.headers)
|
||||
response = await client.get(self.request_url, headers=self.headers)
|
||||
|
||||
if response.status_code == 429:
|
||||
delay = base_delay * (2**attempt)
|
||||
@@ -244,8 +254,13 @@ class PeeringDBFacilityCollector(HTTPCollector):
|
||||
"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
|
||||
"Accept": "application/json",
|
||||
}
|
||||
if PEERINGDB_API_KEY:
|
||||
self.base_url = f"{self.base_url}?key={PEERINGDB_API_KEY}"
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
base = self._resolved_url or self.base_url
|
||||
if not PEERINGDB_API_KEY:
|
||||
return base
|
||||
separator = "&" if "?" in base else "?"
|
||||
return f"{base}{separator}{urlencode({'key': PEERINGDB_API_KEY})}"
|
||||
|
||||
async def fetch_with_retry(
|
||||
self, max_retries: int = 3, base_delay: float = 2.0
|
||||
@@ -256,7 +271,7 @@ class PeeringDBFacilityCollector(HTTPCollector):
|
||||
for attempt in range(max_retries):
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(self.base_url, headers=self.headers)
|
||||
response = await client.get(self.request_url, headers=self.headers)
|
||||
|
||||
if response.status_code == 429:
|
||||
delay = base_delay * (2**attempt)
|
||||
|
||||
@@ -33,7 +33,7 @@ class RISLiveCollector(BaseCollector):
|
||||
|
||||
def _fetch_via_stream(self) -> list[dict[str, Any]]:
|
||||
events: list[dict[str, Any]] = []
|
||||
stream_url = "https://ris-live.ripe.net/v1/stream/?format=json&client=planet-ris-live"
|
||||
stream_url = self._resolved_url or ""
|
||||
subscribe = json.dumps(
|
||||
{
|
||||
"host": "rrc00",
|
||||
|
||||
@@ -7,6 +7,7 @@ API documentation: https://www.space-track.org/documentation
|
||||
import json
|
||||
from typing import Dict, Any, List
|
||||
import httpx
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from app.services.collectors.base import BaseCollector
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
@@ -21,12 +22,30 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
data_type = "satellite_tle"
|
||||
|
||||
@property
|
||||
def base_url(self) -> str:
|
||||
def query_url(self) -> str:
|
||||
config = get_data_sources_config()
|
||||
if self._resolved_url:
|
||||
return self._resolved_url
|
||||
return config.get_yaml_url("spacetrack_tle")
|
||||
|
||||
@property
|
||||
def site_root(self) -> str:
|
||||
config = get_data_sources_config()
|
||||
configured_root = config.get_yaml_value("spacetrack.base_url")
|
||||
if isinstance(configured_root, str) and configured_root:
|
||||
return configured_root.rstrip("/")
|
||||
|
||||
parsed = urlparse(self.query_url)
|
||||
return f"{parsed.scheme}://{parsed.netloc}".rstrip("/")
|
||||
|
||||
@property
|
||||
def login_url(self) -> str:
|
||||
return f"{self.site_root}/ajaxauth/login"
|
||||
|
||||
@property
|
||||
def probe_url(self) -> str:
|
||||
return f"{self.site_root}/basicspacedata/query/class/gp/NORAD_CAT_ID/25544/format/json"
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
from app.core.config import settings
|
||||
|
||||
@@ -47,13 +66,13 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
|
||||
"Accept": "application/json, text/html, */*",
|
||||
"Accept-Language": "en-US,en;q=0.9",
|
||||
"Referer": "https://www.space-track.org/",
|
||||
"Referer": f"{self.site_root}/",
|
||||
},
|
||||
) as client:
|
||||
await client.get("https://www.space-track.org/")
|
||||
await client.get(f"{self.site_root}/")
|
||||
|
||||
login_response = await client.post(
|
||||
"https://www.space-track.org/ajaxauth/login",
|
||||
self.login_url,
|
||||
data={
|
||||
"identity": username,
|
||||
"password": password,
|
||||
@@ -69,7 +88,7 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
timeout=120.0,
|
||||
follow_redirects=True,
|
||||
) as alt_client:
|
||||
await alt_client.get("https://www.space-track.org/")
|
||||
await alt_client.get(f"{self.site_root}/")
|
||||
|
||||
form_data = {
|
||||
"username": username,
|
||||
@@ -77,7 +96,7 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
"query": "class/gp/NORAD_CAT_ID/25544/format/json",
|
||||
}
|
||||
alt_login = await alt_client.post(
|
||||
"https://www.space-track.org/ajaxauth/login",
|
||||
self.login_url,
|
||||
data={
|
||||
"identity": username,
|
||||
"password": password,
|
||||
@@ -86,9 +105,7 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
print(f"SPACETRACK: Alt login status: {alt_login.status_code}")
|
||||
|
||||
if alt_login.status_code == 200:
|
||||
tle_response = await alt_client.get(
|
||||
"https://www.space-track.org/basicspacedata/query/class/gp/NORAD_CAT_ID/25544/format/json"
|
||||
)
|
||||
tle_response = await alt_client.get(self.probe_url)
|
||||
if tle_response.status_code == 200:
|
||||
data = tle_response.json()
|
||||
print(f"SPACETRACK: Received {len(data)} records via alt method")
|
||||
@@ -98,9 +115,7 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
print(f"SPACETRACK: Login failed, using sample data")
|
||||
return self._get_sample_data()
|
||||
|
||||
tle_response = await client.get(
|
||||
"https://www.space-track.org/basicspacedata/query/class/gp/NORAD_CAT_ID/25544/format/json"
|
||||
)
|
||||
tle_response = await client.get(self.probe_url)
|
||||
print(f"SPACETRACK: TLE query status: {tle_response.status_code}")
|
||||
|
||||
if tle_response.status_code != 200:
|
||||
@@ -127,11 +142,11 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
},
|
||||
) as client:
|
||||
# First, visit the main page to get any cookies
|
||||
await client.get("https://www.space-track.org/")
|
||||
await client.get(f"{self.site_root}/")
|
||||
|
||||
# Login to get session cookie
|
||||
login_response = await client.post(
|
||||
"https://www.space-track.org/ajaxauth/login",
|
||||
self.login_url,
|
||||
data={
|
||||
"identity": username,
|
||||
"password": password,
|
||||
@@ -146,13 +161,7 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
return self._get_sample_data()
|
||||
|
||||
# Query for TLE data (get first 1000 satellites)
|
||||
tle_response = await client.get(
|
||||
"https://www.space-track.org/basicspacedata/query"
|
||||
"/class/gp"
|
||||
"/orderby/EPOCH%20desc"
|
||||
"/limit/1000"
|
||||
"/format/json"
|
||||
)
|
||||
tle_response = await client.get(self.query_url)
|
||||
print(f"SPACETRACK: TLE query status: {tle_response.status_code}")
|
||||
|
||||
if tle_response.status_code != 200:
|
||||
|
||||
@@ -11,6 +11,7 @@ from datetime import UTC, datetime
|
||||
from bs4 import BeautifulSoup
|
||||
import httpx
|
||||
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
from app.services.collectors.base import BaseCollector
|
||||
|
||||
|
||||
@@ -24,15 +25,17 @@ class TeleGeographyCableCollector(BaseCollector):
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
"""Fetch submarine cable data from Wayback Machine"""
|
||||
config = get_data_sources_config()
|
||||
# Try multiple data sources
|
||||
sources = [
|
||||
# Wayback Machine archive of TeleGeography
|
||||
"https://web.archive.org/web/2024/https://www.submarinecablemap.com/api/v3/cable",
|
||||
# Alternative: Try scraping the page
|
||||
"https://www.submarinecablemap.com",
|
||||
self._resolved_url or "",
|
||||
str(config.get_yaml_value("telegeography.archived_cable_url") or ""),
|
||||
str(config.get_yaml_value("telegeography.live_map_url") or ""),
|
||||
]
|
||||
|
||||
for url in sources:
|
||||
if not url:
|
||||
continue
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=60.0, follow_redirects=True) as client:
|
||||
response = await client.get(url)
|
||||
@@ -161,7 +164,7 @@ class TeleGeographyLandingPointCollector(BaseCollector):
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
"""Fetch landing point data from GitHub mirror"""
|
||||
url = "https://raw.githubusercontent.com/lintaojlu/submarine_cable_information/main/landing_point.json"
|
||||
url = self._resolved_url or ""
|
||||
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(url)
|
||||
@@ -225,7 +228,7 @@ class TeleGeographyCableSystemCollector(BaseCollector):
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
"""Fetch cable system data"""
|
||||
url = "https://raw.githubusercontent.com/lintaojlu/submarine_cable_information/main/cable.json"
|
||||
url = self._resolved_url or ""
|
||||
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(url)
|
||||
|
||||
@@ -10,6 +10,7 @@ from typing import Dict, Any, List
|
||||
from bs4 import BeautifulSoup
|
||||
import httpx
|
||||
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
from app.services.collectors.base import BaseCollector
|
||||
|
||||
|
||||
@@ -22,7 +23,7 @@ class TOP500Collector(BaseCollector):
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
"""Fetch TOP500 list data and enrich each row with detail-page metadata."""
|
||||
url = "https://top500.org/lists/top500/list/2025/11/"
|
||||
url = self._resolved_url or ""
|
||||
|
||||
async with httpx.AsyncClient(timeout=60.0, follow_redirects=True) as client:
|
||||
response = await client.get(url)
|
||||
@@ -48,11 +49,13 @@ class TOP500Collector(BaseCollector):
|
||||
return await asyncio.gather(*(enrich(entry) for entry in entries))
|
||||
|
||||
def _extract_system_fields(self, system_cell) -> Dict[str, str]:
|
||||
config = get_data_sources_config()
|
||||
top500_base_url = config.get_yaml_value("top500.base_url") or "https://top500.org"
|
||||
link = system_cell.find("a")
|
||||
system_name = link.get_text(" ", strip=True) if link else system_cell.get_text(" ", strip=True)
|
||||
detail_url = ""
|
||||
if link and link.get("href"):
|
||||
detail_url = f"https://top500.org{link.get('href')}"
|
||||
detail_url = f"{str(top500_base_url).rstrip('/')}{link.get('href')}"
|
||||
|
||||
manufacturer = ""
|
||||
if link and link.next_sibling:
|
||||
|
||||
490
backend/app/services/earth_news.py
Normal file
490
backend/app/services/earth_news.py
Normal file
@@ -0,0 +1,490 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
from email.utils import parsedate_to_datetime
|
||||
import hashlib
|
||||
import html
|
||||
import re
|
||||
from typing import Any
|
||||
from urllib.parse import quote
|
||||
import xml.etree.ElementTree as ET
|
||||
|
||||
import httpx
|
||||
from bs4 import BeautifulSoup
|
||||
|
||||
|
||||
USER_AGENT = "PlanetEarthNewsBoard/1.0 (+https://planet.local)"
|
||||
REQUEST_TIMEOUT = 12.0
|
||||
MAX_ITEMS_PER_SOURCE = 6
|
||||
MAX_ITEMS_TOTAL = 12
|
||||
STALE_CACHE_MAX_AGE_SECONDS = 60 * 45
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RegionProfile:
|
||||
key: str
|
||||
label: str
|
||||
query: str
|
||||
accent: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class NewsFeedSource:
|
||||
id: str
|
||||
name: str
|
||||
region: str
|
||||
feed_url: str
|
||||
homepage_url: str
|
||||
source_type: str = "rss"
|
||||
priority: int = 100
|
||||
|
||||
|
||||
@dataclass
|
||||
class ParsedNewsItem:
|
||||
id: str
|
||||
title: str
|
||||
summary: str
|
||||
url: str
|
||||
source: str
|
||||
feed_name: str
|
||||
feed_region: str
|
||||
homepage_url: str
|
||||
published_at: datetime | None
|
||||
|
||||
|
||||
@dataclass
|
||||
class CachedRegionFeed:
|
||||
region: str
|
||||
fetched_at: datetime
|
||||
items: list[ParsedNewsItem]
|
||||
sources: list[NewsFeedSource]
|
||||
|
||||
|
||||
REGION_PROFILES: dict[str, RegionProfile] = {
|
||||
"americas": RegionProfile(
|
||||
key="americas",
|
||||
label="美洲焦点",
|
||||
query='Americas geopolitics OR Latin America OR "United States" OR Canada',
|
||||
accent="#79d3ff",
|
||||
),
|
||||
"europe": RegionProfile(
|
||||
key="europe",
|
||||
label="欧洲焦点",
|
||||
query='Europe geopolitics OR EU OR NATO OR "Eastern Europe"',
|
||||
accent="#8fd4ff",
|
||||
),
|
||||
"middle-east-africa": RegionProfile(
|
||||
key="middle-east-africa",
|
||||
label="中东与非洲焦点",
|
||||
query='"Middle East" OR Africa geopolitics OR Red Sea OR Gulf',
|
||||
accent="#ffb56a",
|
||||
),
|
||||
"asia-pacific": RegionProfile(
|
||||
key="asia-pacific",
|
||||
label="亚太焦点",
|
||||
query='"Asia Pacific" OR Indo-Pacific OR China OR Japan OR Korea OR ASEAN',
|
||||
accent="#78f2cf",
|
||||
),
|
||||
"global": RegionProfile(
|
||||
key="global",
|
||||
label="全球焦点",
|
||||
query='"world news" OR geopolitics OR "global affairs"',
|
||||
accent="#d6e6ff",
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def _google_news_feed(query: str, *, hl: str, gl: str, ceid: str) -> str:
|
||||
return (
|
||||
"https://news.google.com/rss/search?q="
|
||||
+ quote(query, safe="")
|
||||
+ f"&hl={hl}&gl={gl}&ceid={ceid}"
|
||||
)
|
||||
|
||||
|
||||
NEWS_FEED_SOURCES: tuple[NewsFeedSource, ...] = (
|
||||
NewsFeedSource(
|
||||
id="bbc-world",
|
||||
name="BBC World",
|
||||
region="global",
|
||||
feed_url="https://feeds.bbci.co.uk/news/world/rss.xml",
|
||||
homepage_url="https://www.bbc.com/news/world",
|
||||
priority=10,
|
||||
),
|
||||
NewsFeedSource(
|
||||
id="dw-top",
|
||||
name="DW Top Stories",
|
||||
region="europe",
|
||||
feed_url="https://rss.dw.com/rdf/rss-en-top",
|
||||
homepage_url="https://www.dw.com/en/top-stories/s-9097",
|
||||
priority=20,
|
||||
),
|
||||
NewsFeedSource(
|
||||
id="global-scan",
|
||||
name="Global Monitor / World",
|
||||
region="global",
|
||||
feed_url=_google_news_feed(
|
||||
REGION_PROFILES["global"].query,
|
||||
hl="en-US",
|
||||
gl="US",
|
||||
ceid="US:en",
|
||||
),
|
||||
homepage_url="https://news.google.com/",
|
||||
source_type="aggregated",
|
||||
priority=30,
|
||||
),
|
||||
NewsFeedSource(
|
||||
id="google-americas",
|
||||
name="Global Monitor / Americas",
|
||||
region="americas",
|
||||
feed_url=_google_news_feed(
|
||||
REGION_PROFILES["americas"].query,
|
||||
hl="en-US",
|
||||
gl="US",
|
||||
ceid="US:en",
|
||||
),
|
||||
homepage_url="https://news.google.com/",
|
||||
source_type="aggregated",
|
||||
priority=40,
|
||||
),
|
||||
NewsFeedSource(
|
||||
id="google-europe",
|
||||
name="Global Monitor / Europe",
|
||||
region="europe",
|
||||
feed_url=_google_news_feed(
|
||||
REGION_PROFILES["europe"].query,
|
||||
hl="en-GB",
|
||||
gl="GB",
|
||||
ceid="GB:en",
|
||||
),
|
||||
homepage_url="https://news.google.com/",
|
||||
source_type="aggregated",
|
||||
priority=40,
|
||||
),
|
||||
NewsFeedSource(
|
||||
id="google-mea",
|
||||
name="Global Monitor / MEA",
|
||||
region="middle-east-africa",
|
||||
feed_url=_google_news_feed(
|
||||
REGION_PROFILES["middle-east-africa"].query,
|
||||
hl="en-US",
|
||||
gl="US",
|
||||
ceid="US:en",
|
||||
),
|
||||
homepage_url="https://news.google.com/",
|
||||
source_type="aggregated",
|
||||
priority=40,
|
||||
),
|
||||
NewsFeedSource(
|
||||
id="google-apac",
|
||||
name="Global Monitor / APAC",
|
||||
region="asia-pacific",
|
||||
feed_url=_google_news_feed(
|
||||
REGION_PROFILES["asia-pacific"].query,
|
||||
hl="en-SG",
|
||||
gl="SG",
|
||||
ceid="SG:en",
|
||||
),
|
||||
homepage_url="https://news.google.com/",
|
||||
source_type="aggregated",
|
||||
priority=40,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
_REGION_CACHE: dict[str, CachedRegionFeed] = {}
|
||||
|
||||
|
||||
def determine_focus_region(lat: float | None, lon: float | None) -> str:
|
||||
if lat is None or lon is None:
|
||||
return "global"
|
||||
if -170 <= lon <= -30:
|
||||
return "americas"
|
||||
if -30 < lon <= 45:
|
||||
return "europe" if lat >= 30 else "middle-east-africa"
|
||||
if 45 < lon <= 150:
|
||||
return "middle-east-africa" if lat < 10 else "asia-pacific"
|
||||
return "asia-pacific"
|
||||
|
||||
|
||||
def get_region_profile(region: str) -> RegionProfile:
|
||||
return REGION_PROFILES.get(region, REGION_PROFILES["global"])
|
||||
|
||||
|
||||
def get_sources_for_region(region: str) -> list[NewsFeedSource]:
|
||||
return sorted(
|
||||
[source for source in NEWS_FEED_SOURCES if source.region in {"global", region}],
|
||||
key=lambda source: (source.priority, source.name),
|
||||
)
|
||||
|
||||
|
||||
def _strip_html(value: str) -> str:
|
||||
if not value:
|
||||
return ""
|
||||
soup = BeautifulSoup(value, "html.parser")
|
||||
return re.sub(r"\s+", " ", soup.get_text(" ", strip=True)).strip()
|
||||
|
||||
|
||||
def _truncate(value: str, limit: int = 180) -> str:
|
||||
text = value.strip()
|
||||
if len(text) <= limit:
|
||||
return text
|
||||
return text[: limit - 1].rstrip() + "…"
|
||||
|
||||
|
||||
def _normalize_source_name(raw: str, fallback: str) -> str:
|
||||
text = html.unescape((raw or "").strip())
|
||||
if " - " in text:
|
||||
return text.split(" - ")[-1].strip() or fallback
|
||||
return text or fallback
|
||||
|
||||
|
||||
def _parse_datetime(raw: str | None) -> datetime | None:
|
||||
if not raw:
|
||||
return None
|
||||
text = raw.strip()
|
||||
if not text:
|
||||
return None
|
||||
|
||||
for parser in (
|
||||
lambda value: parsedate_to_datetime(value),
|
||||
lambda value: datetime.fromisoformat(value.replace("Z", "+00:00")),
|
||||
):
|
||||
try:
|
||||
parsed = parser(text)
|
||||
if parsed.tzinfo is None:
|
||||
parsed = parsed.replace(tzinfo=UTC)
|
||||
return parsed.astimezone(UTC)
|
||||
except Exception:
|
||||
continue
|
||||
return None
|
||||
|
||||
|
||||
def _extract_item_text(element: ET.Element, *names: str) -> str:
|
||||
for name in names:
|
||||
node = element.find(name)
|
||||
if node is not None and node.text:
|
||||
return node.text.strip()
|
||||
return ""
|
||||
|
||||
|
||||
def _parse_feed_entries(xml_text: str, source: NewsFeedSource) -> list[ParsedNewsItem]:
|
||||
root = ET.fromstring(xml_text)
|
||||
items: list[ParsedNewsItem] = []
|
||||
|
||||
rss_items = root.findall("./channel/item")
|
||||
atom_entries = root.findall("{http://www.w3.org/2005/Atom}entry")
|
||||
nodes = rss_items or atom_entries
|
||||
|
||||
for node in nodes[:MAX_ITEMS_PER_SOURCE]:
|
||||
if node.tag.endswith("entry"):
|
||||
title = _extract_item_text(node, "{http://www.w3.org/2005/Atom}title")
|
||||
summary = _extract_item_text(
|
||||
node,
|
||||
"{http://www.w3.org/2005/Atom}summary",
|
||||
"{http://www.w3.org/2005/Atom}content",
|
||||
)
|
||||
link_node = node.find("{http://www.w3.org/2005/Atom}link")
|
||||
link = link_node.get("href", "").strip() if link_node is not None else ""
|
||||
published = _extract_item_text(
|
||||
node,
|
||||
"{http://www.w3.org/2005/Atom}updated",
|
||||
"{http://www.w3.org/2005/Atom}published",
|
||||
)
|
||||
else:
|
||||
title = _extract_item_text(node, "title")
|
||||
summary = _extract_item_text(node, "description", "content")
|
||||
link = _extract_item_text(node, "link")
|
||||
published = _extract_item_text(node, "pubDate", "published", "updated")
|
||||
|
||||
clean_title = html.unescape(title).strip()
|
||||
clean_summary = _truncate(_strip_html(summary), 180)
|
||||
if not clean_title or not link:
|
||||
continue
|
||||
|
||||
item_source = _normalize_source_name(clean_title, source.name)
|
||||
display_title = clean_title
|
||||
if source.source_type == "aggregated" and " - " in clean_title:
|
||||
parts = clean_title.rsplit(" - ", 1)
|
||||
display_title = parts[0].strip()
|
||||
item_source = _normalize_source_name(parts[1], source.name)
|
||||
|
||||
items.append(
|
||||
ParsedNewsItem(
|
||||
id=f"{source.id}:{hashlib.sha1(link.encode('utf-8')).hexdigest()[:12]}",
|
||||
title=display_title,
|
||||
summary=clean_summary,
|
||||
url=link,
|
||||
source=item_source,
|
||||
feed_name=source.name,
|
||||
feed_region=source.region,
|
||||
homepage_url=source.homepage_url,
|
||||
published_at=_parse_datetime(published),
|
||||
)
|
||||
)
|
||||
|
||||
return items
|
||||
|
||||
|
||||
def _serialize_sources(sources: list[NewsFeedSource]) -> list[dict[str, Any]]:
|
||||
return [
|
||||
{
|
||||
"id": source.id,
|
||||
"name": source.name,
|
||||
"region": source.region,
|
||||
"homepage_url": source.homepage_url,
|
||||
}
|
||||
for source in sources
|
||||
]
|
||||
|
||||
|
||||
def _serialize_item(item: ParsedNewsItem, *, active_region: str) -> dict[str, Any]:
|
||||
published_at = item.published_at
|
||||
return {
|
||||
"id": item.id,
|
||||
"title": item.title,
|
||||
"summary": item.summary,
|
||||
"url": item.url,
|
||||
"source": item.source,
|
||||
"feed_name": item.feed_name,
|
||||
"region": item.feed_region,
|
||||
"homepage_url": item.homepage_url,
|
||||
"published_at": published_at.isoformat().replace("+00:00", "Z") if published_at else None,
|
||||
"is_focus_match": item.feed_region == active_region,
|
||||
}
|
||||
|
||||
|
||||
def _build_payload(
|
||||
*,
|
||||
lat: float | None,
|
||||
lon: float | None,
|
||||
active_region: str,
|
||||
items: list[ParsedNewsItem],
|
||||
sources: list[NewsFeedSource],
|
||||
errors: list[str],
|
||||
stale: bool,
|
||||
generated_at: datetime | None = None,
|
||||
) -> dict[str, Any]:
|
||||
profile = get_region_profile(active_region)
|
||||
timestamp = generated_at or datetime.now(UTC)
|
||||
return {
|
||||
"generated_at": timestamp.isoformat().replace("+00:00", "Z"),
|
||||
"focus": {
|
||||
"lat": lat,
|
||||
"lon": lon,
|
||||
"region": active_region,
|
||||
"label": profile.label,
|
||||
"accent": profile.accent,
|
||||
},
|
||||
"sources": _serialize_sources(sources),
|
||||
"items": [_serialize_item(item, active_region=active_region) for item in items],
|
||||
"errors": errors,
|
||||
"stale": stale,
|
||||
}
|
||||
|
||||
|
||||
def _rank_and_trim_items(items: list[ParsedNewsItem], *, active_region: str) -> list[ParsedNewsItem]:
|
||||
deduped: dict[str, ParsedNewsItem] = {}
|
||||
for item in items:
|
||||
key = item.url.strip() or item.title.strip().lower()
|
||||
if key not in deduped:
|
||||
deduped[key] = item
|
||||
|
||||
return sorted(
|
||||
deduped.values(),
|
||||
key=lambda item: (
|
||||
item.feed_region != active_region,
|
||||
item.published_at is None,
|
||||
-(item.published_at.timestamp() if item.published_at else 0),
|
||||
item.feed_name,
|
||||
),
|
||||
)[:MAX_ITEMS_TOTAL]
|
||||
|
||||
|
||||
def _get_cached_region_feed(region: str) -> CachedRegionFeed | None:
|
||||
cached = _REGION_CACHE.get(region)
|
||||
if not cached:
|
||||
return None
|
||||
age_seconds = (datetime.now(UTC) - cached.fetched_at).total_seconds()
|
||||
if age_seconds > STALE_CACHE_MAX_AGE_SECONDS:
|
||||
return None
|
||||
return cached
|
||||
|
||||
|
||||
def _store_region_cache(region: str, *, items: list[ParsedNewsItem], sources: list[NewsFeedSource]) -> None:
|
||||
_REGION_CACHE[region] = CachedRegionFeed(
|
||||
region=region,
|
||||
fetched_at=datetime.now(UTC),
|
||||
items=list(items),
|
||||
sources=list(sources),
|
||||
)
|
||||
|
||||
|
||||
async def _fetch_source(
|
||||
client: httpx.AsyncClient,
|
||||
source: NewsFeedSource,
|
||||
) -> tuple[NewsFeedSource, list[ParsedNewsItem], str | None]:
|
||||
try:
|
||||
response = await client.get(source.feed_url)
|
||||
response.raise_for_status()
|
||||
return source, _parse_feed_entries(response.text, source), None
|
||||
except Exception as exc:
|
||||
return source, [], str(exc)
|
||||
|
||||
|
||||
async def get_earth_news_payload(lat: float | None = None, lon: float | None = None) -> dict[str, Any]:
|
||||
active_region = determine_focus_region(lat, lon)
|
||||
sources = get_sources_for_region(active_region)
|
||||
errors: list[str] = []
|
||||
|
||||
async with httpx.AsyncClient(
|
||||
timeout=REQUEST_TIMEOUT,
|
||||
follow_redirects=True,
|
||||
headers={"User-Agent": USER_AGENT},
|
||||
) as client:
|
||||
results = await asyncio.gather(*(_fetch_source(client, source) for source in sources))
|
||||
|
||||
fetched_items: list[ParsedNewsItem] = []
|
||||
for source, items, error in results:
|
||||
if error:
|
||||
errors.append(f"{source.name}: {error}")
|
||||
continue
|
||||
fetched_items.extend(items)
|
||||
|
||||
ranked_items = _rank_and_trim_items(fetched_items, active_region=active_region)
|
||||
if ranked_items:
|
||||
_store_region_cache(active_region, items=ranked_items, sources=sources)
|
||||
return _build_payload(
|
||||
lat=lat,
|
||||
lon=lon,
|
||||
active_region=active_region,
|
||||
items=ranked_items,
|
||||
sources=sources,
|
||||
errors=errors,
|
||||
stale=False,
|
||||
)
|
||||
|
||||
cached = _get_cached_region_feed(active_region)
|
||||
if cached:
|
||||
return _build_payload(
|
||||
lat=lat,
|
||||
lon=lon,
|
||||
active_region=active_region,
|
||||
items=cached.items,
|
||||
sources=cached.sources,
|
||||
errors=errors,
|
||||
stale=True,
|
||||
generated_at=cached.fetched_at,
|
||||
)
|
||||
|
||||
return _build_payload(
|
||||
lat=lat,
|
||||
lon=lon,
|
||||
active_region=active_region,
|
||||
items=[],
|
||||
sources=sources,
|
||||
errors=errors,
|
||||
stale=False,
|
||||
)
|
||||
694
backend/app/services/playground_chat_service.py
Normal file
694
backend/app/services/playground_chat_service.py
Normal file
@@ -0,0 +1,694 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from collections.abc import Sequence
|
||||
from datetime import UTC, datetime
|
||||
from time import perf_counter
|
||||
from uuid import uuid4
|
||||
|
||||
from fastapi import HTTPException, status
|
||||
from sqlalchemy import func, select, update
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.db.session import async_session_factory
|
||||
from app.models.playground_message import PlaygroundMessage
|
||||
from app.models.playground_session import PlaygroundSession
|
||||
from app.schemas.ai import (
|
||||
PlaygroundMessageEditRequest,
|
||||
PlaygroundMessageActionResponse,
|
||||
PlaygroundMessageCreateRequest,
|
||||
PlaygroundMessageRecord,
|
||||
PlaygroundMessageResendRequest,
|
||||
PlaygroundMessageStopRequest,
|
||||
PlaygroundSessionResponse,
|
||||
PlaygroundSessionState,
|
||||
PlaygroundSessionUpsertRequest,
|
||||
PlaygroundThreadResponse,
|
||||
SituationalAnalysisRequest,
|
||||
)
|
||||
from app.services.ai_client import AIProviderClient
|
||||
from app.services.playground_session_store import _to_response as session_to_response
|
||||
from app.services.playground_session_store import upsert_playground_session
|
||||
|
||||
STREAM_CHUNK_SIZE = 24
|
||||
STREAM_INTERVAL_SECONDS = 0.08
|
||||
THINKING_PREVIEW_SECONDS = 2.6
|
||||
|
||||
|
||||
class _ActiveRun:
|
||||
def __init__(self, task: asyncio.Task[None]) -> None:
|
||||
self.task = task
|
||||
self.stop_requested = asyncio.Event()
|
||||
|
||||
|
||||
_ACTIVE_RUNS: dict[str, _ActiveRun] = {}
|
||||
|
||||
|
||||
async def _get_session_by_key(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
session_key: str,
|
||||
) -> PlaygroundSession | None:
|
||||
result = await db.execute(
|
||||
select(PlaygroundSession).where(
|
||||
PlaygroundSession.user_id == user_id,
|
||||
PlaygroundSession.session_key == session_key,
|
||||
)
|
||||
)
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
|
||||
async def _require_session(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
session_key: str,
|
||||
) -> PlaygroundSession:
|
||||
session = await _get_session_by_key(db, user_id=user_id, session_key=session_key)
|
||||
if session is None:
|
||||
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Playground session not found")
|
||||
return session
|
||||
|
||||
|
||||
async def _require_visible_message(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
public_id: str,
|
||||
role: str | None = None,
|
||||
) -> PlaygroundMessage:
|
||||
conditions = [
|
||||
PlaygroundMessage.user_id == user_id,
|
||||
PlaygroundMessage.public_id == public_id,
|
||||
PlaygroundMessage.is_visible.is_(True),
|
||||
]
|
||||
if role is not None:
|
||||
conditions.append(PlaygroundMessage.role == role)
|
||||
|
||||
result = await db.execute(select(PlaygroundMessage).where(*conditions))
|
||||
message = result.scalar_one_or_none()
|
||||
if message is None:
|
||||
detail = "User message not found" if role == "user" else "Playground message not found"
|
||||
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail=detail)
|
||||
return message
|
||||
|
||||
|
||||
def _message_to_record(message: PlaygroundMessage, parent_public_id: str | None = None) -> PlaygroundMessageRecord:
|
||||
return PlaygroundMessageRecord(
|
||||
id=message.public_id,
|
||||
role=message.role,
|
||||
kind=message.kind,
|
||||
status=message.status,
|
||||
title=message.title,
|
||||
content=message.content or "",
|
||||
thinking_content=message.thinking_content or "",
|
||||
meta=list(message.meta or []),
|
||||
markdown=message.role != "system",
|
||||
provider=message.provider,
|
||||
model=message.model,
|
||||
request_id=message.request_id,
|
||||
raw_response=dict(message.raw_response or {}),
|
||||
content_blocks=list(message.content_blocks or []),
|
||||
text_blocks=list(message.text_blocks or []),
|
||||
thinking_blocks=list(message.thinking_blocks or []),
|
||||
parent_message_id=parent_public_id,
|
||||
created_at=message.created_at.isoformat(),
|
||||
updated_at=message.updated_at.isoformat(),
|
||||
)
|
||||
|
||||
|
||||
async def _ensure_session(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
session_key: str,
|
||||
title: str,
|
||||
state: PlaygroundSessionState | None = None,
|
||||
) -> PlaygroundSession:
|
||||
result = await db.execute(
|
||||
select(PlaygroundSession).where(
|
||||
PlaygroundSession.user_id == user_id,
|
||||
PlaygroundSession.session_key == session_key,
|
||||
)
|
||||
)
|
||||
session = result.scalar_one_or_none()
|
||||
if session is not None:
|
||||
if title:
|
||||
session.title = title[:200]
|
||||
if state is not None:
|
||||
session.state = state.model_dump(mode="json")
|
||||
await db.flush()
|
||||
await db.refresh(session)
|
||||
return session
|
||||
|
||||
payload = PlaygroundSessionUpsertRequest(
|
||||
session_key=session_key,
|
||||
title=title[:200],
|
||||
state=state or PlaygroundSessionState(title=title[:200]),
|
||||
)
|
||||
await upsert_playground_session(db, user_id=user_id, payload=payload)
|
||||
result = await db.execute(
|
||||
select(PlaygroundSession).where(
|
||||
PlaygroundSession.user_id == user_id,
|
||||
PlaygroundSession.session_key == session_key,
|
||||
)
|
||||
)
|
||||
return result.scalar_one()
|
||||
|
||||
|
||||
async def _list_visible_messages(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
session_id: int,
|
||||
) -> list[PlaygroundMessage]:
|
||||
result = await db.execute(
|
||||
select(PlaygroundMessage)
|
||||
.where(
|
||||
PlaygroundMessage.session_id == session_id,
|
||||
PlaygroundMessage.is_visible.is_(True),
|
||||
)
|
||||
.order_by(PlaygroundMessage.sort_order.asc(), PlaygroundMessage.id.asc())
|
||||
)
|
||||
return list(result.scalars().all())
|
||||
|
||||
|
||||
async def _build_thread_response(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
session: PlaygroundSession,
|
||||
) -> PlaygroundThreadResponse:
|
||||
messages = await _list_visible_messages(db, session_id=session.id)
|
||||
id_map = {item.id: item.public_id for item in messages}
|
||||
return PlaygroundThreadResponse(
|
||||
session=session_to_response(session),
|
||||
messages=[_message_to_record(item, id_map.get(item.parent_message_id)) for item in messages],
|
||||
)
|
||||
|
||||
|
||||
async def get_thread(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
session_key: str,
|
||||
) -> PlaygroundThreadResponse | None:
|
||||
session = await _get_session_by_key(db, user_id=user_id, session_key=session_key)
|
||||
if session is None:
|
||||
return None
|
||||
return await _build_thread_response(db, session=session)
|
||||
|
||||
|
||||
async def _build_action_response(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
session: PlaygroundSession,
|
||||
active_message_id: str | None = None,
|
||||
) -> PlaygroundMessageActionResponse:
|
||||
thread = await _build_thread_response(db, session=session)
|
||||
return PlaygroundMessageActionResponse(
|
||||
session=thread.session,
|
||||
messages=thread.messages,
|
||||
active_message_id=active_message_id,
|
||||
)
|
||||
|
||||
|
||||
def _collect_constraints(raw_constraints: str) -> list[str]:
|
||||
return [item.strip() for item in raw_constraints.split("\n") if item.strip()]
|
||||
|
||||
|
||||
async def _next_sort_order(db: AsyncSession, session_id: int) -> int:
|
||||
result = await db.execute(
|
||||
select(func.max(PlaygroundMessage.sort_order)).where(PlaygroundMessage.session_id == session_id)
|
||||
)
|
||||
current = result.scalar_one_or_none()
|
||||
return int(current or 0)
|
||||
|
||||
|
||||
async def _set_session_state(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
session: PlaygroundSession,
|
||||
payload: PlaygroundMessageCreateRequest,
|
||||
) -> PlaygroundSession:
|
||||
session.state = PlaygroundSessionState(
|
||||
messages=[],
|
||||
selectedPresetKey=payload.selected_preset_key,
|
||||
title=payload.title,
|
||||
objective=payload.objective,
|
||||
constraints=payload.constraints,
|
||||
inputValue="",
|
||||
analysis=None,
|
||||
latestAnalysisMessageId=None,
|
||||
analysisMeta={},
|
||||
helpExpanded=payload.help_expanded,
|
||||
).model_dump(mode="json")
|
||||
session.title = payload.title[:200]
|
||||
await db.flush()
|
||||
await db.refresh(session)
|
||||
return session
|
||||
|
||||
|
||||
def _spawn_assistant_run(
|
||||
*,
|
||||
user_id: int,
|
||||
session_id: int,
|
||||
session_key: str,
|
||||
user_message_id: int,
|
||||
assistant_message_id: int,
|
||||
assistant_public_id: str,
|
||||
payload: PlaygroundMessageCreateRequest,
|
||||
provider_client: AIProviderClient,
|
||||
) -> None:
|
||||
task = asyncio.create_task(
|
||||
_run_assistant_message(
|
||||
user_id=user_id,
|
||||
session_id=session_id,
|
||||
session_key=session_key,
|
||||
user_message_id=user_message_id,
|
||||
assistant_message_id=assistant_message_id,
|
||||
payload=payload,
|
||||
provider_client=provider_client,
|
||||
)
|
||||
)
|
||||
_ACTIVE_RUNS[assistant_public_id] = _ActiveRun(task)
|
||||
|
||||
|
||||
async def create_turn(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
payload: PlaygroundMessageCreateRequest,
|
||||
provider_client: AIProviderClient,
|
||||
) -> PlaygroundMessageActionResponse:
|
||||
session = await _ensure_session(
|
||||
db,
|
||||
user_id=user_id,
|
||||
session_key=payload.session_key,
|
||||
title=payload.title,
|
||||
state=PlaygroundSessionState(
|
||||
selectedPresetKey=payload.selected_preset_key,
|
||||
title=payload.title,
|
||||
objective=payload.objective,
|
||||
constraints=payload.constraints,
|
||||
inputValue="",
|
||||
helpExpanded=payload.help_expanded,
|
||||
),
|
||||
)
|
||||
session = await _set_session_state(db, session=session, payload=payload)
|
||||
base_order = await _next_sort_order(db, session.id)
|
||||
|
||||
user_message = PlaygroundMessage(
|
||||
public_id=uuid4().hex,
|
||||
session_id=session.id,
|
||||
user_id=user_id,
|
||||
role="user",
|
||||
kind="message",
|
||||
status="done",
|
||||
title=payload.selected_preset_key,
|
||||
content=payload.input,
|
||||
meta=[payload.title],
|
||||
sort_order=base_order + 10,
|
||||
)
|
||||
assistant_message = PlaygroundMessage(
|
||||
public_id=uuid4().hex,
|
||||
session_id=session.id,
|
||||
user_id=user_id,
|
||||
parent_message_id=None,
|
||||
role="assistant",
|
||||
kind="thinking",
|
||||
status="pending",
|
||||
title="AI 回应",
|
||||
content="",
|
||||
thinking_content="",
|
||||
meta=[],
|
||||
sort_order=base_order + 20,
|
||||
)
|
||||
db.add(user_message)
|
||||
await db.flush()
|
||||
assistant_message.parent_message_id = user_message.id
|
||||
db.add(assistant_message)
|
||||
await db.flush()
|
||||
await db.refresh(user_message)
|
||||
await db.refresh(assistant_message)
|
||||
await db.commit()
|
||||
await db.refresh(session)
|
||||
await db.refresh(user_message)
|
||||
await db.refresh(assistant_message)
|
||||
|
||||
_spawn_assistant_run(
|
||||
user_id=user_id,
|
||||
session_id=session.id,
|
||||
session_key=payload.session_key,
|
||||
user_message_id=user_message.id,
|
||||
assistant_message_id=assistant_message.id,
|
||||
assistant_public_id=assistant_message.public_id,
|
||||
payload=payload,
|
||||
provider_client=provider_client,
|
||||
)
|
||||
|
||||
return await _build_action_response(
|
||||
db,
|
||||
session=session,
|
||||
active_message_id=assistant_message.public_id,
|
||||
)
|
||||
|
||||
|
||||
async def _create_assistant_retry_turn(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
session: PlaygroundSession,
|
||||
user_message: PlaygroundMessage,
|
||||
payload: PlaygroundMessageCreateRequest,
|
||||
provider_client: AIProviderClient,
|
||||
) -> PlaygroundMessageActionResponse:
|
||||
base_order = await _next_sort_order(db, session.id)
|
||||
assistant_message = PlaygroundMessage(
|
||||
public_id=uuid4().hex,
|
||||
session_id=session.id,
|
||||
user_id=user_id,
|
||||
parent_message_id=user_message.id,
|
||||
role="assistant",
|
||||
kind="thinking",
|
||||
status="pending",
|
||||
title="AI 回应",
|
||||
content="",
|
||||
thinking_content="",
|
||||
meta=[],
|
||||
sort_order=base_order + 10,
|
||||
)
|
||||
db.add(assistant_message)
|
||||
await db.flush()
|
||||
await db.commit()
|
||||
await db.refresh(session)
|
||||
await db.refresh(assistant_message)
|
||||
|
||||
_spawn_assistant_run(
|
||||
user_id=user_id,
|
||||
session_id=session.id,
|
||||
session_key=payload.session_key,
|
||||
user_message_id=user_message.id,
|
||||
assistant_message_id=assistant_message.id,
|
||||
assistant_public_id=assistant_message.public_id,
|
||||
payload=payload,
|
||||
provider_client=provider_client,
|
||||
)
|
||||
|
||||
return await _build_action_response(
|
||||
db,
|
||||
session=session,
|
||||
active_message_id=assistant_message.public_id,
|
||||
)
|
||||
|
||||
|
||||
async def stop_message(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
payload: PlaygroundMessageStopRequest,
|
||||
) -> PlaygroundMessageActionResponse:
|
||||
session = await _require_session(db, user_id=user_id, session_key=payload.session_key)
|
||||
message = await _require_visible_message(db, user_id=user_id, public_id=payload.message_id)
|
||||
|
||||
if message.status not in {"pending", "thinking", "answering"}:
|
||||
return await _build_action_response(db, session=session)
|
||||
|
||||
active_run = _ACTIVE_RUNS.get(message.public_id)
|
||||
if active_run is not None:
|
||||
active_run.stop_requested.set()
|
||||
active_run.task.cancel()
|
||||
|
||||
message.status = "stopped"
|
||||
if "已手动停止生成" not in (message.meta or []):
|
||||
message.meta = [*(message.meta or []), "已手动停止生成"]
|
||||
await db.flush()
|
||||
await db.commit()
|
||||
await db.refresh(message)
|
||||
|
||||
return await _build_action_response(db, session=session)
|
||||
|
||||
|
||||
async def resend_turn(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
payload: PlaygroundMessageResendRequest,
|
||||
provider_client: AIProviderClient,
|
||||
) -> PlaygroundMessageActionResponse:
|
||||
session = await _require_session(db, user_id=user_id, session_key=payload.session_key)
|
||||
user_message = await _require_visible_message(
|
||||
db,
|
||||
user_id=user_id,
|
||||
public_id=payload.user_message_id,
|
||||
role="user",
|
||||
)
|
||||
|
||||
later_messages = await db.execute(
|
||||
select(PlaygroundMessage).where(
|
||||
PlaygroundMessage.session_id == session.id,
|
||||
PlaygroundMessage.sort_order > user_message.sort_order,
|
||||
PlaygroundMessage.is_visible.is_(True),
|
||||
)
|
||||
)
|
||||
for item in later_messages.scalars().all():
|
||||
item.is_visible = False
|
||||
if item.status in {"pending", "thinking", "answering"}:
|
||||
active_run = _ACTIVE_RUNS.get(item.public_id)
|
||||
if active_run is not None:
|
||||
active_run.stop_requested.set()
|
||||
active_run.task.cancel()
|
||||
await db.flush()
|
||||
await db.commit()
|
||||
|
||||
session_state = PlaygroundSessionState.model_validate(session.state or {})
|
||||
create_payload = PlaygroundMessageCreateRequest(
|
||||
session_key=payload.session_key,
|
||||
title=session_state.title or session.title,
|
||||
objective=session_state.objective or "继续当前对话",
|
||||
constraints=session_state.constraints or "",
|
||||
input=user_message.content,
|
||||
selected_preset_key=session_state.selectedPresetKey or "bgp-brief",
|
||||
help_expanded=session_state.helpExpanded,
|
||||
)
|
||||
return await _create_assistant_retry_turn(
|
||||
db,
|
||||
user_id=user_id,
|
||||
session=session,
|
||||
user_message=user_message,
|
||||
payload=create_payload,
|
||||
provider_client=provider_client,
|
||||
)
|
||||
|
||||
|
||||
async def edit_user_message(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
payload: PlaygroundMessageEditRequest,
|
||||
) -> PlaygroundMessageActionResponse:
|
||||
session = await _require_session(db, user_id=user_id, session_key=payload.session_key)
|
||||
user_message = await _require_visible_message(
|
||||
db,
|
||||
user_id=user_id,
|
||||
public_id=payload.user_message_id,
|
||||
role="user",
|
||||
)
|
||||
|
||||
user_message.content = payload.content.strip()
|
||||
await db.flush()
|
||||
await db.commit()
|
||||
await db.refresh(user_message)
|
||||
|
||||
return await _build_action_response(db, session=session)
|
||||
|
||||
|
||||
async def _append_meta_if_missing(db: AsyncSession, message_id: int, meta_line: str) -> None:
|
||||
result = await db.execute(select(PlaygroundMessage).where(PlaygroundMessage.id == message_id))
|
||||
message = result.scalar_one_or_none()
|
||||
if message is None:
|
||||
return
|
||||
if meta_line not in (message.meta or []):
|
||||
message.meta = [*(message.meta or []), meta_line]
|
||||
await db.flush()
|
||||
|
||||
|
||||
async def _should_stop(message_public_id: str) -> bool:
|
||||
active_run = _ACTIVE_RUNS.get(message_public_id)
|
||||
return active_run.stop_requested.is_set() if active_run is not None else False
|
||||
|
||||
|
||||
async def _mark_message_state(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
message_id: int,
|
||||
**updates,
|
||||
) -> PlaygroundMessage:
|
||||
result = await db.execute(select(PlaygroundMessage).where(PlaygroundMessage.id == message_id))
|
||||
message = result.scalar_one()
|
||||
for key, value in updates.items():
|
||||
setattr(message, key, value)
|
||||
await db.flush()
|
||||
await db.refresh(message)
|
||||
return message
|
||||
|
||||
|
||||
def _build_conversation_history(messages: Sequence[PlaygroundMessage], current_user_message_id: int) -> list[dict]:
|
||||
history: list[dict] = []
|
||||
for item in messages:
|
||||
if item.id >= current_user_message_id:
|
||||
break
|
||||
if item.role == "system":
|
||||
continue
|
||||
history.append(
|
||||
{
|
||||
"role": item.role,
|
||||
"kind": item.kind or "message",
|
||||
"title": item.title,
|
||||
"content": item.content or "",
|
||||
}
|
||||
)
|
||||
return history[-8:]
|
||||
|
||||
|
||||
async def _run_assistant_message(
|
||||
*,
|
||||
user_id: int,
|
||||
session_id: int,
|
||||
session_key: str,
|
||||
user_message_id: int,
|
||||
assistant_message_id: int,
|
||||
payload: PlaygroundMessageCreateRequest,
|
||||
provider_client: AIProviderClient,
|
||||
) -> None:
|
||||
request_id = str(uuid4())
|
||||
started_at = perf_counter()
|
||||
assistant_public_id: str | None = None
|
||||
try:
|
||||
async with async_session_factory() as db:
|
||||
session = await db.get(PlaygroundSession, session_id)
|
||||
user_message = await db.get(PlaygroundMessage, user_message_id)
|
||||
assistant_message = await db.get(PlaygroundMessage, assistant_message_id)
|
||||
if session is None or user_message is None or assistant_message is None:
|
||||
return
|
||||
assistant_public_id = assistant_message.public_id
|
||||
|
||||
visible_messages = await _list_visible_messages(db, session_id=session_id)
|
||||
conversation_history = _build_conversation_history(visible_messages, user_message_id)
|
||||
|
||||
request_payload = SituationalAnalysisRequest(
|
||||
title=payload.title,
|
||||
objective=payload.objective,
|
||||
observations=[item.strip() for item in payload.input.split("\n") if item.strip()],
|
||||
constraints=_collect_constraints(payload.constraints),
|
||||
context={
|
||||
"source": "playground",
|
||||
"preset": payload.selected_preset_key,
|
||||
"conversation_history": conversation_history,
|
||||
"history_size": len(conversation_history),
|
||||
},
|
||||
thinking={"type": "enabled"},
|
||||
)
|
||||
|
||||
analysis = await provider_client.analyze(request_payload, request_id=request_id)
|
||||
|
||||
async with async_session_factory() as db:
|
||||
assistant_message = await _mark_message_state(
|
||||
db,
|
||||
message_id=assistant_message_id,
|
||||
status="thinking" if analysis.thinking_blocks else "answering",
|
||||
title=f"{analysis.provider} / {analysis.model}",
|
||||
provider=analysis.provider,
|
||||
model=analysis.model,
|
||||
request_id=request_id,
|
||||
raw_response=analysis.raw_response,
|
||||
content_blocks=[item.model_dump(mode="json") for item in analysis.content_blocks],
|
||||
text_blocks=analysis.text_blocks,
|
||||
thinking_blocks=analysis.thinking_blocks,
|
||||
thinking_content="\n\n".join(analysis.thinking_blocks).strip(),
|
||||
)
|
||||
session = await db.get(PlaygroundSession, session_id)
|
||||
if session is not None:
|
||||
session_state = PlaygroundSessionState.model_validate(session.state or {})
|
||||
session.state = session_state.model_copy(
|
||||
update={
|
||||
"latestAnalysisMessageId": assistant_message.public_id,
|
||||
"analysis": analysis.model_dump(mode="json"),
|
||||
}
|
||||
).model_dump(mode="json")
|
||||
await db.flush()
|
||||
await db.commit()
|
||||
|
||||
if assistant_public_id and analysis.thinking_blocks:
|
||||
await asyncio.sleep(THINKING_PREVIEW_SECONDS)
|
||||
if await _should_stop(assistant_public_id):
|
||||
return
|
||||
|
||||
content = analysis.content or ""
|
||||
cursor = 0
|
||||
while cursor < len(content):
|
||||
if assistant_public_id and await _should_stop(assistant_public_id):
|
||||
return
|
||||
cursor = min(len(content), cursor + STREAM_CHUNK_SIZE)
|
||||
async with async_session_factory() as db:
|
||||
await _mark_message_state(
|
||||
db,
|
||||
message_id=assistant_message_id,
|
||||
status="answering",
|
||||
content=content[:cursor],
|
||||
)
|
||||
await db.commit()
|
||||
await asyncio.sleep(STREAM_INTERVAL_SECONDS)
|
||||
|
||||
duration_ms = round((perf_counter() - started_at) * 1000)
|
||||
async with async_session_factory() as db:
|
||||
assistant_message = await _mark_message_state(
|
||||
db,
|
||||
message_id=assistant_message_id,
|
||||
status="done",
|
||||
content=content,
|
||||
meta=[
|
||||
f"Request ID: {request_id}",
|
||||
f"耗时: {duration_ms} ms",
|
||||
f"完成时间: {datetime.now(UTC).astimezone().isoformat()}",
|
||||
],
|
||||
)
|
||||
session = await db.get(PlaygroundSession, session_id)
|
||||
if session is not None:
|
||||
session_state = PlaygroundSessionState.model_validate(session.state or {})
|
||||
session.state = session_state.model_copy(
|
||||
update={
|
||||
"latestAnalysisMessageId": assistant_message.public_id,
|
||||
"analysis": analysis.model_dump(mode="json"),
|
||||
"analysisMeta": {
|
||||
"requestId": request_id,
|
||||
"durationMs": duration_ms,
|
||||
"completedAt": datetime.now().isoformat(),
|
||||
},
|
||||
}
|
||||
).model_dump(mode="json")
|
||||
await db.flush()
|
||||
await db.commit()
|
||||
except asyncio.CancelledError:
|
||||
async with async_session_factory() as db:
|
||||
result = await db.execute(select(PlaygroundMessage).where(PlaygroundMessage.id == assistant_message_id))
|
||||
message = result.scalar_one_or_none()
|
||||
if message is not None and message.status in {"pending", "thinking", "answering"}:
|
||||
message.status = "stopped"
|
||||
if "已手动停止生成" not in (message.meta or []):
|
||||
message.meta = [*(message.meta or []), "已手动停止生成"]
|
||||
await db.flush()
|
||||
await db.commit()
|
||||
raise
|
||||
except Exception as exc:
|
||||
async with async_session_factory() as db:
|
||||
result = await db.execute(select(PlaygroundMessage).where(PlaygroundMessage.id == assistant_message_id))
|
||||
message = result.scalar_one_or_none()
|
||||
if message is not None:
|
||||
message.status = "error"
|
||||
message.content = message.content or "分析失败,请检查 AI Provider 配置或稍后再试。"
|
||||
message.meta = [*(message.meta or []), f"错误: {type(exc).__name__}"]
|
||||
await db.flush()
|
||||
await db.commit()
|
||||
finally:
|
||||
if assistant_public_id:
|
||||
_ACTIVE_RUNS.pop(assistant_public_id, None)
|
||||
72
backend/app/services/playground_session_store.py
Normal file
72
backend/app/services/playground_session_store.py
Normal file
@@ -0,0 +1,72 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.models.playground_session import PlaygroundSession
|
||||
from app.schemas.ai import (
|
||||
PlaygroundSessionResponse,
|
||||
PlaygroundSessionState,
|
||||
PlaygroundSessionUpsertRequest,
|
||||
)
|
||||
|
||||
|
||||
def _to_response(record: PlaygroundSession) -> PlaygroundSessionResponse:
|
||||
return PlaygroundSessionResponse(
|
||||
id=str(record.id),
|
||||
session_key=record.session_key,
|
||||
title=record.title,
|
||||
state=PlaygroundSessionState.model_validate(record.state or {}),
|
||||
created_at=record.created_at.isoformat(),
|
||||
updated_at=record.updated_at.isoformat(),
|
||||
)
|
||||
|
||||
|
||||
async def get_playground_session(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
session_key: str = "default",
|
||||
) -> PlaygroundSessionResponse | None:
|
||||
result = await db.execute(
|
||||
select(PlaygroundSession).where(
|
||||
PlaygroundSession.user_id == user_id,
|
||||
PlaygroundSession.session_key == session_key,
|
||||
)
|
||||
)
|
||||
record = result.scalar_one_or_none()
|
||||
if record is None:
|
||||
return None
|
||||
return _to_response(record)
|
||||
|
||||
|
||||
async def upsert_playground_session(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
payload: PlaygroundSessionUpsertRequest,
|
||||
) -> PlaygroundSessionResponse:
|
||||
result = await db.execute(
|
||||
select(PlaygroundSession).where(
|
||||
PlaygroundSession.user_id == user_id,
|
||||
PlaygroundSession.session_key == payload.session_key,
|
||||
)
|
||||
)
|
||||
record = result.scalar_one_or_none()
|
||||
title = (payload.title or payload.state.title or "Playground 会话").strip()[:200] or "Playground 会话"
|
||||
|
||||
if record is None:
|
||||
record = PlaygroundSession(
|
||||
user_id=user_id,
|
||||
session_key=payload.session_key,
|
||||
title=title,
|
||||
state=payload.state.model_dump(mode="json"),
|
||||
)
|
||||
db.add(record)
|
||||
else:
|
||||
record.title = title
|
||||
record.state = payload.state.model_dump(mode="json")
|
||||
|
||||
await db.flush()
|
||||
await db.refresh(record)
|
||||
return _to_response(record)
|
||||
@@ -19,6 +19,30 @@ logger = logging.getLogger(__name__)
|
||||
|
||||
scheduler = AsyncIOScheduler()
|
||||
RUNNING_TASK_GUARD_TIMEOUT_MINUTES = 90
|
||||
RUNNING_COLLECTOR_TASKS: dict[str, asyncio.Task[Any]] = {}
|
||||
|
||||
|
||||
def _collector_task_name(collector_name: str) -> str:
|
||||
return f"collector:{collector_name}"
|
||||
|
||||
|
||||
def get_running_collector_task(collector_name: str) -> asyncio.Task[Any] | None:
|
||||
task = RUNNING_COLLECTOR_TASKS.get(collector_name)
|
||||
if task is not None and not task.done():
|
||||
return task
|
||||
|
||||
if task is not None and task.done():
|
||||
RUNNING_COLLECTOR_TASKS.pop(collector_name, None)
|
||||
|
||||
target_name = _collector_task_name(collector_name)
|
||||
for candidate in asyncio.all_tasks():
|
||||
if candidate.done():
|
||||
continue
|
||||
if candidate.get_name() == target_name:
|
||||
RUNNING_COLLECTOR_TASKS[collector_name] = candidate
|
||||
return candidate
|
||||
|
||||
return None
|
||||
|
||||
|
||||
async def _update_next_run_at(datasource: DataSource, session) -> None:
|
||||
@@ -133,6 +157,12 @@ async def run_collector_task(collector_name: str):
|
||||
datasource.last_status = task_result.get("status")
|
||||
await _update_next_run_at(datasource, db)
|
||||
logger.info("Collector %s completed: %s", collector_name, task_result)
|
||||
except asyncio.CancelledError:
|
||||
datasource.last_run_at = datetime.now(UTC)
|
||||
datasource.last_status = "cancelled"
|
||||
await db.commit()
|
||||
logger.warning("Collector %s cancelled by operator", collector_name)
|
||||
raise
|
||||
except Exception as exc:
|
||||
datasource.last_run_at = datetime.now(UTC)
|
||||
datasource.last_status = "failed"
|
||||
@@ -244,10 +274,37 @@ def run_collector_now(collector_name: str) -> bool:
|
||||
logger.error("Collector not found: %s", collector_name)
|
||||
return False
|
||||
|
||||
existing_task = get_running_collector_task(collector_name)
|
||||
if existing_task is not None and not existing_task.done():
|
||||
logger.warning("Collector %s is already running in-memory; skipping duplicate trigger", collector_name)
|
||||
return False
|
||||
|
||||
try:
|
||||
asyncio.create_task(run_collector_task(collector_name))
|
||||
task = asyncio.create_task(run_collector_task(collector_name), name=_collector_task_name(collector_name))
|
||||
RUNNING_COLLECTOR_TASKS[collector_name] = task
|
||||
|
||||
def _cleanup_task(done_task: asyncio.Task[Any]) -> None:
|
||||
current = RUNNING_COLLECTOR_TASKS.get(collector_name)
|
||||
if current is done_task:
|
||||
RUNNING_COLLECTOR_TASKS.pop(collector_name, None)
|
||||
|
||||
task.add_done_callback(_cleanup_task)
|
||||
logger.info("Triggered collector: %s", collector_name)
|
||||
return True
|
||||
except Exception as exc:
|
||||
logger.error("Failed to trigger collector %s: %s", collector_name, exc)
|
||||
return False
|
||||
return False
|
||||
|
||||
|
||||
async def cancel_running_collector_now(collector_name: str) -> bool:
|
||||
task = get_running_collector_task(collector_name)
|
||||
if task is None or task.done():
|
||||
RUNNING_COLLECTOR_TASKS.pop(collector_name, None)
|
||||
return False
|
||||
|
||||
task.cancel()
|
||||
try:
|
||||
await task
|
||||
except asyncio.CancelledError:
|
||||
return True
|
||||
return task.cancelled()
|
||||
|
||||
174
backend/app/services/situational_alert_ai_brief.py
Normal file
174
backend/app/services/situational_alert_ai_brief.py
Normal file
@@ -0,0 +1,174 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections import Counter
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.models.alert import Alert, AlertSeverity, AlertStatus
|
||||
from app.models.bgp_anomaly import BGPAnomaly
|
||||
from app.models.bgp_incident import BGPIncident
|
||||
from app.schemas.ai import SituationalAnalysisRequest
|
||||
from app.services.bgp_ai_brief_store import get_latest_bgp_brief_record
|
||||
|
||||
|
||||
def _format_pairs(pairs: list[tuple[str, int]], empty_text: str = "无") -> str:
|
||||
if not pairs:
|
||||
return empty_text
|
||||
return ",".join(f"{key} {value}" for key, value in pairs if key)
|
||||
|
||||
|
||||
async def build_situational_alert_brief_request(
|
||||
db: AsyncSession,
|
||||
) -> tuple[SituationalAnalysisRequest, list[str], dict[str, Any]]:
|
||||
total_alerts_result = await db.execute(select(func.count(Alert.id)))
|
||||
active_alerts_result = await db.execute(
|
||||
select(func.count(Alert.id)).where(Alert.status == AlertStatus.ACTIVE)
|
||||
)
|
||||
alert_severity_result = await db.execute(
|
||||
select(Alert.severity, func.count(Alert.id))
|
||||
.where(Alert.status == AlertStatus.ACTIVE)
|
||||
.group_by(Alert.severity)
|
||||
)
|
||||
alert_source_result = await db.execute(
|
||||
select(Alert.datasource_name, func.count(Alert.id))
|
||||
.where(Alert.status == AlertStatus.ACTIVE)
|
||||
.group_by(Alert.datasource_name)
|
||||
.order_by(func.count(Alert.id).desc())
|
||||
.limit(6)
|
||||
)
|
||||
recent_alerts_result = await db.execute(
|
||||
select(Alert)
|
||||
.order_by(Alert.created_at.desc(), Alert.id.desc())
|
||||
.limit(6)
|
||||
)
|
||||
|
||||
total_incidents_result = await db.execute(select(func.count(BGPIncident.id)))
|
||||
active_incidents_result = await db.execute(
|
||||
select(func.count(BGPIncident.id)).where(BGPIncident.status == "active")
|
||||
)
|
||||
bgp_severity_result = await db.execute(
|
||||
select(BGPIncident.severity, func.count(BGPIncident.id))
|
||||
.where(BGPIncident.status == "active")
|
||||
.group_by(BGPIncident.severity)
|
||||
)
|
||||
bgp_region_counter: Counter[str] = Counter()
|
||||
recent_incidents_result = await db.execute(
|
||||
select(BGPIncident)
|
||||
.order_by(BGPIncident.created_at.desc(), BGPIncident.id.desc())
|
||||
.limit(5)
|
||||
)
|
||||
|
||||
total_anomalies_result = await db.execute(select(func.count(BGPAnomaly.id)))
|
||||
active_anomalies_result = await db.execute(
|
||||
select(func.count(BGPAnomaly.id)).where(BGPAnomaly.status == "active")
|
||||
)
|
||||
anomaly_type_result = await db.execute(
|
||||
select(BGPAnomaly.anomaly_type, func.count(BGPAnomaly.id))
|
||||
.where(BGPAnomaly.status == "active")
|
||||
.group_by(BGPAnomaly.anomaly_type)
|
||||
.order_by(func.count(BGPAnomaly.id).desc())
|
||||
.limit(6)
|
||||
)
|
||||
|
||||
recent_incidents = recent_incidents_result.scalars().all()
|
||||
for incident in recent_incidents:
|
||||
for region in incident.affected_regions or []:
|
||||
if not isinstance(region, dict):
|
||||
continue
|
||||
label = ", ".join(part for part in [region.get("city"), region.get("country")] if part) or "未知区域"
|
||||
bgp_region_counter[label] += 1
|
||||
|
||||
latest_bgp_brief = get_latest_bgp_brief_record()
|
||||
active_alert_severities = [
|
||||
(item[0].value if isinstance(item[0], AlertSeverity) else str(item[0]), item[1])
|
||||
for item in alert_severity_result.fetchall()
|
||||
if item[0]
|
||||
]
|
||||
active_bgp_severities = [
|
||||
(str(item[0]), item[1])
|
||||
for item in bgp_severity_result.fetchall()
|
||||
if item[0]
|
||||
]
|
||||
active_anomaly_types = [(str(item[0]), item[1]) for item in anomaly_type_result.fetchall() if item[0]]
|
||||
active_alert_sources = [
|
||||
(str(item[0] or "未命名数据源"), item[1])
|
||||
for item in alert_source_result.fetchall()
|
||||
]
|
||||
|
||||
facts = [
|
||||
(
|
||||
f"系统告警侧:总告警 {total_alerts_result.scalar() or 0} 条,active {active_alerts_result.scalar() or 0} 条;"
|
||||
f"活跃告警严重度分布为 {_format_pairs(active_alert_severities)}。"
|
||||
),
|
||||
(
|
||||
f"BGP态势侧:累计 incidents {total_incidents_result.scalar() or 0} 条,active incidents {active_incidents_result.scalar() or 0} 条;"
|
||||
f"活跃 incidents 严重度分布为 {_format_pairs(active_bgp_severities)}。"
|
||||
),
|
||||
(
|
||||
f"BGP异常侧:累计 anomalies {total_anomalies_result.scalar() or 0} 条,active anomalies {active_anomalies_result.scalar() or 0} 条;"
|
||||
f"活跃 anomaly 类型分布为 {_format_pairs(active_anomaly_types)}。"
|
||||
),
|
||||
]
|
||||
|
||||
if active_alert_sources:
|
||||
facts.append(f"当前系统告警主要集中在:{_format_pairs(active_alert_sources)}。")
|
||||
if bgp_region_counter:
|
||||
facts.append(f"BGP近期高风险区域线索:{_format_pairs(bgp_region_counter.most_common(5))}。")
|
||||
|
||||
recent_alerts = recent_alerts_result.scalars().all()
|
||||
if recent_alerts:
|
||||
facts.append(
|
||||
"最近系统告警摘录:"
|
||||
+ ";".join(
|
||||
[
|
||||
f"{alert.datasource_name or '未命名数据源'} / {alert.severity.value if alert.severity else '-'} / {alert.status.value if alert.status else '-'} / {alert.message or '-'}"
|
||||
for alert in recent_alerts
|
||||
]
|
||||
)
|
||||
)
|
||||
|
||||
if recent_incidents:
|
||||
facts.append(
|
||||
"最近BGP事件摘录:"
|
||||
+ ";".join(
|
||||
[
|
||||
f"{incident.incident_type} / {incident.severity} / {incident.status} / {incident.summary}"
|
||||
for incident in recent_incidents
|
||||
]
|
||||
)
|
||||
)
|
||||
|
||||
if latest_bgp_brief:
|
||||
facts.append(
|
||||
f"最近一份 BGP AI 简报生成于 {latest_bgp_brief.generated_at},模型 {latest_bgp_brief.model},可作为当前态势的补充说明。"
|
||||
)
|
||||
|
||||
context = {
|
||||
"source": "situational-alerts",
|
||||
"active_system_alerts": active_alerts_result.scalar() or 0,
|
||||
"active_system_alert_severities": dict(active_alert_severities),
|
||||
"top_system_alert_sources": dict(active_alert_sources),
|
||||
"active_bgp_incidents": active_incidents_result.scalar() or 0,
|
||||
"active_bgp_incident_severities": dict(active_bgp_severities),
|
||||
"active_bgp_anomalies": active_anomalies_result.scalar() or 0,
|
||||
"active_bgp_anomaly_types": dict(active_anomaly_types),
|
||||
"bgp_hot_regions": dict(bgp_region_counter.most_common(5)),
|
||||
"latest_bgp_brief_id": latest_bgp_brief.id if latest_bgp_brief else None,
|
||||
"latest_bgp_brief_generated_at": latest_bgp_brief.generated_at if latest_bgp_brief else None,
|
||||
}
|
||||
|
||||
request = SituationalAnalysisRequest(
|
||||
title="态势告警 AI 简报",
|
||||
objective="综合系统告警、BGP incidents、BGP anomalies 与近期 BGP AI 简报,生成一份面向值班人员的态势告警简报,指出当前最需要关注的风险域、跨模块联动迹象和优先动作。",
|
||||
observations=facts,
|
||||
constraints=[
|
||||
"明确区分事实、推断与建议。",
|
||||
"优先指出仍在 active 状态的系统告警与 BGP 风险是否存在联动。",
|
||||
"不要把单一数据源的局部异常夸大成全局态势。",
|
||||
"如果证据不足,请明确写出仍缺哪些模块或区域信息。",
|
||||
],
|
||||
context=context,
|
||||
)
|
||||
return request, facts, context
|
||||
466
backend/app/services/tv_streams.py
Normal file
466
backend/app/services/tv_streams.py
Normal file
@@ -0,0 +1,466 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.models.system_setting import SystemSetting
|
||||
|
||||
DEFAULT_TV_SOURCE_ID = "cgtn-en"
|
||||
TV_SETTINGS_CATEGORY = "tv"
|
||||
TV_LIVE_SOURCE_COLLECTOR = "news_live_streams"
|
||||
TV_LIVE_SOURCE_DATA_TYPE = "news_live_stream"
|
||||
|
||||
DEFAULT_TV_SETTINGS = {
|
||||
"default_source_id": DEFAULT_TV_SOURCE_ID,
|
||||
"auto_fallback": True,
|
||||
"sources": [
|
||||
{
|
||||
"id": "cctv4",
|
||||
"name": "CCTV-4 中文国际",
|
||||
"provider": "CCTV",
|
||||
"region": "China",
|
||||
"language": "zh-CN",
|
||||
"source_type": "hls",
|
||||
"embed_url": "https://tv.cctv.com/live/cctv4/",
|
||||
"stream_url": "https://ldocctvwbcdtxy.liveplay.myqcloud.com/ldocctvwbcd/cdrmldcctv4_1_td.m3u8",
|
||||
"homepage_url": "https://tv.cctv.com/live/cctv4/",
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": True,
|
||||
"sort_order": 10,
|
||||
"collector_source": None,
|
||||
"notes": "默认兜底新闻直播源。优先尝试 CCTV-4 官方 HLS 播放流,若直播放失败则回退到央视官网直播页。",
|
||||
},
|
||||
{
|
||||
"id": "reuters-tv",
|
||||
"name": "Reuters TV",
|
||||
"provider": "Reuters",
|
||||
"region": "Global",
|
||||
"language": "en",
|
||||
"source_type": "hls",
|
||||
"embed_url": "",
|
||||
"stream_url": "https://reuters-reutersnow-1-eu.rakuten.wurl.tv/playlist.m3u8",
|
||||
"homepage_url": "https://www.reuters.com/video/live/",
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": False,
|
||||
"sort_order": 20,
|
||||
"collector_source": None,
|
||||
"notes": "参考 worldmonitor 的默认新闻频道清单,优先作为全球英文新闻直播放源。",
|
||||
},
|
||||
{
|
||||
"id": "cgtn-en",
|
||||
"name": "CGTN English",
|
||||
"provider": "CGTN",
|
||||
"region": "Global",
|
||||
"language": "en",
|
||||
"source_type": "youtube",
|
||||
"embed_url": "https://www.youtube.com/watch?v=BOy2xDU1LC8",
|
||||
"stream_url": "https://news.cgtn.com/resource/live/english/cgtn-news.m3u8",
|
||||
"youtube_video_id": "BOy2xDU1LC8",
|
||||
"homepage_url": "https://news.cgtn.com/",
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": False,
|
||||
"sort_order": 30,
|
||||
"collector_source": None,
|
||||
"notes": "优先使用官方 YouTube 直播源,保留 HLS 直播放流作为候选信息。",
|
||||
},
|
||||
{
|
||||
"id": "cgtn-es",
|
||||
"name": "CGTN Espanol",
|
||||
"provider": "CGTN",
|
||||
"region": "Latin America",
|
||||
"language": "es",
|
||||
"source_type": "hls",
|
||||
"embed_url": "",
|
||||
"stream_url": "https://news.cgtn.com/resource/live/espanol/cgtn-e.m3u8",
|
||||
"homepage_url": "https://news.cgtn.com/",
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": False,
|
||||
"sort_order": 40,
|
||||
"collector_source": None,
|
||||
"notes": "西语国际新闻频道,覆盖拉美方向态势。",
|
||||
},
|
||||
{
|
||||
"id": "dw-espanol",
|
||||
"name": "DW Espanol",
|
||||
"provider": "Deutsche Welle",
|
||||
"region": "Europe",
|
||||
"language": "es",
|
||||
"source_type": "hls",
|
||||
"embed_url": "",
|
||||
"stream_url": "https://dwamdstream104.akamaized.net/hls/live/2015530/dwstream104/stream04/streamPlaylist.m3u8",
|
||||
"homepage_url": "https://www.dw.com/es/",
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": False,
|
||||
"sort_order": 50,
|
||||
"collector_source": None,
|
||||
"notes": "来自 worldmonitor 可选频道清单的直播放源。",
|
||||
},
|
||||
{
|
||||
"id": "dw-arabic",
|
||||
"name": "DW Arabic",
|
||||
"provider": "Deutsche Welle",
|
||||
"region": "Middle East",
|
||||
"language": "ar",
|
||||
"source_type": "hls",
|
||||
"embed_url": "",
|
||||
"stream_url": "https://dwamdstream103.akamaized.net/hls/live/2015526/dwstream103/index.m3u8",
|
||||
"homepage_url": "https://www.dw.com/ar/",
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": False,
|
||||
"sort_order": 60,
|
||||
"collector_source": None,
|
||||
"notes": "阿拉伯语新闻流,适合作为中东方向新闻补充源。",
|
||||
},
|
||||
{
|
||||
"id": "aljazeera-mubasher",
|
||||
"name": "Al Jazeera Mubasher",
|
||||
"provider": "Al Jazeera",
|
||||
"region": "Middle East",
|
||||
"language": "ar",
|
||||
"source_type": "hls",
|
||||
"embed_url": "",
|
||||
"stream_url": "https://live-hls-web-ajm.getaj.net/AJM/index.m3u8",
|
||||
"homepage_url": "https://www.aljazeera.net/live",
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": False,
|
||||
"sort_order": 70,
|
||||
"collector_source": None,
|
||||
"notes": "中东实时新闻流,来自 worldmonitor HLS 频道目录。",
|
||||
},
|
||||
{
|
||||
"id": "arirang-news",
|
||||
"name": "Arirang News",
|
||||
"provider": "Arirang",
|
||||
"region": "Korea",
|
||||
"language": "en",
|
||||
"source_type": "hls",
|
||||
"embed_url": "",
|
||||
"stream_url": "https://amdlive-ch01-ctnd-com.akamaized.net/arirang_1ch/smil:arirang_1ch.smil/playlist.m3u8",
|
||||
"homepage_url": "https://www.arirang.com/",
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": False,
|
||||
"sort_order": 80,
|
||||
"collector_source": None,
|
||||
"notes": "东北亚英语新闻源,适合补充韩半岛与东亚视角。",
|
||||
},
|
||||
{
|
||||
"id": "abp-news",
|
||||
"name": "ABP News",
|
||||
"provider": "ABP",
|
||||
"region": "India",
|
||||
"language": "hi",
|
||||
"source_type": "hls",
|
||||
"embed_url": "",
|
||||
"stream_url": "https://abplivetv.pc.cdn.bitgravity.com/httppush/abp_livetv/abp_abpnews/master.m3u8",
|
||||
"homepage_url": "https://news.abplive.com/live-tv",
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": False,
|
||||
"sort_order": 90,
|
||||
"collector_source": None,
|
||||
"notes": "印度新闻直播放源,补充南亚区域视角。",
|
||||
},
|
||||
{
|
||||
"id": "sabc-news",
|
||||
"name": "SABC News",
|
||||
"provider": "SABC",
|
||||
"region": "Africa",
|
||||
"language": "en",
|
||||
"source_type": "hls",
|
||||
"embed_url": "",
|
||||
"stream_url": "https://sabconetanw.cdn.mangomolo.com/news/smil:news.stream.smil/playlist.m3u8",
|
||||
"homepage_url": "https://www.sabcnews.com/sabcnews/",
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": False,
|
||||
"sort_order": 100,
|
||||
"collector_source": None,
|
||||
"notes": "非洲英语新闻源,补充非洲区域新闻覆盖。",
|
||||
},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def _clean_text(value: Any) -> str:
|
||||
if value is None:
|
||||
return ""
|
||||
return str(value).strip()
|
||||
|
||||
|
||||
def _clean_url(value: Any) -> str:
|
||||
text = _clean_text(value)
|
||||
if not text:
|
||||
return ""
|
||||
|
||||
parsed = urlparse(text)
|
||||
if parsed.scheme and parsed.scheme not in {"http", "https"}:
|
||||
return ""
|
||||
if parsed.scheme and not parsed.netloc:
|
||||
return ""
|
||||
return text
|
||||
|
||||
|
||||
def _clean_bool(value: Any, *, default: bool) -> bool:
|
||||
if isinstance(value, bool):
|
||||
return value
|
||||
if value in (None, ""):
|
||||
return default
|
||||
if isinstance(value, str):
|
||||
lowered = value.strip().lower()
|
||||
if lowered in {"1", "true", "yes", "on"}:
|
||||
return True
|
||||
if lowered in {"0", "false", "no", "off"}:
|
||||
return False
|
||||
return bool(value)
|
||||
|
||||
|
||||
def _clean_int(value: Any, *, default: int) -> int:
|
||||
try:
|
||||
return int(value)
|
||||
except (TypeError, ValueError):
|
||||
return default
|
||||
|
||||
|
||||
def normalize_tv_source(source: dict[str, Any] | None, *, index: int = 0) -> dict[str, Any]:
|
||||
payload = dict(source or {})
|
||||
source_id = _clean_text(payload.get("id")) or f"tv-source-{index + 1}"
|
||||
source_type = _clean_text(payload.get("source_type")).lower()
|
||||
youtube_video_id = _clean_text(payload.get("youtube_video_id"))
|
||||
youtube_channel = _clean_text(payload.get("youtube_channel"))
|
||||
if source_type not in {"iframe", "hls", "video", "external", "youtube"}:
|
||||
if youtube_video_id or youtube_channel:
|
||||
source_type = "youtube"
|
||||
else:
|
||||
source_type = "iframe" if _clean_text(payload.get("embed_url")) else "external"
|
||||
|
||||
if source_type == "youtube" and not youtube_video_id and not youtube_channel:
|
||||
source_type = "iframe" if _clean_text(payload.get("embed_url")) else "external"
|
||||
|
||||
return {
|
||||
"id": source_id,
|
||||
"name": _clean_text(payload.get("name")) or f"新闻直播源 {index + 1}",
|
||||
"provider": _clean_text(payload.get("provider")) or "Unknown",
|
||||
"region": _clean_text(payload.get("region")) or "Global",
|
||||
"language": _clean_text(payload.get("language")) or "und",
|
||||
"source_type": source_type,
|
||||
"embed_url": _clean_url(payload.get("embed_url")),
|
||||
"stream_url": _clean_url(payload.get("stream_url")),
|
||||
"homepage_url": _clean_url(payload.get("homepage_url")),
|
||||
"poster_url": _clean_url(payload.get("poster_url")),
|
||||
"youtube_video_id": youtube_video_id,
|
||||
"youtube_channel": youtube_channel,
|
||||
"is_enabled": _clean_bool(payload.get("is_enabled"), default=True),
|
||||
"is_fallback": _clean_bool(payload.get("is_fallback"), default=False),
|
||||
"sort_order": _clean_int(payload.get("sort_order"), default=(index + 1) * 10),
|
||||
"collector_source": payload.get("collector_source"),
|
||||
"notes": _clean_text(payload.get("notes")),
|
||||
"updated_at": _clean_text(payload.get("updated_at")),
|
||||
}
|
||||
|
||||
|
||||
def normalize_tv_settings(payload: dict[str, Any] | None) -> dict[str, Any]:
|
||||
merged = {
|
||||
"default_source_id": DEFAULT_TV_SETTINGS["default_source_id"],
|
||||
"auto_fallback": DEFAULT_TV_SETTINGS["auto_fallback"],
|
||||
"sources": [],
|
||||
}
|
||||
|
||||
raw_sources = []
|
||||
if isinstance(payload, dict):
|
||||
merged["default_source_id"] = (
|
||||
_clean_text(payload.get("default_source_id")) or merged["default_source_id"]
|
||||
)
|
||||
merged["auto_fallback"] = _clean_bool(
|
||||
payload.get("auto_fallback"),
|
||||
default=DEFAULT_TV_SETTINGS["auto_fallback"],
|
||||
)
|
||||
if isinstance(payload.get("sources"), list):
|
||||
raw_sources = payload["sources"]
|
||||
|
||||
if not raw_sources:
|
||||
raw_sources = DEFAULT_TV_SETTINGS["sources"]
|
||||
|
||||
normalized_sources = [
|
||||
normalize_tv_source(source, index=index)
|
||||
for index, source in enumerate(raw_sources)
|
||||
]
|
||||
|
||||
if not any(source["id"] == DEFAULT_TV_SOURCE_ID for source in normalized_sources):
|
||||
normalized_sources.append(
|
||||
normalize_tv_source(DEFAULT_TV_SETTINGS["sources"][0], index=len(normalized_sources))
|
||||
)
|
||||
|
||||
default_source_exists = any(
|
||||
source["id"] == merged["default_source_id"] and source["is_enabled"]
|
||||
for source in normalized_sources
|
||||
)
|
||||
if not default_source_exists:
|
||||
fallback_source = next(
|
||||
(source for source in normalized_sources if source["is_fallback"] and source["is_enabled"]),
|
||||
None,
|
||||
)
|
||||
first_enabled_source = next(
|
||||
(source for source in normalized_sources if source["is_enabled"]),
|
||||
None,
|
||||
)
|
||||
merged["default_source_id"] = (
|
||||
fallback_source["id"]
|
||||
if fallback_source
|
||||
else first_enabled_source["id"]
|
||||
if first_enabled_source
|
||||
else DEFAULT_TV_SOURCE_ID
|
||||
)
|
||||
|
||||
merged["sources"] = sorted(
|
||||
normalized_sources,
|
||||
key=lambda item: (item["sort_order"], item["name"], item["id"]),
|
||||
)
|
||||
return merged
|
||||
|
||||
|
||||
async def get_tv_settings_payload(db: AsyncSession) -> dict[str, Any]:
|
||||
result = await db.execute(
|
||||
select(SystemSetting).where(SystemSetting.category == TV_SETTINGS_CATEGORY)
|
||||
)
|
||||
record = result.scalar_one_or_none()
|
||||
payload = record.payload if record else None
|
||||
return normalize_tv_settings(payload)
|
||||
|
||||
|
||||
def _build_collected_tv_source(record: CollectedData, index: int) -> dict[str, Any]:
|
||||
metadata = dict(record.extra_data or {})
|
||||
return normalize_tv_source(
|
||||
{
|
||||
"id": metadata.get("id") or record.source_id or record.entity_key,
|
||||
"name": record.name or record.title or metadata.get("name") or f"采集直播源 {index + 1}",
|
||||
"provider": metadata.get("provider") or metadata.get("publisher") or "Collector",
|
||||
"region": metadata.get("region") or metadata.get("country") or "Global",
|
||||
"language": metadata.get("language") or "und",
|
||||
"source_type": metadata.get("source_type") or "iframe",
|
||||
"embed_url": metadata.get("embed_url") or metadata.get("url") or "",
|
||||
"stream_url": metadata.get("stream_url") or "",
|
||||
"homepage_url": metadata.get("homepage_url") or metadata.get("source_url") or "",
|
||||
"poster_url": metadata.get("poster_url") or "",
|
||||
"youtube_video_id": metadata.get("youtube_video_id") or metadata.get("video_id") or "",
|
||||
"youtube_channel": metadata.get("youtube_channel") or metadata.get("channel_handle") or "",
|
||||
"is_enabled": metadata.get("is_enabled", True),
|
||||
"is_fallback": False,
|
||||
"sort_order": metadata.get("sort_order", 200 + index),
|
||||
"collector_source": record.source,
|
||||
"notes": record.description or metadata.get("notes") or "",
|
||||
"updated_at": to_iso8601_utc(record.updated_at or record.reference_date or datetime.now(UTC)),
|
||||
},
|
||||
index=index,
|
||||
)
|
||||
|
||||
|
||||
async def get_collected_tv_sources(db: AsyncSession) -> list[dict[str, Any]]:
|
||||
result = await db.execute(
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source == TV_LIVE_SOURCE_COLLECTOR)
|
||||
.where(CollectedData.data_type == TV_LIVE_SOURCE_DATA_TYPE)
|
||||
.where(CollectedData.is_current.is_(True))
|
||||
.where(CollectedData.is_valid == 1)
|
||||
.order_by(CollectedData.reference_date.desc().nullslast(), CollectedData.id.desc())
|
||||
)
|
||||
rows = result.scalars().all()
|
||||
return [_build_collected_tv_source(record, index) for index, record in enumerate(rows)]
|
||||
|
||||
|
||||
def build_public_tv_payload(
|
||||
settings_payload: dict[str, Any],
|
||||
collected_sources: list[dict[str, Any]],
|
||||
) -> dict[str, Any]:
|
||||
configured_sources = [
|
||||
source for source in settings_payload["sources"] if source["is_enabled"]
|
||||
]
|
||||
|
||||
merged_by_id = {source["id"]: source for source in configured_sources}
|
||||
for source in collected_sources:
|
||||
if source["id"] in merged_by_id or not source["is_enabled"]:
|
||||
continue
|
||||
merged_by_id[source["id"]] = source
|
||||
|
||||
available_sources = sorted(
|
||||
merged_by_id.values(),
|
||||
key=lambda item: (item["sort_order"], item["name"], item["id"]),
|
||||
)
|
||||
|
||||
default_source = next(
|
||||
(
|
||||
source
|
||||
for source in available_sources
|
||||
if source["id"] == settings_payload["default_source_id"]
|
||||
),
|
||||
None,
|
||||
)
|
||||
fallback_source = next(
|
||||
(source for source in available_sources if source["is_fallback"]),
|
||||
None,
|
||||
)
|
||||
|
||||
resolved_source = default_source or fallback_source or (available_sources[0] if available_sources else None)
|
||||
latest_updated_at = max(
|
||||
(source.get("updated_at") or "" for source in available_sources),
|
||||
default="",
|
||||
)
|
||||
|
||||
return {
|
||||
"default_source_id": settings_payload["default_source_id"],
|
||||
"auto_fallback": settings_payload["auto_fallback"],
|
||||
"selected_source": resolved_source,
|
||||
"fallback_source": fallback_source,
|
||||
"sources": available_sources,
|
||||
"source_count": len(available_sources),
|
||||
"latest_updated_at": latest_updated_at or to_iso8601_utc(datetime.now(UTC)),
|
||||
"generated_at": to_iso8601_utc(datetime.now(UTC)),
|
||||
}
|
||||
|
||||
|
||||
async def get_public_tv_payload(db: AsyncSession) -> dict[str, Any]:
|
||||
settings_payload = await get_tv_settings_payload(db)
|
||||
collected_sources = await get_collected_tv_sources(db)
|
||||
return build_public_tv_payload(settings_payload, collected_sources)
|
||||
|
||||
|
||||
def _extract_allowed_tv_hosts(sources: list[dict[str, Any]]) -> set[str]:
|
||||
hosts: set[str] = set()
|
||||
for source in sources:
|
||||
for field in ("stream_url", "embed_url", "homepage_url", "youtube_channel"):
|
||||
value = _clean_url(source.get(field))
|
||||
if not value:
|
||||
continue
|
||||
parsed = urlparse(value)
|
||||
if parsed.hostname:
|
||||
hosts.add(parsed.hostname.lower())
|
||||
return hosts
|
||||
|
||||
|
||||
def is_allowed_tv_proxy_url(url: str, sources: list[dict[str, Any]]) -> bool:
|
||||
cleaned = _clean_url(url)
|
||||
if not cleaned:
|
||||
return False
|
||||
|
||||
parsed = urlparse(cleaned)
|
||||
hostname = (parsed.hostname or "").lower()
|
||||
if not hostname:
|
||||
return False
|
||||
|
||||
allowed_hosts = _extract_allowed_tv_hosts(sources)
|
||||
if hostname in allowed_hosts:
|
||||
return True
|
||||
return any(hostname.endswith(f".{allowed_host}") for allowed_host in allowed_hosts)
|
||||
@@ -10,7 +10,12 @@ from app.core.config import settings
|
||||
from app.core.security import create_access_token
|
||||
from app.db.session import get_db
|
||||
from app.models.user import User
|
||||
from app.schemas.ai import AIProviderStatusResponse, SituationalAnalysisResponse
|
||||
from app.schemas.ai import (
|
||||
AIProviderStatusResponse,
|
||||
PlaygroundSessionResponse,
|
||||
PlaygroundSessionState,
|
||||
SituationalAnalysisResponse,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
@@ -90,6 +95,15 @@ async def test_alerts_without_auth():
|
||||
assert response.status_code == 401
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_datasource_task_status_without_auth():
|
||||
"""Test datasource task-status endpoint requires authentication"""
|
||||
transport = ASGITransport(app=app)
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/api/v1/datasources/1/task-status")
|
||||
assert response.status_code == 401
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_alerts_endpoint_with_auth(auth_headers):
|
||||
"""Test alerts endpoint with authentication"""
|
||||
@@ -165,7 +179,8 @@ async def test_ai_provider_status_with_auth(auth_headers):
|
||||
class _FakeAIProviderClient:
|
||||
async def get_status(self, request_id=None):
|
||||
return AIProviderStatusResponse(
|
||||
provider="openai_compatible",
|
||||
provider="minimax",
|
||||
api="anthropic-messages",
|
||||
enabled=True,
|
||||
configured=True,
|
||||
model="test-model",
|
||||
@@ -193,6 +208,7 @@ async def test_ai_provider_status_with_auth(auth_headers):
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert "provider" in data
|
||||
assert "api" in data
|
||||
assert "configured" in data
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
@@ -207,6 +223,9 @@ async def test_ai_situational_analysis_returns_503_when_disabled(auth_headers):
|
||||
provider="openai_compatible",
|
||||
model="test-model",
|
||||
content="1) 态势摘要: 测试返回",
|
||||
content_blocks=[],
|
||||
text_blocks=["1) 态势摘要: 测试返回"],
|
||||
thinking_blocks=[],
|
||||
raw_response={"id": "mock-response"},
|
||||
)
|
||||
|
||||
@@ -241,5 +260,297 @@ async def test_ai_situational_analysis_returns_503_when_disabled(auth_headers):
|
||||
data = response.json()
|
||||
assert data["provider"] == "openai_compatible"
|
||||
assert data["content"]
|
||||
assert "content_blocks" in data
|
||||
assert "text_blocks" in data
|
||||
assert "thinking_blocks" in data
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_playground_session_with_auth(auth_headers):
|
||||
"""Test playground session restore endpoint."""
|
||||
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="testuser",
|
||||
email="test@example.com",
|
||||
password_hash="hashed",
|
||||
role="admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
async def override_get_db():
|
||||
yield AsyncMock()
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
get_db: override_get_db,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch(
|
||||
"app.api.v1.ai.get_playground_session",
|
||||
new=AsyncMock(
|
||||
return_value=PlaygroundSessionResponse(
|
||||
id="1",
|
||||
session_key="default",
|
||||
title="Playground 会话",
|
||||
state=PlaygroundSessionState(
|
||||
messages=[{"id": "msg-1", "role": "user", "content": "hello"}],
|
||||
title="测试标题",
|
||||
objective="测试目标",
|
||||
),
|
||||
created_at="2026-04-10T00:00:00+00:00",
|
||||
updated_at="2026-04-10T00:00:00+00:00",
|
||||
)
|
||||
),
|
||||
):
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/api/v1/ai/playground/session", headers=auth_headers)
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["session_key"] == "default"
|
||||
assert data["state"]["messages"][0]["content"] == "hello"
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_save_playground_session_with_auth(auth_headers):
|
||||
"""Test playground session save endpoint."""
|
||||
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="testuser",
|
||||
email="test@example.com",
|
||||
password_hash="hashed",
|
||||
role="admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
async def override_get_db():
|
||||
yield AsyncMock()
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
get_db: override_get_db,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch(
|
||||
"app.api.v1.ai.upsert_playground_session",
|
||||
new=AsyncMock(
|
||||
return_value=PlaygroundSessionResponse(
|
||||
id="1",
|
||||
session_key="default",
|
||||
title="测试标题",
|
||||
state=PlaygroundSessionState(
|
||||
messages=[{"id": "msg-1", "role": "user", "content": "hello"}],
|
||||
title="测试标题",
|
||||
objective="测试目标",
|
||||
),
|
||||
created_at="2026-04-10T00:00:00+00:00",
|
||||
updated_at="2026-04-10T00:00:00+00:00",
|
||||
)
|
||||
),
|
||||
):
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.put(
|
||||
"/api/v1/ai/playground/session",
|
||||
headers=auth_headers,
|
||||
json={
|
||||
"session_key": "default",
|
||||
"title": "测试标题",
|
||||
"state": {
|
||||
"messages": [{"id": "msg-1", "role": "user", "content": "hello"}],
|
||||
"selectedPresetKey": "bgp-brief",
|
||||
"title": "测试标题",
|
||||
"objective": "测试目标",
|
||||
"constraints": "",
|
||||
"inputValue": "",
|
||||
"analysis": None,
|
||||
"latestAnalysisMessageId": None,
|
||||
"analysisMeta": {},
|
||||
"helpExpanded": True,
|
||||
},
|
||||
},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["title"] == "测试标题"
|
||||
assert data["state"]["objective"] == "测试目标"
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ai_bgp_brief_endpoint_persists_fact_snapshot(auth_headers):
|
||||
class _FakeAIProviderClient:
|
||||
async def analyze(self, _payload, request_id=None):
|
||||
return SituationalAnalysisResponse(
|
||||
provider="minimax",
|
||||
model="MiniMax-M2.5",
|
||||
content="# BGP AI 简报\n\n事实摘要:测试",
|
||||
content_blocks=[],
|
||||
text_blocks=["# BGP AI 简报\n\n事实摘要:测试"],
|
||||
thinking_blocks=[],
|
||||
raw_response={"id": "mock-bgp-brief"},
|
||||
)
|
||||
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="testuser",
|
||||
email="test@example.com",
|
||||
password_hash="hashed",
|
||||
role="admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
async def override_get_db():
|
||||
yield AsyncMock()
|
||||
|
||||
async def _fake_build_bgp_brief_request(_db, **_kwargs):
|
||||
request_payload = __import__("app.schemas.ai", fromlist=["SituationalAnalysisRequest"]).SituationalAnalysisRequest(
|
||||
title="BGP 态势 AI 简报",
|
||||
objective="生成值班简报",
|
||||
observations=["事实A", "事实B"],
|
||||
constraints=["不要编造"],
|
||||
context={"incident_total": 2, "active_collectors": 3},
|
||||
)
|
||||
return request_payload, ["事实A", "事实B"], {"incident_total": 2, "active_collectors": 3}
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
__import__("app.services.ai_client", fromlist=["get_ai_provider_client"]).get_ai_provider_client: lambda: _FakeAIProviderClient(),
|
||||
get_db: override_get_db,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch("app.api.v1.ai.build_bgp_brief_request", side_effect=_fake_build_bgp_brief_request):
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.post("/api/v1/ai/bgp/brief", headers=auth_headers, json={})
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["facts"] == ["事实A", "事实B"]
|
||||
assert data["context"]["incident_total"] == 2
|
||||
assert data["content_markdown"]
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ai_alert_brief_endpoint_with_auth(auth_headers):
|
||||
class _FakeAIProviderClient:
|
||||
async def analyze(self, _payload, request_id=None):
|
||||
return SituationalAnalysisResponse(
|
||||
provider="minimax",
|
||||
model="MiniMax-M2.7",
|
||||
content="事实摘要:告警测试。风险研判:告警测试。建议动作:告警测试。",
|
||||
content_blocks=[],
|
||||
text_blocks=["事实摘要:告警测试。风险研判:告警测试。建议动作:告警测试。"],
|
||||
thinking_blocks=[],
|
||||
raw_response={"id": "mock-alert-brief"},
|
||||
)
|
||||
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="testuser",
|
||||
email="test@example.com",
|
||||
password_hash="hashed",
|
||||
role="admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
async def override_get_db():
|
||||
yield AsyncMock()
|
||||
|
||||
async def _fake_build_alert_brief_request(_db, **_kwargs):
|
||||
request_payload = __import__("app.schemas.ai", fromlist=["SituationalAnalysisRequest"]).SituationalAnalysisRequest(
|
||||
title="告警态势 AI 简报",
|
||||
objective="输出告警简报",
|
||||
observations=["告警事实A", "告警事实B"],
|
||||
constraints=["不要编造"],
|
||||
context={"active_alerts": 3, "top_datasources": {"bgp": 2}},
|
||||
)
|
||||
return request_payload, ["告警事实A", "告警事实B"], {"active_alerts": 3, "top_datasources": {"bgp": 2}}
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
__import__("app.services.ai_client", fromlist=["get_ai_provider_client"]).get_ai_provider_client: lambda: _FakeAIProviderClient(),
|
||||
get_db: override_get_db,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch("app.api.v1.ai.build_alert_brief_request", side_effect=_fake_build_alert_brief_request):
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.post("/api/v1/ai/alerts/brief", headers=auth_headers, json={})
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["title"] == "告警态势 AI 简报"
|
||||
assert data["facts"] == ["告警事实A", "告警事实B"]
|
||||
assert data["context"]["active_alerts"] == 3
|
||||
assert data["content"]
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ai_situational_alert_brief_endpoint_with_auth(auth_headers):
|
||||
class _FakeAIProviderClient:
|
||||
async def analyze(self, _payload, request_id=None):
|
||||
return SituationalAnalysisResponse(
|
||||
provider="minimax",
|
||||
model="MiniMax-M2.7",
|
||||
content="事实摘要:态势测试。风险研判:态势测试。建议动作:态势测试。",
|
||||
content_blocks=[],
|
||||
text_blocks=["事实摘要:态势测试。风险研判:态势测试。建议动作:态势测试。"],
|
||||
thinking_blocks=[],
|
||||
raw_response={"id": "mock-situational-brief"},
|
||||
)
|
||||
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="testuser",
|
||||
email="test@example.com",
|
||||
password_hash="hashed",
|
||||
role="admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
async def override_get_db():
|
||||
yield AsyncMock()
|
||||
|
||||
async def _fake_build_situational_alert_brief_request(_db):
|
||||
request_payload = __import__("app.schemas.ai", fromlist=["SituationalAnalysisRequest"]).SituationalAnalysisRequest(
|
||||
title="态势告警 AI 简报",
|
||||
objective="输出态势告警简报",
|
||||
observations=["态势事实A", "态势事实B"],
|
||||
constraints=["不要编造"],
|
||||
context={"active_system_alerts": 2, "active_bgp_incidents": 1},
|
||||
)
|
||||
return request_payload, ["态势事实A", "态势事实B"], {"active_system_alerts": 2, "active_bgp_incidents": 1}
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
__import__("app.services.ai_client", fromlist=["get_ai_provider_client"]).get_ai_provider_client: lambda: _FakeAIProviderClient(),
|
||||
get_db: override_get_db,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch("app.api.v1.ai.build_situational_alert_brief_request", side_effect=_fake_build_situational_alert_brief_request):
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.post("/api/v1/ai/situational-alerts/brief", headers=auth_headers, json={})
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["title"] == "态势告警 AI 简报"
|
||||
assert data["facts"] == ["态势事实A", "态势事实B"]
|
||||
assert data["context"]["active_system_alerts"] == 2
|
||||
assert data["content"]
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
@@ -5,6 +5,8 @@ services:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: aiprovider/Dockerfile
|
||||
env_file:
|
||||
- ./aiprovider/.env
|
||||
container_name: planet_aiprovider
|
||||
ports:
|
||||
- "8010:8010"
|
||||
|
||||
@@ -5,8 +5,684 @@ All notable changes to `planet` are documented here.
|
||||
This project follows the repository versioning rule:
|
||||
|
||||
- `feature` -> `+0.1.0`
|
||||
- `improvement` -> `+0.0.1`(bugfix + 小功能混合)
|
||||
- `bugfix` -> `+0.0.1`
|
||||
|
||||
## [0.31.0] — 2026-04-21
|
||||
|
||||
## [0.31.2] — 2026-04-21
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 巡航模式重构为“通用巡航队列 + 通用连线动画 + BGP 业务适配”三层结构,后续扩到海缆、卫星或新闻巡航时不必再复制一套 `main.js` 状态机
|
||||
- 修复巡航重构后的交互回归:空白点击重新稳定切到下一项,连线按“起点 → 引导线 → 终点”顺序入场
|
||||
|
||||
### 🔧 Improvements
|
||||
- 新增 [cruise-sequencer.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/cruise-sequencer.js) 统一管理队列推进、停留时长、打断与恢复
|
||||
- 新增 [callout-connector.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/callout-connector.js) 统一管理 SVG 连线、折线路径与描边动画
|
||||
- 新增 [bgp-cruise-adapter.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp-cruise-adapter.js) 收口 BGP 巡航目标排序、卡片落点、轮询去重与连线适配
|
||||
- 更新 [earth-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/earth-frontend-context.md) 说明新的巡航分层与复用边界
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复巡航模式下点击空白处无法稳定跳转到下一项、切回旋转再切回巡航后直接卡住的问题
|
||||
- 修复巡航连线被实时重定位覆盖导致“直接出现”而非绘制动画的问题
|
||||
- 修复连线动画节点入场节奏不对的问题,改为先出现起点,再绘制连线,最后出现终点
|
||||
|
||||
---
|
||||
|
||||
## [0.31.1] — 2026-04-21
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 图层开关状态统一成可复用的 `active / loading` 状态机,首次启用地形和卫星时不再像按钮失效
|
||||
- 文档目录重构为 `docs/technical`、`docs/plans`、`docs/deprecated`,并吸收 `.sisyphus/plans` 中有价值的 Earth / 卫星 / UE5 草案
|
||||
|
||||
### 🔧 Improvements
|
||||
- 新增 [layer-button-state.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/layer-button-state.js),统一按钮 tooltip、`aria-busy`、禁用态和状态文本同步
|
||||
- 地形图层支持 hover/focus 预热与空闲预热,首次点击等待前移,加载中状态持续可见
|
||||
- 卫星图层启用前会立即切换为 `loading` 中间态,请求完成后再切回正常开关表现
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复地形首次加载时通知过早消失、开关仍像关闭状态导致用户误判按钮损坏的问题
|
||||
- 修复卫星接口较慢时按钮没有任何中间态反馈的问题
|
||||
|
||||
---
|
||||
|
||||
### ✨ Features
|
||||
- Earth 新增"巡航展示"模式:自动轮播 BGP 异常事件,逐帧追踪连接线位置,支持外部交互立即中断序列(cancel notifier 模式)
|
||||
- 巡航目标事件点高亮显示:hover 外观 + 锁定脉冲动画,并与点击行为统一展示周边受影响卫星与海缆
|
||||
- BGP 事件图标新增填充 W 形波动符号(flap 类型),替换原有难以辨认的贝塞尔细线
|
||||
- 巡航/点击激活时其余卫星自动降饱和度 + 增加透明度以突出焦点;海缆未受影响时同步变暗
|
||||
|
||||
### 🔧 Improvements
|
||||
- 修复巡航轮播期间 BGP 事件 polling 刷新导致标记闪烁消失的问题(clearBGPData 延迟到请求完成后执行)
|
||||
- 点击与巡航锁定颜色统一为 hover 色(0.92, 0.98, 1.0 全透明),移除锁定态脉冲动画
|
||||
- 巡航连接折线转折点从尖角调整为钝角(linkElbowDropPx),提升连线可读性
|
||||
|
||||
---
|
||||
|
||||
## [0.29.1] — 2026-04-20
|
||||
|
||||
## [0.30.0] — 2026-04-21
|
||||
|
||||
### ✨ Features
|
||||
- Earth 新增真实地形图层:后端代理 Terrarium DEM 瓦片(`/api/v1/visualization/terrain/terrarium/{z}/{x}/{y}.png`),前端新增 `terrain.js` 负责瓦片拉取、顶点位移与按海拔着色
|
||||
- 设置弹窗新增"地形"分组,支持通过滑块实时调整地形图层透明度
|
||||
|
||||
### 🔧 Improvements
|
||||
- 地形按钮改为异步加载,首次点击显示进度提示并在失败时自动回退
|
||||
- 启动阶段改用 `applyImmediateView` 直接应用初始视角,`showStatusMessage` / `queueStatusMessage` 区分即时与队列态状态消息,加载中不再被临时状态打断
|
||||
- 控制面板抽取 `applyTerrainUiState` / `getViewRotation` 收敛地形切换与视角旋转的重复 UI 同步逻辑
|
||||
|
||||
---
|
||||
|
||||
## [0.29.2] — 2026-04-21
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 继续收口 HUD 交互与设置面板表现,设置弹窗改成更接近从按钮展开的窗口感,同时加入系统级 admin 入口
|
||||
- 修正天球太阳方向与地球受光解耦后的日照逻辑,地表昼夜判断改为按太阳直射点经纬度落到地球贴图坐标
|
||||
|
||||
### 🔧 Improvements
|
||||
- toolbar 进一步收成更贴近 hub 的浅弓形排列,并统一成与 HUD panel 一致的液态玻璃配色与透明度
|
||||
- 设置弹窗与各 HUD panel 继续统一样式、等比缩放和头部基线,设置列表补充系统分组与 admin 跳转
|
||||
- 所有 HUD panel 增加更统一的液态玻璃高光与 hover / press 反馈
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复设置弹窗仍像旧圆角矩形、标题文案重复和从底边直直飞出的动画问题
|
||||
- 修复天球与太阳方向混用显示校准导致中国白天仍落在夜面的日照错误
|
||||
|
||||
---
|
||||
|
||||
## [0.29.1] — 2026-04-20
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 加载状态条改成单一队列式通知面板,加载阶段不再因为步骤文案变化而回缩,也不会被其他通知打断
|
||||
- 调整 brand panel 的呈现方式与昼夜/选中态可读性,让品牌区更自然、交互高亮在白天和黑夜里都更稳定
|
||||
|
||||
### 🔧 Improvements
|
||||
- 移除旧的地球加载浮层结构,统一由 HUD 状态消息承载三点脉冲加载过程
|
||||
- brand panel 改为无边框品牌层,仅保留轻微氛围光,不再因为非常规尺寸显得像第五块功能面板
|
||||
- 温和收敛地球昼夜材质与主背光强度,保留昼夜辨识度的同时提升白天地表纹理和夜面交互可见性
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复加载地球时通知条在步骤切换中反复缩短、其他状态消息抢占加载流程的问题
|
||||
- 修复海缆、登陆点和 BGP 选中高亮在黑夜中过暗、在高光中过亮导致难以辨识的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.29.0] — 2026-04-20
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 新增天球层第一版:引入真实全天星图、亮星层与太阳/月亮位置计算,地球场景首次具备可校准的天文背景
|
||||
- 地球昼夜分隔升级为更明显的日夜增强效果,夜面、晨昏带和太阳方向联动更容易直接读出来
|
||||
|
||||
### 🔧 Improvements
|
||||
- 新增 `celestial.js` 模块和 `assets/celestial/` 资源目录,统一管理星图、亮星数据以及太阳/月亮与光照同步
|
||||
- 卫星图例改为按倾角分组,严格固定为“赤道轨道 → 低倾角轨道 → 中倾角轨道 → 高倾角轨道 → 逆行轨道”顺序,并全部中文化
|
||||
- 图层面板补齐关闭按钮,拖拽脱离左列后不再被流布局 margin 影响,能够真正贴到品牌面板下沿
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复天球球壳放大后被相机 far plane 裁剪导致的外层黑环问题
|
||||
- 修复图层面板在左侧上移时始终与 brand panel 保持额外间距的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.28.2] — 2026-04-20
|
||||
|
||||
### ✨ Highlights
|
||||
- 修正媒体情报面板在 `电视直播 / 态势聚合` tab 间切换时的尺寸记忆逻辑,切回原 tab 后可恢复各自大小状态
|
||||
- 清理 `docs/` 根目录遗留的旧路径文档,只保留新的分组目录和归档目录,结束同一文档双路径并存状态
|
||||
|
||||
### 🔧 Improvements
|
||||
- `media-panel` 切换逻辑改成按 tab 分别记忆尺寸状态,避免 `A -> B -> A` 时继续共用同一套外层尺寸
|
||||
- 目录整理真正完成收尾:旧的 `docs/*.md` 平铺计划文档删除,继续以 `docs/agents / earth / backend / frontend / ops / ue5 / deprecated` 为唯一入口
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复拉伸 `media-panel` 后切换 tab 时,`news-panel` 高度回退到旧默认值的问题
|
||||
- 修复拉伸后切换 tab 导致面板视觉锚点异常的问题,切换时改为围绕当前卡片自身右下角进行尺寸恢复
|
||||
|
||||
---
|
||||
|
||||
## [0.28.1] — 2026-04-20
|
||||
|
||||
### ✨ Highlights
|
||||
- 收口 Earth 媒体情报面板命名,明确外层 `media-panel` 与内部 `tv-panel / news-panel` 的职责边界
|
||||
- 整理 `docs/` 目录分组,并将已完成或已废弃的计划文档归档到 `docs/deprecated`
|
||||
|
||||
### 🔧 Improvements
|
||||
- 底部 tab 语义统一为 `media-panel-tabs / media-panel-tab`,并将文案更新为“电视直播 / 态势聚合”
|
||||
- 补充媒体面板、聚合新闻模块的注释说明,减少 `tv-panel` 同时指代外层壳和内层直播 pane 的阅读歧义
|
||||
- 更新 README、AI Provider README 与历史文档互链,适配新的 `docs/agents / docs/earth / docs/frontend / docs/backend / docs/ops / docs/ue5` 分组结构
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复媒体情报面板标题组在头部撑出多余空白的问题,去掉 `hud-panel__title-group` 的无效弹性占位
|
||||
- 修复聚合 tab 头部仍保留冗余固定标题的问题,现在仅显示区域标签
|
||||
|
||||
---
|
||||
|
||||
## [0.28.0] — 2026-04-20
|
||||
|
||||
### ✨ Features
|
||||
- 将 Earth 的“新闻直播”和“全球态势聚合”合并为统一的“媒体情报”面板,支持底部 tab 切换与共享标题栏操作区
|
||||
- 聚合新闻不再单独占据一个 HUD 面板,而是作为媒体情报面板内的第二视图与直播协同呈现
|
||||
|
||||
说明:
|
||||
- 当前结构中,外层 HUD 壳为 `media-panel`,内部 tab 内容区分别为 `tv-panel` 和 `news-panel`
|
||||
|
||||
### 🔧 Improvements
|
||||
- TV / News 面板切换加入底边锚定的 reform 动画,并继续保留拖拽、缩放和共享 HUD 行为收口
|
||||
- 聚合新闻视图新增默认高度约束与内部滚动填充逻辑,避免初始高度过度膨胀
|
||||
- `tv.js`、`news.js` 进一步清理共享 HUDPanel 迁移后的残留逻辑,收紧 tab / resize / reform 相关局部 helper
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复媒体情报面板右下角缩放时高度异常抬升、标题栏被顶出视口的问题
|
||||
- 修复直播/聚合 tab 切换时按钮高亮、内容切换和底边基准表现不一致的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.27.7] — 2026-04-16
|
||||
|
||||
## [0.27.8] — 2026-04-20
|
||||
|
||||
### 🔧 Improvements
|
||||
- Earth HUD 共享 `HUDPanel` 默认展开/收缩逻辑继续收口,图例与图层面板统一使用同一套边缘阈值与箭头状态机
|
||||
- 保持新闻直播面板现有特例折叠行为不变,避免播放器区域被默认折叠逻辑影响
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复图例与图层面板展开/收缩箭头方向和实际动作不一致的问题
|
||||
- 修复拖动到屏幕底边附近时初始箭头、拖动中箭头和点击后动作不同步的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.27.7] — 2026-04-16
|
||||
|
||||
### 🔧 Improvements
|
||||
- 用户管理、数据源配置、电视直播源表格统一接入可折叠操作列,窄宽度下自动收起到下拉菜单,减少操作区挤压
|
||||
- 电视直播设置改为表格总览 + 弹窗编辑模式,主表内容更紧凑,适合控制台一屏浏览
|
||||
- Earth TV 面板新增失败源探测与自动回退恢复标记,便于值班时快速识别异常直播源
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复 Settings 电视直播源新增后取消编辑会残留未保存草稿的问题
|
||||
- 修复 Settings 删除直播源只改本地状态、刷新后恢复的问题,删除现在会立即持久化
|
||||
- 修复 Users / DataSources / Settings 表格“备注/状态”和“操作”之间的空白占位列问题
|
||||
- 修复 `useCollapsedActions` 未释放 `ResizeObserver` 导致的潜在内存泄漏与重复回调问题
|
||||
|
||||
---
|
||||
|
||||
## [0.27.6] — 2026-04-15
|
||||
|
||||
### 🔧 Improvements
|
||||
- BGP 告警页表格纵向 overflow 修复:补全 flex 布局链,tabs content-holder 正确撑满剩余高度
|
||||
- 用户管理表格横向滚动修复:采用 flex-fill 方案替换 `height: auto !important`,自定义滚动条 X 轨道位置对齐表格底部
|
||||
- Playground 宽布局隐藏"服务状态"按钮:侧边栏可见时不显示冗余入口
|
||||
- AI Chatbox 输入框失焦收起为单行,聚焦或有内容时展开完整 composer
|
||||
|
||||
---
|
||||
|
||||
## [0.27.4] — 2026-04-14
|
||||
|
||||
### 🔧 Improvements
|
||||
- info-card 改为懒加载动态挂载:页面初始 DOM 不再含隐藏的 `#info-panel` 节点,仅首次点击交互元素时创建
|
||||
|
||||
---
|
||||
|
||||
## [0.27.5] — 2026-04-14
|
||||
|
||||
### 🔧 Improvements
|
||||
- 统一控制台多页面滚动体验:BGP、alerts、采集数据、用户管理、任务、设置、Playground 等区域接入自定义滚动条与表格滚动容器
|
||||
- 优化 BGP 与 alerts 页响应式布局:顶部概览卡在窄宽度下优先重排,必要时才启用横向滚动,避免卡片裁切和全局滚动条接管
|
||||
- 调整 `situational alerts` 布局策略:统计卡按宽度在单行、两列和横滚之间切换,下方详情卡保持单行高度优先
|
||||
- 实时采集进度优化:一键采集完成后在未刷新页面时保留 100% 完成态,不再错误归零
|
||||
- 补充 UE5 MVP 融合方案文档,完善后续集成规划沉淀
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复 BGP summary 与 alerts 顶部卡片在无真实溢出时误出现横向滚动的问题
|
||||
- 修复 alerts 页面缩窄后外层全局竖向滚动被接管的问题,恢复“一屏内、内部滚动”的布局逻辑
|
||||
- 修复 `situational alerts` 在两排布局下详情卡竖向溢出的问题,改为更稳定的分区响应式排版
|
||||
- 修复自定义滚动条交互反馈,悬停、聚焦、拖拽时颜色加深但不再显示多余外圈
|
||||
|
||||
---
|
||||
|
||||
## [0.27.3] — 2026-04-14
|
||||
|
||||
### 🔧 Improvements
|
||||
- TV panel meta 折叠展开方向稳定:底部锚定时向上生长,拖拽后(顶部锚定)通过 JS 补偿 top 保持播放器底部位置不变
|
||||
- 修复 TV panel 展开/折叠时视频区域跳动问题:移除面板 min-height,使播放器高度在两种状态下保持一致
|
||||
- 修正 TV panel meta toggle 箭头方向:展开朝下,折叠朝上
|
||||
- 修复图例面板折叠按钮失效(legend-bar-btn 补充进拖拽排除列表)
|
||||
- 调整图层搜索框图标尺寸为 20px,BR 缩放角标改为直角 L 形
|
||||
|
||||
---
|
||||
|
||||
## [0.27.2] — 2026-04-14
|
||||
|
||||
### 🔧 Improvements
|
||||
- 修复 brand copy 宽度不随内容收缩的问题,现在与 title 图片宽度保持一致
|
||||
- 提取 `--brand-copy-width` CSS 自定义属性,消除 160px / 172px 魔法数字重复
|
||||
|
||||
---
|
||||
|
||||
## [0.27.1] — 2026-04-14
|
||||
|
||||
### 🔧 Improvements
|
||||
- 面板拖拽新增 L 形边界约束,其他面板无法覆盖 brand 面板区域,并从右侧/底部自然卡边
|
||||
- brand 组件引入 `--brand-scale` 整体缩放变量,padding 与内容尺寸独立控制
|
||||
- 图层控制面板宽度收窄(260px),与 brand 面板错落排列,间距调大
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复搜索框 `type="search"` 导致清除按钮重复显示的问题
|
||||
- 修复 `[hidden]` 属性被组件 `display` 规则覆盖的问题
|
||||
|
||||
---
|
||||
|
||||
## 0.27.0
|
||||
|
||||
Released: 2026-04-14
|
||||
|
||||
### Highlights
|
||||
|
||||
- 全面重构 Earth HUD 布局:品牌面板、图层控制面板、信息详情卡片各自独立,支持拖拽与折叠,信息卡片改为跟随点击位置悬浮显示。
|
||||
- 新增图层控制面板(Layer Panel),海缆、卫星、地形、BGP 等图层集中管理,支持关键字搜索过滤。
|
||||
- Earth 大气层渲染升级,引入 Fresnel 着色器双层辉光效果和深度遮挡球体。
|
||||
|
||||
### ✨ Features
|
||||
|
||||
- 新增独立 Layer Panel(`js/controls.js`, `css/layer-panel.css`),图层开关、搜索过滤、折叠收起,取代原工具栏弹出菜单
|
||||
- 信息详情面板(info-panel)改为点击时定位到鼠标附近(`js/info-card.js`),悬停改为轻量 tooltip,降低视觉干扰
|
||||
- Earth 材质重构(`js/earth.js`, `js/constants.js`):新增 `EARTH_MATERIAL_CONFIG`,Fresnel 内外大气层辉光、深度遮挡球体,纹理加载独立为 `loadEarthTexture()`
|
||||
|
||||
### 🔧 Improvements
|
||||
|
||||
- Earth Stats 面板改为 2 列 KPI 网格布局,支持拖拽和关闭(`css/earth-stats.css`)
|
||||
- 数据加载改为分步串行(登陆点 → 海缆 → 卫星 → BGP → 纹理),每步之间 yield 帧,改善视觉渐现体验(`js/main.js`)
|
||||
- 图层按钮状态更新逻辑统一至 `updateLayerButtonState()`,消除重复实现
|
||||
- 海缆状态识别新增 `active` 枚举值(兼容旧 `In Service`)
|
||||
|
||||
---
|
||||
|
||||
## 0.26.1
|
||||
|
||||
Released: 2026-04-12
|
||||
|
||||
### Highlights
|
||||
|
||||
- Cleaned up the first TV follow-up and replaced the dashboard sidebar's brittle one-off scroll behavior with a reusable `Scrollbar` component, so the new live module code is easier to maintain and the console navigation can scroll without layout jitter.
|
||||
|
||||
### Improved
|
||||
|
||||
- Improved [frontend/src/components/Scrollbar/Scrollbar.tsx](/home/ray/dev/linkong/planet/frontend/src/components/Scrollbar/Scrollbar.tsx), [frontend/src/components/AppLayout/AppLayout.tsx](/home/ray/dev/linkong/planet/frontend/src/components/AppLayout/AppLayout.tsx), and [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css) by extracting the sidebar scrollbar into a dedicated component with explicit `x / y / both` axis support, hidden native scrollbars, compact account/version rows, and a sidebar-only vertical setup instead of the earlier patchwork CSS glued directly onto the layout.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fixed [frontend/public/earth/js/tv.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/tv.js) by refactoring repeated iframe/video reset paths into shared helpers, so TV source switching, empty-state fallback, and playback retry handling no longer duplicate cleanup logic across multiple branches.
|
||||
|
||||
## 0.26.2
|
||||
|
||||
Released: 2026-04-12
|
||||
|
||||
### Highlights
|
||||
|
||||
- Stabilized the new reusable scrollbar work by restoring reliable sidebar visibility and extending the same floating scrollbar language to the Data Sources tables without letting scrollbars squeeze layout width or regress the console navigation.
|
||||
|
||||
### Added
|
||||
|
||||
- Added [frontend/src/components/Scrollbar/ScrollbarOverlay.tsx](/home/ray/dev/linkong/planet/frontend/src/components/Scrollbar/ScrollbarOverlay.tsx) as an overlay variant that binds to existing scroll containers such as Ant Table bodies, so heavy data grids can adopt the new scrollbar visuals without replacing their built-in scrolling mechanics.
|
||||
|
||||
### Improved
|
||||
|
||||
- Improved [frontend/src/components/Scrollbar/Scrollbar.tsx](/home/ray/dev/linkong/planet/frontend/src/components/Scrollbar/Scrollbar.tsx), [frontend/src/components/AppLayout/AppLayout.tsx](/home/ray/dev/linkong/planet/frontend/src/components/AppLayout/AppLayout.tsx), and [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css) by reverting the sidebar to a reliable vertical-first scrollbar path, then reintroducing automatic dual-axis support with independent floating tracks that no longer hide the thumb when only the sidebar needs vertical scrolling.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fixed [frontend/src/pages/DataSources/DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/DataSources/DataSources.tsx) and [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css) so the built-in and custom datasource tables now use the new overlay scrollbar instead of native table scrollbars, keeping horizontal and vertical scrolling available without changing Ant Table’s internal layout behavior.
|
||||
|
||||
## 0.26.0
|
||||
|
||||
Released: 2026-04-12
|
||||
|
||||
### Highlights
|
||||
|
||||
- Added an operator-facing TV live module to Earth, including backend-configurable live sources, a draggable/resizable live-news HUD window, default global news channels, and a dedicated settings workflow so the Earth page can open real news playback instead of only static telemetry.
|
||||
|
||||
### Added
|
||||
|
||||
- Added [backend/app/api/v1/tv.py](/home/ray/dev/linkong/planet/backend/app/api/v1/tv.py), [backend/app/services/tv_streams.py](/home/ray/dev/linkong/planet/backend/app/services/tv_streams.py), and [backend/app/services/collectors/news_live_streams.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/news_live_streams.py) to provide TV source configuration, public stream payloads, a guarded HLS proxy path, and a collector entry point for future world-news live-source ingestion.
|
||||
- Added the Earth TV HUD workspace through [frontend/public/earth/index.html](/home/ray/dev/linkong/planet/frontend/public/earth/index.html), [frontend/public/earth/js/tv.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/tv.js), and [frontend/public/earth/css/tv-panel.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/tv-panel.css), including toolbar access, draggable/closable behavior, resize support, direct video/HLS playback, iframe fallback, and per-channel external-open handling.
|
||||
- Added [docs/deprecated/earth-tv-live-module-plan.md](/home/ray/dev/linkong/planet/docs/deprecated/earth-tv-live-module-plan.md) and [docs/earth/technical/news-live-streams-collector-format.md](/home/ray/dev/linkong/planet/docs/technical/earth-news-live-streams-collector-format.md) to document the TV module rollout plan and the expected collector payload format for future curated live-channel ingestion.
|
||||
|
||||
### Improved
|
||||
|
||||
- Improved [frontend/src/pages/Settings/Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Settings/Settings.tsx), [backend/app/api/v1/settings.py](/home/ray/dev/linkong/planet/backend/app/api/v1/settings.py), and [backend/app/core/datasource_defaults.py](/home/ray/dev/linkong/planet/backend/app/core/datasource_defaults.py) by adding TV source administration to system settings and registering the `news_live_streams` datasource as a first-class configurable collector.
|
||||
- Improved [backend/app/services/tv_streams.py](/home/ray/dev/linkong/planet/backend/app/services/tv_streams.py) by seeding a curated first-pass news channel catalog that now defaults to `CGTN English` YouTube playback while keeping `CCTV-4` as a built-in fallback and exposing additional Reuters, CGTN, DW, Al Jazeera, Arirang, ABP, and SABC entries for operator testing.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fixed [frontend/public/earth/js/tv.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/tv.js) and [frontend/public/earth/index.html](/home/ray/dev/linkong/planet/frontend/public/earth/index.html) so HLS/video playback now actively attempts autoplay in the TV panel instead of only loading metadata and leaving the player visually idle.
|
||||
- Fixed [frontend/public/earth/css/tv-panel.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/tv-panel.css) so the TV source selector and action controls better match the Earth HUD dark theme instead of falling back to a bright native dropdown presentation.
|
||||
|
||||
## 0.25.3
|
||||
|
||||
Released: 2026-04-11
|
||||
|
||||
### Highlights
|
||||
|
||||
- Refined the Earth HUD visual system into a calmer operator-facing style, turned the top-left Earth brand into a real reusable component with language-driven rendering, and cleaned up duplicated brand assets so the page now has a single source of truth for HUD branding.
|
||||
|
||||
### Added
|
||||
|
||||
- Added [frontend/public/earth/js/brand.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/brand.js) as a reusable Earth `brand` component that renders the top-left logo/title/subtitle block from a shared config instead of hardcoding the structure in HTML.
|
||||
- Added [frontend/public/earth/assets/brand/earth-logo.svg](/home/ray/dev/linkong/planet/frontend/public/earth/assets/brand/earth-logo.svg), [frontend/public/earth/assets/brand/title-zh.svg](/home/ray/dev/linkong/planet/frontend/public/earth/assets/brand/title-zh.svg), and [frontend/public/earth/assets/brand/title-en.svg](/home/ray/dev/linkong/planet/frontend/public/earth/assets/brand/title-en.svg) as the canonical Earth brand asset set.
|
||||
|
||||
### Improved
|
||||
|
||||
- Improved [frontend/public/earth/css/base.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/base.css), [frontend/public/earth/css/hud.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/hud.css), [frontend/public/earth/css/coordinates-display.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/coordinates-display.css), [frontend/public/earth/css/legend.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/legend.css), [frontend/public/earth/css/earth-stats.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/earth-stats.css), and [frontend/public/earth/css/toolbar.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/toolbar.css) by rebalancing the Earth HUD into a more restrained deep-blue control-room look instead of the earlier over-layered glass-and-neon mix.
|
||||
- Improved [frontend/public/earth/index.html](/home/ray/dev/linkong/planet/frontend/public/earth/index.html), [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js), and [frontend/public/earth/js/constants.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/constants.js) by moving the Earth brand mount to a dedicated root and letting `HUD_CONFIG.brandLanguage` choose between `zh` and `en` without embedding the language decision in the DOM.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fixed [frontend/public/earth/css/info-panel.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/info-panel.css) so the Earth brand area now uses consistent `earth-brand` component selectors and English-specific typography hooks, avoiding the earlier one-off `brand-banner` naming drift and duplicated title styles.
|
||||
|
||||
## 0.25.2
|
||||
|
||||
Released: 2026-04-10
|
||||
|
||||
### Highlights
|
||||
|
||||
- Refined the Earth HUD operator polish so settings now behave like a true bounded menu, HUD panels render at the correct scale from the first frame, and dragged panels animate cleanly into and out of maximized layout targets without drifting to screen edges.
|
||||
|
||||
### Improved
|
||||
|
||||
- Improved [frontend/public/earth/index.html](/home/ray/dev/linkong/planet/frontend/public/earth/index.html) by precomputing the initial `--hud-scale` before Earth CSS loads, so HUD panels no longer flash at full size before shrinking to the target scale.
|
||||
- Improved [frontend/public/earth/css/hud.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/hud.css) by cleaning up duplicated settings-modal close-button styles, aligning the settings title with the shared HUD title system, and constraining the settings sheet to a stable centered width instead of viewport-relative modal sizing.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fixed [frontend/public/earth/js/controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js) so dragged HUD panels now fly from their dragged positions into maximized layout targets, closed panels stay out of the transition, and restoring layout clears drag overrides back to the initial positions.
|
||||
- Fixed [frontend/public/earth/js/controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js) and [frontend/public/earth/css/hud.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/hud.css) so layout transitions no longer visibly stick to screen edges before landing; the FLIP motion now composes with the existing corner transforms instead of fighting them.
|
||||
|
||||
## 0.25.1
|
||||
|
||||
Released: 2026-04-10
|
||||
|
||||
### Highlights
|
||||
|
||||
- Cleaned up the first persistent Playground rollout, fixed sidebar submenu persistence to match the intended operator behavior, and added reusable code-hygiene rules to prevent this class of drift from accumulating again.
|
||||
|
||||
### Improved
|
||||
|
||||
- Improved [backend/app/services/playground_chat_service.py](/home/ray/dev/linkong/planet/backend/app/services/playground_chat_service.py) and [frontend/src/pages/Playground/Playground.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Playground/Playground.tsx) by extracting repeated lookup, response, and request-action paths into shared helpers, reducing duplicated Playground flow code without changing behavior.
|
||||
- Improved [rules.md](/home/ray/dev/linkong/planet/rules.md) by adding a new `Code Hygiene - MANDATORY` section covering single-source-of-truth state, transitional cleanup, repeated-logic extraction, layout debugging order, and post-feature cleanup expectations.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fixed [frontend/src/components/AppLayout/AppLayout.tsx](/home/ray/dev/linkong/planet/frontend/src/components/AppLayout/AppLayout.tsx) so first-level menu expansion now behaves correctly across in-app navigation: `采集与数据` remains the default expanded group after refresh, while manually expanded groups stay open when navigating to their own child routes and reset only on page reload.
|
||||
|
||||
## 0.25.0
|
||||
|
||||
Released: 2026-04-10
|
||||
|
||||
### Highlights
|
||||
|
||||
- Turned `AI Playground` into a persistent backend-backed chat workspace, added dedicated alert workspaces as a foundation for future situational analysis, and aligned the operator UI around a more structured AI + alerts workflow instead of one-off playground calls.
|
||||
|
||||
### Added
|
||||
|
||||
- Added [backend/app/models/playground_session.py](/home/ray/dev/linkong/planet/backend/app/models/playground_session.py), [backend/app/models/playground_message.py](/home/ray/dev/linkong/planet/backend/app/models/playground_message.py), and [backend/app/services/playground_chat_service.py](/home/ray/dev/linkong/planet/backend/app/services/playground_chat_service.py) so Playground conversations, execution state, edits, retries, and stop/resume semantics are persisted in the backend database rather than living only in browser state.
|
||||
- Added [backend/app/services/alert_ai_brief.py](/home/ray/dev/linkong/planet/backend/app/services/alert_ai_brief.py), [backend/app/services/situational_alert_ai_brief.py](/home/ray/dev/linkong/planet/backend/app/services/situational_alert_ai_brief.py), and the dedicated alert pages [frontend/src/pages/Alerts/SystemAlerts.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Alerts/SystemAlerts.tsx), [frontend/src/pages/Alerts/BGPAlerts.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Alerts/BGPAlerts.tsx), and [frontend/src/pages/Alerts/SituationalAlerts.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Alerts/SituationalAlerts.tsx) to establish the alert-analysis foundation for later situational awareness expansion.
|
||||
|
||||
### Improved
|
||||
|
||||
- Improved [frontend/src/pages/Playground/Playground.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Playground/Playground.tsx) and [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css) by rebuilding Playground into a true chatbox workflow with persistent history, edit-and-resend behavior, grounded message actions, responsive composer behavior, bottom-stick scrolling, and tighter mobile layout handling.
|
||||
- Improved [frontend/src/components/AppLayout/AppLayout.tsx](/home/ray/dev/linkong/planet/frontend/src/components/AppLayout/AppLayout.tsx), [frontend/src/App.tsx](/home/ray/dev/linkong/planet/frontend/src/App.tsx), and [frontend/src/pages/Alerts/Alerts.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Alerts/Alerts.tsx) by reorganizing navigation around `采集与数据`, `专题观测`, and split alert entries so the app can scale to more observability and situational modules without turning the top-level UI into a single overloaded page.
|
||||
- Improved [README.md](/home/ray/dev/linkong/planet/README.md) and [docs/agents/situational-awareness-foundation-plan.md](/home/ray/dev/linkong/planet/docs/plans/agents-situational-awareness-foundation-plan.md) by documenting the current AI/alerts base, planned situational-awareness direction, and the new persistent Playground foundation.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fixed [backend/app/services/playground_chat_service.py](/home/ray/dev/linkong/planet/backend/app/services/playground_chat_service.py) so background Playground runs explicitly commit state transitions, allowing frontend polling to observe real pending/thinking/answering/done states instead of seeing stale empty threads.
|
||||
- Fixed [frontend/src/services/situational-awareness/index.ts](/home/ray/dev/linkong/planet/frontend/src/services/situational-awareness/index.ts) by removing the temporary mock gateway path, so Playground and alert-related AI flows now reflect the real backend/provider chain instead of local fake responses.
|
||||
|
||||
## 0.24.8
|
||||
|
||||
Released: 2026-04-10
|
||||
|
||||
### Highlights
|
||||
|
||||
- Refined the BGP AI brief operator workflow so the tab now stays compact and metadata-focused, while the full Markdown brief opens in a bounded modal that respects the repo’s single-screen layout rules.
|
||||
|
||||
### Improved
|
||||
|
||||
- Improved [frontend/src/pages/BGP/BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx) by turning the brief action row into a clearer `历史简报下拉框 + 查看 + 生成` flow, keeping inline metadata visible in the tab while moving full Markdown reading into a dedicated modal workspace.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fixed [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css) and [frontend/src/pages/BGP/BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx) so the BGP brief modal no longer spills below the viewport; long brief content now scrolls inside the modal body instead of extending past the visible screen.
|
||||
|
||||
## 0.24.7
|
||||
|
||||
Released: 2026-04-10
|
||||
|
||||
### Highlights
|
||||
|
||||
- Formalized repository release hygiene and frontend layout guardrails so repeated versioning chores and recurring layout regressions now have explicit repo-level rules instead of living only in conversation context.
|
||||
|
||||
### Added
|
||||
|
||||
- Added [release-workflow/SKILL.md](/home/ray/dev/linkong/planet/.codex/skills/release-workflow/SKILL.md), defining the repository release workflow for version bumps, changelog/version-history updates, minimal validation, and commit/push sequencing.
|
||||
|
||||
### Improved
|
||||
|
||||
- Improved [rules.md](/home/ray/dev/linkong/planet/rules.md) by adding mandatory release-workflow requirements and a new frontend layout constraint section covering single-screen workspaces, overflow ownership, tab-pane behavior, compact-mode expectations, and readable-card fallbacks.
|
||||
- Improved [frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md) by summarizing the recurring Earth, Playground, BGP, and admin-layout regressions into concrete constraints for future frontend work, including “prefer scrollbars over unreadable compression” and “do not treat every tab as a table pane.”
|
||||
|
||||
## 0.24.6
|
||||
|
||||
Released: 2026-04-10
|
||||
|
||||
### Highlights
|
||||
|
||||
- Tightened several backend hot paths outside the original BGP page fixes, stabilized BGP collector coverage after the recent query refactors, and rebuilt the BGP AI brief tab so saved Markdown briefs render and scroll like a proper operator workspace instead of collapsing inside the shared table layout.
|
||||
|
||||
### Improved
|
||||
|
||||
- Improved [backend/app/api/v1/datasources.py](/home/ray/dev/linkong/planet/backend/app/api/v1/datasources.py) by replacing per-datasource task, count, and endpoint lookups with batched prefetch helpers, reducing the worst `1 + N` behavior on the datasource list and `trigger-all` flow.
|
||||
- Improved [backend/app/api/v1/visualization.py](/home/ray/dev/linkong/planet/backend/app/api/v1/visualization.py) by switching the main Earth-facing `CollectedData` endpoints to `is_current` records, batching multi-source loads for aggregate endpoints, and removing stale Python-side dedupe paths from the hot route.
|
||||
- Improved [backend/app/services/bgp_incidents.py](/home/ray/dev/linkong/planet/backend/app/services/bgp_incidents.py) and [backend/app/services/bgp_enrichment.py](/home/ray/dev/linkong/planet/backend/app/services/bgp_enrichment.py) by avoiding historical full-table infrastructure scans, narrowing observation baseline payloads to required columns, and pushing more ASN filtering into the database.
|
||||
- Improved [backend/app/api/v1/alerts.py](/home/ray/dev/linkong/planet/backend/app/api/v1/alerts.py), [backend/app/api/v1/dashboard.py](/home/ray/dev/linkong/planet/backend/app/api/v1/dashboard.py), and [backend/app/api/v1/settings.py](/home/ray/dev/linkong/planet/backend/app/api/v1/settings.py) by collapsing several repeated count and settings queries into fewer aggregate or batched reads.
|
||||
- Improved [frontend/src/pages/BGP/BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx), [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css), and [frontend/src/components/MarkdownRenderer/MarkdownRenderer.tsx](/home/ray/dev/linkong/planet/frontend/src/components/MarkdownRenderer/MarkdownRenderer.tsx) by rebuilding the `AI 简报` tab layout, fixing saved brief scrolling behavior, and extending the renderer to handle tables, separators, and stored metadata comments more gracefully.
|
||||
- Improved [docs/frontend/plans/ai-playground-development-plan.md](/home/ray/dev/linkong/planet/docs/plans/frontend-ai-playground-development-plan.md) by explicitly recording that the current BGP brief is only the first-stage summary flow and that regional prefix-geography analysis remains a planned Phase B follow-up.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fixed [backend/app/services/bgp_collectors.py](/home/ray/dev/linkong/planet/backend/app/services/bgp_collectors.py) and [backend/app/services/bgp_enrichment.py](/home/ray/dev/linkong/planet/backend/app/services/bgp_enrichment.py) so JSON field extraction no longer depends on the less portable `.astext` path that could break BGP collector endpoints in local environments.
|
||||
- Fixed the BGP AI brief tab in [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css) and [frontend/src/pages/BGP/BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx) so long saved briefs are no longer compressed into a tiny clipped viewport by the shared tab/table overflow rules.
|
||||
|
||||
## 0.24.4
|
||||
|
||||
Released: 2026-04-09
|
||||
|
||||
### Highlights
|
||||
|
||||
- Refined the `planet.sh` AI Provider rebuild UX so image rebuilds now feel like first-class scripted tasks, with clearer stage boundaries and cleaner fallback messaging instead of leaking raw Compose output into the terminal.
|
||||
|
||||
### Improved
|
||||
|
||||
- Improved [planet.sh](/home/ray/dev/linkong/planet/planet.sh) by keeping AI Provider image build logs in a temporary file, surfacing stage-specific detail copy for `docker compose v2` and `docker-compose v1`, and showing an explicit success line when the image rebuild finishes before container health checks begin.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fixed [planet.sh](/home/ray/dev/linkong/planet/planet.sh) so the AI Provider image rebuild stage no longer dumps raw Compose build output into the main spinner flow during normal successful runs.
|
||||
- Fixed [planet.sh](/home/ray/dev/linkong/planet/planet.sh) so the `构建 AI Provider 镜像` phase now ends with an explicit completion signal instead of visually blending into the subsequent container health-check phase.
|
||||
|
||||
## 0.24.3
|
||||
|
||||
Released: 2026-04-09
|
||||
|
||||
### Highlights
|
||||
|
||||
- Extended `AI Playground` from a minimal prompt form into a more repeatable diagnostics workspace, and hardened `planet.sh` so AI Provider restarts can rebuild changed images and expose clearer Compose fallback behavior.
|
||||
|
||||
### Improved
|
||||
|
||||
- Improved [frontend/src/pages/Playground/Playground.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Playground/Playground.tsx) by adding preset scenarios, response metadata, `thinking` block visibility, raw JSON inspection, and copy actions so the page works more like a proper AI diagnostics console.
|
||||
- Improved [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css) by styling Playground presets, result metadata, raw response sections, and responsive action groups without breaking the single-screen workspace layout.
|
||||
- Improved [planet.sh](/home/ray/dev/linkong/planet/planet.sh) so AI Provider restarts detect code/config changes, rebuild the image when needed, and surface which Compose path is being used during image and container operations.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fixed the AI Provider restart path in [planet.sh](/home/ray/dev/linkong/planet/planet.sh) so code changes inside `aiprovider/` no longer stay hidden behind an old container image after `restart -a`.
|
||||
- Fixed Compose error reporting in [planet.sh](/home/ray/dev/linkong/planet/planet.sh) by making the script explicitly show `docker compose v2` first, then `docker-compose v1`, and only fail after both execution paths are exhausted.
|
||||
|
||||
## 0.24.2
|
||||
|
||||
Released: 2026-04-09
|
||||
|
||||
### Highlights
|
||||
|
||||
- Fixed the public-entry and AI diagnostics regressions introduced during the recent frontend routing and playground work, while also reducing the main frontend bundle by switching to route-level lazy loading and more targeted vendor chunking.
|
||||
|
||||
### Improved
|
||||
|
||||
- Improved [frontend/src/App.tsx](/home/ray/dev/linkong/planet/frontend/src/App.tsx) by lazy-loading admin pages and large workspaces through `React.lazy()` plus `Suspense`, so the initial frontend entry no longer pulls every route into the first bundle.
|
||||
- Improved [frontend/vite.config.ts](/home/ray/dev/linkong/planet/frontend/vite.config.ts) by adding targeted manual chunking for React, icon, network, and Earth-related vendor dependencies instead of leaving everything in one monolithic application bundle.
|
||||
- Improved [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css) by adding a shared route-loading state and making the Playground help panel size to its content instead of stretching to fill the sidebar.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fixed [frontend/src/App.tsx](/home/ray/dev/linkong/planet/frontend/src/App.tsx) so anonymous visits to `/` once again reach the public Earth entry through the existing `/ -> /earth` redirect instead of being intercepted by the login screen.
|
||||
- Fixed [frontend/src/pages/Playground/Playground.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Playground/Playground.tsx) so cached provider status is shown immediately but still refreshed from the backend, avoiding stale diagnostics after `.env` or provider changes within the same browser tab.
|
||||
- Fixed [frontend/src/pages/Playground/Playground.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Playground/Playground.tsx) and [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css) so the `测试说明` card no longer over-expands and now fits its content more naturally in the sidebar.
|
||||
|
||||
## 0.24.1
|
||||
|
||||
Released: 2026-04-09
|
||||
|
||||
### Highlights
|
||||
|
||||
- Refined the `/earth` HUD into a cleaner class-first structure with responsive scaling, clearer CSS layer boundaries, and lower coupling between HTML, CSS, and runtime UI updates.
|
||||
- Hardened local startup conventions around Bun so frontend tooling, docs, and `planet.sh` behave more predictably in fresh Ubuntu and mixed WSL environments.
|
||||
|
||||
### Added
|
||||
|
||||
- Added [frontend/public/earth/css/hud.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/hud.css), extracting shared HUD panel surfaces, shared HUD typography rows, status messaging, tooltip overlays, and layout-expanded panel transitions out of the old monolithic base layer.
|
||||
- Added [frontend/public/earth/css/toolbar.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/toolbar.css), isolating Earth toolbar, popover, zoom dock, liquid-glass button, and toolbar-tooltip behavior into a dedicated toolbar layer.
|
||||
- Added explicit Bun package-manager metadata to [frontend/package.json](/home/ray/dev/linkong/planet/frontend/package.json) so the frontend package manager choice is declared instead of inferred from the lockfile alone.
|
||||
|
||||
### Improved
|
||||
|
||||
- Improved [frontend/public/earth/css/base.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/base.css) by reducing it to app-shell concerns only: global tokens, Earth app container, loading panel, and shared animation primitives.
|
||||
- Improved [frontend/public/earth/index.html](/home/ray/dev/linkong/planet/frontend/public/earth/index.html) by wiring the new CSS layer order, standardizing HUD utility classes on `hud-panel-*`, and replacing generic toolbar/tooltip hooks with more explicit Earth-specific classes.
|
||||
- Improved [frontend/public/earth/css/info-panel.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/info-panel.css), [frontend/public/earth/css/coordinates-display.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/coordinates-display.css), [frontend/public/earth/css/legend.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/legend.css), and [frontend/public/earth/css/earth-stats.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/earth-stats.css) by leaving only panel-specific responsibilities in each file after the shared HUD and toolbar primitives moved out.
|
||||
- Improved [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js), [frontend/public/earth/js/controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js), [frontend/public/earth/js/ui.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/ui.js), and [frontend/public/earth/js/legend.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/legend.js) by aligning runtime DOM queries and generated markup with the new class-first HUD and toolbar structure.
|
||||
- Improved [frontend/public/earth/js/constants.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/constants.js) and [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js) by keeping HUD scaling configurable through extracted constants instead of scattering scaling assumptions through the main Earth entrypoint.
|
||||
- Improved [planet.sh](/home/ray/dev/linkong/planet/planet.sh) by prepending `~/.local/bin` and `~/.bun/bin` automatically, preferring direct `~/.bun/bin/bun` detection, and auto-installing missing `uv`/`bun` instead of requiring the user's interactive shell config to expose them first.
|
||||
- Improved [README.md](/home/ray/dev/linkong/planet/README.md), [project_context.md](/home/ray/dev/linkong/planet/project_context.md), [rules.md](/home/ray/dev/linkong/planet/rules.md), and [scripts/bootstrap-dev.sh](/home/ray/dev/linkong/planet/scripts/bootstrap-dev.sh) by making the frontend Bun-only workflow explicit in both onboarding docs and command-line guidance.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fixed the Earth HUD cleanup path where splitting styles previously left the page with a missing `base.css` entry and mismatched utility class names during the refactor.
|
||||
- Fixed several Earth UI update paths so toolbar tooltip text, legend re-rendering, status message classes, and mouse coordinate styling continue to work after removing legacy fallback selectors.
|
||||
- Fixed the local developer bootstrap path where `planet.sh start` could fail in a clean Ubuntu or agent shell session simply because Bun or uv were installed outside the current shell's inherited `PATH`.
|
||||
|
||||
## 0.24.0
|
||||
|
||||
Released: 2026-04-09
|
||||
|
||||
### Highlights
|
||||
|
||||
- Added a dedicated `AI Playground` admin entry so operators can validate provider connectivity and run controlled situational-analysis prompts from the main frontend without introducing a separate UI service.
|
||||
- Established a first explicit frontend layout rulebook centered on single-screen workspaces, internal module scrolling, and BGP-style page composition for future admin pages.
|
||||
|
||||
### Added
|
||||
|
||||
- Added [frontend/src/pages/Playground/Playground.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Playground/Playground.tsx), introducing the first dedicated AI testing workspace with provider status visibility, prompt/result tabs, and collapsible operator guidance.
|
||||
- Added [docs/frontend/technical/frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md), documenting the repository standard for one-screen admin workspaces and module-local overflow handling.
|
||||
- Added [docs/frontend/plans/ai-playground-development-plan.md](/home/ray/dev/linkong/planet/docs/plans/frontend-ai-playground-development-plan.md), capturing the completed AI gateway/UI work and the next delivery phases for BGP briefs, evidence-first inputs, and future agent runtime expansion.
|
||||
|
||||
### Improved
|
||||
|
||||
- Improved [frontend/src/components/AppLayout/AppLayout.tsx](/home/ray/dev/linkong/planet/frontend/src/components/AppLayout/AppLayout.tsx) and [frontend/src/App.tsx](/home/ray/dev/linkong/planet/frontend/src/App.tsx) by wiring `AI Playground` into the main admin navigation and route tree instead of pointing operators to a nonexistent `aiprovider` chat page.
|
||||
- Improved [planet.sh](/home/ray/dev/linkong/planet/planet.sh) by replacing the incorrect `localhost:8010/chat` closeout link with the frontend `AI Playground` entry.
|
||||
- Improved [docker-compose.yml](/home/ray/dev/linkong/planet/docker-compose.yml) by attaching `./aiprovider/.env` to the `aiprovider` service so provider identity, model, and credentials actually reach the running container.
|
||||
- Improved [README.md](/home/ray/dev/linkong/planet/README.md) by linking the new frontend layout guidance and AI Playground development plan.
|
||||
- Improved [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css) by refining the Playground workspace into a notebook-friendly left-sidebar plus right-tabbed layout, reusing thin scrollbars, and making provider/help/result regions degrade more gracefully under constrained height.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fixed the local AI status flow so provider configuration no longer appeared permanently `disabled / not configured` merely because `aiprovider/.env` was not mounted into the container.
|
||||
- Fixed repeated `Provider 状态` refetching when switching away from and back to `/playground` by caching the last known provider status within the browser session until the operator explicitly refreshes it.
|
||||
- Fixed several Playground layout regressions where auxiliary panels could push the result area out of view or clip provider details without exposing internal scrolling.
|
||||
|
||||
## 0.23.4
|
||||
|
||||
Released: 2026-04-08
|
||||
|
||||
### Highlights
|
||||
|
||||
- Fixed the BGP overview workspace so summary cards and tabular data now share viewport space more predictably, keeping the table header visible while preserving internal horizontal and vertical scrolling.
|
||||
|
||||
### Improved
|
||||
|
||||
- Improved [BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx) by switching BGP tables from frontend pagination to in-table scrolling, setting explicit horizontal scroll baselines per tab, and reshaping the summary cards into a desktop two-row layout with a compact single-row horizontal strip on tighter screens.
|
||||
- Improved [index.css](/home/ray/dev/linkong/planet/frontend/src/index.css) by hardening the BGP table layout chain from card body to Ant Design table body, so width overflow stays inside the table region and compact summary cards no longer consume unnecessary vertical space above the workspace.
|
||||
- Improved release consistency by aligning [VERSION](/home/ray/dev/linkong/planet/VERSION), [pyproject.toml](/home/ray/dev/linkong/planet/pyproject.toml), [frontend/package.json](/home/ray/dev/linkong/planet/frontend/package.json), and [uv.lock](/home/ray/dev/linkong/planet/uv.lock) on the same `0.23.4` bugfix version.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fixed the BGP collector/events tables so shrinking the browser window no longer clips the table card frame without exposing a usable horizontal scrollbar.
|
||||
- Fixed the BGP table region so the Ant Design header row is no longer obscured by an overgrown table body after the page switched to full-height internal scrolling.
|
||||
- Fixed the BGP summary area so medium and large screens no longer collapse the six KPI cards into overly narrow single-row tiles.
|
||||
|
||||
## 0.23.3
|
||||
|
||||
Released: 2026-04-08
|
||||
|
||||
### Highlights
|
||||
|
||||
- Refined `planet.sh` startup and restart presentation so service bring-up, health checks, and frontend dev-server readiness are easier to follow in real time.
|
||||
|
||||
### Improved
|
||||
|
||||
- Improved [planet.sh](/home/ray/dev/linkong/planet/planet.sh) by standardizing the script on `zsh`, tightening frontend startup stage rendering, and making spinner-driven subtask feedback show up step by step instead of bunching at the end.
|
||||
- Improved [planet.sh](/home/ray/dev/linkong/planet/planet.sh) by clarifying health-check subtask copy, filtering noisy frontend startup lines, and aligning terminal output spacing across spinner and status rows.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fixed several `zsh` compatibility issues in [planet.sh](/home/ray/dev/linkong/planet/planet.sh), including reserved variable name collisions during `restart` and container health checks.
|
||||
- Fixed [planet.sh](/home/ray/dev/linkong/planet/planet.sh) so frontend readiness animation no longer appears frozen while waiting for Vite startup and health probes.
|
||||
|
||||
## 0.23.2
|
||||
|
||||
Released: 2026-04-08
|
||||
|
||||
### Highlights
|
||||
|
||||
- Hardened datasource retrigger handling so operators can safely force reruns without losing control of task state visibility or rollback behavior.
|
||||
|
||||
### Improved
|
||||
|
||||
- Improved [frontend/src/pages/DataSources/DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/DataSources/DataSources.tsx) by mapping phase-local task percentages into a continuous overall progress bar, pre-checking running task status before retriggering, and consolidating the trigger/force-confirm flow.
|
||||
- Improved [planet.sh](/home/ray/dev/linkong/planet/planet.sh) by polishing startup CLI output, reducing duplicated spinner cleanup, and standardizing log/help output around the newer terminal presentation.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fixed forced datasource reruns in [backend/app/services/collectors/base.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/base.py) by rolling back invalid SQLAlchemy session state before marking cancelled or failed task cleanup, preventing `PendingRollbackError` during operator-triggered cancellation.
|
||||
- Fixed datasource trigger UX in [frontend/src/pages/DataSources/DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/DataSources/DataSources.tsx) so batch and single-source progress no longer jump straight to `100%` on frontend-side status coercion while the backend is still reporting real progress.
|
||||
|
||||
## 0.23.1
|
||||
|
||||
Released: 2026-04-07
|
||||
|
||||
### Highlights
|
||||
|
||||
- Fixed the BGP overview page so high-DPI and lower-height screens can keep the observation workspace visible within a single viewport.
|
||||
|
||||
### Improved
|
||||
|
||||
- Improved [BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx) by switching the page to a height-aware shell, compact summary layout, and tabbed data workspace so observation tables retain the majority of the screen.
|
||||
- Improved [index.css](/home/ray/dev/linkong/planet/frontend/src/index.css) by adding responsive BGP-specific compact spacing, denser table paddings, and an internal scroll region for collector coverage and event tables.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fixed the BGP observation page on small or high-scale displays where stacked stats and full-height blocks consumed too much vertical space, leaving only a couple of visible table rows.
|
||||
|
||||
## 0.23.0
|
||||
|
||||
Released: 2026-04-07
|
||||
@@ -23,7 +699,7 @@ Released: 2026-04-07
|
||||
- Added [backend/app/services/ai_client.py](/home/ray/dev/linkong/planet/backend/app/services/ai_client.py), introducing an internal HTTP client for `backend -> aiprovider` calls with request-id propagation and lightweight retry.
|
||||
- Added [aiprovider/main.py](/home/ray/dev/linkong/planet/aiprovider/main.py), [aiprovider/provider_service.py](/home/ray/dev/linkong/planet/aiprovider/provider_service.py), and related config/schema files to stand up the dedicated adapter service.
|
||||
- Added [aiprovider/.env.example](/home/ray/dev/linkong/planet/aiprovider/.env.example) and [docker-compose.local-model.yml](/home/ray/dev/linkong/planet/docker-compose.local-model.yml) as ready-to-edit local-model templates.
|
||||
- Added [docs/aiprovider.md](/home/ray/dev/linkong/planet/docs/aiprovider.md), documenting architecture, configuration, single-machine and multi-machine deployment, and cross-service calling patterns.
|
||||
- Added [docs/agents/aiprovider.md](/home/ray/dev/linkong/planet/docs/technical/agents-aiprovider.md), documenting architecture, configuration, single-machine and multi-machine deployment, and cross-service calling patterns.
|
||||
- Added a dedicated `重启 AI Provider` control path in [Dashboard.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Dashboard/Dashboard.tsx), [system_control.py](/home/ray/dev/linkong/planet/backend/app/services/system_control.py), and [system_restart_runner.py](/home/ray/dev/linkong/planet/backend/scripts/system_restart_runner.py).
|
||||
|
||||
### Improved
|
||||
@@ -31,6 +707,8 @@ Released: 2026-04-07
|
||||
- Improved backend-to-provider tracing by propagating `X-Request-ID` through the AI call chain and returning the same header from both backend and `aiprovider`.
|
||||
- Improved resilience by adding lightweight retry handling to both `backend -> aiprovider` and `aiprovider -> model provider` HTTP calls.
|
||||
- Improved operator workflow by folding `aiprovider` startup, health checks, restart support, and log viewing into [planet.sh](/home/ray/dev/linkong/planet/planet.sh).
|
||||
- Improved [planet.sh](/home/ray/dev/linkong/planet/planet.sh) by adding bounded retry and container-health self-recovery for dependency installs, database startup, `aiprovider` startup, and interactive PostgreSQL boot paths.
|
||||
- Improved [README.md](/home/ray/dev/linkong/planet/README.md) by documenting the new `planet.sh` retry and health-check tuning environment variables with concrete override examples.
|
||||
- Improved container consistency by switching [backend/Dockerfile](/home/ray/dev/linkong/planet/backend/Dockerfile) and [aiprovider/Dockerfile](/home/ray/dev/linkong/planet/aiprovider/Dockerfile) to `uv sync` / `uv run`.
|
||||
|
||||
### Changed
|
||||
@@ -147,7 +825,7 @@ Released: 2026-04-02
|
||||
|
||||
- Added a new `IPtoASN Prefix Geography` collector in [iptoasn.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/iptoasn.py) and registered it through [data_sources.yaml](/home/ray/dev/linkong/planet/backend/app/core/data_sources.yaml), [data_sources.py](/home/ray/dev/linkong/planet/backend/app/core/data_sources.py), [datasource_defaults.py](/home/ray/dev/linkong/planet/backend/app/core/datasource_defaults.py), and [collectors/__init__.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/__init__.py).
|
||||
- Added country centroid helpers in [countries.py](/home/ray/dev/linkong/planet/backend/app/core/countries.py) so country-level prefix geography can produce map coordinates instead of only labels.
|
||||
- Added a dedicated prefix-geography implementation note in [prefix-geography-plan.md](/home/ray/dev/linkong/planet/docs/prefix-geography-plan.md).
|
||||
- Added a dedicated prefix-geography implementation note in [prefix-geography-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-prefix-geography-plan.md).
|
||||
- Added recent `15m` collector activity dimensions to BGP coverage output in [bgp_collectors.py](/home/ray/dev/linkong/planet/backend/app/services/bgp_collectors.py) and [visualization.py](/home/ray/dev/linkong/planet/backend/app/api/v1/visualization.py).
|
||||
- Added additional BGP detector coverage for `route_leak_candidate` and `path_flap` flows in [test_bgp.py](/home/ray/dev/linkong/planet/backend/tests/test_bgp.py).
|
||||
- Added a local Earth cloud texture at [earth_clouds_1024.png](/home/ray/dev/linkong/planet/frontend/public/earth/assets/earth_clouds_1024.png) to avoid remote cloud-map dependency failures.
|
||||
@@ -162,7 +840,7 @@ Released: 2026-04-02
|
||||
- Improved Earth event animation semantics in [bgp.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp.js) by separating icon pulse from ring expansion so the center marker can breathe while the ring expands independently.
|
||||
- Improved Earth texture reliability in [earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js) by switching clouds back to a local static asset under the restored `public/earth` runtime.
|
||||
- Improved frontend boot noise in [frontend/index.html](/home/ray/dev/linkong/planet/frontend/index.html) by removing the default Vite favicon request that was generating irrelevant `vite.svg` timeouts during Earth debugging.
|
||||
- Improved project planning docs in [bgp-context.md](/home/ray/dev/linkong/planet/docs/bgp-context.md) and [TODO.md](/home/ray/dev/linkong/planet/TODO.md) so the roadmap now explicitly prioritizes `activity layer`, `prefix-centric geography`, and follow-up geofeed/whois work.
|
||||
- Improved project planning docs in [bgp-context.md](/home/ray/dev/linkong/planet/docs/technical/earth-bgp-context.md) and [TODO.md](/home/ray/dev/linkong/planet/TODO.md) so the roadmap now explicitly prioritizes `activity layer`, `prefix-centric geography`, and follow-up geofeed/whois work.
|
||||
|
||||
### Fixed
|
||||
|
||||
@@ -320,7 +998,7 @@ Released: 2026-03-31
|
||||
- Added restart-task Redis helpers and whitelist command mapping in [system_control.py](/home/ray/dev/linkong/planet/backend/app/services/system_control.py).
|
||||
- Added detached restart runner orchestration in [system_restart_runner.py](/home/ray/dev/linkong/planet/backend/scripts/system_restart_runner.py).
|
||||
- Added `-d` / `--database` support to [planet.sh](/home/ray/dev/linkong/planet/planet.sh) for database-only restarts.
|
||||
- Added restart control documentation in [system-service-control.md](/home/ray/dev/linkong/planet/docs/system-service-control.md).
|
||||
- Added restart control documentation in [system-service-control.md](/home/ray/dev/linkong/planet/docs/technical/backend-system-service-control.md).
|
||||
|
||||
### Improved
|
||||
|
||||
@@ -523,7 +1201,7 @@ Released: 2026-03-26
|
||||
|
||||
### Added
|
||||
|
||||
- Added a dedicated Earth module remediation plan in [earth-module-plan.md](/home/ray/dev/linkong/planet/docs/earth-module-plan.md).
|
||||
- Added a dedicated Earth module remediation plan in [earth-module-plan.md](/home/ray/dev/linkong/planet/docs/deprecated/earth-module-plan.md).
|
||||
- Added backend TLE helpers in [satellite_tle.py](/home/ray/dev/linkong/planet/backend/app/core/satellite_tle.py).
|
||||
- Added backend support for returning `tle_line1` and `tle_line2` from the satellite visualization API.
|
||||
|
||||
|
||||
22
docs/deprecated/README.md
Normal file
22
docs/deprecated/README.md
Normal file
@@ -0,0 +1,22 @@
|
||||
# Deprecated Docs
|
||||
|
||||
这个目录用于存放两类文档:
|
||||
|
||||
1. 已经完成、主要保留为历史记录的实施计划
|
||||
2. 已被现有实现或新方案替代的旧计划
|
||||
|
||||
放到这里并不代表这些文档“错误”,而是表示:
|
||||
|
||||
- 它们不再适合作为当前开发的主指导文档
|
||||
- 如果要了解历史决策、演进路径或旧设计背景,仍然可以参考
|
||||
|
||||
当前归档原则:
|
||||
|
||||
- 明确写明“已完成”的计划,优先归档
|
||||
- 已被正式实现替代、继续放在 `docs/` 根目录会误导后续开发的计划,归档
|
||||
- 仍然指导未来开发、尚未完成或仍有明确执行价值的文档,继续保留在 `docs/`
|
||||
|
||||
补充说明:
|
||||
|
||||
- 一部分归档文档来自外部或临时工作流草案,例如 sisyphus 生成的初稿
|
||||
- 这类文档如果有可用内容,应先吸收到 `docs/plans/` 或 `docs/technical/`,再归档保留来源记录
|
||||
117
docs/deprecated/earth-tv-live-module-plan.md
Normal file
117
docs/deprecated/earth-tv-live-module-plan.md
Normal file
@@ -0,0 +1,117 @@
|
||||
# Earth 电视直播模块计划
|
||||
|
||||
## 目标
|
||||
|
||||
为 `Earth` 页面增加一个可配置、可扩展、可拖拽的电视直播模块:
|
||||
|
||||
- 后台可配置新闻直播源
|
||||
- 默认兜底源为央视 `CCTV-4`
|
||||
- 未来可通过采集器接入世界各地新闻直播源
|
||||
- Earth 工具栏 `显示控制` 子菜单新增电视按钮
|
||||
- 点击后打开一个与其他 HUD 一致的可拖拽/可关闭窗口
|
||||
- 窗口内部可播放或承载新闻直播页面
|
||||
|
||||
## 设计原则
|
||||
|
||||
- 第一阶段先交付“后台可配 + Earth 可用 + 默认可回退”的版本
|
||||
- 公开读取接口与后台管理接口分离
|
||||
- 手工配置源与采集器源共用统一的前端消费结构
|
||||
- Earth 里的电视窗口必须复用现有 HUD 拖拽、关闭、布局最大化逻辑
|
||||
- 小屏下优先保证窗口完整显示,超出部分在窗口内部滚动
|
||||
|
||||
## 分阶段实现
|
||||
|
||||
### Phase 1:后端配置与公开读取
|
||||
|
||||
- 在系统设置中新增 `tv` 分类
|
||||
- 定义直播源配置结构:
|
||||
- `default_source_id`
|
||||
- `auto_fallback`
|
||||
- `sources[]`
|
||||
- 每个直播源至少包含:
|
||||
- `id`
|
||||
- `name`
|
||||
- `provider`
|
||||
- `region`
|
||||
- `language`
|
||||
- `source_type`
|
||||
- `embed_url`
|
||||
- `stream_url`
|
||||
- `homepage_url`
|
||||
- `is_enabled`
|
||||
- `is_fallback`
|
||||
- `sort_order`
|
||||
- `collector_source`
|
||||
- `notes`
|
||||
- 默认兜底源使用央视官网 `CCTV-4` 直播页
|
||||
- 新增公开读取接口,供 Earth 页面无登录态读取直播源配置
|
||||
|
||||
### Phase 2:采集器扩展位
|
||||
|
||||
- 新增 `news_live_streams` collector 占位
|
||||
- 规范采集器入库数据结构,使其能与后台手工配置源合并
|
||||
- TV 公开接口支持合并:
|
||||
- 后台手工配置源
|
||||
- 采集器入库源
|
||||
- 保持手工配置源优先级更高,避免采集器覆盖人工兜底配置
|
||||
|
||||
### Phase 3:后台配置界面
|
||||
|
||||
- 在系统配置页新增 `电视直播` tab
|
||||
- 支持:
|
||||
- 查看当前默认源
|
||||
- 开关自动回退
|
||||
- 新增直播源
|
||||
- 编辑直播源
|
||||
- 删除直播源
|
||||
- 启用/禁用直播源
|
||||
- 将某个直播源设为默认源
|
||||
- 明确区分:
|
||||
- 手工配置源
|
||||
- 采集器来源
|
||||
|
||||
### Phase 4:Earth HUD 集成
|
||||
|
||||
- 在 `显示控制` 子菜单加入电视按钮
|
||||
- 新增 TV HUD 面板:
|
||||
- 可拖拽
|
||||
- 可关闭
|
||||
- 支持显示/隐藏状态同步
|
||||
- 参与布局最大化与恢复布局
|
||||
- 面板内容至少包含:
|
||||
- 当前频道标题
|
||||
- 源切换下拉菜单
|
||||
- 刷新按钮
|
||||
- 打开官网按钮
|
||||
- 播放区域
|
||||
|
||||
### Phase 5:播放策略
|
||||
|
||||
- 第一版优先支持 `iframe`/嵌入页类直播源
|
||||
- 为未来扩展保留:
|
||||
- `hls`
|
||||
- `video`
|
||||
- `external`
|
||||
- 如果默认源不可用:
|
||||
- 优先回退到标记为 `is_fallback=true` 的源
|
||||
- 若无明确回退源,则回退到第一个可用源
|
||||
- 面板内要有清晰的加载、错误、回退提示
|
||||
|
||||
### Phase 6:打磨与清理
|
||||
|
||||
- 统一 HUD 风格
|
||||
- 小屏下限制窗口尺寸并启用内部滚动
|
||||
- 避免窗口超出屏幕
|
||||
- 补最小验证
|
||||
- 清理临时代码、重复样式和无用资源
|
||||
|
||||
## 首版交付定义
|
||||
|
||||
当以下条件满足时,认为首版可用:
|
||||
|
||||
- 后台可以配置新闻直播源
|
||||
- Earth 可以读取并显示默认直播源
|
||||
- 工具栏可打开电视窗口
|
||||
- 电视窗口可拖拽、可关闭
|
||||
- 央视 `CCTV-4` 作为默认兜底源可被使用
|
||||
- 代码结构已为后续采集器接入预留统一接口
|
||||
165
docs/deprecated/hud-panel-component-plan.md
Normal file
165
docs/deprecated/hud-panel-component-plan.md
Normal file
@@ -0,0 +1,165 @@
|
||||
# HUD Panel Component Plan
|
||||
|
||||
## Goal
|
||||
|
||||
Unify Earth HUD panels into a reusable component layer so new panels can share:
|
||||
|
||||
- a consistent shell
|
||||
- a consistent header
|
||||
- a consistent action-button system
|
||||
- a consistent body and collapse pattern
|
||||
|
||||
## Scope
|
||||
|
||||
Target panels:
|
||||
|
||||
- `tv-panel`
|
||||
- `news-panel`
|
||||
- `legend`
|
||||
- `layer-panel`
|
||||
- `earth-stats`
|
||||
- `info-card`
|
||||
- settings modal header/actions
|
||||
|
||||
## Component Model
|
||||
|
||||
### Base shell
|
||||
|
||||
- `.hud-panel`
|
||||
- `.hud-panel--compact`
|
||||
- `.hud-panel--media`
|
||||
- `.hud-panel--collapsed`
|
||||
- `.hud-panel-hidden`
|
||||
- `.hud-panel.is-dragging`
|
||||
- `.hud-panel.is-layout-animating`
|
||||
|
||||
### Header
|
||||
|
||||
- `.hud-panel__header`
|
||||
- `.hud-panel__title-group`
|
||||
- `.hud-panel__title`
|
||||
- `.hud-panel__subtitle`
|
||||
- `.hud-panel__chip`
|
||||
- `.hud-panel__actions`
|
||||
|
||||
Header baseline rule:
|
||||
|
||||
- Header title styling is fixed by the component layer and should not drift per panel
|
||||
- Title font size, font weight, letter spacing, line height, text color, and vertical alignment come from the shared header tokens and structure
|
||||
- Header divider, border treatment, inner spacing, and title-to-actions alignment are part of the same shared baseline
|
||||
- Panel-specific header differences should be limited to explicit variants such as `compact` or `media`, or token overrides with documented intent
|
||||
- “Looks close enough” local header overrides should be treated as temporary compatibility code and removed during migration
|
||||
|
||||
### Actions
|
||||
|
||||
- `.hud-panel__action`
|
||||
- `.hud-panel__action--icon`
|
||||
- `.hud-panel__action--collapse`
|
||||
- `.hud-panel__action--close`
|
||||
- `.hud-panel__action--refresh`
|
||||
- `.hud-panel__action--external`
|
||||
|
||||
Action-button baseline rule:
|
||||
|
||||
- Header action buttons must have one fixed default style baseline across all HUD panels
|
||||
- Default width behavior, padding, icon size, radius, alignment, hover, and active feedback all come from `.hud-panel__action`
|
||||
- Panel-specific differences must be expressed through explicit variants or token overrides, not ad-hoc local button rewrites
|
||||
- `close` buttons are part of the same default action system and must not silently fall back to a separate legacy box model
|
||||
|
||||
### Body
|
||||
|
||||
- `.hud-panel__body`
|
||||
- `.hud-panel__body--scroll`
|
||||
- `.hud-panel__body--collapsible`
|
||||
|
||||
### Collapse behavior
|
||||
|
||||
- `.hud-panel--collapsed`
|
||||
- `.hud-panel--expand-up`
|
||||
- `.hud-panel--expand-down`
|
||||
|
||||
Adaptive collapse / expand rule:
|
||||
|
||||
- HUD panels support two expansion directions:
|
||||
- top-to-bottom expansion
|
||||
- bottom-to-top expansion
|
||||
- Expansion direction should be decided at runtime from available viewport space rather than hardcoded per panel
|
||||
- Use:
|
||||
- `d` = available distance from the header anchor to the viewport bottom edge
|
||||
- `h` = expected expanded panel height
|
||||
- buffer = `20px`
|
||||
- Collapsed-state direction rule:
|
||||
- if `d > h + 20px`, the next action direction is `expand-up`
|
||||
- if `d <= h + 20px`, the next action direction is `expand-down`
|
||||
- To avoid jitter around the threshold, the shared controller should keep a small hysteresis band:
|
||||
- if the current direction is already `up`, keep it until `d <= h`
|
||||
- if the current direction is already `down`, keep it until `d > h + 20px`
|
||||
- The opposite edge is still a safety guard:
|
||||
- if the chosen side cannot fit at all, fall back to the other side if it can fit
|
||||
- if neither side fully fits, choose the side with more space and let the body scroll
|
||||
- If neither direction fully fits, choose the direction with more available space and let the body scroll
|
||||
- Collapse icon direction must match the active expansion direction so the icon always describes the real open/close motion
|
||||
- The collapse icon describes the next action, not the current state
|
||||
- This mapping is fixed component behavior and must not drift per panel:
|
||||
- collapsed + expand-down => `expand_more`
|
||||
- expanded + expand-down => `expand_less`
|
||||
- collapsed + expand-up => `expand_less`
|
||||
- expanded + expand-up => `expand_more`
|
||||
- Panels must not combine icon-name swapping with extra CSS rotation for the same collapse control
|
||||
- Expansion direction and icon direction must come from one shared source of truth in the component controller
|
||||
- The direction decision should be recomputed when opening, resizing the viewport, or restoring a dragged panel near another edge
|
||||
|
||||
## Tokens
|
||||
|
||||
Promote panel differences into CSS variables instead of duplicating selectors:
|
||||
|
||||
- `--hud-panel-padding`
|
||||
- `--hud-header-padding`
|
||||
- `--hud-header-gap`
|
||||
- `--hud-action-padding`
|
||||
- `--hud-action-gap`
|
||||
- `--hud-action-icon-size`
|
||||
- `--hud-body-gap`
|
||||
- `--hud-body-max-height`
|
||||
- `--hud-chip-radius`
|
||||
- `--hud-title-font-size`
|
||||
- `--hud-title-font-weight`
|
||||
- `--hud-title-letter-spacing`
|
||||
- `--hud-title-line-height`
|
||||
- `--hud-title-color`
|
||||
- `--hud-header-border-color`
|
||||
- `--hud-header-divider-opacity`
|
||||
- `--hud-expand-direction`
|
||||
|
||||
## Migration Order
|
||||
|
||||
1. Build the shared component layer in `frontend/public/earth/css/hud.css`
|
||||
2. Migrate `tv-panel` and `news-panel` first as the reference implementation
|
||||
3. Migrate `legend` and `layer-panel` into a compact variant
|
||||
4. Migrate `earth-stats` and `info-card`
|
||||
5. Align settings modal header/actions with the same action system
|
||||
6. Remove legacy one-off button selectors after verification
|
||||
|
||||
## Guardrails
|
||||
|
||||
- Do not change panel behavior and data flow during the first pass
|
||||
- Keep old class names temporarily as compatibility hooks
|
||||
- Prefer variable overrides over per-panel reimplementation
|
||||
- Treat header action-button default styling as fixed component API, not per-panel design space
|
||||
- Treat header title typography, border, and divider styling as fixed component API, not per-panel design space
|
||||
- Treat collapse direction as a component behavior contract, not a one-off panel trick
|
||||
- Treat collapse icon semantics as a component behavior contract, not a per-panel visual preference
|
||||
- Verify header alignment and drag/collapse behavior after each migration batch
|
||||
|
||||
## First Implementation Batch
|
||||
|
||||
Batch 1 should only do:
|
||||
|
||||
- shared header structure
|
||||
- shared action-button system
|
||||
- shared title typography and header border/divider baseline
|
||||
- shared collapsible body pattern
|
||||
- adaptive collapse direction logic and direction-aware collapse icons
|
||||
- migration of `tv-panel` and `news-panel`
|
||||
|
||||
That keeps risk low while giving the rest of the HUD a stable target to migrate toward.
|
||||
@@ -1,3 +1,5 @@
|
||||
> Archived note: this document was originally created by sisyphus and later reviewed against the main docs set. Useful content has been absorbed into `docs/plans/` where appropriate.
|
||||
|
||||
# 地球3D可视化架构重构计划
|
||||
|
||||
## 背景
|
||||
@@ -1,3 +1,5 @@
|
||||
> Archived note: this document was originally created by sisyphus and later reviewed against the main docs set. Useful content has been absorbed into `docs/plans/` where appropriate.
|
||||
|
||||
# 卫星预测轨道显示功能
|
||||
|
||||
## TL;DR
|
||||
@@ -1,3 +1,5 @@
|
||||
> Archived note: this document was originally created by sisyphus and later reviewed against the main docs set. Useful content has been absorbed into `docs/plans/` where appropriate.
|
||||
|
||||
# UE5 3D 大屏客户端开发计划
|
||||
|
||||
## 项目概述
|
||||
@@ -1,3 +1,5 @@
|
||||
> Archived note: this document was originally created by sisyphus and later reviewed against the main docs set. Useful content has been absorbed into `docs/plans/` where appropriate.
|
||||
|
||||
# WebGL Instancing 卫星渲染优化计划
|
||||
|
||||
## 背景
|
||||
34
docs/plans/README.md
Normal file
34
docs/plans/README.md
Normal file
@@ -0,0 +1,34 @@
|
||||
# Plans Docs
|
||||
|
||||
这里放“未来实施方案和未完成计划”的文档,重点回答:
|
||||
|
||||
- 我们准备做什么
|
||||
- 为什么要做
|
||||
- 分几期做
|
||||
- 当前差距和下一步是什么
|
||||
|
||||
适合放入这里的内容:
|
||||
|
||||
- Earth / BGP / 地形 / 天球实施方案
|
||||
- AI Playground 发展计划
|
||||
- backend / datasource / agent roadmap
|
||||
- UE5 MVP 方案
|
||||
|
||||
当前重点入口:
|
||||
|
||||
- [earth-renderer-architecture-separation-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-renderer-architecture-separation-plan.md)
|
||||
- [earth-predicted-orbit-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-predicted-orbit-plan.md)
|
||||
- [earth-webgl-instancing-satellites-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-webgl-instancing-satellites-plan.md)
|
||||
- [earth-real-terrain-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-real-terrain-plan.md)
|
||||
- [frontend-ai-playground-development-plan.md](/home/ray/dev/linkong/planet/docs/plans/frontend-ai-playground-development-plan.md)
|
||||
- [ue5-mvp-fused-plan.md](/home/ray/dev/linkong/planet/docs/plans/ue5-mvp-fused-plan.md)
|
||||
|
||||
不适合放入这里的内容:
|
||||
|
||||
- 当前代码结构说明
|
||||
- 组件现状和实现入口
|
||||
- 已经落地的技术上下文说明
|
||||
|
||||
这些应放入:
|
||||
|
||||
- [docs/technical/README.md](/home/ray/dev/linkong/planet/docs/technical/README.md)
|
||||
647
docs/plans/agents-agent-architecture-plan.md
Normal file
647
docs/plans/agents-agent-architecture-plan.md
Normal file
@@ -0,0 +1,647 @@
|
||||
# Agent Architecture Plan
|
||||
|
||||
## Overview
|
||||
|
||||
This document defines the agent architecture for Planet.
|
||||
|
||||
The architecture is intentionally broader than datasource health checking.
|
||||
|
||||
It is designed to support both:
|
||||
|
||||
- datasource health governance
|
||||
- future situational-awareness workflows
|
||||
|
||||
The core idea is to avoid building a one-off "repair broken API links" agent.
|
||||
|
||||
Instead, Planet should grow a reusable agent runtime that can:
|
||||
|
||||
- collect evidence
|
||||
- evaluate signals
|
||||
- reason over incomplete information
|
||||
- generate proposals
|
||||
- produce assessments
|
||||
- execute limited actions under policy
|
||||
|
||||
|
||||
## Design Goal
|
||||
|
||||
Build an agent foundation that can evolve in this order:
|
||||
|
||||
1. datasource health checks
|
||||
2. datasource repair proposals
|
||||
3. signal correlation
|
||||
4. situational assessments
|
||||
5. controlled runtime actions
|
||||
|
||||
This means the architecture should treat datasource health as one use case of the larger agent system, not as the whole system.
|
||||
|
||||
|
||||
## Core Principles
|
||||
|
||||
1. Separate evidence from reasoning
|
||||
|
||||
- raw signals should be gathered first
|
||||
- deterministic checks should run before LLM reasoning
|
||||
|
||||
2. Agents do not own the defaults
|
||||
|
||||
- repository defaults remain human-owned
|
||||
- agents operate on runtime state, proposals, and overrides
|
||||
|
||||
3. Reasoning and action are different responsibilities
|
||||
|
||||
- many agents should be read-only or propose-only
|
||||
- only tightly controlled flows may apply changes
|
||||
|
||||
4. Shared runtime, specialized roles
|
||||
|
||||
- multiple agent roles should share the same object model and orchestration patterns
|
||||
- health and situational-awareness agents should not invent incompatible payloads
|
||||
|
||||
5. Auditability is mandatory
|
||||
|
||||
- every proposal, assessment, and applied action should be attributable
|
||||
|
||||
|
||||
## System Layers
|
||||
|
||||
Planet agent architecture should be split into four layers.
|
||||
|
||||
### 1. Signal Layer
|
||||
|
||||
Purpose:
|
||||
|
||||
- gather raw evidence from internal and external systems
|
||||
|
||||
Example sources:
|
||||
|
||||
- collector outputs
|
||||
- datasource health checks
|
||||
- logs
|
||||
- snapshots
|
||||
- alerts
|
||||
- web search results
|
||||
- scraped pages
|
||||
- external APIs
|
||||
- operator inputs
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- fetch
|
||||
- normalize
|
||||
- timestamp
|
||||
- tag with source and trust level
|
||||
|
||||
This layer should not make high-level judgments.
|
||||
|
||||
|
||||
### 2. Evaluation Layer
|
||||
|
||||
Purpose:
|
||||
|
||||
- perform deterministic analysis
|
||||
|
||||
Examples:
|
||||
|
||||
- reachability checks
|
||||
- schema validation
|
||||
- threshold checks
|
||||
- time-window comparisons
|
||||
- anomaly counters
|
||||
- completeness checks
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- classify signals into machine-readable findings
|
||||
- attach deterministic evidence
|
||||
|
||||
This layer should avoid LLM dependency whenever possible.
|
||||
|
||||
|
||||
### 3. Reasoning Layer
|
||||
|
||||
Purpose:
|
||||
|
||||
- use LLMs when semantic interpretation or incomplete-information reasoning is needed
|
||||
|
||||
Examples:
|
||||
|
||||
- endpoint migration inference
|
||||
- multi-source event correlation
|
||||
- causality hypotheses
|
||||
- ambiguity reduction
|
||||
- assessment narrative generation
|
||||
- action recommendation generation
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- synthesize evidence
|
||||
- produce hypotheses
|
||||
- rank confidence
|
||||
- explain reasoning boundaries
|
||||
|
||||
This is the main place where `aiprovider` and web search are used.
|
||||
|
||||
|
||||
### 4. Action Layer
|
||||
|
||||
Purpose:
|
||||
|
||||
- convert proposals or assessments into controlled system actions
|
||||
|
||||
Examples:
|
||||
|
||||
- create runtime override
|
||||
- create proposal
|
||||
- publish alert
|
||||
- update operator task queue
|
||||
- generate summary artifact
|
||||
- trigger follow-up verification
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- enforce policy
|
||||
- enforce approval requirements
|
||||
- verify post-action outcomes
|
||||
- record audit trails
|
||||
|
||||
|
||||
## Architecture Sketch
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A["Collectors / Logs / Snapshots / External APIs"] --> B["Signal Layer"]
|
||||
W["Web Search / Page Fetch / Docs"] --> B
|
||||
B --> C["Evaluation Layer"]
|
||||
C --> D["Findings"]
|
||||
D --> E["Reasoning Layer (LLM + Tools)"]
|
||||
E --> F["Proposals"]
|
||||
E --> G["Assessments"]
|
||||
F --> H["Action Layer"]
|
||||
H --> I["Runtime Overrides / Alerts / Tasks"]
|
||||
H --> J["Verification Loop"]
|
||||
J --> B
|
||||
|
||||
K["Policy Engine"] --> H
|
||||
L["Audit / History Store"] --> H
|
||||
L --> E
|
||||
L --> C
|
||||
```
|
||||
|
||||
|
||||
## Agent Roles
|
||||
|
||||
The first version should define these logical roles.
|
||||
|
||||
### 1. Health Agent
|
||||
|
||||
Primary use case:
|
||||
|
||||
- datasource health governance
|
||||
|
||||
Inputs:
|
||||
|
||||
- datasource metadata
|
||||
- current endpoint
|
||||
- latest health records
|
||||
- latest failures
|
||||
- deterministic findings
|
||||
|
||||
Outputs:
|
||||
|
||||
- health interpretation
|
||||
- repair proposal
|
||||
- confidence
|
||||
- evidence references
|
||||
|
||||
Typical action level:
|
||||
|
||||
- propose-only
|
||||
|
||||
|
||||
### 2. Correlation Agent
|
||||
|
||||
Primary use case:
|
||||
|
||||
- identify whether multiple signals describe the same event or related events
|
||||
|
||||
Inputs:
|
||||
|
||||
- findings from multiple collectors
|
||||
- time windows
|
||||
- region / ASN / prefix / cable relationships
|
||||
- prior incidents
|
||||
|
||||
Outputs:
|
||||
|
||||
- grouped event candidates
|
||||
- correlation rationale
|
||||
- confidence per relationship
|
||||
|
||||
Typical action level:
|
||||
|
||||
- read-only
|
||||
|
||||
|
||||
### 3. Assessment Agent
|
||||
|
||||
Primary use case:
|
||||
|
||||
- produce situational-awareness outputs
|
||||
|
||||
Inputs:
|
||||
|
||||
- grouped events
|
||||
- findings
|
||||
- current context
|
||||
- historical context
|
||||
- operator constraints
|
||||
|
||||
Outputs:
|
||||
|
||||
- structured assessment
|
||||
- risk summary
|
||||
- evidence-backed recommendations
|
||||
- missing-information list
|
||||
|
||||
Typical action level:
|
||||
|
||||
- read-only or propose-only
|
||||
|
||||
|
||||
### 4. Recovery Agent
|
||||
|
||||
Primary use case:
|
||||
|
||||
- carry low-risk proposals into controlled runtime actions
|
||||
|
||||
Inputs:
|
||||
|
||||
- approved proposal
|
||||
- policy constraints
|
||||
- trusted-domain rules
|
||||
- verification checks
|
||||
|
||||
Outputs:
|
||||
|
||||
- applied override
|
||||
- failed application
|
||||
- rollback request
|
||||
|
||||
Typical action level:
|
||||
|
||||
- apply-limited
|
||||
|
||||
|
||||
## Shared Object Model
|
||||
|
||||
All agents should work on a shared object model.
|
||||
|
||||
That prevents the health subsystem and situational-awareness subsystem from drifting into incompatible payloads.
|
||||
|
||||
### Signal
|
||||
|
||||
Represents a raw observed fact.
|
||||
|
||||
Examples:
|
||||
|
||||
- a datasource returned HTTP 404
|
||||
- a collector returned empty results
|
||||
- BGP updates spiked in one region
|
||||
- a known endpoint now redirects elsewhere
|
||||
|
||||
Suggested shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "sig_123",
|
||||
"type": "datasource.http_failure",
|
||||
"source": "ris_live_bgp",
|
||||
"occurred_at": "2026-04-08T10:00:00Z",
|
||||
"severity": "medium",
|
||||
"payload": {},
|
||||
"trust": 0.95
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
### Finding
|
||||
|
||||
Represents a deterministic or semi-deterministic interpretation of one or more signals.
|
||||
|
||||
Examples:
|
||||
|
||||
- `schema_changed`
|
||||
- `endpoint_unreachable`
|
||||
- `data_volume_abnormally_low`
|
||||
- `event_cluster_detected`
|
||||
|
||||
Suggested shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "find_123",
|
||||
"type": "datasource.schema_changed",
|
||||
"source_ids": ["sig_123"],
|
||||
"confidence": 0.92,
|
||||
"evidence": [],
|
||||
"details": {}
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
### Proposal
|
||||
|
||||
Represents a recommended action, not an already-applied action.
|
||||
|
||||
Examples:
|
||||
|
||||
- switch endpoint to new URL
|
||||
- disable bad override
|
||||
- escalate issue for manual review
|
||||
|
||||
Suggested shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "prop_123",
|
||||
"kind": "endpoint_override",
|
||||
"target": "telegeography_cables",
|
||||
"confidence": 0.84,
|
||||
"reason": "Official docs now point to a new API path",
|
||||
"payload": {},
|
||||
"evidence_urls": [],
|
||||
"status": "proposed"
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
### Assessment
|
||||
|
||||
Represents a structured situational-awareness output for operators or downstream systems.
|
||||
|
||||
Examples:
|
||||
|
||||
- current network posture summary
|
||||
- incident impact assessment
|
||||
- risk and response recommendations
|
||||
|
||||
Suggested shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "assess_123",
|
||||
"scope": "regional-network",
|
||||
"risk_level": "high",
|
||||
"summary": "Regional routing instability is increasing.",
|
||||
"key_risks": [],
|
||||
"evidence": [],
|
||||
"recommendations": [],
|
||||
"missing_data": []
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
## State Machine
|
||||
|
||||
The shared orchestration flow should look like this:
|
||||
|
||||
```mermaid
|
||||
stateDiagram-v2
|
||||
[*] --> Collect
|
||||
Collect --> Validate
|
||||
Validate --> Classify
|
||||
Classify --> Reason
|
||||
Reason --> Propose
|
||||
Reason --> Assess
|
||||
Propose --> Review
|
||||
Review --> Apply
|
||||
Apply --> Verify
|
||||
Verify --> Archive
|
||||
Assess --> Archive
|
||||
Archive --> [*]
|
||||
```
|
||||
|
||||
Definitions:
|
||||
|
||||
- `Collect`: gather signals
|
||||
- `Validate`: run deterministic checks
|
||||
- `Classify`: create findings
|
||||
- `Reason`: invoke LLM reasoning when needed
|
||||
- `Propose`: create change proposals
|
||||
- `Review`: policy or human approval
|
||||
- `Apply`: perform limited runtime action
|
||||
- `Verify`: confirm action effect
|
||||
- `Archive`: store artifacts and decisions
|
||||
|
||||
|
||||
## Permission Model
|
||||
|
||||
Each agent role should be assigned one of these action levels.
|
||||
|
||||
### `read-only`
|
||||
|
||||
Allowed:
|
||||
|
||||
- read signals
|
||||
- search web
|
||||
- fetch pages
|
||||
- read internal state
|
||||
- generate findings and assessments
|
||||
|
||||
Not allowed:
|
||||
|
||||
- mutate config
|
||||
- write overrides
|
||||
- change live runtime behavior
|
||||
|
||||
|
||||
### `propose-only`
|
||||
|
||||
Allowed:
|
||||
|
||||
- everything in `read-only`
|
||||
- create proposals
|
||||
- create review tasks
|
||||
|
||||
Not allowed:
|
||||
|
||||
- apply live changes
|
||||
|
||||
|
||||
### `apply-limited`
|
||||
|
||||
Allowed:
|
||||
|
||||
- everything in `propose-only`
|
||||
- write approved runtime overrides
|
||||
- trigger verification checks
|
||||
|
||||
Not allowed:
|
||||
|
||||
- mutate repository defaults
|
||||
- make destructive data changes
|
||||
- bypass policy engine
|
||||
|
||||
|
||||
## Runtime Components
|
||||
|
||||
The first durable architecture should introduce these components.
|
||||
|
||||
### 1. Signal Store
|
||||
|
||||
Stores normalized evidence and health outputs.
|
||||
|
||||
|
||||
### 2. Finding Store
|
||||
|
||||
Stores deterministic classifications that can be reused by multiple agents.
|
||||
|
||||
|
||||
### 3. Proposal Store
|
||||
|
||||
Stores recommended actions with evidence and confidence.
|
||||
|
||||
|
||||
### 4. Assessment Store
|
||||
|
||||
Stores structured situational-awareness outputs.
|
||||
|
||||
|
||||
### 5. Policy Engine
|
||||
|
||||
Decides:
|
||||
|
||||
- whether agent may run
|
||||
- whether proposal requires review
|
||||
- whether proposal may auto-apply
|
||||
- whether post-apply verification passed
|
||||
|
||||
|
||||
### 6. Override Store
|
||||
|
||||
Stores runtime-only configuration changes.
|
||||
|
||||
This is where endpoint repairs should live.
|
||||
|
||||
|
||||
## Relation To `aiprovider`
|
||||
|
||||
`aiprovider` should remain the model gateway.
|
||||
|
||||
It should not become the full agent runtime.
|
||||
|
||||
Recommended split:
|
||||
|
||||
- `aiprovider`
|
||||
- provider adaptation
|
||||
- prompt transport
|
||||
- model execution
|
||||
- protocol compatibility
|
||||
|
||||
- agent runtime
|
||||
- orchestration
|
||||
- signal handling
|
||||
- tool selection
|
||||
- proposal generation
|
||||
- policy and audit
|
||||
|
||||
This keeps provider concerns and agent behavior concerns separate.
|
||||
|
||||
|
||||
## Relation To Datasource Health
|
||||
|
||||
Datasource health becomes one vertical slice of this architecture.
|
||||
|
||||
Mapping:
|
||||
|
||||
- signal:
|
||||
- endpoint unreachable
|
||||
- schema mismatch
|
||||
- bad content type
|
||||
- finding:
|
||||
- `failed`
|
||||
- `schema_changed`
|
||||
- `moved_endpoint_suspected`
|
||||
- proposal:
|
||||
- runtime override suggestion
|
||||
- assessment:
|
||||
- datasource health summary for operators
|
||||
|
||||
|
||||
## Relation To Situational Awareness
|
||||
|
||||
Future situational-awareness capabilities should reuse the same flow:
|
||||
|
||||
- raw telemetry becomes signals
|
||||
- anomaly detection becomes findings
|
||||
- LLM correlation becomes reasoning
|
||||
- operator-facing output becomes assessments
|
||||
- policy-approved mitigations become actions
|
||||
|
||||
This lets the platform evolve from operational health governance into broader cyber/network posture workflows without changing the architecture.
|
||||
|
||||
|
||||
## Suggested Delivery Sequence
|
||||
|
||||
### Phase A
|
||||
|
||||
- finalize shared object model
|
||||
- implement health-oriented signal and finding storage
|
||||
|
||||
### Phase B
|
||||
|
||||
- implement Health Agent
|
||||
- generate proposals only
|
||||
|
||||
### Phase C
|
||||
|
||||
- implement Assessment Agent
|
||||
- expose structured assessments via API
|
||||
|
||||
### Phase D
|
||||
|
||||
- implement Correlation Agent
|
||||
- support multi-source incident grouping
|
||||
|
||||
### Phase E
|
||||
|
||||
- implement Recovery Agent with policy-gated runtime actions
|
||||
|
||||
|
||||
## Recommended First Build
|
||||
|
||||
The first build should not try to implement every agent role.
|
||||
|
||||
Recommended initial slice:
|
||||
|
||||
- shared object model
|
||||
- health signals
|
||||
- health findings
|
||||
- Health Agent
|
||||
- proposal generation only
|
||||
|
||||
This gives immediate value while preserving the longer-term architecture.
|
||||
|
||||
|
||||
## Non-Goals For The First Iteration
|
||||
|
||||
- repository YAML auto-rewrites
|
||||
- unrestricted autonomous action
|
||||
- full incident graph reasoning
|
||||
- automatic large-scale remediation
|
||||
- agent-owned configuration source of truth
|
||||
|
||||
|
||||
## Summary
|
||||
|
||||
Planet should treat agents as a reusable runtime for evidence, reasoning, proposals, and assessments.
|
||||
|
||||
The datasource health use case is the first practical entrypoint, but the architecture should already assume future situational-awareness expansion.
|
||||
|
||||
The safest path is:
|
||||
|
||||
- deterministic checks first
|
||||
- agent reasoning second
|
||||
- proposals before actions
|
||||
- runtime overrides instead of default mutation
|
||||
346
docs/plans/agents-agent-runtime-roadmap.md
Normal file
346
docs/plans/agents-agent-runtime-roadmap.md
Normal file
@@ -0,0 +1,346 @@
|
||||
# Agent Runtime Roadmap
|
||||
|
||||
## Overview
|
||||
|
||||
This document connects three existing planning threads into one implementation roadmap:
|
||||
|
||||
- `aiprovider` as the model gateway
|
||||
- datasource health governance as the first practical agent use case
|
||||
- situational awareness as the broader long-term target
|
||||
|
||||
Related documents:
|
||||
|
||||
- [aiprovider](/home/ray/dev/linkong/planet/docs/technical/agents-aiprovider.md)
|
||||
- [datasource-health-plan](/home/ray/dev/linkong/planet/docs/plans/agents-datasource-health-plan.md)
|
||||
- [agent-architecture-plan](/home/ray/dev/linkong/planet/docs/plans/agents-agent-architecture-plan.md)
|
||||
|
||||
|
||||
## Big Picture
|
||||
|
||||
Planet should evolve in layers:
|
||||
|
||||
1. stable model gateway
|
||||
2. deterministic health and evidence collection
|
||||
3. agent runtime for reasoning and proposal generation
|
||||
4. situational-awareness assessments and controlled actions
|
||||
|
||||
This prevents the system from collapsing into a single giant "AI feature" with unclear boundaries.
|
||||
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
U["Frontend / Backend APIs / Operators"] --> B["Planet Backend"]
|
||||
B --> H["Datasource Health Services"]
|
||||
B --> R["Agent Runtime"]
|
||||
R --> P["aiprovider"]
|
||||
P --> M["OpenAI / Anthropic / MiniMax / Ollama / Local Models"]
|
||||
|
||||
C["Collectors / Snapshots / Logs / Alerts / BGP Signals"] --> S["Signal Store"]
|
||||
H --> S
|
||||
S --> E["Evaluation Layer"]
|
||||
E --> F["Findings"]
|
||||
F --> R
|
||||
|
||||
W["Web Search / Page Fetch / Docs Fetch"] --> R
|
||||
R --> PR["Proposals"]
|
||||
R --> AS["Assessments"]
|
||||
|
||||
PR --> O["Runtime Overrides / Review Queue / Tasks"]
|
||||
AS --> SA["Situational Awareness APIs / UI"]
|
||||
|
||||
O --> V["Verification Loop"]
|
||||
V --> S
|
||||
```
|
||||
|
||||
|
||||
## Role Boundaries
|
||||
|
||||
### `aiprovider`
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- provider compatibility
|
||||
- protocol adaptation
|
||||
- auth and model transport
|
||||
- request/response normalization
|
||||
|
||||
Not responsible for:
|
||||
|
||||
- agent orchestration
|
||||
- business workflows
|
||||
- datasource repair policy
|
||||
- situational-awareness domain logic
|
||||
|
||||
|
||||
### Backend
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- stable business APIs
|
||||
- auth and permissions
|
||||
- task orchestration
|
||||
- health records
|
||||
- proposal and override persistence
|
||||
- assessment exposure
|
||||
|
||||
|
||||
### Agent Runtime
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- consume findings and context
|
||||
- invoke LLMs via `aiprovider`
|
||||
- invoke tools such as web search
|
||||
- create proposals
|
||||
- create assessments
|
||||
- route to policy-controlled action paths
|
||||
|
||||
|
||||
## Delivery Sequence
|
||||
|
||||
## Stage 1: Gateway Foundation
|
||||
|
||||
Status:
|
||||
|
||||
- already in place
|
||||
|
||||
Delivered by current work:
|
||||
|
||||
- `aiprovider`
|
||||
- multi-provider compatibility
|
||||
- backend AI facade
|
||||
- MiniMax / Anthropic-compatible support
|
||||
- request-id propagation
|
||||
|
||||
Primary outcome:
|
||||
|
||||
- the system already has a stable way to call models
|
||||
|
||||
|
||||
## Stage 2: Datasource Health MVP
|
||||
|
||||
Goal:
|
||||
|
||||
- establish deterministic health observability
|
||||
|
||||
Key work:
|
||||
|
||||
- health check task runner
|
||||
- health result table
|
||||
- datasource health APIs
|
||||
- UI visibility
|
||||
- collector endpoint override precedence cleanup
|
||||
|
||||
Primary outcome:
|
||||
|
||||
- Planet knows which collectors are healthy before asking an LLM anything
|
||||
|
||||
|
||||
## Stage 3: Health Agent
|
||||
|
||||
Goal:
|
||||
|
||||
- let the first agent role operate on health failures
|
||||
|
||||
Key work:
|
||||
|
||||
- convert health failures into signals/findings
|
||||
- invoke agent only for failed or suspicious cases
|
||||
- produce repair proposals with evidence and confidence
|
||||
|
||||
Primary outcome:
|
||||
|
||||
- Planet can suggest endpoint repairs without mutating defaults
|
||||
|
||||
|
||||
## Stage 4: Runtime Repair Application
|
||||
|
||||
Goal:
|
||||
|
||||
- safely apply approved datasource repair proposals
|
||||
|
||||
Key work:
|
||||
|
||||
- override storage
|
||||
- policy-gated apply flow
|
||||
- verification after apply
|
||||
- rollback path
|
||||
|
||||
Primary outcome:
|
||||
|
||||
- datasource repair becomes operationally useful without polluting repository defaults
|
||||
|
||||
|
||||
## Stage 5: Situational Awareness Assessments
|
||||
|
||||
Goal:
|
||||
|
||||
- reuse the same runtime for broader operator-facing assessment
|
||||
|
||||
Key work:
|
||||
|
||||
- normalize telemetry and incident evidence into signals/findings
|
||||
- build Assessment Agent
|
||||
- expose structured assessments through backend APIs and UI
|
||||
|
||||
Primary outcome:
|
||||
|
||||
- LLM output becomes evidence-backed situational summary, not just ad hoc chat output
|
||||
|
||||
|
||||
## Stage 6: Correlation and Controlled Actions
|
||||
|
||||
Goal:
|
||||
|
||||
- connect multiple sources into higher-level posture and event groupings
|
||||
|
||||
Key work:
|
||||
|
||||
- event correlation
|
||||
- incident grouping
|
||||
- recommendation scoring
|
||||
- controlled action routing
|
||||
|
||||
Primary outcome:
|
||||
|
||||
- Planet becomes a true agent-assisted situational-awareness system
|
||||
|
||||
|
||||
## Implementation Tracks
|
||||
|
||||
These tracks can progress in parallel, but they should stay loosely coupled.
|
||||
|
||||
### Track A: Config and Runtime Resolution
|
||||
|
||||
Scope:
|
||||
|
||||
- datasource defaults
|
||||
- overrides
|
||||
- runtime precedence
|
||||
- audit trails
|
||||
|
||||
First milestone:
|
||||
|
||||
- health-safe override layer
|
||||
|
||||
|
||||
### Track B: Health and Evidence
|
||||
|
||||
Scope:
|
||||
|
||||
- deterministic checks
|
||||
- failure categorization
|
||||
- signal and finding persistence
|
||||
|
||||
First milestone:
|
||||
|
||||
- datasource health record system
|
||||
|
||||
|
||||
### Track C: Agent Runtime
|
||||
|
||||
Scope:
|
||||
|
||||
- shared object model
|
||||
- orchestration flow
|
||||
- prompt/tool pipeline
|
||||
- policy integration
|
||||
|
||||
First milestone:
|
||||
|
||||
- Health Agent proposal pipeline
|
||||
|
||||
|
||||
### Track D: Situational Awareness
|
||||
|
||||
Scope:
|
||||
|
||||
- assessment schema
|
||||
- multi-source context assembly
|
||||
- operator-facing outputs
|
||||
|
||||
First milestone:
|
||||
|
||||
- structured assessment API
|
||||
|
||||
|
||||
## Shared Artifacts
|
||||
|
||||
To avoid fragmentation, these artifacts should be shared across all future agent work.
|
||||
|
||||
### Shared object model
|
||||
|
||||
- `Signal`
|
||||
- `Finding`
|
||||
- `Proposal`
|
||||
- `Assessment`
|
||||
|
||||
### Shared orchestration flow
|
||||
|
||||
- collect
|
||||
- validate
|
||||
- classify
|
||||
- reason
|
||||
- propose or assess
|
||||
- review or apply
|
||||
- verify
|
||||
- archive
|
||||
|
||||
### Shared policy model
|
||||
|
||||
- read-only
|
||||
- propose-only
|
||||
- apply-limited
|
||||
|
||||
|
||||
## Recommended Next Concrete Steps
|
||||
|
||||
1. Build Stage 2 first
|
||||
|
||||
- datasource health records
|
||||
- deterministic checks
|
||||
- no automatic repair
|
||||
|
||||
2. Then build Stage 3
|
||||
|
||||
- Health Agent
|
||||
- proposal generation only
|
||||
|
||||
3. Then Stage 4
|
||||
|
||||
- override apply flow
|
||||
- rollback and verification
|
||||
|
||||
4. Only after that start Stage 5
|
||||
|
||||
- broader situational-awareness assessment workflows
|
||||
|
||||
|
||||
## Why This Order
|
||||
|
||||
Because situational-awareness quality depends on reliable upstream data.
|
||||
|
||||
If datasource health is weak:
|
||||
|
||||
- agent reasoning quality will degrade
|
||||
- false explanations will increase
|
||||
- assessment trust will drop
|
||||
|
||||
So datasource health is not a side task.
|
||||
|
||||
It is the first operational foundation for the later situational-awareness system.
|
||||
|
||||
|
||||
## Summary
|
||||
|
||||
Planet should be built as:
|
||||
|
||||
- `aiprovider` for model access
|
||||
- backend services for orchestration and persistence
|
||||
- datasource health as the first evidence-governance layer
|
||||
- agent runtime as the reusable reasoning core
|
||||
- situational awareness as the long-term application layer
|
||||
|
||||
That path keeps the architecture coherent and lets each phase produce useful functionality without forcing a rewrite later.
|
||||
486
docs/plans/agents-datasource-health-plan.md
Normal file
486
docs/plans/agents-datasource-health-plan.md
Normal file
@@ -0,0 +1,486 @@
|
||||
# Datasource Health Plan
|
||||
|
||||
## Overview
|
||||
|
||||
This document defines a phased plan for datasource health governance.
|
||||
|
||||
The goal is to make collectors observable, diagnosable, and recoverable when upstream APIs change, while avoiding unsafe automatic mutation of repository defaults.
|
||||
|
||||
The key principle is:
|
||||
|
||||
- do not let runtime automation rewrite repository default config
|
||||
|
||||
Instead, split responsibilities across:
|
||||
|
||||
- default config
|
||||
- runtime overrides
|
||||
- health check records
|
||||
- agent-generated repair proposals
|
||||
|
||||
|
||||
## Problem Statement
|
||||
|
||||
Collectors currently depend on third-party APIs, data downloads, mirrored JSON files, archive links, and web pages.
|
||||
|
||||
These upstream dependencies can fail in several ways:
|
||||
|
||||
- endpoint becomes unreachable
|
||||
- endpoint still responds but schema changes
|
||||
- content-type changes
|
||||
- website shuts down or moves
|
||||
- mirror link disappears
|
||||
- HTML structure changes and scraping fails
|
||||
- endpoint requires a new path or new host
|
||||
|
||||
We want a system that can:
|
||||
|
||||
- detect datasource health degradation early
|
||||
- identify likely cause
|
||||
- search for updated endpoints when reasonable
|
||||
- apply safe runtime fixes without polluting default repo config
|
||||
- preserve auditability and rollback
|
||||
|
||||
|
||||
## Design Principles
|
||||
|
||||
1. Default config is stable
|
||||
|
||||
- `backend/app/core/data_sources.yaml` remains the repository baseline.
|
||||
- It should be changed intentionally through normal development flow, not by autonomous runtime agents.
|
||||
|
||||
2. Runtime fixes are isolated
|
||||
|
||||
- Emergency or adaptive fixes should live in a runtime override layer.
|
||||
- Overrides should be reversible and auditable.
|
||||
|
||||
3. Deterministic checks come first
|
||||
|
||||
- Use normal programmatic health checks before using LLMs.
|
||||
- Only call an agent when deterministic checks indicate a meaningful failure.
|
||||
|
||||
4. Agents suggest before they mutate
|
||||
|
||||
- Agents should produce proposals with evidence and confidence.
|
||||
- Application of a proposal should be controlled by policy.
|
||||
|
||||
5. Every repair is attributable
|
||||
|
||||
- Store what changed, why, who or what suggested it, and when it was applied.
|
||||
|
||||
|
||||
## Configuration Layers
|
||||
|
||||
Recommended runtime precedence:
|
||||
|
||||
1. datasource endpoint override
|
||||
2. datasource DB endpoint override
|
||||
3. repository default YAML
|
||||
4. collector internal fallback logic
|
||||
|
||||
Definitions:
|
||||
|
||||
- repository default YAML:
|
||||
- `backend/app/core/data_sources.yaml`
|
||||
- versioned baseline
|
||||
- datasource DB endpoint override:
|
||||
- existing `DataSourceConfig.endpoint`
|
||||
- current runtime override entrypoint
|
||||
- datasource endpoint override:
|
||||
- a dedicated new override table
|
||||
- used for health-repair and proposal application
|
||||
- collector internal fallback logic:
|
||||
- final defensive fallback
|
||||
- should be minimized over time
|
||||
|
||||
|
||||
## Recommended Architecture
|
||||
|
||||
### 1. Deterministic Health Checks
|
||||
|
||||
Each collector gets a health profile with checks such as:
|
||||
|
||||
- endpoint resolves
|
||||
- HTTP request succeeds
|
||||
- status code is acceptable
|
||||
- content-type is expected
|
||||
- body parses successfully
|
||||
- minimum structural fields exist
|
||||
- sample item count is plausible
|
||||
- latency is within threshold
|
||||
|
||||
Output states:
|
||||
|
||||
- `healthy`
|
||||
- `degraded`
|
||||
- `failed`
|
||||
- `schema_changed`
|
||||
- `rate_limited`
|
||||
- `auth_required`
|
||||
|
||||
|
||||
### 2. Agent-Assisted Repair Discovery
|
||||
|
||||
Only triggered when deterministic health checks fail or return suspicious structure.
|
||||
|
||||
Agent responsibilities:
|
||||
|
||||
- search for current official endpoint or replacement path
|
||||
- inspect likely upstream documentation or landing pages
|
||||
- compare candidate endpoint output to collector expectations
|
||||
- produce a repair proposal with confidence and evidence
|
||||
|
||||
Agent should not directly modify repository defaults.
|
||||
|
||||
|
||||
### 3. Safe Runtime Repair Application
|
||||
|
||||
Repair proposals can be:
|
||||
|
||||
- reviewed manually
|
||||
- auto-applied only under strict low-risk policy
|
||||
|
||||
Auto-apply should be limited to cases like:
|
||||
|
||||
- same trusted domain
|
||||
- highly similar response structure
|
||||
- repeated successful verification
|
||||
- confidence above threshold
|
||||
|
||||
|
||||
## Phased Delivery Plan
|
||||
|
||||
## Phase 1: Deterministic Health MVP
|
||||
|
||||
Goal:
|
||||
|
||||
- build health observability without automated repair
|
||||
|
||||
Scope:
|
||||
|
||||
- datasource health check task runner
|
||||
- datasource health result persistence
|
||||
- endpoint reachability + parse checks
|
||||
- dashboard or API visibility into health status
|
||||
|
||||
Deliverables:
|
||||
|
||||
- health check service
|
||||
- health check record table
|
||||
- status endpoint
|
||||
- scheduled or manual check trigger
|
||||
|
||||
No agent usage yet.
|
||||
|
||||
|
||||
## Phase 2: Agent Repair Proposals
|
||||
|
||||
Goal:
|
||||
|
||||
- let agent investigate failing sources and propose updated endpoints
|
||||
|
||||
Scope:
|
||||
|
||||
- invoke agent only when datasource health is `failed` or `schema_changed`
|
||||
- web search + page inspection
|
||||
- candidate endpoint extraction
|
||||
- proposal persistence
|
||||
|
||||
Deliverables:
|
||||
|
||||
- repair proposal schema
|
||||
- proposal generation pipeline
|
||||
- confidence and evidence model
|
||||
- operator review view or API
|
||||
|
||||
Still no automatic config mutation.
|
||||
|
||||
|
||||
## Phase 3: Runtime Overrides
|
||||
|
||||
Goal:
|
||||
|
||||
- allow approved proposals to take effect safely at runtime
|
||||
|
||||
Scope:
|
||||
|
||||
- add dedicated override storage
|
||||
- runtime resolution prefers override over default config
|
||||
- proposal application writes override only
|
||||
|
||||
Deliverables:
|
||||
|
||||
- endpoint override table
|
||||
- override-aware resolution logic
|
||||
- apply/reject endpoints
|
||||
- rollback endpoint
|
||||
|
||||
Repository default YAML remains untouched.
|
||||
|
||||
|
||||
## Phase 4: Limited Auto-Apply
|
||||
|
||||
Goal:
|
||||
|
||||
- safely automate a narrow slice of low-risk repairs
|
||||
|
||||
Scope:
|
||||
|
||||
- policy engine for auto-apply
|
||||
- same-domain or trusted-domain checks
|
||||
- structure validation
|
||||
- staged verification after apply
|
||||
|
||||
Deliverables:
|
||||
|
||||
- auto-apply rules
|
||||
- audit logs
|
||||
- automatic post-apply health verification
|
||||
- auto-disable or rollback on regression
|
||||
|
||||
|
||||
## Data Model Draft
|
||||
|
||||
### datasource_health_checks
|
||||
|
||||
Purpose:
|
||||
|
||||
- store each health evaluation result
|
||||
|
||||
Suggested fields:
|
||||
|
||||
- `id`
|
||||
- `datasource_id`
|
||||
- `collector_name`
|
||||
- `endpoint_checked`
|
||||
- `status`
|
||||
- `http_status`
|
||||
- `content_type`
|
||||
- `latency_ms`
|
||||
- `sample_count`
|
||||
- `error_message`
|
||||
- `details`
|
||||
- `checked_at`
|
||||
|
||||
`details` can store structured diagnostic data such as:
|
||||
|
||||
- parsed fields
|
||||
- schema mismatch summary
|
||||
- retry count
|
||||
- exception class
|
||||
|
||||
|
||||
### datasource_repair_proposals
|
||||
|
||||
Purpose:
|
||||
|
||||
- store agent-generated repair suggestions
|
||||
|
||||
Suggested fields:
|
||||
|
||||
- `id`
|
||||
- `datasource_id`
|
||||
- `collector_name`
|
||||
- `old_endpoint`
|
||||
- `candidate_endpoint`
|
||||
- `reason`
|
||||
- `confidence`
|
||||
- `evidence_urls`
|
||||
- `evidence_summary`
|
||||
- `status`
|
||||
- `created_by`
|
||||
- `created_at`
|
||||
- `reviewed_at`
|
||||
|
||||
Suggested `status` values:
|
||||
|
||||
- `proposed`
|
||||
- `approved`
|
||||
- `rejected`
|
||||
- `applied`
|
||||
- `expired`
|
||||
|
||||
|
||||
### datasource_endpoint_overrides
|
||||
|
||||
Purpose:
|
||||
|
||||
- runtime endpoint override layer
|
||||
|
||||
Suggested fields:
|
||||
|
||||
- `id`
|
||||
- `datasource_id`
|
||||
- `collector_name`
|
||||
- `endpoint`
|
||||
- `reason`
|
||||
- `source`
|
||||
- `proposal_id`
|
||||
- `enabled`
|
||||
- `created_at`
|
||||
- `updated_at`
|
||||
|
||||
Suggested `source` values:
|
||||
|
||||
- `manual`
|
||||
- `health-agent`
|
||||
- `migration`
|
||||
|
||||
|
||||
## API Draft
|
||||
|
||||
### Health
|
||||
|
||||
- `GET /api/v1/datasources/health`
|
||||
- `GET /api/v1/datasources/{id}/health`
|
||||
- `POST /api/v1/datasources/{id}/health-check`
|
||||
- `POST /api/v1/datasources/health-check-all`
|
||||
|
||||
### Repair proposals
|
||||
|
||||
- `GET /api/v1/datasources/{id}/repair-proposals`
|
||||
- `POST /api/v1/datasources/{id}/repair-proposals/generate`
|
||||
- `POST /api/v1/datasources/{id}/repair-proposals/{proposal_id}/approve`
|
||||
- `POST /api/v1/datasources/{id}/repair-proposals/{proposal_id}/reject`
|
||||
- `POST /api/v1/datasources/{id}/repair-proposals/{proposal_id}/apply`
|
||||
|
||||
### Overrides
|
||||
|
||||
- `GET /api/v1/datasources/{id}/overrides`
|
||||
- `POST /api/v1/datasources/{id}/overrides`
|
||||
- `PUT /api/v1/datasources/{id}/overrides/{override_id}`
|
||||
- `DELETE /api/v1/datasources/{id}/overrides/{override_id}`
|
||||
|
||||
|
||||
## Agent Contract Draft
|
||||
|
||||
When deterministic health fails, the agent should receive:
|
||||
|
||||
- datasource name
|
||||
- collector name
|
||||
- current endpoint
|
||||
- current failure mode
|
||||
- expected response shape summary
|
||||
- known trusted domains
|
||||
|
||||
Expected output:
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "proposal",
|
||||
"candidate_endpoint": "https://example.com/api/v2/data",
|
||||
"confidence": 0.86,
|
||||
"reason": "Official docs now point to v2 endpoint",
|
||||
"evidence_urls": [
|
||||
"https://example.com/docs/api",
|
||||
"https://example.com/changelog"
|
||||
],
|
||||
"notes": "Response shape appears compatible after light field remapping"
|
||||
}
|
||||
```
|
||||
|
||||
The agent should never output "rewrite the default yaml" as its primary action.
|
||||
|
||||
|
||||
## Risk Analysis
|
||||
|
||||
### Risk: wrong endpoint chosen by agent
|
||||
|
||||
Mitigation:
|
||||
|
||||
- use trusted-domain allowlists
|
||||
- require evidence URLs
|
||||
- require confidence threshold
|
||||
- add manual review for medium-risk sources
|
||||
|
||||
|
||||
### Risk: endpoint responds but schema silently changed
|
||||
|
||||
Mitigation:
|
||||
|
||||
- deterministic schema checks
|
||||
- parse and sample validation
|
||||
- content-type checks
|
||||
- collector-specific required fields
|
||||
|
||||
|
||||
### Risk: automatic runtime override causes hidden drift
|
||||
|
||||
Mitigation:
|
||||
|
||||
- store all overrides explicitly
|
||||
- mark source of override
|
||||
- keep default YAML unchanged
|
||||
- expose active overrides in API/UI
|
||||
|
||||
|
||||
### Risk: persistent bad override breaks data collection
|
||||
|
||||
Mitigation:
|
||||
|
||||
- allow rollback
|
||||
- keep parent/default endpoint visible
|
||||
- re-run verification after apply
|
||||
- auto-disable override on repeated failure
|
||||
|
||||
|
||||
## Operational Policy Recommendations
|
||||
|
||||
1. Do not auto-apply for high-value or high-fragility sources initially.
|
||||
|
||||
2. Use manual approval for:
|
||||
|
||||
- scraped HTML sources
|
||||
- unofficial mirrors
|
||||
- sources with auth or rate-limit complexity
|
||||
- sources with legal or trust ambiguity
|
||||
|
||||
3. Allow auto-apply only for:
|
||||
|
||||
- same-domain version bumps
|
||||
- obvious official migration paths
|
||||
- repeated passing verification
|
||||
|
||||
4. Expose health + proposal + override state together in one operator view.
|
||||
|
||||
|
||||
## Suggested Implementation Order
|
||||
|
||||
1. Phase 1
|
||||
- health result table
|
||||
- deterministic checks
|
||||
- API and UI visibility
|
||||
|
||||
2. Phase 2
|
||||
- proposal table
|
||||
- agent prompt/output contract
|
||||
- proposal generation job
|
||||
|
||||
3. Phase 3
|
||||
- runtime override table
|
||||
- resolver precedence update
|
||||
- apply/reject endpoints
|
||||
|
||||
4. Phase 4
|
||||
- auto-apply rules
|
||||
- rollback policy
|
||||
- operator automation
|
||||
|
||||
|
||||
## Out Of Scope For The First Iteration
|
||||
|
||||
- direct automatic mutation of repository default YAML
|
||||
- automatic git commits by repair agents
|
||||
- unrestricted autonomous endpoint replacement
|
||||
- fully generalized schema remapping engine
|
||||
|
||||
|
||||
## Recommended First Milestone
|
||||
|
||||
The first milestone should be:
|
||||
|
||||
- deterministic datasource health checks
|
||||
- persisted results
|
||||
- manual visibility
|
||||
- no automatic repair
|
||||
|
||||
This gives immediate operational value with low risk, and prepares clean inputs for the later agent phase.
|
||||
478
docs/plans/agents-datasource-health-stage2-tasks.md
Normal file
478
docs/plans/agents-datasource-health-stage2-tasks.md
Normal file
@@ -0,0 +1,478 @@
|
||||
# Datasource Health Stage 2 Tasks
|
||||
|
||||
## Goal
|
||||
|
||||
Stage 2 focuses on the first practical operational layer:
|
||||
|
||||
- deterministic datasource health checks
|
||||
- persisted health results
|
||||
- health visibility through API and UI
|
||||
- no agent-assisted repair yet
|
||||
|
||||
This stage should make Planet capable of answering:
|
||||
|
||||
- which collectors are healthy
|
||||
- which collectors are degraded
|
||||
- which collectors are failing
|
||||
- why they are failing at a basic deterministic level
|
||||
|
||||
|
||||
## Scope
|
||||
|
||||
Included:
|
||||
|
||||
- datasource health data model
|
||||
- deterministic health check service
|
||||
- manual and scheduled health check triggers
|
||||
- health result APIs
|
||||
- frontend visibility
|
||||
|
||||
Excluded:
|
||||
|
||||
- LLM reasoning
|
||||
- web-search-based repair proposals
|
||||
- automatic endpoint rewriting
|
||||
- runtime override application
|
||||
|
||||
|
||||
## Delivery Target
|
||||
|
||||
At the end of Stage 2, an operator should be able to:
|
||||
|
||||
1. see health status for each collector
|
||||
2. trigger a health check manually
|
||||
3. inspect the latest failure reason
|
||||
4. inspect the last checked endpoint
|
||||
5. understand whether the problem is:
|
||||
- unreachable
|
||||
- auth-related
|
||||
- rate-limit-related
|
||||
- schema-related
|
||||
- empty-data-related
|
||||
|
||||
|
||||
## Work Breakdown
|
||||
|
||||
## A. Data Model
|
||||
|
||||
### A1. Add datasource health record table
|
||||
|
||||
Create a new model, for example:
|
||||
|
||||
- `backend/app/models/datasource_health_check.py`
|
||||
|
||||
Suggested fields:
|
||||
|
||||
- `id`
|
||||
- `datasource_id`
|
||||
- `collector_name`
|
||||
- `endpoint_checked`
|
||||
- `status`
|
||||
- `http_status`
|
||||
- `content_type`
|
||||
- `latency_ms`
|
||||
- `sample_count`
|
||||
- `error_message`
|
||||
- `details`
|
||||
- `checked_at`
|
||||
|
||||
Suggested status enum values:
|
||||
|
||||
- `healthy`
|
||||
- `degraded`
|
||||
- `failed`
|
||||
- `schema_changed`
|
||||
- `rate_limited`
|
||||
- `auth_required`
|
||||
- `empty_result`
|
||||
|
||||
|
||||
### A2. Add datasource health summary fields
|
||||
|
||||
Option A:
|
||||
|
||||
- keep summary only in the health check table
|
||||
|
||||
Option B:
|
||||
|
||||
- also add summary fields on `data_sources`
|
||||
|
||||
Recommended first step:
|
||||
|
||||
- do not mutate `data_sources` schema yet
|
||||
- derive summary from the latest health record
|
||||
|
||||
|
||||
### A3. Migration task
|
||||
|
||||
Add migration for the health table.
|
||||
|
||||
Deliverables:
|
||||
|
||||
- migration file
|
||||
- model registration
|
||||
|
||||
|
||||
## B. Health Check Engine
|
||||
|
||||
### B1. Define health check service
|
||||
|
||||
Add a new service module, for example:
|
||||
|
||||
- `backend/app/services/datasource_health.py`
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- resolve effective endpoint
|
||||
- execute deterministic check
|
||||
- classify result
|
||||
- persist health record
|
||||
|
||||
|
||||
### B2. Define shared result schema
|
||||
|
||||
Create a typed result object, for example:
|
||||
|
||||
- `HealthCheckResult`
|
||||
|
||||
Suggested fields:
|
||||
|
||||
- `status`
|
||||
- `endpoint_checked`
|
||||
- `http_status`
|
||||
- `content_type`
|
||||
- `latency_ms`
|
||||
- `sample_count`
|
||||
- `error_message`
|
||||
- `details`
|
||||
|
||||
|
||||
### B3. Implement base deterministic checks
|
||||
|
||||
Every datasource should go through a minimal baseline check:
|
||||
|
||||
1. resolve endpoint
|
||||
2. perform request
|
||||
3. measure latency
|
||||
4. inspect status code
|
||||
5. inspect content type
|
||||
6. inspect body shape
|
||||
|
||||
Classification rules:
|
||||
|
||||
- network error -> `failed`
|
||||
- HTTP 401/403 -> `auth_required`
|
||||
- HTTP 429 -> `rate_limited`
|
||||
- HTTP 404/410 -> `failed`
|
||||
- parse failure -> `schema_changed`
|
||||
- zero or suspiciously empty results -> `empty_result` or `degraded`
|
||||
- valid parse -> `healthy`
|
||||
|
||||
|
||||
### B4. Add collector-aware adapters
|
||||
|
||||
Some collectors do not use the same fetch semantics.
|
||||
|
||||
Add adapter profiles such as:
|
||||
|
||||
- `http_json`
|
||||
- `http_csv`
|
||||
- `html_scrape`
|
||||
- `stream_probe`
|
||||
- `auth_session_http`
|
||||
|
||||
Initial mapping suggestion:
|
||||
|
||||
- `huggingface`, `peeringdb`, `cloudflare` -> `http_json`
|
||||
- `fao` -> `http_csv`
|
||||
- `top500`, `epoch_ai`, `telegeography live_map` -> `html_scrape`
|
||||
- `ris_live` -> `stream_probe`
|
||||
- `spacetrack` -> `auth_session_http`
|
||||
|
||||
|
||||
### B5. Add sample validation hooks
|
||||
|
||||
For each adapter, add a lightweight validation rule.
|
||||
|
||||
Examples:
|
||||
|
||||
- JSON array length > 0
|
||||
- CSV rows > 1
|
||||
- HTML page contains expected table or script patterns
|
||||
- stream source yields at least one valid event within timeout
|
||||
|
||||
|
||||
## C. Persistence and Query Layer
|
||||
|
||||
### C1. Save every check run
|
||||
|
||||
Each health check should insert a record.
|
||||
|
||||
Do not overwrite history in Stage 2.
|
||||
|
||||
|
||||
### C2. Add latest-health query helpers
|
||||
|
||||
Add helper functions to fetch:
|
||||
|
||||
- latest health record by datasource
|
||||
- latest failed health record
|
||||
- recent health history
|
||||
|
||||
|
||||
### C3. Optional retention policy
|
||||
|
||||
For Stage 2, retention can be deferred.
|
||||
|
||||
If desired, keep only:
|
||||
|
||||
- last N records per datasource
|
||||
|
||||
|
||||
## D. API Layer
|
||||
|
||||
### D1. Add health list endpoint
|
||||
|
||||
Suggested endpoint:
|
||||
|
||||
- `GET /api/v1/datasources/health`
|
||||
|
||||
Returns:
|
||||
|
||||
- datasource id
|
||||
- collector name
|
||||
- current endpoint
|
||||
- latest health status
|
||||
- last checked time
|
||||
- short reason
|
||||
|
||||
|
||||
### D2. Add per-datasource health detail endpoint
|
||||
|
||||
Suggested endpoint:
|
||||
|
||||
- `GET /api/v1/datasources/{id}/health`
|
||||
|
||||
Returns:
|
||||
|
||||
- latest record
|
||||
- recent history
|
||||
- detailed classification fields
|
||||
|
||||
|
||||
### D3. Add manual health trigger endpoint
|
||||
|
||||
Suggested endpoint:
|
||||
|
||||
- `POST /api/v1/datasources/{id}/health-check`
|
||||
|
||||
Behavior:
|
||||
|
||||
- run a health check now
|
||||
- persist the result
|
||||
- return the new record
|
||||
|
||||
|
||||
### D4. Add bulk health trigger endpoint
|
||||
|
||||
Suggested endpoint:
|
||||
|
||||
- `POST /api/v1/datasources/health-check-all`
|
||||
|
||||
Behavior:
|
||||
|
||||
- enqueue or run health checks for all active datasources
|
||||
|
||||
|
||||
## E. Scheduling
|
||||
|
||||
### E1. Add health scheduler task
|
||||
|
||||
Decide scheduling strategy.
|
||||
|
||||
Recommended first version:
|
||||
|
||||
- run collector jobs and health checks separately
|
||||
- health checks run on a lower frequency
|
||||
|
||||
Suggested frequency:
|
||||
|
||||
- every 6h or 12h for most datasources
|
||||
- optionally on-demand only in the very first cut
|
||||
|
||||
|
||||
### E2. Prevent health check collision with collection
|
||||
|
||||
Rules:
|
||||
|
||||
- health checks should not disrupt active collection
|
||||
- they should use light requests
|
||||
- if a collector is currently running, health check may:
|
||||
- skip
|
||||
- or use a lightweight endpoint probe only
|
||||
|
||||
|
||||
## F. Frontend
|
||||
|
||||
### F1. Add health columns to datasource list
|
||||
|
||||
Update:
|
||||
|
||||
- `frontend/src/pages/DataSources/DataSources.tsx`
|
||||
|
||||
Suggested new columns:
|
||||
|
||||
- health status
|
||||
- last checked
|
||||
- reason summary
|
||||
|
||||
|
||||
### F2. Add manual health check action
|
||||
|
||||
Per datasource:
|
||||
|
||||
- button or dropdown action:
|
||||
- `健康检查`
|
||||
|
||||
|
||||
### F3. Add health detail drawer or modal
|
||||
|
||||
Show:
|
||||
|
||||
- endpoint checked
|
||||
- status
|
||||
- HTTP status
|
||||
- content type
|
||||
- sample count
|
||||
- error message
|
||||
- last few results
|
||||
|
||||
|
||||
### F4. Add basic visual language
|
||||
|
||||
Suggested colors:
|
||||
|
||||
- green -> healthy
|
||||
- yellow -> degraded
|
||||
- orange -> rate-limited / auth-required
|
||||
- red -> failed / schema-changed
|
||||
|
||||
|
||||
## G. Observability
|
||||
|
||||
### G1. Structured logging
|
||||
|
||||
Every health check should log:
|
||||
|
||||
- datasource id
|
||||
- collector name
|
||||
- endpoint
|
||||
- status
|
||||
- latency
|
||||
- failure class
|
||||
|
||||
|
||||
### G2. Optional metrics
|
||||
|
||||
If metrics are added later, useful counters include:
|
||||
|
||||
- health checks total
|
||||
- health checks failed
|
||||
- schema changes detected
|
||||
- rate limited checks
|
||||
|
||||
|
||||
## H. Tests
|
||||
|
||||
### H1. Unit tests
|
||||
|
||||
Add tests for:
|
||||
|
||||
- status classification
|
||||
- content type classification
|
||||
- adapter behavior
|
||||
- latest-health query helpers
|
||||
|
||||
|
||||
### H2. API tests
|
||||
|
||||
Add tests for:
|
||||
|
||||
- health endpoints require auth
|
||||
- manual trigger endpoint works
|
||||
- list endpoint returns latest status
|
||||
|
||||
|
||||
### H3. Failure-path tests
|
||||
|
||||
Add coverage for:
|
||||
|
||||
- HTTP 404
|
||||
- HTTP 429
|
||||
- invalid JSON
|
||||
- empty response
|
||||
- parse mismatch
|
||||
|
||||
|
||||
## Suggested File Plan
|
||||
|
||||
Possible implementation files:
|
||||
|
||||
- `backend/app/models/datasource_health_check.py`
|
||||
- `backend/app/services/datasource_health.py`
|
||||
- `backend/app/schemas/datasource_health.py`
|
||||
- `backend/app/api/v1/datasource_health.py`
|
||||
- migration file under the project migration system
|
||||
|
||||
Likely touched existing files:
|
||||
|
||||
- `backend/app/api/main.py`
|
||||
- `frontend/src/pages/DataSources/DataSources.tsx`
|
||||
- `backend/tests/test_api.py`
|
||||
|
||||
|
||||
## Suggested Execution Order
|
||||
|
||||
1. Add model and migration
|
||||
2. Add service and result schema
|
||||
3. Add deterministic adapters
|
||||
4. Add manual trigger API
|
||||
5. Add list/detail API
|
||||
6. Add frontend visibility
|
||||
7. Add scheduled checks
|
||||
8. Expand tests
|
||||
|
||||
|
||||
## Minimal First Milestone
|
||||
|
||||
If we want the fastest useful slice, do this first:
|
||||
|
||||
1. health table
|
||||
2. deterministic check service
|
||||
3. manual per-datasource health check API
|
||||
4. latest health list API
|
||||
5. frontend status badge column
|
||||
|
||||
That is enough to start operating the system and will provide the input layer for Stage 3.
|
||||
|
||||
|
||||
## Dependency On Later Stages
|
||||
|
||||
Stage 2 outputs become direct inputs for Stage 3.
|
||||
|
||||
Specifically:
|
||||
|
||||
- failed or schema-changed health records become agent triggers
|
||||
- health history becomes repair context
|
||||
- endpoint_checked becomes proposal baseline
|
||||
|
||||
|
||||
## Success Criteria
|
||||
|
||||
Stage 2 is done when:
|
||||
|
||||
- every active datasource can be health-checked deterministically
|
||||
- the latest health state is visible in API and UI
|
||||
- operators can manually trigger checks
|
||||
- failures are categorized into stable machine-readable statuses
|
||||
- no LLM is required for core health visibility
|
||||
309
docs/plans/agents-situational-awareness-foundation-plan.md
Normal file
309
docs/plans/agents-situational-awareness-foundation-plan.md
Normal file
@@ -0,0 +1,309 @@
|
||||
# Situational Awareness Foundation Plan
|
||||
|
||||
## 定位
|
||||
|
||||
当前这套 AI 能力应被视为 `态势感知服务底座`,而不是完整的态势感知产品。
|
||||
|
||||
也就是说,现阶段的目标不是:
|
||||
|
||||
- 做一个“什么都能分析”的万能 AI 页面
|
||||
- 让模型在证据不足时替代人工研判
|
||||
- 过早把页面做成完整指挥大屏
|
||||
|
||||
现阶段真正要做的是:
|
||||
|
||||
- 先把 `model gateway / backend facade / evidence injection / page-specific brief` 这几层边界搭稳
|
||||
- 让系统能够在已有证据上稳定地产出“可读、可回看、可扩展”的摘要
|
||||
- 为后续更强的数据联动、agent 推理和 assessment 结构化输出预留好接口与数据模型
|
||||
|
||||
## 当前现实约束
|
||||
|
||||
### 1. 数据维度不足
|
||||
|
||||
目前系统能提供的主要证据仍集中在:
|
||||
|
||||
- BGP incidents / anomalies / events
|
||||
- collector coverage
|
||||
- datasource health / platform alerts
|
||||
- prefix geography 的部分归属信息
|
||||
|
||||
当前明显还缺:
|
||||
|
||||
- 流量异常与业务指标
|
||||
- 电商、支付、物流等业务侧指标
|
||||
- 更丰富的资产、链路、区域、行业画像
|
||||
- 外部舆情、公告、运营商状态、基础设施事件等背景信息
|
||||
|
||||
这意味着:
|
||||
|
||||
- 模型现在可以做“基于现有证据的摘要与归纳”
|
||||
- 但还不能可靠地做“跨维度因果研判”
|
||||
|
||||
### 2. 维度之间联动还弱
|
||||
|
||||
目前不同模块之间更多是“并列展示”,还不是“强关联分析”:
|
||||
|
||||
- 系统告警和 BGP 事件还没有统一事件模型
|
||||
- collector bias 与真实区域热度还没有完全剥离
|
||||
- datasource health 与 BGP 风险、业务影响之间还没有稳定映射
|
||||
|
||||
这意味着:
|
||||
|
||||
- 当前更适合做 `brief / overview / operator notes`
|
||||
- 还不适合过度承诺“自动态势判断”
|
||||
|
||||
### 3. 结构化 assessment 还未成为主输出
|
||||
|
||||
虽然已经有 BGP brief、系统告警 brief、态势告警 brief,但目前主输出仍偏向:
|
||||
|
||||
- 文本摘要
|
||||
- facts/context 附带证据
|
||||
|
||||
后续真正要服务态势感知,需要更稳定的结构化输出,例如:
|
||||
|
||||
- summary
|
||||
- key risks
|
||||
- evidence
|
||||
- confidence
|
||||
- recommendations
|
||||
- missing data
|
||||
|
||||
## 当前基座已经具备的能力
|
||||
|
||||
### 1. AI 调用边界已经明确
|
||||
|
||||
- `aiprovider` 负责模型协议与 provider 兼容
|
||||
- `backend` 负责业务 API、证据整合和鉴权
|
||||
- `frontend` 负责页面入口与结果展示
|
||||
|
||||
### 2. 页面级 AI 入口已经开始成型
|
||||
|
||||
当前已经有或正在收口的入口:
|
||||
|
||||
- `Playground`
|
||||
- 用于链路验证与 provider 诊断
|
||||
- `BGP AI 简报`
|
||||
- 用于 BGP 事实摘要和区域风险归纳
|
||||
- `Alerts`
|
||||
- 用于系统告警、BGP 告警、态势告警三类入口
|
||||
|
||||
### 3. 证据优先的方向已经建立
|
||||
|
||||
已经不再只依赖人工在 Playground 中手填 prompt,系统开始具备:
|
||||
|
||||
- 从真实业务数据生成事实输入
|
||||
- 保存 facts/context 快照
|
||||
- 回看 AI 输出时同时回看证据
|
||||
|
||||
这一步非常关键,因为它决定后面能否从“玩具 demo”走向“有运维价值的系统”。
|
||||
|
||||
## 近期收尾建议
|
||||
|
||||
这些事情都属于“底座收口”,值得做,但不应该再继续重产品包装。
|
||||
|
||||
### 1. 统一 Alerts 页面
|
||||
|
||||
已采用:
|
||||
|
||||
- 一个 `Alerts` 页面
|
||||
- 三个 tab:
|
||||
- `系统告警`
|
||||
- `BGP 告警`
|
||||
- `态势告警`
|
||||
|
||||
收尾重点:
|
||||
|
||||
- 保持 tab 的文案、摘要卡和 AI 简报交互一致
|
||||
- 不额外扩展成多个独立二级页面
|
||||
|
||||
### 2. 保持 Playground 为测试台
|
||||
|
||||
原则:
|
||||
|
||||
- Playground 只承担链路验证、provider 状态诊断、请求结果观察
|
||||
- 不继续堆“万能业务分析器”式交互
|
||||
|
||||
### 3. 把 brief 能力当服务能力而不是页面特效
|
||||
|
||||
页面现在能看到按钮和结果,这很好,但更重要的是:
|
||||
|
||||
- 后端接口稳定
|
||||
- facts/context 可追踪
|
||||
- 输出结构后续可升级
|
||||
|
||||
### 4. 导航结构先收口,不继续平铺一级菜单
|
||||
|
||||
随着后续能力扩展,系统很可能继续新增:
|
||||
|
||||
- 海缆
|
||||
- 算力中心
|
||||
- 战争信息
|
||||
- 电商分析
|
||||
- 其他专题观测页
|
||||
|
||||
如果继续把这些入口全部平铺在左侧一级菜单中,会带来两个问题:
|
||||
|
||||
- 一级菜单过长,用户难以判断先进入哪个上下文
|
||||
- `观测页 / 告警页 / 研判页 / 运维页` 的职责边界会被混在一起
|
||||
|
||||
因此近期应明确采用分组导航,而不是继续扩展平铺菜单。
|
||||
|
||||
推荐的导航分组如下:
|
||||
|
||||
- `总览`
|
||||
- 仪表盘
|
||||
- Earth
|
||||
- `专题观测`
|
||||
- BGP 观测
|
||||
- 采集数据
|
||||
- 后续可扩展:海缆、算力中心、战争信息、电商分析
|
||||
- `告警与研判`
|
||||
- Alerts
|
||||
- `运维与配置`
|
||||
- 数据源
|
||||
- AI Playground
|
||||
- 用户管理
|
||||
- 系统配置
|
||||
|
||||
这套结构的含义是:
|
||||
|
||||
- `专题观测` 页面负责看某个维度本身
|
||||
- `Alerts` 负责跨模块风险与值班工作台
|
||||
- `Playground` 保持为测试台,不挤占业务导航语义
|
||||
|
||||
短期收尾时,应优先重组现有入口,而不是继续增加新的一级菜单。
|
||||
|
||||
## 后续路线
|
||||
|
||||
## Phase 1:服务底座稳固
|
||||
|
||||
目标:
|
||||
|
||||
- 不追求“更炫的 AI 页面”
|
||||
- 先把当前接口、证据、存储和页面入口收稳
|
||||
|
||||
工作项:
|
||||
|
||||
- 统一页面级 AI 入口模式
|
||||
- 统一 brief response schema
|
||||
- 保证 facts/context 在前后端都可回看
|
||||
- 继续清理 mock 和临时分支逻辑
|
||||
|
||||
完成标准:
|
||||
|
||||
- 每个 AI 入口都是真实链路
|
||||
- 每个 AI 结果都能追溯到证据输入
|
||||
|
||||
## Phase 2:Evidence-first Assessment
|
||||
|
||||
目标:
|
||||
|
||||
- 从“文本摘要”升级成“结构化 assessment”
|
||||
|
||||
工作项:
|
||||
|
||||
- 为 brief/assessment 定义统一 schema
|
||||
- 固化:
|
||||
- summary
|
||||
- key_risks
|
||||
- evidence
|
||||
- confidence
|
||||
- recommendations
|
||||
- missing_data
|
||||
- 页面以结构化区块展示,而不只是大段文本
|
||||
|
||||
完成标准:
|
||||
|
||||
- AI 输出可持久化、可比较、可审计
|
||||
|
||||
## Phase 3:多维证据接入
|
||||
|
||||
目标:
|
||||
|
||||
- 让“态势感知”真正拥有更多维度,而不是只靠 BGP 与系统告警
|
||||
|
||||
优先接入方向:
|
||||
|
||||
- datasource health findings
|
||||
- 流量或业务指标
|
||||
- 区域/资产/链路映射
|
||||
- 外部事件与公告
|
||||
- 业务垂直数据,例如电商分析相关指标
|
||||
|
||||
完成标准:
|
||||
|
||||
- AI 能基于多个维度做交叉说明
|
||||
- 不再只围绕单一模块自说自话
|
||||
|
||||
## Phase 4:Correlation Layer
|
||||
|
||||
目标:
|
||||
|
||||
- 不同来源的信号不再只是并列,而是形成统一的事件关联
|
||||
|
||||
工作项:
|
||||
|
||||
- 统一 signal/finding 模型
|
||||
- 跨模块事件聚合
|
||||
- 证据来源权重
|
||||
- collector bias 与真实热度分离
|
||||
|
||||
完成标准:
|
||||
|
||||
- 系统能回答“这些异常是不是同一件事”
|
||||
- 系统能回答“哪些结论只是观测偏差”
|
||||
|
||||
## Phase 5:Agent-assisted Situational Awareness
|
||||
|
||||
目标:
|
||||
|
||||
- 在证据足够的前提下,再让 agent 负责更复杂的推理与建议
|
||||
|
||||
工作项:
|
||||
|
||||
- 复用现有 agent runtime 规划
|
||||
- 引入 web search / docs fetch / repair proposal 等能力
|
||||
- 但始终坚持:
|
||||
- evidence first
|
||||
- proposal before action
|
||||
- no silent mutation of defaults
|
||||
|
||||
完成标准:
|
||||
|
||||
- agent 成为证据驱动的分析层
|
||||
- 而不是一个“万能猜测层”
|
||||
|
||||
## 设计原则
|
||||
|
||||
### 1. 先底座,后产品化
|
||||
|
||||
先把服务链路和证据模型做好,再做更大的页面表达。
|
||||
|
||||
### 2. 先证据,后判断
|
||||
|
||||
事实输入应先稳定,再让模型做归纳。
|
||||
|
||||
### 3. 先专用 brief,后统一态势层
|
||||
|
||||
先让各业务页有各自可信的 AI 入口,再考虑统一态势页。
|
||||
|
||||
### 4. 先 proposal,后自动动作
|
||||
|
||||
涉及修复、覆盖、写配置、调任务的动作,都应经过 proposal 和审计。
|
||||
|
||||
## 当前建议结论
|
||||
|
||||
对现在这个项目,最合理的定位是:
|
||||
|
||||
- `Playground` 是测试台
|
||||
- `BGP / Alerts` 是第一批业务 AI 入口
|
||||
- `aiprovider + backend AI facade + evidence snapshots` 是核心服务底座
|
||||
|
||||
现阶段不需要追求“已经具备完整态势感知能力”。
|
||||
|
||||
现阶段真正的成功标准是:
|
||||
|
||||
- 这套底座可用
|
||||
- 可回看
|
||||
- 可扩展
|
||||
- 不自欺欺人
|
||||
@@ -17,7 +17,7 @@ It is an aggregation/view-model layer:
|
||||
|
||||
## Why This Layer Exists
|
||||
|
||||
Current product gap from [bgp-context.md](/home/ray/dev/linkong/planet/docs/bgp-context.md):
|
||||
Current product gap from [bgp-context.md](/home/ray/dev/linkong/planet/docs/technical/earth-bgp-context.md):
|
||||
|
||||
- incident density is naturally low
|
||||
- anomaly density is higher, but still not enough to keep the globe expressive all the time
|
||||
@@ -290,7 +290,7 @@ Each feature should include:
|
||||
|
||||
## Earth Rendering Plan
|
||||
|
||||
Detailed visual layering guidance is expanded in [bgp-earth-rendering-plan.md](/home/ray/dev/linkong/planet/docs/bgp-earth-rendering-plan.md).
|
||||
Detailed visual layering guidance is expanded in [bgp-earth-rendering-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-bgp-earth-rendering-plan.md).
|
||||
|
||||
### Layer Relationship
|
||||
|
||||
715
docs/plans/earth-celestial-background-plan.md
Normal file
715
docs/plans/earth-celestial-background-plan.md
Normal file
@@ -0,0 +1,715 @@
|
||||
# Earth 天球背景与日月位置实施方案
|
||||
|
||||
## 目标
|
||||
|
||||
为 Earth 大屏增加一套真正可用的天文背景层,覆盖三件事:
|
||||
|
||||
1. 用真实天球背景替换当前随机星点
|
||||
2. 在当前时间下显示太阳与月亮的相对位置
|
||||
3. 让太阳方向同时驱动地球受光,形成更可信的昼夜关系
|
||||
|
||||
本方案优先追求:
|
||||
|
||||
- 与当前 Three.js Earth 架构兼容
|
||||
- 风险可控
|
||||
- 先落地一版真实感明显提升的 V1
|
||||
- 为后续更严格的天文参考系升级预留余地
|
||||
|
||||
## 当前现状
|
||||
|
||||
当前 Earth 的基础条件已经具备:
|
||||
|
||||
- 地球、云层、地形、网格都基于 Three.js,主渲染入口在 [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js)
|
||||
- 地球实体创建在 [frontend/public/earth/js/earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js)
|
||||
- 当前所谓“宇宙背景”只是 `createStars()` 生成的随机星点,不是真实星图
|
||||
- Earth 已有倾角常量 `EARTH_CONFIG.tiltRad`,位于 [frontend/public/earth/js/constants.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/constants.js)
|
||||
- 主循环 `animate()` 已稳定运行,可在其中接入天体更新逻辑
|
||||
|
||||
这意味着:
|
||||
|
||||
- 不需要重写 Earth
|
||||
- 可以在现有 scene/world 层新增一个 celestial layer
|
||||
- 第一阶段不必拆 Earth / satellite / cable 的参考系
|
||||
|
||||
## 总体策略
|
||||
|
||||
采用“两层现实”设计:
|
||||
|
||||
### 1. 世界层(world-space celestial layer)
|
||||
|
||||
用于放置:
|
||||
|
||||
- 天球背景
|
||||
- 太阳
|
||||
- 月亮
|
||||
- 太阳光方向
|
||||
|
||||
这些对象不挂在 `earthObj` 上,而是直接放在 `scene` 中。
|
||||
|
||||
### 2. 地球层(earth-fixed layer)
|
||||
|
||||
继续保持当前结构:
|
||||
|
||||
- 海缆
|
||||
- 登陆点
|
||||
- 卫星点与轨迹
|
||||
- BGP 覆盖
|
||||
- 地球纹理、云层、地形
|
||||
|
||||
这些对象继续挂在 `earthObj` 下,不打断现有交互。
|
||||
|
||||
## 为什么先这样做
|
||||
|
||||
当前用户交互是“拖动地球本体”,而不是“移动相机绕惯性系观测”。
|
||||
如果现在直接做严格惯性参考系改造,会同时影响:
|
||||
|
||||
- `earthObj.rotation`
|
||||
- 卫星轨迹与锁定逻辑
|
||||
- 海缆与登陆点附着关系
|
||||
- resetView / autoRotate / hover / click 等交互链路
|
||||
|
||||
所以第一阶段只做:
|
||||
|
||||
- 真正的天空
|
||||
- 真正的日月方向
|
||||
- 不碰现有 Earth 附着对象的语义
|
||||
|
||||
## 推荐技术选型
|
||||
|
||||
### 天文计算库
|
||||
|
||||
推荐:
|
||||
|
||||
- [Astronomy Engine](https://github.com/cosinekitty/astronomy)
|
||||
|
||||
原因:
|
||||
|
||||
- 有 JavaScript 版本
|
||||
- 支持 Sun / Moon 的矢量与坐标变换
|
||||
- 精度、可扩展性都比轻量太阳高度角库更适合本项目
|
||||
- 后续若要加行星、月相、黄道、赤道网,也能继续沿用
|
||||
|
||||
不作为主选的库:
|
||||
|
||||
- [SunCalc](https://github.com/mourner/suncalc)
|
||||
|
||||
原因:
|
||||
|
||||
- 更偏本地观察者视角的太阳/月亮高度角
|
||||
- 用于“地面日出日落”很好
|
||||
- 但不如 Astronomy Engine 适合做真实天球与后续空间参考系扩展
|
||||
|
||||
### Three.js 表现层
|
||||
|
||||
推荐组合:
|
||||
|
||||
- 天球:内翻球壳 + 星图纹理
|
||||
- 太阳:`THREE.Sprite`
|
||||
- 月亮:`THREE.Sprite` 或小型 `THREE.Mesh`
|
||||
- 太阳光:`THREE.DirectionalLight`
|
||||
|
||||
参考:
|
||||
|
||||
- [Three.js SpriteMaterial](https://threejs.org/docs/pages/SpriteMaterial.html)
|
||||
|
||||
## 天球背景资源与星体数据来源
|
||||
|
||||
为避免把“视觉背景”和“可计算天体位置”混为一谈,本方案明确分成两类资源:
|
||||
|
||||
### 1. 背景资源:全天星图贴图
|
||||
|
||||
用于 Phase 1 的“真实天空背景”。
|
||||
|
||||
推荐优先来源:
|
||||
|
||||
- NASA SVS 的 Tycho 全天星图
|
||||
- [The Tycho Catalog Skymap - Version 2.0](https://svs.gsfc.nasa.gov/3572/)
|
||||
- NASA Deep Star Maps 2020
|
||||
- SatelliteMap.space 在 credits 中明确提到其使用了 `NASA Deep Star Maps 2020 - High-resolution star field (1.7 billion stars from Gaia DR2)` 作为星空视觉资源
|
||||
- 这说明行业内成熟实现并不一定直接渲染全部星表点,而很可能先使用一张高质量官方深空星图作为背景层
|
||||
- 如需后续替换,也可评估 ESA / Gaia 的全天 sky map 资源
|
||||
- [Gaia DR3 stories](https://www.cosmos.esa.int/web/gaia/dr3-stories)
|
||||
|
||||
建议要求:
|
||||
|
||||
- 使用官方来源或官方衍生可复用资源
|
||||
- 等距矩形投影(equirectangular)
|
||||
- 坐标定义尽量明确为赤道坐标展开
|
||||
- 分辨率建议至少 `4k`
|
||||
- 颜色不要过亮,避免压过 Earth HUD 前景
|
||||
- 尽量优先选择官方天文机构已经生产好的深空图,而不是自行拼接低质量星空纹理
|
||||
|
||||
建议本地资源目录:
|
||||
|
||||
- `frontend/public/earth/assets/celestial/starmap_equatorial_4k.jpg`
|
||||
|
||||
### 2. 位置数据:星表与天体计算
|
||||
|
||||
用于 Phase 2+ 的“位置正确的星体”。
|
||||
|
||||
推荐来源分两层:
|
||||
|
||||
- 太阳、月亮位置
|
||||
- 使用 [Astronomy Engine](https://github.com/cosinekitty/astronomy)
|
||||
- 恒星位置
|
||||
- 第一优先:Hipparcos / Tycho
|
||||
- [Hipparcos overview](https://www.cosmos.esa.int/web/Hipparcos)
|
||||
- [Hipparcos catalogues](https://www.cosmos.esa.int/web/hipparcos/catalogues)
|
||||
- 第二优先:Gaia
|
||||
- [Gaia DR3 stories](https://www.cosmos.esa.int/web/gaia/dr3-stories)
|
||||
|
||||
建议策略:
|
||||
|
||||
- V1:背景球壳只用全天星图,不立即生成全量恒星点
|
||||
- V2:只挑选亮星(例如星等 `< 5.5`)生成恒星点层
|
||||
- V3:如果确实需要更丰富的星场,再逐步扩展到更深星等
|
||||
|
||||
这样做的原因:
|
||||
|
||||
- 背景球壳负责“天球真实感”
|
||||
- 亮星点负责“位置正确、可后续标注和高亮”
|
||||
- 不需要一开始就处理数十万甚至数百万颗星
|
||||
|
||||
### 3. 对外部成熟实现的参考结论
|
||||
|
||||
`SatelliteMap.space` 的公开 credits 提供了一个很有价值的参考样板:
|
||||
|
||||
- 图形渲染使用 `TWGL.js`
|
||||
- 天文计算使用 `Skyfield` 与 `Astronomia`
|
||||
- 星空/天球视觉资源使用 `NASA Deep Star Maps 2020`
|
||||
|
||||
这给本项目的启发是:
|
||||
|
||||
- “真实感强的天球背景”完全可以先依赖官方高质量深空图
|
||||
- “位置正确的动态天体”则应依赖单独的天文计算链路
|
||||
- 没有必要在第一版就直接渲染完整星表
|
||||
|
||||
因此本项目推荐继续坚持两层拆分:
|
||||
|
||||
- 背景层:官方深空图 / 全天星图
|
||||
- 计算层:太阳、月亮与后续亮星点
|
||||
|
||||
## 如何保证星体位置正确
|
||||
|
||||
位置正确不是只看“图看起来像”,而是要统一参考系和转换链路。
|
||||
|
||||
### 1. 统一坐标基准
|
||||
|
||||
本方案推荐统一使用:
|
||||
|
||||
- `J2000` 赤道坐标系作为恒星位置基准
|
||||
|
||||
原因:
|
||||
|
||||
- Hipparcos / Tycho 资料和大量天文可视化都容易映射到该基准
|
||||
- 太阳、月亮也可以通过 Astronomy Engine 转到同一坐标系
|
||||
- 这样背景、恒星点、太阳、月亮就能共用一套 sky orientation
|
||||
|
||||
### 2. 背景贴图与点位必须使用同一展开逻辑
|
||||
|
||||
如果背景球壳使用赤道坐标全天图,那么:
|
||||
|
||||
- 亮星点也必须按赤道坐标贴到同一球面方向
|
||||
- 太阳/月亮 sprite 也必须按赤道坐标转换后落到同一 world-space
|
||||
|
||||
否则会出现:
|
||||
|
||||
- 背景银河带是对的
|
||||
- 但太阳/月亮或亮星点飘到不匹配的位置
|
||||
|
||||
### 3. RA / Dec 到 Three.js 坐标的落点方式
|
||||
|
||||
亮星点和日月方向最终都要转成单位球面向量。
|
||||
|
||||
概念步骤:
|
||||
|
||||
1. 读取赤经 `RA`
|
||||
2. 读取赤纬 `Dec`
|
||||
3. 转成弧度
|
||||
4. 映射到单位球面向量
|
||||
5. 再根据 Three.js 当前世界坐标定义做轴向映射
|
||||
|
||||
参考公式:
|
||||
|
||||
```text
|
||||
x = cos(dec) * cos(ra)
|
||||
y = sin(dec)
|
||||
z = cos(dec) * sin(ra)
|
||||
```
|
||||
|
||||
实际接入 Three.js 时,需要做一次项目内坐标轴校准:
|
||||
|
||||
- 验证 `RA = 0h`
|
||||
- 验证 `RA = 6h`
|
||||
- 验证北天极
|
||||
- 验证银河带主方向
|
||||
|
||||
然后确定最终的:
|
||||
|
||||
- `x/y/z` 对应 Three.js 哪个轴
|
||||
- 是否需要 `z` 取反
|
||||
- 是否需要整体再做一个固定 `rotation`
|
||||
|
||||
建议把这层显式封装在:
|
||||
|
||||
```js
|
||||
function equatorialToWorldVector(raRad, decRad)
|
||||
```
|
||||
|
||||
不要把轴映射散落在不同模块里。
|
||||
|
||||
### 4. 背景球壳与恒星点的关系
|
||||
|
||||
推荐最终组合:
|
||||
|
||||
- 背景层:全天星图球壳
|
||||
- 点位层:亮星点
|
||||
- 动态层:太阳 / 月亮
|
||||
|
||||
这样有三个好处:
|
||||
|
||||
- 背景层提供密集真实的天空纹理
|
||||
- 亮星点提供位置正确、可扩展的标注基础
|
||||
- 太阳/月亮提供与时间相关的真实动态对象
|
||||
|
||||
## 数据与资源建议清单
|
||||
|
||||
### 推荐首批引入资源
|
||||
|
||||
1. 全天星图
|
||||
- 来源:NASA Tycho all-sky map
|
||||
- 用途:背景球壳纹理
|
||||
|
||||
2. 月亮纹理
|
||||
- 用途:Phase 4 月相表现
|
||||
- 路径建议:
|
||||
- `frontend/public/earth/assets/celestial/moon_albedo_2k.jpg`
|
||||
|
||||
3. 太阳 glow 贴图
|
||||
- 用途:太阳 sprite halo
|
||||
- 路径建议:
|
||||
- `frontend/public/earth/assets/celestial/sun_glow.png`
|
||||
|
||||
### 推荐首批数据文件
|
||||
|
||||
如果要上亮星层,建议新增一个预处理后的轻量数据文件:
|
||||
|
||||
- `frontend/public/earth/assets/celestial/bright-stars.json`
|
||||
|
||||
建议字段:
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"id": 32349,
|
||||
"name": "Sirius",
|
||||
"raDeg": 101.2875,
|
||||
"decDeg": -16.7161,
|
||||
"mag": -1.46,
|
||||
"colorIndex": 0.00
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
建议不要在浏览器里直接吞原始 Gaia 大表,而是先离线裁剪成:
|
||||
|
||||
- 只保留亮星
|
||||
- 只保留渲染必需字段
|
||||
- JSON 或二进制轻量格式
|
||||
|
||||
## 资源与数据实施路线
|
||||
|
||||
### 路线 A:先做可用版本(推荐)
|
||||
|
||||
1. 引入 NASA Tycho 全天图
|
||||
- 或评估替换为更接近 SatelliteMap.space 路线的 `NASA Deep Star Maps 2020`
|
||||
2. 实现背景球壳
|
||||
3. 用 Astronomy Engine 计算太阳/月亮方向
|
||||
4. 暂不做亮星点
|
||||
|
||||
优点:
|
||||
|
||||
- 最快见效
|
||||
- 风险最低
|
||||
- 就能明显提升天球真实感
|
||||
|
||||
### 路线 B:在 A 基础上增强
|
||||
|
||||
1. 离线生成 `bright-stars.json`
|
||||
2. 浏览器端渲染亮星点
|
||||
3. 后续可加:
|
||||
- 星座线
|
||||
- 亮星名称
|
||||
- 特定星体高亮
|
||||
|
||||
优点:
|
||||
|
||||
- 背景真实感和“位置正确的可交互星体”同时兼顾
|
||||
|
||||
## 代码模块建议细化
|
||||
|
||||
### 新增模块
|
||||
|
||||
- `frontend/public/earth/js/celestial.js`
|
||||
- 管理天球背景
|
||||
- 管理太阳/月亮
|
||||
- 管理亮星层(后续)
|
||||
|
||||
- `frontend/public/earth/js/celestial-data.js`
|
||||
- 资源路径
|
||||
- 星图方向配置
|
||||
- 亮星数据加载(后续)
|
||||
|
||||
### 建议函数设计
|
||||
|
||||
```js
|
||||
export function initCelestialLayer(scene)
|
||||
export function updateCelestialLayer(date)
|
||||
export function setCelestialVisibility(visible)
|
||||
export function disposeCelestialLayer()
|
||||
|
||||
function loadStarMapTexture()
|
||||
function createSkySphere(texture)
|
||||
function createSunSprite()
|
||||
function createMoonSprite()
|
||||
function getSunEquatorialPosition(date)
|
||||
function getMoonEquatorialPosition(date)
|
||||
function equatorialToWorldVector(raRad, decRad)
|
||||
```
|
||||
|
||||
### 推荐后续预处理脚本
|
||||
|
||||
如要引入亮星层,建议单独做离线脚本:
|
||||
|
||||
- `scripts/build_bright_stars.py`
|
||||
|
||||
职责:
|
||||
|
||||
- 从 Hipparcos / Tycho 源数据读取
|
||||
- 过滤亮星
|
||||
- 生成 `bright-stars.json`
|
||||
|
||||
这样浏览器端只消费轻量结果,不承担大表解析成本。
|
||||
|
||||
## 分阶段实施
|
||||
|
||||
## Phase 1:真实天球背景
|
||||
|
||||
### 目标
|
||||
|
||||
用真实全天星图替换当前随机星点背景。
|
||||
|
||||
### 做法
|
||||
|
||||
1. 新增一张全天星图纹理
|
||||
|
||||
建议路径:
|
||||
|
||||
- `frontend/public/earth/assets/celestial/starmap_equatorial_4k.jpg`
|
||||
|
||||
纹理要求:
|
||||
|
||||
- 等距矩形投影
|
||||
- 赤经/赤纬坐标展开
|
||||
- 无地平线、无地景遮挡
|
||||
- 尽量深色、弱干扰,适合大屏 HUD 叠加
|
||||
|
||||
2. 新增天球球壳
|
||||
|
||||
新增模块:
|
||||
|
||||
- [frontend/public/earth/js/celestial.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/celestial.js)
|
||||
|
||||
建议接口:
|
||||
|
||||
```js
|
||||
export function initCelestialLayer(scene)
|
||||
export function updateCelestialLayer(date, camera, earth)
|
||||
export function disposeCelestialLayer()
|
||||
```
|
||||
|
||||
3. 实现一个大半径内翻球体
|
||||
|
||||
建议参数:
|
||||
|
||||
- 半径:`600 ~ 900`
|
||||
- 材质:`MeshBasicMaterial`
|
||||
- `side: THREE.BackSide`
|
||||
- 不受场景光照影响
|
||||
- 始终围绕场景中心
|
||||
|
||||
### 验收标准
|
||||
|
||||
- 初始加载后背景不再是随机星点
|
||||
- 旋转地球时,背景保持为稳定天球而不是跟地球一起转
|
||||
- 不明显干扰海缆/卫星/BGP 的前景识别
|
||||
|
||||
## Phase 2:太阳与月亮真实位置
|
||||
|
||||
### 目标
|
||||
|
||||
在当前 UTC 时间下,计算太阳与月亮在天球中的方向,并显示出来。
|
||||
|
||||
### 做法
|
||||
|
||||
1. 在 `celestial.js` 内封装天体位置计算
|
||||
|
||||
建议函数:
|
||||
|
||||
```js
|
||||
function getSunDirection(date)
|
||||
function getMoonDirection(date)
|
||||
```
|
||||
|
||||
输出统一为 world-space `THREE.Vector3`
|
||||
|
||||
2. 太阳显示
|
||||
|
||||
- 一个暖色发光 sprite
|
||||
- 比月亮更大、更亮
|
||||
- 可选添加柔和 halo
|
||||
|
||||
3. 月亮显示
|
||||
|
||||
- 一个较小 sprite 或 sphere
|
||||
- 灰白偏冷色
|
||||
- 后续 Phase 3 再做月相
|
||||
|
||||
4. 更新频率
|
||||
|
||||
不要每帧重新做完整天文计算,建议:
|
||||
|
||||
- 每 30 秒或 60 秒重算一次真实位置
|
||||
- 渲染帧内做平滑过渡
|
||||
|
||||
### 验收标准
|
||||
|
||||
- 页面可见太阳与月亮两个对象
|
||||
- 时间变化时位置会更新
|
||||
- 日月不会跟随地球局部旋转而错误附着
|
||||
|
||||
## Phase 3:太阳驱动地球受光
|
||||
|
||||
### 目标
|
||||
|
||||
让地球光照方向与太阳方向一致,不再使用写死的固定主光。
|
||||
|
||||
### 做法
|
||||
|
||||
1. 替换或接管当前主定向光
|
||||
|
||||
当前 [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js) 中 `addLights()` 里用了固定方向的 `DirectionalLight`。
|
||||
|
||||
建议改为:
|
||||
|
||||
- 保留环境补光
|
||||
- 主太阳光方向由 `sunDirection` 决定
|
||||
|
||||
2. 太阳光参数建议
|
||||
|
||||
- `DirectionalLight` 颜色偏暖白
|
||||
- 强度略高于当前主光
|
||||
- 保留一个弱背光作为氛围补偿,避免背面过死黑
|
||||
|
||||
3. 先不做物理级大气散射
|
||||
|
||||
第一版只要求:
|
||||
|
||||
- 亮面与暗面方向真实
|
||||
- 云层和大气仍保持当前风格
|
||||
|
||||
### 验收标准
|
||||
|
||||
- 地球明暗面会随太阳方向改变
|
||||
- 太阳 sprite 和地球亮面方向一致
|
||||
- 不破坏现有海缆、卫星、BGP 的可见性
|
||||
|
||||
## Phase 4:月相与天文细节增强
|
||||
|
||||
### 目标
|
||||
|
||||
在日月真实位置基础上增加更强的“天文可信度”。
|
||||
|
||||
### 可选项
|
||||
|
||||
1. 月相
|
||||
|
||||
- 根据日月夹角计算 illuminated fraction
|
||||
- 用月相纹理或 shader 表达盈亏
|
||||
|
||||
2. 赤道/黄道辅助线
|
||||
|
||||
- 可作为开发调试层,不默认显示
|
||||
|
||||
3. 太阳 terminator 增强
|
||||
|
||||
- 给地球夜面加入更自然的 night tint
|
||||
- 未来可叠加城市夜光纹理
|
||||
|
||||
4. 天文时间入口
|
||||
|
||||
- 设置中加入“当前时刻 / 指定时刻 / 加速时间”模式
|
||||
|
||||
### 验收标准
|
||||
|
||||
- 月亮不再只是一个静态圆点
|
||||
- 后续扩展行星或观测模式时无需推倒重来
|
||||
|
||||
## Phase 5:严格参考系升级(可选,不作为 V1 必做)
|
||||
|
||||
### 目标
|
||||
|
||||
把 Earth 从“用户旋转球体”升级为“真实地球姿态 + 用户观察姿态”的双层模型。
|
||||
|
||||
### 需要处理的问题
|
||||
|
||||
- 地球自转角与 UTC 的一致性
|
||||
- 赤道坐标系、地固坐标系、相机交互层分离
|
||||
- 卫星轨道显示与 Earth 旋转同步关系
|
||||
- resetView 和 autoRotate 的语义重定
|
||||
|
||||
### 风险
|
||||
|
||||
这一步会影响:
|
||||
|
||||
- [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js)
|
||||
- [frontend/public/earth/js/satellites.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/satellites.js)
|
||||
- [frontend/public/earth/js/cables.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/cables.js)
|
||||
- [frontend/public/earth/js/controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js)
|
||||
|
||||
因此不建议与 V1 同时推进。
|
||||
|
||||
## 代码改造清单
|
||||
|
||||
## 1. 新增文件
|
||||
|
||||
- [frontend/public/earth/js/celestial.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/celestial.js)
|
||||
|
||||
职责:
|
||||
|
||||
- 管理天球背景、太阳、月亮
|
||||
- 对外暴露 init/update/dispose
|
||||
|
||||
## 2. 修改 `constants.js`
|
||||
|
||||
文件:
|
||||
|
||||
- [frontend/public/earth/js/constants.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/constants.js)
|
||||
|
||||
新增:
|
||||
|
||||
```js
|
||||
export const CELESTIAL_CONFIG = {
|
||||
sphereRadius: 800,
|
||||
updateIntervalMs: 60000,
|
||||
sunSpriteScale: 28,
|
||||
moonSpriteScale: 16,
|
||||
sunLightIntensity: 1.25,
|
||||
ambientIntensity: 0.28,
|
||||
backLightIntensity: 0.18,
|
||||
};
|
||||
```
|
||||
|
||||
## 3. 修改 `main.js`
|
||||
|
||||
文件:
|
||||
|
||||
- [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js)
|
||||
|
||||
主要改动:
|
||||
|
||||
1. `init()` 中:
|
||||
- 初始化 celestial layer
|
||||
2. `addLights()` 中:
|
||||
- 把固定太阳光改成可更新的 celestial sun light
|
||||
3. `animate()` 中:
|
||||
- 每帧调 `updateCelestialLayer()`
|
||||
4. `destroy()` 中:
|
||||
- 清理 celestial 资源
|
||||
|
||||
## 4. 修改 `earth.js`
|
||||
|
||||
文件:
|
||||
|
||||
- [frontend/public/earth/js/earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js)
|
||||
|
||||
主要改动:
|
||||
|
||||
- `createStars()` 逐步退役
|
||||
- 第一阶段可先保留作为 fallback
|
||||
- 当真实星图加载成功后,不再显示随机星点
|
||||
|
||||
## 5. 新增资源
|
||||
|
||||
目录建议:
|
||||
|
||||
- `frontend/public/earth/assets/celestial/`
|
||||
|
||||
建议至少包含:
|
||||
|
||||
- `starmap_equatorial_4k.jpg`
|
||||
- `sun_glow.png`
|
||||
- `moon_albedo_2k.jpg`
|
||||
|
||||
## 数据流设计
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A["main.js:init()"] --> B["initCelestialLayer(scene)"]
|
||||
B --> C["创建天球球壳"]
|
||||
B --> D["创建太阳 sprite + 主定向光"]
|
||||
B --> E["创建月亮 sprite"]
|
||||
|
||||
F["animate()"] --> G["updateCelestialLayer(now, camera, earth)"]
|
||||
G --> H["Astronomy Engine 计算 Sun/Moon 方向"]
|
||||
H --> I["更新 sun sprite / moon sprite 位置"]
|
||||
H --> J["更新太阳 DirectionalLight 方向"]
|
||||
J --> K["地球昼夜方向变化"]
|
||||
```
|
||||
|
||||
## 风险与注意事项
|
||||
|
||||
### 1. 星图投影方向容易反
|
||||
|
||||
这会表现为:
|
||||
|
||||
- 星图左右镜像
|
||||
- 赤经方向颠倒
|
||||
- 日月位置和背景对不上
|
||||
|
||||
建议:
|
||||
|
||||
- 先做一个开发调试模式
|
||||
- 显示赤经/赤纬参考点,快速校正纹理朝向
|
||||
|
||||
### 2. 不要让天球跟随 Earth 旋转
|
||||
|
||||
天球背景和日月必须属于 scene/world,而不是 `earthObj`。
|
||||
|
||||
### 3. 不要每帧做重型天文计算
|
||||
|
||||
真实位置更新应节流,否则会浪费 CPU。
|
||||
|
||||
### 4. 月亮先求“方向正确”,再求“月相精致”
|
||||
|
||||
月相属于第二步优化,不应阻塞 V1 上线。
|
||||
|
||||
## 推荐实施顺序
|
||||
|
||||
1. 新建 `celestial.js`
|
||||
2. 用星图球壳替换随机星点
|
||||
3. 接入 Astronomy Engine
|
||||
4. 加太阳/月亮 sprite
|
||||
5. 用太阳方向驱动主光
|
||||
6. 再决定要不要做月相和更严格参考系
|
||||
|
||||
## 最终建议
|
||||
|
||||
对于当前 Planet Earth,最稳妥的方案是:
|
||||
|
||||
- 先做真实天球背景
|
||||
- 再做真实太阳/月亮方向
|
||||
- 再让太阳驱动地球受光
|
||||
- 暂时不做 Earth 参考系重构
|
||||
|
||||
这样可以在不破坏现有 Earth 交互和图层系统的前提下,显著提升空间感、真实感和演示说服力。
|
||||
98
docs/plans/earth-predicted-orbit-plan.md
Normal file
98
docs/plans/earth-predicted-orbit-plan.md
Normal file
@@ -0,0 +1,98 @@
|
||||
# Earth Predicted Orbit Plan
|
||||
|
||||
> Source note: this plan absorbs useful ideas from a sisyphus-created draft formerly stored at `.sisyphus/plans/predicted-orbit.md`.
|
||||
|
||||
## Goal
|
||||
|
||||
在 Earth 中锁定卫星时,显示“预测轨道”而不是只有历史尾迹:
|
||||
|
||||
- 从当前时刻开始
|
||||
- 绕地球一圈
|
||||
- 当前点最亮
|
||||
- 向后沿轨道逐步衰减
|
||||
|
||||
## Current State
|
||||
|
||||
当前已经有:
|
||||
|
||||
- 卫星历史轨迹
|
||||
- 锁定卫星
|
||||
- 轨道高亮与相关联动
|
||||
|
||||
但“预测轨道”仍然不是一套稳定、可验证的单独功能计划。
|
||||
|
||||
## Why It Is Valuable
|
||||
|
||||
预测轨道可以明显提升:
|
||||
|
||||
- 锁定卫星后的空间可读性
|
||||
- 轨道类型辨识
|
||||
- 演示解释力
|
||||
|
||||
相比短历史尾迹,预测轨道更符合用户对“这颗卫星接下来会怎么走”的预期。
|
||||
|
||||
## Scope
|
||||
|
||||
### Phase 1
|
||||
|
||||
- 锁定卫星时显示一整圈预测轨道
|
||||
- 解锁时隐藏
|
||||
- 不替代现有普通轨迹系统
|
||||
|
||||
### Phase 2
|
||||
|
||||
- 根据轨道类型调整采样率
|
||||
- GEO / MEO / LEO 不同密度
|
||||
- 进一步减少 fallback 轨迹的比例
|
||||
|
||||
## Implementation Direction
|
||||
|
||||
### 1. Orbit period
|
||||
|
||||
基于 `meanMotion` 估算轨道周期。
|
||||
|
||||
### 2. Predicted samples
|
||||
|
||||
以固定采样步长从 `now -> now + period` 推算轨迹点。
|
||||
|
||||
### 3. Render object lifecycle
|
||||
|
||||
预测轨道应是一个独立渲染对象:
|
||||
|
||||
- show
|
||||
- update
|
||||
- hide
|
||||
- dispose
|
||||
|
||||
### 4. Visual semantics
|
||||
|
||||
预测轨道不应与普通尾迹混淆:
|
||||
|
||||
- 更稳定
|
||||
- 更完整
|
||||
- 透明度沿轨道衰减
|
||||
- 当前点附近更亮
|
||||
|
||||
## Known Risks
|
||||
|
||||
### 1. TLE propagation gaps
|
||||
|
||||
部分卫星可能出现 SGP4 计算不足,需要 fallback。
|
||||
|
||||
### 2. Multiple orbit lines
|
||||
|
||||
必须确保:
|
||||
|
||||
- 锁定切换前先清旧轨道
|
||||
- 页面隐藏/销毁时清理
|
||||
|
||||
### 3. Performance
|
||||
|
||||
GEO 轨道点数高,采样率需要按轨道类型分层。
|
||||
|
||||
## Acceptance
|
||||
|
||||
1. 锁定单颗卫星时只显示一条预测轨道
|
||||
2. 解锁后轨道立即清除
|
||||
3. 不同轨道类型下点数可控
|
||||
4. 页面切换回来不会闪出旧轨道残留
|
||||
472
docs/plans/earth-real-terrain-plan.md
Normal file
472
docs/plans/earth-real-terrain-plan.md
Normal file
@@ -0,0 +1,472 @@
|
||||
# Earth Real Terrain Plan
|
||||
|
||||
## Goal
|
||||
|
||||
将 Earth 页当前的“程序噪声假地形”替换成基于真实 DEM 的可用地形层,使 `地形 terrain` 开关真正显示全球海拔起伏,而不是占位效果。
|
||||
|
||||
当前占位实现位于:
|
||||
|
||||
- [earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js)
|
||||
|
||||
具体问题:
|
||||
|
||||
- `createTerrain()` 直接对球体顶点应用 `simplex noise`
|
||||
- 没有真实海拔数据来源
|
||||
- 没有分辨率分层
|
||||
- 没有和当前相机/视角配套的性能控制
|
||||
|
||||
## Constraints
|
||||
|
||||
本计划必须贴合当前 Earth 架构,而不是引入一套全新的地形引擎:
|
||||
|
||||
- 地球主体仍然是一个 Three.js sphere
|
||||
- 海缆、登陆点、卫星、BGP 都已经建立在当前球体坐标系之上
|
||||
- 不能为了地形把整页改成 Cesium/MapLibre Globe 之类的全栈替换
|
||||
- 第一阶段优先做“真实可用”,不是一步到位做摄影测量级地形
|
||||
|
||||
## Recommended Data Source
|
||||
|
||||
### Primary recommendation
|
||||
|
||||
使用公开的 Terrarium 编码高程瓦片作为浏览器端高度来源,第一阶段优先接入:
|
||||
|
||||
- Mapzen/AWS `Terrarium` elevation tiles
|
||||
参考:[Mapzen terrain tile format / Terrarium](https://www.mapzen.com/blog/terrain-tile-service/)
|
||||
|
||||
原因:
|
||||
|
||||
- 已经是全球瓦片化高程
|
||||
- 浏览器端按 tile 请求,最适合当前 Earth 这种在线 globe
|
||||
- 编码简单稳定:
|
||||
- `heightMeters = (R * 256 + G + B / 256) - 32768`
|
||||
- 不需要我们先离线拼整球 DEM
|
||||
|
||||
### Data quality upgrade path
|
||||
|
||||
如果后面第一阶段效果确认可用,再逐步升级到底层源:
|
||||
|
||||
- Copernicus DEM GLO-30
|
||||
参考:[Copernicus DEM docs](https://documentation.dataspace.copernicus.eu/APIs/SentinelHub/Data/DEM.html)
|
||||
- 或用 Copernicus / SRTM / ASTER 等离线切成我们自己的 terrain tiles
|
||||
|
||||
这条升级路径适合第二阶段,不建议一开始就直接自建全球瓦片服务。
|
||||
|
||||
## Why Not Replace the Engine
|
||||
|
||||
不建议为了地形直接切到 Cesium terrain / quantized mesh 引擎,原因:
|
||||
|
||||
- 现有 Earth 业务对象都依附当前球面坐标
|
||||
- 切引擎会同时波及:
|
||||
- 海缆绘制
|
||||
- 卫星/轨迹
|
||||
- BGP 标记
|
||||
- HUD 与交互
|
||||
- 这是“重做一页”,不是“给地形层接真实数据”
|
||||
|
||||
所以推荐路线是:
|
||||
|
||||
- 保持当前 sphere globe
|
||||
- 为 sphere 增加真实高度位移层
|
||||
|
||||
## Implementation Strategy
|
||||
|
||||
分三期推进。
|
||||
|
||||
### Phase 1 — Global Heightmap Terrain Overlay
|
||||
|
||||
目标:
|
||||
|
||||
- 地形层切换后显示真实海拔起伏
|
||||
- 全球范围可用
|
||||
- 性能可控
|
||||
|
||||
做法:
|
||||
|
||||
1. 新增 terrain 数据模块
|
||||
|
||||
建议文件:
|
||||
|
||||
- `frontend/public/earth/js/terrain.js`
|
||||
|
||||
职责:
|
||||
|
||||
- 选择 DEM zoom level
|
||||
- 请求 Terrarium tiles
|
||||
- 解码 tile 高程
|
||||
- 将高程重采样到当前地形球体网格
|
||||
|
||||
2. 替换 `createTerrain()`
|
||||
|
||||
当前:
|
||||
|
||||
- 在 [earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js) 中同步生成噪声地形
|
||||
|
||||
调整后:
|
||||
|
||||
- `createTerrain()` 只负责创建 terrain mesh 骨架
|
||||
- 真正的顶点位移由 terrain 模块异步注入
|
||||
|
||||
3. 第一阶段采用“整球低分辨率位移”
|
||||
|
||||
不要一上来做动态 patch stitching。第一阶段更稳的办法是:
|
||||
|
||||
- 保留一张全球 terrain sphere
|
||||
- 使用较低分辨率几何
|
||||
- 例如 `SphereGeometry(radius, 192, 192)` 或 `256/256`
|
||||
- 运行时按一个固定地形 zoom(如 `z=4` 或 `z=5`)抓取覆盖全球的 Terrarium tiles
|
||||
- 将 tile 解码后重投影到经纬度采样网格
|
||||
- 将每个球面顶点按真实高度抬升
|
||||
|
||||
这样第一阶段就能做到:
|
||||
|
||||
- 有真实地形
|
||||
- 不需要复杂的局部 LOD
|
||||
- 不会让现有球体对象体系爆炸
|
||||
|
||||
### Phase 2 — View-Aware Refinement
|
||||
|
||||
目标:
|
||||
|
||||
- 正面可见区域更精细
|
||||
- 背面与远处维持低成本
|
||||
|
||||
做法:
|
||||
|
||||
- 引入“基础全球地形 + 当前视角高分局部补丁”
|
||||
- 正面区域额外抓更高 zoom 的高程 tile
|
||||
- 只替换局部顶点位移或局部 overlay mesh
|
||||
|
||||
这一阶段适合在第一阶段稳定后做。
|
||||
|
||||
### Phase 3 — Normals / Shading / Terrain UX
|
||||
|
||||
目标:
|
||||
|
||||
- 地形不仅有起伏,还更好看、更可读
|
||||
|
||||
包括:
|
||||
|
||||
- 根据高度生成更合理的 normals
|
||||
- 调整 terrain material,使山脉/高原更易读
|
||||
- 可选加入:
|
||||
- hillshade
|
||||
- contour lines
|
||||
- snowline / bathymetry tint
|
||||
|
||||
## Calibration Overlay Before More Terrain Tuning
|
||||
|
||||
在当前项目里,terrain 看起来“不像真地形”,不一定只是 DEM 或 exaggeration 不够,也可能是因为缺少稳定参照物。
|
||||
|
||||
没有清晰的海岸线、国界线和地表分层时,人眼很难判断:
|
||||
|
||||
- 山脉是不是在应该高的地方高
|
||||
- terrain 是否真的贴在正确的大陆位置上
|
||||
- 地球纹理、本初子午线、terrain 采样之间是否存在偏移
|
||||
|
||||
这里要明确区分两件事:
|
||||
|
||||
- 国界线不会修好错误的 terrain
|
||||
- 但海岸线 / 国界线会让我们更容易判断 terrain 有没有贴准
|
||||
|
||||
所以在继续盲调 terrain 参数之前,建议先插入一个“校准参照层”阶段。
|
||||
|
||||
### Recommended order for the calibration layer
|
||||
|
||||
1. 海岸线
|
||||
2. 国界线
|
||||
3. 再继续调 terrain
|
||||
|
||||
原因:
|
||||
|
||||
- 海岸线比国界线更基础,也更接近真实地表边界
|
||||
- 判断 terrain 是否贴准,最重要的是大陆边缘和山脉/海岸关系
|
||||
- 国界线更多是政治边界,只能作为辅助参照
|
||||
|
||||
如果只加国界线,不加海岸线,效果仍然可能会怪,因为:
|
||||
|
||||
- 很多国界线本来就是人为直线
|
||||
- 它们并不总是跟真实地形走
|
||||
|
||||
### Suggested layer order during debugging
|
||||
|
||||
建议调试期临时把地球层次明确成:
|
||||
|
||||
1. base earth texture
|
||||
2. coastline / borders overlay
|
||||
3. terrain relief
|
||||
4. cables / landing points / bgp / satellites
|
||||
|
||||
这样会比现在更容易判断:
|
||||
|
||||
- 山脉是否位于正确区域
|
||||
- terrain 是否和地表对齐
|
||||
- 国界/海岸是否漂移
|
||||
|
||||
### Suggested data source for the calibration overlay
|
||||
|
||||
优先用 `Natural Earth` 的轻量全球矢量数据:
|
||||
|
||||
- 海岸线(coastline)
|
||||
- Admin 0 国界线(country borders)
|
||||
|
||||
优点:
|
||||
|
||||
- 全球一致
|
||||
- 轻量
|
||||
- 很适合当前 Three.js globe 做 overlay
|
||||
|
||||
### Recommended execution path
|
||||
|
||||
#### Phase A — Add reference overlays
|
||||
|
||||
先加两层可开关的参考线:
|
||||
|
||||
- 海岸线
|
||||
- 国界线
|
||||
|
||||
这两层的目标不是最终美术表现,而是调试 / 校准。
|
||||
|
||||
#### Phase B — Recalibrate terrain against coastline
|
||||
|
||||
有了海岸线以后,再重新看 terrain:
|
||||
|
||||
- terrain 是否和大陆边缘错位
|
||||
- 地球纹理、本初子午线、terrain 采样之间是否有固定偏移
|
||||
|
||||
#### Phase C — Decide whether to keep the current terrain path
|
||||
|
||||
这时再决定后面的路线:
|
||||
|
||||
- 如果发现真实高程整体是对的,只是缺少 shading / readability
|
||||
继续保留当前 DEM + terrain overlay 路线
|
||||
- 如果发现整球采样投影、本初子午线或 overlay 关系本身就很别扭
|
||||
再考虑重做 terrain pipeline
|
||||
|
||||
### Practical recommendation
|
||||
|
||||
当前阶段不建议“从头开始重做 terrain”。
|
||||
|
||||
更稳的策略是:
|
||||
|
||||
- 暂停继续盲调 terrain 参数
|
||||
- 先补海岸线 / 国界线作为校准参照层
|
||||
- 再基于参照层判断 terrain 是“参数没调好”,还是“整条实现路径有偏移”
|
||||
|
||||
## Recommended Geometry Model
|
||||
|
||||
### First usable model
|
||||
|
||||
保留一层独立 terrain sphere:
|
||||
|
||||
- base earth sphere:贴纹理、昼夜、海洋
|
||||
- terrain sphere:略高于地球半径,真实高程位移
|
||||
|
||||
建议:
|
||||
|
||||
- `terrainBaseRadius = CONFIG.earthRadius + 0.2`
|
||||
- 高度缩放使用真实米制换算,再乘一个可调 exaggeration
|
||||
|
||||
示例关系:
|
||||
|
||||
- `heightWorld = (elevationMeters / 6371000) * CONFIG.earthRadius * exaggeration`
|
||||
|
||||
建议第一阶段 `exaggeration = 1.3 ~ 1.8`
|
||||
|
||||
因为完全真实比例在全球球体上会太平,看不出来。
|
||||
|
||||
## Tile Decoding Plan
|
||||
|
||||
### Terrarium decode
|
||||
|
||||
对于每个高程 tile 像素:
|
||||
|
||||
```text
|
||||
heightMeters = (R * 256 + G + B / 256) - 32768
|
||||
```
|
||||
|
||||
### Sampling path
|
||||
|
||||
对于 terrain mesh 上每个顶点:
|
||||
|
||||
1. 将顶点方向转成经纬度
|
||||
2. 将经纬度映射到 Web Mercator tile 坐标
|
||||
3. 找到对应的 tile 和像素
|
||||
4. 解码高程
|
||||
5. 将顶点沿法线方向抬升
|
||||
|
||||
### Needed helpers
|
||||
|
||||
建议新增:
|
||||
|
||||
- `latLonToTileXY(lat, lon, z)`
|
||||
- `tilePixelFromLatLon(lat, lon, z, tileSize)`
|
||||
- `decodeTerrariumHeight(r, g, b)`
|
||||
|
||||
## Caching Strategy
|
||||
|
||||
为了不让地形开关每次重开都重新抓全量 tile:
|
||||
|
||||
- terrain tile 按 `z/x/y` 存到内存缓存
|
||||
- terrain mesh 结果也缓存一份
|
||||
- 当用户关闭/开启 terrain:
|
||||
- 直接复用已有位移结果
|
||||
|
||||
建议:
|
||||
|
||||
- `Map<string, Float32Array | ImageBitmap>`
|
||||
|
||||
## Material Strategy
|
||||
|
||||
第一阶段不要复杂化。
|
||||
|
||||
建议 terrain material:
|
||||
|
||||
- 半透明低饱和地形色
|
||||
- 比 base earth 稍亮或稍偏冷
|
||||
- 保留当前 HUD 风格下的可读性
|
||||
|
||||
第一阶段不需要:
|
||||
|
||||
- 真实土地覆被纹理
|
||||
- 独立卫星影像贴 terrain
|
||||
|
||||
因为那会和现有地球纹理、云层、昼夜 shader 打架。
|
||||
|
||||
## Integration Points
|
||||
|
||||
### Files to change
|
||||
|
||||
- [earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js)
|
||||
- 重写 `createTerrain()`
|
||||
- 删除 simplex noise 占位逻辑
|
||||
- [main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js)
|
||||
- 初始化 terrain 数据加载
|
||||
- 控制 terrain readiness / loading message
|
||||
- [controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js)
|
||||
- `toggleTerrain` 逻辑保持,但应能区分:
|
||||
- mesh 已就绪
|
||||
- 正在加载
|
||||
- [constants.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/constants.js)
|
||||
- 新增 `TERRAIN_CONFIG`
|
||||
- 新文件:
|
||||
- `frontend/public/earth/js/terrain.js`
|
||||
|
||||
### Suggested new config
|
||||
|
||||
建议新增:
|
||||
|
||||
```js
|
||||
export const TERRAIN_CONFIG = {
|
||||
enabled: true,
|
||||
tileSize: 256,
|
||||
baseZoom: 4,
|
||||
baseRadiusOffset: 0.2,
|
||||
exaggeration: 1.5,
|
||||
opacity: 0.55,
|
||||
color: 0x6c876f,
|
||||
maxConcurrentRequests: 8,
|
||||
cacheEnabled: true,
|
||||
};
|
||||
```
|
||||
|
||||
## Loading UX
|
||||
|
||||
地形第一次开启时,不能像现在一样瞬时切换。
|
||||
|
||||
建议:
|
||||
|
||||
- 如果地形数据尚未准备:
|
||||
- 顶部状态条显示:`正在加载真实地形数据...`
|
||||
- 完成后:
|
||||
- `真实地形已就绪`
|
||||
|
||||
如果加载失败:
|
||||
|
||||
- 保留 base earth
|
||||
- 显示轻量错误提示
|
||||
- 不要让 terrain 开关卡死在“开”状态
|
||||
|
||||
## Risks
|
||||
|
||||
### 1. Global tile count too high
|
||||
|
||||
即使 `z=5` 全球 tile 数也不少。
|
||||
|
||||
缓解:
|
||||
|
||||
- 第一阶段限定低 zoom
|
||||
- 并发上限
|
||||
- 缓存
|
||||
|
||||
### 2. Mesh resolution too low
|
||||
|
||||
如果球面分段太低,山脉会被抹平。
|
||||
|
||||
缓解:
|
||||
|
||||
- 第一阶段先选一个中等分辨率
|
||||
- 用 exaggeration 保证可见性
|
||||
|
||||
### 3. Existing overlays may z-fight with terrain
|
||||
|
||||
海缆、登陆点、BGP、卫星相关对象都假设地球半径固定。
|
||||
|
||||
缓解:
|
||||
|
||||
- terrain sphere 单独作为 overlay
|
||||
- overlay 保持略低或略高的固定 offset
|
||||
- 必要时局部调整 landing point / cable altitude offset
|
||||
|
||||
### 4. Mercator sampling distortion near poles
|
||||
|
||||
Web Mercator 在高纬会有失真。
|
||||
|
||||
缓解:
|
||||
|
||||
- 第一阶段接受
|
||||
- 后续若需要更严格极区质量,再上 geodetic reprojection pipeline
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
第一阶段完成后,应满足:
|
||||
|
||||
1. `地形 terrain` 开关开启时,地表起伏明显不再是随机噪声
|
||||
2. 喜马拉雅、安第斯、落基山、东非高原等全球大尺度地形可辨认
|
||||
3. 关闭/重新开启 terrain 不重复全量请求
|
||||
4. 不破坏:
|
||||
- 海缆
|
||||
- 卫星
|
||||
- BGP
|
||||
- 地球昼夜
|
||||
- 天球层
|
||||
|
||||
## Suggested Execution Order
|
||||
|
||||
1. 引入 `TERRAIN_CONFIG`
|
||||
2. 新建 `terrain.js`
|
||||
3. 实现 Terrarium tile 请求与 decode
|
||||
4. 用低 zoom 全球 tile 构建真实 terrain sphere
|
||||
5. 接管 `toggleTerrain()`
|
||||
6. 调整 terrain material 和高度 exaggeration
|
||||
7. 做缓存
|
||||
8. 再考虑第二阶段局部高分 refinement
|
||||
|
||||
## Source References
|
||||
|
||||
- Mapzen Terrarium / AWS terrain tiles
|
||||
[Mapzen Terrain Tile Service](https://www.mapzen.com/blog/terrain-tile-service/)
|
||||
- Terrarium tile experiments / format background
|
||||
[mapzen/terrarium](https://github.com/mapzen/terrarium)
|
||||
- Copernicus DEM overview
|
||||
[Copernicus DEM docs](https://documentation.dataspace.copernicus.eu/APIs/SentinelHub/Data/DEM.html)
|
||||
|
||||
## Recommendation Summary
|
||||
|
||||
如果现在就要开始做,我建议直接按这条路线开工:
|
||||
|
||||
- 第一阶段接入 Terrarium 全球高程 tile
|
||||
- 替换掉当前 simplex 假地形
|
||||
- 先做一层真实可见的全球 terrain overlay
|
||||
- 等第一阶段稳定,再做视角高分 refinement
|
||||
|
||||
这是对当前项目风险最低、最贴合现有 Earth 架构的一条路。
|
||||
111
docs/plans/earth-renderer-architecture-separation-plan.md
Normal file
111
docs/plans/earth-renderer-architecture-separation-plan.md
Normal file
@@ -0,0 +1,111 @@
|
||||
# Earth Renderer / Logic Separation Plan
|
||||
|
||||
> Source note: this plan absorbs useful ideas from a sisyphus-created draft formerly stored at `.sisyphus/plans/earth-architecture-refactor.md`.
|
||||
|
||||
## Goal
|
||||
|
||||
将 Earth 前端继续往“逻辑层 / 状态层 / 渲染层”分离推进,降低后续这几类工作的耦合成本:
|
||||
|
||||
- Three.js 渲染重构
|
||||
- 部分图层替换实现
|
||||
- 未来 UE / Cesium 客户端迁移
|
||||
- Earth 行为逻辑复用
|
||||
|
||||
## Why This Matters
|
||||
|
||||
当前 Earth 已经有一些良好分层,例如:
|
||||
|
||||
- 图层显隐入口
|
||||
- Cable state 枚举与状态 map
|
||||
- 交互逻辑与实际视觉效果的部分分离
|
||||
|
||||
但还没有形成一套更明确的统一规则。现在的风险是:
|
||||
|
||||
- 同一类对象的 hover / locked / hidden / loading 语义不一致
|
||||
- 状态和渲染更新散落在多个模块
|
||||
- 后续再加新图层时容易复制旧逻辑
|
||||
|
||||
## Target Architecture
|
||||
|
||||
Earth 对每类对象都尽量拆成三层:
|
||||
|
||||
1. `state layer`
|
||||
- 保存对象状态
|
||||
- 例如:`normal / hovered / locked / hidden / loading`
|
||||
|
||||
2. `logic layer`
|
||||
- 处理点击、悬停、锁定、过滤、显隐切换
|
||||
- 不直接关心 Three.js 具体材质怎么改
|
||||
|
||||
3. `renderer layer`
|
||||
- 根据状态更新 Three.js / HUD 外观
|
||||
- 是最容易针对不同渲染引擎替换的一层
|
||||
|
||||
## Current Good Signals
|
||||
|
||||
当前已经接近这条方向的地方:
|
||||
|
||||
- cable 状态管理
|
||||
- 部分 landing point 状态同步
|
||||
- layer button 的统一状态入口
|
||||
- tooltip / legend / info-card 开始朝状态驱动靠拢
|
||||
|
||||
## Next Steps
|
||||
|
||||
### 1. Standardize object state enums
|
||||
|
||||
优先为这些对象建立更稳定的状态语义:
|
||||
|
||||
- cables
|
||||
- satellites
|
||||
- landing points
|
||||
- BGP markers
|
||||
- media / news 面板入口按钮
|
||||
|
||||
### 2. Unify state-to-visual adapters
|
||||
|
||||
为各模块建立更清晰的渲染适配函数,例如:
|
||||
|
||||
- `applyCableVisualState()`
|
||||
- `applySatelliteVisualState()`
|
||||
- `applyBGPVisualState()`
|
||||
|
||||
要求:
|
||||
|
||||
- 逻辑层只改状态
|
||||
- 视觉层负责把状态映射到材质、透明度、发光、尺寸、文字
|
||||
|
||||
### 3. Separate Earth UI state from render state
|
||||
|
||||
HUD / 面板 / 图层按钮状态也需要和渲染状态分离:
|
||||
|
||||
- `loading`
|
||||
- `active`
|
||||
- `locked`
|
||||
- `hidden`
|
||||
- `error`
|
||||
|
||||
不要再让 UI 通过“猜渲染结果”推导业务状态。
|
||||
|
||||
### 4. Prepare migration-safe boundaries
|
||||
|
||||
后续如果做 UE / Cesium 客户端,尽量保留:
|
||||
|
||||
- 状态枚举
|
||||
- 交互规则
|
||||
- 数据层接口
|
||||
|
||||
只替换:
|
||||
|
||||
- Three.js 具体渲染实现
|
||||
- HUD 展示实现
|
||||
|
||||
## Practical Rule
|
||||
|
||||
后续 Earth 新功能开发时,优先问三个问题:
|
||||
|
||||
1. 这个状态由谁持有?
|
||||
2. 这个交互逻辑在哪一层处理?
|
||||
3. 这个视觉变化是否能在不改逻辑的情况下单独替换?
|
||||
|
||||
如果答不上来,就说明还在把状态、逻辑、渲染揉在一起。
|
||||
82
docs/plans/earth-webgl-instancing-satellites-plan.md
Normal file
82
docs/plans/earth-webgl-instancing-satellites-plan.md
Normal file
@@ -0,0 +1,82 @@
|
||||
# Earth WebGL Instancing Satellites Plan
|
||||
|
||||
> Source note: this plan absorbs useful ideas from a sisyphus-created draft formerly stored at `.sisyphus/plans/webgl-instancing-satellites.md`.
|
||||
|
||||
## Goal
|
||||
|
||||
把 Earth 卫星渲染从当前方案继续推进到更适合高数量卫星的 instancing 方向,目标是:
|
||||
|
||||
- 支持更多卫星
|
||||
- 降低渲染压力
|
||||
- 仍然保留当前数据层和交互层
|
||||
|
||||
## Why It Matters
|
||||
|
||||
当前卫星系统已经具备:
|
||||
|
||||
- 数据加载
|
||||
- 轨迹
|
||||
- 选择/锁定
|
||||
- 图例
|
||||
- 相关区域联动
|
||||
|
||||
但当卫星数量持续增加时,渲染层会越来越接近瓶颈。
|
||||
|
||||
## Recommended Direction
|
||||
|
||||
优先调研并原型验证:
|
||||
|
||||
- `InstancedBufferGeometry + custom shader`
|
||||
|
||||
而不是一开始就推倒重写成 raw WebGL。
|
||||
|
||||
原因:
|
||||
|
||||
- 仍能保留 Three.js 主架构
|
||||
- 更容易渐进迁移
|
||||
- 比继续堆普通点渲染更有上限
|
||||
|
||||
## What Should Stay
|
||||
|
||||
尽量保留这些层:
|
||||
|
||||
- 卫星数据获取
|
||||
- 位置计算
|
||||
- 锁定/悬停逻辑
|
||||
- legend / info-card / 相关联动
|
||||
|
||||
主要替换的是:
|
||||
|
||||
- 卫星点渲染实现
|
||||
- 颜色/大小等实例属性更新方式
|
||||
|
||||
## Phases
|
||||
|
||||
### Phase 1: Prototype
|
||||
|
||||
- 用 instancing 做最小原型
|
||||
- 先只渲染卫星点
|
||||
- 不碰轨迹系统
|
||||
|
||||
### Phase 2: Integrate
|
||||
|
||||
- 接入当前 `satellites.js` 数据层
|
||||
- 保留当前选择和高亮语义
|
||||
|
||||
### Phase 3: Tune
|
||||
|
||||
- 调整可视大小
|
||||
- 调整选中高亮方式
|
||||
- 评估是否需要分层 LOD
|
||||
|
||||
## Risks
|
||||
|
||||
1. 透明度排序更复杂
|
||||
2. Shader 调试成本更高
|
||||
3. 选中态和 hover 态不能简单复用旧材质逻辑
|
||||
|
||||
## Acceptance
|
||||
|
||||
1. 在更高卫星数量下保持可接受帧率
|
||||
2. 不破坏现有锁定/高亮语义
|
||||
3. 图例、信息卡、相关卫星联动仍然成立
|
||||
361
docs/plans/frontend-ai-playground-development-plan.md
Normal file
361
docs/plans/frontend-ai-playground-development-plan.md
Normal file
@@ -0,0 +1,361 @@
|
||||
# AI Playground Development Plan
|
||||
|
||||
## 目标
|
||||
|
||||
这份计划用于统一 `aiprovider`、`backend AI facade`、`Playground` 页面,以及后续 `BGP / 告警 / 数据源健康` 等 AI 入口的演进方向。
|
||||
|
||||
当前原则:
|
||||
|
||||
- `aiprovider` 继续作为独立模型网关
|
||||
- `backend` 继续作为稳定业务入口
|
||||
- `frontend` 负责测试台和业务 UI
|
||||
- 先做“可控、可验证、可解释”的 AI 能力,再逐步引入 agent/tool calling
|
||||
|
||||
## 当前已完成
|
||||
|
||||
### 1. AI 网关基础层
|
||||
|
||||
已完成:
|
||||
|
||||
- 独立 `aiprovider` 服务
|
||||
- `backend -> aiprovider -> model provider` 调用链
|
||||
- `provider/status` 与 `situational-awareness/analyze` 稳定接口
|
||||
- `X-Request-ID` 透传
|
||||
- 轻量超时与重试
|
||||
- MiniMax / Anthropic-compatible / OpenAI-compatible / Ollama 适配
|
||||
|
||||
相关文件:
|
||||
|
||||
- [backend/app/api/v1/ai.py](/home/ray/dev/linkong/planet/backend/app/api/v1/ai.py)
|
||||
- [backend/app/services/ai_client.py](/home/ray/dev/linkong/planet/backend/app/services/ai_client.py)
|
||||
- [aiprovider/main.py](/home/ray/dev/linkong/planet/aiprovider/main.py)
|
||||
- [aiprovider/provider_service.py](/home/ray/dev/linkong/planet/aiprovider/provider_service.py)
|
||||
- [docs/technical/agents-aiprovider.md](/home/ray/dev/linkong/planet/docs/technical/agents-aiprovider.md)
|
||||
|
||||
### 2. 本地运行与配置打通
|
||||
|
||||
已完成:
|
||||
|
||||
- `planet.sh` 启动链路纳入 `aiprovider`
|
||||
- `planet.sh` 启动完成后输出 Playground 入口
|
||||
- `docker-compose.yml` 为 `aiprovider` 加入 `env_file`
|
||||
- `backend/.env` 与 `aiprovider/.env` 两侧 service token 对齐
|
||||
- `Playground` 状态缓存,避免页面切换时每次都重新请求 provider 状态
|
||||
|
||||
相关文件:
|
||||
|
||||
- [planet.sh](/home/ray/dev/linkong/planet/planet.sh)
|
||||
- [docker-compose.yml](/home/ray/dev/linkong/planet/docker-compose.yml)
|
||||
- [backend/.env.example](/home/ray/dev/linkong/planet/backend/.env.example)
|
||||
- [aiprovider/.env.example](/home/ray/dev/linkong/planet/aiprovider/.env.example)
|
||||
|
||||
### 3. Playground UI 基础版
|
||||
|
||||
已完成:
|
||||
|
||||
- 新增前端路由 `/playground`
|
||||
- 左侧 `Provider 状态 + 测试说明`
|
||||
- 右侧 `请求 / 结果` Tabs
|
||||
- `Provider 状态` 支持手动刷新
|
||||
- `测试说明` 支持折叠
|
||||
- 内部区域采用细滚动条
|
||||
- 页面布局开始遵循“单屏工作区 + 模块内部滚动”规范
|
||||
|
||||
相关文件:
|
||||
|
||||
- [frontend/src/pages/Playground/Playground.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Playground/Playground.tsx)
|
||||
- [frontend/src/App.tsx](/home/ray/dev/linkong/planet/frontend/src/App.tsx)
|
||||
- [frontend/src/components/AppLayout/AppLayout.tsx](/home/ray/dev/linkong/planet/frontend/src/components/AppLayout/AppLayout.tsx)
|
||||
- [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css)
|
||||
|
||||
### 4. 前端布局规范沉淀
|
||||
|
||||
已完成:
|
||||
|
||||
- 把“一屏工作区、主模块优先、模块内部滚动”的规范文档化
|
||||
- 明确 `BGP` 页面为当前参考实现
|
||||
|
||||
相关文件:
|
||||
|
||||
- [docs/technical/frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md)
|
||||
- [frontend/src/pages/BGP/BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx)
|
||||
|
||||
## 当前限制
|
||||
|
||||
### 1. Playground 还是 prompt playground,不是 agent playground
|
||||
|
||||
当前 `Playground` 的 `观察项 / 目标 / 约束条件` 都是人工输入。
|
||||
|
||||
模型现在拿到的是:
|
||||
|
||||
- 你手工输入的结构化字段
|
||||
- 后端传递的少量静态上下文
|
||||
|
||||
模型现在拿不到:
|
||||
|
||||
- 实时 BGP 事件
|
||||
- 真实告警列表
|
||||
- 数据源健康状态
|
||||
- 自动检索结果
|
||||
- tool calling / skills / 自主取数
|
||||
|
||||
### 2. `situational-awareness/analyze` 还是通用提示词接口
|
||||
|
||||
当前更适合:
|
||||
|
||||
- 测试链路
|
||||
- 测试模型输出风格
|
||||
- 验证不同 provider 是否正常返回
|
||||
|
||||
当前还不适合:
|
||||
|
||||
- 直接当真实态势系统主入口
|
||||
- 让用户手工维护长期分析模板
|
||||
- 代替专用业务研判接口
|
||||
|
||||
### 3. 还没有可验证的真实业务输入注入
|
||||
|
||||
目前最缺的是:
|
||||
|
||||
- 从业务系统自动整理“事实输入”
|
||||
- 再把这些事实喂给 AI
|
||||
|
||||
而不是继续让用户在 Playground 手工输入真实事件摘要。
|
||||
|
||||
## 短期计划
|
||||
|
||||
### Phase A: Playground 收敛为稳定测试台
|
||||
|
||||
目标:
|
||||
|
||||
- 保持 Playground 简洁可用
|
||||
- 不再继续堆“高级参数”
|
||||
|
||||
工作项:
|
||||
|
||||
- 继续微调左侧 `Provider 状态` 与 `测试说明` 的空间策略
|
||||
- 保持 `请求 / 结果` 为单一主工作区
|
||||
- 不引入盲填式高级字段
|
||||
- 统一滚动条、卡片、溢出行为
|
||||
|
||||
完成标准:
|
||||
|
||||
- 笔记本视口下依然可用
|
||||
- 各模块标题可见
|
||||
- 主要阅读区始终是右侧 Tabs
|
||||
|
||||
### Phase B: BGP AI 简报
|
||||
|
||||
目标:
|
||||
|
||||
- 不再依赖手工填写“观察项”
|
||||
- 让系统自动把真实 BGP 数据注入 AI
|
||||
- 让 BGP 页面逐步从“摘要汇总”升级为“证据驱动的区域态势分析”
|
||||
|
||||
建议实现:
|
||||
|
||||
- 新增专用后端接口,例如:
|
||||
- `POST /api/v1/ai/bgp/brief`
|
||||
- 后端自动读取:
|
||||
- incidents summary
|
||||
- anomalies
|
||||
- recent events
|
||||
- collector coverage summary
|
||||
- 后端将结构化事实注入 `context / observations`
|
||||
- 前端在 BGP 页面增加“生成 AI 简报”
|
||||
|
||||
当前阶段说明:
|
||||
|
||||
- 第一版 `BGP AI 简报` 允许先落地为“值班摘要生成器”
|
||||
- 也就是先把 incidents / anomalies / events / collector coverage 自动注入
|
||||
- 允许模型先做事实摘要、风险归纳、建议动作
|
||||
|
||||
但这不应被视为 Phase B 的最终形态。
|
||||
|
||||
Phase B 后续还需要补齐:
|
||||
|
||||
- prefix geography 证据注入
|
||||
- `iptoasn`
|
||||
- `opengeofeed`
|
||||
- `nro_delegated`
|
||||
- 基于 `affected_regions` 与 prefix geography 的区域聚合
|
||||
- 区分“真实区域热度”与“collector coverage 偏差”
|
||||
- 对高风险 prefix / ASN 给出更明确的国家、城市、运营商归属线索
|
||||
- 让 AI 输出明确回答:
|
||||
- 哪些区域正在异常升温
|
||||
- 哪些结论只是观测站偏差
|
||||
- 当前还缺哪些区域证据
|
||||
|
||||
完成标准:
|
||||
|
||||
- 用户不需要手工录入 BGP 观察项
|
||||
- AI 输出能明确区分“事实”和“研判”
|
||||
- AI 不只是复述总量和最近几条事件,还能利用 prefix geography 与 affected regions 做区域态势判断
|
||||
- 输出中能明确指出:
|
||||
- 高风险区域
|
||||
- 区域证据来源
|
||||
- collector coverage 偏差对判断的影响
|
||||
|
||||
### Phase C: 告警 / 数据源健康 AI 简报
|
||||
|
||||
目标:
|
||||
|
||||
- 复用同样模式,扩展到其他模块
|
||||
|
||||
建议入口:
|
||||
|
||||
- `Alerts` 页面:异常与告警摘要
|
||||
- `DataSources` 页面:采集失败与健康状态总结
|
||||
|
||||
原则:
|
||||
|
||||
- 每个业务页优先做“专用 AI 简报”
|
||||
- 不优先做“万能大聊天框”
|
||||
|
||||
## 中期计划
|
||||
|
||||
### 1. Assessment Layer
|
||||
|
||||
目标:
|
||||
|
||||
- 不只返回自由文本
|
||||
- 返回结构化的 assessment
|
||||
|
||||
建议输出字段:
|
||||
|
||||
- summary
|
||||
- key_risks
|
||||
- evidence
|
||||
- recommendations
|
||||
- confidence
|
||||
- missing_data
|
||||
|
||||
这样后续才能:
|
||||
|
||||
- 持久化
|
||||
- 回看
|
||||
- 对比不同时间的 AI 结论
|
||||
- 在 Earth / Dashboard / BGP 页面稳定展示
|
||||
|
||||
### 2. Evidence-first Runtime
|
||||
|
||||
目标:
|
||||
|
||||
- 所有 AI 分析先取真实数据,再调模型
|
||||
|
||||
原则:
|
||||
|
||||
- 先 evidence
|
||||
- 再 prompt
|
||||
- 最后才是自由生成
|
||||
|
||||
优先要做的不是更强聊天,而是:
|
||||
|
||||
- 更稳定的数据注入
|
||||
- 更一致的事实模板
|
||||
- 更清晰的结果结构
|
||||
|
||||
### 3. 按页面提供专用入口
|
||||
|
||||
目标:
|
||||
|
||||
- 让 AI 成为业务视图的一部分,而不是孤立 playground
|
||||
|
||||
优先顺序建议:
|
||||
|
||||
1. `BGP` AI 简报
|
||||
2. `Alerts` AI 简报
|
||||
3. `DataSources` 健康研判
|
||||
4. `Dashboard` 总览总结
|
||||
|
||||
## 长期计划
|
||||
|
||||
### 1. Tool Calling / Agent Runtime
|
||||
|
||||
只有在以下基础稳定后再推进:
|
||||
|
||||
- 数据源健康信号稳定
|
||||
- BGP / Alerts / Datasource evidence 注入稳定
|
||||
- assessment 结构稳定
|
||||
|
||||
长期可做能力:
|
||||
|
||||
- AI 调用受控工具查询业务数据
|
||||
- AI 调用检索/web search 做外部验证
|
||||
- AI 生成建议而不是直接修改系统
|
||||
- 审核后触发受控动作
|
||||
|
||||
### 2. 受控动作与闭环
|
||||
|
||||
潜在方向:
|
||||
|
||||
- 根据健康异常生成修复建议
|
||||
- 根据态势变化生成处理建议
|
||||
- 进入 review queue
|
||||
- 审批后执行
|
||||
- 验证结果并形成闭环
|
||||
|
||||
### 3. 多模块统一 AI 体验
|
||||
|
||||
长期目标不是一个孤立 Playground,而是:
|
||||
|
||||
- 每个业务页都有自己的 AI 入口
|
||||
- 共享统一的 backend AI facade
|
||||
- 共享统一的 assessment 结构
|
||||
- 共享统一的 evidence 注入与审计链路
|
||||
|
||||
## 设计决策总结
|
||||
|
||||
### 为什么保留 `aiprovider`
|
||||
|
||||
因为它已经很好地承担了:
|
||||
|
||||
- provider 适配
|
||||
- 协议兼容
|
||||
- service token 边界
|
||||
- 独立重启与部署
|
||||
|
||||
因此短期内不建议把它并回 `backend`。
|
||||
|
||||
### 为什么 Playground 不做成万能聊天页
|
||||
|
||||
因为当前更需要的是:
|
||||
|
||||
- 稳定测试链路
|
||||
- 可验证业务输入
|
||||
- 专用分析入口
|
||||
|
||||
而不是一个泛化但没有真实数据支撑的聊天框。
|
||||
|
||||
### 为什么优先做专用 AI 简报
|
||||
|
||||
因为:
|
||||
|
||||
- 数据可以自动注入
|
||||
- 用户心智更清晰
|
||||
- 输出更容易结构化
|
||||
- 更容易校验事实与研判是否一致
|
||||
|
||||
## 下一步建议
|
||||
|
||||
按优先级建议接下来这样做:
|
||||
|
||||
1. 稳住 `Playground` 当前布局,不再大幅重做
|
||||
2. 在 `BGP` 页面新增专用 “AI 简报” 入口
|
||||
3. 后端新增 `BGP brief` 专用接口,自动注入真实数据
|
||||
4. 补齐 `BGP brief` 的区域态势证据层
|
||||
5. 把 AI 输出逐步从自由文本升级为结构化 assessment
|
||||
|
||||
### BGP Brief 后续子项
|
||||
|
||||
为避免把“已有 AI 简报”误判成“区域分析已完成”,这里单独记录 `BGP brief` 的后续 backlog:
|
||||
|
||||
1. 把高风险 prefix 命中的 `iptoasn / opengeofeed / nro_delegated` 结果注入 brief context
|
||||
2. 按国家/城市聚合 active incidents、anomalies、affected prefixes,生成区域热点事实层
|
||||
3. 把 collector coverage 与区域热点并排注入,避免模型把观测偏差误判成区域风险
|
||||
4. 对高风险 ASN / prefix 追加归属线索,如国家、城市、可能运营商或注册区域
|
||||
5. 在输出结构中单独增加:
|
||||
- 区域态势
|
||||
- 证据来源
|
||||
- 观测偏差说明
|
||||
- 缺失区域证据
|
||||
1015
docs/plans/ue5-mvp-fused-plan.md
Normal file
1015
docs/plans/ue5-mvp-fused-plan.md
Normal file
File diff suppressed because it is too large
Load Diff
26
docs/technical/README.md
Normal file
26
docs/technical/README.md
Normal file
@@ -0,0 +1,26 @@
|
||||
# Technical Docs
|
||||
|
||||
这里放“当前实现和当前结构”的文档,重点回答:
|
||||
|
||||
- 现在代码是怎么组织的
|
||||
- 当前入口在哪
|
||||
- 状态和组件如何工作
|
||||
- 后续改动应该沿着哪条实现边界继续走
|
||||
|
||||
适合放入这里的内容:
|
||||
|
||||
- 前端上下文
|
||||
- Earth 前端结构
|
||||
- 后端运行控制
|
||||
- collector 现状
|
||||
- 采集格式约定
|
||||
|
||||
不适合放入这里的内容:
|
||||
|
||||
- 尚未完成的 roadmap
|
||||
- 未来迭代方案
|
||||
- 大范围重构计划
|
||||
|
||||
这些应放入:
|
||||
|
||||
- [docs/plans/README.md](/home/ray/dev/linkong/planet/docs/plans/README.md)
|
||||
@@ -31,22 +31,40 @@ The recommended default is:
|
||||
- timeout and lightweight retry
|
||||
- request tracing via `X-Request-ID`
|
||||
|
||||
This now follows an OpenClaw-like seam:
|
||||
|
||||
- `AI_PROVIDER` identifies the vendor or logical provider
|
||||
- `AI_PROVIDER_API` identifies the wire adapter
|
||||
|
||||
That split makes MiniMax, Claude-compatible gateways, and self-hosted OpenAI-compatible services easier to model without overloading one config field.
|
||||
|
||||
## Supported Providers
|
||||
|
||||
`aiprovider` currently supports:
|
||||
`aiprovider` currently supports these provider identities:
|
||||
|
||||
- `openai`
|
||||
- `openai_compatible`
|
||||
- `anthropic`
|
||||
- `minimax`
|
||||
- `ollama`
|
||||
|
||||
Supported request adapters:
|
||||
|
||||
- `openai-completions`
|
||||
- `anthropic-messages`
|
||||
- `ollama-generate`
|
||||
|
||||
Backward-compatible aliases still accepted:
|
||||
|
||||
- `openai_compatible`
|
||||
- `anthropic_compatible`
|
||||
- `claude_compatible`
|
||||
- `ollama`
|
||||
|
||||
Provider mapping:
|
||||
|
||||
- `vLLM`, `LM Studio`, `One API`: `openai_compatible`
|
||||
- `MiniMax`, Claude-compatible gateways: `claude_compatible`
|
||||
- `Ollama`: `ollama`
|
||||
- `vLLM`, `LM Studio`, `One API`: `AI_PROVIDER=openai`, `AI_PROVIDER_API=openai-completions`
|
||||
- `MiniMax`: `AI_PROVIDER=minimax`, `AI_PROVIDER_API=anthropic-messages`
|
||||
- Claude-compatible gateways: `AI_PROVIDER=anthropic`, `AI_PROVIDER_API=anthropic-messages`
|
||||
- `Ollama`: `AI_PROVIDER=ollama`, `AI_PROVIDER_API=ollama-generate`
|
||||
|
||||
## API Surfaces
|
||||
|
||||
@@ -137,9 +155,13 @@ Both backend and `aiprovider` return the same payload shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"provider": "openai_compatible",
|
||||
"model": "gpt-4o-mini",
|
||||
"provider": "minimax",
|
||||
"api": "anthropic-messages",
|
||||
"model": "MiniMax-M2.7",
|
||||
"content": "1) 态势摘要 ...",
|
||||
"content_blocks": [],
|
||||
"text_blocks": [],
|
||||
"thinking_blocks": [],
|
||||
"raw_response": {}
|
||||
}
|
||||
```
|
||||
@@ -189,17 +211,37 @@ AI_ANALYSIS_SYSTEM_PROMPT=你是态势感知分析助手。请基于输入的上
|
||||
### OpenAI-compatible example
|
||||
|
||||
```env
|
||||
AI_PROVIDER=openai_compatible
|
||||
AI_PROVIDER=openai
|
||||
AI_PROVIDER_API=openai-completions
|
||||
AI_BASE_URL=http://127.0.0.1:8001/v1
|
||||
AI_API_KEY=local-key
|
||||
AI_MODEL=your-local-model
|
||||
```
|
||||
|
||||
### Claude-compatible example
|
||||
### MiniMax CN example
|
||||
|
||||
```env
|
||||
AI_PROVIDER=claude_compatible
|
||||
AI_BASE_URL=https://your-claude-compatible-endpoint.example.com
|
||||
AI_PROVIDER=minimax
|
||||
AI_PROVIDER_API=anthropic-messages
|
||||
AI_BASE_URL=https://api.minimaxi.com/anthropic
|
||||
AI_API_KEY=sk-cp-xxxxx
|
||||
AI_MODEL=MiniMax-M2.7
|
||||
AI_MAX_TOKENS=1200
|
||||
AI_ANTHROPIC_VERSION=2023-06-01
|
||||
```
|
||||
|
||||
MiniMax note:
|
||||
|
||||
- This follows the same Anthropic Messages request shape as the official MiniMax examples.
|
||||
- For MiniMax, `aiprovider` now disables `thinking` by default unless the caller explicitly passes a `thinking` object.
|
||||
- This mirrors OpenClaw's caution around MiniMax Anthropic-compatible behavior.
|
||||
|
||||
### Anthropic-compatible example
|
||||
|
||||
```env
|
||||
AI_PROVIDER=anthropic
|
||||
AI_PROVIDER_API=anthropic-messages
|
||||
AI_BASE_URL=https://your-claude-compatible-endpoint.example.com/anthropic
|
||||
AI_API_KEY=your_api_key
|
||||
AI_MODEL=your-model
|
||||
AI_MAX_TOKENS=1200
|
||||
@@ -210,6 +252,7 @@ AI_ANTHROPIC_VERSION=2023-06-01
|
||||
|
||||
```env
|
||||
AI_PROVIDER=ollama
|
||||
AI_PROVIDER_API=ollama-generate
|
||||
AI_BASE_URL=http://127.0.0.1:11434
|
||||
AI_API_KEY=
|
||||
AI_MODEL=qwen2.5:7b
|
||||
@@ -187,7 +187,7 @@ Current reality:
|
||||
- that is expected, because incidents are aggregated and de-noised
|
||||
- but incident-first rendering makes the Earth view look too quiet unless there is another always-available activity layer
|
||||
|
||||
Implementation detail for the recommended `activity layer` is expanded in [bgp-region-aggregation-plan.md](/home/ray/dev/linkong/planet/docs/bgp-region-aggregation-plan.md).
|
||||
Implementation detail for the recommended `activity layer` is expanded in [bgp-region-aggregation-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-bgp-region-aggregation-plan.md).
|
||||
|
||||
So the immediate next milestone is:
|
||||
|
||||
298
docs/technical/earth-frontend-context.md
Normal file
298
docs/technical/earth-frontend-context.md
Normal file
@@ -0,0 +1,298 @@
|
||||
# Earth Frontend Context
|
||||
|
||||
本文件描述当前 Earth 大屏前端的真实结构,重点是帮助后续继续改 HUD、图层、媒体面板、真实地形、BGP 可视化时,不再重复踩结构和状态同步上的坑。
|
||||
|
||||
相关规则建议一起参考:
|
||||
|
||||
- [rules.md](/home/ray/dev/linkong/planet/rules.md)
|
||||
- [frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md)
|
||||
|
||||
## 当前目标
|
||||
|
||||
Earth 前端不是普通管理页,它是独立的大屏展示前端。当前产品目标是:
|
||||
|
||||
- 维持地球视图的空间感和可读性
|
||||
- 让 HUD、图层、媒体面板、BGP、卫星、海缆等保持统一交互
|
||||
- 把加载中、已启用、已隐藏、锁定中这类状态做清楚
|
||||
|
||||
## 当前入口
|
||||
|
||||
React 路由入口:
|
||||
|
||||
- [Earth.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Earth/Earth.tsx)
|
||||
|
||||
当前做法很简单:
|
||||
|
||||
- React 页面只负责提供一个全屏 `iframe`
|
||||
- 真正的 Earth 应用运行在:
|
||||
- [index.html](/home/ray/dev/linkong/planet/frontend/public/earth/index.html)
|
||||
|
||||
所以 Earth 前端本质上是 `public/earth` 下的一套独立静态应用。
|
||||
|
||||
## 当前文件分层
|
||||
|
||||
### 1. 页面入口与结构
|
||||
|
||||
- [index.html](/home/ray/dev/linkong/planet/frontend/public/earth/index.html)
|
||||
|
||||
职责:
|
||||
|
||||
- HUD 基础 DOM
|
||||
- 图层面板
|
||||
- 媒体面板
|
||||
- 工具栏
|
||||
- 设置弹窗
|
||||
- 兼容旧元素 id
|
||||
|
||||
### 2. 主运行时
|
||||
|
||||
- [main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js)
|
||||
|
||||
职责:
|
||||
|
||||
- 地球初始化
|
||||
- Three.js 场景组装
|
||||
- 数据加载与刷新
|
||||
- 各图层集成
|
||||
- Earth 级别状态同步
|
||||
|
||||
### 3. 地球控制层
|
||||
|
||||
- [controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js)
|
||||
|
||||
职责:
|
||||
|
||||
- 工具栏交互
|
||||
- 图层面板交互
|
||||
- 旋转/缩放/布局
|
||||
- HUD 面板拖拽
|
||||
- 图层开关状态机
|
||||
|
||||
这份文件是 Earth 前端当前最核心的 UI 控制入口。
|
||||
|
||||
### 4. UI 与状态消息
|
||||
|
||||
- [ui.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/ui.js)
|
||||
|
||||
职责:
|
||||
|
||||
- loading 面板
|
||||
- status message
|
||||
- tooltip / error / 清理逻辑
|
||||
|
||||
### 5. 地球与地形
|
||||
|
||||
- [earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js)
|
||||
- [terrain.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/terrain.js)
|
||||
|
||||
职责:
|
||||
|
||||
- 地球球体、云层、大气
|
||||
- 真实地形 mesh
|
||||
- terrain tile 拉取、解码、位移、着色
|
||||
|
||||
### 6. 图层模块
|
||||
|
||||
- [satellites.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/satellites.js)
|
||||
- [cables.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/cables.js)
|
||||
- [bgp.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp.js)
|
||||
- [bgp-cruise-adapter.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp-cruise-adapter.js)
|
||||
- [news.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/news.js)
|
||||
- [tv.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/tv.js)
|
||||
- [cruise-sequencer.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/cruise-sequencer.js)
|
||||
- [callout-connector.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/callout-connector.js)
|
||||
|
||||
职责:
|
||||
|
||||
- 各自的数据层
|
||||
- 开关行为
|
||||
- 面板内容
|
||||
- hover/lock/selection 语义
|
||||
|
||||
其中巡航模式现在已经拆成两层:
|
||||
|
||||
- `cruise-sequencer.js`
|
||||
- 负责目标队列顺序、停留时长、切换节奏、打断与恢复
|
||||
- `callout-connector.js`
|
||||
- 负责卡片连线 SVG、路径计算与绘制动画
|
||||
- `bgp-cruise-adapter.js`
|
||||
- 负责 BGP 巡航展示适配:目标排序、卡片落点、连线路径、focus/overlay/info-card 时序
|
||||
|
||||
当前 BGP 巡航只是这套能力的一个调用方,不应再把“按队列巡航”和“BGP 事件展示”混写在同一个状态机里。
|
||||
|
||||
## 当前样式分层
|
||||
|
||||
Earth 的 CSS 不是一份大样式表,而是分层管理:
|
||||
|
||||
- [base.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/base.css)
|
||||
- [hud.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/hud.css)
|
||||
- [toolbar.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/toolbar.css)
|
||||
- [layer-panel.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/layer-panel.css)
|
||||
- [info-panel.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/info-panel.css)
|
||||
- [legend.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/legend.css)
|
||||
- [earth-stats.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/earth-stats.css)
|
||||
- [coordinates-display.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/coordinates-display.css)
|
||||
- [tv-panel.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/tv-panel.css)
|
||||
|
||||
当前建议:
|
||||
|
||||
- 通用 HUD 壳层写进 `hud.css`
|
||||
- 单一面板特性写进各自子文件
|
||||
- 不要把业务状态样式再散回 `index.html`
|
||||
|
||||
## 当前图层开关状态语义
|
||||
|
||||
Earth 图层按钮现在不应再只有“开/关”两态,而应支持:
|
||||
|
||||
- `inactive`
|
||||
- `active`
|
||||
- `loading`
|
||||
|
||||
当前入口在:
|
||||
|
||||
- [controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js)
|
||||
- [layer-button-state.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/layer-button-state.js)
|
||||
|
||||
关键函数:
|
||||
|
||||
- `updateLayerButtonState(button, isActive)`
|
||||
- `setLayerButtonState(button, options)`
|
||||
|
||||
`setLayerButtonState` 负责:
|
||||
|
||||
- `loading` 样式
|
||||
- `aria-busy`
|
||||
- 按钮禁用
|
||||
- tooltip 更新
|
||||
- 绑定状态文本更新
|
||||
- 可选同步 `active`
|
||||
|
||||
因此后续如果别的图层也需要异步启用,应该直接走这套状态机,而不是再手写一套临时 loading class。
|
||||
|
||||
### `data-status-target`
|
||||
|
||||
图层按钮可以通过:
|
||||
|
||||
- `data-status-target`
|
||||
|
||||
指向一个状态文本节点。当前 terrain 已接入:
|
||||
|
||||
- 按钮:`#toggle-terrain`
|
||||
- 状态节点:`#terrain-status`
|
||||
|
||||
以后别的异步图层也可以沿用这套约定。
|
||||
|
||||
## 当前地形链路
|
||||
|
||||
真实地形首次启用会慢,原因不只是一个:
|
||||
|
||||
1. 需要拉取 Terrarium 瓦片
|
||||
2. 需要解码图片
|
||||
3. 需要按顶点采样高程
|
||||
4. 需要重新写入 geometry 和 color
|
||||
5. 需要重新计算法线与包围体
|
||||
|
||||
当前入口在:
|
||||
|
||||
- [terrain.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/terrain.js)
|
||||
|
||||
当前已经做了两层体验优化:
|
||||
|
||||
1. 图层开关 loading 状态持续可见
|
||||
2. 页面空闲时会预热 `ensureTerrainReady()`
|
||||
|
||||
也就是说,后续再继续优化 terrain 时,优先顺序应该是:
|
||||
|
||||
1. 先保证用户感知正确
|
||||
2. 再压缩首次等待
|
||||
3. 最后才做更激进的几何/瓦片优化
|
||||
|
||||
## 当前高频风险点
|
||||
|
||||
### 1. 视觉状态和业务状态不同步
|
||||
|
||||
Earth 里最常见的 bug 不是“没渲染”,而是:
|
||||
|
||||
- 图层关了,tooltip 还在
|
||||
- 锁定对象隐藏了,info card 还在
|
||||
- legend 没跟图层切换
|
||||
- loading 已结束,但按钮还像没开
|
||||
|
||||
后续改动必须优先检查状态同步。
|
||||
|
||||
### 2. HUD 布局问题先查结构,不要先打 CSS 补丁
|
||||
|
||||
Earth HUD 历史上反复出现:
|
||||
|
||||
- 面板只剩一条缝
|
||||
- markdown 被裁掉
|
||||
- tabs/iframe 被 `overflow: hidden` 吃掉
|
||||
|
||||
优先检查:
|
||||
|
||||
1. 谁负责高度
|
||||
2. 谁负责滚动
|
||||
3. 哪一层在裁剪
|
||||
|
||||
不要上来先加 `overflow: hidden` 或额外包装层。
|
||||
|
||||
### 3. Transitional path 必须收口
|
||||
|
||||
Earth 已经经历过多轮 HUD、toolbar、media panel 重构,所以最容易积累:
|
||||
|
||||
- 旧 helper
|
||||
- 旧 class
|
||||
- 旧 fallback 逻辑
|
||||
- 已废弃变体
|
||||
|
||||
每次大功能完成后,都要做一次 cleanup pass。
|
||||
|
||||
### 4. 巡航与业务事件不要再深度耦合
|
||||
|
||||
当前正确边界应该是:
|
||||
|
||||
- 通用巡航层只知道:
|
||||
- 当前目标
|
||||
- 队列顺序
|
||||
- 相机 focus
|
||||
- 停留 / 隐藏 / 切换
|
||||
- 业务模块只负责:
|
||||
- 提供目标队列
|
||||
- 提供 focus 坐标
|
||||
- 提供卡片内容
|
||||
- 提供高亮/图层副作用
|
||||
|
||||
如果以后再给海缆、卫星或新闻做巡航,不应复制一套新的 `main.js` 状态变量,而应复用:
|
||||
|
||||
- [cruise-sequencer.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/cruise-sequencer.js)
|
||||
- [callout-connector.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/callout-connector.js)
|
||||
- [bgp-cruise-adapter.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp-cruise-adapter.js) 这种业务适配层模式
|
||||
|
||||
## 当前推荐改动方式
|
||||
|
||||
如果后续继续改 Earth,建议按这个顺序:
|
||||
|
||||
1. 先确认改的是:
|
||||
- Three.js 渲染层
|
||||
- HUD 结构层
|
||||
- 图层状态层
|
||||
- 面板内容层
|
||||
2. 如果涉及图层按钮,优先接入统一状态机
|
||||
3. 如果涉及可见性切换,检查 tooltip / legend / info-card / lock 是否一起收口
|
||||
4. 如果涉及面板布局,先查结构再动 CSS
|
||||
|
||||
## 当前与控制台前端的边界
|
||||
|
||||
Earth 前端和控制台前端不是同一套 UI 系统:
|
||||
|
||||
- 控制台前端:React + Ant Design 工作台
|
||||
- Earth 前端:`public/earth` 原生 HUD + Three.js 展示面
|
||||
|
||||
因此:
|
||||
|
||||
- Earth 不应该直接复用 Ant Table / AppLayout 语义
|
||||
- 控制台也不应该照搬 Earth HUD 动画和玻璃层语言
|
||||
|
||||
控制台相关结构见:
|
||||
|
||||
- [admin-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/frontend-admin-frontend-context.md)
|
||||
97
docs/technical/earth-news-live-streams-collector-format.md
Normal file
97
docs/technical/earth-news-live-streams-collector-format.md
Normal file
@@ -0,0 +1,97 @@
|
||||
# News Live Streams Collector Format
|
||||
|
||||
`news_live_streams` 采集器面向“频道目录 JSON”输入,而不是直接抓网页。
|
||||
|
||||
这样做的目标是:
|
||||
|
||||
- 让后台能够稳定接入世界各地新闻直播源
|
||||
- 让 `Earth` 页面电视模块始终消费统一结构
|
||||
- 便于后续接入类似 `worldmonitor` 那种 YouTube / HLS / iframe 混合频道目录
|
||||
|
||||
## 推荐 JSON 结构
|
||||
|
||||
```json
|
||||
{
|
||||
"sources": [
|
||||
{
|
||||
"id": "bbc-world-news",
|
||||
"name": "BBC World News",
|
||||
"provider": "BBC",
|
||||
"region": "UK",
|
||||
"language": "en",
|
||||
"source_type": "youtube",
|
||||
"youtube_video_id": "dQw4w9WgXcQ",
|
||||
"youtube_channel": "https://www.youtube.com/@BBCNews",
|
||||
"embed_url": "",
|
||||
"stream_url": "",
|
||||
"homepage_url": "https://www.youtube.com/@BBCNews/live",
|
||||
"poster_url": "",
|
||||
"sort_order": 220,
|
||||
"is_enabled": true,
|
||||
"notes": "Primary English global news channel"
|
||||
},
|
||||
{
|
||||
"id": "france24-en",
|
||||
"name": "France 24 English",
|
||||
"provider": "France 24",
|
||||
"region": "France",
|
||||
"language": "en",
|
||||
"source_type": "hls",
|
||||
"stream_url": "https://example.com/live.m3u8",
|
||||
"homepage_url": "https://www.france24.com/en/live",
|
||||
"sort_order": 230,
|
||||
"is_enabled": true
|
||||
},
|
||||
{
|
||||
"id": "cctv4-page",
|
||||
"name": "CCTV-4 中文国际",
|
||||
"provider": "CCTV",
|
||||
"region": "China",
|
||||
"language": "zh-CN",
|
||||
"source_type": "iframe",
|
||||
"embed_url": "https://tv.cctv.com/live/cctv4/",
|
||||
"homepage_url": "https://tv.cctv.com/live/cctv4/",
|
||||
"sort_order": 10,
|
||||
"is_enabled": true
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## 字段约定
|
||||
|
||||
- `id`: 唯一标识,建议稳定不变
|
||||
- `name`: 频道显示名
|
||||
- `provider`: 提供方
|
||||
- `region`: 国家或地区
|
||||
- `language`: 语言代码
|
||||
- `source_type`: `iframe` / `hls` / `video` / `external` / `youtube`
|
||||
- `embed_url`: 适合 iframe 内嵌的页面
|
||||
- `stream_url`: 直接视频流地址
|
||||
- `homepage_url`: 官网或频道页
|
||||
- `youtube_video_id`: YouTube 直播视频 ID
|
||||
- `youtube_channel`: YouTube 频道 handle 或频道 URL
|
||||
- `poster_url`: 封面图,可选
|
||||
- `sort_order`: 排序值,越小越靠前
|
||||
- `is_enabled`: 是否启用
|
||||
- `notes`: 简短备注
|
||||
|
||||
## 面板行为约定
|
||||
|
||||
- `youtube`
|
||||
- 优先使用 `youtube_video_id`
|
||||
- 无法内嵌时至少保留 `youtube_channel` 或 `homepage_url` 供外部打开
|
||||
- `hls` / `video`
|
||||
- 优先走 `stream_url`
|
||||
- `iframe`
|
||||
- 优先走 `embed_url`
|
||||
- `external`
|
||||
- 不尝试内嵌,只保留外部打开
|
||||
|
||||
## 当前实现状态
|
||||
|
||||
- 后台设置页可以手工维护频道目录
|
||||
- `Earth` 电视模块会合并:
|
||||
- 手工配置源
|
||||
- `news_live_streams` 采集器采集源
|
||||
- 当前默认兜底源为 `CCTV-4 中文国际`
|
||||
236
docs/technical/frontend-admin-frontend-context.md
Normal file
236
docs/technical/frontend-admin-frontend-context.md
Normal file
@@ -0,0 +1,236 @@
|
||||
# Admin Frontend Context
|
||||
|
||||
本文件描述当前控制台前端的真实结构,目标是帮助后续页面开发、表格改造、布局治理和状态收口时快速找到正确入口。
|
||||
|
||||
相关规则建议一起参考:
|
||||
|
||||
- [rules.md](/home/ray/dev/linkong/planet/rules.md)
|
||||
- [frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md)
|
||||
|
||||
## 当前目标
|
||||
|
||||
控制台前端承担的是后台工作台,而不是展示型大屏。当前约束是:
|
||||
|
||||
- 页面默认遵循单屏工作区
|
||||
- 主交互在内部模块滚动,而不是依赖整页无限变长
|
||||
- 列表、表格、分析页优先保证主工作区可见
|
||||
- 通用布局、滚动条、表格滚动行为尽量复用,不要每页各写一套
|
||||
|
||||
## 当前路由入口
|
||||
|
||||
主入口在:
|
||||
|
||||
- [App.tsx](/home/ray/dev/linkong/planet/frontend/src/App.tsx)
|
||||
|
||||
当前后台相关路由包括:
|
||||
|
||||
- `/admin`
|
||||
- `/users`
|
||||
- `/datasources`
|
||||
- `/data`
|
||||
- `/alerts/system`
|
||||
- `/alerts/bgp`
|
||||
- `/alerts/situational`
|
||||
- `/bgp`
|
||||
- `/playground`
|
||||
- `/settings`
|
||||
|
||||
`/earth` 是独立展示页,不属于控制台骨架。
|
||||
|
||||
## 当前页面骨架
|
||||
|
||||
控制台公共壳层在:
|
||||
|
||||
- [AppLayout.tsx](/home/ray/dev/linkong/planet/frontend/src/components/AppLayout/AppLayout.tsx)
|
||||
|
||||
职责:
|
||||
|
||||
- 左侧导航
|
||||
- 折叠与展开
|
||||
- 当前账号/版本信息
|
||||
- 内容区高度闭合
|
||||
- 全站统一侧边栏滚动条
|
||||
|
||||
当前结构是:
|
||||
|
||||
```tsx
|
||||
<Layout className="dashboard-layout">
|
||||
<Sider className="dashboard-sider">...</Sider>
|
||||
<Layout>
|
||||
<Content className="dashboard-content">
|
||||
<div className="dashboard-content-inner">{children}</div>
|
||||
</Content>
|
||||
</Layout>
|
||||
</Layout>
|
||||
```
|
||||
|
||||
后续控制台页面应优先适配这套壳层,而不是重新定义全页高度语义。
|
||||
|
||||
## 当前共享组件
|
||||
|
||||
### 1. `Scrollbar`
|
||||
|
||||
文件:
|
||||
|
||||
- [Scrollbar.tsx](/home/ray/dev/linkong/planet/frontend/src/components/Scrollbar/Scrollbar.tsx)
|
||||
|
||||
用途:
|
||||
|
||||
- 控制台侧边栏这类普通内容容器
|
||||
- 组件内部管理可见性、thumb 尺寸、拖拽和双轴 overflow 判定
|
||||
|
||||
当前约束:
|
||||
|
||||
- 滚动条必须是浮层,不参与布局
|
||||
- 无 overflow 时不应留下可见痕迹
|
||||
- 真实滚动仍交给原生容器,只替换可见层和交互层
|
||||
|
||||
### 2. `ScrollbarOverlay`
|
||||
|
||||
文件:
|
||||
|
||||
- [ScrollbarOverlay.tsx](/home/ray/dev/linkong/planet/frontend/src/components/Scrollbar/ScrollbarOverlay.tsx)
|
||||
|
||||
用途:
|
||||
|
||||
- Ant Table 这类内部已有滚动容器的区域
|
||||
- 不接管滚动语义,只叠加新的滚动条可见层
|
||||
|
||||
当前使用场景:
|
||||
|
||||
- 数据源
|
||||
- 采集数据
|
||||
- 用户管理
|
||||
- 设置页
|
||||
- 告警页
|
||||
- BGP 页面
|
||||
|
||||
### 3. `TableScrollRegion`
|
||||
|
||||
文件:
|
||||
|
||||
- [TableScrollRegion.tsx](/home/ray/dev/linkong/planet/frontend/src/components/Scrollbar/TableScrollRegion.tsx)
|
||||
|
||||
用途:
|
||||
|
||||
- 为表格滚动区提供统一包裹层
|
||||
- 后续新表格页优先复用,不要重复写“表格区域 + overlay scrollbar”样板
|
||||
|
||||
### 4. 其他共享组件
|
||||
|
||||
- [MarkdownRenderer.tsx](/home/ray/dev/linkong/planet/frontend/src/components/MarkdownRenderer/MarkdownRenderer.tsx)
|
||||
- [TableActions.tsx](/home/ray/dev/linkong/planet/frontend/src/components/TableActions/TableActions.tsx)
|
||||
|
||||
## 当前状态来源
|
||||
|
||||
### 1. 认证状态
|
||||
|
||||
文件:
|
||||
|
||||
- [auth.ts](/home/ray/dev/linkong/planet/frontend/src/stores/auth.ts)
|
||||
|
||||
职责:
|
||||
|
||||
- token
|
||||
- 当前用户
|
||||
- 登录/退出
|
||||
|
||||
`App.tsx` 用它判断是否进入登录页。
|
||||
|
||||
### 2. 业务数据网关
|
||||
|
||||
目前 AI / 态势感知相关服务集中在:
|
||||
|
||||
- [http-gateway.ts](/home/ray/dev/linkong/planet/frontend/src/services/situational-awareness/http-gateway.ts)
|
||||
- [port.ts](/home/ray/dev/linkong/planet/frontend/src/services/situational-awareness/port.ts)
|
||||
- [types.ts](/home/ray/dev/linkong/planet/frontend/src/services/situational-awareness/types.ts)
|
||||
|
||||
约束:
|
||||
|
||||
- 页面不要直接散落拼 URL
|
||||
- 先通过 port/types 定义边界
|
||||
- 再由 http/mock gateway 实现
|
||||
|
||||
## 当前页面分层建议
|
||||
|
||||
### 1. 仪表盘和摘要型页面
|
||||
|
||||
例如:
|
||||
|
||||
- [Dashboard.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Dashboard/Dashboard.tsx)
|
||||
|
||||
优先目标:
|
||||
|
||||
- 页头稳定
|
||||
- 摘要卡片先紧凑化
|
||||
- 主工作区占据主要高度
|
||||
|
||||
### 2. 表格型页面
|
||||
|
||||
例如:
|
||||
|
||||
- [DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/DataSources/DataSources.tsx)
|
||||
- [DataList.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/DataList/DataList.tsx)
|
||||
- [Users.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Users/Users.tsx)
|
||||
- [Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Settings/Settings.tsx)
|
||||
|
||||
约束:
|
||||
|
||||
- 优先内部滚动
|
||||
- 不要让表格撑爆整页
|
||||
- 新表格区域优先复用 `TableScrollRegion` / `ScrollbarOverlay`
|
||||
|
||||
### 3. 复杂工作区页面
|
||||
|
||||
例如:
|
||||
|
||||
- [BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx)
|
||||
- [Playground.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Playground/Playground.tsx)
|
||||
|
||||
约束:
|
||||
|
||||
- Tabs 里的内容不能套同一套高度逻辑
|
||||
- 表格 tab、Markdown tab、配置 tab 要各自定义滚动责任
|
||||
- AI 结果区、长文本区优先保证最小可读高度
|
||||
|
||||
## 当前布局约束
|
||||
|
||||
这些原则已经在项目里反复验证过:
|
||||
|
||||
1. 父容器高度链要闭合
|
||||
2. `min-height: 0` 不能漏
|
||||
3. overflow 责任必须明确
|
||||
4. 不要用 `overflow: hidden` 掩盖结构问题
|
||||
5. 不要为了摘要卡完整显示去压缩主工作区
|
||||
6. 自定义滚动条必须是浮层,不得挤压内容宽度
|
||||
|
||||
详细经验见:
|
||||
|
||||
- [frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md)
|
||||
|
||||
## 当前推荐改动方式
|
||||
|
||||
如果后续继续改后台页面,建议按这个顺序:
|
||||
|
||||
1. 先确认页面属于摘要页、表格页还是复杂工作区
|
||||
2. 先接入现有壳层和滚动语义
|
||||
3. 优先复用共享滚动组件
|
||||
4. 最后再改视觉和细节交互
|
||||
|
||||
不要先写局部 CSS 补丁,再回头补结构。
|
||||
|
||||
## 当前明显边界
|
||||
|
||||
控制台前端和 Earth 前端不是一套系统:
|
||||
|
||||
- 控制台前端是 React + Ant Design 工作台
|
||||
- Earth 前端是 `public/earth` 下的独立原生 HUD 系统
|
||||
|
||||
因此:
|
||||
|
||||
- 不要把 Earth 的 HUD/动画/状态机直接挪进控制台
|
||||
- 不要把控制台表格/滚动策略硬套到 Earth HUD
|
||||
|
||||
Earth 相关结构见:
|
||||
|
||||
- [earth-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/earth-frontend-context.md)
|
||||
309
docs/technical/frontend-layout-guidelines.md
Normal file
309
docs/technical/frontend-layout-guidelines.md
Normal file
@@ -0,0 +1,309 @@
|
||||
# Frontend Layout Guidelines
|
||||
|
||||
本项目后台页面默认遵循“单屏工作区”布局规范。目标不是让页面永远不溢出,而是确保在常见桌面视口下:
|
||||
|
||||
- 页面主结构能在一屏内看清
|
||||
- 用户能同时看到页头、摘要区和主工作区
|
||||
- 超出的内容在模块内部滚动,而不是把整页纵向撑爆
|
||||
|
||||
当前推荐参考实现:
|
||||
|
||||
- [frontend/src/pages/BGP/BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx)
|
||||
- [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css)
|
||||
|
||||
## 核心原则
|
||||
|
||||
### 1. 页面优先保证一屏工作区
|
||||
|
||||
管理页默认采用:
|
||||
|
||||
- 页头:标题、说明、主要操作
|
||||
- 主工作区:统计卡、表格、图表、列表、标签页
|
||||
|
||||
推荐结构:
|
||||
|
||||
```tsx
|
||||
<AppLayout>
|
||||
<div className="page-shell">
|
||||
<div className="page-shell__header">...</div>
|
||||
<div className="page-shell__body">...</div>
|
||||
</div>
|
||||
</AppLayout>
|
||||
```
|
||||
|
||||
页面总高度应被限制在 `AppLayout` 内容区内,而不是继续让整个页面自然向下增长。
|
||||
|
||||
### 2. 滚动优先发生在模块内部
|
||||
|
||||
如果表格、日志、长列表、图表明细超出空间:
|
||||
|
||||
- 让卡片内部滚动
|
||||
- 让表格内部滚动
|
||||
- 让标签页内容区内部滚动
|
||||
|
||||
不要默认依赖整个页面滚动去“解决”空间问题。
|
||||
|
||||
### 3. 主工作区必须拿到主要空间
|
||||
|
||||
页面里最重要的模块必须是视觉和空间上的主角。通常应保证:
|
||||
|
||||
- 页头始终可见
|
||||
- 摘要区高度被控制
|
||||
- 主表格 / 主图表 / 主分析区占据 50% 以上可视高度
|
||||
|
||||
如果一个页面有多个大模块,优先顺序是:
|
||||
|
||||
1. 先压缩说明区和摘要区
|
||||
2. 再把次级模块收进标签页或切换视图
|
||||
3. 最后才考虑继续增加整页滚动
|
||||
|
||||
### 4. 小屏幕和高缩放必须进入紧凑模式
|
||||
|
||||
在窗口高度较低、宽度较窄、或系统缩放较高时,应主动切换紧凑布局,例如:
|
||||
|
||||
- 缩小卡片 padding
|
||||
- 缩小表头和单元格间距
|
||||
- 将摘要区改为更紧凑的单行/横向滚动布局
|
||||
- 将次级模块移入标签页、抽屉、折叠区
|
||||
|
||||
紧凑模式的目标是保持可用,不是单纯把文字和控件一股脑缩小。
|
||||
|
||||
### 5. overflow 责任必须明确
|
||||
|
||||
页面中的大块内容必须明确:
|
||||
|
||||
- 谁负责占满剩余高度
|
||||
- 谁负责裁剪
|
||||
- 谁负责滚动
|
||||
|
||||
常见要求:
|
||||
|
||||
- 父容器链路需要 `min-height: 0`
|
||||
- 工作区容器通常需要 `display: flex`
|
||||
- 真正的滚动节点要显式 `overflow: auto`
|
||||
|
||||
### 6. 卡片不能被压到不可读
|
||||
|
||||
历史上我们反复踩到的问题不是“没有滚动条”,而是:
|
||||
|
||||
- 卡片被 `flex` 压缩得只剩一小条可视区域
|
||||
- 文字能渲染,但读不完整
|
||||
- 内容其实存在,却被 `overflow: hidden` 裁掉
|
||||
|
||||
因此后续约束是:
|
||||
|
||||
- 先保证卡片有可读的最小高度
|
||||
- 如果继续压缩会影响阅读,就切换成内部滚动
|
||||
- 不要为了“保持一屏”而把正文、表格、描述区压成无法阅读的条状区域
|
||||
|
||||
### 7. Tabs 不是天然安全的布局容器
|
||||
|
||||
历史上 Tabs 相关回归非常多,典型问题包括:
|
||||
|
||||
- 隐藏 tab pane 因为自定义 `display: flex` 而重新露出来
|
||||
- 所有 tab 被强行套用同一套高度/overflow 规则
|
||||
- 表格 tab 能工作,但 markdown / help / diagnostics tab 被压坏
|
||||
|
||||
因此约束是:
|
||||
|
||||
- `Tabs` 里的每类内容都要单独定义自己的布局策略
|
||||
- 表格 tab 可以是“固定高度 + 内部滚动”
|
||||
- 文档/Markdown tab 更适合“tab pane 自身滚动 + 内容正常文档流”
|
||||
- 如果覆盖组件库样式,必须同时检查 hidden 状态是否仍然成立
|
||||
|
||||
### 8. 摘要区优先进入紧凑模式,而不是挤压正文
|
||||
|
||||
历史经验表明,最容易被误处理的是顶部摘要卡:
|
||||
|
||||
- 它们经常为了“都放下”被强行压窄
|
||||
- 然后正文、表格、AI 结果区一起失去主空间
|
||||
|
||||
后续统一约束:
|
||||
|
||||
- 小屏或高缩放时,摘要卡优先:
|
||||
- 降低 padding
|
||||
- 改成横向滚动
|
||||
- 改成更紧凑的网格
|
||||
- 不要优先牺牲主工作区的可视面积
|
||||
|
||||
### 9. 长文档类内容优先保证阅读体验
|
||||
|
||||
像下面这些内容,不能直接套用“表格工作区”的逻辑:
|
||||
|
||||
- AI 简报
|
||||
- 运行日志
|
||||
- 原始 JSON
|
||||
- 帮助说明
|
||||
- 多段描述性文本
|
||||
|
||||
这些区域应该优先满足:
|
||||
|
||||
- 标题和元信息稳定可见
|
||||
- 正文有明确的最小可读高度
|
||||
- 正文滚动策略单独定义
|
||||
- 支持 Markdown 表格、分隔线、引用、代码块等结构
|
||||
|
||||
### 10. 高度关键路径要少包一层
|
||||
|
||||
历史上不少滚动问题不是组件本身错,而是多包了一层之后:
|
||||
|
||||
- 高度链路断掉
|
||||
- `min-height: 0` 没传下去
|
||||
- `overflow` 责任被吃掉
|
||||
|
||||
因此:
|
||||
|
||||
- 对高度关键区域,优先使用最直接的 DOM 结构
|
||||
- 使用 `Space`、额外包装 `div`、第三方布局容器时,要确认它们不会改变滚动和高度语义
|
||||
- 如果一个区域已经出现“内容明明有,但只剩一条缝”,优先怀疑中间包装层
|
||||
|
||||
## 历史坑位总结
|
||||
|
||||
从 Earth、Playground、BGP、DataSources 这些页面的 bugfix 可以归纳出几类高频坑:
|
||||
|
||||
### 1. 用 `overflow: hidden` 掩盖布局问题
|
||||
|
||||
表面上看页面“整齐了”,实际上会导致:
|
||||
|
||||
- 内容被裁掉
|
||||
- tab 内容只剩一条缝
|
||||
- 面板明明渲染成功,但用户看不见
|
||||
|
||||
正确做法:
|
||||
|
||||
- 让真正的内容节点滚动
|
||||
- 不要让上层容器无差别裁剪所有子内容
|
||||
|
||||
### 2. 把所有 tab 当成同一种内容
|
||||
|
||||
表格、Markdown、帮助卡、日志流的空间需求完全不同。
|
||||
|
||||
正确做法:
|
||||
|
||||
- 表格:固定工作区 + 内部滚动
|
||||
- 文档:普通流式内容 + pane 级滚动
|
||||
- 侧边说明:内容驱动高度,不强行拉满
|
||||
|
||||
### 3. 只做视觉缩小,不做空间重分配
|
||||
|
||||
这会导致:
|
||||
|
||||
- 卡片文字被截断
|
||||
- 表格只剩 1 到 2 行
|
||||
- 按钮和筛选区挤成一团
|
||||
|
||||
正确做法:
|
||||
|
||||
- 紧凑模式优先重排
|
||||
- 横向滚动摘要区
|
||||
- 折叠/收纳次级模块
|
||||
|
||||
### 4. 父容器高度链不完整
|
||||
|
||||
这是最常见的内部滚动失效原因。
|
||||
|
||||
检查顺序:
|
||||
|
||||
1. 外层是否真的有确定高度
|
||||
2. flex 父容器是否带了 `min-height: 0`
|
||||
3. 真正滚动节点是否明确 `overflow: auto`
|
||||
4. 中间包装层是否偷偷改了布局语义
|
||||
|
||||
### 5. UI 状态和显示状态不同步
|
||||
|
||||
Earth 相关改动里反复出现:
|
||||
|
||||
- 图层隐藏了,但 hover/lock 还在
|
||||
- tooltip 还在显示旧对象
|
||||
- legend 没跟着切换
|
||||
|
||||
这类约束同样适用于后台页面:
|
||||
|
||||
- 被隐藏、卸载、切换出视图的内容,不应继续保留活跃交互状态
|
||||
|
||||
## 推荐实现模式
|
||||
|
||||
### 页面骨架
|
||||
|
||||
优先复用项目里已有的通用结构:
|
||||
|
||||
- `.dashboard-content-inner`
|
||||
- `.page-shell`
|
||||
- `.page-shell__header`
|
||||
- `.page-shell__body`
|
||||
- `.table-scroll-region`
|
||||
|
||||
不要每个页面都重新发明一套完全不同的高度和滚动语义。
|
||||
|
||||
### 表格工作区
|
||||
|
||||
推荐模式:
|
||||
|
||||
```tsx
|
||||
<Card>
|
||||
<div className="table-scroll-region" ref={tableRegionRef}>
|
||||
<Table
|
||||
pagination={false}
|
||||
scroll={{ x: 1200, y: tableHeight }}
|
||||
/>
|
||||
</div>
|
||||
</Card>
|
||||
```
|
||||
|
||||
要求:
|
||||
|
||||
- 表格尽量在卡片内部滚动
|
||||
- `scroll.y` 应来自实际可用高度估算,而不是完全静态的魔法数字
|
||||
- 父容器链路要保证 header、body、content 的 overflow 都在表格内部闭合
|
||||
|
||||
### 多模块页面
|
||||
|
||||
如果一个页面同时有:
|
||||
|
||||
- 摘要卡
|
||||
- 表格
|
||||
- 异常明细
|
||||
- 最近事件
|
||||
|
||||
不建议简单纵向堆叠全部模块。优先使用:
|
||||
|
||||
- 顶部摘要 + 底部单一主工作区
|
||||
- 标签页切换多个次级数据视图
|
||||
- 左右分栏,并保证每栏内部独立滚动
|
||||
|
||||
## 不推荐的做法
|
||||
|
||||
以下模式默认视为不符合本项目页面规范:
|
||||
|
||||
- 依赖整页纵向滚动来显示主要工作区
|
||||
- 一个页面纵向堆 3 到 4 个大卡片,每个都想完整展示
|
||||
- 表格没有内部滚动,导致缩放后只能看到 1 到 2 行数据
|
||||
- 父容器缺少 `min-height: 0`,导致内部滚动失效
|
||||
- 只做视觉缩小,不处理真正的空间分配
|
||||
|
||||
## 页面验收检查清单
|
||||
|
||||
提交前至少检查:
|
||||
|
||||
- 页头、摘要区、主工作区能否同时出现
|
||||
- 主工作区是否拿到了页面中最多的高度
|
||||
- 表格或明细溢出时,滚动条是否出现在模块内部
|
||||
- 卡片是否被压缩到文字显示不完整;如果会,是否已经切换为内部滚动
|
||||
- 浏览器缩放到 `125%` / `150%` 时是否仍可用
|
||||
- 低高度窗口下是否还保有合理的可见内容行数
|
||||
- Tabs、Card、Table 在 overflow 时是否仍可操作
|
||||
- 非表格 tab(Markdown、帮助说明、日志)是否有独立且合理的滚动策略
|
||||
|
||||
## 落地顺序
|
||||
|
||||
后续新增或重构后台页时,优先按这个顺序设计:
|
||||
|
||||
1. 先定义主工作区
|
||||
2. 再确定哪些模块必须常驻可见
|
||||
3. 最后再做样式和视觉层次
|
||||
|
||||
简单说:
|
||||
|
||||
- 先保证空间分配正确
|
||||
- 再处理滚动边界
|
||||
- 最后再做美化
|
||||
105
docs/technical/ops-docker-compose-buildx-upgrade.md
Normal file
105
docs/technical/ops-docker-compose-buildx-upgrade.md
Normal file
@@ -0,0 +1,105 @@
|
||||
# Docker + Compose + Buildx 升级教程
|
||||
|
||||
流程:删除旧版 -> 安装新版 -> 验证
|
||||
|
||||
---
|
||||
|
||||
# 1. 删除旧版本
|
||||
|
||||
## 删除 apt 安装的旧包
|
||||
|
||||
```bash
|
||||
sudo apt remove -y docker.io docker-compose docker-compose-v2 docker-doc podman-docker containerd runc
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 删除系统中的 `docker-compose`(V1)
|
||||
|
||||
```bash
|
||||
sudo rm -f "$(which docker-compose 2>/dev/null)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 查找并删除手动安装的 Buildx 插件
|
||||
|
||||
```bash
|
||||
docker info | sed -n '/Plugins:/,/^ Server:/p' | grep -A2 buildx
|
||||
```
|
||||
|
||||
从输出中获取 `Path`,然后执行:
|
||||
|
||||
```bash
|
||||
rm -f <Path中对应的docker-buildx文件>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 清理无用依赖
|
||||
|
||||
```bash
|
||||
sudo apt autoremove -y
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# 2. 安装 Docker 官方版本
|
||||
|
||||
包含 Docker Engine、Docker Compose 插件、Docker Buildx 插件。
|
||||
|
||||
## 安装依赖
|
||||
|
||||
```bash
|
||||
sudo apt update
|
||||
sudo apt install -y ca-certificates curl gnupg
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 添加 Docker GPG key
|
||||
|
||||
```bash
|
||||
sudo install -m 0755 -d /etc/apt/keyrings
|
||||
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | \
|
||||
sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
|
||||
sudo chmod a+r /etc/apt/keyrings/docker.gpg
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 添加官方仓库
|
||||
|
||||
```bash
|
||||
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
|
||||
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 安装 Docker + Compose + Buildx
|
||||
|
||||
```bash
|
||||
sudo apt update
|
||||
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# 3. 验证安装
|
||||
|
||||
```bash
|
||||
docker --version
|
||||
docker compose version
|
||||
docker buildx version
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# 4. 常用命令
|
||||
|
||||
```bash
|
||||
docker compose up -d
|
||||
docker compose down
|
||||
docker buildx build .
|
||||
```
|
||||
@@ -16,12 +16,31 @@
|
||||
## Current Version
|
||||
|
||||
- `main` 当前主线历史推导到:`0.16.5`
|
||||
- `dev` 当前开发分支历史推导到:`0.23.0`
|
||||
- `dev` 当前开发分支历史推导到:`0.31.2`
|
||||
|
||||
## Timeline
|
||||
|
||||
| Version | Type | Branch | Commit | Summary |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| `0.31.2` | bugfix | `dev` | `pending` | 将 Earth 巡航模式拆成通用 sequencer、通用连线和 BGP 巡航适配层,并修复空白点击推进与连线动画回归 |
|
||||
| `0.31.1` | bugfix | `dev` | `pending` | Earth 图层开关统一 loading 状态机,卫星首次加载可见化,并将文档按 technical / plans / deprecated 重构归档 |
|
||||
| `0.31.0` | feature | `dev` | `pending` | Earth 巡航展示模式:自动轮播 BGP 事件,连线逐帧追踪,卫星/海缆联动高亮,视觉状态全面统一 |
|
||||
| `0.30.0` | feature | `dev` | `pending` | Earth 新增真实地形图层(Terrarium DEM 代理 + 前端瓦片解码着色),设置弹窗支持地形透明度滑块 |
|
||||
| `0.29.2` | bugfix | `dev` | `pending` | 修正 Earth 设置弹窗展开表现与系统入口,继续统一液态玻璃 HUD,并校正太阳受光方向 |
|
||||
| `0.29.1` | bugfix | `dev` | `pending` | Earth 加载通知条改为队列式单面板显示,brand panel 去框并收敛昼夜与选中态可读性 |
|
||||
| `0.29.0` | feature | `dev` | `pending` | Earth 新增天球背景与太阳/月亮位置层,强化昼夜分隔并收口卫星图例与图层面板交互 |
|
||||
| `0.28.2` | bugfix | `dev` | `pending` | 修正媒体情报 tab 尺寸记忆与切换锚点逻辑,并清理 docs 根目录遗留旧路径文档 |
|
||||
| `0.28.1` | bugfix | `dev` | `pending` | 收口 Earth 媒体情报面板命名与 tab 文案,整理 docs 分组并归档已完成/废弃计划文档 |
|
||||
| `0.28.0` | feature | `dev` | `pending` | 合并 Earth 媒体情报面板,整合新闻直播与态势聚合 tab,并稳定 TV/news 的 reform、resize 与共享 HUD 行为 |
|
||||
| `0.27.8` | bugfix | `dev` | `pending` | 统一 Earth HUD 默认折叠逻辑,修复图例与图层面板箭头和底边阈值行为 |
|
||||
| `0.27.7` | bugfix | `dev` | `pending` | 修复电视直播源编辑持久化问题,清理表格空白占位列并统一可折叠操作列 |
|
||||
| `0.27.6` | improvement | `dev` | `pending` | BGP/用户表格滚动条修复,Playground 响应式按钮与输入框收起优化 |
|
||||
| `0.27.5` | bugfix | `dev` | `pending` | 统一控制台自定义滚动条,修复 alerts/BGP 响应式滚动与采集进度完成态显示 |
|
||||
| `0.27.4` | improvement | `dev` | — | info-card 懒加载动态挂载,页面初始不再有隐藏节点 |
|
||||
| `0.27.3` | improvement | `dev` | — | TV panel 折叠方向稳定、视频跳动修复、图例折叠按钮修复、搜索图标调整 |
|
||||
| `0.27.2` | improvement | `dev` | — | 修复 brand copy 宽度问题,提取 --brand-copy-width CSS 变量 |
|
||||
| `0.27.1` | improvement | `dev` | — | HUD 面板拖拽 L 形边界约束、brand 组件整体缩放、图层面板宽度优化、搜索叉叉修复 |
|
||||
| `0.27.0` | feature | `dev` | — | Earth HUD 重构:图层面板、信息卡片悬浮定位、Fresnel 大气层渲染 |
|
||||
| `0.0.1-beta` | bootstrap | `main` | `e7033775` | first commit |
|
||||
| `0.1.0` | feature | `main` | `6cb4398f` | Modularize 3D Earth page with ES Modules |
|
||||
| `0.2.0` | feature | `main` | `aaae6a53` | Add cable graph service and data collectors |
|
||||
@@ -71,6 +90,23 @@
|
||||
| `0.21.6` | bugfix | `dev` | `pending` | improve Earth legend generation, info-card interactions, and HUD messaging polish |
|
||||
| `0.22.9` | bugfix | `dev` | `6bfcd053` | simplify `planet.sh` readiness messaging and only show retry counts on actual restart |
|
||||
| `0.23.0` | feature | `dev` | `pending` | add dedicated `aiprovider` service, multi-protocol AI adapters, uv-only Python runtime, and AI Provider restart controls |
|
||||
| `0.23.3` | bugfix | `dev` | `pending` | refine `planet.sh` zsh runtime compatibility, startup logging, and frontend readiness feedback |
|
||||
| `0.24.0` | feature | `dev` | `pending` | add AI Playground entry, provider diagnostics, and frontend layout guidance |
|
||||
| `0.24.1` | bugfix | `dev` | `pending` | refactor Earth HUD into class-first CSS layers, unify Bun-only frontend tooling guidance, and auto-bootstrap Bun/uv in `planet.sh` |
|
||||
| `0.24.2` | bugfix | `dev` | `pending` | restore public Earth entry, refresh Playground provider diagnostics correctly, fit help-card content, and split frontend bundles by route/vendor |
|
||||
| `0.24.3` | bugfix | `dev` | `pending` | expand Playground diagnostics presets and result inspection, and make `planet.sh` rebuild changed AI Provider images with explicit Compose fallback reporting |
|
||||
| `0.24.4` | bugfix | `dev` | `pending` | polish `planet.sh` AI Provider rebuild stage boundaries, hide raw Compose build logs on success, and add explicit image-build completion feedback |
|
||||
| `0.24.5` | bugfix | `dev` | `pending` | add persistent BGP AI briefs with Markdown history, lazy-load BGP tabs, and move BGP hot-path filtering and aggregation back into the database |
|
||||
| `0.24.6` | bugfix | `dev` | `pending` | batch datasource and visualization hot-path queries, fix BGP collector JSON extraction, and rebuild the BGP AI brief tab layout and markdown rendering |
|
||||
| `0.24.7` | bugfix | `dev` | `pending` | formalize release workflow and frontend layout constraints with repo rules and a reusable release skill |
|
||||
| `0.24.8` | bugfix | `dev` | `pending` | move BGP brief markdown into a dedicated modal, keep tab content metadata-focused, and constrain modal scrolling to the viewport |
|
||||
| `0.25.0` | feature | `dev` | `89a71e6f` | add persistent backend-backed AI Playground chat state, split alert workspaces into dedicated pages, and establish the situational-awareness foundation for later multi-signal analysis |
|
||||
| `0.25.1` | bugfix | `dev` | `pending` | clean duplicated Playground flow code, add reusable code-hygiene rules, and fix first-level sidebar menu expansion behavior across route navigation and refresh |
|
||||
| `0.25.2` | bugfix | `dev` | `pending` | refine Earth settings modal sizing, eliminate first-frame HUD scale flicker, and make dragged HUD panels animate cleanly through maximized layout transitions |
|
||||
| `0.25.3` | bugfix | `dev` | `pending` | refactor the Earth HUD visual system, extract the top-left Earth brand into a reusable language-driven component, and consolidate duplicated brand assets into a single canonical set |
|
||||
| `0.26.0` | feature | `dev` | `pending` | add the Earth TV live module with backend-configurable sources, a draggable/resizable TV HUD window, a first curated news channel catalog, and TV source management hooks in system settings |
|
||||
| `0.26.1` | bugfix | `dev` | `pending` | extract the dashboard sidebar scrollbar into a reusable component and clean duplicated Earth TV player reset logic after the first live-module rollout |
|
||||
| `0.26.2` | bugfix | `dev` | `pending` | stabilize the reusable sidebar scrollbar, restore automatic dual-axis floating tracks safely, and apply the same overlay scrollbar system to datasource tables |
|
||||
|
||||
## Maintenance Commits Not Counted as Version Bumps
|
||||
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
{
|
||||
"name": "planet-frontend",
|
||||
"version": "0.23.0",
|
||||
"version": "0.31.2",
|
||||
"private": true,
|
||||
"packageManager": "bun@1",
|
||||
"dependencies": {
|
||||
"@ant-design/icons": "^5.2.6",
|
||||
"antd": "^5.12.5",
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user