Compare commits
79
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
b4e8afb272 | ||
|
|
eeee788530 | ||
|
|
655e2a7d2d | ||
|
|
3ea99a9529 | ||
|
|
f9c1334365 | ||
|
|
5f47ec1659 | ||
|
|
229be0bced | ||
|
|
50a417ca83 | ||
|
|
e9464a9833 | ||
|
|
86807f6af6 | ||
|
|
8b8f7138c0 | ||
|
|
d5f3784ffb | ||
|
|
195a8bf71c | ||
|
|
987c378f99 | ||
|
|
67f82dc41c | ||
|
|
abe04030fb | ||
|
|
6a5f9f7ad4 | ||
|
|
439a512148 | ||
|
|
f73fa1ea6d | ||
|
|
5b623a6385 | ||
|
|
0082cf3fbd | ||
|
|
3ae4acdff8 | ||
|
|
437efc848c | ||
|
|
003a46ac30 | ||
|
|
4b0be4cb76 | ||
|
|
b7647379de | ||
|
|
0f89372d71 | ||
|
|
2b0d4cfc49 | ||
|
|
e6d0332fba | ||
|
|
fe45a99cbd | ||
|
|
ae77b06c3c | ||
|
|
b5dd4f12f8 | ||
|
|
75cb214f23 | ||
|
|
4c21973197 | ||
|
|
51ae5e6ec9 | ||
|
|
1cf1f32ddd | ||
|
|
8f3ab88743 | ||
|
|
f8b43a995b | ||
|
|
d9adaf4134 | ||
|
|
40e51d5b20 | ||
|
|
93c1c1e550 | ||
|
|
48eb13b993 | ||
|
|
11179e7e67 | ||
|
|
07e26d6d5a | ||
|
|
7cd29cf9c0 | ||
|
|
2ee4773f4f | ||
|
|
b1d0624061 | ||
|
|
812c825dc6 | ||
|
|
a359d94127 | ||
|
|
c92be9c054 | ||
|
|
10e2bae8c2 | ||
|
|
60ed88b609 | ||
|
|
e85a9fc614 | ||
|
|
a2210f0f78 | ||
|
|
62ad09e816 | ||
|
|
89a71e6f29 | ||
|
|
60f5ff9bab | ||
|
|
fbb6adfbf5 | ||
|
|
749e6e76b6 | ||
|
|
83839b8b11 | ||
|
|
ed898aef9c | ||
|
|
abe0b5c11b | ||
|
|
306ba7f850 | ||
|
|
39f90bd575 | ||
|
|
c4ea918fac | ||
|
|
34d94a6b6b | ||
|
|
ef65acd49c | ||
|
|
5639546990 | ||
|
|
c8fe8cad59 | ||
|
|
d395769df6 | ||
|
|
8bd9d34376 | ||
|
|
2d43263b9e | ||
|
|
2da6ed166b | ||
|
|
da587398d9 | ||
|
|
f5308340af | ||
|
|
981617ee80 | ||
|
|
7abf391c74 | ||
|
|
f12719914d | ||
|
|
bc90e00e25 |
No files matched your search
@@ -0,0 +1,120 @@
|
||||
---
|
||||
description: 审查当前工作区未提交代码中的垃圾代码,并在不影响逻辑的前提下自动清理
|
||||
argument-hint: 可选:指定要检查的文件或目录(默认检查所有未提交修改)
|
||||
allowed-tools: ["Read", "Edit", "Bash", "Grep", "Glob"]
|
||||
---
|
||||
|
||||
# /cleanup — 垃圾代码审查与清理
|
||||
|
||||
分析当前工作区(git diff)中的未提交代码,找出并修复常见垃圾代码,**不得改变任何运行逻辑**。
|
||||
|
||||
## 检查范围
|
||||
|
||||
若 `$ARGUMENTS` 非空,则只检查指定文件/目录;否则检查所有未提交修改(`git diff HEAD`)。
|
||||
|
||||
## 审查清单
|
||||
|
||||
按优先级检查以下问题(只报告在本次 diff 中**新增或修改**的代码里存在的问题):
|
||||
|
||||
### 1. 重复逻辑 (Duplicate Logic)
|
||||
- 完全相同或高度相似的代码块在多处出现
|
||||
- 同一函数/方法被多个地方各自实现,已有公共版本未被复用
|
||||
- 相同的 DOM 查询、正则、模板字符串在同一文件重复
|
||||
|
||||
### 2. Magic Numbers / Magic Strings
|
||||
- 裸数字直接参与计算(如偏移量、时间、尺寸、阈值),没有命名常量
|
||||
- 硬编码字符串(如 id 名、状态值、URL 片段)散落在逻辑中
|
||||
- 例外:`0`, `1`, `-1`, `100`, `""` 等语义明确的惯用值不算
|
||||
|
||||
### 3. 命名问题
|
||||
- 含义不明的缩写变量(如 `or_`, `tmp2`, `x2`)
|
||||
- 命名与实际用途不符
|
||||
- 同一概念在不同地方用不同名字表达
|
||||
|
||||
### 4. 死代码 / 无效代码
|
||||
- 注释掉的旧代码块(3行以上)
|
||||
- 声明后从未使用的变量/参数/导入
|
||||
- 永远不会执行的条件分支
|
||||
|
||||
### 5. 代码风格问题
|
||||
- 尾部空白字符(trailing whitespace)
|
||||
- 同一文件内风格不一致(如混用单双引号、缩进不统一)
|
||||
- 空行使用不一致(连续多个空行等)
|
||||
|
||||
### 6. 其他常见问题
|
||||
- 私有辅助函数应被 export 但没有,导致调用方重复实现
|
||||
- 类型/接口重复定义
|
||||
- 过于冗长的条件表达式可以简化(不改逻辑)
|
||||
|
||||
## 执行步骤
|
||||
|
||||
### Step 1 — 获取待检查文件列表
|
||||
|
||||
```bash
|
||||
# 无参数时:获取所有未提交修改
|
||||
git diff HEAD --name-only
|
||||
|
||||
# 有参数时:用 $ARGUMENTS 过滤
|
||||
```
|
||||
|
||||
### Step 2 — 逐文件阅读并分析
|
||||
|
||||
- 用 Read 工具读取完整文件(不只读 diff)
|
||||
- 对照审查清单,记录每个问题:文件名、行号、问题类型、建议修复方式
|
||||
|
||||
### Step 3 — 报告问题清单
|
||||
|
||||
在修改前,先以列表形式输出所有发现的问题:
|
||||
|
||||
```
|
||||
发现 N 个问题:
|
||||
|
||||
[文件] js/foo.js
|
||||
· L34, L78: 重复逻辑 — 两处都实现了相同的 DOM 查询,可提取到 getPanel()
|
||||
· L91: Magic number — 硬编码 14 作为偏移量,应命名为 TOOLTIP_OFFSET
|
||||
|
||||
[文件] js/bar.js
|
||||
· L12: 命名问题 — 变量 `or_` 语义不明,应命名为 outerR/outerG/outerB
|
||||
...
|
||||
```
|
||||
|
||||
如果没有发现问题,直接输出"未发现垃圾代码,当前代码质量良好。"并停止。
|
||||
|
||||
### Step 4 — 执行修复
|
||||
|
||||
对每个问题,使用 Edit 工具进行**最小化修改**:
|
||||
|
||||
- **重复逻辑**:提取为共享常量/函数,更新所有调用点
|
||||
- **Magic number**:在文件顶部或逻辑附近声明 `const NAME = value`,替换所有引用
|
||||
- **命名问题**:重命名变量,更新所有使用处
|
||||
- **死代码**:直接删除
|
||||
- **尾部空白/风格**:修正
|
||||
- **未 export 的函数**:添加 `export`,在调用方改为导入(不重复实现)
|
||||
|
||||
**修复原则:**
|
||||
- 只改在审查清单中发现的问题,不做额外优化
|
||||
- 每次 Edit 只修改确实有问题的行,保持 diff 最小
|
||||
- 改完后用 `grep` 验证旧的坏代码已消失
|
||||
|
||||
### Step 5 — 输出总结
|
||||
|
||||
```
|
||||
清理完成:
|
||||
|
||||
修复了 N 个问题:
|
||||
✓ earth.js — 提取重复 vertexShader 为 ATMOS_VERTEX_SHADER 常量
|
||||
✓ main.js — 提取 TOOLTIP_CURSOR_OFFSET = 14(4处引用)
|
||||
✓ controls.js — export updateLayerButtonState,移除 main.js 中的重复实现
|
||||
...
|
||||
|
||||
未修改的问题(需人工确认):
|
||||
! foo.js L45 — 注释代码块较长,建议手动确认是否可删除
|
||||
```
|
||||
|
||||
## 约束
|
||||
|
||||
- **禁止**改变函数签名、接口定义、导出 API(除非问题正是私有函数应被 export)
|
||||
- **禁止**添加新功能、新抽象、新参数
|
||||
- **禁止**修改注释内容(只删除注释掉的死代码)
|
||||
- **禁止**修改测试文件逻辑
|
||||
- 如果一个 Magic number 的语义不完全确定,**跳过**,在总结中标记为"需人工确认"
|
||||
@@ -0,0 +1,91 @@
|
||||
---
|
||||
description: 用 goal-driven 方法推动一个复杂任务持续执行,直到明确成功标准被满足
|
||||
argument-hint: 建议填写任务目标;若同时给出成功标准更好
|
||||
allowed-tools: ["Read", "Edit", "Bash", "Grep", "Glob"]
|
||||
---
|
||||
|
||||
# /goal-driven — 目标驱动执行模式
|
||||
|
||||
使用 `lidangzzz/goal-driven` 的核心思想来推进复杂任务:先固定目标与成功标准,再持续执行和反复验收,直到标准真正满足。
|
||||
|
||||
适用场景:
|
||||
|
||||
- 长周期实现任务
|
||||
- 高复杂度工程任务
|
||||
- 可被明确验收的研究、实现、迁移、验证类工作
|
||||
|
||||
不适用场景:
|
||||
|
||||
- 纯脑暴
|
||||
- 无法定义成功标准的模糊任务
|
||||
- 很小的一次性修改
|
||||
|
||||
## 输入要求
|
||||
|
||||
若 `$ARGUMENTS` 只包含目标,没有成功标准,先补全一版可执行的成功标准再开始。
|
||||
|
||||
启动时先输出:
|
||||
|
||||
```md
|
||||
Goal
|
||||
- ...
|
||||
|
||||
Criteria for success
|
||||
- ...
|
||||
|
||||
Plan
|
||||
1. ...
|
||||
2. ...
|
||||
3. ...
|
||||
|
||||
Verification
|
||||
- ...
|
||||
```
|
||||
|
||||
## 执行规则
|
||||
|
||||
1. 先把任务固化为两个核心块:
|
||||
- `Goal`
|
||||
- `Criteria for success`
|
||||
|
||||
2. 成功标准必须尽量客观,可验证,可落地。
|
||||
优先写成:
|
||||
- 需要交付什么
|
||||
- 需要通过哪些测试或验证
|
||||
- 如何判断结果真的完成
|
||||
|
||||
3. 进入持续执行循环:
|
||||
- 完成一个阶段
|
||||
- 检查当前结果是否满足成功标准
|
||||
- 若未满足,明确剩余差距并继续推进
|
||||
|
||||
4. 任何“完成了”“差不多了”“已实现”之类的结论,都必须经过验证,不能直接接受。
|
||||
|
||||
5. 如果验证失败:
|
||||
- 明确指出哪条成功标准没满足
|
||||
- 继续工作,不要把阶段性进展误判为完成
|
||||
|
||||
6. 只有在以下情况之一才能停止:
|
||||
- 成功标准已满足
|
||||
- 用户明确要求停止
|
||||
|
||||
## 执行风格
|
||||
|
||||
- 重证据,轻口头判断
|
||||
- 重验收,轻自我感觉
|
||||
- 优先用测试、日志、产物、对比结果来证明完成
|
||||
- 对长期任务保持“未达标就继续”的节奏
|
||||
|
||||
## 简版模板
|
||||
|
||||
```md
|
||||
Goal: [[[[[在此填写最终目标]]]]]
|
||||
|
||||
Criteria for success: [[[[[在此填写成功标准]]]]]
|
||||
|
||||
循环执行:
|
||||
1. 推进任务
|
||||
2. 检查是否满足成功标准
|
||||
3. 若未满足,继续工作
|
||||
4. 直到满足标准或用户明确停止
|
||||
```
|
||||
@@ -0,0 +1,146 @@
|
||||
---
|
||||
description: 发版工作流:根据变更类型决定版本号,更新所有版本文件和 changelog,运行验证,commit 并 push
|
||||
argument-hint: 可选:feature | bugfix | 或直接描述本次发布内容
|
||||
allowed-tools: ["Read", "Edit", "Bash", "Glob", "Grep"]
|
||||
---
|
||||
|
||||
# /release — Planet 发版工作流
|
||||
|
||||
## 版本号规则
|
||||
|
||||
| 变更类型 | 版本跳动 | 适用场景 |
|
||||
|---------|---------|---------|
|
||||
| `feature` | `+0.1.0` | 纯新功能,无 bugfix |
|
||||
| `improvement` | `+0.0.1` | UI 调整、小功能增强、bugfix 混合,或以 UI/体验改进为主的迭代 |
|
||||
| `bugfix` | `+0.0.1` | 纯 bug 修复,无新功能 |
|
||||
| `docs` / `maintenance` / `refactor` | 默认不发版,除非用户明确要求 |
|
||||
|
||||
意图混合时以用户明确描述为准;bugfix + 小 feature 混合默认判定为 `improvement`(`+0.0.1`)。
|
||||
|
||||
## 必须同步更新的文件
|
||||
|
||||
使用 `git rev-parse --show-toplevel` 获取仓库根目录,以下路径均相对于根目录:
|
||||
|
||||
- `VERSION`
|
||||
- `frontend/package.json`(`"version"` 字段)
|
||||
- `pyproject.toml`(`version =` 字段)
|
||||
- `uv.lock`(**不要手动编辑**,通过 `uv lock` 重新生成)
|
||||
- `docs/CHANGELOG.md`
|
||||
- `docs/version-history.md`
|
||||
|
||||
## 执行步骤
|
||||
|
||||
### Step 1 — 环境检查
|
||||
|
||||
```bash
|
||||
git branch --show-current # 确认在 dev 分支
|
||||
git status --short # 检查是否有无关的未暂存修改
|
||||
cat VERSION # 读取当前版本
|
||||
```
|
||||
|
||||
若当前**不在 `dev` 分支**,停下来告知用户,不要继续。
|
||||
|
||||
若存在无关的未暂存修改,列出并询问用户是否一并提交,或先 stash。
|
||||
|
||||
### Step 2 — 确定发版类型与新版本号
|
||||
|
||||
- 若 `$ARGUMENTS` 提供了明确类型(`feature` / `bugfix`),直接使用
|
||||
- 否则根据当前 `git diff HEAD` 和 `git log` 推断
|
||||
- 计算新版本号(例:`0.26.2` → bugfix → `0.26.3`)
|
||||
- **先输出发版计划供用户确认**:
|
||||
|
||||
```
|
||||
发版计划:
|
||||
类型:bugfix
|
||||
版本:0.26.2 → 0.26.3
|
||||
分支:dev
|
||||
将更新:VERSION, frontend/package.json, pyproject.toml, uv.lock, CHANGELOG.md, version-history.md
|
||||
```
|
||||
|
||||
### Step 3 — 更新版本号文件
|
||||
|
||||
按顺序更新(每步用 Edit 工具,精确替换,不要重写整个文件):
|
||||
|
||||
1. `VERSION` — 直接替换全部内容为新版本号
|
||||
2. `frontend/package.json` — 替换 `"version": "x.x.x"` 行
|
||||
3. `pyproject.toml` — 替换 `version = "x.x.x"` 行
|
||||
4. 运行 `uv lock` 重新生成 `uv.lock`(在仓库根目录下执行)
|
||||
|
||||
### Step 4 — 更新 CHANGELOG.md
|
||||
|
||||
在文件顶部插入新条目,格式:
|
||||
|
||||
```markdown
|
||||
## [x.x.x] — YYYY-MM-DD
|
||||
|
||||
### ✨ Features / 🐛 Fixes / 🔧 Improvements
|
||||
- ...(只列高信号条目,最多 5 条)
|
||||
- ...
|
||||
|
||||
---
|
||||
```
|
||||
|
||||
日期使用 `date +%Y-%m-%d` 获取今天的日期。
|
||||
|
||||
### Step 5 — 更新 docs/version-history.md
|
||||
|
||||
- 更新文件头部的"当前开发版本"字段
|
||||
- 在时间线表格顶部插入新行:`| vx.x.x | YYYY-MM-DD | 一句话摘要 |`
|
||||
|
||||
### Step 6 — 验证
|
||||
|
||||
针对本次变更范围做最小验证:
|
||||
|
||||
- Python 文件有修改:`python3 -m py_compile <changed_files>`
|
||||
- Frontend 文件有修改:运行项目标准检查(若无则跳过并说明)
|
||||
- 版本号一致性检查:用 grep 确认 VERSION、package.json、pyproject.toml 中的版本号完全一致
|
||||
|
||||
```bash
|
||||
grep -h "version" VERSION frontend/package.json pyproject.toml
|
||||
```
|
||||
|
||||
### Step 7 — 提交前预览
|
||||
|
||||
展示将要提交的文件列表:
|
||||
|
||||
```bash
|
||||
git diff --stat HEAD
|
||||
```
|
||||
|
||||
再次确认所有必须文件都在变更列表中,**不包含**非预期文件(如调试文件、.env 等)。
|
||||
|
||||
### Step 8 — Commit & Push(用户确认后)
|
||||
|
||||
```bash
|
||||
git add VERSION frontend/package.json pyproject.toml uv.lock docs/CHANGELOG.md docs/version-history.md
|
||||
# 若有代码变更也一并 stage
|
||||
git add <code_files>
|
||||
|
||||
git commit -m "release: bump version to x.x.x"
|
||||
git tag vx.x.x
|
||||
git push origin dev
|
||||
git push origin vx.x.x
|
||||
```
|
||||
|
||||
commit message 固定格式:`release: bump version to x.x.x`
|
||||
|
||||
### Step 9 — 完成确认
|
||||
|
||||
输出摘要:
|
||||
|
||||
```
|
||||
✓ 版本号已更新:0.26.2 → 0.26.3
|
||||
✓ CHANGELOG 已更新
|
||||
✓ version-history 已更新
|
||||
✓ uv.lock 已重新生成
|
||||
✓ 验证通过
|
||||
✓ commit: release: bump version to 0.26.3
|
||||
✓ tag: v0.26.3
|
||||
✓ 已 push 到 origin/dev
|
||||
```
|
||||
|
||||
## 注意事项
|
||||
|
||||
- `uv.lock` 只能通过 `uv lock` 生成,绝不手动编辑
|
||||
- 发版 commit 只包含版本文件 + 本次功能代码,不混入无关改动
|
||||
- 若环境中 `uv` 不可用,说明原因并跳过 lockfile 更新,提醒用户手动运行
|
||||
@@ -0,0 +1,3 @@
|
||||
approval_policy = "never"
|
||||
|
||||
sandbox_mode = "danger-full-access"
|
||||
@@ -0,0 +1,124 @@
|
||||
---
|
||||
name: cleanup
|
||||
description: Use when the user asks to clean up, lint, or review uncommitted code for common code smells — duplicate logic, magic numbers, unclear naming, dead code, style inconsistencies. Fixes issues without changing any runtime behavior.
|
||||
---
|
||||
|
||||
# Cleanup
|
||||
|
||||
Review and fix code quality issues in the current working tree without altering any logic or behavior.
|
||||
|
||||
## When To Use
|
||||
|
||||
- The user asks to clean up, tidy, or lint uncommitted changes
|
||||
- The user wants a code smell review before releasing or committing
|
||||
- The user mentions magic numbers, duplicate logic, dead code, or naming issues
|
||||
|
||||
Do not refactor architecture, add features, or change behavior.
|
||||
|
||||
## Scope
|
||||
|
||||
If the user specifies a file or directory, check only that. Otherwise check all uncommitted changes (`git diff HEAD`).
|
||||
|
||||
Only report issues present in **newly added or modified** lines of this diff — do not audit unchanged code.
|
||||
|
||||
## Checklist
|
||||
|
||||
### 1. Duplicate Logic
|
||||
- Identical or near-identical code blocks appearing in multiple places
|
||||
- A function/helper that already exists but is re-implemented elsewhere instead of being reused
|
||||
- Repeated DOM queries, regex literals, or template strings within the same file
|
||||
|
||||
### 2. Magic Numbers / Magic Strings
|
||||
- Bare numeric literals used in calculations (offsets, timeouts, sizes, thresholds) without a named constant
|
||||
- Hardcoded strings (IDs, status values, URL fragments) scattered through logic
|
||||
- Exceptions: `0`, `1`, `-1`, `100`, `""` and other idiomatically clear values are fine
|
||||
|
||||
### 3. Naming Issues
|
||||
- Cryptic abbreviations (`or_`, `tmp2`, `x2`)
|
||||
- Names that do not match actual behavior
|
||||
- The same concept referred to by different names in different places
|
||||
|
||||
### 4. Dead Code
|
||||
- Commented-out code blocks (3+ lines)
|
||||
- Variables, parameters, or imports declared but never used
|
||||
- Branches that can never execute
|
||||
|
||||
### 5. Style Inconsistencies
|
||||
- Trailing whitespace
|
||||
- Mixed quote styles or indentation within the same file
|
||||
- Inconsistent blank-line usage (multiple consecutive blank lines, etc.)
|
||||
|
||||
### 6. Other
|
||||
- Private helper functions that should be exported but are not, causing callers to duplicate the implementation
|
||||
- Overly verbose conditions that can be simplified without changing logic
|
||||
|
||||
## Steps
|
||||
|
||||
### Step 1 — Get the file list
|
||||
|
||||
```bash
|
||||
git diff HEAD --name-only
|
||||
```
|
||||
|
||||
Filter to the user-specified path if one was provided.
|
||||
|
||||
### Step 2 — Read and analyze each file
|
||||
|
||||
Read the full file (not just the diff) with the Read tool. For each file, record every issue found: filename, line number, category, and suggested fix.
|
||||
|
||||
### Step 3 — Report findings before touching anything
|
||||
|
||||
Print a structured list:
|
||||
|
||||
```
|
||||
Found N issues:
|
||||
|
||||
[file] js/foo.js
|
||||
· L34, L78: Duplicate logic — same DOM query implemented twice; extract to getPanel()
|
||||
· L91: Magic number — bare 14 used as pixel offset; name it TOOLTIP_OFFSET
|
||||
|
||||
[file] js/bar.js
|
||||
· L12: Naming — variable `or_` is unclear; rename to outerR, outerG, outerB
|
||||
...
|
||||
```
|
||||
|
||||
If no issues are found, output "No code smells detected. Code quality looks good." and stop.
|
||||
|
||||
### Step 4 — Fix each issue
|
||||
|
||||
Use the Edit tool for **minimal, targeted changes**:
|
||||
|
||||
- **Duplicate logic**: extract to a shared constant or function; update all call sites
|
||||
- **Magic number/string**: declare `const NAME = value` near the top of the relevant scope; replace all usages
|
||||
- **Naming**: rename the variable/function; update all references
|
||||
- **Dead code**: delete it
|
||||
- **Trailing whitespace / style**: fix in place
|
||||
- **Unexported helper**: add `export`; update callers to import instead of re-implementing
|
||||
|
||||
Principles:
|
||||
- Only fix issues identified in the checklist — no extra improvements
|
||||
- Keep each Edit as small as possible
|
||||
- After fixing, verify the old bad pattern is gone with grep
|
||||
|
||||
### Step 5 — Summary
|
||||
|
||||
```
|
||||
Cleanup complete:
|
||||
|
||||
Fixed N issues:
|
||||
✓ earth.js — extracted duplicate vertexShader into ATMOS_VERTEX_SHADER constant
|
||||
✓ main.js — extracted TOOLTIP_CURSOR_OFFSET = 14 (4 references updated)
|
||||
✓ controls.js — exported updateLayerButtonState; removed duplicate implementation in main.js
|
||||
...
|
||||
|
||||
Skipped (needs manual review):
|
||||
! foo.js L45 — large commented-out block; confirm it is safe to delete
|
||||
```
|
||||
|
||||
## Constraints
|
||||
|
||||
- **Do not** change function signatures, exported interfaces, or public APIs (unless the issue is a missing export)
|
||||
- **Do not** add new features, abstractions, or parameters
|
||||
- **Do not** rewrite comments (only delete commented-out dead code)
|
||||
- **Do not** touch test file logic
|
||||
- If a magic number's intent is uncertain, skip it and flag it in the summary
|
||||
Executable
+101
@@ -0,0 +1,101 @@
|
||||
---
|
||||
name: goal-driven
|
||||
description: Run a goal-driven execution loop for very large, long-horizon, rigorously verifiable tasks. Use when the user explicitly wants the lidangzzz/goal-driven method, a master-agent plus worker-agent style workflow, or a persistent loop that keeps working until concrete success criteria are satisfied.
|
||||
---
|
||||
|
||||
# Goal-Driven
|
||||
|
||||
Use this skill when the user wants a strict goal-driven workflow for a hard task with:
|
||||
|
||||
- one clear end goal
|
||||
- explicit success criteria
|
||||
- repeated verification against those criteria
|
||||
- continued execution until the criteria are actually met
|
||||
|
||||
This skill is adapted from `lidangzzz/goal-driven`, but trimmed for local skill use to avoid bloating context.
|
||||
|
||||
## When To Use
|
||||
|
||||
Use it for tasks like:
|
||||
|
||||
- compilers, interpreters, theorem-like proof work, deep refactors
|
||||
- long-running system design or implementation work
|
||||
- problems that are expensive and complex, but still objectively testable
|
||||
|
||||
Do not use it for:
|
||||
|
||||
- vague brainstorming without a success condition
|
||||
- short one-shot edits
|
||||
- tasks where "done" cannot be evaluated in a meaningful way
|
||||
|
||||
## Core Model
|
||||
|
||||
The workflow has two roles:
|
||||
|
||||
1. Master role
|
||||
Defines the goal, defines the success criteria, audits progress, and decides whether the work is actually complete.
|
||||
|
||||
2. Worker role
|
||||
Keeps advancing the task toward the goal. If a result is partial, stalled, or unverifiable, the worker continues.
|
||||
|
||||
In Codex, only use actual subagents when the user explicitly asks for delegation or subagent work and the platform supports it. Otherwise emulate the same loop locally: keep working, checkpointing, and re-verifying until the criteria are satisfied.
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Normalize the task into two blocks:
|
||||
- `Goal`
|
||||
- `Criteria for success`
|
||||
|
||||
2. Make the criteria concrete and testable.
|
||||
Good criteria usually include:
|
||||
- required outputs
|
||||
- required validations or tests
|
||||
- edge cases or coverage thresholds
|
||||
- what evidence proves completion
|
||||
|
||||
3. Break the work into milestones that can each produce evidence.
|
||||
|
||||
4. Execute the next milestone.
|
||||
If subagents are explicitly allowed, the master may delegate bounded worker tasks.
|
||||
If not, do the work locally but keep the master/worker mindset.
|
||||
|
||||
5. Whenever work pauses, stalls, or appears complete, audit against the criteria directly.
|
||||
Check artifacts, tests, logs, diffs, metrics, or other real evidence.
|
||||
|
||||
6. If the criteria are not met, continue with a specific delta:
|
||||
- what is still missing
|
||||
- what evidence failed
|
||||
- what the next worker pass must improve
|
||||
|
||||
7. Stop only when the criteria are met, or when the user explicitly stops the process.
|
||||
|
||||
## Operating Rules
|
||||
|
||||
- Prefer objective checks over self-reported completion.
|
||||
- Do not confuse progress with completion.
|
||||
- If the worker says "done", verify it.
|
||||
- If verification fails, continue from the gap instead of restarting blindly.
|
||||
- Keep the goal stable unless the user changes it.
|
||||
- Tighten fuzzy criteria before sinking large amounts of effort.
|
||||
|
||||
## Recommended Response Shape
|
||||
|
||||
When starting a goal-driven task, structure the kickoff like this:
|
||||
|
||||
```md
|
||||
Goal
|
||||
- ...
|
||||
|
||||
Criteria for success
|
||||
- ...
|
||||
|
||||
Current plan
|
||||
1. ...
|
||||
2. ...
|
||||
3. ...
|
||||
|
||||
Verification
|
||||
- What evidence will prove completion
|
||||
```
|
||||
|
||||
For a reusable prompt template, read [references/prompt-template.md](references/prompt-template.md).
|
||||
@@ -0,0 +1,7 @@
|
||||
interface:
|
||||
display_name: "Goal-Driven"
|
||||
short_description: "Drive complex work until explicit success criteria are met."
|
||||
default_prompt: "Use $goal-driven to turn this task into a concrete goal, explicit success criteria, and a verification-driven execution loop."
|
||||
|
||||
policy:
|
||||
allow_implicit_invocation: true
|
||||
+38
@@ -0,0 +1,38 @@
|
||||
# Goal-Driven Prompt Template
|
||||
|
||||
Use this when you want a reusable kickoff prompt for a master/worker execution loop.
|
||||
|
||||
```md
|
||||
# Goal-Driven System
|
||||
|
||||
Goal: [[[[[DEFINE THE FINAL GOAL HERE]]]]]
|
||||
|
||||
Criteria for success: [[[[[DEFINE THE SUCCESS CRITERIA HERE]]]]]
|
||||
|
||||
You are the master agent.
|
||||
|
||||
Your job is to:
|
||||
1. Keep the goal and criteria fixed.
|
||||
2. Start worker execution toward the goal.
|
||||
3. Audit any claimed progress against the criteria.
|
||||
4. If the criteria are not met, continue the work with a precise next delta.
|
||||
5. Stop only when the criteria are satisfied or the user explicitly stops the process.
|
||||
|
||||
Worker requirements:
|
||||
1. Break the task into subproblems.
|
||||
2. Keep producing concrete progress toward the goal.
|
||||
3. Report evidence, not just claims.
|
||||
4. Continue until the criteria are satisfied.
|
||||
|
||||
Master audit loop:
|
||||
1. Check whether the worker is still making progress.
|
||||
2. If the worker stalls or claims completion, verify against the criteria.
|
||||
3. If verification fails, resume work from the remaining gap.
|
||||
4. Repeat until the criteria are met.
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- Stronger criteria produce better results than stronger rhetoric.
|
||||
- Prefer measurable checks such as tests, parity checks, generated artifacts, benchmarks, or reviewable outputs.
|
||||
- If the environment does not support subagents, emulate the same loop locally.
|
||||
@@ -0,0 +1,157 @@
|
||||
---
|
||||
name: release
|
||||
description: Use when the user asks to release, bump version, update changelog/version files, or commit/push a repository release for the Planet repo. Determines version bump type from changes, updates all required version-bearing files, updates changelog and version-history, runs minimal validation, then commits, tags, and pushes.
|
||||
---
|
||||
|
||||
# Release Workflow
|
||||
|
||||
Use this skill for release-oriented work in this repository.
|
||||
|
||||
## When To Use
|
||||
|
||||
- The user asks to `发版`
|
||||
- The user asks to bump a version
|
||||
- The user asks to update `CHANGELOG`, `version-history`, or version files as part of a release
|
||||
- The user asks to commit/push a release or a publishable bugfix/feature bundle
|
||||
|
||||
Do not use this skill for ordinary commits that are not being released.
|
||||
|
||||
## Versioning Rules
|
||||
|
||||
- `feature` -> bump `+0.1.0`
|
||||
- `bugfix` -> bump `+0.0.1`
|
||||
- `docs`, `maintenance`, and `refactor` do not bump by default unless the user explicitly wants a release
|
||||
|
||||
When intent is mixed, prefer the user's stated release intent.
|
||||
|
||||
## Required Files
|
||||
|
||||
Use `git rev-parse --show-toplevel` to get the repo root. All paths are relative to it:
|
||||
|
||||
- `VERSION`
|
||||
- `frontend/package.json` (`"version"` field)
|
||||
- `pyproject.toml` (`version =` field)
|
||||
- `uv.lock` (**never edit manually** — regenerate by running `uv lock`)
|
||||
- `docs/CHANGELOG.md`
|
||||
- `docs/version-history.md`
|
||||
|
||||
## Workflow
|
||||
|
||||
### Step 1 — Environment check
|
||||
|
||||
```bash
|
||||
git branch --show-current # must be on dev
|
||||
git status --short # check for unrelated uncommitted changes
|
||||
cat VERSION # read current version
|
||||
```
|
||||
|
||||
If not on `dev`, stop and tell the user. Do not proceed.
|
||||
|
||||
If unrelated uncommitted changes exist, list them and ask the user whether to include them or stash first.
|
||||
|
||||
### Step 2 — Determine release type and next version
|
||||
|
||||
- If the user provided an explicit type (`feature` / `bugfix`), use it
|
||||
- Otherwise infer from `git diff HEAD` and recent `git log`
|
||||
- Compute the next version (e.g. `0.26.2` → bugfix → `0.26.3`)
|
||||
- **Show the release plan before making any changes:**
|
||||
|
||||
```
|
||||
Release plan:
|
||||
Type: bugfix
|
||||
Version: 0.26.2 → 0.26.3
|
||||
Branch: dev
|
||||
Will update: VERSION, frontend/package.json, pyproject.toml, uv.lock, CHANGELOG.md, version-history.md
|
||||
```
|
||||
|
||||
### Step 3 — Update version files
|
||||
|
||||
Update in order (use Edit for precise replacement, never rewrite whole files):
|
||||
|
||||
1. `VERSION` — replace entire content with new version string
|
||||
2. `frontend/package.json` — replace `"version": "x.x.x"` line
|
||||
3. `pyproject.toml` — replace `version = "x.x.x"` line
|
||||
4. Run `uv lock` at repo root to regenerate `uv.lock`
|
||||
|
||||
### Step 4 — Update CHANGELOG.md
|
||||
|
||||
Insert a new entry at the top of the file:
|
||||
|
||||
```markdown
|
||||
## x.x.x
|
||||
|
||||
Released: YYYY-MM-DD
|
||||
|
||||
### Highlights
|
||||
|
||||
- ...
|
||||
|
||||
### Added / Fixed / Improved
|
||||
|
||||
- ... (high-signal items only, max 5)
|
||||
|
||||
---
|
||||
```
|
||||
|
||||
Get today's date with `date +%Y-%m-%d`.
|
||||
|
||||
### Step 5 — Update docs/version-history.md
|
||||
|
||||
- Update the "current dev version" field in the file header
|
||||
- Insert a new row at the top of the timeline table: `| vx.x.x | YYYY-MM-DD | one-line summary |`
|
||||
|
||||
### Step 6 — Validate
|
||||
|
||||
Run the smallest relevant validation for the changes in scope:
|
||||
|
||||
- Python files changed: `python3 -m py_compile <changed_files>`
|
||||
- Frontend files changed: run the project-standard check if available; otherwise skip and say so
|
||||
- Version consistency: confirm VERSION, package.json, pyproject.toml, and uv.lock all show the same version
|
||||
|
||||
```bash
|
||||
grep -h "version" VERSION frontend/package.json pyproject.toml
|
||||
```
|
||||
|
||||
### Step 7 — Pre-commit preview
|
||||
|
||||
Show what will be committed:
|
||||
|
||||
```bash
|
||||
git diff --stat HEAD
|
||||
```
|
||||
|
||||
Confirm all required files are present and no unexpected files (debug files, `.env`, etc.) are included.
|
||||
|
||||
### Step 8 — Commit, tag, and push
|
||||
|
||||
```bash
|
||||
git add VERSION frontend/package.json pyproject.toml uv.lock docs/CHANGELOG.md docs/version-history.md
|
||||
# also stage any code changes included in this release
|
||||
git add <code_files>
|
||||
|
||||
git commit -m "release: bump version to x.x.x"
|
||||
git tag vx.x.x
|
||||
git push origin dev
|
||||
git push origin vx.x.x
|
||||
```
|
||||
|
||||
Commit message format is fixed: `release: bump version to x.x.x`
|
||||
|
||||
### Step 9 — Completion summary
|
||||
|
||||
```
|
||||
✓ Version bumped: 0.26.2 → 0.26.3
|
||||
✓ CHANGELOG updated
|
||||
✓ version-history updated
|
||||
✓ uv.lock regenerated
|
||||
✓ Validation passed
|
||||
✓ commit: release: bump version to 0.26.3
|
||||
✓ tag: v0.26.3
|
||||
✓ Pushed to origin/dev
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- `uv.lock` must only be updated by running `uv lock`, never manually
|
||||
- The release commit should include only version files + the code for this release — no unrelated changes
|
||||
- If `uv` is unavailable in the environment, say so explicitly and remind the user to run it manually
|
||||
@@ -145,3 +145,8 @@ docs/.venv/
|
||||
*.temp
|
||||
tmp/
|
||||
temp/
|
||||
|
||||
# ----------------------
|
||||
# Runtime Data
|
||||
# ----------------------
|
||||
data/ai/bgp-briefs/
|
||||
@@ -102,6 +102,13 @@
|
||||
| Axios | HTTP 客户端 |
|
||||
| Socket.io-client | WebSocket 客户端 |
|
||||
| ECharts | 统计图表 |
|
||||
| Bun | 前端包管理与脚本运行 |
|
||||
|
||||
前端工程统一使用 Bun:
|
||||
|
||||
- 安装依赖使用 `bun install`
|
||||
- 运行脚本使用 `bun run <script>`
|
||||
- 不使用 `npm`、`pnpm`、`yarn`
|
||||
|
||||
### 虚幻引擎客户端
|
||||
|
||||
@@ -205,10 +212,161 @@
|
||||
./planet.sh health
|
||||
```
|
||||
|
||||
前端命令约定:
|
||||
|
||||
```bash
|
||||
cd frontend
|
||||
bun install
|
||||
bun run dev
|
||||
bun run build
|
||||
```
|
||||
|
||||
不要使用 `npm run ...`,避免在 WSL/Windows 混合环境里触发 `cmd.exe` 路径兼容问题。
|
||||
|
||||
## API 文档
|
||||
|
||||
启动服务后访问: `http://localhost:8000/docs`
|
||||
|
||||
## WSL / Windows 局域网访问
|
||||
|
||||
如果服务运行在 WSL 中,而你希望:
|
||||
|
||||
- Windows 本机浏览器访问开发服务
|
||||
- 同一局域网内的手机或其他电脑访问开发服务
|
||||
|
||||
推荐按下面顺序排查和配置。
|
||||
|
||||
### 1. 在 WSL 中启动服务
|
||||
|
||||
```bash
|
||||
./planet.sh start --allow-lan
|
||||
```
|
||||
|
||||
这会让前端监听 `0.0.0.0:3000`,后端监听 `0.0.0.0:8000`。
|
||||
|
||||
### 2. 先确认 WSL 内部服务正常
|
||||
|
||||
在 WSL 中执行:
|
||||
|
||||
```bash
|
||||
curl http://localhost:3000
|
||||
curl http://localhost:8000/health
|
||||
ss -ltnp | grep -E ':3000|:8000'
|
||||
```
|
||||
|
||||
预期:
|
||||
|
||||
- `3000` 返回前端 HTML
|
||||
- `8000/health` 返回健康检查 JSON
|
||||
- `ss` 中能看到 `0.0.0.0:3000` 和 `0.0.0.0:8000`
|
||||
|
||||
如果这一步不通,先不要继续做 Windows 转发。
|
||||
|
||||
### 3. 在 Windows 本机验证 localhost 直通
|
||||
|
||||
在 Windows PowerShell 中执行:
|
||||
|
||||
```powershell
|
||||
curl http://localhost:3000
|
||||
curl http://localhost:8000/health
|
||||
```
|
||||
|
||||
在常见的 WSL2 开发环境下,Windows 通常可以直接通过 `localhost` 访问 WSL 中的服务。
|
||||
|
||||
### 4. 如果需要让局域网设备访问,再做 Windows 端口转发
|
||||
|
||||
注意:下面的命令必须在“以管理员身份运行”的 PowerShell 中执行。
|
||||
|
||||
先把 Windows 对外网卡上的 `3000` / `8000` 转发到 Windows 本机 `127.0.0.1`:
|
||||
|
||||
```powershell
|
||||
netsh interface portproxy delete v4tov4 listenaddress=0.0.0.0 listenport=3000
|
||||
netsh interface portproxy delete v4tov4 listenaddress=0.0.0.0 listenport=8000
|
||||
|
||||
netsh interface portproxy add v4tov4 listenaddress=0.0.0.0 listenport=3000 connectaddress=127.0.0.1 connectport=3000
|
||||
netsh interface portproxy add v4tov4 listenaddress=0.0.0.0 listenport=8000 connectaddress=127.0.0.1 connectport=8000
|
||||
```
|
||||
|
||||
再放行 Windows 防火墙:
|
||||
|
||||
```powershell
|
||||
New-NetFirewallRule -DisplayName "WSL Planet 3000" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 3000
|
||||
New-NetFirewallRule -DisplayName "WSL Planet 8000" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 8000
|
||||
```
|
||||
|
||||
检查转发规则是否生效:
|
||||
|
||||
```powershell
|
||||
netsh interface portproxy show all
|
||||
```
|
||||
|
||||
预期能看到:
|
||||
|
||||
- `0.0.0.0:3000 -> 127.0.0.1:3000`
|
||||
- `0.0.0.0:8000 -> 127.0.0.1:8000`
|
||||
|
||||
### 5. 查 Windows 局域网 IP,并让其他设备访问
|
||||
|
||||
在 Windows PowerShell 中执行:
|
||||
|
||||
```powershell
|
||||
ipconfig
|
||||
```
|
||||
|
||||
找到当前联网网卡的 IPv4 地址,例如 `192.168.8.228`。
|
||||
|
||||
局域网其他设备可访问:
|
||||
|
||||
- `http://<Windows局域网IP>:3000/earth`
|
||||
- `http://<Windows局域网IP>:3000/admin`
|
||||
|
||||
例如:
|
||||
|
||||
- `http://192.168.8.228:3000/earth`
|
||||
|
||||
### 6. 常见现象与判断
|
||||
|
||||
- WSL 中 `curl localhost:3000` 能通,但 Windows 访问 `WSL 的局域网 IP:3000` 不通:这是正常现象之一,优先验证 Windows 的 `localhost:3000`
|
||||
- Windows `localhost:3000` 能通,但局域网设备访问 `Windows 局域网 IP:3000` 不通:通常缺少 `portproxy` 或防火墙放行
|
||||
- `whoami /groups` 中 `S-1-5-32-544` 显示 `deny only`:说明当前 PowerShell 不是提权管理员窗口
|
||||
|
||||
### 7. 本项目一次性验证顺序
|
||||
|
||||
建议固定按这个顺序验证:
|
||||
|
||||
1. WSL 中执行 `curl http://localhost:3000`
|
||||
2. WSL 中执行 `curl http://localhost:8000/health`
|
||||
3. Windows 中执行 `curl http://localhost:3000`
|
||||
4. Windows 中执行 `curl http://localhost:8000/health`
|
||||
5. 管理员 PowerShell 配置 `portproxy` 和防火墙
|
||||
6. 用手机或其他电脑访问 `http://<Windows局域网IP>:3000/earth`
|
||||
|
||||
## 启动容错参数
|
||||
|
||||
`planet.sh` 现在为依赖安装、数据库、AI Provider 启动加入了有限次重试,并会在数据库与 `aiprovider` 启动后额外等待 Docker healthcheck。
|
||||
|
||||
可通过环境变量临时调整:
|
||||
|
||||
```bash
|
||||
# 例: 放宽 AI Provider 与数据库在网络抖动下的自愈次数
|
||||
AI_PROVIDER_START_MAX_RETRIES=5 \
|
||||
AI_PROVIDER_RETRY_INTERVAL=10 \
|
||||
DATABASE_START_MAX_RETRIES=5 \
|
||||
DATABASE_RETRY_INTERVAL=10 \
|
||||
./planet.sh restart
|
||||
```
|
||||
|
||||
常用参数:
|
||||
|
||||
- `DEPENDENCY_INSTALL_MAX_RETRIES` / `DEPENDENCY_INSTALL_RETRY_INTERVAL`: 控制 `uv sync`、`bun install` 的重试次数与间隔,默认 `3` 次、`5` 秒
|
||||
- `DATABASE_START_MAX_RETRIES` / `DATABASE_RETRY_INTERVAL`: 控制 `postgres`、`redis` 的启动/重启与健康检查自愈,默认 `3` 次、`5` 秒
|
||||
- `AI_PROVIDER_START_MAX_RETRIES` / `AI_PROVIDER_RETRY_INTERVAL`: 控制 `aiprovider` 的构建/启动与容器重启自愈,默认 `3` 次、`5` 秒
|
||||
- `BACKEND_MAX_RETRIES`: 控制后端进程启动重试次数,默认 `3`
|
||||
- `FRONTEND_MAX_RETRIES`: 控制前端 dev server 启动重试次数,默认 `3`
|
||||
- `BACKEND_HEALTH_CHECK_ATTEMPTS` / `BACKEND_HEALTH_CHECK_INTERVAL`: 控制后端 HTTP 健康检查等待次数与间隔,默认 `10` 次、`2` 秒
|
||||
- `FRONTEND_HEALTH_CHECK_ATTEMPTS` / `FRONTEND_HEALTH_CHECK_INTERVAL`: 控制前端 HTTP 可访问检查等待次数与间隔,默认 `10` 次、`2` 秒
|
||||
- `AI_PROVIDER_HEALTH_CHECK_ATTEMPTS` / `AI_PROVIDER_HEALTH_CHECK_INTERVAL`: 控制 `aiprovider` HTTP 健康检查等待次数与间隔,默认 `10` 次、`2` 秒
|
||||
|
||||
## AI 接口预留
|
||||
|
||||
项目现在采用“两层”设计:
|
||||
@@ -284,8 +442,25 @@ AI_PROVIDER_SERVICE_TOKEN=change_me
|
||||
|
||||
详细文档:
|
||||
|
||||
- [docs/aiprovider.md](/home/ray/dev/linkong/planet/docs/aiprovider.md)
|
||||
- [docs/technical/agents-aiprovider.md](/home/ray/dev/linkong/planet/docs/technical/agents-aiprovider.md)
|
||||
- [aiprovider/README.md](/home/ray/dev/linkong/planet/aiprovider/README.md)
|
||||
- [docs/technical/frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md)
|
||||
- [docs/plans/frontend-ai-playground-development-plan.md](/home/ray/dev/linkong/planet/docs/plans/frontend-ai-playground-development-plan.md)
|
||||
- [docs/plans/agents-situational-awareness-foundation-plan.md](/home/ray/dev/linkong/planet/docs/plans/agents-situational-awareness-foundation-plan.md)
|
||||
|
||||
## 前端页面布局规范
|
||||
|
||||
管理后台页面默认遵循“单屏工作区”原则:
|
||||
|
||||
- 页头、摘要区、主工作区应在一屏内形成稳定结构
|
||||
- 主表格 / 主图表 / 主分析区应占据页面主要可视空间
|
||||
- 模块内容超出时优先在卡片、表格、标签页内部滚动
|
||||
- 不依赖整页纵向撑开来容纳主要工作区
|
||||
|
||||
当前推荐参考实现:
|
||||
|
||||
- [frontend/src/pages/BGP/BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx)
|
||||
- [docs/technical/frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md)
|
||||
|
||||
## License
|
||||
|
||||
|
||||
@@ -13,7 +13,25 @@
|
||||
- [x] 接入 `IPtoASN / IPtoCountry` 作为 prefix-centric geography 的主数据源
|
||||
- [x] 接入 `OpenGeoFeed` 作为 prefix geography 的高质量覆盖/override 数据源
|
||||
- [x] 把 RIR delegated 设计成 prefix geography 的 fallback,而不是主来源
|
||||
- [ ] 为 `aiprovider` 建立 `provider -> api adapter -> compat policy` 的配置中心,优先落成 `json` 或 `yaml` 文件,运行时按 `provider/model` 读取兼容设置,而不是把专项兼容继续散落在 Python 分支里
|
||||
- [ ] 为市面上主流 AI 服务补专项兼容配置并固化到配置文件中,至少覆盖 `OpenAI / Anthropic / MiniMax / Ollama / Moonshot / DeepSeek / Qwen / GLM / Gemini / OpenRouter / vLLM / LM Studio / One API`
|
||||
- [ ] 在兼容配置中补齐可声明项:`api adapter`、`base_url pattern`、`auth header`、`thinking default`、`reasoning block mapping`、`stream path`、`tool-call capability`、`multimodal capability`、`provider-specific request patch`
|
||||
- [ ] 接入 `inetnum` / `inet6num` whois 作为比 RIR 更细粒度的后备层
|
||||
- [x] 在 activity layer 之后继续补 `route leak` 和 `path instability / flap` detector
|
||||
- [ ] 对 [frontend/public/earth/js/bgp.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp.js) 做按职责拆分的小重构,拆成 data / markers / overlays / animation,降低后续维护复杂度
|
||||
- [ ] 可选优化(非必做):将 BGP incident/collector 标点改为 HTML marker(参考 worldmonitor 的 `htmlElementsData` 思路),实现近乎固定屏幕尺寸与更高密度可点击性
|
||||
- [ ] 保持 Earth 当前这批纯个人偏好设置继续走本地持久化:`旋转模式`、HUD 面板显示/隐藏、`地形透明度` 暂不升级到后端系统设置,避免把设备级偏好过早做成全局配置
|
||||
- [ ] 如果后续明确需要“账号级同步 Earth 偏好”,再单独设计 `Earth user preferences`:优先按用户维度而不是全局系统设置保存,并规划 `localStorage -> backend` 的平滑迁移策略
|
||||
- [ ] 为 Planet / Earth 补一个可用的日志查看系统:先明确前后端/AI Provider/采集任务的日志入口、最近日志聚合、筛选与 tail 能力,再决定是先做脚本级统一入口还是控制台内置日志面板
|
||||
- [ ] 把 Earth 态势新闻源从 [earth_news.py](/home/ray/dev/linkong/planet/backend/app/services/earth_news.py) 的硬编码列表抽成可配置目录,优先保持当前“实时聚合”链路不变,只先解决新闻源不可配置的问题
|
||||
- [ ] 为 Earth 态势新闻设计后续采集器化方案:明确新闻数据模型、去重策略、区域映射、过期清理和 Earth/AI 复用方式,再决定何时把新闻从实时抓取升级成正式 collector
|
||||
- [ ] 为 Earth 地球表面增加一层与基础纹理对齐的材质/纹理 overlay,并在同层叠加国界轮廓参考线;要求国界线与底图稳定对齐,且 hover 到国家轮廓时能高亮当前国家,便于校准地表和增强交互
|
||||
- [ ] 把 Earth 新闻接入通用巡航队列:按新闻发生地和时间排序生成巡航目标,巡航聚焦到新闻事件时显示对应新闻卡片,并保持实现边界为“通用巡航层 + 新闻业务适配层”,不要再把新闻逻辑直接耦合回 `main.js` 状态机
|
||||
- [ ] 为未知位置的算力中心建立分层坐标补全链路:优先 `精确坐标 > 站点/园区命中 > 城市 > 州/省 > 国家内主要算力城市 > 国家质心`,并把每次回退的 `confidence / reason / precision` 明确写进统一 GeoJSON
|
||||
- [ ] 为算力中心补一份可维护的本地位置注册表,例如 `canonical_name / aliases / operator / country / region / city / lat / lon / confidence / source_note`,避免把地点知识长期硬编码在 `visualization.py`
|
||||
- [ ] 增强 `epoch_ai_gpu` 和相关算力采集器的源页面解析:即使公开 API 不给坐标,也继续尝试从详情页、HTML、内嵌 JSON、schema.org、OpenGraph、脚本变量和 PDF/新闻稿链接里抽地点线索
|
||||
- [ ] 为未知位置算力中心增加外部富化策略评估:可选接入公开知识源或搜索兜底,只抓“站点名/园区名/城市名”级别线索,不直接抓经纬度结论,并把结果作为候选证据而不是真值
|
||||
- [ ] 为算力中心建立 `operator / cluster name / facility alias` 归一化层,先解决 `xAI / Colossus / Memphis`、`OpenAI / Stargate`、`CoreWeave`、`Lambda`、`Crusoe` 这类同一对象多种写法导致的地点匹配失败
|
||||
- [ ] 为估算位置增加更细的视觉和产品表达:除了问号角标,还要支持 tooltip/详情中的“估算依据”“精度级别”“最后核验时间”,并允许在设置中单独开关“仅看精确位置”
|
||||
- [ ] 为国家级估算点设计更合理的落点策略:优先落在“该国主要算力/数据中心城市候选集”而不是几何质心,必要时同国多节点做稳定散列分配,避免大量节点堆在荒漠或海上
|
||||
- [ ] 为未知位置算力中心建立人工校验工作流:支持导出待核验清单、记录人工确认结果,并把人工确认反哺到位置注册表,逐步减少问号点比例
|
||||
+35
-13
@@ -6,29 +6,51 @@ AI_TIMEOUT_SECONDS=60
|
||||
AI_HTTP_RETRY_ATTEMPTS=2
|
||||
AI_ANALYSIS_SYSTEM_PROMPT=你是态势感知分析助手。请基于输入的上下文、观测与约束,输出结构化、克制、可执行的分析。
|
||||
|
||||
# Select one provider mode:
|
||||
# - openai_compatible
|
||||
# - claude_compatible
|
||||
# Provider identity. Recommended values:
|
||||
# - minimax
|
||||
# - openai
|
||||
# - ollama
|
||||
AI_PROVIDER=ollama
|
||||
# Compatibility aliases still accepted:
|
||||
# - openai_compatible
|
||||
# - anthropic_compatible
|
||||
# - claude_compatible
|
||||
AI_PROVIDER=minimax
|
||||
|
||||
# Request adapter style, following OpenClaw's API-seam pattern:
|
||||
# - auto
|
||||
# - openai-completions
|
||||
# - anthropic-messages
|
||||
# - ollama-generate
|
||||
AI_PROVIDER_API=anthropic-messages
|
||||
|
||||
# Common model selection
|
||||
AI_MODEL=qwen2.5:7b
|
||||
AI_MODEL=MiniMax-M2.7
|
||||
|
||||
# MiniMax CN Anthropic-compatible example
|
||||
AI_BASE_URL=https://api.minimaxi.com/anthropic
|
||||
AI_API_KEY=sk-cp-change-me
|
||||
AI_MAX_TOKENS=1200
|
||||
AI_ANTHROPIC_VERSION=2023-06-01
|
||||
|
||||
# OpenAI-compatible example (vLLM / LM Studio / One API / local gateway)
|
||||
# AI_PROVIDER=openai_compatible
|
||||
# AI_PROVIDER=openai
|
||||
# AI_PROVIDER_API=openai-completions
|
||||
# AI_BASE_URL=http://127.0.0.1:8001/v1
|
||||
# AI_API_KEY=local-key
|
||||
# AI_MODEL=your-local-model
|
||||
|
||||
# Claude-compatible example (Anthropic / MiniMax / Claude-compatible gateway)
|
||||
# AI_PROVIDER=claude_compatible
|
||||
# AI_BASE_URL=http://127.0.0.1:8002
|
||||
# Anthropic-compatible example (Claude-compatible gateway)
|
||||
# AI_PROVIDER=anthropic
|
||||
# AI_PROVIDER_API=anthropic-messages
|
||||
# AI_BASE_URL=http://127.0.0.1:8002/anthropic
|
||||
# AI_API_KEY=local-key
|
||||
# AI_MODEL=your-model
|
||||
# AI_MAX_TOKENS=1200
|
||||
# AI_ANTHROPIC_VERSION=2023-06-01
|
||||
|
||||
# Ollama native example
|
||||
AI_BASE_URL=http://127.0.0.1:11434
|
||||
AI_API_KEY=
|
||||
AI_MAX_TOKENS=1200
|
||||
AI_ANTHROPIC_VERSION=2023-06-01
|
||||
# AI_PROVIDER=ollama
|
||||
# AI_PROVIDER_API=ollama-generate
|
||||
# AI_BASE_URL=http://127.0.0.1:11434
|
||||
# AI_API_KEY=
|
||||
# AI_MODEL=qwen2.5:7b
|
||||
@@ -1,6 +1,10 @@
|
||||
FROM python:3.14-slim
|
||||
ARG PYTHON_IMAGE=python:3.14-slim
|
||||
ARG UV_IMAGE=ghcr.io/astral-sh/uv:latest
|
||||
|
||||
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
|
||||
FROM ${UV_IMAGE} AS uv
|
||||
FROM ${PYTHON_IMAGE}
|
||||
|
||||
COPY --from=uv /uv /uvx /bin/
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
|
||||
+30
-16
@@ -4,21 +4,31 @@
|
||||
|
||||
完整使用说明见:
|
||||
|
||||
- [docs/aiprovider.md](/home/ray/dev/linkong/planet/docs/aiprovider.md)
|
||||
- [docs/agents/aiprovider.md](/home/ray/dev/linkong/planet/docs/agents/aiprovider.md)
|
||||
|
||||
当前支持:
|
||||
|
||||
- `AI_PROVIDER=openai`
|
||||
- `AI_PROVIDER=openai_compatible`
|
||||
- `AI_PROVIDER=anthropic`
|
||||
- `AI_PROVIDER=anthropic_compatible`
|
||||
- `AI_PROVIDER=claude_compatible`
|
||||
- `AI_PROVIDER=ollama`
|
||||
- provider identity:
|
||||
- `AI_PROVIDER=openai`
|
||||
- `AI_PROVIDER=anthropic`
|
||||
- `AI_PROVIDER=minimax`
|
||||
- `AI_PROVIDER=ollama`
|
||||
- request adapter:
|
||||
- `AI_PROVIDER_API=openai-completions`
|
||||
- `AI_PROVIDER_API=anthropic-messages`
|
||||
- `AI_PROVIDER_API=ollama-generate`
|
||||
|
||||
兼容别名仍然保留:
|
||||
|
||||
- `openai_compatible`
|
||||
- `anthropic_compatible`
|
||||
- `claude_compatible`
|
||||
|
||||
典型配置:
|
||||
|
||||
```env
|
||||
AI_PROVIDER=openai_compatible
|
||||
AI_PROVIDER=openai
|
||||
AI_PROVIDER_API=openai-completions
|
||||
AI_BASE_URL=https://api.openai.com/v1
|
||||
AI_API_KEY=your_api_key
|
||||
AI_MODEL=gpt-4o-mini
|
||||
@@ -26,13 +36,14 @@ AI_TIMEOUT_SECONDS=60
|
||||
AI_PROVIDER_SERVICE_TOKEN=change_me
|
||||
```
|
||||
|
||||
Claude 兼容供应商示例:
|
||||
MiniMax 中国大陆节点示例:
|
||||
|
||||
```env
|
||||
AI_PROVIDER=claude_compatible
|
||||
AI_BASE_URL=https://your-claude-compatible-endpoint.example.com
|
||||
AI_API_KEY=your_api_key
|
||||
AI_MODEL=your-claude-compatible-model
|
||||
AI_PROVIDER=minimax
|
||||
AI_PROVIDER_API=anthropic-messages
|
||||
AI_BASE_URL=https://api.minimaxi.com/anthropic
|
||||
AI_API_KEY=sk-cp-xxxxx
|
||||
AI_MODEL=MiniMax-M2.7
|
||||
AI_TIMEOUT_SECONDS=60
|
||||
AI_MAX_TOKENS=1200
|
||||
AI_ANTHROPIC_VERSION=2023-06-01
|
||||
@@ -43,12 +54,15 @@ AI_PROVIDER_SERVICE_TOKEN=change_me
|
||||
|
||||
- Anthropic 官方 Claude API
|
||||
- Claude 兼容网关
|
||||
- MiniMax 等提供 Claude/Anthropic 风格消息接口的服务
|
||||
- MiniMax 等提供 Anthropic Messages 风格接口的服务
|
||||
|
||||
这套命名方式参考了 OpenClaw 的接入模式: provider 负责标识供应商, `AI_PROVIDER_API` 负责标识协议适配层, 避免把“供应商”和“协议”绑死在一起。
|
||||
|
||||
Ollama 原生示例:
|
||||
|
||||
```env
|
||||
AI_PROVIDER=ollama
|
||||
AI_PROVIDER_API=ollama-generate
|
||||
AI_BASE_URL=http://127.0.0.1:11434
|
||||
AI_API_KEY=
|
||||
AI_MODEL=qwen2.5:7b
|
||||
@@ -58,8 +72,8 @@ AI_PROVIDER_SERVICE_TOKEN=change_me
|
||||
|
||||
本地模型接入建议:
|
||||
|
||||
- `vLLM`、`LM Studio`、`One API`:优先使用 `openai_compatible`
|
||||
- `MiniMax`、Claude 兼容网关:使用 `claude_compatible`
|
||||
- `vLLM`、`LM Studio`、`One API`:`AI_PROVIDER=openai` + `AI_PROVIDER_API=openai-completions`
|
||||
- `MiniMax`、Claude 兼容网关:`AI_PROVIDER=minimax|anthropic` + `AI_PROVIDER_API=anthropic-messages`
|
||||
- `Ollama`:可直接使用 `ollama`
|
||||
|
||||
启动模板:
|
||||
|
||||
@@ -9,6 +9,7 @@ class Settings(BaseSettings):
|
||||
SERVICE_VERSION: str = "0.1.0"
|
||||
|
||||
AI_PROVIDER: str = "disabled"
|
||||
AI_PROVIDER_API: str = "auto"
|
||||
AI_BASE_URL: str = "https://api.openai.com/v1"
|
||||
AI_API_KEY: str = ""
|
||||
AI_MODEL: str = ""
|
||||
|
||||
@@ -8,6 +8,7 @@ from fastapi import HTTPException, status
|
||||
|
||||
from aiprovider.config import settings
|
||||
from aiprovider.schemas import (
|
||||
AIContentBlock,
|
||||
AIProviderStatusResponse,
|
||||
SituationalAnalysisRequest,
|
||||
SituationalAnalysisResponse,
|
||||
@@ -18,9 +19,39 @@ def _normalize_provider(value: str) -> str:
|
||||
return (value or "disabled").strip().lower()
|
||||
|
||||
|
||||
def _normalize_provider_api(value: str) -> str:
|
||||
return (value or "auto").strip().lower().replace("_", "-")
|
||||
|
||||
|
||||
def _resolve_provider_api(provider: str, configured_api: str) -> str:
|
||||
if configured_api and configured_api != "auto":
|
||||
return configured_api
|
||||
|
||||
if provider in {"openai", "openai-compatible", "openai_compatible"}:
|
||||
return "openai-completions"
|
||||
if provider in {
|
||||
"anthropic",
|
||||
"anthropic-compatible",
|
||||
"anthropic_compatible",
|
||||
"claude-compatible",
|
||||
"claude_compatible",
|
||||
"minimax",
|
||||
"kimi-coding",
|
||||
"moonshot-anthropic",
|
||||
}:
|
||||
return "anthropic-messages"
|
||||
if provider == "ollama":
|
||||
return "ollama-generate"
|
||||
return "disabled"
|
||||
|
||||
|
||||
class ProviderService:
|
||||
def __init__(self) -> None:
|
||||
self.provider = _normalize_provider(settings.AI_PROVIDER)
|
||||
self.provider_api = _resolve_provider_api(
|
||||
self.provider,
|
||||
_normalize_provider_api(settings.AI_PROVIDER_API),
|
||||
)
|
||||
self.base_url = settings.AI_BASE_URL.rstrip("/")
|
||||
self.api_key = settings.AI_API_KEY
|
||||
self.default_model = settings.AI_MODEL
|
||||
@@ -32,9 +63,11 @@ class ProviderService:
|
||||
|
||||
def get_status(self) -> AIProviderStatusResponse:
|
||||
enabled = self.provider != "disabled"
|
||||
configured = enabled and bool(self.base_url and self.api_key and self.default_model)
|
||||
has_credentials = bool(self.api_key) if self._requires_api_key() else True
|
||||
configured = enabled and bool(self.base_url and has_credentials and self.default_model)
|
||||
return AIProviderStatusResponse(
|
||||
provider=self.provider,
|
||||
api=self.provider_api if enabled else None,
|
||||
enabled=enabled,
|
||||
configured=configured,
|
||||
model=self.default_model or None,
|
||||
@@ -49,7 +82,8 @@ class ProviderService:
|
||||
)
|
||||
|
||||
model = payload.preferred_model or self.default_model
|
||||
if not self.base_url or not self.api_key or not model:
|
||||
has_credentials = bool(self.api_key) if self._requires_api_key() else True
|
||||
if not self.base_url or not has_credentials or not model:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
|
||||
detail="AI provider is not fully configured. Check AI_BASE_URL, AI_API_KEY, and AI_MODEL.",
|
||||
@@ -57,28 +91,40 @@ class ProviderService:
|
||||
|
||||
prompt = self._build_prompt(payload)
|
||||
|
||||
if self.provider in {"openai", "openai_compatible"}:
|
||||
if self.provider_api == "openai-completions":
|
||||
data = await self._request_openai_compatible(model, prompt)
|
||||
content = self._extract_openai_content(data)
|
||||
elif self.provider in {"anthropic", "anthropic_compatible", "claude_compatible"}:
|
||||
data = await self._request_anthropic_compatible(model, prompt)
|
||||
content_blocks = self._extract_openai_blocks(data)
|
||||
elif self.provider_api == "anthropic-messages":
|
||||
data = await self._request_anthropic_messages(model, prompt, payload.thinking)
|
||||
content = self._extract_anthropic_content(data)
|
||||
elif self.provider == "ollama":
|
||||
content_blocks = self._extract_anthropic_blocks(data)
|
||||
elif self.provider_api == "ollama-generate":
|
||||
data = await self._request_ollama(model, prompt)
|
||||
content = self._extract_ollama_content(data)
|
||||
content_blocks = self._extract_ollama_blocks(data)
|
||||
else:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
detail=f"Unsupported AI provider: {self.provider}",
|
||||
detail=f"Unsupported AI provider API: {self.provider_api}",
|
||||
)
|
||||
|
||||
text_blocks = [block.text for block in content_blocks if block.text]
|
||||
thinking_blocks = [block.thinking for block in content_blocks if block.thinking]
|
||||
|
||||
return SituationalAnalysisResponse(
|
||||
provider=self.provider,
|
||||
model=model,
|
||||
content=content,
|
||||
content_blocks=content_blocks,
|
||||
text_blocks=text_blocks,
|
||||
thinking_blocks=thinking_blocks,
|
||||
raw_response=data,
|
||||
)
|
||||
|
||||
def _requires_api_key(self) -> bool:
|
||||
return self.provider_api != "ollama-generate"
|
||||
|
||||
def _build_prompt(self, payload: SituationalAnalysisRequest) -> str:
|
||||
sections = [
|
||||
f"任务标题:\n{payload.title}",
|
||||
@@ -113,7 +159,12 @@ class ProviderService:
|
||||
request_body=request_body,
|
||||
)
|
||||
|
||||
async def _request_anthropic_compatible(self, model: str, prompt: str) -> dict[str, Any]:
|
||||
async def _request_anthropic_messages(
|
||||
self,
|
||||
model: str,
|
||||
prompt: str,
|
||||
thinking: dict[str, Any] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
request_body = {
|
||||
"model": model,
|
||||
"system": self.system_prompt,
|
||||
@@ -131,8 +182,15 @@ class ProviderService:
|
||||
"max_tokens": self.max_tokens,
|
||||
"temperature": 0.2,
|
||||
}
|
||||
resolved_thinking = self._resolve_anthropic_thinking(thinking)
|
||||
if resolved_thinking:
|
||||
request_body["thinking"] = resolved_thinking
|
||||
if self.provider == "minimax" and self.base_url.endswith("/anthropic"):
|
||||
path = "/v1/messages"
|
||||
else:
|
||||
path = "/messages"
|
||||
return await self._post(
|
||||
path="/messages",
|
||||
path=path,
|
||||
headers={
|
||||
"x-api-key": self.api_key,
|
||||
"anthropic-version": self.anthropic_version,
|
||||
@@ -141,6 +199,25 @@ class ProviderService:
|
||||
request_body=request_body,
|
||||
)
|
||||
|
||||
def _resolve_anthropic_thinking(self, thinking: dict[str, Any] | None) -> dict[str, Any] | None:
|
||||
if thinking:
|
||||
return thinking
|
||||
|
||||
# OpenClaw treats MiniMax's Anthropic-compatible path specially:
|
||||
# disable thinking by default unless the caller explicitly opts in.
|
||||
if self.provider == "minimax":
|
||||
return {"type": "disabled"}
|
||||
|
||||
return None
|
||||
|
||||
async def _request_anthropic_compatible(
|
||||
self,
|
||||
model: str,
|
||||
prompt: str,
|
||||
thinking: dict[str, Any] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
return await self._request_anthropic_messages(model, prompt, thinking)
|
||||
|
||||
async def _request_ollama(self, model: str, prompt: str) -> dict[str, Any]:
|
||||
request_body = {
|
||||
"model": model,
|
||||
@@ -218,6 +295,30 @@ class ProviderService:
|
||||
)
|
||||
return ""
|
||||
|
||||
def _extract_openai_blocks(self, payload: dict[str, Any]) -> list[AIContentBlock]:
|
||||
choices = payload.get("choices") or []
|
||||
if not choices:
|
||||
return []
|
||||
|
||||
message = choices[0].get("message") or {}
|
||||
content = message.get("content")
|
||||
if isinstance(content, str):
|
||||
return [AIContentBlock(type="text", text=content)]
|
||||
if not isinstance(content, list):
|
||||
return []
|
||||
|
||||
blocks: list[AIContentBlock] = []
|
||||
for item in content:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
blocks.append(
|
||||
AIContentBlock(
|
||||
type=str(item.get("type", "text")),
|
||||
text=item.get("text") if isinstance(item.get("text"), str) else None,
|
||||
metadata={k: v for k, v in item.items() if k not in {"type", "text"}},
|
||||
)
|
||||
)
|
||||
return blocks
|
||||
def _extract_anthropic_content(self, payload: dict[str, Any]) -> str:
|
||||
content = payload.get("content")
|
||||
if isinstance(content, str):
|
||||
@@ -233,8 +334,39 @@ class ProviderService:
|
||||
fragments.append(item["text"])
|
||||
return "".join(fragments)
|
||||
|
||||
def _extract_anthropic_blocks(self, payload: dict[str, Any]) -> list[AIContentBlock]:
|
||||
content = payload.get("content")
|
||||
if not isinstance(content, list):
|
||||
return []
|
||||
|
||||
blocks: list[AIContentBlock] = []
|
||||
for item in content:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
blocks.append(
|
||||
AIContentBlock(
|
||||
type=str(item.get("type", "unknown")),
|
||||
text=item.get("text") if isinstance(item.get("text"), str) else None,
|
||||
thinking=item.get("thinking") if isinstance(item.get("thinking"), str) else None,
|
||||
signature=item.get("signature") if isinstance(item.get("signature"), str) else None,
|
||||
metadata={
|
||||
k: v
|
||||
for k, v in item.items()
|
||||
if k not in {"type", "text", "thinking", "signature"}
|
||||
},
|
||||
)
|
||||
)
|
||||
return blocks
|
||||
|
||||
|
||||
def _extract_ollama_content(self, payload: dict[str, Any]) -> str:
|
||||
response = payload.get("response")
|
||||
if isinstance(response, str):
|
||||
return response
|
||||
return ""
|
||||
|
||||
def _extract_ollama_blocks(self, payload: dict[str, Any]) -> list[AIContentBlock]:
|
||||
response = payload.get("response")
|
||||
if isinstance(response, str) and response:
|
||||
return [AIContentBlock(type="text", text=response)]
|
||||
return []
|
||||
@@ -3,6 +3,14 @@ from typing import Any
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class AIContentBlock(BaseModel):
|
||||
type: str
|
||||
text: str | None = None
|
||||
thinking: str | None = None
|
||||
signature: str | None = None
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class SituationalAnalysisRequest(BaseModel):
|
||||
title: str = Field(..., min_length=1, max_length=200)
|
||||
objective: str = Field(..., min_length=1, max_length=1000)
|
||||
@@ -10,17 +18,22 @@ class SituationalAnalysisRequest(BaseModel):
|
||||
observations: list[str] = Field(default_factory=list)
|
||||
constraints: list[str] = Field(default_factory=list)
|
||||
preferred_model: str | None = Field(default=None, max_length=200)
|
||||
thinking: dict[str, Any] | None = None
|
||||
|
||||
|
||||
class SituationalAnalysisResponse(BaseModel):
|
||||
provider: str
|
||||
model: str
|
||||
content: str
|
||||
content_blocks: list[AIContentBlock] = Field(default_factory=list)
|
||||
text_blocks: list[str] = Field(default_factory=list)
|
||||
thinking_blocks: list[str] = Field(default_factory=list)
|
||||
raw_response: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class AIProviderStatusResponse(BaseModel):
|
||||
provider: str
|
||||
api: str | None = None
|
||||
enabled: bool
|
||||
configured: bool
|
||||
model: str | None = None
|
||||
|
||||
+6
-2
@@ -1,6 +1,10 @@
|
||||
FROM python:3.14-slim
|
||||
ARG PYTHON_IMAGE=python:3.14-slim
|
||||
ARG UV_IMAGE=ghcr.io/astral-sh/uv:latest
|
||||
|
||||
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
|
||||
FROM ${UV_IMAGE} AS uv
|
||||
FROM ${PYTHON_IMAGE}
|
||||
|
||||
COPY --from=uv /uv /uvx /bin/
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
|
||||
@@ -13,7 +13,9 @@ from app.api.v1 import (
|
||||
collected_data,
|
||||
visualization,
|
||||
bgp,
|
||||
news,
|
||||
system_control,
|
||||
tv,
|
||||
)
|
||||
|
||||
api_router = APIRouter()
|
||||
@@ -33,3 +35,5 @@ api_router.include_router(settings.router, prefix="/settings", tags=["settings"]
|
||||
api_router.include_router(system_control.router, prefix="/system", tags=["system"])
|
||||
api_router.include_router(visualization.router, prefix="/visualization", tags=["visualization"])
|
||||
api_router.include_router(bgp.router, prefix="/bgp", tags=["bgp"])
|
||||
api_router.include_router(tv.router, prefix="/tv", tags=["tv"])
|
||||
api_router.include_router(news.router, prefix="/news", tags=["news"])
|
||||
+243
-1
@@ -1,15 +1,52 @@
|
||||
from uuid import uuid4
|
||||
|
||||
from fastapi import APIRouter, Depends, Request, Response
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request, Response
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.security import get_current_user
|
||||
from app.db.session import get_db
|
||||
from app.models.user import User
|
||||
from app.schemas.ai import (
|
||||
AIProviderStatusResponse,
|
||||
AlertBriefRequest,
|
||||
AlertBriefResponse,
|
||||
BGPBriefRequest,
|
||||
BGPBriefRecordResponse,
|
||||
BGPBriefRecordSummary,
|
||||
PlaygroundMessageActionResponse,
|
||||
PlaygroundMessageCreateRequest,
|
||||
PlaygroundMessageEditRequest,
|
||||
PlaygroundMessageResendRequest,
|
||||
PlaygroundMessageStopRequest,
|
||||
PlaygroundSessionResponse,
|
||||
PlaygroundSessionUpsertRequest,
|
||||
PlaygroundThreadResponse,
|
||||
SituationalAlertBriefRequest,
|
||||
SituationalAlertBriefResponse,
|
||||
SituationalAnalysisRequest,
|
||||
SituationalAnalysisResponse,
|
||||
)
|
||||
from app.services.alert_ai_brief import build_alert_brief_request
|
||||
from app.services.ai_client import AIProviderClient, get_ai_provider_client
|
||||
from app.services.bgp_ai_brief import build_bgp_brief_request
|
||||
from app.services.bgp_ai_brief_store import (
|
||||
get_bgp_brief_record,
|
||||
get_latest_bgp_brief_record,
|
||||
list_bgp_brief_records,
|
||||
save_bgp_brief_record,
|
||||
)
|
||||
from app.services.playground_session_store import (
|
||||
get_playground_session,
|
||||
upsert_playground_session,
|
||||
)
|
||||
from app.services.playground_chat_service import (
|
||||
create_turn,
|
||||
edit_user_message,
|
||||
get_thread,
|
||||
resend_turn,
|
||||
stop_message,
|
||||
)
|
||||
from app.services.situational_alert_ai_brief import build_situational_alert_brief_request
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
@@ -37,3 +74,208 @@ async def analyze_situational_awareness(
|
||||
request_id = request.headers.get("X-Request-ID") or str(uuid4())
|
||||
response.headers["X-Request-ID"] = request_id
|
||||
return await provider_client.analyze(payload, request_id=request_id)
|
||||
|
||||
|
||||
@router.get("/playground/thread", response_model=PlaygroundThreadResponse | None)
|
||||
async def get_playground_thread(
|
||||
session_key: str = "default",
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await get_thread(
|
||||
db,
|
||||
user_id=current_user.id,
|
||||
session_key=session_key,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/playground/session", response_model=PlaygroundSessionResponse | None)
|
||||
async def get_saved_playground_session(
|
||||
session_key: str = "default",
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await get_playground_session(
|
||||
db,
|
||||
user_id=current_user.id,
|
||||
session_key=session_key,
|
||||
)
|
||||
|
||||
|
||||
@router.put("/playground/session", response_model=PlaygroundSessionResponse)
|
||||
async def save_playground_session(
|
||||
payload: PlaygroundSessionUpsertRequest,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await upsert_playground_session(
|
||||
db,
|
||||
user_id=current_user.id,
|
||||
payload=payload,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/playground/messages", response_model=PlaygroundMessageActionResponse)
|
||||
async def create_playground_message(
|
||||
payload: PlaygroundMessageCreateRequest,
|
||||
current_user: User = Depends(get_current_user),
|
||||
provider_client: AIProviderClient = Depends(get_ai_provider_client),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await create_turn(
|
||||
db,
|
||||
user_id=current_user.id,
|
||||
payload=payload,
|
||||
provider_client=provider_client,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/playground/messages/stop", response_model=PlaygroundMessageActionResponse)
|
||||
async def stop_playground_message(
|
||||
payload: PlaygroundMessageStopRequest,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await stop_message(
|
||||
db,
|
||||
user_id=current_user.id,
|
||||
payload=payload,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/playground/messages/resend", response_model=PlaygroundMessageActionResponse)
|
||||
async def resend_playground_message(
|
||||
payload: PlaygroundMessageResendRequest,
|
||||
current_user: User = Depends(get_current_user),
|
||||
provider_client: AIProviderClient = Depends(get_ai_provider_client),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await resend_turn(
|
||||
db,
|
||||
user_id=current_user.id,
|
||||
payload=payload,
|
||||
provider_client=provider_client,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/playground/messages/edit", response_model=PlaygroundMessageActionResponse)
|
||||
async def edit_playground_message(
|
||||
payload: PlaygroundMessageEditRequest,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await edit_user_message(
|
||||
db,
|
||||
user_id=current_user.id,
|
||||
payload=payload,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/bgp/briefs", response_model=list[BGPBriefRecordSummary])
|
||||
async def list_saved_bgp_briefs(
|
||||
current_user: User = Depends(get_current_user),
|
||||
):
|
||||
return list_bgp_brief_records()
|
||||
|
||||
|
||||
@router.get("/bgp/briefs/latest", response_model=BGPBriefRecordResponse | None)
|
||||
async def get_latest_saved_bgp_brief(
|
||||
current_user: User = Depends(get_current_user),
|
||||
):
|
||||
return get_latest_bgp_brief_record()
|
||||
|
||||
|
||||
@router.get("/bgp/briefs/{brief_id}", response_model=BGPBriefRecordResponse)
|
||||
async def get_saved_bgp_brief(
|
||||
brief_id: str,
|
||||
current_user: User = Depends(get_current_user),
|
||||
):
|
||||
record = get_bgp_brief_record(brief_id)
|
||||
if record is None:
|
||||
raise HTTPException(status_code=404, detail="BGP brief not found")
|
||||
return record
|
||||
|
||||
|
||||
@router.post("/bgp/brief", response_model=BGPBriefRecordResponse)
|
||||
async def analyze_bgp_brief(
|
||||
payload: BGPBriefRequest,
|
||||
request: Request,
|
||||
response: Response,
|
||||
current_user: User = Depends(get_current_user),
|
||||
provider_client: AIProviderClient = Depends(get_ai_provider_client),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
request_id = request.headers.get("X-Request-ID") or str(uuid4())
|
||||
response.headers["X-Request-ID"] = request_id
|
||||
|
||||
brief_request, facts, context = await build_bgp_brief_request(
|
||||
db,
|
||||
incident_limit=payload.incident_limit,
|
||||
anomaly_limit=payload.anomaly_limit,
|
||||
collector_limit=payload.collector_limit,
|
||||
)
|
||||
brief_request.preferred_model = payload.preferred_model
|
||||
brief_request.thinking = payload.thinking
|
||||
|
||||
analysis = await provider_client.analyze(brief_request, request_id=request_id)
|
||||
return save_bgp_brief_record(
|
||||
analysis,
|
||||
request_id=request_id,
|
||||
facts=facts,
|
||||
context=context,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/alerts/brief", response_model=AlertBriefResponse)
|
||||
async def analyze_alert_brief(
|
||||
payload: AlertBriefRequest,
|
||||
request: Request,
|
||||
response: Response,
|
||||
current_user: User = Depends(get_current_user),
|
||||
provider_client: AIProviderClient = Depends(get_ai_provider_client),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
request_id = request.headers.get("X-Request-ID") or str(uuid4())
|
||||
response.headers["X-Request-ID"] = request_id
|
||||
|
||||
brief_request, facts, context = await build_alert_brief_request(
|
||||
db,
|
||||
alert_limit=payload.alert_limit,
|
||||
)
|
||||
brief_request.preferred_model = payload.preferred_model
|
||||
brief_request.thinking = payload.thinking
|
||||
|
||||
analysis = await provider_client.analyze(brief_request, request_id=request_id)
|
||||
return AlertBriefResponse(
|
||||
**analysis.model_dump(),
|
||||
title=brief_request.title,
|
||||
objective=brief_request.objective,
|
||||
facts=facts,
|
||||
context=context,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/situational-alerts/brief", response_model=SituationalAlertBriefResponse)
|
||||
async def analyze_situational_alert_brief(
|
||||
payload: SituationalAlertBriefRequest,
|
||||
request: Request,
|
||||
response: Response,
|
||||
current_user: User = Depends(get_current_user),
|
||||
provider_client: AIProviderClient = Depends(get_ai_provider_client),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
request_id = request.headers.get("X-Request-ID") or str(uuid4())
|
||||
response.headers["X-Request-ID"] = request_id
|
||||
|
||||
brief_request, facts, context = await build_situational_alert_brief_request(db)
|
||||
brief_request.preferred_model = payload.preferred_model
|
||||
brief_request.thinking = payload.thinking
|
||||
|
||||
analysis = await provider_client.analyze(brief_request, request_id=request_id)
|
||||
return SituationalAlertBriefResponse(
|
||||
**analysis.model_dump(),
|
||||
title=brief_request.title,
|
||||
objective=brief_request.objective,
|
||||
facts=facts,
|
||||
context=context,
|
||||
)
|
||||
@@ -1,7 +1,7 @@
|
||||
from datetime import UTC, datetime
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, Depends
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from sqlalchemy import select, func, case
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
@@ -9,6 +9,7 @@ from app.db.session import get_db
|
||||
from app.models.user import User
|
||||
from app.core.security import get_current_user
|
||||
from app.models.alert import Alert, AlertSeverity, AlertStatus
|
||||
from app.schemas.alert import AlertResolutionRequest
|
||||
|
||||
|
||||
router = APIRouter()
|
||||
@@ -77,7 +78,7 @@ async def acknowledge_alert(
|
||||
@router.post("/{alert_id}/resolve")
|
||||
async def resolve_alert(
|
||||
alert_id: int,
|
||||
resolution: str,
|
||||
payload: AlertResolutionRequest,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
@@ -85,12 +86,12 @@ async def resolve_alert(
|
||||
alert = result.scalar_one_or_none()
|
||||
|
||||
if not alert:
|
||||
return {"error": "Alert not found"}
|
||||
raise HTTPException(status_code=404, detail="Alert not found")
|
||||
|
||||
alert.status = AlertStatus.RESOLVED
|
||||
alert.resolved_by = current_user.id
|
||||
alert.resolved_at = datetime.now(UTC)
|
||||
alert.resolution_notes = resolution
|
||||
alert.resolution_notes = payload.resolution
|
||||
await db.commit()
|
||||
|
||||
return {"message": "Alert resolved", "alert": alert.to_dict()}
|
||||
@@ -101,25 +102,44 @@ async def get_alert_stats(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
critical_query = select(func.count(Alert.id)).where(
|
||||
Alert.severity == AlertSeverity.CRITICAL,
|
||||
Alert.status == AlertStatus.ACTIVE,
|
||||
result = await db.execute(
|
||||
select(
|
||||
func.sum(
|
||||
case(
|
||||
(
|
||||
(Alert.severity == AlertSeverity.CRITICAL)
|
||||
& (Alert.status == AlertStatus.ACTIVE),
|
||||
1,
|
||||
),
|
||||
else_=0,
|
||||
)
|
||||
).label("critical"),
|
||||
func.sum(
|
||||
case(
|
||||
(
|
||||
(Alert.severity == AlertSeverity.WARNING)
|
||||
& (Alert.status == AlertStatus.ACTIVE),
|
||||
1,
|
||||
),
|
||||
else_=0,
|
||||
)
|
||||
).label("warning"),
|
||||
func.sum(
|
||||
case(
|
||||
(
|
||||
(Alert.severity == AlertSeverity.INFO)
|
||||
& (Alert.status == AlertStatus.ACTIVE),
|
||||
1,
|
||||
),
|
||||
else_=0,
|
||||
)
|
||||
).label("info"),
|
||||
)
|
||||
)
|
||||
warning_query = select(func.count(Alert.id)).where(
|
||||
Alert.severity == AlertSeverity.WARNING,
|
||||
Alert.status == AlertStatus.ACTIVE,
|
||||
)
|
||||
info_query = select(func.count(Alert.id)).where(
|
||||
Alert.severity == AlertSeverity.INFO,
|
||||
Alert.status == AlertStatus.ACTIVE,
|
||||
)
|
||||
|
||||
critical_result = await db.execute(critical_query)
|
||||
warning_result = await db.execute(warning_query)
|
||||
info_result = await db.execute(info_query)
|
||||
row = result.one()
|
||||
|
||||
return {
|
||||
"critical": critical_result.scalar() or 0,
|
||||
"warning": warning_result.scalar() or 0,
|
||||
"info": info_result.scalar() or 0,
|
||||
"critical": row.critical or 0,
|
||||
"warning": row.warning or 0,
|
||||
"info": row.info or 0,
|
||||
}
|
||||
+248
-129
@@ -22,16 +22,161 @@ def _parse_dt(value: Optional[str]) -> Optional[datetime]:
|
||||
if not value:
|
||||
return None
|
||||
return datetime.fromisoformat(value.replace("Z", "+00:00"))
|
||||
def _event_filters(
|
||||
*,
|
||||
prefix: Optional[str],
|
||||
origin_asn: Optional[int],
|
||||
peer_asn: Optional[int],
|
||||
collector: Optional[str],
|
||||
event_type: Optional[str],
|
||||
source: Optional[str],
|
||||
time_from: Optional[datetime],
|
||||
time_to: Optional[datetime],
|
||||
):
|
||||
filters = [BGPObservation.source.in_(BGP_SOURCES)]
|
||||
if source:
|
||||
filters.append(BGPObservation.source == source)
|
||||
if prefix:
|
||||
filters.append(BGPObservation.prefix == prefix)
|
||||
if origin_asn is not None:
|
||||
filters.append(BGPObservation.origin_asn == origin_asn)
|
||||
if peer_asn is not None:
|
||||
filters.append(BGPObservation.peer_asn == peer_asn)
|
||||
if collector:
|
||||
filters.append(BGPObservation.collector == collector)
|
||||
if event_type:
|
||||
filters.append(BGPObservation.event_type == event_type)
|
||||
if time_from:
|
||||
filters.append(BGPObservation.observed_at >= time_from)
|
||||
if time_to:
|
||||
filters.append(BGPObservation.observed_at <= time_to)
|
||||
return filters
|
||||
|
||||
|
||||
def _matches_time(value: Optional[datetime], time_from: Optional[datetime], time_to: Optional[datetime]) -> bool:
|
||||
if value is None:
|
||||
return False
|
||||
if time_from and value < time_from:
|
||||
return False
|
||||
if time_to and value > time_to:
|
||||
return False
|
||||
return True
|
||||
def _anomaly_filters(
|
||||
*,
|
||||
severity: Optional[str],
|
||||
anomaly_type: Optional[str],
|
||||
status: Optional[str],
|
||||
prefix: Optional[str],
|
||||
origin_asn: Optional[int],
|
||||
time_from: Optional[datetime],
|
||||
time_to: Optional[datetime],
|
||||
):
|
||||
filters = []
|
||||
if severity:
|
||||
filters.append(BGPAnomaly.severity == severity)
|
||||
if anomaly_type:
|
||||
filters.append(BGPAnomaly.anomaly_type == anomaly_type)
|
||||
if status:
|
||||
filters.append(BGPAnomaly.status == status)
|
||||
if prefix:
|
||||
filters.append(BGPAnomaly.prefix == prefix)
|
||||
if origin_asn is not None:
|
||||
filters.append(BGPAnomaly.origin_asn == origin_asn)
|
||||
if time_from:
|
||||
filters.append(BGPAnomaly.created_at >= time_from)
|
||||
if time_to:
|
||||
filters.append(BGPAnomaly.created_at <= time_to)
|
||||
return filters
|
||||
|
||||
|
||||
def _incident_filters(
|
||||
*,
|
||||
severity: Optional[str],
|
||||
incident_type: Optional[str],
|
||||
status: Optional[str],
|
||||
):
|
||||
filters = []
|
||||
if severity:
|
||||
filters.append(BGPIncident.severity == severity)
|
||||
if incident_type:
|
||||
filters.append(BGPIncident.incident_type == incident_type)
|
||||
if status:
|
||||
filters.append(BGPIncident.status == status)
|
||||
return filters
|
||||
|
||||
|
||||
async def _build_event_summary_payload(db: AsyncSession) -> dict:
|
||||
base_filters = [BGPObservation.source.in_(BGP_SOURCES)]
|
||||
|
||||
total_result = await db.execute(
|
||||
select(func.count(BGPObservation.id)).where(*base_filters)
|
||||
)
|
||||
collectors_result = await db.execute(
|
||||
select(func.count(func.distinct(BGPObservation.collector))).where(
|
||||
*base_filters, BGPObservation.collector.isnot(None)
|
||||
)
|
||||
)
|
||||
prefixes_result = await db.execute(
|
||||
select(func.count(func.distinct(BGPObservation.prefix))).where(
|
||||
*base_filters, BGPObservation.prefix.isnot(None)
|
||||
)
|
||||
)
|
||||
type_result = await db.execute(
|
||||
select(BGPObservation.event_type, func.count(BGPObservation.id))
|
||||
.where(*base_filters)
|
||||
.group_by(BGPObservation.event_type)
|
||||
)
|
||||
|
||||
return {
|
||||
"total": total_result.scalar() or 0,
|
||||
"collector_count": collectors_result.scalar() or 0,
|
||||
"prefix_count": prefixes_result.scalar() or 0,
|
||||
"by_type": {row[0]: row[1] for row in type_result.fetchall()},
|
||||
}
|
||||
|
||||
|
||||
async def _build_anomaly_summary_payload(db: AsyncSession) -> dict:
|
||||
total_result = await db.execute(select(func.count(BGPAnomaly.id)))
|
||||
type_result = await db.execute(
|
||||
select(BGPAnomaly.anomaly_type, func.count(BGPAnomaly.id))
|
||||
.group_by(BGPAnomaly.anomaly_type)
|
||||
.order_by(func.count(BGPAnomaly.id).desc())
|
||||
)
|
||||
severity_result = await db.execute(
|
||||
select(BGPAnomaly.severity, func.count(BGPAnomaly.id))
|
||||
.group_by(BGPAnomaly.severity)
|
||||
.order_by(func.count(BGPAnomaly.id).desc())
|
||||
)
|
||||
status_result = await db.execute(
|
||||
select(BGPAnomaly.status, func.count(BGPAnomaly.id))
|
||||
.group_by(BGPAnomaly.status)
|
||||
.order_by(func.count(BGPAnomaly.id).desc())
|
||||
)
|
||||
|
||||
return {
|
||||
"total": total_result.scalar() or 0,
|
||||
"by_type": {row[0]: row[1] for row in type_result.fetchall()},
|
||||
"by_severity": {row[0]: row[1] for row in severity_result.fetchall()},
|
||||
"by_status": {row[0]: row[1] for row in status_result.fetchall()},
|
||||
}
|
||||
|
||||
|
||||
async def _build_incident_summary_payload(db: AsyncSession) -> dict:
|
||||
total_result = await db.execute(select(func.count(BGPIncident.id)))
|
||||
type_result = await db.execute(
|
||||
select(BGPIncident.incident_type, func.count(BGPIncident.id))
|
||||
.group_by(BGPIncident.incident_type)
|
||||
.order_by(func.count(BGPIncident.id).desc())
|
||||
)
|
||||
severity_result = await db.execute(
|
||||
select(BGPIncident.severity, func.count(BGPIncident.id))
|
||||
.group_by(BGPIncident.severity)
|
||||
.order_by(func.count(BGPIncident.id).desc())
|
||||
)
|
||||
status_result = await db.execute(
|
||||
select(BGPIncident.status, func.count(BGPIncident.id))
|
||||
.group_by(BGPIncident.status)
|
||||
.order_by(func.count(BGPIncident.id).desc())
|
||||
)
|
||||
|
||||
return {
|
||||
"total": total_result.scalar() or 0,
|
||||
"by_type": {row[0]: row[1] for row in type_result.fetchall()},
|
||||
"by_severity": {row[0]: row[1] for row in severity_result.fetchall()},
|
||||
"by_status": {row[0]: row[1] for row in status_result.fetchall()},
|
||||
}
|
||||
|
||||
|
||||
@router.get("/events")
|
||||
@@ -49,41 +194,36 @@ async def list_bgp_events(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
stmt = (
|
||||
select(BGPObservation)
|
||||
.where(BGPObservation.source.in_(BGP_SOURCES))
|
||||
.order_by(BGPObservation.observed_at.desc(), BGPObservation.id.desc())
|
||||
)
|
||||
if source:
|
||||
stmt = stmt.where(BGPObservation.source == source)
|
||||
|
||||
result = await db.execute(stmt)
|
||||
records = result.scalars().all()
|
||||
dt_from = _parse_dt(time_from)
|
||||
dt_to = _parse_dt(time_to)
|
||||
|
||||
filtered = []
|
||||
for record in records:
|
||||
if prefix and record.prefix != prefix:
|
||||
continue
|
||||
if origin_asn is not None and record.origin_asn != origin_asn:
|
||||
continue
|
||||
if peer_asn is not None and record.peer_asn != peer_asn:
|
||||
continue
|
||||
if collector and record.collector != collector:
|
||||
continue
|
||||
if event_type and record.event_type != event_type:
|
||||
continue
|
||||
if (dt_from or dt_to) and not _matches_time(record.observed_at, dt_from, dt_to):
|
||||
continue
|
||||
filtered.append(record)
|
||||
|
||||
filters = _event_filters(
|
||||
prefix=prefix,
|
||||
origin_asn=origin_asn,
|
||||
peer_asn=peer_asn,
|
||||
collector=collector,
|
||||
event_type=event_type,
|
||||
source=source,
|
||||
time_from=dt_from,
|
||||
time_to=dt_to,
|
||||
)
|
||||
offset = (page - 1) * page_size
|
||||
count_result = await db.execute(
|
||||
select(func.count(BGPObservation.id)).where(*filters)
|
||||
)
|
||||
data_result = await db.execute(
|
||||
select(BGPObservation)
|
||||
.where(*filters)
|
||||
.order_by(BGPObservation.observed_at.desc(), BGPObservation.id.desc())
|
||||
.offset(offset)
|
||||
.limit(page_size)
|
||||
)
|
||||
records = data_result.scalars().all()
|
||||
|
||||
return {
|
||||
"total": len(filtered),
|
||||
"total": count_result.scalar() or 0,
|
||||
"page": page,
|
||||
"page_size": page_size,
|
||||
"data": [record.to_dict() for record in filtered[offset : offset + page_size]],
|
||||
"data": [record.to_dict() for record in records],
|
||||
}
|
||||
|
||||
|
||||
@@ -92,21 +232,7 @@ async def get_bgp_event_summary(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
result = await db.execute(select(BGPObservation).where(BGPObservation.source.in_(BGP_SOURCES)))
|
||||
records = result.scalars().all()
|
||||
|
||||
collectors = sorted({record.collector for record in records if record.collector})
|
||||
prefixes = sorted({record.prefix for record in records if record.prefix})
|
||||
by_type: dict[str, int] = {}
|
||||
for record in records:
|
||||
by_type[record.event_type] = by_type.get(record.event_type, 0) + 1
|
||||
|
||||
return {
|
||||
"total": len(records),
|
||||
"collector_count": len(collectors),
|
||||
"prefix_count": len(prefixes),
|
||||
"by_type": by_type,
|
||||
}
|
||||
return await _build_event_summary_payload(db)
|
||||
|
||||
|
||||
@router.get("/collectors")
|
||||
@@ -138,6 +264,32 @@ async def get_bgp_collector_summary(
|
||||
}
|
||||
|
||||
|
||||
@router.get("/overview/summary")
|
||||
async def get_bgp_overview_summary(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
event_summary = await _build_event_summary_payload(db)
|
||||
anomaly_summary = await _build_anomaly_summary_payload(db)
|
||||
incident_summary = await _build_incident_summary_payload(db)
|
||||
collectors = await build_bgp_collector_coverage(db, source_filter=BGP_SOURCES)
|
||||
active_collectors = [item for item in collectors if item["observation_count"] > 0]
|
||||
|
||||
return {
|
||||
"incidentSummary": incident_summary,
|
||||
"anomalySummary": anomaly_summary,
|
||||
"eventSummary": event_summary,
|
||||
"collectorSummary": {
|
||||
"total": len(collectors),
|
||||
"active_collectors": len(active_collectors),
|
||||
"observed_prefixes": sum(item["prefix_count"] for item in active_collectors),
|
||||
"observed_origins": sum(item["origin_asn_count"] for item in active_collectors),
|
||||
"recent_24h_events": sum(item["recent_24h_observation_count"] for item in active_collectors),
|
||||
"recent_7d_events": sum(item["recent_7d_observation_count"] for item in active_collectors),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@router.get("/events/{event_id}")
|
||||
async def get_bgp_event(
|
||||
event_id: int,
|
||||
@@ -164,31 +316,35 @@ async def list_bgp_anomalies(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
stmt = select(BGPAnomaly).order_by(BGPAnomaly.created_at.desc(), BGPAnomaly.id.desc())
|
||||
if severity:
|
||||
stmt = stmt.where(BGPAnomaly.severity == severity)
|
||||
if anomaly_type:
|
||||
stmt = stmt.where(BGPAnomaly.anomaly_type == anomaly_type)
|
||||
if status:
|
||||
stmt = stmt.where(BGPAnomaly.status == status)
|
||||
if prefix:
|
||||
stmt = stmt.where(BGPAnomaly.prefix == prefix)
|
||||
if origin_asn is not None:
|
||||
stmt = stmt.where(BGPAnomaly.origin_asn == origin_asn)
|
||||
|
||||
result = await db.execute(stmt)
|
||||
records = result.scalars().all()
|
||||
dt_from = _parse_dt(time_from)
|
||||
dt_to = _parse_dt(time_to)
|
||||
if dt_from or dt_to:
|
||||
records = [record for record in records if _matches_time(record.created_at, dt_from, dt_to)]
|
||||
|
||||
filters = _anomaly_filters(
|
||||
severity=severity,
|
||||
anomaly_type=anomaly_type,
|
||||
status=status,
|
||||
prefix=prefix,
|
||||
origin_asn=origin_asn,
|
||||
time_from=dt_from,
|
||||
time_to=dt_to,
|
||||
)
|
||||
offset = (page - 1) * page_size
|
||||
total_result = await db.execute(
|
||||
select(func.count(BGPAnomaly.id)).where(*filters)
|
||||
)
|
||||
data_result = await db.execute(
|
||||
select(BGPAnomaly)
|
||||
.where(*filters)
|
||||
.order_by(BGPAnomaly.created_at.desc(), BGPAnomaly.id.desc())
|
||||
.offset(offset)
|
||||
.limit(page_size)
|
||||
)
|
||||
records = data_result.scalars().all()
|
||||
|
||||
return {
|
||||
"total": len(records),
|
||||
"total": total_result.scalar() or 0,
|
||||
"page": page,
|
||||
"page_size": page_size,
|
||||
"data": [record.to_dict() for record in records[offset : offset + page_size]],
|
||||
"data": [record.to_dict() for record in records],
|
||||
}
|
||||
|
||||
|
||||
@@ -197,29 +353,7 @@ async def get_bgp_anomaly_summary(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
total_result = await db.execute(select(func.count(BGPAnomaly.id)))
|
||||
type_result = await db.execute(
|
||||
select(BGPAnomaly.anomaly_type, func.count(BGPAnomaly.id))
|
||||
.group_by(BGPAnomaly.anomaly_type)
|
||||
.order_by(func.count(BGPAnomaly.id).desc())
|
||||
)
|
||||
severity_result = await db.execute(
|
||||
select(BGPAnomaly.severity, func.count(BGPAnomaly.id))
|
||||
.group_by(BGPAnomaly.severity)
|
||||
.order_by(func.count(BGPAnomaly.id).desc())
|
||||
)
|
||||
status_result = await db.execute(
|
||||
select(BGPAnomaly.status, func.count(BGPAnomaly.id))
|
||||
.group_by(BGPAnomaly.status)
|
||||
.order_by(func.count(BGPAnomaly.id).desc())
|
||||
)
|
||||
|
||||
return {
|
||||
"total": total_result.scalar() or 0,
|
||||
"by_type": {row[0]: row[1] for row in type_result.fetchall()},
|
||||
"by_severity": {row[0]: row[1] for row in severity_result.fetchall()},
|
||||
"by_status": {row[0]: row[1] for row in status_result.fetchall()},
|
||||
}
|
||||
return await _build_anomaly_summary_payload(db)
|
||||
|
||||
|
||||
@router.get("/anomalies/{anomaly_id}")
|
||||
@@ -244,22 +378,29 @@ async def list_bgp_incidents(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
stmt = select(BGPIncident).order_by(BGPIncident.created_at.desc(), BGPIncident.id.desc())
|
||||
if severity:
|
||||
stmt = stmt.where(BGPIncident.severity == severity)
|
||||
if incident_type:
|
||||
stmt = stmt.where(BGPIncident.incident_type == incident_type)
|
||||
if status:
|
||||
stmt = stmt.where(BGPIncident.status == status)
|
||||
|
||||
result = await db.execute(stmt)
|
||||
records = result.scalars().all()
|
||||
filters = _incident_filters(
|
||||
severity=severity,
|
||||
incident_type=incident_type,
|
||||
status=status,
|
||||
)
|
||||
offset = (page - 1) * page_size
|
||||
total_result = await db.execute(
|
||||
select(func.count(BGPIncident.id)).where(*filters)
|
||||
)
|
||||
data_result = await db.execute(
|
||||
select(BGPIncident)
|
||||
.where(*filters)
|
||||
.order_by(BGPIncident.created_at.desc(), BGPIncident.id.desc())
|
||||
.offset(offset)
|
||||
.limit(page_size)
|
||||
)
|
||||
records = data_result.scalars().all()
|
||||
|
||||
return {
|
||||
"total": len(records),
|
||||
"total": total_result.scalar() or 0,
|
||||
"page": page,
|
||||
"page_size": page_size,
|
||||
"data": [record.to_dict() for record in records[offset : offset + page_size]],
|
||||
"data": [record.to_dict() for record in records],
|
||||
}
|
||||
|
||||
|
||||
@@ -268,29 +409,7 @@ async def get_bgp_incident_summary(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
total_result = await db.execute(select(func.count(BGPIncident.id)))
|
||||
type_result = await db.execute(
|
||||
select(BGPIncident.incident_type, func.count(BGPIncident.id))
|
||||
.group_by(BGPIncident.incident_type)
|
||||
.order_by(func.count(BGPIncident.id).desc())
|
||||
)
|
||||
severity_result = await db.execute(
|
||||
select(BGPIncident.severity, func.count(BGPIncident.id))
|
||||
.group_by(BGPIncident.severity)
|
||||
.order_by(func.count(BGPIncident.id).desc())
|
||||
)
|
||||
status_result = await db.execute(
|
||||
select(BGPIncident.status, func.count(BGPIncident.id))
|
||||
.group_by(BGPIncident.status)
|
||||
.order_by(func.count(BGPIncident.id).desc())
|
||||
)
|
||||
|
||||
return {
|
||||
"total": total_result.scalar() or 0,
|
||||
"by_type": {row[0]: row[1] for row in type_result.fetchall()},
|
||||
"by_severity": {row[0]: row[1] for row in severity_result.fetchall()},
|
||||
"by_status": {row[0]: row[1] for row in status_result.fetchall()},
|
||||
}
|
||||
return await _build_incident_summary_payload(db)
|
||||
|
||||
|
||||
@router.get("/incidents/{incident_id}")
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from fastapi import APIRouter, Depends
|
||||
from sqlalchemy import select, func, text
|
||||
from sqlalchemy import case, select, func, text
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.db.session import get_db
|
||||
@@ -118,58 +118,77 @@ async def get_stats(
|
||||
built_in_count = len(COLLECTOR_INFO)
|
||||
built_in_active = built_in_count # Built-in are always "active" for counting purposes
|
||||
|
||||
# Count custom configs from database
|
||||
result = await db.execute(select(func.count(DataSourceConfig.id)))
|
||||
custom_count = result.scalar() or 0
|
||||
|
||||
result = await db.execute(
|
||||
select(func.count(DataSourceConfig.id)).where(DataSourceConfig.is_active == True)
|
||||
select(
|
||||
func.count(DataSourceConfig.id).label("custom_count"),
|
||||
func.sum(
|
||||
case((DataSourceConfig.is_active == True, 1), else_=0)
|
||||
).label("custom_active"),
|
||||
)
|
||||
)
|
||||
custom_active = result.scalar() or 0
|
||||
datasource_stats = result.one()
|
||||
custom_count = datasource_stats.custom_count or 0
|
||||
custom_active = datasource_stats.custom_active or 0
|
||||
|
||||
# Total datasources
|
||||
total_datasources = built_in_count + custom_count
|
||||
active_datasources = built_in_active + custom_active
|
||||
|
||||
# Tasks today (from database)
|
||||
result = await db.execute(
|
||||
select(func.count(CollectionTask.id)).where(CollectionTask.started_at >= today_start)
|
||||
)
|
||||
tasks_today = result.scalar() or 0
|
||||
|
||||
result = await db.execute(
|
||||
select(func.count(CollectionTask.id)).where(
|
||||
CollectionTask.status == "success",
|
||||
CollectionTask.started_at >= today_start,
|
||||
select(
|
||||
func.count(CollectionTask.id).label("tasks_today"),
|
||||
func.sum(
|
||||
case(
|
||||
(CollectionTask.status == "success", 1),
|
||||
else_=0,
|
||||
)
|
||||
).label("success_tasks"),
|
||||
)
|
||||
.where(CollectionTask.started_at >= today_start)
|
||||
)
|
||||
success_tasks = result.scalar() or 0
|
||||
task_stats = result.one()
|
||||
tasks_today = task_stats.tasks_today or 0
|
||||
success_tasks = task_stats.success_tasks or 0
|
||||
success_rate = (success_tasks / tasks_today * 100) if tasks_today > 0 else 0
|
||||
|
||||
# Alerts
|
||||
result = await db.execute(
|
||||
select(func.count(Alert.id)).where(
|
||||
Alert.severity == AlertSeverity.CRITICAL,
|
||||
Alert.status == "active",
|
||||
select(
|
||||
func.sum(
|
||||
case(
|
||||
(
|
||||
(Alert.severity == AlertSeverity.CRITICAL)
|
||||
& (Alert.status == "active"),
|
||||
1,
|
||||
),
|
||||
else_=0,
|
||||
)
|
||||
).label("critical_alerts"),
|
||||
func.sum(
|
||||
case(
|
||||
(
|
||||
(Alert.severity == AlertSeverity.WARNING)
|
||||
& (Alert.status == "active"),
|
||||
1,
|
||||
),
|
||||
else_=0,
|
||||
)
|
||||
).label("warning_alerts"),
|
||||
func.sum(
|
||||
case(
|
||||
(
|
||||
(Alert.severity == AlertSeverity.INFO)
|
||||
& (Alert.status == "active"),
|
||||
1,
|
||||
),
|
||||
else_=0,
|
||||
)
|
||||
).label("info_alerts"),
|
||||
)
|
||||
)
|
||||
critical_alerts = result.scalar() or 0
|
||||
|
||||
result = await db.execute(
|
||||
select(func.count(Alert.id)).where(
|
||||
Alert.severity == AlertSeverity.WARNING,
|
||||
Alert.status == "active",
|
||||
)
|
||||
)
|
||||
warning_alerts = result.scalar() or 0
|
||||
|
||||
result = await db.execute(
|
||||
select(func.count(Alert.id)).where(
|
||||
Alert.severity == AlertSeverity.INFO,
|
||||
Alert.status == "active",
|
||||
)
|
||||
)
|
||||
info_alerts = result.scalar() or 0
|
||||
alert_stats = result.one()
|
||||
critical_alerts = alert_stats.critical_alerts or 0
|
||||
warning_alerts = alert_stats.warning_alerts or 0
|
||||
info_alerts = alert_stats.info_alerts or 0
|
||||
|
||||
response = {
|
||||
"total_datasources": total_datasources,
|
||||
|
||||
@@ -3,7 +3,7 @@ from datetime import datetime, timedelta, timezone
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy import func, select, text
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.time import to_iso8601_utc
|
||||
@@ -11,10 +11,17 @@ from app.core.security import get_current_user
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
from app.db.session import get_db
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.models.data_snapshot import DataSnapshot
|
||||
from app.models.datasource import DataSource
|
||||
from app.models.datasource_config import DataSourceConfig
|
||||
from app.models.task import CollectionTask
|
||||
from app.models.user import User
|
||||
from app.services.scheduler import get_latest_task_id_for_datasource, run_collector_now, sync_datasource_job
|
||||
from app.services.scheduler import (
|
||||
cancel_running_collector_now,
|
||||
get_latest_task_id_for_datasource,
|
||||
run_collector_now,
|
||||
sync_datasource_job,
|
||||
)
|
||||
|
||||
router = APIRouter()
|
||||
STALE_RUNNING_TASK_TIMEOUT_MINUTES = 90
|
||||
@@ -34,6 +41,156 @@ def is_due_for_collection(datasource: DataSource, now: datetime) -> bool:
|
||||
return datasource.last_run_at + timedelta(minutes=datasource.frequency_minutes) <= now
|
||||
|
||||
|
||||
def _task_rank_column(order_column):
|
||||
return func.row_number().over(
|
||||
partition_by=CollectionTask.datasource_id,
|
||||
order_by=(order_column.desc().nullslast(), CollectionTask.id.desc()),
|
||||
).label("row_num")
|
||||
|
||||
|
||||
async def _load_latest_running_tasks(
|
||||
db: AsyncSession,
|
||||
datasource_ids: list[int],
|
||||
) -> dict[int, CollectionTask]:
|
||||
if not datasource_ids:
|
||||
return {}
|
||||
|
||||
ranked_tasks = (
|
||||
select(
|
||||
CollectionTask.id.label("task_id"),
|
||||
_task_rank_column(CollectionTask.started_at),
|
||||
)
|
||||
.where(CollectionTask.datasource_id.in_(datasource_ids))
|
||||
.where(CollectionTask.status == "running")
|
||||
.subquery()
|
||||
)
|
||||
result = await db.execute(
|
||||
select(CollectionTask)
|
||||
.join(ranked_tasks, CollectionTask.id == ranked_tasks.c.task_id)
|
||||
.where(ranked_tasks.c.row_num == 1)
|
||||
)
|
||||
return {task.datasource_id: task for task in result.scalars().all()}
|
||||
|
||||
|
||||
async def _load_latest_completed_tasks(
|
||||
db: AsyncSession,
|
||||
datasource_ids: list[int],
|
||||
) -> dict[int, CollectionTask]:
|
||||
if not datasource_ids:
|
||||
return {}
|
||||
|
||||
ranked_tasks = (
|
||||
select(
|
||||
CollectionTask.id.label("task_id"),
|
||||
_task_rank_column(CollectionTask.completed_at),
|
||||
)
|
||||
.where(CollectionTask.datasource_id.in_(datasource_ids))
|
||||
.where(CollectionTask.completed_at.isnot(None))
|
||||
.where(CollectionTask.status.in_(("success", "failed", "cancelled")))
|
||||
.subquery()
|
||||
)
|
||||
result = await db.execute(
|
||||
select(CollectionTask)
|
||||
.join(ranked_tasks, CollectionTask.id == ranked_tasks.c.task_id)
|
||||
.where(ranked_tasks.c.row_num == 1)
|
||||
)
|
||||
return {task.datasource_id: task for task in result.scalars().all()}
|
||||
|
||||
|
||||
async def _load_latest_task_ids(
|
||||
db: AsyncSession,
|
||||
datasource_ids: list[int],
|
||||
) -> dict[int, int]:
|
||||
if not datasource_ids:
|
||||
return {}
|
||||
|
||||
ranked_tasks = (
|
||||
select(
|
||||
CollectionTask.id.label("task_id"),
|
||||
CollectionTask.datasource_id.label("datasource_id"),
|
||||
func.row_number().over(
|
||||
partition_by=CollectionTask.datasource_id,
|
||||
order_by=CollectionTask.id.desc(),
|
||||
).label("row_num"),
|
||||
)
|
||||
.where(CollectionTask.datasource_id.in_(datasource_ids))
|
||||
.subquery()
|
||||
)
|
||||
result = await db.execute(
|
||||
select(ranked_tasks.c.datasource_id, ranked_tasks.c.task_id)
|
||||
.where(ranked_tasks.c.row_num == 1)
|
||||
)
|
||||
return {datasource_id: task_id for datasource_id, task_id in result.all()}
|
||||
|
||||
|
||||
async def _load_datasource_data_counts(
|
||||
db: AsyncSession,
|
||||
sources: list[str],
|
||||
) -> dict[str, int]:
|
||||
if not sources:
|
||||
return {}
|
||||
|
||||
result = await db.execute(
|
||||
select(CollectedData.source, func.count(CollectedData.id))
|
||||
.where(CollectedData.source.in_(sources))
|
||||
.group_by(CollectedData.source)
|
||||
)
|
||||
return {source: count for source, count in result.all()}
|
||||
|
||||
|
||||
async def _load_datasource_endpoint_overrides(
|
||||
db: AsyncSession,
|
||||
sources: list[str],
|
||||
) -> dict[str, str]:
|
||||
if not sources:
|
||||
return {}
|
||||
|
||||
result = await db.execute(
|
||||
select(DataSourceConfig.name, DataSourceConfig.endpoint)
|
||||
.where(DataSourceConfig.name.in_(sources))
|
||||
.where(DataSourceConfig.is_active.is_(True))
|
||||
.where(DataSourceConfig.endpoint.isnot(None))
|
||||
)
|
||||
return {
|
||||
name: endpoint
|
||||
for name, endpoint in result.all()
|
||||
if endpoint
|
||||
}
|
||||
|
||||
|
||||
async def _load_datasource_list_context(
|
||||
db: AsyncSession,
|
||||
datasources: list[DataSource],
|
||||
) -> tuple[dict[int, CollectionTask], dict[int, CollectionTask], dict[str, int], dict[str, str]]:
|
||||
datasource_ids = [datasource.id for datasource in datasources]
|
||||
sources = [datasource.source for datasource in datasources]
|
||||
|
||||
running_tasks = await _load_latest_running_tasks(db, datasource_ids)
|
||||
datasource_by_id = {datasource.id: datasource for datasource in datasources}
|
||||
now = datetime.now(timezone.utc)
|
||||
|
||||
stale_datasource_ids: list[int] = []
|
||||
for datasource_id, task in running_tasks.items():
|
||||
started_at = task.started_at
|
||||
if started_at is None:
|
||||
continue
|
||||
if started_at.tzinfo is None:
|
||||
started_at = started_at.replace(tzinfo=timezone.utc)
|
||||
if now - started_at > timedelta(minutes=STALE_RUNNING_TASK_TIMEOUT_MINUTES):
|
||||
datasource = datasource_by_id.get(datasource_id)
|
||||
if datasource is not None:
|
||||
await fail_and_rollback_stale_running_task(db, datasource, task)
|
||||
stale_datasource_ids.append(datasource_id)
|
||||
|
||||
if stale_datasource_ids:
|
||||
running_tasks = await _load_latest_running_tasks(db, datasource_ids)
|
||||
|
||||
completed_tasks = await _load_latest_completed_tasks(db, datasource_ids)
|
||||
data_counts = await _load_datasource_data_counts(db, sources)
|
||||
endpoint_overrides = await _load_datasource_endpoint_overrides(db, sources)
|
||||
return running_tasks, completed_tasks, data_counts, endpoint_overrides
|
||||
|
||||
|
||||
async def get_datasource_record(db: AsyncSession, source_id: str) -> Optional[DataSource]:
|
||||
datasource = None
|
||||
try:
|
||||
@@ -52,18 +209,6 @@ async def get_datasource_record(db: AsyncSession, source_id: str) -> Optional[Da
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
|
||||
async def get_last_completed_task(db: AsyncSession, datasource_id: int) -> Optional[CollectionTask]:
|
||||
result = await db.execute(
|
||||
select(CollectionTask)
|
||||
.where(CollectionTask.datasource_id == datasource_id)
|
||||
.where(CollectionTask.completed_at.isnot(None))
|
||||
.where(CollectionTask.status.in_(("success", "failed", "cancelled")))
|
||||
.order_by(CollectionTask.completed_at.desc())
|
||||
.limit(1)
|
||||
)
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
|
||||
async def get_running_task(db: AsyncSession, datasource_id: int) -> Optional[CollectionTask]:
|
||||
result = await db.execute(
|
||||
select(CollectionTask)
|
||||
@@ -87,17 +232,152 @@ async def get_running_task(db: AsyncSession, datasource_id: int) -> Optional[Col
|
||||
if now - started_at <= timedelta(minutes=STALE_RUNNING_TASK_TIMEOUT_MINUTES):
|
||||
return task
|
||||
|
||||
existing_error = (task.error_message or "").strip()
|
||||
datasource = await db.get(DataSource, datasource_id)
|
||||
if datasource is not None:
|
||||
await fail_and_rollback_stale_running_task(db, datasource, task)
|
||||
else:
|
||||
existing_error = (task.error_message or "").strip()
|
||||
stale_reason = (
|
||||
f"Marked failed automatically after stale running timeout "
|
||||
f"({STALE_RUNNING_TASK_TIMEOUT_MINUTES}m)"
|
||||
)
|
||||
task.status = "failed"
|
||||
task.phase = "failed"
|
||||
task.completed_at = now
|
||||
task.error_message = f"{existing_error}\n{stale_reason}".strip() if existing_error else stale_reason
|
||||
await db.commit()
|
||||
return None
|
||||
|
||||
|
||||
async def rollback_orphaned_running_task(
|
||||
db: AsyncSession,
|
||||
datasource: DataSource,
|
||||
running_task: CollectionTask,
|
||||
) -> None:
|
||||
snapshot_result = await db.execute(
|
||||
select(DataSnapshot)
|
||||
.where(
|
||||
DataSnapshot.datasource_id == datasource.id,
|
||||
DataSnapshot.task_id == running_task.id,
|
||||
)
|
||||
.order_by(DataSnapshot.id.desc())
|
||||
.limit(1)
|
||||
)
|
||||
snapshot = snapshot_result.scalar_one_or_none()
|
||||
|
||||
await db.execute(CollectedData.__table__.delete().where(CollectedData.task_id == running_task.id))
|
||||
|
||||
await db.execute(
|
||||
text(
|
||||
"""
|
||||
UPDATE collected_data
|
||||
SET is_current = FALSE
|
||||
WHERE source = :source
|
||||
"""
|
||||
),
|
||||
{"source": datasource.source},
|
||||
)
|
||||
|
||||
if snapshot is not None:
|
||||
snapshot.status = "cancelled"
|
||||
snapshot.is_current = False
|
||||
snapshot.completed_at = datetime.now(timezone.utc)
|
||||
summary = dict(snapshot.summary or {})
|
||||
summary["rollback"] = True
|
||||
summary["rollback_reason"] = "orphaned_running_task_after_backend_restart"
|
||||
snapshot.summary = summary
|
||||
|
||||
if snapshot.parent_snapshot_id is not None:
|
||||
parent_snapshot = await db.get(DataSnapshot, snapshot.parent_snapshot_id)
|
||||
if parent_snapshot:
|
||||
parent_snapshot.is_current = True
|
||||
await db.execute(
|
||||
text(
|
||||
"""
|
||||
UPDATE collected_data
|
||||
SET is_current = TRUE
|
||||
WHERE snapshot_id = :snapshot_id
|
||||
"""
|
||||
),
|
||||
{"snapshot_id": snapshot.parent_snapshot_id},
|
||||
)
|
||||
|
||||
running_task.status = "cancelled"
|
||||
running_task.phase = "cancelled"
|
||||
running_task.completed_at = datetime.now(timezone.utc)
|
||||
existing_error = (running_task.error_message or "").strip()
|
||||
cancel_reason = "Cancelled after backend restart because the running task handle was lost; incomplete writes rolled back"
|
||||
running_task.error_message = f"{existing_error}\n{cancel_reason}".strip() if existing_error else cancel_reason
|
||||
datasource.last_status = "cancelled"
|
||||
datasource.last_run_at = datetime.now(timezone.utc)
|
||||
await db.commit()
|
||||
|
||||
|
||||
async def fail_and_rollback_stale_running_task(
|
||||
db: AsyncSession,
|
||||
datasource: DataSource,
|
||||
running_task: CollectionTask,
|
||||
) -> None:
|
||||
snapshot_result = await db.execute(
|
||||
select(DataSnapshot)
|
||||
.where(
|
||||
DataSnapshot.datasource_id == datasource.id,
|
||||
DataSnapshot.task_id == running_task.id,
|
||||
)
|
||||
.order_by(DataSnapshot.id.desc())
|
||||
.limit(1)
|
||||
)
|
||||
snapshot = snapshot_result.scalar_one_or_none()
|
||||
|
||||
await db.execute(CollectedData.__table__.delete().where(CollectedData.task_id == running_task.id))
|
||||
|
||||
await db.execute(
|
||||
text(
|
||||
"""
|
||||
UPDATE collected_data
|
||||
SET is_current = FALSE
|
||||
WHERE source = :source
|
||||
"""
|
||||
),
|
||||
{"source": datasource.source},
|
||||
)
|
||||
|
||||
if snapshot is not None:
|
||||
snapshot.status = "failed"
|
||||
snapshot.is_current = False
|
||||
snapshot.completed_at = datetime.now(timezone.utc)
|
||||
summary = dict(snapshot.summary or {})
|
||||
summary["rollback"] = True
|
||||
summary["rollback_reason"] = "stale_running_task_timeout"
|
||||
snapshot.summary = summary
|
||||
|
||||
if snapshot.parent_snapshot_id is not None:
|
||||
parent_snapshot = await db.get(DataSnapshot, snapshot.parent_snapshot_id)
|
||||
if parent_snapshot:
|
||||
parent_snapshot.is_current = True
|
||||
await db.execute(
|
||||
text(
|
||||
"""
|
||||
UPDATE collected_data
|
||||
SET is_current = TRUE
|
||||
WHERE snapshot_id = :snapshot_id
|
||||
"""
|
||||
),
|
||||
{"snapshot_id": snapshot.parent_snapshot_id},
|
||||
)
|
||||
|
||||
existing_error = (running_task.error_message or "").strip()
|
||||
stale_reason = (
|
||||
f"Marked failed automatically after stale running timeout "
|
||||
f"({STALE_RUNNING_TASK_TIMEOUT_MINUTES}m)"
|
||||
f"({STALE_RUNNING_TASK_TIMEOUT_MINUTES}m); incomplete writes rolled back"
|
||||
)
|
||||
task.status = "failed"
|
||||
task.phase = "failed"
|
||||
task.completed_at = now
|
||||
task.error_message = f"{existing_error}\n{stale_reason}".strip() if existing_error else stale_reason
|
||||
running_task.status = "failed"
|
||||
running_task.phase = "failed"
|
||||
running_task.completed_at = datetime.now(timezone.utc)
|
||||
running_task.error_message = f"{existing_error}\n{stale_reason}".strip() if existing_error else stale_reason
|
||||
datasource.last_status = "failed"
|
||||
datasource.last_run_at = datetime.now(timezone.utc)
|
||||
await db.commit()
|
||||
return None
|
||||
|
||||
|
||||
@router.get("")
|
||||
@@ -121,14 +401,17 @@ async def list_datasources(
|
||||
|
||||
collector_list = []
|
||||
config = get_data_sources_config()
|
||||
running_tasks, completed_tasks, data_counts, endpoint_overrides = await _load_datasource_list_context(
|
||||
db,
|
||||
datasources,
|
||||
)
|
||||
for datasource in datasources:
|
||||
running_task = await get_running_task(db, datasource.id)
|
||||
last_task = await get_last_completed_task(db, datasource.id)
|
||||
endpoint = await config.get_url(datasource.source, db)
|
||||
data_count_result = await db.execute(
|
||||
select(func.count(CollectedData.id)).where(CollectedData.source == datasource.source)
|
||||
running_task = running_tasks.get(datasource.id)
|
||||
last_task = completed_tasks.get(datasource.id)
|
||||
endpoint = endpoint_overrides.get(datasource.source) or config.get_yaml_url(
|
||||
datasource.source,
|
||||
)
|
||||
data_count = data_count_result.scalar() or 0
|
||||
data_count = data_counts.get(datasource.source, 0)
|
||||
|
||||
last_run_at = datasource.last_run_at or (last_task.completed_at if last_task else None)
|
||||
last_run = to_iso8601_utc(last_run_at)
|
||||
@@ -137,6 +420,7 @@ async def list_datasources(
|
||||
collector_list.append(
|
||||
{
|
||||
"id": datasource.id,
|
||||
"source": datasource.source,
|
||||
"name": datasource.name,
|
||||
"module": datasource.module,
|
||||
"priority": datasource.priority,
|
||||
@@ -189,9 +473,13 @@ async def trigger_all_datasources(
|
||||
skipped_sources: list[dict] = []
|
||||
failed_sources: list[dict] = []
|
||||
now = datetime.now(timezone.utc)
|
||||
running_tasks = await _load_latest_running_tasks(
|
||||
db,
|
||||
[datasource.id for datasource in datasources],
|
||||
)
|
||||
|
||||
for datasource in datasources:
|
||||
running_task = await get_running_task(db, datasource.id)
|
||||
running_task = running_tasks.get(datasource.id)
|
||||
if running_task is not None:
|
||||
skipped_sources.append(
|
||||
{
|
||||
@@ -219,7 +507,7 @@ async def trigger_all_datasources(
|
||||
)
|
||||
continue
|
||||
|
||||
previous_task_ids[datasource.id] = await get_latest_task_id_for_datasource(datasource.id)
|
||||
previous_task_ids[datasource.id] = None
|
||||
success = run_collector_now(datasource.source)
|
||||
if not success:
|
||||
failed_sources.append(
|
||||
@@ -241,13 +529,24 @@ async def trigger_all_datasources(
|
||||
}
|
||||
)
|
||||
|
||||
latest_task_ids = await _load_latest_task_ids(
|
||||
db,
|
||||
[datasource.id for datasource in datasources],
|
||||
)
|
||||
for datasource_id in previous_task_ids:
|
||||
previous_task_ids[datasource_id] = latest_task_ids.get(datasource_id)
|
||||
|
||||
for _ in range(20):
|
||||
await asyncio.sleep(0.1)
|
||||
pending = [item for item in triggered_sources if item["task_id"] is None]
|
||||
if not pending:
|
||||
break
|
||||
latest_task_ids = await _load_latest_task_ids(
|
||||
db,
|
||||
[item["id"] for item in pending],
|
||||
)
|
||||
for item in pending:
|
||||
task_id = await get_latest_task_id_for_datasource(item["id"])
|
||||
task_id = latest_task_ids.get(item["id"])
|
||||
if task_id is not None and task_id != previous_task_ids.get(item["id"]):
|
||||
item["task_id"] = task_id
|
||||
|
||||
@@ -346,6 +645,7 @@ async def get_datasource_stats(
|
||||
@router.post("/{source_id}/trigger")
|
||||
async def trigger_datasource(
|
||||
source_id: str,
|
||||
force: bool = Query(False),
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
@@ -356,6 +656,26 @@ async def trigger_datasource(
|
||||
if not datasource.is_active:
|
||||
raise HTTPException(status_code=400, detail="Data source is disabled")
|
||||
|
||||
running_task = await get_running_task(db, datasource.id)
|
||||
if running_task is not None and not force:
|
||||
raise HTTPException(
|
||||
status_code=409,
|
||||
detail={
|
||||
"reason": "running_task_in_progress",
|
||||
"message": "当前采集任务尚未完成,重新触发会丢失本次未完成进度。是否强制重新采集?",
|
||||
"task_id": running_task.id,
|
||||
"phase": running_task.phase,
|
||||
"progress": running_task.progress,
|
||||
"records_processed": running_task.records_processed,
|
||||
"total_records": running_task.total_records,
|
||||
},
|
||||
)
|
||||
|
||||
if running_task is not None and force:
|
||||
cancelled = await cancel_running_collector_now(datasource.source)
|
||||
if not cancelled:
|
||||
await rollback_orphaned_running_task(db, datasource, running_task)
|
||||
|
||||
previous_task_id = await get_latest_task_id_for_datasource(datasource.id)
|
||||
success = run_collector_now(datasource.source)
|
||||
if not success:
|
||||
@@ -375,6 +695,7 @@ async def trigger_datasource(
|
||||
"source_id": datasource.id,
|
||||
"task_id": task_id,
|
||||
"collector_name": datasource.source,
|
||||
"force": force,
|
||||
"message": f"Collector '{datasource.source}' has been triggered",
|
||||
}
|
||||
|
||||
@@ -412,6 +733,7 @@ async def clear_datasource_data(
|
||||
async def get_task_status(
|
||||
source_id: str,
|
||||
task_id: Optional[int] = None,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
datasource = await get_datasource_record(db, source_id)
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
from fastapi import APIRouter, Query
|
||||
|
||||
from app.services.earth_news import get_earth_news_payload
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
@router.get("/earth-feed")
|
||||
async def get_earth_feed(
|
||||
lat: float | None = Query(None, description="Current Earth view center latitude"),
|
||||
lon: float | None = Query(None, description="Current Earth view center longitude"),
|
||||
):
|
||||
return await get_earth_news_payload(lat=lat, lon=lon)
|
||||
@@ -1,3 +1,4 @@
|
||||
from copy import deepcopy
|
||||
from datetime import UTC, datetime
|
||||
from typing import Optional
|
||||
|
||||
@@ -13,6 +14,7 @@ from app.models.datasource import DataSource
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.models.user import User
|
||||
from app.services.scheduler import sync_datasource_job
|
||||
from app.services.tv_streams import DEFAULT_TV_SETTINGS, get_tv_settings_payload, normalize_tv_settings
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
@@ -36,6 +38,7 @@ DEFAULT_SETTINGS = {
|
||||
"max_login_attempts": 5,
|
||||
"password_policy": "medium",
|
||||
},
|
||||
"tv": DEFAULT_TV_SETTINGS,
|
||||
}
|
||||
|
||||
|
||||
@@ -67,8 +70,34 @@ class CollectorSettingsUpdate(BaseModel):
|
||||
frequency_minutes: int = Field(default=60, ge=1, le=10080)
|
||||
|
||||
|
||||
class TVStreamSourceUpdate(BaseModel):
|
||||
id: str = Field(min_length=1, max_length=100)
|
||||
name: str = Field(min_length=1, max_length=200)
|
||||
provider: str = Field(default="Unknown", max_length=100)
|
||||
region: str = Field(default="Global", max_length=100)
|
||||
language: str = Field(default="und", max_length=32)
|
||||
source_type: str = Field(default="iframe", pattern="^(iframe|hls|video|external|youtube)$")
|
||||
embed_url: str = ""
|
||||
stream_url: str = ""
|
||||
homepage_url: str = ""
|
||||
poster_url: str = ""
|
||||
youtube_video_id: str = ""
|
||||
youtube_channel: str = ""
|
||||
is_enabled: bool = True
|
||||
is_fallback: bool = False
|
||||
sort_order: int = Field(default=10, ge=0, le=9999)
|
||||
collector_source: Optional[str] = None
|
||||
notes: str = ""
|
||||
|
||||
|
||||
class TVSettingsUpdate(BaseModel):
|
||||
default_source_id: str = Field(default=DEFAULT_TV_SETTINGS["default_source_id"], min_length=1)
|
||||
auto_fallback: bool = True
|
||||
sources: list[TVStreamSourceUpdate] = Field(default_factory=list)
|
||||
|
||||
|
||||
def merge_with_defaults(category: str, payload: Optional[dict]) -> dict:
|
||||
merged = DEFAULT_SETTINGS[category].copy()
|
||||
merged = deepcopy(DEFAULT_SETTINGS[category])
|
||||
if payload:
|
||||
merged.update(payload)
|
||||
return merged
|
||||
@@ -79,6 +108,26 @@ async def get_setting_record(db: AsyncSession, category: str) -> Optional[System
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
|
||||
async def get_setting_payloads(db: AsyncSession, categories: list[str]) -> dict[str, dict]:
|
||||
if not categories:
|
||||
return {}
|
||||
|
||||
result = await db.execute(
|
||||
select(SystemSetting).where(SystemSetting.category.in_(categories))
|
||||
)
|
||||
records_by_category = {
|
||||
record.category: record
|
||||
for record in result.scalars().all()
|
||||
}
|
||||
return {
|
||||
category: merge_with_defaults(
|
||||
category,
|
||||
records_by_category.get(category).payload if records_by_category.get(category) else None,
|
||||
)
|
||||
for category in categories
|
||||
}
|
||||
|
||||
|
||||
async def get_setting_payload(db: AsyncSession, category: str) -> dict:
|
||||
record = await get_setting_record(db, category)
|
||||
return merge_with_defaults(category, record.payload if record else None)
|
||||
@@ -175,6 +224,25 @@ async def update_security_settings(
|
||||
return {"status": "updated", "security": payload}
|
||||
|
||||
|
||||
@router.get("/tv")
|
||||
async def get_tv_settings(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return {"tv": await get_tv_settings_payload(db)}
|
||||
|
||||
|
||||
@router.put("/tv")
|
||||
async def update_tv_settings(
|
||||
settings: TVSettingsUpdate,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
payload = normalize_tv_settings(settings.model_dump())
|
||||
saved = await save_setting_payload(db, "tv", payload)
|
||||
return {"status": "updated", "tv": normalize_tv_settings(saved)}
|
||||
|
||||
|
||||
@router.get("/collectors")
|
||||
async def get_collector_settings(
|
||||
current_user: User = Depends(get_current_user),
|
||||
@@ -212,10 +280,15 @@ async def get_all_settings(
|
||||
):
|
||||
result = await db.execute(select(DataSource).order_by(DataSource.module, DataSource.id))
|
||||
datasources = result.scalars().all()
|
||||
setting_payloads = await get_setting_payloads(
|
||||
db,
|
||||
["system", "notifications", "security"],
|
||||
)
|
||||
return {
|
||||
"system": await get_setting_payload(db, "system"),
|
||||
"notifications": await get_setting_payload(db, "notifications"),
|
||||
"security": await get_setting_payload(db, "security"),
|
||||
"system": setting_payloads["system"],
|
||||
"notifications": setting_payloads["notifications"],
|
||||
"security": setting_payloads["security"],
|
||||
"tv": await get_tv_settings_payload(db),
|
||||
"collectors": [serialize_collector(datasource) for datasource in datasources],
|
||||
"generated_at": to_iso8601_utc(datetime.now(UTC)),
|
||||
}
|
||||
@@ -4,12 +4,15 @@ import os
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, status
|
||||
from datetime import datetime
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Request, status
|
||||
from pydantic import BaseModel
|
||||
|
||||
from app.core.config import ROOT_DIR
|
||||
from app.core.security import get_current_user
|
||||
from app.models.user import User
|
||||
from app.services.persistent_logs import record_audit_log, record_system_log
|
||||
from app.services.system_control import (
|
||||
build_task_id,
|
||||
clear_active_task_id,
|
||||
@@ -23,6 +26,15 @@ from app.services.system_control import (
|
||||
set_active_task_id,
|
||||
upsert_task_state,
|
||||
)
|
||||
from app.services.system_logs import (
|
||||
DEFAULT_LOG_LINE_LIMIT,
|
||||
MAX_LOG_LINE_LIMIT,
|
||||
SUPPORTED_LOG_LEVELS,
|
||||
append_buffer_log,
|
||||
list_log_sources,
|
||||
normalize_log_level,
|
||||
read_log_snapshot,
|
||||
)
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
@@ -47,6 +59,59 @@ class RestartTaskLogsResponse(BaseModel):
|
||||
lines: list[str]
|
||||
|
||||
|
||||
class SystemLogSourceSummary(BaseModel):
|
||||
source_id: str
|
||||
name: str
|
||||
kind: str
|
||||
location: str
|
||||
description: str
|
||||
category: str
|
||||
status: str
|
||||
|
||||
|
||||
class SystemLogSourcesResponse(BaseModel):
|
||||
items: list[SystemLogSourceSummary]
|
||||
|
||||
|
||||
class SystemLogDailyMarker(BaseModel):
|
||||
date_token: str
|
||||
total: int
|
||||
dominant_level: str
|
||||
|
||||
|
||||
class SystemLogSnapshotResponse(BaseModel):
|
||||
source_id: str
|
||||
name: str
|
||||
kind: str
|
||||
location: str
|
||||
description: str
|
||||
category: str
|
||||
status: str
|
||||
level: str
|
||||
selected_levels: list[str] = []
|
||||
search_query: str = ""
|
||||
available_levels: list[str]
|
||||
daily_markers: list[SystemLogDailyMarker] = []
|
||||
line_limit: int
|
||||
line_count: int
|
||||
lines: list[str]
|
||||
|
||||
|
||||
class EarthClientLogEventCreate(BaseModel):
|
||||
level: str = "error"
|
||||
message: str
|
||||
category: str | None = None
|
||||
url: str | None = None
|
||||
module: str | None = None
|
||||
detail: str | None = None
|
||||
|
||||
|
||||
class EarthClientLogEventResponse(BaseModel):
|
||||
accepted: bool
|
||||
source_id: str
|
||||
level: str
|
||||
|
||||
|
||||
def ensure_super_admin(current_user: User) -> None:
|
||||
if not require_super_admin(current_user.role):
|
||||
raise HTTPException(
|
||||
@@ -55,9 +120,22 @@ def ensure_super_admin(current_user: User) -> None:
|
||||
)
|
||||
|
||||
|
||||
def validate_log_date(raw_value: str | None, field_name: str) -> str | None:
|
||||
if raw_value in {None, ""}:
|
||||
return None
|
||||
try:
|
||||
return datetime.strptime(raw_value, "%Y-%m-%d").date().isoformat()
|
||||
except ValueError as exc:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
detail=f"{field_name} must be in YYYY-MM-DD format",
|
||||
) from exc
|
||||
|
||||
|
||||
@router.post("/restart-tasks", response_model=RestartTaskResponse)
|
||||
async def create_restart_task(
|
||||
payload: RestartTaskCreate,
|
||||
request: Request,
|
||||
current_user: User = Depends(get_current_user),
|
||||
):
|
||||
ensure_super_admin(current_user)
|
||||
@@ -133,11 +211,31 @@ async def create_restart_task(
|
||||
requested_by=requested_by,
|
||||
)
|
||||
clear_active_task_id(task_id)
|
||||
await record_audit_log(
|
||||
action="system.restart_task.requested",
|
||||
actor_id=current_user.id,
|
||||
actor_name=current_user.username,
|
||||
target_type="restart_task",
|
||||
target_id=task_id,
|
||||
result="failed",
|
||||
ip=request.client.host if request.client else None,
|
||||
details={"action": payload.action, "message": task_state["message"]},
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
|
||||
detail=task_state["message"],
|
||||
) from exc
|
||||
|
||||
await record_audit_log(
|
||||
action="system.restart_task.requested",
|
||||
actor_id=current_user.id,
|
||||
actor_name=current_user.username,
|
||||
target_type="restart_task",
|
||||
target_id=task_id,
|
||||
result="accepted",
|
||||
ip=request.client.host if request.client else None,
|
||||
details={"action": payload.action},
|
||||
)
|
||||
return task_state
|
||||
|
||||
|
||||
@@ -165,3 +263,92 @@ async def get_restart_task_logs(
|
||||
if task is None:
|
||||
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Restart task not found")
|
||||
return {"task_id": task_id, "lines": get_task_logs(task_id)}
|
||||
|
||||
|
||||
@router.get("/logs/sources", response_model=SystemLogSourcesResponse)
|
||||
async def get_system_log_sources(
|
||||
current_user: User = Depends(get_current_user),
|
||||
):
|
||||
ensure_super_admin(current_user)
|
||||
return {"items": list_log_sources()}
|
||||
|
||||
|
||||
@router.get("/logs/{source_id}", response_model=SystemLogSnapshotResponse)
|
||||
async def get_system_log_snapshot(
|
||||
source_id: str,
|
||||
limit: int = DEFAULT_LOG_LINE_LIMIT,
|
||||
level: str = "all",
|
||||
levels: str | None = Query(None, description="Comma-separated log levels"),
|
||||
start_date: str | None = Query(None, description="Filter logs from this date (YYYY-MM-DD)"),
|
||||
end_date: str | None = Query(None, description="Filter logs until this date (YYYY-MM-DD)"),
|
||||
search: str | None = Query(None, description="Case-insensitive substring search"),
|
||||
current_user: User = Depends(get_current_user),
|
||||
):
|
||||
ensure_super_admin(current_user)
|
||||
|
||||
if limit < 1 or limit > MAX_LOG_LINE_LIMIT:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
detail=f"limit must be between 1 and {MAX_LOG_LINE_LIMIT}",
|
||||
)
|
||||
if str(level).strip().lower() not in SUPPORTED_LOG_LEVELS and normalize_log_level(level) == "all" and str(level).strip().lower() not in {"", "all"}:
|
||||
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="Unsupported log level")
|
||||
if levels:
|
||||
for raw_level in str(levels).split(","):
|
||||
normalized_level = str(raw_level).strip().lower()
|
||||
if not normalized_level:
|
||||
continue
|
||||
if normalized_level not in SUPPORTED_LOG_LEVELS and normalize_log_level(normalized_level) == "all":
|
||||
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="Unsupported log level")
|
||||
normalized_start_date = validate_log_date(start_date, "start_date")
|
||||
normalized_end_date = validate_log_date(end_date, "end_date")
|
||||
if normalized_start_date and normalized_end_date and normalized_start_date > normalized_end_date:
|
||||
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="start_date must be earlier than or equal to end_date")
|
||||
|
||||
snapshot = read_log_snapshot(
|
||||
source_id,
|
||||
limit,
|
||||
level=level,
|
||||
levels=levels,
|
||||
start_date=normalized_start_date,
|
||||
end_date=normalized_end_date,
|
||||
search=search,
|
||||
)
|
||||
if snapshot is None:
|
||||
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Log source not found")
|
||||
return snapshot
|
||||
|
||||
|
||||
@router.post("/logs/earth-client", response_model=EarthClientLogEventResponse)
|
||||
async def ingest_earth_client_log(
|
||||
payload: EarthClientLogEventCreate,
|
||||
request: Request,
|
||||
):
|
||||
normalized_level = normalize_log_level(payload.level)
|
||||
append_buffer_log(
|
||||
"earth-client",
|
||||
level=normalized_level,
|
||||
message=payload.message,
|
||||
context={
|
||||
"category": payload.category or "",
|
||||
"url": payload.url or "",
|
||||
"module": payload.module or "",
|
||||
"detail": payload.detail or "",
|
||||
},
|
||||
)
|
||||
await record_system_log(
|
||||
source="earth-client",
|
||||
service="earth",
|
||||
module=payload.module or "earth-client",
|
||||
event="earth.client.runtime_log",
|
||||
level=normalized_level,
|
||||
message=payload.message,
|
||||
category=payload.category or "client-runtime",
|
||||
context={
|
||||
"url": payload.url or "",
|
||||
"detail": payload.detail or "",
|
||||
"module": payload.module or "",
|
||||
"client_ip": request.client.host if request.client else "",
|
||||
},
|
||||
)
|
||||
return {"accepted": True, "source_id": "earth-client", "level": normalized_level}
|
||||
@@ -0,0 +1,70 @@
|
||||
from urllib.parse import quote, urljoin
|
||||
|
||||
import httpx
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query
|
||||
from fastapi.responses import Response
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.db.session import get_db
|
||||
from app.services.tv_streams import get_public_tv_payload, is_allowed_tv_proxy_url
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
@router.get("/streams")
|
||||
async def list_public_tv_streams(
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await get_public_tv_payload(db)
|
||||
|
||||
|
||||
@router.get("/proxy")
|
||||
async def proxy_tv_stream(
|
||||
url: str = Query(..., description="Upstream TV stream or manifest URL"),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
payload = await get_public_tv_payload(db)
|
||||
if not is_allowed_tv_proxy_url(url, payload.get("sources", [])):
|
||||
raise HTTPException(status_code=403, detail="TV proxy target is not allowed")
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(follow_redirects=True, timeout=20.0) as client:
|
||||
upstream = await client.get(
|
||||
url,
|
||||
headers={
|
||||
"User-Agent": "Mozilla/5.0",
|
||||
"Referer": "https://tv.cctv.com/live/cctv4/",
|
||||
},
|
||||
)
|
||||
upstream.raise_for_status()
|
||||
except httpx.HTTPError as exc:
|
||||
raise HTTPException(status_code=502, detail=f"Failed to fetch TV stream: {exc}") from exc
|
||||
|
||||
content_type = upstream.headers.get("content-type", "application/octet-stream")
|
||||
raw_content = upstream.content
|
||||
response_url = str(upstream.url)
|
||||
is_manifest = (
|
||||
response_url.endswith(".m3u8")
|
||||
or "mpegurl" in content_type.lower()
|
||||
or raw_content.lstrip().startswith(b"#EXTM3U")
|
||||
)
|
||||
|
||||
headers = {"Cache-Control": "no-store"}
|
||||
|
||||
if is_manifest:
|
||||
manifest_text = upstream.text
|
||||
rewritten_lines: list[str] = []
|
||||
for line in manifest_text.splitlines():
|
||||
stripped = line.strip()
|
||||
if not stripped or stripped.startswith("#"):
|
||||
rewritten_lines.append(line)
|
||||
continue
|
||||
absolute_url = urljoin(response_url, stripped)
|
||||
rewritten_lines.append(f"/api/v1/tv/proxy?url={quote(absolute_url, safe='')}")
|
||||
return Response(
|
||||
content="\n".join(rewritten_lines),
|
||||
media_type="application/vnd.apple.mpegurl",
|
||||
headers=headers,
|
||||
)
|
||||
|
||||
return Response(content=raw_content, media_type=content_type, headers=headers)
|
||||
+579
-155
@@ -6,12 +6,14 @@ Returns GeoJSON format compatible with Three.js, CesiumJS, and Unreal Cesium.
|
||||
|
||||
from datetime import UTC, datetime
|
||||
import math
|
||||
from fastapi import APIRouter, HTTPException, Depends, Query
|
||||
import httpx
|
||||
from fastapi import APIRouter, HTTPException, Depends, Query, Response
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from sqlalchemy import select, func
|
||||
from typing import List, Dict, Any, Optional
|
||||
|
||||
from app.core.collected_data_fields import get_record_field
|
||||
from app.core.countries import get_country_centroid
|
||||
from app.core.satellite_tle import build_tle_lines_from_elements
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.db.session import get_db
|
||||
@@ -21,8 +23,14 @@ from app.models.collected_data import CollectedData
|
||||
from app.services.bgp_collectors import build_bgp_collector_coverage
|
||||
from app.services.cable_graph import build_graph_from_data, CableGraph, haversine_distance
|
||||
from app.services.collectors.bgp_common import RIPE_RIS_COLLECTOR_COORDS
|
||||
from app.services.persistent_logs import record_system_log
|
||||
from app.core.logging import get_logger
|
||||
|
||||
router = APIRouter()
|
||||
logger = get_logger(__name__, service="api")
|
||||
TERRAIN_TILE_URL_TEMPLATE = (
|
||||
"https://s3.amazonaws.com/elevation-tiles-prod/terrarium/{z}/{x}/{y}.png"
|
||||
)
|
||||
|
||||
|
||||
# ============== Converter Functions ==============
|
||||
@@ -176,6 +184,12 @@ def convert_satellite_to_geojson(records: List[CollectedData]) -> Dict[str, Any]
|
||||
mean_motion=metadata.get("mean_motion"),
|
||||
)
|
||||
|
||||
constellation_group = _normalize_satellite_constellation_group(
|
||||
metadata.get("constellation_group"),
|
||||
record.name,
|
||||
)
|
||||
footprint_policy = _get_satellite_footprint_policy(constellation_group)
|
||||
|
||||
features.append(
|
||||
{
|
||||
"type": "Feature",
|
||||
@@ -185,6 +199,8 @@ def convert_satellite_to_geojson(records: List[CollectedData]) -> Dict[str, Any]
|
||||
"id": record.id,
|
||||
"norad_cat_id": norad_id,
|
||||
"name": record.name,
|
||||
"constellation_group": constellation_group,
|
||||
"footprint_policy": footprint_policy,
|
||||
"international_designator": metadata.get("international_designator"),
|
||||
"epoch": metadata.get("epoch"),
|
||||
"inclination": metadata.get("inclination"),
|
||||
@@ -205,42 +221,109 @@ def convert_satellite_to_geojson(records: List[CollectedData]) -> Dict[str, Any]
|
||||
return {"type": "FeatureCollection", "features": features}
|
||||
|
||||
|
||||
def dedupe_satellite_records(records: List[CollectedData]) -> List[CollectedData]:
|
||||
"""Keep only the newest record for each satellite identity."""
|
||||
latest_by_key: Dict[str, CollectedData] = {}
|
||||
def _normalize_satellite_constellation_group(
|
||||
raw_group: Any,
|
||||
name: Optional[str],
|
||||
) -> Optional[str]:
|
||||
normalized_group = str(raw_group or "").strip().lower()
|
||||
if normalized_group:
|
||||
return normalized_group
|
||||
|
||||
for record in records:
|
||||
metadata = record.extra_data or {}
|
||||
norad_id = metadata.get("norad_cat_id")
|
||||
dedupe_key = (
|
||||
str(norad_id)
|
||||
if norad_id not in (None, "")
|
||||
else str(record.source_id or record.entity_key or record.name or record.id)
|
||||
)
|
||||
normalized_name = str(name or "").strip().upper()
|
||||
if normalized_name.startswith("STARLINK"):
|
||||
return "starlink"
|
||||
if normalized_name.startswith("IRIDIUM"):
|
||||
return "iridium-next"
|
||||
|
||||
existing = latest_by_key.get(dedupe_key)
|
||||
if existing is None or (record.id or 0) > (existing.id or 0):
|
||||
latest_by_key[dedupe_key] = record
|
||||
|
||||
return sorted(latest_by_key.values(), key=lambda item: item.id or 0, reverse=True)
|
||||
return None
|
||||
|
||||
|
||||
def dedupe_collected_records(records: List[CollectedData]) -> List[CollectedData]:
|
||||
"""Keep only the newest record for each collected entity."""
|
||||
latest_by_key: Dict[str, CollectedData] = {}
|
||||
def _get_satellite_footprint_policy(constellation_group: Optional[str]) -> str:
|
||||
if constellation_group == "starlink":
|
||||
return "starlink_ground_footprint"
|
||||
if constellation_group == "iridium-next":
|
||||
return "iridium_coverage_ring"
|
||||
return "none"
|
||||
|
||||
for record in records:
|
||||
dedupe_key = str(
|
||||
record.source_id
|
||||
or record.entity_key
|
||||
or record.name
|
||||
or record.id
|
||||
)
|
||||
existing = latest_by_key.get(dedupe_key)
|
||||
if existing is None or (record.id or 0) > (existing.id or 0):
|
||||
latest_by_key[dedupe_key] = record
|
||||
|
||||
return sorted(latest_by_key.values(), key=lambda item: item.id or 0, reverse=True)
|
||||
def _current_collected_data_stmt(source: str):
|
||||
return (
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source == source)
|
||||
.where(CollectedData.is_current.is_(True))
|
||||
.order_by(CollectedData.id.desc())
|
||||
)
|
||||
|
||||
|
||||
async def _load_current_collected_data(
|
||||
db: AsyncSession,
|
||||
source: str,
|
||||
*,
|
||||
exclude_unknown_name: bool = False,
|
||||
limit: Optional[int] = None,
|
||||
) -> List[CollectedData]:
|
||||
stmt = _current_collected_data_stmt(source)
|
||||
if exclude_unknown_name:
|
||||
stmt = stmt.where(CollectedData.name != "Unknown")
|
||||
if limit is not None:
|
||||
stmt = stmt.limit(limit)
|
||||
|
||||
result = await db.execute(stmt)
|
||||
return list(result.scalars().all())
|
||||
|
||||
|
||||
async def _load_current_collected_data_by_sources(
|
||||
db: AsyncSession,
|
||||
sources: List[str],
|
||||
) -> Dict[str, List[CollectedData]]:
|
||||
if not sources:
|
||||
return {}
|
||||
|
||||
stmt = (
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source.in_(sources))
|
||||
.where(CollectedData.is_current.is_(True))
|
||||
.order_by(CollectedData.source.asc(), CollectedData.id.desc())
|
||||
)
|
||||
result = await db.execute(stmt)
|
||||
|
||||
grouped_records: Dict[str, List[CollectedData]] = {source: [] for source in sources}
|
||||
for record in result.scalars().all():
|
||||
grouped_records.setdefault(record.source, []).append(record)
|
||||
|
||||
return grouped_records
|
||||
|
||||
|
||||
def _build_landing_point_cable_maps(
|
||||
relation_records: List[CollectedData],
|
||||
cable_records: List[CollectedData],
|
||||
) -> tuple[Dict[int, List[int]], Dict[int, str]]:
|
||||
city_to_cable_ids_map: Dict[int, List[int]] = {}
|
||||
for relation_record in relation_records:
|
||||
if not relation_record.extra_data:
|
||||
continue
|
||||
city_id = relation_record.extra_data.get("city_id")
|
||||
cable_id = relation_record.extra_data.get("cable_id")
|
||||
if city_id is None or cable_id is None:
|
||||
continue
|
||||
city_to_cable_ids_map.setdefault(city_id, [])
|
||||
if cable_id not in city_to_cable_ids_map[city_id]:
|
||||
city_to_cable_ids_map[city_id].append(cable_id)
|
||||
|
||||
cable_id_to_name_map: Dict[int, str] = {}
|
||||
for cable_record in cable_records:
|
||||
if not cable_record.extra_data:
|
||||
continue
|
||||
cable_id = cable_record.extra_data.get("cable_id")
|
||||
cable_name = cable_record.name
|
||||
if cable_id and cable_name:
|
||||
cable_id_to_name_map[cable_id] = cable_name
|
||||
|
||||
return city_to_cable_ids_map, cable_id_to_name_map
|
||||
|
||||
|
||||
def _filter_known_records(records: List[CollectedData]) -> List[CollectedData]:
|
||||
return [record for record in records if record.name != "Unknown"]
|
||||
|
||||
|
||||
def convert_supercomputer_to_geojson(records: List[CollectedData]) -> Dict[str, Any]:
|
||||
@@ -317,6 +400,215 @@ def convert_gpu_cluster_to_geojson(records: List[CollectedData]) -> Dict[str, An
|
||||
return {"type": "FeatureCollection", "features": features}
|
||||
|
||||
|
||||
def _parse_float(value: Any) -> Optional[float]:
|
||||
try:
|
||||
if value in (None, ""):
|
||||
return None
|
||||
return float(value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
COMPUTE_CENTER_COORDINATE_HINTS = (
|
||||
("el capitan", 37.6819, -121.7681),
|
||||
("livermore", 37.6819, -121.7681),
|
||||
("llnl", 37.6819, -121.7681),
|
||||
("lawrence livermore", 37.6819, -121.7681),
|
||||
("frontier", 35.9319, -84.3107),
|
||||
("oak ridge", 35.9319, -84.3107),
|
||||
("ornl", 35.9319, -84.3107),
|
||||
("aurora", 41.7130, -87.9820),
|
||||
("argonne", 41.7130, -87.9820),
|
||||
("anl", 41.7130, -87.9820),
|
||||
("fugaku", 34.6953, 135.1974),
|
||||
("kobe", 34.6953, 135.1974),
|
||||
("riken", 34.6953, 135.1974),
|
||||
("summit", 35.9319, -84.3107),
|
||||
("leonardo", 44.4949, 11.3426),
|
||||
("bologna", 44.4949, 11.3426),
|
||||
("alps", 46.0037, 8.9511),
|
||||
("lugano", 46.0037, 8.9511),
|
||||
("sunway taihulight", 31.4912, 120.3119),
|
||||
("wuxi", 31.4912, 120.3119),
|
||||
("tianhe-2", 23.1291, 113.2644),
|
||||
("tianhe-2a", 23.1291, 113.2644),
|
||||
("guangzhou", 23.1291, 113.2644),
|
||||
("colossus", 35.1495, -90.0490),
|
||||
("memphis", 35.1495, -90.0490),
|
||||
("xai", 35.1495, -90.0490),
|
||||
)
|
||||
|
||||
|
||||
def _normalize_hint_text(*parts: Any) -> str:
|
||||
return " ".join(
|
||||
str(part).strip().lower()
|
||||
for part in parts
|
||||
if part not in (None, "")
|
||||
)
|
||||
|
||||
|
||||
def _resolve_compute_center_coordinates(
|
||||
record: CollectedData,
|
||||
metadata: Dict[str, Any],
|
||||
) -> Dict[str, Any]:
|
||||
latitude = _parse_float(get_record_field(record, "latitude"))
|
||||
longitude = _parse_float(get_record_field(record, "longitude"))
|
||||
if latitude not in (None, 0.0) and longitude not in (None, 0.0):
|
||||
return {
|
||||
"latitude": latitude,
|
||||
"longitude": longitude,
|
||||
"location_precision": "precise",
|
||||
"geography_mode": "source_coordinates",
|
||||
"is_estimated": False,
|
||||
"estimated_reason": None,
|
||||
}
|
||||
|
||||
hint_text = _normalize_hint_text(
|
||||
record.name,
|
||||
get_record_field(record, "city"),
|
||||
get_record_field(record, "country"),
|
||||
metadata.get("site"),
|
||||
metadata.get("organization"),
|
||||
metadata.get("operator"),
|
||||
)
|
||||
for needle, resolved_latitude, resolved_longitude in COMPUTE_CENTER_COORDINATE_HINTS:
|
||||
if needle in hint_text:
|
||||
return {
|
||||
"latitude": resolved_latitude,
|
||||
"longitude": resolved_longitude,
|
||||
"location_precision": "estimated_site",
|
||||
"geography_mode": "site_hint",
|
||||
"is_estimated": True,
|
||||
"estimated_reason": f"Matched known site hint: {needle}",
|
||||
}
|
||||
|
||||
centroid = get_country_centroid(get_record_field(record, "country"))
|
||||
if centroid:
|
||||
return {
|
||||
"latitude": centroid.get("latitude"),
|
||||
"longitude": centroid.get("longitude"),
|
||||
"location_precision": "estimated_country",
|
||||
"geography_mode": "country_centroid",
|
||||
"is_estimated": True,
|
||||
"estimated_reason": "Estimated from country centroid",
|
||||
}
|
||||
|
||||
return {
|
||||
"latitude": latitude,
|
||||
"longitude": longitude,
|
||||
"location_precision": "unknown",
|
||||
"geography_mode": "unknown",
|
||||
"is_estimated": True,
|
||||
"estimated_reason": "No resolvable location hints",
|
||||
}
|
||||
|
||||
|
||||
def _normalize_capacity_band(capacity_value: Optional[float], capacity_unit: str) -> str:
|
||||
if capacity_value is None:
|
||||
return "unknown"
|
||||
|
||||
unit = str(capacity_unit or "").strip().lower()
|
||||
if unit in {"pflop/s", "pflops", "pflop"}:
|
||||
normalized_tflops = capacity_value * 1000
|
||||
elif unit in {"gflop/s", "gflops", "gflop"}:
|
||||
normalized_tflops = capacity_value / 1000
|
||||
else:
|
||||
normalized_tflops = capacity_value
|
||||
|
||||
if normalized_tflops >= 1_000_000:
|
||||
return "exascale"
|
||||
if normalized_tflops >= 100_000:
|
||||
return "ultra"
|
||||
if normalized_tflops >= 10_000:
|
||||
return "large"
|
||||
if normalized_tflops > 0:
|
||||
return "regional"
|
||||
return "unknown"
|
||||
|
||||
|
||||
def convert_compute_centers_to_geojson(records: List[CollectedData]) -> Dict[str, Any]:
|
||||
"""Convert compute infrastructure records into a unified GeoJSON layer."""
|
||||
features = []
|
||||
|
||||
for record in records:
|
||||
metadata = record.extra_data or {}
|
||||
coordinate_info = _resolve_compute_center_coordinates(record, metadata)
|
||||
latitude = coordinate_info.get("latitude")
|
||||
longitude = coordinate_info.get("longitude")
|
||||
site_type = (
|
||||
"supercomputer"
|
||||
if record.source == "top500" or record.data_type == "supercomputer"
|
||||
else "gpu_cluster"
|
||||
)
|
||||
if latitude in (None, 0.0) or longitude in (None, 0.0):
|
||||
continue
|
||||
|
||||
if site_type == "supercomputer":
|
||||
capacity_value = _parse_float(get_record_field(record, "rmax"))
|
||||
capacity_unit = "GFlops"
|
||||
else:
|
||||
capacity_value = _parse_float(get_record_field(record, "value"))
|
||||
capacity_unit = str(get_record_field(record, "unit") or "TFlop/s")
|
||||
|
||||
vendor = (
|
||||
metadata.get("manufacturer")
|
||||
or metadata.get("vendor")
|
||||
or metadata.get("gpu_type")
|
||||
)
|
||||
operator = (
|
||||
metadata.get("organization")
|
||||
or metadata.get("operator")
|
||||
or metadata.get("owner")
|
||||
)
|
||||
rank = metadata.get("rank")
|
||||
if rank in (None, "") and site_type == "supercomputer":
|
||||
rank = get_record_field(record, "rank")
|
||||
|
||||
updated_at = to_iso8601_utc(record.reference_date or record.collected_at)
|
||||
|
||||
features.append(
|
||||
{
|
||||
"type": "Feature",
|
||||
"id": record.id,
|
||||
"geometry": {
|
||||
"type": "Point",
|
||||
"coordinates": [longitude or 0, latitude or 0],
|
||||
},
|
||||
"properties": {
|
||||
"id": record.id,
|
||||
"source_id": record.source_id,
|
||||
"name": record.name,
|
||||
"site_type": site_type,
|
||||
"country": get_record_field(record, "country"),
|
||||
"city": get_record_field(record, "city"),
|
||||
"latitude": latitude,
|
||||
"longitude": longitude,
|
||||
"operator": operator,
|
||||
"vendor": vendor,
|
||||
"capacity_value": capacity_value,
|
||||
"capacity_unit": capacity_unit,
|
||||
"capacity_band": _normalize_capacity_band(capacity_value, capacity_unit),
|
||||
"rank": rank,
|
||||
"gpu_count": metadata.get("gpu_count"),
|
||||
"gpu_type": metadata.get("gpu_type"),
|
||||
"cores": get_record_field(record, "cores"),
|
||||
"power": get_record_field(record, "power"),
|
||||
"source": record.source,
|
||||
"updated_at": updated_at,
|
||||
"status": "observed",
|
||||
"location_precision": coordinate_info.get("location_precision"),
|
||||
"geography_mode": coordinate_info.get("geography_mode"),
|
||||
"is_estimated": coordinate_info.get("is_estimated", False),
|
||||
"estimated_reason": coordinate_info.get("estimated_reason"),
|
||||
"data_type": "compute_center",
|
||||
"metadata": metadata,
|
||||
},
|
||||
}
|
||||
)
|
||||
|
||||
return {"type": "FeatureCollection", "features": features}
|
||||
|
||||
|
||||
def convert_bgp_anomalies_to_geojson(
|
||||
records: List[BGPAnomaly],
|
||||
geography_hints: Optional[Dict[str, Dict[str, Any]]] = None,
|
||||
@@ -722,9 +1014,7 @@ def convert_bgp_incidents_to_geojson(
|
||||
async def get_cables_geojson(db: AsyncSession = Depends(get_db)):
|
||||
"""获取海底电缆 GeoJSON 数据 (LineString)"""
|
||||
try:
|
||||
stmt = select(CollectedData).where(CollectedData.source == "arcgis_cables")
|
||||
result = await db.execute(stmt)
|
||||
records = dedupe_collected_records(list(result.scalars().all()))
|
||||
records = await _load_current_collected_data(db, "arcgis_cables")
|
||||
|
||||
if not records:
|
||||
raise HTTPException(
|
||||
@@ -736,42 +1026,46 @@ async def get_cables_geojson(db: AsyncSession = Depends(get_db)):
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.exception_event(
|
||||
"Failed to build cables GeoJSON response",
|
||||
event="visualization.cables.load_failed",
|
||||
context={"error": str(e)},
|
||||
)
|
||||
await record_system_log(
|
||||
source="backend",
|
||||
service="api",
|
||||
module=__name__,
|
||||
event="visualization.cables.load_failed",
|
||||
level="error",
|
||||
message="Failed to build cables GeoJSON response",
|
||||
category="visualization",
|
||||
context={"error": str(e)},
|
||||
)
|
||||
raise HTTPException(status_code=500, detail=f"Internal error: {str(e)}")
|
||||
|
||||
|
||||
@router.get("/geo/landing-points")
|
||||
async def get_landing_points_geojson(db: AsyncSession = Depends(get_db)):
|
||||
try:
|
||||
landing_stmt = select(CollectedData).where(CollectedData.source == "arcgis_landing_points")
|
||||
landing_result = await db.execute(landing_stmt)
|
||||
records = dedupe_collected_records(list(landing_result.scalars().all()))
|
||||
|
||||
relation_stmt = select(CollectedData).where(CollectedData.source == "arcgis_cable_landing_relation")
|
||||
relation_result = await db.execute(relation_stmt)
|
||||
relation_records = dedupe_collected_records(list(relation_result.scalars().all()))
|
||||
|
||||
cable_stmt = select(CollectedData).where(CollectedData.source == "arcgis_cables")
|
||||
cable_result = await db.execute(cable_stmt)
|
||||
cable_records = dedupe_collected_records(list(cable_result.scalars().all()))
|
||||
|
||||
city_to_cable_ids_map = {}
|
||||
for rel in relation_records:
|
||||
if rel.extra_data:
|
||||
city_id = rel.extra_data.get("city_id")
|
||||
cable_id = rel.extra_data.get("cable_id")
|
||||
if city_id is not None and cable_id is not None:
|
||||
if city_id not in city_to_cable_ids_map:
|
||||
city_to_cable_ids_map[city_id] = []
|
||||
if cable_id not in city_to_cable_ids_map[city_id]:
|
||||
city_to_cable_ids_map[city_id].append(cable_id)
|
||||
|
||||
cable_id_to_name_map = {}
|
||||
for cable in cable_records:
|
||||
if cable.extra_data:
|
||||
cable_id = cable.extra_data.get("cable_id")
|
||||
cable_name = cable.name
|
||||
if cable_id and cable_name:
|
||||
cable_id_to_name_map[cable_id] = cable_name
|
||||
records_by_source = await _load_current_collected_data_by_sources(
|
||||
db,
|
||||
[
|
||||
"arcgis_landing_points",
|
||||
"arcgis_cable_landing_relation",
|
||||
"arcgis_cables",
|
||||
],
|
||||
)
|
||||
records = records_by_source.get("arcgis_landing_points", [])
|
||||
relation_records = records_by_source.get(
|
||||
"arcgis_cable_landing_relation",
|
||||
[],
|
||||
)
|
||||
cable_records = records_by_source.get("arcgis_cables", [])
|
||||
|
||||
city_to_cable_ids_map, cable_id_to_name_map = _build_landing_point_cable_maps(
|
||||
relation_records,
|
||||
cable_records,
|
||||
)
|
||||
|
||||
if not records:
|
||||
raise HTTPException(
|
||||
@@ -783,41 +1077,85 @@ async def get_landing_points_geojson(db: AsyncSession = Depends(get_db)):
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.exception_event(
|
||||
"Failed to build landing points GeoJSON response",
|
||||
event="visualization.landing_points.load_failed",
|
||||
context={"error": str(e)},
|
||||
)
|
||||
await record_system_log(
|
||||
source="backend",
|
||||
service="api",
|
||||
module=__name__,
|
||||
event="visualization.landing_points.load_failed",
|
||||
level="error",
|
||||
message="Failed to build landing points GeoJSON response",
|
||||
category="visualization",
|
||||
context={"error": str(e)},
|
||||
)
|
||||
raise HTTPException(status_code=500, detail=f"Internal error: {str(e)}")
|
||||
|
||||
|
||||
@router.get("/terrain/terrarium/{z}/{x}/{y}.png")
|
||||
async def get_terrarium_tile(z: int, x: int, y: int):
|
||||
"""Proxy Terrarium elevation tiles through the backend to avoid browser CORS issues."""
|
||||
if z < 0 or x < 0 or y < 0:
|
||||
raise HTTPException(status_code=400, detail="Invalid terrain tile coordinates")
|
||||
|
||||
url = TERRAIN_TILE_URL_TEMPLATE.format(z=z, x=x, y=y)
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(
|
||||
timeout=20.0,
|
||||
follow_redirects=True,
|
||||
) as client:
|
||||
upstream = await client.get(url)
|
||||
upstream.raise_for_status()
|
||||
except httpx.HTTPStatusError as exc:
|
||||
raise HTTPException(
|
||||
status_code=exc.response.status_code,
|
||||
detail=f"Terrain tile upstream error: {exc.response.status_code}",
|
||||
) from exc
|
||||
except httpx.HTTPError as exc:
|
||||
raise HTTPException(
|
||||
status_code=502,
|
||||
detail=f"Terrain tile fetch failed: {exc}",
|
||||
) from exc
|
||||
|
||||
cache_control = upstream.headers.get("cache-control") or "public, max-age=86400"
|
||||
etag = upstream.headers.get("etag")
|
||||
last_modified = upstream.headers.get("last-modified")
|
||||
headers = {
|
||||
"Cache-Control": cache_control,
|
||||
}
|
||||
if etag:
|
||||
headers["ETag"] = etag
|
||||
if last_modified:
|
||||
headers["Last-Modified"] = last_modified
|
||||
|
||||
return Response(
|
||||
content=upstream.content,
|
||||
media_type=upstream.headers.get("content-type", "image/png"),
|
||||
headers=headers,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/geo/all")
|
||||
async def get_all_geojson(db: AsyncSession = Depends(get_db)):
|
||||
cables_stmt = select(CollectedData).where(CollectedData.source == "arcgis_cables")
|
||||
cables_result = await db.execute(cables_stmt)
|
||||
cables_records = dedupe_collected_records(list(cables_result.scalars().all()))
|
||||
|
||||
points_stmt = select(CollectedData).where(CollectedData.source == "arcgis_landing_points")
|
||||
points_result = await db.execute(points_stmt)
|
||||
points_records = dedupe_collected_records(list(points_result.scalars().all()))
|
||||
|
||||
relation_stmt = select(CollectedData).where(CollectedData.source == "arcgis_cable_landing_relation")
|
||||
relation_result = await db.execute(relation_stmt)
|
||||
relation_records = dedupe_collected_records(list(relation_result.scalars().all()))
|
||||
|
||||
city_to_cable_ids_map = {}
|
||||
for rel in relation_records:
|
||||
if rel.extra_data:
|
||||
city_id = rel.extra_data.get("city_id")
|
||||
cable_id = rel.extra_data.get("cable_id")
|
||||
if city_id is not None and cable_id is not None:
|
||||
if city_id not in city_to_cable_ids_map:
|
||||
city_to_cable_ids_map[city_id] = []
|
||||
if cable_id not in city_to_cable_ids_map[city_id]:
|
||||
city_to_cable_ids_map[city_id].append(cable_id)
|
||||
|
||||
cable_id_to_name_map = {}
|
||||
for cable in cables_records:
|
||||
if cable.extra_data:
|
||||
cable_id = cable.extra_data.get("cable_id")
|
||||
cable_name = cable.name
|
||||
if cable_id and cable_name:
|
||||
cable_id_to_name_map[cable_id] = cable_name
|
||||
records_by_source = await _load_current_collected_data_by_sources(
|
||||
db,
|
||||
[
|
||||
"arcgis_cables",
|
||||
"arcgis_landing_points",
|
||||
"arcgis_cable_landing_relation",
|
||||
],
|
||||
)
|
||||
cables_records = records_by_source.get("arcgis_cables", [])
|
||||
points_records = records_by_source.get("arcgis_landing_points", [])
|
||||
relation_records = records_by_source.get("arcgis_cable_landing_relation", [])
|
||||
city_to_cable_ids_map, cable_id_to_name_map = _build_landing_point_cable_maps(
|
||||
relation_records,
|
||||
cables_records,
|
||||
)
|
||||
|
||||
cables = (
|
||||
convert_cable_to_geojson(cables_records)
|
||||
@@ -850,17 +1188,12 @@ async def get_satellites_geojson(
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""获取卫星 TLE GeoJSON 数据"""
|
||||
stmt = (
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source == "celestrak_tle")
|
||||
.where(CollectedData.name != "Unknown")
|
||||
.order_by(CollectedData.id.desc())
|
||||
records = await _load_current_collected_data(
|
||||
db,
|
||||
"celestrak_tle",
|
||||
exclude_unknown_name=True,
|
||||
limit=limit,
|
||||
)
|
||||
result = await db.execute(stmt)
|
||||
records = dedupe_satellite_records(list(result.scalars().all()))
|
||||
|
||||
if limit is not None:
|
||||
records = records[:limit]
|
||||
|
||||
if not records:
|
||||
return {"type": "FeatureCollection", "features": [], "count": 0}
|
||||
@@ -878,15 +1211,12 @@ async def get_supercomputers_geojson(
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""获取 TOP500 超算中心 GeoJSON 数据"""
|
||||
stmt = (
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source == "top500")
|
||||
.where(CollectedData.name != "Unknown")
|
||||
.order_by(CollectedData.id.desc())
|
||||
records = await _load_current_collected_data(
|
||||
db,
|
||||
"top500",
|
||||
exclude_unknown_name=True,
|
||||
limit=limit,
|
||||
)
|
||||
result = await db.execute(stmt)
|
||||
records = dedupe_collected_records(list(result.scalars().all()))
|
||||
records = records[:limit]
|
||||
|
||||
if not records:
|
||||
return {"type": "FeatureCollection", "features": [], "count": 0}
|
||||
@@ -904,15 +1234,12 @@ async def get_gpu_clusters_geojson(
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""获取 GPU 集群 GeoJSON 数据"""
|
||||
stmt = (
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source == "epoch_ai_gpu")
|
||||
.where(CollectedData.name != "Unknown")
|
||||
.order_by(CollectedData.id.desc())
|
||||
records = await _load_current_collected_data(
|
||||
db,
|
||||
"epoch_ai_gpu",
|
||||
exclude_unknown_name=True,
|
||||
limit=limit,
|
||||
)
|
||||
result = await db.execute(stmt)
|
||||
records = dedupe_collected_records(list(result.scalars().all()))
|
||||
records = records[:limit]
|
||||
|
||||
if not records:
|
||||
return {"type": "FeatureCollection", "features": [], "count": 0}
|
||||
@@ -924,6 +1251,53 @@ async def get_gpu_clusters_geojson(
|
||||
}
|
||||
|
||||
|
||||
@router.get("/geo/compute-centers")
|
||||
async def get_compute_centers_geojson(
|
||||
limit: int = Query(200, ge=1, le=1000),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""获取统一算力中心 GeoJSON 数据"""
|
||||
records_by_source = await _load_current_collected_data_by_sources(
|
||||
db,
|
||||
["top500", "epoch_ai_gpu"],
|
||||
)
|
||||
records = _filter_known_records(
|
||||
records_by_source.get("top500", []) + records_by_source.get("epoch_ai_gpu", []),
|
||||
)
|
||||
if limit is not None:
|
||||
records = records[:limit]
|
||||
|
||||
if not records:
|
||||
return {
|
||||
"type": "FeatureCollection",
|
||||
"features": [],
|
||||
"count": 0,
|
||||
"stats": {
|
||||
"total": 0,
|
||||
"supercomputers": 0,
|
||||
"gpu_clusters": 0,
|
||||
},
|
||||
}
|
||||
|
||||
geojson = convert_compute_centers_to_geojson(records)
|
||||
features = geojson.get("features", [])
|
||||
return {
|
||||
**geojson,
|
||||
"count": len(features),
|
||||
"stats": {
|
||||
"total": len(features),
|
||||
"supercomputers": sum(
|
||||
1 for feature in features
|
||||
if feature.get("properties", {}).get("site_type") == "supercomputer"
|
||||
),
|
||||
"gpu_clusters": sum(
|
||||
1 for feature in features
|
||||
if feature.get("properties", {}).get("site_type") == "gpu_cluster"
|
||||
),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@router.get("/geo/bgp-anomalies")
|
||||
async def get_bgp_anomalies_geojson(
|
||||
severity: Optional[str] = Query(None),
|
||||
@@ -979,6 +1353,71 @@ async def get_bgp_collectors_geojson(db: AsyncSession = Depends(get_db)):
|
||||
return {**geojson, "count": len(geojson.get("features", []))}
|
||||
|
||||
|
||||
@router.get("/geo/summary")
|
||||
async def get_visualization_geo_summary(db: AsyncSession = Depends(get_db)):
|
||||
"""Return lightweight Earth HUD counts without loading layer GeoJSON payloads."""
|
||||
records_by_source = await _load_current_collected_data_by_sources(
|
||||
db,
|
||||
[
|
||||
"arcgis_cables",
|
||||
"arcgis_landing_points",
|
||||
"celestrak_tle",
|
||||
"top500",
|
||||
"epoch_ai_gpu",
|
||||
],
|
||||
)
|
||||
|
||||
cables = convert_cable_to_geojson(records_by_source.get("arcgis_cables", []))
|
||||
landing_points = convert_landing_point_to_geojson(
|
||||
records_by_source.get("arcgis_landing_points", []),
|
||||
)
|
||||
satellites = convert_satellite_to_geojson(
|
||||
_filter_known_records(records_by_source.get("celestrak_tle", [])),
|
||||
)
|
||||
compute_centers = convert_compute_centers_to_geojson(
|
||||
_filter_known_records(
|
||||
records_by_source.get("top500", [])
|
||||
+ records_by_source.get("epoch_ai_gpu", []),
|
||||
),
|
||||
)
|
||||
compute_features = compute_centers.get("features", [])
|
||||
|
||||
active_incident_result = await db.execute(
|
||||
select(func.count(BGPIncident.id)).where(BGPIncident.status == "active"),
|
||||
)
|
||||
active_anomaly_result = await db.execute(
|
||||
select(func.count(BGPAnomaly.id)).where(BGPAnomaly.status == "active"),
|
||||
)
|
||||
active_incident_count = int(active_incident_result.scalar() or 0)
|
||||
active_anomaly_count = int(active_anomaly_result.scalar() or 0)
|
||||
bgp_collectors = await build_bgp_collector_coverage(
|
||||
db,
|
||||
source_filter=("ris_live_bgp", "bgpstream_bgp"),
|
||||
)
|
||||
|
||||
return {
|
||||
"generated_at": to_iso8601_utc(datetime.now(UTC)),
|
||||
"stats": {
|
||||
"cable_count": len(cables.get("features", [])),
|
||||
"landing_point_count": len(landing_points.get("features", [])),
|
||||
"satellite_count": len(satellites.get("features", [])),
|
||||
"compute_center_count": len(compute_features),
|
||||
"supercomputer_count": sum(
|
||||
1 for feature in compute_features
|
||||
if feature.get("properties", {}).get("site_type") == "supercomputer"
|
||||
),
|
||||
"gpu_cluster_count": sum(
|
||||
1 for feature in compute_features
|
||||
if feature.get("properties", {}).get("site_type") == "gpu_cluster"
|
||||
),
|
||||
"bgp_event_count": active_incident_count or active_anomaly_count,
|
||||
"bgp_incident_count": active_incident_count,
|
||||
"bgp_anomaly_count": active_anomaly_count,
|
||||
"bgp_collector_count": len([item for item in bgp_collectors if item.get("collector")]),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@router.get("/all")
|
||||
async def get_all_visualization_data(db: AsyncSession = Depends(get_db)):
|
||||
"""获取所有可视化数据的统一端点
|
||||
@@ -990,37 +1429,27 @@ async def get_all_visualization_data(db: AsyncSession = Depends(get_db)):
|
||||
- supercomputers: TOP500 超算
|
||||
- gpu_clusters: GPU 集群
|
||||
"""
|
||||
cables_stmt = select(CollectedData).where(CollectedData.source == "arcgis_cables")
|
||||
cables_result = await db.execute(cables_stmt)
|
||||
cables_records = dedupe_collected_records(list(cables_result.scalars().all()))
|
||||
|
||||
points_stmt = select(CollectedData).where(CollectedData.source == "arcgis_landing_points")
|
||||
points_result = await db.execute(points_stmt)
|
||||
points_records = dedupe_collected_records(list(points_result.scalars().all()))
|
||||
|
||||
satellites_stmt = (
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source == "celestrak_tle")
|
||||
.where(CollectedData.name != "Unknown")
|
||||
records_by_source = await _load_current_collected_data_by_sources(
|
||||
db,
|
||||
[
|
||||
"arcgis_cables",
|
||||
"arcgis_landing_points",
|
||||
"celestrak_tle",
|
||||
"top500",
|
||||
"epoch_ai_gpu",
|
||||
],
|
||||
)
|
||||
satellites_result = await db.execute(satellites_stmt)
|
||||
satellites_records = dedupe_satellite_records(list(satellites_result.scalars().all()))
|
||||
|
||||
supercomputers_stmt = (
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source == "top500")
|
||||
.where(CollectedData.name != "Unknown")
|
||||
cables_records = records_by_source.get("arcgis_cables", [])
|
||||
points_records = records_by_source.get("arcgis_landing_points", [])
|
||||
satellites_records = _filter_known_records(
|
||||
records_by_source.get("celestrak_tle", []),
|
||||
)
|
||||
supercomputers_result = await db.execute(supercomputers_stmt)
|
||||
supercomputers_records = dedupe_collected_records(list(supercomputers_result.scalars().all()))
|
||||
|
||||
gpu_stmt = (
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source == "epoch_ai_gpu")
|
||||
.where(CollectedData.name != "Unknown")
|
||||
supercomputers_records = _filter_known_records(
|
||||
records_by_source.get("top500", []),
|
||||
)
|
||||
gpu_records = _filter_known_records(
|
||||
records_by_source.get("epoch_ai_gpu", []),
|
||||
)
|
||||
gpu_result = await db.execute(gpu_stmt)
|
||||
gpu_records = dedupe_collected_records(list(gpu_result.scalars().all()))
|
||||
|
||||
cables = (
|
||||
convert_cable_to_geojson(cables_records)
|
||||
@@ -1084,13 +1513,8 @@ async def get_cable_graph(db: AsyncSession) -> CableGraph:
|
||||
global _cable_graph
|
||||
|
||||
if _cable_graph is None:
|
||||
cables_stmt = select(CollectedData).where(CollectedData.source == "arcgis_cables")
|
||||
cables_result = await db.execute(cables_stmt)
|
||||
cables_records = list(cables_result.scalars().all())
|
||||
|
||||
points_stmt = select(CollectedData).where(CollectedData.source == "arcgis_landing_points")
|
||||
points_result = await db.execute(points_stmt)
|
||||
points_records = list(points_result.scalars().all())
|
||||
cables_records = await _load_current_collected_data(db, "arcgis_cables")
|
||||
points_records = await _load_current_collected_data(db, "arcgis_landing_points")
|
||||
|
||||
cables_data = convert_cable_to_geojson(cables_records)
|
||||
points_data = convert_landing_point_to_geojson(points_records)
|
||||
|
||||
@@ -2,7 +2,6 @@
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
from datetime import UTC, datetime
|
||||
from typing import Optional
|
||||
|
||||
@@ -10,10 +9,11 @@ from fastapi import APIRouter, WebSocket, WebSocketDisconnect, Query
|
||||
from jose import jwt, JWTError
|
||||
|
||||
from app.core.config import settings
|
||||
from app.core.logging import get_logger
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.core.websocket.manager import manager
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
logger = get_logger(__name__, service="api")
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
@@ -22,11 +22,18 @@ async def authenticate_token(token: str) -> Optional[dict]:
|
||||
try:
|
||||
payload = jwt.decode(token, settings.SECRET_KEY, algorithms=[settings.ALGORITHM])
|
||||
if payload.get("type") != "access":
|
||||
logger.warning(f"WebSocket auth failed: wrong token type")
|
||||
logger.warning_event(
|
||||
"WebSocket auth failed: wrong token type",
|
||||
event="auth.websocket.invalid_token_type",
|
||||
)
|
||||
return None
|
||||
return payload
|
||||
except JWTError as e:
|
||||
logger.warning(f"WebSocket auth failed: {e}")
|
||||
logger.warning_event(
|
||||
"WebSocket auth failed",
|
||||
event="auth.websocket.decode_failed",
|
||||
context={"error": str(e)},
|
||||
)
|
||||
return None
|
||||
|
||||
|
||||
@@ -36,10 +43,17 @@ async def websocket_endpoint(
|
||||
token: str = Query(...),
|
||||
):
|
||||
"""WebSocket endpoint for real-time data"""
|
||||
logger.info(f"WebSocket connection attempt with token: {token[:20]}...")
|
||||
logger.info_event(
|
||||
"WebSocket connection attempt",
|
||||
event="auth.websocket.connection_attempt",
|
||||
context={"token_preview": f"{token[:8]}..."},
|
||||
)
|
||||
payload = await authenticate_token(token)
|
||||
if payload is None:
|
||||
logger.warning("WebSocket authentication failed, closing connection")
|
||||
logger.warning_event(
|
||||
"WebSocket authentication failed, closing connection",
|
||||
event="auth.websocket.connection_rejected",
|
||||
)
|
||||
await websocket.close(code=4001)
|
||||
return
|
||||
|
||||
|
||||
@@ -1,15 +1,15 @@
|
||||
"""Redis caching service"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
from datetime import timedelta
|
||||
from typing import Optional, Any
|
||||
|
||||
import redis
|
||||
|
||||
from app.core.config import settings
|
||||
from app.core.logging import get_logger
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
# Lazy Redis client initialization
|
||||
@@ -47,7 +47,7 @@ class CacheService:
|
||||
return json.loads(value)
|
||||
return None
|
||||
except Exception as e:
|
||||
logger.warning(f"Cache get error: {e}")
|
||||
logger.warning_event("Cache get error", event="cache.get.failed", context={"error": str(e)})
|
||||
return None
|
||||
|
||||
def set(
|
||||
@@ -61,7 +61,7 @@ class CacheService:
|
||||
serialized = json.dumps(value, default=str)
|
||||
return self.client.setex(key, expire_seconds, serialized)
|
||||
except Exception as e:
|
||||
logger.warning(f"Cache set error: {e}")
|
||||
logger.warning_event("Cache set error", event="cache.set.failed", context={"error": str(e)})
|
||||
return False
|
||||
|
||||
def delete(self, key: str) -> bool:
|
||||
@@ -69,7 +69,7 @@ class CacheService:
|
||||
try:
|
||||
return self.client.delete(key) > 0
|
||||
except Exception as e:
|
||||
logger.warning(f"Cache delete error: {e}")
|
||||
logger.warning_event("Cache delete error", event="cache.delete.failed", context={"error": str(e)})
|
||||
return False
|
||||
|
||||
def delete_pattern(self, pattern: str) -> int:
|
||||
@@ -80,7 +80,7 @@ class CacheService:
|
||||
return self.client.delete(*keys)
|
||||
return 0
|
||||
except Exception as e:
|
||||
logger.warning(f"Cache delete_pattern error: {e}")
|
||||
logger.warning_event("Cache delete_pattern error", event="cache.delete_pattern.failed", context={"error": str(e)})
|
||||
return 0
|
||||
|
||||
def get_or_set(
|
||||
|
||||
@@ -11,6 +11,7 @@ COLLECTOR_URL_KEYS = {
|
||||
"fao_landing_points": "fao.landing_point_url",
|
||||
"telegeography_cables": "telegeography.cable_url",
|
||||
"telegeography_landing": "telegeography.landing_point_url",
|
||||
"telegeography_systems": "telegeography.cable_url",
|
||||
"huggingface_models": "huggingface.models_url",
|
||||
"huggingface_datasets": "huggingface.datasets_url",
|
||||
"huggingface_spaces": "huggingface.spaces_url",
|
||||
@@ -23,11 +24,13 @@ COLLECTOR_URL_KEYS = {
|
||||
"top500": "top500.url",
|
||||
"epoch_ai_gpu": "epoch_ai.gpu_clusters_url",
|
||||
"spacetrack_tle": "spacetrack.tle_query_url",
|
||||
"celestrak_tle": "celestrak.base_url",
|
||||
"ris_live_bgp": "ris_live.url",
|
||||
"bgpstream_bgp": "bgpstream.url",
|
||||
"iptoasn_prefix_geo": "iptoasn.combined_url",
|
||||
"opengeofeed_prefix_geo": "opengeofeed.public_csv_url",
|
||||
"nro_delegated_prefix_geo": "nro.delegated_stats_url",
|
||||
"news_live_streams": "news_live_streams.channels_url",
|
||||
}
|
||||
|
||||
|
||||
@@ -41,18 +44,22 @@ class DataSourcesConfig:
|
||||
with open(config_path, "r") as f:
|
||||
self._yaml_config = yaml.safe_load(f) or {}
|
||||
|
||||
def get_yaml_url(self, collector_name: str) -> str:
|
||||
key = COLLECTOR_URL_KEYS.get(collector_name, "")
|
||||
def get_yaml_value(self, key: str):
|
||||
if not key:
|
||||
return ""
|
||||
return None
|
||||
|
||||
parts = key.split(".")
|
||||
value = self._yaml_config
|
||||
for part in parts:
|
||||
if isinstance(value, dict):
|
||||
value = value.get(part, "")
|
||||
value = value.get(part)
|
||||
else:
|
||||
return ""
|
||||
return None
|
||||
return value
|
||||
|
||||
def get_yaml_url(self, collector_name: str) -> str:
|
||||
key = COLLECTOR_URL_KEYS.get(collector_name, "")
|
||||
value = self.get_yaml_value(key)
|
||||
return value if isinstance(value, str) else ""
|
||||
|
||||
async def get_url(self, collector_name: str, db) -> str:
|
||||
|
||||
@@ -2,53 +2,95 @@
|
||||
# All external data source URLs should be configured here
|
||||
|
||||
arcgis:
|
||||
# ArcGIS 海缆 GeoJSON 查询接口
|
||||
cable_url: "https://services.arcgis.com/6DIQcwlPy8knb6sg/ArcGIS/rest/services/SubmarineCables/FeatureServer/2/query"
|
||||
# ArcGIS 登陆点 GeoJSON 查询接口
|
||||
landing_point_url: "https://services.arcgis.com/6DIQcwlPy8knb6sg/ArcGIS/rest/services/SubmarineCables/FeatureServer/1/query"
|
||||
# ArcGIS 海缆与登陆点关联关系查询接口
|
||||
cable_landing_relation_url: "https://services.arcgis.com/6DIQcwlPy8knb6sg/ArcGIS/rest/services/SubmarineCables/FeatureServer/3/query"
|
||||
|
||||
fao:
|
||||
# FAO 登陆点 CSV 下载地址
|
||||
landing_point_url: "https://data.apps.fao.org/catalog/dataset/1b75ff21-92f2-4b96-9b7b-98e8aa65ad5d/resource/b6071077-d1d4-4e97-aa00-42e902847c87/download/landing-point-geo.csv"
|
||||
|
||||
telegeography:
|
||||
# TeleGeography 海缆/系统主数据源,当前使用 GitHub 镜像 JSON
|
||||
cable_url: "https://raw.githubusercontent.com/lintaojlu/submarine_cable_information/main/cable.json"
|
||||
# TeleGeography 登陆点主数据源,当前使用 GitHub 镜像 JSON
|
||||
landing_point_url: "https://raw.githubusercontent.com/lintaojlu/submarine_cable_information/main/landing_point.json"
|
||||
# TeleGeography 历史 API 存档,用于 cable collector 的 fallback
|
||||
archived_cable_url: "https://web.archive.org/web/2024/https://www.submarinecablemap.com/api/v3/cable"
|
||||
# TeleGeography 官网页面,用于 cable collector 的最终 HTML 抓取 fallback
|
||||
live_map_url: "https://www.submarinecablemap.com"
|
||||
|
||||
huggingface:
|
||||
# Hugging Face 模型目录 API
|
||||
models_url: "https://huggingface.co/api/models"
|
||||
# Hugging Face 数据集目录 API
|
||||
datasets_url: "https://huggingface.co/api/datasets"
|
||||
# Hugging Face Spaces 目录 API
|
||||
spaces_url: "https://huggingface.co/api/spaces"
|
||||
|
||||
cloudflare:
|
||||
# Cloudflare Radar 设备类型摘要接口
|
||||
radar_device_url: "https://api.cloudflare.com/client/v4/radar/http/summary/device_type"
|
||||
# Cloudflare Radar 请求量时间序列接口
|
||||
radar_traffic_url: "https://api.cloudflare.com/client/v4/radar/http/timeseries/requests"
|
||||
# Cloudflare Radar 热点地理位置接口
|
||||
radar_top_locations_url: "https://api.cloudflare.com/client/v4/radar/http/top/locations"
|
||||
|
||||
peeringdb:
|
||||
# PeeringDB IXP API
|
||||
ixp_url: "https://www.peeringdb.com/api/ix"
|
||||
# PeeringDB Network API
|
||||
network_url: "https://www.peeringdb.com/api/net"
|
||||
# PeeringDB Facility API
|
||||
facility_url: "https://www.peeringdb.com/api/fac"
|
||||
|
||||
top500:
|
||||
# TOP500 榜单页面,用于主表抓取
|
||||
url: "https://top500.org/lists/top500/list/2025/11/"
|
||||
# TOP500 站点根地址,用于拼详情页链接
|
||||
base_url: "https://top500.org"
|
||||
|
||||
epoch_ai:
|
||||
# Epoch AI GPU Cluster 页面
|
||||
gpu_clusters_url: "https://epoch.ai/data/gpu-clusters"
|
||||
|
||||
spacetrack:
|
||||
# Space-Track 站点根地址,用于首页访问和登录地址推导
|
||||
base_url: "https://www.space-track.org"
|
||||
# Space-Track TLE 主查询接口
|
||||
tle_query_url: "https://www.space-track.org/basicspacedata/query/class/gp/orderby/EPOCH%20desc/limit/1000/format/json"
|
||||
|
||||
celestrak:
|
||||
# CelesTrak TLE 基础接口,collector 会在其后拼接 GROUP / FORMAT 参数
|
||||
base_url: "https://celestrak.org/NORAD/elements/gp.php"
|
||||
|
||||
ris_live:
|
||||
# RIPE RIS Live 流式订阅地址
|
||||
url: "https://ris-live.ripe.net/v1/stream/?format=json&client=planet-ris-live"
|
||||
|
||||
bgpstream:
|
||||
# CAIDA BGPStream Broker API
|
||||
url: "https://broker.bgpstream.caida.org/v2"
|
||||
|
||||
iptoasn:
|
||||
# IPtoASN prefix geography 合并数据下载地址
|
||||
combined_url: "https://iptoasn.com/data/ip2asn-combined.tsv.gz"
|
||||
|
||||
opengeofeed:
|
||||
# OpenGeoFeed 公共 geofeed CSV
|
||||
public_csv_url: "https://opengeofeed.org/feed/public.csv"
|
||||
|
||||
nro:
|
||||
# NRO delegated stats 下载地址
|
||||
delegated_stats_url: "https://ftp.ripe.net/pub/stats/ripencc/nro-stats/latest/nro-delegated-stats"
|
||||
|
||||
news_live_streams:
|
||||
# IPTV-org 频道元数据 JSON
|
||||
channels_url: "https://iptv-org.github.io/api/channels.json"
|
||||
# IPTV-org 频道播放流 JSON
|
||||
streams_url: "https://iptv-org.github.io/api/streams.json"
|
||||
# IPTV-org 台标 JSON
|
||||
logos_url: "https://iptv-org.github.io/api/logos.json"
|
||||
@@ -155,6 +155,13 @@ DEFAULT_DATASOURCES = {
|
||||
"priority": "P1",
|
||||
"frequency_minutes": 1440,
|
||||
},
|
||||
"news_live_streams": {
|
||||
"id": 26,
|
||||
"name": "News Live Streams",
|
||||
"module": "L4",
|
||||
"priority": "P2",
|
||||
"frequency_minutes": 720,
|
||||
},
|
||||
}
|
||||
|
||||
ID_TO_COLLECTOR = {info["id"]: name for name, info in DEFAULT_DATASOURCES.items()}
|
||||
|
||||
@@ -0,0 +1,161 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
|
||||
from collections.abc import Mapping, Sequence
|
||||
from typing import Any
|
||||
|
||||
from app.core.request_context import get_request_id
|
||||
|
||||
DEFAULT_SERVICE = "backend"
|
||||
DEFAULT_EVENT = "app.log"
|
||||
DEFAULT_LOG_LEVEL = os.getenv("PLANET_LOG_LEVEL", "INFO").upper()
|
||||
REDACTED = "[REDACTED]"
|
||||
SENSITIVE_FIELD_NAMES = {
|
||||
"access_token",
|
||||
"api_key",
|
||||
"authorization",
|
||||
"cookie",
|
||||
"password",
|
||||
"refresh_token",
|
||||
"secret",
|
||||
"token",
|
||||
}
|
||||
SENSITIVE_TEXT_PATTERNS = (
|
||||
re.compile(r"(?i)(authorization\s*[:=]\s*)(.+)"),
|
||||
re.compile(r"(?i)(bearer\s+)([A-Za-z0-9._\-]+)"),
|
||||
re.compile(r"(?i)(token\s*[:=]\s*)(.+)"),
|
||||
re.compile(r"(?i)(password\s*[:=]\s*)(.+)"),
|
||||
re.compile(r"(?i)(cookie\s*[:=]\s*)(.+)"),
|
||||
)
|
||||
|
||||
|
||||
def sanitize_log_value(value: Any) -> Any:
|
||||
if isinstance(value, Mapping):
|
||||
return {
|
||||
str(key): (REDACTED if str(key).lower() in SENSITIVE_FIELD_NAMES else sanitize_log_value(item))
|
||||
for key, item in value.items()
|
||||
}
|
||||
if isinstance(value, Sequence) and not isinstance(value, (str, bytes, bytearray)):
|
||||
return [sanitize_log_value(item) for item in value]
|
||||
if isinstance(value, str):
|
||||
sanitized = value
|
||||
for pattern in SENSITIVE_TEXT_PATTERNS:
|
||||
sanitized = pattern.sub(lambda match: f"{match.group(1)}{REDACTED}", sanitized)
|
||||
return sanitized
|
||||
return value
|
||||
|
||||
|
||||
def _normalize_context(context: Any) -> dict[str, Any]:
|
||||
if context is None:
|
||||
return {}
|
||||
if isinstance(context, Mapping):
|
||||
sanitized = sanitize_log_value(context)
|
||||
return {str(key): value for key, value in sanitized.items()}
|
||||
return {"value": sanitize_log_value(context)}
|
||||
|
||||
|
||||
class PlanetContextFilter(logging.Filter):
|
||||
def filter(self, record: logging.LogRecord) -> bool:
|
||||
record.request_id = getattr(record, "request_id", None) or get_request_id() or "-"
|
||||
record.service = getattr(record, "service", None) or DEFAULT_SERVICE
|
||||
record.event = getattr(record, "event", None) or DEFAULT_EVENT
|
||||
record.context = _normalize_context(getattr(record, "context", None))
|
||||
record.message = sanitize_log_value(record.getMessage())
|
||||
return True
|
||||
|
||||
|
||||
class PlanetFormatter(logging.Formatter):
|
||||
def format(self, record: logging.LogRecord) -> str:
|
||||
timestamp = self.formatTime(record, self.datefmt)
|
||||
level = record.levelname
|
||||
service = getattr(record, "service", DEFAULT_SERVICE)
|
||||
module_name = record.name
|
||||
event = getattr(record, "event", DEFAULT_EVENT)
|
||||
request_id = getattr(record, "request_id", "-")
|
||||
message = sanitize_log_value(record.getMessage())
|
||||
context = _normalize_context(getattr(record, "context", None))
|
||||
context_suffix = ""
|
||||
if context:
|
||||
context_suffix = f" context={json.dumps(context, ensure_ascii=False, sort_keys=True)}"
|
||||
rendered = (
|
||||
f"{timestamp} {level} service={service} module={module_name} "
|
||||
f"event={event} request_id={request_id} message={message}{context_suffix}"
|
||||
)
|
||||
if record.exc_info:
|
||||
rendered = f"{rendered}\n{self.formatException(record.exc_info)}"
|
||||
return rendered
|
||||
|
||||
|
||||
class PlanetLoggerAdapter(logging.LoggerAdapter):
|
||||
def process(self, msg: Any, kwargs: dict[str, Any]) -> tuple[Any, dict[str, Any]]:
|
||||
extra = dict(self.extra)
|
||||
extra.update(kwargs.get("extra", {}))
|
||||
if "context" in extra:
|
||||
extra["context"] = _normalize_context(extra.get("context"))
|
||||
kwargs["extra"] = extra
|
||||
return sanitize_log_value(msg), kwargs
|
||||
|
||||
def log_event(
|
||||
self,
|
||||
level: int,
|
||||
message: str,
|
||||
*,
|
||||
event: str,
|
||||
context: Mapping[str, Any] | None = None,
|
||||
**extra: Any,
|
||||
) -> None:
|
||||
self.log(level, message, extra={"event": event, "context": context or {}, **extra})
|
||||
|
||||
def debug_event(self, message: str, *, event: str, context: Mapping[str, Any] | None = None, **extra: Any) -> None:
|
||||
self.log_event(logging.DEBUG, message, event=event, context=context, **extra)
|
||||
|
||||
def info_event(self, message: str, *, event: str, context: Mapping[str, Any] | None = None, **extra: Any) -> None:
|
||||
self.log_event(logging.INFO, message, event=event, context=context, **extra)
|
||||
|
||||
def warning_event(self, message: str, *, event: str, context: Mapping[str, Any] | None = None, **extra: Any) -> None:
|
||||
self.log_event(logging.WARNING, message, event=event, context=context, **extra)
|
||||
|
||||
def error_event(self, message: str, *, event: str, context: Mapping[str, Any] | None = None, **extra: Any) -> None:
|
||||
self.log_event(logging.ERROR, message, event=event, context=context, **extra)
|
||||
|
||||
def exception_event(
|
||||
self,
|
||||
message: str,
|
||||
*,
|
||||
event: str,
|
||||
context: Mapping[str, Any] | None = None,
|
||||
**extra: Any,
|
||||
) -> None:
|
||||
self.error(message, exc_info=True, extra={"event": event, "context": context or {}, **extra})
|
||||
|
||||
|
||||
def get_logger(name: str, *, service: str = DEFAULT_SERVICE) -> PlanetLoggerAdapter:
|
||||
return PlanetLoggerAdapter(logging.getLogger(name), {"service": service})
|
||||
|
||||
|
||||
def configure_logging(level: str | None = None) -> None:
|
||||
root_logger = logging.getLogger()
|
||||
if getattr(configure_logging, "_configured", False):
|
||||
if level:
|
||||
root_logger.setLevel(level.upper())
|
||||
return
|
||||
|
||||
handler = logging.StreamHandler()
|
||||
handler.setFormatter(PlanetFormatter(datefmt="%Y-%m-%d %H:%M:%S"))
|
||||
handler.addFilter(PlanetContextFilter())
|
||||
|
||||
root_logger.handlers.clear()
|
||||
root_logger.addHandler(handler)
|
||||
root_logger.setLevel((level or DEFAULT_LOG_LEVEL).upper())
|
||||
|
||||
for logger_name in ("uvicorn", "uvicorn.error", "uvicorn.access"):
|
||||
target_logger = logging.getLogger(logger_name)
|
||||
target_logger.handlers.clear()
|
||||
target_logger.propagate = True
|
||||
|
||||
logging.captureWarnings(True)
|
||||
configure_logging._configured = True
|
||||
@@ -0,0 +1,14 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from contextvars import ContextVar
|
||||
|
||||
|
||||
request_id_context: ContextVar[str | None] = ContextVar("request_id", default=None)
|
||||
|
||||
|
||||
def set_request_id(request_id: str | None) -> None:
|
||||
request_id_context.set(request_id)
|
||||
|
||||
|
||||
def get_request_id() -> str | None:
|
||||
return request_id_context.get()
|
||||
@@ -5,10 +5,22 @@ from sqlalchemy.ext.asyncio import AsyncSession, create_async_engine, async_sess
|
||||
from sqlalchemy.orm import declarative_base
|
||||
|
||||
from app.core.config import settings
|
||||
from app.core.logging import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
DB_POOL_CONFIG = {
|
||||
"pool_pre_ping": True,
|
||||
"pool_recycle": 1800,
|
||||
"pool_size": 10,
|
||||
"max_overflow": 20,
|
||||
"pool_timeout": 30,
|
||||
}
|
||||
|
||||
engine = create_async_engine(
|
||||
settings.DATABASE_URL,
|
||||
echo=settings.DEBUG if hasattr(settings, "DEBUG") else False,
|
||||
**DB_POOL_CONFIG,
|
||||
)
|
||||
|
||||
async_session_factory = async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False)
|
||||
@@ -95,6 +107,21 @@ async def init_db():
|
||||
import app.models.bgp_observation # noqa: F401
|
||||
import app.models.collected_data # noqa: F401
|
||||
import app.models.system_setting # noqa: F401
|
||||
import app.models.playground_session # noqa: F401
|
||||
import app.models.playground_message # noqa: F401
|
||||
import app.models.system_log # noqa: F401
|
||||
|
||||
logger.warning_event(
|
||||
"Database pool settings active",
|
||||
event="database.pool.initialized",
|
||||
context={
|
||||
"pool_pre_ping": DB_POOL_CONFIG["pool_pre_ping"],
|
||||
"pool_recycle": DB_POOL_CONFIG["pool_recycle"],
|
||||
"pool_size": DB_POOL_CONFIG["pool_size"],
|
||||
"max_overflow": DB_POOL_CONFIG["max_overflow"],
|
||||
"pool_timeout": DB_POOL_CONFIG["pool_timeout"],
|
||||
},
|
||||
)
|
||||
|
||||
async with engine.begin() as conn:
|
||||
await conn.run_sync(Base.metadata.create_all)
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
from contextlib import asynccontextmanager
|
||||
from uuid import uuid4
|
||||
|
||||
from fastapi import FastAPI
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
@@ -7,6 +8,8 @@ from starlette.middleware.base import BaseHTTPMiddleware
|
||||
from app.api.main import api_router
|
||||
from app.api.v1 import websocket
|
||||
from app.core.config import settings
|
||||
from app.core.logging import configure_logging
|
||||
from app.core.request_context import set_request_id
|
||||
from app.core.websocket.broadcaster import broadcaster
|
||||
from app.db.session import init_db
|
||||
from app.services.scheduler import (
|
||||
@@ -17,6 +20,9 @@ from app.services.scheduler import (
|
||||
)
|
||||
|
||||
|
||||
configure_logging()
|
||||
|
||||
|
||||
class WebSocketCORSMiddleware(BaseHTTPMiddleware):
|
||||
async def dispatch(self, request, call_next):
|
||||
if request.url.path.startswith("/ws") and request.method == "GET":
|
||||
@@ -28,6 +34,18 @@ class WebSocketCORSMiddleware(BaseHTTPMiddleware):
|
||||
return await call_next(request)
|
||||
|
||||
|
||||
class RequestContextMiddleware(BaseHTTPMiddleware):
|
||||
async def dispatch(self, request, call_next):
|
||||
request_id = request.headers.get("X-Request-ID") or uuid4().hex
|
||||
set_request_id(request_id)
|
||||
try:
|
||||
response = await call_next(request)
|
||||
finally:
|
||||
set_request_id(None)
|
||||
response.headers["X-Request-ID"] = request_id
|
||||
return response
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
await init_db()
|
||||
@@ -58,6 +76,7 @@ app.add_middleware(
|
||||
allow_headers=["*"],
|
||||
)
|
||||
|
||||
app.add_middleware(RequestContextMiddleware)
|
||||
app.add_middleware(WebSocketCORSMiddleware)
|
||||
|
||||
app.include_router(api_router, prefix="/api/v1")
|
||||
|
||||
@@ -9,6 +9,9 @@ from app.models.bgp_anomaly import BGPAnomaly
|
||||
from app.models.bgp_incident import BGPIncident
|
||||
from app.models.bgp_observation import BGPObservation
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.models.playground_session import PlaygroundSession
|
||||
from app.models.playground_message import PlaygroundMessage
|
||||
from app.models.system_log import SystemLog, AuditLog
|
||||
|
||||
__all__ = [
|
||||
"User",
|
||||
@@ -24,4 +27,6 @@ __all__ = [
|
||||
"BGPAnomaly",
|
||||
"BGPIncident",
|
||||
"BGPObservation",
|
||||
"SystemLog",
|
||||
"AuditLog",
|
||||
]
|
||||
@@ -0,0 +1,40 @@
|
||||
from sqlalchemy import JSON, Boolean, Column, DateTime, ForeignKey, Integer, String, Text
|
||||
from sqlalchemy.sql import func
|
||||
|
||||
from app.db.session import Base
|
||||
|
||||
|
||||
class PlaygroundMessage(Base):
|
||||
__tablename__ = "playground_messages"
|
||||
|
||||
id = Column(Integer, primary_key=True, autoincrement=True)
|
||||
public_id = Column(String(64), unique=True, index=True, nullable=False)
|
||||
session_id = Column(Integer, ForeignKey("playground_sessions.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
user_id = Column(Integer, ForeignKey("users.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
parent_message_id = Column(Integer, ForeignKey("playground_messages.id", ondelete="SET NULL"), nullable=True)
|
||||
role = Column(String(20), nullable=False)
|
||||
kind = Column(String(20), nullable=False, default="message")
|
||||
status = Column(String(20), nullable=False, default="done")
|
||||
title = Column(String(255), nullable=True)
|
||||
content = Column(Text, nullable=False, default="")
|
||||
thinking_content = Column(Text, nullable=False, default="")
|
||||
meta = Column(JSON, nullable=False, default=list)
|
||||
provider = Column(String(100), nullable=True)
|
||||
model = Column(String(200), nullable=True)
|
||||
request_id = Column(String(100), nullable=True)
|
||||
raw_response = Column(JSON, nullable=False, default=dict)
|
||||
content_blocks = Column(JSON, nullable=False, default=list)
|
||||
text_blocks = Column(JSON, nullable=False, default=list)
|
||||
thinking_blocks = Column(JSON, nullable=False, default=list)
|
||||
sort_order = Column(Integer, nullable=False, default=0, index=True)
|
||||
is_visible = Column(Boolean, nullable=False, default=True)
|
||||
created_at = Column(DateTime(timezone=True), server_default=func.now(), nullable=False)
|
||||
updated_at = Column(
|
||||
DateTime(timezone=True),
|
||||
server_default=func.now(),
|
||||
onupdate=func.now(),
|
||||
nullable=False,
|
||||
)
|
||||
|
||||
def __repr__(self):
|
||||
return f"<PlaygroundMessage public_id={self.public_id} role={self.role} status={self.status}>"
|
||||
@@ -0,0 +1,27 @@
|
||||
from sqlalchemy import JSON, Column, DateTime, ForeignKey, Integer, String, UniqueConstraint
|
||||
from sqlalchemy.sql import func
|
||||
|
||||
from app.db.session import Base
|
||||
|
||||
|
||||
class PlaygroundSession(Base):
|
||||
__tablename__ = "playground_sessions"
|
||||
__table_args__ = (
|
||||
UniqueConstraint("user_id", "session_key", name="uq_playground_sessions_user_session_key"),
|
||||
)
|
||||
|
||||
id = Column(Integer, primary_key=True, autoincrement=True)
|
||||
user_id = Column(Integer, ForeignKey("users.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
session_key = Column(String(100), nullable=False, default="default")
|
||||
title = Column(String(200), nullable=False, default="Playground 会话")
|
||||
state = Column(JSON, nullable=False, default={})
|
||||
created_at = Column(DateTime(timezone=True), server_default=func.now(), nullable=False)
|
||||
updated_at = Column(
|
||||
DateTime(timezone=True),
|
||||
server_default=func.now(),
|
||||
onupdate=func.now(),
|
||||
nullable=False,
|
||||
)
|
||||
|
||||
def __repr__(self):
|
||||
return f"<PlaygroundSession user_id={self.user_id} session_key={self.session_key}>"
|
||||
@@ -0,0 +1,40 @@
|
||||
from sqlalchemy import JSON, Column, DateTime, Integer, String, Text
|
||||
from sqlalchemy.sql import func
|
||||
|
||||
from app.db.session import Base
|
||||
|
||||
|
||||
class SystemLog(Base):
|
||||
__tablename__ = "system_logs"
|
||||
|
||||
id = Column(Integer, primary_key=True, autoincrement=True)
|
||||
occurred_at = Column(DateTime(timezone=True), server_default=func.now(), index=True)
|
||||
source = Column(String(50), nullable=False, index=True)
|
||||
service = Column(String(50), nullable=True)
|
||||
module = Column(String(120), nullable=True)
|
||||
event = Column(String(160), nullable=True, index=True)
|
||||
level = Column(String(20), nullable=False, index=True)
|
||||
message = Column(Text, nullable=False)
|
||||
request_id = Column(String(64), nullable=True, index=True)
|
||||
trace_id = Column(String(64), nullable=True)
|
||||
user_id = Column(Integer, nullable=True, index=True)
|
||||
category = Column(String(80), nullable=True, index=True)
|
||||
context = Column(JSON, nullable=False, default=dict)
|
||||
created_at = Column(DateTime(timezone=True), server_default=func.now())
|
||||
|
||||
|
||||
class AuditLog(Base):
|
||||
__tablename__ = "audit_logs"
|
||||
|
||||
id = Column(Integer, primary_key=True, autoincrement=True)
|
||||
occurred_at = Column(DateTime(timezone=True), server_default=func.now(), index=True)
|
||||
actor_id = Column(Integer, nullable=True, index=True)
|
||||
actor_name = Column(String(255), nullable=True)
|
||||
action = Column(String(120), nullable=False, index=True)
|
||||
target_type = Column(String(80), nullable=True)
|
||||
target_id = Column(String(120), nullable=True)
|
||||
result = Column(String(40), nullable=True, index=True)
|
||||
request_id = Column(String(64), nullable=True, index=True)
|
||||
ip = Column(String(64), nullable=True)
|
||||
details = Column(JSON, nullable=False, default=dict)
|
||||
created_at = Column(DateTime(timezone=True), server_default=func.now())
|
||||
@@ -3,6 +3,14 @@ from typing import Any
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class AIContentBlock(BaseModel):
|
||||
type: str
|
||||
text: str | None = None
|
||||
thinking: str | None = None
|
||||
signature: str | None = None
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class SituationalAnalysisRequest(BaseModel):
|
||||
title: str = Field(..., min_length=1, max_length=200)
|
||||
objective: str = Field(..., min_length=1, max_length=1000)
|
||||
@@ -10,18 +18,158 @@ class SituationalAnalysisRequest(BaseModel):
|
||||
observations: list[str] = Field(default_factory=list)
|
||||
constraints: list[str] = Field(default_factory=list)
|
||||
preferred_model: str | None = Field(default=None, max_length=200)
|
||||
thinking: dict[str, Any] | None = None
|
||||
|
||||
|
||||
class BGPBriefRequest(BaseModel):
|
||||
incident_limit: int = Field(default=5, ge=1, le=10)
|
||||
anomaly_limit: int = Field(default=6, ge=1, le=12)
|
||||
collector_limit: int = Field(default=5, ge=1, le=10)
|
||||
preferred_model: str | None = Field(default=None, max_length=200)
|
||||
thinking: dict[str, Any] | None = None
|
||||
|
||||
|
||||
class AlertBriefRequest(BaseModel):
|
||||
alert_limit: int = Field(default=8, ge=1, le=20)
|
||||
preferred_model: str | None = Field(default=None, max_length=200)
|
||||
thinking: dict[str, Any] | None = None
|
||||
|
||||
|
||||
class SituationalAlertBriefRequest(BaseModel):
|
||||
preferred_model: str | None = Field(default=None, max_length=200)
|
||||
thinking: dict[str, Any] | None = None
|
||||
|
||||
|
||||
class SituationalAnalysisResponse(BaseModel):
|
||||
provider: str
|
||||
model: str
|
||||
content: str
|
||||
content_blocks: list[AIContentBlock] = Field(default_factory=list)
|
||||
text_blocks: list[str] = Field(default_factory=list)
|
||||
thinking_blocks: list[str] = Field(default_factory=list)
|
||||
raw_response: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class BGPBriefRecordSummary(BaseModel):
|
||||
id: str
|
||||
title: str
|
||||
provider: str
|
||||
model: str
|
||||
request_id: str | None = None
|
||||
generated_at: str
|
||||
|
||||
|
||||
class BGPBriefRecordResponse(BGPBriefRecordSummary):
|
||||
content_markdown: str
|
||||
facts: list[str] = Field(default_factory=list)
|
||||
context: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class AlertBriefResponse(SituationalAnalysisResponse):
|
||||
title: str
|
||||
objective: str
|
||||
facts: list[str] = Field(default_factory=list)
|
||||
context: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class SituationalAlertBriefResponse(SituationalAnalysisResponse):
|
||||
title: str
|
||||
objective: str
|
||||
facts: list[str] = Field(default_factory=list)
|
||||
context: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class AIProviderStatusResponse(BaseModel):
|
||||
provider: str
|
||||
api: str | None = None
|
||||
enabled: bool
|
||||
configured: bool
|
||||
model: str | None = None
|
||||
base_url: str | None = None
|
||||
|
||||
|
||||
class PlaygroundSessionState(BaseModel):
|
||||
messages: list[dict[str, Any]] = Field(default_factory=list)
|
||||
selectedPresetKey: str = Field(default="bgp-brief", max_length=100)
|
||||
title: str = Field(default="", max_length=200)
|
||||
objective: str = Field(default="", max_length=1000)
|
||||
constraints: str = Field(default="")
|
||||
inputValue: str = Field(default="")
|
||||
analysis: dict[str, Any] | None = None
|
||||
latestAnalysisMessageId: str | None = Field(default=None, max_length=200)
|
||||
analysisMeta: dict[str, Any] = Field(default_factory=dict)
|
||||
helpExpanded: bool = True
|
||||
|
||||
|
||||
class PlaygroundSessionUpsertRequest(BaseModel):
|
||||
session_key: str = Field(default="default", min_length=1, max_length=100)
|
||||
title: str | None = Field(default=None, max_length=200)
|
||||
state: PlaygroundSessionState
|
||||
|
||||
|
||||
class PlaygroundMessageRecord(BaseModel):
|
||||
id: str
|
||||
role: str
|
||||
kind: str = "message"
|
||||
status: str = "done"
|
||||
title: str | None = None
|
||||
content: str = ""
|
||||
thinking_content: str = ""
|
||||
meta: list[str] = Field(default_factory=list)
|
||||
markdown: bool = True
|
||||
provider: str | None = None
|
||||
model: str | None = None
|
||||
request_id: str | None = None
|
||||
raw_response: dict[str, Any] = Field(default_factory=dict)
|
||||
content_blocks: list[dict[str, Any]] = Field(default_factory=list)
|
||||
text_blocks: list[str] = Field(default_factory=list)
|
||||
thinking_blocks: list[str] = Field(default_factory=list)
|
||||
parent_message_id: str | None = None
|
||||
created_at: str
|
||||
updated_at: str
|
||||
|
||||
|
||||
class PlaygroundSessionResponse(BaseModel):
|
||||
id: str
|
||||
session_key: str
|
||||
title: str
|
||||
state: PlaygroundSessionState
|
||||
created_at: str
|
||||
updated_at: str
|
||||
|
||||
|
||||
class PlaygroundThreadResponse(BaseModel):
|
||||
session: PlaygroundSessionResponse
|
||||
messages: list[PlaygroundMessageRecord] = Field(default_factory=list)
|
||||
|
||||
|
||||
class PlaygroundMessageCreateRequest(BaseModel):
|
||||
session_key: str = Field(default="default", min_length=1, max_length=100)
|
||||
title: str = Field(..., min_length=1, max_length=200)
|
||||
objective: str = Field(..., min_length=1, max_length=1000)
|
||||
constraints: str = Field(default="")
|
||||
input: str = Field(..., min_length=1)
|
||||
selected_preset_key: str = Field(default="bgp-brief", max_length=100)
|
||||
help_expanded: bool = True
|
||||
|
||||
|
||||
class PlaygroundMessageActionResponse(BaseModel):
|
||||
session: PlaygroundSessionResponse
|
||||
messages: list[PlaygroundMessageRecord] = Field(default_factory=list)
|
||||
active_message_id: str | None = None
|
||||
|
||||
|
||||
class PlaygroundMessageStopRequest(BaseModel):
|
||||
session_key: str = Field(default="default", min_length=1, max_length=100)
|
||||
message_id: str = Field(..., min_length=1, max_length=64)
|
||||
|
||||
|
||||
class PlaygroundMessageResendRequest(BaseModel):
|
||||
session_key: str = Field(default="default", min_length=1, max_length=100)
|
||||
user_message_id: str = Field(..., min_length=1, max_length=64)
|
||||
|
||||
|
||||
class PlaygroundMessageEditRequest(BaseModel):
|
||||
session_key: str = Field(default="default", min_length=1, max_length=100)
|
||||
user_message_id: str = Field(..., min_length=1, max_length=64)
|
||||
content: str = Field(..., min_length=1)
|
||||
@@ -0,0 +1,5 @@
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class AlertResolutionRequest(BaseModel):
|
||||
resolution: str = Field(..., min_length=1, max_length=1000)
|
||||
@@ -0,0 +1,103 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections import Counter
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.models.alert import Alert, AlertSeverity, AlertStatus
|
||||
from app.schemas.ai import AlertBriefRequest, SituationalAnalysisRequest
|
||||
|
||||
|
||||
def _format_counter(counter: Counter[str], empty_text: str = "无") -> str:
|
||||
if not counter:
|
||||
return empty_text
|
||||
return ",".join(f"{key} {value}" for key, value in counter.items())
|
||||
|
||||
|
||||
async def build_alert_brief_request(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
alert_limit: int = 8,
|
||||
) -> tuple[SituationalAnalysisRequest, list[str], dict[str, Any]]:
|
||||
recent_alerts_result = await db.execute(
|
||||
select(Alert)
|
||||
.order_by(Alert.created_at.desc(), Alert.id.desc())
|
||||
.limit(max(alert_limit, 1))
|
||||
)
|
||||
total_result = await db.execute(select(func.count(Alert.id)))
|
||||
active_result = await db.execute(select(func.count(Alert.id)).where(Alert.status == AlertStatus.ACTIVE))
|
||||
acknowledged_result = await db.execute(
|
||||
select(func.count(Alert.id)).where(Alert.status == AlertStatus.ACKNOWLEDGED)
|
||||
)
|
||||
resolved_result = await db.execute(select(func.count(Alert.id)).where(Alert.status == AlertStatus.RESOLVED))
|
||||
|
||||
recent_alerts = recent_alerts_result.scalars().all()
|
||||
total_alerts = total_result.scalar() or 0
|
||||
active_alerts = active_result.scalar() or 0
|
||||
acknowledged_alerts = acknowledged_result.scalar() or 0
|
||||
resolved_alerts = resolved_result.scalar() or 0
|
||||
|
||||
severity_counts = Counter((item.severity.value if item.severity else "unknown") for item in recent_alerts)
|
||||
status_counts = Counter((item.status.value if item.status else "unknown") for item in recent_alerts)
|
||||
datasource_counts = Counter((item.datasource_name or "未命名数据源") for item in recent_alerts)
|
||||
active_datasource_counts = Counter(
|
||||
(item.datasource_name or "未命名数据源")
|
||||
for item in recent_alerts
|
||||
if item.status == AlertStatus.ACTIVE
|
||||
)
|
||||
|
||||
facts = [
|
||||
f"告警总量 {total_alerts} 条,其中 active {active_alerts} 条、acknowledged {acknowledged_alerts} 条、resolved {resolved_alerts} 条。",
|
||||
f"最近告警严重度分布:{_format_counter(severity_counts)}。",
|
||||
f"最近告警状态分布:{_format_counter(status_counts)}。",
|
||||
f"最近告警数据源分布:{_format_counter(Counter(dict(datasource_counts.most_common(6))))}。",
|
||||
]
|
||||
|
||||
if active_datasource_counts:
|
||||
facts.append(
|
||||
"当前待处理告警主要集中在:"
|
||||
+ _format_counter(Counter(dict(active_datasource_counts.most_common(5))))
|
||||
+ "。"
|
||||
)
|
||||
|
||||
if recent_alerts:
|
||||
facts.append(
|
||||
"最近告警摘录:"
|
||||
+ ";".join(
|
||||
[
|
||||
f"{item.datasource_name or '未命名数据源'} / {item.severity.value if item.severity else '-'} / {item.status.value if item.status else '-'} / {item.message or '-'}"
|
||||
for item in recent_alerts[:6]
|
||||
]
|
||||
)
|
||||
)
|
||||
|
||||
context = {
|
||||
"source": "alerts",
|
||||
"total_alerts": total_alerts,
|
||||
"active_alerts": active_alerts,
|
||||
"acknowledged_alerts": acknowledged_alerts,
|
||||
"resolved_alerts": resolved_alerts,
|
||||
"severity_distribution": dict(severity_counts),
|
||||
"status_distribution": dict(status_counts),
|
||||
"top_datasources": dict(datasource_counts.most_common(6)),
|
||||
"top_active_datasources": dict(active_datasource_counts.most_common(5)),
|
||||
}
|
||||
|
||||
return (
|
||||
SituationalAnalysisRequest(
|
||||
title="告警态势 AI 简报",
|
||||
objective="基于当前告警总量、严重度、状态、数据源分布与最近告警摘录,生成一份面向值班人员的简明告警态势简报,突出待处理风险、告警集中点和优先动作。",
|
||||
observations=facts,
|
||||
constraints=[
|
||||
"明确区分事实、推断与建议。",
|
||||
"优先指出仍处于 active 状态且高严重度的告警簇。",
|
||||
"不要把 acknowledged 或 resolved 告警误判成当前仍在扩大。",
|
||||
"如果证据不足,请明确指出缺失的上下文。",
|
||||
],
|
||||
context=context,
|
||||
),
|
||||
facts,
|
||||
context,
|
||||
)
|
||||
@@ -0,0 +1,259 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections import Counter
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.api.v1.bgp import BGP_SOURCES
|
||||
from app.models.bgp_anomaly import BGPAnomaly
|
||||
from app.models.bgp_incident import BGPIncident
|
||||
from app.models.bgp_observation import BGPObservation
|
||||
from app.schemas.ai import SituationalAnalysisRequest
|
||||
from app.services.bgp_collectors import build_bgp_collector_coverage
|
||||
from app.services.bgp_enrichment import lookup_prefix_geography
|
||||
|
||||
|
||||
def _format_counter(counter: dict[str, int], empty_text: str = "无") -> str:
|
||||
if not counter:
|
||||
return empty_text
|
||||
return ",".join(f"{key} {value}" for key, value in counter.items())
|
||||
|
||||
|
||||
def _severity_rank(value: str | None) -> int:
|
||||
order = {
|
||||
"critical": 0,
|
||||
"high": 1,
|
||||
"medium": 2,
|
||||
"low": 3,
|
||||
"info": 4,
|
||||
}
|
||||
return order.get((value or "").lower(), 99)
|
||||
|
||||
|
||||
def _normalize_geo_key(country: str | None, city: str | None) -> str:
|
||||
if city and country:
|
||||
return f"{city}, {country}"
|
||||
return city or country or "未知区域"
|
||||
|
||||
|
||||
def _top_counter_items(counter: Counter[str], limit: int = 5) -> dict[str, int]:
|
||||
return {name: count for name, count in counter.most_common(limit) if name}
|
||||
|
||||
|
||||
def _collect_incident_regions(incidents: list[BGPIncident]) -> Counter[str]:
|
||||
counter: Counter[str] = Counter()
|
||||
for item in incidents:
|
||||
for region in item.affected_regions or []:
|
||||
if not isinstance(region, dict):
|
||||
continue
|
||||
counter[_normalize_geo_key(region.get("country"), region.get("city"))] += 1
|
||||
return counter
|
||||
|
||||
|
||||
def _collect_collector_regions(collectors: list[dict[str, Any]]) -> Counter[str]:
|
||||
counter: Counter[str] = Counter()
|
||||
for item in collectors:
|
||||
counter[_normalize_geo_key(item.get("country"), item.get("city"))] += int(item.get("recent_24h_observation_count") or 0)
|
||||
return counter
|
||||
|
||||
|
||||
def _format_geo_evidence(prefix_geographies: dict[str, dict[str, Any]], limit: int = 6) -> str:
|
||||
if not prefix_geographies:
|
||||
return "没有命中 prefix geography 证据。"
|
||||
|
||||
rows = []
|
||||
for prefix, item in list(prefix_geographies.items())[:limit]:
|
||||
region = _normalize_geo_key(item.get("country"), item.get("city"))
|
||||
source = item.get("source") or item.get("geography_mode") or "unknown"
|
||||
as_hint = item.get("asn")
|
||||
as_name = item.get("as_name")
|
||||
as_text = ""
|
||||
if as_hint:
|
||||
as_text = f" / ASN AS{as_hint}"
|
||||
if as_name:
|
||||
as_text += f" ({as_name})"
|
||||
rows.append(f"{prefix} -> {region} / 来源 {source}{as_text}")
|
||||
return ";".join(rows)
|
||||
|
||||
|
||||
async def build_bgp_brief_request(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
incident_limit: int = 5,
|
||||
anomaly_limit: int = 6,
|
||||
collector_limit: int = 5,
|
||||
) -> tuple[SituationalAnalysisRequest, list[str], dict[str, int | str | dict[str, int]]]:
|
||||
incidents_result = await db.execute(
|
||||
select(BGPIncident)
|
||||
.order_by(BGPIncident.created_at.desc(), BGPIncident.id.desc())
|
||||
.limit(max(incident_limit, 1))
|
||||
)
|
||||
anomalies_result = await db.execute(
|
||||
select(BGPAnomaly)
|
||||
.order_by(BGPAnomaly.created_at.desc(), BGPAnomaly.id.desc())
|
||||
.limit(max(anomaly_limit, 1))
|
||||
)
|
||||
observations_result = await db.execute(
|
||||
select(BGPObservation).where(BGPObservation.source.in_(BGP_SOURCES))
|
||||
)
|
||||
incident_count_result = await db.execute(select(func.count(BGPIncident.id)))
|
||||
anomaly_count_result = await db.execute(select(func.count(BGPAnomaly.id)))
|
||||
|
||||
incidents = incidents_result.scalars().all()
|
||||
anomalies = anomalies_result.scalars().all()
|
||||
observations = observations_result.scalars().all()
|
||||
collectors = await build_bgp_collector_coverage(db, source_filter=BGP_SOURCES)
|
||||
|
||||
total_incidents = incident_count_result.scalar() or 0
|
||||
total_anomalies = anomaly_count_result.scalar() or 0
|
||||
total_observations = len(observations)
|
||||
active_collectors = [item for item in collectors if item["observation_count"] > 0]
|
||||
|
||||
incident_status_counts = Counter((item.status or "unknown") for item in incidents)
|
||||
incident_severity_counts = Counter((item.severity or "unknown") for item in incidents)
|
||||
incident_type_counts = Counter((item.incident_type or "unknown") for item in incidents)
|
||||
anomaly_type_counts = Counter((item.anomaly_type or "unknown") for item in anomalies)
|
||||
event_type_counts = Counter((item.event_type or "unknown") for item in observations)
|
||||
incident_region_counts = _collect_incident_regions(incidents)
|
||||
|
||||
top_collectors = sorted(
|
||||
active_collectors,
|
||||
key=lambda item: (
|
||||
-int(item["recent_24h_observation_count"]),
|
||||
-int(item["observation_count"]),
|
||||
str(item["collector"]),
|
||||
),
|
||||
)[: max(collector_limit, 1)]
|
||||
collector_region_counts = _collect_collector_regions(top_collectors)
|
||||
|
||||
prefix_candidates = sorted(
|
||||
{
|
||||
prefix
|
||||
for item in incidents
|
||||
for prefix in (item.affected_prefixes or [])
|
||||
if prefix
|
||||
}
|
||||
| {item.prefix for item in anomalies if item.prefix}
|
||||
)
|
||||
prefix_geographies = await lookup_prefix_geography(db, prefix_candidates) if prefix_candidates else {}
|
||||
geography_region_counts = Counter(
|
||||
_normalize_geo_key(item.get("country"), item.get("city"))
|
||||
for item in prefix_geographies.values()
|
||||
if item.get("country") or item.get("city")
|
||||
)
|
||||
hotspot_region_counts = geography_region_counts + incident_region_counts
|
||||
collector_bias_regions = [
|
||||
region
|
||||
for region, count in collector_region_counts.most_common(3)
|
||||
if count > hotspot_region_counts.get(region, 0)
|
||||
]
|
||||
|
||||
observations_lines: list[str] = [
|
||||
f"当前共有 {total_incidents} 起 BGP incidents、{total_anomalies} 条 anomalies、{total_observations} 条原始观测事件。",
|
||||
f"活跃观测站 {len(active_collectors)} 个;近 24 小时事件数合计 {sum(int(item['recent_24h_observation_count']) for item in active_collectors)}。",
|
||||
f"最近 incidents 严重度分布:{_format_counter(dict(sorted(incident_severity_counts.items(), key=lambda item: _severity_rank(item[0]))))}。",
|
||||
f"最近 incidents 状态分布:{_format_counter(dict(incident_status_counts))}。",
|
||||
f"最近 incidents 类型分布:{_format_counter(dict(incident_type_counts.most_common(5)))}。",
|
||||
f"最近 anomalies 类型分布:{_format_counter(dict(anomaly_type_counts.most_common(6)))}。",
|
||||
f"观测事件类型分布:{_format_counter(dict(event_type_counts.most_common(6)))}。",
|
||||
]
|
||||
|
||||
if hotspot_region_counts:
|
||||
observations_lines.append(
|
||||
"区域热点事实层:"
|
||||
+ _format_counter(_top_counter_items(hotspot_region_counts, limit=5), empty_text="无明显区域聚集")
|
||||
+ "。"
|
||||
)
|
||||
|
||||
if prefix_geographies:
|
||||
observations_lines.append("Prefix geography 证据:" + _format_geo_evidence(prefix_geographies))
|
||||
|
||||
if collector_bias_regions:
|
||||
observations_lines.append(
|
||||
"观测偏差提示:重点观测站最近 24h 活跃度更集中在 "
|
||||
+ "、".join(collector_bias_regions)
|
||||
+ ",这些区域的事件升温结论需要结合 prefix geography 与 affected regions 交叉验证。"
|
||||
)
|
||||
elif top_collectors:
|
||||
observations_lines.append(
|
||||
"观测偏差提示:当前未发现明显高于区域热点事实层的单一观测站集中区域,但仍需区分 collector coverage 与真实区域风险。"
|
||||
)
|
||||
|
||||
if incidents:
|
||||
observations_lines.append(
|
||||
"最近 incident 摘要:" + ";".join(
|
||||
[
|
||||
f"{item.incident_type} / {item.severity} / {item.status}"
|
||||
f" / 前缀 {', '.join(item.affected_prefixes[:2]) if item.affected_prefixes else '-'}"
|
||||
f" / 观测站 {len(item.affected_collectors or [])} 个"
|
||||
for item in incidents
|
||||
]
|
||||
)
|
||||
)
|
||||
|
||||
if anomalies:
|
||||
observations_lines.append(
|
||||
"最近 anomaly 摘要:" + ";".join(
|
||||
[
|
||||
f"{item.anomaly_type} / {item.severity}"
|
||||
f" / 前缀 {item.prefix or '-'}"
|
||||
f" / ASN {item.new_origin_asn or item.origin_asn or '-'}"
|
||||
for item in anomalies
|
||||
]
|
||||
)
|
||||
)
|
||||
|
||||
if top_collectors:
|
||||
observations_lines.append(
|
||||
"重点观测站:" + ";".join(
|
||||
[
|
||||
f"{item['collector']} ({', '.join([part for part in [item.get('city'), item.get('country')] if part]) or '未知位置'})"
|
||||
f" / 近24h {item['recent_24h_observation_count']} 条"
|
||||
f" / 前缀 {item['prefix_count']} 个"
|
||||
for item in top_collectors
|
||||
]
|
||||
)
|
||||
)
|
||||
|
||||
context = {
|
||||
"source": "bgp-overview",
|
||||
"incident_total": total_incidents,
|
||||
"anomaly_total": total_anomalies,
|
||||
"observation_total": total_observations,
|
||||
"active_collectors": len(active_collectors),
|
||||
"top_incident_types": dict(incident_type_counts.most_common(5)),
|
||||
"top_anomaly_types": dict(anomaly_type_counts.most_common(6)),
|
||||
"top_event_types": dict(event_type_counts.most_common(6)),
|
||||
"region_hotspots": _top_counter_items(hotspot_region_counts, limit=6),
|
||||
"incident_regions": _top_counter_items(incident_region_counts, limit=6),
|
||||
"collector_bias_regions": collector_bias_regions,
|
||||
"prefix_geography_sources": dict(
|
||||
Counter(str(item.get("source") or "unknown") for item in prefix_geographies.values()).most_common(5)
|
||||
),
|
||||
"prefix_geography_sample": {
|
||||
prefix: {
|
||||
"country": item.get("country"),
|
||||
"city": item.get("city"),
|
||||
"source": item.get("source"),
|
||||
"asn": item.get("asn"),
|
||||
"as_name": item.get("as_name"),
|
||||
}
|
||||
for prefix, item in list(prefix_geographies.items())[:8]
|
||||
},
|
||||
}
|
||||
|
||||
return SituationalAnalysisRequest(
|
||||
title="BGP 态势 AI 简报",
|
||||
objective="基于当前 BGP incidents、anomalies、原始观测事件、观测站覆盖与 prefix geography 证据,生成一份面向操作员的简明态势简报,突出区域热点、观测偏差、当前风险、证据和优先动作。",
|
||||
observations=observations_lines,
|
||||
constraints=[
|
||||
"明确区分事实、推断与建议。",
|
||||
"优先指出需要立即关注的高严重度 incident 或异常模式。",
|
||||
"需要单独指出哪些区域结论来自 prefix geography / affected regions,哪些可能受 collector coverage 偏差影响。",
|
||||
"结论应服务值班排障,不要写成泛泛的模型演示文案。",
|
||||
"如果证据不足,要明确指出缺失数据。",
|
||||
],
|
||||
context=context,
|
||||
), observations_lines, context
|
||||
@@ -0,0 +1,160 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from uuid import uuid4
|
||||
|
||||
from app.core.config import ROOT_DIR
|
||||
from app.schemas.ai import BGPBriefRecordResponse, BGPBriefRecordSummary, SituationalAnalysisResponse
|
||||
|
||||
|
||||
_BRIEF_STORAGE_DIR = ROOT_DIR / "data" / "ai" / "bgp-briefs"
|
||||
_METADATA_PREFIX = "<!-- planet-bgp-brief-meta "
|
||||
_METADATA_SUFFIX = " -->"
|
||||
_BRIEF_TITLE = "BGP AI 简报"
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class _StoredBrief:
|
||||
id: str
|
||||
title: str
|
||||
provider: str
|
||||
model: str
|
||||
request_id: str | None
|
||||
generated_at: str
|
||||
content_markdown: str
|
||||
facts: list[str]
|
||||
context: dict[str, Any]
|
||||
path: Path
|
||||
|
||||
|
||||
def _ensure_storage_dir() -> Path:
|
||||
_BRIEF_STORAGE_DIR.mkdir(parents=True, exist_ok=True)
|
||||
return _BRIEF_STORAGE_DIR
|
||||
|
||||
|
||||
def _build_metadata_line(metadata: dict[str, Any]) -> str:
|
||||
return f"{_METADATA_PREFIX}{json.dumps(metadata, ensure_ascii=False)}{_METADATA_SUFFIX}"
|
||||
|
||||
|
||||
def _parse_brief_file(path: Path) -> _StoredBrief | None:
|
||||
try:
|
||||
raw_text = path.read_text(encoding="utf-8")
|
||||
except OSError:
|
||||
return None
|
||||
|
||||
first_line, separator, remainder = raw_text.partition("\n")
|
||||
if not separator or not first_line.startswith(_METADATA_PREFIX) or not first_line.endswith(_METADATA_SUFFIX):
|
||||
return None
|
||||
|
||||
metadata_payload = first_line[len(_METADATA_PREFIX) : -len(_METADATA_SUFFIX)]
|
||||
|
||||
try:
|
||||
metadata = json.loads(metadata_payload)
|
||||
except json.JSONDecodeError:
|
||||
return None
|
||||
|
||||
return _StoredBrief(
|
||||
id=str(metadata.get("id") or path.stem),
|
||||
title=str(metadata.get("title") or _BRIEF_TITLE),
|
||||
provider=str(metadata.get("provider") or "-"),
|
||||
model=str(metadata.get("model") or "-"),
|
||||
request_id=metadata.get("request_id"),
|
||||
generated_at=str(metadata.get("generated_at") or datetime.fromtimestamp(path.stat().st_mtime, UTC).isoformat()),
|
||||
content_markdown=remainder.lstrip("\n"),
|
||||
facts=list(metadata.get("facts") or []),
|
||||
context=dict(metadata.get("context") or {}),
|
||||
path=path,
|
||||
)
|
||||
|
||||
|
||||
def list_bgp_brief_records(limit: int = 50) -> list[BGPBriefRecordSummary]:
|
||||
storage_dir = _ensure_storage_dir()
|
||||
records: list[_StoredBrief] = []
|
||||
|
||||
for path in storage_dir.glob("*.md"):
|
||||
parsed = _parse_brief_file(path)
|
||||
if parsed is not None:
|
||||
records.append(parsed)
|
||||
|
||||
records.sort(key=lambda item: item.generated_at, reverse=True)
|
||||
|
||||
return [
|
||||
BGPBriefRecordSummary(
|
||||
id=item.id,
|
||||
title=item.title,
|
||||
provider=item.provider,
|
||||
model=item.model,
|
||||
request_id=item.request_id,
|
||||
generated_at=item.generated_at,
|
||||
)
|
||||
for item in records[: max(limit, 1)]
|
||||
]
|
||||
|
||||
|
||||
def get_bgp_brief_record(brief_id: str) -> BGPBriefRecordResponse | None:
|
||||
path = _ensure_storage_dir() / f"{brief_id}.md"
|
||||
parsed = _parse_brief_file(path)
|
||||
if parsed is None:
|
||||
return None
|
||||
|
||||
return BGPBriefRecordResponse(
|
||||
id=parsed.id,
|
||||
title=parsed.title,
|
||||
provider=parsed.provider,
|
||||
model=parsed.model,
|
||||
request_id=parsed.request_id,
|
||||
generated_at=parsed.generated_at,
|
||||
content_markdown=parsed.content_markdown,
|
||||
facts=parsed.facts,
|
||||
context=parsed.context,
|
||||
)
|
||||
|
||||
|
||||
def get_latest_bgp_brief_record() -> BGPBriefRecordResponse | None:
|
||||
summaries = list_bgp_brief_records(limit=1)
|
||||
if not summaries:
|
||||
return None
|
||||
return get_bgp_brief_record(summaries[0].id)
|
||||
|
||||
|
||||
def save_bgp_brief_record(
|
||||
analysis: SituationalAnalysisResponse,
|
||||
*,
|
||||
request_id: str | None,
|
||||
facts: list[str] | None = None,
|
||||
context: dict[str, Any] | None = None,
|
||||
generated_at: datetime | None = None,
|
||||
) -> BGPBriefRecordResponse:
|
||||
created_at = generated_at or datetime.now(UTC)
|
||||
brief_id = f"{created_at.strftime('%Y%m%dT%H%M%SZ')}-{uuid4().hex[:8]}"
|
||||
path = _ensure_storage_dir() / f"{brief_id}.md"
|
||||
|
||||
metadata = {
|
||||
"id": brief_id,
|
||||
"title": _BRIEF_TITLE,
|
||||
"provider": analysis.provider,
|
||||
"model": analysis.model,
|
||||
"request_id": request_id,
|
||||
"generated_at": created_at.isoformat(),
|
||||
"facts": facts or [],
|
||||
"context": context or {},
|
||||
}
|
||||
|
||||
markdown_text = f"{_build_metadata_line(metadata)}\n\n{analysis.content.rstrip()}\n"
|
||||
path.write_text(markdown_text, encoding="utf-8")
|
||||
|
||||
return BGPBriefRecordResponse(
|
||||
id=brief_id,
|
||||
title=_BRIEF_TITLE,
|
||||
provider=analysis.provider,
|
||||
model=analysis.model,
|
||||
request_id=request_id,
|
||||
generated_at=created_at.isoformat(),
|
||||
content_markdown=analysis.content,
|
||||
facts=facts or [],
|
||||
context=context or {},
|
||||
)
|
||||
@@ -6,7 +6,7 @@ from collections import defaultdict
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy import case, distinct, func, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.time import to_iso8601_utc
|
||||
@@ -14,6 +14,16 @@ from app.models.bgp_observation import BGPObservation
|
||||
from app.services.collectors.bgp_common import RIPE_RIS_COLLECTOR_COORDS
|
||||
|
||||
|
||||
def _collector_base_filters(source_filter: tuple[str, ...] | None) -> list[Any]:
|
||||
filters: list[Any] = [
|
||||
BGPObservation.collector.isnot(None),
|
||||
func.length(func.btrim(BGPObservation.collector)) > 0,
|
||||
]
|
||||
if source_filter:
|
||||
filters.append(BGPObservation.source.in_(source_filter))
|
||||
return filters
|
||||
|
||||
|
||||
async def build_bgp_collector_coverage(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
@@ -24,88 +34,148 @@ async def build_bgp_collector_coverage(
|
||||
recent_24h_threshold = now - timedelta(hours=24)
|
||||
recent_7d_threshold = now - timedelta(days=7)
|
||||
|
||||
stmt = select(BGPObservation).order_by(BGPObservation.observed_at.desc(), BGPObservation.id.desc())
|
||||
if source_filter:
|
||||
stmt = stmt.where(BGPObservation.source.in_(source_filter))
|
||||
filters = _collector_base_filters(source_filter)
|
||||
country_expr = func.nullif(BGPObservation.collector_geo["country"].as_string(), "")
|
||||
city_expr = func.nullif(BGPObservation.collector_geo["city"].as_string(), "")
|
||||
|
||||
result = await db.execute(stmt)
|
||||
records = list(result.scalars().all())
|
||||
aggregate_stmt = (
|
||||
select(
|
||||
BGPObservation.collector.label("collector"),
|
||||
func.count(BGPObservation.id).label("observation_count"),
|
||||
func.count(distinct(BGPObservation.prefix)).label("prefix_count"),
|
||||
func.count(distinct(BGPObservation.origin_asn)).label("origin_asn_count"),
|
||||
func.count(distinct(BGPObservation.peer_asn)).label("peer_asn_count"),
|
||||
func.sum(case((BGPObservation.observed_at >= recent_15m_threshold, 1), else_=0)).label("recent_15m_observation_count"),
|
||||
func.sum(case((BGPObservation.observed_at >= recent_24h_threshold, 1), else_=0)).label("recent_24h_observation_count"),
|
||||
func.sum(case((BGPObservation.observed_at >= recent_7d_threshold, 1), else_=0)).label("recent_7d_observation_count"),
|
||||
func.count(distinct(case((BGPObservation.observed_at >= recent_15m_threshold, BGPObservation.prefix), else_=None))).label("recent_15m_prefix_count"),
|
||||
func.count(distinct(case((BGPObservation.observed_at >= recent_24h_threshold, BGPObservation.prefix), else_=None))).label("recent_24h_prefix_count"),
|
||||
func.count(distinct(case((BGPObservation.observed_at >= recent_7d_threshold, BGPObservation.prefix), else_=None))).label("recent_7d_prefix_count"),
|
||||
func.max(BGPObservation.observed_at).label("latest_observed_at"),
|
||||
)
|
||||
.where(*filters)
|
||||
.group_by(BGPObservation.collector)
|
||||
)
|
||||
aggregate_rows = (await db.execute(aggregate_stmt)).all()
|
||||
|
||||
latest_subquery = (
|
||||
select(
|
||||
BGPObservation.collector.label("collector"),
|
||||
BGPObservation.event_type.label("latest_event_type"),
|
||||
country_expr.label("country"),
|
||||
city_expr.label("city"),
|
||||
func.row_number()
|
||||
.over(
|
||||
partition_by=BGPObservation.collector,
|
||||
order_by=(BGPObservation.observed_at.desc(), BGPObservation.id.desc()),
|
||||
)
|
||||
.label("rn"),
|
||||
)
|
||||
.where(*filters)
|
||||
.subquery()
|
||||
)
|
||||
latest_rows = (
|
||||
await db.execute(
|
||||
select(
|
||||
latest_subquery.c.collector,
|
||||
latest_subquery.c.latest_event_type,
|
||||
latest_subquery.c.country,
|
||||
latest_subquery.c.city,
|
||||
).where(latest_subquery.c.rn == 1)
|
||||
)
|
||||
).all()
|
||||
|
||||
event_counts_subquery = (
|
||||
select(
|
||||
BGPObservation.collector.label("collector"),
|
||||
BGPObservation.event_type.label("event_type"),
|
||||
func.count(BGPObservation.id).label("count"),
|
||||
func.row_number()
|
||||
.over(
|
||||
partition_by=BGPObservation.collector,
|
||||
order_by=(func.count(BGPObservation.id).desc(), BGPObservation.event_type.asc()),
|
||||
)
|
||||
.label("rn"),
|
||||
)
|
||||
.where(*filters)
|
||||
.group_by(BGPObservation.collector, BGPObservation.event_type)
|
||||
.subquery()
|
||||
)
|
||||
top_event_rows = (
|
||||
await db.execute(
|
||||
select(
|
||||
event_counts_subquery.c.collector,
|
||||
event_counts_subquery.c.event_type,
|
||||
event_counts_subquery.c.count,
|
||||
).where(event_counts_subquery.c.rn <= 3)
|
||||
)
|
||||
).all()
|
||||
|
||||
scope_rows = (
|
||||
await db.execute(
|
||||
select(
|
||||
BGPObservation.collector.label("collector"),
|
||||
country_expr.label("country"),
|
||||
city_expr.label("city"),
|
||||
)
|
||||
.where(*filters)
|
||||
.distinct()
|
||||
)
|
||||
).all()
|
||||
|
||||
latest_by_collector = {
|
||||
row.collector: {
|
||||
"latest_event_type": row.latest_event_type,
|
||||
"country": row.country,
|
||||
"city": row.city,
|
||||
}
|
||||
for row in latest_rows
|
||||
}
|
||||
|
||||
scope_by_collector: dict[str, dict[str, set[str]]] = defaultdict(lambda: {"countries": set(), "cities": set()})
|
||||
for row in scope_rows:
|
||||
if row.country:
|
||||
scope_by_collector[row.collector]["countries"].add(row.country)
|
||||
if row.city:
|
||||
scope_by_collector[row.collector]["cities"].add(row.city)
|
||||
|
||||
top_events_by_collector: dict[str, list[dict[str, Any]]] = defaultdict(list)
|
||||
for row in top_event_rows:
|
||||
top_events_by_collector[row.collector].append(
|
||||
{"event_type": row.event_type, "count": row.count}
|
||||
)
|
||||
|
||||
by_collector: dict[str, dict[str, Any]] = {}
|
||||
for record in records:
|
||||
collector = str(record.collector or "").strip()
|
||||
if not collector:
|
||||
continue
|
||||
for row in aggregate_rows:
|
||||
collector = row.collector
|
||||
latest = latest_by_collector.get(collector, {})
|
||||
fallback_location = RIPE_RIS_COLLECTOR_COORDS.get(collector, {})
|
||||
scope = scope_by_collector.get(collector, {"countries": set(), "cities": set()})
|
||||
|
||||
coverage = by_collector.get(collector)
|
||||
if coverage is None:
|
||||
location = record.collector_geo or RIPE_RIS_COLLECTOR_COORDS.get(collector, {})
|
||||
coverage = {
|
||||
"collector": collector,
|
||||
"city": location.get("city"),
|
||||
"country": location.get("country"),
|
||||
"latitude": location.get("latitude"),
|
||||
"longitude": location.get("longitude"),
|
||||
"observation_count": 0,
|
||||
"prefixes": set(),
|
||||
"origin_asns": set(),
|
||||
"peer_asns": set(),
|
||||
"event_types": defaultdict(int),
|
||||
"countries": set(),
|
||||
"cities": set(),
|
||||
"recent_15m_observation_count": 0,
|
||||
"recent_24h_observation_count": 0,
|
||||
"recent_7d_observation_count": 0,
|
||||
"recent_15m_prefixes": set(),
|
||||
"recent_24h_prefixes": set(),
|
||||
"recent_7d_prefixes": set(),
|
||||
"latest_observed_at": None,
|
||||
"latest_event_type": None,
|
||||
}
|
||||
by_collector[collector] = coverage
|
||||
|
||||
coverage["observation_count"] += 1
|
||||
if record.prefix:
|
||||
coverage["prefixes"].add(record.prefix)
|
||||
if record.origin_asn is not None:
|
||||
coverage["origin_asns"].add(record.origin_asn)
|
||||
if record.peer_asn is not None:
|
||||
coverage["peer_asns"].add(record.peer_asn)
|
||||
if record.event_type:
|
||||
coverage["event_types"][record.event_type] += 1
|
||||
|
||||
observed_at = record.observed_at
|
||||
if observed_at is not None:
|
||||
aware_observed_at = (
|
||||
observed_at.astimezone(UTC)
|
||||
if observed_at.tzinfo
|
||||
else observed_at.replace(tzinfo=UTC)
|
||||
)
|
||||
if aware_observed_at >= recent_15m_threshold:
|
||||
coverage["recent_15m_observation_count"] += 1
|
||||
if record.prefix:
|
||||
coverage["recent_15m_prefixes"].add(record.prefix)
|
||||
if aware_observed_at >= recent_24h_threshold:
|
||||
coverage["recent_24h_observation_count"] += 1
|
||||
if record.prefix:
|
||||
coverage["recent_24h_prefixes"].add(record.prefix)
|
||||
if aware_observed_at >= recent_7d_threshold:
|
||||
coverage["recent_7d_observation_count"] += 1
|
||||
if record.prefix:
|
||||
coverage["recent_7d_prefixes"].add(record.prefix)
|
||||
|
||||
geo = record.collector_geo or {}
|
||||
if geo.get("country"):
|
||||
coverage["countries"].add(geo["country"])
|
||||
if geo.get("city"):
|
||||
coverage["cities"].add(geo["city"])
|
||||
|
||||
current_latest = coverage["latest_observed_at"]
|
||||
if current_latest is None or (
|
||||
record.observed_at is not None and record.observed_at > current_latest
|
||||
):
|
||||
coverage["latest_observed_at"] = record.observed_at
|
||||
coverage["latest_event_type"] = record.event_type
|
||||
by_collector[collector] = {
|
||||
"collector": collector,
|
||||
"city": latest.get("city") or fallback_location.get("city"),
|
||||
"country": latest.get("country") or fallback_location.get("country"),
|
||||
"latitude": fallback_location.get("latitude"),
|
||||
"longitude": fallback_location.get("longitude"),
|
||||
"observation_count": row.observation_count or 0,
|
||||
"prefix_count": row.prefix_count or 0,
|
||||
"origin_asn_count": row.origin_asn_count or 0,
|
||||
"peer_asn_count": row.peer_asn_count or 0,
|
||||
"recent_15m_observation_count": row.recent_15m_observation_count or 0,
|
||||
"recent_24h_observation_count": row.recent_24h_observation_count or 0,
|
||||
"recent_7d_observation_count": row.recent_7d_observation_count or 0,
|
||||
"recent_15m_prefix_count": row.recent_15m_prefix_count or 0,
|
||||
"recent_24h_prefix_count": row.recent_24h_prefix_count or 0,
|
||||
"recent_7d_prefix_count": row.recent_7d_prefix_count or 0,
|
||||
"top_event_types": top_events_by_collector.get(collector, []),
|
||||
"latest_observed_at": to_iso8601_utc(row.latest_observed_at),
|
||||
"latest_event_type": latest.get("latest_event_type"),
|
||||
"baseline_scope": {
|
||||
"countries": sorted(scope["countries"]),
|
||||
"cities": sorted(scope["cities"]),
|
||||
},
|
||||
}
|
||||
|
||||
for collector, location in RIPE_RIS_COLLECTOR_COORDS.items():
|
||||
if collector in by_collector:
|
||||
@@ -117,57 +187,22 @@ async def build_bgp_collector_coverage(
|
||||
"latitude": location.get("latitude"),
|
||||
"longitude": location.get("longitude"),
|
||||
"observation_count": 0,
|
||||
"prefixes": set(),
|
||||
"origin_asns": set(),
|
||||
"peer_asns": set(),
|
||||
"event_types": defaultdict(int),
|
||||
"countries": {location.get("country")} if location.get("country") else set(),
|
||||
"cities": {location.get("city")} if location.get("city") else set(),
|
||||
"prefix_count": 0,
|
||||
"origin_asn_count": 0,
|
||||
"peer_asn_count": 0,
|
||||
"recent_15m_observation_count": 0,
|
||||
"recent_24h_observation_count": 0,
|
||||
"recent_7d_observation_count": 0,
|
||||
"recent_15m_prefixes": set(),
|
||||
"recent_24h_prefixes": set(),
|
||||
"recent_7d_prefixes": set(),
|
||||
"recent_15m_prefix_count": 0,
|
||||
"recent_24h_prefix_count": 0,
|
||||
"recent_7d_prefix_count": 0,
|
||||
"top_event_types": [],
|
||||
"latest_observed_at": None,
|
||||
"latest_event_type": None,
|
||||
"baseline_scope": {
|
||||
"countries": [location["country"]] if location.get("country") else [],
|
||||
"cities": [location["city"]] if location.get("city") else [],
|
||||
},
|
||||
}
|
||||
|
||||
results: list[dict[str, Any]] = []
|
||||
for collector in sorted(by_collector.keys()):
|
||||
item = by_collector[collector]
|
||||
top_event_types = sorted(
|
||||
item["event_types"].items(),
|
||||
key=lambda pair: (-pair[1], pair[0]),
|
||||
)
|
||||
results.append(
|
||||
{
|
||||
"collector": item["collector"],
|
||||
"city": item["city"],
|
||||
"country": item["country"],
|
||||
"latitude": item["latitude"],
|
||||
"longitude": item["longitude"],
|
||||
"observation_count": item["observation_count"],
|
||||
"prefix_count": len(item["prefixes"]),
|
||||
"origin_asn_count": len(item["origin_asns"]),
|
||||
"peer_asn_count": len(item["peer_asns"]),
|
||||
"recent_15m_observation_count": item["recent_15m_observation_count"],
|
||||
"recent_24h_observation_count": item["recent_24h_observation_count"],
|
||||
"recent_7d_observation_count": item["recent_7d_observation_count"],
|
||||
"recent_15m_prefix_count": len(item["recent_15m_prefixes"]),
|
||||
"recent_24h_prefix_count": len(item["recent_24h_prefixes"]),
|
||||
"recent_7d_prefix_count": len(item["recent_7d_prefixes"]),
|
||||
"top_event_types": [
|
||||
{"event_type": event_type, "count": count}
|
||||
for event_type, count in top_event_types[:3]
|
||||
],
|
||||
"latest_observed_at": to_iso8601_utc(item["latest_observed_at"]),
|
||||
"latest_event_type": item["latest_event_type"],
|
||||
"baseline_scope": {
|
||||
"countries": sorted(country for country in item["countries"] if country),
|
||||
"cities": sorted(city for city in item["cities"] if city),
|
||||
},
|
||||
}
|
||||
)
|
||||
|
||||
return results
|
||||
return [by_collector[collector] for collector in sorted(by_collector.keys())]
|
||||
@@ -7,7 +7,7 @@ from collections import defaultdict
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import select, text
|
||||
from sqlalchemy import Integer, cast, select, text
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.countries import get_country_centroid, normalize_country
|
||||
@@ -231,6 +231,13 @@ async def _lookup_prefix_geography(
|
||||
return results
|
||||
|
||||
|
||||
async def lookup_prefix_geography(
|
||||
db: AsyncSession,
|
||||
prefix_values: list[str],
|
||||
) -> dict[str, dict[str, Any]]:
|
||||
return await _lookup_prefix_geography(db, prefix_values)
|
||||
|
||||
|
||||
async def enrich_bgp_events_for_batch(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
@@ -261,29 +268,40 @@ async def enrich_bgp_events_for_batch(
|
||||
historical_prefix_baseline: dict[str, dict[str, Any]] = {}
|
||||
if prefix_values:
|
||||
previous_result = await db.execute(
|
||||
select(BGPObservation).where(
|
||||
select(
|
||||
BGPObservation.prefix,
|
||||
BGPObservation.origin_asn,
|
||||
BGPObservation.collector,
|
||||
BGPObservation.collector_geo,
|
||||
).where(
|
||||
BGPObservation.source == source,
|
||||
BGPObservation.prefix.in_(prefix_values),
|
||||
)
|
||||
)
|
||||
by_prefix: defaultdict[str, list[BGPObservation]] = defaultdict(list)
|
||||
for observation in previous_result.scalars().all():
|
||||
if observation.prefix:
|
||||
by_prefix[observation.prefix].append(observation)
|
||||
by_prefix: defaultdict[str, list[dict[str, Any]]] = defaultdict(list)
|
||||
for prefix, origin_asn, collector, collector_geo in previous_result.all():
|
||||
if prefix:
|
||||
by_prefix[str(prefix)].append(
|
||||
{
|
||||
"origin_asn": origin_asn,
|
||||
"collector": collector,
|
||||
"collector_geo": collector_geo or {},
|
||||
}
|
||||
)
|
||||
|
||||
for prefix, observations in by_prefix.items():
|
||||
unique_origins = sorted(
|
||||
{
|
||||
observation.origin_asn
|
||||
observation["origin_asn"]
|
||||
for observation in observations
|
||||
if observation.origin_asn is not None
|
||||
if observation["origin_asn"] is not None
|
||||
}
|
||||
)
|
||||
unique_collectors = sorted(
|
||||
{
|
||||
observation.collector
|
||||
observation["collector"]
|
||||
for observation in observations
|
||||
if observation.collector
|
||||
if observation["collector"]
|
||||
}
|
||||
)
|
||||
historical_prefix_baseline[prefix] = {
|
||||
@@ -292,9 +310,9 @@ async def enrich_bgp_events_for_batch(
|
||||
"historical_observation_count": len(observations),
|
||||
"historical_regions": _compact_locations(
|
||||
[
|
||||
observation.collector_geo or {}
|
||||
observation["collector_geo"] or {}
|
||||
for observation in observations
|
||||
if observation.collector_geo
|
||||
if observation["collector_geo"]
|
||||
]
|
||||
),
|
||||
}
|
||||
@@ -303,7 +321,13 @@ async def enrich_bgp_events_for_batch(
|
||||
prefix_geographies = await _lookup_prefix_geography(db, prefix_values) if prefix_values else {}
|
||||
if origin_asns:
|
||||
peeringdb_result = await db.execute(
|
||||
select(CollectedData).where(CollectedData.source == "peeringdb_network")
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source == "peeringdb_network")
|
||||
.where(CollectedData.is_current.is_(True))
|
||||
.where(
|
||||
cast(CollectedData.extra_data["asn"].as_string(), Integer).in_(origin_asns),
|
||||
)
|
||||
.order_by(CollectedData.id.desc())
|
||||
)
|
||||
for record in peeringdb_result.scalars().all():
|
||||
metadata = record.extra_data or {}
|
||||
|
||||
@@ -48,14 +48,36 @@ def _collector_regions_from_anomaly(anomaly: BGPAnomaly) -> list[dict]:
|
||||
return collected
|
||||
|
||||
|
||||
def _dedupe_collected_records(records: list[CollectedData]) -> list[CollectedData]:
|
||||
latest_by_key: dict[str, CollectedData] = {}
|
||||
for record in records:
|
||||
dedupe_key = str(record.source_id or record.entity_key or record.name or record.id)
|
||||
existing = latest_by_key.get(dedupe_key)
|
||||
if existing is None or (record.id or 0) > (existing.id or 0):
|
||||
latest_by_key[dedupe_key] = record
|
||||
return list(latest_by_key.values())
|
||||
async def _load_current_infrastructure_records(
|
||||
db: AsyncSession,
|
||||
) -> tuple[list[CollectedData], list[CollectedData], list[CollectedData]]:
|
||||
result = await db.execute(
|
||||
select(CollectedData)
|
||||
.where(
|
||||
CollectedData.source.in_(
|
||||
(
|
||||
"arcgis_landing_points",
|
||||
"arcgis_cable_landing_relation",
|
||||
"arcgis_cables",
|
||||
)
|
||||
)
|
||||
)
|
||||
.where(CollectedData.is_current.is_(True))
|
||||
.order_by(CollectedData.source.asc(), CollectedData.id.desc())
|
||||
)
|
||||
grouped_records = {
|
||||
"arcgis_landing_points": [],
|
||||
"arcgis_cable_landing_relation": [],
|
||||
"arcgis_cables": [],
|
||||
}
|
||||
for record in result.scalars().all():
|
||||
grouped_records.setdefault(record.source, []).append(record)
|
||||
|
||||
return (
|
||||
grouped_records["arcgis_landing_points"],
|
||||
grouped_records["arcgis_cable_landing_relation"],
|
||||
grouped_records["arcgis_cables"],
|
||||
)
|
||||
|
||||
|
||||
async def infer_related_infrastructure(
|
||||
@@ -75,19 +97,9 @@ async def infer_related_infrastructure(
|
||||
if not valid_regions:
|
||||
return {"related_cables": [], "related_ixps": []}
|
||||
|
||||
landing_result = await db.execute(
|
||||
select(CollectedData).where(CollectedData.source == "arcgis_landing_points")
|
||||
landing_records, relation_records, cable_records = await _load_current_infrastructure_records(
|
||||
db,
|
||||
)
|
||||
relation_result = await db.execute(
|
||||
select(CollectedData).where(CollectedData.source == "arcgis_cable_landing_relation")
|
||||
)
|
||||
cable_result = await db.execute(
|
||||
select(CollectedData).where(CollectedData.source == "arcgis_cables")
|
||||
)
|
||||
|
||||
landing_records = _dedupe_collected_records(list(landing_result.scalars().all()))
|
||||
relation_records = _dedupe_collected_records(list(relation_result.scalars().all()))
|
||||
cable_records = _dedupe_collected_records(list(cable_result.scalars().all()))
|
||||
|
||||
city_to_cable_ids: dict[int, list[int]] = {}
|
||||
for relation in relation_records:
|
||||
|
||||
@@ -35,6 +35,7 @@ from app.services.collectors.bgpstream import BGPStreamBackfillCollector
|
||||
from app.services.collectors.iptoasn import IPtoASNPrefixGeoCollector
|
||||
from app.services.collectors.opengeofeed import OpenGeoFeedPrefixGeoCollector
|
||||
from app.services.collectors.nro_delegated import NRODelegatedPrefixGeoCollector
|
||||
from app.services.collectors.news_live_streams import NewsLiveStreamsCollector
|
||||
|
||||
collector_registry.register(TOP500Collector())
|
||||
collector_registry.register(EpochAIGPUCollector())
|
||||
@@ -61,3 +62,4 @@ collector_registry.register(BGPStreamBackfillCollector())
|
||||
collector_registry.register(IPtoASNPrefixGeoCollector())
|
||||
collector_registry.register(OpenGeoFeedPrefixGeoCollector())
|
||||
collector_registry.register(NRODelegatedPrefixGeoCollector())
|
||||
collector_registry.register(NewsLiveStreamsCollector())
|
||||
@@ -1,5 +1,6 @@
|
||||
"""Base collector class for all data sources"""
|
||||
|
||||
import asyncio
|
||||
from abc import ABC, abstractmethod
|
||||
from typing import Dict, List, Any, Optional
|
||||
from datetime import UTC, datetime
|
||||
@@ -166,6 +167,59 @@ class BaseCollector(ABC):
|
||||
await db.commit()
|
||||
return snapshot.id
|
||||
|
||||
async def _rollback_incomplete_run(
|
||||
self,
|
||||
db: AsyncSession,
|
||||
*,
|
||||
task_id: int,
|
||||
snapshot_id: Optional[int],
|
||||
reason: str,
|
||||
) -> None:
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.models.data_snapshot import DataSnapshot
|
||||
|
||||
await db.execute(CollectedData.__table__.delete().where(CollectedData.task_id == task_id))
|
||||
|
||||
parent_snapshot_id: Optional[int] = None
|
||||
if snapshot_id is not None:
|
||||
snapshot = await db.get(DataSnapshot, snapshot_id)
|
||||
if snapshot:
|
||||
parent_snapshot_id = snapshot.parent_snapshot_id
|
||||
snapshot.status = "cancelled"
|
||||
snapshot.is_current = False
|
||||
snapshot.completed_at = datetime.now(UTC)
|
||||
summary = dict(snapshot.summary or {})
|
||||
summary["rollback"] = True
|
||||
summary["rollback_reason"] = reason
|
||||
snapshot.summary = summary
|
||||
|
||||
await db.execute(
|
||||
text(
|
||||
"""
|
||||
UPDATE collected_data
|
||||
SET is_current = FALSE
|
||||
WHERE source = :source
|
||||
"""
|
||||
),
|
||||
{"source": self.name},
|
||||
)
|
||||
|
||||
if parent_snapshot_id is not None:
|
||||
parent_snapshot = await db.get(DataSnapshot, parent_snapshot_id)
|
||||
if parent_snapshot:
|
||||
parent_snapshot.is_current = True
|
||||
|
||||
await db.execute(
|
||||
text(
|
||||
"""
|
||||
UPDATE collected_data
|
||||
SET is_current = TRUE
|
||||
WHERE snapshot_id = :snapshot_id
|
||||
"""
|
||||
),
|
||||
{"snapshot_id": parent_snapshot_id},
|
||||
)
|
||||
|
||||
async def run(self, db: AsyncSession) -> Dict[str, Any]:
|
||||
"""Full pipeline: fetch -> transform -> save"""
|
||||
from app.services.collectors.registry import collector_registry
|
||||
@@ -227,7 +281,24 @@ class BaseCollector(ABC):
|
||||
"records_processed": records_count,
|
||||
"execution_time_seconds": (datetime.now(UTC) - start_time).total_seconds(),
|
||||
}
|
||||
except asyncio.CancelledError:
|
||||
await db.rollback()
|
||||
task.status = "cancelled"
|
||||
task.phase = "cancelled"
|
||||
task.error_message = "Collection cancelled by operator and rolled back"
|
||||
task.completed_at = datetime.now(UTC)
|
||||
if snapshot_id is not None:
|
||||
await self._rollback_incomplete_run(
|
||||
db,
|
||||
task_id=task_id,
|
||||
snapshot_id=snapshot_id,
|
||||
reason="cancelled_by_operator",
|
||||
)
|
||||
await db.commit()
|
||||
await self._publish_task_update(force=True)
|
||||
raise
|
||||
except Exception as e:
|
||||
await db.rollback()
|
||||
task.status = "failed"
|
||||
task.phase = "failed"
|
||||
task.error_message = str(e)
|
||||
@@ -276,20 +347,34 @@ class BaseCollector(ABC):
|
||||
updated_count = 0
|
||||
unchanged_count = 0
|
||||
seen_entity_keys: set[str] = set()
|
||||
previous_current_keys: set[str] = set()
|
||||
progress_commit_interval = 1000
|
||||
|
||||
previous_current_result = await db.execute(
|
||||
select(CollectedData.entity_key).where(
|
||||
select(CollectedData)
|
||||
.where(
|
||||
CollectedData.source == self.name,
|
||||
CollectedData.is_current == True,
|
||||
)
|
||||
.order_by(CollectedData.entity_key.asc(), CollectedData.collected_at.desc().nullslast(), CollectedData.id.desc())
|
||||
)
|
||||
previous_current_keys = {row[0] for row in previous_current_result.fetchall() if row[0]}
|
||||
previous_current_records = previous_current_result.scalars().all()
|
||||
previous_current_keys = {record.entity_key for record in previous_current_records if record.entity_key}
|
||||
previous_current_map: dict[str, CollectedData] = {}
|
||||
stale_previous_records: list[CollectedData] = []
|
||||
|
||||
for existing_record in previous_current_records:
|
||||
entity_key = existing_record.entity_key
|
||||
if not entity_key:
|
||||
continue
|
||||
if entity_key not in previous_current_map:
|
||||
previous_current_map[entity_key] = existing_record
|
||||
continue
|
||||
stale_previous_records.append(existing_record)
|
||||
|
||||
for stale_record in stale_previous_records:
|
||||
stale_record.is_current = False
|
||||
|
||||
for i, item in enumerate(data):
|
||||
print(
|
||||
f"DEBUG: Saving item {i}: name={item.get('name')}, metadata={item.get('metadata', 'NOT FOUND')}"
|
||||
)
|
||||
raw_metadata = item.get("metadata", {})
|
||||
extra_data = build_dynamic_metadata(
|
||||
raw_metadata,
|
||||
@@ -318,20 +403,9 @@ class BaseCollector(ABC):
|
||||
previous_record = None
|
||||
|
||||
if entity_key and entity_key not in seen_entity_keys:
|
||||
result = await db.execute(
|
||||
select(CollectedData)
|
||||
.where(
|
||||
CollectedData.source == self.name,
|
||||
CollectedData.entity_key == entity_key,
|
||||
CollectedData.is_current == True,
|
||||
)
|
||||
.order_by(CollectedData.collected_at.desc().nullslast(), CollectedData.id.desc())
|
||||
)
|
||||
previous_records = result.scalars().all()
|
||||
if previous_records:
|
||||
previous_record = previous_records[0]
|
||||
for old_record in previous_records:
|
||||
old_record.is_current = False
|
||||
previous_record = previous_current_map.get(entity_key)
|
||||
if previous_record is not None:
|
||||
previous_record.is_current = False
|
||||
|
||||
record = CollectedData(
|
||||
snapshot_id=snapshot_id,
|
||||
@@ -375,7 +449,7 @@ class BaseCollector(ABC):
|
||||
seen_entity_keys.add(entity_key)
|
||||
records_added += 1
|
||||
|
||||
if i % 100 == 0:
|
||||
if (i + 1) % progress_commit_interval == 0:
|
||||
await self.update_progress(i + 1, commit=True)
|
||||
|
||||
if snapshot_id is not None:
|
||||
|
||||
@@ -21,7 +21,7 @@ class CelesTrakTLECollector(BaseCollector):
|
||||
|
||||
@property
|
||||
def base_url(self) -> str:
|
||||
return "https://celestrak.org/NORAD/elements/gp.php"
|
||||
return self._resolved_url or ""
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
satellite_groups = [
|
||||
@@ -40,12 +40,15 @@ class CelesTrakTLECollector(BaseCollector):
|
||||
async with httpx.AsyncClient(timeout=120.0) as client:
|
||||
for group in satellite_groups:
|
||||
try:
|
||||
url = f"https://celestrak.org/NORAD/elements/gp.php?GROUP={group}&FORMAT=json"
|
||||
url = f"{self.base_url}?GROUP={group}&FORMAT=json"
|
||||
response = await client.get(url)
|
||||
|
||||
if response.status_code == 200:
|
||||
data = response.json()
|
||||
if isinstance(data, list):
|
||||
for item in data:
|
||||
if isinstance(item, dict):
|
||||
item["_celestrak_group"] = group
|
||||
all_satellites.extend(data)
|
||||
print(f"CelesTrak: Fetched {len(data)} satellites from group '{group}'")
|
||||
except Exception as e:
|
||||
@@ -78,6 +81,7 @@ class CelesTrakTLECollector(BaseCollector):
|
||||
"name": item.get("OBJECT_NAME", "Unknown"),
|
||||
"reference_date": item.get("EPOCH", ""),
|
||||
"metadata": {
|
||||
"constellation_group": item.get("_celestrak_group"),
|
||||
"norad_cat_id": item.get("NORAD_CAT_ID"),
|
||||
"international_designator": item.get("OBJECT_ID"),
|
||||
"epoch": item.get("EPOCH"),
|
||||
|
||||
@@ -39,6 +39,16 @@ class CloudflareRadarDeviceCollector(HTTPCollector):
|
||||
if CLOUDFLARE_API_TOKEN:
|
||||
self.headers["Authorization"] = f"Bearer {CLOUDFLARE_API_TOKEN}"
|
||||
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
return self._resolved_url or self.base_url
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(self.request_url, headers=self.headers)
|
||||
response.raise_for_status()
|
||||
return self.parse_response(response.json())
|
||||
|
||||
def parse_response(self, response: Dict[str, Any]) -> List[Dict[str, Any]]:
|
||||
"""Parse Cloudflare Radar device type response"""
|
||||
data = []
|
||||
@@ -87,6 +97,16 @@ class CloudflareRadarTrafficCollector(HTTPCollector):
|
||||
if CLOUDFLARE_API_TOKEN:
|
||||
self.headers["Authorization"] = f"Bearer {CLOUDFLARE_API_TOKEN}"
|
||||
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
return self._resolved_url or self.base_url
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(self.request_url, headers=self.headers)
|
||||
response.raise_for_status()
|
||||
return self.parse_response(response.json())
|
||||
|
||||
def parse_response(self, response: Dict[str, Any]) -> List[Dict[str, Any]]:
|
||||
"""Parse Cloudflare Radar traffic timeseries response"""
|
||||
data = []
|
||||
@@ -135,6 +155,16 @@ class CloudflareRadarTopASCollector(HTTPCollector):
|
||||
if CLOUDFLARE_API_TOKEN:
|
||||
self.headers["Authorization"] = f"Bearer {CLOUDFLARE_API_TOKEN}"
|
||||
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
return self._resolved_url or self.base_url
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(self.request_url, headers=self.headers)
|
||||
response.raise_for_status()
|
||||
return self.parse_response(response.json())
|
||||
|
||||
def parse_response(self, response: Dict[str, Any]) -> List[Dict[str, Any]]:
|
||||
"""Parse Cloudflare Radar top locations response"""
|
||||
data = []
|
||||
|
||||
@@ -23,7 +23,7 @@ class EpochAIGPUCollector(BaseCollector):
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
"""Fetch Epoch AI GPU clusters data from webpage"""
|
||||
url = "https://epoch.ai/data/gpu-clusters"
|
||||
url = self._resolved_url or ""
|
||||
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(url)
|
||||
|
||||
@@ -18,11 +18,9 @@ class FAOLandingPointCollector(BaseCollector):
|
||||
frequency_hours = 168
|
||||
data_type = "landing_point"
|
||||
|
||||
csv_url = "https://data.apps.fao.org/catalog/dataset/1b75ff21-92f2-4b96-9b7b-98e8aa65ad5d/resource/b6071077-d1d4-4e97-aa00-42e902847c87/download/landing-point-geo.csv"
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(self.csv_url)
|
||||
response = await client.get(self._resolved_url or "")
|
||||
response.raise_for_status()
|
||||
return self.parse_csv(response.text)
|
||||
|
||||
|
||||
@@ -21,6 +21,18 @@ class HuggingFaceModelCollector(HTTPCollector):
|
||||
data_type = "model"
|
||||
base_url = "https://huggingface.co/api/models"
|
||||
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
return self._resolved_url or self.base_url
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
from httpx import AsyncClient
|
||||
|
||||
async with AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(self.request_url, headers=self.headers)
|
||||
response.raise_for_status()
|
||||
return self.parse_response(response.json())
|
||||
|
||||
def parse_response(self, response: Dict[str, Any]) -> List[Dict[str, Any]]:
|
||||
"""Parse Hugging Face models API response"""
|
||||
data = []
|
||||
@@ -63,6 +75,18 @@ class HuggingFaceDatasetCollector(HTTPCollector):
|
||||
data_type = "dataset"
|
||||
base_url = "https://huggingface.co/api/datasets"
|
||||
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
return self._resolved_url or self.base_url
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
from httpx import AsyncClient
|
||||
|
||||
async with AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(self.request_url, headers=self.headers)
|
||||
response.raise_for_status()
|
||||
return self.parse_response(response.json())
|
||||
|
||||
def parse_response(self, response: Dict[str, Any]) -> List[Dict[str, Any]]:
|
||||
"""Parse Hugging Face datasets API response"""
|
||||
data = []
|
||||
@@ -104,6 +128,18 @@ class HuggingFaceSpacesCollector(HTTPCollector):
|
||||
data_type = "space"
|
||||
base_url = "https://huggingface.co/api/spaces"
|
||||
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
return self._resolved_url or self.base_url
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
from httpx import AsyncClient
|
||||
|
||||
async with AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(self.request_url, headers=self.headers)
|
||||
response.raise_for_status()
|
||||
return self.parse_response(response.json())
|
||||
|
||||
def parse_response(self, response: Dict[str, Any]) -> List[Dict[str, Any]]:
|
||||
"""Parse Hugging Face Spaces API response"""
|
||||
data = []
|
||||
|
||||
@@ -0,0 +1,570 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
from urllib.parse import urlparse
|
||||
|
||||
import httpx
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
from app.models.datasource_config import DataSourceConfig
|
||||
from app.services.collectors.base import BaseCollector
|
||||
|
||||
|
||||
class NewsLiveStreamsCollector(BaseCollector):
|
||||
"""Collect normalized news live-stream sources from a JSON endpoint."""
|
||||
|
||||
name = "news_live_streams"
|
||||
priority = "P2"
|
||||
module = "L4"
|
||||
frequency_hours = 12
|
||||
data_type = "news_live_stream"
|
||||
fail_on_empty = False
|
||||
|
||||
DEFAULT_TIMEOUT = 45.0
|
||||
DEFAULT_HEADERS = {
|
||||
"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
|
||||
"Accept": "application/json",
|
||||
}
|
||||
RESPONSE_CANDIDATE_KEYS = ("sources", "streams", "channels", "items", "results", "data")
|
||||
DEFAULT_ADAPTER = "iptv_org"
|
||||
DEFAULT_IPTV_ORG_STREAMS_URL = "https://iptv-org.github.io/api/streams.json"
|
||||
DEFAULT_IPTV_ORG_LOGOS_URL = "https://iptv-org.github.io/api/logos.json"
|
||||
DEFAULT_IPTV_ORG_NEWS_CATEGORIES = ("news", "business", "weather")
|
||||
DEFAULT_IPTV_ORG_EXCLUDE_CATEGORIES = ("music", "sports", "kids", "entertainment")
|
||||
DEFAULT_IPTV_ORG_MAX_SOURCES = 120
|
||||
|
||||
async def fetch(self) -> list[dict[str, Any]]:
|
||||
request_url = (self._resolved_url or "").strip()
|
||||
if not request_url:
|
||||
return []
|
||||
|
||||
datasource_config = await self._load_datasource_config()
|
||||
effective_config = self._get_effective_config(datasource_config)
|
||||
adapter = str(effective_config.get("adapter") or "").strip().lower()
|
||||
if adapter == "iptv_org":
|
||||
return await self._fetch_iptv_org(request_url, effective_config)
|
||||
|
||||
request_headers = self._build_request_headers(datasource_config)
|
||||
request_config = self._get_request_config(datasource_config)
|
||||
request_params = self._build_request_params(datasource_config)
|
||||
request_json = self._build_request_json_body(datasource_config)
|
||||
request_data = self._build_request_form_body(datasource_config)
|
||||
timeout = self._get_timeout(datasource_config)
|
||||
|
||||
async with httpx.AsyncClient(timeout=timeout, follow_redirects=True) as client:
|
||||
response = await client.request(
|
||||
request_config["method"],
|
||||
request_url,
|
||||
headers=request_headers,
|
||||
params=request_params or None,
|
||||
json=request_json,
|
||||
data=request_data,
|
||||
)
|
||||
response.raise_for_status()
|
||||
return self.parse_response(
|
||||
response.json(),
|
||||
response_path=request_config["response_path"],
|
||||
)
|
||||
|
||||
async def _load_datasource_config(self) -> DataSourceConfig | None:
|
||||
if not self._db_session:
|
||||
return None
|
||||
|
||||
result = await self._db_session.execute(
|
||||
select(DataSourceConfig)
|
||||
.where(DataSourceConfig.name == self.name)
|
||||
.where(DataSourceConfig.is_active.is_(True))
|
||||
.limit(1)
|
||||
)
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
def _get_effective_config(self, datasource_config: DataSourceConfig | None) -> dict[str, Any]:
|
||||
payload = dict(datasource_config.config or {}) if datasource_config else {}
|
||||
if payload:
|
||||
return payload
|
||||
|
||||
yaml_config = get_data_sources_config()
|
||||
return {
|
||||
"adapter": self.DEFAULT_ADAPTER,
|
||||
"streams_url": yaml_config.get_yaml_value("news_live_streams.streams_url")
|
||||
or self.DEFAULT_IPTV_ORG_STREAMS_URL,
|
||||
"logos_url": yaml_config.get_yaml_value("news_live_streams.logos_url")
|
||||
or self.DEFAULT_IPTV_ORG_LOGOS_URL,
|
||||
"news_categories": list(self.DEFAULT_IPTV_ORG_NEWS_CATEGORIES),
|
||||
"exclude_categories": list(self.DEFAULT_IPTV_ORG_EXCLUDE_CATEGORIES),
|
||||
"max_sources": self.DEFAULT_IPTV_ORG_MAX_SOURCES,
|
||||
}
|
||||
|
||||
def _get_request_config(self, datasource_config: DataSourceConfig | None) -> dict[str, Any]:
|
||||
payload = self._get_effective_config(datasource_config)
|
||||
raw_method = payload.get("method") or payload.get("request_method") or "GET"
|
||||
method = str(raw_method).strip().upper() or "GET"
|
||||
if method not in {"GET", "POST"}:
|
||||
method = "GET"
|
||||
|
||||
response_path = payload.get("response_path") or payload.get("payload_path") or payload.get("items_path")
|
||||
if isinstance(response_path, str):
|
||||
response_path = response_path.strip()
|
||||
else:
|
||||
response_path = None
|
||||
|
||||
return {
|
||||
"method": method,
|
||||
"response_path": response_path or None,
|
||||
}
|
||||
|
||||
def _get_timeout(self, datasource_config: DataSourceConfig | None) -> float:
|
||||
payload = self._get_effective_config(datasource_config)
|
||||
try:
|
||||
return float(payload.get("timeout", self.DEFAULT_TIMEOUT))
|
||||
except (TypeError, ValueError):
|
||||
return self.DEFAULT_TIMEOUT
|
||||
|
||||
def _build_request_headers(self, datasource_config: DataSourceConfig | None) -> dict[str, str]:
|
||||
headers = dict(self.DEFAULT_HEADERS)
|
||||
if datasource_config:
|
||||
headers.update(self._normalize_headers(datasource_config.headers))
|
||||
headers.update(self._build_auth_headers(datasource_config))
|
||||
return headers
|
||||
|
||||
def _build_request_params(self, datasource_config: DataSourceConfig | None) -> dict[str, Any]:
|
||||
params: dict[str, Any] = {}
|
||||
if not datasource_config:
|
||||
return params
|
||||
|
||||
payload = datasource_config.config or {}
|
||||
candidate = payload.get("params") or payload.get("query_params")
|
||||
if isinstance(candidate, dict):
|
||||
params.update(candidate)
|
||||
|
||||
if datasource_config.auth_type == "api_key":
|
||||
auth_config = datasource_config.auth_config or {}
|
||||
if str(auth_config.get("in") or auth_config.get("location") or "header").lower() == "query":
|
||||
api_key = auth_config.get("api_key")
|
||||
key_name = auth_config.get("key_name") or auth_config.get("param_name") or "api_key"
|
||||
if api_key and key_name:
|
||||
params[str(key_name)] = api_key
|
||||
|
||||
return params
|
||||
|
||||
def _build_request_json_body(self, datasource_config: DataSourceConfig | None) -> Any:
|
||||
if not datasource_config:
|
||||
return None
|
||||
|
||||
payload = datasource_config.config or {}
|
||||
body = payload.get("json_body")
|
||||
if body is None and str(payload.get("body_type") or "").lower() in {"json", ""}:
|
||||
candidate = payload.get("body")
|
||||
if isinstance(candidate, (dict, list)):
|
||||
body = candidate
|
||||
return body
|
||||
|
||||
def _build_request_form_body(self, datasource_config: DataSourceConfig | None) -> Any:
|
||||
if not datasource_config:
|
||||
return None
|
||||
|
||||
payload = datasource_config.config or {}
|
||||
form_body = payload.get("form_body")
|
||||
if form_body is not None:
|
||||
return form_body
|
||||
|
||||
if str(payload.get("body_type") or "").lower() == "form":
|
||||
candidate = payload.get("body")
|
||||
if isinstance(candidate, dict):
|
||||
return candidate
|
||||
return None
|
||||
|
||||
def _normalize_headers(self, headers: Any) -> dict[str, str]:
|
||||
if not isinstance(headers, dict):
|
||||
return {}
|
||||
normalized: dict[str, str] = {}
|
||||
for key, value in headers.items():
|
||||
header_name = str(key).strip()
|
||||
if not header_name or value is None:
|
||||
continue
|
||||
normalized[header_name] = str(value)
|
||||
return normalized
|
||||
|
||||
def _build_auth_headers(self, datasource_config: DataSourceConfig | None) -> dict[str, str]:
|
||||
if not datasource_config:
|
||||
return {}
|
||||
|
||||
auth_type = str(datasource_config.auth_type or "none").lower()
|
||||
auth_config = datasource_config.auth_config or {}
|
||||
if auth_type == "bearer" and auth_config.get("token"):
|
||||
return {"Authorization": f"Bearer {auth_config['token']}"}
|
||||
|
||||
if auth_type == "api_key" and auth_config.get("api_key"):
|
||||
location = str(auth_config.get("in") or auth_config.get("location") or "header").lower()
|
||||
if location == "query":
|
||||
return {}
|
||||
key_name = auth_config.get("key_name") or "X-API-Key"
|
||||
return {str(key_name): str(auth_config["api_key"])}
|
||||
|
||||
if auth_type == "basic":
|
||||
username = str(auth_config.get("username") or "")
|
||||
password = str(auth_config.get("password") or "")
|
||||
encoded = base64.b64encode(f"{username}:{password}".encode()).decode()
|
||||
return {"Authorization": f"Basic {encoded}"}
|
||||
|
||||
return {}
|
||||
|
||||
def _extract_candidates(self, response: Any, response_path: str | None) -> list[Any]:
|
||||
if response_path:
|
||||
extracted = self._extract_from_path(response, response_path)
|
||||
if isinstance(extracted, list):
|
||||
return extracted
|
||||
if isinstance(extracted, dict):
|
||||
for key in self.RESPONSE_CANDIDATE_KEYS:
|
||||
nested = extracted.get(key)
|
||||
if isinstance(nested, list):
|
||||
return nested
|
||||
return [extracted]
|
||||
|
||||
if isinstance(response, dict):
|
||||
for key in self.RESPONSE_CANDIDATE_KEYS:
|
||||
nested = response.get(key)
|
||||
if isinstance(nested, list):
|
||||
return nested
|
||||
return []
|
||||
|
||||
if isinstance(response, list):
|
||||
return response
|
||||
return []
|
||||
|
||||
def _extract_from_path(self, payload: Any, path: str) -> Any:
|
||||
current = payload
|
||||
for segment in (part.strip() for part in path.split(".") if part.strip()):
|
||||
if isinstance(current, dict):
|
||||
current = current.get(segment)
|
||||
continue
|
||||
if isinstance(current, list):
|
||||
try:
|
||||
current = current[int(segment)]
|
||||
except (TypeError, ValueError, IndexError):
|
||||
return None
|
||||
continue
|
||||
return None
|
||||
return current
|
||||
|
||||
def _infer_source_type(self, item: dict[str, Any]) -> str:
|
||||
explicit = str(item.get("source_type") or item.get("type") or "").strip().lower()
|
||||
if explicit in {"iframe", "hls", "video", "external", "youtube"}:
|
||||
return explicit
|
||||
|
||||
youtube_video_id = self._clean_text(
|
||||
item.get("youtube_video_id")
|
||||
or item.get("video_id")
|
||||
or item.get("youtubeVideoId")
|
||||
)
|
||||
youtube_channel = self._clean_text(item.get("youtube_channel") or item.get("channel_handle"))
|
||||
embed_url = self._clean_url(item.get("embed_url") or item.get("embed") or item.get("page_url"))
|
||||
stream_url = self._clean_url(item.get("stream_url") or item.get("stream") or item.get("playback_url") or item.get("hls_url"))
|
||||
homepage_url = self._clean_url(item.get("homepage_url") or item.get("source_url") or item.get("website"))
|
||||
|
||||
if youtube_video_id or youtube_channel:
|
||||
return "youtube"
|
||||
if stream_url.endswith(".m3u8"):
|
||||
return "hls"
|
||||
if stream_url:
|
||||
return "video"
|
||||
if embed_url:
|
||||
parsed = urlparse(embed_url)
|
||||
if "youtube.com" in (parsed.netloc or "") or "youtu.be" in (parsed.netloc or ""):
|
||||
return "youtube"
|
||||
return "iframe"
|
||||
if homepage_url:
|
||||
return "external"
|
||||
return "iframe"
|
||||
|
||||
def _parse_enabled(self, item: dict[str, Any]) -> bool:
|
||||
if "is_enabled" in item:
|
||||
return self._to_bool(item.get("is_enabled"), default=True)
|
||||
if "enabled" in item:
|
||||
return self._to_bool(item.get("enabled"), default=True)
|
||||
if "active" in item:
|
||||
return self._to_bool(item.get("active"), default=True)
|
||||
if "status" in item:
|
||||
status = str(item.get("status") or "").strip().lower()
|
||||
if status in {"disabled", "inactive", "offline"}:
|
||||
return False
|
||||
if status in {"enabled", "active", "online", "live"}:
|
||||
return True
|
||||
return True
|
||||
|
||||
def _to_bool(self, value: Any, *, default: bool) -> bool:
|
||||
if isinstance(value, bool):
|
||||
return value
|
||||
if value in (None, ""):
|
||||
return default
|
||||
if isinstance(value, str):
|
||||
lowered = value.strip().lower()
|
||||
if lowered in {"1", "true", "yes", "on", "enabled", "active", "online", "live"}:
|
||||
return True
|
||||
if lowered in {"0", "false", "no", "off", "disabled", "inactive", "offline"}:
|
||||
return False
|
||||
return bool(value)
|
||||
|
||||
def _clean_text(self, value: Any) -> str:
|
||||
if value is None:
|
||||
return ""
|
||||
return str(value).strip()
|
||||
|
||||
def _clean_url(self, value: Any) -> str:
|
||||
text = self._clean_text(value)
|
||||
if not text:
|
||||
return ""
|
||||
parsed = urlparse(text)
|
||||
if parsed.scheme and parsed.scheme not in {"http", "https"}:
|
||||
return ""
|
||||
if parsed.scheme and not parsed.netloc:
|
||||
return ""
|
||||
return text
|
||||
|
||||
async def _fetch_iptv_org(self, channels_url: str, collector_config: dict[str, Any]) -> list[dict[str, Any]]:
|
||||
streams_url = self._clean_url(collector_config.get("streams_url")) or self.DEFAULT_IPTV_ORG_STREAMS_URL
|
||||
logos_url = self._clean_url(collector_config.get("logos_url")) or self.DEFAULT_IPTV_ORG_LOGOS_URL
|
||||
news_categories = {
|
||||
self._clean_text(value).lower()
|
||||
for value in (collector_config.get("news_categories") or self.DEFAULT_IPTV_ORG_NEWS_CATEGORIES)
|
||||
if self._clean_text(value)
|
||||
}
|
||||
exclude_categories = {
|
||||
self._clean_text(value).lower()
|
||||
for value in (collector_config.get("exclude_categories") or self.DEFAULT_IPTV_ORG_EXCLUDE_CATEGORIES)
|
||||
if self._clean_text(value)
|
||||
}
|
||||
try:
|
||||
max_sources = int(collector_config.get("max_sources", self.DEFAULT_IPTV_ORG_MAX_SOURCES))
|
||||
except (TypeError, ValueError):
|
||||
max_sources = self.DEFAULT_IPTV_ORG_MAX_SOURCES
|
||||
|
||||
timeout = self.DEFAULT_TIMEOUT
|
||||
try:
|
||||
timeout = float(collector_config.get("timeout", self.DEFAULT_TIMEOUT))
|
||||
except (TypeError, ValueError):
|
||||
timeout = self.DEFAULT_TIMEOUT
|
||||
|
||||
async with httpx.AsyncClient(timeout=timeout, follow_redirects=True) as client:
|
||||
channels_payload, streams_payload, logos_payload = await self._gather_iptv_org_payloads(
|
||||
client,
|
||||
channels_url,
|
||||
streams_url,
|
||||
logos_url,
|
||||
)
|
||||
|
||||
channels = channels_payload if isinstance(channels_payload, list) else []
|
||||
streams = streams_payload if isinstance(streams_payload, list) else []
|
||||
logos = logos_payload if isinstance(logos_payload, list) else []
|
||||
|
||||
logo_by_channel = {
|
||||
self._clean_text(item.get("channel")): self._clean_url(item.get("url"))
|
||||
for item in logos
|
||||
if isinstance(item, dict) and self._clean_text(item.get("channel")) and self._clean_url(item.get("url"))
|
||||
}
|
||||
|
||||
streams_by_channel: dict[str, list[dict[str, Any]]] = {}
|
||||
for stream in streams:
|
||||
if not isinstance(stream, dict):
|
||||
continue
|
||||
channel_id = self._clean_text(stream.get("channel"))
|
||||
if not channel_id:
|
||||
continue
|
||||
streams_by_channel.setdefault(channel_id, []).append(stream)
|
||||
|
||||
normalized: list[dict[str, Any]] = []
|
||||
for channel in channels:
|
||||
if not isinstance(channel, dict):
|
||||
continue
|
||||
|
||||
categories = [
|
||||
self._clean_text(value).lower()
|
||||
for value in (channel.get("categories") or [])
|
||||
if self._clean_text(value)
|
||||
]
|
||||
if news_categories and not any(category in news_categories for category in categories):
|
||||
continue
|
||||
if exclude_categories and any(category in exclude_categories for category in categories):
|
||||
continue
|
||||
if channel.get("is_nsfw") is True:
|
||||
continue
|
||||
if channel.get("closed"):
|
||||
continue
|
||||
|
||||
channel_id = self._clean_text(channel.get("id"))
|
||||
if not channel_id:
|
||||
continue
|
||||
|
||||
stream = self._pick_iptv_org_stream(streams_by_channel.get(channel_id) or [])
|
||||
if not stream:
|
||||
continue
|
||||
|
||||
stream_url = self._clean_url(stream.get("url"))
|
||||
if not stream_url:
|
||||
continue
|
||||
|
||||
name = self._clean_text(channel.get("name")) or channel_id
|
||||
notes_parts = [
|
||||
f"Imported from IPTV-org catalog ({channel_id})",
|
||||
f"Categories: {', '.join(categories)}" if categories else "",
|
||||
f"Quality: {self._clean_text(stream.get('quality'))}" if self._clean_text(stream.get("quality")) else "",
|
||||
]
|
||||
metadata = {
|
||||
"provider": self._clean_text(channel.get("network")) or "IPTV-org",
|
||||
"region": self._clean_text(channel.get("country")) or "Global",
|
||||
"language": "und",
|
||||
"source_type": "hls" if stream_url.endswith(".m3u8") else "video",
|
||||
"embed_url": "",
|
||||
"stream_url": stream_url,
|
||||
"homepage_url": self._clean_url(channel.get("website")),
|
||||
"poster_url": logo_by_channel.get(channel_id, ""),
|
||||
"youtube_video_id": "",
|
||||
"youtube_channel": "",
|
||||
"sort_order": 400 + len(normalized),
|
||||
"notes": "; ".join(part for part in notes_parts if part),
|
||||
"is_enabled": True,
|
||||
"collector_adapter": "iptv_org",
|
||||
"channel_id": channel_id,
|
||||
"categories": categories,
|
||||
"quality": self._clean_text(stream.get("quality")),
|
||||
"stream_label": self._clean_text(stream.get("label") or stream.get("title")),
|
||||
"stream_referrer": self._clean_text(stream.get("referrer")),
|
||||
"stream_user_agent": self._clean_text(stream.get("user_agent")),
|
||||
}
|
||||
|
||||
normalized.append(
|
||||
{
|
||||
"source_id": channel_id,
|
||||
"name": name,
|
||||
"description": metadata["notes"],
|
||||
"metadata": metadata,
|
||||
"reference_date": datetime.now(UTC).isoformat(),
|
||||
}
|
||||
)
|
||||
if len(normalized) >= max_sources:
|
||||
break
|
||||
|
||||
return normalized
|
||||
|
||||
async def _gather_iptv_org_payloads(
|
||||
self,
|
||||
client: httpx.AsyncClient,
|
||||
channels_url: str,
|
||||
streams_url: str,
|
||||
logos_url: str,
|
||||
) -> tuple[Any, Any, Any]:
|
||||
headers = dict(self.DEFAULT_HEADERS)
|
||||
channels_payload, streams_payload, logos_payload = await asyncio.gather(
|
||||
client.get(channels_url, headers=headers),
|
||||
client.get(streams_url, headers=headers),
|
||||
client.get(logos_url, headers=headers),
|
||||
)
|
||||
channels_payload.raise_for_status()
|
||||
streams_payload.raise_for_status()
|
||||
logos_payload.raise_for_status()
|
||||
return channels_payload.json(), streams_payload.json(), logos_payload.json()
|
||||
|
||||
def _pick_iptv_org_stream(self, streams: list[dict[str, Any]]) -> dict[str, Any] | None:
|
||||
if not streams:
|
||||
return None
|
||||
|
||||
def score(stream: dict[str, Any]) -> tuple[int, int]:
|
||||
url = self._clean_url(stream.get("url"))
|
||||
quality = self._clean_text(stream.get("quality")).lower()
|
||||
quality_score = 0
|
||||
if quality.endswith("p"):
|
||||
try:
|
||||
quality_score = int(quality[:-1])
|
||||
except ValueError:
|
||||
quality_score = 0
|
||||
stream_score = 1000 if url.endswith(".m3u8") else 0
|
||||
return stream_score, quality_score
|
||||
|
||||
sorted_streams = sorted(streams, key=score, reverse=True)
|
||||
return sorted_streams[0]
|
||||
|
||||
def parse_response(self, response: Any, *, response_path: str | None = None) -> list[dict[str, Any]]:
|
||||
candidates = self._extract_candidates(response, response_path)
|
||||
|
||||
normalized: list[dict[str, Any]] = []
|
||||
for index, item in enumerate(candidates):
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
|
||||
stream_id = (
|
||||
item.get("id")
|
||||
or item.get("source_id")
|
||||
or item.get("slug")
|
||||
or item.get("channel_id")
|
||||
or item.get("code")
|
||||
or f"news-live-{index + 1}"
|
||||
)
|
||||
name = self._clean_text(
|
||||
item.get("name")
|
||||
or item.get("title")
|
||||
or item.get("channel")
|
||||
or item.get("display_name")
|
||||
or f"News Live {index + 1}"
|
||||
)
|
||||
if not name:
|
||||
continue
|
||||
|
||||
source_type = self._infer_source_type(item)
|
||||
stream_url = self._clean_url(
|
||||
item.get("stream_url")
|
||||
or item.get("stream")
|
||||
or item.get("playback_url")
|
||||
or item.get("hls_url")
|
||||
or item.get("m3u8_url")
|
||||
)
|
||||
embed_url = self._clean_url(
|
||||
item.get("embed_url")
|
||||
or item.get("embed")
|
||||
or item.get("page_url")
|
||||
or (item.get("url") if source_type == "iframe" else "")
|
||||
)
|
||||
homepage_url = self._clean_url(
|
||||
item.get("homepage_url")
|
||||
or item.get("source_url")
|
||||
or item.get("website")
|
||||
or item.get("url")
|
||||
)
|
||||
metadata = {
|
||||
"provider": self._clean_text(item.get("provider") or item.get("publisher") or item.get("network")) or "Collector",
|
||||
"region": self._clean_text(item.get("region") or item.get("country") or item.get("market")) or "Global",
|
||||
"language": self._clean_text(item.get("language") or item.get("lang") or item.get("locale")) or "und",
|
||||
"source_type": source_type,
|
||||
"embed_url": embed_url,
|
||||
"stream_url": stream_url,
|
||||
"homepage_url": homepage_url,
|
||||
"poster_url": self._clean_url(item.get("poster_url") or item.get("thumbnail_url") or item.get("logo_url")),
|
||||
"youtube_video_id": self._clean_text(
|
||||
item.get("youtube_video_id")
|
||||
or item.get("video_id")
|
||||
or item.get("youtubeVideoId")
|
||||
),
|
||||
"youtube_channel": self._clean_text(
|
||||
item.get("youtube_channel")
|
||||
or item.get("channel_handle")
|
||||
or item.get("youtubeChannel")
|
||||
),
|
||||
"sort_order": item.get("sort_order", 200 + index),
|
||||
"notes": self._clean_text(item.get("notes") or item.get("description") or item.get("summary")),
|
||||
"is_enabled": self._parse_enabled(item),
|
||||
}
|
||||
|
||||
normalized.append(
|
||||
{
|
||||
"source_id": str(stream_id),
|
||||
"name": name,
|
||||
"description": metadata["notes"],
|
||||
"metadata": metadata,
|
||||
"reference_date": item.get("reference_date") or datetime.now(UTC).isoformat(),
|
||||
}
|
||||
)
|
||||
|
||||
return normalized
|
||||
@@ -16,6 +16,7 @@ from typing import Dict, Any, List
|
||||
from datetime import UTC, datetime
|
||||
|
||||
import httpx
|
||||
from urllib.parse import urlencode
|
||||
from app.services.collectors.base import HTTPCollector
|
||||
|
||||
|
||||
@@ -38,9 +39,13 @@ class PeeringDBIXPCollector(HTTPCollector):
|
||||
"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
|
||||
"Accept": "application/json",
|
||||
}
|
||||
# API key is added to URL as query parameter
|
||||
if PEERINGDB_API_KEY:
|
||||
self.base_url = f"{self.base_url}?key={PEERINGDB_API_KEY}"
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
base = self._resolved_url or self.base_url
|
||||
if not PEERINGDB_API_KEY:
|
||||
return base
|
||||
separator = "&" if "?" in base else "?"
|
||||
return f"{base}{separator}{urlencode({'key': PEERINGDB_API_KEY})}"
|
||||
|
||||
async def fetch_with_retry(
|
||||
self, max_retries: int = 3, base_delay: float = 2.0
|
||||
@@ -51,7 +56,7 @@ class PeeringDBIXPCollector(HTTPCollector):
|
||||
for attempt in range(max_retries):
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(self.base_url, headers=self.headers)
|
||||
response = await client.get(self.request_url, headers=self.headers)
|
||||
|
||||
if response.status_code == 429:
|
||||
# Rate limited - wait and retry with exponential backoff
|
||||
@@ -141,8 +146,13 @@ class PeeringDBNetworkCollector(HTTPCollector):
|
||||
"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
|
||||
"Accept": "application/json",
|
||||
}
|
||||
if PEERINGDB_API_KEY:
|
||||
self.base_url = f"{self.base_url}?key={PEERINGDB_API_KEY}"
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
base = self._resolved_url or self.base_url
|
||||
if not PEERINGDB_API_KEY:
|
||||
return base
|
||||
separator = "&" if "?" in base else "?"
|
||||
return f"{base}{separator}{urlencode({'key': PEERINGDB_API_KEY})}"
|
||||
|
||||
async def fetch_with_retry(
|
||||
self, max_retries: int = 3, base_delay: float = 2.0
|
||||
@@ -153,7 +163,7 @@ class PeeringDBNetworkCollector(HTTPCollector):
|
||||
for attempt in range(max_retries):
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(self.base_url, headers=self.headers)
|
||||
response = await client.get(self.request_url, headers=self.headers)
|
||||
|
||||
if response.status_code == 429:
|
||||
delay = base_delay * (2**attempt)
|
||||
@@ -244,8 +254,13 @@ class PeeringDBFacilityCollector(HTTPCollector):
|
||||
"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
|
||||
"Accept": "application/json",
|
||||
}
|
||||
if PEERINGDB_API_KEY:
|
||||
self.base_url = f"{self.base_url}?key={PEERINGDB_API_KEY}"
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
base = self._resolved_url or self.base_url
|
||||
if not PEERINGDB_API_KEY:
|
||||
return base
|
||||
separator = "&" if "?" in base else "?"
|
||||
return f"{base}{separator}{urlencode({'key': PEERINGDB_API_KEY})}"
|
||||
|
||||
async def fetch_with_retry(
|
||||
self, max_retries: int = 3, base_delay: float = 2.0
|
||||
@@ -256,7 +271,7 @@ class PeeringDBFacilityCollector(HTTPCollector):
|
||||
for attempt in range(max_retries):
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(self.base_url, headers=self.headers)
|
||||
response = await client.get(self.request_url, headers=self.headers)
|
||||
|
||||
if response.status_code == 429:
|
||||
delay = base_delay * (2**attempt)
|
||||
|
||||
@@ -33,7 +33,7 @@ class RISLiveCollector(BaseCollector):
|
||||
|
||||
def _fetch_via_stream(self) -> list[dict[str, Any]]:
|
||||
events: list[dict[str, Any]] = []
|
||||
stream_url = "https://ris-live.ripe.net/v1/stream/?format=json&client=planet-ris-live"
|
||||
stream_url = self._resolved_url or ""
|
||||
subscribe = json.dumps(
|
||||
{
|
||||
"host": "rrc00",
|
||||
|
||||
@@ -7,6 +7,7 @@ API documentation: https://www.space-track.org/documentation
|
||||
import json
|
||||
from typing import Dict, Any, List
|
||||
import httpx
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from app.services.collectors.base import BaseCollector
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
@@ -21,12 +22,30 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
data_type = "satellite_tle"
|
||||
|
||||
@property
|
||||
def base_url(self) -> str:
|
||||
def query_url(self) -> str:
|
||||
config = get_data_sources_config()
|
||||
if self._resolved_url:
|
||||
return self._resolved_url
|
||||
return config.get_yaml_url("spacetrack_tle")
|
||||
|
||||
@property
|
||||
def site_root(self) -> str:
|
||||
config = get_data_sources_config()
|
||||
configured_root = config.get_yaml_value("spacetrack.base_url")
|
||||
if isinstance(configured_root, str) and configured_root:
|
||||
return configured_root.rstrip("/")
|
||||
|
||||
parsed = urlparse(self.query_url)
|
||||
return f"{parsed.scheme}://{parsed.netloc}".rstrip("/")
|
||||
|
||||
@property
|
||||
def login_url(self) -> str:
|
||||
return f"{self.site_root}/ajaxauth/login"
|
||||
|
||||
@property
|
||||
def probe_url(self) -> str:
|
||||
return f"{self.site_root}/basicspacedata/query/class/gp/NORAD_CAT_ID/25544/format/json"
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
from app.core.config import settings
|
||||
|
||||
@@ -47,13 +66,13 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
|
||||
"Accept": "application/json, text/html, */*",
|
||||
"Accept-Language": "en-US,en;q=0.9",
|
||||
"Referer": "https://www.space-track.org/",
|
||||
"Referer": f"{self.site_root}/",
|
||||
},
|
||||
) as client:
|
||||
await client.get("https://www.space-track.org/")
|
||||
await client.get(f"{self.site_root}/")
|
||||
|
||||
login_response = await client.post(
|
||||
"https://www.space-track.org/ajaxauth/login",
|
||||
self.login_url,
|
||||
data={
|
||||
"identity": username,
|
||||
"password": password,
|
||||
@@ -69,7 +88,7 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
timeout=120.0,
|
||||
follow_redirects=True,
|
||||
) as alt_client:
|
||||
await alt_client.get("https://www.space-track.org/")
|
||||
await alt_client.get(f"{self.site_root}/")
|
||||
|
||||
form_data = {
|
||||
"username": username,
|
||||
@@ -77,7 +96,7 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
"query": "class/gp/NORAD_CAT_ID/25544/format/json",
|
||||
}
|
||||
alt_login = await alt_client.post(
|
||||
"https://www.space-track.org/ajaxauth/login",
|
||||
self.login_url,
|
||||
data={
|
||||
"identity": username,
|
||||
"password": password,
|
||||
@@ -86,9 +105,7 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
print(f"SPACETRACK: Alt login status: {alt_login.status_code}")
|
||||
|
||||
if alt_login.status_code == 200:
|
||||
tle_response = await alt_client.get(
|
||||
"https://www.space-track.org/basicspacedata/query/class/gp/NORAD_CAT_ID/25544/format/json"
|
||||
)
|
||||
tle_response = await alt_client.get(self.probe_url)
|
||||
if tle_response.status_code == 200:
|
||||
data = tle_response.json()
|
||||
print(f"SPACETRACK: Received {len(data)} records via alt method")
|
||||
@@ -98,9 +115,7 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
print(f"SPACETRACK: Login failed, using sample data")
|
||||
return self._get_sample_data()
|
||||
|
||||
tle_response = await client.get(
|
||||
"https://www.space-track.org/basicspacedata/query/class/gp/NORAD_CAT_ID/25544/format/json"
|
||||
)
|
||||
tle_response = await client.get(self.probe_url)
|
||||
print(f"SPACETRACK: TLE query status: {tle_response.status_code}")
|
||||
|
||||
if tle_response.status_code != 200:
|
||||
@@ -127,11 +142,11 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
},
|
||||
) as client:
|
||||
# First, visit the main page to get any cookies
|
||||
await client.get("https://www.space-track.org/")
|
||||
await client.get(f"{self.site_root}/")
|
||||
|
||||
# Login to get session cookie
|
||||
login_response = await client.post(
|
||||
"https://www.space-track.org/ajaxauth/login",
|
||||
self.login_url,
|
||||
data={
|
||||
"identity": username,
|
||||
"password": password,
|
||||
@@ -146,13 +161,7 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
return self._get_sample_data()
|
||||
|
||||
# Query for TLE data (get first 1000 satellites)
|
||||
tle_response = await client.get(
|
||||
"https://www.space-track.org/basicspacedata/query"
|
||||
"/class/gp"
|
||||
"/orderby/EPOCH%20desc"
|
||||
"/limit/1000"
|
||||
"/format/json"
|
||||
)
|
||||
tle_response = await client.get(self.query_url)
|
||||
print(f"SPACETRACK: TLE query status: {tle_response.status_code}")
|
||||
|
||||
if tle_response.status_code != 200:
|
||||
|
||||
@@ -11,6 +11,7 @@ from datetime import UTC, datetime
|
||||
from bs4 import BeautifulSoup
|
||||
import httpx
|
||||
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
from app.services.collectors.base import BaseCollector
|
||||
|
||||
|
||||
@@ -24,15 +25,17 @@ class TeleGeographyCableCollector(BaseCollector):
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
"""Fetch submarine cable data from Wayback Machine"""
|
||||
config = get_data_sources_config()
|
||||
# Try multiple data sources
|
||||
sources = [
|
||||
# Wayback Machine archive of TeleGeography
|
||||
"https://web.archive.org/web/2024/https://www.submarinecablemap.com/api/v3/cable",
|
||||
# Alternative: Try scraping the page
|
||||
"https://www.submarinecablemap.com",
|
||||
self._resolved_url or "",
|
||||
str(config.get_yaml_value("telegeography.archived_cable_url") or ""),
|
||||
str(config.get_yaml_value("telegeography.live_map_url") or ""),
|
||||
]
|
||||
|
||||
for url in sources:
|
||||
if not url:
|
||||
continue
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=60.0, follow_redirects=True) as client:
|
||||
response = await client.get(url)
|
||||
@@ -161,7 +164,7 @@ class TeleGeographyLandingPointCollector(BaseCollector):
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
"""Fetch landing point data from GitHub mirror"""
|
||||
url = "https://raw.githubusercontent.com/lintaojlu/submarine_cable_information/main/landing_point.json"
|
||||
url = self._resolved_url or ""
|
||||
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(url)
|
||||
@@ -225,7 +228,7 @@ class TeleGeographyCableSystemCollector(BaseCollector):
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
"""Fetch cable system data"""
|
||||
url = "https://raw.githubusercontent.com/lintaojlu/submarine_cable_information/main/cable.json"
|
||||
url = self._resolved_url or ""
|
||||
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(url)
|
||||
|
||||
@@ -10,6 +10,7 @@ from typing import Dict, Any, List
|
||||
from bs4 import BeautifulSoup
|
||||
import httpx
|
||||
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
from app.services.collectors.base import BaseCollector
|
||||
|
||||
|
||||
@@ -22,7 +23,7 @@ class TOP500Collector(BaseCollector):
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
"""Fetch TOP500 list data and enrich each row with detail-page metadata."""
|
||||
url = "https://top500.org/lists/top500/list/2025/11/"
|
||||
url = self._resolved_url or ""
|
||||
|
||||
async with httpx.AsyncClient(timeout=60.0, follow_redirects=True) as client:
|
||||
response = await client.get(url)
|
||||
@@ -48,11 +49,13 @@ class TOP500Collector(BaseCollector):
|
||||
return await asyncio.gather(*(enrich(entry) for entry in entries))
|
||||
|
||||
def _extract_system_fields(self, system_cell) -> Dict[str, str]:
|
||||
config = get_data_sources_config()
|
||||
top500_base_url = config.get_yaml_value("top500.base_url") or "https://top500.org"
|
||||
link = system_cell.find("a")
|
||||
system_name = link.get_text(" ", strip=True) if link else system_cell.get_text(" ", strip=True)
|
||||
detail_url = ""
|
||||
if link and link.get("href"):
|
||||
detail_url = f"https://top500.org{link.get('href')}"
|
||||
detail_url = f"{str(top500_base_url).rstrip('/')}{link.get('href')}"
|
||||
|
||||
manufacturer = ""
|
||||
if link and link.next_sibling:
|
||||
|
||||
@@ -0,0 +1,540 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
from email.utils import parsedate_to_datetime
|
||||
import hashlib
|
||||
import html
|
||||
import re
|
||||
from typing import Any
|
||||
from urllib.parse import quote
|
||||
import xml.etree.ElementTree as ET
|
||||
|
||||
import httpx
|
||||
from bs4 import BeautifulSoup
|
||||
|
||||
|
||||
USER_AGENT = "PlanetEarthNewsBoard/1.0 (+https://planet.local)"
|
||||
REQUEST_TIMEOUT = 12.0
|
||||
MAX_ITEMS_PER_SOURCE = 6
|
||||
MAX_ITEMS_TOTAL = 12
|
||||
STALE_CACHE_MAX_AGE_SECONDS = 60 * 45
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RegionProfile:
|
||||
key: str
|
||||
label: str
|
||||
query: str
|
||||
accent: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RegionAnchor:
|
||||
region: str
|
||||
label: str
|
||||
latitude: float
|
||||
longitude: float
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class NewsFeedSource:
|
||||
id: str
|
||||
name: str
|
||||
region: str
|
||||
feed_url: str
|
||||
homepage_url: str
|
||||
source_type: str = "rss"
|
||||
priority: int = 100
|
||||
|
||||
|
||||
@dataclass
|
||||
class ParsedNewsItem:
|
||||
id: str
|
||||
title: str
|
||||
summary: str
|
||||
url: str
|
||||
source: str
|
||||
feed_name: str
|
||||
feed_region: str
|
||||
homepage_url: str
|
||||
published_at: datetime | None
|
||||
|
||||
|
||||
@dataclass
|
||||
class CachedRegionFeed:
|
||||
region: str
|
||||
fetched_at: datetime
|
||||
items: list[ParsedNewsItem]
|
||||
sources: list[NewsFeedSource]
|
||||
|
||||
|
||||
REGION_PROFILES: dict[str, RegionProfile] = {
|
||||
"americas": RegionProfile(
|
||||
key="americas",
|
||||
label="美洲焦点",
|
||||
query='Americas geopolitics OR Latin America OR "United States" OR Canada',
|
||||
accent="#79d3ff",
|
||||
),
|
||||
"europe": RegionProfile(
|
||||
key="europe",
|
||||
label="欧洲焦点",
|
||||
query='Europe geopolitics OR EU OR NATO OR "Eastern Europe"',
|
||||
accent="#8fd4ff",
|
||||
),
|
||||
"middle-east-africa": RegionProfile(
|
||||
key="middle-east-africa",
|
||||
label="中东与非洲焦点",
|
||||
query='"Middle East" OR Africa geopolitics OR Red Sea OR Gulf',
|
||||
accent="#ffb56a",
|
||||
),
|
||||
"asia-pacific": RegionProfile(
|
||||
key="asia-pacific",
|
||||
label="亚太焦点",
|
||||
query='"Asia Pacific" OR Indo-Pacific OR China OR Japan OR Korea OR ASEAN',
|
||||
accent="#78f2cf",
|
||||
),
|
||||
"global": RegionProfile(
|
||||
key="global",
|
||||
label="全球焦点",
|
||||
query='"world news" OR geopolitics OR "global affairs"',
|
||||
accent="#d6e6ff",
|
||||
),
|
||||
}
|
||||
|
||||
REGION_ANCHORS: dict[str, RegionAnchor] = {
|
||||
"americas": RegionAnchor(
|
||||
region="americas",
|
||||
label="美洲",
|
||||
latitude=37.0902,
|
||||
longitude=-95.7129,
|
||||
),
|
||||
"europe": RegionAnchor(
|
||||
region="europe",
|
||||
label="欧洲",
|
||||
latitude=50.1109,
|
||||
longitude=8.6821,
|
||||
),
|
||||
"middle-east-africa": RegionAnchor(
|
||||
region="middle-east-africa",
|
||||
label="中东与非洲",
|
||||
latitude=25.2048,
|
||||
longitude=55.2708,
|
||||
),
|
||||
"asia-pacific": RegionAnchor(
|
||||
region="asia-pacific",
|
||||
label="亚太",
|
||||
latitude=1.3521,
|
||||
longitude=103.8198,
|
||||
),
|
||||
"global": RegionAnchor(
|
||||
region="global",
|
||||
label="全球",
|
||||
latitude=20.0,
|
||||
longitude=0.0,
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def _google_news_feed(query: str, *, hl: str, gl: str, ceid: str) -> str:
|
||||
return (
|
||||
"https://news.google.com/rss/search?q="
|
||||
+ quote(query, safe="")
|
||||
+ f"&hl={hl}&gl={gl}&ceid={ceid}"
|
||||
)
|
||||
|
||||
|
||||
NEWS_FEED_SOURCES: tuple[NewsFeedSource, ...] = (
|
||||
NewsFeedSource(
|
||||
id="bbc-world",
|
||||
name="BBC World",
|
||||
region="global",
|
||||
feed_url="https://feeds.bbci.co.uk/news/world/rss.xml",
|
||||
homepage_url="https://www.bbc.com/news/world",
|
||||
priority=10,
|
||||
),
|
||||
NewsFeedSource(
|
||||
id="dw-top",
|
||||
name="DW Top Stories",
|
||||
region="europe",
|
||||
feed_url="https://rss.dw.com/rdf/rss-en-top",
|
||||
homepage_url="https://www.dw.com/en/top-stories/s-9097",
|
||||
priority=20,
|
||||
),
|
||||
NewsFeedSource(
|
||||
id="global-scan",
|
||||
name="Global Monitor / World",
|
||||
region="global",
|
||||
feed_url=_google_news_feed(
|
||||
REGION_PROFILES["global"].query,
|
||||
hl="en-US",
|
||||
gl="US",
|
||||
ceid="US:en",
|
||||
),
|
||||
homepage_url="https://news.google.com/",
|
||||
source_type="aggregated",
|
||||
priority=30,
|
||||
),
|
||||
NewsFeedSource(
|
||||
id="google-americas",
|
||||
name="Global Monitor / Americas",
|
||||
region="americas",
|
||||
feed_url=_google_news_feed(
|
||||
REGION_PROFILES["americas"].query,
|
||||
hl="en-US",
|
||||
gl="US",
|
||||
ceid="US:en",
|
||||
),
|
||||
homepage_url="https://news.google.com/",
|
||||
source_type="aggregated",
|
||||
priority=40,
|
||||
),
|
||||
NewsFeedSource(
|
||||
id="google-europe",
|
||||
name="Global Monitor / Europe",
|
||||
region="europe",
|
||||
feed_url=_google_news_feed(
|
||||
REGION_PROFILES["europe"].query,
|
||||
hl="en-GB",
|
||||
gl="GB",
|
||||
ceid="GB:en",
|
||||
),
|
||||
homepage_url="https://news.google.com/",
|
||||
source_type="aggregated",
|
||||
priority=40,
|
||||
),
|
||||
NewsFeedSource(
|
||||
id="google-mea",
|
||||
name="Global Monitor / MEA",
|
||||
region="middle-east-africa",
|
||||
feed_url=_google_news_feed(
|
||||
REGION_PROFILES["middle-east-africa"].query,
|
||||
hl="en-US",
|
||||
gl="US",
|
||||
ceid="US:en",
|
||||
),
|
||||
homepage_url="https://news.google.com/",
|
||||
source_type="aggregated",
|
||||
priority=40,
|
||||
),
|
||||
NewsFeedSource(
|
||||
id="google-apac",
|
||||
name="Global Monitor / APAC",
|
||||
region="asia-pacific",
|
||||
feed_url=_google_news_feed(
|
||||
REGION_PROFILES["asia-pacific"].query,
|
||||
hl="en-SG",
|
||||
gl="SG",
|
||||
ceid="SG:en",
|
||||
),
|
||||
homepage_url="https://news.google.com/",
|
||||
source_type="aggregated",
|
||||
priority=40,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
_REGION_CACHE: dict[str, CachedRegionFeed] = {}
|
||||
|
||||
|
||||
def determine_focus_region(lat: float | None, lon: float | None) -> str:
|
||||
if lat is None or lon is None:
|
||||
return "global"
|
||||
if -170 <= lon <= -30:
|
||||
return "americas"
|
||||
if -30 < lon <= 45:
|
||||
return "europe" if lat >= 30 else "middle-east-africa"
|
||||
if 45 < lon <= 150:
|
||||
return "middle-east-africa" if lat < 10 else "asia-pacific"
|
||||
return "asia-pacific"
|
||||
|
||||
|
||||
def get_region_profile(region: str) -> RegionProfile:
|
||||
return REGION_PROFILES.get(region, REGION_PROFILES["global"])
|
||||
|
||||
|
||||
def get_region_anchor(region: str) -> RegionAnchor:
|
||||
return REGION_ANCHORS.get(region, REGION_ANCHORS["global"])
|
||||
|
||||
|
||||
def get_sources_for_region(region: str) -> list[NewsFeedSource]:
|
||||
return sorted(
|
||||
[source for source in NEWS_FEED_SOURCES if source.region in {"global", region}],
|
||||
key=lambda source: (source.priority, source.name),
|
||||
)
|
||||
|
||||
|
||||
def _strip_html(value: str) -> str:
|
||||
if not value:
|
||||
return ""
|
||||
soup = BeautifulSoup(value, "html.parser")
|
||||
return re.sub(r"\s+", " ", soup.get_text(" ", strip=True)).strip()
|
||||
|
||||
|
||||
def _truncate(value: str, limit: int = 180) -> str:
|
||||
text = value.strip()
|
||||
if len(text) <= limit:
|
||||
return text
|
||||
return text[: limit - 1].rstrip() + "…"
|
||||
|
||||
|
||||
def _normalize_source_name(raw: str, fallback: str) -> str:
|
||||
text = html.unescape((raw or "").strip())
|
||||
if " - " in text:
|
||||
return text.split(" - ")[-1].strip() or fallback
|
||||
return text or fallback
|
||||
|
||||
|
||||
def _parse_datetime(raw: str | None) -> datetime | None:
|
||||
if not raw:
|
||||
return None
|
||||
text = raw.strip()
|
||||
if not text:
|
||||
return None
|
||||
|
||||
for parser in (
|
||||
lambda value: parsedate_to_datetime(value),
|
||||
lambda value: datetime.fromisoformat(value.replace("Z", "+00:00")),
|
||||
):
|
||||
try:
|
||||
parsed = parser(text)
|
||||
if parsed.tzinfo is None:
|
||||
parsed = parsed.replace(tzinfo=UTC)
|
||||
return parsed.astimezone(UTC)
|
||||
except Exception:
|
||||
continue
|
||||
return None
|
||||
|
||||
|
||||
def _extract_item_text(element: ET.Element, *names: str) -> str:
|
||||
for name in names:
|
||||
node = element.find(name)
|
||||
if node is not None and node.text:
|
||||
return node.text.strip()
|
||||
return ""
|
||||
|
||||
|
||||
def _parse_feed_entries(xml_text: str, source: NewsFeedSource) -> list[ParsedNewsItem]:
|
||||
root = ET.fromstring(xml_text)
|
||||
items: list[ParsedNewsItem] = []
|
||||
|
||||
rss_items = root.findall("./channel/item")
|
||||
atom_entries = root.findall("{http://www.w3.org/2005/Atom}entry")
|
||||
nodes = rss_items or atom_entries
|
||||
|
||||
for node in nodes[:MAX_ITEMS_PER_SOURCE]:
|
||||
if node.tag.endswith("entry"):
|
||||
title = _extract_item_text(node, "{http://www.w3.org/2005/Atom}title")
|
||||
summary = _extract_item_text(
|
||||
node,
|
||||
"{http://www.w3.org/2005/Atom}summary",
|
||||
"{http://www.w3.org/2005/Atom}content",
|
||||
)
|
||||
link_node = node.find("{http://www.w3.org/2005/Atom}link")
|
||||
link = link_node.get("href", "").strip() if link_node is not None else ""
|
||||
published = _extract_item_text(
|
||||
node,
|
||||
"{http://www.w3.org/2005/Atom}updated",
|
||||
"{http://www.w3.org/2005/Atom}published",
|
||||
)
|
||||
else:
|
||||
title = _extract_item_text(node, "title")
|
||||
summary = _extract_item_text(node, "description", "content")
|
||||
link = _extract_item_text(node, "link")
|
||||
published = _extract_item_text(node, "pubDate", "published", "updated")
|
||||
|
||||
clean_title = html.unescape(title).strip()
|
||||
clean_summary = _truncate(_strip_html(summary), 180)
|
||||
if not clean_title or not link:
|
||||
continue
|
||||
|
||||
item_source = _normalize_source_name(clean_title, source.name)
|
||||
display_title = clean_title
|
||||
if source.source_type == "aggregated" and " - " in clean_title:
|
||||
parts = clean_title.rsplit(" - ", 1)
|
||||
display_title = parts[0].strip()
|
||||
item_source = _normalize_source_name(parts[1], source.name)
|
||||
|
||||
items.append(
|
||||
ParsedNewsItem(
|
||||
id=f"{source.id}:{hashlib.sha1(link.encode('utf-8')).hexdigest()[:12]}",
|
||||
title=display_title,
|
||||
summary=clean_summary,
|
||||
url=link,
|
||||
source=item_source,
|
||||
feed_name=source.name,
|
||||
feed_region=source.region,
|
||||
homepage_url=source.homepage_url,
|
||||
published_at=_parse_datetime(published),
|
||||
)
|
||||
)
|
||||
|
||||
return items
|
||||
|
||||
|
||||
def _serialize_sources(sources: list[NewsFeedSource]) -> list[dict[str, Any]]:
|
||||
return [
|
||||
{
|
||||
"id": source.id,
|
||||
"name": source.name,
|
||||
"region": source.region,
|
||||
"homepage_url": source.homepage_url,
|
||||
}
|
||||
for source in sources
|
||||
]
|
||||
|
||||
|
||||
def _serialize_item(item: ParsedNewsItem, *, active_region: str) -> dict[str, Any]:
|
||||
published_at = item.published_at
|
||||
anchor = get_region_anchor(item.feed_region)
|
||||
return {
|
||||
"id": item.id,
|
||||
"title": item.title,
|
||||
"summary": item.summary,
|
||||
"url": item.url,
|
||||
"source": item.source,
|
||||
"feed_name": item.feed_name,
|
||||
"region": item.feed_region,
|
||||
"homepage_url": item.homepage_url,
|
||||
"published_at": published_at.isoformat().replace("+00:00", "Z") if published_at else None,
|
||||
"latitude": anchor.latitude,
|
||||
"longitude": anchor.longitude,
|
||||
"location_label": anchor.label,
|
||||
"location_inferred": True,
|
||||
"is_focus_match": item.feed_region == active_region,
|
||||
}
|
||||
|
||||
|
||||
def _build_payload(
|
||||
*,
|
||||
lat: float | None,
|
||||
lon: float | None,
|
||||
active_region: str,
|
||||
items: list[ParsedNewsItem],
|
||||
sources: list[NewsFeedSource],
|
||||
errors: list[str],
|
||||
stale: bool,
|
||||
generated_at: datetime | None = None,
|
||||
) -> dict[str, Any]:
|
||||
profile = get_region_profile(active_region)
|
||||
timestamp = generated_at or datetime.now(UTC)
|
||||
return {
|
||||
"generated_at": timestamp.isoformat().replace("+00:00", "Z"),
|
||||
"focus": {
|
||||
"lat": lat,
|
||||
"lon": lon,
|
||||
"region": active_region,
|
||||
"label": profile.label,
|
||||
"accent": profile.accent,
|
||||
},
|
||||
"sources": _serialize_sources(sources),
|
||||
"items": [_serialize_item(item, active_region=active_region) for item in items],
|
||||
"errors": errors,
|
||||
"stale": stale,
|
||||
}
|
||||
|
||||
|
||||
def _rank_and_trim_items(items: list[ParsedNewsItem], *, active_region: str) -> list[ParsedNewsItem]:
|
||||
deduped: dict[str, ParsedNewsItem] = {}
|
||||
for item in items:
|
||||
key = item.url.strip() or item.title.strip().lower()
|
||||
if key not in deduped:
|
||||
deduped[key] = item
|
||||
|
||||
return sorted(
|
||||
deduped.values(),
|
||||
key=lambda item: (
|
||||
item.feed_region != active_region,
|
||||
item.published_at is None,
|
||||
-(item.published_at.timestamp() if item.published_at else 0),
|
||||
item.feed_name,
|
||||
),
|
||||
)[:MAX_ITEMS_TOTAL]
|
||||
|
||||
|
||||
def _get_cached_region_feed(region: str) -> CachedRegionFeed | None:
|
||||
cached = _REGION_CACHE.get(region)
|
||||
if not cached:
|
||||
return None
|
||||
age_seconds = (datetime.now(UTC) - cached.fetched_at).total_seconds()
|
||||
if age_seconds > STALE_CACHE_MAX_AGE_SECONDS:
|
||||
return None
|
||||
return cached
|
||||
|
||||
|
||||
def _store_region_cache(region: str, *, items: list[ParsedNewsItem], sources: list[NewsFeedSource]) -> None:
|
||||
_REGION_CACHE[region] = CachedRegionFeed(
|
||||
region=region,
|
||||
fetched_at=datetime.now(UTC),
|
||||
items=list(items),
|
||||
sources=list(sources),
|
||||
)
|
||||
|
||||
|
||||
async def _fetch_source(
|
||||
client: httpx.AsyncClient,
|
||||
source: NewsFeedSource,
|
||||
) -> tuple[NewsFeedSource, list[ParsedNewsItem], str | None]:
|
||||
try:
|
||||
response = await client.get(source.feed_url)
|
||||
response.raise_for_status()
|
||||
return source, _parse_feed_entries(response.text, source), None
|
||||
except Exception as exc:
|
||||
return source, [], str(exc)
|
||||
|
||||
|
||||
async def get_earth_news_payload(lat: float | None = None, lon: float | None = None) -> dict[str, Any]:
|
||||
active_region = determine_focus_region(lat, lon)
|
||||
sources = get_sources_for_region(active_region)
|
||||
errors: list[str] = []
|
||||
|
||||
async with httpx.AsyncClient(
|
||||
timeout=REQUEST_TIMEOUT,
|
||||
follow_redirects=True,
|
||||
headers={"User-Agent": USER_AGENT},
|
||||
) as client:
|
||||
results = await asyncio.gather(*(_fetch_source(client, source) for source in sources))
|
||||
|
||||
fetched_items: list[ParsedNewsItem] = []
|
||||
for source, items, error in results:
|
||||
if error:
|
||||
errors.append(f"{source.name}: {error}")
|
||||
continue
|
||||
fetched_items.extend(items)
|
||||
|
||||
ranked_items = _rank_and_trim_items(fetched_items, active_region=active_region)
|
||||
if ranked_items:
|
||||
_store_region_cache(active_region, items=ranked_items, sources=sources)
|
||||
return _build_payload(
|
||||
lat=lat,
|
||||
lon=lon,
|
||||
active_region=active_region,
|
||||
items=ranked_items,
|
||||
sources=sources,
|
||||
errors=errors,
|
||||
stale=False,
|
||||
)
|
||||
|
||||
cached = _get_cached_region_feed(active_region)
|
||||
if cached:
|
||||
return _build_payload(
|
||||
lat=lat,
|
||||
lon=lon,
|
||||
active_region=active_region,
|
||||
items=cached.items,
|
||||
sources=cached.sources,
|
||||
errors=errors,
|
||||
stale=True,
|
||||
generated_at=cached.fetched_at,
|
||||
)
|
||||
|
||||
return _build_payload(
|
||||
lat=lat,
|
||||
lon=lon,
|
||||
active_region=active_region,
|
||||
items=[],
|
||||
sources=sources,
|
||||
errors=errors,
|
||||
stale=False,
|
||||
)
|
||||
@@ -0,0 +1,86 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from app.core.logging import get_logger, sanitize_log_value
|
||||
from app.core.request_context import get_request_id
|
||||
from app.db.session import async_session_factory
|
||||
from app.models.system_log import AuditLog, SystemLog
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
async def record_system_log(
|
||||
*,
|
||||
source: str,
|
||||
level: str,
|
||||
message: str,
|
||||
service: str | None = None,
|
||||
module: str | None = None,
|
||||
event: str | None = None,
|
||||
request_id: str | None = None,
|
||||
trace_id: str | None = None,
|
||||
user_id: int | None = None,
|
||||
category: str | None = None,
|
||||
context: dict[str, Any] | None = None,
|
||||
) -> None:
|
||||
try:
|
||||
async with async_session_factory() as session:
|
||||
session.add(
|
||||
SystemLog(
|
||||
source=source,
|
||||
service=service,
|
||||
module=module,
|
||||
event=event,
|
||||
level=level.lower(),
|
||||
message=str(sanitize_log_value(message)),
|
||||
request_id=request_id or get_request_id(),
|
||||
trace_id=trace_id,
|
||||
user_id=user_id,
|
||||
category=category,
|
||||
context=sanitize_log_value(context or {}),
|
||||
)
|
||||
)
|
||||
await session.commit()
|
||||
except Exception:
|
||||
logger.exception_event(
|
||||
"Failed to persist system log",
|
||||
event="system_log.persist.failed",
|
||||
context={"event_name": event, "source": source},
|
||||
)
|
||||
|
||||
|
||||
async def record_audit_log(
|
||||
*,
|
||||
action: str,
|
||||
actor_id: int | None = None,
|
||||
actor_name: str | None = None,
|
||||
target_type: str | None = None,
|
||||
target_id: str | None = None,
|
||||
result: str | None = None,
|
||||
request_id: str | None = None,
|
||||
ip: str | None = None,
|
||||
details: dict[str, Any] | None = None,
|
||||
) -> None:
|
||||
try:
|
||||
async with async_session_factory() as session:
|
||||
session.add(
|
||||
AuditLog(
|
||||
actor_id=actor_id,
|
||||
actor_name=actor_name,
|
||||
action=action,
|
||||
target_type=target_type,
|
||||
target_id=target_id,
|
||||
result=result,
|
||||
request_id=request_id or get_request_id(),
|
||||
ip=ip,
|
||||
details=sanitize_log_value(details or {}),
|
||||
)
|
||||
)
|
||||
await session.commit()
|
||||
except Exception:
|
||||
logger.exception_event(
|
||||
"Failed to persist audit log",
|
||||
event="audit_log.persist.failed",
|
||||
context={"action": action},
|
||||
)
|
||||
@@ -0,0 +1,694 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from collections.abc import Sequence
|
||||
from datetime import UTC, datetime
|
||||
from time import perf_counter
|
||||
from uuid import uuid4
|
||||
|
||||
from fastapi import HTTPException, status
|
||||
from sqlalchemy import func, select, update
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.db.session import async_session_factory
|
||||
from app.models.playground_message import PlaygroundMessage
|
||||
from app.models.playground_session import PlaygroundSession
|
||||
from app.schemas.ai import (
|
||||
PlaygroundMessageEditRequest,
|
||||
PlaygroundMessageActionResponse,
|
||||
PlaygroundMessageCreateRequest,
|
||||
PlaygroundMessageRecord,
|
||||
PlaygroundMessageResendRequest,
|
||||
PlaygroundMessageStopRequest,
|
||||
PlaygroundSessionResponse,
|
||||
PlaygroundSessionState,
|
||||
PlaygroundSessionUpsertRequest,
|
||||
PlaygroundThreadResponse,
|
||||
SituationalAnalysisRequest,
|
||||
)
|
||||
from app.services.ai_client import AIProviderClient
|
||||
from app.services.playground_session_store import _to_response as session_to_response
|
||||
from app.services.playground_session_store import upsert_playground_session
|
||||
|
||||
STREAM_CHUNK_SIZE = 24
|
||||
STREAM_INTERVAL_SECONDS = 0.08
|
||||
THINKING_PREVIEW_SECONDS = 2.6
|
||||
|
||||
|
||||
class _ActiveRun:
|
||||
def __init__(self, task: asyncio.Task[None]) -> None:
|
||||
self.task = task
|
||||
self.stop_requested = asyncio.Event()
|
||||
|
||||
|
||||
_ACTIVE_RUNS: dict[str, _ActiveRun] = {}
|
||||
|
||||
|
||||
async def _get_session_by_key(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
session_key: str,
|
||||
) -> PlaygroundSession | None:
|
||||
result = await db.execute(
|
||||
select(PlaygroundSession).where(
|
||||
PlaygroundSession.user_id == user_id,
|
||||
PlaygroundSession.session_key == session_key,
|
||||
)
|
||||
)
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
|
||||
async def _require_session(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
session_key: str,
|
||||
) -> PlaygroundSession:
|
||||
session = await _get_session_by_key(db, user_id=user_id, session_key=session_key)
|
||||
if session is None:
|
||||
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Playground session not found")
|
||||
return session
|
||||
|
||||
|
||||
async def _require_visible_message(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
public_id: str,
|
||||
role: str | None = None,
|
||||
) -> PlaygroundMessage:
|
||||
conditions = [
|
||||
PlaygroundMessage.user_id == user_id,
|
||||
PlaygroundMessage.public_id == public_id,
|
||||
PlaygroundMessage.is_visible.is_(True),
|
||||
]
|
||||
if role is not None:
|
||||
conditions.append(PlaygroundMessage.role == role)
|
||||
|
||||
result = await db.execute(select(PlaygroundMessage).where(*conditions))
|
||||
message = result.scalar_one_or_none()
|
||||
if message is None:
|
||||
detail = "User message not found" if role == "user" else "Playground message not found"
|
||||
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail=detail)
|
||||
return message
|
||||
|
||||
|
||||
def _message_to_record(message: PlaygroundMessage, parent_public_id: str | None = None) -> PlaygroundMessageRecord:
|
||||
return PlaygroundMessageRecord(
|
||||
id=message.public_id,
|
||||
role=message.role,
|
||||
kind=message.kind,
|
||||
status=message.status,
|
||||
title=message.title,
|
||||
content=message.content or "",
|
||||
thinking_content=message.thinking_content or "",
|
||||
meta=list(message.meta or []),
|
||||
markdown=message.role != "system",
|
||||
provider=message.provider,
|
||||
model=message.model,
|
||||
request_id=message.request_id,
|
||||
raw_response=dict(message.raw_response or {}),
|
||||
content_blocks=list(message.content_blocks or []),
|
||||
text_blocks=list(message.text_blocks or []),
|
||||
thinking_blocks=list(message.thinking_blocks or []),
|
||||
parent_message_id=parent_public_id,
|
||||
created_at=message.created_at.isoformat(),
|
||||
updated_at=message.updated_at.isoformat(),
|
||||
)
|
||||
|
||||
|
||||
async def _ensure_session(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
session_key: str,
|
||||
title: str,
|
||||
state: PlaygroundSessionState | None = None,
|
||||
) -> PlaygroundSession:
|
||||
result = await db.execute(
|
||||
select(PlaygroundSession).where(
|
||||
PlaygroundSession.user_id == user_id,
|
||||
PlaygroundSession.session_key == session_key,
|
||||
)
|
||||
)
|
||||
session = result.scalar_one_or_none()
|
||||
if session is not None:
|
||||
if title:
|
||||
session.title = title[:200]
|
||||
if state is not None:
|
||||
session.state = state.model_dump(mode="json")
|
||||
await db.flush()
|
||||
await db.refresh(session)
|
||||
return session
|
||||
|
||||
payload = PlaygroundSessionUpsertRequest(
|
||||
session_key=session_key,
|
||||
title=title[:200],
|
||||
state=state or PlaygroundSessionState(title=title[:200]),
|
||||
)
|
||||
await upsert_playground_session(db, user_id=user_id, payload=payload)
|
||||
result = await db.execute(
|
||||
select(PlaygroundSession).where(
|
||||
PlaygroundSession.user_id == user_id,
|
||||
PlaygroundSession.session_key == session_key,
|
||||
)
|
||||
)
|
||||
return result.scalar_one()
|
||||
|
||||
|
||||
async def _list_visible_messages(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
session_id: int,
|
||||
) -> list[PlaygroundMessage]:
|
||||
result = await db.execute(
|
||||
select(PlaygroundMessage)
|
||||
.where(
|
||||
PlaygroundMessage.session_id == session_id,
|
||||
PlaygroundMessage.is_visible.is_(True),
|
||||
)
|
||||
.order_by(PlaygroundMessage.sort_order.asc(), PlaygroundMessage.id.asc())
|
||||
)
|
||||
return list(result.scalars().all())
|
||||
|
||||
|
||||
async def _build_thread_response(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
session: PlaygroundSession,
|
||||
) -> PlaygroundThreadResponse:
|
||||
messages = await _list_visible_messages(db, session_id=session.id)
|
||||
id_map = {item.id: item.public_id for item in messages}
|
||||
return PlaygroundThreadResponse(
|
||||
session=session_to_response(session),
|
||||
messages=[_message_to_record(item, id_map.get(item.parent_message_id)) for item in messages],
|
||||
)
|
||||
|
||||
|
||||
async def get_thread(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
session_key: str,
|
||||
) -> PlaygroundThreadResponse | None:
|
||||
session = await _get_session_by_key(db, user_id=user_id, session_key=session_key)
|
||||
if session is None:
|
||||
return None
|
||||
return await _build_thread_response(db, session=session)
|
||||
|
||||
|
||||
async def _build_action_response(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
session: PlaygroundSession,
|
||||
active_message_id: str | None = None,
|
||||
) -> PlaygroundMessageActionResponse:
|
||||
thread = await _build_thread_response(db, session=session)
|
||||
return PlaygroundMessageActionResponse(
|
||||
session=thread.session,
|
||||
messages=thread.messages,
|
||||
active_message_id=active_message_id,
|
||||
)
|
||||
|
||||
|
||||
def _collect_constraints(raw_constraints: str) -> list[str]:
|
||||
return [item.strip() for item in raw_constraints.split("\n") if item.strip()]
|
||||
|
||||
|
||||
async def _next_sort_order(db: AsyncSession, session_id: int) -> int:
|
||||
result = await db.execute(
|
||||
select(func.max(PlaygroundMessage.sort_order)).where(PlaygroundMessage.session_id == session_id)
|
||||
)
|
||||
current = result.scalar_one_or_none()
|
||||
return int(current or 0)
|
||||
|
||||
|
||||
async def _set_session_state(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
session: PlaygroundSession,
|
||||
payload: PlaygroundMessageCreateRequest,
|
||||
) -> PlaygroundSession:
|
||||
session.state = PlaygroundSessionState(
|
||||
messages=[],
|
||||
selectedPresetKey=payload.selected_preset_key,
|
||||
title=payload.title,
|
||||
objective=payload.objective,
|
||||
constraints=payload.constraints,
|
||||
inputValue="",
|
||||
analysis=None,
|
||||
latestAnalysisMessageId=None,
|
||||
analysisMeta={},
|
||||
helpExpanded=payload.help_expanded,
|
||||
).model_dump(mode="json")
|
||||
session.title = payload.title[:200]
|
||||
await db.flush()
|
||||
await db.refresh(session)
|
||||
return session
|
||||
|
||||
|
||||
def _spawn_assistant_run(
|
||||
*,
|
||||
user_id: int,
|
||||
session_id: int,
|
||||
session_key: str,
|
||||
user_message_id: int,
|
||||
assistant_message_id: int,
|
||||
assistant_public_id: str,
|
||||
payload: PlaygroundMessageCreateRequest,
|
||||
provider_client: AIProviderClient,
|
||||
) -> None:
|
||||
task = asyncio.create_task(
|
||||
_run_assistant_message(
|
||||
user_id=user_id,
|
||||
session_id=session_id,
|
||||
session_key=session_key,
|
||||
user_message_id=user_message_id,
|
||||
assistant_message_id=assistant_message_id,
|
||||
payload=payload,
|
||||
provider_client=provider_client,
|
||||
)
|
||||
)
|
||||
_ACTIVE_RUNS[assistant_public_id] = _ActiveRun(task)
|
||||
|
||||
|
||||
async def create_turn(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
payload: PlaygroundMessageCreateRequest,
|
||||
provider_client: AIProviderClient,
|
||||
) -> PlaygroundMessageActionResponse:
|
||||
session = await _ensure_session(
|
||||
db,
|
||||
user_id=user_id,
|
||||
session_key=payload.session_key,
|
||||
title=payload.title,
|
||||
state=PlaygroundSessionState(
|
||||
selectedPresetKey=payload.selected_preset_key,
|
||||
title=payload.title,
|
||||
objective=payload.objective,
|
||||
constraints=payload.constraints,
|
||||
inputValue="",
|
||||
helpExpanded=payload.help_expanded,
|
||||
),
|
||||
)
|
||||
session = await _set_session_state(db, session=session, payload=payload)
|
||||
base_order = await _next_sort_order(db, session.id)
|
||||
|
||||
user_message = PlaygroundMessage(
|
||||
public_id=uuid4().hex,
|
||||
session_id=session.id,
|
||||
user_id=user_id,
|
||||
role="user",
|
||||
kind="message",
|
||||
status="done",
|
||||
title=payload.selected_preset_key,
|
||||
content=payload.input,
|
||||
meta=[payload.title],
|
||||
sort_order=base_order + 10,
|
||||
)
|
||||
assistant_message = PlaygroundMessage(
|
||||
public_id=uuid4().hex,
|
||||
session_id=session.id,
|
||||
user_id=user_id,
|
||||
parent_message_id=None,
|
||||
role="assistant",
|
||||
kind="thinking",
|
||||
status="pending",
|
||||
title="AI 回应",
|
||||
content="",
|
||||
thinking_content="",
|
||||
meta=[],
|
||||
sort_order=base_order + 20,
|
||||
)
|
||||
db.add(user_message)
|
||||
await db.flush()
|
||||
assistant_message.parent_message_id = user_message.id
|
||||
db.add(assistant_message)
|
||||
await db.flush()
|
||||
await db.refresh(user_message)
|
||||
await db.refresh(assistant_message)
|
||||
await db.commit()
|
||||
await db.refresh(session)
|
||||
await db.refresh(user_message)
|
||||
await db.refresh(assistant_message)
|
||||
|
||||
_spawn_assistant_run(
|
||||
user_id=user_id,
|
||||
session_id=session.id,
|
||||
session_key=payload.session_key,
|
||||
user_message_id=user_message.id,
|
||||
assistant_message_id=assistant_message.id,
|
||||
assistant_public_id=assistant_message.public_id,
|
||||
payload=payload,
|
||||
provider_client=provider_client,
|
||||
)
|
||||
|
||||
return await _build_action_response(
|
||||
db,
|
||||
session=session,
|
||||
active_message_id=assistant_message.public_id,
|
||||
)
|
||||
|
||||
|
||||
async def _create_assistant_retry_turn(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
session: PlaygroundSession,
|
||||
user_message: PlaygroundMessage,
|
||||
payload: PlaygroundMessageCreateRequest,
|
||||
provider_client: AIProviderClient,
|
||||
) -> PlaygroundMessageActionResponse:
|
||||
base_order = await _next_sort_order(db, session.id)
|
||||
assistant_message = PlaygroundMessage(
|
||||
public_id=uuid4().hex,
|
||||
session_id=session.id,
|
||||
user_id=user_id,
|
||||
parent_message_id=user_message.id,
|
||||
role="assistant",
|
||||
kind="thinking",
|
||||
status="pending",
|
||||
title="AI 回应",
|
||||
content="",
|
||||
thinking_content="",
|
||||
meta=[],
|
||||
sort_order=base_order + 10,
|
||||
)
|
||||
db.add(assistant_message)
|
||||
await db.flush()
|
||||
await db.commit()
|
||||
await db.refresh(session)
|
||||
await db.refresh(assistant_message)
|
||||
|
||||
_spawn_assistant_run(
|
||||
user_id=user_id,
|
||||
session_id=session.id,
|
||||
session_key=payload.session_key,
|
||||
user_message_id=user_message.id,
|
||||
assistant_message_id=assistant_message.id,
|
||||
assistant_public_id=assistant_message.public_id,
|
||||
payload=payload,
|
||||
provider_client=provider_client,
|
||||
)
|
||||
|
||||
return await _build_action_response(
|
||||
db,
|
||||
session=session,
|
||||
active_message_id=assistant_message.public_id,
|
||||
)
|
||||
|
||||
|
||||
async def stop_message(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
payload: PlaygroundMessageStopRequest,
|
||||
) -> PlaygroundMessageActionResponse:
|
||||
session = await _require_session(db, user_id=user_id, session_key=payload.session_key)
|
||||
message = await _require_visible_message(db, user_id=user_id, public_id=payload.message_id)
|
||||
|
||||
if message.status not in {"pending", "thinking", "answering"}:
|
||||
return await _build_action_response(db, session=session)
|
||||
|
||||
active_run = _ACTIVE_RUNS.get(message.public_id)
|
||||
if active_run is not None:
|
||||
active_run.stop_requested.set()
|
||||
active_run.task.cancel()
|
||||
|
||||
message.status = "stopped"
|
||||
if "已手动停止生成" not in (message.meta or []):
|
||||
message.meta = [*(message.meta or []), "已手动停止生成"]
|
||||
await db.flush()
|
||||
await db.commit()
|
||||
await db.refresh(message)
|
||||
|
||||
return await _build_action_response(db, session=session)
|
||||
|
||||
|
||||
async def resend_turn(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
payload: PlaygroundMessageResendRequest,
|
||||
provider_client: AIProviderClient,
|
||||
) -> PlaygroundMessageActionResponse:
|
||||
session = await _require_session(db, user_id=user_id, session_key=payload.session_key)
|
||||
user_message = await _require_visible_message(
|
||||
db,
|
||||
user_id=user_id,
|
||||
public_id=payload.user_message_id,
|
||||
role="user",
|
||||
)
|
||||
|
||||
later_messages = await db.execute(
|
||||
select(PlaygroundMessage).where(
|
||||
PlaygroundMessage.session_id == session.id,
|
||||
PlaygroundMessage.sort_order > user_message.sort_order,
|
||||
PlaygroundMessage.is_visible.is_(True),
|
||||
)
|
||||
)
|
||||
for item in later_messages.scalars().all():
|
||||
item.is_visible = False
|
||||
if item.status in {"pending", "thinking", "answering"}:
|
||||
active_run = _ACTIVE_RUNS.get(item.public_id)
|
||||
if active_run is not None:
|
||||
active_run.stop_requested.set()
|
||||
active_run.task.cancel()
|
||||
await db.flush()
|
||||
await db.commit()
|
||||
|
||||
session_state = PlaygroundSessionState.model_validate(session.state or {})
|
||||
create_payload = PlaygroundMessageCreateRequest(
|
||||
session_key=payload.session_key,
|
||||
title=session_state.title or session.title,
|
||||
objective=session_state.objective or "继续当前对话",
|
||||
constraints=session_state.constraints or "",
|
||||
input=user_message.content,
|
||||
selected_preset_key=session_state.selectedPresetKey or "bgp-brief",
|
||||
help_expanded=session_state.helpExpanded,
|
||||
)
|
||||
return await _create_assistant_retry_turn(
|
||||
db,
|
||||
user_id=user_id,
|
||||
session=session,
|
||||
user_message=user_message,
|
||||
payload=create_payload,
|
||||
provider_client=provider_client,
|
||||
)
|
||||
|
||||
|
||||
async def edit_user_message(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
payload: PlaygroundMessageEditRequest,
|
||||
) -> PlaygroundMessageActionResponse:
|
||||
session = await _require_session(db, user_id=user_id, session_key=payload.session_key)
|
||||
user_message = await _require_visible_message(
|
||||
db,
|
||||
user_id=user_id,
|
||||
public_id=payload.user_message_id,
|
||||
role="user",
|
||||
)
|
||||
|
||||
user_message.content = payload.content.strip()
|
||||
await db.flush()
|
||||
await db.commit()
|
||||
await db.refresh(user_message)
|
||||
|
||||
return await _build_action_response(db, session=session)
|
||||
|
||||
|
||||
async def _append_meta_if_missing(db: AsyncSession, message_id: int, meta_line: str) -> None:
|
||||
result = await db.execute(select(PlaygroundMessage).where(PlaygroundMessage.id == message_id))
|
||||
message = result.scalar_one_or_none()
|
||||
if message is None:
|
||||
return
|
||||
if meta_line not in (message.meta or []):
|
||||
message.meta = [*(message.meta or []), meta_line]
|
||||
await db.flush()
|
||||
|
||||
|
||||
async def _should_stop(message_public_id: str) -> bool:
|
||||
active_run = _ACTIVE_RUNS.get(message_public_id)
|
||||
return active_run.stop_requested.is_set() if active_run is not None else False
|
||||
|
||||
|
||||
async def _mark_message_state(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
message_id: int,
|
||||
**updates,
|
||||
) -> PlaygroundMessage:
|
||||
result = await db.execute(select(PlaygroundMessage).where(PlaygroundMessage.id == message_id))
|
||||
message = result.scalar_one()
|
||||
for key, value in updates.items():
|
||||
setattr(message, key, value)
|
||||
await db.flush()
|
||||
await db.refresh(message)
|
||||
return message
|
||||
|
||||
|
||||
def _build_conversation_history(messages: Sequence[PlaygroundMessage], current_user_message_id: int) -> list[dict]:
|
||||
history: list[dict] = []
|
||||
for item in messages:
|
||||
if item.id >= current_user_message_id:
|
||||
break
|
||||
if item.role == "system":
|
||||
continue
|
||||
history.append(
|
||||
{
|
||||
"role": item.role,
|
||||
"kind": item.kind or "message",
|
||||
"title": item.title,
|
||||
"content": item.content or "",
|
||||
}
|
||||
)
|
||||
return history[-8:]
|
||||
|
||||
|
||||
async def _run_assistant_message(
|
||||
*,
|
||||
user_id: int,
|
||||
session_id: int,
|
||||
session_key: str,
|
||||
user_message_id: int,
|
||||
assistant_message_id: int,
|
||||
payload: PlaygroundMessageCreateRequest,
|
||||
provider_client: AIProviderClient,
|
||||
) -> None:
|
||||
request_id = str(uuid4())
|
||||
started_at = perf_counter()
|
||||
assistant_public_id: str | None = None
|
||||
try:
|
||||
async with async_session_factory() as db:
|
||||
session = await db.get(PlaygroundSession, session_id)
|
||||
user_message = await db.get(PlaygroundMessage, user_message_id)
|
||||
assistant_message = await db.get(PlaygroundMessage, assistant_message_id)
|
||||
if session is None or user_message is None or assistant_message is None:
|
||||
return
|
||||
assistant_public_id = assistant_message.public_id
|
||||
|
||||
visible_messages = await _list_visible_messages(db, session_id=session_id)
|
||||
conversation_history = _build_conversation_history(visible_messages, user_message_id)
|
||||
|
||||
request_payload = SituationalAnalysisRequest(
|
||||
title=payload.title,
|
||||
objective=payload.objective,
|
||||
observations=[item.strip() for item in payload.input.split("\n") if item.strip()],
|
||||
constraints=_collect_constraints(payload.constraints),
|
||||
context={
|
||||
"source": "playground",
|
||||
"preset": payload.selected_preset_key,
|
||||
"conversation_history": conversation_history,
|
||||
"history_size": len(conversation_history),
|
||||
},
|
||||
thinking={"type": "enabled"},
|
||||
)
|
||||
|
||||
analysis = await provider_client.analyze(request_payload, request_id=request_id)
|
||||
|
||||
async with async_session_factory() as db:
|
||||
assistant_message = await _mark_message_state(
|
||||
db,
|
||||
message_id=assistant_message_id,
|
||||
status="thinking" if analysis.thinking_blocks else "answering",
|
||||
title=f"{analysis.provider} / {analysis.model}",
|
||||
provider=analysis.provider,
|
||||
model=analysis.model,
|
||||
request_id=request_id,
|
||||
raw_response=analysis.raw_response,
|
||||
content_blocks=[item.model_dump(mode="json") for item in analysis.content_blocks],
|
||||
text_blocks=analysis.text_blocks,
|
||||
thinking_blocks=analysis.thinking_blocks,
|
||||
thinking_content="\n\n".join(analysis.thinking_blocks).strip(),
|
||||
)
|
||||
session = await db.get(PlaygroundSession, session_id)
|
||||
if session is not None:
|
||||
session_state = PlaygroundSessionState.model_validate(session.state or {})
|
||||
session.state = session_state.model_copy(
|
||||
update={
|
||||
"latestAnalysisMessageId": assistant_message.public_id,
|
||||
"analysis": analysis.model_dump(mode="json"),
|
||||
}
|
||||
).model_dump(mode="json")
|
||||
await db.flush()
|
||||
await db.commit()
|
||||
|
||||
if assistant_public_id and analysis.thinking_blocks:
|
||||
await asyncio.sleep(THINKING_PREVIEW_SECONDS)
|
||||
if await _should_stop(assistant_public_id):
|
||||
return
|
||||
|
||||
content = analysis.content or ""
|
||||
cursor = 0
|
||||
while cursor < len(content):
|
||||
if assistant_public_id and await _should_stop(assistant_public_id):
|
||||
return
|
||||
cursor = min(len(content), cursor + STREAM_CHUNK_SIZE)
|
||||
async with async_session_factory() as db:
|
||||
await _mark_message_state(
|
||||
db,
|
||||
message_id=assistant_message_id,
|
||||
status="answering",
|
||||
content=content[:cursor],
|
||||
)
|
||||
await db.commit()
|
||||
await asyncio.sleep(STREAM_INTERVAL_SECONDS)
|
||||
|
||||
duration_ms = round((perf_counter() - started_at) * 1000)
|
||||
async with async_session_factory() as db:
|
||||
assistant_message = await _mark_message_state(
|
||||
db,
|
||||
message_id=assistant_message_id,
|
||||
status="done",
|
||||
content=content,
|
||||
meta=[
|
||||
f"Request ID: {request_id}",
|
||||
f"耗时: {duration_ms} ms",
|
||||
f"完成时间: {datetime.now(UTC).astimezone().isoformat()}",
|
||||
],
|
||||
)
|
||||
session = await db.get(PlaygroundSession, session_id)
|
||||
if session is not None:
|
||||
session_state = PlaygroundSessionState.model_validate(session.state or {})
|
||||
session.state = session_state.model_copy(
|
||||
update={
|
||||
"latestAnalysisMessageId": assistant_message.public_id,
|
||||
"analysis": analysis.model_dump(mode="json"),
|
||||
"analysisMeta": {
|
||||
"requestId": request_id,
|
||||
"durationMs": duration_ms,
|
||||
"completedAt": datetime.now().isoformat(),
|
||||
},
|
||||
}
|
||||
).model_dump(mode="json")
|
||||
await db.flush()
|
||||
await db.commit()
|
||||
except asyncio.CancelledError:
|
||||
async with async_session_factory() as db:
|
||||
result = await db.execute(select(PlaygroundMessage).where(PlaygroundMessage.id == assistant_message_id))
|
||||
message = result.scalar_one_or_none()
|
||||
if message is not None and message.status in {"pending", "thinking", "answering"}:
|
||||
message.status = "stopped"
|
||||
if "已手动停止生成" not in (message.meta or []):
|
||||
message.meta = [*(message.meta or []), "已手动停止生成"]
|
||||
await db.flush()
|
||||
await db.commit()
|
||||
raise
|
||||
except Exception as exc:
|
||||
async with async_session_factory() as db:
|
||||
result = await db.execute(select(PlaygroundMessage).where(PlaygroundMessage.id == assistant_message_id))
|
||||
message = result.scalar_one_or_none()
|
||||
if message is not None:
|
||||
message.status = "error"
|
||||
message.content = message.content or "分析失败,请检查 AI Provider 配置或稍后再试。"
|
||||
message.meta = [*(message.meta or []), f"错误: {type(exc).__name__}"]
|
||||
await db.flush()
|
||||
await db.commit()
|
||||
finally:
|
||||
if assistant_public_id:
|
||||
_ACTIVE_RUNS.pop(assistant_public_id, None)
|
||||
@@ -0,0 +1,72 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.models.playground_session import PlaygroundSession
|
||||
from app.schemas.ai import (
|
||||
PlaygroundSessionResponse,
|
||||
PlaygroundSessionState,
|
||||
PlaygroundSessionUpsertRequest,
|
||||
)
|
||||
|
||||
|
||||
def _to_response(record: PlaygroundSession) -> PlaygroundSessionResponse:
|
||||
return PlaygroundSessionResponse(
|
||||
id=str(record.id),
|
||||
session_key=record.session_key,
|
||||
title=record.title,
|
||||
state=PlaygroundSessionState.model_validate(record.state or {}),
|
||||
created_at=record.created_at.isoformat(),
|
||||
updated_at=record.updated_at.isoformat(),
|
||||
)
|
||||
|
||||
|
||||
async def get_playground_session(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
session_key: str = "default",
|
||||
) -> PlaygroundSessionResponse | None:
|
||||
result = await db.execute(
|
||||
select(PlaygroundSession).where(
|
||||
PlaygroundSession.user_id == user_id,
|
||||
PlaygroundSession.session_key == session_key,
|
||||
)
|
||||
)
|
||||
record = result.scalar_one_or_none()
|
||||
if record is None:
|
||||
return None
|
||||
return _to_response(record)
|
||||
|
||||
|
||||
async def upsert_playground_session(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
payload: PlaygroundSessionUpsertRequest,
|
||||
) -> PlaygroundSessionResponse:
|
||||
result = await db.execute(
|
||||
select(PlaygroundSession).where(
|
||||
PlaygroundSession.user_id == user_id,
|
||||
PlaygroundSession.session_key == payload.session_key,
|
||||
)
|
||||
)
|
||||
record = result.scalar_one_or_none()
|
||||
title = (payload.title or payload.state.title or "Playground 会话").strip()[:200] or "Playground 会话"
|
||||
|
||||
if record is None:
|
||||
record = PlaygroundSession(
|
||||
user_id=user_id,
|
||||
session_key=payload.session_key,
|
||||
title=title,
|
||||
state=payload.state.model_dump(mode="json"),
|
||||
)
|
||||
db.add(record)
|
||||
else:
|
||||
record.title = title
|
||||
record.state = payload.state.model_dump(mode="json")
|
||||
|
||||
await db.flush()
|
||||
await db.refresh(record)
|
||||
return _to_response(record)
|
||||
@@ -1,7 +1,6 @@
|
||||
"""Task Scheduler for running collection jobs."""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
@@ -9,16 +8,41 @@ from apscheduler.schedulers.asyncio import AsyncIOScheduler
|
||||
from apscheduler.triggers.interval import IntervalTrigger
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.core.logging import get_logger
|
||||
from app.db.session import async_session_factory
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.models.datasource import DataSource
|
||||
from app.models.task import CollectionTask
|
||||
from app.services.collectors.registry import collector_registry
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
logger = get_logger(__name__)
|
||||
|
||||
scheduler = AsyncIOScheduler()
|
||||
RUNNING_TASK_GUARD_TIMEOUT_MINUTES = 90
|
||||
RUNNING_COLLECTOR_TASKS: dict[str, asyncio.Task[Any]] = {}
|
||||
|
||||
|
||||
def _collector_task_name(collector_name: str) -> str:
|
||||
return f"collector:{collector_name}"
|
||||
|
||||
|
||||
def get_running_collector_task(collector_name: str) -> asyncio.Task[Any] | None:
|
||||
task = RUNNING_COLLECTOR_TASKS.get(collector_name)
|
||||
if task is not None and not task.done():
|
||||
return task
|
||||
|
||||
if task is not None and task.done():
|
||||
RUNNING_COLLECTOR_TASKS.pop(collector_name, None)
|
||||
|
||||
target_name = _collector_task_name(collector_name)
|
||||
for candidate in asyncio.all_tasks():
|
||||
if candidate.done():
|
||||
continue
|
||||
if candidate.get_name() == target_name:
|
||||
RUNNING_COLLECTOR_TASKS[collector_name] = candidate
|
||||
return candidate
|
||||
|
||||
return None
|
||||
|
||||
|
||||
async def _update_next_run_at(datasource: DataSource, session) -> None:
|
||||
@@ -30,7 +54,11 @@ async def _update_next_run_at(datasource: DataSource, session) -> None:
|
||||
async def _apply_datasource_schedule(datasource: DataSource, session) -> None:
|
||||
collector = collector_registry.get(datasource.source)
|
||||
if not collector:
|
||||
logger.warning("Collector not found for datasource %s", datasource.source)
|
||||
logger.warning_event(
|
||||
"Collector not found for datasource",
|
||||
event="collector.schedule.collector_missing",
|
||||
context={"collector_name": datasource.source},
|
||||
)
|
||||
return
|
||||
|
||||
collector_registry.set_active(datasource.source, datasource.is_active)
|
||||
@@ -48,13 +76,17 @@ async def _apply_datasource_schedule(datasource: DataSource, session) -> None:
|
||||
replace_existing=True,
|
||||
kwargs={"collector_name": datasource.source},
|
||||
)
|
||||
logger.info(
|
||||
"Scheduled collector: %s (every %sm)",
|
||||
datasource.source,
|
||||
datasource.frequency_minutes,
|
||||
logger.info_event(
|
||||
"Scheduled collector",
|
||||
event="collector.schedule.updated",
|
||||
context={"collector_name": datasource.source, "frequency_minutes": datasource.frequency_minutes},
|
||||
)
|
||||
else:
|
||||
logger.info("Collector disabled: %s", datasource.source)
|
||||
logger.info_event(
|
||||
"Collector disabled",
|
||||
event="collector.schedule.disabled",
|
||||
context={"collector_name": datasource.source},
|
||||
)
|
||||
|
||||
await _update_next_run_at(datasource, session)
|
||||
|
||||
@@ -63,18 +95,30 @@ async def run_collector_task(collector_name: str):
|
||||
"""Run a single collector task."""
|
||||
collector = collector_registry.get(collector_name)
|
||||
if not collector:
|
||||
logger.error("Collector not found: %s", collector_name)
|
||||
logger.error_event(
|
||||
"Collector not found",
|
||||
event="collector.run.collector_missing",
|
||||
context={"collector_name": collector_name},
|
||||
)
|
||||
return
|
||||
|
||||
async with async_session_factory() as db:
|
||||
result = await db.execute(select(DataSource).where(DataSource.source == collector_name))
|
||||
datasource = result.scalar_one_or_none()
|
||||
if not datasource:
|
||||
logger.error("Datasource not found for collector: %s", collector_name)
|
||||
logger.error_event(
|
||||
"Datasource not found for collector",
|
||||
event="collector.run.datasource_missing",
|
||||
context={"collector_name": collector_name},
|
||||
)
|
||||
return
|
||||
|
||||
if not datasource.is_active:
|
||||
logger.info("Skipping disabled collector: %s", collector_name)
|
||||
logger.info_event(
|
||||
"Skipping disabled collector",
|
||||
event="collector.run.skipped_disabled",
|
||||
context={"collector_name": collector_name},
|
||||
)
|
||||
return
|
||||
|
||||
running_result = await db.execute(
|
||||
@@ -98,10 +142,10 @@ async def run_collector_task(collector_name: str):
|
||||
and (now - started_at) > timedelta(minutes=RUNNING_TASK_GUARD_TIMEOUT_MINUTES)
|
||||
)
|
||||
if not is_stale:
|
||||
logger.warning(
|
||||
"Skipping collector %s trigger because task %s is already running",
|
||||
collector_name,
|
||||
existing_running.id,
|
||||
logger.warning_event(
|
||||
"Skipping collector trigger because task is already running",
|
||||
event="collector.run.skipped_already_running",
|
||||
context={"collector_name": collector_name, "task_id": existing_running.id},
|
||||
)
|
||||
return
|
||||
|
||||
@@ -119,25 +163,47 @@ async def run_collector_task(collector_name: str):
|
||||
else stale_reason
|
||||
)
|
||||
await db.commit()
|
||||
logger.warning(
|
||||
"Marked stale running task %s as failed before rerun of %s",
|
||||
existing_running.id,
|
||||
collector_name,
|
||||
logger.warning_event(
|
||||
"Marked stale running task as failed before rerun",
|
||||
event="collector.run.stale_task_failed",
|
||||
context={"collector_name": collector_name, "task_id": existing_running.id},
|
||||
)
|
||||
|
||||
try:
|
||||
collector._datasource_id = datasource.id
|
||||
logger.info("Running collector: %s (datasource_id=%s)", collector_name, datasource.id)
|
||||
logger.info_event(
|
||||
"Running collector",
|
||||
event="collector.run.started",
|
||||
context={"collector_name": collector_name, "datasource_id": datasource.id},
|
||||
)
|
||||
task_result = await collector.run(db)
|
||||
datasource.last_run_at = datetime.now(UTC)
|
||||
datasource.last_status = task_result.get("status")
|
||||
await _update_next_run_at(datasource, db)
|
||||
logger.info("Collector %s completed: %s", collector_name, task_result)
|
||||
logger.info_event(
|
||||
"Collector completed",
|
||||
event="collector.run.completed",
|
||||
context={"collector_name": collector_name, "datasource_id": datasource.id, "result": task_result},
|
||||
)
|
||||
except asyncio.CancelledError:
|
||||
datasource.last_run_at = datetime.now(UTC)
|
||||
datasource.last_status = "cancelled"
|
||||
await db.commit()
|
||||
logger.warning_event(
|
||||
"Collector cancelled by operator",
|
||||
event="collector.run.cancelled",
|
||||
context={"collector_name": collector_name, "datasource_id": datasource.id},
|
||||
)
|
||||
raise
|
||||
except Exception as exc:
|
||||
datasource.last_run_at = datetime.now(UTC)
|
||||
datasource.last_status = "failed"
|
||||
await db.commit()
|
||||
logger.exception("Collector %s failed: %s", collector_name, exc)
|
||||
logger.exception_event(
|
||||
"Collector failed",
|
||||
event="collector.run.failed",
|
||||
context={"collector_name": collector_name, "datasource_id": datasource.id, "error": str(exc)},
|
||||
)
|
||||
|
||||
|
||||
async def cleanup_stale_running_tasks(max_age_hours: int = 2) -> int:
|
||||
@@ -164,7 +230,11 @@ async def cleanup_stale_running_tasks(max_age_hours: int = 2) -> int:
|
||||
|
||||
if stale_tasks:
|
||||
await db.commit()
|
||||
logger.warning("Cleaned up %s stale running collection task(s)", len(stale_tasks))
|
||||
logger.warning_event(
|
||||
"Cleaned up stale running collection tasks",
|
||||
event="collector.cleanup.stale_tasks_cleaned",
|
||||
context={"count": len(stale_tasks)},
|
||||
)
|
||||
|
||||
return len(stale_tasks)
|
||||
|
||||
@@ -173,14 +243,14 @@ def start_scheduler() -> None:
|
||||
"""Start the scheduler."""
|
||||
if not scheduler.running:
|
||||
scheduler.start()
|
||||
logger.info("Scheduler started")
|
||||
logger.info_event("Scheduler started", event="scheduler.started")
|
||||
|
||||
|
||||
def stop_scheduler() -> None:
|
||||
"""Stop the scheduler."""
|
||||
if scheduler.running:
|
||||
scheduler.shutdown(wait=False)
|
||||
logger.info("Scheduler stopped")
|
||||
logger.info_event("Scheduler stopped", event="scheduler.stopped")
|
||||
|
||||
|
||||
async def sync_scheduler_with_datasources() -> None:
|
||||
@@ -241,13 +311,56 @@ def run_collector_now(collector_name: str) -> bool:
|
||||
"""Run a collector immediately (not scheduled)."""
|
||||
collector = collector_registry.get(collector_name)
|
||||
if not collector:
|
||||
logger.error("Collector not found: %s", collector_name)
|
||||
logger.error_event(
|
||||
"Collector not found",
|
||||
event="collector.trigger.collector_missing",
|
||||
context={"collector_name": collector_name},
|
||||
)
|
||||
return False
|
||||
|
||||
existing_task = get_running_collector_task(collector_name)
|
||||
if existing_task is not None and not existing_task.done():
|
||||
logger.warning_event(
|
||||
"Collector is already running in-memory; skipping duplicate trigger",
|
||||
event="collector.trigger.skipped_already_running",
|
||||
context={"collector_name": collector_name},
|
||||
)
|
||||
return False
|
||||
|
||||
try:
|
||||
asyncio.create_task(run_collector_task(collector_name))
|
||||
logger.info("Triggered collector: %s", collector_name)
|
||||
task = asyncio.create_task(run_collector_task(collector_name), name=_collector_task_name(collector_name))
|
||||
RUNNING_COLLECTOR_TASKS[collector_name] = task
|
||||
|
||||
def _cleanup_task(done_task: asyncio.Task[Any]) -> None:
|
||||
current = RUNNING_COLLECTOR_TASKS.get(collector_name)
|
||||
if current is done_task:
|
||||
RUNNING_COLLECTOR_TASKS.pop(collector_name, None)
|
||||
|
||||
task.add_done_callback(_cleanup_task)
|
||||
logger.info_event(
|
||||
"Triggered collector",
|
||||
event="collector.trigger.started",
|
||||
context={"collector_name": collector_name},
|
||||
)
|
||||
return True
|
||||
except Exception as exc:
|
||||
logger.error("Failed to trigger collector %s: %s", collector_name, exc)
|
||||
return False
|
||||
logger.error_event(
|
||||
"Failed to trigger collector",
|
||||
event="collector.trigger.failed",
|
||||
context={"collector_name": collector_name, "error": str(exc)},
|
||||
)
|
||||
return False
|
||||
|
||||
|
||||
async def cancel_running_collector_now(collector_name: str) -> bool:
|
||||
task = get_running_collector_task(collector_name)
|
||||
if task is None or task.done():
|
||||
RUNNING_COLLECTOR_TASKS.pop(collector_name, None)
|
||||
return False
|
||||
|
||||
task.cancel()
|
||||
try:
|
||||
await task
|
||||
except asyncio.CancelledError:
|
||||
return True
|
||||
return task.cancelled()
|
||||
@@ -0,0 +1,174 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections import Counter
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.models.alert import Alert, AlertSeverity, AlertStatus
|
||||
from app.models.bgp_anomaly import BGPAnomaly
|
||||
from app.models.bgp_incident import BGPIncident
|
||||
from app.schemas.ai import SituationalAnalysisRequest
|
||||
from app.services.bgp_ai_brief_store import get_latest_bgp_brief_record
|
||||
|
||||
|
||||
def _format_pairs(pairs: list[tuple[str, int]], empty_text: str = "无") -> str:
|
||||
if not pairs:
|
||||
return empty_text
|
||||
return ",".join(f"{key} {value}" for key, value in pairs if key)
|
||||
|
||||
|
||||
async def build_situational_alert_brief_request(
|
||||
db: AsyncSession,
|
||||
) -> tuple[SituationalAnalysisRequest, list[str], dict[str, Any]]:
|
||||
total_alerts_result = await db.execute(select(func.count(Alert.id)))
|
||||
active_alerts_result = await db.execute(
|
||||
select(func.count(Alert.id)).where(Alert.status == AlertStatus.ACTIVE)
|
||||
)
|
||||
alert_severity_result = await db.execute(
|
||||
select(Alert.severity, func.count(Alert.id))
|
||||
.where(Alert.status == AlertStatus.ACTIVE)
|
||||
.group_by(Alert.severity)
|
||||
)
|
||||
alert_source_result = await db.execute(
|
||||
select(Alert.datasource_name, func.count(Alert.id))
|
||||
.where(Alert.status == AlertStatus.ACTIVE)
|
||||
.group_by(Alert.datasource_name)
|
||||
.order_by(func.count(Alert.id).desc())
|
||||
.limit(6)
|
||||
)
|
||||
recent_alerts_result = await db.execute(
|
||||
select(Alert)
|
||||
.order_by(Alert.created_at.desc(), Alert.id.desc())
|
||||
.limit(6)
|
||||
)
|
||||
|
||||
total_incidents_result = await db.execute(select(func.count(BGPIncident.id)))
|
||||
active_incidents_result = await db.execute(
|
||||
select(func.count(BGPIncident.id)).where(BGPIncident.status == "active")
|
||||
)
|
||||
bgp_severity_result = await db.execute(
|
||||
select(BGPIncident.severity, func.count(BGPIncident.id))
|
||||
.where(BGPIncident.status == "active")
|
||||
.group_by(BGPIncident.severity)
|
||||
)
|
||||
bgp_region_counter: Counter[str] = Counter()
|
||||
recent_incidents_result = await db.execute(
|
||||
select(BGPIncident)
|
||||
.order_by(BGPIncident.created_at.desc(), BGPIncident.id.desc())
|
||||
.limit(5)
|
||||
)
|
||||
|
||||
total_anomalies_result = await db.execute(select(func.count(BGPAnomaly.id)))
|
||||
active_anomalies_result = await db.execute(
|
||||
select(func.count(BGPAnomaly.id)).where(BGPAnomaly.status == "active")
|
||||
)
|
||||
anomaly_type_result = await db.execute(
|
||||
select(BGPAnomaly.anomaly_type, func.count(BGPAnomaly.id))
|
||||
.where(BGPAnomaly.status == "active")
|
||||
.group_by(BGPAnomaly.anomaly_type)
|
||||
.order_by(func.count(BGPAnomaly.id).desc())
|
||||
.limit(6)
|
||||
)
|
||||
|
||||
recent_incidents = recent_incidents_result.scalars().all()
|
||||
for incident in recent_incidents:
|
||||
for region in incident.affected_regions or []:
|
||||
if not isinstance(region, dict):
|
||||
continue
|
||||
label = ", ".join(part for part in [region.get("city"), region.get("country")] if part) or "未知区域"
|
||||
bgp_region_counter[label] += 1
|
||||
|
||||
latest_bgp_brief = get_latest_bgp_brief_record()
|
||||
active_alert_severities = [
|
||||
(item[0].value if isinstance(item[0], AlertSeverity) else str(item[0]), item[1])
|
||||
for item in alert_severity_result.fetchall()
|
||||
if item[0]
|
||||
]
|
||||
active_bgp_severities = [
|
||||
(str(item[0]), item[1])
|
||||
for item in bgp_severity_result.fetchall()
|
||||
if item[0]
|
||||
]
|
||||
active_anomaly_types = [(str(item[0]), item[1]) for item in anomaly_type_result.fetchall() if item[0]]
|
||||
active_alert_sources = [
|
||||
(str(item[0] or "未命名数据源"), item[1])
|
||||
for item in alert_source_result.fetchall()
|
||||
]
|
||||
|
||||
facts = [
|
||||
(
|
||||
f"系统告警侧:总告警 {total_alerts_result.scalar() or 0} 条,active {active_alerts_result.scalar() or 0} 条;"
|
||||
f"活跃告警严重度分布为 {_format_pairs(active_alert_severities)}。"
|
||||
),
|
||||
(
|
||||
f"BGP态势侧:累计 incidents {total_incidents_result.scalar() or 0} 条,active incidents {active_incidents_result.scalar() or 0} 条;"
|
||||
f"活跃 incidents 严重度分布为 {_format_pairs(active_bgp_severities)}。"
|
||||
),
|
||||
(
|
||||
f"BGP异常侧:累计 anomalies {total_anomalies_result.scalar() or 0} 条,active anomalies {active_anomalies_result.scalar() or 0} 条;"
|
||||
f"活跃 anomaly 类型分布为 {_format_pairs(active_anomaly_types)}。"
|
||||
),
|
||||
]
|
||||
|
||||
if active_alert_sources:
|
||||
facts.append(f"当前系统告警主要集中在:{_format_pairs(active_alert_sources)}。")
|
||||
if bgp_region_counter:
|
||||
facts.append(f"BGP近期高风险区域线索:{_format_pairs(bgp_region_counter.most_common(5))}。")
|
||||
|
||||
recent_alerts = recent_alerts_result.scalars().all()
|
||||
if recent_alerts:
|
||||
facts.append(
|
||||
"最近系统告警摘录:"
|
||||
+ ";".join(
|
||||
[
|
||||
f"{alert.datasource_name or '未命名数据源'} / {alert.severity.value if alert.severity else '-'} / {alert.status.value if alert.status else '-'} / {alert.message or '-'}"
|
||||
for alert in recent_alerts
|
||||
]
|
||||
)
|
||||
)
|
||||
|
||||
if recent_incidents:
|
||||
facts.append(
|
||||
"最近BGP事件摘录:"
|
||||
+ ";".join(
|
||||
[
|
||||
f"{incident.incident_type} / {incident.severity} / {incident.status} / {incident.summary}"
|
||||
for incident in recent_incidents
|
||||
]
|
||||
)
|
||||
)
|
||||
|
||||
if latest_bgp_brief:
|
||||
facts.append(
|
||||
f"最近一份 BGP AI 简报生成于 {latest_bgp_brief.generated_at},模型 {latest_bgp_brief.model},可作为当前态势的补充说明。"
|
||||
)
|
||||
|
||||
context = {
|
||||
"source": "situational-alerts",
|
||||
"active_system_alerts": active_alerts_result.scalar() or 0,
|
||||
"active_system_alert_severities": dict(active_alert_severities),
|
||||
"top_system_alert_sources": dict(active_alert_sources),
|
||||
"active_bgp_incidents": active_incidents_result.scalar() or 0,
|
||||
"active_bgp_incident_severities": dict(active_bgp_severities),
|
||||
"active_bgp_anomalies": active_anomalies_result.scalar() or 0,
|
||||
"active_bgp_anomaly_types": dict(active_anomaly_types),
|
||||
"bgp_hot_regions": dict(bgp_region_counter.most_common(5)),
|
||||
"latest_bgp_brief_id": latest_bgp_brief.id if latest_bgp_brief else None,
|
||||
"latest_bgp_brief_generated_at": latest_bgp_brief.generated_at if latest_bgp_brief else None,
|
||||
}
|
||||
|
||||
request = SituationalAnalysisRequest(
|
||||
title="态势告警 AI 简报",
|
||||
objective="综合系统告警、BGP incidents、BGP anomalies 与近期 BGP AI 简报,生成一份面向值班人员的态势告警简报,指出当前最需要关注的风险域、跨模块联动迹象和优先动作。",
|
||||
observations=facts,
|
||||
constraints=[
|
||||
"明确区分事实、推断与建议。",
|
||||
"优先指出仍在 active 状态的系统告警与 BGP 风险是否存在联动。",
|
||||
"不要把单一数据源的局部异常夸大成全局态势。",
|
||||
"如果证据不足,请明确写出仍缺哪些模块或区域信息。",
|
||||
],
|
||||
context=context,
|
||||
)
|
||||
return request, facts, context
|
||||
@@ -0,0 +1,532 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
|
||||
from collections import Counter, deque
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from app.core.security import redis_client
|
||||
|
||||
DEFAULT_LOG_LINE_LIMIT = 200
|
||||
MAX_LOG_LINE_LIMIT = 1000
|
||||
BUFFER_LOG_LIMIT = 1000
|
||||
BUFFER_LOG_TTL_SECONDS = 7 * 24 * 60 * 60
|
||||
LOG_BUFFER_KEY_PREFIX = "planet:system_logs"
|
||||
|
||||
LOG_LEVEL_ERROR = "error"
|
||||
LOG_LEVEL_WARNING = "warning"
|
||||
LOG_LEVEL_INFO = "info"
|
||||
LOG_LEVEL_DEBUG = "debug"
|
||||
LOG_LEVEL_ALL = "all"
|
||||
|
||||
SUPPORTED_LOG_LEVELS = {
|
||||
LOG_LEVEL_ALL,
|
||||
LOG_LEVEL_ERROR,
|
||||
LOG_LEVEL_WARNING,
|
||||
LOG_LEVEL_INFO,
|
||||
LOG_LEVEL_DEBUG,
|
||||
}
|
||||
|
||||
LOG_LEVEL_ALIASES = {
|
||||
"warn": LOG_LEVEL_WARNING,
|
||||
"warning": LOG_LEVEL_WARNING,
|
||||
"err": LOG_LEVEL_ERROR,
|
||||
"error": LOG_LEVEL_ERROR,
|
||||
"info": LOG_LEVEL_INFO,
|
||||
"information": LOG_LEVEL_INFO,
|
||||
"debug": LOG_LEVEL_DEBUG,
|
||||
"trace": LOG_LEVEL_DEBUG,
|
||||
"critical": LOG_LEVEL_ERROR,
|
||||
"fatal": LOG_LEVEL_ERROR,
|
||||
}
|
||||
|
||||
TIMESTAMP_FORMATS = (
|
||||
"%Y-%m-%d %H:%M:%S.%f",
|
||||
"%Y-%m-%d %H:%M:%S",
|
||||
"%Y-%m-%dT%H:%M:%S.%f",
|
||||
"%Y-%m-%dT%H:%M:%S",
|
||||
)
|
||||
|
||||
LEVEL_PATTERNS = (
|
||||
("CRITICAL", LOG_LEVEL_ERROR),
|
||||
("FATAL", LOG_LEVEL_ERROR),
|
||||
("ERROR", LOG_LEVEL_ERROR),
|
||||
("WARNING", LOG_LEVEL_WARNING),
|
||||
("WARN", LOG_LEVEL_WARNING),
|
||||
("INFO", LOG_LEVEL_INFO),
|
||||
("DEBUG", LOG_LEVEL_DEBUG),
|
||||
("TRACE", LOG_LEVEL_DEBUG),
|
||||
)
|
||||
|
||||
LEADING_LEVEL_PATTERN = re.compile(
|
||||
r"^\s*(?:\[[^\]]+\]\s*)?(CRITICAL|FATAL|ERROR|WARNING|WARN|INFO|DEBUG|TRACE)\b[:\s-]*",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
EMBEDDED_LEVEL_PATTERN = re.compile(
|
||||
r"\b(CRITICAL|FATAL|ERROR|WARNING|WARN|INFO|DEBUG|TRACE)\b",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
CONTROL_CHAR_PATTERN = re.compile(r"[\x00-\x08\x0b\x0c\x0e-\x1f\x7f]")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class LogSource:
|
||||
source_id: str
|
||||
name: str
|
||||
kind: str
|
||||
location: str
|
||||
description: str
|
||||
category: str
|
||||
status: str = "ok"
|
||||
buffer_key: str | None = None
|
||||
container_name: str | None = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class StructuredLogEntry:
|
||||
timestamp: datetime | None
|
||||
level: str | None
|
||||
display_line: str
|
||||
raw_line: str
|
||||
search_text: str
|
||||
|
||||
|
||||
@dataclass
|
||||
class DailyLogMarker:
|
||||
date_token: str
|
||||
total: int
|
||||
dominant_level: str
|
||||
|
||||
|
||||
LOG_SOURCES: dict[str, LogSource] = {
|
||||
"backend": LogSource(
|
||||
source_id="backend",
|
||||
name="后端服务",
|
||||
kind="file",
|
||||
location="/tmp/planet_backend.log",
|
||||
description="FastAPI 后端、调度器和采集任务共享日志。",
|
||||
category="service",
|
||||
),
|
||||
"frontend": LogSource(
|
||||
source_id="frontend",
|
||||
name="前端开发服务",
|
||||
kind="file",
|
||||
location="/tmp/planet_frontend.log",
|
||||
description="控制台与 Earth 前端开发服务输出。",
|
||||
category="service",
|
||||
),
|
||||
"ai-provider": LogSource(
|
||||
source_id="ai-provider",
|
||||
name="AI Provider",
|
||||
kind="docker",
|
||||
location="docker://planet_aiprovider",
|
||||
description="AI Provider 容器实时输出日志。",
|
||||
category="service",
|
||||
container_name="planet_aiprovider",
|
||||
),
|
||||
"earth-client": LogSource(
|
||||
source_id="earth-client",
|
||||
name="Earth 浏览器端",
|
||||
kind="buffer",
|
||||
location="redis://planet:system_logs:earth-client",
|
||||
description="Earth 浏览器端上报的运行时错误与关键业务日志。",
|
||||
category="client",
|
||||
buffer_key=f"{LOG_BUFFER_KEY_PREFIX}:earth-client",
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def normalize_log_level(level: str | None) -> str:
|
||||
if level is None:
|
||||
return LOG_LEVEL_ALL
|
||||
normalized = str(level).strip().lower()
|
||||
if normalized in {"", LOG_LEVEL_ALL}:
|
||||
return LOG_LEVEL_ALL
|
||||
return LOG_LEVEL_ALIASES.get(normalized, LOG_LEVEL_ALL)
|
||||
|
||||
|
||||
def normalize_log_levels(level: str | None = None, levels: str | None = None) -> tuple[str, ...]:
|
||||
normalized_levels: list[str] = []
|
||||
if levels:
|
||||
for item in str(levels).split(","):
|
||||
normalized = normalize_log_level(item)
|
||||
if normalized != LOG_LEVEL_ALL and normalized not in normalized_levels:
|
||||
normalized_levels.append(normalized)
|
||||
normalized_level = normalize_log_level(level)
|
||||
if normalized_level != LOG_LEVEL_ALL and normalized_level not in normalized_levels:
|
||||
normalized_levels.append(normalized_level)
|
||||
return tuple(normalized_levels)
|
||||
|
||||
|
||||
def get_source_status(source: LogSource) -> str:
|
||||
if source.kind == "file":
|
||||
path = Path(source.location)
|
||||
if not path.exists():
|
||||
return "missing"
|
||||
return "ok" if path.stat().st_size > 0 else "empty"
|
||||
if source.kind == "docker":
|
||||
return "ok" if shutil.which("docker") else "docker_unavailable"
|
||||
if source.kind == "buffer":
|
||||
if not source.buffer_key:
|
||||
return "source_unavailable"
|
||||
try:
|
||||
return "ok" if redis_client.llen(source.buffer_key) > 0 else "empty"
|
||||
except Exception:
|
||||
return "source_unavailable"
|
||||
return "source_unavailable"
|
||||
|
||||
|
||||
def list_log_sources() -> list[dict[str, str]]:
|
||||
items: list[dict[str, str]] = []
|
||||
for source in LOG_SOURCES.values():
|
||||
items.append(
|
||||
{
|
||||
"source_id": source.source_id,
|
||||
"name": source.name,
|
||||
"kind": source.kind,
|
||||
"location": source.location,
|
||||
"description": source.description,
|
||||
"category": source.category,
|
||||
"status": get_source_status(source),
|
||||
}
|
||||
)
|
||||
return items
|
||||
|
||||
|
||||
def get_buffer_log_key(source_id: str) -> str:
|
||||
return f"{LOG_BUFFER_KEY_PREFIX}:{source_id}"
|
||||
|
||||
|
||||
def append_buffer_log(
|
||||
source_id: str,
|
||||
*,
|
||||
level: str,
|
||||
message: str,
|
||||
context: dict[str, Any] | None = None,
|
||||
) -> None:
|
||||
payload = {
|
||||
"timestamp": datetime.now(tz=UTC).isoformat(),
|
||||
"level": normalize_log_level(level),
|
||||
"message": message,
|
||||
"context": context or {},
|
||||
}
|
||||
buffer_key = get_buffer_log_key(source_id)
|
||||
redis_client.rpush(buffer_key, json.dumps(payload, ensure_ascii=False))
|
||||
redis_client.ltrim(buffer_key, -BUFFER_LOG_LIMIT, -1)
|
||||
redis_client.expire(buffer_key, BUFFER_LOG_TTL_SECONDS)
|
||||
|
||||
|
||||
def parse_timestamp(raw_value: str | None) -> datetime | None:
|
||||
if not raw_value:
|
||||
return None
|
||||
candidate = str(raw_value).strip()
|
||||
if not candidate:
|
||||
return None
|
||||
candidate = candidate.replace("Z", "+00:00")
|
||||
try:
|
||||
parsed = datetime.fromisoformat(candidate)
|
||||
return parsed if parsed.tzinfo else parsed.replace(tzinfo=UTC)
|
||||
except ValueError:
|
||||
pass
|
||||
for fmt in TIMESTAMP_FORMATS:
|
||||
try:
|
||||
return datetime.strptime(candidate, fmt).replace(tzinfo=UTC)
|
||||
except ValueError:
|
||||
continue
|
||||
return None
|
||||
|
||||
|
||||
def parse_prefixed_timestamp(line: str) -> tuple[datetime | None, str]:
|
||||
stripped = line.strip()
|
||||
if not stripped:
|
||||
return None, ""
|
||||
for prefix_length in (35, 32, 29, 26, 23, 19):
|
||||
if len(stripped) < prefix_length:
|
||||
continue
|
||||
prefix = stripped[:prefix_length]
|
||||
timestamp = parse_timestamp(prefix)
|
||||
if timestamp is not None:
|
||||
return timestamp, stripped[prefix_length:].lstrip()
|
||||
first_token = stripped.split(maxsplit=1)[0]
|
||||
timestamp = parse_timestamp(first_token)
|
||||
if timestamp is not None:
|
||||
remainder = stripped[len(first_token):].lstrip()
|
||||
return timestamp, remainder
|
||||
return None, stripped
|
||||
|
||||
|
||||
def infer_log_level_from_text(text: str, *, allow_embedded: bool = True) -> str | None:
|
||||
leading_match = LEADING_LEVEL_PATTERN.match(text)
|
||||
if leading_match:
|
||||
return normalize_log_level(leading_match.group(1))
|
||||
|
||||
if allow_embedded:
|
||||
embedded_match = EMBEDDED_LEVEL_PATTERN.search(text)
|
||||
if embedded_match:
|
||||
return normalize_log_level(embedded_match.group(1))
|
||||
upper_text = text.upper()
|
||||
for pattern, normalized in LEVEL_PATTERNS:
|
||||
if f"{pattern}:" in upper_text or f"{pattern} " in upper_text:
|
||||
return normalized
|
||||
return None
|
||||
|
||||
|
||||
def build_display_line(timestamp: datetime | None, level: str | None, message: str) -> str:
|
||||
message_part = message.strip() if message else ""
|
||||
parts = []
|
||||
if timestamp is not None:
|
||||
parts.append(timestamp.astimezone(UTC).strftime("%Y-%m-%d %H:%M:%S"))
|
||||
if level:
|
||||
parts.append(level.upper())
|
||||
if message_part:
|
||||
parts.append(message_part)
|
||||
return " ".join(parts).strip()
|
||||
|
||||
|
||||
def sanitize_text_log_line(line: str) -> str:
|
||||
return CONTROL_CHAR_PATTERN.sub("", line)
|
||||
|
||||
|
||||
def parse_text_log_entry(line: str) -> StructuredLogEntry:
|
||||
sanitized_line = sanitize_text_log_line(line).rstrip("\n")
|
||||
timestamp, remainder = parse_prefixed_timestamp(sanitized_line)
|
||||
level = infer_log_level_from_text(remainder or sanitized_line, allow_embedded=False)
|
||||
display_line = sanitized_line
|
||||
return StructuredLogEntry(
|
||||
timestamp=timestamp,
|
||||
level=level,
|
||||
display_line=display_line,
|
||||
raw_line=display_line,
|
||||
search_text=display_line.lower(),
|
||||
)
|
||||
|
||||
|
||||
def build_buffer_entry(payload: dict[str, Any]) -> StructuredLogEntry:
|
||||
timestamp = parse_timestamp(str(payload.get("timestamp", "")).strip())
|
||||
level = normalize_log_level(payload.get("level"))
|
||||
if level == LOG_LEVEL_ALL:
|
||||
level = None
|
||||
message = str(payload.get("message", "")).strip()
|
||||
context = payload.get("context")
|
||||
context_map = context if isinstance(context, dict) else {}
|
||||
context_fragments = []
|
||||
for key in ("category", "module", "url", "detail"):
|
||||
value = str(context_map.get(key, "")).strip()
|
||||
if value:
|
||||
context_fragments.append(f"{key}={value}")
|
||||
message_with_context = " | ".join([message, *context_fragments]) if context_fragments else message
|
||||
display_line = build_display_line(timestamp, level, message_with_context)
|
||||
search_text = " ".join(
|
||||
[
|
||||
message,
|
||||
json.dumps(context_map, ensure_ascii=False, sort_keys=True),
|
||||
display_line,
|
||||
]
|
||||
).lower()
|
||||
return StructuredLogEntry(
|
||||
timestamp=timestamp,
|
||||
level=level,
|
||||
display_line=display_line,
|
||||
raw_line=json.dumps(payload, ensure_ascii=False, sort_keys=True),
|
||||
search_text=search_text,
|
||||
)
|
||||
|
||||
|
||||
def read_file_entries(source: LogSource, scan_limit: int) -> list[StructuredLogEntry]:
|
||||
path = Path(source.location)
|
||||
if not path.exists():
|
||||
return []
|
||||
with path.open("r", encoding="utf-8", errors="replace") as handle:
|
||||
recent_lines = deque(handle, maxlen=scan_limit)
|
||||
return [
|
||||
parse_text_log_entry(line)
|
||||
for line in recent_lines
|
||||
if sanitize_text_log_line(line).strip()
|
||||
]
|
||||
|
||||
|
||||
def read_docker_entries(source: LogSource, scan_limit: int) -> list[StructuredLogEntry]:
|
||||
if not shutil.which("docker") or not source.container_name:
|
||||
return []
|
||||
try:
|
||||
completed = subprocess.run(
|
||||
[
|
||||
"docker",
|
||||
"logs",
|
||||
"--timestamps",
|
||||
"--tail",
|
||||
str(scan_limit),
|
||||
source.container_name,
|
||||
],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
except OSError:
|
||||
return []
|
||||
if completed.returncode != 0:
|
||||
return []
|
||||
return [
|
||||
parse_text_log_entry(line)
|
||||
for line in completed.stdout.splitlines()
|
||||
if line.strip()
|
||||
]
|
||||
|
||||
|
||||
def read_buffer_entries(source: LogSource, scan_limit: int) -> list[StructuredLogEntry]:
|
||||
if not source.buffer_key:
|
||||
return []
|
||||
try:
|
||||
raw_items = redis_client.lrange(source.buffer_key, -scan_limit, -1)
|
||||
except Exception:
|
||||
return []
|
||||
entries: list[StructuredLogEntry] = []
|
||||
for raw_item in raw_items:
|
||||
try:
|
||||
payload = json.loads(raw_item)
|
||||
except json.JSONDecodeError:
|
||||
entries.append(parse_text_log_entry(str(raw_item)))
|
||||
continue
|
||||
if isinstance(payload, dict):
|
||||
entries.append(build_buffer_entry(payload))
|
||||
else:
|
||||
entries.append(parse_text_log_entry(str(raw_item)))
|
||||
return entries
|
||||
|
||||
|
||||
def read_source_entries(source: LogSource, scan_limit: int) -> list[StructuredLogEntry]:
|
||||
if source.kind == "file":
|
||||
return read_file_entries(source, scan_limit)
|
||||
if source.kind == "docker":
|
||||
return read_docker_entries(source, scan_limit)
|
||||
if source.kind == "buffer":
|
||||
return read_buffer_entries(source, scan_limit)
|
||||
return []
|
||||
|
||||
|
||||
def matches_levels(entry: StructuredLogEntry, selected_levels: tuple[str, ...]) -> bool:
|
||||
if not selected_levels:
|
||||
return True
|
||||
return entry.level in selected_levels
|
||||
|
||||
|
||||
def matches_date_range(
|
||||
entry: StructuredLogEntry,
|
||||
start_date: str | None,
|
||||
end_date: str | None,
|
||||
) -> bool:
|
||||
if not start_date and not end_date:
|
||||
return True
|
||||
if entry.timestamp is None:
|
||||
return False
|
||||
date_token = entry.timestamp.astimezone(UTC).date().isoformat()
|
||||
if start_date and date_token < start_date:
|
||||
return False
|
||||
if end_date and date_token > end_date:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def matches_search(entry: StructuredLogEntry, search: str | None) -> bool:
|
||||
if search is None:
|
||||
return True
|
||||
query = search.strip().lower()
|
||||
if not query:
|
||||
return True
|
||||
return query in entry.search_text
|
||||
|
||||
|
||||
def build_daily_log_markers(entries: list[StructuredLogEntry]) -> list[dict[str, Any]]:
|
||||
grouped: dict[str, list[StructuredLogEntry]] = {}
|
||||
for entry in entries:
|
||||
if entry.timestamp is None:
|
||||
continue
|
||||
date_token = entry.timestamp.astimezone(UTC).date().isoformat()
|
||||
grouped.setdefault(date_token, []).append(entry)
|
||||
|
||||
markers: list[DailyLogMarker] = []
|
||||
for date_token, group in sorted(grouped.items()):
|
||||
level_counts = Counter(
|
||||
entry.level
|
||||
for entry in group
|
||||
if entry.level in SUPPORTED_LOG_LEVELS and entry.level != LOG_LEVEL_ALL
|
||||
)
|
||||
dominant_level = LOG_LEVEL_INFO
|
||||
if level_counts:
|
||||
dominant_level = sorted(
|
||||
level_counts.items(),
|
||||
key=lambda item: (
|
||||
-item[1],
|
||||
("error", "warning", "info", "debug").index(item[0]),
|
||||
),
|
||||
)[0][0]
|
||||
markers.append(
|
||||
DailyLogMarker(
|
||||
date_token=date_token,
|
||||
total=len(group),
|
||||
dominant_level=dominant_level,
|
||||
)
|
||||
)
|
||||
return [marker.__dict__ for marker in markers]
|
||||
|
||||
|
||||
def read_log_snapshot(
|
||||
source_id: str,
|
||||
limit: int,
|
||||
*,
|
||||
level: str = LOG_LEVEL_ALL,
|
||||
levels: str | None = None,
|
||||
start_date: str | None = None,
|
||||
end_date: str | None = None,
|
||||
search: str | None = None,
|
||||
) -> dict[str, Any] | None:
|
||||
source = LOG_SOURCES.get(source_id)
|
||||
if source is None:
|
||||
return None
|
||||
|
||||
selected_levels = normalize_log_levels(level, levels)
|
||||
search_query = (search or "").strip()
|
||||
scan_limit = max(min(MAX_LOG_LINE_LIMIT * 5, 5000), limit * 5, BUFFER_LOG_LIMIT if source.kind == "buffer" else 1000)
|
||||
all_entries = read_source_entries(source, scan_limit)
|
||||
marker_entries = [
|
||||
entry
|
||||
for entry in all_entries
|
||||
if matches_levels(entry, selected_levels) and matches_search(entry, search_query)
|
||||
]
|
||||
filtered_entries = [
|
||||
entry
|
||||
for entry in marker_entries
|
||||
if matches_date_range(entry, start_date, end_date)
|
||||
]
|
||||
visible_entries = filtered_entries[-limit:]
|
||||
|
||||
compatibility_level = selected_levels[0] if len(selected_levels) == 1 else LOG_LEVEL_ALL
|
||||
return {
|
||||
"source_id": source.source_id,
|
||||
"name": source.name,
|
||||
"kind": source.kind,
|
||||
"location": source.location,
|
||||
"description": source.description,
|
||||
"category": source.category,
|
||||
"status": get_source_status(source),
|
||||
"level": compatibility_level,
|
||||
"selected_levels": list(selected_levels),
|
||||
"search_query": search_query,
|
||||
"available_levels": [
|
||||
LOG_LEVEL_ALL,
|
||||
LOG_LEVEL_ERROR,
|
||||
LOG_LEVEL_WARNING,
|
||||
LOG_LEVEL_INFO,
|
||||
LOG_LEVEL_DEBUG,
|
||||
],
|
||||
"daily_markers": build_daily_log_markers(marker_entries),
|
||||
"line_limit": limit,
|
||||
"line_count": len(visible_entries),
|
||||
"lines": [entry.display_line for entry in visible_entries],
|
||||
}
|
||||
@@ -0,0 +1,466 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.models.system_setting import SystemSetting
|
||||
|
||||
DEFAULT_TV_SOURCE_ID = "cgtn-en"
|
||||
TV_SETTINGS_CATEGORY = "tv"
|
||||
TV_LIVE_SOURCE_COLLECTOR = "news_live_streams"
|
||||
TV_LIVE_SOURCE_DATA_TYPE = "news_live_stream"
|
||||
|
||||
DEFAULT_TV_SETTINGS = {
|
||||
"default_source_id": DEFAULT_TV_SOURCE_ID,
|
||||
"auto_fallback": True,
|
||||
"sources": [
|
||||
{
|
||||
"id": "cctv4",
|
||||
"name": "CCTV-4 中文国际",
|
||||
"provider": "CCTV",
|
||||
"region": "China",
|
||||
"language": "zh-CN",
|
||||
"source_type": "hls",
|
||||
"embed_url": "https://tv.cctv.com/live/cctv4/",
|
||||
"stream_url": "https://ldocctvwbcdtxy.liveplay.myqcloud.com/ldocctvwbcd/cdrmldcctv4_1_td.m3u8",
|
||||
"homepage_url": "https://tv.cctv.com/live/cctv4/",
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": True,
|
||||
"sort_order": 10,
|
||||
"collector_source": None,
|
||||
"notes": "默认兜底新闻直播源。优先尝试 CCTV-4 官方 HLS 播放流,若直播放失败则回退到央视官网直播页。",
|
||||
},
|
||||
{
|
||||
"id": "reuters-tv",
|
||||
"name": "Reuters TV",
|
||||
"provider": "Reuters",
|
||||
"region": "Global",
|
||||
"language": "en",
|
||||
"source_type": "hls",
|
||||
"embed_url": "",
|
||||
"stream_url": "https://reuters-reutersnow-1-eu.rakuten.wurl.tv/playlist.m3u8",
|
||||
"homepage_url": "https://www.reuters.com/video/live/",
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": False,
|
||||
"sort_order": 20,
|
||||
"collector_source": None,
|
||||
"notes": "参考 worldmonitor 的默认新闻频道清单,优先作为全球英文新闻直播放源。",
|
||||
},
|
||||
{
|
||||
"id": "cgtn-en",
|
||||
"name": "CGTN English",
|
||||
"provider": "CGTN",
|
||||
"region": "Global",
|
||||
"language": "en",
|
||||
"source_type": "youtube",
|
||||
"embed_url": "https://www.youtube.com/watch?v=BOy2xDU1LC8",
|
||||
"stream_url": "https://news.cgtn.com/resource/live/english/cgtn-news.m3u8",
|
||||
"youtube_video_id": "BOy2xDU1LC8",
|
||||
"homepage_url": "https://news.cgtn.com/",
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": False,
|
||||
"sort_order": 30,
|
||||
"collector_source": None,
|
||||
"notes": "优先使用官方 YouTube 直播源,保留 HLS 直播放流作为候选信息。",
|
||||
},
|
||||
{
|
||||
"id": "cgtn-es",
|
||||
"name": "CGTN Espanol",
|
||||
"provider": "CGTN",
|
||||
"region": "Latin America",
|
||||
"language": "es",
|
||||
"source_type": "hls",
|
||||
"embed_url": "",
|
||||
"stream_url": "https://news.cgtn.com/resource/live/espanol/cgtn-e.m3u8",
|
||||
"homepage_url": "https://news.cgtn.com/",
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": False,
|
||||
"sort_order": 40,
|
||||
"collector_source": None,
|
||||
"notes": "西语国际新闻频道,覆盖拉美方向态势。",
|
||||
},
|
||||
{
|
||||
"id": "dw-espanol",
|
||||
"name": "DW Espanol",
|
||||
"provider": "Deutsche Welle",
|
||||
"region": "Europe",
|
||||
"language": "es",
|
||||
"source_type": "hls",
|
||||
"embed_url": "",
|
||||
"stream_url": "https://dwamdstream104.akamaized.net/hls/live/2015530/dwstream104/stream04/streamPlaylist.m3u8",
|
||||
"homepage_url": "https://www.dw.com/es/",
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": False,
|
||||
"sort_order": 50,
|
||||
"collector_source": None,
|
||||
"notes": "来自 worldmonitor 可选频道清单的直播放源。",
|
||||
},
|
||||
{
|
||||
"id": "dw-arabic",
|
||||
"name": "DW Arabic",
|
||||
"provider": "Deutsche Welle",
|
||||
"region": "Middle East",
|
||||
"language": "ar",
|
||||
"source_type": "hls",
|
||||
"embed_url": "",
|
||||
"stream_url": "https://dwamdstream103.akamaized.net/hls/live/2015526/dwstream103/index.m3u8",
|
||||
"homepage_url": "https://www.dw.com/ar/",
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": False,
|
||||
"sort_order": 60,
|
||||
"collector_source": None,
|
||||
"notes": "阿拉伯语新闻流,适合作为中东方向新闻补充源。",
|
||||
},
|
||||
{
|
||||
"id": "aljazeera-mubasher",
|
||||
"name": "Al Jazeera Mubasher",
|
||||
"provider": "Al Jazeera",
|
||||
"region": "Middle East",
|
||||
"language": "ar",
|
||||
"source_type": "hls",
|
||||
"embed_url": "",
|
||||
"stream_url": "https://live-hls-web-ajm.getaj.net/AJM/index.m3u8",
|
||||
"homepage_url": "https://www.aljazeera.net/live",
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": False,
|
||||
"sort_order": 70,
|
||||
"collector_source": None,
|
||||
"notes": "中东实时新闻流,来自 worldmonitor HLS 频道目录。",
|
||||
},
|
||||
{
|
||||
"id": "arirang-news",
|
||||
"name": "Arirang News",
|
||||
"provider": "Arirang",
|
||||
"region": "Korea",
|
||||
"language": "en",
|
||||
"source_type": "hls",
|
||||
"embed_url": "",
|
||||
"stream_url": "https://amdlive-ch01-ctnd-com.akamaized.net/arirang_1ch/smil:arirang_1ch.smil/playlist.m3u8",
|
||||
"homepage_url": "https://www.arirang.com/",
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": False,
|
||||
"sort_order": 80,
|
||||
"collector_source": None,
|
||||
"notes": "东北亚英语新闻源,适合补充韩半岛与东亚视角。",
|
||||
},
|
||||
{
|
||||
"id": "abp-news",
|
||||
"name": "ABP News",
|
||||
"provider": "ABP",
|
||||
"region": "India",
|
||||
"language": "hi",
|
||||
"source_type": "hls",
|
||||
"embed_url": "",
|
||||
"stream_url": "https://abplivetv.pc.cdn.bitgravity.com/httppush/abp_livetv/abp_abpnews/master.m3u8",
|
||||
"homepage_url": "https://news.abplive.com/live-tv",
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": False,
|
||||
"sort_order": 90,
|
||||
"collector_source": None,
|
||||
"notes": "印度新闻直播放源,补充南亚区域视角。",
|
||||
},
|
||||
{
|
||||
"id": "sabc-news",
|
||||
"name": "SABC News",
|
||||
"provider": "SABC",
|
||||
"region": "Africa",
|
||||
"language": "en",
|
||||
"source_type": "hls",
|
||||
"embed_url": "",
|
||||
"stream_url": "https://sabconetanw.cdn.mangomolo.com/news/smil:news.stream.smil/playlist.m3u8",
|
||||
"homepage_url": "https://www.sabcnews.com/sabcnews/",
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": False,
|
||||
"sort_order": 100,
|
||||
"collector_source": None,
|
||||
"notes": "非洲英语新闻源,补充非洲区域新闻覆盖。",
|
||||
},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def _clean_text(value: Any) -> str:
|
||||
if value is None:
|
||||
return ""
|
||||
return str(value).strip()
|
||||
|
||||
|
||||
def _clean_url(value: Any) -> str:
|
||||
text = _clean_text(value)
|
||||
if not text:
|
||||
return ""
|
||||
|
||||
parsed = urlparse(text)
|
||||
if parsed.scheme and parsed.scheme not in {"http", "https"}:
|
||||
return ""
|
||||
if parsed.scheme and not parsed.netloc:
|
||||
return ""
|
||||
return text
|
||||
|
||||
|
||||
def _clean_bool(value: Any, *, default: bool) -> bool:
|
||||
if isinstance(value, bool):
|
||||
return value
|
||||
if value in (None, ""):
|
||||
return default
|
||||
if isinstance(value, str):
|
||||
lowered = value.strip().lower()
|
||||
if lowered in {"1", "true", "yes", "on"}:
|
||||
return True
|
||||
if lowered in {"0", "false", "no", "off"}:
|
||||
return False
|
||||
return bool(value)
|
||||
|
||||
|
||||
def _clean_int(value: Any, *, default: int) -> int:
|
||||
try:
|
||||
return int(value)
|
||||
except (TypeError, ValueError):
|
||||
return default
|
||||
|
||||
|
||||
def normalize_tv_source(source: dict[str, Any] | None, *, index: int = 0) -> dict[str, Any]:
|
||||
payload = dict(source or {})
|
||||
source_id = _clean_text(payload.get("id")) or f"tv-source-{index + 1}"
|
||||
source_type = _clean_text(payload.get("source_type")).lower()
|
||||
youtube_video_id = _clean_text(payload.get("youtube_video_id"))
|
||||
youtube_channel = _clean_text(payload.get("youtube_channel"))
|
||||
if source_type not in {"iframe", "hls", "video", "external", "youtube"}:
|
||||
if youtube_video_id or youtube_channel:
|
||||
source_type = "youtube"
|
||||
else:
|
||||
source_type = "iframe" if _clean_text(payload.get("embed_url")) else "external"
|
||||
|
||||
if source_type == "youtube" and not youtube_video_id and not youtube_channel:
|
||||
source_type = "iframe" if _clean_text(payload.get("embed_url")) else "external"
|
||||
|
||||
return {
|
||||
"id": source_id,
|
||||
"name": _clean_text(payload.get("name")) or f"新闻直播源 {index + 1}",
|
||||
"provider": _clean_text(payload.get("provider")) or "Unknown",
|
||||
"region": _clean_text(payload.get("region")) or "Global",
|
||||
"language": _clean_text(payload.get("language")) or "und",
|
||||
"source_type": source_type,
|
||||
"embed_url": _clean_url(payload.get("embed_url")),
|
||||
"stream_url": _clean_url(payload.get("stream_url")),
|
||||
"homepage_url": _clean_url(payload.get("homepage_url")),
|
||||
"poster_url": _clean_url(payload.get("poster_url")),
|
||||
"youtube_video_id": youtube_video_id,
|
||||
"youtube_channel": youtube_channel,
|
||||
"is_enabled": _clean_bool(payload.get("is_enabled"), default=True),
|
||||
"is_fallback": _clean_bool(payload.get("is_fallback"), default=False),
|
||||
"sort_order": _clean_int(payload.get("sort_order"), default=(index + 1) * 10),
|
||||
"collector_source": payload.get("collector_source"),
|
||||
"notes": _clean_text(payload.get("notes")),
|
||||
"updated_at": _clean_text(payload.get("updated_at")),
|
||||
}
|
||||
|
||||
|
||||
def normalize_tv_settings(payload: dict[str, Any] | None) -> dict[str, Any]:
|
||||
merged = {
|
||||
"default_source_id": DEFAULT_TV_SETTINGS["default_source_id"],
|
||||
"auto_fallback": DEFAULT_TV_SETTINGS["auto_fallback"],
|
||||
"sources": [],
|
||||
}
|
||||
|
||||
raw_sources = []
|
||||
if isinstance(payload, dict):
|
||||
merged["default_source_id"] = (
|
||||
_clean_text(payload.get("default_source_id")) or merged["default_source_id"]
|
||||
)
|
||||
merged["auto_fallback"] = _clean_bool(
|
||||
payload.get("auto_fallback"),
|
||||
default=DEFAULT_TV_SETTINGS["auto_fallback"],
|
||||
)
|
||||
if isinstance(payload.get("sources"), list):
|
||||
raw_sources = payload["sources"]
|
||||
|
||||
if not raw_sources:
|
||||
raw_sources = DEFAULT_TV_SETTINGS["sources"]
|
||||
|
||||
normalized_sources = [
|
||||
normalize_tv_source(source, index=index)
|
||||
for index, source in enumerate(raw_sources)
|
||||
]
|
||||
|
||||
if not any(source["id"] == DEFAULT_TV_SOURCE_ID for source in normalized_sources):
|
||||
normalized_sources.append(
|
||||
normalize_tv_source(DEFAULT_TV_SETTINGS["sources"][0], index=len(normalized_sources))
|
||||
)
|
||||
|
||||
default_source_exists = any(
|
||||
source["id"] == merged["default_source_id"] and source["is_enabled"]
|
||||
for source in normalized_sources
|
||||
)
|
||||
if not default_source_exists:
|
||||
fallback_source = next(
|
||||
(source for source in normalized_sources if source["is_fallback"] and source["is_enabled"]),
|
||||
None,
|
||||
)
|
||||
first_enabled_source = next(
|
||||
(source for source in normalized_sources if source["is_enabled"]),
|
||||
None,
|
||||
)
|
||||
merged["default_source_id"] = (
|
||||
fallback_source["id"]
|
||||
if fallback_source
|
||||
else first_enabled_source["id"]
|
||||
if first_enabled_source
|
||||
else DEFAULT_TV_SOURCE_ID
|
||||
)
|
||||
|
||||
merged["sources"] = sorted(
|
||||
normalized_sources,
|
||||
key=lambda item: (item["sort_order"], item["name"], item["id"]),
|
||||
)
|
||||
return merged
|
||||
|
||||
|
||||
async def get_tv_settings_payload(db: AsyncSession) -> dict[str, Any]:
|
||||
result = await db.execute(
|
||||
select(SystemSetting).where(SystemSetting.category == TV_SETTINGS_CATEGORY)
|
||||
)
|
||||
record = result.scalar_one_or_none()
|
||||
payload = record.payload if record else None
|
||||
return normalize_tv_settings(payload)
|
||||
|
||||
|
||||
def _build_collected_tv_source(record: CollectedData, index: int) -> dict[str, Any]:
|
||||
metadata = dict(record.extra_data or {})
|
||||
return normalize_tv_source(
|
||||
{
|
||||
"id": metadata.get("id") or record.source_id or record.entity_key,
|
||||
"name": record.name or record.title or metadata.get("name") or f"采集直播源 {index + 1}",
|
||||
"provider": metadata.get("provider") or metadata.get("publisher") or "Collector",
|
||||
"region": metadata.get("region") or metadata.get("country") or "Global",
|
||||
"language": metadata.get("language") or "und",
|
||||
"source_type": metadata.get("source_type") or "iframe",
|
||||
"embed_url": metadata.get("embed_url") or metadata.get("url") or "",
|
||||
"stream_url": metadata.get("stream_url") or "",
|
||||
"homepage_url": metadata.get("homepage_url") or metadata.get("source_url") or "",
|
||||
"poster_url": metadata.get("poster_url") or "",
|
||||
"youtube_video_id": metadata.get("youtube_video_id") or metadata.get("video_id") or "",
|
||||
"youtube_channel": metadata.get("youtube_channel") or metadata.get("channel_handle") or "",
|
||||
"is_enabled": metadata.get("is_enabled", True),
|
||||
"is_fallback": False,
|
||||
"sort_order": metadata.get("sort_order", 200 + index),
|
||||
"collector_source": record.source,
|
||||
"notes": record.description or metadata.get("notes") or "",
|
||||
"updated_at": to_iso8601_utc(record.collected_at or record.reference_date or datetime.now(UTC)),
|
||||
},
|
||||
index=index,
|
||||
)
|
||||
|
||||
|
||||
async def get_collected_tv_sources(db: AsyncSession) -> list[dict[str, Any]]:
|
||||
result = await db.execute(
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source == TV_LIVE_SOURCE_COLLECTOR)
|
||||
.where(CollectedData.data_type == TV_LIVE_SOURCE_DATA_TYPE)
|
||||
.where(CollectedData.is_current.is_(True))
|
||||
.where(CollectedData.is_valid == 1)
|
||||
.order_by(CollectedData.reference_date.desc().nullslast(), CollectedData.id.desc())
|
||||
)
|
||||
rows = result.scalars().all()
|
||||
return [_build_collected_tv_source(record, index) for index, record in enumerate(rows)]
|
||||
|
||||
|
||||
def build_public_tv_payload(
|
||||
settings_payload: dict[str, Any],
|
||||
collected_sources: list[dict[str, Any]],
|
||||
) -> dict[str, Any]:
|
||||
configured_sources = [
|
||||
source for source in settings_payload["sources"] if source["is_enabled"]
|
||||
]
|
||||
|
||||
merged_by_id = {source["id"]: source for source in configured_sources}
|
||||
for source in collected_sources:
|
||||
if source["id"] in merged_by_id or not source["is_enabled"]:
|
||||
continue
|
||||
merged_by_id[source["id"]] = source
|
||||
|
||||
available_sources = sorted(
|
||||
merged_by_id.values(),
|
||||
key=lambda item: (item["sort_order"], item["name"], item["id"]),
|
||||
)
|
||||
|
||||
default_source = next(
|
||||
(
|
||||
source
|
||||
for source in available_sources
|
||||
if source["id"] == settings_payload["default_source_id"]
|
||||
),
|
||||
None,
|
||||
)
|
||||
fallback_source = next(
|
||||
(source for source in available_sources if source["is_fallback"]),
|
||||
None,
|
||||
)
|
||||
|
||||
resolved_source = default_source or fallback_source or (available_sources[0] if available_sources else None)
|
||||
latest_updated_at = max(
|
||||
(source.get("updated_at") or "" for source in available_sources),
|
||||
default="",
|
||||
)
|
||||
|
||||
return {
|
||||
"default_source_id": settings_payload["default_source_id"],
|
||||
"auto_fallback": settings_payload["auto_fallback"],
|
||||
"selected_source": resolved_source,
|
||||
"fallback_source": fallback_source,
|
||||
"sources": available_sources,
|
||||
"source_count": len(available_sources),
|
||||
"latest_updated_at": latest_updated_at or to_iso8601_utc(datetime.now(UTC)),
|
||||
"generated_at": to_iso8601_utc(datetime.now(UTC)),
|
||||
}
|
||||
|
||||
|
||||
async def get_public_tv_payload(db: AsyncSession) -> dict[str, Any]:
|
||||
settings_payload = await get_tv_settings_payload(db)
|
||||
collected_sources = await get_collected_tv_sources(db)
|
||||
return build_public_tv_payload(settings_payload, collected_sources)
|
||||
|
||||
|
||||
def _extract_allowed_tv_hosts(sources: list[dict[str, Any]]) -> set[str]:
|
||||
hosts: set[str] = set()
|
||||
for source in sources:
|
||||
for field in ("stream_url", "embed_url", "homepage_url", "youtube_channel"):
|
||||
value = _clean_url(source.get(field))
|
||||
if not value:
|
||||
continue
|
||||
parsed = urlparse(value)
|
||||
if parsed.hostname:
|
||||
hosts.add(parsed.hostname.lower())
|
||||
return hosts
|
||||
|
||||
|
||||
def is_allowed_tv_proxy_url(url: str, sources: list[dict[str, Any]]) -> bool:
|
||||
cleaned = _clean_url(url)
|
||||
if not cleaned:
|
||||
return False
|
||||
|
||||
parsed = urlparse(cleaned)
|
||||
hostname = (parsed.hostname or "").lower()
|
||||
if not hostname:
|
||||
return False
|
||||
|
||||
allowed_hosts = _extract_allowed_tv_hosts(sources)
|
||||
if hostname in allowed_hosts:
|
||||
return True
|
||||
return any(hostname.endswith(f".{allowed_host}") for allowed_host in allowed_hosts)
|
||||
+653
-2
@@ -10,7 +10,12 @@ from app.core.config import settings
|
||||
from app.core.security import create_access_token
|
||||
from app.db.session import get_db
|
||||
from app.models.user import User
|
||||
from app.schemas.ai import AIProviderStatusResponse, SituationalAnalysisResponse
|
||||
from app.schemas.ai import (
|
||||
AIProviderStatusResponse,
|
||||
PlaygroundSessionResponse,
|
||||
PlaygroundSessionState,
|
||||
SituationalAnalysisResponse,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
@@ -30,6 +35,7 @@ async def test_health_check():
|
||||
data = response.json()
|
||||
assert data["status"] == "healthy"
|
||||
assert "version" in data
|
||||
assert response.headers["x-request-id"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@@ -90,6 +96,15 @@ async def test_alerts_without_auth():
|
||||
assert response.status_code == 401
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_datasource_task_status_without_auth():
|
||||
"""Test datasource task-status endpoint requires authentication"""
|
||||
transport = ASGITransport(app=app)
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/api/v1/datasources/1/task-status")
|
||||
assert response.status_code == 401
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_alerts_endpoint_with_auth(auth_headers):
|
||||
"""Test alerts endpoint with authentication"""
|
||||
@@ -147,6 +162,345 @@ async def test_alerts_endpoint_with_auth(auth_headers):
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_system_log_sources_requires_super_admin(auth_headers):
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="testuser",
|
||||
email="test@example.com",
|
||||
password_hash="hashed",
|
||||
role="admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/api/v1/system/logs/sources", headers=auth_headers)
|
||||
assert response.status_code == 403
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_system_log_sources_with_super_admin(auth_headers):
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="root",
|
||||
email="root@example.com",
|
||||
password_hash="hashed",
|
||||
role="super_admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch(
|
||||
"app.api.v1.system_control.list_log_sources",
|
||||
return_value=[
|
||||
{
|
||||
"source_id": "backend",
|
||||
"name": "后端服务",
|
||||
"kind": "file",
|
||||
"location": "/tmp/planet_backend.log",
|
||||
"description": "FastAPI 后端、调度器和采集任务共享日志。",
|
||||
"category": "service",
|
||||
"status": "ok",
|
||||
}
|
||||
],
|
||||
):
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/api/v1/system/logs/sources", headers=auth_headers)
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["items"][0]["source_id"] == "backend"
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_system_log_snapshot_with_super_admin(auth_headers):
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="root",
|
||||
email="root@example.com",
|
||||
password_hash="hashed",
|
||||
role="super_admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch(
|
||||
"app.api.v1.system_control.read_log_snapshot",
|
||||
return_value={
|
||||
"source_id": "backend",
|
||||
"name": "后端服务",
|
||||
"kind": "file",
|
||||
"location": "/tmp/planet_backend.log",
|
||||
"description": "FastAPI 后端、调度器和采集任务共享日志。",
|
||||
"category": "service",
|
||||
"status": "ok",
|
||||
"level": "all",
|
||||
"selected_levels": [],
|
||||
"search_query": "",
|
||||
"available_levels": ["all", "error", "warning", "info", "debug"],
|
||||
"daily_markers": [],
|
||||
"line_limit": 50,
|
||||
"line_count": 2,
|
||||
"lines": ["line 1", "line 2"],
|
||||
},
|
||||
):
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/api/v1/system/logs/backend?limit=50", headers=auth_headers)
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["source_id"] == "backend"
|
||||
assert data["line_count"] == 2
|
||||
assert data["lines"] == ["line 1", "line 2"]
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_system_log_snapshot_supports_level_filter(auth_headers):
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="root",
|
||||
email="root@example.com",
|
||||
password_hash="hashed",
|
||||
role="super_admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch(
|
||||
"app.api.v1.system_control.read_log_snapshot",
|
||||
return_value={
|
||||
"source_id": "backend",
|
||||
"name": "后端服务",
|
||||
"kind": "file",
|
||||
"location": "/tmp/planet_backend.log",
|
||||
"description": "FastAPI 后端、调度器和采集任务共享日志。",
|
||||
"category": "service",
|
||||
"status": "ok",
|
||||
"level": "error",
|
||||
"selected_levels": ["error"],
|
||||
"search_query": "",
|
||||
"available_levels": ["all", "error", "warning", "info", "debug"],
|
||||
"daily_markers": [],
|
||||
"line_limit": 50,
|
||||
"line_count": 1,
|
||||
"lines": ["ERROR: failed"],
|
||||
},
|
||||
):
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/api/v1/system/logs/backend?limit=50&level=error", headers=auth_headers)
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["level"] == "error"
|
||||
assert data["lines"] == ["ERROR: failed"]
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_system_log_snapshot_supports_date_range_filter(auth_headers):
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="root",
|
||||
email="root@example.com",
|
||||
password_hash="hashed",
|
||||
role="super_admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch(
|
||||
"app.api.v1.system_control.read_log_snapshot",
|
||||
return_value={
|
||||
"source_id": "backend",
|
||||
"name": "后端服务",
|
||||
"kind": "file",
|
||||
"location": "/tmp/planet_backend.log",
|
||||
"description": "FastAPI 后端、调度器和采集任务共享日志。",
|
||||
"category": "service",
|
||||
"status": "ok",
|
||||
"level": "all",
|
||||
"selected_levels": [],
|
||||
"search_query": "",
|
||||
"available_levels": ["all", "error", "warning", "info", "debug"],
|
||||
"daily_markers": [],
|
||||
"line_limit": 50,
|
||||
"line_count": 1,
|
||||
"lines": ["2026-04-23 INFO: service started"],
|
||||
},
|
||||
) as mock_read_log_snapshot:
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get(
|
||||
"/api/v1/system/logs/backend?limit=50&start_date=2026-04-20&end_date=2026-04-23",
|
||||
headers=auth_headers,
|
||||
)
|
||||
assert response.status_code == 200
|
||||
mock_read_log_snapshot.assert_called_once_with(
|
||||
"backend",
|
||||
50,
|
||||
level="all",
|
||||
levels=None,
|
||||
start_date="2026-04-20",
|
||||
end_date="2026-04-23",
|
||||
search=None,
|
||||
)
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_system_log_snapshot_supports_levels_and_search_filter(auth_headers):
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="root",
|
||||
email="root@example.com",
|
||||
password_hash="hashed",
|
||||
role="super_admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch(
|
||||
"app.api.v1.system_control.read_log_snapshot",
|
||||
return_value={
|
||||
"source_id": "backend",
|
||||
"name": "后端服务",
|
||||
"kind": "file",
|
||||
"location": "/tmp/planet_backend.log",
|
||||
"description": "FastAPI 后端、调度器和采集任务共享日志。",
|
||||
"category": "service",
|
||||
"status": "ok",
|
||||
"level": "all",
|
||||
"selected_levels": ["error", "warning"],
|
||||
"search_query": "timeout",
|
||||
"available_levels": ["all", "error", "warning", "info", "debug"],
|
||||
"daily_markers": [],
|
||||
"line_limit": 50,
|
||||
"line_count": 1,
|
||||
"lines": ["2026-04-23 10:00:00 ERROR timeout"],
|
||||
},
|
||||
) as mock_read_log_snapshot:
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get(
|
||||
"/api/v1/system/logs/backend?limit=50&levels=error,warning&search=timeout",
|
||||
headers=auth_headers,
|
||||
)
|
||||
assert response.status_code == 200
|
||||
mock_read_log_snapshot.assert_called_once_with(
|
||||
"backend",
|
||||
50,
|
||||
level="all",
|
||||
levels="error,warning",
|
||||
start_date=None,
|
||||
end_date=None,
|
||||
search="timeout",
|
||||
)
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_system_log_snapshot_rejects_invalid_date_range(auth_headers):
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="root",
|
||||
email="root@example.com",
|
||||
password_hash="hashed",
|
||||
role="super_admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get(
|
||||
"/api/v1/system/logs/backend?start_date=2026-04-31",
|
||||
headers=auth_headers,
|
||||
)
|
||||
assert response.status_code == 400
|
||||
assert "start_date must be in YYYY-MM-DD format" in response.json()["detail"]
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ingest_earth_client_log_accepts_public_events():
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch("app.api.v1.system_control.append_buffer_log") as mock_append_buffer_log:
|
||||
with patch("app.api.v1.system_control.record_system_log", new_callable=AsyncMock) as mock_record_system_log:
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.post(
|
||||
"/api/v1/system/logs/earth-client",
|
||||
json={
|
||||
"level": "error",
|
||||
"message": "登陆点加载失败: 登陆点接口返回 HTTP 500",
|
||||
"category": "startup-load",
|
||||
"module": "layer-startup",
|
||||
},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["accepted"] is True
|
||||
assert data["source_id"] == "earth-client"
|
||||
mock_append_buffer_log.assert_called_once()
|
||||
mock_record_system_log.assert_awaited_once()
|
||||
persisted_kwargs = mock_record_system_log.await_args.kwargs
|
||||
assert persisted_kwargs["source"] == "earth-client"
|
||||
assert persisted_kwargs["event"] == "earth.client.runtime_log"
|
||||
assert persisted_kwargs["category"] == "startup-load"
|
||||
assert persisted_kwargs["level"] == "error"
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_request_id_header_is_echoed_when_provided():
|
||||
transport = ASGITransport(app=app)
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/health", headers={"X-Request-ID": "planet-test-request"})
|
||||
assert response.status_code == 200
|
||||
assert response.headers["x-request-id"] == "planet-test-request"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_invalid_token():
|
||||
"""Test that invalid token is rejected"""
|
||||
@@ -165,7 +519,8 @@ async def test_ai_provider_status_with_auth(auth_headers):
|
||||
class _FakeAIProviderClient:
|
||||
async def get_status(self, request_id=None):
|
||||
return AIProviderStatusResponse(
|
||||
provider="openai_compatible",
|
||||
provider="minimax",
|
||||
api="anthropic-messages",
|
||||
enabled=True,
|
||||
configured=True,
|
||||
model="test-model",
|
||||
@@ -193,6 +548,7 @@ async def test_ai_provider_status_with_auth(auth_headers):
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert "provider" in data
|
||||
assert "api" in data
|
||||
assert "configured" in data
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
@@ -207,6 +563,9 @@ async def test_ai_situational_analysis_returns_503_when_disabled(auth_headers):
|
||||
provider="openai_compatible",
|
||||
model="test-model",
|
||||
content="1) 态势摘要: 测试返回",
|
||||
content_blocks=[],
|
||||
text_blocks=["1) 态势摘要: 测试返回"],
|
||||
thinking_blocks=[],
|
||||
raw_response={"id": "mock-response"},
|
||||
)
|
||||
|
||||
@@ -241,5 +600,297 @@ async def test_ai_situational_analysis_returns_503_when_disabled(auth_headers):
|
||||
data = response.json()
|
||||
assert data["provider"] == "openai_compatible"
|
||||
assert data["content"]
|
||||
assert "content_blocks" in data
|
||||
assert "text_blocks" in data
|
||||
assert "thinking_blocks" in data
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_playground_session_with_auth(auth_headers):
|
||||
"""Test playground session restore endpoint."""
|
||||
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="testuser",
|
||||
email="test@example.com",
|
||||
password_hash="hashed",
|
||||
role="admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
async def override_get_db():
|
||||
yield AsyncMock()
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
get_db: override_get_db,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch(
|
||||
"app.api.v1.ai.get_playground_session",
|
||||
new=AsyncMock(
|
||||
return_value=PlaygroundSessionResponse(
|
||||
id="1",
|
||||
session_key="default",
|
||||
title="Playground 会话",
|
||||
state=PlaygroundSessionState(
|
||||
messages=[{"id": "msg-1", "role": "user", "content": "hello"}],
|
||||
title="测试标题",
|
||||
objective="测试目标",
|
||||
),
|
||||
created_at="2026-04-10T00:00:00+00:00",
|
||||
updated_at="2026-04-10T00:00:00+00:00",
|
||||
)
|
||||
),
|
||||
):
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/api/v1/ai/playground/session", headers=auth_headers)
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["session_key"] == "default"
|
||||
assert data["state"]["messages"][0]["content"] == "hello"
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_save_playground_session_with_auth(auth_headers):
|
||||
"""Test playground session save endpoint."""
|
||||
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="testuser",
|
||||
email="test@example.com",
|
||||
password_hash="hashed",
|
||||
role="admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
async def override_get_db():
|
||||
yield AsyncMock()
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
get_db: override_get_db,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch(
|
||||
"app.api.v1.ai.upsert_playground_session",
|
||||
new=AsyncMock(
|
||||
return_value=PlaygroundSessionResponse(
|
||||
id="1",
|
||||
session_key="default",
|
||||
title="测试标题",
|
||||
state=PlaygroundSessionState(
|
||||
messages=[{"id": "msg-1", "role": "user", "content": "hello"}],
|
||||
title="测试标题",
|
||||
objective="测试目标",
|
||||
),
|
||||
created_at="2026-04-10T00:00:00+00:00",
|
||||
updated_at="2026-04-10T00:00:00+00:00",
|
||||
)
|
||||
),
|
||||
):
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.put(
|
||||
"/api/v1/ai/playground/session",
|
||||
headers=auth_headers,
|
||||
json={
|
||||
"session_key": "default",
|
||||
"title": "测试标题",
|
||||
"state": {
|
||||
"messages": [{"id": "msg-1", "role": "user", "content": "hello"}],
|
||||
"selectedPresetKey": "bgp-brief",
|
||||
"title": "测试标题",
|
||||
"objective": "测试目标",
|
||||
"constraints": "",
|
||||
"inputValue": "",
|
||||
"analysis": None,
|
||||
"latestAnalysisMessageId": None,
|
||||
"analysisMeta": {},
|
||||
"helpExpanded": True,
|
||||
},
|
||||
},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["title"] == "测试标题"
|
||||
assert data["state"]["objective"] == "测试目标"
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ai_bgp_brief_endpoint_persists_fact_snapshot(auth_headers):
|
||||
class _FakeAIProviderClient:
|
||||
async def analyze(self, _payload, request_id=None):
|
||||
return SituationalAnalysisResponse(
|
||||
provider="minimax",
|
||||
model="MiniMax-M2.5",
|
||||
content="# BGP AI 简报\n\n事实摘要:测试",
|
||||
content_blocks=[],
|
||||
text_blocks=["# BGP AI 简报\n\n事实摘要:测试"],
|
||||
thinking_blocks=[],
|
||||
raw_response={"id": "mock-bgp-brief"},
|
||||
)
|
||||
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="testuser",
|
||||
email="test@example.com",
|
||||
password_hash="hashed",
|
||||
role="admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
async def override_get_db():
|
||||
yield AsyncMock()
|
||||
|
||||
async def _fake_build_bgp_brief_request(_db, **_kwargs):
|
||||
request_payload = __import__("app.schemas.ai", fromlist=["SituationalAnalysisRequest"]).SituationalAnalysisRequest(
|
||||
title="BGP 态势 AI 简报",
|
||||
objective="生成值班简报",
|
||||
observations=["事实A", "事实B"],
|
||||
constraints=["不要编造"],
|
||||
context={"incident_total": 2, "active_collectors": 3},
|
||||
)
|
||||
return request_payload, ["事实A", "事实B"], {"incident_total": 2, "active_collectors": 3}
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
__import__("app.services.ai_client", fromlist=["get_ai_provider_client"]).get_ai_provider_client: lambda: _FakeAIProviderClient(),
|
||||
get_db: override_get_db,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch("app.api.v1.ai.build_bgp_brief_request", side_effect=_fake_build_bgp_brief_request):
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.post("/api/v1/ai/bgp/brief", headers=auth_headers, json={})
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["facts"] == ["事实A", "事实B"]
|
||||
assert data["context"]["incident_total"] == 2
|
||||
assert data["content_markdown"]
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ai_alert_brief_endpoint_with_auth(auth_headers):
|
||||
class _FakeAIProviderClient:
|
||||
async def analyze(self, _payload, request_id=None):
|
||||
return SituationalAnalysisResponse(
|
||||
provider="minimax",
|
||||
model="MiniMax-M2.7",
|
||||
content="事实摘要:告警测试。风险研判:告警测试。建议动作:告警测试。",
|
||||
content_blocks=[],
|
||||
text_blocks=["事实摘要:告警测试。风险研判:告警测试。建议动作:告警测试。"],
|
||||
thinking_blocks=[],
|
||||
raw_response={"id": "mock-alert-brief"},
|
||||
)
|
||||
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="testuser",
|
||||
email="test@example.com",
|
||||
password_hash="hashed",
|
||||
role="admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
async def override_get_db():
|
||||
yield AsyncMock()
|
||||
|
||||
async def _fake_build_alert_brief_request(_db, **_kwargs):
|
||||
request_payload = __import__("app.schemas.ai", fromlist=["SituationalAnalysisRequest"]).SituationalAnalysisRequest(
|
||||
title="告警态势 AI 简报",
|
||||
objective="输出告警简报",
|
||||
observations=["告警事实A", "告警事实B"],
|
||||
constraints=["不要编造"],
|
||||
context={"active_alerts": 3, "top_datasources": {"bgp": 2}},
|
||||
)
|
||||
return request_payload, ["告警事实A", "告警事实B"], {"active_alerts": 3, "top_datasources": {"bgp": 2}}
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
__import__("app.services.ai_client", fromlist=["get_ai_provider_client"]).get_ai_provider_client: lambda: _FakeAIProviderClient(),
|
||||
get_db: override_get_db,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch("app.api.v1.ai.build_alert_brief_request", side_effect=_fake_build_alert_brief_request):
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.post("/api/v1/ai/alerts/brief", headers=auth_headers, json={})
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["title"] == "告警态势 AI 简报"
|
||||
assert data["facts"] == ["告警事实A", "告警事实B"]
|
||||
assert data["context"]["active_alerts"] == 3
|
||||
assert data["content"]
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ai_situational_alert_brief_endpoint_with_auth(auth_headers):
|
||||
class _FakeAIProviderClient:
|
||||
async def analyze(self, _payload, request_id=None):
|
||||
return SituationalAnalysisResponse(
|
||||
provider="minimax",
|
||||
model="MiniMax-M2.7",
|
||||
content="事实摘要:态势测试。风险研判:态势测试。建议动作:态势测试。",
|
||||
content_blocks=[],
|
||||
text_blocks=["事实摘要:态势测试。风险研判:态势测试。建议动作:态势测试。"],
|
||||
thinking_blocks=[],
|
||||
raw_response={"id": "mock-situational-brief"},
|
||||
)
|
||||
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="testuser",
|
||||
email="test@example.com",
|
||||
password_hash="hashed",
|
||||
role="admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
async def override_get_db():
|
||||
yield AsyncMock()
|
||||
|
||||
async def _fake_build_situational_alert_brief_request(_db):
|
||||
request_payload = __import__("app.schemas.ai", fromlist=["SituationalAnalysisRequest"]).SituationalAnalysisRequest(
|
||||
title="态势告警 AI 简报",
|
||||
objective="输出态势告警简报",
|
||||
observations=["态势事实A", "态势事实B"],
|
||||
constraints=["不要编造"],
|
||||
context={"active_system_alerts": 2, "active_bgp_incidents": 1},
|
||||
)
|
||||
return request_payload, ["态势事实A", "态势事实B"], {"active_system_alerts": 2, "active_bgp_incidents": 1}
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
__import__("app.services.ai_client", fromlist=["get_ai_provider_client"]).get_ai_provider_client: lambda: _FakeAIProviderClient(),
|
||||
get_db: override_get_db,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch("app.api.v1.ai.build_situational_alert_brief_request", side_effect=_fake_build_situational_alert_brief_request):
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.post("/api/v1/ai/situational-alerts/brief", headers=auth_headers, json={})
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["title"] == "态势告警 AI 简报"
|
||||
assert data["facts"] == ["态势事实A", "态势事实B"]
|
||||
assert data["context"]["active_system_alerts"] == 2
|
||||
assert data["content"]
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
@@ -0,0 +1,49 @@
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from app.services.earth_news import ParsedNewsItem, _serialize_item
|
||||
|
||||
|
||||
def test_serialize_item_includes_region_anchor_for_cruise():
|
||||
item = ParsedNewsItem(
|
||||
id="google-apac:test",
|
||||
title="Example APAC story",
|
||||
summary="Example summary",
|
||||
url="https://example.com/story",
|
||||
source="Example Source",
|
||||
feed_name="Global Monitor / APAC",
|
||||
feed_region="asia-pacific",
|
||||
homepage_url="https://example.com",
|
||||
published_at=datetime(2026, 4, 23, 2, 30, tzinfo=UTC),
|
||||
)
|
||||
|
||||
payload = _serialize_item(item, active_region="asia-pacific")
|
||||
|
||||
assert payload["latitude"] == 1.3521
|
||||
assert payload["longitude"] == 103.8198
|
||||
assert payload["location_label"] == "亚太"
|
||||
assert payload["location_inferred"] is True
|
||||
assert payload["is_focus_match"] is True
|
||||
assert payload["published_at"] == "2026-04-23T02:30:00Z"
|
||||
|
||||
|
||||
def test_serialize_item_falls_back_to_global_anchor():
|
||||
item = ParsedNewsItem(
|
||||
id="custom:test",
|
||||
title="Fallback story",
|
||||
summary="Fallback summary",
|
||||
url="https://example.com/fallback",
|
||||
source="Fallback Source",
|
||||
feed_name="Fallback Feed",
|
||||
feed_region="unknown-region",
|
||||
homepage_url="https://example.com",
|
||||
published_at=None,
|
||||
)
|
||||
|
||||
payload = _serialize_item(item, active_region="americas")
|
||||
|
||||
assert payload["latitude"] == 20.0
|
||||
assert payload["longitude"] == 0.0
|
||||
assert payload["location_label"] == "全球"
|
||||
assert payload["location_inferred"] is True
|
||||
assert payload["is_focus_match"] is False
|
||||
assert payload["published_at"] is None
|
||||
@@ -0,0 +1,78 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
from io import StringIO
|
||||
|
||||
from app.core.logging import PlanetContextFilter, PlanetFormatter, get_logger
|
||||
from app.core.request_context import set_request_id
|
||||
|
||||
|
||||
def _capture_output(callback):
|
||||
stream = StringIO()
|
||||
handler = logging.StreamHandler(stream)
|
||||
handler.setFormatter(PlanetFormatter(datefmt="%Y-%m-%d %H:%M:%S"))
|
||||
handler.addFilter(PlanetContextFilter())
|
||||
|
||||
adapter = get_logger("tests.logging")
|
||||
target_logger = adapter.logger
|
||||
original_handlers = list(target_logger.handlers)
|
||||
original_level = target_logger.level
|
||||
original_propagate = target_logger.propagate
|
||||
|
||||
target_logger.handlers = [handler]
|
||||
target_logger.setLevel(logging.INFO)
|
||||
target_logger.propagate = False
|
||||
|
||||
try:
|
||||
callback(adapter)
|
||||
finally:
|
||||
handler.flush()
|
||||
target_logger.handlers = original_handlers
|
||||
target_logger.setLevel(original_level)
|
||||
target_logger.propagate = original_propagate
|
||||
|
||||
return stream.getvalue()
|
||||
|
||||
|
||||
def test_structured_logger_injects_request_id_and_event():
|
||||
set_request_id("req-test-123")
|
||||
try:
|
||||
output = _capture_output(
|
||||
lambda logger: logger.info_event(
|
||||
"collector started",
|
||||
event="collector.run.started",
|
||||
context={"collector_name": "bgp_news"},
|
||||
)
|
||||
)
|
||||
finally:
|
||||
set_request_id(None)
|
||||
|
||||
assert "request_id=req-test-123" in output
|
||||
assert "event=collector.run.started" in output
|
||||
assert "service=backend" in output
|
||||
assert '"collector_name": "bgp_news"' in output
|
||||
|
||||
|
||||
def test_structured_logger_redacts_sensitive_text_and_context():
|
||||
set_request_id("req-test-redact")
|
||||
try:
|
||||
output = _capture_output(
|
||||
lambda logger: logger.error_event(
|
||||
"Authorization: Bearer super-secret-token",
|
||||
event="auth.token.failed",
|
||||
context={
|
||||
"token": "plain-secret",
|
||||
"nested": {"password": "hunter2"},
|
||||
"safe": "visible",
|
||||
},
|
||||
)
|
||||
)
|
||||
finally:
|
||||
set_request_id(None)
|
||||
|
||||
assert "super-secret-token" not in output
|
||||
assert "plain-secret" not in output
|
||||
assert "hunter2" not in output
|
||||
assert "[REDACTED]" in output
|
||||
assert '"safe": "visible"' in output
|
||||
@@ -0,0 +1,217 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from app.services import system_logs
|
||||
|
||||
|
||||
class FakeRedis:
|
||||
def __init__(self) -> None:
|
||||
self.store: dict[str, list[str]] = {}
|
||||
|
||||
def rpush(self, key: str, value: str) -> None:
|
||||
self.store.setdefault(key, []).append(value)
|
||||
|
||||
def ltrim(self, key: str, start: int, end: int) -> None:
|
||||
items = self.store.get(key, [])
|
||||
normalized_end = None if end == -1 else end + 1
|
||||
self.store[key] = items[start:normalized_end]
|
||||
|
||||
def expire(self, key: str, seconds: int) -> None:
|
||||
return None
|
||||
|
||||
def lrange(self, key: str, start: int, end: int) -> list[str]:
|
||||
items = self.store.get(key, [])
|
||||
normalized_end = None if end == -1 else end + 1
|
||||
return items[start:normalized_end]
|
||||
|
||||
def llen(self, key: str) -> int:
|
||||
return len(self.store.get(key, []))
|
||||
|
||||
|
||||
def test_read_log_snapshot_uses_structured_buffer_timestamp_level_and_search(monkeypatch):
|
||||
fake_redis = FakeRedis()
|
||||
monkeypatch.setattr(system_logs, "redis_client", fake_redis)
|
||||
monkeypatch.setattr(
|
||||
system_logs,
|
||||
"LOG_SOURCES",
|
||||
{
|
||||
"earth-client": system_logs.LogSource(
|
||||
source_id="earth-client",
|
||||
name="Earth 浏览器端",
|
||||
kind="buffer",
|
||||
location="redis://planet:system_logs:earth-client",
|
||||
description="Earth 浏览器端上报日志",
|
||||
category="client",
|
||||
buffer_key=system_logs.get_buffer_log_key("earth-client"),
|
||||
)
|
||||
},
|
||||
)
|
||||
|
||||
fake_redis.rpush(
|
||||
system_logs.get_buffer_log_key("earth-client"),
|
||||
json.dumps(
|
||||
{
|
||||
"timestamp": "2026-04-22T10:15:30Z",
|
||||
"level": "warning",
|
||||
"message": "news feed degraded",
|
||||
"context": {"module": "news", "detail": "timeout"},
|
||||
},
|
||||
ensure_ascii=False,
|
||||
),
|
||||
)
|
||||
fake_redis.rpush(
|
||||
system_logs.get_buffer_log_key("earth-client"),
|
||||
json.dumps(
|
||||
{
|
||||
"timestamp": "2026-04-23T06:01:00Z",
|
||||
"level": "error",
|
||||
"message": "landing points failed",
|
||||
"context": {"module": "layer-startup", "detail": "http 500"},
|
||||
},
|
||||
ensure_ascii=False,
|
||||
),
|
||||
)
|
||||
|
||||
snapshot = system_logs.read_log_snapshot(
|
||||
"earth-client",
|
||||
50,
|
||||
levels="error,warning",
|
||||
start_date="2026-04-23",
|
||||
end_date="2026-04-23",
|
||||
search="landing",
|
||||
)
|
||||
|
||||
assert snapshot is not None
|
||||
assert snapshot["selected_levels"] == ["error", "warning"]
|
||||
assert snapshot["search_query"] == "landing"
|
||||
assert snapshot["line_count"] == 1
|
||||
assert snapshot["lines"][0].startswith("2026-04-23 06:01:00 ERROR landing points failed")
|
||||
assert snapshot["daily_markers"] == [
|
||||
{"date_token": "2026-04-23", "total": 1, "dominant_level": "error"}
|
||||
]
|
||||
|
||||
|
||||
def test_read_log_snapshot_parses_file_timestamp_and_builds_markers(tmp_path: Path, monkeypatch):
|
||||
log_path = tmp_path / "backend.log"
|
||||
log_path.write_text(
|
||||
"\n".join(
|
||||
[
|
||||
"2026-04-22 08:00:00 INFO service booted",
|
||||
"2026-04-23 09:15:00 WARNING disk pressure detected",
|
||||
"2026-04-23 09:16:00 ERROR sync failed",
|
||||
"2026-04-24 10:00:00 DEBUG collector trace",
|
||||
]
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
monkeypatch.setattr(
|
||||
system_logs,
|
||||
"LOG_SOURCES",
|
||||
{
|
||||
"backend": system_logs.LogSource(
|
||||
source_id="backend",
|
||||
name="后端服务",
|
||||
kind="file",
|
||||
location=str(log_path),
|
||||
description="测试文件日志",
|
||||
category="service",
|
||||
)
|
||||
},
|
||||
)
|
||||
|
||||
snapshot = system_logs.read_log_snapshot(
|
||||
"backend",
|
||||
50,
|
||||
levels="warning,error",
|
||||
search="failed",
|
||||
)
|
||||
|
||||
assert snapshot is not None
|
||||
assert snapshot["line_count"] == 1
|
||||
assert snapshot["lines"] == ["2026-04-23 09:16:00 ERROR sync failed"]
|
||||
assert snapshot["daily_markers"] == [
|
||||
{"date_token": "2026-04-23", "total": 1, "dominant_level": "error"}
|
||||
]
|
||||
assert snapshot["status"] == "ok"
|
||||
|
||||
|
||||
def test_append_buffer_log_persists_normalized_level(monkeypatch):
|
||||
fake_redis = FakeRedis()
|
||||
monkeypatch.setattr(system_logs, "redis_client", fake_redis)
|
||||
|
||||
system_logs.append_buffer_log(
|
||||
"earth-client",
|
||||
level="warn",
|
||||
message="feed delayed",
|
||||
context={"module": "news"},
|
||||
)
|
||||
|
||||
stored_items = fake_redis.lrange(system_logs.get_buffer_log_key("earth-client"), 0, -1)
|
||||
payload = json.loads(stored_items[0])
|
||||
assert payload["level"] == "warning"
|
||||
assert payload["message"] == "feed delayed"
|
||||
|
||||
|
||||
def test_infer_log_level_prefers_leading_prefix_over_query_string():
|
||||
line = 'INFO: 127.0.0.1 - "GET /api/v1/system/logs/backend?limit=200&level=error&levels=error HTTP/1.1" 200 OK'
|
||||
|
||||
entry = system_logs.parse_text_log_entry(line)
|
||||
|
||||
assert entry.level == "info"
|
||||
|
||||
|
||||
def test_parse_text_log_entry_does_not_promote_exception_context_to_error():
|
||||
line = "websockets.exceptions.ConnectionClosedError: sent 1011 (internal error) keepalive ping timeout"
|
||||
|
||||
entry = system_logs.parse_text_log_entry(line)
|
||||
|
||||
assert entry.level is None
|
||||
|
||||
|
||||
def test_parse_text_log_entry_still_detects_explicit_error_prefix():
|
||||
line = "ERROR: [Errno 98] Address already in use"
|
||||
|
||||
entry = system_logs.parse_text_log_entry(line)
|
||||
|
||||
assert entry.level == "error"
|
||||
|
||||
|
||||
def test_read_log_snapshot_strips_nul_bytes_from_file_lines(tmp_path: Path, monkeypatch):
|
||||
log_path = tmp_path / "backend.log"
|
||||
log_path.write_bytes(
|
||||
(
|
||||
b"INFO: service booted\n"
|
||||
b"ERROR: bind failed\n"
|
||||
+ b"\x00" * 32
|
||||
+ b"2026-04-23 23:41:32 INFO service=backend message=request served\n"
|
||||
)
|
||||
)
|
||||
|
||||
monkeypatch.setattr(
|
||||
system_logs,
|
||||
"LOG_SOURCES",
|
||||
{
|
||||
"backend": system_logs.LogSource(
|
||||
source_id="backend",
|
||||
name="后端服务",
|
||||
kind="file",
|
||||
location=str(log_path),
|
||||
description="测试文件日志",
|
||||
category="service",
|
||||
)
|
||||
},
|
||||
)
|
||||
|
||||
snapshot = system_logs.read_log_snapshot("backend", 50)
|
||||
|
||||
assert snapshot is not None
|
||||
assert snapshot["line_count"] == 3
|
||||
assert snapshot["lines"] == [
|
||||
"INFO: service booted",
|
||||
"ERROR: bind failed",
|
||||
"2026-04-23 23:41:32 INFO service=backend message=request served",
|
||||
]
|
||||
@@ -0,0 +1,347 @@
|
||||
from datetime import datetime, timezone
|
||||
|
||||
import pytest
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
|
||||
from app.api.v1.visualization import convert_compute_centers_to_geojson
|
||||
from app.db.session import get_db
|
||||
from app.main import app
|
||||
from app.models.collected_data import CollectedData
|
||||
|
||||
|
||||
def _build_record(
|
||||
*,
|
||||
record_id: int,
|
||||
source: str,
|
||||
data_type: str,
|
||||
name: str,
|
||||
country: str,
|
||||
city: str,
|
||||
latitude: float,
|
||||
longitude: float,
|
||||
metadata: dict,
|
||||
):
|
||||
return CollectedData(
|
||||
id=record_id,
|
||||
source=source,
|
||||
data_type=data_type,
|
||||
source_id=f"{source}-{record_id}",
|
||||
name=name,
|
||||
extra_data={
|
||||
"country": country,
|
||||
"city": city,
|
||||
"latitude": latitude,
|
||||
"longitude": longitude,
|
||||
**metadata,
|
||||
},
|
||||
collected_at=datetime(2026, 4, 22, tzinfo=timezone.utc),
|
||||
reference_date=datetime(2026, 4, 21, tzinfo=timezone.utc),
|
||||
is_current=True,
|
||||
)
|
||||
|
||||
|
||||
def test_convert_compute_centers_to_geojson_unifies_sources():
|
||||
top500_record = _build_record(
|
||||
record_id=1,
|
||||
source="top500",
|
||||
data_type="supercomputer",
|
||||
name="Frontier",
|
||||
country="United States",
|
||||
city="Oak Ridge",
|
||||
latitude=35.93,
|
||||
longitude=-84.31,
|
||||
metadata={
|
||||
"rank": 1,
|
||||
"manufacturer": "HPE",
|
||||
"organization": "ORNL",
|
||||
"rmax": 1102000.0,
|
||||
"cores": 8730112,
|
||||
"power": 21510.0,
|
||||
},
|
||||
)
|
||||
gpu_record = _build_record(
|
||||
record_id=2,
|
||||
source="epoch_ai_gpu",
|
||||
data_type="gpu_cluster",
|
||||
name="Colossus",
|
||||
country="United States",
|
||||
city="Memphis",
|
||||
latitude=35.15,
|
||||
longitude=-90.05,
|
||||
metadata={
|
||||
"organization": "xAI",
|
||||
"gpu_type": "H100",
|
||||
"gpu_count": 100000,
|
||||
"value": "20000",
|
||||
"unit": "TFlop/s",
|
||||
},
|
||||
)
|
||||
|
||||
payload = convert_compute_centers_to_geojson([top500_record, gpu_record])
|
||||
|
||||
assert payload["type"] == "FeatureCollection"
|
||||
assert len(payload["features"]) == 2
|
||||
|
||||
supercomputer_feature = payload["features"][0]
|
||||
assert supercomputer_feature["properties"]["site_type"] == "supercomputer"
|
||||
assert supercomputer_feature["properties"]["capacity_unit"] == "GFlops"
|
||||
assert supercomputer_feature["properties"]["capacity_band"] == "exascale"
|
||||
assert supercomputer_feature["properties"]["operator"] == "ORNL"
|
||||
assert supercomputer_feature["properties"]["location_precision"] == "precise"
|
||||
assert supercomputer_feature["properties"]["is_estimated"] is False
|
||||
|
||||
gpu_feature = payload["features"][1]
|
||||
assert gpu_feature["properties"]["site_type"] == "gpu_cluster"
|
||||
assert gpu_feature["properties"]["vendor"] == "H100"
|
||||
assert gpu_feature["properties"]["gpu_count"] == 100000
|
||||
assert gpu_feature["properties"]["capacity_band"] == "large"
|
||||
assert gpu_feature["properties"]["location_precision"] == "precise"
|
||||
|
||||
|
||||
def test_convert_compute_centers_to_geojson_uses_coordinate_hints():
|
||||
hinted_record = _build_record(
|
||||
record_id=3,
|
||||
source="top500",
|
||||
data_type="supercomputer",
|
||||
name="Frontier",
|
||||
country="United States",
|
||||
city="",
|
||||
latitude=0.0,
|
||||
longitude=0.0,
|
||||
metadata={
|
||||
"organization": "Oak Ridge National Laboratory",
|
||||
"rmax": 1102000.0,
|
||||
},
|
||||
)
|
||||
|
||||
payload = convert_compute_centers_to_geojson([hinted_record])
|
||||
|
||||
assert len(payload["features"]) == 1
|
||||
coords = payload["features"][0]["geometry"]["coordinates"]
|
||||
assert coords[0] == pytest.approx(-84.3107)
|
||||
assert coords[1] == pytest.approx(35.9319)
|
||||
assert payload["features"][0]["properties"]["is_estimated"] is True
|
||||
assert payload["features"][0]["properties"]["location_precision"] == "estimated_site"
|
||||
|
||||
|
||||
def test_convert_compute_centers_to_geojson_falls_back_to_country_centroid():
|
||||
centroid_record = _build_record(
|
||||
record_id=4,
|
||||
source="epoch_ai_gpu",
|
||||
data_type="gpu_cluster",
|
||||
name="Unknown Cluster",
|
||||
country="United States",
|
||||
city="",
|
||||
latitude=0.0,
|
||||
longitude=0.0,
|
||||
metadata={
|
||||
"organization": "Unknown Operator",
|
||||
"value": "10000",
|
||||
"unit": "TFlop/s",
|
||||
},
|
||||
)
|
||||
|
||||
payload = convert_compute_centers_to_geojson([centroid_record])
|
||||
|
||||
assert len(payload["features"]) == 1
|
||||
props = payload["features"][0]["properties"]
|
||||
coords = payload["features"][0]["geometry"]["coordinates"]
|
||||
assert coords[0] == pytest.approx(-98.5795)
|
||||
assert coords[1] == pytest.approx(39.8283)
|
||||
assert props["is_estimated"] is True
|
||||
assert props["location_precision"] == "estimated_country"
|
||||
assert props["geography_mode"] == "country_centroid"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_compute_centers_geojson_endpoint_returns_stats():
|
||||
records = [
|
||||
_build_record(
|
||||
record_id=1,
|
||||
source="top500",
|
||||
data_type="supercomputer",
|
||||
name="Frontier",
|
||||
country="United States",
|
||||
city="Oak Ridge",
|
||||
latitude=35.93,
|
||||
longitude=-84.31,
|
||||
metadata={"rank": 1, "rmax": 1102000.0},
|
||||
),
|
||||
_build_record(
|
||||
record_id=2,
|
||||
source="epoch_ai_gpu",
|
||||
data_type="gpu_cluster",
|
||||
name="Colossus",
|
||||
country="United States",
|
||||
city="Memphis",
|
||||
latitude=35.15,
|
||||
longitude=-90.05,
|
||||
metadata={"value": "20000", "unit": "TFlop/s"},
|
||||
),
|
||||
]
|
||||
|
||||
class _ScalarResult:
|
||||
def __init__(self, rows):
|
||||
self._rows = rows
|
||||
|
||||
def scalars(self):
|
||||
class _Scalars:
|
||||
def __init__(self, rows):
|
||||
self._rows = rows
|
||||
|
||||
def all(self):
|
||||
return self._rows
|
||||
|
||||
return _Scalars(self._rows)
|
||||
|
||||
class _FakeSession:
|
||||
async def execute(self, _query):
|
||||
return _ScalarResult(records)
|
||||
|
||||
async def override_get_db():
|
||||
yield _FakeSession()
|
||||
|
||||
app.dependency_overrides[get_db] = override_get_db
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/api/v1/visualization/geo/compute-centers")
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["count"] == 2
|
||||
assert data["stats"]["supercomputers"] == 1
|
||||
assert data["stats"]["gpu_clusters"] == 1
|
||||
assert data["features"][0]["properties"]["data_type"] == "compute_center"
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_visualization_geo_summary_returns_counts(monkeypatch):
|
||||
records = [
|
||||
_build_record(
|
||||
record_id=1,
|
||||
source="arcgis_cables",
|
||||
data_type="submarine_cable",
|
||||
name="Test Cable",
|
||||
country="",
|
||||
city="",
|
||||
latitude=0,
|
||||
longitude=0,
|
||||
metadata={
|
||||
"route_coordinates": [[[0, 0], [1, 1]]],
|
||||
"status": "active",
|
||||
},
|
||||
),
|
||||
_build_record(
|
||||
record_id=2,
|
||||
source="arcgis_landing_points",
|
||||
data_type="landing_point",
|
||||
name="Test Landing",
|
||||
country="United States",
|
||||
city="New York",
|
||||
latitude=40.7,
|
||||
longitude=-74.0,
|
||||
metadata={"city_id": 10},
|
||||
),
|
||||
_build_record(
|
||||
record_id=3,
|
||||
source="celestrak_tle",
|
||||
data_type="satellite_tle",
|
||||
name="TESTSAT",
|
||||
country="",
|
||||
city="",
|
||||
latitude=0,
|
||||
longitude=0,
|
||||
metadata={
|
||||
"norad_cat_id": 12345,
|
||||
"tle_line1": "1 12345U 98067A 24001.00000000 .00000000 00000-0 00000-0 0 9991",
|
||||
"tle_line2": "2 12345 51.6000 100.0000 0001000 10.0000 20.0000 15.50000000 01",
|
||||
},
|
||||
),
|
||||
_build_record(
|
||||
record_id=4,
|
||||
source="top500",
|
||||
data_type="supercomputer",
|
||||
name="Frontier",
|
||||
country="United States",
|
||||
city="Oak Ridge",
|
||||
latitude=35.93,
|
||||
longitude=-84.31,
|
||||
metadata={"rank": 1, "rmax": 1102000.0},
|
||||
),
|
||||
_build_record(
|
||||
record_id=5,
|
||||
source="epoch_ai_gpu",
|
||||
data_type="gpu_cluster",
|
||||
name="Colossus",
|
||||
country="United States",
|
||||
city="Memphis",
|
||||
latitude=35.15,
|
||||
longitude=-90.05,
|
||||
metadata={"value": "20000", "unit": "TFlop/s"},
|
||||
),
|
||||
]
|
||||
|
||||
class _ScalarResult:
|
||||
def __init__(self, rows=None, scalar_value=None):
|
||||
self._rows = rows or []
|
||||
self._scalar_value = scalar_value
|
||||
|
||||
def scalar(self):
|
||||
return self._scalar_value
|
||||
|
||||
def scalars(self):
|
||||
class _Scalars:
|
||||
def __init__(self, rows):
|
||||
self._rows = rows
|
||||
|
||||
def all(self):
|
||||
return self._rows
|
||||
|
||||
return _Scalars(self._rows)
|
||||
|
||||
class _FakeSession:
|
||||
async def execute(self, query):
|
||||
query_text = str(query)
|
||||
if "bgp_incidents" in query_text:
|
||||
return _ScalarResult(scalar_value=2)
|
||||
if "bgp_anomalies" in query_text:
|
||||
return _ScalarResult(scalar_value=3)
|
||||
return _ScalarResult(rows=records)
|
||||
|
||||
async def override_get_db():
|
||||
yield _FakeSession()
|
||||
|
||||
async def _fake_build_bgp_collector_coverage(*_args, **_kwargs):
|
||||
return [
|
||||
{"collector": "rrc00"},
|
||||
{"collector": "rrc01"},
|
||||
]
|
||||
|
||||
monkeypatch.setattr(
|
||||
"app.api.v1.visualization.build_bgp_collector_coverage",
|
||||
_fake_build_bgp_collector_coverage,
|
||||
)
|
||||
|
||||
app.dependency_overrides[get_db] = override_get_db
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/api/v1/visualization/geo/summary")
|
||||
|
||||
assert response.status_code == 200
|
||||
stats = response.json()["stats"]
|
||||
assert stats["cable_count"] == 1
|
||||
assert stats["landing_point_count"] == 1
|
||||
assert stats["satellite_count"] == 1
|
||||
assert stats["compute_center_count"] == 2
|
||||
assert stats["supercomputer_count"] == 1
|
||||
assert stats["gpu_cluster_count"] == 1
|
||||
assert stats["bgp_event_count"] == 2
|
||||
assert stats["bgp_incident_count"] == 2
|
||||
assert stats["bgp_anomaly_count"] == 3
|
||||
assert stats["bgp_collector_count"] == 2
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
@@ -18,6 +18,9 @@ services:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: aiprovider/Dockerfile
|
||||
args:
|
||||
PYTHON_IMAGE: ${PYTHON_IMAGE:-python:3.14-slim}
|
||||
UV_IMAGE: ${UV_IMAGE:-ghcr.io/astral-sh/uv:latest}
|
||||
container_name: planet_aiprovider
|
||||
ports:
|
||||
- "8010:8010"
|
||||
|
||||
@@ -5,6 +5,9 @@ services:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: aiprovider/Dockerfile
|
||||
args:
|
||||
PYTHON_IMAGE: ${PYTHON_IMAGE:-python:3.14-slim}
|
||||
UV_IMAGE: ${UV_IMAGE:-ghcr.io/astral-sh/uv:latest}
|
||||
container_name: planet_aiprovider
|
||||
ports:
|
||||
- "8010:8010"
|
||||
|
||||
@@ -5,6 +5,11 @@ services:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: aiprovider/Dockerfile
|
||||
args:
|
||||
PYTHON_IMAGE: ${PYTHON_IMAGE:-python:3.14-slim}
|
||||
UV_IMAGE: ${UV_IMAGE:-ghcr.io/astral-sh/uv:latest}
|
||||
env_file:
|
||||
- ./aiprovider/.env
|
||||
container_name: planet_aiprovider
|
||||
ports:
|
||||
- "8010:8010"
|
||||
|
||||
+1010
-5
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,22 @@
|
||||
# Deprecated Docs
|
||||
|
||||
这个目录用于存放两类文档:
|
||||
|
||||
1. 已经完成、主要保留为历史记录的实施计划
|
||||
2. 已被现有实现或新方案替代的旧计划
|
||||
|
||||
放到这里并不代表这些文档“错误”,而是表示:
|
||||
|
||||
- 它们不再适合作为当前开发的主指导文档
|
||||
- 如果要了解历史决策、演进路径或旧设计背景,仍然可以参考
|
||||
|
||||
当前归档原则:
|
||||
|
||||
- 明确写明“已完成”的计划,优先归档
|
||||
- 已被正式实现替代、继续放在 `docs/` 根目录会误导后续开发的计划,归档
|
||||
- 仍然指导未来开发、尚未完成或仍有明确执行价值的文档,继续保留在 `docs/`
|
||||
|
||||
补充说明:
|
||||
|
||||
- 一部分归档文档来自外部或临时工作流草案,例如 sisyphus 生成的初稿
|
||||
- 这类文档如果有可用内容,应先吸收到 `docs/plans/` 或 `docs/technical/`,再归档保留来源记录
|
||||
File renamed without changes.
File renamed without changes.
@@ -0,0 +1,117 @@
|
||||
# Earth 电视直播模块计划
|
||||
|
||||
## 目标
|
||||
|
||||
为 `Earth` 页面增加一个可配置、可扩展、可拖拽的电视直播模块:
|
||||
|
||||
- 后台可配置新闻直播源
|
||||
- 默认兜底源为央视 `CCTV-4`
|
||||
- 未来可通过采集器接入世界各地新闻直播源
|
||||
- Earth 工具栏 `显示控制` 子菜单新增电视按钮
|
||||
- 点击后打开一个与其他 HUD 一致的可拖拽/可关闭窗口
|
||||
- 窗口内部可播放或承载新闻直播页面
|
||||
|
||||
## 设计原则
|
||||
|
||||
- 第一阶段先交付“后台可配 + Earth 可用 + 默认可回退”的版本
|
||||
- 公开读取接口与后台管理接口分离
|
||||
- 手工配置源与采集器源共用统一的前端消费结构
|
||||
- Earth 里的电视窗口必须复用现有 HUD 拖拽、关闭、布局最大化逻辑
|
||||
- 小屏下优先保证窗口完整显示,超出部分在窗口内部滚动
|
||||
|
||||
## 分阶段实现
|
||||
|
||||
### Phase 1:后端配置与公开读取
|
||||
|
||||
- 在系统设置中新增 `tv` 分类
|
||||
- 定义直播源配置结构:
|
||||
- `default_source_id`
|
||||
- `auto_fallback`
|
||||
- `sources[]`
|
||||
- 每个直播源至少包含:
|
||||
- `id`
|
||||
- `name`
|
||||
- `provider`
|
||||
- `region`
|
||||
- `language`
|
||||
- `source_type`
|
||||
- `embed_url`
|
||||
- `stream_url`
|
||||
- `homepage_url`
|
||||
- `is_enabled`
|
||||
- `is_fallback`
|
||||
- `sort_order`
|
||||
- `collector_source`
|
||||
- `notes`
|
||||
- 默认兜底源使用央视官网 `CCTV-4` 直播页
|
||||
- 新增公开读取接口,供 Earth 页面无登录态读取直播源配置
|
||||
|
||||
### Phase 2:采集器扩展位
|
||||
|
||||
- 新增 `news_live_streams` collector 占位
|
||||
- 规范采集器入库数据结构,使其能与后台手工配置源合并
|
||||
- TV 公开接口支持合并:
|
||||
- 后台手工配置源
|
||||
- 采集器入库源
|
||||
- 保持手工配置源优先级更高,避免采集器覆盖人工兜底配置
|
||||
|
||||
### Phase 3:后台配置界面
|
||||
|
||||
- 在系统配置页新增 `电视直播` tab
|
||||
- 支持:
|
||||
- 查看当前默认源
|
||||
- 开关自动回退
|
||||
- 新增直播源
|
||||
- 编辑直播源
|
||||
- 删除直播源
|
||||
- 启用/禁用直播源
|
||||
- 将某个直播源设为默认源
|
||||
- 明确区分:
|
||||
- 手工配置源
|
||||
- 采集器来源
|
||||
|
||||
### Phase 4:Earth HUD 集成
|
||||
|
||||
- 在 `显示控制` 子菜单加入电视按钮
|
||||
- 新增 TV HUD 面板:
|
||||
- 可拖拽
|
||||
- 可关闭
|
||||
- 支持显示/隐藏状态同步
|
||||
- 参与布局最大化与恢复布局
|
||||
- 面板内容至少包含:
|
||||
- 当前频道标题
|
||||
- 源切换下拉菜单
|
||||
- 刷新按钮
|
||||
- 打开官网按钮
|
||||
- 播放区域
|
||||
|
||||
### Phase 5:播放策略
|
||||
|
||||
- 第一版优先支持 `iframe`/嵌入页类直播源
|
||||
- 为未来扩展保留:
|
||||
- `hls`
|
||||
- `video`
|
||||
- `external`
|
||||
- 如果默认源不可用:
|
||||
- 优先回退到标记为 `is_fallback=true` 的源
|
||||
- 若无明确回退源,则回退到第一个可用源
|
||||
- 面板内要有清晰的加载、错误、回退提示
|
||||
|
||||
### Phase 6:打磨与清理
|
||||
|
||||
- 统一 HUD 风格
|
||||
- 小屏下限制窗口尺寸并启用内部滚动
|
||||
- 避免窗口超出屏幕
|
||||
- 补最小验证
|
||||
- 清理临时代码、重复样式和无用资源
|
||||
|
||||
## 首版交付定义
|
||||
|
||||
当以下条件满足时,认为首版可用:
|
||||
|
||||
- 后台可以配置新闻直播源
|
||||
- Earth 可以读取并显示默认直播源
|
||||
- 工具栏可打开电视窗口
|
||||
- 电视窗口可拖拽、可关闭
|
||||
- 央视 `CCTV-4` 作为默认兜底源可被使用
|
||||
- 代码结构已为后续采集器接入预留统一接口
|
||||
@@ -0,0 +1,165 @@
|
||||
# HUD Panel Component Plan
|
||||
|
||||
## Goal
|
||||
|
||||
Unify Earth HUD panels into a reusable component layer so new panels can share:
|
||||
|
||||
- a consistent shell
|
||||
- a consistent header
|
||||
- a consistent action-button system
|
||||
- a consistent body and collapse pattern
|
||||
|
||||
## Scope
|
||||
|
||||
Target panels:
|
||||
|
||||
- `tv-panel`
|
||||
- `news-panel`
|
||||
- `legend`
|
||||
- `layer-panel`
|
||||
- `earth-stats`
|
||||
- `info-card`
|
||||
- settings modal header/actions
|
||||
|
||||
## Component Model
|
||||
|
||||
### Base shell
|
||||
|
||||
- `.hud-panel`
|
||||
- `.hud-panel--compact`
|
||||
- `.hud-panel--media`
|
||||
- `.hud-panel--collapsed`
|
||||
- `.hud-panel-hidden`
|
||||
- `.hud-panel.is-dragging`
|
||||
- `.hud-panel.is-layout-animating`
|
||||
|
||||
### Header
|
||||
|
||||
- `.hud-panel__header`
|
||||
- `.hud-panel__title-group`
|
||||
- `.hud-panel__title`
|
||||
- `.hud-panel__subtitle`
|
||||
- `.hud-panel__chip`
|
||||
- `.hud-panel__actions`
|
||||
|
||||
Header baseline rule:
|
||||
|
||||
- Header title styling is fixed by the component layer and should not drift per panel
|
||||
- Title font size, font weight, letter spacing, line height, text color, and vertical alignment come from the shared header tokens and structure
|
||||
- Header divider, border treatment, inner spacing, and title-to-actions alignment are part of the same shared baseline
|
||||
- Panel-specific header differences should be limited to explicit variants such as `compact` or `media`, or token overrides with documented intent
|
||||
- “Looks close enough” local header overrides should be treated as temporary compatibility code and removed during migration
|
||||
|
||||
### Actions
|
||||
|
||||
- `.hud-panel__action`
|
||||
- `.hud-panel__action--icon`
|
||||
- `.hud-panel__action--collapse`
|
||||
- `.hud-panel__action--close`
|
||||
- `.hud-panel__action--refresh`
|
||||
- `.hud-panel__action--external`
|
||||
|
||||
Action-button baseline rule:
|
||||
|
||||
- Header action buttons must have one fixed default style baseline across all HUD panels
|
||||
- Default width behavior, padding, icon size, radius, alignment, hover, and active feedback all come from `.hud-panel__action`
|
||||
- Panel-specific differences must be expressed through explicit variants or token overrides, not ad-hoc local button rewrites
|
||||
- `close` buttons are part of the same default action system and must not silently fall back to a separate legacy box model
|
||||
|
||||
### Body
|
||||
|
||||
- `.hud-panel__body`
|
||||
- `.hud-panel__body--scroll`
|
||||
- `.hud-panel__body--collapsible`
|
||||
|
||||
### Collapse behavior
|
||||
|
||||
- `.hud-panel--collapsed`
|
||||
- `.hud-panel--expand-up`
|
||||
- `.hud-panel--expand-down`
|
||||
|
||||
Adaptive collapse / expand rule:
|
||||
|
||||
- HUD panels support two expansion directions:
|
||||
- top-to-bottom expansion
|
||||
- bottom-to-top expansion
|
||||
- Expansion direction should be decided at runtime from available viewport space rather than hardcoded per panel
|
||||
- Use:
|
||||
- `d` = available distance from the header anchor to the viewport bottom edge
|
||||
- `h` = expected expanded panel height
|
||||
- buffer = `20px`
|
||||
- Collapsed-state direction rule:
|
||||
- if `d > h + 20px`, the next action direction is `expand-up`
|
||||
- if `d <= h + 20px`, the next action direction is `expand-down`
|
||||
- To avoid jitter around the threshold, the shared controller should keep a small hysteresis band:
|
||||
- if the current direction is already `up`, keep it until `d <= h`
|
||||
- if the current direction is already `down`, keep it until `d > h + 20px`
|
||||
- The opposite edge is still a safety guard:
|
||||
- if the chosen side cannot fit at all, fall back to the other side if it can fit
|
||||
- if neither side fully fits, choose the side with more space and let the body scroll
|
||||
- If neither direction fully fits, choose the direction with more available space and let the body scroll
|
||||
- Collapse icon direction must match the active expansion direction so the icon always describes the real open/close motion
|
||||
- The collapse icon describes the next action, not the current state
|
||||
- This mapping is fixed component behavior and must not drift per panel:
|
||||
- collapsed + expand-down => `expand_more`
|
||||
- expanded + expand-down => `expand_less`
|
||||
- collapsed + expand-up => `expand_less`
|
||||
- expanded + expand-up => `expand_more`
|
||||
- Panels must not combine icon-name swapping with extra CSS rotation for the same collapse control
|
||||
- Expansion direction and icon direction must come from one shared source of truth in the component controller
|
||||
- The direction decision should be recomputed when opening, resizing the viewport, or restoring a dragged panel near another edge
|
||||
|
||||
## Tokens
|
||||
|
||||
Promote panel differences into CSS variables instead of duplicating selectors:
|
||||
|
||||
- `--hud-panel-padding`
|
||||
- `--hud-header-padding`
|
||||
- `--hud-header-gap`
|
||||
- `--hud-action-padding`
|
||||
- `--hud-action-gap`
|
||||
- `--hud-action-icon-size`
|
||||
- `--hud-body-gap`
|
||||
- `--hud-body-max-height`
|
||||
- `--hud-chip-radius`
|
||||
- `--hud-title-font-size`
|
||||
- `--hud-title-font-weight`
|
||||
- `--hud-title-letter-spacing`
|
||||
- `--hud-title-line-height`
|
||||
- `--hud-title-color`
|
||||
- `--hud-header-border-color`
|
||||
- `--hud-header-divider-opacity`
|
||||
- `--hud-expand-direction`
|
||||
|
||||
## Migration Order
|
||||
|
||||
1. Build the shared component layer in `frontend/public/earth/css/hud.css`
|
||||
2. Migrate `tv-panel` and `news-panel` first as the reference implementation
|
||||
3. Migrate `legend` and `layer-panel` into a compact variant
|
||||
4. Migrate `earth-stats` and `info-card`
|
||||
5. Align settings modal header/actions with the same action system
|
||||
6. Remove legacy one-off button selectors after verification
|
||||
|
||||
## Guardrails
|
||||
|
||||
- Do not change panel behavior and data flow during the first pass
|
||||
- Keep old class names temporarily as compatibility hooks
|
||||
- Prefer variable overrides over per-panel reimplementation
|
||||
- Treat header action-button default styling as fixed component API, not per-panel design space
|
||||
- Treat header title typography, border, and divider styling as fixed component API, not per-panel design space
|
||||
- Treat collapse direction as a component behavior contract, not a one-off panel trick
|
||||
- Treat collapse icon semantics as a component behavior contract, not a per-panel visual preference
|
||||
- Verify header alignment and drag/collapse behavior after each migration batch
|
||||
|
||||
## First Implementation Batch
|
||||
|
||||
Batch 1 should only do:
|
||||
|
||||
- shared header structure
|
||||
- shared action-button system
|
||||
- shared title typography and header border/divider baseline
|
||||
- shared collapsible body pattern
|
||||
- adaptive collapse direction logic and direction-aware collapse icons
|
||||
- migration of `tv-panel` and `news-panel`
|
||||
|
||||
That keeps risk low while giving the rest of the HUD a stable target to migrate toward.
|
||||
+2
@@ -1,3 +1,5 @@
|
||||
> Archived note: this document was originally created by sisyphus and later reviewed against the main docs set. Useful content has been absorbed into `docs/plans/` where appropriate.
|
||||
|
||||
# 地球3D可视化架构重构计划
|
||||
|
||||
## 背景
|
||||
@@ -1,3 +1,5 @@
|
||||
> Archived note: this document was originally created by sisyphus and later reviewed against the main docs set. Useful content has been absorbed into `docs/plans/` where appropriate.
|
||||
|
||||
# 卫星预测轨道显示功能
|
||||
|
||||
## TL;DR
|
||||
@@ -1,3 +1,5 @@
|
||||
> Archived note: this document was originally created by sisyphus and later reviewed against the main docs set. Useful content has been absorbed into `docs/plans/` where appropriate.
|
||||
|
||||
# UE5 3D 大屏客户端开发计划
|
||||
|
||||
## 项目概述
|
||||
+2
@@ -1,3 +1,5 @@
|
||||
> Archived note: this document was originally created by sisyphus and later reviewed against the main docs set. Useful content has been absorbed into `docs/plans/` where appropriate.
|
||||
|
||||
# WebGL Instancing 卫星渲染优化计划
|
||||
|
||||
## 背景
|
||||
@@ -0,0 +1,39 @@
|
||||
# Plans Docs
|
||||
|
||||
这里放“未来实施方案和未完成计划”的文档,重点回答:
|
||||
|
||||
- 我们准备做什么
|
||||
- 为什么要做
|
||||
- 分几期做
|
||||
- 当前差距和下一步是什么
|
||||
|
||||
适合放入这里的内容:
|
||||
|
||||
- Earth / BGP / 地形 / 天球实施方案
|
||||
- AI Playground 发展计划
|
||||
- backend / datasource / agent roadmap
|
||||
- UE5 MVP 方案
|
||||
|
||||
当前重点入口:
|
||||
|
||||
- [earth-mobile-drawer-ui-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-mobile-drawer-ui-plan.md)
|
||||
- [earth-compute-center-bgp-style-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-compute-center-bgp-style-plan.md)
|
||||
- [earth-renderer-architecture-separation-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-renderer-architecture-separation-plan.md)
|
||||
- [earth-country-boundary-overlay-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-country-boundary-overlay-plan.md)
|
||||
- [earth-predicted-orbit-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-predicted-orbit-plan.md)
|
||||
- [earth-webgl-instancing-satellites-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-webgl-instancing-satellites-plan.md)
|
||||
- [earth-real-terrain-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-real-terrain-plan.md)
|
||||
- [earth-news-source-configuration-and-collector-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-news-source-configuration-and-collector-plan.md)
|
||||
- [frontend-public-docs-site-plan.md](/home/ray/dev/linkong/planet/docs/plans/frontend-public-docs-site-plan.md)
|
||||
- [frontend-ai-playground-development-plan.md](/home/ray/dev/linkong/planet/docs/plans/frontend-ai-playground-development-plan.md)
|
||||
- [ue5-mvp-fused-plan.md](/home/ray/dev/linkong/planet/docs/plans/ue5-mvp-fused-plan.md)
|
||||
|
||||
不适合放入这里的内容:
|
||||
|
||||
- 当前代码结构说明
|
||||
- 组件现状和实现入口
|
||||
- 已经落地的技术上下文说明
|
||||
|
||||
这些应放入:
|
||||
|
||||
- [docs/technical/README.md](/home/ray/dev/linkong/planet/docs/technical/README.md)
|
||||
@@ -0,0 +1,647 @@
|
||||
# Agent Architecture Plan
|
||||
|
||||
## Overview
|
||||
|
||||
This document defines the agent architecture for Planet.
|
||||
|
||||
The architecture is intentionally broader than datasource health checking.
|
||||
|
||||
It is designed to support both:
|
||||
|
||||
- datasource health governance
|
||||
- future situational-awareness workflows
|
||||
|
||||
The core idea is to avoid building a one-off "repair broken API links" agent.
|
||||
|
||||
Instead, Planet should grow a reusable agent runtime that can:
|
||||
|
||||
- collect evidence
|
||||
- evaluate signals
|
||||
- reason over incomplete information
|
||||
- generate proposals
|
||||
- produce assessments
|
||||
- execute limited actions under policy
|
||||
|
||||
|
||||
## Design Goal
|
||||
|
||||
Build an agent foundation that can evolve in this order:
|
||||
|
||||
1. datasource health checks
|
||||
2. datasource repair proposals
|
||||
3. signal correlation
|
||||
4. situational assessments
|
||||
5. controlled runtime actions
|
||||
|
||||
This means the architecture should treat datasource health as one use case of the larger agent system, not as the whole system.
|
||||
|
||||
|
||||
## Core Principles
|
||||
|
||||
1. Separate evidence from reasoning
|
||||
|
||||
- raw signals should be gathered first
|
||||
- deterministic checks should run before LLM reasoning
|
||||
|
||||
2. Agents do not own the defaults
|
||||
|
||||
- repository defaults remain human-owned
|
||||
- agents operate on runtime state, proposals, and overrides
|
||||
|
||||
3. Reasoning and action are different responsibilities
|
||||
|
||||
- many agents should be read-only or propose-only
|
||||
- only tightly controlled flows may apply changes
|
||||
|
||||
4. Shared runtime, specialized roles
|
||||
|
||||
- multiple agent roles should share the same object model and orchestration patterns
|
||||
- health and situational-awareness agents should not invent incompatible payloads
|
||||
|
||||
5. Auditability is mandatory
|
||||
|
||||
- every proposal, assessment, and applied action should be attributable
|
||||
|
||||
|
||||
## System Layers
|
||||
|
||||
Planet agent architecture should be split into four layers.
|
||||
|
||||
### 1. Signal Layer
|
||||
|
||||
Purpose:
|
||||
|
||||
- gather raw evidence from internal and external systems
|
||||
|
||||
Example sources:
|
||||
|
||||
- collector outputs
|
||||
- datasource health checks
|
||||
- logs
|
||||
- snapshots
|
||||
- alerts
|
||||
- web search results
|
||||
- scraped pages
|
||||
- external APIs
|
||||
- operator inputs
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- fetch
|
||||
- normalize
|
||||
- timestamp
|
||||
- tag with source and trust level
|
||||
|
||||
This layer should not make high-level judgments.
|
||||
|
||||
|
||||
### 2. Evaluation Layer
|
||||
|
||||
Purpose:
|
||||
|
||||
- perform deterministic analysis
|
||||
|
||||
Examples:
|
||||
|
||||
- reachability checks
|
||||
- schema validation
|
||||
- threshold checks
|
||||
- time-window comparisons
|
||||
- anomaly counters
|
||||
- completeness checks
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- classify signals into machine-readable findings
|
||||
- attach deterministic evidence
|
||||
|
||||
This layer should avoid LLM dependency whenever possible.
|
||||
|
||||
|
||||
### 3. Reasoning Layer
|
||||
|
||||
Purpose:
|
||||
|
||||
- use LLMs when semantic interpretation or incomplete-information reasoning is needed
|
||||
|
||||
Examples:
|
||||
|
||||
- endpoint migration inference
|
||||
- multi-source event correlation
|
||||
- causality hypotheses
|
||||
- ambiguity reduction
|
||||
- assessment narrative generation
|
||||
- action recommendation generation
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- synthesize evidence
|
||||
- produce hypotheses
|
||||
- rank confidence
|
||||
- explain reasoning boundaries
|
||||
|
||||
This is the main place where `aiprovider` and web search are used.
|
||||
|
||||
|
||||
### 4. Action Layer
|
||||
|
||||
Purpose:
|
||||
|
||||
- convert proposals or assessments into controlled system actions
|
||||
|
||||
Examples:
|
||||
|
||||
- create runtime override
|
||||
- create proposal
|
||||
- publish alert
|
||||
- update operator task queue
|
||||
- generate summary artifact
|
||||
- trigger follow-up verification
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- enforce policy
|
||||
- enforce approval requirements
|
||||
- verify post-action outcomes
|
||||
- record audit trails
|
||||
|
||||
|
||||
## Architecture Sketch
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A["Collectors / Logs / Snapshots / External APIs"] --> B["Signal Layer"]
|
||||
W["Web Search / Page Fetch / Docs"] --> B
|
||||
B --> C["Evaluation Layer"]
|
||||
C --> D["Findings"]
|
||||
D --> E["Reasoning Layer (LLM + Tools)"]
|
||||
E --> F["Proposals"]
|
||||
E --> G["Assessments"]
|
||||
F --> H["Action Layer"]
|
||||
H --> I["Runtime Overrides / Alerts / Tasks"]
|
||||
H --> J["Verification Loop"]
|
||||
J --> B
|
||||
|
||||
K["Policy Engine"] --> H
|
||||
L["Audit / History Store"] --> H
|
||||
L --> E
|
||||
L --> C
|
||||
```
|
||||
|
||||
|
||||
## Agent Roles
|
||||
|
||||
The first version should define these logical roles.
|
||||
|
||||
### 1. Health Agent
|
||||
|
||||
Primary use case:
|
||||
|
||||
- datasource health governance
|
||||
|
||||
Inputs:
|
||||
|
||||
- datasource metadata
|
||||
- current endpoint
|
||||
- latest health records
|
||||
- latest failures
|
||||
- deterministic findings
|
||||
|
||||
Outputs:
|
||||
|
||||
- health interpretation
|
||||
- repair proposal
|
||||
- confidence
|
||||
- evidence references
|
||||
|
||||
Typical action level:
|
||||
|
||||
- propose-only
|
||||
|
||||
|
||||
### 2. Correlation Agent
|
||||
|
||||
Primary use case:
|
||||
|
||||
- identify whether multiple signals describe the same event or related events
|
||||
|
||||
Inputs:
|
||||
|
||||
- findings from multiple collectors
|
||||
- time windows
|
||||
- region / ASN / prefix / cable relationships
|
||||
- prior incidents
|
||||
|
||||
Outputs:
|
||||
|
||||
- grouped event candidates
|
||||
- correlation rationale
|
||||
- confidence per relationship
|
||||
|
||||
Typical action level:
|
||||
|
||||
- read-only
|
||||
|
||||
|
||||
### 3. Assessment Agent
|
||||
|
||||
Primary use case:
|
||||
|
||||
- produce situational-awareness outputs
|
||||
|
||||
Inputs:
|
||||
|
||||
- grouped events
|
||||
- findings
|
||||
- current context
|
||||
- historical context
|
||||
- operator constraints
|
||||
|
||||
Outputs:
|
||||
|
||||
- structured assessment
|
||||
- risk summary
|
||||
- evidence-backed recommendations
|
||||
- missing-information list
|
||||
|
||||
Typical action level:
|
||||
|
||||
- read-only or propose-only
|
||||
|
||||
|
||||
### 4. Recovery Agent
|
||||
|
||||
Primary use case:
|
||||
|
||||
- carry low-risk proposals into controlled runtime actions
|
||||
|
||||
Inputs:
|
||||
|
||||
- approved proposal
|
||||
- policy constraints
|
||||
- trusted-domain rules
|
||||
- verification checks
|
||||
|
||||
Outputs:
|
||||
|
||||
- applied override
|
||||
- failed application
|
||||
- rollback request
|
||||
|
||||
Typical action level:
|
||||
|
||||
- apply-limited
|
||||
|
||||
|
||||
## Shared Object Model
|
||||
|
||||
All agents should work on a shared object model.
|
||||
|
||||
That prevents the health subsystem and situational-awareness subsystem from drifting into incompatible payloads.
|
||||
|
||||
### Signal
|
||||
|
||||
Represents a raw observed fact.
|
||||
|
||||
Examples:
|
||||
|
||||
- a datasource returned HTTP 404
|
||||
- a collector returned empty results
|
||||
- BGP updates spiked in one region
|
||||
- a known endpoint now redirects elsewhere
|
||||
|
||||
Suggested shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "sig_123",
|
||||
"type": "datasource.http_failure",
|
||||
"source": "ris_live_bgp",
|
||||
"occurred_at": "2026-04-08T10:00:00Z",
|
||||
"severity": "medium",
|
||||
"payload": {},
|
||||
"trust": 0.95
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
### Finding
|
||||
|
||||
Represents a deterministic or semi-deterministic interpretation of one or more signals.
|
||||
|
||||
Examples:
|
||||
|
||||
- `schema_changed`
|
||||
- `endpoint_unreachable`
|
||||
- `data_volume_abnormally_low`
|
||||
- `event_cluster_detected`
|
||||
|
||||
Suggested shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "find_123",
|
||||
"type": "datasource.schema_changed",
|
||||
"source_ids": ["sig_123"],
|
||||
"confidence": 0.92,
|
||||
"evidence": [],
|
||||
"details": {}
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
### Proposal
|
||||
|
||||
Represents a recommended action, not an already-applied action.
|
||||
|
||||
Examples:
|
||||
|
||||
- switch endpoint to new URL
|
||||
- disable bad override
|
||||
- escalate issue for manual review
|
||||
|
||||
Suggested shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "prop_123",
|
||||
"kind": "endpoint_override",
|
||||
"target": "telegeography_cables",
|
||||
"confidence": 0.84,
|
||||
"reason": "Official docs now point to a new API path",
|
||||
"payload": {},
|
||||
"evidence_urls": [],
|
||||
"status": "proposed"
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
### Assessment
|
||||
|
||||
Represents a structured situational-awareness output for operators or downstream systems.
|
||||
|
||||
Examples:
|
||||
|
||||
- current network posture summary
|
||||
- incident impact assessment
|
||||
- risk and response recommendations
|
||||
|
||||
Suggested shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "assess_123",
|
||||
"scope": "regional-network",
|
||||
"risk_level": "high",
|
||||
"summary": "Regional routing instability is increasing.",
|
||||
"key_risks": [],
|
||||
"evidence": [],
|
||||
"recommendations": [],
|
||||
"missing_data": []
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
## State Machine
|
||||
|
||||
The shared orchestration flow should look like this:
|
||||
|
||||
```mermaid
|
||||
stateDiagram-v2
|
||||
[*] --> Collect
|
||||
Collect --> Validate
|
||||
Validate --> Classify
|
||||
Classify --> Reason
|
||||
Reason --> Propose
|
||||
Reason --> Assess
|
||||
Propose --> Review
|
||||
Review --> Apply
|
||||
Apply --> Verify
|
||||
Verify --> Archive
|
||||
Assess --> Archive
|
||||
Archive --> [*]
|
||||
```
|
||||
|
||||
Definitions:
|
||||
|
||||
- `Collect`: gather signals
|
||||
- `Validate`: run deterministic checks
|
||||
- `Classify`: create findings
|
||||
- `Reason`: invoke LLM reasoning when needed
|
||||
- `Propose`: create change proposals
|
||||
- `Review`: policy or human approval
|
||||
- `Apply`: perform limited runtime action
|
||||
- `Verify`: confirm action effect
|
||||
- `Archive`: store artifacts and decisions
|
||||
|
||||
|
||||
## Permission Model
|
||||
|
||||
Each agent role should be assigned one of these action levels.
|
||||
|
||||
### `read-only`
|
||||
|
||||
Allowed:
|
||||
|
||||
- read signals
|
||||
- search web
|
||||
- fetch pages
|
||||
- read internal state
|
||||
- generate findings and assessments
|
||||
|
||||
Not allowed:
|
||||
|
||||
- mutate config
|
||||
- write overrides
|
||||
- change live runtime behavior
|
||||
|
||||
|
||||
### `propose-only`
|
||||
|
||||
Allowed:
|
||||
|
||||
- everything in `read-only`
|
||||
- create proposals
|
||||
- create review tasks
|
||||
|
||||
Not allowed:
|
||||
|
||||
- apply live changes
|
||||
|
||||
|
||||
### `apply-limited`
|
||||
|
||||
Allowed:
|
||||
|
||||
- everything in `propose-only`
|
||||
- write approved runtime overrides
|
||||
- trigger verification checks
|
||||
|
||||
Not allowed:
|
||||
|
||||
- mutate repository defaults
|
||||
- make destructive data changes
|
||||
- bypass policy engine
|
||||
|
||||
|
||||
## Runtime Components
|
||||
|
||||
The first durable architecture should introduce these components.
|
||||
|
||||
### 1. Signal Store
|
||||
|
||||
Stores normalized evidence and health outputs.
|
||||
|
||||
|
||||
### 2. Finding Store
|
||||
|
||||
Stores deterministic classifications that can be reused by multiple agents.
|
||||
|
||||
|
||||
### 3. Proposal Store
|
||||
|
||||
Stores recommended actions with evidence and confidence.
|
||||
|
||||
|
||||
### 4. Assessment Store
|
||||
|
||||
Stores structured situational-awareness outputs.
|
||||
|
||||
|
||||
### 5. Policy Engine
|
||||
|
||||
Decides:
|
||||
|
||||
- whether agent may run
|
||||
- whether proposal requires review
|
||||
- whether proposal may auto-apply
|
||||
- whether post-apply verification passed
|
||||
|
||||
|
||||
### 6. Override Store
|
||||
|
||||
Stores runtime-only configuration changes.
|
||||
|
||||
This is where endpoint repairs should live.
|
||||
|
||||
|
||||
## Relation To `aiprovider`
|
||||
|
||||
`aiprovider` should remain the model gateway.
|
||||
|
||||
It should not become the full agent runtime.
|
||||
|
||||
Recommended split:
|
||||
|
||||
- `aiprovider`
|
||||
- provider adaptation
|
||||
- prompt transport
|
||||
- model execution
|
||||
- protocol compatibility
|
||||
|
||||
- agent runtime
|
||||
- orchestration
|
||||
- signal handling
|
||||
- tool selection
|
||||
- proposal generation
|
||||
- policy and audit
|
||||
|
||||
This keeps provider concerns and agent behavior concerns separate.
|
||||
|
||||
|
||||
## Relation To Datasource Health
|
||||
|
||||
Datasource health becomes one vertical slice of this architecture.
|
||||
|
||||
Mapping:
|
||||
|
||||
- signal:
|
||||
- endpoint unreachable
|
||||
- schema mismatch
|
||||
- bad content type
|
||||
- finding:
|
||||
- `failed`
|
||||
- `schema_changed`
|
||||
- `moved_endpoint_suspected`
|
||||
- proposal:
|
||||
- runtime override suggestion
|
||||
- assessment:
|
||||
- datasource health summary for operators
|
||||
|
||||
|
||||
## Relation To Situational Awareness
|
||||
|
||||
Future situational-awareness capabilities should reuse the same flow:
|
||||
|
||||
- raw telemetry becomes signals
|
||||
- anomaly detection becomes findings
|
||||
- LLM correlation becomes reasoning
|
||||
- operator-facing output becomes assessments
|
||||
- policy-approved mitigations become actions
|
||||
|
||||
This lets the platform evolve from operational health governance into broader cyber/network posture workflows without changing the architecture.
|
||||
|
||||
|
||||
## Suggested Delivery Sequence
|
||||
|
||||
### Phase A
|
||||
|
||||
- finalize shared object model
|
||||
- implement health-oriented signal and finding storage
|
||||
|
||||
### Phase B
|
||||
|
||||
- implement Health Agent
|
||||
- generate proposals only
|
||||
|
||||
### Phase C
|
||||
|
||||
- implement Assessment Agent
|
||||
- expose structured assessments via API
|
||||
|
||||
### Phase D
|
||||
|
||||
- implement Correlation Agent
|
||||
- support multi-source incident grouping
|
||||
|
||||
### Phase E
|
||||
|
||||
- implement Recovery Agent with policy-gated runtime actions
|
||||
|
||||
|
||||
## Recommended First Build
|
||||
|
||||
The first build should not try to implement every agent role.
|
||||
|
||||
Recommended initial slice:
|
||||
|
||||
- shared object model
|
||||
- health signals
|
||||
- health findings
|
||||
- Health Agent
|
||||
- proposal generation only
|
||||
|
||||
This gives immediate value while preserving the longer-term architecture.
|
||||
|
||||
|
||||
## Non-Goals For The First Iteration
|
||||
|
||||
- repository YAML auto-rewrites
|
||||
- unrestricted autonomous action
|
||||
- full incident graph reasoning
|
||||
- automatic large-scale remediation
|
||||
- agent-owned configuration source of truth
|
||||
|
||||
|
||||
## Summary
|
||||
|
||||
Planet should treat agents as a reusable runtime for evidence, reasoning, proposals, and assessments.
|
||||
|
||||
The datasource health use case is the first practical entrypoint, but the architecture should already assume future situational-awareness expansion.
|
||||
|
||||
The safest path is:
|
||||
|
||||
- deterministic checks first
|
||||
- agent reasoning second
|
||||
- proposals before actions
|
||||
- runtime overrides instead of default mutation
|
||||
@@ -0,0 +1,346 @@
|
||||
# Agent Runtime Roadmap
|
||||
|
||||
## Overview
|
||||
|
||||
This document connects three existing planning threads into one implementation roadmap:
|
||||
|
||||
- `aiprovider` as the model gateway
|
||||
- datasource health governance as the first practical agent use case
|
||||
- situational awareness as the broader long-term target
|
||||
|
||||
Related documents:
|
||||
|
||||
- [aiprovider](/home/ray/dev/linkong/planet/docs/technical/agents-aiprovider.md)
|
||||
- [datasource-health-plan](/home/ray/dev/linkong/planet/docs/plans/agents-datasource-health-plan.md)
|
||||
- [agent-architecture-plan](/home/ray/dev/linkong/planet/docs/plans/agents-agent-architecture-plan.md)
|
||||
|
||||
|
||||
## Big Picture
|
||||
|
||||
Planet should evolve in layers:
|
||||
|
||||
1. stable model gateway
|
||||
2. deterministic health and evidence collection
|
||||
3. agent runtime for reasoning and proposal generation
|
||||
4. situational-awareness assessments and controlled actions
|
||||
|
||||
This prevents the system from collapsing into a single giant "AI feature" with unclear boundaries.
|
||||
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
U["Frontend / Backend APIs / Operators"] --> B["Planet Backend"]
|
||||
B --> H["Datasource Health Services"]
|
||||
B --> R["Agent Runtime"]
|
||||
R --> P["aiprovider"]
|
||||
P --> M["OpenAI / Anthropic / MiniMax / Ollama / Local Models"]
|
||||
|
||||
C["Collectors / Snapshots / Logs / Alerts / BGP Signals"] --> S["Signal Store"]
|
||||
H --> S
|
||||
S --> E["Evaluation Layer"]
|
||||
E --> F["Findings"]
|
||||
F --> R
|
||||
|
||||
W["Web Search / Page Fetch / Docs Fetch"] --> R
|
||||
R --> PR["Proposals"]
|
||||
R --> AS["Assessments"]
|
||||
|
||||
PR --> O["Runtime Overrides / Review Queue / Tasks"]
|
||||
AS --> SA["Situational Awareness APIs / UI"]
|
||||
|
||||
O --> V["Verification Loop"]
|
||||
V --> S
|
||||
```
|
||||
|
||||
|
||||
## Role Boundaries
|
||||
|
||||
### `aiprovider`
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- provider compatibility
|
||||
- protocol adaptation
|
||||
- auth and model transport
|
||||
- request/response normalization
|
||||
|
||||
Not responsible for:
|
||||
|
||||
- agent orchestration
|
||||
- business workflows
|
||||
- datasource repair policy
|
||||
- situational-awareness domain logic
|
||||
|
||||
|
||||
### Backend
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- stable business APIs
|
||||
- auth and permissions
|
||||
- task orchestration
|
||||
- health records
|
||||
- proposal and override persistence
|
||||
- assessment exposure
|
||||
|
||||
|
||||
### Agent Runtime
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- consume findings and context
|
||||
- invoke LLMs via `aiprovider`
|
||||
- invoke tools such as web search
|
||||
- create proposals
|
||||
- create assessments
|
||||
- route to policy-controlled action paths
|
||||
|
||||
|
||||
## Delivery Sequence
|
||||
|
||||
## Stage 1: Gateway Foundation
|
||||
|
||||
Status:
|
||||
|
||||
- already in place
|
||||
|
||||
Delivered by current work:
|
||||
|
||||
- `aiprovider`
|
||||
- multi-provider compatibility
|
||||
- backend AI facade
|
||||
- MiniMax / Anthropic-compatible support
|
||||
- request-id propagation
|
||||
|
||||
Primary outcome:
|
||||
|
||||
- the system already has a stable way to call models
|
||||
|
||||
|
||||
## Stage 2: Datasource Health MVP
|
||||
|
||||
Goal:
|
||||
|
||||
- establish deterministic health observability
|
||||
|
||||
Key work:
|
||||
|
||||
- health check task runner
|
||||
- health result table
|
||||
- datasource health APIs
|
||||
- UI visibility
|
||||
- collector endpoint override precedence cleanup
|
||||
|
||||
Primary outcome:
|
||||
|
||||
- Planet knows which collectors are healthy before asking an LLM anything
|
||||
|
||||
|
||||
## Stage 3: Health Agent
|
||||
|
||||
Goal:
|
||||
|
||||
- let the first agent role operate on health failures
|
||||
|
||||
Key work:
|
||||
|
||||
- convert health failures into signals/findings
|
||||
- invoke agent only for failed or suspicious cases
|
||||
- produce repair proposals with evidence and confidence
|
||||
|
||||
Primary outcome:
|
||||
|
||||
- Planet can suggest endpoint repairs without mutating defaults
|
||||
|
||||
|
||||
## Stage 4: Runtime Repair Application
|
||||
|
||||
Goal:
|
||||
|
||||
- safely apply approved datasource repair proposals
|
||||
|
||||
Key work:
|
||||
|
||||
- override storage
|
||||
- policy-gated apply flow
|
||||
- verification after apply
|
||||
- rollback path
|
||||
|
||||
Primary outcome:
|
||||
|
||||
- datasource repair becomes operationally useful without polluting repository defaults
|
||||
|
||||
|
||||
## Stage 5: Situational Awareness Assessments
|
||||
|
||||
Goal:
|
||||
|
||||
- reuse the same runtime for broader operator-facing assessment
|
||||
|
||||
Key work:
|
||||
|
||||
- normalize telemetry and incident evidence into signals/findings
|
||||
- build Assessment Agent
|
||||
- expose structured assessments through backend APIs and UI
|
||||
|
||||
Primary outcome:
|
||||
|
||||
- LLM output becomes evidence-backed situational summary, not just ad hoc chat output
|
||||
|
||||
|
||||
## Stage 6: Correlation and Controlled Actions
|
||||
|
||||
Goal:
|
||||
|
||||
- connect multiple sources into higher-level posture and event groupings
|
||||
|
||||
Key work:
|
||||
|
||||
- event correlation
|
||||
- incident grouping
|
||||
- recommendation scoring
|
||||
- controlled action routing
|
||||
|
||||
Primary outcome:
|
||||
|
||||
- Planet becomes a true agent-assisted situational-awareness system
|
||||
|
||||
|
||||
## Implementation Tracks
|
||||
|
||||
These tracks can progress in parallel, but they should stay loosely coupled.
|
||||
|
||||
### Track A: Config and Runtime Resolution
|
||||
|
||||
Scope:
|
||||
|
||||
- datasource defaults
|
||||
- overrides
|
||||
- runtime precedence
|
||||
- audit trails
|
||||
|
||||
First milestone:
|
||||
|
||||
- health-safe override layer
|
||||
|
||||
|
||||
### Track B: Health and Evidence
|
||||
|
||||
Scope:
|
||||
|
||||
- deterministic checks
|
||||
- failure categorization
|
||||
- signal and finding persistence
|
||||
|
||||
First milestone:
|
||||
|
||||
- datasource health record system
|
||||
|
||||
|
||||
### Track C: Agent Runtime
|
||||
|
||||
Scope:
|
||||
|
||||
- shared object model
|
||||
- orchestration flow
|
||||
- prompt/tool pipeline
|
||||
- policy integration
|
||||
|
||||
First milestone:
|
||||
|
||||
- Health Agent proposal pipeline
|
||||
|
||||
|
||||
### Track D: Situational Awareness
|
||||
|
||||
Scope:
|
||||
|
||||
- assessment schema
|
||||
- multi-source context assembly
|
||||
- operator-facing outputs
|
||||
|
||||
First milestone:
|
||||
|
||||
- structured assessment API
|
||||
|
||||
|
||||
## Shared Artifacts
|
||||
|
||||
To avoid fragmentation, these artifacts should be shared across all future agent work.
|
||||
|
||||
### Shared object model
|
||||
|
||||
- `Signal`
|
||||
- `Finding`
|
||||
- `Proposal`
|
||||
- `Assessment`
|
||||
|
||||
### Shared orchestration flow
|
||||
|
||||
- collect
|
||||
- validate
|
||||
- classify
|
||||
- reason
|
||||
- propose or assess
|
||||
- review or apply
|
||||
- verify
|
||||
- archive
|
||||
|
||||
### Shared policy model
|
||||
|
||||
- read-only
|
||||
- propose-only
|
||||
- apply-limited
|
||||
|
||||
|
||||
## Recommended Next Concrete Steps
|
||||
|
||||
1. Build Stage 2 first
|
||||
|
||||
- datasource health records
|
||||
- deterministic checks
|
||||
- no automatic repair
|
||||
|
||||
2. Then build Stage 3
|
||||
|
||||
- Health Agent
|
||||
- proposal generation only
|
||||
|
||||
3. Then Stage 4
|
||||
|
||||
- override apply flow
|
||||
- rollback and verification
|
||||
|
||||
4. Only after that start Stage 5
|
||||
|
||||
- broader situational-awareness assessment workflows
|
||||
|
||||
|
||||
## Why This Order
|
||||
|
||||
Because situational-awareness quality depends on reliable upstream data.
|
||||
|
||||
If datasource health is weak:
|
||||
|
||||
- agent reasoning quality will degrade
|
||||
- false explanations will increase
|
||||
- assessment trust will drop
|
||||
|
||||
So datasource health is not a side task.
|
||||
|
||||
It is the first operational foundation for the later situational-awareness system.
|
||||
|
||||
|
||||
## Summary
|
||||
|
||||
Planet should be built as:
|
||||
|
||||
- `aiprovider` for model access
|
||||
- backend services for orchestration and persistence
|
||||
- datasource health as the first evidence-governance layer
|
||||
- agent runtime as the reusable reasoning core
|
||||
- situational awareness as the long-term application layer
|
||||
|
||||
That path keeps the architecture coherent and lets each phase produce useful functionality without forcing a rewrite later.
|
||||
@@ -0,0 +1,486 @@
|
||||
# Datasource Health Plan
|
||||
|
||||
## Overview
|
||||
|
||||
This document defines a phased plan for datasource health governance.
|
||||
|
||||
The goal is to make collectors observable, diagnosable, and recoverable when upstream APIs change, while avoiding unsafe automatic mutation of repository defaults.
|
||||
|
||||
The key principle is:
|
||||
|
||||
- do not let runtime automation rewrite repository default config
|
||||
|
||||
Instead, split responsibilities across:
|
||||
|
||||
- default config
|
||||
- runtime overrides
|
||||
- health check records
|
||||
- agent-generated repair proposals
|
||||
|
||||
|
||||
## Problem Statement
|
||||
|
||||
Collectors currently depend on third-party APIs, data downloads, mirrored JSON files, archive links, and web pages.
|
||||
|
||||
These upstream dependencies can fail in several ways:
|
||||
|
||||
- endpoint becomes unreachable
|
||||
- endpoint still responds but schema changes
|
||||
- content-type changes
|
||||
- website shuts down or moves
|
||||
- mirror link disappears
|
||||
- HTML structure changes and scraping fails
|
||||
- endpoint requires a new path or new host
|
||||
|
||||
We want a system that can:
|
||||
|
||||
- detect datasource health degradation early
|
||||
- identify likely cause
|
||||
- search for updated endpoints when reasonable
|
||||
- apply safe runtime fixes without polluting default repo config
|
||||
- preserve auditability and rollback
|
||||
|
||||
|
||||
## Design Principles
|
||||
|
||||
1. Default config is stable
|
||||
|
||||
- `backend/app/core/data_sources.yaml` remains the repository baseline.
|
||||
- It should be changed intentionally through normal development flow, not by autonomous runtime agents.
|
||||
|
||||
2. Runtime fixes are isolated
|
||||
|
||||
- Emergency or adaptive fixes should live in a runtime override layer.
|
||||
- Overrides should be reversible and auditable.
|
||||
|
||||
3. Deterministic checks come first
|
||||
|
||||
- Use normal programmatic health checks before using LLMs.
|
||||
- Only call an agent when deterministic checks indicate a meaningful failure.
|
||||
|
||||
4. Agents suggest before they mutate
|
||||
|
||||
- Agents should produce proposals with evidence and confidence.
|
||||
- Application of a proposal should be controlled by policy.
|
||||
|
||||
5. Every repair is attributable
|
||||
|
||||
- Store what changed, why, who or what suggested it, and when it was applied.
|
||||
|
||||
|
||||
## Configuration Layers
|
||||
|
||||
Recommended runtime precedence:
|
||||
|
||||
1. datasource endpoint override
|
||||
2. datasource DB endpoint override
|
||||
3. repository default YAML
|
||||
4. collector internal fallback logic
|
||||
|
||||
Definitions:
|
||||
|
||||
- repository default YAML:
|
||||
- `backend/app/core/data_sources.yaml`
|
||||
- versioned baseline
|
||||
- datasource DB endpoint override:
|
||||
- existing `DataSourceConfig.endpoint`
|
||||
- current runtime override entrypoint
|
||||
- datasource endpoint override:
|
||||
- a dedicated new override table
|
||||
- used for health-repair and proposal application
|
||||
- collector internal fallback logic:
|
||||
- final defensive fallback
|
||||
- should be minimized over time
|
||||
|
||||
|
||||
## Recommended Architecture
|
||||
|
||||
### 1. Deterministic Health Checks
|
||||
|
||||
Each collector gets a health profile with checks such as:
|
||||
|
||||
- endpoint resolves
|
||||
- HTTP request succeeds
|
||||
- status code is acceptable
|
||||
- content-type is expected
|
||||
- body parses successfully
|
||||
- minimum structural fields exist
|
||||
- sample item count is plausible
|
||||
- latency is within threshold
|
||||
|
||||
Output states:
|
||||
|
||||
- `healthy`
|
||||
- `degraded`
|
||||
- `failed`
|
||||
- `schema_changed`
|
||||
- `rate_limited`
|
||||
- `auth_required`
|
||||
|
||||
|
||||
### 2. Agent-Assisted Repair Discovery
|
||||
|
||||
Only triggered when deterministic health checks fail or return suspicious structure.
|
||||
|
||||
Agent responsibilities:
|
||||
|
||||
- search for current official endpoint or replacement path
|
||||
- inspect likely upstream documentation or landing pages
|
||||
- compare candidate endpoint output to collector expectations
|
||||
- produce a repair proposal with confidence and evidence
|
||||
|
||||
Agent should not directly modify repository defaults.
|
||||
|
||||
|
||||
### 3. Safe Runtime Repair Application
|
||||
|
||||
Repair proposals can be:
|
||||
|
||||
- reviewed manually
|
||||
- auto-applied only under strict low-risk policy
|
||||
|
||||
Auto-apply should be limited to cases like:
|
||||
|
||||
- same trusted domain
|
||||
- highly similar response structure
|
||||
- repeated successful verification
|
||||
- confidence above threshold
|
||||
|
||||
|
||||
## Phased Delivery Plan
|
||||
|
||||
## Phase 1: Deterministic Health MVP
|
||||
|
||||
Goal:
|
||||
|
||||
- build health observability without automated repair
|
||||
|
||||
Scope:
|
||||
|
||||
- datasource health check task runner
|
||||
- datasource health result persistence
|
||||
- endpoint reachability + parse checks
|
||||
- dashboard or API visibility into health status
|
||||
|
||||
Deliverables:
|
||||
|
||||
- health check service
|
||||
- health check record table
|
||||
- status endpoint
|
||||
- scheduled or manual check trigger
|
||||
|
||||
No agent usage yet.
|
||||
|
||||
|
||||
## Phase 2: Agent Repair Proposals
|
||||
|
||||
Goal:
|
||||
|
||||
- let agent investigate failing sources and propose updated endpoints
|
||||
|
||||
Scope:
|
||||
|
||||
- invoke agent only when datasource health is `failed` or `schema_changed`
|
||||
- web search + page inspection
|
||||
- candidate endpoint extraction
|
||||
- proposal persistence
|
||||
|
||||
Deliverables:
|
||||
|
||||
- repair proposal schema
|
||||
- proposal generation pipeline
|
||||
- confidence and evidence model
|
||||
- operator review view or API
|
||||
|
||||
Still no automatic config mutation.
|
||||
|
||||
|
||||
## Phase 3: Runtime Overrides
|
||||
|
||||
Goal:
|
||||
|
||||
- allow approved proposals to take effect safely at runtime
|
||||
|
||||
Scope:
|
||||
|
||||
- add dedicated override storage
|
||||
- runtime resolution prefers override over default config
|
||||
- proposal application writes override only
|
||||
|
||||
Deliverables:
|
||||
|
||||
- endpoint override table
|
||||
- override-aware resolution logic
|
||||
- apply/reject endpoints
|
||||
- rollback endpoint
|
||||
|
||||
Repository default YAML remains untouched.
|
||||
|
||||
|
||||
## Phase 4: Limited Auto-Apply
|
||||
|
||||
Goal:
|
||||
|
||||
- safely automate a narrow slice of low-risk repairs
|
||||
|
||||
Scope:
|
||||
|
||||
- policy engine for auto-apply
|
||||
- same-domain or trusted-domain checks
|
||||
- structure validation
|
||||
- staged verification after apply
|
||||
|
||||
Deliverables:
|
||||
|
||||
- auto-apply rules
|
||||
- audit logs
|
||||
- automatic post-apply health verification
|
||||
- auto-disable or rollback on regression
|
||||
|
||||
|
||||
## Data Model Draft
|
||||
|
||||
### datasource_health_checks
|
||||
|
||||
Purpose:
|
||||
|
||||
- store each health evaluation result
|
||||
|
||||
Suggested fields:
|
||||
|
||||
- `id`
|
||||
- `datasource_id`
|
||||
- `collector_name`
|
||||
- `endpoint_checked`
|
||||
- `status`
|
||||
- `http_status`
|
||||
- `content_type`
|
||||
- `latency_ms`
|
||||
- `sample_count`
|
||||
- `error_message`
|
||||
- `details`
|
||||
- `checked_at`
|
||||
|
||||
`details` can store structured diagnostic data such as:
|
||||
|
||||
- parsed fields
|
||||
- schema mismatch summary
|
||||
- retry count
|
||||
- exception class
|
||||
|
||||
|
||||
### datasource_repair_proposals
|
||||
|
||||
Purpose:
|
||||
|
||||
- store agent-generated repair suggestions
|
||||
|
||||
Suggested fields:
|
||||
|
||||
- `id`
|
||||
- `datasource_id`
|
||||
- `collector_name`
|
||||
- `old_endpoint`
|
||||
- `candidate_endpoint`
|
||||
- `reason`
|
||||
- `confidence`
|
||||
- `evidence_urls`
|
||||
- `evidence_summary`
|
||||
- `status`
|
||||
- `created_by`
|
||||
- `created_at`
|
||||
- `reviewed_at`
|
||||
|
||||
Suggested `status` values:
|
||||
|
||||
- `proposed`
|
||||
- `approved`
|
||||
- `rejected`
|
||||
- `applied`
|
||||
- `expired`
|
||||
|
||||
|
||||
### datasource_endpoint_overrides
|
||||
|
||||
Purpose:
|
||||
|
||||
- runtime endpoint override layer
|
||||
|
||||
Suggested fields:
|
||||
|
||||
- `id`
|
||||
- `datasource_id`
|
||||
- `collector_name`
|
||||
- `endpoint`
|
||||
- `reason`
|
||||
- `source`
|
||||
- `proposal_id`
|
||||
- `enabled`
|
||||
- `created_at`
|
||||
- `updated_at`
|
||||
|
||||
Suggested `source` values:
|
||||
|
||||
- `manual`
|
||||
- `health-agent`
|
||||
- `migration`
|
||||
|
||||
|
||||
## API Draft
|
||||
|
||||
### Health
|
||||
|
||||
- `GET /api/v1/datasources/health`
|
||||
- `GET /api/v1/datasources/{id}/health`
|
||||
- `POST /api/v1/datasources/{id}/health-check`
|
||||
- `POST /api/v1/datasources/health-check-all`
|
||||
|
||||
### Repair proposals
|
||||
|
||||
- `GET /api/v1/datasources/{id}/repair-proposals`
|
||||
- `POST /api/v1/datasources/{id}/repair-proposals/generate`
|
||||
- `POST /api/v1/datasources/{id}/repair-proposals/{proposal_id}/approve`
|
||||
- `POST /api/v1/datasources/{id}/repair-proposals/{proposal_id}/reject`
|
||||
- `POST /api/v1/datasources/{id}/repair-proposals/{proposal_id}/apply`
|
||||
|
||||
### Overrides
|
||||
|
||||
- `GET /api/v1/datasources/{id}/overrides`
|
||||
- `POST /api/v1/datasources/{id}/overrides`
|
||||
- `PUT /api/v1/datasources/{id}/overrides/{override_id}`
|
||||
- `DELETE /api/v1/datasources/{id}/overrides/{override_id}`
|
||||
|
||||
|
||||
## Agent Contract Draft
|
||||
|
||||
When deterministic health fails, the agent should receive:
|
||||
|
||||
- datasource name
|
||||
- collector name
|
||||
- current endpoint
|
||||
- current failure mode
|
||||
- expected response shape summary
|
||||
- known trusted domains
|
||||
|
||||
Expected output:
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "proposal",
|
||||
"candidate_endpoint": "https://example.com/api/v2/data",
|
||||
"confidence": 0.86,
|
||||
"reason": "Official docs now point to v2 endpoint",
|
||||
"evidence_urls": [
|
||||
"https://example.com/docs/api",
|
||||
"https://example.com/changelog"
|
||||
],
|
||||
"notes": "Response shape appears compatible after light field remapping"
|
||||
}
|
||||
```
|
||||
|
||||
The agent should never output "rewrite the default yaml" as its primary action.
|
||||
|
||||
|
||||
## Risk Analysis
|
||||
|
||||
### Risk: wrong endpoint chosen by agent
|
||||
|
||||
Mitigation:
|
||||
|
||||
- use trusted-domain allowlists
|
||||
- require evidence URLs
|
||||
- require confidence threshold
|
||||
- add manual review for medium-risk sources
|
||||
|
||||
|
||||
### Risk: endpoint responds but schema silently changed
|
||||
|
||||
Mitigation:
|
||||
|
||||
- deterministic schema checks
|
||||
- parse and sample validation
|
||||
- content-type checks
|
||||
- collector-specific required fields
|
||||
|
||||
|
||||
### Risk: automatic runtime override causes hidden drift
|
||||
|
||||
Mitigation:
|
||||
|
||||
- store all overrides explicitly
|
||||
- mark source of override
|
||||
- keep default YAML unchanged
|
||||
- expose active overrides in API/UI
|
||||
|
||||
|
||||
### Risk: persistent bad override breaks data collection
|
||||
|
||||
Mitigation:
|
||||
|
||||
- allow rollback
|
||||
- keep parent/default endpoint visible
|
||||
- re-run verification after apply
|
||||
- auto-disable override on repeated failure
|
||||
|
||||
|
||||
## Operational Policy Recommendations
|
||||
|
||||
1. Do not auto-apply for high-value or high-fragility sources initially.
|
||||
|
||||
2. Use manual approval for:
|
||||
|
||||
- scraped HTML sources
|
||||
- unofficial mirrors
|
||||
- sources with auth or rate-limit complexity
|
||||
- sources with legal or trust ambiguity
|
||||
|
||||
3. Allow auto-apply only for:
|
||||
|
||||
- same-domain version bumps
|
||||
- obvious official migration paths
|
||||
- repeated passing verification
|
||||
|
||||
4. Expose health + proposal + override state together in one operator view.
|
||||
|
||||
|
||||
## Suggested Implementation Order
|
||||
|
||||
1. Phase 1
|
||||
- health result table
|
||||
- deterministic checks
|
||||
- API and UI visibility
|
||||
|
||||
2. Phase 2
|
||||
- proposal table
|
||||
- agent prompt/output contract
|
||||
- proposal generation job
|
||||
|
||||
3. Phase 3
|
||||
- runtime override table
|
||||
- resolver precedence update
|
||||
- apply/reject endpoints
|
||||
|
||||
4. Phase 4
|
||||
- auto-apply rules
|
||||
- rollback policy
|
||||
- operator automation
|
||||
|
||||
|
||||
## Out Of Scope For The First Iteration
|
||||
|
||||
- direct automatic mutation of repository default YAML
|
||||
- automatic git commits by repair agents
|
||||
- unrestricted autonomous endpoint replacement
|
||||
- fully generalized schema remapping engine
|
||||
|
||||
|
||||
## Recommended First Milestone
|
||||
|
||||
The first milestone should be:
|
||||
|
||||
- deterministic datasource health checks
|
||||
- persisted results
|
||||
- manual visibility
|
||||
- no automatic repair
|
||||
|
||||
This gives immediate operational value with low risk, and prepares clean inputs for the later agent phase.
|
||||
@@ -0,0 +1,478 @@
|
||||
# Datasource Health Stage 2 Tasks
|
||||
|
||||
## Goal
|
||||
|
||||
Stage 2 focuses on the first practical operational layer:
|
||||
|
||||
- deterministic datasource health checks
|
||||
- persisted health results
|
||||
- health visibility through API and UI
|
||||
- no agent-assisted repair yet
|
||||
|
||||
This stage should make Planet capable of answering:
|
||||
|
||||
- which collectors are healthy
|
||||
- which collectors are degraded
|
||||
- which collectors are failing
|
||||
- why they are failing at a basic deterministic level
|
||||
|
||||
|
||||
## Scope
|
||||
|
||||
Included:
|
||||
|
||||
- datasource health data model
|
||||
- deterministic health check service
|
||||
- manual and scheduled health check triggers
|
||||
- health result APIs
|
||||
- frontend visibility
|
||||
|
||||
Excluded:
|
||||
|
||||
- LLM reasoning
|
||||
- web-search-based repair proposals
|
||||
- automatic endpoint rewriting
|
||||
- runtime override application
|
||||
|
||||
|
||||
## Delivery Target
|
||||
|
||||
At the end of Stage 2, an operator should be able to:
|
||||
|
||||
1. see health status for each collector
|
||||
2. trigger a health check manually
|
||||
3. inspect the latest failure reason
|
||||
4. inspect the last checked endpoint
|
||||
5. understand whether the problem is:
|
||||
- unreachable
|
||||
- auth-related
|
||||
- rate-limit-related
|
||||
- schema-related
|
||||
- empty-data-related
|
||||
|
||||
|
||||
## Work Breakdown
|
||||
|
||||
## A. Data Model
|
||||
|
||||
### A1. Add datasource health record table
|
||||
|
||||
Create a new model, for example:
|
||||
|
||||
- `backend/app/models/datasource_health_check.py`
|
||||
|
||||
Suggested fields:
|
||||
|
||||
- `id`
|
||||
- `datasource_id`
|
||||
- `collector_name`
|
||||
- `endpoint_checked`
|
||||
- `status`
|
||||
- `http_status`
|
||||
- `content_type`
|
||||
- `latency_ms`
|
||||
- `sample_count`
|
||||
- `error_message`
|
||||
- `details`
|
||||
- `checked_at`
|
||||
|
||||
Suggested status enum values:
|
||||
|
||||
- `healthy`
|
||||
- `degraded`
|
||||
- `failed`
|
||||
- `schema_changed`
|
||||
- `rate_limited`
|
||||
- `auth_required`
|
||||
- `empty_result`
|
||||
|
||||
|
||||
### A2. Add datasource health summary fields
|
||||
|
||||
Option A:
|
||||
|
||||
- keep summary only in the health check table
|
||||
|
||||
Option B:
|
||||
|
||||
- also add summary fields on `data_sources`
|
||||
|
||||
Recommended first step:
|
||||
|
||||
- do not mutate `data_sources` schema yet
|
||||
- derive summary from the latest health record
|
||||
|
||||
|
||||
### A3. Migration task
|
||||
|
||||
Add migration for the health table.
|
||||
|
||||
Deliverables:
|
||||
|
||||
- migration file
|
||||
- model registration
|
||||
|
||||
|
||||
## B. Health Check Engine
|
||||
|
||||
### B1. Define health check service
|
||||
|
||||
Add a new service module, for example:
|
||||
|
||||
- `backend/app/services/datasource_health.py`
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- resolve effective endpoint
|
||||
- execute deterministic check
|
||||
- classify result
|
||||
- persist health record
|
||||
|
||||
|
||||
### B2. Define shared result schema
|
||||
|
||||
Create a typed result object, for example:
|
||||
|
||||
- `HealthCheckResult`
|
||||
|
||||
Suggested fields:
|
||||
|
||||
- `status`
|
||||
- `endpoint_checked`
|
||||
- `http_status`
|
||||
- `content_type`
|
||||
- `latency_ms`
|
||||
- `sample_count`
|
||||
- `error_message`
|
||||
- `details`
|
||||
|
||||
|
||||
### B3. Implement base deterministic checks
|
||||
|
||||
Every datasource should go through a minimal baseline check:
|
||||
|
||||
1. resolve endpoint
|
||||
2. perform request
|
||||
3. measure latency
|
||||
4. inspect status code
|
||||
5. inspect content type
|
||||
6. inspect body shape
|
||||
|
||||
Classification rules:
|
||||
|
||||
- network error -> `failed`
|
||||
- HTTP 401/403 -> `auth_required`
|
||||
- HTTP 429 -> `rate_limited`
|
||||
- HTTP 404/410 -> `failed`
|
||||
- parse failure -> `schema_changed`
|
||||
- zero or suspiciously empty results -> `empty_result` or `degraded`
|
||||
- valid parse -> `healthy`
|
||||
|
||||
|
||||
### B4. Add collector-aware adapters
|
||||
|
||||
Some collectors do not use the same fetch semantics.
|
||||
|
||||
Add adapter profiles such as:
|
||||
|
||||
- `http_json`
|
||||
- `http_csv`
|
||||
- `html_scrape`
|
||||
- `stream_probe`
|
||||
- `auth_session_http`
|
||||
|
||||
Initial mapping suggestion:
|
||||
|
||||
- `huggingface`, `peeringdb`, `cloudflare` -> `http_json`
|
||||
- `fao` -> `http_csv`
|
||||
- `top500`, `epoch_ai`, `telegeography live_map` -> `html_scrape`
|
||||
- `ris_live` -> `stream_probe`
|
||||
- `spacetrack` -> `auth_session_http`
|
||||
|
||||
|
||||
### B5. Add sample validation hooks
|
||||
|
||||
For each adapter, add a lightweight validation rule.
|
||||
|
||||
Examples:
|
||||
|
||||
- JSON array length > 0
|
||||
- CSV rows > 1
|
||||
- HTML page contains expected table or script patterns
|
||||
- stream source yields at least one valid event within timeout
|
||||
|
||||
|
||||
## C. Persistence and Query Layer
|
||||
|
||||
### C1. Save every check run
|
||||
|
||||
Each health check should insert a record.
|
||||
|
||||
Do not overwrite history in Stage 2.
|
||||
|
||||
|
||||
### C2. Add latest-health query helpers
|
||||
|
||||
Add helper functions to fetch:
|
||||
|
||||
- latest health record by datasource
|
||||
- latest failed health record
|
||||
- recent health history
|
||||
|
||||
|
||||
### C3. Optional retention policy
|
||||
|
||||
For Stage 2, retention can be deferred.
|
||||
|
||||
If desired, keep only:
|
||||
|
||||
- last N records per datasource
|
||||
|
||||
|
||||
## D. API Layer
|
||||
|
||||
### D1. Add health list endpoint
|
||||
|
||||
Suggested endpoint:
|
||||
|
||||
- `GET /api/v1/datasources/health`
|
||||
|
||||
Returns:
|
||||
|
||||
- datasource id
|
||||
- collector name
|
||||
- current endpoint
|
||||
- latest health status
|
||||
- last checked time
|
||||
- short reason
|
||||
|
||||
|
||||
### D2. Add per-datasource health detail endpoint
|
||||
|
||||
Suggested endpoint:
|
||||
|
||||
- `GET /api/v1/datasources/{id}/health`
|
||||
|
||||
Returns:
|
||||
|
||||
- latest record
|
||||
- recent history
|
||||
- detailed classification fields
|
||||
|
||||
|
||||
### D3. Add manual health trigger endpoint
|
||||
|
||||
Suggested endpoint:
|
||||
|
||||
- `POST /api/v1/datasources/{id}/health-check`
|
||||
|
||||
Behavior:
|
||||
|
||||
- run a health check now
|
||||
- persist the result
|
||||
- return the new record
|
||||
|
||||
|
||||
### D4. Add bulk health trigger endpoint
|
||||
|
||||
Suggested endpoint:
|
||||
|
||||
- `POST /api/v1/datasources/health-check-all`
|
||||
|
||||
Behavior:
|
||||
|
||||
- enqueue or run health checks for all active datasources
|
||||
|
||||
|
||||
## E. Scheduling
|
||||
|
||||
### E1. Add health scheduler task
|
||||
|
||||
Decide scheduling strategy.
|
||||
|
||||
Recommended first version:
|
||||
|
||||
- run collector jobs and health checks separately
|
||||
- health checks run on a lower frequency
|
||||
|
||||
Suggested frequency:
|
||||
|
||||
- every 6h or 12h for most datasources
|
||||
- optionally on-demand only in the very first cut
|
||||
|
||||
|
||||
### E2. Prevent health check collision with collection
|
||||
|
||||
Rules:
|
||||
|
||||
- health checks should not disrupt active collection
|
||||
- they should use light requests
|
||||
- if a collector is currently running, health check may:
|
||||
- skip
|
||||
- or use a lightweight endpoint probe only
|
||||
|
||||
|
||||
## F. Frontend
|
||||
|
||||
### F1. Add health columns to datasource list
|
||||
|
||||
Update:
|
||||
|
||||
- `frontend/src/pages/DataSources/DataSources.tsx`
|
||||
|
||||
Suggested new columns:
|
||||
|
||||
- health status
|
||||
- last checked
|
||||
- reason summary
|
||||
|
||||
|
||||
### F2. Add manual health check action
|
||||
|
||||
Per datasource:
|
||||
|
||||
- button or dropdown action:
|
||||
- `健康检查`
|
||||
|
||||
|
||||
### F3. Add health detail drawer or modal
|
||||
|
||||
Show:
|
||||
|
||||
- endpoint checked
|
||||
- status
|
||||
- HTTP status
|
||||
- content type
|
||||
- sample count
|
||||
- error message
|
||||
- last few results
|
||||
|
||||
|
||||
### F4. Add basic visual language
|
||||
|
||||
Suggested colors:
|
||||
|
||||
- green -> healthy
|
||||
- yellow -> degraded
|
||||
- orange -> rate-limited / auth-required
|
||||
- red -> failed / schema-changed
|
||||
|
||||
|
||||
## G. Observability
|
||||
|
||||
### G1. Structured logging
|
||||
|
||||
Every health check should log:
|
||||
|
||||
- datasource id
|
||||
- collector name
|
||||
- endpoint
|
||||
- status
|
||||
- latency
|
||||
- failure class
|
||||
|
||||
|
||||
### G2. Optional metrics
|
||||
|
||||
If metrics are added later, useful counters include:
|
||||
|
||||
- health checks total
|
||||
- health checks failed
|
||||
- schema changes detected
|
||||
- rate limited checks
|
||||
|
||||
|
||||
## H. Tests
|
||||
|
||||
### H1. Unit tests
|
||||
|
||||
Add tests for:
|
||||
|
||||
- status classification
|
||||
- content type classification
|
||||
- adapter behavior
|
||||
- latest-health query helpers
|
||||
|
||||
|
||||
### H2. API tests
|
||||
|
||||
Add tests for:
|
||||
|
||||
- health endpoints require auth
|
||||
- manual trigger endpoint works
|
||||
- list endpoint returns latest status
|
||||
|
||||
|
||||
### H3. Failure-path tests
|
||||
|
||||
Add coverage for:
|
||||
|
||||
- HTTP 404
|
||||
- HTTP 429
|
||||
- invalid JSON
|
||||
- empty response
|
||||
- parse mismatch
|
||||
|
||||
|
||||
## Suggested File Plan
|
||||
|
||||
Possible implementation files:
|
||||
|
||||
- `backend/app/models/datasource_health_check.py`
|
||||
- `backend/app/services/datasource_health.py`
|
||||
- `backend/app/schemas/datasource_health.py`
|
||||
- `backend/app/api/v1/datasource_health.py`
|
||||
- migration file under the project migration system
|
||||
|
||||
Likely touched existing files:
|
||||
|
||||
- `backend/app/api/main.py`
|
||||
- `frontend/src/pages/DataSources/DataSources.tsx`
|
||||
- `backend/tests/test_api.py`
|
||||
|
||||
|
||||
## Suggested Execution Order
|
||||
|
||||
1. Add model and migration
|
||||
2. Add service and result schema
|
||||
3. Add deterministic adapters
|
||||
4. Add manual trigger API
|
||||
5. Add list/detail API
|
||||
6. Add frontend visibility
|
||||
7. Add scheduled checks
|
||||
8. Expand tests
|
||||
|
||||
|
||||
## Minimal First Milestone
|
||||
|
||||
If we want the fastest useful slice, do this first:
|
||||
|
||||
1. health table
|
||||
2. deterministic check service
|
||||
3. manual per-datasource health check API
|
||||
4. latest health list API
|
||||
5. frontend status badge column
|
||||
|
||||
That is enough to start operating the system and will provide the input layer for Stage 3.
|
||||
|
||||
|
||||
## Dependency On Later Stages
|
||||
|
||||
Stage 2 outputs become direct inputs for Stage 3.
|
||||
|
||||
Specifically:
|
||||
|
||||
- failed or schema-changed health records become agent triggers
|
||||
- health history becomes repair context
|
||||
- endpoint_checked becomes proposal baseline
|
||||
|
||||
|
||||
## Success Criteria
|
||||
|
||||
Stage 2 is done when:
|
||||
|
||||
- every active datasource can be health-checked deterministically
|
||||
- the latest health state is visible in API and UI
|
||||
- operators can manually trigger checks
|
||||
- failures are categorized into stable machine-readable statuses
|
||||
- no LLM is required for core health visibility
|
||||
@@ -0,0 +1,309 @@
|
||||
# Situational Awareness Foundation Plan
|
||||
|
||||
## 定位
|
||||
|
||||
当前这套 AI 能力应被视为 `态势感知服务底座`,而不是完整的态势感知产品。
|
||||
|
||||
也就是说,现阶段的目标不是:
|
||||
|
||||
- 做一个“什么都能分析”的万能 AI 页面
|
||||
- 让模型在证据不足时替代人工研判
|
||||
- 过早把页面做成完整指挥大屏
|
||||
|
||||
现阶段真正要做的是:
|
||||
|
||||
- 先把 `model gateway / backend facade / evidence injection / page-specific brief` 这几层边界搭稳
|
||||
- 让系统能够在已有证据上稳定地产出“可读、可回看、可扩展”的摘要
|
||||
- 为后续更强的数据联动、agent 推理和 assessment 结构化输出预留好接口与数据模型
|
||||
|
||||
## 当前现实约束
|
||||
|
||||
### 1. 数据维度不足
|
||||
|
||||
目前系统能提供的主要证据仍集中在:
|
||||
|
||||
- BGP incidents / anomalies / events
|
||||
- collector coverage
|
||||
- datasource health / platform alerts
|
||||
- prefix geography 的部分归属信息
|
||||
|
||||
当前明显还缺:
|
||||
|
||||
- 流量异常与业务指标
|
||||
- 电商、支付、物流等业务侧指标
|
||||
- 更丰富的资产、链路、区域、行业画像
|
||||
- 外部舆情、公告、运营商状态、基础设施事件等背景信息
|
||||
|
||||
这意味着:
|
||||
|
||||
- 模型现在可以做“基于现有证据的摘要与归纳”
|
||||
- 但还不能可靠地做“跨维度因果研判”
|
||||
|
||||
### 2. 维度之间联动还弱
|
||||
|
||||
目前不同模块之间更多是“并列展示”,还不是“强关联分析”:
|
||||
|
||||
- 系统告警和 BGP 事件还没有统一事件模型
|
||||
- collector bias 与真实区域热度还没有完全剥离
|
||||
- datasource health 与 BGP 风险、业务影响之间还没有稳定映射
|
||||
|
||||
这意味着:
|
||||
|
||||
- 当前更适合做 `brief / overview / operator notes`
|
||||
- 还不适合过度承诺“自动态势判断”
|
||||
|
||||
### 3. 结构化 assessment 还未成为主输出
|
||||
|
||||
虽然已经有 BGP brief、系统告警 brief、态势告警 brief,但目前主输出仍偏向:
|
||||
|
||||
- 文本摘要
|
||||
- facts/context 附带证据
|
||||
|
||||
后续真正要服务态势感知,需要更稳定的结构化输出,例如:
|
||||
|
||||
- summary
|
||||
- key risks
|
||||
- evidence
|
||||
- confidence
|
||||
- recommendations
|
||||
- missing data
|
||||
|
||||
## 当前基座已经具备的能力
|
||||
|
||||
### 1. AI 调用边界已经明确
|
||||
|
||||
- `aiprovider` 负责模型协议与 provider 兼容
|
||||
- `backend` 负责业务 API、证据整合和鉴权
|
||||
- `frontend` 负责页面入口与结果展示
|
||||
|
||||
### 2. 页面级 AI 入口已经开始成型
|
||||
|
||||
当前已经有或正在收口的入口:
|
||||
|
||||
- `Playground`
|
||||
- 用于链路验证与 provider 诊断
|
||||
- `BGP AI 简报`
|
||||
- 用于 BGP 事实摘要和区域风险归纳
|
||||
- `Alerts`
|
||||
- 用于系统告警、BGP 告警、态势告警三类入口
|
||||
|
||||
### 3. 证据优先的方向已经建立
|
||||
|
||||
已经不再只依赖人工在 Playground 中手填 prompt,系统开始具备:
|
||||
|
||||
- 从真实业务数据生成事实输入
|
||||
- 保存 facts/context 快照
|
||||
- 回看 AI 输出时同时回看证据
|
||||
|
||||
这一步非常关键,因为它决定后面能否从“玩具 demo”走向“有运维价值的系统”。
|
||||
|
||||
## 近期收尾建议
|
||||
|
||||
这些事情都属于“底座收口”,值得做,但不应该再继续重产品包装。
|
||||
|
||||
### 1. 统一 Alerts 页面
|
||||
|
||||
已采用:
|
||||
|
||||
- 一个 `Alerts` 页面
|
||||
- 三个 tab:
|
||||
- `系统告警`
|
||||
- `BGP 告警`
|
||||
- `态势告警`
|
||||
|
||||
收尾重点:
|
||||
|
||||
- 保持 tab 的文案、摘要卡和 AI 简报交互一致
|
||||
- 不额外扩展成多个独立二级页面
|
||||
|
||||
### 2. 保持 Playground 为测试台
|
||||
|
||||
原则:
|
||||
|
||||
- Playground 只承担链路验证、provider 状态诊断、请求结果观察
|
||||
- 不继续堆“万能业务分析器”式交互
|
||||
|
||||
### 3. 把 brief 能力当服务能力而不是页面特效
|
||||
|
||||
页面现在能看到按钮和结果,这很好,但更重要的是:
|
||||
|
||||
- 后端接口稳定
|
||||
- facts/context 可追踪
|
||||
- 输出结构后续可升级
|
||||
|
||||
### 4. 导航结构先收口,不继续平铺一级菜单
|
||||
|
||||
随着后续能力扩展,系统很可能继续新增:
|
||||
|
||||
- 海缆
|
||||
- 算力中心
|
||||
- 战争信息
|
||||
- 电商分析
|
||||
- 其他专题观测页
|
||||
|
||||
如果继续把这些入口全部平铺在左侧一级菜单中,会带来两个问题:
|
||||
|
||||
- 一级菜单过长,用户难以判断先进入哪个上下文
|
||||
- `观测页 / 告警页 / 研判页 / 运维页` 的职责边界会被混在一起
|
||||
|
||||
因此近期应明确采用分组导航,而不是继续扩展平铺菜单。
|
||||
|
||||
推荐的导航分组如下:
|
||||
|
||||
- `总览`
|
||||
- 仪表盘
|
||||
- Earth
|
||||
- `专题观测`
|
||||
- BGP 观测
|
||||
- 采集数据
|
||||
- 后续可扩展:海缆、算力中心、战争信息、电商分析
|
||||
- `告警与研判`
|
||||
- Alerts
|
||||
- `运维与配置`
|
||||
- 数据源
|
||||
- AI Playground
|
||||
- 用户管理
|
||||
- 系统配置
|
||||
|
||||
这套结构的含义是:
|
||||
|
||||
- `专题观测` 页面负责看某个维度本身
|
||||
- `Alerts` 负责跨模块风险与值班工作台
|
||||
- `Playground` 保持为测试台,不挤占业务导航语义
|
||||
|
||||
短期收尾时,应优先重组现有入口,而不是继续增加新的一级菜单。
|
||||
|
||||
## 后续路线
|
||||
|
||||
## Phase 1:服务底座稳固
|
||||
|
||||
目标:
|
||||
|
||||
- 不追求“更炫的 AI 页面”
|
||||
- 先把当前接口、证据、存储和页面入口收稳
|
||||
|
||||
工作项:
|
||||
|
||||
- 统一页面级 AI 入口模式
|
||||
- 统一 brief response schema
|
||||
- 保证 facts/context 在前后端都可回看
|
||||
- 继续清理 mock 和临时分支逻辑
|
||||
|
||||
完成标准:
|
||||
|
||||
- 每个 AI 入口都是真实链路
|
||||
- 每个 AI 结果都能追溯到证据输入
|
||||
|
||||
## Phase 2:Evidence-first Assessment
|
||||
|
||||
目标:
|
||||
|
||||
- 从“文本摘要”升级成“结构化 assessment”
|
||||
|
||||
工作项:
|
||||
|
||||
- 为 brief/assessment 定义统一 schema
|
||||
- 固化:
|
||||
- summary
|
||||
- key_risks
|
||||
- evidence
|
||||
- confidence
|
||||
- recommendations
|
||||
- missing_data
|
||||
- 页面以结构化区块展示,而不只是大段文本
|
||||
|
||||
完成标准:
|
||||
|
||||
- AI 输出可持久化、可比较、可审计
|
||||
|
||||
## Phase 3:多维证据接入
|
||||
|
||||
目标:
|
||||
|
||||
- 让“态势感知”真正拥有更多维度,而不是只靠 BGP 与系统告警
|
||||
|
||||
优先接入方向:
|
||||
|
||||
- datasource health findings
|
||||
- 流量或业务指标
|
||||
- 区域/资产/链路映射
|
||||
- 外部事件与公告
|
||||
- 业务垂直数据,例如电商分析相关指标
|
||||
|
||||
完成标准:
|
||||
|
||||
- AI 能基于多个维度做交叉说明
|
||||
- 不再只围绕单一模块自说自话
|
||||
|
||||
## Phase 4:Correlation Layer
|
||||
|
||||
目标:
|
||||
|
||||
- 不同来源的信号不再只是并列,而是形成统一的事件关联
|
||||
|
||||
工作项:
|
||||
|
||||
- 统一 signal/finding 模型
|
||||
- 跨模块事件聚合
|
||||
- 证据来源权重
|
||||
- collector bias 与真实热度分离
|
||||
|
||||
完成标准:
|
||||
|
||||
- 系统能回答“这些异常是不是同一件事”
|
||||
- 系统能回答“哪些结论只是观测偏差”
|
||||
|
||||
## Phase 5:Agent-assisted Situational Awareness
|
||||
|
||||
目标:
|
||||
|
||||
- 在证据足够的前提下,再让 agent 负责更复杂的推理与建议
|
||||
|
||||
工作项:
|
||||
|
||||
- 复用现有 agent runtime 规划
|
||||
- 引入 web search / docs fetch / repair proposal 等能力
|
||||
- 但始终坚持:
|
||||
- evidence first
|
||||
- proposal before action
|
||||
- no silent mutation of defaults
|
||||
|
||||
完成标准:
|
||||
|
||||
- agent 成为证据驱动的分析层
|
||||
- 而不是一个“万能猜测层”
|
||||
|
||||
## 设计原则
|
||||
|
||||
### 1. 先底座,后产品化
|
||||
|
||||
先把服务链路和证据模型做好,再做更大的页面表达。
|
||||
|
||||
### 2. 先证据,后判断
|
||||
|
||||
事实输入应先稳定,再让模型做归纳。
|
||||
|
||||
### 3. 先专用 brief,后统一态势层
|
||||
|
||||
先让各业务页有各自可信的 AI 入口,再考虑统一态势页。
|
||||
|
||||
### 4. 先 proposal,后自动动作
|
||||
|
||||
涉及修复、覆盖、写配置、调任务的动作,都应经过 proposal 和审计。
|
||||
|
||||
## 当前建议结论
|
||||
|
||||
对现在这个项目,最合理的定位是:
|
||||
|
||||
- `Playground` 是测试台
|
||||
- `BGP / Alerts` 是第一批业务 AI 入口
|
||||
- `aiprovider + backend AI facade + evidence snapshots` 是核心服务底座
|
||||
|
||||
现阶段不需要追求“已经具备完整态势感知能力”。
|
||||
|
||||
现阶段真正的成功标准是:
|
||||
|
||||
- 这套底座可用
|
||||
- 可回看
|
||||
- 可扩展
|
||||
- 不自欺欺人
|
||||
File renamed without changes.
File renamed without changes.
File renamed without changes.
Loaded 100 of 265 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user