forked from linkong/planet
Compare commits
86
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
9dafbf4f6e | ||
|
|
a87537e903 | ||
|
|
87594a95ff | ||
|
|
2da25376bd | ||
|
|
ac69d5d354 | ||
|
|
1cd2dab0ee | ||
|
|
42d019af36 | ||
|
|
b4e8afb272 | ||
|
|
eeee788530 | ||
|
|
655e2a7d2d | ||
|
|
3ea99a9529 | ||
|
|
f9c1334365 | ||
|
|
5f47ec1659 | ||
|
|
229be0bced | ||
|
|
50a417ca83 | ||
|
|
e9464a9833 | ||
|
|
86807f6af6 | ||
|
|
8b8f7138c0 | ||
|
|
d5f3784ffb | ||
|
|
195a8bf71c | ||
|
|
987c378f99 | ||
|
|
67f82dc41c | ||
|
|
abe04030fb | ||
|
|
6a5f9f7ad4 | ||
|
|
439a512148 | ||
|
|
f73fa1ea6d | ||
|
|
5b623a6385 | ||
|
|
0082cf3fbd | ||
|
|
3ae4acdff8 | ||
|
|
437efc848c | ||
|
|
003a46ac30 | ||
|
|
4b0be4cb76 | ||
|
|
b7647379de | ||
|
|
0f89372d71 | ||
|
|
2b0d4cfc49 | ||
|
|
e6d0332fba | ||
|
|
fe45a99cbd | ||
|
|
ae77b06c3c | ||
|
|
b5dd4f12f8 | ||
|
|
75cb214f23 | ||
|
|
4c21973197 | ||
|
|
51ae5e6ec9 | ||
|
|
1cf1f32ddd | ||
|
|
8f3ab88743 | ||
|
|
f8b43a995b | ||
|
|
d9adaf4134 | ||
|
|
40e51d5b20 | ||
|
|
93c1c1e550 | ||
|
|
48eb13b993 | ||
|
|
11179e7e67 | ||
|
|
07e26d6d5a | ||
|
|
7cd29cf9c0 | ||
|
|
2ee4773f4f | ||
|
|
b1d0624061 | ||
|
|
812c825dc6 | ||
|
|
a359d94127 | ||
|
|
c92be9c054 | ||
|
|
10e2bae8c2 | ||
|
|
60ed88b609 | ||
|
|
e85a9fc614 | ||
|
|
a2210f0f78 | ||
|
|
62ad09e816 | ||
|
|
89a71e6f29 | ||
|
|
60f5ff9bab | ||
|
|
fbb6adfbf5 | ||
|
|
749e6e76b6 | ||
|
|
83839b8b11 | ||
|
|
ed898aef9c | ||
|
|
abe0b5c11b | ||
|
|
306ba7f850 | ||
|
|
39f90bd575 | ||
|
|
c4ea918fac | ||
|
|
34d94a6b6b | ||
|
|
ef65acd49c | ||
|
|
5639546990 | ||
|
|
c8fe8cad59 | ||
|
|
d395769df6 | ||
|
|
8bd9d34376 | ||
|
|
2d43263b9e | ||
|
|
2da6ed166b | ||
|
|
da587398d9 | ||
|
|
f5308340af | ||
|
|
981617ee80 | ||
|
|
7abf391c74 | ||
|
|
f12719914d | ||
|
|
bc90e00e25 |
No files matched your search
@@ -0,0 +1,139 @@
|
||||
---
|
||||
description: 审查当前工作区未提交代码中的垃圾代码,并在不影响逻辑的前提下自动清理
|
||||
argument-hint: 可选:指定要检查的文件或目录(默认检查所有未提交修改)
|
||||
allowed-tools: ["Read", "Edit", "Bash", "Grep", "Glob"]
|
||||
---
|
||||
|
||||
# /cleanup — 垃圾代码审查与清理
|
||||
|
||||
分析当前工作区(git diff)中的未提交代码,找出并修复常见垃圾代码,**不得改变任何运行逻辑**。
|
||||
|
||||
## 检查范围
|
||||
|
||||
若 `$ARGUMENTS` 非空,则只检查指定文件/目录;否则检查所有未提交修改(`git diff HEAD`)。
|
||||
|
||||
## 节省上下文规则
|
||||
|
||||
优先用确定性的 CLI 检查缩小范围,不要一上来把完整文件或大 diff 读入上下文:
|
||||
|
||||
```bash
|
||||
git diff --name-only HEAD
|
||||
git diff --unified=0 HEAD -- <path>
|
||||
git diff --check
|
||||
rg -n "TODO|FIXME|console\.log|debugger|print\(" <changed-paths>
|
||||
```
|
||||
|
||||
只有 focused diff 不足以安全判断或修改时,才读取完整文件。
|
||||
|
||||
## 审查清单
|
||||
|
||||
按优先级检查以下问题(只报告在本次 diff 中**新增或修改**的代码里存在的问题):
|
||||
|
||||
### 1. 重复逻辑 (Duplicate Logic)
|
||||
- 完全相同或高度相似的代码块在多处出现
|
||||
- 同一函数/方法被多个地方各自实现,已有公共版本未被复用
|
||||
- 相同的 DOM 查询、正则、模板字符串在同一文件重复
|
||||
|
||||
### 2. Magic Numbers / Magic Strings
|
||||
- 裸数字直接参与计算(如偏移量、时间、尺寸、阈值),没有命名常量
|
||||
- 硬编码字符串(如 id 名、状态值、URL 片段)散落在逻辑中
|
||||
- 例外:`0`, `1`, `-1`, `100`, `""` 等语义明确的惯用值不算
|
||||
|
||||
### 3. 命名问题
|
||||
- 含义不明的缩写变量(如 `or_`, `tmp2`, `x2`)
|
||||
- 命名与实际用途不符
|
||||
- 同一概念在不同地方用不同名字表达
|
||||
|
||||
### 4. 死代码 / 无效代码
|
||||
- 注释掉的旧代码块(3行以上)
|
||||
- 声明后从未使用的变量/参数/导入
|
||||
- 永远不会执行的条件分支
|
||||
|
||||
### 5. 代码风格问题
|
||||
- 尾部空白字符(trailing whitespace)
|
||||
- 同一文件内风格不一致(如混用单双引号、缩进不统一)
|
||||
- 空行使用不一致(连续多个空行等)
|
||||
|
||||
### 6. 其他常见问题
|
||||
- 私有辅助函数应被 export 但没有,导致调用方重复实现
|
||||
- 类型/接口重复定义
|
||||
- 过于冗长的条件表达式可以简化(不改逻辑)
|
||||
|
||||
## 执行步骤
|
||||
|
||||
### Step 1 — 获取待检查文件列表
|
||||
|
||||
```bash
|
||||
# 无参数时:获取所有未提交修改
|
||||
git diff HEAD --name-only
|
||||
|
||||
# 有参数时:用 $ARGUMENTS 过滤
|
||||
```
|
||||
|
||||
### Step 2 — 逐文件阅读并分析
|
||||
|
||||
先从 focused diff 开始:
|
||||
|
||||
```bash
|
||||
git diff --unified=0 HEAD -- <file>
|
||||
```
|
||||
|
||||
用 `rg`、`git diff --check`、编译器或 linter 输出确认确定性问题。只有需要上下文时才用 Read 读取完整文件。对照审查清单,记录每个问题:文件名、行号、问题类型、建议修复方式。
|
||||
|
||||
### Step 3 — 报告问题清单
|
||||
|
||||
在修改前,先以列表形式输出所有发现的问题:
|
||||
|
||||
```
|
||||
发现 N 个问题:
|
||||
|
||||
[文件] js/foo.js
|
||||
· L34, L78: 重复逻辑 — 两处都实现了相同的 DOM 查询,可提取到 getPanel()
|
||||
· L91: Magic number — 硬编码 14 作为偏移量,应命名为 TOOLTIP_OFFSET
|
||||
|
||||
[文件] js/bar.js
|
||||
· L12: 命名问题 — 变量 `or_` 语义不明,应命名为 outerR/outerG/outerB
|
||||
...
|
||||
```
|
||||
|
||||
如果没有发现问题,直接输出"未发现垃圾代码,当前代码质量良好。"并停止。
|
||||
|
||||
### Step 4 — 执行修复
|
||||
|
||||
对每个问题,使用 Edit 工具进行**最小化修改**:
|
||||
|
||||
- **重复逻辑**:提取为共享常量/函数,更新所有调用点
|
||||
- **Magic number**:在文件顶部或逻辑附近声明 `const NAME = value`,替换所有引用
|
||||
- **命名问题**:重命名变量,更新所有使用处
|
||||
- **死代码**:直接删除
|
||||
- **尾部空白/风格**:修正
|
||||
- **未 export 的函数**:添加 `export`,在调用方改为导入(不重复实现)
|
||||
|
||||
**修复原则:**
|
||||
- 只改在审查清单中发现的问题,不做额外优化
|
||||
- 每次 Edit 只修改确实有问题的行,保持 diff 最小
|
||||
- 改完后用 `grep` 验证旧的坏代码已消失
|
||||
- 优先做精确补丁;只有仓库已有对应格式化流程时,才运行格式化工具
|
||||
|
||||
### Step 5 — 输出总结
|
||||
|
||||
```
|
||||
清理完成:
|
||||
|
||||
修复了 N 个问题:
|
||||
✓ earth.js — 提取重复 vertexShader 为 ATMOS_VERTEX_SHADER 常量
|
||||
✓ main.js — 提取 TOOLTIP_CURSOR_OFFSET = 14(4处引用)
|
||||
✓ controls.js — export updateLayerButtonState,移除 main.js 中的重复实现
|
||||
...
|
||||
|
||||
未修改的问题(需人工确认):
|
||||
! foo.js L45 — 注释代码块较长,建议手动确认是否可删除
|
||||
```
|
||||
|
||||
## 约束
|
||||
|
||||
- **禁止**改变函数签名、接口定义、导出 API(除非问题正是私有函数应被 export)
|
||||
- **禁止**添加新功能、新抽象、新参数
|
||||
- **禁止**修改注释内容(只删除注释掉的死代码)
|
||||
- **禁止**修改测试文件逻辑
|
||||
- 如果一个 Magic number 的语义不完全确定,**跳过**,在总结中标记为"需人工确认"
|
||||
@@ -0,0 +1,170 @@
|
||||
---
|
||||
description: 分析本次 git 变更,在 docs/technical/zh/ 中新建或更新对应的技术文档
|
||||
argument-hint: 可选:指定要记录的主题,或留空自动从 git diff 推断
|
||||
allowed-tools: ["Read", "Edit", "Write", "Bash", "Glob", "Grep"]
|
||||
---
|
||||
|
||||
# /docs — 技术文档写入工作流
|
||||
|
||||
## 目标
|
||||
|
||||
根据当前 git 变更(或用户指定主题)在 `docs/technical/zh/` 中写入或更新技术文档,记录**为什么**这样做,而不只是记录做了什么。
|
||||
|
||||
## 执行步骤
|
||||
|
||||
### Step 1 — 理解变更范围
|
||||
|
||||
```bash
|
||||
git diff HEAD --stat # 变更文件一览
|
||||
git diff HEAD --name-only # 变更文件列表
|
||||
git log --oneline -10 # 近期 commit 上下文
|
||||
```
|
||||
|
||||
若 `$ARGUMENTS` 指定了主题,优先聚焦该主题;否则从文件列表和 diff stat 推断变更主题。不要默认读取完整仓库 diff;只对决定文档主题所需的文件读取 focused diff:
|
||||
|
||||
```bash
|
||||
git diff HEAD -- <path>
|
||||
rg -n "class |def |function |export |router|@router|interface |type " <path>
|
||||
```
|
||||
|
||||
### Step 2 — 确认文档范围
|
||||
|
||||
分析变更,判断:
|
||||
|
||||
1. **应写几篇文档**:单一主题写一篇,跨领域变更可拆分(如后端性能优化 + 运维启动脚本分开写)
|
||||
2. **是新建还是更新**:检查 `docs/technical/zh/` 中是否已有相关文档
|
||||
3. **文档命名**:按 `领域-主题-副题.md` 格式,全小写,用连字符,如:
|
||||
- `backend-datasources-api-performance.md`
|
||||
- `ops-planet-sh-startup.md`
|
||||
- `earth-bgp-context.md`
|
||||
|
||||
```bash
|
||||
ls docs/technical/zh/ # 查看现有文档
|
||||
```
|
||||
|
||||
**先输出写作计划供用户确认**(若变更明确且范围小,可直接执行):
|
||||
|
||||
```
|
||||
文档计划:
|
||||
新建:docs/technical/zh/ops-planet-sh-startup.md — planet.sh 启动性能优化
|
||||
更新:docs/technical/zh/backend-datasources-api-performance.md — 补充并行化细节
|
||||
```
|
||||
|
||||
### Step 2.5 — 覆盖范围检查
|
||||
|
||||
写文档前必须按变更类型检查配套文档,不要只更新一篇专题文档:
|
||||
|
||||
- 用户可见流程变化:更新 `docs/technical/zh/manual.md`,通常也更新 `docs/technical/zh/quickstart.md`。
|
||||
- `manual.md`、`quickstart.md` 这类用户手册存在英文版时,同步更新 `docs/technical/en/...`,至少避免英文版与中文版互相矛盾。
|
||||
- 控制台页面职责、路由入口、表格/抽屉/设置页行为变化:更新 `docs/technical/zh/frontend-admin-frontend-context.md`。
|
||||
- Earth 前端行为、HUD、巡航、图层、图例、交互变化:更新 `docs/technical/zh/earth-frontend-context.md`。
|
||||
- 新增 Earth 图层、调整 `renderOrder`、半径/高度偏移、深度策略、拾取策略、legend mode、图层面板顺序或启动加载顺序:更新 `docs/technical/zh/earth-render-layer-order.md`。
|
||||
- Earth 图层视觉样式、颜色、图例符号语义变化:若影响样式索引,同步更新 `docs/technical/zh/earth-layer-style-reference.md`。
|
||||
- 采集器、数据源、凭证、设置页、连接检查、scheduler、后端 API 变化:更新相关后端文档,优先检查 `docs/technical/zh/backend-collectors.md` 和 datasource/settings 专题文档。
|
||||
- 如果某个旧 plan 的假设已经被当前实现推翻,在对应 `docs/plans/*.md` 增加现状修正或更新该段,不要让计划文档继续给出相反方向。
|
||||
- 新增 technical 文档后,如果需要被发现,更新 `docs/technical/zh/README.md`。
|
||||
- 对本次变更提取旧词做 stale search,例如旧 tab 名、旧路由职责、旧认证假设、改名前 UI 文案:
|
||||
|
||||
```bash
|
||||
rg -n "旧文案|旧路由职责|旧认证假设" docs/technical docs/plans
|
||||
```
|
||||
|
||||
### Step 3 — 写文档
|
||||
|
||||
遵循以下原则:
|
||||
|
||||
**记录 WHY,不只记录 WHAT**
|
||||
- 好:`将戳文件从 /tmp 移到 ~/.cache/planet/,因为 WSL 重启后 /tmp 被清空`
|
||||
- 差:`修改了 AI_PROVIDER_BUILD_STAMP_FILE 的值`
|
||||
|
||||
**必须包含的内容**:
|
||||
- 背景/问题:改动之前存在什么问题,为什么要改
|
||||
- 核心设计决策及其理由
|
||||
- 关键代码片段(用 diff 或 before/after 展示)
|
||||
- 相关文件列表
|
||||
|
||||
**格式要求**:
|
||||
- 使用 `##` 和 `###` 分级,不要超过三级
|
||||
- 代码块注明语言(python / bash / typescript / sql)
|
||||
- 表格用于对比多个选项或列出参数
|
||||
- 中文写作,技术术语保留英文原文
|
||||
- `docs/technical/zh/` 中的文档不得用英文原文占位;如果存在 `docs/technical/en/` 对应文件,禁止逐字复制成中文文件
|
||||
- 中文文档内部链接应指向 `docs/technical/zh/...`,除非明确引用英文专属文档
|
||||
|
||||
**文档结构模板**:
|
||||
|
||||
```markdown
|
||||
# 标题(说明做了什么)
|
||||
|
||||
## 背景
|
||||
|
||||
为什么要做这个改动,改动前存在什么问题。
|
||||
|
||||
## 核心变更
|
||||
|
||||
### 子主题一
|
||||
|
||||
before/after 或决策说明 + 关键代码
|
||||
|
||||
### 子主题二
|
||||
|
||||
...
|
||||
|
||||
## 相关文件
|
||||
|
||||
- `path/to/file.py` — 简短说明
|
||||
```
|
||||
|
||||
### Step 4 — 验证
|
||||
|
||||
- 读一遍写好的文档,确认逻辑清晰、代码片段无明显错误
|
||||
- 用 `rg --files` 或 `test -e` 确认文档中的文件路径在项目中真实存在,避免凭记忆判断:
|
||||
- 检查中文文档没有误复制英文版:
|
||||
|
||||
```bash
|
||||
python - <<'PY'
|
||||
from pathlib import Path
|
||||
same = []
|
||||
for en in sorted(Path("docs/technical/en").glob("*.md")):
|
||||
zh = Path("docs/technical/zh") / en.name
|
||||
if zh.exists() and en.read_text() == zh.read_text():
|
||||
same.append(en.name)
|
||||
if same:
|
||||
raise SystemExit("identical en/zh docs: " + ", ".join(same))
|
||||
print("no identical en/zh docs")
|
||||
PY
|
||||
```
|
||||
|
||||
- 检查中文文档内部链接没有继续指向无语言目录:
|
||||
|
||||
```bash
|
||||
rg -n "/home/ray/dev/linkong/planet/docs/technical/(?!zh|en)" docs/technical/zh --pcre2
|
||||
```
|
||||
|
||||
```bash
|
||||
# 对文档中提到的关键路径做快速验证
|
||||
ls <mentioned_paths>
|
||||
```
|
||||
|
||||
如需检查大量链接,优先用确定性提取:
|
||||
|
||||
```bash
|
||||
rg -n "\]\(([^)]+)\)" docs/technical/zh/<doc>.md
|
||||
```
|
||||
|
||||
### Step 5 — 完成确认
|
||||
|
||||
输出摘要:
|
||||
|
||||
```
|
||||
✓ 新建:docs/technical/zh/ops-planet-sh-startup.md(约 xxx 字)
|
||||
✓ 更新:docs/technical/zh/backend-datasources-api-performance.md
|
||||
```
|
||||
|
||||
## 注意事项
|
||||
|
||||
- 不要写流水账式的"改了 A、改了 B、改了 C",要写改动背后的约束和权衡
|
||||
- 不要在文档中引用 PR 号、issue 号、或当前对话——这些会随时间失效
|
||||
- 代码片段保持简洁,只保留说明问题的关键部分,省略无关样板代码
|
||||
- 如果某个变更已有文档记录,优先在原文档中追加,而不是新建
|
||||
- 文档是给未来的开发者看的,假设读者熟悉项目但不了解这次改动的背景
|
||||
@@ -0,0 +1,93 @@
|
||||
---
|
||||
description: 用 goal-driven 方法推动一个复杂任务持续执行,直到明确成功标准被满足
|
||||
argument-hint: 建议填写任务目标;若同时给出成功标准更好
|
||||
allowed-tools: ["Read", "Edit", "Bash", "Grep", "Glob"]
|
||||
---
|
||||
|
||||
# /goal-driven — 目标驱动执行模式
|
||||
|
||||
使用 `lidangzzz/goal-driven` 的核心思想来推进复杂任务:先固定目标与成功标准,再持续执行和反复验收,直到标准真正满足。
|
||||
|
||||
适用场景:
|
||||
|
||||
- 长周期实现任务
|
||||
- 高复杂度工程任务
|
||||
- 可被明确验收的研究、实现、迁移、验证类工作
|
||||
|
||||
不适用场景:
|
||||
|
||||
- 纯脑暴
|
||||
- 无法定义成功标准的模糊任务
|
||||
- 很小的一次性修改
|
||||
|
||||
## 输入要求
|
||||
|
||||
若 `$ARGUMENTS` 只包含目标,没有成功标准,先补全一版可执行的成功标准再开始。
|
||||
|
||||
启动时先输出:
|
||||
|
||||
```md
|
||||
Goal
|
||||
- ...
|
||||
|
||||
Criteria for success
|
||||
- ...
|
||||
|
||||
Plan
|
||||
1. ...
|
||||
2. ...
|
||||
3. ...
|
||||
|
||||
Verification
|
||||
- ...
|
||||
```
|
||||
|
||||
## 执行规则
|
||||
|
||||
1. 先把任务固化为两个核心块:
|
||||
- `Goal`
|
||||
- `Criteria for success`
|
||||
|
||||
2. 成功标准必须尽量客观,可验证,可落地。
|
||||
优先写成:
|
||||
- 需要交付什么
|
||||
- 需要通过哪些测试或验证
|
||||
- 如何判断结果真的完成
|
||||
|
||||
3. 进入持续执行循环:
|
||||
- 完成一个阶段
|
||||
- 检查当前结果是否满足成功标准
|
||||
- 若未满足,明确剩余差距并继续推进
|
||||
|
||||
4. 任何“完成了”“差不多了”“已实现”之类的结论,都必须经过验证,不能直接接受。
|
||||
|
||||
5. 如果验证失败:
|
||||
- 明确指出哪条成功标准没满足
|
||||
- 继续工作,不要把阶段性进展误判为完成
|
||||
|
||||
6. 只有在以下情况之一才能停止:
|
||||
- 成功标准已满足
|
||||
- 用户明确要求停止
|
||||
|
||||
## 执行风格
|
||||
|
||||
- 重证据,轻口头判断
|
||||
- 优先使用确定性工具证据:`rg`、`git diff --stat`、`git diff -- <path>`、测试、构建、lint、`curl`、数据库查询等能直接证明成功标准的方式
|
||||
- 不把大段命令输出粘进回复;保留在工具调用里,回复只总结关键证据
|
||||
- 重验收,轻自我感觉
|
||||
- 优先用测试、日志、产物、对比结果来证明完成
|
||||
- 对长期任务保持“未达标就继续”的节奏
|
||||
|
||||
## 简版模板
|
||||
|
||||
```md
|
||||
Goal: [[[[[在此填写最终目标]]]]]
|
||||
|
||||
Criteria for success: [[[[[在此填写成功标准]]]]]
|
||||
|
||||
循环执行:
|
||||
1. 推进任务
|
||||
2. 检查是否满足成功标准
|
||||
3. 若未满足,继续工作
|
||||
4. 直到满足标准或用户明确停止
|
||||
```
|
||||
@@ -0,0 +1,160 @@
|
||||
---
|
||||
description: 发版工作流:根据变更类型决定版本号,更新所有版本文件和 changelog,运行验证,commit 并 push
|
||||
argument-hint: 可选:feature | bugfix | 或直接描述本次发布内容
|
||||
allowed-tools: ["Read", "Edit", "Bash", "Glob", "Grep"]
|
||||
---
|
||||
|
||||
# /release — Planet 发版工作流
|
||||
|
||||
## 版本号规则
|
||||
|
||||
| 变更类型 | 版本跳动 | 适用场景 |
|
||||
|---------|---------|---------|
|
||||
| `feature` | `+0.1.0` | 纯新功能,无 bugfix |
|
||||
| `improvement` | `+0.0.1` | UI 调整、小功能增强、bugfix 混合,或以 UI/体验改进为主的迭代 |
|
||||
| `bugfix` | `+0.0.1` | 纯 bug 修复,无新功能 |
|
||||
| `docs` / `maintenance` / `refactor` | 默认不发版,除非用户明确要求 |
|
||||
|
||||
意图混合时以用户明确描述为准;bugfix + 小 feature 混合默认判定为 `improvement`(`+0.0.1`)。
|
||||
|
||||
## 必须同步更新的文件
|
||||
|
||||
使用 `git rev-parse --show-toplevel` 获取仓库根目录,以下路径均相对于根目录:
|
||||
|
||||
- `VERSION`
|
||||
- `frontend/package.json`(`"version"` 字段)
|
||||
- `pyproject.toml`(`version =` 字段)
|
||||
- `uv.lock`(**不要手动编辑**,通过 `uv lock` 重新生成)
|
||||
- `docs/CHANGELOG.md`
|
||||
- `docs/version-history.md`
|
||||
|
||||
## 节省上下文规则
|
||||
|
||||
发版判断应以确定性 CLI 证据为主,优先使用紧凑命令和定点读取:
|
||||
|
||||
```bash
|
||||
git status --short
|
||||
git diff --stat HEAD
|
||||
git diff --name-only HEAD
|
||||
rg -n "version|^## |^Released:|当前开发版本|current" VERSION frontend/package.json pyproject.toml docs/CHANGELOG.md docs/version-history.md
|
||||
```
|
||||
|
||||
除非需要判断某个代码变更是否属于本次发版,否则不要读取完整 diff。
|
||||
|
||||
## 执行步骤
|
||||
|
||||
### Step 1 — 环境检查
|
||||
|
||||
```bash
|
||||
git branch --show-current # 确认在 dev 分支
|
||||
git status --short # 检查是否有无关的未暂存修改
|
||||
cat VERSION # 读取当前版本
|
||||
```
|
||||
|
||||
若当前**不在 `dev` 分支**,停下来告知用户,不要继续。
|
||||
|
||||
若存在无关的未暂存修改,列出并询问用户是否一并提交,或先 stash。
|
||||
|
||||
### Step 2 — 确定发版类型与新版本号
|
||||
|
||||
- 若 `$ARGUMENTS` 提供了明确类型(`feature` / `bugfix`),直接使用
|
||||
- 否则根据 `git diff --stat HEAD`、`git diff --name-only HEAD`、必要的 focused diff 和 `git log` 推断
|
||||
- 计算新版本号(例:`0.26.2` → bugfix → `0.26.3`)
|
||||
- **先输出发版计划供用户确认**:
|
||||
|
||||
```
|
||||
发版计划:
|
||||
类型:bugfix
|
||||
版本:0.26.2 → 0.26.3
|
||||
分支:dev
|
||||
将更新:VERSION, frontend/package.json, pyproject.toml, uv.lock, CHANGELOG.md, version-history.md
|
||||
```
|
||||
|
||||
### Step 3 — 更新版本号文件
|
||||
|
||||
按顺序更新(每步用 Edit 工具,精确替换,不要重写整个文件):
|
||||
|
||||
1. `VERSION` — 直接替换全部内容为新版本号
|
||||
2. `frontend/package.json` — 替换 `"version": "x.x.x"` 行
|
||||
3. `pyproject.toml` — 替换 `version = "x.x.x"` 行
|
||||
4. 运行 `uv lock` 重新生成 `uv.lock`(在仓库根目录下执行)
|
||||
|
||||
### Step 4 — 更新 CHANGELOG.md
|
||||
|
||||
在文件顶部插入新条目,格式:
|
||||
|
||||
```markdown
|
||||
## [x.x.x] — YYYY-MM-DD
|
||||
|
||||
### ✨ Features / 🐛 Fixes / 🔧 Improvements
|
||||
- ...(只列高信号条目,最多 5 条)
|
||||
- ...
|
||||
|
||||
---
|
||||
```
|
||||
|
||||
日期使用 `date +%Y-%m-%d` 获取今天的日期。
|
||||
|
||||
### Step 5 — 更新 docs/version-history.md
|
||||
|
||||
- 更新文件头部的"当前开发版本"字段
|
||||
- 在时间线表格顶部插入新行:`| vx.x.x | YYYY-MM-DD | 一句话摘要 |`
|
||||
|
||||
### Step 6 — 验证
|
||||
|
||||
针对本次变更范围做最小验证:
|
||||
|
||||
- Python 文件有修改:先用 `git diff --name-only HEAD -- '*.py'` 列出,再运行 `python3 -m py_compile <changed_files>`
|
||||
- Frontend 文件有修改:先用 `git diff --name-only HEAD -- frontend` 判断范围,再运行项目标准检查(若无则跳过并说明)
|
||||
- 版本号一致性检查:用 grep 确认 VERSION、package.json、pyproject.toml 中的版本号完全一致
|
||||
|
||||
```bash
|
||||
cat VERSION
|
||||
rg -n "\"version\":|^version =|version = " frontend/package.json pyproject.toml uv.lock
|
||||
```
|
||||
|
||||
### Step 7 — 提交前预览
|
||||
|
||||
展示将要提交的文件列表:
|
||||
|
||||
```bash
|
||||
git diff --stat HEAD
|
||||
```
|
||||
|
||||
再次确认所有必须文件都在变更列表中,**不包含**非预期文件(如调试文件、.env 等)。
|
||||
|
||||
### Step 8 — Commit & Push(用户确认后)
|
||||
|
||||
```bash
|
||||
git add VERSION frontend/package.json pyproject.toml uv.lock docs/CHANGELOG.md docs/version-history.md
|
||||
# 若有代码变更也一并 stage
|
||||
git add <code_files>
|
||||
|
||||
git commit -m "release: bump version to x.x.x"
|
||||
git tag vx.x.x
|
||||
git push origin dev
|
||||
git push origin vx.x.x
|
||||
```
|
||||
|
||||
commit message 固定格式:`release: bump version to x.x.x`
|
||||
|
||||
### Step 9 — 完成确认
|
||||
|
||||
输出摘要:
|
||||
|
||||
```
|
||||
✓ 版本号已更新:0.26.2 → 0.26.3
|
||||
✓ CHANGELOG 已更新
|
||||
✓ version-history 已更新
|
||||
✓ uv.lock 已重新生成
|
||||
✓ 验证通过
|
||||
✓ commit: release: bump version to 0.26.3
|
||||
✓ tag: v0.26.3
|
||||
✓ 已 push 到 origin/dev
|
||||
```
|
||||
|
||||
## 注意事项
|
||||
|
||||
- `uv.lock` 只能通过 `uv lock` 生成,绝不手动编辑
|
||||
- 发版 commit 只包含版本文件 + 本次功能代码,不混入无关改动
|
||||
- 若环境中 `uv` 不可用,说明原因并跳过 lockfile 更新,提醒用户手动运行
|
||||
@@ -0,0 +1,3 @@
|
||||
approval_policy = "never"
|
||||
|
||||
sandbox_mode = "danger-full-access"
|
||||
@@ -0,0 +1,144 @@
|
||||
---
|
||||
name: cleanup
|
||||
description: Use when the user asks to clean up, lint, or review uncommitted code for common code smells — duplicate logic, magic numbers, unclear naming, dead code, style inconsistencies. Fixes issues without changing any runtime behavior.
|
||||
---
|
||||
|
||||
# Cleanup
|
||||
|
||||
Review and fix code quality issues in the current working tree without altering any logic or behavior.
|
||||
|
||||
## When To Use
|
||||
|
||||
- The user asks to clean up, tidy, or lint uncommitted changes
|
||||
- The user wants a code smell review before releasing or committing
|
||||
- The user mentions magic numbers, duplicate logic, dead code, or naming issues
|
||||
|
||||
Do not refactor architecture, add features, or change behavior.
|
||||
|
||||
## Scope
|
||||
|
||||
If the user specifies a file or directory, check only that. Otherwise check all uncommitted changes (`git diff HEAD`).
|
||||
|
||||
Only report issues present in **newly added or modified** lines of this diff — do not audit unchanged code.
|
||||
|
||||
## Token-Saving Rule
|
||||
|
||||
Prefer deterministic CLI checks before reading files into model context:
|
||||
|
||||
```bash
|
||||
git diff --name-only HEAD
|
||||
git diff --unified=0 HEAD -- <path>
|
||||
git diff --check
|
||||
rg -n "TODO|FIXME|console\.log|debugger|print\(" <changed-paths>
|
||||
```
|
||||
|
||||
Read full files only when the focused diff does not provide enough surrounding context to make a safe edit.
|
||||
|
||||
## Checklist
|
||||
|
||||
### 1. Duplicate Logic
|
||||
- Identical or near-identical code blocks appearing in multiple places
|
||||
- A function/helper that already exists but is re-implemented elsewhere instead of being reused
|
||||
- Repeated DOM queries, regex literals, or template strings within the same file
|
||||
|
||||
### 2. Magic Numbers / Magic Strings
|
||||
- Bare numeric literals used in calculations (offsets, timeouts, sizes, thresholds) without a named constant
|
||||
- Hardcoded strings (IDs, status values, URL fragments) scattered through logic
|
||||
- Exceptions: `0`, `1`, `-1`, `100`, `""` and other idiomatically clear values are fine
|
||||
|
||||
### 3. Naming Issues
|
||||
- Cryptic abbreviations (`or_`, `tmp2`, `x2`)
|
||||
- Names that do not match actual behavior
|
||||
- The same concept referred to by different names in different places
|
||||
|
||||
### 4. Dead Code
|
||||
- Commented-out code blocks (3+ lines)
|
||||
- Variables, parameters, or imports declared but never used
|
||||
- Branches that can never execute
|
||||
|
||||
### 5. Style Inconsistencies
|
||||
- Trailing whitespace
|
||||
- Mixed quote styles or indentation within the same file
|
||||
- Inconsistent blank-line usage (multiple consecutive blank lines, etc.)
|
||||
|
||||
### 6. Other
|
||||
- Private helper functions that should be exported but are not, causing callers to duplicate the implementation
|
||||
- Overly verbose conditions that can be simplified without changing logic
|
||||
|
||||
## Steps
|
||||
|
||||
### Step 1 — Get the file list
|
||||
|
||||
```bash
|
||||
git diff HEAD --name-only
|
||||
```
|
||||
|
||||
Filter to the user-specified path if one was provided.
|
||||
|
||||
### Step 2 — Read and analyze each file
|
||||
|
||||
Start with focused diffs:
|
||||
|
||||
```bash
|
||||
git diff --unified=0 HEAD -- <file>
|
||||
```
|
||||
|
||||
Use `rg`, `git diff --check`, and compiler/linter output for deterministic findings. Read the full file only for files that need surrounding context. For each issue found, record filename, line number, category, and suggested fix.
|
||||
|
||||
### Step 3 — Report findings before touching anything
|
||||
|
||||
Print a structured list:
|
||||
|
||||
```
|
||||
Found N issues:
|
||||
|
||||
[file] js/foo.js
|
||||
· L34, L78: Duplicate logic — same DOM query implemented twice; extract to getPanel()
|
||||
· L91: Magic number — bare 14 used as pixel offset; name it TOOLTIP_OFFSET
|
||||
|
||||
[file] js/bar.js
|
||||
· L12: Naming — variable `or_` is unclear; rename to outerR, outerG, outerB
|
||||
...
|
||||
```
|
||||
|
||||
If no issues are found, output "No code smells detected. Code quality looks good." and stop.
|
||||
|
||||
### Step 4 — Fix each issue
|
||||
|
||||
Use the Edit tool for **minimal, targeted changes**:
|
||||
|
||||
- **Duplicate logic**: extract to a shared constant or function; update all call sites
|
||||
- **Magic number/string**: declare `const NAME = value` near the top of the relevant scope; replace all usages
|
||||
- **Naming**: rename the variable/function; update all references
|
||||
- **Dead code**: delete it
|
||||
- **Trailing whitespace / style**: fix in place
|
||||
- **Unexported helper**: add `export`; update callers to import instead of re-implementing
|
||||
|
||||
Principles:
|
||||
- Only fix issues identified in the checklist — no extra improvements
|
||||
- Keep each Edit as small as possible
|
||||
- After fixing, verify the old bad pattern is gone with grep
|
||||
- Prefer `apply_patch` for targeted edits; use formatters only when the repository already uses them for the touched file type
|
||||
|
||||
### Step 5 — Summary
|
||||
|
||||
```
|
||||
Cleanup complete:
|
||||
|
||||
Fixed N issues:
|
||||
✓ earth.js — extracted duplicate vertexShader into ATMOS_VERTEX_SHADER constant
|
||||
✓ main.js — extracted TOOLTIP_CURSOR_OFFSET = 14 (4 references updated)
|
||||
✓ controls.js — exported updateLayerButtonState; removed duplicate implementation in main.js
|
||||
...
|
||||
|
||||
Skipped (needs manual review):
|
||||
! foo.js L45 — large commented-out block; confirm it is safe to delete
|
||||
```
|
||||
|
||||
## Constraints
|
||||
|
||||
- **Do not** change function signatures, exported interfaces, or public APIs (unless the issue is a missing export)
|
||||
- **Do not** add new features, abstractions, or parameters
|
||||
- **Do not** rewrite comments (only delete commented-out dead code)
|
||||
- **Do not** touch test file logic
|
||||
- If a magic number's intent is uncertain, skip it and flag it in the summary
|
||||
@@ -0,0 +1,136 @@
|
||||
---
|
||||
name: docs
|
||||
description: Analyze current Planet repo changes and create or update technical documentation under docs/technical/zh. Use when the user asks to write docs, update technical docs, summarize implementation changes into documentation, or port the Claude docs-codex workflow into Codex.
|
||||
---
|
||||
|
||||
# Docs
|
||||
|
||||
Use this skill when the user asks to create or update Planet technical documentation, especially under `docs/technical/zh/`.
|
||||
|
||||
## Goal
|
||||
|
||||
Write or update technical docs that explain why a change exists, not only what files changed.
|
||||
|
||||
Default target directory:
|
||||
|
||||
- `docs/technical/zh/`
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Gather change context:
|
||||
|
||||
```bash
|
||||
git diff HEAD --stat
|
||||
git diff HEAD --name-only
|
||||
git log --oneline -10
|
||||
ls docs/technical/zh/
|
||||
```
|
||||
|
||||
If the user gives a specific topic, focus on that topic. Otherwise infer the documentation topic from the file list and diff stat. Do **not** read the full repository diff by default; inspect focused diffs only for the files that define the doc topic:
|
||||
|
||||
```bash
|
||||
git diff HEAD -- <path>
|
||||
rg -n "class |def |function |export |router|@router|interface |type " <path>
|
||||
```
|
||||
|
||||
2. Decide document scope:
|
||||
|
||||
- Use one document for one coherent topic.
|
||||
- Split documents when the changes cross meaningful domains, such as backend performance and ops startup behavior.
|
||||
- Prefer updating an existing relevant doc over creating a duplicate.
|
||||
- Name new files as lowercase hyphenated `domain-topic-detail.md`, for example:
|
||||
- `backend-datasources-api-performance.md`
|
||||
- `ops-planet-sh-startup.md`
|
||||
- `earth-bgp-context.md`
|
||||
|
||||
3. Apply the documentation coverage checklist before writing:
|
||||
|
||||
- User-visible workflow changes must update `docs/technical/zh/manual.md` and usually `docs/technical/zh/quickstart.md`.
|
||||
- If an English counterpart exists for user-facing docs such as `manual.md` or `quickstart.md`, update `docs/technical/en/...` enough that it does not contradict the Chinese source.
|
||||
- Control console page responsibility changes must update `docs/technical/zh/frontend-admin-frontend-context.md`.
|
||||
- Earth frontend behavior changes must update `docs/technical/zh/earth-frontend-context.md`.
|
||||
- Earth layer additions, `renderOrder`, altitude/radius offsets, depth strategy, pointer picking, legend modes, or layer panel/startup ordering must update `docs/technical/zh/earth-render-layer-order.md`.
|
||||
- Earth layer visual style or legend symbol/color semantics should also update `docs/technical/zh/earth-layer-style-reference.md` when that reference is affected.
|
||||
- Collector, datasource, credential, settings, connectivity, scheduler, or API changes must update the relevant backend docs, especially `docs/technical/zh/backend-collectors.md` and any datasource/settings-specific doc.
|
||||
- When a change turns an old plan assumption into current behavior, update the relevant `docs/plans/*.md` with a status note instead of leaving contradictory instructions.
|
||||
- If adding a new technical document, add it to `docs/technical/zh/README.md` when it should be discoverable from the technical docs index.
|
||||
- Search docs for stale terms introduced by the change, for example old tab names, old route responsibilities, obsolete auth assumptions, or renamed UI labels.
|
||||
|
||||
4. Write the doc in Chinese:
|
||||
|
||||
- Write Chinese prose for `docs/technical/zh/`.
|
||||
- Keep technical identifiers, API paths, config keys, code symbols, and standard product names in English where appropriate.
|
||||
- Use `##` and `###` headings; avoid going deeper than three levels.
|
||||
- Use fenced code blocks with language tags.
|
||||
- Use tables when comparing options or listing parameters.
|
||||
|
||||
5. Required content:
|
||||
|
||||
- Background/problem: what was wrong before and why the change was needed.
|
||||
- Core design decisions and rationale.
|
||||
- Key code snippets, preferably before/after or focused excerpts.
|
||||
- Related files and what each file contributes.
|
||||
|
||||
6. Verification:
|
||||
|
||||
- Read the completed doc and check that the reasoning is clear.
|
||||
- Verify important referenced paths exist.
|
||||
- Use `rg --files` or `test -e` for path existence instead of relying on memory.
|
||||
- Run a quick duplicate-language check when editing bilingual docs:
|
||||
|
||||
```bash
|
||||
python - <<'PY'
|
||||
from pathlib import Path
|
||||
same = []
|
||||
for en in sorted(Path("docs/technical/en").glob("*.md")):
|
||||
zh = Path("docs/technical/zh") / en.name
|
||||
if zh.exists() and en.read_text() == zh.read_text():
|
||||
same.append(en.name)
|
||||
if same:
|
||||
raise SystemExit("identical en/zh docs: " + ", ".join(same))
|
||||
print("no identical en/zh docs")
|
||||
PY
|
||||
```
|
||||
|
||||
Also check that Chinese docs do not link to the old language-less technical docs path:
|
||||
|
||||
```bash
|
||||
rg -n "/home/ray/dev/linkong/planet/docs/technical/(?!zh|en)" docs/technical/zh --pcre2
|
||||
```
|
||||
|
||||
This command should return no matches.
|
||||
|
||||
If checking many links, prefer deterministic extraction:
|
||||
|
||||
```bash
|
||||
rg -n "\]\(([^)]+)\)" docs/technical/zh/<doc>.md
|
||||
```
|
||||
|
||||
Also run focused stale-term searches derived from the change, for example:
|
||||
|
||||
```bash
|
||||
rg -n "old label|old route purpose|obsolete provider assumption" docs/technical docs/plans
|
||||
```
|
||||
|
||||
## Hard Constraints
|
||||
|
||||
- A file under `docs/technical/zh/` must not be an English source file copied as a placeholder.
|
||||
- Do not leave a Chinese doc with only an English title and English first-screen content.
|
||||
- When an English counterpart exists in `docs/technical/en/`, never duplicate it byte-for-byte into `docs/technical/zh/`.
|
||||
- Internal links inside `docs/technical/zh/` should point to `docs/technical/zh/...` for Chinese docs, unless intentionally linking to an English-only file.
|
||||
- Do not reference PR numbers, issue numbers, or the current conversation.
|
||||
- Do not write changelog-style lists like "changed A, changed B, changed C" without the constraints and tradeoffs behind those changes.
|
||||
- Keep code snippets concise and relevant.
|
||||
|
||||
## Recommended Output
|
||||
|
||||
After editing, summarize:
|
||||
|
||||
```md
|
||||
Updated:
|
||||
- docs/technical/zh/example.md — what changed
|
||||
|
||||
Verified:
|
||||
- no identical en/zh docs
|
||||
- no language-less docs/technical links in zh docs
|
||||
```
|
||||
Executable
+103
@@ -0,0 +1,103 @@
|
||||
---
|
||||
name: goal-driven
|
||||
description: Run a goal-driven execution loop for very large, long-horizon, rigorously verifiable tasks. Use when the user explicitly wants the lidangzzz/goal-driven method, a master-agent plus worker-agent style workflow, or a persistent loop that keeps working until concrete success criteria are satisfied.
|
||||
---
|
||||
|
||||
# Goal-Driven
|
||||
|
||||
Use this skill when the user wants a strict goal-driven workflow for a hard task with:
|
||||
|
||||
- one clear end goal
|
||||
- explicit success criteria
|
||||
- repeated verification against those criteria
|
||||
- continued execution until the criteria are actually met
|
||||
|
||||
This skill is adapted from `lidangzzz/goal-driven`, but trimmed for local skill use to avoid bloating context.
|
||||
|
||||
## When To Use
|
||||
|
||||
Use it for tasks like:
|
||||
|
||||
- compilers, interpreters, theorem-like proof work, deep refactors
|
||||
- long-running system design or implementation work
|
||||
- problems that are expensive and complex, but still objectively testable
|
||||
|
||||
Do not use it for:
|
||||
|
||||
- vague brainstorming without a success condition
|
||||
- short one-shot edits
|
||||
- tasks where "done" cannot be evaluated in a meaningful way
|
||||
|
||||
## Core Model
|
||||
|
||||
The workflow has two roles:
|
||||
|
||||
1. Master role
|
||||
Defines the goal, defines the success criteria, audits progress, and decides whether the work is actually complete.
|
||||
|
||||
2. Worker role
|
||||
Keeps advancing the task toward the goal. If a result is partial, stalled, or unverifiable, the worker continues.
|
||||
|
||||
In Codex, only use actual subagents when the user explicitly asks for delegation or subagent work and the platform supports it. Otherwise emulate the same loop locally: keep working, checkpointing, and re-verifying until the criteria are satisfied.
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Normalize the task into two blocks:
|
||||
- `Goal`
|
||||
- `Criteria for success`
|
||||
|
||||
2. Make the criteria concrete and testable.
|
||||
Good criteria usually include:
|
||||
- required outputs
|
||||
- required validations or tests
|
||||
- edge cases or coverage thresholds
|
||||
- what evidence proves completion
|
||||
|
||||
3. Break the work into milestones that can each produce evidence.
|
||||
|
||||
4. Execute the next milestone.
|
||||
If subagents are explicitly allowed, the master may delegate bounded worker tasks.
|
||||
If not, do the work locally but keep the master/worker mindset.
|
||||
|
||||
5. Whenever work pauses, stalls, or appears complete, audit against the criteria directly.
|
||||
Check artifacts, tests, logs, diffs, metrics, or other real evidence.
|
||||
|
||||
6. If the criteria are not met, continue with a specific delta:
|
||||
- what is still missing
|
||||
- what evidence failed
|
||||
- what the next worker pass must improve
|
||||
|
||||
7. Stop only when the criteria are met, or when the user explicitly stops the process.
|
||||
|
||||
## Operating Rules
|
||||
|
||||
- Prefer objective checks over self-reported completion.
|
||||
- Prefer deterministic tool evidence over long model summaries: use `rg`, `git diff --stat`, targeted `git diff -- <path>`, tests, builds, linters, `curl`, or database queries when they can prove a criterion.
|
||||
- Do not paste large command output into the conversation; summarize the evidence and keep raw output in tool calls.
|
||||
- Do not confuse progress with completion.
|
||||
- If the worker says "done", verify it.
|
||||
- If verification fails, continue from the gap instead of restarting blindly.
|
||||
- Keep the goal stable unless the user changes it.
|
||||
- Tighten fuzzy criteria before sinking large amounts of effort.
|
||||
|
||||
## Recommended Response Shape
|
||||
|
||||
When starting a goal-driven task, structure the kickoff like this:
|
||||
|
||||
```md
|
||||
Goal
|
||||
- ...
|
||||
|
||||
Criteria for success
|
||||
- ...
|
||||
|
||||
Current plan
|
||||
1. ...
|
||||
2. ...
|
||||
3. ...
|
||||
|
||||
Verification
|
||||
- What evidence will prove completion
|
||||
```
|
||||
|
||||
For a reusable prompt template, read [references/prompt-template.md](references/prompt-template.md).
|
||||
@@ -0,0 +1,7 @@
|
||||
interface:
|
||||
display_name: "Goal-Driven"
|
||||
short_description: "Drive complex work until explicit success criteria are met."
|
||||
default_prompt: "Use $goal-driven to turn this task into a concrete goal, explicit success criteria, and a verification-driven execution loop."
|
||||
|
||||
policy:
|
||||
allow_implicit_invocation: true
|
||||
+38
@@ -0,0 +1,38 @@
|
||||
# Goal-Driven Prompt Template
|
||||
|
||||
Use this when you want a reusable kickoff prompt for a master/worker execution loop.
|
||||
|
||||
```md
|
||||
# Goal-Driven System
|
||||
|
||||
Goal: [[[[[DEFINE THE FINAL GOAL HERE]]]]]
|
||||
|
||||
Criteria for success: [[[[[DEFINE THE SUCCESS CRITERIA HERE]]]]]
|
||||
|
||||
You are the master agent.
|
||||
|
||||
Your job is to:
|
||||
1. Keep the goal and criteria fixed.
|
||||
2. Start worker execution toward the goal.
|
||||
3. Audit any claimed progress against the criteria.
|
||||
4. If the criteria are not met, continue the work with a precise next delta.
|
||||
5. Stop only when the criteria are satisfied or the user explicitly stops the process.
|
||||
|
||||
Worker requirements:
|
||||
1. Break the task into subproblems.
|
||||
2. Keep producing concrete progress toward the goal.
|
||||
3. Report evidence, not just claims.
|
||||
4. Continue until the criteria are satisfied.
|
||||
|
||||
Master audit loop:
|
||||
1. Check whether the worker is still making progress.
|
||||
2. If the worker stalls or claims completion, verify against the criteria.
|
||||
3. If verification fails, resume work from the remaining gap.
|
||||
4. Repeat until the criteria are met.
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- Stronger criteria produce better results than stronger rhetoric.
|
||||
- Prefer measurable checks such as tests, parity checks, generated artifacts, benchmarks, or reviewable outputs.
|
||||
- If the environment does not support subagents, emulate the same loop locally.
|
||||
@@ -0,0 +1,173 @@
|
||||
---
|
||||
name: release
|
||||
description: Use when the user asks to release, bump version, update changelog/version files, or commit/push a repository release for the Planet repo. Determines version bump type from changes, updates all required version-bearing files, updates changelog and version-history, runs minimal validation, then commits, tags, and pushes.
|
||||
---
|
||||
|
||||
# Release Workflow
|
||||
|
||||
Use this skill for release-oriented work in this repository.
|
||||
|
||||
## When To Use
|
||||
|
||||
- The user asks to `发版`
|
||||
- The user asks to bump a version
|
||||
- The user asks to update `CHANGELOG`, `version-history`, or version files as part of a release
|
||||
- The user asks to commit/push a release or a publishable bugfix/feature bundle
|
||||
|
||||
Do not use this skill for ordinary commits that are not being released.
|
||||
|
||||
## Versioning Rules
|
||||
|
||||
- `feature` -> bump minor and reset patch to `0` (`x.y.z` → `x.(y+1).0`; for example `0.41.2` → `0.42.0`)
|
||||
- `bugfix` -> bump `+0.0.1`
|
||||
- `docs`, `maintenance`, and `refactor` do not bump by default unless the user explicitly wants a release
|
||||
|
||||
When intent is mixed, prefer the user's stated release intent.
|
||||
|
||||
## Required Files
|
||||
|
||||
Use `git rev-parse --show-toplevel` to get the repo root. All paths are relative to it:
|
||||
|
||||
- `VERSION`
|
||||
- `frontend/package.json` (`"version"` field)
|
||||
- `pyproject.toml` (`version =` field)
|
||||
- `uv.lock` (**never edit manually** — regenerate by running `uv lock`)
|
||||
- `docs/CHANGELOG.md`
|
||||
- `docs/version-history.md`
|
||||
|
||||
## Token-Saving Rule
|
||||
|
||||
Release work should be driven by deterministic CLI evidence. Prefer compact commands and targeted file reads:
|
||||
|
||||
```bash
|
||||
git status --short
|
||||
git diff --stat HEAD
|
||||
git diff --name-only HEAD
|
||||
rg -n "version|^## |^Released:|current" VERSION frontend/package.json pyproject.toml docs/CHANGELOG.md docs/version-history.md
|
||||
```
|
||||
|
||||
Do not inspect full diffs unless deciding whether changed code belongs in the release.
|
||||
|
||||
## Workflow
|
||||
|
||||
### Step 1 — Environment check
|
||||
|
||||
```bash
|
||||
git branch --show-current # must be on dev
|
||||
git status --short # check for unrelated uncommitted changes
|
||||
cat VERSION # read current version
|
||||
```
|
||||
|
||||
If not on `dev`, stop and tell the user. Do not proceed.
|
||||
|
||||
If unrelated uncommitted changes exist, list them and ask the user whether to include them or stash first.
|
||||
|
||||
### Step 2 — Determine release type and next version
|
||||
|
||||
- If the user provided an explicit type (`feature` / `bugfix`), use it
|
||||
- Otherwise infer from `git diff --stat HEAD`, `git diff --name-only HEAD`, focused diffs for changed code, and recent `git log`
|
||||
- Compute the next version:
|
||||
- `feature`: increment minor and reset patch to `0` (e.g. `0.41.2` → `0.42.0`)
|
||||
- `bugfix`: increment patch only (e.g. `0.26.2` → `0.26.3`)
|
||||
- **Show the release plan before making any changes:**
|
||||
|
||||
```
|
||||
Release plan:
|
||||
Type: bugfix
|
||||
Version: 0.26.2 → 0.26.3
|
||||
Branch: dev
|
||||
Will update: VERSION, frontend/package.json, pyproject.toml, uv.lock, CHANGELOG.md, version-history.md
|
||||
```
|
||||
|
||||
### Step 3 — Update version files
|
||||
|
||||
Update in order (use Edit for precise replacement, never rewrite whole files):
|
||||
|
||||
1. `VERSION` — replace entire content with new version string
|
||||
2. `frontend/package.json` — replace `"version": "x.x.x"` line
|
||||
3. `pyproject.toml` — replace `version = "x.x.x"` line
|
||||
4. Run `uv lock` at repo root to regenerate `uv.lock`
|
||||
|
||||
### Step 4 — Update CHANGELOG.md
|
||||
|
||||
Insert a new entry at the top of the file:
|
||||
|
||||
```markdown
|
||||
## x.x.x
|
||||
|
||||
Released: YYYY-MM-DD
|
||||
|
||||
### Highlights
|
||||
|
||||
- ...
|
||||
|
||||
### Added / Fixed / Improved
|
||||
|
||||
- ... (high-signal items only, max 5)
|
||||
|
||||
---
|
||||
```
|
||||
|
||||
Get today's date with `date +%Y-%m-%d`.
|
||||
|
||||
### Step 5 — Update docs/version-history.md
|
||||
|
||||
- Update the "current dev version" field in the file header
|
||||
- Insert a new row at the top of the timeline table: `| vx.x.x | YYYY-MM-DD | one-line summary |`
|
||||
|
||||
### Step 6 — Validate
|
||||
|
||||
Run the smallest relevant validation for the changes in scope:
|
||||
|
||||
- Python files changed: list changed Python files with `git diff --name-only HEAD -- '*.py'`, then run `python3 -m py_compile <changed_files>`
|
||||
- Frontend files changed: list changed frontend files with `git diff --name-only HEAD -- frontend`, then run the project-standard check if available; otherwise skip and say so
|
||||
- Version consistency: confirm VERSION, package.json, pyproject.toml, and uv.lock all show the same version
|
||||
|
||||
```bash
|
||||
cat VERSION
|
||||
rg -n "\"version\":|^version =|version = " frontend/package.json pyproject.toml uv.lock
|
||||
```
|
||||
|
||||
### Step 7 — Pre-commit preview
|
||||
|
||||
Show what will be committed:
|
||||
|
||||
```bash
|
||||
git diff --stat HEAD
|
||||
```
|
||||
|
||||
Confirm all required files are present and no unexpected files (debug files, `.env`, etc.) are included.
|
||||
|
||||
### Step 8 — Commit, tag, and push
|
||||
|
||||
```bash
|
||||
git add VERSION frontend/package.json pyproject.toml uv.lock docs/CHANGELOG.md docs/version-history.md
|
||||
# also stage any code changes included in this release
|
||||
git add <code_files>
|
||||
|
||||
git commit -m "release: bump version to x.x.x"
|
||||
git tag vx.x.x
|
||||
git push origin dev
|
||||
git push origin vx.x.x
|
||||
```
|
||||
|
||||
Commit message format is fixed: `release: bump version to x.x.x`
|
||||
|
||||
### Step 9 — Completion summary
|
||||
|
||||
```
|
||||
✓ Version bumped: 0.26.2 → 0.26.3
|
||||
✓ CHANGELOG updated
|
||||
✓ version-history updated
|
||||
✓ uv.lock regenerated
|
||||
✓ Validation passed
|
||||
✓ commit: release: bump version to 0.26.3
|
||||
✓ tag: v0.26.3
|
||||
✓ Pushed to origin/dev
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- `uv.lock` must only be updated by running `uv lock`, never manually
|
||||
- The release commit should include only version files + the code for this release — no unrelated changes
|
||||
- If `uv` is unavailable in the environment, say so explicitly and remind the user to run it manually
|
||||
@@ -145,3 +145,8 @@ docs/.venv/
|
||||
*.temp
|
||||
tmp/
|
||||
temp/
|
||||
|
||||
# ----------------------
|
||||
# Runtime Data
|
||||
# ----------------------
|
||||
data/ai/bgp-briefs/
|
||||
@@ -102,6 +102,13 @@
|
||||
| Axios | HTTP 客户端 |
|
||||
| Socket.io-client | WebSocket 客户端 |
|
||||
| ECharts | 统计图表 |
|
||||
| Bun | 前端包管理与脚本运行 |
|
||||
|
||||
前端工程统一使用 Bun:
|
||||
|
||||
- 安装依赖使用 `bun install`
|
||||
- 运行脚本使用 `bun run <script>`
|
||||
- 不使用 `npm`、`pnpm`、`yarn`
|
||||
|
||||
### 虚幻引擎客户端
|
||||
|
||||
@@ -205,10 +212,161 @@
|
||||
./planet.sh health
|
||||
```
|
||||
|
||||
前端命令约定:
|
||||
|
||||
```bash
|
||||
cd frontend
|
||||
bun install
|
||||
bun run dev
|
||||
bun run build
|
||||
```
|
||||
|
||||
不要使用 `npm run ...`,避免在 WSL/Windows 混合环境里触发 `cmd.exe` 路径兼容问题。
|
||||
|
||||
## API 文档
|
||||
|
||||
启动服务后访问: `http://localhost:8000/docs`
|
||||
|
||||
## WSL / Windows 局域网访问
|
||||
|
||||
如果服务运行在 WSL 中,而你希望:
|
||||
|
||||
- Windows 本机浏览器访问开发服务
|
||||
- 同一局域网内的手机或其他电脑访问开发服务
|
||||
|
||||
推荐按下面顺序排查和配置。
|
||||
|
||||
### 1. 在 WSL 中启动服务
|
||||
|
||||
```bash
|
||||
./planet.sh start --allow-lan
|
||||
```
|
||||
|
||||
这会让前端监听 `0.0.0.0:3000`,后端监听 `0.0.0.0:8000`。
|
||||
|
||||
### 2. 先确认 WSL 内部服务正常
|
||||
|
||||
在 WSL 中执行:
|
||||
|
||||
```bash
|
||||
curl http://localhost:3000
|
||||
curl http://localhost:8000/health
|
||||
ss -ltnp | grep -E ':3000|:8000'
|
||||
```
|
||||
|
||||
预期:
|
||||
|
||||
- `3000` 返回前端 HTML
|
||||
- `8000/health` 返回健康检查 JSON
|
||||
- `ss` 中能看到 `0.0.0.0:3000` 和 `0.0.0.0:8000`
|
||||
|
||||
如果这一步不通,先不要继续做 Windows 转发。
|
||||
|
||||
### 3. 在 Windows 本机验证 localhost 直通
|
||||
|
||||
在 Windows PowerShell 中执行:
|
||||
|
||||
```powershell
|
||||
curl http://localhost:3000
|
||||
curl http://localhost:8000/health
|
||||
```
|
||||
|
||||
在常见的 WSL2 开发环境下,Windows 通常可以直接通过 `localhost` 访问 WSL 中的服务。
|
||||
|
||||
### 4. 如果需要让局域网设备访问,再做 Windows 端口转发
|
||||
|
||||
注意:下面的命令必须在“以管理员身份运行”的 PowerShell 中执行。
|
||||
|
||||
先把 Windows 对外网卡上的 `3000` / `8000` 转发到 Windows 本机 `127.0.0.1`:
|
||||
|
||||
```powershell
|
||||
netsh interface portproxy delete v4tov4 listenaddress=0.0.0.0 listenport=3000
|
||||
netsh interface portproxy delete v4tov4 listenaddress=0.0.0.0 listenport=8000
|
||||
|
||||
netsh interface portproxy add v4tov4 listenaddress=0.0.0.0 listenport=3000 connectaddress=127.0.0.1 connectport=3000
|
||||
netsh interface portproxy add v4tov4 listenaddress=0.0.0.0 listenport=8000 connectaddress=127.0.0.1 connectport=8000
|
||||
```
|
||||
|
||||
再放行 Windows 防火墙:
|
||||
|
||||
```powershell
|
||||
New-NetFirewallRule -DisplayName "WSL Planet 3000" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 3000
|
||||
New-NetFirewallRule -DisplayName "WSL Planet 8000" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 8000
|
||||
```
|
||||
|
||||
检查转发规则是否生效:
|
||||
|
||||
```powershell
|
||||
netsh interface portproxy show all
|
||||
```
|
||||
|
||||
预期能看到:
|
||||
|
||||
- `0.0.0.0:3000 -> 127.0.0.1:3000`
|
||||
- `0.0.0.0:8000 -> 127.0.0.1:8000`
|
||||
|
||||
### 5. 查 Windows 局域网 IP,并让其他设备访问
|
||||
|
||||
在 Windows PowerShell 中执行:
|
||||
|
||||
```powershell
|
||||
ipconfig
|
||||
```
|
||||
|
||||
找到当前联网网卡的 IPv4 地址,例如 `192.168.8.228`。
|
||||
|
||||
局域网其他设备可访问:
|
||||
|
||||
- `http://<Windows局域网IP>:3000/earth`
|
||||
- `http://<Windows局域网IP>:3000/admin`
|
||||
|
||||
例如:
|
||||
|
||||
- `http://192.168.8.228:3000/earth`
|
||||
|
||||
### 6. 常见现象与判断
|
||||
|
||||
- WSL 中 `curl localhost:3000` 能通,但 Windows 访问 `WSL 的局域网 IP:3000` 不通:这是正常现象之一,优先验证 Windows 的 `localhost:3000`
|
||||
- Windows `localhost:3000` 能通,但局域网设备访问 `Windows 局域网 IP:3000` 不通:通常缺少 `portproxy` 或防火墙放行
|
||||
- `whoami /groups` 中 `S-1-5-32-544` 显示 `deny only`:说明当前 PowerShell 不是提权管理员窗口
|
||||
|
||||
### 7. 本项目一次性验证顺序
|
||||
|
||||
建议固定按这个顺序验证:
|
||||
|
||||
1. WSL 中执行 `curl http://localhost:3000`
|
||||
2. WSL 中执行 `curl http://localhost:8000/health`
|
||||
3. Windows 中执行 `curl http://localhost:3000`
|
||||
4. Windows 中执行 `curl http://localhost:8000/health`
|
||||
5. 管理员 PowerShell 配置 `portproxy` 和防火墙
|
||||
6. 用手机或其他电脑访问 `http://<Windows局域网IP>:3000/earth`
|
||||
|
||||
## 启动容错参数
|
||||
|
||||
`planet.sh` 现在为依赖安装、数据库、AI Provider 启动加入了有限次重试,并会在数据库与 `aiprovider` 启动后额外等待 Docker healthcheck。
|
||||
|
||||
可通过环境变量临时调整:
|
||||
|
||||
```bash
|
||||
# 例: 放宽 AI Provider 与数据库在网络抖动下的自愈次数
|
||||
AI_PROVIDER_START_MAX_RETRIES=5 \
|
||||
AI_PROVIDER_RETRY_INTERVAL=10 \
|
||||
DATABASE_START_MAX_RETRIES=5 \
|
||||
DATABASE_RETRY_INTERVAL=10 \
|
||||
./planet.sh restart
|
||||
```
|
||||
|
||||
常用参数:
|
||||
|
||||
- `DEPENDENCY_INSTALL_MAX_RETRIES` / `DEPENDENCY_INSTALL_RETRY_INTERVAL`: 控制 `uv sync`、`bun install` 的重试次数与间隔,默认 `3` 次、`5` 秒
|
||||
- `DATABASE_START_MAX_RETRIES` / `DATABASE_RETRY_INTERVAL`: 控制 `postgres`、`redis` 的启动/重启与健康检查自愈,默认 `3` 次、`5` 秒
|
||||
- `AI_PROVIDER_START_MAX_RETRIES` / `AI_PROVIDER_RETRY_INTERVAL`: 控制 `aiprovider` 的构建/启动与容器重启自愈,默认 `3` 次、`5` 秒
|
||||
- `BACKEND_MAX_RETRIES`: 控制后端进程启动重试次数,默认 `3`
|
||||
- `FRONTEND_MAX_RETRIES`: 控制前端 dev server 启动重试次数,默认 `3`
|
||||
- `BACKEND_HEALTH_CHECK_ATTEMPTS` / `BACKEND_HEALTH_CHECK_INTERVAL`: 控制后端 HTTP 健康检查等待次数与间隔,默认 `10` 次、`2` 秒
|
||||
- `FRONTEND_HEALTH_CHECK_ATTEMPTS` / `FRONTEND_HEALTH_CHECK_INTERVAL`: 控制前端 HTTP 可访问检查等待次数与间隔,默认 `10` 次、`2` 秒
|
||||
- `AI_PROVIDER_HEALTH_CHECK_ATTEMPTS` / `AI_PROVIDER_HEALTH_CHECK_INTERVAL`: 控制 `aiprovider` HTTP 健康检查等待次数与间隔,默认 `10` 次、`2` 秒
|
||||
|
||||
## AI 接口预留
|
||||
|
||||
项目现在采用“两层”设计:
|
||||
@@ -284,8 +442,25 @@ AI_PROVIDER_SERVICE_TOKEN=change_me
|
||||
|
||||
详细文档:
|
||||
|
||||
- [docs/aiprovider.md](/home/ray/dev/linkong/planet/docs/aiprovider.md)
|
||||
- [docs/technical/agents-aiprovider.md](/home/ray/dev/linkong/planet/docs/technical/agents-aiprovider.md)
|
||||
- [aiprovider/README.md](/home/ray/dev/linkong/planet/aiprovider/README.md)
|
||||
- [docs/technical/frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md)
|
||||
- [docs/plans/frontend-ai-playground-development-plan.md](/home/ray/dev/linkong/planet/docs/plans/frontend-ai-playground-development-plan.md)
|
||||
- [docs/plans/agents-situational-awareness-foundation-plan.md](/home/ray/dev/linkong/planet/docs/plans/agents-situational-awareness-foundation-plan.md)
|
||||
|
||||
## 前端页面布局规范
|
||||
|
||||
管理后台页面默认遵循“单屏工作区”原则:
|
||||
|
||||
- 页头、摘要区、主工作区应在一屏内形成稳定结构
|
||||
- 主表格 / 主图表 / 主分析区应占据页面主要可视空间
|
||||
- 模块内容超出时优先在卡片、表格、标签页内部滚动
|
||||
- 不依赖整页纵向撑开来容纳主要工作区
|
||||
|
||||
当前推荐参考实现:
|
||||
|
||||
- [frontend/src/pages/BGP/BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx)
|
||||
- [docs/technical/frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md)
|
||||
|
||||
## License
|
||||
|
||||
|
||||
@@ -13,7 +13,26 @@
|
||||
- [x] 接入 `IPtoASN / IPtoCountry` 作为 prefix-centric geography 的主数据源
|
||||
- [x] 接入 `OpenGeoFeed` 作为 prefix geography 的高质量覆盖/override 数据源
|
||||
- [x] 把 RIR delegated 设计成 prefix geography 的 fallback,而不是主来源
|
||||
- [ ] 为 `aiprovider` 建立 `provider -> api adapter -> compat policy` 的配置中心,优先落成 `json` 或 `yaml` 文件,运行时按 `provider/model` 读取兼容设置,而不是把专项兼容继续散落在 Python 分支里
|
||||
- [ ] 为市面上主流 AI 服务补专项兼容配置并固化到配置文件中,至少覆盖 `OpenAI / Anthropic / MiniMax / Ollama / Moonshot / DeepSeek / Qwen / GLM / Gemini / OpenRouter / vLLM / LM Studio / One API`
|
||||
- [ ] 在兼容配置中补齐可声明项:`api adapter`、`base_url pattern`、`auth header`、`thinking default`、`reasoning block mapping`、`stream path`、`tool-call capability`、`multimodal capability`、`provider-specific request patch`
|
||||
- [ ] 接入 `inetnum` / `inet6num` whois 作为比 RIR 更细粒度的后备层
|
||||
- [x] 在 activity layer 之后继续补 `route leak` 和 `path instability / flap` detector
|
||||
- [ ] 对 [frontend/public/earth/js/bgp.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp.js) 做按职责拆分的小重构,拆成 data / markers / overlays / animation,降低后续维护复杂度
|
||||
- [ ] 可选优化(非必做):将 BGP incident/collector 标点改为 HTML marker(参考 worldmonitor 的 `htmlElementsData` 思路),实现近乎固定屏幕尺寸与更高密度可点击性
|
||||
- [ ] 保持 Earth 当前这批纯个人偏好设置继续走本地持久化:`旋转模式`、HUD 面板显示/隐藏、`地形透明度` 暂不升级到后端系统设置,避免把设备级偏好过早做成全局配置
|
||||
- [ ] 如果后续明确需要“账号级同步 Earth 偏好”,再单独设计 `Earth user preferences`:优先按用户维度而不是全局系统设置保存,并规划 `localStorage -> backend` 的平滑迁移策略
|
||||
- [ ] 为 Planet / Earth 补一个可用的日志查看系统:先明确前后端/AI Provider/采集任务的日志入口、最近日志聚合、筛选与 tail 能力,再决定是先做脚本级统一入口还是控制台内置日志面板
|
||||
- [ ] 重写控制台 UI,逐步抛弃 Ant Design,建立自有组件体系,并统一采用 `tabler.io` / Tabler Icons 作为控制台主图标库
|
||||
- [ ] 把 Earth 态势新闻源从 [earth_news.py](/home/ray/dev/linkong/planet/backend/app/services/earth_news.py) 的硬编码列表抽成可配置目录,优先保持当前“实时聚合”链路不变,只先解决新闻源不可配置的问题
|
||||
- [ ] 为 Earth 态势新闻设计后续采集器化方案:明确新闻数据模型、去重策略、区域映射、过期清理和 Earth/AI 复用方式,再决定何时把新闻从实时抓取升级成正式 collector
|
||||
- [ ] 为 Earth 地球表面增加一层与基础纹理对齐的材质/纹理 overlay,并在同层叠加国界轮廓参考线;要求国界线与底图稳定对齐,且 hover 到国家轮廓时能高亮当前国家,便于校准地表和增强交互
|
||||
- [ ] 把 Earth 新闻接入通用巡航队列:按新闻发生地和时间排序生成巡航目标,巡航聚焦到新闻事件时显示对应新闻卡片,并保持实现边界为“通用巡航层 + 新闻业务适配层”,不要再把新闻逻辑直接耦合回 `main.js` 状态机
|
||||
- [ ] 为未知位置的算力中心建立分层坐标补全链路:优先 `精确坐标 > 站点/园区命中 > 城市 > 州/省 > 国家内主要算力城市 > 国家质心`,并把每次回退的 `confidence / reason / precision` 明确写进统一 GeoJSON
|
||||
- [ ] 为算力中心补一份可维护的本地位置注册表,例如 `canonical_name / aliases / operator / country / region / city / lat / lon / confidence / source_note`,避免把地点知识长期硬编码在 `visualization.py`
|
||||
- [ ] 增强 `epoch_ai_gpu` 和相关算力采集器的源页面解析:即使公开 API 不给坐标,也继续尝试从详情页、HTML、内嵌 JSON、schema.org、OpenGraph、脚本变量和 PDF/新闻稿链接里抽地点线索
|
||||
- [ ] 为未知位置算力中心增加外部富化策略评估:可选接入公开知识源或搜索兜底,只抓“站点名/园区名/城市名”级别线索,不直接抓经纬度结论,并把结果作为候选证据而不是真值
|
||||
- [ ] 为算力中心建立 `operator / cluster name / facility alias` 归一化层,先解决 `xAI / Colossus / Memphis`、`OpenAI / Stargate`、`CoreWeave`、`Lambda`、`Crusoe` 这类同一对象多种写法导致的地点匹配失败
|
||||
- [ ] 为估算位置增加更细的视觉和产品表达:除了问号角标,还要支持 tooltip/详情中的“估算依据”“精度级别”“最后核验时间”,并允许在设置中单独开关“仅看精确位置”
|
||||
- [ ] 为国家级估算点设计更合理的落点策略:优先落在“该国主要算力/数据中心城市候选集”而不是几何质心,必要时同国多节点做稳定散列分配,避免大量节点堆在荒漠或海上
|
||||
- [ ] 为未知位置算力中心建立人工校验工作流:支持导出待核验清单、记录人工确认结果,并把人工确认反哺到位置注册表,逐步减少问号点比例
|
||||
+35
-13
@@ -6,29 +6,51 @@ AI_TIMEOUT_SECONDS=60
|
||||
AI_HTTP_RETRY_ATTEMPTS=2
|
||||
AI_ANALYSIS_SYSTEM_PROMPT=你是态势感知分析助手。请基于输入的上下文、观测与约束,输出结构化、克制、可执行的分析。
|
||||
|
||||
# Select one provider mode:
|
||||
# - openai_compatible
|
||||
# - claude_compatible
|
||||
# Provider identity. Recommended values:
|
||||
# - minimax
|
||||
# - openai
|
||||
# - ollama
|
||||
AI_PROVIDER=ollama
|
||||
# Compatibility aliases still accepted:
|
||||
# - openai_compatible
|
||||
# - anthropic_compatible
|
||||
# - claude_compatible
|
||||
AI_PROVIDER=minimax
|
||||
|
||||
# Request adapter style, following OpenClaw's API-seam pattern:
|
||||
# - auto
|
||||
# - openai-completions
|
||||
# - anthropic-messages
|
||||
# - ollama-generate
|
||||
AI_PROVIDER_API=anthropic-messages
|
||||
|
||||
# Common model selection
|
||||
AI_MODEL=qwen2.5:7b
|
||||
AI_MODEL=MiniMax-M2.7
|
||||
|
||||
# MiniMax CN Anthropic-compatible example
|
||||
AI_BASE_URL=https://api.minimaxi.com/anthropic
|
||||
AI_API_KEY=sk-cp-change-me
|
||||
AI_MAX_TOKENS=1200
|
||||
AI_ANTHROPIC_VERSION=2023-06-01
|
||||
|
||||
# OpenAI-compatible example (vLLM / LM Studio / One API / local gateway)
|
||||
# AI_PROVIDER=openai_compatible
|
||||
# AI_PROVIDER=openai
|
||||
# AI_PROVIDER_API=openai-completions
|
||||
# AI_BASE_URL=http://127.0.0.1:8001/v1
|
||||
# AI_API_KEY=local-key
|
||||
# AI_MODEL=your-local-model
|
||||
|
||||
# Claude-compatible example (Anthropic / MiniMax / Claude-compatible gateway)
|
||||
# AI_PROVIDER=claude_compatible
|
||||
# AI_BASE_URL=http://127.0.0.1:8002
|
||||
# Anthropic-compatible example (Claude-compatible gateway)
|
||||
# AI_PROVIDER=anthropic
|
||||
# AI_PROVIDER_API=anthropic-messages
|
||||
# AI_BASE_URL=http://127.0.0.1:8002/anthropic
|
||||
# AI_API_KEY=local-key
|
||||
# AI_MODEL=your-model
|
||||
# AI_MAX_TOKENS=1200
|
||||
# AI_ANTHROPIC_VERSION=2023-06-01
|
||||
|
||||
# Ollama native example
|
||||
AI_BASE_URL=http://127.0.0.1:11434
|
||||
AI_API_KEY=
|
||||
AI_MAX_TOKENS=1200
|
||||
AI_ANTHROPIC_VERSION=2023-06-01
|
||||
# AI_PROVIDER=ollama
|
||||
# AI_PROVIDER_API=ollama-generate
|
||||
# AI_BASE_URL=http://127.0.0.1:11434
|
||||
# AI_API_KEY=
|
||||
# AI_MODEL=qwen2.5:7b
|
||||
@@ -1,6 +1,10 @@
|
||||
FROM python:3.14-slim
|
||||
ARG PYTHON_IMAGE=python:3.14-slim
|
||||
ARG UV_IMAGE=ghcr.io/astral-sh/uv:latest
|
||||
|
||||
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
|
||||
FROM ${UV_IMAGE} AS uv
|
||||
FROM ${PYTHON_IMAGE}
|
||||
|
||||
COPY --from=uv /uv /uvx /bin/
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
|
||||
+30
-16
@@ -4,21 +4,31 @@
|
||||
|
||||
完整使用说明见:
|
||||
|
||||
- [docs/aiprovider.md](/home/ray/dev/linkong/planet/docs/aiprovider.md)
|
||||
- [docs/agents/aiprovider.md](/home/ray/dev/linkong/planet/docs/agents/aiprovider.md)
|
||||
|
||||
当前支持:
|
||||
|
||||
- `AI_PROVIDER=openai`
|
||||
- `AI_PROVIDER=openai_compatible`
|
||||
- `AI_PROVIDER=anthropic`
|
||||
- `AI_PROVIDER=anthropic_compatible`
|
||||
- `AI_PROVIDER=claude_compatible`
|
||||
- `AI_PROVIDER=ollama`
|
||||
- provider identity:
|
||||
- `AI_PROVIDER=openai`
|
||||
- `AI_PROVIDER=anthropic`
|
||||
- `AI_PROVIDER=minimax`
|
||||
- `AI_PROVIDER=ollama`
|
||||
- request adapter:
|
||||
- `AI_PROVIDER_API=openai-completions`
|
||||
- `AI_PROVIDER_API=anthropic-messages`
|
||||
- `AI_PROVIDER_API=ollama-generate`
|
||||
|
||||
兼容别名仍然保留:
|
||||
|
||||
- `openai_compatible`
|
||||
- `anthropic_compatible`
|
||||
- `claude_compatible`
|
||||
|
||||
典型配置:
|
||||
|
||||
```env
|
||||
AI_PROVIDER=openai_compatible
|
||||
AI_PROVIDER=openai
|
||||
AI_PROVIDER_API=openai-completions
|
||||
AI_BASE_URL=https://api.openai.com/v1
|
||||
AI_API_KEY=your_api_key
|
||||
AI_MODEL=gpt-4o-mini
|
||||
@@ -26,13 +36,14 @@ AI_TIMEOUT_SECONDS=60
|
||||
AI_PROVIDER_SERVICE_TOKEN=change_me
|
||||
```
|
||||
|
||||
Claude 兼容供应商示例:
|
||||
MiniMax 中国大陆节点示例:
|
||||
|
||||
```env
|
||||
AI_PROVIDER=claude_compatible
|
||||
AI_BASE_URL=https://your-claude-compatible-endpoint.example.com
|
||||
AI_API_KEY=your_api_key
|
||||
AI_MODEL=your-claude-compatible-model
|
||||
AI_PROVIDER=minimax
|
||||
AI_PROVIDER_API=anthropic-messages
|
||||
AI_BASE_URL=https://api.minimaxi.com/anthropic
|
||||
AI_API_KEY=sk-cp-xxxxx
|
||||
AI_MODEL=MiniMax-M2.7
|
||||
AI_TIMEOUT_SECONDS=60
|
||||
AI_MAX_TOKENS=1200
|
||||
AI_ANTHROPIC_VERSION=2023-06-01
|
||||
@@ -43,12 +54,15 @@ AI_PROVIDER_SERVICE_TOKEN=change_me
|
||||
|
||||
- Anthropic 官方 Claude API
|
||||
- Claude 兼容网关
|
||||
- MiniMax 等提供 Claude/Anthropic 风格消息接口的服务
|
||||
- MiniMax 等提供 Anthropic Messages 风格接口的服务
|
||||
|
||||
这套命名方式参考了 OpenClaw 的接入模式: provider 负责标识供应商, `AI_PROVIDER_API` 负责标识协议适配层, 避免把“供应商”和“协议”绑死在一起。
|
||||
|
||||
Ollama 原生示例:
|
||||
|
||||
```env
|
||||
AI_PROVIDER=ollama
|
||||
AI_PROVIDER_API=ollama-generate
|
||||
AI_BASE_URL=http://127.0.0.1:11434
|
||||
AI_API_KEY=
|
||||
AI_MODEL=qwen2.5:7b
|
||||
@@ -58,8 +72,8 @@ AI_PROVIDER_SERVICE_TOKEN=change_me
|
||||
|
||||
本地模型接入建议:
|
||||
|
||||
- `vLLM`、`LM Studio`、`One API`:优先使用 `openai_compatible`
|
||||
- `MiniMax`、Claude 兼容网关:使用 `claude_compatible`
|
||||
- `vLLM`、`LM Studio`、`One API`:`AI_PROVIDER=openai` + `AI_PROVIDER_API=openai-completions`
|
||||
- `MiniMax`、Claude 兼容网关:`AI_PROVIDER=minimax|anthropic` + `AI_PROVIDER_API=anthropic-messages`
|
||||
- `Ollama`:可直接使用 `ollama`
|
||||
|
||||
启动模板:
|
||||
|
||||
@@ -9,6 +9,7 @@ class Settings(BaseSettings):
|
||||
SERVICE_VERSION: str = "0.1.0"
|
||||
|
||||
AI_PROVIDER: str = "disabled"
|
||||
AI_PROVIDER_API: str = "auto"
|
||||
AI_BASE_URL: str = "https://api.openai.com/v1"
|
||||
AI_API_KEY: str = ""
|
||||
AI_MODEL: str = ""
|
||||
|
||||
+20
-2
@@ -37,8 +37,26 @@ def verify_service_token(x_provider_token: str | None = Header(default=None)) ->
|
||||
)
|
||||
|
||||
|
||||
def get_provider_service() -> ProviderService:
|
||||
return ProviderService()
|
||||
def get_provider_service(
|
||||
x_ai_provider: str | None = Header(default=None),
|
||||
x_ai_provider_api: str | None = Header(default=None),
|
||||
x_ai_base_url: str | None = Header(default=None),
|
||||
x_ai_api_key: str | None = Header(default=None),
|
||||
x_ai_model: str | None = Header(default=None),
|
||||
x_ai_max_tokens: str | None = Header(default=None),
|
||||
x_ai_anthropic_version: str | None = Header(default=None),
|
||||
) -> ProviderService:
|
||||
overrides = {
|
||||
"provider": x_ai_provider,
|
||||
"provider_api": x_ai_provider_api,
|
||||
"base_url": x_ai_base_url,
|
||||
"api_key": x_ai_api_key,
|
||||
"model": x_ai_model,
|
||||
"anthropic_version": x_ai_anthropic_version,
|
||||
}
|
||||
if x_ai_max_tokens:
|
||||
overrides["max_tokens"] = x_ai_max_tokens
|
||||
return ProviderService({key: value for key, value in overrides.items() if value not in (None, "")})
|
||||
|
||||
|
||||
@app.get("/health")
|
||||
|
||||
+151
-16
@@ -8,6 +8,7 @@ from fastapi import HTTPException, status
|
||||
|
||||
from aiprovider.config import settings
|
||||
from aiprovider.schemas import (
|
||||
AIContentBlock,
|
||||
AIProviderStatusResponse,
|
||||
SituationalAnalysisRequest,
|
||||
SituationalAnalysisResponse,
|
||||
@@ -18,23 +19,58 @@ def _normalize_provider(value: str) -> str:
|
||||
return (value or "disabled").strip().lower()
|
||||
|
||||
|
||||
def _normalize_provider_api(value: str) -> str:
|
||||
return (value or "auto").strip().lower().replace("_", "-")
|
||||
|
||||
|
||||
def _resolve_provider_api(provider: str, configured_api: str) -> str:
|
||||
if configured_api and configured_api != "auto":
|
||||
return configured_api
|
||||
|
||||
if provider in {"openai", "openai-compatible", "openai_compatible"}:
|
||||
return "openai-completions"
|
||||
if provider in {
|
||||
"anthropic",
|
||||
"anthropic-compatible",
|
||||
"anthropic_compatible",
|
||||
"claude-compatible",
|
||||
"claude_compatible",
|
||||
"minimax",
|
||||
"kimi-coding",
|
||||
"moonshot-anthropic",
|
||||
}:
|
||||
return "anthropic-messages"
|
||||
if provider == "ollama":
|
||||
return "ollama-generate"
|
||||
return "disabled"
|
||||
|
||||
|
||||
class ProviderService:
|
||||
def __init__(self) -> None:
|
||||
self.provider = _normalize_provider(settings.AI_PROVIDER)
|
||||
self.base_url = settings.AI_BASE_URL.rstrip("/")
|
||||
self.api_key = settings.AI_API_KEY
|
||||
self.default_model = settings.AI_MODEL
|
||||
def __init__(self, overrides: dict[str, Any] | None = None) -> None:
|
||||
overrides = overrides or {}
|
||||
self.provider = _normalize_provider(overrides.get("provider") or settings.AI_PROVIDER)
|
||||
self.provider_api = _resolve_provider_api(
|
||||
self.provider,
|
||||
_normalize_provider_api(overrides.get("provider_api") or settings.AI_PROVIDER_API),
|
||||
)
|
||||
self.base_url = str(overrides.get("base_url") or settings.AI_BASE_URL).rstrip("/")
|
||||
self.api_key = str(overrides.get("api_key") or settings.AI_API_KEY)
|
||||
self.default_model = str(overrides.get("model") or settings.AI_MODEL)
|
||||
self.timeout = settings.AI_TIMEOUT_SECONDS
|
||||
self.http_retry_attempts = max(settings.AI_HTTP_RETRY_ATTEMPTS, 1)
|
||||
self.max_tokens = settings.AI_MAX_TOKENS
|
||||
self.anthropic_version = settings.AI_ANTHROPIC_VERSION
|
||||
self.max_tokens = int(overrides.get("max_tokens") or settings.AI_MAX_TOKENS)
|
||||
self.anthropic_version = str(
|
||||
overrides.get("anthropic_version") or settings.AI_ANTHROPIC_VERSION
|
||||
)
|
||||
self.system_prompt = settings.AI_ANALYSIS_SYSTEM_PROMPT
|
||||
|
||||
def get_status(self) -> AIProviderStatusResponse:
|
||||
enabled = self.provider != "disabled"
|
||||
configured = enabled and bool(self.base_url and self.api_key and self.default_model)
|
||||
has_credentials = bool(self.api_key) if self._requires_api_key() else True
|
||||
configured = enabled and bool(self.base_url and has_credentials and self.default_model)
|
||||
return AIProviderStatusResponse(
|
||||
provider=self.provider,
|
||||
api=self.provider_api if enabled else None,
|
||||
enabled=enabled,
|
||||
configured=configured,
|
||||
model=self.default_model or None,
|
||||
@@ -49,7 +85,8 @@ class ProviderService:
|
||||
)
|
||||
|
||||
model = payload.preferred_model or self.default_model
|
||||
if not self.base_url or not self.api_key or not model:
|
||||
has_credentials = bool(self.api_key) if self._requires_api_key() else True
|
||||
if not self.base_url or not has_credentials or not model:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
|
||||
detail="AI provider is not fully configured. Check AI_BASE_URL, AI_API_KEY, and AI_MODEL.",
|
||||
@@ -57,28 +94,40 @@ class ProviderService:
|
||||
|
||||
prompt = self._build_prompt(payload)
|
||||
|
||||
if self.provider in {"openai", "openai_compatible"}:
|
||||
if self.provider_api == "openai-completions":
|
||||
data = await self._request_openai_compatible(model, prompt)
|
||||
content = self._extract_openai_content(data)
|
||||
elif self.provider in {"anthropic", "anthropic_compatible", "claude_compatible"}:
|
||||
data = await self._request_anthropic_compatible(model, prompt)
|
||||
content_blocks = self._extract_openai_blocks(data)
|
||||
elif self.provider_api == "anthropic-messages":
|
||||
data = await self._request_anthropic_messages(model, prompt, payload.thinking)
|
||||
content = self._extract_anthropic_content(data)
|
||||
elif self.provider == "ollama":
|
||||
content_blocks = self._extract_anthropic_blocks(data)
|
||||
elif self.provider_api == "ollama-generate":
|
||||
data = await self._request_ollama(model, prompt)
|
||||
content = self._extract_ollama_content(data)
|
||||
content_blocks = self._extract_ollama_blocks(data)
|
||||
else:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
detail=f"Unsupported AI provider: {self.provider}",
|
||||
detail=f"Unsupported AI provider API: {self.provider_api}",
|
||||
)
|
||||
|
||||
text_blocks = [block.text for block in content_blocks if block.text]
|
||||
thinking_blocks = [block.thinking for block in content_blocks if block.thinking]
|
||||
|
||||
return SituationalAnalysisResponse(
|
||||
provider=self.provider,
|
||||
model=model,
|
||||
content=content,
|
||||
content_blocks=content_blocks,
|
||||
text_blocks=text_blocks,
|
||||
thinking_blocks=thinking_blocks,
|
||||
raw_response=data,
|
||||
)
|
||||
|
||||
def _requires_api_key(self) -> bool:
|
||||
return self.provider_api != "ollama-generate"
|
||||
|
||||
def _build_prompt(self, payload: SituationalAnalysisRequest) -> str:
|
||||
sections = [
|
||||
f"任务标题:\n{payload.title}",
|
||||
@@ -113,7 +162,12 @@ class ProviderService:
|
||||
request_body=request_body,
|
||||
)
|
||||
|
||||
async def _request_anthropic_compatible(self, model: str, prompt: str) -> dict[str, Any]:
|
||||
async def _request_anthropic_messages(
|
||||
self,
|
||||
model: str,
|
||||
prompt: str,
|
||||
thinking: dict[str, Any] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
request_body = {
|
||||
"model": model,
|
||||
"system": self.system_prompt,
|
||||
@@ -131,8 +185,15 @@ class ProviderService:
|
||||
"max_tokens": self.max_tokens,
|
||||
"temperature": 0.2,
|
||||
}
|
||||
resolved_thinking = self._resolve_anthropic_thinking(thinking)
|
||||
if resolved_thinking:
|
||||
request_body["thinking"] = resolved_thinking
|
||||
if self.provider == "minimax" and self.base_url.endswith("/anthropic"):
|
||||
path = "/v1/messages"
|
||||
else:
|
||||
path = "/messages"
|
||||
return await self._post(
|
||||
path="/messages",
|
||||
path=path,
|
||||
headers={
|
||||
"x-api-key": self.api_key,
|
||||
"anthropic-version": self.anthropic_version,
|
||||
@@ -141,6 +202,25 @@ class ProviderService:
|
||||
request_body=request_body,
|
||||
)
|
||||
|
||||
def _resolve_anthropic_thinking(self, thinking: dict[str, Any] | None) -> dict[str, Any] | None:
|
||||
if thinking:
|
||||
return thinking
|
||||
|
||||
# OpenClaw treats MiniMax's Anthropic-compatible path specially:
|
||||
# disable thinking by default unless the caller explicitly opts in.
|
||||
if self.provider == "minimax":
|
||||
return {"type": "disabled"}
|
||||
|
||||
return None
|
||||
|
||||
async def _request_anthropic_compatible(
|
||||
self,
|
||||
model: str,
|
||||
prompt: str,
|
||||
thinking: dict[str, Any] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
return await self._request_anthropic_messages(model, prompt, thinking)
|
||||
|
||||
async def _request_ollama(self, model: str, prompt: str) -> dict[str, Any]:
|
||||
request_body = {
|
||||
"model": model,
|
||||
@@ -218,6 +298,30 @@ class ProviderService:
|
||||
)
|
||||
return ""
|
||||
|
||||
def _extract_openai_blocks(self, payload: dict[str, Any]) -> list[AIContentBlock]:
|
||||
choices = payload.get("choices") or []
|
||||
if not choices:
|
||||
return []
|
||||
|
||||
message = choices[0].get("message") or {}
|
||||
content = message.get("content")
|
||||
if isinstance(content, str):
|
||||
return [AIContentBlock(type="text", text=content)]
|
||||
if not isinstance(content, list):
|
||||
return []
|
||||
|
||||
blocks: list[AIContentBlock] = []
|
||||
for item in content:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
blocks.append(
|
||||
AIContentBlock(
|
||||
type=str(item.get("type", "text")),
|
||||
text=item.get("text") if isinstance(item.get("text"), str) else None,
|
||||
metadata={k: v for k, v in item.items() if k not in {"type", "text"}},
|
||||
)
|
||||
)
|
||||
return blocks
|
||||
def _extract_anthropic_content(self, payload: dict[str, Any]) -> str:
|
||||
content = payload.get("content")
|
||||
if isinstance(content, str):
|
||||
@@ -233,8 +337,39 @@ class ProviderService:
|
||||
fragments.append(item["text"])
|
||||
return "".join(fragments)
|
||||
|
||||
def _extract_anthropic_blocks(self, payload: dict[str, Any]) -> list[AIContentBlock]:
|
||||
content = payload.get("content")
|
||||
if not isinstance(content, list):
|
||||
return []
|
||||
|
||||
blocks: list[AIContentBlock] = []
|
||||
for item in content:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
blocks.append(
|
||||
AIContentBlock(
|
||||
type=str(item.get("type", "unknown")),
|
||||
text=item.get("text") if isinstance(item.get("text"), str) else None,
|
||||
thinking=item.get("thinking") if isinstance(item.get("thinking"), str) else None,
|
||||
signature=item.get("signature") if isinstance(item.get("signature"), str) else None,
|
||||
metadata={
|
||||
k: v
|
||||
for k, v in item.items()
|
||||
if k not in {"type", "text", "thinking", "signature"}
|
||||
},
|
||||
)
|
||||
)
|
||||
return blocks
|
||||
|
||||
|
||||
def _extract_ollama_content(self, payload: dict[str, Any]) -> str:
|
||||
response = payload.get("response")
|
||||
if isinstance(response, str):
|
||||
return response
|
||||
return ""
|
||||
|
||||
def _extract_ollama_blocks(self, payload: dict[str, Any]) -> list[AIContentBlock]:
|
||||
response = payload.get("response")
|
||||
if isinstance(response, str) and response:
|
||||
return [AIContentBlock(type="text", text=response)]
|
||||
return []
|
||||
@@ -3,6 +3,14 @@ from typing import Any
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class AIContentBlock(BaseModel):
|
||||
type: str
|
||||
text: str | None = None
|
||||
thinking: str | None = None
|
||||
signature: str | None = None
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class SituationalAnalysisRequest(BaseModel):
|
||||
title: str = Field(..., min_length=1, max_length=200)
|
||||
objective: str = Field(..., min_length=1, max_length=1000)
|
||||
@@ -10,17 +18,22 @@ class SituationalAnalysisRequest(BaseModel):
|
||||
observations: list[str] = Field(default_factory=list)
|
||||
constraints: list[str] = Field(default_factory=list)
|
||||
preferred_model: str | None = Field(default=None, max_length=200)
|
||||
thinking: dict[str, Any] | None = None
|
||||
|
||||
|
||||
class SituationalAnalysisResponse(BaseModel):
|
||||
provider: str
|
||||
model: str
|
||||
content: str
|
||||
content_blocks: list[AIContentBlock] = Field(default_factory=list)
|
||||
text_blocks: list[str] = Field(default_factory=list)
|
||||
thinking_blocks: list[str] = Field(default_factory=list)
|
||||
raw_response: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class AIProviderStatusResponse(BaseModel):
|
||||
provider: str
|
||||
api: str | None = None
|
||||
enabled: bool
|
||||
configured: bool
|
||||
model: str | None = None
|
||||
|
||||
+6
-2
@@ -1,6 +1,10 @@
|
||||
FROM python:3.14-slim
|
||||
ARG PYTHON_IMAGE=python:3.14-slim
|
||||
ARG UV_IMAGE=ghcr.io/astral-sh/uv:latest
|
||||
|
||||
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
|
||||
FROM ${UV_IMAGE} AS uv
|
||||
FROM ${PYTHON_IMAGE}
|
||||
|
||||
COPY --from=uv /uv /uvx /bin/
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
|
||||
@@ -13,7 +13,9 @@ from app.api.v1 import (
|
||||
collected_data,
|
||||
visualization,
|
||||
bgp,
|
||||
news,
|
||||
system_control,
|
||||
tv,
|
||||
)
|
||||
|
||||
api_router = APIRouter()
|
||||
@@ -33,3 +35,5 @@ api_router.include_router(settings.router, prefix="/settings", tags=["settings"]
|
||||
api_router.include_router(system_control.router, prefix="/system", tags=["system"])
|
||||
api_router.include_router(visualization.router, prefix="/visualization", tags=["visualization"])
|
||||
api_router.include_router(bgp.router, prefix="/bgp", tags=["bgp"])
|
||||
api_router.include_router(tv.router, prefix="/tv", tags=["tv"])
|
||||
api_router.include_router(news.router, prefix="/news", tags=["news"])
|
||||
+243
-1
@@ -1,15 +1,52 @@
|
||||
from uuid import uuid4
|
||||
|
||||
from fastapi import APIRouter, Depends, Request, Response
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request, Response
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.security import get_current_user
|
||||
from app.db.session import get_db
|
||||
from app.models.user import User
|
||||
from app.schemas.ai import (
|
||||
AIProviderStatusResponse,
|
||||
AlertBriefRequest,
|
||||
AlertBriefResponse,
|
||||
BGPBriefRequest,
|
||||
BGPBriefRecordResponse,
|
||||
BGPBriefRecordSummary,
|
||||
PlaygroundMessageActionResponse,
|
||||
PlaygroundMessageCreateRequest,
|
||||
PlaygroundMessageEditRequest,
|
||||
PlaygroundMessageResendRequest,
|
||||
PlaygroundMessageStopRequest,
|
||||
PlaygroundSessionResponse,
|
||||
PlaygroundSessionUpsertRequest,
|
||||
PlaygroundThreadResponse,
|
||||
SituationalAlertBriefRequest,
|
||||
SituationalAlertBriefResponse,
|
||||
SituationalAnalysisRequest,
|
||||
SituationalAnalysisResponse,
|
||||
)
|
||||
from app.services.alert_ai_brief import build_alert_brief_request
|
||||
from app.services.ai_client import AIProviderClient, get_ai_provider_client
|
||||
from app.services.bgp_ai_brief import build_bgp_brief_request
|
||||
from app.services.bgp_ai_brief_store import (
|
||||
get_bgp_brief_record,
|
||||
get_latest_bgp_brief_record,
|
||||
list_bgp_brief_records,
|
||||
save_bgp_brief_record,
|
||||
)
|
||||
from app.services.playground_session_store import (
|
||||
get_playground_session,
|
||||
upsert_playground_session,
|
||||
)
|
||||
from app.services.playground_chat_service import (
|
||||
create_turn,
|
||||
edit_user_message,
|
||||
get_thread,
|
||||
resend_turn,
|
||||
stop_message,
|
||||
)
|
||||
from app.services.situational_alert_ai_brief import build_situational_alert_brief_request
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
@@ -37,3 +74,208 @@ async def analyze_situational_awareness(
|
||||
request_id = request.headers.get("X-Request-ID") or str(uuid4())
|
||||
response.headers["X-Request-ID"] = request_id
|
||||
return await provider_client.analyze(payload, request_id=request_id)
|
||||
|
||||
|
||||
@router.get("/playground/thread", response_model=PlaygroundThreadResponse | None)
|
||||
async def get_playground_thread(
|
||||
session_key: str = "default",
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await get_thread(
|
||||
db,
|
||||
user_id=current_user.id,
|
||||
session_key=session_key,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/playground/session", response_model=PlaygroundSessionResponse | None)
|
||||
async def get_saved_playground_session(
|
||||
session_key: str = "default",
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await get_playground_session(
|
||||
db,
|
||||
user_id=current_user.id,
|
||||
session_key=session_key,
|
||||
)
|
||||
|
||||
|
||||
@router.put("/playground/session", response_model=PlaygroundSessionResponse)
|
||||
async def save_playground_session(
|
||||
payload: PlaygroundSessionUpsertRequest,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await upsert_playground_session(
|
||||
db,
|
||||
user_id=current_user.id,
|
||||
payload=payload,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/playground/messages", response_model=PlaygroundMessageActionResponse)
|
||||
async def create_playground_message(
|
||||
payload: PlaygroundMessageCreateRequest,
|
||||
current_user: User = Depends(get_current_user),
|
||||
provider_client: AIProviderClient = Depends(get_ai_provider_client),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await create_turn(
|
||||
db,
|
||||
user_id=current_user.id,
|
||||
payload=payload,
|
||||
provider_client=provider_client,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/playground/messages/stop", response_model=PlaygroundMessageActionResponse)
|
||||
async def stop_playground_message(
|
||||
payload: PlaygroundMessageStopRequest,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await stop_message(
|
||||
db,
|
||||
user_id=current_user.id,
|
||||
payload=payload,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/playground/messages/resend", response_model=PlaygroundMessageActionResponse)
|
||||
async def resend_playground_message(
|
||||
payload: PlaygroundMessageResendRequest,
|
||||
current_user: User = Depends(get_current_user),
|
||||
provider_client: AIProviderClient = Depends(get_ai_provider_client),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await resend_turn(
|
||||
db,
|
||||
user_id=current_user.id,
|
||||
payload=payload,
|
||||
provider_client=provider_client,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/playground/messages/edit", response_model=PlaygroundMessageActionResponse)
|
||||
async def edit_playground_message(
|
||||
payload: PlaygroundMessageEditRequest,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await edit_user_message(
|
||||
db,
|
||||
user_id=current_user.id,
|
||||
payload=payload,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/bgp/briefs", response_model=list[BGPBriefRecordSummary])
|
||||
async def list_saved_bgp_briefs(
|
||||
current_user: User = Depends(get_current_user),
|
||||
):
|
||||
return list_bgp_brief_records()
|
||||
|
||||
|
||||
@router.get("/bgp/briefs/latest", response_model=BGPBriefRecordResponse | None)
|
||||
async def get_latest_saved_bgp_brief(
|
||||
current_user: User = Depends(get_current_user),
|
||||
):
|
||||
return get_latest_bgp_brief_record()
|
||||
|
||||
|
||||
@router.get("/bgp/briefs/{brief_id}", response_model=BGPBriefRecordResponse)
|
||||
async def get_saved_bgp_brief(
|
||||
brief_id: str,
|
||||
current_user: User = Depends(get_current_user),
|
||||
):
|
||||
record = get_bgp_brief_record(brief_id)
|
||||
if record is None:
|
||||
raise HTTPException(status_code=404, detail="BGP brief not found")
|
||||
return record
|
||||
|
||||
|
||||
@router.post("/bgp/brief", response_model=BGPBriefRecordResponse)
|
||||
async def analyze_bgp_brief(
|
||||
payload: BGPBriefRequest,
|
||||
request: Request,
|
||||
response: Response,
|
||||
current_user: User = Depends(get_current_user),
|
||||
provider_client: AIProviderClient = Depends(get_ai_provider_client),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
request_id = request.headers.get("X-Request-ID") or str(uuid4())
|
||||
response.headers["X-Request-ID"] = request_id
|
||||
|
||||
brief_request, facts, context = await build_bgp_brief_request(
|
||||
db,
|
||||
incident_limit=payload.incident_limit,
|
||||
anomaly_limit=payload.anomaly_limit,
|
||||
collector_limit=payload.collector_limit,
|
||||
)
|
||||
brief_request.preferred_model = payload.preferred_model
|
||||
brief_request.thinking = payload.thinking
|
||||
|
||||
analysis = await provider_client.analyze(brief_request, request_id=request_id)
|
||||
return save_bgp_brief_record(
|
||||
analysis,
|
||||
request_id=request_id,
|
||||
facts=facts,
|
||||
context=context,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/alerts/brief", response_model=AlertBriefResponse)
|
||||
async def analyze_alert_brief(
|
||||
payload: AlertBriefRequest,
|
||||
request: Request,
|
||||
response: Response,
|
||||
current_user: User = Depends(get_current_user),
|
||||
provider_client: AIProviderClient = Depends(get_ai_provider_client),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
request_id = request.headers.get("X-Request-ID") or str(uuid4())
|
||||
response.headers["X-Request-ID"] = request_id
|
||||
|
||||
brief_request, facts, context = await build_alert_brief_request(
|
||||
db,
|
||||
alert_limit=payload.alert_limit,
|
||||
)
|
||||
brief_request.preferred_model = payload.preferred_model
|
||||
brief_request.thinking = payload.thinking
|
||||
|
||||
analysis = await provider_client.analyze(brief_request, request_id=request_id)
|
||||
return AlertBriefResponse(
|
||||
**analysis.model_dump(),
|
||||
title=brief_request.title,
|
||||
objective=brief_request.objective,
|
||||
facts=facts,
|
||||
context=context,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/situational-alerts/brief", response_model=SituationalAlertBriefResponse)
|
||||
async def analyze_situational_alert_brief(
|
||||
payload: SituationalAlertBriefRequest,
|
||||
request: Request,
|
||||
response: Response,
|
||||
current_user: User = Depends(get_current_user),
|
||||
provider_client: AIProviderClient = Depends(get_ai_provider_client),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
request_id = request.headers.get("X-Request-ID") or str(uuid4())
|
||||
response.headers["X-Request-ID"] = request_id
|
||||
|
||||
brief_request, facts, context = await build_situational_alert_brief_request(db)
|
||||
brief_request.preferred_model = payload.preferred_model
|
||||
brief_request.thinking = payload.thinking
|
||||
|
||||
analysis = await provider_client.analyze(brief_request, request_id=request_id)
|
||||
return SituationalAlertBriefResponse(
|
||||
**analysis.model_dump(),
|
||||
title=brief_request.title,
|
||||
objective=brief_request.objective,
|
||||
facts=facts,
|
||||
context=context,
|
||||
)
|
||||
@@ -1,7 +1,7 @@
|
||||
from datetime import UTC, datetime
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, Depends
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from sqlalchemy import select, func, case
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
@@ -9,6 +9,7 @@ from app.db.session import get_db
|
||||
from app.models.user import User
|
||||
from app.core.security import get_current_user
|
||||
from app.models.alert import Alert, AlertSeverity, AlertStatus
|
||||
from app.schemas.alert import AlertResolutionRequest
|
||||
|
||||
|
||||
router = APIRouter()
|
||||
@@ -77,7 +78,7 @@ async def acknowledge_alert(
|
||||
@router.post("/{alert_id}/resolve")
|
||||
async def resolve_alert(
|
||||
alert_id: int,
|
||||
resolution: str,
|
||||
payload: AlertResolutionRequest,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
@@ -85,12 +86,12 @@ async def resolve_alert(
|
||||
alert = result.scalar_one_or_none()
|
||||
|
||||
if not alert:
|
||||
return {"error": "Alert not found"}
|
||||
raise HTTPException(status_code=404, detail="Alert not found")
|
||||
|
||||
alert.status = AlertStatus.RESOLVED
|
||||
alert.resolved_by = current_user.id
|
||||
alert.resolved_at = datetime.now(UTC)
|
||||
alert.resolution_notes = resolution
|
||||
alert.resolution_notes = payload.resolution
|
||||
await db.commit()
|
||||
|
||||
return {"message": "Alert resolved", "alert": alert.to_dict()}
|
||||
@@ -101,25 +102,44 @@ async def get_alert_stats(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
critical_query = select(func.count(Alert.id)).where(
|
||||
Alert.severity == AlertSeverity.CRITICAL,
|
||||
Alert.status == AlertStatus.ACTIVE,
|
||||
result = await db.execute(
|
||||
select(
|
||||
func.sum(
|
||||
case(
|
||||
(
|
||||
(Alert.severity == AlertSeverity.CRITICAL)
|
||||
& (Alert.status == AlertStatus.ACTIVE),
|
||||
1,
|
||||
),
|
||||
else_=0,
|
||||
)
|
||||
).label("critical"),
|
||||
func.sum(
|
||||
case(
|
||||
(
|
||||
(Alert.severity == AlertSeverity.WARNING)
|
||||
& (Alert.status == AlertStatus.ACTIVE),
|
||||
1,
|
||||
),
|
||||
else_=0,
|
||||
)
|
||||
).label("warning"),
|
||||
func.sum(
|
||||
case(
|
||||
(
|
||||
(Alert.severity == AlertSeverity.INFO)
|
||||
& (Alert.status == AlertStatus.ACTIVE),
|
||||
1,
|
||||
),
|
||||
else_=0,
|
||||
)
|
||||
).label("info"),
|
||||
)
|
||||
)
|
||||
warning_query = select(func.count(Alert.id)).where(
|
||||
Alert.severity == AlertSeverity.WARNING,
|
||||
Alert.status == AlertStatus.ACTIVE,
|
||||
)
|
||||
info_query = select(func.count(Alert.id)).where(
|
||||
Alert.severity == AlertSeverity.INFO,
|
||||
Alert.status == AlertStatus.ACTIVE,
|
||||
)
|
||||
|
||||
critical_result = await db.execute(critical_query)
|
||||
warning_result = await db.execute(warning_query)
|
||||
info_result = await db.execute(info_query)
|
||||
row = result.one()
|
||||
|
||||
return {
|
||||
"critical": critical_result.scalar() or 0,
|
||||
"warning": warning_result.scalar() or 0,
|
||||
"info": info_result.scalar() or 0,
|
||||
"critical": row.critical or 0,
|
||||
"warning": row.warning or 0,
|
||||
"info": row.info or 0,
|
||||
}
|
||||
+248
-129
@@ -22,16 +22,161 @@ def _parse_dt(value: Optional[str]) -> Optional[datetime]:
|
||||
if not value:
|
||||
return None
|
||||
return datetime.fromisoformat(value.replace("Z", "+00:00"))
|
||||
def _event_filters(
|
||||
*,
|
||||
prefix: Optional[str],
|
||||
origin_asn: Optional[int],
|
||||
peer_asn: Optional[int],
|
||||
collector: Optional[str],
|
||||
event_type: Optional[str],
|
||||
source: Optional[str],
|
||||
time_from: Optional[datetime],
|
||||
time_to: Optional[datetime],
|
||||
):
|
||||
filters = [BGPObservation.source.in_(BGP_SOURCES)]
|
||||
if source:
|
||||
filters.append(BGPObservation.source == source)
|
||||
if prefix:
|
||||
filters.append(BGPObservation.prefix == prefix)
|
||||
if origin_asn is not None:
|
||||
filters.append(BGPObservation.origin_asn == origin_asn)
|
||||
if peer_asn is not None:
|
||||
filters.append(BGPObservation.peer_asn == peer_asn)
|
||||
if collector:
|
||||
filters.append(BGPObservation.collector == collector)
|
||||
if event_type:
|
||||
filters.append(BGPObservation.event_type == event_type)
|
||||
if time_from:
|
||||
filters.append(BGPObservation.observed_at >= time_from)
|
||||
if time_to:
|
||||
filters.append(BGPObservation.observed_at <= time_to)
|
||||
return filters
|
||||
|
||||
|
||||
def _matches_time(value: Optional[datetime], time_from: Optional[datetime], time_to: Optional[datetime]) -> bool:
|
||||
if value is None:
|
||||
return False
|
||||
if time_from and value < time_from:
|
||||
return False
|
||||
if time_to and value > time_to:
|
||||
return False
|
||||
return True
|
||||
def _anomaly_filters(
|
||||
*,
|
||||
severity: Optional[str],
|
||||
anomaly_type: Optional[str],
|
||||
status: Optional[str],
|
||||
prefix: Optional[str],
|
||||
origin_asn: Optional[int],
|
||||
time_from: Optional[datetime],
|
||||
time_to: Optional[datetime],
|
||||
):
|
||||
filters = []
|
||||
if severity:
|
||||
filters.append(BGPAnomaly.severity == severity)
|
||||
if anomaly_type:
|
||||
filters.append(BGPAnomaly.anomaly_type == anomaly_type)
|
||||
if status:
|
||||
filters.append(BGPAnomaly.status == status)
|
||||
if prefix:
|
||||
filters.append(BGPAnomaly.prefix == prefix)
|
||||
if origin_asn is not None:
|
||||
filters.append(BGPAnomaly.origin_asn == origin_asn)
|
||||
if time_from:
|
||||
filters.append(BGPAnomaly.created_at >= time_from)
|
||||
if time_to:
|
||||
filters.append(BGPAnomaly.created_at <= time_to)
|
||||
return filters
|
||||
|
||||
|
||||
def _incident_filters(
|
||||
*,
|
||||
severity: Optional[str],
|
||||
incident_type: Optional[str],
|
||||
status: Optional[str],
|
||||
):
|
||||
filters = []
|
||||
if severity:
|
||||
filters.append(BGPIncident.severity == severity)
|
||||
if incident_type:
|
||||
filters.append(BGPIncident.incident_type == incident_type)
|
||||
if status:
|
||||
filters.append(BGPIncident.status == status)
|
||||
return filters
|
||||
|
||||
|
||||
async def _build_event_summary_payload(db: AsyncSession) -> dict:
|
||||
base_filters = [BGPObservation.source.in_(BGP_SOURCES)]
|
||||
|
||||
total_result = await db.execute(
|
||||
select(func.count(BGPObservation.id)).where(*base_filters)
|
||||
)
|
||||
collectors_result = await db.execute(
|
||||
select(func.count(func.distinct(BGPObservation.collector))).where(
|
||||
*base_filters, BGPObservation.collector.isnot(None)
|
||||
)
|
||||
)
|
||||
prefixes_result = await db.execute(
|
||||
select(func.count(func.distinct(BGPObservation.prefix))).where(
|
||||
*base_filters, BGPObservation.prefix.isnot(None)
|
||||
)
|
||||
)
|
||||
type_result = await db.execute(
|
||||
select(BGPObservation.event_type, func.count(BGPObservation.id))
|
||||
.where(*base_filters)
|
||||
.group_by(BGPObservation.event_type)
|
||||
)
|
||||
|
||||
return {
|
||||
"total": total_result.scalar() or 0,
|
||||
"collector_count": collectors_result.scalar() or 0,
|
||||
"prefix_count": prefixes_result.scalar() or 0,
|
||||
"by_type": {row[0]: row[1] for row in type_result.fetchall()},
|
||||
}
|
||||
|
||||
|
||||
async def _build_anomaly_summary_payload(db: AsyncSession) -> dict:
|
||||
total_result = await db.execute(select(func.count(BGPAnomaly.id)))
|
||||
type_result = await db.execute(
|
||||
select(BGPAnomaly.anomaly_type, func.count(BGPAnomaly.id))
|
||||
.group_by(BGPAnomaly.anomaly_type)
|
||||
.order_by(func.count(BGPAnomaly.id).desc())
|
||||
)
|
||||
severity_result = await db.execute(
|
||||
select(BGPAnomaly.severity, func.count(BGPAnomaly.id))
|
||||
.group_by(BGPAnomaly.severity)
|
||||
.order_by(func.count(BGPAnomaly.id).desc())
|
||||
)
|
||||
status_result = await db.execute(
|
||||
select(BGPAnomaly.status, func.count(BGPAnomaly.id))
|
||||
.group_by(BGPAnomaly.status)
|
||||
.order_by(func.count(BGPAnomaly.id).desc())
|
||||
)
|
||||
|
||||
return {
|
||||
"total": total_result.scalar() or 0,
|
||||
"by_type": {row[0]: row[1] for row in type_result.fetchall()},
|
||||
"by_severity": {row[0]: row[1] for row in severity_result.fetchall()},
|
||||
"by_status": {row[0]: row[1] for row in status_result.fetchall()},
|
||||
}
|
||||
|
||||
|
||||
async def _build_incident_summary_payload(db: AsyncSession) -> dict:
|
||||
total_result = await db.execute(select(func.count(BGPIncident.id)))
|
||||
type_result = await db.execute(
|
||||
select(BGPIncident.incident_type, func.count(BGPIncident.id))
|
||||
.group_by(BGPIncident.incident_type)
|
||||
.order_by(func.count(BGPIncident.id).desc())
|
||||
)
|
||||
severity_result = await db.execute(
|
||||
select(BGPIncident.severity, func.count(BGPIncident.id))
|
||||
.group_by(BGPIncident.severity)
|
||||
.order_by(func.count(BGPIncident.id).desc())
|
||||
)
|
||||
status_result = await db.execute(
|
||||
select(BGPIncident.status, func.count(BGPIncident.id))
|
||||
.group_by(BGPIncident.status)
|
||||
.order_by(func.count(BGPIncident.id).desc())
|
||||
)
|
||||
|
||||
return {
|
||||
"total": total_result.scalar() or 0,
|
||||
"by_type": {row[0]: row[1] for row in type_result.fetchall()},
|
||||
"by_severity": {row[0]: row[1] for row in severity_result.fetchall()},
|
||||
"by_status": {row[0]: row[1] for row in status_result.fetchall()},
|
||||
}
|
||||
|
||||
|
||||
@router.get("/events")
|
||||
@@ -49,41 +194,36 @@ async def list_bgp_events(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
stmt = (
|
||||
select(BGPObservation)
|
||||
.where(BGPObservation.source.in_(BGP_SOURCES))
|
||||
.order_by(BGPObservation.observed_at.desc(), BGPObservation.id.desc())
|
||||
)
|
||||
if source:
|
||||
stmt = stmt.where(BGPObservation.source == source)
|
||||
|
||||
result = await db.execute(stmt)
|
||||
records = result.scalars().all()
|
||||
dt_from = _parse_dt(time_from)
|
||||
dt_to = _parse_dt(time_to)
|
||||
|
||||
filtered = []
|
||||
for record in records:
|
||||
if prefix and record.prefix != prefix:
|
||||
continue
|
||||
if origin_asn is not None and record.origin_asn != origin_asn:
|
||||
continue
|
||||
if peer_asn is not None and record.peer_asn != peer_asn:
|
||||
continue
|
||||
if collector and record.collector != collector:
|
||||
continue
|
||||
if event_type and record.event_type != event_type:
|
||||
continue
|
||||
if (dt_from or dt_to) and not _matches_time(record.observed_at, dt_from, dt_to):
|
||||
continue
|
||||
filtered.append(record)
|
||||
|
||||
filters = _event_filters(
|
||||
prefix=prefix,
|
||||
origin_asn=origin_asn,
|
||||
peer_asn=peer_asn,
|
||||
collector=collector,
|
||||
event_type=event_type,
|
||||
source=source,
|
||||
time_from=dt_from,
|
||||
time_to=dt_to,
|
||||
)
|
||||
offset = (page - 1) * page_size
|
||||
count_result = await db.execute(
|
||||
select(func.count(BGPObservation.id)).where(*filters)
|
||||
)
|
||||
data_result = await db.execute(
|
||||
select(BGPObservation)
|
||||
.where(*filters)
|
||||
.order_by(BGPObservation.observed_at.desc(), BGPObservation.id.desc())
|
||||
.offset(offset)
|
||||
.limit(page_size)
|
||||
)
|
||||
records = data_result.scalars().all()
|
||||
|
||||
return {
|
||||
"total": len(filtered),
|
||||
"total": count_result.scalar() or 0,
|
||||
"page": page,
|
||||
"page_size": page_size,
|
||||
"data": [record.to_dict() for record in filtered[offset : offset + page_size]],
|
||||
"data": [record.to_dict() for record in records],
|
||||
}
|
||||
|
||||
|
||||
@@ -92,21 +232,7 @@ async def get_bgp_event_summary(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
result = await db.execute(select(BGPObservation).where(BGPObservation.source.in_(BGP_SOURCES)))
|
||||
records = result.scalars().all()
|
||||
|
||||
collectors = sorted({record.collector for record in records if record.collector})
|
||||
prefixes = sorted({record.prefix for record in records if record.prefix})
|
||||
by_type: dict[str, int] = {}
|
||||
for record in records:
|
||||
by_type[record.event_type] = by_type.get(record.event_type, 0) + 1
|
||||
|
||||
return {
|
||||
"total": len(records),
|
||||
"collector_count": len(collectors),
|
||||
"prefix_count": len(prefixes),
|
||||
"by_type": by_type,
|
||||
}
|
||||
return await _build_event_summary_payload(db)
|
||||
|
||||
|
||||
@router.get("/collectors")
|
||||
@@ -138,6 +264,32 @@ async def get_bgp_collector_summary(
|
||||
}
|
||||
|
||||
|
||||
@router.get("/overview/summary")
|
||||
async def get_bgp_overview_summary(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
event_summary = await _build_event_summary_payload(db)
|
||||
anomaly_summary = await _build_anomaly_summary_payload(db)
|
||||
incident_summary = await _build_incident_summary_payload(db)
|
||||
collectors = await build_bgp_collector_coverage(db, source_filter=BGP_SOURCES)
|
||||
active_collectors = [item for item in collectors if item["observation_count"] > 0]
|
||||
|
||||
return {
|
||||
"incidentSummary": incident_summary,
|
||||
"anomalySummary": anomaly_summary,
|
||||
"eventSummary": event_summary,
|
||||
"collectorSummary": {
|
||||
"total": len(collectors),
|
||||
"active_collectors": len(active_collectors),
|
||||
"observed_prefixes": sum(item["prefix_count"] for item in active_collectors),
|
||||
"observed_origins": sum(item["origin_asn_count"] for item in active_collectors),
|
||||
"recent_24h_events": sum(item["recent_24h_observation_count"] for item in active_collectors),
|
||||
"recent_7d_events": sum(item["recent_7d_observation_count"] for item in active_collectors),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@router.get("/events/{event_id}")
|
||||
async def get_bgp_event(
|
||||
event_id: int,
|
||||
@@ -164,31 +316,35 @@ async def list_bgp_anomalies(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
stmt = select(BGPAnomaly).order_by(BGPAnomaly.created_at.desc(), BGPAnomaly.id.desc())
|
||||
if severity:
|
||||
stmt = stmt.where(BGPAnomaly.severity == severity)
|
||||
if anomaly_type:
|
||||
stmt = stmt.where(BGPAnomaly.anomaly_type == anomaly_type)
|
||||
if status:
|
||||
stmt = stmt.where(BGPAnomaly.status == status)
|
||||
if prefix:
|
||||
stmt = stmt.where(BGPAnomaly.prefix == prefix)
|
||||
if origin_asn is not None:
|
||||
stmt = stmt.where(BGPAnomaly.origin_asn == origin_asn)
|
||||
|
||||
result = await db.execute(stmt)
|
||||
records = result.scalars().all()
|
||||
dt_from = _parse_dt(time_from)
|
||||
dt_to = _parse_dt(time_to)
|
||||
if dt_from or dt_to:
|
||||
records = [record for record in records if _matches_time(record.created_at, dt_from, dt_to)]
|
||||
|
||||
filters = _anomaly_filters(
|
||||
severity=severity,
|
||||
anomaly_type=anomaly_type,
|
||||
status=status,
|
||||
prefix=prefix,
|
||||
origin_asn=origin_asn,
|
||||
time_from=dt_from,
|
||||
time_to=dt_to,
|
||||
)
|
||||
offset = (page - 1) * page_size
|
||||
total_result = await db.execute(
|
||||
select(func.count(BGPAnomaly.id)).where(*filters)
|
||||
)
|
||||
data_result = await db.execute(
|
||||
select(BGPAnomaly)
|
||||
.where(*filters)
|
||||
.order_by(BGPAnomaly.created_at.desc(), BGPAnomaly.id.desc())
|
||||
.offset(offset)
|
||||
.limit(page_size)
|
||||
)
|
||||
records = data_result.scalars().all()
|
||||
|
||||
return {
|
||||
"total": len(records),
|
||||
"total": total_result.scalar() or 0,
|
||||
"page": page,
|
||||
"page_size": page_size,
|
||||
"data": [record.to_dict() for record in records[offset : offset + page_size]],
|
||||
"data": [record.to_dict() for record in records],
|
||||
}
|
||||
|
||||
|
||||
@@ -197,29 +353,7 @@ async def get_bgp_anomaly_summary(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
total_result = await db.execute(select(func.count(BGPAnomaly.id)))
|
||||
type_result = await db.execute(
|
||||
select(BGPAnomaly.anomaly_type, func.count(BGPAnomaly.id))
|
||||
.group_by(BGPAnomaly.anomaly_type)
|
||||
.order_by(func.count(BGPAnomaly.id).desc())
|
||||
)
|
||||
severity_result = await db.execute(
|
||||
select(BGPAnomaly.severity, func.count(BGPAnomaly.id))
|
||||
.group_by(BGPAnomaly.severity)
|
||||
.order_by(func.count(BGPAnomaly.id).desc())
|
||||
)
|
||||
status_result = await db.execute(
|
||||
select(BGPAnomaly.status, func.count(BGPAnomaly.id))
|
||||
.group_by(BGPAnomaly.status)
|
||||
.order_by(func.count(BGPAnomaly.id).desc())
|
||||
)
|
||||
|
||||
return {
|
||||
"total": total_result.scalar() or 0,
|
||||
"by_type": {row[0]: row[1] for row in type_result.fetchall()},
|
||||
"by_severity": {row[0]: row[1] for row in severity_result.fetchall()},
|
||||
"by_status": {row[0]: row[1] for row in status_result.fetchall()},
|
||||
}
|
||||
return await _build_anomaly_summary_payload(db)
|
||||
|
||||
|
||||
@router.get("/anomalies/{anomaly_id}")
|
||||
@@ -244,22 +378,29 @@ async def list_bgp_incidents(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
stmt = select(BGPIncident).order_by(BGPIncident.created_at.desc(), BGPIncident.id.desc())
|
||||
if severity:
|
||||
stmt = stmt.where(BGPIncident.severity == severity)
|
||||
if incident_type:
|
||||
stmt = stmt.where(BGPIncident.incident_type == incident_type)
|
||||
if status:
|
||||
stmt = stmt.where(BGPIncident.status == status)
|
||||
|
||||
result = await db.execute(stmt)
|
||||
records = result.scalars().all()
|
||||
filters = _incident_filters(
|
||||
severity=severity,
|
||||
incident_type=incident_type,
|
||||
status=status,
|
||||
)
|
||||
offset = (page - 1) * page_size
|
||||
total_result = await db.execute(
|
||||
select(func.count(BGPIncident.id)).where(*filters)
|
||||
)
|
||||
data_result = await db.execute(
|
||||
select(BGPIncident)
|
||||
.where(*filters)
|
||||
.order_by(BGPIncident.created_at.desc(), BGPIncident.id.desc())
|
||||
.offset(offset)
|
||||
.limit(page_size)
|
||||
)
|
||||
records = data_result.scalars().all()
|
||||
|
||||
return {
|
||||
"total": len(records),
|
||||
"total": total_result.scalar() or 0,
|
||||
"page": page,
|
||||
"page_size": page_size,
|
||||
"data": [record.to_dict() for record in records[offset : offset + page_size]],
|
||||
"data": [record.to_dict() for record in records],
|
||||
}
|
||||
|
||||
|
||||
@@ -268,29 +409,7 @@ async def get_bgp_incident_summary(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
total_result = await db.execute(select(func.count(BGPIncident.id)))
|
||||
type_result = await db.execute(
|
||||
select(BGPIncident.incident_type, func.count(BGPIncident.id))
|
||||
.group_by(BGPIncident.incident_type)
|
||||
.order_by(func.count(BGPIncident.id).desc())
|
||||
)
|
||||
severity_result = await db.execute(
|
||||
select(BGPIncident.severity, func.count(BGPIncident.id))
|
||||
.group_by(BGPIncident.severity)
|
||||
.order_by(func.count(BGPIncident.id).desc())
|
||||
)
|
||||
status_result = await db.execute(
|
||||
select(BGPIncident.status, func.count(BGPIncident.id))
|
||||
.group_by(BGPIncident.status)
|
||||
.order_by(func.count(BGPIncident.id).desc())
|
||||
)
|
||||
|
||||
return {
|
||||
"total": total_result.scalar() or 0,
|
||||
"by_type": {row[0]: row[1] for row in type_result.fetchall()},
|
||||
"by_severity": {row[0]: row[1] for row in severity_result.fetchall()},
|
||||
"by_status": {row[0]: row[1] for row in status_result.fetchall()},
|
||||
}
|
||||
return await _build_incident_summary_payload(db)
|
||||
|
||||
|
||||
@router.get("/incidents/{incident_id}")
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from fastapi import APIRouter, Depends
|
||||
from sqlalchemy import select, func, text
|
||||
from sqlalchemy import case, select, func, text
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.db.session import get_db
|
||||
@@ -118,58 +118,77 @@ async def get_stats(
|
||||
built_in_count = len(COLLECTOR_INFO)
|
||||
built_in_active = built_in_count # Built-in are always "active" for counting purposes
|
||||
|
||||
# Count custom configs from database
|
||||
result = await db.execute(select(func.count(DataSourceConfig.id)))
|
||||
custom_count = result.scalar() or 0
|
||||
|
||||
result = await db.execute(
|
||||
select(func.count(DataSourceConfig.id)).where(DataSourceConfig.is_active == True)
|
||||
select(
|
||||
func.count(DataSourceConfig.id).label("custom_count"),
|
||||
func.sum(
|
||||
case((DataSourceConfig.is_active == True, 1), else_=0)
|
||||
).label("custom_active"),
|
||||
)
|
||||
)
|
||||
custom_active = result.scalar() or 0
|
||||
datasource_stats = result.one()
|
||||
custom_count = datasource_stats.custom_count or 0
|
||||
custom_active = datasource_stats.custom_active or 0
|
||||
|
||||
# Total datasources
|
||||
total_datasources = built_in_count + custom_count
|
||||
active_datasources = built_in_active + custom_active
|
||||
|
||||
# Tasks today (from database)
|
||||
result = await db.execute(
|
||||
select(func.count(CollectionTask.id)).where(CollectionTask.started_at >= today_start)
|
||||
)
|
||||
tasks_today = result.scalar() or 0
|
||||
|
||||
result = await db.execute(
|
||||
select(func.count(CollectionTask.id)).where(
|
||||
CollectionTask.status == "success",
|
||||
CollectionTask.started_at >= today_start,
|
||||
select(
|
||||
func.count(CollectionTask.id).label("tasks_today"),
|
||||
func.sum(
|
||||
case(
|
||||
(CollectionTask.status == "success", 1),
|
||||
else_=0,
|
||||
)
|
||||
).label("success_tasks"),
|
||||
)
|
||||
.where(CollectionTask.started_at >= today_start)
|
||||
)
|
||||
success_tasks = result.scalar() or 0
|
||||
task_stats = result.one()
|
||||
tasks_today = task_stats.tasks_today or 0
|
||||
success_tasks = task_stats.success_tasks or 0
|
||||
success_rate = (success_tasks / tasks_today * 100) if tasks_today > 0 else 0
|
||||
|
||||
# Alerts
|
||||
result = await db.execute(
|
||||
select(func.count(Alert.id)).where(
|
||||
Alert.severity == AlertSeverity.CRITICAL,
|
||||
Alert.status == "active",
|
||||
select(
|
||||
func.sum(
|
||||
case(
|
||||
(
|
||||
(Alert.severity == AlertSeverity.CRITICAL)
|
||||
& (Alert.status == "active"),
|
||||
1,
|
||||
),
|
||||
else_=0,
|
||||
)
|
||||
).label("critical_alerts"),
|
||||
func.sum(
|
||||
case(
|
||||
(
|
||||
(Alert.severity == AlertSeverity.WARNING)
|
||||
& (Alert.status == "active"),
|
||||
1,
|
||||
),
|
||||
else_=0,
|
||||
)
|
||||
).label("warning_alerts"),
|
||||
func.sum(
|
||||
case(
|
||||
(
|
||||
(Alert.severity == AlertSeverity.INFO)
|
||||
& (Alert.status == "active"),
|
||||
1,
|
||||
),
|
||||
else_=0,
|
||||
)
|
||||
).label("info_alerts"),
|
||||
)
|
||||
)
|
||||
critical_alerts = result.scalar() or 0
|
||||
|
||||
result = await db.execute(
|
||||
select(func.count(Alert.id)).where(
|
||||
Alert.severity == AlertSeverity.WARNING,
|
||||
Alert.status == "active",
|
||||
)
|
||||
)
|
||||
warning_alerts = result.scalar() or 0
|
||||
|
||||
result = await db.execute(
|
||||
select(func.count(Alert.id)).where(
|
||||
Alert.severity == AlertSeverity.INFO,
|
||||
Alert.status == "active",
|
||||
)
|
||||
)
|
||||
info_alerts = result.scalar() or 0
|
||||
alert_stats = result.one()
|
||||
critical_alerts = alert_stats.critical_alerts or 0
|
||||
warning_alerts = alert_stats.warning_alerts or 0
|
||||
info_alerts = alert_stats.info_alerts or 0
|
||||
|
||||
response = {
|
||||
"total_datasources": total_datasources,
|
||||
|
||||
@@ -1,20 +1,41 @@
|
||||
"""DataSourceConfig API for user-defined data sources"""
|
||||
|
||||
from typing import Optional
|
||||
from typing import Any, Optional
|
||||
from datetime import datetime
|
||||
import base64
|
||||
import json
|
||||
import re
|
||||
from fastapi import APIRouter, Depends, HTTPException, status
|
||||
from sqlalchemy import select, func
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from pydantic import BaseModel, Field
|
||||
import httpx
|
||||
|
||||
from app.core.target_schema_registry import get_target_schema, list_target_schemas
|
||||
from app.core.datasource_defaults import DEFAULT_DATASOURCES
|
||||
from app.db.session import get_db
|
||||
from app.models.user import User
|
||||
from app.models.datasource_config import DataSourceConfig
|
||||
from app.models.datasource_mapping import DataSourceMappingTemplate
|
||||
from app.core.security import get_current_user
|
||||
from app.core.cache import cache
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.schemas.ai import SituationalAnalysisRequest
|
||||
from app.services.ai_client import AIProviderClient, get_ai_provider_client
|
||||
from app.services.datasource_mapping import (
|
||||
MappingError,
|
||||
build_heuristic_mapping,
|
||||
execute_mapping,
|
||||
persist_mapped_records,
|
||||
redact_for_llm,
|
||||
stable_payload_hash,
|
||||
)
|
||||
from app.services.datasource_connectivity import (
|
||||
get_builtin_connection_status,
|
||||
save_connectivity_success,
|
||||
strip_connectivity_validation,
|
||||
test_builtin_connectivity,
|
||||
)
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
@@ -59,6 +80,70 @@ class DataSourceConfigResponse(BaseModel):
|
||||
from_attributes = True
|
||||
|
||||
|
||||
def _is_builtin_config_name(name: str | None) -> bool:
|
||||
return bool(name and name in DEFAULT_DATASOURCES)
|
||||
|
||||
|
||||
async def _ensure_builtin_connection_verified(
|
||||
db: AsyncSession,
|
||||
config_data: DataSourceConfigCreate,
|
||||
) -> None:
|
||||
if not _is_builtin_config_name(config_data.name):
|
||||
return
|
||||
|
||||
status_result = await get_builtin_connection_status(
|
||||
db,
|
||||
config_data.name,
|
||||
config_data.endpoint,
|
||||
config_data.auth_type,
|
||||
config_data.headers,
|
||||
config_data.config,
|
||||
)
|
||||
if not status_result.get("connected"):
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=status_result.get("message") or "请先完成连接验证,再保存内置采集器配置。",
|
||||
)
|
||||
|
||||
|
||||
class CustomSampleRequest(BaseModel):
|
||||
datasource_config_id: Optional[int] = None
|
||||
config: Optional[DataSourceConfigCreate] = None
|
||||
limit_bytes: int = Field(default=200000, ge=1000, le=1000000)
|
||||
|
||||
|
||||
class MappingProposeRequest(BaseModel):
|
||||
sample_payload: Any
|
||||
target_schema: str
|
||||
use_ai: bool = True
|
||||
|
||||
|
||||
class MappingPreviewRequest(BaseModel):
|
||||
sample_payload: Any
|
||||
target_schema: str
|
||||
mapping_json: dict
|
||||
limit: int = Field(default=20, ge=1, le=100)
|
||||
|
||||
|
||||
class MappingTemplateCreate(BaseModel):
|
||||
datasource_config_id: int
|
||||
target_schema: str
|
||||
mapping_json: dict
|
||||
sample_payload: Any | None = None
|
||||
sample_payload_hash: Optional[str] = None
|
||||
validation_status: str = Field(default="draft", pattern="^(draft|valid|invalid)$")
|
||||
is_active: bool = False
|
||||
|
||||
|
||||
class MappingTemplateUpdate(BaseModel):
|
||||
target_schema: Optional[str] = None
|
||||
mapping_json: Optional[dict] = None
|
||||
sample_payload: Any | None = None
|
||||
sample_payload_hash: Optional[str] = None
|
||||
validation_status: Optional[str] = Field(default=None, pattern="^(draft|valid|invalid)$")
|
||||
is_active: Optional[bool] = None
|
||||
|
||||
|
||||
async def test_endpoint(
|
||||
endpoint: str,
|
||||
auth_type: str,
|
||||
@@ -96,6 +181,134 @@ async def test_endpoint(
|
||||
}
|
||||
|
||||
|
||||
def _build_request_headers(auth_type: str, auth_config: dict, headers: dict) -> dict[str, str]:
|
||||
request_headers = {str(key): str(value) for key, value in (headers or {}).items()}
|
||||
auth_type = str(auth_type or "none").lower()
|
||||
auth_config = auth_config or {}
|
||||
|
||||
if auth_type == "bearer" and auth_config.get("token"):
|
||||
request_headers["Authorization"] = f"Bearer {auth_config['token']}"
|
||||
elif auth_type == "api_key" and auth_config.get("api_key"):
|
||||
location = str(auth_config.get("in") or auth_config.get("location") or "header").lower()
|
||||
if location != "query":
|
||||
key_name = auth_config.get("key_name", "X-API-Key")
|
||||
request_headers[str(key_name)] = str(auth_config["api_key"])
|
||||
elif auth_type == "basic":
|
||||
username = auth_config.get("username", "")
|
||||
password = auth_config.get("password", "")
|
||||
credentials = f"{username}:{password}"
|
||||
encoded = base64.b64encode(credentials.encode()).decode()
|
||||
request_headers["Authorization"] = f"Basic {encoded}"
|
||||
return request_headers
|
||||
|
||||
|
||||
def _build_query_params(auth_type: str, auth_config: dict, config: dict) -> dict[str, Any]:
|
||||
params = {}
|
||||
candidate = (config or {}).get("params") or (config or {}).get("query_params")
|
||||
if isinstance(candidate, dict):
|
||||
params.update(candidate)
|
||||
|
||||
auth_type = str(auth_type or "none").lower()
|
||||
auth_config = auth_config or {}
|
||||
if auth_type == "api_key" and auth_config.get("api_key"):
|
||||
location = str(auth_config.get("in") or auth_config.get("location") or "header").lower()
|
||||
if location == "query":
|
||||
key_name = auth_config.get("key_name") or auth_config.get("param_name") or "api_key"
|
||||
params[str(key_name)] = auth_config["api_key"]
|
||||
return params
|
||||
|
||||
|
||||
async def fetch_custom_sample_from_config(config: DataSourceConfig, limit_bytes: int) -> Any:
|
||||
request_config = config.config or {}
|
||||
method = str(request_config.get("method") or request_config.get("request_method") or "GET").upper()
|
||||
if method not in {"GET", "POST"}:
|
||||
raise HTTPException(status_code=400, detail="Only GET and POST sample requests are supported.")
|
||||
|
||||
headers = _build_request_headers(config.auth_type, config.auth_config or {}, config.headers or {})
|
||||
params = _build_query_params(config.auth_type, config.auth_config or {}, request_config)
|
||||
timeout = float(request_config.get("timeout", 30))
|
||||
json_body = request_config.get("json_body")
|
||||
if json_body is None and str(request_config.get("body_type") or "").lower() in {"json", ""}:
|
||||
candidate = request_config.get("body")
|
||||
if isinstance(candidate, (dict, list)):
|
||||
json_body = candidate
|
||||
|
||||
async with httpx.AsyncClient(timeout=timeout, follow_redirects=True) as client:
|
||||
response = await client.request(
|
||||
method,
|
||||
config.endpoint,
|
||||
headers=headers,
|
||||
params=params or None,
|
||||
json=json_body,
|
||||
)
|
||||
response.raise_for_status()
|
||||
content = response.content[:limit_bytes]
|
||||
if "application/json" in response.headers.get("content-type", ""):
|
||||
return json.loads(content.decode(response.encoding or "utf-8"))
|
||||
return {"text": content.decode(response.encoding or "utf-8", errors="replace")}
|
||||
|
||||
|
||||
def _parse_mapping_from_ai_text(content: str) -> dict[str, Any] | None:
|
||||
if not content:
|
||||
return None
|
||||
|
||||
candidates = [content]
|
||||
fenced = re.findall(r"```(?:json)?\s*(\{.*?\})\s*```", content, flags=re.DOTALL)
|
||||
candidates = fenced + candidates
|
||||
for candidate in candidates:
|
||||
try:
|
||||
parsed = json.loads(candidate)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
if isinstance(parsed, dict) and isinstance(parsed.get("fields"), dict):
|
||||
return parsed
|
||||
return None
|
||||
|
||||
|
||||
async def _get_config_for_sample(
|
||||
payload: CustomSampleRequest,
|
||||
db: AsyncSession,
|
||||
) -> DataSourceConfig:
|
||||
if payload.datasource_config_id is not None:
|
||||
result = await db.execute(
|
||||
select(DataSourceConfig).where(DataSourceConfig.id == payload.datasource_config_id)
|
||||
)
|
||||
config = result.scalar_one_or_none()
|
||||
if not config:
|
||||
raise HTTPException(status_code=404, detail="Configuration not found")
|
||||
return config
|
||||
|
||||
if payload.config is None:
|
||||
raise HTTPException(status_code=400, detail="datasource_config_id or config is required")
|
||||
|
||||
config_data = payload.config
|
||||
return DataSourceConfig(
|
||||
name=config_data.name,
|
||||
description=config_data.description,
|
||||
source_type=config_data.source_type,
|
||||
endpoint=config_data.endpoint,
|
||||
auth_type=config_data.auth_type,
|
||||
auth_config=config_data.auth_config,
|
||||
headers=config_data.headers,
|
||||
config=config_data.config,
|
||||
)
|
||||
|
||||
|
||||
def serialize_mapping_template(template: DataSourceMappingTemplate) -> dict[str, Any]:
|
||||
return {
|
||||
"id": template.id,
|
||||
"datasource_config_id": template.datasource_config_id,
|
||||
"target_schema": template.target_schema,
|
||||
"mapping_json": template.mapping_json,
|
||||
"sample_payload_hash": template.sample_payload_hash,
|
||||
"validation_status": template.validation_status,
|
||||
"version": template.version,
|
||||
"is_active": template.is_active,
|
||||
"created_at": to_iso8601_utc(template.created_at),
|
||||
"updated_at": to_iso8601_utc(template.updated_at),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/configs")
|
||||
async def list_configs(
|
||||
active_only: bool = False,
|
||||
@@ -132,6 +345,47 @@ async def list_configs(
|
||||
}
|
||||
|
||||
|
||||
@router.get("/configs/all")
|
||||
async def list_all_datasources(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""List all data sources: YAML defaults + DB overrides"""
|
||||
from app.core.data_sources import COLLECTOR_URL_KEYS, get_data_sources_config
|
||||
|
||||
config = get_data_sources_config()
|
||||
|
||||
db_query = await db.execute(select(DataSourceConfig))
|
||||
db_configs = {c.name: c for c in db_query.scalars().all()}
|
||||
|
||||
result = []
|
||||
for name, yaml_key in COLLECTOR_URL_KEYS.items():
|
||||
yaml_url = config.get_yaml_url(name)
|
||||
db_config = db_configs.get(name)
|
||||
|
||||
result.append(
|
||||
{
|
||||
"name": name,
|
||||
"default_url": yaml_url,
|
||||
"endpoint": db_config.endpoint if db_config else yaml_url,
|
||||
"is_overridden": db_config is not None and db_config.endpoint != yaml_url
|
||||
if yaml_url
|
||||
else db_config is not None,
|
||||
"is_active": db_config.is_active if db_config else True,
|
||||
"source_type": db_config.source_type if db_config else "http",
|
||||
"auth_type": db_config.auth_type if db_config else "none",
|
||||
"headers": db_config.headers if db_config else {},
|
||||
"config": strip_connectivity_validation(db_config.config if db_config else {}),
|
||||
"config_id": db_config.id if db_config else None,
|
||||
"description": db_config.description
|
||||
if db_config
|
||||
else f"Data source from YAML: {yaml_key}",
|
||||
}
|
||||
)
|
||||
|
||||
return {"total": len(result), "data": result}
|
||||
|
||||
|
||||
@router.get("/configs/{config_id}")
|
||||
async def get_config(
|
||||
config_id: int,
|
||||
@@ -176,7 +430,7 @@ async def create_config(
|
||||
auth_type=config_data.auth_type,
|
||||
auth_config=config_data.auth_config,
|
||||
headers=config_data.headers,
|
||||
config=config_data.config,
|
||||
config=strip_connectivity_validation(config_data.config),
|
||||
)
|
||||
|
||||
db.add(config)
|
||||
@@ -208,6 +462,8 @@ async def update_config(
|
||||
|
||||
update_data = config_data.model_dump(exclude_unset=True)
|
||||
for field, value in update_data.items():
|
||||
if field == "config":
|
||||
value = strip_connectivity_validation(value)
|
||||
setattr(config, field, value)
|
||||
|
||||
await db.commit()
|
||||
@@ -310,38 +566,366 @@ async def test_new_config(
|
||||
}
|
||||
|
||||
|
||||
@router.get("/configs/all")
|
||||
async def list_all_datasources(
|
||||
@router.post("/configs/builtin/connection-status")
|
||||
async def get_builtin_config_connection_status(
|
||||
config_data: DataSourceConfigCreate,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""List all data sources: YAML defaults + DB overrides"""
|
||||
from app.core.data_sources import COLLECTOR_URL_KEYS, get_data_sources_config
|
||||
if not _is_builtin_config_name(config_data.name):
|
||||
raise HTTPException(status_code=400, detail="Only built-in datasource configs are supported.")
|
||||
|
||||
config = get_data_sources_config()
|
||||
return await get_builtin_connection_status(
|
||||
db,
|
||||
config_data.name,
|
||||
config_data.endpoint,
|
||||
config_data.auth_type,
|
||||
config_data.headers,
|
||||
config_data.config,
|
||||
)
|
||||
|
||||
db_query = await db.execute(select(DataSourceConfig))
|
||||
db_configs = {c.name: c for c in db_query.scalars().all()}
|
||||
|
||||
result = []
|
||||
for name, yaml_key in COLLECTOR_URL_KEYS.items():
|
||||
yaml_url = config.get_yaml_url(name)
|
||||
db_config = db_configs.get(name)
|
||||
@router.post("/configs/builtin/connect")
|
||||
async def connect_builtin_config(
|
||||
config_data: DataSourceConfigCreate,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
if not _is_builtin_config_name(config_data.name):
|
||||
raise HTTPException(status_code=400, detail="Only built-in datasource configs are supported.")
|
||||
|
||||
result.append(
|
||||
result = await test_builtin_connectivity(
|
||||
config_data.name,
|
||||
config_data.endpoint,
|
||||
config_data.auth_type,
|
||||
config_data.headers,
|
||||
config_data.config,
|
||||
db,
|
||||
)
|
||||
if result.get("success") and result.get("checksum"):
|
||||
validation = await save_connectivity_success(
|
||||
db,
|
||||
config_data.name,
|
||||
result["checksum"],
|
||||
result,
|
||||
connected_by="connection_button",
|
||||
)
|
||||
await db.commit()
|
||||
return {
|
||||
**result,
|
||||
"connected": True,
|
||||
"validation": validation,
|
||||
}
|
||||
|
||||
return {
|
||||
**result,
|
||||
"connected": False,
|
||||
}
|
||||
|
||||
|
||||
@router.post("/custom/sample")
|
||||
async def fetch_custom_sample(
|
||||
payload: CustomSampleRequest,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""Fetch a sample payload for a saved or draft custom data source."""
|
||||
config = await _get_config_for_sample(payload, db)
|
||||
try:
|
||||
sample = await fetch_custom_sample_from_config(config, payload.limit_bytes)
|
||||
except httpx.HTTPStatusError as exc:
|
||||
raise HTTPException(
|
||||
status_code=exc.response.status_code,
|
||||
detail=f"Sample request failed: HTTP {exc.response.status_code}",
|
||||
) from exc
|
||||
except httpx.HTTPError as exc:
|
||||
raise HTTPException(status_code=502, detail=f"Sample request failed: {exc}") from exc
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"sample_payload": sample,
|
||||
"sample_payload_hash": stable_payload_hash(sample),
|
||||
"redacted_preview": redact_for_llm(sample),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/target-schemas")
|
||||
async def get_datasource_target_schemas(
|
||||
current_user: User = Depends(get_current_user),
|
||||
):
|
||||
"""List target schemas available for custom datasource mapping."""
|
||||
return {"data": list_target_schemas()}
|
||||
|
||||
|
||||
@router.post("/mappings/propose")
|
||||
async def propose_datasource_mapping(
|
||||
payload: MappingProposeRequest,
|
||||
current_user: User = Depends(get_current_user),
|
||||
ai_client: AIProviderClient = Depends(get_ai_provider_client),
|
||||
):
|
||||
"""Generate a mapping draft for a sample payload and target schema."""
|
||||
schema = get_target_schema(payload.target_schema)
|
||||
redacted_sample = redact_for_llm(payload.sample_payload)
|
||||
fallback_mapping = build_heuristic_mapping(redacted_sample, payload.target_schema)
|
||||
|
||||
ai_error: str | None = None
|
||||
mapping = fallback_mapping
|
||||
generated_by = "heuristic"
|
||||
if payload.use_ai:
|
||||
try:
|
||||
response = await ai_client.analyze(
|
||||
SituationalAnalysisRequest(
|
||||
title=f"Generate datasource mapping for {schema.key}",
|
||||
objective=(
|
||||
"Return only JSON for a deterministic mapping DSL. "
|
||||
"The JSON must contain source.items_path and fields. "
|
||||
"Do not include prose or code."
|
||||
),
|
||||
context={
|
||||
"target_schema": schema.to_dict(),
|
||||
"sample_payload": redacted_sample,
|
||||
"mapping_dsl_example": fallback_mapping,
|
||||
},
|
||||
observations=[
|
||||
"Use JSONPath-like paths beginning with $.",
|
||||
"Never generate executable code.",
|
||||
"Use field types from the target schema.",
|
||||
],
|
||||
constraints=[
|
||||
"Return a single JSON object.",
|
||||
"Do not include credentials or secrets.",
|
||||
"Mark uncertain optional fields with default null.",
|
||||
],
|
||||
)
|
||||
)
|
||||
parsed = _parse_mapping_from_ai_text(response.content)
|
||||
if parsed:
|
||||
mapping = parsed
|
||||
generated_by = "ai_provider"
|
||||
else:
|
||||
ai_error = "AI provider did not return a valid mapping JSON object."
|
||||
except HTTPException as exc:
|
||||
ai_error = str(exc.detail)
|
||||
|
||||
mapping.setdefault("meta", {})
|
||||
if isinstance(mapping["meta"], dict):
|
||||
mapping["meta"].update(
|
||||
{
|
||||
"name": name,
|
||||
"default_url": yaml_url,
|
||||
"endpoint": db_config.endpoint if db_config else yaml_url,
|
||||
"is_overridden": db_config is not None and db_config.endpoint != yaml_url
|
||||
if yaml_url
|
||||
else db_config is not None,
|
||||
"is_active": db_config.is_active if db_config else True,
|
||||
"source_type": db_config.source_type if db_config else "http",
|
||||
"description": db_config.description
|
||||
if db_config
|
||||
else f"Data source from YAML: {yaml_key}",
|
||||
"generated_by": generated_by,
|
||||
"requires_review": True,
|
||||
"ai_error": ai_error,
|
||||
}
|
||||
)
|
||||
|
||||
return {"total": len(result), "data": result}
|
||||
return {
|
||||
"target_schema": schema.to_dict(),
|
||||
"mapping_json": mapping,
|
||||
"sample_payload_hash": stable_payload_hash(payload.sample_payload),
|
||||
"redacted_sample_payload": redacted_sample,
|
||||
}
|
||||
|
||||
|
||||
@router.post("/mappings/preview")
|
||||
async def preview_datasource_mapping(
|
||||
payload: MappingPreviewRequest,
|
||||
current_user: User = Depends(get_current_user),
|
||||
):
|
||||
"""Preview deterministic mapping output for a sample payload."""
|
||||
try:
|
||||
preview = execute_mapping(
|
||||
payload.sample_payload,
|
||||
payload.mapping_json,
|
||||
payload.target_schema,
|
||||
limit=payload.limit,
|
||||
)
|
||||
except (MappingError, ValueError) as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
||||
|
||||
return {
|
||||
"success": preview["failed_count"] == 0,
|
||||
"preview": preview,
|
||||
"sample_payload_hash": stable_payload_hash(payload.sample_payload),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/mappings")
|
||||
async def list_datasource_mappings(
|
||||
datasource_config_id: Optional[int] = None,
|
||||
active_only: bool = False,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""List saved mapping templates."""
|
||||
query = select(DataSourceMappingTemplate).order_by(
|
||||
DataSourceMappingTemplate.datasource_config_id,
|
||||
DataSourceMappingTemplate.version.desc(),
|
||||
)
|
||||
if datasource_config_id is not None:
|
||||
query = query.where(DataSourceMappingTemplate.datasource_config_id == datasource_config_id)
|
||||
if active_only:
|
||||
query = query.where(DataSourceMappingTemplate.is_active.is_(True))
|
||||
|
||||
result = await db.execute(query)
|
||||
mappings = result.scalars().all()
|
||||
return {"total": len(mappings), "data": [serialize_mapping_template(item) for item in mappings]}
|
||||
|
||||
|
||||
@router.post("/mappings")
|
||||
async def create_datasource_mapping(
|
||||
payload: MappingTemplateCreate,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""Save a mapping template for a datasource config."""
|
||||
get_target_schema(payload.target_schema)
|
||||
datasource = await db.get(DataSourceConfig, payload.datasource_config_id)
|
||||
if not datasource:
|
||||
raise HTTPException(status_code=404, detail="Configuration not found")
|
||||
|
||||
if payload.sample_payload is not None:
|
||||
try:
|
||||
execute_mapping(payload.sample_payload, payload.mapping_json, payload.target_schema, limit=100)
|
||||
except (MappingError, ValueError) as exc:
|
||||
raise HTTPException(status_code=400, detail=f"Mapping validation failed: {exc}") from exc
|
||||
|
||||
result = await db.execute(
|
||||
select(func.max(DataSourceMappingTemplate.version)).where(
|
||||
DataSourceMappingTemplate.datasource_config_id == payload.datasource_config_id,
|
||||
DataSourceMappingTemplate.target_schema == payload.target_schema,
|
||||
)
|
||||
)
|
||||
next_version = int(result.scalar() or 0) + 1
|
||||
|
||||
if payload.is_active:
|
||||
await db.execute(
|
||||
DataSourceMappingTemplate.__table__.update()
|
||||
.where(DataSourceMappingTemplate.datasource_config_id == payload.datasource_config_id)
|
||||
.values(is_active=False)
|
||||
)
|
||||
|
||||
template = DataSourceMappingTemplate(
|
||||
datasource_config_id=payload.datasource_config_id,
|
||||
target_schema=payload.target_schema,
|
||||
mapping_json=payload.mapping_json,
|
||||
sample_payload_hash=payload.sample_payload_hash
|
||||
or (stable_payload_hash(payload.sample_payload) if payload.sample_payload is not None else None),
|
||||
validation_status=payload.validation_status,
|
||||
version=next_version,
|
||||
is_active=payload.is_active,
|
||||
)
|
||||
db.add(template)
|
||||
await db.commit()
|
||||
await db.refresh(template)
|
||||
return {"message": "Mapping template saved successfully", "data": serialize_mapping_template(template)}
|
||||
|
||||
|
||||
@router.put("/mappings/{mapping_id}")
|
||||
async def update_datasource_mapping(
|
||||
mapping_id: int,
|
||||
payload: MappingTemplateUpdate,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""Update a mapping template in place."""
|
||||
template = await db.get(DataSourceMappingTemplate, mapping_id)
|
||||
if not template:
|
||||
raise HTTPException(status_code=404, detail="Mapping template not found")
|
||||
|
||||
target_schema = payload.target_schema or template.target_schema
|
||||
mapping_json = payload.mapping_json or template.mapping_json
|
||||
get_target_schema(target_schema)
|
||||
if payload.sample_payload is not None:
|
||||
try:
|
||||
execute_mapping(payload.sample_payload, mapping_json, target_schema, limit=100)
|
||||
except (MappingError, ValueError) as exc:
|
||||
raise HTTPException(status_code=400, detail=f"Mapping validation failed: {exc}") from exc
|
||||
|
||||
if payload.is_active is True:
|
||||
await db.execute(
|
||||
DataSourceMappingTemplate.__table__.update()
|
||||
.where(DataSourceMappingTemplate.datasource_config_id == template.datasource_config_id)
|
||||
.where(DataSourceMappingTemplate.id != template.id)
|
||||
.values(is_active=False)
|
||||
)
|
||||
|
||||
template.target_schema = target_schema
|
||||
template.mapping_json = mapping_json
|
||||
if payload.sample_payload_hash is not None:
|
||||
template.sample_payload_hash = payload.sample_payload_hash
|
||||
elif payload.sample_payload is not None:
|
||||
template.sample_payload_hash = stable_payload_hash(payload.sample_payload)
|
||||
if payload.validation_status is not None:
|
||||
template.validation_status = payload.validation_status
|
||||
if payload.is_active is not None:
|
||||
template.is_active = payload.is_active
|
||||
|
||||
await db.commit()
|
||||
await db.refresh(template)
|
||||
return {"message": "Mapping template updated successfully", "data": serialize_mapping_template(template)}
|
||||
|
||||
|
||||
@router.post("/{config_id}/run-mapped")
|
||||
async def run_mapped_datasource(
|
||||
config_id: int,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""Run a saved custom datasource through its active deterministic mapping."""
|
||||
datasource = await db.get(DataSourceConfig, config_id)
|
||||
if not datasource:
|
||||
raise HTTPException(status_code=404, detail="Configuration not found")
|
||||
|
||||
result = await db.execute(
|
||||
select(DataSourceMappingTemplate)
|
||||
.where(DataSourceMappingTemplate.datasource_config_id == config_id)
|
||||
.where(DataSourceMappingTemplate.is_active.is_(True))
|
||||
.order_by(DataSourceMappingTemplate.version.desc())
|
||||
.limit(1)
|
||||
)
|
||||
mapping = result.scalar_one_or_none()
|
||||
if not mapping:
|
||||
raise HTTPException(status_code=404, detail="No active mapping template found")
|
||||
|
||||
try:
|
||||
sample = await fetch_custom_sample_from_config(datasource, 5_000_000)
|
||||
mapped = execute_mapping(sample, mapping.mapping_json, mapping.target_schema)
|
||||
except httpx.HTTPStatusError as exc:
|
||||
raise HTTPException(
|
||||
status_code=exc.response.status_code,
|
||||
detail=f"Datasource request failed: HTTP {exc.response.status_code}",
|
||||
) from exc
|
||||
except httpx.HTTPError as exc:
|
||||
raise HTTPException(status_code=502, detail=f"Datasource request failed: {exc}") from exc
|
||||
except (MappingError, ValueError) as exc:
|
||||
raise HTTPException(status_code=400, detail=f"Mapping failed: {exc}") from exc
|
||||
|
||||
if mapped["failed_count"] > 0:
|
||||
return {
|
||||
"status": "failed",
|
||||
"datasource_config_id": config_id,
|
||||
"mapping_id": mapping.id,
|
||||
"mapping_version": mapping.version,
|
||||
"target_schema": mapping.target_schema,
|
||||
"mapped_count": mapped["mapped_count"],
|
||||
"failed_count": mapped["failed_count"],
|
||||
"errors": mapped["errors"][:20],
|
||||
}
|
||||
|
||||
written_count = await persist_mapped_records(
|
||||
db,
|
||||
datasource_name=datasource.name,
|
||||
datasource_config_id=datasource.id,
|
||||
target_schema=mapping.target_schema,
|
||||
records=mapped["records"],
|
||||
mapping_version=mapping.version,
|
||||
)
|
||||
return {
|
||||
"status": "success",
|
||||
"datasource_config_id": config_id,
|
||||
"mapping_id": mapping.id,
|
||||
"mapping_version": mapping.version,
|
||||
"target_schema": mapping.target_schema,
|
||||
"fetched_count": mapped["total_items"],
|
||||
"mapped_count": mapped["mapped_count"],
|
||||
"written_count": written_count,
|
||||
}
|
||||
@@ -3,18 +3,26 @@ from datetime import datetime, timedelta, timezone
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy import func, select, text
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.core.security import get_current_user
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
from app.core.datasource_defaults import DEFAULT_DATASOURCES
|
||||
from app.db.session import get_db
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.models.data_snapshot import DataSnapshot
|
||||
from app.models.datasource import DataSource
|
||||
from app.models.datasource_config import DataSourceConfig
|
||||
from app.models.task import CollectionTask
|
||||
from app.models.user import User
|
||||
from app.services.scheduler import get_latest_task_id_for_datasource, run_collector_now, sync_datasource_job
|
||||
from app.services.scheduler import (
|
||||
cancel_running_collector_now,
|
||||
get_latest_task_id_for_datasource,
|
||||
run_collector_now,
|
||||
sync_datasource_job,
|
||||
)
|
||||
|
||||
router = APIRouter()
|
||||
STALE_RUNNING_TASK_TIMEOUT_MINUTES = 90
|
||||
@@ -28,12 +36,131 @@ def format_frequency_label(minutes: int) -> str:
|
||||
return f"{minutes}m"
|
||||
|
||||
|
||||
def datasource_metadata(source: str) -> dict:
|
||||
info = DEFAULT_DATASOURCES.get(source, {})
|
||||
return {
|
||||
"display_name": info.get("display_name") or info.get("name") or source,
|
||||
"is_free": bool(info.get("is_free", True)),
|
||||
"requires_credentials": bool(info.get("requires_credentials", False)),
|
||||
"credential_provider": info.get("credential_provider"),
|
||||
"credential_status": info.get("credential_status", "none"),
|
||||
}
|
||||
|
||||
|
||||
def is_due_for_collection(datasource: DataSource, now: datetime) -> bool:
|
||||
if datasource.last_run_at is None:
|
||||
return True
|
||||
return datasource.last_run_at + timedelta(minutes=datasource.frequency_minutes) <= now
|
||||
|
||||
|
||||
def _task_rank_column(order_column):
|
||||
return func.row_number().over(
|
||||
partition_by=CollectionTask.datasource_id,
|
||||
order_by=(order_column.desc().nullslast(), CollectionTask.id.desc()),
|
||||
).label("row_num")
|
||||
|
||||
|
||||
async def _load_latest_running_tasks(
|
||||
db: AsyncSession,
|
||||
datasource_ids: list[int],
|
||||
) -> dict[int, CollectionTask]:
|
||||
if not datasource_ids:
|
||||
return {}
|
||||
|
||||
ranked_tasks = (
|
||||
select(
|
||||
CollectionTask.id.label("task_id"),
|
||||
_task_rank_column(CollectionTask.started_at),
|
||||
)
|
||||
.where(CollectionTask.datasource_id.in_(datasource_ids))
|
||||
.where(CollectionTask.status == "running")
|
||||
.subquery()
|
||||
)
|
||||
result = await db.execute(
|
||||
select(CollectionTask)
|
||||
.join(ranked_tasks, CollectionTask.id == ranked_tasks.c.task_id)
|
||||
.where(ranked_tasks.c.row_num == 1)
|
||||
)
|
||||
return {task.datasource_id: task for task in result.scalars().all()}
|
||||
|
||||
|
||||
async def _load_latest_task_ids(
|
||||
db: AsyncSession,
|
||||
datasource_ids: list[int],
|
||||
) -> dict[int, int]:
|
||||
if not datasource_ids:
|
||||
return {}
|
||||
|
||||
ranked_tasks = (
|
||||
select(
|
||||
CollectionTask.id.label("task_id"),
|
||||
CollectionTask.datasource_id.label("datasource_id"),
|
||||
func.row_number().over(
|
||||
partition_by=CollectionTask.datasource_id,
|
||||
order_by=CollectionTask.id.desc(),
|
||||
).label("row_num"),
|
||||
)
|
||||
.where(CollectionTask.datasource_id.in_(datasource_ids))
|
||||
.subquery()
|
||||
)
|
||||
result = await db.execute(
|
||||
select(ranked_tasks.c.datasource_id, ranked_tasks.c.task_id)
|
||||
.where(ranked_tasks.c.row_num == 1)
|
||||
)
|
||||
return {datasource_id: task_id for datasource_id, task_id in result.all()}
|
||||
|
||||
|
||||
async def _load_datasource_endpoint_overrides(
|
||||
db: AsyncSession,
|
||||
sources: list[str],
|
||||
) -> dict[str, str]:
|
||||
if not sources:
|
||||
return {}
|
||||
|
||||
result = await db.execute(
|
||||
select(DataSourceConfig.name, DataSourceConfig.endpoint)
|
||||
.where(DataSourceConfig.name.in_(sources))
|
||||
.where(DataSourceConfig.is_active.is_(True))
|
||||
.where(DataSourceConfig.endpoint.isnot(None))
|
||||
)
|
||||
return {
|
||||
name: endpoint
|
||||
for name, endpoint in result.all()
|
||||
if endpoint
|
||||
}
|
||||
|
||||
|
||||
async def _load_datasource_list_context(
|
||||
db: AsyncSession,
|
||||
datasources: list[DataSource],
|
||||
) -> tuple[dict[int, CollectionTask], dict[str, str]]:
|
||||
datasource_ids = [datasource.id for datasource in datasources]
|
||||
sources = [datasource.source for datasource in datasources]
|
||||
|
||||
running_tasks = await _load_latest_running_tasks(db, datasource_ids)
|
||||
datasource_by_id = {datasource.id: datasource for datasource in datasources}
|
||||
now = datetime.now(timezone.utc)
|
||||
|
||||
stale_datasource_ids: list[int] = []
|
||||
for datasource_id, task in running_tasks.items():
|
||||
started_at = task.started_at
|
||||
if started_at is None:
|
||||
continue
|
||||
if started_at.tzinfo is None:
|
||||
started_at = started_at.replace(tzinfo=timezone.utc)
|
||||
if now - started_at > timedelta(minutes=STALE_RUNNING_TASK_TIMEOUT_MINUTES):
|
||||
datasource = datasource_by_id.get(datasource_id)
|
||||
if datasource is not None:
|
||||
await fail_and_rollback_stale_running_task(db, datasource, task)
|
||||
stale_datasource_ids.append(datasource_id)
|
||||
|
||||
if stale_datasource_ids:
|
||||
running_tasks = await _load_latest_running_tasks(db, datasource_ids)
|
||||
|
||||
endpoint_overrides = await _load_datasource_endpoint_overrides(db, sources)
|
||||
return running_tasks, endpoint_overrides
|
||||
|
||||
|
||||
async def get_datasource_record(db: AsyncSession, source_id: str) -> Optional[DataSource]:
|
||||
datasource = None
|
||||
try:
|
||||
@@ -52,18 +179,6 @@ async def get_datasource_record(db: AsyncSession, source_id: str) -> Optional[Da
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
|
||||
async def get_last_completed_task(db: AsyncSession, datasource_id: int) -> Optional[CollectionTask]:
|
||||
result = await db.execute(
|
||||
select(CollectionTask)
|
||||
.where(CollectionTask.datasource_id == datasource_id)
|
||||
.where(CollectionTask.completed_at.isnot(None))
|
||||
.where(CollectionTask.status.in_(("success", "failed", "cancelled")))
|
||||
.order_by(CollectionTask.completed_at.desc())
|
||||
.limit(1)
|
||||
)
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
|
||||
async def get_running_task(db: AsyncSession, datasource_id: int) -> Optional[CollectionTask]:
|
||||
result = await db.execute(
|
||||
select(CollectionTask)
|
||||
@@ -87,17 +202,152 @@ async def get_running_task(db: AsyncSession, datasource_id: int) -> Optional[Col
|
||||
if now - started_at <= timedelta(minutes=STALE_RUNNING_TASK_TIMEOUT_MINUTES):
|
||||
return task
|
||||
|
||||
existing_error = (task.error_message or "").strip()
|
||||
datasource = await db.get(DataSource, datasource_id)
|
||||
if datasource is not None:
|
||||
await fail_and_rollback_stale_running_task(db, datasource, task)
|
||||
else:
|
||||
existing_error = (task.error_message or "").strip()
|
||||
stale_reason = (
|
||||
f"Marked failed automatically after stale running timeout "
|
||||
f"({STALE_RUNNING_TASK_TIMEOUT_MINUTES}m)"
|
||||
)
|
||||
task.status = "failed"
|
||||
task.phase = "failed"
|
||||
task.completed_at = now
|
||||
task.error_message = f"{existing_error}\n{stale_reason}".strip() if existing_error else stale_reason
|
||||
await db.commit()
|
||||
return None
|
||||
|
||||
|
||||
async def rollback_orphaned_running_task(
|
||||
db: AsyncSession,
|
||||
datasource: DataSource,
|
||||
running_task: CollectionTask,
|
||||
) -> None:
|
||||
snapshot_result = await db.execute(
|
||||
select(DataSnapshot)
|
||||
.where(
|
||||
DataSnapshot.datasource_id == datasource.id,
|
||||
DataSnapshot.task_id == running_task.id,
|
||||
)
|
||||
.order_by(DataSnapshot.id.desc())
|
||||
.limit(1)
|
||||
)
|
||||
snapshot = snapshot_result.scalar_one_or_none()
|
||||
|
||||
await db.execute(CollectedData.__table__.delete().where(CollectedData.task_id == running_task.id))
|
||||
|
||||
await db.execute(
|
||||
text(
|
||||
"""
|
||||
UPDATE collected_data
|
||||
SET is_current = FALSE
|
||||
WHERE source = :source
|
||||
"""
|
||||
),
|
||||
{"source": datasource.source},
|
||||
)
|
||||
|
||||
if snapshot is not None:
|
||||
snapshot.status = "cancelled"
|
||||
snapshot.is_current = False
|
||||
snapshot.completed_at = datetime.now(timezone.utc)
|
||||
summary = dict(snapshot.summary or {})
|
||||
summary["rollback"] = True
|
||||
summary["rollback_reason"] = "orphaned_running_task_after_backend_restart"
|
||||
snapshot.summary = summary
|
||||
|
||||
if snapshot.parent_snapshot_id is not None:
|
||||
parent_snapshot = await db.get(DataSnapshot, snapshot.parent_snapshot_id)
|
||||
if parent_snapshot:
|
||||
parent_snapshot.is_current = True
|
||||
await db.execute(
|
||||
text(
|
||||
"""
|
||||
UPDATE collected_data
|
||||
SET is_current = TRUE
|
||||
WHERE snapshot_id = :snapshot_id
|
||||
"""
|
||||
),
|
||||
{"snapshot_id": snapshot.parent_snapshot_id},
|
||||
)
|
||||
|
||||
running_task.status = "cancelled"
|
||||
running_task.phase = "cancelled"
|
||||
running_task.completed_at = datetime.now(timezone.utc)
|
||||
existing_error = (running_task.error_message or "").strip()
|
||||
cancel_reason = "Cancelled after backend restart because the running task handle was lost; incomplete writes rolled back"
|
||||
running_task.error_message = f"{existing_error}\n{cancel_reason}".strip() if existing_error else cancel_reason
|
||||
datasource.last_status = "cancelled"
|
||||
datasource.last_run_at = datetime.now(timezone.utc)
|
||||
await db.commit()
|
||||
|
||||
|
||||
async def fail_and_rollback_stale_running_task(
|
||||
db: AsyncSession,
|
||||
datasource: DataSource,
|
||||
running_task: CollectionTask,
|
||||
) -> None:
|
||||
snapshot_result = await db.execute(
|
||||
select(DataSnapshot)
|
||||
.where(
|
||||
DataSnapshot.datasource_id == datasource.id,
|
||||
DataSnapshot.task_id == running_task.id,
|
||||
)
|
||||
.order_by(DataSnapshot.id.desc())
|
||||
.limit(1)
|
||||
)
|
||||
snapshot = snapshot_result.scalar_one_or_none()
|
||||
|
||||
await db.execute(CollectedData.__table__.delete().where(CollectedData.task_id == running_task.id))
|
||||
|
||||
await db.execute(
|
||||
text(
|
||||
"""
|
||||
UPDATE collected_data
|
||||
SET is_current = FALSE
|
||||
WHERE source = :source
|
||||
"""
|
||||
),
|
||||
{"source": datasource.source},
|
||||
)
|
||||
|
||||
if snapshot is not None:
|
||||
snapshot.status = "failed"
|
||||
snapshot.is_current = False
|
||||
snapshot.completed_at = datetime.now(timezone.utc)
|
||||
summary = dict(snapshot.summary or {})
|
||||
summary["rollback"] = True
|
||||
summary["rollback_reason"] = "stale_running_task_timeout"
|
||||
snapshot.summary = summary
|
||||
|
||||
if snapshot.parent_snapshot_id is not None:
|
||||
parent_snapshot = await db.get(DataSnapshot, snapshot.parent_snapshot_id)
|
||||
if parent_snapshot:
|
||||
parent_snapshot.is_current = True
|
||||
await db.execute(
|
||||
text(
|
||||
"""
|
||||
UPDATE collected_data
|
||||
SET is_current = TRUE
|
||||
WHERE snapshot_id = :snapshot_id
|
||||
"""
|
||||
),
|
||||
{"snapshot_id": snapshot.parent_snapshot_id},
|
||||
)
|
||||
|
||||
existing_error = (running_task.error_message or "").strip()
|
||||
stale_reason = (
|
||||
f"Marked failed automatically after stale running timeout "
|
||||
f"({STALE_RUNNING_TASK_TIMEOUT_MINUTES}m)"
|
||||
f"({STALE_RUNNING_TASK_TIMEOUT_MINUTES}m); incomplete writes rolled back"
|
||||
)
|
||||
task.status = "failed"
|
||||
task.phase = "failed"
|
||||
task.completed_at = now
|
||||
task.error_message = f"{existing_error}\n{stale_reason}".strip() if existing_error else stale_reason
|
||||
running_task.status = "failed"
|
||||
running_task.phase = "failed"
|
||||
running_task.completed_at = datetime.now(timezone.utc)
|
||||
running_task.error_message = f"{existing_error}\n{stale_reason}".strip() if existing_error else stale_reason
|
||||
datasource.last_status = "failed"
|
||||
datasource.last_run_at = datetime.now(timezone.utc)
|
||||
await db.commit()
|
||||
return None
|
||||
|
||||
|
||||
@router.get("")
|
||||
@@ -121,23 +371,19 @@ async def list_datasources(
|
||||
|
||||
collector_list = []
|
||||
config = get_data_sources_config()
|
||||
running_tasks, endpoint_overrides = await _load_datasource_list_context(db, datasources)
|
||||
for datasource in datasources:
|
||||
running_task = await get_running_task(db, datasource.id)
|
||||
last_task = await get_last_completed_task(db, datasource.id)
|
||||
endpoint = await config.get_url(datasource.source, db)
|
||||
data_count_result = await db.execute(
|
||||
select(func.count(CollectedData.id)).where(CollectedData.source == datasource.source)
|
||||
)
|
||||
data_count = data_count_result.scalar() or 0
|
||||
|
||||
last_run_at = datasource.last_run_at or (last_task.completed_at if last_task else None)
|
||||
last_run = to_iso8601_utc(last_run_at)
|
||||
last_status = datasource.last_status or (last_task.status if last_task else None)
|
||||
running_task = running_tasks.get(datasource.id)
|
||||
endpoint = endpoint_overrides.get(datasource.source) or config.get_yaml_url(datasource.source)
|
||||
last_run_at = datasource.last_run_at
|
||||
last_status = datasource.last_status
|
||||
|
||||
collector_list.append(
|
||||
{
|
||||
"id": datasource.id,
|
||||
"source": datasource.source,
|
||||
"name": datasource.name,
|
||||
**datasource_metadata(datasource.source),
|
||||
"module": datasource.module,
|
||||
"priority": datasource.priority,
|
||||
"frequency": format_frequency_label(datasource.frequency_minutes),
|
||||
@@ -145,11 +391,9 @@ async def list_datasources(
|
||||
"is_active": datasource.is_active,
|
||||
"collector_class": datasource.collector_class,
|
||||
"endpoint": endpoint,
|
||||
"last_run": last_run,
|
||||
"last_run": to_iso8601_utc(last_run_at),
|
||||
"last_run_at": to_iso8601_utc(last_run_at),
|
||||
"last_status": last_status,
|
||||
"last_records_processed": last_task.records_processed if last_task else None,
|
||||
"data_count": data_count,
|
||||
"is_running": running_task is not None,
|
||||
"task_id": running_task.id if running_task else None,
|
||||
"progress": running_task.progress if running_task else None,
|
||||
@@ -189,9 +433,13 @@ async def trigger_all_datasources(
|
||||
skipped_sources: list[dict] = []
|
||||
failed_sources: list[dict] = []
|
||||
now = datetime.now(timezone.utc)
|
||||
running_tasks = await _load_latest_running_tasks(
|
||||
db,
|
||||
[datasource.id for datasource in datasources],
|
||||
)
|
||||
|
||||
for datasource in datasources:
|
||||
running_task = await get_running_task(db, datasource.id)
|
||||
running_task = running_tasks.get(datasource.id)
|
||||
if running_task is not None:
|
||||
skipped_sources.append(
|
||||
{
|
||||
@@ -219,7 +467,7 @@ async def trigger_all_datasources(
|
||||
)
|
||||
continue
|
||||
|
||||
previous_task_ids[datasource.id] = await get_latest_task_id_for_datasource(datasource.id)
|
||||
previous_task_ids[datasource.id] = None
|
||||
success = run_collector_now(datasource.source)
|
||||
if not success:
|
||||
failed_sources.append(
|
||||
@@ -241,13 +489,24 @@ async def trigger_all_datasources(
|
||||
}
|
||||
)
|
||||
|
||||
latest_task_ids = await _load_latest_task_ids(
|
||||
db,
|
||||
[datasource.id for datasource in datasources],
|
||||
)
|
||||
for datasource_id in previous_task_ids:
|
||||
previous_task_ids[datasource_id] = latest_task_ids.get(datasource_id)
|
||||
|
||||
for _ in range(20):
|
||||
await asyncio.sleep(0.1)
|
||||
pending = [item for item in triggered_sources if item["task_id"] is None]
|
||||
if not pending:
|
||||
break
|
||||
latest_task_ids = await _load_latest_task_ids(
|
||||
db,
|
||||
[item["id"] for item in pending],
|
||||
)
|
||||
for item in pending:
|
||||
task_id = await get_latest_task_id_for_datasource(item["id"])
|
||||
task_id = latest_task_ids.get(item["id"])
|
||||
if task_id is not None and task_id != previous_task_ids.get(item["id"]):
|
||||
item["task_id"] = task_id
|
||||
|
||||
@@ -277,6 +536,7 @@ async def get_datasource(
|
||||
return {
|
||||
"id": datasource.id,
|
||||
"name": datasource.name,
|
||||
**datasource_metadata(datasource.source),
|
||||
"module": datasource.module,
|
||||
"priority": datasource.priority,
|
||||
"frequency": format_frequency_label(datasource.frequency_minutes),
|
||||
@@ -346,6 +606,7 @@ async def get_datasource_stats(
|
||||
@router.post("/{source_id}/trigger")
|
||||
async def trigger_datasource(
|
||||
source_id: str,
|
||||
force: bool = Query(False),
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
@@ -356,6 +617,26 @@ async def trigger_datasource(
|
||||
if not datasource.is_active:
|
||||
raise HTTPException(status_code=400, detail="Data source is disabled")
|
||||
|
||||
running_task = await get_running_task(db, datasource.id)
|
||||
if running_task is not None and not force:
|
||||
raise HTTPException(
|
||||
status_code=409,
|
||||
detail={
|
||||
"reason": "running_task_in_progress",
|
||||
"message": "当前采集任务尚未完成,重新触发会丢失本次未完成进度。是否强制重新采集?",
|
||||
"task_id": running_task.id,
|
||||
"phase": running_task.phase,
|
||||
"progress": running_task.progress,
|
||||
"records_processed": running_task.records_processed,
|
||||
"total_records": running_task.total_records,
|
||||
},
|
||||
)
|
||||
|
||||
if running_task is not None and force:
|
||||
cancelled = await cancel_running_collector_now(datasource.source)
|
||||
if not cancelled:
|
||||
await rollback_orphaned_running_task(db, datasource, running_task)
|
||||
|
||||
previous_task_id = await get_latest_task_id_for_datasource(datasource.id)
|
||||
success = run_collector_now(datasource.source)
|
||||
if not success:
|
||||
@@ -375,6 +656,7 @@ async def trigger_datasource(
|
||||
"source_id": datasource.id,
|
||||
"task_id": task_id,
|
||||
"collector_name": datasource.source,
|
||||
"force": force,
|
||||
"message": f"Collector '{datasource.source}' has been triggered",
|
||||
}
|
||||
|
||||
@@ -412,6 +694,7 @@ async def clear_datasource_data(
|
||||
async def get_task_status(
|
||||
source_id: str,
|
||||
task_id: Optional[int] = None,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
datasource = await get_datasource_record(db, source_id)
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
from fastapi import APIRouter, Query
|
||||
|
||||
from app.services.earth_news import get_earth_news_payload
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
@router.get("/earth-feed")
|
||||
async def get_earth_feed(
|
||||
lat: float | None = Query(None, description="Current Earth view center latitude"),
|
||||
lon: float | None = Query(None, description="Current Earth view center longitude"),
|
||||
):
|
||||
return await get_earth_news_payload(lat=lat, lon=lon)
|
||||
@@ -1,3 +1,4 @@
|
||||
from copy import deepcopy
|
||||
from datetime import UTC, datetime
|
||||
from typing import Optional
|
||||
|
||||
@@ -8,11 +9,38 @@ from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.security import get_current_user
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.core.config import settings as app_settings
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
from app.core.datasource_defaults import DEFAULT_DATASOURCES
|
||||
from app.db.session import get_db
|
||||
from app.models.datasource import DataSource
|
||||
from app.models.datasource_config import DataSourceConfig
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.models.user import User
|
||||
from app.services.barentswatch import (
|
||||
BarentsWatchConfig,
|
||||
check_barentswatch_config,
|
||||
check_barentswatch_connectivity,
|
||||
get_barentswatch_datasource_record,
|
||||
resolve_barentswatch_config,
|
||||
)
|
||||
from app.services.credential_guides import (
|
||||
generate_credential_guide,
|
||||
get_credential_guide,
|
||||
reset_credential_guide,
|
||||
)
|
||||
from app.services.datasource_connectivity import (
|
||||
build_builtin_connectivity_checksum,
|
||||
save_connectivity_success,
|
||||
)
|
||||
from app.services.ai_client import AIProviderClient, get_ai_provider_client
|
||||
from app.services.llm_provider_catalog import (
|
||||
get_fallback_llm_provider_preset,
|
||||
list_fallback_llm_provider_presets,
|
||||
refresh_llm_provider_preset,
|
||||
)
|
||||
from app.services.scheduler import sync_datasource_job
|
||||
from app.services.tv_streams import DEFAULT_TV_SETTINGS, get_tv_settings_payload, normalize_tv_settings
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
@@ -36,6 +64,22 @@ DEFAULT_SETTINGS = {
|
||||
"max_login_attempts": 5,
|
||||
"password_policy": "medium",
|
||||
},
|
||||
"tv": DEFAULT_TV_SETTINGS,
|
||||
"external_integrations": {
|
||||
"ai_provider": {
|
||||
"service_url": "",
|
||||
"service_token": "",
|
||||
"provider": "minimax",
|
||||
"provider_api": "anthropic-messages",
|
||||
"base_url": "https://api.minimaxi.com/anthropic",
|
||||
"model": "MiniMax-M2.7",
|
||||
"api_key": "",
|
||||
"max_tokens": 1200,
|
||||
"anthropic_version": "2023-06-01",
|
||||
"timeout_seconds": 60,
|
||||
"retry_attempts": 2,
|
||||
}
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@@ -67,8 +111,62 @@ class CollectorSettingsUpdate(BaseModel):
|
||||
frequency_minutes: int = Field(default=60, ge=1, le=10080)
|
||||
|
||||
|
||||
class TVStreamSourceUpdate(BaseModel):
|
||||
id: str = Field(min_length=1, max_length=100)
|
||||
name: str = Field(min_length=1, max_length=200)
|
||||
provider: str = Field(default="Unknown", max_length=100)
|
||||
region: str = Field(default="Global", max_length=100)
|
||||
language: str = Field(default="und", max_length=32)
|
||||
source_type: str = Field(default="iframe", pattern="^(iframe|hls|video|external|youtube)$")
|
||||
embed_url: str = ""
|
||||
stream_url: str = ""
|
||||
homepage_url: str = ""
|
||||
poster_url: str = ""
|
||||
youtube_video_id: str = ""
|
||||
youtube_channel: str = ""
|
||||
is_enabled: bool = True
|
||||
is_fallback: bool = False
|
||||
sort_order: int = Field(default=10, ge=0, le=9999)
|
||||
collector_source: Optional[str] = None
|
||||
notes: str = ""
|
||||
|
||||
|
||||
class TVSettingsUpdate(BaseModel):
|
||||
default_source_id: str = Field(default=DEFAULT_TV_SETTINGS["default_source_id"], min_length=1)
|
||||
auto_fallback: bool = True
|
||||
sources: list[TVStreamSourceUpdate] = Field(default_factory=list)
|
||||
|
||||
|
||||
class AIProviderIntegrationUpdate(BaseModel):
|
||||
service_url: str = ""
|
||||
service_token: Optional[str] = None
|
||||
provider: str = Field(default="minimax", max_length=80)
|
||||
provider_api: str = Field(default="anthropic-messages", max_length=80)
|
||||
base_url: str = Field(default="", max_length=500)
|
||||
model: str = Field(default="", max_length=200)
|
||||
api_key: Optional[str] = None
|
||||
max_tokens: int = Field(default=1200, ge=1, le=200000)
|
||||
anthropic_version: str = Field(default="2023-06-01", max_length=40)
|
||||
timeout_seconds: int = Field(default=60, ge=5, le=600)
|
||||
retry_attempts: int = Field(default=2, ge=1, le=10)
|
||||
clear_service_token: bool = False
|
||||
clear_api_key: bool = False
|
||||
|
||||
|
||||
class BarentsWatchIntegrationUpdate(BaseModel):
|
||||
endpoint: str = ""
|
||||
client_id: str = ""
|
||||
client_secret: Optional[str] = None
|
||||
clear_client_secret: bool = False
|
||||
|
||||
|
||||
class ExternalIntegrationsUpdate(BaseModel):
|
||||
ai_provider: AIProviderIntegrationUpdate
|
||||
barentswatch: BarentsWatchIntegrationUpdate
|
||||
|
||||
|
||||
def merge_with_defaults(category: str, payload: Optional[dict]) -> dict:
|
||||
merged = DEFAULT_SETTINGS[category].copy()
|
||||
merged = deepcopy(DEFAULT_SETTINGS[category])
|
||||
if payload:
|
||||
merged.update(payload)
|
||||
return merged
|
||||
@@ -79,6 +177,26 @@ async def get_setting_record(db: AsyncSession, category: str) -> Optional[System
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
|
||||
async def get_setting_payloads(db: AsyncSession, categories: list[str]) -> dict[str, dict]:
|
||||
if not categories:
|
||||
return {}
|
||||
|
||||
result = await db.execute(
|
||||
select(SystemSetting).where(SystemSetting.category.in_(categories))
|
||||
)
|
||||
records_by_category = {
|
||||
record.category: record
|
||||
for record in result.scalars().all()
|
||||
}
|
||||
return {
|
||||
category: merge_with_defaults(
|
||||
category,
|
||||
records_by_category.get(category).payload if records_by_category.get(category) else None,
|
||||
)
|
||||
for category in categories
|
||||
}
|
||||
|
||||
|
||||
async def get_setting_payload(db: AsyncSession, category: str) -> dict:
|
||||
record = await get_setting_record(db, category)
|
||||
return merge_with_defaults(category, record.payload if record else None)
|
||||
@@ -97,6 +215,151 @@ async def save_setting_payload(db: AsyncSession, category: str, payload: dict) -
|
||||
return merge_with_defaults(category, record.payload)
|
||||
|
||||
|
||||
def _mask_secret(value: Optional[str]) -> dict:
|
||||
if not value:
|
||||
return {"configured": False, "preview": ""}
|
||||
text = str(value)
|
||||
if "-" in text:
|
||||
prefix = text.split("-", 1)[0] + "-"
|
||||
preview = prefix + ("*" * max(len(text) - len(prefix), 1))
|
||||
else:
|
||||
prefix_len = min(4, len(text))
|
||||
preview = text[:prefix_len] + ("*" * max(len(text) - prefix_len, 1))
|
||||
return {"configured": True, "preview": preview}
|
||||
|
||||
|
||||
async def get_runtime_ai_provider_config(db: AsyncSession) -> dict:
|
||||
runtime_record = await get_setting_record(db, "external_integrations")
|
||||
payload = merge_with_defaults(
|
||||
"external_integrations",
|
||||
runtime_record.payload if runtime_record else None,
|
||||
)
|
||||
ai_payload = payload.get("ai_provider") or {}
|
||||
has_runtime_llm_config = bool(
|
||||
runtime_record
|
||||
and isinstance(runtime_record.payload, dict)
|
||||
and isinstance(runtime_record.payload.get("ai_provider"), dict)
|
||||
)
|
||||
return {
|
||||
"service_url": ai_payload.get("service_url") or app_settings.AI_PROVIDER_SERVICE_URL,
|
||||
"service_token": ai_payload.get("service_token") or app_settings.AI_PROVIDER_SERVICE_TOKEN,
|
||||
"timeout_seconds": int(
|
||||
ai_payload.get("timeout_seconds") or app_settings.AI_PROVIDER_TIMEOUT_SECONDS
|
||||
),
|
||||
"retry_attempts": int(
|
||||
ai_payload.get("retry_attempts") or app_settings.AI_PROVIDER_RETRY_ATTEMPTS
|
||||
),
|
||||
"llm_config": {
|
||||
"provider": ai_payload.get("provider") or "minimax",
|
||||
"provider_api": ai_payload.get("provider_api") or "anthropic-messages",
|
||||
"base_url": ai_payload.get("base_url") or "https://api.minimaxi.com/anthropic",
|
||||
"model": ai_payload.get("model") or "MiniMax-M2.7",
|
||||
"api_key": ai_payload.get("api_key") or "",
|
||||
"max_tokens": int(ai_payload.get("max_tokens") or 1200),
|
||||
"anthropic_version": ai_payload.get("anthropic_version") or "2023-06-01",
|
||||
} if has_runtime_llm_config else {},
|
||||
}
|
||||
|
||||
|
||||
async def get_barentswatch_config_record(db: AsyncSession) -> Optional[DataSourceConfig]:
|
||||
return await get_barentswatch_datasource_record(db)
|
||||
|
||||
|
||||
async def serialize_external_integrations(db: AsyncSession) -> dict:
|
||||
ai_config = await get_runtime_ai_provider_config(db)
|
||||
runtime_setting = await get_setting_record(db, "external_integrations")
|
||||
display_llm_config = ai_config["llm_config"] or DEFAULT_SETTINGS["external_integrations"]["ai_provider"]
|
||||
barentswatch_record = await get_barentswatch_config_record(db)
|
||||
barentswatch_auth = barentswatch_record.auth_config if barentswatch_record else {}
|
||||
barentswatch_auth = barentswatch_auth or {}
|
||||
resolved_barentswatch = await resolve_barentswatch_config(db)
|
||||
return {
|
||||
"ai_provider": {
|
||||
"service_url": ai_config["service_url"],
|
||||
"service_token": _mask_secret(ai_config["service_token"]),
|
||||
"provider": display_llm_config.get("provider") or "minimax",
|
||||
"provider_api": display_llm_config.get("provider_api") or "anthropic-messages",
|
||||
"base_url": display_llm_config.get("base_url") or "https://api.minimaxi.com/anthropic",
|
||||
"model": display_llm_config.get("model") or "MiniMax-M2.7",
|
||||
"api_key": _mask_secret(display_llm_config.get("api_key")),
|
||||
"max_tokens": int(display_llm_config.get("max_tokens") or 1200),
|
||||
"anthropic_version": display_llm_config.get("anthropic_version") or "2023-06-01",
|
||||
"timeout_seconds": ai_config["timeout_seconds"],
|
||||
"retry_attempts": ai_config["retry_attempts"],
|
||||
"source": "runtime" if runtime_setting else "env",
|
||||
},
|
||||
"barentswatch": {
|
||||
"endpoint": resolved_barentswatch.endpoint,
|
||||
"client_id": barentswatch_auth.get("client_id") or resolved_barentswatch.client_id,
|
||||
"client_secret": _mask_secret(
|
||||
barentswatch_auth.get("client_secret") or resolved_barentswatch.client_secret
|
||||
),
|
||||
"source": resolved_barentswatch.credential_source,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
async def save_external_integrations_payload(
|
||||
db: AsyncSession,
|
||||
update: ExternalIntegrationsUpdate,
|
||||
) -> dict:
|
||||
current_payload = await get_setting_payload(db, "external_integrations")
|
||||
current_ai = current_payload.get("ai_provider") or {}
|
||||
ai_payload = {
|
||||
"service_url": update.ai_provider.service_url.strip()
|
||||
or app_settings.AI_PROVIDER_SERVICE_URL,
|
||||
"service_token": current_ai.get("service_token") or "",
|
||||
"provider": update.ai_provider.provider.strip() or "minimax",
|
||||
"provider_api": update.ai_provider.provider_api.strip() or "anthropic-messages",
|
||||
"base_url": update.ai_provider.base_url.strip(),
|
||||
"model": update.ai_provider.model.strip(),
|
||||
"api_key": current_ai.get("api_key") or "",
|
||||
"max_tokens": update.ai_provider.max_tokens,
|
||||
"anthropic_version": update.ai_provider.anthropic_version.strip() or "2023-06-01",
|
||||
"timeout_seconds": update.ai_provider.timeout_seconds,
|
||||
"retry_attempts": update.ai_provider.retry_attempts,
|
||||
}
|
||||
if update.ai_provider.clear_service_token:
|
||||
ai_payload["service_token"] = ""
|
||||
elif update.ai_provider.service_token not in (None, ""):
|
||||
ai_payload["service_token"] = update.ai_provider.service_token
|
||||
if update.ai_provider.clear_api_key:
|
||||
ai_payload["api_key"] = ""
|
||||
elif update.ai_provider.api_key not in (None, ""):
|
||||
ai_payload["api_key"] = update.ai_provider.api_key
|
||||
|
||||
await save_setting_payload(db, "external_integrations", {"ai_provider": ai_payload})
|
||||
|
||||
default_endpoint = get_data_sources_config().get_yaml_url("barentswatch_vessels")
|
||||
barentswatch_record = await get_barentswatch_config_record(db)
|
||||
if barentswatch_record is None:
|
||||
barentswatch_record = DataSourceConfig(
|
||||
name="barentswatch_vessels",
|
||||
description="BarentsWatch Live AIS credentials",
|
||||
source_type="api",
|
||||
endpoint=update.barentswatch.endpoint.strip() or default_endpoint,
|
||||
auth_type="oauth_client",
|
||||
auth_config={},
|
||||
headers={},
|
||||
config={},
|
||||
is_active=True,
|
||||
)
|
||||
db.add(barentswatch_record)
|
||||
|
||||
current_auth = dict(barentswatch_record.auth_config or {})
|
||||
if update.barentswatch.clear_client_secret:
|
||||
current_auth.pop("client_secret", None)
|
||||
elif update.barentswatch.client_secret not in (None, ""):
|
||||
current_auth["client_secret"] = update.barentswatch.client_secret
|
||||
current_auth["client_id"] = update.barentswatch.client_id.strip()
|
||||
barentswatch_record.endpoint = update.barentswatch.endpoint.strip() or default_endpoint
|
||||
barentswatch_record.auth_type = "oauth_client"
|
||||
barentswatch_record.auth_config = current_auth
|
||||
await db.commit()
|
||||
|
||||
return await serialize_external_integrations(db)
|
||||
|
||||
|
||||
def format_frequency_label(minutes: int) -> str:
|
||||
if minutes % 1440 == 0:
|
||||
return f"{minutes // 1440}d"
|
||||
@@ -106,9 +369,11 @@ def format_frequency_label(minutes: int) -> str:
|
||||
|
||||
|
||||
def serialize_collector(datasource: DataSource) -> dict:
|
||||
defaults = DEFAULT_DATASOURCES.get(datasource.source, {})
|
||||
return {
|
||||
"id": datasource.id,
|
||||
"name": datasource.name,
|
||||
"display_name": defaults.get("display_name") or datasource.name,
|
||||
"source": datasource.source,
|
||||
"module": datasource.module,
|
||||
"priority": datasource.priority,
|
||||
@@ -118,6 +383,10 @@ def serialize_collector(datasource: DataSource) -> dict:
|
||||
"last_run_at": to_iso8601_utc(datasource.last_run_at),
|
||||
"last_status": datasource.last_status,
|
||||
"next_run_at": to_iso8601_utc(datasource.next_run_at),
|
||||
"is_free": bool(defaults.get("is_free", True)),
|
||||
"requires_credentials": bool(defaults.get("requires_credentials", False)),
|
||||
"credential_provider": defaults.get("credential_provider"),
|
||||
"credential_status": defaults.get("credential_status", "none"),
|
||||
}
|
||||
|
||||
|
||||
@@ -175,6 +444,154 @@ async def update_security_settings(
|
||||
return {"status": "updated", "security": payload}
|
||||
|
||||
|
||||
@router.get("/tv")
|
||||
async def get_tv_settings(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return {"tv": await get_tv_settings_payload(db)}
|
||||
|
||||
|
||||
@router.put("/tv")
|
||||
async def update_tv_settings(
|
||||
settings: TVSettingsUpdate,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
payload = normalize_tv_settings(settings.model_dump())
|
||||
saved = await save_setting_payload(db, "tv", payload)
|
||||
return {"status": "updated", "tv": normalize_tv_settings(saved)}
|
||||
|
||||
|
||||
@router.get("/integrations")
|
||||
async def get_external_integrations(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return {"integrations": await serialize_external_integrations(db)}
|
||||
|
||||
|
||||
@router.get("/integrations/barentswatch/connectivity")
|
||||
async def get_barentswatch_connectivity(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await check_barentswatch_connectivity(db)
|
||||
|
||||
|
||||
@router.post("/integrations/barentswatch/connect")
|
||||
async def connect_barentswatch_integration(
|
||||
payload: BarentsWatchIntegrationUpdate,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
current = await resolve_barentswatch_config(db)
|
||||
config = BarentsWatchConfig(
|
||||
endpoint=payload.endpoint.strip() or current.endpoint,
|
||||
client_id=payload.client_id.strip() or current.client_id,
|
||||
client_secret=(
|
||||
""
|
||||
if payload.clear_client_secret
|
||||
else payload.client_secret or current.client_secret
|
||||
),
|
||||
credential_source="draft",
|
||||
endpoint_source="draft",
|
||||
)
|
||||
result = await check_barentswatch_config(config)
|
||||
if result.get("success"):
|
||||
checksum, _context = await build_builtin_connectivity_checksum(
|
||||
"barentswatch_vessels",
|
||||
config.endpoint,
|
||||
"none",
|
||||
{},
|
||||
{},
|
||||
db,
|
||||
credential_override={
|
||||
"client_id": config.client_id,
|
||||
"client_secret": config.client_secret,
|
||||
},
|
||||
)
|
||||
validation = await save_connectivity_success(
|
||||
db,
|
||||
"barentswatch_vessels",
|
||||
checksum,
|
||||
result,
|
||||
connected_by="connection_button",
|
||||
)
|
||||
await db.commit()
|
||||
return {**result, "connected": True, "validation": validation}
|
||||
return {**result, "connected": False}
|
||||
|
||||
|
||||
@router.get("/credential-guides/{provider}")
|
||||
async def read_credential_guide(
|
||||
provider: str,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
try:
|
||||
return {"guide": await get_credential_guide(db, provider)}
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=404, detail=str(exc)) from exc
|
||||
|
||||
|
||||
@router.post("/credential-guides/{provider}/generate")
|
||||
async def generate_provider_credential_guide(
|
||||
provider: str,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
ai_client: AIProviderClient = Depends(get_ai_provider_client),
|
||||
):
|
||||
try:
|
||||
return {"guide": await generate_credential_guide(db, provider, ai_client)}
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=404, detail=str(exc)) from exc
|
||||
|
||||
|
||||
@router.post("/credential-guides/{provider}/reset")
|
||||
async def reset_provider_credential_guide(
|
||||
provider: str,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
try:
|
||||
return {"guide": await reset_credential_guide(db, provider)}
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=404, detail=str(exc)) from exc
|
||||
|
||||
|
||||
@router.get("/integrations/ai-provider/presets")
|
||||
async def get_ai_provider_presets(
|
||||
current_user: User = Depends(get_current_user),
|
||||
):
|
||||
return {"data": list_fallback_llm_provider_presets()}
|
||||
|
||||
|
||||
@router.post("/integrations/ai-provider/presets/{provider}/refresh")
|
||||
async def refresh_ai_provider_preset(
|
||||
provider: str,
|
||||
current_user: User = Depends(get_current_user),
|
||||
):
|
||||
try:
|
||||
return {"data": await refresh_llm_provider_preset(provider)}
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=404, detail=str(exc)) from exc
|
||||
except Exception as exc:
|
||||
fallback = get_fallback_llm_provider_preset(provider)
|
||||
fallback["refresh_error"] = str(exc)
|
||||
return {"data": fallback}
|
||||
|
||||
|
||||
@router.put("/integrations")
|
||||
async def update_external_integrations(
|
||||
payload: ExternalIntegrationsUpdate,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
saved = await save_external_integrations_payload(db, payload)
|
||||
return {"status": "updated", "integrations": saved}
|
||||
|
||||
|
||||
@router.get("/collectors")
|
||||
async def get_collector_settings(
|
||||
current_user: User = Depends(get_current_user),
|
||||
@@ -212,10 +629,16 @@ async def get_all_settings(
|
||||
):
|
||||
result = await db.execute(select(DataSource).order_by(DataSource.module, DataSource.id))
|
||||
datasources = result.scalars().all()
|
||||
setting_payloads = await get_setting_payloads(
|
||||
db,
|
||||
["system", "notifications", "security"],
|
||||
)
|
||||
return {
|
||||
"system": await get_setting_payload(db, "system"),
|
||||
"notifications": await get_setting_payload(db, "notifications"),
|
||||
"security": await get_setting_payload(db, "security"),
|
||||
"system": setting_payloads["system"],
|
||||
"notifications": setting_payloads["notifications"],
|
||||
"security": setting_payloads["security"],
|
||||
"tv": await get_tv_settings_payload(db),
|
||||
"integrations": await serialize_external_integrations(db),
|
||||
"collectors": [serialize_collector(datasource) for datasource in datasources],
|
||||
"generated_at": to_iso8601_utc(datetime.now(UTC)),
|
||||
}
|
||||
@@ -4,12 +4,15 @@ import os
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, status
|
||||
from datetime import datetime
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Request, status
|
||||
from pydantic import BaseModel
|
||||
|
||||
from app.core.config import ROOT_DIR
|
||||
from app.core.security import get_current_user
|
||||
from app.models.user import User
|
||||
from app.services.persistent_logs import record_audit_log, record_system_log
|
||||
from app.services.system_control import (
|
||||
build_task_id,
|
||||
clear_active_task_id,
|
||||
@@ -23,6 +26,15 @@ from app.services.system_control import (
|
||||
set_active_task_id,
|
||||
upsert_task_state,
|
||||
)
|
||||
from app.services.system_logs import (
|
||||
DEFAULT_LOG_LINE_LIMIT,
|
||||
MAX_LOG_LINE_LIMIT,
|
||||
SUPPORTED_LOG_LEVELS,
|
||||
append_buffer_log,
|
||||
list_log_sources,
|
||||
normalize_log_level,
|
||||
read_log_snapshot,
|
||||
)
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
@@ -47,6 +59,59 @@ class RestartTaskLogsResponse(BaseModel):
|
||||
lines: list[str]
|
||||
|
||||
|
||||
class SystemLogSourceSummary(BaseModel):
|
||||
source_id: str
|
||||
name: str
|
||||
kind: str
|
||||
location: str
|
||||
description: str
|
||||
category: str
|
||||
status: str
|
||||
|
||||
|
||||
class SystemLogSourcesResponse(BaseModel):
|
||||
items: list[SystemLogSourceSummary]
|
||||
|
||||
|
||||
class SystemLogDailyMarker(BaseModel):
|
||||
date_token: str
|
||||
total: int
|
||||
dominant_level: str
|
||||
|
||||
|
||||
class SystemLogSnapshotResponse(BaseModel):
|
||||
source_id: str
|
||||
name: str
|
||||
kind: str
|
||||
location: str
|
||||
description: str
|
||||
category: str
|
||||
status: str
|
||||
level: str
|
||||
selected_levels: list[str] = []
|
||||
search_query: str = ""
|
||||
available_levels: list[str]
|
||||
daily_markers: list[SystemLogDailyMarker] = []
|
||||
line_limit: int
|
||||
line_count: int
|
||||
lines: list[str]
|
||||
|
||||
|
||||
class EarthClientLogEventCreate(BaseModel):
|
||||
level: str = "error"
|
||||
message: str
|
||||
category: str | None = None
|
||||
url: str | None = None
|
||||
module: str | None = None
|
||||
detail: str | None = None
|
||||
|
||||
|
||||
class EarthClientLogEventResponse(BaseModel):
|
||||
accepted: bool
|
||||
source_id: str
|
||||
level: str
|
||||
|
||||
|
||||
def ensure_super_admin(current_user: User) -> None:
|
||||
if not require_super_admin(current_user.role):
|
||||
raise HTTPException(
|
||||
@@ -55,9 +120,22 @@ def ensure_super_admin(current_user: User) -> None:
|
||||
)
|
||||
|
||||
|
||||
def validate_log_date(raw_value: str | None, field_name: str) -> str | None:
|
||||
if raw_value in {None, ""}:
|
||||
return None
|
||||
try:
|
||||
return datetime.strptime(raw_value, "%Y-%m-%d").date().isoformat()
|
||||
except ValueError as exc:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
detail=f"{field_name} must be in YYYY-MM-DD format",
|
||||
) from exc
|
||||
|
||||
|
||||
@router.post("/restart-tasks", response_model=RestartTaskResponse)
|
||||
async def create_restart_task(
|
||||
payload: RestartTaskCreate,
|
||||
request: Request,
|
||||
current_user: User = Depends(get_current_user),
|
||||
):
|
||||
ensure_super_admin(current_user)
|
||||
@@ -133,11 +211,31 @@ async def create_restart_task(
|
||||
requested_by=requested_by,
|
||||
)
|
||||
clear_active_task_id(task_id)
|
||||
await record_audit_log(
|
||||
action="system.restart_task.requested",
|
||||
actor_id=current_user.id,
|
||||
actor_name=current_user.username,
|
||||
target_type="restart_task",
|
||||
target_id=task_id,
|
||||
result="failed",
|
||||
ip=request.client.host if request.client else None,
|
||||
details={"action": payload.action, "message": task_state["message"]},
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
|
||||
detail=task_state["message"],
|
||||
) from exc
|
||||
|
||||
await record_audit_log(
|
||||
action="system.restart_task.requested",
|
||||
actor_id=current_user.id,
|
||||
actor_name=current_user.username,
|
||||
target_type="restart_task",
|
||||
target_id=task_id,
|
||||
result="accepted",
|
||||
ip=request.client.host if request.client else None,
|
||||
details={"action": payload.action},
|
||||
)
|
||||
return task_state
|
||||
|
||||
|
||||
@@ -165,3 +263,92 @@ async def get_restart_task_logs(
|
||||
if task is None:
|
||||
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Restart task not found")
|
||||
return {"task_id": task_id, "lines": get_task_logs(task_id)}
|
||||
|
||||
|
||||
@router.get("/logs/sources", response_model=SystemLogSourcesResponse)
|
||||
async def get_system_log_sources(
|
||||
current_user: User = Depends(get_current_user),
|
||||
):
|
||||
ensure_super_admin(current_user)
|
||||
return {"items": list_log_sources()}
|
||||
|
||||
|
||||
@router.get("/logs/{source_id}", response_model=SystemLogSnapshotResponse)
|
||||
async def get_system_log_snapshot(
|
||||
source_id: str,
|
||||
limit: int = DEFAULT_LOG_LINE_LIMIT,
|
||||
level: str = "all",
|
||||
levels: str | None = Query(None, description="Comma-separated log levels"),
|
||||
start_date: str | None = Query(None, description="Filter logs from this date (YYYY-MM-DD)"),
|
||||
end_date: str | None = Query(None, description="Filter logs until this date (YYYY-MM-DD)"),
|
||||
search: str | None = Query(None, description="Case-insensitive substring search"),
|
||||
current_user: User = Depends(get_current_user),
|
||||
):
|
||||
ensure_super_admin(current_user)
|
||||
|
||||
if limit < 1 or limit > MAX_LOG_LINE_LIMIT:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
detail=f"limit must be between 1 and {MAX_LOG_LINE_LIMIT}",
|
||||
)
|
||||
if str(level).strip().lower() not in SUPPORTED_LOG_LEVELS and normalize_log_level(level) == "all" and str(level).strip().lower() not in {"", "all"}:
|
||||
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="Unsupported log level")
|
||||
if levels:
|
||||
for raw_level in str(levels).split(","):
|
||||
normalized_level = str(raw_level).strip().lower()
|
||||
if not normalized_level:
|
||||
continue
|
||||
if normalized_level not in SUPPORTED_LOG_LEVELS and normalize_log_level(normalized_level) == "all":
|
||||
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="Unsupported log level")
|
||||
normalized_start_date = validate_log_date(start_date, "start_date")
|
||||
normalized_end_date = validate_log_date(end_date, "end_date")
|
||||
if normalized_start_date and normalized_end_date and normalized_start_date > normalized_end_date:
|
||||
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="start_date must be earlier than or equal to end_date")
|
||||
|
||||
snapshot = read_log_snapshot(
|
||||
source_id,
|
||||
limit,
|
||||
level=level,
|
||||
levels=levels,
|
||||
start_date=normalized_start_date,
|
||||
end_date=normalized_end_date,
|
||||
search=search,
|
||||
)
|
||||
if snapshot is None:
|
||||
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Log source not found")
|
||||
return snapshot
|
||||
|
||||
|
||||
@router.post("/logs/earth-client", response_model=EarthClientLogEventResponse)
|
||||
async def ingest_earth_client_log(
|
||||
payload: EarthClientLogEventCreate,
|
||||
request: Request,
|
||||
):
|
||||
normalized_level = normalize_log_level(payload.level)
|
||||
append_buffer_log(
|
||||
"earth-client",
|
||||
level=normalized_level,
|
||||
message=payload.message,
|
||||
context={
|
||||
"category": payload.category or "",
|
||||
"url": payload.url or "",
|
||||
"module": payload.module or "",
|
||||
"detail": payload.detail or "",
|
||||
},
|
||||
)
|
||||
await record_system_log(
|
||||
source="earth-client",
|
||||
service="earth",
|
||||
module=payload.module or "earth-client",
|
||||
event="earth.client.runtime_log",
|
||||
level=normalized_level,
|
||||
message=payload.message,
|
||||
category=payload.category or "client-runtime",
|
||||
context={
|
||||
"url": payload.url or "",
|
||||
"detail": payload.detail or "",
|
||||
"module": payload.module or "",
|
||||
"client_ip": request.client.host if request.client else "",
|
||||
},
|
||||
)
|
||||
return {"accepted": True, "source_id": "earth-client", "level": normalized_level}
|
||||
@@ -0,0 +1,70 @@
|
||||
from urllib.parse import quote, urljoin
|
||||
|
||||
import httpx
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query
|
||||
from fastapi.responses import Response
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.db.session import get_db
|
||||
from app.services.tv_streams import get_public_tv_payload, is_allowed_tv_proxy_url
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
@router.get("/streams")
|
||||
async def list_public_tv_streams(
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await get_public_tv_payload(db)
|
||||
|
||||
|
||||
@router.get("/proxy")
|
||||
async def proxy_tv_stream(
|
||||
url: str = Query(..., description="Upstream TV stream or manifest URL"),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
payload = await get_public_tv_payload(db)
|
||||
if not is_allowed_tv_proxy_url(url, payload.get("sources", [])):
|
||||
raise HTTPException(status_code=403, detail="TV proxy target is not allowed")
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(follow_redirects=True, timeout=20.0) as client:
|
||||
upstream = await client.get(
|
||||
url,
|
||||
headers={
|
||||
"User-Agent": "Mozilla/5.0",
|
||||
"Referer": "https://tv.cctv.com/live/cctv4/",
|
||||
},
|
||||
)
|
||||
upstream.raise_for_status()
|
||||
except httpx.HTTPError as exc:
|
||||
raise HTTPException(status_code=502, detail=f"Failed to fetch TV stream: {exc}") from exc
|
||||
|
||||
content_type = upstream.headers.get("content-type", "application/octet-stream")
|
||||
raw_content = upstream.content
|
||||
response_url = str(upstream.url)
|
||||
is_manifest = (
|
||||
response_url.endswith(".m3u8")
|
||||
or "mpegurl" in content_type.lower()
|
||||
or raw_content.lstrip().startswith(b"#EXTM3U")
|
||||
)
|
||||
|
||||
headers = {"Cache-Control": "no-store"}
|
||||
|
||||
if is_manifest:
|
||||
manifest_text = upstream.text
|
||||
rewritten_lines: list[str] = []
|
||||
for line in manifest_text.splitlines():
|
||||
stripped = line.strip()
|
||||
if not stripped or stripped.startswith("#"):
|
||||
rewritten_lines.append(line)
|
||||
continue
|
||||
absolute_url = urljoin(response_url, stripped)
|
||||
rewritten_lines.append(f"/api/v1/tv/proxy?url={quote(absolute_url, safe='')}")
|
||||
return Response(
|
||||
content="\n".join(rewritten_lines),
|
||||
media_type="application/vnd.apple.mpegurl",
|
||||
headers=headers,
|
||||
)
|
||||
|
||||
return Response(content=raw_content, media_type=content_type, headers=headers)
|
||||
+819
-156
File diff suppressed because it is too large.
Load diff
@@ -2,7 +2,6 @@
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
from datetime import UTC, datetime
|
||||
from typing import Optional
|
||||
|
||||
@@ -10,10 +9,11 @@ from fastapi import APIRouter, WebSocket, WebSocketDisconnect, Query
|
||||
from jose import jwt, JWTError
|
||||
|
||||
from app.core.config import settings
|
||||
from app.core.logging import get_logger
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.core.websocket.manager import manager
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
logger = get_logger(__name__, service="api")
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
@@ -22,11 +22,18 @@ async def authenticate_token(token: str) -> Optional[dict]:
|
||||
try:
|
||||
payload = jwt.decode(token, settings.SECRET_KEY, algorithms=[settings.ALGORITHM])
|
||||
if payload.get("type") != "access":
|
||||
logger.warning(f"WebSocket auth failed: wrong token type")
|
||||
logger.warning_event(
|
||||
"WebSocket auth failed: wrong token type",
|
||||
event="auth.websocket.invalid_token_type",
|
||||
)
|
||||
return None
|
||||
return payload
|
||||
except JWTError as e:
|
||||
logger.warning(f"WebSocket auth failed: {e}")
|
||||
logger.warning_event(
|
||||
"WebSocket auth failed",
|
||||
event="auth.websocket.decode_failed",
|
||||
context={"error": str(e)},
|
||||
)
|
||||
return None
|
||||
|
||||
|
||||
@@ -36,10 +43,17 @@ async def websocket_endpoint(
|
||||
token: str = Query(...),
|
||||
):
|
||||
"""WebSocket endpoint for real-time data"""
|
||||
logger.info(f"WebSocket connection attempt with token: {token[:20]}...")
|
||||
logger.info_event(
|
||||
"WebSocket connection attempt",
|
||||
event="auth.websocket.connection_attempt",
|
||||
context={"token_preview": f"{token[:8]}..."},
|
||||
)
|
||||
payload = await authenticate_token(token)
|
||||
if payload is None:
|
||||
logger.warning("WebSocket authentication failed, closing connection")
|
||||
logger.warning_event(
|
||||
"WebSocket authentication failed, closing connection",
|
||||
event="auth.websocket.connection_rejected",
|
||||
)
|
||||
await websocket.close(code=4001)
|
||||
return
|
||||
|
||||
|
||||
@@ -1,15 +1,15 @@
|
||||
"""Redis caching service"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
from datetime import timedelta
|
||||
from typing import Optional, Any
|
||||
|
||||
import redis
|
||||
|
||||
from app.core.config import settings
|
||||
from app.core.logging import get_logger
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
# Lazy Redis client initialization
|
||||
@@ -47,7 +47,7 @@ class CacheService:
|
||||
return json.loads(value)
|
||||
return None
|
||||
except Exception as e:
|
||||
logger.warning(f"Cache get error: {e}")
|
||||
logger.warning_event("Cache get error", event="cache.get.failed", context={"error": str(e)})
|
||||
return None
|
||||
|
||||
def set(
|
||||
@@ -61,7 +61,7 @@ class CacheService:
|
||||
serialized = json.dumps(value, default=str)
|
||||
return self.client.setex(key, expire_seconds, serialized)
|
||||
except Exception as e:
|
||||
logger.warning(f"Cache set error: {e}")
|
||||
logger.warning_event("Cache set error", event="cache.set.failed", context={"error": str(e)})
|
||||
return False
|
||||
|
||||
def delete(self, key: str) -> bool:
|
||||
@@ -69,7 +69,7 @@ class CacheService:
|
||||
try:
|
||||
return self.client.delete(key) > 0
|
||||
except Exception as e:
|
||||
logger.warning(f"Cache delete error: {e}")
|
||||
logger.warning_event("Cache delete error", event="cache.delete.failed", context={"error": str(e)})
|
||||
return False
|
||||
|
||||
def delete_pattern(self, pattern: str) -> int:
|
||||
@@ -80,7 +80,7 @@ class CacheService:
|
||||
return self.client.delete(*keys)
|
||||
return 0
|
||||
except Exception as e:
|
||||
logger.warning(f"Cache delete_pattern error: {e}")
|
||||
logger.warning_event("Cache delete_pattern error", event="cache.delete_pattern.failed", context={"error": str(e)})
|
||||
return 0
|
||||
|
||||
def get_or_set(
|
||||
|
||||
@@ -11,6 +11,7 @@ COLLECTOR_URL_KEYS = {
|
||||
"fao_landing_points": "fao.landing_point_url",
|
||||
"telegeography_cables": "telegeography.cable_url",
|
||||
"telegeography_landing": "telegeography.landing_point_url",
|
||||
"telegeography_systems": "telegeography.cable_url",
|
||||
"huggingface_models": "huggingface.models_url",
|
||||
"huggingface_datasets": "huggingface.datasets_url",
|
||||
"huggingface_spaces": "huggingface.spaces_url",
|
||||
@@ -23,11 +24,14 @@ COLLECTOR_URL_KEYS = {
|
||||
"top500": "top500.url",
|
||||
"epoch_ai_gpu": "epoch_ai.gpu_clusters_url",
|
||||
"spacetrack_tle": "spacetrack.tle_query_url",
|
||||
"celestrak_tle": "celestrak.base_url",
|
||||
"ris_live_bgp": "ris_live.url",
|
||||
"bgpstream_bgp": "bgpstream.url",
|
||||
"iptoasn_prefix_geo": "iptoasn.combined_url",
|
||||
"opengeofeed_prefix_geo": "opengeofeed.public_csv_url",
|
||||
"nro_delegated_prefix_geo": "nro.delegated_stats_url",
|
||||
"news_live_streams": "news_live_streams.channels_url",
|
||||
"barentswatch_vessels": "barentswatch_vessels.url",
|
||||
}
|
||||
|
||||
|
||||
@@ -41,18 +45,22 @@ class DataSourcesConfig:
|
||||
with open(config_path, "r") as f:
|
||||
self._yaml_config = yaml.safe_load(f) or {}
|
||||
|
||||
def get_yaml_url(self, collector_name: str) -> str:
|
||||
key = COLLECTOR_URL_KEYS.get(collector_name, "")
|
||||
def get_yaml_value(self, key: str):
|
||||
if not key:
|
||||
return ""
|
||||
return None
|
||||
|
||||
parts = key.split(".")
|
||||
value = self._yaml_config
|
||||
for part in parts:
|
||||
if isinstance(value, dict):
|
||||
value = value.get(part, "")
|
||||
value = value.get(part)
|
||||
else:
|
||||
return ""
|
||||
return None
|
||||
return value
|
||||
|
||||
def get_yaml_url(self, collector_name: str) -> str:
|
||||
key = COLLECTOR_URL_KEYS.get(collector_name, "")
|
||||
value = self.get_yaml_value(key)
|
||||
return value if isinstance(value, str) else ""
|
||||
|
||||
async def get_url(self, collector_name: str, db) -> str:
|
||||
|
||||
@@ -2,53 +2,99 @@
|
||||
# All external data source URLs should be configured here
|
||||
|
||||
arcgis:
|
||||
# ArcGIS 海缆 GeoJSON 查询接口
|
||||
cable_url: "https://services.arcgis.com/6DIQcwlPy8knb6sg/ArcGIS/rest/services/SubmarineCables/FeatureServer/2/query"
|
||||
# ArcGIS 登陆点 GeoJSON 查询接口
|
||||
landing_point_url: "https://services.arcgis.com/6DIQcwlPy8knb6sg/ArcGIS/rest/services/SubmarineCables/FeatureServer/1/query"
|
||||
# ArcGIS 海缆与登陆点关联关系查询接口
|
||||
cable_landing_relation_url: "https://services.arcgis.com/6DIQcwlPy8knb6sg/ArcGIS/rest/services/SubmarineCables/FeatureServer/3/query"
|
||||
|
||||
fao:
|
||||
# FAO 登陆点 CSV 下载地址
|
||||
landing_point_url: "https://data.apps.fao.org/catalog/dataset/1b75ff21-92f2-4b96-9b7b-98e8aa65ad5d/resource/b6071077-d1d4-4e97-aa00-42e902847c87/download/landing-point-geo.csv"
|
||||
|
||||
telegeography:
|
||||
# TeleGeography 海缆/系统主数据源,当前使用 GitHub 镜像 JSON
|
||||
cable_url: "https://raw.githubusercontent.com/lintaojlu/submarine_cable_information/main/cable.json"
|
||||
# TeleGeography 登陆点主数据源,当前使用 GitHub 镜像 JSON
|
||||
landing_point_url: "https://raw.githubusercontent.com/lintaojlu/submarine_cable_information/main/landing_point.json"
|
||||
# TeleGeography 历史 API 存档,用于 cable collector 的 fallback
|
||||
archived_cable_url: "https://web.archive.org/web/2024/https://www.submarinecablemap.com/api/v3/cable"
|
||||
# TeleGeography 官网页面,用于 cable collector 的最终 HTML 抓取 fallback
|
||||
live_map_url: "https://www.submarinecablemap.com"
|
||||
|
||||
huggingface:
|
||||
# Hugging Face 模型目录 API
|
||||
models_url: "https://huggingface.co/api/models"
|
||||
# Hugging Face 数据集目录 API
|
||||
datasets_url: "https://huggingface.co/api/datasets"
|
||||
# Hugging Face Spaces 目录 API
|
||||
spaces_url: "https://huggingface.co/api/spaces"
|
||||
|
||||
cloudflare:
|
||||
# Cloudflare Radar 设备类型摘要接口
|
||||
radar_device_url: "https://api.cloudflare.com/client/v4/radar/http/summary/device_type"
|
||||
# Cloudflare Radar 请求量时间序列接口
|
||||
radar_traffic_url: "https://api.cloudflare.com/client/v4/radar/http/timeseries/requests"
|
||||
# Cloudflare Radar 热点地理位置接口
|
||||
radar_top_locations_url: "https://api.cloudflare.com/client/v4/radar/http/top/locations"
|
||||
|
||||
peeringdb:
|
||||
# PeeringDB IXP API
|
||||
ixp_url: "https://www.peeringdb.com/api/ix"
|
||||
# PeeringDB Network API
|
||||
network_url: "https://www.peeringdb.com/api/net"
|
||||
# PeeringDB Facility API
|
||||
facility_url: "https://www.peeringdb.com/api/fac"
|
||||
|
||||
top500:
|
||||
# TOP500 榜单页面,用于主表抓取
|
||||
url: "https://top500.org/lists/top500/list/2025/11/"
|
||||
# TOP500 站点根地址,用于拼详情页链接
|
||||
base_url: "https://top500.org"
|
||||
|
||||
epoch_ai:
|
||||
# Epoch AI GPU Cluster 页面
|
||||
gpu_clusters_url: "https://epoch.ai/data/gpu-clusters"
|
||||
|
||||
spacetrack:
|
||||
# Space-Track 站点根地址,用于首页访问和登录地址推导
|
||||
base_url: "https://www.space-track.org"
|
||||
# Space-Track TLE 主查询接口
|
||||
tle_query_url: "https://www.space-track.org/basicspacedata/query/class/gp/orderby/EPOCH%20desc/limit/1000/format/json"
|
||||
|
||||
celestrak:
|
||||
# CelesTrak TLE 基础接口,collector 会在其后拼接 GROUP / FORMAT 参数
|
||||
base_url: "https://celestrak.org/NORAD/elements/gp.php"
|
||||
|
||||
ris_live:
|
||||
# RIPE RIS Live 流式订阅地址
|
||||
url: "https://ris-live.ripe.net/v1/stream/?format=json&client=planet-ris-live"
|
||||
|
||||
bgpstream:
|
||||
# CAIDA BGPStream Broker API
|
||||
url: "https://broker.bgpstream.caida.org/v2"
|
||||
|
||||
iptoasn:
|
||||
# IPtoASN prefix geography 合并数据下载地址
|
||||
combined_url: "https://iptoasn.com/data/ip2asn-combined.tsv.gz"
|
||||
|
||||
opengeofeed:
|
||||
# OpenGeoFeed 公共 geofeed CSV
|
||||
public_csv_url: "https://opengeofeed.org/feed/public.csv"
|
||||
|
||||
nro:
|
||||
# NRO delegated stats 下载地址
|
||||
delegated_stats_url: "https://ftp.ripe.net/pub/stats/ripencc/nro-stats/latest/nro-delegated-stats"
|
||||
|
||||
news_live_streams:
|
||||
# IPTV-org 频道元数据 JSON
|
||||
channels_url: "https://iptv-org.github.io/api/channels.json"
|
||||
# IPTV-org 频道播放流 JSON
|
||||
streams_url: "https://iptv-org.github.io/api/streams.json"
|
||||
# IPTV-org 台标 JSON
|
||||
logos_url: "https://iptv-org.github.io/api/logos.json"
|
||||
|
||||
barentswatch_vessels:
|
||||
# BarentsWatch Live AIS latest combined endpoint. Requires an AIS bearer token.
|
||||
url: "https://live.ais.barentswatch.no/v1/latest/combined"
|
||||
@@ -4,156 +4,246 @@ DEFAULT_DATASOURCES = {
|
||||
"top500": {
|
||||
"id": 1,
|
||||
"name": "TOP500 Supercomputers",
|
||||
"display_name": "TOP500 超算榜单",
|
||||
"module": "L1",
|
||||
"priority": "P0",
|
||||
"frequency_minutes": 240,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"epoch_ai_gpu": {
|
||||
"id": 2,
|
||||
"name": "Epoch AI GPU Clusters",
|
||||
"display_name": "Epoch AI GPU 集群",
|
||||
"module": "L1",
|
||||
"priority": "P0",
|
||||
"frequency_minutes": 360,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"huggingface_models": {
|
||||
"id": 3,
|
||||
"name": "HuggingFace Models",
|
||||
"display_name": "Hugging Face 模型",
|
||||
"module": "L2",
|
||||
"priority": "P1",
|
||||
"frequency_minutes": 720,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"huggingface_datasets": {
|
||||
"id": 4,
|
||||
"name": "HuggingFace Datasets",
|
||||
"display_name": "Hugging Face 数据集",
|
||||
"module": "L2",
|
||||
"priority": "P1",
|
||||
"frequency_minutes": 720,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"huggingface_spaces": {
|
||||
"id": 5,
|
||||
"name": "HuggingFace Spaces",
|
||||
"display_name": "Hugging Face Spaces",
|
||||
"module": "L2",
|
||||
"priority": "P2",
|
||||
"frequency_minutes": 1440,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"peeringdb_ixp": {
|
||||
"id": 6,
|
||||
"name": "PeeringDB IXP",
|
||||
"display_name": "PeeringDB 交换中心",
|
||||
"module": "L2",
|
||||
"priority": "P1",
|
||||
"frequency_minutes": 1440,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"peeringdb_network": {
|
||||
"id": 7,
|
||||
"name": "PeeringDB Networks",
|
||||
"display_name": "PeeringDB 网络",
|
||||
"module": "L2",
|
||||
"priority": "P2",
|
||||
"frequency_minutes": 2880,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"peeringdb_facility": {
|
||||
"id": 8,
|
||||
"name": "PeeringDB Facilities",
|
||||
"display_name": "PeeringDB 设施",
|
||||
"module": "L2",
|
||||
"priority": "P2",
|
||||
"frequency_minutes": 2880,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"telegeography_cables": {
|
||||
"id": 9,
|
||||
"name": "Submarine Cables",
|
||||
"display_name": "海底光缆",
|
||||
"module": "L2",
|
||||
"priority": "P1",
|
||||
"frequency_minutes": 10080,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"telegeography_landing": {
|
||||
"id": 10,
|
||||
"name": "Cable Landing Points",
|
||||
"display_name": "光缆登陆点",
|
||||
"module": "L2",
|
||||
"priority": "P2",
|
||||
"frequency_minutes": 10080,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"telegeography_systems": {
|
||||
"id": 11,
|
||||
"name": "Cable Systems",
|
||||
"display_name": "光缆系统",
|
||||
"module": "L2",
|
||||
"priority": "P2",
|
||||
"frequency_minutes": 10080,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"arcgis_cables": {
|
||||
"id": 15,
|
||||
"name": "ArcGIS Submarine Cables",
|
||||
"display_name": "ArcGIS 海底光缆",
|
||||
"module": "L2",
|
||||
"priority": "P1",
|
||||
"frequency_minutes": 10080,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"arcgis_landing_points": {
|
||||
"id": 16,
|
||||
"name": "ArcGIS Landing Points",
|
||||
"display_name": "ArcGIS 登陆点",
|
||||
"module": "L2",
|
||||
"priority": "P1",
|
||||
"frequency_minutes": 10080,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"arcgis_cable_landing_relation": {
|
||||
"id": 17,
|
||||
"name": "ArcGIS Cable-Landing Relations",
|
||||
"display_name": "ArcGIS 光缆登陆关系",
|
||||
"module": "L2",
|
||||
"priority": "P1",
|
||||
"frequency_minutes": 10080,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"fao_landing_points": {
|
||||
"id": 18,
|
||||
"name": "FAO Landing Points",
|
||||
"display_name": "FAO 登陆点",
|
||||
"module": "L2",
|
||||
"priority": "P1",
|
||||
"frequency_minutes": 10080,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"spacetrack_tle": {
|
||||
"id": 19,
|
||||
"name": "Space-Track TLE",
|
||||
"display_name": "Space-Track 轨道根数",
|
||||
"module": "L3",
|
||||
"priority": "P2",
|
||||
"frequency_minutes": 1440,
|
||||
"is_free": True,
|
||||
"requires_credentials": True,
|
||||
"credential_provider": "spacetrack",
|
||||
"credential_status": "planned",
|
||||
},
|
||||
"celestrak_tle": {
|
||||
"id": 20,
|
||||
"name": "CelesTrak TLE",
|
||||
"display_name": "CelesTrak 轨道根数",
|
||||
"module": "L3",
|
||||
"priority": "P2",
|
||||
"frequency_minutes": 1440,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"ris_live_bgp": {
|
||||
"id": 21,
|
||||
"name": "RIPE RIS Live BGP",
|
||||
"display_name": "RIPE RIS 实时 BGP",
|
||||
"module": "L3",
|
||||
"priority": "P1",
|
||||
"frequency_minutes": 15,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"bgpstream_bgp": {
|
||||
"id": 22,
|
||||
"name": "CAIDA BGPStream Backfill",
|
||||
"display_name": "CAIDA BGPStream 回填",
|
||||
"module": "L3",
|
||||
"priority": "P1",
|
||||
"frequency_minutes": 360,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"iptoasn_prefix_geo": {
|
||||
"id": 23,
|
||||
"name": "IPtoASN Prefix Geography",
|
||||
"display_name": "IPtoASN 前缀地理",
|
||||
"module": "L3",
|
||||
"priority": "P1",
|
||||
"frequency_minutes": 1440,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"opengeofeed_prefix_geo": {
|
||||
"id": 24,
|
||||
"name": "OpenGeoFeed Prefix Geography",
|
||||
"display_name": "OpenGeoFeed 前缀地理",
|
||||
"module": "L3",
|
||||
"priority": "P1",
|
||||
"frequency_minutes": 1440,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"nro_delegated_prefix_geo": {
|
||||
"id": 25,
|
||||
"name": "NRO Delegated Prefix Geography",
|
||||
"display_name": "NRO 分配前缀地理",
|
||||
"module": "L3",
|
||||
"priority": "P1",
|
||||
"frequency_minutes": 1440,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"news_live_streams": {
|
||||
"id": 26,
|
||||
"name": "News Live Streams",
|
||||
"display_name": "新闻直播源",
|
||||
"module": "L4",
|
||||
"priority": "P2",
|
||||
"frequency_minutes": 720,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"barentswatch_vessels": {
|
||||
"id": 27,
|
||||
"name": "BarentsWatch AIS Vessels",
|
||||
"display_name": "BarentsWatch AIS 船舶",
|
||||
"module": "L4",
|
||||
"priority": "P1",
|
||||
"frequency_minutes": 1,
|
||||
"is_free": True,
|
||||
"requires_credentials": True,
|
||||
"credential_provider": "barentswatch",
|
||||
"credential_status": "supported",
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,161 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
|
||||
from collections.abc import Mapping, Sequence
|
||||
from typing import Any
|
||||
|
||||
from app.core.request_context import get_request_id
|
||||
|
||||
DEFAULT_SERVICE = "backend"
|
||||
DEFAULT_EVENT = "app.log"
|
||||
DEFAULT_LOG_LEVEL = os.getenv("PLANET_LOG_LEVEL", "INFO").upper()
|
||||
REDACTED = "[REDACTED]"
|
||||
SENSITIVE_FIELD_NAMES = {
|
||||
"access_token",
|
||||
"api_key",
|
||||
"authorization",
|
||||
"cookie",
|
||||
"password",
|
||||
"refresh_token",
|
||||
"secret",
|
||||
"token",
|
||||
}
|
||||
SENSITIVE_TEXT_PATTERNS = (
|
||||
re.compile(r"(?i)(authorization\s*[:=]\s*)(.+)"),
|
||||
re.compile(r"(?i)(bearer\s+)([A-Za-z0-9._\-]+)"),
|
||||
re.compile(r"(?i)(token\s*[:=]\s*)(.+)"),
|
||||
re.compile(r"(?i)(password\s*[:=]\s*)(.+)"),
|
||||
re.compile(r"(?i)(cookie\s*[:=]\s*)(.+)"),
|
||||
)
|
||||
|
||||
|
||||
def sanitize_log_value(value: Any) -> Any:
|
||||
if isinstance(value, Mapping):
|
||||
return {
|
||||
str(key): (REDACTED if str(key).lower() in SENSITIVE_FIELD_NAMES else sanitize_log_value(item))
|
||||
for key, item in value.items()
|
||||
}
|
||||
if isinstance(value, Sequence) and not isinstance(value, (str, bytes, bytearray)):
|
||||
return [sanitize_log_value(item) for item in value]
|
||||
if isinstance(value, str):
|
||||
sanitized = value
|
||||
for pattern in SENSITIVE_TEXT_PATTERNS:
|
||||
sanitized = pattern.sub(lambda match: f"{match.group(1)}{REDACTED}", sanitized)
|
||||
return sanitized
|
||||
return value
|
||||
|
||||
|
||||
def _normalize_context(context: Any) -> dict[str, Any]:
|
||||
if context is None:
|
||||
return {}
|
||||
if isinstance(context, Mapping):
|
||||
sanitized = sanitize_log_value(context)
|
||||
return {str(key): value for key, value in sanitized.items()}
|
||||
return {"value": sanitize_log_value(context)}
|
||||
|
||||
|
||||
class PlanetContextFilter(logging.Filter):
|
||||
def filter(self, record: logging.LogRecord) -> bool:
|
||||
record.request_id = getattr(record, "request_id", None) or get_request_id() or "-"
|
||||
record.service = getattr(record, "service", None) or DEFAULT_SERVICE
|
||||
record.event = getattr(record, "event", None) or DEFAULT_EVENT
|
||||
record.context = _normalize_context(getattr(record, "context", None))
|
||||
record.message = sanitize_log_value(record.getMessage())
|
||||
return True
|
||||
|
||||
|
||||
class PlanetFormatter(logging.Formatter):
|
||||
def format(self, record: logging.LogRecord) -> str:
|
||||
timestamp = self.formatTime(record, self.datefmt)
|
||||
level = record.levelname
|
||||
service = getattr(record, "service", DEFAULT_SERVICE)
|
||||
module_name = record.name
|
||||
event = getattr(record, "event", DEFAULT_EVENT)
|
||||
request_id = getattr(record, "request_id", "-")
|
||||
message = sanitize_log_value(record.getMessage())
|
||||
context = _normalize_context(getattr(record, "context", None))
|
||||
context_suffix = ""
|
||||
if context:
|
||||
context_suffix = f" context={json.dumps(context, ensure_ascii=False, sort_keys=True)}"
|
||||
rendered = (
|
||||
f"{timestamp} {level} service={service} module={module_name} "
|
||||
f"event={event} request_id={request_id} message={message}{context_suffix}"
|
||||
)
|
||||
if record.exc_info:
|
||||
rendered = f"{rendered}\n{self.formatException(record.exc_info)}"
|
||||
return rendered
|
||||
|
||||
|
||||
class PlanetLoggerAdapter(logging.LoggerAdapter):
|
||||
def process(self, msg: Any, kwargs: dict[str, Any]) -> tuple[Any, dict[str, Any]]:
|
||||
extra = dict(self.extra)
|
||||
extra.update(kwargs.get("extra", {}))
|
||||
if "context" in extra:
|
||||
extra["context"] = _normalize_context(extra.get("context"))
|
||||
kwargs["extra"] = extra
|
||||
return sanitize_log_value(msg), kwargs
|
||||
|
||||
def log_event(
|
||||
self,
|
||||
level: int,
|
||||
message: str,
|
||||
*,
|
||||
event: str,
|
||||
context: Mapping[str, Any] | None = None,
|
||||
**extra: Any,
|
||||
) -> None:
|
||||
self.log(level, message, extra={"event": event, "context": context or {}, **extra})
|
||||
|
||||
def debug_event(self, message: str, *, event: str, context: Mapping[str, Any] | None = None, **extra: Any) -> None:
|
||||
self.log_event(logging.DEBUG, message, event=event, context=context, **extra)
|
||||
|
||||
def info_event(self, message: str, *, event: str, context: Mapping[str, Any] | None = None, **extra: Any) -> None:
|
||||
self.log_event(logging.INFO, message, event=event, context=context, **extra)
|
||||
|
||||
def warning_event(self, message: str, *, event: str, context: Mapping[str, Any] | None = None, **extra: Any) -> None:
|
||||
self.log_event(logging.WARNING, message, event=event, context=context, **extra)
|
||||
|
||||
def error_event(self, message: str, *, event: str, context: Mapping[str, Any] | None = None, **extra: Any) -> None:
|
||||
self.log_event(logging.ERROR, message, event=event, context=context, **extra)
|
||||
|
||||
def exception_event(
|
||||
self,
|
||||
message: str,
|
||||
*,
|
||||
event: str,
|
||||
context: Mapping[str, Any] | None = None,
|
||||
**extra: Any,
|
||||
) -> None:
|
||||
self.error(message, exc_info=True, extra={"event": event, "context": context or {}, **extra})
|
||||
|
||||
|
||||
def get_logger(name: str, *, service: str = DEFAULT_SERVICE) -> PlanetLoggerAdapter:
|
||||
return PlanetLoggerAdapter(logging.getLogger(name), {"service": service})
|
||||
|
||||
|
||||
def configure_logging(level: str | None = None) -> None:
|
||||
root_logger = logging.getLogger()
|
||||
if getattr(configure_logging, "_configured", False):
|
||||
if level:
|
||||
root_logger.setLevel(level.upper())
|
||||
return
|
||||
|
||||
handler = logging.StreamHandler()
|
||||
handler.setFormatter(PlanetFormatter(datefmt="%Y-%m-%d %H:%M:%S"))
|
||||
handler.addFilter(PlanetContextFilter())
|
||||
|
||||
root_logger.handlers.clear()
|
||||
root_logger.addHandler(handler)
|
||||
root_logger.setLevel((level or DEFAULT_LOG_LEVEL).upper())
|
||||
|
||||
for logger_name in ("uvicorn", "uvicorn.error", "uvicorn.access"):
|
||||
target_logger = logging.getLogger(logger_name)
|
||||
target_logger.handlers.clear()
|
||||
target_logger.propagate = True
|
||||
|
||||
logging.captureWarnings(True)
|
||||
configure_logging._configured = True
|
||||
@@ -0,0 +1,14 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from contextvars import ContextVar
|
||||
|
||||
|
||||
request_id_context: ContextVar[str | None] = ContextVar("request_id", default=None)
|
||||
|
||||
|
||||
def set_request_id(request_id: str | None) -> None:
|
||||
request_id_context.set(request_id)
|
||||
|
||||
|
||||
def get_request_id() -> str | None:
|
||||
return request_id_context.get()
|
||||
@@ -0,0 +1,151 @@
|
||||
"""Registry of target schemas supported by mapped custom data sources."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from typing import Any
|
||||
|
||||
from pydantic import BaseModel, Field, ValidationError, field_validator
|
||||
|
||||
|
||||
class VesselAISRecord(BaseModel):
|
||||
mmsi: int = Field(ge=100000000, le=999999999)
|
||||
lat: float = Field(ge=-90, le=90)
|
||||
lon: float = Field(ge=-180, le=180)
|
||||
sog: float | None = None
|
||||
cog: float | None = Field(default=None, ge=0, le=360)
|
||||
heading: int | None = Field(default=None, ge=0, le=511)
|
||||
name: str | None = None
|
||||
vessel_type: str | int | None = None
|
||||
received_at: datetime | None = None
|
||||
|
||||
|
||||
class GeoPointRecord(BaseModel):
|
||||
lat: float = Field(ge=-90, le=90)
|
||||
lon: float = Field(ge=-180, le=180)
|
||||
name: str | None = None
|
||||
type: str | None = None
|
||||
source_id: str | None = None
|
||||
observed_at: datetime | None = None
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class GenericRecord(BaseModel):
|
||||
data: dict[str, Any] = Field(default_factory=dict)
|
||||
source_id: str | None = None
|
||||
observed_at: datetime | None = None
|
||||
|
||||
@field_validator("data")
|
||||
@classmethod
|
||||
def require_payload(cls, value: dict[str, Any]) -> dict[str, Any]:
|
||||
if not value:
|
||||
raise ValueError("generic_records requires a non-empty data object")
|
||||
return value
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class TargetField:
|
||||
name: str
|
||||
type: str
|
||||
required: bool = False
|
||||
description: str = ""
|
||||
example: Any = None
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"name": self.name,
|
||||
"type": self.type,
|
||||
"required": self.required,
|
||||
"description": self.description,
|
||||
"example": self.example,
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class TargetSchema:
|
||||
key: str
|
||||
label: str
|
||||
description: str
|
||||
fields: tuple[TargetField, ...]
|
||||
model: type[BaseModel]
|
||||
destination: str
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"key": self.key,
|
||||
"label": self.label,
|
||||
"description": self.description,
|
||||
"destination": self.destination,
|
||||
"fields": [field.to_dict() for field in self.fields],
|
||||
}
|
||||
|
||||
def validate_record(self, record: dict[str, Any]) -> tuple[dict[str, Any] | None, list[str]]:
|
||||
try:
|
||||
return self.model.model_validate(record).model_dump(mode="json"), []
|
||||
except ValidationError as exc:
|
||||
return None, [
|
||||
".".join(str(part) for part in error["loc"]) + f": {error['msg']}"
|
||||
for error in exc.errors()
|
||||
]
|
||||
|
||||
|
||||
TARGET_SCHEMAS: dict[str, TargetSchema] = {
|
||||
"vessel_ais": TargetSchema(
|
||||
key="vessel_ais",
|
||||
label="船舶 AIS",
|
||||
description="船只位置、航速、航向、MMSI 等 AIS 数据。",
|
||||
destination="vessel_position",
|
||||
model=VesselAISRecord,
|
||||
fields=(
|
||||
TargetField("mmsi", "integer", True, "MMSI 九位船舶标识", 257123000),
|
||||
TargetField("lat", "float", True, "纬度", 59.91),
|
||||
TargetField("lon", "float", True, "经度", 10.75),
|
||||
TargetField("sog", "float", False, "对地航速,单位节", 12.4),
|
||||
TargetField("cog", "float", False, "对地航向,0-360 度", 184.5),
|
||||
TargetField("heading", "integer", False, "船首向,0-511", 186),
|
||||
TargetField("name", "string", False, "船名", "OSLO EXPRESS"),
|
||||
TargetField("vessel_type", "string", False, "船型", "cargo"),
|
||||
TargetField("received_at", "datetime", False, "数据接收时间", "2026-04-28T00:00:00Z"),
|
||||
),
|
||||
),
|
||||
"geo_points": TargetSchema(
|
||||
key="geo_points",
|
||||
label="通用地理点",
|
||||
description="带经纬度的通用实体或事件点位。",
|
||||
destination="generic_geo_points",
|
||||
model=GeoPointRecord,
|
||||
fields=(
|
||||
TargetField("lat", "float", True, "纬度", 1.3),
|
||||
TargetField("lon", "float", True, "经度", 103.8),
|
||||
TargetField("name", "string", False, "点位名称", "Singapore"),
|
||||
TargetField("type", "string", False, "点位类型", "datacenter"),
|
||||
TargetField("source_id", "string", False, "来源侧 ID", "sg-1"),
|
||||
TargetField("observed_at", "datetime", False, "观测时间", "2026-04-28T00:00:00Z"),
|
||||
TargetField("metadata", "object", False, "扩展字段", {"provider": "example"}),
|
||||
),
|
||||
),
|
||||
"generic_records": TargetSchema(
|
||||
key="generic_records",
|
||||
label="通用结构化记录",
|
||||
description="未知结构数据沉淀,不直接进入 Earth 图层。",
|
||||
destination="collected_data",
|
||||
model=GenericRecord,
|
||||
fields=(
|
||||
TargetField("data", "object", True, "结构化记录主体", {"raw": "value"}),
|
||||
TargetField("source_id", "string", False, "来源侧 ID", "record-1"),
|
||||
TargetField("observed_at", "datetime", False, "观测时间", "2026-04-28T00:00:00Z"),
|
||||
),
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def list_target_schemas() -> list[dict[str, Any]]:
|
||||
return [schema.to_dict() for schema in TARGET_SCHEMAS.values()]
|
||||
|
||||
|
||||
def get_target_schema(key: str) -> TargetSchema:
|
||||
try:
|
||||
return TARGET_SCHEMAS[key]
|
||||
except KeyError as exc:
|
||||
raise ValueError(f"Unsupported target schema: {key}") from exc
|
||||
@@ -5,10 +5,22 @@ from sqlalchemy.ext.asyncio import AsyncSession, create_async_engine, async_sess
|
||||
from sqlalchemy.orm import declarative_base
|
||||
|
||||
from app.core.config import settings
|
||||
from app.core.logging import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
DB_POOL_CONFIG = {
|
||||
"pool_pre_ping": True,
|
||||
"pool_recycle": 1800,
|
||||
"pool_size": 10,
|
||||
"max_overflow": 20,
|
||||
"pool_timeout": 30,
|
||||
}
|
||||
|
||||
engine = create_async_engine(
|
||||
settings.DATABASE_URL,
|
||||
echo=settings.DEBUG if hasattr(settings, "DEBUG") else False,
|
||||
**DB_POOL_CONFIG,
|
||||
)
|
||||
|
||||
async_session_factory = async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False)
|
||||
@@ -95,6 +107,23 @@ async def init_db():
|
||||
import app.models.bgp_observation # noqa: F401
|
||||
import app.models.collected_data # noqa: F401
|
||||
import app.models.system_setting # noqa: F401
|
||||
import app.models.playground_session # noqa: F401
|
||||
import app.models.playground_message # noqa: F401
|
||||
import app.models.system_log # noqa: F401
|
||||
import app.models.vessel # noqa: F401
|
||||
import app.models.datasource_mapping # noqa: F401
|
||||
|
||||
logger.warning_event(
|
||||
"Database pool settings active",
|
||||
event="database.pool.initialized",
|
||||
context={
|
||||
"pool_pre_ping": DB_POOL_CONFIG["pool_pre_ping"],
|
||||
"pool_recycle": DB_POOL_CONFIG["pool_recycle"],
|
||||
"pool_size": DB_POOL_CONFIG["pool_size"],
|
||||
"max_overflow": DB_POOL_CONFIG["max_overflow"],
|
||||
"pool_timeout": DB_POOL_CONFIG["pool_timeout"],
|
||||
},
|
||||
)
|
||||
|
||||
async with engine.begin() as conn:
|
||||
await conn.run_sync(Base.metadata.create_all)
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
from contextlib import asynccontextmanager
|
||||
from uuid import uuid4
|
||||
|
||||
from fastapi import FastAPI
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
@@ -7,6 +8,8 @@ from starlette.middleware.base import BaseHTTPMiddleware
|
||||
from app.api.main import api_router
|
||||
from app.api.v1 import websocket
|
||||
from app.core.config import settings
|
||||
from app.core.logging import configure_logging
|
||||
from app.core.request_context import set_request_id
|
||||
from app.core.websocket.broadcaster import broadcaster
|
||||
from app.db.session import init_db
|
||||
from app.services.scheduler import (
|
||||
@@ -17,6 +20,9 @@ from app.services.scheduler import (
|
||||
)
|
||||
|
||||
|
||||
configure_logging()
|
||||
|
||||
|
||||
class WebSocketCORSMiddleware(BaseHTTPMiddleware):
|
||||
async def dispatch(self, request, call_next):
|
||||
if request.url.path.startswith("/ws") and request.method == "GET":
|
||||
@@ -28,6 +34,18 @@ class WebSocketCORSMiddleware(BaseHTTPMiddleware):
|
||||
return await call_next(request)
|
||||
|
||||
|
||||
class RequestContextMiddleware(BaseHTTPMiddleware):
|
||||
async def dispatch(self, request, call_next):
|
||||
request_id = request.headers.get("X-Request-ID") or uuid4().hex
|
||||
set_request_id(request_id)
|
||||
try:
|
||||
response = await call_next(request)
|
||||
finally:
|
||||
set_request_id(None)
|
||||
response.headers["X-Request-ID"] = request_id
|
||||
return response
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
await init_db()
|
||||
@@ -58,6 +76,7 @@ app.add_middleware(
|
||||
allow_headers=["*"],
|
||||
)
|
||||
|
||||
app.add_middleware(RequestContextMiddleware)
|
||||
app.add_middleware(WebSocketCORSMiddleware)
|
||||
|
||||
app.include_router(api_router, prefix="/api/v1")
|
||||
|
||||
@@ -9,6 +9,11 @@ from app.models.bgp_anomaly import BGPAnomaly
|
||||
from app.models.bgp_incident import BGPIncident
|
||||
from app.models.bgp_observation import BGPObservation
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.models.playground_session import PlaygroundSession
|
||||
from app.models.playground_message import PlaygroundMessage
|
||||
from app.models.system_log import SystemLog, AuditLog
|
||||
from app.models.vessel import VesselPosition, VesselStatic
|
||||
from app.models.datasource_mapping import DataSourceMappingTemplate
|
||||
|
||||
__all__ = [
|
||||
"User",
|
||||
@@ -24,4 +29,9 @@ __all__ = [
|
||||
"BGPAnomaly",
|
||||
"BGPIncident",
|
||||
"BGPObservation",
|
||||
"SystemLog",
|
||||
"AuditLog",
|
||||
"VesselPosition",
|
||||
"VesselStatic",
|
||||
"DataSourceMappingTemplate",
|
||||
]
|
||||
@@ -0,0 +1,32 @@
|
||||
"""Mapping templates for user-defined data source payloads."""
|
||||
|
||||
from sqlalchemy import Boolean, Column, DateTime, ForeignKey, Integer, JSON, String
|
||||
from sqlalchemy.sql import func
|
||||
|
||||
from app.db.session import Base
|
||||
|
||||
|
||||
class DataSourceMappingTemplate(Base):
|
||||
__tablename__ = "datasource_mapping_templates"
|
||||
|
||||
id = Column(Integer, primary_key=True, autoincrement=True)
|
||||
datasource_config_id = Column(
|
||||
Integer,
|
||||
ForeignKey("datasource_configs.id"),
|
||||
nullable=False,
|
||||
index=True,
|
||||
)
|
||||
target_schema = Column(String(80), nullable=False, index=True)
|
||||
mapping_json = Column(JSON, nullable=False, default={})
|
||||
sample_payload_hash = Column(String(64), nullable=True)
|
||||
validation_status = Column(String(30), nullable=False, default="draft")
|
||||
version = Column(Integer, nullable=False, default=1)
|
||||
is_active = Column(Boolean, nullable=False, default=False, index=True)
|
||||
created_at = Column(DateTime(timezone=True), server_default=func.now())
|
||||
updated_at = Column(DateTime(timezone=True), server_default=func.now(), onupdate=func.now())
|
||||
|
||||
def __repr__(self):
|
||||
return (
|
||||
f"<DataSourceMappingTemplate {self.id}: "
|
||||
f"{self.datasource_config_id}/{self.target_schema}/v{self.version}>"
|
||||
)
|
||||
@@ -0,0 +1,40 @@
|
||||
from sqlalchemy import JSON, Boolean, Column, DateTime, ForeignKey, Integer, String, Text
|
||||
from sqlalchemy.sql import func
|
||||
|
||||
from app.db.session import Base
|
||||
|
||||
|
||||
class PlaygroundMessage(Base):
|
||||
__tablename__ = "playground_messages"
|
||||
|
||||
id = Column(Integer, primary_key=True, autoincrement=True)
|
||||
public_id = Column(String(64), unique=True, index=True, nullable=False)
|
||||
session_id = Column(Integer, ForeignKey("playground_sessions.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
user_id = Column(Integer, ForeignKey("users.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
parent_message_id = Column(Integer, ForeignKey("playground_messages.id", ondelete="SET NULL"), nullable=True)
|
||||
role = Column(String(20), nullable=False)
|
||||
kind = Column(String(20), nullable=False, default="message")
|
||||
status = Column(String(20), nullable=False, default="done")
|
||||
title = Column(String(255), nullable=True)
|
||||
content = Column(Text, nullable=False, default="")
|
||||
thinking_content = Column(Text, nullable=False, default="")
|
||||
meta = Column(JSON, nullable=False, default=list)
|
||||
provider = Column(String(100), nullable=True)
|
||||
model = Column(String(200), nullable=True)
|
||||
request_id = Column(String(100), nullable=True)
|
||||
raw_response = Column(JSON, nullable=False, default=dict)
|
||||
content_blocks = Column(JSON, nullable=False, default=list)
|
||||
text_blocks = Column(JSON, nullable=False, default=list)
|
||||
thinking_blocks = Column(JSON, nullable=False, default=list)
|
||||
sort_order = Column(Integer, nullable=False, default=0, index=True)
|
||||
is_visible = Column(Boolean, nullable=False, default=True)
|
||||
created_at = Column(DateTime(timezone=True), server_default=func.now(), nullable=False)
|
||||
updated_at = Column(
|
||||
DateTime(timezone=True),
|
||||
server_default=func.now(),
|
||||
onupdate=func.now(),
|
||||
nullable=False,
|
||||
)
|
||||
|
||||
def __repr__(self):
|
||||
return f"<PlaygroundMessage public_id={self.public_id} role={self.role} status={self.status}>"
|
||||
@@ -0,0 +1,27 @@
|
||||
from sqlalchemy import JSON, Column, DateTime, ForeignKey, Integer, String, UniqueConstraint
|
||||
from sqlalchemy.sql import func
|
||||
|
||||
from app.db.session import Base
|
||||
|
||||
|
||||
class PlaygroundSession(Base):
|
||||
__tablename__ = "playground_sessions"
|
||||
__table_args__ = (
|
||||
UniqueConstraint("user_id", "session_key", name="uq_playground_sessions_user_session_key"),
|
||||
)
|
||||
|
||||
id = Column(Integer, primary_key=True, autoincrement=True)
|
||||
user_id = Column(Integer, ForeignKey("users.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
session_key = Column(String(100), nullable=False, default="default")
|
||||
title = Column(String(200), nullable=False, default="Playground 会话")
|
||||
state = Column(JSON, nullable=False, default={})
|
||||
created_at = Column(DateTime(timezone=True), server_default=func.now(), nullable=False)
|
||||
updated_at = Column(
|
||||
DateTime(timezone=True),
|
||||
server_default=func.now(),
|
||||
onupdate=func.now(),
|
||||
nullable=False,
|
||||
)
|
||||
|
||||
def __repr__(self):
|
||||
return f"<PlaygroundSession user_id={self.user_id} session_key={self.session_key}>"
|
||||
@@ -0,0 +1,40 @@
|
||||
from sqlalchemy import JSON, Column, DateTime, Integer, String, Text
|
||||
from sqlalchemy.sql import func
|
||||
|
||||
from app.db.session import Base
|
||||
|
||||
|
||||
class SystemLog(Base):
|
||||
__tablename__ = "system_logs"
|
||||
|
||||
id = Column(Integer, primary_key=True, autoincrement=True)
|
||||
occurred_at = Column(DateTime(timezone=True), server_default=func.now(), index=True)
|
||||
source = Column(String(50), nullable=False, index=True)
|
||||
service = Column(String(50), nullable=True)
|
||||
module = Column(String(120), nullable=True)
|
||||
event = Column(String(160), nullable=True, index=True)
|
||||
level = Column(String(20), nullable=False, index=True)
|
||||
message = Column(Text, nullable=False)
|
||||
request_id = Column(String(64), nullable=True, index=True)
|
||||
trace_id = Column(String(64), nullable=True)
|
||||
user_id = Column(Integer, nullable=True, index=True)
|
||||
category = Column(String(80), nullable=True, index=True)
|
||||
context = Column(JSON, nullable=False, default=dict)
|
||||
created_at = Column(DateTime(timezone=True), server_default=func.now())
|
||||
|
||||
|
||||
class AuditLog(Base):
|
||||
__tablename__ = "audit_logs"
|
||||
|
||||
id = Column(Integer, primary_key=True, autoincrement=True)
|
||||
occurred_at = Column(DateTime(timezone=True), server_default=func.now(), index=True)
|
||||
actor_id = Column(Integer, nullable=True, index=True)
|
||||
actor_name = Column(String(255), nullable=True)
|
||||
action = Column(String(120), nullable=False, index=True)
|
||||
target_type = Column(String(80), nullable=True)
|
||||
target_id = Column(String(120), nullable=True)
|
||||
result = Column(String(40), nullable=True, index=True)
|
||||
request_id = Column(String(64), nullable=True, index=True)
|
||||
ip = Column(String(64), nullable=True)
|
||||
details = Column(JSON, nullable=False, default=dict)
|
||||
created_at = Column(DateTime(timezone=True), server_default=func.now())
|
||||
@@ -0,0 +1,75 @@
|
||||
"""Vessel AIS models for live maritime tracking."""
|
||||
|
||||
from sqlalchemy import BigInteger, Column, DateTime, Float, Index, Integer, SmallInteger, String
|
||||
from sqlalchemy.sql import func
|
||||
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.db.session import Base
|
||||
|
||||
|
||||
class VesselStatic(Base):
|
||||
"""Slow-changing vessel identity and dimensions."""
|
||||
|
||||
__tablename__ = "vessel_static"
|
||||
|
||||
mmsi = Column(BigInteger, primary_key=True)
|
||||
name = Column(String(128), nullable=True)
|
||||
callsign = Column(String(16), nullable=True)
|
||||
vessel_type = Column(SmallInteger, nullable=True, index=True)
|
||||
vessel_type_name = Column(String(64), nullable=True, index=True)
|
||||
flag = Column(String(4), nullable=True, index=True)
|
||||
length = Column(Float, nullable=True)
|
||||
width = Column(Float, nullable=True)
|
||||
draught = Column(Float, nullable=True)
|
||||
imo = Column(BigInteger, nullable=True)
|
||||
updated_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now())
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {
|
||||
"mmsi": self.mmsi,
|
||||
"name": self.name,
|
||||
"callsign": self.callsign,
|
||||
"vessel_type": self.vessel_type,
|
||||
"vessel_type_name": self.vessel_type_name,
|
||||
"flag": self.flag,
|
||||
"length": self.length,
|
||||
"width": self.width,
|
||||
"draught": self.draught,
|
||||
"imo": self.imo,
|
||||
"updated_at": to_iso8601_utc(self.updated_at),
|
||||
}
|
||||
|
||||
|
||||
class VesselPosition(Base):
|
||||
"""Append-only AIS positions retained for short history windows."""
|
||||
|
||||
__tablename__ = "vessel_position"
|
||||
|
||||
id = Column(Integer, primary_key=True, autoincrement=True)
|
||||
mmsi = Column(BigInteger, nullable=False, index=True)
|
||||
lat = Column(Float, nullable=False)
|
||||
lon = Column(Float, nullable=False)
|
||||
sog = Column(Float, nullable=True)
|
||||
cog = Column(Float, nullable=True)
|
||||
heading = Column(SmallInteger, nullable=True)
|
||||
nav_status = Column(SmallInteger, nullable=True, index=True)
|
||||
received_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now(), index=True)
|
||||
|
||||
__table_args__ = (
|
||||
Index("idx_vessel_pos_mmsi_time", "mmsi", "received_at"),
|
||||
Index("idx_vessel_pos_time", "received_at"),
|
||||
Index("idx_vessel_pos_lat_lon", "lat", "lon"),
|
||||
)
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {
|
||||
"id": self.id,
|
||||
"mmsi": self.mmsi,
|
||||
"lat": self.lat,
|
||||
"lon": self.lon,
|
||||
"sog": self.sog,
|
||||
"cog": self.cog,
|
||||
"heading": self.heading,
|
||||
"nav_status": self.nav_status,
|
||||
"received_at": to_iso8601_utc(self.received_at),
|
||||
}
|
||||
@@ -3,6 +3,14 @@ from typing import Any
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class AIContentBlock(BaseModel):
|
||||
type: str
|
||||
text: str | None = None
|
||||
thinking: str | None = None
|
||||
signature: str | None = None
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class SituationalAnalysisRequest(BaseModel):
|
||||
title: str = Field(..., min_length=1, max_length=200)
|
||||
objective: str = Field(..., min_length=1, max_length=1000)
|
||||
@@ -10,18 +18,158 @@ class SituationalAnalysisRequest(BaseModel):
|
||||
observations: list[str] = Field(default_factory=list)
|
||||
constraints: list[str] = Field(default_factory=list)
|
||||
preferred_model: str | None = Field(default=None, max_length=200)
|
||||
thinking: dict[str, Any] | None = None
|
||||
|
||||
|
||||
class BGPBriefRequest(BaseModel):
|
||||
incident_limit: int = Field(default=5, ge=1, le=10)
|
||||
anomaly_limit: int = Field(default=6, ge=1, le=12)
|
||||
collector_limit: int = Field(default=5, ge=1, le=10)
|
||||
preferred_model: str | None = Field(default=None, max_length=200)
|
||||
thinking: dict[str, Any] | None = None
|
||||
|
||||
|
||||
class AlertBriefRequest(BaseModel):
|
||||
alert_limit: int = Field(default=8, ge=1, le=20)
|
||||
preferred_model: str | None = Field(default=None, max_length=200)
|
||||
thinking: dict[str, Any] | None = None
|
||||
|
||||
|
||||
class SituationalAlertBriefRequest(BaseModel):
|
||||
preferred_model: str | None = Field(default=None, max_length=200)
|
||||
thinking: dict[str, Any] | None = None
|
||||
|
||||
|
||||
class SituationalAnalysisResponse(BaseModel):
|
||||
provider: str
|
||||
model: str
|
||||
content: str
|
||||
content_blocks: list[AIContentBlock] = Field(default_factory=list)
|
||||
text_blocks: list[str] = Field(default_factory=list)
|
||||
thinking_blocks: list[str] = Field(default_factory=list)
|
||||
raw_response: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class BGPBriefRecordSummary(BaseModel):
|
||||
id: str
|
||||
title: str
|
||||
provider: str
|
||||
model: str
|
||||
request_id: str | None = None
|
||||
generated_at: str
|
||||
|
||||
|
||||
class BGPBriefRecordResponse(BGPBriefRecordSummary):
|
||||
content_markdown: str
|
||||
facts: list[str] = Field(default_factory=list)
|
||||
context: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class AlertBriefResponse(SituationalAnalysisResponse):
|
||||
title: str
|
||||
objective: str
|
||||
facts: list[str] = Field(default_factory=list)
|
||||
context: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class SituationalAlertBriefResponse(SituationalAnalysisResponse):
|
||||
title: str
|
||||
objective: str
|
||||
facts: list[str] = Field(default_factory=list)
|
||||
context: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class AIProviderStatusResponse(BaseModel):
|
||||
provider: str
|
||||
api: str | None = None
|
||||
enabled: bool
|
||||
configured: bool
|
||||
model: str | None = None
|
||||
base_url: str | None = None
|
||||
|
||||
|
||||
class PlaygroundSessionState(BaseModel):
|
||||
messages: list[dict[str, Any]] = Field(default_factory=list)
|
||||
selectedPresetKey: str = Field(default="bgp-brief", max_length=100)
|
||||
title: str = Field(default="", max_length=200)
|
||||
objective: str = Field(default="", max_length=1000)
|
||||
constraints: str = Field(default="")
|
||||
inputValue: str = Field(default="")
|
||||
analysis: dict[str, Any] | None = None
|
||||
latestAnalysisMessageId: str | None = Field(default=None, max_length=200)
|
||||
analysisMeta: dict[str, Any] = Field(default_factory=dict)
|
||||
helpExpanded: bool = True
|
||||
|
||||
|
||||
class PlaygroundSessionUpsertRequest(BaseModel):
|
||||
session_key: str = Field(default="default", min_length=1, max_length=100)
|
||||
title: str | None = Field(default=None, max_length=200)
|
||||
state: PlaygroundSessionState
|
||||
|
||||
|
||||
class PlaygroundMessageRecord(BaseModel):
|
||||
id: str
|
||||
role: str
|
||||
kind: str = "message"
|
||||
status: str = "done"
|
||||
title: str | None = None
|
||||
content: str = ""
|
||||
thinking_content: str = ""
|
||||
meta: list[str] = Field(default_factory=list)
|
||||
markdown: bool = True
|
||||
provider: str | None = None
|
||||
model: str | None = None
|
||||
request_id: str | None = None
|
||||
raw_response: dict[str, Any] = Field(default_factory=dict)
|
||||
content_blocks: list[dict[str, Any]] = Field(default_factory=list)
|
||||
text_blocks: list[str] = Field(default_factory=list)
|
||||
thinking_blocks: list[str] = Field(default_factory=list)
|
||||
parent_message_id: str | None = None
|
||||
created_at: str
|
||||
updated_at: str
|
||||
|
||||
|
||||
class PlaygroundSessionResponse(BaseModel):
|
||||
id: str
|
||||
session_key: str
|
||||
title: str
|
||||
state: PlaygroundSessionState
|
||||
created_at: str
|
||||
updated_at: str
|
||||
|
||||
|
||||
class PlaygroundThreadResponse(BaseModel):
|
||||
session: PlaygroundSessionResponse
|
||||
messages: list[PlaygroundMessageRecord] = Field(default_factory=list)
|
||||
|
||||
|
||||
class PlaygroundMessageCreateRequest(BaseModel):
|
||||
session_key: str = Field(default="default", min_length=1, max_length=100)
|
||||
title: str = Field(..., min_length=1, max_length=200)
|
||||
objective: str = Field(..., min_length=1, max_length=1000)
|
||||
constraints: str = Field(default="")
|
||||
input: str = Field(..., min_length=1)
|
||||
selected_preset_key: str = Field(default="bgp-brief", max_length=100)
|
||||
help_expanded: bool = True
|
||||
|
||||
|
||||
class PlaygroundMessageActionResponse(BaseModel):
|
||||
session: PlaygroundSessionResponse
|
||||
messages: list[PlaygroundMessageRecord] = Field(default_factory=list)
|
||||
active_message_id: str | None = None
|
||||
|
||||
|
||||
class PlaygroundMessageStopRequest(BaseModel):
|
||||
session_key: str = Field(default="default", min_length=1, max_length=100)
|
||||
message_id: str = Field(..., min_length=1, max_length=64)
|
||||
|
||||
|
||||
class PlaygroundMessageResendRequest(BaseModel):
|
||||
session_key: str = Field(default="default", min_length=1, max_length=100)
|
||||
user_message_id: str = Field(..., min_length=1, max_length=64)
|
||||
|
||||
|
||||
class PlaygroundMessageEditRequest(BaseModel):
|
||||
session_key: str = Field(default="default", min_length=1, max_length=100)
|
||||
user_message_id: str = Field(..., min_length=1, max_length=64)
|
||||
content: str = Field(..., min_length=1)
|
||||
@@ -0,0 +1,5 @@
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class AlertResolutionRequest(BaseModel):
|
||||
resolution: str = Field(..., min_length=1, max_length=1000)
|
||||
@@ -3,9 +3,11 @@ from __future__ import annotations
|
||||
import asyncio
|
||||
|
||||
import httpx
|
||||
from fastapi import HTTPException, status
|
||||
from fastapi import Depends, HTTPException, status
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.config import settings
|
||||
from app.db.session import get_db
|
||||
from app.schemas.ai import (
|
||||
AIProviderStatusResponse,
|
||||
SituationalAnalysisRequest,
|
||||
@@ -14,11 +16,27 @@ from app.schemas.ai import (
|
||||
|
||||
|
||||
class AIProviderClient:
|
||||
def __init__(self) -> None:
|
||||
self.service_url = settings.AI_PROVIDER_SERVICE_URL.rstrip("/")
|
||||
self.service_token = settings.AI_PROVIDER_SERVICE_TOKEN
|
||||
self.timeout = settings.AI_PROVIDER_TIMEOUT_SECONDS
|
||||
self.retry_attempts = max(settings.AI_PROVIDER_RETRY_ATTEMPTS, 1)
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
service_url: str | None = None,
|
||||
service_token: str | None = None,
|
||||
timeout: int | None = None,
|
||||
retry_attempts: int | None = None,
|
||||
llm_config: dict | None = None,
|
||||
) -> None:
|
||||
self.service_url = (
|
||||
service_url if service_url is not None else settings.AI_PROVIDER_SERVICE_URL
|
||||
).rstrip("/")
|
||||
self.service_token = (
|
||||
service_token if service_token is not None else settings.AI_PROVIDER_SERVICE_TOKEN
|
||||
)
|
||||
self.timeout = timeout if timeout is not None else settings.AI_PROVIDER_TIMEOUT_SECONDS
|
||||
self.retry_attempts = max(
|
||||
retry_attempts if retry_attempts is not None else settings.AI_PROVIDER_RETRY_ATTEMPTS,
|
||||
1,
|
||||
)
|
||||
self.llm_config = llm_config or {}
|
||||
|
||||
def _headers(self, request_id: str | None = None) -> dict[str, str]:
|
||||
headers = {"Content-Type": "application/json"}
|
||||
@@ -26,6 +44,19 @@ class AIProviderClient:
|
||||
headers["X-Provider-Token"] = self.service_token
|
||||
if request_id:
|
||||
headers["X-Request-ID"] = request_id
|
||||
llm_header_map = {
|
||||
"provider": "X-AI-Provider",
|
||||
"provider_api": "X-AI-Provider-API",
|
||||
"base_url": "X-AI-Base-URL",
|
||||
"api_key": "X-AI-API-Key",
|
||||
"model": "X-AI-Model",
|
||||
"max_tokens": "X-AI-Max-Tokens",
|
||||
"anthropic_version": "X-AI-Anthropic-Version",
|
||||
}
|
||||
for key, header_name in llm_header_map.items():
|
||||
value = self.llm_config.get(key)
|
||||
if value not in (None, ""):
|
||||
headers[header_name] = str(value)
|
||||
return headers
|
||||
|
||||
async def get_status(self, request_id: str | None = None) -> AIProviderStatusResponse:
|
||||
@@ -105,5 +136,14 @@ class AIProviderClient:
|
||||
)
|
||||
|
||||
|
||||
def get_ai_provider_client() -> AIProviderClient:
|
||||
return AIProviderClient()
|
||||
async def get_ai_provider_client(db: AsyncSession = Depends(get_db)) -> AIProviderClient:
|
||||
from app.api.v1.settings import get_runtime_ai_provider_config
|
||||
|
||||
runtime_config = await get_runtime_ai_provider_config(db)
|
||||
return AIProviderClient(
|
||||
service_url=runtime_config["service_url"],
|
||||
service_token=runtime_config["service_token"],
|
||||
timeout=runtime_config["timeout_seconds"],
|
||||
retry_attempts=runtime_config["retry_attempts"],
|
||||
llm_config=runtime_config.get("llm_config") or {},
|
||||
)
|
||||
@@ -0,0 +1,103 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections import Counter
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.models.alert import Alert, AlertSeverity, AlertStatus
|
||||
from app.schemas.ai import AlertBriefRequest, SituationalAnalysisRequest
|
||||
|
||||
|
||||
def _format_counter(counter: Counter[str], empty_text: str = "无") -> str:
|
||||
if not counter:
|
||||
return empty_text
|
||||
return ",".join(f"{key} {value}" for key, value in counter.items())
|
||||
|
||||
|
||||
async def build_alert_brief_request(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
alert_limit: int = 8,
|
||||
) -> tuple[SituationalAnalysisRequest, list[str], dict[str, Any]]:
|
||||
recent_alerts_result = await db.execute(
|
||||
select(Alert)
|
||||
.order_by(Alert.created_at.desc(), Alert.id.desc())
|
||||
.limit(max(alert_limit, 1))
|
||||
)
|
||||
total_result = await db.execute(select(func.count(Alert.id)))
|
||||
active_result = await db.execute(select(func.count(Alert.id)).where(Alert.status == AlertStatus.ACTIVE))
|
||||
acknowledged_result = await db.execute(
|
||||
select(func.count(Alert.id)).where(Alert.status == AlertStatus.ACKNOWLEDGED)
|
||||
)
|
||||
resolved_result = await db.execute(select(func.count(Alert.id)).where(Alert.status == AlertStatus.RESOLVED))
|
||||
|
||||
recent_alerts = recent_alerts_result.scalars().all()
|
||||
total_alerts = total_result.scalar() or 0
|
||||
active_alerts = active_result.scalar() or 0
|
||||
acknowledged_alerts = acknowledged_result.scalar() or 0
|
||||
resolved_alerts = resolved_result.scalar() or 0
|
||||
|
||||
severity_counts = Counter((item.severity.value if item.severity else "unknown") for item in recent_alerts)
|
||||
status_counts = Counter((item.status.value if item.status else "unknown") for item in recent_alerts)
|
||||
datasource_counts = Counter((item.datasource_name or "未命名数据源") for item in recent_alerts)
|
||||
active_datasource_counts = Counter(
|
||||
(item.datasource_name or "未命名数据源")
|
||||
for item in recent_alerts
|
||||
if item.status == AlertStatus.ACTIVE
|
||||
)
|
||||
|
||||
facts = [
|
||||
f"告警总量 {total_alerts} 条,其中 active {active_alerts} 条、acknowledged {acknowledged_alerts} 条、resolved {resolved_alerts} 条。",
|
||||
f"最近告警严重度分布:{_format_counter(severity_counts)}。",
|
||||
f"最近告警状态分布:{_format_counter(status_counts)}。",
|
||||
f"最近告警数据源分布:{_format_counter(Counter(dict(datasource_counts.most_common(6))))}。",
|
||||
]
|
||||
|
||||
if active_datasource_counts:
|
||||
facts.append(
|
||||
"当前待处理告警主要集中在:"
|
||||
+ _format_counter(Counter(dict(active_datasource_counts.most_common(5))))
|
||||
+ "。"
|
||||
)
|
||||
|
||||
if recent_alerts:
|
||||
facts.append(
|
||||
"最近告警摘录:"
|
||||
+ ";".join(
|
||||
[
|
||||
f"{item.datasource_name or '未命名数据源'} / {item.severity.value if item.severity else '-'} / {item.status.value if item.status else '-'} / {item.message or '-'}"
|
||||
for item in recent_alerts[:6]
|
||||
]
|
||||
)
|
||||
)
|
||||
|
||||
context = {
|
||||
"source": "alerts",
|
||||
"total_alerts": total_alerts,
|
||||
"active_alerts": active_alerts,
|
||||
"acknowledged_alerts": acknowledged_alerts,
|
||||
"resolved_alerts": resolved_alerts,
|
||||
"severity_distribution": dict(severity_counts),
|
||||
"status_distribution": dict(status_counts),
|
||||
"top_datasources": dict(datasource_counts.most_common(6)),
|
||||
"top_active_datasources": dict(active_datasource_counts.most_common(5)),
|
||||
}
|
||||
|
||||
return (
|
||||
SituationalAnalysisRequest(
|
||||
title="告警态势 AI 简报",
|
||||
objective="基于当前告警总量、严重度、状态、数据源分布与最近告警摘录,生成一份面向值班人员的简明告警态势简报,突出待处理风险、告警集中点和优先动作。",
|
||||
observations=facts,
|
||||
constraints=[
|
||||
"明确区分事实、推断与建议。",
|
||||
"优先指出仍处于 active 状态且高严重度的告警簇。",
|
||||
"不要把 acknowledged 或 resolved 告警误判成当前仍在扩大。",
|
||||
"如果证据不足,请明确指出缺失的上下文。",
|
||||
],
|
||||
context=context,
|
||||
),
|
||||
facts,
|
||||
context,
|
||||
)
|
||||
@@ -0,0 +1,209 @@
|
||||
"""BarentsWatch AIS credential resolution and connectivity checks."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import shlex
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
from app.models.datasource_config import DataSourceConfig
|
||||
|
||||
|
||||
BARENTSWATCH_LATEST_URL = "https://live.ais.barentswatch.no/v1/latest/combined"
|
||||
BARENTSWATCH_TOKEN_URL = "https://id.barentswatch.no/connect/token"
|
||||
BARENTSWATCH_DATASOURCE_NAME = "barentswatch_vessels"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class BarentsWatchConfig:
|
||||
endpoint: str
|
||||
client_id: str
|
||||
client_secret: str
|
||||
credential_source: str
|
||||
endpoint_source: str
|
||||
|
||||
|
||||
def _read_zshrc_env(path: Path | None = None) -> dict[str, str]:
|
||||
zshrc_path = path or Path.home() / ".zshrc"
|
||||
if not zshrc_path.exists():
|
||||
return {}
|
||||
|
||||
values: dict[str, str] = {}
|
||||
for raw_line in zshrc_path.read_text(encoding="utf-8", errors="ignore").splitlines():
|
||||
line = raw_line.strip()
|
||||
if not line or line.startswith("#"):
|
||||
continue
|
||||
if line.startswith("export "):
|
||||
line = line[len("export ") :].strip()
|
||||
if "=" not in line:
|
||||
continue
|
||||
|
||||
key, value = line.split("=", 1)
|
||||
key = key.strip()
|
||||
if not key or not key.replace("_", "").isalnum() or not key[0].isalpha():
|
||||
continue
|
||||
|
||||
try:
|
||||
parsed = shlex.split(value, comments=True, posix=True)
|
||||
except ValueError:
|
||||
parsed = [value.strip().strip("'\"")]
|
||||
if parsed:
|
||||
values[key] = parsed[0]
|
||||
return values
|
||||
|
||||
|
||||
def _first_env_value(zshrc_env: dict[str, str], *keys: str) -> tuple[str, str]:
|
||||
for key in keys:
|
||||
value = os.getenv(key)
|
||||
if value:
|
||||
return value, "environment"
|
||||
for key in keys:
|
||||
value = zshrc_env.get(key)
|
||||
if value:
|
||||
return value, "~/.zshrc"
|
||||
return "", ""
|
||||
|
||||
|
||||
async def get_barentswatch_datasource_record(db: AsyncSession) -> DataSourceConfig | None:
|
||||
result = await db.execute(
|
||||
select(DataSourceConfig)
|
||||
.where(DataSourceConfig.name == BARENTSWATCH_DATASOURCE_NAME)
|
||||
.where(DataSourceConfig.is_active.is_(True))
|
||||
)
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
|
||||
async def resolve_barentswatch_config(db: AsyncSession | None = None) -> BarentsWatchConfig:
|
||||
record = await get_barentswatch_datasource_record(db) if db else None
|
||||
auth_config = dict(record.auth_config or {}) if record else {}
|
||||
config = dict(record.config or {}) if record else {}
|
||||
zshrc_env = _read_zshrc_env()
|
||||
|
||||
env_client_id, env_source = _first_env_value(
|
||||
zshrc_env,
|
||||
"BARENTSWATCH_CLIENT_ID",
|
||||
"BARRENTSWATCH_CLIENT_ID",
|
||||
)
|
||||
env_client_secret, secret_env_source = _first_env_value(
|
||||
zshrc_env,
|
||||
"BARENTSWATCH_CLIENT_SECRET",
|
||||
"BARRENTSWATCH_CLIENT_SECRET",
|
||||
)
|
||||
client_id = auth_config.get("client_id") or config.get("client_id") or env_client_id
|
||||
client_secret = (
|
||||
auth_config.get("client_secret") or config.get("client_secret") or env_client_secret
|
||||
)
|
||||
|
||||
credential_source = ""
|
||||
if auth_config.get("client_id") or auth_config.get("client_secret"):
|
||||
credential_source = "datasource_config"
|
||||
elif config.get("client_id") or config.get("client_secret"):
|
||||
credential_source = "datasource_runtime_config"
|
||||
elif env_source or secret_env_source:
|
||||
credential_source = env_source or secret_env_source
|
||||
|
||||
yaml_endpoint = get_data_sources_config().get_yaml_url(BARENTSWATCH_DATASOURCE_NAME)
|
||||
endpoint = record.endpoint if record and record.endpoint else yaml_endpoint
|
||||
return BarentsWatchConfig(
|
||||
endpoint=endpoint or BARENTSWATCH_LATEST_URL,
|
||||
client_id=str(client_id or ""),
|
||||
client_secret=str(client_secret or ""),
|
||||
credential_source=credential_source or "missing",
|
||||
endpoint_source="datasource_config" if record and record.endpoint else "default",
|
||||
)
|
||||
|
||||
|
||||
async def fetch_barentswatch_access_token(
|
||||
client: httpx.AsyncClient,
|
||||
config: BarentsWatchConfig,
|
||||
) -> str | None:
|
||||
if not config.client_id or not config.client_secret:
|
||||
return None
|
||||
|
||||
response = await client.post(
|
||||
BARENTSWATCH_TOKEN_URL,
|
||||
data={
|
||||
"client_id": config.client_id,
|
||||
"client_secret": config.client_secret,
|
||||
"scope": "ais",
|
||||
"grant_type": "client_credentials",
|
||||
},
|
||||
headers={"Content-Type": "application/x-www-form-urlencoded"},
|
||||
)
|
||||
response.raise_for_status()
|
||||
payload = response.json()
|
||||
token = payload.get("access_token")
|
||||
return str(token) if token else None
|
||||
|
||||
|
||||
async def check_barentswatch_connectivity(db: AsyncSession) -> dict[str, Any]:
|
||||
config = await resolve_barentswatch_config(db)
|
||||
return await check_barentswatch_config(config)
|
||||
|
||||
|
||||
async def check_barentswatch_config(config: BarentsWatchConfig) -> dict[str, Any]:
|
||||
if not config.client_id or not config.client_secret:
|
||||
return {
|
||||
"success": False,
|
||||
"stage": "credentials",
|
||||
"message": "未找到 BarentsWatch client id/client secret,请先配置采集器凭证。",
|
||||
"endpoint": config.endpoint,
|
||||
"credential_source": config.credential_source,
|
||||
"settings_tab": "collector_credentials",
|
||||
}
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=20.0) as client:
|
||||
token = await fetch_barentswatch_access_token(client, config)
|
||||
if not token:
|
||||
return {
|
||||
"success": False,
|
||||
"stage": "token",
|
||||
"message": "BarentsWatch token 响应中没有 access_token,请检查凭证。",
|
||||
"endpoint": config.endpoint,
|
||||
"credential_source": config.credential_source,
|
||||
"settings_tab": "collector_credentials",
|
||||
}
|
||||
|
||||
async with client.stream(
|
||||
"GET",
|
||||
config.endpoint,
|
||||
headers={"Authorization": f"Bearer {token}"},
|
||||
) as response:
|
||||
response.raise_for_status()
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"stage": "endpoint",
|
||||
"message": "BarentsWatch AIS token 和数据接口均可连通。",
|
||||
"endpoint": config.endpoint,
|
||||
"credential_source": config.credential_source,
|
||||
"endpoint_source": config.endpoint_source,
|
||||
}
|
||||
except httpx.HTTPStatusError as exc:
|
||||
status_code = exc.response.status_code
|
||||
stage = "token" if str(exc.request.url) == BARENTSWATCH_TOKEN_URL else "endpoint"
|
||||
return {
|
||||
"success": False,
|
||||
"stage": stage,
|
||||
"message": f"BarentsWatch {stage} 请求返回 HTTP {status_code},请检查凭证或接口地址。",
|
||||
"endpoint": config.endpoint,
|
||||
"credential_source": config.credential_source,
|
||||
"settings_tab": "collector_credentials",
|
||||
}
|
||||
except httpx.HTTPError as exc:
|
||||
return {
|
||||
"success": False,
|
||||
"stage": "network",
|
||||
"message": f"BarentsWatch 链路检查失败:{exc.__class__.__name__}",
|
||||
"endpoint": config.endpoint,
|
||||
"credential_source": config.credential_source,
|
||||
"settings_tab": "collector_credentials",
|
||||
}
|
||||
@@ -0,0 +1,259 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections import Counter
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.api.v1.bgp import BGP_SOURCES
|
||||
from app.models.bgp_anomaly import BGPAnomaly
|
||||
from app.models.bgp_incident import BGPIncident
|
||||
from app.models.bgp_observation import BGPObservation
|
||||
from app.schemas.ai import SituationalAnalysisRequest
|
||||
from app.services.bgp_collectors import build_bgp_collector_coverage
|
||||
from app.services.bgp_enrichment import lookup_prefix_geography
|
||||
|
||||
|
||||
def _format_counter(counter: dict[str, int], empty_text: str = "无") -> str:
|
||||
if not counter:
|
||||
return empty_text
|
||||
return ",".join(f"{key} {value}" for key, value in counter.items())
|
||||
|
||||
|
||||
def _severity_rank(value: str | None) -> int:
|
||||
order = {
|
||||
"critical": 0,
|
||||
"high": 1,
|
||||
"medium": 2,
|
||||
"low": 3,
|
||||
"info": 4,
|
||||
}
|
||||
return order.get((value or "").lower(), 99)
|
||||
|
||||
|
||||
def _normalize_geo_key(country: str | None, city: str | None) -> str:
|
||||
if city and country:
|
||||
return f"{city}, {country}"
|
||||
return city or country or "未知区域"
|
||||
|
||||
|
||||
def _top_counter_items(counter: Counter[str], limit: int = 5) -> dict[str, int]:
|
||||
return {name: count for name, count in counter.most_common(limit) if name}
|
||||
|
||||
|
||||
def _collect_incident_regions(incidents: list[BGPIncident]) -> Counter[str]:
|
||||
counter: Counter[str] = Counter()
|
||||
for item in incidents:
|
||||
for region in item.affected_regions or []:
|
||||
if not isinstance(region, dict):
|
||||
continue
|
||||
counter[_normalize_geo_key(region.get("country"), region.get("city"))] += 1
|
||||
return counter
|
||||
|
||||
|
||||
def _collect_collector_regions(collectors: list[dict[str, Any]]) -> Counter[str]:
|
||||
counter: Counter[str] = Counter()
|
||||
for item in collectors:
|
||||
counter[_normalize_geo_key(item.get("country"), item.get("city"))] += int(item.get("recent_24h_observation_count") or 0)
|
||||
return counter
|
||||
|
||||
|
||||
def _format_geo_evidence(prefix_geographies: dict[str, dict[str, Any]], limit: int = 6) -> str:
|
||||
if not prefix_geographies:
|
||||
return "没有命中 prefix geography 证据。"
|
||||
|
||||
rows = []
|
||||
for prefix, item in list(prefix_geographies.items())[:limit]:
|
||||
region = _normalize_geo_key(item.get("country"), item.get("city"))
|
||||
source = item.get("source") or item.get("geography_mode") or "unknown"
|
||||
as_hint = item.get("asn")
|
||||
as_name = item.get("as_name")
|
||||
as_text = ""
|
||||
if as_hint:
|
||||
as_text = f" / ASN AS{as_hint}"
|
||||
if as_name:
|
||||
as_text += f" ({as_name})"
|
||||
rows.append(f"{prefix} -> {region} / 来源 {source}{as_text}")
|
||||
return ";".join(rows)
|
||||
|
||||
|
||||
async def build_bgp_brief_request(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
incident_limit: int = 5,
|
||||
anomaly_limit: int = 6,
|
||||
collector_limit: int = 5,
|
||||
) -> tuple[SituationalAnalysisRequest, list[str], dict[str, int | str | dict[str, int]]]:
|
||||
incidents_result = await db.execute(
|
||||
select(BGPIncident)
|
||||
.order_by(BGPIncident.created_at.desc(), BGPIncident.id.desc())
|
||||
.limit(max(incident_limit, 1))
|
||||
)
|
||||
anomalies_result = await db.execute(
|
||||
select(BGPAnomaly)
|
||||
.order_by(BGPAnomaly.created_at.desc(), BGPAnomaly.id.desc())
|
||||
.limit(max(anomaly_limit, 1))
|
||||
)
|
||||
observations_result = await db.execute(
|
||||
select(BGPObservation).where(BGPObservation.source.in_(BGP_SOURCES))
|
||||
)
|
||||
incident_count_result = await db.execute(select(func.count(BGPIncident.id)))
|
||||
anomaly_count_result = await db.execute(select(func.count(BGPAnomaly.id)))
|
||||
|
||||
incidents = incidents_result.scalars().all()
|
||||
anomalies = anomalies_result.scalars().all()
|
||||
observations = observations_result.scalars().all()
|
||||
collectors = await build_bgp_collector_coverage(db, source_filter=BGP_SOURCES)
|
||||
|
||||
total_incidents = incident_count_result.scalar() or 0
|
||||
total_anomalies = anomaly_count_result.scalar() or 0
|
||||
total_observations = len(observations)
|
||||
active_collectors = [item for item in collectors if item["observation_count"] > 0]
|
||||
|
||||
incident_status_counts = Counter((item.status or "unknown") for item in incidents)
|
||||
incident_severity_counts = Counter((item.severity or "unknown") for item in incidents)
|
||||
incident_type_counts = Counter((item.incident_type or "unknown") for item in incidents)
|
||||
anomaly_type_counts = Counter((item.anomaly_type or "unknown") for item in anomalies)
|
||||
event_type_counts = Counter((item.event_type or "unknown") for item in observations)
|
||||
incident_region_counts = _collect_incident_regions(incidents)
|
||||
|
||||
top_collectors = sorted(
|
||||
active_collectors,
|
||||
key=lambda item: (
|
||||
-int(item["recent_24h_observation_count"]),
|
||||
-int(item["observation_count"]),
|
||||
str(item["collector"]),
|
||||
),
|
||||
)[: max(collector_limit, 1)]
|
||||
collector_region_counts = _collect_collector_regions(top_collectors)
|
||||
|
||||
prefix_candidates = sorted(
|
||||
{
|
||||
prefix
|
||||
for item in incidents
|
||||
for prefix in (item.affected_prefixes or [])
|
||||
if prefix
|
||||
}
|
||||
| {item.prefix for item in anomalies if item.prefix}
|
||||
)
|
||||
prefix_geographies = await lookup_prefix_geography(db, prefix_candidates) if prefix_candidates else {}
|
||||
geography_region_counts = Counter(
|
||||
_normalize_geo_key(item.get("country"), item.get("city"))
|
||||
for item in prefix_geographies.values()
|
||||
if item.get("country") or item.get("city")
|
||||
)
|
||||
hotspot_region_counts = geography_region_counts + incident_region_counts
|
||||
collector_bias_regions = [
|
||||
region
|
||||
for region, count in collector_region_counts.most_common(3)
|
||||
if count > hotspot_region_counts.get(region, 0)
|
||||
]
|
||||
|
||||
observations_lines: list[str] = [
|
||||
f"当前共有 {total_incidents} 起 BGP incidents、{total_anomalies} 条 anomalies、{total_observations} 条原始观测事件。",
|
||||
f"活跃观测站 {len(active_collectors)} 个;近 24 小时事件数合计 {sum(int(item['recent_24h_observation_count']) for item in active_collectors)}。",
|
||||
f"最近 incidents 严重度分布:{_format_counter(dict(sorted(incident_severity_counts.items(), key=lambda item: _severity_rank(item[0]))))}。",
|
||||
f"最近 incidents 状态分布:{_format_counter(dict(incident_status_counts))}。",
|
||||
f"最近 incidents 类型分布:{_format_counter(dict(incident_type_counts.most_common(5)))}。",
|
||||
f"最近 anomalies 类型分布:{_format_counter(dict(anomaly_type_counts.most_common(6)))}。",
|
||||
f"观测事件类型分布:{_format_counter(dict(event_type_counts.most_common(6)))}。",
|
||||
]
|
||||
|
||||
if hotspot_region_counts:
|
||||
observations_lines.append(
|
||||
"区域热点事实层:"
|
||||
+ _format_counter(_top_counter_items(hotspot_region_counts, limit=5), empty_text="无明显区域聚集")
|
||||
+ "。"
|
||||
)
|
||||
|
||||
if prefix_geographies:
|
||||
observations_lines.append("Prefix geography 证据:" + _format_geo_evidence(prefix_geographies))
|
||||
|
||||
if collector_bias_regions:
|
||||
observations_lines.append(
|
||||
"观测偏差提示:重点观测站最近 24h 活跃度更集中在 "
|
||||
+ "、".join(collector_bias_regions)
|
||||
+ ",这些区域的事件升温结论需要结合 prefix geography 与 affected regions 交叉验证。"
|
||||
)
|
||||
elif top_collectors:
|
||||
observations_lines.append(
|
||||
"观测偏差提示:当前未发现明显高于区域热点事实层的单一观测站集中区域,但仍需区分 collector coverage 与真实区域风险。"
|
||||
)
|
||||
|
||||
if incidents:
|
||||
observations_lines.append(
|
||||
"最近 incident 摘要:" + ";".join(
|
||||
[
|
||||
f"{item.incident_type} / {item.severity} / {item.status}"
|
||||
f" / 前缀 {', '.join(item.affected_prefixes[:2]) if item.affected_prefixes else '-'}"
|
||||
f" / 观测站 {len(item.affected_collectors or [])} 个"
|
||||
for item in incidents
|
||||
]
|
||||
)
|
||||
)
|
||||
|
||||
if anomalies:
|
||||
observations_lines.append(
|
||||
"最近 anomaly 摘要:" + ";".join(
|
||||
[
|
||||
f"{item.anomaly_type} / {item.severity}"
|
||||
f" / 前缀 {item.prefix or '-'}"
|
||||
f" / ASN {item.new_origin_asn or item.origin_asn or '-'}"
|
||||
for item in anomalies
|
||||
]
|
||||
)
|
||||
)
|
||||
|
||||
if top_collectors:
|
||||
observations_lines.append(
|
||||
"重点观测站:" + ";".join(
|
||||
[
|
||||
f"{item['collector']} ({', '.join([part for part in [item.get('city'), item.get('country')] if part]) or '未知位置'})"
|
||||
f" / 近24h {item['recent_24h_observation_count']} 条"
|
||||
f" / 前缀 {item['prefix_count']} 个"
|
||||
for item in top_collectors
|
||||
]
|
||||
)
|
||||
)
|
||||
|
||||
context = {
|
||||
"source": "bgp-overview",
|
||||
"incident_total": total_incidents,
|
||||
"anomaly_total": total_anomalies,
|
||||
"observation_total": total_observations,
|
||||
"active_collectors": len(active_collectors),
|
||||
"top_incident_types": dict(incident_type_counts.most_common(5)),
|
||||
"top_anomaly_types": dict(anomaly_type_counts.most_common(6)),
|
||||
"top_event_types": dict(event_type_counts.most_common(6)),
|
||||
"region_hotspots": _top_counter_items(hotspot_region_counts, limit=6),
|
||||
"incident_regions": _top_counter_items(incident_region_counts, limit=6),
|
||||
"collector_bias_regions": collector_bias_regions,
|
||||
"prefix_geography_sources": dict(
|
||||
Counter(str(item.get("source") or "unknown") for item in prefix_geographies.values()).most_common(5)
|
||||
),
|
||||
"prefix_geography_sample": {
|
||||
prefix: {
|
||||
"country": item.get("country"),
|
||||
"city": item.get("city"),
|
||||
"source": item.get("source"),
|
||||
"asn": item.get("asn"),
|
||||
"as_name": item.get("as_name"),
|
||||
}
|
||||
for prefix, item in list(prefix_geographies.items())[:8]
|
||||
},
|
||||
}
|
||||
|
||||
return SituationalAnalysisRequest(
|
||||
title="BGP 态势 AI 简报",
|
||||
objective="基于当前 BGP incidents、anomalies、原始观测事件、观测站覆盖与 prefix geography 证据,生成一份面向操作员的简明态势简报,突出区域热点、观测偏差、当前风险、证据和优先动作。",
|
||||
observations=observations_lines,
|
||||
constraints=[
|
||||
"明确区分事实、推断与建议。",
|
||||
"优先指出需要立即关注的高严重度 incident 或异常模式。",
|
||||
"需要单独指出哪些区域结论来自 prefix geography / affected regions,哪些可能受 collector coverage 偏差影响。",
|
||||
"结论应服务值班排障,不要写成泛泛的模型演示文案。",
|
||||
"如果证据不足,要明确指出缺失数据。",
|
||||
],
|
||||
context=context,
|
||||
), observations_lines, context
|
||||
@@ -0,0 +1,160 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from uuid import uuid4
|
||||
|
||||
from app.core.config import ROOT_DIR
|
||||
from app.schemas.ai import BGPBriefRecordResponse, BGPBriefRecordSummary, SituationalAnalysisResponse
|
||||
|
||||
|
||||
_BRIEF_STORAGE_DIR = ROOT_DIR / "data" / "ai" / "bgp-briefs"
|
||||
_METADATA_PREFIX = "<!-- planet-bgp-brief-meta "
|
||||
_METADATA_SUFFIX = " -->"
|
||||
_BRIEF_TITLE = "BGP AI 简报"
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class _StoredBrief:
|
||||
id: str
|
||||
title: str
|
||||
provider: str
|
||||
model: str
|
||||
request_id: str | None
|
||||
generated_at: str
|
||||
content_markdown: str
|
||||
facts: list[str]
|
||||
context: dict[str, Any]
|
||||
path: Path
|
||||
|
||||
|
||||
def _ensure_storage_dir() -> Path:
|
||||
_BRIEF_STORAGE_DIR.mkdir(parents=True, exist_ok=True)
|
||||
return _BRIEF_STORAGE_DIR
|
||||
|
||||
|
||||
def _build_metadata_line(metadata: dict[str, Any]) -> str:
|
||||
return f"{_METADATA_PREFIX}{json.dumps(metadata, ensure_ascii=False)}{_METADATA_SUFFIX}"
|
||||
|
||||
|
||||
def _parse_brief_file(path: Path) -> _StoredBrief | None:
|
||||
try:
|
||||
raw_text = path.read_text(encoding="utf-8")
|
||||
except OSError:
|
||||
return None
|
||||
|
||||
first_line, separator, remainder = raw_text.partition("\n")
|
||||
if not separator or not first_line.startswith(_METADATA_PREFIX) or not first_line.endswith(_METADATA_SUFFIX):
|
||||
return None
|
||||
|
||||
metadata_payload = first_line[len(_METADATA_PREFIX) : -len(_METADATA_SUFFIX)]
|
||||
|
||||
try:
|
||||
metadata = json.loads(metadata_payload)
|
||||
except json.JSONDecodeError:
|
||||
return None
|
||||
|
||||
return _StoredBrief(
|
||||
id=str(metadata.get("id") or path.stem),
|
||||
title=str(metadata.get("title") or _BRIEF_TITLE),
|
||||
provider=str(metadata.get("provider") or "-"),
|
||||
model=str(metadata.get("model") or "-"),
|
||||
request_id=metadata.get("request_id"),
|
||||
generated_at=str(metadata.get("generated_at") or datetime.fromtimestamp(path.stat().st_mtime, UTC).isoformat()),
|
||||
content_markdown=remainder.lstrip("\n"),
|
||||
facts=list(metadata.get("facts") or []),
|
||||
context=dict(metadata.get("context") or {}),
|
||||
path=path,
|
||||
)
|
||||
|
||||
|
||||
def list_bgp_brief_records(limit: int = 50) -> list[BGPBriefRecordSummary]:
|
||||
storage_dir = _ensure_storage_dir()
|
||||
records: list[_StoredBrief] = []
|
||||
|
||||
for path in storage_dir.glob("*.md"):
|
||||
parsed = _parse_brief_file(path)
|
||||
if parsed is not None:
|
||||
records.append(parsed)
|
||||
|
||||
records.sort(key=lambda item: item.generated_at, reverse=True)
|
||||
|
||||
return [
|
||||
BGPBriefRecordSummary(
|
||||
id=item.id,
|
||||
title=item.title,
|
||||
provider=item.provider,
|
||||
model=item.model,
|
||||
request_id=item.request_id,
|
||||
generated_at=item.generated_at,
|
||||
)
|
||||
for item in records[: max(limit, 1)]
|
||||
]
|
||||
|
||||
|
||||
def get_bgp_brief_record(brief_id: str) -> BGPBriefRecordResponse | None:
|
||||
path = _ensure_storage_dir() / f"{brief_id}.md"
|
||||
parsed = _parse_brief_file(path)
|
||||
if parsed is None:
|
||||
return None
|
||||
|
||||
return BGPBriefRecordResponse(
|
||||
id=parsed.id,
|
||||
title=parsed.title,
|
||||
provider=parsed.provider,
|
||||
model=parsed.model,
|
||||
request_id=parsed.request_id,
|
||||
generated_at=parsed.generated_at,
|
||||
content_markdown=parsed.content_markdown,
|
||||
facts=parsed.facts,
|
||||
context=parsed.context,
|
||||
)
|
||||
|
||||
|
||||
def get_latest_bgp_brief_record() -> BGPBriefRecordResponse | None:
|
||||
summaries = list_bgp_brief_records(limit=1)
|
||||
if not summaries:
|
||||
return None
|
||||
return get_bgp_brief_record(summaries[0].id)
|
||||
|
||||
|
||||
def save_bgp_brief_record(
|
||||
analysis: SituationalAnalysisResponse,
|
||||
*,
|
||||
request_id: str | None,
|
||||
facts: list[str] | None = None,
|
||||
context: dict[str, Any] | None = None,
|
||||
generated_at: datetime | None = None,
|
||||
) -> BGPBriefRecordResponse:
|
||||
created_at = generated_at or datetime.now(UTC)
|
||||
brief_id = f"{created_at.strftime('%Y%m%dT%H%M%SZ')}-{uuid4().hex[:8]}"
|
||||
path = _ensure_storage_dir() / f"{brief_id}.md"
|
||||
|
||||
metadata = {
|
||||
"id": brief_id,
|
||||
"title": _BRIEF_TITLE,
|
||||
"provider": analysis.provider,
|
||||
"model": analysis.model,
|
||||
"request_id": request_id,
|
||||
"generated_at": created_at.isoformat(),
|
||||
"facts": facts or [],
|
||||
"context": context or {},
|
||||
}
|
||||
|
||||
markdown_text = f"{_build_metadata_line(metadata)}\n\n{analysis.content.rstrip()}\n"
|
||||
path.write_text(markdown_text, encoding="utf-8")
|
||||
|
||||
return BGPBriefRecordResponse(
|
||||
id=brief_id,
|
||||
title=_BRIEF_TITLE,
|
||||
provider=analysis.provider,
|
||||
model=analysis.model,
|
||||
request_id=request_id,
|
||||
generated_at=created_at.isoformat(),
|
||||
content_markdown=analysis.content,
|
||||
facts=facts or [],
|
||||
context=context or {},
|
||||
)
|
||||
@@ -6,7 +6,7 @@ from collections import defaultdict
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy import case, distinct, func, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.time import to_iso8601_utc
|
||||
@@ -14,6 +14,16 @@ from app.models.bgp_observation import BGPObservation
|
||||
from app.services.collectors.bgp_common import RIPE_RIS_COLLECTOR_COORDS
|
||||
|
||||
|
||||
def _collector_base_filters(source_filter: tuple[str, ...] | None) -> list[Any]:
|
||||
filters: list[Any] = [
|
||||
BGPObservation.collector.isnot(None),
|
||||
func.length(func.btrim(BGPObservation.collector)) > 0,
|
||||
]
|
||||
if source_filter:
|
||||
filters.append(BGPObservation.source.in_(source_filter))
|
||||
return filters
|
||||
|
||||
|
||||
async def build_bgp_collector_coverage(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
@@ -24,88 +34,148 @@ async def build_bgp_collector_coverage(
|
||||
recent_24h_threshold = now - timedelta(hours=24)
|
||||
recent_7d_threshold = now - timedelta(days=7)
|
||||
|
||||
stmt = select(BGPObservation).order_by(BGPObservation.observed_at.desc(), BGPObservation.id.desc())
|
||||
if source_filter:
|
||||
stmt = stmt.where(BGPObservation.source.in_(source_filter))
|
||||
filters = _collector_base_filters(source_filter)
|
||||
country_expr = func.nullif(BGPObservation.collector_geo["country"].as_string(), "")
|
||||
city_expr = func.nullif(BGPObservation.collector_geo["city"].as_string(), "")
|
||||
|
||||
result = await db.execute(stmt)
|
||||
records = list(result.scalars().all())
|
||||
aggregate_stmt = (
|
||||
select(
|
||||
BGPObservation.collector.label("collector"),
|
||||
func.count(BGPObservation.id).label("observation_count"),
|
||||
func.count(distinct(BGPObservation.prefix)).label("prefix_count"),
|
||||
func.count(distinct(BGPObservation.origin_asn)).label("origin_asn_count"),
|
||||
func.count(distinct(BGPObservation.peer_asn)).label("peer_asn_count"),
|
||||
func.sum(case((BGPObservation.observed_at >= recent_15m_threshold, 1), else_=0)).label("recent_15m_observation_count"),
|
||||
func.sum(case((BGPObservation.observed_at >= recent_24h_threshold, 1), else_=0)).label("recent_24h_observation_count"),
|
||||
func.sum(case((BGPObservation.observed_at >= recent_7d_threshold, 1), else_=0)).label("recent_7d_observation_count"),
|
||||
func.count(distinct(case((BGPObservation.observed_at >= recent_15m_threshold, BGPObservation.prefix), else_=None))).label("recent_15m_prefix_count"),
|
||||
func.count(distinct(case((BGPObservation.observed_at >= recent_24h_threshold, BGPObservation.prefix), else_=None))).label("recent_24h_prefix_count"),
|
||||
func.count(distinct(case((BGPObservation.observed_at >= recent_7d_threshold, BGPObservation.prefix), else_=None))).label("recent_7d_prefix_count"),
|
||||
func.max(BGPObservation.observed_at).label("latest_observed_at"),
|
||||
)
|
||||
.where(*filters)
|
||||
.group_by(BGPObservation.collector)
|
||||
)
|
||||
aggregate_rows = (await db.execute(aggregate_stmt)).all()
|
||||
|
||||
latest_subquery = (
|
||||
select(
|
||||
BGPObservation.collector.label("collector"),
|
||||
BGPObservation.event_type.label("latest_event_type"),
|
||||
country_expr.label("country"),
|
||||
city_expr.label("city"),
|
||||
func.row_number()
|
||||
.over(
|
||||
partition_by=BGPObservation.collector,
|
||||
order_by=(BGPObservation.observed_at.desc(), BGPObservation.id.desc()),
|
||||
)
|
||||
.label("rn"),
|
||||
)
|
||||
.where(*filters)
|
||||
.subquery()
|
||||
)
|
||||
latest_rows = (
|
||||
await db.execute(
|
||||
select(
|
||||
latest_subquery.c.collector,
|
||||
latest_subquery.c.latest_event_type,
|
||||
latest_subquery.c.country,
|
||||
latest_subquery.c.city,
|
||||
).where(latest_subquery.c.rn == 1)
|
||||
)
|
||||
).all()
|
||||
|
||||
event_counts_subquery = (
|
||||
select(
|
||||
BGPObservation.collector.label("collector"),
|
||||
BGPObservation.event_type.label("event_type"),
|
||||
func.count(BGPObservation.id).label("count"),
|
||||
func.row_number()
|
||||
.over(
|
||||
partition_by=BGPObservation.collector,
|
||||
order_by=(func.count(BGPObservation.id).desc(), BGPObservation.event_type.asc()),
|
||||
)
|
||||
.label("rn"),
|
||||
)
|
||||
.where(*filters)
|
||||
.group_by(BGPObservation.collector, BGPObservation.event_type)
|
||||
.subquery()
|
||||
)
|
||||
top_event_rows = (
|
||||
await db.execute(
|
||||
select(
|
||||
event_counts_subquery.c.collector,
|
||||
event_counts_subquery.c.event_type,
|
||||
event_counts_subquery.c.count,
|
||||
).where(event_counts_subquery.c.rn <= 3)
|
||||
)
|
||||
).all()
|
||||
|
||||
scope_rows = (
|
||||
await db.execute(
|
||||
select(
|
||||
BGPObservation.collector.label("collector"),
|
||||
country_expr.label("country"),
|
||||
city_expr.label("city"),
|
||||
)
|
||||
.where(*filters)
|
||||
.distinct()
|
||||
)
|
||||
).all()
|
||||
|
||||
latest_by_collector = {
|
||||
row.collector: {
|
||||
"latest_event_type": row.latest_event_type,
|
||||
"country": row.country,
|
||||
"city": row.city,
|
||||
}
|
||||
for row in latest_rows
|
||||
}
|
||||
|
||||
scope_by_collector: dict[str, dict[str, set[str]]] = defaultdict(lambda: {"countries": set(), "cities": set()})
|
||||
for row in scope_rows:
|
||||
if row.country:
|
||||
scope_by_collector[row.collector]["countries"].add(row.country)
|
||||
if row.city:
|
||||
scope_by_collector[row.collector]["cities"].add(row.city)
|
||||
|
||||
top_events_by_collector: dict[str, list[dict[str, Any]]] = defaultdict(list)
|
||||
for row in top_event_rows:
|
||||
top_events_by_collector[row.collector].append(
|
||||
{"event_type": row.event_type, "count": row.count}
|
||||
)
|
||||
|
||||
by_collector: dict[str, dict[str, Any]] = {}
|
||||
for record in records:
|
||||
collector = str(record.collector or "").strip()
|
||||
if not collector:
|
||||
continue
|
||||
for row in aggregate_rows:
|
||||
collector = row.collector
|
||||
latest = latest_by_collector.get(collector, {})
|
||||
fallback_location = RIPE_RIS_COLLECTOR_COORDS.get(collector, {})
|
||||
scope = scope_by_collector.get(collector, {"countries": set(), "cities": set()})
|
||||
|
||||
coverage = by_collector.get(collector)
|
||||
if coverage is None:
|
||||
location = record.collector_geo or RIPE_RIS_COLLECTOR_COORDS.get(collector, {})
|
||||
coverage = {
|
||||
"collector": collector,
|
||||
"city": location.get("city"),
|
||||
"country": location.get("country"),
|
||||
"latitude": location.get("latitude"),
|
||||
"longitude": location.get("longitude"),
|
||||
"observation_count": 0,
|
||||
"prefixes": set(),
|
||||
"origin_asns": set(),
|
||||
"peer_asns": set(),
|
||||
"event_types": defaultdict(int),
|
||||
"countries": set(),
|
||||
"cities": set(),
|
||||
"recent_15m_observation_count": 0,
|
||||
"recent_24h_observation_count": 0,
|
||||
"recent_7d_observation_count": 0,
|
||||
"recent_15m_prefixes": set(),
|
||||
"recent_24h_prefixes": set(),
|
||||
"recent_7d_prefixes": set(),
|
||||
"latest_observed_at": None,
|
||||
"latest_event_type": None,
|
||||
}
|
||||
by_collector[collector] = coverage
|
||||
|
||||
coverage["observation_count"] += 1
|
||||
if record.prefix:
|
||||
coverage["prefixes"].add(record.prefix)
|
||||
if record.origin_asn is not None:
|
||||
coverage["origin_asns"].add(record.origin_asn)
|
||||
if record.peer_asn is not None:
|
||||
coverage["peer_asns"].add(record.peer_asn)
|
||||
if record.event_type:
|
||||
coverage["event_types"][record.event_type] += 1
|
||||
|
||||
observed_at = record.observed_at
|
||||
if observed_at is not None:
|
||||
aware_observed_at = (
|
||||
observed_at.astimezone(UTC)
|
||||
if observed_at.tzinfo
|
||||
else observed_at.replace(tzinfo=UTC)
|
||||
)
|
||||
if aware_observed_at >= recent_15m_threshold:
|
||||
coverage["recent_15m_observation_count"] += 1
|
||||
if record.prefix:
|
||||
coverage["recent_15m_prefixes"].add(record.prefix)
|
||||
if aware_observed_at >= recent_24h_threshold:
|
||||
coverage["recent_24h_observation_count"] += 1
|
||||
if record.prefix:
|
||||
coverage["recent_24h_prefixes"].add(record.prefix)
|
||||
if aware_observed_at >= recent_7d_threshold:
|
||||
coverage["recent_7d_observation_count"] += 1
|
||||
if record.prefix:
|
||||
coverage["recent_7d_prefixes"].add(record.prefix)
|
||||
|
||||
geo = record.collector_geo or {}
|
||||
if geo.get("country"):
|
||||
coverage["countries"].add(geo["country"])
|
||||
if geo.get("city"):
|
||||
coverage["cities"].add(geo["city"])
|
||||
|
||||
current_latest = coverage["latest_observed_at"]
|
||||
if current_latest is None or (
|
||||
record.observed_at is not None and record.observed_at > current_latest
|
||||
):
|
||||
coverage["latest_observed_at"] = record.observed_at
|
||||
coverage["latest_event_type"] = record.event_type
|
||||
by_collector[collector] = {
|
||||
"collector": collector,
|
||||
"city": latest.get("city") or fallback_location.get("city"),
|
||||
"country": latest.get("country") or fallback_location.get("country"),
|
||||
"latitude": fallback_location.get("latitude"),
|
||||
"longitude": fallback_location.get("longitude"),
|
||||
"observation_count": row.observation_count or 0,
|
||||
"prefix_count": row.prefix_count or 0,
|
||||
"origin_asn_count": row.origin_asn_count or 0,
|
||||
"peer_asn_count": row.peer_asn_count or 0,
|
||||
"recent_15m_observation_count": row.recent_15m_observation_count or 0,
|
||||
"recent_24h_observation_count": row.recent_24h_observation_count or 0,
|
||||
"recent_7d_observation_count": row.recent_7d_observation_count or 0,
|
||||
"recent_15m_prefix_count": row.recent_15m_prefix_count or 0,
|
||||
"recent_24h_prefix_count": row.recent_24h_prefix_count or 0,
|
||||
"recent_7d_prefix_count": row.recent_7d_prefix_count or 0,
|
||||
"top_event_types": top_events_by_collector.get(collector, []),
|
||||
"latest_observed_at": to_iso8601_utc(row.latest_observed_at),
|
||||
"latest_event_type": latest.get("latest_event_type"),
|
||||
"baseline_scope": {
|
||||
"countries": sorted(scope["countries"]),
|
||||
"cities": sorted(scope["cities"]),
|
||||
},
|
||||
}
|
||||
|
||||
for collector, location in RIPE_RIS_COLLECTOR_COORDS.items():
|
||||
if collector in by_collector:
|
||||
@@ -117,57 +187,22 @@ async def build_bgp_collector_coverage(
|
||||
"latitude": location.get("latitude"),
|
||||
"longitude": location.get("longitude"),
|
||||
"observation_count": 0,
|
||||
"prefixes": set(),
|
||||
"origin_asns": set(),
|
||||
"peer_asns": set(),
|
||||
"event_types": defaultdict(int),
|
||||
"countries": {location.get("country")} if location.get("country") else set(),
|
||||
"cities": {location.get("city")} if location.get("city") else set(),
|
||||
"prefix_count": 0,
|
||||
"origin_asn_count": 0,
|
||||
"peer_asn_count": 0,
|
||||
"recent_15m_observation_count": 0,
|
||||
"recent_24h_observation_count": 0,
|
||||
"recent_7d_observation_count": 0,
|
||||
"recent_15m_prefixes": set(),
|
||||
"recent_24h_prefixes": set(),
|
||||
"recent_7d_prefixes": set(),
|
||||
"recent_15m_prefix_count": 0,
|
||||
"recent_24h_prefix_count": 0,
|
||||
"recent_7d_prefix_count": 0,
|
||||
"top_event_types": [],
|
||||
"latest_observed_at": None,
|
||||
"latest_event_type": None,
|
||||
"baseline_scope": {
|
||||
"countries": [location["country"]] if location.get("country") else [],
|
||||
"cities": [location["city"]] if location.get("city") else [],
|
||||
},
|
||||
}
|
||||
|
||||
results: list[dict[str, Any]] = []
|
||||
for collector in sorted(by_collector.keys()):
|
||||
item = by_collector[collector]
|
||||
top_event_types = sorted(
|
||||
item["event_types"].items(),
|
||||
key=lambda pair: (-pair[1], pair[0]),
|
||||
)
|
||||
results.append(
|
||||
{
|
||||
"collector": item["collector"],
|
||||
"city": item["city"],
|
||||
"country": item["country"],
|
||||
"latitude": item["latitude"],
|
||||
"longitude": item["longitude"],
|
||||
"observation_count": item["observation_count"],
|
||||
"prefix_count": len(item["prefixes"]),
|
||||
"origin_asn_count": len(item["origin_asns"]),
|
||||
"peer_asn_count": len(item["peer_asns"]),
|
||||
"recent_15m_observation_count": item["recent_15m_observation_count"],
|
||||
"recent_24h_observation_count": item["recent_24h_observation_count"],
|
||||
"recent_7d_observation_count": item["recent_7d_observation_count"],
|
||||
"recent_15m_prefix_count": len(item["recent_15m_prefixes"]),
|
||||
"recent_24h_prefix_count": len(item["recent_24h_prefixes"]),
|
||||
"recent_7d_prefix_count": len(item["recent_7d_prefixes"]),
|
||||
"top_event_types": [
|
||||
{"event_type": event_type, "count": count}
|
||||
for event_type, count in top_event_types[:3]
|
||||
],
|
||||
"latest_observed_at": to_iso8601_utc(item["latest_observed_at"]),
|
||||
"latest_event_type": item["latest_event_type"],
|
||||
"baseline_scope": {
|
||||
"countries": sorted(country for country in item["countries"] if country),
|
||||
"cities": sorted(city for city in item["cities"] if city),
|
||||
},
|
||||
}
|
||||
)
|
||||
|
||||
return results
|
||||
return [by_collector[collector] for collector in sorted(by_collector.keys())]
|
||||
@@ -7,7 +7,7 @@ from collections import defaultdict
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import select, text
|
||||
from sqlalchemy import Integer, cast, select, text
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.countries import get_country_centroid, normalize_country
|
||||
@@ -231,6 +231,13 @@ async def _lookup_prefix_geography(
|
||||
return results
|
||||
|
||||
|
||||
async def lookup_prefix_geography(
|
||||
db: AsyncSession,
|
||||
prefix_values: list[str],
|
||||
) -> dict[str, dict[str, Any]]:
|
||||
return await _lookup_prefix_geography(db, prefix_values)
|
||||
|
||||
|
||||
async def enrich_bgp_events_for_batch(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
@@ -261,29 +268,40 @@ async def enrich_bgp_events_for_batch(
|
||||
historical_prefix_baseline: dict[str, dict[str, Any]] = {}
|
||||
if prefix_values:
|
||||
previous_result = await db.execute(
|
||||
select(BGPObservation).where(
|
||||
select(
|
||||
BGPObservation.prefix,
|
||||
BGPObservation.origin_asn,
|
||||
BGPObservation.collector,
|
||||
BGPObservation.collector_geo,
|
||||
).where(
|
||||
BGPObservation.source == source,
|
||||
BGPObservation.prefix.in_(prefix_values),
|
||||
)
|
||||
)
|
||||
by_prefix: defaultdict[str, list[BGPObservation]] = defaultdict(list)
|
||||
for observation in previous_result.scalars().all():
|
||||
if observation.prefix:
|
||||
by_prefix[observation.prefix].append(observation)
|
||||
by_prefix: defaultdict[str, list[dict[str, Any]]] = defaultdict(list)
|
||||
for prefix, origin_asn, collector, collector_geo in previous_result.all():
|
||||
if prefix:
|
||||
by_prefix[str(prefix)].append(
|
||||
{
|
||||
"origin_asn": origin_asn,
|
||||
"collector": collector,
|
||||
"collector_geo": collector_geo or {},
|
||||
}
|
||||
)
|
||||
|
||||
for prefix, observations in by_prefix.items():
|
||||
unique_origins = sorted(
|
||||
{
|
||||
observation.origin_asn
|
||||
observation["origin_asn"]
|
||||
for observation in observations
|
||||
if observation.origin_asn is not None
|
||||
if observation["origin_asn"] is not None
|
||||
}
|
||||
)
|
||||
unique_collectors = sorted(
|
||||
{
|
||||
observation.collector
|
||||
observation["collector"]
|
||||
for observation in observations
|
||||
if observation.collector
|
||||
if observation["collector"]
|
||||
}
|
||||
)
|
||||
historical_prefix_baseline[prefix] = {
|
||||
@@ -292,9 +310,9 @@ async def enrich_bgp_events_for_batch(
|
||||
"historical_observation_count": len(observations),
|
||||
"historical_regions": _compact_locations(
|
||||
[
|
||||
observation.collector_geo or {}
|
||||
observation["collector_geo"] or {}
|
||||
for observation in observations
|
||||
if observation.collector_geo
|
||||
if observation["collector_geo"]
|
||||
]
|
||||
),
|
||||
}
|
||||
@@ -303,7 +321,13 @@ async def enrich_bgp_events_for_batch(
|
||||
prefix_geographies = await _lookup_prefix_geography(db, prefix_values) if prefix_values else {}
|
||||
if origin_asns:
|
||||
peeringdb_result = await db.execute(
|
||||
select(CollectedData).where(CollectedData.source == "peeringdb_network")
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source == "peeringdb_network")
|
||||
.where(CollectedData.is_current.is_(True))
|
||||
.where(
|
||||
cast(CollectedData.extra_data["asn"].as_string(), Integer).in_(origin_asns),
|
||||
)
|
||||
.order_by(CollectedData.id.desc())
|
||||
)
|
||||
for record in peeringdb_result.scalars().all():
|
||||
metadata = record.extra_data or {}
|
||||
|
||||
@@ -48,14 +48,36 @@ def _collector_regions_from_anomaly(anomaly: BGPAnomaly) -> list[dict]:
|
||||
return collected
|
||||
|
||||
|
||||
def _dedupe_collected_records(records: list[CollectedData]) -> list[CollectedData]:
|
||||
latest_by_key: dict[str, CollectedData] = {}
|
||||
for record in records:
|
||||
dedupe_key = str(record.source_id or record.entity_key or record.name or record.id)
|
||||
existing = latest_by_key.get(dedupe_key)
|
||||
if existing is None or (record.id or 0) > (existing.id or 0):
|
||||
latest_by_key[dedupe_key] = record
|
||||
return list(latest_by_key.values())
|
||||
async def _load_current_infrastructure_records(
|
||||
db: AsyncSession,
|
||||
) -> tuple[list[CollectedData], list[CollectedData], list[CollectedData]]:
|
||||
result = await db.execute(
|
||||
select(CollectedData)
|
||||
.where(
|
||||
CollectedData.source.in_(
|
||||
(
|
||||
"arcgis_landing_points",
|
||||
"arcgis_cable_landing_relation",
|
||||
"arcgis_cables",
|
||||
)
|
||||
)
|
||||
)
|
||||
.where(CollectedData.is_current.is_(True))
|
||||
.order_by(CollectedData.source.asc(), CollectedData.id.desc())
|
||||
)
|
||||
grouped_records = {
|
||||
"arcgis_landing_points": [],
|
||||
"arcgis_cable_landing_relation": [],
|
||||
"arcgis_cables": [],
|
||||
}
|
||||
for record in result.scalars().all():
|
||||
grouped_records.setdefault(record.source, []).append(record)
|
||||
|
||||
return (
|
||||
grouped_records["arcgis_landing_points"],
|
||||
grouped_records["arcgis_cable_landing_relation"],
|
||||
grouped_records["arcgis_cables"],
|
||||
)
|
||||
|
||||
|
||||
async def infer_related_infrastructure(
|
||||
@@ -75,19 +97,9 @@ async def infer_related_infrastructure(
|
||||
if not valid_regions:
|
||||
return {"related_cables": [], "related_ixps": []}
|
||||
|
||||
landing_result = await db.execute(
|
||||
select(CollectedData).where(CollectedData.source == "arcgis_landing_points")
|
||||
landing_records, relation_records, cable_records = await _load_current_infrastructure_records(
|
||||
db,
|
||||
)
|
||||
relation_result = await db.execute(
|
||||
select(CollectedData).where(CollectedData.source == "arcgis_cable_landing_relation")
|
||||
)
|
||||
cable_result = await db.execute(
|
||||
select(CollectedData).where(CollectedData.source == "arcgis_cables")
|
||||
)
|
||||
|
||||
landing_records = _dedupe_collected_records(list(landing_result.scalars().all()))
|
||||
relation_records = _dedupe_collected_records(list(relation_result.scalars().all()))
|
||||
cable_records = _dedupe_collected_records(list(cable_result.scalars().all()))
|
||||
|
||||
city_to_cable_ids: dict[int, list[int]] = {}
|
||||
for relation in relation_records:
|
||||
|
||||
@@ -35,6 +35,8 @@ from app.services.collectors.bgpstream import BGPStreamBackfillCollector
|
||||
from app.services.collectors.iptoasn import IPtoASNPrefixGeoCollector
|
||||
from app.services.collectors.opengeofeed import OpenGeoFeedPrefixGeoCollector
|
||||
from app.services.collectors.nro_delegated import NRODelegatedPrefixGeoCollector
|
||||
from app.services.collectors.news_live_streams import NewsLiveStreamsCollector
|
||||
from app.services.collectors.vessel_ais import VesselAISCollector
|
||||
|
||||
collector_registry.register(TOP500Collector())
|
||||
collector_registry.register(EpochAIGPUCollector())
|
||||
@@ -61,3 +63,5 @@ collector_registry.register(BGPStreamBackfillCollector())
|
||||
collector_registry.register(IPtoASNPrefixGeoCollector())
|
||||
collector_registry.register(OpenGeoFeedPrefixGeoCollector())
|
||||
collector_registry.register(NRODelegatedPrefixGeoCollector())
|
||||
collector_registry.register(NewsLiveStreamsCollector())
|
||||
collector_registry.register(VesselAISCollector())
|
||||
@@ -1,5 +1,6 @@
|
||||
"""Base collector class for all data sources"""
|
||||
|
||||
import asyncio
|
||||
from abc import ABC, abstractmethod
|
||||
from typing import Dict, List, Any, Optional
|
||||
from datetime import UTC, datetime
|
||||
@@ -166,6 +167,59 @@ class BaseCollector(ABC):
|
||||
await db.commit()
|
||||
return snapshot.id
|
||||
|
||||
async def _rollback_incomplete_run(
|
||||
self,
|
||||
db: AsyncSession,
|
||||
*,
|
||||
task_id: int,
|
||||
snapshot_id: Optional[int],
|
||||
reason: str,
|
||||
) -> None:
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.models.data_snapshot import DataSnapshot
|
||||
|
||||
await db.execute(CollectedData.__table__.delete().where(CollectedData.task_id == task_id))
|
||||
|
||||
parent_snapshot_id: Optional[int] = None
|
||||
if snapshot_id is not None:
|
||||
snapshot = await db.get(DataSnapshot, snapshot_id)
|
||||
if snapshot:
|
||||
parent_snapshot_id = snapshot.parent_snapshot_id
|
||||
snapshot.status = "cancelled"
|
||||
snapshot.is_current = False
|
||||
snapshot.completed_at = datetime.now(UTC)
|
||||
summary = dict(snapshot.summary or {})
|
||||
summary["rollback"] = True
|
||||
summary["rollback_reason"] = reason
|
||||
snapshot.summary = summary
|
||||
|
||||
await db.execute(
|
||||
text(
|
||||
"""
|
||||
UPDATE collected_data
|
||||
SET is_current = FALSE
|
||||
WHERE source = :source
|
||||
"""
|
||||
),
|
||||
{"source": self.name},
|
||||
)
|
||||
|
||||
if parent_snapshot_id is not None:
|
||||
parent_snapshot = await db.get(DataSnapshot, parent_snapshot_id)
|
||||
if parent_snapshot:
|
||||
parent_snapshot.is_current = True
|
||||
|
||||
await db.execute(
|
||||
text(
|
||||
"""
|
||||
UPDATE collected_data
|
||||
SET is_current = TRUE
|
||||
WHERE snapshot_id = :snapshot_id
|
||||
"""
|
||||
),
|
||||
{"snapshot_id": parent_snapshot_id},
|
||||
)
|
||||
|
||||
async def run(self, db: AsyncSession) -> Dict[str, Any]:
|
||||
"""Full pipeline: fetch -> transform -> save"""
|
||||
from app.services.collectors.registry import collector_registry
|
||||
@@ -227,7 +281,24 @@ class BaseCollector(ABC):
|
||||
"records_processed": records_count,
|
||||
"execution_time_seconds": (datetime.now(UTC) - start_time).total_seconds(),
|
||||
}
|
||||
except asyncio.CancelledError:
|
||||
await db.rollback()
|
||||
task.status = "cancelled"
|
||||
task.phase = "cancelled"
|
||||
task.error_message = "Collection cancelled by operator and rolled back"
|
||||
task.completed_at = datetime.now(UTC)
|
||||
if snapshot_id is not None:
|
||||
await self._rollback_incomplete_run(
|
||||
db,
|
||||
task_id=task_id,
|
||||
snapshot_id=snapshot_id,
|
||||
reason="cancelled_by_operator",
|
||||
)
|
||||
await db.commit()
|
||||
await self._publish_task_update(force=True)
|
||||
raise
|
||||
except Exception as e:
|
||||
await db.rollback()
|
||||
task.status = "failed"
|
||||
task.phase = "failed"
|
||||
task.error_message = str(e)
|
||||
@@ -276,20 +347,34 @@ class BaseCollector(ABC):
|
||||
updated_count = 0
|
||||
unchanged_count = 0
|
||||
seen_entity_keys: set[str] = set()
|
||||
previous_current_keys: set[str] = set()
|
||||
progress_commit_interval = 1000
|
||||
|
||||
previous_current_result = await db.execute(
|
||||
select(CollectedData.entity_key).where(
|
||||
select(CollectedData)
|
||||
.where(
|
||||
CollectedData.source == self.name,
|
||||
CollectedData.is_current == True,
|
||||
)
|
||||
.order_by(CollectedData.entity_key.asc(), CollectedData.collected_at.desc().nullslast(), CollectedData.id.desc())
|
||||
)
|
||||
previous_current_keys = {row[0] for row in previous_current_result.fetchall() if row[0]}
|
||||
previous_current_records = previous_current_result.scalars().all()
|
||||
previous_current_keys = {record.entity_key for record in previous_current_records if record.entity_key}
|
||||
previous_current_map: dict[str, CollectedData] = {}
|
||||
stale_previous_records: list[CollectedData] = []
|
||||
|
||||
for existing_record in previous_current_records:
|
||||
entity_key = existing_record.entity_key
|
||||
if not entity_key:
|
||||
continue
|
||||
if entity_key not in previous_current_map:
|
||||
previous_current_map[entity_key] = existing_record
|
||||
continue
|
||||
stale_previous_records.append(existing_record)
|
||||
|
||||
for stale_record in stale_previous_records:
|
||||
stale_record.is_current = False
|
||||
|
||||
for i, item in enumerate(data):
|
||||
print(
|
||||
f"DEBUG: Saving item {i}: name={item.get('name')}, metadata={item.get('metadata', 'NOT FOUND')}"
|
||||
)
|
||||
raw_metadata = item.get("metadata", {})
|
||||
extra_data = build_dynamic_metadata(
|
||||
raw_metadata,
|
||||
@@ -318,20 +403,9 @@ class BaseCollector(ABC):
|
||||
previous_record = None
|
||||
|
||||
if entity_key and entity_key not in seen_entity_keys:
|
||||
result = await db.execute(
|
||||
select(CollectedData)
|
||||
.where(
|
||||
CollectedData.source == self.name,
|
||||
CollectedData.entity_key == entity_key,
|
||||
CollectedData.is_current == True,
|
||||
)
|
||||
.order_by(CollectedData.collected_at.desc().nullslast(), CollectedData.id.desc())
|
||||
)
|
||||
previous_records = result.scalars().all()
|
||||
if previous_records:
|
||||
previous_record = previous_records[0]
|
||||
for old_record in previous_records:
|
||||
old_record.is_current = False
|
||||
previous_record = previous_current_map.get(entity_key)
|
||||
if previous_record is not None:
|
||||
previous_record.is_current = False
|
||||
|
||||
record = CollectedData(
|
||||
snapshot_id=snapshot_id,
|
||||
@@ -375,7 +449,7 @@ class BaseCollector(ABC):
|
||||
seen_entity_keys.add(entity_key)
|
||||
records_added += 1
|
||||
|
||||
if i % 100 == 0:
|
||||
if (i + 1) % progress_commit_interval == 0:
|
||||
await self.update_progress(i + 1, commit=True)
|
||||
|
||||
if snapshot_id is not None:
|
||||
|
||||
@@ -21,7 +21,7 @@ class CelesTrakTLECollector(BaseCollector):
|
||||
|
||||
@property
|
||||
def base_url(self) -> str:
|
||||
return "https://celestrak.org/NORAD/elements/gp.php"
|
||||
return self._resolved_url or ""
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
satellite_groups = [
|
||||
@@ -40,12 +40,15 @@ class CelesTrakTLECollector(BaseCollector):
|
||||
async with httpx.AsyncClient(timeout=120.0) as client:
|
||||
for group in satellite_groups:
|
||||
try:
|
||||
url = f"https://celestrak.org/NORAD/elements/gp.php?GROUP={group}&FORMAT=json"
|
||||
url = f"{self.base_url}?GROUP={group}&FORMAT=json"
|
||||
response = await client.get(url)
|
||||
|
||||
if response.status_code == 200:
|
||||
data = response.json()
|
||||
if isinstance(data, list):
|
||||
for item in data:
|
||||
if isinstance(item, dict):
|
||||
item["_celestrak_group"] = group
|
||||
all_satellites.extend(data)
|
||||
print(f"CelesTrak: Fetched {len(data)} satellites from group '{group}'")
|
||||
except Exception as e:
|
||||
@@ -78,6 +81,7 @@ class CelesTrakTLECollector(BaseCollector):
|
||||
"name": item.get("OBJECT_NAME", "Unknown"),
|
||||
"reference_date": item.get("EPOCH", ""),
|
||||
"metadata": {
|
||||
"constellation_group": item.get("_celestrak_group"),
|
||||
"norad_cat_id": item.get("NORAD_CAT_ID"),
|
||||
"international_designator": item.get("OBJECT_ID"),
|
||||
"epoch": item.get("EPOCH"),
|
||||
|
||||
@@ -39,6 +39,16 @@ class CloudflareRadarDeviceCollector(HTTPCollector):
|
||||
if CLOUDFLARE_API_TOKEN:
|
||||
self.headers["Authorization"] = f"Bearer {CLOUDFLARE_API_TOKEN}"
|
||||
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
return self._resolved_url or self.base_url
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(self.request_url, headers=self.headers)
|
||||
response.raise_for_status()
|
||||
return self.parse_response(response.json())
|
||||
|
||||
def parse_response(self, response: Dict[str, Any]) -> List[Dict[str, Any]]:
|
||||
"""Parse Cloudflare Radar device type response"""
|
||||
data = []
|
||||
@@ -87,6 +97,16 @@ class CloudflareRadarTrafficCollector(HTTPCollector):
|
||||
if CLOUDFLARE_API_TOKEN:
|
||||
self.headers["Authorization"] = f"Bearer {CLOUDFLARE_API_TOKEN}"
|
||||
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
return self._resolved_url or self.base_url
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(self.request_url, headers=self.headers)
|
||||
response.raise_for_status()
|
||||
return self.parse_response(response.json())
|
||||
|
||||
def parse_response(self, response: Dict[str, Any]) -> List[Dict[str, Any]]:
|
||||
"""Parse Cloudflare Radar traffic timeseries response"""
|
||||
data = []
|
||||
@@ -135,6 +155,16 @@ class CloudflareRadarTopASCollector(HTTPCollector):
|
||||
if CLOUDFLARE_API_TOKEN:
|
||||
self.headers["Authorization"] = f"Bearer {CLOUDFLARE_API_TOKEN}"
|
||||
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
return self._resolved_url or self.base_url
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(self.request_url, headers=self.headers)
|
||||
response.raise_for_status()
|
||||
return self.parse_response(response.json())
|
||||
|
||||
def parse_response(self, response: Dict[str, Any]) -> List[Dict[str, Any]]:
|
||||
"""Parse Cloudflare Radar top locations response"""
|
||||
data = []
|
||||
|
||||
@@ -23,7 +23,7 @@ class EpochAIGPUCollector(BaseCollector):
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
"""Fetch Epoch AI GPU clusters data from webpage"""
|
||||
url = "https://epoch.ai/data/gpu-clusters"
|
||||
url = self._resolved_url or ""
|
||||
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(url)
|
||||
|
||||
@@ -18,11 +18,9 @@ class FAOLandingPointCollector(BaseCollector):
|
||||
frequency_hours = 168
|
||||
data_type = "landing_point"
|
||||
|
||||
csv_url = "https://data.apps.fao.org/catalog/dataset/1b75ff21-92f2-4b96-9b7b-98e8aa65ad5d/resource/b6071077-d1d4-4e97-aa00-42e902847c87/download/landing-point-geo.csv"
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(self.csv_url)
|
||||
response = await client.get(self._resolved_url or "")
|
||||
response.raise_for_status()
|
||||
return self.parse_csv(response.text)
|
||||
|
||||
|
||||
@@ -21,6 +21,18 @@ class HuggingFaceModelCollector(HTTPCollector):
|
||||
data_type = "model"
|
||||
base_url = "https://huggingface.co/api/models"
|
||||
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
return self._resolved_url or self.base_url
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
from httpx import AsyncClient
|
||||
|
||||
async with AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(self.request_url, headers=self.headers)
|
||||
response.raise_for_status()
|
||||
return self.parse_response(response.json())
|
||||
|
||||
def parse_response(self, response: Dict[str, Any]) -> List[Dict[str, Any]]:
|
||||
"""Parse Hugging Face models API response"""
|
||||
data = []
|
||||
@@ -63,6 +75,18 @@ class HuggingFaceDatasetCollector(HTTPCollector):
|
||||
data_type = "dataset"
|
||||
base_url = "https://huggingface.co/api/datasets"
|
||||
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
return self._resolved_url or self.base_url
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
from httpx import AsyncClient
|
||||
|
||||
async with AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(self.request_url, headers=self.headers)
|
||||
response.raise_for_status()
|
||||
return self.parse_response(response.json())
|
||||
|
||||
def parse_response(self, response: Dict[str, Any]) -> List[Dict[str, Any]]:
|
||||
"""Parse Hugging Face datasets API response"""
|
||||
data = []
|
||||
@@ -104,6 +128,18 @@ class HuggingFaceSpacesCollector(HTTPCollector):
|
||||
data_type = "space"
|
||||
base_url = "https://huggingface.co/api/spaces"
|
||||
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
return self._resolved_url or self.base_url
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
from httpx import AsyncClient
|
||||
|
||||
async with AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(self.request_url, headers=self.headers)
|
||||
response.raise_for_status()
|
||||
return self.parse_response(response.json())
|
||||
|
||||
def parse_response(self, response: Dict[str, Any]) -> List[Dict[str, Any]]:
|
||||
"""Parse Hugging Face Spaces API response"""
|
||||
data = []
|
||||
|
||||
@@ -0,0 +1,570 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
from urllib.parse import urlparse
|
||||
|
||||
import httpx
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
from app.models.datasource_config import DataSourceConfig
|
||||
from app.services.collectors.base import BaseCollector
|
||||
|
||||
|
||||
class NewsLiveStreamsCollector(BaseCollector):
|
||||
"""Collect normalized news live-stream sources from a JSON endpoint."""
|
||||
|
||||
name = "news_live_streams"
|
||||
priority = "P2"
|
||||
module = "L4"
|
||||
frequency_hours = 12
|
||||
data_type = "news_live_stream"
|
||||
fail_on_empty = False
|
||||
|
||||
DEFAULT_TIMEOUT = 45.0
|
||||
DEFAULT_HEADERS = {
|
||||
"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
|
||||
"Accept": "application/json",
|
||||
}
|
||||
RESPONSE_CANDIDATE_KEYS = ("sources", "streams", "channels", "items", "results", "data")
|
||||
DEFAULT_ADAPTER = "iptv_org"
|
||||
DEFAULT_IPTV_ORG_STREAMS_URL = "https://iptv-org.github.io/api/streams.json"
|
||||
DEFAULT_IPTV_ORG_LOGOS_URL = "https://iptv-org.github.io/api/logos.json"
|
||||
DEFAULT_IPTV_ORG_NEWS_CATEGORIES = ("news", "business", "weather")
|
||||
DEFAULT_IPTV_ORG_EXCLUDE_CATEGORIES = ("music", "sports", "kids", "entertainment")
|
||||
DEFAULT_IPTV_ORG_MAX_SOURCES = 120
|
||||
|
||||
async def fetch(self) -> list[dict[str, Any]]:
|
||||
request_url = (self._resolved_url or "").strip()
|
||||
if not request_url:
|
||||
return []
|
||||
|
||||
datasource_config = await self._load_datasource_config()
|
||||
effective_config = self._get_effective_config(datasource_config)
|
||||
adapter = str(effective_config.get("adapter") or "").strip().lower()
|
||||
if adapter == "iptv_org":
|
||||
return await self._fetch_iptv_org(request_url, effective_config)
|
||||
|
||||
request_headers = self._build_request_headers(datasource_config)
|
||||
request_config = self._get_request_config(datasource_config)
|
||||
request_params = self._build_request_params(datasource_config)
|
||||
request_json = self._build_request_json_body(datasource_config)
|
||||
request_data = self._build_request_form_body(datasource_config)
|
||||
timeout = self._get_timeout(datasource_config)
|
||||
|
||||
async with httpx.AsyncClient(timeout=timeout, follow_redirects=True) as client:
|
||||
response = await client.request(
|
||||
request_config["method"],
|
||||
request_url,
|
||||
headers=request_headers,
|
||||
params=request_params or None,
|
||||
json=request_json,
|
||||
data=request_data,
|
||||
)
|
||||
response.raise_for_status()
|
||||
return self.parse_response(
|
||||
response.json(),
|
||||
response_path=request_config["response_path"],
|
||||
)
|
||||
|
||||
async def _load_datasource_config(self) -> DataSourceConfig | None:
|
||||
if not self._db_session:
|
||||
return None
|
||||
|
||||
result = await self._db_session.execute(
|
||||
select(DataSourceConfig)
|
||||
.where(DataSourceConfig.name == self.name)
|
||||
.where(DataSourceConfig.is_active.is_(True))
|
||||
.limit(1)
|
||||
)
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
def _get_effective_config(self, datasource_config: DataSourceConfig | None) -> dict[str, Any]:
|
||||
payload = dict(datasource_config.config or {}) if datasource_config else {}
|
||||
if payload:
|
||||
return payload
|
||||
|
||||
yaml_config = get_data_sources_config()
|
||||
return {
|
||||
"adapter": self.DEFAULT_ADAPTER,
|
||||
"streams_url": yaml_config.get_yaml_value("news_live_streams.streams_url")
|
||||
or self.DEFAULT_IPTV_ORG_STREAMS_URL,
|
||||
"logos_url": yaml_config.get_yaml_value("news_live_streams.logos_url")
|
||||
or self.DEFAULT_IPTV_ORG_LOGOS_URL,
|
||||
"news_categories": list(self.DEFAULT_IPTV_ORG_NEWS_CATEGORIES),
|
||||
"exclude_categories": list(self.DEFAULT_IPTV_ORG_EXCLUDE_CATEGORIES),
|
||||
"max_sources": self.DEFAULT_IPTV_ORG_MAX_SOURCES,
|
||||
}
|
||||
|
||||
def _get_request_config(self, datasource_config: DataSourceConfig | None) -> dict[str, Any]:
|
||||
payload = self._get_effective_config(datasource_config)
|
||||
raw_method = payload.get("method") or payload.get("request_method") or "GET"
|
||||
method = str(raw_method).strip().upper() or "GET"
|
||||
if method not in {"GET", "POST"}:
|
||||
method = "GET"
|
||||
|
||||
response_path = payload.get("response_path") or payload.get("payload_path") or payload.get("items_path")
|
||||
if isinstance(response_path, str):
|
||||
response_path = response_path.strip()
|
||||
else:
|
||||
response_path = None
|
||||
|
||||
return {
|
||||
"method": method,
|
||||
"response_path": response_path or None,
|
||||
}
|
||||
|
||||
def _get_timeout(self, datasource_config: DataSourceConfig | None) -> float:
|
||||
payload = self._get_effective_config(datasource_config)
|
||||
try:
|
||||
return float(payload.get("timeout", self.DEFAULT_TIMEOUT))
|
||||
except (TypeError, ValueError):
|
||||
return self.DEFAULT_TIMEOUT
|
||||
|
||||
def _build_request_headers(self, datasource_config: DataSourceConfig | None) -> dict[str, str]:
|
||||
headers = dict(self.DEFAULT_HEADERS)
|
||||
if datasource_config:
|
||||
headers.update(self._normalize_headers(datasource_config.headers))
|
||||
headers.update(self._build_auth_headers(datasource_config))
|
||||
return headers
|
||||
|
||||
def _build_request_params(self, datasource_config: DataSourceConfig | None) -> dict[str, Any]:
|
||||
params: dict[str, Any] = {}
|
||||
if not datasource_config:
|
||||
return params
|
||||
|
||||
payload = datasource_config.config or {}
|
||||
candidate = payload.get("params") or payload.get("query_params")
|
||||
if isinstance(candidate, dict):
|
||||
params.update(candidate)
|
||||
|
||||
if datasource_config.auth_type == "api_key":
|
||||
auth_config = datasource_config.auth_config or {}
|
||||
if str(auth_config.get("in") or auth_config.get("location") or "header").lower() == "query":
|
||||
api_key = auth_config.get("api_key")
|
||||
key_name = auth_config.get("key_name") or auth_config.get("param_name") or "api_key"
|
||||
if api_key and key_name:
|
||||
params[str(key_name)] = api_key
|
||||
|
||||
return params
|
||||
|
||||
def _build_request_json_body(self, datasource_config: DataSourceConfig | None) -> Any:
|
||||
if not datasource_config:
|
||||
return None
|
||||
|
||||
payload = datasource_config.config or {}
|
||||
body = payload.get("json_body")
|
||||
if body is None and str(payload.get("body_type") or "").lower() in {"json", ""}:
|
||||
candidate = payload.get("body")
|
||||
if isinstance(candidate, (dict, list)):
|
||||
body = candidate
|
||||
return body
|
||||
|
||||
def _build_request_form_body(self, datasource_config: DataSourceConfig | None) -> Any:
|
||||
if not datasource_config:
|
||||
return None
|
||||
|
||||
payload = datasource_config.config or {}
|
||||
form_body = payload.get("form_body")
|
||||
if form_body is not None:
|
||||
return form_body
|
||||
|
||||
if str(payload.get("body_type") or "").lower() == "form":
|
||||
candidate = payload.get("body")
|
||||
if isinstance(candidate, dict):
|
||||
return candidate
|
||||
return None
|
||||
|
||||
def _normalize_headers(self, headers: Any) -> dict[str, str]:
|
||||
if not isinstance(headers, dict):
|
||||
return {}
|
||||
normalized: dict[str, str] = {}
|
||||
for key, value in headers.items():
|
||||
header_name = str(key).strip()
|
||||
if not header_name or value is None:
|
||||
continue
|
||||
normalized[header_name] = str(value)
|
||||
return normalized
|
||||
|
||||
def _build_auth_headers(self, datasource_config: DataSourceConfig | None) -> dict[str, str]:
|
||||
if not datasource_config:
|
||||
return {}
|
||||
|
||||
auth_type = str(datasource_config.auth_type or "none").lower()
|
||||
auth_config = datasource_config.auth_config or {}
|
||||
if auth_type == "bearer" and auth_config.get("token"):
|
||||
return {"Authorization": f"Bearer {auth_config['token']}"}
|
||||
|
||||
if auth_type == "api_key" and auth_config.get("api_key"):
|
||||
location = str(auth_config.get("in") or auth_config.get("location") or "header").lower()
|
||||
if location == "query":
|
||||
return {}
|
||||
key_name = auth_config.get("key_name") or "X-API-Key"
|
||||
return {str(key_name): str(auth_config["api_key"])}
|
||||
|
||||
if auth_type == "basic":
|
||||
username = str(auth_config.get("username") or "")
|
||||
password = str(auth_config.get("password") or "")
|
||||
encoded = base64.b64encode(f"{username}:{password}".encode()).decode()
|
||||
return {"Authorization": f"Basic {encoded}"}
|
||||
|
||||
return {}
|
||||
|
||||
def _extract_candidates(self, response: Any, response_path: str | None) -> list[Any]:
|
||||
if response_path:
|
||||
extracted = self._extract_from_path(response, response_path)
|
||||
if isinstance(extracted, list):
|
||||
return extracted
|
||||
if isinstance(extracted, dict):
|
||||
for key in self.RESPONSE_CANDIDATE_KEYS:
|
||||
nested = extracted.get(key)
|
||||
if isinstance(nested, list):
|
||||
return nested
|
||||
return [extracted]
|
||||
|
||||
if isinstance(response, dict):
|
||||
for key in self.RESPONSE_CANDIDATE_KEYS:
|
||||
nested = response.get(key)
|
||||
if isinstance(nested, list):
|
||||
return nested
|
||||
return []
|
||||
|
||||
if isinstance(response, list):
|
||||
return response
|
||||
return []
|
||||
|
||||
def _extract_from_path(self, payload: Any, path: str) -> Any:
|
||||
current = payload
|
||||
for segment in (part.strip() for part in path.split(".") if part.strip()):
|
||||
if isinstance(current, dict):
|
||||
current = current.get(segment)
|
||||
continue
|
||||
if isinstance(current, list):
|
||||
try:
|
||||
current = current[int(segment)]
|
||||
except (TypeError, ValueError, IndexError):
|
||||
return None
|
||||
continue
|
||||
return None
|
||||
return current
|
||||
|
||||
def _infer_source_type(self, item: dict[str, Any]) -> str:
|
||||
explicit = str(item.get("source_type") or item.get("type") or "").strip().lower()
|
||||
if explicit in {"iframe", "hls", "video", "external", "youtube"}:
|
||||
return explicit
|
||||
|
||||
youtube_video_id = self._clean_text(
|
||||
item.get("youtube_video_id")
|
||||
or item.get("video_id")
|
||||
or item.get("youtubeVideoId")
|
||||
)
|
||||
youtube_channel = self._clean_text(item.get("youtube_channel") or item.get("channel_handle"))
|
||||
embed_url = self._clean_url(item.get("embed_url") or item.get("embed") or item.get("page_url"))
|
||||
stream_url = self._clean_url(item.get("stream_url") or item.get("stream") or item.get("playback_url") or item.get("hls_url"))
|
||||
homepage_url = self._clean_url(item.get("homepage_url") or item.get("source_url") or item.get("website"))
|
||||
|
||||
if youtube_video_id or youtube_channel:
|
||||
return "youtube"
|
||||
if stream_url.endswith(".m3u8"):
|
||||
return "hls"
|
||||
if stream_url:
|
||||
return "video"
|
||||
if embed_url:
|
||||
parsed = urlparse(embed_url)
|
||||
if "youtube.com" in (parsed.netloc or "") or "youtu.be" in (parsed.netloc or ""):
|
||||
return "youtube"
|
||||
return "iframe"
|
||||
if homepage_url:
|
||||
return "external"
|
||||
return "iframe"
|
||||
|
||||
def _parse_enabled(self, item: dict[str, Any]) -> bool:
|
||||
if "is_enabled" in item:
|
||||
return self._to_bool(item.get("is_enabled"), default=True)
|
||||
if "enabled" in item:
|
||||
return self._to_bool(item.get("enabled"), default=True)
|
||||
if "active" in item:
|
||||
return self._to_bool(item.get("active"), default=True)
|
||||
if "status" in item:
|
||||
status = str(item.get("status") or "").strip().lower()
|
||||
if status in {"disabled", "inactive", "offline"}:
|
||||
return False
|
||||
if status in {"enabled", "active", "online", "live"}:
|
||||
return True
|
||||
return True
|
||||
|
||||
def _to_bool(self, value: Any, *, default: bool) -> bool:
|
||||
if isinstance(value, bool):
|
||||
return value
|
||||
if value in (None, ""):
|
||||
return default
|
||||
if isinstance(value, str):
|
||||
lowered = value.strip().lower()
|
||||
if lowered in {"1", "true", "yes", "on", "enabled", "active", "online", "live"}:
|
||||
return True
|
||||
if lowered in {"0", "false", "no", "off", "disabled", "inactive", "offline"}:
|
||||
return False
|
||||
return bool(value)
|
||||
|
||||
def _clean_text(self, value: Any) -> str:
|
||||
if value is None:
|
||||
return ""
|
||||
return str(value).strip()
|
||||
|
||||
def _clean_url(self, value: Any) -> str:
|
||||
text = self._clean_text(value)
|
||||
if not text:
|
||||
return ""
|
||||
parsed = urlparse(text)
|
||||
if parsed.scheme and parsed.scheme not in {"http", "https"}:
|
||||
return ""
|
||||
if parsed.scheme and not parsed.netloc:
|
||||
return ""
|
||||
return text
|
||||
|
||||
async def _fetch_iptv_org(self, channels_url: str, collector_config: dict[str, Any]) -> list[dict[str, Any]]:
|
||||
streams_url = self._clean_url(collector_config.get("streams_url")) or self.DEFAULT_IPTV_ORG_STREAMS_URL
|
||||
logos_url = self._clean_url(collector_config.get("logos_url")) or self.DEFAULT_IPTV_ORG_LOGOS_URL
|
||||
news_categories = {
|
||||
self._clean_text(value).lower()
|
||||
for value in (collector_config.get("news_categories") or self.DEFAULT_IPTV_ORG_NEWS_CATEGORIES)
|
||||
if self._clean_text(value)
|
||||
}
|
||||
exclude_categories = {
|
||||
self._clean_text(value).lower()
|
||||
for value in (collector_config.get("exclude_categories") or self.DEFAULT_IPTV_ORG_EXCLUDE_CATEGORIES)
|
||||
if self._clean_text(value)
|
||||
}
|
||||
try:
|
||||
max_sources = int(collector_config.get("max_sources", self.DEFAULT_IPTV_ORG_MAX_SOURCES))
|
||||
except (TypeError, ValueError):
|
||||
max_sources = self.DEFAULT_IPTV_ORG_MAX_SOURCES
|
||||
|
||||
timeout = self.DEFAULT_TIMEOUT
|
||||
try:
|
||||
timeout = float(collector_config.get("timeout", self.DEFAULT_TIMEOUT))
|
||||
except (TypeError, ValueError):
|
||||
timeout = self.DEFAULT_TIMEOUT
|
||||
|
||||
async with httpx.AsyncClient(timeout=timeout, follow_redirects=True) as client:
|
||||
channels_payload, streams_payload, logos_payload = await self._gather_iptv_org_payloads(
|
||||
client,
|
||||
channels_url,
|
||||
streams_url,
|
||||
logos_url,
|
||||
)
|
||||
|
||||
channels = channels_payload if isinstance(channels_payload, list) else []
|
||||
streams = streams_payload if isinstance(streams_payload, list) else []
|
||||
logos = logos_payload if isinstance(logos_payload, list) else []
|
||||
|
||||
logo_by_channel = {
|
||||
self._clean_text(item.get("channel")): self._clean_url(item.get("url"))
|
||||
for item in logos
|
||||
if isinstance(item, dict) and self._clean_text(item.get("channel")) and self._clean_url(item.get("url"))
|
||||
}
|
||||
|
||||
streams_by_channel: dict[str, list[dict[str, Any]]] = {}
|
||||
for stream in streams:
|
||||
if not isinstance(stream, dict):
|
||||
continue
|
||||
channel_id = self._clean_text(stream.get("channel"))
|
||||
if not channel_id:
|
||||
continue
|
||||
streams_by_channel.setdefault(channel_id, []).append(stream)
|
||||
|
||||
normalized: list[dict[str, Any]] = []
|
||||
for channel in channels:
|
||||
if not isinstance(channel, dict):
|
||||
continue
|
||||
|
||||
categories = [
|
||||
self._clean_text(value).lower()
|
||||
for value in (channel.get("categories") or [])
|
||||
if self._clean_text(value)
|
||||
]
|
||||
if news_categories and not any(category in news_categories for category in categories):
|
||||
continue
|
||||
if exclude_categories and any(category in exclude_categories for category in categories):
|
||||
continue
|
||||
if channel.get("is_nsfw") is True:
|
||||
continue
|
||||
if channel.get("closed"):
|
||||
continue
|
||||
|
||||
channel_id = self._clean_text(channel.get("id"))
|
||||
if not channel_id:
|
||||
continue
|
||||
|
||||
stream = self._pick_iptv_org_stream(streams_by_channel.get(channel_id) or [])
|
||||
if not stream:
|
||||
continue
|
||||
|
||||
stream_url = self._clean_url(stream.get("url"))
|
||||
if not stream_url:
|
||||
continue
|
||||
|
||||
name = self._clean_text(channel.get("name")) or channel_id
|
||||
notes_parts = [
|
||||
f"Imported from IPTV-org catalog ({channel_id})",
|
||||
f"Categories: {', '.join(categories)}" if categories else "",
|
||||
f"Quality: {self._clean_text(stream.get('quality'))}" if self._clean_text(stream.get("quality")) else "",
|
||||
]
|
||||
metadata = {
|
||||
"provider": self._clean_text(channel.get("network")) or "IPTV-org",
|
||||
"region": self._clean_text(channel.get("country")) or "Global",
|
||||
"language": "und",
|
||||
"source_type": "hls" if stream_url.endswith(".m3u8") else "video",
|
||||
"embed_url": "",
|
||||
"stream_url": stream_url,
|
||||
"homepage_url": self._clean_url(channel.get("website")),
|
||||
"poster_url": logo_by_channel.get(channel_id, ""),
|
||||
"youtube_video_id": "",
|
||||
"youtube_channel": "",
|
||||
"sort_order": 400 + len(normalized),
|
||||
"notes": "; ".join(part for part in notes_parts if part),
|
||||
"is_enabled": True,
|
||||
"collector_adapter": "iptv_org",
|
||||
"channel_id": channel_id,
|
||||
"categories": categories,
|
||||
"quality": self._clean_text(stream.get("quality")),
|
||||
"stream_label": self._clean_text(stream.get("label") or stream.get("title")),
|
||||
"stream_referrer": self._clean_text(stream.get("referrer")),
|
||||
"stream_user_agent": self._clean_text(stream.get("user_agent")),
|
||||
}
|
||||
|
||||
normalized.append(
|
||||
{
|
||||
"source_id": channel_id,
|
||||
"name": name,
|
||||
"description": metadata["notes"],
|
||||
"metadata": metadata,
|
||||
"reference_date": datetime.now(UTC).isoformat(),
|
||||
}
|
||||
)
|
||||
if len(normalized) >= max_sources:
|
||||
break
|
||||
|
||||
return normalized
|
||||
|
||||
async def _gather_iptv_org_payloads(
|
||||
self,
|
||||
client: httpx.AsyncClient,
|
||||
channels_url: str,
|
||||
streams_url: str,
|
||||
logos_url: str,
|
||||
) -> tuple[Any, Any, Any]:
|
||||
headers = dict(self.DEFAULT_HEADERS)
|
||||
channels_payload, streams_payload, logos_payload = await asyncio.gather(
|
||||
client.get(channels_url, headers=headers),
|
||||
client.get(streams_url, headers=headers),
|
||||
client.get(logos_url, headers=headers),
|
||||
)
|
||||
channels_payload.raise_for_status()
|
||||
streams_payload.raise_for_status()
|
||||
logos_payload.raise_for_status()
|
||||
return channels_payload.json(), streams_payload.json(), logos_payload.json()
|
||||
|
||||
def _pick_iptv_org_stream(self, streams: list[dict[str, Any]]) -> dict[str, Any] | None:
|
||||
if not streams:
|
||||
return None
|
||||
|
||||
def score(stream: dict[str, Any]) -> tuple[int, int]:
|
||||
url = self._clean_url(stream.get("url"))
|
||||
quality = self._clean_text(stream.get("quality")).lower()
|
||||
quality_score = 0
|
||||
if quality.endswith("p"):
|
||||
try:
|
||||
quality_score = int(quality[:-1])
|
||||
except ValueError:
|
||||
quality_score = 0
|
||||
stream_score = 1000 if url.endswith(".m3u8") else 0
|
||||
return stream_score, quality_score
|
||||
|
||||
sorted_streams = sorted(streams, key=score, reverse=True)
|
||||
return sorted_streams[0]
|
||||
|
||||
def parse_response(self, response: Any, *, response_path: str | None = None) -> list[dict[str, Any]]:
|
||||
candidates = self._extract_candidates(response, response_path)
|
||||
|
||||
normalized: list[dict[str, Any]] = []
|
||||
for index, item in enumerate(candidates):
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
|
||||
stream_id = (
|
||||
item.get("id")
|
||||
or item.get("source_id")
|
||||
or item.get("slug")
|
||||
or item.get("channel_id")
|
||||
or item.get("code")
|
||||
or f"news-live-{index + 1}"
|
||||
)
|
||||
name = self._clean_text(
|
||||
item.get("name")
|
||||
or item.get("title")
|
||||
or item.get("channel")
|
||||
or item.get("display_name")
|
||||
or f"News Live {index + 1}"
|
||||
)
|
||||
if not name:
|
||||
continue
|
||||
|
||||
source_type = self._infer_source_type(item)
|
||||
stream_url = self._clean_url(
|
||||
item.get("stream_url")
|
||||
or item.get("stream")
|
||||
or item.get("playback_url")
|
||||
or item.get("hls_url")
|
||||
or item.get("m3u8_url")
|
||||
)
|
||||
embed_url = self._clean_url(
|
||||
item.get("embed_url")
|
||||
or item.get("embed")
|
||||
or item.get("page_url")
|
||||
or (item.get("url") if source_type == "iframe" else "")
|
||||
)
|
||||
homepage_url = self._clean_url(
|
||||
item.get("homepage_url")
|
||||
or item.get("source_url")
|
||||
or item.get("website")
|
||||
or item.get("url")
|
||||
)
|
||||
metadata = {
|
||||
"provider": self._clean_text(item.get("provider") or item.get("publisher") or item.get("network")) or "Collector",
|
||||
"region": self._clean_text(item.get("region") or item.get("country") or item.get("market")) or "Global",
|
||||
"language": self._clean_text(item.get("language") or item.get("lang") or item.get("locale")) or "und",
|
||||
"source_type": source_type,
|
||||
"embed_url": embed_url,
|
||||
"stream_url": stream_url,
|
||||
"homepage_url": homepage_url,
|
||||
"poster_url": self._clean_url(item.get("poster_url") or item.get("thumbnail_url") or item.get("logo_url")),
|
||||
"youtube_video_id": self._clean_text(
|
||||
item.get("youtube_video_id")
|
||||
or item.get("video_id")
|
||||
or item.get("youtubeVideoId")
|
||||
),
|
||||
"youtube_channel": self._clean_text(
|
||||
item.get("youtube_channel")
|
||||
or item.get("channel_handle")
|
||||
or item.get("youtubeChannel")
|
||||
),
|
||||
"sort_order": item.get("sort_order", 200 + index),
|
||||
"notes": self._clean_text(item.get("notes") or item.get("description") or item.get("summary")),
|
||||
"is_enabled": self._parse_enabled(item),
|
||||
}
|
||||
|
||||
normalized.append(
|
||||
{
|
||||
"source_id": str(stream_id),
|
||||
"name": name,
|
||||
"description": metadata["notes"],
|
||||
"metadata": metadata,
|
||||
"reference_date": item.get("reference_date") or datetime.now(UTC).isoformat(),
|
||||
}
|
||||
)
|
||||
|
||||
return normalized
|
||||
@@ -16,6 +16,7 @@ from typing import Dict, Any, List
|
||||
from datetime import UTC, datetime
|
||||
|
||||
import httpx
|
||||
from urllib.parse import urlencode
|
||||
from app.services.collectors.base import HTTPCollector
|
||||
|
||||
|
||||
@@ -38,9 +39,13 @@ class PeeringDBIXPCollector(HTTPCollector):
|
||||
"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
|
||||
"Accept": "application/json",
|
||||
}
|
||||
# API key is added to URL as query parameter
|
||||
if PEERINGDB_API_KEY:
|
||||
self.base_url = f"{self.base_url}?key={PEERINGDB_API_KEY}"
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
base = self._resolved_url or self.base_url
|
||||
if not PEERINGDB_API_KEY:
|
||||
return base
|
||||
separator = "&" if "?" in base else "?"
|
||||
return f"{base}{separator}{urlencode({'key': PEERINGDB_API_KEY})}"
|
||||
|
||||
async def fetch_with_retry(
|
||||
self, max_retries: int = 3, base_delay: float = 2.0
|
||||
@@ -51,7 +56,7 @@ class PeeringDBIXPCollector(HTTPCollector):
|
||||
for attempt in range(max_retries):
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(self.base_url, headers=self.headers)
|
||||
response = await client.get(self.request_url, headers=self.headers)
|
||||
|
||||
if response.status_code == 429:
|
||||
# Rate limited - wait and retry with exponential backoff
|
||||
@@ -141,8 +146,13 @@ class PeeringDBNetworkCollector(HTTPCollector):
|
||||
"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
|
||||
"Accept": "application/json",
|
||||
}
|
||||
if PEERINGDB_API_KEY:
|
||||
self.base_url = f"{self.base_url}?key={PEERINGDB_API_KEY}"
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
base = self._resolved_url or self.base_url
|
||||
if not PEERINGDB_API_KEY:
|
||||
return base
|
||||
separator = "&" if "?" in base else "?"
|
||||
return f"{base}{separator}{urlencode({'key': PEERINGDB_API_KEY})}"
|
||||
|
||||
async def fetch_with_retry(
|
||||
self, max_retries: int = 3, base_delay: float = 2.0
|
||||
@@ -153,7 +163,7 @@ class PeeringDBNetworkCollector(HTTPCollector):
|
||||
for attempt in range(max_retries):
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(self.base_url, headers=self.headers)
|
||||
response = await client.get(self.request_url, headers=self.headers)
|
||||
|
||||
if response.status_code == 429:
|
||||
delay = base_delay * (2**attempt)
|
||||
@@ -244,8 +254,13 @@ class PeeringDBFacilityCollector(HTTPCollector):
|
||||
"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
|
||||
"Accept": "application/json",
|
||||
}
|
||||
if PEERINGDB_API_KEY:
|
||||
self.base_url = f"{self.base_url}?key={PEERINGDB_API_KEY}"
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
base = self._resolved_url or self.base_url
|
||||
if not PEERINGDB_API_KEY:
|
||||
return base
|
||||
separator = "&" if "?" in base else "?"
|
||||
return f"{base}{separator}{urlencode({'key': PEERINGDB_API_KEY})}"
|
||||
|
||||
async def fetch_with_retry(
|
||||
self, max_retries: int = 3, base_delay: float = 2.0
|
||||
@@ -256,7 +271,7 @@ class PeeringDBFacilityCollector(HTTPCollector):
|
||||
for attempt in range(max_retries):
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(self.base_url, headers=self.headers)
|
||||
response = await client.get(self.request_url, headers=self.headers)
|
||||
|
||||
if response.status_code == 429:
|
||||
delay = base_delay * (2**attempt)
|
||||
|
||||
@@ -33,7 +33,7 @@ class RISLiveCollector(BaseCollector):
|
||||
|
||||
def _fetch_via_stream(self) -> list[dict[str, Any]]:
|
||||
events: list[dict[str, Any]] = []
|
||||
stream_url = "https://ris-live.ripe.net/v1/stream/?format=json&client=planet-ris-live"
|
||||
stream_url = self._resolved_url or ""
|
||||
subscribe = json.dumps(
|
||||
{
|
||||
"host": "rrc00",
|
||||
|
||||
@@ -7,6 +7,7 @@ API documentation: https://www.space-track.org/documentation
|
||||
import json
|
||||
from typing import Dict, Any, List
|
||||
import httpx
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from app.services.collectors.base import BaseCollector
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
@@ -21,12 +22,30 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
data_type = "satellite_tle"
|
||||
|
||||
@property
|
||||
def base_url(self) -> str:
|
||||
def query_url(self) -> str:
|
||||
config = get_data_sources_config()
|
||||
if self._resolved_url:
|
||||
return self._resolved_url
|
||||
return config.get_yaml_url("spacetrack_tle")
|
||||
|
||||
@property
|
||||
def site_root(self) -> str:
|
||||
config = get_data_sources_config()
|
||||
configured_root = config.get_yaml_value("spacetrack.base_url")
|
||||
if isinstance(configured_root, str) and configured_root:
|
||||
return configured_root.rstrip("/")
|
||||
|
||||
parsed = urlparse(self.query_url)
|
||||
return f"{parsed.scheme}://{parsed.netloc}".rstrip("/")
|
||||
|
||||
@property
|
||||
def login_url(self) -> str:
|
||||
return f"{self.site_root}/ajaxauth/login"
|
||||
|
||||
@property
|
||||
def probe_url(self) -> str:
|
||||
return f"{self.site_root}/basicspacedata/query/class/gp/NORAD_CAT_ID/25544/format/json"
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
from app.core.config import settings
|
||||
|
||||
@@ -47,13 +66,13 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
|
||||
"Accept": "application/json, text/html, */*",
|
||||
"Accept-Language": "en-US,en;q=0.9",
|
||||
"Referer": "https://www.space-track.org/",
|
||||
"Referer": f"{self.site_root}/",
|
||||
},
|
||||
) as client:
|
||||
await client.get("https://www.space-track.org/")
|
||||
await client.get(f"{self.site_root}/")
|
||||
|
||||
login_response = await client.post(
|
||||
"https://www.space-track.org/ajaxauth/login",
|
||||
self.login_url,
|
||||
data={
|
||||
"identity": username,
|
||||
"password": password,
|
||||
@@ -69,7 +88,7 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
timeout=120.0,
|
||||
follow_redirects=True,
|
||||
) as alt_client:
|
||||
await alt_client.get("https://www.space-track.org/")
|
||||
await alt_client.get(f"{self.site_root}/")
|
||||
|
||||
form_data = {
|
||||
"username": username,
|
||||
@@ -77,7 +96,7 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
"query": "class/gp/NORAD_CAT_ID/25544/format/json",
|
||||
}
|
||||
alt_login = await alt_client.post(
|
||||
"https://www.space-track.org/ajaxauth/login",
|
||||
self.login_url,
|
||||
data={
|
||||
"identity": username,
|
||||
"password": password,
|
||||
@@ -86,9 +105,7 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
print(f"SPACETRACK: Alt login status: {alt_login.status_code}")
|
||||
|
||||
if alt_login.status_code == 200:
|
||||
tle_response = await alt_client.get(
|
||||
"https://www.space-track.org/basicspacedata/query/class/gp/NORAD_CAT_ID/25544/format/json"
|
||||
)
|
||||
tle_response = await alt_client.get(self.probe_url)
|
||||
if tle_response.status_code == 200:
|
||||
data = tle_response.json()
|
||||
print(f"SPACETRACK: Received {len(data)} records via alt method")
|
||||
@@ -98,9 +115,7 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
print(f"SPACETRACK: Login failed, using sample data")
|
||||
return self._get_sample_data()
|
||||
|
||||
tle_response = await client.get(
|
||||
"https://www.space-track.org/basicspacedata/query/class/gp/NORAD_CAT_ID/25544/format/json"
|
||||
)
|
||||
tle_response = await client.get(self.probe_url)
|
||||
print(f"SPACETRACK: TLE query status: {tle_response.status_code}")
|
||||
|
||||
if tle_response.status_code != 200:
|
||||
@@ -127,11 +142,11 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
},
|
||||
) as client:
|
||||
# First, visit the main page to get any cookies
|
||||
await client.get("https://www.space-track.org/")
|
||||
await client.get(f"{self.site_root}/")
|
||||
|
||||
# Login to get session cookie
|
||||
login_response = await client.post(
|
||||
"https://www.space-track.org/ajaxauth/login",
|
||||
self.login_url,
|
||||
data={
|
||||
"identity": username,
|
||||
"password": password,
|
||||
@@ -146,13 +161,7 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
return self._get_sample_data()
|
||||
|
||||
# Query for TLE data (get first 1000 satellites)
|
||||
tle_response = await client.get(
|
||||
"https://www.space-track.org/basicspacedata/query"
|
||||
"/class/gp"
|
||||
"/orderby/EPOCH%20desc"
|
||||
"/limit/1000"
|
||||
"/format/json"
|
||||
)
|
||||
tle_response = await client.get(self.query_url)
|
||||
print(f"SPACETRACK: TLE query status: {tle_response.status_code}")
|
||||
|
||||
if tle_response.status_code != 200:
|
||||
|
||||
@@ -11,6 +11,7 @@ from datetime import UTC, datetime
|
||||
from bs4 import BeautifulSoup
|
||||
import httpx
|
||||
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
from app.services.collectors.base import BaseCollector
|
||||
|
||||
|
||||
@@ -24,15 +25,17 @@ class TeleGeographyCableCollector(BaseCollector):
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
"""Fetch submarine cable data from Wayback Machine"""
|
||||
config = get_data_sources_config()
|
||||
# Try multiple data sources
|
||||
sources = [
|
||||
# Wayback Machine archive of TeleGeography
|
||||
"https://web.archive.org/web/2024/https://www.submarinecablemap.com/api/v3/cable",
|
||||
# Alternative: Try scraping the page
|
||||
"https://www.submarinecablemap.com",
|
||||
self._resolved_url or "",
|
||||
str(config.get_yaml_value("telegeography.archived_cable_url") or ""),
|
||||
str(config.get_yaml_value("telegeography.live_map_url") or ""),
|
||||
]
|
||||
|
||||
for url in sources:
|
||||
if not url:
|
||||
continue
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=60.0, follow_redirects=True) as client:
|
||||
response = await client.get(url)
|
||||
@@ -161,7 +164,7 @@ class TeleGeographyLandingPointCollector(BaseCollector):
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
"""Fetch landing point data from GitHub mirror"""
|
||||
url = "https://raw.githubusercontent.com/lintaojlu/submarine_cable_information/main/landing_point.json"
|
||||
url = self._resolved_url or ""
|
||||
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(url)
|
||||
@@ -225,7 +228,7 @@ class TeleGeographyCableSystemCollector(BaseCollector):
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
"""Fetch cable system data"""
|
||||
url = "https://raw.githubusercontent.com/lintaojlu/submarine_cable_information/main/cable.json"
|
||||
url = self._resolved_url or ""
|
||||
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(url)
|
||||
|
||||
@@ -10,6 +10,7 @@ from typing import Dict, Any, List
|
||||
from bs4 import BeautifulSoup
|
||||
import httpx
|
||||
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
from app.services.collectors.base import BaseCollector
|
||||
|
||||
|
||||
@@ -22,7 +23,7 @@ class TOP500Collector(BaseCollector):
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
"""Fetch TOP500 list data and enrich each row with detail-page metadata."""
|
||||
url = "https://top500.org/lists/top500/list/2025/11/"
|
||||
url = self._resolved_url or ""
|
||||
|
||||
async with httpx.AsyncClient(timeout=60.0, follow_redirects=True) as client:
|
||||
response = await client.get(url)
|
||||
@@ -48,11 +49,13 @@ class TOP500Collector(BaseCollector):
|
||||
return await asyncio.gather(*(enrich(entry) for entry in entries))
|
||||
|
||||
def _extract_system_fields(self, system_cell) -> Dict[str, str]:
|
||||
config = get_data_sources_config()
|
||||
top500_base_url = config.get_yaml_value("top500.base_url") or "https://top500.org"
|
||||
link = system_cell.find("a")
|
||||
system_name = link.get_text(" ", strip=True) if link else system_cell.get_text(" ", strip=True)
|
||||
detail_url = ""
|
||||
if link and link.get("href"):
|
||||
detail_url = f"https://top500.org{link.get('href')}"
|
||||
detail_url = f"{str(top500_base_url).rstrip('/')}{link.get('href')}"
|
||||
|
||||
manufacturer = ""
|
||||
if link and link.next_sibling:
|
||||
|
||||
@@ -0,0 +1,273 @@
|
||||
"""BarentsWatch AIS collector for vessel tracking."""
|
||||
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
from sqlalchemy import delete, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.models.vessel import VesselPosition, VesselStatic
|
||||
from app.services.barentswatch import (
|
||||
BARENTSWATCH_LATEST_URL,
|
||||
fetch_barentswatch_access_token,
|
||||
resolve_barentswatch_config,
|
||||
)
|
||||
from app.services.collectors.base import BaseCollector
|
||||
|
||||
|
||||
VESSEL_TYPE_NAMES = {
|
||||
30: "Fishing",
|
||||
35: "Military",
|
||||
60: "Passenger",
|
||||
70: "Cargo",
|
||||
80: "Tanker",
|
||||
}
|
||||
|
||||
|
||||
class VesselAISCollector(BaseCollector):
|
||||
"""Collect latest AIS positions and append them to vessel tables."""
|
||||
|
||||
name = "barentswatch_vessels"
|
||||
priority = "P1"
|
||||
module = "L4"
|
||||
frequency_hours = 1
|
||||
data_type = "vessel_ais"
|
||||
|
||||
@property
|
||||
def base_url(self) -> str:
|
||||
return self._resolved_url or BARENTSWATCH_LATEST_URL
|
||||
|
||||
async def _get_access_token(self, client: httpx.AsyncClient) -> str | None:
|
||||
config = await resolve_barentswatch_config(self._db_session)
|
||||
return await fetch_barentswatch_access_token(client, config)
|
||||
|
||||
async def fetch(self) -> list[dict[str, Any]]:
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
headers: dict[str, str] = {}
|
||||
token = await self._get_access_token(client)
|
||||
if token:
|
||||
headers["Authorization"] = f"Bearer {token}"
|
||||
|
||||
response = await client.get(self.base_url, headers=headers)
|
||||
if response.status_code == 401 and not token:
|
||||
return self._get_sample_data()
|
||||
response.raise_for_status()
|
||||
payload = response.json()
|
||||
|
||||
if isinstance(payload, list):
|
||||
return [item for item in payload if isinstance(item, dict)]
|
||||
if isinstance(payload, dict):
|
||||
for key in ("features", "data", "items", "vessels"):
|
||||
value = payload.get(key)
|
||||
if isinstance(value, list):
|
||||
if key == "features":
|
||||
return [
|
||||
{
|
||||
**(item.get("properties") or {}),
|
||||
"geometry": item.get("geometry"),
|
||||
}
|
||||
for item in value
|
||||
if isinstance(item, dict)
|
||||
]
|
||||
return [item for item in value if isinstance(item, dict)]
|
||||
return self._get_sample_data()
|
||||
|
||||
def transform(self, raw_data: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
transformed = []
|
||||
for item in raw_data:
|
||||
record = self._normalize_record(item)
|
||||
if record:
|
||||
transformed.append(record)
|
||||
return transformed
|
||||
|
||||
async def _save_data(
|
||||
self,
|
||||
db: AsyncSession,
|
||||
data: list[dict[str, Any]],
|
||||
task_id: int | None = None,
|
||||
snapshot_id: int | None = None,
|
||||
) -> int:
|
||||
now = datetime.now(UTC)
|
||||
records_added = 0
|
||||
|
||||
for index, item in enumerate(data):
|
||||
static = await db.get(VesselStatic, item["mmsi"])
|
||||
if static is None:
|
||||
static = VesselStatic(mmsi=item["mmsi"])
|
||||
db.add(static)
|
||||
|
||||
for field in (
|
||||
"name",
|
||||
"callsign",
|
||||
"vessel_type",
|
||||
"vessel_type_name",
|
||||
"flag",
|
||||
"length",
|
||||
"width",
|
||||
"draught",
|
||||
"imo",
|
||||
):
|
||||
value = item.get(field)
|
||||
if value not in (None, ""):
|
||||
setattr(static, field, value)
|
||||
static.updated_at = now
|
||||
|
||||
db.add(
|
||||
VesselPosition(
|
||||
mmsi=item["mmsi"],
|
||||
lat=item["lat"],
|
||||
lon=item["lon"],
|
||||
sog=item.get("sog"),
|
||||
cog=item.get("cog"),
|
||||
heading=item.get("heading"),
|
||||
nav_status=item.get("nav_status"),
|
||||
received_at=item.get("received_at") or now,
|
||||
)
|
||||
)
|
||||
records_added += 1
|
||||
|
||||
if (index + 1) % 1000 == 0:
|
||||
await self.update_progress(index + 1, commit=True)
|
||||
|
||||
await db.execute(
|
||||
delete(VesselPosition).where(VesselPosition.received_at < now - timedelta(hours=24))
|
||||
)
|
||||
await db.commit()
|
||||
await self.update_progress(records_added, force=True)
|
||||
return records_added
|
||||
|
||||
def _normalize_record(self, item: dict[str, Any]) -> dict[str, Any] | None:
|
||||
mmsi = _as_int(_pick(item, "mmsi", "MMSI", "Mmsi"))
|
||||
lat = _as_float(_pick(item, "lat", "latitude", "Latitude"))
|
||||
lon = _as_float(_pick(item, "lon", "lng", "longitude", "Longitude"))
|
||||
|
||||
geometry = item.get("geometry")
|
||||
coordinates = geometry.get("coordinates") if isinstance(geometry, dict) else None
|
||||
if (lat is None or lon is None) and isinstance(coordinates, list) and len(coordinates) >= 2:
|
||||
lon = _as_float(coordinates[0])
|
||||
lat = _as_float(coordinates[1])
|
||||
|
||||
if mmsi is None or lat is None or lon is None:
|
||||
return None
|
||||
if not (-90 <= lat <= 90 and -180 <= lon <= 180):
|
||||
return None
|
||||
|
||||
vessel_type = _as_int(_pick(item, "vessel_type", "shipType", "ship_type", "ShipType"))
|
||||
vessel_type_name = (
|
||||
_pick(item, "vessel_type_name", "shipTypeName", "ship_type_name", "VesselTypeName")
|
||||
or _vessel_type_name(vessel_type)
|
||||
)
|
||||
received_at = _parse_datetime(_pick(item, "received_at", "timestamp", "time", "msgtime"))
|
||||
|
||||
return {
|
||||
"mmsi": mmsi,
|
||||
"name": _pick(item, "name", "shipName", "ship_name", "Name"),
|
||||
"callsign": _pick(item, "callsign", "callSign", "CallSign"),
|
||||
"vessel_type": vessel_type,
|
||||
"vessel_type_name": vessel_type_name,
|
||||
"flag": _pick(item, "flag", "country", "Flag"),
|
||||
"length": _as_float(_pick(item, "length", "shipLength", "Length")),
|
||||
"width": _as_float(_pick(item, "width", "shipWidth", "Width")),
|
||||
"draught": _as_float(_pick(item, "draught", "draft", "Draught")),
|
||||
"imo": _as_int(_pick(item, "imo", "IMO", "imoNumber")),
|
||||
"lat": lat,
|
||||
"lon": lon,
|
||||
"sog": _as_float(_pick(item, "sog", "speedOverGround", "SOG")),
|
||||
"cog": _as_float(_pick(item, "cog", "courseOverGround", "COG")),
|
||||
"heading": _as_int(_pick(item, "heading", "trueHeading", "Heading")),
|
||||
"nav_status": _as_int(_pick(item, "nav_status", "navStatus", "NavigationalStatus")),
|
||||
"received_at": received_at,
|
||||
}
|
||||
|
||||
def _get_sample_data(self) -> list[dict[str, Any]]:
|
||||
return [
|
||||
{
|
||||
"mmsi": 257123000,
|
||||
"name": "OSLO TRADER",
|
||||
"lat": 59.91,
|
||||
"lon": 10.73,
|
||||
"sog": 12.4,
|
||||
"cog": 214,
|
||||
"heading": 215,
|
||||
"nav_status": 0,
|
||||
"vessel_type": 70,
|
||||
"vessel_type_name": "Cargo",
|
||||
"flag": "NO",
|
||||
"length": 185,
|
||||
},
|
||||
{
|
||||
"mmsi": 257456000,
|
||||
"name": "NORDIC FJORD",
|
||||
"lat": 60.39,
|
||||
"lon": 5.32,
|
||||
"sog": 0.2,
|
||||
"cog": 82,
|
||||
"heading": 80,
|
||||
"nav_status": 1,
|
||||
"vessel_type": 60,
|
||||
"vessel_type_name": "Passenger",
|
||||
"flag": "NO",
|
||||
"length": 126,
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
def _pick(item: dict[str, Any], *keys: str) -> Any:
|
||||
for key in keys:
|
||||
if key in item and item[key] not in (None, ""):
|
||||
return item[key]
|
||||
return None
|
||||
|
||||
|
||||
def _as_float(value: Any) -> float | None:
|
||||
try:
|
||||
if value in (None, ""):
|
||||
return None
|
||||
return float(value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def _as_int(value: Any) -> int | None:
|
||||
try:
|
||||
if value in (None, ""):
|
||||
return None
|
||||
return int(float(value))
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def _parse_datetime(value: Any) -> datetime | None:
|
||||
if isinstance(value, datetime):
|
||||
return value if value.tzinfo else value.replace(tzinfo=UTC)
|
||||
if not value:
|
||||
return None
|
||||
if isinstance(value, (int, float)):
|
||||
timestamp = float(value)
|
||||
if timestamp > 10_000_000_000:
|
||||
timestamp /= 1000
|
||||
return datetime.fromtimestamp(timestamp, UTC)
|
||||
if isinstance(value, str):
|
||||
try:
|
||||
parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
|
||||
return parsed if parsed.tzinfo else parsed.replace(tzinfo=UTC)
|
||||
except ValueError:
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
def _vessel_type_name(vessel_type: int | None) -> str:
|
||||
if vessel_type is None:
|
||||
return "Other"
|
||||
if 70 <= vessel_type <= 79:
|
||||
return "Cargo"
|
||||
if 80 <= vessel_type <= 89:
|
||||
return "Tanker"
|
||||
if 60 <= vessel_type <= 69:
|
||||
return "Passenger"
|
||||
if vessel_type == 30:
|
||||
return "Fishing"
|
||||
if vessel_type == 35:
|
||||
return "Military"
|
||||
return VESSEL_TYPE_NAMES.get(vessel_type, "Other")
|
||||
@@ -0,0 +1,165 @@
|
||||
"""Credential setup guides for collector integrations."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.schemas.ai import SituationalAnalysisRequest
|
||||
from app.services.ai_client import AIProviderClient
|
||||
|
||||
|
||||
CREDENTIAL_GUIDES_CATEGORY = "collector_credential_guides"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CredentialGuideDefault:
|
||||
provider: str
|
||||
title: str
|
||||
prompt: str
|
||||
markdown: str
|
||||
|
||||
|
||||
BARENTSWATCH_DEFAULT_GUIDE = CredentialGuideDefault(
|
||||
provider="barentswatch",
|
||||
title="BarentsWatch AIS 凭证获取教程",
|
||||
prompt=(
|
||||
"请生成一份中文教程,指导开发者获取 BarentsWatch Live AIS API 的 "
|
||||
"OAuth client credentials。教程要面向已经有本地开发环境的人,包含注册/登录、"
|
||||
"创建 client、申请或确认 ais scope、复制 client id 和 client secret、"
|
||||
"在系统设置中填写并验证连接、常见失败排查。不要编造具体页面按钮文案,"
|
||||
"必须参考官方 tutorial:https://developer.barentswatch.no/docs/tutorial 。"
|
||||
"必须强调 Live AIS 要选择 AIS-client / AIS - API,而不是普通 API-client。"
|
||||
"如果步骤可能变化,要提醒以 BarentsWatch developer portal 当前页面为准。"
|
||||
),
|
||||
markdown="""## BarentsWatch AIS 凭证获取
|
||||
|
||||
官方教程:https://developer.barentswatch.no/docs/tutorial
|
||||
|
||||
1. 先打开上面的 BarentsWatch 官方 tutorial,按官方流程登录或注册开发者账号。
|
||||
2. 在 Developer access 页面选择 `AIS - API`,不要选择普通的 `BarentsWatch - API`。
|
||||
3. 在 `AIS - API` 下创建用于 Planet 的 AIS client。
|
||||
4. 创建时记下你设置的 password / client secret。
|
||||
5. 回到 My Page 复制完整 `Client ID`。它通常长得像 `your.email@example.com:client-name`。
|
||||
6. 回到 Planet 的 `设置 -> 采集器设置 -> BarentsWatch AIS`,填入 `Client ID` 和 `Client Secret`。
|
||||
7. 点击 `连接` 验证 token 和 AIS endpoint 是否可访问。
|
||||
8. 连接成功后保存凭证。
|
||||
|
||||
### 请求规则
|
||||
|
||||
- Token 地址:`https://id.barentswatch.no/connect/token`
|
||||
- 请求方式:`POST`
|
||||
- Content-Type:`application/x-www-form-urlencoded`
|
||||
- Body 必须包含:`grant_type=client_credentials`、`client_id`、`client_secret`、`scope=ais`
|
||||
- `client_id`、`client_secret`、`scope`、`grant_type` 都要放在 body,不要放在 header。
|
||||
- AIS 数据请求使用 header:`Authorization: Bearer <access_token>`
|
||||
|
||||
### 常见排查
|
||||
|
||||
- `未找到凭证`:确认 `Client ID` 和 `Client Secret` 已填写,或已经写入 `~/.zshrc`。
|
||||
- `HTTP 401/403`:通常是选成了普通 `BarentsWatch - API` client、client secret 错误,或 token 请求没有使用 `scope=ais`。
|
||||
- `network` 错误:检查本机是否能访问 `id.barentswatch.no` 和 `live.ais.barentswatch.no`。
|
||||
- Endpoint 建议保持默认:`https://live.ais.barentswatch.no/v1/latest/combined`。
|
||||
""",
|
||||
)
|
||||
|
||||
|
||||
DEFAULT_CREDENTIAL_GUIDES = {
|
||||
BARENTSWATCH_DEFAULT_GUIDE.provider: BARENTSWATCH_DEFAULT_GUIDE,
|
||||
}
|
||||
|
||||
|
||||
async def _get_guide_store(db) -> tuple[SystemSetting | None, dict[str, Any]]:
|
||||
result = await db.execute(
|
||||
select(SystemSetting).where(SystemSetting.category == CREDENTIAL_GUIDES_CATEGORY)
|
||||
)
|
||||
record = result.scalar_one_or_none()
|
||||
payload = dict(record.payload or {}) if record and isinstance(record.payload, dict) else {}
|
||||
return record, payload
|
||||
|
||||
|
||||
async def get_credential_guide(db, provider: str) -> dict[str, Any]:
|
||||
default = DEFAULT_CREDENTIAL_GUIDES.get(provider)
|
||||
if default is None:
|
||||
raise ValueError(f"Unsupported credential guide provider: {provider}")
|
||||
|
||||
_record, store = await _get_guide_store(db)
|
||||
custom = store.get(provider) if isinstance(store.get(provider), dict) else None
|
||||
return {
|
||||
"provider": provider,
|
||||
"title": custom.get("title") if custom else default.title,
|
||||
"markdown": custom.get("markdown") if custom else default.markdown,
|
||||
"prompt": default.prompt,
|
||||
"source": "ai" if custom else "default",
|
||||
}
|
||||
|
||||
|
||||
async def save_credential_guide(db, provider: str, title: str, markdown: str) -> dict[str, Any]:
|
||||
default = DEFAULT_CREDENTIAL_GUIDES.get(provider)
|
||||
if default is None:
|
||||
raise ValueError(f"Unsupported credential guide provider: {provider}")
|
||||
|
||||
record, store = await _get_guide_store(db)
|
||||
store[provider] = {
|
||||
"title": title or default.title,
|
||||
"markdown": markdown,
|
||||
}
|
||||
if record is None:
|
||||
db.add(SystemSetting(category=CREDENTIAL_GUIDES_CATEGORY, payload=store))
|
||||
else:
|
||||
record.payload = store
|
||||
await db.commit()
|
||||
return await get_credential_guide(db, provider)
|
||||
|
||||
|
||||
async def reset_credential_guide(db, provider: str) -> dict[str, Any]:
|
||||
default = DEFAULT_CREDENTIAL_GUIDES.get(provider)
|
||||
if default is None:
|
||||
raise ValueError(f"Unsupported credential guide provider: {provider}")
|
||||
|
||||
record, store = await _get_guide_store(db)
|
||||
if provider in store:
|
||||
store.pop(provider, None)
|
||||
if record is not None:
|
||||
record.payload = store
|
||||
await db.commit()
|
||||
return await get_credential_guide(db, provider)
|
||||
|
||||
|
||||
async def generate_credential_guide(
|
||||
db,
|
||||
provider: str,
|
||||
ai_client: AIProviderClient,
|
||||
) -> dict[str, Any]:
|
||||
default = DEFAULT_CREDENTIAL_GUIDES.get(provider)
|
||||
if default is None:
|
||||
raise ValueError(f"Unsupported credential guide provider: {provider}")
|
||||
|
||||
response = await ai_client.analyze(
|
||||
SituationalAnalysisRequest(
|
||||
title=f"Generate credential guide for {provider}",
|
||||
objective=default.prompt,
|
||||
context={
|
||||
"provider": provider,
|
||||
"current_default_guide": default.markdown,
|
||||
"product_context": "Planet collector credential settings",
|
||||
},
|
||||
observations=[
|
||||
"Use concise Chinese markdown.",
|
||||
"Prefer stable concepts over brittle UI labels.",
|
||||
"Include verification and troubleshooting steps.",
|
||||
],
|
||||
constraints=[
|
||||
"Do not ask the user for secrets.",
|
||||
"Do not include fabricated screenshots.",
|
||||
"Return markdown only.",
|
||||
],
|
||||
)
|
||||
)
|
||||
markdown = response.content.strip()
|
||||
if not markdown:
|
||||
markdown = default.markdown
|
||||
return await save_credential_guide(db, provider, default.title, markdown)
|
||||
@@ -0,0 +1,384 @@
|
||||
"""Connectivity validation helpers for built-in datasource overrides."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
from sqlalchemy import func, select
|
||||
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
from app.core.datasource_defaults import DEFAULT_DATASOURCES
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.models.datasource import DataSource
|
||||
from app.models.datasource_config import DataSourceConfig
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.services.barentswatch import (
|
||||
_read_zshrc_env,
|
||||
fetch_barentswatch_access_token,
|
||||
resolve_barentswatch_config,
|
||||
)
|
||||
|
||||
|
||||
CONNECTIVITY_VALIDATION_KEY = "connectivity_validation"
|
||||
CONNECTIVITY_STORE_CATEGORY = "datasource_connectivity_validations"
|
||||
|
||||
|
||||
def _sha256_json(payload: Any) -> str:
|
||||
encoded = json.dumps(payload, ensure_ascii=False, sort_keys=True, default=str).encode("utf-8")
|
||||
return hashlib.sha256(encoded).hexdigest()
|
||||
|
||||
|
||||
def _resolve_spacetrack_credentials() -> tuple[str, str, str]:
|
||||
zshrc_env = _read_zshrc_env()
|
||||
username = os.getenv("SPACETRACK_USERNAME") or zshrc_env.get("SPACETRACK_USERNAME") or ""
|
||||
password = os.getenv("SPACETRACK_PASSWORD") or zshrc_env.get("SPACETRACK_PASSWORD") or ""
|
||||
source = "environment" if os.getenv("SPACETRACK_USERNAME") or os.getenv("SPACETRACK_PASSWORD") else ""
|
||||
if not source and (username or password):
|
||||
source = "~/.zshrc"
|
||||
return username, password, source or "missing"
|
||||
|
||||
|
||||
def strip_connectivity_validation(config: dict | None) -> dict:
|
||||
cleaned = dict(config or {})
|
||||
cleaned.pop(CONNECTIVITY_VALIDATION_KEY, None)
|
||||
return cleaned
|
||||
|
||||
|
||||
def merge_connectivity_validation(existing_config: dict | None, next_config: dict | None) -> dict:
|
||||
merged = strip_connectivity_validation(next_config)
|
||||
validation = (existing_config or {}).get(CONNECTIVITY_VALIDATION_KEY)
|
||||
if validation:
|
||||
merged[CONNECTIVITY_VALIDATION_KEY] = validation
|
||||
return merged
|
||||
|
||||
|
||||
def get_connectivity_validation(config: DataSourceConfig | None) -> dict | None:
|
||||
validation = (config.config or {}).get(CONNECTIVITY_VALIDATION_KEY) if config else None
|
||||
return validation if isinstance(validation, dict) else None
|
||||
|
||||
|
||||
async def build_builtin_connectivity_checksum(
|
||||
source: str,
|
||||
endpoint: str,
|
||||
auth_type: str,
|
||||
headers: dict | None,
|
||||
config: dict | None,
|
||||
db=None,
|
||||
credential_override: dict[str, str] | None = None,
|
||||
) -> tuple[str, dict[str, Any]]:
|
||||
defaults = DEFAULT_DATASOURCES.get(source, {})
|
||||
credential_provider = defaults.get("credential_provider")
|
||||
credential_fingerprint = ""
|
||||
credential_source = "none"
|
||||
has_credentials = not defaults.get("requires_credentials", False)
|
||||
|
||||
if credential_provider == "barentswatch":
|
||||
if credential_override:
|
||||
client_id = credential_override.get("client_id", "")
|
||||
client_secret = credential_override.get("client_secret", "")
|
||||
credential_source = "draft"
|
||||
else:
|
||||
barentswatch_config = await resolve_barentswatch_config(db)
|
||||
client_id = barentswatch_config.client_id
|
||||
client_secret = barentswatch_config.client_secret
|
||||
credential_source = barentswatch_config.credential_source
|
||||
has_credentials = bool(client_id and client_secret)
|
||||
credential_fingerprint = _sha256_json(
|
||||
{
|
||||
"client_id": client_id,
|
||||
"client_secret": client_secret,
|
||||
}
|
||||
)
|
||||
elif credential_provider == "spacetrack":
|
||||
username, password, credential_source = _resolve_spacetrack_credentials()
|
||||
has_credentials = bool(username and password)
|
||||
credential_fingerprint = _sha256_json(
|
||||
{
|
||||
"username": username,
|
||||
"password": password,
|
||||
}
|
||||
)
|
||||
elif defaults.get("requires_credentials"):
|
||||
credential_source = str(credential_provider or "unsupported")
|
||||
|
||||
checksum_payload = {
|
||||
"source": source,
|
||||
"endpoint": endpoint,
|
||||
"auth_type": "none",
|
||||
"headers": headers or {},
|
||||
"config": strip_connectivity_validation(config),
|
||||
"credential_provider": credential_provider or "none",
|
||||
"credential_fingerprint": credential_fingerprint,
|
||||
}
|
||||
return _sha256_json(checksum_payload), {
|
||||
"requires_credentials": bool(defaults.get("requires_credentials", False)),
|
||||
"credential_provider": credential_provider,
|
||||
"credential_source": credential_source,
|
||||
"has_credentials": has_credentials,
|
||||
}
|
||||
|
||||
|
||||
async def test_builtin_connectivity(
|
||||
source: str,
|
||||
endpoint: str,
|
||||
auth_type: str,
|
||||
headers: dict | None,
|
||||
config: dict | None,
|
||||
db=None,
|
||||
) -> dict[str, Any]:
|
||||
defaults = DEFAULT_DATASOURCES.get(source)
|
||||
if not defaults:
|
||||
return {
|
||||
"success": False,
|
||||
"message": "未知内置采集器,无法执行连接校验。",
|
||||
}
|
||||
|
||||
checksum, credential_context = await build_builtin_connectivity_checksum(
|
||||
source,
|
||||
endpoint,
|
||||
auth_type,
|
||||
headers,
|
||||
config,
|
||||
db,
|
||||
)
|
||||
if credential_context["requires_credentials"] and not credential_context["has_credentials"]:
|
||||
return {
|
||||
"success": False,
|
||||
"checksum": checksum,
|
||||
"stage": "credentials",
|
||||
"message": "该采集器需要凭证,请先到采集器凭证设置中配置。",
|
||||
"settings_tab": "collector_credentials",
|
||||
**credential_context,
|
||||
}
|
||||
supported_credential_providers = {"barentswatch", "spacetrack"}
|
||||
if (
|
||||
credential_context["requires_credentials"]
|
||||
and credential_context["credential_provider"] not in supported_credential_providers
|
||||
):
|
||||
return {
|
||||
"success": False,
|
||||
"checksum": checksum,
|
||||
"stage": "credentials",
|
||||
"message": "该采集器的凭证链路尚未接入,暂时无法完成连接校验。",
|
||||
"settings_tab": "collector_credentials",
|
||||
**credential_context,
|
||||
}
|
||||
|
||||
request_headers = {str(key): str(value) for key, value in (headers or {}).items()}
|
||||
request_config = strip_connectivity_validation(config)
|
||||
timeout = float(request_config.get("timeout") or 30)
|
||||
request_endpoint = endpoint
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=timeout, follow_redirects=True) as client:
|
||||
if credential_context["credential_provider"] == "barentswatch":
|
||||
barentswatch_config = await resolve_barentswatch_config(db)
|
||||
token = await fetch_barentswatch_access_token(client, barentswatch_config)
|
||||
if not token:
|
||||
return {
|
||||
"success": False,
|
||||
"checksum": checksum,
|
||||
"stage": "token",
|
||||
"message": "凭证可读取,但 token 响应中没有 access_token。",
|
||||
"settings_tab": "collector_credentials",
|
||||
**credential_context,
|
||||
}
|
||||
request_headers["Authorization"] = f"Bearer {token}"
|
||||
elif credential_context["credential_provider"] == "spacetrack":
|
||||
username, password, _source = _resolve_spacetrack_credentials()
|
||||
login_url = "https://www.space-track.org/ajaxauth/login"
|
||||
login_response = await client.post(
|
||||
login_url,
|
||||
data={
|
||||
"identity": username,
|
||||
"password": password,
|
||||
},
|
||||
)
|
||||
login_response.raise_for_status()
|
||||
|
||||
started = datetime.now(UTC)
|
||||
async with client.stream("GET", request_endpoint, headers=request_headers) as response:
|
||||
response.raise_for_status()
|
||||
status_code = response.status_code
|
||||
elapsed_ms = (datetime.now(UTC) - started).total_seconds() * 1000
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"checksum": checksum,
|
||||
"stage": "endpoint",
|
||||
"message": "连接验证成功。",
|
||||
"status_code": status_code,
|
||||
"response_time_ms": elapsed_ms,
|
||||
**credential_context,
|
||||
}
|
||||
except httpx.HTTPStatusError as exc:
|
||||
return {
|
||||
"success": False,
|
||||
"checksum": checksum,
|
||||
"stage": "endpoint",
|
||||
"message": f"连接验证失败:HTTP {exc.response.status_code}",
|
||||
"error": f"HTTP Error: {exc.response.status_code}",
|
||||
**credential_context,
|
||||
}
|
||||
except httpx.HTTPError as exc:
|
||||
return {
|
||||
"success": False,
|
||||
"checksum": checksum,
|
||||
"stage": "network",
|
||||
"message": f"连接验证失败:{exc.__class__.__name__}",
|
||||
"error": str(exc),
|
||||
**credential_context,
|
||||
}
|
||||
|
||||
|
||||
def make_success_validation(checksum: str, result: dict[str, Any]) -> dict[str, Any]:
|
||||
return {
|
||||
"checksum": checksum,
|
||||
"status": "success",
|
||||
"validated_at": datetime.now(UTC).isoformat(),
|
||||
"status_code": result.get("status_code"),
|
||||
"credential_source": result.get("credential_source"),
|
||||
}
|
||||
|
||||
|
||||
def is_builtin_validation_current(config: DataSourceConfig | None, checksum: str) -> bool:
|
||||
validation = get_connectivity_validation(config)
|
||||
return bool(
|
||||
validation
|
||||
and validation.get("status") == "success"
|
||||
and validation.get("checksum") == checksum
|
||||
)
|
||||
|
||||
|
||||
async def get_connectivity_store(db) -> dict[str, Any]:
|
||||
result = await db.execute(
|
||||
select(SystemSetting).where(SystemSetting.category == CONNECTIVITY_STORE_CATEGORY)
|
||||
)
|
||||
record = result.scalar_one_or_none()
|
||||
return dict(record.payload or {}) if record and isinstance(record.payload, dict) else {}
|
||||
|
||||
|
||||
async def save_connectivity_success(
|
||||
db,
|
||||
source: str,
|
||||
checksum: str,
|
||||
result: dict[str, Any],
|
||||
*,
|
||||
connected_by: str,
|
||||
) -> dict[str, Any]:
|
||||
store = await get_connectivity_store(db)
|
||||
validation = {
|
||||
**make_success_validation(checksum, result),
|
||||
"connected_by": connected_by,
|
||||
}
|
||||
store[source] = validation
|
||||
|
||||
existing = await db.execute(
|
||||
select(SystemSetting).where(SystemSetting.category == CONNECTIVITY_STORE_CATEGORY)
|
||||
)
|
||||
record = existing.scalar_one_or_none()
|
||||
if record is None:
|
||||
db.add(SystemSetting(category=CONNECTIVITY_STORE_CATEGORY, payload=store))
|
||||
else:
|
||||
record.payload = store
|
||||
return validation
|
||||
|
||||
|
||||
async def load_builtin_override_config(db, source: str) -> DataSourceConfig | None:
|
||||
result = await db.execute(
|
||||
select(DataSourceConfig)
|
||||
.where(DataSourceConfig.name == source)
|
||||
.where(DataSourceConfig.is_active.is_(True))
|
||||
)
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
|
||||
async def get_builtin_effective_candidate(db, source: str) -> dict[str, Any]:
|
||||
override = await load_builtin_override_config(db, source)
|
||||
default_endpoint = get_data_sources_config().get_yaml_url(source)
|
||||
return {
|
||||
"name": source,
|
||||
"endpoint": (override.endpoint if override and override.endpoint else default_endpoint) or "",
|
||||
"auth_type": override.auth_type if override else "none",
|
||||
"headers": override.headers if override else {},
|
||||
"config": strip_connectivity_validation(override.config if override else {}),
|
||||
}
|
||||
|
||||
|
||||
async def has_collected_data(db, source: str) -> bool:
|
||||
result = await db.execute(select(func.count(CollectedData.id)).where(CollectedData.source == source))
|
||||
if (result.scalar() or 0) > 0:
|
||||
return True
|
||||
|
||||
datasource_result = await db.execute(select(DataSource).where(DataSource.source == source))
|
||||
datasource = datasource_result.scalar_one_or_none()
|
||||
return bool(datasource and datasource.last_status == "success")
|
||||
|
||||
|
||||
async def get_builtin_connection_status(
|
||||
db,
|
||||
source: str,
|
||||
endpoint: str,
|
||||
auth_type: str,
|
||||
headers: dict | None,
|
||||
config: dict | None,
|
||||
) -> dict[str, Any]:
|
||||
checksum, credential_context = await build_builtin_connectivity_checksum(
|
||||
source,
|
||||
endpoint,
|
||||
auth_type,
|
||||
headers,
|
||||
config,
|
||||
db,
|
||||
)
|
||||
store = await get_connectivity_store(db)
|
||||
validation = store.get(source)
|
||||
if isinstance(validation, dict) and validation.get("status") == "success":
|
||||
if validation.get("checksum") == checksum:
|
||||
return {
|
||||
"connected": True,
|
||||
"checksum": checksum,
|
||||
"connected_by": validation.get("connected_by") or "connection_button",
|
||||
"message": "当前配置已完成连接验证。",
|
||||
**credential_context,
|
||||
}
|
||||
|
||||
effective = await get_builtin_effective_candidate(db, source)
|
||||
effective_checksum, _ = await build_builtin_connectivity_checksum(
|
||||
source,
|
||||
effective["endpoint"],
|
||||
effective["auth_type"],
|
||||
effective["headers"],
|
||||
effective["config"],
|
||||
db,
|
||||
)
|
||||
if checksum == effective_checksum and await has_collected_data(db, source):
|
||||
return {
|
||||
"connected": True,
|
||||
"checksum": checksum,
|
||||
"connected_by": "collection",
|
||||
"message": "当前配置已有成功采集数据,视为已连接。",
|
||||
**credential_context,
|
||||
}
|
||||
|
||||
if isinstance(validation, dict) and validation.get("status") == "success":
|
||||
return {
|
||||
"connected": False,
|
||||
"checksum": checksum,
|
||||
"connected_by": None,
|
||||
"message": "接口地址或凭证指纹已变化,请重新点击连接验证。",
|
||||
**credential_context,
|
||||
}
|
||||
|
||||
return {
|
||||
"connected": False,
|
||||
"checksum": checksum,
|
||||
"connected_by": None,
|
||||
"message": "当前配置尚未连接,请点击连接验证。",
|
||||
**credential_context,
|
||||
}
|
||||
@@ -0,0 +1,358 @@
|
||||
"""Deterministic mapping support for custom data sources."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.target_schema_registry import TargetSchema, get_target_schema
|
||||
|
||||
SECRET_KEY_PATTERN = re.compile(
|
||||
r"(token|secret|password|passwd|authorization|api[_-]?key|client[_-]?secret)",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
|
||||
class MappingError(ValueError):
|
||||
"""Raised when a mapping definition cannot be executed."""
|
||||
|
||||
|
||||
def stable_payload_hash(payload: Any) -> str:
|
||||
encoded = json.dumps(payload, ensure_ascii=False, sort_keys=True, default=str).encode()
|
||||
return hashlib.sha256(encoded).hexdigest()
|
||||
|
||||
|
||||
def redact_for_llm(value: Any) -> Any:
|
||||
if isinstance(value, dict):
|
||||
redacted = {}
|
||||
for key, item in value.items():
|
||||
if SECRET_KEY_PATTERN.search(str(key)):
|
||||
redacted[key] = "[REDACTED]"
|
||||
else:
|
||||
redacted[key] = redact_for_llm(item)
|
||||
return redacted
|
||||
if isinstance(value, list):
|
||||
return [redact_for_llm(item) for item in value[:20]]
|
||||
return value
|
||||
|
||||
|
||||
def extract_path(payload: Any, path: str | None) -> Any:
|
||||
if not path or path == "$":
|
||||
return payload
|
||||
|
||||
normalized = path.strip()
|
||||
if normalized.startswith("$."):
|
||||
normalized = normalized[2:]
|
||||
elif normalized.startswith("$"):
|
||||
normalized = normalized[1:]
|
||||
normalized = normalized.strip(".")
|
||||
if not normalized:
|
||||
return payload
|
||||
|
||||
current = payload
|
||||
for raw_segment in normalized.split("."):
|
||||
segment = raw_segment.strip()
|
||||
if not segment:
|
||||
continue
|
||||
|
||||
list_all = segment.endswith("[*]")
|
||||
if list_all:
|
||||
segment = segment[:-3]
|
||||
|
||||
index = None
|
||||
match = re.fullmatch(r"(.+)\[(\d+)\]", segment)
|
||||
if match:
|
||||
segment = match.group(1)
|
||||
index = int(match.group(2))
|
||||
|
||||
if segment:
|
||||
if isinstance(current, dict):
|
||||
current = current.get(segment)
|
||||
else:
|
||||
return None
|
||||
|
||||
if list_all:
|
||||
return current if isinstance(current, list) else []
|
||||
|
||||
if index is not None:
|
||||
if not isinstance(current, list) or index >= len(current):
|
||||
return None
|
||||
current = current[index]
|
||||
|
||||
return current
|
||||
|
||||
|
||||
def _convert_value(value: Any, target_type: str | None) -> Any:
|
||||
if value is None or target_type in (None, "", "any"):
|
||||
return value
|
||||
|
||||
if target_type == "string":
|
||||
return str(value)
|
||||
if target_type == "integer":
|
||||
return int(value)
|
||||
if target_type == "float":
|
||||
return float(value)
|
||||
if target_type == "boolean":
|
||||
if isinstance(value, bool):
|
||||
return value
|
||||
if isinstance(value, str):
|
||||
return value.strip().lower() in {"1", "true", "yes", "y", "on"}
|
||||
return bool(value)
|
||||
if target_type == "datetime":
|
||||
if isinstance(value, datetime):
|
||||
return value
|
||||
if isinstance(value, (int, float)):
|
||||
return datetime.fromtimestamp(value)
|
||||
if isinstance(value, str):
|
||||
return datetime.fromisoformat(value.replace("Z", "+00:00"))
|
||||
return value
|
||||
if target_type == "object":
|
||||
if isinstance(value, dict):
|
||||
return value
|
||||
raise ValueError("expected object")
|
||||
if target_type == "array":
|
||||
if isinstance(value, list):
|
||||
return value
|
||||
raise ValueError("expected array")
|
||||
|
||||
return value
|
||||
|
||||
|
||||
def _apply_enum(value: Any, enum_map: Any) -> Any:
|
||||
if not isinstance(enum_map, dict):
|
||||
return value
|
||||
key = str(value)
|
||||
return enum_map.get(key, enum_map.get(value, value))
|
||||
|
||||
|
||||
def _map_one(item: Any, field_mapping: dict[str, Any]) -> tuple[dict[str, Any], list[str]]:
|
||||
output: dict[str, Any] = {}
|
||||
errors: list[str] = []
|
||||
|
||||
for field_name, rule in field_mapping.items():
|
||||
if isinstance(rule, str):
|
||||
rule = {"path": rule}
|
||||
if not isinstance(rule, dict):
|
||||
errors.append(f"{field_name}: mapping rule must be an object or path string")
|
||||
continue
|
||||
|
||||
value = extract_path(item, rule.get("path"))
|
||||
if value is None and "default" in rule:
|
||||
value = rule.get("default")
|
||||
value = _apply_enum(value, rule.get("enum"))
|
||||
|
||||
try:
|
||||
value = _convert_value(value, rule.get("type"))
|
||||
except (TypeError, ValueError) as exc:
|
||||
errors.append(f"{field_name}: failed to convert value {value!r}: {exc}")
|
||||
continue
|
||||
|
||||
if value is not None or rule.get("include_null", False):
|
||||
output[field_name] = value
|
||||
|
||||
return output, errors
|
||||
|
||||
|
||||
def execute_mapping(
|
||||
payload: Any,
|
||||
mapping_json: dict[str, Any],
|
||||
target_schema: str | TargetSchema,
|
||||
*,
|
||||
limit: int | None = None,
|
||||
) -> dict[str, Any]:
|
||||
schema = get_target_schema(target_schema) if isinstance(target_schema, str) else target_schema
|
||||
source = mapping_json.get("source") or {}
|
||||
fields = mapping_json.get("fields")
|
||||
if not isinstance(fields, dict) or not fields:
|
||||
raise MappingError("mapping_json.fields must be a non-empty object")
|
||||
|
||||
items_path = source.get("items_path") or mapping_json.get("items_path") or "$"
|
||||
items = extract_path(payload, items_path)
|
||||
if isinstance(items, dict):
|
||||
items = [items]
|
||||
elif not isinstance(items, list):
|
||||
items = []
|
||||
|
||||
if limit is not None:
|
||||
items = items[:limit]
|
||||
|
||||
mapped_records: list[dict[str, Any]] = []
|
||||
errors: list[dict[str, Any]] = []
|
||||
for index, item in enumerate(items):
|
||||
mapped, mapping_errors = _map_one(item, fields)
|
||||
validated, validation_errors = schema.validate_record(mapped)
|
||||
all_errors = mapping_errors + validation_errors
|
||||
if all_errors:
|
||||
errors.append({"index": index, "errors": all_errors, "record": mapped})
|
||||
continue
|
||||
if validated is not None:
|
||||
mapped_records.append(validated)
|
||||
|
||||
return {
|
||||
"target_schema": schema.key,
|
||||
"total_items": len(items),
|
||||
"mapped_count": len(mapped_records),
|
||||
"failed_count": len(errors),
|
||||
"records": mapped_records,
|
||||
"errors": errors,
|
||||
}
|
||||
|
||||
|
||||
def build_heuristic_mapping(sample_payload: Any, target_schema_key: str) -> dict[str, Any]:
|
||||
schema = get_target_schema(target_schema_key)
|
||||
items_path = "$"
|
||||
sample_item = sample_payload
|
||||
if isinstance(sample_payload, dict):
|
||||
for key in ("data", "items", "results", "features", "vessels"):
|
||||
candidate = sample_payload.get(key)
|
||||
if isinstance(candidate, list) and candidate:
|
||||
items_path = f"$.{key}[*]"
|
||||
sample_item = candidate[0]
|
||||
break
|
||||
elif isinstance(sample_payload, list) and sample_payload:
|
||||
items_path = "$"
|
||||
sample_item = sample_payload[0]
|
||||
|
||||
available = _flatten_keys(sample_item if isinstance(sample_item, dict) else {})
|
||||
fields: dict[str, Any] = {}
|
||||
for field in schema.fields:
|
||||
candidate = _best_field_match(field.name, available)
|
||||
if candidate:
|
||||
fields[field.name] = {"path": f"$.{candidate}", "type": field.type}
|
||||
elif field.name == "data" and target_schema_key == "generic_records":
|
||||
fields[field.name] = {"path": "$", "type": "object"}
|
||||
elif not field.required:
|
||||
fields[field.name] = {"path": f"$.{field.name}", "type": field.type, "default": None}
|
||||
|
||||
return {
|
||||
"source": {"items_path": items_path},
|
||||
"fields": fields,
|
||||
"meta": {
|
||||
"generated_by": "heuristic",
|
||||
"requires_review": True,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def _flatten_keys(payload: dict[str, Any], prefix: str = "") -> list[str]:
|
||||
keys: list[str] = []
|
||||
for key, value in payload.items():
|
||||
dotted = f"{prefix}.{key}" if prefix else str(key)
|
||||
keys.append(dotted)
|
||||
if isinstance(value, dict):
|
||||
keys.extend(_flatten_keys(value, dotted))
|
||||
return keys
|
||||
|
||||
|
||||
def _best_field_match(field_name: str, candidates: list[str]) -> str | None:
|
||||
aliases = {
|
||||
"lat": ("lat", "latitude", "y"),
|
||||
"lon": ("lon", "lng", "longitude", "x"),
|
||||
"mmsi": ("mmsi",),
|
||||
"sog": ("sog", "speed", "speedOverGround"),
|
||||
"cog": ("cog", "course", "courseOverGround"),
|
||||
"received_at": ("received_at", "timestamp", "time", "updated_at"),
|
||||
"observed_at": ("observed_at", "timestamp", "time", "updated_at"),
|
||||
"source_id": ("id", "source_id", "uuid"),
|
||||
}.get(field_name, (field_name,))
|
||||
|
||||
lowered = {candidate.lower(): candidate for candidate in candidates}
|
||||
for alias in aliases:
|
||||
if alias.lower() in lowered:
|
||||
return lowered[alias.lower()]
|
||||
for candidate in candidates:
|
||||
tail = candidate.split(".")[-1].lower()
|
||||
if tail in {alias.lower() for alias in aliases}:
|
||||
return candidate
|
||||
return None
|
||||
|
||||
|
||||
def _parse_datetime(value: Any) -> datetime | None:
|
||||
if value is None:
|
||||
return None
|
||||
if isinstance(value, datetime):
|
||||
return value
|
||||
if isinstance(value, str):
|
||||
return datetime.fromisoformat(value.replace("Z", "+00:00"))
|
||||
return None
|
||||
|
||||
|
||||
async def persist_mapped_records(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
datasource_name: str,
|
||||
datasource_config_id: int,
|
||||
target_schema: str,
|
||||
records: list[dict[str, Any]],
|
||||
mapping_version: int,
|
||||
) -> int:
|
||||
"""Persist validated mapped records to the destination for a target schema."""
|
||||
if target_schema == "vessel_ais":
|
||||
from app.models.vessel import VesselPosition
|
||||
|
||||
for record in records:
|
||||
db.add(
|
||||
VesselPosition(
|
||||
mmsi=record["mmsi"],
|
||||
lat=record["lat"],
|
||||
lon=record["lon"],
|
||||
sog=record.get("sog"),
|
||||
cog=record.get("cog"),
|
||||
heading=record.get("heading"),
|
||||
received_at=_parse_datetime(record.get("received_at")) or datetime.now(UTC),
|
||||
)
|
||||
)
|
||||
await db.commit()
|
||||
return len(records)
|
||||
|
||||
from app.models.collected_data import CollectedData
|
||||
|
||||
collected_at = datetime.now(UTC)
|
||||
for index, record in enumerate(records):
|
||||
if target_schema == "geo_points":
|
||||
source_id = record.get("source_id") or f"{datasource_config_id}:{index}"
|
||||
name = record.get("name")
|
||||
metadata = {
|
||||
"latitude": record.get("lat"),
|
||||
"longitude": record.get("lon"),
|
||||
"type": record.get("type"),
|
||||
"mapping_version": mapping_version,
|
||||
"target_schema": target_schema,
|
||||
**(record.get("metadata") or {}),
|
||||
}
|
||||
reference_date = _parse_datetime(record.get("observed_at"))
|
||||
else:
|
||||
source_id = record.get("source_id") or f"{datasource_config_id}:{index}"
|
||||
name = None
|
||||
metadata = {
|
||||
"data": record.get("data") or {},
|
||||
"mapping_version": mapping_version,
|
||||
"target_schema": target_schema,
|
||||
}
|
||||
reference_date = _parse_datetime(record.get("observed_at"))
|
||||
|
||||
db.add(
|
||||
CollectedData(
|
||||
source=datasource_name,
|
||||
source_id=str(source_id),
|
||||
entity_key=f"{datasource_name}:{source_id}",
|
||||
data_type=target_schema,
|
||||
name=name,
|
||||
title=name,
|
||||
extra_data=metadata,
|
||||
collected_at=collected_at,
|
||||
reference_date=reference_date,
|
||||
is_valid=1,
|
||||
is_current=True,
|
||||
change_type="created",
|
||||
change_summary={},
|
||||
)
|
||||
)
|
||||
|
||||
await db.commit()
|
||||
return len(records)
|
||||
@@ -0,0 +1,540 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
from email.utils import parsedate_to_datetime
|
||||
import hashlib
|
||||
import html
|
||||
import re
|
||||
from typing import Any
|
||||
from urllib.parse import quote
|
||||
import xml.etree.ElementTree as ET
|
||||
|
||||
import httpx
|
||||
from bs4 import BeautifulSoup
|
||||
|
||||
|
||||
USER_AGENT = "PlanetEarthNewsBoard/1.0 (+https://planet.local)"
|
||||
REQUEST_TIMEOUT = 12.0
|
||||
MAX_ITEMS_PER_SOURCE = 6
|
||||
MAX_ITEMS_TOTAL = 12
|
||||
STALE_CACHE_MAX_AGE_SECONDS = 60 * 45
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RegionProfile:
|
||||
key: str
|
||||
label: str
|
||||
query: str
|
||||
accent: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RegionAnchor:
|
||||
region: str
|
||||
label: str
|
||||
latitude: float
|
||||
longitude: float
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class NewsFeedSource:
|
||||
id: str
|
||||
name: str
|
||||
region: str
|
||||
feed_url: str
|
||||
homepage_url: str
|
||||
source_type: str = "rss"
|
||||
priority: int = 100
|
||||
|
||||
|
||||
@dataclass
|
||||
class ParsedNewsItem:
|
||||
id: str
|
||||
title: str
|
||||
summary: str
|
||||
url: str
|
||||
source: str
|
||||
feed_name: str
|
||||
feed_region: str
|
||||
homepage_url: str
|
||||
published_at: datetime | None
|
||||
|
||||
|
||||
@dataclass
|
||||
class CachedRegionFeed:
|
||||
region: str
|
||||
fetched_at: datetime
|
||||
items: list[ParsedNewsItem]
|
||||
sources: list[NewsFeedSource]
|
||||
|
||||
|
||||
REGION_PROFILES: dict[str, RegionProfile] = {
|
||||
"americas": RegionProfile(
|
||||
key="americas",
|
||||
label="美洲焦点",
|
||||
query='Americas geopolitics OR Latin America OR "United States" OR Canada',
|
||||
accent="#79d3ff",
|
||||
),
|
||||
"europe": RegionProfile(
|
||||
key="europe",
|
||||
label="欧洲焦点",
|
||||
query='Europe geopolitics OR EU OR NATO OR "Eastern Europe"',
|
||||
accent="#8fd4ff",
|
||||
),
|
||||
"middle-east-africa": RegionProfile(
|
||||
key="middle-east-africa",
|
||||
label="中东与非洲焦点",
|
||||
query='"Middle East" OR Africa geopolitics OR Red Sea OR Gulf',
|
||||
accent="#ffb56a",
|
||||
),
|
||||
"asia-pacific": RegionProfile(
|
||||
key="asia-pacific",
|
||||
label="亚太焦点",
|
||||
query='"Asia Pacific" OR Indo-Pacific OR China OR Japan OR Korea OR ASEAN',
|
||||
accent="#78f2cf",
|
||||
),
|
||||
"global": RegionProfile(
|
||||
key="global",
|
||||
label="全球焦点",
|
||||
query='"world news" OR geopolitics OR "global affairs"',
|
||||
accent="#d6e6ff",
|
||||
),
|
||||
}
|
||||
|
||||
REGION_ANCHORS: dict[str, RegionAnchor] = {
|
||||
"americas": RegionAnchor(
|
||||
region="americas",
|
||||
label="美洲",
|
||||
latitude=37.0902,
|
||||
longitude=-95.7129,
|
||||
),
|
||||
"europe": RegionAnchor(
|
||||
region="europe",
|
||||
label="欧洲",
|
||||
latitude=50.1109,
|
||||
longitude=8.6821,
|
||||
),
|
||||
"middle-east-africa": RegionAnchor(
|
||||
region="middle-east-africa",
|
||||
label="中东与非洲",
|
||||
latitude=25.2048,
|
||||
longitude=55.2708,
|
||||
),
|
||||
"asia-pacific": RegionAnchor(
|
||||
region="asia-pacific",
|
||||
label="亚太",
|
||||
latitude=1.3521,
|
||||
longitude=103.8198,
|
||||
),
|
||||
"global": RegionAnchor(
|
||||
region="global",
|
||||
label="全球",
|
||||
latitude=20.0,
|
||||
longitude=0.0,
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def _google_news_feed(query: str, *, hl: str, gl: str, ceid: str) -> str:
|
||||
return (
|
||||
"https://news.google.com/rss/search?q="
|
||||
+ quote(query, safe="")
|
||||
+ f"&hl={hl}&gl={gl}&ceid={ceid}"
|
||||
)
|
||||
|
||||
|
||||
NEWS_FEED_SOURCES: tuple[NewsFeedSource, ...] = (
|
||||
NewsFeedSource(
|
||||
id="bbc-world",
|
||||
name="BBC World",
|
||||
region="global",
|
||||
feed_url="https://feeds.bbci.co.uk/news/world/rss.xml",
|
||||
homepage_url="https://www.bbc.com/news/world",
|
||||
priority=10,
|
||||
),
|
||||
NewsFeedSource(
|
||||
id="dw-top",
|
||||
name="DW Top Stories",
|
||||
region="europe",
|
||||
feed_url="https://rss.dw.com/rdf/rss-en-top",
|
||||
homepage_url="https://www.dw.com/en/top-stories/s-9097",
|
||||
priority=20,
|
||||
),
|
||||
NewsFeedSource(
|
||||
id="global-scan",
|
||||
name="Global Monitor / World",
|
||||
region="global",
|
||||
feed_url=_google_news_feed(
|
||||
REGION_PROFILES["global"].query,
|
||||
hl="en-US",
|
||||
gl="US",
|
||||
ceid="US:en",
|
||||
),
|
||||
homepage_url="https://news.google.com/",
|
||||
source_type="aggregated",
|
||||
priority=30,
|
||||
),
|
||||
NewsFeedSource(
|
||||
id="google-americas",
|
||||
name="Global Monitor / Americas",
|
||||
region="americas",
|
||||
feed_url=_google_news_feed(
|
||||
REGION_PROFILES["americas"].query,
|
||||
hl="en-US",
|
||||
gl="US",
|
||||
ceid="US:en",
|
||||
),
|
||||
homepage_url="https://news.google.com/",
|
||||
source_type="aggregated",
|
||||
priority=40,
|
||||
),
|
||||
NewsFeedSource(
|
||||
id="google-europe",
|
||||
name="Global Monitor / Europe",
|
||||
region="europe",
|
||||
feed_url=_google_news_feed(
|
||||
REGION_PROFILES["europe"].query,
|
||||
hl="en-GB",
|
||||
gl="GB",
|
||||
ceid="GB:en",
|
||||
),
|
||||
homepage_url="https://news.google.com/",
|
||||
source_type="aggregated",
|
||||
priority=40,
|
||||
),
|
||||
NewsFeedSource(
|
||||
id="google-mea",
|
||||
name="Global Monitor / MEA",
|
||||
region="middle-east-africa",
|
||||
feed_url=_google_news_feed(
|
||||
REGION_PROFILES["middle-east-africa"].query,
|
||||
hl="en-US",
|
||||
gl="US",
|
||||
ceid="US:en",
|
||||
),
|
||||
homepage_url="https://news.google.com/",
|
||||
source_type="aggregated",
|
||||
priority=40,
|
||||
),
|
||||
NewsFeedSource(
|
||||
id="google-apac",
|
||||
name="Global Monitor / APAC",
|
||||
region="asia-pacific",
|
||||
feed_url=_google_news_feed(
|
||||
REGION_PROFILES["asia-pacific"].query,
|
||||
hl="en-SG",
|
||||
gl="SG",
|
||||
ceid="SG:en",
|
||||
),
|
||||
homepage_url="https://news.google.com/",
|
||||
source_type="aggregated",
|
||||
priority=40,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
_REGION_CACHE: dict[str, CachedRegionFeed] = {}
|
||||
|
||||
|
||||
def determine_focus_region(lat: float | None, lon: float | None) -> str:
|
||||
if lat is None or lon is None:
|
||||
return "global"
|
||||
if -170 <= lon <= -30:
|
||||
return "americas"
|
||||
if -30 < lon <= 45:
|
||||
return "europe" if lat >= 30 else "middle-east-africa"
|
||||
if 45 < lon <= 150:
|
||||
return "middle-east-africa" if lat < 10 else "asia-pacific"
|
||||
return "asia-pacific"
|
||||
|
||||
|
||||
def get_region_profile(region: str) -> RegionProfile:
|
||||
return REGION_PROFILES.get(region, REGION_PROFILES["global"])
|
||||
|
||||
|
||||
def get_region_anchor(region: str) -> RegionAnchor:
|
||||
return REGION_ANCHORS.get(region, REGION_ANCHORS["global"])
|
||||
|
||||
|
||||
def get_sources_for_region(region: str) -> list[NewsFeedSource]:
|
||||
return sorted(
|
||||
[source for source in NEWS_FEED_SOURCES if source.region in {"global", region}],
|
||||
key=lambda source: (source.priority, source.name),
|
||||
)
|
||||
|
||||
|
||||
def _strip_html(value: str) -> str:
|
||||
if not value:
|
||||
return ""
|
||||
soup = BeautifulSoup(value, "html.parser")
|
||||
return re.sub(r"\s+", " ", soup.get_text(" ", strip=True)).strip()
|
||||
|
||||
|
||||
def _truncate(value: str, limit: int = 180) -> str:
|
||||
text = value.strip()
|
||||
if len(text) <= limit:
|
||||
return text
|
||||
return text[: limit - 1].rstrip() + "…"
|
||||
|
||||
|
||||
def _normalize_source_name(raw: str, fallback: str) -> str:
|
||||
text = html.unescape((raw or "").strip())
|
||||
if " - " in text:
|
||||
return text.split(" - ")[-1].strip() or fallback
|
||||
return text or fallback
|
||||
|
||||
|
||||
def _parse_datetime(raw: str | None) -> datetime | None:
|
||||
if not raw:
|
||||
return None
|
||||
text = raw.strip()
|
||||
if not text:
|
||||
return None
|
||||
|
||||
for parser in (
|
||||
lambda value: parsedate_to_datetime(value),
|
||||
lambda value: datetime.fromisoformat(value.replace("Z", "+00:00")),
|
||||
):
|
||||
try:
|
||||
parsed = parser(text)
|
||||
if parsed.tzinfo is None:
|
||||
parsed = parsed.replace(tzinfo=UTC)
|
||||
return parsed.astimezone(UTC)
|
||||
except Exception:
|
||||
continue
|
||||
return None
|
||||
|
||||
|
||||
def _extract_item_text(element: ET.Element, *names: str) -> str:
|
||||
for name in names:
|
||||
node = element.find(name)
|
||||
if node is not None and node.text:
|
||||
return node.text.strip()
|
||||
return ""
|
||||
|
||||
|
||||
def _parse_feed_entries(xml_text: str, source: NewsFeedSource) -> list[ParsedNewsItem]:
|
||||
root = ET.fromstring(xml_text)
|
||||
items: list[ParsedNewsItem] = []
|
||||
|
||||
rss_items = root.findall("./channel/item")
|
||||
atom_entries = root.findall("{http://www.w3.org/2005/Atom}entry")
|
||||
nodes = rss_items or atom_entries
|
||||
|
||||
for node in nodes[:MAX_ITEMS_PER_SOURCE]:
|
||||
if node.tag.endswith("entry"):
|
||||
title = _extract_item_text(node, "{http://www.w3.org/2005/Atom}title")
|
||||
summary = _extract_item_text(
|
||||
node,
|
||||
"{http://www.w3.org/2005/Atom}summary",
|
||||
"{http://www.w3.org/2005/Atom}content",
|
||||
)
|
||||
link_node = node.find("{http://www.w3.org/2005/Atom}link")
|
||||
link = link_node.get("href", "").strip() if link_node is not None else ""
|
||||
published = _extract_item_text(
|
||||
node,
|
||||
"{http://www.w3.org/2005/Atom}updated",
|
||||
"{http://www.w3.org/2005/Atom}published",
|
||||
)
|
||||
else:
|
||||
title = _extract_item_text(node, "title")
|
||||
summary = _extract_item_text(node, "description", "content")
|
||||
link = _extract_item_text(node, "link")
|
||||
published = _extract_item_text(node, "pubDate", "published", "updated")
|
||||
|
||||
clean_title = html.unescape(title).strip()
|
||||
clean_summary = _truncate(_strip_html(summary), 180)
|
||||
if not clean_title or not link:
|
||||
continue
|
||||
|
||||
item_source = _normalize_source_name(clean_title, source.name)
|
||||
display_title = clean_title
|
||||
if source.source_type == "aggregated" and " - " in clean_title:
|
||||
parts = clean_title.rsplit(" - ", 1)
|
||||
display_title = parts[0].strip()
|
||||
item_source = _normalize_source_name(parts[1], source.name)
|
||||
|
||||
items.append(
|
||||
ParsedNewsItem(
|
||||
id=f"{source.id}:{hashlib.sha1(link.encode('utf-8')).hexdigest()[:12]}",
|
||||
title=display_title,
|
||||
summary=clean_summary,
|
||||
url=link,
|
||||
source=item_source,
|
||||
feed_name=source.name,
|
||||
feed_region=source.region,
|
||||
homepage_url=source.homepage_url,
|
||||
published_at=_parse_datetime(published),
|
||||
)
|
||||
)
|
||||
|
||||
return items
|
||||
|
||||
|
||||
def _serialize_sources(sources: list[NewsFeedSource]) -> list[dict[str, Any]]:
|
||||
return [
|
||||
{
|
||||
"id": source.id,
|
||||
"name": source.name,
|
||||
"region": source.region,
|
||||
"homepage_url": source.homepage_url,
|
||||
}
|
||||
for source in sources
|
||||
]
|
||||
|
||||
|
||||
def _serialize_item(item: ParsedNewsItem, *, active_region: str) -> dict[str, Any]:
|
||||
published_at = item.published_at
|
||||
anchor = get_region_anchor(item.feed_region)
|
||||
return {
|
||||
"id": item.id,
|
||||
"title": item.title,
|
||||
"summary": item.summary,
|
||||
"url": item.url,
|
||||
"source": item.source,
|
||||
"feed_name": item.feed_name,
|
||||
"region": item.feed_region,
|
||||
"homepage_url": item.homepage_url,
|
||||
"published_at": published_at.isoformat().replace("+00:00", "Z") if published_at else None,
|
||||
"latitude": anchor.latitude,
|
||||
"longitude": anchor.longitude,
|
||||
"location_label": anchor.label,
|
||||
"location_inferred": True,
|
||||
"is_focus_match": item.feed_region == active_region,
|
||||
}
|
||||
|
||||
|
||||
def _build_payload(
|
||||
*,
|
||||
lat: float | None,
|
||||
lon: float | None,
|
||||
active_region: str,
|
||||
items: list[ParsedNewsItem],
|
||||
sources: list[NewsFeedSource],
|
||||
errors: list[str],
|
||||
stale: bool,
|
||||
generated_at: datetime | None = None,
|
||||
) -> dict[str, Any]:
|
||||
profile = get_region_profile(active_region)
|
||||
timestamp = generated_at or datetime.now(UTC)
|
||||
return {
|
||||
"generated_at": timestamp.isoformat().replace("+00:00", "Z"),
|
||||
"focus": {
|
||||
"lat": lat,
|
||||
"lon": lon,
|
||||
"region": active_region,
|
||||
"label": profile.label,
|
||||
"accent": profile.accent,
|
||||
},
|
||||
"sources": _serialize_sources(sources),
|
||||
"items": [_serialize_item(item, active_region=active_region) for item in items],
|
||||
"errors": errors,
|
||||
"stale": stale,
|
||||
}
|
||||
|
||||
|
||||
def _rank_and_trim_items(items: list[ParsedNewsItem], *, active_region: str) -> list[ParsedNewsItem]:
|
||||
deduped: dict[str, ParsedNewsItem] = {}
|
||||
for item in items:
|
||||
key = item.url.strip() or item.title.strip().lower()
|
||||
if key not in deduped:
|
||||
deduped[key] = item
|
||||
|
||||
return sorted(
|
||||
deduped.values(),
|
||||
key=lambda item: (
|
||||
item.feed_region != active_region,
|
||||
item.published_at is None,
|
||||
-(item.published_at.timestamp() if item.published_at else 0),
|
||||
item.feed_name,
|
||||
),
|
||||
)[:MAX_ITEMS_TOTAL]
|
||||
|
||||
|
||||
def _get_cached_region_feed(region: str) -> CachedRegionFeed | None:
|
||||
cached = _REGION_CACHE.get(region)
|
||||
if not cached:
|
||||
return None
|
||||
age_seconds = (datetime.now(UTC) - cached.fetched_at).total_seconds()
|
||||
if age_seconds > STALE_CACHE_MAX_AGE_SECONDS:
|
||||
return None
|
||||
return cached
|
||||
|
||||
|
||||
def _store_region_cache(region: str, *, items: list[ParsedNewsItem], sources: list[NewsFeedSource]) -> None:
|
||||
_REGION_CACHE[region] = CachedRegionFeed(
|
||||
region=region,
|
||||
fetched_at=datetime.now(UTC),
|
||||
items=list(items),
|
||||
sources=list(sources),
|
||||
)
|
||||
|
||||
|
||||
async def _fetch_source(
|
||||
client: httpx.AsyncClient,
|
||||
source: NewsFeedSource,
|
||||
) -> tuple[NewsFeedSource, list[ParsedNewsItem], str | None]:
|
||||
try:
|
||||
response = await client.get(source.feed_url)
|
||||
response.raise_for_status()
|
||||
return source, _parse_feed_entries(response.text, source), None
|
||||
except Exception as exc:
|
||||
return source, [], str(exc)
|
||||
|
||||
|
||||
async def get_earth_news_payload(lat: float | None = None, lon: float | None = None) -> dict[str, Any]:
|
||||
active_region = determine_focus_region(lat, lon)
|
||||
sources = get_sources_for_region(active_region)
|
||||
errors: list[str] = []
|
||||
|
||||
async with httpx.AsyncClient(
|
||||
timeout=REQUEST_TIMEOUT,
|
||||
follow_redirects=True,
|
||||
headers={"User-Agent": USER_AGENT},
|
||||
) as client:
|
||||
results = await asyncio.gather(*(_fetch_source(client, source) for source in sources))
|
||||
|
||||
fetched_items: list[ParsedNewsItem] = []
|
||||
for source, items, error in results:
|
||||
if error:
|
||||
errors.append(f"{source.name}: {error}")
|
||||
continue
|
||||
fetched_items.extend(items)
|
||||
|
||||
ranked_items = _rank_and_trim_items(fetched_items, active_region=active_region)
|
||||
if ranked_items:
|
||||
_store_region_cache(active_region, items=ranked_items, sources=sources)
|
||||
return _build_payload(
|
||||
lat=lat,
|
||||
lon=lon,
|
||||
active_region=active_region,
|
||||
items=ranked_items,
|
||||
sources=sources,
|
||||
errors=errors,
|
||||
stale=False,
|
||||
)
|
||||
|
||||
cached = _get_cached_region_feed(active_region)
|
||||
if cached:
|
||||
return _build_payload(
|
||||
lat=lat,
|
||||
lon=lon,
|
||||
active_region=active_region,
|
||||
items=cached.items,
|
||||
sources=cached.sources,
|
||||
errors=errors,
|
||||
stale=True,
|
||||
generated_at=cached.fetched_at,
|
||||
)
|
||||
|
||||
return _build_payload(
|
||||
lat=lat,
|
||||
lon=lon,
|
||||
active_region=active_region,
|
||||
items=[],
|
||||
sources=sources,
|
||||
errors=errors,
|
||||
stale=False,
|
||||
)
|
||||
@@ -0,0 +1,150 @@
|
||||
"""LLM provider presets used by Settings and the runtime AI provider bridge."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
|
||||
MODELS_DEV_URL = "https://models.dev/api.json"
|
||||
|
||||
|
||||
FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
|
||||
"minimax": {
|
||||
"provider": "minimax",
|
||||
"label": "MiniMax",
|
||||
"provider_api": "anthropic-messages",
|
||||
"base_url": "https://api.minimaxi.com/anthropic",
|
||||
"model": "MiniMax-M2.7",
|
||||
"models": ["MiniMax-M2.7", "MiniMax-M2.7-highspeed", "MiniMax-M2.5", "MiniMax-M2"],
|
||||
"api_key_env": "MINIMAX_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
"openai": {
|
||||
"provider": "openai",
|
||||
"label": "OpenAI",
|
||||
"provider_api": "openai-completions",
|
||||
"base_url": "https://api.openai.com/v1",
|
||||
"model": "gpt-5.1",
|
||||
"models": ["gpt-5.1", "gpt-5.1-codex", "gpt-4.1", "gpt-4o"],
|
||||
"api_key_env": "OPENAI_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
"anthropic": {
|
||||
"provider": "anthropic",
|
||||
"label": "Anthropic",
|
||||
"provider_api": "anthropic-messages",
|
||||
"base_url": "https://api.anthropic.com/v1",
|
||||
"model": "claude-sonnet-4-6",
|
||||
"models": ["claude-sonnet-4-6", "claude-opus-4-5", "claude-3-5-haiku-20241022"],
|
||||
"api_key_env": "ANTHROPIC_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
"deepseek": {
|
||||
"provider": "deepseek",
|
||||
"label": "DeepSeek",
|
||||
"provider_api": "openai-completions",
|
||||
"base_url": "https://api.deepseek.com/v1",
|
||||
"model": "deepseek-chat",
|
||||
"models": ["deepseek-chat", "deepseek-reasoner"],
|
||||
"api_key_env": "DEEPSEEK_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
"alibaba": {
|
||||
"provider": "alibaba",
|
||||
"label": "Alibaba Qwen / DashScope",
|
||||
"provider_api": "openai-completions",
|
||||
"base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1",
|
||||
"model": "qwen3-max",
|
||||
"models": ["qwen3-max", "qwen3.5-plus", "qwen-max", "qwen-plus"],
|
||||
"api_key_env": "DASHSCOPE_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
"moonshotai": {
|
||||
"provider": "moonshotai",
|
||||
"label": "Moonshot AI / Kimi",
|
||||
"provider_api": "openai-completions",
|
||||
"base_url": "https://api.moonshot.ai/v1",
|
||||
"model": "kimi-k2.5",
|
||||
"models": ["kimi-k2.5", "kimi-k2-thinking", "kimi-k2-turbo-preview"],
|
||||
"api_key_env": "MOONSHOT_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
"openrouter": {
|
||||
"provider": "openrouter",
|
||||
"label": "OpenRouter",
|
||||
"provider_api": "openai-completions",
|
||||
"base_url": "https://openrouter.ai/api/v1",
|
||||
"model": "openai/gpt-5.1",
|
||||
"models": ["openai/gpt-5.1", "anthropic/claude-sonnet-4.5", "qwen/qwen3-max"],
|
||||
"api_key_env": "OPENROUTER_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
"ollama": {
|
||||
"provider": "ollama",
|
||||
"label": "Ollama Local",
|
||||
"provider_api": "ollama-generate",
|
||||
"base_url": "http://127.0.0.1:11434",
|
||||
"model": "qwen2.5:7b",
|
||||
"models": ["qwen2.5:7b", "llama3.1:8b", "mistral:7b"],
|
||||
"api_key_env": "",
|
||||
"source": "fallback",
|
||||
},
|
||||
}
|
||||
|
||||
MODELS_DEV_PROVIDER_KEYS = {
|
||||
"minimax": "minimax",
|
||||
"openai": "openai",
|
||||
"anthropic": "anthropic",
|
||||
"deepseek": "deepseek",
|
||||
"alibaba": "alibaba",
|
||||
"moonshotai": "moonshotai",
|
||||
"openrouter": "openrouter",
|
||||
}
|
||||
|
||||
|
||||
def list_fallback_llm_provider_presets() -> list[dict[str, Any]]:
|
||||
return [dict(value) for value in FALLBACK_LLM_PROVIDER_PRESETS.values()]
|
||||
|
||||
|
||||
def get_fallback_llm_provider_preset(provider: str) -> dict[str, Any]:
|
||||
key = provider.strip().lower()
|
||||
if key not in FALLBACK_LLM_PROVIDER_PRESETS:
|
||||
raise ValueError(f"Unsupported LLM provider preset: {provider}")
|
||||
return dict(FALLBACK_LLM_PROVIDER_PRESETS[key])
|
||||
|
||||
|
||||
async def refresh_llm_provider_preset(provider: str) -> dict[str, Any]:
|
||||
fallback = get_fallback_llm_provider_preset(provider)
|
||||
models_dev_key = MODELS_DEV_PROVIDER_KEYS.get(fallback["provider"])
|
||||
if not models_dev_key:
|
||||
return fallback
|
||||
|
||||
async with httpx.AsyncClient(timeout=15.0, follow_redirects=True) as client:
|
||||
response = await client.get(
|
||||
MODELS_DEV_URL,
|
||||
headers={"User-Agent": "Planet/1.0"},
|
||||
)
|
||||
response.raise_for_status()
|
||||
catalog = response.json()
|
||||
|
||||
upstream = catalog.get(models_dev_key)
|
||||
if not isinstance(upstream, dict):
|
||||
return fallback
|
||||
|
||||
upstream_models = upstream.get("models") if isinstance(upstream.get("models"), dict) else {}
|
||||
model_ids = list(upstream_models.keys())[:80]
|
||||
base_url = upstream.get("api") or fallback["base_url"]
|
||||
if fallback["provider"] == "deepseek" and base_url == "https://api.deepseek.com":
|
||||
base_url = "https://api.deepseek.com/v1"
|
||||
|
||||
refreshed = {
|
||||
**fallback,
|
||||
"label": upstream.get("name") or fallback["label"],
|
||||
"base_url": base_url,
|
||||
"model": model_ids[0] if model_ids else fallback["model"],
|
||||
"models": model_ids or fallback["models"],
|
||||
"api_key_env": (upstream.get("env") or [fallback["api_key_env"]])[0],
|
||||
"source": MODELS_DEV_URL,
|
||||
}
|
||||
return refreshed
|
||||
@@ -0,0 +1,86 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from app.core.logging import get_logger, sanitize_log_value
|
||||
from app.core.request_context import get_request_id
|
||||
from app.db.session import async_session_factory
|
||||
from app.models.system_log import AuditLog, SystemLog
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
async def record_system_log(
|
||||
*,
|
||||
source: str,
|
||||
level: str,
|
||||
message: str,
|
||||
service: str | None = None,
|
||||
module: str | None = None,
|
||||
event: str | None = None,
|
||||
request_id: str | None = None,
|
||||
trace_id: str | None = None,
|
||||
user_id: int | None = None,
|
||||
category: str | None = None,
|
||||
context: dict[str, Any] | None = None,
|
||||
) -> None:
|
||||
try:
|
||||
async with async_session_factory() as session:
|
||||
session.add(
|
||||
SystemLog(
|
||||
source=source,
|
||||
service=service,
|
||||
module=module,
|
||||
event=event,
|
||||
level=level.lower(),
|
||||
message=str(sanitize_log_value(message)),
|
||||
request_id=request_id or get_request_id(),
|
||||
trace_id=trace_id,
|
||||
user_id=user_id,
|
||||
category=category,
|
||||
context=sanitize_log_value(context or {}),
|
||||
)
|
||||
)
|
||||
await session.commit()
|
||||
except Exception:
|
||||
logger.exception_event(
|
||||
"Failed to persist system log",
|
||||
event="system_log.persist.failed",
|
||||
context={"event_name": event, "source": source},
|
||||
)
|
||||
|
||||
|
||||
async def record_audit_log(
|
||||
*,
|
||||
action: str,
|
||||
actor_id: int | None = None,
|
||||
actor_name: str | None = None,
|
||||
target_type: str | None = None,
|
||||
target_id: str | None = None,
|
||||
result: str | None = None,
|
||||
request_id: str | None = None,
|
||||
ip: str | None = None,
|
||||
details: dict[str, Any] | None = None,
|
||||
) -> None:
|
||||
try:
|
||||
async with async_session_factory() as session:
|
||||
session.add(
|
||||
AuditLog(
|
||||
actor_id=actor_id,
|
||||
actor_name=actor_name,
|
||||
action=action,
|
||||
target_type=target_type,
|
||||
target_id=target_id,
|
||||
result=result,
|
||||
request_id=request_id or get_request_id(),
|
||||
ip=ip,
|
||||
details=sanitize_log_value(details or {}),
|
||||
)
|
||||
)
|
||||
await session.commit()
|
||||
except Exception:
|
||||
logger.exception_event(
|
||||
"Failed to persist audit log",
|
||||
event="audit_log.persist.failed",
|
||||
context={"action": action},
|
||||
)
|
||||
@@ -0,0 +1,694 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from collections.abc import Sequence
|
||||
from datetime import UTC, datetime
|
||||
from time import perf_counter
|
||||
from uuid import uuid4
|
||||
|
||||
from fastapi import HTTPException, status
|
||||
from sqlalchemy import func, select, update
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.db.session import async_session_factory
|
||||
from app.models.playground_message import PlaygroundMessage
|
||||
from app.models.playground_session import PlaygroundSession
|
||||
from app.schemas.ai import (
|
||||
PlaygroundMessageEditRequest,
|
||||
PlaygroundMessageActionResponse,
|
||||
PlaygroundMessageCreateRequest,
|
||||
PlaygroundMessageRecord,
|
||||
PlaygroundMessageResendRequest,
|
||||
PlaygroundMessageStopRequest,
|
||||
PlaygroundSessionResponse,
|
||||
PlaygroundSessionState,
|
||||
PlaygroundSessionUpsertRequest,
|
||||
PlaygroundThreadResponse,
|
||||
SituationalAnalysisRequest,
|
||||
)
|
||||
from app.services.ai_client import AIProviderClient
|
||||
from app.services.playground_session_store import _to_response as session_to_response
|
||||
from app.services.playground_session_store import upsert_playground_session
|
||||
|
||||
STREAM_CHUNK_SIZE = 24
|
||||
STREAM_INTERVAL_SECONDS = 0.08
|
||||
THINKING_PREVIEW_SECONDS = 2.6
|
||||
|
||||
|
||||
class _ActiveRun:
|
||||
def __init__(self, task: asyncio.Task[None]) -> None:
|
||||
self.task = task
|
||||
self.stop_requested = asyncio.Event()
|
||||
|
||||
|
||||
_ACTIVE_RUNS: dict[str, _ActiveRun] = {}
|
||||
|
||||
|
||||
async def _get_session_by_key(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
session_key: str,
|
||||
) -> PlaygroundSession | None:
|
||||
result = await db.execute(
|
||||
select(PlaygroundSession).where(
|
||||
PlaygroundSession.user_id == user_id,
|
||||
PlaygroundSession.session_key == session_key,
|
||||
)
|
||||
)
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
|
||||
async def _require_session(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
session_key: str,
|
||||
) -> PlaygroundSession:
|
||||
session = await _get_session_by_key(db, user_id=user_id, session_key=session_key)
|
||||
if session is None:
|
||||
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Playground session not found")
|
||||
return session
|
||||
|
||||
|
||||
async def _require_visible_message(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
public_id: str,
|
||||
role: str | None = None,
|
||||
) -> PlaygroundMessage:
|
||||
conditions = [
|
||||
PlaygroundMessage.user_id == user_id,
|
||||
PlaygroundMessage.public_id == public_id,
|
||||
PlaygroundMessage.is_visible.is_(True),
|
||||
]
|
||||
if role is not None:
|
||||
conditions.append(PlaygroundMessage.role == role)
|
||||
|
||||
result = await db.execute(select(PlaygroundMessage).where(*conditions))
|
||||
message = result.scalar_one_or_none()
|
||||
if message is None:
|
||||
detail = "User message not found" if role == "user" else "Playground message not found"
|
||||
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail=detail)
|
||||
return message
|
||||
|
||||
|
||||
def _message_to_record(message: PlaygroundMessage, parent_public_id: str | None = None) -> PlaygroundMessageRecord:
|
||||
return PlaygroundMessageRecord(
|
||||
id=message.public_id,
|
||||
role=message.role,
|
||||
kind=message.kind,
|
||||
status=message.status,
|
||||
title=message.title,
|
||||
content=message.content or "",
|
||||
thinking_content=message.thinking_content or "",
|
||||
meta=list(message.meta or []),
|
||||
markdown=message.role != "system",
|
||||
provider=message.provider,
|
||||
model=message.model,
|
||||
request_id=message.request_id,
|
||||
raw_response=dict(message.raw_response or {}),
|
||||
content_blocks=list(message.content_blocks or []),
|
||||
text_blocks=list(message.text_blocks or []),
|
||||
thinking_blocks=list(message.thinking_blocks or []),
|
||||
parent_message_id=parent_public_id,
|
||||
created_at=message.created_at.isoformat(),
|
||||
updated_at=message.updated_at.isoformat(),
|
||||
)
|
||||
|
||||
|
||||
async def _ensure_session(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
session_key: str,
|
||||
title: str,
|
||||
state: PlaygroundSessionState | None = None,
|
||||
) -> PlaygroundSession:
|
||||
result = await db.execute(
|
||||
select(PlaygroundSession).where(
|
||||
PlaygroundSession.user_id == user_id,
|
||||
PlaygroundSession.session_key == session_key,
|
||||
)
|
||||
)
|
||||
session = result.scalar_one_or_none()
|
||||
if session is not None:
|
||||
if title:
|
||||
session.title = title[:200]
|
||||
if state is not None:
|
||||
session.state = state.model_dump(mode="json")
|
||||
await db.flush()
|
||||
await db.refresh(session)
|
||||
return session
|
||||
|
||||
payload = PlaygroundSessionUpsertRequest(
|
||||
session_key=session_key,
|
||||
title=title[:200],
|
||||
state=state or PlaygroundSessionState(title=title[:200]),
|
||||
)
|
||||
await upsert_playground_session(db, user_id=user_id, payload=payload)
|
||||
result = await db.execute(
|
||||
select(PlaygroundSession).where(
|
||||
PlaygroundSession.user_id == user_id,
|
||||
PlaygroundSession.session_key == session_key,
|
||||
)
|
||||
)
|
||||
return result.scalar_one()
|
||||
|
||||
|
||||
async def _list_visible_messages(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
session_id: int,
|
||||
) -> list[PlaygroundMessage]:
|
||||
result = await db.execute(
|
||||
select(PlaygroundMessage)
|
||||
.where(
|
||||
PlaygroundMessage.session_id == session_id,
|
||||
PlaygroundMessage.is_visible.is_(True),
|
||||
)
|
||||
.order_by(PlaygroundMessage.sort_order.asc(), PlaygroundMessage.id.asc())
|
||||
)
|
||||
return list(result.scalars().all())
|
||||
|
||||
|
||||
async def _build_thread_response(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
session: PlaygroundSession,
|
||||
) -> PlaygroundThreadResponse:
|
||||
messages = await _list_visible_messages(db, session_id=session.id)
|
||||
id_map = {item.id: item.public_id for item in messages}
|
||||
return PlaygroundThreadResponse(
|
||||
session=session_to_response(session),
|
||||
messages=[_message_to_record(item, id_map.get(item.parent_message_id)) for item in messages],
|
||||
)
|
||||
|
||||
|
||||
async def get_thread(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
session_key: str,
|
||||
) -> PlaygroundThreadResponse | None:
|
||||
session = await _get_session_by_key(db, user_id=user_id, session_key=session_key)
|
||||
if session is None:
|
||||
return None
|
||||
return await _build_thread_response(db, session=session)
|
||||
|
||||
|
||||
async def _build_action_response(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
session: PlaygroundSession,
|
||||
active_message_id: str | None = None,
|
||||
) -> PlaygroundMessageActionResponse:
|
||||
thread = await _build_thread_response(db, session=session)
|
||||
return PlaygroundMessageActionResponse(
|
||||
session=thread.session,
|
||||
messages=thread.messages,
|
||||
active_message_id=active_message_id,
|
||||
)
|
||||
|
||||
|
||||
def _collect_constraints(raw_constraints: str) -> list[str]:
|
||||
return [item.strip() for item in raw_constraints.split("\n") if item.strip()]
|
||||
|
||||
|
||||
async def _next_sort_order(db: AsyncSession, session_id: int) -> int:
|
||||
result = await db.execute(
|
||||
select(func.max(PlaygroundMessage.sort_order)).where(PlaygroundMessage.session_id == session_id)
|
||||
)
|
||||
current = result.scalar_one_or_none()
|
||||
return int(current or 0)
|
||||
|
||||
|
||||
async def _set_session_state(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
session: PlaygroundSession,
|
||||
payload: PlaygroundMessageCreateRequest,
|
||||
) -> PlaygroundSession:
|
||||
session.state = PlaygroundSessionState(
|
||||
messages=[],
|
||||
selectedPresetKey=payload.selected_preset_key,
|
||||
title=payload.title,
|
||||
objective=payload.objective,
|
||||
constraints=payload.constraints,
|
||||
inputValue="",
|
||||
analysis=None,
|
||||
latestAnalysisMessageId=None,
|
||||
analysisMeta={},
|
||||
helpExpanded=payload.help_expanded,
|
||||
).model_dump(mode="json")
|
||||
session.title = payload.title[:200]
|
||||
await db.flush()
|
||||
await db.refresh(session)
|
||||
return session
|
||||
|
||||
|
||||
def _spawn_assistant_run(
|
||||
*,
|
||||
user_id: int,
|
||||
session_id: int,
|
||||
session_key: str,
|
||||
user_message_id: int,
|
||||
assistant_message_id: int,
|
||||
assistant_public_id: str,
|
||||
payload: PlaygroundMessageCreateRequest,
|
||||
provider_client: AIProviderClient,
|
||||
) -> None:
|
||||
task = asyncio.create_task(
|
||||
_run_assistant_message(
|
||||
user_id=user_id,
|
||||
session_id=session_id,
|
||||
session_key=session_key,
|
||||
user_message_id=user_message_id,
|
||||
assistant_message_id=assistant_message_id,
|
||||
payload=payload,
|
||||
provider_client=provider_client,
|
||||
)
|
||||
)
|
||||
_ACTIVE_RUNS[assistant_public_id] = _ActiveRun(task)
|
||||
|
||||
|
||||
async def create_turn(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
payload: PlaygroundMessageCreateRequest,
|
||||
provider_client: AIProviderClient,
|
||||
) -> PlaygroundMessageActionResponse:
|
||||
session = await _ensure_session(
|
||||
db,
|
||||
user_id=user_id,
|
||||
session_key=payload.session_key,
|
||||
title=payload.title,
|
||||
state=PlaygroundSessionState(
|
||||
selectedPresetKey=payload.selected_preset_key,
|
||||
title=payload.title,
|
||||
objective=payload.objective,
|
||||
constraints=payload.constraints,
|
||||
inputValue="",
|
||||
helpExpanded=payload.help_expanded,
|
||||
),
|
||||
)
|
||||
session = await _set_session_state(db, session=session, payload=payload)
|
||||
base_order = await _next_sort_order(db, session.id)
|
||||
|
||||
user_message = PlaygroundMessage(
|
||||
public_id=uuid4().hex,
|
||||
session_id=session.id,
|
||||
user_id=user_id,
|
||||
role="user",
|
||||
kind="message",
|
||||
status="done",
|
||||
title=payload.selected_preset_key,
|
||||
content=payload.input,
|
||||
meta=[payload.title],
|
||||
sort_order=base_order + 10,
|
||||
)
|
||||
assistant_message = PlaygroundMessage(
|
||||
public_id=uuid4().hex,
|
||||
session_id=session.id,
|
||||
user_id=user_id,
|
||||
parent_message_id=None,
|
||||
role="assistant",
|
||||
kind="thinking",
|
||||
status="pending",
|
||||
title="AI 回应",
|
||||
content="",
|
||||
thinking_content="",
|
||||
meta=[],
|
||||
sort_order=base_order + 20,
|
||||
)
|
||||
db.add(user_message)
|
||||
await db.flush()
|
||||
assistant_message.parent_message_id = user_message.id
|
||||
db.add(assistant_message)
|
||||
await db.flush()
|
||||
await db.refresh(user_message)
|
||||
await db.refresh(assistant_message)
|
||||
await db.commit()
|
||||
await db.refresh(session)
|
||||
await db.refresh(user_message)
|
||||
await db.refresh(assistant_message)
|
||||
|
||||
_spawn_assistant_run(
|
||||
user_id=user_id,
|
||||
session_id=session.id,
|
||||
session_key=payload.session_key,
|
||||
user_message_id=user_message.id,
|
||||
assistant_message_id=assistant_message.id,
|
||||
assistant_public_id=assistant_message.public_id,
|
||||
payload=payload,
|
||||
provider_client=provider_client,
|
||||
)
|
||||
|
||||
return await _build_action_response(
|
||||
db,
|
||||
session=session,
|
||||
active_message_id=assistant_message.public_id,
|
||||
)
|
||||
|
||||
|
||||
async def _create_assistant_retry_turn(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
session: PlaygroundSession,
|
||||
user_message: PlaygroundMessage,
|
||||
payload: PlaygroundMessageCreateRequest,
|
||||
provider_client: AIProviderClient,
|
||||
) -> PlaygroundMessageActionResponse:
|
||||
base_order = await _next_sort_order(db, session.id)
|
||||
assistant_message = PlaygroundMessage(
|
||||
public_id=uuid4().hex,
|
||||
session_id=session.id,
|
||||
user_id=user_id,
|
||||
parent_message_id=user_message.id,
|
||||
role="assistant",
|
||||
kind="thinking",
|
||||
status="pending",
|
||||
title="AI 回应",
|
||||
content="",
|
||||
thinking_content="",
|
||||
meta=[],
|
||||
sort_order=base_order + 10,
|
||||
)
|
||||
db.add(assistant_message)
|
||||
await db.flush()
|
||||
await db.commit()
|
||||
await db.refresh(session)
|
||||
await db.refresh(assistant_message)
|
||||
|
||||
_spawn_assistant_run(
|
||||
user_id=user_id,
|
||||
session_id=session.id,
|
||||
session_key=payload.session_key,
|
||||
user_message_id=user_message.id,
|
||||
assistant_message_id=assistant_message.id,
|
||||
assistant_public_id=assistant_message.public_id,
|
||||
payload=payload,
|
||||
provider_client=provider_client,
|
||||
)
|
||||
|
||||
return await _build_action_response(
|
||||
db,
|
||||
session=session,
|
||||
active_message_id=assistant_message.public_id,
|
||||
)
|
||||
|
||||
|
||||
async def stop_message(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
payload: PlaygroundMessageStopRequest,
|
||||
) -> PlaygroundMessageActionResponse:
|
||||
session = await _require_session(db, user_id=user_id, session_key=payload.session_key)
|
||||
message = await _require_visible_message(db, user_id=user_id, public_id=payload.message_id)
|
||||
|
||||
if message.status not in {"pending", "thinking", "answering"}:
|
||||
return await _build_action_response(db, session=session)
|
||||
|
||||
active_run = _ACTIVE_RUNS.get(message.public_id)
|
||||
if active_run is not None:
|
||||
active_run.stop_requested.set()
|
||||
active_run.task.cancel()
|
||||
|
||||
message.status = "stopped"
|
||||
if "已手动停止生成" not in (message.meta or []):
|
||||
message.meta = [*(message.meta or []), "已手动停止生成"]
|
||||
await db.flush()
|
||||
await db.commit()
|
||||
await db.refresh(message)
|
||||
|
||||
return await _build_action_response(db, session=session)
|
||||
|
||||
|
||||
async def resend_turn(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
payload: PlaygroundMessageResendRequest,
|
||||
provider_client: AIProviderClient,
|
||||
) -> PlaygroundMessageActionResponse:
|
||||
session = await _require_session(db, user_id=user_id, session_key=payload.session_key)
|
||||
user_message = await _require_visible_message(
|
||||
db,
|
||||
user_id=user_id,
|
||||
public_id=payload.user_message_id,
|
||||
role="user",
|
||||
)
|
||||
|
||||
later_messages = await db.execute(
|
||||
select(PlaygroundMessage).where(
|
||||
PlaygroundMessage.session_id == session.id,
|
||||
PlaygroundMessage.sort_order > user_message.sort_order,
|
||||
PlaygroundMessage.is_visible.is_(True),
|
||||
)
|
||||
)
|
||||
for item in later_messages.scalars().all():
|
||||
item.is_visible = False
|
||||
if item.status in {"pending", "thinking", "answering"}:
|
||||
active_run = _ACTIVE_RUNS.get(item.public_id)
|
||||
if active_run is not None:
|
||||
active_run.stop_requested.set()
|
||||
active_run.task.cancel()
|
||||
await db.flush()
|
||||
await db.commit()
|
||||
|
||||
session_state = PlaygroundSessionState.model_validate(session.state or {})
|
||||
create_payload = PlaygroundMessageCreateRequest(
|
||||
session_key=payload.session_key,
|
||||
title=session_state.title or session.title,
|
||||
objective=session_state.objective or "继续当前对话",
|
||||
constraints=session_state.constraints or "",
|
||||
input=user_message.content,
|
||||
selected_preset_key=session_state.selectedPresetKey or "bgp-brief",
|
||||
help_expanded=session_state.helpExpanded,
|
||||
)
|
||||
return await _create_assistant_retry_turn(
|
||||
db,
|
||||
user_id=user_id,
|
||||
session=session,
|
||||
user_message=user_message,
|
||||
payload=create_payload,
|
||||
provider_client=provider_client,
|
||||
)
|
||||
|
||||
|
||||
async def edit_user_message(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
payload: PlaygroundMessageEditRequest,
|
||||
) -> PlaygroundMessageActionResponse:
|
||||
session = await _require_session(db, user_id=user_id, session_key=payload.session_key)
|
||||
user_message = await _require_visible_message(
|
||||
db,
|
||||
user_id=user_id,
|
||||
public_id=payload.user_message_id,
|
||||
role="user",
|
||||
)
|
||||
|
||||
user_message.content = payload.content.strip()
|
||||
await db.flush()
|
||||
await db.commit()
|
||||
await db.refresh(user_message)
|
||||
|
||||
return await _build_action_response(db, session=session)
|
||||
|
||||
|
||||
async def _append_meta_if_missing(db: AsyncSession, message_id: int, meta_line: str) -> None:
|
||||
result = await db.execute(select(PlaygroundMessage).where(PlaygroundMessage.id == message_id))
|
||||
message = result.scalar_one_or_none()
|
||||
if message is None:
|
||||
return
|
||||
if meta_line not in (message.meta or []):
|
||||
message.meta = [*(message.meta or []), meta_line]
|
||||
await db.flush()
|
||||
|
||||
|
||||
async def _should_stop(message_public_id: str) -> bool:
|
||||
active_run = _ACTIVE_RUNS.get(message_public_id)
|
||||
return active_run.stop_requested.is_set() if active_run is not None else False
|
||||
|
||||
|
||||
async def _mark_message_state(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
message_id: int,
|
||||
**updates,
|
||||
) -> PlaygroundMessage:
|
||||
result = await db.execute(select(PlaygroundMessage).where(PlaygroundMessage.id == message_id))
|
||||
message = result.scalar_one()
|
||||
for key, value in updates.items():
|
||||
setattr(message, key, value)
|
||||
await db.flush()
|
||||
await db.refresh(message)
|
||||
return message
|
||||
|
||||
|
||||
def _build_conversation_history(messages: Sequence[PlaygroundMessage], current_user_message_id: int) -> list[dict]:
|
||||
history: list[dict] = []
|
||||
for item in messages:
|
||||
if item.id >= current_user_message_id:
|
||||
break
|
||||
if item.role == "system":
|
||||
continue
|
||||
history.append(
|
||||
{
|
||||
"role": item.role,
|
||||
"kind": item.kind or "message",
|
||||
"title": item.title,
|
||||
"content": item.content or "",
|
||||
}
|
||||
)
|
||||
return history[-8:]
|
||||
|
||||
|
||||
async def _run_assistant_message(
|
||||
*,
|
||||
user_id: int,
|
||||
session_id: int,
|
||||
session_key: str,
|
||||
user_message_id: int,
|
||||
assistant_message_id: int,
|
||||
payload: PlaygroundMessageCreateRequest,
|
||||
provider_client: AIProviderClient,
|
||||
) -> None:
|
||||
request_id = str(uuid4())
|
||||
started_at = perf_counter()
|
||||
assistant_public_id: str | None = None
|
||||
try:
|
||||
async with async_session_factory() as db:
|
||||
session = await db.get(PlaygroundSession, session_id)
|
||||
user_message = await db.get(PlaygroundMessage, user_message_id)
|
||||
assistant_message = await db.get(PlaygroundMessage, assistant_message_id)
|
||||
if session is None or user_message is None or assistant_message is None:
|
||||
return
|
||||
assistant_public_id = assistant_message.public_id
|
||||
|
||||
visible_messages = await _list_visible_messages(db, session_id=session_id)
|
||||
conversation_history = _build_conversation_history(visible_messages, user_message_id)
|
||||
|
||||
request_payload = SituationalAnalysisRequest(
|
||||
title=payload.title,
|
||||
objective=payload.objective,
|
||||
observations=[item.strip() for item in payload.input.split("\n") if item.strip()],
|
||||
constraints=_collect_constraints(payload.constraints),
|
||||
context={
|
||||
"source": "playground",
|
||||
"preset": payload.selected_preset_key,
|
||||
"conversation_history": conversation_history,
|
||||
"history_size": len(conversation_history),
|
||||
},
|
||||
thinking={"type": "enabled"},
|
||||
)
|
||||
|
||||
analysis = await provider_client.analyze(request_payload, request_id=request_id)
|
||||
|
||||
async with async_session_factory() as db:
|
||||
assistant_message = await _mark_message_state(
|
||||
db,
|
||||
message_id=assistant_message_id,
|
||||
status="thinking" if analysis.thinking_blocks else "answering",
|
||||
title=f"{analysis.provider} / {analysis.model}",
|
||||
provider=analysis.provider,
|
||||
model=analysis.model,
|
||||
request_id=request_id,
|
||||
raw_response=analysis.raw_response,
|
||||
content_blocks=[item.model_dump(mode="json") for item in analysis.content_blocks],
|
||||
text_blocks=analysis.text_blocks,
|
||||
thinking_blocks=analysis.thinking_blocks,
|
||||
thinking_content="\n\n".join(analysis.thinking_blocks).strip(),
|
||||
)
|
||||
session = await db.get(PlaygroundSession, session_id)
|
||||
if session is not None:
|
||||
session_state = PlaygroundSessionState.model_validate(session.state or {})
|
||||
session.state = session_state.model_copy(
|
||||
update={
|
||||
"latestAnalysisMessageId": assistant_message.public_id,
|
||||
"analysis": analysis.model_dump(mode="json"),
|
||||
}
|
||||
).model_dump(mode="json")
|
||||
await db.flush()
|
||||
await db.commit()
|
||||
|
||||
if assistant_public_id and analysis.thinking_blocks:
|
||||
await asyncio.sleep(THINKING_PREVIEW_SECONDS)
|
||||
if await _should_stop(assistant_public_id):
|
||||
return
|
||||
|
||||
content = analysis.content or ""
|
||||
cursor = 0
|
||||
while cursor < len(content):
|
||||
if assistant_public_id and await _should_stop(assistant_public_id):
|
||||
return
|
||||
cursor = min(len(content), cursor + STREAM_CHUNK_SIZE)
|
||||
async with async_session_factory() as db:
|
||||
await _mark_message_state(
|
||||
db,
|
||||
message_id=assistant_message_id,
|
||||
status="answering",
|
||||
content=content[:cursor],
|
||||
)
|
||||
await db.commit()
|
||||
await asyncio.sleep(STREAM_INTERVAL_SECONDS)
|
||||
|
||||
duration_ms = round((perf_counter() - started_at) * 1000)
|
||||
async with async_session_factory() as db:
|
||||
assistant_message = await _mark_message_state(
|
||||
db,
|
||||
message_id=assistant_message_id,
|
||||
status="done",
|
||||
content=content,
|
||||
meta=[
|
||||
f"Request ID: {request_id}",
|
||||
f"耗时: {duration_ms} ms",
|
||||
f"完成时间: {datetime.now(UTC).astimezone().isoformat()}",
|
||||
],
|
||||
)
|
||||
session = await db.get(PlaygroundSession, session_id)
|
||||
if session is not None:
|
||||
session_state = PlaygroundSessionState.model_validate(session.state or {})
|
||||
session.state = session_state.model_copy(
|
||||
update={
|
||||
"latestAnalysisMessageId": assistant_message.public_id,
|
||||
"analysis": analysis.model_dump(mode="json"),
|
||||
"analysisMeta": {
|
||||
"requestId": request_id,
|
||||
"durationMs": duration_ms,
|
||||
"completedAt": datetime.now().isoformat(),
|
||||
},
|
||||
}
|
||||
).model_dump(mode="json")
|
||||
await db.flush()
|
||||
await db.commit()
|
||||
except asyncio.CancelledError:
|
||||
async with async_session_factory() as db:
|
||||
result = await db.execute(select(PlaygroundMessage).where(PlaygroundMessage.id == assistant_message_id))
|
||||
message = result.scalar_one_or_none()
|
||||
if message is not None and message.status in {"pending", "thinking", "answering"}:
|
||||
message.status = "stopped"
|
||||
if "已手动停止生成" not in (message.meta or []):
|
||||
message.meta = [*(message.meta or []), "已手动停止生成"]
|
||||
await db.flush()
|
||||
await db.commit()
|
||||
raise
|
||||
except Exception as exc:
|
||||
async with async_session_factory() as db:
|
||||
result = await db.execute(select(PlaygroundMessage).where(PlaygroundMessage.id == assistant_message_id))
|
||||
message = result.scalar_one_or_none()
|
||||
if message is not None:
|
||||
message.status = "error"
|
||||
message.content = message.content or "分析失败,请检查 AI Provider 配置或稍后再试。"
|
||||
message.meta = [*(message.meta or []), f"错误: {type(exc).__name__}"]
|
||||
await db.flush()
|
||||
await db.commit()
|
||||
finally:
|
||||
if assistant_public_id:
|
||||
_ACTIVE_RUNS.pop(assistant_public_id, None)
|
||||
@@ -0,0 +1,72 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.models.playground_session import PlaygroundSession
|
||||
from app.schemas.ai import (
|
||||
PlaygroundSessionResponse,
|
||||
PlaygroundSessionState,
|
||||
PlaygroundSessionUpsertRequest,
|
||||
)
|
||||
|
||||
|
||||
def _to_response(record: PlaygroundSession) -> PlaygroundSessionResponse:
|
||||
return PlaygroundSessionResponse(
|
||||
id=str(record.id),
|
||||
session_key=record.session_key,
|
||||
title=record.title,
|
||||
state=PlaygroundSessionState.model_validate(record.state or {}),
|
||||
created_at=record.created_at.isoformat(),
|
||||
updated_at=record.updated_at.isoformat(),
|
||||
)
|
||||
|
||||
|
||||
async def get_playground_session(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
session_key: str = "default",
|
||||
) -> PlaygroundSessionResponse | None:
|
||||
result = await db.execute(
|
||||
select(PlaygroundSession).where(
|
||||
PlaygroundSession.user_id == user_id,
|
||||
PlaygroundSession.session_key == session_key,
|
||||
)
|
||||
)
|
||||
record = result.scalar_one_or_none()
|
||||
if record is None:
|
||||
return None
|
||||
return _to_response(record)
|
||||
|
||||
|
||||
async def upsert_playground_session(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
payload: PlaygroundSessionUpsertRequest,
|
||||
) -> PlaygroundSessionResponse:
|
||||
result = await db.execute(
|
||||
select(PlaygroundSession).where(
|
||||
PlaygroundSession.user_id == user_id,
|
||||
PlaygroundSession.session_key == payload.session_key,
|
||||
)
|
||||
)
|
||||
record = result.scalar_one_or_none()
|
||||
title = (payload.title or payload.state.title or "Playground 会话").strip()[:200] or "Playground 会话"
|
||||
|
||||
if record is None:
|
||||
record = PlaygroundSession(
|
||||
user_id=user_id,
|
||||
session_key=payload.session_key,
|
||||
title=title,
|
||||
state=payload.state.model_dump(mode="json"),
|
||||
)
|
||||
db.add(record)
|
||||
else:
|
||||
record.title = title
|
||||
record.state = payload.state.model_dump(mode="json")
|
||||
|
||||
await db.flush()
|
||||
await db.refresh(record)
|
||||
return _to_response(record)
|
||||
@@ -1,7 +1,6 @@
|
||||
"""Task Scheduler for running collection jobs."""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
@@ -9,16 +8,46 @@ from apscheduler.schedulers.asyncio import AsyncIOScheduler
|
||||
from apscheduler.triggers.interval import IntervalTrigger
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.core.logging import get_logger
|
||||
from app.db.session import async_session_factory
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.models.datasource import DataSource
|
||||
from app.models.task import CollectionTask
|
||||
from app.services.collectors.registry import collector_registry
|
||||
from app.services.datasource_connectivity import (
|
||||
build_builtin_connectivity_checksum,
|
||||
get_builtin_effective_candidate,
|
||||
save_connectivity_success,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
logger = get_logger(__name__)
|
||||
|
||||
scheduler = AsyncIOScheduler()
|
||||
RUNNING_TASK_GUARD_TIMEOUT_MINUTES = 90
|
||||
RUNNING_COLLECTOR_TASKS: dict[str, asyncio.Task[Any]] = {}
|
||||
|
||||
|
||||
def _collector_task_name(collector_name: str) -> str:
|
||||
return f"collector:{collector_name}"
|
||||
|
||||
|
||||
def get_running_collector_task(collector_name: str) -> asyncio.Task[Any] | None:
|
||||
task = RUNNING_COLLECTOR_TASKS.get(collector_name)
|
||||
if task is not None and not task.done():
|
||||
return task
|
||||
|
||||
if task is not None and task.done():
|
||||
RUNNING_COLLECTOR_TASKS.pop(collector_name, None)
|
||||
|
||||
target_name = _collector_task_name(collector_name)
|
||||
for candidate in asyncio.all_tasks():
|
||||
if candidate.done():
|
||||
continue
|
||||
if candidate.get_name() == target_name:
|
||||
RUNNING_COLLECTOR_TASKS[collector_name] = candidate
|
||||
return candidate
|
||||
|
||||
return None
|
||||
|
||||
|
||||
async def _update_next_run_at(datasource: DataSource, session) -> None:
|
||||
@@ -30,7 +59,11 @@ async def _update_next_run_at(datasource: DataSource, session) -> None:
|
||||
async def _apply_datasource_schedule(datasource: DataSource, session) -> None:
|
||||
collector = collector_registry.get(datasource.source)
|
||||
if not collector:
|
||||
logger.warning("Collector not found for datasource %s", datasource.source)
|
||||
logger.warning_event(
|
||||
"Collector not found for datasource",
|
||||
event="collector.schedule.collector_missing",
|
||||
context={"collector_name": datasource.source},
|
||||
)
|
||||
return
|
||||
|
||||
collector_registry.set_active(datasource.source, datasource.is_active)
|
||||
@@ -48,13 +81,17 @@ async def _apply_datasource_schedule(datasource: DataSource, session) -> None:
|
||||
replace_existing=True,
|
||||
kwargs={"collector_name": datasource.source},
|
||||
)
|
||||
logger.info(
|
||||
"Scheduled collector: %s (every %sm)",
|
||||
datasource.source,
|
||||
datasource.frequency_minutes,
|
||||
logger.info_event(
|
||||
"Scheduled collector",
|
||||
event="collector.schedule.updated",
|
||||
context={"collector_name": datasource.source, "frequency_minutes": datasource.frequency_minutes},
|
||||
)
|
||||
else:
|
||||
logger.info("Collector disabled: %s", datasource.source)
|
||||
logger.info_event(
|
||||
"Collector disabled",
|
||||
event="collector.schedule.disabled",
|
||||
context={"collector_name": datasource.source},
|
||||
)
|
||||
|
||||
await _update_next_run_at(datasource, session)
|
||||
|
||||
@@ -63,18 +100,30 @@ async def run_collector_task(collector_name: str):
|
||||
"""Run a single collector task."""
|
||||
collector = collector_registry.get(collector_name)
|
||||
if not collector:
|
||||
logger.error("Collector not found: %s", collector_name)
|
||||
logger.error_event(
|
||||
"Collector not found",
|
||||
event="collector.run.collector_missing",
|
||||
context={"collector_name": collector_name},
|
||||
)
|
||||
return
|
||||
|
||||
async with async_session_factory() as db:
|
||||
result = await db.execute(select(DataSource).where(DataSource.source == collector_name))
|
||||
datasource = result.scalar_one_or_none()
|
||||
if not datasource:
|
||||
logger.error("Datasource not found for collector: %s", collector_name)
|
||||
logger.error_event(
|
||||
"Datasource not found for collector",
|
||||
event="collector.run.datasource_missing",
|
||||
context={"collector_name": collector_name},
|
||||
)
|
||||
return
|
||||
|
||||
if not datasource.is_active:
|
||||
logger.info("Skipping disabled collector: %s", collector_name)
|
||||
logger.info_event(
|
||||
"Skipping disabled collector",
|
||||
event="collector.run.skipped_disabled",
|
||||
context={"collector_name": collector_name},
|
||||
)
|
||||
return
|
||||
|
||||
running_result = await db.execute(
|
||||
@@ -98,10 +147,10 @@ async def run_collector_task(collector_name: str):
|
||||
and (now - started_at) > timedelta(minutes=RUNNING_TASK_GUARD_TIMEOUT_MINUTES)
|
||||
)
|
||||
if not is_stale:
|
||||
logger.warning(
|
||||
"Skipping collector %s trigger because task %s is already running",
|
||||
collector_name,
|
||||
existing_running.id,
|
||||
logger.warning_event(
|
||||
"Skipping collector trigger because task is already running",
|
||||
event="collector.run.skipped_already_running",
|
||||
context={"collector_name": collector_name, "task_id": existing_running.id},
|
||||
)
|
||||
return
|
||||
|
||||
@@ -119,25 +168,64 @@ async def run_collector_task(collector_name: str):
|
||||
else stale_reason
|
||||
)
|
||||
await db.commit()
|
||||
logger.warning(
|
||||
"Marked stale running task %s as failed before rerun of %s",
|
||||
existing_running.id,
|
||||
collector_name,
|
||||
logger.warning_event(
|
||||
"Marked stale running task as failed before rerun",
|
||||
event="collector.run.stale_task_failed",
|
||||
context={"collector_name": collector_name, "task_id": existing_running.id},
|
||||
)
|
||||
|
||||
try:
|
||||
collector._datasource_id = datasource.id
|
||||
logger.info("Running collector: %s (datasource_id=%s)", collector_name, datasource.id)
|
||||
logger.info_event(
|
||||
"Running collector",
|
||||
event="collector.run.started",
|
||||
context={"collector_name": collector_name, "datasource_id": datasource.id},
|
||||
)
|
||||
task_result = await collector.run(db)
|
||||
datasource.last_run_at = datetime.now(UTC)
|
||||
datasource.last_status = task_result.get("status")
|
||||
if datasource.last_status == "success":
|
||||
effective_candidate = await get_builtin_effective_candidate(db, datasource.source)
|
||||
checksum, _credential_context = await build_builtin_connectivity_checksum(
|
||||
datasource.source,
|
||||
effective_candidate["endpoint"],
|
||||
effective_candidate["auth_type"],
|
||||
effective_candidate["headers"],
|
||||
effective_candidate["config"],
|
||||
db,
|
||||
)
|
||||
await save_connectivity_success(
|
||||
db,
|
||||
datasource.source,
|
||||
checksum,
|
||||
{"status_code": None},
|
||||
connected_by="collection",
|
||||
)
|
||||
await _update_next_run_at(datasource, db)
|
||||
logger.info("Collector %s completed: %s", collector_name, task_result)
|
||||
logger.info_event(
|
||||
"Collector completed",
|
||||
event="collector.run.completed",
|
||||
context={"collector_name": collector_name, "datasource_id": datasource.id, "result": task_result},
|
||||
)
|
||||
except asyncio.CancelledError:
|
||||
datasource.last_run_at = datetime.now(UTC)
|
||||
datasource.last_status = "cancelled"
|
||||
await db.commit()
|
||||
logger.warning_event(
|
||||
"Collector cancelled by operator",
|
||||
event="collector.run.cancelled",
|
||||
context={"collector_name": collector_name, "datasource_id": datasource.id},
|
||||
)
|
||||
raise
|
||||
except Exception as exc:
|
||||
datasource.last_run_at = datetime.now(UTC)
|
||||
datasource.last_status = "failed"
|
||||
await db.commit()
|
||||
logger.exception("Collector %s failed: %s", collector_name, exc)
|
||||
logger.exception_event(
|
||||
"Collector failed",
|
||||
event="collector.run.failed",
|
||||
context={"collector_name": collector_name, "datasource_id": datasource.id, "error": str(exc)},
|
||||
)
|
||||
|
||||
|
||||
async def cleanup_stale_running_tasks(max_age_hours: int = 2) -> int:
|
||||
@@ -164,7 +252,11 @@ async def cleanup_stale_running_tasks(max_age_hours: int = 2) -> int:
|
||||
|
||||
if stale_tasks:
|
||||
await db.commit()
|
||||
logger.warning("Cleaned up %s stale running collection task(s)", len(stale_tasks))
|
||||
logger.warning_event(
|
||||
"Cleaned up stale running collection tasks",
|
||||
event="collector.cleanup.stale_tasks_cleaned",
|
||||
context={"count": len(stale_tasks)},
|
||||
)
|
||||
|
||||
return len(stale_tasks)
|
||||
|
||||
@@ -173,14 +265,14 @@ def start_scheduler() -> None:
|
||||
"""Start the scheduler."""
|
||||
if not scheduler.running:
|
||||
scheduler.start()
|
||||
logger.info("Scheduler started")
|
||||
logger.info_event("Scheduler started", event="scheduler.started")
|
||||
|
||||
|
||||
def stop_scheduler() -> None:
|
||||
"""Stop the scheduler."""
|
||||
if scheduler.running:
|
||||
scheduler.shutdown(wait=False)
|
||||
logger.info("Scheduler stopped")
|
||||
logger.info_event("Scheduler stopped", event="scheduler.stopped")
|
||||
|
||||
|
||||
async def sync_scheduler_with_datasources() -> None:
|
||||
@@ -241,13 +333,56 @@ def run_collector_now(collector_name: str) -> bool:
|
||||
"""Run a collector immediately (not scheduled)."""
|
||||
collector = collector_registry.get(collector_name)
|
||||
if not collector:
|
||||
logger.error("Collector not found: %s", collector_name)
|
||||
logger.error_event(
|
||||
"Collector not found",
|
||||
event="collector.trigger.collector_missing",
|
||||
context={"collector_name": collector_name},
|
||||
)
|
||||
return False
|
||||
|
||||
existing_task = get_running_collector_task(collector_name)
|
||||
if existing_task is not None and not existing_task.done():
|
||||
logger.warning_event(
|
||||
"Collector is already running in-memory; skipping duplicate trigger",
|
||||
event="collector.trigger.skipped_already_running",
|
||||
context={"collector_name": collector_name},
|
||||
)
|
||||
return False
|
||||
|
||||
try:
|
||||
asyncio.create_task(run_collector_task(collector_name))
|
||||
logger.info("Triggered collector: %s", collector_name)
|
||||
task = asyncio.create_task(run_collector_task(collector_name), name=_collector_task_name(collector_name))
|
||||
RUNNING_COLLECTOR_TASKS[collector_name] = task
|
||||
|
||||
def _cleanup_task(done_task: asyncio.Task[Any]) -> None:
|
||||
current = RUNNING_COLLECTOR_TASKS.get(collector_name)
|
||||
if current is done_task:
|
||||
RUNNING_COLLECTOR_TASKS.pop(collector_name, None)
|
||||
|
||||
task.add_done_callback(_cleanup_task)
|
||||
logger.info_event(
|
||||
"Triggered collector",
|
||||
event="collector.trigger.started",
|
||||
context={"collector_name": collector_name},
|
||||
)
|
||||
return True
|
||||
except Exception as exc:
|
||||
logger.error("Failed to trigger collector %s: %s", collector_name, exc)
|
||||
return False
|
||||
logger.error_event(
|
||||
"Failed to trigger collector",
|
||||
event="collector.trigger.failed",
|
||||
context={"collector_name": collector_name, "error": str(exc)},
|
||||
)
|
||||
return False
|
||||
|
||||
|
||||
async def cancel_running_collector_now(collector_name: str) -> bool:
|
||||
task = get_running_collector_task(collector_name)
|
||||
if task is None or task.done():
|
||||
RUNNING_COLLECTOR_TASKS.pop(collector_name, None)
|
||||
return False
|
||||
|
||||
task.cancel()
|
||||
try:
|
||||
await task
|
||||
except asyncio.CancelledError:
|
||||
return True
|
||||
return task.cancelled()
|
||||
@@ -0,0 +1,174 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections import Counter
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.models.alert import Alert, AlertSeverity, AlertStatus
|
||||
from app.models.bgp_anomaly import BGPAnomaly
|
||||
from app.models.bgp_incident import BGPIncident
|
||||
from app.schemas.ai import SituationalAnalysisRequest
|
||||
from app.services.bgp_ai_brief_store import get_latest_bgp_brief_record
|
||||
|
||||
|
||||
def _format_pairs(pairs: list[tuple[str, int]], empty_text: str = "无") -> str:
|
||||
if not pairs:
|
||||
return empty_text
|
||||
return ",".join(f"{key} {value}" for key, value in pairs if key)
|
||||
|
||||
|
||||
async def build_situational_alert_brief_request(
|
||||
db: AsyncSession,
|
||||
) -> tuple[SituationalAnalysisRequest, list[str], dict[str, Any]]:
|
||||
total_alerts_result = await db.execute(select(func.count(Alert.id)))
|
||||
active_alerts_result = await db.execute(
|
||||
select(func.count(Alert.id)).where(Alert.status == AlertStatus.ACTIVE)
|
||||
)
|
||||
alert_severity_result = await db.execute(
|
||||
select(Alert.severity, func.count(Alert.id))
|
||||
.where(Alert.status == AlertStatus.ACTIVE)
|
||||
.group_by(Alert.severity)
|
||||
)
|
||||
alert_source_result = await db.execute(
|
||||
select(Alert.datasource_name, func.count(Alert.id))
|
||||
.where(Alert.status == AlertStatus.ACTIVE)
|
||||
.group_by(Alert.datasource_name)
|
||||
.order_by(func.count(Alert.id).desc())
|
||||
.limit(6)
|
||||
)
|
||||
recent_alerts_result = await db.execute(
|
||||
select(Alert)
|
||||
.order_by(Alert.created_at.desc(), Alert.id.desc())
|
||||
.limit(6)
|
||||
)
|
||||
|
||||
total_incidents_result = await db.execute(select(func.count(BGPIncident.id)))
|
||||
active_incidents_result = await db.execute(
|
||||
select(func.count(BGPIncident.id)).where(BGPIncident.status == "active")
|
||||
)
|
||||
bgp_severity_result = await db.execute(
|
||||
select(BGPIncident.severity, func.count(BGPIncident.id))
|
||||
.where(BGPIncident.status == "active")
|
||||
.group_by(BGPIncident.severity)
|
||||
)
|
||||
bgp_region_counter: Counter[str] = Counter()
|
||||
recent_incidents_result = await db.execute(
|
||||
select(BGPIncident)
|
||||
.order_by(BGPIncident.created_at.desc(), BGPIncident.id.desc())
|
||||
.limit(5)
|
||||
)
|
||||
|
||||
total_anomalies_result = await db.execute(select(func.count(BGPAnomaly.id)))
|
||||
active_anomalies_result = await db.execute(
|
||||
select(func.count(BGPAnomaly.id)).where(BGPAnomaly.status == "active")
|
||||
)
|
||||
anomaly_type_result = await db.execute(
|
||||
select(BGPAnomaly.anomaly_type, func.count(BGPAnomaly.id))
|
||||
.where(BGPAnomaly.status == "active")
|
||||
.group_by(BGPAnomaly.anomaly_type)
|
||||
.order_by(func.count(BGPAnomaly.id).desc())
|
||||
.limit(6)
|
||||
)
|
||||
|
||||
recent_incidents = recent_incidents_result.scalars().all()
|
||||
for incident in recent_incidents:
|
||||
for region in incident.affected_regions or []:
|
||||
if not isinstance(region, dict):
|
||||
continue
|
||||
label = ", ".join(part for part in [region.get("city"), region.get("country")] if part) or "未知区域"
|
||||
bgp_region_counter[label] += 1
|
||||
|
||||
latest_bgp_brief = get_latest_bgp_brief_record()
|
||||
active_alert_severities = [
|
||||
(item[0].value if isinstance(item[0], AlertSeverity) else str(item[0]), item[1])
|
||||
for item in alert_severity_result.fetchall()
|
||||
if item[0]
|
||||
]
|
||||
active_bgp_severities = [
|
||||
(str(item[0]), item[1])
|
||||
for item in bgp_severity_result.fetchall()
|
||||
if item[0]
|
||||
]
|
||||
active_anomaly_types = [(str(item[0]), item[1]) for item in anomaly_type_result.fetchall() if item[0]]
|
||||
active_alert_sources = [
|
||||
(str(item[0] or "未命名数据源"), item[1])
|
||||
for item in alert_source_result.fetchall()
|
||||
]
|
||||
|
||||
facts = [
|
||||
(
|
||||
f"系统告警侧:总告警 {total_alerts_result.scalar() or 0} 条,active {active_alerts_result.scalar() or 0} 条;"
|
||||
f"活跃告警严重度分布为 {_format_pairs(active_alert_severities)}。"
|
||||
),
|
||||
(
|
||||
f"BGP态势侧:累计 incidents {total_incidents_result.scalar() or 0} 条,active incidents {active_incidents_result.scalar() or 0} 条;"
|
||||
f"活跃 incidents 严重度分布为 {_format_pairs(active_bgp_severities)}。"
|
||||
),
|
||||
(
|
||||
f"BGP异常侧:累计 anomalies {total_anomalies_result.scalar() or 0} 条,active anomalies {active_anomalies_result.scalar() or 0} 条;"
|
||||
f"活跃 anomaly 类型分布为 {_format_pairs(active_anomaly_types)}。"
|
||||
),
|
||||
]
|
||||
|
||||
if active_alert_sources:
|
||||
facts.append(f"当前系统告警主要集中在:{_format_pairs(active_alert_sources)}。")
|
||||
if bgp_region_counter:
|
||||
facts.append(f"BGP近期高风险区域线索:{_format_pairs(bgp_region_counter.most_common(5))}。")
|
||||
|
||||
recent_alerts = recent_alerts_result.scalars().all()
|
||||
if recent_alerts:
|
||||
facts.append(
|
||||
"最近系统告警摘录:"
|
||||
+ ";".join(
|
||||
[
|
||||
f"{alert.datasource_name or '未命名数据源'} / {alert.severity.value if alert.severity else '-'} / {alert.status.value if alert.status else '-'} / {alert.message or '-'}"
|
||||
for alert in recent_alerts
|
||||
]
|
||||
)
|
||||
)
|
||||
|
||||
if recent_incidents:
|
||||
facts.append(
|
||||
"最近BGP事件摘录:"
|
||||
+ ";".join(
|
||||
[
|
||||
f"{incident.incident_type} / {incident.severity} / {incident.status} / {incident.summary}"
|
||||
for incident in recent_incidents
|
||||
]
|
||||
)
|
||||
)
|
||||
|
||||
if latest_bgp_brief:
|
||||
facts.append(
|
||||
f"最近一份 BGP AI 简报生成于 {latest_bgp_brief.generated_at},模型 {latest_bgp_brief.model},可作为当前态势的补充说明。"
|
||||
)
|
||||
|
||||
context = {
|
||||
"source": "situational-alerts",
|
||||
"active_system_alerts": active_alerts_result.scalar() or 0,
|
||||
"active_system_alert_severities": dict(active_alert_severities),
|
||||
"top_system_alert_sources": dict(active_alert_sources),
|
||||
"active_bgp_incidents": active_incidents_result.scalar() or 0,
|
||||
"active_bgp_incident_severities": dict(active_bgp_severities),
|
||||
"active_bgp_anomalies": active_anomalies_result.scalar() or 0,
|
||||
"active_bgp_anomaly_types": dict(active_anomaly_types),
|
||||
"bgp_hot_regions": dict(bgp_region_counter.most_common(5)),
|
||||
"latest_bgp_brief_id": latest_bgp_brief.id if latest_bgp_brief else None,
|
||||
"latest_bgp_brief_generated_at": latest_bgp_brief.generated_at if latest_bgp_brief else None,
|
||||
}
|
||||
|
||||
request = SituationalAnalysisRequest(
|
||||
title="态势告警 AI 简报",
|
||||
objective="综合系统告警、BGP incidents、BGP anomalies 与近期 BGP AI 简报,生成一份面向值班人员的态势告警简报,指出当前最需要关注的风险域、跨模块联动迹象和优先动作。",
|
||||
observations=facts,
|
||||
constraints=[
|
||||
"明确区分事实、推断与建议。",
|
||||
"优先指出仍在 active 状态的系统告警与 BGP 风险是否存在联动。",
|
||||
"不要把单一数据源的局部异常夸大成全局态势。",
|
||||
"如果证据不足,请明确写出仍缺哪些模块或区域信息。",
|
||||
],
|
||||
context=context,
|
||||
)
|
||||
return request, facts, context
|
||||
@@ -19,6 +19,10 @@ ALLOWED_ACTIONS: dict[str, dict[str, Any]] = {
|
||||
"command": ["./planet.sh", "restart", "-b"],
|
||||
"recovery_mode": "backend",
|
||||
},
|
||||
"restart-frontend": {
|
||||
"command": ["./planet.sh", "restart", "-f"],
|
||||
"recovery_mode": "frontend",
|
||||
},
|
||||
"restart-ai-provider": {
|
||||
"command": ["./planet.sh", "restart", "-a"],
|
||||
"recovery_mode": "ai-provider",
|
||||
|
||||
@@ -0,0 +1,532 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
|
||||
from collections import Counter, deque
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from app.core.security import redis_client
|
||||
|
||||
DEFAULT_LOG_LINE_LIMIT = 200
|
||||
MAX_LOG_LINE_LIMIT = 1000
|
||||
BUFFER_LOG_LIMIT = 1000
|
||||
BUFFER_LOG_TTL_SECONDS = 7 * 24 * 60 * 60
|
||||
LOG_BUFFER_KEY_PREFIX = "planet:system_logs"
|
||||
|
||||
LOG_LEVEL_ERROR = "error"
|
||||
LOG_LEVEL_WARNING = "warning"
|
||||
LOG_LEVEL_INFO = "info"
|
||||
LOG_LEVEL_DEBUG = "debug"
|
||||
LOG_LEVEL_ALL = "all"
|
||||
|
||||
SUPPORTED_LOG_LEVELS = {
|
||||
LOG_LEVEL_ALL,
|
||||
LOG_LEVEL_ERROR,
|
||||
LOG_LEVEL_WARNING,
|
||||
LOG_LEVEL_INFO,
|
||||
LOG_LEVEL_DEBUG,
|
||||
}
|
||||
|
||||
LOG_LEVEL_ALIASES = {
|
||||
"warn": LOG_LEVEL_WARNING,
|
||||
"warning": LOG_LEVEL_WARNING,
|
||||
"err": LOG_LEVEL_ERROR,
|
||||
"error": LOG_LEVEL_ERROR,
|
||||
"info": LOG_LEVEL_INFO,
|
||||
"information": LOG_LEVEL_INFO,
|
||||
"debug": LOG_LEVEL_DEBUG,
|
||||
"trace": LOG_LEVEL_DEBUG,
|
||||
"critical": LOG_LEVEL_ERROR,
|
||||
"fatal": LOG_LEVEL_ERROR,
|
||||
}
|
||||
|
||||
TIMESTAMP_FORMATS = (
|
||||
"%Y-%m-%d %H:%M:%S.%f",
|
||||
"%Y-%m-%d %H:%M:%S",
|
||||
"%Y-%m-%dT%H:%M:%S.%f",
|
||||
"%Y-%m-%dT%H:%M:%S",
|
||||
)
|
||||
|
||||
LEVEL_PATTERNS = (
|
||||
("CRITICAL", LOG_LEVEL_ERROR),
|
||||
("FATAL", LOG_LEVEL_ERROR),
|
||||
("ERROR", LOG_LEVEL_ERROR),
|
||||
("WARNING", LOG_LEVEL_WARNING),
|
||||
("WARN", LOG_LEVEL_WARNING),
|
||||
("INFO", LOG_LEVEL_INFO),
|
||||
("DEBUG", LOG_LEVEL_DEBUG),
|
||||
("TRACE", LOG_LEVEL_DEBUG),
|
||||
)
|
||||
|
||||
LEADING_LEVEL_PATTERN = re.compile(
|
||||
r"^\s*(?:\[[^\]]+\]\s*)?(CRITICAL|FATAL|ERROR|WARNING|WARN|INFO|DEBUG|TRACE)\b[:\s-]*",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
EMBEDDED_LEVEL_PATTERN = re.compile(
|
||||
r"\b(CRITICAL|FATAL|ERROR|WARNING|WARN|INFO|DEBUG|TRACE)\b",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
CONTROL_CHAR_PATTERN = re.compile(r"[\x00-\x08\x0b\x0c\x0e-\x1f\x7f]")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class LogSource:
|
||||
source_id: str
|
||||
name: str
|
||||
kind: str
|
||||
location: str
|
||||
description: str
|
||||
category: str
|
||||
status: str = "ok"
|
||||
buffer_key: str | None = None
|
||||
container_name: str | None = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class StructuredLogEntry:
|
||||
timestamp: datetime | None
|
||||
level: str | None
|
||||
display_line: str
|
||||
raw_line: str
|
||||
search_text: str
|
||||
|
||||
|
||||
@dataclass
|
||||
class DailyLogMarker:
|
||||
date_token: str
|
||||
total: int
|
||||
dominant_level: str
|
||||
|
||||
|
||||
LOG_SOURCES: dict[str, LogSource] = {
|
||||
"backend": LogSource(
|
||||
source_id="backend",
|
||||
name="后端服务",
|
||||
kind="file",
|
||||
location="/tmp/planet_backend.log",
|
||||
description="FastAPI 后端、调度器和采集任务共享日志。",
|
||||
category="service",
|
||||
),
|
||||
"frontend": LogSource(
|
||||
source_id="frontend",
|
||||
name="前端开发服务",
|
||||
kind="file",
|
||||
location="/tmp/planet_frontend.log",
|
||||
description="控制台与 Earth 前端开发服务输出。",
|
||||
category="service",
|
||||
),
|
||||
"ai-provider": LogSource(
|
||||
source_id="ai-provider",
|
||||
name="AI Provider",
|
||||
kind="docker",
|
||||
location="docker://planet_aiprovider",
|
||||
description="AI Provider 容器实时输出日志。",
|
||||
category="service",
|
||||
container_name="planet_aiprovider",
|
||||
),
|
||||
"earth-client": LogSource(
|
||||
source_id="earth-client",
|
||||
name="Earth 浏览器端",
|
||||
kind="buffer",
|
||||
location="redis://planet:system_logs:earth-client",
|
||||
description="Earth 浏览器端上报的运行时错误与关键业务日志。",
|
||||
category="client",
|
||||
buffer_key=f"{LOG_BUFFER_KEY_PREFIX}:earth-client",
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def normalize_log_level(level: str | None) -> str:
|
||||
if level is None:
|
||||
return LOG_LEVEL_ALL
|
||||
normalized = str(level).strip().lower()
|
||||
if normalized in {"", LOG_LEVEL_ALL}:
|
||||
return LOG_LEVEL_ALL
|
||||
return LOG_LEVEL_ALIASES.get(normalized, LOG_LEVEL_ALL)
|
||||
|
||||
|
||||
def normalize_log_levels(level: str | None = None, levels: str | None = None) -> tuple[str, ...]:
|
||||
normalized_levels: list[str] = []
|
||||
if levels:
|
||||
for item in str(levels).split(","):
|
||||
normalized = normalize_log_level(item)
|
||||
if normalized != LOG_LEVEL_ALL and normalized not in normalized_levels:
|
||||
normalized_levels.append(normalized)
|
||||
normalized_level = normalize_log_level(level)
|
||||
if normalized_level != LOG_LEVEL_ALL and normalized_level not in normalized_levels:
|
||||
normalized_levels.append(normalized_level)
|
||||
return tuple(normalized_levels)
|
||||
|
||||
|
||||
def get_source_status(source: LogSource) -> str:
|
||||
if source.kind == "file":
|
||||
path = Path(source.location)
|
||||
if not path.exists():
|
||||
return "missing"
|
||||
return "ok" if path.stat().st_size > 0 else "empty"
|
||||
if source.kind == "docker":
|
||||
return "ok" if shutil.which("docker") else "docker_unavailable"
|
||||
if source.kind == "buffer":
|
||||
if not source.buffer_key:
|
||||
return "source_unavailable"
|
||||
try:
|
||||
return "ok" if redis_client.llen(source.buffer_key) > 0 else "empty"
|
||||
except Exception:
|
||||
return "source_unavailable"
|
||||
return "source_unavailable"
|
||||
|
||||
|
||||
def list_log_sources() -> list[dict[str, str]]:
|
||||
items: list[dict[str, str]] = []
|
||||
for source in LOG_SOURCES.values():
|
||||
items.append(
|
||||
{
|
||||
"source_id": source.source_id,
|
||||
"name": source.name,
|
||||
"kind": source.kind,
|
||||
"location": source.location,
|
||||
"description": source.description,
|
||||
"category": source.category,
|
||||
"status": get_source_status(source),
|
||||
}
|
||||
)
|
||||
return items
|
||||
|
||||
|
||||
def get_buffer_log_key(source_id: str) -> str:
|
||||
return f"{LOG_BUFFER_KEY_PREFIX}:{source_id}"
|
||||
|
||||
|
||||
def append_buffer_log(
|
||||
source_id: str,
|
||||
*,
|
||||
level: str,
|
||||
message: str,
|
||||
context: dict[str, Any] | None = None,
|
||||
) -> None:
|
||||
payload = {
|
||||
"timestamp": datetime.now(tz=UTC).isoformat(),
|
||||
"level": normalize_log_level(level),
|
||||
"message": message,
|
||||
"context": context or {},
|
||||
}
|
||||
buffer_key = get_buffer_log_key(source_id)
|
||||
redis_client.rpush(buffer_key, json.dumps(payload, ensure_ascii=False))
|
||||
redis_client.ltrim(buffer_key, -BUFFER_LOG_LIMIT, -1)
|
||||
redis_client.expire(buffer_key, BUFFER_LOG_TTL_SECONDS)
|
||||
|
||||
|
||||
def parse_timestamp(raw_value: str | None) -> datetime | None:
|
||||
if not raw_value:
|
||||
return None
|
||||
candidate = str(raw_value).strip()
|
||||
if not candidate:
|
||||
return None
|
||||
candidate = candidate.replace("Z", "+00:00")
|
||||
try:
|
||||
parsed = datetime.fromisoformat(candidate)
|
||||
return parsed if parsed.tzinfo else parsed.replace(tzinfo=UTC)
|
||||
except ValueError:
|
||||
pass
|
||||
for fmt in TIMESTAMP_FORMATS:
|
||||
try:
|
||||
return datetime.strptime(candidate, fmt).replace(tzinfo=UTC)
|
||||
except ValueError:
|
||||
continue
|
||||
return None
|
||||
|
||||
|
||||
def parse_prefixed_timestamp(line: str) -> tuple[datetime | None, str]:
|
||||
stripped = line.strip()
|
||||
if not stripped:
|
||||
return None, ""
|
||||
for prefix_length in (35, 32, 29, 26, 23, 19):
|
||||
if len(stripped) < prefix_length:
|
||||
continue
|
||||
prefix = stripped[:prefix_length]
|
||||
timestamp = parse_timestamp(prefix)
|
||||
if timestamp is not None:
|
||||
return timestamp, stripped[prefix_length:].lstrip()
|
||||
first_token = stripped.split(maxsplit=1)[0]
|
||||
timestamp = parse_timestamp(first_token)
|
||||
if timestamp is not None:
|
||||
remainder = stripped[len(first_token):].lstrip()
|
||||
return timestamp, remainder
|
||||
return None, stripped
|
||||
|
||||
|
||||
def infer_log_level_from_text(text: str, *, allow_embedded: bool = True) -> str | None:
|
||||
leading_match = LEADING_LEVEL_PATTERN.match(text)
|
||||
if leading_match:
|
||||
return normalize_log_level(leading_match.group(1))
|
||||
|
||||
if allow_embedded:
|
||||
embedded_match = EMBEDDED_LEVEL_PATTERN.search(text)
|
||||
if embedded_match:
|
||||
return normalize_log_level(embedded_match.group(1))
|
||||
upper_text = text.upper()
|
||||
for pattern, normalized in LEVEL_PATTERNS:
|
||||
if f"{pattern}:" in upper_text or f"{pattern} " in upper_text:
|
||||
return normalized
|
||||
return None
|
||||
|
||||
|
||||
def build_display_line(timestamp: datetime | None, level: str | None, message: str) -> str:
|
||||
message_part = message.strip() if message else ""
|
||||
parts = []
|
||||
if timestamp is not None:
|
||||
parts.append(timestamp.astimezone(UTC).strftime("%Y-%m-%d %H:%M:%S"))
|
||||
if level:
|
||||
parts.append(level.upper())
|
||||
if message_part:
|
||||
parts.append(message_part)
|
||||
return " ".join(parts).strip()
|
||||
|
||||
|
||||
def sanitize_text_log_line(line: str) -> str:
|
||||
return CONTROL_CHAR_PATTERN.sub("", line)
|
||||
|
||||
|
||||
def parse_text_log_entry(line: str) -> StructuredLogEntry:
|
||||
sanitized_line = sanitize_text_log_line(line).rstrip("\n")
|
||||
timestamp, remainder = parse_prefixed_timestamp(sanitized_line)
|
||||
level = infer_log_level_from_text(remainder or sanitized_line, allow_embedded=False)
|
||||
display_line = sanitized_line
|
||||
return StructuredLogEntry(
|
||||
timestamp=timestamp,
|
||||
level=level,
|
||||
display_line=display_line,
|
||||
raw_line=display_line,
|
||||
search_text=display_line.lower(),
|
||||
)
|
||||
|
||||
|
||||
def build_buffer_entry(payload: dict[str, Any]) -> StructuredLogEntry:
|
||||
timestamp = parse_timestamp(str(payload.get("timestamp", "")).strip())
|
||||
level = normalize_log_level(payload.get("level"))
|
||||
if level == LOG_LEVEL_ALL:
|
||||
level = None
|
||||
message = str(payload.get("message", "")).strip()
|
||||
context = payload.get("context")
|
||||
context_map = context if isinstance(context, dict) else {}
|
||||
context_fragments = []
|
||||
for key in ("category", "module", "url", "detail"):
|
||||
value = str(context_map.get(key, "")).strip()
|
||||
if value:
|
||||
context_fragments.append(f"{key}={value}")
|
||||
message_with_context = " | ".join([message, *context_fragments]) if context_fragments else message
|
||||
display_line = build_display_line(timestamp, level, message_with_context)
|
||||
search_text = " ".join(
|
||||
[
|
||||
message,
|
||||
json.dumps(context_map, ensure_ascii=False, sort_keys=True),
|
||||
display_line,
|
||||
]
|
||||
).lower()
|
||||
return StructuredLogEntry(
|
||||
timestamp=timestamp,
|
||||
level=level,
|
||||
display_line=display_line,
|
||||
raw_line=json.dumps(payload, ensure_ascii=False, sort_keys=True),
|
||||
search_text=search_text,
|
||||
)
|
||||
|
||||
|
||||
def read_file_entries(source: LogSource, scan_limit: int) -> list[StructuredLogEntry]:
|
||||
path = Path(source.location)
|
||||
if not path.exists():
|
||||
return []
|
||||
with path.open("r", encoding="utf-8", errors="replace") as handle:
|
||||
recent_lines = deque(handle, maxlen=scan_limit)
|
||||
return [
|
||||
parse_text_log_entry(line)
|
||||
for line in recent_lines
|
||||
if sanitize_text_log_line(line).strip()
|
||||
]
|
||||
|
||||
|
||||
def read_docker_entries(source: LogSource, scan_limit: int) -> list[StructuredLogEntry]:
|
||||
if not shutil.which("docker") or not source.container_name:
|
||||
return []
|
||||
try:
|
||||
completed = subprocess.run(
|
||||
[
|
||||
"docker",
|
||||
"logs",
|
||||
"--timestamps",
|
||||
"--tail",
|
||||
str(scan_limit),
|
||||
source.container_name,
|
||||
],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
except OSError:
|
||||
return []
|
||||
if completed.returncode != 0:
|
||||
return []
|
||||
return [
|
||||
parse_text_log_entry(line)
|
||||
for line in completed.stdout.splitlines()
|
||||
if line.strip()
|
||||
]
|
||||
|
||||
|
||||
def read_buffer_entries(source: LogSource, scan_limit: int) -> list[StructuredLogEntry]:
|
||||
if not source.buffer_key:
|
||||
return []
|
||||
try:
|
||||
raw_items = redis_client.lrange(source.buffer_key, -scan_limit, -1)
|
||||
except Exception:
|
||||
return []
|
||||
entries: list[StructuredLogEntry] = []
|
||||
for raw_item in raw_items:
|
||||
try:
|
||||
payload = json.loads(raw_item)
|
||||
except json.JSONDecodeError:
|
||||
entries.append(parse_text_log_entry(str(raw_item)))
|
||||
continue
|
||||
if isinstance(payload, dict):
|
||||
entries.append(build_buffer_entry(payload))
|
||||
else:
|
||||
entries.append(parse_text_log_entry(str(raw_item)))
|
||||
return entries
|
||||
|
||||
|
||||
def read_source_entries(source: LogSource, scan_limit: int) -> list[StructuredLogEntry]:
|
||||
if source.kind == "file":
|
||||
return read_file_entries(source, scan_limit)
|
||||
if source.kind == "docker":
|
||||
return read_docker_entries(source, scan_limit)
|
||||
if source.kind == "buffer":
|
||||
return read_buffer_entries(source, scan_limit)
|
||||
return []
|
||||
|
||||
|
||||
def matches_levels(entry: StructuredLogEntry, selected_levels: tuple[str, ...]) -> bool:
|
||||
if not selected_levels:
|
||||
return True
|
||||
return entry.level in selected_levels
|
||||
|
||||
|
||||
def matches_date_range(
|
||||
entry: StructuredLogEntry,
|
||||
start_date: str | None,
|
||||
end_date: str | None,
|
||||
) -> bool:
|
||||
if not start_date and not end_date:
|
||||
return True
|
||||
if entry.timestamp is None:
|
||||
return False
|
||||
date_token = entry.timestamp.astimezone(UTC).date().isoformat()
|
||||
if start_date and date_token < start_date:
|
||||
return False
|
||||
if end_date and date_token > end_date:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def matches_search(entry: StructuredLogEntry, search: str | None) -> bool:
|
||||
if search is None:
|
||||
return True
|
||||
query = search.strip().lower()
|
||||
if not query:
|
||||
return True
|
||||
return query in entry.search_text
|
||||
|
||||
|
||||
def build_daily_log_markers(entries: list[StructuredLogEntry]) -> list[dict[str, Any]]:
|
||||
grouped: dict[str, list[StructuredLogEntry]] = {}
|
||||
for entry in entries:
|
||||
if entry.timestamp is None:
|
||||
continue
|
||||
date_token = entry.timestamp.astimezone(UTC).date().isoformat()
|
||||
grouped.setdefault(date_token, []).append(entry)
|
||||
|
||||
markers: list[DailyLogMarker] = []
|
||||
for date_token, group in sorted(grouped.items()):
|
||||
level_counts = Counter(
|
||||
entry.level
|
||||
for entry in group
|
||||
if entry.level in SUPPORTED_LOG_LEVELS and entry.level != LOG_LEVEL_ALL
|
||||
)
|
||||
dominant_level = LOG_LEVEL_INFO
|
||||
if level_counts:
|
||||
dominant_level = sorted(
|
||||
level_counts.items(),
|
||||
key=lambda item: (
|
||||
-item[1],
|
||||
("error", "warning", "info", "debug").index(item[0]),
|
||||
),
|
||||
)[0][0]
|
||||
markers.append(
|
||||
DailyLogMarker(
|
||||
date_token=date_token,
|
||||
total=len(group),
|
||||
dominant_level=dominant_level,
|
||||
)
|
||||
)
|
||||
return [marker.__dict__ for marker in markers]
|
||||
|
||||
|
||||
def read_log_snapshot(
|
||||
source_id: str,
|
||||
limit: int,
|
||||
*,
|
||||
level: str = LOG_LEVEL_ALL,
|
||||
levels: str | None = None,
|
||||
start_date: str | None = None,
|
||||
end_date: str | None = None,
|
||||
search: str | None = None,
|
||||
) -> dict[str, Any] | None:
|
||||
source = LOG_SOURCES.get(source_id)
|
||||
if source is None:
|
||||
return None
|
||||
|
||||
selected_levels = normalize_log_levels(level, levels)
|
||||
search_query = (search or "").strip()
|
||||
scan_limit = max(min(MAX_LOG_LINE_LIMIT * 5, 5000), limit * 5, BUFFER_LOG_LIMIT if source.kind == "buffer" else 1000)
|
||||
all_entries = read_source_entries(source, scan_limit)
|
||||
marker_entries = [
|
||||
entry
|
||||
for entry in all_entries
|
||||
if matches_levels(entry, selected_levels) and matches_search(entry, search_query)
|
||||
]
|
||||
filtered_entries = [
|
||||
entry
|
||||
for entry in marker_entries
|
||||
if matches_date_range(entry, start_date, end_date)
|
||||
]
|
||||
visible_entries = filtered_entries[-limit:]
|
||||
|
||||
compatibility_level = selected_levels[0] if len(selected_levels) == 1 else LOG_LEVEL_ALL
|
||||
return {
|
||||
"source_id": source.source_id,
|
||||
"name": source.name,
|
||||
"kind": source.kind,
|
||||
"location": source.location,
|
||||
"description": source.description,
|
||||
"category": source.category,
|
||||
"status": get_source_status(source),
|
||||
"level": compatibility_level,
|
||||
"selected_levels": list(selected_levels),
|
||||
"search_query": search_query,
|
||||
"available_levels": [
|
||||
LOG_LEVEL_ALL,
|
||||
LOG_LEVEL_ERROR,
|
||||
LOG_LEVEL_WARNING,
|
||||
LOG_LEVEL_INFO,
|
||||
LOG_LEVEL_DEBUG,
|
||||
],
|
||||
"daily_markers": build_daily_log_markers(marker_entries),
|
||||
"line_limit": limit,
|
||||
"line_count": len(visible_entries),
|
||||
"lines": [entry.display_line for entry in visible_entries],
|
||||
}
|
||||
@@ -0,0 +1,466 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.models.system_setting import SystemSetting
|
||||
|
||||
DEFAULT_TV_SOURCE_ID = "cgtn-en"
|
||||
TV_SETTINGS_CATEGORY = "tv"
|
||||
TV_LIVE_SOURCE_COLLECTOR = "news_live_streams"
|
||||
TV_LIVE_SOURCE_DATA_TYPE = "news_live_stream"
|
||||
|
||||
DEFAULT_TV_SETTINGS = {
|
||||
"default_source_id": DEFAULT_TV_SOURCE_ID,
|
||||
"auto_fallback": True,
|
||||
"sources": [
|
||||
{
|
||||
"id": "cctv4",
|
||||
"name": "CCTV-4 中文国际",
|
||||
"provider": "CCTV",
|
||||
"region": "China",
|
||||
"language": "zh-CN",
|
||||
"source_type": "hls",
|
||||
"embed_url": "https://tv.cctv.com/live/cctv4/",
|
||||
"stream_url": "https://ldocctvwbcdtxy.liveplay.myqcloud.com/ldocctvwbcd/cdrmldcctv4_1_td.m3u8",
|
||||
"homepage_url": "https://tv.cctv.com/live/cctv4/",
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": True,
|
||||
"sort_order": 10,
|
||||
"collector_source": None,
|
||||
"notes": "默认兜底新闻直播源。优先尝试 CCTV-4 官方 HLS 播放流,若直播放失败则回退到央视官网直播页。",
|
||||
},
|
||||
{
|
||||
"id": "reuters-tv",
|
||||
"name": "Reuters TV",
|
||||
"provider": "Reuters",
|
||||
"region": "Global",
|
||||
"language": "en",
|
||||
"source_type": "hls",
|
||||
"embed_url": "",
|
||||
"stream_url": "https://reuters-reutersnow-1-eu.rakuten.wurl.tv/playlist.m3u8",
|
||||
"homepage_url": "https://www.reuters.com/video/live/",
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": False,
|
||||
"sort_order": 20,
|
||||
"collector_source": None,
|
||||
"notes": "参考 worldmonitor 的默认新闻频道清单,优先作为全球英文新闻直播放源。",
|
||||
},
|
||||
{
|
||||
"id": "cgtn-en",
|
||||
"name": "CGTN English",
|
||||
"provider": "CGTN",
|
||||
"region": "Global",
|
||||
"language": "en",
|
||||
"source_type": "youtube",
|
||||
"embed_url": "https://www.youtube.com/watch?v=BOy2xDU1LC8",
|
||||
"stream_url": "https://news.cgtn.com/resource/live/english/cgtn-news.m3u8",
|
||||
"youtube_video_id": "BOy2xDU1LC8",
|
||||
"homepage_url": "https://news.cgtn.com/",
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": False,
|
||||
"sort_order": 30,
|
||||
"collector_source": None,
|
||||
"notes": "优先使用官方 YouTube 直播源,保留 HLS 直播放流作为候选信息。",
|
||||
},
|
||||
{
|
||||
"id": "cgtn-es",
|
||||
"name": "CGTN Espanol",
|
||||
"provider": "CGTN",
|
||||
"region": "Latin America",
|
||||
"language": "es",
|
||||
"source_type": "hls",
|
||||
"embed_url": "",
|
||||
"stream_url": "https://news.cgtn.com/resource/live/espanol/cgtn-e.m3u8",
|
||||
"homepage_url": "https://news.cgtn.com/",
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": False,
|
||||
"sort_order": 40,
|
||||
"collector_source": None,
|
||||
"notes": "西语国际新闻频道,覆盖拉美方向态势。",
|
||||
},
|
||||
{
|
||||
"id": "dw-espanol",
|
||||
"name": "DW Espanol",
|
||||
"provider": "Deutsche Welle",
|
||||
"region": "Europe",
|
||||
"language": "es",
|
||||
"source_type": "hls",
|
||||
"embed_url": "",
|
||||
"stream_url": "https://dwamdstream104.akamaized.net/hls/live/2015530/dwstream104/stream04/streamPlaylist.m3u8",
|
||||
"homepage_url": "https://www.dw.com/es/",
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": False,
|
||||
"sort_order": 50,
|
||||
"collector_source": None,
|
||||
"notes": "来自 worldmonitor 可选频道清单的直播放源。",
|
||||
},
|
||||
{
|
||||
"id": "dw-arabic",
|
||||
"name": "DW Arabic",
|
||||
"provider": "Deutsche Welle",
|
||||
"region": "Middle East",
|
||||
"language": "ar",
|
||||
"source_type": "hls",
|
||||
"embed_url": "",
|
||||
"stream_url": "https://dwamdstream103.akamaized.net/hls/live/2015526/dwstream103/index.m3u8",
|
||||
"homepage_url": "https://www.dw.com/ar/",
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": False,
|
||||
"sort_order": 60,
|
||||
"collector_source": None,
|
||||
"notes": "阿拉伯语新闻流,适合作为中东方向新闻补充源。",
|
||||
},
|
||||
{
|
||||
"id": "aljazeera-mubasher",
|
||||
"name": "Al Jazeera Mubasher",
|
||||
"provider": "Al Jazeera",
|
||||
"region": "Middle East",
|
||||
"language": "ar",
|
||||
"source_type": "hls",
|
||||
"embed_url": "",
|
||||
"stream_url": "https://live-hls-web-ajm.getaj.net/AJM/index.m3u8",
|
||||
"homepage_url": "https://www.aljazeera.net/live",
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": False,
|
||||
"sort_order": 70,
|
||||
"collector_source": None,
|
||||
"notes": "中东实时新闻流,来自 worldmonitor HLS 频道目录。",
|
||||
},
|
||||
{
|
||||
"id": "arirang-news",
|
||||
"name": "Arirang News",
|
||||
"provider": "Arirang",
|
||||
"region": "Korea",
|
||||
"language": "en",
|
||||
"source_type": "hls",
|
||||
"embed_url": "",
|
||||
"stream_url": "https://amdlive-ch01-ctnd-com.akamaized.net/arirang_1ch/smil:arirang_1ch.smil/playlist.m3u8",
|
||||
"homepage_url": "https://www.arirang.com/",
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": False,
|
||||
"sort_order": 80,
|
||||
"collector_source": None,
|
||||
"notes": "东北亚英语新闻源,适合补充韩半岛与东亚视角。",
|
||||
},
|
||||
{
|
||||
"id": "abp-news",
|
||||
"name": "ABP News",
|
||||
"provider": "ABP",
|
||||
"region": "India",
|
||||
"language": "hi",
|
||||
"source_type": "hls",
|
||||
"embed_url": "",
|
||||
"stream_url": "https://abplivetv.pc.cdn.bitgravity.com/httppush/abp_livetv/abp_abpnews/master.m3u8",
|
||||
"homepage_url": "https://news.abplive.com/live-tv",
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": False,
|
||||
"sort_order": 90,
|
||||
"collector_source": None,
|
||||
"notes": "印度新闻直播放源,补充南亚区域视角。",
|
||||
},
|
||||
{
|
||||
"id": "sabc-news",
|
||||
"name": "SABC News",
|
||||
"provider": "SABC",
|
||||
"region": "Africa",
|
||||
"language": "en",
|
||||
"source_type": "hls",
|
||||
"embed_url": "",
|
||||
"stream_url": "https://sabconetanw.cdn.mangomolo.com/news/smil:news.stream.smil/playlist.m3u8",
|
||||
"homepage_url": "https://www.sabcnews.com/sabcnews/",
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": False,
|
||||
"sort_order": 100,
|
||||
"collector_source": None,
|
||||
"notes": "非洲英语新闻源,补充非洲区域新闻覆盖。",
|
||||
},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def _clean_text(value: Any) -> str:
|
||||
if value is None:
|
||||
return ""
|
||||
return str(value).strip()
|
||||
|
||||
|
||||
def _clean_url(value: Any) -> str:
|
||||
text = _clean_text(value)
|
||||
if not text:
|
||||
return ""
|
||||
|
||||
parsed = urlparse(text)
|
||||
if parsed.scheme and parsed.scheme not in {"http", "https"}:
|
||||
return ""
|
||||
if parsed.scheme and not parsed.netloc:
|
||||
return ""
|
||||
return text
|
||||
|
||||
|
||||
def _clean_bool(value: Any, *, default: bool) -> bool:
|
||||
if isinstance(value, bool):
|
||||
return value
|
||||
if value in (None, ""):
|
||||
return default
|
||||
if isinstance(value, str):
|
||||
lowered = value.strip().lower()
|
||||
if lowered in {"1", "true", "yes", "on"}:
|
||||
return True
|
||||
if lowered in {"0", "false", "no", "off"}:
|
||||
return False
|
||||
return bool(value)
|
||||
|
||||
|
||||
def _clean_int(value: Any, *, default: int) -> int:
|
||||
try:
|
||||
return int(value)
|
||||
except (TypeError, ValueError):
|
||||
return default
|
||||
|
||||
|
||||
def normalize_tv_source(source: dict[str, Any] | None, *, index: int = 0) -> dict[str, Any]:
|
||||
payload = dict(source or {})
|
||||
source_id = _clean_text(payload.get("id")) or f"tv-source-{index + 1}"
|
||||
source_type = _clean_text(payload.get("source_type")).lower()
|
||||
youtube_video_id = _clean_text(payload.get("youtube_video_id"))
|
||||
youtube_channel = _clean_text(payload.get("youtube_channel"))
|
||||
if source_type not in {"iframe", "hls", "video", "external", "youtube"}:
|
||||
if youtube_video_id or youtube_channel:
|
||||
source_type = "youtube"
|
||||
else:
|
||||
source_type = "iframe" if _clean_text(payload.get("embed_url")) else "external"
|
||||
|
||||
if source_type == "youtube" and not youtube_video_id and not youtube_channel:
|
||||
source_type = "iframe" if _clean_text(payload.get("embed_url")) else "external"
|
||||
|
||||
return {
|
||||
"id": source_id,
|
||||
"name": _clean_text(payload.get("name")) or f"新闻直播源 {index + 1}",
|
||||
"provider": _clean_text(payload.get("provider")) or "Unknown",
|
||||
"region": _clean_text(payload.get("region")) or "Global",
|
||||
"language": _clean_text(payload.get("language")) or "und",
|
||||
"source_type": source_type,
|
||||
"embed_url": _clean_url(payload.get("embed_url")),
|
||||
"stream_url": _clean_url(payload.get("stream_url")),
|
||||
"homepage_url": _clean_url(payload.get("homepage_url")),
|
||||
"poster_url": _clean_url(payload.get("poster_url")),
|
||||
"youtube_video_id": youtube_video_id,
|
||||
"youtube_channel": youtube_channel,
|
||||
"is_enabled": _clean_bool(payload.get("is_enabled"), default=True),
|
||||
"is_fallback": _clean_bool(payload.get("is_fallback"), default=False),
|
||||
"sort_order": _clean_int(payload.get("sort_order"), default=(index + 1) * 10),
|
||||
"collector_source": payload.get("collector_source"),
|
||||
"notes": _clean_text(payload.get("notes")),
|
||||
"updated_at": _clean_text(payload.get("updated_at")),
|
||||
}
|
||||
|
||||
|
||||
def normalize_tv_settings(payload: dict[str, Any] | None) -> dict[str, Any]:
|
||||
merged = {
|
||||
"default_source_id": DEFAULT_TV_SETTINGS["default_source_id"],
|
||||
"auto_fallback": DEFAULT_TV_SETTINGS["auto_fallback"],
|
||||
"sources": [],
|
||||
}
|
||||
|
||||
raw_sources = []
|
||||
if isinstance(payload, dict):
|
||||
merged["default_source_id"] = (
|
||||
_clean_text(payload.get("default_source_id")) or merged["default_source_id"]
|
||||
)
|
||||
merged["auto_fallback"] = _clean_bool(
|
||||
payload.get("auto_fallback"),
|
||||
default=DEFAULT_TV_SETTINGS["auto_fallback"],
|
||||
)
|
||||
if isinstance(payload.get("sources"), list):
|
||||
raw_sources = payload["sources"]
|
||||
|
||||
if not raw_sources:
|
||||
raw_sources = DEFAULT_TV_SETTINGS["sources"]
|
||||
|
||||
normalized_sources = [
|
||||
normalize_tv_source(source, index=index)
|
||||
for index, source in enumerate(raw_sources)
|
||||
]
|
||||
|
||||
if not any(source["id"] == DEFAULT_TV_SOURCE_ID for source in normalized_sources):
|
||||
normalized_sources.append(
|
||||
normalize_tv_source(DEFAULT_TV_SETTINGS["sources"][0], index=len(normalized_sources))
|
||||
)
|
||||
|
||||
default_source_exists = any(
|
||||
source["id"] == merged["default_source_id"] and source["is_enabled"]
|
||||
for source in normalized_sources
|
||||
)
|
||||
if not default_source_exists:
|
||||
fallback_source = next(
|
||||
(source for source in normalized_sources if source["is_fallback"] and source["is_enabled"]),
|
||||
None,
|
||||
)
|
||||
first_enabled_source = next(
|
||||
(source for source in normalized_sources if source["is_enabled"]),
|
||||
None,
|
||||
)
|
||||
merged["default_source_id"] = (
|
||||
fallback_source["id"]
|
||||
if fallback_source
|
||||
else first_enabled_source["id"]
|
||||
if first_enabled_source
|
||||
else DEFAULT_TV_SOURCE_ID
|
||||
)
|
||||
|
||||
merged["sources"] = sorted(
|
||||
normalized_sources,
|
||||
key=lambda item: (item["sort_order"], item["name"], item["id"]),
|
||||
)
|
||||
return merged
|
||||
|
||||
|
||||
async def get_tv_settings_payload(db: AsyncSession) -> dict[str, Any]:
|
||||
result = await db.execute(
|
||||
select(SystemSetting).where(SystemSetting.category == TV_SETTINGS_CATEGORY)
|
||||
)
|
||||
record = result.scalar_one_or_none()
|
||||
payload = record.payload if record else None
|
||||
return normalize_tv_settings(payload)
|
||||
|
||||
|
||||
def _build_collected_tv_source(record: CollectedData, index: int) -> dict[str, Any]:
|
||||
metadata = dict(record.extra_data or {})
|
||||
return normalize_tv_source(
|
||||
{
|
||||
"id": metadata.get("id") or record.source_id or record.entity_key,
|
||||
"name": record.name or record.title or metadata.get("name") or f"采集直播源 {index + 1}",
|
||||
"provider": metadata.get("provider") or metadata.get("publisher") or "Collector",
|
||||
"region": metadata.get("region") or metadata.get("country") or "Global",
|
||||
"language": metadata.get("language") or "und",
|
||||
"source_type": metadata.get("source_type") or "iframe",
|
||||
"embed_url": metadata.get("embed_url") or metadata.get("url") or "",
|
||||
"stream_url": metadata.get("stream_url") or "",
|
||||
"homepage_url": metadata.get("homepage_url") or metadata.get("source_url") or "",
|
||||
"poster_url": metadata.get("poster_url") or "",
|
||||
"youtube_video_id": metadata.get("youtube_video_id") or metadata.get("video_id") or "",
|
||||
"youtube_channel": metadata.get("youtube_channel") or metadata.get("channel_handle") or "",
|
||||
"is_enabled": metadata.get("is_enabled", True),
|
||||
"is_fallback": False,
|
||||
"sort_order": metadata.get("sort_order", 200 + index),
|
||||
"collector_source": record.source,
|
||||
"notes": record.description or metadata.get("notes") or "",
|
||||
"updated_at": to_iso8601_utc(record.collected_at or record.reference_date or datetime.now(UTC)),
|
||||
},
|
||||
index=index,
|
||||
)
|
||||
|
||||
|
||||
async def get_collected_tv_sources(db: AsyncSession) -> list[dict[str, Any]]:
|
||||
result = await db.execute(
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source == TV_LIVE_SOURCE_COLLECTOR)
|
||||
.where(CollectedData.data_type == TV_LIVE_SOURCE_DATA_TYPE)
|
||||
.where(CollectedData.is_current.is_(True))
|
||||
.where(CollectedData.is_valid == 1)
|
||||
.order_by(CollectedData.reference_date.desc().nullslast(), CollectedData.id.desc())
|
||||
)
|
||||
rows = result.scalars().all()
|
||||
return [_build_collected_tv_source(record, index) for index, record in enumerate(rows)]
|
||||
|
||||
|
||||
def build_public_tv_payload(
|
||||
settings_payload: dict[str, Any],
|
||||
collected_sources: list[dict[str, Any]],
|
||||
) -> dict[str, Any]:
|
||||
configured_sources = [
|
||||
source for source in settings_payload["sources"] if source["is_enabled"]
|
||||
]
|
||||
|
||||
merged_by_id = {source["id"]: source for source in configured_sources}
|
||||
for source in collected_sources:
|
||||
if source["id"] in merged_by_id or not source["is_enabled"]:
|
||||
continue
|
||||
merged_by_id[source["id"]] = source
|
||||
|
||||
available_sources = sorted(
|
||||
merged_by_id.values(),
|
||||
key=lambda item: (item["sort_order"], item["name"], item["id"]),
|
||||
)
|
||||
|
||||
default_source = next(
|
||||
(
|
||||
source
|
||||
for source in available_sources
|
||||
if source["id"] == settings_payload["default_source_id"]
|
||||
),
|
||||
None,
|
||||
)
|
||||
fallback_source = next(
|
||||
(source for source in available_sources if source["is_fallback"]),
|
||||
None,
|
||||
)
|
||||
|
||||
resolved_source = default_source or fallback_source or (available_sources[0] if available_sources else None)
|
||||
latest_updated_at = max(
|
||||
(source.get("updated_at") or "" for source in available_sources),
|
||||
default="",
|
||||
)
|
||||
|
||||
return {
|
||||
"default_source_id": settings_payload["default_source_id"],
|
||||
"auto_fallback": settings_payload["auto_fallback"],
|
||||
"selected_source": resolved_source,
|
||||
"fallback_source": fallback_source,
|
||||
"sources": available_sources,
|
||||
"source_count": len(available_sources),
|
||||
"latest_updated_at": latest_updated_at or to_iso8601_utc(datetime.now(UTC)),
|
||||
"generated_at": to_iso8601_utc(datetime.now(UTC)),
|
||||
}
|
||||
|
||||
|
||||
async def get_public_tv_payload(db: AsyncSession) -> dict[str, Any]:
|
||||
settings_payload = await get_tv_settings_payload(db)
|
||||
collected_sources = await get_collected_tv_sources(db)
|
||||
return build_public_tv_payload(settings_payload, collected_sources)
|
||||
|
||||
|
||||
def _extract_allowed_tv_hosts(sources: list[dict[str, Any]]) -> set[str]:
|
||||
hosts: set[str] = set()
|
||||
for source in sources:
|
||||
for field in ("stream_url", "embed_url", "homepage_url", "youtube_channel"):
|
||||
value = _clean_url(source.get(field))
|
||||
if not value:
|
||||
continue
|
||||
parsed = urlparse(value)
|
||||
if parsed.hostname:
|
||||
hosts.add(parsed.hostname.lower())
|
||||
return hosts
|
||||
|
||||
|
||||
def is_allowed_tv_proxy_url(url: str, sources: list[dict[str, Any]]) -> bool:
|
||||
cleaned = _clean_url(url)
|
||||
if not cleaned:
|
||||
return False
|
||||
|
||||
parsed = urlparse(cleaned)
|
||||
hostname = (parsed.hostname or "").lower()
|
||||
if not hostname:
|
||||
return False
|
||||
|
||||
allowed_hosts = _extract_allowed_tv_hosts(sources)
|
||||
if hostname in allowed_hosts:
|
||||
return True
|
||||
return any(hostname.endswith(f".{allowed_host}") for allowed_host in allowed_hosts)
|
||||
@@ -2,6 +2,7 @@ from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import shlex
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
@@ -59,6 +60,8 @@ def wait_for_recovery(action: str) -> tuple[bool, str]:
|
||||
recovery_mode = get_action_recovery_mode(action)
|
||||
if recovery_mode == "backend":
|
||||
return wait_for_http("http://localhost:8000/health"), "backend health recovery"
|
||||
if recovery_mode == "frontend":
|
||||
return wait_for_http("http://localhost:3000"), "frontend entrypoint recovery"
|
||||
if recovery_mode == "ai-provider":
|
||||
return wait_for_http("http://localhost:8010/health"), "ai provider health recovery"
|
||||
if recovery_mode == "database":
|
||||
@@ -108,8 +111,9 @@ def main() -> int:
|
||||
)
|
||||
append_task_log(args.task_id, "restart command started")
|
||||
|
||||
shell_command = shlex.join(command)
|
||||
completed = subprocess.run(
|
||||
command,
|
||||
["zsh", "-ic", shell_command],
|
||||
cwd=str(ROOT_DIR),
|
||||
env=env,
|
||||
capture_output=True,
|
||||
|
||||
+653
-2
@@ -10,7 +10,12 @@ from app.core.config import settings
|
||||
from app.core.security import create_access_token
|
||||
from app.db.session import get_db
|
||||
from app.models.user import User
|
||||
from app.schemas.ai import AIProviderStatusResponse, SituationalAnalysisResponse
|
||||
from app.schemas.ai import (
|
||||
AIProviderStatusResponse,
|
||||
PlaygroundSessionResponse,
|
||||
PlaygroundSessionState,
|
||||
SituationalAnalysisResponse,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
@@ -30,6 +35,7 @@ async def test_health_check():
|
||||
data = response.json()
|
||||
assert data["status"] == "healthy"
|
||||
assert "version" in data
|
||||
assert response.headers["x-request-id"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@@ -90,6 +96,15 @@ async def test_alerts_without_auth():
|
||||
assert response.status_code == 401
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_datasource_task_status_without_auth():
|
||||
"""Test datasource task-status endpoint requires authentication"""
|
||||
transport = ASGITransport(app=app)
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/api/v1/datasources/1/task-status")
|
||||
assert response.status_code == 401
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_alerts_endpoint_with_auth(auth_headers):
|
||||
"""Test alerts endpoint with authentication"""
|
||||
@@ -147,6 +162,345 @@ async def test_alerts_endpoint_with_auth(auth_headers):
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_system_log_sources_requires_super_admin(auth_headers):
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="testuser",
|
||||
email="test@example.com",
|
||||
password_hash="hashed",
|
||||
role="admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/api/v1/system/logs/sources", headers=auth_headers)
|
||||
assert response.status_code == 403
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_system_log_sources_with_super_admin(auth_headers):
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="root",
|
||||
email="root@example.com",
|
||||
password_hash="hashed",
|
||||
role="super_admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch(
|
||||
"app.api.v1.system_control.list_log_sources",
|
||||
return_value=[
|
||||
{
|
||||
"source_id": "backend",
|
||||
"name": "后端服务",
|
||||
"kind": "file",
|
||||
"location": "/tmp/planet_backend.log",
|
||||
"description": "FastAPI 后端、调度器和采集任务共享日志。",
|
||||
"category": "service",
|
||||
"status": "ok",
|
||||
}
|
||||
],
|
||||
):
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/api/v1/system/logs/sources", headers=auth_headers)
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["items"][0]["source_id"] == "backend"
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_system_log_snapshot_with_super_admin(auth_headers):
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="root",
|
||||
email="root@example.com",
|
||||
password_hash="hashed",
|
||||
role="super_admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch(
|
||||
"app.api.v1.system_control.read_log_snapshot",
|
||||
return_value={
|
||||
"source_id": "backend",
|
||||
"name": "后端服务",
|
||||
"kind": "file",
|
||||
"location": "/tmp/planet_backend.log",
|
||||
"description": "FastAPI 后端、调度器和采集任务共享日志。",
|
||||
"category": "service",
|
||||
"status": "ok",
|
||||
"level": "all",
|
||||
"selected_levels": [],
|
||||
"search_query": "",
|
||||
"available_levels": ["all", "error", "warning", "info", "debug"],
|
||||
"daily_markers": [],
|
||||
"line_limit": 50,
|
||||
"line_count": 2,
|
||||
"lines": ["line 1", "line 2"],
|
||||
},
|
||||
):
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/api/v1/system/logs/backend?limit=50", headers=auth_headers)
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["source_id"] == "backend"
|
||||
assert data["line_count"] == 2
|
||||
assert data["lines"] == ["line 1", "line 2"]
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_system_log_snapshot_supports_level_filter(auth_headers):
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="root",
|
||||
email="root@example.com",
|
||||
password_hash="hashed",
|
||||
role="super_admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch(
|
||||
"app.api.v1.system_control.read_log_snapshot",
|
||||
return_value={
|
||||
"source_id": "backend",
|
||||
"name": "后端服务",
|
||||
"kind": "file",
|
||||
"location": "/tmp/planet_backend.log",
|
||||
"description": "FastAPI 后端、调度器和采集任务共享日志。",
|
||||
"category": "service",
|
||||
"status": "ok",
|
||||
"level": "error",
|
||||
"selected_levels": ["error"],
|
||||
"search_query": "",
|
||||
"available_levels": ["all", "error", "warning", "info", "debug"],
|
||||
"daily_markers": [],
|
||||
"line_limit": 50,
|
||||
"line_count": 1,
|
||||
"lines": ["ERROR: failed"],
|
||||
},
|
||||
):
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/api/v1/system/logs/backend?limit=50&level=error", headers=auth_headers)
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["level"] == "error"
|
||||
assert data["lines"] == ["ERROR: failed"]
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_system_log_snapshot_supports_date_range_filter(auth_headers):
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="root",
|
||||
email="root@example.com",
|
||||
password_hash="hashed",
|
||||
role="super_admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch(
|
||||
"app.api.v1.system_control.read_log_snapshot",
|
||||
return_value={
|
||||
"source_id": "backend",
|
||||
"name": "后端服务",
|
||||
"kind": "file",
|
||||
"location": "/tmp/planet_backend.log",
|
||||
"description": "FastAPI 后端、调度器和采集任务共享日志。",
|
||||
"category": "service",
|
||||
"status": "ok",
|
||||
"level": "all",
|
||||
"selected_levels": [],
|
||||
"search_query": "",
|
||||
"available_levels": ["all", "error", "warning", "info", "debug"],
|
||||
"daily_markers": [],
|
||||
"line_limit": 50,
|
||||
"line_count": 1,
|
||||
"lines": ["2026-04-23 INFO: service started"],
|
||||
},
|
||||
) as mock_read_log_snapshot:
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get(
|
||||
"/api/v1/system/logs/backend?limit=50&start_date=2026-04-20&end_date=2026-04-23",
|
||||
headers=auth_headers,
|
||||
)
|
||||
assert response.status_code == 200
|
||||
mock_read_log_snapshot.assert_called_once_with(
|
||||
"backend",
|
||||
50,
|
||||
level="all",
|
||||
levels=None,
|
||||
start_date="2026-04-20",
|
||||
end_date="2026-04-23",
|
||||
search=None,
|
||||
)
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_system_log_snapshot_supports_levels_and_search_filter(auth_headers):
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="root",
|
||||
email="root@example.com",
|
||||
password_hash="hashed",
|
||||
role="super_admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch(
|
||||
"app.api.v1.system_control.read_log_snapshot",
|
||||
return_value={
|
||||
"source_id": "backend",
|
||||
"name": "后端服务",
|
||||
"kind": "file",
|
||||
"location": "/tmp/planet_backend.log",
|
||||
"description": "FastAPI 后端、调度器和采集任务共享日志。",
|
||||
"category": "service",
|
||||
"status": "ok",
|
||||
"level": "all",
|
||||
"selected_levels": ["error", "warning"],
|
||||
"search_query": "timeout",
|
||||
"available_levels": ["all", "error", "warning", "info", "debug"],
|
||||
"daily_markers": [],
|
||||
"line_limit": 50,
|
||||
"line_count": 1,
|
||||
"lines": ["2026-04-23 10:00:00 ERROR timeout"],
|
||||
},
|
||||
) as mock_read_log_snapshot:
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get(
|
||||
"/api/v1/system/logs/backend?limit=50&levels=error,warning&search=timeout",
|
||||
headers=auth_headers,
|
||||
)
|
||||
assert response.status_code == 200
|
||||
mock_read_log_snapshot.assert_called_once_with(
|
||||
"backend",
|
||||
50,
|
||||
level="all",
|
||||
levels="error,warning",
|
||||
start_date=None,
|
||||
end_date=None,
|
||||
search="timeout",
|
||||
)
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_system_log_snapshot_rejects_invalid_date_range(auth_headers):
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="root",
|
||||
email="root@example.com",
|
||||
password_hash="hashed",
|
||||
role="super_admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get(
|
||||
"/api/v1/system/logs/backend?start_date=2026-04-31",
|
||||
headers=auth_headers,
|
||||
)
|
||||
assert response.status_code == 400
|
||||
assert "start_date must be in YYYY-MM-DD format" in response.json()["detail"]
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ingest_earth_client_log_accepts_public_events():
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch("app.api.v1.system_control.append_buffer_log") as mock_append_buffer_log:
|
||||
with patch("app.api.v1.system_control.record_system_log", new_callable=AsyncMock) as mock_record_system_log:
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.post(
|
||||
"/api/v1/system/logs/earth-client",
|
||||
json={
|
||||
"level": "error",
|
||||
"message": "登陆点加载失败: 登陆点接口返回 HTTP 500",
|
||||
"category": "startup-load",
|
||||
"module": "layer-startup",
|
||||
},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["accepted"] is True
|
||||
assert data["source_id"] == "earth-client"
|
||||
mock_append_buffer_log.assert_called_once()
|
||||
mock_record_system_log.assert_awaited_once()
|
||||
persisted_kwargs = mock_record_system_log.await_args.kwargs
|
||||
assert persisted_kwargs["source"] == "earth-client"
|
||||
assert persisted_kwargs["event"] == "earth.client.runtime_log"
|
||||
assert persisted_kwargs["category"] == "startup-load"
|
||||
assert persisted_kwargs["level"] == "error"
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_request_id_header_is_echoed_when_provided():
|
||||
transport = ASGITransport(app=app)
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/health", headers={"X-Request-ID": "planet-test-request"})
|
||||
assert response.status_code == 200
|
||||
assert response.headers["x-request-id"] == "planet-test-request"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_invalid_token():
|
||||
"""Test that invalid token is rejected"""
|
||||
@@ -165,7 +519,8 @@ async def test_ai_provider_status_with_auth(auth_headers):
|
||||
class _FakeAIProviderClient:
|
||||
async def get_status(self, request_id=None):
|
||||
return AIProviderStatusResponse(
|
||||
provider="openai_compatible",
|
||||
provider="minimax",
|
||||
api="anthropic-messages",
|
||||
enabled=True,
|
||||
configured=True,
|
||||
model="test-model",
|
||||
@@ -193,6 +548,7 @@ async def test_ai_provider_status_with_auth(auth_headers):
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert "provider" in data
|
||||
assert "api" in data
|
||||
assert "configured" in data
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
@@ -207,6 +563,9 @@ async def test_ai_situational_analysis_returns_503_when_disabled(auth_headers):
|
||||
provider="openai_compatible",
|
||||
model="test-model",
|
||||
content="1) 态势摘要: 测试返回",
|
||||
content_blocks=[],
|
||||
text_blocks=["1) 态势摘要: 测试返回"],
|
||||
thinking_blocks=[],
|
||||
raw_response={"id": "mock-response"},
|
||||
)
|
||||
|
||||
@@ -241,5 +600,297 @@ async def test_ai_situational_analysis_returns_503_when_disabled(auth_headers):
|
||||
data = response.json()
|
||||
assert data["provider"] == "openai_compatible"
|
||||
assert data["content"]
|
||||
assert "content_blocks" in data
|
||||
assert "text_blocks" in data
|
||||
assert "thinking_blocks" in data
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_playground_session_with_auth(auth_headers):
|
||||
"""Test playground session restore endpoint."""
|
||||
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="testuser",
|
||||
email="test@example.com",
|
||||
password_hash="hashed",
|
||||
role="admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
async def override_get_db():
|
||||
yield AsyncMock()
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
get_db: override_get_db,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch(
|
||||
"app.api.v1.ai.get_playground_session",
|
||||
new=AsyncMock(
|
||||
return_value=PlaygroundSessionResponse(
|
||||
id="1",
|
||||
session_key="default",
|
||||
title="Playground 会话",
|
||||
state=PlaygroundSessionState(
|
||||
messages=[{"id": "msg-1", "role": "user", "content": "hello"}],
|
||||
title="测试标题",
|
||||
objective="测试目标",
|
||||
),
|
||||
created_at="2026-04-10T00:00:00+00:00",
|
||||
updated_at="2026-04-10T00:00:00+00:00",
|
||||
)
|
||||
),
|
||||
):
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/api/v1/ai/playground/session", headers=auth_headers)
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["session_key"] == "default"
|
||||
assert data["state"]["messages"][0]["content"] == "hello"
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_save_playground_session_with_auth(auth_headers):
|
||||
"""Test playground session save endpoint."""
|
||||
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="testuser",
|
||||
email="test@example.com",
|
||||
password_hash="hashed",
|
||||
role="admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
async def override_get_db():
|
||||
yield AsyncMock()
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
get_db: override_get_db,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch(
|
||||
"app.api.v1.ai.upsert_playground_session",
|
||||
new=AsyncMock(
|
||||
return_value=PlaygroundSessionResponse(
|
||||
id="1",
|
||||
session_key="default",
|
||||
title="测试标题",
|
||||
state=PlaygroundSessionState(
|
||||
messages=[{"id": "msg-1", "role": "user", "content": "hello"}],
|
||||
title="测试标题",
|
||||
objective="测试目标",
|
||||
),
|
||||
created_at="2026-04-10T00:00:00+00:00",
|
||||
updated_at="2026-04-10T00:00:00+00:00",
|
||||
)
|
||||
),
|
||||
):
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.put(
|
||||
"/api/v1/ai/playground/session",
|
||||
headers=auth_headers,
|
||||
json={
|
||||
"session_key": "default",
|
||||
"title": "测试标题",
|
||||
"state": {
|
||||
"messages": [{"id": "msg-1", "role": "user", "content": "hello"}],
|
||||
"selectedPresetKey": "bgp-brief",
|
||||
"title": "测试标题",
|
||||
"objective": "测试目标",
|
||||
"constraints": "",
|
||||
"inputValue": "",
|
||||
"analysis": None,
|
||||
"latestAnalysisMessageId": None,
|
||||
"analysisMeta": {},
|
||||
"helpExpanded": True,
|
||||
},
|
||||
},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["title"] == "测试标题"
|
||||
assert data["state"]["objective"] == "测试目标"
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ai_bgp_brief_endpoint_persists_fact_snapshot(auth_headers):
|
||||
class _FakeAIProviderClient:
|
||||
async def analyze(self, _payload, request_id=None):
|
||||
return SituationalAnalysisResponse(
|
||||
provider="minimax",
|
||||
model="MiniMax-M2.5",
|
||||
content="# BGP AI 简报\n\n事实摘要:测试",
|
||||
content_blocks=[],
|
||||
text_blocks=["# BGP AI 简报\n\n事实摘要:测试"],
|
||||
thinking_blocks=[],
|
||||
raw_response={"id": "mock-bgp-brief"},
|
||||
)
|
||||
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="testuser",
|
||||
email="test@example.com",
|
||||
password_hash="hashed",
|
||||
role="admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
async def override_get_db():
|
||||
yield AsyncMock()
|
||||
|
||||
async def _fake_build_bgp_brief_request(_db, **_kwargs):
|
||||
request_payload = __import__("app.schemas.ai", fromlist=["SituationalAnalysisRequest"]).SituationalAnalysisRequest(
|
||||
title="BGP 态势 AI 简报",
|
||||
objective="生成值班简报",
|
||||
observations=["事实A", "事实B"],
|
||||
constraints=["不要编造"],
|
||||
context={"incident_total": 2, "active_collectors": 3},
|
||||
)
|
||||
return request_payload, ["事实A", "事实B"], {"incident_total": 2, "active_collectors": 3}
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
__import__("app.services.ai_client", fromlist=["get_ai_provider_client"]).get_ai_provider_client: lambda: _FakeAIProviderClient(),
|
||||
get_db: override_get_db,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch("app.api.v1.ai.build_bgp_brief_request", side_effect=_fake_build_bgp_brief_request):
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.post("/api/v1/ai/bgp/brief", headers=auth_headers, json={})
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["facts"] == ["事实A", "事实B"]
|
||||
assert data["context"]["incident_total"] == 2
|
||||
assert data["content_markdown"]
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ai_alert_brief_endpoint_with_auth(auth_headers):
|
||||
class _FakeAIProviderClient:
|
||||
async def analyze(self, _payload, request_id=None):
|
||||
return SituationalAnalysisResponse(
|
||||
provider="minimax",
|
||||
model="MiniMax-M2.7",
|
||||
content="事实摘要:告警测试。风险研判:告警测试。建议动作:告警测试。",
|
||||
content_blocks=[],
|
||||
text_blocks=["事实摘要:告警测试。风险研判:告警测试。建议动作:告警测试。"],
|
||||
thinking_blocks=[],
|
||||
raw_response={"id": "mock-alert-brief"},
|
||||
)
|
||||
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="testuser",
|
||||
email="test@example.com",
|
||||
password_hash="hashed",
|
||||
role="admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
async def override_get_db():
|
||||
yield AsyncMock()
|
||||
|
||||
async def _fake_build_alert_brief_request(_db, **_kwargs):
|
||||
request_payload = __import__("app.schemas.ai", fromlist=["SituationalAnalysisRequest"]).SituationalAnalysisRequest(
|
||||
title="告警态势 AI 简报",
|
||||
objective="输出告警简报",
|
||||
observations=["告警事实A", "告警事实B"],
|
||||
constraints=["不要编造"],
|
||||
context={"active_alerts": 3, "top_datasources": {"bgp": 2}},
|
||||
)
|
||||
return request_payload, ["告警事实A", "告警事实B"], {"active_alerts": 3, "top_datasources": {"bgp": 2}}
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
__import__("app.services.ai_client", fromlist=["get_ai_provider_client"]).get_ai_provider_client: lambda: _FakeAIProviderClient(),
|
||||
get_db: override_get_db,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch("app.api.v1.ai.build_alert_brief_request", side_effect=_fake_build_alert_brief_request):
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.post("/api/v1/ai/alerts/brief", headers=auth_headers, json={})
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["title"] == "告警态势 AI 简报"
|
||||
assert data["facts"] == ["告警事实A", "告警事实B"]
|
||||
assert data["context"]["active_alerts"] == 3
|
||||
assert data["content"]
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ai_situational_alert_brief_endpoint_with_auth(auth_headers):
|
||||
class _FakeAIProviderClient:
|
||||
async def analyze(self, _payload, request_id=None):
|
||||
return SituationalAnalysisResponse(
|
||||
provider="minimax",
|
||||
model="MiniMax-M2.7",
|
||||
content="事实摘要:态势测试。风险研判:态势测试。建议动作:态势测试。",
|
||||
content_blocks=[],
|
||||
text_blocks=["事实摘要:态势测试。风险研判:态势测试。建议动作:态势测试。"],
|
||||
thinking_blocks=[],
|
||||
raw_response={"id": "mock-situational-brief"},
|
||||
)
|
||||
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="testuser",
|
||||
email="test@example.com",
|
||||
password_hash="hashed",
|
||||
role="admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
async def override_get_db():
|
||||
yield AsyncMock()
|
||||
|
||||
async def _fake_build_situational_alert_brief_request(_db):
|
||||
request_payload = __import__("app.schemas.ai", fromlist=["SituationalAnalysisRequest"]).SituationalAnalysisRequest(
|
||||
title="态势告警 AI 简报",
|
||||
objective="输出态势告警简报",
|
||||
observations=["态势事实A", "态势事实B"],
|
||||
constraints=["不要编造"],
|
||||
context={"active_system_alerts": 2, "active_bgp_incidents": 1},
|
||||
)
|
||||
return request_payload, ["态势事实A", "态势事实B"], {"active_system_alerts": 2, "active_bgp_incidents": 1}
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
__import__("app.services.ai_client", fromlist=["get_ai_provider_client"]).get_ai_provider_client: lambda: _FakeAIProviderClient(),
|
||||
get_db: override_get_db,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch("app.api.v1.ai.build_situational_alert_brief_request", side_effect=_fake_build_situational_alert_brief_request):
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.post("/api/v1/ai/situational-alerts/brief", headers=auth_headers, json={})
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["title"] == "态势告警 AI 简报"
|
||||
assert data["facts"] == ["态势事实A", "态势事实B"]
|
||||
assert data["context"]["active_system_alerts"] == 2
|
||||
assert data["content"]
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
@@ -0,0 +1,199 @@
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
|
||||
from app.api.v1.datasource_config import get_ai_provider_client
|
||||
from app.core.security import get_current_user
|
||||
from app.core.target_schema_registry import get_target_schema, list_target_schemas
|
||||
from app.main import app
|
||||
from app.models.user import User
|
||||
from app.services.datasource_mapping import execute_mapping, persist_mapped_records, redact_for_llm
|
||||
|
||||
|
||||
SAMPLE_AIS = {
|
||||
"data": [
|
||||
{
|
||||
"mmsi": "257123000",
|
||||
"latitude": "59.91",
|
||||
"longitude": "10.75",
|
||||
"speedOverGround": "12.4",
|
||||
"timestamp": "2026-04-28T00:00:00Z",
|
||||
"api_token": "secret-value",
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
def test_registry_exposes_v1_target_schemas():
|
||||
keys = {schema["key"] for schema in list_target_schemas()}
|
||||
|
||||
assert {"vessel_ais", "geo_points", "generic_records"}.issubset(keys)
|
||||
assert get_target_schema("vessel_ais").destination == "vessel_position"
|
||||
|
||||
|
||||
def test_mapping_engine_maps_and_validates_vessel_ais():
|
||||
mapping = {
|
||||
"source": {"items_path": "$.data[*]"},
|
||||
"fields": {
|
||||
"mmsi": {"path": "$.mmsi", "type": "integer"},
|
||||
"lat": {"path": "$.latitude", "type": "float"},
|
||||
"lon": {"path": "$.longitude", "type": "float"},
|
||||
"sog": {"path": "$.speedOverGround", "type": "float"},
|
||||
"received_at": {"path": "$.timestamp", "type": "datetime"},
|
||||
},
|
||||
}
|
||||
|
||||
result = execute_mapping(SAMPLE_AIS, mapping, "vessel_ais")
|
||||
|
||||
assert result["mapped_count"] == 1
|
||||
assert result["failed_count"] == 0
|
||||
assert result["records"][0]["mmsi"] == 257123000
|
||||
assert result["records"][0]["lat"] == 59.91
|
||||
|
||||
|
||||
def test_mapping_engine_reports_schema_errors():
|
||||
mapping = {
|
||||
"source": {"items_path": "$.data[*]"},
|
||||
"fields": {
|
||||
"mmsi": {"path": "$.mmsi", "type": "integer"},
|
||||
"lat": {"path": "$.missing_lat", "type": "float"},
|
||||
"lon": {"path": "$.longitude", "type": "float"},
|
||||
},
|
||||
}
|
||||
|
||||
result = execute_mapping(SAMPLE_AIS, mapping, "vessel_ais")
|
||||
|
||||
assert result["mapped_count"] == 0
|
||||
assert result["failed_count"] == 1
|
||||
assert any("lat" in error for error in result["errors"][0]["errors"])
|
||||
|
||||
|
||||
def test_redact_for_llm_masks_secret_like_fields():
|
||||
redacted = redact_for_llm(SAMPLE_AIS)
|
||||
|
||||
assert redacted["data"][0]["api_token"] == "[REDACTED]"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_persist_mapped_records_writes_generic_records():
|
||||
class FakeDB:
|
||||
def __init__(self):
|
||||
self.added = []
|
||||
self.committed = False
|
||||
|
||||
def add(self, value):
|
||||
self.added.append(value)
|
||||
|
||||
async def commit(self):
|
||||
self.committed = True
|
||||
|
||||
db = FakeDB()
|
||||
|
||||
count = await persist_mapped_records(
|
||||
db,
|
||||
datasource_name="custom_weather",
|
||||
datasource_config_id=42,
|
||||
target_schema="generic_records",
|
||||
records=[{"source_id": "row-1", "data": {"temp": 25}}],
|
||||
mapping_version=3,
|
||||
)
|
||||
|
||||
assert count == 1
|
||||
assert db.committed is True
|
||||
assert db.added[0].source == "custom_weather"
|
||||
assert db.added[0].data_type == "generic_records"
|
||||
assert db.added[0].extra_data["mapping_version"] == 3
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_mapping_preview_api_uses_deterministic_engine():
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="testuser",
|
||||
email="test@example.com",
|
||||
password_hash="hashed",
|
||||
role="admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
app.dependency_overrides = {get_current_user: override_get_current_user}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.post(
|
||||
"/api/v1/datasources/mappings/preview",
|
||||
json={
|
||||
"sample_payload": SAMPLE_AIS,
|
||||
"target_schema": "vessel_ais",
|
||||
"mapping_json": {
|
||||
"source": {"items_path": "$.data[*]"},
|
||||
"fields": {
|
||||
"mmsi": {"path": "$.mmsi", "type": "integer"},
|
||||
"lat": {"path": "$.latitude", "type": "float"},
|
||||
"lon": {"path": "$.longitude", "type": "float"},
|
||||
},
|
||||
},
|
||||
},
|
||||
)
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
assert response.status_code == 200
|
||||
payload = response.json()
|
||||
assert payload["success"] is True
|
||||
assert payload["preview"]["records"][0]["mmsi"] == 257123000
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_mapping_propose_api_redacts_sample_before_ai():
|
||||
seen_context = {}
|
||||
|
||||
class FakeAIClient:
|
||||
async def analyze(self, request, request_id=None):
|
||||
seen_context.update(request.context)
|
||||
return SimpleNamespace(
|
||||
content=(
|
||||
'{"source":{"items_path":"$.data[*]"},"fields":{'
|
||||
'"mmsi":{"path":"$.mmsi","type":"integer"},'
|
||||
'"lat":{"path":"$.latitude","type":"float"},'
|
||||
'"lon":{"path":"$.longitude","type":"float"}}}'
|
||||
)
|
||||
)
|
||||
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="testuser",
|
||||
email="test@example.com",
|
||||
password_hash="hashed",
|
||||
role="admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
def override_ai_client():
|
||||
return FakeAIClient()
|
||||
|
||||
app.dependency_overrides = {
|
||||
get_current_user: override_get_current_user,
|
||||
get_ai_provider_client: override_ai_client,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.post(
|
||||
"/api/v1/datasources/mappings/propose",
|
||||
json={
|
||||
"sample_payload": SAMPLE_AIS,
|
||||
"target_schema": "vessel_ais",
|
||||
"use_ai": True,
|
||||
},
|
||||
)
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
assert response.status_code == 200
|
||||
payload = response.json()
|
||||
assert payload["mapping_json"]["meta"]["generated_by"] == "ai_provider"
|
||||
assert seen_context["sample_payload"]["data"][0]["api_token"] == "[REDACTED]"
|
||||
@@ -0,0 +1,49 @@
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from app.services.earth_news import ParsedNewsItem, _serialize_item
|
||||
|
||||
|
||||
def test_serialize_item_includes_region_anchor_for_cruise():
|
||||
item = ParsedNewsItem(
|
||||
id="google-apac:test",
|
||||
title="Example APAC story",
|
||||
summary="Example summary",
|
||||
url="https://example.com/story",
|
||||
source="Example Source",
|
||||
feed_name="Global Monitor / APAC",
|
||||
feed_region="asia-pacific",
|
||||
homepage_url="https://example.com",
|
||||
published_at=datetime(2026, 4, 23, 2, 30, tzinfo=UTC),
|
||||
)
|
||||
|
||||
payload = _serialize_item(item, active_region="asia-pacific")
|
||||
|
||||
assert payload["latitude"] == 1.3521
|
||||
assert payload["longitude"] == 103.8198
|
||||
assert payload["location_label"] == "亚太"
|
||||
assert payload["location_inferred"] is True
|
||||
assert payload["is_focus_match"] is True
|
||||
assert payload["published_at"] == "2026-04-23T02:30:00Z"
|
||||
|
||||
|
||||
def test_serialize_item_falls_back_to_global_anchor():
|
||||
item = ParsedNewsItem(
|
||||
id="custom:test",
|
||||
title="Fallback story",
|
||||
summary="Fallback summary",
|
||||
url="https://example.com/fallback",
|
||||
source="Fallback Source",
|
||||
feed_name="Fallback Feed",
|
||||
feed_region="unknown-region",
|
||||
homepage_url="https://example.com",
|
||||
published_at=None,
|
||||
)
|
||||
|
||||
payload = _serialize_item(item, active_region="americas")
|
||||
|
||||
assert payload["latitude"] == 20.0
|
||||
assert payload["longitude"] == 0.0
|
||||
assert payload["location_label"] == "全球"
|
||||
assert payload["location_inferred"] is True
|
||||
assert payload["is_focus_match"] is False
|
||||
assert payload["published_at"] is None
|
||||
@@ -0,0 +1,78 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
from io import StringIO
|
||||
|
||||
from app.core.logging import PlanetContextFilter, PlanetFormatter, get_logger
|
||||
from app.core.request_context import set_request_id
|
||||
|
||||
|
||||
def _capture_output(callback):
|
||||
stream = StringIO()
|
||||
handler = logging.StreamHandler(stream)
|
||||
handler.setFormatter(PlanetFormatter(datefmt="%Y-%m-%d %H:%M:%S"))
|
||||
handler.addFilter(PlanetContextFilter())
|
||||
|
||||
adapter = get_logger("tests.logging")
|
||||
target_logger = adapter.logger
|
||||
original_handlers = list(target_logger.handlers)
|
||||
original_level = target_logger.level
|
||||
original_propagate = target_logger.propagate
|
||||
|
||||
target_logger.handlers = [handler]
|
||||
target_logger.setLevel(logging.INFO)
|
||||
target_logger.propagate = False
|
||||
|
||||
try:
|
||||
callback(adapter)
|
||||
finally:
|
||||
handler.flush()
|
||||
target_logger.handlers = original_handlers
|
||||
target_logger.setLevel(original_level)
|
||||
target_logger.propagate = original_propagate
|
||||
|
||||
return stream.getvalue()
|
||||
|
||||
|
||||
def test_structured_logger_injects_request_id_and_event():
|
||||
set_request_id("req-test-123")
|
||||
try:
|
||||
output = _capture_output(
|
||||
lambda logger: logger.info_event(
|
||||
"collector started",
|
||||
event="collector.run.started",
|
||||
context={"collector_name": "bgp_news"},
|
||||
)
|
||||
)
|
||||
finally:
|
||||
set_request_id(None)
|
||||
|
||||
assert "request_id=req-test-123" in output
|
||||
assert "event=collector.run.started" in output
|
||||
assert "service=backend" in output
|
||||
assert '"collector_name": "bgp_news"' in output
|
||||
|
||||
|
||||
def test_structured_logger_redacts_sensitive_text_and_context():
|
||||
set_request_id("req-test-redact")
|
||||
try:
|
||||
output = _capture_output(
|
||||
lambda logger: logger.error_event(
|
||||
"Authorization: Bearer super-secret-token",
|
||||
event="auth.token.failed",
|
||||
context={
|
||||
"token": "plain-secret",
|
||||
"nested": {"password": "hunter2"},
|
||||
"safe": "visible",
|
||||
},
|
||||
)
|
||||
)
|
||||
finally:
|
||||
set_request_id(None)
|
||||
|
||||
assert "super-secret-token" not in output
|
||||
assert "plain-secret" not in output
|
||||
assert "hunter2" not in output
|
||||
assert "[REDACTED]" in output
|
||||
assert '"safe": "visible"' in output
|
||||
@@ -0,0 +1,217 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from app.services import system_logs
|
||||
|
||||
|
||||
class FakeRedis:
|
||||
def __init__(self) -> None:
|
||||
self.store: dict[str, list[str]] = {}
|
||||
|
||||
def rpush(self, key: str, value: str) -> None:
|
||||
self.store.setdefault(key, []).append(value)
|
||||
|
||||
def ltrim(self, key: str, start: int, end: int) -> None:
|
||||
items = self.store.get(key, [])
|
||||
normalized_end = None if end == -1 else end + 1
|
||||
self.store[key] = items[start:normalized_end]
|
||||
|
||||
def expire(self, key: str, seconds: int) -> None:
|
||||
return None
|
||||
|
||||
def lrange(self, key: str, start: int, end: int) -> list[str]:
|
||||
items = self.store.get(key, [])
|
||||
normalized_end = None if end == -1 else end + 1
|
||||
return items[start:normalized_end]
|
||||
|
||||
def llen(self, key: str) -> int:
|
||||
return len(self.store.get(key, []))
|
||||
|
||||
|
||||
def test_read_log_snapshot_uses_structured_buffer_timestamp_level_and_search(monkeypatch):
|
||||
fake_redis = FakeRedis()
|
||||
monkeypatch.setattr(system_logs, "redis_client", fake_redis)
|
||||
monkeypatch.setattr(
|
||||
system_logs,
|
||||
"LOG_SOURCES",
|
||||
{
|
||||
"earth-client": system_logs.LogSource(
|
||||
source_id="earth-client",
|
||||
name="Earth 浏览器端",
|
||||
kind="buffer",
|
||||
location="redis://planet:system_logs:earth-client",
|
||||
description="Earth 浏览器端上报日志",
|
||||
category="client",
|
||||
buffer_key=system_logs.get_buffer_log_key("earth-client"),
|
||||
)
|
||||
},
|
||||
)
|
||||
|
||||
fake_redis.rpush(
|
||||
system_logs.get_buffer_log_key("earth-client"),
|
||||
json.dumps(
|
||||
{
|
||||
"timestamp": "2026-04-22T10:15:30Z",
|
||||
"level": "warning",
|
||||
"message": "news feed degraded",
|
||||
"context": {"module": "news", "detail": "timeout"},
|
||||
},
|
||||
ensure_ascii=False,
|
||||
),
|
||||
)
|
||||
fake_redis.rpush(
|
||||
system_logs.get_buffer_log_key("earth-client"),
|
||||
json.dumps(
|
||||
{
|
||||
"timestamp": "2026-04-23T06:01:00Z",
|
||||
"level": "error",
|
||||
"message": "landing points failed",
|
||||
"context": {"module": "layer-startup", "detail": "http 500"},
|
||||
},
|
||||
ensure_ascii=False,
|
||||
),
|
||||
)
|
||||
|
||||
snapshot = system_logs.read_log_snapshot(
|
||||
"earth-client",
|
||||
50,
|
||||
levels="error,warning",
|
||||
start_date="2026-04-23",
|
||||
end_date="2026-04-23",
|
||||
search="landing",
|
||||
)
|
||||
|
||||
assert snapshot is not None
|
||||
assert snapshot["selected_levels"] == ["error", "warning"]
|
||||
assert snapshot["search_query"] == "landing"
|
||||
assert snapshot["line_count"] == 1
|
||||
assert snapshot["lines"][0].startswith("2026-04-23 06:01:00 ERROR landing points failed")
|
||||
assert snapshot["daily_markers"] == [
|
||||
{"date_token": "2026-04-23", "total": 1, "dominant_level": "error"}
|
||||
]
|
||||
|
||||
|
||||
def test_read_log_snapshot_parses_file_timestamp_and_builds_markers(tmp_path: Path, monkeypatch):
|
||||
log_path = tmp_path / "backend.log"
|
||||
log_path.write_text(
|
||||
"\n".join(
|
||||
[
|
||||
"2026-04-22 08:00:00 INFO service booted",
|
||||
"2026-04-23 09:15:00 WARNING disk pressure detected",
|
||||
"2026-04-23 09:16:00 ERROR sync failed",
|
||||
"2026-04-24 10:00:00 DEBUG collector trace",
|
||||
]
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
monkeypatch.setattr(
|
||||
system_logs,
|
||||
"LOG_SOURCES",
|
||||
{
|
||||
"backend": system_logs.LogSource(
|
||||
source_id="backend",
|
||||
name="后端服务",
|
||||
kind="file",
|
||||
location=str(log_path),
|
||||
description="测试文件日志",
|
||||
category="service",
|
||||
)
|
||||
},
|
||||
)
|
||||
|
||||
snapshot = system_logs.read_log_snapshot(
|
||||
"backend",
|
||||
50,
|
||||
levels="warning,error",
|
||||
search="failed",
|
||||
)
|
||||
|
||||
assert snapshot is not None
|
||||
assert snapshot["line_count"] == 1
|
||||
assert snapshot["lines"] == ["2026-04-23 09:16:00 ERROR sync failed"]
|
||||
assert snapshot["daily_markers"] == [
|
||||
{"date_token": "2026-04-23", "total": 1, "dominant_level": "error"}
|
||||
]
|
||||
assert snapshot["status"] == "ok"
|
||||
|
||||
|
||||
def test_append_buffer_log_persists_normalized_level(monkeypatch):
|
||||
fake_redis = FakeRedis()
|
||||
monkeypatch.setattr(system_logs, "redis_client", fake_redis)
|
||||
|
||||
system_logs.append_buffer_log(
|
||||
"earth-client",
|
||||
level="warn",
|
||||
message="feed delayed",
|
||||
context={"module": "news"},
|
||||
)
|
||||
|
||||
stored_items = fake_redis.lrange(system_logs.get_buffer_log_key("earth-client"), 0, -1)
|
||||
payload = json.loads(stored_items[0])
|
||||
assert payload["level"] == "warning"
|
||||
assert payload["message"] == "feed delayed"
|
||||
|
||||
|
||||
def test_infer_log_level_prefers_leading_prefix_over_query_string():
|
||||
line = 'INFO: 127.0.0.1 - "GET /api/v1/system/logs/backend?limit=200&level=error&levels=error HTTP/1.1" 200 OK'
|
||||
|
||||
entry = system_logs.parse_text_log_entry(line)
|
||||
|
||||
assert entry.level == "info"
|
||||
|
||||
|
||||
def test_parse_text_log_entry_does_not_promote_exception_context_to_error():
|
||||
line = "websockets.exceptions.ConnectionClosedError: sent 1011 (internal error) keepalive ping timeout"
|
||||
|
||||
entry = system_logs.parse_text_log_entry(line)
|
||||
|
||||
assert entry.level is None
|
||||
|
||||
|
||||
def test_parse_text_log_entry_still_detects_explicit_error_prefix():
|
||||
line = "ERROR: [Errno 98] Address already in use"
|
||||
|
||||
entry = system_logs.parse_text_log_entry(line)
|
||||
|
||||
assert entry.level == "error"
|
||||
|
||||
|
||||
def test_read_log_snapshot_strips_nul_bytes_from_file_lines(tmp_path: Path, monkeypatch):
|
||||
log_path = tmp_path / "backend.log"
|
||||
log_path.write_bytes(
|
||||
(
|
||||
b"INFO: service booted\n"
|
||||
b"ERROR: bind failed\n"
|
||||
+ b"\x00" * 32
|
||||
+ b"2026-04-23 23:41:32 INFO service=backend message=request served\n"
|
||||
)
|
||||
)
|
||||
|
||||
monkeypatch.setattr(
|
||||
system_logs,
|
||||
"LOG_SOURCES",
|
||||
{
|
||||
"backend": system_logs.LogSource(
|
||||
source_id="backend",
|
||||
name="后端服务",
|
||||
kind="file",
|
||||
location=str(log_path),
|
||||
description="测试文件日志",
|
||||
category="service",
|
||||
)
|
||||
},
|
||||
)
|
||||
|
||||
snapshot = system_logs.read_log_snapshot("backend", 50)
|
||||
|
||||
assert snapshot is not None
|
||||
assert snapshot["line_count"] == 3
|
||||
assert snapshot["lines"] == [
|
||||
"INFO: service booted",
|
||||
"ERROR: bind failed",
|
||||
"2026-04-23 23:41:32 INFO service=backend message=request served",
|
||||
]
|
||||
@@ -0,0 +1,149 @@
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import pytest
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
|
||||
from app.api.v1.visualization import convert_vessels_to_geojson
|
||||
from app.db.session import get_db
|
||||
from app.main import app
|
||||
from app.models.vessel import VesselPosition, VesselStatic
|
||||
from app.services import barentswatch
|
||||
from app.services.collectors.vessel_ais import VesselAISCollector
|
||||
|
||||
|
||||
def test_vessel_collector_transforms_barentswatch_like_records():
|
||||
collector = VesselAISCollector()
|
||||
records = collector.transform(
|
||||
[
|
||||
{
|
||||
"mmsi": "257123000",
|
||||
"lat": "59.91",
|
||||
"lon": "10.73",
|
||||
"sog": 12.4,
|
||||
"cog": 214,
|
||||
"nav_status": 0,
|
||||
"shipType": 70,
|
||||
"name": "OSLO TRADER",
|
||||
},
|
||||
{"mmsi": "bad", "lat": 120, "lon": 10},
|
||||
]
|
||||
)
|
||||
|
||||
assert len(records) == 1
|
||||
assert records[0]["mmsi"] == 257123000
|
||||
assert records[0]["vessel_type_name"] == "Cargo"
|
||||
assert records[0]["lat"] == pytest.approx(59.91)
|
||||
|
||||
|
||||
def test_barentswatch_reads_credentials_from_zshrc(tmp_path):
|
||||
zshrc = tmp_path / ".zshrc"
|
||||
zshrc.write_text(
|
||||
"\n".join(
|
||||
[
|
||||
"export BARENTSWATCH_CLIENT_ID='client-from-zshrc'",
|
||||
'export BARENTSWATCH_CLIENT_SECRET="secret-from-zshrc" # local dev credential',
|
||||
]
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
values = barentswatch._read_zshrc_env(zshrc)
|
||||
|
||||
assert values["BARENTSWATCH_CLIENT_ID"] == "client-from-zshrc"
|
||||
assert values["BARENTSWATCH_CLIENT_SECRET"] == "secret-from-zshrc"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_barentswatch_resolves_config_from_zshrc(tmp_path, monkeypatch):
|
||||
zshrc = tmp_path / ".zshrc"
|
||||
zshrc.write_text(
|
||||
"\n".join(
|
||||
[
|
||||
"export BARENTSWATCH_CLIENT_ID=client-from-zshrc",
|
||||
"export BARENTSWATCH_CLIENT_SECRET=secret-from-zshrc",
|
||||
]
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
monkeypatch.delenv("BARENTSWATCH_CLIENT_ID", raising=False)
|
||||
monkeypatch.delenv("BARENTSWATCH_CLIENT_SECRET", raising=False)
|
||||
monkeypatch.delenv("BARRENTSWATCH_CLIENT_ID", raising=False)
|
||||
monkeypatch.delenv("BARRENTSWATCH_CLIENT_SECRET", raising=False)
|
||||
monkeypatch.setattr(barentswatch.Path, "home", lambda: tmp_path)
|
||||
|
||||
config = await barentswatch.resolve_barentswatch_config(None)
|
||||
|
||||
assert config.client_id == "client-from-zshrc"
|
||||
assert config.client_secret == "secret-from-zshrc"
|
||||
assert config.credential_source == "~/.zshrc"
|
||||
|
||||
|
||||
def test_convert_vessels_to_geojson():
|
||||
position = VesselPosition(
|
||||
mmsi=257123000,
|
||||
lat=59.91,
|
||||
lon=10.73,
|
||||
sog=12.4,
|
||||
cog=214,
|
||||
heading=215,
|
||||
nav_status=0,
|
||||
received_at=datetime(2026, 4, 28, 1, 0, tzinfo=timezone.utc),
|
||||
)
|
||||
static = VesselStatic(
|
||||
mmsi=257123000,
|
||||
name="OSLO TRADER",
|
||||
vessel_type=70,
|
||||
vessel_type_name="Cargo",
|
||||
flag="NO",
|
||||
length=185,
|
||||
)
|
||||
|
||||
payload = convert_vessels_to_geojson([(position, static)])
|
||||
|
||||
assert payload["type"] == "FeatureCollection"
|
||||
assert payload["features"][0]["geometry"]["coordinates"] == [10.73, 59.91]
|
||||
assert payload["features"][0]["properties"]["mmsi"] == 257123000
|
||||
assert payload["features"][0]["properties"]["vessel_type_name"] == "Cargo"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_vessels_geojson_endpoint_filters_type_and_bbox():
|
||||
now = datetime(2026, 4, 28, 1, 0, tzinfo=timezone.utc)
|
||||
rows = [
|
||||
(
|
||||
VesselPosition(mmsi=1, lat=59.9, lon=10.7, received_at=now),
|
||||
VesselStatic(mmsi=1, name="Cargo Ship", vessel_type=70, vessel_type_name="Cargo"),
|
||||
),
|
||||
(
|
||||
VesselPosition(mmsi=2, lat=60.3, lon=5.3, received_at=now - timedelta(minutes=1)),
|
||||
VesselStatic(mmsi=2, name="Passenger Ship", vessel_type=60, vessel_type_name="Passenger"),
|
||||
),
|
||||
]
|
||||
|
||||
class _Result:
|
||||
def all(self):
|
||||
return rows
|
||||
|
||||
class _FakeSession:
|
||||
async def execute(self, _query):
|
||||
return _Result()
|
||||
|
||||
async def override_get_db():
|
||||
yield _FakeSession()
|
||||
|
||||
app.dependency_overrides[get_db] = override_get_db
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get(
|
||||
"/api/v1/visualization/geo/vessels",
|
||||
params={"bbox": "0,50,20,70", "type": "cargo"},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["count"] == 1
|
||||
assert data["features"][0]["properties"]["name"] == "Cargo Ship"
|
||||
assert data["stats"]["by_type"]["Cargo"] == 1
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
@@ -0,0 +1,347 @@
|
||||
from datetime import datetime, timezone
|
||||
|
||||
import pytest
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
|
||||
from app.api.v1.visualization import convert_compute_centers_to_geojson
|
||||
from app.db.session import get_db
|
||||
from app.main import app
|
||||
from app.models.collected_data import CollectedData
|
||||
|
||||
|
||||
def _build_record(
|
||||
*,
|
||||
record_id: int,
|
||||
source: str,
|
||||
data_type: str,
|
||||
name: str,
|
||||
country: str,
|
||||
city: str,
|
||||
latitude: float,
|
||||
longitude: float,
|
||||
metadata: dict,
|
||||
):
|
||||
return CollectedData(
|
||||
id=record_id,
|
||||
source=source,
|
||||
data_type=data_type,
|
||||
source_id=f"{source}-{record_id}",
|
||||
name=name,
|
||||
extra_data={
|
||||
"country": country,
|
||||
"city": city,
|
||||
"latitude": latitude,
|
||||
"longitude": longitude,
|
||||
**metadata,
|
||||
},
|
||||
collected_at=datetime(2026, 4, 22, tzinfo=timezone.utc),
|
||||
reference_date=datetime(2026, 4, 21, tzinfo=timezone.utc),
|
||||
is_current=True,
|
||||
)
|
||||
|
||||
|
||||
def test_convert_compute_centers_to_geojson_unifies_sources():
|
||||
top500_record = _build_record(
|
||||
record_id=1,
|
||||
source="top500",
|
||||
data_type="supercomputer",
|
||||
name="Frontier",
|
||||
country="United States",
|
||||
city="Oak Ridge",
|
||||
latitude=35.93,
|
||||
longitude=-84.31,
|
||||
metadata={
|
||||
"rank": 1,
|
||||
"manufacturer": "HPE",
|
||||
"organization": "ORNL",
|
||||
"rmax": 1102000.0,
|
||||
"cores": 8730112,
|
||||
"power": 21510.0,
|
||||
},
|
||||
)
|
||||
gpu_record = _build_record(
|
||||
record_id=2,
|
||||
source="epoch_ai_gpu",
|
||||
data_type="gpu_cluster",
|
||||
name="Colossus",
|
||||
country="United States",
|
||||
city="Memphis",
|
||||
latitude=35.15,
|
||||
longitude=-90.05,
|
||||
metadata={
|
||||
"organization": "xAI",
|
||||
"gpu_type": "H100",
|
||||
"gpu_count": 100000,
|
||||
"value": "20000",
|
||||
"unit": "TFlop/s",
|
||||
},
|
||||
)
|
||||
|
||||
payload = convert_compute_centers_to_geojson([top500_record, gpu_record])
|
||||
|
||||
assert payload["type"] == "FeatureCollection"
|
||||
assert len(payload["features"]) == 2
|
||||
|
||||
supercomputer_feature = payload["features"][0]
|
||||
assert supercomputer_feature["properties"]["site_type"] == "supercomputer"
|
||||
assert supercomputer_feature["properties"]["capacity_unit"] == "GFlops"
|
||||
assert supercomputer_feature["properties"]["capacity_band"] == "exascale"
|
||||
assert supercomputer_feature["properties"]["operator"] == "ORNL"
|
||||
assert supercomputer_feature["properties"]["location_precision"] == "precise"
|
||||
assert supercomputer_feature["properties"]["is_estimated"] is False
|
||||
|
||||
gpu_feature = payload["features"][1]
|
||||
assert gpu_feature["properties"]["site_type"] == "gpu_cluster"
|
||||
assert gpu_feature["properties"]["vendor"] == "H100"
|
||||
assert gpu_feature["properties"]["gpu_count"] == 100000
|
||||
assert gpu_feature["properties"]["capacity_band"] == "large"
|
||||
assert gpu_feature["properties"]["location_precision"] == "precise"
|
||||
|
||||
|
||||
def test_convert_compute_centers_to_geojson_uses_coordinate_hints():
|
||||
hinted_record = _build_record(
|
||||
record_id=3,
|
||||
source="top500",
|
||||
data_type="supercomputer",
|
||||
name="Frontier",
|
||||
country="United States",
|
||||
city="",
|
||||
latitude=0.0,
|
||||
longitude=0.0,
|
||||
metadata={
|
||||
"organization": "Oak Ridge National Laboratory",
|
||||
"rmax": 1102000.0,
|
||||
},
|
||||
)
|
||||
|
||||
payload = convert_compute_centers_to_geojson([hinted_record])
|
||||
|
||||
assert len(payload["features"]) == 1
|
||||
coords = payload["features"][0]["geometry"]["coordinates"]
|
||||
assert coords[0] == pytest.approx(-84.3107)
|
||||
assert coords[1] == pytest.approx(35.9319)
|
||||
assert payload["features"][0]["properties"]["is_estimated"] is True
|
||||
assert payload["features"][0]["properties"]["location_precision"] == "estimated_site"
|
||||
|
||||
|
||||
def test_convert_compute_centers_to_geojson_falls_back_to_country_centroid():
|
||||
centroid_record = _build_record(
|
||||
record_id=4,
|
||||
source="epoch_ai_gpu",
|
||||
data_type="gpu_cluster",
|
||||
name="Unknown Cluster",
|
||||
country="United States",
|
||||
city="",
|
||||
latitude=0.0,
|
||||
longitude=0.0,
|
||||
metadata={
|
||||
"organization": "Unknown Operator",
|
||||
"value": "10000",
|
||||
"unit": "TFlop/s",
|
||||
},
|
||||
)
|
||||
|
||||
payload = convert_compute_centers_to_geojson([centroid_record])
|
||||
|
||||
assert len(payload["features"]) == 1
|
||||
props = payload["features"][0]["properties"]
|
||||
coords = payload["features"][0]["geometry"]["coordinates"]
|
||||
assert coords[0] == pytest.approx(-98.5795)
|
||||
assert coords[1] == pytest.approx(39.8283)
|
||||
assert props["is_estimated"] is True
|
||||
assert props["location_precision"] == "estimated_country"
|
||||
assert props["geography_mode"] == "country_centroid"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_compute_centers_geojson_endpoint_returns_stats():
|
||||
records = [
|
||||
_build_record(
|
||||
record_id=1,
|
||||
source="top500",
|
||||
data_type="supercomputer",
|
||||
name="Frontier",
|
||||
country="United States",
|
||||
city="Oak Ridge",
|
||||
latitude=35.93,
|
||||
longitude=-84.31,
|
||||
metadata={"rank": 1, "rmax": 1102000.0},
|
||||
),
|
||||
_build_record(
|
||||
record_id=2,
|
||||
source="epoch_ai_gpu",
|
||||
data_type="gpu_cluster",
|
||||
name="Colossus",
|
||||
country="United States",
|
||||
city="Memphis",
|
||||
latitude=35.15,
|
||||
longitude=-90.05,
|
||||
metadata={"value": "20000", "unit": "TFlop/s"},
|
||||
),
|
||||
]
|
||||
|
||||
class _ScalarResult:
|
||||
def __init__(self, rows):
|
||||
self._rows = rows
|
||||
|
||||
def scalars(self):
|
||||
class _Scalars:
|
||||
def __init__(self, rows):
|
||||
self._rows = rows
|
||||
|
||||
def all(self):
|
||||
return self._rows
|
||||
|
||||
return _Scalars(self._rows)
|
||||
|
||||
class _FakeSession:
|
||||
async def execute(self, _query):
|
||||
return _ScalarResult(records)
|
||||
|
||||
async def override_get_db():
|
||||
yield _FakeSession()
|
||||
|
||||
app.dependency_overrides[get_db] = override_get_db
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/api/v1/visualization/geo/compute-centers")
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["count"] == 2
|
||||
assert data["stats"]["supercomputers"] == 1
|
||||
assert data["stats"]["gpu_clusters"] == 1
|
||||
assert data["features"][0]["properties"]["data_type"] == "compute_center"
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_visualization_geo_summary_returns_counts(monkeypatch):
|
||||
records = [
|
||||
_build_record(
|
||||
record_id=1,
|
||||
source="arcgis_cables",
|
||||
data_type="submarine_cable",
|
||||
name="Test Cable",
|
||||
country="",
|
||||
city="",
|
||||
latitude=0,
|
||||
longitude=0,
|
||||
metadata={
|
||||
"route_coordinates": [[[0, 0], [1, 1]]],
|
||||
"status": "active",
|
||||
},
|
||||
),
|
||||
_build_record(
|
||||
record_id=2,
|
||||
source="arcgis_landing_points",
|
||||
data_type="landing_point",
|
||||
name="Test Landing",
|
||||
country="United States",
|
||||
city="New York",
|
||||
latitude=40.7,
|
||||
longitude=-74.0,
|
||||
metadata={"city_id": 10},
|
||||
),
|
||||
_build_record(
|
||||
record_id=3,
|
||||
source="celestrak_tle",
|
||||
data_type="satellite_tle",
|
||||
name="TESTSAT",
|
||||
country="",
|
||||
city="",
|
||||
latitude=0,
|
||||
longitude=0,
|
||||
metadata={
|
||||
"norad_cat_id": 12345,
|
||||
"tle_line1": "1 12345U 98067A 24001.00000000 .00000000 00000-0 00000-0 0 9991",
|
||||
"tle_line2": "2 12345 51.6000 100.0000 0001000 10.0000 20.0000 15.50000000 01",
|
||||
},
|
||||
),
|
||||
_build_record(
|
||||
record_id=4,
|
||||
source="top500",
|
||||
data_type="supercomputer",
|
||||
name="Frontier",
|
||||
country="United States",
|
||||
city="Oak Ridge",
|
||||
latitude=35.93,
|
||||
longitude=-84.31,
|
||||
metadata={"rank": 1, "rmax": 1102000.0},
|
||||
),
|
||||
_build_record(
|
||||
record_id=5,
|
||||
source="epoch_ai_gpu",
|
||||
data_type="gpu_cluster",
|
||||
name="Colossus",
|
||||
country="United States",
|
||||
city="Memphis",
|
||||
latitude=35.15,
|
||||
longitude=-90.05,
|
||||
metadata={"value": "20000", "unit": "TFlop/s"},
|
||||
),
|
||||
]
|
||||
|
||||
class _ScalarResult:
|
||||
def __init__(self, rows=None, scalar_value=None):
|
||||
self._rows = rows or []
|
||||
self._scalar_value = scalar_value
|
||||
|
||||
def scalar(self):
|
||||
return self._scalar_value
|
||||
|
||||
def scalars(self):
|
||||
class _Scalars:
|
||||
def __init__(self, rows):
|
||||
self._rows = rows
|
||||
|
||||
def all(self):
|
||||
return self._rows
|
||||
|
||||
return _Scalars(self._rows)
|
||||
|
||||
class _FakeSession:
|
||||
async def execute(self, query):
|
||||
query_text = str(query)
|
||||
if "bgp_incidents" in query_text:
|
||||
return _ScalarResult(scalar_value=2)
|
||||
if "bgp_anomalies" in query_text:
|
||||
return _ScalarResult(scalar_value=3)
|
||||
return _ScalarResult(rows=records)
|
||||
|
||||
async def override_get_db():
|
||||
yield _FakeSession()
|
||||
|
||||
async def _fake_build_bgp_collector_coverage(*_args, **_kwargs):
|
||||
return [
|
||||
{"collector": "rrc00"},
|
||||
{"collector": "rrc01"},
|
||||
]
|
||||
|
||||
monkeypatch.setattr(
|
||||
"app.api.v1.visualization.build_bgp_collector_coverage",
|
||||
_fake_build_bgp_collector_coverage,
|
||||
)
|
||||
|
||||
app.dependency_overrides[get_db] = override_get_db
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/api/v1/visualization/geo/summary")
|
||||
|
||||
assert response.status_code == 200
|
||||
stats = response.json()["stats"]
|
||||
assert stats["cable_count"] == 1
|
||||
assert stats["landing_point_count"] == 1
|
||||
assert stats["satellite_count"] == 1
|
||||
assert stats["compute_center_count"] == 2
|
||||
assert stats["supercomputer_count"] == 1
|
||||
assert stats["gpu_cluster_count"] == 1
|
||||
assert stats["bgp_event_count"] == 2
|
||||
assert stats["bgp_incident_count"] == 2
|
||||
assert stats["bgp_anomaly_count"] == 3
|
||||
assert stats["bgp_collector_count"] == 2
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
@@ -18,6 +18,9 @@ services:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: aiprovider/Dockerfile
|
||||
args:
|
||||
PYTHON_IMAGE: ${PYTHON_IMAGE:-python:3.14-slim}
|
||||
UV_IMAGE: ${UV_IMAGE:-ghcr.io/astral-sh/uv:latest}
|
||||
container_name: planet_aiprovider
|
||||
ports:
|
||||
- "8010:8010"
|
||||
|
||||
@@ -5,6 +5,9 @@ services:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: aiprovider/Dockerfile
|
||||
args:
|
||||
PYTHON_IMAGE: ${PYTHON_IMAGE:-python:3.14-slim}
|
||||
UV_IMAGE: ${UV_IMAGE:-ghcr.io/astral-sh/uv:latest}
|
||||
container_name: planet_aiprovider
|
||||
ports:
|
||||
- "8010:8010"
|
||||
|
||||
@@ -5,6 +5,11 @@ services:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: aiprovider/Dockerfile
|
||||
args:
|
||||
PYTHON_IMAGE: ${PYTHON_IMAGE:-python:3.14-slim}
|
||||
UV_IMAGE: ${UV_IMAGE:-ghcr.io/astral-sh/uv:latest}
|
||||
env_file:
|
||||
- ./aiprovider/.env
|
||||
container_name: planet_aiprovider
|
||||
ports:
|
||||
- "8010:8010"
|
||||
|
||||
+1081
-5
File diff suppressed because it is too large.
Load diff
Loaded 100 of 292 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user