Compare commits
5
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
c97dd83f3e | ||
|
|
a4e6ce7489 | ||
|
|
7ffc8537e4 | ||
|
|
4dd396ea65 | ||
|
|
1e6f4b338b |
No files matched your search
@@ -12,19 +12,6 @@ allowed-tools: ["Read", "Edit", "Bash", "Grep", "Glob"]
|
||||
|
||||
若 `$ARGUMENTS` 非空,则只检查指定文件/目录;否则检查所有未提交修改(`git diff HEAD`)。
|
||||
|
||||
## 节省上下文规则
|
||||
|
||||
优先用确定性的 CLI 检查缩小范围,不要一上来把完整文件或大 diff 读入上下文:
|
||||
|
||||
```bash
|
||||
git diff --name-only HEAD
|
||||
git diff --unified=0 HEAD -- <path>
|
||||
git diff --check
|
||||
rg -n "TODO|FIXME|console\.log|debugger|print\(" <changed-paths>
|
||||
```
|
||||
|
||||
只有 focused diff 不足以安全判断或修改时,才读取完整文件。
|
||||
|
||||
## 审查清单
|
||||
|
||||
按优先级检查以下问题(只报告在本次 diff 中**新增或修改**的代码里存在的问题):
|
||||
@@ -72,13 +59,8 @@ git diff HEAD --name-only
|
||||
|
||||
### Step 2 — 逐文件阅读并分析
|
||||
|
||||
先从 focused diff 开始:
|
||||
|
||||
```bash
|
||||
git diff --unified=0 HEAD -- <file>
|
||||
```
|
||||
|
||||
用 `rg`、`git diff --check`、编译器或 linter 输出确认确定性问题。只有需要上下文时才用 Read 读取完整文件。对照审查清单,记录每个问题:文件名、行号、问题类型、建议修复方式。
|
||||
- 用 Read 工具读取完整文件(不只读 diff)
|
||||
- 对照审查清单,记录每个问题:文件名、行号、问题类型、建议修复方式
|
||||
|
||||
### Step 3 — 报告问题清单
|
||||
|
||||
@@ -113,7 +95,6 @@ git diff --unified=0 HEAD -- <file>
|
||||
- 只改在审查清单中发现的问题,不做额外优化
|
||||
- 每次 Edit 只修改确实有问题的行,保持 diff 最小
|
||||
- 改完后用 `grep` 验证旧的坏代码已消失
|
||||
- 优先做精确补丁;只有仓库已有对应格式化流程时,才运行格式化工具
|
||||
|
||||
### Step 5 — 输出总结
|
||||
|
||||
|
||||
@@ -1,232 +0,0 @@
|
||||
---
|
||||
description: 分析本次 git 变更,在 docs/technical/zh/ 中新建或更新对应的技术文档
|
||||
argument-hint: 可选:指定要记录的主题,或留空自动从 git diff 推断
|
||||
allowed-tools: ["Read", "Edit", "Write", "Bash", "Glob", "Grep"]
|
||||
---
|
||||
|
||||
# /docs — 技术文档写入工作流
|
||||
|
||||
## 目标
|
||||
|
||||
根据当前 git 变更(或用户指定主题)在 `docs/technical/zh/` 中写入或更新技术文档,记录**为什么**这样做,而不只是记录做了什么。
|
||||
|
||||
## 执行步骤
|
||||
|
||||
### Step 1 — 理解变更范围
|
||||
|
||||
```bash
|
||||
git diff HEAD --stat # 变更文件一览
|
||||
git diff HEAD --name-only # 变更文件列表
|
||||
git log --oneline -10 # 近期 commit 上下文
|
||||
```
|
||||
|
||||
若 `$ARGUMENTS` 指定了主题,优先聚焦该主题;否则从文件列表和 diff stat 推断变更主题。不要默认读取完整仓库 diff;只对决定文档主题所需的文件读取 focused diff:
|
||||
|
||||
```bash
|
||||
git diff HEAD -- <path>
|
||||
rg -n "class |def |function |export |router|@router|interface |type " <path>
|
||||
```
|
||||
|
||||
### Step 2 — 确认文档范围
|
||||
|
||||
分析变更,判断:
|
||||
|
||||
1. **应写几篇文档**:单一主题写一篇,跨领域变更可拆分(如后端性能优化 + 运维启动脚本分开写)
|
||||
2. **是新建还是更新**:检查 `docs/technical/zh/` 中是否已有相关文档
|
||||
3. **文档命名**:按 `领域-主题-副题.md` 格式,全小写,用连字符,如:
|
||||
- `backend-datasources-api-performance.md`
|
||||
- `ops-planet-sh-startup.md`
|
||||
- `earth-bgp-context.md`
|
||||
|
||||
```bash
|
||||
ls docs/technical/zh/ # 查看现有文档
|
||||
```
|
||||
|
||||
**先输出写作计划供用户确认**(若变更明确且范围小,可直接执行):
|
||||
|
||||
```
|
||||
文档计划:
|
||||
新建:docs/technical/zh/ops-planet-sh-startup.md — planet.sh 启动性能优化
|
||||
更新:docs/technical/zh/backend-datasources-api-performance.md — 补充并行化细节
|
||||
```
|
||||
|
||||
### Step 2.5 — 覆盖范围检查
|
||||
|
||||
写文档前必须按变更类型检查配套文档,不要只更新一篇专题文档:
|
||||
|
||||
- 用户可见流程变化:更新 `docs/technical/zh/manual.md`,通常也更新 `docs/technical/zh/quickstart.md`。
|
||||
- `manual.md`、`quickstart.md` 这类用户手册存在英文版时,同步更新 `docs/technical/en/...`,至少避免英文版与中文版互相矛盾。
|
||||
- 控制台页面职责、路由入口、表格/抽屉/设置页行为变化:更新 `docs/technical/zh/frontend-admin-frontend-context.md`。
|
||||
- Earth 前端行为、HUD、巡航、图层、图例、交互变化:更新 `docs/technical/zh/earth-frontend-context.md`。
|
||||
- 新增 Earth 图层、调整 `renderOrder`、半径/高度偏移、深度策略、拾取策略、legend mode、图层面板顺序或启动加载顺序:更新 `docs/technical/zh/earth-render-layer-order.md`。
|
||||
- Earth 图层视觉样式、颜色、图例符号语义变化:若影响样式索引,同步更新 `docs/technical/zh/earth-layer-style-reference.md`。
|
||||
- 采集器、数据源、凭证、设置页、连接检查、scheduler、后端 API 变化:更新相关后端文档,优先检查 `docs/technical/zh/backend-collectors.md` 和 datasource/settings 专题文档。
|
||||
- 如果某个旧 plan 的假设已经被当前实现推翻,在对应 `docs/plans/*.md` 增加现状修正或更新该段,不要让计划文档继续给出相反方向。
|
||||
- 新增 technical 文档后,如果需要被发现,更新 `docs/technical/zh/README.md`。
|
||||
- 如果 technical 文档需要在公开 Docs 页面显示,或从 technical README 链接进入,必须同步更新 `frontend/src/pages/Docs/docs-content.ts` 的 `DOCS_METADATA`。前端使用这份白名单,`docs/technical/{zh,en}/` 中存在 `.md` 文件并不会自动生成路由。
|
||||
- 公开 technical 文档必须按同名文件维护中英文双语版本:`docs/technical/zh/<name>.md` 与 `docs/technical/en/<name>.md`。如果某篇文档刻意只保留单语,完成说明中必须明确写出原因。
|
||||
- 对本次变更提取旧词做 stale search,例如旧 tab 名、旧路由职责、旧认证假设、改名前 UI 文案:
|
||||
|
||||
```bash
|
||||
rg -n "旧文案|旧路由职责|旧认证假设" docs/technical docs/plans
|
||||
```
|
||||
|
||||
### Step 3 — 写文档
|
||||
|
||||
遵循以下原则:
|
||||
|
||||
**记录 WHY,不只记录 WHAT**
|
||||
- 好:`将戳文件从 /tmp 移到 ~/.cache/planet/,因为 WSL 重启后 /tmp 被清空`
|
||||
- 差:`修改了 AI_PROVIDER_BUILD_STAMP_FILE 的值`
|
||||
|
||||
**必须包含的内容**:
|
||||
- 背景/问题:改动之前存在什么问题,为什么要改
|
||||
- 核心设计决策及其理由
|
||||
- 关键代码片段(用 diff 或 before/after 展示)
|
||||
- 相关文件列表
|
||||
|
||||
**格式要求**:
|
||||
- 使用 `##` 和 `###` 分级,不要超过三级
|
||||
- 代码块注明语言(python / bash / typescript / sql)
|
||||
- 表格用于对比多个选项或列出参数
|
||||
- 中文写作,技术术语保留英文原文
|
||||
- `docs/technical/zh/` 中的文档不得用英文原文占位;如果存在 `docs/technical/en/` 对应文件,禁止逐字复制成中文文件
|
||||
- 中文文档内部链接应指向 `docs/technical/zh/...`,除非明确引用英文专属文档
|
||||
- 公开文档的 Markdown 链接显示文字应使用可读标题,不要直接暴露 `manual.md`、`earth-frontend-context.md` 这类裸文件名
|
||||
|
||||
**文档结构模板**:
|
||||
|
||||
```markdown
|
||||
# 标题(说明做了什么)
|
||||
|
||||
## 背景
|
||||
|
||||
为什么要做这个改动,改动前存在什么问题。
|
||||
|
||||
## 核心变更
|
||||
|
||||
### 子主题一
|
||||
|
||||
before/after 或决策说明 + 关键代码
|
||||
|
||||
### 子主题二
|
||||
|
||||
...
|
||||
|
||||
## 相关文件
|
||||
|
||||
- `path/to/file.py` — 简短说明
|
||||
```
|
||||
|
||||
### Step 4 — 验证
|
||||
|
||||
- 读一遍写好的文档,确认逻辑清晰、代码片段无明显错误
|
||||
- 用 `rg --files` 或 `test -e` 确认文档中的文件路径在项目中真实存在,避免凭记忆判断:
|
||||
- 检查中文文档没有误复制英文版:
|
||||
|
||||
```bash
|
||||
python - <<'PY'
|
||||
from pathlib import Path
|
||||
same = []
|
||||
for en in sorted(Path("docs/technical/en").glob("*.md")):
|
||||
zh = Path("docs/technical/zh") / en.name
|
||||
if zh.exists() and en.read_text() == zh.read_text():
|
||||
same.append(en.name)
|
||||
if same:
|
||||
raise SystemExit("identical en/zh docs: " + ", ".join(same))
|
||||
print("no identical en/zh docs")
|
||||
PY
|
||||
```
|
||||
|
||||
- 检查中文文档内部链接没有继续指向无语言目录:
|
||||
|
||||
```bash
|
||||
rg -n "/home/ray/dev/linkong/planet/docs/technical/(?!zh|en)" docs/technical/zh --pcre2
|
||||
```
|
||||
|
||||
- 检查公开文档链接已进入 Docs 前端白名单。凡是 `docs/technical/{zh,en}/README.md` 中链接到的 technical `.md`,都必须存在于 `DOCS_METADATA`:
|
||||
|
||||
```bash
|
||||
python - <<'PY'
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
metadata = Path("frontend/src/pages/Docs/docs-content.ts").read_text()
|
||||
known = set(re.findall(r"'([^']+\.md)':\s*\{", metadata))
|
||||
known.add("README.md")
|
||||
|
||||
missing = []
|
||||
for readme in [Path("docs/technical/zh/README.md"), Path("docs/technical/en/README.md")]:
|
||||
if not readme.exists():
|
||||
continue
|
||||
for href in re.findall(r"\]\(([^)]+\.md)\)", readme.read_text()):
|
||||
path = Path(href)
|
||||
if "docs/technical/" not in href:
|
||||
continue
|
||||
filename = path.name
|
||||
if filename not in known:
|
||||
missing.append(f"{readme}: {filename}")
|
||||
|
||||
if missing:
|
||||
raise SystemExit("docs README links missing DOCS_METADATA: " + ", ".join(missing))
|
||||
print("docs README links are whitelisted")
|
||||
PY
|
||||
```
|
||||
|
||||
- 检查公开文档双语同名文件齐备。除 `README.md` 外,所有白名单文档都应同时存在 zh/en 文件,除非本次说明中明确豁免:
|
||||
|
||||
```bash
|
||||
python - <<'PY'
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
metadata = Path("frontend/src/pages/Docs/docs-content.ts").read_text()
|
||||
filenames = sorted(set(re.findall(r"'([^']+\.md)':\s*\{", metadata)) - {"README.md"})
|
||||
missing = []
|
||||
for filename in filenames:
|
||||
for lang in ("zh", "en"):
|
||||
path = Path("docs/technical") / lang / filename
|
||||
if not path.exists():
|
||||
missing.append(str(path))
|
||||
if missing:
|
||||
raise SystemExit("missing bilingual docs: " + ", ".join(missing))
|
||||
print("public docs have zh/en file pairs")
|
||||
PY
|
||||
```
|
||||
|
||||
- 检查公开文档里没有用裸 `.md` 文件名当链接标题。这个命令在 polished public docs 中应无输出:
|
||||
|
||||
```bash
|
||||
rg -n "\[[^]]+\.md\]\(" docs/technical/zh docs/technical/en
|
||||
```
|
||||
|
||||
```bash
|
||||
# 对文档中提到的关键路径做快速验证
|
||||
ls <mentioned_paths>
|
||||
```
|
||||
|
||||
如需检查大量链接,优先用确定性提取:
|
||||
|
||||
```bash
|
||||
rg -n "\]\(([^)]+)\)" docs/technical/zh/<doc>.md
|
||||
```
|
||||
|
||||
### Step 5 — 完成确认
|
||||
|
||||
输出摘要:
|
||||
|
||||
```
|
||||
✓ 新建:docs/technical/zh/ops-planet-sh-startup.md(约 xxx 字)
|
||||
✓ 更新:docs/technical/zh/backend-datasources-api-performance.md
|
||||
```
|
||||
|
||||
## 注意事项
|
||||
|
||||
- 不要写流水账式的"改了 A、改了 B、改了 C",要写改动背后的约束和权衡
|
||||
- 不要在文档中引用 PR 号、issue 号、或当前对话——这些会随时间失效
|
||||
- 代码片段保持简洁,只保留说明问题的关键部分,省略无关样板代码
|
||||
- 如果某个变更已有文档记录,优先在原文档中追加,而不是新建
|
||||
- 公开 technical 文档没有注册 `DOCS_METADATA` 时,Docs 页面不会显示;不要只创建 `.md` 文件就结束。
|
||||
- 公开 technical 文档默认需要 zh/en 同名文件,不要只补一个语言版本。
|
||||
- 链接可见文字使用文档标题或语义标题,不要使用裸文件名。
|
||||
- 文档是给未来的开发者看的,假设读者熟悉项目但不了解这次改动的背景
|
||||
@@ -1,93 +0,0 @@
|
||||
---
|
||||
description: 用 goal-driven 方法推动一个复杂任务持续执行,直到明确成功标准被满足
|
||||
argument-hint: 建议填写任务目标;若同时给出成功标准更好
|
||||
allowed-tools: ["Read", "Edit", "Bash", "Grep", "Glob"]
|
||||
---
|
||||
|
||||
# /goal-driven — 目标驱动执行模式
|
||||
|
||||
使用 `lidangzzz/goal-driven` 的核心思想来推进复杂任务:先固定目标与成功标准,再持续执行和反复验收,直到标准真正满足。
|
||||
|
||||
适用场景:
|
||||
|
||||
- 长周期实现任务
|
||||
- 高复杂度工程任务
|
||||
- 可被明确验收的研究、实现、迁移、验证类工作
|
||||
|
||||
不适用场景:
|
||||
|
||||
- 纯脑暴
|
||||
- 无法定义成功标准的模糊任务
|
||||
- 很小的一次性修改
|
||||
|
||||
## 输入要求
|
||||
|
||||
若 `$ARGUMENTS` 只包含目标,没有成功标准,先补全一版可执行的成功标准再开始。
|
||||
|
||||
启动时先输出:
|
||||
|
||||
```md
|
||||
Goal
|
||||
- ...
|
||||
|
||||
Criteria for success
|
||||
- ...
|
||||
|
||||
Plan
|
||||
1. ...
|
||||
2. ...
|
||||
3. ...
|
||||
|
||||
Verification
|
||||
- ...
|
||||
```
|
||||
|
||||
## 执行规则
|
||||
|
||||
1. 先把任务固化为两个核心块:
|
||||
- `Goal`
|
||||
- `Criteria for success`
|
||||
|
||||
2. 成功标准必须尽量客观,可验证,可落地。
|
||||
优先写成:
|
||||
- 需要交付什么
|
||||
- 需要通过哪些测试或验证
|
||||
- 如何判断结果真的完成
|
||||
|
||||
3. 进入持续执行循环:
|
||||
- 完成一个阶段
|
||||
- 检查当前结果是否满足成功标准
|
||||
- 若未满足,明确剩余差距并继续推进
|
||||
|
||||
4. 任何“完成了”“差不多了”“已实现”之类的结论,都必须经过验证,不能直接接受。
|
||||
|
||||
5. 如果验证失败:
|
||||
- 明确指出哪条成功标准没满足
|
||||
- 继续工作,不要把阶段性进展误判为完成
|
||||
|
||||
6. 只有在以下情况之一才能停止:
|
||||
- 成功标准已满足
|
||||
- 用户明确要求停止
|
||||
|
||||
## 执行风格
|
||||
|
||||
- 重证据,轻口头判断
|
||||
- 优先使用确定性工具证据:`rg`、`git diff --stat`、`git diff -- <path>`、测试、构建、lint、`curl`、数据库查询等能直接证明成功标准的方式
|
||||
- 不把大段命令输出粘进回复;保留在工具调用里,回复只总结关键证据
|
||||
- 重验收,轻自我感觉
|
||||
- 优先用测试、日志、产物、对比结果来证明完成
|
||||
- 对长期任务保持“未达标就继续”的节奏
|
||||
|
||||
## 简版模板
|
||||
|
||||
```md
|
||||
Goal: [[[[[在此填写最终目标]]]]]
|
||||
|
||||
Criteria for success: [[[[[在此填写成功标准]]]]]
|
||||
|
||||
循环执行:
|
||||
1. 推进任务
|
||||
2. 检查是否满足成功标准
|
||||
3. 若未满足,继续工作
|
||||
4. 直到满足标准或用户明确停止
|
||||
```
|
||||
@@ -28,19 +28,6 @@ allowed-tools: ["Read", "Edit", "Bash", "Glob", "Grep"]
|
||||
- `docs/CHANGELOG.md`
|
||||
- `docs/version-history.md`
|
||||
|
||||
## 节省上下文规则
|
||||
|
||||
发版判断应以确定性 CLI 证据为主,优先使用紧凑命令和定点读取:
|
||||
|
||||
```bash
|
||||
git status --short
|
||||
git diff --stat HEAD
|
||||
git diff --name-only HEAD
|
||||
rg -n "version|^## |^Released:|当前开发版本|current" VERSION frontend/package.json pyproject.toml docs/CHANGELOG.md docs/version-history.md
|
||||
```
|
||||
|
||||
除非需要判断某个代码变更是否属于本次发版,否则不要读取完整 diff。
|
||||
|
||||
## 执行步骤
|
||||
|
||||
### Step 1 — 环境检查
|
||||
@@ -58,7 +45,7 @@ cat VERSION # 读取当前版本
|
||||
### Step 2 — 确定发版类型与新版本号
|
||||
|
||||
- 若 `$ARGUMENTS` 提供了明确类型(`feature` / `bugfix`),直接使用
|
||||
- 否则根据 `git diff --stat HEAD`、`git diff --name-only HEAD`、必要的 focused diff 和 `git log` 推断
|
||||
- 否则根据当前 `git diff HEAD` 和 `git log` 推断
|
||||
- 计算新版本号(例:`0.26.2` → bugfix → `0.26.3`)
|
||||
- **先输出发版计划供用户确认**:
|
||||
|
||||
@@ -104,13 +91,12 @@ cat VERSION # 读取当前版本
|
||||
|
||||
针对本次变更范围做最小验证:
|
||||
|
||||
- Python 文件有修改:先用 `git diff --name-only HEAD -- '*.py'` 列出,再运行 `python3 -m py_compile <changed_files>`
|
||||
- Frontend 文件有修改:先用 `git diff --name-only HEAD -- frontend` 判断范围,再运行项目标准检查(若无则跳过并说明)
|
||||
- Python 文件有修改:`python3 -m py_compile <changed_files>`
|
||||
- Frontend 文件有修改:运行项目标准检查(若无则跳过并说明)
|
||||
- 版本号一致性检查:用 grep 确认 VERSION、package.json、pyproject.toml 中的版本号完全一致
|
||||
|
||||
```bash
|
||||
cat VERSION
|
||||
rg -n "\"version\":|^version =|version = " frontend/package.json pyproject.toml uv.lock
|
||||
grep -h "version" VERSION frontend/package.json pyproject.toml
|
||||
```
|
||||
|
||||
### Step 7 — 提交前预览
|
||||
|
||||
@@ -1,3 +0,0 @@
|
||||
approval_policy = "never"
|
||||
|
||||
sandbox_mode = "danger-full-access"
|
||||
@@ -21,19 +21,6 @@ If the user specifies a file or directory, check only that. Otherwise check all
|
||||
|
||||
Only report issues present in **newly added or modified** lines of this diff — do not audit unchanged code.
|
||||
|
||||
## Token-Saving Rule
|
||||
|
||||
Prefer deterministic CLI checks before reading files into model context:
|
||||
|
||||
```bash
|
||||
git diff --name-only HEAD
|
||||
git diff --unified=0 HEAD -- <path>
|
||||
git diff --check
|
||||
rg -n "TODO|FIXME|console\.log|debugger|print\(" <changed-paths>
|
||||
```
|
||||
|
||||
Read full files only when the focused diff does not provide enough surrounding context to make a safe edit.
|
||||
|
||||
## Checklist
|
||||
|
||||
### 1. Duplicate Logic
|
||||
@@ -77,13 +64,7 @@ Filter to the user-specified path if one was provided.
|
||||
|
||||
### Step 2 — Read and analyze each file
|
||||
|
||||
Start with focused diffs:
|
||||
|
||||
```bash
|
||||
git diff --unified=0 HEAD -- <file>
|
||||
```
|
||||
|
||||
Use `rg`, `git diff --check`, and compiler/linter output for deterministic findings. Read the full file only for files that need surrounding context. For each issue found, record filename, line number, category, and suggested fix.
|
||||
Read the full file (not just the diff) with the Read tool. For each file, record every issue found: filename, line number, category, and suggested fix.
|
||||
|
||||
### Step 3 — Report findings before touching anything
|
||||
|
||||
@@ -118,7 +99,6 @@ Principles:
|
||||
- Only fix issues identified in the checklist — no extra improvements
|
||||
- Keep each Edit as small as possible
|
||||
- After fixing, verify the old bad pattern is gone with grep
|
||||
- Prefer `apply_patch` for targeted edits; use formatters only when the repository already uses them for the touched file type
|
||||
|
||||
### Step 5 — Summary
|
||||
|
||||
|
||||
@@ -1,200 +0,0 @@
|
||||
---
|
||||
name: docs
|
||||
description: Analyze current Planet repo changes and create or update technical documentation under docs/technical/zh. Use when the user asks to write docs, update technical docs, summarize implementation changes into documentation, or port the Claude docs-codex workflow into Codex.
|
||||
---
|
||||
|
||||
# Docs
|
||||
|
||||
Use this skill when the user asks to create or update Planet technical documentation, especially under `docs/technical/zh/`.
|
||||
|
||||
## Goal
|
||||
|
||||
Write or update technical docs that explain why a change exists, not only what files changed.
|
||||
|
||||
Default target directory:
|
||||
|
||||
- `docs/technical/zh/`
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Gather change context:
|
||||
|
||||
```bash
|
||||
git diff HEAD --stat
|
||||
git diff HEAD --name-only
|
||||
git log --oneline -10
|
||||
ls docs/technical/zh/
|
||||
```
|
||||
|
||||
If the user gives a specific topic, focus on that topic. Otherwise infer the documentation topic from the file list and diff stat. Do **not** read the full repository diff by default; inspect focused diffs only for the files that define the doc topic:
|
||||
|
||||
```bash
|
||||
git diff HEAD -- <path>
|
||||
rg -n "class |def |function |export |router|@router|interface |type " <path>
|
||||
```
|
||||
|
||||
2. Decide document scope:
|
||||
|
||||
- Use one document for one coherent topic.
|
||||
- Split documents when the changes cross meaningful domains, such as backend performance and ops startup behavior.
|
||||
- Prefer updating an existing relevant doc over creating a duplicate.
|
||||
- Name new files as lowercase hyphenated `domain-topic-detail.md`, for example:
|
||||
- `backend-datasources-api-performance.md`
|
||||
- `ops-planet-sh-startup.md`
|
||||
- `earth-bgp-context.md`
|
||||
|
||||
3. Apply the documentation coverage checklist before writing:
|
||||
|
||||
- User-visible workflow changes must update `docs/technical/zh/manual.md` and usually `docs/technical/zh/quickstart.md`.
|
||||
- If an English counterpart exists for user-facing docs such as `manual.md` or `quickstart.md`, update `docs/technical/en/...` enough that it does not contradict the Chinese source.
|
||||
- Control console page responsibility changes must update `docs/technical/zh/frontend-admin-frontend-context.md`.
|
||||
- Earth frontend behavior changes must update `docs/technical/zh/earth-frontend-context.md`.
|
||||
- Earth layer additions, `renderOrder`, altitude/radius offsets, depth strategy, pointer picking, legend modes, or layer panel/startup ordering must update `docs/technical/zh/earth-render-layer-order.md`.
|
||||
- Earth layer visual style or legend symbol/color semantics should also update `docs/technical/zh/earth-layer-style-reference.md` when that reference is affected.
|
||||
- Collector, datasource, credential, settings, connectivity, scheduler, or API changes must update the relevant backend docs, especially `docs/technical/zh/backend-collectors.md` and any datasource/settings-specific doc.
|
||||
- When a change turns an old plan assumption into current behavior, update the relevant `docs/plans/*.md` with a status note instead of leaving contradictory instructions.
|
||||
- If adding a new technical document, add it to `docs/technical/zh/README.md` when it should be discoverable from the technical docs index.
|
||||
- If a technical document should be visible in the public Docs page or linked from a technical README, register it in `frontend/src/pages/Docs/docs-content.ts` under `DOCS_METADATA`. The frontend uses this whitelist; files under `docs/technical/{zh,en}/` are not automatically routable.
|
||||
- For every public technical doc, keep the bilingual file pair in sync by filename: `docs/technical/zh/<name>.md` and `docs/technical/en/<name>.md`. If the content is intentionally Chinese-only or English-only, state that intentionally in the final note.
|
||||
- Search docs for stale terms introduced by the change, for example old tab names, old route responsibilities, obsolete auth assumptions, or renamed UI labels.
|
||||
|
||||
4. Write the doc in Chinese:
|
||||
|
||||
- Write Chinese prose for `docs/technical/zh/`.
|
||||
- Keep technical identifiers, API paths, config keys, code symbols, and standard product names in English where appropriate.
|
||||
- Use `##` and `###` headings; avoid going deeper than three levels.
|
||||
- Use fenced code blocks with language tags.
|
||||
- Use tables when comparing options or listing parameters.
|
||||
|
||||
5. Required content:
|
||||
|
||||
- Background/problem: what was wrong before and why the change was needed.
|
||||
- Core design decisions and rationale.
|
||||
- Key code snippets, preferably before/after or focused excerpts.
|
||||
- Related files and what each file contributes.
|
||||
|
||||
6. Verification:
|
||||
|
||||
- Read the completed doc and check that the reasoning is clear.
|
||||
- Verify important referenced paths exist.
|
||||
- Use `rg --files` or `test -e` for path existence instead of relying on memory.
|
||||
- Run a quick duplicate-language check when editing bilingual docs:
|
||||
|
||||
```bash
|
||||
python - <<'PY'
|
||||
from pathlib import Path
|
||||
same = []
|
||||
for en in sorted(Path("docs/technical/en").glob("*.md")):
|
||||
zh = Path("docs/technical/zh") / en.name
|
||||
if zh.exists() and en.read_text() == zh.read_text():
|
||||
same.append(en.name)
|
||||
if same:
|
||||
raise SystemExit("identical en/zh docs: " + ", ".join(same))
|
||||
print("no identical en/zh docs")
|
||||
PY
|
||||
```
|
||||
|
||||
Also check that Chinese docs do not link to the old language-less technical docs path:
|
||||
|
||||
```bash
|
||||
rg -n "/home/ray/dev/linkong/planet/docs/technical/(?!zh|en)" docs/technical/zh --pcre2
|
||||
```
|
||||
|
||||
This command should return no matches.
|
||||
|
||||
Check that public docs are whitelisted in the frontend Docs registry. Any `.md` linked from `docs/technical/{zh,en}/README.md` and located under `docs/technical/{zh,en}/` must have a matching `DOCS_METADATA` key:
|
||||
|
||||
```bash
|
||||
python - <<'PY'
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
metadata = Path("frontend/src/pages/Docs/docs-content.ts").read_text()
|
||||
known = set(re.findall(r"'([^']+\.md)':\s*\{", metadata))
|
||||
known.add("README.md")
|
||||
|
||||
missing = []
|
||||
for readme in [Path("docs/technical/zh/README.md"), Path("docs/technical/en/README.md")]:
|
||||
if not readme.exists():
|
||||
continue
|
||||
for href in re.findall(r"\]\(([^)]+\.md)\)", readme.read_text()):
|
||||
path = Path(href)
|
||||
if "docs/technical/" not in href:
|
||||
continue
|
||||
filename = path.name
|
||||
if filename not in known:
|
||||
missing.append(f"{readme}: {filename}")
|
||||
|
||||
if missing:
|
||||
raise SystemExit("docs README links missing DOCS_METADATA: " + ", ".join(missing))
|
||||
print("docs README links are whitelisted")
|
||||
PY
|
||||
```
|
||||
|
||||
Check bilingual parity for public docs. Every whitelisted document except `README.md` should exist in both language directories unless intentionally documented otherwise:
|
||||
|
||||
```bash
|
||||
python - <<'PY'
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
metadata = Path("frontend/src/pages/Docs/docs-content.ts").read_text()
|
||||
filenames = sorted(set(re.findall(r"'([^']+\.md)':\s*\{", metadata)) - {"README.md"})
|
||||
missing = []
|
||||
for filename in filenames:
|
||||
for lang in ("zh", "en"):
|
||||
path = Path("docs/technical") / lang / filename
|
||||
if not path.exists():
|
||||
missing.append(str(path))
|
||||
if missing:
|
||||
raise SystemExit("missing bilingual docs: " + ", ".join(missing))
|
||||
print("public docs have zh/en file pairs")
|
||||
PY
|
||||
```
|
||||
|
||||
Check that Markdown links do not expose raw filenames as user-facing titles. This should return no matches for polished public docs:
|
||||
|
||||
```bash
|
||||
rg -n "\[[^]]+\.md\]\(" docs/technical/zh docs/technical/en
|
||||
```
|
||||
|
||||
If checking many links, prefer deterministic extraction:
|
||||
|
||||
```bash
|
||||
rg -n "\]\(([^)]+)\)" docs/technical/zh/<doc>.md
|
||||
```
|
||||
|
||||
Also run focused stale-term searches derived from the change, for example:
|
||||
|
||||
```bash
|
||||
rg -n "old label|old route purpose|obsolete provider assumption" docs/technical docs/plans
|
||||
```
|
||||
|
||||
## Hard Constraints
|
||||
|
||||
- A file under `docs/technical/zh/` must not be an English source file copied as a placeholder.
|
||||
- Do not leave a Chinese doc with only an English title and English first-screen content.
|
||||
- When an English counterpart exists in `docs/technical/en/`, never duplicate it byte-for-byte into `docs/technical/zh/`.
|
||||
- Internal links inside `docs/technical/zh/` should point to `docs/technical/zh/...` for Chinese docs, unless intentionally linking to an English-only file.
|
||||
- Public technical documents must be registered in `frontend/src/pages/Docs/docs-content.ts` before considering them available in the Docs UI.
|
||||
- Public technical documents should have both zh and en files with the same filename, unless intentionally exempted.
|
||||
- Markdown link text in public docs should be a readable title, not a raw filename such as `manual.md`.
|
||||
- Do not reference PR numbers, issue numbers, or the current conversation.
|
||||
- Do not write changelog-style lists like "changed A, changed B, changed C" without the constraints and tradeoffs behind those changes.
|
||||
- Keep code snippets concise and relevant.
|
||||
|
||||
## Recommended Output
|
||||
|
||||
After editing, summarize:
|
||||
|
||||
```md
|
||||
Updated:
|
||||
- docs/technical/zh/example.md — what changed
|
||||
|
||||
Verified:
|
||||
- no identical en/zh docs
|
||||
- no language-less docs/technical links in zh docs
|
||||
- public docs are registered in DOCS_METADATA
|
||||
- public docs have zh/en file pairs
|
||||
- no raw `.md` filenames as public link titles
|
||||
```
|
||||
@@ -1,103 +0,0 @@
|
||||
---
|
||||
name: goal-driven
|
||||
description: Run a goal-driven execution loop for very large, long-horizon, rigorously verifiable tasks. Use when the user explicitly wants the lidangzzz/goal-driven method, a master-agent plus worker-agent style workflow, or a persistent loop that keeps working until concrete success criteria are satisfied.
|
||||
---
|
||||
|
||||
# Goal-Driven
|
||||
|
||||
Use this skill when the user wants a strict goal-driven workflow for a hard task with:
|
||||
|
||||
- one clear end goal
|
||||
- explicit success criteria
|
||||
- repeated verification against those criteria
|
||||
- continued execution until the criteria are actually met
|
||||
|
||||
This skill is adapted from `lidangzzz/goal-driven`, but trimmed for local skill use to avoid bloating context.
|
||||
|
||||
## When To Use
|
||||
|
||||
Use it for tasks like:
|
||||
|
||||
- compilers, interpreters, theorem-like proof work, deep refactors
|
||||
- long-running system design or implementation work
|
||||
- problems that are expensive and complex, but still objectively testable
|
||||
|
||||
Do not use it for:
|
||||
|
||||
- vague brainstorming without a success condition
|
||||
- short one-shot edits
|
||||
- tasks where "done" cannot be evaluated in a meaningful way
|
||||
|
||||
## Core Model
|
||||
|
||||
The workflow has two roles:
|
||||
|
||||
1. Master role
|
||||
Defines the goal, defines the success criteria, audits progress, and decides whether the work is actually complete.
|
||||
|
||||
2. Worker role
|
||||
Keeps advancing the task toward the goal. If a result is partial, stalled, or unverifiable, the worker continues.
|
||||
|
||||
In Codex, only use actual subagents when the user explicitly asks for delegation or subagent work and the platform supports it. Otherwise emulate the same loop locally: keep working, checkpointing, and re-verifying until the criteria are satisfied.
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Normalize the task into two blocks:
|
||||
- `Goal`
|
||||
- `Criteria for success`
|
||||
|
||||
2. Make the criteria concrete and testable.
|
||||
Good criteria usually include:
|
||||
- required outputs
|
||||
- required validations or tests
|
||||
- edge cases or coverage thresholds
|
||||
- what evidence proves completion
|
||||
|
||||
3. Break the work into milestones that can each produce evidence.
|
||||
|
||||
4. Execute the next milestone.
|
||||
If subagents are explicitly allowed, the master may delegate bounded worker tasks.
|
||||
If not, do the work locally but keep the master/worker mindset.
|
||||
|
||||
5. Whenever work pauses, stalls, or appears complete, audit against the criteria directly.
|
||||
Check artifacts, tests, logs, diffs, metrics, or other real evidence.
|
||||
|
||||
6. If the criteria are not met, continue with a specific delta:
|
||||
- what is still missing
|
||||
- what evidence failed
|
||||
- what the next worker pass must improve
|
||||
|
||||
7. Stop only when the criteria are met, or when the user explicitly stops the process.
|
||||
|
||||
## Operating Rules
|
||||
|
||||
- Prefer objective checks over self-reported completion.
|
||||
- Prefer deterministic tool evidence over long model summaries: use `rg`, `git diff --stat`, targeted `git diff -- <path>`, tests, builds, linters, `curl`, or database queries when they can prove a criterion.
|
||||
- Do not paste large command output into the conversation; summarize the evidence and keep raw output in tool calls.
|
||||
- Do not confuse progress with completion.
|
||||
- If the worker says "done", verify it.
|
||||
- If verification fails, continue from the gap instead of restarting blindly.
|
||||
- Keep the goal stable unless the user changes it.
|
||||
- Tighten fuzzy criteria before sinking large amounts of effort.
|
||||
|
||||
## Recommended Response Shape
|
||||
|
||||
When starting a goal-driven task, structure the kickoff like this:
|
||||
|
||||
```md
|
||||
Goal
|
||||
- ...
|
||||
|
||||
Criteria for success
|
||||
- ...
|
||||
|
||||
Current plan
|
||||
1. ...
|
||||
2. ...
|
||||
3. ...
|
||||
|
||||
Verification
|
||||
- What evidence will prove completion
|
||||
```
|
||||
|
||||
For a reusable prompt template, read [references/prompt-template.md](references/prompt-template.md).
|
||||
@@ -1,7 +0,0 @@
|
||||
interface:
|
||||
display_name: "Goal-Driven"
|
||||
short_description: "Drive complex work until explicit success criteria are met."
|
||||
default_prompt: "Use $goal-driven to turn this task into a concrete goal, explicit success criteria, and a verification-driven execution loop."
|
||||
|
||||
policy:
|
||||
allow_implicit_invocation: true
|
||||
@@ -1,38 +0,0 @@
|
||||
# Goal-Driven Prompt Template
|
||||
|
||||
Use this when you want a reusable kickoff prompt for a master/worker execution loop.
|
||||
|
||||
```md
|
||||
# Goal-Driven System
|
||||
|
||||
Goal: [[[[[DEFINE THE FINAL GOAL HERE]]]]]
|
||||
|
||||
Criteria for success: [[[[[DEFINE THE SUCCESS CRITERIA HERE]]]]]
|
||||
|
||||
You are the master agent.
|
||||
|
||||
Your job is to:
|
||||
1. Keep the goal and criteria fixed.
|
||||
2. Start worker execution toward the goal.
|
||||
3. Audit any claimed progress against the criteria.
|
||||
4. If the criteria are not met, continue the work with a precise next delta.
|
||||
5. Stop only when the criteria are satisfied or the user explicitly stops the process.
|
||||
|
||||
Worker requirements:
|
||||
1. Break the task into subproblems.
|
||||
2. Keep producing concrete progress toward the goal.
|
||||
3. Report evidence, not just claims.
|
||||
4. Continue until the criteria are satisfied.
|
||||
|
||||
Master audit loop:
|
||||
1. Check whether the worker is still making progress.
|
||||
2. If the worker stalls or claims completion, verify against the criteria.
|
||||
3. If verification fails, resume work from the remaining gap.
|
||||
4. Repeat until the criteria are met.
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- Stronger criteria produce better results than stronger rhetoric.
|
||||
- Prefer measurable checks such as tests, parity checks, generated artifacts, benchmarks, or reviewable outputs.
|
||||
- If the environment does not support subagents, emulate the same loop locally.
|
||||
@@ -18,7 +18,7 @@ Do not use this skill for ordinary commits that are not being released.
|
||||
|
||||
## Versioning Rules
|
||||
|
||||
- `feature` -> bump minor and reset patch to `0` (`x.y.z` → `x.(y+1).0`; for example `0.41.2` → `0.42.0`)
|
||||
- `feature` -> bump `+0.1.0`
|
||||
- `bugfix` -> bump `+0.0.1`
|
||||
- `docs`, `maintenance`, and `refactor` do not bump by default unless the user explicitly wants a release
|
||||
|
||||
@@ -35,19 +35,6 @@ Use `git rev-parse --show-toplevel` to get the repo root. All paths are relative
|
||||
- `docs/CHANGELOG.md`
|
||||
- `docs/version-history.md`
|
||||
|
||||
## Token-Saving Rule
|
||||
|
||||
Release work should be driven by deterministic CLI evidence. Prefer compact commands and targeted file reads:
|
||||
|
||||
```bash
|
||||
git status --short
|
||||
git diff --stat HEAD
|
||||
git diff --name-only HEAD
|
||||
rg -n "version|^## |^Released:|current" VERSION frontend/package.json pyproject.toml docs/CHANGELOG.md docs/version-history.md
|
||||
```
|
||||
|
||||
Do not inspect full diffs unless deciding whether changed code belongs in the release.
|
||||
|
||||
## Workflow
|
||||
|
||||
### Step 1 — Environment check
|
||||
@@ -65,10 +52,8 @@ If unrelated uncommitted changes exist, list them and ask the user whether to in
|
||||
### Step 2 — Determine release type and next version
|
||||
|
||||
- If the user provided an explicit type (`feature` / `bugfix`), use it
|
||||
- Otherwise infer from `git diff --stat HEAD`, `git diff --name-only HEAD`, focused diffs for changed code, and recent `git log`
|
||||
- Compute the next version:
|
||||
- `feature`: increment minor and reset patch to `0` (e.g. `0.41.2` → `0.42.0`)
|
||||
- `bugfix`: increment patch only (e.g. `0.26.2` → `0.26.3`)
|
||||
- Otherwise infer from `git diff HEAD` and recent `git log`
|
||||
- Compute the next version (e.g. `0.26.2` → bugfix → `0.26.3`)
|
||||
- **Show the release plan before making any changes:**
|
||||
|
||||
```
|
||||
@@ -119,13 +104,12 @@ Get today's date with `date +%Y-%m-%d`.
|
||||
|
||||
Run the smallest relevant validation for the changes in scope:
|
||||
|
||||
- Python files changed: list changed Python files with `git diff --name-only HEAD -- '*.py'`, then run `python3 -m py_compile <changed_files>`
|
||||
- Frontend files changed: list changed frontend files with `git diff --name-only HEAD -- frontend`, then run the project-standard check if available; otherwise skip and say so
|
||||
- Python files changed: `python3 -m py_compile <changed_files>`
|
||||
- Frontend files changed: run the project-standard check if available; otherwise skip and say so
|
||||
- Version consistency: confirm VERSION, package.json, pyproject.toml, and uv.lock all show the same version
|
||||
|
||||
```bash
|
||||
cat VERSION
|
||||
rg -n "\"version\":|^version =|version = " frontend/package.json pyproject.toml uv.lock
|
||||
grep -h "version" VERSION frontend/package.json pyproject.toml
|
||||
```
|
||||
|
||||
### Step 7 — Pre-commit preview
|
||||
|
||||
@@ -1,13 +0,0 @@
|
||||
**
|
||||
|
||||
!pyproject.toml
|
||||
!uv.lock
|
||||
!aiprovider/
|
||||
!aiprovider/**
|
||||
|
||||
aiprovider/.env
|
||||
aiprovider/.env.*
|
||||
!aiprovider/.env.example
|
||||
**/__pycache__/
|
||||
**/*.pyc
|
||||
**/*.pyo
|
||||
-2
@@ -1,5 +1,3 @@
|
||||
> Archived note: this document was originally created by sisyphus and later reviewed against the main docs set. Useful content has been absorbed into `docs/plans/` where appropriate.
|
||||
|
||||
# 地球3D可视化架构重构计划
|
||||
|
||||
## 背景
|
||||
@@ -1,5 +1,3 @@
|
||||
> Archived note: this document was originally created by sisyphus and later reviewed against the main docs set. Useful content has been absorbed into `docs/plans/` where appropriate.
|
||||
|
||||
# 卫星预测轨道显示功能
|
||||
|
||||
## TL;DR
|
||||
@@ -1,5 +1,3 @@
|
||||
> Archived note: this document was originally created by sisyphus and later reviewed against the main docs set. Useful content has been absorbed into `docs/plans/` where appropriate.
|
||||
|
||||
# UE5 3D 大屏客户端开发计划
|
||||
|
||||
## 项目概述
|
||||
-2
@@ -1,5 +1,3 @@
|
||||
> Archived note: this document was originally created by sisyphus and later reviewed against the main docs set. Useful content has been absorbed into `docs/plans/` where appropriate.
|
||||
|
||||
# WebGL Instancing 卫星渲染优化计划
|
||||
|
||||
## 背景
|
||||
@@ -227,120 +227,6 @@ bun run build
|
||||
|
||||
启动服务后访问: `http://localhost:8000/docs`
|
||||
|
||||
## WSL / Windows 局域网访问
|
||||
|
||||
如果服务运行在 WSL 中,而你希望:
|
||||
|
||||
- Windows 本机浏览器访问开发服务
|
||||
- 同一局域网内的手机或其他电脑访问开发服务
|
||||
|
||||
推荐按下面顺序排查和配置。
|
||||
|
||||
### 1. 在 WSL 中启动服务
|
||||
|
||||
```bash
|
||||
./planet.sh start --allow-lan
|
||||
```
|
||||
|
||||
这会让前端监听 `0.0.0.0:3000`,后端监听 `0.0.0.0:8000`。
|
||||
|
||||
### 2. 先确认 WSL 内部服务正常
|
||||
|
||||
在 WSL 中执行:
|
||||
|
||||
```bash
|
||||
curl http://localhost:3000
|
||||
curl http://localhost:8000/health
|
||||
ss -ltnp | grep -E ':3000|:8000'
|
||||
```
|
||||
|
||||
预期:
|
||||
|
||||
- `3000` 返回前端 HTML
|
||||
- `8000/health` 返回健康检查 JSON
|
||||
- `ss` 中能看到 `0.0.0.0:3000` 和 `0.0.0.0:8000`
|
||||
|
||||
如果这一步不通,先不要继续做 Windows 转发。
|
||||
|
||||
### 3. 在 Windows 本机验证 localhost 直通
|
||||
|
||||
在 Windows PowerShell 中执行:
|
||||
|
||||
```powershell
|
||||
curl http://localhost:3000
|
||||
curl http://localhost:8000/health
|
||||
```
|
||||
|
||||
在常见的 WSL2 开发环境下,Windows 通常可以直接通过 `localhost` 访问 WSL 中的服务。
|
||||
|
||||
### 4. 如果需要让局域网设备访问,再做 Windows 端口转发
|
||||
|
||||
注意:下面的命令必须在“以管理员身份运行”的 PowerShell 中执行。
|
||||
|
||||
先把 Windows 对外网卡上的 `3000` / `8000` 转发到 Windows 本机 `127.0.0.1`:
|
||||
|
||||
```powershell
|
||||
netsh interface portproxy delete v4tov4 listenaddress=0.0.0.0 listenport=3000
|
||||
netsh interface portproxy delete v4tov4 listenaddress=0.0.0.0 listenport=8000
|
||||
|
||||
netsh interface portproxy add v4tov4 listenaddress=0.0.0.0 listenport=3000 connectaddress=127.0.0.1 connectport=3000
|
||||
netsh interface portproxy add v4tov4 listenaddress=0.0.0.0 listenport=8000 connectaddress=127.0.0.1 connectport=8000
|
||||
```
|
||||
|
||||
再放行 Windows 防火墙:
|
||||
|
||||
```powershell
|
||||
New-NetFirewallRule -DisplayName "WSL Planet 3000" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 3000
|
||||
New-NetFirewallRule -DisplayName "WSL Planet 8000" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 8000
|
||||
```
|
||||
|
||||
检查转发规则是否生效:
|
||||
|
||||
```powershell
|
||||
netsh interface portproxy show all
|
||||
```
|
||||
|
||||
预期能看到:
|
||||
|
||||
- `0.0.0.0:3000 -> 127.0.0.1:3000`
|
||||
- `0.0.0.0:8000 -> 127.0.0.1:8000`
|
||||
|
||||
### 5. 查 Windows 局域网 IP,并让其他设备访问
|
||||
|
||||
在 Windows PowerShell 中执行:
|
||||
|
||||
```powershell
|
||||
ipconfig
|
||||
```
|
||||
|
||||
找到当前联网网卡的 IPv4 地址,例如 `192.168.8.228`。
|
||||
|
||||
局域网其他设备可访问:
|
||||
|
||||
- `http://<Windows局域网IP>:3000/earth`
|
||||
- `http://<Windows局域网IP>:3000/admin`
|
||||
|
||||
例如:
|
||||
|
||||
- `http://192.168.8.228:3000/earth`
|
||||
|
||||
### 6. 常见现象与判断
|
||||
|
||||
- WSL 中 `curl localhost:3000` 能通,但 Windows 访问 `WSL 的局域网 IP:3000` 不通:这是正常现象之一,优先验证 Windows 的 `localhost:3000`
|
||||
- Windows `localhost:3000` 能通,但局域网设备访问 `Windows 局域网 IP:3000` 不通:通常缺少 `portproxy` 或防火墙放行
|
||||
- `whoami /groups` 中 `S-1-5-32-544` 显示 `deny only`:说明当前 PowerShell 不是提权管理员窗口
|
||||
|
||||
### 7. 本项目一次性验证顺序
|
||||
|
||||
建议固定按这个顺序验证:
|
||||
|
||||
1. WSL 中执行 `curl http://localhost:3000`
|
||||
2. WSL 中执行 `curl http://localhost:8000/health`
|
||||
3. Windows 中执行 `curl http://localhost:3000`
|
||||
4. Windows 中执行 `curl http://localhost:8000/health`
|
||||
5. 管理员 PowerShell 配置 `portproxy` 和防火墙
|
||||
6. 用手机或其他电脑访问 `http://<Windows局域网IP>:3000/earth`
|
||||
|
||||
## 启动容错参数
|
||||
|
||||
`planet.sh` 现在为依赖安装、数据库、AI Provider 启动加入了有限次重试,并会在数据库与 `aiprovider` 启动后额外等待 Docker healthcheck。
|
||||
@@ -442,11 +328,11 @@ AI_PROVIDER_SERVICE_TOKEN=change_me
|
||||
|
||||
详细文档:
|
||||
|
||||
- [docs/technical/agents-aiprovider.md](/home/ray/dev/linkong/planet/docs/technical/agents-aiprovider.md)
|
||||
- [docs/aiprovider.md](/home/ray/dev/linkong/planet/docs/aiprovider.md)
|
||||
- [aiprovider/README.md](/home/ray/dev/linkong/planet/aiprovider/README.md)
|
||||
- [docs/technical/frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md)
|
||||
- [docs/plans/frontend-ai-playground-development-plan.md](/home/ray/dev/linkong/planet/docs/plans/frontend-ai-playground-development-plan.md)
|
||||
- [docs/plans/agents-situational-awareness-foundation-plan.md](/home/ray/dev/linkong/planet/docs/plans/agents-situational-awareness-foundation-plan.md)
|
||||
- [docs/frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/frontend-layout-guidelines.md)
|
||||
- [docs/ai-playground-development-plan.md](/home/ray/dev/linkong/planet/docs/ai-playground-development-plan.md)
|
||||
- [docs/situational-awareness-foundation-plan.md](/home/ray/dev/linkong/planet/docs/situational-awareness-foundation-plan.md)
|
||||
|
||||
## 前端页面布局规范
|
||||
|
||||
@@ -460,7 +346,7 @@ AI_PROVIDER_SERVICE_TOKEN=change_me
|
||||
当前推荐参考实现:
|
||||
|
||||
- [frontend/src/pages/BGP/BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx)
|
||||
- [docs/technical/frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md)
|
||||
- [docs/frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/frontend-layout-guidelines.md)
|
||||
|
||||
## License
|
||||
|
||||
|
||||
@@ -20,19 +20,3 @@
|
||||
- [x] 在 activity layer 之后继续补 `route leak` 和 `path instability / flap` detector
|
||||
- [ ] 对 [frontend/public/earth/js/bgp.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp.js) 做按职责拆分的小重构,拆成 data / markers / overlays / animation,降低后续维护复杂度
|
||||
- [ ] 可选优化(非必做):将 BGP incident/collector 标点改为 HTML marker(参考 worldmonitor 的 `htmlElementsData` 思路),实现近乎固定屏幕尺寸与更高密度可点击性
|
||||
- [ ] 保持 Earth 当前这批纯个人偏好设置继续走本地持久化:`旋转模式`、HUD 面板显示/隐藏、`地形透明度` 暂不升级到后端系统设置,避免把设备级偏好过早做成全局配置
|
||||
- [ ] 如果后续明确需要“账号级同步 Earth 偏好”,再单独设计 `Earth user preferences`:优先按用户维度而不是全局系统设置保存,并规划 `localStorage -> backend` 的平滑迁移策略
|
||||
- [ ] 为 Planet / Earth 补一个可用的日志查看系统:先明确前后端/AI Provider/采集任务的日志入口、最近日志聚合、筛选与 tail 能力,再决定是先做脚本级统一入口还是控制台内置日志面板
|
||||
- [ ] 重写控制台 UI,逐步抛弃 Ant Design,建立自有组件体系,并统一采用 `tabler.io` / Tabler Icons 作为控制台主图标库
|
||||
- [ ] 把 Earth 态势新闻源从 [earth_news.py](/home/ray/dev/linkong/planet/backend/app/services/earth_news.py) 的硬编码列表抽成可配置目录,优先保持当前“实时聚合”链路不变,只先解决新闻源不可配置的问题
|
||||
- [ ] 为 Earth 态势新闻设计后续采集器化方案:明确新闻数据模型、去重策略、区域映射、过期清理和 Earth/AI 复用方式,再决定何时把新闻从实时抓取升级成正式 collector
|
||||
- [ ] 为 Earth 地球表面增加一层与基础纹理对齐的材质/纹理 overlay,并在同层叠加国界轮廓参考线;要求国界线与底图稳定对齐,且 hover 到国家轮廓时能高亮当前国家,便于校准地表和增强交互
|
||||
- [ ] 把 Earth 新闻接入通用巡航队列:按新闻发生地和时间排序生成巡航目标,巡航聚焦到新闻事件时显示对应新闻卡片,并保持实现边界为“通用巡航层 + 新闻业务适配层”,不要再把新闻逻辑直接耦合回 `main.js` 状态机
|
||||
- [ ] 为未知位置的算力中心建立分层坐标补全链路:优先 `精确坐标 > 站点/园区命中 > 城市 > 州/省 > 国家内主要算力城市 > 国家质心`,并把每次回退的 `confidence / reason / precision` 明确写进统一 GeoJSON
|
||||
- [ ] 为算力中心补一份可维护的本地位置注册表,例如 `canonical_name / aliases / operator / country / region / city / lat / lon / confidence / source_note`,避免把地点知识长期硬编码在 `visualization.py`
|
||||
- [ ] 增强 `epoch_ai_gpu` 和相关算力采集器的源页面解析:即使公开 API 不给坐标,也继续尝试从详情页、HTML、内嵌 JSON、schema.org、OpenGraph、脚本变量和 PDF/新闻稿链接里抽地点线索
|
||||
- [ ] 为未知位置算力中心增加外部富化策略评估:可选接入公开知识源或搜索兜底,只抓“站点名/园区名/城市名”级别线索,不直接抓经纬度结论,并把结果作为候选证据而不是真值
|
||||
- [ ] 为算力中心建立 `operator / cluster name / facility alias` 归一化层,先解决 `xAI / Colossus / Memphis`、`OpenAI / Stargate`、`CoreWeave`、`Lambda`、`Crusoe` 这类同一对象多种写法导致的地点匹配失败
|
||||
- [ ] 为估算位置增加更细的视觉和产品表达:除了问号角标,还要支持 tooltip/详情中的“估算依据”“精度级别”“最后核验时间”,并允许在设置中单独开关“仅看精确位置”
|
||||
- [ ] 为国家级估算点设计更合理的落点策略:优先落在“该国主要算力/数据中心城市候选集”而不是几何质心,必要时同国多节点做稳定散列分配,避免大量节点堆在荒漠或海上
|
||||
- [ ] 为未知位置算力中心建立人工校验工作流:支持导出待核验清单、记录人工确认结果,并把人工确认反哺到位置注册表,逐步减少问号点比例
|
||||
+4
-11
@@ -1,12 +1,6 @@
|
||||
# syntax=docker/dockerfile:1.7
|
||||
FROM python:3.14-slim
|
||||
|
||||
ARG PYTHON_IMAGE=python:3.14-slim
|
||||
ARG UV_IMAGE=ghcr.io/astral-sh/uv:latest
|
||||
|
||||
FROM ${UV_IMAGE} AS uv
|
||||
FROM ${PYTHON_IMAGE}
|
||||
|
||||
COPY --from=uv /uv /uvx /bin/
|
||||
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
@@ -20,10 +14,9 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
COPY pyproject.toml uv.lock /app/
|
||||
RUN --mount=type=cache,target=/root/.cache/uv \
|
||||
uv sync --frozen --no-dev
|
||||
RUN uv sync --frozen --no-dev
|
||||
|
||||
COPY aiprovider /app/aiprovider
|
||||
COPY . /app
|
||||
|
||||
EXPOSE 8010
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
|
||||
完整使用说明见:
|
||||
|
||||
- [docs/agents/aiprovider.md](/home/ray/dev/linkong/planet/docs/agents/aiprovider.md)
|
||||
- [docs/aiprovider.md](/home/ray/dev/linkong/planet/docs/aiprovider.md)
|
||||
|
||||
当前支持:
|
||||
|
||||
|
||||
+2
-20
@@ -37,26 +37,8 @@ def verify_service_token(x_provider_token: str | None = Header(default=None)) ->
|
||||
)
|
||||
|
||||
|
||||
def get_provider_service(
|
||||
x_ai_provider: str | None = Header(default=None),
|
||||
x_ai_provider_api: str | None = Header(default=None),
|
||||
x_ai_base_url: str | None = Header(default=None),
|
||||
x_ai_api_key: str | None = Header(default=None),
|
||||
x_ai_model: str | None = Header(default=None),
|
||||
x_ai_max_tokens: str | None = Header(default=None),
|
||||
x_ai_anthropic_version: str | None = Header(default=None),
|
||||
) -> ProviderService:
|
||||
overrides = {
|
||||
"provider": x_ai_provider,
|
||||
"provider_api": x_ai_provider_api,
|
||||
"base_url": x_ai_base_url,
|
||||
"api_key": x_ai_api_key,
|
||||
"model": x_ai_model,
|
||||
"anthropic_version": x_ai_anthropic_version,
|
||||
}
|
||||
if x_ai_max_tokens:
|
||||
overrides["max_tokens"] = x_ai_max_tokens
|
||||
return ProviderService({key: value for key, value in overrides.items() if value not in (None, "")})
|
||||
def get_provider_service() -> ProviderService:
|
||||
return ProviderService()
|
||||
|
||||
|
||||
@app.get("/health")
|
||||
|
||||
@@ -46,22 +46,19 @@ def _resolve_provider_api(provider: str, configured_api: str) -> str:
|
||||
|
||||
|
||||
class ProviderService:
|
||||
def __init__(self, overrides: dict[str, Any] | None = None) -> None:
|
||||
overrides = overrides or {}
|
||||
self.provider = _normalize_provider(overrides.get("provider") or settings.AI_PROVIDER)
|
||||
def __init__(self) -> None:
|
||||
self.provider = _normalize_provider(settings.AI_PROVIDER)
|
||||
self.provider_api = _resolve_provider_api(
|
||||
self.provider,
|
||||
_normalize_provider_api(overrides.get("provider_api") or settings.AI_PROVIDER_API),
|
||||
_normalize_provider_api(settings.AI_PROVIDER_API),
|
||||
)
|
||||
self.base_url = str(overrides.get("base_url") or settings.AI_BASE_URL).rstrip("/")
|
||||
self.api_key = str(overrides.get("api_key") or settings.AI_API_KEY)
|
||||
self.default_model = str(overrides.get("model") or settings.AI_MODEL)
|
||||
self.base_url = settings.AI_BASE_URL.rstrip("/")
|
||||
self.api_key = settings.AI_API_KEY
|
||||
self.default_model = settings.AI_MODEL
|
||||
self.timeout = settings.AI_TIMEOUT_SECONDS
|
||||
self.http_retry_attempts = max(settings.AI_HTTP_RETRY_ATTEMPTS, 1)
|
||||
self.max_tokens = int(overrides.get("max_tokens") or settings.AI_MAX_TOKENS)
|
||||
self.anthropic_version = str(
|
||||
overrides.get("anthropic_version") or settings.AI_ANTHROPIC_VERSION
|
||||
)
|
||||
self.max_tokens = settings.AI_MAX_TOKENS
|
||||
self.anthropic_version = settings.AI_ANTHROPIC_VERSION
|
||||
self.system_prompt = settings.AI_ANALYSIS_SYSTEM_PROMPT
|
||||
|
||||
def get_status(self) -> AIProviderStatusResponse:
|
||||
|
||||
+2
-6
@@ -1,10 +1,6 @@
|
||||
ARG PYTHON_IMAGE=python:3.14-slim
|
||||
ARG UV_IMAGE=ghcr.io/astral-sh/uv:latest
|
||||
FROM python:3.14-slim
|
||||
|
||||
FROM ${UV_IMAGE} AS uv
|
||||
FROM ${PYTHON_IMAGE}
|
||||
|
||||
COPY --from=uv /uv /uvx /bin/
|
||||
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
|
||||
@@ -13,9 +13,9 @@ from app.api.v1 import (
|
||||
collected_data,
|
||||
visualization,
|
||||
bgp,
|
||||
news,
|
||||
system_control,
|
||||
tv,
|
||||
ue_data,
|
||||
)
|
||||
|
||||
api_router = APIRouter()
|
||||
@@ -36,4 +36,4 @@ api_router.include_router(system_control.router, prefix="/system", tags=["system
|
||||
api_router.include_router(visualization.router, prefix="/visualization", tags=["visualization"])
|
||||
api_router.include_router(bgp.router, prefix="/bgp", tags=["bgp"])
|
||||
api_router.include_router(tv.router, prefix="/tv", tags=["tv"])
|
||||
api_router.include_router(news.router, prefix="/news", tags=["news"])
|
||||
api_router.include_router(ue_data.router, prefix="/ue", tags=["ue-client"])
|
||||
@@ -1,41 +1,20 @@
|
||||
"""DataSourceConfig API for user-defined data sources"""
|
||||
|
||||
from typing import Any, Optional
|
||||
from typing import Optional
|
||||
from datetime import datetime
|
||||
import base64
|
||||
import json
|
||||
import re
|
||||
from fastapi import APIRouter, Depends, HTTPException, status
|
||||
from sqlalchemy import select, func
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from pydantic import BaseModel, Field
|
||||
import httpx
|
||||
|
||||
from app.core.target_schema_registry import get_target_schema, list_target_schemas
|
||||
from app.core.datasource_defaults import DEFAULT_DATASOURCES
|
||||
from app.db.session import get_db
|
||||
from app.models.user import User
|
||||
from app.models.datasource_config import DataSourceConfig
|
||||
from app.models.datasource_mapping import DataSourceMappingTemplate
|
||||
from app.core.security import get_current_user
|
||||
from app.core.cache import cache
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.schemas.ai import SituationalAnalysisRequest
|
||||
from app.services.ai_client import AIProviderClient, get_ai_provider_client
|
||||
from app.services.datasource_mapping import (
|
||||
MappingError,
|
||||
build_heuristic_mapping,
|
||||
execute_mapping,
|
||||
persist_mapped_records,
|
||||
redact_for_llm,
|
||||
stable_payload_hash,
|
||||
)
|
||||
from app.services.datasource_connectivity import (
|
||||
get_builtin_connection_status,
|
||||
save_connectivity_success,
|
||||
strip_connectivity_validation,
|
||||
test_builtin_connectivity,
|
||||
)
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
@@ -80,70 +59,6 @@ class DataSourceConfigResponse(BaseModel):
|
||||
from_attributes = True
|
||||
|
||||
|
||||
def _is_builtin_config_name(name: str | None) -> bool:
|
||||
return bool(name and name in DEFAULT_DATASOURCES)
|
||||
|
||||
|
||||
async def _ensure_builtin_connection_verified(
|
||||
db: AsyncSession,
|
||||
config_data: DataSourceConfigCreate,
|
||||
) -> None:
|
||||
if not _is_builtin_config_name(config_data.name):
|
||||
return
|
||||
|
||||
status_result = await get_builtin_connection_status(
|
||||
db,
|
||||
config_data.name,
|
||||
config_data.endpoint,
|
||||
config_data.auth_type,
|
||||
config_data.headers,
|
||||
config_data.config,
|
||||
)
|
||||
if not status_result.get("connected"):
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=status_result.get("message") or "请先完成连接验证,再保存内置采集器配置。",
|
||||
)
|
||||
|
||||
|
||||
class CustomSampleRequest(BaseModel):
|
||||
datasource_config_id: Optional[int] = None
|
||||
config: Optional[DataSourceConfigCreate] = None
|
||||
limit_bytes: int = Field(default=200000, ge=1000, le=1000000)
|
||||
|
||||
|
||||
class MappingProposeRequest(BaseModel):
|
||||
sample_payload: Any
|
||||
target_schema: str
|
||||
use_ai: bool = True
|
||||
|
||||
|
||||
class MappingPreviewRequest(BaseModel):
|
||||
sample_payload: Any
|
||||
target_schema: str
|
||||
mapping_json: dict
|
||||
limit: int = Field(default=20, ge=1, le=100)
|
||||
|
||||
|
||||
class MappingTemplateCreate(BaseModel):
|
||||
datasource_config_id: int
|
||||
target_schema: str
|
||||
mapping_json: dict
|
||||
sample_payload: Any | None = None
|
||||
sample_payload_hash: Optional[str] = None
|
||||
validation_status: str = Field(default="draft", pattern="^(draft|valid|invalid)$")
|
||||
is_active: bool = False
|
||||
|
||||
|
||||
class MappingTemplateUpdate(BaseModel):
|
||||
target_schema: Optional[str] = None
|
||||
mapping_json: Optional[dict] = None
|
||||
sample_payload: Any | None = None
|
||||
sample_payload_hash: Optional[str] = None
|
||||
validation_status: Optional[str] = Field(default=None, pattern="^(draft|valid|invalid)$")
|
||||
is_active: Optional[bool] = None
|
||||
|
||||
|
||||
async def test_endpoint(
|
||||
endpoint: str,
|
||||
auth_type: str,
|
||||
@@ -181,134 +96,6 @@ async def test_endpoint(
|
||||
}
|
||||
|
||||
|
||||
def _build_request_headers(auth_type: str, auth_config: dict, headers: dict) -> dict[str, str]:
|
||||
request_headers = {str(key): str(value) for key, value in (headers or {}).items()}
|
||||
auth_type = str(auth_type or "none").lower()
|
||||
auth_config = auth_config or {}
|
||||
|
||||
if auth_type == "bearer" and auth_config.get("token"):
|
||||
request_headers["Authorization"] = f"Bearer {auth_config['token']}"
|
||||
elif auth_type == "api_key" and auth_config.get("api_key"):
|
||||
location = str(auth_config.get("in") or auth_config.get("location") or "header").lower()
|
||||
if location != "query":
|
||||
key_name = auth_config.get("key_name", "X-API-Key")
|
||||
request_headers[str(key_name)] = str(auth_config["api_key"])
|
||||
elif auth_type == "basic":
|
||||
username = auth_config.get("username", "")
|
||||
password = auth_config.get("password", "")
|
||||
credentials = f"{username}:{password}"
|
||||
encoded = base64.b64encode(credentials.encode()).decode()
|
||||
request_headers["Authorization"] = f"Basic {encoded}"
|
||||
return request_headers
|
||||
|
||||
|
||||
def _build_query_params(auth_type: str, auth_config: dict, config: dict) -> dict[str, Any]:
|
||||
params = {}
|
||||
candidate = (config or {}).get("params") or (config or {}).get("query_params")
|
||||
if isinstance(candidate, dict):
|
||||
params.update(candidate)
|
||||
|
||||
auth_type = str(auth_type or "none").lower()
|
||||
auth_config = auth_config or {}
|
||||
if auth_type == "api_key" and auth_config.get("api_key"):
|
||||
location = str(auth_config.get("in") or auth_config.get("location") or "header").lower()
|
||||
if location == "query":
|
||||
key_name = auth_config.get("key_name") or auth_config.get("param_name") or "api_key"
|
||||
params[str(key_name)] = auth_config["api_key"]
|
||||
return params
|
||||
|
||||
|
||||
async def fetch_custom_sample_from_config(config: DataSourceConfig, limit_bytes: int) -> Any:
|
||||
request_config = config.config or {}
|
||||
method = str(request_config.get("method") or request_config.get("request_method") or "GET").upper()
|
||||
if method not in {"GET", "POST"}:
|
||||
raise HTTPException(status_code=400, detail="Only GET and POST sample requests are supported.")
|
||||
|
||||
headers = _build_request_headers(config.auth_type, config.auth_config or {}, config.headers or {})
|
||||
params = _build_query_params(config.auth_type, config.auth_config or {}, request_config)
|
||||
timeout = float(request_config.get("timeout", 30))
|
||||
json_body = request_config.get("json_body")
|
||||
if json_body is None and str(request_config.get("body_type") or "").lower() in {"json", ""}:
|
||||
candidate = request_config.get("body")
|
||||
if isinstance(candidate, (dict, list)):
|
||||
json_body = candidate
|
||||
|
||||
async with httpx.AsyncClient(timeout=timeout, follow_redirects=True) as client:
|
||||
response = await client.request(
|
||||
method,
|
||||
config.endpoint,
|
||||
headers=headers,
|
||||
params=params or None,
|
||||
json=json_body,
|
||||
)
|
||||
response.raise_for_status()
|
||||
content = response.content[:limit_bytes]
|
||||
if "application/json" in response.headers.get("content-type", ""):
|
||||
return json.loads(content.decode(response.encoding or "utf-8"))
|
||||
return {"text": content.decode(response.encoding or "utf-8", errors="replace")}
|
||||
|
||||
|
||||
def _parse_mapping_from_ai_text(content: str) -> dict[str, Any] | None:
|
||||
if not content:
|
||||
return None
|
||||
|
||||
candidates = [content]
|
||||
fenced = re.findall(r"```(?:json)?\s*(\{.*?\})\s*```", content, flags=re.DOTALL)
|
||||
candidates = fenced + candidates
|
||||
for candidate in candidates:
|
||||
try:
|
||||
parsed = json.loads(candidate)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
if isinstance(parsed, dict) and isinstance(parsed.get("fields"), dict):
|
||||
return parsed
|
||||
return None
|
||||
|
||||
|
||||
async def _get_config_for_sample(
|
||||
payload: CustomSampleRequest,
|
||||
db: AsyncSession,
|
||||
) -> DataSourceConfig:
|
||||
if payload.datasource_config_id is not None:
|
||||
result = await db.execute(
|
||||
select(DataSourceConfig).where(DataSourceConfig.id == payload.datasource_config_id)
|
||||
)
|
||||
config = result.scalar_one_or_none()
|
||||
if not config:
|
||||
raise HTTPException(status_code=404, detail="Configuration not found")
|
||||
return config
|
||||
|
||||
if payload.config is None:
|
||||
raise HTTPException(status_code=400, detail="datasource_config_id or config is required")
|
||||
|
||||
config_data = payload.config
|
||||
return DataSourceConfig(
|
||||
name=config_data.name,
|
||||
description=config_data.description,
|
||||
source_type=config_data.source_type,
|
||||
endpoint=config_data.endpoint,
|
||||
auth_type=config_data.auth_type,
|
||||
auth_config=config_data.auth_config,
|
||||
headers=config_data.headers,
|
||||
config=config_data.config,
|
||||
)
|
||||
|
||||
|
||||
def serialize_mapping_template(template: DataSourceMappingTemplate) -> dict[str, Any]:
|
||||
return {
|
||||
"id": template.id,
|
||||
"datasource_config_id": template.datasource_config_id,
|
||||
"target_schema": template.target_schema,
|
||||
"mapping_json": template.mapping_json,
|
||||
"sample_payload_hash": template.sample_payload_hash,
|
||||
"validation_status": template.validation_status,
|
||||
"version": template.version,
|
||||
"is_active": template.is_active,
|
||||
"created_at": to_iso8601_utc(template.created_at),
|
||||
"updated_at": to_iso8601_utc(template.updated_at),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/configs")
|
||||
async def list_configs(
|
||||
active_only: bool = False,
|
||||
@@ -345,47 +132,6 @@ async def list_configs(
|
||||
}
|
||||
|
||||
|
||||
@router.get("/configs/all")
|
||||
async def list_all_datasources(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""List all data sources: YAML defaults + DB overrides"""
|
||||
from app.core.data_sources import COLLECTOR_URL_KEYS, get_data_sources_config
|
||||
|
||||
config = get_data_sources_config()
|
||||
|
||||
db_query = await db.execute(select(DataSourceConfig))
|
||||
db_configs = {c.name: c for c in db_query.scalars().all()}
|
||||
|
||||
result = []
|
||||
for name, yaml_key in COLLECTOR_URL_KEYS.items():
|
||||
yaml_url = config.get_yaml_url(name)
|
||||
db_config = db_configs.get(name)
|
||||
|
||||
result.append(
|
||||
{
|
||||
"name": name,
|
||||
"default_url": yaml_url,
|
||||
"endpoint": db_config.endpoint if db_config else yaml_url,
|
||||
"is_overridden": db_config is not None and db_config.endpoint != yaml_url
|
||||
if yaml_url
|
||||
else db_config is not None,
|
||||
"is_active": db_config.is_active if db_config else True,
|
||||
"source_type": db_config.source_type if db_config else "http",
|
||||
"auth_type": db_config.auth_type if db_config else "none",
|
||||
"headers": db_config.headers if db_config else {},
|
||||
"config": strip_connectivity_validation(db_config.config if db_config else {}),
|
||||
"config_id": db_config.id if db_config else None,
|
||||
"description": db_config.description
|
||||
if db_config
|
||||
else f"Data source from YAML: {yaml_key}",
|
||||
}
|
||||
)
|
||||
|
||||
return {"total": len(result), "data": result}
|
||||
|
||||
|
||||
@router.get("/configs/{config_id}")
|
||||
async def get_config(
|
||||
config_id: int,
|
||||
@@ -430,7 +176,7 @@ async def create_config(
|
||||
auth_type=config_data.auth_type,
|
||||
auth_config=config_data.auth_config,
|
||||
headers=config_data.headers,
|
||||
config=strip_connectivity_validation(config_data.config),
|
||||
config=config_data.config,
|
||||
)
|
||||
|
||||
db.add(config)
|
||||
@@ -462,8 +208,6 @@ async def update_config(
|
||||
|
||||
update_data = config_data.model_dump(exclude_unset=True)
|
||||
for field, value in update_data.items():
|
||||
if field == "config":
|
||||
value = strip_connectivity_validation(value)
|
||||
setattr(config, field, value)
|
||||
|
||||
await db.commit()
|
||||
@@ -566,366 +310,38 @@ async def test_new_config(
|
||||
}
|
||||
|
||||
|
||||
@router.post("/configs/builtin/connection-status")
|
||||
async def get_builtin_config_connection_status(
|
||||
config_data: DataSourceConfigCreate,
|
||||
@router.get("/configs/all")
|
||||
async def list_all_datasources(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
if not _is_builtin_config_name(config_data.name):
|
||||
raise HTTPException(status_code=400, detail="Only built-in datasource configs are supported.")
|
||||
"""List all data sources: YAML defaults + DB overrides"""
|
||||
from app.core.data_sources import COLLECTOR_URL_KEYS, get_data_sources_config
|
||||
|
||||
return await get_builtin_connection_status(
|
||||
db,
|
||||
config_data.name,
|
||||
config_data.endpoint,
|
||||
config_data.auth_type,
|
||||
config_data.headers,
|
||||
config_data.config,
|
||||
)
|
||||
config = get_data_sources_config()
|
||||
|
||||
db_query = await db.execute(select(DataSourceConfig))
|
||||
db_configs = {c.name: c for c in db_query.scalars().all()}
|
||||
|
||||
@router.post("/configs/builtin/connect")
|
||||
async def connect_builtin_config(
|
||||
config_data: DataSourceConfigCreate,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
if not _is_builtin_config_name(config_data.name):
|
||||
raise HTTPException(status_code=400, detail="Only built-in datasource configs are supported.")
|
||||
result = []
|
||||
for name, yaml_key in COLLECTOR_URL_KEYS.items():
|
||||
yaml_url = config.get_yaml_url(name)
|
||||
db_config = db_configs.get(name)
|
||||
|
||||
result = await test_builtin_connectivity(
|
||||
config_data.name,
|
||||
config_data.endpoint,
|
||||
config_data.auth_type,
|
||||
config_data.headers,
|
||||
config_data.config,
|
||||
db,
|
||||
)
|
||||
if result.get("success") and result.get("checksum"):
|
||||
validation = await save_connectivity_success(
|
||||
db,
|
||||
config_data.name,
|
||||
result["checksum"],
|
||||
result,
|
||||
connected_by="connection_button",
|
||||
)
|
||||
await db.commit()
|
||||
return {
|
||||
**result,
|
||||
"connected": True,
|
||||
"validation": validation,
|
||||
}
|
||||
|
||||
return {
|
||||
**result,
|
||||
"connected": False,
|
||||
}
|
||||
|
||||
|
||||
@router.post("/custom/sample")
|
||||
async def fetch_custom_sample(
|
||||
payload: CustomSampleRequest,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""Fetch a sample payload for a saved or draft custom data source."""
|
||||
config = await _get_config_for_sample(payload, db)
|
||||
try:
|
||||
sample = await fetch_custom_sample_from_config(config, payload.limit_bytes)
|
||||
except httpx.HTTPStatusError as exc:
|
||||
raise HTTPException(
|
||||
status_code=exc.response.status_code,
|
||||
detail=f"Sample request failed: HTTP {exc.response.status_code}",
|
||||
) from exc
|
||||
except httpx.HTTPError as exc:
|
||||
raise HTTPException(status_code=502, detail=f"Sample request failed: {exc}") from exc
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"sample_payload": sample,
|
||||
"sample_payload_hash": stable_payload_hash(sample),
|
||||
"redacted_preview": redact_for_llm(sample),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/target-schemas")
|
||||
async def get_datasource_target_schemas(
|
||||
current_user: User = Depends(get_current_user),
|
||||
):
|
||||
"""List target schemas available for custom datasource mapping."""
|
||||
return {"data": list_target_schemas()}
|
||||
|
||||
|
||||
@router.post("/mappings/propose")
|
||||
async def propose_datasource_mapping(
|
||||
payload: MappingProposeRequest,
|
||||
current_user: User = Depends(get_current_user),
|
||||
ai_client: AIProviderClient = Depends(get_ai_provider_client),
|
||||
):
|
||||
"""Generate a mapping draft for a sample payload and target schema."""
|
||||
schema = get_target_schema(payload.target_schema)
|
||||
redacted_sample = redact_for_llm(payload.sample_payload)
|
||||
fallback_mapping = build_heuristic_mapping(redacted_sample, payload.target_schema)
|
||||
|
||||
ai_error: str | None = None
|
||||
mapping = fallback_mapping
|
||||
generated_by = "heuristic"
|
||||
if payload.use_ai:
|
||||
try:
|
||||
response = await ai_client.analyze(
|
||||
SituationalAnalysisRequest(
|
||||
title=f"Generate datasource mapping for {schema.key}",
|
||||
objective=(
|
||||
"Return only JSON for a deterministic mapping DSL. "
|
||||
"The JSON must contain source.items_path and fields. "
|
||||
"Do not include prose or code."
|
||||
),
|
||||
context={
|
||||
"target_schema": schema.to_dict(),
|
||||
"sample_payload": redacted_sample,
|
||||
"mapping_dsl_example": fallback_mapping,
|
||||
},
|
||||
observations=[
|
||||
"Use JSONPath-like paths beginning with $.",
|
||||
"Never generate executable code.",
|
||||
"Use field types from the target schema.",
|
||||
],
|
||||
constraints=[
|
||||
"Return a single JSON object.",
|
||||
"Do not include credentials or secrets.",
|
||||
"Mark uncertain optional fields with default null.",
|
||||
],
|
||||
)
|
||||
)
|
||||
parsed = _parse_mapping_from_ai_text(response.content)
|
||||
if parsed:
|
||||
mapping = parsed
|
||||
generated_by = "ai_provider"
|
||||
else:
|
||||
ai_error = "AI provider did not return a valid mapping JSON object."
|
||||
except HTTPException as exc:
|
||||
ai_error = str(exc.detail)
|
||||
|
||||
mapping.setdefault("meta", {})
|
||||
if isinstance(mapping["meta"], dict):
|
||||
mapping["meta"].update(
|
||||
result.append(
|
||||
{
|
||||
"generated_by": generated_by,
|
||||
"requires_review": True,
|
||||
"ai_error": ai_error,
|
||||
"name": name,
|
||||
"default_url": yaml_url,
|
||||
"endpoint": db_config.endpoint if db_config else yaml_url,
|
||||
"is_overridden": db_config is not None and db_config.endpoint != yaml_url
|
||||
if yaml_url
|
||||
else db_config is not None,
|
||||
"is_active": db_config.is_active if db_config else True,
|
||||
"source_type": db_config.source_type if db_config else "http",
|
||||
"description": db_config.description
|
||||
if db_config
|
||||
else f"Data source from YAML: {yaml_key}",
|
||||
}
|
||||
)
|
||||
|
||||
return {
|
||||
"target_schema": schema.to_dict(),
|
||||
"mapping_json": mapping,
|
||||
"sample_payload_hash": stable_payload_hash(payload.sample_payload),
|
||||
"redacted_sample_payload": redacted_sample,
|
||||
}
|
||||
|
||||
|
||||
@router.post("/mappings/preview")
|
||||
async def preview_datasource_mapping(
|
||||
payload: MappingPreviewRequest,
|
||||
current_user: User = Depends(get_current_user),
|
||||
):
|
||||
"""Preview deterministic mapping output for a sample payload."""
|
||||
try:
|
||||
preview = execute_mapping(
|
||||
payload.sample_payload,
|
||||
payload.mapping_json,
|
||||
payload.target_schema,
|
||||
limit=payload.limit,
|
||||
)
|
||||
except (MappingError, ValueError) as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
||||
|
||||
return {
|
||||
"success": preview["failed_count"] == 0,
|
||||
"preview": preview,
|
||||
"sample_payload_hash": stable_payload_hash(payload.sample_payload),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/mappings")
|
||||
async def list_datasource_mappings(
|
||||
datasource_config_id: Optional[int] = None,
|
||||
active_only: bool = False,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""List saved mapping templates."""
|
||||
query = select(DataSourceMappingTemplate).order_by(
|
||||
DataSourceMappingTemplate.datasource_config_id,
|
||||
DataSourceMappingTemplate.version.desc(),
|
||||
)
|
||||
if datasource_config_id is not None:
|
||||
query = query.where(DataSourceMappingTemplate.datasource_config_id == datasource_config_id)
|
||||
if active_only:
|
||||
query = query.where(DataSourceMappingTemplate.is_active.is_(True))
|
||||
|
||||
result = await db.execute(query)
|
||||
mappings = result.scalars().all()
|
||||
return {"total": len(mappings), "data": [serialize_mapping_template(item) for item in mappings]}
|
||||
|
||||
|
||||
@router.post("/mappings")
|
||||
async def create_datasource_mapping(
|
||||
payload: MappingTemplateCreate,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""Save a mapping template for a datasource config."""
|
||||
get_target_schema(payload.target_schema)
|
||||
datasource = await db.get(DataSourceConfig, payload.datasource_config_id)
|
||||
if not datasource:
|
||||
raise HTTPException(status_code=404, detail="Configuration not found")
|
||||
|
||||
if payload.sample_payload is not None:
|
||||
try:
|
||||
execute_mapping(payload.sample_payload, payload.mapping_json, payload.target_schema, limit=100)
|
||||
except (MappingError, ValueError) as exc:
|
||||
raise HTTPException(status_code=400, detail=f"Mapping validation failed: {exc}") from exc
|
||||
|
||||
result = await db.execute(
|
||||
select(func.max(DataSourceMappingTemplate.version)).where(
|
||||
DataSourceMappingTemplate.datasource_config_id == payload.datasource_config_id,
|
||||
DataSourceMappingTemplate.target_schema == payload.target_schema,
|
||||
)
|
||||
)
|
||||
next_version = int(result.scalar() or 0) + 1
|
||||
|
||||
if payload.is_active:
|
||||
await db.execute(
|
||||
DataSourceMappingTemplate.__table__.update()
|
||||
.where(DataSourceMappingTemplate.datasource_config_id == payload.datasource_config_id)
|
||||
.values(is_active=False)
|
||||
)
|
||||
|
||||
template = DataSourceMappingTemplate(
|
||||
datasource_config_id=payload.datasource_config_id,
|
||||
target_schema=payload.target_schema,
|
||||
mapping_json=payload.mapping_json,
|
||||
sample_payload_hash=payload.sample_payload_hash
|
||||
or (stable_payload_hash(payload.sample_payload) if payload.sample_payload is not None else None),
|
||||
validation_status=payload.validation_status,
|
||||
version=next_version,
|
||||
is_active=payload.is_active,
|
||||
)
|
||||
db.add(template)
|
||||
await db.commit()
|
||||
await db.refresh(template)
|
||||
return {"message": "Mapping template saved successfully", "data": serialize_mapping_template(template)}
|
||||
|
||||
|
||||
@router.put("/mappings/{mapping_id}")
|
||||
async def update_datasource_mapping(
|
||||
mapping_id: int,
|
||||
payload: MappingTemplateUpdate,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""Update a mapping template in place."""
|
||||
template = await db.get(DataSourceMappingTemplate, mapping_id)
|
||||
if not template:
|
||||
raise HTTPException(status_code=404, detail="Mapping template not found")
|
||||
|
||||
target_schema = payload.target_schema or template.target_schema
|
||||
mapping_json = payload.mapping_json or template.mapping_json
|
||||
get_target_schema(target_schema)
|
||||
if payload.sample_payload is not None:
|
||||
try:
|
||||
execute_mapping(payload.sample_payload, mapping_json, target_schema, limit=100)
|
||||
except (MappingError, ValueError) as exc:
|
||||
raise HTTPException(status_code=400, detail=f"Mapping validation failed: {exc}") from exc
|
||||
|
||||
if payload.is_active is True:
|
||||
await db.execute(
|
||||
DataSourceMappingTemplate.__table__.update()
|
||||
.where(DataSourceMappingTemplate.datasource_config_id == template.datasource_config_id)
|
||||
.where(DataSourceMappingTemplate.id != template.id)
|
||||
.values(is_active=False)
|
||||
)
|
||||
|
||||
template.target_schema = target_schema
|
||||
template.mapping_json = mapping_json
|
||||
if payload.sample_payload_hash is not None:
|
||||
template.sample_payload_hash = payload.sample_payload_hash
|
||||
elif payload.sample_payload is not None:
|
||||
template.sample_payload_hash = stable_payload_hash(payload.sample_payload)
|
||||
if payload.validation_status is not None:
|
||||
template.validation_status = payload.validation_status
|
||||
if payload.is_active is not None:
|
||||
template.is_active = payload.is_active
|
||||
|
||||
await db.commit()
|
||||
await db.refresh(template)
|
||||
return {"message": "Mapping template updated successfully", "data": serialize_mapping_template(template)}
|
||||
|
||||
|
||||
@router.post("/{config_id}/run-mapped")
|
||||
async def run_mapped_datasource(
|
||||
config_id: int,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""Run a saved custom datasource through its active deterministic mapping."""
|
||||
datasource = await db.get(DataSourceConfig, config_id)
|
||||
if not datasource:
|
||||
raise HTTPException(status_code=404, detail="Configuration not found")
|
||||
|
||||
result = await db.execute(
|
||||
select(DataSourceMappingTemplate)
|
||||
.where(DataSourceMappingTemplate.datasource_config_id == config_id)
|
||||
.where(DataSourceMappingTemplate.is_active.is_(True))
|
||||
.order_by(DataSourceMappingTemplate.version.desc())
|
||||
.limit(1)
|
||||
)
|
||||
mapping = result.scalar_one_or_none()
|
||||
if not mapping:
|
||||
raise HTTPException(status_code=404, detail="No active mapping template found")
|
||||
|
||||
try:
|
||||
sample = await fetch_custom_sample_from_config(datasource, 5_000_000)
|
||||
mapped = execute_mapping(sample, mapping.mapping_json, mapping.target_schema)
|
||||
except httpx.HTTPStatusError as exc:
|
||||
raise HTTPException(
|
||||
status_code=exc.response.status_code,
|
||||
detail=f"Datasource request failed: HTTP {exc.response.status_code}",
|
||||
) from exc
|
||||
except httpx.HTTPError as exc:
|
||||
raise HTTPException(status_code=502, detail=f"Datasource request failed: {exc}") from exc
|
||||
except (MappingError, ValueError) as exc:
|
||||
raise HTTPException(status_code=400, detail=f"Mapping failed: {exc}") from exc
|
||||
|
||||
if mapped["failed_count"] > 0:
|
||||
return {
|
||||
"status": "failed",
|
||||
"datasource_config_id": config_id,
|
||||
"mapping_id": mapping.id,
|
||||
"mapping_version": mapping.version,
|
||||
"target_schema": mapping.target_schema,
|
||||
"mapped_count": mapped["mapped_count"],
|
||||
"failed_count": mapped["failed_count"],
|
||||
"errors": mapped["errors"][:20],
|
||||
}
|
||||
|
||||
written_count = await persist_mapped_records(
|
||||
db,
|
||||
datasource_name=datasource.name,
|
||||
datasource_config_id=datasource.id,
|
||||
target_schema=mapping.target_schema,
|
||||
records=mapped["records"],
|
||||
mapping_version=mapping.version,
|
||||
)
|
||||
return {
|
||||
"status": "success",
|
||||
"datasource_config_id": config_id,
|
||||
"mapping_id": mapping.id,
|
||||
"mapping_version": mapping.version,
|
||||
"target_schema": mapping.target_schema,
|
||||
"fetched_count": mapped["total_items"],
|
||||
"mapped_count": mapped["mapped_count"],
|
||||
"written_count": written_count,
|
||||
}
|
||||
return {"total": len(result), "data": result}
|
||||
@@ -9,7 +9,6 @@ from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.core.security import get_current_user
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
from app.core.datasource_defaults import DEFAULT_DATASOURCES
|
||||
from app.db.session import get_db
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.models.data_snapshot import DataSnapshot
|
||||
@@ -36,17 +35,6 @@ def format_frequency_label(minutes: int) -> str:
|
||||
return f"{minutes}m"
|
||||
|
||||
|
||||
def datasource_metadata(source: str) -> dict:
|
||||
info = DEFAULT_DATASOURCES.get(source, {})
|
||||
return {
|
||||
"display_name": info.get("display_name") or info.get("name") or source,
|
||||
"is_free": bool(info.get("is_free", True)),
|
||||
"requires_credentials": bool(info.get("requires_credentials", False)),
|
||||
"credential_provider": info.get("credential_provider"),
|
||||
"credential_status": info.get("credential_status", "none"),
|
||||
}
|
||||
|
||||
|
||||
def is_due_for_collection(datasource: DataSource, now: datetime) -> bool:
|
||||
if datasource.last_run_at is None:
|
||||
return True
|
||||
@@ -84,6 +72,31 @@ async def _load_latest_running_tasks(
|
||||
return {task.datasource_id: task for task in result.scalars().all()}
|
||||
|
||||
|
||||
async def _load_latest_completed_tasks(
|
||||
db: AsyncSession,
|
||||
datasource_ids: list[int],
|
||||
) -> dict[int, CollectionTask]:
|
||||
if not datasource_ids:
|
||||
return {}
|
||||
|
||||
ranked_tasks = (
|
||||
select(
|
||||
CollectionTask.id.label("task_id"),
|
||||
_task_rank_column(CollectionTask.completed_at),
|
||||
)
|
||||
.where(CollectionTask.datasource_id.in_(datasource_ids))
|
||||
.where(CollectionTask.completed_at.isnot(None))
|
||||
.where(CollectionTask.status.in_(("success", "failed", "cancelled")))
|
||||
.subquery()
|
||||
)
|
||||
result = await db.execute(
|
||||
select(CollectionTask)
|
||||
.join(ranked_tasks, CollectionTask.id == ranked_tasks.c.task_id)
|
||||
.where(ranked_tasks.c.row_num == 1)
|
||||
)
|
||||
return {task.datasource_id: task for task in result.scalars().all()}
|
||||
|
||||
|
||||
async def _load_latest_task_ids(
|
||||
db: AsyncSession,
|
||||
datasource_ids: list[int],
|
||||
@@ -110,6 +123,21 @@ async def _load_latest_task_ids(
|
||||
return {datasource_id: task_id for datasource_id, task_id in result.all()}
|
||||
|
||||
|
||||
async def _load_datasource_data_counts(
|
||||
db: AsyncSession,
|
||||
sources: list[str],
|
||||
) -> dict[str, int]:
|
||||
if not sources:
|
||||
return {}
|
||||
|
||||
result = await db.execute(
|
||||
select(CollectedData.source, func.count(CollectedData.id))
|
||||
.where(CollectedData.source.in_(sources))
|
||||
.group_by(CollectedData.source)
|
||||
)
|
||||
return {source: count for source, count in result.all()}
|
||||
|
||||
|
||||
async def _load_datasource_endpoint_overrides(
|
||||
db: AsyncSession,
|
||||
sources: list[str],
|
||||
@@ -133,7 +161,7 @@ async def _load_datasource_endpoint_overrides(
|
||||
async def _load_datasource_list_context(
|
||||
db: AsyncSession,
|
||||
datasources: list[DataSource],
|
||||
) -> tuple[dict[int, CollectionTask], dict[str, str]]:
|
||||
) -> tuple[dict[int, CollectionTask], dict[int, CollectionTask], dict[str, int], dict[str, str]]:
|
||||
datasource_ids = [datasource.id for datasource in datasources]
|
||||
sources = [datasource.source for datasource in datasources]
|
||||
|
||||
@@ -157,8 +185,10 @@ async def _load_datasource_list_context(
|
||||
if stale_datasource_ids:
|
||||
running_tasks = await _load_latest_running_tasks(db, datasource_ids)
|
||||
|
||||
completed_tasks = await _load_latest_completed_tasks(db, datasource_ids)
|
||||
data_counts = await _load_datasource_data_counts(db, sources)
|
||||
endpoint_overrides = await _load_datasource_endpoint_overrides(db, sources)
|
||||
return running_tasks, endpoint_overrides
|
||||
return running_tasks, completed_tasks, data_counts, endpoint_overrides
|
||||
|
||||
|
||||
async def get_datasource_record(db: AsyncSession, source_id: str) -> Optional[DataSource]:
|
||||
@@ -371,19 +401,26 @@ async def list_datasources(
|
||||
|
||||
collector_list = []
|
||||
config = get_data_sources_config()
|
||||
running_tasks, endpoint_overrides = await _load_datasource_list_context(db, datasources)
|
||||
running_tasks, completed_tasks, data_counts, endpoint_overrides = await _load_datasource_list_context(
|
||||
db,
|
||||
datasources,
|
||||
)
|
||||
for datasource in datasources:
|
||||
running_task = running_tasks.get(datasource.id)
|
||||
endpoint = endpoint_overrides.get(datasource.source) or config.get_yaml_url(datasource.source)
|
||||
last_run_at = datasource.last_run_at
|
||||
last_status = datasource.last_status
|
||||
last_task = completed_tasks.get(datasource.id)
|
||||
endpoint = endpoint_overrides.get(datasource.source) or config.get_yaml_url(
|
||||
datasource.source,
|
||||
)
|
||||
data_count = data_counts.get(datasource.source, 0)
|
||||
|
||||
last_run_at = datasource.last_run_at or (last_task.completed_at if last_task else None)
|
||||
last_run = to_iso8601_utc(last_run_at)
|
||||
last_status = datasource.last_status or (last_task.status if last_task else None)
|
||||
|
||||
collector_list.append(
|
||||
{
|
||||
"id": datasource.id,
|
||||
"source": datasource.source,
|
||||
"name": datasource.name,
|
||||
**datasource_metadata(datasource.source),
|
||||
"module": datasource.module,
|
||||
"priority": datasource.priority,
|
||||
"frequency": format_frequency_label(datasource.frequency_minutes),
|
||||
@@ -391,18 +428,15 @@ async def list_datasources(
|
||||
"is_active": datasource.is_active,
|
||||
"collector_class": datasource.collector_class,
|
||||
"endpoint": endpoint,
|
||||
"last_run": to_iso8601_utc(last_run_at),
|
||||
"last_run": last_run,
|
||||
"last_run_at": to_iso8601_utc(last_run_at),
|
||||
"last_status": last_status,
|
||||
"last_records_processed": last_task.records_processed if last_task else None,
|
||||
"data_count": data_count,
|
||||
"is_running": running_task is not None,
|
||||
"task_id": running_task.id if running_task else None,
|
||||
"progress": running_task.progress if running_task else None,
|
||||
"phase": running_task.phase if running_task else None,
|
||||
"phase_progress": running_task.phase_progress if running_task else None,
|
||||
"phase_message": running_task.phase_message if running_task else None,
|
||||
"phase_current": running_task.phase_current if running_task else None,
|
||||
"phase_total": running_task.phase_total if running_task else None,
|
||||
"phase_unit": running_task.phase_unit if running_task else None,
|
||||
"records_processed": running_task.records_processed if running_task else None,
|
||||
"total_records": running_task.total_records if running_task else None,
|
||||
}
|
||||
@@ -541,7 +575,6 @@ async def get_datasource(
|
||||
return {
|
||||
"id": datasource.id,
|
||||
"name": datasource.name,
|
||||
**datasource_metadata(datasource.source),
|
||||
"module": datasource.module,
|
||||
"priority": datasource.priority,
|
||||
"frequency": format_frequency_label(datasource.frequency_minutes),
|
||||
@@ -631,11 +664,6 @@ async def trigger_datasource(
|
||||
"message": "当前采集任务尚未完成,重新触发会丢失本次未完成进度。是否强制重新采集?",
|
||||
"task_id": running_task.id,
|
||||
"phase": running_task.phase,
|
||||
"phase_progress": running_task.phase_progress,
|
||||
"phase_message": running_task.phase_message,
|
||||
"phase_current": running_task.phase_current,
|
||||
"phase_total": running_task.phase_total,
|
||||
"phase_unit": running_task.phase_unit,
|
||||
"progress": running_task.progress,
|
||||
"records_processed": running_task.records_processed,
|
||||
"total_records": running_task.total_records,
|
||||
@@ -719,29 +747,13 @@ async def get_task_status(
|
||||
task = await get_running_task(db, datasource.id)
|
||||
|
||||
if not task:
|
||||
return {
|
||||
"is_running": False,
|
||||
"task_id": None,
|
||||
"progress": None,
|
||||
"phase": None,
|
||||
"phase_progress": None,
|
||||
"phase_message": None,
|
||||
"phase_current": None,
|
||||
"phase_total": None,
|
||||
"phase_unit": None,
|
||||
"status": "idle",
|
||||
}
|
||||
return {"is_running": False, "task_id": None, "progress": None, "phase": None, "status": "idle"}
|
||||
|
||||
return {
|
||||
"is_running": task.status == "running",
|
||||
"task_id": task.id,
|
||||
"progress": task.progress,
|
||||
"phase": task.phase,
|
||||
"phase_progress": task.phase_progress,
|
||||
"phase_message": task.phase_message,
|
||||
"phase_current": task.phase_current,
|
||||
"phase_total": task.phase_total,
|
||||
"phase_unit": task.phase_unit,
|
||||
"records_processed": task.records_processed,
|
||||
"total_records": task.total_records,
|
||||
"status": task.status,
|
||||
|
||||
@@ -1,13 +0,0 @@
|
||||
from fastapi import APIRouter, Query
|
||||
|
||||
from app.services.earth_news import get_earth_news_payload
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
@router.get("/earth-feed")
|
||||
async def get_earth_feed(
|
||||
lat: float | None = Query(None, description="Current Earth view center latitude"),
|
||||
lon: float | None = Query(None, description="Current Earth view center longitude"),
|
||||
):
|
||||
return await get_earth_news_payload(lat=lat, lon=lon)
|
||||
@@ -9,36 +9,10 @@ from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.security import get_current_user
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.core.config import settings as app_settings
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
from app.core.datasource_defaults import DEFAULT_DATASOURCES
|
||||
from app.db.session import get_db
|
||||
from app.models.datasource import DataSource
|
||||
from app.models.datasource_config import DataSourceConfig
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.models.user import User
|
||||
from app.services.barentswatch import (
|
||||
BarentsWatchConfig,
|
||||
check_barentswatch_config,
|
||||
check_barentswatch_connectivity,
|
||||
get_barentswatch_datasource_record,
|
||||
resolve_barentswatch_config,
|
||||
)
|
||||
from app.services.credential_guides import (
|
||||
generate_credential_guide,
|
||||
get_credential_guide,
|
||||
reset_credential_guide,
|
||||
)
|
||||
from app.services.datasource_connectivity import (
|
||||
build_builtin_connectivity_checksum,
|
||||
save_connectivity_success,
|
||||
)
|
||||
from app.services.ai_client import AIProviderClient, get_ai_provider_client
|
||||
from app.services.llm_provider_catalog import (
|
||||
get_fallback_llm_provider_preset,
|
||||
list_fallback_llm_provider_presets,
|
||||
refresh_llm_provider_preset,
|
||||
)
|
||||
from app.services.scheduler import sync_datasource_job
|
||||
from app.services.tv_streams import DEFAULT_TV_SETTINGS, get_tv_settings_payload, normalize_tv_settings
|
||||
|
||||
@@ -65,21 +39,6 @@ DEFAULT_SETTINGS = {
|
||||
"password_policy": "medium",
|
||||
},
|
||||
"tv": DEFAULT_TV_SETTINGS,
|
||||
"external_integrations": {
|
||||
"ai_provider": {
|
||||
"service_url": "",
|
||||
"service_token": "",
|
||||
"provider": "minimax",
|
||||
"provider_api": "anthropic-messages",
|
||||
"base_url": "https://api.minimaxi.com/anthropic",
|
||||
"model": "MiniMax-M2.7",
|
||||
"api_key": "",
|
||||
"max_tokens": 1200,
|
||||
"anthropic_version": "2023-06-01",
|
||||
"timeout_seconds": 60,
|
||||
"retry_attempts": 2,
|
||||
}
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@@ -137,34 +96,6 @@ class TVSettingsUpdate(BaseModel):
|
||||
sources: list[TVStreamSourceUpdate] = Field(default_factory=list)
|
||||
|
||||
|
||||
class AIProviderIntegrationUpdate(BaseModel):
|
||||
service_url: str = ""
|
||||
service_token: Optional[str] = None
|
||||
provider: str = Field(default="minimax", max_length=80)
|
||||
provider_api: str = Field(default="anthropic-messages", max_length=80)
|
||||
base_url: str = Field(default="", max_length=500)
|
||||
model: str = Field(default="", max_length=200)
|
||||
api_key: Optional[str] = None
|
||||
max_tokens: int = Field(default=1200, ge=1, le=200000)
|
||||
anthropic_version: str = Field(default="2023-06-01", max_length=40)
|
||||
timeout_seconds: int = Field(default=60, ge=5, le=600)
|
||||
retry_attempts: int = Field(default=2, ge=1, le=10)
|
||||
clear_service_token: bool = False
|
||||
clear_api_key: bool = False
|
||||
|
||||
|
||||
class BarentsWatchIntegrationUpdate(BaseModel):
|
||||
endpoint: str = ""
|
||||
client_id: str = ""
|
||||
client_secret: Optional[str] = None
|
||||
clear_client_secret: bool = False
|
||||
|
||||
|
||||
class ExternalIntegrationsUpdate(BaseModel):
|
||||
ai_provider: AIProviderIntegrationUpdate
|
||||
barentswatch: BarentsWatchIntegrationUpdate
|
||||
|
||||
|
||||
def merge_with_defaults(category: str, payload: Optional[dict]) -> dict:
|
||||
merged = deepcopy(DEFAULT_SETTINGS[category])
|
||||
if payload:
|
||||
@@ -215,151 +146,6 @@ async def save_setting_payload(db: AsyncSession, category: str, payload: dict) -
|
||||
return merge_with_defaults(category, record.payload)
|
||||
|
||||
|
||||
def _mask_secret(value: Optional[str]) -> dict:
|
||||
if not value:
|
||||
return {"configured": False, "preview": ""}
|
||||
text = str(value)
|
||||
if "-" in text:
|
||||
prefix = text.split("-", 1)[0] + "-"
|
||||
preview = prefix + ("*" * max(len(text) - len(prefix), 1))
|
||||
else:
|
||||
prefix_len = min(4, len(text))
|
||||
preview = text[:prefix_len] + ("*" * max(len(text) - prefix_len, 1))
|
||||
return {"configured": True, "preview": preview}
|
||||
|
||||
|
||||
async def get_runtime_ai_provider_config(db: AsyncSession) -> dict:
|
||||
runtime_record = await get_setting_record(db, "external_integrations")
|
||||
payload = merge_with_defaults(
|
||||
"external_integrations",
|
||||
runtime_record.payload if runtime_record else None,
|
||||
)
|
||||
ai_payload = payload.get("ai_provider") or {}
|
||||
has_runtime_llm_config = bool(
|
||||
runtime_record
|
||||
and isinstance(runtime_record.payload, dict)
|
||||
and isinstance(runtime_record.payload.get("ai_provider"), dict)
|
||||
)
|
||||
return {
|
||||
"service_url": ai_payload.get("service_url") or app_settings.AI_PROVIDER_SERVICE_URL,
|
||||
"service_token": ai_payload.get("service_token") or app_settings.AI_PROVIDER_SERVICE_TOKEN,
|
||||
"timeout_seconds": int(
|
||||
ai_payload.get("timeout_seconds") or app_settings.AI_PROVIDER_TIMEOUT_SECONDS
|
||||
),
|
||||
"retry_attempts": int(
|
||||
ai_payload.get("retry_attempts") or app_settings.AI_PROVIDER_RETRY_ATTEMPTS
|
||||
),
|
||||
"llm_config": {
|
||||
"provider": ai_payload.get("provider") or "minimax",
|
||||
"provider_api": ai_payload.get("provider_api") or "anthropic-messages",
|
||||
"base_url": ai_payload.get("base_url") or "https://api.minimaxi.com/anthropic",
|
||||
"model": ai_payload.get("model") or "MiniMax-M2.7",
|
||||
"api_key": ai_payload.get("api_key") or "",
|
||||
"max_tokens": int(ai_payload.get("max_tokens") or 1200),
|
||||
"anthropic_version": ai_payload.get("anthropic_version") or "2023-06-01",
|
||||
} if has_runtime_llm_config else {},
|
||||
}
|
||||
|
||||
|
||||
async def get_barentswatch_config_record(db: AsyncSession) -> Optional[DataSourceConfig]:
|
||||
return await get_barentswatch_datasource_record(db)
|
||||
|
||||
|
||||
async def serialize_external_integrations(db: AsyncSession) -> dict:
|
||||
ai_config = await get_runtime_ai_provider_config(db)
|
||||
runtime_setting = await get_setting_record(db, "external_integrations")
|
||||
display_llm_config = ai_config["llm_config"] or DEFAULT_SETTINGS["external_integrations"]["ai_provider"]
|
||||
barentswatch_record = await get_barentswatch_config_record(db)
|
||||
barentswatch_auth = barentswatch_record.auth_config if barentswatch_record else {}
|
||||
barentswatch_auth = barentswatch_auth or {}
|
||||
resolved_barentswatch = await resolve_barentswatch_config(db)
|
||||
return {
|
||||
"ai_provider": {
|
||||
"service_url": ai_config["service_url"],
|
||||
"service_token": _mask_secret(ai_config["service_token"]),
|
||||
"provider": display_llm_config.get("provider") or "minimax",
|
||||
"provider_api": display_llm_config.get("provider_api") or "anthropic-messages",
|
||||
"base_url": display_llm_config.get("base_url") or "https://api.minimaxi.com/anthropic",
|
||||
"model": display_llm_config.get("model") or "MiniMax-M2.7",
|
||||
"api_key": _mask_secret(display_llm_config.get("api_key")),
|
||||
"max_tokens": int(display_llm_config.get("max_tokens") or 1200),
|
||||
"anthropic_version": display_llm_config.get("anthropic_version") or "2023-06-01",
|
||||
"timeout_seconds": ai_config["timeout_seconds"],
|
||||
"retry_attempts": ai_config["retry_attempts"],
|
||||
"source": "runtime" if runtime_setting else "env",
|
||||
},
|
||||
"barentswatch": {
|
||||
"endpoint": resolved_barentswatch.endpoint,
|
||||
"client_id": barentswatch_auth.get("client_id") or resolved_barentswatch.client_id,
|
||||
"client_secret": _mask_secret(
|
||||
barentswatch_auth.get("client_secret") or resolved_barentswatch.client_secret
|
||||
),
|
||||
"source": resolved_barentswatch.credential_source,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
async def save_external_integrations_payload(
|
||||
db: AsyncSession,
|
||||
update: ExternalIntegrationsUpdate,
|
||||
) -> dict:
|
||||
current_payload = await get_setting_payload(db, "external_integrations")
|
||||
current_ai = current_payload.get("ai_provider") or {}
|
||||
ai_payload = {
|
||||
"service_url": update.ai_provider.service_url.strip()
|
||||
or app_settings.AI_PROVIDER_SERVICE_URL,
|
||||
"service_token": current_ai.get("service_token") or "",
|
||||
"provider": update.ai_provider.provider.strip() or "minimax",
|
||||
"provider_api": update.ai_provider.provider_api.strip() or "anthropic-messages",
|
||||
"base_url": update.ai_provider.base_url.strip(),
|
||||
"model": update.ai_provider.model.strip(),
|
||||
"api_key": current_ai.get("api_key") or "",
|
||||
"max_tokens": update.ai_provider.max_tokens,
|
||||
"anthropic_version": update.ai_provider.anthropic_version.strip() or "2023-06-01",
|
||||
"timeout_seconds": update.ai_provider.timeout_seconds,
|
||||
"retry_attempts": update.ai_provider.retry_attempts,
|
||||
}
|
||||
if update.ai_provider.clear_service_token:
|
||||
ai_payload["service_token"] = ""
|
||||
elif update.ai_provider.service_token not in (None, ""):
|
||||
ai_payload["service_token"] = update.ai_provider.service_token
|
||||
if update.ai_provider.clear_api_key:
|
||||
ai_payload["api_key"] = ""
|
||||
elif update.ai_provider.api_key not in (None, ""):
|
||||
ai_payload["api_key"] = update.ai_provider.api_key
|
||||
|
||||
await save_setting_payload(db, "external_integrations", {"ai_provider": ai_payload})
|
||||
|
||||
default_endpoint = get_data_sources_config().get_yaml_url("barentswatch_vessels")
|
||||
barentswatch_record = await get_barentswatch_config_record(db)
|
||||
if barentswatch_record is None:
|
||||
barentswatch_record = DataSourceConfig(
|
||||
name="barentswatch_vessels",
|
||||
description="BarentsWatch Live AIS credentials",
|
||||
source_type="api",
|
||||
endpoint=update.barentswatch.endpoint.strip() or default_endpoint,
|
||||
auth_type="oauth_client",
|
||||
auth_config={},
|
||||
headers={},
|
||||
config={},
|
||||
is_active=True,
|
||||
)
|
||||
db.add(barentswatch_record)
|
||||
|
||||
current_auth = dict(barentswatch_record.auth_config or {})
|
||||
if update.barentswatch.clear_client_secret:
|
||||
current_auth.pop("client_secret", None)
|
||||
elif update.barentswatch.client_secret not in (None, ""):
|
||||
current_auth["client_secret"] = update.barentswatch.client_secret
|
||||
current_auth["client_id"] = update.barentswatch.client_id.strip()
|
||||
barentswatch_record.endpoint = update.barentswatch.endpoint.strip() or default_endpoint
|
||||
barentswatch_record.auth_type = "oauth_client"
|
||||
barentswatch_record.auth_config = current_auth
|
||||
await db.commit()
|
||||
|
||||
return await serialize_external_integrations(db)
|
||||
|
||||
|
||||
def format_frequency_label(minutes: int) -> str:
|
||||
if minutes % 1440 == 0:
|
||||
return f"{minutes // 1440}d"
|
||||
@@ -369,11 +155,9 @@ def format_frequency_label(minutes: int) -> str:
|
||||
|
||||
|
||||
def serialize_collector(datasource: DataSource) -> dict:
|
||||
defaults = DEFAULT_DATASOURCES.get(datasource.source, {})
|
||||
return {
|
||||
"id": datasource.id,
|
||||
"name": datasource.name,
|
||||
"display_name": defaults.get("display_name") or datasource.name,
|
||||
"source": datasource.source,
|
||||
"module": datasource.module,
|
||||
"priority": datasource.priority,
|
||||
@@ -383,10 +167,6 @@ def serialize_collector(datasource: DataSource) -> dict:
|
||||
"last_run_at": to_iso8601_utc(datasource.last_run_at),
|
||||
"last_status": datasource.last_status,
|
||||
"next_run_at": to_iso8601_utc(datasource.next_run_at),
|
||||
"is_free": bool(defaults.get("is_free", True)),
|
||||
"requires_credentials": bool(defaults.get("requires_credentials", False)),
|
||||
"credential_provider": defaults.get("credential_provider"),
|
||||
"credential_status": defaults.get("credential_status", "none"),
|
||||
}
|
||||
|
||||
|
||||
@@ -463,135 +243,6 @@ async def update_tv_settings(
|
||||
return {"status": "updated", "tv": normalize_tv_settings(saved)}
|
||||
|
||||
|
||||
@router.get("/integrations")
|
||||
async def get_external_integrations(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return {"integrations": await serialize_external_integrations(db)}
|
||||
|
||||
|
||||
@router.get("/integrations/barentswatch/connectivity")
|
||||
async def get_barentswatch_connectivity(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await check_barentswatch_connectivity(db)
|
||||
|
||||
|
||||
@router.post("/integrations/barentswatch/connect")
|
||||
async def connect_barentswatch_integration(
|
||||
payload: BarentsWatchIntegrationUpdate,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
current = await resolve_barentswatch_config(db)
|
||||
config = BarentsWatchConfig(
|
||||
endpoint=payload.endpoint.strip() or current.endpoint,
|
||||
client_id=payload.client_id.strip() or current.client_id,
|
||||
client_secret=(
|
||||
""
|
||||
if payload.clear_client_secret
|
||||
else payload.client_secret or current.client_secret
|
||||
),
|
||||
credential_source="draft",
|
||||
endpoint_source="draft",
|
||||
)
|
||||
result = await check_barentswatch_config(config)
|
||||
if result.get("success"):
|
||||
checksum, _context = await build_builtin_connectivity_checksum(
|
||||
"barentswatch_vessels",
|
||||
config.endpoint,
|
||||
"none",
|
||||
{},
|
||||
{},
|
||||
db,
|
||||
credential_override={
|
||||
"client_id": config.client_id,
|
||||
"client_secret": config.client_secret,
|
||||
},
|
||||
)
|
||||
validation = await save_connectivity_success(
|
||||
db,
|
||||
"barentswatch_vessels",
|
||||
checksum,
|
||||
result,
|
||||
connected_by="connection_button",
|
||||
)
|
||||
await db.commit()
|
||||
return {**result, "connected": True, "validation": validation}
|
||||
return {**result, "connected": False}
|
||||
|
||||
|
||||
@router.get("/credential-guides/{provider}")
|
||||
async def read_credential_guide(
|
||||
provider: str,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
try:
|
||||
return {"guide": await get_credential_guide(db, provider)}
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=404, detail=str(exc)) from exc
|
||||
|
||||
|
||||
@router.post("/credential-guides/{provider}/generate")
|
||||
async def generate_provider_credential_guide(
|
||||
provider: str,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
ai_client: AIProviderClient = Depends(get_ai_provider_client),
|
||||
):
|
||||
try:
|
||||
return {"guide": await generate_credential_guide(db, provider, ai_client)}
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=404, detail=str(exc)) from exc
|
||||
|
||||
|
||||
@router.post("/credential-guides/{provider}/reset")
|
||||
async def reset_provider_credential_guide(
|
||||
provider: str,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
try:
|
||||
return {"guide": await reset_credential_guide(db, provider)}
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=404, detail=str(exc)) from exc
|
||||
|
||||
|
||||
@router.get("/integrations/ai-provider/presets")
|
||||
async def get_ai_provider_presets(
|
||||
current_user: User = Depends(get_current_user),
|
||||
):
|
||||
return {"data": list_fallback_llm_provider_presets()}
|
||||
|
||||
|
||||
@router.post("/integrations/ai-provider/presets/{provider}/refresh")
|
||||
async def refresh_ai_provider_preset(
|
||||
provider: str,
|
||||
current_user: User = Depends(get_current_user),
|
||||
):
|
||||
try:
|
||||
return {"data": await refresh_llm_provider_preset(provider)}
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=404, detail=str(exc)) from exc
|
||||
except Exception as exc:
|
||||
fallback = get_fallback_llm_provider_preset(provider)
|
||||
fallback["refresh_error"] = str(exc)
|
||||
return {"data": fallback}
|
||||
|
||||
|
||||
@router.put("/integrations")
|
||||
async def update_external_integrations(
|
||||
payload: ExternalIntegrationsUpdate,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
saved = await save_external_integrations_payload(db, payload)
|
||||
return {"status": "updated", "integrations": saved}
|
||||
|
||||
|
||||
@router.get("/collectors")
|
||||
async def get_collector_settings(
|
||||
current_user: User = Depends(get_current_user),
|
||||
@@ -638,7 +289,6 @@ async def get_all_settings(
|
||||
"notifications": setting_payloads["notifications"],
|
||||
"security": setting_payloads["security"],
|
||||
"tv": await get_tv_settings_payload(db),
|
||||
"integrations": await serialize_external_integrations(db),
|
||||
"collectors": [serialize_collector(datasource) for datasource in datasources],
|
||||
"generated_at": to_iso8601_utc(datetime.now(UTC)),
|
||||
}
|
||||
@@ -4,15 +4,12 @@ import os
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
from datetime import datetime
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Request, status
|
||||
from fastapi import APIRouter, Depends, HTTPException, status
|
||||
from pydantic import BaseModel
|
||||
|
||||
from app.core.config import ROOT_DIR
|
||||
from app.core.security import get_current_user
|
||||
from app.models.user import User
|
||||
from app.services.persistent_logs import record_audit_log, record_system_log
|
||||
from app.services.system_control import (
|
||||
build_task_id,
|
||||
clear_active_task_id,
|
||||
@@ -26,15 +23,6 @@ from app.services.system_control import (
|
||||
set_active_task_id,
|
||||
upsert_task_state,
|
||||
)
|
||||
from app.services.system_logs import (
|
||||
DEFAULT_LOG_LINE_LIMIT,
|
||||
MAX_LOG_LINE_LIMIT,
|
||||
SUPPORTED_LOG_LEVELS,
|
||||
append_buffer_log,
|
||||
list_log_sources,
|
||||
normalize_log_level,
|
||||
read_log_snapshot,
|
||||
)
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
@@ -59,59 +47,6 @@ class RestartTaskLogsResponse(BaseModel):
|
||||
lines: list[str]
|
||||
|
||||
|
||||
class SystemLogSourceSummary(BaseModel):
|
||||
source_id: str
|
||||
name: str
|
||||
kind: str
|
||||
location: str
|
||||
description: str
|
||||
category: str
|
||||
status: str
|
||||
|
||||
|
||||
class SystemLogSourcesResponse(BaseModel):
|
||||
items: list[SystemLogSourceSummary]
|
||||
|
||||
|
||||
class SystemLogDailyMarker(BaseModel):
|
||||
date_token: str
|
||||
total: int
|
||||
dominant_level: str
|
||||
|
||||
|
||||
class SystemLogSnapshotResponse(BaseModel):
|
||||
source_id: str
|
||||
name: str
|
||||
kind: str
|
||||
location: str
|
||||
description: str
|
||||
category: str
|
||||
status: str
|
||||
level: str
|
||||
selected_levels: list[str] = []
|
||||
search_query: str = ""
|
||||
available_levels: list[str]
|
||||
daily_markers: list[SystemLogDailyMarker] = []
|
||||
line_limit: int
|
||||
line_count: int
|
||||
lines: list[str]
|
||||
|
||||
|
||||
class EarthClientLogEventCreate(BaseModel):
|
||||
level: str = "error"
|
||||
message: str
|
||||
category: str | None = None
|
||||
url: str | None = None
|
||||
module: str | None = None
|
||||
detail: str | None = None
|
||||
|
||||
|
||||
class EarthClientLogEventResponse(BaseModel):
|
||||
accepted: bool
|
||||
source_id: str
|
||||
level: str
|
||||
|
||||
|
||||
def ensure_super_admin(current_user: User) -> None:
|
||||
if not require_super_admin(current_user.role):
|
||||
raise HTTPException(
|
||||
@@ -120,22 +55,9 @@ def ensure_super_admin(current_user: User) -> None:
|
||||
)
|
||||
|
||||
|
||||
def validate_log_date(raw_value: str | None, field_name: str) -> str | None:
|
||||
if raw_value in {None, ""}:
|
||||
return None
|
||||
try:
|
||||
return datetime.strptime(raw_value, "%Y-%m-%d").date().isoformat()
|
||||
except ValueError as exc:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
detail=f"{field_name} must be in YYYY-MM-DD format",
|
||||
) from exc
|
||||
|
||||
|
||||
@router.post("/restart-tasks", response_model=RestartTaskResponse)
|
||||
async def create_restart_task(
|
||||
payload: RestartTaskCreate,
|
||||
request: Request,
|
||||
current_user: User = Depends(get_current_user),
|
||||
):
|
||||
ensure_super_admin(current_user)
|
||||
@@ -211,31 +133,11 @@ async def create_restart_task(
|
||||
requested_by=requested_by,
|
||||
)
|
||||
clear_active_task_id(task_id)
|
||||
await record_audit_log(
|
||||
action="system.restart_task.requested",
|
||||
actor_id=current_user.id,
|
||||
actor_name=current_user.username,
|
||||
target_type="restart_task",
|
||||
target_id=task_id,
|
||||
result="failed",
|
||||
ip=request.client.host if request.client else None,
|
||||
details={"action": payload.action, "message": task_state["message"]},
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
|
||||
detail=task_state["message"],
|
||||
) from exc
|
||||
|
||||
await record_audit_log(
|
||||
action="system.restart_task.requested",
|
||||
actor_id=current_user.id,
|
||||
actor_name=current_user.username,
|
||||
target_type="restart_task",
|
||||
target_id=task_id,
|
||||
result="accepted",
|
||||
ip=request.client.host if request.client else None,
|
||||
details={"action": payload.action},
|
||||
)
|
||||
return task_state
|
||||
|
||||
|
||||
@@ -263,92 +165,3 @@ async def get_restart_task_logs(
|
||||
if task is None:
|
||||
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Restart task not found")
|
||||
return {"task_id": task_id, "lines": get_task_logs(task_id)}
|
||||
|
||||
|
||||
@router.get("/logs/sources", response_model=SystemLogSourcesResponse)
|
||||
async def get_system_log_sources(
|
||||
current_user: User = Depends(get_current_user),
|
||||
):
|
||||
ensure_super_admin(current_user)
|
||||
return {"items": list_log_sources()}
|
||||
|
||||
|
||||
@router.get("/logs/{source_id}", response_model=SystemLogSnapshotResponse)
|
||||
async def get_system_log_snapshot(
|
||||
source_id: str,
|
||||
limit: int = DEFAULT_LOG_LINE_LIMIT,
|
||||
level: str = "all",
|
||||
levels: str | None = Query(None, description="Comma-separated log levels"),
|
||||
start_date: str | None = Query(None, description="Filter logs from this date (YYYY-MM-DD)"),
|
||||
end_date: str | None = Query(None, description="Filter logs until this date (YYYY-MM-DD)"),
|
||||
search: str | None = Query(None, description="Case-insensitive substring search"),
|
||||
current_user: User = Depends(get_current_user),
|
||||
):
|
||||
ensure_super_admin(current_user)
|
||||
|
||||
if limit < 1 or limit > MAX_LOG_LINE_LIMIT:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
detail=f"limit must be between 1 and {MAX_LOG_LINE_LIMIT}",
|
||||
)
|
||||
if str(level).strip().lower() not in SUPPORTED_LOG_LEVELS and normalize_log_level(level) == "all" and str(level).strip().lower() not in {"", "all"}:
|
||||
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="Unsupported log level")
|
||||
if levels:
|
||||
for raw_level in str(levels).split(","):
|
||||
normalized_level = str(raw_level).strip().lower()
|
||||
if not normalized_level:
|
||||
continue
|
||||
if normalized_level not in SUPPORTED_LOG_LEVELS and normalize_log_level(normalized_level) == "all":
|
||||
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="Unsupported log level")
|
||||
normalized_start_date = validate_log_date(start_date, "start_date")
|
||||
normalized_end_date = validate_log_date(end_date, "end_date")
|
||||
if normalized_start_date and normalized_end_date and normalized_start_date > normalized_end_date:
|
||||
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="start_date must be earlier than or equal to end_date")
|
||||
|
||||
snapshot = read_log_snapshot(
|
||||
source_id,
|
||||
limit,
|
||||
level=level,
|
||||
levels=levels,
|
||||
start_date=normalized_start_date,
|
||||
end_date=normalized_end_date,
|
||||
search=search,
|
||||
)
|
||||
if snapshot is None:
|
||||
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Log source not found")
|
||||
return snapshot
|
||||
|
||||
|
||||
@router.post("/logs/earth-client", response_model=EarthClientLogEventResponse)
|
||||
async def ingest_earth_client_log(
|
||||
payload: EarthClientLogEventCreate,
|
||||
request: Request,
|
||||
):
|
||||
normalized_level = normalize_log_level(payload.level)
|
||||
append_buffer_log(
|
||||
"earth-client",
|
||||
level=normalized_level,
|
||||
message=payload.message,
|
||||
context={
|
||||
"category": payload.category or "",
|
||||
"url": payload.url or "",
|
||||
"module": payload.module or "",
|
||||
"detail": payload.detail or "",
|
||||
},
|
||||
)
|
||||
await record_system_log(
|
||||
source="earth-client",
|
||||
service="earth",
|
||||
module=payload.module or "earth-client",
|
||||
event="earth.client.runtime_log",
|
||||
level=normalized_level,
|
||||
message=payload.message,
|
||||
category=payload.category or "client-runtime",
|
||||
context={
|
||||
"url": payload.url or "",
|
||||
"detail": payload.detail or "",
|
||||
"module": payload.module or "",
|
||||
"client_ip": request.client.host if request.client else "",
|
||||
},
|
||||
)
|
||||
return {"accepted": True, "source_id": "earth-client", "level": normalized_level}
|
||||
@@ -27,9 +27,7 @@ async def list_tasks(
|
||||
offset = (page - 1) * page_size
|
||||
query = """
|
||||
SELECT ct.id, ct.datasource_id, ds.name as datasource_name, ct.status,
|
||||
ct.started_at, ct.completed_at, ct.records_processed, ct.error_message,
|
||||
ct.phase, ct.phase_progress, ct.phase_message, ct.phase_current,
|
||||
ct.phase_total, ct.phase_unit, ct.total_records, ct.progress
|
||||
ct.started_at, ct.completed_at, ct.records_processed, ct.error_message
|
||||
FROM collection_tasks ct
|
||||
JOIN data_sources ds ON ct.datasource_id = ds.id
|
||||
WHERE 1=1
|
||||
@@ -68,14 +66,6 @@ async def list_tasks(
|
||||
"completed_at": to_iso8601_utc(t[5]),
|
||||
"records_processed": t[6],
|
||||
"error_message": t[7],
|
||||
"phase": t[8],
|
||||
"phase_progress": t[9],
|
||||
"phase_message": t[10],
|
||||
"phase_current": t[11],
|
||||
"phase_total": t[12],
|
||||
"phase_unit": t[13],
|
||||
"total_records": t[14],
|
||||
"progress": t[15],
|
||||
}
|
||||
for t in tasks
|
||||
],
|
||||
@@ -91,9 +81,7 @@ async def get_task(
|
||||
result = await db.execute(
|
||||
text("""
|
||||
SELECT ct.id, ct.datasource_id, ds.name as datasource_name, ct.status,
|
||||
ct.started_at, ct.completed_at, ct.records_processed, ct.error_message,
|
||||
ct.phase, ct.phase_progress, ct.phase_message, ct.phase_current,
|
||||
ct.phase_total, ct.phase_unit, ct.total_records, ct.progress
|
||||
ct.started_at, ct.completed_at, ct.records_processed, ct.error_message
|
||||
FROM collection_tasks ct
|
||||
JOIN data_sources ds ON ct.datasource_id = ds.id
|
||||
WHERE ct.id = :id
|
||||
@@ -117,14 +105,6 @@ async def get_task(
|
||||
"completed_at": to_iso8601_utc(task[5]),
|
||||
"records_processed": task[6],
|
||||
"error_message": task[7],
|
||||
"phase": task[8],
|
||||
"phase_progress": task[9],
|
||||
"phase_message": task[10],
|
||||
"phase_current": task[11],
|
||||
"phase_total": task[12],
|
||||
"phase_unit": task[13],
|
||||
"total_records": task[14],
|
||||
"progress": task[15],
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,351 @@
|
||||
"""UE Client Data API
|
||||
|
||||
Flat JSON endpoints designed for easy parsing in Unreal Engine C++/Blueprint.
|
||||
Avoids GeoJSON nesting — every field is at the top level of each item.
|
||||
"""
|
||||
|
||||
from typing import Any, Dict, List, Optional
|
||||
from fastapi import APIRouter, Depends, Query
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from sqlalchemy import select, func
|
||||
|
||||
from app.core.collected_data_fields import get_record_field
|
||||
from app.core.satellite_tle import build_tle_lines_from_elements
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.db.session import get_db
|
||||
from app.models.collected_data import CollectedData
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _current_stmt(source: str, limit: Optional[int] = None):
|
||||
stmt = (
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source == source)
|
||||
.where(CollectedData.is_current.is_(True))
|
||||
.order_by(CollectedData.id.desc())
|
||||
)
|
||||
if limit:
|
||||
stmt = stmt.limit(limit)
|
||||
return stmt
|
||||
|
||||
|
||||
async def _fetch(db: AsyncSession, source: str, limit: Optional[int] = None) -> List[CollectedData]:
|
||||
result = await db.execute(_current_stmt(source, limit))
|
||||
return list(result.scalars().all())
|
||||
|
||||
|
||||
def _safe_float(value: Any) -> Optional[float]:
|
||||
try:
|
||||
v = float(value)
|
||||
return v if v == v else None # reject NaN
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Status endpoint
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@router.get("/status")
|
||||
async def ue_status(db: AsyncSession = Depends(get_db)):
|
||||
"""Quick health-check + data counts for the UE client."""
|
||||
from datetime import UTC, datetime
|
||||
|
||||
async def count_source(source: str) -> int:
|
||||
result = await db.execute(
|
||||
select(func.count())
|
||||
.select_from(CollectedData)
|
||||
.where(CollectedData.source == source)
|
||||
.where(CollectedData.is_current.is_(True))
|
||||
)
|
||||
return result.scalar() or 0
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"server_time": to_iso8601_utc(datetime.now(UTC)),
|
||||
"compute_points_count": await count_source("top500"),
|
||||
"cables_count": await count_source("telegeography_cables"),
|
||||
"landing_points_count": await count_source("arcgis_landing"),
|
||||
"satellites_count": await count_source("celestrak"),
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Compute points (TOP500 supercomputers)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@router.get("/compute-points")
|
||||
async def ue_compute_points(
|
||||
limit: int = Query(default=500, ge=1, le=2000),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""
|
||||
Returns TOP500 supercomputer data as a flat JSON array.
|
||||
|
||||
Response shape:
|
||||
{
|
||||
"count": 500,
|
||||
"items": [
|
||||
{
|
||||
"id": "top500_1",
|
||||
"name": "Frontier",
|
||||
"latitude": 36.01,
|
||||
"longitude": -84.26,
|
||||
"country": "United States",
|
||||
"city": "Oak Ridge",
|
||||
"rank": 1,
|
||||
"rmax_tflops": 1194000.0,
|
||||
"rpeak_tflops": 1679616.0,
|
||||
"cores": 8730112,
|
||||
"power_kw": 22703.0
|
||||
}
|
||||
]
|
||||
}
|
||||
"""
|
||||
records = await _fetch(db, "top500", limit)
|
||||
items = []
|
||||
for record in records:
|
||||
meta = record.extra_data or {}
|
||||
lat = _safe_float(get_record_field(record, "latitude"))
|
||||
lon = _safe_float(get_record_field(record, "longitude"))
|
||||
if lat is None or lon is None:
|
||||
continue
|
||||
items.append({
|
||||
"id": f"top500_{record.id}",
|
||||
"name": record.name or "Unknown",
|
||||
"latitude": lat,
|
||||
"longitude": lon,
|
||||
"country": get_record_field(record, "country") or "",
|
||||
"city": get_record_field(record, "city") or "",
|
||||
"rank": meta.get("rank"),
|
||||
"rmax_tflops": _safe_float(get_record_field(record, "rmax")),
|
||||
"rpeak_tflops": _safe_float(get_record_field(record, "rpeak")),
|
||||
"cores": meta.get("cores"),
|
||||
"power_kw": _safe_float(get_record_field(record, "power")),
|
||||
})
|
||||
return {"count": len(items), "items": items}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Cable landing points
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@router.get("/landing-points")
|
||||
async def ue_landing_points(db: AsyncSession = Depends(get_db)):
|
||||
"""
|
||||
Returns cable landing points as a flat JSON array.
|
||||
|
||||
Response shape:
|
||||
{
|
||||
"count": 1200,
|
||||
"items": [
|
||||
{
|
||||
"id": "lp_42",
|
||||
"name": "Shoreham",
|
||||
"latitude": 50.83,
|
||||
"longitude": -0.28,
|
||||
"country": "United Kingdom",
|
||||
"city": "Shoreham-by-Sea",
|
||||
"cable_names": ["FLAG", "TAT-14"]
|
||||
}
|
||||
]
|
||||
}
|
||||
"""
|
||||
# Load landing points
|
||||
lp_records = await _fetch(db, "arcgis_landing")
|
||||
|
||||
# Load relation + cable data for cable_names mapping
|
||||
rel_result = await db.execute(
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source == "arcgis_relation")
|
||||
.where(CollectedData.is_current.is_(True))
|
||||
)
|
||||
rel_records = list(rel_result.scalars().all())
|
||||
|
||||
cable_result = await db.execute(
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source == "telegeography_cables")
|
||||
.where(CollectedData.is_current.is_(True))
|
||||
)
|
||||
cable_records = list(cable_result.scalars().all())
|
||||
|
||||
# Build mapping: city_id → list of cable names
|
||||
city_to_cable_ids: Dict[int, List[int]] = {}
|
||||
for r in rel_records:
|
||||
meta = r.extra_data or {}
|
||||
city_id = meta.get("city_id")
|
||||
cable_id = meta.get("cable_id")
|
||||
if city_id is not None and cable_id is not None:
|
||||
city_to_cable_ids.setdefault(city_id, [])
|
||||
if cable_id not in city_to_cable_ids[city_id]:
|
||||
city_to_cable_ids[city_id].append(cable_id)
|
||||
|
||||
cable_id_to_name: Dict[int, str] = {}
|
||||
for r in cable_records:
|
||||
meta = r.extra_data or {}
|
||||
cable_id = meta.get("cable_id")
|
||||
if cable_id and r.name:
|
||||
cable_id_to_name[cable_id] = r.name
|
||||
|
||||
items = []
|
||||
for record in lp_records:
|
||||
lat = _safe_float(get_record_field(record, "latitude"))
|
||||
lon = _safe_float(get_record_field(record, "longitude"))
|
||||
if lat is None or lon is None:
|
||||
continue
|
||||
meta = record.extra_data or {}
|
||||
city_id = meta.get("city_id")
|
||||
cable_names = []
|
||||
if city_id in city_to_cable_ids:
|
||||
cable_names = [
|
||||
cable_id_to_name[cid]
|
||||
for cid in city_to_cable_ids[city_id]
|
||||
if cid in cable_id_to_name
|
||||
]
|
||||
items.append({
|
||||
"id": f"lp_{record.id}",
|
||||
"name": record.name or "Unknown",
|
||||
"latitude": lat,
|
||||
"longitude": lon,
|
||||
"country": get_record_field(record, "country") or "",
|
||||
"city": get_record_field(record, "city") or "",
|
||||
"cable_names": cable_names,
|
||||
})
|
||||
return {"count": len(items), "items": items}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Cables (route geometry)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@router.get("/cables")
|
||||
async def ue_cables(db: AsyncSession = Depends(get_db)):
|
||||
"""
|
||||
Returns cable route geometry.
|
||||
|
||||
Each segment is a flat array of [lon, lat] pairs.
|
||||
|
||||
Response shape:
|
||||
{
|
||||
"count": 100,
|
||||
"items": [
|
||||
{
|
||||
"id": "cable_42",
|
||||
"cable_id": "flag",
|
||||
"name": "FLAG",
|
||||
"status": "active",
|
||||
"length_km": 28000,
|
||||
"segments": [
|
||||
[[lon, lat], [lon, lat], ...]
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
"""
|
||||
records = await _fetch(db, "telegeography_cables")
|
||||
items = []
|
||||
for record in records:
|
||||
meta = record.extra_data or {}
|
||||
route_coords = meta.get("route_coordinates", [])
|
||||
segments: List[List[List[float]]] = []
|
||||
|
||||
if route_coords:
|
||||
# Support both flat [lon,lat] array and array-of-arrays
|
||||
if route_coords and isinstance(route_coords[0][0], list):
|
||||
raw_lines = route_coords
|
||||
else:
|
||||
raw_lines = [route_coords]
|
||||
|
||||
for raw_line in raw_lines:
|
||||
line = []
|
||||
for pt in raw_line:
|
||||
try:
|
||||
line.append([float(pt[0]), float(pt[1])])
|
||||
except (TypeError, ValueError, IndexError):
|
||||
continue
|
||||
if len(line) >= 2:
|
||||
segments.append(line)
|
||||
|
||||
if not segments:
|
||||
continue
|
||||
|
||||
items.append({
|
||||
"id": f"cable_{record.id}",
|
||||
"cable_id": record.source_id or record.name or "",
|
||||
"name": record.name or "Unknown",
|
||||
"status": meta.get("status", "active"),
|
||||
"length_km": _safe_float(get_record_field(record, "value")),
|
||||
"owners": meta.get("owners") or [],
|
||||
"rfs": meta.get("rfs"),
|
||||
"color": meta.get("color"),
|
||||
"segments": segments,
|
||||
})
|
||||
return {"count": len(items), "items": items}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Satellites (TLE data)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@router.get("/satellites")
|
||||
async def ue_satellites(
|
||||
limit: int = Query(default=200, ge=1, le=5000),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""
|
||||
Returns satellite TLE data for orbit propagation in UE.
|
||||
|
||||
Response shape:
|
||||
{
|
||||
"count": 200,
|
||||
"items": [
|
||||
{
|
||||
"id": "sat_42",
|
||||
"norad_id": "25544",
|
||||
"name": "ISS (ZARYA)",
|
||||
"tle_line1": "1 25544U ...",
|
||||
"tle_line2": "2 25544 ...",
|
||||
"epoch": "2026-04-14T00:00:00Z",
|
||||
"inclination": 51.6,
|
||||
"mean_motion": 15.5
|
||||
}
|
||||
]
|
||||
}
|
||||
"""
|
||||
records = await _fetch(db, "celestrak", limit)
|
||||
items = []
|
||||
for record in records:
|
||||
meta = record.extra_data or {}
|
||||
norad_id = meta.get("norad_cat_id")
|
||||
if not norad_id:
|
||||
continue
|
||||
tle1 = meta.get("tle_line1")
|
||||
tle2 = meta.get("tle_line2")
|
||||
if not tle1 or not tle2:
|
||||
tle1, tle2 = build_tle_lines_from_elements(
|
||||
norad_cat_id=norad_id,
|
||||
epoch=meta.get("epoch"),
|
||||
inclination=meta.get("inclination"),
|
||||
raan=meta.get("raan"),
|
||||
eccentricity=meta.get("eccentricity"),
|
||||
arg_of_perigee=meta.get("arg_of_perigee"),
|
||||
mean_anomaly=meta.get("mean_anomaly"),
|
||||
mean_motion=meta.get("mean_motion"),
|
||||
)
|
||||
items.append({
|
||||
"id": f"sat_{record.id}",
|
||||
"norad_id": str(norad_id),
|
||||
"name": record.name or "Unknown",
|
||||
"tle_line1": tle1 or "",
|
||||
"tle_line2": tle2 or "",
|
||||
"epoch": meta.get("epoch") or "",
|
||||
"inclination": _safe_float(meta.get("inclination")),
|
||||
"raan": _safe_float(meta.get("raan")),
|
||||
"eccentricity": _safe_float(meta.get("eccentricity")),
|
||||
"mean_motion": _safe_float(meta.get("mean_motion")),
|
||||
})
|
||||
return {"count": len(items), "items": items}
|
||||
@@ -4,34 +4,25 @@ Unified API for all visualization data sources.
|
||||
Returns GeoJSON format compatible with Three.js, CesiumJS, and Unreal Cesium.
|
||||
"""
|
||||
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from datetime import UTC, datetime
|
||||
import math
|
||||
import httpx
|
||||
from fastapi import APIRouter, HTTPException, Depends, Query, Response
|
||||
from fastapi import APIRouter, HTTPException, Depends, Query
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from sqlalchemy import select, func
|
||||
from typing import List, Dict, Any, Optional
|
||||
|
||||
from app.core.collected_data_fields import get_record_field
|
||||
from app.core.countries import get_country_centroid
|
||||
from app.core.satellite_tle import build_tle_lines_from_elements
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.db.session import get_db
|
||||
from app.models.bgp_anomaly import BGPAnomaly
|
||||
from app.models.bgp_incident import BGPIncident
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.models.vessel import VesselPosition, VesselStatic
|
||||
from app.services.bgp_collectors import build_bgp_collector_coverage
|
||||
from app.services.cable_graph import build_graph_from_data, CableGraph, haversine_distance
|
||||
from app.services.collectors.bgp_common import RIPE_RIS_COLLECTOR_COORDS
|
||||
from app.services.persistent_logs import record_system_log
|
||||
from app.core.logging import get_logger
|
||||
|
||||
router = APIRouter()
|
||||
logger = get_logger(__name__, service="api")
|
||||
TERRAIN_TILE_URL_TEMPLATE = (
|
||||
"https://s3.amazonaws.com/elevation-tiles-prod/terrarium/{z}/{x}/{y}.png"
|
||||
)
|
||||
|
||||
|
||||
# ============== Converter Functions ==============
|
||||
@@ -185,12 +176,6 @@ def convert_satellite_to_geojson(records: List[CollectedData]) -> Dict[str, Any]
|
||||
mean_motion=metadata.get("mean_motion"),
|
||||
)
|
||||
|
||||
constellation_group = _normalize_satellite_constellation_group(
|
||||
metadata.get("constellation_group"),
|
||||
record.name,
|
||||
)
|
||||
footprint_policy = _get_satellite_footprint_policy(constellation_group)
|
||||
|
||||
features.append(
|
||||
{
|
||||
"type": "Feature",
|
||||
@@ -200,8 +185,6 @@ def convert_satellite_to_geojson(records: List[CollectedData]) -> Dict[str, Any]
|
||||
"id": record.id,
|
||||
"norad_cat_id": norad_id,
|
||||
"name": record.name,
|
||||
"constellation_group": constellation_group,
|
||||
"footprint_policy": footprint_policy,
|
||||
"international_designator": metadata.get("international_designator"),
|
||||
"epoch": metadata.get("epoch"),
|
||||
"inclination": metadata.get("inclination"),
|
||||
@@ -222,31 +205,6 @@ def convert_satellite_to_geojson(records: List[CollectedData]) -> Dict[str, Any]
|
||||
return {"type": "FeatureCollection", "features": features}
|
||||
|
||||
|
||||
def _normalize_satellite_constellation_group(
|
||||
raw_group: Any,
|
||||
name: Optional[str],
|
||||
) -> Optional[str]:
|
||||
normalized_group = str(raw_group or "").strip().lower()
|
||||
if normalized_group:
|
||||
return normalized_group
|
||||
|
||||
normalized_name = str(name or "").strip().upper()
|
||||
if normalized_name.startswith("STARLINK"):
|
||||
return "starlink"
|
||||
if normalized_name.startswith("IRIDIUM"):
|
||||
return "iridium-next"
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def _get_satellite_footprint_policy(constellation_group: Optional[str]) -> str:
|
||||
if constellation_group == "starlink":
|
||||
return "starlink_ground_footprint"
|
||||
if constellation_group == "iridium-next":
|
||||
return "iridium_coverage_ring"
|
||||
return "none"
|
||||
|
||||
|
||||
def _current_collected_data_stmt(source: str):
|
||||
return (
|
||||
select(CollectedData)
|
||||
@@ -401,317 +359,6 @@ def convert_gpu_cluster_to_geojson(records: List[CollectedData]) -> Dict[str, An
|
||||
return {"type": "FeatureCollection", "features": features}
|
||||
|
||||
|
||||
def _parse_float(value: Any) -> Optional[float]:
|
||||
try:
|
||||
if value in (None, ""):
|
||||
return None
|
||||
return float(value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
COMPUTE_CENTER_COORDINATE_HINTS = (
|
||||
("el capitan", 37.6819, -121.7681),
|
||||
("livermore", 37.6819, -121.7681),
|
||||
("llnl", 37.6819, -121.7681),
|
||||
("lawrence livermore", 37.6819, -121.7681),
|
||||
("frontier", 35.9319, -84.3107),
|
||||
("oak ridge", 35.9319, -84.3107),
|
||||
("ornl", 35.9319, -84.3107),
|
||||
("aurora", 41.7130, -87.9820),
|
||||
("argonne", 41.7130, -87.9820),
|
||||
("anl", 41.7130, -87.9820),
|
||||
("fugaku", 34.6953, 135.1974),
|
||||
("kobe", 34.6953, 135.1974),
|
||||
("riken", 34.6953, 135.1974),
|
||||
("summit", 35.9319, -84.3107),
|
||||
("leonardo", 44.4949, 11.3426),
|
||||
("bologna", 44.4949, 11.3426),
|
||||
("alps", 46.0037, 8.9511),
|
||||
("lugano", 46.0037, 8.9511),
|
||||
("sunway taihulight", 31.4912, 120.3119),
|
||||
("wuxi", 31.4912, 120.3119),
|
||||
("tianhe-2", 23.1291, 113.2644),
|
||||
("tianhe-2a", 23.1291, 113.2644),
|
||||
("guangzhou", 23.1291, 113.2644),
|
||||
("colossus", 35.1495, -90.0490),
|
||||
("memphis", 35.1495, -90.0490),
|
||||
("xai", 35.1495, -90.0490),
|
||||
)
|
||||
|
||||
|
||||
def _normalize_hint_text(*parts: Any) -> str:
|
||||
return " ".join(
|
||||
str(part).strip().lower()
|
||||
for part in parts
|
||||
if part not in (None, "")
|
||||
)
|
||||
|
||||
|
||||
def _resolve_compute_center_coordinates(
|
||||
record: CollectedData,
|
||||
metadata: Dict[str, Any],
|
||||
) -> Dict[str, Any]:
|
||||
latitude = _parse_float(get_record_field(record, "latitude"))
|
||||
longitude = _parse_float(get_record_field(record, "longitude"))
|
||||
if latitude not in (None, 0.0) and longitude not in (None, 0.0):
|
||||
return {
|
||||
"latitude": latitude,
|
||||
"longitude": longitude,
|
||||
"location_precision": "precise",
|
||||
"geography_mode": "source_coordinates",
|
||||
"is_estimated": False,
|
||||
"estimated_reason": None,
|
||||
}
|
||||
|
||||
hint_text = _normalize_hint_text(
|
||||
record.name,
|
||||
get_record_field(record, "city"),
|
||||
get_record_field(record, "country"),
|
||||
metadata.get("site"),
|
||||
metadata.get("organization"),
|
||||
metadata.get("operator"),
|
||||
)
|
||||
for needle, resolved_latitude, resolved_longitude in COMPUTE_CENTER_COORDINATE_HINTS:
|
||||
if needle in hint_text:
|
||||
return {
|
||||
"latitude": resolved_latitude,
|
||||
"longitude": resolved_longitude,
|
||||
"location_precision": "estimated_site",
|
||||
"geography_mode": "site_hint",
|
||||
"is_estimated": True,
|
||||
"estimated_reason": f"Matched known site hint: {needle}",
|
||||
}
|
||||
|
||||
centroid = get_country_centroid(get_record_field(record, "country"))
|
||||
if centroid:
|
||||
return {
|
||||
"latitude": centroid.get("latitude"),
|
||||
"longitude": centroid.get("longitude"),
|
||||
"location_precision": "estimated_country",
|
||||
"geography_mode": "country_centroid",
|
||||
"is_estimated": True,
|
||||
"estimated_reason": "Estimated from country centroid",
|
||||
}
|
||||
|
||||
return {
|
||||
"latitude": latitude,
|
||||
"longitude": longitude,
|
||||
"location_precision": "unknown",
|
||||
"geography_mode": "unknown",
|
||||
"is_estimated": True,
|
||||
"estimated_reason": "No resolvable location hints",
|
||||
}
|
||||
|
||||
|
||||
def _normalize_capacity_band(capacity_value: Optional[float], capacity_unit: str) -> str:
|
||||
if capacity_value is None:
|
||||
return "unknown"
|
||||
|
||||
unit = str(capacity_unit or "").strip().lower()
|
||||
if unit in {"pflop/s", "pflops", "pflop"}:
|
||||
normalized_tflops = capacity_value * 1000
|
||||
elif unit in {"gflop/s", "gflops", "gflop"}:
|
||||
normalized_tflops = capacity_value
|
||||
else:
|
||||
normalized_tflops = capacity_value
|
||||
|
||||
if normalized_tflops >= 1_000_000:
|
||||
return "exascale"
|
||||
if normalized_tflops >= 100_000:
|
||||
return "ultra"
|
||||
if normalized_tflops >= 10_000:
|
||||
return "large"
|
||||
if normalized_tflops > 0:
|
||||
return "regional"
|
||||
return "unknown"
|
||||
|
||||
|
||||
def convert_compute_centers_to_geojson(records: List[CollectedData]) -> Dict[str, Any]:
|
||||
"""Convert compute infrastructure records into a unified GeoJSON layer."""
|
||||
features = []
|
||||
|
||||
for record in records:
|
||||
metadata = record.extra_data or {}
|
||||
coordinate_info = _resolve_compute_center_coordinates(record, metadata)
|
||||
latitude = coordinate_info.get("latitude")
|
||||
longitude = coordinate_info.get("longitude")
|
||||
site_type = (
|
||||
"supercomputer"
|
||||
if record.source == "top500" or record.data_type == "supercomputer"
|
||||
else "gpu_cluster"
|
||||
)
|
||||
if latitude in (None, 0.0) or longitude in (None, 0.0):
|
||||
continue
|
||||
|
||||
if site_type == "supercomputer":
|
||||
capacity_value = _parse_float(get_record_field(record, "rmax"))
|
||||
capacity_unit = "GFlops"
|
||||
else:
|
||||
capacity_value = _parse_float(get_record_field(record, "value"))
|
||||
capacity_unit = str(get_record_field(record, "unit") or "TFlop/s")
|
||||
|
||||
vendor = (
|
||||
metadata.get("manufacturer")
|
||||
or metadata.get("vendor")
|
||||
or metadata.get("gpu_type")
|
||||
)
|
||||
operator = (
|
||||
metadata.get("organization")
|
||||
or metadata.get("operator")
|
||||
or metadata.get("owner")
|
||||
)
|
||||
rank = metadata.get("rank")
|
||||
if rank in (None, "") and site_type == "supercomputer":
|
||||
rank = get_record_field(record, "rank")
|
||||
|
||||
updated_at = to_iso8601_utc(record.reference_date or record.collected_at)
|
||||
|
||||
features.append(
|
||||
{
|
||||
"type": "Feature",
|
||||
"id": record.id,
|
||||
"geometry": {
|
||||
"type": "Point",
|
||||
"coordinates": [longitude or 0, latitude or 0],
|
||||
},
|
||||
"properties": {
|
||||
"id": record.id,
|
||||
"source_id": record.source_id,
|
||||
"name": record.name,
|
||||
"site_type": site_type,
|
||||
"country": get_record_field(record, "country"),
|
||||
"city": get_record_field(record, "city"),
|
||||
"latitude": latitude,
|
||||
"longitude": longitude,
|
||||
"operator": operator,
|
||||
"vendor": vendor,
|
||||
"capacity_value": capacity_value,
|
||||
"capacity_unit": capacity_unit,
|
||||
"capacity_band": _normalize_capacity_band(capacity_value, capacity_unit),
|
||||
"rank": rank,
|
||||
"gpu_count": metadata.get("gpu_count"),
|
||||
"gpu_type": metadata.get("gpu_type"),
|
||||
"cores": get_record_field(record, "cores"),
|
||||
"power": get_record_field(record, "power"),
|
||||
"source": record.source,
|
||||
"updated_at": updated_at,
|
||||
"status": "observed",
|
||||
"location_precision": coordinate_info.get("location_precision"),
|
||||
"geography_mode": coordinate_info.get("geography_mode"),
|
||||
"is_estimated": coordinate_info.get("is_estimated", False),
|
||||
"estimated_reason": coordinate_info.get("estimated_reason"),
|
||||
"data_type": "compute_center",
|
||||
"metadata": metadata,
|
||||
},
|
||||
}
|
||||
)
|
||||
|
||||
return {"type": "FeatureCollection", "features": features}
|
||||
|
||||
|
||||
VESSEL_TYPE_FILTERS = {
|
||||
"cargo": lambda props: str(props.get("vessel_type_name", "")).lower() == "cargo"
|
||||
or 70 <= int(props.get("vessel_type") or -1) <= 79,
|
||||
"tanker": lambda props: str(props.get("vessel_type_name", "")).lower() == "tanker"
|
||||
or 80 <= int(props.get("vessel_type") or -1) <= 89,
|
||||
"passenger": lambda props: str(props.get("vessel_type_name", "")).lower() == "passenger"
|
||||
or 60 <= int(props.get("vessel_type") or -1) <= 69,
|
||||
"fishing": lambda props: str(props.get("vessel_type_name", "")).lower() == "fishing"
|
||||
or int(props.get("vessel_type") or -1) == 30,
|
||||
"military": lambda props: str(props.get("vessel_type_name", "")).lower() == "military"
|
||||
or int(props.get("vessel_type") or -1) == 35,
|
||||
"other": lambda props: str(props.get("vessel_type_name", "")).lower()
|
||||
not in {"cargo", "tanker", "passenger", "fishing", "military"},
|
||||
}
|
||||
|
||||
|
||||
def convert_vessels_to_geojson(rows: List[Any]) -> Dict[str, Any]:
|
||||
features = []
|
||||
for position, static in rows:
|
||||
if position.lat is None or position.lon is None:
|
||||
continue
|
||||
props = {
|
||||
"mmsi": position.mmsi,
|
||||
"name": getattr(static, "name", None) or f"MMSI {position.mmsi}",
|
||||
"callsign": getattr(static, "callsign", None),
|
||||
"imo": getattr(static, "imo", None),
|
||||
"vessel_type": getattr(static, "vessel_type", None),
|
||||
"vessel_type_name": getattr(static, "vessel_type_name", None) or "Other",
|
||||
"flag": getattr(static, "flag", None),
|
||||
"length": getattr(static, "length", None),
|
||||
"width": getattr(static, "width", None),
|
||||
"draught": getattr(static, "draught", None),
|
||||
"sog": position.sog,
|
||||
"cog": position.cog,
|
||||
"heading": position.heading,
|
||||
"nav_status": position.nav_status,
|
||||
"received_at": to_iso8601_utc(position.received_at),
|
||||
"data_type": "vessel",
|
||||
}
|
||||
features.append(
|
||||
{
|
||||
"type": "Feature",
|
||||
"id": position.mmsi,
|
||||
"geometry": {
|
||||
"type": "Point",
|
||||
"coordinates": [position.lon, position.lat],
|
||||
},
|
||||
"properties": props,
|
||||
}
|
||||
)
|
||||
|
||||
return {"type": "FeatureCollection", "features": features}
|
||||
|
||||
|
||||
def _parse_bbox(value: Optional[str]) -> tuple[float, float, float, float] | None:
|
||||
if not value:
|
||||
return None
|
||||
parts = [part.strip() for part in value.split(",")]
|
||||
if len(parts) != 4:
|
||||
raise HTTPException(status_code=400, detail="bbox must be lon_min,lat_min,lon_max,lat_max")
|
||||
try:
|
||||
lon_min, lat_min, lon_max, lat_max = [float(part) for part in parts]
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=400, detail="bbox values must be numbers") from exc
|
||||
if lat_min > lat_max:
|
||||
lat_min, lat_max = lat_max, lat_min
|
||||
if lon_min > lon_max:
|
||||
lon_min, lon_max = lon_max, lon_min
|
||||
return lon_min, lat_min, lon_max, lat_max
|
||||
|
||||
|
||||
def _matches_vessel_type(props: dict[str, Any], requested_types: set[str]) -> bool:
|
||||
if not requested_types:
|
||||
return True
|
||||
for requested_type in requested_types:
|
||||
predicate = VESSEL_TYPE_FILTERS.get(requested_type)
|
||||
if predicate and predicate(props):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _build_vessel_stats(features: List[dict[str, Any]]) -> dict[str, Any]:
|
||||
by_type: dict[str, int] = {}
|
||||
underway = 0
|
||||
anchored_or_moored = 0
|
||||
for feature in features:
|
||||
props = feature.get("properties", {})
|
||||
vessel_type = str(props.get("vessel_type_name") or "Other")
|
||||
by_type[vessel_type] = by_type.get(vessel_type, 0) + 1
|
||||
nav_status = props.get("nav_status")
|
||||
if nav_status in (1, 5):
|
||||
anchored_or_moored += 1
|
||||
else:
|
||||
underway += 1
|
||||
return {
|
||||
"total": len(features),
|
||||
"by_type": by_type,
|
||||
"underway": underway,
|
||||
"anchored_or_moored": anchored_or_moored,
|
||||
}
|
||||
|
||||
|
||||
def convert_bgp_anomalies_to_geojson(
|
||||
records: List[BGPAnomaly],
|
||||
geography_hints: Optional[Dict[str, Dict[str, Any]]] = None,
|
||||
@@ -1129,41 +776,15 @@ async def get_cables_geojson(db: AsyncSession = Depends(get_db)):
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.exception_event(
|
||||
"Failed to build cables GeoJSON response",
|
||||
event="visualization.cables.load_failed",
|
||||
context={"error": str(e)},
|
||||
)
|
||||
await record_system_log(
|
||||
source="backend",
|
||||
service="api",
|
||||
module=__name__,
|
||||
event="visualization.cables.load_failed",
|
||||
level="error",
|
||||
message="Failed to build cables GeoJSON response",
|
||||
category="visualization",
|
||||
context={"error": str(e)},
|
||||
)
|
||||
raise HTTPException(status_code=500, detail=f"Internal error: {str(e)}")
|
||||
|
||||
|
||||
@router.get("/geo/landing-points")
|
||||
async def get_landing_points_geojson(db: AsyncSession = Depends(get_db)):
|
||||
try:
|
||||
records_by_source = await _load_current_collected_data_by_sources(
|
||||
db,
|
||||
[
|
||||
"arcgis_landing_points",
|
||||
"arcgis_cable_landing_relation",
|
||||
"arcgis_cables",
|
||||
],
|
||||
)
|
||||
records = records_by_source.get("arcgis_landing_points", [])
|
||||
relation_records = records_by_source.get(
|
||||
"arcgis_cable_landing_relation",
|
||||
[],
|
||||
)
|
||||
cable_records = records_by_source.get("arcgis_cables", [])
|
||||
records = await _load_current_collected_data(db, "arcgis_landing_points")
|
||||
relation_records = await _load_current_collected_data(db, "arcgis_cable_landing_relation")
|
||||
cable_records = await _load_current_collected_data(db, "arcgis_cables")
|
||||
|
||||
city_to_cable_ids_map, cable_id_to_name_map = _build_landing_point_cable_maps(
|
||||
relation_records,
|
||||
@@ -1180,68 +801,9 @@ async def get_landing_points_geojson(db: AsyncSession = Depends(get_db)):
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.exception_event(
|
||||
"Failed to build landing points GeoJSON response",
|
||||
event="visualization.landing_points.load_failed",
|
||||
context={"error": str(e)},
|
||||
)
|
||||
await record_system_log(
|
||||
source="backend",
|
||||
service="api",
|
||||
module=__name__,
|
||||
event="visualization.landing_points.load_failed",
|
||||
level="error",
|
||||
message="Failed to build landing points GeoJSON response",
|
||||
category="visualization",
|
||||
context={"error": str(e)},
|
||||
)
|
||||
raise HTTPException(status_code=500, detail=f"Internal error: {str(e)}")
|
||||
|
||||
|
||||
@router.get("/terrain/terrarium/{z}/{x}/{y}.png")
|
||||
async def get_terrarium_tile(z: int, x: int, y: int):
|
||||
"""Proxy Terrarium elevation tiles through the backend to avoid browser CORS issues."""
|
||||
if z < 0 or x < 0 or y < 0:
|
||||
raise HTTPException(status_code=400, detail="Invalid terrain tile coordinates")
|
||||
|
||||
url = TERRAIN_TILE_URL_TEMPLATE.format(z=z, x=x, y=y)
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(
|
||||
timeout=20.0,
|
||||
follow_redirects=True,
|
||||
) as client:
|
||||
upstream = await client.get(url)
|
||||
upstream.raise_for_status()
|
||||
except httpx.HTTPStatusError as exc:
|
||||
raise HTTPException(
|
||||
status_code=exc.response.status_code,
|
||||
detail=f"Terrain tile upstream error: {exc.response.status_code}",
|
||||
) from exc
|
||||
except httpx.HTTPError as exc:
|
||||
raise HTTPException(
|
||||
status_code=502,
|
||||
detail=f"Terrain tile fetch failed: {exc}",
|
||||
) from exc
|
||||
|
||||
cache_control = upstream.headers.get("cache-control") or "public, max-age=86400"
|
||||
etag = upstream.headers.get("etag")
|
||||
last_modified = upstream.headers.get("last-modified")
|
||||
headers = {
|
||||
"Cache-Control": cache_control,
|
||||
}
|
||||
if etag:
|
||||
headers["ETag"] = etag
|
||||
if last_modified:
|
||||
headers["Last-Modified"] = last_modified
|
||||
|
||||
return Response(
|
||||
content=upstream.content,
|
||||
media_type=upstream.headers.get("content-type", "image/png"),
|
||||
headers=headers,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/geo/all")
|
||||
async def get_all_geojson(db: AsyncSession = Depends(get_db)):
|
||||
records_by_source = await _load_current_collected_data_by_sources(
|
||||
@@ -1354,184 +916,6 @@ async def get_gpu_clusters_geojson(
|
||||
}
|
||||
|
||||
|
||||
@router.get("/geo/compute-centers")
|
||||
async def get_compute_centers_geojson(
|
||||
limit: int = Query(200, ge=1, le=1000),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""获取统一算力中心 GeoJSON 数据"""
|
||||
records_by_source = await _load_current_collected_data_by_sources(
|
||||
db,
|
||||
["top500", "epoch_ai_gpu"],
|
||||
)
|
||||
records = _filter_known_records(
|
||||
records_by_source.get("top500", []) + records_by_source.get("epoch_ai_gpu", []),
|
||||
)
|
||||
if limit is not None:
|
||||
records = records[:limit]
|
||||
|
||||
if not records:
|
||||
return {
|
||||
"type": "FeatureCollection",
|
||||
"features": [],
|
||||
"count": 0,
|
||||
"stats": {
|
||||
"total": 0,
|
||||
"supercomputers": 0,
|
||||
"gpu_clusters": 0,
|
||||
},
|
||||
}
|
||||
|
||||
geojson = convert_compute_centers_to_geojson(records)
|
||||
features = geojson.get("features", [])
|
||||
return {
|
||||
**geojson,
|
||||
"count": len(features),
|
||||
"stats": {
|
||||
"total": len(features),
|
||||
"supercomputers": sum(
|
||||
1 for feature in features
|
||||
if feature.get("properties", {}).get("site_type") == "supercomputer"
|
||||
),
|
||||
"gpu_clusters": sum(
|
||||
1 for feature in features
|
||||
if feature.get("properties", {}).get("site_type") == "gpu_cluster"
|
||||
),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@router.get("/geo/vessels")
|
||||
async def get_vessels_geojson(
|
||||
bbox: Optional[str] = Query(
|
||||
None,
|
||||
description="Viewport bbox as lon_min,lat_min,lon_max,lat_max",
|
||||
),
|
||||
type: Optional[str] = Query(
|
||||
None,
|
||||
description="Comma-separated vessel types: cargo,tanker,passenger,fishing,military,other",
|
||||
),
|
||||
limit: int = Query(5000, ge=1, le=50000),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""Return latest vessel positions as GeoJSON points."""
|
||||
latest_times = (
|
||||
select(
|
||||
VesselPosition.mmsi.label("mmsi"),
|
||||
func.max(VesselPosition.received_at).label("received_at"),
|
||||
)
|
||||
.group_by(VesselPosition.mmsi)
|
||||
.subquery()
|
||||
)
|
||||
stmt = (
|
||||
select(VesselPosition, VesselStatic)
|
||||
.join(
|
||||
latest_times,
|
||||
(VesselPosition.mmsi == latest_times.c.mmsi)
|
||||
& (VesselPosition.received_at == latest_times.c.received_at),
|
||||
)
|
||||
.outerjoin(VesselStatic, VesselStatic.mmsi == VesselPosition.mmsi)
|
||||
.order_by(VesselPosition.received_at.desc())
|
||||
.limit(limit)
|
||||
)
|
||||
|
||||
parsed_bbox = _parse_bbox(bbox)
|
||||
if parsed_bbox is not None:
|
||||
lon_min, lat_min, lon_max, lat_max = parsed_bbox
|
||||
stmt = stmt.where(
|
||||
VesselPosition.lon >= lon_min,
|
||||
VesselPosition.lon <= lon_max,
|
||||
VesselPosition.lat >= lat_min,
|
||||
VesselPosition.lat <= lat_max,
|
||||
)
|
||||
|
||||
result = await db.execute(stmt)
|
||||
rows = list(result.all())
|
||||
geojson = convert_vessels_to_geojson(rows)
|
||||
requested_types = {
|
||||
item.strip().lower()
|
||||
for item in (type or "").split(",")
|
||||
if item.strip()
|
||||
}
|
||||
if requested_types:
|
||||
geojson["features"] = [
|
||||
feature
|
||||
for feature in geojson.get("features", [])
|
||||
if _matches_vessel_type(feature.get("properties", {}), requested_types)
|
||||
]
|
||||
|
||||
features = geojson.get("features", [])
|
||||
return {
|
||||
**geojson,
|
||||
"count": len(features),
|
||||
"stats": _build_vessel_stats(features),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/vessels/{mmsi}")
|
||||
async def get_vessel_detail(mmsi: int, db: AsyncSession = Depends(get_db)):
|
||||
latest_position_stmt = (
|
||||
select(VesselPosition)
|
||||
.where(VesselPosition.mmsi == mmsi)
|
||||
.order_by(VesselPosition.received_at.desc())
|
||||
.limit(1)
|
||||
)
|
||||
static = await db.get(VesselStatic, mmsi)
|
||||
result = await db.execute(latest_position_stmt)
|
||||
position = result.scalar_one_or_none()
|
||||
if position is None:
|
||||
raise HTTPException(status_code=404, detail="Vessel not found")
|
||||
geojson = convert_vessels_to_geojson([(position, static)])
|
||||
return {
|
||||
**(geojson["features"][0]["properties"]),
|
||||
"latitude": position.lat,
|
||||
"longitude": position.lon,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/vessels/{mmsi}/track")
|
||||
async def get_vessel_track(
|
||||
mmsi: int,
|
||||
hours: int = Query(6, ge=1, le=24),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
cutoff = datetime.now(UTC) - timedelta(hours=hours)
|
||||
result = await db.execute(
|
||||
select(VesselPosition)
|
||||
.where(VesselPosition.mmsi == mmsi)
|
||||
.where(VesselPosition.received_at >= cutoff)
|
||||
.order_by(VesselPosition.received_at.asc())
|
||||
)
|
||||
positions = list(result.scalars().all())
|
||||
if not positions:
|
||||
return {
|
||||
"type": "FeatureCollection",
|
||||
"features": [],
|
||||
"count": 0,
|
||||
}
|
||||
|
||||
return {
|
||||
"type": "FeatureCollection",
|
||||
"features": [
|
||||
{
|
||||
"type": "Feature",
|
||||
"geometry": {
|
||||
"type": "LineString",
|
||||
"coordinates": [[position.lon, position.lat] for position in positions],
|
||||
},
|
||||
"properties": {
|
||||
"mmsi": mmsi,
|
||||
"hours": hours,
|
||||
"point_count": len(positions),
|
||||
"start_at": to_iso8601_utc(positions[0].received_at),
|
||||
"end_at": to_iso8601_utc(positions[-1].received_at),
|
||||
},
|
||||
}
|
||||
],
|
||||
"count": 1,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/geo/bgp-anomalies")
|
||||
async def get_bgp_anomalies_geojson(
|
||||
severity: Optional[str] = Query(None),
|
||||
@@ -1587,76 +971,6 @@ async def get_bgp_collectors_geojson(db: AsyncSession = Depends(get_db)):
|
||||
return {**geojson, "count": len(geojson.get("features", []))}
|
||||
|
||||
|
||||
@router.get("/geo/summary")
|
||||
async def get_visualization_geo_summary(db: AsyncSession = Depends(get_db)):
|
||||
"""Return lightweight Earth HUD counts without loading layer GeoJSON payloads."""
|
||||
records_by_source = await _load_current_collected_data_by_sources(
|
||||
db,
|
||||
[
|
||||
"arcgis_cables",
|
||||
"arcgis_landing_points",
|
||||
"celestrak_tle",
|
||||
"top500",
|
||||
"epoch_ai_gpu",
|
||||
],
|
||||
)
|
||||
|
||||
cables = convert_cable_to_geojson(records_by_source.get("arcgis_cables", []))
|
||||
landing_points = convert_landing_point_to_geojson(
|
||||
records_by_source.get("arcgis_landing_points", []),
|
||||
)
|
||||
satellites = convert_satellite_to_geojson(
|
||||
_filter_known_records(records_by_source.get("celestrak_tle", [])),
|
||||
)
|
||||
compute_centers = convert_compute_centers_to_geojson(
|
||||
_filter_known_records(
|
||||
records_by_source.get("top500", [])
|
||||
+ records_by_source.get("epoch_ai_gpu", []),
|
||||
),
|
||||
)
|
||||
compute_features = compute_centers.get("features", [])
|
||||
|
||||
active_incident_result = await db.execute(
|
||||
select(func.count(BGPIncident.id)).where(BGPIncident.status == "active"),
|
||||
)
|
||||
active_anomaly_result = await db.execute(
|
||||
select(func.count(BGPAnomaly.id)).where(BGPAnomaly.status == "active"),
|
||||
)
|
||||
active_incident_count = int(active_incident_result.scalar() or 0)
|
||||
active_anomaly_count = int(active_anomaly_result.scalar() or 0)
|
||||
bgp_collectors = await build_bgp_collector_coverage(
|
||||
db,
|
||||
source_filter=("ris_live_bgp", "bgpstream_bgp"),
|
||||
)
|
||||
vessel_count_result = await db.execute(
|
||||
select(func.count(func.distinct(VesselPosition.mmsi))),
|
||||
)
|
||||
vessel_count = int(vessel_count_result.scalar() or 0)
|
||||
|
||||
return {
|
||||
"generated_at": to_iso8601_utc(datetime.now(UTC)),
|
||||
"stats": {
|
||||
"cable_count": len(cables.get("features", [])),
|
||||
"landing_point_count": len(landing_points.get("features", [])),
|
||||
"satellite_count": len(satellites.get("features", [])),
|
||||
"compute_center_count": len(compute_features),
|
||||
"vessel_count": vessel_count,
|
||||
"supercomputer_count": sum(
|
||||
1 for feature in compute_features
|
||||
if feature.get("properties", {}).get("site_type") == "supercomputer"
|
||||
),
|
||||
"gpu_cluster_count": sum(
|
||||
1 for feature in compute_features
|
||||
if feature.get("properties", {}).get("site_type") == "gpu_cluster"
|
||||
),
|
||||
"bgp_event_count": active_incident_count or active_anomaly_count,
|
||||
"bgp_incident_count": active_incident_count,
|
||||
"bgp_anomaly_count": active_anomaly_count,
|
||||
"bgp_collector_count": len([item for item in bgp_collectors if item.get("collector")]),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@router.get("/all")
|
||||
async def get_all_visualization_data(db: AsyncSession = Depends(get_db)):
|
||||
"""获取所有可视化数据的统一端点
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
from datetime import UTC, datetime
|
||||
from typing import Optional
|
||||
|
||||
@@ -9,11 +10,10 @@ from fastapi import APIRouter, WebSocket, WebSocketDisconnect, Query
|
||||
from jose import jwt, JWTError
|
||||
|
||||
from app.core.config import settings
|
||||
from app.core.logging import get_logger
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.core.websocket.manager import manager
|
||||
|
||||
logger = get_logger(__name__, service="api")
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
@@ -22,18 +22,11 @@ async def authenticate_token(token: str) -> Optional[dict]:
|
||||
try:
|
||||
payload = jwt.decode(token, settings.SECRET_KEY, algorithms=[settings.ALGORITHM])
|
||||
if payload.get("type") != "access":
|
||||
logger.warning_event(
|
||||
"WebSocket auth failed: wrong token type",
|
||||
event="auth.websocket.invalid_token_type",
|
||||
)
|
||||
logger.warning(f"WebSocket auth failed: wrong token type")
|
||||
return None
|
||||
return payload
|
||||
except JWTError as e:
|
||||
logger.warning_event(
|
||||
"WebSocket auth failed",
|
||||
event="auth.websocket.decode_failed",
|
||||
context={"error": str(e)},
|
||||
)
|
||||
logger.warning(f"WebSocket auth failed: {e}")
|
||||
return None
|
||||
|
||||
|
||||
@@ -43,17 +36,10 @@ async def websocket_endpoint(
|
||||
token: str = Query(...),
|
||||
):
|
||||
"""WebSocket endpoint for real-time data"""
|
||||
logger.info_event(
|
||||
"WebSocket connection attempt",
|
||||
event="auth.websocket.connection_attempt",
|
||||
context={"token_preview": f"{token[:8]}..."},
|
||||
)
|
||||
logger.info(f"WebSocket connection attempt with token: {token[:20]}...")
|
||||
payload = await authenticate_token(token)
|
||||
if payload is None:
|
||||
logger.warning_event(
|
||||
"WebSocket authentication failed, closing connection",
|
||||
event="auth.websocket.connection_rejected",
|
||||
)
|
||||
logger.warning("WebSocket authentication failed, closing connection")
|
||||
await websocket.close(code=4001)
|
||||
return
|
||||
|
||||
|
||||
@@ -1,15 +1,15 @@
|
||||
"""Redis caching service"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
from datetime import timedelta
|
||||
from typing import Optional, Any
|
||||
|
||||
import redis
|
||||
|
||||
from app.core.config import settings
|
||||
from app.core.logging import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# Lazy Redis client initialization
|
||||
@@ -47,7 +47,7 @@ class CacheService:
|
||||
return json.loads(value)
|
||||
return None
|
||||
except Exception as e:
|
||||
logger.warning_event("Cache get error", event="cache.get.failed", context={"error": str(e)})
|
||||
logger.warning(f"Cache get error: {e}")
|
||||
return None
|
||||
|
||||
def set(
|
||||
@@ -61,7 +61,7 @@ class CacheService:
|
||||
serialized = json.dumps(value, default=str)
|
||||
return self.client.setex(key, expire_seconds, serialized)
|
||||
except Exception as e:
|
||||
logger.warning_event("Cache set error", event="cache.set.failed", context={"error": str(e)})
|
||||
logger.warning(f"Cache set error: {e}")
|
||||
return False
|
||||
|
||||
def delete(self, key: str) -> bool:
|
||||
@@ -69,7 +69,7 @@ class CacheService:
|
||||
try:
|
||||
return self.client.delete(key) > 0
|
||||
except Exception as e:
|
||||
logger.warning_event("Cache delete error", event="cache.delete.failed", context={"error": str(e)})
|
||||
logger.warning(f"Cache delete error: {e}")
|
||||
return False
|
||||
|
||||
def delete_pattern(self, pattern: str) -> int:
|
||||
@@ -80,7 +80,7 @@ class CacheService:
|
||||
return self.client.delete(*keys)
|
||||
return 0
|
||||
except Exception as e:
|
||||
logger.warning_event("Cache delete_pattern error", event="cache.delete_pattern.failed", context={"error": str(e)})
|
||||
logger.warning(f"Cache delete_pattern error: {e}")
|
||||
return 0
|
||||
|
||||
def get_or_set(
|
||||
|
||||
@@ -30,8 +30,6 @@ COLLECTOR_URL_KEYS = {
|
||||
"iptoasn_prefix_geo": "iptoasn.combined_url",
|
||||
"opengeofeed_prefix_geo": "opengeofeed.public_csv_url",
|
||||
"nro_delegated_prefix_geo": "nro.delegated_stats_url",
|
||||
"news_live_streams": "news_live_streams.channels_url",
|
||||
"barentswatch_vessels": "barentswatch_vessels.url",
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -86,15 +86,3 @@ opengeofeed:
|
||||
nro:
|
||||
# NRO delegated stats 下载地址
|
||||
delegated_stats_url: "https://ftp.ripe.net/pub/stats/ripencc/nro-stats/latest/nro-delegated-stats"
|
||||
|
||||
news_live_streams:
|
||||
# IPTV-org 频道元数据 JSON
|
||||
channels_url: "https://iptv-org.github.io/api/channels.json"
|
||||
# IPTV-org 频道播放流 JSON
|
||||
streams_url: "https://iptv-org.github.io/api/streams.json"
|
||||
# IPTV-org 台标 JSON
|
||||
logos_url: "https://iptv-org.github.io/api/logos.json"
|
||||
|
||||
barentswatch_vessels:
|
||||
# BarentsWatch Live AIS latest combined endpoint. Requires an AIS bearer token.
|
||||
url: "https://live.ais.barentswatch.no/v1/latest/combined"
|
||||
@@ -4,246 +4,163 @@ DEFAULT_DATASOURCES = {
|
||||
"top500": {
|
||||
"id": 1,
|
||||
"name": "TOP500 Supercomputers",
|
||||
"display_name": "TOP500 超算榜单",
|
||||
"module": "L1",
|
||||
"priority": "P0",
|
||||
"frequency_minutes": 240,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"epoch_ai_gpu": {
|
||||
"id": 2,
|
||||
"name": "Epoch AI GPU Clusters",
|
||||
"display_name": "Epoch AI GPU 集群",
|
||||
"module": "L1",
|
||||
"priority": "P0",
|
||||
"frequency_minutes": 360,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"huggingface_models": {
|
||||
"id": 3,
|
||||
"name": "HuggingFace Models",
|
||||
"display_name": "Hugging Face 模型",
|
||||
"module": "L2",
|
||||
"priority": "P1",
|
||||
"frequency_minutes": 720,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"huggingface_datasets": {
|
||||
"id": 4,
|
||||
"name": "HuggingFace Datasets",
|
||||
"display_name": "Hugging Face 数据集",
|
||||
"module": "L2",
|
||||
"priority": "P1",
|
||||
"frequency_minutes": 720,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"huggingface_spaces": {
|
||||
"id": 5,
|
||||
"name": "HuggingFace Spaces",
|
||||
"display_name": "Hugging Face Spaces",
|
||||
"module": "L2",
|
||||
"priority": "P2",
|
||||
"frequency_minutes": 1440,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"peeringdb_ixp": {
|
||||
"id": 6,
|
||||
"name": "PeeringDB IXP",
|
||||
"display_name": "PeeringDB 交换中心",
|
||||
"module": "L2",
|
||||
"priority": "P1",
|
||||
"frequency_minutes": 1440,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"peeringdb_network": {
|
||||
"id": 7,
|
||||
"name": "PeeringDB Networks",
|
||||
"display_name": "PeeringDB 网络",
|
||||
"module": "L2",
|
||||
"priority": "P2",
|
||||
"frequency_minutes": 2880,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"peeringdb_facility": {
|
||||
"id": 8,
|
||||
"name": "PeeringDB Facilities",
|
||||
"display_name": "PeeringDB 设施",
|
||||
"module": "L2",
|
||||
"priority": "P2",
|
||||
"frequency_minutes": 2880,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"telegeography_cables": {
|
||||
"id": 9,
|
||||
"name": "Submarine Cables",
|
||||
"display_name": "海底光缆",
|
||||
"module": "L2",
|
||||
"priority": "P1",
|
||||
"frequency_minutes": 10080,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"telegeography_landing": {
|
||||
"id": 10,
|
||||
"name": "Cable Landing Points",
|
||||
"display_name": "光缆登陆点",
|
||||
"module": "L2",
|
||||
"priority": "P2",
|
||||
"frequency_minutes": 10080,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"telegeography_systems": {
|
||||
"id": 11,
|
||||
"name": "Cable Systems",
|
||||
"display_name": "光缆系统",
|
||||
"module": "L2",
|
||||
"priority": "P2",
|
||||
"frequency_minutes": 10080,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"arcgis_cables": {
|
||||
"id": 15,
|
||||
"name": "ArcGIS Submarine Cables",
|
||||
"display_name": "ArcGIS 海底光缆",
|
||||
"module": "L2",
|
||||
"priority": "P1",
|
||||
"frequency_minutes": 10080,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"arcgis_landing_points": {
|
||||
"id": 16,
|
||||
"name": "ArcGIS Landing Points",
|
||||
"display_name": "ArcGIS 登陆点",
|
||||
"module": "L2",
|
||||
"priority": "P1",
|
||||
"frequency_minutes": 10080,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"arcgis_cable_landing_relation": {
|
||||
"id": 17,
|
||||
"name": "ArcGIS Cable-Landing Relations",
|
||||
"display_name": "ArcGIS 光缆登陆关系",
|
||||
"module": "L2",
|
||||
"priority": "P1",
|
||||
"frequency_minutes": 10080,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"fao_landing_points": {
|
||||
"id": 18,
|
||||
"name": "FAO Landing Points",
|
||||
"display_name": "FAO 登陆点",
|
||||
"module": "L2",
|
||||
"priority": "P1",
|
||||
"frequency_minutes": 10080,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"spacetrack_tle": {
|
||||
"id": 19,
|
||||
"name": "Space-Track TLE",
|
||||
"display_name": "Space-Track 轨道根数",
|
||||
"module": "L3",
|
||||
"priority": "P2",
|
||||
"frequency_minutes": 1440,
|
||||
"is_free": True,
|
||||
"requires_credentials": True,
|
||||
"credential_provider": "spacetrack",
|
||||
"credential_status": "planned",
|
||||
},
|
||||
"celestrak_tle": {
|
||||
"id": 20,
|
||||
"name": "CelesTrak TLE",
|
||||
"display_name": "CelesTrak 轨道根数",
|
||||
"module": "L3",
|
||||
"priority": "P2",
|
||||
"frequency_minutes": 1440,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"ris_live_bgp": {
|
||||
"id": 21,
|
||||
"name": "RIPE RIS Live BGP",
|
||||
"display_name": "RIPE RIS 实时 BGP",
|
||||
"module": "L3",
|
||||
"priority": "P1",
|
||||
"frequency_minutes": 15,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"bgpstream_bgp": {
|
||||
"id": 22,
|
||||
"name": "CAIDA BGPStream Backfill",
|
||||
"display_name": "CAIDA BGPStream 回填",
|
||||
"module": "L3",
|
||||
"priority": "P1",
|
||||
"frequency_minutes": 360,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"iptoasn_prefix_geo": {
|
||||
"id": 23,
|
||||
"name": "IPtoASN Prefix Geography",
|
||||
"display_name": "IPtoASN 前缀地理",
|
||||
"module": "L3",
|
||||
"priority": "P1",
|
||||
"frequency_minutes": 1440,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"opengeofeed_prefix_geo": {
|
||||
"id": 24,
|
||||
"name": "OpenGeoFeed Prefix Geography",
|
||||
"display_name": "OpenGeoFeed 前缀地理",
|
||||
"module": "L3",
|
||||
"priority": "P1",
|
||||
"frequency_minutes": 1440,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"nro_delegated_prefix_geo": {
|
||||
"id": 25,
|
||||
"name": "NRO Delegated Prefix Geography",
|
||||
"display_name": "NRO 分配前缀地理",
|
||||
"module": "L3",
|
||||
"priority": "P1",
|
||||
"frequency_minutes": 1440,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"news_live_streams": {
|
||||
"id": 26,
|
||||
"name": "News Live Streams",
|
||||
"display_name": "新闻直播源",
|
||||
"module": "L4",
|
||||
"priority": "P2",
|
||||
"frequency_minutes": 720,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
"barentswatch_vessels": {
|
||||
"id": 27,
|
||||
"name": "BarentsWatch AIS Vessels",
|
||||
"display_name": "BarentsWatch AIS 船舶",
|
||||
"module": "L4",
|
||||
"priority": "P1",
|
||||
"frequency_minutes": 1,
|
||||
"is_free": True,
|
||||
"requires_credentials": True,
|
||||
"credential_provider": "barentswatch",
|
||||
"credential_status": "supported",
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@@ -1,161 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
|
||||
from collections.abc import Mapping, Sequence
|
||||
from typing import Any
|
||||
|
||||
from app.core.request_context import get_request_id
|
||||
|
||||
DEFAULT_SERVICE = "backend"
|
||||
DEFAULT_EVENT = "app.log"
|
||||
DEFAULT_LOG_LEVEL = os.getenv("PLANET_LOG_LEVEL", "INFO").upper()
|
||||
REDACTED = "[REDACTED]"
|
||||
SENSITIVE_FIELD_NAMES = {
|
||||
"access_token",
|
||||
"api_key",
|
||||
"authorization",
|
||||
"cookie",
|
||||
"password",
|
||||
"refresh_token",
|
||||
"secret",
|
||||
"token",
|
||||
}
|
||||
SENSITIVE_TEXT_PATTERNS = (
|
||||
re.compile(r"(?i)(authorization\s*[:=]\s*)(.+)"),
|
||||
re.compile(r"(?i)(bearer\s+)([A-Za-z0-9._\-]+)"),
|
||||
re.compile(r"(?i)(token\s*[:=]\s*)(.+)"),
|
||||
re.compile(r"(?i)(password\s*[:=]\s*)(.+)"),
|
||||
re.compile(r"(?i)(cookie\s*[:=]\s*)(.+)"),
|
||||
)
|
||||
|
||||
|
||||
def sanitize_log_value(value: Any) -> Any:
|
||||
if isinstance(value, Mapping):
|
||||
return {
|
||||
str(key): (REDACTED if str(key).lower() in SENSITIVE_FIELD_NAMES else sanitize_log_value(item))
|
||||
for key, item in value.items()
|
||||
}
|
||||
if isinstance(value, Sequence) and not isinstance(value, (str, bytes, bytearray)):
|
||||
return [sanitize_log_value(item) for item in value]
|
||||
if isinstance(value, str):
|
||||
sanitized = value
|
||||
for pattern in SENSITIVE_TEXT_PATTERNS:
|
||||
sanitized = pattern.sub(lambda match: f"{match.group(1)}{REDACTED}", sanitized)
|
||||
return sanitized
|
||||
return value
|
||||
|
||||
|
||||
def _normalize_context(context: Any) -> dict[str, Any]:
|
||||
if context is None:
|
||||
return {}
|
||||
if isinstance(context, Mapping):
|
||||
sanitized = sanitize_log_value(context)
|
||||
return {str(key): value for key, value in sanitized.items()}
|
||||
return {"value": sanitize_log_value(context)}
|
||||
|
||||
|
||||
class PlanetContextFilter(logging.Filter):
|
||||
def filter(self, record: logging.LogRecord) -> bool:
|
||||
record.request_id = getattr(record, "request_id", None) or get_request_id() or "-"
|
||||
record.service = getattr(record, "service", None) or DEFAULT_SERVICE
|
||||
record.event = getattr(record, "event", None) or DEFAULT_EVENT
|
||||
record.context = _normalize_context(getattr(record, "context", None))
|
||||
record.message = sanitize_log_value(record.getMessage())
|
||||
return True
|
||||
|
||||
|
||||
class PlanetFormatter(logging.Formatter):
|
||||
def format(self, record: logging.LogRecord) -> str:
|
||||
timestamp = self.formatTime(record, self.datefmt)
|
||||
level = record.levelname
|
||||
service = getattr(record, "service", DEFAULT_SERVICE)
|
||||
module_name = record.name
|
||||
event = getattr(record, "event", DEFAULT_EVENT)
|
||||
request_id = getattr(record, "request_id", "-")
|
||||
message = sanitize_log_value(record.getMessage())
|
||||
context = _normalize_context(getattr(record, "context", None))
|
||||
context_suffix = ""
|
||||
if context:
|
||||
context_suffix = f" context={json.dumps(context, ensure_ascii=False, sort_keys=True)}"
|
||||
rendered = (
|
||||
f"{timestamp} {level} service={service} module={module_name} "
|
||||
f"event={event} request_id={request_id} message={message}{context_suffix}"
|
||||
)
|
||||
if record.exc_info:
|
||||
rendered = f"{rendered}\n{self.formatException(record.exc_info)}"
|
||||
return rendered
|
||||
|
||||
|
||||
class PlanetLoggerAdapter(logging.LoggerAdapter):
|
||||
def process(self, msg: Any, kwargs: dict[str, Any]) -> tuple[Any, dict[str, Any]]:
|
||||
extra = dict(self.extra)
|
||||
extra.update(kwargs.get("extra", {}))
|
||||
if "context" in extra:
|
||||
extra["context"] = _normalize_context(extra.get("context"))
|
||||
kwargs["extra"] = extra
|
||||
return sanitize_log_value(msg), kwargs
|
||||
|
||||
def log_event(
|
||||
self,
|
||||
level: int,
|
||||
message: str,
|
||||
*,
|
||||
event: str,
|
||||
context: Mapping[str, Any] | None = None,
|
||||
**extra: Any,
|
||||
) -> None:
|
||||
self.log(level, message, extra={"event": event, "context": context or {}, **extra})
|
||||
|
||||
def debug_event(self, message: str, *, event: str, context: Mapping[str, Any] | None = None, **extra: Any) -> None:
|
||||
self.log_event(logging.DEBUG, message, event=event, context=context, **extra)
|
||||
|
||||
def info_event(self, message: str, *, event: str, context: Mapping[str, Any] | None = None, **extra: Any) -> None:
|
||||
self.log_event(logging.INFO, message, event=event, context=context, **extra)
|
||||
|
||||
def warning_event(self, message: str, *, event: str, context: Mapping[str, Any] | None = None, **extra: Any) -> None:
|
||||
self.log_event(logging.WARNING, message, event=event, context=context, **extra)
|
||||
|
||||
def error_event(self, message: str, *, event: str, context: Mapping[str, Any] | None = None, **extra: Any) -> None:
|
||||
self.log_event(logging.ERROR, message, event=event, context=context, **extra)
|
||||
|
||||
def exception_event(
|
||||
self,
|
||||
message: str,
|
||||
*,
|
||||
event: str,
|
||||
context: Mapping[str, Any] | None = None,
|
||||
**extra: Any,
|
||||
) -> None:
|
||||
self.error(message, exc_info=True, extra={"event": event, "context": context or {}, **extra})
|
||||
|
||||
|
||||
def get_logger(name: str, *, service: str = DEFAULT_SERVICE) -> PlanetLoggerAdapter:
|
||||
return PlanetLoggerAdapter(logging.getLogger(name), {"service": service})
|
||||
|
||||
|
||||
def configure_logging(level: str | None = None) -> None:
|
||||
root_logger = logging.getLogger()
|
||||
if getattr(configure_logging, "_configured", False):
|
||||
if level:
|
||||
root_logger.setLevel(level.upper())
|
||||
return
|
||||
|
||||
handler = logging.StreamHandler()
|
||||
handler.setFormatter(PlanetFormatter(datefmt="%Y-%m-%d %H:%M:%S"))
|
||||
handler.addFilter(PlanetContextFilter())
|
||||
|
||||
root_logger.handlers.clear()
|
||||
root_logger.addHandler(handler)
|
||||
root_logger.setLevel((level or DEFAULT_LOG_LEVEL).upper())
|
||||
|
||||
for logger_name in ("uvicorn", "uvicorn.error", "uvicorn.access"):
|
||||
target_logger = logging.getLogger(logger_name)
|
||||
target_logger.handlers.clear()
|
||||
target_logger.propagate = True
|
||||
|
||||
logging.captureWarnings(True)
|
||||
configure_logging._configured = True
|
||||
@@ -1,14 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from contextvars import ContextVar
|
||||
|
||||
|
||||
request_id_context: ContextVar[str | None] = ContextVar("request_id", default=None)
|
||||
|
||||
|
||||
def set_request_id(request_id: str | None) -> None:
|
||||
request_id_context.set(request_id)
|
||||
|
||||
|
||||
def get_request_id() -> str | None:
|
||||
return request_id_context.get()
|
||||
@@ -1,151 +0,0 @@
|
||||
"""Registry of target schemas supported by mapped custom data sources."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from typing import Any
|
||||
|
||||
from pydantic import BaseModel, Field, ValidationError, field_validator
|
||||
|
||||
|
||||
class VesselAISRecord(BaseModel):
|
||||
mmsi: int = Field(ge=100000000, le=999999999)
|
||||
lat: float = Field(ge=-90, le=90)
|
||||
lon: float = Field(ge=-180, le=180)
|
||||
sog: float | None = None
|
||||
cog: float | None = Field(default=None, ge=0, le=360)
|
||||
heading: int | None = Field(default=None, ge=0, le=511)
|
||||
name: str | None = None
|
||||
vessel_type: str | int | None = None
|
||||
received_at: datetime | None = None
|
||||
|
||||
|
||||
class GeoPointRecord(BaseModel):
|
||||
lat: float = Field(ge=-90, le=90)
|
||||
lon: float = Field(ge=-180, le=180)
|
||||
name: str | None = None
|
||||
type: str | None = None
|
||||
source_id: str | None = None
|
||||
observed_at: datetime | None = None
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class GenericRecord(BaseModel):
|
||||
data: dict[str, Any] = Field(default_factory=dict)
|
||||
source_id: str | None = None
|
||||
observed_at: datetime | None = None
|
||||
|
||||
@field_validator("data")
|
||||
@classmethod
|
||||
def require_payload(cls, value: dict[str, Any]) -> dict[str, Any]:
|
||||
if not value:
|
||||
raise ValueError("generic_records requires a non-empty data object")
|
||||
return value
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class TargetField:
|
||||
name: str
|
||||
type: str
|
||||
required: bool = False
|
||||
description: str = ""
|
||||
example: Any = None
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"name": self.name,
|
||||
"type": self.type,
|
||||
"required": self.required,
|
||||
"description": self.description,
|
||||
"example": self.example,
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class TargetSchema:
|
||||
key: str
|
||||
label: str
|
||||
description: str
|
||||
fields: tuple[TargetField, ...]
|
||||
model: type[BaseModel]
|
||||
destination: str
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"key": self.key,
|
||||
"label": self.label,
|
||||
"description": self.description,
|
||||
"destination": self.destination,
|
||||
"fields": [field.to_dict() for field in self.fields],
|
||||
}
|
||||
|
||||
def validate_record(self, record: dict[str, Any]) -> tuple[dict[str, Any] | None, list[str]]:
|
||||
try:
|
||||
return self.model.model_validate(record).model_dump(mode="json"), []
|
||||
except ValidationError as exc:
|
||||
return None, [
|
||||
".".join(str(part) for part in error["loc"]) + f": {error['msg']}"
|
||||
for error in exc.errors()
|
||||
]
|
||||
|
||||
|
||||
TARGET_SCHEMAS: dict[str, TargetSchema] = {
|
||||
"vessel_ais": TargetSchema(
|
||||
key="vessel_ais",
|
||||
label="船舶 AIS",
|
||||
description="船只位置、航速、航向、MMSI 等 AIS 数据。",
|
||||
destination="vessel_position",
|
||||
model=VesselAISRecord,
|
||||
fields=(
|
||||
TargetField("mmsi", "integer", True, "MMSI 九位船舶标识", 257123000),
|
||||
TargetField("lat", "float", True, "纬度", 59.91),
|
||||
TargetField("lon", "float", True, "经度", 10.75),
|
||||
TargetField("sog", "float", False, "对地航速,单位节", 12.4),
|
||||
TargetField("cog", "float", False, "对地航向,0-360 度", 184.5),
|
||||
TargetField("heading", "integer", False, "船首向,0-511", 186),
|
||||
TargetField("name", "string", False, "船名", "OSLO EXPRESS"),
|
||||
TargetField("vessel_type", "string", False, "船型", "cargo"),
|
||||
TargetField("received_at", "datetime", False, "数据接收时间", "2026-04-28T00:00:00Z"),
|
||||
),
|
||||
),
|
||||
"geo_points": TargetSchema(
|
||||
key="geo_points",
|
||||
label="通用地理点",
|
||||
description="带经纬度的通用实体或事件点位。",
|
||||
destination="generic_geo_points",
|
||||
model=GeoPointRecord,
|
||||
fields=(
|
||||
TargetField("lat", "float", True, "纬度", 1.3),
|
||||
TargetField("lon", "float", True, "经度", 103.8),
|
||||
TargetField("name", "string", False, "点位名称", "Singapore"),
|
||||
TargetField("type", "string", False, "点位类型", "datacenter"),
|
||||
TargetField("source_id", "string", False, "来源侧 ID", "sg-1"),
|
||||
TargetField("observed_at", "datetime", False, "观测时间", "2026-04-28T00:00:00Z"),
|
||||
TargetField("metadata", "object", False, "扩展字段", {"provider": "example"}),
|
||||
),
|
||||
),
|
||||
"generic_records": TargetSchema(
|
||||
key="generic_records",
|
||||
label="通用结构化记录",
|
||||
description="未知结构数据沉淀,不直接进入 Earth 图层。",
|
||||
destination="collected_data",
|
||||
model=GenericRecord,
|
||||
fields=(
|
||||
TargetField("data", "object", True, "结构化记录主体", {"raw": "value"}),
|
||||
TargetField("source_id", "string", False, "来源侧 ID", "record-1"),
|
||||
TargetField("observed_at", "datetime", False, "观测时间", "2026-04-28T00:00:00Z"),
|
||||
),
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def list_target_schemas() -> list[dict[str, Any]]:
|
||||
return [schema.to_dict() for schema in TARGET_SCHEMAS.values()]
|
||||
|
||||
|
||||
def get_target_schema(key: str) -> TargetSchema:
|
||||
try:
|
||||
return TARGET_SCHEMAS[key]
|
||||
except KeyError as exc:
|
||||
raise ValueError(f"Unsupported target schema: {key}") from exc
|
||||
@@ -5,22 +5,10 @@ from sqlalchemy.ext.asyncio import AsyncSession, create_async_engine, async_sess
|
||||
from sqlalchemy.orm import declarative_base
|
||||
|
||||
from app.core.config import settings
|
||||
from app.core.logging import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
DB_POOL_CONFIG = {
|
||||
"pool_pre_ping": True,
|
||||
"pool_recycle": 1800,
|
||||
"pool_size": 10,
|
||||
"max_overflow": 20,
|
||||
"pool_timeout": 30,
|
||||
}
|
||||
|
||||
engine = create_async_engine(
|
||||
settings.DATABASE_URL,
|
||||
echo=settings.DEBUG if hasattr(settings, "DEBUG") else False,
|
||||
**DB_POOL_CONFIG,
|
||||
)
|
||||
|
||||
async_session_factory = async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False)
|
||||
@@ -109,21 +97,6 @@ async def init_db():
|
||||
import app.models.system_setting # noqa: F401
|
||||
import app.models.playground_session # noqa: F401
|
||||
import app.models.playground_message # noqa: F401
|
||||
import app.models.system_log # noqa: F401
|
||||
import app.models.vessel # noqa: F401
|
||||
import app.models.datasource_mapping # noqa: F401
|
||||
|
||||
logger.warning_event(
|
||||
"Database pool settings active",
|
||||
event="database.pool.initialized",
|
||||
context={
|
||||
"pool_pre_ping": DB_POOL_CONFIG["pool_pre_ping"],
|
||||
"pool_recycle": DB_POOL_CONFIG["pool_recycle"],
|
||||
"pool_size": DB_POOL_CONFIG["pool_size"],
|
||||
"max_overflow": DB_POOL_CONFIG["max_overflow"],
|
||||
"pool_timeout": DB_POOL_CONFIG["pool_timeout"],
|
||||
},
|
||||
)
|
||||
|
||||
async with engine.begin() as conn:
|
||||
await conn.run_sync(Base.metadata.create_all)
|
||||
@@ -146,12 +119,7 @@ async def init_db():
|
||||
text(
|
||||
"""
|
||||
ALTER TABLE collection_tasks
|
||||
ADD COLUMN IF NOT EXISTS phase VARCHAR(30) DEFAULT 'queued',
|
||||
ADD COLUMN IF NOT EXISTS phase_progress DOUBLE PRECISION,
|
||||
ADD COLUMN IF NOT EXISTS phase_message VARCHAR(255),
|
||||
ADD COLUMN IF NOT EXISTS phase_current BIGINT,
|
||||
ADD COLUMN IF NOT EXISTS phase_total BIGINT,
|
||||
ADD COLUMN IF NOT EXISTS phase_unit VARCHAR(30)
|
||||
ADD COLUMN IF NOT EXISTS phase VARCHAR(30) DEFAULT 'queued'
|
||||
"""
|
||||
)
|
||||
)
|
||||
|
||||
@@ -1,5 +1,4 @@
|
||||
from contextlib import asynccontextmanager
|
||||
from uuid import uuid4
|
||||
|
||||
from fastapi import FastAPI
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
@@ -8,8 +7,6 @@ from starlette.middleware.base import BaseHTTPMiddleware
|
||||
from app.api.main import api_router
|
||||
from app.api.v1 import websocket
|
||||
from app.core.config import settings
|
||||
from app.core.logging import configure_logging
|
||||
from app.core.request_context import set_request_id
|
||||
from app.core.websocket.broadcaster import broadcaster
|
||||
from app.db.session import init_db
|
||||
from app.services.scheduler import (
|
||||
@@ -20,9 +17,6 @@ from app.services.scheduler import (
|
||||
)
|
||||
|
||||
|
||||
configure_logging()
|
||||
|
||||
|
||||
class WebSocketCORSMiddleware(BaseHTTPMiddleware):
|
||||
async def dispatch(self, request, call_next):
|
||||
if request.url.path.startswith("/ws") and request.method == "GET":
|
||||
@@ -34,18 +28,6 @@ class WebSocketCORSMiddleware(BaseHTTPMiddleware):
|
||||
return await call_next(request)
|
||||
|
||||
|
||||
class RequestContextMiddleware(BaseHTTPMiddleware):
|
||||
async def dispatch(self, request, call_next):
|
||||
request_id = request.headers.get("X-Request-ID") or uuid4().hex
|
||||
set_request_id(request_id)
|
||||
try:
|
||||
response = await call_next(request)
|
||||
finally:
|
||||
set_request_id(None)
|
||||
response.headers["X-Request-ID"] = request_id
|
||||
return response
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
await init_db()
|
||||
@@ -76,7 +58,6 @@ app.add_middleware(
|
||||
allow_headers=["*"],
|
||||
)
|
||||
|
||||
app.add_middleware(RequestContextMiddleware)
|
||||
app.add_middleware(WebSocketCORSMiddleware)
|
||||
|
||||
app.include_router(api_router, prefix="/api/v1")
|
||||
|
||||
@@ -11,9 +11,6 @@ from app.models.bgp_observation import BGPObservation
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.models.playground_session import PlaygroundSession
|
||||
from app.models.playground_message import PlaygroundMessage
|
||||
from app.models.system_log import SystemLog, AuditLog
|
||||
from app.models.vessel import VesselPosition, VesselStatic
|
||||
from app.models.datasource_mapping import DataSourceMappingTemplate
|
||||
|
||||
__all__ = [
|
||||
"User",
|
||||
@@ -29,9 +26,4 @@ __all__ = [
|
||||
"BGPAnomaly",
|
||||
"BGPIncident",
|
||||
"BGPObservation",
|
||||
"SystemLog",
|
||||
"AuditLog",
|
||||
"VesselPosition",
|
||||
"VesselStatic",
|
||||
"DataSourceMappingTemplate",
|
||||
]
|
||||
@@ -1,32 +0,0 @@
|
||||
"""Mapping templates for user-defined data source payloads."""
|
||||
|
||||
from sqlalchemy import Boolean, Column, DateTime, ForeignKey, Integer, JSON, String
|
||||
from sqlalchemy.sql import func
|
||||
|
||||
from app.db.session import Base
|
||||
|
||||
|
||||
class DataSourceMappingTemplate(Base):
|
||||
__tablename__ = "datasource_mapping_templates"
|
||||
|
||||
id = Column(Integer, primary_key=True, autoincrement=True)
|
||||
datasource_config_id = Column(
|
||||
Integer,
|
||||
ForeignKey("datasource_configs.id"),
|
||||
nullable=False,
|
||||
index=True,
|
||||
)
|
||||
target_schema = Column(String(80), nullable=False, index=True)
|
||||
mapping_json = Column(JSON, nullable=False, default={})
|
||||
sample_payload_hash = Column(String(64), nullable=True)
|
||||
validation_status = Column(String(30), nullable=False, default="draft")
|
||||
version = Column(Integer, nullable=False, default=1)
|
||||
is_active = Column(Boolean, nullable=False, default=False, index=True)
|
||||
created_at = Column(DateTime(timezone=True), server_default=func.now())
|
||||
updated_at = Column(DateTime(timezone=True), server_default=func.now(), onupdate=func.now())
|
||||
|
||||
def __repr__(self):
|
||||
return (
|
||||
f"<DataSourceMappingTemplate {self.id}: "
|
||||
f"{self.datasource_config_id}/{self.target_schema}/v{self.version}>"
|
||||
)
|
||||
@@ -1,40 +0,0 @@
|
||||
from sqlalchemy import JSON, Column, DateTime, Integer, String, Text
|
||||
from sqlalchemy.sql import func
|
||||
|
||||
from app.db.session import Base
|
||||
|
||||
|
||||
class SystemLog(Base):
|
||||
__tablename__ = "system_logs"
|
||||
|
||||
id = Column(Integer, primary_key=True, autoincrement=True)
|
||||
occurred_at = Column(DateTime(timezone=True), server_default=func.now(), index=True)
|
||||
source = Column(String(50), nullable=False, index=True)
|
||||
service = Column(String(50), nullable=True)
|
||||
module = Column(String(120), nullable=True)
|
||||
event = Column(String(160), nullable=True, index=True)
|
||||
level = Column(String(20), nullable=False, index=True)
|
||||
message = Column(Text, nullable=False)
|
||||
request_id = Column(String(64), nullable=True, index=True)
|
||||
trace_id = Column(String(64), nullable=True)
|
||||
user_id = Column(Integer, nullable=True, index=True)
|
||||
category = Column(String(80), nullable=True, index=True)
|
||||
context = Column(JSON, nullable=False, default=dict)
|
||||
created_at = Column(DateTime(timezone=True), server_default=func.now())
|
||||
|
||||
|
||||
class AuditLog(Base):
|
||||
__tablename__ = "audit_logs"
|
||||
|
||||
id = Column(Integer, primary_key=True, autoincrement=True)
|
||||
occurred_at = Column(DateTime(timezone=True), server_default=func.now(), index=True)
|
||||
actor_id = Column(Integer, nullable=True, index=True)
|
||||
actor_name = Column(String(255), nullable=True)
|
||||
action = Column(String(120), nullable=False, index=True)
|
||||
target_type = Column(String(80), nullable=True)
|
||||
target_id = Column(String(120), nullable=True)
|
||||
result = Column(String(40), nullable=True, index=True)
|
||||
request_id = Column(String(64), nullable=True, index=True)
|
||||
ip = Column(String(64), nullable=True)
|
||||
details = Column(JSON, nullable=False, default=dict)
|
||||
created_at = Column(DateTime(timezone=True), server_default=func.now())
|
||||
@@ -1,6 +1,6 @@
|
||||
"""Collection Task model"""
|
||||
|
||||
from sqlalchemy import BigInteger, Column, DateTime, Integer, String, Text, Float
|
||||
from sqlalchemy import Column, DateTime, Integer, String, Text, Float
|
||||
from sqlalchemy.sql import func
|
||||
|
||||
from app.db.session import Base
|
||||
@@ -13,11 +13,6 @@ class CollectionTask(Base):
|
||||
datasource_id = Column(Integer, nullable=False, index=True)
|
||||
status = Column(String(20), nullable=False) # pending, running, success, failed, cancelled
|
||||
phase = Column(String(30), default="queued")
|
||||
phase_progress = Column(Float)
|
||||
phase_message = Column(String(255))
|
||||
phase_current = Column(BigInteger)
|
||||
phase_total = Column(BigInteger)
|
||||
phase_unit = Column(String(30))
|
||||
started_at = Column(DateTime(timezone=True))
|
||||
completed_at = Column(DateTime(timezone=True))
|
||||
records_processed = Column(Integer, default=0)
|
||||
|
||||
@@ -1,75 +0,0 @@
|
||||
"""Vessel AIS models for live maritime tracking."""
|
||||
|
||||
from sqlalchemy import BigInteger, Column, DateTime, Float, Index, Integer, SmallInteger, String
|
||||
from sqlalchemy.sql import func
|
||||
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.db.session import Base
|
||||
|
||||
|
||||
class VesselStatic(Base):
|
||||
"""Slow-changing vessel identity and dimensions."""
|
||||
|
||||
__tablename__ = "vessel_static"
|
||||
|
||||
mmsi = Column(BigInteger, primary_key=True)
|
||||
name = Column(String(128), nullable=True)
|
||||
callsign = Column(String(16), nullable=True)
|
||||
vessel_type = Column(SmallInteger, nullable=True, index=True)
|
||||
vessel_type_name = Column(String(64), nullable=True, index=True)
|
||||
flag = Column(String(4), nullable=True, index=True)
|
||||
length = Column(Float, nullable=True)
|
||||
width = Column(Float, nullable=True)
|
||||
draught = Column(Float, nullable=True)
|
||||
imo = Column(BigInteger, nullable=True)
|
||||
updated_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now())
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {
|
||||
"mmsi": self.mmsi,
|
||||
"name": self.name,
|
||||
"callsign": self.callsign,
|
||||
"vessel_type": self.vessel_type,
|
||||
"vessel_type_name": self.vessel_type_name,
|
||||
"flag": self.flag,
|
||||
"length": self.length,
|
||||
"width": self.width,
|
||||
"draught": self.draught,
|
||||
"imo": self.imo,
|
||||
"updated_at": to_iso8601_utc(self.updated_at),
|
||||
}
|
||||
|
||||
|
||||
class VesselPosition(Base):
|
||||
"""Append-only AIS positions retained for short history windows."""
|
||||
|
||||
__tablename__ = "vessel_position"
|
||||
|
||||
id = Column(Integer, primary_key=True, autoincrement=True)
|
||||
mmsi = Column(BigInteger, nullable=False, index=True)
|
||||
lat = Column(Float, nullable=False)
|
||||
lon = Column(Float, nullable=False)
|
||||
sog = Column(Float, nullable=True)
|
||||
cog = Column(Float, nullable=True)
|
||||
heading = Column(SmallInteger, nullable=True)
|
||||
nav_status = Column(SmallInteger, nullable=True, index=True)
|
||||
received_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now(), index=True)
|
||||
|
||||
__table_args__ = (
|
||||
Index("idx_vessel_pos_mmsi_time", "mmsi", "received_at"),
|
||||
Index("idx_vessel_pos_time", "received_at"),
|
||||
Index("idx_vessel_pos_lat_lon", "lat", "lon"),
|
||||
)
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {
|
||||
"id": self.id,
|
||||
"mmsi": self.mmsi,
|
||||
"lat": self.lat,
|
||||
"lon": self.lon,
|
||||
"sog": self.sog,
|
||||
"cog": self.cog,
|
||||
"heading": self.heading,
|
||||
"nav_status": self.nav_status,
|
||||
"received_at": to_iso8601_utc(self.received_at),
|
||||
}
|
||||
@@ -3,11 +3,9 @@ from __future__ import annotations
|
||||
import asyncio
|
||||
|
||||
import httpx
|
||||
from fastapi import Depends, HTTPException, status
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from fastapi import HTTPException, status
|
||||
|
||||
from app.core.config import settings
|
||||
from app.db.session import get_db
|
||||
from app.schemas.ai import (
|
||||
AIProviderStatusResponse,
|
||||
SituationalAnalysisRequest,
|
||||
@@ -16,27 +14,11 @@ from app.schemas.ai import (
|
||||
|
||||
|
||||
class AIProviderClient:
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
service_url: str | None = None,
|
||||
service_token: str | None = None,
|
||||
timeout: int | None = None,
|
||||
retry_attempts: int | None = None,
|
||||
llm_config: dict | None = None,
|
||||
) -> None:
|
||||
self.service_url = (
|
||||
service_url if service_url is not None else settings.AI_PROVIDER_SERVICE_URL
|
||||
).rstrip("/")
|
||||
self.service_token = (
|
||||
service_token if service_token is not None else settings.AI_PROVIDER_SERVICE_TOKEN
|
||||
)
|
||||
self.timeout = timeout if timeout is not None else settings.AI_PROVIDER_TIMEOUT_SECONDS
|
||||
self.retry_attempts = max(
|
||||
retry_attempts if retry_attempts is not None else settings.AI_PROVIDER_RETRY_ATTEMPTS,
|
||||
1,
|
||||
)
|
||||
self.llm_config = llm_config or {}
|
||||
def __init__(self) -> None:
|
||||
self.service_url = settings.AI_PROVIDER_SERVICE_URL.rstrip("/")
|
||||
self.service_token = settings.AI_PROVIDER_SERVICE_TOKEN
|
||||
self.timeout = settings.AI_PROVIDER_TIMEOUT_SECONDS
|
||||
self.retry_attempts = max(settings.AI_PROVIDER_RETRY_ATTEMPTS, 1)
|
||||
|
||||
def _headers(self, request_id: str | None = None) -> dict[str, str]:
|
||||
headers = {"Content-Type": "application/json"}
|
||||
@@ -44,19 +26,6 @@ class AIProviderClient:
|
||||
headers["X-Provider-Token"] = self.service_token
|
||||
if request_id:
|
||||
headers["X-Request-ID"] = request_id
|
||||
llm_header_map = {
|
||||
"provider": "X-AI-Provider",
|
||||
"provider_api": "X-AI-Provider-API",
|
||||
"base_url": "X-AI-Base-URL",
|
||||
"api_key": "X-AI-API-Key",
|
||||
"model": "X-AI-Model",
|
||||
"max_tokens": "X-AI-Max-Tokens",
|
||||
"anthropic_version": "X-AI-Anthropic-Version",
|
||||
}
|
||||
for key, header_name in llm_header_map.items():
|
||||
value = self.llm_config.get(key)
|
||||
if value not in (None, ""):
|
||||
headers[header_name] = str(value)
|
||||
return headers
|
||||
|
||||
async def get_status(self, request_id: str | None = None) -> AIProviderStatusResponse:
|
||||
@@ -136,14 +105,5 @@ class AIProviderClient:
|
||||
)
|
||||
|
||||
|
||||
async def get_ai_provider_client(db: AsyncSession = Depends(get_db)) -> AIProviderClient:
|
||||
from app.api.v1.settings import get_runtime_ai_provider_config
|
||||
|
||||
runtime_config = await get_runtime_ai_provider_config(db)
|
||||
return AIProviderClient(
|
||||
service_url=runtime_config["service_url"],
|
||||
service_token=runtime_config["service_token"],
|
||||
timeout=runtime_config["timeout_seconds"],
|
||||
retry_attempts=runtime_config["retry_attempts"],
|
||||
llm_config=runtime_config.get("llm_config") or {},
|
||||
)
|
||||
def get_ai_provider_client() -> AIProviderClient:
|
||||
return AIProviderClient()
|
||||
@@ -1,209 +0,0 @@
|
||||
"""BarentsWatch AIS credential resolution and connectivity checks."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import shlex
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
from app.models.datasource_config import DataSourceConfig
|
||||
|
||||
|
||||
BARENTSWATCH_LATEST_URL = "https://live.ais.barentswatch.no/v1/latest/combined"
|
||||
BARENTSWATCH_TOKEN_URL = "https://id.barentswatch.no/connect/token"
|
||||
BARENTSWATCH_DATASOURCE_NAME = "barentswatch_vessels"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class BarentsWatchConfig:
|
||||
endpoint: str
|
||||
client_id: str
|
||||
client_secret: str
|
||||
credential_source: str
|
||||
endpoint_source: str
|
||||
|
||||
|
||||
def _read_zshrc_env(path: Path | None = None) -> dict[str, str]:
|
||||
zshrc_path = path or Path.home() / ".zshrc"
|
||||
if not zshrc_path.exists():
|
||||
return {}
|
||||
|
||||
values: dict[str, str] = {}
|
||||
for raw_line in zshrc_path.read_text(encoding="utf-8", errors="ignore").splitlines():
|
||||
line = raw_line.strip()
|
||||
if not line or line.startswith("#"):
|
||||
continue
|
||||
if line.startswith("export "):
|
||||
line = line[len("export ") :].strip()
|
||||
if "=" not in line:
|
||||
continue
|
||||
|
||||
key, value = line.split("=", 1)
|
||||
key = key.strip()
|
||||
if not key or not key.replace("_", "").isalnum() or not key[0].isalpha():
|
||||
continue
|
||||
|
||||
try:
|
||||
parsed = shlex.split(value, comments=True, posix=True)
|
||||
except ValueError:
|
||||
parsed = [value.strip().strip("'\"")]
|
||||
if parsed:
|
||||
values[key] = parsed[0]
|
||||
return values
|
||||
|
||||
|
||||
def _first_env_value(zshrc_env: dict[str, str], *keys: str) -> tuple[str, str]:
|
||||
for key in keys:
|
||||
value = os.getenv(key)
|
||||
if value:
|
||||
return value, "environment"
|
||||
for key in keys:
|
||||
value = zshrc_env.get(key)
|
||||
if value:
|
||||
return value, "~/.zshrc"
|
||||
return "", ""
|
||||
|
||||
|
||||
async def get_barentswatch_datasource_record(db: AsyncSession) -> DataSourceConfig | None:
|
||||
result = await db.execute(
|
||||
select(DataSourceConfig)
|
||||
.where(DataSourceConfig.name == BARENTSWATCH_DATASOURCE_NAME)
|
||||
.where(DataSourceConfig.is_active.is_(True))
|
||||
)
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
|
||||
async def resolve_barentswatch_config(db: AsyncSession | None = None) -> BarentsWatchConfig:
|
||||
record = await get_barentswatch_datasource_record(db) if db else None
|
||||
auth_config = dict(record.auth_config or {}) if record else {}
|
||||
config = dict(record.config or {}) if record else {}
|
||||
zshrc_env = _read_zshrc_env()
|
||||
|
||||
env_client_id, env_source = _first_env_value(
|
||||
zshrc_env,
|
||||
"BARENTSWATCH_CLIENT_ID",
|
||||
"BARRENTSWATCH_CLIENT_ID",
|
||||
)
|
||||
env_client_secret, secret_env_source = _first_env_value(
|
||||
zshrc_env,
|
||||
"BARENTSWATCH_CLIENT_SECRET",
|
||||
"BARRENTSWATCH_CLIENT_SECRET",
|
||||
)
|
||||
client_id = auth_config.get("client_id") or config.get("client_id") or env_client_id
|
||||
client_secret = (
|
||||
auth_config.get("client_secret") or config.get("client_secret") or env_client_secret
|
||||
)
|
||||
|
||||
credential_source = ""
|
||||
if auth_config.get("client_id") or auth_config.get("client_secret"):
|
||||
credential_source = "datasource_config"
|
||||
elif config.get("client_id") or config.get("client_secret"):
|
||||
credential_source = "datasource_runtime_config"
|
||||
elif env_source or secret_env_source:
|
||||
credential_source = env_source or secret_env_source
|
||||
|
||||
yaml_endpoint = get_data_sources_config().get_yaml_url(BARENTSWATCH_DATASOURCE_NAME)
|
||||
endpoint = record.endpoint if record and record.endpoint else yaml_endpoint
|
||||
return BarentsWatchConfig(
|
||||
endpoint=endpoint or BARENTSWATCH_LATEST_URL,
|
||||
client_id=str(client_id or ""),
|
||||
client_secret=str(client_secret or ""),
|
||||
credential_source=credential_source or "missing",
|
||||
endpoint_source="datasource_config" if record and record.endpoint else "default",
|
||||
)
|
||||
|
||||
|
||||
async def fetch_barentswatch_access_token(
|
||||
client: httpx.AsyncClient,
|
||||
config: BarentsWatchConfig,
|
||||
) -> str | None:
|
||||
if not config.client_id or not config.client_secret:
|
||||
return None
|
||||
|
||||
response = await client.post(
|
||||
BARENTSWATCH_TOKEN_URL,
|
||||
data={
|
||||
"client_id": config.client_id,
|
||||
"client_secret": config.client_secret,
|
||||
"scope": "ais",
|
||||
"grant_type": "client_credentials",
|
||||
},
|
||||
headers={"Content-Type": "application/x-www-form-urlencoded"},
|
||||
)
|
||||
response.raise_for_status()
|
||||
payload = response.json()
|
||||
token = payload.get("access_token")
|
||||
return str(token) if token else None
|
||||
|
||||
|
||||
async def check_barentswatch_connectivity(db: AsyncSession) -> dict[str, Any]:
|
||||
config = await resolve_barentswatch_config(db)
|
||||
return await check_barentswatch_config(config)
|
||||
|
||||
|
||||
async def check_barentswatch_config(config: BarentsWatchConfig) -> dict[str, Any]:
|
||||
if not config.client_id or not config.client_secret:
|
||||
return {
|
||||
"success": False,
|
||||
"stage": "credentials",
|
||||
"message": "未找到 BarentsWatch client id/client secret,请先配置采集器凭证。",
|
||||
"endpoint": config.endpoint,
|
||||
"credential_source": config.credential_source,
|
||||
"settings_tab": "collector_credentials",
|
||||
}
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=20.0) as client:
|
||||
token = await fetch_barentswatch_access_token(client, config)
|
||||
if not token:
|
||||
return {
|
||||
"success": False,
|
||||
"stage": "token",
|
||||
"message": "BarentsWatch token 响应中没有 access_token,请检查凭证。",
|
||||
"endpoint": config.endpoint,
|
||||
"credential_source": config.credential_source,
|
||||
"settings_tab": "collector_credentials",
|
||||
}
|
||||
|
||||
async with client.stream(
|
||||
"GET",
|
||||
config.endpoint,
|
||||
headers={"Authorization": f"Bearer {token}"},
|
||||
) as response:
|
||||
response.raise_for_status()
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"stage": "endpoint",
|
||||
"message": "BarentsWatch AIS token 和数据接口均可连通。",
|
||||
"endpoint": config.endpoint,
|
||||
"credential_source": config.credential_source,
|
||||
"endpoint_source": config.endpoint_source,
|
||||
}
|
||||
except httpx.HTTPStatusError as exc:
|
||||
status_code = exc.response.status_code
|
||||
stage = "token" if str(exc.request.url) == BARENTSWATCH_TOKEN_URL else "endpoint"
|
||||
return {
|
||||
"success": False,
|
||||
"stage": stage,
|
||||
"message": f"BarentsWatch {stage} 请求返回 HTTP {status_code},请检查凭证或接口地址。",
|
||||
"endpoint": config.endpoint,
|
||||
"credential_source": config.credential_source,
|
||||
"settings_tab": "collector_credentials",
|
||||
}
|
||||
except httpx.HTTPError as exc:
|
||||
return {
|
||||
"success": False,
|
||||
"stage": "network",
|
||||
"message": f"BarentsWatch 链路检查失败:{exc.__class__.__name__}",
|
||||
"endpoint": config.endpoint,
|
||||
"credential_source": config.credential_source,
|
||||
"settings_tab": "collector_credentials",
|
||||
}
|
||||
@@ -36,7 +36,6 @@ from app.services.collectors.iptoasn import IPtoASNPrefixGeoCollector
|
||||
from app.services.collectors.opengeofeed import OpenGeoFeedPrefixGeoCollector
|
||||
from app.services.collectors.nro_delegated import NRODelegatedPrefixGeoCollector
|
||||
from app.services.collectors.news_live_streams import NewsLiveStreamsCollector
|
||||
from app.services.collectors.vessel_ais import VesselAISCollector
|
||||
|
||||
collector_registry.register(TOP500Collector())
|
||||
collector_registry.register(EpochAIGPUCollector())
|
||||
@@ -64,4 +63,3 @@ collector_registry.register(IPtoASNPrefixGeoCollector())
|
||||
collector_registry.register(OpenGeoFeedPrefixGeoCollector())
|
||||
collector_registry.register(NRODelegatedPrefixGeoCollector())
|
||||
collector_registry.register(NewsLiveStreamsCollector())
|
||||
collector_registry.register(VesselAISCollector())
|
||||
@@ -54,11 +54,6 @@ class BaseCollector(ABC):
|
||||
"task_id": self._current_task.id,
|
||||
"status": self._current_task.status,
|
||||
"phase": self._current_task.phase,
|
||||
"phase_progress": self._current_task.phase_progress,
|
||||
"phase_message": self._current_task.phase_message,
|
||||
"phase_current": self._current_task.phase_current,
|
||||
"phase_total": self._current_task.phase_total,
|
||||
"phase_unit": self._current_task.phase_unit,
|
||||
"progress": progress,
|
||||
"records_processed": self._current_task.records_processed,
|
||||
"total_records": self._current_task.total_records,
|
||||
@@ -85,52 +80,12 @@ class BaseCollector(ABC):
|
||||
|
||||
await self._publish_task_update(force=force)
|
||||
|
||||
async def set_phase(self, phase: str, *, message: str | None = None, reset_progress: bool = True):
|
||||
async def set_phase(self, phase: str):
|
||||
if self._current_task and self._db_session:
|
||||
self._current_task.phase = phase
|
||||
self._current_task.phase_message = message
|
||||
if reset_progress:
|
||||
self._current_task.phase_progress = None
|
||||
self._current_task.phase_current = None
|
||||
self._current_task.phase_total = None
|
||||
self._current_task.phase_unit = None
|
||||
await self._db_session.commit()
|
||||
await self._publish_task_update(force=True)
|
||||
|
||||
async def update_phase_progress(
|
||||
self,
|
||||
*,
|
||||
current: int | None = None,
|
||||
total: int | None = None,
|
||||
unit: str | None = None,
|
||||
message: str | None = None,
|
||||
progress: float | None = None,
|
||||
commit: bool = False,
|
||||
force: bool = False,
|
||||
):
|
||||
"""Update progress for the current phase without changing task totals."""
|
||||
if not self._current_task or not self._db_session:
|
||||
return
|
||||
|
||||
if progress is None and current is not None and total and total > 0:
|
||||
progress = (current / total) * 100
|
||||
|
||||
if progress is not None:
|
||||
self._current_task.phase_progress = max(0.0, min(float(progress), 100.0))
|
||||
if current is not None:
|
||||
self._current_task.phase_current = max(0, int(current))
|
||||
if total is not None:
|
||||
self._current_task.phase_total = max(0, int(total))
|
||||
if unit is not None:
|
||||
self._current_task.phase_unit = unit
|
||||
if message is not None:
|
||||
self._current_task.phase_message = message
|
||||
|
||||
if commit:
|
||||
await self._db_session.commit()
|
||||
|
||||
await self._publish_task_update(force=force)
|
||||
|
||||
@abstractmethod
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
"""Fetch raw data from source"""
|
||||
@@ -296,7 +251,7 @@ class BaseCollector(ABC):
|
||||
await self._publish_task_update(force=True)
|
||||
|
||||
try:
|
||||
await self.set_phase("fetching", message="正在拉取原始数据")
|
||||
await self.set_phase("fetching")
|
||||
raw_data = await self.fetch()
|
||||
task.total_records = len(raw_data)
|
||||
await db.commit()
|
||||
@@ -305,20 +260,15 @@ class BaseCollector(ABC):
|
||||
if self.fail_on_empty and not raw_data:
|
||||
raise RuntimeError(f"Collector {self.name} returned no data")
|
||||
|
||||
await self.set_phase("transforming", message="正在转换采集数据")
|
||||
await self.set_phase("transforming")
|
||||
data = self.transform(raw_data)
|
||||
snapshot_id = await self._create_snapshot(db, task_id, data, start_time)
|
||||
|
||||
await self.set_phase("saving", message="正在保存采集数据")
|
||||
await self.set_phase("saving")
|
||||
records_count = await self._save_data(db, data, task_id=task_id, snapshot_id=snapshot_id)
|
||||
|
||||
task.status = "success"
|
||||
task.phase = "completed"
|
||||
task.phase_progress = 100.0
|
||||
task.phase_message = "采集完成"
|
||||
task.phase_current = records_count
|
||||
task.phase_total = records_count
|
||||
task.phase_unit = "records"
|
||||
task.records_processed = records_count
|
||||
task.progress = 100.0
|
||||
task.completed_at = datetime.now(UTC)
|
||||
@@ -335,7 +285,6 @@ class BaseCollector(ABC):
|
||||
await db.rollback()
|
||||
task.status = "cancelled"
|
||||
task.phase = "cancelled"
|
||||
task.phase_message = "采集已取消"
|
||||
task.error_message = "Collection cancelled by operator and rolled back"
|
||||
task.completed_at = datetime.now(UTC)
|
||||
if snapshot_id is not None:
|
||||
@@ -352,7 +301,6 @@ class BaseCollector(ABC):
|
||||
await db.rollback()
|
||||
task.status = "failed"
|
||||
task.phase = "failed"
|
||||
task.phase_message = str(e)
|
||||
task.error_message = str(e)
|
||||
task.completed_at = datetime.now(UTC)
|
||||
if snapshot_id is not None:
|
||||
|
||||
@@ -46,9 +46,6 @@ class CelesTrakTLECollector(BaseCollector):
|
||||
if response.status_code == 200:
|
||||
data = response.json()
|
||||
if isinstance(data, list):
|
||||
for item in data:
|
||||
if isinstance(item, dict):
|
||||
item["_celestrak_group"] = group
|
||||
all_satellites.extend(data)
|
||||
print(f"CelesTrak: Fetched {len(data)} satellites from group '{group}'")
|
||||
except Exception as e:
|
||||
@@ -81,7 +78,6 @@ class CelesTrakTLECollector(BaseCollector):
|
||||
"name": item.get("OBJECT_NAME", "Unknown"),
|
||||
"reference_date": item.get("EPOCH", ""),
|
||||
"metadata": {
|
||||
"constellation_group": item.get("_celestrak_group"),
|
||||
"norad_cat_id": item.get("NORAD_CAT_ID"),
|
||||
"international_designator": item.get("OBJECT_ID"),
|
||||
"epoch": item.get("EPOCH"),
|
||||
|
||||
@@ -108,11 +108,6 @@ class IPtoASNPrefixGeoCollector(BaseCollector):
|
||||
self._current_task.total_records = total_expected
|
||||
self._current_task.records_processed = 0
|
||||
self._current_task.progress = 0.0
|
||||
self._current_task.phase_progress = 0.0
|
||||
self._current_task.phase_message = "正在下载 IPtoASN 数据"
|
||||
self._current_task.phase_current = 0
|
||||
self._current_task.phase_total = total_expected
|
||||
self._current_task.phase_unit = "bytes"
|
||||
await self._db_session.commit()
|
||||
await self._publish_task_update(force=True)
|
||||
|
||||
@@ -140,14 +135,7 @@ class IPtoASNPrefixGeoCollector(BaseCollector):
|
||||
return
|
||||
last_emit["value"] = aggregated
|
||||
last_emit["t"] = now
|
||||
current = min(aggregated, total_expected)
|
||||
await self.update_phase_progress(
|
||||
current=current,
|
||||
total=total_expected,
|
||||
unit="bytes",
|
||||
message="正在下载 IPtoASN 数据",
|
||||
)
|
||||
await self.update_progress(current, commit=True)
|
||||
await self.update_progress(min(aggregated, total_expected), commit=True)
|
||||
|
||||
batches = await asyncio.gather(
|
||||
*(
|
||||
@@ -160,12 +148,6 @@ class IPtoASNPrefixGeoCollector(BaseCollector):
|
||||
)
|
||||
)
|
||||
if total_expected > 0:
|
||||
await self.update_phase_progress(
|
||||
current=total_expected,
|
||||
total=total_expected,
|
||||
unit="bytes",
|
||||
message="IPtoASN 数据下载完成",
|
||||
)
|
||||
await self.update_progress(total_expected, commit=True, force=True)
|
||||
|
||||
rows: list[dict[str, Any]] = []
|
||||
|
||||
@@ -1,16 +1,10 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
from urllib.parse import urlparse
|
||||
|
||||
import httpx
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
from app.models.datasource_config import DataSourceConfig
|
||||
from app.services.collectors.base import BaseCollector
|
||||
|
||||
|
||||
@@ -24,537 +18,52 @@ class NewsLiveStreamsCollector(BaseCollector):
|
||||
data_type = "news_live_stream"
|
||||
fail_on_empty = False
|
||||
|
||||
DEFAULT_TIMEOUT = 45.0
|
||||
DEFAULT_HEADERS = {
|
||||
"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
|
||||
"Accept": "application/json",
|
||||
}
|
||||
RESPONSE_CANDIDATE_KEYS = ("sources", "streams", "channels", "items", "results", "data")
|
||||
DEFAULT_ADAPTER = "iptv_org"
|
||||
DEFAULT_IPTV_ORG_STREAMS_URL = "https://iptv-org.github.io/api/streams.json"
|
||||
DEFAULT_IPTV_ORG_LOGOS_URL = "https://iptv-org.github.io/api/logos.json"
|
||||
DEFAULT_IPTV_ORG_NEWS_CATEGORIES = ("news", "business", "weather")
|
||||
DEFAULT_IPTV_ORG_EXCLUDE_CATEGORIES = ("music", "sports", "kids", "entertainment")
|
||||
DEFAULT_IPTV_ORG_MAX_SOURCES = 120
|
||||
|
||||
async def fetch(self) -> list[dict[str, Any]]:
|
||||
request_url = (self._resolved_url or "").strip()
|
||||
if not request_url:
|
||||
return []
|
||||
|
||||
datasource_config = await self._load_datasource_config()
|
||||
effective_config = self._get_effective_config(datasource_config)
|
||||
adapter = str(effective_config.get("adapter") or "").strip().lower()
|
||||
if adapter == "iptv_org":
|
||||
return await self._fetch_iptv_org(request_url, effective_config)
|
||||
|
||||
request_headers = self._build_request_headers(datasource_config)
|
||||
request_config = self._get_request_config(datasource_config)
|
||||
request_params = self._build_request_params(datasource_config)
|
||||
request_json = self._build_request_json_body(datasource_config)
|
||||
request_data = self._build_request_form_body(datasource_config)
|
||||
timeout = self._get_timeout(datasource_config)
|
||||
|
||||
async with httpx.AsyncClient(timeout=timeout, follow_redirects=True) as client:
|
||||
response = await client.request(
|
||||
request_config["method"],
|
||||
async with httpx.AsyncClient(timeout=45.0, follow_redirects=True) as client:
|
||||
response = await client.get(
|
||||
request_url,
|
||||
headers=request_headers,
|
||||
params=request_params or None,
|
||||
json=request_json,
|
||||
data=request_data,
|
||||
headers={
|
||||
"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
|
||||
"Accept": "application/json",
|
||||
},
|
||||
)
|
||||
response.raise_for_status()
|
||||
return self.parse_response(
|
||||
response.json(),
|
||||
response_path=request_config["response_path"],
|
||||
)
|
||||
|
||||
async def _load_datasource_config(self) -> DataSourceConfig | None:
|
||||
if not self._db_session:
|
||||
return None
|
||||
|
||||
result = await self._db_session.execute(
|
||||
select(DataSourceConfig)
|
||||
.where(DataSourceConfig.name == self.name)
|
||||
.where(DataSourceConfig.is_active.is_(True))
|
||||
.limit(1)
|
||||
)
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
def _get_effective_config(self, datasource_config: DataSourceConfig | None) -> dict[str, Any]:
|
||||
payload = dict(datasource_config.config or {}) if datasource_config else {}
|
||||
if payload:
|
||||
return payload
|
||||
|
||||
yaml_config = get_data_sources_config()
|
||||
return {
|
||||
"adapter": self.DEFAULT_ADAPTER,
|
||||
"streams_url": yaml_config.get_yaml_value("news_live_streams.streams_url")
|
||||
or self.DEFAULT_IPTV_ORG_STREAMS_URL,
|
||||
"logos_url": yaml_config.get_yaml_value("news_live_streams.logos_url")
|
||||
or self.DEFAULT_IPTV_ORG_LOGOS_URL,
|
||||
"news_categories": list(self.DEFAULT_IPTV_ORG_NEWS_CATEGORIES),
|
||||
"exclude_categories": list(self.DEFAULT_IPTV_ORG_EXCLUDE_CATEGORIES),
|
||||
"max_sources": self.DEFAULT_IPTV_ORG_MAX_SOURCES,
|
||||
}
|
||||
|
||||
def _get_request_config(self, datasource_config: DataSourceConfig | None) -> dict[str, Any]:
|
||||
payload = self._get_effective_config(datasource_config)
|
||||
raw_method = payload.get("method") or payload.get("request_method") or "GET"
|
||||
method = str(raw_method).strip().upper() or "GET"
|
||||
if method not in {"GET", "POST"}:
|
||||
method = "GET"
|
||||
|
||||
response_path = payload.get("response_path") or payload.get("payload_path") or payload.get("items_path")
|
||||
if isinstance(response_path, str):
|
||||
response_path = response_path.strip()
|
||||
else:
|
||||
response_path = None
|
||||
|
||||
return {
|
||||
"method": method,
|
||||
"response_path": response_path or None,
|
||||
}
|
||||
|
||||
def _get_timeout(self, datasource_config: DataSourceConfig | None) -> float:
|
||||
payload = self._get_effective_config(datasource_config)
|
||||
try:
|
||||
return float(payload.get("timeout", self.DEFAULT_TIMEOUT))
|
||||
except (TypeError, ValueError):
|
||||
return self.DEFAULT_TIMEOUT
|
||||
|
||||
def _build_request_headers(self, datasource_config: DataSourceConfig | None) -> dict[str, str]:
|
||||
headers = dict(self.DEFAULT_HEADERS)
|
||||
if datasource_config:
|
||||
headers.update(self._normalize_headers(datasource_config.headers))
|
||||
headers.update(self._build_auth_headers(datasource_config))
|
||||
return headers
|
||||
|
||||
def _build_request_params(self, datasource_config: DataSourceConfig | None) -> dict[str, Any]:
|
||||
params: dict[str, Any] = {}
|
||||
if not datasource_config:
|
||||
return params
|
||||
|
||||
payload = datasource_config.config or {}
|
||||
candidate = payload.get("params") or payload.get("query_params")
|
||||
if isinstance(candidate, dict):
|
||||
params.update(candidate)
|
||||
|
||||
if datasource_config.auth_type == "api_key":
|
||||
auth_config = datasource_config.auth_config or {}
|
||||
if str(auth_config.get("in") or auth_config.get("location") or "header").lower() == "query":
|
||||
api_key = auth_config.get("api_key")
|
||||
key_name = auth_config.get("key_name") or auth_config.get("param_name") or "api_key"
|
||||
if api_key and key_name:
|
||||
params[str(key_name)] = api_key
|
||||
|
||||
return params
|
||||
|
||||
def _build_request_json_body(self, datasource_config: DataSourceConfig | None) -> Any:
|
||||
if not datasource_config:
|
||||
return None
|
||||
|
||||
payload = datasource_config.config or {}
|
||||
body = payload.get("json_body")
|
||||
if body is None and str(payload.get("body_type") or "").lower() in {"json", ""}:
|
||||
candidate = payload.get("body")
|
||||
if isinstance(candidate, (dict, list)):
|
||||
body = candidate
|
||||
return body
|
||||
|
||||
def _build_request_form_body(self, datasource_config: DataSourceConfig | None) -> Any:
|
||||
if not datasource_config:
|
||||
return None
|
||||
|
||||
payload = datasource_config.config or {}
|
||||
form_body = payload.get("form_body")
|
||||
if form_body is not None:
|
||||
return form_body
|
||||
|
||||
if str(payload.get("body_type") or "").lower() == "form":
|
||||
candidate = payload.get("body")
|
||||
if isinstance(candidate, dict):
|
||||
return candidate
|
||||
return None
|
||||
|
||||
def _normalize_headers(self, headers: Any) -> dict[str, str]:
|
||||
if not isinstance(headers, dict):
|
||||
return {}
|
||||
normalized: dict[str, str] = {}
|
||||
for key, value in headers.items():
|
||||
header_name = str(key).strip()
|
||||
if not header_name or value is None:
|
||||
continue
|
||||
normalized[header_name] = str(value)
|
||||
return normalized
|
||||
|
||||
def _build_auth_headers(self, datasource_config: DataSourceConfig | None) -> dict[str, str]:
|
||||
if not datasource_config:
|
||||
return {}
|
||||
|
||||
auth_type = str(datasource_config.auth_type or "none").lower()
|
||||
auth_config = datasource_config.auth_config or {}
|
||||
if auth_type == "bearer" and auth_config.get("token"):
|
||||
return {"Authorization": f"Bearer {auth_config['token']}"}
|
||||
|
||||
if auth_type == "api_key" and auth_config.get("api_key"):
|
||||
location = str(auth_config.get("in") or auth_config.get("location") or "header").lower()
|
||||
if location == "query":
|
||||
return {}
|
||||
key_name = auth_config.get("key_name") or "X-API-Key"
|
||||
return {str(key_name): str(auth_config["api_key"])}
|
||||
|
||||
if auth_type == "basic":
|
||||
username = str(auth_config.get("username") or "")
|
||||
password = str(auth_config.get("password") or "")
|
||||
encoded = base64.b64encode(f"{username}:{password}".encode()).decode()
|
||||
return {"Authorization": f"Basic {encoded}"}
|
||||
|
||||
return {}
|
||||
|
||||
def _extract_candidates(self, response: Any, response_path: str | None) -> list[Any]:
|
||||
if response_path:
|
||||
extracted = self._extract_from_path(response, response_path)
|
||||
if isinstance(extracted, list):
|
||||
return extracted
|
||||
if isinstance(extracted, dict):
|
||||
for key in self.RESPONSE_CANDIDATE_KEYS:
|
||||
nested = extracted.get(key)
|
||||
if isinstance(nested, list):
|
||||
return nested
|
||||
return [extracted]
|
||||
return self.parse_response(response.json())
|
||||
|
||||
def parse_response(self, response: Any) -> list[dict[str, Any]]:
|
||||
if isinstance(response, dict):
|
||||
for key in self.RESPONSE_CANDIDATE_KEYS:
|
||||
nested = response.get(key)
|
||||
if isinstance(nested, list):
|
||||
return nested
|
||||
return []
|
||||
|
||||
if isinstance(response, list):
|
||||
return response
|
||||
return []
|
||||
|
||||
def _extract_from_path(self, payload: Any, path: str) -> Any:
|
||||
current = payload
|
||||
for segment in (part.strip() for part in path.split(".") if part.strip()):
|
||||
if isinstance(current, dict):
|
||||
current = current.get(segment)
|
||||
continue
|
||||
if isinstance(current, list):
|
||||
try:
|
||||
current = current[int(segment)]
|
||||
except (TypeError, ValueError, IndexError):
|
||||
return None
|
||||
continue
|
||||
return None
|
||||
return current
|
||||
|
||||
def _infer_source_type(self, item: dict[str, Any]) -> str:
|
||||
explicit = str(item.get("source_type") or item.get("type") or "").strip().lower()
|
||||
if explicit in {"iframe", "hls", "video", "external", "youtube"}:
|
||||
return explicit
|
||||
|
||||
youtube_video_id = self._clean_text(
|
||||
item.get("youtube_video_id")
|
||||
or item.get("video_id")
|
||||
or item.get("youtubeVideoId")
|
||||
)
|
||||
youtube_channel = self._clean_text(item.get("youtube_channel") or item.get("channel_handle"))
|
||||
embed_url = self._clean_url(item.get("embed_url") or item.get("embed") or item.get("page_url"))
|
||||
stream_url = self._clean_url(item.get("stream_url") or item.get("stream") or item.get("playback_url") or item.get("hls_url"))
|
||||
homepage_url = self._clean_url(item.get("homepage_url") or item.get("source_url") or item.get("website"))
|
||||
|
||||
if youtube_video_id or youtube_channel:
|
||||
return "youtube"
|
||||
if stream_url.endswith(".m3u8"):
|
||||
return "hls"
|
||||
if stream_url:
|
||||
return "video"
|
||||
if embed_url:
|
||||
parsed = urlparse(embed_url)
|
||||
if "youtube.com" in (parsed.netloc or "") or "youtu.be" in (parsed.netloc or ""):
|
||||
return "youtube"
|
||||
return "iframe"
|
||||
if homepage_url:
|
||||
return "external"
|
||||
return "iframe"
|
||||
|
||||
def _parse_enabled(self, item: dict[str, Any]) -> bool:
|
||||
if "is_enabled" in item:
|
||||
return self._to_bool(item.get("is_enabled"), default=True)
|
||||
if "enabled" in item:
|
||||
return self._to_bool(item.get("enabled"), default=True)
|
||||
if "active" in item:
|
||||
return self._to_bool(item.get("active"), default=True)
|
||||
if "status" in item:
|
||||
status = str(item.get("status") or "").strip().lower()
|
||||
if status in {"disabled", "inactive", "offline"}:
|
||||
return False
|
||||
if status in {"enabled", "active", "online", "live"}:
|
||||
return True
|
||||
return True
|
||||
|
||||
def _to_bool(self, value: Any, *, default: bool) -> bool:
|
||||
if isinstance(value, bool):
|
||||
return value
|
||||
if value in (None, ""):
|
||||
return default
|
||||
if isinstance(value, str):
|
||||
lowered = value.strip().lower()
|
||||
if lowered in {"1", "true", "yes", "on", "enabled", "active", "online", "live"}:
|
||||
return True
|
||||
if lowered in {"0", "false", "no", "off", "disabled", "inactive", "offline"}:
|
||||
return False
|
||||
return bool(value)
|
||||
|
||||
def _clean_text(self, value: Any) -> str:
|
||||
if value is None:
|
||||
return ""
|
||||
return str(value).strip()
|
||||
|
||||
def _clean_url(self, value: Any) -> str:
|
||||
text = self._clean_text(value)
|
||||
if not text:
|
||||
return ""
|
||||
parsed = urlparse(text)
|
||||
if parsed.scheme and parsed.scheme not in {"http", "https"}:
|
||||
return ""
|
||||
if parsed.scheme and not parsed.netloc:
|
||||
return ""
|
||||
return text
|
||||
|
||||
async def _fetch_iptv_org(self, channels_url: str, collector_config: dict[str, Any]) -> list[dict[str, Any]]:
|
||||
streams_url = self._clean_url(collector_config.get("streams_url")) or self.DEFAULT_IPTV_ORG_STREAMS_URL
|
||||
logos_url = self._clean_url(collector_config.get("logos_url")) or self.DEFAULT_IPTV_ORG_LOGOS_URL
|
||||
news_categories = {
|
||||
self._clean_text(value).lower()
|
||||
for value in (collector_config.get("news_categories") or self.DEFAULT_IPTV_ORG_NEWS_CATEGORIES)
|
||||
if self._clean_text(value)
|
||||
}
|
||||
exclude_categories = {
|
||||
self._clean_text(value).lower()
|
||||
for value in (collector_config.get("exclude_categories") or self.DEFAULT_IPTV_ORG_EXCLUDE_CATEGORIES)
|
||||
if self._clean_text(value)
|
||||
}
|
||||
try:
|
||||
max_sources = int(collector_config.get("max_sources", self.DEFAULT_IPTV_ORG_MAX_SOURCES))
|
||||
except (TypeError, ValueError):
|
||||
max_sources = self.DEFAULT_IPTV_ORG_MAX_SOURCES
|
||||
|
||||
timeout = self.DEFAULT_TIMEOUT
|
||||
try:
|
||||
timeout = float(collector_config.get("timeout", self.DEFAULT_TIMEOUT))
|
||||
except (TypeError, ValueError):
|
||||
timeout = self.DEFAULT_TIMEOUT
|
||||
|
||||
async with httpx.AsyncClient(timeout=timeout, follow_redirects=True) as client:
|
||||
channels_payload, streams_payload, logos_payload = await self._gather_iptv_org_payloads(
|
||||
client,
|
||||
channels_url,
|
||||
streams_url,
|
||||
logos_url,
|
||||
)
|
||||
|
||||
channels = channels_payload if isinstance(channels_payload, list) else []
|
||||
streams = streams_payload if isinstance(streams_payload, list) else []
|
||||
logos = logos_payload if isinstance(logos_payload, list) else []
|
||||
|
||||
logo_by_channel = {
|
||||
self._clean_text(item.get("channel")): self._clean_url(item.get("url"))
|
||||
for item in logos
|
||||
if isinstance(item, dict) and self._clean_text(item.get("channel")) and self._clean_url(item.get("url"))
|
||||
}
|
||||
|
||||
streams_by_channel: dict[str, list[dict[str, Any]]] = {}
|
||||
for stream in streams:
|
||||
if not isinstance(stream, dict):
|
||||
continue
|
||||
channel_id = self._clean_text(stream.get("channel"))
|
||||
if not channel_id:
|
||||
continue
|
||||
streams_by_channel.setdefault(channel_id, []).append(stream)
|
||||
|
||||
normalized: list[dict[str, Any]] = []
|
||||
for channel in channels:
|
||||
if not isinstance(channel, dict):
|
||||
continue
|
||||
|
||||
categories = [
|
||||
self._clean_text(value).lower()
|
||||
for value in (channel.get("categories") or [])
|
||||
if self._clean_text(value)
|
||||
]
|
||||
if news_categories and not any(category in news_categories for category in categories):
|
||||
continue
|
||||
if exclude_categories and any(category in exclude_categories for category in categories):
|
||||
continue
|
||||
if channel.get("is_nsfw") is True:
|
||||
continue
|
||||
if channel.get("closed"):
|
||||
continue
|
||||
|
||||
channel_id = self._clean_text(channel.get("id"))
|
||||
if not channel_id:
|
||||
continue
|
||||
|
||||
stream = self._pick_iptv_org_stream(streams_by_channel.get(channel_id) or [])
|
||||
if not stream:
|
||||
continue
|
||||
|
||||
stream_url = self._clean_url(stream.get("url"))
|
||||
if not stream_url:
|
||||
continue
|
||||
|
||||
name = self._clean_text(channel.get("name")) or channel_id
|
||||
notes_parts = [
|
||||
f"Imported from IPTV-org catalog ({channel_id})",
|
||||
f"Categories: {', '.join(categories)}" if categories else "",
|
||||
f"Quality: {self._clean_text(stream.get('quality'))}" if self._clean_text(stream.get("quality")) else "",
|
||||
]
|
||||
metadata = {
|
||||
"provider": self._clean_text(channel.get("network")) or "IPTV-org",
|
||||
"region": self._clean_text(channel.get("country")) or "Global",
|
||||
"language": "und",
|
||||
"source_type": "hls" if stream_url.endswith(".m3u8") else "video",
|
||||
"embed_url": "",
|
||||
"stream_url": stream_url,
|
||||
"homepage_url": self._clean_url(channel.get("website")),
|
||||
"poster_url": logo_by_channel.get(channel_id, ""),
|
||||
"youtube_video_id": "",
|
||||
"youtube_channel": "",
|
||||
"sort_order": 400 + len(normalized),
|
||||
"notes": "; ".join(part for part in notes_parts if part),
|
||||
"is_enabled": True,
|
||||
"collector_adapter": "iptv_org",
|
||||
"channel_id": channel_id,
|
||||
"categories": categories,
|
||||
"quality": self._clean_text(stream.get("quality")),
|
||||
"stream_label": self._clean_text(stream.get("label") or stream.get("title")),
|
||||
"stream_referrer": self._clean_text(stream.get("referrer")),
|
||||
"stream_user_agent": self._clean_text(stream.get("user_agent")),
|
||||
}
|
||||
|
||||
normalized.append(
|
||||
{
|
||||
"source_id": channel_id,
|
||||
"name": name,
|
||||
"description": metadata["notes"],
|
||||
"metadata": metadata,
|
||||
"reference_date": datetime.now(UTC).isoformat(),
|
||||
}
|
||||
)
|
||||
if len(normalized) >= max_sources:
|
||||
break
|
||||
|
||||
return normalized
|
||||
|
||||
async def _gather_iptv_org_payloads(
|
||||
self,
|
||||
client: httpx.AsyncClient,
|
||||
channels_url: str,
|
||||
streams_url: str,
|
||||
logos_url: str,
|
||||
) -> tuple[Any, Any, Any]:
|
||||
headers = dict(self.DEFAULT_HEADERS)
|
||||
channels_payload, streams_payload, logos_payload = await asyncio.gather(
|
||||
client.get(channels_url, headers=headers),
|
||||
client.get(streams_url, headers=headers),
|
||||
client.get(logos_url, headers=headers),
|
||||
)
|
||||
channels_payload.raise_for_status()
|
||||
streams_payload.raise_for_status()
|
||||
logos_payload.raise_for_status()
|
||||
return channels_payload.json(), streams_payload.json(), logos_payload.json()
|
||||
|
||||
def _pick_iptv_org_stream(self, streams: list[dict[str, Any]]) -> dict[str, Any] | None:
|
||||
if not streams:
|
||||
return None
|
||||
|
||||
def score(stream: dict[str, Any]) -> tuple[int, int]:
|
||||
url = self._clean_url(stream.get("url"))
|
||||
quality = self._clean_text(stream.get("quality")).lower()
|
||||
quality_score = 0
|
||||
if quality.endswith("p"):
|
||||
try:
|
||||
quality_score = int(quality[:-1])
|
||||
except ValueError:
|
||||
quality_score = 0
|
||||
stream_score = 1000 if url.endswith(".m3u8") else 0
|
||||
return stream_score, quality_score
|
||||
|
||||
sorted_streams = sorted(streams, key=score, reverse=True)
|
||||
return sorted_streams[0]
|
||||
|
||||
def parse_response(self, response: Any, *, response_path: str | None = None) -> list[dict[str, Any]]:
|
||||
candidates = self._extract_candidates(response, response_path)
|
||||
candidates = response.get("sources") or response.get("streams") or response.get("data") or []
|
||||
elif isinstance(response, list):
|
||||
candidates = response
|
||||
else:
|
||||
candidates = []
|
||||
|
||||
normalized: list[dict[str, Any]] = []
|
||||
for index, item in enumerate(candidates):
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
|
||||
stream_id = (
|
||||
item.get("id")
|
||||
or item.get("source_id")
|
||||
or item.get("slug")
|
||||
or item.get("channel_id")
|
||||
or item.get("code")
|
||||
or f"news-live-{index + 1}"
|
||||
)
|
||||
name = self._clean_text(
|
||||
item.get("name")
|
||||
or item.get("title")
|
||||
or item.get("channel")
|
||||
or item.get("display_name")
|
||||
or f"News Live {index + 1}"
|
||||
)
|
||||
stream_id = item.get("id") or item.get("source_id") or item.get("slug") or f"news-live-{index + 1}"
|
||||
name = str(item.get("name") or item.get("title") or f"News Live {index + 1}").strip()
|
||||
if not name:
|
||||
continue
|
||||
|
||||
source_type = self._infer_source_type(item)
|
||||
stream_url = self._clean_url(
|
||||
item.get("stream_url")
|
||||
or item.get("stream")
|
||||
or item.get("playback_url")
|
||||
or item.get("hls_url")
|
||||
or item.get("m3u8_url")
|
||||
)
|
||||
embed_url = self._clean_url(
|
||||
item.get("embed_url")
|
||||
or item.get("embed")
|
||||
or item.get("page_url")
|
||||
or (item.get("url") if source_type == "iframe" else "")
|
||||
)
|
||||
homepage_url = self._clean_url(
|
||||
item.get("homepage_url")
|
||||
or item.get("source_url")
|
||||
or item.get("website")
|
||||
or item.get("url")
|
||||
)
|
||||
metadata = {
|
||||
"provider": self._clean_text(item.get("provider") or item.get("publisher") or item.get("network")) or "Collector",
|
||||
"region": self._clean_text(item.get("region") or item.get("country") or item.get("market")) or "Global",
|
||||
"language": self._clean_text(item.get("language") or item.get("lang") or item.get("locale")) or "und",
|
||||
"source_type": source_type,
|
||||
"embed_url": embed_url,
|
||||
"stream_url": stream_url,
|
||||
"homepage_url": homepage_url,
|
||||
"poster_url": self._clean_url(item.get("poster_url") or item.get("thumbnail_url") or item.get("logo_url")),
|
||||
"youtube_video_id": self._clean_text(
|
||||
item.get("youtube_video_id")
|
||||
or item.get("video_id")
|
||||
or item.get("youtubeVideoId")
|
||||
),
|
||||
"youtube_channel": self._clean_text(
|
||||
item.get("youtube_channel")
|
||||
or item.get("channel_handle")
|
||||
or item.get("youtubeChannel")
|
||||
),
|
||||
"provider": item.get("provider") or item.get("publisher") or "Collector",
|
||||
"region": item.get("region") or item.get("country") or "Global",
|
||||
"language": item.get("language") or "und",
|
||||
"source_type": item.get("source_type") or "iframe",
|
||||
"embed_url": item.get("embed_url") or item.get("url") or "",
|
||||
"stream_url": item.get("stream_url") or "",
|
||||
"homepage_url": item.get("homepage_url") or item.get("source_url") or "",
|
||||
"poster_url": item.get("poster_url") or "",
|
||||
"sort_order": item.get("sort_order", 200 + index),
|
||||
"notes": self._clean_text(item.get("notes") or item.get("description") or item.get("summary")),
|
||||
"is_enabled": self._parse_enabled(item),
|
||||
"notes": item.get("notes") or item.get("description") or "",
|
||||
"is_enabled": item.get("is_enabled", True),
|
||||
}
|
||||
|
||||
normalized.append(
|
||||
@@ -563,7 +72,7 @@ class NewsLiveStreamsCollector(BaseCollector):
|
||||
"name": name,
|
||||
"description": metadata["notes"],
|
||||
"metadata": metadata,
|
||||
"reference_date": item.get("reference_date") or datetime.now(UTC).isoformat(),
|
||||
"reference_date": item.get("reference_date", datetime.now(UTC).isoformat()),
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
@@ -39,23 +39,12 @@ class NRODelegatedPrefixGeoCollector(BaseCollector):
|
||||
self._current_task.total_records = total_expected
|
||||
self._current_task.records_processed = 0
|
||||
self._current_task.progress = 0.0
|
||||
self._current_task.phase_progress = 0.0
|
||||
self._current_task.phase_message = "正在下载 NRO delegated 数据"
|
||||
self._current_task.phase_current = 0
|
||||
self._current_task.phase_total = total_expected
|
||||
self._current_task.phase_unit = "bytes"
|
||||
await self._db_session.commit()
|
||||
await self._publish_task_update(force=True)
|
||||
|
||||
async def on_progress(downloaded: int, total: int | None) -> None:
|
||||
if not total or total <= 0:
|
||||
return
|
||||
await self.update_phase_progress(
|
||||
current=min(downloaded, total),
|
||||
total=total,
|
||||
unit="bytes",
|
||||
message="正在下载 NRO delegated 数据",
|
||||
)
|
||||
await self.update_progress(min(downloaded, total), commit=True)
|
||||
|
||||
body_path = await self._downloader.download_file(
|
||||
|
||||
@@ -40,23 +40,12 @@ class OpenGeoFeedPrefixGeoCollector(BaseCollector):
|
||||
self._current_task.total_records = total_expected
|
||||
self._current_task.records_processed = 0
|
||||
self._current_task.progress = 0.0
|
||||
self._current_task.phase_progress = 0.0
|
||||
self._current_task.phase_message = "正在下载 OpenGeoFeed 数据"
|
||||
self._current_task.phase_current = 0
|
||||
self._current_task.phase_total = total_expected
|
||||
self._current_task.phase_unit = "bytes"
|
||||
await self._db_session.commit()
|
||||
await self._publish_task_update(force=True)
|
||||
|
||||
async def on_progress(downloaded: int, total: int | None) -> None:
|
||||
if not total or total <= 0:
|
||||
return
|
||||
await self.update_phase_progress(
|
||||
current=min(downloaded, total),
|
||||
total=total,
|
||||
unit="bytes",
|
||||
message="正在下载 OpenGeoFeed 数据",
|
||||
)
|
||||
await self.update_progress(min(downloaded, total), commit=True)
|
||||
|
||||
body_path = await self._downloader.download_file(
|
||||
|
||||
@@ -1,273 +0,0 @@
|
||||
"""BarentsWatch AIS collector for vessel tracking."""
|
||||
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
from sqlalchemy import delete, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.models.vessel import VesselPosition, VesselStatic
|
||||
from app.services.barentswatch import (
|
||||
BARENTSWATCH_LATEST_URL,
|
||||
fetch_barentswatch_access_token,
|
||||
resolve_barentswatch_config,
|
||||
)
|
||||
from app.services.collectors.base import BaseCollector
|
||||
|
||||
|
||||
VESSEL_TYPE_NAMES = {
|
||||
30: "Fishing",
|
||||
35: "Military",
|
||||
60: "Passenger",
|
||||
70: "Cargo",
|
||||
80: "Tanker",
|
||||
}
|
||||
|
||||
|
||||
class VesselAISCollector(BaseCollector):
|
||||
"""Collect latest AIS positions and append them to vessel tables."""
|
||||
|
||||
name = "barentswatch_vessels"
|
||||
priority = "P1"
|
||||
module = "L4"
|
||||
frequency_hours = 1
|
||||
data_type = "vessel_ais"
|
||||
|
||||
@property
|
||||
def base_url(self) -> str:
|
||||
return self._resolved_url or BARENTSWATCH_LATEST_URL
|
||||
|
||||
async def _get_access_token(self, client: httpx.AsyncClient) -> str | None:
|
||||
config = await resolve_barentswatch_config(self._db_session)
|
||||
return await fetch_barentswatch_access_token(client, config)
|
||||
|
||||
async def fetch(self) -> list[dict[str, Any]]:
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
headers: dict[str, str] = {}
|
||||
token = await self._get_access_token(client)
|
||||
if token:
|
||||
headers["Authorization"] = f"Bearer {token}"
|
||||
|
||||
response = await client.get(self.base_url, headers=headers)
|
||||
if response.status_code == 401 and not token:
|
||||
return self._get_sample_data()
|
||||
response.raise_for_status()
|
||||
payload = response.json()
|
||||
|
||||
if isinstance(payload, list):
|
||||
return [item for item in payload if isinstance(item, dict)]
|
||||
if isinstance(payload, dict):
|
||||
for key in ("features", "data", "items", "vessels"):
|
||||
value = payload.get(key)
|
||||
if isinstance(value, list):
|
||||
if key == "features":
|
||||
return [
|
||||
{
|
||||
**(item.get("properties") or {}),
|
||||
"geometry": item.get("geometry"),
|
||||
}
|
||||
for item in value
|
||||
if isinstance(item, dict)
|
||||
]
|
||||
return [item for item in value if isinstance(item, dict)]
|
||||
return self._get_sample_data()
|
||||
|
||||
def transform(self, raw_data: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
transformed = []
|
||||
for item in raw_data:
|
||||
record = self._normalize_record(item)
|
||||
if record:
|
||||
transformed.append(record)
|
||||
return transformed
|
||||
|
||||
async def _save_data(
|
||||
self,
|
||||
db: AsyncSession,
|
||||
data: list[dict[str, Any]],
|
||||
task_id: int | None = None,
|
||||
snapshot_id: int | None = None,
|
||||
) -> int:
|
||||
now = datetime.now(UTC)
|
||||
records_added = 0
|
||||
|
||||
for index, item in enumerate(data):
|
||||
static = await db.get(VesselStatic, item["mmsi"])
|
||||
if static is None:
|
||||
static = VesselStatic(mmsi=item["mmsi"])
|
||||
db.add(static)
|
||||
|
||||
for field in (
|
||||
"name",
|
||||
"callsign",
|
||||
"vessel_type",
|
||||
"vessel_type_name",
|
||||
"flag",
|
||||
"length",
|
||||
"width",
|
||||
"draught",
|
||||
"imo",
|
||||
):
|
||||
value = item.get(field)
|
||||
if value not in (None, ""):
|
||||
setattr(static, field, value)
|
||||
static.updated_at = now
|
||||
|
||||
db.add(
|
||||
VesselPosition(
|
||||
mmsi=item["mmsi"],
|
||||
lat=item["lat"],
|
||||
lon=item["lon"],
|
||||
sog=item.get("sog"),
|
||||
cog=item.get("cog"),
|
||||
heading=item.get("heading"),
|
||||
nav_status=item.get("nav_status"),
|
||||
received_at=item.get("received_at") or now,
|
||||
)
|
||||
)
|
||||
records_added += 1
|
||||
|
||||
if (index + 1) % 1000 == 0:
|
||||
await self.update_progress(index + 1, commit=True)
|
||||
|
||||
await db.execute(
|
||||
delete(VesselPosition).where(VesselPosition.received_at < now - timedelta(hours=24))
|
||||
)
|
||||
await db.commit()
|
||||
await self.update_progress(records_added, force=True)
|
||||
return records_added
|
||||
|
||||
def _normalize_record(self, item: dict[str, Any]) -> dict[str, Any] | None:
|
||||
mmsi = _as_int(_pick(item, "mmsi", "MMSI", "Mmsi"))
|
||||
lat = _as_float(_pick(item, "lat", "latitude", "Latitude"))
|
||||
lon = _as_float(_pick(item, "lon", "lng", "longitude", "Longitude"))
|
||||
|
||||
geometry = item.get("geometry")
|
||||
coordinates = geometry.get("coordinates") if isinstance(geometry, dict) else None
|
||||
if (lat is None or lon is None) and isinstance(coordinates, list) and len(coordinates) >= 2:
|
||||
lon = _as_float(coordinates[0])
|
||||
lat = _as_float(coordinates[1])
|
||||
|
||||
if mmsi is None or lat is None or lon is None:
|
||||
return None
|
||||
if not (-90 <= lat <= 90 and -180 <= lon <= 180):
|
||||
return None
|
||||
|
||||
vessel_type = _as_int(_pick(item, "vessel_type", "shipType", "ship_type", "ShipType"))
|
||||
vessel_type_name = (
|
||||
_pick(item, "vessel_type_name", "shipTypeName", "ship_type_name", "VesselTypeName")
|
||||
or _vessel_type_name(vessel_type)
|
||||
)
|
||||
received_at = _parse_datetime(_pick(item, "received_at", "timestamp", "time", "msgtime"))
|
||||
|
||||
return {
|
||||
"mmsi": mmsi,
|
||||
"name": _pick(item, "name", "shipName", "ship_name", "Name"),
|
||||
"callsign": _pick(item, "callsign", "callSign", "CallSign"),
|
||||
"vessel_type": vessel_type,
|
||||
"vessel_type_name": vessel_type_name,
|
||||
"flag": _pick(item, "flag", "country", "Flag"),
|
||||
"length": _as_float(_pick(item, "length", "shipLength", "Length")),
|
||||
"width": _as_float(_pick(item, "width", "shipWidth", "Width")),
|
||||
"draught": _as_float(_pick(item, "draught", "draft", "Draught")),
|
||||
"imo": _as_int(_pick(item, "imo", "IMO", "imoNumber")),
|
||||
"lat": lat,
|
||||
"lon": lon,
|
||||
"sog": _as_float(_pick(item, "sog", "speedOverGround", "SOG")),
|
||||
"cog": _as_float(_pick(item, "cog", "courseOverGround", "COG")),
|
||||
"heading": _as_int(_pick(item, "heading", "trueHeading", "Heading")),
|
||||
"nav_status": _as_int(_pick(item, "nav_status", "navStatus", "NavigationalStatus")),
|
||||
"received_at": received_at,
|
||||
}
|
||||
|
||||
def _get_sample_data(self) -> list[dict[str, Any]]:
|
||||
return [
|
||||
{
|
||||
"mmsi": 257123000,
|
||||
"name": "OSLO TRADER",
|
||||
"lat": 59.91,
|
||||
"lon": 10.73,
|
||||
"sog": 12.4,
|
||||
"cog": 214,
|
||||
"heading": 215,
|
||||
"nav_status": 0,
|
||||
"vessel_type": 70,
|
||||
"vessel_type_name": "Cargo",
|
||||
"flag": "NO",
|
||||
"length": 185,
|
||||
},
|
||||
{
|
||||
"mmsi": 257456000,
|
||||
"name": "NORDIC FJORD",
|
||||
"lat": 60.39,
|
||||
"lon": 5.32,
|
||||
"sog": 0.2,
|
||||
"cog": 82,
|
||||
"heading": 80,
|
||||
"nav_status": 1,
|
||||
"vessel_type": 60,
|
||||
"vessel_type_name": "Passenger",
|
||||
"flag": "NO",
|
||||
"length": 126,
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
def _pick(item: dict[str, Any], *keys: str) -> Any:
|
||||
for key in keys:
|
||||
if key in item and item[key] not in (None, ""):
|
||||
return item[key]
|
||||
return None
|
||||
|
||||
|
||||
def _as_float(value: Any) -> float | None:
|
||||
try:
|
||||
if value in (None, ""):
|
||||
return None
|
||||
return float(value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def _as_int(value: Any) -> int | None:
|
||||
try:
|
||||
if value in (None, ""):
|
||||
return None
|
||||
return int(float(value))
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def _parse_datetime(value: Any) -> datetime | None:
|
||||
if isinstance(value, datetime):
|
||||
return value if value.tzinfo else value.replace(tzinfo=UTC)
|
||||
if not value:
|
||||
return None
|
||||
if isinstance(value, (int, float)):
|
||||
timestamp = float(value)
|
||||
if timestamp > 10_000_000_000:
|
||||
timestamp /= 1000
|
||||
return datetime.fromtimestamp(timestamp, UTC)
|
||||
if isinstance(value, str):
|
||||
try:
|
||||
parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
|
||||
return parsed if parsed.tzinfo else parsed.replace(tzinfo=UTC)
|
||||
except ValueError:
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
def _vessel_type_name(vessel_type: int | None) -> str:
|
||||
if vessel_type is None:
|
||||
return "Other"
|
||||
if 70 <= vessel_type <= 79:
|
||||
return "Cargo"
|
||||
if 80 <= vessel_type <= 89:
|
||||
return "Tanker"
|
||||
if 60 <= vessel_type <= 69:
|
||||
return "Passenger"
|
||||
if vessel_type == 30:
|
||||
return "Fishing"
|
||||
if vessel_type == 35:
|
||||
return "Military"
|
||||
return VESSEL_TYPE_NAMES.get(vessel_type, "Other")
|
||||
@@ -1,165 +0,0 @@
|
||||
"""Credential setup guides for collector integrations."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.schemas.ai import SituationalAnalysisRequest
|
||||
from app.services.ai_client import AIProviderClient
|
||||
|
||||
|
||||
CREDENTIAL_GUIDES_CATEGORY = "collector_credential_guides"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CredentialGuideDefault:
|
||||
provider: str
|
||||
title: str
|
||||
prompt: str
|
||||
markdown: str
|
||||
|
||||
|
||||
BARENTSWATCH_DEFAULT_GUIDE = CredentialGuideDefault(
|
||||
provider="barentswatch",
|
||||
title="BarentsWatch AIS 凭证获取教程",
|
||||
prompt=(
|
||||
"请生成一份中文教程,指导开发者获取 BarentsWatch Live AIS API 的 "
|
||||
"OAuth client credentials。教程要面向已经有本地开发环境的人,包含注册/登录、"
|
||||
"创建 client、申请或确认 ais scope、复制 client id 和 client secret、"
|
||||
"在系统设置中填写并验证连接、常见失败排查。不要编造具体页面按钮文案,"
|
||||
"必须参考官方 tutorial:https://developer.barentswatch.no/docs/tutorial 。"
|
||||
"必须强调 Live AIS 要选择 AIS-client / AIS - API,而不是普通 API-client。"
|
||||
"如果步骤可能变化,要提醒以 BarentsWatch developer portal 当前页面为准。"
|
||||
),
|
||||
markdown="""## BarentsWatch AIS 凭证获取
|
||||
|
||||
官方教程:https://developer.barentswatch.no/docs/tutorial
|
||||
|
||||
1. 先打开上面的 BarentsWatch 官方 tutorial,按官方流程登录或注册开发者账号。
|
||||
2. 在 Developer access 页面选择 `AIS - API`,不要选择普通的 `BarentsWatch - API`。
|
||||
3. 在 `AIS - API` 下创建用于 Planet 的 AIS client。
|
||||
4. 创建时记下你设置的 password / client secret。
|
||||
5. 回到 My Page 复制完整 `Client ID`。它通常长得像 `your.email@example.com:client-name`。
|
||||
6. 回到 Planet 的 `设置 -> 采集器设置 -> BarentsWatch AIS`,填入 `Client ID` 和 `Client Secret`。
|
||||
7. 点击 `连接` 验证 token 和 AIS endpoint 是否可访问。
|
||||
8. 连接成功后保存凭证。
|
||||
|
||||
### 请求规则
|
||||
|
||||
- Token 地址:`https://id.barentswatch.no/connect/token`
|
||||
- 请求方式:`POST`
|
||||
- Content-Type:`application/x-www-form-urlencoded`
|
||||
- Body 必须包含:`grant_type=client_credentials`、`client_id`、`client_secret`、`scope=ais`
|
||||
- `client_id`、`client_secret`、`scope`、`grant_type` 都要放在 body,不要放在 header。
|
||||
- AIS 数据请求使用 header:`Authorization: Bearer <access_token>`
|
||||
|
||||
### 常见排查
|
||||
|
||||
- `未找到凭证`:确认 `Client ID` 和 `Client Secret` 已填写,或已经写入 `~/.zshrc`。
|
||||
- `HTTP 401/403`:通常是选成了普通 `BarentsWatch - API` client、client secret 错误,或 token 请求没有使用 `scope=ais`。
|
||||
- `network` 错误:检查本机是否能访问 `id.barentswatch.no` 和 `live.ais.barentswatch.no`。
|
||||
- Endpoint 建议保持默认:`https://live.ais.barentswatch.no/v1/latest/combined`。
|
||||
""",
|
||||
)
|
||||
|
||||
|
||||
DEFAULT_CREDENTIAL_GUIDES = {
|
||||
BARENTSWATCH_DEFAULT_GUIDE.provider: BARENTSWATCH_DEFAULT_GUIDE,
|
||||
}
|
||||
|
||||
|
||||
async def _get_guide_store(db) -> tuple[SystemSetting | None, dict[str, Any]]:
|
||||
result = await db.execute(
|
||||
select(SystemSetting).where(SystemSetting.category == CREDENTIAL_GUIDES_CATEGORY)
|
||||
)
|
||||
record = result.scalar_one_or_none()
|
||||
payload = dict(record.payload or {}) if record and isinstance(record.payload, dict) else {}
|
||||
return record, payload
|
||||
|
||||
|
||||
async def get_credential_guide(db, provider: str) -> dict[str, Any]:
|
||||
default = DEFAULT_CREDENTIAL_GUIDES.get(provider)
|
||||
if default is None:
|
||||
raise ValueError(f"Unsupported credential guide provider: {provider}")
|
||||
|
||||
_record, store = await _get_guide_store(db)
|
||||
custom = store.get(provider) if isinstance(store.get(provider), dict) else None
|
||||
return {
|
||||
"provider": provider,
|
||||
"title": custom.get("title") if custom else default.title,
|
||||
"markdown": custom.get("markdown") if custom else default.markdown,
|
||||
"prompt": default.prompt,
|
||||
"source": "ai" if custom else "default",
|
||||
}
|
||||
|
||||
|
||||
async def save_credential_guide(db, provider: str, title: str, markdown: str) -> dict[str, Any]:
|
||||
default = DEFAULT_CREDENTIAL_GUIDES.get(provider)
|
||||
if default is None:
|
||||
raise ValueError(f"Unsupported credential guide provider: {provider}")
|
||||
|
||||
record, store = await _get_guide_store(db)
|
||||
store[provider] = {
|
||||
"title": title or default.title,
|
||||
"markdown": markdown,
|
||||
}
|
||||
if record is None:
|
||||
db.add(SystemSetting(category=CREDENTIAL_GUIDES_CATEGORY, payload=store))
|
||||
else:
|
||||
record.payload = store
|
||||
await db.commit()
|
||||
return await get_credential_guide(db, provider)
|
||||
|
||||
|
||||
async def reset_credential_guide(db, provider: str) -> dict[str, Any]:
|
||||
default = DEFAULT_CREDENTIAL_GUIDES.get(provider)
|
||||
if default is None:
|
||||
raise ValueError(f"Unsupported credential guide provider: {provider}")
|
||||
|
||||
record, store = await _get_guide_store(db)
|
||||
if provider in store:
|
||||
store.pop(provider, None)
|
||||
if record is not None:
|
||||
record.payload = store
|
||||
await db.commit()
|
||||
return await get_credential_guide(db, provider)
|
||||
|
||||
|
||||
async def generate_credential_guide(
|
||||
db,
|
||||
provider: str,
|
||||
ai_client: AIProviderClient,
|
||||
) -> dict[str, Any]:
|
||||
default = DEFAULT_CREDENTIAL_GUIDES.get(provider)
|
||||
if default is None:
|
||||
raise ValueError(f"Unsupported credential guide provider: {provider}")
|
||||
|
||||
response = await ai_client.analyze(
|
||||
SituationalAnalysisRequest(
|
||||
title=f"Generate credential guide for {provider}",
|
||||
objective=default.prompt,
|
||||
context={
|
||||
"provider": provider,
|
||||
"current_default_guide": default.markdown,
|
||||
"product_context": "Planet collector credential settings",
|
||||
},
|
||||
observations=[
|
||||
"Use concise Chinese markdown.",
|
||||
"Prefer stable concepts over brittle UI labels.",
|
||||
"Include verification and troubleshooting steps.",
|
||||
],
|
||||
constraints=[
|
||||
"Do not ask the user for secrets.",
|
||||
"Do not include fabricated screenshots.",
|
||||
"Return markdown only.",
|
||||
],
|
||||
)
|
||||
)
|
||||
markdown = response.content.strip()
|
||||
if not markdown:
|
||||
markdown = default.markdown
|
||||
return await save_credential_guide(db, provider, default.title, markdown)
|
||||
@@ -1,384 +0,0 @@
|
||||
"""Connectivity validation helpers for built-in datasource overrides."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
from sqlalchemy import func, select
|
||||
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
from app.core.datasource_defaults import DEFAULT_DATASOURCES
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.models.datasource import DataSource
|
||||
from app.models.datasource_config import DataSourceConfig
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.services.barentswatch import (
|
||||
_read_zshrc_env,
|
||||
fetch_barentswatch_access_token,
|
||||
resolve_barentswatch_config,
|
||||
)
|
||||
|
||||
|
||||
CONNECTIVITY_VALIDATION_KEY = "connectivity_validation"
|
||||
CONNECTIVITY_STORE_CATEGORY = "datasource_connectivity_validations"
|
||||
|
||||
|
||||
def _sha256_json(payload: Any) -> str:
|
||||
encoded = json.dumps(payload, ensure_ascii=False, sort_keys=True, default=str).encode("utf-8")
|
||||
return hashlib.sha256(encoded).hexdigest()
|
||||
|
||||
|
||||
def _resolve_spacetrack_credentials() -> tuple[str, str, str]:
|
||||
zshrc_env = _read_zshrc_env()
|
||||
username = os.getenv("SPACETRACK_USERNAME") or zshrc_env.get("SPACETRACK_USERNAME") or ""
|
||||
password = os.getenv("SPACETRACK_PASSWORD") or zshrc_env.get("SPACETRACK_PASSWORD") or ""
|
||||
source = "environment" if os.getenv("SPACETRACK_USERNAME") or os.getenv("SPACETRACK_PASSWORD") else ""
|
||||
if not source and (username or password):
|
||||
source = "~/.zshrc"
|
||||
return username, password, source or "missing"
|
||||
|
||||
|
||||
def strip_connectivity_validation(config: dict | None) -> dict:
|
||||
cleaned = dict(config or {})
|
||||
cleaned.pop(CONNECTIVITY_VALIDATION_KEY, None)
|
||||
return cleaned
|
||||
|
||||
|
||||
def merge_connectivity_validation(existing_config: dict | None, next_config: dict | None) -> dict:
|
||||
merged = strip_connectivity_validation(next_config)
|
||||
validation = (existing_config or {}).get(CONNECTIVITY_VALIDATION_KEY)
|
||||
if validation:
|
||||
merged[CONNECTIVITY_VALIDATION_KEY] = validation
|
||||
return merged
|
||||
|
||||
|
||||
def get_connectivity_validation(config: DataSourceConfig | None) -> dict | None:
|
||||
validation = (config.config or {}).get(CONNECTIVITY_VALIDATION_KEY) if config else None
|
||||
return validation if isinstance(validation, dict) else None
|
||||
|
||||
|
||||
async def build_builtin_connectivity_checksum(
|
||||
source: str,
|
||||
endpoint: str,
|
||||
auth_type: str,
|
||||
headers: dict | None,
|
||||
config: dict | None,
|
||||
db=None,
|
||||
credential_override: dict[str, str] | None = None,
|
||||
) -> tuple[str, dict[str, Any]]:
|
||||
defaults = DEFAULT_DATASOURCES.get(source, {})
|
||||
credential_provider = defaults.get("credential_provider")
|
||||
credential_fingerprint = ""
|
||||
credential_source = "none"
|
||||
has_credentials = not defaults.get("requires_credentials", False)
|
||||
|
||||
if credential_provider == "barentswatch":
|
||||
if credential_override:
|
||||
client_id = credential_override.get("client_id", "")
|
||||
client_secret = credential_override.get("client_secret", "")
|
||||
credential_source = "draft"
|
||||
else:
|
||||
barentswatch_config = await resolve_barentswatch_config(db)
|
||||
client_id = barentswatch_config.client_id
|
||||
client_secret = barentswatch_config.client_secret
|
||||
credential_source = barentswatch_config.credential_source
|
||||
has_credentials = bool(client_id and client_secret)
|
||||
credential_fingerprint = _sha256_json(
|
||||
{
|
||||
"client_id": client_id,
|
||||
"client_secret": client_secret,
|
||||
}
|
||||
)
|
||||
elif credential_provider == "spacetrack":
|
||||
username, password, credential_source = _resolve_spacetrack_credentials()
|
||||
has_credentials = bool(username and password)
|
||||
credential_fingerprint = _sha256_json(
|
||||
{
|
||||
"username": username,
|
||||
"password": password,
|
||||
}
|
||||
)
|
||||
elif defaults.get("requires_credentials"):
|
||||
credential_source = str(credential_provider or "unsupported")
|
||||
|
||||
checksum_payload = {
|
||||
"source": source,
|
||||
"endpoint": endpoint,
|
||||
"auth_type": "none",
|
||||
"headers": headers or {},
|
||||
"config": strip_connectivity_validation(config),
|
||||
"credential_provider": credential_provider or "none",
|
||||
"credential_fingerprint": credential_fingerprint,
|
||||
}
|
||||
return _sha256_json(checksum_payload), {
|
||||
"requires_credentials": bool(defaults.get("requires_credentials", False)),
|
||||
"credential_provider": credential_provider,
|
||||
"credential_source": credential_source,
|
||||
"has_credentials": has_credentials,
|
||||
}
|
||||
|
||||
|
||||
async def test_builtin_connectivity(
|
||||
source: str,
|
||||
endpoint: str,
|
||||
auth_type: str,
|
||||
headers: dict | None,
|
||||
config: dict | None,
|
||||
db=None,
|
||||
) -> dict[str, Any]:
|
||||
defaults = DEFAULT_DATASOURCES.get(source)
|
||||
if not defaults:
|
||||
return {
|
||||
"success": False,
|
||||
"message": "未知内置采集器,无法执行连接校验。",
|
||||
}
|
||||
|
||||
checksum, credential_context = await build_builtin_connectivity_checksum(
|
||||
source,
|
||||
endpoint,
|
||||
auth_type,
|
||||
headers,
|
||||
config,
|
||||
db,
|
||||
)
|
||||
if credential_context["requires_credentials"] and not credential_context["has_credentials"]:
|
||||
return {
|
||||
"success": False,
|
||||
"checksum": checksum,
|
||||
"stage": "credentials",
|
||||
"message": "该采集器需要凭证,请先到采集器凭证设置中配置。",
|
||||
"settings_tab": "collector_credentials",
|
||||
**credential_context,
|
||||
}
|
||||
supported_credential_providers = {"barentswatch", "spacetrack"}
|
||||
if (
|
||||
credential_context["requires_credentials"]
|
||||
and credential_context["credential_provider"] not in supported_credential_providers
|
||||
):
|
||||
return {
|
||||
"success": False,
|
||||
"checksum": checksum,
|
||||
"stage": "credentials",
|
||||
"message": "该采集器的凭证链路尚未接入,暂时无法完成连接校验。",
|
||||
"settings_tab": "collector_credentials",
|
||||
**credential_context,
|
||||
}
|
||||
|
||||
request_headers = {str(key): str(value) for key, value in (headers or {}).items()}
|
||||
request_config = strip_connectivity_validation(config)
|
||||
timeout = float(request_config.get("timeout") or 30)
|
||||
request_endpoint = endpoint
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=timeout, follow_redirects=True) as client:
|
||||
if credential_context["credential_provider"] == "barentswatch":
|
||||
barentswatch_config = await resolve_barentswatch_config(db)
|
||||
token = await fetch_barentswatch_access_token(client, barentswatch_config)
|
||||
if not token:
|
||||
return {
|
||||
"success": False,
|
||||
"checksum": checksum,
|
||||
"stage": "token",
|
||||
"message": "凭证可读取,但 token 响应中没有 access_token。",
|
||||
"settings_tab": "collector_credentials",
|
||||
**credential_context,
|
||||
}
|
||||
request_headers["Authorization"] = f"Bearer {token}"
|
||||
elif credential_context["credential_provider"] == "spacetrack":
|
||||
username, password, _source = _resolve_spacetrack_credentials()
|
||||
login_url = "https://www.space-track.org/ajaxauth/login"
|
||||
login_response = await client.post(
|
||||
login_url,
|
||||
data={
|
||||
"identity": username,
|
||||
"password": password,
|
||||
},
|
||||
)
|
||||
login_response.raise_for_status()
|
||||
|
||||
started = datetime.now(UTC)
|
||||
async with client.stream("GET", request_endpoint, headers=request_headers) as response:
|
||||
response.raise_for_status()
|
||||
status_code = response.status_code
|
||||
elapsed_ms = (datetime.now(UTC) - started).total_seconds() * 1000
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"checksum": checksum,
|
||||
"stage": "endpoint",
|
||||
"message": "连接验证成功。",
|
||||
"status_code": status_code,
|
||||
"response_time_ms": elapsed_ms,
|
||||
**credential_context,
|
||||
}
|
||||
except httpx.HTTPStatusError as exc:
|
||||
return {
|
||||
"success": False,
|
||||
"checksum": checksum,
|
||||
"stage": "endpoint",
|
||||
"message": f"连接验证失败:HTTP {exc.response.status_code}",
|
||||
"error": f"HTTP Error: {exc.response.status_code}",
|
||||
**credential_context,
|
||||
}
|
||||
except httpx.HTTPError as exc:
|
||||
return {
|
||||
"success": False,
|
||||
"checksum": checksum,
|
||||
"stage": "network",
|
||||
"message": f"连接验证失败:{exc.__class__.__name__}",
|
||||
"error": str(exc),
|
||||
**credential_context,
|
||||
}
|
||||
|
||||
|
||||
def make_success_validation(checksum: str, result: dict[str, Any]) -> dict[str, Any]:
|
||||
return {
|
||||
"checksum": checksum,
|
||||
"status": "success",
|
||||
"validated_at": datetime.now(UTC).isoformat(),
|
||||
"status_code": result.get("status_code"),
|
||||
"credential_source": result.get("credential_source"),
|
||||
}
|
||||
|
||||
|
||||
def is_builtin_validation_current(config: DataSourceConfig | None, checksum: str) -> bool:
|
||||
validation = get_connectivity_validation(config)
|
||||
return bool(
|
||||
validation
|
||||
and validation.get("status") == "success"
|
||||
and validation.get("checksum") == checksum
|
||||
)
|
||||
|
||||
|
||||
async def get_connectivity_store(db) -> dict[str, Any]:
|
||||
result = await db.execute(
|
||||
select(SystemSetting).where(SystemSetting.category == CONNECTIVITY_STORE_CATEGORY)
|
||||
)
|
||||
record = result.scalar_one_or_none()
|
||||
return dict(record.payload or {}) if record and isinstance(record.payload, dict) else {}
|
||||
|
||||
|
||||
async def save_connectivity_success(
|
||||
db,
|
||||
source: str,
|
||||
checksum: str,
|
||||
result: dict[str, Any],
|
||||
*,
|
||||
connected_by: str,
|
||||
) -> dict[str, Any]:
|
||||
store = await get_connectivity_store(db)
|
||||
validation = {
|
||||
**make_success_validation(checksum, result),
|
||||
"connected_by": connected_by,
|
||||
}
|
||||
store[source] = validation
|
||||
|
||||
existing = await db.execute(
|
||||
select(SystemSetting).where(SystemSetting.category == CONNECTIVITY_STORE_CATEGORY)
|
||||
)
|
||||
record = existing.scalar_one_or_none()
|
||||
if record is None:
|
||||
db.add(SystemSetting(category=CONNECTIVITY_STORE_CATEGORY, payload=store))
|
||||
else:
|
||||
record.payload = store
|
||||
return validation
|
||||
|
||||
|
||||
async def load_builtin_override_config(db, source: str) -> DataSourceConfig | None:
|
||||
result = await db.execute(
|
||||
select(DataSourceConfig)
|
||||
.where(DataSourceConfig.name == source)
|
||||
.where(DataSourceConfig.is_active.is_(True))
|
||||
)
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
|
||||
async def get_builtin_effective_candidate(db, source: str) -> dict[str, Any]:
|
||||
override = await load_builtin_override_config(db, source)
|
||||
default_endpoint = get_data_sources_config().get_yaml_url(source)
|
||||
return {
|
||||
"name": source,
|
||||
"endpoint": (override.endpoint if override and override.endpoint else default_endpoint) or "",
|
||||
"auth_type": override.auth_type if override else "none",
|
||||
"headers": override.headers if override else {},
|
||||
"config": strip_connectivity_validation(override.config if override else {}),
|
||||
}
|
||||
|
||||
|
||||
async def has_collected_data(db, source: str) -> bool:
|
||||
result = await db.execute(select(func.count(CollectedData.id)).where(CollectedData.source == source))
|
||||
if (result.scalar() or 0) > 0:
|
||||
return True
|
||||
|
||||
datasource_result = await db.execute(select(DataSource).where(DataSource.source == source))
|
||||
datasource = datasource_result.scalar_one_or_none()
|
||||
return bool(datasource and datasource.last_status == "success")
|
||||
|
||||
|
||||
async def get_builtin_connection_status(
|
||||
db,
|
||||
source: str,
|
||||
endpoint: str,
|
||||
auth_type: str,
|
||||
headers: dict | None,
|
||||
config: dict | None,
|
||||
) -> dict[str, Any]:
|
||||
checksum, credential_context = await build_builtin_connectivity_checksum(
|
||||
source,
|
||||
endpoint,
|
||||
auth_type,
|
||||
headers,
|
||||
config,
|
||||
db,
|
||||
)
|
||||
store = await get_connectivity_store(db)
|
||||
validation = store.get(source)
|
||||
if isinstance(validation, dict) and validation.get("status") == "success":
|
||||
if validation.get("checksum") == checksum:
|
||||
return {
|
||||
"connected": True,
|
||||
"checksum": checksum,
|
||||
"connected_by": validation.get("connected_by") or "connection_button",
|
||||
"message": "当前配置已完成连接验证。",
|
||||
**credential_context,
|
||||
}
|
||||
|
||||
effective = await get_builtin_effective_candidate(db, source)
|
||||
effective_checksum, _ = await build_builtin_connectivity_checksum(
|
||||
source,
|
||||
effective["endpoint"],
|
||||
effective["auth_type"],
|
||||
effective["headers"],
|
||||
effective["config"],
|
||||
db,
|
||||
)
|
||||
if checksum == effective_checksum and await has_collected_data(db, source):
|
||||
return {
|
||||
"connected": True,
|
||||
"checksum": checksum,
|
||||
"connected_by": "collection",
|
||||
"message": "当前配置已有成功采集数据,视为已连接。",
|
||||
**credential_context,
|
||||
}
|
||||
|
||||
if isinstance(validation, dict) and validation.get("status") == "success":
|
||||
return {
|
||||
"connected": False,
|
||||
"checksum": checksum,
|
||||
"connected_by": None,
|
||||
"message": "接口地址或凭证指纹已变化,请重新点击连接验证。",
|
||||
**credential_context,
|
||||
}
|
||||
|
||||
return {
|
||||
"connected": False,
|
||||
"checksum": checksum,
|
||||
"connected_by": None,
|
||||
"message": "当前配置尚未连接,请点击连接验证。",
|
||||
**credential_context,
|
||||
}
|
||||
@@ -1,358 +0,0 @@
|
||||
"""Deterministic mapping support for custom data sources."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.target_schema_registry import TargetSchema, get_target_schema
|
||||
|
||||
SECRET_KEY_PATTERN = re.compile(
|
||||
r"(token|secret|password|passwd|authorization|api[_-]?key|client[_-]?secret)",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
|
||||
class MappingError(ValueError):
|
||||
"""Raised when a mapping definition cannot be executed."""
|
||||
|
||||
|
||||
def stable_payload_hash(payload: Any) -> str:
|
||||
encoded = json.dumps(payload, ensure_ascii=False, sort_keys=True, default=str).encode()
|
||||
return hashlib.sha256(encoded).hexdigest()
|
||||
|
||||
|
||||
def redact_for_llm(value: Any) -> Any:
|
||||
if isinstance(value, dict):
|
||||
redacted = {}
|
||||
for key, item in value.items():
|
||||
if SECRET_KEY_PATTERN.search(str(key)):
|
||||
redacted[key] = "[REDACTED]"
|
||||
else:
|
||||
redacted[key] = redact_for_llm(item)
|
||||
return redacted
|
||||
if isinstance(value, list):
|
||||
return [redact_for_llm(item) for item in value[:20]]
|
||||
return value
|
||||
|
||||
|
||||
def extract_path(payload: Any, path: str | None) -> Any:
|
||||
if not path or path == "$":
|
||||
return payload
|
||||
|
||||
normalized = path.strip()
|
||||
if normalized.startswith("$."):
|
||||
normalized = normalized[2:]
|
||||
elif normalized.startswith("$"):
|
||||
normalized = normalized[1:]
|
||||
normalized = normalized.strip(".")
|
||||
if not normalized:
|
||||
return payload
|
||||
|
||||
current = payload
|
||||
for raw_segment in normalized.split("."):
|
||||
segment = raw_segment.strip()
|
||||
if not segment:
|
||||
continue
|
||||
|
||||
list_all = segment.endswith("[*]")
|
||||
if list_all:
|
||||
segment = segment[:-3]
|
||||
|
||||
index = None
|
||||
match = re.fullmatch(r"(.+)\[(\d+)\]", segment)
|
||||
if match:
|
||||
segment = match.group(1)
|
||||
index = int(match.group(2))
|
||||
|
||||
if segment:
|
||||
if isinstance(current, dict):
|
||||
current = current.get(segment)
|
||||
else:
|
||||
return None
|
||||
|
||||
if list_all:
|
||||
return current if isinstance(current, list) else []
|
||||
|
||||
if index is not None:
|
||||
if not isinstance(current, list) or index >= len(current):
|
||||
return None
|
||||
current = current[index]
|
||||
|
||||
return current
|
||||
|
||||
|
||||
def _convert_value(value: Any, target_type: str | None) -> Any:
|
||||
if value is None or target_type in (None, "", "any"):
|
||||
return value
|
||||
|
||||
if target_type == "string":
|
||||
return str(value)
|
||||
if target_type == "integer":
|
||||
return int(value)
|
||||
if target_type == "float":
|
||||
return float(value)
|
||||
if target_type == "boolean":
|
||||
if isinstance(value, bool):
|
||||
return value
|
||||
if isinstance(value, str):
|
||||
return value.strip().lower() in {"1", "true", "yes", "y", "on"}
|
||||
return bool(value)
|
||||
if target_type == "datetime":
|
||||
if isinstance(value, datetime):
|
||||
return value
|
||||
if isinstance(value, (int, float)):
|
||||
return datetime.fromtimestamp(value)
|
||||
if isinstance(value, str):
|
||||
return datetime.fromisoformat(value.replace("Z", "+00:00"))
|
||||
return value
|
||||
if target_type == "object":
|
||||
if isinstance(value, dict):
|
||||
return value
|
||||
raise ValueError("expected object")
|
||||
if target_type == "array":
|
||||
if isinstance(value, list):
|
||||
return value
|
||||
raise ValueError("expected array")
|
||||
|
||||
return value
|
||||
|
||||
|
||||
def _apply_enum(value: Any, enum_map: Any) -> Any:
|
||||
if not isinstance(enum_map, dict):
|
||||
return value
|
||||
key = str(value)
|
||||
return enum_map.get(key, enum_map.get(value, value))
|
||||
|
||||
|
||||
def _map_one(item: Any, field_mapping: dict[str, Any]) -> tuple[dict[str, Any], list[str]]:
|
||||
output: dict[str, Any] = {}
|
||||
errors: list[str] = []
|
||||
|
||||
for field_name, rule in field_mapping.items():
|
||||
if isinstance(rule, str):
|
||||
rule = {"path": rule}
|
||||
if not isinstance(rule, dict):
|
||||
errors.append(f"{field_name}: mapping rule must be an object or path string")
|
||||
continue
|
||||
|
||||
value = extract_path(item, rule.get("path"))
|
||||
if value is None and "default" in rule:
|
||||
value = rule.get("default")
|
||||
value = _apply_enum(value, rule.get("enum"))
|
||||
|
||||
try:
|
||||
value = _convert_value(value, rule.get("type"))
|
||||
except (TypeError, ValueError) as exc:
|
||||
errors.append(f"{field_name}: failed to convert value {value!r}: {exc}")
|
||||
continue
|
||||
|
||||
if value is not None or rule.get("include_null", False):
|
||||
output[field_name] = value
|
||||
|
||||
return output, errors
|
||||
|
||||
|
||||
def execute_mapping(
|
||||
payload: Any,
|
||||
mapping_json: dict[str, Any],
|
||||
target_schema: str | TargetSchema,
|
||||
*,
|
||||
limit: int | None = None,
|
||||
) -> dict[str, Any]:
|
||||
schema = get_target_schema(target_schema) if isinstance(target_schema, str) else target_schema
|
||||
source = mapping_json.get("source") or {}
|
||||
fields = mapping_json.get("fields")
|
||||
if not isinstance(fields, dict) or not fields:
|
||||
raise MappingError("mapping_json.fields must be a non-empty object")
|
||||
|
||||
items_path = source.get("items_path") or mapping_json.get("items_path") or "$"
|
||||
items = extract_path(payload, items_path)
|
||||
if isinstance(items, dict):
|
||||
items = [items]
|
||||
elif not isinstance(items, list):
|
||||
items = []
|
||||
|
||||
if limit is not None:
|
||||
items = items[:limit]
|
||||
|
||||
mapped_records: list[dict[str, Any]] = []
|
||||
errors: list[dict[str, Any]] = []
|
||||
for index, item in enumerate(items):
|
||||
mapped, mapping_errors = _map_one(item, fields)
|
||||
validated, validation_errors = schema.validate_record(mapped)
|
||||
all_errors = mapping_errors + validation_errors
|
||||
if all_errors:
|
||||
errors.append({"index": index, "errors": all_errors, "record": mapped})
|
||||
continue
|
||||
if validated is not None:
|
||||
mapped_records.append(validated)
|
||||
|
||||
return {
|
||||
"target_schema": schema.key,
|
||||
"total_items": len(items),
|
||||
"mapped_count": len(mapped_records),
|
||||
"failed_count": len(errors),
|
||||
"records": mapped_records,
|
||||
"errors": errors,
|
||||
}
|
||||
|
||||
|
||||
def build_heuristic_mapping(sample_payload: Any, target_schema_key: str) -> dict[str, Any]:
|
||||
schema = get_target_schema(target_schema_key)
|
||||
items_path = "$"
|
||||
sample_item = sample_payload
|
||||
if isinstance(sample_payload, dict):
|
||||
for key in ("data", "items", "results", "features", "vessels"):
|
||||
candidate = sample_payload.get(key)
|
||||
if isinstance(candidate, list) and candidate:
|
||||
items_path = f"$.{key}[*]"
|
||||
sample_item = candidate[0]
|
||||
break
|
||||
elif isinstance(sample_payload, list) and sample_payload:
|
||||
items_path = "$"
|
||||
sample_item = sample_payload[0]
|
||||
|
||||
available = _flatten_keys(sample_item if isinstance(sample_item, dict) else {})
|
||||
fields: dict[str, Any] = {}
|
||||
for field in schema.fields:
|
||||
candidate = _best_field_match(field.name, available)
|
||||
if candidate:
|
||||
fields[field.name] = {"path": f"$.{candidate}", "type": field.type}
|
||||
elif field.name == "data" and target_schema_key == "generic_records":
|
||||
fields[field.name] = {"path": "$", "type": "object"}
|
||||
elif not field.required:
|
||||
fields[field.name] = {"path": f"$.{field.name}", "type": field.type, "default": None}
|
||||
|
||||
return {
|
||||
"source": {"items_path": items_path},
|
||||
"fields": fields,
|
||||
"meta": {
|
||||
"generated_by": "heuristic",
|
||||
"requires_review": True,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def _flatten_keys(payload: dict[str, Any], prefix: str = "") -> list[str]:
|
||||
keys: list[str] = []
|
||||
for key, value in payload.items():
|
||||
dotted = f"{prefix}.{key}" if prefix else str(key)
|
||||
keys.append(dotted)
|
||||
if isinstance(value, dict):
|
||||
keys.extend(_flatten_keys(value, dotted))
|
||||
return keys
|
||||
|
||||
|
||||
def _best_field_match(field_name: str, candidates: list[str]) -> str | None:
|
||||
aliases = {
|
||||
"lat": ("lat", "latitude", "y"),
|
||||
"lon": ("lon", "lng", "longitude", "x"),
|
||||
"mmsi": ("mmsi",),
|
||||
"sog": ("sog", "speed", "speedOverGround"),
|
||||
"cog": ("cog", "course", "courseOverGround"),
|
||||
"received_at": ("received_at", "timestamp", "time", "updated_at"),
|
||||
"observed_at": ("observed_at", "timestamp", "time", "updated_at"),
|
||||
"source_id": ("id", "source_id", "uuid"),
|
||||
}.get(field_name, (field_name,))
|
||||
|
||||
lowered = {candidate.lower(): candidate for candidate in candidates}
|
||||
for alias in aliases:
|
||||
if alias.lower() in lowered:
|
||||
return lowered[alias.lower()]
|
||||
for candidate in candidates:
|
||||
tail = candidate.split(".")[-1].lower()
|
||||
if tail in {alias.lower() for alias in aliases}:
|
||||
return candidate
|
||||
return None
|
||||
|
||||
|
||||
def _parse_datetime(value: Any) -> datetime | None:
|
||||
if value is None:
|
||||
return None
|
||||
if isinstance(value, datetime):
|
||||
return value
|
||||
if isinstance(value, str):
|
||||
return datetime.fromisoformat(value.replace("Z", "+00:00"))
|
||||
return None
|
||||
|
||||
|
||||
async def persist_mapped_records(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
datasource_name: str,
|
||||
datasource_config_id: int,
|
||||
target_schema: str,
|
||||
records: list[dict[str, Any]],
|
||||
mapping_version: int,
|
||||
) -> int:
|
||||
"""Persist validated mapped records to the destination for a target schema."""
|
||||
if target_schema == "vessel_ais":
|
||||
from app.models.vessel import VesselPosition
|
||||
|
||||
for record in records:
|
||||
db.add(
|
||||
VesselPosition(
|
||||
mmsi=record["mmsi"],
|
||||
lat=record["lat"],
|
||||
lon=record["lon"],
|
||||
sog=record.get("sog"),
|
||||
cog=record.get("cog"),
|
||||
heading=record.get("heading"),
|
||||
received_at=_parse_datetime(record.get("received_at")) or datetime.now(UTC),
|
||||
)
|
||||
)
|
||||
await db.commit()
|
||||
return len(records)
|
||||
|
||||
from app.models.collected_data import CollectedData
|
||||
|
||||
collected_at = datetime.now(UTC)
|
||||
for index, record in enumerate(records):
|
||||
if target_schema == "geo_points":
|
||||
source_id = record.get("source_id") or f"{datasource_config_id}:{index}"
|
||||
name = record.get("name")
|
||||
metadata = {
|
||||
"latitude": record.get("lat"),
|
||||
"longitude": record.get("lon"),
|
||||
"type": record.get("type"),
|
||||
"mapping_version": mapping_version,
|
||||
"target_schema": target_schema,
|
||||
**(record.get("metadata") or {}),
|
||||
}
|
||||
reference_date = _parse_datetime(record.get("observed_at"))
|
||||
else:
|
||||
source_id = record.get("source_id") or f"{datasource_config_id}:{index}"
|
||||
name = None
|
||||
metadata = {
|
||||
"data": record.get("data") or {},
|
||||
"mapping_version": mapping_version,
|
||||
"target_schema": target_schema,
|
||||
}
|
||||
reference_date = _parse_datetime(record.get("observed_at"))
|
||||
|
||||
db.add(
|
||||
CollectedData(
|
||||
source=datasource_name,
|
||||
source_id=str(source_id),
|
||||
entity_key=f"{datasource_name}:{source_id}",
|
||||
data_type=target_schema,
|
||||
name=name,
|
||||
title=name,
|
||||
extra_data=metadata,
|
||||
collected_at=collected_at,
|
||||
reference_date=reference_date,
|
||||
is_valid=1,
|
||||
is_current=True,
|
||||
change_type="created",
|
||||
change_summary={},
|
||||
)
|
||||
)
|
||||
|
||||
await db.commit()
|
||||
return len(records)
|
||||
@@ -1,540 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
from email.utils import parsedate_to_datetime
|
||||
import hashlib
|
||||
import html
|
||||
import re
|
||||
from typing import Any
|
||||
from urllib.parse import quote
|
||||
import xml.etree.ElementTree as ET
|
||||
|
||||
import httpx
|
||||
from bs4 import BeautifulSoup
|
||||
|
||||
|
||||
USER_AGENT = "PlanetEarthNewsBoard/1.0 (+https://planet.local)"
|
||||
REQUEST_TIMEOUT = 12.0
|
||||
MAX_ITEMS_PER_SOURCE = 6
|
||||
MAX_ITEMS_TOTAL = 12
|
||||
STALE_CACHE_MAX_AGE_SECONDS = 60 * 45
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RegionProfile:
|
||||
key: str
|
||||
label: str
|
||||
query: str
|
||||
accent: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RegionAnchor:
|
||||
region: str
|
||||
label: str
|
||||
latitude: float
|
||||
longitude: float
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class NewsFeedSource:
|
||||
id: str
|
||||
name: str
|
||||
region: str
|
||||
feed_url: str
|
||||
homepage_url: str
|
||||
source_type: str = "rss"
|
||||
priority: int = 100
|
||||
|
||||
|
||||
@dataclass
|
||||
class ParsedNewsItem:
|
||||
id: str
|
||||
title: str
|
||||
summary: str
|
||||
url: str
|
||||
source: str
|
||||
feed_name: str
|
||||
feed_region: str
|
||||
homepage_url: str
|
||||
published_at: datetime | None
|
||||
|
||||
|
||||
@dataclass
|
||||
class CachedRegionFeed:
|
||||
region: str
|
||||
fetched_at: datetime
|
||||
items: list[ParsedNewsItem]
|
||||
sources: list[NewsFeedSource]
|
||||
|
||||
|
||||
REGION_PROFILES: dict[str, RegionProfile] = {
|
||||
"americas": RegionProfile(
|
||||
key="americas",
|
||||
label="美洲焦点",
|
||||
query='Americas geopolitics OR Latin America OR "United States" OR Canada',
|
||||
accent="#79d3ff",
|
||||
),
|
||||
"europe": RegionProfile(
|
||||
key="europe",
|
||||
label="欧洲焦点",
|
||||
query='Europe geopolitics OR EU OR NATO OR "Eastern Europe"',
|
||||
accent="#8fd4ff",
|
||||
),
|
||||
"middle-east-africa": RegionProfile(
|
||||
key="middle-east-africa",
|
||||
label="中东与非洲焦点",
|
||||
query='"Middle East" OR Africa geopolitics OR Red Sea OR Gulf',
|
||||
accent="#ffb56a",
|
||||
),
|
||||
"asia-pacific": RegionProfile(
|
||||
key="asia-pacific",
|
||||
label="亚太焦点",
|
||||
query='"Asia Pacific" OR Indo-Pacific OR China OR Japan OR Korea OR ASEAN',
|
||||
accent="#78f2cf",
|
||||
),
|
||||
"global": RegionProfile(
|
||||
key="global",
|
||||
label="全球焦点",
|
||||
query='"world news" OR geopolitics OR "global affairs"',
|
||||
accent="#d6e6ff",
|
||||
),
|
||||
}
|
||||
|
||||
REGION_ANCHORS: dict[str, RegionAnchor] = {
|
||||
"americas": RegionAnchor(
|
||||
region="americas",
|
||||
label="美洲",
|
||||
latitude=37.0902,
|
||||
longitude=-95.7129,
|
||||
),
|
||||
"europe": RegionAnchor(
|
||||
region="europe",
|
||||
label="欧洲",
|
||||
latitude=50.1109,
|
||||
longitude=8.6821,
|
||||
),
|
||||
"middle-east-africa": RegionAnchor(
|
||||
region="middle-east-africa",
|
||||
label="中东与非洲",
|
||||
latitude=25.2048,
|
||||
longitude=55.2708,
|
||||
),
|
||||
"asia-pacific": RegionAnchor(
|
||||
region="asia-pacific",
|
||||
label="亚太",
|
||||
latitude=1.3521,
|
||||
longitude=103.8198,
|
||||
),
|
||||
"global": RegionAnchor(
|
||||
region="global",
|
||||
label="全球",
|
||||
latitude=20.0,
|
||||
longitude=0.0,
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def _google_news_feed(query: str, *, hl: str, gl: str, ceid: str) -> str:
|
||||
return (
|
||||
"https://news.google.com/rss/search?q="
|
||||
+ quote(query, safe="")
|
||||
+ f"&hl={hl}&gl={gl}&ceid={ceid}"
|
||||
)
|
||||
|
||||
|
||||
NEWS_FEED_SOURCES: tuple[NewsFeedSource, ...] = (
|
||||
NewsFeedSource(
|
||||
id="bbc-world",
|
||||
name="BBC World",
|
||||
region="global",
|
||||
feed_url="https://feeds.bbci.co.uk/news/world/rss.xml",
|
||||
homepage_url="https://www.bbc.com/news/world",
|
||||
priority=10,
|
||||
),
|
||||
NewsFeedSource(
|
||||
id="dw-top",
|
||||
name="DW Top Stories",
|
||||
region="europe",
|
||||
feed_url="https://rss.dw.com/rdf/rss-en-top",
|
||||
homepage_url="https://www.dw.com/en/top-stories/s-9097",
|
||||
priority=20,
|
||||
),
|
||||
NewsFeedSource(
|
||||
id="global-scan",
|
||||
name="Global Monitor / World",
|
||||
region="global",
|
||||
feed_url=_google_news_feed(
|
||||
REGION_PROFILES["global"].query,
|
||||
hl="en-US",
|
||||
gl="US",
|
||||
ceid="US:en",
|
||||
),
|
||||
homepage_url="https://news.google.com/",
|
||||
source_type="aggregated",
|
||||
priority=30,
|
||||
),
|
||||
NewsFeedSource(
|
||||
id="google-americas",
|
||||
name="Global Monitor / Americas",
|
||||
region="americas",
|
||||
feed_url=_google_news_feed(
|
||||
REGION_PROFILES["americas"].query,
|
||||
hl="en-US",
|
||||
gl="US",
|
||||
ceid="US:en",
|
||||
),
|
||||
homepage_url="https://news.google.com/",
|
||||
source_type="aggregated",
|
||||
priority=40,
|
||||
),
|
||||
NewsFeedSource(
|
||||
id="google-europe",
|
||||
name="Global Monitor / Europe",
|
||||
region="europe",
|
||||
feed_url=_google_news_feed(
|
||||
REGION_PROFILES["europe"].query,
|
||||
hl="en-GB",
|
||||
gl="GB",
|
||||
ceid="GB:en",
|
||||
),
|
||||
homepage_url="https://news.google.com/",
|
||||
source_type="aggregated",
|
||||
priority=40,
|
||||
),
|
||||
NewsFeedSource(
|
||||
id="google-mea",
|
||||
name="Global Monitor / MEA",
|
||||
region="middle-east-africa",
|
||||
feed_url=_google_news_feed(
|
||||
REGION_PROFILES["middle-east-africa"].query,
|
||||
hl="en-US",
|
||||
gl="US",
|
||||
ceid="US:en",
|
||||
),
|
||||
homepage_url="https://news.google.com/",
|
||||
source_type="aggregated",
|
||||
priority=40,
|
||||
),
|
||||
NewsFeedSource(
|
||||
id="google-apac",
|
||||
name="Global Monitor / APAC",
|
||||
region="asia-pacific",
|
||||
feed_url=_google_news_feed(
|
||||
REGION_PROFILES["asia-pacific"].query,
|
||||
hl="en-SG",
|
||||
gl="SG",
|
||||
ceid="SG:en",
|
||||
),
|
||||
homepage_url="https://news.google.com/",
|
||||
source_type="aggregated",
|
||||
priority=40,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
_REGION_CACHE: dict[str, CachedRegionFeed] = {}
|
||||
|
||||
|
||||
def determine_focus_region(lat: float | None, lon: float | None) -> str:
|
||||
if lat is None or lon is None:
|
||||
return "global"
|
||||
if -170 <= lon <= -30:
|
||||
return "americas"
|
||||
if -30 < lon <= 45:
|
||||
return "europe" if lat >= 30 else "middle-east-africa"
|
||||
if 45 < lon <= 150:
|
||||
return "middle-east-africa" if lat < 10 else "asia-pacific"
|
||||
return "asia-pacific"
|
||||
|
||||
|
||||
def get_region_profile(region: str) -> RegionProfile:
|
||||
return REGION_PROFILES.get(region, REGION_PROFILES["global"])
|
||||
|
||||
|
||||
def get_region_anchor(region: str) -> RegionAnchor:
|
||||
return REGION_ANCHORS.get(region, REGION_ANCHORS["global"])
|
||||
|
||||
|
||||
def get_sources_for_region(region: str) -> list[NewsFeedSource]:
|
||||
return sorted(
|
||||
[source for source in NEWS_FEED_SOURCES if source.region in {"global", region}],
|
||||
key=lambda source: (source.priority, source.name),
|
||||
)
|
||||
|
||||
|
||||
def _strip_html(value: str) -> str:
|
||||
if not value:
|
||||
return ""
|
||||
soup = BeautifulSoup(value, "html.parser")
|
||||
return re.sub(r"\s+", " ", soup.get_text(" ", strip=True)).strip()
|
||||
|
||||
|
||||
def _truncate(value: str, limit: int = 180) -> str:
|
||||
text = value.strip()
|
||||
if len(text) <= limit:
|
||||
return text
|
||||
return text[: limit - 1].rstrip() + "…"
|
||||
|
||||
|
||||
def _normalize_source_name(raw: str, fallback: str) -> str:
|
||||
text = html.unescape((raw or "").strip())
|
||||
if " - " in text:
|
||||
return text.split(" - ")[-1].strip() or fallback
|
||||
return text or fallback
|
||||
|
||||
|
||||
def _parse_datetime(raw: str | None) -> datetime | None:
|
||||
if not raw:
|
||||
return None
|
||||
text = raw.strip()
|
||||
if not text:
|
||||
return None
|
||||
|
||||
for parser in (
|
||||
lambda value: parsedate_to_datetime(value),
|
||||
lambda value: datetime.fromisoformat(value.replace("Z", "+00:00")),
|
||||
):
|
||||
try:
|
||||
parsed = parser(text)
|
||||
if parsed.tzinfo is None:
|
||||
parsed = parsed.replace(tzinfo=UTC)
|
||||
return parsed.astimezone(UTC)
|
||||
except Exception:
|
||||
continue
|
||||
return None
|
||||
|
||||
|
||||
def _extract_item_text(element: ET.Element, *names: str) -> str:
|
||||
for name in names:
|
||||
node = element.find(name)
|
||||
if node is not None and node.text:
|
||||
return node.text.strip()
|
||||
return ""
|
||||
|
||||
|
||||
def _parse_feed_entries(xml_text: str, source: NewsFeedSource) -> list[ParsedNewsItem]:
|
||||
root = ET.fromstring(xml_text)
|
||||
items: list[ParsedNewsItem] = []
|
||||
|
||||
rss_items = root.findall("./channel/item")
|
||||
atom_entries = root.findall("{http://www.w3.org/2005/Atom}entry")
|
||||
nodes = rss_items or atom_entries
|
||||
|
||||
for node in nodes[:MAX_ITEMS_PER_SOURCE]:
|
||||
if node.tag.endswith("entry"):
|
||||
title = _extract_item_text(node, "{http://www.w3.org/2005/Atom}title")
|
||||
summary = _extract_item_text(
|
||||
node,
|
||||
"{http://www.w3.org/2005/Atom}summary",
|
||||
"{http://www.w3.org/2005/Atom}content",
|
||||
)
|
||||
link_node = node.find("{http://www.w3.org/2005/Atom}link")
|
||||
link = link_node.get("href", "").strip() if link_node is not None else ""
|
||||
published = _extract_item_text(
|
||||
node,
|
||||
"{http://www.w3.org/2005/Atom}updated",
|
||||
"{http://www.w3.org/2005/Atom}published",
|
||||
)
|
||||
else:
|
||||
title = _extract_item_text(node, "title")
|
||||
summary = _extract_item_text(node, "description", "content")
|
||||
link = _extract_item_text(node, "link")
|
||||
published = _extract_item_text(node, "pubDate", "published", "updated")
|
||||
|
||||
clean_title = html.unescape(title).strip()
|
||||
clean_summary = _truncate(_strip_html(summary), 180)
|
||||
if not clean_title or not link:
|
||||
continue
|
||||
|
||||
item_source = _normalize_source_name(clean_title, source.name)
|
||||
display_title = clean_title
|
||||
if source.source_type == "aggregated" and " - " in clean_title:
|
||||
parts = clean_title.rsplit(" - ", 1)
|
||||
display_title = parts[0].strip()
|
||||
item_source = _normalize_source_name(parts[1], source.name)
|
||||
|
||||
items.append(
|
||||
ParsedNewsItem(
|
||||
id=f"{source.id}:{hashlib.sha1(link.encode('utf-8')).hexdigest()[:12]}",
|
||||
title=display_title,
|
||||
summary=clean_summary,
|
||||
url=link,
|
||||
source=item_source,
|
||||
feed_name=source.name,
|
||||
feed_region=source.region,
|
||||
homepage_url=source.homepage_url,
|
||||
published_at=_parse_datetime(published),
|
||||
)
|
||||
)
|
||||
|
||||
return items
|
||||
|
||||
|
||||
def _serialize_sources(sources: list[NewsFeedSource]) -> list[dict[str, Any]]:
|
||||
return [
|
||||
{
|
||||
"id": source.id,
|
||||
"name": source.name,
|
||||
"region": source.region,
|
||||
"homepage_url": source.homepage_url,
|
||||
}
|
||||
for source in sources
|
||||
]
|
||||
|
||||
|
||||
def _serialize_item(item: ParsedNewsItem, *, active_region: str) -> dict[str, Any]:
|
||||
published_at = item.published_at
|
||||
anchor = get_region_anchor(item.feed_region)
|
||||
return {
|
||||
"id": item.id,
|
||||
"title": item.title,
|
||||
"summary": item.summary,
|
||||
"url": item.url,
|
||||
"source": item.source,
|
||||
"feed_name": item.feed_name,
|
||||
"region": item.feed_region,
|
||||
"homepage_url": item.homepage_url,
|
||||
"published_at": published_at.isoformat().replace("+00:00", "Z") if published_at else None,
|
||||
"latitude": anchor.latitude,
|
||||
"longitude": anchor.longitude,
|
||||
"location_label": anchor.label,
|
||||
"location_inferred": True,
|
||||
"is_focus_match": item.feed_region == active_region,
|
||||
}
|
||||
|
||||
|
||||
def _build_payload(
|
||||
*,
|
||||
lat: float | None,
|
||||
lon: float | None,
|
||||
active_region: str,
|
||||
items: list[ParsedNewsItem],
|
||||
sources: list[NewsFeedSource],
|
||||
errors: list[str],
|
||||
stale: bool,
|
||||
generated_at: datetime | None = None,
|
||||
) -> dict[str, Any]:
|
||||
profile = get_region_profile(active_region)
|
||||
timestamp = generated_at or datetime.now(UTC)
|
||||
return {
|
||||
"generated_at": timestamp.isoformat().replace("+00:00", "Z"),
|
||||
"focus": {
|
||||
"lat": lat,
|
||||
"lon": lon,
|
||||
"region": active_region,
|
||||
"label": profile.label,
|
||||
"accent": profile.accent,
|
||||
},
|
||||
"sources": _serialize_sources(sources),
|
||||
"items": [_serialize_item(item, active_region=active_region) for item in items],
|
||||
"errors": errors,
|
||||
"stale": stale,
|
||||
}
|
||||
|
||||
|
||||
def _rank_and_trim_items(items: list[ParsedNewsItem], *, active_region: str) -> list[ParsedNewsItem]:
|
||||
deduped: dict[str, ParsedNewsItem] = {}
|
||||
for item in items:
|
||||
key = item.url.strip() or item.title.strip().lower()
|
||||
if key not in deduped:
|
||||
deduped[key] = item
|
||||
|
||||
return sorted(
|
||||
deduped.values(),
|
||||
key=lambda item: (
|
||||
item.feed_region != active_region,
|
||||
item.published_at is None,
|
||||
-(item.published_at.timestamp() if item.published_at else 0),
|
||||
item.feed_name,
|
||||
),
|
||||
)[:MAX_ITEMS_TOTAL]
|
||||
|
||||
|
||||
def _get_cached_region_feed(region: str) -> CachedRegionFeed | None:
|
||||
cached = _REGION_CACHE.get(region)
|
||||
if not cached:
|
||||
return None
|
||||
age_seconds = (datetime.now(UTC) - cached.fetched_at).total_seconds()
|
||||
if age_seconds > STALE_CACHE_MAX_AGE_SECONDS:
|
||||
return None
|
||||
return cached
|
||||
|
||||
|
||||
def _store_region_cache(region: str, *, items: list[ParsedNewsItem], sources: list[NewsFeedSource]) -> None:
|
||||
_REGION_CACHE[region] = CachedRegionFeed(
|
||||
region=region,
|
||||
fetched_at=datetime.now(UTC),
|
||||
items=list(items),
|
||||
sources=list(sources),
|
||||
)
|
||||
|
||||
|
||||
async def _fetch_source(
|
||||
client: httpx.AsyncClient,
|
||||
source: NewsFeedSource,
|
||||
) -> tuple[NewsFeedSource, list[ParsedNewsItem], str | None]:
|
||||
try:
|
||||
response = await client.get(source.feed_url)
|
||||
response.raise_for_status()
|
||||
return source, _parse_feed_entries(response.text, source), None
|
||||
except Exception as exc:
|
||||
return source, [], str(exc)
|
||||
|
||||
|
||||
async def get_earth_news_payload(lat: float | None = None, lon: float | None = None) -> dict[str, Any]:
|
||||
active_region = determine_focus_region(lat, lon)
|
||||
sources = get_sources_for_region(active_region)
|
||||
errors: list[str] = []
|
||||
|
||||
async with httpx.AsyncClient(
|
||||
timeout=REQUEST_TIMEOUT,
|
||||
follow_redirects=True,
|
||||
headers={"User-Agent": USER_AGENT},
|
||||
) as client:
|
||||
results = await asyncio.gather(*(_fetch_source(client, source) for source in sources))
|
||||
|
||||
fetched_items: list[ParsedNewsItem] = []
|
||||
for source, items, error in results:
|
||||
if error:
|
||||
errors.append(f"{source.name}: {error}")
|
||||
continue
|
||||
fetched_items.extend(items)
|
||||
|
||||
ranked_items = _rank_and_trim_items(fetched_items, active_region=active_region)
|
||||
if ranked_items:
|
||||
_store_region_cache(active_region, items=ranked_items, sources=sources)
|
||||
return _build_payload(
|
||||
lat=lat,
|
||||
lon=lon,
|
||||
active_region=active_region,
|
||||
items=ranked_items,
|
||||
sources=sources,
|
||||
errors=errors,
|
||||
stale=False,
|
||||
)
|
||||
|
||||
cached = _get_cached_region_feed(active_region)
|
||||
if cached:
|
||||
return _build_payload(
|
||||
lat=lat,
|
||||
lon=lon,
|
||||
active_region=active_region,
|
||||
items=cached.items,
|
||||
sources=cached.sources,
|
||||
errors=errors,
|
||||
stale=True,
|
||||
generated_at=cached.fetched_at,
|
||||
)
|
||||
|
||||
return _build_payload(
|
||||
lat=lat,
|
||||
lon=lon,
|
||||
active_region=active_region,
|
||||
items=[],
|
||||
sources=sources,
|
||||
errors=errors,
|
||||
stale=False,
|
||||
)
|
||||
@@ -1,150 +0,0 @@
|
||||
"""LLM provider presets used by Settings and the runtime AI provider bridge."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
|
||||
MODELS_DEV_URL = "https://models.dev/api.json"
|
||||
|
||||
|
||||
FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
|
||||
"minimax": {
|
||||
"provider": "minimax",
|
||||
"label": "MiniMax",
|
||||
"provider_api": "anthropic-messages",
|
||||
"base_url": "https://api.minimaxi.com/anthropic",
|
||||
"model": "MiniMax-M2.7",
|
||||
"models": ["MiniMax-M2.7", "MiniMax-M2.7-highspeed", "MiniMax-M2.5", "MiniMax-M2"],
|
||||
"api_key_env": "MINIMAX_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
"openai": {
|
||||
"provider": "openai",
|
||||
"label": "OpenAI",
|
||||
"provider_api": "openai-completions",
|
||||
"base_url": "https://api.openai.com/v1",
|
||||
"model": "gpt-5.1",
|
||||
"models": ["gpt-5.1", "gpt-5.1-codex", "gpt-4.1", "gpt-4o"],
|
||||
"api_key_env": "OPENAI_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
"anthropic": {
|
||||
"provider": "anthropic",
|
||||
"label": "Anthropic",
|
||||
"provider_api": "anthropic-messages",
|
||||
"base_url": "https://api.anthropic.com/v1",
|
||||
"model": "claude-sonnet-4-6",
|
||||
"models": ["claude-sonnet-4-6", "claude-opus-4-5", "claude-3-5-haiku-20241022"],
|
||||
"api_key_env": "ANTHROPIC_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
"deepseek": {
|
||||
"provider": "deepseek",
|
||||
"label": "DeepSeek",
|
||||
"provider_api": "openai-completions",
|
||||
"base_url": "https://api.deepseek.com/v1",
|
||||
"model": "deepseek-chat",
|
||||
"models": ["deepseek-chat", "deepseek-reasoner"],
|
||||
"api_key_env": "DEEPSEEK_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
"alibaba": {
|
||||
"provider": "alibaba",
|
||||
"label": "Alibaba Qwen / DashScope",
|
||||
"provider_api": "openai-completions",
|
||||
"base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1",
|
||||
"model": "qwen3-max",
|
||||
"models": ["qwen3-max", "qwen3.5-plus", "qwen-max", "qwen-plus"],
|
||||
"api_key_env": "DASHSCOPE_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
"moonshotai": {
|
||||
"provider": "moonshotai",
|
||||
"label": "Moonshot AI / Kimi",
|
||||
"provider_api": "openai-completions",
|
||||
"base_url": "https://api.moonshot.ai/v1",
|
||||
"model": "kimi-k2.5",
|
||||
"models": ["kimi-k2.5", "kimi-k2-thinking", "kimi-k2-turbo-preview"],
|
||||
"api_key_env": "MOONSHOT_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
"openrouter": {
|
||||
"provider": "openrouter",
|
||||
"label": "OpenRouter",
|
||||
"provider_api": "openai-completions",
|
||||
"base_url": "https://openrouter.ai/api/v1",
|
||||
"model": "openai/gpt-5.1",
|
||||
"models": ["openai/gpt-5.1", "anthropic/claude-sonnet-4.5", "qwen/qwen3-max"],
|
||||
"api_key_env": "OPENROUTER_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
"ollama": {
|
||||
"provider": "ollama",
|
||||
"label": "Ollama Local",
|
||||
"provider_api": "ollama-generate",
|
||||
"base_url": "http://127.0.0.1:11434",
|
||||
"model": "qwen2.5:7b",
|
||||
"models": ["qwen2.5:7b", "llama3.1:8b", "mistral:7b"],
|
||||
"api_key_env": "",
|
||||
"source": "fallback",
|
||||
},
|
||||
}
|
||||
|
||||
MODELS_DEV_PROVIDER_KEYS = {
|
||||
"minimax": "minimax",
|
||||
"openai": "openai",
|
||||
"anthropic": "anthropic",
|
||||
"deepseek": "deepseek",
|
||||
"alibaba": "alibaba",
|
||||
"moonshotai": "moonshotai",
|
||||
"openrouter": "openrouter",
|
||||
}
|
||||
|
||||
|
||||
def list_fallback_llm_provider_presets() -> list[dict[str, Any]]:
|
||||
return [dict(value) for value in FALLBACK_LLM_PROVIDER_PRESETS.values()]
|
||||
|
||||
|
||||
def get_fallback_llm_provider_preset(provider: str) -> dict[str, Any]:
|
||||
key = provider.strip().lower()
|
||||
if key not in FALLBACK_LLM_PROVIDER_PRESETS:
|
||||
raise ValueError(f"Unsupported LLM provider preset: {provider}")
|
||||
return dict(FALLBACK_LLM_PROVIDER_PRESETS[key])
|
||||
|
||||
|
||||
async def refresh_llm_provider_preset(provider: str) -> dict[str, Any]:
|
||||
fallback = get_fallback_llm_provider_preset(provider)
|
||||
models_dev_key = MODELS_DEV_PROVIDER_KEYS.get(fallback["provider"])
|
||||
if not models_dev_key:
|
||||
return fallback
|
||||
|
||||
async with httpx.AsyncClient(timeout=15.0, follow_redirects=True) as client:
|
||||
response = await client.get(
|
||||
MODELS_DEV_URL,
|
||||
headers={"User-Agent": "Planet/1.0"},
|
||||
)
|
||||
response.raise_for_status()
|
||||
catalog = response.json()
|
||||
|
||||
upstream = catalog.get(models_dev_key)
|
||||
if not isinstance(upstream, dict):
|
||||
return fallback
|
||||
|
||||
upstream_models = upstream.get("models") if isinstance(upstream.get("models"), dict) else {}
|
||||
model_ids = list(upstream_models.keys())[:80]
|
||||
base_url = upstream.get("api") or fallback["base_url"]
|
||||
if fallback["provider"] == "deepseek" and base_url == "https://api.deepseek.com":
|
||||
base_url = "https://api.deepseek.com/v1"
|
||||
|
||||
refreshed = {
|
||||
**fallback,
|
||||
"label": upstream.get("name") or fallback["label"],
|
||||
"base_url": base_url,
|
||||
"model": model_ids[0] if model_ids else fallback["model"],
|
||||
"models": model_ids or fallback["models"],
|
||||
"api_key_env": (upstream.get("env") or [fallback["api_key_env"]])[0],
|
||||
"source": MODELS_DEV_URL,
|
||||
}
|
||||
return refreshed
|
||||
@@ -1,86 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from app.core.logging import get_logger, sanitize_log_value
|
||||
from app.core.request_context import get_request_id
|
||||
from app.db.session import async_session_factory
|
||||
from app.models.system_log import AuditLog, SystemLog
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
async def record_system_log(
|
||||
*,
|
||||
source: str,
|
||||
level: str,
|
||||
message: str,
|
||||
service: str | None = None,
|
||||
module: str | None = None,
|
||||
event: str | None = None,
|
||||
request_id: str | None = None,
|
||||
trace_id: str | None = None,
|
||||
user_id: int | None = None,
|
||||
category: str | None = None,
|
||||
context: dict[str, Any] | None = None,
|
||||
) -> None:
|
||||
try:
|
||||
async with async_session_factory() as session:
|
||||
session.add(
|
||||
SystemLog(
|
||||
source=source,
|
||||
service=service,
|
||||
module=module,
|
||||
event=event,
|
||||
level=level.lower(),
|
||||
message=str(sanitize_log_value(message)),
|
||||
request_id=request_id or get_request_id(),
|
||||
trace_id=trace_id,
|
||||
user_id=user_id,
|
||||
category=category,
|
||||
context=sanitize_log_value(context or {}),
|
||||
)
|
||||
)
|
||||
await session.commit()
|
||||
except Exception:
|
||||
logger.exception_event(
|
||||
"Failed to persist system log",
|
||||
event="system_log.persist.failed",
|
||||
context={"event_name": event, "source": source},
|
||||
)
|
||||
|
||||
|
||||
async def record_audit_log(
|
||||
*,
|
||||
action: str,
|
||||
actor_id: int | None = None,
|
||||
actor_name: str | None = None,
|
||||
target_type: str | None = None,
|
||||
target_id: str | None = None,
|
||||
result: str | None = None,
|
||||
request_id: str | None = None,
|
||||
ip: str | None = None,
|
||||
details: dict[str, Any] | None = None,
|
||||
) -> None:
|
||||
try:
|
||||
async with async_session_factory() as session:
|
||||
session.add(
|
||||
AuditLog(
|
||||
actor_id=actor_id,
|
||||
actor_name=actor_name,
|
||||
action=action,
|
||||
target_type=target_type,
|
||||
target_id=target_id,
|
||||
result=result,
|
||||
request_id=request_id or get_request_id(),
|
||||
ip=ip,
|
||||
details=sanitize_log_value(details or {}),
|
||||
)
|
||||
)
|
||||
await session.commit()
|
||||
except Exception:
|
||||
logger.exception_event(
|
||||
"Failed to persist audit log",
|
||||
event="audit_log.persist.failed",
|
||||
context={"action": action},
|
||||
)
|
||||
@@ -1,6 +1,7 @@
|
||||
"""Task Scheduler for running collection jobs."""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
@@ -8,19 +9,13 @@ from apscheduler.schedulers.asyncio import AsyncIOScheduler
|
||||
from apscheduler.triggers.interval import IntervalTrigger
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.core.logging import get_logger
|
||||
from app.db.session import async_session_factory
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.models.datasource import DataSource
|
||||
from app.models.task import CollectionTask
|
||||
from app.services.collectors.registry import collector_registry
|
||||
from app.services.datasource_connectivity import (
|
||||
build_builtin_connectivity_checksum,
|
||||
get_builtin_effective_candidate,
|
||||
save_connectivity_success,
|
||||
)
|
||||
|
||||
logger = get_logger(__name__)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
scheduler = AsyncIOScheduler()
|
||||
RUNNING_TASK_GUARD_TIMEOUT_MINUTES = 90
|
||||
@@ -59,11 +54,7 @@ async def _update_next_run_at(datasource: DataSource, session) -> None:
|
||||
async def _apply_datasource_schedule(datasource: DataSource, session) -> None:
|
||||
collector = collector_registry.get(datasource.source)
|
||||
if not collector:
|
||||
logger.warning_event(
|
||||
"Collector not found for datasource",
|
||||
event="collector.schedule.collector_missing",
|
||||
context={"collector_name": datasource.source},
|
||||
)
|
||||
logger.warning("Collector not found for datasource %s", datasource.source)
|
||||
return
|
||||
|
||||
collector_registry.set_active(datasource.source, datasource.is_active)
|
||||
@@ -81,17 +72,13 @@ async def _apply_datasource_schedule(datasource: DataSource, session) -> None:
|
||||
replace_existing=True,
|
||||
kwargs={"collector_name": datasource.source},
|
||||
)
|
||||
logger.info_event(
|
||||
"Scheduled collector",
|
||||
event="collector.schedule.updated",
|
||||
context={"collector_name": datasource.source, "frequency_minutes": datasource.frequency_minutes},
|
||||
logger.info(
|
||||
"Scheduled collector: %s (every %sm)",
|
||||
datasource.source,
|
||||
datasource.frequency_minutes,
|
||||
)
|
||||
else:
|
||||
logger.info_event(
|
||||
"Collector disabled",
|
||||
event="collector.schedule.disabled",
|
||||
context={"collector_name": datasource.source},
|
||||
)
|
||||
logger.info("Collector disabled: %s", datasource.source)
|
||||
|
||||
await _update_next_run_at(datasource, session)
|
||||
|
||||
@@ -100,30 +87,18 @@ async def run_collector_task(collector_name: str):
|
||||
"""Run a single collector task."""
|
||||
collector = collector_registry.get(collector_name)
|
||||
if not collector:
|
||||
logger.error_event(
|
||||
"Collector not found",
|
||||
event="collector.run.collector_missing",
|
||||
context={"collector_name": collector_name},
|
||||
)
|
||||
logger.error("Collector not found: %s", collector_name)
|
||||
return
|
||||
|
||||
async with async_session_factory() as db:
|
||||
result = await db.execute(select(DataSource).where(DataSource.source == collector_name))
|
||||
datasource = result.scalar_one_or_none()
|
||||
if not datasource:
|
||||
logger.error_event(
|
||||
"Datasource not found for collector",
|
||||
event="collector.run.datasource_missing",
|
||||
context={"collector_name": collector_name},
|
||||
)
|
||||
logger.error("Datasource not found for collector: %s", collector_name)
|
||||
return
|
||||
|
||||
if not datasource.is_active:
|
||||
logger.info_event(
|
||||
"Skipping disabled collector",
|
||||
event="collector.run.skipped_disabled",
|
||||
context={"collector_name": collector_name},
|
||||
)
|
||||
logger.info("Skipping disabled collector: %s", collector_name)
|
||||
return
|
||||
|
||||
running_result = await db.execute(
|
||||
@@ -147,10 +122,10 @@ async def run_collector_task(collector_name: str):
|
||||
and (now - started_at) > timedelta(minutes=RUNNING_TASK_GUARD_TIMEOUT_MINUTES)
|
||||
)
|
||||
if not is_stale:
|
||||
logger.warning_event(
|
||||
"Skipping collector trigger because task is already running",
|
||||
event="collector.run.skipped_already_running",
|
||||
context={"collector_name": collector_name, "task_id": existing_running.id},
|
||||
logger.warning(
|
||||
"Skipping collector %s trigger because task %s is already running",
|
||||
collector_name,
|
||||
existing_running.id,
|
||||
)
|
||||
return
|
||||
|
||||
@@ -168,64 +143,31 @@ async def run_collector_task(collector_name: str):
|
||||
else stale_reason
|
||||
)
|
||||
await db.commit()
|
||||
logger.warning_event(
|
||||
"Marked stale running task as failed before rerun",
|
||||
event="collector.run.stale_task_failed",
|
||||
context={"collector_name": collector_name, "task_id": existing_running.id},
|
||||
logger.warning(
|
||||
"Marked stale running task %s as failed before rerun of %s",
|
||||
existing_running.id,
|
||||
collector_name,
|
||||
)
|
||||
|
||||
try:
|
||||
collector._datasource_id = datasource.id
|
||||
logger.info_event(
|
||||
"Running collector",
|
||||
event="collector.run.started",
|
||||
context={"collector_name": collector_name, "datasource_id": datasource.id},
|
||||
)
|
||||
logger.info("Running collector: %s (datasource_id=%s)", collector_name, datasource.id)
|
||||
task_result = await collector.run(db)
|
||||
datasource.last_run_at = datetime.now(UTC)
|
||||
datasource.last_status = task_result.get("status")
|
||||
if datasource.last_status == "success":
|
||||
effective_candidate = await get_builtin_effective_candidate(db, datasource.source)
|
||||
checksum, _credential_context = await build_builtin_connectivity_checksum(
|
||||
datasource.source,
|
||||
effective_candidate["endpoint"],
|
||||
effective_candidate["auth_type"],
|
||||
effective_candidate["headers"],
|
||||
effective_candidate["config"],
|
||||
db,
|
||||
)
|
||||
await save_connectivity_success(
|
||||
db,
|
||||
datasource.source,
|
||||
checksum,
|
||||
{"status_code": None},
|
||||
connected_by="collection",
|
||||
)
|
||||
await _update_next_run_at(datasource, db)
|
||||
logger.info_event(
|
||||
"Collector completed",
|
||||
event="collector.run.completed",
|
||||
context={"collector_name": collector_name, "datasource_id": datasource.id, "result": task_result},
|
||||
)
|
||||
logger.info("Collector %s completed: %s", collector_name, task_result)
|
||||
except asyncio.CancelledError:
|
||||
datasource.last_run_at = datetime.now(UTC)
|
||||
datasource.last_status = "cancelled"
|
||||
await db.commit()
|
||||
logger.warning_event(
|
||||
"Collector cancelled by operator",
|
||||
event="collector.run.cancelled",
|
||||
context={"collector_name": collector_name, "datasource_id": datasource.id},
|
||||
)
|
||||
logger.warning("Collector %s cancelled by operator", collector_name)
|
||||
raise
|
||||
except Exception as exc:
|
||||
datasource.last_run_at = datetime.now(UTC)
|
||||
datasource.last_status = "failed"
|
||||
await db.commit()
|
||||
logger.exception_event(
|
||||
"Collector failed",
|
||||
event="collector.run.failed",
|
||||
context={"collector_name": collector_name, "datasource_id": datasource.id, "error": str(exc)},
|
||||
)
|
||||
logger.exception("Collector %s failed: %s", collector_name, exc)
|
||||
|
||||
|
||||
async def cleanup_stale_running_tasks(max_age_hours: int = 2) -> int:
|
||||
@@ -252,11 +194,7 @@ async def cleanup_stale_running_tasks(max_age_hours: int = 2) -> int:
|
||||
|
||||
if stale_tasks:
|
||||
await db.commit()
|
||||
logger.warning_event(
|
||||
"Cleaned up stale running collection tasks",
|
||||
event="collector.cleanup.stale_tasks_cleaned",
|
||||
context={"count": len(stale_tasks)},
|
||||
)
|
||||
logger.warning("Cleaned up %s stale running collection task(s)", len(stale_tasks))
|
||||
|
||||
return len(stale_tasks)
|
||||
|
||||
@@ -265,14 +203,14 @@ def start_scheduler() -> None:
|
||||
"""Start the scheduler."""
|
||||
if not scheduler.running:
|
||||
scheduler.start()
|
||||
logger.info_event("Scheduler started", event="scheduler.started")
|
||||
logger.info("Scheduler started")
|
||||
|
||||
|
||||
def stop_scheduler() -> None:
|
||||
"""Stop the scheduler."""
|
||||
if scheduler.running:
|
||||
scheduler.shutdown(wait=False)
|
||||
logger.info_event("Scheduler stopped", event="scheduler.stopped")
|
||||
logger.info("Scheduler stopped")
|
||||
|
||||
|
||||
async def sync_scheduler_with_datasources() -> None:
|
||||
@@ -333,20 +271,12 @@ def run_collector_now(collector_name: str) -> bool:
|
||||
"""Run a collector immediately (not scheduled)."""
|
||||
collector = collector_registry.get(collector_name)
|
||||
if not collector:
|
||||
logger.error_event(
|
||||
"Collector not found",
|
||||
event="collector.trigger.collector_missing",
|
||||
context={"collector_name": collector_name},
|
||||
)
|
||||
logger.error("Collector not found: %s", collector_name)
|
||||
return False
|
||||
|
||||
existing_task = get_running_collector_task(collector_name)
|
||||
if existing_task is not None and not existing_task.done():
|
||||
logger.warning_event(
|
||||
"Collector is already running in-memory; skipping duplicate trigger",
|
||||
event="collector.trigger.skipped_already_running",
|
||||
context={"collector_name": collector_name},
|
||||
)
|
||||
logger.warning("Collector %s is already running in-memory; skipping duplicate trigger", collector_name)
|
||||
return False
|
||||
|
||||
try:
|
||||
@@ -359,18 +289,10 @@ def run_collector_now(collector_name: str) -> bool:
|
||||
RUNNING_COLLECTOR_TASKS.pop(collector_name, None)
|
||||
|
||||
task.add_done_callback(_cleanup_task)
|
||||
logger.info_event(
|
||||
"Triggered collector",
|
||||
event="collector.trigger.started",
|
||||
context={"collector_name": collector_name},
|
||||
)
|
||||
logger.info("Triggered collector: %s", collector_name)
|
||||
return True
|
||||
except Exception as exc:
|
||||
logger.error_event(
|
||||
"Failed to trigger collector",
|
||||
event="collector.trigger.failed",
|
||||
context={"collector_name": collector_name, "error": str(exc)},
|
||||
)
|
||||
logger.error("Failed to trigger collector %s: %s", collector_name, exc)
|
||||
return False
|
||||
|
||||
|
||||
|
||||
@@ -19,10 +19,6 @@ ALLOWED_ACTIONS: dict[str, dict[str, Any]] = {
|
||||
"command": ["./planet.sh", "restart", "-b"],
|
||||
"recovery_mode": "backend",
|
||||
},
|
||||
"restart-frontend": {
|
||||
"command": ["./planet.sh", "restart", "-f"],
|
||||
"recovery_mode": "frontend",
|
||||
},
|
||||
"restart-ai-provider": {
|
||||
"command": ["./planet.sh", "restart", "-a"],
|
||||
"recovery_mode": "ai-provider",
|
||||
|
||||
@@ -1,532 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
|
||||
from collections import Counter, deque
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from app.core.security import redis_client
|
||||
|
||||
DEFAULT_LOG_LINE_LIMIT = 200
|
||||
MAX_LOG_LINE_LIMIT = 1000
|
||||
BUFFER_LOG_LIMIT = 1000
|
||||
BUFFER_LOG_TTL_SECONDS = 7 * 24 * 60 * 60
|
||||
LOG_BUFFER_KEY_PREFIX = "planet:system_logs"
|
||||
|
||||
LOG_LEVEL_ERROR = "error"
|
||||
LOG_LEVEL_WARNING = "warning"
|
||||
LOG_LEVEL_INFO = "info"
|
||||
LOG_LEVEL_DEBUG = "debug"
|
||||
LOG_LEVEL_ALL = "all"
|
||||
|
||||
SUPPORTED_LOG_LEVELS = {
|
||||
LOG_LEVEL_ALL,
|
||||
LOG_LEVEL_ERROR,
|
||||
LOG_LEVEL_WARNING,
|
||||
LOG_LEVEL_INFO,
|
||||
LOG_LEVEL_DEBUG,
|
||||
}
|
||||
|
||||
LOG_LEVEL_ALIASES = {
|
||||
"warn": LOG_LEVEL_WARNING,
|
||||
"warning": LOG_LEVEL_WARNING,
|
||||
"err": LOG_LEVEL_ERROR,
|
||||
"error": LOG_LEVEL_ERROR,
|
||||
"info": LOG_LEVEL_INFO,
|
||||
"information": LOG_LEVEL_INFO,
|
||||
"debug": LOG_LEVEL_DEBUG,
|
||||
"trace": LOG_LEVEL_DEBUG,
|
||||
"critical": LOG_LEVEL_ERROR,
|
||||
"fatal": LOG_LEVEL_ERROR,
|
||||
}
|
||||
|
||||
TIMESTAMP_FORMATS = (
|
||||
"%Y-%m-%d %H:%M:%S.%f",
|
||||
"%Y-%m-%d %H:%M:%S",
|
||||
"%Y-%m-%dT%H:%M:%S.%f",
|
||||
"%Y-%m-%dT%H:%M:%S",
|
||||
)
|
||||
|
||||
LEVEL_PATTERNS = (
|
||||
("CRITICAL", LOG_LEVEL_ERROR),
|
||||
("FATAL", LOG_LEVEL_ERROR),
|
||||
("ERROR", LOG_LEVEL_ERROR),
|
||||
("WARNING", LOG_LEVEL_WARNING),
|
||||
("WARN", LOG_LEVEL_WARNING),
|
||||
("INFO", LOG_LEVEL_INFO),
|
||||
("DEBUG", LOG_LEVEL_DEBUG),
|
||||
("TRACE", LOG_LEVEL_DEBUG),
|
||||
)
|
||||
|
||||
LEADING_LEVEL_PATTERN = re.compile(
|
||||
r"^\s*(?:\[[^\]]+\]\s*)?(CRITICAL|FATAL|ERROR|WARNING|WARN|INFO|DEBUG|TRACE)\b[:\s-]*",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
EMBEDDED_LEVEL_PATTERN = re.compile(
|
||||
r"\b(CRITICAL|FATAL|ERROR|WARNING|WARN|INFO|DEBUG|TRACE)\b",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
CONTROL_CHAR_PATTERN = re.compile(r"[\x00-\x08\x0b\x0c\x0e-\x1f\x7f]")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class LogSource:
|
||||
source_id: str
|
||||
name: str
|
||||
kind: str
|
||||
location: str
|
||||
description: str
|
||||
category: str
|
||||
status: str = "ok"
|
||||
buffer_key: str | None = None
|
||||
container_name: str | None = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class StructuredLogEntry:
|
||||
timestamp: datetime | None
|
||||
level: str | None
|
||||
display_line: str
|
||||
raw_line: str
|
||||
search_text: str
|
||||
|
||||
|
||||
@dataclass
|
||||
class DailyLogMarker:
|
||||
date_token: str
|
||||
total: int
|
||||
dominant_level: str
|
||||
|
||||
|
||||
LOG_SOURCES: dict[str, LogSource] = {
|
||||
"backend": LogSource(
|
||||
source_id="backend",
|
||||
name="后端服务",
|
||||
kind="file",
|
||||
location="/tmp/planet_backend.log",
|
||||
description="FastAPI 后端、调度器和采集任务共享日志。",
|
||||
category="service",
|
||||
),
|
||||
"frontend": LogSource(
|
||||
source_id="frontend",
|
||||
name="前端开发服务",
|
||||
kind="file",
|
||||
location="/tmp/planet_frontend.log",
|
||||
description="控制台与 Earth 前端开发服务输出。",
|
||||
category="service",
|
||||
),
|
||||
"ai-provider": LogSource(
|
||||
source_id="ai-provider",
|
||||
name="AI Provider",
|
||||
kind="docker",
|
||||
location="docker://planet_aiprovider",
|
||||
description="AI Provider 容器实时输出日志。",
|
||||
category="service",
|
||||
container_name="planet_aiprovider",
|
||||
),
|
||||
"earth-client": LogSource(
|
||||
source_id="earth-client",
|
||||
name="Earth 浏览器端",
|
||||
kind="buffer",
|
||||
location="redis://planet:system_logs:earth-client",
|
||||
description="Earth 浏览器端上报的运行时错误与关键业务日志。",
|
||||
category="client",
|
||||
buffer_key=f"{LOG_BUFFER_KEY_PREFIX}:earth-client",
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def normalize_log_level(level: str | None) -> str:
|
||||
if level is None:
|
||||
return LOG_LEVEL_ALL
|
||||
normalized = str(level).strip().lower()
|
||||
if normalized in {"", LOG_LEVEL_ALL}:
|
||||
return LOG_LEVEL_ALL
|
||||
return LOG_LEVEL_ALIASES.get(normalized, LOG_LEVEL_ALL)
|
||||
|
||||
|
||||
def normalize_log_levels(level: str | None = None, levels: str | None = None) -> tuple[str, ...]:
|
||||
normalized_levels: list[str] = []
|
||||
if levels:
|
||||
for item in str(levels).split(","):
|
||||
normalized = normalize_log_level(item)
|
||||
if normalized != LOG_LEVEL_ALL and normalized not in normalized_levels:
|
||||
normalized_levels.append(normalized)
|
||||
normalized_level = normalize_log_level(level)
|
||||
if normalized_level != LOG_LEVEL_ALL and normalized_level not in normalized_levels:
|
||||
normalized_levels.append(normalized_level)
|
||||
return tuple(normalized_levels)
|
||||
|
||||
|
||||
def get_source_status(source: LogSource) -> str:
|
||||
if source.kind == "file":
|
||||
path = Path(source.location)
|
||||
if not path.exists():
|
||||
return "missing"
|
||||
return "ok" if path.stat().st_size > 0 else "empty"
|
||||
if source.kind == "docker":
|
||||
return "ok" if shutil.which("docker") else "docker_unavailable"
|
||||
if source.kind == "buffer":
|
||||
if not source.buffer_key:
|
||||
return "source_unavailable"
|
||||
try:
|
||||
return "ok" if redis_client.llen(source.buffer_key) > 0 else "empty"
|
||||
except Exception:
|
||||
return "source_unavailable"
|
||||
return "source_unavailable"
|
||||
|
||||
|
||||
def list_log_sources() -> list[dict[str, str]]:
|
||||
items: list[dict[str, str]] = []
|
||||
for source in LOG_SOURCES.values():
|
||||
items.append(
|
||||
{
|
||||
"source_id": source.source_id,
|
||||
"name": source.name,
|
||||
"kind": source.kind,
|
||||
"location": source.location,
|
||||
"description": source.description,
|
||||
"category": source.category,
|
||||
"status": get_source_status(source),
|
||||
}
|
||||
)
|
||||
return items
|
||||
|
||||
|
||||
def get_buffer_log_key(source_id: str) -> str:
|
||||
return f"{LOG_BUFFER_KEY_PREFIX}:{source_id}"
|
||||
|
||||
|
||||
def append_buffer_log(
|
||||
source_id: str,
|
||||
*,
|
||||
level: str,
|
||||
message: str,
|
||||
context: dict[str, Any] | None = None,
|
||||
) -> None:
|
||||
payload = {
|
||||
"timestamp": datetime.now(tz=UTC).isoformat(),
|
||||
"level": normalize_log_level(level),
|
||||
"message": message,
|
||||
"context": context or {},
|
||||
}
|
||||
buffer_key = get_buffer_log_key(source_id)
|
||||
redis_client.rpush(buffer_key, json.dumps(payload, ensure_ascii=False))
|
||||
redis_client.ltrim(buffer_key, -BUFFER_LOG_LIMIT, -1)
|
||||
redis_client.expire(buffer_key, BUFFER_LOG_TTL_SECONDS)
|
||||
|
||||
|
||||
def parse_timestamp(raw_value: str | None) -> datetime | None:
|
||||
if not raw_value:
|
||||
return None
|
||||
candidate = str(raw_value).strip()
|
||||
if not candidate:
|
||||
return None
|
||||
candidate = candidate.replace("Z", "+00:00")
|
||||
try:
|
||||
parsed = datetime.fromisoformat(candidate)
|
||||
return parsed if parsed.tzinfo else parsed.replace(tzinfo=UTC)
|
||||
except ValueError:
|
||||
pass
|
||||
for fmt in TIMESTAMP_FORMATS:
|
||||
try:
|
||||
return datetime.strptime(candidate, fmt).replace(tzinfo=UTC)
|
||||
except ValueError:
|
||||
continue
|
||||
return None
|
||||
|
||||
|
||||
def parse_prefixed_timestamp(line: str) -> tuple[datetime | None, str]:
|
||||
stripped = line.strip()
|
||||
if not stripped:
|
||||
return None, ""
|
||||
for prefix_length in (35, 32, 29, 26, 23, 19):
|
||||
if len(stripped) < prefix_length:
|
||||
continue
|
||||
prefix = stripped[:prefix_length]
|
||||
timestamp = parse_timestamp(prefix)
|
||||
if timestamp is not None:
|
||||
return timestamp, stripped[prefix_length:].lstrip()
|
||||
first_token = stripped.split(maxsplit=1)[0]
|
||||
timestamp = parse_timestamp(first_token)
|
||||
if timestamp is not None:
|
||||
remainder = stripped[len(first_token):].lstrip()
|
||||
return timestamp, remainder
|
||||
return None, stripped
|
||||
|
||||
|
||||
def infer_log_level_from_text(text: str, *, allow_embedded: bool = True) -> str | None:
|
||||
leading_match = LEADING_LEVEL_PATTERN.match(text)
|
||||
if leading_match:
|
||||
return normalize_log_level(leading_match.group(1))
|
||||
|
||||
if allow_embedded:
|
||||
embedded_match = EMBEDDED_LEVEL_PATTERN.search(text)
|
||||
if embedded_match:
|
||||
return normalize_log_level(embedded_match.group(1))
|
||||
upper_text = text.upper()
|
||||
for pattern, normalized in LEVEL_PATTERNS:
|
||||
if f"{pattern}:" in upper_text or f"{pattern} " in upper_text:
|
||||
return normalized
|
||||
return None
|
||||
|
||||
|
||||
def build_display_line(timestamp: datetime | None, level: str | None, message: str) -> str:
|
||||
message_part = message.strip() if message else ""
|
||||
parts = []
|
||||
if timestamp is not None:
|
||||
parts.append(timestamp.astimezone(UTC).strftime("%Y-%m-%d %H:%M:%S"))
|
||||
if level:
|
||||
parts.append(level.upper())
|
||||
if message_part:
|
||||
parts.append(message_part)
|
||||
return " ".join(parts).strip()
|
||||
|
||||
|
||||
def sanitize_text_log_line(line: str) -> str:
|
||||
return CONTROL_CHAR_PATTERN.sub("", line)
|
||||
|
||||
|
||||
def parse_text_log_entry(line: str) -> StructuredLogEntry:
|
||||
sanitized_line = sanitize_text_log_line(line).rstrip("\n")
|
||||
timestamp, remainder = parse_prefixed_timestamp(sanitized_line)
|
||||
level = infer_log_level_from_text(remainder or sanitized_line, allow_embedded=False)
|
||||
display_line = sanitized_line
|
||||
return StructuredLogEntry(
|
||||
timestamp=timestamp,
|
||||
level=level,
|
||||
display_line=display_line,
|
||||
raw_line=display_line,
|
||||
search_text=display_line.lower(),
|
||||
)
|
||||
|
||||
|
||||
def build_buffer_entry(payload: dict[str, Any]) -> StructuredLogEntry:
|
||||
timestamp = parse_timestamp(str(payload.get("timestamp", "")).strip())
|
||||
level = normalize_log_level(payload.get("level"))
|
||||
if level == LOG_LEVEL_ALL:
|
||||
level = None
|
||||
message = str(payload.get("message", "")).strip()
|
||||
context = payload.get("context")
|
||||
context_map = context if isinstance(context, dict) else {}
|
||||
context_fragments = []
|
||||
for key in ("category", "module", "url", "detail"):
|
||||
value = str(context_map.get(key, "")).strip()
|
||||
if value:
|
||||
context_fragments.append(f"{key}={value}")
|
||||
message_with_context = " | ".join([message, *context_fragments]) if context_fragments else message
|
||||
display_line = build_display_line(timestamp, level, message_with_context)
|
||||
search_text = " ".join(
|
||||
[
|
||||
message,
|
||||
json.dumps(context_map, ensure_ascii=False, sort_keys=True),
|
||||
display_line,
|
||||
]
|
||||
).lower()
|
||||
return StructuredLogEntry(
|
||||
timestamp=timestamp,
|
||||
level=level,
|
||||
display_line=display_line,
|
||||
raw_line=json.dumps(payload, ensure_ascii=False, sort_keys=True),
|
||||
search_text=search_text,
|
||||
)
|
||||
|
||||
|
||||
def read_file_entries(source: LogSource, scan_limit: int) -> list[StructuredLogEntry]:
|
||||
path = Path(source.location)
|
||||
if not path.exists():
|
||||
return []
|
||||
with path.open("r", encoding="utf-8", errors="replace") as handle:
|
||||
recent_lines = deque(handle, maxlen=scan_limit)
|
||||
return [
|
||||
parse_text_log_entry(line)
|
||||
for line in recent_lines
|
||||
if sanitize_text_log_line(line).strip()
|
||||
]
|
||||
|
||||
|
||||
def read_docker_entries(source: LogSource, scan_limit: int) -> list[StructuredLogEntry]:
|
||||
if not shutil.which("docker") or not source.container_name:
|
||||
return []
|
||||
try:
|
||||
completed = subprocess.run(
|
||||
[
|
||||
"docker",
|
||||
"logs",
|
||||
"--timestamps",
|
||||
"--tail",
|
||||
str(scan_limit),
|
||||
source.container_name,
|
||||
],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
except OSError:
|
||||
return []
|
||||
if completed.returncode != 0:
|
||||
return []
|
||||
return [
|
||||
parse_text_log_entry(line)
|
||||
for line in completed.stdout.splitlines()
|
||||
if line.strip()
|
||||
]
|
||||
|
||||
|
||||
def read_buffer_entries(source: LogSource, scan_limit: int) -> list[StructuredLogEntry]:
|
||||
if not source.buffer_key:
|
||||
return []
|
||||
try:
|
||||
raw_items = redis_client.lrange(source.buffer_key, -scan_limit, -1)
|
||||
except Exception:
|
||||
return []
|
||||
entries: list[StructuredLogEntry] = []
|
||||
for raw_item in raw_items:
|
||||
try:
|
||||
payload = json.loads(raw_item)
|
||||
except json.JSONDecodeError:
|
||||
entries.append(parse_text_log_entry(str(raw_item)))
|
||||
continue
|
||||
if isinstance(payload, dict):
|
||||
entries.append(build_buffer_entry(payload))
|
||||
else:
|
||||
entries.append(parse_text_log_entry(str(raw_item)))
|
||||
return entries
|
||||
|
||||
|
||||
def read_source_entries(source: LogSource, scan_limit: int) -> list[StructuredLogEntry]:
|
||||
if source.kind == "file":
|
||||
return read_file_entries(source, scan_limit)
|
||||
if source.kind == "docker":
|
||||
return read_docker_entries(source, scan_limit)
|
||||
if source.kind == "buffer":
|
||||
return read_buffer_entries(source, scan_limit)
|
||||
return []
|
||||
|
||||
|
||||
def matches_levels(entry: StructuredLogEntry, selected_levels: tuple[str, ...]) -> bool:
|
||||
if not selected_levels:
|
||||
return True
|
||||
return entry.level in selected_levels
|
||||
|
||||
|
||||
def matches_date_range(
|
||||
entry: StructuredLogEntry,
|
||||
start_date: str | None,
|
||||
end_date: str | None,
|
||||
) -> bool:
|
||||
if not start_date and not end_date:
|
||||
return True
|
||||
if entry.timestamp is None:
|
||||
return False
|
||||
date_token = entry.timestamp.astimezone(UTC).date().isoformat()
|
||||
if start_date and date_token < start_date:
|
||||
return False
|
||||
if end_date and date_token > end_date:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def matches_search(entry: StructuredLogEntry, search: str | None) -> bool:
|
||||
if search is None:
|
||||
return True
|
||||
query = search.strip().lower()
|
||||
if not query:
|
||||
return True
|
||||
return query in entry.search_text
|
||||
|
||||
|
||||
def build_daily_log_markers(entries: list[StructuredLogEntry]) -> list[dict[str, Any]]:
|
||||
grouped: dict[str, list[StructuredLogEntry]] = {}
|
||||
for entry in entries:
|
||||
if entry.timestamp is None:
|
||||
continue
|
||||
date_token = entry.timestamp.astimezone(UTC).date().isoformat()
|
||||
grouped.setdefault(date_token, []).append(entry)
|
||||
|
||||
markers: list[DailyLogMarker] = []
|
||||
for date_token, group in sorted(grouped.items()):
|
||||
level_counts = Counter(
|
||||
entry.level
|
||||
for entry in group
|
||||
if entry.level in SUPPORTED_LOG_LEVELS and entry.level != LOG_LEVEL_ALL
|
||||
)
|
||||
dominant_level = LOG_LEVEL_INFO
|
||||
if level_counts:
|
||||
dominant_level = sorted(
|
||||
level_counts.items(),
|
||||
key=lambda item: (
|
||||
-item[1],
|
||||
("error", "warning", "info", "debug").index(item[0]),
|
||||
),
|
||||
)[0][0]
|
||||
markers.append(
|
||||
DailyLogMarker(
|
||||
date_token=date_token,
|
||||
total=len(group),
|
||||
dominant_level=dominant_level,
|
||||
)
|
||||
)
|
||||
return [marker.__dict__ for marker in markers]
|
||||
|
||||
|
||||
def read_log_snapshot(
|
||||
source_id: str,
|
||||
limit: int,
|
||||
*,
|
||||
level: str = LOG_LEVEL_ALL,
|
||||
levels: str | None = None,
|
||||
start_date: str | None = None,
|
||||
end_date: str | None = None,
|
||||
search: str | None = None,
|
||||
) -> dict[str, Any] | None:
|
||||
source = LOG_SOURCES.get(source_id)
|
||||
if source is None:
|
||||
return None
|
||||
|
||||
selected_levels = normalize_log_levels(level, levels)
|
||||
search_query = (search or "").strip()
|
||||
scan_limit = max(min(MAX_LOG_LINE_LIMIT * 5, 5000), limit * 5, BUFFER_LOG_LIMIT if source.kind == "buffer" else 1000)
|
||||
all_entries = read_source_entries(source, scan_limit)
|
||||
marker_entries = [
|
||||
entry
|
||||
for entry in all_entries
|
||||
if matches_levels(entry, selected_levels) and matches_search(entry, search_query)
|
||||
]
|
||||
filtered_entries = [
|
||||
entry
|
||||
for entry in marker_entries
|
||||
if matches_date_range(entry, start_date, end_date)
|
||||
]
|
||||
visible_entries = filtered_entries[-limit:]
|
||||
|
||||
compatibility_level = selected_levels[0] if len(selected_levels) == 1 else LOG_LEVEL_ALL
|
||||
return {
|
||||
"source_id": source.source_id,
|
||||
"name": source.name,
|
||||
"kind": source.kind,
|
||||
"location": source.location,
|
||||
"description": source.description,
|
||||
"category": source.category,
|
||||
"status": get_source_status(source),
|
||||
"level": compatibility_level,
|
||||
"selected_levels": list(selected_levels),
|
||||
"search_query": search_query,
|
||||
"available_levels": [
|
||||
LOG_LEVEL_ALL,
|
||||
LOG_LEVEL_ERROR,
|
||||
LOG_LEVEL_WARNING,
|
||||
LOG_LEVEL_INFO,
|
||||
LOG_LEVEL_DEBUG,
|
||||
],
|
||||
"daily_markers": build_daily_log_markers(marker_entries),
|
||||
"line_limit": limit,
|
||||
"line_count": len(visible_entries),
|
||||
"lines": [entry.display_line for entry in visible_entries],
|
||||
}
|
||||
@@ -17,7 +17,7 @@ TV_LIVE_SOURCE_COLLECTOR = "news_live_streams"
|
||||
TV_LIVE_SOURCE_DATA_TYPE = "news_live_stream"
|
||||
|
||||
DEFAULT_TV_SETTINGS = {
|
||||
"default_source_id": DEFAULT_TV_SOURCE_ID,
|
||||
"default_source_id": DEFAULT_TV_SOURCE_ID,
|
||||
"auto_fallback": True,
|
||||
"sources": [
|
||||
{
|
||||
@@ -362,7 +362,7 @@ def _build_collected_tv_source(record: CollectedData, index: int) -> dict[str, A
|
||||
"sort_order": metadata.get("sort_order", 200 + index),
|
||||
"collector_source": record.source,
|
||||
"notes": record.description or metadata.get("notes") or "",
|
||||
"updated_at": to_iso8601_utc(record.collected_at or record.reference_date or datetime.now(UTC)),
|
||||
"updated_at": to_iso8601_utc(record.updated_at or record.reference_date or datetime.now(UTC)),
|
||||
},
|
||||
index=index,
|
||||
)
|
||||
|
||||
@@ -2,7 +2,6 @@ from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import shlex
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
@@ -60,8 +59,6 @@ def wait_for_recovery(action: str) -> tuple[bool, str]:
|
||||
recovery_mode = get_action_recovery_mode(action)
|
||||
if recovery_mode == "backend":
|
||||
return wait_for_http("http://localhost:8000/health"), "backend health recovery"
|
||||
if recovery_mode == "frontend":
|
||||
return wait_for_http("http://localhost:3000"), "frontend entrypoint recovery"
|
||||
if recovery_mode == "ai-provider":
|
||||
return wait_for_http("http://localhost:8010/health"), "ai provider health recovery"
|
||||
if recovery_mode == "database":
|
||||
@@ -111,9 +108,8 @@ def main() -> int:
|
||||
)
|
||||
append_task_log(args.task_id, "restart command started")
|
||||
|
||||
shell_command = shlex.join(command)
|
||||
completed = subprocess.run(
|
||||
["zsh", "-ic", shell_command],
|
||||
command,
|
||||
cwd=str(ROOT_DIR),
|
||||
env=env,
|
||||
capture_output=True,
|
||||
|
||||
+2
-342
@@ -35,7 +35,6 @@ async def test_health_check():
|
||||
data = response.json()
|
||||
assert data["status"] == "healthy"
|
||||
assert "version" in data
|
||||
assert response.headers["x-request-id"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@@ -162,345 +161,6 @@ async def test_alerts_endpoint_with_auth(auth_headers):
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_system_log_sources_requires_super_admin(auth_headers):
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="testuser",
|
||||
email="test@example.com",
|
||||
password_hash="hashed",
|
||||
role="admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/api/v1/system/logs/sources", headers=auth_headers)
|
||||
assert response.status_code == 403
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_system_log_sources_with_super_admin(auth_headers):
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="root",
|
||||
email="root@example.com",
|
||||
password_hash="hashed",
|
||||
role="super_admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch(
|
||||
"app.api.v1.system_control.list_log_sources",
|
||||
return_value=[
|
||||
{
|
||||
"source_id": "backend",
|
||||
"name": "后端服务",
|
||||
"kind": "file",
|
||||
"location": "/tmp/planet_backend.log",
|
||||
"description": "FastAPI 后端、调度器和采集任务共享日志。",
|
||||
"category": "service",
|
||||
"status": "ok",
|
||||
}
|
||||
],
|
||||
):
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/api/v1/system/logs/sources", headers=auth_headers)
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["items"][0]["source_id"] == "backend"
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_system_log_snapshot_with_super_admin(auth_headers):
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="root",
|
||||
email="root@example.com",
|
||||
password_hash="hashed",
|
||||
role="super_admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch(
|
||||
"app.api.v1.system_control.read_log_snapshot",
|
||||
return_value={
|
||||
"source_id": "backend",
|
||||
"name": "后端服务",
|
||||
"kind": "file",
|
||||
"location": "/tmp/planet_backend.log",
|
||||
"description": "FastAPI 后端、调度器和采集任务共享日志。",
|
||||
"category": "service",
|
||||
"status": "ok",
|
||||
"level": "all",
|
||||
"selected_levels": [],
|
||||
"search_query": "",
|
||||
"available_levels": ["all", "error", "warning", "info", "debug"],
|
||||
"daily_markers": [],
|
||||
"line_limit": 50,
|
||||
"line_count": 2,
|
||||
"lines": ["line 1", "line 2"],
|
||||
},
|
||||
):
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/api/v1/system/logs/backend?limit=50", headers=auth_headers)
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["source_id"] == "backend"
|
||||
assert data["line_count"] == 2
|
||||
assert data["lines"] == ["line 1", "line 2"]
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_system_log_snapshot_supports_level_filter(auth_headers):
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="root",
|
||||
email="root@example.com",
|
||||
password_hash="hashed",
|
||||
role="super_admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch(
|
||||
"app.api.v1.system_control.read_log_snapshot",
|
||||
return_value={
|
||||
"source_id": "backend",
|
||||
"name": "后端服务",
|
||||
"kind": "file",
|
||||
"location": "/tmp/planet_backend.log",
|
||||
"description": "FastAPI 后端、调度器和采集任务共享日志。",
|
||||
"category": "service",
|
||||
"status": "ok",
|
||||
"level": "error",
|
||||
"selected_levels": ["error"],
|
||||
"search_query": "",
|
||||
"available_levels": ["all", "error", "warning", "info", "debug"],
|
||||
"daily_markers": [],
|
||||
"line_limit": 50,
|
||||
"line_count": 1,
|
||||
"lines": ["ERROR: failed"],
|
||||
},
|
||||
):
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/api/v1/system/logs/backend?limit=50&level=error", headers=auth_headers)
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["level"] == "error"
|
||||
assert data["lines"] == ["ERROR: failed"]
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_system_log_snapshot_supports_date_range_filter(auth_headers):
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="root",
|
||||
email="root@example.com",
|
||||
password_hash="hashed",
|
||||
role="super_admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch(
|
||||
"app.api.v1.system_control.read_log_snapshot",
|
||||
return_value={
|
||||
"source_id": "backend",
|
||||
"name": "后端服务",
|
||||
"kind": "file",
|
||||
"location": "/tmp/planet_backend.log",
|
||||
"description": "FastAPI 后端、调度器和采集任务共享日志。",
|
||||
"category": "service",
|
||||
"status": "ok",
|
||||
"level": "all",
|
||||
"selected_levels": [],
|
||||
"search_query": "",
|
||||
"available_levels": ["all", "error", "warning", "info", "debug"],
|
||||
"daily_markers": [],
|
||||
"line_limit": 50,
|
||||
"line_count": 1,
|
||||
"lines": ["2026-04-23 INFO: service started"],
|
||||
},
|
||||
) as mock_read_log_snapshot:
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get(
|
||||
"/api/v1/system/logs/backend?limit=50&start_date=2026-04-20&end_date=2026-04-23",
|
||||
headers=auth_headers,
|
||||
)
|
||||
assert response.status_code == 200
|
||||
mock_read_log_snapshot.assert_called_once_with(
|
||||
"backend",
|
||||
50,
|
||||
level="all",
|
||||
levels=None,
|
||||
start_date="2026-04-20",
|
||||
end_date="2026-04-23",
|
||||
search=None,
|
||||
)
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_system_log_snapshot_supports_levels_and_search_filter(auth_headers):
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="root",
|
||||
email="root@example.com",
|
||||
password_hash="hashed",
|
||||
role="super_admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch(
|
||||
"app.api.v1.system_control.read_log_snapshot",
|
||||
return_value={
|
||||
"source_id": "backend",
|
||||
"name": "后端服务",
|
||||
"kind": "file",
|
||||
"location": "/tmp/planet_backend.log",
|
||||
"description": "FastAPI 后端、调度器和采集任务共享日志。",
|
||||
"category": "service",
|
||||
"status": "ok",
|
||||
"level": "all",
|
||||
"selected_levels": ["error", "warning"],
|
||||
"search_query": "timeout",
|
||||
"available_levels": ["all", "error", "warning", "info", "debug"],
|
||||
"daily_markers": [],
|
||||
"line_limit": 50,
|
||||
"line_count": 1,
|
||||
"lines": ["2026-04-23 10:00:00 ERROR timeout"],
|
||||
},
|
||||
) as mock_read_log_snapshot:
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get(
|
||||
"/api/v1/system/logs/backend?limit=50&levels=error,warning&search=timeout",
|
||||
headers=auth_headers,
|
||||
)
|
||||
assert response.status_code == 200
|
||||
mock_read_log_snapshot.assert_called_once_with(
|
||||
"backend",
|
||||
50,
|
||||
level="all",
|
||||
levels="error,warning",
|
||||
start_date=None,
|
||||
end_date=None,
|
||||
search="timeout",
|
||||
)
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_system_log_snapshot_rejects_invalid_date_range(auth_headers):
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="root",
|
||||
email="root@example.com",
|
||||
password_hash="hashed",
|
||||
role="super_admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get(
|
||||
"/api/v1/system/logs/backend?start_date=2026-04-31",
|
||||
headers=auth_headers,
|
||||
)
|
||||
assert response.status_code == 400
|
||||
assert "start_date must be in YYYY-MM-DD format" in response.json()["detail"]
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ingest_earth_client_log_accepts_public_events():
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch("app.api.v1.system_control.append_buffer_log") as mock_append_buffer_log:
|
||||
with patch("app.api.v1.system_control.record_system_log", new_callable=AsyncMock) as mock_record_system_log:
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.post(
|
||||
"/api/v1/system/logs/earth-client",
|
||||
json={
|
||||
"level": "error",
|
||||
"message": "登陆点加载失败: 登陆点接口返回 HTTP 500",
|
||||
"category": "startup-load",
|
||||
"module": "layer-startup",
|
||||
},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["accepted"] is True
|
||||
assert data["source_id"] == "earth-client"
|
||||
mock_append_buffer_log.assert_called_once()
|
||||
mock_record_system_log.assert_awaited_once()
|
||||
persisted_kwargs = mock_record_system_log.await_args.kwargs
|
||||
assert persisted_kwargs["source"] == "earth-client"
|
||||
assert persisted_kwargs["event"] == "earth.client.runtime_log"
|
||||
assert persisted_kwargs["category"] == "startup-load"
|
||||
assert persisted_kwargs["level"] == "error"
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_request_id_header_is_echoed_when_provided():
|
||||
transport = ASGITransport(app=app)
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/health", headers={"X-Request-ID": "planet-test-request"})
|
||||
assert response.status_code == 200
|
||||
assert response.headers["x-request-id"] == "planet-test-request"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_invalid_token():
|
||||
"""Test that invalid token is rejected"""
|
||||
@@ -603,8 +263,6 @@ async def test_ai_situational_analysis_returns_503_when_disabled(auth_headers):
|
||||
assert "content_blocks" in data
|
||||
assert "text_blocks" in data
|
||||
assert "thinking_blocks" in data
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@@ -724,6 +382,8 @@ async def test_save_playground_session_with_auth(auth_headers):
|
||||
assert data["state"]["objective"] == "测试目标"
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
|
||||
@@ -6,7 +6,6 @@ from unittest.mock import AsyncMock, MagicMock, patch
|
||||
|
||||
from app.services.collectors.top500 import TOP500Collector
|
||||
from app.services.collectors.base import BaseCollector, HTTPCollector
|
||||
from app.models.task import CollectionTask
|
||||
|
||||
|
||||
class TestBaseCollector:
|
||||
@@ -20,31 +19,6 @@ class TestBaseCollector:
|
||||
assert collector.module == "L1"
|
||||
assert collector.frequency_hours == 4
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_update_phase_progress_tracks_phase_fields(self, mock_db_session):
|
||||
"""Test phase-level progress updates independently from record totals"""
|
||||
collector = TOP500Collector()
|
||||
task = CollectionTask(datasource_id=1, status="running", phase="fetching")
|
||||
collector._current_task = task
|
||||
collector._db_session = mock_db_session
|
||||
|
||||
with patch.object(collector, "_publish_task_update", new=AsyncMock()) as publish:
|
||||
await collector.update_phase_progress(
|
||||
current=512,
|
||||
total=1024,
|
||||
unit="bytes",
|
||||
message="Downloading dataset",
|
||||
commit=True,
|
||||
)
|
||||
|
||||
assert task.phase_progress == 50.0
|
||||
assert task.phase_current == 512
|
||||
assert task.phase_total == 1024
|
||||
assert task.phase_unit == "bytes"
|
||||
assert task.phase_message == "Downloading dataset"
|
||||
mock_db_session.commit.assert_awaited_once()
|
||||
publish.assert_awaited_once()
|
||||
|
||||
|
||||
class TestTOP500Collector:
|
||||
"""Tests for TOP500Collector"""
|
||||
|
||||
@@ -1,199 +0,0 @@
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
|
||||
from app.api.v1.datasource_config import get_ai_provider_client
|
||||
from app.core.security import get_current_user
|
||||
from app.core.target_schema_registry import get_target_schema, list_target_schemas
|
||||
from app.main import app
|
||||
from app.models.user import User
|
||||
from app.services.datasource_mapping import execute_mapping, persist_mapped_records, redact_for_llm
|
||||
|
||||
|
||||
SAMPLE_AIS = {
|
||||
"data": [
|
||||
{
|
||||
"mmsi": "257123000",
|
||||
"latitude": "59.91",
|
||||
"longitude": "10.75",
|
||||
"speedOverGround": "12.4",
|
||||
"timestamp": "2026-04-28T00:00:00Z",
|
||||
"api_token": "secret-value",
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
def test_registry_exposes_v1_target_schemas():
|
||||
keys = {schema["key"] for schema in list_target_schemas()}
|
||||
|
||||
assert {"vessel_ais", "geo_points", "generic_records"}.issubset(keys)
|
||||
assert get_target_schema("vessel_ais").destination == "vessel_position"
|
||||
|
||||
|
||||
def test_mapping_engine_maps_and_validates_vessel_ais():
|
||||
mapping = {
|
||||
"source": {"items_path": "$.data[*]"},
|
||||
"fields": {
|
||||
"mmsi": {"path": "$.mmsi", "type": "integer"},
|
||||
"lat": {"path": "$.latitude", "type": "float"},
|
||||
"lon": {"path": "$.longitude", "type": "float"},
|
||||
"sog": {"path": "$.speedOverGround", "type": "float"},
|
||||
"received_at": {"path": "$.timestamp", "type": "datetime"},
|
||||
},
|
||||
}
|
||||
|
||||
result = execute_mapping(SAMPLE_AIS, mapping, "vessel_ais")
|
||||
|
||||
assert result["mapped_count"] == 1
|
||||
assert result["failed_count"] == 0
|
||||
assert result["records"][0]["mmsi"] == 257123000
|
||||
assert result["records"][0]["lat"] == 59.91
|
||||
|
||||
|
||||
def test_mapping_engine_reports_schema_errors():
|
||||
mapping = {
|
||||
"source": {"items_path": "$.data[*]"},
|
||||
"fields": {
|
||||
"mmsi": {"path": "$.mmsi", "type": "integer"},
|
||||
"lat": {"path": "$.missing_lat", "type": "float"},
|
||||
"lon": {"path": "$.longitude", "type": "float"},
|
||||
},
|
||||
}
|
||||
|
||||
result = execute_mapping(SAMPLE_AIS, mapping, "vessel_ais")
|
||||
|
||||
assert result["mapped_count"] == 0
|
||||
assert result["failed_count"] == 1
|
||||
assert any("lat" in error for error in result["errors"][0]["errors"])
|
||||
|
||||
|
||||
def test_redact_for_llm_masks_secret_like_fields():
|
||||
redacted = redact_for_llm(SAMPLE_AIS)
|
||||
|
||||
assert redacted["data"][0]["api_token"] == "[REDACTED]"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_persist_mapped_records_writes_generic_records():
|
||||
class FakeDB:
|
||||
def __init__(self):
|
||||
self.added = []
|
||||
self.committed = False
|
||||
|
||||
def add(self, value):
|
||||
self.added.append(value)
|
||||
|
||||
async def commit(self):
|
||||
self.committed = True
|
||||
|
||||
db = FakeDB()
|
||||
|
||||
count = await persist_mapped_records(
|
||||
db,
|
||||
datasource_name="custom_weather",
|
||||
datasource_config_id=42,
|
||||
target_schema="generic_records",
|
||||
records=[{"source_id": "row-1", "data": {"temp": 25}}],
|
||||
mapping_version=3,
|
||||
)
|
||||
|
||||
assert count == 1
|
||||
assert db.committed is True
|
||||
assert db.added[0].source == "custom_weather"
|
||||
assert db.added[0].data_type == "generic_records"
|
||||
assert db.added[0].extra_data["mapping_version"] == 3
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_mapping_preview_api_uses_deterministic_engine():
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="testuser",
|
||||
email="test@example.com",
|
||||
password_hash="hashed",
|
||||
role="admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
app.dependency_overrides = {get_current_user: override_get_current_user}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.post(
|
||||
"/api/v1/datasources/mappings/preview",
|
||||
json={
|
||||
"sample_payload": SAMPLE_AIS,
|
||||
"target_schema": "vessel_ais",
|
||||
"mapping_json": {
|
||||
"source": {"items_path": "$.data[*]"},
|
||||
"fields": {
|
||||
"mmsi": {"path": "$.mmsi", "type": "integer"},
|
||||
"lat": {"path": "$.latitude", "type": "float"},
|
||||
"lon": {"path": "$.longitude", "type": "float"},
|
||||
},
|
||||
},
|
||||
},
|
||||
)
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
assert response.status_code == 200
|
||||
payload = response.json()
|
||||
assert payload["success"] is True
|
||||
assert payload["preview"]["records"][0]["mmsi"] == 257123000
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_mapping_propose_api_redacts_sample_before_ai():
|
||||
seen_context = {}
|
||||
|
||||
class FakeAIClient:
|
||||
async def analyze(self, request, request_id=None):
|
||||
seen_context.update(request.context)
|
||||
return SimpleNamespace(
|
||||
content=(
|
||||
'{"source":{"items_path":"$.data[*]"},"fields":{'
|
||||
'"mmsi":{"path":"$.mmsi","type":"integer"},'
|
||||
'"lat":{"path":"$.latitude","type":"float"},'
|
||||
'"lon":{"path":"$.longitude","type":"float"}}}'
|
||||
)
|
||||
)
|
||||
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="testuser",
|
||||
email="test@example.com",
|
||||
password_hash="hashed",
|
||||
role="admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
def override_ai_client():
|
||||
return FakeAIClient()
|
||||
|
||||
app.dependency_overrides = {
|
||||
get_current_user: override_get_current_user,
|
||||
get_ai_provider_client: override_ai_client,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.post(
|
||||
"/api/v1/datasources/mappings/propose",
|
||||
json={
|
||||
"sample_payload": SAMPLE_AIS,
|
||||
"target_schema": "vessel_ais",
|
||||
"use_ai": True,
|
||||
},
|
||||
)
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
assert response.status_code == 200
|
||||
payload = response.json()
|
||||
assert payload["mapping_json"]["meta"]["generated_by"] == "ai_provider"
|
||||
assert seen_context["sample_payload"]["data"][0]["api_token"] == "[REDACTED]"
|
||||
@@ -1,49 +0,0 @@
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from app.services.earth_news import ParsedNewsItem, _serialize_item
|
||||
|
||||
|
||||
def test_serialize_item_includes_region_anchor_for_cruise():
|
||||
item = ParsedNewsItem(
|
||||
id="google-apac:test",
|
||||
title="Example APAC story",
|
||||
summary="Example summary",
|
||||
url="https://example.com/story",
|
||||
source="Example Source",
|
||||
feed_name="Global Monitor / APAC",
|
||||
feed_region="asia-pacific",
|
||||
homepage_url="https://example.com",
|
||||
published_at=datetime(2026, 4, 23, 2, 30, tzinfo=UTC),
|
||||
)
|
||||
|
||||
payload = _serialize_item(item, active_region="asia-pacific")
|
||||
|
||||
assert payload["latitude"] == 1.3521
|
||||
assert payload["longitude"] == 103.8198
|
||||
assert payload["location_label"] == "亚太"
|
||||
assert payload["location_inferred"] is True
|
||||
assert payload["is_focus_match"] is True
|
||||
assert payload["published_at"] == "2026-04-23T02:30:00Z"
|
||||
|
||||
|
||||
def test_serialize_item_falls_back_to_global_anchor():
|
||||
item = ParsedNewsItem(
|
||||
id="custom:test",
|
||||
title="Fallback story",
|
||||
summary="Fallback summary",
|
||||
url="https://example.com/fallback",
|
||||
source="Fallback Source",
|
||||
feed_name="Fallback Feed",
|
||||
feed_region="unknown-region",
|
||||
homepage_url="https://example.com",
|
||||
published_at=None,
|
||||
)
|
||||
|
||||
payload = _serialize_item(item, active_region="americas")
|
||||
|
||||
assert payload["latitude"] == 20.0
|
||||
assert payload["longitude"] == 0.0
|
||||
assert payload["location_label"] == "全球"
|
||||
assert payload["location_inferred"] is True
|
||||
assert payload["is_focus_match"] is False
|
||||
assert payload["published_at"] is None
|
||||
@@ -1,78 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
from io import StringIO
|
||||
|
||||
from app.core.logging import PlanetContextFilter, PlanetFormatter, get_logger
|
||||
from app.core.request_context import set_request_id
|
||||
|
||||
|
||||
def _capture_output(callback):
|
||||
stream = StringIO()
|
||||
handler = logging.StreamHandler(stream)
|
||||
handler.setFormatter(PlanetFormatter(datefmt="%Y-%m-%d %H:%M:%S"))
|
||||
handler.addFilter(PlanetContextFilter())
|
||||
|
||||
adapter = get_logger("tests.logging")
|
||||
target_logger = adapter.logger
|
||||
original_handlers = list(target_logger.handlers)
|
||||
original_level = target_logger.level
|
||||
original_propagate = target_logger.propagate
|
||||
|
||||
target_logger.handlers = [handler]
|
||||
target_logger.setLevel(logging.INFO)
|
||||
target_logger.propagate = False
|
||||
|
||||
try:
|
||||
callback(adapter)
|
||||
finally:
|
||||
handler.flush()
|
||||
target_logger.handlers = original_handlers
|
||||
target_logger.setLevel(original_level)
|
||||
target_logger.propagate = original_propagate
|
||||
|
||||
return stream.getvalue()
|
||||
|
||||
|
||||
def test_structured_logger_injects_request_id_and_event():
|
||||
set_request_id("req-test-123")
|
||||
try:
|
||||
output = _capture_output(
|
||||
lambda logger: logger.info_event(
|
||||
"collector started",
|
||||
event="collector.run.started",
|
||||
context={"collector_name": "bgp_news"},
|
||||
)
|
||||
)
|
||||
finally:
|
||||
set_request_id(None)
|
||||
|
||||
assert "request_id=req-test-123" in output
|
||||
assert "event=collector.run.started" in output
|
||||
assert "service=backend" in output
|
||||
assert '"collector_name": "bgp_news"' in output
|
||||
|
||||
|
||||
def test_structured_logger_redacts_sensitive_text_and_context():
|
||||
set_request_id("req-test-redact")
|
||||
try:
|
||||
output = _capture_output(
|
||||
lambda logger: logger.error_event(
|
||||
"Authorization: Bearer super-secret-token",
|
||||
event="auth.token.failed",
|
||||
context={
|
||||
"token": "plain-secret",
|
||||
"nested": {"password": "hunter2"},
|
||||
"safe": "visible",
|
||||
},
|
||||
)
|
||||
)
|
||||
finally:
|
||||
set_request_id(None)
|
||||
|
||||
assert "super-secret-token" not in output
|
||||
assert "plain-secret" not in output
|
||||
assert "hunter2" not in output
|
||||
assert "[REDACTED]" in output
|
||||
assert '"safe": "visible"' in output
|
||||
@@ -121,25 +121,6 @@ class TestCollectionTaskModel:
|
||||
)
|
||||
assert task.records_processed == 100
|
||||
|
||||
def test_task_with_phase_progress(self):
|
||||
"""Test collection task phase-level progress fields"""
|
||||
task = CollectionTask(
|
||||
datasource_id=1,
|
||||
status="running",
|
||||
phase="fetching",
|
||||
phase_progress=42.5,
|
||||
phase_message="Downloading dataset",
|
||||
phase_current=1024,
|
||||
phase_total=4096,
|
||||
phase_unit="bytes",
|
||||
)
|
||||
assert task.phase == "fetching"
|
||||
assert task.phase_progress == 42.5
|
||||
assert task.phase_message == "Downloading dataset"
|
||||
assert task.phase_current == 1024
|
||||
assert task.phase_total == 4096
|
||||
assert task.phase_unit == "bytes"
|
||||
|
||||
def test_task_error_message(self):
|
||||
"""Test collection task with error message"""
|
||||
task = CollectionTask(
|
||||
|
||||
@@ -1,217 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from app.services import system_logs
|
||||
|
||||
|
||||
class FakeRedis:
|
||||
def __init__(self) -> None:
|
||||
self.store: dict[str, list[str]] = {}
|
||||
|
||||
def rpush(self, key: str, value: str) -> None:
|
||||
self.store.setdefault(key, []).append(value)
|
||||
|
||||
def ltrim(self, key: str, start: int, end: int) -> None:
|
||||
items = self.store.get(key, [])
|
||||
normalized_end = None if end == -1 else end + 1
|
||||
self.store[key] = items[start:normalized_end]
|
||||
|
||||
def expire(self, key: str, seconds: int) -> None:
|
||||
return None
|
||||
|
||||
def lrange(self, key: str, start: int, end: int) -> list[str]:
|
||||
items = self.store.get(key, [])
|
||||
normalized_end = None if end == -1 else end + 1
|
||||
return items[start:normalized_end]
|
||||
|
||||
def llen(self, key: str) -> int:
|
||||
return len(self.store.get(key, []))
|
||||
|
||||
|
||||
def test_read_log_snapshot_uses_structured_buffer_timestamp_level_and_search(monkeypatch):
|
||||
fake_redis = FakeRedis()
|
||||
monkeypatch.setattr(system_logs, "redis_client", fake_redis)
|
||||
monkeypatch.setattr(
|
||||
system_logs,
|
||||
"LOG_SOURCES",
|
||||
{
|
||||
"earth-client": system_logs.LogSource(
|
||||
source_id="earth-client",
|
||||
name="Earth 浏览器端",
|
||||
kind="buffer",
|
||||
location="redis://planet:system_logs:earth-client",
|
||||
description="Earth 浏览器端上报日志",
|
||||
category="client",
|
||||
buffer_key=system_logs.get_buffer_log_key("earth-client"),
|
||||
)
|
||||
},
|
||||
)
|
||||
|
||||
fake_redis.rpush(
|
||||
system_logs.get_buffer_log_key("earth-client"),
|
||||
json.dumps(
|
||||
{
|
||||
"timestamp": "2026-04-22T10:15:30Z",
|
||||
"level": "warning",
|
||||
"message": "news feed degraded",
|
||||
"context": {"module": "news", "detail": "timeout"},
|
||||
},
|
||||
ensure_ascii=False,
|
||||
),
|
||||
)
|
||||
fake_redis.rpush(
|
||||
system_logs.get_buffer_log_key("earth-client"),
|
||||
json.dumps(
|
||||
{
|
||||
"timestamp": "2026-04-23T06:01:00Z",
|
||||
"level": "error",
|
||||
"message": "landing points failed",
|
||||
"context": {"module": "layer-startup", "detail": "http 500"},
|
||||
},
|
||||
ensure_ascii=False,
|
||||
),
|
||||
)
|
||||
|
||||
snapshot = system_logs.read_log_snapshot(
|
||||
"earth-client",
|
||||
50,
|
||||
levels="error,warning",
|
||||
start_date="2026-04-23",
|
||||
end_date="2026-04-23",
|
||||
search="landing",
|
||||
)
|
||||
|
||||
assert snapshot is not None
|
||||
assert snapshot["selected_levels"] == ["error", "warning"]
|
||||
assert snapshot["search_query"] == "landing"
|
||||
assert snapshot["line_count"] == 1
|
||||
assert snapshot["lines"][0].startswith("2026-04-23 06:01:00 ERROR landing points failed")
|
||||
assert snapshot["daily_markers"] == [
|
||||
{"date_token": "2026-04-23", "total": 1, "dominant_level": "error"}
|
||||
]
|
||||
|
||||
|
||||
def test_read_log_snapshot_parses_file_timestamp_and_builds_markers(tmp_path: Path, monkeypatch):
|
||||
log_path = tmp_path / "backend.log"
|
||||
log_path.write_text(
|
||||
"\n".join(
|
||||
[
|
||||
"2026-04-22 08:00:00 INFO service booted",
|
||||
"2026-04-23 09:15:00 WARNING disk pressure detected",
|
||||
"2026-04-23 09:16:00 ERROR sync failed",
|
||||
"2026-04-24 10:00:00 DEBUG collector trace",
|
||||
]
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
monkeypatch.setattr(
|
||||
system_logs,
|
||||
"LOG_SOURCES",
|
||||
{
|
||||
"backend": system_logs.LogSource(
|
||||
source_id="backend",
|
||||
name="后端服务",
|
||||
kind="file",
|
||||
location=str(log_path),
|
||||
description="测试文件日志",
|
||||
category="service",
|
||||
)
|
||||
},
|
||||
)
|
||||
|
||||
snapshot = system_logs.read_log_snapshot(
|
||||
"backend",
|
||||
50,
|
||||
levels="warning,error",
|
||||
search="failed",
|
||||
)
|
||||
|
||||
assert snapshot is not None
|
||||
assert snapshot["line_count"] == 1
|
||||
assert snapshot["lines"] == ["2026-04-23 09:16:00 ERROR sync failed"]
|
||||
assert snapshot["daily_markers"] == [
|
||||
{"date_token": "2026-04-23", "total": 1, "dominant_level": "error"}
|
||||
]
|
||||
assert snapshot["status"] == "ok"
|
||||
|
||||
|
||||
def test_append_buffer_log_persists_normalized_level(monkeypatch):
|
||||
fake_redis = FakeRedis()
|
||||
monkeypatch.setattr(system_logs, "redis_client", fake_redis)
|
||||
|
||||
system_logs.append_buffer_log(
|
||||
"earth-client",
|
||||
level="warn",
|
||||
message="feed delayed",
|
||||
context={"module": "news"},
|
||||
)
|
||||
|
||||
stored_items = fake_redis.lrange(system_logs.get_buffer_log_key("earth-client"), 0, -1)
|
||||
payload = json.loads(stored_items[0])
|
||||
assert payload["level"] == "warning"
|
||||
assert payload["message"] == "feed delayed"
|
||||
|
||||
|
||||
def test_infer_log_level_prefers_leading_prefix_over_query_string():
|
||||
line = 'INFO: 127.0.0.1 - "GET /api/v1/system/logs/backend?limit=200&level=error&levels=error HTTP/1.1" 200 OK'
|
||||
|
||||
entry = system_logs.parse_text_log_entry(line)
|
||||
|
||||
assert entry.level == "info"
|
||||
|
||||
|
||||
def test_parse_text_log_entry_does_not_promote_exception_context_to_error():
|
||||
line = "websockets.exceptions.ConnectionClosedError: sent 1011 (internal error) keepalive ping timeout"
|
||||
|
||||
entry = system_logs.parse_text_log_entry(line)
|
||||
|
||||
assert entry.level is None
|
||||
|
||||
|
||||
def test_parse_text_log_entry_still_detects_explicit_error_prefix():
|
||||
line = "ERROR: [Errno 98] Address already in use"
|
||||
|
||||
entry = system_logs.parse_text_log_entry(line)
|
||||
|
||||
assert entry.level == "error"
|
||||
|
||||
|
||||
def test_read_log_snapshot_strips_nul_bytes_from_file_lines(tmp_path: Path, monkeypatch):
|
||||
log_path = tmp_path / "backend.log"
|
||||
log_path.write_bytes(
|
||||
(
|
||||
b"INFO: service booted\n"
|
||||
b"ERROR: bind failed\n"
|
||||
+ b"\x00" * 32
|
||||
+ b"2026-04-23 23:41:32 INFO service=backend message=request served\n"
|
||||
)
|
||||
)
|
||||
|
||||
monkeypatch.setattr(
|
||||
system_logs,
|
||||
"LOG_SOURCES",
|
||||
{
|
||||
"backend": system_logs.LogSource(
|
||||
source_id="backend",
|
||||
name="后端服务",
|
||||
kind="file",
|
||||
location=str(log_path),
|
||||
description="测试文件日志",
|
||||
category="service",
|
||||
)
|
||||
},
|
||||
)
|
||||
|
||||
snapshot = system_logs.read_log_snapshot("backend", 50)
|
||||
|
||||
assert snapshot is not None
|
||||
assert snapshot["line_count"] == 3
|
||||
assert snapshot["lines"] == [
|
||||
"INFO: service booted",
|
||||
"ERROR: bind failed",
|
||||
"2026-04-23 23:41:32 INFO service=backend message=request served",
|
||||
]
|
||||
@@ -1,149 +0,0 @@
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import pytest
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
|
||||
from app.api.v1.visualization import convert_vessels_to_geojson
|
||||
from app.db.session import get_db
|
||||
from app.main import app
|
||||
from app.models.vessel import VesselPosition, VesselStatic
|
||||
from app.services import barentswatch
|
||||
from app.services.collectors.vessel_ais import VesselAISCollector
|
||||
|
||||
|
||||
def test_vessel_collector_transforms_barentswatch_like_records():
|
||||
collector = VesselAISCollector()
|
||||
records = collector.transform(
|
||||
[
|
||||
{
|
||||
"mmsi": "257123000",
|
||||
"lat": "59.91",
|
||||
"lon": "10.73",
|
||||
"sog": 12.4,
|
||||
"cog": 214,
|
||||
"nav_status": 0,
|
||||
"shipType": 70,
|
||||
"name": "OSLO TRADER",
|
||||
},
|
||||
{"mmsi": "bad", "lat": 120, "lon": 10},
|
||||
]
|
||||
)
|
||||
|
||||
assert len(records) == 1
|
||||
assert records[0]["mmsi"] == 257123000
|
||||
assert records[0]["vessel_type_name"] == "Cargo"
|
||||
assert records[0]["lat"] == pytest.approx(59.91)
|
||||
|
||||
|
||||
def test_barentswatch_reads_credentials_from_zshrc(tmp_path):
|
||||
zshrc = tmp_path / ".zshrc"
|
||||
zshrc.write_text(
|
||||
"\n".join(
|
||||
[
|
||||
"export BARENTSWATCH_CLIENT_ID='client-from-zshrc'",
|
||||
'export BARENTSWATCH_CLIENT_SECRET="secret-from-zshrc" # local dev credential',
|
||||
]
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
values = barentswatch._read_zshrc_env(zshrc)
|
||||
|
||||
assert values["BARENTSWATCH_CLIENT_ID"] == "client-from-zshrc"
|
||||
assert values["BARENTSWATCH_CLIENT_SECRET"] == "secret-from-zshrc"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_barentswatch_resolves_config_from_zshrc(tmp_path, monkeypatch):
|
||||
zshrc = tmp_path / ".zshrc"
|
||||
zshrc.write_text(
|
||||
"\n".join(
|
||||
[
|
||||
"export BARENTSWATCH_CLIENT_ID=client-from-zshrc",
|
||||
"export BARENTSWATCH_CLIENT_SECRET=secret-from-zshrc",
|
||||
]
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
monkeypatch.delenv("BARENTSWATCH_CLIENT_ID", raising=False)
|
||||
monkeypatch.delenv("BARENTSWATCH_CLIENT_SECRET", raising=False)
|
||||
monkeypatch.delenv("BARRENTSWATCH_CLIENT_ID", raising=False)
|
||||
monkeypatch.delenv("BARRENTSWATCH_CLIENT_SECRET", raising=False)
|
||||
monkeypatch.setattr(barentswatch.Path, "home", lambda: tmp_path)
|
||||
|
||||
config = await barentswatch.resolve_barentswatch_config(None)
|
||||
|
||||
assert config.client_id == "client-from-zshrc"
|
||||
assert config.client_secret == "secret-from-zshrc"
|
||||
assert config.credential_source == "~/.zshrc"
|
||||
|
||||
|
||||
def test_convert_vessels_to_geojson():
|
||||
position = VesselPosition(
|
||||
mmsi=257123000,
|
||||
lat=59.91,
|
||||
lon=10.73,
|
||||
sog=12.4,
|
||||
cog=214,
|
||||
heading=215,
|
||||
nav_status=0,
|
||||
received_at=datetime(2026, 4, 28, 1, 0, tzinfo=timezone.utc),
|
||||
)
|
||||
static = VesselStatic(
|
||||
mmsi=257123000,
|
||||
name="OSLO TRADER",
|
||||
vessel_type=70,
|
||||
vessel_type_name="Cargo",
|
||||
flag="NO",
|
||||
length=185,
|
||||
)
|
||||
|
||||
payload = convert_vessels_to_geojson([(position, static)])
|
||||
|
||||
assert payload["type"] == "FeatureCollection"
|
||||
assert payload["features"][0]["geometry"]["coordinates"] == [10.73, 59.91]
|
||||
assert payload["features"][0]["properties"]["mmsi"] == 257123000
|
||||
assert payload["features"][0]["properties"]["vessel_type_name"] == "Cargo"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_vessels_geojson_endpoint_filters_type_and_bbox():
|
||||
now = datetime(2026, 4, 28, 1, 0, tzinfo=timezone.utc)
|
||||
rows = [
|
||||
(
|
||||
VesselPosition(mmsi=1, lat=59.9, lon=10.7, received_at=now),
|
||||
VesselStatic(mmsi=1, name="Cargo Ship", vessel_type=70, vessel_type_name="Cargo"),
|
||||
),
|
||||
(
|
||||
VesselPosition(mmsi=2, lat=60.3, lon=5.3, received_at=now - timedelta(minutes=1)),
|
||||
VesselStatic(mmsi=2, name="Passenger Ship", vessel_type=60, vessel_type_name="Passenger"),
|
||||
),
|
||||
]
|
||||
|
||||
class _Result:
|
||||
def all(self):
|
||||
return rows
|
||||
|
||||
class _FakeSession:
|
||||
async def execute(self, _query):
|
||||
return _Result()
|
||||
|
||||
async def override_get_db():
|
||||
yield _FakeSession()
|
||||
|
||||
app.dependency_overrides[get_db] = override_get_db
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get(
|
||||
"/api/v1/visualization/geo/vessels",
|
||||
params={"bbox": "0,50,20,70", "type": "cargo"},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["count"] == 1
|
||||
assert data["features"][0]["properties"]["name"] == "Cargo Ship"
|
||||
assert data["stats"]["by_type"]["Cargo"] == 1
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
@@ -1,347 +0,0 @@
|
||||
from datetime import datetime, timezone
|
||||
|
||||
import pytest
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
|
||||
from app.api.v1.visualization import convert_compute_centers_to_geojson
|
||||
from app.db.session import get_db
|
||||
from app.main import app
|
||||
from app.models.collected_data import CollectedData
|
||||
|
||||
|
||||
def _build_record(
|
||||
*,
|
||||
record_id: int,
|
||||
source: str,
|
||||
data_type: str,
|
||||
name: str,
|
||||
country: str,
|
||||
city: str,
|
||||
latitude: float,
|
||||
longitude: float,
|
||||
metadata: dict,
|
||||
):
|
||||
return CollectedData(
|
||||
id=record_id,
|
||||
source=source,
|
||||
data_type=data_type,
|
||||
source_id=f"{source}-{record_id}",
|
||||
name=name,
|
||||
extra_data={
|
||||
"country": country,
|
||||
"city": city,
|
||||
"latitude": latitude,
|
||||
"longitude": longitude,
|
||||
**metadata,
|
||||
},
|
||||
collected_at=datetime(2026, 4, 22, tzinfo=timezone.utc),
|
||||
reference_date=datetime(2026, 4, 21, tzinfo=timezone.utc),
|
||||
is_current=True,
|
||||
)
|
||||
|
||||
|
||||
def test_convert_compute_centers_to_geojson_unifies_sources():
|
||||
top500_record = _build_record(
|
||||
record_id=1,
|
||||
source="top500",
|
||||
data_type="supercomputer",
|
||||
name="Frontier",
|
||||
country="United States",
|
||||
city="Oak Ridge",
|
||||
latitude=35.93,
|
||||
longitude=-84.31,
|
||||
metadata={
|
||||
"rank": 1,
|
||||
"manufacturer": "HPE",
|
||||
"organization": "ORNL",
|
||||
"rmax": 1102000.0,
|
||||
"cores": 8730112,
|
||||
"power": 21510.0,
|
||||
},
|
||||
)
|
||||
gpu_record = _build_record(
|
||||
record_id=2,
|
||||
source="epoch_ai_gpu",
|
||||
data_type="gpu_cluster",
|
||||
name="Colossus",
|
||||
country="United States",
|
||||
city="Memphis",
|
||||
latitude=35.15,
|
||||
longitude=-90.05,
|
||||
metadata={
|
||||
"organization": "xAI",
|
||||
"gpu_type": "H100",
|
||||
"gpu_count": 100000,
|
||||
"value": "20000",
|
||||
"unit": "TFlop/s",
|
||||
},
|
||||
)
|
||||
|
||||
payload = convert_compute_centers_to_geojson([top500_record, gpu_record])
|
||||
|
||||
assert payload["type"] == "FeatureCollection"
|
||||
assert len(payload["features"]) == 2
|
||||
|
||||
supercomputer_feature = payload["features"][0]
|
||||
assert supercomputer_feature["properties"]["site_type"] == "supercomputer"
|
||||
assert supercomputer_feature["properties"]["capacity_unit"] == "GFlops"
|
||||
assert supercomputer_feature["properties"]["capacity_band"] == "exascale"
|
||||
assert supercomputer_feature["properties"]["operator"] == "ORNL"
|
||||
assert supercomputer_feature["properties"]["location_precision"] == "precise"
|
||||
assert supercomputer_feature["properties"]["is_estimated"] is False
|
||||
|
||||
gpu_feature = payload["features"][1]
|
||||
assert gpu_feature["properties"]["site_type"] == "gpu_cluster"
|
||||
assert gpu_feature["properties"]["vendor"] == "H100"
|
||||
assert gpu_feature["properties"]["gpu_count"] == 100000
|
||||
assert gpu_feature["properties"]["capacity_band"] == "large"
|
||||
assert gpu_feature["properties"]["location_precision"] == "precise"
|
||||
|
||||
|
||||
def test_convert_compute_centers_to_geojson_uses_coordinate_hints():
|
||||
hinted_record = _build_record(
|
||||
record_id=3,
|
||||
source="top500",
|
||||
data_type="supercomputer",
|
||||
name="Frontier",
|
||||
country="United States",
|
||||
city="",
|
||||
latitude=0.0,
|
||||
longitude=0.0,
|
||||
metadata={
|
||||
"organization": "Oak Ridge National Laboratory",
|
||||
"rmax": 1102000.0,
|
||||
},
|
||||
)
|
||||
|
||||
payload = convert_compute_centers_to_geojson([hinted_record])
|
||||
|
||||
assert len(payload["features"]) == 1
|
||||
coords = payload["features"][0]["geometry"]["coordinates"]
|
||||
assert coords[0] == pytest.approx(-84.3107)
|
||||
assert coords[1] == pytest.approx(35.9319)
|
||||
assert payload["features"][0]["properties"]["is_estimated"] is True
|
||||
assert payload["features"][0]["properties"]["location_precision"] == "estimated_site"
|
||||
|
||||
|
||||
def test_convert_compute_centers_to_geojson_falls_back_to_country_centroid():
|
||||
centroid_record = _build_record(
|
||||
record_id=4,
|
||||
source="epoch_ai_gpu",
|
||||
data_type="gpu_cluster",
|
||||
name="Unknown Cluster",
|
||||
country="United States",
|
||||
city="",
|
||||
latitude=0.0,
|
||||
longitude=0.0,
|
||||
metadata={
|
||||
"organization": "Unknown Operator",
|
||||
"value": "10000",
|
||||
"unit": "TFlop/s",
|
||||
},
|
||||
)
|
||||
|
||||
payload = convert_compute_centers_to_geojson([centroid_record])
|
||||
|
||||
assert len(payload["features"]) == 1
|
||||
props = payload["features"][0]["properties"]
|
||||
coords = payload["features"][0]["geometry"]["coordinates"]
|
||||
assert coords[0] == pytest.approx(-98.5795)
|
||||
assert coords[1] == pytest.approx(39.8283)
|
||||
assert props["is_estimated"] is True
|
||||
assert props["location_precision"] == "estimated_country"
|
||||
assert props["geography_mode"] == "country_centroid"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_compute_centers_geojson_endpoint_returns_stats():
|
||||
records = [
|
||||
_build_record(
|
||||
record_id=1,
|
||||
source="top500",
|
||||
data_type="supercomputer",
|
||||
name="Frontier",
|
||||
country="United States",
|
||||
city="Oak Ridge",
|
||||
latitude=35.93,
|
||||
longitude=-84.31,
|
||||
metadata={"rank": 1, "rmax": 1102000.0},
|
||||
),
|
||||
_build_record(
|
||||
record_id=2,
|
||||
source="epoch_ai_gpu",
|
||||
data_type="gpu_cluster",
|
||||
name="Colossus",
|
||||
country="United States",
|
||||
city="Memphis",
|
||||
latitude=35.15,
|
||||
longitude=-90.05,
|
||||
metadata={"value": "20000", "unit": "TFlop/s"},
|
||||
),
|
||||
]
|
||||
|
||||
class _ScalarResult:
|
||||
def __init__(self, rows):
|
||||
self._rows = rows
|
||||
|
||||
def scalars(self):
|
||||
class _Scalars:
|
||||
def __init__(self, rows):
|
||||
self._rows = rows
|
||||
|
||||
def all(self):
|
||||
return self._rows
|
||||
|
||||
return _Scalars(self._rows)
|
||||
|
||||
class _FakeSession:
|
||||
async def execute(self, _query):
|
||||
return _ScalarResult(records)
|
||||
|
||||
async def override_get_db():
|
||||
yield _FakeSession()
|
||||
|
||||
app.dependency_overrides[get_db] = override_get_db
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/api/v1/visualization/geo/compute-centers")
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["count"] == 2
|
||||
assert data["stats"]["supercomputers"] == 1
|
||||
assert data["stats"]["gpu_clusters"] == 1
|
||||
assert data["features"][0]["properties"]["data_type"] == "compute_center"
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_visualization_geo_summary_returns_counts(monkeypatch):
|
||||
records = [
|
||||
_build_record(
|
||||
record_id=1,
|
||||
source="arcgis_cables",
|
||||
data_type="submarine_cable",
|
||||
name="Test Cable",
|
||||
country="",
|
||||
city="",
|
||||
latitude=0,
|
||||
longitude=0,
|
||||
metadata={
|
||||
"route_coordinates": [[[0, 0], [1, 1]]],
|
||||
"status": "active",
|
||||
},
|
||||
),
|
||||
_build_record(
|
||||
record_id=2,
|
||||
source="arcgis_landing_points",
|
||||
data_type="landing_point",
|
||||
name="Test Landing",
|
||||
country="United States",
|
||||
city="New York",
|
||||
latitude=40.7,
|
||||
longitude=-74.0,
|
||||
metadata={"city_id": 10},
|
||||
),
|
||||
_build_record(
|
||||
record_id=3,
|
||||
source="celestrak_tle",
|
||||
data_type="satellite_tle",
|
||||
name="TESTSAT",
|
||||
country="",
|
||||
city="",
|
||||
latitude=0,
|
||||
longitude=0,
|
||||
metadata={
|
||||
"norad_cat_id": 12345,
|
||||
"tle_line1": "1 12345U 98067A 24001.00000000 .00000000 00000-0 00000-0 0 9991",
|
||||
"tle_line2": "2 12345 51.6000 100.0000 0001000 10.0000 20.0000 15.50000000 01",
|
||||
},
|
||||
),
|
||||
_build_record(
|
||||
record_id=4,
|
||||
source="top500",
|
||||
data_type="supercomputer",
|
||||
name="Frontier",
|
||||
country="United States",
|
||||
city="Oak Ridge",
|
||||
latitude=35.93,
|
||||
longitude=-84.31,
|
||||
metadata={"rank": 1, "rmax": 1102000.0},
|
||||
),
|
||||
_build_record(
|
||||
record_id=5,
|
||||
source="epoch_ai_gpu",
|
||||
data_type="gpu_cluster",
|
||||
name="Colossus",
|
||||
country="United States",
|
||||
city="Memphis",
|
||||
latitude=35.15,
|
||||
longitude=-90.05,
|
||||
metadata={"value": "20000", "unit": "TFlop/s"},
|
||||
),
|
||||
]
|
||||
|
||||
class _ScalarResult:
|
||||
def __init__(self, rows=None, scalar_value=None):
|
||||
self._rows = rows or []
|
||||
self._scalar_value = scalar_value
|
||||
|
||||
def scalar(self):
|
||||
return self._scalar_value
|
||||
|
||||
def scalars(self):
|
||||
class _Scalars:
|
||||
def __init__(self, rows):
|
||||
self._rows = rows
|
||||
|
||||
def all(self):
|
||||
return self._rows
|
||||
|
||||
return _Scalars(self._rows)
|
||||
|
||||
class _FakeSession:
|
||||
async def execute(self, query):
|
||||
query_text = str(query)
|
||||
if "bgp_incidents" in query_text:
|
||||
return _ScalarResult(scalar_value=2)
|
||||
if "bgp_anomalies" in query_text:
|
||||
return _ScalarResult(scalar_value=3)
|
||||
return _ScalarResult(rows=records)
|
||||
|
||||
async def override_get_db():
|
||||
yield _FakeSession()
|
||||
|
||||
async def _fake_build_bgp_collector_coverage(*_args, **_kwargs):
|
||||
return [
|
||||
{"collector": "rrc00"},
|
||||
{"collector": "rrc01"},
|
||||
]
|
||||
|
||||
monkeypatch.setattr(
|
||||
"app.api.v1.visualization.build_bgp_collector_coverage",
|
||||
_fake_build_bgp_collector_coverage,
|
||||
)
|
||||
|
||||
app.dependency_overrides[get_db] = override_get_db
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/api/v1/visualization/geo/summary")
|
||||
|
||||
assert response.status_code == 200
|
||||
stats = response.json()["stats"]
|
||||
assert stats["cable_count"] == 1
|
||||
assert stats["landing_point_count"] == 1
|
||||
assert stats["satellite_count"] == 1
|
||||
assert stats["compute_center_count"] == 2
|
||||
assert stats["supercomputer_count"] == 1
|
||||
assert stats["gpu_cluster_count"] == 1
|
||||
assert stats["bgp_event_count"] == 2
|
||||
assert stats["bgp_incident_count"] == 2
|
||||
assert stats["bgp_anomaly_count"] == 3
|
||||
assert stats["bgp_collector_count"] == 2
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
@@ -86,12 +86,6 @@ CREATE TABLE collection_tasks (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
datasource_id INTEGER NOT NULL REFERENCES data_sources(id) ON DELETE CASCADE,
|
||||
status task_status NOT NULL DEFAULT 'pending',
|
||||
phase VARCHAR(30) DEFAULT 'queued',
|
||||
phase_progress FLOAT,
|
||||
phase_message VARCHAR(255),
|
||||
phase_current BIGINT,
|
||||
phase_total BIGINT,
|
||||
phase_unit VARCHAR(30),
|
||||
started_at TIMESTAMP WITH TIME ZONE,
|
||||
completed_at TIMESTAMP WITH TIME ZONE,
|
||||
records_processed INTEGER DEFAULT 0,
|
||||
|
||||
@@ -18,9 +18,6 @@ services:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: aiprovider/Dockerfile
|
||||
args:
|
||||
PYTHON_IMAGE: ${PYTHON_IMAGE:-python:3.14-slim}
|
||||
UV_IMAGE: ${UV_IMAGE:-ghcr.io/astral-sh/uv:latest}
|
||||
container_name: planet_aiprovider
|
||||
ports:
|
||||
- "8010:8010"
|
||||
|
||||
@@ -5,12 +5,6 @@ services:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: aiprovider/Dockerfile
|
||||
args:
|
||||
PYTHON_IMAGE: ${PYTHON_IMAGE:-python:3.14-slim}
|
||||
UV_IMAGE: ${UV_IMAGE:-ghcr.io/astral-sh/uv:latest}
|
||||
env_file:
|
||||
- ./aiprovider/.env
|
||||
- ${PLANET_AI_PROVIDER_RUNTIME_ENV_FILE:-./aiprovider/.env}
|
||||
container_name: planet_aiprovider
|
||||
ports:
|
||||
- "8010:8010"
|
||||
|
||||
@@ -5,12 +5,8 @@ services:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: aiprovider/Dockerfile
|
||||
args:
|
||||
PYTHON_IMAGE: ${PYTHON_IMAGE:-python:3.14-slim}
|
||||
UV_IMAGE: ${UV_IMAGE:-ghcr.io/astral-sh/uv:latest}
|
||||
env_file:
|
||||
- ./aiprovider/.env
|
||||
- ${PLANET_AI_PROVIDER_RUNTIME_ENV_FILE:-./aiprovider/.env}
|
||||
container_name: planet_aiprovider
|
||||
ports:
|
||||
- "8010:8010"
|
||||
|
||||
+11
-678
@@ -8,673 +8,6 @@ This project follows the repository versioning rule:
|
||||
- `improvement` -> `+0.0.1`(bugfix + 小功能混合)
|
||||
- `bugfix` -> `+0.0.1`
|
||||
|
||||
## [0.46.3] — 2026-04-30
|
||||
|
||||
Released: 2026-04-30
|
||||
|
||||
### 🐛 Fixes
|
||||
- 优化 Starlink footprint 显示后的地球拖拽性能,避免旋转地球时每帧重建 footprint 大网格,同时保持现有视觉效果不变。
|
||||
- 恢复点击线缆后的呼吸透明度动画,让 locked / hover 线缆重新使用既有 pulse 配置。
|
||||
|
||||
---
|
||||
|
||||
## [0.46.2] — 2026-04-30
|
||||
|
||||
Released: 2026-04-30
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复 Earth 启动时高清材质、云图和图层可见性绕过 `startupPriority` 的问题,统一由启动队列按文档顺序加载。
|
||||
- 修复保存为关闭的高清材质/图层仍会先加载再关闭的问题,并保持海陆基座作为国界线图层的常驻底图。
|
||||
- 修复搜索跳转会误关媒体面板、船只轨迹末端不贴合当前船只、Iridium footprint 被地表层遮挡等 Earth 交互问题。
|
||||
|
||||
### 📝 Documentation
|
||||
- 更新 Earth 图层顺序、样式参考、使用手册和 AIS 聚合计划,补齐中英文说明与后续接入策略。
|
||||
|
||||
---
|
||||
|
||||
## [0.46.1] — 2026-04-30
|
||||
|
||||
Released: 2026-04-30
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复新增 technical docs 文件存在但未进入 Docs 前端白名单时,侧栏不显示且 Markdown 链接无法解析到 `/docs/<slug>` 的问题。
|
||||
- 补齐数据源/采集器连接验证与 Earth Interactable 使用说明的英文文档,保证公开 Docs 切换 EN 时同名页面可访问。
|
||||
- 清理中英文 technical docs 中裸 `.md` 文件名链接标题,改为面向读者的语义标题。
|
||||
|
||||
### 📝 Documentation
|
||||
- 将 Docs 前端白名单、公开文档双语配对、裸文件名链接标题三项检查写入 Claude 与 Codex 的 docs 技能流程。
|
||||
|
||||
---
|
||||
|
||||
## [0.46.0] — 2026-04-30
|
||||
|
||||
Released: 2026-04-30
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 新增通用 Interactable 图标层,船只、算力中心、BGP 事件与观测站统一使用批量 Points、屏幕拾取、状态 glow 和状态缩放。
|
||||
- BGP 事件保留向外扩散圈,观测站保留雷达扫描层,并与 Interactable 主图标解耦到稳定的地表渲染层级。
|
||||
- 登陆点回归黄色球形 Sprite,贴近海缆层级并保持更稳定的地表显示和遮挡表现。
|
||||
|
||||
### 🔧 Improvements
|
||||
- 新增 SVG asset 到 canvas texture 的 Interactable 资产加载路径,支持统一图标资源、缓存和可选染色。
|
||||
- 同坐标 Interactable 自动做地表切向避让,降低重叠物件无法选择的问题。
|
||||
- 优化 Earth toolbar 初始尺寸注入,避免首次显示原始尺寸后再跳到缩放尺寸。
|
||||
- 补充 Interactable 计划、使用说明、图层顺序和 Earth 前端上下文文档。
|
||||
- 修复船只 hover/locked 状态仅发光但放大反馈不明显的问题,将已有状态缩放接入通用图标层。
|
||||
|
||||
---
|
||||
|
||||
## [0.45.0] — 2026-04-29
|
||||
|
||||
### ✨ Highlights
|
||||
- 采集任务新增阶段级量化进度,`fetching` 可展示百分比、阶段说明和字节下载量。
|
||||
- AI Provider 启动链路支持从 `aiprovider/.env` 与 `~/.zshrc` 注入运行期配置,并避免密钥/模型变化触发镜像重建。
|
||||
- AI Provider Docker build context 收敛到服务必需文件,`uv sync` 接入 BuildKit 缓存以减少重复下载。
|
||||
|
||||
### 🔧 Improvements
|
||||
- IPtoASN、OpenGeoFeed、NRO delegated 下载型采集器接入真实字节进度上报。
|
||||
- 数据源页、采集中任务弹窗和任务历史页展示阶段摘要,并在 tooltip 中保留完整进度细节。
|
||||
- 调整 Earth 船只默认高度偏移,进一步贴近地表展示。
|
||||
|
||||
---
|
||||
|
||||
## [0.44.2] — 2026-04-29
|
||||
|
||||
### 📝 Documentation
|
||||
- 补充 Earth 船只图层技术文档,记录分桶 `THREE.Points` 批量渲染、同尺寸交互 overlay 和屏幕空间 picking 的设计约束。
|
||||
- 同步 Earth 渲染图层顺序和样式参考,明确 AIS 船只 renderOrder、depthTest、图标尺寸、航向分桶与 hover 命中半径。
|
||||
- 更新船只渲染性能计划状态,标注 `0.44.1` 已落地的实现与后续全球 AIS / LOD 演进方向。
|
||||
|
||||
---
|
||||
|
||||
## [0.44.1] — 2026-04-29
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复 Earth 船只图层拖动不跟手的问题,将普通船只从独立 Sprite 切换为按航向分桶的批量 Points 渲染。
|
||||
- 修正船只 hover/click 拾取错位,改为屏幕空间命中检测并在拖拽/惯性期间跳过 hover 拾取。
|
||||
- 统一 AIS 船只普通态与交互态方向,并让 hover/locked glow 与普通图标保持同尺寸覆盖。
|
||||
- 恢复船只深度测试并收敛默认图标尺寸,避免北部岛屿/冰面附近出现明显压盖陆地的视觉问题。
|
||||
|
||||
---
|
||||
|
||||
## [0.44.0] — 2026-04-29
|
||||
|
||||
### ✨ Highlights
|
||||
- 重构数据源与采集器设置边界:数据源页回归目录和采集触发,采集器 endpoint、请求头、凭证与连接验证统一进入设置页
|
||||
- BarentsWatch AIS 完整接入凭证解析、连接检查、默认教程、AI 生成教程和船只采集/可视化链路
|
||||
- Earth 新增船只图例、缩放反馈胶囊、缩放感知拖拽灵敏度,并将船只渲染性能优化方案沉淀到 plans
|
||||
- 仪表盘重启服务新增前端重启 action,并让 runner 通过 `~/.zshrc` 继承本地环境变量
|
||||
|
||||
### 🔧 Improvements
|
||||
- 采集器连接状态改为基于成功采集或手动连接校验 checksum 判断,避免只依赖前端样式状态
|
||||
- 数据源页新增采集中任务标签和任务进度弹窗,内置与自定义数据源统一展示
|
||||
- Earth 国界线进一步贴近地表,并补充船只图层渲染顺序、样式和用户手册说明
|
||||
- docs skill 与 Claude/Codex 文档流程补齐技术文档和 plans 的职责边界
|
||||
|
||||
---
|
||||
|
||||
## [0.43.1] — 2026-04-28
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修正 `planet.sh` 在全量 restart 后启动 AI Provider 时的提示语义,避免把预期内未就绪描述成异常不健康
|
||||
|
||||
---
|
||||
|
||||
## [0.43.0] — 2026-04-28
|
||||
|
||||
### ✨ Highlights
|
||||
- 新增 Earth 船舶追踪链路,接入 BarentsWatch AIS 凭证配置、采集器、后端 vessel 模型/API 与前端 Earth 船舶图层
|
||||
- 新增自定义数据源映射流程,支持样本抓取、目标 schema、AI 辅助生成映射、预览校验和映射执行
|
||||
- Settings 拆分 AI Provider 与采集器凭证配置,DataSources 只保留采集状态、运行参数和必要引导
|
||||
|
||||
### 🔧 Improvements
|
||||
- AI Provider 支持运行时 LLM 配置、provider preset 下拉与刷新,并在 Playground 中引导到 AI 配置页
|
||||
- Markdown 渲染器补齐代码块复制按钮、语言标签、任务列表、图片、自动链接、删除线和文档主题样式
|
||||
- Docs 公开导航改为显式元数据白名单,避免开发任务文档自动出现在“其他”分组
|
||||
- 将 Codex/Claude cleanup、docs、goal-driven、release 流程补充 CLI-first 约束,并把 `rules.md` 整理成可按模块加载的工程规则
|
||||
|
||||
---
|
||||
|
||||
## [0.42.2] — 2026-04-28
|
||||
|
||||
### 🐛 Fixes
|
||||
- Docs 中文模式下补齐左侧分组、文档标题、页头分类与搜索结果分类翻译,并更新文档站品牌标题/副标题文案
|
||||
|
||||
---
|
||||
|
||||
## [0.42.1] — 2026-04-28
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修正 release skill 的 feature 版本计算规则:minor 进位时 patch 必须重置为 `0`,例如 `0.41.2` 应发布为 `0.42.0`
|
||||
|
||||
---
|
||||
|
||||
## [0.42.0] — 2026-04-28
|
||||
|
||||
### ✨ Highlights
|
||||
- 新增公开 `/docs` 文档站,支持中英文技术文档、使用手册、Quickstart、搜索、目录锚点与浅色/深色/跟随系统主题
|
||||
- Earth 在无高清材质时新增轻量 Fresnel 边缘提示,并调整卫星覆盖默认显示与地表材质可读性
|
||||
|
||||
### 🔧 Improvements
|
||||
- 将技术文档整理为 `docs/technical/zh` 与 `docs/technical/en`,并补充控制台、`planet.sh`、Earth 与公共组件使用说明
|
||||
- 新增 `SegmentedControl` 公共滑块组件,支持缩放参数,复用到 docs 语言与主题切换
|
||||
- Markdown 渲染器接入自定义滚动条,表格与代码块在深色模式和 overflow 场景下保持可读
|
||||
- Docs 搜索结果支持内部滚动、点击外部关闭、重新聚焦恢复上次搜索结果
|
||||
- Earth 工具栏展开状态与设置持久化版本迁移继续收口,改善默认面板和快捷关闭行为
|
||||
|
||||
---
|
||||
|
||||
## [0.41.2] — 2026-04-27
|
||||
|
||||
### 🔧 Improvements
|
||||
- `planet.sh` 启动链路新增 verbose 滚动输出窗口,并在后端端口占用时打印目标地址和监听进程诊断
|
||||
- Docker 构建支持通过 build args 覆盖 Python 与 uv 镜像,方便 Docker Hub 不稳定时切换镜像源
|
||||
|
||||
### 🐛 Fixes
|
||||
- Earth 海缆登陆点改为基于相机射线与地球遮挡判断可见性,修复旋转后 pin 可见性滞后一帧的问题
|
||||
|
||||
### 🔧 Improvements
|
||||
- `docker-compose*.yml` 为 AI Provider 构建传入 `PYTHON_IMAGE` / `UV_IMAGE` 参数,默认仍使用官方镜像
|
||||
- 后端启动失败遇到 `Address already in use` 时输出 `lsof`、`ss` 与 PID 命令行信息
|
||||
- verbose 模式下 AI Provider build、后端与前端启动日志会在 spinner 下方保留最新 5 行滚动展示
|
||||
|
||||
---
|
||||
|
||||
## [0.41.1] — 2026-04-27
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复新闻直播面板设置项持久化失效:`closeTransientMobileOverlays` 通过旁路路径隐藏面板导致下次 persist 快照到错误状态,改为不重新从 DOM 读取面板可见性
|
||||
- 修复登陆点 pin 在地球侧面被半截遮挡:改为在接近地平线前(dot < 0.05)主动隐藏,避免深度测试切片
|
||||
|
||||
### 🔧 Improvements
|
||||
- 将所有画布绘制的图标抽取为 SVG,存入 `frontend/public/earth/assets/icons/`,新增图标规范到 `rules.md`
|
||||
|
||||
---
|
||||
|
||||
## [0.41.0] — 2026-04-27
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 图层系统完成地表到天空的注册顺序与关注优先的面板顺序拆分,支持基座海陆色块、国界、高清材质、云图、地形、算力、BGP、卫星、轨迹与海缆的稳定层级
|
||||
- 国界层新增真实行政区轮廓交互与中国/台湾联动高亮,修复高清材质、地形、footprint、卫星与经纬线之间的遮挡和 hover 竞争
|
||||
|
||||
### 🔧 Improvements
|
||||
- 新增无轮廓基座地图,所有图层关闭时仍保留 `#010609` 海洋与 `#080f1b` 陆地色块
|
||||
- 将大气云图抽象为独立图层并接入桌面/移动端图层开关、持久化状态与启动同步
|
||||
- 高清材质改为独立纹理覆盖层,地形显示在高清材质上方,并在高清材质关闭/恢复时保持原地形开关意图
|
||||
- 补充 Earth 渲染层级与图层样式文档,记录正式图层名、变量名、材质颜色、线宽与 renderOrder
|
||||
|
||||
---
|
||||
|
||||
## [0.40.5] — 2026-04-26
|
||||
|
||||
### 🔧 Improvements
|
||||
- 卫星拖尾改用 Instanced screen-space ribbon,单 draw call 渲染所有轨迹段,支持像素级宽度控制
|
||||
- Iridium 地面覆盖重写为球面投影径向网格,修复填充光晕不可见问题;新增外圈 LineLoop
|
||||
- 搜索面板打开时改用双 rAF 延迟聚焦输入框,确保 CSS 过渡完成后焦点可靠触发
|
||||
- 代码清理:提取 `IRIDIUM_OVERLAY_COLOR`、`IRIDIUM_REFERENCE_ALTITUDE_KM` 常量,消除重复三角函数调用
|
||||
|
||||
---
|
||||
|
||||
## [0.39.0] — 2026-04-24
|
||||
|
||||
## [0.40.4] — 2026-04-26
|
||||
|
||||
### 🔧 Improvements
|
||||
- 新增页面可见性恢复处理,页面从后台切回前台时主动刷新卫星位置,避免累积后台时间在下一帧一次性回放
|
||||
- 抽出卫星轨迹状态与轨迹几何清理 helper,统一后台恢复与清空数据时的轨迹重置路径
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复页面在后台停留较久后恢复前台时,卫星轨迹因超大 `deltaTime` 突然跳变、拖尾异常拉长的问题
|
||||
- 修复后台恢复后首帧仍沿用旧轨迹缓存,导致轨迹与当前卫星位置短时错位的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.40.3] — 2026-04-25
|
||||
|
||||
### 🔧 Improvements
|
||||
- 卫星点云升级为自定义 ShaderMaterial,支持 per-point alpha 控制,锁定/悬停卫星从点云中精确隐藏
|
||||
- 修复锁定环与自发光选中标记的 depthTest 错误(false → true),消除远端渲染穿透 artifact
|
||||
- 新增锁定环悬停态缩放与线宽(LOCKED_RING_HOVER_SCALE / LOCKED_RING_HOVER_LINE_WIDTH)
|
||||
- 修复 updateLockedDotWorldTransform / updateLockedHaloWorldTransform 未强制刷新 matrixWorld 导致的位置漂移
|
||||
|
||||
---
|
||||
|
||||
## [0.40.2] — 2026-04-24
|
||||
|
||||
### 🔧 Improvements
|
||||
- 卫星点大小随镜头缩放动态调整,拉近变大、拉远变小,响应与相机距离线性对应
|
||||
- 调小卫星点默认基础尺寸(dotSize 2.8),缩放范围更合理
|
||||
|
||||
---
|
||||
|
||||
## [0.40.1] — 2026-04-24
|
||||
|
||||
### 🔧 Improvements
|
||||
- 卫星选中标记(lockedring / lockeddot / 光晕)颜色统一跟随图例轨道倾角分类配色
|
||||
- 修复 Starlink footprint 在特定视角下遮蔽卫星点的渲染顺序问题(Group renderOrder 影响子 Mesh 排序)
|
||||
- footprint 材质改为 `depthTest: false` + 相机朝向 limbFade,替代 polygonOffset 深度竞争方案
|
||||
- 修复选中海缆时误触发附近卫星高亮(该行为属于 BGP 事件点逻辑,不应用于海缆)
|
||||
|
||||
---
|
||||
|
||||
## [0.40.0] — 2026-04-24
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 卫星 footprint 正式按星座能力分层:Starlink 保留专用地表覆盖,Iridium 改为独立外圈覆盖表达,其它非 Starlink 星座不再误用同一套 footprint
|
||||
- Earth 卫星详情卡补齐覆盖能力与当前显示说明,用户现在可以直接看见每颗卫星为什么显示 footprint、为何回退为自身发光
|
||||
|
||||
### 🔧 Improvements
|
||||
- 后端可视化接口新增并透传 `constellation_group` 与 `footprint_policy`,前端据此执行 capability-gated footprint renderer
|
||||
- 新增 Iridium 独立 coverage ring adapter,并继续保留 Starlink 专用 footprint 调校与昼夜可读性增强
|
||||
- 新增 Earth 卫星 footprint 策略技术文档,明确 GNSS、generic LEO、GEO 与 Iridium 的显示边界
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复前后端对 Iridium footprint policy 命名不一致,导致策略分发语义含混的问题
|
||||
- 清理 Starlink footprint 渲染中的未使用常量与过时命名,减少后续继续调校时的歧义
|
||||
|
||||
---
|
||||
|
||||
## [0.39.0] — 2026-04-24
|
||||
|
||||
### ✨ Highlights
|
||||
- 后端正式落下统一结构化日志地基:请求上下文、事件名、脱敏与持久化链路开始收口为可扩展的企业级日志体系
|
||||
- 系统日志页重构为真正的日志工作台:顶部筛选更紧凑,终端日志区成为主视觉,移动端 Earth 新闻/态势细节交互继续补稳
|
||||
|
||||
### 🔧 Improvements
|
||||
- 新增 `backend/app/core/logging.py`,统一 `request_id`、`service`、`event` 注入与敏感字段脱敏,并接入后端主入口、调度器、缓存、数据库和可视化链路
|
||||
- 系统日志页筛选区重排为更紧凑的两层结构,信息摘要并入终端工具栏 tooltip,日志终端区留出更稳定的按钮避让空间
|
||||
- Earth 移动端态势抽屉补齐宽度约束与图例换行规则,新闻详情抽屉在巡航切换时可同步更新标题和摘要
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复 `/tmp/planet_backend.log` 中混入空字节时,日志摘要条行数与实际可见日志不一致的问题
|
||||
- 修复移动端“态势”tab 在内容渲染后被图例文本撑宽、超出一屏的问题
|
||||
- 修复移动端新闻详情抽屉在巡航切换下一条新闻时标题更新但 summary 不同步的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.38.0] — 2026-04-23
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 新闻正式接入通用巡航层:新闻和 BGP 统一进入可配置巡航模块,桌面端与移动端都能在巡航聚焦时展示对应新闻卡片
|
||||
- 系统日志页升级为结构化过滤链路:按真实时间戳、结构化级别和字符串检索统一筛选,不再依赖前端或后端从日志文本里猜结果
|
||||
|
||||
### 🔧 Improvements
|
||||
- 新闻巡航补齐业务适配层:按发生地与时间生成巡航目标,桌面端与移动端统一标题 + summary 卡片风格,并增加连线与打字机摘要展示
|
||||
- 日志页筛选体验重排,统一服务源、级别、行数、时间和检索布局,日历标记改为由后端返回的结构化每日聚合结果驱动
|
||||
- 后端补充 `system_logs` 结构化解析与多级别精确过滤能力,Earth 浏览器端日志缓冲与系统日志 API 现在走同一套筛选语义
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复新闻巡航模块开启后难以关闭、桌面/移动端设置状态互相污染的问题
|
||||
- 修复新闻巡航卡片缺少摘要、移动端详情样式不统一、新闻巡航缺少连线的问题
|
||||
- 修复日志级别筛选会被访问日志 query string 中的 `level=error` 等参数污染,从而把 `INFO` 行误判为 `ERROR` 的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.37.2] — 2026-04-23
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 图层系统新增经纬线开关,桌面图层面板与移动端抽屉都可直接控制
|
||||
|
||||
### 🔧 Improvements
|
||||
- 经纬线正式接入 Earth layer registry,复用现有图层切换、移动端图层卡片与设置持久化流
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复经纬线只能默认常驻、无法作为独立图层开关控制的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.37.1] — 2026-04-23
|
||||
|
||||
### ✨ Highlights
|
||||
- `planet.sh` 后端重启链路修复 `uvicorn --reload` 残留 worker 场景,`restart` 现在能真正替换旧实例
|
||||
|
||||
### 🔧 Improvements
|
||||
- 收口后端清理逻辑,统一按 `uvicorn` 进程、端口占用进程和进程组执行清理,减少 reload 场景漏杀分支
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复部分机器执行 `./planet.sh restart --allow-lan` 后后端仍停留旧实例,导致 `/api/v1/visualization/geo/compute-centers` 返回 `404` 的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.35.1] — 2026-04-22
|
||||
## [0.37.0] — 2026-04-23
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 连线系统正式从巡航里解耦成通用 callout connector:桌面端和移动端统一支持对象级锚点、四边切换与临界区边缘滑动
|
||||
- BGP 巡航展示继续收口为稳定的“先定位卡片、再连真实锚点、再展示卡片”链路,移动端 popup 与桌面 info panel 的路线规则统一
|
||||
|
||||
### 🔧 Improvements
|
||||
- connector 配置从 `CRUISE_CONFIG` 拆到独立 `CONNECTOR_CONFIG`,默认类名、动画名和实例命名也全部去 cruise 语义
|
||||
- 移动端 popup 增加更稳定的 dock/obstacle 处理,拖动卡片时连线起终点会持续按几何关系自适应刷新
|
||||
- Earth 多个图层与控制逻辑继续收口,补充算力中心/BGP 风格对齐、layer panel 与相关交互细节调整
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复巡航模式下终点只像“视觉锚点”而不是真实绑定对象的问题,卡片拖动后终点现在会跟随
|
||||
- 修复移动端与桌面端多类连线路线异常:压线、反向、临界区折返、起点遮挡事件点等问题
|
||||
- 修复对象矩形临界区内连线仍强制中点到中点导致路线像“先钻进 source 内部”再出去的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.35.1] — 2026-04-22
|
||||
## [0.36.0] — 2026-04-22
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 新增统一“算力中心”图层:接入超算与 GPU 集群,支持搜索、统计、图例、详情卡与独立图层开关
|
||||
- 算力中心支持精确位置与估算位置两种状态,估算点会以问号角标区分,避免数据不全时整批节点在地图上消失
|
||||
|
||||
### 🔧 Improvements
|
||||
- Earth 详情卡拖拽与地球拖拽交互继续收口,减少拖动卡片和旋转地球时的选中文本与 pointer 竞争
|
||||
- `planet.sh` 改为通过独立脚本计算 AI Provider 依赖指纹,降低与根仓库依赖版本文件的无关耦合
|
||||
- README 补充 WSL / Windows 局域网访问排查与转发配置说明,便于开发环境联调
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复 Earth 算力中心图层在无原始坐标时无法显示的问题,支持站点提示和国家级估算回退
|
||||
- 修复信息卡拖拽事件可能被卡片级 stopPropagation 吞掉,导致拖拽流中断的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.35.1] — 2026-04-22
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 统计展示改为统一 `data-earth-stat` 绑定机制,桌面 HUD 和移动端抽屉复用同一套状态更新入口
|
||||
|
||||
### 🔧 Improvements
|
||||
- 收口海缆、登陆点、卫星、BGP 事件与 BGP 状态的统计写入逻辑,减少后续继续补桌面/移动双写分支的成本
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复移动端态势抽屉中的海缆、登陆点与 BGP 统计在图层切换后可能停留旧值的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.35.0] — 2026-04-22
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 移动端底部抽屉系统全面上线:响应式布局自动切换、Tab 导航、手势上拉/下滑开合、惯性速度判定
|
||||
- 移动端点击可交互物件(海缆、登陆点、卫星、BGP)后弹出智能定位悬浮卡片,可拖动,点击跳转详情
|
||||
|
||||
### 🔧 Improvements
|
||||
- 抽屉把手区域缩小至 36px(collapsed 时仅露出把手,不遮挡地球操作区)
|
||||
- 抽屉定期弹跳动画提示用户可上拉,5 秒间隔,打开后自动停止
|
||||
- 通知胶囊位置调整,不再覆盖品牌 logo
|
||||
- 移动端单指旋转、双指捏合缩放地球,触控事件冲突修复(pointer-events 级联)
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复移动端抽屉 shell 因 layout 高度(240px+)遮挡地球触控区域,pointer-events 改为按层级精确控制
|
||||
- 修复悬浮卡片因 setPointerCapture 在 iOS Safari 抑制合成 click 事件导致无法点击的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.34.0] — 2026-04-22
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 搜索面板正式接入,支持搜索海缆、登陆点、卫星、BGP 事件与观测站,并可直接聚焦到对应对象
|
||||
- `planet.sh --allow-lan` 打通 Bun + Vite 的局域网开放链路,启动成功后自动打印推荐访问地址与后端健康检查地址
|
||||
|
||||
### 🔧 Improvements
|
||||
- 前端开发启动链统一改成 Bun 直接执行 Vite 入口,不再依赖 shell 中额外暴露的 Node 路径
|
||||
- Earth 搜索结果接入登陆点详情卡片与对象聚焦,搜索后可直接进入对应详情流
|
||||
- `planet.sh` 补充局域网 IPv4 自动识别与推荐地址输出,减少 WSL 局域网调试成本
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复 `./planet.sh restart --allow-lan` 全量重启时未把 `--allow-lan` 继续传给 `start()`,导致前端退回本机监听的问题
|
||||
- 修复 WSL + Bun 环境下前端偶发因 Vite 启动链不稳定而无法正确监听 `0.0.0.0:3000` 的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.33.0] — 2026-04-22
|
||||
|
||||
### ✨ Highlights
|
||||
- `news_live_streams` 采集器默认接入 `iptv-org` 频道目录,并将采集结果稳定并入 Earth TV 直播源列表
|
||||
- 数据源页支持直接编辑内置数据源 override,并为内置源提供一键恢复默认配置入口
|
||||
|
||||
### 🔧 Improvements
|
||||
- `News Live Streams` 现在作为可直接触发的内置默认数据源提供,无需先手工补 override 才能采集
|
||||
- TV 播放源菜单会直接区分 `[内置]` 和 `[采集]` 来源,频道来源信息也会同步展示
|
||||
- 新增 [earth-news-source-configuration-and-collector-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-news-source-configuration-and-collector-plan.md),正式规划 Earth 态势新闻源配置化与后续采集器化路线
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复 `news_live_streams` 采集完成后 `/api/v1/tv/streams` 因读取不存在的 `updated_at` 字段而导致默认频道全部消失的问题
|
||||
- 修复内置数据源操作列按钮显示不全,以及编辑抽屉中多个 `Collapse` 紧贴的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.32.0] — 2026-04-22
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 设置新增“地球默认大小”持久化项,重置视角、缩放百分比重置和 BGP 巡航视图现在统一复用这一份默认 zoom
|
||||
- 卫星焦点层次继续收口:巡航进入 presentation 前不再过早 dim,非焦点卫星改成“降亮度/尾迹/背板”而不是去饱和度
|
||||
|
||||
### 🔧 Improvements
|
||||
- Earth 设置面板区块和左右留白进一步收紧,整体更贴近 HUD 面板的密度
|
||||
- toolbar 展开边界缓存改为按需刷新,减少 document 级 mousemove 期间的重复布局读取
|
||||
- Scrollbar 和 ScrollbarOverlay 收窄 observer 范围,减少大表格和动态菜单下的额外刷新成本
|
||||
- 更新 [earth-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/earth-frontend-context.md),补充默认视图大小已进入 Earth 设置持久化真源
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复开启巡航后,尚未进入连线/presentation 时卫星已经整体变暗的问题
|
||||
- 修复默认大小重置链路分散在多个入口、实际 reset/cruise/缩放提示不一致的问题
|
||||
- 修复开启地形后卫星反馈层与地球背面可见性之间的一组表现问题,保留正面反馈同时恢复背面轨道遮挡
|
||||
|
||||
---
|
||||
|
||||
## [0.31.3] — 2026-04-22
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 图层注册表和启动任务框架继续收口,启动顺序、启动模式、启动提示和任务注册现在都能从统一入口扩展
|
||||
- 修复 Earth 普通旋转模式与巡航模式切换时的一组交互回归,同时让卫星/地形/昼夜模式的表现更稳定
|
||||
|
||||
### 🔧 Improvements
|
||||
- 新增 [layer-startup-tasks.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/layer-startup-tasks.js) 启动任务注册表,支持 `registerLayerStartupTask(id, taskFactory)`,并拆成海缆 / 卫星 / BGP 独立注册函数
|
||||
- Earth 图层控制改成注册表驱动,统一承载 `startupPriority`、`startupMode`、`startupLabel`、`startupMessage` 与图层持久化元信息
|
||||
- Earth 设置支持持久化图层开关、旋转模式、HUD 面板显示状态、地形透明度与日夜模式,并提供一键重置
|
||||
- 更新 [earth-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/earth-frontend-context.md) 记录图层注册表、启动任务、设置持久化与巡航适配边界
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复普通旋转模式下点击海缆 / 卫星 / BGP 后卡片和选中表现会被异常清空的问题
|
||||
- 修复巡航模式切回旋转再切回巡航后无法继续自动巡航的问题
|
||||
- 修复开启地形后卫星选中反馈层被高海拔区域吞掉的问题,并恢复轨道只在地球前半侧可见
|
||||
- 修复关闭日夜模式后地球照明仍沿真实昼夜切换、亮部过曝和偏色的问题,改成更中性的 inspection lighting
|
||||
- 修复 toolbar 收起态仍挡住地球交互,以及首帧短暂展开闪现的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.31.2] — 2026-04-21
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 巡航模式重构为“通用巡航队列 + 通用连线动画 + BGP 业务适配”三层结构,后续扩到海缆、卫星或新闻巡航时不必再复制一套 `main.js` 状态机
|
||||
- 修复巡航重构后的交互回归:空白点击重新稳定切到下一项,连线按“起点 → 引导线 → 终点”顺序入场
|
||||
|
||||
### 🔧 Improvements
|
||||
- 新增 [cruise-sequencer.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/cruise-sequencer.js) 统一管理队列推进、停留时长、打断与恢复
|
||||
- 新增 [callout-connector.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/callout-connector.js) 统一管理 SVG 连线、折线路径与描边动画
|
||||
- 新增 [bgp-cruise-adapter.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp-cruise-adapter.js) 收口 BGP 巡航目标排序、卡片落点、轮询去重与连线适配
|
||||
- 更新 [earth-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/earth-frontend-context.md) 说明新的巡航分层与复用边界
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复巡航模式下点击空白处无法稳定跳转到下一项、切回旋转再切回巡航后直接卡住的问题
|
||||
- 修复巡航连线被实时重定位覆盖导致“直接出现”而非绘制动画的问题
|
||||
- 修复连线动画节点入场节奏不对的问题,改为先出现起点,再绘制连线,最后出现终点
|
||||
|
||||
---
|
||||
|
||||
## [0.31.1] — 2026-04-21
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 图层开关状态统一成可复用的 `active / loading` 状态机,首次启用地形和卫星时不再像按钮失效
|
||||
- 文档目录重构为 `docs/technical`、`docs/plans`、`docs/deprecated`,并吸收 `.sisyphus/plans` 中有价值的 Earth / 卫星 / UE5 草案
|
||||
|
||||
### 🔧 Improvements
|
||||
- 新增 [layer-button-state.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/layer-button-state.js),统一按钮 tooltip、`aria-busy`、禁用态和状态文本同步
|
||||
- 地形图层支持 hover/focus 预热与空闲预热,首次点击等待前移,加载中状态持续可见
|
||||
- 卫星图层启用前会立即切换为 `loading` 中间态,请求完成后再切回正常开关表现
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复地形首次加载时通知过早消失、开关仍像关闭状态导致用户误判按钮损坏的问题
|
||||
- 修复卫星接口较慢时按钮没有任何中间态反馈的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.31.0] — 2026-04-21
|
||||
|
||||
### ✨ Features
|
||||
- Earth 新增"巡航展示"模式:自动轮播 BGP 异常事件,逐帧追踪连接线位置,支持外部交互立即中断序列(cancel notifier 模式)
|
||||
- 巡航目标事件点高亮显示:hover 外观 + 锁定脉冲动画,并与点击行为统一展示周边受影响卫星与海缆
|
||||
- BGP 事件图标新增填充 W 形波动符号(flap 类型),替换原有难以辨认的贝塞尔细线
|
||||
- 巡航/点击激活时其余卫星自动降饱和度 + 增加透明度以突出焦点;海缆未受影响时同步变暗
|
||||
|
||||
### 🔧 Improvements
|
||||
- 修复巡航轮播期间 BGP 事件 polling 刷新导致标记闪烁消失的问题(clearBGPData 延迟到请求完成后执行)
|
||||
- 点击与巡航锁定颜色统一为 hover 色(0.92, 0.98, 1.0 全透明),移除锁定态脉冲动画
|
||||
- 巡航连接折线转折点从尖角调整为钝角(linkElbowDropPx),提升连线可读性
|
||||
|
||||
---
|
||||
|
||||
## [0.30.0] — 2026-04-21
|
||||
|
||||
### ✨ Features
|
||||
- Earth 新增真实地形图层:后端代理 Terrarium DEM 瓦片(`/api/v1/visualization/terrain/terrarium/{z}/{x}/{y}.png`),前端新增 `terrain.js` 负责瓦片拉取、顶点位移与按海拔着色
|
||||
- 设置弹窗新增"地形"分组,支持通过滑块实时调整地形图层透明度
|
||||
|
||||
### 🔧 Improvements
|
||||
- 地形按钮改为异步加载,首次点击显示进度提示并在失败时自动回退
|
||||
- 启动阶段改用 `applyImmediateView` 直接应用初始视角,`showStatusMessage` / `queueStatusMessage` 区分即时与队列态状态消息,加载中不再被临时状态打断
|
||||
- 控制面板抽取 `applyTerrainUiState` / `getViewRotation` 收敛地形切换与视角旋转的重复 UI 同步逻辑
|
||||
|
||||
---
|
||||
|
||||
## [0.29.2] — 2026-04-21
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 继续收口 HUD 交互与设置面板表现,设置弹窗改成更接近从按钮展开的窗口感,同时加入系统级 admin 入口
|
||||
- 修正天球太阳方向与地球受光解耦后的日照逻辑,地表昼夜判断改为按太阳直射点经纬度落到地球贴图坐标
|
||||
|
||||
### 🔧 Improvements
|
||||
- toolbar 进一步收成更贴近 hub 的浅弓形排列,并统一成与 HUD panel 一致的液态玻璃配色与透明度
|
||||
- 设置弹窗与各 HUD panel 继续统一样式、等比缩放和头部基线,设置列表补充系统分组与 admin 跳转
|
||||
- 所有 HUD panel 增加更统一的液态玻璃高光与 hover / press 反馈
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复设置弹窗仍像旧圆角矩形、标题文案重复和从底边直直飞出的动画问题
|
||||
- 修复天球与太阳方向混用显示校准导致中国白天仍落在夜面的日照错误
|
||||
|
||||
---
|
||||
|
||||
## [0.29.1] — 2026-04-20
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 加载状态条改成单一队列式通知面板,加载阶段不再因为步骤文案变化而回缩,也不会被其他通知打断
|
||||
- 调整 brand panel 的呈现方式与昼夜/选中态可读性,让品牌区更自然、交互高亮在白天和黑夜里都更稳定
|
||||
|
||||
### 🔧 Improvements
|
||||
- 移除旧的地球加载浮层结构,统一由 HUD 状态消息承载三点脉冲加载过程
|
||||
- brand panel 改为无边框品牌层,仅保留轻微氛围光,不再因为非常规尺寸显得像第五块功能面板
|
||||
- 温和收敛地球昼夜材质与主背光强度,保留昼夜辨识度的同时提升白天地表纹理和夜面交互可见性
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复加载地球时通知条在步骤切换中反复缩短、其他状态消息抢占加载流程的问题
|
||||
- 修复海缆、登陆点和 BGP 选中高亮在黑夜中过暗、在高光中过亮导致难以辨识的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.29.0] — 2026-04-20
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 新增天球层第一版:引入真实全天星图、亮星层与太阳/月亮位置计算,地球场景首次具备可校准的天文背景
|
||||
- 地球昼夜分隔升级为更明显的日夜增强效果,夜面、晨昏带和太阳方向联动更容易直接读出来
|
||||
|
||||
### 🔧 Improvements
|
||||
- 新增 `celestial.js` 模块和 `assets/celestial/` 资源目录,统一管理星图、亮星数据以及太阳/月亮与光照同步
|
||||
- 卫星图例改为按倾角分组,严格固定为“赤道轨道 → 低倾角轨道 → 中倾角轨道 → 高倾角轨道 → 逆行轨道”顺序,并全部中文化
|
||||
- 图层面板补齐关闭按钮,拖拽脱离左列后不再被流布局 margin 影响,能够真正贴到品牌面板下沿
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复天球球壳放大后被相机 far plane 裁剪导致的外层黑环问题
|
||||
- 修复图层面板在左侧上移时始终与 brand panel 保持额外间距的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.28.2] — 2026-04-20
|
||||
|
||||
### ✨ Highlights
|
||||
- 修正媒体情报面板在 `电视直播 / 态势聚合` tab 间切换时的尺寸记忆逻辑,切回原 tab 后可恢复各自大小状态
|
||||
- 清理 `docs/` 根目录遗留的旧路径文档,只保留新的分组目录和归档目录,结束同一文档双路径并存状态
|
||||
|
||||
### 🔧 Improvements
|
||||
- `media-panel` 切换逻辑改成按 tab 分别记忆尺寸状态,避免 `A -> B -> A` 时继续共用同一套外层尺寸
|
||||
- 目录整理真正完成收尾:旧的 `docs/*.md` 平铺计划文档删除,继续以 `docs/agents / earth / backend / frontend / ops / ue5 / deprecated` 为唯一入口
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复拉伸 `media-panel` 后切换 tab 时,`news-panel` 高度回退到旧默认值的问题
|
||||
- 修复拉伸后切换 tab 导致面板视觉锚点异常的问题,切换时改为围绕当前卡片自身右下角进行尺寸恢复
|
||||
|
||||
---
|
||||
|
||||
## [0.28.1] — 2026-04-20
|
||||
|
||||
### ✨ Highlights
|
||||
- 收口 Earth 媒体情报面板命名,明确外层 `media-panel` 与内部 `tv-panel / news-panel` 的职责边界
|
||||
- 整理 `docs/` 目录分组,并将已完成或已废弃的计划文档归档到 `docs/deprecated`
|
||||
|
||||
### 🔧 Improvements
|
||||
- 底部 tab 语义统一为 `media-panel-tabs / media-panel-tab`,并将文案更新为“电视直播 / 态势聚合”
|
||||
- 补充媒体面板、聚合新闻模块的注释说明,减少 `tv-panel` 同时指代外层壳和内层直播 pane 的阅读歧义
|
||||
- 更新 README、AI Provider README 与历史文档互链,适配新的 `docs/agents / docs/earth / docs/frontend / docs/backend / docs/ops / docs/ue5` 分组结构
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复媒体情报面板标题组在头部撑出多余空白的问题,去掉 `hud-panel__title-group` 的无效弹性占位
|
||||
- 修复聚合 tab 头部仍保留冗余固定标题的问题,现在仅显示区域标签
|
||||
|
||||
---
|
||||
|
||||
## [0.28.0] — 2026-04-20
|
||||
|
||||
### ✨ Features
|
||||
- 将 Earth 的“新闻直播”和“全球态势聚合”合并为统一的“媒体情报”面板,支持底部 tab 切换与共享标题栏操作区
|
||||
- 聚合新闻不再单独占据一个 HUD 面板,而是作为媒体情报面板内的第二视图与直播协同呈现
|
||||
|
||||
说明:
|
||||
- 当前结构中,外层 HUD 壳为 `media-panel`,内部 tab 内容区分别为 `tv-panel` 和 `news-panel`
|
||||
|
||||
### 🔧 Improvements
|
||||
- TV / News 面板切换加入底边锚定的 reform 动画,并继续保留拖拽、缩放和共享 HUD 行为收口
|
||||
- 聚合新闻视图新增默认高度约束与内部滚动填充逻辑,避免初始高度过度膨胀
|
||||
- `tv.js`、`news.js` 进一步清理共享 HUDPanel 迁移后的残留逻辑,收紧 tab / resize / reform 相关局部 helper
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复媒体情报面板右下角缩放时高度异常抬升、标题栏被顶出视口的问题
|
||||
- 修复直播/聚合 tab 切换时按钮高亮、内容切换和底边基准表现不一致的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.27.7] — 2026-04-16
|
||||
|
||||
## [0.27.8] — 2026-04-20
|
||||
|
||||
### 🔧 Improvements
|
||||
- Earth HUD 共享 `HUDPanel` 默认展开/收缩逻辑继续收口,图例与图层面板统一使用同一套边缘阈值与箭头状态机
|
||||
- 保持新闻直播面板现有特例折叠行为不变,避免播放器区域被默认折叠逻辑影响
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复图例与图层面板展开/收缩箭头方向和实际动作不一致的问题
|
||||
- 修复拖动到屏幕底边附近时初始箭头、拖动中箭头和点击后动作不同步的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.27.7] — 2026-04-16
|
||||
|
||||
### 🔧 Improvements
|
||||
- 用户管理、数据源配置、电视直播源表格统一接入可折叠操作列,窄宽度下自动收起到下拉菜单,减少操作区挤压
|
||||
- 电视直播设置改为表格总览 + 弹窗编辑模式,主表内容更紧凑,适合控制台一屏浏览
|
||||
- Earth TV 面板新增失败源探测与自动回退恢复标记,便于值班时快速识别异常直播源
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复 Settings 电视直播源新增后取消编辑会残留未保存草稿的问题
|
||||
- 修复 Settings 删除直播源只改本地状态、刷新后恢复的问题,删除现在会立即持久化
|
||||
- 修复 Users / DataSources / Settings 表格“备注/状态”和“操作”之间的空白占位列问题
|
||||
- 修复 `useCollapsedActions` 未释放 `ResizeObserver` 导致的潜在内存泄漏与重复回调问题
|
||||
|
||||
---
|
||||
|
||||
## [0.27.6] — 2026-04-15
|
||||
|
||||
### 🔧 Improvements
|
||||
@@ -814,7 +147,7 @@ Released: 2026-04-12
|
||||
|
||||
- Added [backend/app/api/v1/tv.py](/home/ray/dev/linkong/planet/backend/app/api/v1/tv.py), [backend/app/services/tv_streams.py](/home/ray/dev/linkong/planet/backend/app/services/tv_streams.py), and [backend/app/services/collectors/news_live_streams.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/news_live_streams.py) to provide TV source configuration, public stream payloads, a guarded HLS proxy path, and a collector entry point for future world-news live-source ingestion.
|
||||
- Added the Earth TV HUD workspace through [frontend/public/earth/index.html](/home/ray/dev/linkong/planet/frontend/public/earth/index.html), [frontend/public/earth/js/tv.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/tv.js), and [frontend/public/earth/css/tv-panel.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/tv-panel.css), including toolbar access, draggable/closable behavior, resize support, direct video/HLS playback, iframe fallback, and per-channel external-open handling.
|
||||
- Added [docs/deprecated/earth-tv-live-module-plan.md](/home/ray/dev/linkong/planet/docs/deprecated/earth-tv-live-module-plan.md) and [docs/earth/technical/news-live-streams-collector-format.md](/home/ray/dev/linkong/planet/docs/technical/earth-news-live-streams-collector-format.md) to document the TV module rollout plan and the expected collector payload format for future curated live-channel ingestion.
|
||||
- Added [docs/earth-tv-live-module-plan.md](/home/ray/dev/linkong/planet/docs/earth-tv-live-module-plan.md) and [docs/news-live-streams-collector-format.md](/home/ray/dev/linkong/planet/docs/news-live-streams-collector-format.md) to document the TV module rollout plan and the expected collector payload format for future curated live-channel ingestion.
|
||||
|
||||
### Improved
|
||||
|
||||
@@ -900,7 +233,7 @@ Released: 2026-04-10
|
||||
|
||||
- Improved [frontend/src/pages/Playground/Playground.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Playground/Playground.tsx) and [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css) by rebuilding Playground into a true chatbox workflow with persistent history, edit-and-resend behavior, grounded message actions, responsive composer behavior, bottom-stick scrolling, and tighter mobile layout handling.
|
||||
- Improved [frontend/src/components/AppLayout/AppLayout.tsx](/home/ray/dev/linkong/planet/frontend/src/components/AppLayout/AppLayout.tsx), [frontend/src/App.tsx](/home/ray/dev/linkong/planet/frontend/src/App.tsx), and [frontend/src/pages/Alerts/Alerts.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Alerts/Alerts.tsx) by reorganizing navigation around `采集与数据`, `专题观测`, and split alert entries so the app can scale to more observability and situational modules without turning the top-level UI into a single overloaded page.
|
||||
- Improved [README.md](/home/ray/dev/linkong/planet/README.md) and [docs/agents/situational-awareness-foundation-plan.md](/home/ray/dev/linkong/planet/docs/plans/agents-situational-awareness-foundation-plan.md) by documenting the current AI/alerts base, planned situational-awareness direction, and the new persistent Playground foundation.
|
||||
- Improved [README.md](/home/ray/dev/linkong/planet/README.md) and [docs/situational-awareness-foundation-plan.md](/home/ray/dev/linkong/planet/docs/situational-awareness-foundation-plan.md) by documenting the current AI/alerts base, planned situational-awareness direction, and the new persistent Playground foundation.
|
||||
|
||||
### Fixed
|
||||
|
||||
@@ -938,7 +271,7 @@ Released: 2026-04-10
|
||||
### Improved
|
||||
|
||||
- Improved [rules.md](/home/ray/dev/linkong/planet/rules.md) by adding mandatory release-workflow requirements and a new frontend layout constraint section covering single-screen workspaces, overflow ownership, tab-pane behavior, compact-mode expectations, and readable-card fallbacks.
|
||||
- Improved [frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md) by summarizing the recurring Earth, Playground, BGP, and admin-layout regressions into concrete constraints for future frontend work, including “prefer scrollbars over unreadable compression” and “do not treat every tab as a table pane.”
|
||||
- Improved [frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/frontend-layout-guidelines.md) by summarizing the recurring Earth, Playground, BGP, and admin-layout regressions into concrete constraints for future frontend work, including “prefer scrollbars over unreadable compression” and “do not treat every tab as a table pane.”
|
||||
|
||||
## 0.24.6
|
||||
|
||||
@@ -955,7 +288,7 @@ Released: 2026-04-10
|
||||
- Improved [backend/app/services/bgp_incidents.py](/home/ray/dev/linkong/planet/backend/app/services/bgp_incidents.py) and [backend/app/services/bgp_enrichment.py](/home/ray/dev/linkong/planet/backend/app/services/bgp_enrichment.py) by avoiding historical full-table infrastructure scans, narrowing observation baseline payloads to required columns, and pushing more ASN filtering into the database.
|
||||
- Improved [backend/app/api/v1/alerts.py](/home/ray/dev/linkong/planet/backend/app/api/v1/alerts.py), [backend/app/api/v1/dashboard.py](/home/ray/dev/linkong/planet/backend/app/api/v1/dashboard.py), and [backend/app/api/v1/settings.py](/home/ray/dev/linkong/planet/backend/app/api/v1/settings.py) by collapsing several repeated count and settings queries into fewer aggregate or batched reads.
|
||||
- Improved [frontend/src/pages/BGP/BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx), [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css), and [frontend/src/components/MarkdownRenderer/MarkdownRenderer.tsx](/home/ray/dev/linkong/planet/frontend/src/components/MarkdownRenderer/MarkdownRenderer.tsx) by rebuilding the `AI 简报` tab layout, fixing saved brief scrolling behavior, and extending the renderer to handle tables, separators, and stored metadata comments more gracefully.
|
||||
- Improved [docs/frontend/plans/ai-playground-development-plan.md](/home/ray/dev/linkong/planet/docs/plans/frontend-ai-playground-development-plan.md) by explicitly recording that the current BGP brief is only the first-stage summary flow and that regional prefix-geography analysis remains a planned Phase B follow-up.
|
||||
- Improved [docs/ai-playground-development-plan.md](/home/ray/dev/linkong/planet/docs/ai-playground-development-plan.md) by explicitly recording that the current BGP brief is only the first-stage summary flow and that regional prefix-geography analysis remains a planned Phase B follow-up.
|
||||
|
||||
### Fixed
|
||||
|
||||
@@ -1061,8 +394,8 @@ Released: 2026-04-09
|
||||
### Added
|
||||
|
||||
- Added [frontend/src/pages/Playground/Playground.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Playground/Playground.tsx), introducing the first dedicated AI testing workspace with provider status visibility, prompt/result tabs, and collapsible operator guidance.
|
||||
- Added [docs/frontend/technical/frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md), documenting the repository standard for one-screen admin workspaces and module-local overflow handling.
|
||||
- Added [docs/frontend/plans/ai-playground-development-plan.md](/home/ray/dev/linkong/planet/docs/plans/frontend-ai-playground-development-plan.md), capturing the completed AI gateway/UI work and the next delivery phases for BGP briefs, evidence-first inputs, and future agent runtime expansion.
|
||||
- Added [docs/frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/frontend-layout-guidelines.md), documenting the repository standard for one-screen admin workspaces and module-local overflow handling.
|
||||
- Added [docs/ai-playground-development-plan.md](/home/ray/dev/linkong/planet/docs/ai-playground-development-plan.md), capturing the completed AI gateway/UI work and the next delivery phases for BGP briefs, evidence-first inputs, and future agent runtime expansion.
|
||||
|
||||
### Improved
|
||||
|
||||
@@ -1167,7 +500,7 @@ Released: 2026-04-07
|
||||
- Added [backend/app/services/ai_client.py](/home/ray/dev/linkong/planet/backend/app/services/ai_client.py), introducing an internal HTTP client for `backend -> aiprovider` calls with request-id propagation and lightweight retry.
|
||||
- Added [aiprovider/main.py](/home/ray/dev/linkong/planet/aiprovider/main.py), [aiprovider/provider_service.py](/home/ray/dev/linkong/planet/aiprovider/provider_service.py), and related config/schema files to stand up the dedicated adapter service.
|
||||
- Added [aiprovider/.env.example](/home/ray/dev/linkong/planet/aiprovider/.env.example) and [docker-compose.local-model.yml](/home/ray/dev/linkong/planet/docker-compose.local-model.yml) as ready-to-edit local-model templates.
|
||||
- Added [docs/agents/aiprovider.md](/home/ray/dev/linkong/planet/docs/technical/agents-aiprovider.md), documenting architecture, configuration, single-machine and multi-machine deployment, and cross-service calling patterns.
|
||||
- Added [docs/aiprovider.md](/home/ray/dev/linkong/planet/docs/aiprovider.md), documenting architecture, configuration, single-machine and multi-machine deployment, and cross-service calling patterns.
|
||||
- Added a dedicated `重启 AI Provider` control path in [Dashboard.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Dashboard/Dashboard.tsx), [system_control.py](/home/ray/dev/linkong/planet/backend/app/services/system_control.py), and [system_restart_runner.py](/home/ray/dev/linkong/planet/backend/scripts/system_restart_runner.py).
|
||||
|
||||
### Improved
|
||||
@@ -1293,7 +626,7 @@ Released: 2026-04-02
|
||||
|
||||
- Added a new `IPtoASN Prefix Geography` collector in [iptoasn.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/iptoasn.py) and registered it through [data_sources.yaml](/home/ray/dev/linkong/planet/backend/app/core/data_sources.yaml), [data_sources.py](/home/ray/dev/linkong/planet/backend/app/core/data_sources.py), [datasource_defaults.py](/home/ray/dev/linkong/planet/backend/app/core/datasource_defaults.py), and [collectors/__init__.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/__init__.py).
|
||||
- Added country centroid helpers in [countries.py](/home/ray/dev/linkong/planet/backend/app/core/countries.py) so country-level prefix geography can produce map coordinates instead of only labels.
|
||||
- Added a dedicated prefix-geography implementation note in [prefix-geography-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-prefix-geography-plan.md).
|
||||
- Added a dedicated prefix-geography implementation note in [prefix-geography-plan.md](/home/ray/dev/linkong/planet/docs/prefix-geography-plan.md).
|
||||
- Added recent `15m` collector activity dimensions to BGP coverage output in [bgp_collectors.py](/home/ray/dev/linkong/planet/backend/app/services/bgp_collectors.py) and [visualization.py](/home/ray/dev/linkong/planet/backend/app/api/v1/visualization.py).
|
||||
- Added additional BGP detector coverage for `route_leak_candidate` and `path_flap` flows in [test_bgp.py](/home/ray/dev/linkong/planet/backend/tests/test_bgp.py).
|
||||
- Added a local Earth cloud texture at [earth_clouds_1024.png](/home/ray/dev/linkong/planet/frontend/public/earth/assets/earth_clouds_1024.png) to avoid remote cloud-map dependency failures.
|
||||
@@ -1308,7 +641,7 @@ Released: 2026-04-02
|
||||
- Improved Earth event animation semantics in [bgp.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp.js) by separating icon pulse from ring expansion so the center marker can breathe while the ring expands independently.
|
||||
- Improved Earth texture reliability in [earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js) by switching clouds back to a local static asset under the restored `public/earth` runtime.
|
||||
- Improved frontend boot noise in [frontend/index.html](/home/ray/dev/linkong/planet/frontend/index.html) by removing the default Vite favicon request that was generating irrelevant `vite.svg` timeouts during Earth debugging.
|
||||
- Improved project planning docs in [bgp-context.md](/home/ray/dev/linkong/planet/docs/technical/earth-bgp-context.md) and [TODO.md](/home/ray/dev/linkong/planet/TODO.md) so the roadmap now explicitly prioritizes `activity layer`, `prefix-centric geography`, and follow-up geofeed/whois work.
|
||||
- Improved project planning docs in [bgp-context.md](/home/ray/dev/linkong/planet/docs/bgp-context.md) and [TODO.md](/home/ray/dev/linkong/planet/TODO.md) so the roadmap now explicitly prioritizes `activity layer`, `prefix-centric geography`, and follow-up geofeed/whois work.
|
||||
|
||||
### Fixed
|
||||
|
||||
@@ -1466,7 +799,7 @@ Released: 2026-03-31
|
||||
- Added restart-task Redis helpers and whitelist command mapping in [system_control.py](/home/ray/dev/linkong/planet/backend/app/services/system_control.py).
|
||||
- Added detached restart runner orchestration in [system_restart_runner.py](/home/ray/dev/linkong/planet/backend/scripts/system_restart_runner.py).
|
||||
- Added `-d` / `--database` support to [planet.sh](/home/ray/dev/linkong/planet/planet.sh) for database-only restarts.
|
||||
- Added restart control documentation in [system-service-control.md](/home/ray/dev/linkong/planet/docs/technical/backend-system-service-control.md).
|
||||
- Added restart control documentation in [system-service-control.md](/home/ray/dev/linkong/planet/docs/system-service-control.md).
|
||||
|
||||
### Improved
|
||||
|
||||
@@ -1669,7 +1002,7 @@ Released: 2026-03-26
|
||||
|
||||
### Added
|
||||
|
||||
- Added a dedicated Earth module remediation plan in [earth-module-plan.md](/home/ray/dev/linkong/planet/docs/deprecated/earth-module-plan.md).
|
||||
- Added a dedicated Earth module remediation plan in [earth-module-plan.md](/home/ray/dev/linkong/planet/docs/earth-module-plan.md).
|
||||
- Added backend TLE helpers in [satellite_tle.py](/home/ray/dev/linkong/planet/backend/app/core/satellite_tle.py).
|
||||
- Added backend support for returning `tle_line1` and `tle_line2` from the satellite visualization API.
|
||||
|
||||
|
||||
File renamed without changes.
@@ -10,9 +10,9 @@ This document connects three existing planning threads into one implementation r
|
||||
|
||||
Related documents:
|
||||
|
||||
- [aiprovider](/home/ray/dev/linkong/planet/docs/technical/agents-aiprovider.md)
|
||||
- [datasource-health-plan](/home/ray/dev/linkong/planet/docs/plans/agents-datasource-health-plan.md)
|
||||
- [agent-architecture-plan](/home/ray/dev/linkong/planet/docs/plans/agents-agent-architecture-plan.md)
|
||||
- [aiprovider](/home/ray/dev/linkong/planet/docs/aiprovider.md)
|
||||
- [datasource-health-plan](/home/ray/dev/linkong/planet/docs/datasource-health-plan.md)
|
||||
- [agent-architecture-plan](/home/ray/dev/linkong/planet/docs/agent-architecture-plan.md)
|
||||
|
||||
|
||||
## Big Picture
|
||||
+2
-2
@@ -30,7 +30,7 @@
|
||||
- [backend/app/services/ai_client.py](/home/ray/dev/linkong/planet/backend/app/services/ai_client.py)
|
||||
- [aiprovider/main.py](/home/ray/dev/linkong/planet/aiprovider/main.py)
|
||||
- [aiprovider/provider_service.py](/home/ray/dev/linkong/planet/aiprovider/provider_service.py)
|
||||
- [docs/technical/agents-aiprovider.md](/home/ray/dev/linkong/planet/docs/technical/agents-aiprovider.md)
|
||||
- [docs/aiprovider.md](/home/ray/dev/linkong/planet/docs/aiprovider.md)
|
||||
|
||||
### 2. 本地运行与配置打通
|
||||
|
||||
@@ -77,7 +77,7 @@
|
||||
|
||||
相关文件:
|
||||
|
||||
- [docs/technical/frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md)
|
||||
- [docs/frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/frontend-layout-guidelines.md)
|
||||
- [frontend/src/pages/BGP/BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx)
|
||||
|
||||
## 当前限制
|
||||
File renamed without changes.
@@ -187,7 +187,7 @@ Current reality:
|
||||
- that is expected, because incidents are aggregated and de-noised
|
||||
- but incident-first rendering makes the Earth view look too quiet unless there is another always-available activity layer
|
||||
|
||||
Implementation detail for the recommended `activity layer` is expanded in the [BGP Region Aggregation Plan](/home/ray/dev/linkong/planet/docs/plans/earth-bgp-region-aggregation-plan.md).
|
||||
Implementation detail for the recommended `activity layer` is expanded in [bgp-region-aggregation-plan.md](/home/ray/dev/linkong/planet/docs/bgp-region-aggregation-plan.md).
|
||||
|
||||
So the immediate next milestone is:
|
||||
|
||||
File renamed without changes.
File renamed without changes.
+2
-2
@@ -17,7 +17,7 @@ It is an aggregation/view-model layer:
|
||||
|
||||
## Why This Layer Exists
|
||||
|
||||
Current product gap from [bgp-context.md](/home/ray/dev/linkong/planet/docs/technical/earth-bgp-context.md):
|
||||
Current product gap from [bgp-context.md](/home/ray/dev/linkong/planet/docs/bgp-context.md):
|
||||
|
||||
- incident density is naturally low
|
||||
- anomaly density is higher, but still not enough to keep the globe expressive all the time
|
||||
@@ -290,7 +290,7 @@ Each feature should include:
|
||||
|
||||
## Earth Rendering Plan
|
||||
|
||||
Detailed visual layering guidance is expanded in [bgp-earth-rendering-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-bgp-earth-rendering-plan.md).
|
||||
Detailed visual layering guidance is expanded in [bgp-earth-rendering-plan.md](/home/ray/dev/linkong/planet/docs/bgp-earth-rendering-plan.md).
|
||||
|
||||
### Layer Relationship
|
||||
|
||||
File renamed without changes.
File renamed without changes.
@@ -84,8 +84,6 @@ async def run(self, db):
|
||||
| HuggingFace Spaces | space | Demo应用 | 1天 |
|
||||
| PeeringDB | ixp/network/facility | 互联网交换点/网络/机房 | 1-2天 |
|
||||
| TeleGeography | submarine_cable | 海底光缆信息 | 7天 |
|
||||
| Space-Track TLE | satellite_tle | 卫星轨道 TLE 数据 | 依采集器配置 |
|
||||
| BarentsWatch AIS | vessel | 船只位置、航速、航向、MMSI 等 AIS 数据 | 依采集器配置 |
|
||||
|
||||
## 四、数据格式 (统一存储到 CollectedData 表)
|
||||
|
||||
@@ -202,30 +200,6 @@ def start_scheduler():
|
||||
|
||||
**核心文件**: `backend/app/services/scheduler.py`
|
||||
|
||||
### 成功采集与连接状态
|
||||
|
||||
内置采集器成功采集后,调度器会记录当前有效配置已通过连接验证:
|
||||
|
||||
```python
|
||||
if datasource.last_status == "success":
|
||||
effective_candidate = await get_builtin_effective_candidate(db, datasource.source)
|
||||
checksum, _ = await build_builtin_connectivity_checksum(...)
|
||||
await save_connectivity_success(
|
||||
db,
|
||||
datasource.source,
|
||||
checksum,
|
||||
{"status_code": None},
|
||||
connected_by="collection",
|
||||
)
|
||||
```
|
||||
|
||||
这个记录用于控制台“采集器设置”中的连接状态判断:如果当前配置和成功采集时的 checksum 一致,就视为已连接,不要求用户再手动点击连接按钮。只有 endpoint、请求头、基础配置或凭证指纹变化时,才需要重新验证。
|
||||
|
||||
相关实现见:
|
||||
|
||||
- [datasource_connectivity.py](/home/ray/dev/linkong/planet/backend/app/services/datasource_connectivity.py)
|
||||
- [数据源、采集器设置与连接验证](/home/ray/dev/linkong/planet/docs/technical/zh/datasource-collector-settings-connectivity.md)
|
||||
|
||||
## 八、相关代码文件
|
||||
|
||||
```
|
||||
@@ -237,86 +211,13 @@ backend/app/services/collectors/
|
||||
├── epoch_ai.py # Epoch AI采集器
|
||||
├── huggingface.py # HuggingFace采集器
|
||||
├── peeringdb.py # PeeringDB采集器
|
||||
├── telegeraphy.py # TeleGeography海底光缆采集器
|
||||
└── vessel_ais.py # BarentsWatch AIS 船只采集器
|
||||
└── telegeraphy.py # TeleGeography海底光缆采集器
|
||||
|
||||
backend/app/models/
|
||||
└── collected_data.py # 统一数据模型
|
||||
```
|
||||
|
||||
## 九、凭证型采集器
|
||||
|
||||
部分采集器需要外部服务凭证,例如:
|
||||
|
||||
| 采集器 | credential provider | 凭证来源 |
|
||||
| --- | --- | --- |
|
||||
| `barentswatch_vessels` | `barentswatch` | 控制台采集器设置、环境变量、`~/.zshrc` |
|
||||
| `spacetrack_tle` | `spacetrack` | 环境变量、`~/.zshrc` |
|
||||
|
||||
### BarentsWatch AIS
|
||||
|
||||
BarentsWatch AIS 的凭证解析统一在:
|
||||
|
||||
- [barentswatch.py](/home/ray/dev/linkong/planet/backend/app/services/barentswatch.py)
|
||||
|
||||
`VesselAISCollector` 只负责采集和转换 AIS 数据,不再自己读取环境变量或拼 token 请求。它通过:
|
||||
|
||||
- `resolve_barentswatch_config()`
|
||||
- `fetch_barentswatch_access_token()`
|
||||
|
||||
获取运行时配置。
|
||||
|
||||
解析优先级:
|
||||
|
||||
1. `DataSourceConfig.auth_config`
|
||||
2. `DataSourceConfig.config`
|
||||
3. 环境变量
|
||||
4. `~/.zshrc`
|
||||
|
||||
支持变量:
|
||||
|
||||
```bash
|
||||
export BARENTSWATCH_CLIENT_ID="..."
|
||||
export BARENTSWATCH_CLIENT_SECRET="..."
|
||||
```
|
||||
|
||||
并兼容历史拼写:
|
||||
|
||||
```bash
|
||||
export BARRENTSWATCH_CLIENT_ID="..."
|
||||
export BARRENTSWATCH_CLIENT_SECRET="..."
|
||||
```
|
||||
|
||||
连接验证会先请求 `https://id.barentswatch.no/connect/token` 获取 `scope=ais` 的 access token,再用 `Authorization: Bearer <token>` 请求 AIS endpoint。
|
||||
|
||||
## 十、采集器设置与连接验证
|
||||
|
||||
控制台的“采集器设置”页提供所有内置采集器的 endpoint、请求头、超时、重试和凭证配置。连接验证不是只看前端按钮状态,而是由后端计算 checksum:
|
||||
|
||||
- endpoint
|
||||
- auth type
|
||||
- headers
|
||||
- config
|
||||
- credential provider
|
||||
- 凭证指纹
|
||||
|
||||
相关 API:
|
||||
|
||||
```http
|
||||
GET /api/v1/datasources/configs/all
|
||||
POST /api/v1/datasources/configs/builtin/connection-status
|
||||
POST /api/v1/datasources/configs/builtin/connect
|
||||
POST /api/v1/settings/integrations/barentswatch/connect
|
||||
GET /api/v1/settings/credential-guides/{provider}
|
||||
POST /api/v1/settings/credential-guides/{provider}/generate
|
||||
POST /api/v1/settings/credential-guides/{provider}/reset
|
||||
```
|
||||
|
||||
更多细节见:
|
||||
|
||||
- [数据源、采集器设置与连接验证](/home/ray/dev/linkong/planet/docs/technical/zh/datasource-collector-settings-connectivity.md)
|
||||
|
||||
## 十一、数据使用场景
|
||||
## 九、数据使用场景
|
||||
|
||||
采集的数据最终会:
|
||||
|
||||
@@ -324,7 +225,7 @@ POST /api/v1/settings/credential-guides/{provider}/reset
|
||||
2. **态势分析** - 统计全球算力分布、增长趋势
|
||||
3. **告警系统** - 检测重要节点变化
|
||||
|
||||
## 十二、采集器注册机制
|
||||
## 十、采集器注册机制
|
||||
|
||||
采集器在应用启动时自动注册:
|
||||
|
||||
@@ -346,7 +247,7 @@ collector_registry.register(TeleGeographyCableSystemCollector())
|
||||
|
||||
**核心文件**: `backend/app/services/collectors/registry.py`
|
||||
|
||||
## 十三、触发采集
|
||||
## 十一、触发采集
|
||||
|
||||
### 方式一:定时触发
|
||||
系统启动时,APScheduler会自动根据各采集器的`frequency_hours`设置定时任务。
|
||||
File renamed without changes.
File renamed without changes.
@@ -1,22 +0,0 @@
|
||||
# Deprecated Docs
|
||||
|
||||
这个目录用于存放两类文档:
|
||||
|
||||
1. 已经完成、主要保留为历史记录的实施计划
|
||||
2. 已被现有实现或新方案替代的旧计划
|
||||
|
||||
放到这里并不代表这些文档“错误”,而是表示:
|
||||
|
||||
- 它们不再适合作为当前开发的主指导文档
|
||||
- 如果要了解历史决策、演进路径或旧设计背景,仍然可以参考
|
||||
|
||||
当前归档原则:
|
||||
|
||||
- 明确写明“已完成”的计划,优先归档
|
||||
- 已被正式实现替代、继续放在 `docs/` 根目录会误导后续开发的计划,归档
|
||||
- 仍然指导未来开发、尚未完成或仍有明确执行价值的文档,继续保留在 `docs/`
|
||||
|
||||
补充说明:
|
||||
|
||||
- 一部分归档文档来自外部或临时工作流草案,例如 sisyphus 生成的初稿
|
||||
- 这类文档如果有可用内容,应先吸收到 `docs/plans/` 或 `docs/technical/`,再归档保留来源记录
|
||||
@@ -1,165 +0,0 @@
|
||||
# HUD Panel Component Plan
|
||||
|
||||
## Goal
|
||||
|
||||
Unify Earth HUD panels into a reusable component layer so new panels can share:
|
||||
|
||||
- a consistent shell
|
||||
- a consistent header
|
||||
- a consistent action-button system
|
||||
- a consistent body and collapse pattern
|
||||
|
||||
## Scope
|
||||
|
||||
Target panels:
|
||||
|
||||
- `tv-panel`
|
||||
- `news-panel`
|
||||
- `legend`
|
||||
- `layer-panel`
|
||||
- `earth-stats`
|
||||
- `info-card`
|
||||
- settings modal header/actions
|
||||
|
||||
## Component Model
|
||||
|
||||
### Base shell
|
||||
|
||||
- `.hud-panel`
|
||||
- `.hud-panel--compact`
|
||||
- `.hud-panel--media`
|
||||
- `.hud-panel--collapsed`
|
||||
- `.hud-panel-hidden`
|
||||
- `.hud-panel.is-dragging`
|
||||
- `.hud-panel.is-layout-animating`
|
||||
|
||||
### Header
|
||||
|
||||
- `.hud-panel__header`
|
||||
- `.hud-panel__title-group`
|
||||
- `.hud-panel__title`
|
||||
- `.hud-panel__subtitle`
|
||||
- `.hud-panel__chip`
|
||||
- `.hud-panel__actions`
|
||||
|
||||
Header baseline rule:
|
||||
|
||||
- Header title styling is fixed by the component layer and should not drift per panel
|
||||
- Title font size, font weight, letter spacing, line height, text color, and vertical alignment come from the shared header tokens and structure
|
||||
- Header divider, border treatment, inner spacing, and title-to-actions alignment are part of the same shared baseline
|
||||
- Panel-specific header differences should be limited to explicit variants such as `compact` or `media`, or token overrides with documented intent
|
||||
- “Looks close enough” local header overrides should be treated as temporary compatibility code and removed during migration
|
||||
|
||||
### Actions
|
||||
|
||||
- `.hud-panel__action`
|
||||
- `.hud-panel__action--icon`
|
||||
- `.hud-panel__action--collapse`
|
||||
- `.hud-panel__action--close`
|
||||
- `.hud-panel__action--refresh`
|
||||
- `.hud-panel__action--external`
|
||||
|
||||
Action-button baseline rule:
|
||||
|
||||
- Header action buttons must have one fixed default style baseline across all HUD panels
|
||||
- Default width behavior, padding, icon size, radius, alignment, hover, and active feedback all come from `.hud-panel__action`
|
||||
- Panel-specific differences must be expressed through explicit variants or token overrides, not ad-hoc local button rewrites
|
||||
- `close` buttons are part of the same default action system and must not silently fall back to a separate legacy box model
|
||||
|
||||
### Body
|
||||
|
||||
- `.hud-panel__body`
|
||||
- `.hud-panel__body--scroll`
|
||||
- `.hud-panel__body--collapsible`
|
||||
|
||||
### Collapse behavior
|
||||
|
||||
- `.hud-panel--collapsed`
|
||||
- `.hud-panel--expand-up`
|
||||
- `.hud-panel--expand-down`
|
||||
|
||||
Adaptive collapse / expand rule:
|
||||
|
||||
- HUD panels support two expansion directions:
|
||||
- top-to-bottom expansion
|
||||
- bottom-to-top expansion
|
||||
- Expansion direction should be decided at runtime from available viewport space rather than hardcoded per panel
|
||||
- Use:
|
||||
- `d` = available distance from the header anchor to the viewport bottom edge
|
||||
- `h` = expected expanded panel height
|
||||
- buffer = `20px`
|
||||
- Collapsed-state direction rule:
|
||||
- if `d > h + 20px`, the next action direction is `expand-up`
|
||||
- if `d <= h + 20px`, the next action direction is `expand-down`
|
||||
- To avoid jitter around the threshold, the shared controller should keep a small hysteresis band:
|
||||
- if the current direction is already `up`, keep it until `d <= h`
|
||||
- if the current direction is already `down`, keep it until `d > h + 20px`
|
||||
- The opposite edge is still a safety guard:
|
||||
- if the chosen side cannot fit at all, fall back to the other side if it can fit
|
||||
- if neither side fully fits, choose the side with more space and let the body scroll
|
||||
- If neither direction fully fits, choose the direction with more available space and let the body scroll
|
||||
- Collapse icon direction must match the active expansion direction so the icon always describes the real open/close motion
|
||||
- The collapse icon describes the next action, not the current state
|
||||
- This mapping is fixed component behavior and must not drift per panel:
|
||||
- collapsed + expand-down => `expand_more`
|
||||
- expanded + expand-down => `expand_less`
|
||||
- collapsed + expand-up => `expand_less`
|
||||
- expanded + expand-up => `expand_more`
|
||||
- Panels must not combine icon-name swapping with extra CSS rotation for the same collapse control
|
||||
- Expansion direction and icon direction must come from one shared source of truth in the component controller
|
||||
- The direction decision should be recomputed when opening, resizing the viewport, or restoring a dragged panel near another edge
|
||||
|
||||
## Tokens
|
||||
|
||||
Promote panel differences into CSS variables instead of duplicating selectors:
|
||||
|
||||
- `--hud-panel-padding`
|
||||
- `--hud-header-padding`
|
||||
- `--hud-header-gap`
|
||||
- `--hud-action-padding`
|
||||
- `--hud-action-gap`
|
||||
- `--hud-action-icon-size`
|
||||
- `--hud-body-gap`
|
||||
- `--hud-body-max-height`
|
||||
- `--hud-chip-radius`
|
||||
- `--hud-title-font-size`
|
||||
- `--hud-title-font-weight`
|
||||
- `--hud-title-letter-spacing`
|
||||
- `--hud-title-line-height`
|
||||
- `--hud-title-color`
|
||||
- `--hud-header-border-color`
|
||||
- `--hud-header-divider-opacity`
|
||||
- `--hud-expand-direction`
|
||||
|
||||
## Migration Order
|
||||
|
||||
1. Build the shared component layer in `frontend/public/earth/css/hud.css`
|
||||
2. Migrate `tv-panel` and `news-panel` first as the reference implementation
|
||||
3. Migrate `legend` and `layer-panel` into a compact variant
|
||||
4. Migrate `earth-stats` and `info-card`
|
||||
5. Align settings modal header/actions with the same action system
|
||||
6. Remove legacy one-off button selectors after verification
|
||||
|
||||
## Guardrails
|
||||
|
||||
- Do not change panel behavior and data flow during the first pass
|
||||
- Keep old class names temporarily as compatibility hooks
|
||||
- Prefer variable overrides over per-panel reimplementation
|
||||
- Treat header action-button default styling as fixed component API, not per-panel design space
|
||||
- Treat header title typography, border, and divider styling as fixed component API, not per-panel design space
|
||||
- Treat collapse direction as a component behavior contract, not a one-off panel trick
|
||||
- Treat collapse icon semantics as a component behavior contract, not a per-panel visual preference
|
||||
- Verify header alignment and drag/collapse behavior after each migration batch
|
||||
|
||||
## First Implementation Batch
|
||||
|
||||
Batch 1 should only do:
|
||||
|
||||
- shared header structure
|
||||
- shared action-button system
|
||||
- shared title typography and header border/divider baseline
|
||||
- shared collapsible body pattern
|
||||
- adaptive collapse direction logic and direction-aware collapse icons
|
||||
- migration of `tv-panel` and `news-panel`
|
||||
|
||||
That keeps risk low while giving the rest of the HUD a stable target to migrate toward.
|
||||
File renamed without changes.
File renamed without changes.
Loaded 100 of 273 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user