Compare commits
31 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
eeee788530 | ||
|
|
655e2a7d2d | ||
|
|
3ea99a9529 | ||
|
|
f9c1334365 | ||
|
|
5f47ec1659 | ||
|
|
229be0bced | ||
|
|
50a417ca83 | ||
|
|
e9464a9833 | ||
|
|
86807f6af6 | ||
|
|
8b8f7138c0 | ||
|
|
d5f3784ffb | ||
|
|
195a8bf71c | ||
|
|
987c378f99 | ||
|
|
67f82dc41c | ||
|
|
abe04030fb | ||
|
|
6a5f9f7ad4 | ||
|
|
439a512148 | ||
|
|
f73fa1ea6d | ||
|
|
5b623a6385 | ||
|
|
0082cf3fbd | ||
|
|
3ae4acdff8 | ||
|
|
437efc848c | ||
|
|
003a46ac30 | ||
|
|
4b0be4cb76 | ||
|
|
b7647379de | ||
|
|
0f89372d71 | ||
|
|
2b0d4cfc49 | ||
|
|
e6d0332fba | ||
|
|
fe45a99cbd | ||
|
|
ae77b06c3c | ||
|
|
b5dd4f12f8 |
91
.claude/commands/goal-driven.md
Normal file
91
.claude/commands/goal-driven.md
Normal file
@@ -0,0 +1,91 @@
|
||||
---
|
||||
description: 用 goal-driven 方法推动一个复杂任务持续执行,直到明确成功标准被满足
|
||||
argument-hint: 建议填写任务目标;若同时给出成功标准更好
|
||||
allowed-tools: ["Read", "Edit", "Bash", "Grep", "Glob"]
|
||||
---
|
||||
|
||||
# /goal-driven — 目标驱动执行模式
|
||||
|
||||
使用 `lidangzzz/goal-driven` 的核心思想来推进复杂任务:先固定目标与成功标准,再持续执行和反复验收,直到标准真正满足。
|
||||
|
||||
适用场景:
|
||||
|
||||
- 长周期实现任务
|
||||
- 高复杂度工程任务
|
||||
- 可被明确验收的研究、实现、迁移、验证类工作
|
||||
|
||||
不适用场景:
|
||||
|
||||
- 纯脑暴
|
||||
- 无法定义成功标准的模糊任务
|
||||
- 很小的一次性修改
|
||||
|
||||
## 输入要求
|
||||
|
||||
若 `$ARGUMENTS` 只包含目标,没有成功标准,先补全一版可执行的成功标准再开始。
|
||||
|
||||
启动时先输出:
|
||||
|
||||
```md
|
||||
Goal
|
||||
- ...
|
||||
|
||||
Criteria for success
|
||||
- ...
|
||||
|
||||
Plan
|
||||
1. ...
|
||||
2. ...
|
||||
3. ...
|
||||
|
||||
Verification
|
||||
- ...
|
||||
```
|
||||
|
||||
## 执行规则
|
||||
|
||||
1. 先把任务固化为两个核心块:
|
||||
- `Goal`
|
||||
- `Criteria for success`
|
||||
|
||||
2. 成功标准必须尽量客观,可验证,可落地。
|
||||
优先写成:
|
||||
- 需要交付什么
|
||||
- 需要通过哪些测试或验证
|
||||
- 如何判断结果真的完成
|
||||
|
||||
3. 进入持续执行循环:
|
||||
- 完成一个阶段
|
||||
- 检查当前结果是否满足成功标准
|
||||
- 若未满足,明确剩余差距并继续推进
|
||||
|
||||
4. 任何“完成了”“差不多了”“已实现”之类的结论,都必须经过验证,不能直接接受。
|
||||
|
||||
5. 如果验证失败:
|
||||
- 明确指出哪条成功标准没满足
|
||||
- 继续工作,不要把阶段性进展误判为完成
|
||||
|
||||
6. 只有在以下情况之一才能停止:
|
||||
- 成功标准已满足
|
||||
- 用户明确要求停止
|
||||
|
||||
## 执行风格
|
||||
|
||||
- 重证据,轻口头判断
|
||||
- 重验收,轻自我感觉
|
||||
- 优先用测试、日志、产物、对比结果来证明完成
|
||||
- 对长期任务保持“未达标就继续”的节奏
|
||||
|
||||
## 简版模板
|
||||
|
||||
```md
|
||||
Goal: [[[[[在此填写最终目标]]]]]
|
||||
|
||||
Criteria for success: [[[[[在此填写成功标准]]]]]
|
||||
|
||||
循环执行:
|
||||
1. 推进任务
|
||||
2. 检查是否满足成功标准
|
||||
3. 若未满足,继续工作
|
||||
4. 直到满足标准或用户明确停止
|
||||
```
|
||||
3
.codex/config.toml
Normal file
3
.codex/config.toml
Normal file
@@ -0,0 +1,3 @@
|
||||
approval_policy = "never"
|
||||
|
||||
sandbox_mode = "danger-full-access"
|
||||
101
.codex/skills/goal-driven/SKILL.md
Executable file
101
.codex/skills/goal-driven/SKILL.md
Executable file
@@ -0,0 +1,101 @@
|
||||
---
|
||||
name: goal-driven
|
||||
description: Run a goal-driven execution loop for very large, long-horizon, rigorously verifiable tasks. Use when the user explicitly wants the lidangzzz/goal-driven method, a master-agent plus worker-agent style workflow, or a persistent loop that keeps working until concrete success criteria are satisfied.
|
||||
---
|
||||
|
||||
# Goal-Driven
|
||||
|
||||
Use this skill when the user wants a strict goal-driven workflow for a hard task with:
|
||||
|
||||
- one clear end goal
|
||||
- explicit success criteria
|
||||
- repeated verification against those criteria
|
||||
- continued execution until the criteria are actually met
|
||||
|
||||
This skill is adapted from `lidangzzz/goal-driven`, but trimmed for local skill use to avoid bloating context.
|
||||
|
||||
## When To Use
|
||||
|
||||
Use it for tasks like:
|
||||
|
||||
- compilers, interpreters, theorem-like proof work, deep refactors
|
||||
- long-running system design or implementation work
|
||||
- problems that are expensive and complex, but still objectively testable
|
||||
|
||||
Do not use it for:
|
||||
|
||||
- vague brainstorming without a success condition
|
||||
- short one-shot edits
|
||||
- tasks where "done" cannot be evaluated in a meaningful way
|
||||
|
||||
## Core Model
|
||||
|
||||
The workflow has two roles:
|
||||
|
||||
1. Master role
|
||||
Defines the goal, defines the success criteria, audits progress, and decides whether the work is actually complete.
|
||||
|
||||
2. Worker role
|
||||
Keeps advancing the task toward the goal. If a result is partial, stalled, or unverifiable, the worker continues.
|
||||
|
||||
In Codex, only use actual subagents when the user explicitly asks for delegation or subagent work and the platform supports it. Otherwise emulate the same loop locally: keep working, checkpointing, and re-verifying until the criteria are satisfied.
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Normalize the task into two blocks:
|
||||
- `Goal`
|
||||
- `Criteria for success`
|
||||
|
||||
2. Make the criteria concrete and testable.
|
||||
Good criteria usually include:
|
||||
- required outputs
|
||||
- required validations or tests
|
||||
- edge cases or coverage thresholds
|
||||
- what evidence proves completion
|
||||
|
||||
3. Break the work into milestones that can each produce evidence.
|
||||
|
||||
4. Execute the next milestone.
|
||||
If subagents are explicitly allowed, the master may delegate bounded worker tasks.
|
||||
If not, do the work locally but keep the master/worker mindset.
|
||||
|
||||
5. Whenever work pauses, stalls, or appears complete, audit against the criteria directly.
|
||||
Check artifacts, tests, logs, diffs, metrics, or other real evidence.
|
||||
|
||||
6. If the criteria are not met, continue with a specific delta:
|
||||
- what is still missing
|
||||
- what evidence failed
|
||||
- what the next worker pass must improve
|
||||
|
||||
7. Stop only when the criteria are met, or when the user explicitly stops the process.
|
||||
|
||||
## Operating Rules
|
||||
|
||||
- Prefer objective checks over self-reported completion.
|
||||
- Do not confuse progress with completion.
|
||||
- If the worker says "done", verify it.
|
||||
- If verification fails, continue from the gap instead of restarting blindly.
|
||||
- Keep the goal stable unless the user changes it.
|
||||
- Tighten fuzzy criteria before sinking large amounts of effort.
|
||||
|
||||
## Recommended Response Shape
|
||||
|
||||
When starting a goal-driven task, structure the kickoff like this:
|
||||
|
||||
```md
|
||||
Goal
|
||||
- ...
|
||||
|
||||
Criteria for success
|
||||
- ...
|
||||
|
||||
Current plan
|
||||
1. ...
|
||||
2. ...
|
||||
3. ...
|
||||
|
||||
Verification
|
||||
- What evidence will prove completion
|
||||
```
|
||||
|
||||
For a reusable prompt template, read [references/prompt-template.md](references/prompt-template.md).
|
||||
7
.codex/skills/goal-driven/agents/openai.yaml
Normal file
7
.codex/skills/goal-driven/agents/openai.yaml
Normal file
@@ -0,0 +1,7 @@
|
||||
interface:
|
||||
display_name: "Goal-Driven"
|
||||
short_description: "Drive complex work until explicit success criteria are met."
|
||||
default_prompt: "Use $goal-driven to turn this task into a concrete goal, explicit success criteria, and a verification-driven execution loop."
|
||||
|
||||
policy:
|
||||
allow_implicit_invocation: true
|
||||
38
.codex/skills/goal-driven/references/prompt-template.md
Executable file
38
.codex/skills/goal-driven/references/prompt-template.md
Executable file
@@ -0,0 +1,38 @@
|
||||
# Goal-Driven Prompt Template
|
||||
|
||||
Use this when you want a reusable kickoff prompt for a master/worker execution loop.
|
||||
|
||||
```md
|
||||
# Goal-Driven System
|
||||
|
||||
Goal: [[[[[DEFINE THE FINAL GOAL HERE]]]]]
|
||||
|
||||
Criteria for success: [[[[[DEFINE THE SUCCESS CRITERIA HERE]]]]]
|
||||
|
||||
You are the master agent.
|
||||
|
||||
Your job is to:
|
||||
1. Keep the goal and criteria fixed.
|
||||
2. Start worker execution toward the goal.
|
||||
3. Audit any claimed progress against the criteria.
|
||||
4. If the criteria are not met, continue the work with a precise next delta.
|
||||
5. Stop only when the criteria are satisfied or the user explicitly stops the process.
|
||||
|
||||
Worker requirements:
|
||||
1. Break the task into subproblems.
|
||||
2. Keep producing concrete progress toward the goal.
|
||||
3. Report evidence, not just claims.
|
||||
4. Continue until the criteria are satisfied.
|
||||
|
||||
Master audit loop:
|
||||
1. Check whether the worker is still making progress.
|
||||
2. If the worker stalls or claims completion, verify against the criteria.
|
||||
3. If verification fails, resume work from the remaining gap.
|
||||
4. Repeat until the criteria are met.
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- Stronger criteria produce better results than stronger rhetoric.
|
||||
- Prefer measurable checks such as tests, parity checks, generated artifacts, benchmarks, or reviewable outputs.
|
||||
- If the environment does not support subagents, emulate the same loop locally.
|
||||
124
README.md
124
README.md
@@ -227,6 +227,120 @@ bun run build
|
||||
|
||||
启动服务后访问: `http://localhost:8000/docs`
|
||||
|
||||
## WSL / Windows 局域网访问
|
||||
|
||||
如果服务运行在 WSL 中,而你希望:
|
||||
|
||||
- Windows 本机浏览器访问开发服务
|
||||
- 同一局域网内的手机或其他电脑访问开发服务
|
||||
|
||||
推荐按下面顺序排查和配置。
|
||||
|
||||
### 1. 在 WSL 中启动服务
|
||||
|
||||
```bash
|
||||
./planet.sh start --allow-lan
|
||||
```
|
||||
|
||||
这会让前端监听 `0.0.0.0:3000`,后端监听 `0.0.0.0:8000`。
|
||||
|
||||
### 2. 先确认 WSL 内部服务正常
|
||||
|
||||
在 WSL 中执行:
|
||||
|
||||
```bash
|
||||
curl http://localhost:3000
|
||||
curl http://localhost:8000/health
|
||||
ss -ltnp | grep -E ':3000|:8000'
|
||||
```
|
||||
|
||||
预期:
|
||||
|
||||
- `3000` 返回前端 HTML
|
||||
- `8000/health` 返回健康检查 JSON
|
||||
- `ss` 中能看到 `0.0.0.0:3000` 和 `0.0.0.0:8000`
|
||||
|
||||
如果这一步不通,先不要继续做 Windows 转发。
|
||||
|
||||
### 3. 在 Windows 本机验证 localhost 直通
|
||||
|
||||
在 Windows PowerShell 中执行:
|
||||
|
||||
```powershell
|
||||
curl http://localhost:3000
|
||||
curl http://localhost:8000/health
|
||||
```
|
||||
|
||||
在常见的 WSL2 开发环境下,Windows 通常可以直接通过 `localhost` 访问 WSL 中的服务。
|
||||
|
||||
### 4. 如果需要让局域网设备访问,再做 Windows 端口转发
|
||||
|
||||
注意:下面的命令必须在“以管理员身份运行”的 PowerShell 中执行。
|
||||
|
||||
先把 Windows 对外网卡上的 `3000` / `8000` 转发到 Windows 本机 `127.0.0.1`:
|
||||
|
||||
```powershell
|
||||
netsh interface portproxy delete v4tov4 listenaddress=0.0.0.0 listenport=3000
|
||||
netsh interface portproxy delete v4tov4 listenaddress=0.0.0.0 listenport=8000
|
||||
|
||||
netsh interface portproxy add v4tov4 listenaddress=0.0.0.0 listenport=3000 connectaddress=127.0.0.1 connectport=3000
|
||||
netsh interface portproxy add v4tov4 listenaddress=0.0.0.0 listenport=8000 connectaddress=127.0.0.1 connectport=8000
|
||||
```
|
||||
|
||||
再放行 Windows 防火墙:
|
||||
|
||||
```powershell
|
||||
New-NetFirewallRule -DisplayName "WSL Planet 3000" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 3000
|
||||
New-NetFirewallRule -DisplayName "WSL Planet 8000" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 8000
|
||||
```
|
||||
|
||||
检查转发规则是否生效:
|
||||
|
||||
```powershell
|
||||
netsh interface portproxy show all
|
||||
```
|
||||
|
||||
预期能看到:
|
||||
|
||||
- `0.0.0.0:3000 -> 127.0.0.1:3000`
|
||||
- `0.0.0.0:8000 -> 127.0.0.1:8000`
|
||||
|
||||
### 5. 查 Windows 局域网 IP,并让其他设备访问
|
||||
|
||||
在 Windows PowerShell 中执行:
|
||||
|
||||
```powershell
|
||||
ipconfig
|
||||
```
|
||||
|
||||
找到当前联网网卡的 IPv4 地址,例如 `192.168.8.228`。
|
||||
|
||||
局域网其他设备可访问:
|
||||
|
||||
- `http://<Windows局域网IP>:3000/earth`
|
||||
- `http://<Windows局域网IP>:3000/admin`
|
||||
|
||||
例如:
|
||||
|
||||
- `http://192.168.8.228:3000/earth`
|
||||
|
||||
### 6. 常见现象与判断
|
||||
|
||||
- WSL 中 `curl localhost:3000` 能通,但 Windows 访问 `WSL 的局域网 IP:3000` 不通:这是正常现象之一,优先验证 Windows 的 `localhost:3000`
|
||||
- Windows `localhost:3000` 能通,但局域网设备访问 `Windows 局域网 IP:3000` 不通:通常缺少 `portproxy` 或防火墙放行
|
||||
- `whoami /groups` 中 `S-1-5-32-544` 显示 `deny only`:说明当前 PowerShell 不是提权管理员窗口
|
||||
|
||||
### 7. 本项目一次性验证顺序
|
||||
|
||||
建议固定按这个顺序验证:
|
||||
|
||||
1. WSL 中执行 `curl http://localhost:3000`
|
||||
2. WSL 中执行 `curl http://localhost:8000/health`
|
||||
3. Windows 中执行 `curl http://localhost:3000`
|
||||
4. Windows 中执行 `curl http://localhost:8000/health`
|
||||
5. 管理员 PowerShell 配置 `portproxy` 和防火墙
|
||||
6. 用手机或其他电脑访问 `http://<Windows局域网IP>:3000/earth`
|
||||
|
||||
## 启动容错参数
|
||||
|
||||
`planet.sh` 现在为依赖安装、数据库、AI Provider 启动加入了有限次重试,并会在数据库与 `aiprovider` 启动后额外等待 Docker healthcheck。
|
||||
@@ -328,11 +442,11 @@ AI_PROVIDER_SERVICE_TOKEN=change_me
|
||||
|
||||
详细文档:
|
||||
|
||||
- [docs/agents/aiprovider.md](/home/ray/dev/linkong/planet/docs/agents/aiprovider.md)
|
||||
- [docs/technical/agents-aiprovider.md](/home/ray/dev/linkong/planet/docs/technical/agents-aiprovider.md)
|
||||
- [aiprovider/README.md](/home/ray/dev/linkong/planet/aiprovider/README.md)
|
||||
- [docs/frontend/frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/frontend/frontend-layout-guidelines.md)
|
||||
- [docs/frontend/ai-playground-development-plan.md](/home/ray/dev/linkong/planet/docs/frontend/ai-playground-development-plan.md)
|
||||
- [docs/agents/situational-awareness-foundation-plan.md](/home/ray/dev/linkong/planet/docs/agents/situational-awareness-foundation-plan.md)
|
||||
- [docs/technical/frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md)
|
||||
- [docs/plans/frontend-ai-playground-development-plan.md](/home/ray/dev/linkong/planet/docs/plans/frontend-ai-playground-development-plan.md)
|
||||
- [docs/plans/agents-situational-awareness-foundation-plan.md](/home/ray/dev/linkong/planet/docs/plans/agents-situational-awareness-foundation-plan.md)
|
||||
|
||||
## 前端页面布局规范
|
||||
|
||||
@@ -346,7 +460,7 @@ AI_PROVIDER_SERVICE_TOKEN=change_me
|
||||
当前推荐参考实现:
|
||||
|
||||
- [frontend/src/pages/BGP/BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx)
|
||||
- [docs/frontend/frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/frontend/frontend-layout-guidelines.md)
|
||||
- [docs/technical/frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md)
|
||||
|
||||
## License
|
||||
|
||||
|
||||
15
TODO.md
15
TODO.md
@@ -20,3 +20,18 @@
|
||||
- [x] 在 activity layer 之后继续补 `route leak` 和 `path instability / flap` detector
|
||||
- [ ] 对 [frontend/public/earth/js/bgp.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp.js) 做按职责拆分的小重构,拆成 data / markers / overlays / animation,降低后续维护复杂度
|
||||
- [ ] 可选优化(非必做):将 BGP incident/collector 标点改为 HTML marker(参考 worldmonitor 的 `htmlElementsData` 思路),实现近乎固定屏幕尺寸与更高密度可点击性
|
||||
- [ ] 保持 Earth 当前这批纯个人偏好设置继续走本地持久化:`旋转模式`、HUD 面板显示/隐藏、`地形透明度` 暂不升级到后端系统设置,避免把设备级偏好过早做成全局配置
|
||||
- [ ] 如果后续明确需要“账号级同步 Earth 偏好”,再单独设计 `Earth user preferences`:优先按用户维度而不是全局系统设置保存,并规划 `localStorage -> backend` 的平滑迁移策略
|
||||
- [ ] 为 Planet / Earth 补一个可用的日志查看系统:先明确前后端/AI Provider/采集任务的日志入口、最近日志聚合、筛选与 tail 能力,再决定是先做脚本级统一入口还是控制台内置日志面板
|
||||
- [ ] 把 Earth 态势新闻源从 [earth_news.py](/home/ray/dev/linkong/planet/backend/app/services/earth_news.py) 的硬编码列表抽成可配置目录,优先保持当前“实时聚合”链路不变,只先解决新闻源不可配置的问题
|
||||
- [ ] 为 Earth 态势新闻设计后续采集器化方案:明确新闻数据模型、去重策略、区域映射、过期清理和 Earth/AI 复用方式,再决定何时把新闻从实时抓取升级成正式 collector
|
||||
- [ ] 为 Earth 地球表面增加一层与基础纹理对齐的材质/纹理 overlay,并在同层叠加国界轮廓参考线;要求国界线与底图稳定对齐,且 hover 到国家轮廓时能高亮当前国家,便于校准地表和增强交互
|
||||
- [ ] 把 Earth 新闻接入通用巡航队列:按新闻发生地和时间排序生成巡航目标,巡航聚焦到新闻事件时显示对应新闻卡片,并保持实现边界为“通用巡航层 + 新闻业务适配层”,不要再把新闻逻辑直接耦合回 `main.js` 状态机
|
||||
- [ ] 为未知位置的算力中心建立分层坐标补全链路:优先 `精确坐标 > 站点/园区命中 > 城市 > 州/省 > 国家内主要算力城市 > 国家质心`,并把每次回退的 `confidence / reason / precision` 明确写进统一 GeoJSON
|
||||
- [ ] 为算力中心补一份可维护的本地位置注册表,例如 `canonical_name / aliases / operator / country / region / city / lat / lon / confidence / source_note`,避免把地点知识长期硬编码在 `visualization.py`
|
||||
- [ ] 增强 `epoch_ai_gpu` 和相关算力采集器的源页面解析:即使公开 API 不给坐标,也继续尝试从详情页、HTML、内嵌 JSON、schema.org、OpenGraph、脚本变量和 PDF/新闻稿链接里抽地点线索
|
||||
- [ ] 为未知位置算力中心增加外部富化策略评估:可选接入公开知识源或搜索兜底,只抓“站点名/园区名/城市名”级别线索,不直接抓经纬度结论,并把结果作为候选证据而不是真值
|
||||
- [ ] 为算力中心建立 `operator / cluster name / facility alias` 归一化层,先解决 `xAI / Colossus / Memphis`、`OpenAI / Stargate`、`CoreWeave`、`Lambda`、`Crusoe` 这类同一对象多种写法导致的地点匹配失败
|
||||
- [ ] 为估算位置增加更细的视觉和产品表达:除了问号角标,还要支持 tooltip/详情中的“估算依据”“精度级别”“最后核验时间”,并允许在设置中单独开关“仅看精确位置”
|
||||
- [ ] 为国家级估算点设计更合理的落点策略:优先落在“该国主要算力/数据中心城市候选集”而不是几何质心,必要时同国多节点做稳定散列分配,避免大量节点堆在荒漠或海上
|
||||
- [ ] 为未知位置算力中心建立人工校验工作流:支持导出待核验清单、记录人工确认结果,并把人工确认反哺到位置注册表,逐步减少问号点比例
|
||||
|
||||
@@ -1,6 +1,10 @@
|
||||
FROM python:3.14-slim
|
||||
ARG PYTHON_IMAGE=python:3.14-slim
|
||||
ARG UV_IMAGE=ghcr.io/astral-sh/uv:latest
|
||||
|
||||
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
|
||||
FROM ${UV_IMAGE} AS uv
|
||||
FROM ${PYTHON_IMAGE}
|
||||
|
||||
COPY --from=uv /uv /uvx /bin/
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
|
||||
@@ -1,6 +1,10 @@
|
||||
FROM python:3.14-slim
|
||||
ARG PYTHON_IMAGE=python:3.14-slim
|
||||
ARG UV_IMAGE=ghcr.io/astral-sh/uv:latest
|
||||
|
||||
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
|
||||
FROM ${UV_IMAGE} AS uv
|
||||
FROM ${PYTHON_IMAGE}
|
||||
|
||||
COPY --from=uv /uv /uvx /bin/
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
|
||||
@@ -420,6 +420,7 @@ async def list_datasources(
|
||||
collector_list.append(
|
||||
{
|
||||
"id": datasource.id,
|
||||
"source": datasource.source,
|
||||
"name": datasource.name,
|
||||
"module": datasource.module,
|
||||
"priority": datasource.priority,
|
||||
|
||||
@@ -4,12 +4,15 @@ import os
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, status
|
||||
from datetime import datetime
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Request, status
|
||||
from pydantic import BaseModel
|
||||
|
||||
from app.core.config import ROOT_DIR
|
||||
from app.core.security import get_current_user
|
||||
from app.models.user import User
|
||||
from app.services.persistent_logs import record_audit_log, record_system_log
|
||||
from app.services.system_control import (
|
||||
build_task_id,
|
||||
clear_active_task_id,
|
||||
@@ -23,6 +26,15 @@ from app.services.system_control import (
|
||||
set_active_task_id,
|
||||
upsert_task_state,
|
||||
)
|
||||
from app.services.system_logs import (
|
||||
DEFAULT_LOG_LINE_LIMIT,
|
||||
MAX_LOG_LINE_LIMIT,
|
||||
SUPPORTED_LOG_LEVELS,
|
||||
append_buffer_log,
|
||||
list_log_sources,
|
||||
normalize_log_level,
|
||||
read_log_snapshot,
|
||||
)
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
@@ -47,6 +59,59 @@ class RestartTaskLogsResponse(BaseModel):
|
||||
lines: list[str]
|
||||
|
||||
|
||||
class SystemLogSourceSummary(BaseModel):
|
||||
source_id: str
|
||||
name: str
|
||||
kind: str
|
||||
location: str
|
||||
description: str
|
||||
category: str
|
||||
status: str
|
||||
|
||||
|
||||
class SystemLogSourcesResponse(BaseModel):
|
||||
items: list[SystemLogSourceSummary]
|
||||
|
||||
|
||||
class SystemLogDailyMarker(BaseModel):
|
||||
date_token: str
|
||||
total: int
|
||||
dominant_level: str
|
||||
|
||||
|
||||
class SystemLogSnapshotResponse(BaseModel):
|
||||
source_id: str
|
||||
name: str
|
||||
kind: str
|
||||
location: str
|
||||
description: str
|
||||
category: str
|
||||
status: str
|
||||
level: str
|
||||
selected_levels: list[str] = []
|
||||
search_query: str = ""
|
||||
available_levels: list[str]
|
||||
daily_markers: list[SystemLogDailyMarker] = []
|
||||
line_limit: int
|
||||
line_count: int
|
||||
lines: list[str]
|
||||
|
||||
|
||||
class EarthClientLogEventCreate(BaseModel):
|
||||
level: str = "error"
|
||||
message: str
|
||||
category: str | None = None
|
||||
url: str | None = None
|
||||
module: str | None = None
|
||||
detail: str | None = None
|
||||
|
||||
|
||||
class EarthClientLogEventResponse(BaseModel):
|
||||
accepted: bool
|
||||
source_id: str
|
||||
level: str
|
||||
|
||||
|
||||
def ensure_super_admin(current_user: User) -> None:
|
||||
if not require_super_admin(current_user.role):
|
||||
raise HTTPException(
|
||||
@@ -55,9 +120,22 @@ def ensure_super_admin(current_user: User) -> None:
|
||||
)
|
||||
|
||||
|
||||
def validate_log_date(raw_value: str | None, field_name: str) -> str | None:
|
||||
if raw_value in {None, ""}:
|
||||
return None
|
||||
try:
|
||||
return datetime.strptime(raw_value, "%Y-%m-%d").date().isoformat()
|
||||
except ValueError as exc:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
detail=f"{field_name} must be in YYYY-MM-DD format",
|
||||
) from exc
|
||||
|
||||
|
||||
@router.post("/restart-tasks", response_model=RestartTaskResponse)
|
||||
async def create_restart_task(
|
||||
payload: RestartTaskCreate,
|
||||
request: Request,
|
||||
current_user: User = Depends(get_current_user),
|
||||
):
|
||||
ensure_super_admin(current_user)
|
||||
@@ -133,11 +211,31 @@ async def create_restart_task(
|
||||
requested_by=requested_by,
|
||||
)
|
||||
clear_active_task_id(task_id)
|
||||
await record_audit_log(
|
||||
action="system.restart_task.requested",
|
||||
actor_id=current_user.id,
|
||||
actor_name=current_user.username,
|
||||
target_type="restart_task",
|
||||
target_id=task_id,
|
||||
result="failed",
|
||||
ip=request.client.host if request.client else None,
|
||||
details={"action": payload.action, "message": task_state["message"]},
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
|
||||
detail=task_state["message"],
|
||||
) from exc
|
||||
|
||||
await record_audit_log(
|
||||
action="system.restart_task.requested",
|
||||
actor_id=current_user.id,
|
||||
actor_name=current_user.username,
|
||||
target_type="restart_task",
|
||||
target_id=task_id,
|
||||
result="accepted",
|
||||
ip=request.client.host if request.client else None,
|
||||
details={"action": payload.action},
|
||||
)
|
||||
return task_state
|
||||
|
||||
|
||||
@@ -165,3 +263,92 @@ async def get_restart_task_logs(
|
||||
if task is None:
|
||||
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Restart task not found")
|
||||
return {"task_id": task_id, "lines": get_task_logs(task_id)}
|
||||
|
||||
|
||||
@router.get("/logs/sources", response_model=SystemLogSourcesResponse)
|
||||
async def get_system_log_sources(
|
||||
current_user: User = Depends(get_current_user),
|
||||
):
|
||||
ensure_super_admin(current_user)
|
||||
return {"items": list_log_sources()}
|
||||
|
||||
|
||||
@router.get("/logs/{source_id}", response_model=SystemLogSnapshotResponse)
|
||||
async def get_system_log_snapshot(
|
||||
source_id: str,
|
||||
limit: int = DEFAULT_LOG_LINE_LIMIT,
|
||||
level: str = "all",
|
||||
levels: str | None = Query(None, description="Comma-separated log levels"),
|
||||
start_date: str | None = Query(None, description="Filter logs from this date (YYYY-MM-DD)"),
|
||||
end_date: str | None = Query(None, description="Filter logs until this date (YYYY-MM-DD)"),
|
||||
search: str | None = Query(None, description="Case-insensitive substring search"),
|
||||
current_user: User = Depends(get_current_user),
|
||||
):
|
||||
ensure_super_admin(current_user)
|
||||
|
||||
if limit < 1 or limit > MAX_LOG_LINE_LIMIT:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
detail=f"limit must be between 1 and {MAX_LOG_LINE_LIMIT}",
|
||||
)
|
||||
if str(level).strip().lower() not in SUPPORTED_LOG_LEVELS and normalize_log_level(level) == "all" and str(level).strip().lower() not in {"", "all"}:
|
||||
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="Unsupported log level")
|
||||
if levels:
|
||||
for raw_level in str(levels).split(","):
|
||||
normalized_level = str(raw_level).strip().lower()
|
||||
if not normalized_level:
|
||||
continue
|
||||
if normalized_level not in SUPPORTED_LOG_LEVELS and normalize_log_level(normalized_level) == "all":
|
||||
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="Unsupported log level")
|
||||
normalized_start_date = validate_log_date(start_date, "start_date")
|
||||
normalized_end_date = validate_log_date(end_date, "end_date")
|
||||
if normalized_start_date and normalized_end_date and normalized_start_date > normalized_end_date:
|
||||
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="start_date must be earlier than or equal to end_date")
|
||||
|
||||
snapshot = read_log_snapshot(
|
||||
source_id,
|
||||
limit,
|
||||
level=level,
|
||||
levels=levels,
|
||||
start_date=normalized_start_date,
|
||||
end_date=normalized_end_date,
|
||||
search=search,
|
||||
)
|
||||
if snapshot is None:
|
||||
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Log source not found")
|
||||
return snapshot
|
||||
|
||||
|
||||
@router.post("/logs/earth-client", response_model=EarthClientLogEventResponse)
|
||||
async def ingest_earth_client_log(
|
||||
payload: EarthClientLogEventCreate,
|
||||
request: Request,
|
||||
):
|
||||
normalized_level = normalize_log_level(payload.level)
|
||||
append_buffer_log(
|
||||
"earth-client",
|
||||
level=normalized_level,
|
||||
message=payload.message,
|
||||
context={
|
||||
"category": payload.category or "",
|
||||
"url": payload.url or "",
|
||||
"module": payload.module or "",
|
||||
"detail": payload.detail or "",
|
||||
},
|
||||
)
|
||||
await record_system_log(
|
||||
source="earth-client",
|
||||
service="earth",
|
||||
module=payload.module or "earth-client",
|
||||
event="earth.client.runtime_log",
|
||||
level=normalized_level,
|
||||
message=payload.message,
|
||||
category=payload.category or "client-runtime",
|
||||
context={
|
||||
"url": payload.url or "",
|
||||
"detail": payload.detail or "",
|
||||
"module": payload.module or "",
|
||||
"client_ip": request.client.host if request.client else "",
|
||||
},
|
||||
)
|
||||
return {"accepted": True, "source_id": "earth-client", "level": normalized_level}
|
||||
|
||||
@@ -6,12 +6,14 @@ Returns GeoJSON format compatible with Three.js, CesiumJS, and Unreal Cesium.
|
||||
|
||||
from datetime import UTC, datetime
|
||||
import math
|
||||
from fastapi import APIRouter, HTTPException, Depends, Query
|
||||
import httpx
|
||||
from fastapi import APIRouter, HTTPException, Depends, Query, Response
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from sqlalchemy import select, func
|
||||
from typing import List, Dict, Any, Optional
|
||||
|
||||
from app.core.collected_data_fields import get_record_field
|
||||
from app.core.countries import get_country_centroid
|
||||
from app.core.satellite_tle import build_tle_lines_from_elements
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.db.session import get_db
|
||||
@@ -21,8 +23,14 @@ from app.models.collected_data import CollectedData
|
||||
from app.services.bgp_collectors import build_bgp_collector_coverage
|
||||
from app.services.cable_graph import build_graph_from_data, CableGraph, haversine_distance
|
||||
from app.services.collectors.bgp_common import RIPE_RIS_COLLECTOR_COORDS
|
||||
from app.services.persistent_logs import record_system_log
|
||||
from app.core.logging import get_logger
|
||||
|
||||
router = APIRouter()
|
||||
logger = get_logger(__name__, service="api")
|
||||
TERRAIN_TILE_URL_TEMPLATE = (
|
||||
"https://s3.amazonaws.com/elevation-tiles-prod/terrarium/{z}/{x}/{y}.png"
|
||||
)
|
||||
|
||||
|
||||
# ============== Converter Functions ==============
|
||||
@@ -176,6 +184,12 @@ def convert_satellite_to_geojson(records: List[CollectedData]) -> Dict[str, Any]
|
||||
mean_motion=metadata.get("mean_motion"),
|
||||
)
|
||||
|
||||
constellation_group = _normalize_satellite_constellation_group(
|
||||
metadata.get("constellation_group"),
|
||||
record.name,
|
||||
)
|
||||
footprint_policy = _get_satellite_footprint_policy(constellation_group)
|
||||
|
||||
features.append(
|
||||
{
|
||||
"type": "Feature",
|
||||
@@ -185,6 +199,8 @@ def convert_satellite_to_geojson(records: List[CollectedData]) -> Dict[str, Any]
|
||||
"id": record.id,
|
||||
"norad_cat_id": norad_id,
|
||||
"name": record.name,
|
||||
"constellation_group": constellation_group,
|
||||
"footprint_policy": footprint_policy,
|
||||
"international_designator": metadata.get("international_designator"),
|
||||
"epoch": metadata.get("epoch"),
|
||||
"inclination": metadata.get("inclination"),
|
||||
@@ -205,6 +221,31 @@ def convert_satellite_to_geojson(records: List[CollectedData]) -> Dict[str, Any]
|
||||
return {"type": "FeatureCollection", "features": features}
|
||||
|
||||
|
||||
def _normalize_satellite_constellation_group(
|
||||
raw_group: Any,
|
||||
name: Optional[str],
|
||||
) -> Optional[str]:
|
||||
normalized_group = str(raw_group or "").strip().lower()
|
||||
if normalized_group:
|
||||
return normalized_group
|
||||
|
||||
normalized_name = str(name or "").strip().upper()
|
||||
if normalized_name.startswith("STARLINK"):
|
||||
return "starlink"
|
||||
if normalized_name.startswith("IRIDIUM"):
|
||||
return "iridium-next"
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def _get_satellite_footprint_policy(constellation_group: Optional[str]) -> str:
|
||||
if constellation_group == "starlink":
|
||||
return "starlink_ground_footprint"
|
||||
if constellation_group == "iridium-next":
|
||||
return "iridium_coverage_ring"
|
||||
return "none"
|
||||
|
||||
|
||||
def _current_collected_data_stmt(source: str):
|
||||
return (
|
||||
select(CollectedData)
|
||||
@@ -359,6 +400,215 @@ def convert_gpu_cluster_to_geojson(records: List[CollectedData]) -> Dict[str, An
|
||||
return {"type": "FeatureCollection", "features": features}
|
||||
|
||||
|
||||
def _parse_float(value: Any) -> Optional[float]:
|
||||
try:
|
||||
if value in (None, ""):
|
||||
return None
|
||||
return float(value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
COMPUTE_CENTER_COORDINATE_HINTS = (
|
||||
("el capitan", 37.6819, -121.7681),
|
||||
("livermore", 37.6819, -121.7681),
|
||||
("llnl", 37.6819, -121.7681),
|
||||
("lawrence livermore", 37.6819, -121.7681),
|
||||
("frontier", 35.9319, -84.3107),
|
||||
("oak ridge", 35.9319, -84.3107),
|
||||
("ornl", 35.9319, -84.3107),
|
||||
("aurora", 41.7130, -87.9820),
|
||||
("argonne", 41.7130, -87.9820),
|
||||
("anl", 41.7130, -87.9820),
|
||||
("fugaku", 34.6953, 135.1974),
|
||||
("kobe", 34.6953, 135.1974),
|
||||
("riken", 34.6953, 135.1974),
|
||||
("summit", 35.9319, -84.3107),
|
||||
("leonardo", 44.4949, 11.3426),
|
||||
("bologna", 44.4949, 11.3426),
|
||||
("alps", 46.0037, 8.9511),
|
||||
("lugano", 46.0037, 8.9511),
|
||||
("sunway taihulight", 31.4912, 120.3119),
|
||||
("wuxi", 31.4912, 120.3119),
|
||||
("tianhe-2", 23.1291, 113.2644),
|
||||
("tianhe-2a", 23.1291, 113.2644),
|
||||
("guangzhou", 23.1291, 113.2644),
|
||||
("colossus", 35.1495, -90.0490),
|
||||
("memphis", 35.1495, -90.0490),
|
||||
("xai", 35.1495, -90.0490),
|
||||
)
|
||||
|
||||
|
||||
def _normalize_hint_text(*parts: Any) -> str:
|
||||
return " ".join(
|
||||
str(part).strip().lower()
|
||||
for part in parts
|
||||
if part not in (None, "")
|
||||
)
|
||||
|
||||
|
||||
def _resolve_compute_center_coordinates(
|
||||
record: CollectedData,
|
||||
metadata: Dict[str, Any],
|
||||
) -> Dict[str, Any]:
|
||||
latitude = _parse_float(get_record_field(record, "latitude"))
|
||||
longitude = _parse_float(get_record_field(record, "longitude"))
|
||||
if latitude not in (None, 0.0) and longitude not in (None, 0.0):
|
||||
return {
|
||||
"latitude": latitude,
|
||||
"longitude": longitude,
|
||||
"location_precision": "precise",
|
||||
"geography_mode": "source_coordinates",
|
||||
"is_estimated": False,
|
||||
"estimated_reason": None,
|
||||
}
|
||||
|
||||
hint_text = _normalize_hint_text(
|
||||
record.name,
|
||||
get_record_field(record, "city"),
|
||||
get_record_field(record, "country"),
|
||||
metadata.get("site"),
|
||||
metadata.get("organization"),
|
||||
metadata.get("operator"),
|
||||
)
|
||||
for needle, resolved_latitude, resolved_longitude in COMPUTE_CENTER_COORDINATE_HINTS:
|
||||
if needle in hint_text:
|
||||
return {
|
||||
"latitude": resolved_latitude,
|
||||
"longitude": resolved_longitude,
|
||||
"location_precision": "estimated_site",
|
||||
"geography_mode": "site_hint",
|
||||
"is_estimated": True,
|
||||
"estimated_reason": f"Matched known site hint: {needle}",
|
||||
}
|
||||
|
||||
centroid = get_country_centroid(get_record_field(record, "country"))
|
||||
if centroid:
|
||||
return {
|
||||
"latitude": centroid.get("latitude"),
|
||||
"longitude": centroid.get("longitude"),
|
||||
"location_precision": "estimated_country",
|
||||
"geography_mode": "country_centroid",
|
||||
"is_estimated": True,
|
||||
"estimated_reason": "Estimated from country centroid",
|
||||
}
|
||||
|
||||
return {
|
||||
"latitude": latitude,
|
||||
"longitude": longitude,
|
||||
"location_precision": "unknown",
|
||||
"geography_mode": "unknown",
|
||||
"is_estimated": True,
|
||||
"estimated_reason": "No resolvable location hints",
|
||||
}
|
||||
|
||||
|
||||
def _normalize_capacity_band(capacity_value: Optional[float], capacity_unit: str) -> str:
|
||||
if capacity_value is None:
|
||||
return "unknown"
|
||||
|
||||
unit = str(capacity_unit or "").strip().lower()
|
||||
if unit in {"pflop/s", "pflops", "pflop"}:
|
||||
normalized_tflops = capacity_value * 1000
|
||||
elif unit in {"gflop/s", "gflops", "gflop"}:
|
||||
normalized_tflops = capacity_value / 1000
|
||||
else:
|
||||
normalized_tflops = capacity_value
|
||||
|
||||
if normalized_tflops >= 1_000_000:
|
||||
return "exascale"
|
||||
if normalized_tflops >= 100_000:
|
||||
return "ultra"
|
||||
if normalized_tflops >= 10_000:
|
||||
return "large"
|
||||
if normalized_tflops > 0:
|
||||
return "regional"
|
||||
return "unknown"
|
||||
|
||||
|
||||
def convert_compute_centers_to_geojson(records: List[CollectedData]) -> Dict[str, Any]:
|
||||
"""Convert compute infrastructure records into a unified GeoJSON layer."""
|
||||
features = []
|
||||
|
||||
for record in records:
|
||||
metadata = record.extra_data or {}
|
||||
coordinate_info = _resolve_compute_center_coordinates(record, metadata)
|
||||
latitude = coordinate_info.get("latitude")
|
||||
longitude = coordinate_info.get("longitude")
|
||||
site_type = (
|
||||
"supercomputer"
|
||||
if record.source == "top500" or record.data_type == "supercomputer"
|
||||
else "gpu_cluster"
|
||||
)
|
||||
if latitude in (None, 0.0) or longitude in (None, 0.0):
|
||||
continue
|
||||
|
||||
if site_type == "supercomputer":
|
||||
capacity_value = _parse_float(get_record_field(record, "rmax"))
|
||||
capacity_unit = "GFlops"
|
||||
else:
|
||||
capacity_value = _parse_float(get_record_field(record, "value"))
|
||||
capacity_unit = str(get_record_field(record, "unit") or "TFlop/s")
|
||||
|
||||
vendor = (
|
||||
metadata.get("manufacturer")
|
||||
or metadata.get("vendor")
|
||||
or metadata.get("gpu_type")
|
||||
)
|
||||
operator = (
|
||||
metadata.get("organization")
|
||||
or metadata.get("operator")
|
||||
or metadata.get("owner")
|
||||
)
|
||||
rank = metadata.get("rank")
|
||||
if rank in (None, "") and site_type == "supercomputer":
|
||||
rank = get_record_field(record, "rank")
|
||||
|
||||
updated_at = to_iso8601_utc(record.reference_date or record.collected_at)
|
||||
|
||||
features.append(
|
||||
{
|
||||
"type": "Feature",
|
||||
"id": record.id,
|
||||
"geometry": {
|
||||
"type": "Point",
|
||||
"coordinates": [longitude or 0, latitude or 0],
|
||||
},
|
||||
"properties": {
|
||||
"id": record.id,
|
||||
"source_id": record.source_id,
|
||||
"name": record.name,
|
||||
"site_type": site_type,
|
||||
"country": get_record_field(record, "country"),
|
||||
"city": get_record_field(record, "city"),
|
||||
"latitude": latitude,
|
||||
"longitude": longitude,
|
||||
"operator": operator,
|
||||
"vendor": vendor,
|
||||
"capacity_value": capacity_value,
|
||||
"capacity_unit": capacity_unit,
|
||||
"capacity_band": _normalize_capacity_band(capacity_value, capacity_unit),
|
||||
"rank": rank,
|
||||
"gpu_count": metadata.get("gpu_count"),
|
||||
"gpu_type": metadata.get("gpu_type"),
|
||||
"cores": get_record_field(record, "cores"),
|
||||
"power": get_record_field(record, "power"),
|
||||
"source": record.source,
|
||||
"updated_at": updated_at,
|
||||
"status": "observed",
|
||||
"location_precision": coordinate_info.get("location_precision"),
|
||||
"geography_mode": coordinate_info.get("geography_mode"),
|
||||
"is_estimated": coordinate_info.get("is_estimated", False),
|
||||
"estimated_reason": coordinate_info.get("estimated_reason"),
|
||||
"data_type": "compute_center",
|
||||
"metadata": metadata,
|
||||
},
|
||||
}
|
||||
)
|
||||
|
||||
return {"type": "FeatureCollection", "features": features}
|
||||
|
||||
|
||||
def convert_bgp_anomalies_to_geojson(
|
||||
records: List[BGPAnomaly],
|
||||
geography_hints: Optional[Dict[str, Dict[str, Any]]] = None,
|
||||
@@ -776,15 +1026,41 @@ async def get_cables_geojson(db: AsyncSession = Depends(get_db)):
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.exception_event(
|
||||
"Failed to build cables GeoJSON response",
|
||||
event="visualization.cables.load_failed",
|
||||
context={"error": str(e)},
|
||||
)
|
||||
await record_system_log(
|
||||
source="backend",
|
||||
service="api",
|
||||
module=__name__,
|
||||
event="visualization.cables.load_failed",
|
||||
level="error",
|
||||
message="Failed to build cables GeoJSON response",
|
||||
category="visualization",
|
||||
context={"error": str(e)},
|
||||
)
|
||||
raise HTTPException(status_code=500, detail=f"Internal error: {str(e)}")
|
||||
|
||||
|
||||
@router.get("/geo/landing-points")
|
||||
async def get_landing_points_geojson(db: AsyncSession = Depends(get_db)):
|
||||
try:
|
||||
records = await _load_current_collected_data(db, "arcgis_landing_points")
|
||||
relation_records = await _load_current_collected_data(db, "arcgis_cable_landing_relation")
|
||||
cable_records = await _load_current_collected_data(db, "arcgis_cables")
|
||||
records_by_source = await _load_current_collected_data_by_sources(
|
||||
db,
|
||||
[
|
||||
"arcgis_landing_points",
|
||||
"arcgis_cable_landing_relation",
|
||||
"arcgis_cables",
|
||||
],
|
||||
)
|
||||
records = records_by_source.get("arcgis_landing_points", [])
|
||||
relation_records = records_by_source.get(
|
||||
"arcgis_cable_landing_relation",
|
||||
[],
|
||||
)
|
||||
cable_records = records_by_source.get("arcgis_cables", [])
|
||||
|
||||
city_to_cable_ids_map, cable_id_to_name_map = _build_landing_point_cable_maps(
|
||||
relation_records,
|
||||
@@ -801,9 +1077,68 @@ async def get_landing_points_geojson(db: AsyncSession = Depends(get_db)):
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.exception_event(
|
||||
"Failed to build landing points GeoJSON response",
|
||||
event="visualization.landing_points.load_failed",
|
||||
context={"error": str(e)},
|
||||
)
|
||||
await record_system_log(
|
||||
source="backend",
|
||||
service="api",
|
||||
module=__name__,
|
||||
event="visualization.landing_points.load_failed",
|
||||
level="error",
|
||||
message="Failed to build landing points GeoJSON response",
|
||||
category="visualization",
|
||||
context={"error": str(e)},
|
||||
)
|
||||
raise HTTPException(status_code=500, detail=f"Internal error: {str(e)}")
|
||||
|
||||
|
||||
@router.get("/terrain/terrarium/{z}/{x}/{y}.png")
|
||||
async def get_terrarium_tile(z: int, x: int, y: int):
|
||||
"""Proxy Terrarium elevation tiles through the backend to avoid browser CORS issues."""
|
||||
if z < 0 or x < 0 or y < 0:
|
||||
raise HTTPException(status_code=400, detail="Invalid terrain tile coordinates")
|
||||
|
||||
url = TERRAIN_TILE_URL_TEMPLATE.format(z=z, x=x, y=y)
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(
|
||||
timeout=20.0,
|
||||
follow_redirects=True,
|
||||
) as client:
|
||||
upstream = await client.get(url)
|
||||
upstream.raise_for_status()
|
||||
except httpx.HTTPStatusError as exc:
|
||||
raise HTTPException(
|
||||
status_code=exc.response.status_code,
|
||||
detail=f"Terrain tile upstream error: {exc.response.status_code}",
|
||||
) from exc
|
||||
except httpx.HTTPError as exc:
|
||||
raise HTTPException(
|
||||
status_code=502,
|
||||
detail=f"Terrain tile fetch failed: {exc}",
|
||||
) from exc
|
||||
|
||||
cache_control = upstream.headers.get("cache-control") or "public, max-age=86400"
|
||||
etag = upstream.headers.get("etag")
|
||||
last_modified = upstream.headers.get("last-modified")
|
||||
headers = {
|
||||
"Cache-Control": cache_control,
|
||||
}
|
||||
if etag:
|
||||
headers["ETag"] = etag
|
||||
if last_modified:
|
||||
headers["Last-Modified"] = last_modified
|
||||
|
||||
return Response(
|
||||
content=upstream.content,
|
||||
media_type=upstream.headers.get("content-type", "image/png"),
|
||||
headers=headers,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/geo/all")
|
||||
async def get_all_geojson(db: AsyncSession = Depends(get_db)):
|
||||
records_by_source = await _load_current_collected_data_by_sources(
|
||||
@@ -916,6 +1251,53 @@ async def get_gpu_clusters_geojson(
|
||||
}
|
||||
|
||||
|
||||
@router.get("/geo/compute-centers")
|
||||
async def get_compute_centers_geojson(
|
||||
limit: int = Query(200, ge=1, le=1000),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""获取统一算力中心 GeoJSON 数据"""
|
||||
records_by_source = await _load_current_collected_data_by_sources(
|
||||
db,
|
||||
["top500", "epoch_ai_gpu"],
|
||||
)
|
||||
records = _filter_known_records(
|
||||
records_by_source.get("top500", []) + records_by_source.get("epoch_ai_gpu", []),
|
||||
)
|
||||
if limit is not None:
|
||||
records = records[:limit]
|
||||
|
||||
if not records:
|
||||
return {
|
||||
"type": "FeatureCollection",
|
||||
"features": [],
|
||||
"count": 0,
|
||||
"stats": {
|
||||
"total": 0,
|
||||
"supercomputers": 0,
|
||||
"gpu_clusters": 0,
|
||||
},
|
||||
}
|
||||
|
||||
geojson = convert_compute_centers_to_geojson(records)
|
||||
features = geojson.get("features", [])
|
||||
return {
|
||||
**geojson,
|
||||
"count": len(features),
|
||||
"stats": {
|
||||
"total": len(features),
|
||||
"supercomputers": sum(
|
||||
1 for feature in features
|
||||
if feature.get("properties", {}).get("site_type") == "supercomputer"
|
||||
),
|
||||
"gpu_clusters": sum(
|
||||
1 for feature in features
|
||||
if feature.get("properties", {}).get("site_type") == "gpu_cluster"
|
||||
),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@router.get("/geo/bgp-anomalies")
|
||||
async def get_bgp_anomalies_geojson(
|
||||
severity: Optional[str] = Query(None),
|
||||
@@ -971,6 +1353,71 @@ async def get_bgp_collectors_geojson(db: AsyncSession = Depends(get_db)):
|
||||
return {**geojson, "count": len(geojson.get("features", []))}
|
||||
|
||||
|
||||
@router.get("/geo/summary")
|
||||
async def get_visualization_geo_summary(db: AsyncSession = Depends(get_db)):
|
||||
"""Return lightweight Earth HUD counts without loading layer GeoJSON payloads."""
|
||||
records_by_source = await _load_current_collected_data_by_sources(
|
||||
db,
|
||||
[
|
||||
"arcgis_cables",
|
||||
"arcgis_landing_points",
|
||||
"celestrak_tle",
|
||||
"top500",
|
||||
"epoch_ai_gpu",
|
||||
],
|
||||
)
|
||||
|
||||
cables = convert_cable_to_geojson(records_by_source.get("arcgis_cables", []))
|
||||
landing_points = convert_landing_point_to_geojson(
|
||||
records_by_source.get("arcgis_landing_points", []),
|
||||
)
|
||||
satellites = convert_satellite_to_geojson(
|
||||
_filter_known_records(records_by_source.get("celestrak_tle", [])),
|
||||
)
|
||||
compute_centers = convert_compute_centers_to_geojson(
|
||||
_filter_known_records(
|
||||
records_by_source.get("top500", [])
|
||||
+ records_by_source.get("epoch_ai_gpu", []),
|
||||
),
|
||||
)
|
||||
compute_features = compute_centers.get("features", [])
|
||||
|
||||
active_incident_result = await db.execute(
|
||||
select(func.count(BGPIncident.id)).where(BGPIncident.status == "active"),
|
||||
)
|
||||
active_anomaly_result = await db.execute(
|
||||
select(func.count(BGPAnomaly.id)).where(BGPAnomaly.status == "active"),
|
||||
)
|
||||
active_incident_count = int(active_incident_result.scalar() or 0)
|
||||
active_anomaly_count = int(active_anomaly_result.scalar() or 0)
|
||||
bgp_collectors = await build_bgp_collector_coverage(
|
||||
db,
|
||||
source_filter=("ris_live_bgp", "bgpstream_bgp"),
|
||||
)
|
||||
|
||||
return {
|
||||
"generated_at": to_iso8601_utc(datetime.now(UTC)),
|
||||
"stats": {
|
||||
"cable_count": len(cables.get("features", [])),
|
||||
"landing_point_count": len(landing_points.get("features", [])),
|
||||
"satellite_count": len(satellites.get("features", [])),
|
||||
"compute_center_count": len(compute_features),
|
||||
"supercomputer_count": sum(
|
||||
1 for feature in compute_features
|
||||
if feature.get("properties", {}).get("site_type") == "supercomputer"
|
||||
),
|
||||
"gpu_cluster_count": sum(
|
||||
1 for feature in compute_features
|
||||
if feature.get("properties", {}).get("site_type") == "gpu_cluster"
|
||||
),
|
||||
"bgp_event_count": active_incident_count or active_anomaly_count,
|
||||
"bgp_incident_count": active_incident_count,
|
||||
"bgp_anomaly_count": active_anomaly_count,
|
||||
"bgp_collector_count": len([item for item in bgp_collectors if item.get("collector")]),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@router.get("/all")
|
||||
async def get_all_visualization_data(db: AsyncSession = Depends(get_db)):
|
||||
"""获取所有可视化数据的统一端点
|
||||
|
||||
@@ -2,7 +2,6 @@
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
from datetime import UTC, datetime
|
||||
from typing import Optional
|
||||
|
||||
@@ -10,10 +9,11 @@ from fastapi import APIRouter, WebSocket, WebSocketDisconnect, Query
|
||||
from jose import jwt, JWTError
|
||||
|
||||
from app.core.config import settings
|
||||
from app.core.logging import get_logger
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.core.websocket.manager import manager
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
logger = get_logger(__name__, service="api")
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
@@ -22,11 +22,18 @@ async def authenticate_token(token: str) -> Optional[dict]:
|
||||
try:
|
||||
payload = jwt.decode(token, settings.SECRET_KEY, algorithms=[settings.ALGORITHM])
|
||||
if payload.get("type") != "access":
|
||||
logger.warning(f"WebSocket auth failed: wrong token type")
|
||||
logger.warning_event(
|
||||
"WebSocket auth failed: wrong token type",
|
||||
event="auth.websocket.invalid_token_type",
|
||||
)
|
||||
return None
|
||||
return payload
|
||||
except JWTError as e:
|
||||
logger.warning(f"WebSocket auth failed: {e}")
|
||||
logger.warning_event(
|
||||
"WebSocket auth failed",
|
||||
event="auth.websocket.decode_failed",
|
||||
context={"error": str(e)},
|
||||
)
|
||||
return None
|
||||
|
||||
|
||||
@@ -36,10 +43,17 @@ async def websocket_endpoint(
|
||||
token: str = Query(...),
|
||||
):
|
||||
"""WebSocket endpoint for real-time data"""
|
||||
logger.info(f"WebSocket connection attempt with token: {token[:20]}...")
|
||||
logger.info_event(
|
||||
"WebSocket connection attempt",
|
||||
event="auth.websocket.connection_attempt",
|
||||
context={"token_preview": f"{token[:8]}..."},
|
||||
)
|
||||
payload = await authenticate_token(token)
|
||||
if payload is None:
|
||||
logger.warning("WebSocket authentication failed, closing connection")
|
||||
logger.warning_event(
|
||||
"WebSocket authentication failed, closing connection",
|
||||
event="auth.websocket.connection_rejected",
|
||||
)
|
||||
await websocket.close(code=4001)
|
||||
return
|
||||
|
||||
|
||||
@@ -1,15 +1,15 @@
|
||||
"""Redis caching service"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
from datetime import timedelta
|
||||
from typing import Optional, Any
|
||||
|
||||
import redis
|
||||
|
||||
from app.core.config import settings
|
||||
from app.core.logging import get_logger
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
# Lazy Redis client initialization
|
||||
@@ -47,7 +47,7 @@ class CacheService:
|
||||
return json.loads(value)
|
||||
return None
|
||||
except Exception as e:
|
||||
logger.warning(f"Cache get error: {e}")
|
||||
logger.warning_event("Cache get error", event="cache.get.failed", context={"error": str(e)})
|
||||
return None
|
||||
|
||||
def set(
|
||||
@@ -61,7 +61,7 @@ class CacheService:
|
||||
serialized = json.dumps(value, default=str)
|
||||
return self.client.setex(key, expire_seconds, serialized)
|
||||
except Exception as e:
|
||||
logger.warning(f"Cache set error: {e}")
|
||||
logger.warning_event("Cache set error", event="cache.set.failed", context={"error": str(e)})
|
||||
return False
|
||||
|
||||
def delete(self, key: str) -> bool:
|
||||
@@ -69,7 +69,7 @@ class CacheService:
|
||||
try:
|
||||
return self.client.delete(key) > 0
|
||||
except Exception as e:
|
||||
logger.warning(f"Cache delete error: {e}")
|
||||
logger.warning_event("Cache delete error", event="cache.delete.failed", context={"error": str(e)})
|
||||
return False
|
||||
|
||||
def delete_pattern(self, pattern: str) -> int:
|
||||
@@ -80,7 +80,7 @@ class CacheService:
|
||||
return self.client.delete(*keys)
|
||||
return 0
|
||||
except Exception as e:
|
||||
logger.warning(f"Cache delete_pattern error: {e}")
|
||||
logger.warning_event("Cache delete_pattern error", event="cache.delete_pattern.failed", context={"error": str(e)})
|
||||
return 0
|
||||
|
||||
def get_or_set(
|
||||
|
||||
@@ -30,6 +30,7 @@ COLLECTOR_URL_KEYS = {
|
||||
"iptoasn_prefix_geo": "iptoasn.combined_url",
|
||||
"opengeofeed_prefix_geo": "opengeofeed.public_csv_url",
|
||||
"nro_delegated_prefix_geo": "nro.delegated_stats_url",
|
||||
"news_live_streams": "news_live_streams.channels_url",
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -86,3 +86,11 @@ opengeofeed:
|
||||
nro:
|
||||
# NRO delegated stats 下载地址
|
||||
delegated_stats_url: "https://ftp.ripe.net/pub/stats/ripencc/nro-stats/latest/nro-delegated-stats"
|
||||
|
||||
news_live_streams:
|
||||
# IPTV-org 频道元数据 JSON
|
||||
channels_url: "https://iptv-org.github.io/api/channels.json"
|
||||
# IPTV-org 频道播放流 JSON
|
||||
streams_url: "https://iptv-org.github.io/api/streams.json"
|
||||
# IPTV-org 台标 JSON
|
||||
logos_url: "https://iptv-org.github.io/api/logos.json"
|
||||
|
||||
161
backend/app/core/logging.py
Normal file
161
backend/app/core/logging.py
Normal file
@@ -0,0 +1,161 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
|
||||
from collections.abc import Mapping, Sequence
|
||||
from typing import Any
|
||||
|
||||
from app.core.request_context import get_request_id
|
||||
|
||||
DEFAULT_SERVICE = "backend"
|
||||
DEFAULT_EVENT = "app.log"
|
||||
DEFAULT_LOG_LEVEL = os.getenv("PLANET_LOG_LEVEL", "INFO").upper()
|
||||
REDACTED = "[REDACTED]"
|
||||
SENSITIVE_FIELD_NAMES = {
|
||||
"access_token",
|
||||
"api_key",
|
||||
"authorization",
|
||||
"cookie",
|
||||
"password",
|
||||
"refresh_token",
|
||||
"secret",
|
||||
"token",
|
||||
}
|
||||
SENSITIVE_TEXT_PATTERNS = (
|
||||
re.compile(r"(?i)(authorization\s*[:=]\s*)(.+)"),
|
||||
re.compile(r"(?i)(bearer\s+)([A-Za-z0-9._\-]+)"),
|
||||
re.compile(r"(?i)(token\s*[:=]\s*)(.+)"),
|
||||
re.compile(r"(?i)(password\s*[:=]\s*)(.+)"),
|
||||
re.compile(r"(?i)(cookie\s*[:=]\s*)(.+)"),
|
||||
)
|
||||
|
||||
|
||||
def sanitize_log_value(value: Any) -> Any:
|
||||
if isinstance(value, Mapping):
|
||||
return {
|
||||
str(key): (REDACTED if str(key).lower() in SENSITIVE_FIELD_NAMES else sanitize_log_value(item))
|
||||
for key, item in value.items()
|
||||
}
|
||||
if isinstance(value, Sequence) and not isinstance(value, (str, bytes, bytearray)):
|
||||
return [sanitize_log_value(item) for item in value]
|
||||
if isinstance(value, str):
|
||||
sanitized = value
|
||||
for pattern in SENSITIVE_TEXT_PATTERNS:
|
||||
sanitized = pattern.sub(lambda match: f"{match.group(1)}{REDACTED}", sanitized)
|
||||
return sanitized
|
||||
return value
|
||||
|
||||
|
||||
def _normalize_context(context: Any) -> dict[str, Any]:
|
||||
if context is None:
|
||||
return {}
|
||||
if isinstance(context, Mapping):
|
||||
sanitized = sanitize_log_value(context)
|
||||
return {str(key): value for key, value in sanitized.items()}
|
||||
return {"value": sanitize_log_value(context)}
|
||||
|
||||
|
||||
class PlanetContextFilter(logging.Filter):
|
||||
def filter(self, record: logging.LogRecord) -> bool:
|
||||
record.request_id = getattr(record, "request_id", None) or get_request_id() or "-"
|
||||
record.service = getattr(record, "service", None) or DEFAULT_SERVICE
|
||||
record.event = getattr(record, "event", None) or DEFAULT_EVENT
|
||||
record.context = _normalize_context(getattr(record, "context", None))
|
||||
record.message = sanitize_log_value(record.getMessage())
|
||||
return True
|
||||
|
||||
|
||||
class PlanetFormatter(logging.Formatter):
|
||||
def format(self, record: logging.LogRecord) -> str:
|
||||
timestamp = self.formatTime(record, self.datefmt)
|
||||
level = record.levelname
|
||||
service = getattr(record, "service", DEFAULT_SERVICE)
|
||||
module_name = record.name
|
||||
event = getattr(record, "event", DEFAULT_EVENT)
|
||||
request_id = getattr(record, "request_id", "-")
|
||||
message = sanitize_log_value(record.getMessage())
|
||||
context = _normalize_context(getattr(record, "context", None))
|
||||
context_suffix = ""
|
||||
if context:
|
||||
context_suffix = f" context={json.dumps(context, ensure_ascii=False, sort_keys=True)}"
|
||||
rendered = (
|
||||
f"{timestamp} {level} service={service} module={module_name} "
|
||||
f"event={event} request_id={request_id} message={message}{context_suffix}"
|
||||
)
|
||||
if record.exc_info:
|
||||
rendered = f"{rendered}\n{self.formatException(record.exc_info)}"
|
||||
return rendered
|
||||
|
||||
|
||||
class PlanetLoggerAdapter(logging.LoggerAdapter):
|
||||
def process(self, msg: Any, kwargs: dict[str, Any]) -> tuple[Any, dict[str, Any]]:
|
||||
extra = dict(self.extra)
|
||||
extra.update(kwargs.get("extra", {}))
|
||||
if "context" in extra:
|
||||
extra["context"] = _normalize_context(extra.get("context"))
|
||||
kwargs["extra"] = extra
|
||||
return sanitize_log_value(msg), kwargs
|
||||
|
||||
def log_event(
|
||||
self,
|
||||
level: int,
|
||||
message: str,
|
||||
*,
|
||||
event: str,
|
||||
context: Mapping[str, Any] | None = None,
|
||||
**extra: Any,
|
||||
) -> None:
|
||||
self.log(level, message, extra={"event": event, "context": context or {}, **extra})
|
||||
|
||||
def debug_event(self, message: str, *, event: str, context: Mapping[str, Any] | None = None, **extra: Any) -> None:
|
||||
self.log_event(logging.DEBUG, message, event=event, context=context, **extra)
|
||||
|
||||
def info_event(self, message: str, *, event: str, context: Mapping[str, Any] | None = None, **extra: Any) -> None:
|
||||
self.log_event(logging.INFO, message, event=event, context=context, **extra)
|
||||
|
||||
def warning_event(self, message: str, *, event: str, context: Mapping[str, Any] | None = None, **extra: Any) -> None:
|
||||
self.log_event(logging.WARNING, message, event=event, context=context, **extra)
|
||||
|
||||
def error_event(self, message: str, *, event: str, context: Mapping[str, Any] | None = None, **extra: Any) -> None:
|
||||
self.log_event(logging.ERROR, message, event=event, context=context, **extra)
|
||||
|
||||
def exception_event(
|
||||
self,
|
||||
message: str,
|
||||
*,
|
||||
event: str,
|
||||
context: Mapping[str, Any] | None = None,
|
||||
**extra: Any,
|
||||
) -> None:
|
||||
self.error(message, exc_info=True, extra={"event": event, "context": context or {}, **extra})
|
||||
|
||||
|
||||
def get_logger(name: str, *, service: str = DEFAULT_SERVICE) -> PlanetLoggerAdapter:
|
||||
return PlanetLoggerAdapter(logging.getLogger(name), {"service": service})
|
||||
|
||||
|
||||
def configure_logging(level: str | None = None) -> None:
|
||||
root_logger = logging.getLogger()
|
||||
if getattr(configure_logging, "_configured", False):
|
||||
if level:
|
||||
root_logger.setLevel(level.upper())
|
||||
return
|
||||
|
||||
handler = logging.StreamHandler()
|
||||
handler.setFormatter(PlanetFormatter(datefmt="%Y-%m-%d %H:%M:%S"))
|
||||
handler.addFilter(PlanetContextFilter())
|
||||
|
||||
root_logger.handlers.clear()
|
||||
root_logger.addHandler(handler)
|
||||
root_logger.setLevel((level or DEFAULT_LOG_LEVEL).upper())
|
||||
|
||||
for logger_name in ("uvicorn", "uvicorn.error", "uvicorn.access"):
|
||||
target_logger = logging.getLogger(logger_name)
|
||||
target_logger.handlers.clear()
|
||||
target_logger.propagate = True
|
||||
|
||||
logging.captureWarnings(True)
|
||||
configure_logging._configured = True
|
||||
14
backend/app/core/request_context.py
Normal file
14
backend/app/core/request_context.py
Normal file
@@ -0,0 +1,14 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from contextvars import ContextVar
|
||||
|
||||
|
||||
request_id_context: ContextVar[str | None] = ContextVar("request_id", default=None)
|
||||
|
||||
|
||||
def set_request_id(request_id: str | None) -> None:
|
||||
request_id_context.set(request_id)
|
||||
|
||||
|
||||
def get_request_id() -> str | None:
|
||||
return request_id_context.get()
|
||||
@@ -5,10 +5,22 @@ from sqlalchemy.ext.asyncio import AsyncSession, create_async_engine, async_sess
|
||||
from sqlalchemy.orm import declarative_base
|
||||
|
||||
from app.core.config import settings
|
||||
from app.core.logging import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
DB_POOL_CONFIG = {
|
||||
"pool_pre_ping": True,
|
||||
"pool_recycle": 1800,
|
||||
"pool_size": 10,
|
||||
"max_overflow": 20,
|
||||
"pool_timeout": 30,
|
||||
}
|
||||
|
||||
engine = create_async_engine(
|
||||
settings.DATABASE_URL,
|
||||
echo=settings.DEBUG if hasattr(settings, "DEBUG") else False,
|
||||
**DB_POOL_CONFIG,
|
||||
)
|
||||
|
||||
async_session_factory = async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False)
|
||||
@@ -97,6 +109,19 @@ async def init_db():
|
||||
import app.models.system_setting # noqa: F401
|
||||
import app.models.playground_session # noqa: F401
|
||||
import app.models.playground_message # noqa: F401
|
||||
import app.models.system_log # noqa: F401
|
||||
|
||||
logger.warning_event(
|
||||
"Database pool settings active",
|
||||
event="database.pool.initialized",
|
||||
context={
|
||||
"pool_pre_ping": DB_POOL_CONFIG["pool_pre_ping"],
|
||||
"pool_recycle": DB_POOL_CONFIG["pool_recycle"],
|
||||
"pool_size": DB_POOL_CONFIG["pool_size"],
|
||||
"max_overflow": DB_POOL_CONFIG["max_overflow"],
|
||||
"pool_timeout": DB_POOL_CONFIG["pool_timeout"],
|
||||
},
|
||||
)
|
||||
|
||||
async with engine.begin() as conn:
|
||||
await conn.run_sync(Base.metadata.create_all)
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
from contextlib import asynccontextmanager
|
||||
from uuid import uuid4
|
||||
|
||||
from fastapi import FastAPI
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
@@ -7,6 +8,8 @@ from starlette.middleware.base import BaseHTTPMiddleware
|
||||
from app.api.main import api_router
|
||||
from app.api.v1 import websocket
|
||||
from app.core.config import settings
|
||||
from app.core.logging import configure_logging
|
||||
from app.core.request_context import set_request_id
|
||||
from app.core.websocket.broadcaster import broadcaster
|
||||
from app.db.session import init_db
|
||||
from app.services.scheduler import (
|
||||
@@ -17,6 +20,9 @@ from app.services.scheduler import (
|
||||
)
|
||||
|
||||
|
||||
configure_logging()
|
||||
|
||||
|
||||
class WebSocketCORSMiddleware(BaseHTTPMiddleware):
|
||||
async def dispatch(self, request, call_next):
|
||||
if request.url.path.startswith("/ws") and request.method == "GET":
|
||||
@@ -28,6 +34,18 @@ class WebSocketCORSMiddleware(BaseHTTPMiddleware):
|
||||
return await call_next(request)
|
||||
|
||||
|
||||
class RequestContextMiddleware(BaseHTTPMiddleware):
|
||||
async def dispatch(self, request, call_next):
|
||||
request_id = request.headers.get("X-Request-ID") or uuid4().hex
|
||||
set_request_id(request_id)
|
||||
try:
|
||||
response = await call_next(request)
|
||||
finally:
|
||||
set_request_id(None)
|
||||
response.headers["X-Request-ID"] = request_id
|
||||
return response
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
await init_db()
|
||||
@@ -58,6 +76,7 @@ app.add_middleware(
|
||||
allow_headers=["*"],
|
||||
)
|
||||
|
||||
app.add_middleware(RequestContextMiddleware)
|
||||
app.add_middleware(WebSocketCORSMiddleware)
|
||||
|
||||
app.include_router(api_router, prefix="/api/v1")
|
||||
|
||||
@@ -11,6 +11,7 @@ from app.models.bgp_observation import BGPObservation
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.models.playground_session import PlaygroundSession
|
||||
from app.models.playground_message import PlaygroundMessage
|
||||
from app.models.system_log import SystemLog, AuditLog
|
||||
|
||||
__all__ = [
|
||||
"User",
|
||||
@@ -26,4 +27,6 @@ __all__ = [
|
||||
"BGPAnomaly",
|
||||
"BGPIncident",
|
||||
"BGPObservation",
|
||||
"SystemLog",
|
||||
"AuditLog",
|
||||
]
|
||||
|
||||
40
backend/app/models/system_log.py
Normal file
40
backend/app/models/system_log.py
Normal file
@@ -0,0 +1,40 @@
|
||||
from sqlalchemy import JSON, Column, DateTime, Integer, String, Text
|
||||
from sqlalchemy.sql import func
|
||||
|
||||
from app.db.session import Base
|
||||
|
||||
|
||||
class SystemLog(Base):
|
||||
__tablename__ = "system_logs"
|
||||
|
||||
id = Column(Integer, primary_key=True, autoincrement=True)
|
||||
occurred_at = Column(DateTime(timezone=True), server_default=func.now(), index=True)
|
||||
source = Column(String(50), nullable=False, index=True)
|
||||
service = Column(String(50), nullable=True)
|
||||
module = Column(String(120), nullable=True)
|
||||
event = Column(String(160), nullable=True, index=True)
|
||||
level = Column(String(20), nullable=False, index=True)
|
||||
message = Column(Text, nullable=False)
|
||||
request_id = Column(String(64), nullable=True, index=True)
|
||||
trace_id = Column(String(64), nullable=True)
|
||||
user_id = Column(Integer, nullable=True, index=True)
|
||||
category = Column(String(80), nullable=True, index=True)
|
||||
context = Column(JSON, nullable=False, default=dict)
|
||||
created_at = Column(DateTime(timezone=True), server_default=func.now())
|
||||
|
||||
|
||||
class AuditLog(Base):
|
||||
__tablename__ = "audit_logs"
|
||||
|
||||
id = Column(Integer, primary_key=True, autoincrement=True)
|
||||
occurred_at = Column(DateTime(timezone=True), server_default=func.now(), index=True)
|
||||
actor_id = Column(Integer, nullable=True, index=True)
|
||||
actor_name = Column(String(255), nullable=True)
|
||||
action = Column(String(120), nullable=False, index=True)
|
||||
target_type = Column(String(80), nullable=True)
|
||||
target_id = Column(String(120), nullable=True)
|
||||
result = Column(String(40), nullable=True, index=True)
|
||||
request_id = Column(String(64), nullable=True, index=True)
|
||||
ip = Column(String(64), nullable=True)
|
||||
details = Column(JSON, nullable=False, default=dict)
|
||||
created_at = Column(DateTime(timezone=True), server_default=func.now())
|
||||
@@ -46,6 +46,9 @@ class CelesTrakTLECollector(BaseCollector):
|
||||
if response.status_code == 200:
|
||||
data = response.json()
|
||||
if isinstance(data, list):
|
||||
for item in data:
|
||||
if isinstance(item, dict):
|
||||
item["_celestrak_group"] = group
|
||||
all_satellites.extend(data)
|
||||
print(f"CelesTrak: Fetched {len(data)} satellites from group '{group}'")
|
||||
except Exception as e:
|
||||
@@ -78,6 +81,7 @@ class CelesTrakTLECollector(BaseCollector):
|
||||
"name": item.get("OBJECT_NAME", "Unknown"),
|
||||
"reference_date": item.get("EPOCH", ""),
|
||||
"metadata": {
|
||||
"constellation_group": item.get("_celestrak_group"),
|
||||
"norad_cat_id": item.get("NORAD_CAT_ID"),
|
||||
"international_designator": item.get("OBJECT_ID"),
|
||||
"epoch": item.get("EPOCH"),
|
||||
|
||||
@@ -1,10 +1,16 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
from urllib.parse import urlparse
|
||||
|
||||
import httpx
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
from app.models.datasource_config import DataSourceConfig
|
||||
from app.services.collectors.base import BaseCollector
|
||||
|
||||
|
||||
@@ -18,52 +24,537 @@ class NewsLiveStreamsCollector(BaseCollector):
|
||||
data_type = "news_live_stream"
|
||||
fail_on_empty = False
|
||||
|
||||
DEFAULT_TIMEOUT = 45.0
|
||||
DEFAULT_HEADERS = {
|
||||
"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
|
||||
"Accept": "application/json",
|
||||
}
|
||||
RESPONSE_CANDIDATE_KEYS = ("sources", "streams", "channels", "items", "results", "data")
|
||||
DEFAULT_ADAPTER = "iptv_org"
|
||||
DEFAULT_IPTV_ORG_STREAMS_URL = "https://iptv-org.github.io/api/streams.json"
|
||||
DEFAULT_IPTV_ORG_LOGOS_URL = "https://iptv-org.github.io/api/logos.json"
|
||||
DEFAULT_IPTV_ORG_NEWS_CATEGORIES = ("news", "business", "weather")
|
||||
DEFAULT_IPTV_ORG_EXCLUDE_CATEGORIES = ("music", "sports", "kids", "entertainment")
|
||||
DEFAULT_IPTV_ORG_MAX_SOURCES = 120
|
||||
|
||||
async def fetch(self) -> list[dict[str, Any]]:
|
||||
request_url = (self._resolved_url or "").strip()
|
||||
if not request_url:
|
||||
return []
|
||||
|
||||
async with httpx.AsyncClient(timeout=45.0, follow_redirects=True) as client:
|
||||
response = await client.get(
|
||||
datasource_config = await self._load_datasource_config()
|
||||
effective_config = self._get_effective_config(datasource_config)
|
||||
adapter = str(effective_config.get("adapter") or "").strip().lower()
|
||||
if adapter == "iptv_org":
|
||||
return await self._fetch_iptv_org(request_url, effective_config)
|
||||
|
||||
request_headers = self._build_request_headers(datasource_config)
|
||||
request_config = self._get_request_config(datasource_config)
|
||||
request_params = self._build_request_params(datasource_config)
|
||||
request_json = self._build_request_json_body(datasource_config)
|
||||
request_data = self._build_request_form_body(datasource_config)
|
||||
timeout = self._get_timeout(datasource_config)
|
||||
|
||||
async with httpx.AsyncClient(timeout=timeout, follow_redirects=True) as client:
|
||||
response = await client.request(
|
||||
request_config["method"],
|
||||
request_url,
|
||||
headers={
|
||||
"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
|
||||
"Accept": "application/json",
|
||||
},
|
||||
headers=request_headers,
|
||||
params=request_params or None,
|
||||
json=request_json,
|
||||
data=request_data,
|
||||
)
|
||||
response.raise_for_status()
|
||||
return self.parse_response(response.json())
|
||||
return self.parse_response(
|
||||
response.json(),
|
||||
response_path=request_config["response_path"],
|
||||
)
|
||||
|
||||
def parse_response(self, response: Any) -> list[dict[str, Any]]:
|
||||
if isinstance(response, dict):
|
||||
candidates = response.get("sources") or response.get("streams") or response.get("data") or []
|
||||
elif isinstance(response, list):
|
||||
candidates = response
|
||||
async def _load_datasource_config(self) -> DataSourceConfig | None:
|
||||
if not self._db_session:
|
||||
return None
|
||||
|
||||
result = await self._db_session.execute(
|
||||
select(DataSourceConfig)
|
||||
.where(DataSourceConfig.name == self.name)
|
||||
.where(DataSourceConfig.is_active.is_(True))
|
||||
.limit(1)
|
||||
)
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
def _get_effective_config(self, datasource_config: DataSourceConfig | None) -> dict[str, Any]:
|
||||
payload = dict(datasource_config.config or {}) if datasource_config else {}
|
||||
if payload:
|
||||
return payload
|
||||
|
||||
yaml_config = get_data_sources_config()
|
||||
return {
|
||||
"adapter": self.DEFAULT_ADAPTER,
|
||||
"streams_url": yaml_config.get_yaml_value("news_live_streams.streams_url")
|
||||
or self.DEFAULT_IPTV_ORG_STREAMS_URL,
|
||||
"logos_url": yaml_config.get_yaml_value("news_live_streams.logos_url")
|
||||
or self.DEFAULT_IPTV_ORG_LOGOS_URL,
|
||||
"news_categories": list(self.DEFAULT_IPTV_ORG_NEWS_CATEGORIES),
|
||||
"exclude_categories": list(self.DEFAULT_IPTV_ORG_EXCLUDE_CATEGORIES),
|
||||
"max_sources": self.DEFAULT_IPTV_ORG_MAX_SOURCES,
|
||||
}
|
||||
|
||||
def _get_request_config(self, datasource_config: DataSourceConfig | None) -> dict[str, Any]:
|
||||
payload = self._get_effective_config(datasource_config)
|
||||
raw_method = payload.get("method") or payload.get("request_method") or "GET"
|
||||
method = str(raw_method).strip().upper() or "GET"
|
||||
if method not in {"GET", "POST"}:
|
||||
method = "GET"
|
||||
|
||||
response_path = payload.get("response_path") or payload.get("payload_path") or payload.get("items_path")
|
||||
if isinstance(response_path, str):
|
||||
response_path = response_path.strip()
|
||||
else:
|
||||
candidates = []
|
||||
response_path = None
|
||||
|
||||
return {
|
||||
"method": method,
|
||||
"response_path": response_path or None,
|
||||
}
|
||||
|
||||
def _get_timeout(self, datasource_config: DataSourceConfig | None) -> float:
|
||||
payload = self._get_effective_config(datasource_config)
|
||||
try:
|
||||
return float(payload.get("timeout", self.DEFAULT_TIMEOUT))
|
||||
except (TypeError, ValueError):
|
||||
return self.DEFAULT_TIMEOUT
|
||||
|
||||
def _build_request_headers(self, datasource_config: DataSourceConfig | None) -> dict[str, str]:
|
||||
headers = dict(self.DEFAULT_HEADERS)
|
||||
if datasource_config:
|
||||
headers.update(self._normalize_headers(datasource_config.headers))
|
||||
headers.update(self._build_auth_headers(datasource_config))
|
||||
return headers
|
||||
|
||||
def _build_request_params(self, datasource_config: DataSourceConfig | None) -> dict[str, Any]:
|
||||
params: dict[str, Any] = {}
|
||||
if not datasource_config:
|
||||
return params
|
||||
|
||||
payload = datasource_config.config or {}
|
||||
candidate = payload.get("params") or payload.get("query_params")
|
||||
if isinstance(candidate, dict):
|
||||
params.update(candidate)
|
||||
|
||||
if datasource_config.auth_type == "api_key":
|
||||
auth_config = datasource_config.auth_config or {}
|
||||
if str(auth_config.get("in") or auth_config.get("location") or "header").lower() == "query":
|
||||
api_key = auth_config.get("api_key")
|
||||
key_name = auth_config.get("key_name") or auth_config.get("param_name") or "api_key"
|
||||
if api_key and key_name:
|
||||
params[str(key_name)] = api_key
|
||||
|
||||
return params
|
||||
|
||||
def _build_request_json_body(self, datasource_config: DataSourceConfig | None) -> Any:
|
||||
if not datasource_config:
|
||||
return None
|
||||
|
||||
payload = datasource_config.config or {}
|
||||
body = payload.get("json_body")
|
||||
if body is None and str(payload.get("body_type") or "").lower() in {"json", ""}:
|
||||
candidate = payload.get("body")
|
||||
if isinstance(candidate, (dict, list)):
|
||||
body = candidate
|
||||
return body
|
||||
|
||||
def _build_request_form_body(self, datasource_config: DataSourceConfig | None) -> Any:
|
||||
if not datasource_config:
|
||||
return None
|
||||
|
||||
payload = datasource_config.config or {}
|
||||
form_body = payload.get("form_body")
|
||||
if form_body is not None:
|
||||
return form_body
|
||||
|
||||
if str(payload.get("body_type") or "").lower() == "form":
|
||||
candidate = payload.get("body")
|
||||
if isinstance(candidate, dict):
|
||||
return candidate
|
||||
return None
|
||||
|
||||
def _normalize_headers(self, headers: Any) -> dict[str, str]:
|
||||
if not isinstance(headers, dict):
|
||||
return {}
|
||||
normalized: dict[str, str] = {}
|
||||
for key, value in headers.items():
|
||||
header_name = str(key).strip()
|
||||
if not header_name or value is None:
|
||||
continue
|
||||
normalized[header_name] = str(value)
|
||||
return normalized
|
||||
|
||||
def _build_auth_headers(self, datasource_config: DataSourceConfig | None) -> dict[str, str]:
|
||||
if not datasource_config:
|
||||
return {}
|
||||
|
||||
auth_type = str(datasource_config.auth_type or "none").lower()
|
||||
auth_config = datasource_config.auth_config or {}
|
||||
if auth_type == "bearer" and auth_config.get("token"):
|
||||
return {"Authorization": f"Bearer {auth_config['token']}"}
|
||||
|
||||
if auth_type == "api_key" and auth_config.get("api_key"):
|
||||
location = str(auth_config.get("in") or auth_config.get("location") or "header").lower()
|
||||
if location == "query":
|
||||
return {}
|
||||
key_name = auth_config.get("key_name") or "X-API-Key"
|
||||
return {str(key_name): str(auth_config["api_key"])}
|
||||
|
||||
if auth_type == "basic":
|
||||
username = str(auth_config.get("username") or "")
|
||||
password = str(auth_config.get("password") or "")
|
||||
encoded = base64.b64encode(f"{username}:{password}".encode()).decode()
|
||||
return {"Authorization": f"Basic {encoded}"}
|
||||
|
||||
return {}
|
||||
|
||||
def _extract_candidates(self, response: Any, response_path: str | None) -> list[Any]:
|
||||
if response_path:
|
||||
extracted = self._extract_from_path(response, response_path)
|
||||
if isinstance(extracted, list):
|
||||
return extracted
|
||||
if isinstance(extracted, dict):
|
||||
for key in self.RESPONSE_CANDIDATE_KEYS:
|
||||
nested = extracted.get(key)
|
||||
if isinstance(nested, list):
|
||||
return nested
|
||||
return [extracted]
|
||||
|
||||
if isinstance(response, dict):
|
||||
for key in self.RESPONSE_CANDIDATE_KEYS:
|
||||
nested = response.get(key)
|
||||
if isinstance(nested, list):
|
||||
return nested
|
||||
return []
|
||||
|
||||
if isinstance(response, list):
|
||||
return response
|
||||
return []
|
||||
|
||||
def _extract_from_path(self, payload: Any, path: str) -> Any:
|
||||
current = payload
|
||||
for segment in (part.strip() for part in path.split(".") if part.strip()):
|
||||
if isinstance(current, dict):
|
||||
current = current.get(segment)
|
||||
continue
|
||||
if isinstance(current, list):
|
||||
try:
|
||||
current = current[int(segment)]
|
||||
except (TypeError, ValueError, IndexError):
|
||||
return None
|
||||
continue
|
||||
return None
|
||||
return current
|
||||
|
||||
def _infer_source_type(self, item: dict[str, Any]) -> str:
|
||||
explicit = str(item.get("source_type") or item.get("type") or "").strip().lower()
|
||||
if explicit in {"iframe", "hls", "video", "external", "youtube"}:
|
||||
return explicit
|
||||
|
||||
youtube_video_id = self._clean_text(
|
||||
item.get("youtube_video_id")
|
||||
or item.get("video_id")
|
||||
or item.get("youtubeVideoId")
|
||||
)
|
||||
youtube_channel = self._clean_text(item.get("youtube_channel") or item.get("channel_handle"))
|
||||
embed_url = self._clean_url(item.get("embed_url") or item.get("embed") or item.get("page_url"))
|
||||
stream_url = self._clean_url(item.get("stream_url") or item.get("stream") or item.get("playback_url") or item.get("hls_url"))
|
||||
homepage_url = self._clean_url(item.get("homepage_url") or item.get("source_url") or item.get("website"))
|
||||
|
||||
if youtube_video_id or youtube_channel:
|
||||
return "youtube"
|
||||
if stream_url.endswith(".m3u8"):
|
||||
return "hls"
|
||||
if stream_url:
|
||||
return "video"
|
||||
if embed_url:
|
||||
parsed = urlparse(embed_url)
|
||||
if "youtube.com" in (parsed.netloc or "") or "youtu.be" in (parsed.netloc or ""):
|
||||
return "youtube"
|
||||
return "iframe"
|
||||
if homepage_url:
|
||||
return "external"
|
||||
return "iframe"
|
||||
|
||||
def _parse_enabled(self, item: dict[str, Any]) -> bool:
|
||||
if "is_enabled" in item:
|
||||
return self._to_bool(item.get("is_enabled"), default=True)
|
||||
if "enabled" in item:
|
||||
return self._to_bool(item.get("enabled"), default=True)
|
||||
if "active" in item:
|
||||
return self._to_bool(item.get("active"), default=True)
|
||||
if "status" in item:
|
||||
status = str(item.get("status") or "").strip().lower()
|
||||
if status in {"disabled", "inactive", "offline"}:
|
||||
return False
|
||||
if status in {"enabled", "active", "online", "live"}:
|
||||
return True
|
||||
return True
|
||||
|
||||
def _to_bool(self, value: Any, *, default: bool) -> bool:
|
||||
if isinstance(value, bool):
|
||||
return value
|
||||
if value in (None, ""):
|
||||
return default
|
||||
if isinstance(value, str):
|
||||
lowered = value.strip().lower()
|
||||
if lowered in {"1", "true", "yes", "on", "enabled", "active", "online", "live"}:
|
||||
return True
|
||||
if lowered in {"0", "false", "no", "off", "disabled", "inactive", "offline"}:
|
||||
return False
|
||||
return bool(value)
|
||||
|
||||
def _clean_text(self, value: Any) -> str:
|
||||
if value is None:
|
||||
return ""
|
||||
return str(value).strip()
|
||||
|
||||
def _clean_url(self, value: Any) -> str:
|
||||
text = self._clean_text(value)
|
||||
if not text:
|
||||
return ""
|
||||
parsed = urlparse(text)
|
||||
if parsed.scheme and parsed.scheme not in {"http", "https"}:
|
||||
return ""
|
||||
if parsed.scheme and not parsed.netloc:
|
||||
return ""
|
||||
return text
|
||||
|
||||
async def _fetch_iptv_org(self, channels_url: str, collector_config: dict[str, Any]) -> list[dict[str, Any]]:
|
||||
streams_url = self._clean_url(collector_config.get("streams_url")) or self.DEFAULT_IPTV_ORG_STREAMS_URL
|
||||
logos_url = self._clean_url(collector_config.get("logos_url")) or self.DEFAULT_IPTV_ORG_LOGOS_URL
|
||||
news_categories = {
|
||||
self._clean_text(value).lower()
|
||||
for value in (collector_config.get("news_categories") or self.DEFAULT_IPTV_ORG_NEWS_CATEGORIES)
|
||||
if self._clean_text(value)
|
||||
}
|
||||
exclude_categories = {
|
||||
self._clean_text(value).lower()
|
||||
for value in (collector_config.get("exclude_categories") or self.DEFAULT_IPTV_ORG_EXCLUDE_CATEGORIES)
|
||||
if self._clean_text(value)
|
||||
}
|
||||
try:
|
||||
max_sources = int(collector_config.get("max_sources", self.DEFAULT_IPTV_ORG_MAX_SOURCES))
|
||||
except (TypeError, ValueError):
|
||||
max_sources = self.DEFAULT_IPTV_ORG_MAX_SOURCES
|
||||
|
||||
timeout = self.DEFAULT_TIMEOUT
|
||||
try:
|
||||
timeout = float(collector_config.get("timeout", self.DEFAULT_TIMEOUT))
|
||||
except (TypeError, ValueError):
|
||||
timeout = self.DEFAULT_TIMEOUT
|
||||
|
||||
async with httpx.AsyncClient(timeout=timeout, follow_redirects=True) as client:
|
||||
channels_payload, streams_payload, logos_payload = await self._gather_iptv_org_payloads(
|
||||
client,
|
||||
channels_url,
|
||||
streams_url,
|
||||
logos_url,
|
||||
)
|
||||
|
||||
channels = channels_payload if isinstance(channels_payload, list) else []
|
||||
streams = streams_payload if isinstance(streams_payload, list) else []
|
||||
logos = logos_payload if isinstance(logos_payload, list) else []
|
||||
|
||||
logo_by_channel = {
|
||||
self._clean_text(item.get("channel")): self._clean_url(item.get("url"))
|
||||
for item in logos
|
||||
if isinstance(item, dict) and self._clean_text(item.get("channel")) and self._clean_url(item.get("url"))
|
||||
}
|
||||
|
||||
streams_by_channel: dict[str, list[dict[str, Any]]] = {}
|
||||
for stream in streams:
|
||||
if not isinstance(stream, dict):
|
||||
continue
|
||||
channel_id = self._clean_text(stream.get("channel"))
|
||||
if not channel_id:
|
||||
continue
|
||||
streams_by_channel.setdefault(channel_id, []).append(stream)
|
||||
|
||||
normalized: list[dict[str, Any]] = []
|
||||
for channel in channels:
|
||||
if not isinstance(channel, dict):
|
||||
continue
|
||||
|
||||
categories = [
|
||||
self._clean_text(value).lower()
|
||||
for value in (channel.get("categories") or [])
|
||||
if self._clean_text(value)
|
||||
]
|
||||
if news_categories and not any(category in news_categories for category in categories):
|
||||
continue
|
||||
if exclude_categories and any(category in exclude_categories for category in categories):
|
||||
continue
|
||||
if channel.get("is_nsfw") is True:
|
||||
continue
|
||||
if channel.get("closed"):
|
||||
continue
|
||||
|
||||
channel_id = self._clean_text(channel.get("id"))
|
||||
if not channel_id:
|
||||
continue
|
||||
|
||||
stream = self._pick_iptv_org_stream(streams_by_channel.get(channel_id) or [])
|
||||
if not stream:
|
||||
continue
|
||||
|
||||
stream_url = self._clean_url(stream.get("url"))
|
||||
if not stream_url:
|
||||
continue
|
||||
|
||||
name = self._clean_text(channel.get("name")) or channel_id
|
||||
notes_parts = [
|
||||
f"Imported from IPTV-org catalog ({channel_id})",
|
||||
f"Categories: {', '.join(categories)}" if categories else "",
|
||||
f"Quality: {self._clean_text(stream.get('quality'))}" if self._clean_text(stream.get("quality")) else "",
|
||||
]
|
||||
metadata = {
|
||||
"provider": self._clean_text(channel.get("network")) or "IPTV-org",
|
||||
"region": self._clean_text(channel.get("country")) or "Global",
|
||||
"language": "und",
|
||||
"source_type": "hls" if stream_url.endswith(".m3u8") else "video",
|
||||
"embed_url": "",
|
||||
"stream_url": stream_url,
|
||||
"homepage_url": self._clean_url(channel.get("website")),
|
||||
"poster_url": logo_by_channel.get(channel_id, ""),
|
||||
"youtube_video_id": "",
|
||||
"youtube_channel": "",
|
||||
"sort_order": 400 + len(normalized),
|
||||
"notes": "; ".join(part for part in notes_parts if part),
|
||||
"is_enabled": True,
|
||||
"collector_adapter": "iptv_org",
|
||||
"channel_id": channel_id,
|
||||
"categories": categories,
|
||||
"quality": self._clean_text(stream.get("quality")),
|
||||
"stream_label": self._clean_text(stream.get("label") or stream.get("title")),
|
||||
"stream_referrer": self._clean_text(stream.get("referrer")),
|
||||
"stream_user_agent": self._clean_text(stream.get("user_agent")),
|
||||
}
|
||||
|
||||
normalized.append(
|
||||
{
|
||||
"source_id": channel_id,
|
||||
"name": name,
|
||||
"description": metadata["notes"],
|
||||
"metadata": metadata,
|
||||
"reference_date": datetime.now(UTC).isoformat(),
|
||||
}
|
||||
)
|
||||
if len(normalized) >= max_sources:
|
||||
break
|
||||
|
||||
return normalized
|
||||
|
||||
async def _gather_iptv_org_payloads(
|
||||
self,
|
||||
client: httpx.AsyncClient,
|
||||
channels_url: str,
|
||||
streams_url: str,
|
||||
logos_url: str,
|
||||
) -> tuple[Any, Any, Any]:
|
||||
headers = dict(self.DEFAULT_HEADERS)
|
||||
channels_payload, streams_payload, logos_payload = await asyncio.gather(
|
||||
client.get(channels_url, headers=headers),
|
||||
client.get(streams_url, headers=headers),
|
||||
client.get(logos_url, headers=headers),
|
||||
)
|
||||
channels_payload.raise_for_status()
|
||||
streams_payload.raise_for_status()
|
||||
logos_payload.raise_for_status()
|
||||
return channels_payload.json(), streams_payload.json(), logos_payload.json()
|
||||
|
||||
def _pick_iptv_org_stream(self, streams: list[dict[str, Any]]) -> dict[str, Any] | None:
|
||||
if not streams:
|
||||
return None
|
||||
|
||||
def score(stream: dict[str, Any]) -> tuple[int, int]:
|
||||
url = self._clean_url(stream.get("url"))
|
||||
quality = self._clean_text(stream.get("quality")).lower()
|
||||
quality_score = 0
|
||||
if quality.endswith("p"):
|
||||
try:
|
||||
quality_score = int(quality[:-1])
|
||||
except ValueError:
|
||||
quality_score = 0
|
||||
stream_score = 1000 if url.endswith(".m3u8") else 0
|
||||
return stream_score, quality_score
|
||||
|
||||
sorted_streams = sorted(streams, key=score, reverse=True)
|
||||
return sorted_streams[0]
|
||||
|
||||
def parse_response(self, response: Any, *, response_path: str | None = None) -> list[dict[str, Any]]:
|
||||
candidates = self._extract_candidates(response, response_path)
|
||||
|
||||
normalized: list[dict[str, Any]] = []
|
||||
for index, item in enumerate(candidates):
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
|
||||
stream_id = item.get("id") or item.get("source_id") or item.get("slug") or f"news-live-{index + 1}"
|
||||
name = str(item.get("name") or item.get("title") or f"News Live {index + 1}").strip()
|
||||
stream_id = (
|
||||
item.get("id")
|
||||
or item.get("source_id")
|
||||
or item.get("slug")
|
||||
or item.get("channel_id")
|
||||
or item.get("code")
|
||||
or f"news-live-{index + 1}"
|
||||
)
|
||||
name = self._clean_text(
|
||||
item.get("name")
|
||||
or item.get("title")
|
||||
or item.get("channel")
|
||||
or item.get("display_name")
|
||||
or f"News Live {index + 1}"
|
||||
)
|
||||
if not name:
|
||||
continue
|
||||
|
||||
source_type = self._infer_source_type(item)
|
||||
stream_url = self._clean_url(
|
||||
item.get("stream_url")
|
||||
or item.get("stream")
|
||||
or item.get("playback_url")
|
||||
or item.get("hls_url")
|
||||
or item.get("m3u8_url")
|
||||
)
|
||||
embed_url = self._clean_url(
|
||||
item.get("embed_url")
|
||||
or item.get("embed")
|
||||
or item.get("page_url")
|
||||
or (item.get("url") if source_type == "iframe" else "")
|
||||
)
|
||||
homepage_url = self._clean_url(
|
||||
item.get("homepage_url")
|
||||
or item.get("source_url")
|
||||
or item.get("website")
|
||||
or item.get("url")
|
||||
)
|
||||
metadata = {
|
||||
"provider": item.get("provider") or item.get("publisher") or "Collector",
|
||||
"region": item.get("region") or item.get("country") or "Global",
|
||||
"language": item.get("language") or "und",
|
||||
"source_type": item.get("source_type") or "iframe",
|
||||
"embed_url": item.get("embed_url") or item.get("url") or "",
|
||||
"stream_url": item.get("stream_url") or "",
|
||||
"homepage_url": item.get("homepage_url") or item.get("source_url") or "",
|
||||
"poster_url": item.get("poster_url") or "",
|
||||
"provider": self._clean_text(item.get("provider") or item.get("publisher") or item.get("network")) or "Collector",
|
||||
"region": self._clean_text(item.get("region") or item.get("country") or item.get("market")) or "Global",
|
||||
"language": self._clean_text(item.get("language") or item.get("lang") or item.get("locale")) or "und",
|
||||
"source_type": source_type,
|
||||
"embed_url": embed_url,
|
||||
"stream_url": stream_url,
|
||||
"homepage_url": homepage_url,
|
||||
"poster_url": self._clean_url(item.get("poster_url") or item.get("thumbnail_url") or item.get("logo_url")),
|
||||
"youtube_video_id": self._clean_text(
|
||||
item.get("youtube_video_id")
|
||||
or item.get("video_id")
|
||||
or item.get("youtubeVideoId")
|
||||
),
|
||||
"youtube_channel": self._clean_text(
|
||||
item.get("youtube_channel")
|
||||
or item.get("channel_handle")
|
||||
or item.get("youtubeChannel")
|
||||
),
|
||||
"sort_order": item.get("sort_order", 200 + index),
|
||||
"notes": item.get("notes") or item.get("description") or "",
|
||||
"is_enabled": item.get("is_enabled", True),
|
||||
"notes": self._clean_text(item.get("notes") or item.get("description") or item.get("summary")),
|
||||
"is_enabled": self._parse_enabled(item),
|
||||
}
|
||||
|
||||
normalized.append(
|
||||
@@ -72,7 +563,7 @@ class NewsLiveStreamsCollector(BaseCollector):
|
||||
"name": name,
|
||||
"description": metadata["notes"],
|
||||
"metadata": metadata,
|
||||
"reference_date": item.get("reference_date", datetime.now(UTC).isoformat()),
|
||||
"reference_date": item.get("reference_date") or datetime.now(UTC).isoformat(),
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
@@ -30,6 +30,14 @@ class RegionProfile:
|
||||
accent: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RegionAnchor:
|
||||
region: str
|
||||
label: str
|
||||
latitude: float
|
||||
longitude: float
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class NewsFeedSource:
|
||||
id: str
|
||||
@@ -95,6 +103,39 @@ REGION_PROFILES: dict[str, RegionProfile] = {
|
||||
),
|
||||
}
|
||||
|
||||
REGION_ANCHORS: dict[str, RegionAnchor] = {
|
||||
"americas": RegionAnchor(
|
||||
region="americas",
|
||||
label="美洲",
|
||||
latitude=37.0902,
|
||||
longitude=-95.7129,
|
||||
),
|
||||
"europe": RegionAnchor(
|
||||
region="europe",
|
||||
label="欧洲",
|
||||
latitude=50.1109,
|
||||
longitude=8.6821,
|
||||
),
|
||||
"middle-east-africa": RegionAnchor(
|
||||
region="middle-east-africa",
|
||||
label="中东与非洲",
|
||||
latitude=25.2048,
|
||||
longitude=55.2708,
|
||||
),
|
||||
"asia-pacific": RegionAnchor(
|
||||
region="asia-pacific",
|
||||
label="亚太",
|
||||
latitude=1.3521,
|
||||
longitude=103.8198,
|
||||
),
|
||||
"global": RegionAnchor(
|
||||
region="global",
|
||||
label="全球",
|
||||
latitude=20.0,
|
||||
longitude=0.0,
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def _google_news_feed(query: str, *, hl: str, gl: str, ceid: str) -> str:
|
||||
return (
|
||||
@@ -213,6 +254,10 @@ def get_region_profile(region: str) -> RegionProfile:
|
||||
return REGION_PROFILES.get(region, REGION_PROFILES["global"])
|
||||
|
||||
|
||||
def get_region_anchor(region: str) -> RegionAnchor:
|
||||
return REGION_ANCHORS.get(region, REGION_ANCHORS["global"])
|
||||
|
||||
|
||||
def get_sources_for_region(region: str) -> list[NewsFeedSource]:
|
||||
return sorted(
|
||||
[source for source in NEWS_FEED_SOURCES if source.region in {"global", region}],
|
||||
@@ -342,6 +387,7 @@ def _serialize_sources(sources: list[NewsFeedSource]) -> list[dict[str, Any]]:
|
||||
|
||||
def _serialize_item(item: ParsedNewsItem, *, active_region: str) -> dict[str, Any]:
|
||||
published_at = item.published_at
|
||||
anchor = get_region_anchor(item.feed_region)
|
||||
return {
|
||||
"id": item.id,
|
||||
"title": item.title,
|
||||
@@ -352,6 +398,10 @@ def _serialize_item(item: ParsedNewsItem, *, active_region: str) -> dict[str, An
|
||||
"region": item.feed_region,
|
||||
"homepage_url": item.homepage_url,
|
||||
"published_at": published_at.isoformat().replace("+00:00", "Z") if published_at else None,
|
||||
"latitude": anchor.latitude,
|
||||
"longitude": anchor.longitude,
|
||||
"location_label": anchor.label,
|
||||
"location_inferred": True,
|
||||
"is_focus_match": item.feed_region == active_region,
|
||||
}
|
||||
|
||||
|
||||
86
backend/app/services/persistent_logs.py
Normal file
86
backend/app/services/persistent_logs.py
Normal file
@@ -0,0 +1,86 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from app.core.logging import get_logger, sanitize_log_value
|
||||
from app.core.request_context import get_request_id
|
||||
from app.db.session import async_session_factory
|
||||
from app.models.system_log import AuditLog, SystemLog
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
async def record_system_log(
|
||||
*,
|
||||
source: str,
|
||||
level: str,
|
||||
message: str,
|
||||
service: str | None = None,
|
||||
module: str | None = None,
|
||||
event: str | None = None,
|
||||
request_id: str | None = None,
|
||||
trace_id: str | None = None,
|
||||
user_id: int | None = None,
|
||||
category: str | None = None,
|
||||
context: dict[str, Any] | None = None,
|
||||
) -> None:
|
||||
try:
|
||||
async with async_session_factory() as session:
|
||||
session.add(
|
||||
SystemLog(
|
||||
source=source,
|
||||
service=service,
|
||||
module=module,
|
||||
event=event,
|
||||
level=level.lower(),
|
||||
message=str(sanitize_log_value(message)),
|
||||
request_id=request_id or get_request_id(),
|
||||
trace_id=trace_id,
|
||||
user_id=user_id,
|
||||
category=category,
|
||||
context=sanitize_log_value(context or {}),
|
||||
)
|
||||
)
|
||||
await session.commit()
|
||||
except Exception:
|
||||
logger.exception_event(
|
||||
"Failed to persist system log",
|
||||
event="system_log.persist.failed",
|
||||
context={"event_name": event, "source": source},
|
||||
)
|
||||
|
||||
|
||||
async def record_audit_log(
|
||||
*,
|
||||
action: str,
|
||||
actor_id: int | None = None,
|
||||
actor_name: str | None = None,
|
||||
target_type: str | None = None,
|
||||
target_id: str | None = None,
|
||||
result: str | None = None,
|
||||
request_id: str | None = None,
|
||||
ip: str | None = None,
|
||||
details: dict[str, Any] | None = None,
|
||||
) -> None:
|
||||
try:
|
||||
async with async_session_factory() as session:
|
||||
session.add(
|
||||
AuditLog(
|
||||
actor_id=actor_id,
|
||||
actor_name=actor_name,
|
||||
action=action,
|
||||
target_type=target_type,
|
||||
target_id=target_id,
|
||||
result=result,
|
||||
request_id=request_id or get_request_id(),
|
||||
ip=ip,
|
||||
details=sanitize_log_value(details or {}),
|
||||
)
|
||||
)
|
||||
await session.commit()
|
||||
except Exception:
|
||||
logger.exception_event(
|
||||
"Failed to persist audit log",
|
||||
event="audit_log.persist.failed",
|
||||
context={"action": action},
|
||||
)
|
||||
@@ -1,7 +1,6 @@
|
||||
"""Task Scheduler for running collection jobs."""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
@@ -9,13 +8,14 @@ from apscheduler.schedulers.asyncio import AsyncIOScheduler
|
||||
from apscheduler.triggers.interval import IntervalTrigger
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.core.logging import get_logger
|
||||
from app.db.session import async_session_factory
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.models.datasource import DataSource
|
||||
from app.models.task import CollectionTask
|
||||
from app.services.collectors.registry import collector_registry
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
logger = get_logger(__name__)
|
||||
|
||||
scheduler = AsyncIOScheduler()
|
||||
RUNNING_TASK_GUARD_TIMEOUT_MINUTES = 90
|
||||
@@ -54,7 +54,11 @@ async def _update_next_run_at(datasource: DataSource, session) -> None:
|
||||
async def _apply_datasource_schedule(datasource: DataSource, session) -> None:
|
||||
collector = collector_registry.get(datasource.source)
|
||||
if not collector:
|
||||
logger.warning("Collector not found for datasource %s", datasource.source)
|
||||
logger.warning_event(
|
||||
"Collector not found for datasource",
|
||||
event="collector.schedule.collector_missing",
|
||||
context={"collector_name": datasource.source},
|
||||
)
|
||||
return
|
||||
|
||||
collector_registry.set_active(datasource.source, datasource.is_active)
|
||||
@@ -72,13 +76,17 @@ async def _apply_datasource_schedule(datasource: DataSource, session) -> None:
|
||||
replace_existing=True,
|
||||
kwargs={"collector_name": datasource.source},
|
||||
)
|
||||
logger.info(
|
||||
"Scheduled collector: %s (every %sm)",
|
||||
datasource.source,
|
||||
datasource.frequency_minutes,
|
||||
logger.info_event(
|
||||
"Scheduled collector",
|
||||
event="collector.schedule.updated",
|
||||
context={"collector_name": datasource.source, "frequency_minutes": datasource.frequency_minutes},
|
||||
)
|
||||
else:
|
||||
logger.info("Collector disabled: %s", datasource.source)
|
||||
logger.info_event(
|
||||
"Collector disabled",
|
||||
event="collector.schedule.disabled",
|
||||
context={"collector_name": datasource.source},
|
||||
)
|
||||
|
||||
await _update_next_run_at(datasource, session)
|
||||
|
||||
@@ -87,18 +95,30 @@ async def run_collector_task(collector_name: str):
|
||||
"""Run a single collector task."""
|
||||
collector = collector_registry.get(collector_name)
|
||||
if not collector:
|
||||
logger.error("Collector not found: %s", collector_name)
|
||||
logger.error_event(
|
||||
"Collector not found",
|
||||
event="collector.run.collector_missing",
|
||||
context={"collector_name": collector_name},
|
||||
)
|
||||
return
|
||||
|
||||
async with async_session_factory() as db:
|
||||
result = await db.execute(select(DataSource).where(DataSource.source == collector_name))
|
||||
datasource = result.scalar_one_or_none()
|
||||
if not datasource:
|
||||
logger.error("Datasource not found for collector: %s", collector_name)
|
||||
logger.error_event(
|
||||
"Datasource not found for collector",
|
||||
event="collector.run.datasource_missing",
|
||||
context={"collector_name": collector_name},
|
||||
)
|
||||
return
|
||||
|
||||
if not datasource.is_active:
|
||||
logger.info("Skipping disabled collector: %s", collector_name)
|
||||
logger.info_event(
|
||||
"Skipping disabled collector",
|
||||
event="collector.run.skipped_disabled",
|
||||
context={"collector_name": collector_name},
|
||||
)
|
||||
return
|
||||
|
||||
running_result = await db.execute(
|
||||
@@ -122,10 +142,10 @@ async def run_collector_task(collector_name: str):
|
||||
and (now - started_at) > timedelta(minutes=RUNNING_TASK_GUARD_TIMEOUT_MINUTES)
|
||||
)
|
||||
if not is_stale:
|
||||
logger.warning(
|
||||
"Skipping collector %s trigger because task %s is already running",
|
||||
collector_name,
|
||||
existing_running.id,
|
||||
logger.warning_event(
|
||||
"Skipping collector trigger because task is already running",
|
||||
event="collector.run.skipped_already_running",
|
||||
context={"collector_name": collector_name, "task_id": existing_running.id},
|
||||
)
|
||||
return
|
||||
|
||||
@@ -143,31 +163,47 @@ async def run_collector_task(collector_name: str):
|
||||
else stale_reason
|
||||
)
|
||||
await db.commit()
|
||||
logger.warning(
|
||||
"Marked stale running task %s as failed before rerun of %s",
|
||||
existing_running.id,
|
||||
collector_name,
|
||||
logger.warning_event(
|
||||
"Marked stale running task as failed before rerun",
|
||||
event="collector.run.stale_task_failed",
|
||||
context={"collector_name": collector_name, "task_id": existing_running.id},
|
||||
)
|
||||
|
||||
try:
|
||||
collector._datasource_id = datasource.id
|
||||
logger.info("Running collector: %s (datasource_id=%s)", collector_name, datasource.id)
|
||||
logger.info_event(
|
||||
"Running collector",
|
||||
event="collector.run.started",
|
||||
context={"collector_name": collector_name, "datasource_id": datasource.id},
|
||||
)
|
||||
task_result = await collector.run(db)
|
||||
datasource.last_run_at = datetime.now(UTC)
|
||||
datasource.last_status = task_result.get("status")
|
||||
await _update_next_run_at(datasource, db)
|
||||
logger.info("Collector %s completed: %s", collector_name, task_result)
|
||||
logger.info_event(
|
||||
"Collector completed",
|
||||
event="collector.run.completed",
|
||||
context={"collector_name": collector_name, "datasource_id": datasource.id, "result": task_result},
|
||||
)
|
||||
except asyncio.CancelledError:
|
||||
datasource.last_run_at = datetime.now(UTC)
|
||||
datasource.last_status = "cancelled"
|
||||
await db.commit()
|
||||
logger.warning("Collector %s cancelled by operator", collector_name)
|
||||
logger.warning_event(
|
||||
"Collector cancelled by operator",
|
||||
event="collector.run.cancelled",
|
||||
context={"collector_name": collector_name, "datasource_id": datasource.id},
|
||||
)
|
||||
raise
|
||||
except Exception as exc:
|
||||
datasource.last_run_at = datetime.now(UTC)
|
||||
datasource.last_status = "failed"
|
||||
await db.commit()
|
||||
logger.exception("Collector %s failed: %s", collector_name, exc)
|
||||
logger.exception_event(
|
||||
"Collector failed",
|
||||
event="collector.run.failed",
|
||||
context={"collector_name": collector_name, "datasource_id": datasource.id, "error": str(exc)},
|
||||
)
|
||||
|
||||
|
||||
async def cleanup_stale_running_tasks(max_age_hours: int = 2) -> int:
|
||||
@@ -194,7 +230,11 @@ async def cleanup_stale_running_tasks(max_age_hours: int = 2) -> int:
|
||||
|
||||
if stale_tasks:
|
||||
await db.commit()
|
||||
logger.warning("Cleaned up %s stale running collection task(s)", len(stale_tasks))
|
||||
logger.warning_event(
|
||||
"Cleaned up stale running collection tasks",
|
||||
event="collector.cleanup.stale_tasks_cleaned",
|
||||
context={"count": len(stale_tasks)},
|
||||
)
|
||||
|
||||
return len(stale_tasks)
|
||||
|
||||
@@ -203,14 +243,14 @@ def start_scheduler() -> None:
|
||||
"""Start the scheduler."""
|
||||
if not scheduler.running:
|
||||
scheduler.start()
|
||||
logger.info("Scheduler started")
|
||||
logger.info_event("Scheduler started", event="scheduler.started")
|
||||
|
||||
|
||||
def stop_scheduler() -> None:
|
||||
"""Stop the scheduler."""
|
||||
if scheduler.running:
|
||||
scheduler.shutdown(wait=False)
|
||||
logger.info("Scheduler stopped")
|
||||
logger.info_event("Scheduler stopped", event="scheduler.stopped")
|
||||
|
||||
|
||||
async def sync_scheduler_with_datasources() -> None:
|
||||
@@ -271,12 +311,20 @@ def run_collector_now(collector_name: str) -> bool:
|
||||
"""Run a collector immediately (not scheduled)."""
|
||||
collector = collector_registry.get(collector_name)
|
||||
if not collector:
|
||||
logger.error("Collector not found: %s", collector_name)
|
||||
logger.error_event(
|
||||
"Collector not found",
|
||||
event="collector.trigger.collector_missing",
|
||||
context={"collector_name": collector_name},
|
||||
)
|
||||
return False
|
||||
|
||||
existing_task = get_running_collector_task(collector_name)
|
||||
if existing_task is not None and not existing_task.done():
|
||||
logger.warning("Collector %s is already running in-memory; skipping duplicate trigger", collector_name)
|
||||
logger.warning_event(
|
||||
"Collector is already running in-memory; skipping duplicate trigger",
|
||||
event="collector.trigger.skipped_already_running",
|
||||
context={"collector_name": collector_name},
|
||||
)
|
||||
return False
|
||||
|
||||
try:
|
||||
@@ -289,10 +337,18 @@ def run_collector_now(collector_name: str) -> bool:
|
||||
RUNNING_COLLECTOR_TASKS.pop(collector_name, None)
|
||||
|
||||
task.add_done_callback(_cleanup_task)
|
||||
logger.info("Triggered collector: %s", collector_name)
|
||||
logger.info_event(
|
||||
"Triggered collector",
|
||||
event="collector.trigger.started",
|
||||
context={"collector_name": collector_name},
|
||||
)
|
||||
return True
|
||||
except Exception as exc:
|
||||
logger.error("Failed to trigger collector %s: %s", collector_name, exc)
|
||||
logger.error_event(
|
||||
"Failed to trigger collector",
|
||||
event="collector.trigger.failed",
|
||||
context={"collector_name": collector_name, "error": str(exc)},
|
||||
)
|
||||
return False
|
||||
|
||||
|
||||
|
||||
532
backend/app/services/system_logs.py
Normal file
532
backend/app/services/system_logs.py
Normal file
@@ -0,0 +1,532 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
|
||||
from collections import Counter, deque
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from app.core.security import redis_client
|
||||
|
||||
DEFAULT_LOG_LINE_LIMIT = 200
|
||||
MAX_LOG_LINE_LIMIT = 1000
|
||||
BUFFER_LOG_LIMIT = 1000
|
||||
BUFFER_LOG_TTL_SECONDS = 7 * 24 * 60 * 60
|
||||
LOG_BUFFER_KEY_PREFIX = "planet:system_logs"
|
||||
|
||||
LOG_LEVEL_ERROR = "error"
|
||||
LOG_LEVEL_WARNING = "warning"
|
||||
LOG_LEVEL_INFO = "info"
|
||||
LOG_LEVEL_DEBUG = "debug"
|
||||
LOG_LEVEL_ALL = "all"
|
||||
|
||||
SUPPORTED_LOG_LEVELS = {
|
||||
LOG_LEVEL_ALL,
|
||||
LOG_LEVEL_ERROR,
|
||||
LOG_LEVEL_WARNING,
|
||||
LOG_LEVEL_INFO,
|
||||
LOG_LEVEL_DEBUG,
|
||||
}
|
||||
|
||||
LOG_LEVEL_ALIASES = {
|
||||
"warn": LOG_LEVEL_WARNING,
|
||||
"warning": LOG_LEVEL_WARNING,
|
||||
"err": LOG_LEVEL_ERROR,
|
||||
"error": LOG_LEVEL_ERROR,
|
||||
"info": LOG_LEVEL_INFO,
|
||||
"information": LOG_LEVEL_INFO,
|
||||
"debug": LOG_LEVEL_DEBUG,
|
||||
"trace": LOG_LEVEL_DEBUG,
|
||||
"critical": LOG_LEVEL_ERROR,
|
||||
"fatal": LOG_LEVEL_ERROR,
|
||||
}
|
||||
|
||||
TIMESTAMP_FORMATS = (
|
||||
"%Y-%m-%d %H:%M:%S.%f",
|
||||
"%Y-%m-%d %H:%M:%S",
|
||||
"%Y-%m-%dT%H:%M:%S.%f",
|
||||
"%Y-%m-%dT%H:%M:%S",
|
||||
)
|
||||
|
||||
LEVEL_PATTERNS = (
|
||||
("CRITICAL", LOG_LEVEL_ERROR),
|
||||
("FATAL", LOG_LEVEL_ERROR),
|
||||
("ERROR", LOG_LEVEL_ERROR),
|
||||
("WARNING", LOG_LEVEL_WARNING),
|
||||
("WARN", LOG_LEVEL_WARNING),
|
||||
("INFO", LOG_LEVEL_INFO),
|
||||
("DEBUG", LOG_LEVEL_DEBUG),
|
||||
("TRACE", LOG_LEVEL_DEBUG),
|
||||
)
|
||||
|
||||
LEADING_LEVEL_PATTERN = re.compile(
|
||||
r"^\s*(?:\[[^\]]+\]\s*)?(CRITICAL|FATAL|ERROR|WARNING|WARN|INFO|DEBUG|TRACE)\b[:\s-]*",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
EMBEDDED_LEVEL_PATTERN = re.compile(
|
||||
r"\b(CRITICAL|FATAL|ERROR|WARNING|WARN|INFO|DEBUG|TRACE)\b",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
CONTROL_CHAR_PATTERN = re.compile(r"[\x00-\x08\x0b\x0c\x0e-\x1f\x7f]")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class LogSource:
|
||||
source_id: str
|
||||
name: str
|
||||
kind: str
|
||||
location: str
|
||||
description: str
|
||||
category: str
|
||||
status: str = "ok"
|
||||
buffer_key: str | None = None
|
||||
container_name: str | None = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class StructuredLogEntry:
|
||||
timestamp: datetime | None
|
||||
level: str | None
|
||||
display_line: str
|
||||
raw_line: str
|
||||
search_text: str
|
||||
|
||||
|
||||
@dataclass
|
||||
class DailyLogMarker:
|
||||
date_token: str
|
||||
total: int
|
||||
dominant_level: str
|
||||
|
||||
|
||||
LOG_SOURCES: dict[str, LogSource] = {
|
||||
"backend": LogSource(
|
||||
source_id="backend",
|
||||
name="后端服务",
|
||||
kind="file",
|
||||
location="/tmp/planet_backend.log",
|
||||
description="FastAPI 后端、调度器和采集任务共享日志。",
|
||||
category="service",
|
||||
),
|
||||
"frontend": LogSource(
|
||||
source_id="frontend",
|
||||
name="前端开发服务",
|
||||
kind="file",
|
||||
location="/tmp/planet_frontend.log",
|
||||
description="控制台与 Earth 前端开发服务输出。",
|
||||
category="service",
|
||||
),
|
||||
"ai-provider": LogSource(
|
||||
source_id="ai-provider",
|
||||
name="AI Provider",
|
||||
kind="docker",
|
||||
location="docker://planet_aiprovider",
|
||||
description="AI Provider 容器实时输出日志。",
|
||||
category="service",
|
||||
container_name="planet_aiprovider",
|
||||
),
|
||||
"earth-client": LogSource(
|
||||
source_id="earth-client",
|
||||
name="Earth 浏览器端",
|
||||
kind="buffer",
|
||||
location="redis://planet:system_logs:earth-client",
|
||||
description="Earth 浏览器端上报的运行时错误与关键业务日志。",
|
||||
category="client",
|
||||
buffer_key=f"{LOG_BUFFER_KEY_PREFIX}:earth-client",
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def normalize_log_level(level: str | None) -> str:
|
||||
if level is None:
|
||||
return LOG_LEVEL_ALL
|
||||
normalized = str(level).strip().lower()
|
||||
if normalized in {"", LOG_LEVEL_ALL}:
|
||||
return LOG_LEVEL_ALL
|
||||
return LOG_LEVEL_ALIASES.get(normalized, LOG_LEVEL_ALL)
|
||||
|
||||
|
||||
def normalize_log_levels(level: str | None = None, levels: str | None = None) -> tuple[str, ...]:
|
||||
normalized_levels: list[str] = []
|
||||
if levels:
|
||||
for item in str(levels).split(","):
|
||||
normalized = normalize_log_level(item)
|
||||
if normalized != LOG_LEVEL_ALL and normalized not in normalized_levels:
|
||||
normalized_levels.append(normalized)
|
||||
normalized_level = normalize_log_level(level)
|
||||
if normalized_level != LOG_LEVEL_ALL and normalized_level not in normalized_levels:
|
||||
normalized_levels.append(normalized_level)
|
||||
return tuple(normalized_levels)
|
||||
|
||||
|
||||
def get_source_status(source: LogSource) -> str:
|
||||
if source.kind == "file":
|
||||
path = Path(source.location)
|
||||
if not path.exists():
|
||||
return "missing"
|
||||
return "ok" if path.stat().st_size > 0 else "empty"
|
||||
if source.kind == "docker":
|
||||
return "ok" if shutil.which("docker") else "docker_unavailable"
|
||||
if source.kind == "buffer":
|
||||
if not source.buffer_key:
|
||||
return "source_unavailable"
|
||||
try:
|
||||
return "ok" if redis_client.llen(source.buffer_key) > 0 else "empty"
|
||||
except Exception:
|
||||
return "source_unavailable"
|
||||
return "source_unavailable"
|
||||
|
||||
|
||||
def list_log_sources() -> list[dict[str, str]]:
|
||||
items: list[dict[str, str]] = []
|
||||
for source in LOG_SOURCES.values():
|
||||
items.append(
|
||||
{
|
||||
"source_id": source.source_id,
|
||||
"name": source.name,
|
||||
"kind": source.kind,
|
||||
"location": source.location,
|
||||
"description": source.description,
|
||||
"category": source.category,
|
||||
"status": get_source_status(source),
|
||||
}
|
||||
)
|
||||
return items
|
||||
|
||||
|
||||
def get_buffer_log_key(source_id: str) -> str:
|
||||
return f"{LOG_BUFFER_KEY_PREFIX}:{source_id}"
|
||||
|
||||
|
||||
def append_buffer_log(
|
||||
source_id: str,
|
||||
*,
|
||||
level: str,
|
||||
message: str,
|
||||
context: dict[str, Any] | None = None,
|
||||
) -> None:
|
||||
payload = {
|
||||
"timestamp": datetime.now(tz=UTC).isoformat(),
|
||||
"level": normalize_log_level(level),
|
||||
"message": message,
|
||||
"context": context or {},
|
||||
}
|
||||
buffer_key = get_buffer_log_key(source_id)
|
||||
redis_client.rpush(buffer_key, json.dumps(payload, ensure_ascii=False))
|
||||
redis_client.ltrim(buffer_key, -BUFFER_LOG_LIMIT, -1)
|
||||
redis_client.expire(buffer_key, BUFFER_LOG_TTL_SECONDS)
|
||||
|
||||
|
||||
def parse_timestamp(raw_value: str | None) -> datetime | None:
|
||||
if not raw_value:
|
||||
return None
|
||||
candidate = str(raw_value).strip()
|
||||
if not candidate:
|
||||
return None
|
||||
candidate = candidate.replace("Z", "+00:00")
|
||||
try:
|
||||
parsed = datetime.fromisoformat(candidate)
|
||||
return parsed if parsed.tzinfo else parsed.replace(tzinfo=UTC)
|
||||
except ValueError:
|
||||
pass
|
||||
for fmt in TIMESTAMP_FORMATS:
|
||||
try:
|
||||
return datetime.strptime(candidate, fmt).replace(tzinfo=UTC)
|
||||
except ValueError:
|
||||
continue
|
||||
return None
|
||||
|
||||
|
||||
def parse_prefixed_timestamp(line: str) -> tuple[datetime | None, str]:
|
||||
stripped = line.strip()
|
||||
if not stripped:
|
||||
return None, ""
|
||||
for prefix_length in (35, 32, 29, 26, 23, 19):
|
||||
if len(stripped) < prefix_length:
|
||||
continue
|
||||
prefix = stripped[:prefix_length]
|
||||
timestamp = parse_timestamp(prefix)
|
||||
if timestamp is not None:
|
||||
return timestamp, stripped[prefix_length:].lstrip()
|
||||
first_token = stripped.split(maxsplit=1)[0]
|
||||
timestamp = parse_timestamp(first_token)
|
||||
if timestamp is not None:
|
||||
remainder = stripped[len(first_token):].lstrip()
|
||||
return timestamp, remainder
|
||||
return None, stripped
|
||||
|
||||
|
||||
def infer_log_level_from_text(text: str, *, allow_embedded: bool = True) -> str | None:
|
||||
leading_match = LEADING_LEVEL_PATTERN.match(text)
|
||||
if leading_match:
|
||||
return normalize_log_level(leading_match.group(1))
|
||||
|
||||
if allow_embedded:
|
||||
embedded_match = EMBEDDED_LEVEL_PATTERN.search(text)
|
||||
if embedded_match:
|
||||
return normalize_log_level(embedded_match.group(1))
|
||||
upper_text = text.upper()
|
||||
for pattern, normalized in LEVEL_PATTERNS:
|
||||
if f"{pattern}:" in upper_text or f"{pattern} " in upper_text:
|
||||
return normalized
|
||||
return None
|
||||
|
||||
|
||||
def build_display_line(timestamp: datetime | None, level: str | None, message: str) -> str:
|
||||
message_part = message.strip() if message else ""
|
||||
parts = []
|
||||
if timestamp is not None:
|
||||
parts.append(timestamp.astimezone(UTC).strftime("%Y-%m-%d %H:%M:%S"))
|
||||
if level:
|
||||
parts.append(level.upper())
|
||||
if message_part:
|
||||
parts.append(message_part)
|
||||
return " ".join(parts).strip()
|
||||
|
||||
|
||||
def sanitize_text_log_line(line: str) -> str:
|
||||
return CONTROL_CHAR_PATTERN.sub("", line)
|
||||
|
||||
|
||||
def parse_text_log_entry(line: str) -> StructuredLogEntry:
|
||||
sanitized_line = sanitize_text_log_line(line).rstrip("\n")
|
||||
timestamp, remainder = parse_prefixed_timestamp(sanitized_line)
|
||||
level = infer_log_level_from_text(remainder or sanitized_line, allow_embedded=False)
|
||||
display_line = sanitized_line
|
||||
return StructuredLogEntry(
|
||||
timestamp=timestamp,
|
||||
level=level,
|
||||
display_line=display_line,
|
||||
raw_line=display_line,
|
||||
search_text=display_line.lower(),
|
||||
)
|
||||
|
||||
|
||||
def build_buffer_entry(payload: dict[str, Any]) -> StructuredLogEntry:
|
||||
timestamp = parse_timestamp(str(payload.get("timestamp", "")).strip())
|
||||
level = normalize_log_level(payload.get("level"))
|
||||
if level == LOG_LEVEL_ALL:
|
||||
level = None
|
||||
message = str(payload.get("message", "")).strip()
|
||||
context = payload.get("context")
|
||||
context_map = context if isinstance(context, dict) else {}
|
||||
context_fragments = []
|
||||
for key in ("category", "module", "url", "detail"):
|
||||
value = str(context_map.get(key, "")).strip()
|
||||
if value:
|
||||
context_fragments.append(f"{key}={value}")
|
||||
message_with_context = " | ".join([message, *context_fragments]) if context_fragments else message
|
||||
display_line = build_display_line(timestamp, level, message_with_context)
|
||||
search_text = " ".join(
|
||||
[
|
||||
message,
|
||||
json.dumps(context_map, ensure_ascii=False, sort_keys=True),
|
||||
display_line,
|
||||
]
|
||||
).lower()
|
||||
return StructuredLogEntry(
|
||||
timestamp=timestamp,
|
||||
level=level,
|
||||
display_line=display_line,
|
||||
raw_line=json.dumps(payload, ensure_ascii=False, sort_keys=True),
|
||||
search_text=search_text,
|
||||
)
|
||||
|
||||
|
||||
def read_file_entries(source: LogSource, scan_limit: int) -> list[StructuredLogEntry]:
|
||||
path = Path(source.location)
|
||||
if not path.exists():
|
||||
return []
|
||||
with path.open("r", encoding="utf-8", errors="replace") as handle:
|
||||
recent_lines = deque(handle, maxlen=scan_limit)
|
||||
return [
|
||||
parse_text_log_entry(line)
|
||||
for line in recent_lines
|
||||
if sanitize_text_log_line(line).strip()
|
||||
]
|
||||
|
||||
|
||||
def read_docker_entries(source: LogSource, scan_limit: int) -> list[StructuredLogEntry]:
|
||||
if not shutil.which("docker") or not source.container_name:
|
||||
return []
|
||||
try:
|
||||
completed = subprocess.run(
|
||||
[
|
||||
"docker",
|
||||
"logs",
|
||||
"--timestamps",
|
||||
"--tail",
|
||||
str(scan_limit),
|
||||
source.container_name,
|
||||
],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
except OSError:
|
||||
return []
|
||||
if completed.returncode != 0:
|
||||
return []
|
||||
return [
|
||||
parse_text_log_entry(line)
|
||||
for line in completed.stdout.splitlines()
|
||||
if line.strip()
|
||||
]
|
||||
|
||||
|
||||
def read_buffer_entries(source: LogSource, scan_limit: int) -> list[StructuredLogEntry]:
|
||||
if not source.buffer_key:
|
||||
return []
|
||||
try:
|
||||
raw_items = redis_client.lrange(source.buffer_key, -scan_limit, -1)
|
||||
except Exception:
|
||||
return []
|
||||
entries: list[StructuredLogEntry] = []
|
||||
for raw_item in raw_items:
|
||||
try:
|
||||
payload = json.loads(raw_item)
|
||||
except json.JSONDecodeError:
|
||||
entries.append(parse_text_log_entry(str(raw_item)))
|
||||
continue
|
||||
if isinstance(payload, dict):
|
||||
entries.append(build_buffer_entry(payload))
|
||||
else:
|
||||
entries.append(parse_text_log_entry(str(raw_item)))
|
||||
return entries
|
||||
|
||||
|
||||
def read_source_entries(source: LogSource, scan_limit: int) -> list[StructuredLogEntry]:
|
||||
if source.kind == "file":
|
||||
return read_file_entries(source, scan_limit)
|
||||
if source.kind == "docker":
|
||||
return read_docker_entries(source, scan_limit)
|
||||
if source.kind == "buffer":
|
||||
return read_buffer_entries(source, scan_limit)
|
||||
return []
|
||||
|
||||
|
||||
def matches_levels(entry: StructuredLogEntry, selected_levels: tuple[str, ...]) -> bool:
|
||||
if not selected_levels:
|
||||
return True
|
||||
return entry.level in selected_levels
|
||||
|
||||
|
||||
def matches_date_range(
|
||||
entry: StructuredLogEntry,
|
||||
start_date: str | None,
|
||||
end_date: str | None,
|
||||
) -> bool:
|
||||
if not start_date and not end_date:
|
||||
return True
|
||||
if entry.timestamp is None:
|
||||
return False
|
||||
date_token = entry.timestamp.astimezone(UTC).date().isoformat()
|
||||
if start_date and date_token < start_date:
|
||||
return False
|
||||
if end_date and date_token > end_date:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def matches_search(entry: StructuredLogEntry, search: str | None) -> bool:
|
||||
if search is None:
|
||||
return True
|
||||
query = search.strip().lower()
|
||||
if not query:
|
||||
return True
|
||||
return query in entry.search_text
|
||||
|
||||
|
||||
def build_daily_log_markers(entries: list[StructuredLogEntry]) -> list[dict[str, Any]]:
|
||||
grouped: dict[str, list[StructuredLogEntry]] = {}
|
||||
for entry in entries:
|
||||
if entry.timestamp is None:
|
||||
continue
|
||||
date_token = entry.timestamp.astimezone(UTC).date().isoformat()
|
||||
grouped.setdefault(date_token, []).append(entry)
|
||||
|
||||
markers: list[DailyLogMarker] = []
|
||||
for date_token, group in sorted(grouped.items()):
|
||||
level_counts = Counter(
|
||||
entry.level
|
||||
for entry in group
|
||||
if entry.level in SUPPORTED_LOG_LEVELS and entry.level != LOG_LEVEL_ALL
|
||||
)
|
||||
dominant_level = LOG_LEVEL_INFO
|
||||
if level_counts:
|
||||
dominant_level = sorted(
|
||||
level_counts.items(),
|
||||
key=lambda item: (
|
||||
-item[1],
|
||||
("error", "warning", "info", "debug").index(item[0]),
|
||||
),
|
||||
)[0][0]
|
||||
markers.append(
|
||||
DailyLogMarker(
|
||||
date_token=date_token,
|
||||
total=len(group),
|
||||
dominant_level=dominant_level,
|
||||
)
|
||||
)
|
||||
return [marker.__dict__ for marker in markers]
|
||||
|
||||
|
||||
def read_log_snapshot(
|
||||
source_id: str,
|
||||
limit: int,
|
||||
*,
|
||||
level: str = LOG_LEVEL_ALL,
|
||||
levels: str | None = None,
|
||||
start_date: str | None = None,
|
||||
end_date: str | None = None,
|
||||
search: str | None = None,
|
||||
) -> dict[str, Any] | None:
|
||||
source = LOG_SOURCES.get(source_id)
|
||||
if source is None:
|
||||
return None
|
||||
|
||||
selected_levels = normalize_log_levels(level, levels)
|
||||
search_query = (search or "").strip()
|
||||
scan_limit = max(min(MAX_LOG_LINE_LIMIT * 5, 5000), limit * 5, BUFFER_LOG_LIMIT if source.kind == "buffer" else 1000)
|
||||
all_entries = read_source_entries(source, scan_limit)
|
||||
marker_entries = [
|
||||
entry
|
||||
for entry in all_entries
|
||||
if matches_levels(entry, selected_levels) and matches_search(entry, search_query)
|
||||
]
|
||||
filtered_entries = [
|
||||
entry
|
||||
for entry in marker_entries
|
||||
if matches_date_range(entry, start_date, end_date)
|
||||
]
|
||||
visible_entries = filtered_entries[-limit:]
|
||||
|
||||
compatibility_level = selected_levels[0] if len(selected_levels) == 1 else LOG_LEVEL_ALL
|
||||
return {
|
||||
"source_id": source.source_id,
|
||||
"name": source.name,
|
||||
"kind": source.kind,
|
||||
"location": source.location,
|
||||
"description": source.description,
|
||||
"category": source.category,
|
||||
"status": get_source_status(source),
|
||||
"level": compatibility_level,
|
||||
"selected_levels": list(selected_levels),
|
||||
"search_query": search_query,
|
||||
"available_levels": [
|
||||
LOG_LEVEL_ALL,
|
||||
LOG_LEVEL_ERROR,
|
||||
LOG_LEVEL_WARNING,
|
||||
LOG_LEVEL_INFO,
|
||||
LOG_LEVEL_DEBUG,
|
||||
],
|
||||
"daily_markers": build_daily_log_markers(marker_entries),
|
||||
"line_limit": limit,
|
||||
"line_count": len(visible_entries),
|
||||
"lines": [entry.display_line for entry in visible_entries],
|
||||
}
|
||||
@@ -17,7 +17,7 @@ TV_LIVE_SOURCE_COLLECTOR = "news_live_streams"
|
||||
TV_LIVE_SOURCE_DATA_TYPE = "news_live_stream"
|
||||
|
||||
DEFAULT_TV_SETTINGS = {
|
||||
"default_source_id": DEFAULT_TV_SOURCE_ID,
|
||||
"default_source_id": DEFAULT_TV_SOURCE_ID,
|
||||
"auto_fallback": True,
|
||||
"sources": [
|
||||
{
|
||||
@@ -362,7 +362,7 @@ def _build_collected_tv_source(record: CollectedData, index: int) -> dict[str, A
|
||||
"sort_order": metadata.get("sort_order", 200 + index),
|
||||
"collector_source": record.source,
|
||||
"notes": record.description or metadata.get("notes") or "",
|
||||
"updated_at": to_iso8601_utc(record.updated_at or record.reference_date or datetime.now(UTC)),
|
||||
"updated_at": to_iso8601_utc(record.collected_at or record.reference_date or datetime.now(UTC)),
|
||||
},
|
||||
index=index,
|
||||
)
|
||||
|
||||
@@ -35,6 +35,7 @@ async def test_health_check():
|
||||
data = response.json()
|
||||
assert data["status"] == "healthy"
|
||||
assert "version" in data
|
||||
assert response.headers["x-request-id"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@@ -161,6 +162,345 @@ async def test_alerts_endpoint_with_auth(auth_headers):
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_system_log_sources_requires_super_admin(auth_headers):
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="testuser",
|
||||
email="test@example.com",
|
||||
password_hash="hashed",
|
||||
role="admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/api/v1/system/logs/sources", headers=auth_headers)
|
||||
assert response.status_code == 403
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_system_log_sources_with_super_admin(auth_headers):
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="root",
|
||||
email="root@example.com",
|
||||
password_hash="hashed",
|
||||
role="super_admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch(
|
||||
"app.api.v1.system_control.list_log_sources",
|
||||
return_value=[
|
||||
{
|
||||
"source_id": "backend",
|
||||
"name": "后端服务",
|
||||
"kind": "file",
|
||||
"location": "/tmp/planet_backend.log",
|
||||
"description": "FastAPI 后端、调度器和采集任务共享日志。",
|
||||
"category": "service",
|
||||
"status": "ok",
|
||||
}
|
||||
],
|
||||
):
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/api/v1/system/logs/sources", headers=auth_headers)
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["items"][0]["source_id"] == "backend"
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_system_log_snapshot_with_super_admin(auth_headers):
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="root",
|
||||
email="root@example.com",
|
||||
password_hash="hashed",
|
||||
role="super_admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch(
|
||||
"app.api.v1.system_control.read_log_snapshot",
|
||||
return_value={
|
||||
"source_id": "backend",
|
||||
"name": "后端服务",
|
||||
"kind": "file",
|
||||
"location": "/tmp/planet_backend.log",
|
||||
"description": "FastAPI 后端、调度器和采集任务共享日志。",
|
||||
"category": "service",
|
||||
"status": "ok",
|
||||
"level": "all",
|
||||
"selected_levels": [],
|
||||
"search_query": "",
|
||||
"available_levels": ["all", "error", "warning", "info", "debug"],
|
||||
"daily_markers": [],
|
||||
"line_limit": 50,
|
||||
"line_count": 2,
|
||||
"lines": ["line 1", "line 2"],
|
||||
},
|
||||
):
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/api/v1/system/logs/backend?limit=50", headers=auth_headers)
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["source_id"] == "backend"
|
||||
assert data["line_count"] == 2
|
||||
assert data["lines"] == ["line 1", "line 2"]
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_system_log_snapshot_supports_level_filter(auth_headers):
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="root",
|
||||
email="root@example.com",
|
||||
password_hash="hashed",
|
||||
role="super_admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch(
|
||||
"app.api.v1.system_control.read_log_snapshot",
|
||||
return_value={
|
||||
"source_id": "backend",
|
||||
"name": "后端服务",
|
||||
"kind": "file",
|
||||
"location": "/tmp/planet_backend.log",
|
||||
"description": "FastAPI 后端、调度器和采集任务共享日志。",
|
||||
"category": "service",
|
||||
"status": "ok",
|
||||
"level": "error",
|
||||
"selected_levels": ["error"],
|
||||
"search_query": "",
|
||||
"available_levels": ["all", "error", "warning", "info", "debug"],
|
||||
"daily_markers": [],
|
||||
"line_limit": 50,
|
||||
"line_count": 1,
|
||||
"lines": ["ERROR: failed"],
|
||||
},
|
||||
):
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/api/v1/system/logs/backend?limit=50&level=error", headers=auth_headers)
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["level"] == "error"
|
||||
assert data["lines"] == ["ERROR: failed"]
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_system_log_snapshot_supports_date_range_filter(auth_headers):
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="root",
|
||||
email="root@example.com",
|
||||
password_hash="hashed",
|
||||
role="super_admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch(
|
||||
"app.api.v1.system_control.read_log_snapshot",
|
||||
return_value={
|
||||
"source_id": "backend",
|
||||
"name": "后端服务",
|
||||
"kind": "file",
|
||||
"location": "/tmp/planet_backend.log",
|
||||
"description": "FastAPI 后端、调度器和采集任务共享日志。",
|
||||
"category": "service",
|
||||
"status": "ok",
|
||||
"level": "all",
|
||||
"selected_levels": [],
|
||||
"search_query": "",
|
||||
"available_levels": ["all", "error", "warning", "info", "debug"],
|
||||
"daily_markers": [],
|
||||
"line_limit": 50,
|
||||
"line_count": 1,
|
||||
"lines": ["2026-04-23 INFO: service started"],
|
||||
},
|
||||
) as mock_read_log_snapshot:
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get(
|
||||
"/api/v1/system/logs/backend?limit=50&start_date=2026-04-20&end_date=2026-04-23",
|
||||
headers=auth_headers,
|
||||
)
|
||||
assert response.status_code == 200
|
||||
mock_read_log_snapshot.assert_called_once_with(
|
||||
"backend",
|
||||
50,
|
||||
level="all",
|
||||
levels=None,
|
||||
start_date="2026-04-20",
|
||||
end_date="2026-04-23",
|
||||
search=None,
|
||||
)
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_system_log_snapshot_supports_levels_and_search_filter(auth_headers):
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="root",
|
||||
email="root@example.com",
|
||||
password_hash="hashed",
|
||||
role="super_admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch(
|
||||
"app.api.v1.system_control.read_log_snapshot",
|
||||
return_value={
|
||||
"source_id": "backend",
|
||||
"name": "后端服务",
|
||||
"kind": "file",
|
||||
"location": "/tmp/planet_backend.log",
|
||||
"description": "FastAPI 后端、调度器和采集任务共享日志。",
|
||||
"category": "service",
|
||||
"status": "ok",
|
||||
"level": "all",
|
||||
"selected_levels": ["error", "warning"],
|
||||
"search_query": "timeout",
|
||||
"available_levels": ["all", "error", "warning", "info", "debug"],
|
||||
"daily_markers": [],
|
||||
"line_limit": 50,
|
||||
"line_count": 1,
|
||||
"lines": ["2026-04-23 10:00:00 ERROR timeout"],
|
||||
},
|
||||
) as mock_read_log_snapshot:
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get(
|
||||
"/api/v1/system/logs/backend?limit=50&levels=error,warning&search=timeout",
|
||||
headers=auth_headers,
|
||||
)
|
||||
assert response.status_code == 200
|
||||
mock_read_log_snapshot.assert_called_once_with(
|
||||
"backend",
|
||||
50,
|
||||
level="all",
|
||||
levels="error,warning",
|
||||
start_date=None,
|
||||
end_date=None,
|
||||
search="timeout",
|
||||
)
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_system_log_snapshot_rejects_invalid_date_range(auth_headers):
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="root",
|
||||
email="root@example.com",
|
||||
password_hash="hashed",
|
||||
role="super_admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get(
|
||||
"/api/v1/system/logs/backend?start_date=2026-04-31",
|
||||
headers=auth_headers,
|
||||
)
|
||||
assert response.status_code == 400
|
||||
assert "start_date must be in YYYY-MM-DD format" in response.json()["detail"]
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ingest_earth_client_log_accepts_public_events():
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch("app.api.v1.system_control.append_buffer_log") as mock_append_buffer_log:
|
||||
with patch("app.api.v1.system_control.record_system_log", new_callable=AsyncMock) as mock_record_system_log:
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.post(
|
||||
"/api/v1/system/logs/earth-client",
|
||||
json={
|
||||
"level": "error",
|
||||
"message": "登陆点加载失败: 登陆点接口返回 HTTP 500",
|
||||
"category": "startup-load",
|
||||
"module": "layer-startup",
|
||||
},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["accepted"] is True
|
||||
assert data["source_id"] == "earth-client"
|
||||
mock_append_buffer_log.assert_called_once()
|
||||
mock_record_system_log.assert_awaited_once()
|
||||
persisted_kwargs = mock_record_system_log.await_args.kwargs
|
||||
assert persisted_kwargs["source"] == "earth-client"
|
||||
assert persisted_kwargs["event"] == "earth.client.runtime_log"
|
||||
assert persisted_kwargs["category"] == "startup-load"
|
||||
assert persisted_kwargs["level"] == "error"
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_request_id_header_is_echoed_when_provided():
|
||||
transport = ASGITransport(app=app)
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/health", headers={"X-Request-ID": "planet-test-request"})
|
||||
assert response.status_code == 200
|
||||
assert response.headers["x-request-id"] == "planet-test-request"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_invalid_token():
|
||||
"""Test that invalid token is rejected"""
|
||||
@@ -263,6 +603,8 @@ async def test_ai_situational_analysis_returns_503_when_disabled(auth_headers):
|
||||
assert "content_blocks" in data
|
||||
assert "text_blocks" in data
|
||||
assert "thinking_blocks" in data
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@@ -382,8 +724,6 @@ async def test_save_playground_session_with_auth(auth_headers):
|
||||
assert data["state"]["objective"] == "测试目标"
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
|
||||
49
backend/tests/test_earth_news.py
Normal file
49
backend/tests/test_earth_news.py
Normal file
@@ -0,0 +1,49 @@
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from app.services.earth_news import ParsedNewsItem, _serialize_item
|
||||
|
||||
|
||||
def test_serialize_item_includes_region_anchor_for_cruise():
|
||||
item = ParsedNewsItem(
|
||||
id="google-apac:test",
|
||||
title="Example APAC story",
|
||||
summary="Example summary",
|
||||
url="https://example.com/story",
|
||||
source="Example Source",
|
||||
feed_name="Global Monitor / APAC",
|
||||
feed_region="asia-pacific",
|
||||
homepage_url="https://example.com",
|
||||
published_at=datetime(2026, 4, 23, 2, 30, tzinfo=UTC),
|
||||
)
|
||||
|
||||
payload = _serialize_item(item, active_region="asia-pacific")
|
||||
|
||||
assert payload["latitude"] == 1.3521
|
||||
assert payload["longitude"] == 103.8198
|
||||
assert payload["location_label"] == "亚太"
|
||||
assert payload["location_inferred"] is True
|
||||
assert payload["is_focus_match"] is True
|
||||
assert payload["published_at"] == "2026-04-23T02:30:00Z"
|
||||
|
||||
|
||||
def test_serialize_item_falls_back_to_global_anchor():
|
||||
item = ParsedNewsItem(
|
||||
id="custom:test",
|
||||
title="Fallback story",
|
||||
summary="Fallback summary",
|
||||
url="https://example.com/fallback",
|
||||
source="Fallback Source",
|
||||
feed_name="Fallback Feed",
|
||||
feed_region="unknown-region",
|
||||
homepage_url="https://example.com",
|
||||
published_at=None,
|
||||
)
|
||||
|
||||
payload = _serialize_item(item, active_region="americas")
|
||||
|
||||
assert payload["latitude"] == 20.0
|
||||
assert payload["longitude"] == 0.0
|
||||
assert payload["location_label"] == "全球"
|
||||
assert payload["location_inferred"] is True
|
||||
assert payload["is_focus_match"] is False
|
||||
assert payload["published_at"] is None
|
||||
78
backend/tests/test_logging.py
Normal file
78
backend/tests/test_logging.py
Normal file
@@ -0,0 +1,78 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
from io import StringIO
|
||||
|
||||
from app.core.logging import PlanetContextFilter, PlanetFormatter, get_logger
|
||||
from app.core.request_context import set_request_id
|
||||
|
||||
|
||||
def _capture_output(callback):
|
||||
stream = StringIO()
|
||||
handler = logging.StreamHandler(stream)
|
||||
handler.setFormatter(PlanetFormatter(datefmt="%Y-%m-%d %H:%M:%S"))
|
||||
handler.addFilter(PlanetContextFilter())
|
||||
|
||||
adapter = get_logger("tests.logging")
|
||||
target_logger = adapter.logger
|
||||
original_handlers = list(target_logger.handlers)
|
||||
original_level = target_logger.level
|
||||
original_propagate = target_logger.propagate
|
||||
|
||||
target_logger.handlers = [handler]
|
||||
target_logger.setLevel(logging.INFO)
|
||||
target_logger.propagate = False
|
||||
|
||||
try:
|
||||
callback(adapter)
|
||||
finally:
|
||||
handler.flush()
|
||||
target_logger.handlers = original_handlers
|
||||
target_logger.setLevel(original_level)
|
||||
target_logger.propagate = original_propagate
|
||||
|
||||
return stream.getvalue()
|
||||
|
||||
|
||||
def test_structured_logger_injects_request_id_and_event():
|
||||
set_request_id("req-test-123")
|
||||
try:
|
||||
output = _capture_output(
|
||||
lambda logger: logger.info_event(
|
||||
"collector started",
|
||||
event="collector.run.started",
|
||||
context={"collector_name": "bgp_news"},
|
||||
)
|
||||
)
|
||||
finally:
|
||||
set_request_id(None)
|
||||
|
||||
assert "request_id=req-test-123" in output
|
||||
assert "event=collector.run.started" in output
|
||||
assert "service=backend" in output
|
||||
assert '"collector_name": "bgp_news"' in output
|
||||
|
||||
|
||||
def test_structured_logger_redacts_sensitive_text_and_context():
|
||||
set_request_id("req-test-redact")
|
||||
try:
|
||||
output = _capture_output(
|
||||
lambda logger: logger.error_event(
|
||||
"Authorization: Bearer super-secret-token",
|
||||
event="auth.token.failed",
|
||||
context={
|
||||
"token": "plain-secret",
|
||||
"nested": {"password": "hunter2"},
|
||||
"safe": "visible",
|
||||
},
|
||||
)
|
||||
)
|
||||
finally:
|
||||
set_request_id(None)
|
||||
|
||||
assert "super-secret-token" not in output
|
||||
assert "plain-secret" not in output
|
||||
assert "hunter2" not in output
|
||||
assert "[REDACTED]" in output
|
||||
assert '"safe": "visible"' in output
|
||||
217
backend/tests/test_system_logs.py
Normal file
217
backend/tests/test_system_logs.py
Normal file
@@ -0,0 +1,217 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from app.services import system_logs
|
||||
|
||||
|
||||
class FakeRedis:
|
||||
def __init__(self) -> None:
|
||||
self.store: dict[str, list[str]] = {}
|
||||
|
||||
def rpush(self, key: str, value: str) -> None:
|
||||
self.store.setdefault(key, []).append(value)
|
||||
|
||||
def ltrim(self, key: str, start: int, end: int) -> None:
|
||||
items = self.store.get(key, [])
|
||||
normalized_end = None if end == -1 else end + 1
|
||||
self.store[key] = items[start:normalized_end]
|
||||
|
||||
def expire(self, key: str, seconds: int) -> None:
|
||||
return None
|
||||
|
||||
def lrange(self, key: str, start: int, end: int) -> list[str]:
|
||||
items = self.store.get(key, [])
|
||||
normalized_end = None if end == -1 else end + 1
|
||||
return items[start:normalized_end]
|
||||
|
||||
def llen(self, key: str) -> int:
|
||||
return len(self.store.get(key, []))
|
||||
|
||||
|
||||
def test_read_log_snapshot_uses_structured_buffer_timestamp_level_and_search(monkeypatch):
|
||||
fake_redis = FakeRedis()
|
||||
monkeypatch.setattr(system_logs, "redis_client", fake_redis)
|
||||
monkeypatch.setattr(
|
||||
system_logs,
|
||||
"LOG_SOURCES",
|
||||
{
|
||||
"earth-client": system_logs.LogSource(
|
||||
source_id="earth-client",
|
||||
name="Earth 浏览器端",
|
||||
kind="buffer",
|
||||
location="redis://planet:system_logs:earth-client",
|
||||
description="Earth 浏览器端上报日志",
|
||||
category="client",
|
||||
buffer_key=system_logs.get_buffer_log_key("earth-client"),
|
||||
)
|
||||
},
|
||||
)
|
||||
|
||||
fake_redis.rpush(
|
||||
system_logs.get_buffer_log_key("earth-client"),
|
||||
json.dumps(
|
||||
{
|
||||
"timestamp": "2026-04-22T10:15:30Z",
|
||||
"level": "warning",
|
||||
"message": "news feed degraded",
|
||||
"context": {"module": "news", "detail": "timeout"},
|
||||
},
|
||||
ensure_ascii=False,
|
||||
),
|
||||
)
|
||||
fake_redis.rpush(
|
||||
system_logs.get_buffer_log_key("earth-client"),
|
||||
json.dumps(
|
||||
{
|
||||
"timestamp": "2026-04-23T06:01:00Z",
|
||||
"level": "error",
|
||||
"message": "landing points failed",
|
||||
"context": {"module": "layer-startup", "detail": "http 500"},
|
||||
},
|
||||
ensure_ascii=False,
|
||||
),
|
||||
)
|
||||
|
||||
snapshot = system_logs.read_log_snapshot(
|
||||
"earth-client",
|
||||
50,
|
||||
levels="error,warning",
|
||||
start_date="2026-04-23",
|
||||
end_date="2026-04-23",
|
||||
search="landing",
|
||||
)
|
||||
|
||||
assert snapshot is not None
|
||||
assert snapshot["selected_levels"] == ["error", "warning"]
|
||||
assert snapshot["search_query"] == "landing"
|
||||
assert snapshot["line_count"] == 1
|
||||
assert snapshot["lines"][0].startswith("2026-04-23 06:01:00 ERROR landing points failed")
|
||||
assert snapshot["daily_markers"] == [
|
||||
{"date_token": "2026-04-23", "total": 1, "dominant_level": "error"}
|
||||
]
|
||||
|
||||
|
||||
def test_read_log_snapshot_parses_file_timestamp_and_builds_markers(tmp_path: Path, monkeypatch):
|
||||
log_path = tmp_path / "backend.log"
|
||||
log_path.write_text(
|
||||
"\n".join(
|
||||
[
|
||||
"2026-04-22 08:00:00 INFO service booted",
|
||||
"2026-04-23 09:15:00 WARNING disk pressure detected",
|
||||
"2026-04-23 09:16:00 ERROR sync failed",
|
||||
"2026-04-24 10:00:00 DEBUG collector trace",
|
||||
]
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
monkeypatch.setattr(
|
||||
system_logs,
|
||||
"LOG_SOURCES",
|
||||
{
|
||||
"backend": system_logs.LogSource(
|
||||
source_id="backend",
|
||||
name="后端服务",
|
||||
kind="file",
|
||||
location=str(log_path),
|
||||
description="测试文件日志",
|
||||
category="service",
|
||||
)
|
||||
},
|
||||
)
|
||||
|
||||
snapshot = system_logs.read_log_snapshot(
|
||||
"backend",
|
||||
50,
|
||||
levels="warning,error",
|
||||
search="failed",
|
||||
)
|
||||
|
||||
assert snapshot is not None
|
||||
assert snapshot["line_count"] == 1
|
||||
assert snapshot["lines"] == ["2026-04-23 09:16:00 ERROR sync failed"]
|
||||
assert snapshot["daily_markers"] == [
|
||||
{"date_token": "2026-04-23", "total": 1, "dominant_level": "error"}
|
||||
]
|
||||
assert snapshot["status"] == "ok"
|
||||
|
||||
|
||||
def test_append_buffer_log_persists_normalized_level(monkeypatch):
|
||||
fake_redis = FakeRedis()
|
||||
monkeypatch.setattr(system_logs, "redis_client", fake_redis)
|
||||
|
||||
system_logs.append_buffer_log(
|
||||
"earth-client",
|
||||
level="warn",
|
||||
message="feed delayed",
|
||||
context={"module": "news"},
|
||||
)
|
||||
|
||||
stored_items = fake_redis.lrange(system_logs.get_buffer_log_key("earth-client"), 0, -1)
|
||||
payload = json.loads(stored_items[0])
|
||||
assert payload["level"] == "warning"
|
||||
assert payload["message"] == "feed delayed"
|
||||
|
||||
|
||||
def test_infer_log_level_prefers_leading_prefix_over_query_string():
|
||||
line = 'INFO: 127.0.0.1 - "GET /api/v1/system/logs/backend?limit=200&level=error&levels=error HTTP/1.1" 200 OK'
|
||||
|
||||
entry = system_logs.parse_text_log_entry(line)
|
||||
|
||||
assert entry.level == "info"
|
||||
|
||||
|
||||
def test_parse_text_log_entry_does_not_promote_exception_context_to_error():
|
||||
line = "websockets.exceptions.ConnectionClosedError: sent 1011 (internal error) keepalive ping timeout"
|
||||
|
||||
entry = system_logs.parse_text_log_entry(line)
|
||||
|
||||
assert entry.level is None
|
||||
|
||||
|
||||
def test_parse_text_log_entry_still_detects_explicit_error_prefix():
|
||||
line = "ERROR: [Errno 98] Address already in use"
|
||||
|
||||
entry = system_logs.parse_text_log_entry(line)
|
||||
|
||||
assert entry.level == "error"
|
||||
|
||||
|
||||
def test_read_log_snapshot_strips_nul_bytes_from_file_lines(tmp_path: Path, monkeypatch):
|
||||
log_path = tmp_path / "backend.log"
|
||||
log_path.write_bytes(
|
||||
(
|
||||
b"INFO: service booted\n"
|
||||
b"ERROR: bind failed\n"
|
||||
+ b"\x00" * 32
|
||||
+ b"2026-04-23 23:41:32 INFO service=backend message=request served\n"
|
||||
)
|
||||
)
|
||||
|
||||
monkeypatch.setattr(
|
||||
system_logs,
|
||||
"LOG_SOURCES",
|
||||
{
|
||||
"backend": system_logs.LogSource(
|
||||
source_id="backend",
|
||||
name="后端服务",
|
||||
kind="file",
|
||||
location=str(log_path),
|
||||
description="测试文件日志",
|
||||
category="service",
|
||||
)
|
||||
},
|
||||
)
|
||||
|
||||
snapshot = system_logs.read_log_snapshot("backend", 50)
|
||||
|
||||
assert snapshot is not None
|
||||
assert snapshot["line_count"] == 3
|
||||
assert snapshot["lines"] == [
|
||||
"INFO: service booted",
|
||||
"ERROR: bind failed",
|
||||
"2026-04-23 23:41:32 INFO service=backend message=request served",
|
||||
]
|
||||
347
backend/tests/test_visualization_compute_centers.py
Normal file
347
backend/tests/test_visualization_compute_centers.py
Normal file
@@ -0,0 +1,347 @@
|
||||
from datetime import datetime, timezone
|
||||
|
||||
import pytest
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
|
||||
from app.api.v1.visualization import convert_compute_centers_to_geojson
|
||||
from app.db.session import get_db
|
||||
from app.main import app
|
||||
from app.models.collected_data import CollectedData
|
||||
|
||||
|
||||
def _build_record(
|
||||
*,
|
||||
record_id: int,
|
||||
source: str,
|
||||
data_type: str,
|
||||
name: str,
|
||||
country: str,
|
||||
city: str,
|
||||
latitude: float,
|
||||
longitude: float,
|
||||
metadata: dict,
|
||||
):
|
||||
return CollectedData(
|
||||
id=record_id,
|
||||
source=source,
|
||||
data_type=data_type,
|
||||
source_id=f"{source}-{record_id}",
|
||||
name=name,
|
||||
extra_data={
|
||||
"country": country,
|
||||
"city": city,
|
||||
"latitude": latitude,
|
||||
"longitude": longitude,
|
||||
**metadata,
|
||||
},
|
||||
collected_at=datetime(2026, 4, 22, tzinfo=timezone.utc),
|
||||
reference_date=datetime(2026, 4, 21, tzinfo=timezone.utc),
|
||||
is_current=True,
|
||||
)
|
||||
|
||||
|
||||
def test_convert_compute_centers_to_geojson_unifies_sources():
|
||||
top500_record = _build_record(
|
||||
record_id=1,
|
||||
source="top500",
|
||||
data_type="supercomputer",
|
||||
name="Frontier",
|
||||
country="United States",
|
||||
city="Oak Ridge",
|
||||
latitude=35.93,
|
||||
longitude=-84.31,
|
||||
metadata={
|
||||
"rank": 1,
|
||||
"manufacturer": "HPE",
|
||||
"organization": "ORNL",
|
||||
"rmax": 1102000.0,
|
||||
"cores": 8730112,
|
||||
"power": 21510.0,
|
||||
},
|
||||
)
|
||||
gpu_record = _build_record(
|
||||
record_id=2,
|
||||
source="epoch_ai_gpu",
|
||||
data_type="gpu_cluster",
|
||||
name="Colossus",
|
||||
country="United States",
|
||||
city="Memphis",
|
||||
latitude=35.15,
|
||||
longitude=-90.05,
|
||||
metadata={
|
||||
"organization": "xAI",
|
||||
"gpu_type": "H100",
|
||||
"gpu_count": 100000,
|
||||
"value": "20000",
|
||||
"unit": "TFlop/s",
|
||||
},
|
||||
)
|
||||
|
||||
payload = convert_compute_centers_to_geojson([top500_record, gpu_record])
|
||||
|
||||
assert payload["type"] == "FeatureCollection"
|
||||
assert len(payload["features"]) == 2
|
||||
|
||||
supercomputer_feature = payload["features"][0]
|
||||
assert supercomputer_feature["properties"]["site_type"] == "supercomputer"
|
||||
assert supercomputer_feature["properties"]["capacity_unit"] == "GFlops"
|
||||
assert supercomputer_feature["properties"]["capacity_band"] == "exascale"
|
||||
assert supercomputer_feature["properties"]["operator"] == "ORNL"
|
||||
assert supercomputer_feature["properties"]["location_precision"] == "precise"
|
||||
assert supercomputer_feature["properties"]["is_estimated"] is False
|
||||
|
||||
gpu_feature = payload["features"][1]
|
||||
assert gpu_feature["properties"]["site_type"] == "gpu_cluster"
|
||||
assert gpu_feature["properties"]["vendor"] == "H100"
|
||||
assert gpu_feature["properties"]["gpu_count"] == 100000
|
||||
assert gpu_feature["properties"]["capacity_band"] == "large"
|
||||
assert gpu_feature["properties"]["location_precision"] == "precise"
|
||||
|
||||
|
||||
def test_convert_compute_centers_to_geojson_uses_coordinate_hints():
|
||||
hinted_record = _build_record(
|
||||
record_id=3,
|
||||
source="top500",
|
||||
data_type="supercomputer",
|
||||
name="Frontier",
|
||||
country="United States",
|
||||
city="",
|
||||
latitude=0.0,
|
||||
longitude=0.0,
|
||||
metadata={
|
||||
"organization": "Oak Ridge National Laboratory",
|
||||
"rmax": 1102000.0,
|
||||
},
|
||||
)
|
||||
|
||||
payload = convert_compute_centers_to_geojson([hinted_record])
|
||||
|
||||
assert len(payload["features"]) == 1
|
||||
coords = payload["features"][0]["geometry"]["coordinates"]
|
||||
assert coords[0] == pytest.approx(-84.3107)
|
||||
assert coords[1] == pytest.approx(35.9319)
|
||||
assert payload["features"][0]["properties"]["is_estimated"] is True
|
||||
assert payload["features"][0]["properties"]["location_precision"] == "estimated_site"
|
||||
|
||||
|
||||
def test_convert_compute_centers_to_geojson_falls_back_to_country_centroid():
|
||||
centroid_record = _build_record(
|
||||
record_id=4,
|
||||
source="epoch_ai_gpu",
|
||||
data_type="gpu_cluster",
|
||||
name="Unknown Cluster",
|
||||
country="United States",
|
||||
city="",
|
||||
latitude=0.0,
|
||||
longitude=0.0,
|
||||
metadata={
|
||||
"organization": "Unknown Operator",
|
||||
"value": "10000",
|
||||
"unit": "TFlop/s",
|
||||
},
|
||||
)
|
||||
|
||||
payload = convert_compute_centers_to_geojson([centroid_record])
|
||||
|
||||
assert len(payload["features"]) == 1
|
||||
props = payload["features"][0]["properties"]
|
||||
coords = payload["features"][0]["geometry"]["coordinates"]
|
||||
assert coords[0] == pytest.approx(-98.5795)
|
||||
assert coords[1] == pytest.approx(39.8283)
|
||||
assert props["is_estimated"] is True
|
||||
assert props["location_precision"] == "estimated_country"
|
||||
assert props["geography_mode"] == "country_centroid"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_compute_centers_geojson_endpoint_returns_stats():
|
||||
records = [
|
||||
_build_record(
|
||||
record_id=1,
|
||||
source="top500",
|
||||
data_type="supercomputer",
|
||||
name="Frontier",
|
||||
country="United States",
|
||||
city="Oak Ridge",
|
||||
latitude=35.93,
|
||||
longitude=-84.31,
|
||||
metadata={"rank": 1, "rmax": 1102000.0},
|
||||
),
|
||||
_build_record(
|
||||
record_id=2,
|
||||
source="epoch_ai_gpu",
|
||||
data_type="gpu_cluster",
|
||||
name="Colossus",
|
||||
country="United States",
|
||||
city="Memphis",
|
||||
latitude=35.15,
|
||||
longitude=-90.05,
|
||||
metadata={"value": "20000", "unit": "TFlop/s"},
|
||||
),
|
||||
]
|
||||
|
||||
class _ScalarResult:
|
||||
def __init__(self, rows):
|
||||
self._rows = rows
|
||||
|
||||
def scalars(self):
|
||||
class _Scalars:
|
||||
def __init__(self, rows):
|
||||
self._rows = rows
|
||||
|
||||
def all(self):
|
||||
return self._rows
|
||||
|
||||
return _Scalars(self._rows)
|
||||
|
||||
class _FakeSession:
|
||||
async def execute(self, _query):
|
||||
return _ScalarResult(records)
|
||||
|
||||
async def override_get_db():
|
||||
yield _FakeSession()
|
||||
|
||||
app.dependency_overrides[get_db] = override_get_db
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/api/v1/visualization/geo/compute-centers")
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["count"] == 2
|
||||
assert data["stats"]["supercomputers"] == 1
|
||||
assert data["stats"]["gpu_clusters"] == 1
|
||||
assert data["features"][0]["properties"]["data_type"] == "compute_center"
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_visualization_geo_summary_returns_counts(monkeypatch):
|
||||
records = [
|
||||
_build_record(
|
||||
record_id=1,
|
||||
source="arcgis_cables",
|
||||
data_type="submarine_cable",
|
||||
name="Test Cable",
|
||||
country="",
|
||||
city="",
|
||||
latitude=0,
|
||||
longitude=0,
|
||||
metadata={
|
||||
"route_coordinates": [[[0, 0], [1, 1]]],
|
||||
"status": "active",
|
||||
},
|
||||
),
|
||||
_build_record(
|
||||
record_id=2,
|
||||
source="arcgis_landing_points",
|
||||
data_type="landing_point",
|
||||
name="Test Landing",
|
||||
country="United States",
|
||||
city="New York",
|
||||
latitude=40.7,
|
||||
longitude=-74.0,
|
||||
metadata={"city_id": 10},
|
||||
),
|
||||
_build_record(
|
||||
record_id=3,
|
||||
source="celestrak_tle",
|
||||
data_type="satellite_tle",
|
||||
name="TESTSAT",
|
||||
country="",
|
||||
city="",
|
||||
latitude=0,
|
||||
longitude=0,
|
||||
metadata={
|
||||
"norad_cat_id": 12345,
|
||||
"tle_line1": "1 12345U 98067A 24001.00000000 .00000000 00000-0 00000-0 0 9991",
|
||||
"tle_line2": "2 12345 51.6000 100.0000 0001000 10.0000 20.0000 15.50000000 01",
|
||||
},
|
||||
),
|
||||
_build_record(
|
||||
record_id=4,
|
||||
source="top500",
|
||||
data_type="supercomputer",
|
||||
name="Frontier",
|
||||
country="United States",
|
||||
city="Oak Ridge",
|
||||
latitude=35.93,
|
||||
longitude=-84.31,
|
||||
metadata={"rank": 1, "rmax": 1102000.0},
|
||||
),
|
||||
_build_record(
|
||||
record_id=5,
|
||||
source="epoch_ai_gpu",
|
||||
data_type="gpu_cluster",
|
||||
name="Colossus",
|
||||
country="United States",
|
||||
city="Memphis",
|
||||
latitude=35.15,
|
||||
longitude=-90.05,
|
||||
metadata={"value": "20000", "unit": "TFlop/s"},
|
||||
),
|
||||
]
|
||||
|
||||
class _ScalarResult:
|
||||
def __init__(self, rows=None, scalar_value=None):
|
||||
self._rows = rows or []
|
||||
self._scalar_value = scalar_value
|
||||
|
||||
def scalar(self):
|
||||
return self._scalar_value
|
||||
|
||||
def scalars(self):
|
||||
class _Scalars:
|
||||
def __init__(self, rows):
|
||||
self._rows = rows
|
||||
|
||||
def all(self):
|
||||
return self._rows
|
||||
|
||||
return _Scalars(self._rows)
|
||||
|
||||
class _FakeSession:
|
||||
async def execute(self, query):
|
||||
query_text = str(query)
|
||||
if "bgp_incidents" in query_text:
|
||||
return _ScalarResult(scalar_value=2)
|
||||
if "bgp_anomalies" in query_text:
|
||||
return _ScalarResult(scalar_value=3)
|
||||
return _ScalarResult(rows=records)
|
||||
|
||||
async def override_get_db():
|
||||
yield _FakeSession()
|
||||
|
||||
async def _fake_build_bgp_collector_coverage(*_args, **_kwargs):
|
||||
return [
|
||||
{"collector": "rrc00"},
|
||||
{"collector": "rrc01"},
|
||||
]
|
||||
|
||||
monkeypatch.setattr(
|
||||
"app.api.v1.visualization.build_bgp_collector_coverage",
|
||||
_fake_build_bgp_collector_coverage,
|
||||
)
|
||||
|
||||
app.dependency_overrides[get_db] = override_get_db
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/api/v1/visualization/geo/summary")
|
||||
|
||||
assert response.status_code == 200
|
||||
stats = response.json()["stats"]
|
||||
assert stats["cable_count"] == 1
|
||||
assert stats["landing_point_count"] == 1
|
||||
assert stats["satellite_count"] == 1
|
||||
assert stats["compute_center_count"] == 2
|
||||
assert stats["supercomputer_count"] == 1
|
||||
assert stats["gpu_cluster_count"] == 1
|
||||
assert stats["bgp_event_count"] == 2
|
||||
assert stats["bgp_incident_count"] == 2
|
||||
assert stats["bgp_anomaly_count"] == 3
|
||||
assert stats["bgp_collector_count"] == 2
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
@@ -18,6 +18,9 @@ services:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: aiprovider/Dockerfile
|
||||
args:
|
||||
PYTHON_IMAGE: ${PYTHON_IMAGE:-python:3.14-slim}
|
||||
UV_IMAGE: ${UV_IMAGE:-ghcr.io/astral-sh/uv:latest}
|
||||
container_name: planet_aiprovider
|
||||
ports:
|
||||
- "8010:8010"
|
||||
|
||||
@@ -5,6 +5,9 @@ services:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: aiprovider/Dockerfile
|
||||
args:
|
||||
PYTHON_IMAGE: ${PYTHON_IMAGE:-python:3.14-slim}
|
||||
UV_IMAGE: ${UV_IMAGE:-ghcr.io/astral-sh/uv:latest}
|
||||
container_name: planet_aiprovider
|
||||
ports:
|
||||
- "8010:8010"
|
||||
|
||||
@@ -5,6 +5,9 @@ services:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: aiprovider/Dockerfile
|
||||
args:
|
||||
PYTHON_IMAGE: ${PYTHON_IMAGE:-python:3.14-slim}
|
||||
UV_IMAGE: ${UV_IMAGE:-ghcr.io/astral-sh/uv:latest}
|
||||
env_file:
|
||||
- ./aiprovider/.env
|
||||
container_name: planet_aiprovider
|
||||
|
||||
@@ -8,6 +8,451 @@ This project follows the repository versioning rule:
|
||||
- `improvement` -> `+0.0.1`(bugfix + 小功能混合)
|
||||
- `bugfix` -> `+0.0.1`
|
||||
|
||||
## [0.41.2] — 2026-04-27
|
||||
|
||||
### 🔧 Improvements
|
||||
- `planet.sh` 启动链路新增 verbose 滚动输出窗口,并在后端端口占用时打印目标地址和监听进程诊断
|
||||
- Docker 构建支持通过 build args 覆盖 Python 与 uv 镜像,方便 Docker Hub 不稳定时切换镜像源
|
||||
|
||||
### 🐛 Fixes
|
||||
- Earth 海缆登陆点改为基于相机射线与地球遮挡判断可见性,修复旋转后 pin 可见性滞后一帧的问题
|
||||
|
||||
### 🔧 Improvements
|
||||
- `docker-compose*.yml` 为 AI Provider 构建传入 `PYTHON_IMAGE` / `UV_IMAGE` 参数,默认仍使用官方镜像
|
||||
- 后端启动失败遇到 `Address already in use` 时输出 `lsof`、`ss` 与 PID 命令行信息
|
||||
- verbose 模式下 AI Provider build、后端与前端启动日志会在 spinner 下方保留最新 5 行滚动展示
|
||||
|
||||
---
|
||||
|
||||
## [0.41.1] — 2026-04-27
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复新闻直播面板设置项持久化失效:`closeTransientMobileOverlays` 通过旁路路径隐藏面板导致下次 persist 快照到错误状态,改为不重新从 DOM 读取面板可见性
|
||||
- 修复登陆点 pin 在地球侧面被半截遮挡:改为在接近地平线前(dot < 0.05)主动隐藏,避免深度测试切片
|
||||
|
||||
### 🔧 Improvements
|
||||
- 将所有画布绘制的图标抽取为 SVG,存入 `frontend/public/earth/assets/icons/`,新增图标规范到 `rules.md`
|
||||
|
||||
---
|
||||
|
||||
## [0.41.0] — 2026-04-27
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 图层系统完成地表到天空的注册顺序与关注优先的面板顺序拆分,支持基座海陆色块、国界、高清材质、云图、地形、算力、BGP、卫星、轨迹与海缆的稳定层级
|
||||
- 国界层新增真实行政区轮廓交互与中国/台湾联动高亮,修复高清材质、地形、footprint、卫星与经纬线之间的遮挡和 hover 竞争
|
||||
|
||||
### 🔧 Improvements
|
||||
- 新增无轮廓基座地图,所有图层关闭时仍保留 `#010609` 海洋与 `#080f1b` 陆地色块
|
||||
- 将大气云图抽象为独立图层并接入桌面/移动端图层开关、持久化状态与启动同步
|
||||
- 高清材质改为独立纹理覆盖层,地形显示在高清材质上方,并在高清材质关闭/恢复时保持原地形开关意图
|
||||
- 补充 Earth 渲染层级与图层样式文档,记录正式图层名、变量名、材质颜色、线宽与 renderOrder
|
||||
|
||||
---
|
||||
|
||||
## [0.40.5] — 2026-04-26
|
||||
|
||||
### 🔧 Improvements
|
||||
- 卫星拖尾改用 Instanced screen-space ribbon,单 draw call 渲染所有轨迹段,支持像素级宽度控制
|
||||
- Iridium 地面覆盖重写为球面投影径向网格,修复填充光晕不可见问题;新增外圈 LineLoop
|
||||
- 搜索面板打开时改用双 rAF 延迟聚焦输入框,确保 CSS 过渡完成后焦点可靠触发
|
||||
- 代码清理:提取 `IRIDIUM_OVERLAY_COLOR`、`IRIDIUM_REFERENCE_ALTITUDE_KM` 常量,消除重复三角函数调用
|
||||
|
||||
---
|
||||
|
||||
## [0.39.0] — 2026-04-24
|
||||
|
||||
## [0.40.4] — 2026-04-26
|
||||
|
||||
### 🔧 Improvements
|
||||
- 新增页面可见性恢复处理,页面从后台切回前台时主动刷新卫星位置,避免累积后台时间在下一帧一次性回放
|
||||
- 抽出卫星轨迹状态与轨迹几何清理 helper,统一后台恢复与清空数据时的轨迹重置路径
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复页面在后台停留较久后恢复前台时,卫星轨迹因超大 `deltaTime` 突然跳变、拖尾异常拉长的问题
|
||||
- 修复后台恢复后首帧仍沿用旧轨迹缓存,导致轨迹与当前卫星位置短时错位的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.40.3] — 2026-04-25
|
||||
|
||||
### 🔧 Improvements
|
||||
- 卫星点云升级为自定义 ShaderMaterial,支持 per-point alpha 控制,锁定/悬停卫星从点云中精确隐藏
|
||||
- 修复锁定环与自发光选中标记的 depthTest 错误(false → true),消除远端渲染穿透 artifact
|
||||
- 新增锁定环悬停态缩放与线宽(LOCKED_RING_HOVER_SCALE / LOCKED_RING_HOVER_LINE_WIDTH)
|
||||
- 修复 updateLockedDotWorldTransform / updateLockedHaloWorldTransform 未强制刷新 matrixWorld 导致的位置漂移
|
||||
|
||||
---
|
||||
|
||||
## [0.40.2] — 2026-04-24
|
||||
|
||||
### 🔧 Improvements
|
||||
- 卫星点大小随镜头缩放动态调整,拉近变大、拉远变小,响应与相机距离线性对应
|
||||
- 调小卫星点默认基础尺寸(dotSize 2.8),缩放范围更合理
|
||||
|
||||
---
|
||||
|
||||
## [0.40.1] — 2026-04-24
|
||||
|
||||
### 🔧 Improvements
|
||||
- 卫星选中标记(lockedring / lockeddot / 光晕)颜色统一跟随图例轨道倾角分类配色
|
||||
- 修复 Starlink footprint 在特定视角下遮蔽卫星点的渲染顺序问题(Group renderOrder 影响子 Mesh 排序)
|
||||
- footprint 材质改为 `depthTest: false` + 相机朝向 limbFade,替代 polygonOffset 深度竞争方案
|
||||
- 修复选中海缆时误触发附近卫星高亮(该行为属于 BGP 事件点逻辑,不应用于海缆)
|
||||
|
||||
---
|
||||
|
||||
## [0.40.0] — 2026-04-24
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 卫星 footprint 正式按星座能力分层:Starlink 保留专用地表覆盖,Iridium 改为独立外圈覆盖表达,其它非 Starlink 星座不再误用同一套 footprint
|
||||
- Earth 卫星详情卡补齐覆盖能力与当前显示说明,用户现在可以直接看见每颗卫星为什么显示 footprint、为何回退为自身发光
|
||||
|
||||
### 🔧 Improvements
|
||||
- 后端可视化接口新增并透传 `constellation_group` 与 `footprint_policy`,前端据此执行 capability-gated footprint renderer
|
||||
- 新增 Iridium 独立 coverage ring adapter,并继续保留 Starlink 专用 footprint 调校与昼夜可读性增强
|
||||
- 新增 Earth 卫星 footprint 策略技术文档,明确 GNSS、generic LEO、GEO 与 Iridium 的显示边界
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复前后端对 Iridium footprint policy 命名不一致,导致策略分发语义含混的问题
|
||||
- 清理 Starlink footprint 渲染中的未使用常量与过时命名,减少后续继续调校时的歧义
|
||||
|
||||
---
|
||||
|
||||
## [0.39.0] — 2026-04-24
|
||||
|
||||
### ✨ Highlights
|
||||
- 后端正式落下统一结构化日志地基:请求上下文、事件名、脱敏与持久化链路开始收口为可扩展的企业级日志体系
|
||||
- 系统日志页重构为真正的日志工作台:顶部筛选更紧凑,终端日志区成为主视觉,移动端 Earth 新闻/态势细节交互继续补稳
|
||||
|
||||
### 🔧 Improvements
|
||||
- 新增 `backend/app/core/logging.py`,统一 `request_id`、`service`、`event` 注入与敏感字段脱敏,并接入后端主入口、调度器、缓存、数据库和可视化链路
|
||||
- 系统日志页筛选区重排为更紧凑的两层结构,信息摘要并入终端工具栏 tooltip,日志终端区留出更稳定的按钮避让空间
|
||||
- Earth 移动端态势抽屉补齐宽度约束与图例换行规则,新闻详情抽屉在巡航切换时可同步更新标题和摘要
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复 `/tmp/planet_backend.log` 中混入空字节时,日志摘要条行数与实际可见日志不一致的问题
|
||||
- 修复移动端“态势”tab 在内容渲染后被图例文本撑宽、超出一屏的问题
|
||||
- 修复移动端新闻详情抽屉在巡航切换下一条新闻时标题更新但 summary 不同步的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.38.0] — 2026-04-23
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 新闻正式接入通用巡航层:新闻和 BGP 统一进入可配置巡航模块,桌面端与移动端都能在巡航聚焦时展示对应新闻卡片
|
||||
- 系统日志页升级为结构化过滤链路:按真实时间戳、结构化级别和字符串检索统一筛选,不再依赖前端或后端从日志文本里猜结果
|
||||
|
||||
### 🔧 Improvements
|
||||
- 新闻巡航补齐业务适配层:按发生地与时间生成巡航目标,桌面端与移动端统一标题 + summary 卡片风格,并增加连线与打字机摘要展示
|
||||
- 日志页筛选体验重排,统一服务源、级别、行数、时间和检索布局,日历标记改为由后端返回的结构化每日聚合结果驱动
|
||||
- 后端补充 `system_logs` 结构化解析与多级别精确过滤能力,Earth 浏览器端日志缓冲与系统日志 API 现在走同一套筛选语义
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复新闻巡航模块开启后难以关闭、桌面/移动端设置状态互相污染的问题
|
||||
- 修复新闻巡航卡片缺少摘要、移动端详情样式不统一、新闻巡航缺少连线的问题
|
||||
- 修复日志级别筛选会被访问日志 query string 中的 `level=error` 等参数污染,从而把 `INFO` 行误判为 `ERROR` 的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.37.2] — 2026-04-23
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 图层系统新增经纬线开关,桌面图层面板与移动端抽屉都可直接控制
|
||||
|
||||
### 🔧 Improvements
|
||||
- 经纬线正式接入 Earth layer registry,复用现有图层切换、移动端图层卡片与设置持久化流
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复经纬线只能默认常驻、无法作为独立图层开关控制的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.37.1] — 2026-04-23
|
||||
|
||||
### ✨ Highlights
|
||||
- `planet.sh` 后端重启链路修复 `uvicorn --reload` 残留 worker 场景,`restart` 现在能真正替换旧实例
|
||||
|
||||
### 🔧 Improvements
|
||||
- 收口后端清理逻辑,统一按 `uvicorn` 进程、端口占用进程和进程组执行清理,减少 reload 场景漏杀分支
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复部分机器执行 `./planet.sh restart --allow-lan` 后后端仍停留旧实例,导致 `/api/v1/visualization/geo/compute-centers` 返回 `404` 的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.35.1] — 2026-04-22
|
||||
## [0.37.0] — 2026-04-23
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 连线系统正式从巡航里解耦成通用 callout connector:桌面端和移动端统一支持对象级锚点、四边切换与临界区边缘滑动
|
||||
- BGP 巡航展示继续收口为稳定的“先定位卡片、再连真实锚点、再展示卡片”链路,移动端 popup 与桌面 info panel 的路线规则统一
|
||||
|
||||
### 🔧 Improvements
|
||||
- connector 配置从 `CRUISE_CONFIG` 拆到独立 `CONNECTOR_CONFIG`,默认类名、动画名和实例命名也全部去 cruise 语义
|
||||
- 移动端 popup 增加更稳定的 dock/obstacle 处理,拖动卡片时连线起终点会持续按几何关系自适应刷新
|
||||
- Earth 多个图层与控制逻辑继续收口,补充算力中心/BGP 风格对齐、layer panel 与相关交互细节调整
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复巡航模式下终点只像“视觉锚点”而不是真实绑定对象的问题,卡片拖动后终点现在会跟随
|
||||
- 修复移动端与桌面端多类连线路线异常:压线、反向、临界区折返、起点遮挡事件点等问题
|
||||
- 修复对象矩形临界区内连线仍强制中点到中点导致路线像“先钻进 source 内部”再出去的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.35.1] — 2026-04-22
|
||||
## [0.36.0] — 2026-04-22
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 新增统一“算力中心”图层:接入超算与 GPU 集群,支持搜索、统计、图例、详情卡与独立图层开关
|
||||
- 算力中心支持精确位置与估算位置两种状态,估算点会以问号角标区分,避免数据不全时整批节点在地图上消失
|
||||
|
||||
### 🔧 Improvements
|
||||
- Earth 详情卡拖拽与地球拖拽交互继续收口,减少拖动卡片和旋转地球时的选中文本与 pointer 竞争
|
||||
- `planet.sh` 改为通过独立脚本计算 AI Provider 依赖指纹,降低与根仓库依赖版本文件的无关耦合
|
||||
- README 补充 WSL / Windows 局域网访问排查与转发配置说明,便于开发环境联调
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复 Earth 算力中心图层在无原始坐标时无法显示的问题,支持站点提示和国家级估算回退
|
||||
- 修复信息卡拖拽事件可能被卡片级 stopPropagation 吞掉,导致拖拽流中断的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.35.1] — 2026-04-22
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 统计展示改为统一 `data-earth-stat` 绑定机制,桌面 HUD 和移动端抽屉复用同一套状态更新入口
|
||||
|
||||
### 🔧 Improvements
|
||||
- 收口海缆、登陆点、卫星、BGP 事件与 BGP 状态的统计写入逻辑,减少后续继续补桌面/移动双写分支的成本
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复移动端态势抽屉中的海缆、登陆点与 BGP 统计在图层切换后可能停留旧值的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.35.0] — 2026-04-22
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 移动端底部抽屉系统全面上线:响应式布局自动切换、Tab 导航、手势上拉/下滑开合、惯性速度判定
|
||||
- 移动端点击可交互物件(海缆、登陆点、卫星、BGP)后弹出智能定位悬浮卡片,可拖动,点击跳转详情
|
||||
|
||||
### 🔧 Improvements
|
||||
- 抽屉把手区域缩小至 36px(collapsed 时仅露出把手,不遮挡地球操作区)
|
||||
- 抽屉定期弹跳动画提示用户可上拉,5 秒间隔,打开后自动停止
|
||||
- 通知胶囊位置调整,不再覆盖品牌 logo
|
||||
- 移动端单指旋转、双指捏合缩放地球,触控事件冲突修复(pointer-events 级联)
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复移动端抽屉 shell 因 layout 高度(240px+)遮挡地球触控区域,pointer-events 改为按层级精确控制
|
||||
- 修复悬浮卡片因 setPointerCapture 在 iOS Safari 抑制合成 click 事件导致无法点击的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.34.0] — 2026-04-22
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 搜索面板正式接入,支持搜索海缆、登陆点、卫星、BGP 事件与观测站,并可直接聚焦到对应对象
|
||||
- `planet.sh --allow-lan` 打通 Bun + Vite 的局域网开放链路,启动成功后自动打印推荐访问地址与后端健康检查地址
|
||||
|
||||
### 🔧 Improvements
|
||||
- 前端开发启动链统一改成 Bun 直接执行 Vite 入口,不再依赖 shell 中额外暴露的 Node 路径
|
||||
- Earth 搜索结果接入登陆点详情卡片与对象聚焦,搜索后可直接进入对应详情流
|
||||
- `planet.sh` 补充局域网 IPv4 自动识别与推荐地址输出,减少 WSL 局域网调试成本
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复 `./planet.sh restart --allow-lan` 全量重启时未把 `--allow-lan` 继续传给 `start()`,导致前端退回本机监听的问题
|
||||
- 修复 WSL + Bun 环境下前端偶发因 Vite 启动链不稳定而无法正确监听 `0.0.0.0:3000` 的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.33.0] — 2026-04-22
|
||||
|
||||
### ✨ Highlights
|
||||
- `news_live_streams` 采集器默认接入 `iptv-org` 频道目录,并将采集结果稳定并入 Earth TV 直播源列表
|
||||
- 数据源页支持直接编辑内置数据源 override,并为内置源提供一键恢复默认配置入口
|
||||
|
||||
### 🔧 Improvements
|
||||
- `News Live Streams` 现在作为可直接触发的内置默认数据源提供,无需先手工补 override 才能采集
|
||||
- TV 播放源菜单会直接区分 `[内置]` 和 `[采集]` 来源,频道来源信息也会同步展示
|
||||
- 新增 [earth-news-source-configuration-and-collector-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-news-source-configuration-and-collector-plan.md),正式规划 Earth 态势新闻源配置化与后续采集器化路线
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复 `news_live_streams` 采集完成后 `/api/v1/tv/streams` 因读取不存在的 `updated_at` 字段而导致默认频道全部消失的问题
|
||||
- 修复内置数据源操作列按钮显示不全,以及编辑抽屉中多个 `Collapse` 紧贴的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.32.0] — 2026-04-22
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 设置新增“地球默认大小”持久化项,重置视角、缩放百分比重置和 BGP 巡航视图现在统一复用这一份默认 zoom
|
||||
- 卫星焦点层次继续收口:巡航进入 presentation 前不再过早 dim,非焦点卫星改成“降亮度/尾迹/背板”而不是去饱和度
|
||||
|
||||
### 🔧 Improvements
|
||||
- Earth 设置面板区块和左右留白进一步收紧,整体更贴近 HUD 面板的密度
|
||||
- toolbar 展开边界缓存改为按需刷新,减少 document 级 mousemove 期间的重复布局读取
|
||||
- Scrollbar 和 ScrollbarOverlay 收窄 observer 范围,减少大表格和动态菜单下的额外刷新成本
|
||||
- 更新 [earth-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/earth-frontend-context.md),补充默认视图大小已进入 Earth 设置持久化真源
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复开启巡航后,尚未进入连线/presentation 时卫星已经整体变暗的问题
|
||||
- 修复默认大小重置链路分散在多个入口、实际 reset/cruise/缩放提示不一致的问题
|
||||
- 修复开启地形后卫星反馈层与地球背面可见性之间的一组表现问题,保留正面反馈同时恢复背面轨道遮挡
|
||||
|
||||
---
|
||||
|
||||
## [0.31.3] — 2026-04-22
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 图层注册表和启动任务框架继续收口,启动顺序、启动模式、启动提示和任务注册现在都能从统一入口扩展
|
||||
- 修复 Earth 普通旋转模式与巡航模式切换时的一组交互回归,同时让卫星/地形/昼夜模式的表现更稳定
|
||||
|
||||
### 🔧 Improvements
|
||||
- 新增 [layer-startup-tasks.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/layer-startup-tasks.js) 启动任务注册表,支持 `registerLayerStartupTask(id, taskFactory)`,并拆成海缆 / 卫星 / BGP 独立注册函数
|
||||
- Earth 图层控制改成注册表驱动,统一承载 `startupPriority`、`startupMode`、`startupLabel`、`startupMessage` 与图层持久化元信息
|
||||
- Earth 设置支持持久化图层开关、旋转模式、HUD 面板显示状态、地形透明度与日夜模式,并提供一键重置
|
||||
- 更新 [earth-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/earth-frontend-context.md) 记录图层注册表、启动任务、设置持久化与巡航适配边界
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复普通旋转模式下点击海缆 / 卫星 / BGP 后卡片和选中表现会被异常清空的问题
|
||||
- 修复巡航模式切回旋转再切回巡航后无法继续自动巡航的问题
|
||||
- 修复开启地形后卫星选中反馈层被高海拔区域吞掉的问题,并恢复轨道只在地球前半侧可见
|
||||
- 修复关闭日夜模式后地球照明仍沿真实昼夜切换、亮部过曝和偏色的问题,改成更中性的 inspection lighting
|
||||
- 修复 toolbar 收起态仍挡住地球交互,以及首帧短暂展开闪现的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.31.2] — 2026-04-21
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 巡航模式重构为“通用巡航队列 + 通用连线动画 + BGP 业务适配”三层结构,后续扩到海缆、卫星或新闻巡航时不必再复制一套 `main.js` 状态机
|
||||
- 修复巡航重构后的交互回归:空白点击重新稳定切到下一项,连线按“起点 → 引导线 → 终点”顺序入场
|
||||
|
||||
### 🔧 Improvements
|
||||
- 新增 [cruise-sequencer.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/cruise-sequencer.js) 统一管理队列推进、停留时长、打断与恢复
|
||||
- 新增 [callout-connector.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/callout-connector.js) 统一管理 SVG 连线、折线路径与描边动画
|
||||
- 新增 [bgp-cruise-adapter.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp-cruise-adapter.js) 收口 BGP 巡航目标排序、卡片落点、轮询去重与连线适配
|
||||
- 更新 [earth-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/earth-frontend-context.md) 说明新的巡航分层与复用边界
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复巡航模式下点击空白处无法稳定跳转到下一项、切回旋转再切回巡航后直接卡住的问题
|
||||
- 修复巡航连线被实时重定位覆盖导致“直接出现”而非绘制动画的问题
|
||||
- 修复连线动画节点入场节奏不对的问题,改为先出现起点,再绘制连线,最后出现终点
|
||||
|
||||
---
|
||||
|
||||
## [0.31.1] — 2026-04-21
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 图层开关状态统一成可复用的 `active / loading` 状态机,首次启用地形和卫星时不再像按钮失效
|
||||
- 文档目录重构为 `docs/technical`、`docs/plans`、`docs/deprecated`,并吸收 `.sisyphus/plans` 中有价值的 Earth / 卫星 / UE5 草案
|
||||
|
||||
### 🔧 Improvements
|
||||
- 新增 [layer-button-state.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/layer-button-state.js),统一按钮 tooltip、`aria-busy`、禁用态和状态文本同步
|
||||
- 地形图层支持 hover/focus 预热与空闲预热,首次点击等待前移,加载中状态持续可见
|
||||
- 卫星图层启用前会立即切换为 `loading` 中间态,请求完成后再切回正常开关表现
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复地形首次加载时通知过早消失、开关仍像关闭状态导致用户误判按钮损坏的问题
|
||||
- 修复卫星接口较慢时按钮没有任何中间态反馈的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.31.0] — 2026-04-21
|
||||
|
||||
### ✨ Features
|
||||
- Earth 新增"巡航展示"模式:自动轮播 BGP 异常事件,逐帧追踪连接线位置,支持外部交互立即中断序列(cancel notifier 模式)
|
||||
- 巡航目标事件点高亮显示:hover 外观 + 锁定脉冲动画,并与点击行为统一展示周边受影响卫星与海缆
|
||||
- BGP 事件图标新增填充 W 形波动符号(flap 类型),替换原有难以辨认的贝塞尔细线
|
||||
- 巡航/点击激活时其余卫星自动降饱和度 + 增加透明度以突出焦点;海缆未受影响时同步变暗
|
||||
|
||||
### 🔧 Improvements
|
||||
- 修复巡航轮播期间 BGP 事件 polling 刷新导致标记闪烁消失的问题(clearBGPData 延迟到请求完成后执行)
|
||||
- 点击与巡航锁定颜色统一为 hover 色(0.92, 0.98, 1.0 全透明),移除锁定态脉冲动画
|
||||
- 巡航连接折线转折点从尖角调整为钝角(linkElbowDropPx),提升连线可读性
|
||||
|
||||
---
|
||||
|
||||
## [0.30.0] — 2026-04-21
|
||||
|
||||
### ✨ Features
|
||||
- Earth 新增真实地形图层:后端代理 Terrarium DEM 瓦片(`/api/v1/visualization/terrain/terrarium/{z}/{x}/{y}.png`),前端新增 `terrain.js` 负责瓦片拉取、顶点位移与按海拔着色
|
||||
- 设置弹窗新增"地形"分组,支持通过滑块实时调整地形图层透明度
|
||||
|
||||
### 🔧 Improvements
|
||||
- 地形按钮改为异步加载,首次点击显示进度提示并在失败时自动回退
|
||||
- 启动阶段改用 `applyImmediateView` 直接应用初始视角,`showStatusMessage` / `queueStatusMessage` 区分即时与队列态状态消息,加载中不再被临时状态打断
|
||||
- 控制面板抽取 `applyTerrainUiState` / `getViewRotation` 收敛地形切换与视角旋转的重复 UI 同步逻辑
|
||||
|
||||
---
|
||||
|
||||
## [0.29.2] — 2026-04-21
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 继续收口 HUD 交互与设置面板表现,设置弹窗改成更接近从按钮展开的窗口感,同时加入系统级 admin 入口
|
||||
- 修正天球太阳方向与地球受光解耦后的日照逻辑,地表昼夜判断改为按太阳直射点经纬度落到地球贴图坐标
|
||||
|
||||
### 🔧 Improvements
|
||||
- toolbar 进一步收成更贴近 hub 的浅弓形排列,并统一成与 HUD panel 一致的液态玻璃配色与透明度
|
||||
- 设置弹窗与各 HUD panel 继续统一样式、等比缩放和头部基线,设置列表补充系统分组与 admin 跳转
|
||||
- 所有 HUD panel 增加更统一的液态玻璃高光与 hover / press 反馈
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复设置弹窗仍像旧圆角矩形、标题文案重复和从底边直直飞出的动画问题
|
||||
- 修复天球与太阳方向混用显示校准导致中国白天仍落在夜面的日照错误
|
||||
|
||||
---
|
||||
|
||||
## [0.29.1] — 2026-04-20
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 加载状态条改成单一队列式通知面板,加载阶段不再因为步骤文案变化而回缩,也不会被其他通知打断
|
||||
- 调整 brand panel 的呈现方式与昼夜/选中态可读性,让品牌区更自然、交互高亮在白天和黑夜里都更稳定
|
||||
|
||||
### 🔧 Improvements
|
||||
- 移除旧的地球加载浮层结构,统一由 HUD 状态消息承载三点脉冲加载过程
|
||||
- brand panel 改为无边框品牌层,仅保留轻微氛围光,不再因为非常规尺寸显得像第五块功能面板
|
||||
- 温和收敛地球昼夜材质与主背光强度,保留昼夜辨识度的同时提升白天地表纹理和夜面交互可见性
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复加载地球时通知条在步骤切换中反复缩短、其他状态消息抢占加载流程的问题
|
||||
- 修复海缆、登陆点和 BGP 选中高亮在黑夜中过暗、在高光中过亮导致难以辨识的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.29.0] — 2026-04-20
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 新增天球层第一版:引入真实全天星图、亮星层与太阳/月亮位置计算,地球场景首次具备可校准的天文背景
|
||||
- 地球昼夜分隔升级为更明显的日夜增强效果,夜面、晨昏带和太阳方向联动更容易直接读出来
|
||||
|
||||
### 🔧 Improvements
|
||||
- 新增 `celestial.js` 模块和 `assets/celestial/` 资源目录,统一管理星图、亮星数据以及太阳/月亮与光照同步
|
||||
- 卫星图例改为按倾角分组,严格固定为“赤道轨道 → 低倾角轨道 → 中倾角轨道 → 高倾角轨道 → 逆行轨道”顺序,并全部中文化
|
||||
- 图层面板补齐关闭按钮,拖拽脱离左列后不再被流布局 margin 影响,能够真正贴到品牌面板下沿
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复天球球壳放大后被相机 far plane 裁剪导致的外层黑环问题
|
||||
- 修复图层面板在左侧上移时始终与 brand panel 保持额外间距的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.28.2] — 2026-04-20
|
||||
|
||||
### ✨ Highlights
|
||||
- 修正媒体情报面板在 `电视直播 / 态势聚合` tab 间切换时的尺寸记忆逻辑,切回原 tab 后可恢复各自大小状态
|
||||
- 清理 `docs/` 根目录遗留的旧路径文档,只保留新的分组目录和归档目录,结束同一文档双路径并存状态
|
||||
|
||||
### 🔧 Improvements
|
||||
- `media-panel` 切换逻辑改成按 tab 分别记忆尺寸状态,避免 `A -> B -> A` 时继续共用同一套外层尺寸
|
||||
- 目录整理真正完成收尾:旧的 `docs/*.md` 平铺计划文档删除,继续以 `docs/agents / earth / backend / frontend / ops / ue5 / deprecated` 为唯一入口
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复拉伸 `media-panel` 后切换 tab 时,`news-panel` 高度回退到旧默认值的问题
|
||||
- 修复拉伸后切换 tab 导致面板视觉锚点异常的问题,切换时改为围绕当前卡片自身右下角进行尺寸恢复
|
||||
|
||||
---
|
||||
|
||||
## [0.28.1] — 2026-04-20
|
||||
|
||||
### ✨ Highlights
|
||||
@@ -213,7 +658,7 @@ Released: 2026-04-12
|
||||
|
||||
- Added [backend/app/api/v1/tv.py](/home/ray/dev/linkong/planet/backend/app/api/v1/tv.py), [backend/app/services/tv_streams.py](/home/ray/dev/linkong/planet/backend/app/services/tv_streams.py), and [backend/app/services/collectors/news_live_streams.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/news_live_streams.py) to provide TV source configuration, public stream payloads, a guarded HLS proxy path, and a collector entry point for future world-news live-source ingestion.
|
||||
- Added the Earth TV HUD workspace through [frontend/public/earth/index.html](/home/ray/dev/linkong/planet/frontend/public/earth/index.html), [frontend/public/earth/js/tv.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/tv.js), and [frontend/public/earth/css/tv-panel.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/tv-panel.css), including toolbar access, draggable/closable behavior, resize support, direct video/HLS playback, iframe fallback, and per-channel external-open handling.
|
||||
- Added [docs/deprecated/earth-tv-live-module-plan.md](/home/ray/dev/linkong/planet/docs/deprecated/earth-tv-live-module-plan.md) and [docs/earth/news-live-streams-collector-format.md](/home/ray/dev/linkong/planet/docs/earth/news-live-streams-collector-format.md) to document the TV module rollout plan and the expected collector payload format for future curated live-channel ingestion.
|
||||
- Added [docs/deprecated/earth-tv-live-module-plan.md](/home/ray/dev/linkong/planet/docs/deprecated/earth-tv-live-module-plan.md) and [docs/earth/technical/news-live-streams-collector-format.md](/home/ray/dev/linkong/planet/docs/technical/earth-news-live-streams-collector-format.md) to document the TV module rollout plan and the expected collector payload format for future curated live-channel ingestion.
|
||||
|
||||
### Improved
|
||||
|
||||
@@ -299,7 +744,7 @@ Released: 2026-04-10
|
||||
|
||||
- Improved [frontend/src/pages/Playground/Playground.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Playground/Playground.tsx) and [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css) by rebuilding Playground into a true chatbox workflow with persistent history, edit-and-resend behavior, grounded message actions, responsive composer behavior, bottom-stick scrolling, and tighter mobile layout handling.
|
||||
- Improved [frontend/src/components/AppLayout/AppLayout.tsx](/home/ray/dev/linkong/planet/frontend/src/components/AppLayout/AppLayout.tsx), [frontend/src/App.tsx](/home/ray/dev/linkong/planet/frontend/src/App.tsx), and [frontend/src/pages/Alerts/Alerts.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Alerts/Alerts.tsx) by reorganizing navigation around `采集与数据`, `专题观测`, and split alert entries so the app can scale to more observability and situational modules without turning the top-level UI into a single overloaded page.
|
||||
- Improved [README.md](/home/ray/dev/linkong/planet/README.md) and [docs/agents/situational-awareness-foundation-plan.md](/home/ray/dev/linkong/planet/docs/agents/situational-awareness-foundation-plan.md) by documenting the current AI/alerts base, planned situational-awareness direction, and the new persistent Playground foundation.
|
||||
- Improved [README.md](/home/ray/dev/linkong/planet/README.md) and [docs/agents/situational-awareness-foundation-plan.md](/home/ray/dev/linkong/planet/docs/plans/agents-situational-awareness-foundation-plan.md) by documenting the current AI/alerts base, planned situational-awareness direction, and the new persistent Playground foundation.
|
||||
|
||||
### Fixed
|
||||
|
||||
@@ -337,7 +782,7 @@ Released: 2026-04-10
|
||||
### Improved
|
||||
|
||||
- Improved [rules.md](/home/ray/dev/linkong/planet/rules.md) by adding mandatory release-workflow requirements and a new frontend layout constraint section covering single-screen workspaces, overflow ownership, tab-pane behavior, compact-mode expectations, and readable-card fallbacks.
|
||||
- Improved [frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/frontend/frontend-layout-guidelines.md) by summarizing the recurring Earth, Playground, BGP, and admin-layout regressions into concrete constraints for future frontend work, including “prefer scrollbars over unreadable compression” and “do not treat every tab as a table pane.”
|
||||
- Improved [frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md) by summarizing the recurring Earth, Playground, BGP, and admin-layout regressions into concrete constraints for future frontend work, including “prefer scrollbars over unreadable compression” and “do not treat every tab as a table pane.”
|
||||
|
||||
## 0.24.6
|
||||
|
||||
@@ -354,7 +799,7 @@ Released: 2026-04-10
|
||||
- Improved [backend/app/services/bgp_incidents.py](/home/ray/dev/linkong/planet/backend/app/services/bgp_incidents.py) and [backend/app/services/bgp_enrichment.py](/home/ray/dev/linkong/planet/backend/app/services/bgp_enrichment.py) by avoiding historical full-table infrastructure scans, narrowing observation baseline payloads to required columns, and pushing more ASN filtering into the database.
|
||||
- Improved [backend/app/api/v1/alerts.py](/home/ray/dev/linkong/planet/backend/app/api/v1/alerts.py), [backend/app/api/v1/dashboard.py](/home/ray/dev/linkong/planet/backend/app/api/v1/dashboard.py), and [backend/app/api/v1/settings.py](/home/ray/dev/linkong/planet/backend/app/api/v1/settings.py) by collapsing several repeated count and settings queries into fewer aggregate or batched reads.
|
||||
- Improved [frontend/src/pages/BGP/BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx), [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css), and [frontend/src/components/MarkdownRenderer/MarkdownRenderer.tsx](/home/ray/dev/linkong/planet/frontend/src/components/MarkdownRenderer/MarkdownRenderer.tsx) by rebuilding the `AI 简报` tab layout, fixing saved brief scrolling behavior, and extending the renderer to handle tables, separators, and stored metadata comments more gracefully.
|
||||
- Improved [docs/frontend/ai-playground-development-plan.md](/home/ray/dev/linkong/planet/docs/frontend/ai-playground-development-plan.md) by explicitly recording that the current BGP brief is only the first-stage summary flow and that regional prefix-geography analysis remains a planned Phase B follow-up.
|
||||
- Improved [docs/frontend/plans/ai-playground-development-plan.md](/home/ray/dev/linkong/planet/docs/plans/frontend-ai-playground-development-plan.md) by explicitly recording that the current BGP brief is only the first-stage summary flow and that regional prefix-geography analysis remains a planned Phase B follow-up.
|
||||
|
||||
### Fixed
|
||||
|
||||
@@ -460,8 +905,8 @@ Released: 2026-04-09
|
||||
### Added
|
||||
|
||||
- Added [frontend/src/pages/Playground/Playground.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Playground/Playground.tsx), introducing the first dedicated AI testing workspace with provider status visibility, prompt/result tabs, and collapsible operator guidance.
|
||||
- Added [docs/frontend/frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/frontend/frontend-layout-guidelines.md), documenting the repository standard for one-screen admin workspaces and module-local overflow handling.
|
||||
- Added [docs/frontend/ai-playground-development-plan.md](/home/ray/dev/linkong/planet/docs/frontend/ai-playground-development-plan.md), capturing the completed AI gateway/UI work and the next delivery phases for BGP briefs, evidence-first inputs, and future agent runtime expansion.
|
||||
- Added [docs/frontend/technical/frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md), documenting the repository standard for one-screen admin workspaces and module-local overflow handling.
|
||||
- Added [docs/frontend/plans/ai-playground-development-plan.md](/home/ray/dev/linkong/planet/docs/plans/frontend-ai-playground-development-plan.md), capturing the completed AI gateway/UI work and the next delivery phases for BGP briefs, evidence-first inputs, and future agent runtime expansion.
|
||||
|
||||
### Improved
|
||||
|
||||
@@ -566,7 +1011,7 @@ Released: 2026-04-07
|
||||
- Added [backend/app/services/ai_client.py](/home/ray/dev/linkong/planet/backend/app/services/ai_client.py), introducing an internal HTTP client for `backend -> aiprovider` calls with request-id propagation and lightweight retry.
|
||||
- Added [aiprovider/main.py](/home/ray/dev/linkong/planet/aiprovider/main.py), [aiprovider/provider_service.py](/home/ray/dev/linkong/planet/aiprovider/provider_service.py), and related config/schema files to stand up the dedicated adapter service.
|
||||
- Added [aiprovider/.env.example](/home/ray/dev/linkong/planet/aiprovider/.env.example) and [docker-compose.local-model.yml](/home/ray/dev/linkong/planet/docker-compose.local-model.yml) as ready-to-edit local-model templates.
|
||||
- Added [docs/agents/aiprovider.md](/home/ray/dev/linkong/planet/docs/agents/aiprovider.md), documenting architecture, configuration, single-machine and multi-machine deployment, and cross-service calling patterns.
|
||||
- Added [docs/agents/aiprovider.md](/home/ray/dev/linkong/planet/docs/technical/agents-aiprovider.md), documenting architecture, configuration, single-machine and multi-machine deployment, and cross-service calling patterns.
|
||||
- Added a dedicated `重启 AI Provider` control path in [Dashboard.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Dashboard/Dashboard.tsx), [system_control.py](/home/ray/dev/linkong/planet/backend/app/services/system_control.py), and [system_restart_runner.py](/home/ray/dev/linkong/planet/backend/scripts/system_restart_runner.py).
|
||||
|
||||
### Improved
|
||||
@@ -692,7 +1137,7 @@ Released: 2026-04-02
|
||||
|
||||
- Added a new `IPtoASN Prefix Geography` collector in [iptoasn.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/iptoasn.py) and registered it through [data_sources.yaml](/home/ray/dev/linkong/planet/backend/app/core/data_sources.yaml), [data_sources.py](/home/ray/dev/linkong/planet/backend/app/core/data_sources.py), [datasource_defaults.py](/home/ray/dev/linkong/planet/backend/app/core/datasource_defaults.py), and [collectors/__init__.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/__init__.py).
|
||||
- Added country centroid helpers in [countries.py](/home/ray/dev/linkong/planet/backend/app/core/countries.py) so country-level prefix geography can produce map coordinates instead of only labels.
|
||||
- Added a dedicated prefix-geography implementation note in [prefix-geography-plan.md](/home/ray/dev/linkong/planet/docs/earth/prefix-geography-plan.md).
|
||||
- Added a dedicated prefix-geography implementation note in [prefix-geography-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-prefix-geography-plan.md).
|
||||
- Added recent `15m` collector activity dimensions to BGP coverage output in [bgp_collectors.py](/home/ray/dev/linkong/planet/backend/app/services/bgp_collectors.py) and [visualization.py](/home/ray/dev/linkong/planet/backend/app/api/v1/visualization.py).
|
||||
- Added additional BGP detector coverage for `route_leak_candidate` and `path_flap` flows in [test_bgp.py](/home/ray/dev/linkong/planet/backend/tests/test_bgp.py).
|
||||
- Added a local Earth cloud texture at [earth_clouds_1024.png](/home/ray/dev/linkong/planet/frontend/public/earth/assets/earth_clouds_1024.png) to avoid remote cloud-map dependency failures.
|
||||
@@ -707,7 +1152,7 @@ Released: 2026-04-02
|
||||
- Improved Earth event animation semantics in [bgp.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp.js) by separating icon pulse from ring expansion so the center marker can breathe while the ring expands independently.
|
||||
- Improved Earth texture reliability in [earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js) by switching clouds back to a local static asset under the restored `public/earth` runtime.
|
||||
- Improved frontend boot noise in [frontend/index.html](/home/ray/dev/linkong/planet/frontend/index.html) by removing the default Vite favicon request that was generating irrelevant `vite.svg` timeouts during Earth debugging.
|
||||
- Improved project planning docs in [bgp-context.md](/home/ray/dev/linkong/planet/docs/earth/bgp-context.md) and [TODO.md](/home/ray/dev/linkong/planet/TODO.md) so the roadmap now explicitly prioritizes `activity layer`, `prefix-centric geography`, and follow-up geofeed/whois work.
|
||||
- Improved project planning docs in [bgp-context.md](/home/ray/dev/linkong/planet/docs/technical/earth-bgp-context.md) and [TODO.md](/home/ray/dev/linkong/planet/TODO.md) so the roadmap now explicitly prioritizes `activity layer`, `prefix-centric geography`, and follow-up geofeed/whois work.
|
||||
|
||||
### Fixed
|
||||
|
||||
@@ -865,7 +1310,7 @@ Released: 2026-03-31
|
||||
- Added restart-task Redis helpers and whitelist command mapping in [system_control.py](/home/ray/dev/linkong/planet/backend/app/services/system_control.py).
|
||||
- Added detached restart runner orchestration in [system_restart_runner.py](/home/ray/dev/linkong/planet/backend/scripts/system_restart_runner.py).
|
||||
- Added `-d` / `--database` support to [planet.sh](/home/ray/dev/linkong/planet/planet.sh) for database-only restarts.
|
||||
- Added restart control documentation in [system-service-control.md](/home/ray/dev/linkong/planet/docs/backend/system-service-control.md).
|
||||
- Added restart control documentation in [system-service-control.md](/home/ray/dev/linkong/planet/docs/technical/backend-system-service-control.md).
|
||||
|
||||
### Improved
|
||||
|
||||
|
||||
@@ -1,647 +0,0 @@
|
||||
# Agent Architecture Plan
|
||||
|
||||
## Overview
|
||||
|
||||
This document defines the agent architecture for Planet.
|
||||
|
||||
The architecture is intentionally broader than datasource health checking.
|
||||
|
||||
It is designed to support both:
|
||||
|
||||
- datasource health governance
|
||||
- future situational-awareness workflows
|
||||
|
||||
The core idea is to avoid building a one-off "repair broken API links" agent.
|
||||
|
||||
Instead, Planet should grow a reusable agent runtime that can:
|
||||
|
||||
- collect evidence
|
||||
- evaluate signals
|
||||
- reason over incomplete information
|
||||
- generate proposals
|
||||
- produce assessments
|
||||
- execute limited actions under policy
|
||||
|
||||
|
||||
## Design Goal
|
||||
|
||||
Build an agent foundation that can evolve in this order:
|
||||
|
||||
1. datasource health checks
|
||||
2. datasource repair proposals
|
||||
3. signal correlation
|
||||
4. situational assessments
|
||||
5. controlled runtime actions
|
||||
|
||||
This means the architecture should treat datasource health as one use case of the larger agent system, not as the whole system.
|
||||
|
||||
|
||||
## Core Principles
|
||||
|
||||
1. Separate evidence from reasoning
|
||||
|
||||
- raw signals should be gathered first
|
||||
- deterministic checks should run before LLM reasoning
|
||||
|
||||
2. Agents do not own the defaults
|
||||
|
||||
- repository defaults remain human-owned
|
||||
- agents operate on runtime state, proposals, and overrides
|
||||
|
||||
3. Reasoning and action are different responsibilities
|
||||
|
||||
- many agents should be read-only or propose-only
|
||||
- only tightly controlled flows may apply changes
|
||||
|
||||
4. Shared runtime, specialized roles
|
||||
|
||||
- multiple agent roles should share the same object model and orchestration patterns
|
||||
- health and situational-awareness agents should not invent incompatible payloads
|
||||
|
||||
5. Auditability is mandatory
|
||||
|
||||
- every proposal, assessment, and applied action should be attributable
|
||||
|
||||
|
||||
## System Layers
|
||||
|
||||
Planet agent architecture should be split into four layers.
|
||||
|
||||
### 1. Signal Layer
|
||||
|
||||
Purpose:
|
||||
|
||||
- gather raw evidence from internal and external systems
|
||||
|
||||
Example sources:
|
||||
|
||||
- collector outputs
|
||||
- datasource health checks
|
||||
- logs
|
||||
- snapshots
|
||||
- alerts
|
||||
- web search results
|
||||
- scraped pages
|
||||
- external APIs
|
||||
- operator inputs
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- fetch
|
||||
- normalize
|
||||
- timestamp
|
||||
- tag with source and trust level
|
||||
|
||||
This layer should not make high-level judgments.
|
||||
|
||||
|
||||
### 2. Evaluation Layer
|
||||
|
||||
Purpose:
|
||||
|
||||
- perform deterministic analysis
|
||||
|
||||
Examples:
|
||||
|
||||
- reachability checks
|
||||
- schema validation
|
||||
- threshold checks
|
||||
- time-window comparisons
|
||||
- anomaly counters
|
||||
- completeness checks
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- classify signals into machine-readable findings
|
||||
- attach deterministic evidence
|
||||
|
||||
This layer should avoid LLM dependency whenever possible.
|
||||
|
||||
|
||||
### 3. Reasoning Layer
|
||||
|
||||
Purpose:
|
||||
|
||||
- use LLMs when semantic interpretation or incomplete-information reasoning is needed
|
||||
|
||||
Examples:
|
||||
|
||||
- endpoint migration inference
|
||||
- multi-source event correlation
|
||||
- causality hypotheses
|
||||
- ambiguity reduction
|
||||
- assessment narrative generation
|
||||
- action recommendation generation
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- synthesize evidence
|
||||
- produce hypotheses
|
||||
- rank confidence
|
||||
- explain reasoning boundaries
|
||||
|
||||
This is the main place where `aiprovider` and web search are used.
|
||||
|
||||
|
||||
### 4. Action Layer
|
||||
|
||||
Purpose:
|
||||
|
||||
- convert proposals or assessments into controlled system actions
|
||||
|
||||
Examples:
|
||||
|
||||
- create runtime override
|
||||
- create proposal
|
||||
- publish alert
|
||||
- update operator task queue
|
||||
- generate summary artifact
|
||||
- trigger follow-up verification
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- enforce policy
|
||||
- enforce approval requirements
|
||||
- verify post-action outcomes
|
||||
- record audit trails
|
||||
|
||||
|
||||
## Architecture Sketch
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A["Collectors / Logs / Snapshots / External APIs"] --> B["Signal Layer"]
|
||||
W["Web Search / Page Fetch / Docs"] --> B
|
||||
B --> C["Evaluation Layer"]
|
||||
C --> D["Findings"]
|
||||
D --> E["Reasoning Layer (LLM + Tools)"]
|
||||
E --> F["Proposals"]
|
||||
E --> G["Assessments"]
|
||||
F --> H["Action Layer"]
|
||||
H --> I["Runtime Overrides / Alerts / Tasks"]
|
||||
H --> J["Verification Loop"]
|
||||
J --> B
|
||||
|
||||
K["Policy Engine"] --> H
|
||||
L["Audit / History Store"] --> H
|
||||
L --> E
|
||||
L --> C
|
||||
```
|
||||
|
||||
|
||||
## Agent Roles
|
||||
|
||||
The first version should define these logical roles.
|
||||
|
||||
### 1. Health Agent
|
||||
|
||||
Primary use case:
|
||||
|
||||
- datasource health governance
|
||||
|
||||
Inputs:
|
||||
|
||||
- datasource metadata
|
||||
- current endpoint
|
||||
- latest health records
|
||||
- latest failures
|
||||
- deterministic findings
|
||||
|
||||
Outputs:
|
||||
|
||||
- health interpretation
|
||||
- repair proposal
|
||||
- confidence
|
||||
- evidence references
|
||||
|
||||
Typical action level:
|
||||
|
||||
- propose-only
|
||||
|
||||
|
||||
### 2. Correlation Agent
|
||||
|
||||
Primary use case:
|
||||
|
||||
- identify whether multiple signals describe the same event or related events
|
||||
|
||||
Inputs:
|
||||
|
||||
- findings from multiple collectors
|
||||
- time windows
|
||||
- region / ASN / prefix / cable relationships
|
||||
- prior incidents
|
||||
|
||||
Outputs:
|
||||
|
||||
- grouped event candidates
|
||||
- correlation rationale
|
||||
- confidence per relationship
|
||||
|
||||
Typical action level:
|
||||
|
||||
- read-only
|
||||
|
||||
|
||||
### 3. Assessment Agent
|
||||
|
||||
Primary use case:
|
||||
|
||||
- produce situational-awareness outputs
|
||||
|
||||
Inputs:
|
||||
|
||||
- grouped events
|
||||
- findings
|
||||
- current context
|
||||
- historical context
|
||||
- operator constraints
|
||||
|
||||
Outputs:
|
||||
|
||||
- structured assessment
|
||||
- risk summary
|
||||
- evidence-backed recommendations
|
||||
- missing-information list
|
||||
|
||||
Typical action level:
|
||||
|
||||
- read-only or propose-only
|
||||
|
||||
|
||||
### 4. Recovery Agent
|
||||
|
||||
Primary use case:
|
||||
|
||||
- carry low-risk proposals into controlled runtime actions
|
||||
|
||||
Inputs:
|
||||
|
||||
- approved proposal
|
||||
- policy constraints
|
||||
- trusted-domain rules
|
||||
- verification checks
|
||||
|
||||
Outputs:
|
||||
|
||||
- applied override
|
||||
- failed application
|
||||
- rollback request
|
||||
|
||||
Typical action level:
|
||||
|
||||
- apply-limited
|
||||
|
||||
|
||||
## Shared Object Model
|
||||
|
||||
All agents should work on a shared object model.
|
||||
|
||||
That prevents the health subsystem and situational-awareness subsystem from drifting into incompatible payloads.
|
||||
|
||||
### Signal
|
||||
|
||||
Represents a raw observed fact.
|
||||
|
||||
Examples:
|
||||
|
||||
- a datasource returned HTTP 404
|
||||
- a collector returned empty results
|
||||
- BGP updates spiked in one region
|
||||
- a known endpoint now redirects elsewhere
|
||||
|
||||
Suggested shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "sig_123",
|
||||
"type": "datasource.http_failure",
|
||||
"source": "ris_live_bgp",
|
||||
"occurred_at": "2026-04-08T10:00:00Z",
|
||||
"severity": "medium",
|
||||
"payload": {},
|
||||
"trust": 0.95
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
### Finding
|
||||
|
||||
Represents a deterministic or semi-deterministic interpretation of one or more signals.
|
||||
|
||||
Examples:
|
||||
|
||||
- `schema_changed`
|
||||
- `endpoint_unreachable`
|
||||
- `data_volume_abnormally_low`
|
||||
- `event_cluster_detected`
|
||||
|
||||
Suggested shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "find_123",
|
||||
"type": "datasource.schema_changed",
|
||||
"source_ids": ["sig_123"],
|
||||
"confidence": 0.92,
|
||||
"evidence": [],
|
||||
"details": {}
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
### Proposal
|
||||
|
||||
Represents a recommended action, not an already-applied action.
|
||||
|
||||
Examples:
|
||||
|
||||
- switch endpoint to new URL
|
||||
- disable bad override
|
||||
- escalate issue for manual review
|
||||
|
||||
Suggested shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "prop_123",
|
||||
"kind": "endpoint_override",
|
||||
"target": "telegeography_cables",
|
||||
"confidence": 0.84,
|
||||
"reason": "Official docs now point to a new API path",
|
||||
"payload": {},
|
||||
"evidence_urls": [],
|
||||
"status": "proposed"
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
### Assessment
|
||||
|
||||
Represents a structured situational-awareness output for operators or downstream systems.
|
||||
|
||||
Examples:
|
||||
|
||||
- current network posture summary
|
||||
- incident impact assessment
|
||||
- risk and response recommendations
|
||||
|
||||
Suggested shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "assess_123",
|
||||
"scope": "regional-network",
|
||||
"risk_level": "high",
|
||||
"summary": "Regional routing instability is increasing.",
|
||||
"key_risks": [],
|
||||
"evidence": [],
|
||||
"recommendations": [],
|
||||
"missing_data": []
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
## State Machine
|
||||
|
||||
The shared orchestration flow should look like this:
|
||||
|
||||
```mermaid
|
||||
stateDiagram-v2
|
||||
[*] --> Collect
|
||||
Collect --> Validate
|
||||
Validate --> Classify
|
||||
Classify --> Reason
|
||||
Reason --> Propose
|
||||
Reason --> Assess
|
||||
Propose --> Review
|
||||
Review --> Apply
|
||||
Apply --> Verify
|
||||
Verify --> Archive
|
||||
Assess --> Archive
|
||||
Archive --> [*]
|
||||
```
|
||||
|
||||
Definitions:
|
||||
|
||||
- `Collect`: gather signals
|
||||
- `Validate`: run deterministic checks
|
||||
- `Classify`: create findings
|
||||
- `Reason`: invoke LLM reasoning when needed
|
||||
- `Propose`: create change proposals
|
||||
- `Review`: policy or human approval
|
||||
- `Apply`: perform limited runtime action
|
||||
- `Verify`: confirm action effect
|
||||
- `Archive`: store artifacts and decisions
|
||||
|
||||
|
||||
## Permission Model
|
||||
|
||||
Each agent role should be assigned one of these action levels.
|
||||
|
||||
### `read-only`
|
||||
|
||||
Allowed:
|
||||
|
||||
- read signals
|
||||
- search web
|
||||
- fetch pages
|
||||
- read internal state
|
||||
- generate findings and assessments
|
||||
|
||||
Not allowed:
|
||||
|
||||
- mutate config
|
||||
- write overrides
|
||||
- change live runtime behavior
|
||||
|
||||
|
||||
### `propose-only`
|
||||
|
||||
Allowed:
|
||||
|
||||
- everything in `read-only`
|
||||
- create proposals
|
||||
- create review tasks
|
||||
|
||||
Not allowed:
|
||||
|
||||
- apply live changes
|
||||
|
||||
|
||||
### `apply-limited`
|
||||
|
||||
Allowed:
|
||||
|
||||
- everything in `propose-only`
|
||||
- write approved runtime overrides
|
||||
- trigger verification checks
|
||||
|
||||
Not allowed:
|
||||
|
||||
- mutate repository defaults
|
||||
- make destructive data changes
|
||||
- bypass policy engine
|
||||
|
||||
|
||||
## Runtime Components
|
||||
|
||||
The first durable architecture should introduce these components.
|
||||
|
||||
### 1. Signal Store
|
||||
|
||||
Stores normalized evidence and health outputs.
|
||||
|
||||
|
||||
### 2. Finding Store
|
||||
|
||||
Stores deterministic classifications that can be reused by multiple agents.
|
||||
|
||||
|
||||
### 3. Proposal Store
|
||||
|
||||
Stores recommended actions with evidence and confidence.
|
||||
|
||||
|
||||
### 4. Assessment Store
|
||||
|
||||
Stores structured situational-awareness outputs.
|
||||
|
||||
|
||||
### 5. Policy Engine
|
||||
|
||||
Decides:
|
||||
|
||||
- whether agent may run
|
||||
- whether proposal requires review
|
||||
- whether proposal may auto-apply
|
||||
- whether post-apply verification passed
|
||||
|
||||
|
||||
### 6. Override Store
|
||||
|
||||
Stores runtime-only configuration changes.
|
||||
|
||||
This is where endpoint repairs should live.
|
||||
|
||||
|
||||
## Relation To `aiprovider`
|
||||
|
||||
`aiprovider` should remain the model gateway.
|
||||
|
||||
It should not become the full agent runtime.
|
||||
|
||||
Recommended split:
|
||||
|
||||
- `aiprovider`
|
||||
- provider adaptation
|
||||
- prompt transport
|
||||
- model execution
|
||||
- protocol compatibility
|
||||
|
||||
- agent runtime
|
||||
- orchestration
|
||||
- signal handling
|
||||
- tool selection
|
||||
- proposal generation
|
||||
- policy and audit
|
||||
|
||||
This keeps provider concerns and agent behavior concerns separate.
|
||||
|
||||
|
||||
## Relation To Datasource Health
|
||||
|
||||
Datasource health becomes one vertical slice of this architecture.
|
||||
|
||||
Mapping:
|
||||
|
||||
- signal:
|
||||
- endpoint unreachable
|
||||
- schema mismatch
|
||||
- bad content type
|
||||
- finding:
|
||||
- `failed`
|
||||
- `schema_changed`
|
||||
- `moved_endpoint_suspected`
|
||||
- proposal:
|
||||
- runtime override suggestion
|
||||
- assessment:
|
||||
- datasource health summary for operators
|
||||
|
||||
|
||||
## Relation To Situational Awareness
|
||||
|
||||
Future situational-awareness capabilities should reuse the same flow:
|
||||
|
||||
- raw telemetry becomes signals
|
||||
- anomaly detection becomes findings
|
||||
- LLM correlation becomes reasoning
|
||||
- operator-facing output becomes assessments
|
||||
- policy-approved mitigations become actions
|
||||
|
||||
This lets the platform evolve from operational health governance into broader cyber/network posture workflows without changing the architecture.
|
||||
|
||||
|
||||
## Suggested Delivery Sequence
|
||||
|
||||
### Phase A
|
||||
|
||||
- finalize shared object model
|
||||
- implement health-oriented signal and finding storage
|
||||
|
||||
### Phase B
|
||||
|
||||
- implement Health Agent
|
||||
- generate proposals only
|
||||
|
||||
### Phase C
|
||||
|
||||
- implement Assessment Agent
|
||||
- expose structured assessments via API
|
||||
|
||||
### Phase D
|
||||
|
||||
- implement Correlation Agent
|
||||
- support multi-source incident grouping
|
||||
|
||||
### Phase E
|
||||
|
||||
- implement Recovery Agent with policy-gated runtime actions
|
||||
|
||||
|
||||
## Recommended First Build
|
||||
|
||||
The first build should not try to implement every agent role.
|
||||
|
||||
Recommended initial slice:
|
||||
|
||||
- shared object model
|
||||
- health signals
|
||||
- health findings
|
||||
- Health Agent
|
||||
- proposal generation only
|
||||
|
||||
This gives immediate value while preserving the longer-term architecture.
|
||||
|
||||
|
||||
## Non-Goals For The First Iteration
|
||||
|
||||
- repository YAML auto-rewrites
|
||||
- unrestricted autonomous action
|
||||
- full incident graph reasoning
|
||||
- automatic large-scale remediation
|
||||
- agent-owned configuration source of truth
|
||||
|
||||
|
||||
## Summary
|
||||
|
||||
Planet should treat agents as a reusable runtime for evidence, reasoning, proposals, and assessments.
|
||||
|
||||
The datasource health use case is the first practical entrypoint, but the architecture should already assume future situational-awareness expansion.
|
||||
|
||||
The safest path is:
|
||||
|
||||
- deterministic checks first
|
||||
- agent reasoning second
|
||||
- proposals before actions
|
||||
- runtime overrides instead of default mutation
|
||||
@@ -1,346 +0,0 @@
|
||||
# Agent Runtime Roadmap
|
||||
|
||||
## Overview
|
||||
|
||||
This document connects three existing planning threads into one implementation roadmap:
|
||||
|
||||
- `aiprovider` as the model gateway
|
||||
- datasource health governance as the first practical agent use case
|
||||
- situational awareness as the broader long-term target
|
||||
|
||||
Related documents:
|
||||
|
||||
- [aiprovider](/home/ray/dev/linkong/planet/docs/agents/aiprovider.md)
|
||||
- [datasource-health-plan](/home/ray/dev/linkong/planet/docs/agents/datasource-health-plan.md)
|
||||
- [agent-architecture-plan](/home/ray/dev/linkong/planet/docs/agents/agent-architecture-plan.md)
|
||||
|
||||
|
||||
## Big Picture
|
||||
|
||||
Planet should evolve in layers:
|
||||
|
||||
1. stable model gateway
|
||||
2. deterministic health and evidence collection
|
||||
3. agent runtime for reasoning and proposal generation
|
||||
4. situational-awareness assessments and controlled actions
|
||||
|
||||
This prevents the system from collapsing into a single giant "AI feature" with unclear boundaries.
|
||||
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
U["Frontend / Backend APIs / Operators"] --> B["Planet Backend"]
|
||||
B --> H["Datasource Health Services"]
|
||||
B --> R["Agent Runtime"]
|
||||
R --> P["aiprovider"]
|
||||
P --> M["OpenAI / Anthropic / MiniMax / Ollama / Local Models"]
|
||||
|
||||
C["Collectors / Snapshots / Logs / Alerts / BGP Signals"] --> S["Signal Store"]
|
||||
H --> S
|
||||
S --> E["Evaluation Layer"]
|
||||
E --> F["Findings"]
|
||||
F --> R
|
||||
|
||||
W["Web Search / Page Fetch / Docs Fetch"] --> R
|
||||
R --> PR["Proposals"]
|
||||
R --> AS["Assessments"]
|
||||
|
||||
PR --> O["Runtime Overrides / Review Queue / Tasks"]
|
||||
AS --> SA["Situational Awareness APIs / UI"]
|
||||
|
||||
O --> V["Verification Loop"]
|
||||
V --> S
|
||||
```
|
||||
|
||||
|
||||
## Role Boundaries
|
||||
|
||||
### `aiprovider`
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- provider compatibility
|
||||
- protocol adaptation
|
||||
- auth and model transport
|
||||
- request/response normalization
|
||||
|
||||
Not responsible for:
|
||||
|
||||
- agent orchestration
|
||||
- business workflows
|
||||
- datasource repair policy
|
||||
- situational-awareness domain logic
|
||||
|
||||
|
||||
### Backend
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- stable business APIs
|
||||
- auth and permissions
|
||||
- task orchestration
|
||||
- health records
|
||||
- proposal and override persistence
|
||||
- assessment exposure
|
||||
|
||||
|
||||
### Agent Runtime
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- consume findings and context
|
||||
- invoke LLMs via `aiprovider`
|
||||
- invoke tools such as web search
|
||||
- create proposals
|
||||
- create assessments
|
||||
- route to policy-controlled action paths
|
||||
|
||||
|
||||
## Delivery Sequence
|
||||
|
||||
## Stage 1: Gateway Foundation
|
||||
|
||||
Status:
|
||||
|
||||
- already in place
|
||||
|
||||
Delivered by current work:
|
||||
|
||||
- `aiprovider`
|
||||
- multi-provider compatibility
|
||||
- backend AI facade
|
||||
- MiniMax / Anthropic-compatible support
|
||||
- request-id propagation
|
||||
|
||||
Primary outcome:
|
||||
|
||||
- the system already has a stable way to call models
|
||||
|
||||
|
||||
## Stage 2: Datasource Health MVP
|
||||
|
||||
Goal:
|
||||
|
||||
- establish deterministic health observability
|
||||
|
||||
Key work:
|
||||
|
||||
- health check task runner
|
||||
- health result table
|
||||
- datasource health APIs
|
||||
- UI visibility
|
||||
- collector endpoint override precedence cleanup
|
||||
|
||||
Primary outcome:
|
||||
|
||||
- Planet knows which collectors are healthy before asking an LLM anything
|
||||
|
||||
|
||||
## Stage 3: Health Agent
|
||||
|
||||
Goal:
|
||||
|
||||
- let the first agent role operate on health failures
|
||||
|
||||
Key work:
|
||||
|
||||
- convert health failures into signals/findings
|
||||
- invoke agent only for failed or suspicious cases
|
||||
- produce repair proposals with evidence and confidence
|
||||
|
||||
Primary outcome:
|
||||
|
||||
- Planet can suggest endpoint repairs without mutating defaults
|
||||
|
||||
|
||||
## Stage 4: Runtime Repair Application
|
||||
|
||||
Goal:
|
||||
|
||||
- safely apply approved datasource repair proposals
|
||||
|
||||
Key work:
|
||||
|
||||
- override storage
|
||||
- policy-gated apply flow
|
||||
- verification after apply
|
||||
- rollback path
|
||||
|
||||
Primary outcome:
|
||||
|
||||
- datasource repair becomes operationally useful without polluting repository defaults
|
||||
|
||||
|
||||
## Stage 5: Situational Awareness Assessments
|
||||
|
||||
Goal:
|
||||
|
||||
- reuse the same runtime for broader operator-facing assessment
|
||||
|
||||
Key work:
|
||||
|
||||
- normalize telemetry and incident evidence into signals/findings
|
||||
- build Assessment Agent
|
||||
- expose structured assessments through backend APIs and UI
|
||||
|
||||
Primary outcome:
|
||||
|
||||
- LLM output becomes evidence-backed situational summary, not just ad hoc chat output
|
||||
|
||||
|
||||
## Stage 6: Correlation and Controlled Actions
|
||||
|
||||
Goal:
|
||||
|
||||
- connect multiple sources into higher-level posture and event groupings
|
||||
|
||||
Key work:
|
||||
|
||||
- event correlation
|
||||
- incident grouping
|
||||
- recommendation scoring
|
||||
- controlled action routing
|
||||
|
||||
Primary outcome:
|
||||
|
||||
- Planet becomes a true agent-assisted situational-awareness system
|
||||
|
||||
|
||||
## Implementation Tracks
|
||||
|
||||
These tracks can progress in parallel, but they should stay loosely coupled.
|
||||
|
||||
### Track A: Config and Runtime Resolution
|
||||
|
||||
Scope:
|
||||
|
||||
- datasource defaults
|
||||
- overrides
|
||||
- runtime precedence
|
||||
- audit trails
|
||||
|
||||
First milestone:
|
||||
|
||||
- health-safe override layer
|
||||
|
||||
|
||||
### Track B: Health and Evidence
|
||||
|
||||
Scope:
|
||||
|
||||
- deterministic checks
|
||||
- failure categorization
|
||||
- signal and finding persistence
|
||||
|
||||
First milestone:
|
||||
|
||||
- datasource health record system
|
||||
|
||||
|
||||
### Track C: Agent Runtime
|
||||
|
||||
Scope:
|
||||
|
||||
- shared object model
|
||||
- orchestration flow
|
||||
- prompt/tool pipeline
|
||||
- policy integration
|
||||
|
||||
First milestone:
|
||||
|
||||
- Health Agent proposal pipeline
|
||||
|
||||
|
||||
### Track D: Situational Awareness
|
||||
|
||||
Scope:
|
||||
|
||||
- assessment schema
|
||||
- multi-source context assembly
|
||||
- operator-facing outputs
|
||||
|
||||
First milestone:
|
||||
|
||||
- structured assessment API
|
||||
|
||||
|
||||
## Shared Artifacts
|
||||
|
||||
To avoid fragmentation, these artifacts should be shared across all future agent work.
|
||||
|
||||
### Shared object model
|
||||
|
||||
- `Signal`
|
||||
- `Finding`
|
||||
- `Proposal`
|
||||
- `Assessment`
|
||||
|
||||
### Shared orchestration flow
|
||||
|
||||
- collect
|
||||
- validate
|
||||
- classify
|
||||
- reason
|
||||
- propose or assess
|
||||
- review or apply
|
||||
- verify
|
||||
- archive
|
||||
|
||||
### Shared policy model
|
||||
|
||||
- read-only
|
||||
- propose-only
|
||||
- apply-limited
|
||||
|
||||
|
||||
## Recommended Next Concrete Steps
|
||||
|
||||
1. Build Stage 2 first
|
||||
|
||||
- datasource health records
|
||||
- deterministic checks
|
||||
- no automatic repair
|
||||
|
||||
2. Then build Stage 3
|
||||
|
||||
- Health Agent
|
||||
- proposal generation only
|
||||
|
||||
3. Then Stage 4
|
||||
|
||||
- override apply flow
|
||||
- rollback and verification
|
||||
|
||||
4. Only after that start Stage 5
|
||||
|
||||
- broader situational-awareness assessment workflows
|
||||
|
||||
|
||||
## Why This Order
|
||||
|
||||
Because situational-awareness quality depends on reliable upstream data.
|
||||
|
||||
If datasource health is weak:
|
||||
|
||||
- agent reasoning quality will degrade
|
||||
- false explanations will increase
|
||||
- assessment trust will drop
|
||||
|
||||
So datasource health is not a side task.
|
||||
|
||||
It is the first operational foundation for the later situational-awareness system.
|
||||
|
||||
|
||||
## Summary
|
||||
|
||||
Planet should be built as:
|
||||
|
||||
- `aiprovider` for model access
|
||||
- backend services for orchestration and persistence
|
||||
- datasource health as the first evidence-governance layer
|
||||
- agent runtime as the reusable reasoning core
|
||||
- situational awareness as the long-term application layer
|
||||
|
||||
That path keeps the architecture coherent and lets each phase produce useful functionality without forcing a rewrite later.
|
||||
@@ -1,361 +0,0 @@
|
||||
# AI Playground Development Plan
|
||||
|
||||
## 目标
|
||||
|
||||
这份计划用于统一 `aiprovider`、`backend AI facade`、`Playground` 页面,以及后续 `BGP / 告警 / 数据源健康` 等 AI 入口的演进方向。
|
||||
|
||||
当前原则:
|
||||
|
||||
- `aiprovider` 继续作为独立模型网关
|
||||
- `backend` 继续作为稳定业务入口
|
||||
- `frontend` 负责测试台和业务 UI
|
||||
- 先做“可控、可验证、可解释”的 AI 能力,再逐步引入 agent/tool calling
|
||||
|
||||
## 当前已完成
|
||||
|
||||
### 1. AI 网关基础层
|
||||
|
||||
已完成:
|
||||
|
||||
- 独立 `aiprovider` 服务
|
||||
- `backend -> aiprovider -> model provider` 调用链
|
||||
- `provider/status` 与 `situational-awareness/analyze` 稳定接口
|
||||
- `X-Request-ID` 透传
|
||||
- 轻量超时与重试
|
||||
- MiniMax / Anthropic-compatible / OpenAI-compatible / Ollama 适配
|
||||
|
||||
相关文件:
|
||||
|
||||
- [backend/app/api/v1/ai.py](/home/ray/dev/linkong/planet/backend/app/api/v1/ai.py)
|
||||
- [backend/app/services/ai_client.py](/home/ray/dev/linkong/planet/backend/app/services/ai_client.py)
|
||||
- [aiprovider/main.py](/home/ray/dev/linkong/planet/aiprovider/main.py)
|
||||
- [aiprovider/provider_service.py](/home/ray/dev/linkong/planet/aiprovider/provider_service.py)
|
||||
- [docs/aiprovider.md](/home/ray/dev/linkong/planet/docs/aiprovider.md)
|
||||
|
||||
### 2. 本地运行与配置打通
|
||||
|
||||
已完成:
|
||||
|
||||
- `planet.sh` 启动链路纳入 `aiprovider`
|
||||
- `planet.sh` 启动完成后输出 Playground 入口
|
||||
- `docker-compose.yml` 为 `aiprovider` 加入 `env_file`
|
||||
- `backend/.env` 与 `aiprovider/.env` 两侧 service token 对齐
|
||||
- `Playground` 状态缓存,避免页面切换时每次都重新请求 provider 状态
|
||||
|
||||
相关文件:
|
||||
|
||||
- [planet.sh](/home/ray/dev/linkong/planet/planet.sh)
|
||||
- [docker-compose.yml](/home/ray/dev/linkong/planet/docker-compose.yml)
|
||||
- [backend/.env.example](/home/ray/dev/linkong/planet/backend/.env.example)
|
||||
- [aiprovider/.env.example](/home/ray/dev/linkong/planet/aiprovider/.env.example)
|
||||
|
||||
### 3. Playground UI 基础版
|
||||
|
||||
已完成:
|
||||
|
||||
- 新增前端路由 `/playground`
|
||||
- 左侧 `Provider 状态 + 测试说明`
|
||||
- 右侧 `请求 / 结果` Tabs
|
||||
- `Provider 状态` 支持手动刷新
|
||||
- `测试说明` 支持折叠
|
||||
- 内部区域采用细滚动条
|
||||
- 页面布局开始遵循“单屏工作区 + 模块内部滚动”规范
|
||||
|
||||
相关文件:
|
||||
|
||||
- [frontend/src/pages/Playground/Playground.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Playground/Playground.tsx)
|
||||
- [frontend/src/App.tsx](/home/ray/dev/linkong/planet/frontend/src/App.tsx)
|
||||
- [frontend/src/components/AppLayout/AppLayout.tsx](/home/ray/dev/linkong/planet/frontend/src/components/AppLayout/AppLayout.tsx)
|
||||
- [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css)
|
||||
|
||||
### 4. 前端布局规范沉淀
|
||||
|
||||
已完成:
|
||||
|
||||
- 把“一屏工作区、主模块优先、模块内部滚动”的规范文档化
|
||||
- 明确 `BGP` 页面为当前参考实现
|
||||
|
||||
相关文件:
|
||||
|
||||
- [docs/frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/frontend-layout-guidelines.md)
|
||||
- [frontend/src/pages/BGP/BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx)
|
||||
|
||||
## 当前限制
|
||||
|
||||
### 1. Playground 还是 prompt playground,不是 agent playground
|
||||
|
||||
当前 `Playground` 的 `观察项 / 目标 / 约束条件` 都是人工输入。
|
||||
|
||||
模型现在拿到的是:
|
||||
|
||||
- 你手工输入的结构化字段
|
||||
- 后端传递的少量静态上下文
|
||||
|
||||
模型现在拿不到:
|
||||
|
||||
- 实时 BGP 事件
|
||||
- 真实告警列表
|
||||
- 数据源健康状态
|
||||
- 自动检索结果
|
||||
- tool calling / skills / 自主取数
|
||||
|
||||
### 2. `situational-awareness/analyze` 还是通用提示词接口
|
||||
|
||||
当前更适合:
|
||||
|
||||
- 测试链路
|
||||
- 测试模型输出风格
|
||||
- 验证不同 provider 是否正常返回
|
||||
|
||||
当前还不适合:
|
||||
|
||||
- 直接当真实态势系统主入口
|
||||
- 让用户手工维护长期分析模板
|
||||
- 代替专用业务研判接口
|
||||
|
||||
### 3. 还没有可验证的真实业务输入注入
|
||||
|
||||
目前最缺的是:
|
||||
|
||||
- 从业务系统自动整理“事实输入”
|
||||
- 再把这些事实喂给 AI
|
||||
|
||||
而不是继续让用户在 Playground 手工输入真实事件摘要。
|
||||
|
||||
## 短期计划
|
||||
|
||||
### Phase A: Playground 收敛为稳定测试台
|
||||
|
||||
目标:
|
||||
|
||||
- 保持 Playground 简洁可用
|
||||
- 不再继续堆“高级参数”
|
||||
|
||||
工作项:
|
||||
|
||||
- 继续微调左侧 `Provider 状态` 与 `测试说明` 的空间策略
|
||||
- 保持 `请求 / 结果` 为单一主工作区
|
||||
- 不引入盲填式高级字段
|
||||
- 统一滚动条、卡片、溢出行为
|
||||
|
||||
完成标准:
|
||||
|
||||
- 笔记本视口下依然可用
|
||||
- 各模块标题可见
|
||||
- 主要阅读区始终是右侧 Tabs
|
||||
|
||||
### Phase B: BGP AI 简报
|
||||
|
||||
目标:
|
||||
|
||||
- 不再依赖手工填写“观察项”
|
||||
- 让系统自动把真实 BGP 数据注入 AI
|
||||
- 让 BGP 页面逐步从“摘要汇总”升级为“证据驱动的区域态势分析”
|
||||
|
||||
建议实现:
|
||||
|
||||
- 新增专用后端接口,例如:
|
||||
- `POST /api/v1/ai/bgp/brief`
|
||||
- 后端自动读取:
|
||||
- incidents summary
|
||||
- anomalies
|
||||
- recent events
|
||||
- collector coverage summary
|
||||
- 后端将结构化事实注入 `context / observations`
|
||||
- 前端在 BGP 页面增加“生成 AI 简报”
|
||||
|
||||
当前阶段说明:
|
||||
|
||||
- 第一版 `BGP AI 简报` 允许先落地为“值班摘要生成器”
|
||||
- 也就是先把 incidents / anomalies / events / collector coverage 自动注入
|
||||
- 允许模型先做事实摘要、风险归纳、建议动作
|
||||
|
||||
但这不应被视为 Phase B 的最终形态。
|
||||
|
||||
Phase B 后续还需要补齐:
|
||||
|
||||
- prefix geography 证据注入
|
||||
- `iptoasn`
|
||||
- `opengeofeed`
|
||||
- `nro_delegated`
|
||||
- 基于 `affected_regions` 与 prefix geography 的区域聚合
|
||||
- 区分“真实区域热度”与“collector coverage 偏差”
|
||||
- 对高风险 prefix / ASN 给出更明确的国家、城市、运营商归属线索
|
||||
- 让 AI 输出明确回答:
|
||||
- 哪些区域正在异常升温
|
||||
- 哪些结论只是观测站偏差
|
||||
- 当前还缺哪些区域证据
|
||||
|
||||
完成标准:
|
||||
|
||||
- 用户不需要手工录入 BGP 观察项
|
||||
- AI 输出能明确区分“事实”和“研判”
|
||||
- AI 不只是复述总量和最近几条事件,还能利用 prefix geography 与 affected regions 做区域态势判断
|
||||
- 输出中能明确指出:
|
||||
- 高风险区域
|
||||
- 区域证据来源
|
||||
- collector coverage 偏差对判断的影响
|
||||
|
||||
### Phase C: 告警 / 数据源健康 AI 简报
|
||||
|
||||
目标:
|
||||
|
||||
- 复用同样模式,扩展到其他模块
|
||||
|
||||
建议入口:
|
||||
|
||||
- `Alerts` 页面:异常与告警摘要
|
||||
- `DataSources` 页面:采集失败与健康状态总结
|
||||
|
||||
原则:
|
||||
|
||||
- 每个业务页优先做“专用 AI 简报”
|
||||
- 不优先做“万能大聊天框”
|
||||
|
||||
## 中期计划
|
||||
|
||||
### 1. Assessment Layer
|
||||
|
||||
目标:
|
||||
|
||||
- 不只返回自由文本
|
||||
- 返回结构化的 assessment
|
||||
|
||||
建议输出字段:
|
||||
|
||||
- summary
|
||||
- key_risks
|
||||
- evidence
|
||||
- recommendations
|
||||
- confidence
|
||||
- missing_data
|
||||
|
||||
这样后续才能:
|
||||
|
||||
- 持久化
|
||||
- 回看
|
||||
- 对比不同时间的 AI 结论
|
||||
- 在 Earth / Dashboard / BGP 页面稳定展示
|
||||
|
||||
### 2. Evidence-first Runtime
|
||||
|
||||
目标:
|
||||
|
||||
- 所有 AI 分析先取真实数据,再调模型
|
||||
|
||||
原则:
|
||||
|
||||
- 先 evidence
|
||||
- 再 prompt
|
||||
- 最后才是自由生成
|
||||
|
||||
优先要做的不是更强聊天,而是:
|
||||
|
||||
- 更稳定的数据注入
|
||||
- 更一致的事实模板
|
||||
- 更清晰的结果结构
|
||||
|
||||
### 3. 按页面提供专用入口
|
||||
|
||||
目标:
|
||||
|
||||
- 让 AI 成为业务视图的一部分,而不是孤立 playground
|
||||
|
||||
优先顺序建议:
|
||||
|
||||
1. `BGP` AI 简报
|
||||
2. `Alerts` AI 简报
|
||||
3. `DataSources` 健康研判
|
||||
4. `Dashboard` 总览总结
|
||||
|
||||
## 长期计划
|
||||
|
||||
### 1. Tool Calling / Agent Runtime
|
||||
|
||||
只有在以下基础稳定后再推进:
|
||||
|
||||
- 数据源健康信号稳定
|
||||
- BGP / Alerts / Datasource evidence 注入稳定
|
||||
- assessment 结构稳定
|
||||
|
||||
长期可做能力:
|
||||
|
||||
- AI 调用受控工具查询业务数据
|
||||
- AI 调用检索/web search 做外部验证
|
||||
- AI 生成建议而不是直接修改系统
|
||||
- 审核后触发受控动作
|
||||
|
||||
### 2. 受控动作与闭环
|
||||
|
||||
潜在方向:
|
||||
|
||||
- 根据健康异常生成修复建议
|
||||
- 根据态势变化生成处理建议
|
||||
- 进入 review queue
|
||||
- 审批后执行
|
||||
- 验证结果并形成闭环
|
||||
|
||||
### 3. 多模块统一 AI 体验
|
||||
|
||||
长期目标不是一个孤立 Playground,而是:
|
||||
|
||||
- 每个业务页都有自己的 AI 入口
|
||||
- 共享统一的 backend AI facade
|
||||
- 共享统一的 assessment 结构
|
||||
- 共享统一的 evidence 注入与审计链路
|
||||
|
||||
## 设计决策总结
|
||||
|
||||
### 为什么保留 `aiprovider`
|
||||
|
||||
因为它已经很好地承担了:
|
||||
|
||||
- provider 适配
|
||||
- 协议兼容
|
||||
- service token 边界
|
||||
- 独立重启与部署
|
||||
|
||||
因此短期内不建议把它并回 `backend`。
|
||||
|
||||
### 为什么 Playground 不做成万能聊天页
|
||||
|
||||
因为当前更需要的是:
|
||||
|
||||
- 稳定测试链路
|
||||
- 可验证业务输入
|
||||
- 专用分析入口
|
||||
|
||||
而不是一个泛化但没有真实数据支撑的聊天框。
|
||||
|
||||
### 为什么优先做专用 AI 简报
|
||||
|
||||
因为:
|
||||
|
||||
- 数据可以自动注入
|
||||
- 用户心智更清晰
|
||||
- 输出更容易结构化
|
||||
- 更容易校验事实与研判是否一致
|
||||
|
||||
## 下一步建议
|
||||
|
||||
按优先级建议接下来这样做:
|
||||
|
||||
1. 稳住 `Playground` 当前布局,不再大幅重做
|
||||
2. 在 `BGP` 页面新增专用 “AI 简报” 入口
|
||||
3. 后端新增 `BGP brief` 专用接口,自动注入真实数据
|
||||
4. 补齐 `BGP brief` 的区域态势证据层
|
||||
5. 把 AI 输出逐步从自由文本升级为结构化 assessment
|
||||
|
||||
### BGP Brief 后续子项
|
||||
|
||||
为避免把“已有 AI 简报”误判成“区域分析已完成”,这里单独记录 `BGP brief` 的后续 backlog:
|
||||
|
||||
1. 把高风险 prefix 命中的 `iptoasn / opengeofeed / nro_delegated` 结果注入 brief context
|
||||
2. 按国家/城市聚合 active incidents、anomalies、affected prefixes,生成区域热点事实层
|
||||
3. 把 collector coverage 与区域热点并排注入,避免模型把观测偏差误判成区域风险
|
||||
4. 对高风险 ASN / prefix 追加归属线索,如国家、城市、可能运营商或注册区域
|
||||
5. 在输出结构中单独增加:
|
||||
- 区域态势
|
||||
- 证据来源
|
||||
- 观测偏差说明
|
||||
- 缺失区域证据
|
||||
@@ -1,333 +0,0 @@
|
||||
# AI Provider Guide
|
||||
|
||||
## Overview
|
||||
|
||||
`aiprovider` is the model-adapter service for Planet.
|
||||
|
||||
It isolates model-vendor details from the main backend so the rest of the system can call a stable business API:
|
||||
|
||||
- Caller service -> `planet backend`
|
||||
- `planet backend` -> `aiprovider`
|
||||
- `aiprovider` -> concrete model provider
|
||||
|
||||
The recommended default is:
|
||||
|
||||
- External and cross-service callers use `planet backend`
|
||||
- Only infrastructure-grade internal jobs call `aiprovider` directly
|
||||
|
||||
## Responsibilities
|
||||
|
||||
`backend` is responsible for:
|
||||
|
||||
- authentication and authorization
|
||||
- business-level request shaping
|
||||
- stable `/api/v1/ai/...` endpoints
|
||||
- internal service-to-service authentication toward `aiprovider`
|
||||
|
||||
`aiprovider` is responsible for:
|
||||
|
||||
- model protocol adaptation
|
||||
- provider selection by `.env`
|
||||
- timeout and lightweight retry
|
||||
- request tracing via `X-Request-ID`
|
||||
|
||||
This now follows an OpenClaw-like seam:
|
||||
|
||||
- `AI_PROVIDER` identifies the vendor or logical provider
|
||||
- `AI_PROVIDER_API` identifies the wire adapter
|
||||
|
||||
That split makes MiniMax, Claude-compatible gateways, and self-hosted OpenAI-compatible services easier to model without overloading one config field.
|
||||
|
||||
## Supported Providers
|
||||
|
||||
`aiprovider` currently supports these provider identities:
|
||||
|
||||
- `openai`
|
||||
- `anthropic`
|
||||
- `minimax`
|
||||
- `ollama`
|
||||
|
||||
Supported request adapters:
|
||||
|
||||
- `openai-completions`
|
||||
- `anthropic-messages`
|
||||
- `ollama-generate`
|
||||
|
||||
Backward-compatible aliases still accepted:
|
||||
|
||||
- `openai_compatible`
|
||||
- `anthropic_compatible`
|
||||
- `claude_compatible`
|
||||
|
||||
Provider mapping:
|
||||
|
||||
- `vLLM`, `LM Studio`, `One API`: `AI_PROVIDER=openai`, `AI_PROVIDER_API=openai-completions`
|
||||
- `MiniMax`: `AI_PROVIDER=minimax`, `AI_PROVIDER_API=anthropic-messages`
|
||||
- Claude-compatible gateways: `AI_PROVIDER=anthropic`, `AI_PROVIDER_API=anthropic-messages`
|
||||
- `Ollama`: `AI_PROVIDER=ollama`, `AI_PROVIDER_API=ollama-generate`
|
||||
|
||||
## API Surfaces
|
||||
|
||||
### Main backend API
|
||||
|
||||
Preferred stable entrypoints:
|
||||
|
||||
- `GET /api/v1/ai/provider/status`
|
||||
- `POST /api/v1/ai/situational-awareness/analyze`
|
||||
|
||||
Authentication:
|
||||
|
||||
- `Authorization: Bearer <jwt>`
|
||||
|
||||
Optional tracing header:
|
||||
|
||||
- `X-Request-ID: <caller-generated-id>`
|
||||
|
||||
The backend will propagate `X-Request-ID` to `aiprovider` and return the same header in the response.
|
||||
|
||||
### AI provider internal API
|
||||
|
||||
Internal-only endpoints:
|
||||
|
||||
- `GET /v1/provider/status`
|
||||
- `POST /v1/analyze`
|
||||
|
||||
Authentication:
|
||||
|
||||
- `X-Provider-Token: <shared-secret>`
|
||||
|
||||
Optional tracing header:
|
||||
|
||||
- `X-Request-ID: <caller-generated-id>`
|
||||
|
||||
## Request Example
|
||||
|
||||
### Call through backend
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8000/api/v1/ai/situational-awareness/analyze \
|
||||
-H "Authorization: Bearer <access_token>" \
|
||||
-H "X-Request-ID: bgp-incident-20260407-001" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"title": "BGP异常研判",
|
||||
"objective": "总结当前风险并给出处置建议",
|
||||
"observations": [
|
||||
"collector A 在 5 分钟内出现多次 origin 变更",
|
||||
"异常集中在同一地区前缀"
|
||||
],
|
||||
"constraints": [
|
||||
"不要编造不存在的数据",
|
||||
"区分事实和推断"
|
||||
],
|
||||
"context": {
|
||||
"source": "bgp-monitor",
|
||||
"severity": "high"
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
### Call `aiprovider` directly
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8010/v1/analyze \
|
||||
-H "X-Provider-Token: change_me" \
|
||||
-H "X-Request-ID: ai-batch-job-001" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"title": "链路波动分析",
|
||||
"objective": "给出简要态势摘要和下一步建议",
|
||||
"observations": [
|
||||
"多个节点出现延迟上升"
|
||||
],
|
||||
"constraints": [
|
||||
"不要假设根因已经确认"
|
||||
],
|
||||
"context": {
|
||||
"region": "APAC"
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
## Response Shape
|
||||
|
||||
Both backend and `aiprovider` return the same payload shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"provider": "minimax",
|
||||
"api": "anthropic-messages",
|
||||
"model": "MiniMax-M2.7",
|
||||
"content": "1) 态势摘要 ...",
|
||||
"content_blocks": [],
|
||||
"text_blocks": [],
|
||||
"thinking_blocks": [],
|
||||
"raw_response": {}
|
||||
}
|
||||
```
|
||||
|
||||
Both services also return:
|
||||
|
||||
- `X-Request-ID: <id>`
|
||||
|
||||
## Configuration
|
||||
|
||||
### Backend
|
||||
|
||||
Recommended backend `.env`:
|
||||
|
||||
```env
|
||||
AI_PROVIDER_SERVICE_URL=http://localhost:8010
|
||||
AI_PROVIDER_SERVICE_TOKEN=change_me
|
||||
AI_PROVIDER_TIMEOUT_SECONDS=60
|
||||
AI_PROVIDER_RETRY_ATTEMPTS=2
|
||||
```
|
||||
|
||||
Reference file:
|
||||
|
||||
- [backend/.env.example](/home/ray/dev/linkong/planet/backend/.env.example)
|
||||
|
||||
### AI Provider
|
||||
|
||||
Reference file:
|
||||
|
||||
- [aiprovider/.env.example](/home/ray/dev/linkong/planet/aiprovider/.env.example)
|
||||
|
||||
Frontend local reference:
|
||||
|
||||
- [frontend/.env.example](/home/ray/dev/linkong/planet/frontend/.env.example)
|
||||
|
||||
Common settings:
|
||||
|
||||
```env
|
||||
SERVICE_NAME=planet-ai-provider
|
||||
SERVICE_VERSION=0.1.0
|
||||
AI_PROVIDER_SERVICE_TOKEN=change_me
|
||||
AI_TIMEOUT_SECONDS=60
|
||||
AI_HTTP_RETRY_ATTEMPTS=2
|
||||
AI_ANALYSIS_SYSTEM_PROMPT=你是态势感知分析助手。请基于输入的上下文、观测与约束,输出结构化、克制、可执行的分析。
|
||||
```
|
||||
|
||||
### OpenAI-compatible example
|
||||
|
||||
```env
|
||||
AI_PROVIDER=openai
|
||||
AI_PROVIDER_API=openai-completions
|
||||
AI_BASE_URL=http://127.0.0.1:8001/v1
|
||||
AI_API_KEY=local-key
|
||||
AI_MODEL=your-local-model
|
||||
```
|
||||
|
||||
### MiniMax CN example
|
||||
|
||||
```env
|
||||
AI_PROVIDER=minimax
|
||||
AI_PROVIDER_API=anthropic-messages
|
||||
AI_BASE_URL=https://api.minimaxi.com/anthropic
|
||||
AI_API_KEY=sk-cp-xxxxx
|
||||
AI_MODEL=MiniMax-M2.7
|
||||
AI_MAX_TOKENS=1200
|
||||
AI_ANTHROPIC_VERSION=2023-06-01
|
||||
```
|
||||
|
||||
MiniMax note:
|
||||
|
||||
- This follows the same Anthropic Messages request shape as the official MiniMax examples.
|
||||
- For MiniMax, `aiprovider` now disables `thinking` by default unless the caller explicitly passes a `thinking` object.
|
||||
- This mirrors OpenClaw's caution around MiniMax Anthropic-compatible behavior.
|
||||
|
||||
### Anthropic-compatible example
|
||||
|
||||
```env
|
||||
AI_PROVIDER=anthropic
|
||||
AI_PROVIDER_API=anthropic-messages
|
||||
AI_BASE_URL=https://your-claude-compatible-endpoint.example.com/anthropic
|
||||
AI_API_KEY=your_api_key
|
||||
AI_MODEL=your-model
|
||||
AI_MAX_TOKENS=1200
|
||||
AI_ANTHROPIC_VERSION=2023-06-01
|
||||
```
|
||||
|
||||
### Ollama example
|
||||
|
||||
```env
|
||||
AI_PROVIDER=ollama
|
||||
AI_PROVIDER_API=ollama-generate
|
||||
AI_BASE_URL=http://127.0.0.1:11434
|
||||
AI_API_KEY=
|
||||
AI_MODEL=qwen2.5:7b
|
||||
```
|
||||
|
||||
## Deployment Modes
|
||||
|
||||
### Single machine
|
||||
|
||||
Recommended local flow:
|
||||
|
||||
- `backend` on `localhost:8000`
|
||||
- `aiprovider` on `localhost:8010`
|
||||
- local model gateway on `localhost:11434` or another local port
|
||||
|
||||
Helpers already included:
|
||||
|
||||
- [planet.sh](/home/ray/dev/linkong/planet/planet.sh)
|
||||
- [docker-compose.local-model.yml](/home/ray/dev/linkong/planet/docker-compose.local-model.yml)
|
||||
|
||||
### Multi-machine
|
||||
|
||||
Example topology:
|
||||
|
||||
- app machine: `backend`
|
||||
- AI gateway machine: `aiprovider`
|
||||
- model machine: local model service or cloud proxy
|
||||
|
||||
In that case, this becomes service-to-service HTTP RPC:
|
||||
|
||||
- caller -> backend
|
||||
- backend -> `http://10.0.0.12:8010`
|
||||
- `aiprovider` -> model endpoint
|
||||
|
||||
Recommended cross-machine backend config:
|
||||
|
||||
```env
|
||||
AI_PROVIDER_SERVICE_URL=http://10.0.0.12:8010
|
||||
AI_PROVIDER_SERVICE_TOKEN=change_me
|
||||
AI_PROVIDER_TIMEOUT_SECONDS=60
|
||||
AI_PROVIDER_RETRY_ATTEMPTS=2
|
||||
```
|
||||
|
||||
Recommended operating rules:
|
||||
|
||||
- keep `aiprovider` on a private network
|
||||
- protect it with `X-Provider-Token` at minimum
|
||||
- always send `X-Request-ID`
|
||||
- keep callers on the backend API unless they are infrastructure jobs
|
||||
|
||||
## Retry And Failure Behavior
|
||||
|
||||
`backend -> aiprovider`:
|
||||
|
||||
- retries lightweight network / 5xx failures
|
||||
- returns `502` when the provider service is unavailable
|
||||
|
||||
`aiprovider -> model provider`:
|
||||
|
||||
- retries lightweight network / 5xx failures
|
||||
- returns `502` when the model provider is unavailable
|
||||
|
||||
This is intentionally conservative. It avoids masking persistent errors while still absorbing short hiccups.
|
||||
|
||||
## Operational Notes
|
||||
|
||||
- `./planet.sh start` now starts `aiprovider` automatically
|
||||
- `./planet.sh restart -a` restarts only `aiprovider`
|
||||
- `./planet.sh log -a` tails `aiprovider` logs
|
||||
- `./planet.sh health` reports `aiprovider` health
|
||||
|
||||
## Recommended Calling Policy
|
||||
|
||||
- Frontend and application services: call `backend`
|
||||
- Scheduled infra jobs and diagnostics: optionally call `aiprovider`
|
||||
- Do not let multiple business services integrate model vendors independently
|
||||
|
||||
That keeps provider switching centralized and avoids model-specific drift across the system.
|
||||
@@ -1,422 +0,0 @@
|
||||
# BGP Region Aggregation Plan
|
||||
|
||||
## Goal
|
||||
|
||||
This document refines the current BGP `activity layer` into an implementation-ready regional aggregation design.
|
||||
|
||||
Primary product goal:
|
||||
|
||||
- turn sparse prefix-level observations, anomalies, and incidents into a readable `regional observability layer`
|
||||
- keep Earth visually alive during low-incident periods
|
||||
- make `incident markers` remain the highest-confidence foreground layer instead of replacing them
|
||||
|
||||
This layer is not a new collector, detector, or raw storage table.
|
||||
It is an aggregation/view-model layer:
|
||||
|
||||
`observations -> enrichment -> anomalies/incidents -> geography mapping -> region aggregation -> Earth/UI activity layer`
|
||||
|
||||
## Why This Layer Exists
|
||||
|
||||
Current product gap from [bgp-context.md](/home/ray/dev/linkong/planet/docs/bgp-context.md):
|
||||
|
||||
- incident density is naturally low
|
||||
- anomaly density is higher, but still not enough to keep the globe expressive all the time
|
||||
- collector presence alone proves coverage, but does not communicate `where routing is currently active or noisy`
|
||||
|
||||
So the missing middle layer is:
|
||||
|
||||
- `collectors` show that observation exists
|
||||
- `regions` show where activity is building up
|
||||
- `incidents` show the specific high-confidence focus events
|
||||
|
||||
## Scope
|
||||
|
||||
This plan is specifically for:
|
||||
|
||||
- a backend aggregation service
|
||||
- a summary API for console/stats
|
||||
- a GeoJSON API for Earth rendering
|
||||
- an Earth background activity layer that supports, but does not replace, incident markers
|
||||
|
||||
This plan does not attempt to solve:
|
||||
|
||||
- exact prefix geolocation quality
|
||||
- polygon-heavy geopolitical visualization
|
||||
- persistent materialized region tables in v1
|
||||
|
||||
## Region Layer Definition
|
||||
|
||||
Recommended conceptual model:
|
||||
|
||||
- `region layer` = background situational awareness
|
||||
- `incident layer` = focal event markers
|
||||
|
||||
That means:
|
||||
|
||||
- region activity should answer `where is routing behavior currently active or abnormal`
|
||||
- incident markers should answer `which concrete event should the user click`
|
||||
|
||||
## Recommended Output Model
|
||||
|
||||
Suggested backend output object:
|
||||
|
||||
## `BGPRegionActivity`
|
||||
|
||||
```json
|
||||
{
|
||||
"region_key": "sea",
|
||||
"region_name": "Southeast Asia",
|
||||
"center_lat": 1.3521,
|
||||
"center_lon": 103.8198,
|
||||
"observation_count": 128,
|
||||
"anomaly_count": 9,
|
||||
"incident_count": 2,
|
||||
"activity_score": 17.6,
|
||||
"status": "incident",
|
||||
"affected_prefix_count": 14,
|
||||
"affected_asn_count": 6,
|
||||
"collector_count": 5,
|
||||
"first_seen_at": "2026-04-02T10:00:00Z",
|
||||
"last_seen_at": "2026-04-02T10:12:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
### Fields To Keep In MVP
|
||||
|
||||
- `region_key`
|
||||
- `region_name`
|
||||
- `center_lat`
|
||||
- `center_lon`
|
||||
- `observation_count`
|
||||
- `anomaly_count`
|
||||
- `incident_count`
|
||||
- `activity_score`
|
||||
- `status`
|
||||
- `affected_prefix_count`
|
||||
- `affected_asn_count`
|
||||
- `collector_count`
|
||||
- `first_seen_at`
|
||||
- `last_seen_at`
|
||||
|
||||
### Fields To Delay
|
||||
|
||||
These are useful, but not required for the first implementation:
|
||||
|
||||
- `bounding_box`
|
||||
- `top_incident_types`
|
||||
- `top_prefixes`
|
||||
- polygon geometry
|
||||
|
||||
## Region Definition Strategy
|
||||
|
||||
### Recommendation
|
||||
|
||||
Use a static region-definition table first.
|
||||
|
||||
Examples:
|
||||
|
||||
- `north_america`
|
||||
- `south_america`
|
||||
- `western_europe`
|
||||
- `eastern_europe`
|
||||
- `east_asia`
|
||||
- `southeast_asia`
|
||||
- `south_asia`
|
||||
- `middle_east`
|
||||
- `north_africa`
|
||||
- `sub_saharan_africa`
|
||||
- `oceania`
|
||||
|
||||
Why this is the right v1 choice:
|
||||
|
||||
- stable UI semantics
|
||||
- strong readability on Earth
|
||||
- easier debugging and explanation
|
||||
- lower implementation cost than geohash or H3 grids
|
||||
|
||||
### Not Recommended For V1
|
||||
|
||||
- geohash cell aggregation
|
||||
- H3 aggregation
|
||||
- fine-grained lat/lon bucket maps
|
||||
|
||||
Those are more flexible, but they make the map feel fragmented and less explainable.
|
||||
|
||||
## Geography Mapping Strategy
|
||||
|
||||
Do not reduce the implementation to only `prefix -> exact geo`.
|
||||
|
||||
The region layer should follow the same geography-priority logic already implied by the current BGP direction:
|
||||
|
||||
1. `prefix_geography`
|
||||
2. `prefix_scope`
|
||||
3. `ASN organization region`
|
||||
4. `collector centroid` fallback
|
||||
|
||||
This matters because exact prefix geography will often be incomplete or approximate.
|
||||
The region layer should stay robust even when only partial enrichment is available.
|
||||
|
||||
## Backend Design
|
||||
|
||||
Recommended new service file:
|
||||
|
||||
- [backend/app/services/bgp_regions.py](/home/ray/dev/linkong/planet/backend/app/services/bgp_regions.py)
|
||||
|
||||
Suggested responsibilities:
|
||||
|
||||
- `map_record_to_region(...)`
|
||||
- `aggregate_region_activity(...)`
|
||||
- `build_region_geojson(...)`
|
||||
- `resolve_activity_status(...)`
|
||||
- `compute_activity_score(...)`
|
||||
|
||||
### Data Source Inputs
|
||||
|
||||
Use a recent rolling window, default `15 minutes`, and aggregate from:
|
||||
|
||||
- `BGPObservation`
|
||||
- `BGPAnomaly`
|
||||
- active `BGPIncident`
|
||||
|
||||
### Aggregation Flow
|
||||
|
||||
1. query observations in the time window
|
||||
2. query anomalies in the same window
|
||||
3. query active incidents in the same window or active status set
|
||||
4. resolve each record to a best-effort region
|
||||
5. accumulate per-region counters
|
||||
6. compute score and status
|
||||
7. return region activity list
|
||||
|
||||
## Status Model
|
||||
|
||||
Recommended status buckets:
|
||||
|
||||
- `idle`
|
||||
- `observing`
|
||||
- `anomaly`
|
||||
- `incident`
|
||||
|
||||
Suggested rule:
|
||||
|
||||
```text
|
||||
if incident_count > 0: incident
|
||||
elif anomaly_count > 0: anomaly
|
||||
elif observation_count > 0: observing
|
||||
else: idle
|
||||
```
|
||||
|
||||
This aligns well with the current Earth status language and keeps the visual mapping simple.
|
||||
|
||||
## Activity Score
|
||||
|
||||
The score should be a tunable heuristic, not a fixed truth model.
|
||||
|
||||
Recommended v1 formula:
|
||||
|
||||
```text
|
||||
activity_score =
|
||||
min(observation_count, 50) * 0.03
|
||||
+ anomaly_count * 1.2
|
||||
+ incident_count * 5.0
|
||||
```
|
||||
|
||||
Why cap observations:
|
||||
|
||||
- observation volume is usually much larger than anomaly or incident volume
|
||||
- uncapped observation counts would overwhelm the score
|
||||
- capped observation counts preserve baseline presence without drowning real abnormality
|
||||
|
||||
### Practical Guidance
|
||||
|
||||
- treat coefficients as configuration-like constants
|
||||
- expect to retune after looking at real data
|
||||
- keep `incident` weight dominant
|
||||
|
||||
## API Design
|
||||
|
||||
### 1. Summary/List API
|
||||
|
||||
Suggested endpoint:
|
||||
|
||||
- `/api/v1/bgp/regions/activity`
|
||||
|
||||
Response shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"window_minutes": 15,
|
||||
"regions": []
|
||||
}
|
||||
```
|
||||
|
||||
Use cases:
|
||||
|
||||
- BGP console summaries
|
||||
- right-side Earth stats
|
||||
- future region list panels
|
||||
|
||||
### 2. GeoJSON API
|
||||
|
||||
Suggested endpoint:
|
||||
|
||||
- `/api/v1/visualization/geo/bgp-regions`
|
||||
|
||||
Response shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "FeatureCollection",
|
||||
"features": []
|
||||
}
|
||||
```
|
||||
|
||||
Each feature should include:
|
||||
|
||||
- `geometry`
|
||||
- v1: `Point`
|
||||
- later: optional `Polygon`
|
||||
- `properties`
|
||||
- `region_key`
|
||||
- `region_name`
|
||||
- `status`
|
||||
- `activity_score`
|
||||
- `observation_count`
|
||||
- `anomaly_count`
|
||||
- `incident_count`
|
||||
- `affected_prefix_count`
|
||||
- `affected_asn_count`
|
||||
- `collector_count`
|
||||
|
||||
## Earth Rendering Plan
|
||||
|
||||
Detailed visual layering guidance is expanded in [bgp-earth-rendering-plan.md](/home/ray/dev/linkong/planet/docs/bgp-earth-rendering-plan.md).
|
||||
|
||||
### Layer Relationship
|
||||
|
||||
- `region layer` = ambient background activity
|
||||
- `incident marker` = focal event object
|
||||
|
||||
Do not replace incident markers with region markers.
|
||||
|
||||
### Region Visual Rules
|
||||
|
||||
Suggested mapping:
|
||||
|
||||
- `observing`
|
||||
- weak glow
|
||||
- low pulse or no pulse
|
||||
- `anomaly`
|
||||
- stronger glow
|
||||
- more visible pulse
|
||||
- `incident`
|
||||
- strongest regional emphasis
|
||||
- but still visually secondary to the incident marker itself
|
||||
|
||||
### Region Labels
|
||||
|
||||
Good v2 enhancement:
|
||||
|
||||
- show region name
|
||||
- show counts like `2 incidents / 5 anomalies`
|
||||
|
||||
This is useful, but should come after the core aggregation and Earth glow layer are working.
|
||||
|
||||
## Interaction Model
|
||||
|
||||
### Click Region
|
||||
|
||||
Recommended detail payload:
|
||||
|
||||
- region name
|
||||
- observation/anomaly/incident counts in the selected window
|
||||
- affected prefix count
|
||||
- affected ASN count
|
||||
- collector count
|
||||
- recent incidents in the region
|
||||
|
||||
### Click Incident
|
||||
|
||||
Keep the current incident-detail flow.
|
||||
|
||||
Interaction should feel hierarchical:
|
||||
|
||||
1. region gives situational context
|
||||
2. incident gives event focus
|
||||
|
||||
## MVP Implementation Order
|
||||
|
||||
### Step 1
|
||||
|
||||
Define static `REGIONS` in code or config.
|
||||
|
||||
### Step 2
|
||||
|
||||
Map geography-enriched BGP records into regions using the fallback chain.
|
||||
|
||||
### Step 3
|
||||
|
||||
Aggregate recent window counts:
|
||||
|
||||
- `observation_count`
|
||||
- `anomaly_count`
|
||||
- `incident_count`
|
||||
|
||||
### Step 4
|
||||
|
||||
Compute `activity_score` and `status`.
|
||||
|
||||
### Step 5
|
||||
|
||||
Expose:
|
||||
|
||||
- `/api/v1/bgp/regions/activity`
|
||||
- `/api/v1/visualization/geo/bgp-regions`
|
||||
|
||||
### Step 6
|
||||
|
||||
Render region glows on Earth behind incident markers.
|
||||
|
||||
## Out Of Scope For MVP
|
||||
|
||||
- persistent materialized region tables
|
||||
- geohash or H3 support
|
||||
- polygon-filled regional overlays
|
||||
- detailed top-prefix ranking in the first release
|
||||
- complicated scoring personalization
|
||||
|
||||
## Risks And Constraints
|
||||
|
||||
### Geography Quality
|
||||
|
||||
Prefix geography is approximate and incomplete.
|
||||
The region layer must tolerate fallback-based placement.
|
||||
|
||||
### Query Cost
|
||||
|
||||
Dynamic aggregation is the right v1 choice, but repeated short-window queries may eventually need:
|
||||
|
||||
- in-process caching
|
||||
- scheduled pre-aggregation
|
||||
- materialized summaries
|
||||
|
||||
### UI Overcrowding
|
||||
|
||||
If region glow, collector activity, and incidents all become too strong at once, Earth readability will regress.
|
||||
The region layer must remain supportive, not dominant.
|
||||
|
||||
## Final Recommendation
|
||||
|
||||
The current BGP roadmap should explicitly add:
|
||||
|
||||
- `region aggregation` as the concrete implementation of the missing `activity layer`
|
||||
|
||||
The recommended product interpretation is:
|
||||
|
||||
- `collectors` prove observation coverage
|
||||
- `regions` communicate live routing activity and abnormality
|
||||
- `incidents` remain the clearest high-confidence event objects
|
||||
|
||||
In one sentence:
|
||||
|
||||
`region aggregation is not a replacement for incidents; it is the situational background that makes sparse incidents feel legible on Earth.`
|
||||
@@ -1,207 +0,0 @@
|
||||
# collected_data 强耦合列拆除计划
|
||||
|
||||
## 背景
|
||||
|
||||
当前 `collected_data` 同时承担了两类职责:
|
||||
|
||||
1. 通用采集事实表
|
||||
2. 少数数据源的宽表字段承载
|
||||
|
||||
典型强耦合列包括:
|
||||
|
||||
- `country`
|
||||
- `city`
|
||||
- `latitude`
|
||||
- `longitude`
|
||||
- `value`
|
||||
- `unit`
|
||||
|
||||
以及 API 层临时平铺出来的:
|
||||
|
||||
- `cores`
|
||||
- `rmax`
|
||||
- `rpeak`
|
||||
- `power`
|
||||
|
||||
这些字段并不适合作为统一事实表的长期 schema。
|
||||
推荐方向是:
|
||||
|
||||
- 表内保留通用稳定字段
|
||||
- 业务差异字段全部归入 `metadata`
|
||||
- API 和前端动态读取 `metadata`
|
||||
|
||||
## 拆除目标
|
||||
|
||||
最终希望 `collected_data` 只保留:
|
||||
|
||||
- `id`
|
||||
- `snapshot_id`
|
||||
- `task_id`
|
||||
- `source`
|
||||
- `source_id`
|
||||
- `entity_key`
|
||||
- `data_type`
|
||||
- `name`
|
||||
- `title`
|
||||
- `description`
|
||||
- `metadata`
|
||||
- `collected_at`
|
||||
- `reference_date`
|
||||
- `is_valid`
|
||||
- `is_current`
|
||||
- `previous_record_id`
|
||||
- `change_type`
|
||||
- `change_summary`
|
||||
- `deleted_at`
|
||||
|
||||
## 计划阶段
|
||||
|
||||
### Phase 1:读取层去依赖
|
||||
|
||||
目标:
|
||||
|
||||
- API / 可视化 / 前端不再优先依赖宽列表字段
|
||||
- 所有动态字段优先从 `metadata` 取
|
||||
|
||||
当前已完成:
|
||||
|
||||
- 新写入数据时,将 `country/city/latitude/longitude/value/unit` 自动镜像到 `metadata`
|
||||
- `/api/v1/collected` 优先从 `metadata` 取动态字段
|
||||
- `visualization` 接口优先从 `metadata` 取动态字段
|
||||
- 国家筛选已改成只走 `metadata->>'country'`
|
||||
- `CollectedData.to_dict()` 已切到 metadata-first
|
||||
- 变更比较逻辑已切到 metadata-first
|
||||
- 已新增历史回填脚本:
|
||||
[scripts/backfill_collected_data_metadata.py](/home/ray/dev/linkong/planet/scripts/backfill_collected_data_metadata.py)
|
||||
- 已新增删列脚本:
|
||||
[scripts/drop_collected_data_legacy_columns.py](/home/ray/dev/linkong/planet/scripts/drop_collected_data_legacy_columns.py)
|
||||
|
||||
涉及文件:
|
||||
|
||||
- [backend/app/core/collected_data_fields.py](/home/ray/dev/linkong/planet/backend/app/core/collected_data_fields.py)
|
||||
- [backend/app/services/collectors/base.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/base.py)
|
||||
- [backend/app/api/v1/collected_data.py](/home/ray/dev/linkong/planet/backend/app/api/v1/collected_data.py)
|
||||
- [backend/app/api/v1/visualization.py](/home/ray/dev/linkong/planet/backend/app/api/v1/visualization.py)
|
||||
|
||||
### Phase 2:写入层去依赖
|
||||
|
||||
目标:
|
||||
|
||||
- 采集器内部不再把这些字段当作数据库一级列来理解
|
||||
- 统一只写:
|
||||
- 通用主字段
|
||||
- `metadata`
|
||||
|
||||
建议动作:
|
||||
|
||||
1. Collector 内部仍可使用 `country/city/value` 这种临时字段作为采集过程变量
|
||||
2. 进入 `BaseCollector._save_data()` 后统一归档到 `metadata`
|
||||
3. `CollectedData` 模型中的强耦合列已从 ORM 移除,写入统一归档到 `metadata`
|
||||
|
||||
### Phase 3:数据库删列
|
||||
|
||||
目标:
|
||||
|
||||
- 从 `collected_data` 真正移除以下列:
|
||||
- `country`
|
||||
- `city`
|
||||
- `latitude`
|
||||
- `longitude`
|
||||
- `value`
|
||||
- `unit`
|
||||
|
||||
注意:
|
||||
|
||||
- `cores / rmax / rpeak / power` 当前本来就在 `metadata` 里,不是表列
|
||||
- 这四个主要是 API 平铺字段,不需要数据库删列
|
||||
|
||||
## 当前阻塞点
|
||||
|
||||
在正式删列前,还需要确认这些地方已经完全不再直接依赖数据库列:
|
||||
|
||||
### 1. `CollectedData.to_dict()`
|
||||
|
||||
文件:
|
||||
|
||||
- [backend/app/models/collected_data.py](/home/ray/dev/linkong/planet/backend/app/models/collected_data.py)
|
||||
|
||||
状态:
|
||||
|
||||
- 已完成
|
||||
|
||||
### 2. 差异计算逻辑
|
||||
|
||||
文件:
|
||||
|
||||
- [backend/app/services/collectors/base.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/base.py)
|
||||
|
||||
状态:
|
||||
|
||||
- 已完成
|
||||
- 当前已改成比较归一化后的 metadata-first payload
|
||||
|
||||
### 3. 历史数据回填
|
||||
|
||||
问题:
|
||||
|
||||
- 老数据可能只有列值,没有对应 `metadata`
|
||||
|
||||
当前方案:
|
||||
|
||||
- 在删列前执行一次回填脚本:
|
||||
- [scripts/backfill_collected_data_metadata.py](/home/ray/dev/linkong/planet/scripts/backfill_collected_data_metadata.py)
|
||||
|
||||
### 4. 导出格式兼容
|
||||
|
||||
文件:
|
||||
|
||||
- [backend/app/api/v1/collected_data.py](/home/ray/dev/linkong/planet/backend/app/api/v1/collected_data.py)
|
||||
|
||||
现状:
|
||||
|
||||
- CSV/JSON 导出已基本切成 metadata-first
|
||||
|
||||
建议:
|
||||
|
||||
- 删列前再回归检查一次导出字段是否一致
|
||||
|
||||
## 推荐执行顺序
|
||||
|
||||
1. 保持新数据写入时 `metadata` 完整
|
||||
2. 把模型和 diff 逻辑完全切成 metadata-first
|
||||
3. 写一条历史回填脚本
|
||||
4. 回填后观察一轮
|
||||
5. 正式执行删列迁移
|
||||
|
||||
## 推荐迁移 SQL
|
||||
|
||||
仅在确认全部读取链路已去依赖后执行:
|
||||
|
||||
```sql
|
||||
ALTER TABLE collected_data
|
||||
DROP COLUMN IF EXISTS country,
|
||||
DROP COLUMN IF EXISTS city,
|
||||
DROP COLUMN IF EXISTS latitude,
|
||||
DROP COLUMN IF EXISTS longitude,
|
||||
DROP COLUMN IF EXISTS value,
|
||||
DROP COLUMN IF EXISTS unit;
|
||||
```
|
||||
|
||||
## 风险提示
|
||||
|
||||
1. 地图类接口对经纬度最敏感
|
||||
必须确保所有地图需要的记录,其 `metadata.latitude/longitude` 已回填完整。
|
||||
|
||||
2. 历史老数据如果没有回填,删列后会直接丢失这些信息。
|
||||
|
||||
3. 某些 collector 可能仍隐式依赖这些宽字段做差异比较,删列前必须做一次全量回归。
|
||||
|
||||
## 当前判断
|
||||
|
||||
当前项目已经完成“代码去依赖 + 历史回填 + readiness 检查”。
|
||||
下一步执行顺序建议固定为:
|
||||
|
||||
1. 先部署当前代码版本并重启后端
|
||||
2. 再做一轮功能回归
|
||||
3. 最后执行:
|
||||
`uv run python scripts/drop_collected_data_legacy_columns.py`
|
||||
@@ -1,402 +0,0 @@
|
||||
# 采集数据历史快照化改造方案
|
||||
|
||||
## 背景
|
||||
|
||||
当前系统的 `collected_data` 更接近“当前结果表”:
|
||||
|
||||
- 同一个 `source + source_id` 会被更新覆盖
|
||||
- 前端列表页默认读取这张表
|
||||
- `collection_tasks` 只记录任务执行状态,不直接承载数据版本语义
|
||||
|
||||
这套方式适合管理后台,但不利于后续做态势感知、时间回放、趋势分析和版本对比。
|
||||
如果后面需要回答下面这类问题,当前模型会比较吃力:
|
||||
|
||||
- 某条实体在过去 7 天如何变化
|
||||
- 某次采集相比上次新增了什么、删除了什么、值变了什么
|
||||
- 某个时刻地图上“当时的世界状态”是什么
|
||||
- 告警是在第几次采集后触发的
|
||||
|
||||
因此建议把采集数据改造成“历史快照 + 当前视图”模型。
|
||||
|
||||
## 目标
|
||||
|
||||
1. 每次触发采集都保留一份独立快照,历史可追溯。
|
||||
2. 管理后台默认仍然只看“当前最新状态”,不增加使用复杂度。
|
||||
3. 后续支持:
|
||||
- 时间线回放
|
||||
- 两次采集差异对比
|
||||
- 趋势分析
|
||||
- 按快照回溯告警和地图状态
|
||||
4. 尽量兼容现有接口,降低改造成本。
|
||||
|
||||
## 结论
|
||||
|
||||
不建议继续用以下两种单一模式:
|
||||
|
||||
- 直接覆盖旧数据
|
||||
问题:没有历史,无法回溯。
|
||||
|
||||
- 软删除旧数据再全量新增
|
||||
问题:语义不清,历史和“当前无效”混在一起,后续统计复杂。
|
||||
|
||||
推荐方案:
|
||||
|
||||
- 保留历史事实表
|
||||
- 维护当前视图
|
||||
- 每次采集对应一个明确的快照批次
|
||||
|
||||
## 推荐数据模型
|
||||
|
||||
### 方案概览
|
||||
|
||||
建议拆成三层:
|
||||
|
||||
1. `collection_tasks`
|
||||
继续作为采集任务表,表示“这次采集任务”。
|
||||
|
||||
2. `data_snapshots`
|
||||
新增快照表,表示“某个数据源在某次任务中产出的一个快照批次”。
|
||||
|
||||
3. `collected_data`
|
||||
从“当前结果表”升级为“历史事实表”,每一行归属于一个快照。
|
||||
|
||||
同时再提供一个“当前视图”:
|
||||
|
||||
- SQL View / 物化视图 / API 查询层封装均可
|
||||
- 语义是“每个 `source + source_id` 的最新有效记录”
|
||||
|
||||
### 新增表:`data_snapshots`
|
||||
|
||||
建议字段:
|
||||
|
||||
| 字段 | 类型 | 含义 |
|
||||
|---|---|---|
|
||||
| `id` | bigint PK | 快照主键 |
|
||||
| `datasource_id` | int | 对应数据源 |
|
||||
| `task_id` | int | 对应采集任务 |
|
||||
| `source` | varchar(100) | 数据源名,如 `top500` |
|
||||
| `snapshot_key` | varchar(100) | 可选,业务快照标识 |
|
||||
| `reference_date` | timestamptz nullable | 这批数据的参考时间 |
|
||||
| `started_at` | timestamptz | 快照开始时间 |
|
||||
| `completed_at` | timestamptz | 快照完成时间 |
|
||||
| `record_count` | int | 快照总记录数 |
|
||||
| `status` | varchar(20) | `running/success/failed/partial` |
|
||||
| `is_current` | bool | 当前是否是该数据源最新快照 |
|
||||
| `parent_snapshot_id` | bigint nullable | 上一版快照,可用于 diff |
|
||||
| `summary` | jsonb | 本次快照统计摘要 |
|
||||
|
||||
说明:
|
||||
|
||||
- `collection_tasks` 偏“执行过程”
|
||||
- `data_snapshots` 偏“数据版本”
|
||||
- 一个任务通常对应一个快照,但保留分层更清晰
|
||||
|
||||
### 升级表:`collected_data`
|
||||
|
||||
建议新增字段:
|
||||
|
||||
| 字段 | 类型 | 含义 |
|
||||
|---|---|---|
|
||||
| `snapshot_id` | bigint not null | 归属快照 |
|
||||
| `task_id` | int nullable | 归属任务,便于追查 |
|
||||
| `entity_key` | varchar(255) | 实体稳定键,通常可由 `source + source_id` 派生 |
|
||||
| `is_current` | bool | 当前是否为该实体最新记录 |
|
||||
| `previous_record_id` | bigint nullable | 上一个版本的记录 |
|
||||
| `change_type` | varchar(20) | `created/updated/unchanged/deleted` |
|
||||
| `change_summary` | jsonb | 字段变化摘要 |
|
||||
| `deleted_at` | timestamptz nullable | 对应“本次快照中消失”的实体 |
|
||||
|
||||
保留现有字段:
|
||||
|
||||
- `source`
|
||||
- `source_id`
|
||||
- `data_type`
|
||||
- `name`
|
||||
- `title`
|
||||
- `description`
|
||||
- `country`
|
||||
- `city`
|
||||
- `latitude`
|
||||
- `longitude`
|
||||
- `value`
|
||||
- `unit`
|
||||
- `metadata`
|
||||
- `collected_at`
|
||||
- `reference_date`
|
||||
- `is_valid`
|
||||
|
||||
### 当前视图
|
||||
|
||||
建议新增一个只读视图:
|
||||
|
||||
`current_collected_data`
|
||||
|
||||
语义:
|
||||
|
||||
- 对每个 `source + source_id` 只保留最新一条 `is_current = true` 且 `deleted_at is null` 的记录
|
||||
|
||||
这样:
|
||||
|
||||
- 管理后台继续像现在一样查“当前数据”
|
||||
- 历史分析查 `collected_data`
|
||||
|
||||
## 写入策略
|
||||
|
||||
### 触发按钮语义
|
||||
|
||||
“触发”不再理解为“覆盖旧表”,而是:
|
||||
|
||||
- 启动一次新的采集任务
|
||||
- 生成一个新的快照
|
||||
- 将本次结果写入历史事实表
|
||||
- 再更新当前视图标记
|
||||
|
||||
### 写入流程
|
||||
|
||||
1. 创建 `collection_tasks` 记录,状态 `running`
|
||||
2. 创建 `data_snapshots` 记录,状态 `running`
|
||||
3. 采集器拉取原始数据并标准化
|
||||
4. 为每条记录生成 `entity_key`
|
||||
- 推荐:`{source}:{source_id}`
|
||||
5. 将本次记录批量写入 `collected_data`
|
||||
6. 与上一个快照做比对,计算:
|
||||
- 新增
|
||||
- 更新
|
||||
- 未变
|
||||
- 删除
|
||||
7. 更新本批记录的:
|
||||
- `change_type`
|
||||
- `previous_record_id`
|
||||
- `is_current`
|
||||
8. 将上一批同实体记录的 `is_current` 置为 `false`
|
||||
9. 将本次快照未出现但上一版存在的实体标记为 `deleted`
|
||||
10. 更新 `data_snapshots.status = success`
|
||||
11. 更新 `collection_tasks.status = success`
|
||||
|
||||
### 删除语义
|
||||
|
||||
这里不建议真的删记录。
|
||||
建议采用“逻辑消失”模型:
|
||||
|
||||
- 历史行永远保留
|
||||
- 如果某实体在新快照里消失:
|
||||
- 上一条历史记录补一条“删除状态记录”或标记 `change_type = deleted`
|
||||
- 同时该实体不再出现在当前视图
|
||||
|
||||
这样最适合态势感知。
|
||||
|
||||
## API 改造建议
|
||||
|
||||
### 保持现有接口默认行为
|
||||
|
||||
现有接口:
|
||||
|
||||
- `GET /api/v1/collected`
|
||||
- `GET /api/v1/collected/{id}`
|
||||
- `GET /api/v1/collected/summary`
|
||||
|
||||
建议默认仍返回“当前视图”,避免前端全面重写。
|
||||
|
||||
### 新增历史查询能力
|
||||
|
||||
建议新增参数或新接口:
|
||||
|
||||
#### 1. 当前/历史切换
|
||||
|
||||
`GET /api/v1/collected?mode=current|history`
|
||||
|
||||
- `current`:默认,查当前视图
|
||||
- `history`:查历史事实表
|
||||
|
||||
#### 2. 按快照查询
|
||||
|
||||
`GET /api/v1/collected?snapshot_id=123`
|
||||
|
||||
#### 3. 快照列表
|
||||
|
||||
`GET /api/v1/snapshots`
|
||||
|
||||
支持筛选:
|
||||
|
||||
- `datasource_id`
|
||||
- `source`
|
||||
- `status`
|
||||
- `date_from/date_to`
|
||||
|
||||
#### 4. 快照详情
|
||||
|
||||
`GET /api/v1/snapshots/{id}`
|
||||
|
||||
返回:
|
||||
|
||||
- 快照基础信息
|
||||
- 统计摘要
|
||||
- 与上一版的 diff 摘要
|
||||
|
||||
#### 5. 快照 diff
|
||||
|
||||
`GET /api/v1/snapshots/{id}/diff?base_snapshot_id=122`
|
||||
|
||||
返回:
|
||||
|
||||
- `created`
|
||||
- `updated`
|
||||
- `deleted`
|
||||
- `unchanged`
|
||||
|
||||
## 前端改造建议
|
||||
|
||||
### 1. 数据列表页
|
||||
|
||||
默认仍看当前数据,不改用户使用习惯。
|
||||
|
||||
建议新增:
|
||||
|
||||
- “视图模式”
|
||||
- 当前数据
|
||||
- 历史数据
|
||||
- “快照时间”筛选
|
||||
- “只看变化项”筛选
|
||||
|
||||
### 2. 数据详情页
|
||||
|
||||
详情页建议展示:
|
||||
|
||||
- 当前记录基础信息
|
||||
- 元数据动态字段
|
||||
- 所属快照
|
||||
- 上一版本对比入口
|
||||
- 历史版本时间线
|
||||
|
||||
### 3. 数据源管理页
|
||||
|
||||
“触发”按钮文案建议改成更准确的:
|
||||
|
||||
- `立即采集`
|
||||
|
||||
并在详情里补:
|
||||
|
||||
- 最近一次快照时间
|
||||
- 最近一次快照记录数
|
||||
- 最近一次变化数
|
||||
|
||||
## 迁移方案
|
||||
|
||||
### Phase 1:兼容式落地
|
||||
|
||||
目标:先保留当前页面可用。
|
||||
|
||||
改动:
|
||||
|
||||
1. 新增 `data_snapshots`
|
||||
2. 给 `collected_data` 增加:
|
||||
- `snapshot_id`
|
||||
- `task_id`
|
||||
- `entity_key`
|
||||
- `is_current`
|
||||
- `previous_record_id`
|
||||
- `change_type`
|
||||
- `change_summary`
|
||||
- `deleted_at`
|
||||
3. 现有数据全部补成一个“初始化快照”
|
||||
4. 现有 `/collected` 默认改查当前视图
|
||||
|
||||
优点:
|
||||
|
||||
- 前端几乎无感
|
||||
- 风险最小
|
||||
|
||||
### Phase 2:启用差异计算
|
||||
|
||||
目标:采集后可知道本次改了什么。
|
||||
|
||||
改动:
|
||||
|
||||
1. 写入时做新旧快照比对
|
||||
2. 写 `change_type`
|
||||
3. 生成快照摘要
|
||||
|
||||
### Phase 3:前端态势感知能力
|
||||
|
||||
目标:支持历史回放和趋势分析。
|
||||
|
||||
改动:
|
||||
|
||||
1. 快照时间线
|
||||
2. 版本 diff 页面
|
||||
3. 地图时间回放
|
||||
4. 告警和快照关联
|
||||
|
||||
## 唯一性与索引建议
|
||||
|
||||
### 建议保留的业务唯一性
|
||||
|
||||
在“同一个快照内部”,建议唯一:
|
||||
|
||||
- `(snapshot_id, source, source_id)`
|
||||
|
||||
不要在整张历史表上强加:
|
||||
|
||||
- `(source, source_id)` 唯一
|
||||
|
||||
因为历史表本来就应该允许同一实体跨快照存在多条版本。
|
||||
|
||||
### 建议索引
|
||||
|
||||
- `idx_collected_data_snapshot_id`
|
||||
- `idx_collected_data_source_source_id`
|
||||
- `idx_collected_data_entity_key`
|
||||
- `idx_collected_data_is_current`
|
||||
- `idx_collected_data_reference_date`
|
||||
- `idx_snapshots_source_completed_at`
|
||||
|
||||
## 风险点
|
||||
|
||||
1. 存储量会明显增加
|
||||
- 需要评估保留周期
|
||||
- 可以考虑冷热分层
|
||||
|
||||
2. 写入复杂度上升
|
||||
- 需要批量 upsert / diff 逻辑
|
||||
|
||||
3. 当前接口语义会从“表”变成“视图”
|
||||
- 文档必须同步
|
||||
|
||||
4. 某些采集器缺稳定 `source_id`
|
||||
- 需要补齐实体稳定键策略
|
||||
|
||||
## 对当前项目的具体建议
|
||||
|
||||
结合当前代码,推荐这样落地:
|
||||
|
||||
### 短期
|
||||
|
||||
1. 先设计并落表:
|
||||
- `data_snapshots`
|
||||
- `collected_data` 新字段
|
||||
2. 采集完成后每次新增快照
|
||||
3. `/api/v1/collected` 默认查 `is_current = true`
|
||||
|
||||
### 中期
|
||||
|
||||
1. 在 `BaseCollector._save_data()` 中改成:
|
||||
- 生成快照
|
||||
- 批量写历史
|
||||
- 标记当前
|
||||
2. 将 `CollectionTask.id` 关联到 `snapshot.task_id`
|
||||
|
||||
### 长期
|
||||
|
||||
1. 地图接口支持按 `snapshot_id` 查询
|
||||
2. 仪表盘支持“最近一次快照变化量”
|
||||
3. 告警支持绑定到快照版本
|
||||
|
||||
## 最终建议
|
||||
|
||||
最终建议采用:
|
||||
|
||||
- 历史事实表:保存每次采集结果
|
||||
- 当前视图:服务管理后台默认查询
|
||||
- 快照表:承载版本批次和 diff 语义
|
||||
|
||||
这样既能保留历史,又不会把当前页面全部推翻重做,是最适合后续做态势感知的一条路径。
|
||||
@@ -1,263 +0,0 @@
|
||||
# 数据采集系统 (Collectors)
|
||||
|
||||
## 一、系统架构
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ 数据采集系统架构 │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
|
||||
│ │ TOP500 │ │ Epoch AI │ │ HuggingFace │ │
|
||||
│ │ 采集器 │ │ 采集器 │ │ 采集器 │ │
|
||||
│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │
|
||||
│ │ │ │ │
|
||||
│ └───────────────────┼───────────────────┘ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────┐ │
|
||||
│ │ BaseCollector │◄── 基类 (统一处理) │
|
||||
│ │ run() 方法 │ │
|
||||
│ └─────────┬───────────┘ │
|
||||
│ │ │
|
||||
│ ┌─────────────────┼─────────────────┐ │
|
||||
│ ▼ ▼ ▼ │
|
||||
│ ┌───────────┐ ┌───────────┐ ┌───────────┐ │
|
||||
│ │ fetch() │ │transform()│ │ _save_data│ │
|
||||
│ │ 获取原始数据 │ │ 数据转换 │ │ 保存到DB │ │
|
||||
│ └───────────┘ └───────────┘ └───────────┘ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────┐ │
|
||||
│ │ CollectedData 表 │◄── 统一存储 │
|
||||
│ └─────────────────────┘ │
|
||||
│ │
|
||||
│ ┌─────────────────────────────────────────────────────────┐ │
|
||||
│ │ Scheduler (APScheduler) │ │
|
||||
│ │ 定时任务调度: 每4小时/6小时/12小时/1天 自动执行 │ │
|
||||
│ └─────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## 二、工作流程 (Pipeline)
|
||||
|
||||
```python
|
||||
# 1. Scheduler 触发 (定时 或 手动触发)
|
||||
# ↓
|
||||
|
||||
# 2. run() 方法执行完整流水线
|
||||
async def run(self, db):
|
||||
# 2.1 检查采集器是否启用
|
||||
if not collector_registry.is_active(self.name):
|
||||
return {"status": "skipped"}
|
||||
|
||||
# 2.2 记录任务开始
|
||||
task = CollectionTask(status="running")
|
||||
db.add(task)
|
||||
await db.commit()
|
||||
|
||||
# 2.3 FETCH - 获取原始数据 (由子类实现)
|
||||
raw_data = await self.fetch()
|
||||
|
||||
# 2.4 TRANSFORM - 转换为统一格式
|
||||
data = self.transform(raw_data)
|
||||
|
||||
# 2.5 SAVE - 保存到数据库
|
||||
records_count = await self._save_data(db, data)
|
||||
|
||||
# 2.6 记录任务完成
|
||||
task.status = "success"
|
||||
task.records_processed = records_count
|
||||
await db.commit()
|
||||
```
|
||||
|
||||
**核心文件**: `backend/app/services/collectors/base.py`
|
||||
|
||||
## 三、采集器列表
|
||||
|
||||
| 采集器 | 数据类型 | 数据内容 | 采集频率 |
|
||||
|--------|----------|----------|----------|
|
||||
| TOP500 | supercomputer | 全球超级计算机排名 (算力、性能) | 4小时 |
|
||||
| Epoch AI | gpu_cluster | GPU算力集群信息 | 6小时 |
|
||||
| HuggingFace Models | model | AI模型信息 | 12小时 |
|
||||
| HuggingFace Datasets | dataset | 数据集信息 | 12小时 |
|
||||
| HuggingFace Spaces | space | Demo应用 | 1天 |
|
||||
| PeeringDB | ixp/network/facility | 互联网交换点/网络/机房 | 1-2天 |
|
||||
| TeleGeography | submarine_cable | 海底光缆信息 | 7天 |
|
||||
|
||||
## 四、数据格式 (统一存储到 CollectedData 表)
|
||||
|
||||
```python
|
||||
# 每个采集器 parse_response() 返回格式
|
||||
{
|
||||
"source_id": "top500_1", # 原始系统ID (必填)
|
||||
"name": "El Capitan", # 名称 (必填)
|
||||
"description": "系统描述...", # 描述
|
||||
"country": "United States", # 国家
|
||||
"city": "Livermore, CA", # 城市
|
||||
"latitude": "37.6819", # 纬度 (字符串)
|
||||
"longitude": "-121.7681", # 经度 (字符串)
|
||||
"value": "1742.00", # 性能值 (如算力)
|
||||
"unit": "PFlop/s", # 单位
|
||||
"metadata": { # 额外数据 (JSON)
|
||||
"rank": 1,
|
||||
"r_peak": 2746.38,
|
||||
"cores": 11039616
|
||||
},
|
||||
"reference_date": "2025-11-01" # 数据参考日期
|
||||
}
|
||||
```
|
||||
|
||||
## 五、数据库表结构
|
||||
|
||||
**CollectedData 表** (`collected_data`)
|
||||
|
||||
| 字段 | 类型 | 说明 |
|
||||
|------|------|------|
|
||||
| id | SERIAL | 主键 |
|
||||
| source | VARCHAR(100) | 数据源名称 (top500, huggingface等) |
|
||||
| source_id | VARCHAR(100) | 原始数据ID |
|
||||
| data_type | VARCHAR(50) | 数据类型 (supercomputer, model等) |
|
||||
| name | VARCHAR(500) | 名称 |
|
||||
| title | VARCHAR(500) | 标题 |
|
||||
| description | TEXT | 描述 |
|
||||
| country | VARCHAR(100) | 国家 |
|
||||
| city | VARCHAR(100) | 城市 |
|
||||
| latitude | VARCHAR(50) | 纬度 |
|
||||
| longitude | VARCHAR(50) | 经度 |
|
||||
| value | VARCHAR(100) | 性能值 |
|
||||
| unit | VARCHAR(20) | 单位 |
|
||||
| metadata | JSONB | 额外元数据 |
|
||||
| collected_at | TIMESTAMP | 采集时间 |
|
||||
| reference_date | TIMESTAMP | 数据参考日期 |
|
||||
| is_valid | INTEGER | 是否有效 |
|
||||
|
||||
**核心文件**: `backend/app/models/collected_data.py`
|
||||
|
||||
## 六、TOP500 采集器示例 (完整流程)
|
||||
|
||||
```python
|
||||
# 1. fetch() - 从网页获取HTML
|
||||
async def fetch(self):
|
||||
url = "https://top500.org/lists/top500/list/2025/11/"
|
||||
response = await client.get(url)
|
||||
return response.text # 返回HTML
|
||||
|
||||
# 2. parse_response() - 解析HTML为统一格式
|
||||
def parse_response(self, html):
|
||||
soup = BeautifulSoup(html, "html.parser")
|
||||
table = soup.find("table")
|
||||
|
||||
for row in table.find_all("tr")[1:]: # 跳过表头
|
||||
cells = row.find_all("td")
|
||||
|
||||
entry = {
|
||||
"source_id": f"top500_{cells[0].text}", # "top500_1"
|
||||
"name": cells[1].text.strip(), # "El Capitan"
|
||||
"country": cells[2].text.strip(), # "United States"
|
||||
"city": "", # 城市
|
||||
"latitude": "", # 需进一步解析
|
||||
"longitude": "",
|
||||
"value": "1742.00", # Rmax
|
||||
"unit": "PFlop/s",
|
||||
"metadata": {
|
||||
"rank": 1,
|
||||
"cores": "11340000"
|
||||
},
|
||||
"reference_date": "2025-11-01"
|
||||
}
|
||||
data.append(entry)
|
||||
|
||||
return data
|
||||
|
||||
# 3. run() 自动调用 _save_data() 保存到数据库
|
||||
```
|
||||
|
||||
**核心文件**: `backend/app/services/collectors/top500.py`
|
||||
|
||||
## 七、调度机制
|
||||
|
||||
```python
|
||||
# 启动时注册所有采集器到定时任务
|
||||
def start_scheduler():
|
||||
for name, collector in collectors.items():
|
||||
if collector_registry.is_active(name):
|
||||
scheduler.add_job(
|
||||
run_collector_task,
|
||||
trigger=IntervalTrigger(hours=collector.frequency_hours),
|
||||
id=name,
|
||||
name=name
|
||||
)
|
||||
```
|
||||
|
||||
| 采集器 | 采集频率 |
|
||||
|--------|----------|
|
||||
| TOP500 | 每4小时 |
|
||||
| Epoch AI | 每6小时 |
|
||||
| HuggingFace | 每12小时 |
|
||||
| PeeringDB | 每1-2天 |
|
||||
| TeleGeography | 每7天 |
|
||||
|
||||
**核心文件**: `backend/app/services/scheduler.py`
|
||||
|
||||
## 八、相关代码文件
|
||||
|
||||
```
|
||||
backend/app/services/collectors/
|
||||
├── base.py # 基类: run() 流水线, _save_data() 保存
|
||||
├── registry.py # 采集器注册表
|
||||
├── scheduler.py # 定时任务调度 (APScheduler)
|
||||
├── top500.py # TOP500采集器
|
||||
├── epoch_ai.py # Epoch AI采集器
|
||||
├── huggingface.py # HuggingFace采集器
|
||||
├── peeringdb.py # PeeringDB采集器
|
||||
└── telegeraphy.py # TeleGeography海底光缆采集器
|
||||
|
||||
backend/app/models/
|
||||
└── collected_data.py # 统一数据模型
|
||||
```
|
||||
|
||||
## 九、数据使用场景
|
||||
|
||||
采集的数据最终会:
|
||||
|
||||
1. **可视化展示** - 在UE5大屏上显示超级计算机、GPU集群、海底光缆的地理位置
|
||||
2. **态势分析** - 统计全球算力分布、增长趋势
|
||||
3. **告警系统** - 检测重要节点变化
|
||||
|
||||
## 十、采集器注册机制
|
||||
|
||||
采集器在应用启动时自动注册:
|
||||
|
||||
```python
|
||||
# backend/app/services/collectors/__init__.py
|
||||
|
||||
collector_registry.register(TOP500Collector())
|
||||
collector_registry.register(EpochAIGPUCollector())
|
||||
collector_registry.register(HuggingFaceModelCollector())
|
||||
collector_registry.register(HuggingFaceDatasetCollector())
|
||||
collector_registry.register(HuggingFaceSpacesCollector())
|
||||
collector_registry.register(PeeringDBIXPCollector())
|
||||
collector_registry.register(PeeringDBNetworkCollector())
|
||||
collector_registry.register(PeeringDBFacilityCollector())
|
||||
collector_registry.register(TeleGeographyCableCollector())
|
||||
collector_registry.register(TeleGeographyLandingPointCollector())
|
||||
collector_registry.register(TeleGeographyCableSystemCollector())
|
||||
```
|
||||
|
||||
**核心文件**: `backend/app/services/collectors/registry.py`
|
||||
|
||||
## 十一、触发采集
|
||||
|
||||
### 方式一:定时触发
|
||||
系统启动时,APScheduler会自动根据各采集器的`frequency_hours`设置定时任务。
|
||||
|
||||
### 方式二:手动触发 API
|
||||
|
||||
```bash
|
||||
# 触发TOP500采集
|
||||
curl -X POST http://localhost:8000/api/v1/datasources/1/trigger \
|
||||
-H "Authorization: Bearer <token>"
|
||||
```
|
||||
|
||||
**核心文件**: `backend/app/api/v1/datasources.py`
|
||||
@@ -1,486 +0,0 @@
|
||||
# Datasource Health Plan
|
||||
|
||||
## Overview
|
||||
|
||||
This document defines a phased plan for datasource health governance.
|
||||
|
||||
The goal is to make collectors observable, diagnosable, and recoverable when upstream APIs change, while avoiding unsafe automatic mutation of repository defaults.
|
||||
|
||||
The key principle is:
|
||||
|
||||
- do not let runtime automation rewrite repository default config
|
||||
|
||||
Instead, split responsibilities across:
|
||||
|
||||
- default config
|
||||
- runtime overrides
|
||||
- health check records
|
||||
- agent-generated repair proposals
|
||||
|
||||
|
||||
## Problem Statement
|
||||
|
||||
Collectors currently depend on third-party APIs, data downloads, mirrored JSON files, archive links, and web pages.
|
||||
|
||||
These upstream dependencies can fail in several ways:
|
||||
|
||||
- endpoint becomes unreachable
|
||||
- endpoint still responds but schema changes
|
||||
- content-type changes
|
||||
- website shuts down or moves
|
||||
- mirror link disappears
|
||||
- HTML structure changes and scraping fails
|
||||
- endpoint requires a new path or new host
|
||||
|
||||
We want a system that can:
|
||||
|
||||
- detect datasource health degradation early
|
||||
- identify likely cause
|
||||
- search for updated endpoints when reasonable
|
||||
- apply safe runtime fixes without polluting default repo config
|
||||
- preserve auditability and rollback
|
||||
|
||||
|
||||
## Design Principles
|
||||
|
||||
1. Default config is stable
|
||||
|
||||
- `backend/app/core/data_sources.yaml` remains the repository baseline.
|
||||
- It should be changed intentionally through normal development flow, not by autonomous runtime agents.
|
||||
|
||||
2. Runtime fixes are isolated
|
||||
|
||||
- Emergency or adaptive fixes should live in a runtime override layer.
|
||||
- Overrides should be reversible and auditable.
|
||||
|
||||
3. Deterministic checks come first
|
||||
|
||||
- Use normal programmatic health checks before using LLMs.
|
||||
- Only call an agent when deterministic checks indicate a meaningful failure.
|
||||
|
||||
4. Agents suggest before they mutate
|
||||
|
||||
- Agents should produce proposals with evidence and confidence.
|
||||
- Application of a proposal should be controlled by policy.
|
||||
|
||||
5. Every repair is attributable
|
||||
|
||||
- Store what changed, why, who or what suggested it, and when it was applied.
|
||||
|
||||
|
||||
## Configuration Layers
|
||||
|
||||
Recommended runtime precedence:
|
||||
|
||||
1. datasource endpoint override
|
||||
2. datasource DB endpoint override
|
||||
3. repository default YAML
|
||||
4. collector internal fallback logic
|
||||
|
||||
Definitions:
|
||||
|
||||
- repository default YAML:
|
||||
- `backend/app/core/data_sources.yaml`
|
||||
- versioned baseline
|
||||
- datasource DB endpoint override:
|
||||
- existing `DataSourceConfig.endpoint`
|
||||
- current runtime override entrypoint
|
||||
- datasource endpoint override:
|
||||
- a dedicated new override table
|
||||
- used for health-repair and proposal application
|
||||
- collector internal fallback logic:
|
||||
- final defensive fallback
|
||||
- should be minimized over time
|
||||
|
||||
|
||||
## Recommended Architecture
|
||||
|
||||
### 1. Deterministic Health Checks
|
||||
|
||||
Each collector gets a health profile with checks such as:
|
||||
|
||||
- endpoint resolves
|
||||
- HTTP request succeeds
|
||||
- status code is acceptable
|
||||
- content-type is expected
|
||||
- body parses successfully
|
||||
- minimum structural fields exist
|
||||
- sample item count is plausible
|
||||
- latency is within threshold
|
||||
|
||||
Output states:
|
||||
|
||||
- `healthy`
|
||||
- `degraded`
|
||||
- `failed`
|
||||
- `schema_changed`
|
||||
- `rate_limited`
|
||||
- `auth_required`
|
||||
|
||||
|
||||
### 2. Agent-Assisted Repair Discovery
|
||||
|
||||
Only triggered when deterministic health checks fail or return suspicious structure.
|
||||
|
||||
Agent responsibilities:
|
||||
|
||||
- search for current official endpoint or replacement path
|
||||
- inspect likely upstream documentation or landing pages
|
||||
- compare candidate endpoint output to collector expectations
|
||||
- produce a repair proposal with confidence and evidence
|
||||
|
||||
Agent should not directly modify repository defaults.
|
||||
|
||||
|
||||
### 3. Safe Runtime Repair Application
|
||||
|
||||
Repair proposals can be:
|
||||
|
||||
- reviewed manually
|
||||
- auto-applied only under strict low-risk policy
|
||||
|
||||
Auto-apply should be limited to cases like:
|
||||
|
||||
- same trusted domain
|
||||
- highly similar response structure
|
||||
- repeated successful verification
|
||||
- confidence above threshold
|
||||
|
||||
|
||||
## Phased Delivery Plan
|
||||
|
||||
## Phase 1: Deterministic Health MVP
|
||||
|
||||
Goal:
|
||||
|
||||
- build health observability without automated repair
|
||||
|
||||
Scope:
|
||||
|
||||
- datasource health check task runner
|
||||
- datasource health result persistence
|
||||
- endpoint reachability + parse checks
|
||||
- dashboard or API visibility into health status
|
||||
|
||||
Deliverables:
|
||||
|
||||
- health check service
|
||||
- health check record table
|
||||
- status endpoint
|
||||
- scheduled or manual check trigger
|
||||
|
||||
No agent usage yet.
|
||||
|
||||
|
||||
## Phase 2: Agent Repair Proposals
|
||||
|
||||
Goal:
|
||||
|
||||
- let agent investigate failing sources and propose updated endpoints
|
||||
|
||||
Scope:
|
||||
|
||||
- invoke agent only when datasource health is `failed` or `schema_changed`
|
||||
- web search + page inspection
|
||||
- candidate endpoint extraction
|
||||
- proposal persistence
|
||||
|
||||
Deliverables:
|
||||
|
||||
- repair proposal schema
|
||||
- proposal generation pipeline
|
||||
- confidence and evidence model
|
||||
- operator review view or API
|
||||
|
||||
Still no automatic config mutation.
|
||||
|
||||
|
||||
## Phase 3: Runtime Overrides
|
||||
|
||||
Goal:
|
||||
|
||||
- allow approved proposals to take effect safely at runtime
|
||||
|
||||
Scope:
|
||||
|
||||
- add dedicated override storage
|
||||
- runtime resolution prefers override over default config
|
||||
- proposal application writes override only
|
||||
|
||||
Deliverables:
|
||||
|
||||
- endpoint override table
|
||||
- override-aware resolution logic
|
||||
- apply/reject endpoints
|
||||
- rollback endpoint
|
||||
|
||||
Repository default YAML remains untouched.
|
||||
|
||||
|
||||
## Phase 4: Limited Auto-Apply
|
||||
|
||||
Goal:
|
||||
|
||||
- safely automate a narrow slice of low-risk repairs
|
||||
|
||||
Scope:
|
||||
|
||||
- policy engine for auto-apply
|
||||
- same-domain or trusted-domain checks
|
||||
- structure validation
|
||||
- staged verification after apply
|
||||
|
||||
Deliverables:
|
||||
|
||||
- auto-apply rules
|
||||
- audit logs
|
||||
- automatic post-apply health verification
|
||||
- auto-disable or rollback on regression
|
||||
|
||||
|
||||
## Data Model Draft
|
||||
|
||||
### datasource_health_checks
|
||||
|
||||
Purpose:
|
||||
|
||||
- store each health evaluation result
|
||||
|
||||
Suggested fields:
|
||||
|
||||
- `id`
|
||||
- `datasource_id`
|
||||
- `collector_name`
|
||||
- `endpoint_checked`
|
||||
- `status`
|
||||
- `http_status`
|
||||
- `content_type`
|
||||
- `latency_ms`
|
||||
- `sample_count`
|
||||
- `error_message`
|
||||
- `details`
|
||||
- `checked_at`
|
||||
|
||||
`details` can store structured diagnostic data such as:
|
||||
|
||||
- parsed fields
|
||||
- schema mismatch summary
|
||||
- retry count
|
||||
- exception class
|
||||
|
||||
|
||||
### datasource_repair_proposals
|
||||
|
||||
Purpose:
|
||||
|
||||
- store agent-generated repair suggestions
|
||||
|
||||
Suggested fields:
|
||||
|
||||
- `id`
|
||||
- `datasource_id`
|
||||
- `collector_name`
|
||||
- `old_endpoint`
|
||||
- `candidate_endpoint`
|
||||
- `reason`
|
||||
- `confidence`
|
||||
- `evidence_urls`
|
||||
- `evidence_summary`
|
||||
- `status`
|
||||
- `created_by`
|
||||
- `created_at`
|
||||
- `reviewed_at`
|
||||
|
||||
Suggested `status` values:
|
||||
|
||||
- `proposed`
|
||||
- `approved`
|
||||
- `rejected`
|
||||
- `applied`
|
||||
- `expired`
|
||||
|
||||
|
||||
### datasource_endpoint_overrides
|
||||
|
||||
Purpose:
|
||||
|
||||
- runtime endpoint override layer
|
||||
|
||||
Suggested fields:
|
||||
|
||||
- `id`
|
||||
- `datasource_id`
|
||||
- `collector_name`
|
||||
- `endpoint`
|
||||
- `reason`
|
||||
- `source`
|
||||
- `proposal_id`
|
||||
- `enabled`
|
||||
- `created_at`
|
||||
- `updated_at`
|
||||
|
||||
Suggested `source` values:
|
||||
|
||||
- `manual`
|
||||
- `health-agent`
|
||||
- `migration`
|
||||
|
||||
|
||||
## API Draft
|
||||
|
||||
### Health
|
||||
|
||||
- `GET /api/v1/datasources/health`
|
||||
- `GET /api/v1/datasources/{id}/health`
|
||||
- `POST /api/v1/datasources/{id}/health-check`
|
||||
- `POST /api/v1/datasources/health-check-all`
|
||||
|
||||
### Repair proposals
|
||||
|
||||
- `GET /api/v1/datasources/{id}/repair-proposals`
|
||||
- `POST /api/v1/datasources/{id}/repair-proposals/generate`
|
||||
- `POST /api/v1/datasources/{id}/repair-proposals/{proposal_id}/approve`
|
||||
- `POST /api/v1/datasources/{id}/repair-proposals/{proposal_id}/reject`
|
||||
- `POST /api/v1/datasources/{id}/repair-proposals/{proposal_id}/apply`
|
||||
|
||||
### Overrides
|
||||
|
||||
- `GET /api/v1/datasources/{id}/overrides`
|
||||
- `POST /api/v1/datasources/{id}/overrides`
|
||||
- `PUT /api/v1/datasources/{id}/overrides/{override_id}`
|
||||
- `DELETE /api/v1/datasources/{id}/overrides/{override_id}`
|
||||
|
||||
|
||||
## Agent Contract Draft
|
||||
|
||||
When deterministic health fails, the agent should receive:
|
||||
|
||||
- datasource name
|
||||
- collector name
|
||||
- current endpoint
|
||||
- current failure mode
|
||||
- expected response shape summary
|
||||
- known trusted domains
|
||||
|
||||
Expected output:
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "proposal",
|
||||
"candidate_endpoint": "https://example.com/api/v2/data",
|
||||
"confidence": 0.86,
|
||||
"reason": "Official docs now point to v2 endpoint",
|
||||
"evidence_urls": [
|
||||
"https://example.com/docs/api",
|
||||
"https://example.com/changelog"
|
||||
],
|
||||
"notes": "Response shape appears compatible after light field remapping"
|
||||
}
|
||||
```
|
||||
|
||||
The agent should never output "rewrite the default yaml" as its primary action.
|
||||
|
||||
|
||||
## Risk Analysis
|
||||
|
||||
### Risk: wrong endpoint chosen by agent
|
||||
|
||||
Mitigation:
|
||||
|
||||
- use trusted-domain allowlists
|
||||
- require evidence URLs
|
||||
- require confidence threshold
|
||||
- add manual review for medium-risk sources
|
||||
|
||||
|
||||
### Risk: endpoint responds but schema silently changed
|
||||
|
||||
Mitigation:
|
||||
|
||||
- deterministic schema checks
|
||||
- parse and sample validation
|
||||
- content-type checks
|
||||
- collector-specific required fields
|
||||
|
||||
|
||||
### Risk: automatic runtime override causes hidden drift
|
||||
|
||||
Mitigation:
|
||||
|
||||
- store all overrides explicitly
|
||||
- mark source of override
|
||||
- keep default YAML unchanged
|
||||
- expose active overrides in API/UI
|
||||
|
||||
|
||||
### Risk: persistent bad override breaks data collection
|
||||
|
||||
Mitigation:
|
||||
|
||||
- allow rollback
|
||||
- keep parent/default endpoint visible
|
||||
- re-run verification after apply
|
||||
- auto-disable override on repeated failure
|
||||
|
||||
|
||||
## Operational Policy Recommendations
|
||||
|
||||
1. Do not auto-apply for high-value or high-fragility sources initially.
|
||||
|
||||
2. Use manual approval for:
|
||||
|
||||
- scraped HTML sources
|
||||
- unofficial mirrors
|
||||
- sources with auth or rate-limit complexity
|
||||
- sources with legal or trust ambiguity
|
||||
|
||||
3. Allow auto-apply only for:
|
||||
|
||||
- same-domain version bumps
|
||||
- obvious official migration paths
|
||||
- repeated passing verification
|
||||
|
||||
4. Expose health + proposal + override state together in one operator view.
|
||||
|
||||
|
||||
## Suggested Implementation Order
|
||||
|
||||
1. Phase 1
|
||||
- health result table
|
||||
- deterministic checks
|
||||
- API and UI visibility
|
||||
|
||||
2. Phase 2
|
||||
- proposal table
|
||||
- agent prompt/output contract
|
||||
- proposal generation job
|
||||
|
||||
3. Phase 3
|
||||
- runtime override table
|
||||
- resolver precedence update
|
||||
- apply/reject endpoints
|
||||
|
||||
4. Phase 4
|
||||
- auto-apply rules
|
||||
- rollback policy
|
||||
- operator automation
|
||||
|
||||
|
||||
## Out Of Scope For The First Iteration
|
||||
|
||||
- direct automatic mutation of repository default YAML
|
||||
- automatic git commits by repair agents
|
||||
- unrestricted autonomous endpoint replacement
|
||||
- fully generalized schema remapping engine
|
||||
|
||||
|
||||
## Recommended First Milestone
|
||||
|
||||
The first milestone should be:
|
||||
|
||||
- deterministic datasource health checks
|
||||
- persisted results
|
||||
- manual visibility
|
||||
- no automatic repair
|
||||
|
||||
This gives immediate operational value with low risk, and prepares clean inputs for the later agent phase.
|
||||
@@ -1,478 +0,0 @@
|
||||
# Datasource Health Stage 2 Tasks
|
||||
|
||||
## Goal
|
||||
|
||||
Stage 2 focuses on the first practical operational layer:
|
||||
|
||||
- deterministic datasource health checks
|
||||
- persisted health results
|
||||
- health visibility through API and UI
|
||||
- no agent-assisted repair yet
|
||||
|
||||
This stage should make Planet capable of answering:
|
||||
|
||||
- which collectors are healthy
|
||||
- which collectors are degraded
|
||||
- which collectors are failing
|
||||
- why they are failing at a basic deterministic level
|
||||
|
||||
|
||||
## Scope
|
||||
|
||||
Included:
|
||||
|
||||
- datasource health data model
|
||||
- deterministic health check service
|
||||
- manual and scheduled health check triggers
|
||||
- health result APIs
|
||||
- frontend visibility
|
||||
|
||||
Excluded:
|
||||
|
||||
- LLM reasoning
|
||||
- web-search-based repair proposals
|
||||
- automatic endpoint rewriting
|
||||
- runtime override application
|
||||
|
||||
|
||||
## Delivery Target
|
||||
|
||||
At the end of Stage 2, an operator should be able to:
|
||||
|
||||
1. see health status for each collector
|
||||
2. trigger a health check manually
|
||||
3. inspect the latest failure reason
|
||||
4. inspect the last checked endpoint
|
||||
5. understand whether the problem is:
|
||||
- unreachable
|
||||
- auth-related
|
||||
- rate-limit-related
|
||||
- schema-related
|
||||
- empty-data-related
|
||||
|
||||
|
||||
## Work Breakdown
|
||||
|
||||
## A. Data Model
|
||||
|
||||
### A1. Add datasource health record table
|
||||
|
||||
Create a new model, for example:
|
||||
|
||||
- `backend/app/models/datasource_health_check.py`
|
||||
|
||||
Suggested fields:
|
||||
|
||||
- `id`
|
||||
- `datasource_id`
|
||||
- `collector_name`
|
||||
- `endpoint_checked`
|
||||
- `status`
|
||||
- `http_status`
|
||||
- `content_type`
|
||||
- `latency_ms`
|
||||
- `sample_count`
|
||||
- `error_message`
|
||||
- `details`
|
||||
- `checked_at`
|
||||
|
||||
Suggested status enum values:
|
||||
|
||||
- `healthy`
|
||||
- `degraded`
|
||||
- `failed`
|
||||
- `schema_changed`
|
||||
- `rate_limited`
|
||||
- `auth_required`
|
||||
- `empty_result`
|
||||
|
||||
|
||||
### A2. Add datasource health summary fields
|
||||
|
||||
Option A:
|
||||
|
||||
- keep summary only in the health check table
|
||||
|
||||
Option B:
|
||||
|
||||
- also add summary fields on `data_sources`
|
||||
|
||||
Recommended first step:
|
||||
|
||||
- do not mutate `data_sources` schema yet
|
||||
- derive summary from the latest health record
|
||||
|
||||
|
||||
### A3. Migration task
|
||||
|
||||
Add migration for the health table.
|
||||
|
||||
Deliverables:
|
||||
|
||||
- migration file
|
||||
- model registration
|
||||
|
||||
|
||||
## B. Health Check Engine
|
||||
|
||||
### B1. Define health check service
|
||||
|
||||
Add a new service module, for example:
|
||||
|
||||
- `backend/app/services/datasource_health.py`
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- resolve effective endpoint
|
||||
- execute deterministic check
|
||||
- classify result
|
||||
- persist health record
|
||||
|
||||
|
||||
### B2. Define shared result schema
|
||||
|
||||
Create a typed result object, for example:
|
||||
|
||||
- `HealthCheckResult`
|
||||
|
||||
Suggested fields:
|
||||
|
||||
- `status`
|
||||
- `endpoint_checked`
|
||||
- `http_status`
|
||||
- `content_type`
|
||||
- `latency_ms`
|
||||
- `sample_count`
|
||||
- `error_message`
|
||||
- `details`
|
||||
|
||||
|
||||
### B3. Implement base deterministic checks
|
||||
|
||||
Every datasource should go through a minimal baseline check:
|
||||
|
||||
1. resolve endpoint
|
||||
2. perform request
|
||||
3. measure latency
|
||||
4. inspect status code
|
||||
5. inspect content type
|
||||
6. inspect body shape
|
||||
|
||||
Classification rules:
|
||||
|
||||
- network error -> `failed`
|
||||
- HTTP 401/403 -> `auth_required`
|
||||
- HTTP 429 -> `rate_limited`
|
||||
- HTTP 404/410 -> `failed`
|
||||
- parse failure -> `schema_changed`
|
||||
- zero or suspiciously empty results -> `empty_result` or `degraded`
|
||||
- valid parse -> `healthy`
|
||||
|
||||
|
||||
### B4. Add collector-aware adapters
|
||||
|
||||
Some collectors do not use the same fetch semantics.
|
||||
|
||||
Add adapter profiles such as:
|
||||
|
||||
- `http_json`
|
||||
- `http_csv`
|
||||
- `html_scrape`
|
||||
- `stream_probe`
|
||||
- `auth_session_http`
|
||||
|
||||
Initial mapping suggestion:
|
||||
|
||||
- `huggingface`, `peeringdb`, `cloudflare` -> `http_json`
|
||||
- `fao` -> `http_csv`
|
||||
- `top500`, `epoch_ai`, `telegeography live_map` -> `html_scrape`
|
||||
- `ris_live` -> `stream_probe`
|
||||
- `spacetrack` -> `auth_session_http`
|
||||
|
||||
|
||||
### B5. Add sample validation hooks
|
||||
|
||||
For each adapter, add a lightweight validation rule.
|
||||
|
||||
Examples:
|
||||
|
||||
- JSON array length > 0
|
||||
- CSV rows > 1
|
||||
- HTML page contains expected table or script patterns
|
||||
- stream source yields at least one valid event within timeout
|
||||
|
||||
|
||||
## C. Persistence and Query Layer
|
||||
|
||||
### C1. Save every check run
|
||||
|
||||
Each health check should insert a record.
|
||||
|
||||
Do not overwrite history in Stage 2.
|
||||
|
||||
|
||||
### C2. Add latest-health query helpers
|
||||
|
||||
Add helper functions to fetch:
|
||||
|
||||
- latest health record by datasource
|
||||
- latest failed health record
|
||||
- recent health history
|
||||
|
||||
|
||||
### C3. Optional retention policy
|
||||
|
||||
For Stage 2, retention can be deferred.
|
||||
|
||||
If desired, keep only:
|
||||
|
||||
- last N records per datasource
|
||||
|
||||
|
||||
## D. API Layer
|
||||
|
||||
### D1. Add health list endpoint
|
||||
|
||||
Suggested endpoint:
|
||||
|
||||
- `GET /api/v1/datasources/health`
|
||||
|
||||
Returns:
|
||||
|
||||
- datasource id
|
||||
- collector name
|
||||
- current endpoint
|
||||
- latest health status
|
||||
- last checked time
|
||||
- short reason
|
||||
|
||||
|
||||
### D2. Add per-datasource health detail endpoint
|
||||
|
||||
Suggested endpoint:
|
||||
|
||||
- `GET /api/v1/datasources/{id}/health`
|
||||
|
||||
Returns:
|
||||
|
||||
- latest record
|
||||
- recent history
|
||||
- detailed classification fields
|
||||
|
||||
|
||||
### D3. Add manual health trigger endpoint
|
||||
|
||||
Suggested endpoint:
|
||||
|
||||
- `POST /api/v1/datasources/{id}/health-check`
|
||||
|
||||
Behavior:
|
||||
|
||||
- run a health check now
|
||||
- persist the result
|
||||
- return the new record
|
||||
|
||||
|
||||
### D4. Add bulk health trigger endpoint
|
||||
|
||||
Suggested endpoint:
|
||||
|
||||
- `POST /api/v1/datasources/health-check-all`
|
||||
|
||||
Behavior:
|
||||
|
||||
- enqueue or run health checks for all active datasources
|
||||
|
||||
|
||||
## E. Scheduling
|
||||
|
||||
### E1. Add health scheduler task
|
||||
|
||||
Decide scheduling strategy.
|
||||
|
||||
Recommended first version:
|
||||
|
||||
- run collector jobs and health checks separately
|
||||
- health checks run on a lower frequency
|
||||
|
||||
Suggested frequency:
|
||||
|
||||
- every 6h or 12h for most datasources
|
||||
- optionally on-demand only in the very first cut
|
||||
|
||||
|
||||
### E2. Prevent health check collision with collection
|
||||
|
||||
Rules:
|
||||
|
||||
- health checks should not disrupt active collection
|
||||
- they should use light requests
|
||||
- if a collector is currently running, health check may:
|
||||
- skip
|
||||
- or use a lightweight endpoint probe only
|
||||
|
||||
|
||||
## F. Frontend
|
||||
|
||||
### F1. Add health columns to datasource list
|
||||
|
||||
Update:
|
||||
|
||||
- `frontend/src/pages/DataSources/DataSources.tsx`
|
||||
|
||||
Suggested new columns:
|
||||
|
||||
- health status
|
||||
- last checked
|
||||
- reason summary
|
||||
|
||||
|
||||
### F2. Add manual health check action
|
||||
|
||||
Per datasource:
|
||||
|
||||
- button or dropdown action:
|
||||
- `健康检查`
|
||||
|
||||
|
||||
### F3. Add health detail drawer or modal
|
||||
|
||||
Show:
|
||||
|
||||
- endpoint checked
|
||||
- status
|
||||
- HTTP status
|
||||
- content type
|
||||
- sample count
|
||||
- error message
|
||||
- last few results
|
||||
|
||||
|
||||
### F4. Add basic visual language
|
||||
|
||||
Suggested colors:
|
||||
|
||||
- green -> healthy
|
||||
- yellow -> degraded
|
||||
- orange -> rate-limited / auth-required
|
||||
- red -> failed / schema-changed
|
||||
|
||||
|
||||
## G. Observability
|
||||
|
||||
### G1. Structured logging
|
||||
|
||||
Every health check should log:
|
||||
|
||||
- datasource id
|
||||
- collector name
|
||||
- endpoint
|
||||
- status
|
||||
- latency
|
||||
- failure class
|
||||
|
||||
|
||||
### G2. Optional metrics
|
||||
|
||||
If metrics are added later, useful counters include:
|
||||
|
||||
- health checks total
|
||||
- health checks failed
|
||||
- schema changes detected
|
||||
- rate limited checks
|
||||
|
||||
|
||||
## H. Tests
|
||||
|
||||
### H1. Unit tests
|
||||
|
||||
Add tests for:
|
||||
|
||||
- status classification
|
||||
- content type classification
|
||||
- adapter behavior
|
||||
- latest-health query helpers
|
||||
|
||||
|
||||
### H2. API tests
|
||||
|
||||
Add tests for:
|
||||
|
||||
- health endpoints require auth
|
||||
- manual trigger endpoint works
|
||||
- list endpoint returns latest status
|
||||
|
||||
|
||||
### H3. Failure-path tests
|
||||
|
||||
Add coverage for:
|
||||
|
||||
- HTTP 404
|
||||
- HTTP 429
|
||||
- invalid JSON
|
||||
- empty response
|
||||
- parse mismatch
|
||||
|
||||
|
||||
## Suggested File Plan
|
||||
|
||||
Possible implementation files:
|
||||
|
||||
- `backend/app/models/datasource_health_check.py`
|
||||
- `backend/app/services/datasource_health.py`
|
||||
- `backend/app/schemas/datasource_health.py`
|
||||
- `backend/app/api/v1/datasource_health.py`
|
||||
- migration file under the project migration system
|
||||
|
||||
Likely touched existing files:
|
||||
|
||||
- `backend/app/api/main.py`
|
||||
- `frontend/src/pages/DataSources/DataSources.tsx`
|
||||
- `backend/tests/test_api.py`
|
||||
|
||||
|
||||
## Suggested Execution Order
|
||||
|
||||
1. Add model and migration
|
||||
2. Add service and result schema
|
||||
3. Add deterministic adapters
|
||||
4. Add manual trigger API
|
||||
5. Add list/detail API
|
||||
6. Add frontend visibility
|
||||
7. Add scheduled checks
|
||||
8. Expand tests
|
||||
|
||||
|
||||
## Minimal First Milestone
|
||||
|
||||
If we want the fastest useful slice, do this first:
|
||||
|
||||
1. health table
|
||||
2. deterministic check service
|
||||
3. manual per-datasource health check API
|
||||
4. latest health list API
|
||||
5. frontend status badge column
|
||||
|
||||
That is enough to start operating the system and will provide the input layer for Stage 3.
|
||||
|
||||
|
||||
## Dependency On Later Stages
|
||||
|
||||
Stage 2 outputs become direct inputs for Stage 3.
|
||||
|
||||
Specifically:
|
||||
|
||||
- failed or schema-changed health records become agent triggers
|
||||
- health history becomes repair context
|
||||
- endpoint_checked becomes proposal baseline
|
||||
|
||||
|
||||
## Success Criteria
|
||||
|
||||
Stage 2 is done when:
|
||||
|
||||
- every active datasource can be health-checked deterministically
|
||||
- the latest health state is visible in API and UI
|
||||
- operators can manually trigger checks
|
||||
- failures are categorized into stable machine-readable statuses
|
||||
- no LLM is required for core health visibility
|
||||
@@ -15,3 +15,8 @@
|
||||
- 明确写明“已完成”的计划,优先归档
|
||||
- 已被正式实现替代、继续放在 `docs/` 根目录会误导后续开发的计划,归档
|
||||
- 仍然指导未来开发、尚未完成或仍有明确执行价值的文档,继续保留在 `docs/`
|
||||
|
||||
补充说明:
|
||||
|
||||
- 一部分归档文档来自外部或临时工作流草案,例如 sisyphus 生成的初稿
|
||||
- 这类文档如果有可用内容,应先吸收到 `docs/plans/` 或 `docs/technical/`,再归档保留来源记录
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
> Archived note: this document was originally created by sisyphus and later reviewed against the main docs set. Useful content has been absorbed into `docs/plans/` where appropriate.
|
||||
|
||||
# 地球3D可视化架构重构计划
|
||||
|
||||
## 背景
|
||||
@@ -1,3 +1,5 @@
|
||||
> Archived note: this document was originally created by sisyphus and later reviewed against the main docs set. Useful content has been absorbed into `docs/plans/` where appropriate.
|
||||
|
||||
# 卫星预测轨道显示功能
|
||||
|
||||
## TL;DR
|
||||
@@ -1,3 +1,5 @@
|
||||
> Archived note: this document was originally created by sisyphus and later reviewed against the main docs set. Useful content has been absorbed into `docs/plans/` where appropriate.
|
||||
|
||||
# UE5 3D 大屏客户端开发计划
|
||||
|
||||
## 项目概述
|
||||
@@ -1,3 +1,5 @@
|
||||
> Archived note: this document was originally created by sisyphus and later reviewed against the main docs set. Useful content has been absorbed into `docs/plans/` where appropriate.
|
||||
|
||||
# WebGL Instancing 卫星渲染优化计划
|
||||
|
||||
## 背景
|
||||
@@ -1,210 +0,0 @@
|
||||
# Earth 模块整治计划
|
||||
|
||||
## 背景
|
||||
|
||||
`planet` 前端中的 Earth 模块是当前最重要的大屏 3D 星球展示能力,但它仍以 legacy iframe 页面形式存在:
|
||||
|
||||
- React 页面入口仅为 [Earth.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Earth/Earth.tsx)
|
||||
- 实际 3D 实现位于 [frontend/public/earth](/home/ray/dev/linkong/planet/frontend/public/earth)
|
||||
|
||||
当前模块已经具备基础展示能力,但在生命周期、性能、可恢复性、可维护性方面存在明显隐患,不适合长期无人值守的大屏场景直接扩展。
|
||||
|
||||
## 目标
|
||||
|
||||
本计划的目标不是立刻重写 Earth,而是分阶段把它从“能跑的 legacy 展示页”提升到“可稳定运行、可持续演进的大屏核心模块”。
|
||||
|
||||
核心目标:
|
||||
|
||||
1. 先止血,解决资源泄漏、重载污染、假性卡顿等稳定性问题
|
||||
2. 再梳理数据加载、交互和渲染循环,降低性能风险
|
||||
3. 最后逐步从 iframe legacy 向可控模块化架构迁移
|
||||
|
||||
## 现阶段主要问题
|
||||
|
||||
### 1. 生命周期缺失
|
||||
|
||||
- 没有统一 `destroy()` / 卸载清理逻辑
|
||||
- `requestAnimationFrame`
|
||||
- `window/document/dom listeners`
|
||||
- `THREE` geometry / material / texture
|
||||
- 运行时全局状态
|
||||
都没有系统回收
|
||||
|
||||
### 2. 数据重载不完整
|
||||
|
||||
- `reloadData()` 没有彻底清理旧场景对象
|
||||
- cable、landing point、satellite 相关缓存与对象存在累积风险
|
||||
|
||||
### 3. 渲染与命中检测成本高
|
||||
|
||||
- 鼠标移动时频繁创建 `Raycaster` / `Vector2`
|
||||
- cable 命中前会重复做 bounding box 计算
|
||||
- 卫星每帧计算量偏高
|
||||
|
||||
### 4. 状态管理分裂
|
||||
|
||||
- 大量依赖 `window.*` 全局桥接
|
||||
- 模块之间靠隐式共享状态通信
|
||||
- React 外层无法有效感知 Earth 内部状态
|
||||
|
||||
### 5. 错误恢复弱
|
||||
|
||||
- 数据加载失败主要依赖 `console` 和轻提示
|
||||
- 缺少统一重试、降级、局部失败隔离机制
|
||||
|
||||
## 分阶段计划
|
||||
|
||||
## Phase 1:稳定性止血
|
||||
|
||||
目标:
|
||||
|
||||
- 不改视觉主形态
|
||||
- 优先解决泄漏、卡死、重载污染
|
||||
|
||||
### 任务
|
||||
|
||||
1. 补 Earth 生命周期管理
|
||||
|
||||
- 为 [main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js) 增加:
|
||||
- `init()`
|
||||
- `destroy()`
|
||||
- `reloadData()`
|
||||
三类明确入口
|
||||
- 统一记录并释放:
|
||||
- animation frame id
|
||||
- interval / timeout
|
||||
- DOM 事件监听
|
||||
- `window` 暴露对象
|
||||
|
||||
2. 增加场景对象清理层
|
||||
|
||||
- 为 cable / landing point / satellite sprite / orbit line 提供统一清理函数
|
||||
- reload 前先 dispose 旧对象,再重新加载
|
||||
|
||||
3. 增加 stale 状态恢复
|
||||
|
||||
- 页面重新进入时,先清理上一次遗留选择态、hover 态、锁定态
|
||||
- 避免 iframe reload 后出现旧状态残留
|
||||
|
||||
4. 加强失败提示
|
||||
|
||||
- 电缆、登陆点、卫星加载拆分为独立状态
|
||||
- 某一类数据失败时,其它类型仍可继续显示
|
||||
- 提供明确的页面内提示而不是只打 console
|
||||
|
||||
### 验收标准
|
||||
|
||||
- 页面重复进入 / 离开后内存不持续上涨
|
||||
- 连续多次点“重新加载数据”后对象数量不异常增加
|
||||
- 单一数据源加载失败时页面不整体失效
|
||||
|
||||
## Phase 2:性能优化
|
||||
|
||||
目标:
|
||||
|
||||
- 控制鼠标交互和动画循环成本
|
||||
- 提升大屏长时间运行的稳定帧率
|
||||
|
||||
### 任务
|
||||
|
||||
1. 复用交互对象
|
||||
|
||||
- 复用 `Raycaster`、`Vector2`、中间 `Vector3`
|
||||
- 避免 `mousemove` 热路径中频繁 new 对象
|
||||
|
||||
2. 优化 cable 命中逻辑
|
||||
|
||||
- 提前缓存 cable 中心点 / bounding 数据
|
||||
- 移除 `mousemove` 内重复 `computeBoundingBox()`
|
||||
- 必要时增加分层命中:
|
||||
- 先粗筛
|
||||
- 再精确相交
|
||||
|
||||
3. 改造动画循环
|
||||
|
||||
- 使用真实 `deltaTime`
|
||||
- 把卫星位置更新、呼吸动画、视觉状态更新拆成独立阶段
|
||||
- 为不可见对象减少无意义更新
|
||||
|
||||
4. 卫星轨迹与预测轨道优化
|
||||
|
||||
- 评估轨迹更新频率
|
||||
- 对高开销几何计算增加缓存
|
||||
- 限制预测轨道生成频次
|
||||
|
||||
### 验收标准
|
||||
|
||||
- 鼠标移动时不明显掉帧
|
||||
- 中高数据量下动画速度不受帧率明显影响
|
||||
- 长时间运行 CPU/GPU 占用更平稳
|
||||
|
||||
## Phase 3:架构收编
|
||||
|
||||
目标:
|
||||
|
||||
- 降低 legacy iframe 架构带来的维护成本
|
||||
- 让 React 主应用重新获得对 Earth 模块的控制力
|
||||
|
||||
### 任务
|
||||
|
||||
1. 抽离 Earth App Shell
|
||||
|
||||
- 将数据加载、错误状态、控制面板状态抽到更明确的模块边界
|
||||
- 减少 `window.*` 全局依赖
|
||||
|
||||
2. 规范模块通信
|
||||
|
||||
- 统一 `main / controls / cables / satellites / ui` 的状态流
|
||||
- 明确只读配置、运行时状态、渲染对象的职责分层
|
||||
|
||||
3. 评估去 iframe 迁移
|
||||
|
||||
- 中期可以保留 public/legacy 资源目录
|
||||
- 但逐步把 Earth 作为前端内嵌模块而不是完全孤立页面
|
||||
|
||||
### 验收标准
|
||||
|
||||
- Earth 内部状态不再大量依赖全局变量
|
||||
- React 外层可以感知 Earth 加载状态和错误状态
|
||||
- 后续功能开发不再必须修改多个 legacy 文件才能完成
|
||||
|
||||
## 优先级建议
|
||||
|
||||
### P0
|
||||
|
||||
- 生命周期清理
|
||||
- reload 清理
|
||||
- stale 状态恢复
|
||||
|
||||
### P1
|
||||
|
||||
- 命中检测优化
|
||||
- 动画 `deltaTime`
|
||||
- 数据加载失败隔离
|
||||
|
||||
### P2
|
||||
|
||||
- 全局状态收编
|
||||
- iframe 架构迁移
|
||||
|
||||
## 推荐实施顺序
|
||||
|
||||
1. 先做 Phase 1
|
||||
2. 再做交互热路径与动画循环优化
|
||||
3. 最后再考虑架构迁移
|
||||
|
||||
## 风险提示
|
||||
|
||||
1. Earth 是 legacy 模块,修复时容易牵一发而动全身
|
||||
2. 如果不先补清理逻辑,后续所有性能优化收益都会被泄漏问题吃掉
|
||||
3. 如果过早重写而不先止血,短期会影响现有演示稳定性
|
||||
|
||||
## 当前建议
|
||||
|
||||
最值得马上启动的是一个小范围稳定性 sprint:
|
||||
|
||||
- 生命周期清理
|
||||
- reload 全量清理
|
||||
- 错误状态隔离
|
||||
|
||||
这个阶段不追求“更炫”,先追求“更稳”。稳定下来之后,再进入性能和架构层的优化。
|
||||
@@ -1,117 +0,0 @@
|
||||
# Earth 电视直播模块计划
|
||||
|
||||
## 目标
|
||||
|
||||
为 `Earth` 页面增加一个可配置、可扩展、可拖拽的电视直播模块:
|
||||
|
||||
- 后台可配置新闻直播源
|
||||
- 默认兜底源为央视 `CCTV-4`
|
||||
- 未来可通过采集器接入世界各地新闻直播源
|
||||
- Earth 工具栏 `显示控制` 子菜单新增电视按钮
|
||||
- 点击后打开一个与其他 HUD 一致的可拖拽/可关闭窗口
|
||||
- 窗口内部可播放或承载新闻直播页面
|
||||
|
||||
## 设计原则
|
||||
|
||||
- 第一阶段先交付“后台可配 + Earth 可用 + 默认可回退”的版本
|
||||
- 公开读取接口与后台管理接口分离
|
||||
- 手工配置源与采集器源共用统一的前端消费结构
|
||||
- Earth 里的电视窗口必须复用现有 HUD 拖拽、关闭、布局最大化逻辑
|
||||
- 小屏下优先保证窗口完整显示,超出部分在窗口内部滚动
|
||||
|
||||
## 分阶段实现
|
||||
|
||||
### Phase 1:后端配置与公开读取
|
||||
|
||||
- 在系统设置中新增 `tv` 分类
|
||||
- 定义直播源配置结构:
|
||||
- `default_source_id`
|
||||
- `auto_fallback`
|
||||
- `sources[]`
|
||||
- 每个直播源至少包含:
|
||||
- `id`
|
||||
- `name`
|
||||
- `provider`
|
||||
- `region`
|
||||
- `language`
|
||||
- `source_type`
|
||||
- `embed_url`
|
||||
- `stream_url`
|
||||
- `homepage_url`
|
||||
- `is_enabled`
|
||||
- `is_fallback`
|
||||
- `sort_order`
|
||||
- `collector_source`
|
||||
- `notes`
|
||||
- 默认兜底源使用央视官网 `CCTV-4` 直播页
|
||||
- 新增公开读取接口,供 Earth 页面无登录态读取直播源配置
|
||||
|
||||
### Phase 2:采集器扩展位
|
||||
|
||||
- 新增 `news_live_streams` collector 占位
|
||||
- 规范采集器入库数据结构,使其能与后台手工配置源合并
|
||||
- TV 公开接口支持合并:
|
||||
- 后台手工配置源
|
||||
- 采集器入库源
|
||||
- 保持手工配置源优先级更高,避免采集器覆盖人工兜底配置
|
||||
|
||||
### Phase 3:后台配置界面
|
||||
|
||||
- 在系统配置页新增 `电视直播` tab
|
||||
- 支持:
|
||||
- 查看当前默认源
|
||||
- 开关自动回退
|
||||
- 新增直播源
|
||||
- 编辑直播源
|
||||
- 删除直播源
|
||||
- 启用/禁用直播源
|
||||
- 将某个直播源设为默认源
|
||||
- 明确区分:
|
||||
- 手工配置源
|
||||
- 采集器来源
|
||||
|
||||
### Phase 4:Earth HUD 集成
|
||||
|
||||
- 在 `显示控制` 子菜单加入电视按钮
|
||||
- 新增 TV HUD 面板:
|
||||
- 可拖拽
|
||||
- 可关闭
|
||||
- 支持显示/隐藏状态同步
|
||||
- 参与布局最大化与恢复布局
|
||||
- 面板内容至少包含:
|
||||
- 当前频道标题
|
||||
- 源切换下拉菜单
|
||||
- 刷新按钮
|
||||
- 打开官网按钮
|
||||
- 播放区域
|
||||
|
||||
### Phase 5:播放策略
|
||||
|
||||
- 第一版优先支持 `iframe`/嵌入页类直播源
|
||||
- 为未来扩展保留:
|
||||
- `hls`
|
||||
- `video`
|
||||
- `external`
|
||||
- 如果默认源不可用:
|
||||
- 优先回退到标记为 `is_fallback=true` 的源
|
||||
- 若无明确回退源,则回退到第一个可用源
|
||||
- 面板内要有清晰的加载、错误、回退提示
|
||||
|
||||
### Phase 6:打磨与清理
|
||||
|
||||
- 统一 HUD 风格
|
||||
- 小屏下限制窗口尺寸并启用内部滚动
|
||||
- 避免窗口超出屏幕
|
||||
- 补最小验证
|
||||
- 清理临时代码、重复样式和无用资源
|
||||
|
||||
## 首版交付定义
|
||||
|
||||
当以下条件满足时,认为首版可用:
|
||||
|
||||
- 后台可以配置新闻直播源
|
||||
- Earth 可以读取并显示默认直播源
|
||||
- 工具栏可打开电视窗口
|
||||
- 电视窗口可拖拽、可关闭
|
||||
- 央视 `CCTV-4` 作为默认兜底源可被使用
|
||||
- 代码结构已为后续采集器接入预留统一接口
|
||||
@@ -1,355 +0,0 @@
|
||||
# BGP Context
|
||||
|
||||
## Current Goal
|
||||
|
||||
The BGP module is being evolved from an anomaly-only demo into a layered observability pipeline:
|
||||
|
||||
`raw observations -> enrichment -> detectors -> incidents -> console/Earth visualization`
|
||||
|
||||
The practical product goal is no longer just to "show incidents on the globe". The current product objective is:
|
||||
|
||||
1. keep BGP visually present on Earth even when incident density is low
|
||||
2. make incidents clearly feel like a higher-confidence layer than anomalies
|
||||
3. show that the observation network is still active even when there are no active incidents
|
||||
|
||||
In practice, that means Earth should behave like an observability surface, not only an incident map:
|
||||
|
||||
- `collectors` show that observation is happening
|
||||
- `activity` shows where routing state is currently active or noisy
|
||||
- `incidents` become the highest-confidence focus layer
|
||||
|
||||
## Current Backend Architecture
|
||||
|
||||
### Data Layers
|
||||
|
||||
1. `BGPObservation`
|
||||
- File: `backend/app/models/bgp_observation.py`
|
||||
- Purpose: store normalized raw routing observations from live/history sources.
|
||||
- Typical fields:
|
||||
- `source`
|
||||
- `collector`
|
||||
- `peer_asn`
|
||||
- `peer_ip`
|
||||
- `prefix`
|
||||
- `event_type`
|
||||
- `as_path`
|
||||
- `origin_asn`
|
||||
- `next_hop`
|
||||
- `communities`
|
||||
- `observed_at`
|
||||
- `raw_payload`
|
||||
- `collector_geo`
|
||||
- `ingest_batch_id`
|
||||
|
||||
2. `BGPAnomaly`
|
||||
- File: `backend/app/models/bgp_anomaly.py`
|
||||
- Purpose: hold atomic detector outputs.
|
||||
- Current detector output types include:
|
||||
- `origin_change`
|
||||
- `more_specific_burst`
|
||||
- `mass_withdrawal`
|
||||
|
||||
3. `BGPIncident`
|
||||
- File: `backend/app/models/bgp_incident.py`
|
||||
- Purpose: aggregate atomic anomalies into incident-level objects for humans and the UI.
|
||||
|
||||
### Pipeline
|
||||
|
||||
Main flow is currently anchored in:
|
||||
|
||||
- `backend/app/services/collectors/bgp_common.py`
|
||||
- `backend/app/services/bgp_enrichment.py`
|
||||
- `backend/app/services/bgp_detectors.py`
|
||||
- `backend/app/services/bgp_incidents.py`
|
||||
|
||||
Operational flow:
|
||||
|
||||
1. collectors fetch raw BGP data
|
||||
2. `normalize_bgp_event()` standardizes payloads
|
||||
3. observations are persisted to `bgp_observations`
|
||||
4. enrichment augments events with analysis context
|
||||
5. detectors create `bgp_anomalies`
|
||||
6. incident aggregation rolls anomalies up into `bgp_incidents`
|
||||
|
||||
### Current Ingest Sources
|
||||
|
||||
1. `RIPE RIS Live`
|
||||
- Collector file: `backend/app/services/collectors/ris_live.py`
|
||||
- Used for realtime observation flow.
|
||||
|
||||
2. `CAIDA BGPStream Backfill`
|
||||
- Collector file: `backend/app/services/collectors/bgpstream.py`
|
||||
- Used as history/backfill entry point.
|
||||
|
||||
## Current Enrichment Status
|
||||
|
||||
Implemented enrichment skeleton in:
|
||||
|
||||
- `backend/app/services/bgp_enrichment.py`
|
||||
|
||||
Current enrichments:
|
||||
|
||||
- prefix family / prefix length
|
||||
- supernet / more-specific derivation
|
||||
- deduplicated AS path
|
||||
- path prepending hints
|
||||
- collector region info
|
||||
- prefix baseline hints
|
||||
- new-origin detection
|
||||
- ASN organization profile from PeeringDB where available
|
||||
- prefix scope / impacted region hints
|
||||
- prefix geography source priority:
|
||||
- `OpenGeoFeed` (override/high confidence)
|
||||
- `IPtoASN` (country-range baseline)
|
||||
- `NRO delegated stats` (registry-allocation fallback)
|
||||
|
||||
Current limitation:
|
||||
|
||||
- `RPKI` is still placeholder-only and returns `unknown`
|
||||
- no real ROA validation source is integrated yet
|
||||
- `inetnum` / `inet6num` whois fallback is still pending
|
||||
|
||||
## Current API Surface
|
||||
|
||||
Primary API file:
|
||||
|
||||
- `backend/app/api/v1/bgp.py`
|
||||
|
||||
Available endpoints:
|
||||
|
||||
- `/api/v1/bgp/events`
|
||||
- `/api/v1/bgp/events/summary`
|
||||
- `/api/v1/bgp/events/{id}`
|
||||
- `/api/v1/bgp/anomalies`
|
||||
- `/api/v1/bgp/anomalies/summary`
|
||||
- `/api/v1/bgp/anomalies/{id}`
|
||||
- `/api/v1/bgp/incidents`
|
||||
- `/api/v1/bgp/incidents/summary`
|
||||
- `/api/v1/bgp/incidents/{id}`
|
||||
|
||||
Visualization GeoJSON endpoints:
|
||||
|
||||
- `backend/app/api/v1/visualization.py`
|
||||
- `/api/v1/visualization/geo/bgp-collectors`
|
||||
- `/api/v1/visualization/geo/bgp-anomalies`
|
||||
- `/api/v1/visualization/geo/bgp-incidents`
|
||||
|
||||
## Current Earth Behavior
|
||||
|
||||
Relevant files:
|
||||
|
||||
- `frontend/public/earth/js/bgp.js`
|
||||
- `frontend/public/earth/js/main.js`
|
||||
- `frontend/public/earth/js/info-card.js`
|
||||
- `frontend/public/earth/js/constants.js`
|
||||
- `frontend/public/earth/index.html`
|
||||
|
||||
Current design:
|
||||
|
||||
1. Collectors are always shown when BGP is enabled.
|
||||
2. Incident markers are now the primary Earth BGP markers.
|
||||
3. If there are no incidents, Earth falls back to anomaly markers.
|
||||
4. If there are no anomalies either, collectors still provide presence.
|
||||
5. A dedicated `activity layer` now adds:
|
||||
- per-collector recent 15-minute activity halos
|
||||
- clustered regional activity hints derived from active collectors
|
||||
6. Incident markers now use:
|
||||
- symbol-driven event cores
|
||||
- outward ring pulses
|
||||
- reduced diffuse glow compared with older Earth builds
|
||||
5. The right-side stats now show:
|
||||
- BGP events
|
||||
- collector count
|
||||
- BGP status summary
|
||||
|
||||
This is directionally correct, but still incomplete for low-event-density periods. Right now Earth can still feel too quiet when incidents are sparse because the system lacks a dedicated `activity layer` between raw observation and incident focus.
|
||||
|
||||
Current BGP status strategy:
|
||||
|
||||
- incidents present: show active incident count
|
||||
- no incidents but anomalies present: show active anomaly count, plus active observation regions when available
|
||||
- no incidents/anomalies but activity present: show `观测网络运行中`
|
||||
- no incidents/anomalies but collectors present: show `观测网络运行中 · 当前未发现聚合级事件`
|
||||
- no BGP data at all: show `暂无观测数据`
|
||||
|
||||
Earth info-card strategy:
|
||||
|
||||
- `bgp` card is now incident-centric in wording
|
||||
- `bgp_collector` card shows collector location and current event count
|
||||
|
||||
## Current Product Gap
|
||||
|
||||
The main product gap is not architecture correctness. It is low-density visualization strategy.
|
||||
|
||||
Current reality:
|
||||
|
||||
- incident count is naturally much lower than anomaly count
|
||||
- that is expected, because incidents are aggregated and de-noised
|
||||
- but incident-first rendering makes the Earth view look too quiet unless there is another always-available activity layer
|
||||
|
||||
Implementation detail for the recommended `activity layer` is expanded in [bgp-region-aggregation-plan.md](/home/ray/dev/linkong/planet/docs/earth/bgp-region-aggregation-plan.md).
|
||||
|
||||
So the immediate next milestone is:
|
||||
|
||||
`event map -> observability map`
|
||||
|
||||
That means Earth needs three simultaneously readable layers:
|
||||
|
||||
1. `observation layer`
|
||||
- collectors
|
||||
- recent collector activity
|
||||
- baseline coverage
|
||||
2. `activity layer`
|
||||
- recent event density
|
||||
- anomaly/noise hotspots
|
||||
- regional activity scoring
|
||||
- incident presence bonus
|
||||
3. `incident layer`
|
||||
- sparse but highly legible, high-confidence event objects
|
||||
- symbol-driven markers
|
||||
- outward ring pulse instead of broad diffuse glow
|
||||
|
||||
## Incident Visual Direction
|
||||
|
||||
The Earth `incident` layer should not read like a large glowing patch. It should read like a compact, high-confidence event focus.
|
||||
|
||||
Design principles:
|
||||
|
||||
1. `incident` markers should use a strong primary symbol
|
||||
- the symbol shape should carry type meaning where possible
|
||||
- examples:
|
||||
- `origin_change`: triangle-like warning marker
|
||||
- `mass_withdrawal`: alert/exclamation-style marker
|
||||
- `more_specific_burst`: split/radiating marker
|
||||
|
||||
2. emphasis should come from outward ring pulses, not area flooding
|
||||
- use a compact hot core
|
||||
- use one or more expanding ring pulses
|
||||
- avoid broad luminous blobs that make the event center feel vague
|
||||
|
||||
3. `collector` and `incident` must stay visually distinct
|
||||
- collectors are observation infrastructure
|
||||
- incidents are extracted event focus
|
||||
- collector activity should stay quieter than incident pulse language
|
||||
|
||||
4. calm periods still need observability presence
|
||||
- collectors and activity layers should keep the map alive
|
||||
- once incidents appear, they should clearly dominate nearby BGP visuals
|
||||
|
||||
5. incident geography should become `prefix-centric`
|
||||
- collectors should remain evidence sources, not the primary event location
|
||||
- preferred geography priority:
|
||||
- `prefix_geography`
|
||||
- `prefix_scope`
|
||||
- `ASN organization region`
|
||||
- `collector centroid` as final fallback
|
||||
- `prefix_scope` should remain an observation-derived scope hint
|
||||
- a new `prefix_geography` layer should be introduced for actual prefix-centric placement
|
||||
|
||||
Reference inspiration:
|
||||
|
||||
- `World Monitor`
|
||||
- sparse event symbols
|
||||
- compact centers
|
||||
- ring-like outward pulses
|
||||
- stronger incident legibility than diffuse glow
|
||||
|
||||
## Current Console Behavior
|
||||
|
||||
Relevant page:
|
||||
|
||||
- `frontend/src/pages/BGP/BGP.tsx`
|
||||
|
||||
Current BGP console page has three levels:
|
||||
|
||||
1. observation summary
|
||||
- total events
|
||||
- collector count
|
||||
- prefix count
|
||||
|
||||
2. incident summary and incident table
|
||||
|
||||
3. anomaly detail table plus recent observation events
|
||||
|
||||
This means the BGP page still has useful signal even when there are zero anomalies.
|
||||
|
||||
## Known Product/Engineering Boundaries
|
||||
|
||||
1. The current system is still closer to an event board than a full BGP sensing platform.
|
||||
2. RIS coverage still needs to expand beyond narrow subscription scope.
|
||||
3. BGPStream history is still not full MRT-to-prefix decoded analytics.
|
||||
4. Collector geography still depends heavily on static RIPE RIS mappings.
|
||||
5. Incident-to-cable/IXP/region association is still weak and early-stage.
|
||||
6. Earth currently visualizes logical observation/impact structure, not true physical traffic paths.
|
||||
|
||||
## Test Status
|
||||
|
||||
BGP-specific tests live in:
|
||||
|
||||
- `backend/tests/test_bgp.py`
|
||||
|
||||
Verified status at this point:
|
||||
|
||||
- `25 passed` for `backend/tests/test_bgp.py`
|
||||
- `62 passed` for `backend/tests`
|
||||
|
||||
Covered areas include:
|
||||
|
||||
- normalization
|
||||
- observation serialization
|
||||
- enrichment
|
||||
- detectors, including route leak candidate and path flap
|
||||
- incident aggregation
|
||||
- batch anomaly creation
|
||||
- BGP events/incidents API
|
||||
- summary endpoints
|
||||
|
||||
## Most Relevant Files
|
||||
|
||||
Backend:
|
||||
|
||||
- `backend/app/models/bgp_observation.py`
|
||||
- `backend/app/models/bgp_anomaly.py`
|
||||
- `backend/app/models/bgp_incident.py`
|
||||
- `backend/app/services/collectors/bgp_common.py`
|
||||
- `backend/app/services/bgp_enrichment.py`
|
||||
- `backend/app/services/bgp_detectors.py`
|
||||
- `backend/app/services/bgp_incidents.py`
|
||||
- `backend/app/api/v1/bgp.py`
|
||||
- `backend/app/api/v1/visualization.py`
|
||||
|
||||
Frontend:
|
||||
|
||||
- `frontend/src/pages/BGP/BGP.tsx`
|
||||
- `frontend/public/earth/js/bgp.js`
|
||||
- `frontend/public/earth/js/main.js`
|
||||
- `frontend/public/earth/js/info-card.js`
|
||||
- `frontend/public/earth/js/constants.js`
|
||||
- `frontend/public/earth/index.html`
|
||||
|
||||
## Recommended Next Steps
|
||||
|
||||
### Next Backend / Detection Priority
|
||||
|
||||
1. Integrate real RPKI validation data.
|
||||
2. Expand realtime collector coverage and include withdrawals more broadly.
|
||||
3. Continue refining route leak and path instability detectors with stronger heuristics.
|
||||
|
||||
### Next Correlation / Storytelling Priority
|
||||
|
||||
4. Strengthen incident aggregation semantics and titles.
|
||||
5. Add weak correlation from incidents to:
|
||||
- cable corridors
|
||||
- landing points
|
||||
- IXPs
|
||||
- other traffic anomaly sources
|
||||
6. Refine Earth hover/click handoff between collectors and incidents.
|
||||
|
||||
### Next Visualization Priority
|
||||
|
||||
7. Refine regional activity scoring so the activity layer is informative without becoming noisy.
|
||||
8. Add more incident symbol types as new detectors land.
|
||||
9. Add a real prefix geography source:
|
||||
- `IPtoASN / IPtoCountry` as the first practical dataset
|
||||
- `OpenGeoFeed` as a higher-quality override layer
|
||||
- registry/whois only as fallback
|
||||
@@ -1,296 +0,0 @@
|
||||
# BGP Earth Rendering Plan
|
||||
|
||||
## Goal
|
||||
|
||||
This document defines how the BGP `region activity layer` and `incident layer` should coexist on Earth without conflicting.
|
||||
|
||||
The main question it answers is:
|
||||
|
||||
- how to add a regional observability background layer
|
||||
- without weakening the current incident-first event focus
|
||||
|
||||
## Core Principle
|
||||
|
||||
The Earth design should follow a strict semantic hierarchy:
|
||||
|
||||
- `collector layer` = observation infrastructure
|
||||
- `region activity layer` = background situational awareness
|
||||
- `incident layer` = focal high-confidence event objects
|
||||
|
||||
In short:
|
||||
|
||||
- collectors prove the network is observing
|
||||
- regions show where routing behavior is active or abnormal
|
||||
- incidents show the concrete event worth clicking
|
||||
|
||||
Region aggregation is therefore not a replacement for incident rendering.
|
||||
It is the context layer that makes sparse incident markers legible.
|
||||
|
||||
## Rendering Hierarchy
|
||||
|
||||
Recommended visual stack order:
|
||||
|
||||
1. collector network / collector halos
|
||||
2. region activity glow
|
||||
3. incident markers and incident pulses
|
||||
|
||||
This ordering should always hold.
|
||||
|
||||
Why:
|
||||
|
||||
- collectors should stay visible but quiet
|
||||
- regions should create ambient activity presence
|
||||
- incidents must remain the first thing users notice as a concrete event
|
||||
|
||||
## Role Separation
|
||||
|
||||
### Region Layer
|
||||
|
||||
The region layer answers:
|
||||
|
||||
- where is routing activity building up
|
||||
- where is there current noise or instability
|
||||
- which part of the world is currently worth looking at
|
||||
|
||||
The region layer should feel:
|
||||
|
||||
- broad
|
||||
- ambient
|
||||
- low-frequency
|
||||
- contextual
|
||||
|
||||
### Incident Layer
|
||||
|
||||
The incident layer answers:
|
||||
|
||||
- which exact event should the user inspect
|
||||
- where is the highest-confidence routing event located right now
|
||||
|
||||
The incident layer should feel:
|
||||
|
||||
- sharp
|
||||
- compact
|
||||
- high-contrast
|
||||
- intentionally clickable
|
||||
|
||||
## Non-Conflict Rules
|
||||
|
||||
To avoid visual and semantic conflict, these implementation rules should be treated as hard constraints:
|
||||
|
||||
1. region markers must not use the same symbol language as incidents
|
||||
2. region emphasis must stay weaker than incident emphasis
|
||||
3. region animation frequency must stay lower than incident animation frequency
|
||||
4. incident markers must always render above region glows
|
||||
5. region layer should support the event, not compete with it
|
||||
|
||||
If a user notices the region layer first but misses the incident marker, the region layer is too strong.
|
||||
|
||||
If a user only sees isolated incident points and cannot feel broader activity context, the region layer is too weak.
|
||||
|
||||
## Region Rendering Rules
|
||||
|
||||
The region layer should not be rendered as a second kind of incident point.
|
||||
|
||||
Recommended representation:
|
||||
|
||||
- diffuse glow
|
||||
- halo
|
||||
- low-detail pulse
|
||||
- soft center, not a sharp icon
|
||||
|
||||
### Status Mapping
|
||||
|
||||
#### `observing`
|
||||
|
||||
- weak glow
|
||||
- cool color, such as cyan or blue
|
||||
- little to no pulse
|
||||
- purpose: keep the globe alive during calm periods
|
||||
|
||||
#### `anomaly`
|
||||
|
||||
- stronger glow
|
||||
- warmer color, such as amber
|
||||
- gentle breathing or low-frequency pulse
|
||||
- purpose: show that a region is experiencing abnormal routing noise
|
||||
|
||||
#### `incident`
|
||||
|
||||
- strongest regional background emphasis
|
||||
- still clearly weaker than the incident marker itself
|
||||
- purpose: lift the surrounding area so the focal event does not feel isolated
|
||||
|
||||
### Region Visual Characteristics
|
||||
|
||||
Recommended properties:
|
||||
|
||||
- large radius
|
||||
- low opacity
|
||||
- soft edge
|
||||
- low-contrast outline or no outline
|
||||
- low pulse amplitude
|
||||
|
||||
Avoid:
|
||||
|
||||
- sharp symbol shapes
|
||||
- strong icon silhouettes
|
||||
- bright hard-edged centers
|
||||
- incident-like pulse language
|
||||
|
||||
## Incident Rendering Rules
|
||||
|
||||
The incident layer should remain visually sharper and more explicit than region activity.
|
||||
|
||||
Recommended qualities:
|
||||
|
||||
- clear event symbol
|
||||
- compact hot core
|
||||
- one or two outward ring pulses
|
||||
- high contrast
|
||||
- clear click target
|
||||
|
||||
The incident layer should read as:
|
||||
|
||||
- focal
|
||||
- deliberate
|
||||
- high-confidence
|
||||
|
||||
while the region layer should read as:
|
||||
|
||||
- contextual
|
||||
- ambient
|
||||
- supporting
|
||||
|
||||
## Region And Incident In The Same Area
|
||||
|
||||
When a region contains one or more incidents:
|
||||
|
||||
- the region glow may intensify
|
||||
- but the incident marker must remain the dominant local feature
|
||||
|
||||
Interpretation should be:
|
||||
|
||||
- `region` says this area is in an event state
|
||||
- `incident marker` says this is the concrete event object
|
||||
|
||||
So a region with `incident` status is not itself the event marker.
|
||||
It is the background state around the event.
|
||||
|
||||
## Interaction Model
|
||||
|
||||
Interaction should also preserve hierarchy.
|
||||
|
||||
### Click Region
|
||||
|
||||
Open a regional situation view, such as:
|
||||
|
||||
- region name
|
||||
- observation count
|
||||
- anomaly count
|
||||
- incident count
|
||||
- affected prefix count
|
||||
- affected ASN count
|
||||
- recent incidents in the region
|
||||
|
||||
### Click Incident
|
||||
|
||||
Keep the current incident-focused detail interaction.
|
||||
|
||||
This creates a natural two-step flow:
|
||||
|
||||
1. region gives context
|
||||
2. incident gives detail
|
||||
|
||||
## Layer Relationship To Existing BGP Elements
|
||||
|
||||
### Collector Layer
|
||||
|
||||
Collectors should remain:
|
||||
|
||||
- quieter than regions
|
||||
- more infrastructural than semantic
|
||||
- proof of coverage, not proof of incident
|
||||
|
||||
### Region Layer
|
||||
|
||||
Regions should become:
|
||||
|
||||
- the main ambient activity layer
|
||||
- the bridge between collectors and incidents
|
||||
- the answer to low-density map quietness
|
||||
|
||||
### Incident Layer
|
||||
|
||||
Incidents should remain:
|
||||
|
||||
- the most legible event layer
|
||||
- sparse but dominant
|
||||
- compact and symbol-driven
|
||||
|
||||
## Practical Visual Test
|
||||
|
||||
Use this test when tuning the Earth implementation:
|
||||
|
||||
### Calm Period
|
||||
|
||||
Expected result:
|
||||
|
||||
- collectors visible
|
||||
- some weak region glows present
|
||||
- no region feels alarm-heavy
|
||||
- globe still feels alive
|
||||
|
||||
### Anomaly Period
|
||||
|
||||
Expected result:
|
||||
|
||||
- one or more regions brighten noticeably
|
||||
- user can sense the active area before clicking
|
||||
- still no confusion between region background and incident objects
|
||||
|
||||
### Incident Period
|
||||
|
||||
Expected result:
|
||||
|
||||
- region provides broader context
|
||||
- incident marker is the first explicit focal object the eye lands on
|
||||
- user can immediately tell both:
|
||||
- which region is active
|
||||
- which specific event to inspect
|
||||
|
||||
## Failure Modes To Avoid
|
||||
|
||||
### Region Too Strong
|
||||
|
||||
Symptoms:
|
||||
|
||||
- incident markers disappear into the glow
|
||||
- users treat the region center as the main event
|
||||
- the map feels like area flooding instead of event focus
|
||||
|
||||
### Region Too Weak
|
||||
|
||||
Symptoms:
|
||||
|
||||
- incident markers still feel isolated
|
||||
- low-incident periods still look visually empty
|
||||
- users cannot tell where routing activity is generally happening
|
||||
|
||||
### Region Uses Incident Language
|
||||
|
||||
Symptoms:
|
||||
|
||||
- region and incident both look like event markers
|
||||
- users cannot distinguish context from event
|
||||
|
||||
## Final Design Rule
|
||||
|
||||
The desired reading order is:
|
||||
|
||||
1. see the specific incident marker
|
||||
2. feel the active region around it
|
||||
3. understand that collectors and background activity keep the globe alive even during quieter periods
|
||||
|
||||
In one sentence:
|
||||
|
||||
`incident is the point; region is the field.`
|
||||
@@ -1,487 +0,0 @@
|
||||
# BGP Observability Plan
|
||||
|
||||
## Goal
|
||||
|
||||
Build a global routing observability capability on top of:
|
||||
|
||||
- [RIPE RIS Live](https://ris-live.ripe.net/)
|
||||
- [CAIDA BGPStream data access overview](https://bgpstream.caida.org/docs/overview/data-access)
|
||||
|
||||
The target is to support:
|
||||
|
||||
- real-time routing event ingestion
|
||||
- historical replay and baseline analysis
|
||||
- anomaly detection
|
||||
- Earth big-screen visualization
|
||||
|
||||
## Important Scope Note
|
||||
|
||||
These data sources expose the BGP control plane, not user traffic itself.
|
||||
|
||||
That means the system can infer:
|
||||
|
||||
- route propagation direction
|
||||
- prefix reachability changes
|
||||
- AS path changes
|
||||
- visibility changes across collectors
|
||||
|
||||
But it cannot directly measure:
|
||||
|
||||
- exact application traffic volume
|
||||
- exact user packet path
|
||||
- real bandwidth consumption between countries or operators
|
||||
|
||||
Product wording should therefore use phrases like:
|
||||
|
||||
- global routing propagation
|
||||
- route visibility
|
||||
- control-plane anomalies
|
||||
- suspected path diversion
|
||||
|
||||
Instead of claiming direct traffic measurement.
|
||||
|
||||
## Data Source Roles
|
||||
|
||||
### RIS Live
|
||||
|
||||
Use RIS Live as the real-time feed.
|
||||
|
||||
Recommended usage:
|
||||
|
||||
- subscribe to update streams over WebSocket
|
||||
- ingest announcements and withdrawals continuously
|
||||
- trigger low-latency alerts
|
||||
|
||||
Best suited for:
|
||||
|
||||
- hijack suspicion
|
||||
- withdrawal bursts
|
||||
- real-time path changes
|
||||
- live Earth event overlay
|
||||
|
||||
### BGPStream
|
||||
|
||||
Use BGPStream as the historical and replay layer.
|
||||
|
||||
Recommended usage:
|
||||
|
||||
- backfill time windows
|
||||
- build normal baselines
|
||||
- compare current events against history
|
||||
- support investigations and playback
|
||||
|
||||
Best suited for:
|
||||
|
||||
- historical anomaly confirmation
|
||||
- baseline path frequency
|
||||
- visibility baselines
|
||||
- postmortem analysis
|
||||
|
||||
## Recommended Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A["RIS Live WebSocket"] --> B["Realtime Collector"]
|
||||
C["BGPStream Historical Access"] --> D["Backfill Collector"]
|
||||
B --> E["Normalization Layer"]
|
||||
D --> E
|
||||
E --> F["data_snapshots"]
|
||||
E --> G["collected_data"]
|
||||
E --> H["bgp_anomalies"]
|
||||
H --> I["Alerts API"]
|
||||
G --> J["Visualization API"]
|
||||
H --> J
|
||||
J --> K["Earth Big Screen"]
|
||||
```
|
||||
|
||||
## Storage Design
|
||||
|
||||
The current project already has:
|
||||
|
||||
- [data_snapshot.py](/home/ray/dev/linkong/planet/backend/app/models/data_snapshot.py)
|
||||
- [collected_data.py](/home/ray/dev/linkong/planet/backend/app/models/collected_data.py)
|
||||
|
||||
So the lowest-risk path is:
|
||||
|
||||
1. keep raw and normalized BGP events in `collected_data`
|
||||
2. use `data_snapshots` to group each ingest window
|
||||
3. add a dedicated anomaly table for higher-value derived events
|
||||
|
||||
## Proposed Data Types
|
||||
|
||||
### `collected_data`
|
||||
|
||||
Use these `source` values:
|
||||
|
||||
- `ris_live_bgp`
|
||||
- `bgpstream_bgp`
|
||||
|
||||
Use these `data_type` values:
|
||||
|
||||
- `bgp_update`
|
||||
- `bgp_rib`
|
||||
- `bgp_visibility`
|
||||
- `bgp_path_change`
|
||||
|
||||
Recommended stable fields:
|
||||
|
||||
- `source`
|
||||
- `source_id`
|
||||
- `entity_key`
|
||||
- `data_type`
|
||||
- `name`
|
||||
- `reference_date`
|
||||
- `metadata`
|
||||
|
||||
Recommended `entity_key` strategy:
|
||||
|
||||
- event entity: `collector|peer|prefix|event_time`
|
||||
- prefix state entity: `collector|peer|prefix`
|
||||
- origin state entity: `prefix|origin_asn`
|
||||
|
||||
### `metadata` schema for raw events
|
||||
|
||||
Store the normalized event payload in `metadata`:
|
||||
|
||||
```json
|
||||
{
|
||||
"project": "ris-live",
|
||||
"collector": "rrc00",
|
||||
"peer_asn": 3333,
|
||||
"peer_ip": "2001:db8::1",
|
||||
"event_type": "announcement",
|
||||
"prefix": "203.0.113.0/24",
|
||||
"origin_asn": 64496,
|
||||
"as_path": [3333, 64500, 64496],
|
||||
"communities": ["3333:100", "64500:1"],
|
||||
"next_hop": "192.0.2.1",
|
||||
"med": 0,
|
||||
"local_pref": null,
|
||||
"timestamp": "2026-03-26T08:00:00Z",
|
||||
"raw_message": {}
|
||||
}
|
||||
```
|
||||
|
||||
### New anomaly table
|
||||
|
||||
Add a new table, recommended name: `bgp_anomalies`
|
||||
|
||||
Suggested columns:
|
||||
|
||||
- `id`
|
||||
- `snapshot_id`
|
||||
- `task_id`
|
||||
- `source`
|
||||
- `anomaly_type`
|
||||
- `severity`
|
||||
- `status`
|
||||
- `entity_key`
|
||||
- `prefix`
|
||||
- `origin_asn`
|
||||
- `new_origin_asn`
|
||||
- `peer_scope`
|
||||
- `started_at`
|
||||
- `ended_at`
|
||||
- `confidence`
|
||||
- `summary`
|
||||
- `evidence`
|
||||
- `created_at`
|
||||
|
||||
This table should represent derived intelligence, not raw updates.
|
||||
|
||||
## Collector Design
|
||||
|
||||
## 1. `RISLiveCollector`
|
||||
|
||||
Responsibility:
|
||||
|
||||
- maintain WebSocket connection
|
||||
- subscribe to relevant message types
|
||||
- normalize messages
|
||||
- write event batches into snapshots
|
||||
- optionally emit derived anomalies in near real time
|
||||
|
||||
Suggested runtime mode:
|
||||
|
||||
- long-running background task
|
||||
|
||||
Suggested snapshot strategy:
|
||||
|
||||
- one snapshot per rolling time window
|
||||
- for example every 1 minute or every 5 minutes
|
||||
|
||||
## 2. `BGPStreamBackfillCollector`
|
||||
|
||||
Responsibility:
|
||||
|
||||
- fetch historical data windows
|
||||
- normalize to the same schema as real-time data
|
||||
- build baselines
|
||||
- re-run anomaly rules on past windows if needed
|
||||
|
||||
Suggested runtime mode:
|
||||
|
||||
- scheduled task
|
||||
- or ad hoc task for investigations
|
||||
|
||||
Suggested snapshot strategy:
|
||||
|
||||
- one snapshot per historical query window
|
||||
|
||||
## Normalization Rules
|
||||
|
||||
Normalize both sources into the same internal event model.
|
||||
|
||||
Required normalized fields:
|
||||
|
||||
- `collector`
|
||||
- `peer_asn`
|
||||
- `peer_ip`
|
||||
- `event_type`
|
||||
- `prefix`
|
||||
- `origin_asn`
|
||||
- `as_path`
|
||||
- `timestamp`
|
||||
|
||||
Derived normalized fields:
|
||||
|
||||
- `as_path_length`
|
||||
- `country_guess`
|
||||
- `prefix_length`
|
||||
- `is_more_specific`
|
||||
- `visibility_weight`
|
||||
|
||||
## Anomaly Detection Rules
|
||||
|
||||
Start with these five rules first.
|
||||
|
||||
### 1. Origin ASN Change
|
||||
|
||||
Trigger when:
|
||||
|
||||
- the same prefix is announced by a new origin ASN not seen in the baseline window
|
||||
|
||||
Use for:
|
||||
|
||||
- hijack suspicion
|
||||
- origin drift detection
|
||||
|
||||
### 2. More-Specific Burst
|
||||
|
||||
Trigger when:
|
||||
|
||||
- a more-specific prefix appears suddenly
|
||||
- especially from an unexpected origin ASN
|
||||
|
||||
Use for:
|
||||
|
||||
- subprefix hijack suspicion
|
||||
|
||||
### 3. Mass Withdrawal
|
||||
|
||||
Trigger when:
|
||||
|
||||
- the same prefix or ASN sees many withdrawals across collectors within a short window
|
||||
|
||||
Use for:
|
||||
|
||||
- outage suspicion
|
||||
- regional incident detection
|
||||
|
||||
### 4. Path Deviation
|
||||
|
||||
Trigger when:
|
||||
|
||||
- AS path length jumps sharply
|
||||
- or a rarely seen transit ASN appears
|
||||
- or path frequency drops below baseline norms
|
||||
|
||||
Use for:
|
||||
|
||||
- route leak suspicion
|
||||
- unusual path diversion
|
||||
|
||||
### 5. Visibility Drop
|
||||
|
||||
Trigger when:
|
||||
|
||||
- a prefix is visible from far fewer collectors/peers than its baseline
|
||||
|
||||
Use for:
|
||||
|
||||
- regional reachability degradation
|
||||
|
||||
## Baseline Strategy
|
||||
|
||||
Use BGPStream historical data to build:
|
||||
|
||||
- common origin ASN per prefix
|
||||
- common AS path patterns
|
||||
- collector visibility distribution
|
||||
- normal withdrawal frequency
|
||||
|
||||
Recommended baseline windows:
|
||||
|
||||
- short baseline: last 24 hours
|
||||
- medium baseline: last 7 days
|
||||
- long baseline: last 30 days
|
||||
|
||||
The first implementation can start with only the 7-day baseline.
|
||||
|
||||
## API Design
|
||||
|
||||
### Raw event API
|
||||
|
||||
Add endpoints like:
|
||||
|
||||
- `GET /api/v1/bgp/events`
|
||||
- `GET /api/v1/bgp/events/{id}`
|
||||
|
||||
Suggested filters:
|
||||
|
||||
- `prefix`
|
||||
- `origin_asn`
|
||||
- `peer_asn`
|
||||
- `collector`
|
||||
- `event_type`
|
||||
- `time_from`
|
||||
- `time_to`
|
||||
- `source`
|
||||
|
||||
### Anomaly API
|
||||
|
||||
Add endpoints like:
|
||||
|
||||
- `GET /api/v1/bgp/anomalies`
|
||||
- `GET /api/v1/bgp/anomalies/{id}`
|
||||
- `GET /api/v1/bgp/anomalies/summary`
|
||||
|
||||
Suggested filters:
|
||||
|
||||
- `severity`
|
||||
- `anomaly_type`
|
||||
- `status`
|
||||
- `prefix`
|
||||
- `origin_asn`
|
||||
- `time_from`
|
||||
- `time_to`
|
||||
|
||||
### Visualization API
|
||||
|
||||
Add an Earth-oriented endpoint like:
|
||||
|
||||
- `GET /api/v1/visualization/geo/bgp-anomalies`
|
||||
|
||||
Recommended feature shapes:
|
||||
|
||||
- point: collector locations
|
||||
- arc: inferred propagation or suspicious path edge
|
||||
- pulse point: active anomaly hotspot
|
||||
|
||||
## Earth Big-Screen Design
|
||||
|
||||
Recommended layers:
|
||||
|
||||
### Layer 1: Collector layer
|
||||
|
||||
Show known collector locations and current activity intensity.
|
||||
|
||||
### Layer 2: Route propagation arcs
|
||||
|
||||
Use arcs for:
|
||||
|
||||
- origin ASN country to collector country
|
||||
- or collector-to-collector visibility edges
|
||||
|
||||
Important note:
|
||||
|
||||
This is an inferred propagation view, not real packet flow.
|
||||
|
||||
### Layer 3: Active anomaly overlay
|
||||
|
||||
Show:
|
||||
|
||||
- hijack suspicion in red
|
||||
- mass withdrawal in orange
|
||||
- visibility drop in yellow
|
||||
- path deviation in blue
|
||||
|
||||
### Layer 4: Time playback
|
||||
|
||||
Use `data_snapshots` to replay:
|
||||
|
||||
- minute-by-minute route changes
|
||||
- anomaly expansion
|
||||
- recovery timeline
|
||||
|
||||
## Alerting Strategy
|
||||
|
||||
Map anomaly severity to the current alert system.
|
||||
|
||||
Recommended severity mapping:
|
||||
|
||||
- `critical`
|
||||
- likely hijack
|
||||
- very large withdrawal burst
|
||||
- `high`
|
||||
- clear origin change
|
||||
- large visibility drop
|
||||
- `medium`
|
||||
- unusual path change
|
||||
- moderate more-specific burst
|
||||
- `low`
|
||||
- weak or localized anomalies
|
||||
|
||||
## Delivery Plan
|
||||
|
||||
### Phase 1
|
||||
|
||||
- add `RISLiveCollector`
|
||||
- normalize updates into `collected_data`
|
||||
- create `bgp_anomalies`
|
||||
- implement 3 rules:
|
||||
- origin change
|
||||
- more-specific burst
|
||||
- mass withdrawal
|
||||
|
||||
### Phase 2
|
||||
|
||||
- add `BGPStreamBackfillCollector`
|
||||
- build 7-day baseline
|
||||
- implement:
|
||||
- path deviation
|
||||
- visibility drop
|
||||
|
||||
### Phase 3
|
||||
|
||||
- add Earth visualization layer
|
||||
- add time playback
|
||||
- add anomaly filtering and drilldown
|
||||
|
||||
## Practical Implementation Notes
|
||||
|
||||
- Start with IPv4 first, then add IPv6 after the event schema is stable.
|
||||
- Store the original raw payload in `metadata.raw_message` for traceability.
|
||||
- Deduplicate events by a stable hash of collector, peer, prefix, type, and timestamp.
|
||||
- Keep anomaly generation idempotent so replay and backfill do not create duplicate alerts.
|
||||
- Expect noisy data and partial views; confidence scoring matters.
|
||||
|
||||
## Recommended First Patch Set
|
||||
|
||||
The first code milestone should include:
|
||||
|
||||
1. `backend/app/services/collectors/ris_live.py`
|
||||
2. `backend/app/services/collectors/bgpstream.py`
|
||||
3. `backend/app/models/bgp_anomaly.py`
|
||||
4. `backend/app/api/v1/bgp.py`
|
||||
5. `backend/app/api/v1/visualization.py`
|
||||
add BGP anomaly geo endpoint
|
||||
6. `frontend/src/pages`
|
||||
add a BGP anomaly list or summary page
|
||||
7. `frontend/public/earth/js`
|
||||
add BGP anomaly rendering layer
|
||||
|
||||
## Sources
|
||||
|
||||
- [RIPE RIS Live](https://ris-live.ripe.net/)
|
||||
- [CAIDA BGPStream Data Access Overview](https://bgpstream.caida.org/docs/overview/data-access)
|
||||
@@ -1,97 +0,0 @@
|
||||
# News Live Streams Collector Format
|
||||
|
||||
`news_live_streams` 采集器面向“频道目录 JSON”输入,而不是直接抓网页。
|
||||
|
||||
这样做的目标是:
|
||||
|
||||
- 让后台能够稳定接入世界各地新闻直播源
|
||||
- 让 `Earth` 页面电视模块始终消费统一结构
|
||||
- 便于后续接入类似 `worldmonitor` 那种 YouTube / HLS / iframe 混合频道目录
|
||||
|
||||
## 推荐 JSON 结构
|
||||
|
||||
```json
|
||||
{
|
||||
"sources": [
|
||||
{
|
||||
"id": "bbc-world-news",
|
||||
"name": "BBC World News",
|
||||
"provider": "BBC",
|
||||
"region": "UK",
|
||||
"language": "en",
|
||||
"source_type": "youtube",
|
||||
"youtube_video_id": "dQw4w9WgXcQ",
|
||||
"youtube_channel": "https://www.youtube.com/@BBCNews",
|
||||
"embed_url": "",
|
||||
"stream_url": "",
|
||||
"homepage_url": "https://www.youtube.com/@BBCNews/live",
|
||||
"poster_url": "",
|
||||
"sort_order": 220,
|
||||
"is_enabled": true,
|
||||
"notes": "Primary English global news channel"
|
||||
},
|
||||
{
|
||||
"id": "france24-en",
|
||||
"name": "France 24 English",
|
||||
"provider": "France 24",
|
||||
"region": "France",
|
||||
"language": "en",
|
||||
"source_type": "hls",
|
||||
"stream_url": "https://example.com/live.m3u8",
|
||||
"homepage_url": "https://www.france24.com/en/live",
|
||||
"sort_order": 230,
|
||||
"is_enabled": true
|
||||
},
|
||||
{
|
||||
"id": "cctv4-page",
|
||||
"name": "CCTV-4 中文国际",
|
||||
"provider": "CCTV",
|
||||
"region": "China",
|
||||
"language": "zh-CN",
|
||||
"source_type": "iframe",
|
||||
"embed_url": "https://tv.cctv.com/live/cctv4/",
|
||||
"homepage_url": "https://tv.cctv.com/live/cctv4/",
|
||||
"sort_order": 10,
|
||||
"is_enabled": true
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## 字段约定
|
||||
|
||||
- `id`: 唯一标识,建议稳定不变
|
||||
- `name`: 频道显示名
|
||||
- `provider`: 提供方
|
||||
- `region`: 国家或地区
|
||||
- `language`: 语言代码
|
||||
- `source_type`: `iframe` / `hls` / `video` / `external` / `youtube`
|
||||
- `embed_url`: 适合 iframe 内嵌的页面
|
||||
- `stream_url`: 直接视频流地址
|
||||
- `homepage_url`: 官网或频道页
|
||||
- `youtube_video_id`: YouTube 直播视频 ID
|
||||
- `youtube_channel`: YouTube 频道 handle 或频道 URL
|
||||
- `poster_url`: 封面图,可选
|
||||
- `sort_order`: 排序值,越小越靠前
|
||||
- `is_enabled`: 是否启用
|
||||
- `notes`: 简短备注
|
||||
|
||||
## 面板行为约定
|
||||
|
||||
- `youtube`
|
||||
- 优先使用 `youtube_video_id`
|
||||
- 无法内嵌时至少保留 `youtube_channel` 或 `homepage_url` 供外部打开
|
||||
- `hls` / `video`
|
||||
- 优先走 `stream_url`
|
||||
- `iframe`
|
||||
- 优先走 `embed_url`
|
||||
- `external`
|
||||
- 不尝试内嵌,只保留外部打开
|
||||
|
||||
## 当前实现状态
|
||||
|
||||
- 后台设置页可以手工维护频道目录
|
||||
- `Earth` 电视模块会合并:
|
||||
- 手工配置源
|
||||
- `news_live_streams` 采集器采集源
|
||||
- 当前默认兜底源为 `CCTV-4 中文国际`
|
||||
@@ -1,309 +0,0 @@
|
||||
# Frontend Layout Guidelines
|
||||
|
||||
本项目后台页面默认遵循“单屏工作区”布局规范。目标不是让页面永远不溢出,而是确保在常见桌面视口下:
|
||||
|
||||
- 页面主结构能在一屏内看清
|
||||
- 用户能同时看到页头、摘要区和主工作区
|
||||
- 超出的内容在模块内部滚动,而不是把整页纵向撑爆
|
||||
|
||||
当前推荐参考实现:
|
||||
|
||||
- [frontend/src/pages/BGP/BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx)
|
||||
- [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css)
|
||||
|
||||
## 核心原则
|
||||
|
||||
### 1. 页面优先保证一屏工作区
|
||||
|
||||
管理页默认采用:
|
||||
|
||||
- 页头:标题、说明、主要操作
|
||||
- 主工作区:统计卡、表格、图表、列表、标签页
|
||||
|
||||
推荐结构:
|
||||
|
||||
```tsx
|
||||
<AppLayout>
|
||||
<div className="page-shell">
|
||||
<div className="page-shell__header">...</div>
|
||||
<div className="page-shell__body">...</div>
|
||||
</div>
|
||||
</AppLayout>
|
||||
```
|
||||
|
||||
页面总高度应被限制在 `AppLayout` 内容区内,而不是继续让整个页面自然向下增长。
|
||||
|
||||
### 2. 滚动优先发生在模块内部
|
||||
|
||||
如果表格、日志、长列表、图表明细超出空间:
|
||||
|
||||
- 让卡片内部滚动
|
||||
- 让表格内部滚动
|
||||
- 让标签页内容区内部滚动
|
||||
|
||||
不要默认依赖整个页面滚动去“解决”空间问题。
|
||||
|
||||
### 3. 主工作区必须拿到主要空间
|
||||
|
||||
页面里最重要的模块必须是视觉和空间上的主角。通常应保证:
|
||||
|
||||
- 页头始终可见
|
||||
- 摘要区高度被控制
|
||||
- 主表格 / 主图表 / 主分析区占据 50% 以上可视高度
|
||||
|
||||
如果一个页面有多个大模块,优先顺序是:
|
||||
|
||||
1. 先压缩说明区和摘要区
|
||||
2. 再把次级模块收进标签页或切换视图
|
||||
3. 最后才考虑继续增加整页滚动
|
||||
|
||||
### 4. 小屏幕和高缩放必须进入紧凑模式
|
||||
|
||||
在窗口高度较低、宽度较窄、或系统缩放较高时,应主动切换紧凑布局,例如:
|
||||
|
||||
- 缩小卡片 padding
|
||||
- 缩小表头和单元格间距
|
||||
- 将摘要区改为更紧凑的单行/横向滚动布局
|
||||
- 将次级模块移入标签页、抽屉、折叠区
|
||||
|
||||
紧凑模式的目标是保持可用,不是单纯把文字和控件一股脑缩小。
|
||||
|
||||
### 5. overflow 责任必须明确
|
||||
|
||||
页面中的大块内容必须明确:
|
||||
|
||||
- 谁负责占满剩余高度
|
||||
- 谁负责裁剪
|
||||
- 谁负责滚动
|
||||
|
||||
常见要求:
|
||||
|
||||
- 父容器链路需要 `min-height: 0`
|
||||
- 工作区容器通常需要 `display: flex`
|
||||
- 真正的滚动节点要显式 `overflow: auto`
|
||||
|
||||
### 6. 卡片不能被压到不可读
|
||||
|
||||
历史上我们反复踩到的问题不是“没有滚动条”,而是:
|
||||
|
||||
- 卡片被 `flex` 压缩得只剩一小条可视区域
|
||||
- 文字能渲染,但读不完整
|
||||
- 内容其实存在,却被 `overflow: hidden` 裁掉
|
||||
|
||||
因此后续约束是:
|
||||
|
||||
- 先保证卡片有可读的最小高度
|
||||
- 如果继续压缩会影响阅读,就切换成内部滚动
|
||||
- 不要为了“保持一屏”而把正文、表格、描述区压成无法阅读的条状区域
|
||||
|
||||
### 7. Tabs 不是天然安全的布局容器
|
||||
|
||||
历史上 Tabs 相关回归非常多,典型问题包括:
|
||||
|
||||
- 隐藏 tab pane 因为自定义 `display: flex` 而重新露出来
|
||||
- 所有 tab 被强行套用同一套高度/overflow 规则
|
||||
- 表格 tab 能工作,但 markdown / help / diagnostics tab 被压坏
|
||||
|
||||
因此约束是:
|
||||
|
||||
- `Tabs` 里的每类内容都要单独定义自己的布局策略
|
||||
- 表格 tab 可以是“固定高度 + 内部滚动”
|
||||
- 文档/Markdown tab 更适合“tab pane 自身滚动 + 内容正常文档流”
|
||||
- 如果覆盖组件库样式,必须同时检查 hidden 状态是否仍然成立
|
||||
|
||||
### 8. 摘要区优先进入紧凑模式,而不是挤压正文
|
||||
|
||||
历史经验表明,最容易被误处理的是顶部摘要卡:
|
||||
|
||||
- 它们经常为了“都放下”被强行压窄
|
||||
- 然后正文、表格、AI 结果区一起失去主空间
|
||||
|
||||
后续统一约束:
|
||||
|
||||
- 小屏或高缩放时,摘要卡优先:
|
||||
- 降低 padding
|
||||
- 改成横向滚动
|
||||
- 改成更紧凑的网格
|
||||
- 不要优先牺牲主工作区的可视面积
|
||||
|
||||
### 9. 长文档类内容优先保证阅读体验
|
||||
|
||||
像下面这些内容,不能直接套用“表格工作区”的逻辑:
|
||||
|
||||
- AI 简报
|
||||
- 运行日志
|
||||
- 原始 JSON
|
||||
- 帮助说明
|
||||
- 多段描述性文本
|
||||
|
||||
这些区域应该优先满足:
|
||||
|
||||
- 标题和元信息稳定可见
|
||||
- 正文有明确的最小可读高度
|
||||
- 正文滚动策略单独定义
|
||||
- 支持 Markdown 表格、分隔线、引用、代码块等结构
|
||||
|
||||
### 10. 高度关键路径要少包一层
|
||||
|
||||
历史上不少滚动问题不是组件本身错,而是多包了一层之后:
|
||||
|
||||
- 高度链路断掉
|
||||
- `min-height: 0` 没传下去
|
||||
- `overflow` 责任被吃掉
|
||||
|
||||
因此:
|
||||
|
||||
- 对高度关键区域,优先使用最直接的 DOM 结构
|
||||
- 使用 `Space`、额外包装 `div`、第三方布局容器时,要确认它们不会改变滚动和高度语义
|
||||
- 如果一个区域已经出现“内容明明有,但只剩一条缝”,优先怀疑中间包装层
|
||||
|
||||
## 历史坑位总结
|
||||
|
||||
从 Earth、Playground、BGP、DataSources 这些页面的 bugfix 可以归纳出几类高频坑:
|
||||
|
||||
### 1. 用 `overflow: hidden` 掩盖布局问题
|
||||
|
||||
表面上看页面“整齐了”,实际上会导致:
|
||||
|
||||
- 内容被裁掉
|
||||
- tab 内容只剩一条缝
|
||||
- 面板明明渲染成功,但用户看不见
|
||||
|
||||
正确做法:
|
||||
|
||||
- 让真正的内容节点滚动
|
||||
- 不要让上层容器无差别裁剪所有子内容
|
||||
|
||||
### 2. 把所有 tab 当成同一种内容
|
||||
|
||||
表格、Markdown、帮助卡、日志流的空间需求完全不同。
|
||||
|
||||
正确做法:
|
||||
|
||||
- 表格:固定工作区 + 内部滚动
|
||||
- 文档:普通流式内容 + pane 级滚动
|
||||
- 侧边说明:内容驱动高度,不强行拉满
|
||||
|
||||
### 3. 只做视觉缩小,不做空间重分配
|
||||
|
||||
这会导致:
|
||||
|
||||
- 卡片文字被截断
|
||||
- 表格只剩 1 到 2 行
|
||||
- 按钮和筛选区挤成一团
|
||||
|
||||
正确做法:
|
||||
|
||||
- 紧凑模式优先重排
|
||||
- 横向滚动摘要区
|
||||
- 折叠/收纳次级模块
|
||||
|
||||
### 4. 父容器高度链不完整
|
||||
|
||||
这是最常见的内部滚动失效原因。
|
||||
|
||||
检查顺序:
|
||||
|
||||
1. 外层是否真的有确定高度
|
||||
2. flex 父容器是否带了 `min-height: 0`
|
||||
3. 真正滚动节点是否明确 `overflow: auto`
|
||||
4. 中间包装层是否偷偷改了布局语义
|
||||
|
||||
### 5. UI 状态和显示状态不同步
|
||||
|
||||
Earth 相关改动里反复出现:
|
||||
|
||||
- 图层隐藏了,但 hover/lock 还在
|
||||
- tooltip 还在显示旧对象
|
||||
- legend 没跟着切换
|
||||
|
||||
这类约束同样适用于后台页面:
|
||||
|
||||
- 被隐藏、卸载、切换出视图的内容,不应继续保留活跃交互状态
|
||||
|
||||
## 推荐实现模式
|
||||
|
||||
### 页面骨架
|
||||
|
||||
优先复用项目里已有的通用结构:
|
||||
|
||||
- `.dashboard-content-inner`
|
||||
- `.page-shell`
|
||||
- `.page-shell__header`
|
||||
- `.page-shell__body`
|
||||
- `.table-scroll-region`
|
||||
|
||||
不要每个页面都重新发明一套完全不同的高度和滚动语义。
|
||||
|
||||
### 表格工作区
|
||||
|
||||
推荐模式:
|
||||
|
||||
```tsx
|
||||
<Card>
|
||||
<div className="table-scroll-region" ref={tableRegionRef}>
|
||||
<Table
|
||||
pagination={false}
|
||||
scroll={{ x: 1200, y: tableHeight }}
|
||||
/>
|
||||
</div>
|
||||
</Card>
|
||||
```
|
||||
|
||||
要求:
|
||||
|
||||
- 表格尽量在卡片内部滚动
|
||||
- `scroll.y` 应来自实际可用高度估算,而不是完全静态的魔法数字
|
||||
- 父容器链路要保证 header、body、content 的 overflow 都在表格内部闭合
|
||||
|
||||
### 多模块页面
|
||||
|
||||
如果一个页面同时有:
|
||||
|
||||
- 摘要卡
|
||||
- 表格
|
||||
- 异常明细
|
||||
- 最近事件
|
||||
|
||||
不建议简单纵向堆叠全部模块。优先使用:
|
||||
|
||||
- 顶部摘要 + 底部单一主工作区
|
||||
- 标签页切换多个次级数据视图
|
||||
- 左右分栏,并保证每栏内部独立滚动
|
||||
|
||||
## 不推荐的做法
|
||||
|
||||
以下模式默认视为不符合本项目页面规范:
|
||||
|
||||
- 依赖整页纵向滚动来显示主要工作区
|
||||
- 一个页面纵向堆 3 到 4 个大卡片,每个都想完整展示
|
||||
- 表格没有内部滚动,导致缩放后只能看到 1 到 2 行数据
|
||||
- 父容器缺少 `min-height: 0`,导致内部滚动失效
|
||||
- 只做视觉缩小,不处理真正的空间分配
|
||||
|
||||
## 页面验收检查清单
|
||||
|
||||
提交前至少检查:
|
||||
|
||||
- 页头、摘要区、主工作区能否同时出现
|
||||
- 主工作区是否拿到了页面中最多的高度
|
||||
- 表格或明细溢出时,滚动条是否出现在模块内部
|
||||
- 卡片是否被压缩到文字显示不完整;如果会,是否已经切换为内部滚动
|
||||
- 浏览器缩放到 `125%` / `150%` 时是否仍可用
|
||||
- 低高度窗口下是否还保有合理的可见内容行数
|
||||
- Tabs、Card、Table 在 overflow 时是否仍可操作
|
||||
- 非表格 tab(Markdown、帮助说明、日志)是否有独立且合理的滚动策略
|
||||
|
||||
## 落地顺序
|
||||
|
||||
后续新增或重构后台页时,优先按这个顺序设计:
|
||||
|
||||
1. 先定义主工作区
|
||||
2. 再确定哪些模块必须常驻可见
|
||||
3. 最后再做样式和视觉层次
|
||||
|
||||
简单说:
|
||||
|
||||
- 先保证空间分配正确
|
||||
- 再处理滚动边界
|
||||
- 最后再做美化
|
||||
@@ -1,165 +0,0 @@
|
||||
# HUD Panel Component Plan
|
||||
|
||||
## Goal
|
||||
|
||||
Unify Earth HUD panels into a reusable component layer so new panels can share:
|
||||
|
||||
- a consistent shell
|
||||
- a consistent header
|
||||
- a consistent action-button system
|
||||
- a consistent body and collapse pattern
|
||||
|
||||
## Scope
|
||||
|
||||
Target panels:
|
||||
|
||||
- `tv-panel`
|
||||
- `news-panel`
|
||||
- `legend`
|
||||
- `layer-panel`
|
||||
- `earth-stats`
|
||||
- `info-card`
|
||||
- settings modal header/actions
|
||||
|
||||
## Component Model
|
||||
|
||||
### Base shell
|
||||
|
||||
- `.hud-panel`
|
||||
- `.hud-panel--compact`
|
||||
- `.hud-panel--media`
|
||||
- `.hud-panel--collapsed`
|
||||
- `.hud-panel-hidden`
|
||||
- `.hud-panel.is-dragging`
|
||||
- `.hud-panel.is-layout-animating`
|
||||
|
||||
### Header
|
||||
|
||||
- `.hud-panel__header`
|
||||
- `.hud-panel__title-group`
|
||||
- `.hud-panel__title`
|
||||
- `.hud-panel__subtitle`
|
||||
- `.hud-panel__chip`
|
||||
- `.hud-panel__actions`
|
||||
|
||||
Header baseline rule:
|
||||
|
||||
- Header title styling is fixed by the component layer and should not drift per panel
|
||||
- Title font size, font weight, letter spacing, line height, text color, and vertical alignment come from the shared header tokens and structure
|
||||
- Header divider, border treatment, inner spacing, and title-to-actions alignment are part of the same shared baseline
|
||||
- Panel-specific header differences should be limited to explicit variants such as `compact` or `media`, or token overrides with documented intent
|
||||
- “Looks close enough” local header overrides should be treated as temporary compatibility code and removed during migration
|
||||
|
||||
### Actions
|
||||
|
||||
- `.hud-panel__action`
|
||||
- `.hud-panel__action--icon`
|
||||
- `.hud-panel__action--collapse`
|
||||
- `.hud-panel__action--close`
|
||||
- `.hud-panel__action--refresh`
|
||||
- `.hud-panel__action--external`
|
||||
|
||||
Action-button baseline rule:
|
||||
|
||||
- Header action buttons must have one fixed default style baseline across all HUD panels
|
||||
- Default width behavior, padding, icon size, radius, alignment, hover, and active feedback all come from `.hud-panel__action`
|
||||
- Panel-specific differences must be expressed through explicit variants or token overrides, not ad-hoc local button rewrites
|
||||
- `close` buttons are part of the same default action system and must not silently fall back to a separate legacy box model
|
||||
|
||||
### Body
|
||||
|
||||
- `.hud-panel__body`
|
||||
- `.hud-panel__body--scroll`
|
||||
- `.hud-panel__body--collapsible`
|
||||
|
||||
### Collapse behavior
|
||||
|
||||
- `.hud-panel--collapsed`
|
||||
- `.hud-panel--expand-up`
|
||||
- `.hud-panel--expand-down`
|
||||
|
||||
Adaptive collapse / expand rule:
|
||||
|
||||
- HUD panels support two expansion directions:
|
||||
- top-to-bottom expansion
|
||||
- bottom-to-top expansion
|
||||
- Expansion direction should be decided at runtime from available viewport space rather than hardcoded per panel
|
||||
- Use:
|
||||
- `d` = available distance from the header anchor to the viewport bottom edge
|
||||
- `h` = expected expanded panel height
|
||||
- buffer = `20px`
|
||||
- Collapsed-state direction rule:
|
||||
- if `d > h + 20px`, the next action direction is `expand-up`
|
||||
- if `d <= h + 20px`, the next action direction is `expand-down`
|
||||
- To avoid jitter around the threshold, the shared controller should keep a small hysteresis band:
|
||||
- if the current direction is already `up`, keep it until `d <= h`
|
||||
- if the current direction is already `down`, keep it until `d > h + 20px`
|
||||
- The opposite edge is still a safety guard:
|
||||
- if the chosen side cannot fit at all, fall back to the other side if it can fit
|
||||
- if neither side fully fits, choose the side with more space and let the body scroll
|
||||
- If neither direction fully fits, choose the direction with more available space and let the body scroll
|
||||
- Collapse icon direction must match the active expansion direction so the icon always describes the real open/close motion
|
||||
- The collapse icon describes the next action, not the current state
|
||||
- This mapping is fixed component behavior and must not drift per panel:
|
||||
- collapsed + expand-down => `expand_more`
|
||||
- expanded + expand-down => `expand_less`
|
||||
- collapsed + expand-up => `expand_less`
|
||||
- expanded + expand-up => `expand_more`
|
||||
- Panels must not combine icon-name swapping with extra CSS rotation for the same collapse control
|
||||
- Expansion direction and icon direction must come from one shared source of truth in the component controller
|
||||
- The direction decision should be recomputed when opening, resizing the viewport, or restoring a dragged panel near another edge
|
||||
|
||||
## Tokens
|
||||
|
||||
Promote panel differences into CSS variables instead of duplicating selectors:
|
||||
|
||||
- `--hud-panel-padding`
|
||||
- `--hud-header-padding`
|
||||
- `--hud-header-gap`
|
||||
- `--hud-action-padding`
|
||||
- `--hud-action-gap`
|
||||
- `--hud-action-icon-size`
|
||||
- `--hud-body-gap`
|
||||
- `--hud-body-max-height`
|
||||
- `--hud-chip-radius`
|
||||
- `--hud-title-font-size`
|
||||
- `--hud-title-font-weight`
|
||||
- `--hud-title-letter-spacing`
|
||||
- `--hud-title-line-height`
|
||||
- `--hud-title-color`
|
||||
- `--hud-header-border-color`
|
||||
- `--hud-header-divider-opacity`
|
||||
- `--hud-expand-direction`
|
||||
|
||||
## Migration Order
|
||||
|
||||
1. Build the shared component layer in `frontend/public/earth/css/hud.css`
|
||||
2. Migrate `tv-panel` and `news-panel` first as the reference implementation
|
||||
3. Migrate `legend` and `layer-panel` into a compact variant
|
||||
4. Migrate `earth-stats` and `info-card`
|
||||
5. Align settings modal header/actions with the same action system
|
||||
6. Remove legacy one-off button selectors after verification
|
||||
|
||||
## Guardrails
|
||||
|
||||
- Do not change panel behavior and data flow during the first pass
|
||||
- Keep old class names temporarily as compatibility hooks
|
||||
- Prefer variable overrides over per-panel reimplementation
|
||||
- Treat header action-button default styling as fixed component API, not per-panel design space
|
||||
- Treat header title typography, border, and divider styling as fixed component API, not per-panel design space
|
||||
- Treat collapse direction as a component behavior contract, not a one-off panel trick
|
||||
- Treat collapse icon semantics as a component behavior contract, not a per-panel visual preference
|
||||
- Verify header alignment and drag/collapse behavior after each migration batch
|
||||
|
||||
## First Implementation Batch
|
||||
|
||||
Batch 1 should only do:
|
||||
|
||||
- shared header structure
|
||||
- shared action-button system
|
||||
- shared title typography and header border/divider baseline
|
||||
- shared collapsible body pattern
|
||||
- adaptive collapse direction logic and direction-aware collapse icons
|
||||
- migration of `tv-panel` and `news-panel`
|
||||
|
||||
That keeps risk low while giving the rest of the HUD a stable target to migrate toward.
|
||||
@@ -1,105 +0,0 @@
|
||||
# Docker + Compose + Buildx 升级教程
|
||||
|
||||
流程:删除旧版 -> 安装新版 -> 验证
|
||||
|
||||
---
|
||||
|
||||
# 1. 删除旧版本
|
||||
|
||||
## 删除 apt 安装的旧包
|
||||
|
||||
```bash
|
||||
sudo apt remove -y docker.io docker-compose docker-compose-v2 docker-doc podman-docker containerd runc
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 删除系统中的 `docker-compose`(V1)
|
||||
|
||||
```bash
|
||||
sudo rm -f "$(which docker-compose 2>/dev/null)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 查找并删除手动安装的 Buildx 插件
|
||||
|
||||
```bash
|
||||
docker info | sed -n '/Plugins:/,/^ Server:/p' | grep -A2 buildx
|
||||
```
|
||||
|
||||
从输出中获取 `Path`,然后执行:
|
||||
|
||||
```bash
|
||||
rm -f <Path中对应的docker-buildx文件>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 清理无用依赖
|
||||
|
||||
```bash
|
||||
sudo apt autoremove -y
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# 2. 安装 Docker 官方版本
|
||||
|
||||
包含 Docker Engine、Docker Compose 插件、Docker Buildx 插件。
|
||||
|
||||
## 安装依赖
|
||||
|
||||
```bash
|
||||
sudo apt update
|
||||
sudo apt install -y ca-certificates curl gnupg
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 添加 Docker GPG key
|
||||
|
||||
```bash
|
||||
sudo install -m 0755 -d /etc/apt/keyrings
|
||||
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | \
|
||||
sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
|
||||
sudo chmod a+r /etc/apt/keyrings/docker.gpg
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 添加官方仓库
|
||||
|
||||
```bash
|
||||
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
|
||||
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 安装 Docker + Compose + Buildx
|
||||
|
||||
```bash
|
||||
sudo apt update
|
||||
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# 3. 验证安装
|
||||
|
||||
```bash
|
||||
docker --version
|
||||
docker compose version
|
||||
docker buildx version
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# 4. 常用命令
|
||||
|
||||
```bash
|
||||
docker compose up -d
|
||||
docker compose down
|
||||
docker buildx build .
|
||||
```
|
||||
38
docs/plans/README.md
Normal file
38
docs/plans/README.md
Normal file
@@ -0,0 +1,38 @@
|
||||
# Plans Docs
|
||||
|
||||
这里放“未来实施方案和未完成计划”的文档,重点回答:
|
||||
|
||||
- 我们准备做什么
|
||||
- 为什么要做
|
||||
- 分几期做
|
||||
- 当前差距和下一步是什么
|
||||
|
||||
适合放入这里的内容:
|
||||
|
||||
- Earth / BGP / 地形 / 天球实施方案
|
||||
- AI Playground 发展计划
|
||||
- backend / datasource / agent roadmap
|
||||
- UE5 MVP 方案
|
||||
|
||||
当前重点入口:
|
||||
|
||||
- [earth-mobile-drawer-ui-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-mobile-drawer-ui-plan.md)
|
||||
- [earth-compute-center-bgp-style-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-compute-center-bgp-style-plan.md)
|
||||
- [earth-renderer-architecture-separation-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-renderer-architecture-separation-plan.md)
|
||||
- [earth-country-boundary-overlay-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-country-boundary-overlay-plan.md)
|
||||
- [earth-predicted-orbit-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-predicted-orbit-plan.md)
|
||||
- [earth-webgl-instancing-satellites-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-webgl-instancing-satellites-plan.md)
|
||||
- [earth-real-terrain-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-real-terrain-plan.md)
|
||||
- [earth-news-source-configuration-and-collector-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-news-source-configuration-and-collector-plan.md)
|
||||
- [frontend-ai-playground-development-plan.md](/home/ray/dev/linkong/planet/docs/plans/frontend-ai-playground-development-plan.md)
|
||||
- [ue5-mvp-fused-plan.md](/home/ray/dev/linkong/planet/docs/plans/ue5-mvp-fused-plan.md)
|
||||
|
||||
不适合放入这里的内容:
|
||||
|
||||
- 当前代码结构说明
|
||||
- 组件现状和实现入口
|
||||
- 已经落地的技术上下文说明
|
||||
|
||||
这些应放入:
|
||||
|
||||
- [docs/technical/README.md](/home/ray/dev/linkong/planet/docs/technical/README.md)
|
||||
@@ -10,9 +10,9 @@ This document connects three existing planning threads into one implementation r
|
||||
|
||||
Related documents:
|
||||
|
||||
- [aiprovider](/home/ray/dev/linkong/planet/docs/aiprovider.md)
|
||||
- [datasource-health-plan](/home/ray/dev/linkong/planet/docs/datasource-health-plan.md)
|
||||
- [agent-architecture-plan](/home/ray/dev/linkong/planet/docs/agent-architecture-plan.md)
|
||||
- [aiprovider](/home/ray/dev/linkong/planet/docs/technical/agents-aiprovider.md)
|
||||
- [datasource-health-plan](/home/ray/dev/linkong/planet/docs/plans/agents-datasource-health-plan.md)
|
||||
- [agent-architecture-plan](/home/ray/dev/linkong/planet/docs/plans/agents-agent-architecture-plan.md)
|
||||
|
||||
|
||||
## Big Picture
|
||||
@@ -17,7 +17,7 @@ It is an aggregation/view-model layer:
|
||||
|
||||
## Why This Layer Exists
|
||||
|
||||
Current product gap from [bgp-context.md](/home/ray/dev/linkong/planet/docs/earth/bgp-context.md):
|
||||
Current product gap from [bgp-context.md](/home/ray/dev/linkong/planet/docs/technical/earth-bgp-context.md):
|
||||
|
||||
- incident density is naturally low
|
||||
- anomaly density is higher, but still not enough to keep the globe expressive all the time
|
||||
@@ -290,7 +290,7 @@ Each feature should include:
|
||||
|
||||
## Earth Rendering Plan
|
||||
|
||||
Detailed visual layering guidance is expanded in [bgp-earth-rendering-plan.md](/home/ray/dev/linkong/planet/docs/earth/bgp-earth-rendering-plan.md).
|
||||
Detailed visual layering guidance is expanded in [bgp-earth-rendering-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-bgp-earth-rendering-plan.md).
|
||||
|
||||
### Layer Relationship
|
||||
|
||||
@@ -112,6 +112,285 @@
|
||||
|
||||
- [Three.js SpriteMaterial](https://threejs.org/docs/pages/SpriteMaterial.html)
|
||||
|
||||
## 天球背景资源与星体数据来源
|
||||
|
||||
为避免把“视觉背景”和“可计算天体位置”混为一谈,本方案明确分成两类资源:
|
||||
|
||||
### 1. 背景资源:全天星图贴图
|
||||
|
||||
用于 Phase 1 的“真实天空背景”。
|
||||
|
||||
推荐优先来源:
|
||||
|
||||
- NASA SVS 的 Tycho 全天星图
|
||||
- [The Tycho Catalog Skymap - Version 2.0](https://svs.gsfc.nasa.gov/3572/)
|
||||
- NASA Deep Star Maps 2020
|
||||
- SatelliteMap.space 在 credits 中明确提到其使用了 `NASA Deep Star Maps 2020 - High-resolution star field (1.7 billion stars from Gaia DR2)` 作为星空视觉资源
|
||||
- 这说明行业内成熟实现并不一定直接渲染全部星表点,而很可能先使用一张高质量官方深空星图作为背景层
|
||||
- 如需后续替换,也可评估 ESA / Gaia 的全天 sky map 资源
|
||||
- [Gaia DR3 stories](https://www.cosmos.esa.int/web/gaia/dr3-stories)
|
||||
|
||||
建议要求:
|
||||
|
||||
- 使用官方来源或官方衍生可复用资源
|
||||
- 等距矩形投影(equirectangular)
|
||||
- 坐标定义尽量明确为赤道坐标展开
|
||||
- 分辨率建议至少 `4k`
|
||||
- 颜色不要过亮,避免压过 Earth HUD 前景
|
||||
- 尽量优先选择官方天文机构已经生产好的深空图,而不是自行拼接低质量星空纹理
|
||||
|
||||
建议本地资源目录:
|
||||
|
||||
- `frontend/public/earth/assets/celestial/starmap_equatorial_4k.jpg`
|
||||
|
||||
### 2. 位置数据:星表与天体计算
|
||||
|
||||
用于 Phase 2+ 的“位置正确的星体”。
|
||||
|
||||
推荐来源分两层:
|
||||
|
||||
- 太阳、月亮位置
|
||||
- 使用 [Astronomy Engine](https://github.com/cosinekitty/astronomy)
|
||||
- 恒星位置
|
||||
- 第一优先:Hipparcos / Tycho
|
||||
- [Hipparcos overview](https://www.cosmos.esa.int/web/Hipparcos)
|
||||
- [Hipparcos catalogues](https://www.cosmos.esa.int/web/hipparcos/catalogues)
|
||||
- 第二优先:Gaia
|
||||
- [Gaia DR3 stories](https://www.cosmos.esa.int/web/gaia/dr3-stories)
|
||||
|
||||
建议策略:
|
||||
|
||||
- V1:背景球壳只用全天星图,不立即生成全量恒星点
|
||||
- V2:只挑选亮星(例如星等 `< 5.5`)生成恒星点层
|
||||
- V3:如果确实需要更丰富的星场,再逐步扩展到更深星等
|
||||
|
||||
这样做的原因:
|
||||
|
||||
- 背景球壳负责“天球真实感”
|
||||
- 亮星点负责“位置正确、可后续标注和高亮”
|
||||
- 不需要一开始就处理数十万甚至数百万颗星
|
||||
|
||||
### 3. 对外部成熟实现的参考结论
|
||||
|
||||
`SatelliteMap.space` 的公开 credits 提供了一个很有价值的参考样板:
|
||||
|
||||
- 图形渲染使用 `TWGL.js`
|
||||
- 天文计算使用 `Skyfield` 与 `Astronomia`
|
||||
- 星空/天球视觉资源使用 `NASA Deep Star Maps 2020`
|
||||
|
||||
这给本项目的启发是:
|
||||
|
||||
- “真实感强的天球背景”完全可以先依赖官方高质量深空图
|
||||
- “位置正确的动态天体”则应依赖单独的天文计算链路
|
||||
- 没有必要在第一版就直接渲染完整星表
|
||||
|
||||
因此本项目推荐继续坚持两层拆分:
|
||||
|
||||
- 背景层:官方深空图 / 全天星图
|
||||
- 计算层:太阳、月亮与后续亮星点
|
||||
|
||||
## 如何保证星体位置正确
|
||||
|
||||
位置正确不是只看“图看起来像”,而是要统一参考系和转换链路。
|
||||
|
||||
### 1. 统一坐标基准
|
||||
|
||||
本方案推荐统一使用:
|
||||
|
||||
- `J2000` 赤道坐标系作为恒星位置基准
|
||||
|
||||
原因:
|
||||
|
||||
- Hipparcos / Tycho 资料和大量天文可视化都容易映射到该基准
|
||||
- 太阳、月亮也可以通过 Astronomy Engine 转到同一坐标系
|
||||
- 这样背景、恒星点、太阳、月亮就能共用一套 sky orientation
|
||||
|
||||
### 2. 背景贴图与点位必须使用同一展开逻辑
|
||||
|
||||
如果背景球壳使用赤道坐标全天图,那么:
|
||||
|
||||
- 亮星点也必须按赤道坐标贴到同一球面方向
|
||||
- 太阳/月亮 sprite 也必须按赤道坐标转换后落到同一 world-space
|
||||
|
||||
否则会出现:
|
||||
|
||||
- 背景银河带是对的
|
||||
- 但太阳/月亮或亮星点飘到不匹配的位置
|
||||
|
||||
### 3. RA / Dec 到 Three.js 坐标的落点方式
|
||||
|
||||
亮星点和日月方向最终都要转成单位球面向量。
|
||||
|
||||
概念步骤:
|
||||
|
||||
1. 读取赤经 `RA`
|
||||
2. 读取赤纬 `Dec`
|
||||
3. 转成弧度
|
||||
4. 映射到单位球面向量
|
||||
5. 再根据 Three.js 当前世界坐标定义做轴向映射
|
||||
|
||||
参考公式:
|
||||
|
||||
```text
|
||||
x = cos(dec) * cos(ra)
|
||||
y = sin(dec)
|
||||
z = cos(dec) * sin(ra)
|
||||
```
|
||||
|
||||
实际接入 Three.js 时,需要做一次项目内坐标轴校准:
|
||||
|
||||
- 验证 `RA = 0h`
|
||||
- 验证 `RA = 6h`
|
||||
- 验证北天极
|
||||
- 验证银河带主方向
|
||||
|
||||
然后确定最终的:
|
||||
|
||||
- `x/y/z` 对应 Three.js 哪个轴
|
||||
- 是否需要 `z` 取反
|
||||
- 是否需要整体再做一个固定 `rotation`
|
||||
|
||||
建议把这层显式封装在:
|
||||
|
||||
```js
|
||||
function equatorialToWorldVector(raRad, decRad)
|
||||
```
|
||||
|
||||
不要把轴映射散落在不同模块里。
|
||||
|
||||
### 4. 背景球壳与恒星点的关系
|
||||
|
||||
推荐最终组合:
|
||||
|
||||
- 背景层:全天星图球壳
|
||||
- 点位层:亮星点
|
||||
- 动态层:太阳 / 月亮
|
||||
|
||||
这样有三个好处:
|
||||
|
||||
- 背景层提供密集真实的天空纹理
|
||||
- 亮星点提供位置正确、可扩展的标注基础
|
||||
- 太阳/月亮提供与时间相关的真实动态对象
|
||||
|
||||
## 数据与资源建议清单
|
||||
|
||||
### 推荐首批引入资源
|
||||
|
||||
1. 全天星图
|
||||
- 来源:NASA Tycho all-sky map
|
||||
- 用途:背景球壳纹理
|
||||
|
||||
2. 月亮纹理
|
||||
- 用途:Phase 4 月相表现
|
||||
- 路径建议:
|
||||
- `frontend/public/earth/assets/celestial/moon_albedo_2k.jpg`
|
||||
|
||||
3. 太阳 glow 贴图
|
||||
- 用途:太阳 sprite halo
|
||||
- 路径建议:
|
||||
- `frontend/public/earth/assets/celestial/sun_glow.png`
|
||||
|
||||
### 推荐首批数据文件
|
||||
|
||||
如果要上亮星层,建议新增一个预处理后的轻量数据文件:
|
||||
|
||||
- `frontend/public/earth/assets/celestial/bright-stars.json`
|
||||
|
||||
建议字段:
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"id": 32349,
|
||||
"name": "Sirius",
|
||||
"raDeg": 101.2875,
|
||||
"decDeg": -16.7161,
|
||||
"mag": -1.46,
|
||||
"colorIndex": 0.00
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
建议不要在浏览器里直接吞原始 Gaia 大表,而是先离线裁剪成:
|
||||
|
||||
- 只保留亮星
|
||||
- 只保留渲染必需字段
|
||||
- JSON 或二进制轻量格式
|
||||
|
||||
## 资源与数据实施路线
|
||||
|
||||
### 路线 A:先做可用版本(推荐)
|
||||
|
||||
1. 引入 NASA Tycho 全天图
|
||||
- 或评估替换为更接近 SatelliteMap.space 路线的 `NASA Deep Star Maps 2020`
|
||||
2. 实现背景球壳
|
||||
3. 用 Astronomy Engine 计算太阳/月亮方向
|
||||
4. 暂不做亮星点
|
||||
|
||||
优点:
|
||||
|
||||
- 最快见效
|
||||
- 风险最低
|
||||
- 就能明显提升天球真实感
|
||||
|
||||
### 路线 B:在 A 基础上增强
|
||||
|
||||
1. 离线生成 `bright-stars.json`
|
||||
2. 浏览器端渲染亮星点
|
||||
3. 后续可加:
|
||||
- 星座线
|
||||
- 亮星名称
|
||||
- 特定星体高亮
|
||||
|
||||
优点:
|
||||
|
||||
- 背景真实感和“位置正确的可交互星体”同时兼顾
|
||||
|
||||
## 代码模块建议细化
|
||||
|
||||
### 新增模块
|
||||
|
||||
- `frontend/public/earth/js/celestial.js`
|
||||
- 管理天球背景
|
||||
- 管理太阳/月亮
|
||||
- 管理亮星层(后续)
|
||||
|
||||
- `frontend/public/earth/js/celestial-data.js`
|
||||
- 资源路径
|
||||
- 星图方向配置
|
||||
- 亮星数据加载(后续)
|
||||
|
||||
### 建议函数设计
|
||||
|
||||
```js
|
||||
export function initCelestialLayer(scene)
|
||||
export function updateCelestialLayer(date)
|
||||
export function setCelestialVisibility(visible)
|
||||
export function disposeCelestialLayer()
|
||||
|
||||
function loadStarMapTexture()
|
||||
function createSkySphere(texture)
|
||||
function createSunSprite()
|
||||
function createMoonSprite()
|
||||
function getSunEquatorialPosition(date)
|
||||
function getMoonEquatorialPosition(date)
|
||||
function equatorialToWorldVector(raRad, decRad)
|
||||
```
|
||||
|
||||
### 推荐后续预处理脚本
|
||||
|
||||
如要引入亮星层,建议单独做离线脚本:
|
||||
|
||||
- `scripts/build_bright_stars.py`
|
||||
|
||||
职责:
|
||||
|
||||
- 从 Hipparcos / Tycho 源数据读取
|
||||
- 过滤亮星
|
||||
- 生成 `bright-stars.json`
|
||||
|
||||
这样浏览器端只消费轻量结果,不承担大表解析成本。
|
||||
|
||||
## 分阶段实施
|
||||
|
||||
## Phase 1:真实天球背景
|
||||
372
docs/plans/earth-compute-center-bgp-style-plan.md
Normal file
372
docs/plans/earth-compute-center-bgp-style-plan.md
Normal file
@@ -0,0 +1,372 @@
|
||||
# Earth Compute Center BGP-Style Plan
|
||||
|
||||
## Goal
|
||||
|
||||
这份文档定义如何按照 BGP 模块的产品方式,把“算力中心”提升为 Earth 上的一级能力。
|
||||
|
||||
这里的“按 BGP 方式”指的是:
|
||||
|
||||
- 有独立的数据语义和接口入口
|
||||
- 有独立的 Earth 图层与图例
|
||||
- 有独立的 hover / click / 选中态 / 详情卡
|
||||
- 有独立的统计口径与后续专题页扩展空间
|
||||
|
||||
这里的“按 BGP 方式”不指:
|
||||
|
||||
- 机械复制 BGP 的 anomaly / incident / collector 三层事件模型
|
||||
- 为静态算力设施强行引入不必要的复杂告警语义
|
||||
|
||||
算力中心本质上更接近“长期基础设施分布层”,不是“高频动态异常层”。
|
||||
因此应该复用 BGP 的模块化方法,而不是照搬 BGP 的事件结构。
|
||||
|
||||
## Why
|
||||
|
||||
当前仓库里已经有算力相关基础:
|
||||
|
||||
- 后端已有 `top500` 和 `epoch_ai_gpu` 数据采集
|
||||
- 可视化接口已有 `/api/v1/visualization/geo/supercomputers` 和 `/api/v1/visualization/geo/gpu-clusters`
|
||||
- Earth 信息卡已对 `supercomputer` 和 `gpu_cluster` 做了基础类型兼容
|
||||
|
||||
但当前能力还停留在“数据可取到”的阶段,没有形成像 BGP 那样完整的可视化模块:
|
||||
|
||||
- Earth 缺少独立的算力图层加载模块
|
||||
- 缺少算力 marker 体系和视觉层级
|
||||
- 缺少算力图例、统计、开关和搜索接入
|
||||
- 缺少与海缆、BGP、卫星的关系表达
|
||||
- 缺少算力专题页和后续告警/研判扩展入口
|
||||
|
||||
所以当前真正的缺口不是“有没有数据”,而是“有没有产品级模块”。
|
||||
|
||||
## Core Principle
|
||||
|
||||
算力中心应当采用和 BGP 一致的模块化分层:
|
||||
|
||||
1. 数据层:稳定的数据契约和 GeoJSON 输出
|
||||
2. 渲染层:独立的 Earth 图层、marker 和视觉状态管理
|
||||
3. 交互层:hover、click、锁定态、详情卡、图例和统计
|
||||
4. 扩展层:后续专题页、关系分析、告警和 AI 研判
|
||||
|
||||
但语义上必须保持算力中心自身的特点:
|
||||
|
||||
- `site / center` 是主对象,不是事件
|
||||
- `capacity / rank / vendor / operator / status` 是主信息,不是异常严重度
|
||||
- `distribution / concentration / dependency` 是后续分析方向,不是第一阶段必须项
|
||||
|
||||
## Recommended Scope
|
||||
|
||||
第一版“算力中心”建议统一承载两类对象:
|
||||
|
||||
- `supercomputer`
|
||||
- `gpu_cluster`
|
||||
|
||||
并在 Earth 上收口为一个主题层:`compute_centers`
|
||||
|
||||
这样做有几个好处:
|
||||
|
||||
- 用户看到的是统一的“算力基础设施”语义,而不是零散数据源
|
||||
- 后端仍可保留 `top500` 和 `epoch_ai_gpu` 的来源差异
|
||||
- 前端可以在一个图层里再细分两种 marker 语言
|
||||
|
||||
## Current Gap
|
||||
|
||||
和 BGP 对比,当前差距主要在下面几层。
|
||||
|
||||
### 1. Data Contract Gap
|
||||
|
||||
现在的算力 GeoJSON 还是通用 `collected_data` 输出思路,字段较轻:
|
||||
|
||||
- `gpu_cluster` 只有基础名称和地点
|
||||
- `supercomputer` 只暴露一部分性能字段
|
||||
- 缺少统一的 `site_type / operator / capacity_band / source / updated_at / confidence`
|
||||
- 缺少统一的算力层聚合出口
|
||||
|
||||
### 2. Earth Rendering Gap
|
||||
|
||||
当前 Earth 里没有类似 `bgp.js` 的算力模块:
|
||||
|
||||
- `constants.js` 没有算力 API 路径和视觉配置
|
||||
- `main.js` 没有算力加载、拾取、状态同步和 HUD 更新
|
||||
- `controls.js` 没有算力图层开关和启动加载优先级
|
||||
- `layer-startup-tasks.js` 没有算力启动任务
|
||||
- `legend.js` / `ui.js` 没有算力统计与图例模式
|
||||
|
||||
### 3. Interaction Gap
|
||||
|
||||
虽然 `info-card.js` 支持基础字段,但还没有形成 BGP 那种完整交互链路:
|
||||
|
||||
- 没有 hover / selected / dimmed 的视觉状态
|
||||
- 没有算力对象专属 tooltip 与摘要文案
|
||||
- 没有锁定后与其他基础设施的联动高亮
|
||||
- 没有搜索、统计卡和详情组织方式
|
||||
|
||||
### 4. Product Expansion Gap
|
||||
|
||||
当前还没有“算力中心”专题页与分析语义:
|
||||
|
||||
- 没有全球分布/国家聚合/厂商聚合视图
|
||||
- 没有算力与海缆/BGP/区域的关系表达
|
||||
- 没有 AI brief / assessment 的后续落点
|
||||
|
||||
## Architecture Direction
|
||||
|
||||
推荐把算力中心做成“BGP 同级能力”,但采用更适合静态基础设施的结构。
|
||||
|
||||
### Backend
|
||||
|
||||
建议新增统一聚合接口,例如:
|
||||
|
||||
- `/api/v1/visualization/geo/compute-centers`
|
||||
|
||||
它的职责是把:
|
||||
|
||||
- `top500`
|
||||
- `epoch_ai_gpu`
|
||||
|
||||
统一转换成一个主题层输出,同时保留对象细分类型:
|
||||
|
||||
- `site_type: supercomputer | gpu_cluster`
|
||||
|
||||
建议统一字段至少包括:
|
||||
|
||||
- `id`
|
||||
- `name`
|
||||
- `site_type`
|
||||
- `country`
|
||||
- `city`
|
||||
- `latitude`
|
||||
- `longitude`
|
||||
- `operator`
|
||||
- `vendor`
|
||||
- `capacity_value`
|
||||
- `capacity_unit`
|
||||
- `capacity_band`
|
||||
- `rank`
|
||||
- `source`
|
||||
- `updated_at`
|
||||
- `location_precision`
|
||||
- `geography_mode`
|
||||
- `is_estimated`
|
||||
- `estimated_reason`
|
||||
- `metadata`
|
||||
|
||||
这里建议优先做“统一聚合出口”,而不是一开始就新增独立数据库表。
|
||||
|
||||
原因:
|
||||
|
||||
- 当前源数据更新频率低,先复用 `collected_data` 成本更低
|
||||
- 可以先把 Earth 产品体验做完整
|
||||
- 如果后续要做历史趋势、关系推断、告警,再评估是否拆成独立模型
|
||||
|
||||
### Frontend Earth
|
||||
|
||||
建议新增独立模块,例如:
|
||||
|
||||
- `frontend/public/earth/js/compute-centers.js`
|
||||
|
||||
职责参照 `bgp.js`:
|
||||
|
||||
- 拉取算力中心 GeoJSON
|
||||
- 创建 marker
|
||||
- 管理 hover / selected / dimmed 状态
|
||||
- 输出图例项
|
||||
- 输出统计摘要
|
||||
- 提供 overlay 和详情格式化辅助函数
|
||||
|
||||
推荐视觉分层:
|
||||
|
||||
1. `supercomputer` 用更稳定、更规整的设施型符号
|
||||
2. `gpu_cluster` 用更活跃、更现代的密度型符号
|
||||
3. 选中态通过 halo / ring / related infrastructure highlight 表达
|
||||
|
||||
视觉上应避免把算力中心做成“BGP 事件点”那种高频脉冲风格。
|
||||
它应该更像长期存在的高价值设施。
|
||||
|
||||
## Phases
|
||||
|
||||
## Phase 1: Unified Earth Layer
|
||||
|
||||
目标:
|
||||
|
||||
- 先把算力中心做成 Earth 上可用、可点、可解释的一级图层
|
||||
|
||||
工作项:
|
||||
|
||||
- 新增统一算力 GeoJSON 接口
|
||||
- 新增 `compute-centers.js`
|
||||
- 在 `constants.js` 增加 API 路径和视觉配置
|
||||
- 在 `controls.js` 增加算力图层开关与启动元数据
|
||||
- 在 `layer-startup-tasks.js` 增加算力启动加载任务
|
||||
- 在 `main.js` 接入算力拾取、hover、click、锁定态和 HUD 统计
|
||||
- 在 `ui.js` / `legend.js` / `index.html` 增加算力统计与图例入口
|
||||
- 在 `info-card.js` 提升算力详情字段组织
|
||||
- 对无法精确定位、但可按国家或弱线索推测的大概位置,仍然生成地图点位
|
||||
- 这类对象必须带显式“估算位置”状态,例如图标问号角标与详情说明
|
||||
|
||||
完成标准:
|
||||
|
||||
- Earth 上能独立显示/隐藏算力中心
|
||||
- 两类对象有可区分的视觉表达
|
||||
- hover / click / 详情卡 / 图例 / 统计全部打通
|
||||
- 精确位置与估算位置在图标或文案上可区分,不会误导为同一精度
|
||||
- 不干扰现有海缆、卫星、BGP 的交互链路
|
||||
|
||||
## Phase 2: Relationship Layer
|
||||
|
||||
目标:
|
||||
|
||||
- 让算力中心不只是“点”,而是和其他基础设施产生上下文关系
|
||||
|
||||
工作项:
|
||||
|
||||
- 建立算力中心与国家/区域聚合摘要
|
||||
- 增加与附近海缆登陆点的关系提示
|
||||
- 增加与 BGP 事件/观测范围的空间邻近提示
|
||||
- 增加与卫星覆盖或区域连通性的实验性提示
|
||||
|
||||
完成标准:
|
||||
|
||||
- 点击算力中心时,用户能看到“它和哪些基础设施相关”
|
||||
- 信息表达以辅助判断为主,不做夸张推断
|
||||
|
||||
## Phase 3: Compute Center Observatory
|
||||
|
||||
目标:
|
||||
|
||||
- 把算力中心从 Earth 图层扩展成独立专题观测能力
|
||||
|
||||
工作项:
|
||||
|
||||
- 新增算力中心专题页
|
||||
- 提供国家/厂商/类型/容量分布统计
|
||||
- 支持列表、筛选、详情和历史快照
|
||||
- 预留 AI brief / assessment 入口
|
||||
|
||||
完成标准:
|
||||
|
||||
- 算力中心不再只是 Earth 上的视觉点位
|
||||
- 能作为独立业务上下文进入日常观察与研判
|
||||
|
||||
## Phase 4: Alerts And Assessment
|
||||
|
||||
目标:
|
||||
|
||||
- 在不滥造“假动态告警”的前提下,引入真正有价值的变化感知
|
||||
|
||||
候选方向:
|
||||
|
||||
- 新增大规模算力中心
|
||||
- 既有中心容量显著变化
|
||||
- 国家/区域集中度显著变化
|
||||
- 高价值中心与关键网络基础设施关系变化
|
||||
|
||||
完成标准:
|
||||
|
||||
- 告警来自可解释的结构变化
|
||||
- 不把静态数据硬做成噪声式实时事件流
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
建议按下面顺序推进:
|
||||
|
||||
1. 先统一 GeoJSON 契约
|
||||
2. 再做 Earth 独立模块和图层开关
|
||||
3. 再补详情卡、图例和统计
|
||||
4. 最后才做关系层和专题页
|
||||
|
||||
这样可以避免一开始把范围摊得过大。
|
||||
|
||||
## Unknown Location Strategy
|
||||
|
||||
由于部分算力数据源不会直接提供经纬度,未知位置补全不能只依赖“继续找 API 字段”。
|
||||
更稳妥的方式是做成一条分层富化链路,而不是单一猜测规则。
|
||||
|
||||
推荐按下面优先级推进:
|
||||
|
||||
1. 直接源信息
|
||||
|
||||
- 源记录显式给出 `latitude / longitude`
|
||||
- 源记录给出 `city / region / facility / campus / operator`
|
||||
- 源页面详情、内嵌 JSON、结构化元数据、新闻稿链接里能抽出地点线索
|
||||
|
||||
2. 名称与机构归一化
|
||||
|
||||
- 建立 `canonical_name / aliases / operator / facility` 归一化表
|
||||
- 把 `cluster name`、`operator`、`campus name` 归一到同一个实体
|
||||
- 优先解决同一对象多写法导致的命中失败,而不是先扩大猜测范围
|
||||
|
||||
3. 本地位置注册表
|
||||
|
||||
- 用仓库内可维护的 registry 保存高价值对象的位置知识
|
||||
- 每条记录至少包含:`canonical_name`、`aliases`、`operator`、`country`、`region`、`city`、`lat`、`lon`、`confidence`、`source_note`
|
||||
- 转换层优先读取 registry,避免地点知识长期散落在转换代码里
|
||||
|
||||
4. 分层回退定位
|
||||
|
||||
- `precise`
|
||||
- `estimated_site`
|
||||
- `estimated_city`
|
||||
- `estimated_region`
|
||||
- `estimated_national_hub`
|
||||
- `estimated_country`
|
||||
|
||||
这里建议把“国家内主要算力城市”作为国家质心之前的一层。
|
||||
例如没有美国精确位置时,优先考虑已知的主要算力/数据中心城市候选,而不是直接落在几何质心。
|
||||
|
||||
5. 候选证据富化
|
||||
|
||||
- 如果源 API 无地点信息,可以允许采集链路读取公开辅助证据
|
||||
- 例如机构官网、数据中心介绍页、新闻稿、百科型页面、公开 PDF
|
||||
- 但只提取“地点线索”,不把外部页面上的经纬度当真值直接写回
|
||||
|
||||
6. 人工校验闭环
|
||||
|
||||
- 对高价值且仍然未知的对象输出待核验清单
|
||||
- 把人工确认结果回写到位置注册表
|
||||
- 后续采集继续优先复用这层人工确认结果
|
||||
|
||||
### Additional Solution Paths
|
||||
|
||||
除了静态映射表,还可以考虑下面这些办法:
|
||||
|
||||
- 基于国家和运营方建立“主要园区候选集”,用稳定散列把同国未知节点分散到若干可信城市,而不是全部压到一个点
|
||||
- 基于数据中心/云厂商公开 region 列表建立 `operator -> city set` 候选映射,用于云 GPU 集群类对象
|
||||
- 把“估算依据”结构化,例如 `matched_alias`、`matched_operator`、`matched_city_text`、`fallback_country_hub`
|
||||
- 给位置补全增加 `last_verified_at`,便于后续按时间重新校验老旧映射
|
||||
- 单独维护“不可可靠定位”状态;这类对象仍可在国家级聚合统计中出现,但可以允许用户在地图上过滤掉
|
||||
- 后续如果你们愿意投入更多,可把这条链路做成小型 enrichment pipeline,而不是仅在 API 转换时临时判断
|
||||
|
||||
## Non-Goals
|
||||
|
||||
第一阶段不建议做这些内容:
|
||||
|
||||
- 不复制 BGP 巡航模式到算力中心
|
||||
- 不先做复杂实时 websocket 推送
|
||||
- 不先引入独立 `compute_center_incident` 一类模型
|
||||
- 不先做全量 AI 分析面板
|
||||
|
||||
原因是算力中心的第一需求是“被看清楚”,不是“被实时播报”。
|
||||
但“被看清楚”不等于“只显示精确坐标对象”。
|
||||
对于没有精确经纬度、但能推测到国家或区域级位置的算力中心,应优先以上图并标注估算状态的方式处理,而不是直接在地图上消失。
|
||||
|
||||
## Acceptance Checklist
|
||||
|
||||
- 后端存在统一的算力中心 GeoJSON 出口
|
||||
- Earth 有独立算力图层模块,而不是散落在 `main.js`
|
||||
- 页面上有清晰的算力开关、图例和统计
|
||||
- `supercomputer` 和 `gpu_cluster` 在视觉和详情上都可区分
|
||||
- 估算位置对象在地图和详情中都有明确状态提示
|
||||
- 现有 BGP / 海缆 / 卫星功能无回归
|
||||
- 代码结构上为后续专题页和关系分析留出了明确扩展点
|
||||
|
||||
## Summary
|
||||
|
||||
这项工作的本质不是“再多画几个点”。
|
||||
|
||||
它应该把算力中心从已有数据源,升级成与 BGP 同级的 Earth 观测主题:
|
||||
|
||||
- 有独立语义
|
||||
- 有独立图层
|
||||
- 有独立交互
|
||||
- 有后续分析扩展能力
|
||||
|
||||
推荐先完成 Phase 1,把算力中心做成真正可用的 Earth 一级模块,再继续推进关系层和专题页。
|
||||
400
docs/plans/earth-mobile-drawer-ui-plan.md
Normal file
400
docs/plans/earth-mobile-drawer-ui-plan.md
Normal file
@@ -0,0 +1,400 @@
|
||||
# Earth Mobile Drawer UI Plan
|
||||
|
||||
## 背景
|
||||
|
||||
当前 Earth 移动端已经补上了基础触控能力,例如:
|
||||
|
||||
- 单指拖拽旋转地球
|
||||
- 双指缩放
|
||||
- 点击阈值和基础事件隔离
|
||||
|
||||
但移动端 UI 仍然存在一个根本问题:
|
||||
|
||||
它还在沿用桌面 HUD 的内容切分方式,只是把原来的 panel、modal、toolbar 改位置、改层级、改容器。这样虽然能快速复用旧代码,但手机端体验仍然是生硬的,因为:
|
||||
|
||||
- 信息密度和结构是按桌面设计的
|
||||
- 面板标题、关闭、折叠、开关项是桌面心智,不是手机心智
|
||||
- 很多内容只是“被塞进抽屉”,而不是为抽屉重新设计
|
||||
- 设置里仍然带有“显示/隐藏某些 panel”的思路,但移动端本来就不应该存在那些独立 panel
|
||||
|
||||
因此本计划进一步收紧:
|
||||
|
||||
移动端不只是“底部抽屉化”,而是**重新设计一套 fit 抽屉体系的 mobile-first UI**。
|
||||
|
||||
## 新目标
|
||||
|
||||
1. 手机端不再使用现有 `toolbar` 作为主入口。
|
||||
2. 手机端不再使用现有独立 `panel / modal / sheet` 作为直接 UI 单元。
|
||||
3. 手机端统一采用“底部抽屉 + 顶部标题 + tab 切换 + 卡片内容”的单前景模式。
|
||||
4. 抽屉内部每个 tab 页面都按移动端重新设计内容结构,而不是直接复用旧 panel 结构。
|
||||
5. 设置页移除“显示/隐藏 panel”的桌面遗留配置。
|
||||
6. 媒体页拆成两个移动端页面:`新闻` 与 `TV`,都归入抽屉体系。
|
||||
7. 桌面端保持现有 HUD 体系,不回退。
|
||||
|
||||
## 核心原则
|
||||
|
||||
### 1. 只复用数据和状态,不复用桌面 UI 结构
|
||||
|
||||
可复用:
|
||||
|
||||
- 图层注册表
|
||||
- 搜索结果数据
|
||||
- BGP / 海缆 / 卫星详情数据
|
||||
- 媒体数据
|
||||
- 旋转、缩放、选择、高亮等运行时状态
|
||||
|
||||
不直接复用:
|
||||
|
||||
- 桌面 panel DOM 结构
|
||||
- 桌面 panel header / close / collapse 交互
|
||||
- 桌面 settings 项里的“显示某 panel”逻辑
|
||||
- 桌面媒体面板布局
|
||||
|
||||
### 2. 抽屉是唯一主前景层
|
||||
|
||||
移动端同一时刻只有一个主前景层:底部抽屉。
|
||||
|
||||
抽屉内部切换内容页,而不是多个悬浮层互相覆盖。
|
||||
|
||||
### 3. 每个 tab 都是移动端页面,而不是 panel 容器
|
||||
|
||||
抽屉中的每一项都应视为一个移动端子页面:
|
||||
|
||||
- 有自己的标题
|
||||
- 有自己的内容层次
|
||||
- 有自己的滚动区域
|
||||
- 有自己的主操作
|
||||
|
||||
而不是简单挂一个旧面板进去。
|
||||
|
||||
### 4. 移动端状态提示不占据屏幕正中
|
||||
|
||||
桌面端当前很多通知、状态提示、胶囊消息更适合在屏幕上方居中出现,但移动端不应继续沿用这套布局。
|
||||
|
||||
移动端统一改为:
|
||||
|
||||
- 通知栏放在右上角安全区
|
||||
- 胶囊提示放在右上角堆叠
|
||||
- 不遮挡地球中心视野
|
||||
- 不与底部抽屉主交互区冲突
|
||||
|
||||
## 交互模型
|
||||
|
||||
### 默认态
|
||||
|
||||
移动端默认只显示:
|
||||
|
||||
- 地球主画布
|
||||
- 底部半露出的抽屉头部
|
||||
|
||||
不再单独显示上箭头按钮。
|
||||
|
||||
### 展开态
|
||||
|
||||
用户从底边直接上拉抽屉,或点击抽屉头部展开。
|
||||
|
||||
展开后显示:
|
||||
|
||||
- 当前页面标题
|
||||
- tab 导航
|
||||
- 当前页面内容
|
||||
|
||||
### 收起态
|
||||
|
||||
用户下拉抽屉头部收起,或点击背景收起。
|
||||
|
||||
## 信息架构
|
||||
|
||||
移动端抽屉内的一级页面重定为:
|
||||
|
||||
1. 图层
|
||||
2. 搜索
|
||||
3. 态势
|
||||
4. 新闻
|
||||
5. TV
|
||||
6. 设置
|
||||
7. 详情(按需出现,不固定常驻 tab)
|
||||
|
||||
其中 `新闻` 和 `TV` 不再共享同一个移动端媒体面板。
|
||||
|
||||
## 页面重设计要求
|
||||
|
||||
### 图层页
|
||||
|
||||
目标:
|
||||
|
||||
- 成为移动端最核心的控制页
|
||||
- 强调快速开关,不强调桌面 panel 感
|
||||
|
||||
内容建议:
|
||||
|
||||
- 顶部摘要:当前已启用图层数量
|
||||
- 图层列表卡片
|
||||
- 每个图层项只保留:
|
||||
- 图标
|
||||
- 中文名
|
||||
- 英文副标题
|
||||
- 开关
|
||||
- 去掉桌面式 header / collapse / close 结构
|
||||
|
||||
### 搜索页
|
||||
|
||||
目标:
|
||||
|
||||
- 成为抽屉中的完整搜索页
|
||||
- 避免看起来像桌面 modal 被塞进抽屉
|
||||
|
||||
内容建议:
|
||||
|
||||
- 顶部搜索输入框
|
||||
- 搜索提示文案
|
||||
- 结果列表
|
||||
- 结果项更适合手指点击
|
||||
- 结果点击后:
|
||||
- 聚焦地球对象
|
||||
- 自动切换到详情页
|
||||
|
||||
### 态势页
|
||||
|
||||
目标:
|
||||
|
||||
- 合并原来的 `stats + legend` 思路
|
||||
- 成为移动端全局态势页
|
||||
|
||||
内容建议:
|
||||
|
||||
- 顶部核心统计卡
|
||||
- 海缆数量
|
||||
- 登陆点数量
|
||||
- 卫星数量
|
||||
- BGP 事件数量
|
||||
- 当前关注层图例
|
||||
- BGP 状态摘要
|
||||
- 不再出现独立 legend 面板和独立 stats 面板
|
||||
|
||||
### 新闻页
|
||||
|
||||
目标:
|
||||
|
||||
- 从原媒体面板中拆出单独的移动端新闻页
|
||||
|
||||
内容建议:
|
||||
|
||||
- 当前区域焦点
|
||||
- 新闻源数量
|
||||
- 新闻卡片列表
|
||||
- 卡片内显示标题、来源、时间、区域
|
||||
- 外链操作更清晰
|
||||
|
||||
### TV 页
|
||||
|
||||
目标:
|
||||
|
||||
- 从原媒体面板中拆出单独的移动端 TV 页
|
||||
|
||||
内容建议:
|
||||
|
||||
- 顶部频道选择
|
||||
- 直播状态
|
||||
- 当前频道说明
|
||||
- 视频播放器区域
|
||||
- 刷新和外链按钮
|
||||
|
||||
不再保留桌面式“新闻/TV tab 共处一个 panel”的结构。
|
||||
|
||||
### 设置页
|
||||
|
||||
目标:
|
||||
|
||||
- 只保留对移动端仍有意义的系统配置
|
||||
|
||||
必须移除:
|
||||
|
||||
- 图层控制 panel 显示/隐藏
|
||||
- 图例 panel 显示/隐藏
|
||||
- 全球态势 panel 显示/隐藏
|
||||
- 媒体 panel 显示/隐藏
|
||||
|
||||
保留项建议:
|
||||
|
||||
- 旋转模式
|
||||
- 日夜模式
|
||||
- 地球默认大小
|
||||
- 地形透明度
|
||||
- 系统入口
|
||||
|
||||
原因:
|
||||
|
||||
移动端已经没有这些独立 panel 了,所以继续保留这些开关会制造错误心智。
|
||||
|
||||
### 详情页
|
||||
|
||||
目标:
|
||||
|
||||
- 成为海缆 / BGP / 卫星对象的统一移动端详情页
|
||||
|
||||
内容建议:
|
||||
|
||||
- 标题区
|
||||
- 类型标签
|
||||
- 关键属性列表
|
||||
- 相关对象摘要
|
||||
- 相关图层或态势提示
|
||||
|
||||
行为建议:
|
||||
|
||||
- 点击对象后自动切入详情页
|
||||
- 搜索结果点击后也切入详情页
|
||||
|
||||
## 阶段重定义
|
||||
|
||||
### 阶段 2:抽屉壳层
|
||||
|
||||
目标:
|
||||
|
||||
1. 实现底部抽屉基本壳层。
|
||||
2. 支持上拉展开、下拉收起、背景点击关闭。
|
||||
3. `mobile` 模式下隐藏旧 toolbar。
|
||||
4. `mobile` 模式下不再直接显示旧 panel。
|
||||
|
||||
完成标准:
|
||||
|
||||
1. 手机端只有地球主视图和抽屉。
|
||||
2. 抽屉开合稳定。
|
||||
|
||||
### 阶段 3:基础页面重做
|
||||
|
||||
目标:
|
||||
|
||||
1. 重新设计并实现图层页。
|
||||
2. 重新设计并实现搜索页。
|
||||
3. 重新设计并实现设置页。
|
||||
|
||||
完成标准:
|
||||
|
||||
1. 这三个页面不再是旧 panel 原样移植。
|
||||
2. 设置页已移除 panel 可见性开关。
|
||||
|
||||
### 阶段 4:态势与详情重做
|
||||
|
||||
目标:
|
||||
|
||||
1. 将 stats 和 legend 合并为新的态势页。
|
||||
2. 实现统一详情页。
|
||||
3. 对象点击与搜索结果点击都可切入详情页。
|
||||
|
||||
完成标准:
|
||||
|
||||
1. 不再存在移动端独立 legend / stats 面板。
|
||||
2. 详情页成为统一对象信息入口。
|
||||
|
||||
### 阶段 5:媒体拆分重做
|
||||
|
||||
目标:
|
||||
|
||||
1. 将原媒体面板拆成两个移动端页面:新闻页、TV 页。
|
||||
2. 分别重做这两个页面的布局。
|
||||
3. 保留各自必要操作,但不继续共享桌面 panel 结构。
|
||||
|
||||
完成标准:
|
||||
|
||||
1. 新闻与 TV 各自成为独立移动端页面。
|
||||
2. 不再使用桌面媒体 panel 的 tab 结构作为移动端主体。
|
||||
|
||||
### 阶段 6:手感与真机修正
|
||||
|
||||
目标:
|
||||
|
||||
1. 调整抽屉高度、节奏、手势阈值。
|
||||
2. 调整 tab 密度与文字层级。
|
||||
3. 优化 iPhone / Android 安全区。
|
||||
4. 优化抽屉滚动与地球拖拽边界。
|
||||
|
||||
完成标准:
|
||||
|
||||
1. 抽屉和地球不会抢手势。
|
||||
2. 手机端各页面信息层次清晰。
|
||||
3. 真机下无遮挡、无死层、无错误交互心智。
|
||||
|
||||
## 技术落点调整
|
||||
|
||||
### [frontend/public/earth/index.html](/home/ray/dev/linkong/planet/frontend/public/earth/index.html)
|
||||
|
||||
职责:
|
||||
|
||||
- 只保留移动端抽屉壳层
|
||||
- 为各页面提供新的页面容器
|
||||
|
||||
不再把旧 panel 作为最终结构直接塞进抽屉。
|
||||
|
||||
### [frontend/public/earth/js/controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js)
|
||||
|
||||
职责:
|
||||
|
||||
- 管理抽屉开合
|
||||
- 管理 tab 切换
|
||||
- 管理详情页切入
|
||||
- 管理 mobile / desktop 分流
|
||||
|
||||
### [frontend/public/earth/js/search.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/search.js)
|
||||
|
||||
职责:
|
||||
|
||||
- 保留搜索能力和结果逻辑
|
||||
- 输出给新的移动端搜索页
|
||||
|
||||
### [frontend/public/earth/js/info-card.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/info-card.js)
|
||||
|
||||
职责:
|
||||
|
||||
- 从桌面 info-card 逻辑中提取可复用的数据层
|
||||
- 服务新的移动端详情页
|
||||
|
||||
### [frontend/public/earth/js/tv.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/tv.js)
|
||||
|
||||
职责:
|
||||
|
||||
- 为新的 TV 页面提供数据和状态
|
||||
- 不再直接主导移动端媒体 panel 壳层
|
||||
|
||||
### [frontend/public/earth/js/news.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/news.js)
|
||||
|
||||
职责:
|
||||
|
||||
- 为新的新闻页面提供列表和区域焦点数据
|
||||
|
||||
### CSS
|
||||
|
||||
需要新增真正的移动端页面样式,而不是继续在旧 panel class 上堆条件分支:
|
||||
|
||||
- 图层页样式
|
||||
- 搜索页样式
|
||||
- 态势页样式
|
||||
- 新闻页样式
|
||||
- TV 页样式
|
||||
- 设置页样式
|
||||
- 详情页样式
|
||||
- 移动端右上角通知 / 胶囊提示样式
|
||||
|
||||
## 验收标准
|
||||
|
||||
1. `mobile` 模式下不再显示旧 toolbar。
|
||||
2. `mobile` 模式下不再把旧 panel 直接作为最终 UI。
|
||||
3. 图层、搜索、态势、新闻、TV、设置都是重新设计的移动端页面。
|
||||
4. 设置页不再包含移动端无意义的 panel 显示/隐藏项。
|
||||
5. 新闻与 TV 已拆分为两个移动端页面。
|
||||
6. legend / stats 已整合为态势页。
|
||||
7. 详情页成为统一对象详情入口。
|
||||
8. 移动端通知栏和胶囊提示已统一放到右上角安全区,而不是屏幕正中。
|
||||
|
||||
## 结论
|
||||
|
||||
本计划进一步明确:
|
||||
|
||||
移动端目标不是“把桌面 HUD 放进抽屉”,而是“以抽屉为载体,重做一套适合手机端的信息页面”。
|
||||
|
||||
后续开发必须以此为准:
|
||||
|
||||
- 复用数据
|
||||
- 重做界面
|
||||
- 清除桌面遗留心智
|
||||
156
docs/plans/earth-news-source-configuration-and-collector-plan.md
Normal file
156
docs/plans/earth-news-source-configuration-and-collector-plan.md
Normal file
@@ -0,0 +1,156 @@
|
||||
# Earth News Source Configuration And Collector Plan
|
||||
|
||||
## Why
|
||||
|
||||
当前 Earth 的“态势新闻”由 [earth_news.py](/home/ray/dev/linkong/planet/backend/app/services/earth_news.py) 直接在请求时抓取 RSS / Google News feed,再按当前地球视角中心区域聚合返回。
|
||||
|
||||
这条链已经可用,但存在两个明显限制:
|
||||
|
||||
- 新闻源写死在代码里,不能像 TV 直播源一样从后台维护
|
||||
- 新闻并未进入统一采集体系,没有采集状态、失败监控、历史数据和后续 AI 复用能力
|
||||
|
||||
因此这块更合理的路线不是一步到位重写,而是分阶段推进:
|
||||
|
||||
1. 先做“新闻源配置化”
|
||||
2. 再做“新闻采集器化”
|
||||
|
||||
## Current State
|
||||
|
||||
当前实现分布在:
|
||||
|
||||
- 新闻接口
|
||||
- [news.py](/home/ray/dev/linkong/planet/backend/app/api/v1/news.py)
|
||||
- 实时聚合逻辑
|
||||
- [earth_news.py](/home/ray/dev/linkong/planet/backend/app/services/earth_news.py)
|
||||
- 前端消费
|
||||
- [news.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/news.js)
|
||||
|
||||
当前新闻源包含:
|
||||
|
||||
- `BBC World` RSS
|
||||
- `DW Top Stories` RSS
|
||||
- 按区域关键词拼出来的 `Google News RSS`
|
||||
- `Global`
|
||||
- `Americas`
|
||||
- `Europe`
|
||||
- `Middle East / Africa`
|
||||
- `Asia Pacific`
|
||||
|
||||
当前不是采集器,也不落库,只做内存缓存。
|
||||
|
||||
## Phase 1: Source Configuration
|
||||
|
||||
### Goal
|
||||
|
||||
把 `NEWS_FEED_SOURCES` 从硬编码列表升级成可配置新闻源目录,但继续保留当前“实时聚合”的工作方式。
|
||||
|
||||
### Scope
|
||||
|
||||
- 为 Earth news 建立独立配置结构
|
||||
- 支持后台维护 feed 源
|
||||
- 支持启用/禁用、优先级、区域、源类型
|
||||
- 保持现有 `/api/v1/news/earth-feed` 输出协议不变
|
||||
|
||||
### Proposed Shape
|
||||
|
||||
建议配置字段至少包括:
|
||||
|
||||
- `id`
|
||||
- `name`
|
||||
- `region`
|
||||
- `feed_url`
|
||||
- `homepage_url`
|
||||
- `source_type`
|
||||
- `priority`
|
||||
- `is_enabled`
|
||||
- 可选 `query_profile`
|
||||
- 可选 `language`
|
||||
- 可选 `notes`
|
||||
|
||||
### Suggested Storage
|
||||
|
||||
优先走系统设置或单独的 news source settings payload,而不是先建复杂新表。
|
||||
|
||||
推荐原因:
|
||||
|
||||
- 改动小
|
||||
- 易上线
|
||||
- 和当前 TV settings 维护体验更接近
|
||||
- 先解决“写死在代码里”的问题
|
||||
|
||||
### Non-goals
|
||||
|
||||
这一阶段不做:
|
||||
|
||||
- 新闻入库
|
||||
- 新闻历史回看
|
||||
- 新闻采集任务监控
|
||||
- 新闻去重流水线
|
||||
|
||||
## Phase 2: News Collectorization
|
||||
|
||||
### Goal
|
||||
|
||||
把“态势新闻”升级为真正的采集器链路,使其进入采集系统和数据层。
|
||||
|
||||
### Scope
|
||||
|
||||
- 新增专用 news collector
|
||||
- 按配置源定时采集 RSS / feed
|
||||
- 做标题/链接级去重
|
||||
- 建立统一新闻记录模型
|
||||
- 为 Earth、控制台、AI 研判复用同一份新闻数据
|
||||
|
||||
### Benefits
|
||||
|
||||
- 有采集状态
|
||||
- 有失败监控
|
||||
- 有历史缓存
|
||||
- 可以做时间轴 / 区域新闻基线
|
||||
- 可以作为 AI 引用证据
|
||||
|
||||
### Required Design Work
|
||||
|
||||
需要提前明确:
|
||||
|
||||
- 新闻数据模型
|
||||
- 去重策略
|
||||
- 过期清理策略
|
||||
- 区域映射策略
|
||||
- 聚合排序策略
|
||||
- 新闻与 Earth 当前视角/区域的关联方式
|
||||
|
||||
### Candidate Output Model
|
||||
|
||||
至少应包含:
|
||||
|
||||
- `source_id`
|
||||
- `headline`
|
||||
- `summary`
|
||||
- `url`
|
||||
- `publisher`
|
||||
- `region`
|
||||
- `published_at`
|
||||
- `language`
|
||||
- `tags`
|
||||
- `raw_feed_source`
|
||||
- `reference_date`
|
||||
|
||||
## Recommended Order
|
||||
|
||||
推荐执行顺序:
|
||||
|
||||
1. 先完成 Phase 1 配置化
|
||||
2. 保持 Earth 继续实时聚合,但改为读取配置源
|
||||
3. 等新闻源稳定后,再设计 Phase 2 的 collector / storage / dedupe
|
||||
|
||||
## Decision
|
||||
|
||||
当前结论:
|
||||
|
||||
- TV 直播源:优先采集器化
|
||||
- 态势新闻:优先配置化,再采集器化
|
||||
|
||||
## Source Note
|
||||
|
||||
This plan is newly created for the Planet repo to separate the short-term "configurable source directory" work from the longer-term "collectorized news pipeline" work.
|
||||
98
docs/plans/earth-predicted-orbit-plan.md
Normal file
98
docs/plans/earth-predicted-orbit-plan.md
Normal file
@@ -0,0 +1,98 @@
|
||||
# Earth Predicted Orbit Plan
|
||||
|
||||
> Source note: this plan absorbs useful ideas from a sisyphus-created draft formerly stored at `.sisyphus/plans/predicted-orbit.md`.
|
||||
|
||||
## Goal
|
||||
|
||||
在 Earth 中锁定卫星时,显示“预测轨道”而不是只有历史尾迹:
|
||||
|
||||
- 从当前时刻开始
|
||||
- 绕地球一圈
|
||||
- 当前点最亮
|
||||
- 向后沿轨道逐步衰减
|
||||
|
||||
## Current State
|
||||
|
||||
当前已经有:
|
||||
|
||||
- 卫星历史轨迹
|
||||
- 锁定卫星
|
||||
- 轨道高亮与相关联动
|
||||
|
||||
但“预测轨道”仍然不是一套稳定、可验证的单独功能计划。
|
||||
|
||||
## Why It Is Valuable
|
||||
|
||||
预测轨道可以明显提升:
|
||||
|
||||
- 锁定卫星后的空间可读性
|
||||
- 轨道类型辨识
|
||||
- 演示解释力
|
||||
|
||||
相比短历史尾迹,预测轨道更符合用户对“这颗卫星接下来会怎么走”的预期。
|
||||
|
||||
## Scope
|
||||
|
||||
### Phase 1
|
||||
|
||||
- 锁定卫星时显示一整圈预测轨道
|
||||
- 解锁时隐藏
|
||||
- 不替代现有普通轨迹系统
|
||||
|
||||
### Phase 2
|
||||
|
||||
- 根据轨道类型调整采样率
|
||||
- GEO / MEO / LEO 不同密度
|
||||
- 进一步减少 fallback 轨迹的比例
|
||||
|
||||
## Implementation Direction
|
||||
|
||||
### 1. Orbit period
|
||||
|
||||
基于 `meanMotion` 估算轨道周期。
|
||||
|
||||
### 2. Predicted samples
|
||||
|
||||
以固定采样步长从 `now -> now + period` 推算轨迹点。
|
||||
|
||||
### 3. Render object lifecycle
|
||||
|
||||
预测轨道应是一个独立渲染对象:
|
||||
|
||||
- show
|
||||
- update
|
||||
- hide
|
||||
- dispose
|
||||
|
||||
### 4. Visual semantics
|
||||
|
||||
预测轨道不应与普通尾迹混淆:
|
||||
|
||||
- 更稳定
|
||||
- 更完整
|
||||
- 透明度沿轨道衰减
|
||||
- 当前点附近更亮
|
||||
|
||||
## Known Risks
|
||||
|
||||
### 1. TLE propagation gaps
|
||||
|
||||
部分卫星可能出现 SGP4 计算不足,需要 fallback。
|
||||
|
||||
### 2. Multiple orbit lines
|
||||
|
||||
必须确保:
|
||||
|
||||
- 锁定切换前先清旧轨道
|
||||
- 页面隐藏/销毁时清理
|
||||
|
||||
### 3. Performance
|
||||
|
||||
GEO 轨道点数高,采样率需要按轨道类型分层。
|
||||
|
||||
## Acceptance
|
||||
|
||||
1. 锁定单颗卫星时只显示一条预测轨道
|
||||
2. 解锁后轨道立即清除
|
||||
3. 不同轨道类型下点数可控
|
||||
4. 页面切换回来不会闪出旧轨道残留
|
||||
472
docs/plans/earth-real-terrain-plan.md
Normal file
472
docs/plans/earth-real-terrain-plan.md
Normal file
@@ -0,0 +1,472 @@
|
||||
# Earth Real Terrain Plan
|
||||
|
||||
## Goal
|
||||
|
||||
将 Earth 页当前的“程序噪声假地形”替换成基于真实 DEM 的可用地形层,使 `地形 terrain` 开关真正显示全球海拔起伏,而不是占位效果。
|
||||
|
||||
当前占位实现位于:
|
||||
|
||||
- [earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js)
|
||||
|
||||
具体问题:
|
||||
|
||||
- `createTerrain()` 直接对球体顶点应用 `simplex noise`
|
||||
- 没有真实海拔数据来源
|
||||
- 没有分辨率分层
|
||||
- 没有和当前相机/视角配套的性能控制
|
||||
|
||||
## Constraints
|
||||
|
||||
本计划必须贴合当前 Earth 架构,而不是引入一套全新的地形引擎:
|
||||
|
||||
- 地球主体仍然是一个 Three.js sphere
|
||||
- 海缆、登陆点、卫星、BGP 都已经建立在当前球体坐标系之上
|
||||
- 不能为了地形把整页改成 Cesium/MapLibre Globe 之类的全栈替换
|
||||
- 第一阶段优先做“真实可用”,不是一步到位做摄影测量级地形
|
||||
|
||||
## Recommended Data Source
|
||||
|
||||
### Primary recommendation
|
||||
|
||||
使用公开的 Terrarium 编码高程瓦片作为浏览器端高度来源,第一阶段优先接入:
|
||||
|
||||
- Mapzen/AWS `Terrarium` elevation tiles
|
||||
参考:[Mapzen terrain tile format / Terrarium](https://www.mapzen.com/blog/terrain-tile-service/)
|
||||
|
||||
原因:
|
||||
|
||||
- 已经是全球瓦片化高程
|
||||
- 浏览器端按 tile 请求,最适合当前 Earth 这种在线 globe
|
||||
- 编码简单稳定:
|
||||
- `heightMeters = (R * 256 + G + B / 256) - 32768`
|
||||
- 不需要我们先离线拼整球 DEM
|
||||
|
||||
### Data quality upgrade path
|
||||
|
||||
如果后面第一阶段效果确认可用,再逐步升级到底层源:
|
||||
|
||||
- Copernicus DEM GLO-30
|
||||
参考:[Copernicus DEM docs](https://documentation.dataspace.copernicus.eu/APIs/SentinelHub/Data/DEM.html)
|
||||
- 或用 Copernicus / SRTM / ASTER 等离线切成我们自己的 terrain tiles
|
||||
|
||||
这条升级路径适合第二阶段,不建议一开始就直接自建全球瓦片服务。
|
||||
|
||||
## Why Not Replace the Engine
|
||||
|
||||
不建议为了地形直接切到 Cesium terrain / quantized mesh 引擎,原因:
|
||||
|
||||
- 现有 Earth 业务对象都依附当前球面坐标
|
||||
- 切引擎会同时波及:
|
||||
- 海缆绘制
|
||||
- 卫星/轨迹
|
||||
- BGP 标记
|
||||
- HUD 与交互
|
||||
- 这是“重做一页”,不是“给地形层接真实数据”
|
||||
|
||||
所以推荐路线是:
|
||||
|
||||
- 保持当前 sphere globe
|
||||
- 为 sphere 增加真实高度位移层
|
||||
|
||||
## Implementation Strategy
|
||||
|
||||
分三期推进。
|
||||
|
||||
### Phase 1 — Global Heightmap Terrain Overlay
|
||||
|
||||
目标:
|
||||
|
||||
- 地形层切换后显示真实海拔起伏
|
||||
- 全球范围可用
|
||||
- 性能可控
|
||||
|
||||
做法:
|
||||
|
||||
1. 新增 terrain 数据模块
|
||||
|
||||
建议文件:
|
||||
|
||||
- `frontend/public/earth/js/terrain.js`
|
||||
|
||||
职责:
|
||||
|
||||
- 选择 DEM zoom level
|
||||
- 请求 Terrarium tiles
|
||||
- 解码 tile 高程
|
||||
- 将高程重采样到当前地形球体网格
|
||||
|
||||
2. 替换 `createTerrain()`
|
||||
|
||||
当前:
|
||||
|
||||
- 在 [earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js) 中同步生成噪声地形
|
||||
|
||||
调整后:
|
||||
|
||||
- `createTerrain()` 只负责创建 terrain mesh 骨架
|
||||
- 真正的顶点位移由 terrain 模块异步注入
|
||||
|
||||
3. 第一阶段采用“整球低分辨率位移”
|
||||
|
||||
不要一上来做动态 patch stitching。第一阶段更稳的办法是:
|
||||
|
||||
- 保留一张全球 terrain sphere
|
||||
- 使用较低分辨率几何
|
||||
- 例如 `SphereGeometry(radius, 192, 192)` 或 `256/256`
|
||||
- 运行时按一个固定地形 zoom(如 `z=4` 或 `z=5`)抓取覆盖全球的 Terrarium tiles
|
||||
- 将 tile 解码后重投影到经纬度采样网格
|
||||
- 将每个球面顶点按真实高度抬升
|
||||
|
||||
这样第一阶段就能做到:
|
||||
|
||||
- 有真实地形
|
||||
- 不需要复杂的局部 LOD
|
||||
- 不会让现有球体对象体系爆炸
|
||||
|
||||
### Phase 2 — View-Aware Refinement
|
||||
|
||||
目标:
|
||||
|
||||
- 正面可见区域更精细
|
||||
- 背面与远处维持低成本
|
||||
|
||||
做法:
|
||||
|
||||
- 引入“基础全球地形 + 当前视角高分局部补丁”
|
||||
- 正面区域额外抓更高 zoom 的高程 tile
|
||||
- 只替换局部顶点位移或局部 overlay mesh
|
||||
|
||||
这一阶段适合在第一阶段稳定后做。
|
||||
|
||||
### Phase 3 — Normals / Shading / Terrain UX
|
||||
|
||||
目标:
|
||||
|
||||
- 地形不仅有起伏,还更好看、更可读
|
||||
|
||||
包括:
|
||||
|
||||
- 根据高度生成更合理的 normals
|
||||
- 调整 terrain material,使山脉/高原更易读
|
||||
- 可选加入:
|
||||
- hillshade
|
||||
- contour lines
|
||||
- snowline / bathymetry tint
|
||||
|
||||
## Calibration Overlay Before More Terrain Tuning
|
||||
|
||||
在当前项目里,terrain 看起来“不像真地形”,不一定只是 DEM 或 exaggeration 不够,也可能是因为缺少稳定参照物。
|
||||
|
||||
没有清晰的海岸线、国界线和地表分层时,人眼很难判断:
|
||||
|
||||
- 山脉是不是在应该高的地方高
|
||||
- terrain 是否真的贴在正确的大陆位置上
|
||||
- 地球纹理、本初子午线、terrain 采样之间是否存在偏移
|
||||
|
||||
这里要明确区分两件事:
|
||||
|
||||
- 国界线不会修好错误的 terrain
|
||||
- 但海岸线 / 国界线会让我们更容易判断 terrain 有没有贴准
|
||||
|
||||
所以在继续盲调 terrain 参数之前,建议先插入一个“校准参照层”阶段。
|
||||
|
||||
### Recommended order for the calibration layer
|
||||
|
||||
1. 海岸线
|
||||
2. 国界线
|
||||
3. 再继续调 terrain
|
||||
|
||||
原因:
|
||||
|
||||
- 海岸线比国界线更基础,也更接近真实地表边界
|
||||
- 判断 terrain 是否贴准,最重要的是大陆边缘和山脉/海岸关系
|
||||
- 国界线更多是政治边界,只能作为辅助参照
|
||||
|
||||
如果只加国界线,不加海岸线,效果仍然可能会怪,因为:
|
||||
|
||||
- 很多国界线本来就是人为直线
|
||||
- 它们并不总是跟真实地形走
|
||||
|
||||
### Suggested layer order during debugging
|
||||
|
||||
建议调试期临时把地球层次明确成:
|
||||
|
||||
1. base earth texture
|
||||
2. coastline / borders overlay
|
||||
3. terrain relief
|
||||
4. cables / landing points / bgp / satellites
|
||||
|
||||
这样会比现在更容易判断:
|
||||
|
||||
- 山脉是否位于正确区域
|
||||
- terrain 是否和地表对齐
|
||||
- 国界/海岸是否漂移
|
||||
|
||||
### Suggested data source for the calibration overlay
|
||||
|
||||
优先用 `Natural Earth` 的轻量全球矢量数据:
|
||||
|
||||
- 海岸线(coastline)
|
||||
- Admin 0 国界线(country borders)
|
||||
|
||||
优点:
|
||||
|
||||
- 全球一致
|
||||
- 轻量
|
||||
- 很适合当前 Three.js globe 做 overlay
|
||||
|
||||
### Recommended execution path
|
||||
|
||||
#### Phase A — Add reference overlays
|
||||
|
||||
先加两层可开关的参考线:
|
||||
|
||||
- 海岸线
|
||||
- 国界线
|
||||
|
||||
这两层的目标不是最终美术表现,而是调试 / 校准。
|
||||
|
||||
#### Phase B — Recalibrate terrain against coastline
|
||||
|
||||
有了海岸线以后,再重新看 terrain:
|
||||
|
||||
- terrain 是否和大陆边缘错位
|
||||
- 地球纹理、本初子午线、terrain 采样之间是否有固定偏移
|
||||
|
||||
#### Phase C — Decide whether to keep the current terrain path
|
||||
|
||||
这时再决定后面的路线:
|
||||
|
||||
- 如果发现真实高程整体是对的,只是缺少 shading / readability
|
||||
继续保留当前 DEM + terrain overlay 路线
|
||||
- 如果发现整球采样投影、本初子午线或 overlay 关系本身就很别扭
|
||||
再考虑重做 terrain pipeline
|
||||
|
||||
### Practical recommendation
|
||||
|
||||
当前阶段不建议“从头开始重做 terrain”。
|
||||
|
||||
更稳的策略是:
|
||||
|
||||
- 暂停继续盲调 terrain 参数
|
||||
- 先补海岸线 / 国界线作为校准参照层
|
||||
- 再基于参照层判断 terrain 是“参数没调好”,还是“整条实现路径有偏移”
|
||||
|
||||
## Recommended Geometry Model
|
||||
|
||||
### First usable model
|
||||
|
||||
保留一层独立 terrain sphere:
|
||||
|
||||
- base earth sphere:贴纹理、昼夜、海洋
|
||||
- terrain sphere:略高于地球半径,真实高程位移
|
||||
|
||||
建议:
|
||||
|
||||
- `terrainBaseRadius = CONFIG.earthRadius + 0.2`
|
||||
- 高度缩放使用真实米制换算,再乘一个可调 exaggeration
|
||||
|
||||
示例关系:
|
||||
|
||||
- `heightWorld = (elevationMeters / 6371000) * CONFIG.earthRadius * exaggeration`
|
||||
|
||||
建议第一阶段 `exaggeration = 1.3 ~ 1.8`
|
||||
|
||||
因为完全真实比例在全球球体上会太平,看不出来。
|
||||
|
||||
## Tile Decoding Plan
|
||||
|
||||
### Terrarium decode
|
||||
|
||||
对于每个高程 tile 像素:
|
||||
|
||||
```text
|
||||
heightMeters = (R * 256 + G + B / 256) - 32768
|
||||
```
|
||||
|
||||
### Sampling path
|
||||
|
||||
对于 terrain mesh 上每个顶点:
|
||||
|
||||
1. 将顶点方向转成经纬度
|
||||
2. 将经纬度映射到 Web Mercator tile 坐标
|
||||
3. 找到对应的 tile 和像素
|
||||
4. 解码高程
|
||||
5. 将顶点沿法线方向抬升
|
||||
|
||||
### Needed helpers
|
||||
|
||||
建议新增:
|
||||
|
||||
- `latLonToTileXY(lat, lon, z)`
|
||||
- `tilePixelFromLatLon(lat, lon, z, tileSize)`
|
||||
- `decodeTerrariumHeight(r, g, b)`
|
||||
|
||||
## Caching Strategy
|
||||
|
||||
为了不让地形开关每次重开都重新抓全量 tile:
|
||||
|
||||
- terrain tile 按 `z/x/y` 存到内存缓存
|
||||
- terrain mesh 结果也缓存一份
|
||||
- 当用户关闭/开启 terrain:
|
||||
- 直接复用已有位移结果
|
||||
|
||||
建议:
|
||||
|
||||
- `Map<string, Float32Array | ImageBitmap>`
|
||||
|
||||
## Material Strategy
|
||||
|
||||
第一阶段不要复杂化。
|
||||
|
||||
建议 terrain material:
|
||||
|
||||
- 半透明低饱和地形色
|
||||
- 比 base earth 稍亮或稍偏冷
|
||||
- 保留当前 HUD 风格下的可读性
|
||||
|
||||
第一阶段不需要:
|
||||
|
||||
- 真实土地覆被纹理
|
||||
- 独立卫星影像贴 terrain
|
||||
|
||||
因为那会和现有地球纹理、云层、昼夜 shader 打架。
|
||||
|
||||
## Integration Points
|
||||
|
||||
### Files to change
|
||||
|
||||
- [earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js)
|
||||
- 重写 `createTerrain()`
|
||||
- 删除 simplex noise 占位逻辑
|
||||
- [main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js)
|
||||
- 初始化 terrain 数据加载
|
||||
- 控制 terrain readiness / loading message
|
||||
- [controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js)
|
||||
- `toggleTerrain` 逻辑保持,但应能区分:
|
||||
- mesh 已就绪
|
||||
- 正在加载
|
||||
- [constants.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/constants.js)
|
||||
- 新增 `TERRAIN_CONFIG`
|
||||
- 新文件:
|
||||
- `frontend/public/earth/js/terrain.js`
|
||||
|
||||
### Suggested new config
|
||||
|
||||
建议新增:
|
||||
|
||||
```js
|
||||
export const TERRAIN_CONFIG = {
|
||||
enabled: true,
|
||||
tileSize: 256,
|
||||
baseZoom: 4,
|
||||
baseRadiusOffset: 0.2,
|
||||
exaggeration: 1.5,
|
||||
opacity: 0.55,
|
||||
color: 0x6c876f,
|
||||
maxConcurrentRequests: 8,
|
||||
cacheEnabled: true,
|
||||
};
|
||||
```
|
||||
|
||||
## Loading UX
|
||||
|
||||
地形第一次开启时,不能像现在一样瞬时切换。
|
||||
|
||||
建议:
|
||||
|
||||
- 如果地形数据尚未准备:
|
||||
- 顶部状态条显示:`正在加载真实地形数据...`
|
||||
- 完成后:
|
||||
- `真实地形已就绪`
|
||||
|
||||
如果加载失败:
|
||||
|
||||
- 保留 base earth
|
||||
- 显示轻量错误提示
|
||||
- 不要让 terrain 开关卡死在“开”状态
|
||||
|
||||
## Risks
|
||||
|
||||
### 1. Global tile count too high
|
||||
|
||||
即使 `z=5` 全球 tile 数也不少。
|
||||
|
||||
缓解:
|
||||
|
||||
- 第一阶段限定低 zoom
|
||||
- 并发上限
|
||||
- 缓存
|
||||
|
||||
### 2. Mesh resolution too low
|
||||
|
||||
如果球面分段太低,山脉会被抹平。
|
||||
|
||||
缓解:
|
||||
|
||||
- 第一阶段先选一个中等分辨率
|
||||
- 用 exaggeration 保证可见性
|
||||
|
||||
### 3. Existing overlays may z-fight with terrain
|
||||
|
||||
海缆、登陆点、BGP、卫星相关对象都假设地球半径固定。
|
||||
|
||||
缓解:
|
||||
|
||||
- terrain sphere 单独作为 overlay
|
||||
- overlay 保持略低或略高的固定 offset
|
||||
- 必要时局部调整 landing point / cable altitude offset
|
||||
|
||||
### 4. Mercator sampling distortion near poles
|
||||
|
||||
Web Mercator 在高纬会有失真。
|
||||
|
||||
缓解:
|
||||
|
||||
- 第一阶段接受
|
||||
- 后续若需要更严格极区质量,再上 geodetic reprojection pipeline
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
第一阶段完成后,应满足:
|
||||
|
||||
1. `地形 terrain` 开关开启时,地表起伏明显不再是随机噪声
|
||||
2. 喜马拉雅、安第斯、落基山、东非高原等全球大尺度地形可辨认
|
||||
3. 关闭/重新开启 terrain 不重复全量请求
|
||||
4. 不破坏:
|
||||
- 海缆
|
||||
- 卫星
|
||||
- BGP
|
||||
- 地球昼夜
|
||||
- 天球层
|
||||
|
||||
## Suggested Execution Order
|
||||
|
||||
1. 引入 `TERRAIN_CONFIG`
|
||||
2. 新建 `terrain.js`
|
||||
3. 实现 Terrarium tile 请求与 decode
|
||||
4. 用低 zoom 全球 tile 构建真实 terrain sphere
|
||||
5. 接管 `toggleTerrain()`
|
||||
6. 调整 terrain material 和高度 exaggeration
|
||||
7. 做缓存
|
||||
8. 再考虑第二阶段局部高分 refinement
|
||||
|
||||
## Source References
|
||||
|
||||
- Mapzen Terrarium / AWS terrain tiles
|
||||
[Mapzen Terrain Tile Service](https://www.mapzen.com/blog/terrain-tile-service/)
|
||||
- Terrarium tile experiments / format background
|
||||
[mapzen/terrarium](https://github.com/mapzen/terrarium)
|
||||
- Copernicus DEM overview
|
||||
[Copernicus DEM docs](https://documentation.dataspace.copernicus.eu/APIs/SentinelHub/Data/DEM.html)
|
||||
|
||||
## Recommendation Summary
|
||||
|
||||
如果现在就要开始做,我建议直接按这条路线开工:
|
||||
|
||||
- 第一阶段接入 Terrarium 全球高程 tile
|
||||
- 替换掉当前 simplex 假地形
|
||||
- 先做一层真实可见的全球 terrain overlay
|
||||
- 等第一阶段稳定,再做视角高分 refinement
|
||||
|
||||
这是对当前项目风险最低、最贴合现有 Earth 架构的一条路。
|
||||
111
docs/plans/earth-renderer-architecture-separation-plan.md
Normal file
111
docs/plans/earth-renderer-architecture-separation-plan.md
Normal file
@@ -0,0 +1,111 @@
|
||||
# Earth Renderer / Logic Separation Plan
|
||||
|
||||
> Source note: this plan absorbs useful ideas from a sisyphus-created draft formerly stored at `.sisyphus/plans/earth-architecture-refactor.md`.
|
||||
|
||||
## Goal
|
||||
|
||||
将 Earth 前端继续往“逻辑层 / 状态层 / 渲染层”分离推进,降低后续这几类工作的耦合成本:
|
||||
|
||||
- Three.js 渲染重构
|
||||
- 部分图层替换实现
|
||||
- 未来 UE / Cesium 客户端迁移
|
||||
- Earth 行为逻辑复用
|
||||
|
||||
## Why This Matters
|
||||
|
||||
当前 Earth 已经有一些良好分层,例如:
|
||||
|
||||
- 图层显隐入口
|
||||
- Cable state 枚举与状态 map
|
||||
- 交互逻辑与实际视觉效果的部分分离
|
||||
|
||||
但还没有形成一套更明确的统一规则。现在的风险是:
|
||||
|
||||
- 同一类对象的 hover / locked / hidden / loading 语义不一致
|
||||
- 状态和渲染更新散落在多个模块
|
||||
- 后续再加新图层时容易复制旧逻辑
|
||||
|
||||
## Target Architecture
|
||||
|
||||
Earth 对每类对象都尽量拆成三层:
|
||||
|
||||
1. `state layer`
|
||||
- 保存对象状态
|
||||
- 例如:`normal / hovered / locked / hidden / loading`
|
||||
|
||||
2. `logic layer`
|
||||
- 处理点击、悬停、锁定、过滤、显隐切换
|
||||
- 不直接关心 Three.js 具体材质怎么改
|
||||
|
||||
3. `renderer layer`
|
||||
- 根据状态更新 Three.js / HUD 外观
|
||||
- 是最容易针对不同渲染引擎替换的一层
|
||||
|
||||
## Current Good Signals
|
||||
|
||||
当前已经接近这条方向的地方:
|
||||
|
||||
- cable 状态管理
|
||||
- 部分 landing point 状态同步
|
||||
- layer button 的统一状态入口
|
||||
- tooltip / legend / info-card 开始朝状态驱动靠拢
|
||||
|
||||
## Next Steps
|
||||
|
||||
### 1. Standardize object state enums
|
||||
|
||||
优先为这些对象建立更稳定的状态语义:
|
||||
|
||||
- cables
|
||||
- satellites
|
||||
- landing points
|
||||
- BGP markers
|
||||
- media / news 面板入口按钮
|
||||
|
||||
### 2. Unify state-to-visual adapters
|
||||
|
||||
为各模块建立更清晰的渲染适配函数,例如:
|
||||
|
||||
- `applyCableVisualState()`
|
||||
- `applySatelliteVisualState()`
|
||||
- `applyBGPVisualState()`
|
||||
|
||||
要求:
|
||||
|
||||
- 逻辑层只改状态
|
||||
- 视觉层负责把状态映射到材质、透明度、发光、尺寸、文字
|
||||
|
||||
### 3. Separate Earth UI state from render state
|
||||
|
||||
HUD / 面板 / 图层按钮状态也需要和渲染状态分离:
|
||||
|
||||
- `loading`
|
||||
- `active`
|
||||
- `locked`
|
||||
- `hidden`
|
||||
- `error`
|
||||
|
||||
不要再让 UI 通过“猜渲染结果”推导业务状态。
|
||||
|
||||
### 4. Prepare migration-safe boundaries
|
||||
|
||||
后续如果做 UE / Cesium 客户端,尽量保留:
|
||||
|
||||
- 状态枚举
|
||||
- 交互规则
|
||||
- 数据层接口
|
||||
|
||||
只替换:
|
||||
|
||||
- Three.js 具体渲染实现
|
||||
- HUD 展示实现
|
||||
|
||||
## Practical Rule
|
||||
|
||||
后续 Earth 新功能开发时,优先问三个问题:
|
||||
|
||||
1. 这个状态由谁持有?
|
||||
2. 这个交互逻辑在哪一层处理?
|
||||
3. 这个视觉变化是否能在不改逻辑的情况下单独替换?
|
||||
|
||||
如果答不上来,就说明还在把状态、逻辑、渲染揉在一起。
|
||||
82
docs/plans/earth-webgl-instancing-satellites-plan.md
Normal file
82
docs/plans/earth-webgl-instancing-satellites-plan.md
Normal file
@@ -0,0 +1,82 @@
|
||||
# Earth WebGL Instancing Satellites Plan
|
||||
|
||||
> Source note: this plan absorbs useful ideas from a sisyphus-created draft formerly stored at `.sisyphus/plans/webgl-instancing-satellites.md`.
|
||||
|
||||
## Goal
|
||||
|
||||
把 Earth 卫星渲染从当前方案继续推进到更适合高数量卫星的 instancing 方向,目标是:
|
||||
|
||||
- 支持更多卫星
|
||||
- 降低渲染压力
|
||||
- 仍然保留当前数据层和交互层
|
||||
|
||||
## Why It Matters
|
||||
|
||||
当前卫星系统已经具备:
|
||||
|
||||
- 数据加载
|
||||
- 轨迹
|
||||
- 选择/锁定
|
||||
- 图例
|
||||
- 相关区域联动
|
||||
|
||||
但当卫星数量持续增加时,渲染层会越来越接近瓶颈。
|
||||
|
||||
## Recommended Direction
|
||||
|
||||
优先调研并原型验证:
|
||||
|
||||
- `InstancedBufferGeometry + custom shader`
|
||||
|
||||
而不是一开始就推倒重写成 raw WebGL。
|
||||
|
||||
原因:
|
||||
|
||||
- 仍能保留 Three.js 主架构
|
||||
- 更容易渐进迁移
|
||||
- 比继续堆普通点渲染更有上限
|
||||
|
||||
## What Should Stay
|
||||
|
||||
尽量保留这些层:
|
||||
|
||||
- 卫星数据获取
|
||||
- 位置计算
|
||||
- 锁定/悬停逻辑
|
||||
- legend / info-card / 相关联动
|
||||
|
||||
主要替换的是:
|
||||
|
||||
- 卫星点渲染实现
|
||||
- 颜色/大小等实例属性更新方式
|
||||
|
||||
## Phases
|
||||
|
||||
### Phase 1: Prototype
|
||||
|
||||
- 用 instancing 做最小原型
|
||||
- 先只渲染卫星点
|
||||
- 不碰轨迹系统
|
||||
|
||||
### Phase 2: Integrate
|
||||
|
||||
- 接入当前 `satellites.js` 数据层
|
||||
- 保留当前选择和高亮语义
|
||||
|
||||
### Phase 3: Tune
|
||||
|
||||
- 调整可视大小
|
||||
- 调整选中高亮方式
|
||||
- 评估是否需要分层 LOD
|
||||
|
||||
## Risks
|
||||
|
||||
1. 透明度排序更复杂
|
||||
2. Shader 调试成本更高
|
||||
3. 选中态和 hover 态不能简单复用旧材质逻辑
|
||||
|
||||
## Acceptance
|
||||
|
||||
1. 在更高卫星数量下保持可接受帧率
|
||||
2. 不破坏现有锁定/高亮语义
|
||||
3. 图例、信息卡、相关卫星联动仍然成立
|
||||
793
docs/plans/enterprise-logging-system-plan.md
Normal file
793
docs/plans/enterprise-logging-system-plan.md
Normal file
@@ -0,0 +1,793 @@
|
||||
# Planet 企业级日志系统实施计划
|
||||
|
||||
## Goal
|
||||
|
||||
把 Planet 当前“能看一点运行输出”的日志能力,升级为一套真正可用、可定位、可纠错、可追责、可演进的企业级日志系统。
|
||||
|
||||
这里的“企业级”不是指一上来就接入很重的外部平台,而是指这套系统需要同时满足下面五件事:
|
||||
|
||||
1. 排障可用
|
||||
2. 历史可查
|
||||
3. 业务可解释
|
||||
4. 权限操作可追责
|
||||
5. 出错后能够反向定位到请求、任务、模块和操作者
|
||||
|
||||
最终目标不是“把更多 stdout 放到日志页里”,而是建立一套统一的日志契约与落地链路:
|
||||
|
||||
- 统一日志字段
|
||||
- 统一事件命名
|
||||
- 统一采集入口
|
||||
- 统一查询视图
|
||||
- 清晰的实时日志、持久化事件、审计日志分层
|
||||
|
||||
## Why
|
||||
|
||||
当前仓库已经有一些日志基础,但离真正可用的日志系统还有明显距离。
|
||||
|
||||
已有基础:
|
||||
|
||||
- 后端运行日志可通过 `/tmp/planet_backend.log` 查看
|
||||
- 前端开发服务日志可通过 `/tmp/planet_frontend.log` 查看
|
||||
- AI Provider 可从 Docker 容器读取日志
|
||||
- Earth 浏览器端关键日志可上报到后端并进入 Redis 缓冲
|
||||
- 已有 `system_logs` / `audit_logs` 持久化能力
|
||||
- 管理台已有“系统日志”页面,支持来源、级别、日期、搜索
|
||||
|
||||
当前缺口:
|
||||
|
||||
- 后端日志仍以 `uvicorn` / 文本输出为主,不是统一结构化事件流
|
||||
- 不同模块的日志格式不一致,很多地方只有 message,没有 event 语义
|
||||
- 还没有统一的后端 logger 封装与字段注入机制
|
||||
- 前端虽然能上报错误,但还没有统一 logger API 和统一事件词汇
|
||||
- Earth 与管理台之间的错误事件还没有形成可串联的事件链路
|
||||
- 历史持久化还偏点状,很多高价值失败并没有系统性落库
|
||||
- 系统日志页当前更像“运行输出查看器”,不是“多层日志查询台”
|
||||
- 审计日志与运行日志尚未形成明确的产品级联动
|
||||
|
||||
所以当前真正的问题不是“有没有日志页”,而是:
|
||||
|
||||
**当前系统能看见输出,但还不能稳定回答“发生了什么、影响了谁、在哪条链路上坏了、是否已修复、是谁触发的”。**
|
||||
|
||||
## Current State
|
||||
|
||||
截至 2026-04-23,当前代码中的日志相关能力大致如下。
|
||||
|
||||
### 1. 日志来源
|
||||
|
||||
当前系统日志页主要读取以下来源:
|
||||
|
||||
- `backend`
|
||||
读取 `/tmp/planet_backend.log`
|
||||
- `frontend`
|
||||
读取 `/tmp/planet_frontend.log`
|
||||
- `ai-provider`
|
||||
读取 Docker 容器日志
|
||||
- `earth-client`
|
||||
读取 Redis 缓冲的浏览器端日志
|
||||
|
||||
这些来源定义在:
|
||||
|
||||
- [backend/app/services/system_logs.py](/home/ray/dev/linkong/planet/backend/app/services/system_logs.py)
|
||||
|
||||
### 2. 当前日志读取模型
|
||||
|
||||
当前 `read_log_snapshot()` 的职责是:
|
||||
|
||||
- 读取某个来源的最近若干行
|
||||
- 解析基础级别与时间
|
||||
- 按级别、日期、搜索进行过滤
|
||||
- 返回用于日志页展示的快照
|
||||
|
||||
这个模型适合“运维查看器”,但不适合企业级日志系统,原因是:
|
||||
|
||||
- 读取基于文本尾部扫描,不是基于事件模型
|
||||
- 不同来源的结构粒度完全不同
|
||||
- 过滤依赖文本解析,准确率有限
|
||||
- 没有请求、任务、用户、资源、动作等核心关联字段
|
||||
|
||||
### 3. 已有持久化能力
|
||||
|
||||
当前已经存在两个持久化入口:
|
||||
|
||||
- `record_system_log(...)`
|
||||
- `record_audit_log(...)`
|
||||
|
||||
位置:
|
||||
|
||||
- [backend/app/services/persistent_logs.py](/home/ray/dev/linkong/planet/backend/app/services/persistent_logs.py)
|
||||
|
||||
这说明系统并不是从 0 开始,但也说明当前最大的问题是:
|
||||
|
||||
**持久化能力存在,但没有成为统一默认路径。**
|
||||
|
||||
### 4. 已有 request_id 基础
|
||||
|
||||
当前系统已具备 `request_id` 相关基础,部分持久化能力也会尝试写入 `request_id`。
|
||||
|
||||
这为后续做:
|
||||
|
||||
- 请求链路排障
|
||||
- 前后端关联查询
|
||||
- 任务执行追踪
|
||||
|
||||
提供了很好的基础。
|
||||
|
||||
### 5. 当前日志页定位
|
||||
|
||||
当前日志页已经具备:
|
||||
|
||||
- 来源切换
|
||||
- 级别筛选
|
||||
- 日期筛选
|
||||
- 搜索
|
||||
- 文本控制台视图
|
||||
|
||||
但它仍然是“单层视图”:
|
||||
|
||||
- 上面是筛选器
|
||||
- 下面是一块文本控制台
|
||||
|
||||
它还不是:
|
||||
|
||||
- 运行日志 + 事件日志 + 审计日志 的统一入口
|
||||
- 也没有事件详情、关联跳转、纠错建议、链路追踪能力
|
||||
|
||||
## Core Principles
|
||||
|
||||
这套日志系统后续必须遵循下面几个原则。
|
||||
|
||||
### 1. 分层,而不是混存
|
||||
|
||||
日志必须拆成三层:
|
||||
|
||||
1. 运行日志
|
||||
2. 持久化事件日志
|
||||
3. 审计日志
|
||||
|
||||
它们的用途不同,绝不能继续混成一个概念。
|
||||
|
||||
#### 运行日志
|
||||
|
||||
用于:
|
||||
|
||||
- 实时排障
|
||||
- 观察服务运行状态
|
||||
- 看 stdout / stderr / exception / collector 输出
|
||||
|
||||
特点:
|
||||
|
||||
- 数据量大
|
||||
- 时效性强
|
||||
- 保留周期短
|
||||
- 不要求每条都落库
|
||||
|
||||
#### 持久化事件日志
|
||||
|
||||
用于:
|
||||
|
||||
- 记录高价值错误
|
||||
- 记录关键业务失败
|
||||
- 支撑历史追溯
|
||||
- 支撑趋势分析
|
||||
|
||||
特点:
|
||||
|
||||
- 只持久化有价值事件
|
||||
- 必须结构化
|
||||
- 必须有统一 event 命名
|
||||
|
||||
#### 审计日志
|
||||
|
||||
用于:
|
||||
|
||||
- 留痕
|
||||
- 追责
|
||||
- 还原高权限操作
|
||||
|
||||
特点:
|
||||
|
||||
- 必须单独建模
|
||||
- 不与普通运行日志混用
|
||||
|
||||
### 2. 结构化优先
|
||||
|
||||
正式日志必须可拆字段,不能长期依赖自由文本。
|
||||
|
||||
最低要求至少能拿到:
|
||||
|
||||
- `timestamp`
|
||||
- `level`
|
||||
- `service`
|
||||
- `module`
|
||||
- `event`
|
||||
- `message`
|
||||
- `request_id`
|
||||
- `trace_id`
|
||||
- `user_id` / `actor`
|
||||
- `context`
|
||||
|
||||
### 3. 事件命名优先于 message 命名
|
||||
|
||||
人看的 message 可以变化,但机器查询和跨模块关联必须依赖稳定事件名。
|
||||
|
||||
例如:
|
||||
|
||||
- `collector.run.started`
|
||||
- `collector.run.completed`
|
||||
- `collector.run.failed`
|
||||
- `earth.layer.load_failed`
|
||||
- `earth.cruise.route_build_failed`
|
||||
- `system.restart_task.failed`
|
||||
- `auth.websocket.invalid_token`
|
||||
|
||||
### 4. 查询链路必须可串联
|
||||
|
||||
企业级日志系统的核心不是“有很多日志”,而是“能串起来”。
|
||||
|
||||
最终一条高价值事件,至少要能回链到下面任意几类对象:
|
||||
|
||||
- 某个请求
|
||||
- 某个任务
|
||||
- 某个用户
|
||||
- 某个数据源
|
||||
- 某个 Earth 模块
|
||||
- 某个管理动作
|
||||
|
||||
### 5. 默认脱敏
|
||||
|
||||
日志体系必须明确禁止记录:
|
||||
|
||||
- token
|
||||
- password
|
||||
- Authorization header
|
||||
- cookie
|
||||
- session
|
||||
- 明文敏感个人信息
|
||||
|
||||
并且需要有统一脱敏器,而不是靠调用者自觉。
|
||||
|
||||
### 6. “可纠错”不是一句口号
|
||||
|
||||
这里的“可纠错”至少包含三层:
|
||||
|
||||
1. 日志字段足够解释错误,方便人排查
|
||||
2. 系统能识别常见错误模式并给出纠偏建议
|
||||
3. 关键错误支持闭环动作,例如重试、重建索引、重新触发采集、跳转到对应对象
|
||||
|
||||
也就是说,这套日志系统最终不只是“告诉你出错了”,而要尽量接近“告诉你为什么出错、怎么修、去哪修”。
|
||||
|
||||
## Non-Goals
|
||||
|
||||
第一阶段不追求:
|
||||
|
||||
- 全量接入 ELK / Loki / Datadog / OpenTelemetry 全家桶
|
||||
- 做分布式 trace 全链路可视化大屏
|
||||
- 把所有历史日志都迁进数据库
|
||||
- 先做特别复杂的规则引擎
|
||||
|
||||
第一阶段追求的是:
|
||||
|
||||
- 在当前仓库和当前部署方式下,先把基础日志体系做正确
|
||||
- 再为后续平台化接入预留好接口
|
||||
|
||||
## Target Architecture
|
||||
|
||||
推荐目标架构如下。
|
||||
|
||||
### Layer 1: Runtime Logs
|
||||
|
||||
职责:
|
||||
|
||||
- 承载后端、前端开发服务、容器输出、浏览器端缓冲事件
|
||||
- 提供最近窗口内的实时查看能力
|
||||
|
||||
来源:
|
||||
|
||||
- 文件
|
||||
- Docker
|
||||
- Redis 缓冲
|
||||
- 后续可扩展到 stdout collector
|
||||
|
||||
接口:
|
||||
|
||||
- `GET /api/v1/system/logs/sources`
|
||||
- `GET /api/v1/system/logs/{source_id}`
|
||||
|
||||
这层继续保留,但需要做结构化增强和来源补强。
|
||||
|
||||
### Layer 2: Persistent System Events
|
||||
|
||||
职责:
|
||||
|
||||
- 只存高价值事件
|
||||
- 供历史追溯、事件列表、趋势和纠错使用
|
||||
|
||||
数据来源:
|
||||
|
||||
- 后端关键异常
|
||||
- 浏览器端关键失败
|
||||
- 采集器/调度器关键失败
|
||||
- 业务关键告警与降级事件
|
||||
|
||||
接口建议:
|
||||
|
||||
- `GET /api/v1/system/events`
|
||||
- `GET /api/v1/system/events/{id}`
|
||||
- `POST /api/v1/system/events/{id}/actions/...`(后续)
|
||||
|
||||
### Layer 3: Audit Logs
|
||||
|
||||
职责:
|
||||
|
||||
- 留痕高权限操作
|
||||
- 记录操作者、对象、结果、请求号
|
||||
|
||||
接口建议:
|
||||
|
||||
- `GET /api/v1/system/audit-logs`
|
||||
|
||||
### Layer 4: Error Intelligence / Triage
|
||||
|
||||
职责:
|
||||
|
||||
- 对高频错误做归类
|
||||
- 对已知错误给出解释与建议动作
|
||||
- 对相同错误进行 fingerprint 聚合
|
||||
|
||||
这是“可纠错”能力的关键层。
|
||||
|
||||
建议字段:
|
||||
|
||||
- `fingerprint`
|
||||
- `root_cause_type`
|
||||
- `known_fix_hint`
|
||||
- `runbook_url`
|
||||
- `related_resource_type`
|
||||
- `related_resource_id`
|
||||
|
||||
## Canonical Event Model
|
||||
|
||||
推荐统一事件字段模型如下。
|
||||
|
||||
### Runtime Log Record
|
||||
|
||||
```json
|
||||
{
|
||||
"timestamp": "2026-04-23T10:15:30Z",
|
||||
"level": "error",
|
||||
"service": "backend",
|
||||
"module": "app.services.scheduler",
|
||||
"event": "collector.run.failed",
|
||||
"message": "Collector bgp_news failed",
|
||||
"request_id": "req_xxx",
|
||||
"trace_id": "trace_xxx",
|
||||
"user_id": null,
|
||||
"actor": null,
|
||||
"resource_type": "collector",
|
||||
"resource_id": "bgp_news",
|
||||
"context": {
|
||||
"datasource_id": 12,
|
||||
"exception_type": "TimeoutError"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Persistent System Event
|
||||
|
||||
```json
|
||||
{
|
||||
"id": 1024,
|
||||
"event": "earth.layer.load_failed",
|
||||
"level": "error",
|
||||
"source": "earth-client",
|
||||
"service": "earth",
|
||||
"module": "cables",
|
||||
"message": "Failed to load cable layer",
|
||||
"fingerprint": "earth.layer.load_failed:cables:network_timeout",
|
||||
"request_id": "req_xxx",
|
||||
"trace_id": null,
|
||||
"user_id": 1,
|
||||
"resource_type": "earth_layer",
|
||||
"resource_id": "cables",
|
||||
"category": "visualization",
|
||||
"status": "open",
|
||||
"context": {
|
||||
"url": "/api/v1/visualization/geo/cables"
|
||||
},
|
||||
"created_at": "2026-04-23T10:15:30Z"
|
||||
}
|
||||
```
|
||||
|
||||
### Audit Log
|
||||
|
||||
```json
|
||||
{
|
||||
"id": 88,
|
||||
"action": "system.restart_task.requested",
|
||||
"actor_id": 1,
|
||||
"actor_name": "root",
|
||||
"target_type": "restart_task",
|
||||
"target_id": "restart_20260423_xxx",
|
||||
"result": "success",
|
||||
"request_id": "req_xxx",
|
||||
"ip": "127.0.0.1",
|
||||
"details": {
|
||||
"action": "restart_backend"
|
||||
},
|
||||
"created_at": "2026-04-23T10:15:30Z"
|
||||
}
|
||||
```
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
## Phase 0: Logging Inventory And Naming Freeze
|
||||
|
||||
目标:
|
||||
|
||||
- 先统一“记录什么”和“怎么命名”,避免后面越做越乱
|
||||
|
||||
工作项:
|
||||
|
||||
- 盘点当前所有 `logging.getLogger` 使用点
|
||||
- 盘点裸 `print`
|
||||
- 盘点 `record_system_log` / `record_audit_log` 已落点位
|
||||
- 建立统一事件命名表
|
||||
- 定义 service / module / category / resource 字段枚举
|
||||
- 输出日志字段白名单和脱敏规范
|
||||
|
||||
完成标准:
|
||||
|
||||
- 有一份稳定的事件命名清单
|
||||
- 有一份字段规范清单
|
||||
- 后续新增日志不再“临时起名”
|
||||
|
||||
## Phase 1: Backend Structured Logging Foundation
|
||||
|
||||
目标:
|
||||
|
||||
- 把后端从“散落 logging + 文本输出”升级成“统一结构化 logger”
|
||||
|
||||
工作项:
|
||||
|
||||
- 新增统一后端 logger helper,例如 `app/core/logging.py`
|
||||
- 自动注入:
|
||||
- `service`
|
||||
- `module`
|
||||
- `request_id`
|
||||
- `trace_id`
|
||||
- 增加统一脱敏 filter
|
||||
- 把关键模块先切到统一 logger:
|
||||
- API 层
|
||||
- scheduler
|
||||
- collectors
|
||||
- websocket
|
||||
- visualization
|
||||
- system control
|
||||
- 约束:
|
||||
- 正式路径禁止裸 `print`
|
||||
- 正式异常优先 `logger.exception(..., extra={...})`
|
||||
|
||||
完成标准:
|
||||
|
||||
- 后端关键模块都有稳定 `event`
|
||||
- request 日志和异常日志能挂上 `request_id`
|
||||
- 不再依赖只看 `uvicorn` 原生文本输出来定位问题
|
||||
|
||||
## Phase 2: Persistent Event Layer
|
||||
|
||||
目标:
|
||||
|
||||
- 把“值得长期保留的错误和关键事件”系统性落库
|
||||
|
||||
工作项:
|
||||
|
||||
- 重新定义 `record_system_log()` 的使用边界
|
||||
- 明确哪些事件必须持久化:
|
||||
- API 关键失败
|
||||
- 调度器失败
|
||||
- 采集器失败
|
||||
- Earth 客户端关键错误
|
||||
- 数据源不可用
|
||||
- 业务降级与恢复
|
||||
- 补齐字段:
|
||||
- `event`
|
||||
- `resource_type`
|
||||
- `resource_id`
|
||||
- `category`
|
||||
- `fingerprint`
|
||||
- `status`
|
||||
- 增加高频错误去重/聚合策略
|
||||
|
||||
完成标准:
|
||||
|
||||
- 高价值错误不再只存在于运行日志里
|
||||
- 能查询最近一周/一月的关键失败事件
|
||||
- 相同错误具备聚合基础
|
||||
|
||||
## Phase 3: Frontend And Earth Unified Logger
|
||||
|
||||
目标:
|
||||
|
||||
- 把前端从“点状 error 上报”升级成统一前端事件流
|
||||
|
||||
工作项:
|
||||
|
||||
- 在前端新增统一 logger API
|
||||
- 统一方法:
|
||||
- `debug`
|
||||
- `info`
|
||||
- `warn`
|
||||
- `error`
|
||||
- 统一字段:
|
||||
- `page`
|
||||
- `module`
|
||||
- `event`
|
||||
- `message`
|
||||
- `url`
|
||||
- `user_agent`
|
||||
- `context`
|
||||
- Earth 模块优先接入:
|
||||
- layer load failed
|
||||
- cruise build failed
|
||||
- popup render failed
|
||||
- connector render failed
|
||||
- websocket dropped
|
||||
- 管理台优先接入:
|
||||
- settings save failed
|
||||
- datasource toggle failed
|
||||
- restart task submit failed
|
||||
|
||||
完成标准:
|
||||
|
||||
- 前端日志事件名与后端可对齐
|
||||
- Earth 和管理台关键失败不再只停留在 console
|
||||
- 浏览器端关键问题能进入统一系统日志/事件层
|
||||
|
||||
## Phase 4: Audit Logging Completion
|
||||
|
||||
目标:
|
||||
|
||||
- 把管理员与高权限操作真正做成企业级审计
|
||||
|
||||
工作项:
|
||||
|
||||
- 扩大审计覆盖面:
|
||||
- 系统重启
|
||||
- 数据源启停
|
||||
- 调度规则变更
|
||||
- 配置变更
|
||||
- 人工触发采集
|
||||
- 删除/修改关键配置
|
||||
- 增加字段:
|
||||
- actor
|
||||
- target
|
||||
- before / after
|
||||
- request_id
|
||||
- IP
|
||||
- 审计页支持:
|
||||
- 动作筛选
|
||||
- 操作者筛选
|
||||
- 时间筛选
|
||||
- 目标对象筛选
|
||||
|
||||
完成标准:
|
||||
|
||||
- 所有高权限操作都能追到人、时间、对象、结果
|
||||
|
||||
## Phase 5: Log Console To Enterprise Observability UI
|
||||
|
||||
目标:
|
||||
|
||||
- 把当前“系统日志”页升级为真正的多层日志工作台
|
||||
|
||||
工作项:
|
||||
|
||||
- 将页面拆为三个主视图:
|
||||
1. 运行日志
|
||||
2. 关键事件
|
||||
3. 审计日志
|
||||
- 运行日志视图:
|
||||
- 保留大控制台
|
||||
- 支持来源、级别、日期、搜索
|
||||
- 关键事件视图:
|
||||
- 列表化展示高价值事件
|
||||
- 支持聚合、状态、指纹、对象筛选
|
||||
- 审计视图:
|
||||
- 列表化展示管理员动作
|
||||
- 增加详情抽屉:
|
||||
- 原始 message
|
||||
- context
|
||||
- request_id
|
||||
- related resource
|
||||
- recommended action
|
||||
|
||||
完成标准:
|
||||
|
||||
- 日志页不再只是“终端文本窗口”
|
||||
- 运维排障、历史追溯、审计留痕三者分层清晰
|
||||
|
||||
## Phase 6: Corrective Intelligence
|
||||
|
||||
目标:
|
||||
|
||||
- 让系统从“能看日志”进化到“能辅助修错”
|
||||
|
||||
工作项:
|
||||
|
||||
- 引入错误 fingerprint
|
||||
- 对已知错误配置:
|
||||
- 根因类型
|
||||
- 修复建议
|
||||
- runbook 链接
|
||||
- 推荐动作
|
||||
- 支持常见纠错动作:
|
||||
- 重试采集任务
|
||||
- 重载配置
|
||||
- 跳转到对应模块/资源
|
||||
- 打开相关日志过滤视图
|
||||
- 高频错误支持聚合与静默窗口
|
||||
|
||||
完成标准:
|
||||
|
||||
- 已知错误能给出明确建议
|
||||
- 运维不需要每次都从零猜
|
||||
|
||||
## Recommended Module Changes
|
||||
|
||||
### Backend
|
||||
|
||||
建议新增/增强的模块:
|
||||
|
||||
- `backend/app/core/logging.py`
|
||||
- 统一 logger 封装
|
||||
- formatter
|
||||
- filter
|
||||
- request/trace 注入
|
||||
- `backend/app/services/persistent_logs.py`
|
||||
- 扩展字段
|
||||
- 统一持久化策略
|
||||
- `backend/app/services/system_logs.py`
|
||||
- 逐步从“文本尾部查看器”升级为“运行日志聚合器”
|
||||
- `backend/app/services/log_classification.py`
|
||||
- 指纹
|
||||
- 根因分类
|
||||
- 纠错建议
|
||||
- `backend/app/api/v1/system_control.py`
|
||||
- 补充事件 / 审计 / 日志多视图接口
|
||||
|
||||
### Frontend
|
||||
|
||||
建议新增/增强:
|
||||
|
||||
- `frontend/src/lib/logger.ts`
|
||||
- 统一前端 logger API
|
||||
- `frontend/src/pages/Logs/Logs.tsx`
|
||||
- 升级为多层工作台
|
||||
- `frontend/public/earth/js/...`
|
||||
- 各 Earth 模块接入统一事件 logger
|
||||
|
||||
## Event Naming Convention
|
||||
|
||||
建议采用:
|
||||
|
||||
`<domain>.<resource>.<action>.<result>`
|
||||
|
||||
示例:
|
||||
|
||||
- `collector.datasource.run.started`
|
||||
- `collector.datasource.run.failed`
|
||||
- `earth.layer.cables.load.failed`
|
||||
- `earth.cruise.route.build.failed`
|
||||
- `system.restart_task.requested`
|
||||
- `system.restart_task.completed`
|
||||
- `auth.websocket.connect.failed`
|
||||
- `settings.datasource.priority.updated`
|
||||
|
||||
规则:
|
||||
|
||||
- 不用自然语言句子
|
||||
- 不把 ID 塞进 event 名里
|
||||
- 资源对象通过字段承载,不通过 event 名承载
|
||||
|
||||
## Query Model
|
||||
|
||||
最终推荐支持的查询维度:
|
||||
|
||||
- 时间范围
|
||||
- level
|
||||
- source
|
||||
- service
|
||||
- module
|
||||
- event
|
||||
- request_id
|
||||
- trace_id
|
||||
- user_id / actor
|
||||
- resource_type / resource_id
|
||||
- category
|
||||
- fingerprint
|
||||
- status
|
||||
- full-text search
|
||||
|
||||
## Retention Strategy
|
||||
|
||||
推荐保留策略:
|
||||
|
||||
- 运行日志:
|
||||
- 文件 / 容器 / Redis 缓冲保留短周期
|
||||
- 持久化事件:
|
||||
- 保留中长期
|
||||
- 审计日志:
|
||||
- 长期保留
|
||||
|
||||
初版可以先这样:
|
||||
|
||||
- 运行日志:7 到 14 天
|
||||
- 关键事件:90 到 180 天
|
||||
- 审计日志:180 天以上
|
||||
|
||||
后续再根据存储与合规要求调整。
|
||||
|
||||
## Security And Compliance
|
||||
|
||||
必须落实:
|
||||
|
||||
- 敏感字段脱敏
|
||||
- 前端上报白名单
|
||||
- 防止日志注入
|
||||
- 审计日志不可被普通管理员随意篡改
|
||||
- 高敏感纠错动作必须再次鉴权
|
||||
|
||||
## Success Criteria
|
||||
|
||||
当下面这些条件成立时,才算这套日志系统真的“成了”:
|
||||
|
||||
1. 一个后端请求失败时,能通过 `request_id` 在运行日志、持久化事件、审计日志之间串联查询
|
||||
2. 一个 Earth 前端错误能定位到页面、模块、事件名和上下文
|
||||
3. 一个采集器失败能同时看到运行日志、持久化事件和可执行纠错动作
|
||||
4. 一个管理员操作能查到操作者、目标对象、结果和 request_id
|
||||
5. 日志页不再只是文本控制台,而是完整的“运行日志 / 关键事件 / 审计日志”工作台
|
||||
6. 高频已知错误能聚合并给出修复建议
|
||||
|
||||
## Delivery Order
|
||||
|
||||
推荐严格按下面顺序做,不要乱跳:
|
||||
|
||||
1. Phase 0 命名与字段规范冻结
|
||||
2. Phase 1 后端结构化 logging 基础
|
||||
3. Phase 2 高价值事件持久化
|
||||
4. Phase 3 前端 / Earth 统一 logger
|
||||
5. Phase 4 审计覆盖补齐
|
||||
6. Phase 5 日志工作台 UI 重构
|
||||
7. Phase 6 指纹 / 纠错 / runbook
|
||||
|
||||
原因:
|
||||
|
||||
- 如果不先统一字段和命名,后面 UI 和持久化会越来越乱
|
||||
- 如果不先做后端结构化基础,前端上报再多也串不起来
|
||||
- 如果不先补持久化层,就只有“实时可看”,没有“历史可查”
|
||||
|
||||
## First Actionable Milestone
|
||||
|
||||
如果要从明天就开始做,最合理的第一个里程碑是:
|
||||
|
||||
### M1: 让后端关键路径全部拥有统一结构化事件
|
||||
|
||||
范围:
|
||||
|
||||
- API 请求入口/出口
|
||||
- scheduler
|
||||
- collectors
|
||||
- websocket
|
||||
- visualization
|
||||
- system control
|
||||
|
||||
交付物:
|
||||
|
||||
- 统一 logger helper
|
||||
- 统一 event naming 表
|
||||
- 统一 request_id 注入
|
||||
- 统一脱敏策略
|
||||
- 关键模块替换完成
|
||||
|
||||
完成这个里程碑后,Planet 才算真正拥有了“企业级日志系统的地基”。
|
||||
|
||||
@@ -30,7 +30,7 @@
|
||||
- [backend/app/services/ai_client.py](/home/ray/dev/linkong/planet/backend/app/services/ai_client.py)
|
||||
- [aiprovider/main.py](/home/ray/dev/linkong/planet/aiprovider/main.py)
|
||||
- [aiprovider/provider_service.py](/home/ray/dev/linkong/planet/aiprovider/provider_service.py)
|
||||
- [docs/agents/aiprovider.md](/home/ray/dev/linkong/planet/docs/agents/aiprovider.md)
|
||||
- [docs/technical/agents-aiprovider.md](/home/ray/dev/linkong/planet/docs/technical/agents-aiprovider.md)
|
||||
|
||||
### 2. 本地运行与配置打通
|
||||
|
||||
@@ -77,7 +77,7 @@
|
||||
|
||||
相关文件:
|
||||
|
||||
- [docs/frontend/frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/frontend/frontend-layout-guidelines.md)
|
||||
- [docs/technical/frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md)
|
||||
- [frontend/src/pages/BGP/BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx)
|
||||
|
||||
## 当前限制
|
||||
@@ -979,3 +979,37 @@ Content/
|
||||
如果你按这份方案推进,一期最现实的目标不是“立刻做出完整 UE 大屏”,而是:
|
||||
|
||||
**在 14 天左右,做出一个能显示真实地球、能显示超算点、能点击看详情、能接后端的可用 UE 客户端 MVP。**
|
||||
|
||||
---
|
||||
|
||||
# 附录:来自 sisyphus 草案的补充
|
||||
|
||||
> 这部分吸收自一个 sisyphus-created draft,原始草案已归档,不再单独维护为主计划。
|
||||
|
||||
## 1. 项目骨架建议
|
||||
|
||||
原草案给过一个更偏“工程初始化”的目录示意,适合拿来做一期的命名参考:
|
||||
|
||||
- `Levels/`
|
||||
- `Blueprints/`
|
||||
- `Materials/`
|
||||
- `Widgets/`
|
||||
- `Source/PlanetAPI/`
|
||||
- `Source/CesiumIntegration/`
|
||||
- `Source/Visualization/`
|
||||
|
||||
这不是强制结构,但对 UE 初期整理目录很有帮助。
|
||||
|
||||
## 2. API 契约意识
|
||||
|
||||
原草案有一个很对的提醒:
|
||||
|
||||
- 一期虽然可以先走 HTTP
|
||||
- 但数据模型命名不应只服务于一次性演示
|
||||
- 后续 WebSocket 接入时,字段设计最好能沿用
|
||||
|
||||
所以当前主计划继续建议:
|
||||
|
||||
- 先做 HTTP 拉取
|
||||
- 尽量把 UE 侧数据模型定义清楚
|
||||
- 不要在蓝图各处散写临时 JSON 字段解析
|
||||
@@ -1,216 +0,0 @@
|
||||
# Prefix Geography Plan
|
||||
|
||||
## Goal
|
||||
|
||||
Make Earth BGP incidents `prefix-centric` instead of `collector-centric`.
|
||||
|
||||
The map should primarily answer:
|
||||
|
||||
- where a prefix-related event is likely centered
|
||||
- which regions the prefix is likely associated with
|
||||
- which collectors observed the event as evidence
|
||||
|
||||
It should not continue to imply that the event is located at the collector itself unless no better geography is available.
|
||||
|
||||
## Why Current Geography Is Not Enough
|
||||
|
||||
Current incident geography can still collapse back to collector-derived regions because:
|
||||
|
||||
1. `prefix_scope` is currently built mostly from observed collector regions and historical observation regions.
|
||||
2. `origin_asn_profile` currently comes from `peeringdb_network`, which is useful for ASN footprint hints but not sufficient as a primary prefix location source.
|
||||
3. `collector centroid` is still a common fallback and therefore dominates sparse incidents.
|
||||
|
||||
This makes Earth feel like a collector map with event decorations instead of a prefix impact map.
|
||||
|
||||
## Data Source Layers
|
||||
|
||||
Prefix geography should be built from four layers, ordered by confidence.
|
||||
|
||||
### Layer 1. Prefix-to-country / prefix-to-region
|
||||
|
||||
This is the primary source layer and the current missing piece.
|
||||
|
||||
Recommended sources:
|
||||
|
||||
1. `IPtoASN / IPtoCountry`
|
||||
- URL: <https://iptoasn.com/>
|
||||
- Good fit for this project because it provides downloadable IPv4/IPv6 range-to-ASN and range-to-country mappings.
|
||||
- Best use:
|
||||
- map a prefix to country code
|
||||
- enrich prefixes with coarse regional placement
|
||||
|
||||
2. `OpenGeoFeed`
|
||||
- URL: <https://opengeofeed.org/faq/>
|
||||
- Best use:
|
||||
- override coarse country mappings when the prefix holder publishes a geofeed
|
||||
- provide a more realistic deployment/service region than whois-style registration country
|
||||
|
||||
### Layer 2. Registry allocation fallback
|
||||
|
||||
Use these only as fallback signals, not as a ground-truth physical location.
|
||||
|
||||
Candidate inputs:
|
||||
|
||||
- RIR delegated stats
|
||||
- `inetnum` / `inet6num` whois
|
||||
|
||||
Best use:
|
||||
|
||||
- detect registration country / allocation region
|
||||
- provide fallback when no direct prefix geolocation dataset is available
|
||||
|
||||
### Layer 3. ASN footprint hints
|
||||
|
||||
Existing in this project:
|
||||
|
||||
- `peeringdb_network`
|
||||
- `peeringdb_facility`
|
||||
- `peeringdb_ixp`
|
||||
|
||||
Best use:
|
||||
|
||||
- derive ASN city/country footprint
|
||||
- identify likely exchange/facility regions
|
||||
- act as secondary evidence when prefix-specific geography is unavailable
|
||||
|
||||
### Layer 4. Observation evidence
|
||||
|
||||
Existing in this project:
|
||||
|
||||
- `RIPE RIS Live`
|
||||
- `CAIDA BGPStream Backfill`
|
||||
|
||||
Best use:
|
||||
|
||||
- prove who observed the event
|
||||
- derive affected observation regions
|
||||
- support impact evidence
|
||||
|
||||
This should remain the final fallback and evidence layer, not the primary event geography.
|
||||
|
||||
## Recommended Geography Priority
|
||||
|
||||
The backend should compute incident geography with this order:
|
||||
|
||||
1. `prefix_geography`
|
||||
- prefix-to-country / region / geofeed-backed result
|
||||
2. `asn_region`
|
||||
- ASN organization / facility / IXP footprint
|
||||
3. `collector_centroid`
|
||||
- observed collector regions only as final fallback
|
||||
|
||||
Returned GeoJSON should keep exposing the selected mode through:
|
||||
|
||||
- `geography_mode = prefix_geography | asn_region | collector_centroid`
|
||||
|
||||
## Proposed Backend Changes
|
||||
|
||||
### 1. Add a dedicated prefix geography dataset
|
||||
|
||||
New datasource candidates:
|
||||
|
||||
- `ip2asn_prefix_geo`
|
||||
- optionally `opengeofeed_prefix_geo`
|
||||
|
||||
Suggested storage model:
|
||||
|
||||
- keep downloaded rows in `CollectedData` first for speed of integration
|
||||
- later move to a dedicated table if lookup volume grows
|
||||
|
||||
Minimum normalized fields:
|
||||
|
||||
- `range_start`
|
||||
- `range_end`
|
||||
- `prefix`
|
||||
- `country`
|
||||
- `continent`
|
||||
- `asn`
|
||||
- `as_name`
|
||||
- `source`
|
||||
- `confidence`
|
||||
|
||||
### 2. Add prefix geography enrichment
|
||||
|
||||
Extend:
|
||||
|
||||
- `backend/app/services/bgp_enrichment.py`
|
||||
|
||||
New enrichment payload should include:
|
||||
|
||||
- `prefix_geography`
|
||||
- `country`
|
||||
- `continent`
|
||||
- `regions`
|
||||
- `source`
|
||||
- `confidence`
|
||||
|
||||
This should be separate from the current `prefix_scope`.
|
||||
|
||||
Suggested distinction:
|
||||
|
||||
- `prefix_scope`
|
||||
- observation-derived scope hint
|
||||
- `prefix_geography`
|
||||
- prefix-centric geography estimate
|
||||
|
||||
### 3. Update incident visualization geography selection
|
||||
|
||||
Extend:
|
||||
|
||||
- `backend/app/api/v1/visualization.py`
|
||||
|
||||
Selection order:
|
||||
|
||||
1. `prefix_geography.regions`
|
||||
2. ASN geography hints from PeeringDB-derived profile
|
||||
3. observation-derived `affected_regions`
|
||||
|
||||
### 4. Keep evidence visible in the frontend
|
||||
|
||||
Earth should distinguish:
|
||||
|
||||
- event center = prefix geography estimate
|
||||
- evidence lines / collectors = observation proof
|
||||
|
||||
This keeps the event meaningful for non-expert users without losing collector evidence.
|
||||
|
||||
## Earth UX Result
|
||||
|
||||
After this change, a user should see:
|
||||
|
||||
- an incident marker near the estimated affected prefix region
|
||||
- collectors as supporting evidence, not as the event center itself
|
||||
- cables / landing points / nearby infrastructure as weak correlation around the estimated region
|
||||
|
||||
This makes BGP incidents readable as “where the event is likely happening or affecting”, instead of “which station saw it”.
|
||||
|
||||
## Implementation Order
|
||||
|
||||
### Phase 1
|
||||
|
||||
1. Add `IPtoASN / IPtoCountry` datasource support
|
||||
2. Normalize rows into lookup-friendly format
|
||||
3. Enrich BGP events with `prefix_geography`
|
||||
4. Switch incident geography priority to prefer `prefix_geography`
|
||||
|
||||
### Phase 2
|
||||
|
||||
5. Add `OpenGeoFeed` support
|
||||
6. Let geofeed override coarse country-level prefix geography
|
||||
7. Add confidence scoring per geography source
|
||||
|
||||
### Phase 3
|
||||
|
||||
8. Add RIR / whois fallback
|
||||
9. Add better ASN regional footprint from PeeringDB facilities / IXPs
|
||||
10. Refine Earth visual semantics for prefix geography vs observation evidence
|
||||
|
||||
## Recommendation
|
||||
|
||||
The best next engineering move is:
|
||||
|
||||
1. integrate `IPtoASN / IPtoCountry`
|
||||
2. model `prefix_geography` separately from `prefix_scope`
|
||||
3. only then continue refining incident map placement
|
||||
|
||||
Without this layer, any further Earth tuning will still be constrained by collector-centric data.
|
||||
@@ -1,309 +0,0 @@
|
||||
# Situational Awareness Foundation Plan
|
||||
|
||||
## 定位
|
||||
|
||||
当前这套 AI 能力应被视为 `态势感知服务底座`,而不是完整的态势感知产品。
|
||||
|
||||
也就是说,现阶段的目标不是:
|
||||
|
||||
- 做一个“什么都能分析”的万能 AI 页面
|
||||
- 让模型在证据不足时替代人工研判
|
||||
- 过早把页面做成完整指挥大屏
|
||||
|
||||
现阶段真正要做的是:
|
||||
|
||||
- 先把 `model gateway / backend facade / evidence injection / page-specific brief` 这几层边界搭稳
|
||||
- 让系统能够在已有证据上稳定地产出“可读、可回看、可扩展”的摘要
|
||||
- 为后续更强的数据联动、agent 推理和 assessment 结构化输出预留好接口与数据模型
|
||||
|
||||
## 当前现实约束
|
||||
|
||||
### 1. 数据维度不足
|
||||
|
||||
目前系统能提供的主要证据仍集中在:
|
||||
|
||||
- BGP incidents / anomalies / events
|
||||
- collector coverage
|
||||
- datasource health / platform alerts
|
||||
- prefix geography 的部分归属信息
|
||||
|
||||
当前明显还缺:
|
||||
|
||||
- 流量异常与业务指标
|
||||
- 电商、支付、物流等业务侧指标
|
||||
- 更丰富的资产、链路、区域、行业画像
|
||||
- 外部舆情、公告、运营商状态、基础设施事件等背景信息
|
||||
|
||||
这意味着:
|
||||
|
||||
- 模型现在可以做“基于现有证据的摘要与归纳”
|
||||
- 但还不能可靠地做“跨维度因果研判”
|
||||
|
||||
### 2. 维度之间联动还弱
|
||||
|
||||
目前不同模块之间更多是“并列展示”,还不是“强关联分析”:
|
||||
|
||||
- 系统告警和 BGP 事件还没有统一事件模型
|
||||
- collector bias 与真实区域热度还没有完全剥离
|
||||
- datasource health 与 BGP 风险、业务影响之间还没有稳定映射
|
||||
|
||||
这意味着:
|
||||
|
||||
- 当前更适合做 `brief / overview / operator notes`
|
||||
- 还不适合过度承诺“自动态势判断”
|
||||
|
||||
### 3. 结构化 assessment 还未成为主输出
|
||||
|
||||
虽然已经有 BGP brief、系统告警 brief、态势告警 brief,但目前主输出仍偏向:
|
||||
|
||||
- 文本摘要
|
||||
- facts/context 附带证据
|
||||
|
||||
后续真正要服务态势感知,需要更稳定的结构化输出,例如:
|
||||
|
||||
- summary
|
||||
- key risks
|
||||
- evidence
|
||||
- confidence
|
||||
- recommendations
|
||||
- missing data
|
||||
|
||||
## 当前基座已经具备的能力
|
||||
|
||||
### 1. AI 调用边界已经明确
|
||||
|
||||
- `aiprovider` 负责模型协议与 provider 兼容
|
||||
- `backend` 负责业务 API、证据整合和鉴权
|
||||
- `frontend` 负责页面入口与结果展示
|
||||
|
||||
### 2. 页面级 AI 入口已经开始成型
|
||||
|
||||
当前已经有或正在收口的入口:
|
||||
|
||||
- `Playground`
|
||||
- 用于链路验证与 provider 诊断
|
||||
- `BGP AI 简报`
|
||||
- 用于 BGP 事实摘要和区域风险归纳
|
||||
- `Alerts`
|
||||
- 用于系统告警、BGP 告警、态势告警三类入口
|
||||
|
||||
### 3. 证据优先的方向已经建立
|
||||
|
||||
已经不再只依赖人工在 Playground 中手填 prompt,系统开始具备:
|
||||
|
||||
- 从真实业务数据生成事实输入
|
||||
- 保存 facts/context 快照
|
||||
- 回看 AI 输出时同时回看证据
|
||||
|
||||
这一步非常关键,因为它决定后面能否从“玩具 demo”走向“有运维价值的系统”。
|
||||
|
||||
## 近期收尾建议
|
||||
|
||||
这些事情都属于“底座收口”,值得做,但不应该再继续重产品包装。
|
||||
|
||||
### 1. 统一 Alerts 页面
|
||||
|
||||
已采用:
|
||||
|
||||
- 一个 `Alerts` 页面
|
||||
- 三个 tab:
|
||||
- `系统告警`
|
||||
- `BGP 告警`
|
||||
- `态势告警`
|
||||
|
||||
收尾重点:
|
||||
|
||||
- 保持 tab 的文案、摘要卡和 AI 简报交互一致
|
||||
- 不额外扩展成多个独立二级页面
|
||||
|
||||
### 2. 保持 Playground 为测试台
|
||||
|
||||
原则:
|
||||
|
||||
- Playground 只承担链路验证、provider 状态诊断、请求结果观察
|
||||
- 不继续堆“万能业务分析器”式交互
|
||||
|
||||
### 3. 把 brief 能力当服务能力而不是页面特效
|
||||
|
||||
页面现在能看到按钮和结果,这很好,但更重要的是:
|
||||
|
||||
- 后端接口稳定
|
||||
- facts/context 可追踪
|
||||
- 输出结构后续可升级
|
||||
|
||||
### 4. 导航结构先收口,不继续平铺一级菜单
|
||||
|
||||
随着后续能力扩展,系统很可能继续新增:
|
||||
|
||||
- 海缆
|
||||
- 算力中心
|
||||
- 战争信息
|
||||
- 电商分析
|
||||
- 其他专题观测页
|
||||
|
||||
如果继续把这些入口全部平铺在左侧一级菜单中,会带来两个问题:
|
||||
|
||||
- 一级菜单过长,用户难以判断先进入哪个上下文
|
||||
- `观测页 / 告警页 / 研判页 / 运维页` 的职责边界会被混在一起
|
||||
|
||||
因此近期应明确采用分组导航,而不是继续扩展平铺菜单。
|
||||
|
||||
推荐的导航分组如下:
|
||||
|
||||
- `总览`
|
||||
- 仪表盘
|
||||
- Earth
|
||||
- `专题观测`
|
||||
- BGP 观测
|
||||
- 采集数据
|
||||
- 后续可扩展:海缆、算力中心、战争信息、电商分析
|
||||
- `告警与研判`
|
||||
- Alerts
|
||||
- `运维与配置`
|
||||
- 数据源
|
||||
- AI Playground
|
||||
- 用户管理
|
||||
- 系统配置
|
||||
|
||||
这套结构的含义是:
|
||||
|
||||
- `专题观测` 页面负责看某个维度本身
|
||||
- `Alerts` 负责跨模块风险与值班工作台
|
||||
- `Playground` 保持为测试台,不挤占业务导航语义
|
||||
|
||||
短期收尾时,应优先重组现有入口,而不是继续增加新的一级菜单。
|
||||
|
||||
## 后续路线
|
||||
|
||||
## Phase 1:服务底座稳固
|
||||
|
||||
目标:
|
||||
|
||||
- 不追求“更炫的 AI 页面”
|
||||
- 先把当前接口、证据、存储和页面入口收稳
|
||||
|
||||
工作项:
|
||||
|
||||
- 统一页面级 AI 入口模式
|
||||
- 统一 brief response schema
|
||||
- 保证 facts/context 在前后端都可回看
|
||||
- 继续清理 mock 和临时分支逻辑
|
||||
|
||||
完成标准:
|
||||
|
||||
- 每个 AI 入口都是真实链路
|
||||
- 每个 AI 结果都能追溯到证据输入
|
||||
|
||||
## Phase 2:Evidence-first Assessment
|
||||
|
||||
目标:
|
||||
|
||||
- 从“文本摘要”升级成“结构化 assessment”
|
||||
|
||||
工作项:
|
||||
|
||||
- 为 brief/assessment 定义统一 schema
|
||||
- 固化:
|
||||
- summary
|
||||
- key_risks
|
||||
- evidence
|
||||
- confidence
|
||||
- recommendations
|
||||
- missing_data
|
||||
- 页面以结构化区块展示,而不只是大段文本
|
||||
|
||||
完成标准:
|
||||
|
||||
- AI 输出可持久化、可比较、可审计
|
||||
|
||||
## Phase 3:多维证据接入
|
||||
|
||||
目标:
|
||||
|
||||
- 让“态势感知”真正拥有更多维度,而不是只靠 BGP 与系统告警
|
||||
|
||||
优先接入方向:
|
||||
|
||||
- datasource health findings
|
||||
- 流量或业务指标
|
||||
- 区域/资产/链路映射
|
||||
- 外部事件与公告
|
||||
- 业务垂直数据,例如电商分析相关指标
|
||||
|
||||
完成标准:
|
||||
|
||||
- AI 能基于多个维度做交叉说明
|
||||
- 不再只围绕单一模块自说自话
|
||||
|
||||
## Phase 4:Correlation Layer
|
||||
|
||||
目标:
|
||||
|
||||
- 不同来源的信号不再只是并列,而是形成统一的事件关联
|
||||
|
||||
工作项:
|
||||
|
||||
- 统一 signal/finding 模型
|
||||
- 跨模块事件聚合
|
||||
- 证据来源权重
|
||||
- collector bias 与真实热度分离
|
||||
|
||||
完成标准:
|
||||
|
||||
- 系统能回答“这些异常是不是同一件事”
|
||||
- 系统能回答“哪些结论只是观测偏差”
|
||||
|
||||
## Phase 5:Agent-assisted Situational Awareness
|
||||
|
||||
目标:
|
||||
|
||||
- 在证据足够的前提下,再让 agent 负责更复杂的推理与建议
|
||||
|
||||
工作项:
|
||||
|
||||
- 复用现有 agent runtime 规划
|
||||
- 引入 web search / docs fetch / repair proposal 等能力
|
||||
- 但始终坚持:
|
||||
- evidence first
|
||||
- proposal before action
|
||||
- no silent mutation of defaults
|
||||
|
||||
完成标准:
|
||||
|
||||
- agent 成为证据驱动的分析层
|
||||
- 而不是一个“万能猜测层”
|
||||
|
||||
## 设计原则
|
||||
|
||||
### 1. 先底座,后产品化
|
||||
|
||||
先把服务链路和证据模型做好,再做更大的页面表达。
|
||||
|
||||
### 2. 先证据,后判断
|
||||
|
||||
事实输入应先稳定,再让模型做归纳。
|
||||
|
||||
### 3. 先专用 brief,后统一态势层
|
||||
|
||||
先让各业务页有各自可信的 AI 入口,再考虑统一态势页。
|
||||
|
||||
### 4. 先 proposal,后自动动作
|
||||
|
||||
涉及修复、覆盖、写配置、调任务的动作,都应经过 proposal 和审计。
|
||||
|
||||
## 当前建议结论
|
||||
|
||||
对现在这个项目,最合理的定位是:
|
||||
|
||||
- `Playground` 是测试台
|
||||
- `BGP / Alerts` 是第一批业务 AI 入口
|
||||
- `aiprovider + backend AI facade + evidence snapshots` 是核心服务底座
|
||||
|
||||
现阶段不需要追求“已经具备完整态势感知能力”。
|
||||
|
||||
现阶段真正的成功标准是:
|
||||
|
||||
- 这套底座可用
|
||||
- 可回看
|
||||
- 可扩展
|
||||
- 不自欺欺人
|
||||
@@ -1,347 +0,0 @@
|
||||
# System Service Control
|
||||
|
||||
This document defines the fixed mapping between admin control-plane actions and
|
||||
the existing `planet.sh` service-management commands.
|
||||
|
||||
The goal is to reuse the current operational script semantics without exposing
|
||||
arbitrary shell execution to the frontend or API callers.
|
||||
|
||||
## Scope
|
||||
|
||||
- This mapping is for admin-side operational controls only.
|
||||
- The control plane must submit a fixed action name, not a raw shell command.
|
||||
- The backend is responsible for translating an allowed action into a fixed
|
||||
`planet.sh` invocation.
|
||||
|
||||
## Design Rules
|
||||
|
||||
- Only whitelist actions may be executed.
|
||||
- The frontend must never send arbitrary shell strings.
|
||||
- The backend must build command arguments from a fixed mapping table.
|
||||
- High-risk actions should be restricted to `super_admin`.
|
||||
- Prefer partial restarts over full-stack restarts when UI continuity matters.
|
||||
|
||||
## Action Mapping
|
||||
|
||||
| Action name | Intended use | `planet.sh` command | Notes |
|
||||
| --- | --- | --- | --- |
|
||||
| `restart-backend` | Restart backend API only | `./planet.sh restart -b` | Recommended first implementation for UI-triggered restart flows. |
|
||||
| `restart-database` | Restart PostgreSQL and Redis containers | `./planet.sh restart -d` | Useful when database/cache services need a controlled bounce without restarting the UI. |
|
||||
| `restart-system` | Restart the whole application stack | `./planet.sh restart` | Frontend continuity breaks briefly; UI should switch to guided recovery mode. |
|
||||
| `restart-frontend` | Restart frontend dev server only | `./planet.sh restart -f` | Use with caution; UI continuity is weaker than backend-only restart. |
|
||||
| `restart-backend-port` | Restart backend on a specific port | `./planet.sh restart -b <port>` | Port must be backend-validated before execution. |
|
||||
| `restart-frontend-port` | Restart frontend on a specific port | `./planet.sh restart -f <port>` | Port must be backend-validated before execution. |
|
||||
| `health-check` | Read current service health | `./planet.sh health` | Safe read-only operational action. |
|
||||
| `show-logs-backend` | Inspect backend logs | `./planet.sh log -b` | Best used for CLI/operator tooling, not normal Web UI streaming. |
|
||||
| `show-logs-frontend` | Inspect frontend logs | `./planet.sh log -f` | Best used for CLI/operator tooling, not normal Web UI streaming. |
|
||||
|
||||
## Not Exposed In UI By Default
|
||||
|
||||
The following existing script capabilities should not be exposed directly in the
|
||||
Web UI unless there is an explicit product need and an additional safety review:
|
||||
|
||||
- `./planet.sh restart`
|
||||
- `./planet.sh start`
|
||||
- `./planet.sh stop`
|
||||
- `./planet.sh createuser`
|
||||
- any future raw shell passthrough
|
||||
|
||||
Reason:
|
||||
|
||||
- full restart can break the current control session;
|
||||
- stop/start have larger blast radius;
|
||||
- user creation is not a service-control operation;
|
||||
- raw shell passthrough creates unnecessary privilege risk.
|
||||
|
||||
## Recommended First-Phase UI Contract
|
||||
|
||||
### Frontend action payload
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "restart-backend"
|
||||
}
|
||||
```
|
||||
|
||||
### Backend command resolution
|
||||
|
||||
```text
|
||||
restart-backend -> ["./planet.sh", "restart", "-b"]
|
||||
restart-database -> ["./planet.sh", "restart", "-d"]
|
||||
restart-system -> ["./planet.sh", "restart"]
|
||||
restart-frontend -> ["./planet.sh", "restart", "-f"]
|
||||
health-check -> ["./planet.sh", "health"]
|
||||
```
|
||||
|
||||
## API Draft
|
||||
|
||||
### Primary Endpoint
|
||||
|
||||
- `POST /api/v1/system/restart-tasks`
|
||||
|
||||
Purpose:
|
||||
|
||||
- create a controlled restart task;
|
||||
- resolve a whitelist action into a fixed `planet.sh` command;
|
||||
- hand execution off to an external runner or detached subprocess.
|
||||
|
||||
### Request Body
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "restart-backend"
|
||||
}
|
||||
```
|
||||
|
||||
Optional future shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "restart-backend-port",
|
||||
"port": 8000
|
||||
}
|
||||
```
|
||||
|
||||
### Response
|
||||
|
||||
```json
|
||||
{
|
||||
"task_id": "restart_20260331_153000_ab12cd",
|
||||
"action": "restart-backend",
|
||||
"status": "queued",
|
||||
"stage": "accepted",
|
||||
"message": "Restart task accepted"
|
||||
}
|
||||
```
|
||||
|
||||
### Task Query Endpoint
|
||||
|
||||
- `GET /api/v1/system/restart-tasks/{task_id}`
|
||||
|
||||
Response shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"task_id": "restart_20260331_153000_ab12cd",
|
||||
"action": "restart-backend",
|
||||
"status": "queued",
|
||||
"stage": "accepted",
|
||||
"message": "Waiting for execution",
|
||||
"requested_by": {
|
||||
"id": 1,
|
||||
"username": "admin"
|
||||
},
|
||||
"created_at": "2026-03-31T15:30:00+08:00",
|
||||
"updated_at": "2026-03-31T15:30:02+08:00"
|
||||
}
|
||||
```
|
||||
|
||||
### Optional Log Endpoint
|
||||
|
||||
- `GET /api/v1/system/restart-tasks/{task_id}/logs`
|
||||
|
||||
Suggested response:
|
||||
|
||||
```json
|
||||
{
|
||||
"task_id": "restart_20260331_153000_ab12cd",
|
||||
"lines": [
|
||||
"accepted restart-backend request",
|
||||
"spawning restart command",
|
||||
"waiting for backend shutdown",
|
||||
"waiting for backend health recovery"
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
This log endpoint is optional for phase one. The first version can work with
|
||||
task state plus `/health` polling alone.
|
||||
|
||||
## Task State Model
|
||||
|
||||
### Status
|
||||
|
||||
- `queued`
|
||||
- `running`
|
||||
- `succeeded`
|
||||
- `failed`
|
||||
- `timeout`
|
||||
|
||||
### Stage
|
||||
|
||||
- `accepted`
|
||||
- `spawning`
|
||||
- `stopping`
|
||||
- `starting`
|
||||
- `waiting_for_health`
|
||||
- `healthy`
|
||||
- `failed`
|
||||
|
||||
### Interpretation
|
||||
|
||||
- `status` is the high-level terminal or non-terminal state.
|
||||
- `stage` is the operator-facing execution phase for the UI.
|
||||
- `message` is the short human-readable line shown in the modal or full-screen
|
||||
overlay.
|
||||
|
||||
## Permission Model
|
||||
|
||||
- `restart-backend` should require `super_admin`.
|
||||
- Permission checks should follow the same role pattern already used in
|
||||
[users.py](/home/ray/dev/linkong/planet/backend/app/api/v1/users.py).
|
||||
- Frontend visibility may hide controls for non-`super_admin`, but backend must
|
||||
still enforce authorization.
|
||||
|
||||
## Storage Model
|
||||
|
||||
Recommended first implementation:
|
||||
|
||||
- store restart task state in Redis;
|
||||
- keep task lifetime short;
|
||||
- keep recent logs as a bounded list.
|
||||
|
||||
Suggested keys:
|
||||
|
||||
- `system:restart_task:{task_id}`
|
||||
- `system:restart_task:{task_id}:logs`
|
||||
|
||||
Suggested stored fields:
|
||||
|
||||
- `task_id`
|
||||
- `action`
|
||||
- `status`
|
||||
- `stage`
|
||||
- `message`
|
||||
- `requested_by_id`
|
||||
- `requested_by_username`
|
||||
- `created_at`
|
||||
- `updated_at`
|
||||
|
||||
## Execution Model
|
||||
|
||||
The request-handling API process should not depend on itself surviving long
|
||||
enough to stream the whole restart output.
|
||||
|
||||
Recommended execution flow:
|
||||
|
||||
1. validate caller and action
|
||||
2. create task state in Redis
|
||||
3. resolve action to fixed `planet.sh` argv
|
||||
4. spawn detached executor
|
||||
5. return `task_id`
|
||||
6. executor updates task state while restart is in progress
|
||||
7. frontend polls health and/or task state until recovery
|
||||
|
||||
Recommended command resolution examples:
|
||||
|
||||
```text
|
||||
restart-backend -> ["./planet.sh", "restart", "-b"]
|
||||
restart-frontend -> ["./planet.sh", "restart", "-f"]
|
||||
restart-backend-port -> ["./planet.sh", "restart", "-b", "<port>"]
|
||||
health-check -> ["./planet.sh", "health"]
|
||||
```
|
||||
|
||||
## Frontend Polling Flow
|
||||
|
||||
Recommended first-phase UX:
|
||||
|
||||
1. user clicks `重启后端`
|
||||
2. confirmation modal explains temporary unavailability
|
||||
3. frontend calls `POST /api/v1/system/restart-tasks`
|
||||
4. UI enters blocking restart state
|
||||
5. frontend polls `/health` every `1-2s`
|
||||
6. temporary request failures are treated as expected
|
||||
7. after `2-3` consecutive successful health checks, frontend reloads page
|
||||
|
||||
Optional richer polling:
|
||||
|
||||
1. poll task status endpoint while backend is still reachable
|
||||
2. switch to `/health` recovery polling after disconnect begins
|
||||
3. refresh page after health recovery
|
||||
|
||||
## Frontend State Machine
|
||||
|
||||
- `idle`
|
||||
- `confirming`
|
||||
- `submitting`
|
||||
- `waiting_for_shutdown`
|
||||
- `waiting_for_recovery`
|
||||
- `recovered`
|
||||
- `failed`
|
||||
- `timeout`
|
||||
|
||||
Suggested UI messages:
|
||||
|
||||
- `已发送重启指令`
|
||||
- `正在停止后端服务`
|
||||
- `正在等待服务恢复`
|
||||
- `服务已恢复,正在刷新页面`
|
||||
- `恢复超时,请手动检查服务状态`
|
||||
|
||||
## Phase-One Recommendation
|
||||
|
||||
Implement only the following in phase one:
|
||||
|
||||
- `restart-backend`
|
||||
- `super_admin` permission gate
|
||||
- task creation endpoint
|
||||
- Redis-backed task state
|
||||
- frontend confirmation modal
|
||||
- frontend `/health` polling
|
||||
- automatic page reload after recovery
|
||||
|
||||
Do not implement in phase one:
|
||||
|
||||
- full `./planet.sh restart`
|
||||
- raw shell command passthrough
|
||||
- arbitrary service control
|
||||
- full terminal stdout streaming
|
||||
- multi-action concurrent restart queueing
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
### Backend
|
||||
|
||||
1. add a dedicated system-control API module under `backend/app/api/v1/`
|
||||
2. add a whitelist-based action resolver for `planet.sh`
|
||||
3. store restart task state in Redis
|
||||
4. add detached restart-runner script execution
|
||||
5. expose:
|
||||
- `POST /api/v1/system/restart-tasks`
|
||||
- `GET /api/v1/system/restart-tasks/{task_id}`
|
||||
- optional task log endpoint
|
||||
6. enforce `super_admin` permission on all restart-task endpoints
|
||||
|
||||
### Frontend
|
||||
|
||||
1. add a `重启后端` control on the dashboard for `super_admin`
|
||||
2. show a confirmation modal before dispatch
|
||||
3. after submission, switch modal into blocking restart state
|
||||
4. poll `/health` until backend recovery is confirmed
|
||||
5. auto-refresh page after consecutive successful health checks
|
||||
6. show short stage-oriented logs instead of raw terminal streaming
|
||||
|
||||
### Operational Notes
|
||||
|
||||
1. phase one should target backend-only restart
|
||||
2. frontend restart should remain out of scope initially
|
||||
3. command execution must always originate from repository root
|
||||
4. only fixed action names may cross the API boundary
|
||||
|
||||
## Validation Requirements
|
||||
|
||||
- Reject any action not present in the whitelist.
|
||||
- If a port-bearing action is added, validate the port as an integer in
|
||||
`1..65535`.
|
||||
- Resolve commands from the repository root so `planet.sh` runs with a stable
|
||||
working directory.
|
||||
- Record the requested action, operator identity, execution start time, and
|
||||
result.
|
||||
|
||||
## Implementation Guidance
|
||||
|
||||
- For UI-triggered restart flows, prefer `restart-backend` first.
|
||||
- Do not rely on the current API request process to stream full restart output
|
||||
after it triggers its own restart.
|
||||
- Use a task record plus polling/health-check recovery flow instead of raw
|
||||
terminal streaming as the primary UX.
|
||||
@@ -1,48 +0,0 @@
|
||||
# 系统配置中心开发计划
|
||||
|
||||
## 目标
|
||||
|
||||
将当前仅保存于内存中的“系统配置”页面升级为真正可用的配置中心,优先服务以下两类能力:
|
||||
|
||||
1. 系统级配置持久化
|
||||
2. 采集调度配置管理
|
||||
|
||||
## 第一阶段范围
|
||||
|
||||
### 1. 系统配置持久化
|
||||
|
||||
- 新增 `system_settings` 表,用于保存分类配置
|
||||
- 将系统、通知、安全配置从进程内存迁移到数据库
|
||||
- 提供统一读取接口,页面刷新和服务重启后保持不丢失
|
||||
|
||||
### 2. 采集调度配置接入真实数据源
|
||||
|
||||
- 统一内置采集器默认定义
|
||||
- 启动时自动初始化 `data_sources` 表
|
||||
- 配置页允许修改:
|
||||
- 是否启用
|
||||
- 采集频率(分钟)
|
||||
- 优先级
|
||||
- 修改后实时同步到调度器
|
||||
|
||||
### 3. 前端配置页重构
|
||||
|
||||
- 将当前通用模板页调整为项目专用配置中心
|
||||
- 增加“采集调度”Tab
|
||||
- 保留“系统显示 / 通知 / 安全”三类配置
|
||||
- 将设置页正式接入主路由
|
||||
|
||||
## 非本阶段内容
|
||||
|
||||
- 邮件发送能力本身
|
||||
- 配置审计历史
|
||||
- 敏感凭证加密管理
|
||||
- 多租户或按角色细粒度配置
|
||||
|
||||
## 验收标准
|
||||
|
||||
- 设置项修改后重启服务仍然存在
|
||||
- 配置页可以查看并修改所有内置采集器的启停与采集频率
|
||||
- 调整采集频率后,调度器任务随之更新
|
||||
- `/settings` 页面可从主导航进入并正常工作
|
||||
|
||||
29
docs/technical/README.md
Normal file
29
docs/technical/README.md
Normal file
@@ -0,0 +1,29 @@
|
||||
# Technical Docs
|
||||
|
||||
这里放“当前实现和当前结构”的文档,重点回答:
|
||||
|
||||
- 现在代码是怎么组织的
|
||||
- 当前入口在哪
|
||||
- 状态和组件如何工作
|
||||
- 后续改动应该沿着哪条实现边界继续走
|
||||
|
||||
适合放入这里的内容:
|
||||
|
||||
- 前端上下文
|
||||
- Earth 前端结构
|
||||
- Earth 卫星 footprint 策略
|
||||
- Earth 渲染图层顺序
|
||||
- Earth 图层样式属性索引
|
||||
- 后端运行控制
|
||||
- collector 现状
|
||||
- 采集格式约定
|
||||
|
||||
不适合放入这里的内容:
|
||||
|
||||
- 尚未完成的 roadmap
|
||||
- 未来迭代方案
|
||||
- 大范围重构计划
|
||||
|
||||
这些应放入:
|
||||
|
||||
- [docs/plans/README.md](/home/ray/dev/linkong/planet/docs/plans/README.md)
|
||||
@@ -187,7 +187,7 @@ Current reality:
|
||||
- that is expected, because incidents are aggregated and de-noised
|
||||
- but incident-first rendering makes the Earth view look too quiet unless there is another always-available activity layer
|
||||
|
||||
Implementation detail for the recommended `activity layer` is expanded in [bgp-region-aggregation-plan.md](/home/ray/dev/linkong/planet/docs/bgp-region-aggregation-plan.md).
|
||||
Implementation detail for the recommended `activity layer` is expanded in [bgp-region-aggregation-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-bgp-region-aggregation-plan.md).
|
||||
|
||||
So the immediate next milestone is:
|
||||
|
||||
381
docs/technical/earth-frontend-context.md
Normal file
381
docs/technical/earth-frontend-context.md
Normal file
@@ -0,0 +1,381 @@
|
||||
# Earth Frontend Context
|
||||
|
||||
本文件描述当前 Earth 大屏前端的真实结构,重点是帮助后续继续改 HUD、图层、媒体面板、真实地形、BGP 可视化时,不再重复踩结构和状态同步上的坑。
|
||||
|
||||
相关规则建议一起参考:
|
||||
|
||||
- [rules.md](/home/ray/dev/linkong/planet/rules.md)
|
||||
- [frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md)
|
||||
|
||||
## 当前目标
|
||||
|
||||
Earth 前端不是普通管理页,它是独立的大屏展示前端。当前产品目标是:
|
||||
|
||||
- 维持地球视图的空间感和可读性
|
||||
- 让 HUD、图层、媒体面板、BGP、卫星、海缆等保持统一交互
|
||||
- 把加载中、已启用、已隐藏、锁定中这类状态做清楚
|
||||
|
||||
## 当前入口
|
||||
|
||||
React 路由入口:
|
||||
|
||||
- [Earth.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Earth/Earth.tsx)
|
||||
|
||||
当前做法很简单:
|
||||
|
||||
- React 页面只负责提供一个全屏 `iframe`
|
||||
- 真正的 Earth 应用运行在:
|
||||
- [index.html](/home/ray/dev/linkong/planet/frontend/public/earth/index.html)
|
||||
|
||||
所以 Earth 前端本质上是 `public/earth` 下的一套独立静态应用。
|
||||
|
||||
## 当前文件分层
|
||||
|
||||
### 1. 页面入口与结构
|
||||
|
||||
- [index.html](/home/ray/dev/linkong/planet/frontend/public/earth/index.html)
|
||||
|
||||
职责:
|
||||
|
||||
- HUD 基础 DOM
|
||||
- 图层面板
|
||||
- 媒体面板
|
||||
- 工具栏
|
||||
- 设置弹窗
|
||||
- 兼容旧元素 id
|
||||
|
||||
### 2. 主运行时
|
||||
|
||||
- [main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js)
|
||||
|
||||
职责:
|
||||
|
||||
- 地球初始化
|
||||
- Three.js 场景组装
|
||||
- 数据加载与刷新
|
||||
- 各图层集成
|
||||
- Earth 级别状态同步
|
||||
|
||||
### 3. 地球控制层
|
||||
|
||||
- [controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js)
|
||||
|
||||
职责:
|
||||
|
||||
- 工具栏交互
|
||||
- 图层面板交互
|
||||
- 旋转/缩放/布局
|
||||
- HUD 面板拖拽
|
||||
- 图层开关状态机
|
||||
- Earth 设置读取、持久化与重置
|
||||
|
||||
这份文件是 Earth 前端当前最核心的 UI 控制入口。
|
||||
|
||||
### 4. UI 与状态消息
|
||||
|
||||
- [ui.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/ui.js)
|
||||
|
||||
职责:
|
||||
|
||||
- loading 面板
|
||||
- status message
|
||||
- tooltip / error / 清理逻辑
|
||||
|
||||
### 5. 地球与地形
|
||||
|
||||
- [earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js)
|
||||
- [terrain.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/terrain.js)
|
||||
|
||||
职责:
|
||||
|
||||
- 地球球体、云层、大气
|
||||
- 真实地形 mesh
|
||||
- terrain tile 拉取、解码、位移、着色
|
||||
|
||||
### 6. 图层模块
|
||||
|
||||
- [satellites.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/satellites.js)
|
||||
- [cables.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/cables.js)
|
||||
- [bgp.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp.js)
|
||||
- [bgp-cruise-adapter.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp-cruise-adapter.js)
|
||||
- [news.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/news.js)
|
||||
- [tv.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/tv.js)
|
||||
- [layer-startup-tasks.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/layer-startup-tasks.js)
|
||||
- [cruise-sequencer.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/cruise-sequencer.js)
|
||||
- [callout-connector.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/callout-connector.js)
|
||||
|
||||
职责:
|
||||
|
||||
- 各自的数据层
|
||||
- 开关行为
|
||||
- 面板内容
|
||||
- hover/lock/selection 语义
|
||||
|
||||
其中 Earth 启动加载链现在也拆成了两层:
|
||||
|
||||
- `controls.js`
|
||||
- 提供图层注册表与启动元信息
|
||||
- `layer-startup-tasks.js`
|
||||
- 提供图层启动任务注册表
|
||||
- 通过 `registerLayerStartupTask(id, taskFactory)` 扩展启动任务
|
||||
- `main.js`
|
||||
- 只负责读取排序后的启动图层,再按映射执行队列
|
||||
|
||||
其中巡航模式现在已经拆成两层:
|
||||
|
||||
- `cruise-sequencer.js`
|
||||
- 负责目标队列顺序、停留时长、切换节奏、打断与恢复
|
||||
- `callout-connector.js`
|
||||
- 负责卡片连线 SVG、路径计算与绘制动画
|
||||
- `bgp-cruise-adapter.js`
|
||||
- 负责 BGP 巡航展示适配:目标排序、卡片落点、连线路径、focus/overlay/info-card 时序
|
||||
|
||||
当前 BGP 巡航只是这套能力的一个调用方,不应再把“按队列巡航”和“BGP 事件展示”混写在同一个状态机里。
|
||||
|
||||
## 当前样式分层
|
||||
|
||||
Earth 的 CSS 不是一份大样式表,而是分层管理:
|
||||
|
||||
- [base.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/base.css)
|
||||
- [hud.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/hud.css)
|
||||
- [toolbar.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/toolbar.css)
|
||||
- [layer-panel.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/layer-panel.css)
|
||||
- [info-panel.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/info-panel.css)
|
||||
- [legend.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/legend.css)
|
||||
- [earth-stats.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/earth-stats.css)
|
||||
- [coordinates-display.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/coordinates-display.css)
|
||||
- [tv-panel.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/tv-panel.css)
|
||||
|
||||
当前建议:
|
||||
|
||||
- 通用 HUD 壳层写进 `hud.css`
|
||||
- 单一面板特性写进各自子文件
|
||||
- 不要把业务状态样式再散回 `index.html`
|
||||
|
||||
## 当前图层开关状态语义
|
||||
|
||||
Earth 图层按钮现在不应再只有“开/关”两态,而应支持:
|
||||
|
||||
- `inactive`
|
||||
- `active`
|
||||
- `loading`
|
||||
|
||||
当前入口在:
|
||||
|
||||
- [controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js)
|
||||
- [layer-button-state.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/layer-button-state.js)
|
||||
|
||||
关键函数:
|
||||
|
||||
- `updateLayerButtonState(button, isActive)`
|
||||
- `setLayerButtonState(button, options)`
|
||||
|
||||
`setLayerButtonState` 负责:
|
||||
|
||||
- `loading` 样式
|
||||
- `aria-busy`
|
||||
- 按钮禁用
|
||||
- tooltip 更新
|
||||
- 绑定状态文本更新
|
||||
- 可选同步 `active`
|
||||
|
||||
因此后续如果别的图层也需要异步启用,应该直接走这套状态机,而不是再手写一套临时 loading class。
|
||||
|
||||
另外,Earth 图层控制现在已经收成“注册表驱动”:
|
||||
|
||||
- 图层元数据
|
||||
- `id`
|
||||
- `icon`
|
||||
- `label`
|
||||
- `meta`
|
||||
- `buttonId`
|
||||
- `persist`
|
||||
- `startupPriority`
|
||||
- `startupMode`
|
||||
- `startupLabel`
|
||||
- `startupMessage`
|
||||
- 图层行为
|
||||
- `getVisible()`
|
||||
- `setVisible(next, options)`
|
||||
|
||||
当前入口仍在 [controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js)。
|
||||
|
||||
这意味着后续新增图层时,优先应补一条图层注册定义,而不是同时去改:
|
||||
|
||||
- 图层面板 HTML
|
||||
- 持久化快照
|
||||
- 初始化恢复
|
||||
- click 绑定
|
||||
|
||||
这四处现在都应该由注册表派生。
|
||||
|
||||
其中:
|
||||
|
||||
- `startupPriority`
|
||||
- 描述图层参与启动加载时的顺序
|
||||
- `startupMode`
|
||||
- `visible`
|
||||
- 仅当前图层处于启用/可见状态时,才加入启动加载队列
|
||||
- `preload`
|
||||
- 即使当前图层未显示,也会参与启动预加载
|
||||
|
||||
当前 `main.js` 会通过注册表读取排序后的启动图层列表,再动态拼装启动加载队列,而不是手写一串固定步骤。像 BGP 这类需要尽早准备数据、但不一定默认显示的图层,应该优先走 `startupMode: "preload"`,而不是在启动流程里写隐式特判。
|
||||
|
||||
此外,启动阶段给用户看的提示文案也应尽量从注册表派生:
|
||||
|
||||
- `startupLabel`
|
||||
- 用于描述当前启动任务的业务名称
|
||||
- `startupMessage`
|
||||
- 用于描述启动中的提示文案
|
||||
- 可以是字符串
|
||||
- 也可以是对象,用于像海缆这种“准备阶段 / 主加载阶段”两段式文案
|
||||
|
||||
这样后续新增会参与启动加载的图层时,顺序、模式和提示文案都在同一处定义,不需要再去 `main.js` 里补第二套常量。
|
||||
|
||||
### `data-status-target`
|
||||
|
||||
图层按钮可以通过:
|
||||
|
||||
- `data-status-target`
|
||||
|
||||
指向一个状态文本节点。当前 terrain 已接入:
|
||||
|
||||
- 按钮:`#toggle-terrain`
|
||||
- 状态节点:`#terrain-status`
|
||||
|
||||
以后别的异步图层也可以沿用这套约定。
|
||||
|
||||
## 当前设置持久化
|
||||
|
||||
Earth 设置面板当前由 [controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js) 统一负责:
|
||||
|
||||
- 捕获默认值
|
||||
- 从 `localStorage` 读取上次设置
|
||||
- 初始化应用当前设置
|
||||
- 用户变更后即时持久化
|
||||
- 一键重置回默认值
|
||||
|
||||
当前持久化的范围是:
|
||||
|
||||
- 旋转模式
|
||||
- 地球默认大小(作为重置视角、缩放重置和巡航视图的默认 zoom 真源)
|
||||
- HUD 面板显示/隐藏
|
||||
- 图层控制开关:`地形 / 卫星 / 轨迹 / 海缆 / BGP`
|
||||
- 地形透明度
|
||||
|
||||
也就是说,Earth 设置不是一次性 UI 状态了,而是本地设备级偏好。后续如果再加入新的设置项,应优先接入同一条持久化链,而不是各自散着写 `localStorage`。
|
||||
|
||||
## 当前地形链路
|
||||
|
||||
真实地形首次启用会慢,原因不只是一个:
|
||||
|
||||
1. 需要拉取 Terrarium 瓦片
|
||||
2. 需要解码图片
|
||||
3. 需要按顶点采样高程
|
||||
4. 需要重新写入 geometry 和 color
|
||||
5. 需要重新计算法线与包围体
|
||||
|
||||
当前入口在:
|
||||
|
||||
- [terrain.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/terrain.js)
|
||||
|
||||
当前已经做了两层体验优化:
|
||||
|
||||
1. 图层开关 loading 状态持续可见
|
||||
2. 页面空闲时会预热 `ensureTerrainReady()`
|
||||
|
||||
也就是说,后续再继续优化 terrain 时,优先顺序应该是:
|
||||
|
||||
1. 先保证用户感知正确
|
||||
2. 再压缩首次等待
|
||||
3. 最后才做更激进的几何/瓦片优化
|
||||
|
||||
## 当前高频风险点
|
||||
|
||||
### 1. 视觉状态和业务状态不同步
|
||||
|
||||
Earth 里最常见的 bug 不是“没渲染”,而是:
|
||||
|
||||
- 图层关了,tooltip 还在
|
||||
- 锁定对象隐藏了,info card 还在
|
||||
- legend 没跟图层切换
|
||||
- loading 已结束,但按钮还像没开
|
||||
|
||||
后续改动必须优先检查状态同步。
|
||||
|
||||
### 2. HUD 布局问题先查结构,不要先打 CSS 补丁
|
||||
|
||||
Earth HUD 历史上反复出现:
|
||||
|
||||
- 面板只剩一条缝
|
||||
- markdown 被裁掉
|
||||
- tabs/iframe 被 `overflow: hidden` 吃掉
|
||||
|
||||
优先检查:
|
||||
|
||||
1. 谁负责高度
|
||||
2. 谁负责滚动
|
||||
3. 哪一层在裁剪
|
||||
|
||||
不要上来先加 `overflow: hidden` 或额外包装层。
|
||||
|
||||
### 3. Transitional path 必须收口
|
||||
|
||||
Earth 已经经历过多轮 HUD、toolbar、media panel 重构,所以最容易积累:
|
||||
|
||||
- 旧 helper
|
||||
- 旧 class
|
||||
- 旧 fallback 逻辑
|
||||
- 已废弃变体
|
||||
|
||||
每次大功能完成后,都要做一次 cleanup pass。
|
||||
|
||||
### 4. 巡航与业务事件不要再深度耦合
|
||||
|
||||
当前正确边界应该是:
|
||||
|
||||
- 通用巡航层只知道:
|
||||
- 当前目标
|
||||
- 队列顺序
|
||||
- 相机 focus
|
||||
- 停留 / 隐藏 / 切换
|
||||
- 业务模块只负责:
|
||||
- 提供目标队列
|
||||
- 提供 focus 坐标
|
||||
- 提供卡片内容
|
||||
- 提供高亮/图层副作用
|
||||
|
||||
如果以后再给海缆、卫星或新闻做巡航,不应复制一套新的 `main.js` 状态变量,而应复用:
|
||||
|
||||
- [cruise-sequencer.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/cruise-sequencer.js)
|
||||
- [callout-connector.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/callout-connector.js)
|
||||
- [bgp-cruise-adapter.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp-cruise-adapter.js) 这种业务适配层模式
|
||||
|
||||
## 当前推荐改动方式
|
||||
|
||||
如果后续继续改 Earth,建议按这个顺序:
|
||||
|
||||
1. 先确认改的是:
|
||||
- Three.js 渲染层
|
||||
- HUD 结构层
|
||||
- 图层状态层
|
||||
- 面板内容层
|
||||
2. 如果涉及图层按钮,优先接入统一状态机
|
||||
3. 如果涉及可见性切换,检查 tooltip / legend / info-card / lock 是否一起收口
|
||||
4. 如果涉及面板布局,先查结构再动 CSS
|
||||
|
||||
## 当前与控制台前端的边界
|
||||
|
||||
Earth 前端和控制台前端不是同一套 UI 系统:
|
||||
|
||||
- 控制台前端:React + Ant Design 工作台
|
||||
- Earth 前端:`public/earth` 原生 HUD + Three.js 展示面
|
||||
|
||||
因此:
|
||||
|
||||
- Earth 不应该直接复用 Ant Table / AppLayout 语义
|
||||
- 控制台也不应该照搬 Earth HUD 动画和玻璃层语言
|
||||
|
||||
控制台相关结构见:
|
||||
|
||||
- [admin-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/frontend-admin-frontend-context.md)
|
||||
235
docs/technical/earth-layer-style-reference.md
Normal file
235
docs/technical/earth-layer-style-reference.md
Normal file
@@ -0,0 +1,235 @@
|
||||
# Earth 图层样式属性索引
|
||||
|
||||
本文记录当前 Earth 前端各图层的材质、颜色、透明度、线宽、半径偏移和
|
||||
`renderOrder` 等样式属性。层级关系请配合
|
||||
[earth-render-layer-order.md](/home/ray/dev/linkong/planet/docs/technical/earth-render-layer-order.md)
|
||||
查看。
|
||||
|
||||
## 命名约定
|
||||
|
||||
| 类别 | 约定 | 示例 |
|
||||
| --- | --- | --- |
|
||||
| 全局配置对象 | `*_CONFIG` | `COUNTRY_BOUNDARY_CONFIG` |
|
||||
| 图层半径偏移 | `*AltitudeOffset` / `radiusOffset` | `lineAltitudeOffset`, `GRID_CONFIG.radiusOffset` |
|
||||
| 透明度 | `*Opacity` | `hoverLineOpacity` |
|
||||
| 渲染顺序 | `*RenderOrder` | `textureOverlayRenderOrder` |
|
||||
| 颜色 | `*Color`,十六进制数字或 CSS 色值 | `lineColor`, `colors.supercomputer` |
|
||||
| 线宽 | `lineWidth` / `*LineWidth` | `GRID_CONFIG.lineWidth` |
|
||||
|
||||
## Earth 基座与高清材质
|
||||
|
||||
| 正式名称 | 变量名 | 当前值 | 使用位置 / 说明 |
|
||||
| --- | --- | --- | --- |
|
||||
| Earth 基座半径 | `CONFIG.earthRadius` | `100` | `earth.js:createEarth()` |
|
||||
| Earth 基座颜色 | `EARTH_MATERIAL_CONFIG.color` | `0x010609` | `MeshPhongMaterial.color` |
|
||||
| Earth 基座 emissive | `EARTH_MATERIAL_CONFIG.emissive` | `0x010609` | `MeshPhongMaterial.emissive` |
|
||||
| Earth 基座 specular | `EARTH_MATERIAL_CONFIG.specular` | `0x1a2d45` | `MeshPhongMaterial.specular` |
|
||||
| Earth 基座 shininess | `EARTH_MATERIAL_CONFIG.shininess` | `12` | `MeshPhongMaterial.shininess` |
|
||||
| Earth 基座 opacity | `EARTH_MATERIAL_CONFIG.opacity` | `1` | `MeshPhongMaterial.opacity` |
|
||||
| 高清材质半径偏移 | `EARTH_MATERIAL_CONFIG.textureOverlayAltitudeOffset` | `0.1` | 独立高清材质球半径 |
|
||||
| 高清材质透明度 | `EARTH_MATERIAL_CONFIG.textureOverlayOpacity` | `0.88` | 高清材质 `MeshPhongMaterial.opacity` |
|
||||
| 高清材质 renderOrder | `EARTH_MATERIAL_CONFIG.textureOverlayRenderOrder` | `0.96` | `_earthTextureOverlay.renderOrder` |
|
||||
| 高清材质颜色乘色 | inline | `0xffffff` | `_earthTextureOverlayMaterial.color` |
|
||||
|
||||
## Earth 遮挡与昼夜
|
||||
|
||||
| 正式名称 | 变量名 | 当前值 | 使用位置 / 说明 |
|
||||
| --- | --- | --- | --- |
|
||||
| 遮挡球半径系数 | `EARTH_MATERIAL_CONFIG.occluderRadiusFactor` | `0.999` | 深度遮挡球半径 |
|
||||
| 遮挡球分段 | `EARTH_MATERIAL_CONFIG.occluderSegments` | `48` | 遮挡球几何分段 |
|
||||
| 遮挡球 renderOrder | inline | `-1` | `occluder.renderOrder` |
|
||||
| 昼夜太阳方向 | `EARTH_MATERIAL_CONFIG.dayNight.sunDirection` | `{ x: 1, y: 0.2, z: 0.4 }` | 自定义 day/night shader |
|
||||
| 夜侧最低亮度 | `EARTH_MATERIAL_CONFIG.dayNight.nightFloor` | `0.32` | shader uniform |
|
||||
| 日侧增强 | `EARTH_MATERIAL_CONFIG.dayNight.dayBoost` | `0.94` | shader uniform |
|
||||
| 暮光宽度 | `EARTH_MATERIAL_CONFIG.dayNight.twilightWidth` | `0.24` | shader uniform |
|
||||
| 暮光强度 | `EARTH_MATERIAL_CONFIG.dayNight.twilightIntensity` | `0.14` | shader uniform |
|
||||
| 暮光颜色 | `EARTH_MATERIAL_CONFIG.dayNight.twilightColor` | `0x4ea0ff` | shader uniform |
|
||||
| 夜侧 tint 颜色 | `EARTH_MATERIAL_CONFIG.dayNight.nightTintColor` | `0x0b1830` | shader uniform |
|
||||
| 夜侧 tint 强度 | `EARTH_MATERIAL_CONFIG.dayNight.nightTintIntensity` | `0.05` | shader uniform |
|
||||
|
||||
## 大气辉光与云图
|
||||
|
||||
| 正式名称 | 变量名 | 当前值 | 使用位置 / 说明 |
|
||||
| --- | --- | --- | --- |
|
||||
| 内层大气半径系数 | `EARTH_MATERIAL_CONFIG.atmosInnerRadiusFactor` | `1.01` | `atmosInnerGeo` |
|
||||
| 内层大气分段 | `EARTH_MATERIAL_CONFIG.atmosInnerSegments` | `64` | `atmosInnerGeo` |
|
||||
| 内层大气颜色 | `EARTH_MATERIAL_CONFIG.atmosInnerColor` | `[0.25, 0.62, 1.0]` | shader RGB |
|
||||
| 内层大气 rim power | `EARTH_MATERIAL_CONFIG.atmosInnerRimPower` | `3.2` | shader rim |
|
||||
| 内层大气强度 | `EARTH_MATERIAL_CONFIG.atmosInnerIntensity` | `0.18` | shader alpha multiplier |
|
||||
| 外层大气半径系数 | `EARTH_MATERIAL_CONFIG.atmosOuterRadiusFactor` | `1.016` | `atmosOuterGeo` |
|
||||
| 外层大气分段 | `EARTH_MATERIAL_CONFIG.atmosOuterSegments` | `48` | `atmosOuterGeo` |
|
||||
| 外层大气颜色 | `EARTH_MATERIAL_CONFIG.atmosOuterColor` | `[0.18, 0.45, 0.9]` | shader RGB |
|
||||
| 外层大气 rim power | `EARTH_MATERIAL_CONFIG.atmosOuterRimPower` | `5.0` | shader rim |
|
||||
| 外层大气强度 | `EARTH_MATERIAL_CONFIG.atmosOuterIntensity` | `0.02` | shader alpha multiplier |
|
||||
| 大气辉光 blending | inline | `THREE.AdditiveBlending` | `ShaderMaterial.blending` |
|
||||
| 大气辉光 renderOrder | inline | `1` | `atmosInner/Outer.renderOrder` |
|
||||
| 云图半径偏移 | `CLOUD_LAYER_CONFIG.radiusOffset` | `3` | 云层球半径 |
|
||||
| 云图分段 | `CLOUD_LAYER_CONFIG.widthSegments / heightSegments` | `64 / 64` | 云层球几何分段 |
|
||||
| 云图透明度 | `CLOUD_LAYER_CONFIG.opacity` | `0.15` | `MeshPhongMaterial.opacity` |
|
||||
| 云图贴图 | `CLOUD_LAYER_CONFIG.textureUrl` | `"./assets/earth_clouds_1024.png"` | 云层贴图 |
|
||||
| 云图 blending | inline | `THREE.AdditiveBlending` | `MeshPhongMaterial.blending` |
|
||||
|
||||
## 海陆基座与国界
|
||||
|
||||
| 正式名称 | 变量名 | 当前值 | 使用位置 / 说明 |
|
||||
| --- | --- | --- | --- |
|
||||
| 国界数据路径 | `COUNTRY_BOUNDARY_CONFIG.dataPath` | `"/earth/data/countries-admin0.min.geojson"` | GeoJSON 输入 |
|
||||
| 海洋填充色 | local `OCEAN_HEX` | `0x010609` | 海陆基座 canvas 背景 |
|
||||
| 陆地填充色 | `COUNTRY_BOUNDARY_CONFIG.landColor` | `0x080f1b` | 海陆基座 canvas 陆地 |
|
||||
| 海陆基座透明度 | `COUNTRY_BOUNDARY_CONFIG.landOpacity` | `1.0` | `MeshBasicMaterial.opacity` |
|
||||
| 海陆基座半径偏移 | `COUNTRY_BOUNDARY_CONFIG.landAltitudeOffset` | `0.08` | `country-land-ocean` 半径 |
|
||||
| 海陆基座 renderOrder | `COUNTRY_BOUNDARY_CONFIG.landRenderOrder` | `0.86` | `country-land-ocean.renderOrder` |
|
||||
| 海陆 mask 尺寸 | `landMaskWidth / landMaskHeight` | `2048 / 1024` | canvas / DataTexture 尺寸 |
|
||||
| 国界 tint 颜色 | `COUNTRY_BOUNDARY_CONFIG.tintColor` | `0x0b1830` | 高清材质关闭时 tint |
|
||||
| 国界 tint 半径偏移 | `COUNTRY_BOUNDARY_CONFIG.tintAltitudeOffset` | `0.04` | `country-tint` 半径 |
|
||||
| 国界 tint renderOrder | `COUNTRY_BOUNDARY_CONFIG.tintRenderOrder` | `0.2` | `country-tint.renderOrder` |
|
||||
| 国界线颜色 | `COUNTRY_BOUNDARY_CONFIG.lineColor` | `0x7fc7ff` | 普通国界线 |
|
||||
| 国界线透明度 | `COUNTRY_BOUNDARY_CONFIG.lineOpacity` | `0.58` | 普通国界线 opacity |
|
||||
| 国界线 hover 时压暗透明度 | `COUNTRY_BOUNDARY_CONFIG.dimmedLineOpacity` | `0.18` | hover 时普通国界线 opacity |
|
||||
| 国界线半径偏移 | `COUNTRY_BOUNDARY_CONFIG.lineAltitudeOffset` | `0.24` | 普通国界线半径 |
|
||||
| 国界线 renderOrder | `COUNTRY_BOUNDARY_CONFIG.lineRenderOrder` | `2.2` | 普通国界线层级 |
|
||||
| 国界 hover 颜色 | `COUNTRY_BOUNDARY_CONFIG.hoverLineColor` | `0xff3b1f` | 霓虹红橘 |
|
||||
| 国界 hover 透明度 | `COUNTRY_BOUNDARY_CONFIG.hoverLineOpacity` | `1.0` | hover 实线 opacity |
|
||||
| 国界 hover 半径偏移 | `COUNTRY_BOUNDARY_CONFIG.hoverAltitudeOffset` | `0.32` | hover 实线半径 |
|
||||
| 国界 hover renderOrder | `COUNTRY_BOUNDARY_CONFIG.hoverLineRenderOrder` | `2.3` | hover 实线层级 |
|
||||
| 国界 hover glow 透明度 | `COUNTRY_BOUNDARY_CONFIG.hoverGlowOpacity` | `0.38` | glow 线 opacity |
|
||||
| 国界 hover glow 线宽 | `COUNTRY_BOUNDARY_CONFIG.hoverGlowLineWidth` | `3` | glow `LineBasicMaterial.linewidth` |
|
||||
| 国界 hover glow 层级偏移 | `COUNTRY_BOUNDARY_CONFIG.hoverGlowRenderOrderOffset` | `0.01` | glow renderOrder = `2.29` |
|
||||
| 国界 hover glow 半径偏移 | `COUNTRY_BOUNDARY_CONFIG.hoverGlowRadiusOffset` | `0.04` | glow 半径 = hover 半径 + 0.04 |
|
||||
|
||||
## 真实地形
|
||||
|
||||
| 正式名称 | 变量名 | 当前值 | 使用位置 / 说明 |
|
||||
| --- | --- | --- | --- |
|
||||
| 地形 tile size | `TERRAIN_CONFIG.tileSize` | `256` | Terrarium tile 读取 |
|
||||
| 地形 base zoom | `TERRAIN_CONFIG.baseZoom` | `4` | 地形采样 zoom |
|
||||
| 地形几何分段 | `geometryWidthSegments / geometryHeightSegments` | `320 / 320` | 地形球几何 |
|
||||
| 地形基准半径偏移 | `TERRAIN_CONFIG.baseRadiusOffset` | `0.16` | 地形压过高清材质 |
|
||||
| 地形夸张系数 | `TERRAIN_CONFIG.exaggeration` | `34` | 海拔转世界单位 |
|
||||
| 地形陆地淡入高度 | `TERRAIN_CONFIG.landRevealFadeMeters` | `220` | 顶点 alpha |
|
||||
| 地形透明度 | `TERRAIN_CONFIG.opacity` | `0.62` | `MeshPhongMaterial.opacity` |
|
||||
| 地形颜色 | `TERRAIN_CONFIG.color` | `0x7f9d7f` | `MeshPhongMaterial.color` |
|
||||
| 地形 emissive | `TERRAIN_CONFIG.emissive` | `0x061008` | `MeshPhongMaterial.emissive` |
|
||||
| 地形 specular | `TERRAIN_CONFIG.specular` | `0x233126` | `MeshPhongMaterial.specular` |
|
||||
| 地形 shininess | `TERRAIN_CONFIG.shininess` | `10` | `MeshPhongMaterial.shininess` |
|
||||
| 地形 renderOrder | inline | `1.2` | `terrain.renderOrder` |
|
||||
| 地形 polygonOffset | inline | `factor -1`, `units -1` | 降低贴近球面时的闪烁 |
|
||||
|
||||
## 经纬线
|
||||
|
||||
| 正式名称 | 变量名 | 当前值 | 使用位置 / 说明 |
|
||||
| --- | --- | --- | --- |
|
||||
| 经纬线半径偏移 | `GRID_CONFIG.radiusOffset` | `0.14` | 经纬线球面半径 |
|
||||
| 经纬线颜色 | `GRID_CONFIG.color` | `0xc0e0ff` | `LineBasicMaterial.color` |
|
||||
| 经纬线透明度 | `GRID_CONFIG.opacity` | `0.08` | `LineBasicMaterial.opacity` |
|
||||
| 经纬线线宽 | `GRID_CONFIG.lineWidth` | `1` | `LineBasicMaterial.linewidth` |
|
||||
| 经纬线 renderOrder | `GRID_CONFIG.renderOrder` | `2.05` | 经纬线层级 |
|
||||
| 纬线间隔 | `GRID_CONFIG.latitudeStep` | `15` | 纬线生成步长 |
|
||||
| 经线间隔 | `GRID_CONFIG.longitudeStep` | `30` | 经线生成步长 |
|
||||
| 线段采样步长 | `GRID_CONFIG.segmentStep` | `5` | 经纬线采样步长 |
|
||||
|
||||
## 海缆与登陆点
|
||||
|
||||
| 正式名称 | 变量名 | 当前值 | 使用位置 / 说明 |
|
||||
| --- | --- | --- | --- |
|
||||
| 默认海缆颜色 | `CABLE_COLORS.default` | `0xffff44` | 无数据颜色时使用 |
|
||||
| 海缆半径偏移 | `CABLE_CONFIG.line.altitudeOffset` | `0.2` | 海缆线半径 |
|
||||
| 海缆线宽 | `CABLE_CONFIG.line.lineWidth` | `1` | `LineBasicMaterial.linewidth` |
|
||||
| 海缆透明度 | `CABLE_CONFIG.line.opacity` | `1.0` | 海缆线 opacity |
|
||||
| 海缆 renderOrder | `CABLE_CONFIG.line.renderOrder` | `1` | 海缆线层级 |
|
||||
| 登陆点半径偏移 | `CABLE_CONFIG.landingPoint.altitudeOffset` | `0.48` | 对齐算力中心贴地表 marker 高度 |
|
||||
| 登陆点 icon 贴图尺寸 | `CABLE_CONFIG.landingPoint.textureSize` | `256` | canvas 渲染 EPS 参考图的实心 map-pin,中间圆孔透明镂空 |
|
||||
| 登陆点 icon 宽高比 | `CABLE_CONFIG.landingPoint.iconAspectRatio` | `0.82` | `Sprite.scale.x = height * aspect` |
|
||||
| 登陆点 icon 锚点 | `CABLE_CONFIG.landingPoint.anchorX / anchorY` | `0.52 / 0.276` | `Sprite.center`,将 pin 下端点对齐登陆点经纬度 |
|
||||
| 登陆点基础缩放 | `CABLE_CONFIG.landingPoint.baseScale` | `12` | 对齐算力中心等地表 icon 的 sprite 高度 |
|
||||
| 登陆点颜色 | `CABLE_CONFIG.landingPoint.color` | `0xffaa00` | `SpriteMaterial.color` |
|
||||
| 登陆点 emissive | `CABLE_CONFIG.landingPoint.emissive` | `0x442200` | 兼容旧球体材质;sprite 不使用 |
|
||||
| 登陆点 emissive 强度 | `CABLE_CONFIG.landingPoint.emissiveIntensity` | `0.5` | 兼容旧球体材质;sprite 不使用 |
|
||||
| 登陆点透明度 | `CABLE_CONFIG.landingPoint.opacity` | `1.0` | `SpriteMaterial.opacity` |
|
||||
| 登陆点 renderOrder | `CABLE_CONFIG.landingPoint.renderOrder` | `4.5` | 对齐算力中心地表设施层级 |
|
||||
| 登陆点 dim 亮度系数 | `landingPointVisual.dimBrightness` | `0.62` | dim 状态颜色乘数 |
|
||||
| 相关登陆点高亮 opacity | `landingPointVisual.related.opacityBase / opacityPulse` | `0.8 / 0.2` | 高亮脉冲 |
|
||||
| 非相关登陆点颜色 | `landingPointVisual.dimmed.colorRGB` | `{ r: 180, g: 116, b: 28 }` | dim 状态颜色,避免黑色基座透出成暗洞 |
|
||||
| 非相关登陆点 emissive | `landingPointVisual.dimmed.emissive` | `0x3a2200` | dim 状态弱琥珀自发光 |
|
||||
| 非相关登陆点 emissive 强度 | `landingPointVisual.dimmed.emissiveIntensity` | `0.18` | dim 状态弱发光强度 |
|
||||
| 非相关登陆点 opacity | `landingPointVisual.dimmed.opacity` | `0.78` | dim 状态透明度,不再用低 alpha 混黑底 |
|
||||
|
||||
## 卫星、轨迹和 footprint
|
||||
|
||||
| 正式名称 | 变量名 | 当前值 | 使用位置 / 说明 |
|
||||
| --- | --- | --- | --- |
|
||||
| 卫星显示半径偏移 | `SATELLITE_CONFIG.displayAltitudeOffset` | `8` | 卫星点位置 |
|
||||
| 卫星点基础像素大小 | `SATELLITE_CONFIG.dotBaseSize` | `2.8` | 点 shader size |
|
||||
| 卫星背景点缩放 | `SATELLITE_CONFIG.dotBackdropScale` | `1.28` | 背景点大小 |
|
||||
| 卫星点透明度范围 | `dotOpacityMin / dotOpacityMax` | `0.7 / 1.0` | 呼吸动画 |
|
||||
| 卫星点呼吸速度 | `SATELLITE_CONFIG.dotBreathingSpeed` | `0.12` | 点 opacity 动画 |
|
||||
| 卫星背景点颜色 | inline | `0x0b1626` | backdrop point baseColor |
|
||||
| 卫星背景点透明度 | inline | `0.42` | backdrop point opacity |
|
||||
| 卫星点透明度 | inline | `0.9` | point material opacity |
|
||||
| 卫星背景点 renderOrder | inline | `5` | `satelliteBackdropPoints.renderOrder` |
|
||||
| 卫星点 renderOrder | inline | `6` | `satellitePoints.renderOrder` |
|
||||
| 卫星轨迹长度 | `SATELLITE_CONFIG.trailLength` | `10` | trail buffer |
|
||||
| 卫星轨迹线宽 | `SATELLITE_CONFIG.trailLineWidth` | `3` | ribbon shader uniform |
|
||||
| 选中 ring 大小 | `SATELLITE_CONFIG.ringSize` | `0.07` | hover / locked ring sprite |
|
||||
| 卫星覆盖层 renderOrder | `SATELLITE_CONFIG.overlayRenderOrder` | `12` | locked ring / halo / orbit |
|
||||
| 自发光选中点颜色 | inline default | `"#ffd25a"` | `showSelfGlowStyle()` |
|
||||
| 自发光选中点透明度 | inline | `0.96` | locked dot material |
|
||||
| footprint renderOrder | local `GROUND_FOOTPRINT_RENDER_ORDER` | `3` | footprint fill |
|
||||
| footprint group renderOrder | inline | `0` | 避免 Group 排序盖过卫星点 |
|
||||
|
||||
## 算力中心
|
||||
|
||||
| 正式名称 | 变量名 | 当前值 | 使用位置 / 说明 |
|
||||
| --- | --- | --- | --- |
|
||||
| 算力中心半径偏移 | `COMPUTE_CENTER_CONFIG.altitudeOffset` | `0.48` | marker 位置 |
|
||||
| 算力中心基础透明度 | `COMPUTE_CENTER_CONFIG.marker.baseOpacity` | `0.88` | `SpriteMaterial.opacity` |
|
||||
| 超算 marker 缩放 | `COMPUTE_CENTER_CONFIG.marker.supercomputerScale` | `12` | 超算 marker |
|
||||
| GPU 集群 marker 缩放 | `COMPUTE_CENTER_CONFIG.marker.gpuClusterScale` | `12` | GPU marker |
|
||||
| hover 缩放 | `COMPUTE_CENTER_CONFIG.marker.hoverScale` | `1.16` | hover 状态 |
|
||||
| locked 缩放 | `COMPUTE_CENTER_CONFIG.marker.lockedScale` | `1.22` | locked 状态 |
|
||||
| dimmed 缩放 / 透明度 | `dimmedScale / dimmedOpacity` | `0.82 / 0.34` | dim 状态 |
|
||||
| 超算颜色 | `COMPUTE_CENTER_CONFIG.colors.supercomputer` | `"#38bdf8"` | marker texture |
|
||||
| GPU 集群颜色 | `COMPUTE_CENTER_CONFIG.colors.gpu_cluster` | `"#2dd4bf"` | marker texture |
|
||||
| 关联颜色 | `COMPUTE_CENTER_CONFIG.colors.linked` | `"#f8fafc"` | 关联态 |
|
||||
| 算力中心 renderOrder | local `COMPUTE_CENTER_RENDER_ORDER` | `4.5` | 地表设施低于卫星点 |
|
||||
|
||||
## BGP 观测
|
||||
|
||||
| 正式名称 | 变量名 | 当前值 | 使用位置 / 说明 |
|
||||
| --- | --- | --- | --- |
|
||||
| BGP 事件半径偏移 | `BGP_CONFIG.altitudeOffset` | `2.1` | anomaly marker |
|
||||
| BGP collector 半径偏移 | `BGP_CONFIG.collectorAltitudeOffset` | `1.6` | collector marker |
|
||||
| 事件基础缩放 | `BGP_CONFIG.marker.eventBaseScale` | `6.2` | anomaly sprite |
|
||||
| collector 基础缩放 | `BGP_CONFIG.marker.collectorBaseScale` | `7.4` | collector plane |
|
||||
| hover / dim 缩放 | `hoverScale / dimmedScale` | `1.16 / 0.92` | 交互状态 |
|
||||
| 普通事件透明度 | `BGP_CONFIG.opacity.normal` | `0.78` | anomaly sprite |
|
||||
| hover 透明度 | `BGP_CONFIG.opacity.hover` | `1.0` | hover 状态 |
|
||||
| dimmed 透明度 | `BGP_CONFIG.opacity.dimmed` | `0.24` | dim 状态 |
|
||||
| collector 透明度 | `BGP_CONFIG.opacity.collector` | `0.62` | collector 状态 |
|
||||
| critical 颜色 | `BGP_CONFIG.severityColors.critical` | `0xff4d4f` | 严重事件 |
|
||||
| high 颜色 | `BGP_CONFIG.severityColors.high` | `0xff9f43` | 高危事件 |
|
||||
| medium 颜色 | `BGP_CONFIG.severityColors.medium` | `0xffd166` | 中危事件 |
|
||||
| low 颜色 | `BGP_CONFIG.severityColors.low` | `0x4dabf7` | 低危事件 |
|
||||
| collector 基础色 | `BGP_CONFIG.collectorColor` | `0x6db7ff` | collector 默认色 |
|
||||
| region 色 | `BGP_CONFIG.regionColor` | `0x2dd4bf` | 区域覆盖 |
|
||||
| BGP ring 缩放 | `BGP_CONFIG.ring.scaleA / scaleB` | `2.5 / 3.4` | anomaly ring |
|
||||
| BGP ring 透明度 | `BGP_CONFIG.ring.opacity` | `0.5` | anomaly ring |
|
||||
| collector marker renderOrder | inline | `3` | `marker.renderOrder` |
|
||||
| anomaly marker renderOrder | inline | `5` normal, `7` active | `marker.renderOrder` |
|
||||
|
||||
## 天体与星空
|
||||
|
||||
| 正式名称 | 变量名 | 当前值 | 使用位置 / 说明 |
|
||||
| --- | --- | --- | --- |
|
||||
| 天球半径 | `CELESTIAL_CONFIG.skyRadius` | `2600` | 天体背景 |
|
||||
| 天球透明度 | `CELESTIAL_CONFIG.skyOpacity` | `1` | 背景材质 |
|
||||
| 太阳距离 / 缩放 | `sunDistance / sunScale` | `2150 / 78` | 太阳 sprite |
|
||||
| 月亮距离 / 缩放 | `moonDistance / moonScale` | `2050 / 38` | 月亮 sprite |
|
||||
| 太阳 halo 缩放 | `CELESTIAL_CONFIG.sunHaloScale` | `136` | 太阳 halo |
|
||||
| 月亮 halo 缩放 | `CELESTIAL_CONFIG.moonHaloScale` | `62` | 月亮 halo |
|
||||
| 太阳光颜色 / 强度 | `sunLightColor / sunLightIntensity` | `0xfff4df / 1.02` | scene light |
|
||||
| 背光颜色 / 强度 | `backLightColor / backLightIntensity` | `0x2b4c78 / 0.3` | scene light |
|
||||
| 星空点数量 | `STARFIELD_CONFIG.count` | `8000` | `createStars()` |
|
||||
| 星空半径范围 | `minRadius + radiusJitter` | `800 + 200` | 随机分布 |
|
||||
| 星空点颜色 | `STARFIELD_CONFIG.color` | `0xffffff` | `PointsMaterial.color` |
|
||||
| 星空点大小 | `STARFIELD_CONFIG.size` | `0.5` | `PointsMaterial.size` |
|
||||
@@ -95,3 +95,93 @@
|
||||
- 手工配置源
|
||||
- `news_live_streams` 采集器采集源
|
||||
- 当前默认兜底源为 `CCTV-4 中文国际`
|
||||
- `news_live_streams` 在未配置 override 时,默认使用 `iptv-org`:
|
||||
- `channels.json`
|
||||
- `streams.json`
|
||||
- `logos.json`
|
||||
并自动筛出新闻类频道目录
|
||||
|
||||
## 采集器配置方式
|
||||
|
||||
`news_live_streams` 不需要单独新页面,直接复用现有数据源配置:
|
||||
|
||||
- `endpoint`
|
||||
- 频道目录 JSON API 地址
|
||||
- `auth_type`
|
||||
- `none` / `bearer` / `api_key` / `basic`
|
||||
- `headers`
|
||||
- 额外请求头
|
||||
- `config`
|
||||
- 采集器请求与解析行为
|
||||
|
||||
### 支持的 `config` 字段
|
||||
|
||||
```json
|
||||
{
|
||||
"timeout": 30,
|
||||
"method": "GET",
|
||||
"params": {
|
||||
"region": "global"
|
||||
},
|
||||
"body_type": "json",
|
||||
"body": {
|
||||
"include_disabled": false
|
||||
},
|
||||
"response_path": "payload.channels"
|
||||
}
|
||||
```
|
||||
|
||||
- `timeout`
|
||||
- 请求超时秒数
|
||||
- `method`
|
||||
- `GET` 或 `POST`
|
||||
- `params`
|
||||
- 查询参数对象
|
||||
- `body_type`
|
||||
- `json` 或 `form`
|
||||
- `body`
|
||||
- 配合 `POST` 使用的请求体
|
||||
- `json_body`
|
||||
- 显式 JSON 请求体,优先级高于 `body`
|
||||
- `form_body`
|
||||
- 显式表单请求体,优先级高于 `body`
|
||||
- `response_path`
|
||||
- 返回 JSON 中频道数组所在路径,支持点路径,例如:
|
||||
- `payload.channels`
|
||||
- `data.items`
|
||||
- `result.streams`
|
||||
|
||||
### 认证补充
|
||||
|
||||
- `bearer`
|
||||
- 使用 `Authorization: Bearer <token>`
|
||||
- `api_key`
|
||||
- 默认作为请求头发送
|
||||
- 如果 `auth_config.in = "query"`,则作为 query param 发送
|
||||
- `basic`
|
||||
- 使用 HTTP Basic Authorization
|
||||
|
||||
## 兼容的响应结构
|
||||
|
||||
采集器会优先读取:
|
||||
|
||||
- 顶层数组
|
||||
- 或这些常见字段下的数组:
|
||||
- `sources`
|
||||
- `streams`
|
||||
- `channels`
|
||||
- `items`
|
||||
- `results`
|
||||
- `data`
|
||||
|
||||
同时会兼容这些字段别名:
|
||||
|
||||
- `id` / `source_id` / `slug` / `channel_id` / `code`
|
||||
- `name` / `title` / `channel` / `display_name`
|
||||
- `provider` / `publisher` / `network`
|
||||
- `stream_url` / `stream` / `playback_url` / `hls_url` / `m3u8_url`
|
||||
- `embed_url` / `embed` / `page_url`
|
||||
- `homepage_url` / `source_url` / `website`
|
||||
- `language` / `lang` / `locale`
|
||||
- `youtube_video_id` / `video_id`
|
||||
- `youtube_channel` / `channel_handle`
|
||||
57
docs/technical/earth-render-layer-order.md
Normal file
57
docs/technical/earth-render-layer-order.md
Normal file
@@ -0,0 +1,57 @@
|
||||
# Earth 渲染图层顺序
|
||||
|
||||
本文记录当前 Earth 渲染器的图层顺序和每层意图。后续调整
|
||||
`renderOrder`、半径偏移、深度策略或指针交互时,需要同步更新这里。
|
||||
|
||||
注意:图层控制面板顺序和注册 / 启动加载顺序是两套语义。
|
||||
|
||||
| 顺序类型 | 当前顺序 | 说明 |
|
||||
| --- | --- | --- |
|
||||
| 控制面板顺序 | 海缆 → 轨迹 → 卫星 → 算力中心 → BGP → 地形 → 高清材质 → 大气云图 → 国界 → 经纬线 | 由 `displayOrder` 控制,按操作关注度排列。 |
|
||||
| 注册 / 启动加载顺序 | 经纬线 → 国界 → 高清材质 → 大气云图 → 海缆 → 算力中心 → BGP → 卫星 | 由注册顺序和 `startupPriority` 控制,按地表到天空排列;轨迹和地形是依赖/可选显示层,不参与常规启动数据加载。 |
|
||||
|
||||
## 地表图层栈
|
||||
|
||||
| 顺序 | 图层 | 来源 | 渲染 / 半径策略 | 深度 / 交互策略 | 备注 |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| -1000 | 天体背景 mesh | `celestial.js` | 背景球 | 不参与地表拾取 | 位于所有 Earth 内容之后。 |
|
||||
| -1 | Earth 遮挡球 | `earth.js` | 地球内侧不可见球 | 写入深度 | 遮挡地球背面的对象。 |
|
||||
| 0 | Earth 基座球 | `earth.js` | `CONFIG.earthRadius` | 地表拾取兜底目标 | 深色基座,所有可选地图层关闭时仍可见。 |
|
||||
| 0.2 | 国界暗色 tint | `country-boundaries.js` | `tintAltitudeOffset` | 禁用 raycast | 高清材质关闭时使用。 |
|
||||
| 0.86 | 海陆基座填充 | `country-boundaries.js` | `landAltitudeOffset`; 海洋 `#010609`,陆地 `#080f1b` | 禁用 raycast | 即使国界线关闭,基座地图仍保持可用。 |
|
||||
| 0.96 | 高清 Earth 材质 | `earth.js` | `textureOverlayAltitudeOffset` | 可见时作为地表拾取目标 | 高清材质始终压过海陆基座填充。 |
|
||||
| 1 | 大气辉光和云图 | `earth.js` | 大气 / 云层球 | 不走普通对象选择路径 | 云图由“大气云图”图层开关控制。 |
|
||||
| 1 | 海缆 | `cables.js` | `CABLE_CONFIG.line.renderOrder` | 海缆拾取路径 | 保持现有海缆层级。 |
|
||||
| 1.2 | 真实地形 | `earth.js`, `terrain.js` | `TERRAIN_CONFIG.baseRadiusOffset` 加地形位移 | 禁用 raycast | 地形压过高清材质;高清材质关闭时临时隐藏,重新开启后恢复原状态。 |
|
||||
| 2.05 | 经纬线 | `earth.js` | `CONFIG.earthRadius + 0.14` | 禁用 raycast | 低透明度显示在高清材质上。 |
|
||||
| 2.2 | 国界线 | `country-boundaries.js` | `lineAltitudeOffset` | 禁用 raycast | 只保证压过高清材质。 |
|
||||
| 2.29 | 国界 hover 光晕 | `country-boundaries.js` | hover 半径加 glow 偏移 | `depthTest: false`,禁用 raycast | 用 additive 光晕增强交界边和地形开启时的 hover 可见性。 |
|
||||
| 2.3 | 国界 hover 实线 | `country-boundaries.js` | `hoverAltitudeOffset` | `depthTest: false`,禁用 raycast | 霓虹红橘 hover 线;中国和中国(台湾)共享高亮组。 |
|
||||
| 3 | 卫星 footprint 填充 | `satellites.js` | `GROUND_FOOTPRINT_RENDER_ORDER` | depth-tested,Group renderOrder 保持 0 | Footprint 在国界线之上,但在算力中心和卫星之下。 |
|
||||
| 3-5 | BGP 标记和覆盖层 | `bgp.js` | 各 marker 自身 renderOrder | BGP 拾取路径 | 保持现有 BGP 视觉层级。 |
|
||||
| 4.5 | 算力中心 | `compute-centers.js` | `COMPUTE_CENTER_RENDER_ORDER` | 算力中心拾取路径 | 地表设施,保持在卫星下方。 |
|
||||
| 5 | 卫星背景点 | `satellites.js` | 固定 renderOrder | 屏幕空间卫星拾取 | 位于卫星点下方。 |
|
||||
| 6 | 卫星点 | `satellites.js` | 固定 renderOrder | 屏幕空间卫星拾取 | 卫星点压过 footprint 和算力中心。 |
|
||||
| 12+ | 卫星锁定 ring、halo、预测轨道 | `satellites.js` | `SATELLITE_CONFIG.overlayRenderOrder` 及偏移 | 卫星覆盖层路径 | 用于选中 / 锁定卫星强调。 |
|
||||
| 98-100 | 太阳 / 月亮 halo 和 sprite | `celestial.js` | 固定 renderOrder | 天体拾取禁用 | 前景天体 sprite。 |
|
||||
|
||||
## 开关联动
|
||||
|
||||
| 开关 | 行为 |
|
||||
| --- | --- |
|
||||
| 高清材质 off | 隐藏高清材质,启用国界 tint / 基座表面,禁用地形和昼夜开关交互,并记住地形和昼夜之前状态。 |
|
||||
| 高清材质 on | 恢复高清材质,并恢复记住的地形 / 昼夜状态。 |
|
||||
| 地形 on | 显示在高清材质之上,但低于国界 hover、footprint、卫星等强调层。 |
|
||||
| 大气云图 | 只控制云图 mesh 显隐。 |
|
||||
| 国界 | 控制国界线和 hover 线显隐;海陆基座填充独立存在,作为 Earth 基座地图使用。 |
|
||||
|
||||
## 交互规则
|
||||
|
||||
| 交互 | 当前规则 |
|
||||
| --- | --- |
|
||||
| Earth 坐标 hover | 高清材质可见时使用高清材质 overlay 作为地表拾取目标,否则使用 Earth 基座球。 |
|
||||
| 国界 hover | 先把地表拾取坐标转成经纬度,再用 GeoJSON 点面判断;国界 hover 线本身不接收 raycast。 |
|
||||
| 国界 hover 视觉 | hover 时压暗普通国界线,并绘制无深度测试的光晕和实线。 |
|
||||
| 中国 / 台湾 hover | `CHN` 和 `TWN` 被归到同一个 hover 高亮组;tooltip 仍显示鼠标实际命中的 feature。 |
|
||||
| 地形 | 只作为视觉层参与,`terrain.raycast` 已禁用。 |
|
||||
| 卫星 | 使用屏幕空间卫星拾取,避免 footprint 或地表层挡住卫星点击。 |
|
||||
198
docs/technical/earth-satellite-footprint-policy.md
Normal file
198
docs/technical/earth-satellite-footprint-policy.md
Normal file
@@ -0,0 +1,198 @@
|
||||
# Earth Satellite Footprint Policy
|
||||
|
||||
本文件记录 Earth 卫星图层当前关于 `footprint` 的产品边界、资料依据和已落地实现,目标是避免把 Starlink 这套专用地表覆盖模型误用到其它星座上。
|
||||
|
||||
相关上下文:
|
||||
|
||||
- [earth-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/earth-frontend-context.md)
|
||||
- [backend-collectors.md](/home/ray/dev/linkong/planet/docs/technical/backend-collectors.md)
|
||||
- [backend/app/services/collectors/celestrak.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/celestrak.py)
|
||||
- [frontend/public/earth/js/satellites.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/satellites.js)
|
||||
|
||||
## 当前目标
|
||||
|
||||
- 明确哪些非 Starlink 卫星不该显示贴地 footprint
|
||||
- 明确哪些星座未来可以有独立 footprint,但不能复用 Starlink bowtie / GSO-gap 模型
|
||||
- 把这条策略沉淀成可执行实现边界,而不是继续散落在视觉参数里
|
||||
|
||||
## 本地实际类别
|
||||
|
||||
当前 CelesTrak 卫星分组在 [backend/app/services/collectors/celestrak.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/celestrak.py) 中包括:
|
||||
|
||||
- `starlink`
|
||||
- `gps-ops`
|
||||
- `galileo`
|
||||
- `glonass`
|
||||
- `beidou`
|
||||
- `leo`
|
||||
- `geo`
|
||||
- `iridium-next`
|
||||
|
||||
其中非 Starlink 类别是:
|
||||
|
||||
- `gps-ops`
|
||||
- `galileo`
|
||||
- `glonass`
|
||||
- `beidou`
|
||||
- `leo`
|
||||
- `geo`
|
||||
- `iridium-next`
|
||||
|
||||
## 资料结论
|
||||
|
||||
### 1. GNSS / RNSS: `gps-ops`, `galileo`, `glonass`, `beidou`
|
||||
|
||||
默认不要画局部地表 footprint。
|
||||
|
||||
原因:
|
||||
|
||||
- 公开资料强调的是 `Earth-pointing`、`Earth coverage`、`continuous global coverage`
|
||||
- 这类系统的公开语义是全球导航 / 授时覆盖,不是 Starlink 那种面向终端业务的局部 spot footprint
|
||||
|
||||
更合适的表示:
|
||||
|
||||
- 默认只显示卫星本体和轨道
|
||||
- 如果后续要强调“服务可达性”,只能做很弱的 global coverage 语义,不应画贴地局部光斑
|
||||
|
||||
资料:
|
||||
|
||||
- [GPS III EC Antenna Patterns](https://www.navcen.uscg.gov/sites/default/files/pdf/gps/GPS_ZIP/GPS_III_EC_Antenna_Patterns_SVN_74_75_76_77_78.pdf)
|
||||
- [ESA Galileo satellites](https://www.esa.int/Applications/Satellite_navigation/Galileo/Galileo_satellites)
|
||||
- [Navipedia Galileo General Introduction](https://gssc.esa.int/navipedia/index.php/Galileo_General_Introduction)
|
||||
- [BeiDou official overview](https://www.beidou.gov.cn/xt/gfxz/201812/P020190117356387956569.pdf)
|
||||
- [GPS.gov GNSS overview](https://www.gps.gov/systems/gnss/)
|
||||
|
||||
### 2. `iridium-next`
|
||||
|
||||
可以有 footprint,但不能复用 Starlink 的单一 bowtie footprint。
|
||||
|
||||
原因:
|
||||
|
||||
- Iridium NEXT 公开资料强调的是固定多 spot beam 体系
|
||||
- 公开示例里常见的是 `48 fixed spot beams in 4 tiers`
|
||||
- 这和 Starlink 当前这套“单星、单主 footprint、带 GSO 缺口”的业务可视化不是同一个问题
|
||||
|
||||
更合适的表示:
|
||||
|
||||
- 默认:仍然不画 Starlink 式地表 footprint
|
||||
- 后续如果要做:单独接入 Iridium 多波束适配层
|
||||
- 在视觉上更接近多束 cluster / 蜂窝 / 分层束,而不是单个 bowtie 光斑
|
||||
|
||||
资料:
|
||||
|
||||
- [Iridium Satellite Spot Beam Coverage on the US](https://www.mathworks.com/help/phased/ug/iridium-satellite-spot-beam-coverage-on-the-us-1.html)
|
||||
|
||||
### 3. `geo`
|
||||
|
||||
默认不要画统一 footprint。
|
||||
|
||||
原因:
|
||||
|
||||
- GEO 通信星公开上可能是 global beam、zone beam、spot beam、steerable spot beam
|
||||
- 没有 operator / payload / beam contour 元数据时,统一画一个 footprint 很容易错
|
||||
|
||||
更合适的表示:
|
||||
|
||||
- 默认只显示 GEO belt 和卫星驻点语义
|
||||
- 只有拿到 beam contour / operator metadata 时才允许画 footprint
|
||||
|
||||
资料:
|
||||
|
||||
- [ITU Handbook on Satellite](https://www.itu.int/dms_pub/itu-r/opb/hdb/R-HDB-42-2002-PDF-E.pdf)
|
||||
|
||||
### 4. `leo`(generic)
|
||||
|
||||
默认不要画 footprint。
|
||||
|
||||
原因:
|
||||
|
||||
- `leo` 组过于混杂,可能同时包含通信、遥感、试验、观测等不同任务
|
||||
- 没有 mission / payload / antenna pattern 元数据时,无法判断是否存在可视化意义上的服务覆盖面
|
||||
|
||||
更合适的表示:
|
||||
|
||||
- 默认只显示卫星和轨道
|
||||
- 后续如果按 operator / mission subtype 细分,再决定是否引入独立 coverage mode
|
||||
|
||||
## 产品策略
|
||||
|
||||
当前统一策略如下:
|
||||
|
||||
- `Starlink`
|
||||
- 保留当前专用 `ground_footprint` 逻辑
|
||||
- `Iridium NEXT`
|
||||
- 预留独立适配层
|
||||
- 当前不复用 Starlink footprint
|
||||
- `GPS / Galileo / GLONASS / BeiDou`
|
||||
- 不显示贴地 footprint
|
||||
- `GEO`
|
||||
- 无 beam metadata 不显示 footprint
|
||||
- `generic LEO`
|
||||
- 无 mission metadata 不显示 footprint
|
||||
|
||||
## 已落地实现
|
||||
|
||||
本次实现只做最小可执行版本,不改现有 Starlink 视觉参数:
|
||||
|
||||
1. 后端把星座分组和 footprint 策略提示透给前端
|
||||
|
||||
- CelesTrak collector 会把 `GROUP` 记入 `metadata.constellation_group`
|
||||
- Visualization API 会输出:
|
||||
- `properties.constellation_group`
|
||||
- `properties.footprint_policy`
|
||||
|
||||
当前策略值:
|
||||
|
||||
- `starlink_ground_footprint`
|
||||
- `iridium_coverage_ring`
|
||||
- `none`
|
||||
|
||||
对应代码:
|
||||
|
||||
- [backend/app/services/collectors/celestrak.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/celestrak.py)
|
||||
- [backend/app/api/v1/visualization.py](/home/ray/dev/linkong/planet/backend/app/api/v1/visualization.py)
|
||||
|
||||
2. 前端把 footprint 变成 capability-gated renderer
|
||||
|
||||
- `ground_footprint` 只有在 `footprint_policy === starlink_ground_footprint` 时才真正启用
|
||||
- `iridium-next` 不再回退成占位分支,而是走独立的 Iridium coverage ring adapter
|
||||
- 其它非 Starlink 即使用户全局选择了 `ground_footprint`,也会自动回退到 `self_glow`
|
||||
|
||||
对应代码:
|
||||
|
||||
- [frontend/public/earth/js/satellites.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/satellites.js)
|
||||
- [frontend/public/earth/js/iridium-footprint-adapter.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/iridium-footprint-adapter.js)
|
||||
|
||||
3. 卫星信息卡显示 capability,而不是只显示轨道参数
|
||||
|
||||
- 卫星详情现在会明确显示:
|
||||
- `星座/分组`
|
||||
- `覆盖能力`
|
||||
- `当前显示`
|
||||
- `覆盖模型`
|
||||
- 这样用户能直接看到:
|
||||
- 当前卫星是否支持 footprint
|
||||
- 当前显示是不是因为 capability gating 被回退
|
||||
- Iridium 和 Starlink 使用的不是同一种模型
|
||||
|
||||
对应代码:
|
||||
|
||||
- [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js)
|
||||
- [frontend/public/earth/js/info-card.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/info-card.js)
|
||||
|
||||
## 当前实现边界
|
||||
|
||||
这条边界需要继续保持:
|
||||
|
||||
- `Starlink` 的 footprint 参数和 shader 逻辑只服务于 Starlink
|
||||
- 非 Starlink 的能力判断属于“策略层 / 适配层”
|
||||
- 不要把不同星座的覆盖模型再混写进同一套参数里
|
||||
- `iridium-next` 已经切成独立 adapter,应继续沿这条边界演进,而不是给现有 Starlink bowtie 增加更多 if/else
|
||||
|
||||
## 后续建议
|
||||
|
||||
如果继续往前做,推荐顺序是:
|
||||
|
||||
1. 为 `iridium-next` 新建独立 footprint adapter
|
||||
2. 在 UI 上补一个只读提示,让用户知道当前卫星是否支持 footprint
|
||||
3. 如果未来拿到 GEO beam contour / operator metadata,再为 GEO 开 operator-specific footprint
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user