Merge pull request 'release: bump version to 0.27.6' (#7) from dev into feature/ue5-led-client

Reviewed-on: #7

.claude/commands/cleanup.md

@@ -0,0 +1,120 @@
+---
+description: 审查当前工作区未提交代码中的垃圾代码，并在不影响逻辑的前提下自动清理
+argument-hint: 可选：指定要检查的文件或目录（默认检查所有未提交修改）
+allowed-tools: ["Read", "Edit", "Bash", "Grep", "Glob"]
+---
+
+# /cleanup — 垃圾代码审查与清理
+
+分析当前工作区（git diff）中的未提交代码，找出并修复常见垃圾代码，**不得改变任何运行逻辑**。
+
+## 检查范围
+
+若 `$ARGUMENTS` 非空，则只检查指定文件/目录；否则检查所有未提交修改（`git diff HEAD`）。
+
+## 审查清单
+
+按优先级检查以下问题（只报告在本次 diff 中**新增或修改**的代码里存在的问题）：
+
+### 1. 重复逻辑 (Duplicate Logic)
+- 完全相同或高度相似的代码块在多处出现
+- 同一函数/方法被多个地方各自实现，已有公共版本未被复用
+- 相同的 DOM 查询、正则、模板字符串在同一文件重复
+
+### 2. Magic Numbers / Magic Strings
+- 裸数字直接参与计算（如偏移量、时间、尺寸、阈值），没有命名常量
+- 硬编码字符串（如 id 名、状态值、URL 片段）散落在逻辑中
+- 例外：`0`, `1`, `-1`, `100`, `""` 等语义明确的惯用值不算
+
+### 3. 命名问题
+- 含义不明的缩写变量（如 `or_`, `tmp2`, `x2`）
+- 命名与实际用途不符
+- 同一概念在不同地方用不同名字表达
+
+### 4. 死代码 / 无效代码
+- 注释掉的旧代码块（3行以上）
+- 声明后从未使用的变量/参数/导入
+- 永远不会执行的条件分支
+
+### 5. 代码风格问题
+- 尾部空白字符（trailing whitespace）
+- 同一文件内风格不一致（如混用单双引号、缩进不统一）
+- 空行使用不一致（连续多个空行等）
+
+### 6. 其他常见问题
+- 私有辅助函数应被 export 但没有，导致调用方重复实现
+- 类型/接口重复定义
+- 过于冗长的条件表达式可以简化（不改逻辑）
+
+## 执行步骤
+
+### Step 1 — 获取待检查文件列表
+
+```bash
+# 无参数时：获取所有未提交修改
+git diff HEAD --name-only
+
+# 有参数时：用 $ARGUMENTS 过滤
+```
+
+### Step 2 — 逐文件阅读并分析
+
+- 用 Read 工具读取完整文件（不只读 diff）
+- 对照审查清单，记录每个问题：文件名、行号、问题类型、建议修复方式
+
+### Step 3 — 报告问题清单
+
+在修改前，先以列表形式输出所有发现的问题：
+
+```
+发现 N 个问题：
+
+[文件] js/foo.js
+  · L34, L78: 重复逻辑 — 两处都实现了相同的 DOM 查询，可提取到 getPanel()
+  · L91: Magic number — 硬编码 14 作为偏移量，应命名为 TOOLTIP_OFFSET
+
+[文件] js/bar.js
+  · L12: 命名问题 — 变量 `or_` 语义不明，应命名为 outerR/outerG/outerB
+  ...
+```
+
+如果没有发现问题，直接输出"未发现垃圾代码，当前代码质量良好。"并停止。
+
+### Step 4 — 执行修复
+
+对每个问题，使用 Edit 工具进行**最小化修改**：
+
+- **重复逻辑**：提取为共享常量/函数，更新所有调用点
+- **Magic number**：在文件顶部或逻辑附近声明 `const NAME = value`，替换所有引用
+- **命名问题**：重命名变量，更新所有使用处
+- **死代码**：直接删除
+- **尾部空白/风格**：修正
+- **未 export 的函数**：添加 `export`，在调用方改为导入（不重复实现）
+
+**修复原则：**
+- 只改在审查清单中发现的问题，不做额外优化
+- 每次 Edit 只修改确实有问题的行，保持 diff 最小
+- 改完后用 `grep` 验证旧的坏代码已消失
+
+### Step 5 — 输出总结
+
+```
+清理完成：
+
+修复了 N 个问题：
+  ✓ earth.js — 提取重复 vertexShader 为 ATMOS_VERTEX_SHADER 常量
+  ✓ main.js — 提取 TOOLTIP_CURSOR_OFFSET = 14（4处引用）
+  ✓ controls.js — export updateLayerButtonState，移除 main.js 中的重复实现
+  ...
+
+未修改的问题（需人工确认）：
+  ! foo.js L45 — 注释代码块较长，建议手动确认是否可删除
+```
+
+## 约束
+
+- **禁止**改变函数签名、接口定义、导出 API（除非问题正是私有函数应被 export）
+- **禁止**添加新功能、新抽象、新参数
+- **禁止**修改注释内容（只删除注释掉的死代码）
+- **禁止**修改测试文件逻辑
+- 如果一个 Magic number 的语义不完全确定，**跳过**，在总结中标记为"需人工确认"

.claude/commands/release.md

@@ -0,0 +1,146 @@
+---
+description: 发版工作流：根据变更类型决定版本号，更新所有版本文件和 changelog，运行验证，commit 并 push
+argument-hint: 可选：feature | bugfix | 或直接描述本次发布内容
+allowed-tools: ["Read", "Edit", "Bash", "Glob", "Grep"]
+---
+
+# /release — Planet 发版工作流
+
+## 版本号规则
+
+| 变更类型 | 版本跳动 | 适用场景 |
+|---------|---------|---------|
+| `feature` | `+0.1.0` | 纯新功能，无 bugfix |
+| `improvement` | `+0.0.1` | UI 调整、小功能增强、bugfix 混合，或以 UI/体验改进为主的迭代 |
+| `bugfix` | `+0.0.1` | 纯 bug 修复，无新功能 |
+| `docs` / `maintenance` / `refactor` | 默认不发版，除非用户明确要求 |
+
+意图混合时以用户明确描述为准；bugfix + 小 feature 混合默认判定为 `improvement`（`+0.0.1`）。
+
+## 必须同步更新的文件
+
+使用 `git rev-parse --show-toplevel` 获取仓库根目录，以下路径均相对于根目录：
+
+- `VERSION`
+- `frontend/package.json`（`"version"` 字段）
+- `pyproject.toml`（`version =` 字段）
+- `uv.lock`（**不要手动编辑**，通过 `uv lock` 重新生成）
+- `docs/CHANGELOG.md`
+- `docs/version-history.md`
+
+## 执行步骤
+
+### Step 1 — 环境检查
+
+```bash
+git branch --show-current          # 确认在 dev 分支
+git status --short                 # 检查是否有无关的未暂存修改
+cat VERSION                        # 读取当前版本
+```
+
+若当前**不在 `dev` 分支**，停下来告知用户，不要继续。
+
+若存在无关的未暂存修改，列出并询问用户是否一并提交，或先 stash。
+
+### Step 2 — 确定发版类型与新版本号
+
+- 若 `$ARGUMENTS` 提供了明确类型（`feature` / `bugfix`），直接使用
+- 否则根据当前 `git diff HEAD` 和 `git log` 推断
+- 计算新版本号（例：`0.26.2` → bugfix → `0.26.3`）
+- **先输出发版计划供用户确认**：
+
+```
+发版计划：
+  类型：bugfix
+  版本：0.26.2 → 0.26.3
+  分支：dev
+  将更新：VERSION, frontend/package.json, pyproject.toml, uv.lock, CHANGELOG.md, version-history.md
+```
+
+### Step 3 — 更新版本号文件
+
+按顺序更新（每步用 Edit 工具，精确替换，不要重写整个文件）：
+
+1. `VERSION` — 直接替换全部内容为新版本号
+2. `frontend/package.json` — 替换 `"version": "x.x.x"` 行
+3. `pyproject.toml` — 替换 `version = "x.x.x"` 行
+4. 运行 `uv lock` 重新生成 `uv.lock`（在仓库根目录下执行）
+
+### Step 4 — 更新 CHANGELOG.md
+
+在文件顶部插入新条目，格式：
+
+```markdown
+## [x.x.x] — YYYY-MM-DD
+
+### ✨ Features / 🐛 Fixes / 🔧 Improvements
+- ...（只列高信号条目，最多 5 条）
+- ...
+
+---
+```
+
+日期使用 `date +%Y-%m-%d` 获取今天的日期。
+
+### Step 5 — 更新 docs/version-history.md
+
+- 更新文件头部的"当前开发版本"字段
+- 在时间线表格顶部插入新行：`| vx.x.x | YYYY-MM-DD | 一句话摘要 |`
+
+### Step 6 — 验证
+
+针对本次变更范围做最小验证：
+
+- Python 文件有修改：`python3 -m py_compile <changed_files>`
+- Frontend 文件有修改：运行项目标准检查（若无则跳过并说明）
+- 版本号一致性检查：用 grep 确认 VERSION、package.json、pyproject.toml 中的版本号完全一致
+
+```bash
+grep -h "version" VERSION frontend/package.json pyproject.toml
+```
+
+### Step 7 — 提交前预览
+
+展示将要提交的文件列表：
+
+```bash
+git diff --stat HEAD
+```
+
+再次确认所有必须文件都在变更列表中，**不包含**非预期文件（如调试文件、.env 等）。
+
+### Step 8 — Commit & Push（用户确认后）
+
+```bash
+git add VERSION frontend/package.json pyproject.toml uv.lock docs/CHANGELOG.md docs/version-history.md
+# 若有代码变更也一并 stage
+git add <code_files>
+
+git commit -m "release: bump version to x.x.x"
+git tag vx.x.x
+git push origin dev
+git push origin vx.x.x
+```
+
+commit message 固定格式：`release: bump version to x.x.x`
+
+### Step 9 — 完成确认
+
+输出摘要：
+
+```
+✓ 版本号已更新：0.26.2 → 0.26.3
+✓ CHANGELOG 已更新
+✓ version-history 已更新
+✓ uv.lock 已重新生成
+✓ 验证通过
+✓ commit: release: bump version to 0.26.3
+✓ tag: v0.26.3
+✓ 已 push 到 origin/dev
+```
+
+## 注意事项
+
+- `uv.lock` 只能通过 `uv lock` 生成，绝不手动编辑
+- 发版 commit 只包含版本文件 + 本次功能代码，不混入无关改动
+- 若环境中 `uv` 不可用，说明原因并跳过 lockfile 更新，提醒用户手动运行

.codex/skills/cleanup/SKILL.md

@@ -0,0 +1,124 @@
+---
+name: cleanup
+description: Use when the user asks to clean up, lint, or review uncommitted code for common code smells — duplicate logic, magic numbers, unclear naming, dead code, style inconsistencies. Fixes issues without changing any runtime behavior.
+---
+
+# Cleanup
+
+Review and fix code quality issues in the current working tree without altering any logic or behavior.
+
+## When To Use
+
+- The user asks to clean up, tidy, or lint uncommitted changes
+- The user wants a code smell review before releasing or committing
+- The user mentions magic numbers, duplicate logic, dead code, or naming issues
+
+Do not refactor architecture, add features, or change behavior.
+
+## Scope
+
+If the user specifies a file or directory, check only that. Otherwise check all uncommitted changes (`git diff HEAD`).
+
+Only report issues present in **newly added or modified** lines of this diff — do not audit unchanged code.
+
+## Checklist
+
+### 1. Duplicate Logic
+- Identical or near-identical code blocks appearing in multiple places
+- A function/helper that already exists but is re-implemented elsewhere instead of being reused
+- Repeated DOM queries, regex literals, or template strings within the same file
+
+### 2. Magic Numbers / Magic Strings
+- Bare numeric literals used in calculations (offsets, timeouts, sizes, thresholds) without a named constant
+- Hardcoded strings (IDs, status values, URL fragments) scattered through logic
+- Exceptions: `0`, `1`, `-1`, `100`, `""` and other idiomatically clear values are fine
+
+### 3. Naming Issues
+- Cryptic abbreviations (`or_`, `tmp2`, `x2`)
+- Names that do not match actual behavior
+- The same concept referred to by different names in different places
+
+### 4. Dead Code
+- Commented-out code blocks (3+ lines)
+- Variables, parameters, or imports declared but never used
+- Branches that can never execute
+
+### 5. Style Inconsistencies
+- Trailing whitespace
+- Mixed quote styles or indentation within the same file
+- Inconsistent blank-line usage (multiple consecutive blank lines, etc.)
+
+### 6. Other
+- Private helper functions that should be exported but are not, causing callers to duplicate the implementation
+- Overly verbose conditions that can be simplified without changing logic
+
+## Steps
+
+### Step 1 — Get the file list
+
+```bash
+git diff HEAD --name-only
+```
+
+Filter to the user-specified path if one was provided.
+
+### Step 2 — Read and analyze each file
+
+Read the full file (not just the diff) with the Read tool. For each file, record every issue found: filename, line number, category, and suggested fix.
+
+### Step 3 — Report findings before touching anything
+
+Print a structured list:
+
+```
+Found N issues:
+
+[file] js/foo.js
+  · L34, L78: Duplicate logic — same DOM query implemented twice; extract to getPanel()
+  · L91: Magic number — bare 14 used as pixel offset; name it TOOLTIP_OFFSET
+
+[file] js/bar.js
+  · L12: Naming — variable `or_` is unclear; rename to outerR, outerG, outerB
+  ...
+```
+
+If no issues are found, output "No code smells detected. Code quality looks good." and stop.
+
+### Step 4 — Fix each issue
+
+Use the Edit tool for **minimal, targeted changes**:
+
+- **Duplicate logic**: extract to a shared constant or function; update all call sites
+- **Magic number/string**: declare `const NAME = value` near the top of the relevant scope; replace all usages
+- **Naming**: rename the variable/function; update all references
+- **Dead code**: delete it
+- **Trailing whitespace / style**: fix in place
+- **Unexported helper**: add `export`; update callers to import instead of re-implementing
+
+Principles:
+- Only fix issues identified in the checklist — no extra improvements
+- Keep each Edit as small as possible
+- After fixing, verify the old bad pattern is gone with grep
+
+### Step 5 — Summary
+
+```
+Cleanup complete:
+
+Fixed N issues:
+  ✓ earth.js — extracted duplicate vertexShader into ATMOS_VERTEX_SHADER constant
+  ✓ main.js — extracted TOOLTIP_CURSOR_OFFSET = 14 (4 references updated)
+  ✓ controls.js — exported updateLayerButtonState; removed duplicate implementation in main.js
+  ...
+
+Skipped (needs manual review):
+  ! foo.js L45 — large commented-out block; confirm it is safe to delete
+```
+
+## Constraints
+
+- **Do not** change function signatures, exported interfaces, or public APIs (unless the issue is a missing export)
+- **Do not** add new features, abstractions, or parameters
+- **Do not** rewrite comments (only delete commented-out dead code)
+- **Do not** touch test file logic
+- If a magic number's intent is uncertain, skip it and flag it in the summary

.codex/skills/release/SKILL.md

@@ -0,0 +1,157 @@
+---
+name: release
+description: Use when the user asks to release, bump version, update changelog/version files, or commit/push a repository release for the Planet repo. Determines version bump type from changes, updates all required version-bearing files, updates changelog and version-history, runs minimal validation, then commits, tags, and pushes.
+---
+
+# Release Workflow
+
+Use this skill for release-oriented work in this repository.
+
+## When To Use
+
+- The user asks to `发版`
+- The user asks to bump a version
+- The user asks to update `CHANGELOG`, `version-history`, or version files as part of a release
+- The user asks to commit/push a release or a publishable bugfix/feature bundle
+
+Do not use this skill for ordinary commits that are not being released.
+
+## Versioning Rules
+
+- `feature` -> bump `+0.1.0`
+- `bugfix` -> bump `+0.0.1`
+- `docs`, `maintenance`, and `refactor` do not bump by default unless the user explicitly wants a release
+
+When intent is mixed, prefer the user's stated release intent.
+
+## Required Files
+
+Use `git rev-parse --show-toplevel` to get the repo root. All paths are relative to it:
+
+- `VERSION`
+- `frontend/package.json` (`"version"` field)
+- `pyproject.toml` (`version =` field)
+- `uv.lock` (**never edit manually** — regenerate by running `uv lock`)
+- `docs/CHANGELOG.md`
+- `docs/version-history.md`
+
+## Workflow
+
+### Step 1 — Environment check
+
+```bash
+git branch --show-current   # must be on dev
+git status --short           # check for unrelated uncommitted changes
+cat VERSION                  # read current version
+```
+
+If not on `dev`, stop and tell the user. Do not proceed.
+
+If unrelated uncommitted changes exist, list them and ask the user whether to include them or stash first.
+
+### Step 2 — Determine release type and next version
+
+- If the user provided an explicit type (`feature` / `bugfix`), use it
+- Otherwise infer from `git diff HEAD` and recent `git log`
+- Compute the next version (e.g. `0.26.2` → bugfix → `0.26.3`)
+- **Show the release plan before making any changes:**
+
+```
+Release plan:
+  Type:    bugfix
+  Version: 0.26.2 → 0.26.3
+  Branch:  dev
+  Will update: VERSION, frontend/package.json, pyproject.toml, uv.lock, CHANGELOG.md, version-history.md
+```
+
+### Step 3 — Update version files
+
+Update in order (use Edit for precise replacement, never rewrite whole files):
+
+1. `VERSION` — replace entire content with new version string
+2. `frontend/package.json` — replace `"version": "x.x.x"` line
+3. `pyproject.toml` — replace `version = "x.x.x"` line
+4. Run `uv lock` at repo root to regenerate `uv.lock`
+
+### Step 4 — Update CHANGELOG.md
+
+Insert a new entry at the top of the file:
+
+```markdown
+## x.x.x
+
+Released: YYYY-MM-DD
+
+### Highlights
+
+- ...
+
+### Added / Fixed / Improved
+
+- ... (high-signal items only, max 5)
+
+---
+```
+
+Get today's date with `date +%Y-%m-%d`.
+
+### Step 5 — Update docs/version-history.md
+
+- Update the "current dev version" field in the file header
+- Insert a new row at the top of the timeline table: `| vx.x.x | YYYY-MM-DD | one-line summary |`
+
+### Step 6 — Validate
+
+Run the smallest relevant validation for the changes in scope:
+
+- Python files changed: `python3 -m py_compile <changed_files>`
+- Frontend files changed: run the project-standard check if available; otherwise skip and say so
+- Version consistency: confirm VERSION, package.json, pyproject.toml, and uv.lock all show the same version
+
+```bash
+grep -h "version" VERSION frontend/package.json pyproject.toml
+```
+
+### Step 7 — Pre-commit preview
+
+Show what will be committed:
+
+```bash
+git diff --stat HEAD
+```
+
+Confirm all required files are present and no unexpected files (debug files, `.env`, etc.) are included.
+
+### Step 8 — Commit, tag, and push
+
+```bash
+git add VERSION frontend/package.json pyproject.toml uv.lock docs/CHANGELOG.md docs/version-history.md
+# also stage any code changes included in this release
+git add <code_files>
+
+git commit -m "release: bump version to x.x.x"
+git tag vx.x.x
+git push origin dev
+git push origin vx.x.x
+```
+
+Commit message format is fixed: `release: bump version to x.x.x`
+
+### Step 9 — Completion summary
+
+```
+✓ Version bumped: 0.26.2 → 0.26.3
+✓ CHANGELOG updated
+✓ version-history updated
+✓ uv.lock regenerated
+✓ Validation passed
+✓ commit: release: bump version to 0.26.3
+✓ tag: v0.26.3
+✓ Pushed to origin/dev
+```
+
+## Notes
+
+- `uv.lock` must only be updated by running `uv lock`, never manually
+- The release commit should include only version files + the code for this release — no unrelated changes
+- If `uv` is unavailable in the environment, say so explicitly and remind the user to run it manually

.gitignore

@@ -41,6 +41,8 @@ MANIFEST
 venv/
 ENV/
 env/
+.uv/
+.uv-cache/
 .ruff_cache/
 *.db
 *.sqlite
@@ -143,3 +145,8 @@ docs/.venv/
 *.temp
 tmp/
 temp/
+
+# ----------------------
+# Runtime Data
+# ----------------------
+data/ai/bgp-briefs/

.python-version

@@ -0,0 +1 @@
+3.14

.sisyphus/plans/predicted-orbit.md

@@ -0,0 +1,136 @@
+# 卫星预测轨道显示功能
+
+## TL;DR
+> 锁定卫星时显示绕地球完整一圈的预测轨道轨迹，从当前位置向外渐变消失
+
+## Context
+
+### 目标
+点击锁定卫星 → 显示该卫星绕地球一周的完整预测轨道（而非当前的历史轨迹）
+
+### 当前实现
+- `TRAIL_LENGTH = 30` - 历史轨迹点数，每帧 push 当前位置
+- 显示最近30帧历史轨迹（类似彗星尾巴）
+
+### 参考: SatelliteMap.space
+- 锁定时显示预测轨道
+- 颜色从当前位置向外渐变消失
+- 使用 satellite.js（与本项目相同）
+
+## 实现状态
+
+### ✅ 已完成
+- [x] 计算卫星轨道周期（基于 `meanMotion`）
+- [x] 生成预测轨道点（10秒采样间隔）
+- [x] 创建独立预测轨道渲染对象
+- [x] 锁定卫星时显示预测轨道
+- [x] 解除锁定时隐藏预测轨道
+- [x] 颜色渐变：当前位置(亮) → 轨道终点(暗)
+- [x] 页面隐藏时清除轨迹（防止切回时闪现）
+
+### 🚧 进行中
+- [ ] 完整圆环轨道（部分卫星因 SGP4 计算问题使用 fallback 圆形轨道）
+- [ ] 每颗卫星只显示一条轨道
+
+## 技术细节
+
+### 轨道周期计算
+```javascript
+function calculateOrbitalPeriod(meanMotion) {
+  return 86400 / meanMotion;
+}
+```
+
+### 预测轨道计算
+```javascript
+function calculatePredictedOrbit(satellite, periodSeconds, sampleInterval = 10) {
+  const points = [];
+  const samples = Math.ceil(periodSeconds / sampleInterval);
+  const now = new Date();
+  
+  // Full orbit: from now to now+period
+  for (let i = 0; i <= samples; i++) {
+    const time = new Date(now.getTime() + i * sampleInterval * 1000);
+    const pos = computeSatellitePosition(satellite, time);
+    if (pos) points.push(pos);
+  }
+  
+  // Fallback: 如果真实位置计算点太少，使用圆形 fallback
+  if (points.length < samples * 0.5) {
+    points.length = 0;
+    // ... 圆形轨道生成
+  }
+  
+  return points;
+}
+```
+
+### 渲染对象
+```javascript
+let predictedOrbitLine = null;
+
+export function showPredictedOrbit(satellite) {
+  hidePredictedOrbit();
+  // ... 计算并渲染轨道
+}
+
+export function hidePredictedOrbit() {
+  if (predictedOrbitLine) {
+    earthObjRef.remove(predictedOrbitLine);
+    predictedOrbitLine.geometry.dispose();
+    predictedOrbitLine.material.dispose();
+    predictedOrbitLine = null;
+  }
+}
+```
+
+## 已知问题
+
+### 1. TLE 格式问题
+`computeSatellitePosition` 使用自行构建的 TLE 格式，对某些卫星返回 null。当前使用 fallback 圆形轨道作为补偿。
+
+### 2. 多条轨道
+部分情况下锁定时会显示多条轨道。需要确保 `hidePredictedOrbit()` 被正确调用。
+
+## 性能考虑
+
+### 点数估算
+| 卫星类型 | 周期 | 10秒采样 | 点数 |
+|---------|------|---------|------|
+| LEO | 90分钟 | 540秒 | ~54点 |
+| MEO | 12小时 | 4320秒 | ~432点 |
+| GEO | 24小时 | 8640秒 | ~864点 |
+
+### 优化策略
+- 当前方案（~900点 GEO）性能可接受
+- 如遇性能问题：GEO 降低采样率到 30秒
+
+## 验证方案
+
+### QA Scenarios
+
+**Scenario: 锁定 Starlink 卫星显示预测轨道**
+1. 打开浏览器，进入 Earth 页面
+2. 显示卫星（点击按钮）
+3. 点击一颗 Starlink 卫星（低轨道 LEO）
+4. 验证：出现黄色预测轨道线，从卫星向外绕行
+5. 验证：颜色从亮黄渐变到暗蓝
+6. 验证：轨道完整闭环
+
+**Scenario: 锁定 GEO 卫星显示预测轨道**
+1. 筛选一颗 GEO 卫星（倾斜角 0-10° 或高轨道）
+2. 点击锁定
+3. 验证：显示完整 24 小时轨道（或 fallback 圆形轨道）
+4. 验证：点数合理（~864点或 fallback）
+
+**Scenario: 解除锁定隐藏预测轨道**
+1. 锁定一颗卫星，显示预测轨道
+2. 点击地球空白处解除锁定
+3. 验证：预测轨道消失
+
+**Scenario: 切换页面后轨迹不闪现**
+1. 锁定一颗卫星
+2. 切换到其他标签页
+3. 等待几秒
+4. 切回页面
+5. 验证：轨迹不突然闪现累积

README.md

@@ -102,6 +102,13 @@
 | Axios | HTTP 客户端 |
 | Socket.io-client | WebSocket 客户端 |
 | ECharts | 统计图表 |
+| Bun | 前端包管理与脚本运行 |
+
+前端工程统一使用 Bun：
+
+- 安装依赖使用 `bun install`
+- 运行脚本使用 `bun run <script>`
+- 不使用 `npm`、`pnpm`、`yarn`
 
 ### 虚幻引擎客户端
 
@@ -184,20 +191,163 @@
 ## 快速启动
 
 ```bash
-# 启动全部服务
-docker-compose up -d
+# 新机器首次初始化
+./scripts/bootstrap-dev.sh
+# 会自动安装/检查 uv、bun，并同步 Python/前端依赖
+# 会在缺少时生成 backend/.env、aiprovider/.env、frontend/.env.local
 
-# 仅启动后端
-cd backend && python -m uvicorn app.main:app --reload
+# 启动前后端服务
+./planet.sh start
 
-# 仅启动前端
-cd frontend && npm run dev
+# 仅重启后端
+./planet.sh restart -b
+
+# 仅重启前端
+./planet.sh restart -f
+
+# 交互创建用户
+./planet.sh createuser
+
+# 查看服务状态
+./planet.sh health
 ```
 
+前端命令约定：
+
+```bash
+cd frontend
+bun install
+bun run dev
+bun run build
+```
+
+不要使用 `npm run ...`，避免在 WSL/Windows 混合环境里触发 `cmd.exe` 路径兼容问题。
+
 ## API 文档
 
 启动服务后访问: `http://localhost:8000/docs`
 
+## 启动容错参数
+
+`planet.sh` 现在为依赖安装、数据库、AI Provider 启动加入了有限次重试，并会在数据库与 `aiprovider` 启动后额外等待 Docker healthcheck。
+
+可通过环境变量临时调整:
+
+```bash
+# 例: 放宽 AI Provider 与数据库在网络抖动下的自愈次数
+AI_PROVIDER_START_MAX_RETRIES=5 \
+AI_PROVIDER_RETRY_INTERVAL=10 \
+DATABASE_START_MAX_RETRIES=5 \
+DATABASE_RETRY_INTERVAL=10 \
+./planet.sh restart
+```
+
+常用参数:
+
+- `DEPENDENCY_INSTALL_MAX_RETRIES` / `DEPENDENCY_INSTALL_RETRY_INTERVAL`: 控制 `uv sync`、`bun install` 的重试次数与间隔，默认 `3` 次、`5` 秒
+- `DATABASE_START_MAX_RETRIES` / `DATABASE_RETRY_INTERVAL`: 控制 `postgres`、`redis` 的启动/重启与健康检查自愈，默认 `3` 次、`5` 秒
+- `AI_PROVIDER_START_MAX_RETRIES` / `AI_PROVIDER_RETRY_INTERVAL`: 控制 `aiprovider` 的构建/启动与容器重启自愈，默认 `3` 次、`5` 秒
+- `BACKEND_MAX_RETRIES`: 控制后端进程启动重试次数，默认 `3`
+- `FRONTEND_MAX_RETRIES`: 控制前端 dev server 启动重试次数，默认 `3`
+- `BACKEND_HEALTH_CHECK_ATTEMPTS` / `BACKEND_HEALTH_CHECK_INTERVAL`: 控制后端 HTTP 健康检查等待次数与间隔，默认 `10` 次、`2` 秒
+- `FRONTEND_HEALTH_CHECK_ATTEMPTS` / `FRONTEND_HEALTH_CHECK_INTERVAL`: 控制前端 HTTP 可访问检查等待次数与间隔，默认 `10` 次、`2` 秒
+- `AI_PROVIDER_HEALTH_CHECK_ATTEMPTS` / `AI_PROVIDER_HEALTH_CHECK_INTERVAL`: 控制 `aiprovider` HTTP 健康检查等待次数与间隔，默认 `10` 次、`2` 秒
+
+## AI 接口预留
+
+项目现在采用“两层”设计:
+
+- 主后端暴露稳定业务接口: `GET /api/v1/ai/provider/status`、`POST /api/v1/ai/situational-awareness/analyze`
+- 独立 `aiprovider` 服务负责适配具体模型供应商
+
+这样前端和业务代码不直接依赖 OpenAI、本地模型网关或其他订阅服务，后续切换部署方式只需要调整环境变量。
+
+主后端建议配置:
+
+```env
+AI_PROVIDER_SERVICE_URL=http://localhost:8010
+AI_PROVIDER_SERVICE_TOKEN=change_me
+AI_PROVIDER_TIMEOUT_SECONDS=60
+```
+
+`aiprovider` 服务建议配置:
+
+```env
+AI_PROVIDER=openai_compatible
+AI_BASE_URL=https://api.openai.com/v1
+AI_API_KEY=your_api_key
+AI_MODEL=gpt-4o-mini
+AI_TIMEOUT_SECONDS=60
+AI_PROVIDER_SERVICE_TOKEN=change_me
+```
+
+OpenAI 兼容场景推荐使用:
+
+- `AI_PROVIDER=openai_compatible`
+
+Claude 兼容场景推荐使用:
+
+- `AI_PROVIDER=anthropic`
+- `AI_PROVIDER=anthropic_compatible`
+- `AI_PROVIDER=claude_compatible`
+
+Ollama 原生场景推荐使用:
+
+- `AI_PROVIDER=ollama`
+
+比如 MiniMax 或其他 Claude 兼容网关，可以这样配置:
+
+```env
+AI_PROVIDER=claude_compatible
+AI_BASE_URL=https://your-claude-compatible-endpoint.example.com
+AI_API_KEY=your_api_key
+AI_MODEL=your-claude-compatible-model
+AI_TIMEOUT_SECONDS=60
+AI_MAX_TOKENS=1200
+AI_ANTHROPIC_VERSION=2023-06-01
+AI_PROVIDER_SERVICE_TOKEN=change_me
+```
+
+如果你要本地直接起模型适配层，项目里已经补了模板:
+
+- [aiprovider/.env.example](/home/ray/dev/linkong/planet/aiprovider/.env.example)
+- [docker-compose.local-model.yml](/home/ray/dev/linkong/planet/docker-compose.local-model.yml)
+
+推荐映射关系:
+
+- `vLLM` / `LM Studio` / `One API`: `AI_PROVIDER=openai_compatible`
+- `MiniMax` / Claude 兼容网关: `AI_PROVIDER=claude_compatible`
+- `Ollama`: `AI_PROVIDER=ollama`
+
+运行与调用补充:
+
+- `./planet.sh start` 默认会启动 `aiprovider`
+- 其他服务优先调用主后端 `POST /api/v1/ai/situational-awareness/analyze`
+- `backend -> aiprovider` 会透传 `X-Request-ID`
+- `backend -> aiprovider` 与 `aiprovider -> 模型供应商` 都带轻量重试
+
+详细文档:
+
+- [docs/aiprovider.md](/home/ray/dev/linkong/planet/docs/aiprovider.md)
+- [aiprovider/README.md](/home/ray/dev/linkong/planet/aiprovider/README.md)
+- [docs/frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/frontend-layout-guidelines.md)
+- [docs/ai-playground-development-plan.md](/home/ray/dev/linkong/planet/docs/ai-playground-development-plan.md)
+- [docs/situational-awareness-foundation-plan.md](/home/ray/dev/linkong/planet/docs/situational-awareness-foundation-plan.md)
+
+## 前端页面布局规范
+
+管理后台页面默认遵循“单屏工作区”原则：
+
+- 页头、摘要区、主工作区应在一屏内形成稳定结构
+- 主表格 / 主图表 / 主分析区应占据页面主要可视空间
+- 模块内容超出时优先在卡片、表格、标签页内部滚动
+- 不依赖整页纵向撑开来容纳主要工作区
+
+当前推荐参考实现：
+
+- [frontend/src/pages/BGP/BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx)
+- [docs/frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/frontend-layout-guidelines.md)
+
 ## License
 
 待定

TODO.md

@@ -0,0 +1,22 @@
+# TODO
+
+- [x] 把 BGP 观测站和异常点的 `hover/click` 手感再磨细一点
+- [x] 开始做 BGP 异常和海缆/区域的关联展示
+- [x] 做 Earth 侧的 `BGP activity layer`，让低 incident 密度时地图仍然有持续可感知的观测存在感
+- [x] 给 Earth BGP 补三层状态表达：`平稳观测态 / 局部波动态 / 事件活跃态`
+- [x] 把“当前无活跃事件”改造成“观测网络仍在运行、当前未发现聚合级事件”的状态表达
+- [x] 做 collector / region 近 15 分钟 activity score 聚合接口或动态聚合逻辑
+- [x] 把 Earth 的 BGP incident 改成 `紧凑事件核 + 向外扩张环形 pulse`，替换当前大面积 glow
+- [x] 为 BGP incident 建立符号系统：按事件类型用不同 marker，而不是都用同一种亮点
+- [x] 把 incident 地理定位从 `collector-centric` 改成 `prefix-centric`，优先使用 `prefix_geography`，其次 `prefix_scope`，再次 ASN 区域，最后才回退到观测区域质心
+- [x] 新增 `prefix_geography` 数据层，不再把 `prefix_scope` 当成 prefix 地理归属本身
+- [x] 接入 `IPtoASN / IPtoCountry` 作为 prefix-centric geography 的主数据源
+- [x] 接入 `OpenGeoFeed` 作为 prefix geography 的高质量覆盖/override 数据源
+- [x] 把 RIR delegated 设计成 prefix geography 的 fallback，而不是主来源
+- [ ] 为 `aiprovider` 建立 `provider -> api adapter -> compat policy` 的配置中心，优先落成 `json` 或 `yaml` 文件，运行时按 `provider/model` 读取兼容设置，而不是把专项兼容继续散落在 Python 分支里
+- [ ] 为市面上主流 AI 服务补专项兼容配置并固化到配置文件中，至少覆盖 `OpenAI / Anthropic / MiniMax / Ollama / Moonshot / DeepSeek / Qwen / GLM / Gemini / OpenRouter / vLLM / LM Studio / One API`
+- [ ] 在兼容配置中补齐可声明项：`api adapter`、`base_url pattern`、`auth header`、`thinking default`、`reasoning block mapping`、`stream path`、`tool-call capability`、`multimodal capability`、`provider-specific request patch`
+- [ ] 接入 `inetnum` / `inet6num` whois 作为比 RIR 更细粒度的后备层
+- [x] 在 activity layer 之后继续补 `route leak` 和 `path instability / flap` detector
+- [ ] 对 [frontend/public/earth/js/bgp.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp.js) 做按职责拆分的小重构，拆成 data / markers / overlays / animation，降低后续维护复杂度
+- [ ] 可选优化（非必做）：将 BGP incident/collector 标点改为 HTML marker（参考 worldmonitor 的 `htmlElementsData` 思路），实现近乎固定屏幕尺寸与更高密度可点击性

VERSION

@@ -0,0 +1 @@
+0.27.6

aiprovider/.env.example

@@ -0,0 +1,56 @@
+# Shared service settings
+SERVICE_NAME=planet-ai-provider
+SERVICE_VERSION=0.1.0
+AI_PROVIDER_SERVICE_TOKEN=change_me
+AI_TIMEOUT_SECONDS=60
+AI_HTTP_RETRY_ATTEMPTS=2
+AI_ANALYSIS_SYSTEM_PROMPT=你是态势感知分析助手。请基于输入的上下文、观测与约束，输出结构化、克制、可执行的分析。
+
+# Provider identity. Recommended values:
+# - minimax
+# - openai
+# - ollama
+# Compatibility aliases still accepted:
+# - openai_compatible
+# - anthropic_compatible
+# - claude_compatible
+AI_PROVIDER=minimax
+
+# Request adapter style, following OpenClaw's API-seam pattern:
+# - auto
+# - openai-completions
+# - anthropic-messages
+# - ollama-generate
+AI_PROVIDER_API=anthropic-messages
+
+# Common model selection
+AI_MODEL=MiniMax-M2.7
+
+# MiniMax CN Anthropic-compatible example
+AI_BASE_URL=https://api.minimaxi.com/anthropic
+AI_API_KEY=sk-cp-change-me
+AI_MAX_TOKENS=1200
+AI_ANTHROPIC_VERSION=2023-06-01
+
+# OpenAI-compatible example (vLLM / LM Studio / One API / local gateway)
+# AI_PROVIDER=openai
+# AI_PROVIDER_API=openai-completions
+# AI_BASE_URL=http://127.0.0.1:8001/v1
+# AI_API_KEY=local-key
+# AI_MODEL=your-local-model
+
+# Anthropic-compatible example (Claude-compatible gateway)
+# AI_PROVIDER=anthropic
+# AI_PROVIDER_API=anthropic-messages
+# AI_BASE_URL=http://127.0.0.1:8002/anthropic
+# AI_API_KEY=local-key
+# AI_MODEL=your-model
+# AI_MAX_TOKENS=1200
+# AI_ANTHROPIC_VERSION=2023-06-01
+
+# Ollama native example
+# AI_PROVIDER=ollama
+# AI_PROVIDER_API=ollama-generate
+# AI_BASE_URL=http://127.0.0.1:11434
+# AI_API_KEY=
+# AI_MODEL=qwen2.5:7b

aiprovider/Dockerfile

@@ -0,0 +1,23 @@
+FROM python:3.14-slim
+
+COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
+
+WORKDIR /app
+
+ENV PYTHONDONTWRITEBYTECODE=1
+ENV PYTHONUNBUFFERED=1
+ENV UV_COMPILE_BYTECODE=1
+ENV UV_LINK_MODE=copy
+
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    curl \
+    && rm -rf /var/lib/apt/lists/*
+
+COPY pyproject.toml uv.lock /app/
+RUN uv sync --frozen --no-dev
+
+COPY . /app
+
+EXPOSE 8010
+
+CMD ["uv", "run", "--frozen", "--no-dev", "--project", "/app", "python", "-m", "uvicorn", "aiprovider.main:app", "--host", "0.0.0.0", "--port", "8010", "--reload"]

aiprovider/README.md

@@ -0,0 +1,95 @@
+# AI Provider Service
+
+`aiprovider` 是独立的模型适配服务，负责把项目内部的分析请求转发到具体的大模型供应商。
+
+完整使用说明见:
+
+- [docs/aiprovider.md](/home/ray/dev/linkong/planet/docs/aiprovider.md)
+
+当前支持:
+
+- provider identity:
+  - `AI_PROVIDER=openai`
+  - `AI_PROVIDER=anthropic`
+  - `AI_PROVIDER=minimax`
+  - `AI_PROVIDER=ollama`
+- request adapter:
+  - `AI_PROVIDER_API=openai-completions`
+  - `AI_PROVIDER_API=anthropic-messages`
+  - `AI_PROVIDER_API=ollama-generate`
+
+兼容别名仍然保留:
+
+- `openai_compatible`
+- `anthropic_compatible`
+- `claude_compatible`
+
+典型配置:
+
+```env
+AI_PROVIDER=openai
+AI_PROVIDER_API=openai-completions
+AI_BASE_URL=https://api.openai.com/v1
+AI_API_KEY=your_api_key
+AI_MODEL=gpt-4o-mini
+AI_TIMEOUT_SECONDS=60
+AI_PROVIDER_SERVICE_TOKEN=change_me
+```
+
+MiniMax 中国大陆节点示例:
+
+```env
+AI_PROVIDER=minimax
+AI_PROVIDER_API=anthropic-messages
+AI_BASE_URL=https://api.minimaxi.com/anthropic
+AI_API_KEY=sk-cp-xxxxx
+AI_MODEL=MiniMax-M2.7
+AI_TIMEOUT_SECONDS=60
+AI_MAX_TOKENS=1200
+AI_ANTHROPIC_VERSION=2023-06-01
+AI_PROVIDER_SERVICE_TOKEN=change_me
+```
+
+适用场景:
+
+- Anthropic 官方 Claude API
+- Claude 兼容网关
+- MiniMax 等提供 Anthropic Messages 风格接口的服务
+
+这套命名方式参考了 OpenClaw 的接入模式: provider 负责标识供应商, `AI_PROVIDER_API` 负责标识协议适配层, 避免把“供应商”和“协议”绑死在一起。
+
+Ollama 原生示例:
+
+```env
+AI_PROVIDER=ollama
+AI_PROVIDER_API=ollama-generate
+AI_BASE_URL=http://127.0.0.1:11434
+AI_API_KEY=
+AI_MODEL=qwen2.5:7b
+AI_TIMEOUT_SECONDS=60
+AI_PROVIDER_SERVICE_TOKEN=change_me
+```
+
+本地模型接入建议:
+
+- `vLLM`、`LM Studio`、`One API`：`AI_PROVIDER=openai` + `AI_PROVIDER_API=openai-completions`
+- `MiniMax`、Claude 兼容网关：`AI_PROVIDER=minimax|anthropic` + `AI_PROVIDER_API=anthropic-messages`
+- `Ollama`：可直接使用 `ollama`
+
+启动模板:
+
+- `aiprovider/.env.example`
+- `docker-compose.local-model.yml`
+
+跨服务调用补充:
+
+- 业务服务优先调用主后端 `/api/v1/ai/...`
+- 直接调用 `aiprovider` 时使用 `X-Provider-Token`
+- 支持 `X-Request-ID` 透传
+- 内置轻量重试，适合跨机器 HTTP RPC 场景
+
+接口:
+
+- `GET /health`
+- `GET /v1/provider/status`
+- `POST /v1/analyze`

aiprovider/__init__.py

@@ -0,0 +1 @@
+"""AI provider adapter service package."""

aiprovider/config.py

@@ -0,0 +1,36 @@
+from functools import lru_cache
+from pathlib import Path
+
+from pydantic_settings import BaseSettings
+
+
+class Settings(BaseSettings):
+    SERVICE_NAME: str = "planet-ai-provider"
+    SERVICE_VERSION: str = "0.1.0"
+
+    AI_PROVIDER: str = "disabled"
+    AI_PROVIDER_API: str = "auto"
+    AI_BASE_URL: str = "https://api.openai.com/v1"
+    AI_API_KEY: str = ""
+    AI_MODEL: str = ""
+    AI_TIMEOUT_SECONDS: int = 60
+    AI_HTTP_RETRY_ATTEMPTS: int = 2
+    AI_MAX_TOKENS: int = 1200
+    AI_ANTHROPIC_VERSION: str = "2023-06-01"
+    AI_ANALYSIS_SYSTEM_PROMPT: str = (
+        "你是态势感知分析助手。请基于输入的上下文、观测与约束，输出结构化、克制、可执行的分析。"
+    )
+
+    AI_PROVIDER_SERVICE_TOKEN: str = ""
+
+    class Config:
+        env_file = Path(__file__).parent / ".env"
+        case_sensitive = True
+
+
+@lru_cache()
+def get_settings() -> Settings:
+    return Settings()
+
+
+settings = get_settings()

aiprovider/main.py

@@ -0,0 +1,79 @@
+from uuid import uuid4
+
+from fastapi import Depends, FastAPI, Header, HTTPException, Request, Response, status
+
+from aiprovider.config import settings
+from aiprovider.provider_service import ProviderService
+from aiprovider.schemas import (
+    AIProviderStatusResponse,
+    SituationalAnalysisRequest,
+    SituationalAnalysisResponse,
+)
+
+app = FastAPI(
+    title=settings.SERVICE_NAME,
+    version=settings.SERVICE_VERSION,
+    description="AI provider adapter service for Planet",
+)
+
+
+@app.middleware("http")
+async def request_id_middleware(request: Request, call_next):
+    request_id = request.headers.get("X-Request-ID") or str(uuid4())
+    request.state.request_id = request_id
+    response = await call_next(request)
+    response.headers["X-Request-ID"] = request_id
+    return response
+
+
+def verify_service_token(x_provider_token: str | None = Header(default=None)) -> None:
+    expected = settings.AI_PROVIDER_SERVICE_TOKEN
+    if not expected:
+        return
+    if x_provider_token != expected:
+        raise HTTPException(
+            status_code=status.HTTP_401_UNAUTHORIZED,
+            detail="Invalid provider service token",
+        )
+
+
+def get_provider_service() -> ProviderService:
+    return ProviderService()
+
+
+@app.get("/health")
+async def health_check():
+    return {
+        "status": "healthy",
+        "service": settings.SERVICE_NAME,
+        "version": settings.SERVICE_VERSION,
+    }
+
+
+@app.get(
+    "/v1/provider/status",
+    response_model=AIProviderStatusResponse,
+    dependencies=[Depends(verify_service_token)],
+)
+async def get_provider_status(
+    response: Response,
+    request: Request,
+    provider_service: ProviderService = Depends(get_provider_service),
+):
+    response.headers["X-Request-ID"] = request.state.request_id
+    return provider_service.get_status()
+
+
+@app.post(
+    "/v1/analyze",
+    response_model=SituationalAnalysisResponse,
+    dependencies=[Depends(verify_service_token)],
+)
+async def analyze(
+    payload: SituationalAnalysisRequest,
+    response: Response,
+    request: Request,
+    provider_service: ProviderService = Depends(get_provider_service),
+):
+    response.headers["X-Request-ID"] = request.state.request_id
+    return await provider_service.analyze(payload)

aiprovider/provider_service.py

@@ -0,0 +1,372 @@
+from __future__ import annotations
+
+import asyncio
+from typing import Any
+
+import httpx
+from fastapi import HTTPException, status
+
+from aiprovider.config import settings
+from aiprovider.schemas import (
+    AIContentBlock,
+    AIProviderStatusResponse,
+    SituationalAnalysisRequest,
+    SituationalAnalysisResponse,
+)
+
+
+def _normalize_provider(value: str) -> str:
+    return (value or "disabled").strip().lower()
+
+
+def _normalize_provider_api(value: str) -> str:
+    return (value or "auto").strip().lower().replace("_", "-")
+
+
+def _resolve_provider_api(provider: str, configured_api: str) -> str:
+    if configured_api and configured_api != "auto":
+        return configured_api
+
+    if provider in {"openai", "openai-compatible", "openai_compatible"}:
+        return "openai-completions"
+    if provider in {
+        "anthropic",
+        "anthropic-compatible",
+        "anthropic_compatible",
+        "claude-compatible",
+        "claude_compatible",
+        "minimax",
+        "kimi-coding",
+        "moonshot-anthropic",
+    }:
+        return "anthropic-messages"
+    if provider == "ollama":
+        return "ollama-generate"
+    return "disabled"
+
+
+class ProviderService:
+    def __init__(self) -> None:
+        self.provider = _normalize_provider(settings.AI_PROVIDER)
+        self.provider_api = _resolve_provider_api(
+            self.provider,
+            _normalize_provider_api(settings.AI_PROVIDER_API),
+        )
+        self.base_url = settings.AI_BASE_URL.rstrip("/")
+        self.api_key = settings.AI_API_KEY
+        self.default_model = settings.AI_MODEL
+        self.timeout = settings.AI_TIMEOUT_SECONDS
+        self.http_retry_attempts = max(settings.AI_HTTP_RETRY_ATTEMPTS, 1)
+        self.max_tokens = settings.AI_MAX_TOKENS
+        self.anthropic_version = settings.AI_ANTHROPIC_VERSION
+        self.system_prompt = settings.AI_ANALYSIS_SYSTEM_PROMPT
+
+    def get_status(self) -> AIProviderStatusResponse:
+        enabled = self.provider != "disabled"
+        has_credentials = bool(self.api_key) if self._requires_api_key() else True
+        configured = enabled and bool(self.base_url and has_credentials and self.default_model)
+        return AIProviderStatusResponse(
+            provider=self.provider,
+            api=self.provider_api if enabled else None,
+            enabled=enabled,
+            configured=configured,
+            model=self.default_model or None,
+            base_url=self.base_url if enabled else None,
+        )
+
+    async def analyze(self, payload: SituationalAnalysisRequest) -> SituationalAnalysisResponse:
+        if self.provider == "disabled":
+            raise HTTPException(
+                status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
+                detail="AI provider is disabled. Configure AI_PROVIDER in .env to enable analysis.",
+            )
+
+        model = payload.preferred_model or self.default_model
+        has_credentials = bool(self.api_key) if self._requires_api_key() else True
+        if not self.base_url or not has_credentials or not model:
+            raise HTTPException(
+                status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
+                detail="AI provider is not fully configured. Check AI_BASE_URL, AI_API_KEY, and AI_MODEL.",
+            )
+
+        prompt = self._build_prompt(payload)
+
+        if self.provider_api == "openai-completions":
+            data = await self._request_openai_compatible(model, prompt)
+            content = self._extract_openai_content(data)
+            content_blocks = self._extract_openai_blocks(data)
+        elif self.provider_api == "anthropic-messages":
+            data = await self._request_anthropic_messages(model, prompt, payload.thinking)
+            content = self._extract_anthropic_content(data)
+            content_blocks = self._extract_anthropic_blocks(data)
+        elif self.provider_api == "ollama-generate":
+            data = await self._request_ollama(model, prompt)
+            content = self._extract_ollama_content(data)
+            content_blocks = self._extract_ollama_blocks(data)
+        else:
+            raise HTTPException(
+                status_code=status.HTTP_400_BAD_REQUEST,
+                detail=f"Unsupported AI provider API: {self.provider_api}",
+            )
+
+        text_blocks = [block.text for block in content_blocks if block.text]
+        thinking_blocks = [block.thinking for block in content_blocks if block.thinking]
+
+        return SituationalAnalysisResponse(
+            provider=self.provider,
+            model=model,
+            content=content,
+            content_blocks=content_blocks,
+            text_blocks=text_blocks,
+            thinking_blocks=thinking_blocks,
+            raw_response=data,
+        )
+
+    def _requires_api_key(self) -> bool:
+        return self.provider_api != "ollama-generate"
+
+    def _build_prompt(self, payload: SituationalAnalysisRequest) -> str:
+        sections = [
+            f"任务标题:\n{payload.title}",
+            f"分析目标:\n{payload.objective}",
+        ]
+        if payload.observations:
+            sections.append("观测事实:\n" + "\n".join(f"- {item}" for item in payload.observations))
+        if payload.constraints:
+            sections.append("约束条件:\n" + "\n".join(f"- {item}" for item in payload.constraints))
+        if payload.context:
+            sections.append(f"附加上下文:\n{payload.context}")
+        sections.append(
+            "请输出: 1) 态势摘要 2) 关键风险 3) 研判依据 4) 建议动作 5) 还缺少的数据。"
+        )
+        return "\n\n".join(sections)
+
+    async def _request_openai_compatible(self, model: str, prompt: str) -> dict[str, Any]:
+        request_body = {
+            "model": model,
+            "messages": [
+                {"role": "system", "content": self.system_prompt},
+                {"role": "user", "content": prompt},
+            ],
+            "temperature": 0.2,
+        }
+        return await self._post(
+            path="/chat/completions",
+            headers={
+                "Authorization": f"Bearer {self.api_key}",
+                "Content-Type": "application/json",
+            },
+            request_body=request_body,
+        )
+
+    async def _request_anthropic_messages(
+        self,
+        model: str,
+        prompt: str,
+        thinking: dict[str, Any] | None = None,
+    ) -> dict[str, Any]:
+        request_body = {
+            "model": model,
+            "system": self.system_prompt,
+            "messages": [
+                {
+                    "role": "user",
+                    "content": [
+                        {
+                            "type": "text",
+                            "text": prompt,
+                        }
+                    ],
+                }
+            ],
+            "max_tokens": self.max_tokens,
+            "temperature": 0.2,
+        }
+        resolved_thinking = self._resolve_anthropic_thinking(thinking)
+        if resolved_thinking:
+            request_body["thinking"] = resolved_thinking
+        if self.provider == "minimax" and self.base_url.endswith("/anthropic"):
+            path = "/v1/messages"
+        else:
+            path = "/messages"
+        return await self._post(
+            path=path,
+            headers={
+                "x-api-key": self.api_key,
+                "anthropic-version": self.anthropic_version,
+                "Content-Type": "application/json",
+            },
+            request_body=request_body,
+        )
+
+    def _resolve_anthropic_thinking(self, thinking: dict[str, Any] | None) -> dict[str, Any] | None:
+        if thinking:
+            return thinking
+
+        # OpenClaw treats MiniMax's Anthropic-compatible path specially:
+        # disable thinking by default unless the caller explicitly opts in.
+        if self.provider == "minimax":
+            return {"type": "disabled"}
+
+        return None
+
+    async def _request_anthropic_compatible(
+        self,
+        model: str,
+        prompt: str,
+        thinking: dict[str, Any] | None = None,
+    ) -> dict[str, Any]:
+        return await self._request_anthropic_messages(model, prompt, thinking)
+
+    async def _request_ollama(self, model: str, prompt: str) -> dict[str, Any]:
+        request_body = {
+            "model": model,
+            "stream": False,
+            "system": self.system_prompt,
+            "prompt": prompt,
+            "options": {
+                "temperature": 0.2,
+            },
+        }
+        return await self._post(
+            path="/api/generate",
+            headers={
+                "Content-Type": "application/json",
+            },
+            request_body=request_body,
+        )
+
+    async def _post(
+        self,
+        path: str,
+        headers: dict[str, str],
+        request_body: dict[str, Any],
+    ) -> dict[str, Any]:
+        last_error: Exception | None = None
+        for attempt in range(1, self.http_retry_attempts + 1):
+            try:
+                async with httpx.AsyncClient(timeout=self.timeout) as client:
+                    response = await client.post(
+                        f"{self.base_url}{path}",
+                        headers=headers,
+                        json=request_body,
+                    )
+                    response.raise_for_status()
+                    return response.json()
+            except httpx.HTTPStatusError as exc:
+                last_error = exc
+                if attempt < self.http_retry_attempts and exc.response.status_code >= 500:
+                    await asyncio.sleep(0.3 * attempt)
+                    continue
+                detail = exc.response.text or "AI provider returned an error"
+                raise HTTPException(
+                    status_code=status.HTTP_502_BAD_GATEWAY,
+                    detail=f"AI provider request failed: {detail}",
+                ) from exc
+            except httpx.HTTPError as exc:
+                last_error = exc
+                if attempt < self.http_retry_attempts:
+                    await asyncio.sleep(0.3 * attempt)
+                    continue
+                raise HTTPException(
+                    status_code=status.HTTP_502_BAD_GATEWAY,
+                    detail=f"Failed to reach AI provider: {exc}",
+                ) from exc
+
+        raise HTTPException(
+            status_code=status.HTTP_502_BAD_GATEWAY,
+            detail=f"AI provider request failed: {last_error}",
+        )
+
+    def _extract_openai_content(self, payload: dict[str, Any]) -> str:
+        choices = payload.get("choices") or []
+        if not choices:
+            return ""
+
+        message = choices[0].get("message") or {}
+        content = message.get("content")
+        if isinstance(content, str):
+            return content
+        if isinstance(content, list):
+            return "".join(
+                item.get("text", "")
+                for item in content
+                if isinstance(item, dict)
+            )
+        return ""
+
+    def _extract_openai_blocks(self, payload: dict[str, Any]) -> list[AIContentBlock]:
+        choices = payload.get("choices") or []
+        if not choices:
+            return []
+
+        message = choices[0].get("message") or {}
+        content = message.get("content")
+        if isinstance(content, str):
+            return [AIContentBlock(type="text", text=content)]
+        if not isinstance(content, list):
+            return []
+
+        blocks: list[AIContentBlock] = []
+        for item in content:
+            if not isinstance(item, dict):
+                continue
+            blocks.append(
+                AIContentBlock(
+                    type=str(item.get("type", "text")),
+                    text=item.get("text") if isinstance(item.get("text"), str) else None,
+                    metadata={k: v for k, v in item.items() if k not in {"type", "text"}},
+                )
+            )
+        return blocks
+    def _extract_anthropic_content(self, payload: dict[str, Any]) -> str:
+        content = payload.get("content")
+        if isinstance(content, str):
+            return content
+        if not isinstance(content, list):
+            return ""
+
+        fragments: list[str] = []
+        for item in content:
+            if not isinstance(item, dict):
+                continue
+            if item.get("type") == "text" and isinstance(item.get("text"), str):
+                fragments.append(item["text"])
+        return "".join(fragments)
+
+    def _extract_anthropic_blocks(self, payload: dict[str, Any]) -> list[AIContentBlock]:
+        content = payload.get("content")
+        if not isinstance(content, list):
+            return []
+
+        blocks: list[AIContentBlock] = []
+        for item in content:
+            if not isinstance(item, dict):
+                continue
+            blocks.append(
+                AIContentBlock(
+                    type=str(item.get("type", "unknown")),
+                    text=item.get("text") if isinstance(item.get("text"), str) else None,
+                    thinking=item.get("thinking") if isinstance(item.get("thinking"), str) else None,
+                    signature=item.get("signature") if isinstance(item.get("signature"), str) else None,
+                    metadata={
+                        k: v
+                        for k, v in item.items()
+                        if k not in {"type", "text", "thinking", "signature"}
+                    },
+                )
+            )
+        return blocks
+
+
+    def _extract_ollama_content(self, payload: dict[str, Any]) -> str:
+        response = payload.get("response")
+        if isinstance(response, str):
+            return response
+        return ""
+
+    def _extract_ollama_blocks(self, payload: dict[str, Any]) -> list[AIContentBlock]:
+        response = payload.get("response")
+        if isinstance(response, str) and response:
+            return [AIContentBlock(type="text", text=response)]
+        return []

aiprovider/schemas.py

@@ -0,0 +1,40 @@
+from typing import Any
+
+from pydantic import BaseModel, Field
+
+
+class AIContentBlock(BaseModel):
+    type: str
+    text: str | None = None
+    thinking: str | None = None
+    signature: str | None = None
+    metadata: dict[str, Any] = Field(default_factory=dict)
+
+
+class SituationalAnalysisRequest(BaseModel):
+    title: str = Field(..., min_length=1, max_length=200)
+    objective: str = Field(..., min_length=1, max_length=1000)
+    context: dict[str, Any] = Field(default_factory=dict)
+    observations: list[str] = Field(default_factory=list)
+    constraints: list[str] = Field(default_factory=list)
+    preferred_model: str | None = Field(default=None, max_length=200)
+    thinking: dict[str, Any] | None = None
+
+
+class SituationalAnalysisResponse(BaseModel):
+    provider: str
+    model: str
+    content: str
+    content_blocks: list[AIContentBlock] = Field(default_factory=list)
+    text_blocks: list[str] = Field(default_factory=list)
+    thinking_blocks: list[str] = Field(default_factory=list)
+    raw_response: dict[str, Any] = Field(default_factory=dict)
+
+
+class AIProviderStatusResponse(BaseModel):
+    provider: str
+    api: str | None = None
+    enabled: bool
+    configured: bool
+    model: str | None = None
+    base_url: str | None = None

backend/.env.example

@@ -1,23 +1,26 @@
-# Database
+PROJECT_NAME=Intelligent Planet Plan
+APP_VERSION=0.23.0
+
+SECRET_KEY=change_me_to_a_random_secret
+ALGORITHM=HS256
+ACCESS_TOKEN_EXPIRE_MINUTES=0
+REFRESH_TOKEN_EXPIRE_DAYS=0
+
 POSTGRES_SERVER=localhost
 POSTGRES_USER=postgres
 POSTGRES_PASSWORD=postgres
 POSTGRES_DB=planet_db
+DATABASE_URL=postgresql+asyncpg://postgres:postgres@localhost:5432/planet_db
 
-# Redis
 REDIS_SERVER=localhost
 REDIS_PORT=6379
+REDIS_DB=0
+REDIS_URL=redis://localhost:6379/0
 
-# Security
-SECRET_KEY=your-secret-key-change-in-production
-ALGORITHM=HS256
-ACCESS_TOKEN_EXPIRE_MINUTES=15
-REFRESH_TOKEN_EXPIRE_DAYS=7
+AI_PROVIDER_SERVICE_URL=http://localhost:8010
+AI_PROVIDER_SERVICE_TOKEN=change_me
+AI_PROVIDER_TIMEOUT_SECONDS=60
+AI_PROVIDER_RETRY_ATTEMPTS=2
 
-# API
-API_V1_STR=/api/v1
-PROJECT_NAME="Intelligent Planet Plan"
-VERSION=1.0.0
-
-# CORS
-CORS_ORIGINS=["http://localhost:3000", "http://localhost:8000"]
+SPACETRACK_USERNAME=
+SPACETRACK_PASSWORD=

backend/Dockerfile

@@ -1,19 +1,24 @@
-FROM python:3.11-slim
+FROM python:3.14-slim
+
+COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
 
 WORKDIR /app
 
 ENV PYTHONDONTWRITEBYTECODE=1
 ENV PYTHONUNBUFFERED=1
+ENV UV_COMPILE_BYTECODE=1
+ENV UV_LINK_MODE=copy
 
 RUN apt-get update && apt-get install -y --no-install-recommends \
     curl \
     && rm -rf /var/lib/apt/lists/*
 
-COPY requirements.txt .
-RUN pip install --no-cache-dir -r requirements.txt
+COPY pyproject.toml uv.lock /app/
+RUN uv sync --frozen --no-dev
 
-COPY . .
+COPY backend /app/backend
+COPY VERSION /app/VERSION
 
 EXPOSE 8000
 
-CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000", "--reload"]
+CMD ["uv", "run", "--frozen", "--no-dev", "--project", "/app", "python", "-m", "uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000", "--reload"]

backend/app/api/main.py

@@ -1,5 +1,6 @@
 from fastapi import APIRouter
 from app.api.v1 import (
+    ai,
     auth,
     users,
     datasource_config,
@@ -11,11 +12,16 @@ from app.api.v1 import (
     settings,
     collected_data,
     visualization,
+    bgp,
+    system_control,
+    tv,
+    ue_data,
 )
 
 api_router = APIRouter()
 
 api_router.include_router(auth.router, prefix="/auth", tags=["auth"])
+api_router.include_router(ai.router, prefix="/ai", tags=["ai"])
 api_router.include_router(users.router, prefix="/users", tags=["users"])
 api_router.include_router(
     datasource_config.router, prefix="/datasources", tags=["datasource-config"]
@@ -26,4 +32,8 @@ api_router.include_router(tasks.router, prefix="/tasks", tags=["tasks"])
 api_router.include_router(dashboard.router, prefix="/dashboard", tags=["dashboard"])
 api_router.include_router(alerts.router, prefix="/alerts", tags=["alerts"])
 api_router.include_router(settings.router, prefix="/settings", tags=["settings"])
+api_router.include_router(system_control.router, prefix="/system", tags=["system"])
 api_router.include_router(visualization.router, prefix="/visualization", tags=["visualization"])
+api_router.include_router(bgp.router, prefix="/bgp", tags=["bgp"])
+api_router.include_router(tv.router, prefix="/tv", tags=["tv"])
+api_router.include_router(ue_data.router, prefix="/ue", tags=["ue-client"])

backend/app/api/v1/ai.py

@@ -0,0 +1,281 @@
+from uuid import uuid4
+
+from fastapi import APIRouter, Depends, HTTPException, Request, Response
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.core.security import get_current_user
+from app.db.session import get_db
+from app.models.user import User
+from app.schemas.ai import (
+    AIProviderStatusResponse,
+    AlertBriefRequest,
+    AlertBriefResponse,
+    BGPBriefRequest,
+    BGPBriefRecordResponse,
+    BGPBriefRecordSummary,
+    PlaygroundMessageActionResponse,
+    PlaygroundMessageCreateRequest,
+    PlaygroundMessageEditRequest,
+    PlaygroundMessageResendRequest,
+    PlaygroundMessageStopRequest,
+    PlaygroundSessionResponse,
+    PlaygroundSessionUpsertRequest,
+    PlaygroundThreadResponse,
+    SituationalAlertBriefRequest,
+    SituationalAlertBriefResponse,
+    SituationalAnalysisRequest,
+    SituationalAnalysisResponse,
+)
+from app.services.alert_ai_brief import build_alert_brief_request
+from app.services.ai_client import AIProviderClient, get_ai_provider_client
+from app.services.bgp_ai_brief import build_bgp_brief_request
+from app.services.bgp_ai_brief_store import (
+    get_bgp_brief_record,
+    get_latest_bgp_brief_record,
+    list_bgp_brief_records,
+    save_bgp_brief_record,
+)
+from app.services.playground_session_store import (
+    get_playground_session,
+    upsert_playground_session,
+)
+from app.services.playground_chat_service import (
+    create_turn,
+    edit_user_message,
+    get_thread,
+    resend_turn,
+    stop_message,
+)
+from app.services.situational_alert_ai_brief import build_situational_alert_brief_request
+
+router = APIRouter()
+
+
+@router.get("/provider/status", response_model=AIProviderStatusResponse)
+async def get_ai_provider_status(
+    request: Request,
+    response: Response,
+    current_user: User = Depends(get_current_user),
+    provider_client: AIProviderClient = Depends(get_ai_provider_client),
+):
+    request_id = request.headers.get("X-Request-ID") or str(uuid4())
+    response.headers["X-Request-ID"] = request_id
+    return await provider_client.get_status(request_id=request_id)
+
+
+@router.post("/situational-awareness/analyze", response_model=SituationalAnalysisResponse)
+async def analyze_situational_awareness(
+    payload: SituationalAnalysisRequest,
+    request: Request,
+    response: Response,
+    current_user: User = Depends(get_current_user),
+    provider_client: AIProviderClient = Depends(get_ai_provider_client),
+):
+    request_id = request.headers.get("X-Request-ID") or str(uuid4())
+    response.headers["X-Request-ID"] = request_id
+    return await provider_client.analyze(payload, request_id=request_id)
+
+
+@router.get("/playground/thread", response_model=PlaygroundThreadResponse | None)
+async def get_playground_thread(
+    session_key: str = "default",
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    return await get_thread(
+        db,
+        user_id=current_user.id,
+        session_key=session_key,
+    )
+
+
+@router.get("/playground/session", response_model=PlaygroundSessionResponse | None)
+async def get_saved_playground_session(
+    session_key: str = "default",
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    return await get_playground_session(
+        db,
+        user_id=current_user.id,
+        session_key=session_key,
+    )
+
+
+@router.put("/playground/session", response_model=PlaygroundSessionResponse)
+async def save_playground_session(
+    payload: PlaygroundSessionUpsertRequest,
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    return await upsert_playground_session(
+        db,
+        user_id=current_user.id,
+        payload=payload,
+    )
+
+
+@router.post("/playground/messages", response_model=PlaygroundMessageActionResponse)
+async def create_playground_message(
+    payload: PlaygroundMessageCreateRequest,
+    current_user: User = Depends(get_current_user),
+    provider_client: AIProviderClient = Depends(get_ai_provider_client),
+    db: AsyncSession = Depends(get_db),
+):
+    return await create_turn(
+        db,
+        user_id=current_user.id,
+        payload=payload,
+        provider_client=provider_client,
+    )
+
+
+@router.post("/playground/messages/stop", response_model=PlaygroundMessageActionResponse)
+async def stop_playground_message(
+    payload: PlaygroundMessageStopRequest,
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    return await stop_message(
+        db,
+        user_id=current_user.id,
+        payload=payload,
+    )
+
+
+@router.post("/playground/messages/resend", response_model=PlaygroundMessageActionResponse)
+async def resend_playground_message(
+    payload: PlaygroundMessageResendRequest,
+    current_user: User = Depends(get_current_user),
+    provider_client: AIProviderClient = Depends(get_ai_provider_client),
+    db: AsyncSession = Depends(get_db),
+):
+    return await resend_turn(
+        db,
+        user_id=current_user.id,
+        payload=payload,
+        provider_client=provider_client,
+    )
+
+
+@router.post("/playground/messages/edit", response_model=PlaygroundMessageActionResponse)
+async def edit_playground_message(
+    payload: PlaygroundMessageEditRequest,
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    return await edit_user_message(
+        db,
+        user_id=current_user.id,
+        payload=payload,
+    )
+
+
+@router.get("/bgp/briefs", response_model=list[BGPBriefRecordSummary])
+async def list_saved_bgp_briefs(
+    current_user: User = Depends(get_current_user),
+):
+    return list_bgp_brief_records()
+
+
+@router.get("/bgp/briefs/latest", response_model=BGPBriefRecordResponse | None)
+async def get_latest_saved_bgp_brief(
+    current_user: User = Depends(get_current_user),
+):
+    return get_latest_bgp_brief_record()
+
+
+@router.get("/bgp/briefs/{brief_id}", response_model=BGPBriefRecordResponse)
+async def get_saved_bgp_brief(
+    brief_id: str,
+    current_user: User = Depends(get_current_user),
+):
+    record = get_bgp_brief_record(brief_id)
+    if record is None:
+        raise HTTPException(status_code=404, detail="BGP brief not found")
+    return record
+
+
+@router.post("/bgp/brief", response_model=BGPBriefRecordResponse)
+async def analyze_bgp_brief(
+    payload: BGPBriefRequest,
+    request: Request,
+    response: Response,
+    current_user: User = Depends(get_current_user),
+    provider_client: AIProviderClient = Depends(get_ai_provider_client),
+    db: AsyncSession = Depends(get_db),
+):
+    request_id = request.headers.get("X-Request-ID") or str(uuid4())
+    response.headers["X-Request-ID"] = request_id
+
+    brief_request, facts, context = await build_bgp_brief_request(
+        db,
+        incident_limit=payload.incident_limit,
+        anomaly_limit=payload.anomaly_limit,
+        collector_limit=payload.collector_limit,
+    )
+    brief_request.preferred_model = payload.preferred_model
+    brief_request.thinking = payload.thinking
+
+    analysis = await provider_client.analyze(brief_request, request_id=request_id)
+    return save_bgp_brief_record(
+        analysis,
+        request_id=request_id,
+        facts=facts,
+        context=context,
+    )
+
+
+@router.post("/alerts/brief", response_model=AlertBriefResponse)
+async def analyze_alert_brief(
+    payload: AlertBriefRequest,
+    request: Request,
+    response: Response,
+    current_user: User = Depends(get_current_user),
+    provider_client: AIProviderClient = Depends(get_ai_provider_client),
+    db: AsyncSession = Depends(get_db),
+):
+    request_id = request.headers.get("X-Request-ID") or str(uuid4())
+    response.headers["X-Request-ID"] = request_id
+
+    brief_request, facts, context = await build_alert_brief_request(
+        db,
+        alert_limit=payload.alert_limit,
+    )
+    brief_request.preferred_model = payload.preferred_model
+    brief_request.thinking = payload.thinking
+
+    analysis = await provider_client.analyze(brief_request, request_id=request_id)
+    return AlertBriefResponse(
+        **analysis.model_dump(),
+        title=brief_request.title,
+        objective=brief_request.objective,
+        facts=facts,
+        context=context,
+    )
+
+
+@router.post("/situational-alerts/brief", response_model=SituationalAlertBriefResponse)
+async def analyze_situational_alert_brief(
+    payload: SituationalAlertBriefRequest,
+    request: Request,
+    response: Response,
+    current_user: User = Depends(get_current_user),
+    provider_client: AIProviderClient = Depends(get_ai_provider_client),
+    db: AsyncSession = Depends(get_db),
+):
+    request_id = request.headers.get("X-Request-ID") or str(uuid4())
+    response.headers["X-Request-ID"] = request_id
+
+    brief_request, facts, context = await build_situational_alert_brief_request(db)
+    brief_request.preferred_model = payload.preferred_model
+    brief_request.thinking = payload.thinking
+
+    analysis = await provider_client.analyze(brief_request, request_id=request_id)
+    return SituationalAlertBriefResponse(
+        **analysis.model_dump(),
+        title=brief_request.title,
+        objective=brief_request.objective,
+        facts=facts,
+        context=context,
+    )

backend/app/api/v1/alerts.py

@@ -1,7 +1,7 @@
-from datetime import datetime
+from datetime import UTC, datetime
 from typing import Optional
 
-from fastapi import APIRouter, Depends
+from fastapi import APIRouter, Depends, HTTPException
 from sqlalchemy import select, func, case
 from sqlalchemy.ext.asyncio import AsyncSession
 
@@ -9,6 +9,7 @@ from app.db.session import get_db
 from app.models.user import User
 from app.core.security import get_current_user
 from app.models.alert import Alert, AlertSeverity, AlertStatus
+from app.schemas.alert import AlertResolutionRequest
 
 
 router = APIRouter()
@@ -68,7 +69,7 @@ async def acknowledge_alert(
 
     alert.status = AlertStatus.ACKNOWLEDGED
     alert.acknowledged_by = current_user.id
-    alert.acknowledged_at = datetime.utcnow()
+    alert.acknowledged_at = datetime.now(UTC)
     await db.commit()
 
     return {"message": "Alert acknowledged", "alert": alert.to_dict()}
@@ -77,7 +78,7 @@ async def acknowledge_alert(
 @router.post("/{alert_id}/resolve")
 async def resolve_alert(
     alert_id: int,
-    resolution: str,
+    payload: AlertResolutionRequest,
     current_user: User = Depends(get_current_user),
     db: AsyncSession = Depends(get_db),
 ):
@@ -85,12 +86,12 @@ async def resolve_alert(
     alert = result.scalar_one_or_none()
 
     if not alert:
-        return {"error": "Alert not found"}
+        raise HTTPException(status_code=404, detail="Alert not found")
 
     alert.status = AlertStatus.RESOLVED
     alert.resolved_by = current_user.id
-    alert.resolved_at = datetime.utcnow()
-    alert.resolution_notes = resolution
+    alert.resolved_at = datetime.now(UTC)
+    alert.resolution_notes = payload.resolution
     await db.commit()
 
     return {"message": "Alert resolved", "alert": alert.to_dict()}
@@ -101,25 +102,44 @@ async def get_alert_stats(
     current_user: User = Depends(get_current_user),
     db: AsyncSession = Depends(get_db),
 ):
-    critical_query = select(func.count(Alert.id)).where(
-        Alert.severity == AlertSeverity.CRITICAL,
-        Alert.status == AlertStatus.ACTIVE,
+    result = await db.execute(
+        select(
+            func.sum(
+                case(
+                    (
+                        (Alert.severity == AlertSeverity.CRITICAL)
+                        & (Alert.status == AlertStatus.ACTIVE),
+                        1,
+                    ),
+                    else_=0,
+                )
+            ).label("critical"),
+            func.sum(
+                case(
+                    (
+                        (Alert.severity == AlertSeverity.WARNING)
+                        & (Alert.status == AlertStatus.ACTIVE),
+                        1,
+                    ),
+                    else_=0,
+                )
+            ).label("warning"),
+            func.sum(
+                case(
+                    (
+                        (Alert.severity == AlertSeverity.INFO)
+                        & (Alert.status == AlertStatus.ACTIVE),
+                        1,
+                    ),
+                    else_=0,
+                )
+            ).label("info"),
+        )
     )
-    warning_query = select(func.count(Alert.id)).where(
-        Alert.severity == AlertSeverity.WARNING,
-        Alert.status == AlertStatus.ACTIVE,
-    )
-    info_query = select(func.count(Alert.id)).where(
-        Alert.severity == AlertSeverity.INFO,
-        Alert.status == AlertStatus.ACTIVE,
-    )
-
-    critical_result = await db.execute(critical_query)
-    warning_result = await db.execute(warning_query)
-    info_result = await db.execute(info_query)
+    row = result.one()
 
     return {
-        "critical": critical_result.scalar() or 0,
-        "warning": warning_result.scalar() or 0,
-        "info": info_result.scalar() or 0,
+        "critical": row.critical or 0,
+        "warning": row.warning or 0,
+        "info": row.info or 0,
     }

backend/app/api/v1/bgp.py

@@ -0,0 +1,424 @@
+from datetime import datetime
+from typing import Optional
+
+from fastapi import APIRouter, Depends, HTTPException, Query
+from sqlalchemy import func, select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.core.security import get_current_user
+from app.db.session import get_db
+from app.models.bgp_anomaly import BGPAnomaly
+from app.models.bgp_incident import BGPIncident
+from app.models.bgp_observation import BGPObservation
+from app.models.user import User
+from app.services.bgp_collectors import build_bgp_collector_coverage
+
+router = APIRouter()
+
+BGP_SOURCES = ("ris_live_bgp", "bgpstream_bgp")
+
+
+def _parse_dt(value: Optional[str]) -> Optional[datetime]:
+    if not value:
+        return None
+    return datetime.fromisoformat(value.replace("Z", "+00:00"))
+def _event_filters(
+    *,
+    prefix: Optional[str],
+    origin_asn: Optional[int],
+    peer_asn: Optional[int],
+    collector: Optional[str],
+    event_type: Optional[str],
+    source: Optional[str],
+    time_from: Optional[datetime],
+    time_to: Optional[datetime],
+):
+    filters = [BGPObservation.source.in_(BGP_SOURCES)]
+    if source:
+        filters.append(BGPObservation.source == source)
+    if prefix:
+        filters.append(BGPObservation.prefix == prefix)
+    if origin_asn is not None:
+        filters.append(BGPObservation.origin_asn == origin_asn)
+    if peer_asn is not None:
+        filters.append(BGPObservation.peer_asn == peer_asn)
+    if collector:
+        filters.append(BGPObservation.collector == collector)
+    if event_type:
+        filters.append(BGPObservation.event_type == event_type)
+    if time_from:
+        filters.append(BGPObservation.observed_at >= time_from)
+    if time_to:
+        filters.append(BGPObservation.observed_at <= time_to)
+    return filters
+
+
+def _anomaly_filters(
+    *,
+    severity: Optional[str],
+    anomaly_type: Optional[str],
+    status: Optional[str],
+    prefix: Optional[str],
+    origin_asn: Optional[int],
+    time_from: Optional[datetime],
+    time_to: Optional[datetime],
+):
+    filters = []
+    if severity:
+        filters.append(BGPAnomaly.severity == severity)
+    if anomaly_type:
+        filters.append(BGPAnomaly.anomaly_type == anomaly_type)
+    if status:
+        filters.append(BGPAnomaly.status == status)
+    if prefix:
+        filters.append(BGPAnomaly.prefix == prefix)
+    if origin_asn is not None:
+        filters.append(BGPAnomaly.origin_asn == origin_asn)
+    if time_from:
+        filters.append(BGPAnomaly.created_at >= time_from)
+    if time_to:
+        filters.append(BGPAnomaly.created_at <= time_to)
+    return filters
+
+
+def _incident_filters(
+    *,
+    severity: Optional[str],
+    incident_type: Optional[str],
+    status: Optional[str],
+):
+    filters = []
+    if severity:
+        filters.append(BGPIncident.severity == severity)
+    if incident_type:
+        filters.append(BGPIncident.incident_type == incident_type)
+    if status:
+        filters.append(BGPIncident.status == status)
+    return filters
+
+
+async def _build_event_summary_payload(db: AsyncSession) -> dict:
+    base_filters = [BGPObservation.source.in_(BGP_SOURCES)]
+
+    total_result = await db.execute(
+        select(func.count(BGPObservation.id)).where(*base_filters)
+    )
+    collectors_result = await db.execute(
+        select(func.count(func.distinct(BGPObservation.collector))).where(
+            *base_filters, BGPObservation.collector.isnot(None)
+        )
+    )
+    prefixes_result = await db.execute(
+        select(func.count(func.distinct(BGPObservation.prefix))).where(
+            *base_filters, BGPObservation.prefix.isnot(None)
+        )
+    )
+    type_result = await db.execute(
+        select(BGPObservation.event_type, func.count(BGPObservation.id))
+        .where(*base_filters)
+        .group_by(BGPObservation.event_type)
+    )
+
+    return {
+        "total": total_result.scalar() or 0,
+        "collector_count": collectors_result.scalar() or 0,
+        "prefix_count": prefixes_result.scalar() or 0,
+        "by_type": {row[0]: row[1] for row in type_result.fetchall()},
+    }
+
+
+async def _build_anomaly_summary_payload(db: AsyncSession) -> dict:
+    total_result = await db.execute(select(func.count(BGPAnomaly.id)))
+    type_result = await db.execute(
+        select(BGPAnomaly.anomaly_type, func.count(BGPAnomaly.id))
+        .group_by(BGPAnomaly.anomaly_type)
+        .order_by(func.count(BGPAnomaly.id).desc())
+    )
+    severity_result = await db.execute(
+        select(BGPAnomaly.severity, func.count(BGPAnomaly.id))
+        .group_by(BGPAnomaly.severity)
+        .order_by(func.count(BGPAnomaly.id).desc())
+    )
+    status_result = await db.execute(
+        select(BGPAnomaly.status, func.count(BGPAnomaly.id))
+        .group_by(BGPAnomaly.status)
+        .order_by(func.count(BGPAnomaly.id).desc())
+    )
+
+    return {
+        "total": total_result.scalar() or 0,
+        "by_type": {row[0]: row[1] for row in type_result.fetchall()},
+        "by_severity": {row[0]: row[1] for row in severity_result.fetchall()},
+        "by_status": {row[0]: row[1] for row in status_result.fetchall()},
+    }
+
+
+async def _build_incident_summary_payload(db: AsyncSession) -> dict:
+    total_result = await db.execute(select(func.count(BGPIncident.id)))
+    type_result = await db.execute(
+        select(BGPIncident.incident_type, func.count(BGPIncident.id))
+        .group_by(BGPIncident.incident_type)
+        .order_by(func.count(BGPIncident.id).desc())
+    )
+    severity_result = await db.execute(
+        select(BGPIncident.severity, func.count(BGPIncident.id))
+        .group_by(BGPIncident.severity)
+        .order_by(func.count(BGPIncident.id).desc())
+    )
+    status_result = await db.execute(
+        select(BGPIncident.status, func.count(BGPIncident.id))
+        .group_by(BGPIncident.status)
+        .order_by(func.count(BGPIncident.id).desc())
+    )
+
+    return {
+        "total": total_result.scalar() or 0,
+        "by_type": {row[0]: row[1] for row in type_result.fetchall()},
+        "by_severity": {row[0]: row[1] for row in severity_result.fetchall()},
+        "by_status": {row[0]: row[1] for row in status_result.fetchall()},
+    }
+
+
+@router.get("/events")
+async def list_bgp_events(
+    prefix: Optional[str] = Query(None),
+    origin_asn: Optional[int] = Query(None),
+    peer_asn: Optional[int] = Query(None),
+    collector: Optional[str] = Query(None),
+    event_type: Optional[str] = Query(None),
+    source: Optional[str] = Query(None),
+    time_from: Optional[str] = Query(None),
+    time_to: Optional[str] = Query(None),
+    page: int = Query(1, ge=1),
+    page_size: int = Query(50, ge=1, le=200),
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    dt_from = _parse_dt(time_from)
+    dt_to = _parse_dt(time_to)
+    filters = _event_filters(
+        prefix=prefix,
+        origin_asn=origin_asn,
+        peer_asn=peer_asn,
+        collector=collector,
+        event_type=event_type,
+        source=source,
+        time_from=dt_from,
+        time_to=dt_to,
+    )
+    offset = (page - 1) * page_size
+    count_result = await db.execute(
+        select(func.count(BGPObservation.id)).where(*filters)
+    )
+    data_result = await db.execute(
+        select(BGPObservation)
+        .where(*filters)
+        .order_by(BGPObservation.observed_at.desc(), BGPObservation.id.desc())
+        .offset(offset)
+        .limit(page_size)
+    )
+    records = data_result.scalars().all()
+
+    return {
+        "total": count_result.scalar() or 0,
+        "page": page,
+        "page_size": page_size,
+        "data": [record.to_dict() for record in records],
+    }
+
+
+@router.get("/events/summary")
+async def get_bgp_event_summary(
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    return await _build_event_summary_payload(db)
+
+
+@router.get("/collectors")
+async def list_bgp_collectors(
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    data = await build_bgp_collector_coverage(db, source_filter=BGP_SOURCES)
+    return {
+        "total": len(data),
+        "data": data,
+    }
+
+
+@router.get("/collectors/summary")
+async def get_bgp_collector_summary(
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    collectors = await build_bgp_collector_coverage(db, source_filter=BGP_SOURCES)
+    active_collectors = [item for item in collectors if item["observation_count"] > 0]
+    return {
+        "total": len(collectors),
+        "active_collectors": len(active_collectors),
+        "observed_prefixes": sum(item["prefix_count"] for item in active_collectors),
+        "observed_origins": sum(item["origin_asn_count"] for item in active_collectors),
+        "recent_24h_events": sum(item["recent_24h_observation_count"] for item in active_collectors),
+        "recent_7d_events": sum(item["recent_7d_observation_count"] for item in active_collectors),
+    }
+
+
+@router.get("/overview/summary")
+async def get_bgp_overview_summary(
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    event_summary = await _build_event_summary_payload(db)
+    anomaly_summary = await _build_anomaly_summary_payload(db)
+    incident_summary = await _build_incident_summary_payload(db)
+    collectors = await build_bgp_collector_coverage(db, source_filter=BGP_SOURCES)
+    active_collectors = [item for item in collectors if item["observation_count"] > 0]
+
+    return {
+        "incidentSummary": incident_summary,
+        "anomalySummary": anomaly_summary,
+        "eventSummary": event_summary,
+        "collectorSummary": {
+            "total": len(collectors),
+            "active_collectors": len(active_collectors),
+            "observed_prefixes": sum(item["prefix_count"] for item in active_collectors),
+            "observed_origins": sum(item["origin_asn_count"] for item in active_collectors),
+            "recent_24h_events": sum(item["recent_24h_observation_count"] for item in active_collectors),
+            "recent_7d_events": sum(item["recent_7d_observation_count"] for item in active_collectors),
+        },
+    }
+
+
+@router.get("/events/{event_id}")
+async def get_bgp_event(
+    event_id: int,
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    record = await db.get(BGPObservation, event_id)
+    if not record or record.source not in BGP_SOURCES:
+        raise HTTPException(status_code=404, detail="BGP event not found")
+    return record.to_dict()
+
+
+@router.get("/anomalies")
+async def list_bgp_anomalies(
+    severity: Optional[str] = Query(None),
+    anomaly_type: Optional[str] = Query(None),
+    status: Optional[str] = Query(None),
+    prefix: Optional[str] = Query(None),
+    origin_asn: Optional[int] = Query(None),
+    time_from: Optional[str] = Query(None),
+    time_to: Optional[str] = Query(None),
+    page: int = Query(1, ge=1),
+    page_size: int = Query(50, ge=1, le=200),
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    dt_from = _parse_dt(time_from)
+    dt_to = _parse_dt(time_to)
+    filters = _anomaly_filters(
+        severity=severity,
+        anomaly_type=anomaly_type,
+        status=status,
+        prefix=prefix,
+        origin_asn=origin_asn,
+        time_from=dt_from,
+        time_to=dt_to,
+    )
+    offset = (page - 1) * page_size
+    total_result = await db.execute(
+        select(func.count(BGPAnomaly.id)).where(*filters)
+    )
+    data_result = await db.execute(
+        select(BGPAnomaly)
+        .where(*filters)
+        .order_by(BGPAnomaly.created_at.desc(), BGPAnomaly.id.desc())
+        .offset(offset)
+        .limit(page_size)
+    )
+    records = data_result.scalars().all()
+
+    return {
+        "total": total_result.scalar() or 0,
+        "page": page,
+        "page_size": page_size,
+        "data": [record.to_dict() for record in records],
+    }
+
+
+@router.get("/anomalies/summary")
+async def get_bgp_anomaly_summary(
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    return await _build_anomaly_summary_payload(db)
+
+
+@router.get("/anomalies/{anomaly_id}")
+async def get_bgp_anomaly(
+    anomaly_id: int,
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    record = await db.get(BGPAnomaly, anomaly_id)
+    if not record:
+        raise HTTPException(status_code=404, detail="BGP anomaly not found")
+    return record.to_dict()
+
+
+@router.get("/incidents")
+async def list_bgp_incidents(
+    severity: Optional[str] = Query(None),
+    incident_type: Optional[str] = Query(None),
+    status: Optional[str] = Query(None),
+    page: int = Query(1, ge=1),
+    page_size: int = Query(50, ge=1, le=200),
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    filters = _incident_filters(
+        severity=severity,
+        incident_type=incident_type,
+        status=status,
+    )
+    offset = (page - 1) * page_size
+    total_result = await db.execute(
+        select(func.count(BGPIncident.id)).where(*filters)
+    )
+    data_result = await db.execute(
+        select(BGPIncident)
+        .where(*filters)
+        .order_by(BGPIncident.created_at.desc(), BGPIncident.id.desc())
+        .offset(offset)
+        .limit(page_size)
+    )
+    records = data_result.scalars().all()
+
+    return {
+        "total": total_result.scalar() or 0,
+        "page": page,
+        "page_size": page_size,
+        "data": [record.to_dict() for record in records],
+    }
+
+
+@router.get("/incidents/summary")
+async def get_bgp_incident_summary(
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    return await _build_incident_summary_payload(db)
+
+
+@router.get("/incidents/{incident_id}")
+async def get_bgp_incident(
+    incident_id: int,
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    record = await db.get(BGPIncident, incident_id)
+    if not record:
+        raise HTTPException(status_code=404, detail="BGP incident not found")
+    return record.to_dict()

backend/app/api/v1/collected_data.py

@@ -7,16 +7,138 @@ import json
 import csv
 import io
 
+from app.core.collected_data_fields import get_metadata_field
+from app.core.countries import COUNTRY_OPTIONS, get_country_search_variants, normalize_country
+from app.core.time import to_iso8601_utc
 from app.db.session import get_db
 from app.models.user import User
 from app.core.security import get_current_user
 from app.models.collected_data import CollectedData
+from app.models.datasource import DataSource
 
 router = APIRouter()
 
 
+COUNTRY_SQL = "metadata->>'country'"
+SEARCHABLE_SQL = [
+    "name",
+    "title",
+    "description",
+    "source",
+    "data_type",
+    "source_id",
+    "metadata::text",
+]
+
+
+def parse_multi_values(value: Optional[str]) -> list[str]:
+    if not value:
+        return []
+    return [item.strip() for item in value.split(",") if item.strip()]
+
+
+def build_in_condition(field_sql: str, values: list[str], param_prefix: str, params: dict) -> str:
+    placeholders = []
+    for index, value in enumerate(values):
+        key = f"{param_prefix}_{index}"
+        params[key] = value
+        placeholders.append(f":{key}")
+    return f"{field_sql} IN ({', '.join(placeholders)})"
+
+
+def build_search_condition(search: Optional[str], params: dict) -> Optional[str]:
+    if not search:
+        return None
+
+    normalized = search.strip()
+    if not normalized:
+        return None
+
+    search_terms = [normalized]
+    for variant in get_country_search_variants(normalized):
+        if variant.casefold() not in {term.casefold() for term in search_terms}:
+            search_terms.append(variant)
+
+    conditions = []
+    for index, term in enumerate(search_terms):
+        params[f"search_{index}"] = f"%{term}%"
+        conditions.extend(f"{field} ILIKE :search_{index}" for field in SEARCHABLE_SQL)
+
+    params["search_exact"] = normalized
+    params["search_prefix"] = f"{normalized}%"
+
+    canonical_variants = get_country_search_variants(normalized)
+    canonical = canonical_variants[0] if canonical_variants else None
+    params["country_search_exact"] = canonical or normalized
+    params["country_search_prefix"] = f"{(canonical or normalized)}%"
+
+    return "(" + " OR ".join(conditions) + ")"
+
+
+def build_search_rank_sql(search: Optional[str]) -> str:
+    if not search or not search.strip():
+        return "0"
+
+    return """
+        CASE
+            WHEN name ILIKE :search_exact THEN 700
+            WHEN name ILIKE :search_prefix THEN 600
+            WHEN title ILIKE :search_exact THEN 500
+            WHEN title ILIKE :search_prefix THEN 400
+            WHEN metadata->>'country' ILIKE :country_search_exact THEN 380
+            WHEN metadata->>'country' ILIKE :country_search_prefix THEN 340
+            WHEN source_id ILIKE :search_exact THEN 350
+            WHEN source ILIKE :search_exact THEN 300
+            WHEN data_type ILIKE :search_exact THEN 250
+            WHEN description ILIKE :search_0 THEN 150
+            WHEN metadata::text ILIKE :search_0 THEN 100
+            WHEN title ILIKE :search_0 THEN 80
+            WHEN name ILIKE :search_0 THEN 60
+            WHEN source ILIKE :search_0 THEN 40
+            WHEN data_type ILIKE :search_0 THEN 30
+            WHEN source_id ILIKE :search_0 THEN 20
+            ELSE 0
+        END
+    """
+
+
+def serialize_collected_row(row, source_name_map: dict[str, str] | None = None) -> dict:
+    metadata = row[7]
+    source = row[1]
+    return {
+        "id": row[0],
+        "source": source,
+        "source_name": source_name_map.get(source, source) if source_name_map else source,
+        "source_id": row[2],
+        "data_type": row[3],
+        "name": row[4],
+        "title": row[5],
+        "description": row[6],
+        "country": get_metadata_field(metadata, "country"),
+        "city": get_metadata_field(metadata, "city"),
+        "latitude": get_metadata_field(metadata, "latitude"),
+        "longitude": get_metadata_field(metadata, "longitude"),
+        "value": get_metadata_field(metadata, "value"),
+        "unit": get_metadata_field(metadata, "unit"),
+        "metadata": metadata,
+        "cores": get_metadata_field(metadata, "cores"),
+        "rmax": get_metadata_field(metadata, "rmax"),
+        "rpeak": get_metadata_field(metadata, "rpeak"),
+        "power": get_metadata_field(metadata, "power"),
+        "collected_at": to_iso8601_utc(row[8]),
+        "reference_date": to_iso8601_utc(row[9]),
+        "is_valid": row[10],
+    }
+
+
+async def get_source_name_map(db: AsyncSession) -> dict[str, str]:
+    result = await db.execute(select(DataSource.source, DataSource.name))
+    return {row[0]: row[1] for row in result.fetchall()}
+
+
 @router.get("")
 async def list_collected_data(
+    mode: str = Query("current", description="查询模式: current/history"),
     source: Optional[str] = Query(None, description="数据源过滤"),
     data_type: Optional[str] = Query(None, description="数据类型过滤"),
     country: Optional[str] = Query(None, description="国家过滤"),
@@ -27,25 +149,30 @@ async def list_collected_data(
     db: AsyncSession = Depends(get_db),
 ):
     """查询采集的数据列表"""
+    normalized_country = normalize_country(country) if country else None
+    source_values = parse_multi_values(source)
+    data_type_values = parse_multi_values(data_type)
 
     # Build WHERE clause
     conditions = []
     params = {}
 
-    if source:
-        conditions.append("source = :source")
-        params["source"] = source
-    if data_type:
-        conditions.append("data_type = :data_type")
-        params["data_type"] = data_type
-    if country:
-        conditions.append("country = :country")
-        params["country"] = country
-    if search:
-        conditions.append("(name ILIKE :search OR title ILIKE :search)")
-        params["search"] = f"%{search}%"
+    if mode != "history":
+        conditions.append("COALESCE(is_current, TRUE) = TRUE")
+
+    if source_values:
+        conditions.append(build_in_condition("source", source_values, "source", params))
+    if data_type_values:
+        conditions.append(build_in_condition("data_type", data_type_values, "data_type", params))
+    if normalized_country:
+        conditions.append(f"{COUNTRY_SQL} = :country")
+        params["country"] = normalized_country
+    search_condition = build_search_condition(search, params)
+    if search_condition:
+        conditions.append(search_condition)
 
     where_sql = " AND ".join(conditions) if conditions else "1=1"
+    search_rank_sql = build_search_rank_sql(search)
 
     # Calculate offset
     offset = (page - 1) * page_size
@@ -58,11 +185,11 @@ async def list_collected_data(
     # Query data
     query = text(f"""
         SELECT id, source, source_id, data_type, name, title, description,
-               country, city, latitude, longitude, value, unit,
-               metadata, collected_at, reference_date, is_valid
+               metadata, collected_at, reference_date, is_valid,
+               {search_rank_sql} AS search_rank
         FROM collected_data
         WHERE {where_sql}
-        ORDER BY collected_at DESC
+        ORDER BY search_rank DESC, collected_at DESC
         LIMIT :limit OFFSET :offset
     """)
     params["limit"] = page_size
@@ -70,30 +197,11 @@ async def list_collected_data(
 
     result = await db.execute(query, params)
     rows = result.fetchall()
+    source_name_map = await get_source_name_map(db)
 
     data = []
     for row in rows:
-        data.append(
-            {
-                "id": row[0],
-                "source": row[1],
-                "source_id": row[2],
-                "data_type": row[3],
-                "name": row[4],
-                "title": row[5],
-                "description": row[6],
-                "country": row[7],
-                "city": row[8],
-                "latitude": row[9],
-                "longitude": row[10],
-                "value": row[11],
-                "unit": row[12],
-                "metadata": row[13],
-                "collected_at": row[14].isoformat() if row[14] else None,
-                "reference_date": row[15].isoformat() if row[15] else None,
-                "is_valid": row[16],
-            }
-        )
+        data.append(serialize_collected_row(row[:11], source_name_map))
 
     return {
         "total": total,
@@ -105,21 +213,39 @@ async def list_collected_data(
 
 @router.get("/summary")
 async def get_data_summary(
+    mode: str = Query("current", description="查询模式: current/history"),
+    source: Optional[str] = Query(None, description="数据源过滤"),
+    data_type: Optional[str] = Query(None, description="数据类型过滤"),
+    country: Optional[str] = Query(None, description="国家过滤"),
+    search: Optional[str] = Query(None, description="搜索名称"),
     current_user: User = Depends(get_current_user),
     db: AsyncSession = Depends(get_db),
 ):
     """获取数据汇总统计"""
+    where_sql, params = build_where_clause(source, data_type, country, search)
+    if mode != "history":
+        where_sql = f"({where_sql}) AND COALESCE(is_current, TRUE) = TRUE"
+
+    overall_where_sql = "COALESCE(is_current, TRUE) = TRUE" if mode != "history" else "1=1"
+
+    overall_total_result = await db.execute(
+        text(f"SELECT COUNT(*) FROM collected_data WHERE {overall_where_sql}")
+    )
+    overall_total = overall_total_result.scalar() or 0
 
     # By source and data_type
     result = await db.execute(
-        text("""
+        text(f"""
         SELECT source, data_type, COUNT(*) as count
         FROM collected_data
+        WHERE {where_sql}
         GROUP BY source, data_type
         ORDER BY source, data_type
-    """)
+    """),
+        params,
     )
     rows = result.fetchall()
+    source_name_map = await get_source_name_map(db)
 
     by_source = {}
     total = 0
@@ -128,31 +254,62 @@ async def get_data_summary(
         data_type = row[1]
         count = row[2]
 
-        if source not in by_source:
-            by_source[source] = {}
-        by_source[source][data_type] = count
+        source_key = source_name_map.get(source, source)
+        if source_key not in by_source:
+            by_source[source_key] = {}
+        by_source[source_key][data_type] = count
         total += count
 
     # Total by source
     source_totals = await db.execute(
-        text("""
+        text(f"""
         SELECT source, COUNT(*) as count
         FROM collected_data
+        WHERE {where_sql}
         GROUP BY source
         ORDER BY count DESC
-    """)
+    """),
+        params,
     )
     source_rows = source_totals.fetchall()
 
+    type_totals = await db.execute(
+        text(f"""
+        SELECT data_type, COUNT(*) as count
+        FROM collected_data
+        WHERE {where_sql}
+        GROUP BY data_type
+        ORDER BY count DESC, data_type
+    """),
+        params,
+    )
+    type_rows = type_totals.fetchall()
+
     return {
         "total_records": total,
+        "overall_total_records": overall_total,
         "by_source": by_source,
-        "source_totals": [{"source": row[0], "count": row[1]} for row in source_rows],
+        "source_totals": [
+            {
+                "source": row[0],
+                "source_name": source_name_map.get(row[0], row[0]),
+                "count": row[1],
+            }
+            for row in source_rows
+        ],
+        "type_totals": [
+            {
+                "data_type": row[0],
+                "count": row[1],
+            }
+            for row in type_rows
+        ],
     }
 
 
 @router.get("/sources")
 async def get_data_sources(
+    mode: str = Query("current", description="查询模式: current/history"),
     current_user: User = Depends(get_current_user),
     db: AsyncSession = Depends(get_db),
 ):
@@ -160,18 +317,25 @@ async def get_data_sources(
 
     result = await db.execute(
         text("""
-        SELECT DISTINCT source FROM collected_data ORDER BY source
+        SELECT DISTINCT source FROM collected_data
+        """ + ("WHERE COALESCE(is_current, TRUE) = TRUE " if mode != "history" else "") + """
+        ORDER BY source
     """)
     )
     rows = result.fetchall()
+    source_name_map = await get_source_name_map(db)
 
     return {
-        "sources": [row[0] for row in rows],
+        "sources": [
+            {"source": row[0], "source_name": source_name_map.get(row[0], row[0])}
+            for row in rows
+        ],
     }
 
 
 @router.get("/types")
 async def get_data_types(
+    mode: str = Query("current", description="查询模式: current/history"),
     current_user: User = Depends(get_current_user),
     db: AsyncSession = Depends(get_db),
 ):
@@ -179,7 +343,9 @@ async def get_data_types(
 
     result = await db.execute(
         text("""
-        SELECT DISTINCT data_type FROM collected_data ORDER BY data_type
+        SELECT DISTINCT data_type FROM collected_data
+        """ + ("WHERE COALESCE(is_current, TRUE) = TRUE " if mode != "history" else "") + """
+        ORDER BY data_type
     """)
     )
     rows = result.fetchall()
@@ -196,17 +362,8 @@ async def get_countries(
 ):
     """获取所有国家列表"""
 
-    result = await db.execute(
-        text("""
-        SELECT DISTINCT country FROM collected_data
-        WHERE country IS NOT NULL AND country != ''
-        ORDER BY country
-    """)
-    )
-    rows = result.fetchall()
-
     return {
-        "countries": [row[0] for row in rows],
+        "countries": COUNTRY_OPTIONS,
     }
 
 
@@ -221,7 +378,6 @@ async def get_collected_data(
     result = await db.execute(
         text("""
             SELECT id, source, source_id, data_type, name, title, description,
-                   country, city, latitude, longitude, value, unit,
                    metadata, collected_at, reference_date, is_valid
             FROM collected_data
             WHERE id = :id
@@ -236,25 +392,8 @@ async def get_collected_data(
             detail="数据不存在",
         )
 
-    return {
-        "id": row[0],
-        "source": row[1],
-        "source_id": row[2],
-        "data_type": row[3],
-        "name": row[4],
-        "title": row[5],
-        "description": row[6],
-        "country": row[7],
-        "city": row[8],
-        "latitude": row[9],
-        "longitude": row[10],
-        "value": row[11],
-        "unit": row[12],
-        "metadata": row[13],
-        "collected_at": row[14].isoformat() if row[14] else None,
-        "reference_date": row[15].isoformat() if row[15] else None,
-        "is_valid": row[16],
-    }
+    source_name_map = await get_source_name_map(db)
+    return serialize_collected_row(row, source_name_map)
 
 
 def build_where_clause(
@@ -263,19 +402,21 @@ def build_where_clause(
     """Build WHERE clause and params for queries"""
     conditions = []
     params = {}
+    source_values = parse_multi_values(source)
+    data_type_values = parse_multi_values(data_type)
 
-    if source:
-        conditions.append("source = :source")
-        params["source"] = source
-    if data_type:
-        conditions.append("data_type = :data_type")
-        params["data_type"] = data_type
-    if country:
-        conditions.append("country = :country")
-        params["country"] = country
-    if search:
-        conditions.append("(name ILIKE :search OR title ILIKE :search)")
-        params["search"] = f"%{search}%"
+    if source_values:
+        conditions.append(build_in_condition("source", source_values, "source", params))
+    if data_type_values:
+        conditions.append(build_in_condition("data_type", data_type_values, "data_type", params))
+    normalized_country = normalize_country(country) if country else None
+
+    if normalized_country:
+        conditions.append(f"{COUNTRY_SQL} = :country")
+        params["country"] = normalized_country
+    search_condition = build_search_condition(search, params)
+    if search_condition:
+        conditions.append(search_condition)
 
     where_sql = " AND ".join(conditions) if conditions else "1=1"
     return where_sql, params
@@ -283,6 +424,7 @@ def build_where_clause(
 
 @router.get("/export/json")
 async def export_json(
+    mode: str = Query("current", description="查询模式: current/history"),
     source: Optional[str] = Query(None, description="数据源过滤"),
     data_type: Optional[str] = Query(None, description="数据类型过滤"),
     country: Optional[str] = Query(None, description="国家过滤"),
@@ -294,11 +436,12 @@ async def export_json(
     """导出数据为 JSON 格式"""
 
     where_sql, params = build_where_clause(source, data_type, country, search)
+    if mode != "history":
+        where_sql = f"({where_sql}) AND COALESCE(is_current, TRUE) = TRUE"
     params["limit"] = limit
 
     query = text(f"""
         SELECT id, source, source_id, data_type, name, title, description,
-               country, city, latitude, longitude, value, unit,
                metadata, collected_at, reference_date, is_valid
         FROM collected_data
         WHERE {where_sql}
@@ -311,27 +454,7 @@ async def export_json(
 
     data = []
     for row in rows:
-        data.append(
-            {
-                "id": row[0],
-                "source": row[1],
-                "source_id": row[2],
-                "data_type": row[3],
-                "name": row[4],
-                "title": row[5],
-                "description": row[6],
-                "country": row[7],
-                "city": row[8],
-                "latitude": row[9],
-                "longitude": row[10],
-                "value": row[11],
-                "unit": row[12],
-                "metadata": row[13],
-                "collected_at": row[14].isoformat() if row[14] else None,
-                "reference_date": row[15].isoformat() if row[15] else None,
-                "is_valid": row[16],
-            }
-        )
+        data.append(serialize_collected_row(row))
 
     json_str = json.dumps({"data": data, "total": len(data)}, ensure_ascii=False, indent=2)
 
@@ -346,6 +469,7 @@ async def export_json(
 
 @router.get("/export/csv")
 async def export_csv(
+    mode: str = Query("current", description="查询模式: current/history"),
     source: Optional[str] = Query(None, description="数据源过滤"),
     data_type: Optional[str] = Query(None, description="数据类型过滤"),
     country: Optional[str] = Query(None, description="国家过滤"),
@@ -357,11 +481,12 @@ async def export_csv(
     """导出数据为 CSV 格式"""
 
     where_sql, params = build_where_clause(source, data_type, country, search)
+    if mode != "history":
+        where_sql = f"({where_sql}) AND COALESCE(is_current, TRUE) = TRUE"
     params["limit"] = limit
 
     query = text(f"""
         SELECT id, source, source_id, data_type, name, title, description,
-               country, city, latitude, longitude, value, unit,
                metadata, collected_at, reference_date, is_valid
         FROM collected_data
         WHERE {where_sql}
@@ -409,16 +534,16 @@ async def export_csv(
                 row[4],
                 row[5],
                 row[6],
-                row[7],
-                row[8],
-                row[9],
+                get_metadata_field(row[7], "country"),
+                get_metadata_field(row[7], "city"),
+                get_metadata_field(row[7], "latitude"),
+                get_metadata_field(row[7], "longitude"),
+                get_metadata_field(row[7], "value"),
+                get_metadata_field(row[7], "unit"),
+                json.dumps(row[7]) if row[7] else "",
+                to_iso8601_utc(row[8]) or "",
+                to_iso8601_utc(row[9]) or "",
                 row[10],
-                row[11],
-                row[12],
-                json.dumps(row[13]) if row[13] else "",
-                row[14].isoformat() if row[14] else "",
-                row[15].isoformat() if row[15] else "",
-                row[16],
             ]
         )
 

backend/app/api/v1/dashboard.py

@@ -1,8 +1,8 @@
 """Dashboard API with caching and optimizations"""
 
-from datetime import datetime, timedelta
+from datetime import UTC, datetime, timedelta
 from fastapi import APIRouter, Depends
-from sqlalchemy import select, func, text
+from sqlalchemy import case, select, func, text
 from sqlalchemy.ext.asyncio import AsyncSession
 
 from app.db.session import get_db
@@ -13,6 +13,7 @@ from app.models.alert import Alert, AlertSeverity
 from app.models.task import CollectionTask
 from app.core.security import get_current_user
 from app.core.cache import cache
+from app.core.time import to_iso8601_utc
 
 
 # Built-in collectors info (mirrored from datasources.py)
@@ -111,71 +112,90 @@ async def get_stats(
     if cached_result:
         return cached_result
 
-    today_start = datetime.utcnow().replace(hour=0, minute=0, second=0, microsecond=0)
+    today_start = datetime.now(UTC).replace(hour=0, minute=0, second=0, microsecond=0)
 
     # Count built-in collectors
     built_in_count = len(COLLECTOR_INFO)
     built_in_active = built_in_count  # Built-in are always "active" for counting purposes
 
-    # Count custom configs from database
-    result = await db.execute(select(func.count(DataSourceConfig.id)))
-    custom_count = result.scalar() or 0
-
     result = await db.execute(
-        select(func.count(DataSourceConfig.id)).where(DataSourceConfig.is_active == True)
+        select(
+            func.count(DataSourceConfig.id).label("custom_count"),
+            func.sum(
+                case((DataSourceConfig.is_active == True, 1), else_=0)
+            ).label("custom_active"),
+        )
     )
-    custom_active = result.scalar() or 0
+    datasource_stats = result.one()
+    custom_count = datasource_stats.custom_count or 0
+    custom_active = datasource_stats.custom_active or 0
 
     # Total datasources
     total_datasources = built_in_count + custom_count
     active_datasources = built_in_active + custom_active
 
-    # Tasks today (from database)
     result = await db.execute(
-        select(func.count(CollectionTask.id)).where(CollectionTask.started_at >= today_start)
-    )
-    tasks_today = result.scalar() or 0
-
-    result = await db.execute(
-        select(func.count(CollectionTask.id)).where(
-            CollectionTask.status == "success",
-            CollectionTask.started_at >= today_start,
+        select(
+            func.count(CollectionTask.id).label("tasks_today"),
+            func.sum(
+                case(
+                    (CollectionTask.status == "success", 1),
+                    else_=0,
+                )
+            ).label("success_tasks"),
         )
+        .where(CollectionTask.started_at >= today_start)
     )
-    success_tasks = result.scalar() or 0
+    task_stats = result.one()
+    tasks_today = task_stats.tasks_today or 0
+    success_tasks = task_stats.success_tasks or 0
     success_rate = (success_tasks / tasks_today * 100) if tasks_today > 0 else 0
 
-    # Alerts
     result = await db.execute(
-        select(func.count(Alert.id)).where(
-            Alert.severity == AlertSeverity.CRITICAL,
-            Alert.status == "active",
+        select(
+            func.sum(
+                case(
+                    (
+                        (Alert.severity == AlertSeverity.CRITICAL)
+                        & (Alert.status == "active"),
+                        1,
+                    ),
+                    else_=0,
+                )
+            ).label("critical_alerts"),
+            func.sum(
+                case(
+                    (
+                        (Alert.severity == AlertSeverity.WARNING)
+                        & (Alert.status == "active"),
+                        1,
+                    ),
+                    else_=0,
+                )
+            ).label("warning_alerts"),
+            func.sum(
+                case(
+                    (
+                        (Alert.severity == AlertSeverity.INFO)
+                        & (Alert.status == "active"),
+                        1,
+                    ),
+                    else_=0,
+                )
+            ).label("info_alerts"),
         )
     )
-    critical_alerts = result.scalar() or 0
-
-    result = await db.execute(
-        select(func.count(Alert.id)).where(
-            Alert.severity == AlertSeverity.WARNING,
-            Alert.status == "active",
-        )
-    )
-    warning_alerts = result.scalar() or 0
-
-    result = await db.execute(
-        select(func.count(Alert.id)).where(
-            Alert.severity == AlertSeverity.INFO,
-            Alert.status == "active",
-        )
-    )
-    info_alerts = result.scalar() or 0
+    alert_stats = result.one()
+    critical_alerts = alert_stats.critical_alerts or 0
+    warning_alerts = alert_stats.warning_alerts or 0
+    info_alerts = alert_stats.info_alerts or 0
 
     response = {
         "total_datasources": total_datasources,
         "active_datasources": active_datasources,
         "tasks_today": tasks_today,
         "success_rate": round(success_rate, 1),
-        "last_updated": datetime.utcnow().isoformat(),
+        "last_updated": to_iso8601_utc(datetime.now(UTC)),
         "alerts": {
             "critical": critical_alerts,
             "warning": warning_alerts,
@@ -230,10 +250,10 @@ async def get_summary(
         summary[module] = {
             "datasources": data["datasources"],
             "total_records": 0,  # Built-in don't track this in dashboard stats
-            "last_updated": datetime.utcnow().isoformat(),
+            "last_updated": to_iso8601_utc(datetime.now(UTC)),
         }
 
-    response = {"modules": summary, "last_updated": datetime.utcnow().isoformat()}
+    response = {"modules": summary, "last_updated": to_iso8601_utc(datetime.now(UTC))}
 
     cache.set(cache_key, response, expire_seconds=300)
 

backend/app/api/v1/datasource_config.py

@@ -14,6 +14,7 @@ from app.models.user import User
 from app.models.datasource_config import DataSourceConfig
 from app.core.security import get_current_user
 from app.core.cache import cache
+from app.core.time import to_iso8601_utc
 
 router = APIRouter()
 
@@ -123,8 +124,8 @@ async def list_configs(
                 "headers": c.headers,
                 "config": c.config,
                 "is_active": c.is_active,
-                "created_at": c.created_at.isoformat() if c.created_at else None,
-                "updated_at": c.updated_at.isoformat() if c.updated_at else None,
+                "created_at": to_iso8601_utc(c.created_at),
+                "updated_at": to_iso8601_utc(c.updated_at),
             }
             for c in configs
         ],
@@ -155,8 +156,8 @@ async def get_config(
         "headers": config.headers,
         "config": config.config,
         "is_active": config.is_active,
-        "created_at": config.created_at.isoformat() if config.created_at else None,
-        "updated_at": config.updated_at.isoformat() if config.updated_at else None,
+        "created_at": to_iso8601_utc(config.created_at),
+        "updated_at": to_iso8601_utc(config.updated_at),
     }
 
 

backend/app/api/v1/settings.py

@@ -1,13 +1,24 @@
+from copy import deepcopy
+from datetime import UTC, datetime
 from typing import Optional
-from fastapi import APIRouter, Depends, HTTPException
-from pydantic import BaseModel, EmailStr
 
-from app.models.user import User
+from fastapi import APIRouter, Depends, HTTPException
+from pydantic import BaseModel, EmailStr, Field
+from sqlalchemy import select
+from sqlalchemy.ext.asyncio import AsyncSession
+
 from app.core.security import get_current_user
+from app.core.time import to_iso8601_utc
+from app.db.session import get_db
+from app.models.datasource import DataSource
+from app.models.system_setting import SystemSetting
+from app.models.user import User
+from app.services.scheduler import sync_datasource_job
+from app.services.tv_streams import DEFAULT_TV_SETTINGS, get_tv_settings_payload, normalize_tv_settings
 
 router = APIRouter()
 
-default_settings = {
+DEFAULT_SETTINGS = {
     "system": {
         "system_name": "智能星球",
         "refresh_interval": 60,
@@ -27,19 +38,16 @@ default_settings = {
         "max_login_attempts": 5,
         "password_policy": "medium",
     },
+    "tv": DEFAULT_TV_SETTINGS,
 }
 
-system_settings = default_settings["system"].copy()
-notification_settings = default_settings["notifications"].copy()
-security_settings = default_settings["security"].copy()
-
 
 class SystemSettingsUpdate(BaseModel):
     system_name: str = "智能星球"
-    refresh_interval: int = 60
+    refresh_interval: int = Field(default=60, ge=10, le=3600)
     auto_refresh: bool = True
-    data_retention_days: int = 30
-    max_concurrent_tasks: int = 5
+    data_retention_days: int = Field(default=30, ge=1, le=3650)
+    max_concurrent_tasks: int = Field(default=5, ge=1, le=50)
 
 
 class NotificationSettingsUpdate(BaseModel):
@@ -51,60 +59,236 @@ class NotificationSettingsUpdate(BaseModel):
 
 
 class SecuritySettingsUpdate(BaseModel):
-    session_timeout: int = 60
-    max_login_attempts: int = 5
-    password_policy: str = "medium"
+    session_timeout: int = Field(default=60, ge=5, le=1440)
+    max_login_attempts: int = Field(default=5, ge=1, le=20)
+    password_policy: str = Field(default="medium")
+
+
+class CollectorSettingsUpdate(BaseModel):
+    is_active: bool
+    priority: str = Field(default="P1")
+    frequency_minutes: int = Field(default=60, ge=1, le=10080)
+
+
+class TVStreamSourceUpdate(BaseModel):
+    id: str = Field(min_length=1, max_length=100)
+    name: str = Field(min_length=1, max_length=200)
+    provider: str = Field(default="Unknown", max_length=100)
+    region: str = Field(default="Global", max_length=100)
+    language: str = Field(default="und", max_length=32)
+    source_type: str = Field(default="iframe", pattern="^(iframe|hls|video|external|youtube)$")
+    embed_url: str = ""
+    stream_url: str = ""
+    homepage_url: str = ""
+    poster_url: str = ""
+    youtube_video_id: str = ""
+    youtube_channel: str = ""
+    is_enabled: bool = True
+    is_fallback: bool = False
+    sort_order: int = Field(default=10, ge=0, le=9999)
+    collector_source: Optional[str] = None
+    notes: str = ""
+
+
+class TVSettingsUpdate(BaseModel):
+    default_source_id: str = Field(default=DEFAULT_TV_SETTINGS["default_source_id"], min_length=1)
+    auto_fallback: bool = True
+    sources: list[TVStreamSourceUpdate] = Field(default_factory=list)
+
+
+def merge_with_defaults(category: str, payload: Optional[dict]) -> dict:
+    merged = deepcopy(DEFAULT_SETTINGS[category])
+    if payload:
+        merged.update(payload)
+    return merged
+
+
+async def get_setting_record(db: AsyncSession, category: str) -> Optional[SystemSetting]:
+    result = await db.execute(select(SystemSetting).where(SystemSetting.category == category))
+    return result.scalar_one_or_none()
+
+
+async def get_setting_payloads(db: AsyncSession, categories: list[str]) -> dict[str, dict]:
+    if not categories:
+        return {}
+
+    result = await db.execute(
+        select(SystemSetting).where(SystemSetting.category.in_(categories))
+    )
+    records_by_category = {
+        record.category: record
+        for record in result.scalars().all()
+    }
+    return {
+        category: merge_with_defaults(
+            category,
+            records_by_category.get(category).payload if records_by_category.get(category) else None,
+        )
+        for category in categories
+    }
+
+
+async def get_setting_payload(db: AsyncSession, category: str) -> dict:
+    record = await get_setting_record(db, category)
+    return merge_with_defaults(category, record.payload if record else None)
+
+
+async def save_setting_payload(db: AsyncSession, category: str, payload: dict) -> dict:
+    record = await get_setting_record(db, category)
+    if record is None:
+        record = SystemSetting(category=category, payload=payload)
+        db.add(record)
+    else:
+        record.payload = payload
+
+    await db.commit()
+    await db.refresh(record)
+    return merge_with_defaults(category, record.payload)
+
+
+def format_frequency_label(minutes: int) -> str:
+    if minutes % 1440 == 0:
+        return f"{minutes // 1440}d"
+    if minutes % 60 == 0:
+        return f"{minutes // 60}h"
+    return f"{minutes}m"
+
+
+def serialize_collector(datasource: DataSource) -> dict:
+    return {
+        "id": datasource.id,
+        "name": datasource.name,
+        "source": datasource.source,
+        "module": datasource.module,
+        "priority": datasource.priority,
+        "frequency_minutes": datasource.frequency_minutes,
+        "frequency": format_frequency_label(datasource.frequency_minutes),
+        "is_active": datasource.is_active,
+        "last_run_at": to_iso8601_utc(datasource.last_run_at),
+        "last_status": datasource.last_status,
+        "next_run_at": to_iso8601_utc(datasource.next_run_at),
+    }
 
 
 @router.get("/system")
-async def get_system_settings(current_user: User = Depends(get_current_user)):
-    return {"system": system_settings}
+async def get_system_settings(
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    return {"system": await get_setting_payload(db, "system")}
 
 
 @router.put("/system")
 async def update_system_settings(
     settings: SystemSettingsUpdate,
     current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
 ):
-    global system_settings
-    system_settings = settings.model_dump()
-    return {"status": "updated", "system": system_settings}
+    payload = await save_setting_payload(db, "system", settings.model_dump())
+    return {"status": "updated", "system": payload}
 
 
 @router.get("/notifications")
-async def get_notification_settings(current_user: User = Depends(get_current_user)):
-    return {"notifications": notification_settings}
+async def get_notification_settings(
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    return {"notifications": await get_setting_payload(db, "notifications")}
 
 
 @router.put("/notifications")
 async def update_notification_settings(
     settings: NotificationSettingsUpdate,
     current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
 ):
-    global notification_settings
-    notification_settings = settings.model_dump()
-    return {"status": "updated", "notifications": notification_settings}
+    payload = await save_setting_payload(db, "notifications", settings.model_dump())
+    return {"status": "updated", "notifications": payload}
 
 
 @router.get("/security")
-async def get_security_settings(current_user: User = Depends(get_current_user)):
-    return {"security": security_settings}
+async def get_security_settings(
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    return {"security": await get_setting_payload(db, "security")}
 
 
 @router.put("/security")
 async def update_security_settings(
     settings: SecuritySettingsUpdate,
     current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
 ):
-    global security_settings
-    security_settings = settings.model_dump()
-    return {"status": "updated", "security": security_settings}
+    payload = await save_setting_payload(db, "security", settings.model_dump())
+    return {"status": "updated", "security": payload}
+
+
+@router.get("/tv")
+async def get_tv_settings(
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    return {"tv": await get_tv_settings_payload(db)}
+
+
+@router.put("/tv")
+async def update_tv_settings(
+    settings: TVSettingsUpdate,
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    payload = normalize_tv_settings(settings.model_dump())
+    saved = await save_setting_payload(db, "tv", payload)
+    return {"status": "updated", "tv": normalize_tv_settings(saved)}
+
+
+@router.get("/collectors")
+async def get_collector_settings(
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    result = await db.execute(select(DataSource).order_by(DataSource.module, DataSource.id))
+    datasources = result.scalars().all()
+    return {"collectors": [serialize_collector(datasource) for datasource in datasources]}
+
+
+@router.put("/collectors/{datasource_id}")
+async def update_collector_settings(
+    datasource_id: int,
+    settings: CollectorSettingsUpdate,
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    datasource = await db.get(DataSource, datasource_id)
+    if not datasource:
+        raise HTTPException(status_code=404, detail="Data source not found")
+
+    datasource.is_active = settings.is_active
+    datasource.priority = settings.priority
+    datasource.frequency_minutes = settings.frequency_minutes
+    await db.commit()
+    await db.refresh(datasource)
+    await sync_datasource_job(datasource.id)
+    return {"status": "updated", "collector": serialize_collector(datasource)}
 
 
 @router.get("")
-async def get_all_settings(current_user: User = Depends(get_current_user)):
+async def get_all_settings(
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    result = await db.execute(select(DataSource).order_by(DataSource.module, DataSource.id))
+    datasources = result.scalars().all()
+    setting_payloads = await get_setting_payloads(
+        db,
+        ["system", "notifications", "security"],
+    )
     return {
-        "system": system_settings,
-        "notifications": notification_settings,
-        "security": security_settings,
+        "system": setting_payloads["system"],
+        "notifications": setting_payloads["notifications"],
+        "security": setting_payloads["security"],
+        "tv": await get_tv_settings_payload(db),
+        "collectors": [serialize_collector(datasource) for datasource in datasources],
+        "generated_at": to_iso8601_utc(datetime.now(UTC)),
     }

backend/app/api/v1/system_control.py

@@ -0,0 +1,167 @@
+from __future__ import annotations
+
+import os
+import subprocess
+import sys
+
+from fastapi import APIRouter, Depends, HTTPException, status
+from pydantic import BaseModel
+
+from app.core.config import ROOT_DIR
+from app.core.security import get_current_user
+from app.models.user import User
+from app.services.system_control import (
+    build_task_id,
+    clear_active_task_id,
+    get_active_task_id,
+    get_allowed_command,
+    get_runner_script_path,
+    is_task_stale,
+    get_task_logs,
+    require_super_admin,
+    serialize_task,
+    set_active_task_id,
+    upsert_task_state,
+)
+
+router = APIRouter()
+
+
+class RestartTaskCreate(BaseModel):
+    action: str
+
+
+class RestartTaskResponse(BaseModel):
+    task_id: str
+    action: str
+    status: str
+    stage: str
+    message: str
+    created_at: str
+    updated_at: str
+    requested_by: dict[str, object] | None = None
+
+
+class RestartTaskLogsResponse(BaseModel):
+    task_id: str
+    lines: list[str]
+
+
+def ensure_super_admin(current_user: User) -> None:
+    if not require_super_admin(current_user.role):
+        raise HTTPException(
+            status_code=status.HTTP_403_FORBIDDEN,
+            detail="Only super_admin can restart services",
+        )
+
+
+@router.post("/restart-tasks", response_model=RestartTaskResponse)
+async def create_restart_task(
+    payload: RestartTaskCreate,
+    current_user: User = Depends(get_current_user),
+):
+    ensure_super_admin(current_user)
+
+    command = get_allowed_command(payload.action)
+    if command is None:
+        raise HTTPException(
+            status_code=status.HTTP_400_BAD_REQUEST,
+            detail="Unsupported system action",
+        )
+
+    active_task_id = get_active_task_id()
+    if active_task_id:
+        active_task = serialize_task(active_task_id)
+        if active_task and is_task_stale(active_task):
+            upsert_task_state(
+                active_task_id,
+                status="failed",
+                stage="failed",
+                message="Previous restart task became stale and was released",
+            )
+            clear_active_task_id(active_task_id)
+        elif active_task and active_task.get("status") in {"queued", "running"}:
+            raise HTTPException(
+                status_code=status.HTTP_409_CONFLICT,
+                detail="Another restart task is already in progress",
+            )
+
+    task_id = build_task_id()
+    requested_by = {"id": current_user.id, "username": current_user.username}
+    task_state = upsert_task_state(
+        task_id,
+        action=payload.action,
+        status="queued",
+        stage="accepted",
+        message="Restart task accepted",
+        requested_by=requested_by,
+    )
+    set_active_task_id(task_id)
+
+    env = os.environ.copy()
+    backend_path = str(ROOT_DIR / "backend")
+    existing_pythonpath = env.get("PYTHONPATH", "")
+    env["PYTHONPATH"] = (
+        f"{backend_path}{os.pathsep}{existing_pythonpath}"
+        if existing_pythonpath
+        else backend_path
+    )
+
+    try:
+        subprocess.Popen(
+            [
+                sys.executable,
+                str(get_runner_script_path()),
+                "--task-id",
+                task_id,
+                "--action",
+                payload.action,
+            ],
+            cwd=str(ROOT_DIR),
+            env=env,
+            stdout=subprocess.DEVNULL,
+            stderr=subprocess.DEVNULL,
+            start_new_session=True,
+        )
+    except OSError as exc:
+        task_state = upsert_task_state(
+            task_id,
+            action=payload.action,
+            status="failed",
+            stage="failed",
+            message=f"Unable to start restart runner: {exc}",
+            requested_by=requested_by,
+        )
+        clear_active_task_id(task_id)
+        raise HTTPException(
+            status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
+            detail=task_state["message"],
+        ) from exc
+
+    return task_state
+
+
+@router.get("/restart-tasks/{task_id}", response_model=RestartTaskResponse)
+async def get_restart_task(
+    task_id: str,
+    current_user: User = Depends(get_current_user),
+):
+    ensure_super_admin(current_user)
+
+    task = serialize_task(task_id)
+    if task is None:
+        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Restart task not found")
+    return task
+
+
+@router.get("/restart-tasks/{task_id}/logs", response_model=RestartTaskLogsResponse)
+async def get_restart_task_logs(
+    task_id: str,
+    current_user: User = Depends(get_current_user),
+):
+    ensure_super_admin(current_user)
+
+    task = serialize_task(task_id)
+    if task is None:
+        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Restart task not found")
+    return {"task_id": task_id, "lines": get_task_logs(task_id)}

backend/app/api/v1/tasks.py

@@ -1,4 +1,4 @@
-from datetime import datetime
+from datetime import UTC, datetime
 from typing import Optional
 
 from fastapi import APIRouter, Depends, HTTPException, status
@@ -8,6 +8,7 @@ from sqlalchemy import text
 from app.db.session import get_db
 from app.models.user import User
 from app.core.security import get_current_user
+from app.core.time import to_iso8601_utc
 from app.services.collectors.registry import collector_registry
 
 
@@ -61,8 +62,8 @@ async def list_tasks(
                 "datasource_id": t[1],
                 "datasource_name": t[2],
                 "status": t[3],
-                "started_at": t[4].isoformat() if t[4] else None,
-                "completed_at": t[5].isoformat() if t[5] else None,
+                "started_at": to_iso8601_utc(t[4]),
+                "completed_at": to_iso8601_utc(t[5]),
                 "records_processed": t[6],
                 "error_message": t[7],
             }
@@ -100,8 +101,8 @@ async def get_task(
         "datasource_id": task[1],
         "datasource_name": task[2],
         "status": task[3],
-        "started_at": task[4].isoformat() if task[4] else None,
-        "completed_at": task[5].isoformat() if task[5] else None,
+        "started_at": to_iso8601_utc(task[4]),
+        "completed_at": to_iso8601_utc(task[5]),
         "records_processed": task[6],
         "error_message": task[7],
     }
@@ -147,8 +148,8 @@ async def trigger_collection(
             "status": result.get("status", "unknown"),
             "records_processed": result.get("records_processed", 0),
             "error_message": result.get("error"),
-            "started_at": datetime.utcnow(),
-            "completed_at": datetime.utcnow(),
+            "started_at": datetime.now(UTC),
+            "completed_at": datetime.now(UTC),
         },
     )
 

backend/app/api/v1/tv.py

@@ -0,0 +1,70 @@
+from urllib.parse import quote, urljoin
+
+import httpx
+from fastapi import APIRouter, Depends, HTTPException, Query
+from fastapi.responses import Response
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.db.session import get_db
+from app.services.tv_streams import get_public_tv_payload, is_allowed_tv_proxy_url
+
+router = APIRouter()
+
+
+@router.get("/streams")
+async def list_public_tv_streams(
+    db: AsyncSession = Depends(get_db),
+):
+    return await get_public_tv_payload(db)
+
+
+@router.get("/proxy")
+async def proxy_tv_stream(
+    url: str = Query(..., description="Upstream TV stream or manifest URL"),
+    db: AsyncSession = Depends(get_db),
+):
+    payload = await get_public_tv_payload(db)
+    if not is_allowed_tv_proxy_url(url, payload.get("sources", [])):
+        raise HTTPException(status_code=403, detail="TV proxy target is not allowed")
+
+    try:
+        async with httpx.AsyncClient(follow_redirects=True, timeout=20.0) as client:
+            upstream = await client.get(
+                url,
+                headers={
+                    "User-Agent": "Mozilla/5.0",
+                    "Referer": "https://tv.cctv.com/live/cctv4/",
+                },
+            )
+            upstream.raise_for_status()
+    except httpx.HTTPError as exc:
+        raise HTTPException(status_code=502, detail=f"Failed to fetch TV stream: {exc}") from exc
+
+    content_type = upstream.headers.get("content-type", "application/octet-stream")
+    raw_content = upstream.content
+    response_url = str(upstream.url)
+    is_manifest = (
+        response_url.endswith(".m3u8")
+        or "mpegurl" in content_type.lower()
+        or raw_content.lstrip().startswith(b"#EXTM3U")
+    )
+
+    headers = {"Cache-Control": "no-store"}
+
+    if is_manifest:
+        manifest_text = upstream.text
+        rewritten_lines: list[str] = []
+        for line in manifest_text.splitlines():
+            stripped = line.strip()
+            if not stripped or stripped.startswith("#"):
+                rewritten_lines.append(line)
+                continue
+            absolute_url = urljoin(response_url, stripped)
+            rewritten_lines.append(f"/api/v1/tv/proxy?url={quote(absolute_url, safe='')}")
+        return Response(
+            content="\n".join(rewritten_lines),
+            media_type="application/vnd.apple.mpegurl",
+            headers=headers,
+        )
+
+    return Response(content=raw_content, media_type=content_type, headers=headers)

backend/app/api/v1/ue_data.py

@@ -0,0 +1,351 @@
+"""UE Client Data API
+
+Flat JSON endpoints designed for easy parsing in Unreal Engine C++/Blueprint.
+Avoids GeoJSON nesting — every field is at the top level of each item.
+"""
+
+from typing import Any, Dict, List, Optional
+from fastapi import APIRouter, Depends, Query
+from sqlalchemy.ext.asyncio import AsyncSession
+from sqlalchemy import select, func
+
+from app.core.collected_data_fields import get_record_field
+from app.core.satellite_tle import build_tle_lines_from_elements
+from app.core.time import to_iso8601_utc
+from app.db.session import get_db
+from app.models.collected_data import CollectedData
+
+router = APIRouter()
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+def _current_stmt(source: str, limit: Optional[int] = None):
+    stmt = (
+        select(CollectedData)
+        .where(CollectedData.source == source)
+        .where(CollectedData.is_current.is_(True))
+        .order_by(CollectedData.id.desc())
+    )
+    if limit:
+        stmt = stmt.limit(limit)
+    return stmt
+
+
+async def _fetch(db: AsyncSession, source: str, limit: Optional[int] = None) -> List[CollectedData]:
+    result = await db.execute(_current_stmt(source, limit))
+    return list(result.scalars().all())
+
+
+def _safe_float(value: Any) -> Optional[float]:
+    try:
+        v = float(value)
+        return v if v == v else None  # reject NaN
+    except (TypeError, ValueError):
+        return None
+
+
+# ---------------------------------------------------------------------------
+# Status endpoint
+# ---------------------------------------------------------------------------
+
+@router.get("/status")
+async def ue_status(db: AsyncSession = Depends(get_db)):
+    """Quick health-check + data counts for the UE client."""
+    from datetime import UTC, datetime
+
+    async def count_source(source: str) -> int:
+        result = await db.execute(
+            select(func.count())
+            .select_from(CollectedData)
+            .where(CollectedData.source == source)
+            .where(CollectedData.is_current.is_(True))
+        )
+        return result.scalar() or 0
+
+    return {
+        "ok": True,
+        "server_time": to_iso8601_utc(datetime.now(UTC)),
+        "compute_points_count": await count_source("top500"),
+        "cables_count": await count_source("telegeography_cables"),
+        "landing_points_count": await count_source("arcgis_landing"),
+        "satellites_count": await count_source("celestrak"),
+    }
+
+
+# ---------------------------------------------------------------------------
+# Compute points (TOP500 supercomputers)
+# ---------------------------------------------------------------------------
+
+@router.get("/compute-points")
+async def ue_compute_points(
+    limit: int = Query(default=500, ge=1, le=2000),
+    db: AsyncSession = Depends(get_db),
+):
+    """
+    Returns TOP500 supercomputer data as a flat JSON array.
+
+    Response shape:
+    {
+      "count": 500,
+      "items": [
+        {
+          "id": "top500_1",
+          "name": "Frontier",
+          "latitude": 36.01,
+          "longitude": -84.26,
+          "country": "United States",
+          "city": "Oak Ridge",
+          "rank": 1,
+          "rmax_tflops": 1194000.0,
+          "rpeak_tflops": 1679616.0,
+          "cores": 8730112,
+          "power_kw": 22703.0
+        }
+      ]
+    }
+    """
+    records = await _fetch(db, "top500", limit)
+    items = []
+    for record in records:
+        meta = record.extra_data or {}
+        lat = _safe_float(get_record_field(record, "latitude"))
+        lon = _safe_float(get_record_field(record, "longitude"))
+        if lat is None or lon is None:
+            continue
+        items.append({
+            "id": f"top500_{record.id}",
+            "name": record.name or "Unknown",
+            "latitude": lat,
+            "longitude": lon,
+            "country": get_record_field(record, "country") or "",
+            "city": get_record_field(record, "city") or "",
+            "rank": meta.get("rank"),
+            "rmax_tflops": _safe_float(get_record_field(record, "rmax")),
+            "rpeak_tflops": _safe_float(get_record_field(record, "rpeak")),
+            "cores": meta.get("cores"),
+            "power_kw": _safe_float(get_record_field(record, "power")),
+        })
+    return {"count": len(items), "items": items}
+
+
+# ---------------------------------------------------------------------------
+# Cable landing points
+# ---------------------------------------------------------------------------
+
+@router.get("/landing-points")
+async def ue_landing_points(db: AsyncSession = Depends(get_db)):
+    """
+    Returns cable landing points as a flat JSON array.
+
+    Response shape:
+    {
+      "count": 1200,
+      "items": [
+        {
+          "id": "lp_42",
+          "name": "Shoreham",
+          "latitude": 50.83,
+          "longitude": -0.28,
+          "country": "United Kingdom",
+          "city": "Shoreham-by-Sea",
+          "cable_names": ["FLAG", "TAT-14"]
+        }
+      ]
+    }
+    """
+    # Load landing points
+    lp_records = await _fetch(db, "arcgis_landing")
+
+    # Load relation + cable data for cable_names mapping
+    rel_result = await db.execute(
+        select(CollectedData)
+        .where(CollectedData.source == "arcgis_relation")
+        .where(CollectedData.is_current.is_(True))
+    )
+    rel_records = list(rel_result.scalars().all())
+
+    cable_result = await db.execute(
+        select(CollectedData)
+        .where(CollectedData.source == "telegeography_cables")
+        .where(CollectedData.is_current.is_(True))
+    )
+    cable_records = list(cable_result.scalars().all())
+
+    # Build mapping: city_id → list of cable names
+    city_to_cable_ids: Dict[int, List[int]] = {}
+    for r in rel_records:
+        meta = r.extra_data or {}
+        city_id = meta.get("city_id")
+        cable_id = meta.get("cable_id")
+        if city_id is not None and cable_id is not None:
+            city_to_cable_ids.setdefault(city_id, [])
+            if cable_id not in city_to_cable_ids[city_id]:
+                city_to_cable_ids[city_id].append(cable_id)
+
+    cable_id_to_name: Dict[int, str] = {}
+    for r in cable_records:
+        meta = r.extra_data or {}
+        cable_id = meta.get("cable_id")
+        if cable_id and r.name:
+            cable_id_to_name[cable_id] = r.name
+
+    items = []
+    for record in lp_records:
+        lat = _safe_float(get_record_field(record, "latitude"))
+        lon = _safe_float(get_record_field(record, "longitude"))
+        if lat is None or lon is None:
+            continue
+        meta = record.extra_data or {}
+        city_id = meta.get("city_id")
+        cable_names = []
+        if city_id in city_to_cable_ids:
+            cable_names = [
+                cable_id_to_name[cid]
+                for cid in city_to_cable_ids[city_id]
+                if cid in cable_id_to_name
+            ]
+        items.append({
+            "id": f"lp_{record.id}",
+            "name": record.name or "Unknown",
+            "latitude": lat,
+            "longitude": lon,
+            "country": get_record_field(record, "country") or "",
+            "city": get_record_field(record, "city") or "",
+            "cable_names": cable_names,
+        })
+    return {"count": len(items), "items": items}
+
+
+# ---------------------------------------------------------------------------
+# Cables (route geometry)
+# ---------------------------------------------------------------------------
+
+@router.get("/cables")
+async def ue_cables(db: AsyncSession = Depends(get_db)):
+    """
+    Returns cable route geometry.
+
+    Each segment is a flat array of [lon, lat] pairs.
+
+    Response shape:
+    {
+      "count": 100,
+      "items": [
+        {
+          "id": "cable_42",
+          "cable_id": "flag",
+          "name": "FLAG",
+          "status": "active",
+          "length_km": 28000,
+          "segments": [
+            [[lon, lat], [lon, lat], ...]
+          ]
+        }
+      ]
+    }
+    """
+    records = await _fetch(db, "telegeography_cables")
+    items = []
+    for record in records:
+        meta = record.extra_data or {}
+        route_coords = meta.get("route_coordinates", [])
+        segments: List[List[List[float]]] = []
+
+        if route_coords:
+            # Support both flat [lon,lat] array and array-of-arrays
+            if route_coords and isinstance(route_coords[0][0], list):
+                raw_lines = route_coords
+            else:
+                raw_lines = [route_coords]
+
+            for raw_line in raw_lines:
+                line = []
+                for pt in raw_line:
+                    try:
+                        line.append([float(pt[0]), float(pt[1])])
+                    except (TypeError, ValueError, IndexError):
+                        continue
+                if len(line) >= 2:
+                    segments.append(line)
+
+        if not segments:
+            continue
+
+        items.append({
+            "id": f"cable_{record.id}",
+            "cable_id": record.source_id or record.name or "",
+            "name": record.name or "Unknown",
+            "status": meta.get("status", "active"),
+            "length_km": _safe_float(get_record_field(record, "value")),
+            "owners": meta.get("owners") or [],
+            "rfs": meta.get("rfs"),
+            "color": meta.get("color"),
+            "segments": segments,
+        })
+    return {"count": len(items), "items": items}
+
+
+# ---------------------------------------------------------------------------
+# Satellites (TLE data)
+# ---------------------------------------------------------------------------
+
+@router.get("/satellites")
+async def ue_satellites(
+    limit: int = Query(default=200, ge=1, le=5000),
+    db: AsyncSession = Depends(get_db),
+):
+    """
+    Returns satellite TLE data for orbit propagation in UE.
+
+    Response shape:
+    {
+      "count": 200,
+      "items": [
+        {
+          "id": "sat_42",
+          "norad_id": "25544",
+          "name": "ISS (ZARYA)",
+          "tle_line1": "1 25544U ...",
+          "tle_line2": "2 25544 ...",
+          "epoch": "2026-04-14T00:00:00Z",
+          "inclination": 51.6,
+          "mean_motion": 15.5
+        }
+      ]
+    }
+    """
+    records = await _fetch(db, "celestrak", limit)
+    items = []
+    for record in records:
+        meta = record.extra_data or {}
+        norad_id = meta.get("norad_cat_id")
+        if not norad_id:
+            continue
+        tle1 = meta.get("tle_line1")
+        tle2 = meta.get("tle_line2")
+        if not tle1 or not tle2:
+            tle1, tle2 = build_tle_lines_from_elements(
+                norad_cat_id=norad_id,
+                epoch=meta.get("epoch"),
+                inclination=meta.get("inclination"),
+                raan=meta.get("raan"),
+                eccentricity=meta.get("eccentricity"),
+                arg_of_perigee=meta.get("arg_of_perigee"),
+                mean_anomaly=meta.get("mean_anomaly"),
+                mean_motion=meta.get("mean_motion"),
+            )
+        items.append({
+            "id": f"sat_{record.id}",
+            "norad_id": str(norad_id),
+            "name": record.name or "Unknown",
+            "tle_line1": tle1 or "",
+            "tle_line2": tle2 or "",
+            "epoch": meta.get("epoch") or "",
+            "inclination": _safe_float(meta.get("inclination")),
+            "raan": _safe_float(meta.get("raan")),
+            "eccentricity": _safe_float(meta.get("eccentricity")),
+            "mean_motion": _safe_float(meta.get("mean_motion")),
+        })
+    return {"count": len(items), "items": items}

backend/app/api/v1/visualization.py

@@ -4,15 +4,23 @@ Unified API for all visualization data sources.
 Returns GeoJSON format compatible with Three.js, CesiumJS, and Unreal Cesium.
 """
 
-from datetime import datetime
-from fastapi import APIRouter, HTTPException, Depends
+from datetime import UTC, datetime
+import math
+from fastapi import APIRouter, HTTPException, Depends, Query
 from sqlalchemy.ext.asyncio import AsyncSession
 from sqlalchemy import select, func
 from typing import List, Dict, Any, Optional
 
+from app.core.collected_data_fields import get_record_field
+from app.core.satellite_tle import build_tle_lines_from_elements
+from app.core.time import to_iso8601_utc
 from app.db.session import get_db
+from app.models.bgp_anomaly import BGPAnomaly
+from app.models.bgp_incident import BGPIncident
 from app.models.collected_data import CollectedData
-from app.services.cable_graph import build_graph_from_data, CableGraph
+from app.services.bgp_collectors import build_bgp_collector_coverage
+from app.services.cable_graph import build_graph_from_data, CableGraph, haversine_distance
+from app.services.collectors.bgp_common import RIPE_RIS_COLLECTOR_COORDS
 
 router = APIRouter()
 
@@ -83,9 +91,9 @@ def convert_cable_to_geojson(records: List[CollectedData]) -> Dict[str, Any]:
                     "rfs": metadata.get("rfs"),
                     "RFS": metadata.get("rfs"),
                     "status": metadata.get("status", "active"),
-                    "length": record.value,
-                    "length_km": record.value,
-                    "SHAPE__Length": record.value,
+                    "length": get_record_field(record, "value"),
+                    "length_km": get_record_field(record, "value"),
+                    "SHAPE__Length": get_record_field(record, "value"),
                     "url": metadata.get("url"),
                     "color": metadata.get("color"),
                     "year": metadata.get("year"),
@@ -101,8 +109,10 @@ def convert_landing_point_to_geojson(records: List[CollectedData], city_to_cable
     
     for record in records:
         try:
-            lat = float(record.latitude) if record.latitude else None
-            lon = float(record.longitude) if record.longitude else None
+            latitude = get_record_field(record, "latitude")
+            longitude = get_record_field(record, "longitude")
+            lat = float(latitude) if latitude else None
+            lon = float(longitude) if longitude else None
         except (ValueError, TypeError):
             continue
         
@@ -116,8 +126,8 @@ def convert_landing_point_to_geojson(records: List[CollectedData], city_to_cable
             "id": record.id,
             "source_id": record.source_id,
             "name": record.name,
-            "country": record.country,
-            "city": record.city,
+            "country": get_record_field(record, "country"),
+            "city": get_record_field(record, "city"),
             "is_tbd": metadata.get("is_tbd", False),
         }
         
@@ -152,6 +162,20 @@ def convert_satellite_to_geojson(records: List[CollectedData]) -> Dict[str, Any]
         if not norad_id:
             continue
 
+        tle_line1 = metadata.get("tle_line1")
+        tle_line2 = metadata.get("tle_line2")
+        if not tle_line1 or not tle_line2:
+            tle_line1, tle_line2 = build_tle_lines_from_elements(
+                norad_cat_id=norad_id,
+                epoch=metadata.get("epoch"),
+                inclination=metadata.get("inclination"),
+                raan=metadata.get("raan"),
+                eccentricity=metadata.get("eccentricity"),
+                arg_of_perigee=metadata.get("arg_of_perigee"),
+                mean_anomaly=metadata.get("mean_anomaly"),
+                mean_motion=metadata.get("mean_motion"),
+            )
+
         features.append(
             {
                 "type": "Feature",
@@ -171,6 +195,8 @@ def convert_satellite_to_geojson(records: List[CollectedData]) -> Dict[str, Any]
                     "mean_motion": metadata.get("mean_motion"),
                     "bstar": metadata.get("bstar"),
                     "classification_type": metadata.get("classification_type"),
+                    "tle_line1": tle_line1,
+                    "tle_line2": tle_line2,
                     "data_type": "satellite_tle",
                 },
             }
@@ -179,15 +205,97 @@ def convert_satellite_to_geojson(records: List[CollectedData]) -> Dict[str, Any]
     return {"type": "FeatureCollection", "features": features}
 
 
+def _current_collected_data_stmt(source: str):
+    return (
+        select(CollectedData)
+        .where(CollectedData.source == source)
+        .where(CollectedData.is_current.is_(True))
+        .order_by(CollectedData.id.desc())
+    )
+
+
+async def _load_current_collected_data(
+    db: AsyncSession,
+    source: str,
+    *,
+    exclude_unknown_name: bool = False,
+    limit: Optional[int] = None,
+) -> List[CollectedData]:
+    stmt = _current_collected_data_stmt(source)
+    if exclude_unknown_name:
+        stmt = stmt.where(CollectedData.name != "Unknown")
+    if limit is not None:
+        stmt = stmt.limit(limit)
+
+    result = await db.execute(stmt)
+    return list(result.scalars().all())
+
+
+async def _load_current_collected_data_by_sources(
+    db: AsyncSession,
+    sources: List[str],
+) -> Dict[str, List[CollectedData]]:
+    if not sources:
+        return {}
+
+    stmt = (
+        select(CollectedData)
+        .where(CollectedData.source.in_(sources))
+        .where(CollectedData.is_current.is_(True))
+        .order_by(CollectedData.source.asc(), CollectedData.id.desc())
+    )
+    result = await db.execute(stmt)
+
+    grouped_records: Dict[str, List[CollectedData]] = {source: [] for source in sources}
+    for record in result.scalars().all():
+        grouped_records.setdefault(record.source, []).append(record)
+
+    return grouped_records
+
+
+def _build_landing_point_cable_maps(
+    relation_records: List[CollectedData],
+    cable_records: List[CollectedData],
+) -> tuple[Dict[int, List[int]], Dict[int, str]]:
+    city_to_cable_ids_map: Dict[int, List[int]] = {}
+    for relation_record in relation_records:
+        if not relation_record.extra_data:
+            continue
+        city_id = relation_record.extra_data.get("city_id")
+        cable_id = relation_record.extra_data.get("cable_id")
+        if city_id is None or cable_id is None:
+            continue
+        city_to_cable_ids_map.setdefault(city_id, [])
+        if cable_id not in city_to_cable_ids_map[city_id]:
+            city_to_cable_ids_map[city_id].append(cable_id)
+
+    cable_id_to_name_map: Dict[int, str] = {}
+    for cable_record in cable_records:
+        if not cable_record.extra_data:
+            continue
+        cable_id = cable_record.extra_data.get("cable_id")
+        cable_name = cable_record.name
+        if cable_id and cable_name:
+            cable_id_to_name_map[cable_id] = cable_name
+
+    return city_to_cable_ids_map, cable_id_to_name_map
+
+
+def _filter_known_records(records: List[CollectedData]) -> List[CollectedData]:
+    return [record for record in records if record.name != "Unknown"]
+
+
 def convert_supercomputer_to_geojson(records: List[CollectedData]) -> Dict[str, Any]:
     """Convert TOP500 supercomputer records to GeoJSON"""
     features = []
 
     for record in records:
         try:
-            lat = float(record.latitude) if record.latitude and record.latitude != "0.0" else None
+            latitude = get_record_field(record, "latitude")
+            longitude = get_record_field(record, "longitude")
+            lat = float(latitude) if latitude and latitude != "0.0" else None
             lon = (
-                float(record.longitude) if record.longitude and record.longitude != "0.0" else None
+                float(longitude) if longitude and longitude != "0.0" else None
             )
         except (ValueError, TypeError):
             lat, lon = None, None
@@ -203,12 +311,12 @@ def convert_supercomputer_to_geojson(records: List[CollectedData]) -> Dict[str,
                     "id": record.id,
                     "name": record.name,
                     "rank": metadata.get("rank"),
-                    "r_max": record.value,
-                    "r_peak": metadata.get("r_peak"),
-                    "cores": metadata.get("cores"),
-                    "power": metadata.get("power"),
-                    "country": record.country,
-                    "city": record.city,
+                    "r_max": get_record_field(record, "rmax"),
+                    "r_peak": get_record_field(record, "rpeak"),
+                    "cores": get_record_field(record, "cores"),
+                    "power": get_record_field(record, "power"),
+                    "country": get_record_field(record, "country"),
+                    "city": get_record_field(record, "city"),
                     "data_type": "supercomputer",
                 },
             }
@@ -223,8 +331,10 @@ def convert_gpu_cluster_to_geojson(records: List[CollectedData]) -> Dict[str, An
 
     for record in records:
         try:
-            lat = float(record.latitude) if record.latitude else None
-            lon = float(record.longitude) if record.longitude else None
+            latitude = get_record_field(record, "latitude")
+            longitude = get_record_field(record, "longitude")
+            lat = float(latitude) if latitude else None
+            lon = float(longitude) if longitude else None
         except (ValueError, TypeError):
             lat, lon = None, None
 
@@ -238,8 +348,8 @@ def convert_gpu_cluster_to_geojson(records: List[CollectedData]) -> Dict[str, An
                 "properties": {
                     "id": record.id,
                     "name": record.name,
-                    "country": record.country,
-                    "city": record.city,
+                    "country": get_record_field(record, "country"),
+                    "city": get_record_field(record, "city"),
                     "metadata": metadata,
                     "data_type": "gpu_cluster",
                 },
@@ -249,6 +359,404 @@ def convert_gpu_cluster_to_geojson(records: List[CollectedData]) -> Dict[str, An
     return {"type": "FeatureCollection", "features": features}
 
 
+def convert_bgp_anomalies_to_geojson(
+    records: List[BGPAnomaly],
+    geography_hints: Optional[Dict[str, Dict[str, Any]]] = None,
+) -> Dict[str, Any]:
+    features = []
+    geography_hints = geography_hints or {}
+
+    for record in records:
+        evidence = record.evidence or {}
+        hint = geography_hints.get(str(record.entity_key or record.id), {})
+        collectors = evidence.get("collectors") or record.peer_scope or []
+        if not collectors:
+            nested = evidence.get("events") or []
+            collectors = [
+                str((item or {}).get("collector") or "").strip()
+                for item in nested
+                if (item or {}).get("collector")
+            ]
+
+        collectors = [collector for collector in collectors if collector]
+        if not collectors:
+            collectors = []
+
+        as_path = []
+        if isinstance(evidence.get("as_path"), list):
+            as_path = evidence.get("as_path") or []
+        if not as_path:
+            nested = evidence.get("events") or []
+            for item in nested:
+                candidate_path = (item or {}).get("as_path")
+                if isinstance(candidate_path, list) and candidate_path:
+                    as_path = candidate_path
+                    break
+
+        impacted_regions = []
+        seen_regions = set()
+        for collector_name in collectors:
+            collector_location = RIPE_RIS_COLLECTOR_COORDS.get(str(collector_name))
+            if not collector_location:
+                continue
+            region_key = (
+                collector_location.get("country"),
+                collector_location.get("city"),
+            )
+            if region_key in seen_regions:
+                continue
+            seen_regions.add(region_key)
+            impacted_regions.append(
+                {
+                    "collector": collector_name,
+                    "country": collector_location.get("country"),
+                    "city": collector_location.get("city"),
+                    "latitude": collector_location.get("latitude"),
+                    "longitude": collector_location.get("longitude"),
+                }
+            )
+
+        geography_regions = _normalize_geo_regions(hint.get("regions") or [])
+        geography_mode = hint.get("geography_mode") or "collector_centroid"
+
+        collector = collectors[0] if collectors else None
+        location = geography_regions[0] if geography_regions else None
+
+        if location is None and collector:
+            location = RIPE_RIS_COLLECTOR_COORDS.get(str(collector))
+
+        if location is None:
+            nested = evidence.get("events") or []
+            for item in nested:
+                collector_name = (item or {}).get("collector")
+                if collector_name and collector_name in RIPE_RIS_COLLECTOR_COORDS:
+                    location = RIPE_RIS_COLLECTOR_COORDS[collector_name]
+                    collector = collector_name
+                    geography_mode = "collector_centroid"
+                    break
+
+        if location is None:
+            continue
+
+        features.append(
+            {
+                "type": "Feature",
+                "geometry": {
+                    "type": "Point",
+                    "coordinates": [location["longitude"], location["latitude"]],
+                },
+                "properties": {
+                    "id": record.id,
+                    "collector": collector,
+                    "city": location.get("city"),
+                    "country": location.get("country"),
+                    "source": record.source,
+                    "anomaly_type": record.anomaly_type,
+                    "severity": record.severity,
+                    "status": record.status,
+                    "prefix": record.prefix,
+                    "origin_asn": record.origin_asn,
+                    "new_origin_asn": record.new_origin_asn,
+                    "collectors": collectors,
+                    "collector_count": len(collectors) or 1,
+                    "as_path": as_path,
+                    "impacted_regions": impacted_regions,
+                    "geography_mode": geography_mode,
+                    "confidence": record.confidence,
+                    "summary": record.summary,
+                    "created_at": to_iso8601_utc(record.created_at),
+                },
+            }
+        )
+
+    return {"type": "FeatureCollection", "features": features}
+
+
+async def build_anomaly_geography_hints(
+    db: AsyncSession,
+    records: List[BGPAnomaly],
+) -> Dict[str, Dict[str, Any]]:
+    hints: Dict[str, Dict[str, Any]] = {}
+    for record in records:
+        hint = _extract_evidence_geography_hint(record.evidence or {})
+        if hint:
+            hints[str(record.entity_key or record.id)] = hint
+
+    return hints
+
+
+def convert_bgp_collectors_to_geojson(
+    coverage_by_collector: Dict[str, Dict[str, Any]] | None = None,
+) -> Dict[str, Any]:
+    features = []
+    coverage_by_collector = coverage_by_collector or {}
+
+    for collector, location in sorted(RIPE_RIS_COLLECTOR_COORDS.items()):
+        coverage = coverage_by_collector.get(collector, {})
+        features.append(
+            {
+                "type": "Feature",
+                "geometry": {
+                    "type": "Point",
+                    "coordinates": [location["longitude"], location["latitude"]],
+                },
+                "properties": {
+                    "collector": collector,
+                    "city": coverage.get("city") or location.get("city"),
+                    "country": coverage.get("country") or location.get("country"),
+                    "status": "online",
+                    "observation_count": coverage.get("observation_count", 0),
+                    "prefix_count": coverage.get("prefix_count", 0),
+                    "origin_asn_count": coverage.get("origin_asn_count", 0),
+                    "peer_asn_count": coverage.get("peer_asn_count", 0),
+                    "recent_15m_observation_count": coverage.get("recent_15m_observation_count", 0),
+                    "recent_24h_observation_count": coverage.get("recent_24h_observation_count", 0),
+                    "recent_7d_observation_count": coverage.get("recent_7d_observation_count", 0),
+                    "recent_15m_prefix_count": coverage.get("recent_15m_prefix_count", 0),
+                    "recent_24h_prefix_count": coverage.get("recent_24h_prefix_count", 0),
+                    "recent_7d_prefix_count": coverage.get("recent_7d_prefix_count", 0),
+                    "top_event_types": coverage.get("top_event_types", []),
+                    "latest_observed_at": coverage.get("latest_observed_at"),
+                    "latest_event_type": coverage.get("latest_event_type"),
+                    "baseline_scope": coverage.get(
+                        "baseline_scope",
+                        {
+                            "countries": [location.get("country")] if location.get("country") else [],
+                            "cities": [location.get("city")] if location.get("city") else [],
+                        },
+                    ),
+                },
+            }
+        )
+
+    return {"type": "FeatureCollection", "features": features}
+
+
+def _incident_estimated_center(valid_regions: List[Dict[str, Any]]) -> Dict[str, float]:
+    x = 0.0
+    y = 0.0
+    z = 0.0
+    for region in valid_regions:
+        lat_rad = math.radians(float(region["latitude"]))
+        lon_rad = math.radians(float(region["longitude"]))
+        x += math.cos(lat_rad) * math.cos(lon_rad)
+        y += math.cos(lat_rad) * math.sin(lon_rad)
+        z += math.sin(lat_rad)
+
+    total = float(len(valid_regions))
+    if total <= 0:
+        return {"latitude": 0.0, "longitude": 0.0}
+
+    x /= total
+    y /= total
+    z /= total
+    hyp = math.sqrt((x * x) + (y * y))
+    if hyp == 0:
+        return {"latitude": 0.0, "longitude": 0.0}
+
+    return {
+        "latitude": math.degrees(math.atan2(z, hyp)),
+        "longitude": math.degrees(math.atan2(y, x)),
+    }
+
+
+def _incident_estimated_radius_km(center: Dict[str, float], valid_regions: List[Dict[str, Any]]) -> float:
+    center_coords = (float(center["longitude"]), float(center["latitude"]))
+    distances = [
+        haversine_distance(
+            center_coords,
+            (float(region["longitude"]), float(region["latitude"])),
+        )
+        for region in valid_regions
+    ]
+    return round(max(distances) if distances else 0.0, 1)
+
+
+def _normalize_geo_regions(regions: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
+    normalized: list[dict[str, Any]] = []
+    seen: set[tuple[Any, ...]] = set()
+    for region in regions:
+        if not isinstance(region, dict):
+            continue
+        latitude = region.get("latitude")
+        longitude = region.get("longitude")
+        if not isinstance(latitude, (int, float)) or not isinstance(longitude, (int, float)):
+            continue
+        item = {
+            "collector": region.get("collector"),
+            "country": region.get("country"),
+            "city": region.get("city"),
+            "latitude": float(latitude),
+            "longitude": float(longitude),
+        }
+        key = (
+            item["collector"],
+            item["country"],
+            item["city"],
+            item["latitude"],
+            item["longitude"],
+        )
+        if key in seen:
+            continue
+        seen.add(key)
+        normalized.append(item)
+    return normalized
+
+
+def _extract_evidence_geography_hint(evidence: Dict[str, Any]) -> Dict[str, Any] | None:
+    prefix_geo_regions = []
+    prefix_regions = []
+    asn_regions = []
+
+    evidence_prefix_geography = evidence.get("prefix_geography") or {}
+    prefix_geo_regions.extend(
+        _normalize_geo_regions(evidence_prefix_geography.get("regions") or [])
+    )
+
+    prefix_scope = evidence.get("prefix_scope") or {}
+    prefix_regions.extend(_normalize_geo_regions(prefix_scope.get("regions") or []))
+
+    for profile_key in ("origin_asn_profile", "new_origin_asn_profile"):
+        profile = evidence.get(profile_key) or {}
+        latitude = profile.get("latitude")
+        longitude = profile.get("longitude")
+        if isinstance(latitude, (int, float)) and isinstance(longitude, (int, float)):
+            asn_regions.append(
+                {
+                    "country": profile.get("country"),
+                    "city": profile.get("city"),
+                    "latitude": float(latitude),
+                    "longitude": float(longitude),
+                }
+            )
+
+    prefix_geo_regions = _normalize_geo_regions(prefix_geo_regions)
+    prefix_regions = _normalize_geo_regions(prefix_regions)
+    asn_regions = _normalize_geo_regions(asn_regions)
+
+    if prefix_geo_regions:
+        return {"regions": prefix_geo_regions, "geography_mode": "prefix_geography"}
+    if prefix_regions:
+        return {"regions": prefix_regions, "geography_mode": "prefix_scope"}
+    if asn_regions:
+        return {"regions": asn_regions, "geography_mode": "asn_region"}
+    return None
+
+
+async def build_incident_geography_hints(
+    db: AsyncSession,
+    records: List[BGPIncident],
+) -> Dict[str, Dict[str, Any]]:
+    evidence_refs = sorted(
+        {
+            str(ref)
+            for record in records
+            for ref in (record.evidence_refs or [])
+            if ref
+        }
+    )
+
+    anomalies = []
+    if evidence_refs:
+        result = await db.execute(
+            select(BGPAnomaly).where(BGPAnomaly.entity_key.in_(evidence_refs))
+        )
+        anomalies = result.scalars().all()
+    anomaly_by_key = {
+        str(anomaly.entity_key): anomaly
+        for anomaly in anomalies
+        if anomaly.entity_key
+    }
+
+    hints: Dict[str, Dict[str, Any]] = {}
+    for record in records:
+        merged_hint: Dict[str, Any] | None = None
+        priority = {"prefix_geography": 3, "prefix_scope": 2, "asn_region": 1}
+
+        for ref in record.evidence_refs or []:
+            anomaly = anomaly_by_key.get(str(ref))
+            if anomaly is None:
+                continue
+            hint = _extract_evidence_geography_hint(anomaly.evidence or {})
+            if hint is None:
+                continue
+            if merged_hint is None:
+                merged_hint = {
+                    "regions": list(hint["regions"]),
+                    "geography_mode": hint["geography_mode"],
+                }
+                continue
+            if priority[hint["geography_mode"]] > priority[merged_hint["geography_mode"]]:
+                merged_hint = {
+                    "regions": list(hint["regions"]),
+                    "geography_mode": hint["geography_mode"],
+                }
+            elif priority[hint["geography_mode"]] == priority[merged_hint["geography_mode"]]:
+                merged_hint["regions"].extend(hint["regions"])
+
+        if merged_hint:
+            merged_hint["regions"] = _normalize_geo_regions(merged_hint["regions"])
+            hints[record.incident_key] = merged_hint
+
+    return hints
+
+
+def convert_bgp_incidents_to_geojson(
+    records: List[BGPIncident],
+    geography_hints: Optional[Dict[str, Dict[str, Any]]] = None,
+) -> Dict[str, Any]:
+    features = []
+
+    for record in records:
+        hint = (geography_hints or {}).get(record.incident_key, {})
+        regions = hint.get("regions") or (record.affected_regions or [])
+        if not regions:
+            continue
+
+        valid_regions = _normalize_geo_regions(regions)
+        if not valid_regions:
+            continue
+
+        estimated_center = _incident_estimated_center(valid_regions)
+        estimated_radius_km = _incident_estimated_radius_km(estimated_center, valid_regions)
+
+        features.append(
+            {
+                "type": "Feature",
+                "geometry": {
+                    "type": "Point",
+                    "coordinates": [
+                        estimated_center["longitude"],
+                        estimated_center["latitude"],
+                    ],
+                },
+                "properties": {
+                    "id": record.id,
+                    "incident_key": record.incident_key,
+                    "incident_type": record.incident_type,
+                    "title": record.title,
+                    "summary": record.summary,
+                    "severity": record.severity,
+                    "status": record.status,
+                    "confidence": record.confidence,
+                    "affected_prefixes": record.affected_prefixes or [],
+                    "affected_asns": record.affected_asns or [],
+                    "affected_collectors": record.affected_collectors or [],
+                    "affected_regions": valid_regions,
+                    "estimated_center": estimated_center,
+                    "estimated_radius_km": estimated_radius_km,
+                    "geography_mode": hint.get("geography_mode") or "collector_centroid",
+                    "related_cables": record.related_cables or [],
+                    "related_ixps": record.related_ixps or [],
+                    "created_at": to_iso8601_utc(record.created_at),
+                    "started_at": to_iso8601_utc(record.started_at),
+                },
+            }
+        )
+
+    return {"type": "FeatureCollection", "features": features}
+
+
 # ============== API Endpoints ==============
 
 
@@ -256,9 +764,7 @@ def convert_gpu_cluster_to_geojson(records: List[CollectedData]) -> Dict[str, An
 async def get_cables_geojson(db: AsyncSession = Depends(get_db)):
     """获取海底电缆 GeoJSON 数据 (LineString)"""
     try:
-        stmt = select(CollectedData).where(CollectedData.source == "arcgis_cables")
-        result = await db.execute(stmt)
-        records = result.scalars().all()
+        records = await _load_current_collected_data(db, "arcgis_cables")
 
         if not records:
             raise HTTPException(
@@ -276,36 +782,14 @@ async def get_cables_geojson(db: AsyncSession = Depends(get_db)):
 @router.get("/geo/landing-points")
 async def get_landing_points_geojson(db: AsyncSession = Depends(get_db)):
     try:
-        landing_stmt = select(CollectedData).where(CollectedData.source == "arcgis_landing_points")
-        landing_result = await db.execute(landing_stmt)
-        records = landing_result.scalars().all()
-        
-        relation_stmt = select(CollectedData).where(CollectedData.source == "arcgis_cable_landing_relation")
-        relation_result = await db.execute(relation_stmt)
-        relation_records = relation_result.scalars().all()
-        
-        cable_stmt = select(CollectedData).where(CollectedData.source == "arcgis_cables")
-        cable_result = await db.execute(cable_stmt)
-        cable_records = cable_result.scalars().all()
-        
-        city_to_cable_ids_map = {}
-        for rel in relation_records:
-            if rel.extra_data:
-                city_id = rel.extra_data.get("city_id")
-                cable_id = rel.extra_data.get("cable_id")
-                if city_id is not None and cable_id is not None:
-                    if city_id not in city_to_cable_ids_map:
-                        city_to_cable_ids_map[city_id] = []
-                    if cable_id not in city_to_cable_ids_map[city_id]:
-                        city_to_cable_ids_map[city_id].append(cable_id)
-        
-        cable_id_to_name_map = {}
-        for cable in cable_records:
-            if cable.extra_data:
-                cable_id = cable.extra_data.get("cable_id")
-                cable_name = cable.name
-                if cable_id and cable_name:
-                    cable_id_to_name_map[cable_id] = cable_name
+        records = await _load_current_collected_data(db, "arcgis_landing_points")
+        relation_records = await _load_current_collected_data(db, "arcgis_cable_landing_relation")
+        cable_records = await _load_current_collected_data(db, "arcgis_cables")
+
+        city_to_cable_ids_map, cable_id_to_name_map = _build_landing_point_cable_maps(
+            relation_records,
+            cable_records,
+        )
         
         if not records:
             raise HTTPException(
@@ -322,36 +806,21 @@ async def get_landing_points_geojson(db: AsyncSession = Depends(get_db)):
 
 @router.get("/geo/all")
 async def get_all_geojson(db: AsyncSession = Depends(get_db)):
-    cables_stmt = select(CollectedData).where(CollectedData.source == "arcgis_cables")
-    cables_result = await db.execute(cables_stmt)
-    cables_records = cables_result.scalars().all()
-
-    points_stmt = select(CollectedData).where(CollectedData.source == "arcgis_landing_points")
-    points_result = await db.execute(points_stmt)
-    points_records = points_result.scalars().all()
-    
-    relation_stmt = select(CollectedData).where(CollectedData.source == "arcgis_cable_landing_relation")
-    relation_result = await db.execute(relation_stmt)
-    relation_records = relation_result.scalars().all()
-    
-    city_to_cable_ids_map = {}
-    for rel in relation_records:
-        if rel.extra_data:
-            city_id = rel.extra_data.get("city_id")
-            cable_id = rel.extra_data.get("cable_id")
-            if city_id is not None and cable_id is not None:
-                if city_id not in city_to_cable_ids_map:
-                    city_to_cable_ids_map[city_id] = []
-                if cable_id not in city_to_cable_ids_map[city_id]:
-                    city_to_cable_ids_map[city_id].append(cable_id)
-    
-    cable_id_to_name_map = {}
-    for cable in cables_records:
-        if cable.extra_data:
-            cable_id = cable.extra_data.get("cable_id")
-            cable_name = cable.name
-            if cable_id and cable_name:
-                cable_id_to_name_map[cable_id] = cable_name
+    records_by_source = await _load_current_collected_data_by_sources(
+        db,
+        [
+            "arcgis_cables",
+            "arcgis_landing_points",
+            "arcgis_cable_landing_relation",
+        ],
+    )
+    cables_records = records_by_source.get("arcgis_cables", [])
+    points_records = records_by_source.get("arcgis_landing_points", [])
+    relation_records = records_by_source.get("arcgis_cable_landing_relation", [])
+    city_to_cable_ids_map, cable_id_to_name_map = _build_landing_point_cable_maps(
+        relation_records,
+        cables_records,
+    )
 
     cables = (
         convert_cable_to_geojson(cables_records)
@@ -376,19 +845,20 @@ async def get_all_geojson(db: AsyncSession = Depends(get_db)):
 
 @router.get("/geo/satellites")
 async def get_satellites_geojson(
-    limit: int = 10000,
+    limit: Optional[int] = Query(
+        None,
+        ge=1,
+        description="Maximum number of satellites to return. Omit for no limit.",
+    ),
     db: AsyncSession = Depends(get_db),
 ):
     """获取卫星 TLE GeoJSON 数据"""
-    stmt = (
-        select(CollectedData)
-        .where(CollectedData.source == "celestrak_tle")
-        .where(CollectedData.name != "Unknown")
-        .order_by(CollectedData.id.desc())
-        .limit(limit)
+    records = await _load_current_collected_data(
+        db,
+        "celestrak_tle",
+        exclude_unknown_name=True,
+        limit=limit,
     )
-    result = await db.execute(stmt)
-    records = result.scalars().all()
 
     if not records:
         return {"type": "FeatureCollection", "features": [], "count": 0}
@@ -406,14 +876,12 @@ async def get_supercomputers_geojson(
     db: AsyncSession = Depends(get_db),
 ):
     """获取 TOP500 超算中心 GeoJSON 数据"""
-    stmt = (
-        select(CollectedData)
-        .where(CollectedData.source == "top500")
-        .where(CollectedData.name != "Unknown")
-        .limit(limit)
+    records = await _load_current_collected_data(
+        db,
+        "top500",
+        exclude_unknown_name=True,
+        limit=limit,
     )
-    result = await db.execute(stmt)
-    records = result.scalars().all()
 
     if not records:
         return {"type": "FeatureCollection", "features": [], "count": 0}
@@ -431,14 +899,12 @@ async def get_gpu_clusters_geojson(
     db: AsyncSession = Depends(get_db),
 ):
     """获取 GPU 集群 GeoJSON 数据"""
-    stmt = (
-        select(CollectedData)
-        .where(CollectedData.source == "epoch_ai_gpu")
-        .where(CollectedData.name != "Unknown")
-        .limit(limit)
+    records = await _load_current_collected_data(
+        db,
+        "epoch_ai_gpu",
+        exclude_unknown_name=True,
+        limit=limit,
     )
-    result = await db.execute(stmt)
-    records = result.scalars().all()
 
     if not records:
         return {"type": "FeatureCollection", "features": [], "count": 0}
@@ -450,6 +916,61 @@ async def get_gpu_clusters_geojson(
     }
 
 
+@router.get("/geo/bgp-anomalies")
+async def get_bgp_anomalies_geojson(
+    severity: Optional[str] = Query(None),
+    status: Optional[str] = Query("active"),
+    limit: int = Query(200, ge=1, le=1000),
+    db: AsyncSession = Depends(get_db),
+):
+    stmt = select(BGPAnomaly).order_by(BGPAnomaly.created_at.desc()).limit(limit)
+    if severity:
+        stmt = stmt.where(BGPAnomaly.severity == severity)
+    if status:
+        stmt = stmt.where(BGPAnomaly.status == status)
+
+    result = await db.execute(stmt)
+    records = list(result.scalars().all())
+    geography_hints = await build_anomaly_geography_hints(db, records)
+    geojson = convert_bgp_anomalies_to_geojson(records, geography_hints)
+    return {**geojson, "count": len(geojson.get("features", []))}
+
+
+@router.get("/geo/bgp-incidents")
+async def get_bgp_incidents_geojson(
+    severity: Optional[str] = Query(None),
+    status: Optional[str] = Query("active"),
+    limit: int = Query(100, ge=1, le=500),
+    db: AsyncSession = Depends(get_db),
+):
+    stmt = select(BGPIncident).order_by(BGPIncident.created_at.desc()).limit(limit)
+    if severity:
+        stmt = stmt.where(BGPIncident.severity == severity)
+    if status:
+        stmt = stmt.where(BGPIncident.status == status)
+
+    result = await db.execute(stmt)
+    records = list(result.scalars().all())
+    geography_hints = await build_incident_geography_hints(db, records)
+    geojson = convert_bgp_incidents_to_geojson(records, geography_hints)
+    return {**geojson, "count": len(geojson.get("features", []))}
+
+
+@router.get("/geo/bgp-collectors")
+async def get_bgp_collectors_geojson(db: AsyncSession = Depends(get_db)):
+    coverage = await build_bgp_collector_coverage(
+        db,
+        source_filter=("ris_live_bgp", "bgpstream_bgp"),
+    )
+    coverage_by_collector = {
+        item["collector"]: item
+        for item in coverage
+        if item.get("collector")
+    }
+    geojson = convert_bgp_collectors_to_geojson(coverage_by_collector)
+    return {**geojson, "count": len(geojson.get("features", []))}
+
+
 @router.get("/all")
 async def get_all_visualization_data(db: AsyncSession = Depends(get_db)):
     """获取所有可视化数据的统一端点
@@ -461,37 +982,27 @@ async def get_all_visualization_data(db: AsyncSession = Depends(get_db)):
     - supercomputers: TOP500 超算
     - gpu_clusters: GPU 集群
     """
-    cables_stmt = select(CollectedData).where(CollectedData.source == "arcgis_cables")
-    cables_result = await db.execute(cables_stmt)
-    cables_records = list(cables_result.scalars().all())
-
-    points_stmt = select(CollectedData).where(CollectedData.source == "arcgis_landing_points")
-    points_result = await db.execute(points_stmt)
-    points_records = list(points_result.scalars().all())
-
-    satellites_stmt = (
-        select(CollectedData)
-        .where(CollectedData.source == "celestrak_tle")
-        .where(CollectedData.name != "Unknown")
+    records_by_source = await _load_current_collected_data_by_sources(
+        db,
+        [
+            "arcgis_cables",
+            "arcgis_landing_points",
+            "celestrak_tle",
+            "top500",
+            "epoch_ai_gpu",
+        ],
     )
-    satellites_result = await db.execute(satellites_stmt)
-    satellites_records = list(satellites_result.scalars().all())
-
-    supercomputers_stmt = (
-        select(CollectedData)
-        .where(CollectedData.source == "top500")
-        .where(CollectedData.name != "Unknown")
+    cables_records = records_by_source.get("arcgis_cables", [])
+    points_records = records_by_source.get("arcgis_landing_points", [])
+    satellites_records = _filter_known_records(
+        records_by_source.get("celestrak_tle", []),
     )
-    supercomputers_result = await db.execute(supercomputers_stmt)
-    supercomputers_records = list(supercomputers_result.scalars().all())
-
-    gpu_stmt = (
-        select(CollectedData)
-        .where(CollectedData.source == "epoch_ai_gpu")
-        .where(CollectedData.name != "Unknown")
+    supercomputers_records = _filter_known_records(
+        records_by_source.get("top500", []),
+    )
+    gpu_records = _filter_known_records(
+        records_by_source.get("epoch_ai_gpu", []),
     )
-    gpu_result = await db.execute(gpu_stmt)
-    gpu_records = list(gpu_result.scalars().all())
 
     cables = (
         convert_cable_to_geojson(cables_records)
@@ -520,7 +1031,7 @@ async def get_all_visualization_data(db: AsyncSession = Depends(get_db)):
     )
 
     return {
-        "generated_at": datetime.utcnow().isoformat() + "Z",
+        "generated_at": to_iso8601_utc(datetime.now(UTC)),
         "version": "1.0",
         "data": {
             "satellites": satellites,
@@ -555,13 +1066,8 @@ async def get_cable_graph(db: AsyncSession) -> CableGraph:
     global _cable_graph
 
     if _cable_graph is None:
-        cables_stmt = select(CollectedData).where(CollectedData.source == "arcgis_cables")
-        cables_result = await db.execute(cables_stmt)
-        cables_records = list(cables_result.scalars().all())
-
-        points_stmt = select(CollectedData).where(CollectedData.source == "arcgis_landing_points")
-        points_result = await db.execute(points_stmt)
-        points_records = list(points_result.scalars().all())
+        cables_records = await _load_current_collected_data(db, "arcgis_cables")
+        points_records = await _load_current_collected_data(db, "arcgis_landing_points")
 
         cables_data = convert_cable_to_geojson(cables_records)
         points_data = convert_landing_point_to_geojson(points_records)

backend/app/api/v1/websocket.py

@@ -3,13 +3,14 @@
 import asyncio
 import json
 import logging
-from datetime import datetime
+from datetime import UTC, datetime
 from typing import Optional
 
 from fastapi import APIRouter, WebSocket, WebSocketDisconnect, Query
 from jose import jwt, JWTError
 
 from app.core.config import settings
+from app.core.time import to_iso8601_utc
 from app.core.websocket.manager import manager
 
 logger = logging.getLogger(__name__)
@@ -59,6 +60,7 @@ async def websocket_endpoint(
                         "ixp_nodes",
                         "alerts",
                         "dashboard",
+                        "datasource_tasks",
                     ],
                 },
             }
@@ -72,7 +74,7 @@ async def websocket_endpoint(
                     await websocket.send_json(
                         {
                             "type": "heartbeat",
-                            "data": {"action": "pong", "timestamp": datetime.utcnow().isoformat()},
+                            "data": {"action": "pong", "timestamp": to_iso8601_utc(datetime.now(UTC))},
                         }
                     )
                 elif data.get("type") == "subscribe":

backend/app/core/collected_data_fields.py

@@ -0,0 +1,62 @@
+from typing import Any, Dict, Optional
+
+
+FIELD_ALIASES = {
+    "country": ("country",),
+    "city": ("city",),
+    "latitude": ("latitude",),
+    "longitude": ("longitude",),
+    "value": ("value",),
+    "unit": ("unit",),
+    "cores": ("cores",),
+    "rmax": ("rmax", "r_max"),
+    "rpeak": ("rpeak", "r_peak"),
+    "power": ("power",),
+}
+
+
+def get_metadata_field(metadata: Optional[Dict[str, Any]], field: str, fallback: Any = None) -> Any:
+    if isinstance(metadata, dict):
+        for key in FIELD_ALIASES.get(field, (field,)):
+            value = metadata.get(key)
+            if value not in (None, ""):
+                return value
+    return fallback
+
+
+def build_dynamic_metadata(
+    metadata: Optional[Dict[str, Any]],
+    *,
+    country: Any = None,
+    city: Any = None,
+    latitude: Any = None,
+    longitude: Any = None,
+    value: Any = None,
+    unit: Any = None,
+) -> Dict[str, Any]:
+    merged = dict(metadata) if isinstance(metadata, dict) else {}
+
+    fallbacks = {
+        "country": country,
+        "city": city,
+        "latitude": latitude,
+        "longitude": longitude,
+        "value": value,
+        "unit": unit,
+    }
+
+    for field, fallback in fallbacks.items():
+        if fallback not in (None, "") and get_metadata_field(merged, field) in (None, ""):
+            merged[field] = fallback
+
+    return merged
+
+
+def get_record_field(record: Any, field: str) -> Any:
+    metadata = getattr(record, "extra_data", None) or {}
+    fallback_attr = field
+    if field in {"cores", "rmax", "rpeak", "power"}:
+        fallback = None
+    else:
+        fallback = getattr(record, fallback_attr, None)
+    return get_metadata_field(metadata, field, fallback=fallback)

backend/app/core/config.py

@@ -6,9 +6,16 @@ import os
 from pydantic_settings import BaseSettings
 
 
+ROOT_DIR = Path(__file__).parent.parent.parent.parent
+VERSION_FILE = ROOT_DIR / "VERSION"
+
+
 class Settings(BaseSettings):
     PROJECT_NAME: str = "Intelligent Planet Plan"
-    VERSION: str = "1.0.0"
+    VERSION: str = (
+        os.getenv("APP_VERSION")
+        or (VERSION_FILE.read_text(encoding="utf-8").strip() if VERSION_FILE.exists() else "0.19.0")
+    )
     API_V1_STR: str = "/api/v1"
     SECRET_KEY: str = "your-secret-key-change-in-production"
     ALGORITHM: str = "HS256"
@@ -30,6 +37,11 @@ class Settings(BaseSettings):
     SPACETRACK_USERNAME: str = ""
     SPACETRACK_PASSWORD: str = ""
 
+    AI_PROVIDER_SERVICE_URL: str = "http://localhost:8010"
+    AI_PROVIDER_SERVICE_TOKEN: str = ""
+    AI_PROVIDER_TIMEOUT_SECONDS: int = 60
+    AI_PROVIDER_RETRY_ATTEMPTS: int = 2
+
     @property
     def REDIS_URL(self) -> str:
         return os.getenv(
@@ -39,6 +51,7 @@ class Settings(BaseSettings):
     class Config:
         env_file = Path(__file__).parent.parent.parent / ".env"
         case_sensitive = True
+        extra = "ignore"
 
 
 @lru_cache()

backend/app/core/countries.py

@@ -0,0 +1,338 @@
+import re
+from typing import Any, Optional
+
+
+COUNTRY_ENTRIES = [
+    ("阿富汗", ["Afghanistan", "AF", "AFG"]),
+    ("阿尔巴尼亚", ["Albania", "AL", "ALB"]),
+    ("阿尔及利亚", ["Algeria", "DZ", "DZA"]),
+    ("安道尔", ["Andorra", "AD", "AND"]),
+    ("安哥拉", ["Angola", "AO", "AGO"]),
+    ("安提瓜和巴布达", ["Antigua and Barbuda", "AG", "ATG"]),
+    ("阿根廷", ["Argentina", "AR", "ARG"]),
+    ("亚美尼亚", ["Armenia", "AM", "ARM"]),
+    ("澳大利亚", ["Australia", "AU", "AUS"]),
+    ("奥地利", ["Austria", "AT", "AUT"]),
+    ("阿塞拜疆", ["Azerbaijan", "AZ", "AZE"]),
+    ("巴哈马", ["Bahamas", "BS", "BHS"]),
+    ("巴林", ["Bahrain", "BH", "BHR"]),
+    ("孟加拉国", ["Bangladesh", "BD", "BGD"]),
+    ("巴巴多斯", ["Barbados", "BB", "BRB"]),
+    ("白俄罗斯", ["Belarus", "BY", "BLR"]),
+    ("比利时", ["Belgium", "BE", "BEL"]),
+    ("伯利兹", ["Belize", "BZ", "BLZ"]),
+    ("贝宁", ["Benin", "BJ", "BEN"]),
+    ("不丹", ["Bhutan", "BT", "BTN"]),
+    ("玻利维亚", ["Bolivia", "BO", "BOL", "Bolivia (Plurinational State of)"]),
+    ("波斯尼亚和黑塞哥维那", ["Bosnia and Herzegovina", "BA", "BIH"]),
+    ("博茨瓦纳", ["Botswana", "BW", "BWA"]),
+    ("巴西", ["Brazil", "BR", "BRA"]),
+    ("文莱", ["Brunei", "BN", "BRN", "Brunei Darussalam"]),
+    ("保加利亚", ["Bulgaria", "BG", "BGR"]),
+    ("布基纳法索", ["Burkina Faso", "BF", "BFA"]),
+    ("布隆迪", ["Burundi", "BI", "BDI"]),
+    ("柬埔寨", ["Cambodia", "KH", "KHM"]),
+    ("喀麦隆", ["Cameroon", "CM", "CMR"]),
+    ("加拿大", ["Canada", "CA", "CAN"]),
+    ("佛得角", ["Cape Verde", "CV", "CPV", "Cabo Verde"]),
+    ("中非", ["Central African Republic", "CF", "CAF"]),
+    ("乍得", ["Chad", "TD", "TCD"]),
+    ("智利", ["Chile", "CL", "CHL"]),
+    ("中国", ["China", "CN", "CHN", "Mainland China", "PRC", "People's Republic of China"]),
+    ("中国（香港）", ["Hong Kong", "HK", "HKG", "Hong Kong SAR", "China Hong Kong", "Hong Kong, China"]),
+    ("中国（澳门）", ["Macao", "Macau", "MO", "MAC", "Macao SAR", "China Macao", "Macau, China"]),
+    ("中国（台湾）", ["Taiwan", "TW", "TWN", "Chinese Taipei", "Taiwan, China"]),
+    ("哥伦比亚", ["Colombia", "CO", "COL"]),
+    ("科摩罗", ["Comoros", "KM", "COM"]),
+    ("刚果（布）", ["Republic of the Congo", "Congo", "Congo-Brazzaville", "CG", "COG"]),
+    ("刚果（金）", ["Democratic Republic of the Congo", "DR Congo", "Congo-Kinshasa", "CD", "COD"]),
+    ("哥斯达黎加", ["Costa Rica", "CR", "CRI"]),
+    ("科特迪瓦", ["Cote d'Ivoire", "Côte d'Ivoire", "Ivory Coast", "CI", "CIV"]),
+    ("克罗地亚", ["Croatia", "HR", "HRV"]),
+    ("古巴", ["Cuba", "CU", "CUB"]),
+    ("塞浦路斯", ["Cyprus", "CY", "CYP"]),
+    ("捷克", ["Czech Republic", "Czechia", "CZ", "CZE"]),
+    ("丹麦", ["Denmark", "DK", "DNK"]),
+    ("吉布提", ["Djibouti", "DJ", "DJI"]),
+    ("多米尼克", ["Dominica", "DM", "DMA"]),
+    ("多米尼加", ["Dominican Republic", "DO", "DOM"]),
+    ("厄瓜多尔", ["Ecuador", "EC", "ECU"]),
+    ("埃及", ["Egypt", "EG", "EGY"]),
+    ("萨尔瓦多", ["El Salvador", "SV", "SLV"]),
+    ("赤道几内亚", ["Equatorial Guinea", "GQ", "GNQ"]),
+    ("厄立特里亚", ["Eritrea", "ER", "ERI"]),
+    ("爱沙尼亚", ["Estonia", "EE", "EST"]),
+    ("埃斯瓦蒂尼", ["Eswatini", "SZ", "SWZ", "Swaziland"]),
+    ("埃塞俄比亚", ["Ethiopia", "ET", "ETH"]),
+    ("斐济", ["Fiji", "FJ", "FJI"]),
+    ("芬兰", ["Finland", "FI", "FIN"]),
+    ("法国", ["France", "FR", "FRA"]),
+    ("加蓬", ["Gabon", "GA", "GAB"]),
+    ("冈比亚", ["Gambia", "GM", "GMB"]),
+    ("格鲁吉亚", ["Georgia", "GE", "GEO"]),
+    ("德国", ["Germany", "DE", "DEU"]),
+    ("加纳", ["Ghana", "GH", "GHA"]),
+    ("希腊", ["Greece", "GR", "GRC"]),
+    ("格林纳达", ["Grenada", "GD", "GRD"]),
+    ("危地马拉", ["Guatemala", "GT", "GTM"]),
+    ("几内亚", ["Guinea", "GN", "GIN"]),
+    ("几内亚比绍", ["Guinea-Bissau", "GW", "GNB"]),
+    ("圭亚那", ["Guyana", "GY", "GUY"]),
+    ("海地", ["Haiti", "HT", "HTI"]),
+    ("洪都拉斯", ["Honduras", "HN", "HND"]),
+    ("匈牙利", ["Hungary", "HU", "HUN"]),
+    ("冰岛", ["Iceland", "IS", "ISL"]),
+    ("印度", ["India", "IN", "IND"]),
+    ("印度尼西亚", ["Indonesia", "ID", "IDN"]),
+    ("伊朗", ["Iran", "IR", "IRN", "Iran (Islamic Republic of)"]),
+    ("伊拉克", ["Iraq", "IQ", "IRQ"]),
+    ("爱尔兰", ["Ireland", "IE", "IRL"]),
+    ("以色列", ["Israel", "IL", "ISR"]),
+    ("意大利", ["Italy", "IT", "ITA"]),
+    ("牙买加", ["Jamaica", "JM", "JAM"]),
+    ("日本", ["Japan", "JP", "JPN"]),
+    ("约旦", ["Jordan", "JO", "JOR"]),
+    ("哈萨克斯坦", ["Kazakhstan", "KZ", "KAZ"]),
+    ("肯尼亚", ["Kenya", "KE", "KEN"]),
+    ("基里巴斯", ["Kiribati", "KI", "KIR"]),
+    ("朝鲜", ["North Korea", "Korea, DPRK", "Democratic People's Republic of Korea", "KP", "PRK"]),
+    ("韩国", ["South Korea", "Republic of Korea", "Korea", "KR", "KOR"]),
+    ("科威特", ["Kuwait", "KW", "KWT"]),
+    ("吉尔吉斯斯坦", ["Kyrgyzstan", "KG", "KGZ"]),
+    ("老挝", ["Laos", "Lao PDR", "Lao People's Democratic Republic", "LA", "LAO"]),
+    ("拉脱维亚", ["Latvia", "LV", "LVA"]),
+    ("黎巴嫩", ["Lebanon", "LB", "LBN"]),
+    ("莱索托", ["Lesotho", "LS", "LSO"]),
+    ("利比里亚", ["Liberia", "LR", "LBR"]),
+    ("利比亚", ["Libya", "LY", "LBY"]),
+    ("列支敦士登", ["Liechtenstein", "LI", "LIE"]),
+    ("立陶宛", ["Lithuania", "LT", "LTU"]),
+    ("卢森堡", ["Luxembourg", "LU", "LUX"]),
+    ("马达加斯加", ["Madagascar", "MG", "MDG"]),
+    ("马拉维", ["Malawi", "MW", "MWI"]),
+    ("马来西亚", ["Malaysia", "MY", "MYS"]),
+    ("马尔代夫", ["Maldives", "MV", "MDV"]),
+    ("马里", ["Mali", "ML", "MLI"]),
+    ("马耳他", ["Malta", "MT", "MLT"]),
+    ("马绍尔群岛", ["Marshall Islands", "MH", "MHL"]),
+    ("毛里塔尼亚", ["Mauritania", "MR", "MRT"]),
+    ("毛里求斯", ["Mauritius", "MU", "MUS"]),
+    ("墨西哥", ["Mexico", "MX", "MEX"]),
+    ("密克罗尼西亚", ["Micronesia", "FM", "FSM", "Federated States of Micronesia"]),
+    ("摩尔多瓦", ["Moldova", "MD", "MDA", "Republic of Moldova"]),
+    ("摩纳哥", ["Monaco", "MC", "MCO"]),
+    ("蒙古", ["Mongolia", "MN", "MNG"]),
+    ("黑山", ["Montenegro", "ME", "MNE"]),
+    ("摩洛哥", ["Morocco", "MA", "MAR"]),
+    ("莫桑比克", ["Mozambique", "MZ", "MOZ"]),
+    ("缅甸", ["Myanmar", "MM", "MMR", "Burma"]),
+    ("纳米比亚", ["Namibia", "NA", "NAM"]),
+    ("瑙鲁", ["Nauru", "NR", "NRU"]),
+    ("尼泊尔", ["Nepal", "NP", "NPL"]),
+    ("荷兰", ["Netherlands", "NL", "NLD"]),
+    ("新西兰", ["New Zealand", "NZ", "NZL"]),
+    ("尼加拉瓜", ["Nicaragua", "NI", "NIC"]),
+    ("尼日尔", ["Niger", "NE", "NER"]),
+    ("尼日利亚", ["Nigeria", "NG", "NGA"]),
+    ("北马其顿", ["North Macedonia", "MK", "MKD", "Macedonia"]),
+    ("挪威", ["Norway", "NO", "NOR"]),
+    ("阿曼", ["Oman", "OM", "OMN"]),
+    ("巴基斯坦", ["Pakistan", "PK", "PAK"]),
+    ("帕劳", ["Palau", "PW", "PLW"]),
+    ("巴勒斯坦", ["Palestine", "PS", "PSE", "State of Palestine"]),
+    ("巴拿马", ["Panama", "PA", "PAN"]),
+    ("巴布亚新几内亚", ["Papua New Guinea", "PG", "PNG"]),
+    ("巴拉圭", ["Paraguay", "PY", "PRY"]),
+    ("秘鲁", ["Peru", "PE", "PER"]),
+    ("菲律宾", ["Philippines", "PH", "PHL"]),
+    ("波兰", ["Poland", "PL", "POL"]),
+    ("葡萄牙", ["Portugal", "PT", "PRT"]),
+    ("卡塔尔", ["Qatar", "QA", "QAT"]),
+    ("罗马尼亚", ["Romania", "RO", "ROU"]),
+    ("俄罗斯", ["Russia", "Russian Federation", "RU", "RUS"]),
+    ("卢旺达", ["Rwanda", "RW", "RWA"]),
+    ("圣基茨和尼维斯", ["Saint Kitts and Nevis", "KN", "KNA"]),
+    ("圣卢西亚", ["Saint Lucia", "LC", "LCA"]),
+    ("圣文森特和格林纳丁斯", ["Saint Vincent and the Grenadines", "VC", "VCT"]),
+    ("萨摩亚", ["Samoa", "WS", "WSM"]),
+    ("圣马力诺", ["San Marino", "SM", "SMR"]),
+    ("圣多美和普林西比", ["Sao Tome and Principe", "ST", "STP", "São Tomé and Príncipe"]),
+    ("沙特阿拉伯", ["Saudi Arabia", "SA", "SAU"]),
+    ("塞内加尔", ["Senegal", "SN", "SEN"]),
+    ("塞尔维亚", ["Serbia", "RS", "SRB", "Kosovo", "XK", "XKS", "Republic of Kosovo"]),
+    ("塞舌尔", ["Seychelles", "SC", "SYC"]),
+    ("塞拉利昂", ["Sierra Leone", "SL", "SLE"]),
+    ("新加坡", ["Singapore", "SG", "SGP"]),
+    ("斯洛伐克", ["Slovakia", "SK", "SVK"]),
+    ("斯洛文尼亚", ["Slovenia", "SI", "SVN"]),
+    ("所罗门群岛", ["Solomon Islands", "SB", "SLB"]),
+    ("索马里", ["Somalia", "SO", "SOM"]),
+    ("南非", ["South Africa", "ZA", "ZAF"]),
+    ("南苏丹", ["South Sudan", "SS", "SSD"]),
+    ("西班牙", ["Spain", "ES", "ESP"]),
+    ("斯里兰卡", ["Sri Lanka", "LK", "LKA"]),
+    ("苏丹", ["Sudan", "SD", "SDN"]),
+    ("苏里南", ["Suriname", "SR", "SUR"]),
+    ("瑞典", ["Sweden", "SE", "SWE"]),
+    ("瑞士", ["Switzerland", "CH", "CHE"]),
+    ("叙利亚", ["Syria", "SY", "SYR", "Syrian Arab Republic"]),
+    ("塔吉克斯坦", ["Tajikistan", "TJ", "TJK"]),
+    ("坦桑尼亚", ["Tanzania", "TZ", "TZA", "United Republic of Tanzania"]),
+    ("泰国", ["Thailand", "TH", "THA"]),
+    ("东帝汶", ["Timor-Leste", "East Timor", "TL", "TLS"]),
+    ("多哥", ["Togo", "TG", "TGO"]),
+    ("汤加", ["Tonga", "TO", "TON"]),
+    ("特立尼达和多巴哥", ["Trinidad and Tobago", "TT", "TTO"]),
+    ("突尼斯", ["Tunisia", "TN", "TUN"]),
+    ("土耳其", ["Turkey", "TR", "TUR", "Türkiye"]),
+    ("土库曼斯坦", ["Turkmenistan", "TM", "TKM"]),
+    ("图瓦卢", ["Tuvalu", "TV", "TUV"]),
+    ("乌干达", ["Uganda", "UG", "UGA"]),
+    ("乌克兰", ["Ukraine", "UA", "UKR"]),
+    ("阿联酋", ["United Arab Emirates", "AE", "ARE", "UAE"]),
+    ("英国", ["United Kingdom", "UK", "GB", "GBR", "Great Britain", "Britain", "England"]),
+    ("美国", ["United States", "United States of America", "US", "USA", "U.S.", "U.S.A."]),
+    ("乌拉圭", ["Uruguay", "UY", "URY"]),
+    ("乌兹别克斯坦", ["Uzbekistan", "UZ", "UZB"]),
+    ("瓦努阿图", ["Vanuatu", "VU", "VUT"]),
+    ("梵蒂冈", ["Vatican City", "Holy See", "VA", "VAT"]),
+    ("委内瑞拉", ["Venezuela", "VE", "VEN", "Venezuela (Bolivarian Republic of)"]),
+    ("越南", ["Vietnam", "Viet Nam", "VN", "VNM"]),
+    ("也门", ["Yemen", "YE", "YEM"]),
+    ("赞比亚", ["Zambia", "ZM", "ZMB"]),
+    ("津巴布韦", ["Zimbabwe", "ZW", "ZWE"]),
+]
+
+
+COUNTRY_OPTIONS = [entry[0] for entry in COUNTRY_ENTRIES]
+CANONICAL_COUNTRY_SET = set(COUNTRY_OPTIONS)
+INVALID_COUNTRY_VALUES = {
+    "",
+    "-",
+    "--",
+    "unknown",
+    "n/a",
+    "na",
+    "none",
+    "null",
+    "global",
+    "world",
+    "worldwide",
+    "xx",
+}
+NUMERIC_LIKE_PATTERN = re.compile(r"^[\d\s,._%+\-]+$")
+
+COUNTRY_ALIAS_MAP = {}
+COUNTRY_VARIANTS_MAP = {}
+for canonical, aliases in COUNTRY_ENTRIES:
+    COUNTRY_ALIAS_MAP[canonical.casefold()] = canonical
+    variants = [canonical, *aliases]
+    COUNTRY_VARIANTS_MAP[canonical] = variants
+    for alias in aliases:
+        COUNTRY_ALIAS_MAP[alias.casefold()] = canonical
+
+
+COUNTRY_CENTROIDS = {
+    "美国": {"latitude": 39.8283, "longitude": -98.5795},
+    "英国": {"latitude": 55.3781, "longitude": -3.4360},
+    "荷兰": {"latitude": 52.1326, "longitude": 5.2913},
+    "日本": {"latitude": 36.2048, "longitude": 138.2529},
+    "德国": {"latitude": 51.1657, "longitude": 10.4515},
+    "法国": {"latitude": 46.2276, "longitude": 2.2137},
+    "新加坡": {"latitude": 1.3521, "longitude": 103.8198},
+    "中国": {"latitude": 35.8617, "longitude": 104.1954},
+    "中国（香港）": {"latitude": 22.3193, "longitude": 114.1694},
+    "中国（台湾）": {"latitude": 23.6978, "longitude": 120.9605},
+    "韩国": {"latitude": 35.9078, "longitude": 127.7669},
+    "俄罗斯": {"latitude": 61.5240, "longitude": 105.3188},
+    "加拿大": {"latitude": 56.1304, "longitude": -106.3468},
+    "澳大利亚": {"latitude": -25.2744, "longitude": 133.7751},
+    "巴西": {"latitude": -14.2350, "longitude": -51.9253},
+    "南非": {"latitude": -30.5595, "longitude": 22.9375},
+    "西班牙": {"latitude": 40.4637, "longitude": -3.7492},
+    "意大利": {"latitude": 41.8719, "longitude": 12.5674},
+    "瑞士": {"latitude": 46.8182, "longitude": 8.2275},
+    "阿联酋": {"latitude": 23.4241, "longitude": 53.8478},
+    "莫桑比克": {"latitude": -18.6657, "longitude": 35.5296},
+    "哥斯达黎加": {"latitude": 9.7489, "longitude": -83.7534},
+    "尼日利亚": {"latitude": 9.0820, "longitude": 8.6753},
+    "印度尼西亚": {"latitude": -0.7893, "longitude": 113.9213},
+    "芬兰": {"latitude": 61.9241, "longitude": 25.7482},
+    "巴基斯坦": {"latitude": 30.3753, "longitude": 69.3451},
+    "泰国": {"latitude": 15.8700, "longitude": 100.9925},
+    "墨西哥": {"latitude": 23.6345, "longitude": -102.5528},
+    "安哥拉": {"latitude": -11.2027, "longitude": 17.8739},
+    "摩尔多瓦": {"latitude": 47.4116, "longitude": 28.3699},
+    "印度": {"latitude": 20.5937, "longitude": 78.9629},
+    "乌克兰": {"latitude": 48.3794, "longitude": 31.1656},
+    "阿富汗": {"latitude": 33.9391, "longitude": 67.7100},
+    "肯尼亚": {"latitude": -0.0236, "longitude": 37.9062},
+    "土耳其": {"latitude": 38.9637, "longitude": 35.2433},
+    "多米尼加": {"latitude": 18.7357, "longitude": -70.1627},
+    "叙利亚": {"latitude": 34.8021, "longitude": 38.9968},
+    "乌干达": {"latitude": 1.3733, "longitude": 32.2903},
+    "卢森堡": {"latitude": 49.8153, "longitude": 6.1296},
+    "罗马尼亚": {"latitude": 45.9432, "longitude": 24.9668},
+    "尼泊尔": {"latitude": 28.3949, "longitude": 84.1240},
+    "匈牙利": {"latitude": 47.1625, "longitude": 19.5033},
+    "埃及": {"latitude": 26.8206, "longitude": 30.8025},
+    "波兰": {"latitude": 51.9194, "longitude": 19.1451},
+    "哥伦比亚": {"latitude": 4.5709, "longitude": -74.2973},
+    "爱尔兰": {"latitude": 53.1424, "longitude": -7.6921},
+    "菲律宾": {"latitude": 12.8797, "longitude": 121.7740},
+}
+
+
+def normalize_country(value: Any) -> Optional[str]:
+    if value is None:
+        return None
+
+    if not isinstance(value, str):
+        return None
+
+    normalized = re.sub(r"\s+", " ", value.strip())
+    normalized = normalized.replace("(", "（").replace(")", "）")
+
+    if not normalized:
+        return None
+
+    lowered = normalized.casefold()
+    if lowered in INVALID_COUNTRY_VALUES:
+        return None
+
+    if NUMERIC_LIKE_PATTERN.fullmatch(normalized):
+        return None
+
+    if normalized in CANONICAL_COUNTRY_SET:
+        return normalized
+
+    return COUNTRY_ALIAS_MAP.get(lowered)
+
+
+def get_country_centroid(value: Any) -> Optional[dict[str, float]]:
+    canonical = normalize_country(value)
+    if not canonical:
+        return None
+    return COUNTRY_CENTROIDS.get(canonical)
+
+
+def get_country_search_variants(value: Any) -> list[str]:
+    canonical = normalize_country(value)
+    if canonical is None:
+        return []
+
+    variants = []
+    seen = set()
+    for item in COUNTRY_VARIANTS_MAP.get(canonical, [canonical]):
+        if not isinstance(item, str):
+            continue
+        normalized = re.sub(r"\s+", " ", item.strip())
+        if not normalized:
+            continue
+        key = normalized.casefold()
+        if key in seen:
+            continue
+        seen.add(key)
+        variants.append(normalized)
+
+    return variants

backend/app/core/data_sources.py

@@ -11,6 +11,7 @@ COLLECTOR_URL_KEYS = {
     "fao_landing_points": "fao.landing_point_url",
     "telegeography_cables": "telegeography.cable_url",
     "telegeography_landing": "telegeography.landing_point_url",
+    "telegeography_systems": "telegeography.cable_url",
     "huggingface_models": "huggingface.models_url",
     "huggingface_datasets": "huggingface.datasets_url",
     "huggingface_spaces": "huggingface.spaces_url",
@@ -23,6 +24,12 @@ COLLECTOR_URL_KEYS = {
     "top500": "top500.url",
     "epoch_ai_gpu": "epoch_ai.gpu_clusters_url",
     "spacetrack_tle": "spacetrack.tle_query_url",
+    "celestrak_tle": "celestrak.base_url",
+    "ris_live_bgp": "ris_live.url",
+    "bgpstream_bgp": "bgpstream.url",
+    "iptoasn_prefix_geo": "iptoasn.combined_url",
+    "opengeofeed_prefix_geo": "opengeofeed.public_csv_url",
+    "nro_delegated_prefix_geo": "nro.delegated_stats_url",
 }
 
 
@@ -36,18 +43,22 @@ class DataSourcesConfig:
             with open(config_path, "r") as f:
                 self._yaml_config = yaml.safe_load(f) or {}
 
-    def get_yaml_url(self, collector_name: str) -> str:
-        key = COLLECTOR_URL_KEYS.get(collector_name, "")
+    def get_yaml_value(self, key: str):
         if not key:
-            return ""
+            return None
 
         parts = key.split(".")
         value = self._yaml_config
         for part in parts:
             if isinstance(value, dict):
-                value = value.get(part, "")
+                value = value.get(part)
             else:
-                return ""
+                return None
+        return value
+
+    def get_yaml_url(self, collector_name: str) -> str:
+        key = COLLECTOR_URL_KEYS.get(collector_name, "")
+        value = self.get_yaml_value(key)
         return value if isinstance(value, str) else ""
 
     async def get_url(self, collector_name: str, db) -> str:

backend/app/core/data_sources.yaml

@@ -2,38 +2,87 @@
 # All external data source URLs should be configured here
 
 arcgis:
+  # ArcGIS 海缆 GeoJSON 查询接口
   cable_url: "https://services.arcgis.com/6DIQcwlPy8knb6sg/ArcGIS/rest/services/SubmarineCables/FeatureServer/2/query"
+  # ArcGIS 登陆点 GeoJSON 查询接口
   landing_point_url: "https://services.arcgis.com/6DIQcwlPy8knb6sg/ArcGIS/rest/services/SubmarineCables/FeatureServer/1/query"
+  # ArcGIS 海缆与登陆点关联关系查询接口
   cable_landing_relation_url: "https://services.arcgis.com/6DIQcwlPy8knb6sg/ArcGIS/rest/services/SubmarineCables/FeatureServer/3/query"
 
 fao:
+  # FAO 登陆点 CSV 下载地址
   landing_point_url: "https://data.apps.fao.org/catalog/dataset/1b75ff21-92f2-4b96-9b7b-98e8aa65ad5d/resource/b6071077-d1d4-4e97-aa00-42e902847c87/download/landing-point-geo.csv"
 
 telegeography:
+  # TeleGeography 海缆/系统主数据源，当前使用 GitHub 镜像 JSON
   cable_url: "https://raw.githubusercontent.com/lintaojlu/submarine_cable_information/main/cable.json"
+  # TeleGeography 登陆点主数据源，当前使用 GitHub 镜像 JSON
   landing_point_url: "https://raw.githubusercontent.com/lintaojlu/submarine_cable_information/main/landing_point.json"
+  # TeleGeography 历史 API 存档，用于 cable collector 的 fallback
+  archived_cable_url: "https://web.archive.org/web/2024/https://www.submarinecablemap.com/api/v3/cable"
+  # TeleGeography 官网页面，用于 cable collector 的最终 HTML 抓取 fallback
+  live_map_url: "https://www.submarinecablemap.com"
 
 huggingface:
+  # Hugging Face 模型目录 API
   models_url: "https://huggingface.co/api/models"
+  # Hugging Face 数据集目录 API
   datasets_url: "https://huggingface.co/api/datasets"
+  # Hugging Face Spaces 目录 API
   spaces_url: "https://huggingface.co/api/spaces"
 
 cloudflare:
+  # Cloudflare Radar 设备类型摘要接口
   radar_device_url: "https://api.cloudflare.com/client/v4/radar/http/summary/device_type"
+  # Cloudflare Radar 请求量时间序列接口
   radar_traffic_url: "https://api.cloudflare.com/client/v4/radar/http/timeseries/requests"
+  # Cloudflare Radar 热点地理位置接口
   radar_top_locations_url: "https://api.cloudflare.com/client/v4/radar/http/top/locations"
 
 peeringdb:
+  # PeeringDB IXP API
   ixp_url: "https://www.peeringdb.com/api/ix"
+  # PeeringDB Network API
   network_url: "https://www.peeringdb.com/api/net"
+  # PeeringDB Facility API
   facility_url: "https://www.peeringdb.com/api/fac"
 
 top500:
+  # TOP500 榜单页面，用于主表抓取
   url: "https://top500.org/lists/top500/list/2025/11/"
+  # TOP500 站点根地址，用于拼详情页链接
+  base_url: "https://top500.org"
 
 epoch_ai:
+  # Epoch AI GPU Cluster 页面
   gpu_clusters_url: "https://epoch.ai/data/gpu-clusters"
 
 spacetrack:
+  # Space-Track 站点根地址，用于首页访问和登录地址推导
   base_url: "https://www.space-track.org"
+  # Space-Track TLE 主查询接口
   tle_query_url: "https://www.space-track.org/basicspacedata/query/class/gp/orderby/EPOCH%20desc/limit/1000/format/json"
+
+celestrak:
+  # CelesTrak TLE 基础接口，collector 会在其后拼接 GROUP / FORMAT 参数
+  base_url: "https://celestrak.org/NORAD/elements/gp.php"
+
+ris_live:
+  # RIPE RIS Live 流式订阅地址
+  url: "https://ris-live.ripe.net/v1/stream/?format=json&client=planet-ris-live"
+
+bgpstream:
+  # CAIDA BGPStream Broker API
+  url: "https://broker.bgpstream.caida.org/v2"
+
+iptoasn:
+  # IPtoASN prefix geography 合并数据下载地址
+  combined_url: "https://iptoasn.com/data/ip2asn-combined.tsv.gz"
+
+opengeofeed:
+  # OpenGeoFeed 公共 geofeed CSV
+  public_csv_url: "https://opengeofeed.org/feed/public.csv"
+
+nro:
+  # NRO delegated stats 下载地址
+  delegated_stats_url: "https://ftp.ripe.net/pub/stats/ripencc/nro-stats/latest/nro-delegated-stats"

backend/app/core/datasource_defaults.py

@@ -0,0 +1,168 @@
+"""Default built-in datasource definitions."""
+
+DEFAULT_DATASOURCES = {
+    "top500": {
+        "id": 1,
+        "name": "TOP500 Supercomputers",
+        "module": "L1",
+        "priority": "P0",
+        "frequency_minutes": 240,
+    },
+    "epoch_ai_gpu": {
+        "id": 2,
+        "name": "Epoch AI GPU Clusters",
+        "module": "L1",
+        "priority": "P0",
+        "frequency_minutes": 360,
+    },
+    "huggingface_models": {
+        "id": 3,
+        "name": "HuggingFace Models",
+        "module": "L2",
+        "priority": "P1",
+        "frequency_minutes": 720,
+    },
+    "huggingface_datasets": {
+        "id": 4,
+        "name": "HuggingFace Datasets",
+        "module": "L2",
+        "priority": "P1",
+        "frequency_minutes": 720,
+    },
+    "huggingface_spaces": {
+        "id": 5,
+        "name": "HuggingFace Spaces",
+        "module": "L2",
+        "priority": "P2",
+        "frequency_minutes": 1440,
+    },
+    "peeringdb_ixp": {
+        "id": 6,
+        "name": "PeeringDB IXP",
+        "module": "L2",
+        "priority": "P1",
+        "frequency_minutes": 1440,
+    },
+    "peeringdb_network": {
+        "id": 7,
+        "name": "PeeringDB Networks",
+        "module": "L2",
+        "priority": "P2",
+        "frequency_minutes": 2880,
+    },
+    "peeringdb_facility": {
+        "id": 8,
+        "name": "PeeringDB Facilities",
+        "module": "L2",
+        "priority": "P2",
+        "frequency_minutes": 2880,
+    },
+    "telegeography_cables": {
+        "id": 9,
+        "name": "Submarine Cables",
+        "module": "L2",
+        "priority": "P1",
+        "frequency_minutes": 10080,
+    },
+    "telegeography_landing": {
+        "id": 10,
+        "name": "Cable Landing Points",
+        "module": "L2",
+        "priority": "P2",
+        "frequency_minutes": 10080,
+    },
+    "telegeography_systems": {
+        "id": 11,
+        "name": "Cable Systems",
+        "module": "L2",
+        "priority": "P2",
+        "frequency_minutes": 10080,
+    },
+    "arcgis_cables": {
+        "id": 15,
+        "name": "ArcGIS Submarine Cables",
+        "module": "L2",
+        "priority": "P1",
+        "frequency_minutes": 10080,
+    },
+    "arcgis_landing_points": {
+        "id": 16,
+        "name": "ArcGIS Landing Points",
+        "module": "L2",
+        "priority": "P1",
+        "frequency_minutes": 10080,
+    },
+    "arcgis_cable_landing_relation": {
+        "id": 17,
+        "name": "ArcGIS Cable-Landing Relations",
+        "module": "L2",
+        "priority": "P1",
+        "frequency_minutes": 10080,
+    },
+    "fao_landing_points": {
+        "id": 18,
+        "name": "FAO Landing Points",
+        "module": "L2",
+        "priority": "P1",
+        "frequency_minutes": 10080,
+    },
+    "spacetrack_tle": {
+        "id": 19,
+        "name": "Space-Track TLE",
+        "module": "L3",
+        "priority": "P2",
+        "frequency_minutes": 1440,
+    },
+    "celestrak_tle": {
+        "id": 20,
+        "name": "CelesTrak TLE",
+        "module": "L3",
+        "priority": "P2",
+        "frequency_minutes": 1440,
+    },
+    "ris_live_bgp": {
+        "id": 21,
+        "name": "RIPE RIS Live BGP",
+        "module": "L3",
+        "priority": "P1",
+        "frequency_minutes": 15,
+    },
+    "bgpstream_bgp": {
+        "id": 22,
+        "name": "CAIDA BGPStream Backfill",
+        "module": "L3",
+        "priority": "P1",
+        "frequency_minutes": 360,
+    },
+    "iptoasn_prefix_geo": {
+        "id": 23,
+        "name": "IPtoASN Prefix Geography",
+        "module": "L3",
+        "priority": "P1",
+        "frequency_minutes": 1440,
+    },
+    "opengeofeed_prefix_geo": {
+        "id": 24,
+        "name": "OpenGeoFeed Prefix Geography",
+        "module": "L3",
+        "priority": "P1",
+        "frequency_minutes": 1440,
+    },
+    "nro_delegated_prefix_geo": {
+        "id": 25,
+        "name": "NRO Delegated Prefix Geography",
+        "module": "L3",
+        "priority": "P1",
+        "frequency_minutes": 1440,
+    },
+    "news_live_streams": {
+        "id": 26,
+        "name": "News Live Streams",
+        "module": "L4",
+        "priority": "P2",
+        "frequency_minutes": 720,
+    },
+}
+
+ID_TO_COLLECTOR = {info["id"]: name for name, info in DEFAULT_DATASOURCES.items()}
+COLLECTOR_TO_ID = {name: info["id"] for name, info in DEFAULT_DATASOURCES.items()}

backend/app/core/satellite_tle.py

@@ -0,0 +1,116 @@
+"""Helpers for building stable TLE lines from orbital elements."""
+
+from __future__ import annotations
+
+from datetime import datetime
+from typing import Any, Optional
+
+
+def compute_tle_checksum(line: str) -> str:
+    """Compute the standard modulo-10 checksum for a TLE line."""
+    total = 0
+
+    for char in line[:68]:
+        if char.isdigit():
+            total += int(char)
+        elif char == "-":
+            total += 1
+
+    return str(total % 10)
+
+
+def _parse_epoch(value: Any) -> Optional[datetime]:
+    if not value:
+        return None
+    if isinstance(value, datetime):
+        return value
+    if isinstance(value, str):
+        return datetime.fromisoformat(value.replace("Z", "+00:00"))
+    return None
+
+
+def build_tle_line1(norad_cat_id: Any, epoch: Any) -> Optional[str]:
+    """Build a valid TLE line 1 from the NORAD id and epoch."""
+    epoch_date = _parse_epoch(epoch)
+    if not norad_cat_id or epoch_date is None:
+        return None
+
+    epoch_year = epoch_date.year % 100
+    start_of_year = datetime(epoch_date.year, 1, 1, tzinfo=epoch_date.tzinfo)
+    day_of_year = (epoch_date - start_of_year).days + 1
+    ms_of_day = (
+        epoch_date.hour * 3600000
+        + epoch_date.minute * 60000
+        + epoch_date.second * 1000
+        + int(epoch_date.microsecond / 1000)
+    )
+    day_fraction = ms_of_day / 86400000
+    decimal_fraction = f"{day_fraction:.8f}"[1:]
+    epoch_str = f"{epoch_year:02d}{day_of_year:03d}{decimal_fraction}"
+
+    core = (
+        f"1 {int(norad_cat_id):05d}U 00001A   {epoch_str}"
+        "  .00000000  00000-0  00000-0 0  999"
+    )
+    return core + compute_tle_checksum(core)
+
+
+def build_tle_line2(
+    norad_cat_id: Any,
+    inclination: Any,
+    raan: Any,
+    eccentricity: Any,
+    arg_of_perigee: Any,
+    mean_anomaly: Any,
+    mean_motion: Any,
+) -> Optional[str]:
+    """Build a valid TLE line 2 from the standard orbital elements."""
+    required = [
+        norad_cat_id,
+        inclination,
+        raan,
+        eccentricity,
+        arg_of_perigee,
+        mean_anomaly,
+        mean_motion,
+    ]
+    if any(value is None for value in required):
+        return None
+
+    eccentricity_digits = str(round(float(eccentricity) * 10_000_000)).zfill(7)
+    core = (
+        f"2 {int(norad_cat_id):05d}"
+        f" {float(inclination):8.4f}"
+        f" {float(raan):8.4f}"
+        f" {eccentricity_digits}"
+        f" {float(arg_of_perigee):8.4f}"
+        f" {float(mean_anomaly):8.4f}"
+        f" {float(mean_motion):11.8f}"
+        "00000"
+    )
+    return core + compute_tle_checksum(core)
+
+
+def build_tle_lines_from_elements(
+    *,
+    norad_cat_id: Any,
+    epoch: Any,
+    inclination: Any,
+    raan: Any,
+    eccentricity: Any,
+    arg_of_perigee: Any,
+    mean_anomaly: Any,
+    mean_motion: Any,
+) -> tuple[Optional[str], Optional[str]]:
+    """Build both TLE lines from a metadata payload."""
+    line1 = build_tle_line1(norad_cat_id, epoch)
+    line2 = build_tle_line2(
+        norad_cat_id,
+        inclination,
+        raan,
+        eccentricity,
+        arg_of_perigee,
+        mean_anomaly,
+        mean_motion,
+    )
+    return line1, line2

backend/app/core/security.py

@@ -1,4 +1,4 @@
-from datetime import datetime, timedelta
+from datetime import UTC, datetime, timedelta
 from typing import Optional
 
 import bcrypt
@@ -49,9 +49,9 @@ def get_password_hash(password: str) -> str:
 def create_access_token(data: dict, expires_delta: Optional[timedelta] = None) -> str:
     to_encode = data.copy()
     if expires_delta:
-        expire = datetime.utcnow() + expires_delta
+        expire = datetime.now(UTC) + expires_delta
     elif settings.ACCESS_TOKEN_EXPIRE_MINUTES > 0:
-        expire = datetime.utcnow() + timedelta(minutes=settings.ACCESS_TOKEN_EXPIRE_MINUTES)
+        expire = datetime.now(UTC) + timedelta(minutes=settings.ACCESS_TOKEN_EXPIRE_MINUTES)
     else:
         expire = None
     if expire:
@@ -65,7 +65,7 @@ def create_access_token(data: dict, expires_delta: Optional[timedelta] = None) -
 def create_refresh_token(data: dict) -> str:
     to_encode = data.copy()
     if settings.REFRESH_TOKEN_EXPIRE_DAYS > 0:
-        expire = datetime.utcnow() + timedelta(days=settings.REFRESH_TOKEN_EXPIRE_DAYS)
+        expire = datetime.now(UTC) + timedelta(days=settings.REFRESH_TOKEN_EXPIRE_DAYS)
         to_encode.update({"exp": expire})
     to_encode.update({"type": "refresh"})
     if "sub" in to_encode:

backend/app/core/time.py

@@ -0,0 +1,20 @@
+"""Time helpers for API serialization."""
+
+from __future__ import annotations
+
+from datetime import UTC, datetime
+
+
+def ensure_utc(value: datetime | None) -> datetime | None:
+    if value is None:
+        return None
+    if value.tzinfo is None:
+        return value.replace(tzinfo=UTC)
+    return value.astimezone(UTC)
+
+
+def to_iso8601_utc(value: datetime | None) -> str | None:
+    normalized = ensure_utc(value)
+    if normalized is None:
+        return None
+    return normalized.isoformat().replace("+00:00", "Z")

backend/app/core/websocket/broadcaster.py

@@ -1,9 +1,10 @@
 """Data broadcaster for WebSocket connections"""
 
 import asyncio
-from datetime import datetime
+from datetime import UTC, datetime
 from typing import Dict, Any, Optional
 
+from app.core.time import to_iso8601_utc
 from app.core.websocket.manager import manager
 
 
@@ -22,7 +23,7 @@ class DataBroadcaster:
             "active_datasources": 8,
             "tasks_today": 45,
             "success_rate": 97.8,
-            "last_updated": datetime.utcnow().isoformat(),
+            "last_updated": to_iso8601_utc(datetime.now(UTC)),
             "alerts": {"critical": 0, "warning": 2, "info": 5},
         }
 
@@ -35,7 +36,7 @@ class DataBroadcaster:
                     {
                         "type": "data_frame",
                         "channel": "dashboard",
-                        "timestamp": datetime.utcnow().isoformat(),
+                        "timestamp": to_iso8601_utc(datetime.now(UTC)),
                         "payload": {"stats": stats},
                     },
                     channel="dashboard",
@@ -49,7 +50,7 @@ class DataBroadcaster:
         await manager.broadcast(
             {
                 "type": "alert_notification",
-                "timestamp": datetime.utcnow().isoformat(),
+                "timestamp": to_iso8601_utc(datetime.now(UTC)),
                 "data": {"alert": alert},
             }
         )
@@ -60,7 +61,7 @@ class DataBroadcaster:
             {
                 "type": "data_frame",
                 "channel": "gpu_clusters",
-                "timestamp": datetime.utcnow().isoformat(),
+                "timestamp": to_iso8601_utc(datetime.now(UTC)),
                 "payload": data,
             }
         )
@@ -71,12 +72,24 @@ class DataBroadcaster:
             {
                 "type": "data_frame",
                 "channel": channel,
-                "timestamp": datetime.utcnow().isoformat(),
+                "timestamp": to_iso8601_utc(datetime.now(UTC)),
                 "payload": data,
             },
             channel=channel if channel in manager.active_connections else "all",
         )
 
+    async def broadcast_datasource_task_update(self, data: Dict[str, Any]):
+        """Broadcast datasource task progress updates to connected clients."""
+        await manager.broadcast(
+            {
+                "type": "data_frame",
+                "channel": "datasource_tasks",
+                "timestamp": to_iso8601_utc(datetime.now(UTC)),
+                "payload": data,
+            },
+            channel="all",
+        )
+
     def start(self):
         """Start all broadcasters"""
         if not self.running:

backend/app/db/session.py

@@ -1,5 +1,6 @@
 from typing import AsyncGenerator
 
+from sqlalchemy import text
 from sqlalchemy.ext.asyncio import AsyncSession, create_async_engine, async_sessionmaker
 from sqlalchemy.orm import declarative_base
 
@@ -25,11 +26,130 @@ async def get_db() -> AsyncGenerator[AsyncSession, None]:
             raise
 
 
+async def seed_default_datasources(session: AsyncSession):
+    from app.core.datasource_defaults import DEFAULT_DATASOURCES
+    from app.models.datasource import DataSource
+
+    for source, info in DEFAULT_DATASOURCES.items():
+        existing = await session.get(DataSource, info["id"])
+        if existing:
+            existing.name = info["name"]
+            existing.source = source
+            existing.module = info["module"]
+            existing.priority = info["priority"]
+            existing.frequency_minutes = info["frequency_minutes"]
+            existing.collector_class = source
+            if existing.config is None:
+                existing.config = "{}"
+            continue
+
+        session.add(
+            DataSource(
+                id=info["id"],
+                name=info["name"],
+                source=source,
+                module=info["module"],
+                priority=info["priority"],
+                frequency_minutes=info["frequency_minutes"],
+                collector_class=source,
+                config="{}",
+                is_active=True,
+            )
+        )
+
+    await session.commit()
+
+
+async def ensure_default_admin_user(session: AsyncSession):
+    from app.core.security import get_password_hash
+    from app.models.user import User
+
+    result = await session.execute(
+        text("SELECT id FROM users WHERE username = 'admin'")
+    )
+    if result.fetchone():
+        return
+
+    session.add(
+        User(
+            username="admin",
+            email="admin@planet.local",
+            password_hash=get_password_hash("admin123"),
+            role="super_admin",
+            is_active=True,
+        )
+    )
+    await session.commit()
+
+
 async def init_db():
     import app.models.user  # noqa: F401
     import app.models.gpu_cluster  # noqa: F401
     import app.models.task  # noqa: F401
+    import app.models.data_snapshot  # noqa: F401
     import app.models.datasource  # noqa: F401
+    import app.models.datasource_config  # noqa: F401
+    import app.models.alert  # noqa: F401
+    import app.models.bgp_anomaly  # noqa: F401
+    import app.models.bgp_incident  # noqa: F401
+    import app.models.bgp_observation  # noqa: F401
+    import app.models.collected_data  # noqa: F401
+    import app.models.system_setting  # noqa: F401
+    import app.models.playground_session  # noqa: F401
+    import app.models.playground_message  # noqa: F401
 
     async with engine.begin() as conn:
         await conn.run_sync(Base.metadata.create_all)
+        await conn.execute(
+            text(
+                """
+                ALTER TABLE collected_data
+                ADD COLUMN IF NOT EXISTS snapshot_id INTEGER,
+                ADD COLUMN IF NOT EXISTS task_id INTEGER,
+                ADD COLUMN IF NOT EXISTS entity_key VARCHAR(255),
+                ADD COLUMN IF NOT EXISTS is_current BOOLEAN DEFAULT TRUE,
+                ADD COLUMN IF NOT EXISTS previous_record_id INTEGER,
+                ADD COLUMN IF NOT EXISTS change_type VARCHAR(20),
+                ADD COLUMN IF NOT EXISTS change_summary JSONB DEFAULT '{}'::jsonb,
+                ADD COLUMN IF NOT EXISTS deleted_at TIMESTAMPTZ
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                ALTER TABLE collection_tasks
+                ADD COLUMN IF NOT EXISTS phase VARCHAR(30) DEFAULT 'queued'
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                CREATE INDEX IF NOT EXISTS idx_collected_data_source_source_id
+                ON collected_data (source, source_id)
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                UPDATE collected_data
+                SET entity_key = source || ':' || COALESCE(source_id, id::text)
+                WHERE entity_key IS NULL
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                UPDATE collected_data
+                SET is_current = TRUE
+                WHERE is_current IS NULL
+                """
+            )
+        )
+
+    async with async_session_factory() as session:
+        await seed_default_datasources(session)
+        await ensure_default_admin_user(session)

backend/app/main.py

@@ -2,15 +2,19 @@ from contextlib import asynccontextmanager
 
 from fastapi import FastAPI
 from fastapi.middleware.cors import CORSMiddleware
-from fastapi.staticfiles import StaticFiles
 from starlette.middleware.base import BaseHTTPMiddleware
 
-from app.core.config import settings
-from app.core.websocket.broadcaster import broadcaster
-from app.db.session import init_db, async_session_factory
 from app.api.main import api_router
 from app.api.v1 import websocket
-from app.services.scheduler import start_scheduler, stop_scheduler
+from app.core.config import settings
+from app.core.websocket.broadcaster import broadcaster
+from app.db.session import init_db
+from app.services.scheduler import (
+    cleanup_stale_running_tasks,
+    start_scheduler,
+    stop_scheduler,
+    sync_scheduler_with_datasources,
+)
 
 
 class WebSocketCORSMiddleware(BaseHTTPMiddleware):
@@ -27,7 +31,9 @@ class WebSocketCORSMiddleware(BaseHTTPMiddleware):
 @asynccontextmanager
 async def lifespan(app: FastAPI):
     await init_db()
+    await cleanup_stale_running_tasks()
     start_scheduler()
+    await sync_scheduler_with_datasources()
     broadcaster.start()
     yield
     broadcaster.stop()
@@ -60,16 +66,11 @@ app.include_router(websocket.router)
 
 @app.get("/health")
 async def health_check():
-    """健康检查端点"""
-    return {
-        "status": "healthy",
-        "version": settings.VERSION,
-    }
+    return {"status": "healthy", "version": settings.VERSION}
 
 
 @app.get("/")
 async def root():
-    """API根目录"""
     return {
         "name": settings.PROJECT_NAME,
         "version": settings.VERSION,
@@ -80,7 +81,6 @@ async def root():
 
 @app.get("/api/v1/scheduler/jobs")
 async def get_scheduler_jobs():
-    """获取调度任务列表"""
     from app.services.scheduler import get_scheduler_jobs
 
-    return {"jobs": get_scheduler_jobs()}
+    return {"jobs": get_scheduler_jobs()}

backend/app/models/__init__.py

@@ -1,15 +1,29 @@
 from app.models.user import User
 from app.models.gpu_cluster import GPUCluster
 from app.models.task import CollectionTask
+from app.models.data_snapshot import DataSnapshot
 from app.models.datasource import DataSource
+from app.models.datasource_config import DataSourceConfig
 from app.models.alert import Alert, AlertSeverity, AlertStatus
+from app.models.bgp_anomaly import BGPAnomaly
+from app.models.bgp_incident import BGPIncident
+from app.models.bgp_observation import BGPObservation
+from app.models.system_setting import SystemSetting
+from app.models.playground_session import PlaygroundSession
+from app.models.playground_message import PlaygroundMessage
 
 __all__ = [
     "User",
     "GPUCluster",
     "CollectionTask",
+    "DataSnapshot",
     "DataSource",
+    "DataSourceConfig",
+    "SystemSetting",
     "Alert",
     "AlertSeverity",
     "AlertStatus",
+    "BGPAnomaly",
+    "BGPIncident",
+    "BGPObservation",
 ]

backend/app/models/alert.py

@@ -5,6 +5,7 @@ from typing import Optional
 from sqlalchemy import Column, Integer, String, DateTime, Text, ForeignKey, Enum as SQLEnum
 from sqlalchemy.orm import relationship
 
+from app.core.time import to_iso8601_utc
 from app.db.session import Base
 
 
@@ -50,8 +51,8 @@ class Alert(Base):
             "acknowledged_by": self.acknowledged_by,
             "resolved_by": self.resolved_by,
             "resolution_notes": self.resolution_notes,
-            "created_at": self.created_at.isoformat() if self.created_at else None,
-            "updated_at": self.updated_at.isoformat() if self.updated_at else None,
-            "acknowledged_at": self.acknowledged_at.isoformat() if self.acknowledged_at else None,
-            "resolved_at": self.resolved_at.isoformat() if self.resolved_at else None,
+            "created_at": to_iso8601_utc(self.created_at),
+            "updated_at": to_iso8601_utc(self.updated_at),
+            "acknowledged_at": to_iso8601_utc(self.acknowledged_at),
+            "resolved_at": to_iso8601_utc(self.resolved_at),
         }

backend/app/models/bgp_anomaly.py

@@ -0,0 +1,58 @@
+"""BGP anomaly model for derived routing intelligence."""
+
+from datetime import datetime
+
+from sqlalchemy import Column, DateTime, Float, ForeignKey, Index, Integer, JSON, String, Text
+
+from app.core.time import to_iso8601_utc
+from app.db.session import Base
+
+
+class BGPAnomaly(Base):
+    __tablename__ = "bgp_anomalies"
+
+    id = Column(Integer, primary_key=True, index=True)
+    snapshot_id = Column(Integer, ForeignKey("data_snapshots.id"), nullable=True, index=True)
+    task_id = Column(Integer, ForeignKey("collection_tasks.id"), nullable=True, index=True)
+    source = Column(String(100), nullable=False, index=True)
+    anomaly_type = Column(String(50), nullable=False, index=True)
+    severity = Column(String(20), nullable=False, index=True)
+    status = Column(String(20), nullable=False, default="active", index=True)
+    entity_key = Column(String(255), nullable=False, index=True)
+    prefix = Column(String(64), nullable=True, index=True)
+    origin_asn = Column(Integer, nullable=True, index=True)
+    new_origin_asn = Column(Integer, nullable=True, index=True)
+    peer_scope = Column(JSON, default=list)
+    started_at = Column(DateTime(timezone=True), nullable=False, default=datetime.utcnow, index=True)
+    ended_at = Column(DateTime(timezone=True), nullable=True)
+    confidence = Column(Float, nullable=False, default=0.5)
+    summary = Column(Text, nullable=False)
+    evidence = Column(JSON, default=dict)
+    created_at = Column(DateTime(timezone=True), nullable=False, default=datetime.utcnow, index=True)
+
+    __table_args__ = (
+        Index("idx_bgp_anomalies_source_created", "source", "created_at"),
+        Index("idx_bgp_anomalies_type_status", "anomaly_type", "status"),
+    )
+
+    def to_dict(self) -> dict:
+        return {
+            "id": self.id,
+            "snapshot_id": self.snapshot_id,
+            "task_id": self.task_id,
+            "source": self.source,
+            "anomaly_type": self.anomaly_type,
+            "severity": self.severity,
+            "status": self.status,
+            "entity_key": self.entity_key,
+            "prefix": self.prefix,
+            "origin_asn": self.origin_asn,
+            "new_origin_asn": self.new_origin_asn,
+            "peer_scope": self.peer_scope or [],
+            "started_at": to_iso8601_utc(self.started_at),
+            "ended_at": to_iso8601_utc(self.ended_at),
+            "confidence": self.confidence,
+            "summary": self.summary,
+            "evidence": self.evidence or {},
+            "created_at": to_iso8601_utc(self.created_at),
+        }

backend/app/models/bgp_incident.py

@@ -0,0 +1,64 @@
+"""BGP incident model for aggregated routing events."""
+
+from datetime import datetime
+
+from sqlalchemy import Column, DateTime, Float, ForeignKey, Index, Integer, JSON, String, Text
+
+from app.core.time import to_iso8601_utc
+from app.db.session import Base
+
+
+class BGPIncident(Base):
+    __tablename__ = "bgp_incidents"
+
+    id = Column(Integer, primary_key=True, index=True)
+    snapshot_id = Column(Integer, ForeignKey("data_snapshots.id"), nullable=True, index=True)
+    task_id = Column(Integer, ForeignKey("collection_tasks.id"), nullable=True, index=True)
+    source = Column(String(100), nullable=False, index=True)
+    incident_key = Column(String(255), nullable=False, index=True)
+    incident_type = Column(String(50), nullable=False, index=True)
+    title = Column(String(255), nullable=False)
+    summary = Column(Text, nullable=False)
+    severity = Column(String(20), nullable=False, index=True)
+    status = Column(String(20), nullable=False, default="active", index=True)
+    confidence = Column(Float, nullable=False, default=0.5)
+    started_at = Column(DateTime(timezone=True), nullable=False, default=datetime.utcnow, index=True)
+    ended_at = Column(DateTime(timezone=True), nullable=True)
+    affected_prefixes = Column(JSON, default=list)
+    affected_asns = Column(JSON, default=list)
+    affected_collectors = Column(JSON, default=list)
+    affected_regions = Column(JSON, default=list)
+    related_cables = Column(JSON, default=list)
+    related_ixps = Column(JSON, default=list)
+    evidence_refs = Column(JSON, default=list)
+    created_at = Column(DateTime(timezone=True), nullable=False, default=datetime.utcnow, index=True)
+
+    __table_args__ = (
+        Index("idx_bgp_incidents_source_created", "source", "created_at"),
+        Index("idx_bgp_incidents_type_status", "incident_type", "status"),
+    )
+
+    def to_dict(self) -> dict:
+        return {
+            "id": self.id,
+            "snapshot_id": self.snapshot_id,
+            "task_id": self.task_id,
+            "source": self.source,
+            "incident_key": self.incident_key,
+            "incident_type": self.incident_type,
+            "title": self.title,
+            "summary": self.summary,
+            "severity": self.severity,
+            "status": self.status,
+            "confidence": self.confidence,
+            "started_at": to_iso8601_utc(self.started_at),
+            "ended_at": to_iso8601_utc(self.ended_at),
+            "affected_prefixes": self.affected_prefixes or [],
+            "affected_asns": self.affected_asns or [],
+            "affected_collectors": self.affected_collectors or [],
+            "affected_regions": self.affected_regions or [],
+            "related_cables": self.related_cables or [],
+            "related_ixps": self.related_ixps or [],
+            "evidence_refs": self.evidence_refs or [],
+            "created_at": to_iso8601_utc(self.created_at),
+        }

backend/app/models/bgp_observation.py

@@ -0,0 +1,62 @@
+"""BGP raw observation model for routing event ingestion."""
+
+from sqlalchemy import Column, DateTime, ForeignKey, Index, Integer, JSON, String, Text
+from sqlalchemy.sql import func
+
+from app.core.time import to_iso8601_utc
+from app.db.session import Base
+
+
+class BGPObservation(Base):
+    __tablename__ = "bgp_observations"
+
+    id = Column(Integer, primary_key=True, index=True)
+    snapshot_id = Column(Integer, ForeignKey("data_snapshots.id"), nullable=True, index=True)
+    task_id = Column(Integer, ForeignKey("collection_tasks.id"), nullable=True, index=True)
+    source = Column(String(100), nullable=False, index=True)
+    ingest_batch_id = Column(String(100), nullable=True, index=True)
+    source_event_id = Column(String(100), nullable=True, index=True)
+    collector = Column(String(100), nullable=True, index=True)
+    peer_asn = Column(Integer, nullable=True, index=True)
+    peer_ip = Column(String(100), nullable=True)
+    prefix = Column(String(64), nullable=True, index=True)
+    event_type = Column(String(32), nullable=False, index=True)
+    as_path = Column(JSON, default=list)
+    origin_asn = Column(Integer, nullable=True, index=True)
+    next_hop = Column(String(100), nullable=True)
+    communities = Column(JSON, default=list)
+    observed_at = Column(DateTime(timezone=True), nullable=False, index=True)
+    collector_geo = Column(JSON, default=dict)
+    raw_payload = Column(JSON, default=dict)
+    created_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now(), index=True)
+    note = Column(Text, nullable=True)
+
+    __table_args__ = (
+        Index("idx_bgp_obs_source_observed", "source", "observed_at"),
+        Index("idx_bgp_obs_collector_prefix", "collector", "prefix"),
+        Index("idx_bgp_obs_task_source_event", "task_id", "source_event_id"),
+    )
+
+    def to_dict(self) -> dict:
+        return {
+            "id": self.id,
+            "snapshot_id": self.snapshot_id,
+            "task_id": self.task_id,
+            "source": self.source,
+            "ingest_batch_id": self.ingest_batch_id,
+            "source_event_id": self.source_event_id,
+            "collector": self.collector,
+            "peer_asn": self.peer_asn,
+            "peer_ip": self.peer_ip,
+            "prefix": self.prefix,
+            "event_type": self.event_type,
+            "as_path": self.as_path or [],
+            "origin_asn": self.origin_asn,
+            "next_hop": self.next_hop,
+            "communities": self.communities or [],
+            "observed_at": to_iso8601_utc(self.observed_at),
+            "collector_geo": self.collector_geo or {},
+            "raw_payload": self.raw_payload or {},
+            "created_at": to_iso8601_utc(self.created_at),
+            "note": self.note,
+        }

backend/app/models/collected_data.py

@@ -1,8 +1,10 @@
 """Collected Data model for storing data from all collectors"""
 
-from sqlalchemy import Column, DateTime, Integer, String, Text, JSON, Index
+from sqlalchemy import Boolean, Column, DateTime, ForeignKey, Integer, String, Text, JSON, Index
 from sqlalchemy.sql import func
 
+from app.core.collected_data_fields import get_record_field
+from app.core.time import to_iso8601_utc
 from app.db.session import Base
 
 
@@ -12,8 +14,11 @@ class CollectedData(Base):
     __tablename__ = "collected_data"
 
     id = Column(Integer, primary_key=True, autoincrement=True)
+    snapshot_id = Column(Integer, ForeignKey("data_snapshots.id"), nullable=True, index=True)
+    task_id = Column(Integer, ForeignKey("collection_tasks.id"), nullable=True, index=True)
     source = Column(String(100), nullable=False, index=True)  # e.g., "top500", "huggingface_models"
     source_id = Column(String(100), index=True)  # Original ID from source, e.g., "rank_1"
+    entity_key = Column(String(255), index=True)
     data_type = Column(
         String(50), nullable=False, index=True
     )  # e.g., "supercomputer", "model", "dataset"
@@ -23,16 +28,6 @@ class CollectedData(Base):
     title = Column(String(500))
     description = Column(Text)
 
-    # Location data (for geo visualization)
-    country = Column(String(100))
-    city = Column(String(100))
-    latitude = Column(String(50))
-    longitude = Column(String(50))
-
-    # Performance metrics
-    value = Column(String(100))  # Generic value field (Rmax, Rpeak, etc.)
-    unit = Column(String(20))
-
     # Additional metadata as JSON
     extra_data = Column(
         "metadata", JSON, default={}
@@ -44,11 +39,17 @@ class CollectedData(Base):
 
     # Status
     is_valid = Column(Integer, default=1)  # 1=valid, 0=invalid
+    is_current = Column(Boolean, default=True, index=True)
+    previous_record_id = Column(Integer, ForeignKey("collected_data.id"), nullable=True, index=True)
+    change_type = Column(String(20), nullable=True)
+    change_summary = Column(JSON, default={})
+    deleted_at = Column(DateTime(timezone=True), nullable=True)
 
     # Indexes for common queries
     __table_args__ = (
         Index("idx_collected_data_source_collected", "source", "collected_at"),
         Index("idx_collected_data_source_type", "source", "data_type"),
+        Index("idx_collected_data_source_source_id", "source", "source_id"),
     )
 
     def __repr__(self):
@@ -58,23 +59,27 @@ class CollectedData(Base):
         """Convert to dictionary"""
         return {
             "id": self.id,
+            "snapshot_id": self.snapshot_id,
+            "task_id": self.task_id,
             "source": self.source,
             "source_id": self.source_id,
+            "entity_key": self.entity_key,
             "data_type": self.data_type,
             "name": self.name,
             "title": self.title,
             "description": self.description,
-            "country": self.country,
-            "city": self.city,
-            "latitude": self.latitude,
-            "longitude": self.longitude,
-            "value": self.value,
-            "unit": self.unit,
+            "country": get_record_field(self, "country"),
+            "city": get_record_field(self, "city"),
+            "latitude": get_record_field(self, "latitude"),
+            "longitude": get_record_field(self, "longitude"),
+            "value": get_record_field(self, "value"),
+            "unit": get_record_field(self, "unit"),
             "metadata": self.extra_data,
-            "collected_at": self.collected_at.isoformat()
-            if self.collected_at is not None
-            else None,
-            "reference_date": self.reference_date.isoformat()
-            if self.reference_date is not None
-            else None,
+            "collected_at": to_iso8601_utc(self.collected_at),
+            "reference_date": to_iso8601_utc(self.reference_date),
+            "is_current": self.is_current,
+            "previous_record_id": self.previous_record_id,
+            "change_type": self.change_type,
+            "change_summary": self.change_summary,
+            "deleted_at": to_iso8601_utc(self.deleted_at),
         }

backend/app/models/data_snapshot.py

@@ -0,0 +1,26 @@
+from sqlalchemy import Boolean, Column, DateTime, ForeignKey, Integer, JSON, String
+from sqlalchemy.sql import func
+
+from app.db.session import Base
+
+
+class DataSnapshot(Base):
+    __tablename__ = "data_snapshots"
+
+    id = Column(Integer, primary_key=True, autoincrement=True)
+    datasource_id = Column(Integer, nullable=False, index=True)
+    task_id = Column(Integer, ForeignKey("collection_tasks.id"), nullable=True, index=True)
+    source = Column(String(100), nullable=False, index=True)
+    snapshot_key = Column(String(100), nullable=True, index=True)
+    reference_date = Column(DateTime(timezone=True), nullable=True)
+    started_at = Column(DateTime(timezone=True), server_default=func.now())
+    completed_at = Column(DateTime(timezone=True), nullable=True)
+    record_count = Column(Integer, default=0)
+    status = Column(String(20), nullable=False, default="running")
+    is_current = Column(Boolean, default=True, index=True)
+    parent_snapshot_id = Column(Integer, ForeignKey("data_snapshots.id"), nullable=True, index=True)
+    summary = Column(JSON, default={})
+    created_at = Column(DateTime(timezone=True), server_default=func.now())
+
+    def __repr__(self):
+        return f"<DataSnapshot {self.id}: {self.source}/{self.status}>"

backend/app/models/playground_message.py

@@ -0,0 +1,40 @@
+from sqlalchemy import JSON, Boolean, Column, DateTime, ForeignKey, Integer, String, Text
+from sqlalchemy.sql import func
+
+from app.db.session import Base
+
+
+class PlaygroundMessage(Base):
+    __tablename__ = "playground_messages"
+
+    id = Column(Integer, primary_key=True, autoincrement=True)
+    public_id = Column(String(64), unique=True, index=True, nullable=False)
+    session_id = Column(Integer, ForeignKey("playground_sessions.id", ondelete="CASCADE"), nullable=False, index=True)
+    user_id = Column(Integer, ForeignKey("users.id", ondelete="CASCADE"), nullable=False, index=True)
+    parent_message_id = Column(Integer, ForeignKey("playground_messages.id", ondelete="SET NULL"), nullable=True)
+    role = Column(String(20), nullable=False)
+    kind = Column(String(20), nullable=False, default="message")
+    status = Column(String(20), nullable=False, default="done")
+    title = Column(String(255), nullable=True)
+    content = Column(Text, nullable=False, default="")
+    thinking_content = Column(Text, nullable=False, default="")
+    meta = Column(JSON, nullable=False, default=list)
+    provider = Column(String(100), nullable=True)
+    model = Column(String(200), nullable=True)
+    request_id = Column(String(100), nullable=True)
+    raw_response = Column(JSON, nullable=False, default=dict)
+    content_blocks = Column(JSON, nullable=False, default=list)
+    text_blocks = Column(JSON, nullable=False, default=list)
+    thinking_blocks = Column(JSON, nullable=False, default=list)
+    sort_order = Column(Integer, nullable=False, default=0, index=True)
+    is_visible = Column(Boolean, nullable=False, default=True)
+    created_at = Column(DateTime(timezone=True), server_default=func.now(), nullable=False)
+    updated_at = Column(
+        DateTime(timezone=True),
+        server_default=func.now(),
+        onupdate=func.now(),
+        nullable=False,
+    )
+
+    def __repr__(self):
+        return f"<PlaygroundMessage public_id={self.public_id} role={self.role} status={self.status}>"

backend/app/models/playground_session.py

@@ -0,0 +1,27 @@
+from sqlalchemy import JSON, Column, DateTime, ForeignKey, Integer, String, UniqueConstraint
+from sqlalchemy.sql import func
+
+from app.db.session import Base
+
+
+class PlaygroundSession(Base):
+    __tablename__ = "playground_sessions"
+    __table_args__ = (
+        UniqueConstraint("user_id", "session_key", name="uq_playground_sessions_user_session_key"),
+    )
+
+    id = Column(Integer, primary_key=True, autoincrement=True)
+    user_id = Column(Integer, ForeignKey("users.id", ondelete="CASCADE"), nullable=False, index=True)
+    session_key = Column(String(100), nullable=False, default="default")
+    title = Column(String(200), nullable=False, default="Playground 会话")
+    state = Column(JSON, nullable=False, default={})
+    created_at = Column(DateTime(timezone=True), server_default=func.now(), nullable=False)
+    updated_at = Column(
+        DateTime(timezone=True),
+        server_default=func.now(),
+        onupdate=func.now(),
+        nullable=False,
+    )
+
+    def __repr__(self):
+        return f"<PlaygroundSession user_id={self.user_id} session_key={self.session_key}>"

backend/app/models/system_setting.py

@@ -0,0 +1,19 @@
+"""Persistent system settings model."""
+
+from sqlalchemy import JSON, Column, DateTime, Integer, String, UniqueConstraint
+from sqlalchemy.sql import func
+
+from app.db.session import Base
+
+
+class SystemSetting(Base):
+    __tablename__ = "system_settings"
+    __table_args__ = (UniqueConstraint("category", name="uq_system_settings_category"),)
+
+    id = Column(Integer, primary_key=True, autoincrement=True)
+    category = Column(String(50), nullable=False)
+    payload = Column(JSON, nullable=False, default={})
+    updated_at = Column(DateTime(timezone=True), server_default=func.now(), onupdate=func.now())
+
+    def __repr__(self):
+        return f"<SystemSetting {self.category}>"

backend/app/models/task.py

@@ -12,6 +12,7 @@ class CollectionTask(Base):
     id = Column(Integer, primary_key=True, autoincrement=True)
     datasource_id = Column(Integer, nullable=False, index=True)
     status = Column(String(20), nullable=False)  # pending, running, success, failed, cancelled
+    phase = Column(String(30), default="queued")
     started_at = Column(DateTime(timezone=True))
     completed_at = Column(DateTime(timezone=True))
     records_processed = Column(Integer, default=0)

backend/app/schemas/ai.py

@@ -0,0 +1,175 @@
+from typing import Any
+
+from pydantic import BaseModel, Field
+
+
+class AIContentBlock(BaseModel):
+    type: str
+    text: str | None = None
+    thinking: str | None = None
+    signature: str | None = None
+    metadata: dict[str, Any] = Field(default_factory=dict)
+
+
+class SituationalAnalysisRequest(BaseModel):
+    title: str = Field(..., min_length=1, max_length=200)
+    objective: str = Field(..., min_length=1, max_length=1000)
+    context: dict[str, Any] = Field(default_factory=dict)
+    observations: list[str] = Field(default_factory=list)
+    constraints: list[str] = Field(default_factory=list)
+    preferred_model: str | None = Field(default=None, max_length=200)
+    thinking: dict[str, Any] | None = None
+
+
+class BGPBriefRequest(BaseModel):
+    incident_limit: int = Field(default=5, ge=1, le=10)
+    anomaly_limit: int = Field(default=6, ge=1, le=12)
+    collector_limit: int = Field(default=5, ge=1, le=10)
+    preferred_model: str | None = Field(default=None, max_length=200)
+    thinking: dict[str, Any] | None = None
+
+
+class AlertBriefRequest(BaseModel):
+    alert_limit: int = Field(default=8, ge=1, le=20)
+    preferred_model: str | None = Field(default=None, max_length=200)
+    thinking: dict[str, Any] | None = None
+
+
+class SituationalAlertBriefRequest(BaseModel):
+    preferred_model: str | None = Field(default=None, max_length=200)
+    thinking: dict[str, Any] | None = None
+
+
+class SituationalAnalysisResponse(BaseModel):
+    provider: str
+    model: str
+    content: str
+    content_blocks: list[AIContentBlock] = Field(default_factory=list)
+    text_blocks: list[str] = Field(default_factory=list)
+    thinking_blocks: list[str] = Field(default_factory=list)
+    raw_response: dict[str, Any] = Field(default_factory=dict)
+
+
+class BGPBriefRecordSummary(BaseModel):
+    id: str
+    title: str
+    provider: str
+    model: str
+    request_id: str | None = None
+    generated_at: str
+
+
+class BGPBriefRecordResponse(BGPBriefRecordSummary):
+    content_markdown: str
+    facts: list[str] = Field(default_factory=list)
+    context: dict[str, Any] = Field(default_factory=dict)
+
+
+class AlertBriefResponse(SituationalAnalysisResponse):
+    title: str
+    objective: str
+    facts: list[str] = Field(default_factory=list)
+    context: dict[str, Any] = Field(default_factory=dict)
+
+
+class SituationalAlertBriefResponse(SituationalAnalysisResponse):
+    title: str
+    objective: str
+    facts: list[str] = Field(default_factory=list)
+    context: dict[str, Any] = Field(default_factory=dict)
+
+
+class AIProviderStatusResponse(BaseModel):
+    provider: str
+    api: str | None = None
+    enabled: bool
+    configured: bool
+    model: str | None = None
+    base_url: str | None = None
+
+
+class PlaygroundSessionState(BaseModel):
+    messages: list[dict[str, Any]] = Field(default_factory=list)
+    selectedPresetKey: str = Field(default="bgp-brief", max_length=100)
+    title: str = Field(default="", max_length=200)
+    objective: str = Field(default="", max_length=1000)
+    constraints: str = Field(default="")
+    inputValue: str = Field(default="")
+    analysis: dict[str, Any] | None = None
+    latestAnalysisMessageId: str | None = Field(default=None, max_length=200)
+    analysisMeta: dict[str, Any] = Field(default_factory=dict)
+    helpExpanded: bool = True
+
+
+class PlaygroundSessionUpsertRequest(BaseModel):
+    session_key: str = Field(default="default", min_length=1, max_length=100)
+    title: str | None = Field(default=None, max_length=200)
+    state: PlaygroundSessionState
+
+
+class PlaygroundMessageRecord(BaseModel):
+    id: str
+    role: str
+    kind: str = "message"
+    status: str = "done"
+    title: str | None = None
+    content: str = ""
+    thinking_content: str = ""
+    meta: list[str] = Field(default_factory=list)
+    markdown: bool = True
+    provider: str | None = None
+    model: str | None = None
+    request_id: str | None = None
+    raw_response: dict[str, Any] = Field(default_factory=dict)
+    content_blocks: list[dict[str, Any]] = Field(default_factory=list)
+    text_blocks: list[str] = Field(default_factory=list)
+    thinking_blocks: list[str] = Field(default_factory=list)
+    parent_message_id: str | None = None
+    created_at: str
+    updated_at: str
+
+
+class PlaygroundSessionResponse(BaseModel):
+    id: str
+    session_key: str
+    title: str
+    state: PlaygroundSessionState
+    created_at: str
+    updated_at: str
+
+
+class PlaygroundThreadResponse(BaseModel):
+    session: PlaygroundSessionResponse
+    messages: list[PlaygroundMessageRecord] = Field(default_factory=list)
+
+
+class PlaygroundMessageCreateRequest(BaseModel):
+    session_key: str = Field(default="default", min_length=1, max_length=100)
+    title: str = Field(..., min_length=1, max_length=200)
+    objective: str = Field(..., min_length=1, max_length=1000)
+    constraints: str = Field(default="")
+    input: str = Field(..., min_length=1)
+    selected_preset_key: str = Field(default="bgp-brief", max_length=100)
+    help_expanded: bool = True
+
+
+class PlaygroundMessageActionResponse(BaseModel):
+    session: PlaygroundSessionResponse
+    messages: list[PlaygroundMessageRecord] = Field(default_factory=list)
+    active_message_id: str | None = None
+
+
+class PlaygroundMessageStopRequest(BaseModel):
+    session_key: str = Field(default="default", min_length=1, max_length=100)
+    message_id: str = Field(..., min_length=1, max_length=64)
+
+
+class PlaygroundMessageResendRequest(BaseModel):
+    session_key: str = Field(default="default", min_length=1, max_length=100)
+    user_message_id: str = Field(..., min_length=1, max_length=64)
+
+
+class PlaygroundMessageEditRequest(BaseModel):
+    session_key: str = Field(default="default", min_length=1, max_length=100)
+    user_message_id: str = Field(..., min_length=1, max_length=64)
+    content: str = Field(..., min_length=1)

backend/app/schemas/alert.py

@@ -0,0 +1,5 @@
+from pydantic import BaseModel, Field
+
+
+class AlertResolutionRequest(BaseModel):
+    resolution: str = Field(..., min_length=1, max_length=1000)

backend/app/services/ai_client.py

@@ -0,0 +1,109 @@
+from __future__ import annotations
+
+import asyncio
+
+import httpx
+from fastapi import HTTPException, status
+
+from app.core.config import settings
+from app.schemas.ai import (
+    AIProviderStatusResponse,
+    SituationalAnalysisRequest,
+    SituationalAnalysisResponse,
+)
+
+
+class AIProviderClient:
+    def __init__(self) -> None:
+        self.service_url = settings.AI_PROVIDER_SERVICE_URL.rstrip("/")
+        self.service_token = settings.AI_PROVIDER_SERVICE_TOKEN
+        self.timeout = settings.AI_PROVIDER_TIMEOUT_SECONDS
+        self.retry_attempts = max(settings.AI_PROVIDER_RETRY_ATTEMPTS, 1)
+
+    def _headers(self, request_id: str | None = None) -> dict[str, str]:
+        headers = {"Content-Type": "application/json"}
+        if self.service_token:
+            headers["X-Provider-Token"] = self.service_token
+        if request_id:
+            headers["X-Request-ID"] = request_id
+        return headers
+
+    async def get_status(self, request_id: str | None = None) -> AIProviderStatusResponse:
+        if not self.service_url:
+            return AIProviderStatusResponse(
+                provider="unconfigured",
+                enabled=False,
+                configured=False,
+                model=None,
+                base_url=None,
+            )
+
+        data = await self._request("GET", "/v1/provider/status", request_id=request_id)
+        return AIProviderStatusResponse.model_validate(data)
+
+    async def analyze(
+        self,
+        payload: SituationalAnalysisRequest,
+        request_id: str | None = None,
+    ) -> SituationalAnalysisResponse:
+        if not self.service_url:
+            raise HTTPException(
+                status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
+                detail="AI provider service URL is not configured.",
+            )
+
+        data = await self._request(
+            "POST",
+            "/v1/analyze",
+            json=payload.model_dump(),
+            request_id=request_id,
+        )
+        return SituationalAnalysisResponse.model_validate(data)
+
+    async def _request(
+        self,
+        method: str,
+        path: str,
+        json: dict | None = None,
+        request_id: str | None = None,
+    ) -> dict:
+        last_error: Exception | None = None
+        for attempt in range(1, self.retry_attempts + 1):
+            try:
+                async with httpx.AsyncClient(timeout=self.timeout) as client:
+                    response = await client.request(
+                        method,
+                        f"{self.service_url}{path}",
+                        headers=self._headers(request_id),
+                        json=json,
+                    )
+                    response.raise_for_status()
+                    return response.json()
+            except httpx.HTTPStatusError as exc:
+                last_error = exc
+                if attempt < self.retry_attempts and exc.response.status_code >= 500:
+                    await asyncio.sleep(0.3 * attempt)
+                    continue
+                detail = exc.response.text or "AI provider service returned an error"
+                raise HTTPException(
+                    status_code=status.HTTP_502_BAD_GATEWAY,
+                    detail=f"AI provider service request failed: {detail}",
+                ) from exc
+            except httpx.HTTPError as exc:
+                last_error = exc
+                if attempt < self.retry_attempts:
+                    await asyncio.sleep(0.3 * attempt)
+                    continue
+                raise HTTPException(
+                    status_code=status.HTTP_502_BAD_GATEWAY,
+                    detail=f"Failed to reach AI provider service: {exc}",
+                ) from exc
+
+        raise HTTPException(
+            status_code=status.HTTP_502_BAD_GATEWAY,
+            detail=f"AI provider service request failed: {last_error}",
+        )
+
+
+def get_ai_provider_client() -> AIProviderClient:
+    return AIProviderClient()

backend/app/services/alert_ai_brief.py

@@ -0,0 +1,103 @@
+from __future__ import annotations
+
+from collections import Counter
+from typing import Any
+
+from sqlalchemy import func, select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.models.alert import Alert, AlertSeverity, AlertStatus
+from app.schemas.ai import AlertBriefRequest, SituationalAnalysisRequest
+
+
+def _format_counter(counter: Counter[str], empty_text: str = "无") -> str:
+    if not counter:
+        return empty_text
+    return "，".join(f"{key} {value}" for key, value in counter.items())
+
+
+async def build_alert_brief_request(
+    db: AsyncSession,
+    *,
+    alert_limit: int = 8,
+) -> tuple[SituationalAnalysisRequest, list[str], dict[str, Any]]:
+    recent_alerts_result = await db.execute(
+        select(Alert)
+        .order_by(Alert.created_at.desc(), Alert.id.desc())
+        .limit(max(alert_limit, 1))
+    )
+    total_result = await db.execute(select(func.count(Alert.id)))
+    active_result = await db.execute(select(func.count(Alert.id)).where(Alert.status == AlertStatus.ACTIVE))
+    acknowledged_result = await db.execute(
+        select(func.count(Alert.id)).where(Alert.status == AlertStatus.ACKNOWLEDGED)
+    )
+    resolved_result = await db.execute(select(func.count(Alert.id)).where(Alert.status == AlertStatus.RESOLVED))
+
+    recent_alerts = recent_alerts_result.scalars().all()
+    total_alerts = total_result.scalar() or 0
+    active_alerts = active_result.scalar() or 0
+    acknowledged_alerts = acknowledged_result.scalar() or 0
+    resolved_alerts = resolved_result.scalar() or 0
+
+    severity_counts = Counter((item.severity.value if item.severity else "unknown") for item in recent_alerts)
+    status_counts = Counter((item.status.value if item.status else "unknown") for item in recent_alerts)
+    datasource_counts = Counter((item.datasource_name or "未命名数据源") for item in recent_alerts)
+    active_datasource_counts = Counter(
+        (item.datasource_name or "未命名数据源")
+        for item in recent_alerts
+        if item.status == AlertStatus.ACTIVE
+    )
+
+    facts = [
+        f"告警总量 {total_alerts} 条，其中 active {active_alerts} 条、acknowledged {acknowledged_alerts} 条、resolved {resolved_alerts} 条。",
+        f"最近告警严重度分布：{_format_counter(severity_counts)}。",
+        f"最近告警状态分布：{_format_counter(status_counts)}。",
+        f"最近告警数据源分布：{_format_counter(Counter(dict(datasource_counts.most_common(6))))}。",
+    ]
+
+    if active_datasource_counts:
+        facts.append(
+            "当前待处理告警主要集中在："
+            + _format_counter(Counter(dict(active_datasource_counts.most_common(5))))
+            + "。"
+        )
+
+    if recent_alerts:
+        facts.append(
+            "最近告警摘录："
+            + "；".join(
+                [
+                    f"{item.datasource_name or '未命名数据源'} / {item.severity.value if item.severity else '-'} / {item.status.value if item.status else '-'} / {item.message or '-'}"
+                    for item in recent_alerts[:6]
+                ]
+            )
+        )
+
+    context = {
+        "source": "alerts",
+        "total_alerts": total_alerts,
+        "active_alerts": active_alerts,
+        "acknowledged_alerts": acknowledged_alerts,
+        "resolved_alerts": resolved_alerts,
+        "severity_distribution": dict(severity_counts),
+        "status_distribution": dict(status_counts),
+        "top_datasources": dict(datasource_counts.most_common(6)),
+        "top_active_datasources": dict(active_datasource_counts.most_common(5)),
+    }
+
+    return (
+        SituationalAnalysisRequest(
+            title="告警态势 AI 简报",
+            objective="基于当前告警总量、严重度、状态、数据源分布与最近告警摘录，生成一份面向值班人员的简明告警态势简报，突出待处理风险、告警集中点和优先动作。",
+            observations=facts,
+            constraints=[
+                "明确区分事实、推断与建议。",
+                "优先指出仍处于 active 状态且高严重度的告警簇。",
+                "不要把 acknowledged 或 resolved 告警误判成当前仍在扩大。",
+                "如果证据不足，请明确指出缺失的上下文。",
+            ],
+            context=context,
+        ),
+        facts,
+        context,
+    )

backend/app/services/bgp_ai_brief.py

@@ -0,0 +1,259 @@
+from __future__ import annotations
+
+from collections import Counter
+from typing import Any
+
+from sqlalchemy import func, select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.api.v1.bgp import BGP_SOURCES
+from app.models.bgp_anomaly import BGPAnomaly
+from app.models.bgp_incident import BGPIncident
+from app.models.bgp_observation import BGPObservation
+from app.schemas.ai import SituationalAnalysisRequest
+from app.services.bgp_collectors import build_bgp_collector_coverage
+from app.services.bgp_enrichment import lookup_prefix_geography
+
+
+def _format_counter(counter: dict[str, int], empty_text: str = "无") -> str:
+    if not counter:
+        return empty_text
+    return "，".join(f"{key} {value}" for key, value in counter.items())
+
+
+def _severity_rank(value: str | None) -> int:
+    order = {
+        "critical": 0,
+        "high": 1,
+        "medium": 2,
+        "low": 3,
+        "info": 4,
+    }
+    return order.get((value or "").lower(), 99)
+
+
+def _normalize_geo_key(country: str | None, city: str | None) -> str:
+    if city and country:
+        return f"{city}, {country}"
+    return city or country or "未知区域"
+
+
+def _top_counter_items(counter: Counter[str], limit: int = 5) -> dict[str, int]:
+    return {name: count for name, count in counter.most_common(limit) if name}
+
+
+def _collect_incident_regions(incidents: list[BGPIncident]) -> Counter[str]:
+    counter: Counter[str] = Counter()
+    for item in incidents:
+        for region in item.affected_regions or []:
+            if not isinstance(region, dict):
+                continue
+            counter[_normalize_geo_key(region.get("country"), region.get("city"))] += 1
+    return counter
+
+
+def _collect_collector_regions(collectors: list[dict[str, Any]]) -> Counter[str]:
+    counter: Counter[str] = Counter()
+    for item in collectors:
+        counter[_normalize_geo_key(item.get("country"), item.get("city"))] += int(item.get("recent_24h_observation_count") or 0)
+    return counter
+
+
+def _format_geo_evidence(prefix_geographies: dict[str, dict[str, Any]], limit: int = 6) -> str:
+    if not prefix_geographies:
+        return "没有命中 prefix geography 证据。"
+
+    rows = []
+    for prefix, item in list(prefix_geographies.items())[:limit]:
+        region = _normalize_geo_key(item.get("country"), item.get("city"))
+        source = item.get("source") or item.get("geography_mode") or "unknown"
+        as_hint = item.get("asn")
+        as_name = item.get("as_name")
+        as_text = ""
+        if as_hint:
+            as_text = f" / ASN AS{as_hint}"
+            if as_name:
+                as_text += f" ({as_name})"
+        rows.append(f"{prefix} -> {region} / 来源 {source}{as_text}")
+    return "；".join(rows)
+
+
+async def build_bgp_brief_request(
+    db: AsyncSession,
+    *,
+    incident_limit: int = 5,
+    anomaly_limit: int = 6,
+    collector_limit: int = 5,
+) -> tuple[SituationalAnalysisRequest, list[str], dict[str, int | str | dict[str, int]]]:
+    incidents_result = await db.execute(
+        select(BGPIncident)
+        .order_by(BGPIncident.created_at.desc(), BGPIncident.id.desc())
+        .limit(max(incident_limit, 1))
+    )
+    anomalies_result = await db.execute(
+        select(BGPAnomaly)
+        .order_by(BGPAnomaly.created_at.desc(), BGPAnomaly.id.desc())
+        .limit(max(anomaly_limit, 1))
+    )
+    observations_result = await db.execute(
+        select(BGPObservation).where(BGPObservation.source.in_(BGP_SOURCES))
+    )
+    incident_count_result = await db.execute(select(func.count(BGPIncident.id)))
+    anomaly_count_result = await db.execute(select(func.count(BGPAnomaly.id)))
+
+    incidents = incidents_result.scalars().all()
+    anomalies = anomalies_result.scalars().all()
+    observations = observations_result.scalars().all()
+    collectors = await build_bgp_collector_coverage(db, source_filter=BGP_SOURCES)
+
+    total_incidents = incident_count_result.scalar() or 0
+    total_anomalies = anomaly_count_result.scalar() or 0
+    total_observations = len(observations)
+    active_collectors = [item for item in collectors if item["observation_count"] > 0]
+
+    incident_status_counts = Counter((item.status or "unknown") for item in incidents)
+    incident_severity_counts = Counter((item.severity or "unknown") for item in incidents)
+    incident_type_counts = Counter((item.incident_type or "unknown") for item in incidents)
+    anomaly_type_counts = Counter((item.anomaly_type or "unknown") for item in anomalies)
+    event_type_counts = Counter((item.event_type or "unknown") for item in observations)
+    incident_region_counts = _collect_incident_regions(incidents)
+
+    top_collectors = sorted(
+        active_collectors,
+        key=lambda item: (
+            -int(item["recent_24h_observation_count"]),
+            -int(item["observation_count"]),
+            str(item["collector"]),
+        ),
+    )[: max(collector_limit, 1)]
+    collector_region_counts = _collect_collector_regions(top_collectors)
+
+    prefix_candidates = sorted(
+        {
+            prefix
+            for item in incidents
+            for prefix in (item.affected_prefixes or [])
+            if prefix
+        }
+        | {item.prefix for item in anomalies if item.prefix}
+    )
+    prefix_geographies = await lookup_prefix_geography(db, prefix_candidates) if prefix_candidates else {}
+    geography_region_counts = Counter(
+        _normalize_geo_key(item.get("country"), item.get("city"))
+        for item in prefix_geographies.values()
+        if item.get("country") or item.get("city")
+    )
+    hotspot_region_counts = geography_region_counts + incident_region_counts
+    collector_bias_regions = [
+        region
+        for region, count in collector_region_counts.most_common(3)
+        if count > hotspot_region_counts.get(region, 0)
+    ]
+
+    observations_lines: list[str] = [
+        f"当前共有 {total_incidents} 起 BGP incidents、{total_anomalies} 条 anomalies、{total_observations} 条原始观测事件。",
+        f"活跃观测站 {len(active_collectors)} 个；近 24 小时事件数合计 {sum(int(item['recent_24h_observation_count']) for item in active_collectors)}。",
+        f"最近 incidents 严重度分布：{_format_counter(dict(sorted(incident_severity_counts.items(), key=lambda item: _severity_rank(item[0]))))}。",
+        f"最近 incidents 状态分布：{_format_counter(dict(incident_status_counts))}。",
+        f"最近 incidents 类型分布：{_format_counter(dict(incident_type_counts.most_common(5)))}。",
+        f"最近 anomalies 类型分布：{_format_counter(dict(anomaly_type_counts.most_common(6)))}。",
+        f"观测事件类型分布：{_format_counter(dict(event_type_counts.most_common(6)))}。",
+    ]
+
+    if hotspot_region_counts:
+        observations_lines.append(
+            "区域热点事实层："
+            + _format_counter(_top_counter_items(hotspot_region_counts, limit=5), empty_text="无明显区域聚集")
+            + "。"
+        )
+
+    if prefix_geographies:
+        observations_lines.append("Prefix geography 证据：" + _format_geo_evidence(prefix_geographies))
+
+    if collector_bias_regions:
+        observations_lines.append(
+            "观测偏差提示：重点观测站最近 24h 活跃度更集中在 "
+            + "、".join(collector_bias_regions)
+            + "，这些区域的事件升温结论需要结合 prefix geography 与 affected regions 交叉验证。"
+        )
+    elif top_collectors:
+        observations_lines.append(
+            "观测偏差提示：当前未发现明显高于区域热点事实层的单一观测站集中区域，但仍需区分 collector coverage 与真实区域风险。"
+        )
+
+    if incidents:
+        observations_lines.append(
+            "最近 incident 摘要：" + "；".join(
+                [
+                    f"{item.incident_type} / {item.severity} / {item.status}"
+                    f" / 前缀 {', '.join(item.affected_prefixes[:2]) if item.affected_prefixes else '-'}"
+                    f" / 观测站 {len(item.affected_collectors or [])} 个"
+                    for item in incidents
+                ]
+            )
+        )
+
+    if anomalies:
+        observations_lines.append(
+            "最近 anomaly 摘要：" + "；".join(
+                [
+                    f"{item.anomaly_type} / {item.severity}"
+                    f" / 前缀 {item.prefix or '-'}"
+                    f" / ASN {item.new_origin_asn or item.origin_asn or '-'}"
+                    for item in anomalies
+                ]
+            )
+        )
+
+    if top_collectors:
+        observations_lines.append(
+            "重点观测站：" + "；".join(
+                [
+                    f"{item['collector']} ({', '.join([part for part in [item.get('city'), item.get('country')] if part]) or '未知位置'})"
+                    f" / 近24h {item['recent_24h_observation_count']} 条"
+                    f" / 前缀 {item['prefix_count']} 个"
+                    for item in top_collectors
+                ]
+            )
+        )
+
+    context = {
+        "source": "bgp-overview",
+        "incident_total": total_incidents,
+        "anomaly_total": total_anomalies,
+        "observation_total": total_observations,
+        "active_collectors": len(active_collectors),
+        "top_incident_types": dict(incident_type_counts.most_common(5)),
+        "top_anomaly_types": dict(anomaly_type_counts.most_common(6)),
+        "top_event_types": dict(event_type_counts.most_common(6)),
+        "region_hotspots": _top_counter_items(hotspot_region_counts, limit=6),
+        "incident_regions": _top_counter_items(incident_region_counts, limit=6),
+        "collector_bias_regions": collector_bias_regions,
+        "prefix_geography_sources": dict(
+            Counter(str(item.get("source") or "unknown") for item in prefix_geographies.values()).most_common(5)
+        ),
+        "prefix_geography_sample": {
+            prefix: {
+                "country": item.get("country"),
+                "city": item.get("city"),
+                "source": item.get("source"),
+                "asn": item.get("asn"),
+                "as_name": item.get("as_name"),
+            }
+            for prefix, item in list(prefix_geographies.items())[:8]
+        },
+    }
+
+    return SituationalAnalysisRequest(
+        title="BGP 态势 AI 简报",
+        objective="基于当前 BGP incidents、anomalies、原始观测事件、观测站覆盖与 prefix geography 证据，生成一份面向操作员的简明态势简报，突出区域热点、观测偏差、当前风险、证据和优先动作。",
+        observations=observations_lines,
+        constraints=[
+            "明确区分事实、推断与建议。",
+            "优先指出需要立即关注的高严重度 incident 或异常模式。",
+            "需要单独指出哪些区域结论来自 prefix geography / affected regions，哪些可能受 collector coverage 偏差影响。",
+            "结论应服务值班排障，不要写成泛泛的模型演示文案。",
+            "如果证据不足，要明确指出缺失数据。",
+        ],
+        context=context,
+    ), observations_lines, context

backend/app/services/bgp_ai_brief_store.py

@@ -0,0 +1,160 @@
+from __future__ import annotations
+
+import json
+from dataclasses import dataclass
+from datetime import UTC, datetime
+from pathlib import Path
+from typing import Any
+from uuid import uuid4
+
+from app.core.config import ROOT_DIR
+from app.schemas.ai import BGPBriefRecordResponse, BGPBriefRecordSummary, SituationalAnalysisResponse
+
+
+_BRIEF_STORAGE_DIR = ROOT_DIR / "data" / "ai" / "bgp-briefs"
+_METADATA_PREFIX = "<!-- planet-bgp-brief-meta "
+_METADATA_SUFFIX = " -->"
+_BRIEF_TITLE = "BGP AI 简报"
+
+
+@dataclass(slots=True)
+class _StoredBrief:
+    id: str
+    title: str
+    provider: str
+    model: str
+    request_id: str | None
+    generated_at: str
+    content_markdown: str
+    facts: list[str]
+    context: dict[str, Any]
+    path: Path
+
+
+def _ensure_storage_dir() -> Path:
+    _BRIEF_STORAGE_DIR.mkdir(parents=True, exist_ok=True)
+    return _BRIEF_STORAGE_DIR
+
+
+def _build_metadata_line(metadata: dict[str, Any]) -> str:
+    return f"{_METADATA_PREFIX}{json.dumps(metadata, ensure_ascii=False)}{_METADATA_SUFFIX}"
+
+
+def _parse_brief_file(path: Path) -> _StoredBrief | None:
+    try:
+        raw_text = path.read_text(encoding="utf-8")
+    except OSError:
+        return None
+
+    first_line, separator, remainder = raw_text.partition("\n")
+    if not separator or not first_line.startswith(_METADATA_PREFIX) or not first_line.endswith(_METADATA_SUFFIX):
+        return None
+
+    metadata_payload = first_line[len(_METADATA_PREFIX) : -len(_METADATA_SUFFIX)]
+
+    try:
+        metadata = json.loads(metadata_payload)
+    except json.JSONDecodeError:
+        return None
+
+    return _StoredBrief(
+        id=str(metadata.get("id") or path.stem),
+        title=str(metadata.get("title") or _BRIEF_TITLE),
+        provider=str(metadata.get("provider") or "-"),
+        model=str(metadata.get("model") or "-"),
+        request_id=metadata.get("request_id"),
+        generated_at=str(metadata.get("generated_at") or datetime.fromtimestamp(path.stat().st_mtime, UTC).isoformat()),
+        content_markdown=remainder.lstrip("\n"),
+        facts=list(metadata.get("facts") or []),
+        context=dict(metadata.get("context") or {}),
+        path=path,
+    )
+
+
+def list_bgp_brief_records(limit: int = 50) -> list[BGPBriefRecordSummary]:
+    storage_dir = _ensure_storage_dir()
+    records: list[_StoredBrief] = []
+
+    for path in storage_dir.glob("*.md"):
+        parsed = _parse_brief_file(path)
+        if parsed is not None:
+            records.append(parsed)
+
+    records.sort(key=lambda item: item.generated_at, reverse=True)
+
+    return [
+        BGPBriefRecordSummary(
+            id=item.id,
+            title=item.title,
+            provider=item.provider,
+            model=item.model,
+            request_id=item.request_id,
+            generated_at=item.generated_at,
+        )
+        for item in records[: max(limit, 1)]
+    ]
+
+
+def get_bgp_brief_record(brief_id: str) -> BGPBriefRecordResponse | None:
+    path = _ensure_storage_dir() / f"{brief_id}.md"
+    parsed = _parse_brief_file(path)
+    if parsed is None:
+        return None
+
+    return BGPBriefRecordResponse(
+        id=parsed.id,
+        title=parsed.title,
+        provider=parsed.provider,
+        model=parsed.model,
+        request_id=parsed.request_id,
+        generated_at=parsed.generated_at,
+        content_markdown=parsed.content_markdown,
+        facts=parsed.facts,
+        context=parsed.context,
+    )
+
+
+def get_latest_bgp_brief_record() -> BGPBriefRecordResponse | None:
+    summaries = list_bgp_brief_records(limit=1)
+    if not summaries:
+        return None
+    return get_bgp_brief_record(summaries[0].id)
+
+
+def save_bgp_brief_record(
+    analysis: SituationalAnalysisResponse,
+    *,
+    request_id: str | None,
+    facts: list[str] | None = None,
+    context: dict[str, Any] | None = None,
+    generated_at: datetime | None = None,
+) -> BGPBriefRecordResponse:
+    created_at = generated_at or datetime.now(UTC)
+    brief_id = f"{created_at.strftime('%Y%m%dT%H%M%SZ')}-{uuid4().hex[:8]}"
+    path = _ensure_storage_dir() / f"{brief_id}.md"
+
+    metadata = {
+        "id": brief_id,
+        "title": _BRIEF_TITLE,
+        "provider": analysis.provider,
+        "model": analysis.model,
+        "request_id": request_id,
+        "generated_at": created_at.isoformat(),
+        "facts": facts or [],
+        "context": context or {},
+    }
+
+    markdown_text = f"{_build_metadata_line(metadata)}\n\n{analysis.content.rstrip()}\n"
+    path.write_text(markdown_text, encoding="utf-8")
+
+    return BGPBriefRecordResponse(
+        id=brief_id,
+        title=_BRIEF_TITLE,
+        provider=analysis.provider,
+        model=analysis.model,
+        request_id=request_id,
+        generated_at=created_at.isoformat(),
+        content_markdown=analysis.content,
+        facts=facts or [],
+        context=context or {},
+    )

backend/app/services/bgp_collectors.py

@@ -0,0 +1,208 @@
+"""Collector baseline and coverage helpers for BGP observations."""
+
+from __future__ import annotations
+
+from collections import defaultdict
+from datetime import UTC, datetime, timedelta
+from typing import Any
+
+from sqlalchemy import case, distinct, func, select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.core.time import to_iso8601_utc
+from app.models.bgp_observation import BGPObservation
+from app.services.collectors.bgp_common import RIPE_RIS_COLLECTOR_COORDS
+
+
+def _collector_base_filters(source_filter: tuple[str, ...] | None) -> list[Any]:
+    filters: list[Any] = [
+        BGPObservation.collector.isnot(None),
+        func.length(func.btrim(BGPObservation.collector)) > 0,
+    ]
+    if source_filter:
+        filters.append(BGPObservation.source.in_(source_filter))
+    return filters
+
+
+async def build_bgp_collector_coverage(
+    db: AsyncSession,
+    *,
+    source_filter: tuple[str, ...] | None = None,
+) -> list[dict[str, Any]]:
+    now = datetime.now(UTC)
+    recent_15m_threshold = now - timedelta(minutes=15)
+    recent_24h_threshold = now - timedelta(hours=24)
+    recent_7d_threshold = now - timedelta(days=7)
+
+    filters = _collector_base_filters(source_filter)
+    country_expr = func.nullif(BGPObservation.collector_geo["country"].as_string(), "")
+    city_expr = func.nullif(BGPObservation.collector_geo["city"].as_string(), "")
+
+    aggregate_stmt = (
+        select(
+            BGPObservation.collector.label("collector"),
+            func.count(BGPObservation.id).label("observation_count"),
+            func.count(distinct(BGPObservation.prefix)).label("prefix_count"),
+            func.count(distinct(BGPObservation.origin_asn)).label("origin_asn_count"),
+            func.count(distinct(BGPObservation.peer_asn)).label("peer_asn_count"),
+            func.sum(case((BGPObservation.observed_at >= recent_15m_threshold, 1), else_=0)).label("recent_15m_observation_count"),
+            func.sum(case((BGPObservation.observed_at >= recent_24h_threshold, 1), else_=0)).label("recent_24h_observation_count"),
+            func.sum(case((BGPObservation.observed_at >= recent_7d_threshold, 1), else_=0)).label("recent_7d_observation_count"),
+            func.count(distinct(case((BGPObservation.observed_at >= recent_15m_threshold, BGPObservation.prefix), else_=None))).label("recent_15m_prefix_count"),
+            func.count(distinct(case((BGPObservation.observed_at >= recent_24h_threshold, BGPObservation.prefix), else_=None))).label("recent_24h_prefix_count"),
+            func.count(distinct(case((BGPObservation.observed_at >= recent_7d_threshold, BGPObservation.prefix), else_=None))).label("recent_7d_prefix_count"),
+            func.max(BGPObservation.observed_at).label("latest_observed_at"),
+        )
+        .where(*filters)
+        .group_by(BGPObservation.collector)
+    )
+    aggregate_rows = (await db.execute(aggregate_stmt)).all()
+
+    latest_subquery = (
+        select(
+            BGPObservation.collector.label("collector"),
+            BGPObservation.event_type.label("latest_event_type"),
+            country_expr.label("country"),
+            city_expr.label("city"),
+            func.row_number()
+            .over(
+                partition_by=BGPObservation.collector,
+                order_by=(BGPObservation.observed_at.desc(), BGPObservation.id.desc()),
+            )
+            .label("rn"),
+        )
+        .where(*filters)
+        .subquery()
+    )
+    latest_rows = (
+        await db.execute(
+            select(
+                latest_subquery.c.collector,
+                latest_subquery.c.latest_event_type,
+                latest_subquery.c.country,
+                latest_subquery.c.city,
+            ).where(latest_subquery.c.rn == 1)
+        )
+    ).all()
+
+    event_counts_subquery = (
+        select(
+            BGPObservation.collector.label("collector"),
+            BGPObservation.event_type.label("event_type"),
+            func.count(BGPObservation.id).label("count"),
+            func.row_number()
+            .over(
+                partition_by=BGPObservation.collector,
+                order_by=(func.count(BGPObservation.id).desc(), BGPObservation.event_type.asc()),
+            )
+            .label("rn"),
+        )
+        .where(*filters)
+        .group_by(BGPObservation.collector, BGPObservation.event_type)
+        .subquery()
+    )
+    top_event_rows = (
+        await db.execute(
+            select(
+                event_counts_subquery.c.collector,
+                event_counts_subquery.c.event_type,
+                event_counts_subquery.c.count,
+            ).where(event_counts_subquery.c.rn <= 3)
+        )
+    ).all()
+
+    scope_rows = (
+        await db.execute(
+            select(
+                BGPObservation.collector.label("collector"),
+                country_expr.label("country"),
+                city_expr.label("city"),
+            )
+            .where(*filters)
+            .distinct()
+        )
+    ).all()
+
+    latest_by_collector = {
+        row.collector: {
+            "latest_event_type": row.latest_event_type,
+            "country": row.country,
+            "city": row.city,
+        }
+        for row in latest_rows
+    }
+
+    scope_by_collector: dict[str, dict[str, set[str]]] = defaultdict(lambda: {"countries": set(), "cities": set()})
+    for row in scope_rows:
+        if row.country:
+            scope_by_collector[row.collector]["countries"].add(row.country)
+        if row.city:
+            scope_by_collector[row.collector]["cities"].add(row.city)
+
+    top_events_by_collector: dict[str, list[dict[str, Any]]] = defaultdict(list)
+    for row in top_event_rows:
+        top_events_by_collector[row.collector].append(
+            {"event_type": row.event_type, "count": row.count}
+        )
+
+    by_collector: dict[str, dict[str, Any]] = {}
+    for row in aggregate_rows:
+        collector = row.collector
+        latest = latest_by_collector.get(collector, {})
+        fallback_location = RIPE_RIS_COLLECTOR_COORDS.get(collector, {})
+        scope = scope_by_collector.get(collector, {"countries": set(), "cities": set()})
+
+        by_collector[collector] = {
+            "collector": collector,
+            "city": latest.get("city") or fallback_location.get("city"),
+            "country": latest.get("country") or fallback_location.get("country"),
+            "latitude": fallback_location.get("latitude"),
+            "longitude": fallback_location.get("longitude"),
+            "observation_count": row.observation_count or 0,
+            "prefix_count": row.prefix_count or 0,
+            "origin_asn_count": row.origin_asn_count or 0,
+            "peer_asn_count": row.peer_asn_count or 0,
+            "recent_15m_observation_count": row.recent_15m_observation_count or 0,
+            "recent_24h_observation_count": row.recent_24h_observation_count or 0,
+            "recent_7d_observation_count": row.recent_7d_observation_count or 0,
+            "recent_15m_prefix_count": row.recent_15m_prefix_count or 0,
+            "recent_24h_prefix_count": row.recent_24h_prefix_count or 0,
+            "recent_7d_prefix_count": row.recent_7d_prefix_count or 0,
+            "top_event_types": top_events_by_collector.get(collector, []),
+            "latest_observed_at": to_iso8601_utc(row.latest_observed_at),
+            "latest_event_type": latest.get("latest_event_type"),
+            "baseline_scope": {
+                "countries": sorted(scope["countries"]),
+                "cities": sorted(scope["cities"]),
+            },
+        }
+
+    for collector, location in RIPE_RIS_COLLECTOR_COORDS.items():
+        if collector in by_collector:
+            continue
+        by_collector[collector] = {
+            "collector": collector,
+            "city": location.get("city"),
+            "country": location.get("country"),
+            "latitude": location.get("latitude"),
+            "longitude": location.get("longitude"),
+            "observation_count": 0,
+            "prefix_count": 0,
+            "origin_asn_count": 0,
+            "peer_asn_count": 0,
+            "recent_15m_observation_count": 0,
+            "recent_24h_observation_count": 0,
+            "recent_7d_observation_count": 0,
+            "recent_15m_prefix_count": 0,
+            "recent_24h_prefix_count": 0,
+            "recent_7d_prefix_count": 0,
+            "top_event_types": [],
+            "latest_observed_at": None,
+            "latest_event_type": None,
+            "baseline_scope": {
+                "countries": [location["country"]] if location.get("country") else [],
+                "cities": [location["city"]] if location.get("city") else [],
+            },
+        }
+
+    return [by_collector[collector] for collector in sorted(by_collector.keys())]

backend/app/services/bgp_detectors.py

@@ -0,0 +1,466 @@
+"""Detector helpers for BGP anomaly generation."""
+
+from __future__ import annotations
+
+from collections import Counter, defaultdict
+from datetime import UTC, datetime
+from typing import Any
+
+from app.models.bgp_anomaly import BGPAnomaly
+
+
+def _iter_event_regions(events: list[dict[str, Any]]) -> list[dict[str, Any]]:
+    regions: list[dict[str, Any]] = []
+    seen: set[tuple[Any, ...]] = set()
+    for event in events:
+        metadata = event.get("metadata") or {}
+        location = metadata.get("collector_location") or {}
+        region = {
+            "collector": metadata.get("collector"),
+            "country": location.get("country"),
+            "city": location.get("city"),
+            "latitude": location.get("latitude"),
+            "longitude": location.get("longitude"),
+        }
+        region_key = (
+            region.get("collector"),
+            region.get("country"),
+            region.get("city"),
+            region.get("latitude"),
+            region.get("longitude"),
+        )
+        if region_key in seen:
+            continue
+        seen.add(region_key)
+        regions.append(region)
+    return regions
+
+
+def _unique_collectors(events: list[dict[str, Any]]) -> list[str]:
+    return sorted(
+        {
+            str((event.get("metadata") or {}).get("collector"))
+            for event in events
+            if (event.get("metadata") or {}).get("collector")
+        }
+    )
+
+
+def _unique_peers(events: list[dict[str, Any]]) -> list[int]:
+    peers: set[int] = set()
+    for event in events:
+        peer_asn = (event.get("metadata") or {}).get("peer_asn")
+        if peer_asn is not None:
+            peers.add(int(peer_asn))
+    return sorted(peers)
+
+
+def _path_signature(metadata: dict[str, Any]) -> tuple[int, ...]:
+    path = metadata.get("as_path") or []
+    return tuple(int(asn) for asn in path if asn is not None)
+
+
+def detect_origin_change_anomalies(
+    *,
+    source: str,
+    snapshot_id: int | None,
+    task_id: int | None,
+    events: list[dict[str, Any]],
+    previous_origin_map: dict[str, set[int]],
+) -> list[BGPAnomaly]:
+    prefix_to_origins: defaultdict[str, set[int]] = defaultdict(set)
+    for event in events:
+        metadata = event.get("metadata") or {}
+        prefix = metadata.get("prefix")
+        origin_asn = metadata.get("origin_asn")
+        if prefix and origin_asn is not None:
+            prefix_to_origins[str(prefix)].add(int(origin_asn))
+
+    anomalies: list[BGPAnomaly] = []
+    for prefix, origins in prefix_to_origins.items():
+        historic = previous_origin_map.get(prefix, set())
+        new_origins = sorted(origin for origin in origins if origin not in historic)
+        related_events = [
+            event
+            for event in events
+            if (event.get("metadata") or {}).get("prefix") == prefix
+        ]
+        related_collectors = _unique_collectors(related_events)
+        related_regions = _iter_event_regions(related_events)
+
+        moas_candidate = not historic and len(origins) >= 2 and len(related_collectors) >= 2
+        if (not historic or not new_origins) and not moas_candidate:
+            continue
+
+        target_origins = new_origins or sorted(origins)
+        for new_origin in target_origins:
+            sample_event = next(
+                (
+                    event
+                    for event in related_events
+                    if (event.get("metadata") or {}).get("prefix") == prefix
+                    and int((event.get("metadata") or {}).get("origin_asn") or -1) == new_origin
+                ),
+                {},
+            )
+            sample_metadata = sample_event.get("metadata") or {}
+            sample_enrichment = sample_metadata.get("enrichment") or {}
+            sample_prefix_geography = sample_enrichment.get("prefix_geography") or {}
+            anomaly_type = "origin_change"
+            severity = "critical"
+            confidence = 0.86
+            summary = f"Prefix {prefix} is now originated by AS{new_origin}, outside the current baseline."
+            evidence_previous_origins = sorted(historic)
+            if moas_candidate and not historic:
+                anomaly_type = "origin_conflict"
+                severity = "high"
+                confidence = 0.74
+                summary = (
+                    f"Prefix {prefix} is being originated by multiple ASNs "
+                    f"{sorted(origins)} across {len(related_collectors)} collectors."
+                )
+                evidence_previous_origins = []
+            anomalies.append(
+                BGPAnomaly(
+                    snapshot_id=snapshot_id,
+                    task_id=task_id,
+                    source=source,
+                    anomaly_type=anomaly_type,
+                    severity=severity,
+                    status="active",
+                    entity_key=f"{anomaly_type}:{prefix}:{new_origin}",
+                    prefix=prefix,
+                    origin_asn=sorted(historic)[0] if historic else None,
+                    new_origin_asn=new_origin,
+                    peer_scope=related_collectors,
+                    started_at=datetime.now(UTC),
+                    confidence=confidence,
+                    summary=summary,
+                    evidence={
+                        "previous_origins": evidence_previous_origins,
+                        "current_origins": sorted(origins),
+                        "events": [
+                            (item.get("metadata") or {})
+                            for item in related_events[:10]
+                        ],
+                        "origin_asn_profile": sample_enrichment.get("origin_asn_profile"),
+                        "new_origin_asn_profile": sample_enrichment.get("new_origin_asn_profile"),
+                        "rpki_validation": sample_enrichment.get("rpki_validation"),
+                        "prefix_geography": sample_prefix_geography,
+                        "prefix_scope": sample_enrichment.get("prefix_scope"),
+                        "impacted_regions": sample_prefix_geography.get("regions")
+                        or related_regions
+                        or sample_enrichment.get("prefix_scope", {}).get("regions", []),
+                    },
+                )
+            )
+
+    return anomalies
+
+
+def detect_more_specific_burst_anomalies(
+    *,
+    source: str,
+    snapshot_id: int | None,
+    task_id: int | None,
+    events: list[dict[str, Any]],
+) -> list[BGPAnomaly]:
+    prefix_to_more_specifics: defaultdict[str, list[dict[str, Any]]] = defaultdict(list)
+    for event in events:
+        metadata = event.get("metadata") or {}
+        enrichment = metadata.get("enrichment") or {}
+        root_prefix = enrichment.get("prefix_supernet")
+        if root_prefix and enrichment.get("is_more_specific"):
+            prefix_to_more_specifics[str(root_prefix)].append(event)
+
+    anomalies: list[BGPAnomaly] = []
+    for root_prefix, more_specifics in prefix_to_more_specifics.items():
+        unique_prefixes = sorted(
+            {
+                str((item.get("metadata") or {}).get("prefix"))
+                for item in more_specifics
+                if (item.get("metadata") or {}).get("prefix")
+            }
+        )
+        related_collectors = _unique_collectors(more_specifics)
+        if len(unique_prefixes) < 2 and len(related_collectors) < 2:
+            continue
+
+        sample = more_specifics[0].get("metadata") or {}
+        sample_enrichment = sample.get("enrichment") or {}
+        sample_prefix_geography = sample_enrichment.get("prefix_geography") or {}
+        event_count = len(more_specifics)
+        anomalies.append(
+            BGPAnomaly(
+                snapshot_id=snapshot_id,
+                task_id=task_id,
+                source=source,
+                anomaly_type="more_specific_burst",
+                severity="high",
+                status="active",
+                entity_key=f"more_specific_burst:{root_prefix}:{len(unique_prefixes)}:{len(related_collectors)}",
+                prefix=sample.get("prefix"),
+                origin_asn=sample.get("origin_asn"),
+                new_origin_asn=None,
+                peer_scope=related_collectors,
+                started_at=datetime.now(UTC),
+                confidence=min(0.64 + (0.04 * min(event_count, 5)), 0.88),
+                summary=(
+                    f"{len(unique_prefixes)} more-specific prefixes clustered under {root_prefix} "
+                    f"across {len(related_collectors) or 1} collectors."
+                ),
+                evidence={
+                    "events": [item.get("metadata") for item in more_specifics[:10]],
+                    "unique_prefixes": unique_prefixes,
+                    "rpki_validation": sample_enrichment.get("rpki_validation"),
+                    "origin_asn_profile": sample_enrichment.get("origin_asn_profile"),
+                    "prefix_geography": sample_prefix_geography,
+                    "prefix_scope": sample_enrichment.get("prefix_scope"),
+                    "impacted_regions": sample_prefix_geography.get("regions")
+                    or _iter_event_regions(more_specifics)
+                    or sample_enrichment.get("prefix_scope", {}).get("regions", []),
+                },
+            )
+        )
+
+    return anomalies
+
+
+def detect_mass_withdrawal_anomalies(
+    *,
+    source: str,
+    snapshot_id: int | None,
+    task_id: int | None,
+    events: list[dict[str, Any]],
+) -> list[BGPAnomaly]:
+    withdrawal_counter: Counter[tuple[str, int | None]] = Counter()
+    withdrawal_events_by_key: defaultdict[tuple[str, int | None], list[dict[str, Any]]] = defaultdict(list)
+    for event in events:
+        metadata = event.get("metadata") or {}
+        prefix = metadata.get("prefix")
+        if prefix and metadata.get("event_type") == "withdrawal":
+            key = (str(prefix), metadata.get("origin_asn"))
+            withdrawal_counter[key] += 1
+            withdrawal_events_by_key[key].append(event)
+
+    anomalies: list[BGPAnomaly] = []
+    for (prefix, origin_asn), count in withdrawal_counter.items():
+        related_events = withdrawal_events_by_key[(prefix, origin_asn)]
+        related_collectors = _unique_collectors(related_events)
+        related_peers = _unique_peers(related_events)
+        if count < 3 and not (count >= 2 and len(related_collectors) >= 2):
+            continue
+        sample_event = related_events[0] if related_events else {}
+        sample_metadata = sample_event.get("metadata") or {}
+        sample_enrichment = sample_metadata.get("enrichment") or {}
+        sample_prefix_geography = sample_enrichment.get("prefix_geography") or {}
+        severity = "medium"
+        if count >= 4 or len(related_collectors) >= 3:
+            severity = "high"
+        if count >= 8:
+            severity = "critical"
+
+        anomalies.append(
+            BGPAnomaly(
+                snapshot_id=snapshot_id,
+                task_id=task_id,
+                source=source,
+                anomaly_type="mass_withdrawal",
+                severity=severity,
+                status="active",
+                entity_key=f"mass_withdrawal:{prefix}:{origin_asn}:{len(related_collectors)}:{count}",
+                prefix=prefix,
+                origin_asn=origin_asn,
+                new_origin_asn=None,
+                peer_scope=related_collectors,
+                started_at=datetime.now(UTC),
+                confidence=min(0.5 + (count * 0.06) + (0.04 * max(len(related_collectors) - 1, 0)), 0.95),
+                summary=(
+                    f"{count} withdrawal events observed for {prefix} "
+                    f"across {len(related_collectors) or 1} collectors in the current ingest window."
+                ),
+                evidence={
+                    "withdrawal_count": count,
+                    "collector_count": len(related_collectors),
+                    "peer_count": len(related_peers),
+                    "events": [
+                        (item.get("metadata") or {})
+                        for item in related_events[:10]
+                    ],
+                    "origin_asn_profile": sample_enrichment.get("origin_asn_profile"),
+                    "rpki_validation": sample_enrichment.get("rpki_validation"),
+                    "prefix_geography": sample_prefix_geography,
+                    "prefix_scope": sample_enrichment.get("prefix_scope"),
+                    "impacted_regions": sample_prefix_geography.get("regions")
+                    or _iter_event_regions(related_events)
+                    or sample_enrichment.get("prefix_scope", {}).get("regions", []),
+                },
+            )
+        )
+
+    return anomalies
+
+
+def detect_route_leak_anomalies(
+    *,
+    source: str,
+    snapshot_id: int | None,
+    task_id: int | None,
+    events: list[dict[str, Any]],
+) -> list[BGPAnomaly]:
+    events_by_prefix: defaultdict[str, list[dict[str, Any]]] = defaultdict(list)
+    for event in events:
+        metadata = event.get("metadata") or {}
+        prefix = metadata.get("prefix")
+        if prefix and metadata.get("event_type") == "announcement":
+            events_by_prefix[str(prefix)].append(event)
+
+    anomalies: list[BGPAnomaly] = []
+    for prefix, related_events in events_by_prefix.items():
+        related_collectors = _unique_collectors(related_events)
+        if len(related_collectors) < 2:
+            continue
+
+        path_signatures = Counter()
+        max_path_length = 0
+        for event in related_events:
+            metadata = event.get("metadata") or {}
+            signature = _path_signature(metadata)
+            if signature:
+                path_signatures[signature] += 1
+                max_path_length = max(max_path_length, len(signature))
+
+        if len(path_signatures) < 2:
+            continue
+
+        dominant_length = len(path_signatures.most_common(1)[0][0])
+        if max_path_length < max(dominant_length + 2, 5):
+            continue
+
+        sample_event = max(
+            related_events,
+            key=lambda event: len(_path_signature((event.get("metadata") or {}))),
+        )
+        sample_metadata = sample_event.get("metadata") or {}
+        sample_enrichment = sample_metadata.get("enrichment") or {}
+        sample_prefix_geography = sample_enrichment.get("prefix_geography") or {}
+        peer_scope = related_collectors
+        path_lengths = sorted({len(signature) for signature in path_signatures if signature})
+
+        anomalies.append(
+            BGPAnomaly(
+                snapshot_id=snapshot_id,
+                task_id=task_id,
+                source=source,
+                anomaly_type="route_leak_candidate",
+                severity="high" if max_path_length >= dominant_length + 3 else "medium",
+                status="active",
+                entity_key=f"route_leak_candidate:{prefix}:{max_path_length}:{len(related_collectors)}",
+                prefix=prefix,
+                origin_asn=sample_metadata.get("origin_asn"),
+                new_origin_asn=None,
+                peer_scope=peer_scope,
+                started_at=datetime.now(UTC),
+                confidence=min(0.58 + (0.05 * min(len(related_collectors), 4)) + (0.03 * min(max_path_length - dominant_length, 4)), 0.88),
+                summary=(
+                    f"Prefix {prefix} shows divergent long AS paths across "
+                    f"{len(related_collectors)} collectors, suggesting a possible route leak."
+                ),
+                evidence={
+                    "path_lengths": path_lengths,
+                    "dominant_path_length": dominant_length,
+                    "max_path_length": max_path_length,
+                    "path_signatures": [
+                        {"path": list(signature), "count": count}
+                        for signature, count in path_signatures.most_common(5)
+                    ],
+                    "events": [(item.get("metadata") or {}) for item in related_events[:10]],
+                    "origin_asn_profile": sample_enrichment.get("origin_asn_profile"),
+                    "rpki_validation": sample_enrichment.get("rpki_validation"),
+                    "prefix_geography": sample_prefix_geography,
+                    "prefix_scope": sample_enrichment.get("prefix_scope"),
+                    "impacted_regions": sample_prefix_geography.get("regions")
+                    or _iter_event_regions(related_events)
+                    or sample_enrichment.get("prefix_scope", {}).get("regions", []),
+                },
+            )
+        )
+
+    return anomalies
+
+
+def detect_path_flap_anomalies(
+    *,
+    source: str,
+    snapshot_id: int | None,
+    task_id: int | None,
+    events: list[dict[str, Any]],
+) -> list[BGPAnomaly]:
+    events_by_prefix: defaultdict[str, list[dict[str, Any]]] = defaultdict(list)
+    for event in events:
+        metadata = event.get("metadata") or {}
+        prefix = metadata.get("prefix")
+        if prefix:
+            events_by_prefix[str(prefix)].append(event)
+
+    anomalies: list[BGPAnomaly] = []
+    for prefix, related_events in events_by_prefix.items():
+        ordered = sorted(
+            related_events,
+            key=lambda event: str((event.get("metadata") or {}).get("timestamp") or ""),
+        )
+        event_types = [str((item.get("metadata") or {}).get("event_type") or "") for item in ordered]
+        transitions = sum(1 for index in range(1, len(event_types)) if event_types[index] != event_types[index - 1])
+        distinct_paths = {
+            _path_signature(item.get("metadata") or {})
+            for item in ordered
+            if _path_signature(item.get("metadata") or {})
+        }
+        related_collectors = _unique_collectors(ordered)
+
+        if transitions < 3 and len(distinct_paths) < 3:
+            continue
+
+        sample_metadata = (ordered[0].get("metadata") or {}) if ordered else {}
+        sample_enrichment = sample_metadata.get("enrichment") or {}
+        sample_prefix_geography = sample_enrichment.get("prefix_geography") or {}
+        severity = "medium"
+        if transitions >= 5 or len(distinct_paths) >= 4:
+            severity = "high"
+
+        anomalies.append(
+            BGPAnomaly(
+                snapshot_id=snapshot_id,
+                task_id=task_id,
+                source=source,
+                anomaly_type="path_flap",
+                severity=severity,
+                status="active",
+                entity_key=f"path_flap:{prefix}:{transitions}:{len(distinct_paths)}",
+                prefix=prefix,
+                origin_asn=sample_metadata.get("origin_asn"),
+                new_origin_asn=None,
+                peer_scope=related_collectors,
+                started_at=datetime.now(UTC),
+                confidence=min(0.54 + (0.05 * min(transitions, 5)) + (0.03 * min(len(distinct_paths), 4)), 0.9),
+                summary=(
+                    f"Prefix {prefix} shows repeated state/path changes "
+                    f"({transitions} transitions, {len(distinct_paths)} distinct paths) in the current window."
+                ),
+                evidence={
+                    "transitions": transitions,
+                    "event_types": event_types[:12],
+                    "distinct_paths": [list(path) for path in list(distinct_paths)[:6]],
+                    "events": [(item.get("metadata") or {}) for item in ordered[:10]],
+                    "origin_asn_profile": sample_enrichment.get("origin_asn_profile"),
+                    "rpki_validation": sample_enrichment.get("rpki_validation"),
+                    "prefix_geography": sample_prefix_geography,
+                    "prefix_scope": sample_enrichment.get("prefix_scope"),
+                    "impacted_regions": sample_prefix_geography.get("regions")
+                    or _iter_event_regions(ordered)
+                    or sample_enrichment.get("prefix_scope", {}).get("regions", []),
+                },
+            )
+        )
+
+    return anomalies

backend/app/services/bgp_enrichment.py

@@ -0,0 +1,432 @@
+"""Enrichment helpers for BGP observation and anomaly pipelines."""
+
+from __future__ import annotations
+
+import ipaddress
+from collections import defaultdict
+from datetime import UTC, datetime
+from typing import Any
+
+from sqlalchemy import Integer, cast, select, text
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.core.countries import get_country_centroid, normalize_country
+from app.models.bgp_observation import BGPObservation
+from app.models.collected_data import CollectedData
+
+
+def _safe_int(value: Any) -> int | None:
+    try:
+        if value in (None, ""):
+            return None
+        return int(value)
+    except (TypeError, ValueError):
+        return None
+
+
+def _parse_timestamp(value: Any) -> datetime:
+    if isinstance(value, datetime):
+        return value.astimezone(UTC) if value.tzinfo else value.replace(tzinfo=UTC)
+
+    if isinstance(value, (int, float)):
+        return datetime.fromtimestamp(value, tz=UTC)
+
+    if isinstance(value, str) and value:
+        normalized = value.replace("Z", "+00:00")
+        parsed = datetime.fromisoformat(normalized)
+        return parsed.astimezone(UTC) if parsed.tzinfo else parsed.replace(tzinfo=UTC)
+
+    return datetime.now(UTC)
+
+
+def _dedupe_as_path(as_path: list[int]) -> list[int]:
+    deduped: list[int] = []
+    for asn in as_path:
+        if not deduped or deduped[-1] != asn:
+            deduped.append(asn)
+    return deduped
+
+
+def _compact_locations(items: list[dict[str, Any]]) -> list[dict[str, Any]]:
+    results: list[dict[str, Any]] = []
+    seen: set[tuple[Any, ...]] = set()
+    for item in items:
+        key = (
+            item.get("country"),
+            item.get("city"),
+            item.get("latitude"),
+            item.get("longitude"),
+        )
+        if key in seen:
+            continue
+        seen.add(key)
+        results.append(item)
+    return results
+
+
+def extract_bgp_network_fields(prefix: str) -> dict[str, Any]:
+    if not prefix:
+        return {
+            "prefix_family": None,
+            "prefix_length": None,
+            "prefix_supernet": None,
+            "is_more_specific": False,
+        }
+
+    try:
+        network = ipaddress.ip_network(prefix, strict=False)
+    except ValueError:
+        return {
+            "prefix_family": None,
+            "prefix_length": None,
+            "prefix_supernet": None,
+            "is_more_specific": False,
+        }
+
+    supernet_prefix = 16 if network.version == 4 else 32
+    if network.prefixlen > supernet_prefix:
+        prefix_supernet = str(network.supernet(new_prefix=supernet_prefix))
+    else:
+        prefix_supernet = str(network)
+
+    return {
+        "prefix_family": f"ipv{network.version}",
+        "prefix_length": int(network.prefixlen),
+        "prefix_supernet": prefix_supernet,
+        "is_more_specific": network.prefixlen > (24 if network.version == 4 else 48),
+    }
+
+
+async def _lookup_prefix_geography(
+    db: AsyncSession,
+    prefix_values: list[str],
+) -> dict[str, dict[str, Any]]:
+    async def _query_prefix_metadata(
+        *,
+        source: str,
+        family: str,
+        range_start: str,
+        range_end: str,
+    ) -> dict[str, Any] | None:
+        result = await db.execute(
+            text(
+                """
+                SELECT metadata
+                FROM collected_data
+                WHERE source = :source
+                  AND COALESCE(is_current, TRUE) = TRUE
+                  AND metadata->>'family' = :family
+                  AND CAST(metadata->>'range_start' AS inet) <= CAST(:range_start AS inet)
+                  AND CAST(metadata->>'range_end' AS inet) >= CAST(:range_end AS inet)
+                ORDER BY
+                  masklen(CAST(metadata->>'prefix' AS cidr)) DESC NULLS LAST,
+                  id DESC
+                LIMIT 1
+                """
+            ),
+            {
+                "source": source,
+                "family": family,
+                "range_start": range_start,
+                "range_end": range_end,
+            },
+        )
+        row = result.fetchone()
+        if not row:
+            return None
+
+        if isinstance(row, dict):
+            payload = row.get("metadata") or row.get("extra_data")
+        elif hasattr(row, "_mapping"):
+            payload = row._mapping.get("metadata") or row._mapping.get("extra_data")
+        else:
+            payload = row[0]
+
+        return payload if isinstance(payload, dict) else None
+
+    results: dict[str, dict[str, Any]] = {}
+
+    for prefix in prefix_values:
+        try:
+            network = ipaddress.ip_network(prefix, strict=False)
+        except ValueError:
+            continue
+
+        family = f"ipv{network.version}"
+        range_start = str(network.network_address)
+        range_end = str(network.broadcast_address)
+        payload = await _query_prefix_metadata(
+            source="opengeofeed_prefix_geo",
+            family=family,
+            range_start=range_start,
+            range_end=range_end,
+        )
+        selected_source = "opengeofeed"
+        if not payload:
+            payload = await _query_prefix_metadata(
+                source="iptoasn_prefix_geo",
+                family=family,
+                range_start=range_start,
+                range_end=range_end,
+            )
+            selected_source = "iptoasn"
+        if not payload:
+            payload = await _query_prefix_metadata(
+                source="nro_delegated_prefix_geo",
+                family=family,
+                range_start=range_start,
+                range_end=range_end,
+            )
+            selected_source = "nro_delegated"
+        if not payload:
+            continue
+
+        country = normalize_country(payload.get("country") or payload.get("country_code"))
+        prefix_hint = payload.get("prefix") or prefix
+        asn = _safe_int(payload.get("asn"))
+        as_name = payload.get("as_name")
+        city = payload.get("city")
+        centroid = get_country_centroid(country)
+        regions = []
+        if country:
+            regions.append(
+                {
+                    "country": country,
+                    "city": city,
+                    "latitude": centroid.get("latitude") if centroid else None,
+                    "longitude": centroid.get("longitude") if centroid else None,
+                }
+            )
+
+        results[prefix] = {
+            "prefix": prefix_hint,
+            "country": country,
+            "city": city,
+            "asn": asn,
+            "as_name": as_name,
+            "source": payload.get("source_dataset")
+            or (
+                "opengeofeed_public"
+                if selected_source == "opengeofeed"
+                else (
+                    "iptoasn_combined"
+                    if selected_source == "iptoasn"
+                    else "nro_delegated_stats"
+                )
+            ),
+            "confidence": payload.get("confidence")
+            or (
+                "geofeed"
+                if selected_source == "opengeofeed"
+                else (
+                    "country_range"
+                    if selected_source == "iptoasn"
+                    else "registry_allocated"
+                )
+            ),
+            "geography_mode": "prefix_geography",
+            "regions": regions,
+        }
+
+    return results
+
+
+async def lookup_prefix_geography(
+    db: AsyncSession,
+    prefix_values: list[str],
+) -> dict[str, dict[str, Any]]:
+    return await _lookup_prefix_geography(db, prefix_values)
+
+
+async def enrich_bgp_events_for_batch(
+    db: AsyncSession,
+    *,
+    source: str,
+    events: list[dict[str, Any]],
+) -> list[dict[str, Any]]:
+    if not events:
+        return []
+
+    prefixes = {
+        str((event.get("metadata") or {}).get("prefix") or "").strip()
+        for event in events
+        if (event.get("metadata") or {}).get("prefix")
+    }
+    prefix_values = sorted(prefix for prefix in prefixes if prefix)
+    origin_asns = sorted(
+        {
+            asn
+            for event in events
+            for asn in [
+                _safe_int((event.get("metadata") or {}).get("origin_asn")),
+                _safe_int((event.get("metadata") or {}).get("new_origin_asn")),
+            ]
+            if asn is not None
+        }
+    )
+
+    historical_prefix_baseline: dict[str, dict[str, Any]] = {}
+    if prefix_values:
+        previous_result = await db.execute(
+            select(
+                BGPObservation.prefix,
+                BGPObservation.origin_asn,
+                BGPObservation.collector,
+                BGPObservation.collector_geo,
+            ).where(
+                BGPObservation.source == source,
+                BGPObservation.prefix.in_(prefix_values),
+            )
+        )
+        by_prefix: defaultdict[str, list[dict[str, Any]]] = defaultdict(list)
+        for prefix, origin_asn, collector, collector_geo in previous_result.all():
+            if prefix:
+                by_prefix[str(prefix)].append(
+                    {
+                        "origin_asn": origin_asn,
+                        "collector": collector,
+                        "collector_geo": collector_geo or {},
+                    }
+                )
+
+        for prefix, observations in by_prefix.items():
+            unique_origins = sorted(
+                {
+                    observation["origin_asn"]
+                    for observation in observations
+                    if observation["origin_asn"] is not None
+                }
+            )
+            unique_collectors = sorted(
+                {
+                    observation["collector"]
+                    for observation in observations
+                    if observation["collector"]
+                }
+            )
+            historical_prefix_baseline[prefix] = {
+                "historical_origin_asns": unique_origins,
+                "historical_collectors": unique_collectors,
+                "historical_observation_count": len(observations),
+                "historical_regions": _compact_locations(
+                    [
+                        observation["collector_geo"] or {}
+                        for observation in observations
+                        if observation["collector_geo"]
+                    ]
+                ),
+            }
+
+    asn_profiles: dict[int, dict[str, Any]] = {}
+    prefix_geographies = await _lookup_prefix_geography(db, prefix_values) if prefix_values else {}
+    if origin_asns:
+        peeringdb_result = await db.execute(
+            select(CollectedData)
+            .where(CollectedData.source == "peeringdb_network")
+            .where(CollectedData.is_current.is_(True))
+            .where(
+                cast(CollectedData.extra_data["asn"].as_string(), Integer).in_(origin_asns),
+            )
+            .order_by(CollectedData.id.desc())
+        )
+        for record in peeringdb_result.scalars().all():
+            metadata = record.extra_data or {}
+            asn = _safe_int(metadata.get("asn"))
+            if asn is None or asn not in origin_asns:
+                continue
+            current = asn_profiles.get(asn)
+            if current and (current.get("id") or 0) > (record.id or 0):
+                continue
+            asn_profiles[asn] = {
+                "id": record.id,
+                "asn": asn,
+                "name": record.name,
+                "country": metadata.get("country"),
+                "city": metadata.get("city"),
+                "source": "peeringdb_network",
+                "info_type": metadata.get("info_type"),
+                "info_traffic": metadata.get("info_traffic"),
+                "info_ratio": metadata.get("info_ratio"),
+                "ix_count": metadata.get("ix_count"),
+                "url": metadata.get("url"),
+            }
+
+    collector_counts: defaultdict[str, int] = defaultdict(int)
+    for event in events:
+        collector = (event.get("metadata") or {}).get("collector")
+        if collector:
+            collector_counts[str(collector)] += 1
+
+    enriched: list[dict[str, Any]] = []
+    for event in events:
+        metadata = dict(event.get("metadata") or {})
+        prefix = str(metadata.get("prefix") or "").strip()
+        as_path = metadata.get("as_path") or []
+        normalized_as_path = [asn for asn in (_safe_int(item) for item in as_path) if asn is not None]
+        deduped_as_path = _dedupe_as_path(normalized_as_path)
+        collector = str(metadata.get("collector") or "").strip()
+        collector_location = metadata.get("collector_location") or {}
+        baseline = historical_prefix_baseline.get(prefix, {})
+        prefix_geography = prefix_geographies.get(prefix)
+        observed_at = _parse_timestamp(metadata.get("timestamp") or event.get("reference_date"))
+        origin_asn = _safe_int(metadata.get("origin_asn"))
+        new_origin_asn = _safe_int(metadata.get("new_origin_asn"))
+        baseline_regions = baseline.get("historical_regions", [])
+        prefix_scope_regions = _compact_locations([*baseline_regions])
+
+        enrichment = {
+            **extract_bgp_network_fields(prefix),
+            "observed_at": observed_at.isoformat(),
+            "normalized_as_path": normalized_as_path,
+            "deduped_as_path": deduped_as_path,
+            "deduped_as_path_length": len(deduped_as_path),
+            "path_prepending": len(normalized_as_path) > len(deduped_as_path),
+            "collector_region": {
+                "city": collector_location.get("city"),
+                "country": collector_location.get("country"),
+            },
+            "collector_observation_count_in_batch": collector_counts.get(collector, 0),
+            "batch_visibility_collectors": sorted(collector_counts.keys()),
+            "prefix_baseline": baseline,
+            "is_new_origin_for_prefix": (
+                origin_asn is not None
+                and origin_asn
+                not in set(baseline.get("historical_origin_asns", []))
+            ),
+            "rpki_validation": {
+                "status": "unknown",
+                "reason": "no_rpki_roa_dataset_configured",
+            },
+            "origin_asn_profile": asn_profiles.get(origin_asn),
+            "new_origin_asn_profile": asn_profiles.get(new_origin_asn),
+            "prefix_geography": prefix_geography,
+            "prefix_scope": {
+                "countries": sorted(
+                    {
+                        item.get("country")
+                        for item in prefix_scope_regions
+                        if item.get("country")
+                    }
+                ),
+                "cities": sorted(
+                    {
+                        item.get("city")
+                        for item in prefix_scope_regions
+                        if item.get("city")
+                    }
+                ),
+                "regions": prefix_scope_regions,
+            },
+        }
+
+        enriched.append(
+            {
+                **event,
+                "metadata": {
+                    **metadata,
+                    "enrichment": enrichment,
+                },
+            }
+        )
+
+    return enriched

backend/app/services/bgp_incidents.py

@@ -0,0 +1,333 @@
+"""Incident aggregation helpers for BGP anomalies."""
+
+from __future__ import annotations
+
+from datetime import UTC, datetime
+from typing import Any
+
+from sqlalchemy import select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.core.collected_data_fields import get_record_field
+from app.models.bgp_anomaly import BGPAnomaly
+from app.models.bgp_incident import BGPIncident
+from app.models.collected_data import CollectedData
+from app.services.cable_graph import haversine_distance
+
+
+def _severity_rank(value: str | None) -> int:
+    mapping = {"critical": 4, "high": 3, "medium": 2, "low": 1, "info": 0}
+    return mapping.get(str(value or "").lower(), 0)
+
+
+def _pick_severity(values: list[str]) -> str:
+    ordered = sorted(values, key=_severity_rank, reverse=True)
+    return ordered[0] if ordered else "medium"
+
+
+def _collector_regions_from_anomaly(anomaly: BGPAnomaly) -> list[dict]:
+    evidence = anomaly.evidence or {}
+    regions = evidence.get("impacted_regions") or []
+    if regions:
+        return regions
+
+    collected = []
+    for item in evidence.get("events") or []:
+        collector = item.get("collector")
+        location = item.get("collector_location") or {}
+        if collector or location:
+            collected.append(
+                {
+                    "collector": collector,
+                    "country": location.get("country"),
+                    "city": location.get("city"),
+                    "latitude": location.get("latitude"),
+                    "longitude": location.get("longitude"),
+                }
+            )
+    return collected
+
+
+async def _load_current_infrastructure_records(
+    db: AsyncSession,
+) -> tuple[list[CollectedData], list[CollectedData], list[CollectedData]]:
+    result = await db.execute(
+        select(CollectedData)
+        .where(
+            CollectedData.source.in_(
+                (
+                    "arcgis_landing_points",
+                    "arcgis_cable_landing_relation",
+                    "arcgis_cables",
+                )
+            )
+        )
+        .where(CollectedData.is_current.is_(True))
+        .order_by(CollectedData.source.asc(), CollectedData.id.desc())
+    )
+    grouped_records = {
+        "arcgis_landing_points": [],
+        "arcgis_cable_landing_relation": [],
+        "arcgis_cables": [],
+    }
+    for record in result.scalars().all():
+        grouped_records.setdefault(record.source, []).append(record)
+
+    return (
+        grouped_records["arcgis_landing_points"],
+        grouped_records["arcgis_cable_landing_relation"],
+        grouped_records["arcgis_cables"],
+    )
+
+
+async def infer_related_infrastructure(
+    db: AsyncSession,
+    affected_regions: list[dict],
+    *,
+    max_matches: int = 6,
+    max_distance_km: float = 450.0,
+) -> dict[str, list[dict[str, Any]]]:
+    valid_regions = [
+        region
+        for region in affected_regions
+        if isinstance(region, dict)
+        and isinstance(region.get("latitude"), (int, float))
+        and isinstance(region.get("longitude"), (int, float))
+    ]
+    if not valid_regions:
+        return {"related_cables": [], "related_ixps": []}
+
+    landing_records, relation_records, cable_records = await _load_current_infrastructure_records(
+        db,
+    )
+
+    city_to_cable_ids: dict[int, list[int]] = {}
+    for relation in relation_records:
+        metadata = relation.extra_data or {}
+        city_id = metadata.get("city_id")
+        cable_id = metadata.get("cable_id")
+        if city_id is None or cable_id is None:
+            continue
+        city_key = int(city_id)
+        cable_key = int(cable_id)
+        city_to_cable_ids.setdefault(city_key, [])
+        if cable_key not in city_to_cable_ids[city_key]:
+            city_to_cable_ids[city_key].append(cable_key)
+
+    cable_id_to_name: dict[int, str] = {}
+    for cable in cable_records:
+        metadata = cable.extra_data or {}
+        cable_id = metadata.get("cable_id")
+        if cable_id is None or not cable.name:
+            continue
+        cable_id_to_name[int(cable_id)] = cable.name
+
+    matches: list[dict[str, Any]] = []
+    seen_match_keys: set[tuple[Any, ...]] = set()
+
+    for region in valid_regions:
+        region_coords = (float(region["longitude"]), float(region["latitude"]))
+
+        for landing in landing_records:
+            try:
+                latitude = get_record_field(landing, "latitude")
+                longitude = get_record_field(landing, "longitude")
+                landing_lat = float(latitude) if latitude is not None else None
+                landing_lon = float(longitude) if longitude is not None else None
+            except (TypeError, ValueError):
+                landing_lat = None
+                landing_lon = None
+
+            if landing_lat is None or landing_lon is None:
+                continue
+
+            distance_km = haversine_distance(region_coords, (landing_lon, landing_lat))
+            if distance_km > max_distance_km:
+                continue
+
+            landing_meta = landing.extra_data or {}
+            city_id = landing_meta.get("city_id")
+            cable_names = []
+            if city_id is not None:
+                for cable_id in city_to_cable_ids.get(int(city_id), []):
+                    cable_name = cable_id_to_name.get(int(cable_id))
+                    if cable_name and cable_name not in cable_names:
+                        cable_names.append(cable_name)
+
+            match = {
+                "landing_point": landing.name or "Unknown",
+                "city": get_record_field(landing, "city"),
+                "country": get_record_field(landing, "country"),
+                "distance_km": round(distance_km, 1),
+                "collector": region.get("collector"),
+                "cable_names": cable_names,
+            }
+            match_key = (
+                match["landing_point"],
+                match["city"],
+                match["country"],
+            )
+            if match_key in seen_match_keys:
+                continue
+            seen_match_keys.add(match_key)
+            matches.append(match)
+
+    matches.sort(
+        key=lambda item: (
+            item.get("distance_km", 999999),
+            str(item.get("landing_point") or ""),
+        )
+    )
+    matches = matches[:max_matches]
+
+    related_ixps = []
+    seen_ixp_keys: set[tuple[str, str]] = set()
+    for item in matches:
+        city = str(item.get("city") or "").strip()
+        country = str(item.get("country") or "").strip()
+        if not city and not country:
+            continue
+        key = (city, country)
+        if key in seen_ixp_keys:
+            continue
+        seen_ixp_keys.add(key)
+        related_ixps.append(
+            {
+                "name": ", ".join(part for part in [city, country] if part),
+                "type": "regional_exchange_hint",
+            }
+        )
+
+    return {
+        "related_cables": matches,
+        "related_ixps": related_ixps,
+    }
+
+
+async def create_bgp_incidents_for_anomalies(
+    db: AsyncSession,
+    *,
+    source: str,
+    snapshot_id: int | None,
+    task_id: int | None,
+    anomalies: list[BGPAnomaly],
+) -> int:
+    if not anomalies:
+        return 0
+
+    grouped: dict[str, list[BGPAnomaly]] = {}
+    for anomaly in anomalies:
+        incident_key = f"{anomaly.anomaly_type}:{anomaly.prefix or 'unknown'}:{anomaly.new_origin_asn or anomaly.origin_asn or 'na'}"
+        grouped.setdefault(incident_key, []).append(anomaly)
+
+    existing_result = await db.execute(
+        select(BGPIncident).where(BGPIncident.incident_key.in_(sorted(grouped.keys())))
+    )
+    existing_incidents = {
+        incident.incident_key: incident for incident in existing_result.scalars().all()
+    }
+
+    created = 0
+    for incident_key, items in grouped.items():
+        items = sorted(items, key=lambda item: item.created_at or item.started_at or datetime.now(UTC))
+        primary = items[0]
+        prefixes = sorted({item.prefix for item in items if item.prefix})
+        asns = sorted(
+            {
+                asn
+                for item in items
+                for asn in [item.origin_asn, item.new_origin_asn]
+                if asn is not None
+            }
+        )
+        collectors = sorted(
+            {
+                collector
+                for item in items
+                for collector in (item.peer_scope or [])
+                if collector
+            }
+        )
+        regions: list[dict] = []
+        seen_regions: set[tuple] = set()
+        for item in items:
+            for region in _collector_regions_from_anomaly(item):
+                region_key = (
+                    region.get("collector"),
+                    region.get("country"),
+                    region.get("city"),
+                )
+                if region_key in seen_regions:
+                    continue
+                seen_regions.add(region_key)
+                regions.append(region)
+
+        if not collectors:
+            collectors = sorted(
+                {
+                    region.get("collector")
+                    for region in regions
+                    if region.get("collector")
+                }
+            )
+
+        evidence_refs = [item.entity_key for item in items if item.entity_key]
+        severity = _pick_severity([item.severity for item in items])
+        confidence = max((item.confidence or 0.0) for item in items)
+        title = f"{primary.anomaly_type.replace('_', ' ').title()} incident on {primary.prefix or 'unknown prefix'}"
+        summary = (
+            f"{len(items)} anomaly signal(s) grouped into one {primary.anomaly_type} incident, "
+            f"affecting {len(prefixes) or 1} prefix scope(s) across {len(collectors)} collector(s)."
+        )
+        related_infrastructure = await infer_related_infrastructure(db, regions)
+
+        existing = existing_incidents.get(incident_key)
+        if existing is not None:
+            existing.snapshot_id = snapshot_id
+            existing.task_id = task_id
+            existing.source = source
+            existing.incident_type = primary.anomaly_type
+            existing.title = title
+            existing.summary = summary
+            existing.severity = severity
+            existing.status = "active"
+            existing.confidence = confidence
+            existing.started_at = primary.started_at or existing.started_at or datetime.now(UTC)
+            existing.ended_at = None
+            existing.affected_prefixes = prefixes
+            existing.affected_asns = asns
+            existing.affected_collectors = collectors
+            existing.affected_regions = regions
+            existing.related_cables = related_infrastructure["related_cables"]
+            existing.related_ixps = related_infrastructure["related_ixps"]
+            existing.evidence_refs = evidence_refs
+            continue
+
+        db.add(
+            BGPIncident(
+                snapshot_id=snapshot_id,
+                task_id=task_id,
+                source=source,
+                incident_key=incident_key,
+                incident_type=primary.anomaly_type,
+                title=title,
+                summary=summary,
+                severity=severity,
+                status="active",
+                confidence=confidence,
+                started_at=primary.started_at or datetime.now(UTC),
+                affected_prefixes=prefixes,
+                affected_asns=asns,
+                affected_collectors=collectors,
+                affected_regions=regions,
+                related_cables=related_infrastructure["related_cables"],
+                related_ixps=related_infrastructure["related_ixps"],
+                evidence_refs=evidence_refs,
+            )
+        )
+        created += 1
+
+    if created or existing_incidents:
+        await db.commit()
+
+    return created

backend/app/services/collectors/__init__.py

@@ -30,6 +30,12 @@ from app.services.collectors.arcgis_landing import ArcGISLandingPointCollector
 from app.services.collectors.arcgis_relation import ArcGISCableLandingRelationCollector
 from app.services.collectors.spacetrack import SpaceTrackTLECollector
 from app.services.collectors.celestrak import CelesTrakTLECollector
+from app.services.collectors.ris_live import RISLiveCollector
+from app.services.collectors.bgpstream import BGPStreamBackfillCollector
+from app.services.collectors.iptoasn import IPtoASNPrefixGeoCollector
+from app.services.collectors.opengeofeed import OpenGeoFeedPrefixGeoCollector
+from app.services.collectors.nro_delegated import NRODelegatedPrefixGeoCollector
+from app.services.collectors.news_live_streams import NewsLiveStreamsCollector
 
 collector_registry.register(TOP500Collector())
 collector_registry.register(EpochAIGPUCollector())
@@ -51,3 +57,9 @@ collector_registry.register(ArcGISLandingPointCollector())
 collector_registry.register(ArcGISCableLandingRelationCollector())
 collector_registry.register(SpaceTrackTLECollector())
 collector_registry.register(CelesTrakTLECollector())
+collector_registry.register(RISLiveCollector())
+collector_registry.register(BGPStreamBackfillCollector())
+collector_registry.register(IPtoASNPrefixGeoCollector())
+collector_registry.register(OpenGeoFeedPrefixGeoCollector())
+collector_registry.register(NRODelegatedPrefixGeoCollector())
+collector_registry.register(NewsLiveStreamsCollector())

backend/app/services/collectors/arcgis_cables.py

@@ -5,7 +5,7 @@ Collects submarine cable data from ArcGIS GeoJSON API.
 
 import json
 from typing import Dict, Any, List
-from datetime import datetime
+from datetime import UTC, datetime
 import httpx
 
 from app.services.collectors.base import BaseCollector
@@ -84,7 +84,7 @@ class ArcGISCableCollector(BaseCollector):
                         "color": props.get("color"),
                         "route_coordinates": route_coordinates,
                     },
-                    "reference_date": datetime.utcnow().strftime("%Y-%m-%d"),
+                    "reference_date": datetime.now(UTC).strftime("%Y-%m-%d"),
                 }
                 result.append(entry)
             except (ValueError, TypeError, KeyError):

backend/app/services/collectors/arcgis_landing.py

@@ -1,5 +1,5 @@
 from typing import Dict, Any, List
-from datetime import datetime
+from datetime import UTC, datetime
 import httpx
 
 from app.services.collectors.base import BaseCollector
@@ -67,7 +67,7 @@ class ArcGISLandingPointCollector(BaseCollector):
                         "status": props.get("status"),
                         "landing_point_id": props.get("landing_point_id"),
                     },
-                    "reference_date": datetime.utcnow().strftime("%Y-%m-%d"),
+                    "reference_date": datetime.now(UTC).strftime("%Y-%m-%d"),
                 }
                 result.append(entry)
             except (ValueError, TypeError, KeyError):

backend/app/services/collectors/arcgis_relation.py

@@ -1,10 +1,11 @@
-from typing import Dict, Any, List
-from datetime import datetime
+import asyncio
+from datetime import UTC, datetime
+from typing import Any, Dict, List, Optional
+
 import httpx
 
-from app.services.collectors.base import BaseCollector
 from app.core.data_sources import get_data_sources_config
-
+from app.services.collectors.base import BaseCollector
 
 
 class ArcGISCableLandingRelationCollector(BaseCollector):
@@ -18,47 +19,131 @@ class ArcGISCableLandingRelationCollector(BaseCollector):
     def base_url(self) -> str:
         if self._resolved_url:
             return self._resolved_url
-        from app.core.data_sources import get_data_sources_config
-
         config = get_data_sources_config()
         return config.get_yaml_url("arcgis_cable_landing_relation")
 
+    def _layer_url(self, layer_id: int) -> str:
+        if "/FeatureServer/" not in self.base_url:
+            return self.base_url
+        prefix = self.base_url.split("/FeatureServer/")[0]
+        return f"{prefix}/FeatureServer/{layer_id}/query"
+
+    async def _fetch_layer_attributes(
+        self, client: httpx.AsyncClient, layer_id: int
+    ) -> List[Dict[str, Any]]:
+        response = await client.get(
+            self._layer_url(layer_id),
+            params={
+                "where": "1=1",
+                "outFields": "*",
+                "returnGeometry": "false",
+                "f": "json",
+            },
+        )
+        response.raise_for_status()
+        data = response.json()
+        return [feature.get("attributes", {}) for feature in data.get("features", [])]
+
+    async def _fetch_relation_features(self, client: httpx.AsyncClient) -> List[Dict[str, Any]]:
+        response = await client.get(
+            self.base_url,
+            params={
+                "where": "1=1",
+                "outFields": "*",
+                "returnGeometry": "true",
+                "f": "geojson",
+            },
+        )
+        response.raise_for_status()
+        data = response.json()
+        return data.get("features", [])
+
     async def fetch(self) -> List[Dict[str, Any]]:
-        params = {"where": "1=1", "outFields": "*", "returnGeometry": "true", "f": "geojson"}
-
         async with httpx.AsyncClient(timeout=60.0) as client:
-            response = await client.get(self.base_url, params=params)
-            response.raise_for_status()
-            return self.parse_response(response.json())
+            relation_features, landing_rows, cable_rows = await asyncio.gather(
+                self._fetch_relation_features(client),
+                self._fetch_layer_attributes(client, 1),
+                self._fetch_layer_attributes(client, 2),
+            )
+        return self.parse_response(relation_features, landing_rows, cable_rows)
 
-    def parse_response(self, data: Dict[str, Any]) -> List[Dict[str, Any]]:
-        result = []
+    def _build_landing_lookup(self, landing_rows: List[Dict[str, Any]]) -> Dict[int, Dict[str, Any]]:
+        lookup: Dict[int, Dict[str, Any]] = {}
+        for row in landing_rows:
+            city_id = row.get("city_id")
+            if city_id is None:
+                continue
+            lookup[int(city_id)] = {
+                "landing_point_id": row.get("landing_point_id") or city_id,
+                "landing_point_name": row.get("Name") or row.get("name") or "",
+                "facility": row.get("facility") or "",
+                "status": row.get("status") or "",
+                "country": row.get("country") or "",
+            }
+        return lookup
 
-        features = data.get("features", [])
-        for feature in features:
+    def _build_cable_lookup(self, cable_rows: List[Dict[str, Any]]) -> Dict[int, Dict[str, Any]]:
+        lookup: Dict[int, Dict[str, Any]] = {}
+        for row in cable_rows:
+            cable_id = row.get("cable_id")
+            if cable_id is None:
+                continue
+            lookup[int(cable_id)] = {
+                "cable_name": row.get("Name") or "",
+                "status": row.get("status") or "active",
+            }
+        return lookup
+
+    def parse_response(
+        self,
+        relation_features: List[Dict[str, Any]],
+        landing_rows: List[Dict[str, Any]],
+        cable_rows: List[Dict[str, Any]],
+    ) -> List[Dict[str, Any]]:
+        result: List[Dict[str, Any]] = []
+        landing_lookup = self._build_landing_lookup(landing_rows)
+        cable_lookup = self._build_cable_lookup(cable_rows)
+
+        for feature in relation_features:
             props = feature.get("properties", {})
 
             try:
+                city_id = props.get("city_id")
+                cable_id = props.get("cable_id")
+                landing_info = landing_lookup.get(int(city_id), {}) if city_id is not None else {}
+                cable_info = cable_lookup.get(int(cable_id), {}) if cable_id is not None else {}
+
+                cable_name = cable_info.get("cable_name") or props.get("cable_name") or "Unknown"
+                landing_point_name = (
+                    landing_info.get("landing_point_name")
+                    or props.get("landing_point_name")
+                    or "Unknown"
+                )
+                facility = landing_info.get("facility") or props.get("facility") or "-"
+                status = cable_info.get("status") or landing_info.get("status") or props.get("status") or "-"
+                country = landing_info.get("country") or props.get("country") or ""
+                landing_point_id = landing_info.get("landing_point_id") or props.get("landing_point_id") or city_id
+
                 entry = {
                     "source_id": f"arcgis_relation_{props.get('OBJECTID', props.get('id', ''))}",
-                    "name": f"{props.get('cable_name', 'Unknown')} - {props.get('landing_point_name', 'Unknown')}",
-                    "country": props.get("country", ""),
-                    "city": props.get("landing_point_name", ""),
+                    "name": f"{cable_name} - {landing_point_name}",
+                    "country": country,
+                    "city": landing_point_name,
                     "latitude": str(props.get("latitude", "")) if props.get("latitude") else "",
                     "longitude": str(props.get("longitude", "")) if props.get("longitude") else "",
                     "value": "",
                     "unit": "",
                     "metadata": {
                         "objectid": props.get("OBJECTID"),
-                        "city_id": props.get("city_id"),
-                        "cable_id": props.get("cable_id"),
-                        "cable_name": props.get("cable_name"),
-                        "landing_point_id": props.get("landing_point_id"),
-                        "landing_point_name": props.get("landing_point_name"),
-                        "facility": props.get("facility"),
-                        "status": props.get("status"),
+                        "city_id": city_id,
+                        "cable_id": cable_id,
+                        "cable_name": cable_name,
+                        "landing_point_id": landing_point_id,
+                        "landing_point_name": landing_point_name,
+                        "facility": facility,
+                        "status": status,
                     },
-                    "reference_date": datetime.utcnow().strftime("%Y-%m-%d"),
+                    "reference_date": datetime.now(UTC).strftime("%Y-%m-%d"),
                 }
                 result.append(entry)
             except (ValueError, TypeError, KeyError):

backend/app/services/collectors/base.py

@@ -1,13 +1,18 @@
 """Base collector class for all data sources"""
 
+import asyncio
 from abc import ABC, abstractmethod
 from typing import Dict, List, Any, Optional
-from datetime import datetime
+from datetime import UTC, datetime
 import httpx
-from sqlalchemy import text
+from sqlalchemy import select, text
 from sqlalchemy.ext.asyncio import AsyncSession
 
+from app.core.collected_data_fields import build_dynamic_metadata, get_record_field
 from app.core.config import settings
+from app.core.countries import normalize_country
+from app.core.time import to_iso8601_utc
+from app.core.websocket.broadcaster import broadcaster
 
 
 class BaseCollector(ABC):
@@ -18,12 +23,14 @@ class BaseCollector(ABC):
     module: str = "L1"
     frequency_hours: int = 4
     data_type: str = "generic"
+    fail_on_empty: bool = False
 
     def __init__(self):
         self._current_task = None
         self._db_session = None
         self._datasource_id = 1
         self._resolved_url: Optional[str] = None
+        self._last_broadcast_progress: Optional[int] = None
 
     async def resolve_url(self, db: AsyncSession) -> None:
         from app.core.data_sources import get_data_sources_config
@@ -31,13 +38,53 @@ class BaseCollector(ABC):
         config = get_data_sources_config()
         self._resolved_url = await config.get_url(self.name, db)
 
-    def update_progress(self, records_processed: int):
+    async def _publish_task_update(self, force: bool = False):
+        if not self._current_task:
+            return
+
+        progress = float(self._current_task.progress or 0.0)
+        rounded_progress = int(round(progress))
+        if not force and self._last_broadcast_progress == rounded_progress:
+            return
+
+        await broadcaster.broadcast_datasource_task_update(
+            {
+                "datasource_id": getattr(self, "_datasource_id", None),
+                "collector_name": self.name,
+                "task_id": self._current_task.id,
+                "status": self._current_task.status,
+                "phase": self._current_task.phase,
+                "progress": progress,
+                "records_processed": self._current_task.records_processed,
+                "total_records": self._current_task.total_records,
+                "started_at": to_iso8601_utc(self._current_task.started_at),
+                "completed_at": to_iso8601_utc(self._current_task.completed_at),
+                "error_message": self._current_task.error_message,
+            }
+        )
+        self._last_broadcast_progress = rounded_progress
+
+    async def update_progress(self, records_processed: int, *, commit: bool = False, force: bool = False):
         """Update task progress - call this during data processing"""
-        if self._current_task and self._db_session and self._current_task.total_records > 0:
+        if self._current_task and self._db_session:
             self._current_task.records_processed = records_processed
-            self._current_task.progress = (
-                records_processed / self._current_task.total_records
-            ) * 100
+            if self._current_task.total_records and self._current_task.total_records > 0:
+                self._current_task.progress = (
+                    records_processed / self._current_task.total_records
+                ) * 100
+            else:
+                self._current_task.progress = 0.0
+
+            if commit:
+                await self._db_session.commit()
+
+            await self._publish_task_update(force=force)
+
+    async def set_phase(self, phase: str):
+        if self._current_task and self._db_session:
+            self._current_task.phase = phase
+            await self._db_session.commit()
+            await self._publish_task_update(force=True)
 
     @abstractmethod
     async def fetch(self) -> List[Dict[str, Any]]:
@@ -48,14 +95,140 @@ class BaseCollector(ABC):
         """Transform raw data to internal format (default: pass through)"""
         return raw_data
 
+    def _parse_reference_date(self, value: Any) -> Optional[datetime]:
+        if not value:
+            return None
+        if isinstance(value, datetime):
+            return value
+        if isinstance(value, str):
+            return datetime.fromisoformat(value.replace("Z", "+00:00"))
+        return None
+
+    def _build_comparable_payload(self, record: Any) -> Dict[str, Any]:
+        return {
+            "name": getattr(record, "name", None),
+            "title": getattr(record, "title", None),
+            "description": getattr(record, "description", None),
+            "country": get_record_field(record, "country"),
+            "city": get_record_field(record, "city"),
+            "latitude": get_record_field(record, "latitude"),
+            "longitude": get_record_field(record, "longitude"),
+            "value": get_record_field(record, "value"),
+            "unit": get_record_field(record, "unit"),
+            "metadata": getattr(record, "extra_data", None) or {},
+            "reference_date": (
+                getattr(record, "reference_date", None).isoformat()
+                if getattr(record, "reference_date", None)
+                else None
+            ),
+        }
+
+    async def _create_snapshot(
+        self,
+        db: AsyncSession,
+        task_id: int,
+        data: List[Dict[str, Any]],
+        started_at: datetime,
+    ) -> int:
+        from app.models.data_snapshot import DataSnapshot
+
+        reference_dates = [
+            parsed
+            for parsed in (self._parse_reference_date(item.get("reference_date")) for item in data)
+            if parsed is not None
+        ]
+        reference_date = max(reference_dates) if reference_dates else None
+
+        result = await db.execute(
+            select(DataSnapshot)
+            .where(DataSnapshot.source == self.name, DataSnapshot.is_current == True)
+            .order_by(DataSnapshot.completed_at.desc().nullslast(), DataSnapshot.id.desc())
+            .limit(1)
+        )
+        previous_snapshot = result.scalar_one_or_none()
+
+        snapshot = DataSnapshot(
+            datasource_id=getattr(self, "_datasource_id", 1),
+            task_id=task_id,
+            source=self.name,
+            snapshot_key=f"{self.name}:{task_id}",
+            reference_date=reference_date,
+            started_at=started_at,
+            status="running",
+            is_current=True,
+            parent_snapshot_id=previous_snapshot.id if previous_snapshot else None,
+            summary={},
+        )
+        db.add(snapshot)
+
+        if previous_snapshot:
+            previous_snapshot.is_current = False
+
+        await db.commit()
+        return snapshot.id
+
+    async def _rollback_incomplete_run(
+        self,
+        db: AsyncSession,
+        *,
+        task_id: int,
+        snapshot_id: Optional[int],
+        reason: str,
+    ) -> None:
+        from app.models.collected_data import CollectedData
+        from app.models.data_snapshot import DataSnapshot
+
+        await db.execute(CollectedData.__table__.delete().where(CollectedData.task_id == task_id))
+
+        parent_snapshot_id: Optional[int] = None
+        if snapshot_id is not None:
+            snapshot = await db.get(DataSnapshot, snapshot_id)
+            if snapshot:
+                parent_snapshot_id = snapshot.parent_snapshot_id
+                snapshot.status = "cancelled"
+                snapshot.is_current = False
+                snapshot.completed_at = datetime.now(UTC)
+                summary = dict(snapshot.summary or {})
+                summary["rollback"] = True
+                summary["rollback_reason"] = reason
+                snapshot.summary = summary
+
+        await db.execute(
+            text(
+                """
+                UPDATE collected_data
+                SET is_current = FALSE
+                WHERE source = :source
+                """
+            ),
+            {"source": self.name},
+        )
+
+        if parent_snapshot_id is not None:
+            parent_snapshot = await db.get(DataSnapshot, parent_snapshot_id)
+            if parent_snapshot:
+                parent_snapshot.is_current = True
+
+            await db.execute(
+                text(
+                    """
+                    UPDATE collected_data
+                    SET is_current = TRUE
+                    WHERE snapshot_id = :snapshot_id
+                    """
+                ),
+                {"snapshot_id": parent_snapshot_id},
+            )
+
     async def run(self, db: AsyncSession) -> Dict[str, Any]:
         """Full pipeline: fetch -> transform -> save"""
         from app.services.collectors.registry import collector_registry
         from app.models.task import CollectionTask
-        from app.models.collected_data import CollectedData
+        from app.models.data_snapshot import DataSnapshot
 
-        start_time = datetime.utcnow()
+        start_time = datetime.now(UTC)
         datasource_id = getattr(self, "_datasource_id", 1)
+        snapshot_id: Optional[int] = None
 
         if not collector_registry.is_active(self.name):
             return {"status": "skipped", "reason": "Collector is disabled"}
@@ -63,6 +236,7 @@ class BaseCollector(ABC):
         task = CollectionTask(
             datasource_id=datasource_id,
             status="running",
+            phase="queued",
             started_at=start_time,
         )
         db.add(task)
@@ -71,92 +245,241 @@ class BaseCollector(ABC):
 
         self._current_task = task
         self._db_session = db
+        self._last_broadcast_progress = None
 
         await self.resolve_url(db)
+        await self._publish_task_update(force=True)
 
         try:
+            await self.set_phase("fetching")
             raw_data = await self.fetch()
             task.total_records = len(raw_data)
             await db.commit()
+            await self._publish_task_update(force=True)
 
+            if self.fail_on_empty and not raw_data:
+                raise RuntimeError(f"Collector {self.name} returned no data")
+
+            await self.set_phase("transforming")
             data = self.transform(raw_data)
+            snapshot_id = await self._create_snapshot(db, task_id, data, start_time)
 
-            records_count = await self._save_data(db, data)
+            await self.set_phase("saving")
+            records_count = await self._save_data(db, data, task_id=task_id, snapshot_id=snapshot_id)
 
             task.status = "success"
+            task.phase = "completed"
             task.records_processed = records_count
             task.progress = 100.0
-            task.completed_at = datetime.utcnow()
+            task.completed_at = datetime.now(UTC)
             await db.commit()
+            await self._publish_task_update(force=True)
 
             return {
                 "status": "success",
                 "task_id": task_id,
                 "records_processed": records_count,
-                "execution_time_seconds": (datetime.utcnow() - start_time).total_seconds(),
+                "execution_time_seconds": (datetime.now(UTC) - start_time).total_seconds(),
             }
-        except Exception as e:
-            task.status = "failed"
-            task.error_message = str(e)
-            task.completed_at = datetime.utcnow()
+        except asyncio.CancelledError:
+            await db.rollback()
+            task.status = "cancelled"
+            task.phase = "cancelled"
+            task.error_message = "Collection cancelled by operator and rolled back"
+            task.completed_at = datetime.now(UTC)
+            if snapshot_id is not None:
+                await self._rollback_incomplete_run(
+                    db,
+                    task_id=task_id,
+                    snapshot_id=snapshot_id,
+                    reason="cancelled_by_operator",
+                )
             await db.commit()
+            await self._publish_task_update(force=True)
+            raise
+        except Exception as e:
+            await db.rollback()
+            task.status = "failed"
+            task.phase = "failed"
+            task.error_message = str(e)
+            task.completed_at = datetime.now(UTC)
+            if snapshot_id is not None:
+                snapshot = await db.get(DataSnapshot, snapshot_id)
+                if snapshot:
+                    snapshot.status = "failed"
+                    snapshot.completed_at = datetime.now(UTC)
+                    snapshot.summary = {"error": str(e)}
+            await db.commit()
+            await self._publish_task_update(force=True)
 
             return {
                 "status": "failed",
                 "task_id": task_id,
                 "error": str(e),
-                "execution_time_seconds": (datetime.utcnow() - start_time).total_seconds(),
+                "execution_time_seconds": (datetime.now(UTC) - start_time).total_seconds(),
             }
 
-    async def _save_data(self, db: AsyncSession, data: List[Dict[str, Any]]) -> int:
+    async def _save_data(
+        self,
+        db: AsyncSession,
+        data: List[Dict[str, Any]],
+        task_id: Optional[int] = None,
+        snapshot_id: Optional[int] = None,
+    ) -> int:
         """Save transformed data to database"""
         from app.models.collected_data import CollectedData
+        from app.models.data_snapshot import DataSnapshot
 
         if not data:
+            if snapshot_id is not None:
+                snapshot = await db.get(DataSnapshot, snapshot_id)
+                if snapshot:
+                    snapshot.record_count = 0
+                    snapshot.summary = {"created": 0, "updated": 0, "unchanged": 0}
+                    snapshot.status = "success"
+                    snapshot.completed_at = datetime.now(UTC)
+                    await db.commit()
             return 0
 
-        collected_at = datetime.utcnow()
+        collected_at = datetime.now(UTC)
         records_added = 0
+        created_count = 0
+        updated_count = 0
+        unchanged_count = 0
+        seen_entity_keys: set[str] = set()
+        progress_commit_interval = 1000
+
+        previous_current_result = await db.execute(
+            select(CollectedData)
+            .where(
+                CollectedData.source == self.name,
+                CollectedData.is_current == True,
+            )
+            .order_by(CollectedData.entity_key.asc(), CollectedData.collected_at.desc().nullslast(), CollectedData.id.desc())
+        )
+        previous_current_records = previous_current_result.scalars().all()
+        previous_current_keys = {record.entity_key for record in previous_current_records if record.entity_key}
+        previous_current_map: dict[str, CollectedData] = {}
+        stale_previous_records: list[CollectedData] = []
+
+        for existing_record in previous_current_records:
+            entity_key = existing_record.entity_key
+            if not entity_key:
+                continue
+            if entity_key not in previous_current_map:
+                previous_current_map[entity_key] = existing_record
+                continue
+            stale_previous_records.append(existing_record)
+
+        for stale_record in stale_previous_records:
+            stale_record.is_current = False
 
         for i, item in enumerate(data):
-            print(
-                f"DEBUG: Saving item {i}: name={item.get('name')}, metadata={item.get('metadata', 'NOT FOUND')}"
+            raw_metadata = item.get("metadata", {})
+            extra_data = build_dynamic_metadata(
+                raw_metadata,
+                country=item.get("country"),
+                city=item.get("city"),
+                latitude=item.get("latitude"),
+                longitude=item.get("longitude"),
+                value=item.get("value"),
+                unit=item.get("unit"),
             )
+            normalized_country = normalize_country(item.get("country"))
+            if normalized_country is not None:
+                extra_data["country"] = normalized_country
+
+            if item.get("country") and normalized_country != item.get("country"):
+                extra_data["raw_country"] = item.get("country")
+                if normalized_country is None:
+                    extra_data["country_validation"] = "invalid"
+
+            source_id = item.get("source_id") or item.get("id")
+            reference_date = (
+                self._parse_reference_date(item.get("reference_date"))
+            )
+            source_id_str = str(source_id) if source_id is not None else None
+            entity_key = f"{self.name}:{source_id_str}" if source_id_str else f"{self.name}:{i}"
+            previous_record = None
+
+            if entity_key and entity_key not in seen_entity_keys:
+                previous_record = previous_current_map.get(entity_key)
+                if previous_record is not None:
+                    previous_record.is_current = False
+
             record = CollectedData(
+                snapshot_id=snapshot_id,
+                task_id=task_id,
                 source=self.name,
-                source_id=item.get("source_id") or item.get("id"),
+                source_id=source_id_str,
+                entity_key=entity_key,
                 data_type=self.data_type,
                 name=item.get("name"),
                 title=item.get("title"),
                 description=item.get("description"),
-                country=item.get("country"),
-                city=item.get("city"),
-                latitude=str(item.get("latitude", ""))
-                if item.get("latitude") is not None
-                else None,
-                longitude=str(item.get("longitude", ""))
-                if item.get("longitude") is not None
-                else None,
-                value=item.get("value"),
-                unit=item.get("unit"),
-                extra_data=item.get("metadata", {}),
+                extra_data=extra_data,
                 collected_at=collected_at,
-                reference_date=datetime.fromisoformat(
-                    item.get("reference_date").replace("Z", "+00:00")
-                )
-                if item.get("reference_date")
-                else None,
+                reference_date=reference_date,
                 is_valid=1,
+                is_current=True,
+                previous_record_id=previous_record.id if previous_record else None,
+                deleted_at=None,
             )
+
+            if previous_record is None:
+                record.change_type = "created"
+                record.change_summary = {}
+                created_count += 1
+            else:
+                previous_payload = self._build_comparable_payload(previous_record)
+                current_payload = self._build_comparable_payload(record)
+                if current_payload == previous_payload:
+                    record.change_type = "unchanged"
+                    record.change_summary = {}
+                    unchanged_count += 1
+                else:
+                    changed_fields = [
+                        key for key in current_payload.keys() if current_payload[key] != previous_payload.get(key)
+                    ]
+                    record.change_type = "updated"
+                    record.change_summary = {"changed_fields": changed_fields}
+                    updated_count += 1
+
             db.add(record)
+            seen_entity_keys.add(entity_key)
             records_added += 1
 
-            if i % 100 == 0:
-                self.update_progress(i + 1)
-                await db.commit()
+            if (i + 1) % progress_commit_interval == 0:
+                await self.update_progress(i + 1, commit=True)
+
+        if snapshot_id is not None:
+            deleted_keys = previous_current_keys - seen_entity_keys
+            await db.execute(
+                text(
+                    """
+                    UPDATE collected_data
+                    SET is_current = FALSE
+                    WHERE source = :source
+                      AND snapshot_id IS DISTINCT FROM :snapshot_id
+                      AND COALESCE(is_current, TRUE) = TRUE
+                    """
+                ),
+                {"source": self.name, "snapshot_id": snapshot_id},
+            )
+            snapshot = await db.get(DataSnapshot, snapshot_id)
+            if snapshot:
+                snapshot.record_count = records_added
+                snapshot.status = "success"
+                snapshot.completed_at = datetime.now(UTC)
+                snapshot.summary = {
+                    "created": created_count,
+                    "updated": updated_count,
+                    "unchanged": unchanged_count,
+                    "deleted": len(deleted_keys),
+                }
 
         await db.commit()
-        self.update_progress(len(data))
+        await self.update_progress(len(data), force=True)
         return records_added
 
     async def save(self, db: AsyncSession, data: List[Dict[str, Any]]) -> int:
@@ -203,8 +526,8 @@ async def log_task(
         status=status,
         records_processed=records_processed,
         error_message=error_message,
-        started_at=datetime.utcnow(),
-        completed_at=datetime.utcnow(),
+        started_at=datetime.now(UTC),
+        completed_at=datetime.now(UTC),
     )
     db.add(task)
     await db.commit()

backend/app/services/collectors/bgp_common.py

@@ -0,0 +1,350 @@
+"""Shared helpers for BGP collectors."""
+
+from __future__ import annotations
+
+import hashlib
+from collections import defaultdict
+from datetime import UTC, datetime
+from typing import Any
+
+from sqlalchemy import select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.models.bgp_anomaly import BGPAnomaly
+from app.models.bgp_observation import BGPObservation
+from app.models.collected_data import CollectedData
+from app.services.bgp_incidents import create_bgp_incidents_for_anomalies
+from app.services.bgp_detectors import (
+    detect_mass_withdrawal_anomalies,
+    detect_more_specific_burst_anomalies,
+    detect_origin_change_anomalies,
+    detect_path_flap_anomalies,
+    detect_route_leak_anomalies,
+)
+from app.services.bgp_enrichment import enrich_bgp_events_for_batch, extract_bgp_network_fields
+
+
+RIPE_RIS_COLLECTOR_COORDS: dict[str, dict[str, Any]] = {
+    "rrc00": {"city": "Amsterdam", "country": "Netherlands", "latitude": 52.3676, "longitude": 4.9041},
+    "rrc01": {"city": "London", "country": "United Kingdom", "latitude": 51.5072, "longitude": -0.1276},
+    "rrc03": {"city": "Amsterdam", "country": "Netherlands", "latitude": 52.3676, "longitude": 4.9041},
+    "rrc04": {"city": "Geneva", "country": "Switzerland", "latitude": 46.2044, "longitude": 6.1432},
+    "rrc05": {"city": "Vienna", "country": "Austria", "latitude": 48.2082, "longitude": 16.3738},
+    "rrc06": {"city": "Otemachi", "country": "Japan", "latitude": 35.686, "longitude": 139.7671},
+    "rrc07": {"city": "Stockholm", "country": "Sweden", "latitude": 59.3293, "longitude": 18.0686},
+    "rrc10": {"city": "Milan", "country": "Italy", "latitude": 45.4642, "longitude": 9.19},
+    "rrc11": {"city": "New York", "country": "United States", "latitude": 40.7128, "longitude": -74.006},
+    "rrc12": {"city": "Frankfurt", "country": "Germany", "latitude": 50.1109, "longitude": 8.6821},
+    "rrc13": {"city": "Moscow", "country": "Russia", "latitude": 55.7558, "longitude": 37.6173},
+    "rrc14": {"city": "Palo Alto", "country": "United States", "latitude": 37.4419, "longitude": -122.143},
+    "rrc15": {"city": "Sao Paulo", "country": "Brazil", "latitude": -23.5558, "longitude": -46.6396},
+    "rrc16": {"city": "Miami", "country": "United States", "latitude": 25.7617, "longitude": -80.1918},
+    "rrc18": {"city": "Barcelona", "country": "Spain", "latitude": 41.3874, "longitude": 2.1686},
+    "rrc19": {"city": "Johannesburg", "country": "South Africa", "latitude": -26.2041, "longitude": 28.0473},
+    "rrc20": {"city": "Zurich", "country": "Switzerland", "latitude": 47.3769, "longitude": 8.5417},
+    "rrc21": {"city": "Paris", "country": "France", "latitude": 48.8566, "longitude": 2.3522},
+    "rrc22": {"city": "Bucharest", "country": "Romania", "latitude": 44.4268, "longitude": 26.1025},
+    "rrc23": {"city": "Singapore", "country": "Singapore", "latitude": 1.3521, "longitude": 103.8198},
+    "rrc24": {"city": "Montevideo", "country": "Uruguay", "latitude": -34.9011, "longitude": -56.1645},
+    "rrc25": {"city": "Amsterdam", "country": "Netherlands", "latitude": 52.3676, "longitude": 4.9041},
+    "rrc26": {"city": "Dubai", "country": "United Arab Emirates", "latitude": 25.2048, "longitude": 55.2708},
+}
+
+
+def _safe_int(value: Any) -> int | None:
+    try:
+        if value in (None, ""):
+            return None
+        return int(value)
+    except (TypeError, ValueError):
+        return None
+
+
+def _parse_timestamp(value: Any) -> datetime:
+    if isinstance(value, datetime):
+        return value.astimezone(UTC) if value.tzinfo else value.replace(tzinfo=UTC)
+
+    if isinstance(value, (int, float)):
+        return datetime.fromtimestamp(value, tz=UTC)
+
+    if isinstance(value, str) and value:
+        normalized = value.replace("Z", "+00:00")
+        parsed = datetime.fromisoformat(normalized)
+        return parsed.astimezone(UTC) if parsed.tzinfo else parsed.replace(tzinfo=UTC)
+
+    return datetime.now(UTC)
+
+
+def _normalize_as_path(raw_path: Any) -> list[int]:
+    if raw_path in (None, ""):
+        return []
+    if isinstance(raw_path, list):
+        return [asn for asn in (_safe_int(item) for item in raw_path) if asn is not None]
+    if isinstance(raw_path, str):
+        parts = raw_path.replace("{", "").replace("}", "").split()
+        return [asn for asn in (_safe_int(item) for item in parts) if asn is not None]
+    return []
+
+
+def normalize_bgp_event(payload: dict[str, Any], *, project: str) -> dict[str, Any]:
+    raw_message = payload.get("raw_message", payload)
+    raw_path = (
+        payload.get("path")
+        or payload.get("as_path")
+        or payload.get("attrs", {}).get("path")
+        or payload.get("attrs", {}).get("as_path")
+        or []
+    )
+    as_path = _normalize_as_path(raw_path)
+
+    raw_type = str(payload.get("event_type") or payload.get("type") or payload.get("msg_type") or "").lower()
+    if raw_type in {"a", "announce", "announcement"}:
+        event_type = "announcement"
+    elif raw_type in {"w", "withdraw", "withdrawal"}:
+        event_type = "withdrawal"
+    elif raw_type in {"r", "rib"}:
+        event_type = "rib"
+    else:
+        event_type = raw_type or "announcement"
+
+    prefix = str(payload.get("prefix") or payload.get("prefixes") or payload.get("target_prefix") or "").strip()
+    if prefix.startswith("[") and prefix.endswith("]"):
+        prefix = prefix[1:-1]
+
+    timestamp = _parse_timestamp(payload.get("timestamp") or payload.get("time") or payload.get("ts"))
+    collector = str(payload.get("collector") or payload.get("host") or payload.get("router") or "unknown")
+    peer_asn = _safe_int(payload.get("peer_asn") or payload.get("peer"))
+    peer_ip = payload.get("peer_ip") or payload.get("peer_address")
+    if peer_ip in (None, ""):
+        peer_candidate = payload.get("peer")
+        peer_ip = str(peer_candidate) if isinstance(peer_candidate, str) and ":" in peer_candidate else peer_candidate
+    origin_asn = _safe_int(payload.get("origin_asn")) or (as_path[-1] if as_path else None)
+    source_material = "|".join(
+        [
+            collector,
+            str(peer_asn or ""),
+            prefix,
+            event_type,
+            timestamp.isoformat(),
+            ",".join(str(asn) for asn in as_path),
+        ]
+    )
+    source_id = hashlib.sha1(source_material.encode("utf-8")).hexdigest()[:24]
+
+    collector_location = RIPE_RIS_COLLECTOR_COORDS.get(collector, {})
+    network_fields = extract_bgp_network_fields(prefix)
+    metadata = {
+        "project": project,
+        "collector": collector,
+        "peer_asn": peer_asn,
+        "peer_ip": peer_ip,
+        "event_type": event_type,
+        "prefix": prefix,
+        "origin_asn": origin_asn,
+        "as_path": as_path,
+        "communities": payload.get("communities")
+        or payload.get("community")
+        or payload.get("attrs", {}).get("communities")
+        or [],
+        "next_hop": payload.get("next_hop") or payload.get("attrs", {}).get("next_hop"),
+        "med": payload.get("med") or payload.get("attrs", {}).get("med"),
+        "local_pref": payload.get("local_pref") or payload.get("attrs", {}).get("local_pref"),
+        "timestamp": timestamp.isoformat(),
+        "as_path_length": len(as_path),
+        "visibility_weight": 1,
+        "collector_location": collector_location,
+        "raw_message": raw_message,
+        "prefix_family": network_fields.get("prefix_family"),
+        "prefix_length": network_fields.get("prefix_length"),
+        "prefix_supernet": network_fields.get("prefix_supernet"),
+        "is_more_specific": network_fields.get("is_more_specific", False),
+    }
+
+    return {
+        "source_id": source_id,
+        "name": prefix or f"{collector}:{event_type}",
+        "title": f"{event_type} {prefix}".strip(),
+        "description": f"{collector} observed {event_type} for {prefix}".strip(),
+        "reference_date": timestamp.isoformat(),
+        "country": collector_location.get("country"),
+        "city": collector_location.get("city"),
+        "latitude": collector_location.get("latitude"),
+        "longitude": collector_location.get("longitude"),
+        "metadata": metadata,
+    }
+
+
+async def save_bgp_observations_for_batch(
+    db: AsyncSession,
+    *,
+    source: str,
+    snapshot_id: int | None,
+    task_id: int | None,
+    events: list[dict[str, Any]],
+) -> int:
+    if not events:
+        return 0
+
+    ingest_batch_id = f"{source}:{task_id or 'adhoc'}:{snapshot_id or 'nosnapshot'}"
+    created = 0
+
+    for event in events:
+        metadata = event.get("metadata", {}) or {}
+        collector_location = metadata.get("collector_location") or {}
+        observed_at = _parse_timestamp(
+            metadata.get("timestamp") or event.get("reference_date")
+        )
+
+        db.add(
+            BGPObservation(
+                snapshot_id=snapshot_id,
+                task_id=task_id,
+                source=source,
+                ingest_batch_id=ingest_batch_id,
+                source_event_id=event.get("source_id"),
+                collector=metadata.get("collector"),
+                peer_asn=_safe_int(metadata.get("peer_asn")),
+                peer_ip=metadata.get("peer_ip"),
+                prefix=metadata.get("prefix"),
+                event_type=str(metadata.get("event_type") or "announcement"),
+                as_path=metadata.get("as_path") or [],
+                origin_asn=_safe_int(metadata.get("origin_asn")),
+                next_hop=metadata.get("next_hop"),
+                communities=metadata.get("communities") or [],
+                observed_at=observed_at,
+                collector_geo=collector_location,
+                raw_payload=metadata.get("raw_message") or {},
+                note=event.get("description"),
+            )
+        )
+        created += 1
+
+    if created:
+        await db.commit()
+
+    return created
+
+
+async def create_bgp_anomalies_for_batch(
+    db: AsyncSession,
+    *,
+    source: str,
+    snapshot_id: int | None,
+    task_id: int | None,
+    events: list[dict[str, Any]],
+) -> int:
+    if not events:
+        return 0
+
+    enriched_events = await enrich_bgp_events_for_batch(
+        db,
+        source=source,
+        events=events,
+    )
+
+    prefixes = {
+        event["metadata"].get("prefix")
+        for event in enriched_events
+        if event.get("metadata", {}).get("prefix")
+    }
+    previous_origin_map: dict[str, set[int]] = defaultdict(set)
+
+    if prefixes:
+        previous_query = await db.execute(
+            select(CollectedData).where(
+                CollectedData.source == source,
+                CollectedData.snapshot_id != snapshot_id,
+                CollectedData.extra_data["prefix"].as_string().in_(sorted(prefixes)),
+            )
+        )
+        for record in previous_query.scalars().all():
+            metadata = record.extra_data or {}
+            prefix = metadata.get("prefix")
+            origin = _safe_int(metadata.get("origin_asn"))
+            if prefix and origin is not None:
+                previous_origin_map[prefix].add(origin)
+
+    pending_anomalies = [
+        *detect_origin_change_anomalies(
+            source=source,
+            snapshot_id=snapshot_id,
+            task_id=task_id,
+            events=enriched_events,
+            previous_origin_map=previous_origin_map,
+        ),
+        *detect_more_specific_burst_anomalies(
+            source=source,
+            snapshot_id=snapshot_id,
+            task_id=task_id,
+            events=enriched_events,
+        ),
+        *detect_mass_withdrawal_anomalies(
+            source=source,
+            snapshot_id=snapshot_id,
+            task_id=task_id,
+            events=enriched_events,
+        ),
+        *detect_route_leak_anomalies(
+            source=source,
+            snapshot_id=snapshot_id,
+            task_id=task_id,
+            events=enriched_events,
+        ),
+        *detect_path_flap_anomalies(
+            source=source,
+            snapshot_id=snapshot_id,
+            task_id=task_id,
+            events=enriched_events,
+        ),
+    ]
+
+    if not pending_anomalies:
+        return 0
+
+    existing_result = await db.execute(
+        select(BGPAnomaly.entity_key).where(
+            BGPAnomaly.entity_key.in_([item.entity_key for item in pending_anomalies])
+        )
+    )
+    existing_keys = {row[0] for row in existing_result.fetchall()}
+    existing_anomalies: list[BGPAnomaly] = []
+    if existing_keys:
+        existing_anomaly_result = await db.execute(
+            select(BGPAnomaly).where(BGPAnomaly.entity_key.in_(sorted(existing_keys)))
+        )
+        existing_anomalies = existing_anomaly_result.scalars().all()
+
+    created = 0
+    created_anomalies: list[BGPAnomaly] = []
+    refreshed_anomalies: list[BGPAnomaly] = []
+    existing_map = {item.entity_key: item for item in existing_anomalies if item.entity_key}
+    for anomaly in pending_anomalies:
+        if anomaly.entity_key in existing_keys:
+            existing = existing_map.get(anomaly.entity_key)
+            if existing is not None:
+                existing.severity = anomaly.severity
+                existing.status = anomaly.status
+                existing.summary = anomaly.summary
+                existing.confidence = anomaly.confidence
+                existing.peer_scope = anomaly.peer_scope
+                existing.evidence = anomaly.evidence
+                existing.new_origin_asn = anomaly.new_origin_asn
+                existing.origin_asn = anomaly.origin_asn
+                refreshed_anomalies.append(existing)
+            continue
+        db.add(anomaly)
+        created_anomalies.append(anomaly)
+        created += 1
+
+    if created or refreshed_anomalies:
+        await db.commit()
+    incident_seed_anomalies = [*created_anomalies, *refreshed_anomalies]
+    if incident_seed_anomalies:
+        await create_bgp_incidents_for_anomalies(
+            db,
+            source=source,
+            snapshot_id=snapshot_id,
+            task_id=task_id,
+            anomalies=incident_seed_anomalies,
+        )
+    return created

backend/app/services/collectors/bgpstream.py

@@ -0,0 +1,132 @@
+"""BGPStream backfill collector."""
+
+from __future__ import annotations
+
+import asyncio
+import json
+import time
+import urllib.parse
+import urllib.request
+from typing import Any
+
+from app.services.collectors.base import BaseCollector
+from app.services.collectors.bgp_common import (
+    create_bgp_anomalies_for_batch,
+    normalize_bgp_event,
+    save_bgp_observations_for_batch,
+)
+
+
+class BGPStreamBackfillCollector(BaseCollector):
+    name = "bgpstream_bgp"
+    priority = "P1"
+    module = "L3"
+    frequency_hours = 6
+    data_type = "bgp_rib"
+    fail_on_empty = True
+
+    async def fetch(self) -> list[dict[str, Any]]:
+        if not self._resolved_url:
+            raise RuntimeError("BGPStream URL is not configured")
+
+        return await asyncio.to_thread(self._fetch_resource_windows)
+
+    def _fetch_resource_windows(self) -> list[dict[str, Any]]:
+        end = int(time.time()) - 3600
+        start = end - 86400
+        params = [
+            ("projects[]", "routeviews"),
+            ("collectors[]", "route-views2"),
+            ("types[]", "updates"),
+            ("intervals[]", f"{start},{end}"),
+        ]
+        url = f"{self._resolved_url}/data?{urllib.parse.urlencode(params)}"
+        request = urllib.request.Request(
+            url,
+            headers={"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)"},
+        )
+        with urllib.request.urlopen(request, timeout=30) as response:
+            body = json.loads(response.read().decode())
+
+        if body.get("error"):
+            raise RuntimeError(f"BGPStream broker error: {body['error']}")
+
+        return body.get("data", {}).get("resources", [])
+
+    def transform(self, raw_data: list[dict[str, Any]]) -> list[dict[str, Any]]:
+        transformed: list[dict[str, Any]] = []
+        for item in raw_data:
+            if not isinstance(item, dict):
+                continue
+
+            is_broker_window = any(key in item for key in ("filename", "url", "startTime", "start_time"))
+
+            if {"collector", "prefix"} <= set(item.keys()) and not is_broker_window:
+                transformed.append(normalize_bgp_event(item, project="bgpstream"))
+                continue
+
+            # Broker responses provide file windows rather than decoded events.
+            collector = item.get("collector") or item.get("project") or "bgpstream"
+            timestamp = item.get("time") or item.get("startTime") or item.get("start_time")
+            name = item.get("filename") or item.get("url") or f"{collector}-window"
+            normalized = normalize_bgp_event(
+                {
+                    "collector": collector,
+                    "event_type": "rib",
+                    "prefix": item.get("prefix") or "historical-window",
+                    "timestamp": timestamp,
+                    "origin_asn": item.get("origin_asn"),
+                    "path": item.get("path") or [],
+                    "raw_message": item,
+                },
+                project="bgpstream",
+            )
+            transformed.append(
+                normalized
+                | {
+                    "name": name,
+                    "title": f"BGPStream {collector}",
+                    "description": "Historical BGPStream backfill window",
+                    "metadata": {
+                        **normalized["metadata"],
+                        "broker_record": item,
+                    },
+                }
+            )
+        self._latest_transformed_batch = transformed
+        return transformed
+
+    async def run(self, db):
+        result = await super().run(db)
+        if result.get("status") != "success":
+            return result
+
+        snapshot_id = await self._resolve_snapshot_id(db, result.get("task_id"))
+        observation_count = await save_bgp_observations_for_batch(
+            db,
+            source=self.name,
+            snapshot_id=snapshot_id,
+            task_id=result.get("task_id"),
+            events=getattr(self, "_latest_transformed_batch", []),
+        )
+        anomaly_count = await create_bgp_anomalies_for_batch(
+            db,
+            source=self.name,
+            snapshot_id=snapshot_id,
+            task_id=result.get("task_id"),
+            events=getattr(self, "_latest_transformed_batch", []),
+        )
+        result["observations_created"] = observation_count
+        result["anomalies_created"] = anomaly_count
+        return result
+
+    async def _resolve_snapshot_id(self, db, task_id: int | None) -> int | None:
+        if task_id is None:
+            return None
+        from sqlalchemy import select
+        from app.models.data_snapshot import DataSnapshot
+
+        result = await db.execute(
+            select(DataSnapshot.id).where(DataSnapshot.task_id == task_id).order_by(DataSnapshot.id.desc())
+        )
+        return result.scalar_one_or_none()

backend/app/services/collectors/celestrak.py

@@ -8,6 +8,7 @@ import json
 from typing import Dict, Any, List
 import httpx
 
+from app.core.satellite_tle import build_tle_lines_from_elements
 from app.services.collectors.base import BaseCollector
 
 
@@ -20,7 +21,7 @@ class CelesTrakTLECollector(BaseCollector):
 
     @property
     def base_url(self) -> str:
-        return "https://celestrak.org/NORAD/elements/gp.php"
+        return self._resolved_url or ""
 
     async def fetch(self) -> List[Dict[str, Any]]:
         satellite_groups = [
@@ -39,7 +40,7 @@ class CelesTrakTLECollector(BaseCollector):
         async with httpx.AsyncClient(timeout=120.0) as client:
             for group in satellite_groups:
                 try:
-                    url = f"https://celestrak.org/NORAD/elements/gp.php?GROUP={group}&FORMAT=json"
+                    url = f"{self.base_url}?GROUP={group}&FORMAT=json"
                     response = await client.get(url)
 
                     if response.status_code == 200:
@@ -61,6 +62,17 @@ class CelesTrakTLECollector(BaseCollector):
     def transform(self, raw_data: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
         transformed = []
         for item in raw_data:
+            tle_line1, tle_line2 = build_tle_lines_from_elements(
+                norad_cat_id=item.get("NORAD_CAT_ID"),
+                epoch=item.get("EPOCH"),
+                inclination=item.get("INCLINATION"),
+                raan=item.get("RA_OF_ASC_NODE"),
+                eccentricity=item.get("ECCENTRICITY"),
+                arg_of_perigee=item.get("ARG_OF_PERICENTER"),
+                mean_anomaly=item.get("MEAN_ANOMALY"),
+                mean_motion=item.get("MEAN_MOTION"),
+            )
+
             transformed.append(
                 {
                     "name": item.get("OBJECT_NAME", "Unknown"),
@@ -80,6 +92,10 @@ class CelesTrakTLECollector(BaseCollector):
                         "mean_motion_dot": item.get("MEAN_MOTION_DOT"),
                         "mean_motion_ddot": item.get("MEAN_MOTION_DDOT"),
                         "ephemeris_type": item.get("EPHEMERIS_TYPE"),
+                        # Prefer the original TLE lines when the source provides them.
+                        # If they are missing, store a normalized TLE pair built once on the backend.
+                        "tle_line1": item.get("TLE_LINE1") or tle_line1,
+                        "tle_line2": item.get("TLE_LINE2") or tle_line2,
                     },
                 }
             )

backend/app/services/collectors/cloudflare.py

@@ -10,7 +10,7 @@ Some endpoints require authentication for higher rate limits.
 import asyncio
 import os
 from typing import Dict, Any, List
-from datetime import datetime
+from datetime import UTC, datetime
 
 import httpx
 from app.services.collectors.base import HTTPCollector
@@ -39,6 +39,16 @@ class CloudflareRadarDeviceCollector(HTTPCollector):
         if CLOUDFLARE_API_TOKEN:
             self.headers["Authorization"] = f"Bearer {CLOUDFLARE_API_TOKEN}"
 
+    @property
+    def request_url(self) -> str:
+        return self._resolved_url or self.base_url
+
+    async def fetch(self) -> List[Dict[str, Any]]:
+        async with httpx.AsyncClient(timeout=60.0) as client:
+            response = await client.get(self.request_url, headers=self.headers)
+            response.raise_for_status()
+            return self.parse_response(response.json())
+
     def parse_response(self, response: Dict[str, Any]) -> List[Dict[str, Any]]:
         """Parse Cloudflare Radar device type response"""
         data = []
@@ -59,7 +69,7 @@ class CloudflareRadarDeviceCollector(HTTPCollector):
                     "other_percent": float(summary.get("other", 0)),
                     "date_range": result.get("meta", {}).get("dateRange", {}),
                 },
-                "reference_date": datetime.utcnow().isoformat(),
+                "reference_date": datetime.now(UTC).isoformat(),
             }
             data.append(entry)
         except (ValueError, TypeError, KeyError):
@@ -87,6 +97,16 @@ class CloudflareRadarTrafficCollector(HTTPCollector):
         if CLOUDFLARE_API_TOKEN:
             self.headers["Authorization"] = f"Bearer {CLOUDFLARE_API_TOKEN}"
 
+    @property
+    def request_url(self) -> str:
+        return self._resolved_url or self.base_url
+
+    async def fetch(self) -> List[Dict[str, Any]]:
+        async with httpx.AsyncClient(timeout=60.0) as client:
+            response = await client.get(self.request_url, headers=self.headers)
+            response.raise_for_status()
+            return self.parse_response(response.json())
+
     def parse_response(self, response: Dict[str, Any]) -> List[Dict[str, Any]]:
         """Parse Cloudflare Radar traffic timeseries response"""
         data = []
@@ -107,7 +127,7 @@ class CloudflareRadarTrafficCollector(HTTPCollector):
                         "requests": item.get("requests"),
                         "visit_duration": item.get("visitDuration"),
                     },
-                    "reference_date": item.get("datetime", datetime.utcnow().isoformat()),
+                    "reference_date": item.get("datetime", datetime.now(UTC).isoformat()),
                 }
                 data.append(entry)
             except (ValueError, TypeError, KeyError):
@@ -135,6 +155,16 @@ class CloudflareRadarTopASCollector(HTTPCollector):
         if CLOUDFLARE_API_TOKEN:
             self.headers["Authorization"] = f"Bearer {CLOUDFLARE_API_TOKEN}"
 
+    @property
+    def request_url(self) -> str:
+        return self._resolved_url or self.base_url
+
+    async def fetch(self) -> List[Dict[str, Any]]:
+        async with httpx.AsyncClient(timeout=60.0) as client:
+            response = await client.get(self.request_url, headers=self.headers)
+            response.raise_for_status()
+            return self.parse_response(response.json())
+
     def parse_response(self, response: Dict[str, Any]) -> List[Dict[str, Any]]:
         """Parse Cloudflare Radar top locations response"""
         data = []
@@ -155,7 +185,7 @@ class CloudflareRadarTopASCollector(HTTPCollector):
                         "traffic_share": item.get("trafficShare"),
                         "country_code": item.get("location", {}).get("countryCode"),
                     },
-                    "reference_date": datetime.utcnow().isoformat(),
+                    "reference_date": datetime.now(UTC).isoformat(),
                 }
                 data.append(entry)
             except (ValueError, TypeError, KeyError):

backend/app/services/collectors/downloads.py

@@ -0,0 +1,204 @@
+"""Shared resumable download helpers for collectors."""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import tempfile
+import time
+from datetime import UTC, datetime
+from pathlib import Path
+from typing import Any, Awaitable, Callable
+
+import httpx
+
+
+ProgressCallback = Callable[[int, int | None], Awaitable[None]]
+ValidateCallback = Callable[[Path], bool]
+
+
+class ResumableFileDownloader:
+    """Download files with cache validators and byte-range resume support."""
+
+    def __init__(
+        self,
+        *,
+        cache_namespace: str,
+        user_agent: str = "Planet-Intelligence-System/1.0 (Python/collector)",
+        default_accept: str = "*/*",
+    ) -> None:
+        self._cache_dir = Path(tempfile.gettempdir()) / "planet-download-cache" / cache_namespace
+        self._user_agent = user_agent
+        self._default_accept = default_accept
+
+    @staticmethod
+    def _cache_key(url: str) -> str:
+        return hashlib.sha1(url.encode("utf-8")).hexdigest()[:16]
+
+    def _cache_paths(self, url: str, extension: str) -> tuple[Path, Path, Path]:
+        key = self._cache_key(url)
+        normalized_ext = extension if extension.startswith(".") else f".{extension}"
+        final_path = self._cache_dir / f"{key}{normalized_ext}"
+        part_path = self._cache_dir / f"{key}{normalized_ext}.part"
+        meta_path = self._cache_dir / f"{key}.meta.json"
+        return final_path, part_path, meta_path
+
+    @staticmethod
+    def _load_meta(meta_path: Path) -> dict[str, Any]:
+        if not meta_path.exists():
+            return {}
+        try:
+            return json.loads(meta_path.read_text(encoding="utf-8"))
+        except (json.JSONDecodeError, OSError):
+            return {}
+
+    @staticmethod
+    def _save_meta(meta_path: Path, payload: dict[str, Any]) -> None:
+        meta_path.write_text(json.dumps(payload, ensure_ascii=False), encoding="utf-8")
+
+    @staticmethod
+    def _validators_match(meta: dict[str, Any], remote: dict[str, Any]) -> bool:
+        etag = str(remote.get("etag") or "").strip()
+        last_modified = str(remote.get("last_modified") or "").strip()
+        if etag:
+            return etag == str(meta.get("etag") or "").strip()
+        if last_modified:
+            return last_modified == str(meta.get("last_modified") or "").strip()
+        return True
+
+    async def fetch_remote_info(self, client: httpx.AsyncClient, url: str) -> dict[str, Any]:
+        try:
+            response = await client.head(url)
+            if response.status_code >= 400:
+                return {}
+            content_length_raw = response.headers.get("content-length")
+            content_length = int(content_length_raw) if content_length_raw else None
+            return {
+                "etag": response.headers.get("etag"),
+                "last_modified": response.headers.get("last-modified"),
+                "content_length": content_length,
+                "accept_ranges": (response.headers.get("accept-ranges") or "").lower(),
+            }
+        except (httpx.HTTPError, ValueError):
+            return {}
+
+    async def download_file(
+        self,
+        client: httpx.AsyncClient,
+        url: str,
+        *,
+        extension: str,
+        accept: str | None = None,
+        progress_callback: ProgressCallback | None = None,
+        validate_existing: ValidateCallback | None = None,
+    ) -> Path:
+        self._cache_dir.mkdir(parents=True, exist_ok=True)
+        final_path, part_path, meta_path = self._cache_paths(url, extension)
+        meta = self._load_meta(meta_path)
+        remote = await self.fetch_remote_info(client, url)
+        expected_size = remote.get("content_length")
+
+        if final_path.exists():
+            local_size = final_path.stat().st_size
+            size_match = expected_size is None or local_size == expected_size
+            if self._validators_match(meta, remote) and size_match:
+                if validate_existing and not validate_existing(final_path):
+                    final_path.unlink(missing_ok=True)
+                else:
+                    if progress_callback and expected_size and expected_size > 0:
+                        await progress_callback(expected_size, expected_size)
+                    return final_path
+
+        can_resume = (remote.get("accept_ranges") or "") == "bytes"
+        resume_from = part_path.stat().st_size if part_path.exists() else 0
+        if expected_size is not None and resume_from > expected_size:
+            part_path.unlink(missing_ok=True)
+            resume_from = 0
+        if not self._validators_match(meta, remote):
+            part_path.unlink(missing_ok=True)
+            resume_from = 0
+
+        headers = {
+            "User-Agent": self._user_agent,
+            "Accept": accept or self._default_accept,
+        }
+        if final_path.exists():
+            if meta.get("etag"):
+                headers["If-None-Match"] = str(meta.get("etag"))
+            elif meta.get("last_modified"):
+                headers["If-Modified-Since"] = str(meta.get("last_modified"))
+
+        if can_resume and resume_from > 0:
+            headers["Range"] = f"bytes={resume_from}-"
+            if remote.get("etag"):
+                headers["If-Range"] = str(remote.get("etag"))
+            elif remote.get("last_modified"):
+                headers["If-Range"] = str(remote.get("last_modified"))
+
+        async with client.stream("GET", url, headers=headers) as response:
+            if response.status_code == 304 and final_path.exists():
+                if progress_callback and expected_size and expected_size > 0:
+                    await progress_callback(expected_size, expected_size)
+                return final_path
+            response.raise_for_status()
+
+            if response.status_code == 206 and resume_from > 0:
+                mode = "ab"
+            else:
+                mode = "wb"
+                resume_from = 0
+
+            downloaded = resume_from
+            last_emit_bytes = 0
+            last_emit_time = time.monotonic()
+            min_emit_bytes = (
+                max(expected_size // 150, 512 * 1024) if expected_size and expected_size > 0 else 1024 * 1024
+            )
+
+            with part_path.open(mode) as f:
+                if progress_callback and downloaded > 0:
+                    await progress_callback(downloaded, expected_size)
+                async for chunk in response.aiter_bytes():
+                    if not chunk:
+                        continue
+                    f.write(chunk)
+                    downloaded += len(chunk)
+                    if not progress_callback:
+                        continue
+                    now = time.monotonic()
+                    should_emit = (
+                        expected_size is None
+                        or downloaded >= expected_size
+                        or downloaded - last_emit_bytes >= min_emit_bytes
+                        or now - last_emit_time >= 2.0
+                    )
+                    if should_emit:
+                        last_emit_bytes = downloaded
+                        last_emit_time = now
+                        await progress_callback(downloaded, expected_size)
+
+        final_size = part_path.stat().st_size if part_path.exists() else 0
+        if expected_size is not None and final_size != expected_size:
+            raise RuntimeError(
+                f"Resumable download incomplete for {url}: expected={expected_size}, got={final_size}"
+            )
+
+        part_path.replace(final_path)
+        self._save_meta(
+            meta_path,
+            {
+                "url": url,
+                "etag": remote.get("etag"),
+                "last_modified": remote.get("last_modified"),
+                "content_length": expected_size,
+                "updated_at": datetime.now(UTC).isoformat(),
+            },
+        )
+
+        if validate_existing and not validate_existing(final_path):
+            raise RuntimeError(f"Downloaded file validation failed for {url}")
+
+        if progress_callback and expected_size and expected_size > 0:
+            await progress_callback(expected_size, expected_size)
+
+        return final_path

backend/app/services/collectors/epoch_ai.py

@@ -6,7 +6,7 @@ https://epoch.ai/data/gpu-clusters
 
 import re
 from typing import Dict, Any, List
-from datetime import datetime
+from datetime import UTC, datetime
 from bs4 import BeautifulSoup
 import httpx
 
@@ -23,7 +23,7 @@ class EpochAIGPUCollector(BaseCollector):
 
     async def fetch(self) -> List[Dict[str, Any]]:
         """Fetch Epoch AI GPU clusters data from webpage"""
-        url = "https://epoch.ai/data/gpu-clusters"
+        url = self._resolved_url or ""
 
         async with httpx.AsyncClient(timeout=60.0) as client:
             response = await client.get(url)
@@ -64,7 +64,7 @@ class EpochAIGPUCollector(BaseCollector):
                             "metadata": {
                                 "raw_data": perf_cell,
                             },
-                            "reference_date": datetime.utcnow().strftime("%Y-%m-%d"),
+                            "reference_date": datetime.now(UTC).strftime("%Y-%m-%d"),
                         }
                         data.append(entry)
                     except (ValueError, IndexError, AttributeError):
@@ -114,6 +114,6 @@ class EpochAIGPUCollector(BaseCollector):
                 "metadata": {
                     "note": "Sample data - Epoch AI page structure may vary",
                 },
-                "reference_date": datetime.utcnow().strftime("%Y-%m-%d"),
+                "reference_date": datetime.now(UTC).strftime("%Y-%m-%d"),
             },
         ]

backend/app/services/collectors/fao_landing.py

@@ -4,7 +4,7 @@ Collects landing point data from FAO CSV API.
 """
 
 from typing import Dict, Any, List
-from datetime import datetime
+from datetime import UTC, datetime
 import httpx
 
 from app.services.collectors.base import BaseCollector
@@ -18,11 +18,9 @@ class FAOLandingPointCollector(BaseCollector):
     frequency_hours = 168
     data_type = "landing_point"
 
-    csv_url = "https://data.apps.fao.org/catalog/dataset/1b75ff21-92f2-4b96-9b7b-98e8aa65ad5d/resource/b6071077-d1d4-4e97-aa00-42e902847c87/download/landing-point-geo.csv"
-
     async def fetch(self) -> List[Dict[str, Any]]:
         async with httpx.AsyncClient(timeout=60.0) as client:
-            response = await client.get(self.csv_url)
+            response = await client.get(self._resolved_url or "")
             response.raise_for_status()
             return self.parse_csv(response.text)
 
@@ -58,7 +56,7 @@ class FAOLandingPointCollector(BaseCollector):
                             "is_tbd": is_tbd,
                             "original_id": feature_id,
                         },
-                        "reference_date": datetime.utcnow().strftime("%Y-%m-%d"),
+                        "reference_date": datetime.now(UTC).strftime("%Y-%m-%d"),
                     }
                     result.append(entry)
                 except (ValueError, IndexError):

backend/app/services/collectors/huggingface.py

@@ -7,7 +7,7 @@ https://huggingface.co/spaces
 """
 
 from typing import Dict, Any, List
-from datetime import datetime
+from datetime import UTC, datetime
 
 from app.services.collectors.base import HTTPCollector
 
@@ -21,6 +21,18 @@ class HuggingFaceModelCollector(HTTPCollector):
     data_type = "model"
     base_url = "https://huggingface.co/api/models"
 
+    @property
+    def request_url(self) -> str:
+        return self._resolved_url or self.base_url
+
+    async def fetch(self) -> List[Dict[str, Any]]:
+        from httpx import AsyncClient
+
+        async with AsyncClient(timeout=60.0) as client:
+            response = await client.get(self.request_url, headers=self.headers)
+            response.raise_for_status()
+            return self.parse_response(response.json())
+
     def parse_response(self, response: Dict[str, Any]) -> List[Dict[str, Any]]:
         """Parse Hugging Face models API response"""
         data = []
@@ -46,7 +58,7 @@ class HuggingFaceModelCollector(HTTPCollector):
                         "library_name": item.get("library_name"),
                         "created_at": item.get("createdAt"),
                     },
-                    "reference_date": datetime.utcnow().strftime("%Y-%m-%d"),
+                    "reference_date": datetime.now(UTC).strftime("%Y-%m-%d"),
                 }
                 data.append(entry)
             except (ValueError, TypeError, KeyError):
@@ -63,6 +75,18 @@ class HuggingFaceDatasetCollector(HTTPCollector):
     data_type = "dataset"
     base_url = "https://huggingface.co/api/datasets"
 
+    @property
+    def request_url(self) -> str:
+        return self._resolved_url or self.base_url
+
+    async def fetch(self) -> List[Dict[str, Any]]:
+        from httpx import AsyncClient
+
+        async with AsyncClient(timeout=60.0) as client:
+            response = await client.get(self.request_url, headers=self.headers)
+            response.raise_for_status()
+            return self.parse_response(response.json())
+
     def parse_response(self, response: Dict[str, Any]) -> List[Dict[str, Any]]:
         """Parse Hugging Face datasets API response"""
         data = []
@@ -87,7 +111,7 @@ class HuggingFaceDatasetCollector(HTTPCollector):
                         "tags": (item.get("tags", []) or [])[:10],
                         "created_at": item.get("createdAt"),
                     },
-                    "reference_date": datetime.utcnow().strftime("%Y-%m-%d"),
+                    "reference_date": datetime.now(UTC).strftime("%Y-%m-%d"),
                 }
                 data.append(entry)
             except (ValueError, TypeError, KeyError):
@@ -104,6 +128,18 @@ class HuggingFaceSpacesCollector(HTTPCollector):
     data_type = "space"
     base_url = "https://huggingface.co/api/spaces"
 
+    @property
+    def request_url(self) -> str:
+        return self._resolved_url or self.base_url
+
+    async def fetch(self) -> List[Dict[str, Any]]:
+        from httpx import AsyncClient
+
+        async with AsyncClient(timeout=60.0) as client:
+            response = await client.get(self.request_url, headers=self.headers)
+            response.raise_for_status()
+            return self.parse_response(response.json())
+
     def parse_response(self, response: Dict[str, Any]) -> List[Dict[str, Any]]:
         """Parse Hugging Face Spaces API response"""
         data = []
@@ -128,7 +164,7 @@ class HuggingFaceSpacesCollector(HTTPCollector):
                         "tags": (item.get("tags", []) or [])[:10],
                         "created_at": item.get("createdAt"),
                     },
-                    "reference_date": datetime.utcnow().strftime("%Y-%m-%d"),
+                    "reference_date": datetime.now(UTC).strftime("%Y-%m-%d"),
                 }
                 data.append(entry)
             except (ValueError, TypeError, KeyError):

backend/app/services/collectors/iptoasn.py

@@ -0,0 +1,207 @@
+"""IPtoASN prefix geography collector.
+
+Downloads the public combined IPv4+IPv6 TSV database and stores coarse
+prefix-to-country/ASN geography hints for BGP enrichment.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import gzip
+import time
+from datetime import UTC, datetime
+from ipaddress import summarize_address_range, ip_address
+from pathlib import Path
+from typing import Any
+
+import httpx
+
+from app.services.collectors.base import BaseCollector
+from app.services.collectors.downloads import ResumableFileDownloader
+
+
+class IPtoASNPrefixGeoCollector(BaseCollector):
+    name = "iptoasn_prefix_geo"
+    priority = "P1"
+    module = "L3"
+    frequency_hours = 24
+    data_type = "prefix_geography"
+    fail_on_empty = True
+    _downloader = ResumableFileDownloader(
+        cache_namespace="iptoasn",
+        default_accept="application/gzip,application/octet-stream,*/*",
+    )
+
+    @staticmethod
+    def _build_dataset_urls(resolved_url: str) -> list[str]:
+        if "ip2asn-combined.tsv.gz" in resolved_url:
+            return [
+                resolved_url.replace("ip2asn-combined.tsv.gz", "ip2asn-v4.tsv.gz"),
+                resolved_url.replace("ip2asn-combined.tsv.gz", "ip2asn-v6.tsv.gz"),
+            ]
+        return [resolved_url]
+
+    def _parse_rows_from_gzip_file(self, file_path: Path) -> list[dict[str, Any]]:
+        rows: list[dict[str, Any]] = []
+        with gzip.open(file_path, "rt", encoding="utf-8", errors="replace") as f:
+            for raw_line in f:
+                line = raw_line.strip()
+                if not line or line.startswith("#"):
+                    continue
+                parts = line.split("\t")
+                if len(parts) < 5:
+                    continue
+                range_start, range_end, asn, country_code, as_name = parts[:5]
+                rows.append(
+                    {
+                        "range_start": range_start,
+                        "range_end": range_end,
+                        "asn": asn,
+                        "country_code": country_code,
+                        "as_name": as_name,
+                    }
+                )
+        if not rows:
+            raise RuntimeError(f"IPtoASN dataset parsed empty rows: {file_path.name}")
+        return rows
+
+    async def _fetch_dataset_rows(
+        self,
+        client: httpx.AsyncClient,
+        url: str,
+        *,
+        progress_callback=None,
+    ) -> list[dict[str, Any]]:
+        file_path = await self._downloader.download_file(
+            client,
+            url,
+            extension=".tsv.gz",
+            progress_callback=progress_callback,
+            validate_existing=lambda p: self._validate_gzip_dataset(p),
+        )
+        return self._parse_rows_from_gzip_file(file_path)
+
+    def _validate_gzip_dataset(self, file_path: Path) -> bool:
+        try:
+            self._parse_rows_from_gzip_file(file_path)
+            return True
+        except Exception:
+            return False
+
+    async def fetch(self) -> list[dict[str, Any]]:
+        if not self._resolved_url:
+            raise RuntimeError("IPtoASN combined URL is not configured")
+
+        dataset_urls = self._build_dataset_urls(self._resolved_url)
+
+        async with httpx.AsyncClient(timeout=180.0, follow_redirects=True) as client:
+            remote_infos = await asyncio.gather(
+                *(self._downloader.fetch_remote_info(client, url) for url in dataset_urls)
+            )
+            expected_sizes = [
+                info.get("content_length")
+                for info in remote_infos
+                if isinstance(info.get("content_length"), int)
+            ]
+            total_expected = sum(expected_sizes) if expected_sizes else 0
+            if total_expected > 0 and self._current_task and self._db_session:
+                self._current_task.total_records = total_expected
+                self._current_task.records_processed = 0
+                self._current_task.progress = 0.0
+                await self._db_session.commit()
+                await self._publish_task_update(force=True)
+
+            url_progress: dict[str, int] = {url: 0 for url in dataset_urls}
+            progress_lock = asyncio.Lock()
+            last_emit = {"t": 0.0, "value": 0}
+            min_emit_bytes = max(total_expected // 200, 2 * 1024 * 1024) if total_expected > 0 else 4 * 1024 * 1024
+
+            async def on_url_progress(url: str, downloaded_bytes: int, total_bytes: int | None) -> None:
+                if total_expected <= 0:
+                    return
+                async with progress_lock:
+                    current = max(0, downloaded_bytes)
+                    if current < url_progress[url]:
+                        return
+                    url_progress[url] = current
+                    aggregated = sum(url_progress.values())
+                    now = time.monotonic()
+                    should_emit = (
+                        aggregated >= total_expected
+                        or aggregated - last_emit["value"] >= min_emit_bytes
+                        or now - last_emit["t"] >= 2.0
+                    )
+                    if not should_emit:
+                        return
+                    last_emit["value"] = aggregated
+                    last_emit["t"] = now
+                await self.update_progress(min(aggregated, total_expected), commit=True)
+
+            batches = await asyncio.gather(
+                *(
+                    self._fetch_dataset_rows(
+                        client,
+                        url,
+                        progress_callback=lambda downloaded, total, u=url: on_url_progress(u, downloaded, total),
+                    )
+                    for url in dataset_urls
+                )
+            )
+            if total_expected > 0:
+                await self.update_progress(total_expected, commit=True, force=True)
+
+        rows: list[dict[str, Any]] = []
+        for batch in batches:
+            rows.extend(batch)
+        return rows
+
+    def transform(self, raw_data: list[dict[str, Any]]) -> list[dict[str, Any]]:
+        reference_date = datetime.now(UTC).isoformat()
+        transformed: list[dict[str, Any]] = []
+
+        for item in raw_data:
+            try:
+                start_ip = ip_address(str(item["range_start"]))
+                end_ip = ip_address(str(item["range_end"]))
+            except ValueError:
+                continue
+
+            if start_ip.version != end_ip.version:
+                continue
+
+            summarized = list(summarize_address_range(start_ip, end_ip))
+            primary_prefix = str(summarized[0]) if summarized else f"{start_ip}/{32 if start_ip.version == 4 else 128}"
+            family = f"ipv{start_ip.version}"
+
+            asn_value = item.get("asn")
+            try:
+                normalized_asn = int(str(asn_value))
+            except (TypeError, ValueError):
+                normalized_asn = None
+
+            transformed.append(
+                {
+                    "source_id": f"{family}:{item['range_start']}-{item['range_end']}",
+                    "name": primary_prefix,
+                    "title": f"{primary_prefix} {item.get('country_code', '').strip()}".strip(),
+                    "country": item.get("country_code"),
+                    "city": "",
+                    "latitude": None,
+                    "longitude": None,
+                    "metadata": {
+                        "family": family,
+                        "range_start": item["range_start"],
+                        "range_end": item["range_end"],
+                        "prefix": primary_prefix,
+                        "prefixes": [str(prefix) for prefix in summarized[:8]],
+                        "range_prefix_count": len(summarized),
+                        "country_code": item.get("country_code"),
+                        "asn": normalized_asn,
+                        "as_name": item.get("as_name"),
+                        "source_dataset": "iptoasn_combined",
+                    },
+                    "reference_date": reference_date,
+                }
+            )
+
+        return transformed

backend/app/services/collectors/news_live_streams.py

@@ -0,0 +1,79 @@
+from __future__ import annotations
+
+from datetime import UTC, datetime
+from typing import Any
+
+import httpx
+
+from app.services.collectors.base import BaseCollector
+
+
+class NewsLiveStreamsCollector(BaseCollector):
+    """Collect normalized news live-stream sources from a JSON endpoint."""
+
+    name = "news_live_streams"
+    priority = "P2"
+    module = "L4"
+    frequency_hours = 12
+    data_type = "news_live_stream"
+    fail_on_empty = False
+
+    async def fetch(self) -> list[dict[str, Any]]:
+        request_url = (self._resolved_url or "").strip()
+        if not request_url:
+            return []
+
+        async with httpx.AsyncClient(timeout=45.0, follow_redirects=True) as client:
+            response = await client.get(
+                request_url,
+                headers={
+                    "User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
+                    "Accept": "application/json",
+                },
+            )
+            response.raise_for_status()
+            return self.parse_response(response.json())
+
+    def parse_response(self, response: Any) -> list[dict[str, Any]]:
+        if isinstance(response, dict):
+            candidates = response.get("sources") or response.get("streams") or response.get("data") or []
+        elif isinstance(response, list):
+            candidates = response
+        else:
+            candidates = []
+
+        normalized: list[dict[str, Any]] = []
+        for index, item in enumerate(candidates):
+            if not isinstance(item, dict):
+                continue
+
+            stream_id = item.get("id") or item.get("source_id") or item.get("slug") or f"news-live-{index + 1}"
+            name = str(item.get("name") or item.get("title") or f"News Live {index + 1}").strip()
+            if not name:
+                continue
+
+            metadata = {
+                "provider": item.get("provider") or item.get("publisher") or "Collector",
+                "region": item.get("region") or item.get("country") or "Global",
+                "language": item.get("language") or "und",
+                "source_type": item.get("source_type") or "iframe",
+                "embed_url": item.get("embed_url") or item.get("url") or "",
+                "stream_url": item.get("stream_url") or "",
+                "homepage_url": item.get("homepage_url") or item.get("source_url") or "",
+                "poster_url": item.get("poster_url") or "",
+                "sort_order": item.get("sort_order", 200 + index),
+                "notes": item.get("notes") or item.get("description") or "",
+                "is_enabled": item.get("is_enabled", True),
+            }
+
+            normalized.append(
+                {
+                    "source_id": str(stream_id),
+                    "name": name,
+                    "description": metadata["notes"],
+                    "metadata": metadata,
+                    "reference_date": item.get("reference_date", datetime.now(UTC).isoformat()),
+                }
+            )
+
+        return normalized

backend/app/services/collectors/nro_delegated.py

@@ -0,0 +1,152 @@
+"""NRO delegated stats prefix geography collector.
+
+Parses the delegated extended/statistics file and stores coarse registry
+allocation geography as prefix-centric fallback hints.
+"""
+
+from __future__ import annotations
+
+import ipaddress
+from datetime import UTC, datetime
+from typing import Any
+
+import httpx
+
+from app.services.collectors.base import BaseCollector
+from app.services.collectors.downloads import ResumableFileDownloader
+
+
+class NRODelegatedPrefixGeoCollector(BaseCollector):
+    name = "nro_delegated_prefix_geo"
+    priority = "P1"
+    module = "L3"
+    frequency_hours = 24
+    data_type = "prefix_geography"
+    fail_on_empty = True
+    _downloader = ResumableFileDownloader(
+        cache_namespace="nro",
+        default_accept="text/plain,*/*",
+    )
+
+    async def fetch(self) -> list[dict[str, Any]]:
+        if not self._resolved_url:
+            raise RuntimeError("NRO delegated stats URL is not configured")
+
+        async with httpx.AsyncClient(timeout=180.0, follow_redirects=True) as client:
+            remote = await self._downloader.fetch_remote_info(client, self._resolved_url)
+            total_expected = remote.get("content_length") or 0
+            if total_expected > 0 and self._current_task and self._db_session:
+                self._current_task.total_records = total_expected
+                self._current_task.records_processed = 0
+                self._current_task.progress = 0.0
+                await self._db_session.commit()
+                await self._publish_task_update(force=True)
+
+            async def on_progress(downloaded: int, total: int | None) -> None:
+                if not total or total <= 0:
+                    return
+                await self.update_progress(min(downloaded, total), commit=True)
+
+            body_path = await self._downloader.download_file(
+                client,
+                self._resolved_url,
+                extension=".txt",
+                progress_callback=on_progress,
+            )
+            body = body_path.read_text(encoding="utf-8", errors="replace")
+
+        rows: list[dict[str, Any]] = []
+        for raw_line in body.splitlines():
+            line = raw_line.strip()
+            if not line or line.startswith("#"):
+                continue
+
+            parts = line.split("|")
+            if len(parts) < 7:
+                continue
+
+            rir = (parts[0] or "").strip().lower()
+            country_code = (parts[1] or "").strip().upper()
+            record_type = (parts[2] or "").strip().lower()
+            start = (parts[3] or "").strip()
+            value = (parts[4] or "").strip()
+            allocated_date = (parts[5] or "").strip()
+            status = (parts[6] or "").strip().lower()
+
+            if record_type not in {"ipv4", "ipv6"}:
+                continue
+            if not start or not value:
+                continue
+
+            rows.append(
+                {
+                    "rir": rir,
+                    "country_code": country_code,
+                    "type": record_type,
+                    "start": start,
+                    "value": value,
+                    "allocated_date": allocated_date,
+                    "status": status,
+                }
+            )
+
+        return rows
+
+    def transform(self, raw_data: list[dict[str, Any]]) -> list[dict[str, Any]]:
+        reference_date = datetime.now(UTC).isoformat()
+        transformed: list[dict[str, Any]] = []
+
+        for item in raw_data:
+            record_type = str(item.get("type") or "").strip().lower()
+            start = str(item.get("start") or "").strip()
+            value = str(item.get("value") or "").strip()
+            country_code = str(item.get("country_code") or "").strip().upper()
+
+            try:
+                if record_type == "ipv4":
+                    start_ip = ipaddress.ip_address(start)
+                    count = int(value)
+                    if count <= 0:
+                        continue
+                    end_ip_int = int(start_ip) + count - 1
+                    end_ip = ipaddress.ip_address(end_ip_int)
+                    network = list(ipaddress.summarize_address_range(start_ip, end_ip))[0]
+                elif record_type == "ipv6":
+                    prefixlen = int(value)
+                    network = ipaddress.ip_network(f"{start}/{prefixlen}", strict=False)
+                    start_ip = network.network_address
+                    end_ip = network.broadcast_address
+                else:
+                    continue
+            except (ValueError, TypeError):
+                continue
+
+            family = f"ipv{network.version}"
+            prefix = str(network)
+
+            transformed.append(
+                {
+                    "source_id": f"{item.get('rir')}:{family}:{prefix}:{country_code}",
+                    "name": prefix,
+                    "title": f"{prefix} {country_code}".strip(),
+                    "country": country_code,
+                    "city": "",
+                    "latitude": None,
+                    "longitude": None,
+                    "metadata": {
+                        "family": family,
+                        "prefix": prefix,
+                        "range_start": str(start_ip),
+                        "range_end": str(end_ip),
+                        "country_code": country_code,
+                        "rir": item.get("rir"),
+                        "status": item.get("status"),
+                        "allocated_date": item.get("allocated_date"),
+                        "source_dataset": "nro_delegated_stats",
+                        "confidence": "registry_allocated",
+                    },
+                    "reference_date": reference_date,
+                }
+            )
+
+        return transformed

backend/app/services/collectors/opengeofeed.py

@@ -0,0 +1,135 @@
+"""OpenGeoFeed prefix geography collector.
+
+Fetches public OpenGeoFeed CSV data and stores higher-confidence
+prefix-to-location hints for BGP prefix-centric enrichment.
+"""
+
+from __future__ import annotations
+
+import csv
+import ipaddress
+from datetime import UTC, datetime
+from typing import Any
+
+import httpx
+
+from app.services.collectors.base import BaseCollector
+from app.services.collectors.downloads import ResumableFileDownloader
+
+
+class OpenGeoFeedPrefixGeoCollector(BaseCollector):
+    name = "opengeofeed_prefix_geo"
+    priority = "P1"
+    module = "L3"
+    frequency_hours = 24
+    data_type = "prefix_geography"
+    fail_on_empty = True
+    _downloader = ResumableFileDownloader(
+        cache_namespace="opengeofeed",
+        default_accept="text/csv,*/*",
+    )
+
+    async def fetch(self) -> list[dict[str, Any]]:
+        if not self._resolved_url:
+            raise RuntimeError("OpenGeoFeed URL is not configured")
+
+        async with httpx.AsyncClient(timeout=180.0, follow_redirects=True) as client:
+            remote = await self._downloader.fetch_remote_info(client, self._resolved_url)
+            total_expected = remote.get("content_length") or 0
+            if total_expected > 0 and self._current_task and self._db_session:
+                self._current_task.total_records = total_expected
+                self._current_task.records_processed = 0
+                self._current_task.progress = 0.0
+                await self._db_session.commit()
+                await self._publish_task_update(force=True)
+
+            async def on_progress(downloaded: int, total: int | None) -> None:
+                if not total or total <= 0:
+                    return
+                await self.update_progress(min(downloaded, total), commit=True)
+
+            body_path = await self._downloader.download_file(
+                client,
+                self._resolved_url,
+                extension=".csv",
+                progress_callback=on_progress,
+            )
+            body = body_path.read_text(encoding="utf-8", errors="replace")
+
+        rows: list[dict[str, Any]] = []
+        reader = csv.reader(body.splitlines())
+        for fields in reader:
+            if not fields:
+                continue
+            first = (fields[0] or "").strip().lower()
+            if not first or first.startswith("#") or first == "prefix":
+                continue
+
+            prefix = (fields[0] or "").strip()
+            country_code = (fields[1] if len(fields) > 1 else "").strip()
+            region = (fields[2] if len(fields) > 2 else "").strip()
+            city = (fields[3] if len(fields) > 3 else "").strip()
+            postal_code = (fields[4] if len(fields) > 4 else "").strip()
+
+            # Keep additional columns for future enrichment without breaking
+            # current normalized schema.
+            extras = [value.strip() for value in fields[5:]] if len(fields) > 5 else []
+
+            rows.append(
+                {
+                    "prefix": prefix,
+                    "country_code": country_code,
+                    "region": region,
+                    "city": city,
+                    "postal_code": postal_code,
+                    "extra_columns": extras,
+                }
+            )
+        return rows
+
+    def transform(self, raw_data: list[dict[str, Any]]) -> list[dict[str, Any]]:
+        reference_date = datetime.now(UTC).isoformat()
+        transformed: list[dict[str, Any]] = []
+
+        for item in raw_data:
+            prefix = str(item.get("prefix") or "").strip()
+            if not prefix:
+                continue
+            try:
+                network = ipaddress.ip_network(prefix, strict=False)
+            except ValueError:
+                continue
+
+            family = f"ipv{network.version}"
+            country_code = str(item.get("country_code") or "").strip().upper()
+            region = str(item.get("region") or "").strip()
+            city = str(item.get("city") or "").strip()
+            postal_code = str(item.get("postal_code") or "").strip()
+
+            transformed.append(
+                {
+                    "source_id": f"{family}:{prefix}:{country_code}:{region}:{city}",
+                    "name": prefix,
+                    "title": f"{prefix} {country_code}".strip(),
+                    "country": country_code,
+                    "city": city,
+                    "latitude": None,
+                    "longitude": None,
+                    "metadata": {
+                        "family": family,
+                        "prefix": prefix,
+                        "range_start": str(network.network_address),
+                        "range_end": str(network.broadcast_address),
+                        "country_code": country_code,
+                        "region": region,
+                        "city": city,
+                        "postal_code": postal_code,
+                        "extra_columns": item.get("extra_columns") or [],
+                        "source_dataset": "opengeofeed_public",
+                        "confidence": "geofeed",
+                    },
+                    "reference_date": reference_date,
+                }
+            )
+
+        return transformed

backend/app/services/collectors/peeringdb.py

@@ -13,9 +13,10 @@ To get higher limits, set PEERINGDB_API_KEY environment variable.
 import asyncio
 import os
 from typing import Dict, Any, List
-from datetime import datetime
+from datetime import UTC, datetime
 
 import httpx
+from urllib.parse import urlencode
 from app.services.collectors.base import HTTPCollector
 
 
@@ -38,9 +39,13 @@ class PeeringDBIXPCollector(HTTPCollector):
             "User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
             "Accept": "application/json",
         }
-        # API key is added to URL as query parameter
-        if PEERINGDB_API_KEY:
-            self.base_url = f"{self.base_url}?key={PEERINGDB_API_KEY}"
+    @property
+    def request_url(self) -> str:
+        base = self._resolved_url or self.base_url
+        if not PEERINGDB_API_KEY:
+            return base
+        separator = "&" if "?" in base else "?"
+        return f"{base}{separator}{urlencode({'key': PEERINGDB_API_KEY})}"
 
     async def fetch_with_retry(
         self, max_retries: int = 3, base_delay: float = 2.0
@@ -51,7 +56,7 @@ class PeeringDBIXPCollector(HTTPCollector):
         for attempt in range(max_retries):
             try:
                 async with httpx.AsyncClient(timeout=60.0) as client:
-                    response = await client.get(self.base_url, headers=self.headers)
+                    response = await client.get(self.request_url, headers=self.headers)
 
                     if response.status_code == 429:
                         # Rate limited - wait and retry with exponential backoff
@@ -76,7 +81,7 @@ class PeeringDBIXPCollector(HTTPCollector):
         print(f"Warning: PeeringDB collection failed after {max_retries} retries: {last_error}")
         return {}
 
-    async def collect(self) -> List[Dict[str, Any]]:
+    async def fetch(self) -> List[Dict[str, Any]]:
         """Collect IXP data from PeeringDB with rate limit handling"""
         response_data = await self.fetch_with_retry()
         if not response_data:
@@ -106,7 +111,7 @@ class PeeringDBIXPCollector(HTTPCollector):
                         "created": item.get("created"),
                         "updated": item.get("updated"),
                     },
-                    "reference_date": datetime.utcnow().isoformat(),
+                    "reference_date": datetime.now(UTC).isoformat(),
                 }
                 data.append(entry)
             except (ValueError, TypeError, KeyError):
@@ -141,8 +146,13 @@ class PeeringDBNetworkCollector(HTTPCollector):
             "User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
             "Accept": "application/json",
         }
-        if PEERINGDB_API_KEY:
-            self.base_url = f"{self.base_url}?key={PEERINGDB_API_KEY}"
+    @property
+    def request_url(self) -> str:
+        base = self._resolved_url or self.base_url
+        if not PEERINGDB_API_KEY:
+            return base
+        separator = "&" if "?" in base else "?"
+        return f"{base}{separator}{urlencode({'key': PEERINGDB_API_KEY})}"
 
     async def fetch_with_retry(
         self, max_retries: int = 3, base_delay: float = 2.0
@@ -153,7 +163,7 @@ class PeeringDBNetworkCollector(HTTPCollector):
         for attempt in range(max_retries):
             try:
                 async with httpx.AsyncClient(timeout=60.0) as client:
-                    response = await client.get(self.base_url, headers=self.headers)
+                    response = await client.get(self.request_url, headers=self.headers)
 
                     if response.status_code == 429:
                         delay = base_delay * (2**attempt)
@@ -177,7 +187,7 @@ class PeeringDBNetworkCollector(HTTPCollector):
         print(f"Warning: PeeringDB collection failed after {max_retries} retries: {last_error}")
         return {}
 
-    async def collect(self) -> List[Dict[str, Any]]:
+    async def fetch(self) -> List[Dict[str, Any]]:
         """Collect Network data from PeeringDB with rate limit handling"""
         response_data = await self.fetch_with_retry()
         if not response_data:
@@ -209,7 +219,7 @@ class PeeringDBNetworkCollector(HTTPCollector):
                         "created": item.get("created"),
                         "updated": item.get("updated"),
                     },
-                    "reference_date": datetime.utcnow().isoformat(),
+                    "reference_date": datetime.now(UTC).isoformat(),
                 }
                 data.append(entry)
             except (ValueError, TypeError, KeyError):
@@ -244,8 +254,13 @@ class PeeringDBFacilityCollector(HTTPCollector):
             "User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
             "Accept": "application/json",
         }
-        if PEERINGDB_API_KEY:
-            self.base_url = f"{self.base_url}?key={PEERINGDB_API_KEY}"
+    @property
+    def request_url(self) -> str:
+        base = self._resolved_url or self.base_url
+        if not PEERINGDB_API_KEY:
+            return base
+        separator = "&" if "?" in base else "?"
+        return f"{base}{separator}{urlencode({'key': PEERINGDB_API_KEY})}"
 
     async def fetch_with_retry(
         self, max_retries: int = 3, base_delay: float = 2.0
@@ -256,7 +271,7 @@ class PeeringDBFacilityCollector(HTTPCollector):
         for attempt in range(max_retries):
             try:
                 async with httpx.AsyncClient(timeout=60.0) as client:
-                    response = await client.get(self.base_url, headers=self.headers)
+                    response = await client.get(self.request_url, headers=self.headers)
 
                     if response.status_code == 429:
                         delay = base_delay * (2**attempt)
@@ -280,7 +295,7 @@ class PeeringDBFacilityCollector(HTTPCollector):
         print(f"Warning: PeeringDB collection failed after {max_retries} retries: {last_error}")
         return {}
 
-    async def collect(self) -> List[Dict[str, Any]]:
+    async def fetch(self) -> List[Dict[str, Any]]:
         """Collect Facility data from PeeringDB with rate limit handling"""
         response_data = await self.fetch_with_retry()
         if not response_data:
@@ -311,7 +326,7 @@ class PeeringDBFacilityCollector(HTTPCollector):
                         "created": item.get("created"),
                         "updated": item.get("updated"),
                     },
-                    "reference_date": datetime.utcnow().isoformat(),
+                    "reference_date": datetime.now(UTC).isoformat(),
                 }
                 data.append(entry)
             except (ValueError, TypeError, KeyError):

backend/app/services/collectors/ris_live.py

@@ -0,0 +1,143 @@
+"""RIPE RIS Live collector."""
+
+from __future__ import annotations
+
+import asyncio
+import json
+import urllib.request
+from typing import Any
+
+from app.services.collectors.base import BaseCollector
+from app.services.collectors.bgp_common import (
+    create_bgp_anomalies_for_batch,
+    normalize_bgp_event,
+    save_bgp_observations_for_batch,
+)
+
+
+class RISLiveCollector(BaseCollector):
+    name = "ris_live_bgp"
+    priority = "P1"
+    module = "L3"
+    frequency_hours = 1
+    data_type = "bgp_update"
+    fail_on_empty = True
+    max_messages = 100
+    idle_timeout_seconds = 15
+
+    async def fetch(self) -> list[dict[str, Any]]:
+        if not self._resolved_url:
+            raise RuntimeError("RIS Live URL is not configured")
+
+        return await asyncio.to_thread(self._fetch_via_stream)
+
+    def _fetch_via_stream(self) -> list[dict[str, Any]]:
+        events: list[dict[str, Any]] = []
+        stream_url = self._resolved_url or ""
+        subscribe = json.dumps(
+            {
+                "host": "rrc00",
+                "type": "UPDATE",
+                "require": "announcements",
+            }
+        )
+        request = urllib.request.Request(
+            stream_url,
+            headers={"X-RIS-Subscribe": subscribe},
+        )
+        with urllib.request.urlopen(request, timeout=20) as response:
+            while len(events) < self.max_messages:
+                line = response.readline().decode().strip()
+                if not line:
+                    break
+                payload = json.loads(line)
+                if payload.get("type") != "ris_message":
+                    continue
+                data = payload.get("data", {})
+                if isinstance(data, dict):
+                    events.append(data)
+        return events
+
+    def transform(self, raw_data: list[dict[str, Any]]) -> list[dict[str, Any]]:
+        transformed: list[dict[str, Any]] = []
+        for item in raw_data:
+            announcements = item.get("announcements") or []
+            withdrawals = item.get("withdrawals") or []
+
+            for announcement in announcements:
+                next_hop = announcement.get("next_hop")
+                for prefix in announcement.get("prefixes") or []:
+                    transformed.append(
+                        normalize_bgp_event(
+                            {
+                                **item,
+                                "collector": item.get("host", "").replace(".ripe.net", ""),
+                                "event_type": "announcement",
+                                "prefix": prefix,
+                                "next_hop": next_hop,
+                            },
+                            project="ris-live",
+                        )
+                    )
+
+            for prefix in withdrawals:
+                transformed.append(
+                    normalize_bgp_event(
+                        {
+                            **item,
+                            "collector": item.get("host", "").replace(".ripe.net", ""),
+                            "event_type": "withdrawal",
+                            "prefix": prefix,
+                        },
+                        project="ris-live",
+                    )
+                )
+
+            if not announcements and not withdrawals:
+                transformed.append(
+                    normalize_bgp_event(
+                        {
+                            **item,
+                            "collector": item.get("host", "").replace(".ripe.net", ""),
+                        },
+                        project="ris-live",
+                    )
+                )
+
+        self._latest_transformed_batch = transformed
+        return transformed
+
+    async def run(self, db):
+        result = await super().run(db)
+        if result.get("status") != "success":
+            return result
+
+        snapshot_id = await self._resolve_snapshot_id(db, result.get("task_id"))
+        observation_count = await save_bgp_observations_for_batch(
+            db,
+            source=self.name,
+            snapshot_id=snapshot_id,
+            task_id=result.get("task_id"),
+            events=getattr(self, "_latest_transformed_batch", []),
+        )
+        anomaly_count = await create_bgp_anomalies_for_batch(
+            db,
+            source=self.name,
+            snapshot_id=snapshot_id,
+            task_id=result.get("task_id"),
+            events=getattr(self, "_latest_transformed_batch", []),
+        )
+        result["observations_created"] = observation_count
+        result["anomalies_created"] = anomaly_count
+        return result
+
+    async def _resolve_snapshot_id(self, db, task_id: int | None) -> int | None:
+        if task_id is None:
+            return None
+        from sqlalchemy import select
+        from app.models.data_snapshot import DataSnapshot
+
+        result = await db.execute(
+            select(DataSnapshot.id).where(DataSnapshot.task_id == task_id).order_by(DataSnapshot.id.desc())
+        )
+        return result.scalar_one_or_none()

backend/app/services/collectors/spacetrack.py

@@ -7,9 +7,11 @@ API documentation: https://www.space-track.org/documentation
 import json
 from typing import Dict, Any, List
 import httpx
+from urllib.parse import urlparse
 
 from app.services.collectors.base import BaseCollector
 from app.core.data_sources import get_data_sources_config
+from app.core.satellite_tle import build_tle_lines_from_elements
 
 
 class SpaceTrackTLECollector(BaseCollector):
@@ -20,12 +22,30 @@ class SpaceTrackTLECollector(BaseCollector):
     data_type = "satellite_tle"
 
     @property
-    def base_url(self) -> str:
+    def query_url(self) -> str:
         config = get_data_sources_config()
         if self._resolved_url:
             return self._resolved_url
         return config.get_yaml_url("spacetrack_tle")
 
+    @property
+    def site_root(self) -> str:
+        config = get_data_sources_config()
+        configured_root = config.get_yaml_value("spacetrack.base_url")
+        if isinstance(configured_root, str) and configured_root:
+            return configured_root.rstrip("/")
+
+        parsed = urlparse(self.query_url)
+        return f"{parsed.scheme}://{parsed.netloc}".rstrip("/")
+
+    @property
+    def login_url(self) -> str:
+        return f"{self.site_root}/ajaxauth/login"
+
+    @property
+    def probe_url(self) -> str:
+        return f"{self.site_root}/basicspacedata/query/class/gp/NORAD_CAT_ID/25544/format/json"
+
     async def fetch(self) -> List[Dict[str, Any]]:
         from app.core.config import settings
 
@@ -46,13 +66,13 @@ class SpaceTrackTLECollector(BaseCollector):
                     "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
                     "Accept": "application/json, text/html, */*",
                     "Accept-Language": "en-US,en;q=0.9",
-                    "Referer": "https://www.space-track.org/",
+                    "Referer": f"{self.site_root}/",
                 },
             ) as client:
-                await client.get("https://www.space-track.org/")
+                await client.get(f"{self.site_root}/")
 
                 login_response = await client.post(
-                    "https://www.space-track.org/ajaxauth/login",
+                    self.login_url,
                     data={
                         "identity": username,
                         "password": password,
@@ -68,7 +88,7 @@ class SpaceTrackTLECollector(BaseCollector):
                         timeout=120.0,
                         follow_redirects=True,
                     ) as alt_client:
-                        await alt_client.get("https://www.space-track.org/")
+                        await alt_client.get(f"{self.site_root}/")
 
                         form_data = {
                             "username": username,
@@ -76,7 +96,7 @@ class SpaceTrackTLECollector(BaseCollector):
                             "query": "class/gp/NORAD_CAT_ID/25544/format/json",
                         }
                         alt_login = await alt_client.post(
-                            "https://www.space-track.org/ajaxauth/login",
+                            self.login_url,
                             data={
                                 "identity": username,
                                 "password": password,
@@ -85,9 +105,7 @@ class SpaceTrackTLECollector(BaseCollector):
                         print(f"SPACETRACK: Alt login status: {alt_login.status_code}")
 
                         if alt_login.status_code == 200:
-                            tle_response = await alt_client.get(
-                                "https://www.space-track.org/basicspacedata/query/class/gp/NORAD_CAT_ID/25544/format/json"
-                            )
+                            tle_response = await alt_client.get(self.probe_url)
                             if tle_response.status_code == 200:
                                 data = tle_response.json()
                                 print(f"SPACETRACK: Received {len(data)} records via alt method")
@@ -97,9 +115,7 @@ class SpaceTrackTLECollector(BaseCollector):
                     print(f"SPACETRACK: Login failed, using sample data")
                     return self._get_sample_data()
 
-                tle_response = await client.get(
-                    "https://www.space-track.org/basicspacedata/query/class/gp/NORAD_CAT_ID/25544/format/json"
-                )
+                tle_response = await client.get(self.probe_url)
                 print(f"SPACETRACK: TLE query status: {tle_response.status_code}")
 
                 if tle_response.status_code != 200:
@@ -126,11 +142,11 @@ class SpaceTrackTLECollector(BaseCollector):
                 },
             ) as client:
                 # First, visit the main page to get any cookies
-                await client.get("https://www.space-track.org/")
+                await client.get(f"{self.site_root}/")
 
                 # Login to get session cookie
                 login_response = await client.post(
-                    "https://www.space-track.org/ajaxauth/login",
+                    self.login_url,
                     data={
                         "identity": username,
                         "password": password,
@@ -145,13 +161,7 @@ class SpaceTrackTLECollector(BaseCollector):
                     return self._get_sample_data()
 
                 # Query for TLE data (get first 1000 satellites)
-                tle_response = await client.get(
-                    "https://www.space-track.org/basicspacedata/query"
-                    "/class/gp"
-                    "/orderby/EPOCH%20desc"
-                    "/limit/1000"
-                    "/format/json"
-                )
+                tle_response = await client.get(self.query_url)
                 print(f"SPACETRACK: TLE query status: {tle_response.status_code}")
 
                 if tle_response.status_code != 200:
@@ -169,25 +179,41 @@ class SpaceTrackTLECollector(BaseCollector):
         """Transform TLE data to internal format"""
         transformed = []
         for item in raw_data:
+            tle_line1, tle_line2 = build_tle_lines_from_elements(
+                norad_cat_id=item.get("NORAD_CAT_ID"),
+                epoch=item.get("EPOCH"),
+                inclination=item.get("INCLINATION"),
+                raan=item.get("RAAN"),
+                eccentricity=item.get("ECCENTRICITY"),
+                arg_of_perigee=item.get("ARG_OF_PERIGEE"),
+                mean_anomaly=item.get("MEAN_ANOMALY"),
+                mean_motion=item.get("MEAN_MOTION"),
+            )
             transformed.append(
                 {
                     "name": item.get("OBJECT_NAME", "Unknown"),
-                    "norad_cat_id": item.get("NORAD_CAT_ID"),
-                    "international_designator": item.get("INTL_DESIGNATOR"),
-                    "epoch": item.get("EPOCH"),
-                    "mean_motion": item.get("MEAN_MOTION"),
-                    "eccentricity": item.get("ECCENTRICITY"),
-                    "inclination": item.get("INCLINATION"),
-                    "raan": item.get("RAAN"),
-                    "arg_of_perigee": item.get("ARG_OF_PERIGEE"),
-                    "mean_anomaly": item.get("MEAN_ANOMALY"),
-                    "ephemeris_type": item.get("EPHEMERIS_TYPE"),
-                    "classification_type": item.get("CLASSIFICATION_TYPE"),
-                    "element_set_no": item.get("ELEMENT_SET_NO"),
-                    "rev_at_epoch": item.get("REV_AT_EPOCH"),
-                    "bstar": item.get("BSTAR"),
-                    "mean_motion_dot": item.get("MEAN_MOTION_DOT"),
-                    "mean_motion_ddot": item.get("MEAN_MOTION_DDOT"),
+                    "reference_date": item.get("EPOCH", ""),
+                    "metadata": {
+                        "norad_cat_id": item.get("NORAD_CAT_ID"),
+                        "international_designator": item.get("INTL_DESIGNATOR"),
+                        "epoch": item.get("EPOCH"),
+                        "mean_motion": item.get("MEAN_MOTION"),
+                        "eccentricity": item.get("ECCENTRICITY"),
+                        "inclination": item.get("INCLINATION"),
+                        "raan": item.get("RAAN"),
+                        "arg_of_perigee": item.get("ARG_OF_PERIGEE"),
+                        "mean_anomaly": item.get("MEAN_ANOMALY"),
+                        "ephemeris_type": item.get("EPHEMERIS_TYPE"),
+                        "classification_type": item.get("CLASSIFICATION_TYPE"),
+                        "element_set_no": item.get("ELEMENT_SET_NO"),
+                        "rev_at_epoch": item.get("REV_AT_EPOCH"),
+                        "bstar": item.get("BSTAR"),
+                        "mean_motion_dot": item.get("MEAN_MOTION_DOT"),
+                        "mean_motion_ddot": item.get("MEAN_MOTION_DDOT"),
+                        # Prefer original lines from the source, but keep a backend-built pair as a stable fallback.
+                        "tle_line1": item.get("TLE_LINE1") or item.get("TLE1") or tle_line1,
+                        "tle_line2": item.get("TLE_LINE2") or item.get("TLE2") or tle_line2,
+                    },
                 }
             )
         return transformed

backend/app/services/collectors/telegeography.py

@@ -7,10 +7,11 @@ Uses Wayback Machine as backup data source since live data requires JavaScript r
 import json
 import re
 from typing import Dict, Any, List
-from datetime import datetime
+from datetime import UTC, datetime
 from bs4 import BeautifulSoup
 import httpx
 
+from app.core.data_sources import get_data_sources_config
 from app.services.collectors.base import BaseCollector
 
 
@@ -24,15 +25,17 @@ class TeleGeographyCableCollector(BaseCollector):
 
     async def fetch(self) -> List[Dict[str, Any]]:
         """Fetch submarine cable data from Wayback Machine"""
+        config = get_data_sources_config()
         # Try multiple data sources
         sources = [
-            # Wayback Machine archive of TeleGeography
-            "https://web.archive.org/web/2024/https://www.submarinecablemap.com/api/v3/cable",
-            # Alternative: Try scraping the page
-            "https://www.submarinecablemap.com",
+            self._resolved_url or "",
+            str(config.get_yaml_value("telegeography.archived_cable_url") or ""),
+            str(config.get_yaml_value("telegeography.live_map_url") or ""),
         ]
 
         for url in sources:
+            if not url:
+                continue
             try:
                 async with httpx.AsyncClient(timeout=60.0, follow_redirects=True) as client:
                     response = await client.get(url)
@@ -103,7 +106,7 @@ class TeleGeographyCableCollector(BaseCollector):
                         "capacity_tbps": item.get("capacity"),
                         "url": item.get("url"),
                     },
-                    "reference_date": datetime.utcnow().strftime("%Y-%m-%d"),
+                    "reference_date": datetime.now(UTC).strftime("%Y-%m-%d"),
                 }
                 result.append(entry)
             except (ValueError, TypeError, KeyError):
@@ -131,7 +134,7 @@ class TeleGeographyCableCollector(BaseCollector):
                     "owner": "Meta, Orange, Vodafone, etc.",
                     "status": "active",
                 },
-                "reference_date": datetime.utcnow().strftime("%Y-%m-%d"),
+                "reference_date": datetime.now(UTC).strftime("%Y-%m-%d"),
             },
             {
                 "source_id": "telegeo_sample_2",
@@ -147,7 +150,7 @@ class TeleGeographyCableCollector(BaseCollector):
                     "owner": "Alibaba, NEC",
                     "status": "planned",
                 },
-                "reference_date": datetime.utcnow().strftime("%Y-%m-%d"),
+                "reference_date": datetime.now(UTC).strftime("%Y-%m-%d"),
             },
         ]
 
@@ -161,7 +164,7 @@ class TeleGeographyLandingPointCollector(BaseCollector):
 
     async def fetch(self) -> List[Dict[str, Any]]:
         """Fetch landing point data from GitHub mirror"""
-        url = "https://raw.githubusercontent.com/lintaojlu/submarine_cable_information/main/landing_point.json"
+        url = self._resolved_url or ""
 
         async with httpx.AsyncClient(timeout=60.0) as client:
             response = await client.get(url)
@@ -187,7 +190,7 @@ class TeleGeographyLandingPointCollector(BaseCollector):
                         "cable_count": len(item.get("cables", [])),
                         "url": item.get("url"),
                     },
-                    "reference_date": datetime.utcnow().strftime("%Y-%m-%d"),
+                    "reference_date": datetime.now(UTC).strftime("%Y-%m-%d"),
                 }
                 result.append(entry)
             except (ValueError, TypeError, KeyError):
@@ -211,7 +214,7 @@ class TeleGeographyLandingPointCollector(BaseCollector):
                 "value": "",
                 "unit": "",
                 "metadata": {"note": "Sample data"},
-                "reference_date": datetime.utcnow().strftime("%Y-%m-%d"),
+                "reference_date": datetime.now(UTC).strftime("%Y-%m-%d"),
             },
         ]
 
@@ -225,7 +228,7 @@ class TeleGeographyCableSystemCollector(BaseCollector):
 
     async def fetch(self) -> List[Dict[str, Any]]:
         """Fetch cable system data"""
-        url = "https://raw.githubusercontent.com/lintaojlu/submarine_cable_information/main/cable.json"
+        url = self._resolved_url or ""
 
         async with httpx.AsyncClient(timeout=60.0) as client:
             response = await client.get(url)
@@ -258,7 +261,7 @@ class TeleGeographyCableSystemCollector(BaseCollector):
                         "investment": item.get("investment"),
                         "url": item.get("url"),
                     },
-                    "reference_date": datetime.utcnow().strftime("%Y-%m-%d"),
+                    "reference_date": datetime.now(UTC).strftime("%Y-%m-%d"),
                 }
                 result.append(entry)
             except (ValueError, TypeError, KeyError):
@@ -282,6 +285,6 @@ class TeleGeographyCableSystemCollector(BaseCollector):
                 "value": "5000",
                 "unit": "km",
                 "metadata": {"note": "Sample data"},
-                "reference_date": datetime.utcnow().strftime("%Y-%m-%d"),
+                "reference_date": datetime.now(UTC).strftime("%Y-%m-%d"),
             },
         ]

backend/app/services/collectors/top500.py

@@ -4,12 +4,13 @@ Collects data from TOP500 supercomputer rankings.
 https://top500.org/lists/top500/
 """
 
+import asyncio
 import re
 from typing import Dict, Any, List
-from datetime import datetime
 from bs4 import BeautifulSoup
 import httpx
 
+from app.core.data_sources import get_data_sources_config
 from app.services.collectors.base import BaseCollector
 
 
@@ -21,14 +22,110 @@ class TOP500Collector(BaseCollector):
     data_type = "supercomputer"
 
     async def fetch(self) -> List[Dict[str, Any]]:
-        """Fetch TOP500 data from website (scraping)"""
-        # Get the latest list page
-        url = "https://top500.org/lists/top500/list/2025/11/"
+        """Fetch TOP500 list data and enrich each row with detail-page metadata."""
+        url = self._resolved_url or ""
 
-        async with httpx.AsyncClient(timeout=60.0) as client:
+        async with httpx.AsyncClient(timeout=60.0, follow_redirects=True) as client:
             response = await client.get(url)
             response.raise_for_status()
-            return self.parse_response(response.text)
+            entries = self.parse_response(response.text)
+
+            semaphore = asyncio.Semaphore(8)
+
+            async def enrich(entry: Dict[str, Any]) -> Dict[str, Any]:
+                detail_url = entry.pop("_detail_url", "")
+                if not detail_url:
+                    return entry
+
+                async with semaphore:
+                    try:
+                        detail_response = await client.get(detail_url)
+                        detail_response.raise_for_status()
+                        entry["metadata"].update(self.parse_detail_response(detail_response.text))
+                    except Exception:
+                        entry["metadata"]["detail_fetch_failed"] = True
+                return entry
+
+            return await asyncio.gather(*(enrich(entry) for entry in entries))
+
+    def _extract_system_fields(self, system_cell) -> Dict[str, str]:
+        config = get_data_sources_config()
+        top500_base_url = config.get_yaml_value("top500.base_url") or "https://top500.org"
+        link = system_cell.find("a")
+        system_name = link.get_text(" ", strip=True) if link else system_cell.get_text(" ", strip=True)
+        detail_url = ""
+        if link and link.get("href"):
+            detail_url = f"{str(top500_base_url).rstrip('/')}{link.get('href')}"
+
+        manufacturer = ""
+        if link and link.next_sibling:
+            manufacturer = str(link.next_sibling).strip(" ,\n\t")
+
+        cell_text = system_cell.get_text("\n", strip=True)
+        lines = [line.strip(" ,") for line in cell_text.splitlines() if line.strip()]
+
+        site = ""
+        country = ""
+        if lines:
+            system_name = lines[0]
+        if len(lines) >= 3:
+            site = lines[-2]
+            country = lines[-1]
+        elif len(lines) == 2:
+            country = lines[-1]
+
+        if not manufacturer and len(lines) >= 2:
+            manufacturer = lines[1]
+
+        return {
+            "name": system_name,
+            "manufacturer": manufacturer,
+            "site": site,
+            "country": country,
+            "detail_url": detail_url,
+        }
+
+    def parse_detail_response(self, html: str) -> Dict[str, Any]:
+        soup = BeautifulSoup(html, "html.parser")
+        detail_table = soup.find("table", {"class": "table table-condensed"})
+        if not detail_table:
+            return {}
+
+        detail_map: Dict[str, Any] = {}
+        label_aliases = {
+            "Site": "site",
+            "Manufacturer": "manufacturer",
+            "Cores": "cores",
+            "Processor": "processor",
+            "Interconnect": "interconnect",
+            "Installation Year": "installation_year",
+            "Linpack Performance (Rmax)": "rmax",
+            "Theoretical Peak (Rpeak)": "rpeak",
+            "Nmax": "nmax",
+            "HPCG": "hpcg",
+            "Power": "power",
+            "Power Measurement Level": "power_measurement_level",
+            "Operating System": "operating_system",
+            "Compiler": "compiler",
+            "Math Library": "math_library",
+            "MPI": "mpi",
+        }
+
+        for row in detail_table.find_all("tr"):
+            header = row.find("th")
+            value_cell = row.find("td")
+            if not header or not value_cell:
+                continue
+
+            label = header.get_text(" ", strip=True).rstrip(":")
+            key = label_aliases.get(label)
+            if not key:
+                continue
+
+            value = value_cell.get_text(" ", strip=True)
+            detail_map[key] = value
+
+        return detail_map
 
     def parse_response(self, html: str) -> List[Dict[str, Any]]:
         """Parse TOP500 HTML response"""
@@ -36,27 +133,26 @@ class TOP500Collector(BaseCollector):
         soup = BeautifulSoup(html, "html.parser")
 
         # Find the table with TOP500 data
-        table = soup.find("table", {"class": "top500-table"})
-        if not table:
-            # Try alternative table selector
-            table = soup.find("table", {"id": "top500"})
+        table = None
+        for candidate in soup.find_all("table"):
+            header_cells = [
+                cell.get_text(" ", strip=True) for cell in candidate.select("thead th")
+            ]
+            normalized_headers = [header.lower() for header in header_cells]
+            if (
+                "rank" in normalized_headers
+                and "system" in normalized_headers
+                and any("cores" in header for header in normalized_headers)
+                and any("rmax" in header for header in normalized_headers)
+            ):
+                table = candidate
+                break
 
         if not table:
-            # Try to find any table with rank data
-            tables = soup.find_all("table")
-            for t in tables:
-                if t.find(string=re.compile(r"Rank.*System.*Cores.*Rmax", re.I)):
-                    table = t
-                    break
-
-        if not table:
-            # Fallback: try to extract data from any table
-            tables = soup.find_all("table")
-            if tables:
-                table = tables[0]
+            table = soup.find("table", {"class": "top500-table"}) or soup.find("table", {"id": "top500"})
 
         if table:
-            rows = table.find_all("tr")
+            rows = table.select("tr")
             for row in rows[1:]:  # Skip header row
                 cells = row.find_all(["td", "th"])
                 if len(cells) >= 6:
@@ -68,43 +164,26 @@ class TOP500Collector(BaseCollector):
 
                         rank = int(rank_text)
 
-                        # System name (may contain link)
                         system_cell = cells[1]
-                        system_name = system_cell.get_text(strip=True)
-                        # Try to get full name from link title or data attribute
-                        link = system_cell.find("a")
-                        if link and link.get("title"):
-                            system_name = link.get("title")
+                        system_fields = self._extract_system_fields(system_cell)
+                        system_name = system_fields["name"]
+                        manufacturer = system_fields["manufacturer"]
+                        site = system_fields["site"]
+                        country = system_fields["country"]
+                        detail_url = system_fields["detail_url"]
 
-                        # Country
-                        country_cell = cells[2]
-                        country = country_cell.get_text(strip=True)
-                        # Try to get country from data attribute or image alt
-                        img = country_cell.find("img")
-                        if img and img.get("alt"):
-                            country = img.get("alt")
-
-                        # Extract location (city)
                         city = ""
-                        location_text = country_cell.get_text(strip=True)
-                        if "(" in location_text and ")" in location_text:
-                            city = location_text.split("(")[0].strip()
+                        cores = cells[2].get_text(strip=True).replace(",", "")
 
-                        # Cores
-                        cores = cells[3].get_text(strip=True).replace(",", "")
-
-                        # Rmax
-                        rmax_text = cells[4].get_text(strip=True)
+                        rmax_text = cells[3].get_text(strip=True)
                         rmax = self._parse_performance(rmax_text)
 
-                        # Rpeak
-                        rpeak_text = cells[5].get_text(strip=True)
+                        rpeak_text = cells[4].get_text(strip=True)
                         rpeak = self._parse_performance(rpeak_text)
 
-                        # Power (optional)
                         power = ""
-                        if len(cells) >= 7:
-                            power = cells[6].get_text(strip=True)
+                        if len(cells) >= 6:
+                            power = cells[5].get_text(strip=True).replace(",", "")
 
                         entry = {
                             "source_id": f"top500_{rank}",
@@ -117,10 +196,14 @@ class TOP500Collector(BaseCollector):
                             "unit": "PFlop/s",
                             "metadata": {
                                 "rank": rank,
-                                "r_peak": rpeak,
-                                "power": power,
                                 "cores": cores,
+                                "rmax": rmax_text,
+                                "rpeak": rpeak_text,
+                                "power": power,
+                                "manufacturer": manufacturer,
+                                "site": site,
                             },
+                            "_detail_url": detail_url,
                             "reference_date": "2025-11-01",
                         }
                         data.append(entry)
@@ -184,10 +267,15 @@ class TOP500Collector(BaseCollector):
                 "unit": "PFlop/s",
                 "metadata": {
                     "rank": 1,
-                    "r_peak": 2746.38,
-                    "power": 29581,
-                    "cores": 11039616,
+                    "cores": "11039616",
+                    "rmax": "1742.00",
+                    "rpeak": "2746.38",
+                    "power": "29581",
                     "manufacturer": "HPE",
+                    "site": "DOE/NNSA/LLNL",
+                    "processor": "AMD 4th Gen EPYC 24C 1.8GHz",
+                    "interconnect": "Slingshot-11",
+                    "installation_year": "2025",
                 },
                 "reference_date": "2025-11-01",
             },
@@ -202,10 +290,12 @@ class TOP500Collector(BaseCollector):
                 "unit": "PFlop/s",
                 "metadata": {
                     "rank": 2,
-                    "r_peak": 2055.72,
-                    "power": 24607,
-                    "cores": 9066176,
+                    "cores": "9066176",
+                    "rmax": "1353.00",
+                    "rpeak": "2055.72",
+                    "power": "24607",
                     "manufacturer": "HPE",
+                    "site": "DOE/SC/Oak Ridge National Laboratory",
                 },
                 "reference_date": "2025-11-01",
             },
@@ -220,9 +310,10 @@ class TOP500Collector(BaseCollector):
                 "unit": "PFlop/s",
                 "metadata": {
                     "rank": 3,
-                    "r_peak": 1980.01,
-                    "power": 38698,
-                    "cores": 9264128,
+                    "cores": "9264128",
+                    "rmax": "1012.00",
+                    "rpeak": "1980.01",
+                    "power": "38698",
                     "manufacturer": "Intel",
                 },
                 "reference_date": "2025-11-01",

backend/app/services/playground_chat_service.py

@@ -0,0 +1,694 @@
+from __future__ import annotations
+
+import asyncio
+from collections.abc import Sequence
+from datetime import UTC, datetime
+from time import perf_counter
+from uuid import uuid4
+
+from fastapi import HTTPException, status
+from sqlalchemy import func, select, update
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.db.session import async_session_factory
+from app.models.playground_message import PlaygroundMessage
+from app.models.playground_session import PlaygroundSession
+from app.schemas.ai import (
+    PlaygroundMessageEditRequest,
+    PlaygroundMessageActionResponse,
+    PlaygroundMessageCreateRequest,
+    PlaygroundMessageRecord,
+    PlaygroundMessageResendRequest,
+    PlaygroundMessageStopRequest,
+    PlaygroundSessionResponse,
+    PlaygroundSessionState,
+    PlaygroundSessionUpsertRequest,
+    PlaygroundThreadResponse,
+    SituationalAnalysisRequest,
+)
+from app.services.ai_client import AIProviderClient
+from app.services.playground_session_store import _to_response as session_to_response
+from app.services.playground_session_store import upsert_playground_session
+
+STREAM_CHUNK_SIZE = 24
+STREAM_INTERVAL_SECONDS = 0.08
+THINKING_PREVIEW_SECONDS = 2.6
+
+
+class _ActiveRun:
+    def __init__(self, task: asyncio.Task[None]) -> None:
+        self.task = task
+        self.stop_requested = asyncio.Event()
+
+
+_ACTIVE_RUNS: dict[str, _ActiveRun] = {}
+
+
+async def _get_session_by_key(
+    db: AsyncSession,
+    *,
+    user_id: int,
+    session_key: str,
+) -> PlaygroundSession | None:
+    result = await db.execute(
+        select(PlaygroundSession).where(
+            PlaygroundSession.user_id == user_id,
+            PlaygroundSession.session_key == session_key,
+        )
+    )
+    return result.scalar_one_or_none()
+
+
+async def _require_session(
+    db: AsyncSession,
+    *,
+    user_id: int,
+    session_key: str,
+) -> PlaygroundSession:
+    session = await _get_session_by_key(db, user_id=user_id, session_key=session_key)
+    if session is None:
+        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Playground session not found")
+    return session
+
+
+async def _require_visible_message(
+    db: AsyncSession,
+    *,
+    user_id: int,
+    public_id: str,
+    role: str | None = None,
+) -> PlaygroundMessage:
+    conditions = [
+        PlaygroundMessage.user_id == user_id,
+        PlaygroundMessage.public_id == public_id,
+        PlaygroundMessage.is_visible.is_(True),
+    ]
+    if role is not None:
+        conditions.append(PlaygroundMessage.role == role)
+
+    result = await db.execute(select(PlaygroundMessage).where(*conditions))
+    message = result.scalar_one_or_none()
+    if message is None:
+        detail = "User message not found" if role == "user" else "Playground message not found"
+        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail=detail)
+    return message
+
+
+def _message_to_record(message: PlaygroundMessage, parent_public_id: str | None = None) -> PlaygroundMessageRecord:
+    return PlaygroundMessageRecord(
+        id=message.public_id,
+        role=message.role,
+        kind=message.kind,
+        status=message.status,
+        title=message.title,
+        content=message.content or "",
+        thinking_content=message.thinking_content or "",
+        meta=list(message.meta or []),
+        markdown=message.role != "system",
+        provider=message.provider,
+        model=message.model,
+        request_id=message.request_id,
+        raw_response=dict(message.raw_response or {}),
+        content_blocks=list(message.content_blocks or []),
+        text_blocks=list(message.text_blocks or []),
+        thinking_blocks=list(message.thinking_blocks or []),
+        parent_message_id=parent_public_id,
+        created_at=message.created_at.isoformat(),
+        updated_at=message.updated_at.isoformat(),
+    )
+
+
+async def _ensure_session(
+    db: AsyncSession,
+    *,
+    user_id: int,
+    session_key: str,
+    title: str,
+    state: PlaygroundSessionState | None = None,
+) -> PlaygroundSession:
+    result = await db.execute(
+        select(PlaygroundSession).where(
+            PlaygroundSession.user_id == user_id,
+            PlaygroundSession.session_key == session_key,
+        )
+    )
+    session = result.scalar_one_or_none()
+    if session is not None:
+        if title:
+            session.title = title[:200]
+        if state is not None:
+            session.state = state.model_dump(mode="json")
+        await db.flush()
+        await db.refresh(session)
+        return session
+
+    payload = PlaygroundSessionUpsertRequest(
+        session_key=session_key,
+        title=title[:200],
+        state=state or PlaygroundSessionState(title=title[:200]),
+    )
+    await upsert_playground_session(db, user_id=user_id, payload=payload)
+    result = await db.execute(
+        select(PlaygroundSession).where(
+            PlaygroundSession.user_id == user_id,
+            PlaygroundSession.session_key == session_key,
+        )
+    )
+    return result.scalar_one()
+
+
+async def _list_visible_messages(
+    db: AsyncSession,
+    *,
+    session_id: int,
+) -> list[PlaygroundMessage]:
+    result = await db.execute(
+        select(PlaygroundMessage)
+        .where(
+            PlaygroundMessage.session_id == session_id,
+            PlaygroundMessage.is_visible.is_(True),
+        )
+        .order_by(PlaygroundMessage.sort_order.asc(), PlaygroundMessage.id.asc())
+    )
+    return list(result.scalars().all())
+
+
+async def _build_thread_response(
+    db: AsyncSession,
+    *,
+    session: PlaygroundSession,
+) -> PlaygroundThreadResponse:
+    messages = await _list_visible_messages(db, session_id=session.id)
+    id_map = {item.id: item.public_id for item in messages}
+    return PlaygroundThreadResponse(
+        session=session_to_response(session),
+        messages=[_message_to_record(item, id_map.get(item.parent_message_id)) for item in messages],
+    )
+
+
+async def get_thread(
+    db: AsyncSession,
+    *,
+    user_id: int,
+    session_key: str,
+) -> PlaygroundThreadResponse | None:
+    session = await _get_session_by_key(db, user_id=user_id, session_key=session_key)
+    if session is None:
+        return None
+    return await _build_thread_response(db, session=session)
+
+
+async def _build_action_response(
+    db: AsyncSession,
+    *,
+    session: PlaygroundSession,
+    active_message_id: str | None = None,
+) -> PlaygroundMessageActionResponse:
+    thread = await _build_thread_response(db, session=session)
+    return PlaygroundMessageActionResponse(
+        session=thread.session,
+        messages=thread.messages,
+        active_message_id=active_message_id,
+    )
+
+
+def _collect_constraints(raw_constraints: str) -> list[str]:
+    return [item.strip() for item in raw_constraints.split("\n") if item.strip()]
+
+
+async def _next_sort_order(db: AsyncSession, session_id: int) -> int:
+    result = await db.execute(
+        select(func.max(PlaygroundMessage.sort_order)).where(PlaygroundMessage.session_id == session_id)
+    )
+    current = result.scalar_one_or_none()
+    return int(current or 0)
+
+
+async def _set_session_state(
+    db: AsyncSession,
+    *,
+    session: PlaygroundSession,
+    payload: PlaygroundMessageCreateRequest,
+) -> PlaygroundSession:
+    session.state = PlaygroundSessionState(
+        messages=[],
+        selectedPresetKey=payload.selected_preset_key,
+        title=payload.title,
+        objective=payload.objective,
+        constraints=payload.constraints,
+        inputValue="",
+        analysis=None,
+        latestAnalysisMessageId=None,
+        analysisMeta={},
+        helpExpanded=payload.help_expanded,
+    ).model_dump(mode="json")
+    session.title = payload.title[:200]
+    await db.flush()
+    await db.refresh(session)
+    return session
+
+
+def _spawn_assistant_run(
+    *,
+    user_id: int,
+    session_id: int,
+    session_key: str,
+    user_message_id: int,
+    assistant_message_id: int,
+    assistant_public_id: str,
+    payload: PlaygroundMessageCreateRequest,
+    provider_client: AIProviderClient,
+) -> None:
+    task = asyncio.create_task(
+        _run_assistant_message(
+            user_id=user_id,
+            session_id=session_id,
+            session_key=session_key,
+            user_message_id=user_message_id,
+            assistant_message_id=assistant_message_id,
+            payload=payload,
+            provider_client=provider_client,
+        )
+    )
+    _ACTIVE_RUNS[assistant_public_id] = _ActiveRun(task)
+
+
+async def create_turn(
+    db: AsyncSession,
+    *,
+    user_id: int,
+    payload: PlaygroundMessageCreateRequest,
+    provider_client: AIProviderClient,
+) -> PlaygroundMessageActionResponse:
+    session = await _ensure_session(
+        db,
+        user_id=user_id,
+        session_key=payload.session_key,
+        title=payload.title,
+        state=PlaygroundSessionState(
+            selectedPresetKey=payload.selected_preset_key,
+            title=payload.title,
+            objective=payload.objective,
+            constraints=payload.constraints,
+            inputValue="",
+            helpExpanded=payload.help_expanded,
+        ),
+    )
+    session = await _set_session_state(db, session=session, payload=payload)
+    base_order = await _next_sort_order(db, session.id)
+
+    user_message = PlaygroundMessage(
+        public_id=uuid4().hex,
+        session_id=session.id,
+        user_id=user_id,
+        role="user",
+        kind="message",
+        status="done",
+        title=payload.selected_preset_key,
+        content=payload.input,
+        meta=[payload.title],
+        sort_order=base_order + 10,
+    )
+    assistant_message = PlaygroundMessage(
+        public_id=uuid4().hex,
+        session_id=session.id,
+        user_id=user_id,
+        parent_message_id=None,
+        role="assistant",
+        kind="thinking",
+        status="pending",
+        title="AI 回应",
+        content="",
+        thinking_content="",
+        meta=[],
+        sort_order=base_order + 20,
+    )
+    db.add(user_message)
+    await db.flush()
+    assistant_message.parent_message_id = user_message.id
+    db.add(assistant_message)
+    await db.flush()
+    await db.refresh(user_message)
+    await db.refresh(assistant_message)
+    await db.commit()
+    await db.refresh(session)
+    await db.refresh(user_message)
+    await db.refresh(assistant_message)
+
+    _spawn_assistant_run(
+        user_id=user_id,
+        session_id=session.id,
+        session_key=payload.session_key,
+        user_message_id=user_message.id,
+        assistant_message_id=assistant_message.id,
+        assistant_public_id=assistant_message.public_id,
+        payload=payload,
+        provider_client=provider_client,
+    )
+
+    return await _build_action_response(
+        db,
+        session=session,
+        active_message_id=assistant_message.public_id,
+    )
+
+
+async def _create_assistant_retry_turn(
+    db: AsyncSession,
+    *,
+    user_id: int,
+    session: PlaygroundSession,
+    user_message: PlaygroundMessage,
+    payload: PlaygroundMessageCreateRequest,
+    provider_client: AIProviderClient,
+) -> PlaygroundMessageActionResponse:
+    base_order = await _next_sort_order(db, session.id)
+    assistant_message = PlaygroundMessage(
+        public_id=uuid4().hex,
+        session_id=session.id,
+        user_id=user_id,
+        parent_message_id=user_message.id,
+        role="assistant",
+        kind="thinking",
+        status="pending",
+        title="AI 回应",
+        content="",
+        thinking_content="",
+        meta=[],
+        sort_order=base_order + 10,
+    )
+    db.add(assistant_message)
+    await db.flush()
+    await db.commit()
+    await db.refresh(session)
+    await db.refresh(assistant_message)
+
+    _spawn_assistant_run(
+        user_id=user_id,
+        session_id=session.id,
+        session_key=payload.session_key,
+        user_message_id=user_message.id,
+        assistant_message_id=assistant_message.id,
+        assistant_public_id=assistant_message.public_id,
+        payload=payload,
+        provider_client=provider_client,
+    )
+
+    return await _build_action_response(
+        db,
+        session=session,
+        active_message_id=assistant_message.public_id,
+    )
+
+
+async def stop_message(
+    db: AsyncSession,
+    *,
+    user_id: int,
+    payload: PlaygroundMessageStopRequest,
+) -> PlaygroundMessageActionResponse:
+    session = await _require_session(db, user_id=user_id, session_key=payload.session_key)
+    message = await _require_visible_message(db, user_id=user_id, public_id=payload.message_id)
+
+    if message.status not in {"pending", "thinking", "answering"}:
+        return await _build_action_response(db, session=session)
+
+    active_run = _ACTIVE_RUNS.get(message.public_id)
+    if active_run is not None:
+        active_run.stop_requested.set()
+        active_run.task.cancel()
+
+    message.status = "stopped"
+    if "已手动停止生成" not in (message.meta or []):
+        message.meta = [*(message.meta or []), "已手动停止生成"]
+    await db.flush()
+    await db.commit()
+    await db.refresh(message)
+
+    return await _build_action_response(db, session=session)
+
+
+async def resend_turn(
+    db: AsyncSession,
+    *,
+    user_id: int,
+    payload: PlaygroundMessageResendRequest,
+    provider_client: AIProviderClient,
+) -> PlaygroundMessageActionResponse:
+    session = await _require_session(db, user_id=user_id, session_key=payload.session_key)
+    user_message = await _require_visible_message(
+        db,
+        user_id=user_id,
+        public_id=payload.user_message_id,
+        role="user",
+    )
+
+    later_messages = await db.execute(
+        select(PlaygroundMessage).where(
+            PlaygroundMessage.session_id == session.id,
+            PlaygroundMessage.sort_order > user_message.sort_order,
+            PlaygroundMessage.is_visible.is_(True),
+        )
+    )
+    for item in later_messages.scalars().all():
+        item.is_visible = False
+        if item.status in {"pending", "thinking", "answering"}:
+            active_run = _ACTIVE_RUNS.get(item.public_id)
+            if active_run is not None:
+                active_run.stop_requested.set()
+                active_run.task.cancel()
+    await db.flush()
+    await db.commit()
+
+    session_state = PlaygroundSessionState.model_validate(session.state or {})
+    create_payload = PlaygroundMessageCreateRequest(
+        session_key=payload.session_key,
+        title=session_state.title or session.title,
+        objective=session_state.objective or "继续当前对话",
+        constraints=session_state.constraints or "",
+        input=user_message.content,
+        selected_preset_key=session_state.selectedPresetKey or "bgp-brief",
+        help_expanded=session_state.helpExpanded,
+    )
+    return await _create_assistant_retry_turn(
+        db,
+        user_id=user_id,
+        session=session,
+        user_message=user_message,
+        payload=create_payload,
+        provider_client=provider_client,
+    )
+
+
+async def edit_user_message(
+    db: AsyncSession,
+    *,
+    user_id: int,
+    payload: PlaygroundMessageEditRequest,
+) -> PlaygroundMessageActionResponse:
+    session = await _require_session(db, user_id=user_id, session_key=payload.session_key)
+    user_message = await _require_visible_message(
+        db,
+        user_id=user_id,
+        public_id=payload.user_message_id,
+        role="user",
+    )
+
+    user_message.content = payload.content.strip()
+    await db.flush()
+    await db.commit()
+    await db.refresh(user_message)
+
+    return await _build_action_response(db, session=session)
+
+
+async def _append_meta_if_missing(db: AsyncSession, message_id: int, meta_line: str) -> None:
+    result = await db.execute(select(PlaygroundMessage).where(PlaygroundMessage.id == message_id))
+    message = result.scalar_one_or_none()
+    if message is None:
+        return
+    if meta_line not in (message.meta or []):
+        message.meta = [*(message.meta or []), meta_line]
+        await db.flush()
+
+
+async def _should_stop(message_public_id: str) -> bool:
+    active_run = _ACTIVE_RUNS.get(message_public_id)
+    return active_run.stop_requested.is_set() if active_run is not None else False
+
+
+async def _mark_message_state(
+    db: AsyncSession,
+    *,
+    message_id: int,
+    **updates,
+) -> PlaygroundMessage:
+    result = await db.execute(select(PlaygroundMessage).where(PlaygroundMessage.id == message_id))
+    message = result.scalar_one()
+    for key, value in updates.items():
+        setattr(message, key, value)
+    await db.flush()
+    await db.refresh(message)
+    return message
+
+
+def _build_conversation_history(messages: Sequence[PlaygroundMessage], current_user_message_id: int) -> list[dict]:
+    history: list[dict] = []
+    for item in messages:
+        if item.id >= current_user_message_id:
+            break
+        if item.role == "system":
+            continue
+        history.append(
+            {
+                "role": item.role,
+                "kind": item.kind or "message",
+                "title": item.title,
+                "content": item.content or "",
+            }
+        )
+    return history[-8:]
+
+
+async def _run_assistant_message(
+    *,
+    user_id: int,
+    session_id: int,
+    session_key: str,
+    user_message_id: int,
+    assistant_message_id: int,
+    payload: PlaygroundMessageCreateRequest,
+    provider_client: AIProviderClient,
+) -> None:
+    request_id = str(uuid4())
+    started_at = perf_counter()
+    assistant_public_id: str | None = None
+    try:
+        async with async_session_factory() as db:
+            session = await db.get(PlaygroundSession, session_id)
+            user_message = await db.get(PlaygroundMessage, user_message_id)
+            assistant_message = await db.get(PlaygroundMessage, assistant_message_id)
+            if session is None or user_message is None or assistant_message is None:
+                return
+            assistant_public_id = assistant_message.public_id
+
+            visible_messages = await _list_visible_messages(db, session_id=session_id)
+            conversation_history = _build_conversation_history(visible_messages, user_message_id)
+
+            request_payload = SituationalAnalysisRequest(
+                title=payload.title,
+                objective=payload.objective,
+                observations=[item.strip() for item in payload.input.split("\n") if item.strip()],
+                constraints=_collect_constraints(payload.constraints),
+                context={
+                    "source": "playground",
+                    "preset": payload.selected_preset_key,
+                    "conversation_history": conversation_history,
+                    "history_size": len(conversation_history),
+                },
+                thinking={"type": "enabled"},
+            )
+
+        analysis = await provider_client.analyze(request_payload, request_id=request_id)
+
+        async with async_session_factory() as db:
+            assistant_message = await _mark_message_state(
+                db,
+                message_id=assistant_message_id,
+                status="thinking" if analysis.thinking_blocks else "answering",
+                title=f"{analysis.provider} / {analysis.model}",
+                provider=analysis.provider,
+                model=analysis.model,
+                request_id=request_id,
+                raw_response=analysis.raw_response,
+                content_blocks=[item.model_dump(mode="json") for item in analysis.content_blocks],
+                text_blocks=analysis.text_blocks,
+                thinking_blocks=analysis.thinking_blocks,
+                thinking_content="\n\n".join(analysis.thinking_blocks).strip(),
+            )
+            session = await db.get(PlaygroundSession, session_id)
+            if session is not None:
+                session_state = PlaygroundSessionState.model_validate(session.state or {})
+                session.state = session_state.model_copy(
+                    update={
+                        "latestAnalysisMessageId": assistant_message.public_id,
+                        "analysis": analysis.model_dump(mode="json"),
+                    }
+                ).model_dump(mode="json")
+                await db.flush()
+            await db.commit()
+
+        if assistant_public_id and analysis.thinking_blocks:
+            await asyncio.sleep(THINKING_PREVIEW_SECONDS)
+            if await _should_stop(assistant_public_id):
+                return
+
+        content = analysis.content or ""
+        cursor = 0
+        while cursor < len(content):
+            if assistant_public_id and await _should_stop(assistant_public_id):
+                return
+            cursor = min(len(content), cursor + STREAM_CHUNK_SIZE)
+            async with async_session_factory() as db:
+                await _mark_message_state(
+                    db,
+                    message_id=assistant_message_id,
+                    status="answering",
+                    content=content[:cursor],
+                )
+                await db.commit()
+            await asyncio.sleep(STREAM_INTERVAL_SECONDS)
+
+        duration_ms = round((perf_counter() - started_at) * 1000)
+        async with async_session_factory() as db:
+            assistant_message = await _mark_message_state(
+                db,
+                message_id=assistant_message_id,
+                status="done",
+                content=content,
+                meta=[
+                    f"Request ID: {request_id}",
+                    f"耗时: {duration_ms} ms",
+                    f"完成时间: {datetime.now(UTC).astimezone().isoformat()}",
+                ],
+            )
+            session = await db.get(PlaygroundSession, session_id)
+            if session is not None:
+                session_state = PlaygroundSessionState.model_validate(session.state or {})
+                session.state = session_state.model_copy(
+                    update={
+                        "latestAnalysisMessageId": assistant_message.public_id,
+                        "analysis": analysis.model_dump(mode="json"),
+                        "analysisMeta": {
+                            "requestId": request_id,
+                            "durationMs": duration_ms,
+                            "completedAt": datetime.now().isoformat(),
+                        },
+                    }
+                ).model_dump(mode="json")
+                await db.flush()
+            await db.commit()
+    except asyncio.CancelledError:
+        async with async_session_factory() as db:
+            result = await db.execute(select(PlaygroundMessage).where(PlaygroundMessage.id == assistant_message_id))
+            message = result.scalar_one_or_none()
+            if message is not None and message.status in {"pending", "thinking", "answering"}:
+                message.status = "stopped"
+                if "已手动停止生成" not in (message.meta or []):
+                    message.meta = [*(message.meta or []), "已手动停止生成"]
+                await db.flush()
+                await db.commit()
+        raise
+    except Exception as exc:
+        async with async_session_factory() as db:
+            result = await db.execute(select(PlaygroundMessage).where(PlaygroundMessage.id == assistant_message_id))
+            message = result.scalar_one_or_none()
+            if message is not None:
+                message.status = "error"
+                message.content = message.content or "分析失败，请检查 AI Provider 配置或稍后再试。"
+                message.meta = [*(message.meta or []), f"错误: {type(exc).__name__}"]
+                await db.flush()
+                await db.commit()
+    finally:
+        if assistant_public_id:
+            _ACTIVE_RUNS.pop(assistant_public_id, None)

backend/app/services/playground_session_store.py

@@ -0,0 +1,72 @@
+from __future__ import annotations
+
+from sqlalchemy import select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.models.playground_session import PlaygroundSession
+from app.schemas.ai import (
+    PlaygroundSessionResponse,
+    PlaygroundSessionState,
+    PlaygroundSessionUpsertRequest,
+)
+
+
+def _to_response(record: PlaygroundSession) -> PlaygroundSessionResponse:
+    return PlaygroundSessionResponse(
+        id=str(record.id),
+        session_key=record.session_key,
+        title=record.title,
+        state=PlaygroundSessionState.model_validate(record.state or {}),
+        created_at=record.created_at.isoformat(),
+        updated_at=record.updated_at.isoformat(),
+    )
+
+
+async def get_playground_session(
+    db: AsyncSession,
+    *,
+    user_id: int,
+    session_key: str = "default",
+) -> PlaygroundSessionResponse | None:
+    result = await db.execute(
+        select(PlaygroundSession).where(
+            PlaygroundSession.user_id == user_id,
+            PlaygroundSession.session_key == session_key,
+        )
+    )
+    record = result.scalar_one_or_none()
+    if record is None:
+        return None
+    return _to_response(record)
+
+
+async def upsert_playground_session(
+    db: AsyncSession,
+    *,
+    user_id: int,
+    payload: PlaygroundSessionUpsertRequest,
+) -> PlaygroundSessionResponse:
+    result = await db.execute(
+        select(PlaygroundSession).where(
+            PlaygroundSession.user_id == user_id,
+            PlaygroundSession.session_key == payload.session_key,
+        )
+    )
+    record = result.scalar_one_or_none()
+    title = (payload.title or payload.state.title or "Playground 会话").strip()[:200] or "Playground 会话"
+
+    if record is None:
+        record = PlaygroundSession(
+            user_id=user_id,
+            session_key=payload.session_key,
+            title=title,
+            state=payload.state.model_dump(mode="json"),
+        )
+        db.add(record)
+    else:
+        record.title = title
+        record.state = payload.state.model_dump(mode="json")
+
+    await db.flush()
+    await db.refresh(record)
+    return _to_response(record)

backend/app/services/scheduler.py

@@ -1,152 +1,310 @@
-"""Task Scheduler for running collection jobs"""
+"""Task Scheduler for running collection jobs."""
 
 import asyncio
 import logging
-from datetime import datetime
-from typing import Dict, Any
+from datetime import UTC, datetime, timedelta
+from typing import Any, Dict, Optional
 
 from apscheduler.schedulers.asyncio import AsyncIOScheduler
 from apscheduler.triggers.interval import IntervalTrigger
-from sqlalchemy.ext.asyncio import AsyncSession
+from sqlalchemy import select
 
 from app.db.session import async_session_factory
+from app.core.time import to_iso8601_utc
+from app.models.datasource import DataSource
+from app.models.task import CollectionTask
 from app.services.collectors.registry import collector_registry
 
 logger = logging.getLogger(__name__)
 
 scheduler = AsyncIOScheduler()
+RUNNING_TASK_GUARD_TIMEOUT_MINUTES = 90
+RUNNING_COLLECTOR_TASKS: dict[str, asyncio.Task[Any]] = {}
 
 
-COLLECTOR_TO_ID = {
-    "top500": 1,
-    "epoch_ai_gpu": 2,
-    "huggingface_models": 3,
-    "huggingface_datasets": 4,
-    "huggingface_spaces": 5,
-    "peeringdb_ixp": 6,
-    "peeringdb_network": 7,
-    "peeringdb_facility": 8,
-    "telegeography_cables": 9,
-    "telegeography_landing": 10,
-    "telegeography_systems": 11,
-    "arcgis_cables": 15,
-    "arcgis_landing_points": 16,
-    "arcgis_cable_landing_relation": 17,
-    "fao_landing_points": 18,
-    "spacetrack_tle": 19,
-    "celestrak_tle": 20,
-}
+def _collector_task_name(collector_name: str) -> str:
+    return f"collector:{collector_name}"
+
+
+def get_running_collector_task(collector_name: str) -> asyncio.Task[Any] | None:
+    task = RUNNING_COLLECTOR_TASKS.get(collector_name)
+    if task is not None and not task.done():
+        return task
+
+    if task is not None and task.done():
+        RUNNING_COLLECTOR_TASKS.pop(collector_name, None)
+
+    target_name = _collector_task_name(collector_name)
+    for candidate in asyncio.all_tasks():
+        if candidate.done():
+            continue
+        if candidate.get_name() == target_name:
+            RUNNING_COLLECTOR_TASKS[collector_name] = candidate
+            return candidate
+
+    return None
+
+
+async def _update_next_run_at(datasource: DataSource, session) -> None:
+    job = scheduler.get_job(datasource.source)
+    datasource.next_run_at = job.next_run_time if job else None
+    await session.commit()
+
+
+async def _apply_datasource_schedule(datasource: DataSource, session) -> None:
+    collector = collector_registry.get(datasource.source)
+    if not collector:
+        logger.warning("Collector not found for datasource %s", datasource.source)
+        return
+
+    collector_registry.set_active(datasource.source, datasource.is_active)
+
+    existing_job = scheduler.get_job(datasource.source)
+    if existing_job:
+        scheduler.remove_job(datasource.source)
+
+    if datasource.is_active:
+        scheduler.add_job(
+            run_collector_task,
+            trigger=IntervalTrigger(minutes=max(1, datasource.frequency_minutes)),
+            id=datasource.source,
+            name=datasource.name,
+            replace_existing=True,
+            kwargs={"collector_name": datasource.source},
+        )
+        logger.info(
+            "Scheduled collector: %s (every %sm)",
+            datasource.source,
+            datasource.frequency_minutes,
+        )
+    else:
+        logger.info("Collector disabled: %s", datasource.source)
+
+    await _update_next_run_at(datasource, session)
 
 
 async def run_collector_task(collector_name: str):
-    """Run a single collector task"""
+    """Run a single collector task."""
     collector = collector_registry.get(collector_name)
     if not collector:
-        logger.error(f"Collector not found: {collector_name}")
+        logger.error("Collector not found: %s", collector_name)
         return
 
-    # Get the correct datasource_id
-    datasource_id = COLLECTOR_TO_ID.get(collector_name, 1)
+    async with async_session_factory() as db:
+        result = await db.execute(select(DataSource).where(DataSource.source == collector_name))
+        datasource = result.scalar_one_or_none()
+        if not datasource:
+            logger.error("Datasource not found for collector: %s", collector_name)
+            return
+
+        if not datasource.is_active:
+            logger.info("Skipping disabled collector: %s", collector_name)
+            return
+
+        running_result = await db.execute(
+            select(CollectionTask)
+            .where(
+                CollectionTask.datasource_id == datasource.id,
+                CollectionTask.status == "running",
+            )
+            .order_by(CollectionTask.started_at.desc(), CollectionTask.id.desc())
+            .limit(1)
+        )
+        existing_running = running_result.scalar_one_or_none()
+        if existing_running is not None:
+            now = datetime.now(UTC)
+            started_at = existing_running.started_at
+            if started_at is not None and started_at.tzinfo is None:
+                started_at = started_at.replace(tzinfo=UTC)
+
+            is_stale = (
+                started_at is not None
+                and (now - started_at) > timedelta(minutes=RUNNING_TASK_GUARD_TIMEOUT_MINUTES)
+            )
+            if not is_stale:
+                logger.warning(
+                    "Skipping collector %s trigger because task %s is already running",
+                    collector_name,
+                    existing_running.id,
+                )
+                return
+
+            existing_error = (existing_running.error_message or "").strip()
+            stale_reason = (
+                f"Marked failed automatically after stale running timeout "
+                f"({RUNNING_TASK_GUARD_TIMEOUT_MINUTES}m) in scheduler guard"
+            )
+            existing_running.status = "failed"
+            existing_running.phase = "failed"
+            existing_running.completed_at = now
+            existing_running.error_message = (
+                f"{existing_error}\n{stale_reason}".strip()
+                if existing_error
+                else stale_reason
+            )
+            await db.commit()
+            logger.warning(
+                "Marked stale running task %s as failed before rerun of %s",
+                existing_running.id,
+                collector_name,
+            )
+
+        try:
+            collector._datasource_id = datasource.id
+            logger.info("Running collector: %s (datasource_id=%s)", collector_name, datasource.id)
+            task_result = await collector.run(db)
+            datasource.last_run_at = datetime.now(UTC)
+            datasource.last_status = task_result.get("status")
+            await _update_next_run_at(datasource, db)
+            logger.info("Collector %s completed: %s", collector_name, task_result)
+        except asyncio.CancelledError:
+            datasource.last_run_at = datetime.now(UTC)
+            datasource.last_status = "cancelled"
+            await db.commit()
+            logger.warning("Collector %s cancelled by operator", collector_name)
+            raise
+        except Exception as exc:
+            datasource.last_run_at = datetime.now(UTC)
+            datasource.last_status = "failed"
+            await db.commit()
+            logger.exception("Collector %s failed: %s", collector_name, exc)
+
+
+async def cleanup_stale_running_tasks(max_age_hours: int = 2) -> int:
+    """Mark stale running tasks as failed after restarts or collector hangs."""
+    cutoff = datetime.now(UTC) - timedelta(hours=max_age_hours)
 
     async with async_session_factory() as db:
-        try:
-            # Set the datasource_id on the collector instance
-            collector._datasource_id = datasource_id
-
-            logger.info(f"Running collector: {collector_name} (datasource_id={datasource_id})")
-            result = await collector.run(db)
-            logger.info(f"Collector {collector_name} completed: {result}")
-        except Exception as e:
-            logger.error(f"Collector {collector_name} failed: {e}")
-
-
-def start_scheduler():
-    """Start the scheduler with all registered collectors"""
-    collectors = collector_registry.all()
-
-    for name, collector in collectors.items():
-        if collector_registry.is_active(name):
-            scheduler.add_job(
-                run_collector_task,
-                trigger=IntervalTrigger(hours=collector.frequency_hours),
-                id=name,
-                name=name,
-                replace_existing=True,
-                kwargs={"collector_name": name},
+        result = await db.execute(
+            select(CollectionTask).where(
+                CollectionTask.status == "running",
+                CollectionTask.started_at.is_not(None),
+                CollectionTask.started_at < cutoff,
             )
-            logger.info(f"Scheduled collector: {name} (every {collector.frequency_hours}h)")
+        )
+        stale_tasks = result.scalars().all()
 
-    scheduler.start()
-    logger.info("Scheduler started")
+        for task in stale_tasks:
+            task.status = "failed"
+            task.phase = "failed"
+            task.completed_at = datetime.now(UTC)
+            existing_error = (task.error_message or "").strip()
+            cleanup_error = "Marked failed automatically after stale running task cleanup"
+            task.error_message = f"{existing_error}\n{cleanup_error}".strip() if existing_error else cleanup_error
+
+        if stale_tasks:
+            await db.commit()
+            logger.warning("Cleaned up %s stale running collection task(s)", len(stale_tasks))
+
+        return len(stale_tasks)
 
 
-def stop_scheduler():
-    """Stop the scheduler"""
-    scheduler.shutdown()
-    logger.info("Scheduler stopped")
+def start_scheduler() -> None:
+    """Start the scheduler."""
+    if not scheduler.running:
+        scheduler.start()
+        logger.info("Scheduler started")
+
+
+def stop_scheduler() -> None:
+    """Stop the scheduler."""
+    if scheduler.running:
+        scheduler.shutdown(wait=False)
+        logger.info("Scheduler stopped")
+
+
+async def sync_scheduler_with_datasources() -> None:
+    """Synchronize scheduler jobs with datasource table."""
+    async with async_session_factory() as db:
+        result = await db.execute(select(DataSource).order_by(DataSource.id))
+        datasources = result.scalars().all()
+
+        configured_sources = {datasource.source for datasource in datasources}
+        for job in list(scheduler.get_jobs()):
+            if job.id not in configured_sources:
+                scheduler.remove_job(job.id)
+
+        for datasource in datasources:
+            await _apply_datasource_schedule(datasource, db)
+
+
+async def sync_datasource_job(datasource_id: int) -> bool:
+    """Synchronize a single datasource job after settings changes."""
+    async with async_session_factory() as db:
+        datasource = await db.get(DataSource, datasource_id)
+        if not datasource:
+            return False
+
+        await _apply_datasource_schedule(datasource, db)
+        return True
 
 
 def get_scheduler_jobs() -> list[Dict[str, Any]]:
-    """Get all scheduled jobs"""
+    """Get all scheduled jobs."""
     jobs = []
     for job in scheduler.get_jobs():
         jobs.append(
             {
                 "id": job.id,
                 "name": job.name,
-                "next_run_time": job.next_run_time.isoformat() if job.next_run_time else None,
+                "next_run_time": to_iso8601_utc(job.next_run_time),
                 "trigger": str(job.trigger),
             }
         )
     return jobs
 
 
-def add_job(collector_name: str, hours: int = 4):
-    """Add a new scheduled job"""
-    collector = collector_registry.get(collector_name)
-    if not collector:
-        raise ValueError(f"Collector not found: {collector_name}")
+async def get_latest_task_id_for_datasource(datasource_id: int) -> Optional[int]:
+    from app.models.task import CollectionTask
 
-    scheduler.add_job(
-        run_collector_task,
-        trigger=IntervalTrigger(hours=hours),
-        id=collector_name,
-        name=collector_name,
-        replace_existing=True,
-        kwargs={"collector_name": collector_name},
-    )
-    logger.info(f"Added scheduled job: {collector_name} (every {hours}h)")
-
-
-def remove_job(collector_name: str):
-    """Remove a scheduled job"""
-    scheduler.remove_job(collector_name)
-    logger.info(f"Removed scheduled job: {collector_name}")
-
-
-def pause_job(collector_name: str):
-    """Pause a scheduled job"""
-    scheduler.pause_job(collector_name)
-    logger.info(f"Paused job: {collector_name}")
-
-
-def resume_job(collector_name: str):
-    """Resume a scheduled job"""
-    scheduler.resume_job(collector_name)
-    logger.info(f"Resumed job: {collector_name}")
+    async with async_session_factory() as db:
+        result = await db.execute(
+            select(CollectionTask.id)
+            .where(CollectionTask.datasource_id == datasource_id)
+            .order_by(CollectionTask.created_at.desc(), CollectionTask.id.desc())
+            .limit(1)
+        )
+        return result.scalar_one_or_none()
 
 
 def run_collector_now(collector_name: str) -> bool:
-    """Run a collector immediately (not scheduled)"""
+    """Run a collector immediately (not scheduled)."""
     collector = collector_registry.get(collector_name)
     if not collector:
-        logger.error(f"Collector not found: {collector_name}")
+        logger.error("Collector not found: %s", collector_name)
+        return False
+
+    existing_task = get_running_collector_task(collector_name)
+    if existing_task is not None and not existing_task.done():
+        logger.warning("Collector %s is already running in-memory; skipping duplicate trigger", collector_name)
         return False
 
     try:
-        asyncio.create_task(run_collector_task(collector_name))
-        logger.info(f"Triggered collector: {collector_name}")
+        task = asyncio.create_task(run_collector_task(collector_name), name=_collector_task_name(collector_name))
+        RUNNING_COLLECTOR_TASKS[collector_name] = task
+
+        def _cleanup_task(done_task: asyncio.Task[Any]) -> None:
+            current = RUNNING_COLLECTOR_TASKS.get(collector_name)
+            if current is done_task:
+                RUNNING_COLLECTOR_TASKS.pop(collector_name, None)
+
+        task.add_done_callback(_cleanup_task)
+        logger.info("Triggered collector: %s", collector_name)
         return True
-    except Exception as e:
-        logger.error(f"Failed to trigger collector {collector_name}: {e}")
+    except Exception as exc:
+        logger.error("Failed to trigger collector %s: %s", collector_name, exc)
         return False
+
+
+async def cancel_running_collector_now(collector_name: str) -> bool:
+    task = get_running_collector_task(collector_name)
+    if task is None or task.done():
+        RUNNING_COLLECTOR_TASKS.pop(collector_name, None)
+        return False
+
+    task.cancel()
+    try:
+        await task
+    except asyncio.CancelledError:
+        return True
+    return task.cancelled()

backend/app/services/situational_alert_ai_brief.py

@@ -0,0 +1,174 @@
+from __future__ import annotations
+
+from collections import Counter
+from typing import Any
+
+from sqlalchemy import func, select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.models.alert import Alert, AlertSeverity, AlertStatus
+from app.models.bgp_anomaly import BGPAnomaly
+from app.models.bgp_incident import BGPIncident
+from app.schemas.ai import SituationalAnalysisRequest
+from app.services.bgp_ai_brief_store import get_latest_bgp_brief_record
+
+
+def _format_pairs(pairs: list[tuple[str, int]], empty_text: str = "无") -> str:
+    if not pairs:
+        return empty_text
+    return "，".join(f"{key} {value}" for key, value in pairs if key)
+
+
+async def build_situational_alert_brief_request(
+    db: AsyncSession,
+) -> tuple[SituationalAnalysisRequest, list[str], dict[str, Any]]:
+    total_alerts_result = await db.execute(select(func.count(Alert.id)))
+    active_alerts_result = await db.execute(
+        select(func.count(Alert.id)).where(Alert.status == AlertStatus.ACTIVE)
+    )
+    alert_severity_result = await db.execute(
+        select(Alert.severity, func.count(Alert.id))
+        .where(Alert.status == AlertStatus.ACTIVE)
+        .group_by(Alert.severity)
+    )
+    alert_source_result = await db.execute(
+        select(Alert.datasource_name, func.count(Alert.id))
+        .where(Alert.status == AlertStatus.ACTIVE)
+        .group_by(Alert.datasource_name)
+        .order_by(func.count(Alert.id).desc())
+        .limit(6)
+    )
+    recent_alerts_result = await db.execute(
+        select(Alert)
+        .order_by(Alert.created_at.desc(), Alert.id.desc())
+        .limit(6)
+    )
+
+    total_incidents_result = await db.execute(select(func.count(BGPIncident.id)))
+    active_incidents_result = await db.execute(
+        select(func.count(BGPIncident.id)).where(BGPIncident.status == "active")
+    )
+    bgp_severity_result = await db.execute(
+        select(BGPIncident.severity, func.count(BGPIncident.id))
+        .where(BGPIncident.status == "active")
+        .group_by(BGPIncident.severity)
+    )
+    bgp_region_counter: Counter[str] = Counter()
+    recent_incidents_result = await db.execute(
+        select(BGPIncident)
+        .order_by(BGPIncident.created_at.desc(), BGPIncident.id.desc())
+        .limit(5)
+    )
+
+    total_anomalies_result = await db.execute(select(func.count(BGPAnomaly.id)))
+    active_anomalies_result = await db.execute(
+        select(func.count(BGPAnomaly.id)).where(BGPAnomaly.status == "active")
+    )
+    anomaly_type_result = await db.execute(
+        select(BGPAnomaly.anomaly_type, func.count(BGPAnomaly.id))
+        .where(BGPAnomaly.status == "active")
+        .group_by(BGPAnomaly.anomaly_type)
+        .order_by(func.count(BGPAnomaly.id).desc())
+        .limit(6)
+    )
+
+    recent_incidents = recent_incidents_result.scalars().all()
+    for incident in recent_incidents:
+        for region in incident.affected_regions or []:
+            if not isinstance(region, dict):
+                continue
+            label = ", ".join(part for part in [region.get("city"), region.get("country")] if part) or "未知区域"
+            bgp_region_counter[label] += 1
+
+    latest_bgp_brief = get_latest_bgp_brief_record()
+    active_alert_severities = [
+        (item[0].value if isinstance(item[0], AlertSeverity) else str(item[0]), item[1])
+        for item in alert_severity_result.fetchall()
+        if item[0]
+    ]
+    active_bgp_severities = [
+        (str(item[0]), item[1])
+        for item in bgp_severity_result.fetchall()
+        if item[0]
+    ]
+    active_anomaly_types = [(str(item[0]), item[1]) for item in anomaly_type_result.fetchall() if item[0]]
+    active_alert_sources = [
+        (str(item[0] or "未命名数据源"), item[1])
+        for item in alert_source_result.fetchall()
+    ]
+
+    facts = [
+        (
+            f"系统告警侧：总告警 {total_alerts_result.scalar() or 0} 条，active {active_alerts_result.scalar() or 0} 条；"
+            f"活跃告警严重度分布为 {_format_pairs(active_alert_severities)}。"
+        ),
+        (
+            f"BGP态势侧：累计 incidents {total_incidents_result.scalar() or 0} 条，active incidents {active_incidents_result.scalar() or 0} 条；"
+            f"活跃 incidents 严重度分布为 {_format_pairs(active_bgp_severities)}。"
+        ),
+        (
+            f"BGP异常侧：累计 anomalies {total_anomalies_result.scalar() or 0} 条，active anomalies {active_anomalies_result.scalar() or 0} 条；"
+            f"活跃 anomaly 类型分布为 {_format_pairs(active_anomaly_types)}。"
+        ),
+    ]
+
+    if active_alert_sources:
+        facts.append(f"当前系统告警主要集中在：{_format_pairs(active_alert_sources)}。")
+    if bgp_region_counter:
+        facts.append(f"BGP近期高风险区域线索：{_format_pairs(bgp_region_counter.most_common(5))}。")
+
+    recent_alerts = recent_alerts_result.scalars().all()
+    if recent_alerts:
+        facts.append(
+            "最近系统告警摘录："
+            + "；".join(
+                [
+                    f"{alert.datasource_name or '未命名数据源'} / {alert.severity.value if alert.severity else '-'} / {alert.status.value if alert.status else '-'} / {alert.message or '-'}"
+                    for alert in recent_alerts
+                ]
+            )
+        )
+
+    if recent_incidents:
+        facts.append(
+            "最近BGP事件摘录："
+            + "；".join(
+                [
+                    f"{incident.incident_type} / {incident.severity} / {incident.status} / {incident.summary}"
+                    for incident in recent_incidents
+                ]
+            )
+        )
+
+    if latest_bgp_brief:
+        facts.append(
+            f"最近一份 BGP AI 简报生成于 {latest_bgp_brief.generated_at}，模型 {latest_bgp_brief.model}，可作为当前态势的补充说明。"
+        )
+
+    context = {
+        "source": "situational-alerts",
+        "active_system_alerts": active_alerts_result.scalar() or 0,
+        "active_system_alert_severities": dict(active_alert_severities),
+        "top_system_alert_sources": dict(active_alert_sources),
+        "active_bgp_incidents": active_incidents_result.scalar() or 0,
+        "active_bgp_incident_severities": dict(active_bgp_severities),
+        "active_bgp_anomalies": active_anomalies_result.scalar() or 0,
+        "active_bgp_anomaly_types": dict(active_anomaly_types),
+        "bgp_hot_regions": dict(bgp_region_counter.most_common(5)),
+        "latest_bgp_brief_id": latest_bgp_brief.id if latest_bgp_brief else None,
+        "latest_bgp_brief_generated_at": latest_bgp_brief.generated_at if latest_bgp_brief else None,
+    }
+
+    request = SituationalAnalysisRequest(
+        title="态势告警 AI 简报",
+        objective="综合系统告警、BGP incidents、BGP anomalies 与近期 BGP AI 简报，生成一份面向值班人员的态势告警简报，指出当前最需要关注的风险域、跨模块联动迹象和优先动作。",
+        observations=facts,
+        constraints=[
+            "明确区分事实、推断与建议。",
+            "优先指出仍在 active 状态的系统告警与 BGP 风险是否存在联动。",
+            "不要把单一数据源的局部异常夸大成全局态势。",
+            "如果证据不足，请明确写出仍缺哪些模块或区域信息。",
+        ],
+        context=context,
+    )
+    return request, facts, context

backend/app/services/system_control.py

@@ -0,0 +1,178 @@
+from __future__ import annotations
+
+import json
+import secrets
+from datetime import UTC, datetime, timedelta
+from pathlib import Path
+from typing import Any
+
+from app.core.config import ROOT_DIR
+from app.core.security import redis_client
+
+SYSTEM_TASK_TTL_SECONDS = 24 * 60 * 60
+SYSTEM_TASK_LOG_LIMIT = 100
+SYSTEM_TASK_ACTIVE_KEY = "system:restart_task:active"
+SYSTEM_TASK_STALE_SECONDS = 5 * 60
+
+ALLOWED_ACTIONS: dict[str, dict[str, Any]] = {
+    "restart-backend": {
+        "command": ["./planet.sh", "restart", "-b"],
+        "recovery_mode": "backend",
+    },
+    "restart-ai-provider": {
+        "command": ["./planet.sh", "restart", "-a"],
+        "recovery_mode": "ai-provider",
+    },
+    "restart-database": {
+        "command": ["./planet.sh", "restart", "-d"],
+        "recovery_mode": "database",
+    },
+    "restart-system": {
+        "command": ["./planet.sh", "restart"],
+        "recovery_mode": "system",
+    },
+}
+
+
+def utc_now_iso() -> str:
+    return datetime.now(UTC).isoformat()
+
+
+def normalize_user_role(role: Any) -> str:
+    return role.value if hasattr(role, "value") else str(role)
+
+
+def require_super_admin(user_role: Any) -> bool:
+    return normalize_user_role(user_role) == "super_admin"
+
+
+def build_task_id(prefix: str = "restart") -> str:
+    timestamp = datetime.now(UTC).strftime("%Y%m%d_%H%M%S")
+    return f"{prefix}_{timestamp}_{secrets.token_hex(3)}"
+
+
+def get_task_key(task_id: str) -> str:
+    return f"system:restart_task:{task_id}"
+
+
+def get_task_logs_key(task_id: str) -> str:
+    return f"{get_task_key(task_id)}:logs"
+
+
+def get_allowed_command(action: str) -> list[str] | None:
+    config = ALLOWED_ACTIONS.get(action)
+    if config is None:
+        return None
+    return list(config["command"])
+
+
+def get_action_recovery_mode(action: str) -> str | None:
+    config = ALLOWED_ACTIONS.get(action)
+    if config is None:
+        return None
+    return str(config["recovery_mode"])
+
+
+def serialize_task(task_id: str) -> dict[str, Any] | None:
+    payload = redis_client.hgetall(get_task_key(task_id))
+    if not payload:
+        return None
+
+    if payload.get("requested_by"):
+        try:
+            payload["requested_by"] = json.loads(payload["requested_by"])
+        except json.JSONDecodeError:
+            payload["requested_by"] = {"username": payload["requested_by"]}
+
+    return payload
+
+
+def append_task_log(task_id: str, line: str) -> None:
+    logs_key = get_task_logs_key(task_id)
+    redis_client.rpush(logs_key, line)
+    redis_client.ltrim(logs_key, -SYSTEM_TASK_LOG_LIMIT, -1)
+    redis_client.expire(logs_key, SYSTEM_TASK_TTL_SECONDS)
+
+
+def upsert_task_state(
+    task_id: str,
+    *,
+    action: str | None = None,
+    status: str,
+    stage: str,
+    message: str,
+    requested_by: dict[str, Any] | None = None,
+) -> dict[str, Any]:
+    existing = serialize_task(task_id) or {}
+    now = utc_now_iso()
+
+    payload: dict[str, Any] = {
+        "task_id": task_id,
+        "action": action or existing.get("action") or "",
+        "status": status,
+        "stage": stage,
+        "message": message,
+        "created_at": existing.get("created_at") or now,
+        "updated_at": now,
+    }
+
+    if requested_by is not None:
+        payload["requested_by"] = requested_by
+    elif existing.get("requested_by") is not None:
+        payload["requested_by"] = existing["requested_by"]
+
+    redis_payload = {
+        key: json.dumps(value, ensure_ascii=False) if key == "requested_by" else str(value)
+        for key, value in payload.items()
+        if value is not None
+    }
+    task_key = get_task_key(task_id)
+    redis_client.hset(task_key, mapping=redis_payload)
+    redis_client.expire(task_key, SYSTEM_TASK_TTL_SECONDS)
+    return payload
+
+
+def get_task_logs(task_id: str) -> list[str]:
+    return [str(item) for item in redis_client.lrange(get_task_logs_key(task_id), 0, -1)]
+
+
+def get_active_task_id() -> str | None:
+    value = redis_client.get(SYSTEM_TASK_ACTIVE_KEY)
+    return str(value) if value else None
+
+
+def set_active_task_id(task_id: str) -> None:
+    redis_client.set(SYSTEM_TASK_ACTIVE_KEY, task_id, ex=SYSTEM_TASK_TTL_SECONDS)
+
+
+def clear_active_task_id(task_id: str) -> None:
+    current = get_active_task_id()
+    if current == task_id:
+        redis_client.delete(SYSTEM_TASK_ACTIVE_KEY)
+
+
+def parse_task_timestamp(value: str | None) -> datetime | None:
+    if not value:
+        return None
+    try:
+        return datetime.fromisoformat(value)
+    except ValueError:
+        return None
+
+
+def is_task_stale(task: dict[str, Any], *, max_age_seconds: int = SYSTEM_TASK_STALE_SECONDS) -> bool:
+    if task.get("status") not in {"queued", "running"}:
+        return False
+
+    updated_at = parse_task_timestamp(str(task.get("updated_at") or ""))
+    if updated_at is None:
+        return False
+
+    if updated_at.tzinfo is None:
+        updated_at = updated_at.replace(tzinfo=UTC)
+
+    return datetime.now(UTC) - updated_at > timedelta(seconds=max_age_seconds)
+
+
+def get_runner_script_path() -> Path:
+    return ROOT_DIR / "backend" / "scripts" / "system_restart_runner.py"

backend/app/services/tv_streams.py

@@ -0,0 +1,466 @@
+from __future__ import annotations
+
+from datetime import UTC, datetime
+from typing import Any
+from urllib.parse import urlparse
+
+from sqlalchemy import select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.core.time import to_iso8601_utc
+from app.models.collected_data import CollectedData
+from app.models.system_setting import SystemSetting
+
+DEFAULT_TV_SOURCE_ID = "cgtn-en"
+TV_SETTINGS_CATEGORY = "tv"
+TV_LIVE_SOURCE_COLLECTOR = "news_live_streams"
+TV_LIVE_SOURCE_DATA_TYPE = "news_live_stream"
+
+DEFAULT_TV_SETTINGS = {
+        "default_source_id": DEFAULT_TV_SOURCE_ID,
+    "auto_fallback": True,
+    "sources": [
+        {
+            "id": "cctv4",
+            "name": "CCTV-4 中文国际",
+            "provider": "CCTV",
+            "region": "China",
+            "language": "zh-CN",
+            "source_type": "hls",
+            "embed_url": "https://tv.cctv.com/live/cctv4/",
+            "stream_url": "https://ldocctvwbcdtxy.liveplay.myqcloud.com/ldocctvwbcd/cdrmldcctv4_1_td.m3u8",
+            "homepage_url": "https://tv.cctv.com/live/cctv4/",
+            "poster_url": "",
+            "is_enabled": True,
+            "is_fallback": True,
+            "sort_order": 10,
+            "collector_source": None,
+            "notes": "默认兜底新闻直播源。优先尝试 CCTV-4 官方 HLS 播放流，若直播放失败则回退到央视官网直播页。",
+        },
+        {
+            "id": "reuters-tv",
+            "name": "Reuters TV",
+            "provider": "Reuters",
+            "region": "Global",
+            "language": "en",
+            "source_type": "hls",
+            "embed_url": "",
+            "stream_url": "https://reuters-reutersnow-1-eu.rakuten.wurl.tv/playlist.m3u8",
+            "homepage_url": "https://www.reuters.com/video/live/",
+            "poster_url": "",
+            "is_enabled": True,
+            "is_fallback": False,
+            "sort_order": 20,
+            "collector_source": None,
+            "notes": "参考 worldmonitor 的默认新闻频道清单，优先作为全球英文新闻直播放源。",
+        },
+        {
+            "id": "cgtn-en",
+            "name": "CGTN English",
+            "provider": "CGTN",
+            "region": "Global",
+            "language": "en",
+            "source_type": "youtube",
+            "embed_url": "https://www.youtube.com/watch?v=BOy2xDU1LC8",
+            "stream_url": "https://news.cgtn.com/resource/live/english/cgtn-news.m3u8",
+            "youtube_video_id": "BOy2xDU1LC8",
+            "homepage_url": "https://news.cgtn.com/",
+            "poster_url": "",
+            "is_enabled": True,
+            "is_fallback": False,
+            "sort_order": 30,
+            "collector_source": None,
+            "notes": "优先使用官方 YouTube 直播源，保留 HLS 直播放流作为候选信息。",
+        },
+        {
+            "id": "cgtn-es",
+            "name": "CGTN Espanol",
+            "provider": "CGTN",
+            "region": "Latin America",
+            "language": "es",
+            "source_type": "hls",
+            "embed_url": "",
+            "stream_url": "https://news.cgtn.com/resource/live/espanol/cgtn-e.m3u8",
+            "homepage_url": "https://news.cgtn.com/",
+            "poster_url": "",
+            "is_enabled": True,
+            "is_fallback": False,
+            "sort_order": 40,
+            "collector_source": None,
+            "notes": "西语国际新闻频道，覆盖拉美方向态势。",
+        },
+        {
+            "id": "dw-espanol",
+            "name": "DW Espanol",
+            "provider": "Deutsche Welle",
+            "region": "Europe",
+            "language": "es",
+            "source_type": "hls",
+            "embed_url": "",
+            "stream_url": "https://dwamdstream104.akamaized.net/hls/live/2015530/dwstream104/stream04/streamPlaylist.m3u8",
+            "homepage_url": "https://www.dw.com/es/",
+            "poster_url": "",
+            "is_enabled": True,
+            "is_fallback": False,
+            "sort_order": 50,
+            "collector_source": None,
+            "notes": "来自 worldmonitor 可选频道清单的直播放源。",
+        },
+        {
+            "id": "dw-arabic",
+            "name": "DW Arabic",
+            "provider": "Deutsche Welle",
+            "region": "Middle East",
+            "language": "ar",
+            "source_type": "hls",
+            "embed_url": "",
+            "stream_url": "https://dwamdstream103.akamaized.net/hls/live/2015526/dwstream103/index.m3u8",
+            "homepage_url": "https://www.dw.com/ar/",
+            "poster_url": "",
+            "is_enabled": True,
+            "is_fallback": False,
+            "sort_order": 60,
+            "collector_source": None,
+            "notes": "阿拉伯语新闻流，适合作为中东方向新闻补充源。",
+        },
+        {
+            "id": "aljazeera-mubasher",
+            "name": "Al Jazeera Mubasher",
+            "provider": "Al Jazeera",
+            "region": "Middle East",
+            "language": "ar",
+            "source_type": "hls",
+            "embed_url": "",
+            "stream_url": "https://live-hls-web-ajm.getaj.net/AJM/index.m3u8",
+            "homepage_url": "https://www.aljazeera.net/live",
+            "poster_url": "",
+            "is_enabled": True,
+            "is_fallback": False,
+            "sort_order": 70,
+            "collector_source": None,
+            "notes": "中东实时新闻流，来自 worldmonitor HLS 频道目录。",
+        },
+        {
+            "id": "arirang-news",
+            "name": "Arirang News",
+            "provider": "Arirang",
+            "region": "Korea",
+            "language": "en",
+            "source_type": "hls",
+            "embed_url": "",
+            "stream_url": "https://amdlive-ch01-ctnd-com.akamaized.net/arirang_1ch/smil:arirang_1ch.smil/playlist.m3u8",
+            "homepage_url": "https://www.arirang.com/",
+            "poster_url": "",
+            "is_enabled": True,
+            "is_fallback": False,
+            "sort_order": 80,
+            "collector_source": None,
+            "notes": "东北亚英语新闻源，适合补充韩半岛与东亚视角。",
+        },
+        {
+            "id": "abp-news",
+            "name": "ABP News",
+            "provider": "ABP",
+            "region": "India",
+            "language": "hi",
+            "source_type": "hls",
+            "embed_url": "",
+            "stream_url": "https://abplivetv.pc.cdn.bitgravity.com/httppush/abp_livetv/abp_abpnews/master.m3u8",
+            "homepage_url": "https://news.abplive.com/live-tv",
+            "poster_url": "",
+            "is_enabled": True,
+            "is_fallback": False,
+            "sort_order": 90,
+            "collector_source": None,
+            "notes": "印度新闻直播放源，补充南亚区域视角。",
+        },
+        {
+            "id": "sabc-news",
+            "name": "SABC News",
+            "provider": "SABC",
+            "region": "Africa",
+            "language": "en",
+            "source_type": "hls",
+            "embed_url": "",
+            "stream_url": "https://sabconetanw.cdn.mangomolo.com/news/smil:news.stream.smil/playlist.m3u8",
+            "homepage_url": "https://www.sabcnews.com/sabcnews/",
+            "poster_url": "",
+            "is_enabled": True,
+            "is_fallback": False,
+            "sort_order": 100,
+            "collector_source": None,
+            "notes": "非洲英语新闻源，补充非洲区域新闻覆盖。",
+        },
+    ],
+}
+
+
+def _clean_text(value: Any) -> str:
+    if value is None:
+        return ""
+    return str(value).strip()
+
+
+def _clean_url(value: Any) -> str:
+    text = _clean_text(value)
+    if not text:
+        return ""
+
+    parsed = urlparse(text)
+    if parsed.scheme and parsed.scheme not in {"http", "https"}:
+        return ""
+    if parsed.scheme and not parsed.netloc:
+        return ""
+    return text
+
+
+def _clean_bool(value: Any, *, default: bool) -> bool:
+    if isinstance(value, bool):
+        return value
+    if value in (None, ""):
+        return default
+    if isinstance(value, str):
+        lowered = value.strip().lower()
+        if lowered in {"1", "true", "yes", "on"}:
+            return True
+        if lowered in {"0", "false", "no", "off"}:
+            return False
+    return bool(value)
+
+
+def _clean_int(value: Any, *, default: int) -> int:
+    try:
+        return int(value)
+    except (TypeError, ValueError):
+        return default
+
+
+def normalize_tv_source(source: dict[str, Any] | None, *, index: int = 0) -> dict[str, Any]:
+    payload = dict(source or {})
+    source_id = _clean_text(payload.get("id")) or f"tv-source-{index + 1}"
+    source_type = _clean_text(payload.get("source_type")).lower()
+    youtube_video_id = _clean_text(payload.get("youtube_video_id"))
+    youtube_channel = _clean_text(payload.get("youtube_channel"))
+    if source_type not in {"iframe", "hls", "video", "external", "youtube"}:
+        if youtube_video_id or youtube_channel:
+            source_type = "youtube"
+        else:
+            source_type = "iframe" if _clean_text(payload.get("embed_url")) else "external"
+
+    if source_type == "youtube" and not youtube_video_id and not youtube_channel:
+        source_type = "iframe" if _clean_text(payload.get("embed_url")) else "external"
+
+    return {
+        "id": source_id,
+        "name": _clean_text(payload.get("name")) or f"新闻直播源 {index + 1}",
+        "provider": _clean_text(payload.get("provider")) or "Unknown",
+        "region": _clean_text(payload.get("region")) or "Global",
+        "language": _clean_text(payload.get("language")) or "und",
+        "source_type": source_type,
+        "embed_url": _clean_url(payload.get("embed_url")),
+        "stream_url": _clean_url(payload.get("stream_url")),
+        "homepage_url": _clean_url(payload.get("homepage_url")),
+        "poster_url": _clean_url(payload.get("poster_url")),
+        "youtube_video_id": youtube_video_id,
+        "youtube_channel": youtube_channel,
+        "is_enabled": _clean_bool(payload.get("is_enabled"), default=True),
+        "is_fallback": _clean_bool(payload.get("is_fallback"), default=False),
+        "sort_order": _clean_int(payload.get("sort_order"), default=(index + 1) * 10),
+        "collector_source": payload.get("collector_source"),
+        "notes": _clean_text(payload.get("notes")),
+        "updated_at": _clean_text(payload.get("updated_at")),
+    }
+
+
+def normalize_tv_settings(payload: dict[str, Any] | None) -> dict[str, Any]:
+    merged = {
+        "default_source_id": DEFAULT_TV_SETTINGS["default_source_id"],
+        "auto_fallback": DEFAULT_TV_SETTINGS["auto_fallback"],
+        "sources": [],
+    }
+
+    raw_sources = []
+    if isinstance(payload, dict):
+        merged["default_source_id"] = (
+            _clean_text(payload.get("default_source_id")) or merged["default_source_id"]
+        )
+        merged["auto_fallback"] = _clean_bool(
+            payload.get("auto_fallback"),
+            default=DEFAULT_TV_SETTINGS["auto_fallback"],
+        )
+        if isinstance(payload.get("sources"), list):
+            raw_sources = payload["sources"]
+
+    if not raw_sources:
+        raw_sources = DEFAULT_TV_SETTINGS["sources"]
+
+    normalized_sources = [
+        normalize_tv_source(source, index=index)
+        for index, source in enumerate(raw_sources)
+    ]
+
+    if not any(source["id"] == DEFAULT_TV_SOURCE_ID for source in normalized_sources):
+        normalized_sources.append(
+            normalize_tv_source(DEFAULT_TV_SETTINGS["sources"][0], index=len(normalized_sources))
+        )
+
+    default_source_exists = any(
+        source["id"] == merged["default_source_id"] and source["is_enabled"]
+        for source in normalized_sources
+    )
+    if not default_source_exists:
+        fallback_source = next(
+            (source for source in normalized_sources if source["is_fallback"] and source["is_enabled"]),
+            None,
+        )
+        first_enabled_source = next(
+            (source for source in normalized_sources if source["is_enabled"]),
+            None,
+        )
+        merged["default_source_id"] = (
+            fallback_source["id"]
+            if fallback_source
+            else first_enabled_source["id"]
+            if first_enabled_source
+            else DEFAULT_TV_SOURCE_ID
+        )
+
+    merged["sources"] = sorted(
+        normalized_sources,
+        key=lambda item: (item["sort_order"], item["name"], item["id"]),
+    )
+    return merged
+
+
+async def get_tv_settings_payload(db: AsyncSession) -> dict[str, Any]:
+    result = await db.execute(
+        select(SystemSetting).where(SystemSetting.category == TV_SETTINGS_CATEGORY)
+    )
+    record = result.scalar_one_or_none()
+    payload = record.payload if record else None
+    return normalize_tv_settings(payload)
+
+
+def _build_collected_tv_source(record: CollectedData, index: int) -> dict[str, Any]:
+    metadata = dict(record.extra_data or {})
+    return normalize_tv_source(
+        {
+            "id": metadata.get("id") or record.source_id or record.entity_key,
+            "name": record.name or record.title or metadata.get("name") or f"采集直播源 {index + 1}",
+            "provider": metadata.get("provider") or metadata.get("publisher") or "Collector",
+            "region": metadata.get("region") or metadata.get("country") or "Global",
+            "language": metadata.get("language") or "und",
+            "source_type": metadata.get("source_type") or "iframe",
+            "embed_url": metadata.get("embed_url") or metadata.get("url") or "",
+            "stream_url": metadata.get("stream_url") or "",
+            "homepage_url": metadata.get("homepage_url") or metadata.get("source_url") or "",
+            "poster_url": metadata.get("poster_url") or "",
+            "youtube_video_id": metadata.get("youtube_video_id") or metadata.get("video_id") or "",
+            "youtube_channel": metadata.get("youtube_channel") or metadata.get("channel_handle") or "",
+            "is_enabled": metadata.get("is_enabled", True),
+            "is_fallback": False,
+            "sort_order": metadata.get("sort_order", 200 + index),
+            "collector_source": record.source,
+            "notes": record.description or metadata.get("notes") or "",
+            "updated_at": to_iso8601_utc(record.updated_at or record.reference_date or datetime.now(UTC)),
+        },
+        index=index,
+    )
+
+
+async def get_collected_tv_sources(db: AsyncSession) -> list[dict[str, Any]]:
+    result = await db.execute(
+        select(CollectedData)
+        .where(CollectedData.source == TV_LIVE_SOURCE_COLLECTOR)
+        .where(CollectedData.data_type == TV_LIVE_SOURCE_DATA_TYPE)
+        .where(CollectedData.is_current.is_(True))
+        .where(CollectedData.is_valid == 1)
+        .order_by(CollectedData.reference_date.desc().nullslast(), CollectedData.id.desc())
+    )
+    rows = result.scalars().all()
+    return [_build_collected_tv_source(record, index) for index, record in enumerate(rows)]
+
+
+def build_public_tv_payload(
+    settings_payload: dict[str, Any],
+    collected_sources: list[dict[str, Any]],
+) -> dict[str, Any]:
+    configured_sources = [
+        source for source in settings_payload["sources"] if source["is_enabled"]
+    ]
+
+    merged_by_id = {source["id"]: source for source in configured_sources}
+    for source in collected_sources:
+        if source["id"] in merged_by_id or not source["is_enabled"]:
+            continue
+        merged_by_id[source["id"]] = source
+
+    available_sources = sorted(
+        merged_by_id.values(),
+        key=lambda item: (item["sort_order"], item["name"], item["id"]),
+    )
+
+    default_source = next(
+        (
+            source
+            for source in available_sources
+            if source["id"] == settings_payload["default_source_id"]
+        ),
+        None,
+    )
+    fallback_source = next(
+        (source for source in available_sources if source["is_fallback"]),
+        None,
+    )
+
+    resolved_source = default_source or fallback_source or (available_sources[0] if available_sources else None)
+    latest_updated_at = max(
+        (source.get("updated_at") or "" for source in available_sources),
+        default="",
+    )
+
+    return {
+        "default_source_id": settings_payload["default_source_id"],
+        "auto_fallback": settings_payload["auto_fallback"],
+        "selected_source": resolved_source,
+        "fallback_source": fallback_source,
+        "sources": available_sources,
+        "source_count": len(available_sources),
+        "latest_updated_at": latest_updated_at or to_iso8601_utc(datetime.now(UTC)),
+        "generated_at": to_iso8601_utc(datetime.now(UTC)),
+    }
+
+
+async def get_public_tv_payload(db: AsyncSession) -> dict[str, Any]:
+    settings_payload = await get_tv_settings_payload(db)
+    collected_sources = await get_collected_tv_sources(db)
+    return build_public_tv_payload(settings_payload, collected_sources)
+
+
+def _extract_allowed_tv_hosts(sources: list[dict[str, Any]]) -> set[str]:
+    hosts: set[str] = set()
+    for source in sources:
+        for field in ("stream_url", "embed_url", "homepage_url", "youtube_channel"):
+            value = _clean_url(source.get(field))
+            if not value:
+                continue
+            parsed = urlparse(value)
+            if parsed.hostname:
+                hosts.add(parsed.hostname.lower())
+    return hosts
+
+
+def is_allowed_tv_proxy_url(url: str, sources: list[dict[str, Any]]) -> bool:
+    cleaned = _clean_url(url)
+    if not cleaned:
+        return False
+
+    parsed = urlparse(cleaned)
+    hostname = (parsed.hostname or "").lower()
+    if not hostname:
+        return False
+
+    allowed_hosts = _extract_allowed_tv_hosts(sources)
+    if hostname in allowed_hosts:
+        return True
+    return any(hostname.endswith(f".{allowed_host}") for allowed_host in allowed_hosts)

backend/requirements.txt

@@ -1,19 +0,0 @@
-fastapi>=0.109.0
-uvicorn[standard]>=0.27.0
-sqlalchemy[asyncio]>=2.0.25
-asyncpg>=0.29.0
-redis>=5.0.1
-pydantic>=2.5.0
-pydantic-settings>=2.1.0
-python-jose[cryptography]>=3.3.0
-passlib[bcrypt]>=1.7.4
-python-multipart>=0.0.6
-httpx>=0.26.0
-beautifulsoup4>=4.12.0
-aiofiles>=23.2.1
-python-dotenv>=1.0.0
-email-validator
-apscheduler>=3.10.4
-pytest>=7.4.0
-pytest-asyncio>=0.23.0
-networkx>=3.0

backend/scripts/system_restart_runner.py

@@ -0,0 +1,184 @@
+from __future__ import annotations
+
+import argparse
+import os
+import subprocess
+import sys
+import time
+from pathlib import Path
+from urllib.error import URLError
+from urllib.request import urlopen
+
+
+ROOT_DIR = Path(__file__).resolve().parents[2]
+BACKEND_DIR = ROOT_DIR / "backend"
+
+if str(BACKEND_DIR) not in sys.path:
+    sys.path.insert(0, str(BACKEND_DIR))
+
+from app.services.system_control import (  # noqa: E402
+    append_task_log,
+    clear_active_task_id,
+    get_allowed_command,
+    get_action_recovery_mode,
+    upsert_task_state,
+)
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--task-id", required=True)
+    parser.add_argument("--action", required=True)
+    return parser.parse_args()
+
+
+def wait_for_http(url: str, timeout_seconds: int = 90, interval_seconds: float = 2.0) -> bool:
+    deadline = time.time() + timeout_seconds
+    success_streak = 0
+
+    while time.time() < deadline:
+        try:
+            with urlopen(url, timeout=2) as response:
+                if response.status == 200:
+                    success_streak += 1
+                    if success_streak >= 2:
+                        return True
+                else:
+                    success_streak = 0
+        except URLError:
+            success_streak = 0
+        except Exception:
+            success_streak = 0
+
+        time.sleep(interval_seconds)
+
+    return False
+
+
+def wait_for_recovery(action: str) -> tuple[bool, str]:
+    recovery_mode = get_action_recovery_mode(action)
+    if recovery_mode == "backend":
+        return wait_for_http("http://localhost:8000/health"), "backend health recovery"
+    if recovery_mode == "ai-provider":
+        return wait_for_http("http://localhost:8010/health"), "ai provider health recovery"
+    if recovery_mode == "database":
+        return True, "database container restart completion"
+    if recovery_mode == "system":
+        backend_ok = wait_for_http("http://localhost:8000/health")
+        frontend_ok = wait_for_http("http://localhost:3000")
+        return backend_ok and frontend_ok, "system service recovery"
+    return False, "unsupported recovery mode"
+
+
+def main() -> int:
+    args = parse_args()
+    command = get_allowed_command(args.action)
+    recovery_mode = get_action_recovery_mode(args.action)
+    if command is None or recovery_mode is None:
+        upsert_task_state(
+            args.task_id,
+            action=args.action,
+            status="failed",
+            stage="failed",
+            message="Unsupported system action",
+        )
+        clear_active_task_id(args.task_id)
+        return 1
+
+    upsert_task_state(
+        args.task_id,
+        action=args.action,
+        status="running",
+        stage="spawning",
+        message="Spawning restart command",
+    )
+    append_task_log(args.task_id, f"accepted {args.action} request")
+    append_task_log(args.task_id, f"resolved command: {' '.join(command)}")
+
+    env = os.environ.copy()
+    env["PATH"] = f"{Path.home() / '.bun' / 'bin'}:{Path.home() / '.local' / 'bin'}:{env.get('PATH', '')}"
+
+    try:
+        upsert_task_state(
+            args.task_id,
+            action=args.action,
+            status="running",
+            stage="stopping",
+            message="Restart command is running",
+        )
+        append_task_log(args.task_id, "restart command started")
+
+        completed = subprocess.run(
+            command,
+            cwd=str(ROOT_DIR),
+            env=env,
+            capture_output=True,
+            text=True,
+            check=False,
+        )
+
+        if completed.stdout.strip():
+            for line in completed.stdout.strip().splitlines()[-20:]:
+                append_task_log(args.task_id, line)
+        if completed.stderr.strip():
+            for line in completed.stderr.strip().splitlines()[-20:]:
+                append_task_log(args.task_id, line)
+
+        if completed.returncode != 0:
+            upsert_task_state(
+                args.task_id,
+                action=args.action,
+                status="failed",
+                stage="failed",
+                message=f"Restart command failed with exit code {completed.returncode}",
+            )
+            clear_active_task_id(args.task_id)
+            return completed.returncode
+
+        upsert_task_state(
+            args.task_id,
+            action=args.action,
+            status="running",
+            stage="waiting_for_health",
+            message=f"Waiting for {recovery_mode} recovery",
+        )
+        append_task_log(args.task_id, f"waiting for {recovery_mode} recovery")
+
+        recovered, recovery_label = wait_for_recovery(args.action)
+        if recovered:
+            upsert_task_state(
+                args.task_id,
+                action=args.action,
+                status="succeeded",
+                stage="healthy",
+                message="Restart completed successfully",
+            )
+            append_task_log(args.task_id, f"{recovery_label} completed")
+            clear_active_task_id(args.task_id)
+            return 0
+
+        upsert_task_state(
+            args.task_id,
+            action=args.action,
+            status="timeout",
+            stage="failed",
+            message="Restart timed out waiting for recovery",
+        )
+        append_task_log(args.task_id, f"{recovery_label} timed out")
+        clear_active_task_id(args.task_id)
+        return 2
+    except Exception as exc:
+        append_task_log(args.task_id, f"runner exception: {exc}")
+        upsert_task_state(
+            args.task_id,
+            action=args.action,
+            status="failed",
+            stage="failed",
+            message=f"Restart runner failed: {exc}",
+        )
+        clear_active_task_id(args.task_id)
+        return 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

backend/tests/test_api.py

@@ -8,6 +8,14 @@ from httpx import AsyncClient, ASGITransport
 from app.main import app
 from app.core.config import settings
 from app.core.security import create_access_token
+from app.db.session import get_db
+from app.models.user import User
+from app.schemas.ai import (
+    AIProviderStatusResponse,
+    PlaygroundSessionResponse,
+    PlaygroundSessionState,
+    SituationalAnalysisResponse,
+)
 
 
 @pytest.fixture
@@ -88,12 +96,69 @@ async def test_alerts_without_auth():
 
 
 @pytest.mark.asyncio
-async def test_alerts_endpoint_with_auth(auth_headers):
-    """Test alerts endpoint with authentication"""
+async def test_datasource_task_status_without_auth():
+    """Test datasource task-status endpoint requires authentication"""
     transport = ASGITransport(app=app)
     async with AsyncClient(transport=transport, base_url="http://test") as client:
-        response = await client.get("/api/v1/alerts", headers=auth_headers)
-        assert response.status_code == 200
+        response = await client.get("/api/v1/datasources/1/task-status")
+    assert response.status_code == 401
+
+
+@pytest.mark.asyncio
+async def test_alerts_endpoint_with_auth(auth_headers):
+    """Test alerts endpoint with authentication"""
+    class _ScalarResult:
+        def __init__(self, rows=None, scalar_value=0):
+            self._rows = rows or []
+            self._scalar_value = scalar_value
+
+        def scalars(self):
+            class _Scalars:
+                def __init__(self, rows):
+                    self._rows = rows
+
+                def all(self):
+                    return self._rows
+
+            return _Scalars(self._rows)
+
+        def scalar(self):
+            return self._scalar_value
+
+    class _FakeAlertsSession:
+        def __init__(self):
+            self.calls = 0
+
+        async def execute(self, _query):
+            self.calls += 1
+            if self.calls == 1:
+                return _ScalarResult(rows=[])
+            return _ScalarResult(rows=[], scalar_value=0)
+
+    def override_get_current_user():
+        return User(
+            id=1,
+            username="testuser",
+            email="test@example.com",
+            password_hash="hashed",
+            role="admin",
+            is_active=True,
+        )
+
+    async def override_get_db():
+        yield _FakeAlertsSession()
+
+    app.dependency_overrides = {
+        __import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
+        get_db: override_get_db,
+    }
+    transport = ASGITransport(app=app)
+    try:
+        async with AsyncClient(transport=transport, base_url="http://test") as client:
+            response = await client.get("/api/v1/alerts", headers=auth_headers)
+            assert response.status_code == 200
+    finally:
+        app.dependency_overrides.clear()
 
 
 @pytest.mark.asyncio
@@ -106,3 +171,386 @@ async def test_invalid_token():
             headers={"Authorization": "Bearer invalid_token"},
         )
     assert response.status_code == 401
+
+
+@pytest.mark.asyncio
+async def test_ai_provider_status_with_auth(auth_headers):
+    """Test AI provider status endpoint"""
+    class _FakeAIProviderClient:
+        async def get_status(self, request_id=None):
+            return AIProviderStatusResponse(
+                provider="minimax",
+                api="anthropic-messages",
+                enabled=True,
+                configured=True,
+                model="test-model",
+                base_url="http://aiprovider:8010",
+            )
+
+    def override_get_current_user():
+        return User(
+            id=1,
+            username="testuser",
+            email="test@example.com",
+            password_hash="hashed",
+            role="admin",
+            is_active=True,
+        )
+
+    app.dependency_overrides = {
+        __import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
+        __import__("app.services.ai_client", fromlist=["get_ai_provider_client"]).get_ai_provider_client: lambda: _FakeAIProviderClient(),
+    }
+    transport = ASGITransport(app=app)
+    try:
+        async with AsyncClient(transport=transport, base_url="http://test") as client:
+            response = await client.get("/api/v1/ai/provider/status", headers=auth_headers)
+        assert response.status_code == 200
+        data = response.json()
+        assert "provider" in data
+        assert "api" in data
+        assert "configured" in data
+    finally:
+        app.dependency_overrides.clear()
+
+
+@pytest.mark.asyncio
+async def test_ai_situational_analysis_returns_503_when_disabled(auth_headers):
+    """Test AI analysis endpoint proxies provider service response"""
+    class _FakeAIProviderClient:
+        async def analyze(self, _payload, request_id=None):
+            return SituationalAnalysisResponse(
+                provider="openai_compatible",
+                model="test-model",
+                content="1) 态势摘要: 测试返回",
+                content_blocks=[],
+                text_blocks=["1) 态势摘要: 测试返回"],
+                thinking_blocks=[],
+                raw_response={"id": "mock-response"},
+            )
+
+    def override_get_current_user():
+        return User(
+            id=1,
+            username="testuser",
+            email="test@example.com",
+            password_hash="hashed",
+            role="admin",
+            is_active=True,
+        )
+
+    app.dependency_overrides = {
+        __import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
+        __import__("app.services.ai_client", fromlist=["get_ai_provider_client"]).get_ai_provider_client: lambda: _FakeAIProviderClient(),
+    }
+    transport = ASGITransport(app=app)
+    try:
+        async with AsyncClient(transport=transport, base_url="http://test") as client:
+            response = await client.post(
+                "/api/v1/ai/situational-awareness/analyze",
+                headers=auth_headers,
+                json={
+                    "title": "BGP 异常研判",
+                    "objective": "给出当前异常的风险摘要和建议动作",
+                    "observations": ["collector A 在 5 分钟内出现多个 origin 变更"],
+                    "constraints": ["不要假设缺失数据"],
+                },
+            )
+        assert response.status_code == 200
+        data = response.json()
+        assert data["provider"] == "openai_compatible"
+        assert data["content"]
+        assert "content_blocks" in data
+        assert "text_blocks" in data
+        assert "thinking_blocks" in data
+
+
+@pytest.mark.asyncio
+async def test_get_playground_session_with_auth(auth_headers):
+    """Test playground session restore endpoint."""
+
+    def override_get_current_user():
+        return User(
+            id=1,
+            username="testuser",
+            email="test@example.com",
+            password_hash="hashed",
+            role="admin",
+            is_active=True,
+        )
+
+    async def override_get_db():
+        yield AsyncMock()
+
+    app.dependency_overrides = {
+        __import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
+        get_db: override_get_db,
+    }
+    transport = ASGITransport(app=app)
+    try:
+        with patch(
+            "app.api.v1.ai.get_playground_session",
+            new=AsyncMock(
+                return_value=PlaygroundSessionResponse(
+                    id="1",
+                    session_key="default",
+                    title="Playground 会话",
+                    state=PlaygroundSessionState(
+                        messages=[{"id": "msg-1", "role": "user", "content": "hello"}],
+                        title="测试标题",
+                        objective="测试目标",
+                    ),
+                    created_at="2026-04-10T00:00:00+00:00",
+                    updated_at="2026-04-10T00:00:00+00:00",
+                )
+            ),
+        ):
+            async with AsyncClient(transport=transport, base_url="http://test") as client:
+                response = await client.get("/api/v1/ai/playground/session", headers=auth_headers)
+        assert response.status_code == 200
+        data = response.json()
+        assert data["session_key"] == "default"
+        assert data["state"]["messages"][0]["content"] == "hello"
+    finally:
+        app.dependency_overrides.clear()
+
+
+@pytest.mark.asyncio
+async def test_save_playground_session_with_auth(auth_headers):
+    """Test playground session save endpoint."""
+
+    def override_get_current_user():
+        return User(
+            id=1,
+            username="testuser",
+            email="test@example.com",
+            password_hash="hashed",
+            role="admin",
+            is_active=True,
+        )
+
+    async def override_get_db():
+        yield AsyncMock()
+
+    app.dependency_overrides = {
+        __import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
+        get_db: override_get_db,
+    }
+    transport = ASGITransport(app=app)
+    try:
+        with patch(
+            "app.api.v1.ai.upsert_playground_session",
+            new=AsyncMock(
+                return_value=PlaygroundSessionResponse(
+                    id="1",
+                    session_key="default",
+                    title="测试标题",
+                    state=PlaygroundSessionState(
+                        messages=[{"id": "msg-1", "role": "user", "content": "hello"}],
+                        title="测试标题",
+                        objective="测试目标",
+                    ),
+                    created_at="2026-04-10T00:00:00+00:00",
+                    updated_at="2026-04-10T00:00:00+00:00",
+                )
+            ),
+        ):
+            async with AsyncClient(transport=transport, base_url="http://test") as client:
+                response = await client.put(
+                    "/api/v1/ai/playground/session",
+                    headers=auth_headers,
+                    json={
+                        "session_key": "default",
+                        "title": "测试标题",
+                        "state": {
+                            "messages": [{"id": "msg-1", "role": "user", "content": "hello"}],
+                            "selectedPresetKey": "bgp-brief",
+                            "title": "测试标题",
+                            "objective": "测试目标",
+                            "constraints": "",
+                            "inputValue": "",
+                            "analysis": None,
+                            "latestAnalysisMessageId": None,
+                            "analysisMeta": {},
+                            "helpExpanded": True,
+                        },
+                    },
+                )
+        assert response.status_code == 200
+        data = response.json()
+        assert data["title"] == "测试标题"
+        assert data["state"]["objective"] == "测试目标"
+    finally:
+        app.dependency_overrides.clear()
+    finally:
+        app.dependency_overrides.clear()
+
+
+@pytest.mark.asyncio
+async def test_ai_bgp_brief_endpoint_persists_fact_snapshot(auth_headers):
+    class _FakeAIProviderClient:
+        async def analyze(self, _payload, request_id=None):
+            return SituationalAnalysisResponse(
+                provider="minimax",
+                model="MiniMax-M2.5",
+                content="# BGP AI 简报\n\n事实摘要：测试",
+                content_blocks=[],
+                text_blocks=["# BGP AI 简报\n\n事实摘要：测试"],
+                thinking_blocks=[],
+                raw_response={"id": "mock-bgp-brief"},
+            )
+
+    def override_get_current_user():
+        return User(
+            id=1,
+            username="testuser",
+            email="test@example.com",
+            password_hash="hashed",
+            role="admin",
+            is_active=True,
+        )
+
+    async def override_get_db():
+        yield AsyncMock()
+
+    async def _fake_build_bgp_brief_request(_db, **_kwargs):
+        request_payload = __import__("app.schemas.ai", fromlist=["SituationalAnalysisRequest"]).SituationalAnalysisRequest(
+            title="BGP 态势 AI 简报",
+            objective="生成值班简报",
+            observations=["事实A", "事实B"],
+            constraints=["不要编造"],
+            context={"incident_total": 2, "active_collectors": 3},
+        )
+        return request_payload, ["事实A", "事实B"], {"incident_total": 2, "active_collectors": 3}
+
+    app.dependency_overrides = {
+        __import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
+        __import__("app.services.ai_client", fromlist=["get_ai_provider_client"]).get_ai_provider_client: lambda: _FakeAIProviderClient(),
+        get_db: override_get_db,
+    }
+    transport = ASGITransport(app=app)
+    try:
+        with patch("app.api.v1.ai.build_bgp_brief_request", side_effect=_fake_build_bgp_brief_request):
+            async with AsyncClient(transport=transport, base_url="http://test") as client:
+                response = await client.post("/api/v1/ai/bgp/brief", headers=auth_headers, json={})
+        assert response.status_code == 200
+        data = response.json()
+        assert data["facts"] == ["事实A", "事实B"]
+        assert data["context"]["incident_total"] == 2
+        assert data["content_markdown"]
+    finally:
+        app.dependency_overrides.clear()
+
+
+@pytest.mark.asyncio
+async def test_ai_alert_brief_endpoint_with_auth(auth_headers):
+    class _FakeAIProviderClient:
+        async def analyze(self, _payload, request_id=None):
+            return SituationalAnalysisResponse(
+                provider="minimax",
+                model="MiniMax-M2.7",
+                content="事实摘要：告警测试。风险研判：告警测试。建议动作：告警测试。",
+                content_blocks=[],
+                text_blocks=["事实摘要：告警测试。风险研判：告警测试。建议动作：告警测试。"],
+                thinking_blocks=[],
+                raw_response={"id": "mock-alert-brief"},
+            )
+
+    def override_get_current_user():
+        return User(
+            id=1,
+            username="testuser",
+            email="test@example.com",
+            password_hash="hashed",
+            role="admin",
+            is_active=True,
+        )
+
+    async def override_get_db():
+        yield AsyncMock()
+
+    async def _fake_build_alert_brief_request(_db, **_kwargs):
+        request_payload = __import__("app.schemas.ai", fromlist=["SituationalAnalysisRequest"]).SituationalAnalysisRequest(
+            title="告警态势 AI 简报",
+            objective="输出告警简报",
+            observations=["告警事实A", "告警事实B"],
+            constraints=["不要编造"],
+            context={"active_alerts": 3, "top_datasources": {"bgp": 2}},
+        )
+        return request_payload, ["告警事实A", "告警事实B"], {"active_alerts": 3, "top_datasources": {"bgp": 2}}
+
+    app.dependency_overrides = {
+        __import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
+        __import__("app.services.ai_client", fromlist=["get_ai_provider_client"]).get_ai_provider_client: lambda: _FakeAIProviderClient(),
+        get_db: override_get_db,
+    }
+    transport = ASGITransport(app=app)
+    try:
+        with patch("app.api.v1.ai.build_alert_brief_request", side_effect=_fake_build_alert_brief_request):
+            async with AsyncClient(transport=transport, base_url="http://test") as client:
+                response = await client.post("/api/v1/ai/alerts/brief", headers=auth_headers, json={})
+        assert response.status_code == 200
+        data = response.json()
+        assert data["title"] == "告警态势 AI 简报"
+        assert data["facts"] == ["告警事实A", "告警事实B"]
+        assert data["context"]["active_alerts"] == 3
+        assert data["content"]
+    finally:
+        app.dependency_overrides.clear()
+
+
+@pytest.mark.asyncio
+async def test_ai_situational_alert_brief_endpoint_with_auth(auth_headers):
+    class _FakeAIProviderClient:
+        async def analyze(self, _payload, request_id=None):
+            return SituationalAnalysisResponse(
+                provider="minimax",
+                model="MiniMax-M2.7",
+                content="事实摘要：态势测试。风险研判：态势测试。建议动作：态势测试。",
+                content_blocks=[],
+                text_blocks=["事实摘要：态势测试。风险研判：态势测试。建议动作：态势测试。"],
+                thinking_blocks=[],
+                raw_response={"id": "mock-situational-brief"},
+            )
+
+    def override_get_current_user():
+        return User(
+            id=1,
+            username="testuser",
+            email="test@example.com",
+            password_hash="hashed",
+            role="admin",
+            is_active=True,
+        )
+
+    async def override_get_db():
+        yield AsyncMock()
+
+    async def _fake_build_situational_alert_brief_request(_db):
+        request_payload = __import__("app.schemas.ai", fromlist=["SituationalAnalysisRequest"]).SituationalAnalysisRequest(
+            title="态势告警 AI 简报",
+            objective="输出态势告警简报",
+            observations=["态势事实A", "态势事实B"],
+            constraints=["不要编造"],
+            context={"active_system_alerts": 2, "active_bgp_incidents": 1},
+        )
+        return request_payload, ["态势事实A", "态势事实B"], {"active_system_alerts": 2, "active_bgp_incidents": 1}
+
+    app.dependency_overrides = {
+        __import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
+        __import__("app.services.ai_client", fromlist=["get_ai_provider_client"]).get_ai_provider_client: lambda: _FakeAIProviderClient(),
+        get_db: override_get_db,
+    }
+    transport = ASGITransport(app=app)
+    try:
+        with patch("app.api.v1.ai.build_situational_alert_brief_request", side_effect=_fake_build_situational_alert_brief_request):
+            async with AsyncClient(transport=transport, base_url="http://test") as client:
+                response = await client.post("/api/v1/ai/situational-alerts/brief", headers=auth_headers, json={})
+        assert response.status_code == 200
+        data = response.json()
+        assert data["title"] == "态势告警 AI 简报"
+        assert data["facts"] == ["态势事实A", "态势事实B"]
+        assert data["context"]["active_system_alerts"] == 2
+        assert data["content"]
+    finally:
+        app.dependency_overrides.clear()