Compare commits

...

11 Commits

Author SHA1 Message Date
linkong
f22079d33a release: bump version to 0.46.3 2026-04-30 14:46:19 +08:00
linkong
9f737fdb89 release: bump version to 0.46.2 2026-04-30 14:30:12 +08:00
linkong
7418ce2fc1 release: bump version to 0.46.1 2026-04-30 09:41:08 +08:00
rayd1o
b1a5934b80 release: bump version to 0.46.0 2026-04-30 04:42:29 +08:00
rayd1o
ba54545ac7 release: bump version to 0.45.0 2026-04-29 23:43:54 +08:00
linkong
9dafbf4f6e release: bump version to 0.44.2 2026-04-29 18:11:37 +08:00
linkong
a87537e903 release: bump version to 0.44.1 2026-04-29 18:07:35 +08:00
linkong
87594a95ff release: bump version to 0.44.0 2026-04-29 17:27:44 +08:00
linkong
2da25376bd release: bump version to 0.43.1 2026-04-28 16:21:33 +08:00
linkong
ac69d5d354 release: bump version to 0.43.0 2026-04-28 16:10:17 +08:00
rayd1o
1cd2dab0ee release: bump version to 0.42.2 2026-04-28 04:35:13 +08:00
131 changed files with 14364 additions and 3524 deletions

View File

@@ -12,6 +12,19 @@ allowed-tools: ["Read", "Edit", "Bash", "Grep", "Glob"]
`$ARGUMENTS` 非空,则只检查指定文件/目录;否则检查所有未提交修改(`git diff HEAD`)。
## 节省上下文规则
优先用确定性的 CLI 检查缩小范围,不要一上来把完整文件或大 diff 读入上下文:
```bash
git diff --name-only HEAD
git diff --unified=0 HEAD -- <path>
git diff --check
rg -n "TODO|FIXME|console\.log|debugger|print\(" <changed-paths>
```
只有 focused diff 不足以安全判断或修改时,才读取完整文件。
## 审查清单
按优先级检查以下问题(只报告在本次 diff 中**新增或修改**的代码里存在的问题):
@@ -59,8 +72,13 @@ git diff HEAD --name-only
### Step 2 — 逐文件阅读并分析
- 用 Read 工具读取完整文件(不只读 diff
- 对照审查清单,记录每个问题:文件名、行号、问题类型、建议修复方式
先从 focused diff 开始:
```bash
git diff --unified=0 HEAD -- <file>
```
`rg``git diff --check`、编译器或 linter 输出确认确定性问题。只有需要上下文时才用 Read 读取完整文件。对照审查清单,记录每个问题:文件名、行号、问题类型、建议修复方式。
### Step 3 — 报告问题清单
@@ -95,6 +113,7 @@ git diff HEAD --name-only
- 只改在审查清单中发现的问题,不做额外优化
- 每次 Edit 只修改确实有问题的行,保持 diff 最小
- 改完后用 `grep` 验证旧的坏代码已消失
- 优先做精确补丁;只有仓库已有对应格式化流程时,才运行格式化工具
### Step 5 — 输出总结

232
.claude/commands/docs.md Normal file
View File

@@ -0,0 +1,232 @@
---
description: 分析本次 git 变更,在 docs/technical/zh/ 中新建或更新对应的技术文档
argument-hint: 可选:指定要记录的主题,或留空自动从 git diff 推断
allowed-tools: ["Read", "Edit", "Write", "Bash", "Glob", "Grep"]
---
# /docs — 技术文档写入工作流
## 目标
根据当前 git 变更(或用户指定主题)在 `docs/technical/zh/` 中写入或更新技术文档,记录**为什么**这样做,而不只是记录做了什么。
## 执行步骤
### Step 1 — 理解变更范围
```bash
git diff HEAD --stat # 变更文件一览
git diff HEAD --name-only # 变更文件列表
git log --oneline -10 # 近期 commit 上下文
```
`$ARGUMENTS` 指定了主题,优先聚焦该主题;否则从文件列表和 diff stat 推断变更主题。不要默认读取完整仓库 diff只对决定文档主题所需的文件读取 focused diff
```bash
git diff HEAD -- <path>
rg -n "class |def |function |export |router|@router|interface |type " <path>
```
### Step 2 — 确认文档范围
分析变更,判断:
1. **应写几篇文档**:单一主题写一篇,跨领域变更可拆分(如后端性能优化 + 运维启动脚本分开写)
2. **是新建还是更新**:检查 `docs/technical/zh/` 中是否已有相关文档
3. **文档命名**:按 `领域-主题-副题.md` 格式,全小写,用连字符,如:
- `backend-datasources-api-performance.md`
- `ops-planet-sh-startup.md`
- `earth-bgp-context.md`
```bash
ls docs/technical/zh/ # 查看现有文档
```
**先输出写作计划供用户确认**(若变更明确且范围小,可直接执行):
```
文档计划:
新建docs/technical/zh/ops-planet-sh-startup.md — planet.sh 启动性能优化
更新docs/technical/zh/backend-datasources-api-performance.md — 补充并行化细节
```
### Step 2.5 — 覆盖范围检查
写文档前必须按变更类型检查配套文档,不要只更新一篇专题文档:
- 用户可见流程变化:更新 `docs/technical/zh/manual.md`,通常也更新 `docs/technical/zh/quickstart.md`
- `manual.md``quickstart.md` 这类用户手册存在英文版时,同步更新 `docs/technical/en/...`,至少避免英文版与中文版互相矛盾。
- 控制台页面职责、路由入口、表格/抽屉/设置页行为变化:更新 `docs/technical/zh/frontend-admin-frontend-context.md`
- Earth 前端行为、HUD、巡航、图层、图例、交互变化更新 `docs/technical/zh/earth-frontend-context.md`
- 新增 Earth 图层、调整 `renderOrder`、半径/高度偏移、深度策略、拾取策略、legend mode、图层面板顺序或启动加载顺序更新 `docs/technical/zh/earth-render-layer-order.md`
- Earth 图层视觉样式、颜色、图例符号语义变化:若影响样式索引,同步更新 `docs/technical/zh/earth-layer-style-reference.md`
- 采集器、数据源、凭证、设置页、连接检查、scheduler、后端 API 变化:更新相关后端文档,优先检查 `docs/technical/zh/backend-collectors.md` 和 datasource/settings 专题文档。
- 如果某个旧 plan 的假设已经被当前实现推翻,在对应 `docs/plans/*.md` 增加现状修正或更新该段,不要让计划文档继续给出相反方向。
- 新增 technical 文档后,如果需要被发现,更新 `docs/technical/zh/README.md`
- 如果 technical 文档需要在公开 Docs 页面显示,或从 technical README 链接进入,必须同步更新 `frontend/src/pages/Docs/docs-content.ts``DOCS_METADATA`。前端使用这份白名单,`docs/technical/{zh,en}/` 中存在 `.md` 文件并不会自动生成路由。
- 公开 technical 文档必须按同名文件维护中英文双语版本:`docs/technical/zh/<name>.md``docs/technical/en/<name>.md`。如果某篇文档刻意只保留单语,完成说明中必须明确写出原因。
- 对本次变更提取旧词做 stale search例如旧 tab 名、旧路由职责、旧认证假设、改名前 UI 文案:
```bash
rg -n "旧文案|旧路由职责|旧认证假设" docs/technical docs/plans
```
### Step 3 — 写文档
遵循以下原则:
**记录 WHY不只记录 WHAT**
- 好:`将戳文件从 /tmp 移到 ~/.cache/planet/,因为 WSL 重启后 /tmp 被清空`
- 差:`修改了 AI_PROVIDER_BUILD_STAMP_FILE 的值`
**必须包含的内容**
- 背景/问题:改动之前存在什么问题,为什么要改
- 核心设计决策及其理由
- 关键代码片段(用 diff 或 before/after 展示)
- 相关文件列表
**格式要求**
- 使用 `##``###` 分级,不要超过三级
- 代码块注明语言python / bash / typescript / sql
- 表格用于对比多个选项或列出参数
- 中文写作,技术术语保留英文原文
- `docs/technical/zh/` 中的文档不得用英文原文占位;如果存在 `docs/technical/en/` 对应文件,禁止逐字复制成中文文件
- 中文文档内部链接应指向 `docs/technical/zh/...`,除非明确引用英文专属文档
- 公开文档的 Markdown 链接显示文字应使用可读标题,不要直接暴露 `manual.md``earth-frontend-context.md` 这类裸文件名
**文档结构模板**
```markdown
# 标题(说明做了什么)
## 背景
为什么要做这个改动,改动前存在什么问题。
## 核心变更
### 子主题一
before/after 或决策说明 + 关键代码
### 子主题二
...
## 相关文件
- `path/to/file.py` — 简短说明
```
### Step 4 — 验证
- 读一遍写好的文档,确认逻辑清晰、代码片段无明显错误
-`rg --files``test -e` 确认文档中的文件路径在项目中真实存在,避免凭记忆判断:
- 检查中文文档没有误复制英文版:
```bash
python - <<'PY'
from pathlib import Path
same = []
for en in sorted(Path("docs/technical/en").glob("*.md")):
zh = Path("docs/technical/zh") / en.name
if zh.exists() and en.read_text() == zh.read_text():
same.append(en.name)
if same:
raise SystemExit("identical en/zh docs: " + ", ".join(same))
print("no identical en/zh docs")
PY
```
- 检查中文文档内部链接没有继续指向无语言目录:
```bash
rg -n "/home/ray/dev/linkong/planet/docs/technical/(?!zh|en)" docs/technical/zh --pcre2
```
- 检查公开文档链接已进入 Docs 前端白名单。凡是 `docs/technical/{zh,en}/README.md` 中链接到的 technical `.md`,都必须存在于 `DOCS_METADATA`
```bash
python - <<'PY'
import re
from pathlib import Path
metadata = Path("frontend/src/pages/Docs/docs-content.ts").read_text()
known = set(re.findall(r"'([^']+\.md)':\s*\{", metadata))
known.add("README.md")
missing = []
for readme in [Path("docs/technical/zh/README.md"), Path("docs/technical/en/README.md")]:
if not readme.exists():
continue
for href in re.findall(r"\]\(([^)]+\.md)\)", readme.read_text()):
path = Path(href)
if "docs/technical/" not in href:
continue
filename = path.name
if filename not in known:
missing.append(f"{readme}: {filename}")
if missing:
raise SystemExit("docs README links missing DOCS_METADATA: " + ", ".join(missing))
print("docs README links are whitelisted")
PY
```
- 检查公开文档双语同名文件齐备。除 `README.md` 外,所有白名单文档都应同时存在 zh/en 文件,除非本次说明中明确豁免:
```bash
python - <<'PY'
import re
from pathlib import Path
metadata = Path("frontend/src/pages/Docs/docs-content.ts").read_text()
filenames = sorted(set(re.findall(r"'([^']+\.md)':\s*\{", metadata)) - {"README.md"})
missing = []
for filename in filenames:
for lang in ("zh", "en"):
path = Path("docs/technical") / lang / filename
if not path.exists():
missing.append(str(path))
if missing:
raise SystemExit("missing bilingual docs: " + ", ".join(missing))
print("public docs have zh/en file pairs")
PY
```
- 检查公开文档里没有用裸 `.md` 文件名当链接标题。这个命令在 polished public docs 中应无输出:
```bash
rg -n "\[[^]]+\.md\]\(" docs/technical/zh docs/technical/en
```
```bash
# 对文档中提到的关键路径做快速验证
ls <mentioned_paths>
```
如需检查大量链接,优先用确定性提取:
```bash
rg -n "\]\(([^)]+)\)" docs/technical/zh/<doc>.md
```
### Step 5 — 完成确认
输出摘要:
```
✓ 新建docs/technical/zh/ops-planet-sh-startup.md约 xxx 字)
✓ 更新docs/technical/zh/backend-datasources-api-performance.md
```
## 注意事项
- 不要写流水账式的"改了 A、改了 B、改了 C",要写改动背后的约束和权衡
- 不要在文档中引用 PR 号、issue 号、或当前对话——这些会随时间失效
- 代码片段保持简洁,只保留说明问题的关键部分,省略无关样板代码
- 如果某个变更已有文档记录,优先在原文档中追加,而不是新建
- 公开 technical 文档没有注册 `DOCS_METADATA`Docs 页面不会显示;不要只创建 `.md` 文件就结束。
- 公开 technical 文档默认需要 zh/en 同名文件,不要只补一个语言版本。
- 链接可见文字使用文档标题或语义标题,不要使用裸文件名。
- 文档是给未来的开发者看的,假设读者熟悉项目但不了解这次改动的背景

View File

@@ -72,6 +72,8 @@ Verification
## 执行风格
- 重证据,轻口头判断
- 优先使用确定性工具证据:`rg``git diff --stat``git diff -- <path>`、测试、构建、lint、`curl`、数据库查询等能直接证明成功标准的方式
- 不把大段命令输出粘进回复;保留在工具调用里,回复只总结关键证据
- 重验收,轻自我感觉
- 优先用测试、日志、产物、对比结果来证明完成
- 对长期任务保持“未达标就继续”的节奏

View File

@@ -28,6 +28,19 @@ allowed-tools: ["Read", "Edit", "Bash", "Glob", "Grep"]
- `docs/CHANGELOG.md`
- `docs/version-history.md`
## 节省上下文规则
发版判断应以确定性 CLI 证据为主,优先使用紧凑命令和定点读取:
```bash
git status --short
git diff --stat HEAD
git diff --name-only HEAD
rg -n "version|^## |^Released:|当前开发版本|current" VERSION frontend/package.json pyproject.toml docs/CHANGELOG.md docs/version-history.md
```
除非需要判断某个代码变更是否属于本次发版,否则不要读取完整 diff。
## 执行步骤
### Step 1 — 环境检查
@@ -45,7 +58,7 @@ cat VERSION # 读取当前版本
### Step 2 — 确定发版类型与新版本号
-`$ARGUMENTS` 提供了明确类型(`feature` / `bugfix`),直接使用
- 否则根据当前 `git diff HEAD``git log` 推断
- 否则根据 `git diff --stat HEAD``git diff --name-only HEAD`、必要的 focused diff`git log` 推断
- 计算新版本号(例:`0.26.2` → bugfix → `0.26.3`
- **先输出发版计划供用户确认**
@@ -91,12 +104,13 @@ cat VERSION # 读取当前版本
针对本次变更范围做最小验证:
- Python 文件有修改:`python3 -m py_compile <changed_files>`
- Frontend 文件有修改:运行项目标准检查(若无则跳过并说明)
- Python 文件有修改:先用 `git diff --name-only HEAD -- '*.py'` 列出,再运行 `python3 -m py_compile <changed_files>`
- Frontend 文件有修改:先用 `git diff --name-only HEAD -- frontend` 判断范围,再运行项目标准检查(若无则跳过并说明)
- 版本号一致性检查:用 grep 确认 VERSION、package.json、pyproject.toml 中的版本号完全一致
```bash
grep -h "version" VERSION frontend/package.json pyproject.toml
cat VERSION
rg -n "\"version\":|^version =|version = " frontend/package.json pyproject.toml uv.lock
```
### Step 7 — 提交前预览

View File

@@ -21,6 +21,19 @@ If the user specifies a file or directory, check only that. Otherwise check all
Only report issues present in **newly added or modified** lines of this diff — do not audit unchanged code.
## Token-Saving Rule
Prefer deterministic CLI checks before reading files into model context:
```bash
git diff --name-only HEAD
git diff --unified=0 HEAD -- <path>
git diff --check
rg -n "TODO|FIXME|console\.log|debugger|print\(" <changed-paths>
```
Read full files only when the focused diff does not provide enough surrounding context to make a safe edit.
## Checklist
### 1. Duplicate Logic
@@ -64,7 +77,13 @@ Filter to the user-specified path if one was provided.
### Step 2 — Read and analyze each file
Read the full file (not just the diff) with the Read tool. For each file, record every issue found: filename, line number, category, and suggested fix.
Start with focused diffs:
```bash
git diff --unified=0 HEAD -- <file>
```
Use `rg`, `git diff --check`, and compiler/linter output for deterministic findings. Read the full file only for files that need surrounding context. For each issue found, record filename, line number, category, and suggested fix.
### Step 3 — Report findings before touching anything
@@ -99,6 +118,7 @@ Principles:
- Only fix issues identified in the checklist — no extra improvements
- Keep each Edit as small as possible
- After fixing, verify the old bad pattern is gone with grep
- Prefer `apply_patch` for targeted edits; use formatters only when the repository already uses them for the touched file type
### Step 5 — Summary

200
.codex/skills/docs/SKILL.md Normal file
View File

@@ -0,0 +1,200 @@
---
name: docs
description: Analyze current Planet repo changes and create or update technical documentation under docs/technical/zh. Use when the user asks to write docs, update technical docs, summarize implementation changes into documentation, or port the Claude docs-codex workflow into Codex.
---
# Docs
Use this skill when the user asks to create or update Planet technical documentation, especially under `docs/technical/zh/`.
## Goal
Write or update technical docs that explain why a change exists, not only what files changed.
Default target directory:
- `docs/technical/zh/`
## Workflow
1. Gather change context:
```bash
git diff HEAD --stat
git diff HEAD --name-only
git log --oneline -10
ls docs/technical/zh/
```
If the user gives a specific topic, focus on that topic. Otherwise infer the documentation topic from the file list and diff stat. Do **not** read the full repository diff by default; inspect focused diffs only for the files that define the doc topic:
```bash
git diff HEAD -- <path>
rg -n "class |def |function |export |router|@router|interface |type " <path>
```
2. Decide document scope:
- Use one document for one coherent topic.
- Split documents when the changes cross meaningful domains, such as backend performance and ops startup behavior.
- Prefer updating an existing relevant doc over creating a duplicate.
- Name new files as lowercase hyphenated `domain-topic-detail.md`, for example:
- `backend-datasources-api-performance.md`
- `ops-planet-sh-startup.md`
- `earth-bgp-context.md`
3. Apply the documentation coverage checklist before writing:
- User-visible workflow changes must update `docs/technical/zh/manual.md` and usually `docs/technical/zh/quickstart.md`.
- If an English counterpart exists for user-facing docs such as `manual.md` or `quickstart.md`, update `docs/technical/en/...` enough that it does not contradict the Chinese source.
- Control console page responsibility changes must update `docs/technical/zh/frontend-admin-frontend-context.md`.
- Earth frontend behavior changes must update `docs/technical/zh/earth-frontend-context.md`.
- Earth layer additions, `renderOrder`, altitude/radius offsets, depth strategy, pointer picking, legend modes, or layer panel/startup ordering must update `docs/technical/zh/earth-render-layer-order.md`.
- Earth layer visual style or legend symbol/color semantics should also update `docs/technical/zh/earth-layer-style-reference.md` when that reference is affected.
- Collector, datasource, credential, settings, connectivity, scheduler, or API changes must update the relevant backend docs, especially `docs/technical/zh/backend-collectors.md` and any datasource/settings-specific doc.
- When a change turns an old plan assumption into current behavior, update the relevant `docs/plans/*.md` with a status note instead of leaving contradictory instructions.
- If adding a new technical document, add it to `docs/technical/zh/README.md` when it should be discoverable from the technical docs index.
- If a technical document should be visible in the public Docs page or linked from a technical README, register it in `frontend/src/pages/Docs/docs-content.ts` under `DOCS_METADATA`. The frontend uses this whitelist; files under `docs/technical/{zh,en}/` are not automatically routable.
- For every public technical doc, keep the bilingual file pair in sync by filename: `docs/technical/zh/<name>.md` and `docs/technical/en/<name>.md`. If the content is intentionally Chinese-only or English-only, state that intentionally in the final note.
- Search docs for stale terms introduced by the change, for example old tab names, old route responsibilities, obsolete auth assumptions, or renamed UI labels.
4. Write the doc in Chinese:
- Write Chinese prose for `docs/technical/zh/`.
- Keep technical identifiers, API paths, config keys, code symbols, and standard product names in English where appropriate.
- Use `##` and `###` headings; avoid going deeper than three levels.
- Use fenced code blocks with language tags.
- Use tables when comparing options or listing parameters.
5. Required content:
- Background/problem: what was wrong before and why the change was needed.
- Core design decisions and rationale.
- Key code snippets, preferably before/after or focused excerpts.
- Related files and what each file contributes.
6. Verification:
- Read the completed doc and check that the reasoning is clear.
- Verify important referenced paths exist.
- Use `rg --files` or `test -e` for path existence instead of relying on memory.
- Run a quick duplicate-language check when editing bilingual docs:
```bash
python - <<'PY'
from pathlib import Path
same = []
for en in sorted(Path("docs/technical/en").glob("*.md")):
zh = Path("docs/technical/zh") / en.name
if zh.exists() and en.read_text() == zh.read_text():
same.append(en.name)
if same:
raise SystemExit("identical en/zh docs: " + ", ".join(same))
print("no identical en/zh docs")
PY
```
Also check that Chinese docs do not link to the old language-less technical docs path:
```bash
rg -n "/home/ray/dev/linkong/planet/docs/technical/(?!zh|en)" docs/technical/zh --pcre2
```
This command should return no matches.
Check that public docs are whitelisted in the frontend Docs registry. Any `.md` linked from `docs/technical/{zh,en}/README.md` and located under `docs/technical/{zh,en}/` must have a matching `DOCS_METADATA` key:
```bash
python - <<'PY'
import re
from pathlib import Path
metadata = Path("frontend/src/pages/Docs/docs-content.ts").read_text()
known = set(re.findall(r"'([^']+\.md)':\s*\{", metadata))
known.add("README.md")
missing = []
for readme in [Path("docs/technical/zh/README.md"), Path("docs/technical/en/README.md")]:
if not readme.exists():
continue
for href in re.findall(r"\]\(([^)]+\.md)\)", readme.read_text()):
path = Path(href)
if "docs/technical/" not in href:
continue
filename = path.name
if filename not in known:
missing.append(f"{readme}: {filename}")
if missing:
raise SystemExit("docs README links missing DOCS_METADATA: " + ", ".join(missing))
print("docs README links are whitelisted")
PY
```
Check bilingual parity for public docs. Every whitelisted document except `README.md` should exist in both language directories unless intentionally documented otherwise:
```bash
python - <<'PY'
import re
from pathlib import Path
metadata = Path("frontend/src/pages/Docs/docs-content.ts").read_text()
filenames = sorted(set(re.findall(r"'([^']+\.md)':\s*\{", metadata)) - {"README.md"})
missing = []
for filename in filenames:
for lang in ("zh", "en"):
path = Path("docs/technical") / lang / filename
if not path.exists():
missing.append(str(path))
if missing:
raise SystemExit("missing bilingual docs: " + ", ".join(missing))
print("public docs have zh/en file pairs")
PY
```
Check that Markdown links do not expose raw filenames as user-facing titles. This should return no matches for polished public docs:
```bash
rg -n "\[[^]]+\.md\]\(" docs/technical/zh docs/technical/en
```
If checking many links, prefer deterministic extraction:
```bash
rg -n "\]\(([^)]+)\)" docs/technical/zh/<doc>.md
```
Also run focused stale-term searches derived from the change, for example:
```bash
rg -n "old label|old route purpose|obsolete provider assumption" docs/technical docs/plans
```
## Hard Constraints
- A file under `docs/technical/zh/` must not be an English source file copied as a placeholder.
- Do not leave a Chinese doc with only an English title and English first-screen content.
- When an English counterpart exists in `docs/technical/en/`, never duplicate it byte-for-byte into `docs/technical/zh/`.
- Internal links inside `docs/technical/zh/` should point to `docs/technical/zh/...` for Chinese docs, unless intentionally linking to an English-only file.
- Public technical documents must be registered in `frontend/src/pages/Docs/docs-content.ts` before considering them available in the Docs UI.
- Public technical documents should have both zh and en files with the same filename, unless intentionally exempted.
- Markdown link text in public docs should be a readable title, not a raw filename such as `manual.md`.
- Do not reference PR numbers, issue numbers, or the current conversation.
- Do not write changelog-style lists like "changed A, changed B, changed C" without the constraints and tradeoffs behind those changes.
- Keep code snippets concise and relevant.
## Recommended Output
After editing, summarize:
```md
Updated:
- docs/technical/zh/example.md — what changed
Verified:
- no identical en/zh docs
- no language-less docs/technical links in zh docs
- public docs are registered in DOCS_METADATA
- public docs have zh/en file pairs
- no raw `.md` filenames as public link titles
```

View File

@@ -72,6 +72,8 @@ In Codex, only use actual subagents when the user explicitly asks for delegation
## Operating Rules
- Prefer objective checks over self-reported completion.
- Prefer deterministic tool evidence over long model summaries: use `rg`, `git diff --stat`, targeted `git diff -- <path>`, tests, builds, linters, `curl`, or database queries when they can prove a criterion.
- Do not paste large command output into the conversation; summarize the evidence and keep raw output in tool calls.
- Do not confuse progress with completion.
- If the worker says "done", verify it.
- If verification fails, continue from the gap instead of restarting blindly.

View File

@@ -35,6 +35,19 @@ Use `git rev-parse --show-toplevel` to get the repo root. All paths are relative
- `docs/CHANGELOG.md`
- `docs/version-history.md`
## Token-Saving Rule
Release work should be driven by deterministic CLI evidence. Prefer compact commands and targeted file reads:
```bash
git status --short
git diff --stat HEAD
git diff --name-only HEAD
rg -n "version|^## |^Released:|current" VERSION frontend/package.json pyproject.toml docs/CHANGELOG.md docs/version-history.md
```
Do not inspect full diffs unless deciding whether changed code belongs in the release.
## Workflow
### Step 1 — Environment check
@@ -52,7 +65,7 @@ If unrelated uncommitted changes exist, list them and ask the user whether to in
### Step 2 — Determine release type and next version
- If the user provided an explicit type (`feature` / `bugfix`), use it
- Otherwise infer from `git diff HEAD` and recent `git log`
- Otherwise infer from `git diff --stat HEAD`, `git diff --name-only HEAD`, focused diffs for changed code, and recent `git log`
- Compute the next version:
- `feature`: increment minor and reset patch to `0` (e.g. `0.41.2``0.42.0`)
- `bugfix`: increment patch only (e.g. `0.26.2``0.26.3`)
@@ -106,12 +119,13 @@ Get today's date with `date +%Y-%m-%d`.
Run the smallest relevant validation for the changes in scope:
- Python files changed: `python3 -m py_compile <changed_files>`
- Frontend files changed: run the project-standard check if available; otherwise skip and say so
- Python files changed: list changed Python files with `git diff --name-only HEAD -- '*.py'`, then run `python3 -m py_compile <changed_files>`
- Frontend files changed: list changed frontend files with `git diff --name-only HEAD -- frontend`, then run the project-standard check if available; otherwise skip and say so
- Version consistency: confirm VERSION, package.json, pyproject.toml, and uv.lock all show the same version
```bash
grep -h "version" VERSION frontend/package.json pyproject.toml
cat VERSION
rg -n "\"version\":|^version =|version = " frontend/package.json pyproject.toml uv.lock
```
### Step 7 — Pre-commit preview

13
.dockerignore Normal file
View File

@@ -0,0 +1,13 @@
**
!pyproject.toml
!uv.lock
!aiprovider/
!aiprovider/**
aiprovider/.env
aiprovider/.env.*
!aiprovider/.env.example
**/__pycache__/
**/*.pyc
**/*.pyo

View File

@@ -23,6 +23,7 @@
- [ ] 保持 Earth 当前这批纯个人偏好设置继续走本地持久化:`旋转模式`、HUD 面板显示/隐藏、`地形透明度` 暂不升级到后端系统设置,避免把设备级偏好过早做成全局配置
- [ ] 如果后续明确需要“账号级同步 Earth 偏好”,再单独设计 `Earth user preferences`:优先按用户维度而不是全局系统设置保存,并规划 `localStorage -> backend` 的平滑迁移策略
- [ ] 为 Planet / Earth 补一个可用的日志查看系统:先明确前后端/AI Provider/采集任务的日志入口、最近日志聚合、筛选与 tail 能力,再决定是先做脚本级统一入口还是控制台内置日志面板
- [ ] 重写控制台 UI逐步抛弃 Ant Design建立自有组件体系并统一采用 `tabler.io` / Tabler Icons 作为控制台主图标库
- [ ] 把 Earth 态势新闻源从 [earth_news.py](/home/ray/dev/linkong/planet/backend/app/services/earth_news.py) 的硬编码列表抽成可配置目录,优先保持当前“实时聚合”链路不变,只先解决新闻源不可配置的问题
- [ ] 为 Earth 态势新闻设计后续采集器化方案:明确新闻数据模型、去重策略、区域映射、过期清理和 Earth/AI 复用方式,再决定何时把新闻从实时抓取升级成正式 collector
- [ ] 为 Earth 地球表面增加一层与基础纹理对齐的材质/纹理 overlay并在同层叠加国界轮廓参考线要求国界线与底图稳定对齐且 hover 到国家轮廓时能高亮当前国家,便于校准地表和增强交互

View File

@@ -1 +1 @@
0.42.1
0.46.3

View File

@@ -1,3 +1,5 @@
# syntax=docker/dockerfile:1.7
ARG PYTHON_IMAGE=python:3.14-slim
ARG UV_IMAGE=ghcr.io/astral-sh/uv:latest
@@ -18,9 +20,10 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
&& rm -rf /var/lib/apt/lists/*
COPY pyproject.toml uv.lock /app/
RUN uv sync --frozen --no-dev
RUN --mount=type=cache,target=/root/.cache/uv \
uv sync --frozen --no-dev
COPY . /app
COPY aiprovider /app/aiprovider
EXPOSE 8010

View File

@@ -37,8 +37,26 @@ def verify_service_token(x_provider_token: str | None = Header(default=None)) ->
)
def get_provider_service() -> ProviderService:
return ProviderService()
def get_provider_service(
x_ai_provider: str | None = Header(default=None),
x_ai_provider_api: str | None = Header(default=None),
x_ai_base_url: str | None = Header(default=None),
x_ai_api_key: str | None = Header(default=None),
x_ai_model: str | None = Header(default=None),
x_ai_max_tokens: str | None = Header(default=None),
x_ai_anthropic_version: str | None = Header(default=None),
) -> ProviderService:
overrides = {
"provider": x_ai_provider,
"provider_api": x_ai_provider_api,
"base_url": x_ai_base_url,
"api_key": x_ai_api_key,
"model": x_ai_model,
"anthropic_version": x_ai_anthropic_version,
}
if x_ai_max_tokens:
overrides["max_tokens"] = x_ai_max_tokens
return ProviderService({key: value for key, value in overrides.items() if value not in (None, "")})
@app.get("/health")

View File

@@ -46,19 +46,22 @@ def _resolve_provider_api(provider: str, configured_api: str) -> str:
class ProviderService:
def __init__(self) -> None:
self.provider = _normalize_provider(settings.AI_PROVIDER)
def __init__(self, overrides: dict[str, Any] | None = None) -> None:
overrides = overrides or {}
self.provider = _normalize_provider(overrides.get("provider") or settings.AI_PROVIDER)
self.provider_api = _resolve_provider_api(
self.provider,
_normalize_provider_api(settings.AI_PROVIDER_API),
_normalize_provider_api(overrides.get("provider_api") or settings.AI_PROVIDER_API),
)
self.base_url = settings.AI_BASE_URL.rstrip("/")
self.api_key = settings.AI_API_KEY
self.default_model = settings.AI_MODEL
self.base_url = str(overrides.get("base_url") or settings.AI_BASE_URL).rstrip("/")
self.api_key = str(overrides.get("api_key") or settings.AI_API_KEY)
self.default_model = str(overrides.get("model") or settings.AI_MODEL)
self.timeout = settings.AI_TIMEOUT_SECONDS
self.http_retry_attempts = max(settings.AI_HTTP_RETRY_ATTEMPTS, 1)
self.max_tokens = settings.AI_MAX_TOKENS
self.anthropic_version = settings.AI_ANTHROPIC_VERSION
self.max_tokens = int(overrides.get("max_tokens") or settings.AI_MAX_TOKENS)
self.anthropic_version = str(
overrides.get("anthropic_version") or settings.AI_ANTHROPIC_VERSION
)
self.system_prompt = settings.AI_ANALYSIS_SYSTEM_PROMPT
def get_status(self) -> AIProviderStatusResponse:

View File

@@ -1,20 +1,41 @@
"""DataSourceConfig API for user-defined data sources"""
from typing import Optional
from typing import Any, Optional
from datetime import datetime
import base64
import json
import re
from fastapi import APIRouter, Depends, HTTPException, status
from sqlalchemy import select, func
from sqlalchemy.ext.asyncio import AsyncSession
from pydantic import BaseModel, Field
import httpx
from app.core.target_schema_registry import get_target_schema, list_target_schemas
from app.core.datasource_defaults import DEFAULT_DATASOURCES
from app.db.session import get_db
from app.models.user import User
from app.models.datasource_config import DataSourceConfig
from app.models.datasource_mapping import DataSourceMappingTemplate
from app.core.security import get_current_user
from app.core.cache import cache
from app.core.time import to_iso8601_utc
from app.schemas.ai import SituationalAnalysisRequest
from app.services.ai_client import AIProviderClient, get_ai_provider_client
from app.services.datasource_mapping import (
MappingError,
build_heuristic_mapping,
execute_mapping,
persist_mapped_records,
redact_for_llm,
stable_payload_hash,
)
from app.services.datasource_connectivity import (
get_builtin_connection_status,
save_connectivity_success,
strip_connectivity_validation,
test_builtin_connectivity,
)
router = APIRouter()
@@ -59,6 +80,70 @@ class DataSourceConfigResponse(BaseModel):
from_attributes = True
def _is_builtin_config_name(name: str | None) -> bool:
return bool(name and name in DEFAULT_DATASOURCES)
async def _ensure_builtin_connection_verified(
db: AsyncSession,
config_data: DataSourceConfigCreate,
) -> None:
if not _is_builtin_config_name(config_data.name):
return
status_result = await get_builtin_connection_status(
db,
config_data.name,
config_data.endpoint,
config_data.auth_type,
config_data.headers,
config_data.config,
)
if not status_result.get("connected"):
raise HTTPException(
status_code=400,
detail=status_result.get("message") or "请先完成连接验证,再保存内置采集器配置。",
)
class CustomSampleRequest(BaseModel):
datasource_config_id: Optional[int] = None
config: Optional[DataSourceConfigCreate] = None
limit_bytes: int = Field(default=200000, ge=1000, le=1000000)
class MappingProposeRequest(BaseModel):
sample_payload: Any
target_schema: str
use_ai: bool = True
class MappingPreviewRequest(BaseModel):
sample_payload: Any
target_schema: str
mapping_json: dict
limit: int = Field(default=20, ge=1, le=100)
class MappingTemplateCreate(BaseModel):
datasource_config_id: int
target_schema: str
mapping_json: dict
sample_payload: Any | None = None
sample_payload_hash: Optional[str] = None
validation_status: str = Field(default="draft", pattern="^(draft|valid|invalid)$")
is_active: bool = False
class MappingTemplateUpdate(BaseModel):
target_schema: Optional[str] = None
mapping_json: Optional[dict] = None
sample_payload: Any | None = None
sample_payload_hash: Optional[str] = None
validation_status: Optional[str] = Field(default=None, pattern="^(draft|valid|invalid)$")
is_active: Optional[bool] = None
async def test_endpoint(
endpoint: str,
auth_type: str,
@@ -96,6 +181,134 @@ async def test_endpoint(
}
def _build_request_headers(auth_type: str, auth_config: dict, headers: dict) -> dict[str, str]:
request_headers = {str(key): str(value) for key, value in (headers or {}).items()}
auth_type = str(auth_type or "none").lower()
auth_config = auth_config or {}
if auth_type == "bearer" and auth_config.get("token"):
request_headers["Authorization"] = f"Bearer {auth_config['token']}"
elif auth_type == "api_key" and auth_config.get("api_key"):
location = str(auth_config.get("in") or auth_config.get("location") or "header").lower()
if location != "query":
key_name = auth_config.get("key_name", "X-API-Key")
request_headers[str(key_name)] = str(auth_config["api_key"])
elif auth_type == "basic":
username = auth_config.get("username", "")
password = auth_config.get("password", "")
credentials = f"{username}:{password}"
encoded = base64.b64encode(credentials.encode()).decode()
request_headers["Authorization"] = f"Basic {encoded}"
return request_headers
def _build_query_params(auth_type: str, auth_config: dict, config: dict) -> dict[str, Any]:
params = {}
candidate = (config or {}).get("params") or (config or {}).get("query_params")
if isinstance(candidate, dict):
params.update(candidate)
auth_type = str(auth_type or "none").lower()
auth_config = auth_config or {}
if auth_type == "api_key" and auth_config.get("api_key"):
location = str(auth_config.get("in") or auth_config.get("location") or "header").lower()
if location == "query":
key_name = auth_config.get("key_name") or auth_config.get("param_name") or "api_key"
params[str(key_name)] = auth_config["api_key"]
return params
async def fetch_custom_sample_from_config(config: DataSourceConfig, limit_bytes: int) -> Any:
request_config = config.config or {}
method = str(request_config.get("method") or request_config.get("request_method") or "GET").upper()
if method not in {"GET", "POST"}:
raise HTTPException(status_code=400, detail="Only GET and POST sample requests are supported.")
headers = _build_request_headers(config.auth_type, config.auth_config or {}, config.headers or {})
params = _build_query_params(config.auth_type, config.auth_config or {}, request_config)
timeout = float(request_config.get("timeout", 30))
json_body = request_config.get("json_body")
if json_body is None and str(request_config.get("body_type") or "").lower() in {"json", ""}:
candidate = request_config.get("body")
if isinstance(candidate, (dict, list)):
json_body = candidate
async with httpx.AsyncClient(timeout=timeout, follow_redirects=True) as client:
response = await client.request(
method,
config.endpoint,
headers=headers,
params=params or None,
json=json_body,
)
response.raise_for_status()
content = response.content[:limit_bytes]
if "application/json" in response.headers.get("content-type", ""):
return json.loads(content.decode(response.encoding or "utf-8"))
return {"text": content.decode(response.encoding or "utf-8", errors="replace")}
def _parse_mapping_from_ai_text(content: str) -> dict[str, Any] | None:
if not content:
return None
candidates = [content]
fenced = re.findall(r"```(?:json)?\s*(\{.*?\})\s*```", content, flags=re.DOTALL)
candidates = fenced + candidates
for candidate in candidates:
try:
parsed = json.loads(candidate)
except json.JSONDecodeError:
continue
if isinstance(parsed, dict) and isinstance(parsed.get("fields"), dict):
return parsed
return None
async def _get_config_for_sample(
payload: CustomSampleRequest,
db: AsyncSession,
) -> DataSourceConfig:
if payload.datasource_config_id is not None:
result = await db.execute(
select(DataSourceConfig).where(DataSourceConfig.id == payload.datasource_config_id)
)
config = result.scalar_one_or_none()
if not config:
raise HTTPException(status_code=404, detail="Configuration not found")
return config
if payload.config is None:
raise HTTPException(status_code=400, detail="datasource_config_id or config is required")
config_data = payload.config
return DataSourceConfig(
name=config_data.name,
description=config_data.description,
source_type=config_data.source_type,
endpoint=config_data.endpoint,
auth_type=config_data.auth_type,
auth_config=config_data.auth_config,
headers=config_data.headers,
config=config_data.config,
)
def serialize_mapping_template(template: DataSourceMappingTemplate) -> dict[str, Any]:
return {
"id": template.id,
"datasource_config_id": template.datasource_config_id,
"target_schema": template.target_schema,
"mapping_json": template.mapping_json,
"sample_payload_hash": template.sample_payload_hash,
"validation_status": template.validation_status,
"version": template.version,
"is_active": template.is_active,
"created_at": to_iso8601_utc(template.created_at),
"updated_at": to_iso8601_utc(template.updated_at),
}
@router.get("/configs")
async def list_configs(
active_only: bool = False,
@@ -132,6 +345,47 @@ async def list_configs(
}
@router.get("/configs/all")
async def list_all_datasources(
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
"""List all data sources: YAML defaults + DB overrides"""
from app.core.data_sources import COLLECTOR_URL_KEYS, get_data_sources_config
config = get_data_sources_config()
db_query = await db.execute(select(DataSourceConfig))
db_configs = {c.name: c for c in db_query.scalars().all()}
result = []
for name, yaml_key in COLLECTOR_URL_KEYS.items():
yaml_url = config.get_yaml_url(name)
db_config = db_configs.get(name)
result.append(
{
"name": name,
"default_url": yaml_url,
"endpoint": db_config.endpoint if db_config else yaml_url,
"is_overridden": db_config is not None and db_config.endpoint != yaml_url
if yaml_url
else db_config is not None,
"is_active": db_config.is_active if db_config else True,
"source_type": db_config.source_type if db_config else "http",
"auth_type": db_config.auth_type if db_config else "none",
"headers": db_config.headers if db_config else {},
"config": strip_connectivity_validation(db_config.config if db_config else {}),
"config_id": db_config.id if db_config else None,
"description": db_config.description
if db_config
else f"Data source from YAML: {yaml_key}",
}
)
return {"total": len(result), "data": result}
@router.get("/configs/{config_id}")
async def get_config(
config_id: int,
@@ -176,7 +430,7 @@ async def create_config(
auth_type=config_data.auth_type,
auth_config=config_data.auth_config,
headers=config_data.headers,
config=config_data.config,
config=strip_connectivity_validation(config_data.config),
)
db.add(config)
@@ -208,6 +462,8 @@ async def update_config(
update_data = config_data.model_dump(exclude_unset=True)
for field, value in update_data.items():
if field == "config":
value = strip_connectivity_validation(value)
setattr(config, field, value)
await db.commit()
@@ -310,38 +566,366 @@ async def test_new_config(
}
@router.get("/configs/all")
async def list_all_datasources(
@router.post("/configs/builtin/connection-status")
async def get_builtin_config_connection_status(
config_data: DataSourceConfigCreate,
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
"""List all data sources: YAML defaults + DB overrides"""
from app.core.data_sources import COLLECTOR_URL_KEYS, get_data_sources_config
if not _is_builtin_config_name(config_data.name):
raise HTTPException(status_code=400, detail="Only built-in datasource configs are supported.")
config = get_data_sources_config()
return await get_builtin_connection_status(
db,
config_data.name,
config_data.endpoint,
config_data.auth_type,
config_data.headers,
config_data.config,
)
db_query = await db.execute(select(DataSourceConfig))
db_configs = {c.name: c for c in db_query.scalars().all()}
result = []
for name, yaml_key in COLLECTOR_URL_KEYS.items():
yaml_url = config.get_yaml_url(name)
db_config = db_configs.get(name)
@router.post("/configs/builtin/connect")
async def connect_builtin_config(
config_data: DataSourceConfigCreate,
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
if not _is_builtin_config_name(config_data.name):
raise HTTPException(status_code=400, detail="Only built-in datasource configs are supported.")
result.append(
result = await test_builtin_connectivity(
config_data.name,
config_data.endpoint,
config_data.auth_type,
config_data.headers,
config_data.config,
db,
)
if result.get("success") and result.get("checksum"):
validation = await save_connectivity_success(
db,
config_data.name,
result["checksum"],
result,
connected_by="connection_button",
)
await db.commit()
return {
**result,
"connected": True,
"validation": validation,
}
return {
**result,
"connected": False,
}
@router.post("/custom/sample")
async def fetch_custom_sample(
payload: CustomSampleRequest,
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
"""Fetch a sample payload for a saved or draft custom data source."""
config = await _get_config_for_sample(payload, db)
try:
sample = await fetch_custom_sample_from_config(config, payload.limit_bytes)
except httpx.HTTPStatusError as exc:
raise HTTPException(
status_code=exc.response.status_code,
detail=f"Sample request failed: HTTP {exc.response.status_code}",
) from exc
except httpx.HTTPError as exc:
raise HTTPException(status_code=502, detail=f"Sample request failed: {exc}") from exc
return {
"success": True,
"sample_payload": sample,
"sample_payload_hash": stable_payload_hash(sample),
"redacted_preview": redact_for_llm(sample),
}
@router.get("/target-schemas")
async def get_datasource_target_schemas(
current_user: User = Depends(get_current_user),
):
"""List target schemas available for custom datasource mapping."""
return {"data": list_target_schemas()}
@router.post("/mappings/propose")
async def propose_datasource_mapping(
payload: MappingProposeRequest,
current_user: User = Depends(get_current_user),
ai_client: AIProviderClient = Depends(get_ai_provider_client),
):
"""Generate a mapping draft for a sample payload and target schema."""
schema = get_target_schema(payload.target_schema)
redacted_sample = redact_for_llm(payload.sample_payload)
fallback_mapping = build_heuristic_mapping(redacted_sample, payload.target_schema)
ai_error: str | None = None
mapping = fallback_mapping
generated_by = "heuristic"
if payload.use_ai:
try:
response = await ai_client.analyze(
SituationalAnalysisRequest(
title=f"Generate datasource mapping for {schema.key}",
objective=(
"Return only JSON for a deterministic mapping DSL. "
"The JSON must contain source.items_path and fields. "
"Do not include prose or code."
),
context={
"target_schema": schema.to_dict(),
"sample_payload": redacted_sample,
"mapping_dsl_example": fallback_mapping,
},
observations=[
"Use JSONPath-like paths beginning with $.",
"Never generate executable code.",
"Use field types from the target schema.",
],
constraints=[
"Return a single JSON object.",
"Do not include credentials or secrets.",
"Mark uncertain optional fields with default null.",
],
)
)
parsed = _parse_mapping_from_ai_text(response.content)
if parsed:
mapping = parsed
generated_by = "ai_provider"
else:
ai_error = "AI provider did not return a valid mapping JSON object."
except HTTPException as exc:
ai_error = str(exc.detail)
mapping.setdefault("meta", {})
if isinstance(mapping["meta"], dict):
mapping["meta"].update(
{
"name": name,
"default_url": yaml_url,
"endpoint": db_config.endpoint if db_config else yaml_url,
"is_overridden": db_config is not None and db_config.endpoint != yaml_url
if yaml_url
else db_config is not None,
"is_active": db_config.is_active if db_config else True,
"source_type": db_config.source_type if db_config else "http",
"description": db_config.description
if db_config
else f"Data source from YAML: {yaml_key}",
"generated_by": generated_by,
"requires_review": True,
"ai_error": ai_error,
}
)
return {"total": len(result), "data": result}
return {
"target_schema": schema.to_dict(),
"mapping_json": mapping,
"sample_payload_hash": stable_payload_hash(payload.sample_payload),
"redacted_sample_payload": redacted_sample,
}
@router.post("/mappings/preview")
async def preview_datasource_mapping(
payload: MappingPreviewRequest,
current_user: User = Depends(get_current_user),
):
"""Preview deterministic mapping output for a sample payload."""
try:
preview = execute_mapping(
payload.sample_payload,
payload.mapping_json,
payload.target_schema,
limit=payload.limit,
)
except (MappingError, ValueError) as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
return {
"success": preview["failed_count"] == 0,
"preview": preview,
"sample_payload_hash": stable_payload_hash(payload.sample_payload),
}
@router.get("/mappings")
async def list_datasource_mappings(
datasource_config_id: Optional[int] = None,
active_only: bool = False,
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
"""List saved mapping templates."""
query = select(DataSourceMappingTemplate).order_by(
DataSourceMappingTemplate.datasource_config_id,
DataSourceMappingTemplate.version.desc(),
)
if datasource_config_id is not None:
query = query.where(DataSourceMappingTemplate.datasource_config_id == datasource_config_id)
if active_only:
query = query.where(DataSourceMappingTemplate.is_active.is_(True))
result = await db.execute(query)
mappings = result.scalars().all()
return {"total": len(mappings), "data": [serialize_mapping_template(item) for item in mappings]}
@router.post("/mappings")
async def create_datasource_mapping(
payload: MappingTemplateCreate,
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
"""Save a mapping template for a datasource config."""
get_target_schema(payload.target_schema)
datasource = await db.get(DataSourceConfig, payload.datasource_config_id)
if not datasource:
raise HTTPException(status_code=404, detail="Configuration not found")
if payload.sample_payload is not None:
try:
execute_mapping(payload.sample_payload, payload.mapping_json, payload.target_schema, limit=100)
except (MappingError, ValueError) as exc:
raise HTTPException(status_code=400, detail=f"Mapping validation failed: {exc}") from exc
result = await db.execute(
select(func.max(DataSourceMappingTemplate.version)).where(
DataSourceMappingTemplate.datasource_config_id == payload.datasource_config_id,
DataSourceMappingTemplate.target_schema == payload.target_schema,
)
)
next_version = int(result.scalar() or 0) + 1
if payload.is_active:
await db.execute(
DataSourceMappingTemplate.__table__.update()
.where(DataSourceMappingTemplate.datasource_config_id == payload.datasource_config_id)
.values(is_active=False)
)
template = DataSourceMappingTemplate(
datasource_config_id=payload.datasource_config_id,
target_schema=payload.target_schema,
mapping_json=payload.mapping_json,
sample_payload_hash=payload.sample_payload_hash
or (stable_payload_hash(payload.sample_payload) if payload.sample_payload is not None else None),
validation_status=payload.validation_status,
version=next_version,
is_active=payload.is_active,
)
db.add(template)
await db.commit()
await db.refresh(template)
return {"message": "Mapping template saved successfully", "data": serialize_mapping_template(template)}
@router.put("/mappings/{mapping_id}")
async def update_datasource_mapping(
mapping_id: int,
payload: MappingTemplateUpdate,
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
"""Update a mapping template in place."""
template = await db.get(DataSourceMappingTemplate, mapping_id)
if not template:
raise HTTPException(status_code=404, detail="Mapping template not found")
target_schema = payload.target_schema or template.target_schema
mapping_json = payload.mapping_json or template.mapping_json
get_target_schema(target_schema)
if payload.sample_payload is not None:
try:
execute_mapping(payload.sample_payload, mapping_json, target_schema, limit=100)
except (MappingError, ValueError) as exc:
raise HTTPException(status_code=400, detail=f"Mapping validation failed: {exc}") from exc
if payload.is_active is True:
await db.execute(
DataSourceMappingTemplate.__table__.update()
.where(DataSourceMappingTemplate.datasource_config_id == template.datasource_config_id)
.where(DataSourceMappingTemplate.id != template.id)
.values(is_active=False)
)
template.target_schema = target_schema
template.mapping_json = mapping_json
if payload.sample_payload_hash is not None:
template.sample_payload_hash = payload.sample_payload_hash
elif payload.sample_payload is not None:
template.sample_payload_hash = stable_payload_hash(payload.sample_payload)
if payload.validation_status is not None:
template.validation_status = payload.validation_status
if payload.is_active is not None:
template.is_active = payload.is_active
await db.commit()
await db.refresh(template)
return {"message": "Mapping template updated successfully", "data": serialize_mapping_template(template)}
@router.post("/{config_id}/run-mapped")
async def run_mapped_datasource(
config_id: int,
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
"""Run a saved custom datasource through its active deterministic mapping."""
datasource = await db.get(DataSourceConfig, config_id)
if not datasource:
raise HTTPException(status_code=404, detail="Configuration not found")
result = await db.execute(
select(DataSourceMappingTemplate)
.where(DataSourceMappingTemplate.datasource_config_id == config_id)
.where(DataSourceMappingTemplate.is_active.is_(True))
.order_by(DataSourceMappingTemplate.version.desc())
.limit(1)
)
mapping = result.scalar_one_or_none()
if not mapping:
raise HTTPException(status_code=404, detail="No active mapping template found")
try:
sample = await fetch_custom_sample_from_config(datasource, 5_000_000)
mapped = execute_mapping(sample, mapping.mapping_json, mapping.target_schema)
except httpx.HTTPStatusError as exc:
raise HTTPException(
status_code=exc.response.status_code,
detail=f"Datasource request failed: HTTP {exc.response.status_code}",
) from exc
except httpx.HTTPError as exc:
raise HTTPException(status_code=502, detail=f"Datasource request failed: {exc}") from exc
except (MappingError, ValueError) as exc:
raise HTTPException(status_code=400, detail=f"Mapping failed: {exc}") from exc
if mapped["failed_count"] > 0:
return {
"status": "failed",
"datasource_config_id": config_id,
"mapping_id": mapping.id,
"mapping_version": mapping.version,
"target_schema": mapping.target_schema,
"mapped_count": mapped["mapped_count"],
"failed_count": mapped["failed_count"],
"errors": mapped["errors"][:20],
}
written_count = await persist_mapped_records(
db,
datasource_name=datasource.name,
datasource_config_id=datasource.id,
target_schema=mapping.target_schema,
records=mapped["records"],
mapping_version=mapping.version,
)
return {
"status": "success",
"datasource_config_id": config_id,
"mapping_id": mapping.id,
"mapping_version": mapping.version,
"target_schema": mapping.target_schema,
"fetched_count": mapped["total_items"],
"mapped_count": mapped["mapped_count"],
"written_count": written_count,
}

View File

@@ -9,6 +9,7 @@ from sqlalchemy.ext.asyncio import AsyncSession
from app.core.time import to_iso8601_utc
from app.core.security import get_current_user
from app.core.data_sources import get_data_sources_config
from app.core.datasource_defaults import DEFAULT_DATASOURCES
from app.db.session import get_db
from app.models.collected_data import CollectedData
from app.models.data_snapshot import DataSnapshot
@@ -35,6 +36,17 @@ def format_frequency_label(minutes: int) -> str:
return f"{minutes}m"
def datasource_metadata(source: str) -> dict:
info = DEFAULT_DATASOURCES.get(source, {})
return {
"display_name": info.get("display_name") or info.get("name") or source,
"is_free": bool(info.get("is_free", True)),
"requires_credentials": bool(info.get("requires_credentials", False)),
"credential_provider": info.get("credential_provider"),
"credential_status": info.get("credential_status", "none"),
}
def is_due_for_collection(datasource: DataSource, now: datetime) -> bool:
if datasource.last_run_at is None:
return True
@@ -72,31 +84,6 @@ async def _load_latest_running_tasks(
return {task.datasource_id: task for task in result.scalars().all()}
async def _load_latest_completed_tasks(
db: AsyncSession,
datasource_ids: list[int],
) -> dict[int, CollectionTask]:
if not datasource_ids:
return {}
ranked_tasks = (
select(
CollectionTask.id.label("task_id"),
_task_rank_column(CollectionTask.completed_at),
)
.where(CollectionTask.datasource_id.in_(datasource_ids))
.where(CollectionTask.completed_at.isnot(None))
.where(CollectionTask.status.in_(("success", "failed", "cancelled")))
.subquery()
)
result = await db.execute(
select(CollectionTask)
.join(ranked_tasks, CollectionTask.id == ranked_tasks.c.task_id)
.where(ranked_tasks.c.row_num == 1)
)
return {task.datasource_id: task for task in result.scalars().all()}
async def _load_latest_task_ids(
db: AsyncSession,
datasource_ids: list[int],
@@ -123,21 +110,6 @@ async def _load_latest_task_ids(
return {datasource_id: task_id for datasource_id, task_id in result.all()}
async def _load_datasource_data_counts(
db: AsyncSession,
sources: list[str],
) -> dict[str, int]:
if not sources:
return {}
result = await db.execute(
select(CollectedData.source, func.count(CollectedData.id))
.where(CollectedData.source.in_(sources))
.group_by(CollectedData.source)
)
return {source: count for source, count in result.all()}
async def _load_datasource_endpoint_overrides(
db: AsyncSession,
sources: list[str],
@@ -161,7 +133,7 @@ async def _load_datasource_endpoint_overrides(
async def _load_datasource_list_context(
db: AsyncSession,
datasources: list[DataSource],
) -> tuple[dict[int, CollectionTask], dict[int, CollectionTask], dict[str, int], dict[str, str]]:
) -> tuple[dict[int, CollectionTask], dict[str, str]]:
datasource_ids = [datasource.id for datasource in datasources]
sources = [datasource.source for datasource in datasources]
@@ -185,10 +157,8 @@ async def _load_datasource_list_context(
if stale_datasource_ids:
running_tasks = await _load_latest_running_tasks(db, datasource_ids)
completed_tasks = await _load_latest_completed_tasks(db, datasource_ids)
data_counts = await _load_datasource_data_counts(db, sources)
endpoint_overrides = await _load_datasource_endpoint_overrides(db, sources)
return running_tasks, completed_tasks, data_counts, endpoint_overrides
return running_tasks, endpoint_overrides
async def get_datasource_record(db: AsyncSession, source_id: str) -> Optional[DataSource]:
@@ -401,27 +371,19 @@ async def list_datasources(
collector_list = []
config = get_data_sources_config()
running_tasks, completed_tasks, data_counts, endpoint_overrides = await _load_datasource_list_context(
db,
datasources,
)
running_tasks, endpoint_overrides = await _load_datasource_list_context(db, datasources)
for datasource in datasources:
running_task = running_tasks.get(datasource.id)
last_task = completed_tasks.get(datasource.id)
endpoint = endpoint_overrides.get(datasource.source) or config.get_yaml_url(
datasource.source,
)
data_count = data_counts.get(datasource.source, 0)
last_run_at = datasource.last_run_at or (last_task.completed_at if last_task else None)
last_run = to_iso8601_utc(last_run_at)
last_status = datasource.last_status or (last_task.status if last_task else None)
endpoint = endpoint_overrides.get(datasource.source) or config.get_yaml_url(datasource.source)
last_run_at = datasource.last_run_at
last_status = datasource.last_status
collector_list.append(
{
"id": datasource.id,
"source": datasource.source,
"name": datasource.name,
**datasource_metadata(datasource.source),
"module": datasource.module,
"priority": datasource.priority,
"frequency": format_frequency_label(datasource.frequency_minutes),
@@ -429,15 +391,18 @@ async def list_datasources(
"is_active": datasource.is_active,
"collector_class": datasource.collector_class,
"endpoint": endpoint,
"last_run": last_run,
"last_run": to_iso8601_utc(last_run_at),
"last_run_at": to_iso8601_utc(last_run_at),
"last_status": last_status,
"last_records_processed": last_task.records_processed if last_task else None,
"data_count": data_count,
"is_running": running_task is not None,
"task_id": running_task.id if running_task else None,
"progress": running_task.progress if running_task else None,
"phase": running_task.phase if running_task else None,
"phase_progress": running_task.phase_progress if running_task else None,
"phase_message": running_task.phase_message if running_task else None,
"phase_current": running_task.phase_current if running_task else None,
"phase_total": running_task.phase_total if running_task else None,
"phase_unit": running_task.phase_unit if running_task else None,
"records_processed": running_task.records_processed if running_task else None,
"total_records": running_task.total_records if running_task else None,
}
@@ -576,6 +541,7 @@ async def get_datasource(
return {
"id": datasource.id,
"name": datasource.name,
**datasource_metadata(datasource.source),
"module": datasource.module,
"priority": datasource.priority,
"frequency": format_frequency_label(datasource.frequency_minutes),
@@ -665,6 +631,11 @@ async def trigger_datasource(
"message": "当前采集任务尚未完成,重新触发会丢失本次未完成进度。是否强制重新采集?",
"task_id": running_task.id,
"phase": running_task.phase,
"phase_progress": running_task.phase_progress,
"phase_message": running_task.phase_message,
"phase_current": running_task.phase_current,
"phase_total": running_task.phase_total,
"phase_unit": running_task.phase_unit,
"progress": running_task.progress,
"records_processed": running_task.records_processed,
"total_records": running_task.total_records,
@@ -748,13 +719,29 @@ async def get_task_status(
task = await get_running_task(db, datasource.id)
if not task:
return {"is_running": False, "task_id": None, "progress": None, "phase": None, "status": "idle"}
return {
"is_running": False,
"task_id": None,
"progress": None,
"phase": None,
"phase_progress": None,
"phase_message": None,
"phase_current": None,
"phase_total": None,
"phase_unit": None,
"status": "idle",
}
return {
"is_running": task.status == "running",
"task_id": task.id,
"progress": task.progress,
"phase": task.phase,
"phase_progress": task.phase_progress,
"phase_message": task.phase_message,
"phase_current": task.phase_current,
"phase_total": task.phase_total,
"phase_unit": task.phase_unit,
"records_processed": task.records_processed,
"total_records": task.total_records,
"status": task.status,

View File

@@ -9,10 +9,36 @@ from sqlalchemy.ext.asyncio import AsyncSession
from app.core.security import get_current_user
from app.core.time import to_iso8601_utc
from app.core.config import settings as app_settings
from app.core.data_sources import get_data_sources_config
from app.core.datasource_defaults import DEFAULT_DATASOURCES
from app.db.session import get_db
from app.models.datasource import DataSource
from app.models.datasource_config import DataSourceConfig
from app.models.system_setting import SystemSetting
from app.models.user import User
from app.services.barentswatch import (
BarentsWatchConfig,
check_barentswatch_config,
check_barentswatch_connectivity,
get_barentswatch_datasource_record,
resolve_barentswatch_config,
)
from app.services.credential_guides import (
generate_credential_guide,
get_credential_guide,
reset_credential_guide,
)
from app.services.datasource_connectivity import (
build_builtin_connectivity_checksum,
save_connectivity_success,
)
from app.services.ai_client import AIProviderClient, get_ai_provider_client
from app.services.llm_provider_catalog import (
get_fallback_llm_provider_preset,
list_fallback_llm_provider_presets,
refresh_llm_provider_preset,
)
from app.services.scheduler import sync_datasource_job
from app.services.tv_streams import DEFAULT_TV_SETTINGS, get_tv_settings_payload, normalize_tv_settings
@@ -39,6 +65,21 @@ DEFAULT_SETTINGS = {
"password_policy": "medium",
},
"tv": DEFAULT_TV_SETTINGS,
"external_integrations": {
"ai_provider": {
"service_url": "",
"service_token": "",
"provider": "minimax",
"provider_api": "anthropic-messages",
"base_url": "https://api.minimaxi.com/anthropic",
"model": "MiniMax-M2.7",
"api_key": "",
"max_tokens": 1200,
"anthropic_version": "2023-06-01",
"timeout_seconds": 60,
"retry_attempts": 2,
}
},
}
@@ -96,6 +137,34 @@ class TVSettingsUpdate(BaseModel):
sources: list[TVStreamSourceUpdate] = Field(default_factory=list)
class AIProviderIntegrationUpdate(BaseModel):
service_url: str = ""
service_token: Optional[str] = None
provider: str = Field(default="minimax", max_length=80)
provider_api: str = Field(default="anthropic-messages", max_length=80)
base_url: str = Field(default="", max_length=500)
model: str = Field(default="", max_length=200)
api_key: Optional[str] = None
max_tokens: int = Field(default=1200, ge=1, le=200000)
anthropic_version: str = Field(default="2023-06-01", max_length=40)
timeout_seconds: int = Field(default=60, ge=5, le=600)
retry_attempts: int = Field(default=2, ge=1, le=10)
clear_service_token: bool = False
clear_api_key: bool = False
class BarentsWatchIntegrationUpdate(BaseModel):
endpoint: str = ""
client_id: str = ""
client_secret: Optional[str] = None
clear_client_secret: bool = False
class ExternalIntegrationsUpdate(BaseModel):
ai_provider: AIProviderIntegrationUpdate
barentswatch: BarentsWatchIntegrationUpdate
def merge_with_defaults(category: str, payload: Optional[dict]) -> dict:
merged = deepcopy(DEFAULT_SETTINGS[category])
if payload:
@@ -146,6 +215,151 @@ async def save_setting_payload(db: AsyncSession, category: str, payload: dict) -
return merge_with_defaults(category, record.payload)
def _mask_secret(value: Optional[str]) -> dict:
if not value:
return {"configured": False, "preview": ""}
text = str(value)
if "-" in text:
prefix = text.split("-", 1)[0] + "-"
preview = prefix + ("*" * max(len(text) - len(prefix), 1))
else:
prefix_len = min(4, len(text))
preview = text[:prefix_len] + ("*" * max(len(text) - prefix_len, 1))
return {"configured": True, "preview": preview}
async def get_runtime_ai_provider_config(db: AsyncSession) -> dict:
runtime_record = await get_setting_record(db, "external_integrations")
payload = merge_with_defaults(
"external_integrations",
runtime_record.payload if runtime_record else None,
)
ai_payload = payload.get("ai_provider") or {}
has_runtime_llm_config = bool(
runtime_record
and isinstance(runtime_record.payload, dict)
and isinstance(runtime_record.payload.get("ai_provider"), dict)
)
return {
"service_url": ai_payload.get("service_url") or app_settings.AI_PROVIDER_SERVICE_URL,
"service_token": ai_payload.get("service_token") or app_settings.AI_PROVIDER_SERVICE_TOKEN,
"timeout_seconds": int(
ai_payload.get("timeout_seconds") or app_settings.AI_PROVIDER_TIMEOUT_SECONDS
),
"retry_attempts": int(
ai_payload.get("retry_attempts") or app_settings.AI_PROVIDER_RETRY_ATTEMPTS
),
"llm_config": {
"provider": ai_payload.get("provider") or "minimax",
"provider_api": ai_payload.get("provider_api") or "anthropic-messages",
"base_url": ai_payload.get("base_url") or "https://api.minimaxi.com/anthropic",
"model": ai_payload.get("model") or "MiniMax-M2.7",
"api_key": ai_payload.get("api_key") or "",
"max_tokens": int(ai_payload.get("max_tokens") or 1200),
"anthropic_version": ai_payload.get("anthropic_version") or "2023-06-01",
} if has_runtime_llm_config else {},
}
async def get_barentswatch_config_record(db: AsyncSession) -> Optional[DataSourceConfig]:
return await get_barentswatch_datasource_record(db)
async def serialize_external_integrations(db: AsyncSession) -> dict:
ai_config = await get_runtime_ai_provider_config(db)
runtime_setting = await get_setting_record(db, "external_integrations")
display_llm_config = ai_config["llm_config"] or DEFAULT_SETTINGS["external_integrations"]["ai_provider"]
barentswatch_record = await get_barentswatch_config_record(db)
barentswatch_auth = barentswatch_record.auth_config if barentswatch_record else {}
barentswatch_auth = barentswatch_auth or {}
resolved_barentswatch = await resolve_barentswatch_config(db)
return {
"ai_provider": {
"service_url": ai_config["service_url"],
"service_token": _mask_secret(ai_config["service_token"]),
"provider": display_llm_config.get("provider") or "minimax",
"provider_api": display_llm_config.get("provider_api") or "anthropic-messages",
"base_url": display_llm_config.get("base_url") or "https://api.minimaxi.com/anthropic",
"model": display_llm_config.get("model") or "MiniMax-M2.7",
"api_key": _mask_secret(display_llm_config.get("api_key")),
"max_tokens": int(display_llm_config.get("max_tokens") or 1200),
"anthropic_version": display_llm_config.get("anthropic_version") or "2023-06-01",
"timeout_seconds": ai_config["timeout_seconds"],
"retry_attempts": ai_config["retry_attempts"],
"source": "runtime" if runtime_setting else "env",
},
"barentswatch": {
"endpoint": resolved_barentswatch.endpoint,
"client_id": barentswatch_auth.get("client_id") or resolved_barentswatch.client_id,
"client_secret": _mask_secret(
barentswatch_auth.get("client_secret") or resolved_barentswatch.client_secret
),
"source": resolved_barentswatch.credential_source,
},
}
async def save_external_integrations_payload(
db: AsyncSession,
update: ExternalIntegrationsUpdate,
) -> dict:
current_payload = await get_setting_payload(db, "external_integrations")
current_ai = current_payload.get("ai_provider") or {}
ai_payload = {
"service_url": update.ai_provider.service_url.strip()
or app_settings.AI_PROVIDER_SERVICE_URL,
"service_token": current_ai.get("service_token") or "",
"provider": update.ai_provider.provider.strip() or "minimax",
"provider_api": update.ai_provider.provider_api.strip() or "anthropic-messages",
"base_url": update.ai_provider.base_url.strip(),
"model": update.ai_provider.model.strip(),
"api_key": current_ai.get("api_key") or "",
"max_tokens": update.ai_provider.max_tokens,
"anthropic_version": update.ai_provider.anthropic_version.strip() or "2023-06-01",
"timeout_seconds": update.ai_provider.timeout_seconds,
"retry_attempts": update.ai_provider.retry_attempts,
}
if update.ai_provider.clear_service_token:
ai_payload["service_token"] = ""
elif update.ai_provider.service_token not in (None, ""):
ai_payload["service_token"] = update.ai_provider.service_token
if update.ai_provider.clear_api_key:
ai_payload["api_key"] = ""
elif update.ai_provider.api_key not in (None, ""):
ai_payload["api_key"] = update.ai_provider.api_key
await save_setting_payload(db, "external_integrations", {"ai_provider": ai_payload})
default_endpoint = get_data_sources_config().get_yaml_url("barentswatch_vessels")
barentswatch_record = await get_barentswatch_config_record(db)
if barentswatch_record is None:
barentswatch_record = DataSourceConfig(
name="barentswatch_vessels",
description="BarentsWatch Live AIS credentials",
source_type="api",
endpoint=update.barentswatch.endpoint.strip() or default_endpoint,
auth_type="oauth_client",
auth_config={},
headers={},
config={},
is_active=True,
)
db.add(barentswatch_record)
current_auth = dict(barentswatch_record.auth_config or {})
if update.barentswatch.clear_client_secret:
current_auth.pop("client_secret", None)
elif update.barentswatch.client_secret not in (None, ""):
current_auth["client_secret"] = update.barentswatch.client_secret
current_auth["client_id"] = update.barentswatch.client_id.strip()
barentswatch_record.endpoint = update.barentswatch.endpoint.strip() or default_endpoint
barentswatch_record.auth_type = "oauth_client"
barentswatch_record.auth_config = current_auth
await db.commit()
return await serialize_external_integrations(db)
def format_frequency_label(minutes: int) -> str:
if minutes % 1440 == 0:
return f"{minutes // 1440}d"
@@ -155,9 +369,11 @@ def format_frequency_label(minutes: int) -> str:
def serialize_collector(datasource: DataSource) -> dict:
defaults = DEFAULT_DATASOURCES.get(datasource.source, {})
return {
"id": datasource.id,
"name": datasource.name,
"display_name": defaults.get("display_name") or datasource.name,
"source": datasource.source,
"module": datasource.module,
"priority": datasource.priority,
@@ -167,6 +383,10 @@ def serialize_collector(datasource: DataSource) -> dict:
"last_run_at": to_iso8601_utc(datasource.last_run_at),
"last_status": datasource.last_status,
"next_run_at": to_iso8601_utc(datasource.next_run_at),
"is_free": bool(defaults.get("is_free", True)),
"requires_credentials": bool(defaults.get("requires_credentials", False)),
"credential_provider": defaults.get("credential_provider"),
"credential_status": defaults.get("credential_status", "none"),
}
@@ -243,6 +463,135 @@ async def update_tv_settings(
return {"status": "updated", "tv": normalize_tv_settings(saved)}
@router.get("/integrations")
async def get_external_integrations(
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
return {"integrations": await serialize_external_integrations(db)}
@router.get("/integrations/barentswatch/connectivity")
async def get_barentswatch_connectivity(
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
return await check_barentswatch_connectivity(db)
@router.post("/integrations/barentswatch/connect")
async def connect_barentswatch_integration(
payload: BarentsWatchIntegrationUpdate,
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
current = await resolve_barentswatch_config(db)
config = BarentsWatchConfig(
endpoint=payload.endpoint.strip() or current.endpoint,
client_id=payload.client_id.strip() or current.client_id,
client_secret=(
""
if payload.clear_client_secret
else payload.client_secret or current.client_secret
),
credential_source="draft",
endpoint_source="draft",
)
result = await check_barentswatch_config(config)
if result.get("success"):
checksum, _context = await build_builtin_connectivity_checksum(
"barentswatch_vessels",
config.endpoint,
"none",
{},
{},
db,
credential_override={
"client_id": config.client_id,
"client_secret": config.client_secret,
},
)
validation = await save_connectivity_success(
db,
"barentswatch_vessels",
checksum,
result,
connected_by="connection_button",
)
await db.commit()
return {**result, "connected": True, "validation": validation}
return {**result, "connected": False}
@router.get("/credential-guides/{provider}")
async def read_credential_guide(
provider: str,
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
try:
return {"guide": await get_credential_guide(db, provider)}
except ValueError as exc:
raise HTTPException(status_code=404, detail=str(exc)) from exc
@router.post("/credential-guides/{provider}/generate")
async def generate_provider_credential_guide(
provider: str,
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
ai_client: AIProviderClient = Depends(get_ai_provider_client),
):
try:
return {"guide": await generate_credential_guide(db, provider, ai_client)}
except ValueError as exc:
raise HTTPException(status_code=404, detail=str(exc)) from exc
@router.post("/credential-guides/{provider}/reset")
async def reset_provider_credential_guide(
provider: str,
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
try:
return {"guide": await reset_credential_guide(db, provider)}
except ValueError as exc:
raise HTTPException(status_code=404, detail=str(exc)) from exc
@router.get("/integrations/ai-provider/presets")
async def get_ai_provider_presets(
current_user: User = Depends(get_current_user),
):
return {"data": list_fallback_llm_provider_presets()}
@router.post("/integrations/ai-provider/presets/{provider}/refresh")
async def refresh_ai_provider_preset(
provider: str,
current_user: User = Depends(get_current_user),
):
try:
return {"data": await refresh_llm_provider_preset(provider)}
except ValueError as exc:
raise HTTPException(status_code=404, detail=str(exc)) from exc
except Exception as exc:
fallback = get_fallback_llm_provider_preset(provider)
fallback["refresh_error"] = str(exc)
return {"data": fallback}
@router.put("/integrations")
async def update_external_integrations(
payload: ExternalIntegrationsUpdate,
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
saved = await save_external_integrations_payload(db, payload)
return {"status": "updated", "integrations": saved}
@router.get("/collectors")
async def get_collector_settings(
current_user: User = Depends(get_current_user),
@@ -289,6 +638,7 @@ async def get_all_settings(
"notifications": setting_payloads["notifications"],
"security": setting_payloads["security"],
"tv": await get_tv_settings_payload(db),
"integrations": await serialize_external_integrations(db),
"collectors": [serialize_collector(datasource) for datasource in datasources],
"generated_at": to_iso8601_utc(datetime.now(UTC)),
}

View File

@@ -27,7 +27,9 @@ async def list_tasks(
offset = (page - 1) * page_size
query = """
SELECT ct.id, ct.datasource_id, ds.name as datasource_name, ct.status,
ct.started_at, ct.completed_at, ct.records_processed, ct.error_message
ct.started_at, ct.completed_at, ct.records_processed, ct.error_message,
ct.phase, ct.phase_progress, ct.phase_message, ct.phase_current,
ct.phase_total, ct.phase_unit, ct.total_records, ct.progress
FROM collection_tasks ct
JOIN data_sources ds ON ct.datasource_id = ds.id
WHERE 1=1
@@ -66,6 +68,14 @@ async def list_tasks(
"completed_at": to_iso8601_utc(t[5]),
"records_processed": t[6],
"error_message": t[7],
"phase": t[8],
"phase_progress": t[9],
"phase_message": t[10],
"phase_current": t[11],
"phase_total": t[12],
"phase_unit": t[13],
"total_records": t[14],
"progress": t[15],
}
for t in tasks
],
@@ -81,7 +91,9 @@ async def get_task(
result = await db.execute(
text("""
SELECT ct.id, ct.datasource_id, ds.name as datasource_name, ct.status,
ct.started_at, ct.completed_at, ct.records_processed, ct.error_message
ct.started_at, ct.completed_at, ct.records_processed, ct.error_message,
ct.phase, ct.phase_progress, ct.phase_message, ct.phase_current,
ct.phase_total, ct.phase_unit, ct.total_records, ct.progress
FROM collection_tasks ct
JOIN data_sources ds ON ct.datasource_id = ds.id
WHERE ct.id = :id
@@ -105,6 +117,14 @@ async def get_task(
"completed_at": to_iso8601_utc(task[5]),
"records_processed": task[6],
"error_message": task[7],
"phase": task[8],
"phase_progress": task[9],
"phase_message": task[10],
"phase_current": task[11],
"phase_total": task[12],
"phase_unit": task[13],
"total_records": task[14],
"progress": task[15],
}

View File

@@ -4,7 +4,7 @@ Unified API for all visualization data sources.
Returns GeoJSON format compatible with Three.js, CesiumJS, and Unreal Cesium.
"""
from datetime import UTC, datetime
from datetime import UTC, datetime, timedelta
import math
import httpx
from fastapi import APIRouter, HTTPException, Depends, Query, Response
@@ -20,6 +20,7 @@ from app.db.session import get_db
from app.models.bgp_anomaly import BGPAnomaly
from app.models.bgp_incident import BGPIncident
from app.models.collected_data import CollectedData
from app.models.vessel import VesselPosition, VesselStatic
from app.services.bgp_collectors import build_bgp_collector_coverage
from app.services.cable_graph import build_graph_from_data, CableGraph, haversine_distance
from app.services.collectors.bgp_common import RIPE_RIS_COLLECTOR_COORDS
@@ -511,7 +512,7 @@ def _normalize_capacity_band(capacity_value: Optional[float], capacity_unit: str
if unit in {"pflop/s", "pflops", "pflop"}:
normalized_tflops = capacity_value * 1000
elif unit in {"gflop/s", "gflops", "gflop"}:
normalized_tflops = capacity_value / 1000
normalized_tflops = capacity_value
else:
normalized_tflops = capacity_value
@@ -609,6 +610,108 @@ def convert_compute_centers_to_geojson(records: List[CollectedData]) -> Dict[str
return {"type": "FeatureCollection", "features": features}
VESSEL_TYPE_FILTERS = {
"cargo": lambda props: str(props.get("vessel_type_name", "")).lower() == "cargo"
or 70 <= int(props.get("vessel_type") or -1) <= 79,
"tanker": lambda props: str(props.get("vessel_type_name", "")).lower() == "tanker"
or 80 <= int(props.get("vessel_type") or -1) <= 89,
"passenger": lambda props: str(props.get("vessel_type_name", "")).lower() == "passenger"
or 60 <= int(props.get("vessel_type") or -1) <= 69,
"fishing": lambda props: str(props.get("vessel_type_name", "")).lower() == "fishing"
or int(props.get("vessel_type") or -1) == 30,
"military": lambda props: str(props.get("vessel_type_name", "")).lower() == "military"
or int(props.get("vessel_type") or -1) == 35,
"other": lambda props: str(props.get("vessel_type_name", "")).lower()
not in {"cargo", "tanker", "passenger", "fishing", "military"},
}
def convert_vessels_to_geojson(rows: List[Any]) -> Dict[str, Any]:
features = []
for position, static in rows:
if position.lat is None or position.lon is None:
continue
props = {
"mmsi": position.mmsi,
"name": getattr(static, "name", None) or f"MMSI {position.mmsi}",
"callsign": getattr(static, "callsign", None),
"imo": getattr(static, "imo", None),
"vessel_type": getattr(static, "vessel_type", None),
"vessel_type_name": getattr(static, "vessel_type_name", None) or "Other",
"flag": getattr(static, "flag", None),
"length": getattr(static, "length", None),
"width": getattr(static, "width", None),
"draught": getattr(static, "draught", None),
"sog": position.sog,
"cog": position.cog,
"heading": position.heading,
"nav_status": position.nav_status,
"received_at": to_iso8601_utc(position.received_at),
"data_type": "vessel",
}
features.append(
{
"type": "Feature",
"id": position.mmsi,
"geometry": {
"type": "Point",
"coordinates": [position.lon, position.lat],
},
"properties": props,
}
)
return {"type": "FeatureCollection", "features": features}
def _parse_bbox(value: Optional[str]) -> tuple[float, float, float, float] | None:
if not value:
return None
parts = [part.strip() for part in value.split(",")]
if len(parts) != 4:
raise HTTPException(status_code=400, detail="bbox must be lon_min,lat_min,lon_max,lat_max")
try:
lon_min, lat_min, lon_max, lat_max = [float(part) for part in parts]
except ValueError as exc:
raise HTTPException(status_code=400, detail="bbox values must be numbers") from exc
if lat_min > lat_max:
lat_min, lat_max = lat_max, lat_min
if lon_min > lon_max:
lon_min, lon_max = lon_max, lon_min
return lon_min, lat_min, lon_max, lat_max
def _matches_vessel_type(props: dict[str, Any], requested_types: set[str]) -> bool:
if not requested_types:
return True
for requested_type in requested_types:
predicate = VESSEL_TYPE_FILTERS.get(requested_type)
if predicate and predicate(props):
return True
return False
def _build_vessel_stats(features: List[dict[str, Any]]) -> dict[str, Any]:
by_type: dict[str, int] = {}
underway = 0
anchored_or_moored = 0
for feature in features:
props = feature.get("properties", {})
vessel_type = str(props.get("vessel_type_name") or "Other")
by_type[vessel_type] = by_type.get(vessel_type, 0) + 1
nav_status = props.get("nav_status")
if nav_status in (1, 5):
anchored_or_moored += 1
else:
underway += 1
return {
"total": len(features),
"by_type": by_type,
"underway": underway,
"anchored_or_moored": anchored_or_moored,
}
def convert_bgp_anomalies_to_geojson(
records: List[BGPAnomaly],
geography_hints: Optional[Dict[str, Dict[str, Any]]] = None,
@@ -1298,6 +1401,137 @@ async def get_compute_centers_geojson(
}
@router.get("/geo/vessels")
async def get_vessels_geojson(
bbox: Optional[str] = Query(
None,
description="Viewport bbox as lon_min,lat_min,lon_max,lat_max",
),
type: Optional[str] = Query(
None,
description="Comma-separated vessel types: cargo,tanker,passenger,fishing,military,other",
),
limit: int = Query(5000, ge=1, le=50000),
db: AsyncSession = Depends(get_db),
):
"""Return latest vessel positions as GeoJSON points."""
latest_times = (
select(
VesselPosition.mmsi.label("mmsi"),
func.max(VesselPosition.received_at).label("received_at"),
)
.group_by(VesselPosition.mmsi)
.subquery()
)
stmt = (
select(VesselPosition, VesselStatic)
.join(
latest_times,
(VesselPosition.mmsi == latest_times.c.mmsi)
& (VesselPosition.received_at == latest_times.c.received_at),
)
.outerjoin(VesselStatic, VesselStatic.mmsi == VesselPosition.mmsi)
.order_by(VesselPosition.received_at.desc())
.limit(limit)
)
parsed_bbox = _parse_bbox(bbox)
if parsed_bbox is not None:
lon_min, lat_min, lon_max, lat_max = parsed_bbox
stmt = stmt.where(
VesselPosition.lon >= lon_min,
VesselPosition.lon <= lon_max,
VesselPosition.lat >= lat_min,
VesselPosition.lat <= lat_max,
)
result = await db.execute(stmt)
rows = list(result.all())
geojson = convert_vessels_to_geojson(rows)
requested_types = {
item.strip().lower()
for item in (type or "").split(",")
if item.strip()
}
if requested_types:
geojson["features"] = [
feature
for feature in geojson.get("features", [])
if _matches_vessel_type(feature.get("properties", {}), requested_types)
]
features = geojson.get("features", [])
return {
**geojson,
"count": len(features),
"stats": _build_vessel_stats(features),
}
@router.get("/vessels/{mmsi}")
async def get_vessel_detail(mmsi: int, db: AsyncSession = Depends(get_db)):
latest_position_stmt = (
select(VesselPosition)
.where(VesselPosition.mmsi == mmsi)
.order_by(VesselPosition.received_at.desc())
.limit(1)
)
static = await db.get(VesselStatic, mmsi)
result = await db.execute(latest_position_stmt)
position = result.scalar_one_or_none()
if position is None:
raise HTTPException(status_code=404, detail="Vessel not found")
geojson = convert_vessels_to_geojson([(position, static)])
return {
**(geojson["features"][0]["properties"]),
"latitude": position.lat,
"longitude": position.lon,
}
@router.get("/vessels/{mmsi}/track")
async def get_vessel_track(
mmsi: int,
hours: int = Query(6, ge=1, le=24),
db: AsyncSession = Depends(get_db),
):
cutoff = datetime.now(UTC) - timedelta(hours=hours)
result = await db.execute(
select(VesselPosition)
.where(VesselPosition.mmsi == mmsi)
.where(VesselPosition.received_at >= cutoff)
.order_by(VesselPosition.received_at.asc())
)
positions = list(result.scalars().all())
if not positions:
return {
"type": "FeatureCollection",
"features": [],
"count": 0,
}
return {
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"geometry": {
"type": "LineString",
"coordinates": [[position.lon, position.lat] for position in positions],
},
"properties": {
"mmsi": mmsi,
"hours": hours,
"point_count": len(positions),
"start_at": to_iso8601_utc(positions[0].received_at),
"end_at": to_iso8601_utc(positions[-1].received_at),
},
}
],
"count": 1,
}
@router.get("/geo/bgp-anomalies")
async def get_bgp_anomalies_geojson(
severity: Optional[str] = Query(None),
@@ -1394,6 +1628,10 @@ async def get_visualization_geo_summary(db: AsyncSession = Depends(get_db)):
db,
source_filter=("ris_live_bgp", "bgpstream_bgp"),
)
vessel_count_result = await db.execute(
select(func.count(func.distinct(VesselPosition.mmsi))),
)
vessel_count = int(vessel_count_result.scalar() or 0)
return {
"generated_at": to_iso8601_utc(datetime.now(UTC)),
@@ -1402,6 +1640,7 @@ async def get_visualization_geo_summary(db: AsyncSession = Depends(get_db)):
"landing_point_count": len(landing_points.get("features", [])),
"satellite_count": len(satellites.get("features", [])),
"compute_center_count": len(compute_features),
"vessel_count": vessel_count,
"supercomputer_count": sum(
1 for feature in compute_features
if feature.get("properties", {}).get("site_type") == "supercomputer"

View File

@@ -31,6 +31,7 @@ COLLECTOR_URL_KEYS = {
"opengeofeed_prefix_geo": "opengeofeed.public_csv_url",
"nro_delegated_prefix_geo": "nro.delegated_stats_url",
"news_live_streams": "news_live_streams.channels_url",
"barentswatch_vessels": "barentswatch_vessels.url",
}

View File

@@ -94,3 +94,7 @@ news_live_streams:
streams_url: "https://iptv-org.github.io/api/streams.json"
# IPTV-org 台标 JSON
logos_url: "https://iptv-org.github.io/api/logos.json"
barentswatch_vessels:
# BarentsWatch Live AIS latest combined endpoint. Requires an AIS bearer token.
url: "https://live.ais.barentswatch.no/v1/latest/combined"

View File

@@ -4,163 +4,246 @@ DEFAULT_DATASOURCES = {
"top500": {
"id": 1,
"name": "TOP500 Supercomputers",
"display_name": "TOP500 超算榜单",
"module": "L1",
"priority": "P0",
"frequency_minutes": 240,
"is_free": True,
"requires_credentials": False,
},
"epoch_ai_gpu": {
"id": 2,
"name": "Epoch AI GPU Clusters",
"display_name": "Epoch AI GPU 集群",
"module": "L1",
"priority": "P0",
"frequency_minutes": 360,
"is_free": True,
"requires_credentials": False,
},
"huggingface_models": {
"id": 3,
"name": "HuggingFace Models",
"display_name": "Hugging Face 模型",
"module": "L2",
"priority": "P1",
"frequency_minutes": 720,
"is_free": True,
"requires_credentials": False,
},
"huggingface_datasets": {
"id": 4,
"name": "HuggingFace Datasets",
"display_name": "Hugging Face 数据集",
"module": "L2",
"priority": "P1",
"frequency_minutes": 720,
"is_free": True,
"requires_credentials": False,
},
"huggingface_spaces": {
"id": 5,
"name": "HuggingFace Spaces",
"display_name": "Hugging Face Spaces",
"module": "L2",
"priority": "P2",
"frequency_minutes": 1440,
"is_free": True,
"requires_credentials": False,
},
"peeringdb_ixp": {
"id": 6,
"name": "PeeringDB IXP",
"display_name": "PeeringDB 交换中心",
"module": "L2",
"priority": "P1",
"frequency_minutes": 1440,
"is_free": True,
"requires_credentials": False,
},
"peeringdb_network": {
"id": 7,
"name": "PeeringDB Networks",
"display_name": "PeeringDB 网络",
"module": "L2",
"priority": "P2",
"frequency_minutes": 2880,
"is_free": True,
"requires_credentials": False,
},
"peeringdb_facility": {
"id": 8,
"name": "PeeringDB Facilities",
"display_name": "PeeringDB 设施",
"module": "L2",
"priority": "P2",
"frequency_minutes": 2880,
"is_free": True,
"requires_credentials": False,
},
"telegeography_cables": {
"id": 9,
"name": "Submarine Cables",
"display_name": "海底光缆",
"module": "L2",
"priority": "P1",
"frequency_minutes": 10080,
"is_free": True,
"requires_credentials": False,
},
"telegeography_landing": {
"id": 10,
"name": "Cable Landing Points",
"display_name": "光缆登陆点",
"module": "L2",
"priority": "P2",
"frequency_minutes": 10080,
"is_free": True,
"requires_credentials": False,
},
"telegeography_systems": {
"id": 11,
"name": "Cable Systems",
"display_name": "光缆系统",
"module": "L2",
"priority": "P2",
"frequency_minutes": 10080,
"is_free": True,
"requires_credentials": False,
},
"arcgis_cables": {
"id": 15,
"name": "ArcGIS Submarine Cables",
"display_name": "ArcGIS 海底光缆",
"module": "L2",
"priority": "P1",
"frequency_minutes": 10080,
"is_free": True,
"requires_credentials": False,
},
"arcgis_landing_points": {
"id": 16,
"name": "ArcGIS Landing Points",
"display_name": "ArcGIS 登陆点",
"module": "L2",
"priority": "P1",
"frequency_minutes": 10080,
"is_free": True,
"requires_credentials": False,
},
"arcgis_cable_landing_relation": {
"id": 17,
"name": "ArcGIS Cable-Landing Relations",
"display_name": "ArcGIS 光缆登陆关系",
"module": "L2",
"priority": "P1",
"frequency_minutes": 10080,
"is_free": True,
"requires_credentials": False,
},
"fao_landing_points": {
"id": 18,
"name": "FAO Landing Points",
"display_name": "FAO 登陆点",
"module": "L2",
"priority": "P1",
"frequency_minutes": 10080,
"is_free": True,
"requires_credentials": False,
},
"spacetrack_tle": {
"id": 19,
"name": "Space-Track TLE",
"display_name": "Space-Track 轨道根数",
"module": "L3",
"priority": "P2",
"frequency_minutes": 1440,
"is_free": True,
"requires_credentials": True,
"credential_provider": "spacetrack",
"credential_status": "planned",
},
"celestrak_tle": {
"id": 20,
"name": "CelesTrak TLE",
"display_name": "CelesTrak 轨道根数",
"module": "L3",
"priority": "P2",
"frequency_minutes": 1440,
"is_free": True,
"requires_credentials": False,
},
"ris_live_bgp": {
"id": 21,
"name": "RIPE RIS Live BGP",
"display_name": "RIPE RIS 实时 BGP",
"module": "L3",
"priority": "P1",
"frequency_minutes": 15,
"is_free": True,
"requires_credentials": False,
},
"bgpstream_bgp": {
"id": 22,
"name": "CAIDA BGPStream Backfill",
"display_name": "CAIDA BGPStream 回填",
"module": "L3",
"priority": "P1",
"frequency_minutes": 360,
"is_free": True,
"requires_credentials": False,
},
"iptoasn_prefix_geo": {
"id": 23,
"name": "IPtoASN Prefix Geography",
"display_name": "IPtoASN 前缀地理",
"module": "L3",
"priority": "P1",
"frequency_minutes": 1440,
"is_free": True,
"requires_credentials": False,
},
"opengeofeed_prefix_geo": {
"id": 24,
"name": "OpenGeoFeed Prefix Geography",
"display_name": "OpenGeoFeed 前缀地理",
"module": "L3",
"priority": "P1",
"frequency_minutes": 1440,
"is_free": True,
"requires_credentials": False,
},
"nro_delegated_prefix_geo": {
"id": 25,
"name": "NRO Delegated Prefix Geography",
"display_name": "NRO 分配前缀地理",
"module": "L3",
"priority": "P1",
"frequency_minutes": 1440,
"is_free": True,
"requires_credentials": False,
},
"news_live_streams": {
"id": 26,
"name": "News Live Streams",
"display_name": "新闻直播源",
"module": "L4",
"priority": "P2",
"frequency_minutes": 720,
"is_free": True,
"requires_credentials": False,
},
"barentswatch_vessels": {
"id": 27,
"name": "BarentsWatch AIS Vessels",
"display_name": "BarentsWatch AIS 船舶",
"module": "L4",
"priority": "P1",
"frequency_minutes": 1,
"is_free": True,
"requires_credentials": True,
"credential_provider": "barentswatch",
"credential_status": "supported",
},
}

View File

@@ -0,0 +1,151 @@
"""Registry of target schemas supported by mapped custom data sources."""
from __future__ import annotations
from dataclasses import dataclass
from datetime import datetime
from typing import Any
from pydantic import BaseModel, Field, ValidationError, field_validator
class VesselAISRecord(BaseModel):
mmsi: int = Field(ge=100000000, le=999999999)
lat: float = Field(ge=-90, le=90)
lon: float = Field(ge=-180, le=180)
sog: float | None = None
cog: float | None = Field(default=None, ge=0, le=360)
heading: int | None = Field(default=None, ge=0, le=511)
name: str | None = None
vessel_type: str | int | None = None
received_at: datetime | None = None
class GeoPointRecord(BaseModel):
lat: float = Field(ge=-90, le=90)
lon: float = Field(ge=-180, le=180)
name: str | None = None
type: str | None = None
source_id: str | None = None
observed_at: datetime | None = None
metadata: dict[str, Any] = Field(default_factory=dict)
class GenericRecord(BaseModel):
data: dict[str, Any] = Field(default_factory=dict)
source_id: str | None = None
observed_at: datetime | None = None
@field_validator("data")
@classmethod
def require_payload(cls, value: dict[str, Any]) -> dict[str, Any]:
if not value:
raise ValueError("generic_records requires a non-empty data object")
return value
@dataclass(frozen=True)
class TargetField:
name: str
type: str
required: bool = False
description: str = ""
example: Any = None
def to_dict(self) -> dict[str, Any]:
return {
"name": self.name,
"type": self.type,
"required": self.required,
"description": self.description,
"example": self.example,
}
@dataclass(frozen=True)
class TargetSchema:
key: str
label: str
description: str
fields: tuple[TargetField, ...]
model: type[BaseModel]
destination: str
def to_dict(self) -> dict[str, Any]:
return {
"key": self.key,
"label": self.label,
"description": self.description,
"destination": self.destination,
"fields": [field.to_dict() for field in self.fields],
}
def validate_record(self, record: dict[str, Any]) -> tuple[dict[str, Any] | None, list[str]]:
try:
return self.model.model_validate(record).model_dump(mode="json"), []
except ValidationError as exc:
return None, [
".".join(str(part) for part in error["loc"]) + f": {error['msg']}"
for error in exc.errors()
]
TARGET_SCHEMAS: dict[str, TargetSchema] = {
"vessel_ais": TargetSchema(
key="vessel_ais",
label="船舶 AIS",
description="船只位置、航速、航向、MMSI 等 AIS 数据。",
destination="vessel_position",
model=VesselAISRecord,
fields=(
TargetField("mmsi", "integer", True, "MMSI 九位船舶标识", 257123000),
TargetField("lat", "float", True, "纬度", 59.91),
TargetField("lon", "float", True, "经度", 10.75),
TargetField("sog", "float", False, "对地航速,单位节", 12.4),
TargetField("cog", "float", False, "对地航向0-360 度", 184.5),
TargetField("heading", "integer", False, "船首向0-511", 186),
TargetField("name", "string", False, "船名", "OSLO EXPRESS"),
TargetField("vessel_type", "string", False, "船型", "cargo"),
TargetField("received_at", "datetime", False, "数据接收时间", "2026-04-28T00:00:00Z"),
),
),
"geo_points": TargetSchema(
key="geo_points",
label="通用地理点",
description="带经纬度的通用实体或事件点位。",
destination="generic_geo_points",
model=GeoPointRecord,
fields=(
TargetField("lat", "float", True, "纬度", 1.3),
TargetField("lon", "float", True, "经度", 103.8),
TargetField("name", "string", False, "点位名称", "Singapore"),
TargetField("type", "string", False, "点位类型", "datacenter"),
TargetField("source_id", "string", False, "来源侧 ID", "sg-1"),
TargetField("observed_at", "datetime", False, "观测时间", "2026-04-28T00:00:00Z"),
TargetField("metadata", "object", False, "扩展字段", {"provider": "example"}),
),
),
"generic_records": TargetSchema(
key="generic_records",
label="通用结构化记录",
description="未知结构数据沉淀,不直接进入 Earth 图层。",
destination="collected_data",
model=GenericRecord,
fields=(
TargetField("data", "object", True, "结构化记录主体", {"raw": "value"}),
TargetField("source_id", "string", False, "来源侧 ID", "record-1"),
TargetField("observed_at", "datetime", False, "观测时间", "2026-04-28T00:00:00Z"),
),
),
}
def list_target_schemas() -> list[dict[str, Any]]:
return [schema.to_dict() for schema in TARGET_SCHEMAS.values()]
def get_target_schema(key: str) -> TargetSchema:
try:
return TARGET_SCHEMAS[key]
except KeyError as exc:
raise ValueError(f"Unsupported target schema: {key}") from exc

View File

@@ -110,6 +110,8 @@ async def init_db():
import app.models.playground_session # noqa: F401
import app.models.playground_message # noqa: F401
import app.models.system_log # noqa: F401
import app.models.vessel # noqa: F401
import app.models.datasource_mapping # noqa: F401
logger.warning_event(
"Database pool settings active",
@@ -144,7 +146,12 @@ async def init_db():
text(
"""
ALTER TABLE collection_tasks
ADD COLUMN IF NOT EXISTS phase VARCHAR(30) DEFAULT 'queued'
ADD COLUMN IF NOT EXISTS phase VARCHAR(30) DEFAULT 'queued',
ADD COLUMN IF NOT EXISTS phase_progress DOUBLE PRECISION,
ADD COLUMN IF NOT EXISTS phase_message VARCHAR(255),
ADD COLUMN IF NOT EXISTS phase_current BIGINT,
ADD COLUMN IF NOT EXISTS phase_total BIGINT,
ADD COLUMN IF NOT EXISTS phase_unit VARCHAR(30)
"""
)
)

View File

@@ -12,6 +12,8 @@ from app.models.system_setting import SystemSetting
from app.models.playground_session import PlaygroundSession
from app.models.playground_message import PlaygroundMessage
from app.models.system_log import SystemLog, AuditLog
from app.models.vessel import VesselPosition, VesselStatic
from app.models.datasource_mapping import DataSourceMappingTemplate
__all__ = [
"User",
@@ -29,4 +31,7 @@ __all__ = [
"BGPObservation",
"SystemLog",
"AuditLog",
"VesselPosition",
"VesselStatic",
"DataSourceMappingTemplate",
]

View File

@@ -0,0 +1,32 @@
"""Mapping templates for user-defined data source payloads."""
from sqlalchemy import Boolean, Column, DateTime, ForeignKey, Integer, JSON, String
from sqlalchemy.sql import func
from app.db.session import Base
class DataSourceMappingTemplate(Base):
__tablename__ = "datasource_mapping_templates"
id = Column(Integer, primary_key=True, autoincrement=True)
datasource_config_id = Column(
Integer,
ForeignKey("datasource_configs.id"),
nullable=False,
index=True,
)
target_schema = Column(String(80), nullable=False, index=True)
mapping_json = Column(JSON, nullable=False, default={})
sample_payload_hash = Column(String(64), nullable=True)
validation_status = Column(String(30), nullable=False, default="draft")
version = Column(Integer, nullable=False, default=1)
is_active = Column(Boolean, nullable=False, default=False, index=True)
created_at = Column(DateTime(timezone=True), server_default=func.now())
updated_at = Column(DateTime(timezone=True), server_default=func.now(), onupdate=func.now())
def __repr__(self):
return (
f"<DataSourceMappingTemplate {self.id}: "
f"{self.datasource_config_id}/{self.target_schema}/v{self.version}>"
)

View File

@@ -1,6 +1,6 @@
"""Collection Task model"""
from sqlalchemy import Column, DateTime, Integer, String, Text, Float
from sqlalchemy import BigInteger, Column, DateTime, Integer, String, Text, Float
from sqlalchemy.sql import func
from app.db.session import Base
@@ -13,6 +13,11 @@ class CollectionTask(Base):
datasource_id = Column(Integer, nullable=False, index=True)
status = Column(String(20), nullable=False) # pending, running, success, failed, cancelled
phase = Column(String(30), default="queued")
phase_progress = Column(Float)
phase_message = Column(String(255))
phase_current = Column(BigInteger)
phase_total = Column(BigInteger)
phase_unit = Column(String(30))
started_at = Column(DateTime(timezone=True))
completed_at = Column(DateTime(timezone=True))
records_processed = Column(Integer, default=0)

View File

@@ -0,0 +1,75 @@
"""Vessel AIS models for live maritime tracking."""
from sqlalchemy import BigInteger, Column, DateTime, Float, Index, Integer, SmallInteger, String
from sqlalchemy.sql import func
from app.core.time import to_iso8601_utc
from app.db.session import Base
class VesselStatic(Base):
"""Slow-changing vessel identity and dimensions."""
__tablename__ = "vessel_static"
mmsi = Column(BigInteger, primary_key=True)
name = Column(String(128), nullable=True)
callsign = Column(String(16), nullable=True)
vessel_type = Column(SmallInteger, nullable=True, index=True)
vessel_type_name = Column(String(64), nullable=True, index=True)
flag = Column(String(4), nullable=True, index=True)
length = Column(Float, nullable=True)
width = Column(Float, nullable=True)
draught = Column(Float, nullable=True)
imo = Column(BigInteger, nullable=True)
updated_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now())
def to_dict(self) -> dict:
return {
"mmsi": self.mmsi,
"name": self.name,
"callsign": self.callsign,
"vessel_type": self.vessel_type,
"vessel_type_name": self.vessel_type_name,
"flag": self.flag,
"length": self.length,
"width": self.width,
"draught": self.draught,
"imo": self.imo,
"updated_at": to_iso8601_utc(self.updated_at),
}
class VesselPosition(Base):
"""Append-only AIS positions retained for short history windows."""
__tablename__ = "vessel_position"
id = Column(Integer, primary_key=True, autoincrement=True)
mmsi = Column(BigInteger, nullable=False, index=True)
lat = Column(Float, nullable=False)
lon = Column(Float, nullable=False)
sog = Column(Float, nullable=True)
cog = Column(Float, nullable=True)
heading = Column(SmallInteger, nullable=True)
nav_status = Column(SmallInteger, nullable=True, index=True)
received_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now(), index=True)
__table_args__ = (
Index("idx_vessel_pos_mmsi_time", "mmsi", "received_at"),
Index("idx_vessel_pos_time", "received_at"),
Index("idx_vessel_pos_lat_lon", "lat", "lon"),
)
def to_dict(self) -> dict:
return {
"id": self.id,
"mmsi": self.mmsi,
"lat": self.lat,
"lon": self.lon,
"sog": self.sog,
"cog": self.cog,
"heading": self.heading,
"nav_status": self.nav_status,
"received_at": to_iso8601_utc(self.received_at),
}

View File

@@ -3,9 +3,11 @@ from __future__ import annotations
import asyncio
import httpx
from fastapi import HTTPException, status
from fastapi import Depends, HTTPException, status
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.config import settings
from app.db.session import get_db
from app.schemas.ai import (
AIProviderStatusResponse,
SituationalAnalysisRequest,
@@ -14,11 +16,27 @@ from app.schemas.ai import (
class AIProviderClient:
def __init__(self) -> None:
self.service_url = settings.AI_PROVIDER_SERVICE_URL.rstrip("/")
self.service_token = settings.AI_PROVIDER_SERVICE_TOKEN
self.timeout = settings.AI_PROVIDER_TIMEOUT_SECONDS
self.retry_attempts = max(settings.AI_PROVIDER_RETRY_ATTEMPTS, 1)
def __init__(
self,
*,
service_url: str | None = None,
service_token: str | None = None,
timeout: int | None = None,
retry_attempts: int | None = None,
llm_config: dict | None = None,
) -> None:
self.service_url = (
service_url if service_url is not None else settings.AI_PROVIDER_SERVICE_URL
).rstrip("/")
self.service_token = (
service_token if service_token is not None else settings.AI_PROVIDER_SERVICE_TOKEN
)
self.timeout = timeout if timeout is not None else settings.AI_PROVIDER_TIMEOUT_SECONDS
self.retry_attempts = max(
retry_attempts if retry_attempts is not None else settings.AI_PROVIDER_RETRY_ATTEMPTS,
1,
)
self.llm_config = llm_config or {}
def _headers(self, request_id: str | None = None) -> dict[str, str]:
headers = {"Content-Type": "application/json"}
@@ -26,6 +44,19 @@ class AIProviderClient:
headers["X-Provider-Token"] = self.service_token
if request_id:
headers["X-Request-ID"] = request_id
llm_header_map = {
"provider": "X-AI-Provider",
"provider_api": "X-AI-Provider-API",
"base_url": "X-AI-Base-URL",
"api_key": "X-AI-API-Key",
"model": "X-AI-Model",
"max_tokens": "X-AI-Max-Tokens",
"anthropic_version": "X-AI-Anthropic-Version",
}
for key, header_name in llm_header_map.items():
value = self.llm_config.get(key)
if value not in (None, ""):
headers[header_name] = str(value)
return headers
async def get_status(self, request_id: str | None = None) -> AIProviderStatusResponse:
@@ -105,5 +136,14 @@ class AIProviderClient:
)
def get_ai_provider_client() -> AIProviderClient:
return AIProviderClient()
async def get_ai_provider_client(db: AsyncSession = Depends(get_db)) -> AIProviderClient:
from app.api.v1.settings import get_runtime_ai_provider_config
runtime_config = await get_runtime_ai_provider_config(db)
return AIProviderClient(
service_url=runtime_config["service_url"],
service_token=runtime_config["service_token"],
timeout=runtime_config["timeout_seconds"],
retry_attempts=runtime_config["retry_attempts"],
llm_config=runtime_config.get("llm_config") or {},
)

View File

@@ -0,0 +1,209 @@
"""BarentsWatch AIS credential resolution and connectivity checks."""
from __future__ import annotations
import os
import shlex
from dataclasses import dataclass
from pathlib import Path
from typing import Any
import httpx
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.data_sources import get_data_sources_config
from app.models.datasource_config import DataSourceConfig
BARENTSWATCH_LATEST_URL = "https://live.ais.barentswatch.no/v1/latest/combined"
BARENTSWATCH_TOKEN_URL = "https://id.barentswatch.no/connect/token"
BARENTSWATCH_DATASOURCE_NAME = "barentswatch_vessels"
@dataclass(frozen=True)
class BarentsWatchConfig:
endpoint: str
client_id: str
client_secret: str
credential_source: str
endpoint_source: str
def _read_zshrc_env(path: Path | None = None) -> dict[str, str]:
zshrc_path = path or Path.home() / ".zshrc"
if not zshrc_path.exists():
return {}
values: dict[str, str] = {}
for raw_line in zshrc_path.read_text(encoding="utf-8", errors="ignore").splitlines():
line = raw_line.strip()
if not line or line.startswith("#"):
continue
if line.startswith("export "):
line = line[len("export ") :].strip()
if "=" not in line:
continue
key, value = line.split("=", 1)
key = key.strip()
if not key or not key.replace("_", "").isalnum() or not key[0].isalpha():
continue
try:
parsed = shlex.split(value, comments=True, posix=True)
except ValueError:
parsed = [value.strip().strip("'\"")]
if parsed:
values[key] = parsed[0]
return values
def _first_env_value(zshrc_env: dict[str, str], *keys: str) -> tuple[str, str]:
for key in keys:
value = os.getenv(key)
if value:
return value, "environment"
for key in keys:
value = zshrc_env.get(key)
if value:
return value, "~/.zshrc"
return "", ""
async def get_barentswatch_datasource_record(db: AsyncSession) -> DataSourceConfig | None:
result = await db.execute(
select(DataSourceConfig)
.where(DataSourceConfig.name == BARENTSWATCH_DATASOURCE_NAME)
.where(DataSourceConfig.is_active.is_(True))
)
return result.scalar_one_or_none()
async def resolve_barentswatch_config(db: AsyncSession | None = None) -> BarentsWatchConfig:
record = await get_barentswatch_datasource_record(db) if db else None
auth_config = dict(record.auth_config or {}) if record else {}
config = dict(record.config or {}) if record else {}
zshrc_env = _read_zshrc_env()
env_client_id, env_source = _first_env_value(
zshrc_env,
"BARENTSWATCH_CLIENT_ID",
"BARRENTSWATCH_CLIENT_ID",
)
env_client_secret, secret_env_source = _first_env_value(
zshrc_env,
"BARENTSWATCH_CLIENT_SECRET",
"BARRENTSWATCH_CLIENT_SECRET",
)
client_id = auth_config.get("client_id") or config.get("client_id") or env_client_id
client_secret = (
auth_config.get("client_secret") or config.get("client_secret") or env_client_secret
)
credential_source = ""
if auth_config.get("client_id") or auth_config.get("client_secret"):
credential_source = "datasource_config"
elif config.get("client_id") or config.get("client_secret"):
credential_source = "datasource_runtime_config"
elif env_source or secret_env_source:
credential_source = env_source or secret_env_source
yaml_endpoint = get_data_sources_config().get_yaml_url(BARENTSWATCH_DATASOURCE_NAME)
endpoint = record.endpoint if record and record.endpoint else yaml_endpoint
return BarentsWatchConfig(
endpoint=endpoint or BARENTSWATCH_LATEST_URL,
client_id=str(client_id or ""),
client_secret=str(client_secret or ""),
credential_source=credential_source or "missing",
endpoint_source="datasource_config" if record and record.endpoint else "default",
)
async def fetch_barentswatch_access_token(
client: httpx.AsyncClient,
config: BarentsWatchConfig,
) -> str | None:
if not config.client_id or not config.client_secret:
return None
response = await client.post(
BARENTSWATCH_TOKEN_URL,
data={
"client_id": config.client_id,
"client_secret": config.client_secret,
"scope": "ais",
"grant_type": "client_credentials",
},
headers={"Content-Type": "application/x-www-form-urlencoded"},
)
response.raise_for_status()
payload = response.json()
token = payload.get("access_token")
return str(token) if token else None
async def check_barentswatch_connectivity(db: AsyncSession) -> dict[str, Any]:
config = await resolve_barentswatch_config(db)
return await check_barentswatch_config(config)
async def check_barentswatch_config(config: BarentsWatchConfig) -> dict[str, Any]:
if not config.client_id or not config.client_secret:
return {
"success": False,
"stage": "credentials",
"message": "未找到 BarentsWatch client id/client secret请先配置采集器凭证。",
"endpoint": config.endpoint,
"credential_source": config.credential_source,
"settings_tab": "collector_credentials",
}
try:
async with httpx.AsyncClient(timeout=20.0) as client:
token = await fetch_barentswatch_access_token(client, config)
if not token:
return {
"success": False,
"stage": "token",
"message": "BarentsWatch token 响应中没有 access_token请检查凭证。",
"endpoint": config.endpoint,
"credential_source": config.credential_source,
"settings_tab": "collector_credentials",
}
async with client.stream(
"GET",
config.endpoint,
headers={"Authorization": f"Bearer {token}"},
) as response:
response.raise_for_status()
return {
"success": True,
"stage": "endpoint",
"message": "BarentsWatch AIS token 和数据接口均可连通。",
"endpoint": config.endpoint,
"credential_source": config.credential_source,
"endpoint_source": config.endpoint_source,
}
except httpx.HTTPStatusError as exc:
status_code = exc.response.status_code
stage = "token" if str(exc.request.url) == BARENTSWATCH_TOKEN_URL else "endpoint"
return {
"success": False,
"stage": stage,
"message": f"BarentsWatch {stage} 请求返回 HTTP {status_code},请检查凭证或接口地址。",
"endpoint": config.endpoint,
"credential_source": config.credential_source,
"settings_tab": "collector_credentials",
}
except httpx.HTTPError as exc:
return {
"success": False,
"stage": "network",
"message": f"BarentsWatch 链路检查失败:{exc.__class__.__name__}",
"endpoint": config.endpoint,
"credential_source": config.credential_source,
"settings_tab": "collector_credentials",
}

View File

@@ -36,6 +36,7 @@ from app.services.collectors.iptoasn import IPtoASNPrefixGeoCollector
from app.services.collectors.opengeofeed import OpenGeoFeedPrefixGeoCollector
from app.services.collectors.nro_delegated import NRODelegatedPrefixGeoCollector
from app.services.collectors.news_live_streams import NewsLiveStreamsCollector
from app.services.collectors.vessel_ais import VesselAISCollector
collector_registry.register(TOP500Collector())
collector_registry.register(EpochAIGPUCollector())
@@ -63,3 +64,4 @@ collector_registry.register(IPtoASNPrefixGeoCollector())
collector_registry.register(OpenGeoFeedPrefixGeoCollector())
collector_registry.register(NRODelegatedPrefixGeoCollector())
collector_registry.register(NewsLiveStreamsCollector())
collector_registry.register(VesselAISCollector())

View File

@@ -54,6 +54,11 @@ class BaseCollector(ABC):
"task_id": self._current_task.id,
"status": self._current_task.status,
"phase": self._current_task.phase,
"phase_progress": self._current_task.phase_progress,
"phase_message": self._current_task.phase_message,
"phase_current": self._current_task.phase_current,
"phase_total": self._current_task.phase_total,
"phase_unit": self._current_task.phase_unit,
"progress": progress,
"records_processed": self._current_task.records_processed,
"total_records": self._current_task.total_records,
@@ -80,12 +85,52 @@ class BaseCollector(ABC):
await self._publish_task_update(force=force)
async def set_phase(self, phase: str):
async def set_phase(self, phase: str, *, message: str | None = None, reset_progress: bool = True):
if self._current_task and self._db_session:
self._current_task.phase = phase
self._current_task.phase_message = message
if reset_progress:
self._current_task.phase_progress = None
self._current_task.phase_current = None
self._current_task.phase_total = None
self._current_task.phase_unit = None
await self._db_session.commit()
await self._publish_task_update(force=True)
async def update_phase_progress(
self,
*,
current: int | None = None,
total: int | None = None,
unit: str | None = None,
message: str | None = None,
progress: float | None = None,
commit: bool = False,
force: bool = False,
):
"""Update progress for the current phase without changing task totals."""
if not self._current_task or not self._db_session:
return
if progress is None and current is not None and total and total > 0:
progress = (current / total) * 100
if progress is not None:
self._current_task.phase_progress = max(0.0, min(float(progress), 100.0))
if current is not None:
self._current_task.phase_current = max(0, int(current))
if total is not None:
self._current_task.phase_total = max(0, int(total))
if unit is not None:
self._current_task.phase_unit = unit
if message is not None:
self._current_task.phase_message = message
if commit:
await self._db_session.commit()
await self._publish_task_update(force=force)
@abstractmethod
async def fetch(self) -> List[Dict[str, Any]]:
"""Fetch raw data from source"""
@@ -251,7 +296,7 @@ class BaseCollector(ABC):
await self._publish_task_update(force=True)
try:
await self.set_phase("fetching")
await self.set_phase("fetching", message="正在拉取原始数据")
raw_data = await self.fetch()
task.total_records = len(raw_data)
await db.commit()
@@ -260,15 +305,20 @@ class BaseCollector(ABC):
if self.fail_on_empty and not raw_data:
raise RuntimeError(f"Collector {self.name} returned no data")
await self.set_phase("transforming")
await self.set_phase("transforming", message="正在转换采集数据")
data = self.transform(raw_data)
snapshot_id = await self._create_snapshot(db, task_id, data, start_time)
await self.set_phase("saving")
await self.set_phase("saving", message="正在保存采集数据")
records_count = await self._save_data(db, data, task_id=task_id, snapshot_id=snapshot_id)
task.status = "success"
task.phase = "completed"
task.phase_progress = 100.0
task.phase_message = "采集完成"
task.phase_current = records_count
task.phase_total = records_count
task.phase_unit = "records"
task.records_processed = records_count
task.progress = 100.0
task.completed_at = datetime.now(UTC)
@@ -285,6 +335,7 @@ class BaseCollector(ABC):
await db.rollback()
task.status = "cancelled"
task.phase = "cancelled"
task.phase_message = "采集已取消"
task.error_message = "Collection cancelled by operator and rolled back"
task.completed_at = datetime.now(UTC)
if snapshot_id is not None:
@@ -301,6 +352,7 @@ class BaseCollector(ABC):
await db.rollback()
task.status = "failed"
task.phase = "failed"
task.phase_message = str(e)
task.error_message = str(e)
task.completed_at = datetime.now(UTC)
if snapshot_id is not None:

View File

@@ -108,6 +108,11 @@ class IPtoASNPrefixGeoCollector(BaseCollector):
self._current_task.total_records = total_expected
self._current_task.records_processed = 0
self._current_task.progress = 0.0
self._current_task.phase_progress = 0.0
self._current_task.phase_message = "正在下载 IPtoASN 数据"
self._current_task.phase_current = 0
self._current_task.phase_total = total_expected
self._current_task.phase_unit = "bytes"
await self._db_session.commit()
await self._publish_task_update(force=True)
@@ -135,7 +140,14 @@ class IPtoASNPrefixGeoCollector(BaseCollector):
return
last_emit["value"] = aggregated
last_emit["t"] = now
await self.update_progress(min(aggregated, total_expected), commit=True)
current = min(aggregated, total_expected)
await self.update_phase_progress(
current=current,
total=total_expected,
unit="bytes",
message="正在下载 IPtoASN 数据",
)
await self.update_progress(current, commit=True)
batches = await asyncio.gather(
*(
@@ -148,6 +160,12 @@ class IPtoASNPrefixGeoCollector(BaseCollector):
)
)
if total_expected > 0:
await self.update_phase_progress(
current=total_expected,
total=total_expected,
unit="bytes",
message="IPtoASN 数据下载完成",
)
await self.update_progress(total_expected, commit=True, force=True)
rows: list[dict[str, Any]] = []

View File

@@ -39,12 +39,23 @@ class NRODelegatedPrefixGeoCollector(BaseCollector):
self._current_task.total_records = total_expected
self._current_task.records_processed = 0
self._current_task.progress = 0.0
self._current_task.phase_progress = 0.0
self._current_task.phase_message = "正在下载 NRO delegated 数据"
self._current_task.phase_current = 0
self._current_task.phase_total = total_expected
self._current_task.phase_unit = "bytes"
await self._db_session.commit()
await self._publish_task_update(force=True)
async def on_progress(downloaded: int, total: int | None) -> None:
if not total or total <= 0:
return
await self.update_phase_progress(
current=min(downloaded, total),
total=total,
unit="bytes",
message="正在下载 NRO delegated 数据",
)
await self.update_progress(min(downloaded, total), commit=True)
body_path = await self._downloader.download_file(

View File

@@ -40,12 +40,23 @@ class OpenGeoFeedPrefixGeoCollector(BaseCollector):
self._current_task.total_records = total_expected
self._current_task.records_processed = 0
self._current_task.progress = 0.0
self._current_task.phase_progress = 0.0
self._current_task.phase_message = "正在下载 OpenGeoFeed 数据"
self._current_task.phase_current = 0
self._current_task.phase_total = total_expected
self._current_task.phase_unit = "bytes"
await self._db_session.commit()
await self._publish_task_update(force=True)
async def on_progress(downloaded: int, total: int | None) -> None:
if not total or total <= 0:
return
await self.update_phase_progress(
current=min(downloaded, total),
total=total,
unit="bytes",
message="正在下载 OpenGeoFeed 数据",
)
await self.update_progress(min(downloaded, total), commit=True)
body_path = await self._downloader.download_file(

View File

@@ -0,0 +1,273 @@
"""BarentsWatch AIS collector for vessel tracking."""
from datetime import UTC, datetime, timedelta
from typing import Any
import httpx
from sqlalchemy import delete, select
from sqlalchemy.ext.asyncio import AsyncSession
from app.models.vessel import VesselPosition, VesselStatic
from app.services.barentswatch import (
BARENTSWATCH_LATEST_URL,
fetch_barentswatch_access_token,
resolve_barentswatch_config,
)
from app.services.collectors.base import BaseCollector
VESSEL_TYPE_NAMES = {
30: "Fishing",
35: "Military",
60: "Passenger",
70: "Cargo",
80: "Tanker",
}
class VesselAISCollector(BaseCollector):
"""Collect latest AIS positions and append them to vessel tables."""
name = "barentswatch_vessels"
priority = "P1"
module = "L4"
frequency_hours = 1
data_type = "vessel_ais"
@property
def base_url(self) -> str:
return self._resolved_url or BARENTSWATCH_LATEST_URL
async def _get_access_token(self, client: httpx.AsyncClient) -> str | None:
config = await resolve_barentswatch_config(self._db_session)
return await fetch_barentswatch_access_token(client, config)
async def fetch(self) -> list[dict[str, Any]]:
async with httpx.AsyncClient(timeout=60.0) as client:
headers: dict[str, str] = {}
token = await self._get_access_token(client)
if token:
headers["Authorization"] = f"Bearer {token}"
response = await client.get(self.base_url, headers=headers)
if response.status_code == 401 and not token:
return self._get_sample_data()
response.raise_for_status()
payload = response.json()
if isinstance(payload, list):
return [item for item in payload if isinstance(item, dict)]
if isinstance(payload, dict):
for key in ("features", "data", "items", "vessels"):
value = payload.get(key)
if isinstance(value, list):
if key == "features":
return [
{
**(item.get("properties") or {}),
"geometry": item.get("geometry"),
}
for item in value
if isinstance(item, dict)
]
return [item for item in value if isinstance(item, dict)]
return self._get_sample_data()
def transform(self, raw_data: list[dict[str, Any]]) -> list[dict[str, Any]]:
transformed = []
for item in raw_data:
record = self._normalize_record(item)
if record:
transformed.append(record)
return transformed
async def _save_data(
self,
db: AsyncSession,
data: list[dict[str, Any]],
task_id: int | None = None,
snapshot_id: int | None = None,
) -> int:
now = datetime.now(UTC)
records_added = 0
for index, item in enumerate(data):
static = await db.get(VesselStatic, item["mmsi"])
if static is None:
static = VesselStatic(mmsi=item["mmsi"])
db.add(static)
for field in (
"name",
"callsign",
"vessel_type",
"vessel_type_name",
"flag",
"length",
"width",
"draught",
"imo",
):
value = item.get(field)
if value not in (None, ""):
setattr(static, field, value)
static.updated_at = now
db.add(
VesselPosition(
mmsi=item["mmsi"],
lat=item["lat"],
lon=item["lon"],
sog=item.get("sog"),
cog=item.get("cog"),
heading=item.get("heading"),
nav_status=item.get("nav_status"),
received_at=item.get("received_at") or now,
)
)
records_added += 1
if (index + 1) % 1000 == 0:
await self.update_progress(index + 1, commit=True)
await db.execute(
delete(VesselPosition).where(VesselPosition.received_at < now - timedelta(hours=24))
)
await db.commit()
await self.update_progress(records_added, force=True)
return records_added
def _normalize_record(self, item: dict[str, Any]) -> dict[str, Any] | None:
mmsi = _as_int(_pick(item, "mmsi", "MMSI", "Mmsi"))
lat = _as_float(_pick(item, "lat", "latitude", "Latitude"))
lon = _as_float(_pick(item, "lon", "lng", "longitude", "Longitude"))
geometry = item.get("geometry")
coordinates = geometry.get("coordinates") if isinstance(geometry, dict) else None
if (lat is None or lon is None) and isinstance(coordinates, list) and len(coordinates) >= 2:
lon = _as_float(coordinates[0])
lat = _as_float(coordinates[1])
if mmsi is None or lat is None or lon is None:
return None
if not (-90 <= lat <= 90 and -180 <= lon <= 180):
return None
vessel_type = _as_int(_pick(item, "vessel_type", "shipType", "ship_type", "ShipType"))
vessel_type_name = (
_pick(item, "vessel_type_name", "shipTypeName", "ship_type_name", "VesselTypeName")
or _vessel_type_name(vessel_type)
)
received_at = _parse_datetime(_pick(item, "received_at", "timestamp", "time", "msgtime"))
return {
"mmsi": mmsi,
"name": _pick(item, "name", "shipName", "ship_name", "Name"),
"callsign": _pick(item, "callsign", "callSign", "CallSign"),
"vessel_type": vessel_type,
"vessel_type_name": vessel_type_name,
"flag": _pick(item, "flag", "country", "Flag"),
"length": _as_float(_pick(item, "length", "shipLength", "Length")),
"width": _as_float(_pick(item, "width", "shipWidth", "Width")),
"draught": _as_float(_pick(item, "draught", "draft", "Draught")),
"imo": _as_int(_pick(item, "imo", "IMO", "imoNumber")),
"lat": lat,
"lon": lon,
"sog": _as_float(_pick(item, "sog", "speedOverGround", "SOG")),
"cog": _as_float(_pick(item, "cog", "courseOverGround", "COG")),
"heading": _as_int(_pick(item, "heading", "trueHeading", "Heading")),
"nav_status": _as_int(_pick(item, "nav_status", "navStatus", "NavigationalStatus")),
"received_at": received_at,
}
def _get_sample_data(self) -> list[dict[str, Any]]:
return [
{
"mmsi": 257123000,
"name": "OSLO TRADER",
"lat": 59.91,
"lon": 10.73,
"sog": 12.4,
"cog": 214,
"heading": 215,
"nav_status": 0,
"vessel_type": 70,
"vessel_type_name": "Cargo",
"flag": "NO",
"length": 185,
},
{
"mmsi": 257456000,
"name": "NORDIC FJORD",
"lat": 60.39,
"lon": 5.32,
"sog": 0.2,
"cog": 82,
"heading": 80,
"nav_status": 1,
"vessel_type": 60,
"vessel_type_name": "Passenger",
"flag": "NO",
"length": 126,
},
]
def _pick(item: dict[str, Any], *keys: str) -> Any:
for key in keys:
if key in item and item[key] not in (None, ""):
return item[key]
return None
def _as_float(value: Any) -> float | None:
try:
if value in (None, ""):
return None
return float(value)
except (TypeError, ValueError):
return None
def _as_int(value: Any) -> int | None:
try:
if value in (None, ""):
return None
return int(float(value))
except (TypeError, ValueError):
return None
def _parse_datetime(value: Any) -> datetime | None:
if isinstance(value, datetime):
return value if value.tzinfo else value.replace(tzinfo=UTC)
if not value:
return None
if isinstance(value, (int, float)):
timestamp = float(value)
if timestamp > 10_000_000_000:
timestamp /= 1000
return datetime.fromtimestamp(timestamp, UTC)
if isinstance(value, str):
try:
parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
return parsed if parsed.tzinfo else parsed.replace(tzinfo=UTC)
except ValueError:
return None
return None
def _vessel_type_name(vessel_type: int | None) -> str:
if vessel_type is None:
return "Other"
if 70 <= vessel_type <= 79:
return "Cargo"
if 80 <= vessel_type <= 89:
return "Tanker"
if 60 <= vessel_type <= 69:
return "Passenger"
if vessel_type == 30:
return "Fishing"
if vessel_type == 35:
return "Military"
return VESSEL_TYPE_NAMES.get(vessel_type, "Other")

View File

@@ -0,0 +1,165 @@
"""Credential setup guides for collector integrations."""
from __future__ import annotations
from dataclasses import dataclass
from typing import Any
from sqlalchemy import select
from app.models.system_setting import SystemSetting
from app.schemas.ai import SituationalAnalysisRequest
from app.services.ai_client import AIProviderClient
CREDENTIAL_GUIDES_CATEGORY = "collector_credential_guides"
@dataclass(frozen=True)
class CredentialGuideDefault:
provider: str
title: str
prompt: str
markdown: str
BARENTSWATCH_DEFAULT_GUIDE = CredentialGuideDefault(
provider="barentswatch",
title="BarentsWatch AIS 凭证获取教程",
prompt=(
"请生成一份中文教程,指导开发者获取 BarentsWatch Live AIS API 的 "
"OAuth client credentials。教程要面向已经有本地开发环境的人包含注册/登录、"
"创建 client、申请或确认 ais scope、复制 client id 和 client secret、"
"在系统设置中填写并验证连接、常见失败排查。不要编造具体页面按钮文案,"
"必须参考官方 tutorialhttps://developer.barentswatch.no/docs/tutorial 。"
"必须强调 Live AIS 要选择 AIS-client / AIS - API而不是普通 API-client。"
"如果步骤可能变化,要提醒以 BarentsWatch developer portal 当前页面为准。"
),
markdown="""## BarentsWatch AIS 凭证获取
官方教程https://developer.barentswatch.no/docs/tutorial
1. 先打开上面的 BarentsWatch 官方 tutorial按官方流程登录或注册开发者账号。
2. 在 Developer access 页面选择 `AIS - API`,不要选择普通的 `BarentsWatch - API`。
3. 在 `AIS - API` 下创建用于 Planet 的 AIS client。
4. 创建时记下你设置的 password / client secret。
5. 回到 My Page 复制完整 `Client ID`。它通常长得像 `your.email@example.com:client-name`。
6. 回到 Planet 的 `设置 -> 采集器设置 -> BarentsWatch AIS`,填入 `Client ID` 和 `Client Secret`。
7. 点击 `连接` 验证 token 和 AIS endpoint 是否可访问。
8. 连接成功后保存凭证。
### 请求规则
- Token 地址:`https://id.barentswatch.no/connect/token`
- 请求方式:`POST`
- Content-Type`application/x-www-form-urlencoded`
- Body 必须包含:`grant_type=client_credentials`、`client_id`、`client_secret`、`scope=ais`
- `client_id`、`client_secret`、`scope`、`grant_type` 都要放在 body不要放在 header。
- AIS 数据请求使用 header`Authorization: Bearer <access_token>`
### 常见排查
- `未找到凭证`:确认 `Client ID` 和 `Client Secret` 已填写,或已经写入 `~/.zshrc`。
- `HTTP 401/403`:通常是选成了普通 `BarentsWatch - API` client、client secret 错误,或 token 请求没有使用 `scope=ais`。
- `network` 错误:检查本机是否能访问 `id.barentswatch.no` 和 `live.ais.barentswatch.no`。
- Endpoint 建议保持默认:`https://live.ais.barentswatch.no/v1/latest/combined`。
""",
)
DEFAULT_CREDENTIAL_GUIDES = {
BARENTSWATCH_DEFAULT_GUIDE.provider: BARENTSWATCH_DEFAULT_GUIDE,
}
async def _get_guide_store(db) -> tuple[SystemSetting | None, dict[str, Any]]:
result = await db.execute(
select(SystemSetting).where(SystemSetting.category == CREDENTIAL_GUIDES_CATEGORY)
)
record = result.scalar_one_or_none()
payload = dict(record.payload or {}) if record and isinstance(record.payload, dict) else {}
return record, payload
async def get_credential_guide(db, provider: str) -> dict[str, Any]:
default = DEFAULT_CREDENTIAL_GUIDES.get(provider)
if default is None:
raise ValueError(f"Unsupported credential guide provider: {provider}")
_record, store = await _get_guide_store(db)
custom = store.get(provider) if isinstance(store.get(provider), dict) else None
return {
"provider": provider,
"title": custom.get("title") if custom else default.title,
"markdown": custom.get("markdown") if custom else default.markdown,
"prompt": default.prompt,
"source": "ai" if custom else "default",
}
async def save_credential_guide(db, provider: str, title: str, markdown: str) -> dict[str, Any]:
default = DEFAULT_CREDENTIAL_GUIDES.get(provider)
if default is None:
raise ValueError(f"Unsupported credential guide provider: {provider}")
record, store = await _get_guide_store(db)
store[provider] = {
"title": title or default.title,
"markdown": markdown,
}
if record is None:
db.add(SystemSetting(category=CREDENTIAL_GUIDES_CATEGORY, payload=store))
else:
record.payload = store
await db.commit()
return await get_credential_guide(db, provider)
async def reset_credential_guide(db, provider: str) -> dict[str, Any]:
default = DEFAULT_CREDENTIAL_GUIDES.get(provider)
if default is None:
raise ValueError(f"Unsupported credential guide provider: {provider}")
record, store = await _get_guide_store(db)
if provider in store:
store.pop(provider, None)
if record is not None:
record.payload = store
await db.commit()
return await get_credential_guide(db, provider)
async def generate_credential_guide(
db,
provider: str,
ai_client: AIProviderClient,
) -> dict[str, Any]:
default = DEFAULT_CREDENTIAL_GUIDES.get(provider)
if default is None:
raise ValueError(f"Unsupported credential guide provider: {provider}")
response = await ai_client.analyze(
SituationalAnalysisRequest(
title=f"Generate credential guide for {provider}",
objective=default.prompt,
context={
"provider": provider,
"current_default_guide": default.markdown,
"product_context": "Planet collector credential settings",
},
observations=[
"Use concise Chinese markdown.",
"Prefer stable concepts over brittle UI labels.",
"Include verification and troubleshooting steps.",
],
constraints=[
"Do not ask the user for secrets.",
"Do not include fabricated screenshots.",
"Return markdown only.",
],
)
)
markdown = response.content.strip()
if not markdown:
markdown = default.markdown
return await save_credential_guide(db, provider, default.title, markdown)

View File

@@ -0,0 +1,384 @@
"""Connectivity validation helpers for built-in datasource overrides."""
from __future__ import annotations
from datetime import UTC, datetime
import hashlib
import json
import os
from typing import Any
import httpx
from sqlalchemy import func, select
from app.core.data_sources import get_data_sources_config
from app.core.datasource_defaults import DEFAULT_DATASOURCES
from app.models.collected_data import CollectedData
from app.models.datasource import DataSource
from app.models.datasource_config import DataSourceConfig
from app.models.system_setting import SystemSetting
from app.services.barentswatch import (
_read_zshrc_env,
fetch_barentswatch_access_token,
resolve_barentswatch_config,
)
CONNECTIVITY_VALIDATION_KEY = "connectivity_validation"
CONNECTIVITY_STORE_CATEGORY = "datasource_connectivity_validations"
def _sha256_json(payload: Any) -> str:
encoded = json.dumps(payload, ensure_ascii=False, sort_keys=True, default=str).encode("utf-8")
return hashlib.sha256(encoded).hexdigest()
def _resolve_spacetrack_credentials() -> tuple[str, str, str]:
zshrc_env = _read_zshrc_env()
username = os.getenv("SPACETRACK_USERNAME") or zshrc_env.get("SPACETRACK_USERNAME") or ""
password = os.getenv("SPACETRACK_PASSWORD") or zshrc_env.get("SPACETRACK_PASSWORD") or ""
source = "environment" if os.getenv("SPACETRACK_USERNAME") or os.getenv("SPACETRACK_PASSWORD") else ""
if not source and (username or password):
source = "~/.zshrc"
return username, password, source or "missing"
def strip_connectivity_validation(config: dict | None) -> dict:
cleaned = dict(config or {})
cleaned.pop(CONNECTIVITY_VALIDATION_KEY, None)
return cleaned
def merge_connectivity_validation(existing_config: dict | None, next_config: dict | None) -> dict:
merged = strip_connectivity_validation(next_config)
validation = (existing_config or {}).get(CONNECTIVITY_VALIDATION_KEY)
if validation:
merged[CONNECTIVITY_VALIDATION_KEY] = validation
return merged
def get_connectivity_validation(config: DataSourceConfig | None) -> dict | None:
validation = (config.config or {}).get(CONNECTIVITY_VALIDATION_KEY) if config else None
return validation if isinstance(validation, dict) else None
async def build_builtin_connectivity_checksum(
source: str,
endpoint: str,
auth_type: str,
headers: dict | None,
config: dict | None,
db=None,
credential_override: dict[str, str] | None = None,
) -> tuple[str, dict[str, Any]]:
defaults = DEFAULT_DATASOURCES.get(source, {})
credential_provider = defaults.get("credential_provider")
credential_fingerprint = ""
credential_source = "none"
has_credentials = not defaults.get("requires_credentials", False)
if credential_provider == "barentswatch":
if credential_override:
client_id = credential_override.get("client_id", "")
client_secret = credential_override.get("client_secret", "")
credential_source = "draft"
else:
barentswatch_config = await resolve_barentswatch_config(db)
client_id = barentswatch_config.client_id
client_secret = barentswatch_config.client_secret
credential_source = barentswatch_config.credential_source
has_credentials = bool(client_id and client_secret)
credential_fingerprint = _sha256_json(
{
"client_id": client_id,
"client_secret": client_secret,
}
)
elif credential_provider == "spacetrack":
username, password, credential_source = _resolve_spacetrack_credentials()
has_credentials = bool(username and password)
credential_fingerprint = _sha256_json(
{
"username": username,
"password": password,
}
)
elif defaults.get("requires_credentials"):
credential_source = str(credential_provider or "unsupported")
checksum_payload = {
"source": source,
"endpoint": endpoint,
"auth_type": "none",
"headers": headers or {},
"config": strip_connectivity_validation(config),
"credential_provider": credential_provider or "none",
"credential_fingerprint": credential_fingerprint,
}
return _sha256_json(checksum_payload), {
"requires_credentials": bool(defaults.get("requires_credentials", False)),
"credential_provider": credential_provider,
"credential_source": credential_source,
"has_credentials": has_credentials,
}
async def test_builtin_connectivity(
source: str,
endpoint: str,
auth_type: str,
headers: dict | None,
config: dict | None,
db=None,
) -> dict[str, Any]:
defaults = DEFAULT_DATASOURCES.get(source)
if not defaults:
return {
"success": False,
"message": "未知内置采集器,无法执行连接校验。",
}
checksum, credential_context = await build_builtin_connectivity_checksum(
source,
endpoint,
auth_type,
headers,
config,
db,
)
if credential_context["requires_credentials"] and not credential_context["has_credentials"]:
return {
"success": False,
"checksum": checksum,
"stage": "credentials",
"message": "该采集器需要凭证,请先到采集器凭证设置中配置。",
"settings_tab": "collector_credentials",
**credential_context,
}
supported_credential_providers = {"barentswatch", "spacetrack"}
if (
credential_context["requires_credentials"]
and credential_context["credential_provider"] not in supported_credential_providers
):
return {
"success": False,
"checksum": checksum,
"stage": "credentials",
"message": "该采集器的凭证链路尚未接入,暂时无法完成连接校验。",
"settings_tab": "collector_credentials",
**credential_context,
}
request_headers = {str(key): str(value) for key, value in (headers or {}).items()}
request_config = strip_connectivity_validation(config)
timeout = float(request_config.get("timeout") or 30)
request_endpoint = endpoint
try:
async with httpx.AsyncClient(timeout=timeout, follow_redirects=True) as client:
if credential_context["credential_provider"] == "barentswatch":
barentswatch_config = await resolve_barentswatch_config(db)
token = await fetch_barentswatch_access_token(client, barentswatch_config)
if not token:
return {
"success": False,
"checksum": checksum,
"stage": "token",
"message": "凭证可读取,但 token 响应中没有 access_token。",
"settings_tab": "collector_credentials",
**credential_context,
}
request_headers["Authorization"] = f"Bearer {token}"
elif credential_context["credential_provider"] == "spacetrack":
username, password, _source = _resolve_spacetrack_credentials()
login_url = "https://www.space-track.org/ajaxauth/login"
login_response = await client.post(
login_url,
data={
"identity": username,
"password": password,
},
)
login_response.raise_for_status()
started = datetime.now(UTC)
async with client.stream("GET", request_endpoint, headers=request_headers) as response:
response.raise_for_status()
status_code = response.status_code
elapsed_ms = (datetime.now(UTC) - started).total_seconds() * 1000
return {
"success": True,
"checksum": checksum,
"stage": "endpoint",
"message": "连接验证成功。",
"status_code": status_code,
"response_time_ms": elapsed_ms,
**credential_context,
}
except httpx.HTTPStatusError as exc:
return {
"success": False,
"checksum": checksum,
"stage": "endpoint",
"message": f"连接验证失败HTTP {exc.response.status_code}",
"error": f"HTTP Error: {exc.response.status_code}",
**credential_context,
}
except httpx.HTTPError as exc:
return {
"success": False,
"checksum": checksum,
"stage": "network",
"message": f"连接验证失败:{exc.__class__.__name__}",
"error": str(exc),
**credential_context,
}
def make_success_validation(checksum: str, result: dict[str, Any]) -> dict[str, Any]:
return {
"checksum": checksum,
"status": "success",
"validated_at": datetime.now(UTC).isoformat(),
"status_code": result.get("status_code"),
"credential_source": result.get("credential_source"),
}
def is_builtin_validation_current(config: DataSourceConfig | None, checksum: str) -> bool:
validation = get_connectivity_validation(config)
return bool(
validation
and validation.get("status") == "success"
and validation.get("checksum") == checksum
)
async def get_connectivity_store(db) -> dict[str, Any]:
result = await db.execute(
select(SystemSetting).where(SystemSetting.category == CONNECTIVITY_STORE_CATEGORY)
)
record = result.scalar_one_or_none()
return dict(record.payload or {}) if record and isinstance(record.payload, dict) else {}
async def save_connectivity_success(
db,
source: str,
checksum: str,
result: dict[str, Any],
*,
connected_by: str,
) -> dict[str, Any]:
store = await get_connectivity_store(db)
validation = {
**make_success_validation(checksum, result),
"connected_by": connected_by,
}
store[source] = validation
existing = await db.execute(
select(SystemSetting).where(SystemSetting.category == CONNECTIVITY_STORE_CATEGORY)
)
record = existing.scalar_one_or_none()
if record is None:
db.add(SystemSetting(category=CONNECTIVITY_STORE_CATEGORY, payload=store))
else:
record.payload = store
return validation
async def load_builtin_override_config(db, source: str) -> DataSourceConfig | None:
result = await db.execute(
select(DataSourceConfig)
.where(DataSourceConfig.name == source)
.where(DataSourceConfig.is_active.is_(True))
)
return result.scalar_one_or_none()
async def get_builtin_effective_candidate(db, source: str) -> dict[str, Any]:
override = await load_builtin_override_config(db, source)
default_endpoint = get_data_sources_config().get_yaml_url(source)
return {
"name": source,
"endpoint": (override.endpoint if override and override.endpoint else default_endpoint) or "",
"auth_type": override.auth_type if override else "none",
"headers": override.headers if override else {},
"config": strip_connectivity_validation(override.config if override else {}),
}
async def has_collected_data(db, source: str) -> bool:
result = await db.execute(select(func.count(CollectedData.id)).where(CollectedData.source == source))
if (result.scalar() or 0) > 0:
return True
datasource_result = await db.execute(select(DataSource).where(DataSource.source == source))
datasource = datasource_result.scalar_one_or_none()
return bool(datasource and datasource.last_status == "success")
async def get_builtin_connection_status(
db,
source: str,
endpoint: str,
auth_type: str,
headers: dict | None,
config: dict | None,
) -> dict[str, Any]:
checksum, credential_context = await build_builtin_connectivity_checksum(
source,
endpoint,
auth_type,
headers,
config,
db,
)
store = await get_connectivity_store(db)
validation = store.get(source)
if isinstance(validation, dict) and validation.get("status") == "success":
if validation.get("checksum") == checksum:
return {
"connected": True,
"checksum": checksum,
"connected_by": validation.get("connected_by") or "connection_button",
"message": "当前配置已完成连接验证。",
**credential_context,
}
effective = await get_builtin_effective_candidate(db, source)
effective_checksum, _ = await build_builtin_connectivity_checksum(
source,
effective["endpoint"],
effective["auth_type"],
effective["headers"],
effective["config"],
db,
)
if checksum == effective_checksum and await has_collected_data(db, source):
return {
"connected": True,
"checksum": checksum,
"connected_by": "collection",
"message": "当前配置已有成功采集数据,视为已连接。",
**credential_context,
}
if isinstance(validation, dict) and validation.get("status") == "success":
return {
"connected": False,
"checksum": checksum,
"connected_by": None,
"message": "接口地址或凭证指纹已变化,请重新点击连接验证。",
**credential_context,
}
return {
"connected": False,
"checksum": checksum,
"connected_by": None,
"message": "当前配置尚未连接,请点击连接验证。",
**credential_context,
}

View File

@@ -0,0 +1,358 @@
"""Deterministic mapping support for custom data sources."""
from __future__ import annotations
import hashlib
import json
import re
from datetime import UTC, datetime
from typing import Any
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.target_schema_registry import TargetSchema, get_target_schema
SECRET_KEY_PATTERN = re.compile(
r"(token|secret|password|passwd|authorization|api[_-]?key|client[_-]?secret)",
re.IGNORECASE,
)
class MappingError(ValueError):
"""Raised when a mapping definition cannot be executed."""
def stable_payload_hash(payload: Any) -> str:
encoded = json.dumps(payload, ensure_ascii=False, sort_keys=True, default=str).encode()
return hashlib.sha256(encoded).hexdigest()
def redact_for_llm(value: Any) -> Any:
if isinstance(value, dict):
redacted = {}
for key, item in value.items():
if SECRET_KEY_PATTERN.search(str(key)):
redacted[key] = "[REDACTED]"
else:
redacted[key] = redact_for_llm(item)
return redacted
if isinstance(value, list):
return [redact_for_llm(item) for item in value[:20]]
return value
def extract_path(payload: Any, path: str | None) -> Any:
if not path or path == "$":
return payload
normalized = path.strip()
if normalized.startswith("$."):
normalized = normalized[2:]
elif normalized.startswith("$"):
normalized = normalized[1:]
normalized = normalized.strip(".")
if not normalized:
return payload
current = payload
for raw_segment in normalized.split("."):
segment = raw_segment.strip()
if not segment:
continue
list_all = segment.endswith("[*]")
if list_all:
segment = segment[:-3]
index = None
match = re.fullmatch(r"(.+)\[(\d+)\]", segment)
if match:
segment = match.group(1)
index = int(match.group(2))
if segment:
if isinstance(current, dict):
current = current.get(segment)
else:
return None
if list_all:
return current if isinstance(current, list) else []
if index is not None:
if not isinstance(current, list) or index >= len(current):
return None
current = current[index]
return current
def _convert_value(value: Any, target_type: str | None) -> Any:
if value is None or target_type in (None, "", "any"):
return value
if target_type == "string":
return str(value)
if target_type == "integer":
return int(value)
if target_type == "float":
return float(value)
if target_type == "boolean":
if isinstance(value, bool):
return value
if isinstance(value, str):
return value.strip().lower() in {"1", "true", "yes", "y", "on"}
return bool(value)
if target_type == "datetime":
if isinstance(value, datetime):
return value
if isinstance(value, (int, float)):
return datetime.fromtimestamp(value)
if isinstance(value, str):
return datetime.fromisoformat(value.replace("Z", "+00:00"))
return value
if target_type == "object":
if isinstance(value, dict):
return value
raise ValueError("expected object")
if target_type == "array":
if isinstance(value, list):
return value
raise ValueError("expected array")
return value
def _apply_enum(value: Any, enum_map: Any) -> Any:
if not isinstance(enum_map, dict):
return value
key = str(value)
return enum_map.get(key, enum_map.get(value, value))
def _map_one(item: Any, field_mapping: dict[str, Any]) -> tuple[dict[str, Any], list[str]]:
output: dict[str, Any] = {}
errors: list[str] = []
for field_name, rule in field_mapping.items():
if isinstance(rule, str):
rule = {"path": rule}
if not isinstance(rule, dict):
errors.append(f"{field_name}: mapping rule must be an object or path string")
continue
value = extract_path(item, rule.get("path"))
if value is None and "default" in rule:
value = rule.get("default")
value = _apply_enum(value, rule.get("enum"))
try:
value = _convert_value(value, rule.get("type"))
except (TypeError, ValueError) as exc:
errors.append(f"{field_name}: failed to convert value {value!r}: {exc}")
continue
if value is not None or rule.get("include_null", False):
output[field_name] = value
return output, errors
def execute_mapping(
payload: Any,
mapping_json: dict[str, Any],
target_schema: str | TargetSchema,
*,
limit: int | None = None,
) -> dict[str, Any]:
schema = get_target_schema(target_schema) if isinstance(target_schema, str) else target_schema
source = mapping_json.get("source") or {}
fields = mapping_json.get("fields")
if not isinstance(fields, dict) or not fields:
raise MappingError("mapping_json.fields must be a non-empty object")
items_path = source.get("items_path") or mapping_json.get("items_path") or "$"
items = extract_path(payload, items_path)
if isinstance(items, dict):
items = [items]
elif not isinstance(items, list):
items = []
if limit is not None:
items = items[:limit]
mapped_records: list[dict[str, Any]] = []
errors: list[dict[str, Any]] = []
for index, item in enumerate(items):
mapped, mapping_errors = _map_one(item, fields)
validated, validation_errors = schema.validate_record(mapped)
all_errors = mapping_errors + validation_errors
if all_errors:
errors.append({"index": index, "errors": all_errors, "record": mapped})
continue
if validated is not None:
mapped_records.append(validated)
return {
"target_schema": schema.key,
"total_items": len(items),
"mapped_count": len(mapped_records),
"failed_count": len(errors),
"records": mapped_records,
"errors": errors,
}
def build_heuristic_mapping(sample_payload: Any, target_schema_key: str) -> dict[str, Any]:
schema = get_target_schema(target_schema_key)
items_path = "$"
sample_item = sample_payload
if isinstance(sample_payload, dict):
for key in ("data", "items", "results", "features", "vessels"):
candidate = sample_payload.get(key)
if isinstance(candidate, list) and candidate:
items_path = f"$.{key}[*]"
sample_item = candidate[0]
break
elif isinstance(sample_payload, list) and sample_payload:
items_path = "$"
sample_item = sample_payload[0]
available = _flatten_keys(sample_item if isinstance(sample_item, dict) else {})
fields: dict[str, Any] = {}
for field in schema.fields:
candidate = _best_field_match(field.name, available)
if candidate:
fields[field.name] = {"path": f"$.{candidate}", "type": field.type}
elif field.name == "data" and target_schema_key == "generic_records":
fields[field.name] = {"path": "$", "type": "object"}
elif not field.required:
fields[field.name] = {"path": f"$.{field.name}", "type": field.type, "default": None}
return {
"source": {"items_path": items_path},
"fields": fields,
"meta": {
"generated_by": "heuristic",
"requires_review": True,
},
}
def _flatten_keys(payload: dict[str, Any], prefix: str = "") -> list[str]:
keys: list[str] = []
for key, value in payload.items():
dotted = f"{prefix}.{key}" if prefix else str(key)
keys.append(dotted)
if isinstance(value, dict):
keys.extend(_flatten_keys(value, dotted))
return keys
def _best_field_match(field_name: str, candidates: list[str]) -> str | None:
aliases = {
"lat": ("lat", "latitude", "y"),
"lon": ("lon", "lng", "longitude", "x"),
"mmsi": ("mmsi",),
"sog": ("sog", "speed", "speedOverGround"),
"cog": ("cog", "course", "courseOverGround"),
"received_at": ("received_at", "timestamp", "time", "updated_at"),
"observed_at": ("observed_at", "timestamp", "time", "updated_at"),
"source_id": ("id", "source_id", "uuid"),
}.get(field_name, (field_name,))
lowered = {candidate.lower(): candidate for candidate in candidates}
for alias in aliases:
if alias.lower() in lowered:
return lowered[alias.lower()]
for candidate in candidates:
tail = candidate.split(".")[-1].lower()
if tail in {alias.lower() for alias in aliases}:
return candidate
return None
def _parse_datetime(value: Any) -> datetime | None:
if value is None:
return None
if isinstance(value, datetime):
return value
if isinstance(value, str):
return datetime.fromisoformat(value.replace("Z", "+00:00"))
return None
async def persist_mapped_records(
db: AsyncSession,
*,
datasource_name: str,
datasource_config_id: int,
target_schema: str,
records: list[dict[str, Any]],
mapping_version: int,
) -> int:
"""Persist validated mapped records to the destination for a target schema."""
if target_schema == "vessel_ais":
from app.models.vessel import VesselPosition
for record in records:
db.add(
VesselPosition(
mmsi=record["mmsi"],
lat=record["lat"],
lon=record["lon"],
sog=record.get("sog"),
cog=record.get("cog"),
heading=record.get("heading"),
received_at=_parse_datetime(record.get("received_at")) or datetime.now(UTC),
)
)
await db.commit()
return len(records)
from app.models.collected_data import CollectedData
collected_at = datetime.now(UTC)
for index, record in enumerate(records):
if target_schema == "geo_points":
source_id = record.get("source_id") or f"{datasource_config_id}:{index}"
name = record.get("name")
metadata = {
"latitude": record.get("lat"),
"longitude": record.get("lon"),
"type": record.get("type"),
"mapping_version": mapping_version,
"target_schema": target_schema,
**(record.get("metadata") or {}),
}
reference_date = _parse_datetime(record.get("observed_at"))
else:
source_id = record.get("source_id") or f"{datasource_config_id}:{index}"
name = None
metadata = {
"data": record.get("data") or {},
"mapping_version": mapping_version,
"target_schema": target_schema,
}
reference_date = _parse_datetime(record.get("observed_at"))
db.add(
CollectedData(
source=datasource_name,
source_id=str(source_id),
entity_key=f"{datasource_name}:{source_id}",
data_type=target_schema,
name=name,
title=name,
extra_data=metadata,
collected_at=collected_at,
reference_date=reference_date,
is_valid=1,
is_current=True,
change_type="created",
change_summary={},
)
)
await db.commit()
return len(records)

View File

@@ -0,0 +1,150 @@
"""LLM provider presets used by Settings and the runtime AI provider bridge."""
from __future__ import annotations
from typing import Any
import httpx
MODELS_DEV_URL = "https://models.dev/api.json"
FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
"minimax": {
"provider": "minimax",
"label": "MiniMax",
"provider_api": "anthropic-messages",
"base_url": "https://api.minimaxi.com/anthropic",
"model": "MiniMax-M2.7",
"models": ["MiniMax-M2.7", "MiniMax-M2.7-highspeed", "MiniMax-M2.5", "MiniMax-M2"],
"api_key_env": "MINIMAX_API_KEY",
"source": "fallback",
},
"openai": {
"provider": "openai",
"label": "OpenAI",
"provider_api": "openai-completions",
"base_url": "https://api.openai.com/v1",
"model": "gpt-5.1",
"models": ["gpt-5.1", "gpt-5.1-codex", "gpt-4.1", "gpt-4o"],
"api_key_env": "OPENAI_API_KEY",
"source": "fallback",
},
"anthropic": {
"provider": "anthropic",
"label": "Anthropic",
"provider_api": "anthropic-messages",
"base_url": "https://api.anthropic.com/v1",
"model": "claude-sonnet-4-6",
"models": ["claude-sonnet-4-6", "claude-opus-4-5", "claude-3-5-haiku-20241022"],
"api_key_env": "ANTHROPIC_API_KEY",
"source": "fallback",
},
"deepseek": {
"provider": "deepseek",
"label": "DeepSeek",
"provider_api": "openai-completions",
"base_url": "https://api.deepseek.com/v1",
"model": "deepseek-chat",
"models": ["deepseek-chat", "deepseek-reasoner"],
"api_key_env": "DEEPSEEK_API_KEY",
"source": "fallback",
},
"alibaba": {
"provider": "alibaba",
"label": "Alibaba Qwen / DashScope",
"provider_api": "openai-completions",
"base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1",
"model": "qwen3-max",
"models": ["qwen3-max", "qwen3.5-plus", "qwen-max", "qwen-plus"],
"api_key_env": "DASHSCOPE_API_KEY",
"source": "fallback",
},
"moonshotai": {
"provider": "moonshotai",
"label": "Moonshot AI / Kimi",
"provider_api": "openai-completions",
"base_url": "https://api.moonshot.ai/v1",
"model": "kimi-k2.5",
"models": ["kimi-k2.5", "kimi-k2-thinking", "kimi-k2-turbo-preview"],
"api_key_env": "MOONSHOT_API_KEY",
"source": "fallback",
},
"openrouter": {
"provider": "openrouter",
"label": "OpenRouter",
"provider_api": "openai-completions",
"base_url": "https://openrouter.ai/api/v1",
"model": "openai/gpt-5.1",
"models": ["openai/gpt-5.1", "anthropic/claude-sonnet-4.5", "qwen/qwen3-max"],
"api_key_env": "OPENROUTER_API_KEY",
"source": "fallback",
},
"ollama": {
"provider": "ollama",
"label": "Ollama Local",
"provider_api": "ollama-generate",
"base_url": "http://127.0.0.1:11434",
"model": "qwen2.5:7b",
"models": ["qwen2.5:7b", "llama3.1:8b", "mistral:7b"],
"api_key_env": "",
"source": "fallback",
},
}
MODELS_DEV_PROVIDER_KEYS = {
"minimax": "minimax",
"openai": "openai",
"anthropic": "anthropic",
"deepseek": "deepseek",
"alibaba": "alibaba",
"moonshotai": "moonshotai",
"openrouter": "openrouter",
}
def list_fallback_llm_provider_presets() -> list[dict[str, Any]]:
return [dict(value) for value in FALLBACK_LLM_PROVIDER_PRESETS.values()]
def get_fallback_llm_provider_preset(provider: str) -> dict[str, Any]:
key = provider.strip().lower()
if key not in FALLBACK_LLM_PROVIDER_PRESETS:
raise ValueError(f"Unsupported LLM provider preset: {provider}")
return dict(FALLBACK_LLM_PROVIDER_PRESETS[key])
async def refresh_llm_provider_preset(provider: str) -> dict[str, Any]:
fallback = get_fallback_llm_provider_preset(provider)
models_dev_key = MODELS_DEV_PROVIDER_KEYS.get(fallback["provider"])
if not models_dev_key:
return fallback
async with httpx.AsyncClient(timeout=15.0, follow_redirects=True) as client:
response = await client.get(
MODELS_DEV_URL,
headers={"User-Agent": "Planet/1.0"},
)
response.raise_for_status()
catalog = response.json()
upstream = catalog.get(models_dev_key)
if not isinstance(upstream, dict):
return fallback
upstream_models = upstream.get("models") if isinstance(upstream.get("models"), dict) else {}
model_ids = list(upstream_models.keys())[:80]
base_url = upstream.get("api") or fallback["base_url"]
if fallback["provider"] == "deepseek" and base_url == "https://api.deepseek.com":
base_url = "https://api.deepseek.com/v1"
refreshed = {
**fallback,
"label": upstream.get("name") or fallback["label"],
"base_url": base_url,
"model": model_ids[0] if model_ids else fallback["model"],
"models": model_ids or fallback["models"],
"api_key_env": (upstream.get("env") or [fallback["api_key_env"]])[0],
"source": MODELS_DEV_URL,
}
return refreshed

View File

@@ -14,6 +14,11 @@ from app.core.time import to_iso8601_utc
from app.models.datasource import DataSource
from app.models.task import CollectionTask
from app.services.collectors.registry import collector_registry
from app.services.datasource_connectivity import (
build_builtin_connectivity_checksum,
get_builtin_effective_candidate,
save_connectivity_success,
)
logger = get_logger(__name__)
@@ -179,6 +184,23 @@ async def run_collector_task(collector_name: str):
task_result = await collector.run(db)
datasource.last_run_at = datetime.now(UTC)
datasource.last_status = task_result.get("status")
if datasource.last_status == "success":
effective_candidate = await get_builtin_effective_candidate(db, datasource.source)
checksum, _credential_context = await build_builtin_connectivity_checksum(
datasource.source,
effective_candidate["endpoint"],
effective_candidate["auth_type"],
effective_candidate["headers"],
effective_candidate["config"],
db,
)
await save_connectivity_success(
db,
datasource.source,
checksum,
{"status_code": None},
connected_by="collection",
)
await _update_next_run_at(datasource, db)
logger.info_event(
"Collector completed",

View File

@@ -19,6 +19,10 @@ ALLOWED_ACTIONS: dict[str, dict[str, Any]] = {
"command": ["./planet.sh", "restart", "-b"],
"recovery_mode": "backend",
},
"restart-frontend": {
"command": ["./planet.sh", "restart", "-f"],
"recovery_mode": "frontend",
},
"restart-ai-provider": {
"command": ["./planet.sh", "restart", "-a"],
"recovery_mode": "ai-provider",

View File

@@ -2,6 +2,7 @@ from __future__ import annotations
import argparse
import os
import shlex
import subprocess
import sys
import time
@@ -59,6 +60,8 @@ def wait_for_recovery(action: str) -> tuple[bool, str]:
recovery_mode = get_action_recovery_mode(action)
if recovery_mode == "backend":
return wait_for_http("http://localhost:8000/health"), "backend health recovery"
if recovery_mode == "frontend":
return wait_for_http("http://localhost:3000"), "frontend entrypoint recovery"
if recovery_mode == "ai-provider":
return wait_for_http("http://localhost:8010/health"), "ai provider health recovery"
if recovery_mode == "database":
@@ -108,8 +111,9 @@ def main() -> int:
)
append_task_log(args.task_id, "restart command started")
shell_command = shlex.join(command)
completed = subprocess.run(
command,
["zsh", "-ic", shell_command],
cwd=str(ROOT_DIR),
env=env,
capture_output=True,

View File

@@ -6,6 +6,7 @@ from unittest.mock import AsyncMock, MagicMock, patch
from app.services.collectors.top500 import TOP500Collector
from app.services.collectors.base import BaseCollector, HTTPCollector
from app.models.task import CollectionTask
class TestBaseCollector:
@@ -19,6 +20,31 @@ class TestBaseCollector:
assert collector.module == "L1"
assert collector.frequency_hours == 4
@pytest.mark.asyncio
async def test_update_phase_progress_tracks_phase_fields(self, mock_db_session):
"""Test phase-level progress updates independently from record totals"""
collector = TOP500Collector()
task = CollectionTask(datasource_id=1, status="running", phase="fetching")
collector._current_task = task
collector._db_session = mock_db_session
with patch.object(collector, "_publish_task_update", new=AsyncMock()) as publish:
await collector.update_phase_progress(
current=512,
total=1024,
unit="bytes",
message="Downloading dataset",
commit=True,
)
assert task.phase_progress == 50.0
assert task.phase_current == 512
assert task.phase_total == 1024
assert task.phase_unit == "bytes"
assert task.phase_message == "Downloading dataset"
mock_db_session.commit.assert_awaited_once()
publish.assert_awaited_once()
class TestTOP500Collector:
"""Tests for TOP500Collector"""

View File

@@ -0,0 +1,199 @@
from types import SimpleNamespace
import pytest
from httpx import ASGITransport, AsyncClient
from app.api.v1.datasource_config import get_ai_provider_client
from app.core.security import get_current_user
from app.core.target_schema_registry import get_target_schema, list_target_schemas
from app.main import app
from app.models.user import User
from app.services.datasource_mapping import execute_mapping, persist_mapped_records, redact_for_llm
SAMPLE_AIS = {
"data": [
{
"mmsi": "257123000",
"latitude": "59.91",
"longitude": "10.75",
"speedOverGround": "12.4",
"timestamp": "2026-04-28T00:00:00Z",
"api_token": "secret-value",
}
]
}
def test_registry_exposes_v1_target_schemas():
keys = {schema["key"] for schema in list_target_schemas()}
assert {"vessel_ais", "geo_points", "generic_records"}.issubset(keys)
assert get_target_schema("vessel_ais").destination == "vessel_position"
def test_mapping_engine_maps_and_validates_vessel_ais():
mapping = {
"source": {"items_path": "$.data[*]"},
"fields": {
"mmsi": {"path": "$.mmsi", "type": "integer"},
"lat": {"path": "$.latitude", "type": "float"},
"lon": {"path": "$.longitude", "type": "float"},
"sog": {"path": "$.speedOverGround", "type": "float"},
"received_at": {"path": "$.timestamp", "type": "datetime"},
},
}
result = execute_mapping(SAMPLE_AIS, mapping, "vessel_ais")
assert result["mapped_count"] == 1
assert result["failed_count"] == 0
assert result["records"][0]["mmsi"] == 257123000
assert result["records"][0]["lat"] == 59.91
def test_mapping_engine_reports_schema_errors():
mapping = {
"source": {"items_path": "$.data[*]"},
"fields": {
"mmsi": {"path": "$.mmsi", "type": "integer"},
"lat": {"path": "$.missing_lat", "type": "float"},
"lon": {"path": "$.longitude", "type": "float"},
},
}
result = execute_mapping(SAMPLE_AIS, mapping, "vessel_ais")
assert result["mapped_count"] == 0
assert result["failed_count"] == 1
assert any("lat" in error for error in result["errors"][0]["errors"])
def test_redact_for_llm_masks_secret_like_fields():
redacted = redact_for_llm(SAMPLE_AIS)
assert redacted["data"][0]["api_token"] == "[REDACTED]"
@pytest.mark.asyncio
async def test_persist_mapped_records_writes_generic_records():
class FakeDB:
def __init__(self):
self.added = []
self.committed = False
def add(self, value):
self.added.append(value)
async def commit(self):
self.committed = True
db = FakeDB()
count = await persist_mapped_records(
db,
datasource_name="custom_weather",
datasource_config_id=42,
target_schema="generic_records",
records=[{"source_id": "row-1", "data": {"temp": 25}}],
mapping_version=3,
)
assert count == 1
assert db.committed is True
assert db.added[0].source == "custom_weather"
assert db.added[0].data_type == "generic_records"
assert db.added[0].extra_data["mapping_version"] == 3
@pytest.mark.asyncio
async def test_mapping_preview_api_uses_deterministic_engine():
def override_get_current_user():
return User(
id=1,
username="testuser",
email="test@example.com",
password_hash="hashed",
role="admin",
is_active=True,
)
app.dependency_overrides = {get_current_user: override_get_current_user}
transport = ASGITransport(app=app)
try:
async with AsyncClient(transport=transport, base_url="http://test") as client:
response = await client.post(
"/api/v1/datasources/mappings/preview",
json={
"sample_payload": SAMPLE_AIS,
"target_schema": "vessel_ais",
"mapping_json": {
"source": {"items_path": "$.data[*]"},
"fields": {
"mmsi": {"path": "$.mmsi", "type": "integer"},
"lat": {"path": "$.latitude", "type": "float"},
"lon": {"path": "$.longitude", "type": "float"},
},
},
},
)
finally:
app.dependency_overrides.clear()
assert response.status_code == 200
payload = response.json()
assert payload["success"] is True
assert payload["preview"]["records"][0]["mmsi"] == 257123000
@pytest.mark.asyncio
async def test_mapping_propose_api_redacts_sample_before_ai():
seen_context = {}
class FakeAIClient:
async def analyze(self, request, request_id=None):
seen_context.update(request.context)
return SimpleNamespace(
content=(
'{"source":{"items_path":"$.data[*]"},"fields":{'
'"mmsi":{"path":"$.mmsi","type":"integer"},'
'"lat":{"path":"$.latitude","type":"float"},'
'"lon":{"path":"$.longitude","type":"float"}}}'
)
)
def override_get_current_user():
return User(
id=1,
username="testuser",
email="test@example.com",
password_hash="hashed",
role="admin",
is_active=True,
)
def override_ai_client():
return FakeAIClient()
app.dependency_overrides = {
get_current_user: override_get_current_user,
get_ai_provider_client: override_ai_client,
}
transport = ASGITransport(app=app)
try:
async with AsyncClient(transport=transport, base_url="http://test") as client:
response = await client.post(
"/api/v1/datasources/mappings/propose",
json={
"sample_payload": SAMPLE_AIS,
"target_schema": "vessel_ais",
"use_ai": True,
},
)
finally:
app.dependency_overrides.clear()
assert response.status_code == 200
payload = response.json()
assert payload["mapping_json"]["meta"]["generated_by"] == "ai_provider"
assert seen_context["sample_payload"]["data"][0]["api_token"] == "[REDACTED]"

View File

@@ -121,6 +121,25 @@ class TestCollectionTaskModel:
)
assert task.records_processed == 100
def test_task_with_phase_progress(self):
"""Test collection task phase-level progress fields"""
task = CollectionTask(
datasource_id=1,
status="running",
phase="fetching",
phase_progress=42.5,
phase_message="Downloading dataset",
phase_current=1024,
phase_total=4096,
phase_unit="bytes",
)
assert task.phase == "fetching"
assert task.phase_progress == 42.5
assert task.phase_message == "Downloading dataset"
assert task.phase_current == 1024
assert task.phase_total == 4096
assert task.phase_unit == "bytes"
def test_task_error_message(self):
"""Test collection task with error message"""
task = CollectionTask(

View File

@@ -0,0 +1,149 @@
from datetime import datetime, timedelta, timezone
import pytest
from httpx import ASGITransport, AsyncClient
from app.api.v1.visualization import convert_vessels_to_geojson
from app.db.session import get_db
from app.main import app
from app.models.vessel import VesselPosition, VesselStatic
from app.services import barentswatch
from app.services.collectors.vessel_ais import VesselAISCollector
def test_vessel_collector_transforms_barentswatch_like_records():
collector = VesselAISCollector()
records = collector.transform(
[
{
"mmsi": "257123000",
"lat": "59.91",
"lon": "10.73",
"sog": 12.4,
"cog": 214,
"nav_status": 0,
"shipType": 70,
"name": "OSLO TRADER",
},
{"mmsi": "bad", "lat": 120, "lon": 10},
]
)
assert len(records) == 1
assert records[0]["mmsi"] == 257123000
assert records[0]["vessel_type_name"] == "Cargo"
assert records[0]["lat"] == pytest.approx(59.91)
def test_barentswatch_reads_credentials_from_zshrc(tmp_path):
zshrc = tmp_path / ".zshrc"
zshrc.write_text(
"\n".join(
[
"export BARENTSWATCH_CLIENT_ID='client-from-zshrc'",
'export BARENTSWATCH_CLIENT_SECRET="secret-from-zshrc" # local dev credential',
]
),
encoding="utf-8",
)
values = barentswatch._read_zshrc_env(zshrc)
assert values["BARENTSWATCH_CLIENT_ID"] == "client-from-zshrc"
assert values["BARENTSWATCH_CLIENT_SECRET"] == "secret-from-zshrc"
@pytest.mark.asyncio
async def test_barentswatch_resolves_config_from_zshrc(tmp_path, monkeypatch):
zshrc = tmp_path / ".zshrc"
zshrc.write_text(
"\n".join(
[
"export BARENTSWATCH_CLIENT_ID=client-from-zshrc",
"export BARENTSWATCH_CLIENT_SECRET=secret-from-zshrc",
]
),
encoding="utf-8",
)
monkeypatch.delenv("BARENTSWATCH_CLIENT_ID", raising=False)
monkeypatch.delenv("BARENTSWATCH_CLIENT_SECRET", raising=False)
monkeypatch.delenv("BARRENTSWATCH_CLIENT_ID", raising=False)
monkeypatch.delenv("BARRENTSWATCH_CLIENT_SECRET", raising=False)
monkeypatch.setattr(barentswatch.Path, "home", lambda: tmp_path)
config = await barentswatch.resolve_barentswatch_config(None)
assert config.client_id == "client-from-zshrc"
assert config.client_secret == "secret-from-zshrc"
assert config.credential_source == "~/.zshrc"
def test_convert_vessels_to_geojson():
position = VesselPosition(
mmsi=257123000,
lat=59.91,
lon=10.73,
sog=12.4,
cog=214,
heading=215,
nav_status=0,
received_at=datetime(2026, 4, 28, 1, 0, tzinfo=timezone.utc),
)
static = VesselStatic(
mmsi=257123000,
name="OSLO TRADER",
vessel_type=70,
vessel_type_name="Cargo",
flag="NO",
length=185,
)
payload = convert_vessels_to_geojson([(position, static)])
assert payload["type"] == "FeatureCollection"
assert payload["features"][0]["geometry"]["coordinates"] == [10.73, 59.91]
assert payload["features"][0]["properties"]["mmsi"] == 257123000
assert payload["features"][0]["properties"]["vessel_type_name"] == "Cargo"
@pytest.mark.asyncio
async def test_vessels_geojson_endpoint_filters_type_and_bbox():
now = datetime(2026, 4, 28, 1, 0, tzinfo=timezone.utc)
rows = [
(
VesselPosition(mmsi=1, lat=59.9, lon=10.7, received_at=now),
VesselStatic(mmsi=1, name="Cargo Ship", vessel_type=70, vessel_type_name="Cargo"),
),
(
VesselPosition(mmsi=2, lat=60.3, lon=5.3, received_at=now - timedelta(minutes=1)),
VesselStatic(mmsi=2, name="Passenger Ship", vessel_type=60, vessel_type_name="Passenger"),
),
]
class _Result:
def all(self):
return rows
class _FakeSession:
async def execute(self, _query):
return _Result()
async def override_get_db():
yield _FakeSession()
app.dependency_overrides[get_db] = override_get_db
transport = ASGITransport(app=app)
try:
async with AsyncClient(transport=transport, base_url="http://test") as client:
response = await client.get(
"/api/v1/visualization/geo/vessels",
params={"bbox": "0,50,20,70", "type": "cargo"},
)
assert response.status_code == 200
data = response.json()
assert data["count"] == 1
assert data["features"][0]["properties"]["name"] == "Cargo Ship"
assert data["stats"]["by_type"]["Cargo"] == 1
finally:
app.dependency_overrides.clear()

View File

@@ -86,6 +86,12 @@ CREATE TABLE collection_tasks (
id BIGSERIAL PRIMARY KEY,
datasource_id INTEGER NOT NULL REFERENCES data_sources(id) ON DELETE CASCADE,
status task_status NOT NULL DEFAULT 'pending',
phase VARCHAR(30) DEFAULT 'queued',
phase_progress FLOAT,
phase_message VARCHAR(255),
phase_current BIGINT,
phase_total BIGINT,
phase_unit VARCHAR(30),
started_at TIMESTAMP WITH TIME ZONE,
completed_at TIMESTAMP WITH TIME ZONE,
records_processed INTEGER DEFAULT 0,

View File

@@ -8,6 +8,9 @@ services:
args:
PYTHON_IMAGE: ${PYTHON_IMAGE:-python:3.14-slim}
UV_IMAGE: ${UV_IMAGE:-ghcr.io/astral-sh/uv:latest}
env_file:
- ./aiprovider/.env
- ${PLANET_AI_PROVIDER_RUNTIME_ENV_FILE:-./aiprovider/.env}
container_name: planet_aiprovider
ports:
- "8010:8010"

View File

@@ -10,6 +10,7 @@ services:
UV_IMAGE: ${UV_IMAGE:-ghcr.io/astral-sh/uv:latest}
env_file:
- ./aiprovider/.env
- ${PLANET_AI_PROVIDER_RUNTIME_ENV_FILE:-./aiprovider/.env}
container_name: planet_aiprovider
ports:
- "8010:8010"

View File

@@ -8,6 +8,140 @@ This project follows the repository versioning rule:
- `improvement` -> `+0.0.1`bugfix + 小功能混合)
- `bugfix` -> `+0.0.1`
## [0.46.3] — 2026-04-30
Released: 2026-04-30
### 🐛 Fixes
- 优化 Starlink footprint 显示后的地球拖拽性能,避免旋转地球时每帧重建 footprint 大网格,同时保持现有视觉效果不变。
- 恢复点击线缆后的呼吸透明度动画,让 locked / hover 线缆重新使用既有 pulse 配置。
---
## [0.46.2] — 2026-04-30
Released: 2026-04-30
### 🐛 Fixes
- 修复 Earth 启动时高清材质、云图和图层可见性绕过 `startupPriority` 的问题,统一由启动队列按文档顺序加载。
- 修复保存为关闭的高清材质/图层仍会先加载再关闭的问题,并保持海陆基座作为国界线图层的常驻底图。
- 修复搜索跳转会误关媒体面板、船只轨迹末端不贴合当前船只、Iridium footprint 被地表层遮挡等 Earth 交互问题。
### 📝 Documentation
- 更新 Earth 图层顺序、样式参考、使用手册和 AIS 聚合计划,补齐中英文说明与后续接入策略。
---
## [0.46.1] — 2026-04-30
Released: 2026-04-30
### 🐛 Fixes
- 修复新增 technical docs 文件存在但未进入 Docs 前端白名单时,侧栏不显示且 Markdown 链接无法解析到 `/docs/<slug>` 的问题。
- 补齐数据源/采集器连接验证与 Earth Interactable 使用说明的英文文档,保证公开 Docs 切换 EN 时同名页面可访问。
- 清理中英文 technical docs 中裸 `.md` 文件名链接标题,改为面向读者的语义标题。
### 📝 Documentation
- 将 Docs 前端白名单、公开文档双语配对、裸文件名链接标题三项检查写入 Claude 与 Codex 的 docs 技能流程。
---
## [0.46.0] — 2026-04-30
Released: 2026-04-30
### ✨ Highlights
- Earth 新增通用 Interactable 图标层船只、算力中心、BGP 事件与观测站统一使用批量 Points、屏幕拾取、状态 glow 和状态缩放。
- BGP 事件保留向外扩散圈,观测站保留雷达扫描层,并与 Interactable 主图标解耦到稳定的地表渲染层级。
- 登陆点回归黄色球形 Sprite贴近海缆层级并保持更稳定的地表显示和遮挡表现。
### 🔧 Improvements
- 新增 SVG asset 到 canvas texture 的 Interactable 资产加载路径,支持统一图标资源、缓存和可选染色。
- 同坐标 Interactable 自动做地表切向避让,降低重叠物件无法选择的问题。
- 优化 Earth toolbar 初始尺寸注入,避免首次显示原始尺寸后再跳到缩放尺寸。
- 补充 Interactable 计划、使用说明、图层顺序和 Earth 前端上下文文档。
- 修复船只 hover/locked 状态仅发光但放大反馈不明显的问题,将已有状态缩放接入通用图标层。
---
## [0.45.0] — 2026-04-29
### ✨ Highlights
- 采集任务新增阶段级量化进度,`fetching` 可展示百分比、阶段说明和字节下载量。
- AI Provider 启动链路支持从 `aiprovider/.env``~/.zshrc` 注入运行期配置,并避免密钥/模型变化触发镜像重建。
- AI Provider Docker build context 收敛到服务必需文件,`uv sync` 接入 BuildKit 缓存以减少重复下载。
### 🔧 Improvements
- IPtoASN、OpenGeoFeed、NRO delegated 下载型采集器接入真实字节进度上报。
- 数据源页、采集中任务弹窗和任务历史页展示阶段摘要,并在 tooltip 中保留完整进度细节。
- 调整 Earth 船只默认高度偏移,进一步贴近地表展示。
---
## [0.44.2] — 2026-04-29
### 📝 Documentation
- 补充 Earth 船只图层技术文档,记录分桶 `THREE.Points` 批量渲染、同尺寸交互 overlay 和屏幕空间 picking 的设计约束。
- 同步 Earth 渲染图层顺序和样式参考,明确 AIS 船只 renderOrder、depthTest、图标尺寸、航向分桶与 hover 命中半径。
- 更新船只渲染性能计划状态,标注 `0.44.1` 已落地的实现与后续全球 AIS / LOD 演进方向。
---
## [0.44.1] — 2026-04-29
### 🐛 Fixes
- 修复 Earth 船只图层拖动不跟手的问题,将普通船只从独立 Sprite 切换为按航向分桶的批量 Points 渲染。
- 修正船只 hover/click 拾取错位,改为屏幕空间命中检测并在拖拽/惯性期间跳过 hover 拾取。
- 统一 AIS 船只普通态与交互态方向,并让 hover/locked glow 与普通图标保持同尺寸覆盖。
- 恢复船只深度测试并收敛默认图标尺寸,避免北部岛屿/冰面附近出现明显压盖陆地的视觉问题。
---
## [0.44.0] — 2026-04-29
### ✨ Highlights
- 重构数据源与采集器设置边界:数据源页回归目录和采集触发,采集器 endpoint、请求头、凭证与连接验证统一进入设置页
- BarentsWatch AIS 完整接入凭证解析、连接检查、默认教程、AI 生成教程和船只采集/可视化链路
- Earth 新增船只图例、缩放反馈胶囊、缩放感知拖拽灵敏度,并将船只渲染性能优化方案沉淀到 plans
- 仪表盘重启服务新增前端重启 action并让 runner 通过 `~/.zshrc` 继承本地环境变量
### 🔧 Improvements
- 采集器连接状态改为基于成功采集或手动连接校验 checksum 判断,避免只依赖前端样式状态
- 数据源页新增采集中任务标签和任务进度弹窗,内置与自定义数据源统一展示
- Earth 国界线进一步贴近地表,并补充船只图层渲染顺序、样式和用户手册说明
- docs skill 与 Claude/Codex 文档流程补齐技术文档和 plans 的职责边界
---
## [0.43.1] — 2026-04-28
### 🐛 Fixes
- 修正 `planet.sh` 在全量 restart 后启动 AI Provider 时的提示语义,避免把预期内未就绪描述成异常不健康
---
## [0.43.0] — 2026-04-28
### ✨ Highlights
- 新增 Earth 船舶追踪链路,接入 BarentsWatch AIS 凭证配置、采集器、后端 vessel 模型/API 与前端 Earth 船舶图层
- 新增自定义数据源映射流程,支持样本抓取、目标 schema、AI 辅助生成映射、预览校验和映射执行
- Settings 拆分 AI Provider 与采集器凭证配置DataSources 只保留采集状态、运行参数和必要引导
### 🔧 Improvements
- AI Provider 支持运行时 LLM 配置、provider preset 下拉与刷新,并在 Playground 中引导到 AI 配置页
- Markdown 渲染器补齐代码块复制按钮、语言标签、任务列表、图片、自动链接、删除线和文档主题样式
- Docs 公开导航改为显式元数据白名单,避免开发任务文档自动出现在“其他”分组
- 将 Codex/Claude cleanup、docs、goal-driven、release 流程补充 CLI-first 约束,并把 `rules.md` 整理成可按模块加载的工程规则
---
## [0.42.2] — 2026-04-28
### 🐛 Fixes
- Docs 中文模式下补齐左侧分组、文档标题、页头分类与搜索结果分类翻译,并更新文档站品牌标题/副标题文案
---
## [0.42.1] — 2026-04-28
### 🐛 Fixes

View File

@@ -24,6 +24,10 @@
- [earth-webgl-instancing-satellites-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-webgl-instancing-satellites-plan.md)
- [earth-real-terrain-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-real-terrain-plan.md)
- [earth-news-source-configuration-and-collector-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-news-source-configuration-and-collector-plan.md)
- [earth-news-cruise-summary-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-news-cruise-summary-plan.md)
- [earth-vessel-rendering-performance-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-vessel-rendering-performance-plan.md)
- [AIS 多源采集、冲突记录与聚合接口计划](/home/ray/dev/linkong/planet/docs/plans/earth-vessel-ais-aggregation-plan.md)
- [earth-interactable-layer-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-interactable-layer-plan.md)
- [frontend-public-docs-site-plan.md](/home/ray/dev/linkong/planet/docs/plans/frontend-public-docs-site-plan.md)
- [frontend-ai-playground-development-plan.md](/home/ray/dev/linkong/planet/docs/plans/frontend-ai-playground-development-plan.md)
- [ue5-mvp-fused-plan.md](/home/ray/dev/linkong/planet/docs/plans/ue5-mvp-fused-plan.md)

View File

@@ -0,0 +1,426 @@
# 自定义 API 数据源与 LLM 映射系统 — 实施计划
**状态**:规划中
**创建日期**2026-04-28
**核心原则**LLM 辅助生成映射配置;生产采集使用确定性转换引擎
## 已确认决策
| 项目 | 决策 |
|-----|------|
| 自定义 API 的定位 | 作为内置数据源的补充入口,不直接等同于 Earth 新功能 |
| LLM 的职责 | 探索未知 API、分析样本 JSON、生成 mapping 草案 |
| 采集时是否调用 LLM | 不调用;采集链路必须确定性、可审计、可复现 |
| 自定义数据如何进入 Earth | 必须映射到已支持的目标 schema或先进入通用数据沉淀 |
| 外部凭证放置位置 | Settings / 外部集成统一管理 provider tokenDataSources 引用 provider profile |
| TimescaleDB | 放入 TODO高频时序数据稳定后再评估迁移 |
---
## 一、背景与问题
当前系统已经有 `datasource_configs`,可以配置自定义数据源的 endpoint、auth、headers、config也已经有部分 collector 会读取这些配置。但这只能解决“怎么请求数据”,还没有解决以下问题:
- API 返回 JSON 后,如何转换成系统已有领域模型。
- 自定义数据源是补充已有能力,还是全新数据沉淀。
- 转换规则由谁生成、谁校验、谁执行。
- 未知数据是否能自动在 Earth 上展示。
- 外部 token 是放在全局配置中心,还是放在每个 datasource 下。
专业做法是把“请求配置”“外部凭证”“目标 schema”“字段映射”“采集执行”拆开
- Settings 管外部集成凭证,例如 AI Provider、BarentsWatch、未来付费 AIS API。
- DataSources 管具体数据源实例,例如 endpoint、调度频率、目标 schema、mapping 版本。
- LLM 只在配置阶段辅助生成 mapping不进入生产采集链路。
- Earth 只消费明确 schema 的数据,不消费任意未知 JSON。
---
## 二、目标架构
### 2.1 自定义 API 数据源生命周期
```mermaid
flowchart LR
A[配置 endpoint/auth/request] --> B[抓取 sample JSON]
B --> C[选择目标 schema]
C --> D[LLM 生成 mapping 草案]
D --> E[确定性 mapping engine 预览]
E --> F[schema validation]
F --> G[保存 mapping version]
G --> H[scheduler 执行 mapped collector]
H --> I[写入目标表或 generic_records]
```
### 2.2 目标 schema 分层
| schema | 用途 | Earth 可视化 |
|-------|------|-------------|
| `vessel_ais` | 船只 AIS 位置、航速、航向、MMSI 等 | 进入船舶图层 |
| `geo_points` | 通用点位数据,包含经纬度、名称、类型、时间 | 进入通用 geo layerTODO |
| `news_events` | 新闻/事件类数据,带时间、地点、摘要、来源 | 复用新闻/事件链路 |
| `compute_centers` | 算力中心、机房、数据中心数据 | 复用算力中心图层 |
| `generic_records` | 未知结构化数据沉淀 | 不直接展示 |
v1 建议优先实现:
- `vessel_ais`
- `geo_points`
- `generic_records`
其他 schema 可先在 registry 中预留名称,但不承诺完整落库与可视化。
### 2.3 LLM 的边界
LLM 可以做:
- 根据 API 文档或 sample JSON 解释字段含义。
- 推荐目标 schema。
- 生成 mapping JSON 草案。
- 给出字段置信度和需要人工确认的字段。
- 帮用户发现分页、数组路径、时间字段、坐标字段。
LLM 不应该做:
- 在正式采集时参与每批数据转换。
- 生成并执行 Python/JavaScript 代码。
- 接触 API key、bearer token、basic auth password。
- 自动创建新的 Earth 图层或数据库表。
---
## 三、后端实施计划
### Phase 1 — Target Schema Registry
新增代码级 registry统一描述系统支持的目标数据类型。
每个 target schema 至少包含:
- `key`:例如 `vessel_ais`
- `label`:前端展示名称。
- `description`:适用场景。
- `fields`:字段名、类型、是否必填、说明、示例。
- `validator`Pydantic 或等价校验器。
- `destination`:写入目标,例如 vessel 表、generic_records、future geo layer。
示例概念:
```json
{
"key": "vessel_ais",
"fields": [
{"name": "mmsi", "type": "integer", "required": true},
{"name": "lat", "type": "float", "required": true},
{"name": "lon", "type": "float", "required": true},
{"name": "sog", "type": "float", "required": false},
{"name": "cog", "type": "float", "required": false},
{"name": "received_at", "type": "datetime", "required": false}
]
}
```
### Phase 2 — Mapping Template Model
新增 mapping 配置持久化表,建议命名为 `datasource_mapping_templates`
关键字段:
- `id`
- `datasource_config_id`
- `target_schema`
- `mapping_json`
- `sample_payload_hash`
- `validation_status`
- `version`
- `is_active`
- `created_at`
- `updated_at`
`mapping_json` 是声明式 DSL不允许任意代码执行。
示例:
```json
{
"source": {
"items_path": "$.data.vessels[*]"
},
"fields": {
"mmsi": {"path": "$.mmsi", "type": "integer"},
"lat": {"path": "$.latitude", "type": "float"},
"lon": {"path": "$.longitude", "type": "float"},
"sog": {"path": "$.speedOverGround", "type": "float", "default": null},
"received_at": {"path": "$.timestamp", "type": "datetime"}
}
}
```
### Phase 3 — Deterministic Mapping Engine
实现独立 mapping engine输入 sample/raw payload 和 mapping JSON输出目标 schema 记录。
v1 支持能力:
- JSONPath/JMESPath 风格路径提取。
- 数组展开。
- 默认值。
- 基础类型转换string、integer、float、boolean、datetime。
- 坐标范围校验。
- 简单枚举映射。
- 错误收集:缺字段、类型转换失败、路径不存在。
明确不支持:
- 任意表达式执行。
- 用户提交脚本。
- LLM runtime 修复。
### Phase 4 — LLM Mapping Assistant API
新增配置阶段 API
- `POST /api/v1/datasources/custom/sample`
- 按 datasource 请求配置抓取 sample JSON。
- `GET /api/v1/datasources/target-schemas`
- 返回可选目标 schema 和字段说明。
- `POST /api/v1/datasources/mappings/propose`
- 输入 sample JSON + target schema调用 AI provider 生成 mapping 草案。
- `POST /api/v1/datasources/mappings/preview`
- 使用确定性 mapping engine 预览转换结果。
- `POST /api/v1/datasources/mappings`
- 保存 mapping 版本。
- `PUT /api/v1/datasources/mappings/{id}`
- 更新 mapping生成新版本或覆盖草稿。
- `POST /api/v1/datasources/{id}/run-mapped`
- 手动触发一次 mapped collector。
安全要求:
- `propose` 请求发送给 LLM 前必须脱敏 sample。
- auth headers、token、password 不进入 prompt。
- LLM 返回结果必须再经过 mapping schema 校验。
### Phase 5 — Generic Mapped HTTP Collector
新增通用 collector
- 读取 `DataSourceConfig` 请求配置。
- 读取 active mapping template。
- 拉取 API 数据。
- 使用 mapping engine 转换。
- 使用 target schema validator 校验。
- 调用 destination handler 写入目标表或 generic storage。
- 将失败记录写入错误日志或 dead-letter 结构。
对于 `generic_records`
- 保存 datasource id。
- 保存 target schema。
- 保存 normalized JSON。
- 保存 raw payload 摘要或 raw reference。
- 保存采集时间、source timestamp、mapping version。
---
## 四、前端实施计划
### Phase 1 — Settings 外部集成
Settings 中保留统一外部集成配置:
- AI Providerbase URL、model、API key。
- BarentsWatchclient id/client secret 或 bearer token。
- 未来付费接口AISHub、MarineTraffic、VesselFinder 等 provider profile。
DataSources 不直接管理全局 secret只引用 provider profile。
### Phase 2 — Settings 自定义源向导
自定义数据源配置入口应放在 `/settings` 的“采集器设置”或后续专门的自定义采集器设置区。`/datasources` 保持数据源目录和采集触发职责,不再承载编辑入口。
自定义数据源配置改成向导或右侧 drawer
1. Request
- endpoint
- method
- auth profile
- headers
- query/body config
- schedule
2. Sample
- 点击抓取 sample
- 展示 JSON tree
- 支持选择数组根路径
3. Target Schema
- 选择 `vessel_ais``geo_points``generic_records`
- 展示该 schema 必填字段
4. Mapping Proposal
- 调用 LLM 生成 mapping 草案
- 显示字段匹配置信度
- 标出需要人工确认的字段
5. Preview
- 用确定性 engine 预览前 N 条转换结果
- 展示校验错误
6. Save & Enable
- 保存 mapping version
- 启用调度或仅保存草稿
### Phase 3 — 运维视图
为 mapped datasource 展示:
- 上次运行时间。
- 成功记录数。
- 失败记录数。
- 当前 mapping version。
- 目标 schema。
- 最近错误。
- 手动运行按钮。
---
## 五、数据库与存储策略
### v1继续使用 PostgreSQL
PostgreSQL 可以承载当前规模的采集、关系查询、JSONB 沉淀和基础时序查询。v1 不必因为“时序数据”立刻引入 TimescaleDB。
适合继续用 PostgreSQL 的场景:
- 数据量可控。
- 最近状态查询为主。
- 历史保留窗口较短。
- 查询模式还没稳定。
- 需要快速迭代 schema 与 mapping。
### TODOTimescaleDB
以下条件满足后,再评估 TimescaleDB
- AIS、遥测、轨迹类数据达到高频持续写入。
- 需要按时间窗口做聚合、降采样、retention policy。
- 单表时间序列查询明显成为瓶颈。
- 历史轨迹保留从 24h 扩展到数周或数月。
候选迁移对象:
- `vessel_position`
- future telemetry tables
- future generic time-series records
备选方案:
- PostgreSQL 原生按天/月分区。
- TimescaleDB hypertable。
- 热数据 PostgreSQL冷数据对象存储。
---
## 六、安全与治理
### Secret 管理
- Settings 中保存 provider credentials。
- API 返回配置时必须 mask secret。
- LLM prompt 只能包含脱敏 sample 和 schema 说明。
- 后续 TODO引入字段级加密或 KMS。
### Mapping 治理
- 每次 mapping 变更保留版本。
- active mapping 只能有一个。
- 允许保存 draft mapping。
- 运行记录关联 mapping version。
- 校验失败不能自动启用。
### 错误处理
常见错误类型:
- API 401/403凭证错误或过期。
- API 429限流需要调整 schedule。
- JSON path 不存在:上游结构变化。
- 类型转换失败mapping 规则错误。
- schema validation failed转换结果不满足目标模型。
每次运行需要记录:
- datasource id。
- mapping version。
- started_at / finished_at。
- fetched count。
- mapped count。
- written count。
- failed count。
- error summary。
---
## 七、测试计划
### Backend Unit Tests
- mapping engine
- path 提取。
- 数组展开。
- 默认值。
- 类型转换。
- datetime parse。
- 枚举映射。
- 缺字段错误。
- target schema registry
- `vessel_ais` 必填字段校验。
- `geo_points` 经纬度范围校验。
- `generic_records` 接受未知结构。
- LLM assistant
- mock provider 返回 mapping。
- 验证 secret 不进入 prompt。
- 验证非法 mapping 被拒绝。
### Backend Integration Tests
- sample JSON -> propose mapping -> preview -> save mapping。
- mapped collector 使用保存的 mapping 写入 `generic_records`
- `vessel_ais` sample 写入船舶相关目标结构。
- 上游 JSON 结构变化时,运行失败并记录错误。
### Frontend Tests
- 自定义数据源向导完整流程。
- 未配置 AI Provider 时,提示去 Settings 配置,但允许手写 mapping。
- LLM 返回不完整 mapping 时Preview 阶段显示校验错误。
- 保存 mapping 后展示 active version 和运行状态。
---
## 八、分期工作量
| 阶段 | 内容 | 估算 |
|-----|------|------|
| Phase 0 | 完成本规划、确认 schema registry 设计 | 0.5 天 |
| Phase 1 | target schema registry + mapping template model | 12 天 |
| Phase 2 | deterministic mapping engine | 23 天 |
| Phase 3 | sample/propose/preview/save API | 23 天 |
| Phase 4 | DataSources 自定义源向导 | 35 天 |
| Phase 5 | generic mapped collector + run history | 24 天 |
| Phase 6 | vessel_ais / geo_points destination handler | 24 天 |
---
## 九、当前差距与下一步
当前差距:
- `datasource_configs` 只描述请求配置,不描述目标 schema 和 mapping。
- 自定义源没有 sample -> schema -> mapping -> preview -> save 的闭环。
- 生产采集还没有通用 mapped collector。
- Settings 与 DataSources 的职责边界需要在 UI 上进一步明确。
- Earth 还没有通用 `geo_points` 图层。
下一步建议:
1. 先实现 target schema registry 和 mapping engine不急着接 LLM。
2. 用固定 sample JSON 做 `vessel_ais``generic_records` 的单元测试。
3. 再接 LLM propose API让 LLM 产出的只是 mapping 草案。
4. 最后做前端向导,把人工确认和 preview 放到启用之前。

View File

@@ -0,0 +1,313 @@
# Earth Interactable Layer Plan
## 背景
状态Phase 1 已经开始落地Phase 2 的 BGP 事件 / 观测站迁移和 Phase 3 的算力中心迁移也已完成。`frontend/public/earth/js/interactable.js` 已新增AIS 船只、BGP 事件、BGP 观测站和算力中心图层已经改为通过 `createInteractableLayer()` 使用通用批量 `Points`、hover / locked overlay、默认 glow、状态更新、asset icon 预加载、屏幕空间 picking、固定 / 距离缩放和跨 Interactable 同坐标避让。登陆点因 `THREE.Points` 边缘深度裁切和贴地层级要求,已退回专用 `THREE.Sprite` 黄色球路径,并与海缆同高度同 renderOrder。后续阶段聚焦把可复用的扩圈 / 雷达扇形动画正式沉淀成 `animations` 扩展。
当前实现说明和接入示例见:
- [earth-interactable-usage.md](/home/ray/dev/linkong/planet/docs/technical/zh/earth-interactable-usage.md)
当前 AIS 船只图层已经形成了一个适合作为基准的交互图标模式:
- 普通态使用批量 `THREE.Points` 渲染,避免每个对象一个 `Sprite` 带来的 draw call 和透明排序压力。
- hover / locked 态使用单点 overlay 叠加 glow不改变普通批次交互反馈清晰且成本低。
- moving / anchored 船只通过 canvas 点纹理表达不同形状moving 船只还按航向分桶。
- 拾取走屏幕空间命中,拖动和惯性期间跳过高频 hover picking。
- 图层高度贴近地表,仅保留很小的深度余量,避免“浮在表面层”的观感。
这个模式不应该只服务船只。后续 BGP 事件、BGP 观测站、算力中心、新闻事件、告警、地面传感器等都可能需要“图标类可交互元素”。如果每个图层继续各写一套 icon、glow、hover、locked、动画、picking 和图例逻辑,视觉会漂移,性能策略也会重复分叉。登陆点已经验证为例外:需要完整贴地且不被球面边缘裁切时,专用 Sprite 路径比通用 `Points` 更合适。
目标是把船只图层的成功做法抽象成一个通用接口:业务图层只描述“要画什么、在哪里、怎么交互”,底层统一负责批量渲染、默认 glow、状态 overlay、动画槽位、拾取和生命周期。
## 目标
1. 建立统一的 Earth 交互图标接口,作为未来地表图标类元素的默认入口。
2. 以 AIS 船只 glow 为默认 glow 视觉,其它图标默认沿用同一套 glow 质感。
3. 保留图标颜色、状态颜色、hover 放大、locked 强调、dimmed 聚焦、动画扩展等能力。
4. 支持 canvas / SVG / image icon不强行要求所有图标都可重着色。
5. 保持船只当前性能路线:批量绘制普通态,少量 overlay 处理交互态。
6. 给 BGP 事件扩圈、BGP 观测站雷达扇形等补充动画留出正式扩展点。
## 非目标
- 不在第一阶段重写所有 Earth 图层。
- 不把卫星、海缆、国家边界、真实地形这类非图标图层纳入同一个接口。
- 不为了抽象牺牲业务图标的差异表达例如船只航向、BGP 事件严重级别、观测站雷达扫掠。
- 不要求图片图标支持运行时重着色;图片图标只能通过预制多状态图片或 overlay tint 做有限表达。
## 核心设计
建议新增一个通用模块,例如:
```text
frontend/public/earth/js/interactable.js
```
它导出一个工厂或注册函数:
```js
createInteractableLayer({
id,
earth,
renderOrder,
altitudeOffset,
icon,
scale,
glow,
colors,
states,
animations,
picking,
data,
getPosition,
getKind,
getRotation,
getPayload,
});
```
业务模块仍保留自己的数据加载、图例、详情卡字段和业务语义。例如 `vessels.js` 负责 AIS 数据和船型映射,但 icon 渲染、hover overlay、locked overlay、默认 glow 和屏幕空间 picking 可以逐步迁入 `interactable.js`
## 参数草案
| 参数 | 类型 / 示例 | 默认值 | 说明 |
| --- | --- | --- | --- |
| `id` | `"vessels"` | 必填 | 图层唯一标识,用于 debug、picking、legend 和状态缓存。 |
| `earth` | `THREE.Object3D` | 必填 | 图层挂载目标,通常是 Earth root。 |
| `renderOrder` | `4.4` | `4` | 普通 icon 批次和 overlay 的基础渲染顺序。 |
| `altitudeOffset` | `0.2` | `0.2` | 图层高度,语义为 `CONFIG.earthRadius + altitudeOffset`。地表图标默认贴近真实地形基础层。 |
| `icon` | `{ type, source, draw, size, bins }` | 必填 | 图标来源。支持 canvas draw、SVG URL、image URL、内置 shape。 |
| `icon.fitSize` | `60``{ width: 60, height: 60 }` | `atlasCellSize` | asset 图标在 atlas canvas 内的最大绘制尺寸,默认居中等比 contain。SVG / 图片文件只负责原始形状,不需要为了显示大小手写 transform。 |
| `scale` | `{ base, min, max }` | `{ base: 1 }` | 基础缩放和距离稳定范围。当前船只可映射到 `VESSEL_POINT_SIZE` / `baseScale`。 |
| `sizeMode` | `"fixed" / "distance"` | `"fixed"` | 是否固定屏幕像素尺寸;非 fixed 时按相机到地表距离做比例缩放。 |
| `sizeScale` | `{ min, max, referenceFov }` | `{ min: 0.12, max: 3, referenceFov: 75 }` | `sizeMode !== "fixed"` 时的缩放限制和参考视角。 |
| `glow.enabled` | `true / false` | `true` | 是否启用默认 glow。默认 glow 以船只 hover / locked overlay 为基准。 |
| `glow.intensity` | `0.0 - 2.0` | `1` | glow 强度,内部映射到 canvas `shadowBlur`、opacity 或 shader uniform。 |
| `glow.colorMode` | `"state" / "icon" / "fixed"` | `"state"` | glow 颜色来源,默认跟随状态颜色。 |
| `hover.scale` | `1.0 - 2.0` | `1.18` | hover 放大倍率。当前船只保持同尺寸 glow overlay接口仍保留放大能力供其它图层使用。 |
| `hover.mode` | `"scale" / "glow-only" / "custom"` | `"scale"` | hover 反馈方式。船只可用 `"glow-only"`,其它图标默认放大。 |
| `colors.normal` | `"#4A90D9"` | icon 原色 | 普通态颜色。只有可上色 icon 生效。 |
| `colors.hover` | `"#7dd3fc"` | normal | hover 态颜色。 |
| `colors.locked` | `"#ffffff"` | hover | locked 态颜色。 |
| `colors.dimmed` | `"#9B9B9B"` | normal | 聚焦其它对象时的弱化颜色。 |
| `colors.byKind` | `{ cargo: "#4A90D9" }` | `{}` | 按业务类型着色如船型、BGP 严重级别。 |
| `colorable` | `true / false` | 由 icon 类型推断 | canvas shape 和 SVG mask 通常可上色;图片默认不可上色。 |
| `opacity` | `{ normal, hover, locked, dimmed }` | 船只当前值 | 各状态透明度。 |
| `rotation` | `{ enabled, bins, getAngle }` | disabled | 是否按角度分桶,例如船只按 COG 分 32 桶。 |
| `animations` | `IconAnimationSpec[]` | `[]` | 补充动画列表,例如扩圈、雷达扇形、脉冲、轨迹尾迹。 |
| `picking.radiusPx` | `22` | `20` | 屏幕空间命中半径。 |
| `picking.throttleMs` | `100` | `80` | hover picking 节流。 |
| `picking.skipWhileDragging` | `true` | `true` | 拖动和惯性期间跳过 hover picking。 |
| `zIndexPolicy` | `"surface-icon"` | `"surface-icon"` | 预设层级策略,避免每个业务图层手写高度和 renderOrder。 |
| `avoidance.enabled` | `true / false` | `true` | 是否参与跨 Interactable 的同坐标避让。默认开启,同一经纬度下的图标会沿地表切平面小幅排开,方便辨认和选择。 |
| `avoidance.radius` | `number` | `1.1` | 同坐标避让的第一圈半径,单位为地球本地坐标单位。 |
| `avoidance.precision` | `number` | `4` | 经纬度归并精度,默认约等于只处理几乎完全重叠的图标。 |
| `legend` | `{ label, color, shape }[]` | `[]` | 可选图例声明,业务层也可以继续自己导出。 |
| `metadata` | object | `{}` | 业务扩展数据,不参与渲染但参与 tooltip / info-card / search。 |
## Icon 规格
图标输入建议分三类:
```js
{
type: "canvas-shape",
size: 128,
draw(ctx, state) {
// draw triangle / dot / custom shape
},
}
```
```js
{
type: "svg-mask",
source: "/earth/assets/icons/bgp-event-dot.svg",
colorable: true,
}
```
```js
{
type: "image",
source: "/earth/assets/icons/vendor-logo.png",
colorable: false,
stateSources: {
hover: "/earth/assets/icons/vendor-logo-hover.png",
},
}
```
颜色策略:
- `canvas-shape` 默认可上色,适合船只、事件点、雷达站这类符号。
- `svg-mask` 如果能作为 mask 使用,则可上色;如果是完整多色 SVG则按图片处理。
- `image` 默认不可上色;需要状态变化时使用 `stateSources` 或额外 glow / ring。
## 默认 Glow 规范
默认 glow 以当前船只 overlay 为视觉基准:
- 普通态尽量不启用 glow保持地图干净。
- hover / locked 态叠加同位置 overlay。
- glow 颜色默认跟随状态颜色或业务类型颜色。
- glow blur 应该稳定,不随 camera zoom 夸张膨胀。
- 允许通过 `glow.intensity` 控制强度,但不要让业务图层各自发明完全不同的光晕语言。
建议内部把 glow 拆成两个层次:
1. `textureGlow`canvas texture 里的 `shadowBlur`,适合小图标 hover / locked。
2. `effectGlow`:额外 ring / halo / pulse适合告警、BGP 事件和锁定强调。
## 状态模型
通用状态至少包含:
| 状态 | 触发 | 默认表现 |
| --- | --- | --- |
| `normal` | 普通显示 | 批量 Points使用 normal 颜色和 opacity。 |
| `hover` | 指针悬停 | 默认放大并显示 glow船只可配置为同尺寸 glow-only。 |
| `locked` | 点击锁定 / 详情打开 | 强 glow、更高 opacity可选 ring 或 pulse。 |
| `dimmed` | 聚焦其它对象 | 降低 opacity保留上下文。 |
| `hidden` | 图层关闭或过滤 | 不参与绘制和 picking。 |
| `alert` | 业务告警 | 可叠加动画,不替代 locked 状态。 |
状态更新需要增量化:只在 hover 目标、locked 目标、过滤条件、数据版本或相机距离阈值变化时更新,不在每帧遍历全部 icon 写材质属性。
## 动画扩展
动画不直接塞进 icon 基础参数,而是作为 `animations` 列表注册。每个动画声明自己的 geometry / material / update 策略:
```js
{
type: "expanding-ring",
when: ["alert", "locked"],
color: "state",
radiusPx: [10, 42],
durationMs: 1400,
opacity: [0.8, 0],
}
```
```js
{
type: "radar-sweep",
when: ["normal", "hover", "locked"],
angleDeg: 72,
rotationMs: 2600,
opacity: 0.36,
}
```
首批建议内置动画:
| 动画 | 用例 | 说明 |
| --- | --- | --- |
| `pulse-ring` | locked、告警点 | 原地呼吸环,强调选中对象。 |
| `expanding-ring` | BGP 事件 | 向外扩散的事件波纹。 |
| `radar-sweep` | BGP 观测站 | 扇形扫描,可持续旋转。 |
| `orbiting-dot` | 数据流 / collector 活跃态 | 小点绕 icon 环绕,表达活动状态。 |
| `trail` | 移动目标 | 可选短尾迹,船只或飞机类目标使用。 |
动画必须支持批量或分组绘制,避免为每个对象创建独立的高频更新对象。只有 locked / hover / 少量 alert 对象可以使用单对象 overlay。
## 渲染策略
### 普通态
普通态优先使用分桶 `THREE.Points`
- 按 icon 类型、可上色策略、旋转分桶、纹理 key 分组。
- 每组一个 `BufferGeometry`,存 `position``color`、必要的 `payloadIndex`
- `PointsMaterial.sizeAttenuation = false`,保持屏幕尺寸稳定。
- `depthTest = true``depthWrite = false`,避免遮挡关系破坏地表。
### 交互态
hover / locked 使用少量 overlay
- overlay 复用 `THREE.Points` 单点对象或小型 ring mesh。
- overlay texture 从统一 cache 获取。
- overlay 更新只写当前 hover / locked 的 position、texture、opacity、size。
### 高密度升级
当某类图标超过分桶 Points 的舒适区,才考虑升级:
- `InstancedBufferGeometry` billboard。
- 自定义 shader 支持 per-instance rotation / scale / opacity。
- 视口 bbox / LOD / cluster。
这个升级不应该改变业务接口,只替换底层 renderer。
## Picking 策略
沿用船只当前方向:
- 默认屏幕空间 picking而不是 Three.js 对每个 Sprite / Points 做 raycast。
- 每个 icon 保留世界坐标和业务 payload。
- 每次 pointer move 将候选点投影到屏幕,按半径和深度判断命中。
- 拖动、惯性旋转、相机剧烈变化期间跳过 hover picking。
- click 时允许做一次更精确的 picking。
后续可以按图层或经纬度网格增加空间索引,减少候选点数量。
## 与现有图层的迁移路径
### Phase 1抽出船只基准能力
-`vessels.js` 提取 texture cache、canvas icon draw、overlay glow、分桶 Points 创建、状态增量更新。
- 保持 `vessels.js` 的公开 API 不变:`loadVessels()``toggleVessels()``getVesselMarkers()` 等继续可用。
- 新模块先只服务船只,确保视觉没有回退。
### Phase 2迁移 BGP 事件和观测站
- BGP 事件使用 `canvas-shape`,已接入 `Interactable`
- 严重级别映射到 `colors.byKind`,并通过通用 `getPointSizeMultiplier` 保留严重级别尺寸倍率。
- 当前扩圈效果保留在 BGP 业务动画中,并跟随 `Interactable` marker 位置更新。
- BGP 观测站主图标已接入 `Interactable`,活跃度映射到颜色和 `getPointSizeMultiplier`
- BGP 观测站 halo / 覆盖扇形继续由 BGP 业务动画表达扫描,并跟随 `Interactable` marker 位置更新。
### Phase 3迁移算力中心并评估登陆点
- 算力中心保留现有业务 icon但接入统一 hover / locked / glow。已完成
- 登陆点曾接入同一套 `Points` 渲染,但 pin 类 SVG 在地球边缘会被深度测试裁切;当前保留专用 `THREE.Sprite`,并使用 canvas 生成黄色扁平球,贴到海缆层级。
- TODO登陆点暂不迁移到完整 Interactable。后续若要统一交互接口优先考虑 Sprite-backed adapter只对齐 `getMarkers()``getPointerIntersections()``setMarkerState()``updateVisualState()` 等外观协议,不强行复用 `THREE.Points`、atlas 和跨图层避让。
- 检查图例、搜索和 info-card 是否只依赖业务 payload而不是依赖渲染对象类型。
### Phase 4形成 Earth 图标层规范
-`docs/technical/zh/earth-frontend-context.md` 记录当前实现入口。
-`docs/technical/zh/earth-layer-style-reference.md` 记录默认 glow、状态颜色、默认高度和动画参数。
-`docs/technical/zh/earth-render-layer-order.md` 记录 surface icon renderOrder 范围。
## 风险与约束
- 过早抽象可能让船只这种高质量基准被平均化,因此第一阶段必须以船只视觉不回退为验收标准。
- 图片 icon 不可上色,接口需要明确 `colorable = false` 的行为,避免业务层误以为颜色一定生效。
- 动画如果默认开启过多,会重新引入 overdraw 和每帧更新压力;默认只给 hover / locked 或少量 alert 使用。
- 地形开启时,贴地 icon 需要在高度、`depthTest``polygonOffset` 和 renderOrder 之间保持平衡。
- 统一 glow 不等于所有图标一模一样;业务可以调强度和颜色,但不应破坏整体视觉语言。
## 验收标准
1. 船只迁入通用接口后普通态、hover、locked、航向、颜色、轨迹和 picking 行为保持一致。
2. 新增一个 BGP 事件示例图层配置,不需要复制船只渲染代码即可得到 icon、glow、hover 和扩圈动画。
3. 新增一个 BGP 观测站示例图层配置,不需要自写独立动画循环即可得到雷达扇形。
4. 关闭图层后对应 icon、overlay、动画和 picking 全部停止。
5. 高密度数据下普通态仍走批量绘制hover / locked 只更新少量 overlay。
6. 文档同步说明默认高度、默认 glow、状态模型和动画扩展点。
## 相关文件
| 文件 | 当前角色 | 未来关系 |
| --- | --- | --- |
| `frontend/public/earth/js/vessels.js` | 船只基准实现,包含分桶 Points、hover / locked overlay、默认 glow 形态 | Phase 1 的抽象来源 |
| `frontend/public/earth/js/constants.js` | 保存船只高度、颜色、透明度、轨迹参数 | 后续可加入通用 surface icon 默认配置 |
| `frontend/public/earth/js/bgp.js` | BGP 事件和观测站视觉逻辑 | BGP 事件和观测站主图标已接入 Interactable扩圈、halo 和覆盖扇形仍保留业务动画 |
| `frontend/public/earth/js/compute-centers.js` | 算力中心 icon 和交互 | 已通过 Interactable 接入统一 Points、overlay、glow 和 picking |
| `frontend/public/earth/js/cables.js` | 登陆点 icon 和海缆线 | 登陆点当前使用专用 `THREE.Sprite` 黄色球,不再走 Interactable海缆线仍独立渲染 |
| `frontend/public/earth/js/main.js` | 当前集中处理 hover、click、locked 和 info-card 入口 | 后续需要接入通用 icon picking 结果 |
| `docs/technical/zh/earth-layer-style-reference.md` | 当前视觉参数参考 | 实现后同步默认 glow 和通用参数 |
| `docs/technical/zh/earth-render-layer-order.md` | 当前层级参考 | 实现后同步 surface icon 层级范围 |

View File

@@ -0,0 +1,366 @@
# Earth 新闻巡航摘要增强计划
## 背景
`Earth` 的新闻巡航模式目前直接消费 `/api/v1/news/earth-feed` 返回的 `items[].summary`。这个字段主要来自 RSS/Atom 的 `description``summary``content`,再经过 HTML 清理与长度截断。
这个实现足够轻量,但在巡航展示里有三个问题:
- 不是所有新闻源都会提供摘要,部分源只返回标题和链接。
- 聚合源的摘要质量不稳定,可能只是重复标题、来源署名或片段文本。
- 巡航模式需要更稳定的“态势说明”,否则新闻卡片的 `SUMMARY` 区域会显得空或信息密度不足。
目标不是把所有新闻都交给大模型,而是建立一个分层摘要管线:能用新闻源自带内容时零成本处理,需要增强时优先用本地模型,云端 LLM 只作为可控兜底。
## 当前相关实现
| 文件 | 作用 |
| --- | --- |
| `backend/app/services/earth_news.py` | 拉取 RSS/Atom 新闻源、解析标题/摘要、按区域聚合并返回 Earth 新闻 payload |
| `backend/app/api/v1/news.py` | 暴露 `/api/v1/news/earth-feed` |
| `frontend/public/earth/js/news.js` | 拉取新闻 payload 并渲染媒体面板新闻列表 |
| `frontend/public/earth/js/news-cruise-adapter.js` | 将新闻条目映射为巡航事件,并把 `summary` 传给信息卡 |
| `frontend/public/earth/js/info-card.js` | 展示新闻巡航卡片中的 `SUMMARY` |
当前摘要生成逻辑集中在 `earth_news.py`
```python
summary = _extract_item_text(node, "description", "content")
clean_summary = _truncate(_strip_html(summary), 180)
```
这意味着后端还没有区分“摘要来自哪里”“质量是否足够”“是否需要异步增强”。
## 总体方案
采用四级摘要来源:
| 优先级 | 来源 | 成本 | 适用情况 | 风险 |
| --- | --- | --- | --- | --- |
| 1 | 新闻源自带 summary / description | 低 | RSS/Atom 已提供可读摘要 | 字段可能为空或重复标题 |
| 2 | 本地规则提取 | 低 | 有正文片段但没有可靠摘要 | 只能抽取,不能真正概括 |
| 3 | 本地 Gemma/Ollama 摘要 | 中 | 巡航会展示且前两级质量不足 | 本地模型质量与机器性能相关 |
| 4 | 云端 LLM 兜底 | 高 | 用户手动增强、重点新闻、失败补偿 | 成本与网络依赖 |
推荐默认策略:
```text
provider summary -> extractive summary -> cached local model summary -> async local model summary -> optional cloud LLM
```
巡航 UI 永远先展示已有摘要,不等待模型调用。模型摘要在后台补齐,写入缓存后下一轮巡航或刷新时使用。
## 数据结构
后端应把原来的 `summary: str` 升级为可追踪的摘要元信息,同时为了兼容前端保留顶层 `summary` 字段。
建议新增结构:
```json
{
"summary": "短摘要文本",
"summary_meta": {
"source": "provider",
"quality": "good",
"generated_at": "2026-04-29T00:00:00Z",
"content_hash": "sha256:...",
"model": null,
"language": "zh-CN"
}
}
```
字段说明:
| 字段 | 可选值 | 说明 |
| --- | --- | --- |
| `source` | `provider` / `extractive` / `local_llm` / `cloud_llm` / `fallback` | 摘要来源 |
| `quality` | `good` / `partial` / `poor` | 后端对摘要可用性的判断 |
| `generated_at` | ISO 时间 | 模型或规则生成时间 |
| `content_hash` | SHA-256 | 用于缓存命中和判断内容变化 |
| `model` | 字符串或 `null` | 例如 `gemma3:4b` |
| `language` | 语言代码 | 默认 `zh-CN`,也可跟随新闻语言 |
前端第一阶段不需要显示 `summary_meta`,但可以用于后续调试面板或质量标记。
## 后端设计
### NewsSummaryService
新增 `backend/app/services/news_summary.py`,提供统一入口:
```python
async def resolve_news_summary(item: ParsedNewsItem, *, mode: str) -> NewsSummaryResult:
...
```
核心职责:
- 标准化新闻输入标题、URL、来源、发布时间、摘要片段、正文片段。
- 判断 provider summary 是否可用。
- 生成本地规则摘要。
- 查询模型摘要缓存。
- 在允许时调用本地 Ollama/Gemma。
- 在增强模式或手动触发时调用云端 LLM。
- 返回摘要文本与 `summary_meta`
### 摘要质量判断
第一版可以用轻量规则:
- 少于 30 个字符:`poor`
- 与标题高度重复:`partial`
- 包含明显来源署名或聚合噪声:`partial`
- 60-180 个字符且不重复标题:`good`
伪代码:
```python
def score_summary(title: str, summary: str) -> SummaryQuality:
if len(summary.strip()) < 30:
return "poor"
if normalized_overlap(title, summary) > 0.75:
return "partial"
if looks_like_source_attribution(summary):
return "partial"
return "good"
```
### 本地规则摘要
如果新闻源没有摘要,但有 `content``description``snippet` 或正文片段:
- 清理 HTML。
- 去掉标题重复内容。
- 去掉来源署名、发布时间、图片说明。
- 优先取前 1-2 个完整句子。
- 控制在 80-140 个中文字符或 40-80 个英文词。
### 本地 Gemma/Ollama Provider
不要在业务里写死 Gemma抽象为 `LocalLLMSummaryProvider`,默认可以指向 Ollama
```env
NEWS_SUMMARY_PROVIDER=ollama
OLLAMA_BASE_URL=http://localhost:11434
NEWS_SUMMARY_MODEL=gemma3:4b
NEWS_SUMMARY_TIMEOUT_SECONDS=20
NEWS_SUMMARY_MAX_INPUT_CHARS=5000
NEWS_SUMMARY_MAX_PER_HOUR=60
```
Ollama 请求示例:
```http
POST /api/generate
Content-Type: application/json
{
"model": "gemma3:4b",
"prompt": "...",
"stream": false,
"options": {
"temperature": 0.2,
"num_predict": 180
}
}
```
摘要 prompt 要强调“只基于原文”,避免模型补事实:
```text
你是新闻摘要器。只根据输入新闻内容生成摘要,不要添加原文没有的信息。
输出中文1-2 句话80-140 字。
如果原文信息不足,只概括已知事实,不要推测。
标题:{title}
来源:{source}
发布时间:{published_at}
正文或片段:
{content}
```
### 缓存
需要缓存模型摘要,避免重复花时间和费用。
缓存 key
```text
sha256(url + title + published_at + normalized_content)
```
建议新增表或复用系统设置缓存。若要可查询与清理,推荐独立表:
```text
news_summary_cache
- id
- cache_key
- url
- title
- content_hash
- summary
- source
- quality
- provider
- model
- generated_at
- expires_at
- failure_count
- last_error
```
缓存策略:
- 同一 `cache_key` 命中后直接返回。
- `provider` / `extractive` 可以短期缓存。
- `local_llm` / `cloud_llm` 可以长缓存,内容 hash 变化才重算。
- LLM 失败后记录 `failure_count`,短时间内不重复调用。
## 前端与巡航行为
前端第一阶段只需要继续使用 `item.summary`,不阻塞现有逻辑。
后续可选增强:
- `news.js` 在渲染新闻列表时,如果 `summary_meta.quality === "poor"`,可以用更紧凑的标题卡样式。
- `news-cruise-adapter.js` 选择巡航项时,可以优先选择 `summary_meta.quality !== "poor"` 的新闻。
- `info-card.js` 不显示“AI 生成中”这类文案,避免把系统内部状态暴露给用户。
如果后端异步生成完成,可以通过下一次 `/api/v1/news/earth-feed` 刷新自然更新。第一版不需要 WebSocket。
## 调用策略
默认使用“省钱模式”:
- 只处理本次 payload 中即将进入巡航队列的前 N 条。
- `provider``extractive` 达到 `good` 时不调用模型。
- 本地模型失败时不影响新闻 payload。
- 云端 LLM 默认关闭,只允许手动增强或后台配置开启。
推荐限制:
| 配置 | 默认值 | 说明 |
| --- | --- | --- |
| `NEWS_SUMMARY_MODE` | `economy` | `off` / `economy` / `enhanced` / `manual` |
| `NEWS_SUMMARY_CRUISE_PREFETCH_LIMIT` | `10` | 每次新闻 payload 预热多少条巡航摘要 |
| `NEWS_SUMMARY_MAX_PER_HOUR` | `60` | 本地模型每小时最多处理数量 |
| `NEWS_SUMMARY_CLOUD_MAX_PER_DAY` | `20` | 云端 LLM 每天最多处理数量 |
| `NEWS_SUMMARY_TIMEOUT_SECONDS` | `20` | 单条模型摘要超时 |
| `NEWS_SUMMARY_MAX_INPUT_CHARS` | `5000` | 输入截断上限 |
## Gemma 本地部署建议
Gemma 适合作为“本地省钱层”,但不应成为强绑定依赖。建议通过 Ollama 接入,未来可切换 Qwen、Llama 或其他本地模型。
开发环境:
```bash
ollama pull gemma3:4b
ollama serve
```
集成原则:
- 后端只依赖 Ollama HTTP API不直接依赖 Gemma SDK。
- 模型名称来自配置,不写死在代码里。
- 健康检查访问 `/api/tags` 或执行一条极短测试 prompt。
- 如果 Ollama 不可用,摘要管线自动退回 `provider` / `extractive`
中文新闻较多时,需要单独评估 Gemma 与 Qwen 系本地模型的中文摘要质量。不要只看单条效果,至少抽样 50 条新闻比较:
- 事实准确性
- 中文自然度
- 长度稳定性
- 延迟
- 是否会补充原文没有的信息
## 云端 LLM 兜底
云端 LLM 不作为默认路径,只用于:
- 用户点击“增强摘要”。
- 管理员开启增强模式。
- 本地模型连续失败且新闻进入重点巡航队列。
云端结果同样写入 `news_summary_cache`,并受每日限额控制。
## 分阶段实施
### 第一阶段:零成本摘要质量增强
-`earth_news.py` 中引入 `summary_meta`
- 增加 provider summary 质量判断。
- 增加本地规则摘要兜底。
- `/api/v1/news/earth-feed` 保持兼容,继续返回顶层 `summary`
- 前端无需大改。
验收标准:
- 没有摘要的新闻也能尽量得到短摘要。
- `summary_meta.source``summary_meta.quality` 可用于调试。
- 现有新闻面板和巡航模式不破坏。
### 第二阶段:本地 Gemma/Ollama 摘要
- 新增 `LocalLLMSummaryProvider`
- 接入 Ollama `/api/generate`
- 添加超时、输入截断、错误退避。
- 增加模型摘要缓存。
- 巡航 payload 后台预热前 N 条摘要。
验收标准:
- Ollama 可用时,低质量摘要能被本地模型增强。
- Ollama 不可用时,新闻接口仍然正常返回。
- 同一新闻不会重复调用模型。
### 第三阶段:设置与可观测性
- 在设置中增加新闻摘要模式:
- `关闭`
- `省钱模式`
- `增强模式`
- `仅手动`
- 增加本地模型连通性检查。
- 暴露缓存命中率、模型调用次数、失败次数。
- 日志记录摘要来源和失败原因。
验收标准:
- 用户可以不改环境变量就知道本地摘要服务是否可用。
- 管理员能看出成本和失败情况。
### 第四阶段:云端 LLM 兜底
- 接入现有 AI Provider 或新增 cloud summary provider。
- 增加每日限额与手动增强入口。
- 对云端生成结果落缓存。
验收标准:
- 云端调用可控、可关闭、可限流。
- 云端失败不影响巡航。
## 风险与防护
| 风险 | 防护 |
| --- | --- |
| 本地模型生成不存在的事实 | prompt 明确禁止扩写;摘要只作为原文概括;保留来源链接 |
| 本地模型慢导致新闻接口卡住 | 模型摘要异步化;接口先返回已有摘要 |
| 成本失控 | 默认不启用云端;按小时/天限流;缓存命中优先 |
| 摘要语言不一致 | 配置目标语言,默认 `zh-CN` |
| 新闻源正文不足 | 只概括标题和片段,不强行扩写 |
| 模型服务不可用 | 自动回退,不影响巡航主流程 |
## 推荐优先级
先做第一阶段和第二阶段的最小闭环:
1. `summary_meta` + 质量判断。
2. 本地规则摘要。
3. Ollama provider。
4. 缓存。
5. 巡航前 N 条异步预热。
云端 LLM 和设置页可以后置。这样能先验证“摘要缺失比例、本地模型质量、实际延迟”三个关键问题,再决定是否投入更重的 UI 与云端增强。

View File

@@ -96,3 +96,11 @@ GEO 轨道点数高,采样率需要按轨道类型分层。
2. 解锁后轨道立即清除
3. 不同轨道类型下点数可控
4. 页面切换回来不会闪出旧轨道残留
## Satellite Footprint Follow-Up Items
从技术文档迁出的 footprint 后续项,作为卫星覆盖能力的计划 backlog
1.`iridium-next` 新建独立 footprint adapter。
2. 在 UI 上补一个只读提示,让用户知道当前卫星是否支持 footprint。
3. 如果未来拿到 GEO beam contour / operator metadata再为 GEO 开 operator-specific footprint。

View File

@@ -25,6 +25,49 @@
- 状态和渲染更新散落在多个模块
- 后续再加新图层时容易复制旧逻辑
## Current High-Frequency Risks
### 1. Visual state and business state drift apart
Earth 里最常见的 bug 不是“没渲染”,而是状态没有一起收口:
- 图层关了tooltip 还在
- 锁定对象隐藏了info card 还在
- legend 没跟图层切换
- loading 已结束,但按钮还像没开
后续架构治理需要把这类同步责任从临时 UI patch 转为统一状态流。
### 2. HUD layout fixes skip structure analysis
Earth HUD 历史上反复出现:
- 面板只剩一条缝
- markdown 被裁掉
- tabs / iframe 被 `overflow: hidden` 吃掉
这类问题应纳入布局治理计划,而不是散落在单个功能改动里临时修。
### 3. Transitional paths keep accumulating
Earth 已经经历过多轮 HUD、toolbar、media panel 重构,容易留下:
- 旧 helper
- 旧 class
- 旧 fallback 逻辑
- 已废弃变体
架构分离阶段需要把 cleanup pass 作为计划项,而不是让技术上下文承担提醒职责。
### 4. Cruise logic and business events couple too deeply
巡航相关风险是通用巡航层继续混入业务事件细节,导致 BGP、新闻、卫星、海缆各自复制一套状态机。
架构目标应保持:
- 通用巡航层管理目标、队列、focus、停留、隐藏和切换
- 业务模块只提供队列、坐标、卡片内容和高亮副作用
## Target Architecture
Earth 对每类对象都尽量拆成三层:

View File

@@ -0,0 +1,261 @@
# AIS 多源采集、冲突记录与聚合接口计划
**状态**:规划中
**创建日期**2026-04-30
**核心原则**:采集器只写原始观测;去重、合并、冲突解释放在聚合接口中完成
## 已确认决策
| 项目 | 决策 |
|-----|------|
| AISStream 接入方式 | 单独实现 WebSocket 采集器,不塞进现有 BarentsWatch HTTP collector |
| 采集器职责 | 连接上游、标准化字段、写入原始观测,不直接决定最终展示值 |
| 去重合并位置 | 放在聚合服务和聚合 API 中,而不是散落在每个 collector 的保存逻辑里 |
| 冲突处理 | 先记录冲突事实和当前选择原因,后续再开放用户规则配置 |
| 默认可信度 | 同类 AIS 数据源优先按 `delivery_mode` 评估:`realtime_stream` 优于 `batch_stream`,再优于 `polling``snapshot` |
| 过期保护 | 实时流源断流超过 freshness 窗口后,不能仅凭“实时源”身份压过更新的轮询数据 |
## 背景
当前 AIS 链路以 BarentsWatch 为主。它是 HTTP polling 模式,覆盖挪威附近海域,适合作为稳定的免费起点,但不适合承担全球实时船只数据的全部职责。后续接入 AISStream 后,会出现同一个 MMSI 被多个来源同时上报的情况:
- 位置、航速、航向可能在多个来源之间存在秒级差异。
- 船名、IMO、呼号、船型、尺寸等静态字段可能不完整甚至互相冲突。
- WebSocket 或其他实时流通常更接近实时,但也可能断流或批量延迟。
- 如果每个 collector 自己做去重合并,规则会分散、不可审计,也很难让用户后续配置“某个字段信任哪个来源”。
因此 v1 不应让采集器直接覆盖最终船只表。更稳的方式是先保留观测事实,再由聚合接口统一给出当前展示视图。
## 目标架构
```mermaid
flowchart LR
A[BarentsWatch HTTP collector] --> D[AIS raw observations]
B[AISStream WebSocket collector] --> D
C[Custom mapped vessel_ais sources] --> D
D --> E[AIS aggregation service]
E --> F[Conflict records]
E --> G[GeoJSON vessels API]
E --> H[Vessel detail API]
I[Aggregation strategy config] --> E
```
### 原始观测层
原始观测层保存每个来源看到的事实。建议模型包含:
| 字段 | 用途 |
|-----|------|
| `target_schema` | 例如 `vessel_ais` |
| `source` | 例如 `barentswatch_vessels``aisstream_vessels` |
| `entity_key` | AIS 使用 MMSI |
| `delivery_mode` | `realtime_stream``batch_stream``polling``snapshot` |
| `transport` | `websocket``sse``http``file` 等 |
| `observed_at` | 上游数据时间,优先使用 AIS 消息时间 |
| `collected_at` | 本系统接收或采集时间 |
| `normalized_payload` | 标准化后的 AIS JSON |
| `raw_payload` | 可选,保存原始或裁剪后的上游记录 |
`delivery_mode``transport` 不应混为一谈。WebSocket 是传输方式streaming 是交付模式。聚合可信度主要看 `delivery_mode``transport` 只作为辅助信息。
### 冲突记录层
聚合服务发现同一个实体、同一个字段存在多个非空不同值时,写入冲突记录。冲突记录不代表错误,只代表“有多个可用候选值”。
```json
{
"target_schema": "vessel_ais",
"entity_key": "257123000",
"field": "name",
"candidates": {
"barentswatch_vessels": "OSLO TRADER",
"aisstream_vessels": "OSLO TRADER II"
},
"selected_source": "aisstream_vessels",
"selected_value": "OSLO TRADER II",
"selected_reason": "delivery_mode_priority",
"resolved_by": "system",
"status": "open"
}
```
第一阶段只需要记录冲突和当前选择原因,不需要做人工逐条确认。后续 UI 的目标也不是让用户处理每条冲突,而是把冲突沉淀成字段级规则。
## 聚合规则
### 字段分类
| 类型 | 字段 | 默认策略 |
|-----|------|----------|
| 动态位置 | `lat``lon``sog``cog``heading``nav_status` | 优先最新 `observed_at`,同时间再按来源优先级 |
| 静态身份 | `name``callsign``imo``flag` | 非空优先,再按字段策略或来源优先级 |
| 静态规格 | `vessel_type``vessel_type_name``length``width``draught` | 非空优先;冲突时记录候选值 |
| 元信息 | `field_sources``conflict_count``selected_reasons` | 聚合接口生成,便于调试和后续 UI 展示 |
### 默认优先级
默认优先级应使用两个维度:
```yaml
delivery_mode_priority:
- realtime_stream
- batch_stream
- polling
- snapshot
transport_priority:
- websocket
- sse
- http
- file
```
`delivery_mode_priority` 是主判断。比如 AISStream 如果提供实时推送,应标记为 `realtime_stream + websocket`BarentsWatch 当前是 `polling + http`
### 断流保护
实时流不能永久凭身份占优。聚合时需要 freshness 窗口:
```yaml
freshness:
realtime_stream_seconds: 900
polling_seconds: 3600
```
如果 `aisstream_vessels` 最近 15 分钟没有该 MMSI 的新观测,而 BarentsWatch 轮询源有更新位置,则位置类字段应采用 BarentsWatch 的更新观测,并记录选择原因 `newest_observation``freshness_fallback`
## 聚合接口
现有展示接口应逐步改为消费聚合服务,而不是自己直接拼 `VesselPosition + VesselStatic`
```text
GET /api/v1/visualization/geo/vessels
GET /api/v1/visualization/vessels/{mmsi}
GET /api/v1/visualization/vessels/{mmsi}/track
GET /api/v1/visualization/vessels/{mmsi}/conflicts
```
GeoJSON properties 建议增加:
```json
{
"mmsi": 257123000,
"name": "OSLO TRADER",
"lat": 59.91,
"lon": 10.73,
"received_at": "2026-04-30T10:00:00Z",
"field_sources": {
"name": "aisstream_vessels",
"lat": "aisstream_vessels",
"lon": "aisstream_vessels",
"vessel_type": "barentswatch_vessels"
},
"selected_reasons": {
"name": "delivery_mode_priority",
"lat": "newest_observation",
"vessel_type": "non_empty_priority"
},
"conflict_count": 2
}
```
## 开放配置计划
### Phase 1 — 内置默认策略和只读解释
- 实现后端默认策略。
- 聚合接口返回 `field_sources``selected_reasons``conflict_count`
- 冲突记录可查询,但不允许用户修改。
- 保持现有前端船只图层接口形状基本兼容,新增字段只作为调试和后续 UI 输入。
### Phase 2 — 系统设置中的 JSON/YAML 策略配置
新增系统设置项,例如:
```yaml
collector_aggregation:
vessel_ais:
source_priority:
- aisstream_vessels
- barentswatch_vessels
field_rules:
name:
mode: source_priority
vessel_type:
mode: source_priority
source_priority:
- barentswatch_vessels
- aisstream_vessels
lat:
mode: newest
lon:
mode: newest
```
配置校验要求:
- 未知 source 只警告,不阻断保存,便于先配置后启用。
- 未知 field 必须拒绝,避免拼写错误悄悄失效。
- 动态位置字段默认不允许被固定来源永久锁死,除非显式开启高级选项。
- 空值不覆盖非空值是全局保护,不建议开放关闭。
### Phase 3 — 冲突治理 UI
基于冲突记录提供页面或 drawer
- 查看某个 MMSI 的冲突字段。
- 查看每个字段的候选来源和值。
- 查看当前选择原因。
- 将一次人工选择保存成字段规则,而不是只处理单条冲突。
- 支持恢复默认策略。
## AISStream 采集器计划
AISStream 采集器单独实现,建议命名为 `aisstream_vessels`。它的职责是:
- 维护 WebSocket 连接、订阅范围和重连。
- 将上游 AIS 消息标准化为 `vessel_ais` payload。
- 标记 `delivery_mode = realtime_stream``transport = websocket`
- 写入原始观测层。
- 不直接 upsert 最终展示数据。
配置应放入采集器设置,而不是硬编码:
```yaml
aisstream_vessels:
api_key: "${AISSTREAM_API_KEY}"
bounding_boxes:
- [[-180, -90], [180, 90]]
message_types:
- PositionReport
- ShipStaticData
```
## 实施顺序
1. 新增原始观测模型和冲突记录模型。
2. 实现 AIS 聚合服务,先从现有 `vessel_position` / `vessel_static` 兼容读取,再逐步切换到原始观测层。
3.`/geo/vessels``/vessels/{mmsi}` 改为走聚合服务。
4. 改造 BarentsWatch 保存逻辑,让它写入原始观测,同时保留现有表作为兼容缓存。
5. 实现 AISStream WebSocket collector。
6. 接入系统设置中的聚合策略配置。
7. 做冲突治理 UI。
## 测试计划
- 同一来源同一 `mmsi + observed_at + lat + lon` 重复记录只聚合一次。
- 多来源同一 MMSI 的位置字段优先选择最新观测。
- 实时流和轮询源同时间冲突时,实时流优先。
- 实时流过期后,更新的轮询源可以接管动态字段。
- 静态字段不会被空值覆盖。
- 静态字段冲突会写入冲突记录。
- 字段级配置可以覆盖默认来源优先级。
- 聚合接口在没有冲突表时仍可返回兼容 GeoJSON。
## 相关文件
- [实时船只监控系统计划](/home/ray/dev/linkong/planet/docs/plans/earth-vessel-tracking-plan.md)
- [自定义 API 数据源与 LLM 映射系统计划](/home/ray/dev/linkong/planet/docs/plans/datasource-custom-api-mapping-plan.md)
- [BarentsWatch AIS collector](/home/ray/dev/linkong/planet/backend/app/services/collectors/vessel_ais.py)
- [船只模型](/home/ray/dev/linkong/planet/backend/app/models/vessel.py)
- [可视化 API](/home/ray/dev/linkong/planet/backend/app/api/v1/visualization.py)

View File

@@ -0,0 +1,249 @@
# Earth Vessel Rendering Performance Plan
## 当前状态
该计划的前端核心部分已经在 `0.44.1` 落地,但最终实现不是原文设想的 `InstancedBufferGeometry` quad而是更稳的分桶 `THREE.Points` 方案:
- 普通船只按 moving / anchored 和 `VESSEL_COURSE_BINS` 航向分桶,使用 `PointsMaterial` 批量绘制。
- 航行船只仍是带方向的三角形,停泊或低速船只仍是圆点。
- hover / locked 不再放大成世界尺寸 Sprite而是在原点位叠加同尺寸单点 glow overlay。
- picking 改为屏幕空间命中,拖动和惯性期间跳过 hover picking。
- 普通态关闭 glow交互态才显示 glow降低 overdraw 并让默认地图更干净。
后续如果需要全球 AIS 或更高船只密度,再评估是否从分桶 `Points` 升级到真正 instanced quad 或视口 bbox / LOD。
## 背景
Earth 船只图层已经形成了一套较好的视觉语言:
- 航行船只使用三角形标记
- 标记按航向旋转
- 停泊或低速船只使用圆点
- 不同船型使用不同颜色
- hover / locked 状态有 glow、透明度和聚焦反馈
- 标记带有轻微 glow / soft edge和 Earth HUD 的观感一致
当前性能问题不应通过降级成无方向、无船型语义的普通小点来解决。目标是在保留现有观赏性的前提下,把底层从“每艘船一个 Sprite 对象”优化为批量绘制和轻量交互。
## 当前问题判断
卫星图层能承载几万个对象,是因为它主要走 `THREE.Points` / `BufferGeometry` / instanced trail 路径。船只图层目前每艘船创建一个 `THREE.Sprite` 和独立 `SpriteMaterial`,这会带来:
- draw call 随船只数量增长
- 透明 sprite 排序和 overdraw 成本上升
- 每帧遍历所有船只更新 opacity / scale / visible
- pointer move 时对船只 sprite 做对象级 raycast
- hover reset 时全量遍历 marker
因此,即使免费 BarentsWatch AIS 只开放挪威周边数据,前端仍可能因为对象级 sprite、raycast 和每帧全量更新出现地球拖动卡顿。
## 目标
1. 保留当前船只标记的视觉质量。
2. 保留 hover tooltip、点击详情、lock、轨迹等交互。
3. 显著降低 draw call、每帧 JS 遍历和 pointer picking 成本。
4. 为后续全球 AIS 或更高船只数量预留扩展空间。
## 非目标
- 不把船只降级为普通无方向 `Points`
- 不取消船型颜色、航向三角和停泊圆点。
- 不为了短期性能直接删除 hover / click 交互。
## Phase 1交互路径止血
这一阶段不改视觉,只减少 pointer move 和 hover 状态开销。
### 1. 拖动和惯性期间跳过船只 picking
地球拖动时用户主要关注视角变化,不需要每个 pointer move 都命中船只。
处理方式:
- `isDragging === true` 时跳过船只 hover picking。
- 惯性旋转期间也跳过船只 hover picking。
- 拖动结束后再恢复 hover 检测。
### 2. vessel hover picking 节流
对船只 hover 命中增加节流,例如 `80ms ~ 120ms` 一次。鼠标高速移动时复用上一次 hover 状态,不在每个 pointer event 上都做 raycast。
### 3. hover reset 从全量遍历改为增量更新
当前 `resetTransientVesselStates()` 会遍历所有船只。改为记录:
- `hoveredVessel`
- `lockedObject`
当 hover 目标变化时,只更新旧 hover 和新 hover。
### 4. 点击路径只在 click 时做一次精确 picking
点击仍保留精确命中,但只在 click 事件里执行,不参与拖动和高频 pointer move。
## Phase 2每帧更新减负
这一阶段仍保留 `Sprite` 外观,但减少每帧对全部 marker 的写操作。
### 1. `updateVesselVisualState()` 增量化
当前每帧都会遍历船只并写:
- `marker.material.opacity`
- `marker.scale`
- `marker.visible`
优化方向:
- 图层关闭时直接 return。
- 没有船只时直接 return。
- 只有以下状态变化时才更新 marker
- show/hide 变化
- hover 变化
- locked 变化
- camera zoom / distance scale 变化超过阈值
- focus dim 状态变化
### 2. 缓存 distance scale
`getDistanceScale(camera)` 可以按 camera distance 或 zoom 阈值缓存。缩放没有明显变化时,不必每帧重设所有船只 scale。
### 3. 降低透明 overdraw
在不破坏视觉的前提下微调:
- marker 基础尺寸
- glow blur 半径
- 最大 size stabilization
目标是减少屏幕空间重叠面积,而不是改变符号设计。
## Phase 3保留视觉的批量渲染已落地为分桶 Points
原设想是把每艘船的视觉从 `THREE.Sprite` 迁移为 instanced sprite batch。实际落地时选择了更稳的分桶 `THREE.Points`
- 不依赖自定义 shader。
- 不依赖 `Points` 自带 raycaster。
- 用 canvas 纹理保留三角、圆点、船型颜色和航向。
- 用 hover / locked 单点 overlay 保留交互 glow。
如果未来全球 AIS 导致分桶 `Points` 仍不够,再升级到 instanced quad。
### 1. 原候选方案instanced quad
每艘船仍然显示为带贴图/软边的 billboard但底层使用
- `THREE.InstancedBufferGeometry`
- 每类船只一个或少量 material
- per-instance attributes
可按形状和船型拆 batch
- moving cargo
- moving tanker
- moving passenger
- moving fishing
- moving military
- moving other
- anchored / slow dot
这样 draw call 从“每艘船一个”变为“每类船只一个”。
### 2. 原候选方案per-instance attributes
每个 instance 存:
- position
- color
- rotation
- scale
- opacity
- state
- mmsi / data index
hover、locked、dimmed 可通过更新少量 instance attribute 实现,不再逐个修改 material。
### 3. 当前落地方案:分桶 `THREE.Points`
当前实现按以下方式复刻视觉:
- moving 船只按 `VESSEL_COURSE_BINS` 做航向分桶。
- anchored / slow 船只使用圆点分桶。
- 每个分桶生成一组 `THREE.Points`,共享 `PointsMaterial` 和 canvas 点纹理。
- `VESSEL_CONFIG.colors` 仍通过 vertex colors 表示船型颜色。
- hover / locked 在原位置叠加同尺寸单点 overlay普通态不带 glow交互态才带 glow。
这样 draw call 从“每艘船一个”变为“每个形状 / 航向分桶一组”,同时避免自定义 shader 的兼容风险。
### 4. 复刻当前视觉
视觉上继续使用当前 canvas texture 或等效 shader
- moving 使用三角形纹理
- anchored 使用圆点纹理
- 保留 soft glow
- 保留航向 rotation
- 保留 hover / locked 放大
因此用户看到的效果应与当前船只图层基本一致。
## Phase 4picking 改造
批量渲染后不再适合对所有 sprite object 做 `raycaster.intersectObjects()`
### 1. 屏幕空间 picking
参考卫星 picking
1. 过滤背面船只。
2. 将候选船只世界坐标投影到屏幕。
3. 用鼠标位置计算距离。
4. 取距离最近且小于半径阈值的船只。
### 2. 可选空间索引
如果后续船只数量明显上升,可增加轻量空间索引:
- 经纬度网格 bucket
- 屏幕空间 bucket
- viewport bbox 过滤
第一阶段不必引入复杂索引。
## Phase 5数据层和 LOD
当接入全球 AIS 或船只数量显著增加时,再做数据层优化。
### 1. 请求视口范围
前端请求 `/api/v1/visualization/geo/vessels` 时带上当前视口 `bbox`,减少无关船只。
### 2. 后端排序策略
从单纯 `received_at desc` 改为综合排序:
- 数据新鲜度
- 船型优先级
- 当前视口相关性
- 是否正在航行
### 3. 远景聚合
远景可显示聚合或 top N近景展开单船。
## 验收指标
1. 船只视觉效果保持当前质量三角、圆点、颜色、航向、hover、lock 都保留。
2. 开启船只图层后拖动地球不应明显掉帧。
3. pointer move 不应因为船只 hover 导致卡顿。
4. 船只数量达到 `1000` 级别时仍可顺畅旋转地球。
5. `renderer.info.render.calls` 相比 Sprite 版本显著下降。
6. hover / click 命中体验不低于当前版本。
## 建议落地顺序
1. 先做 Phase 1快速恢复地球拖动手感。
2. 再做 Phase 2减少每帧 JS 写操作。
3. Phase 3 和 Phase 4 已按分桶 `THREE.Points` + 屏幕空间 picking 落地。
4. Phase 5 等全球船只数据或数量压力出现后再推进。
5. 如果分桶 `THREE.Points` 达到瓶颈,再评估 instanced quad。

View File

@@ -0,0 +1,279 @@
# 实时船只监控系统 — 实施计划
**状态**:规划中
**创建日期**2026-04-27
**优先数据源**BarentsWatch AIS免费但需要 OAuth client credentials→ AISHub / MarineTrafficTODO付费
## 已确认决策
| 项目 | 决策 |
|-----|------|
| 数据源 | BarentsWatch 先行AISHub / MarineTraffic TODO |
| 船只规模 | BarentsWatch 阶段全部显示;全球数据接入后按需加船型过滤(默认 Cargo + Tanker + Passenger |
| 更新频率 | 准实时:前端 5 分钟轮询,后端 Collector 每分钟拉取写库 |
| 历史轨迹 | 保留(`vessel_position` 表保留 24h后期按需扩展 |
| 推送方式 | 前端展示仍可先用 HTTP 拉取聚合结果AISStream 等实时源应单独实现 WebSocket 采集器 |
---
## 一、技术背景
船只通过 AIS自动识别系统每 210 秒广播位置、航速、航向、目的地等信息。全球约 50 万艘持证船只在线,实时数据通过以下方式获取:
| 来源类型 | 典型服务 | 覆盖范围 | 成本 | 状态 |
|---------|---------|---------|------|------|
| **BarentsWatch AIS API** | live.ais.barentswatch.no | 挪威海域实时 | 免费,需要 AIS API client credentials | **当前使用** |
| **AISHub** | aishub.net | 全球实时 | 免费/小额 | TODO付费接入 |
| **MarineTraffic API** | marinetraffic.com | 全球实时 | $50$500/月 | TODO评估 tier |
| **VesselFinder API** | vesselfinder.com | 全球实时 | $50$300/月 | TODO备选 |
| **自建 SDR 接收** | RTL-SDR + AIS-catcher | 仅本地 3050km | 硬件 $30 | 不考虑 |
| **NOAA 历史数据** | Marine Cadastre | 美国近海历史 | 免费 | 可用于冷启动 |
### BarentsWatch AIS API
- 端点:`https://live.ais.barentswatch.no/v1/latest/combined`
- 需要在 BarentsWatch developer portal 创建 `AIS - API` client通过 client credentials 获取 `scope=ais` 的 access token 后请求 AIS endpoint
- 字段mmsi, lat, lon, sog, cog, heading, nav_status, name, vessel_type, flag
- 刷新频率:数据约 3060s 更新一次,可随意轮询
### TODO多源 AIS 与实时流接入
- [ ] 接入 AISStream WebSocket 采集器,作为 BarentsWatch 覆盖不足的实时补充
- [ ] 将 BarentsWatch、AISStream、自定义 `vessel_ais` 映射源统一写入原始观测层
- [ ] 通过聚合接口做去重、字段合并、冲突记录和默认来源选择
- [ ] 开放字段级聚合策略配置,让用户决定不同字段优先信任哪个来源
- [ ] 评估 AISHub 订阅(全球覆盖,约 $30/月),接入全球实时流
- [ ] 评估 MarineTraffic API tier对比 AISHub 数据质量与成本
- [ ] 实现多数据源适配器,通过 `datasource_config` 切换
- [ ] 真实高频 AIS 稳定接入后,评估将 `vessel_position` 迁移为 TimescaleDB hypertable保留 Postgres 原生分区作为备选)
多源 AIS 的详细设计见 [AIS 多源采集、冲突记录与聚合接口计划](/home/ray/dev/linkong/planet/docs/plans/earth-vessel-ais-aggregation-plan.md)。
---
## 二、实施计划
### Phase 0 — 数据源验证与链路打通12 天)
- 接入 BarentsWatch AIS API验证 OAuth token、数据格式与字段
- 构建全球 mock 数据生成器(用于前端渲染压测,补充 BarentsWatch 的地域限制)
- 确认前端可渲染船只点,整条链路走通
### Phase 1 — 后端基础设施34 天)
#### 1.1 数据库 Schema
```sql
-- 船只静态信息(每 6h 刷新一次)
CREATE TABLE vessel_static (
mmsi BIGINT PRIMARY KEY,
name VARCHAR(128),
callsign VARCHAR(16),
vessel_type SMALLINT,
vessel_type_name VARCHAR(64),
flag VARCHAR(4), -- ISO 国家码
length FLOAT,
width FLOAT,
draught FLOAT,
imo BIGINT,
updated_at TIMESTAMPTZ
);
-- 船只实时位置(高频写入,保留 24h 轨迹)
CREATE TABLE vessel_position (
id BIGSERIAL PRIMARY KEY,
mmsi BIGINT NOT NULL,
lat FLOAT NOT NULL,
lon FLOAT NOT NULL,
sog FLOAT, -- Speed over ground
cog FLOAT, -- Course over ground
heading SMALLINT, -- 真北航向
nav_status SMALLINT, -- 0=航行 1=锚泊 5=停靠 ...
received_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX idx_vessel_pos_mmsi_time ON vessel_position(mmsi, received_at DESC);
CREATE INDEX idx_vessel_pos_time ON vessel_position(received_at DESC);
-- 最新位置物化视图(地图渲染主数据源,避免全表扫描)
CREATE MATERIALIZED VIEW vessel_latest AS
SELECT DISTINCT ON (mmsi)
vp.*, vs.name, vs.vessel_type_name, vs.flag, vs.length
FROM vessel_position vp
LEFT JOIN vessel_static vs USING (mmsi)
ORDER BY mmsi, received_at DESC;
CREATE UNIQUE INDEX ON vessel_latest(mmsi);
```
> 后期如需完整历史轨迹查询,迁移 `vessel_position` 到 TimescaleDB 或按天分区。
#### 1.2 CollectorVesselAISCollector
文件:`backend/app/services/collectors/vessel_ais.py`
- 继承 `BaseCollector`,注册到 `collector_registry`
- 轮询间隔3060s由数据源限速决定
- 支持多数据源切换,通过 `datasource_config` 配置 URL + API Key
- 写入逻辑upsert `vessel_latest`append `vessel_position`
- 接入现有调度系统(`scheduler.py`
#### 1.3 API 端点
```
GET /api/v1/visualization/geo/vessels
?bbox=lon_min,lat_min,lon_max,lat_max # 视口裁剪
?type=cargo,tanker,passenger # 船型过滤
?limit=5000
→ GeoJSON FeatureCollectionPoint
GET /api/v1/visualization/vessels/{mmsi} # 单船详情
GET /api/v1/visualization/vessels/{mmsi}/track # 历史轨迹(默认 6h
?hours=6
→ GeoJSON LineString
```
GeoJSON Feature 格式:
```json
{
"type": "Feature",
"geometry": { "type": "Point", "coordinates": [lon, lat] },
"properties": {
"mmsi": 123456789,
"name": "EVER GIVEN",
"vessel_type": 70,
"vessel_type_name": "Cargo",
"flag": "PA",
"sog": 12.4,
"cog": 247.0,
"heading": 245,
"nav_status": 0,
"length": 400,
"received_at": "2026-04-27T10:00:00Z"
}
}
```
#### 1.4 更新机制
**前端聚合结果拉取 + 后端实时采集**
- 前端 `setInterval(fetchVessels, 5 * 60 * 1000)` 定期拉取最新快照
- 后端 BarentsWatch collector 继续以 HTTP polling 方式采集
- AISStream 等实时源以独立 WebSocket collector 写入原始观测层
- 展示接口从聚合服务读取当前船只视图,而不是由单个 collector 决定最终展示值
- 前端是否升级为 WebSocket delta push 是独立优化,不影响后端采集器可以使用 WebSocket 接上游实时源
---
### Phase 2 — 前端渲染34 天)
文件:`frontend/public/earth/js/vessels.js`
#### 2.1 渲染方案
参考现有卫星系统(`satellites.js`)的 InstancedMesh 模式:
- `THREE.InstancedMesh`:每个实例 = 一艘船,矩阵包含位置 + 旋转(朝向 COG
- 行进船:三角箭头图标,朝向 COG 方向
- 静止/锚泊船:圆点图标
- SVG 图标输出到 `frontend/public/earth/assets/icons/vessel-arrow.svg``vessel-dot.svg`
#### 2.2 船型颜色规范
| 船型 | 颜色 |
|-----|------|
| 货轮 Cargo | `#4A90D9` 蓝 |
| 油轮 Tanker | `#E85D04` 橙红 |
| 客船 Passenger | `#06D6A0` 绿 |
| 渔船 Fishing | `#FFD166` 黄 |
| 军舰 Military | `#73797E` 灰 |
| 其他 | `#9B9B9B` 浅灰 |
| 锚泊/停靠 | 降低饱和度 0.4x |
#### 2.3 LOD相机距离细节层次
| 相机距离 | 渲染策略 |
|---------|---------|
| > 400 | 仅渲染 top 1000 艘(按数据新鲜度 + 船型优先级) |
| 200400 | 渲染 top 5000 艘 |
| < 200 | 渲染当前视口 bbox 内全部船只 |
前端根据相机位置动态计算 bbox附加到 API 请求中。
#### 2.4 图层集成
接入现有图层系统,新增"船只"图层项,支持:
- 图层开/关,状态持久化
- 子过滤(按船型选择显示哪类,可在图例或设置面板中配置)
- 与海缆、BGP、卫星层级共存renderOrder 待定,参考现有层级文档)
#### 2.5 Info Card
复用 `showInfoCard` 机制,点击船只弹出:
```
EVER GIVEN 🚢
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
MMSI 123456789
IMO 9811000
旗帜 巴拿马 🇵🇦
船型 散货轮
当前航速 12.4 kn
航向 247°
状态 航行中
目的地 ROTTERDAM
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ 查看轨迹 ] [ MarineTraffic ↗ ]
```
#### 2.6 轨迹可视化
点击"查看轨迹" → 请求 `/vessels/{mmsi}/track` → 用 `THREE.CatmullRomCurve3` 渲染插值轨迹线,风格与海缆一致。
---
### Phase 3 — 功能完善23 天)
| 功能 | 说明 |
|-----|------|
| **船只搜索** | 接入现有搜索面板,按名称 / MMSI 搜索 |
| **统计 HUD** | 显示当前在线船只数、各类型分布 |
| **密度热图** | 超低 zoom 时切换为 hex-bin 热力图(避免点云爆炸) |
| **港口标注** | 加载 WorldPorts 数据集,显示主要港口标记 |
| **关键水道监控** | 马六甲、霍尔木兹、苏伊士等高亮 + 流量统计 |
---
### Phase 4 — 性能与生产化23 天)
- `vessel_position` 按天分区7 天自动清理
- TODO真实数据量达到百万级/日后,将 `vessel_position` 升级为 TimescaleDB hypertable配置 retention policy 与压缩策略
- GeoJSON endpoint 用 Redis 缓存 15s
- 若需 bbox 精确查询,引入 PostGIS `geography` + `ST_DWithin`
- InstancedMesh + frustum culling目标 5 万船只 60fps
---
## 三、工作量估算
| Phase | 内容 | 估计时间 |
|-------|-----|---------|
| Phase 0 | 数据源验证、mock | 12 天 |
| Phase 1 | 后端 Schema + Collector + API | 34 天 |
| Phase 2 | 前端渲染InstancedMesh + 图层 + Info Card | 34 天 |
| Phase 3 | 搜索 + 统计 + 轨迹 | 23 天 |
| Phase 4 | 性能优化 + 生产数据源接入 | 23 天 |
| **合计** | | **约 23 周** |
---
## 四、参考资料
- BarentsWatch AIS API 文档https://www.barentswatch.no/en/developer/ais-api/
- MarineTraffic APIhttps://www.marinetraffic.com/en/ais-api-services
- AISHubhttps://www.aishub.net/api
- AIS 导航状态码ITU-R M.1371-5
- 船型编码vessel_typeITU/IMO AIS Message 5 Type and Cargo
- WorldPorts 数据集https://msi.nga.mil/Publications/WPI

View File

@@ -0,0 +1,97 @@
# Markdown 渲染器完善计划
## 背景
Planet 控制台当前有三类主要 Markdown 使用场景:
- 文档中心:技术文档、计划文档、运行手册。
- AI Playground模型回复、分析结果、代码片段。
- BGP 简报:由系统生成并保存的态势报告。
这些场景都复用 `frontend/src/components/MarkdownRenderer/MarkdownRenderer.tsx`。因此 Markdown 能力应该集中在共享渲染器内完成,页面只负责传入内容、链接转换和布局约束,不能让每篇文档或每个页面手写复制按钮、表格样式、列表样式等交互细节。
## 目标
建设一个稳定、可复用、适合技术文档和 AI 输出的 Markdown 渲染器,优先覆盖常用语法、代码块操作和清晰的阅读样式,并为后续语法高亮、锚点导航、内容安全策略留出接口。
## 成功标准
- 代码块支持 fenced language、语言标签、复制按钮、复制成功状态和横向滚动。
- 常用块语法稳定渲染:标题 1-6、段落、引用、分割线、表格、无序列表、有序列表、任务列表。
- 常用行内语法稳定渲染:链接、自动链接、图片、行内代码、粗体、斜体、删除线。
- 文档中心、AI Playground、BGP 简报继续复用同一个组件,不出现页面级重复实现。
- 样式在普通业务面板和文档中心都有合理表现,文档中心可以通过 `.docs-markdown` 覆盖主题变量。
- 前端 TypeScript build 通过,`git diff --check` 无空白错误。
## 当前实施范围
### 第一阶段:共享渲染器补齐
-`MarkdownRenderer` 内解析 fenced code block 的语言信息。
- 引入 `MarkdownCodeBlock` 子组件,负责语言标签、复制按钮和复制状态。
- 保留现有 `Scrollbar` 横向滚动能力,避免长代码撑破页面。
- 扩展标题渲染到 h1-h6并保留 `getHeadingId` 对文档目录的支持。
- 扩展列表解析,支持 `-``*``+``1.``1)` 和 GitHub 风格任务列表。
- 扩展行内解析,支持图片、自动链接、删除线。
### 第二阶段:样式统一
- 全局 Markdown 样式覆盖业务场景,保持紧凑、清晰、可扫描。
- 文档中心用 `.docs-markdown` 适配主题变量,避免硬编码颜色破坏明暗主题。
- 代码块 toolbar 和 copy button 不依赖具体页面。
- 图片默认响应式展示,避免超出内容区域。
### 第三阶段:验证
- 使用前端 build 验证 TypeScript 和 Vite 构建。
- 使用 `git diff --check` 验证补丁格式。
- 手动检查至少一个文档页中代码块复制按钮、语言标签和表格滚动是否出现。
## 后续增强项
### 语法高亮
当前不新增高亮依赖,避免一次性引入过重运行时代码。后续可以在以下方案中二选一:
- `shiki`:适合文档中心,视觉质量高,但包体和初始化成本更高。
- `highlight.js`:接入简单,覆盖语言广,但样式控制需要额外约束。
建议当文档代码块数量稳定增加后再引入,并做按需加载或懒加载。
### 更完整 CommonMark 支持
当前渲染器覆盖 Planet 常见内容,不追求完整 CommonMark 兼容。后续如果需要完整规范,建议切换到成熟生态:
- `react-markdown`
- `remark-gfm`
- `rehype-sanitize`
- `rehype-slug`
切换前需要评估:链接转换、目录 ID、现有样式、AI 输出安全策略和包体影响。
### 安全策略
目前渲染器不解析原始 HTML这是正确默认值。后续如需支持 HTML必须先明确
- 是否允许用户输入 Markdown。
- 是否需要 HTML 白名单。
- 是否需要 `rehype-sanitize`
- 图片和链接是否需要域名策略。
### 文档页能力
可继续补齐:
- 标题锚点悬浮复制。
- Mermaid 图表。
- 代码块折叠。
- 文档内搜索结果定位到代码块。
- 复制按钮埋点,用于判断文档片段是否真正被使用。
## 维护约束
- Markdown 语法能力优先放在共享渲染器,不在具体文档页面散落实现。
- 文档内容只表达内容,不承载 UI 行为。
- 新增 Markdown 能力必须同时考虑文档中心、AI Playground、BGP 简报三个调用方。
- 不解析原始 HTML除非同步引入明确的 sanitize 策略。
- 与主题相关的样式优先走页面容器变量覆盖,不在组件内写死文档中心颜色。

View File

@@ -17,12 +17,16 @@ What belongs here:
- Earth layer style property index
- Backend runtime control
- Collector status
- Collector settings and connectivity validation
- Earth Interactable integration
- Collection format conventions
## Entry Points
- [quickstart.md](/home/ray/dev/linkong/planet/docs/technical/en/quickstart.md): The shortest path to getting Planet running from scratch
- [manual.md](/home/ray/dev/linkong/planet/docs/technical/en/manual.md): Complete usage guide for the console, `planet.sh`, Earth, and Docs
- [Quickstart](/home/ray/dev/linkong/planet/docs/technical/en/quickstart.md): The shortest path to getting Planet running from scratch
- [Planet Manual](/home/ray/dev/linkong/planet/docs/technical/en/manual.md): Complete usage guide for the console, `planet.sh`, Earth, and Docs
- [Collector Settings and Connectivity Validation](/home/ray/dev/linkong/planet/docs/technical/en/datasource-collector-settings-connectivity.md): Data source catalog, collector settings, connectivity validation, and BarentsWatch credentials
- [Earth Interactable Usage](/home/ray/dev/linkong/planet/docs/technical/en/earth-interactable-usage.md): API, lifecycle, and integration examples for Earth surface icon Interactable
What does not belong here:
@@ -32,4 +36,4 @@ What does not belong here:
Those belong in:
- [docs/plans/README.md](/home/ray/dev/linkong/planet/docs/plans/README.md)
- [Plans Index](/home/ray/dev/linkong/planet/docs/plans/README.md)

View File

@@ -0,0 +1,325 @@
# Collector Settings and Connectivity Validation
## Background
The console now separates the "data source catalog" from "collector configuration":
- `/datasources`
- Lists all data sources, including built-in and custom sources.
- Clicking a name only opens an information drawer.
- Focuses on status, manual collection, and running collection tasks.
- `/settings?tab=collector_credentials`
- Displays as "Collector Settings".
- Owns endpoint, headers, base parameters, and credentials.
- Every collector exposes a connection button for health checks.
This reduces first-use confusion: API endpoints, headers, credentials, and custom source configuration all belong to collector settings instead of being scattered across the data source list and system settings.
## User-Facing Rules
Connection state is not a frontend styling state. The backend derives it from the current configuration checksum and previously validated records.
A built-in collector is considered "connected" when either condition is true:
- The current configuration has successfully collected data.
- The user clicked the connection button for the current configuration and backend validation succeeded.
If endpoint, headers, base configuration, or credential fingerprint changes after the last successful validation, the state returns to "needs reconnection".
## Frontend Entry Points
### Data Source Catalog
Files:
- [DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/DataSources/DataSources.tsx)
- [index.css](/home/ray/dev/linkong/planet/frontend/src/index.css)
Current behavior:
- Built-in and custom data sources are merged into a `UnifiedDataSource` list.
- The table only keeps view, collect, and status actions.
- Clicking the name opens a read-only drawer.
- The drawer shows:
- Whether the source is built in
- Whether it is enabled
- Module, priority, and frequency
- Endpoint
- Headers
- Base configuration
- Whether credentials are required
- When tasks are running, the top progress area shows a clickable `Collecting N` pill.
- Clicking `Collecting N` opens a task list modal with per-task progress.
`data-source-bulk-toolbar__running-pill` is the styling entry point for the "Collecting" pill. It is aligned with other status tags, while hover treatment, arrow affordance, and blue outline indicate interactivity.
### Collector Settings
File:
- [Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Settings/Settings.tsx)
Current behavior:
- The `collector_credentials` tab is displayed as "Collector Settings".
- A select lists all built-in collectors.
- The only button beside the select is a plug icon for health checks.
- Status tags below the select show:
- `Credentials required` / `No credentials required`
- Module
- `Enabled` / `Disabled`
- `Unchecked` / `Available` / `Unavailable`
- Whether the endpoint is overridden
- Collectors that require credentials place the credential card above base configuration.
- Collectors without credentials only show base configuration.
The connection button uses an inline Tabler-style plug icon with `plug-connected` semantics, avoiding the older refresh icon for a connection action.
## Backend APIs
### Data Source Configuration List
```http
GET /api/v1/datasources/configs/all
```
Returns a merged view of YAML default data sources and database overrides. This route must be declared before `/configs/{config_id}`; otherwise FastAPI treats `all` as a path parameter and returns 422.
Returned fields include:
- `name`
- `default_url`
- `endpoint`
- `is_overridden`
- `is_active`
- `source_type`
- `auth_type`
- `headers`
- `config`
- `config_id`
- `description`
Before returning `config`, internal connectivity validation fields are removed so the frontend does not display validation metadata as user configuration.
### Built-In Collector Connection Status
```http
POST /api/v1/datasources/configs/builtin/connection-status
```
Purpose:
- Accept a candidate configuration.
- Compute its checksum.
- Determine whether the current configuration is already connected.
The current frontend mostly performs an immediate check through the connection button and does not strongly depend on this endpoint. It remains the backend basis for future save-button disabling and restoring initial page state.
### Built-In Collector Connectivity Validation
```http
POST /api/v1/datasources/configs/builtin/connect
```
Purpose:
- Free collectors request the endpoint directly.
- Credentialed collectors go through their credential provider.
- Successful validation writes a system-level connection record.
Successful responses include:
- `success`
- `connected`
- `checksum`
- `stage`
- `message`
- `response_time_ms`
- `credential_provider`
- `credential_source`
### BarentsWatch AIS Connectivity Validation
```http
POST /api/v1/settings/integrations/barentswatch/connect
GET /api/v1/settings/integrations/barentswatch/connectivity
```
BarentsWatch uses separate endpoints because draft credentials must be validated before saving:
- Use draft `client_id` / `client_secret` to fetch a token.
- Use that token to request the AIS endpoint.
- After success, write a built-in collector connection record using the draft credential fingerprint.
## Connectivity Validation Service
File:
- [datasource_connectivity.py](/home/ray/dev/linkong/planet/backend/app/services/datasource_connectivity.py)
Core responsibilities:
- Compute built-in collector configuration checksums.
- Read credentials from environment variables and `~/.zshrc`.
- Determine whether the current configuration is already connected.
- Run endpoint health checks.
- Save successful connection records.
### Checksum Inputs
The checksum includes:
- Collector name
- Endpoint
- Auth type
- Headers
- Config after removing internal validation fields
- Credential provider
- Credential fingerprint
The credential fingerprint is a hash of credential content. Plaintext credentials are not written into connection records.
### Connection Records
Successful connection records are written to `SystemSetting`:
```text
category = datasource_connectivity_validations
```
The payload uses collector source as the key:
```json
{
"barentswatch_vessels": {
"checksum": "...",
"status": "success",
"validated_at": "2026-04-29T00:00:00+00:00",
"status_code": 200,
"credential_source": "datasource_config",
"connected_by": "connection_button"
}
}
```
`connected_by` currently has two sources:
- `connection_button`: the user manually clicked the connection button.
- `collection`: a collection task completed successfully, so the system recorded the current effective configuration as connected.
### Successful Collection Means Connected
After a successful collection, the scheduler writes a connection record:
- [scheduler.py](/home/ray/dev/linkong/planet/backend/app/services/scheduler.py)
This prevents collectors that already have data from asking the user to validate again. Reconnection is only required when the configuration checksum changes.
## BarentsWatch AIS Credential Chain
Files:
- [barentswatch.py](/home/ray/dev/linkong/planet/backend/app/services/barentswatch.py)
- [vessel_ais.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/vessel_ais.py)
Resolution priority:
1. `DataSourceConfig.auth_config`
2. `DataSourceConfig.config`
3. Environment variables
4. `~/.zshrc`
Supported environment variables:
```bash
export BARENTSWATCH_CLIENT_ID="..."
export BARENTSWATCH_CLIENT_SECRET="..."
```
Historical misspellings are also supported:
```bash
export BARRENTSWATCH_CLIENT_ID="..."
export BARRENTSWATCH_CLIENT_SECRET="..."
```
Token request rules:
- Token URL: `https://id.barentswatch.no/connect/token`
- `Content-Type`: `application/x-www-form-urlencoded`
- Body:
- `grant_type=client_credentials`
- `client_id`
- `client_secret`
- `scope=ais`
AIS request rules:
- Default endpoint: `https://live.ais.barentswatch.no/v1/latest/combined`
- Header: `Authorization: Bearer <access_token>`
`VesselAISCollector` no longer reads environment variables directly. It goes through `resolve_barentswatch_config()` and `fetch_barentswatch_access_token()` so settings, connectivity validation, and collection do not fork into three credential flows.
## Credential Guide
File:
- [credential_guides.py](/home/ray/dev/linkong/planet/backend/app/services/credential_guides.py)
APIs:
```http
GET /api/v1/settings/credential-guides/{provider}
POST /api/v1/settings/credential-guides/{provider}/generate
POST /api/v1/settings/credential-guides/{provider}/reset
```
Currently supported:
- `barentswatch`
The default guide includes the official BarentsWatch tutorial:
```text
https://developer.barentswatch.no/docs/tutorial
```
If the user clicks that the tutorial is not useful, the backend sends the default prompt to AI Provider, generates a new Chinese tutorial, and saves it to `SystemSetting`:
```text
category = collector_credential_guides
```
Reset deletes the custom tutorial and restores the default guide.
## Save Rules
When built-in collector configuration is saved, the internal `connectivity_validation` field is removed so validation state does not mix with user configuration.
BarentsWatch `client_secret` has special handling:
- The input shows a masked preview.
- If the submitted value still matches the masked preview, the backend keeps the old secret.
- If a new value is submitted, the secret is replaced.
- The previous separate "clear current secret" checkbox is no longer provided.
## Test Coverage
Related tests:
- [test_vessels.py](/home/ray/dev/linkong/planet/backend/tests/test_vessels.py)
Added coverage:
- BarentsWatch credentials can be parsed from `~/.zshrc`.
- When environment variables are empty, `resolve_barentswatch_config()` can fall back to `~/.zshrc`.
- Vessel data conversion and GeoJSON output remain compatible.
## Current Provider Coverage
Credential providers currently supported:
- `barentswatch`
- `spacetrack`
Other collectors with `requires_credentials=true` return that their credential chain has not been wired yet, and the frontend shows `Unavailable`.

View File

@@ -187,7 +187,7 @@ Current reality:
- that is expected, because incidents are aggregated and de-noised
- but incident-first rendering makes the Earth view look too quiet unless there is another always-available activity layer
Implementation detail for the recommended `activity layer` is expanded in [bgp-region-aggregation-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-bgp-region-aggregation-plan.md).
Implementation detail for the recommended `activity layer` is expanded in the [BGP Region Aggregation Plan](/home/ray/dev/linkong/planet/docs/plans/earth-bgp-region-aggregation-plan.md).
So the immediate next milestone is:

View File

@@ -4,8 +4,8 @@ This document describes the current real structure of the Earth display frontend
Related references:
- [rules.md](/home/ray/dev/linkong/planet/rules.md)
- [frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/en/frontend-layout-guidelines.md)
- [Project Rules](/home/ray/dev/linkong/planet/rules.md)
- [Frontend Layout Guidelines](/home/ray/dev/linkong/planet/docs/technical/en/frontend-layout-guidelines.md)
## Current Goal
@@ -249,4 +249,4 @@ Therefore:
For console structure, see:
- [frontend-admin-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/en/frontend-admin-frontend-context.md)
- [Admin Frontend Context](/home/ray/dev/linkong/planet/docs/technical/en/frontend-admin-frontend-context.md)

View File

@@ -0,0 +1,270 @@
# Earth Interactable Usage
`Interactable` is the shared rendering entry point for icon-like interactive elements on the Earth surface. It extracts the pattern proven by the vessel layer into reusable behavior: normal state uses batched `THREE.Points`, hover and locked states use small overlays, picking uses screen-space hit testing, icon assets are normalized into canvas textures, and the shared layer handles glow, state, size, ground rendering, and same-coordinate avoidance.
Currently integrated layers:
| Layer | Business File | Icon Source | Extra Animation |
| --- | --- | --- | --- |
| AIS vessels | `frontend/public/earth/js/vessels.js` | canvas draw, moving triangle / anchored dot | Vessel tracks are still maintained by the business layer |
| Compute centers | `frontend/public/earth/js/compute-centers.js` | `assets/icons/compute-*.svg` | Estimated-location `?` badge is added through `icon.afterDraw()` |
| BGP events | `frontend/public/earth/js/bgp.js` | canvas draw, symbol by event type | Expanding rings are still maintained by the BGP business layer |
| BGP observers | `frontend/public/earth/js/bgp.js` | `assets/icons/bgp-broadcast-pin.svg` | Halo, activity core, coverage wedge, and radar sweep remain in the BGP business layer |
Landing sites were previously attempted on Interactable, but pin-style SVGs were fragmented by `THREE.Points` depth testing near the Earth edge. They now use a dedicated `THREE.Sprite` path with a yellow flat-sphere texture generated by canvas. The old SVG assets remain in `assets/icons/`, but landing sites no longer depend on SVG at runtime.
## Why Interactable Exists
Before this layer, each surface icon layer could easily reimplement its own version of:
- icon texture generation
- hover / locked state
- glow styling
- picking radius
- zoom-dependent size strategy
- overlap avoidance for identical coordinates
When this logic is scattered across business files, visual behavior drifts and later tuning becomes layer-by-layer repair. The boundary of `Interactable` is: the shared layer owns how icons remain stable on Earth and how they are selected; the business layer owns where data comes from, what the icon means, what detail cards show, and whether extra animation exists.
## Entry Point
```javascript
import { createInteractableLayer } from "./interactable.js";
```
Core call shape:
```javascript
const layer = createInteractableLayer({
id: "example",
objectType: "example_object",
renderOrder: 4.4,
altitudeOffset: 0.2,
pointSize: 34,
icon: {
draw(context, options) {
// draw canvas icon
},
},
getPosition: (item) => ({
latitude: item.latitude,
longitude: item.longitude,
}),
getKind: (item) => item.kind || "default",
});
```
Business modules usually expose only a thin wrapper:
```javascript
export function getExampleMarkers() {
return layer.getMarkers();
}
export function getExamplePointerIntersections(options) {
return layer.getPointerIntersections(options);
}
export function setExampleMarkerState(marker, state = "normal") {
layer.setMarkerState(marker, state);
}
export function updateExampleVisualState(lockedObjectType, lockedObject, camera) {
layer.updateVisualState(lockedObjectType, lockedObject, camera);
}
```
## Configuration
| Option | Default | Description |
| --- | --- | --- |
| `id` | required | Unique layer id used for group name, avoidance registration, and debug. |
| `objectType` | `id` | Business type written to `marker.userData.type`; the main interaction layer uses it to identify locked objects. |
| `renderOrder` | `4` | Base render order for normal points and hover / locked overlays. |
| `altitudeOffset` | `0.2` | Business altitude, used as `CONFIG.earthRadius + altitudeOffset` for the original surface position. |
| `pointSize` | `32` | Base screen pixel size used by both normal points and overlays. |
| `sizeMode` | `"fixed"` | Fixed screen size by default; non-`"fixed"` modes scale by camera distance. |
| `sizeScale` | `{ referenceFov: 75, min: 0.12, max: 3 }` | Scaling bounds when `sizeMode !== "fixed"`. |
| `atlasCellSize` | `128` | Canvas texture cell size for icons. |
| `colors` | `{}` | Supports `normal`, flattened kind keys, and `byKind`. |
| `opacity` | `{ normal: 0.88, dimmed: 0.26, hover: 0.98, locked: 1 }` | Opacity per state. |
| `stateScale` | `{ hover: 1, locked: 1, dimmed: 1 }` | Size multiplier per state. |
| `pulse` | `{}` | Optional locked-state breathing scale, with `enabled`, `speed`, and `amplitude`. |
| `avoidance` | `{ enabled: true, precision: 4, radius: 1.1, step: 0.35 }` | Same-coordinate avoidance across Interactable layers. |
| `icon` | required | Icon source, supporting canvas draw, SVG / image asset, state asset, anchor, and post-processing. |
| `getPosition(item)` | required | Returns `{ latitude, longitude }` or `THREE.Vector3`. |
| `getKind(item)` | `item.type || "default"` | Returns a business kind for color and texture buckets. |
| `getRotationBin(marker)` | `0` | Returns a rotation bucket, such as 32 heading buckets for vessels. |
| `getBucketKey(marker)` | `String(getRotationBin(marker))` | Returns a texture / geometry bucket key. |
| `getPointSizeMultiplier(marker)` | `1` | Per-marker size multiplier. BGP events use severity; observers use activity. |
| `getUserData(item)` | `item` | Business fields written onto the marker. |
## Icon Configuration
`icon.anchor` is optional and defaults to `{ x: 0.5, y: 0.5 }`, meaning the texture center aligns with the marker coordinate. It is only suitable for small visual anchor offsets. If the icon body is large and must remain fully visible at the Earth edge, such as the old landing-site pin, it should not be forced through `THREE.Points + depthTest`; the body will be clipped by Earth depth.
### Canvas Icons
Canvas icons fit vessels and BGP events where symbols need to be drawn dynamically by state or rotation:
```javascript
const vesselIconLayer = createInteractableLayer({
id: "vessels",
objectType: "vessel",
pointSize: 34,
icon: {
draw(context, { marker, rotationBin = 0, glow = false, color = "#ffffff" }) {
if (!marker.userData.anchored) {
context.rotate((rotationBin / 32) * Math.PI * 2);
}
context.fillStyle = color;
context.shadowColor = color;
context.shadowBlur = glow ? 14 : 0;
context.beginPath();
context.moveTo(0, -37);
context.lineTo(28, 32);
context.lineTo(0, 17);
context.lineTo(-28, 32);
context.closePath();
context.fill();
},
},
getRotationBin: getCourseBin,
getBucketKey: (marker) => `${marker.userData.anchored ? "anchored" : "moving"}:${getCourseBin(marker)}`,
});
```
When `icon.coordinates !== "canvas"`, `Interactable` translates the context to the atlas center first. Vessel-style icons that already draw around center coordinates do not need to declare `coordinates`.
### SVG / Image Asset Icons
Asset icons fit facilities such as compute centers and BGP observers:
```javascript
const computeCenterIconLayer = createInteractableLayer({
id: "computeCenters",
objectType: "compute_center",
pointSize: 36,
atlasCellSize: 128,
icon: {
coordinates: "canvas",
colorable: false,
fitSize: 60,
glowBlur: 16,
getSource({ marker, item }) {
const siteType = marker?.userData?.site_type || item?.site_type || "gpu_cluster";
return COMPUTE_CENTER_ICON_SOURCES[siteType];
},
afterDraw(context, { marker, item }) {
if (marker?.userData?.is_estimated ?? item?.is_estimated) {
drawComputeCenterEstimatedBadge(context, true);
}
},
},
});
```
Asset conventions:
- SVG / image files live in `frontend/public/earth/assets/icons/` and are referenced as `/earth/assets/icons/name.svg`.
- Original SVGs should keep a standard `viewBox` and paths; avoid hard-coding transform only for display size.
- Display size is controlled by `icon.fitSize`; it can be a number, `{ width, height }`, or a function.
- If `icon.colorable !== false` and state colors are provided, the shared layer first draws the asset to a temporary canvas and then tints it with `source-in`.
- Multicolor images or SVGs that should not be tinted must set `colorable: false`.
## Lifecycle
Typical load flow:
```javascript
export async function loadExampleLayer(_scene, earth) {
clearExampleData(earth);
const markerData = await fetchExampleData();
await layer.preloadAssets(markerData);
layer.setData(markerData);
layer.attach(earth);
layer.setVisible(showExampleLayer);
return { totalCount: layer.getCount() };
}
```
Method responsibilities:
| Method | Description |
| --- | --- |
| `preloadAssets(items)` | Collects asset sources that may be used by normal / hover / locked states and preloads them with browser `Image`. Canvas-drawn icons can skip this. |
| `setData(items)` | Clears old points, creates markers, registers avoidance, and rebuilds `THREE.Points` by bucket. |
| `attach(parent)` | Mounts the layer group onto the Earth root. |
| `setVisible(next)` | Controls visibility for the group, points, and overlays. |
| `setMarkerState(marker, state)` | Sets `normal` / `hover` and other states, then invalidates visual state. |
| `updateVisualState(focusType, focusObject, camera)` | Updates normal opacity / size and refreshes hover / locked overlays. |
| `getPointerIntersections(options)` | Runs screen-space picking and returns hits sorted by pixel distance. |
| `clearData(parent)` | Unregisters avoidance, disposes geometry / material, clears markers, and removes the group from the parent. |
## Picking Integration
`Interactable` does not depend on the default Three.js raycast for `Points`. The main interaction layer passes Earth, camera, pointer, and hit radius:
```javascript
const intersects = getVesselPointerIntersections({
earth,
camera,
pointer,
radiusPx: 22,
width: window.innerWidth,
height: window.innerHeight,
});
```
The shared layer:
1. Converts the camera position into Earth-local coordinates.
2. Skips markers on the back side.
3. Projects marker world position into screen coordinates.
4. Uses `radiusPx` for pixel-distance hits.
5. Returns the nearest candidate objects.
Earth dragging, inertia, and hover throttling still belong to `main.js` because they depend on global input state.
## Same-Coordinate Avoidance
Avoidance is enabled by default and applies to all layers created through `createInteractableLayer()`. The shared layer builds an `icon_avoidance_key` from latitude / longitude or `THREE.Vector3`, then arranges markers with the same key into a small circle along the surface tangent plane.
Key points:
- `icon_base_position` keeps the original business position.
- Avoidance only changes rendering and picking position. It does not change business latitude / longitude.
- When a single marker returns to its original position, it uses the business surface position computed from `altitudeOffset`.
- When multiple markers share coordinates, the first ring uses `avoidance.radius`; later rings add `avoidance.step`.
If a business layer must stay exactly on the original point, disable avoidance explicitly:
```javascript
createInteractableLayer({
id: "strict-layer",
avoidance: { enabled: false },
});
```
## Business Animation Boundary
`Interactable` currently owns only the icon body and common hover / locked overlays. Complex animations remain in business modules, but should follow the Interactable marker position:
- BGP event expanding rings are independent ring sprites created by `bgp.js`, updated every frame with `position.copy(marker.position)`.
- BGP observer halo, status core, coverage halo, and coverage wedge are managed by `bgp.js`; the icon body is managed by Interactable.
- Vessel tracks remain in `vessels.js` because they depend on track data loaded after a click.
This boundary avoids pushing every animation type into the shared interface too early. If multiple layers reuse the same animation type later, it can move into an Interactable `animations` extension.
## New Layer Checklist
1. Prepare marker data in the business file and keep required business fields.
2. Choose an icon type: canvas draw, SVG / image asset, or dynamic `getSource()`.
3. Configure `pointSize`, `icon.fitSize`, `colors`, `opacity`, and `stateScale`.
4. Provide `getPointSizeMultiplier()` if business-specific size variation is needed.
5. Provide `getRotationBin()` and a stable `getBucketKey()` if rotation exists.
6. During load, call `preloadAssets()` before `setData()`, `attach()`, and `setVisible()`.
7. Wire `getPointerIntersections()` in `main.js` and reuse the existing hover / locked state update flow.
8. Record altitude, `renderOrder`, `pointSize`, and animation ordering in the layer style index and render order documents.

View File

@@ -1,6 +1,6 @@
# Earth Layer Style Property Index
This document records the material, color, opacity, line width, radius offset, and `renderOrder` style properties of all Earth frontend layers. For layer ordering relationships, see [earth-render-layer-order.md](/home/ray/dev/linkong/planet/docs/technical/en/earth-render-layer-order.md).
This document records the material, color, opacity, line width, radius offset, and `renderOrder` style properties of all Earth frontend layers. For layer ordering relationships, see [Earth Render Layer Order](/home/ray/dev/linkong/planet/docs/technical/en/earth-render-layer-order.md).
## Naming Conventions
@@ -74,6 +74,8 @@ This document records the material, color, opacity, line width, radius offset, a
## Land/Ocean Base and Country Borders
The land/ocean base is an Earth base-map asset and preloads at startup; the "Border Lines" layer toggle only controls normal border lines, hover lines, and interactive hover.
| Name | Variable | Current Value | Location / Notes |
| --- | --- | --- | --- |
| Country border data path | `COUNTRY_BOUNDARY_CONFIG.dataPath` | `"/earth/data/countries-admin0.min.geojson"` | GeoJSON input |
@@ -89,11 +91,11 @@ This document records the material, color, opacity, line width, radius offset, a
| Border line color | `COUNTRY_BOUNDARY_CONFIG.lineColor` | `0x7fc7ff` | Normal border line |
| Border line opacity | `COUNTRY_BOUNDARY_CONFIG.lineOpacity` | `0.58` | Normal border line opacity |
| Border dimmed opacity on hover | `COUNTRY_BOUNDARY_CONFIG.dimmedLineOpacity` | `0.18` | Normal border opacity during hover |
| Border line radius offset | `COUNTRY_BOUNDARY_CONFIG.lineAltitudeOffset` | `0.24` | Normal border line radius |
| Border line radius offset | `COUNTRY_BOUNDARY_CONFIG.lineAltitudeOffset` | `0.115` | Normal border line radius; slightly above HD texture `0.10` and below terrain base `0.16` to reduce floating |
| Border line renderOrder | `COUNTRY_BOUNDARY_CONFIG.lineRenderOrder` | `2.2` | Normal border line level |
| Border hover color | `COUNTRY_BOUNDARY_CONFIG.hoverLineColor` | `0xff3b1f` | Neon red-orange |
| Border hover opacity | `COUNTRY_BOUNDARY_CONFIG.hoverLineOpacity` | `1.0` | Hover line opacity |
| Border hover radius offset | `COUNTRY_BOUNDARY_CONFIG.hoverAltitudeOffset` | `0.32` | Hover line radius |
| Border hover radius offset | `COUNTRY_BOUNDARY_CONFIG.hoverAltitudeOffset` | `0.14` | Hover line radius; close to the surface but above normal border lines |
| Border hover renderOrder | `COUNTRY_BOUNDARY_CONFIG.hoverLineRenderOrder` | `2.3` | Hover line level |
| Border hover glow opacity | `COUNTRY_BOUNDARY_CONFIG.hoverGlowOpacity` | `0.38` | Glow line opacity |
| Border hover glow line width | `COUNTRY_BOUNDARY_CONFIG.hoverGlowLineWidth` | `3` | Glow `LineBasicMaterial.linewidth` |
@@ -140,17 +142,18 @@ This document records the material, color, opacity, line width, radius offset, a
| Cable line width | `CABLE_CONFIG.line.lineWidth` | `1` | `LineBasicMaterial.linewidth` |
| Cable opacity | `CABLE_CONFIG.line.opacity` | `1.0` | Cable line opacity |
| Cable renderOrder | `CABLE_CONFIG.line.renderOrder` | `1` | Cable line level |
| Landing point radius offset | `CABLE_CONFIG.landingPoint.altitudeOffset` | `0.48` | Aligns with compute center marker height |
| Landing point icon texture size | `CABLE_CONFIG.landingPoint.textureSize` | `256` | Canvas size for solid map-pin icon |
| Landing point icon aspect ratio | `CABLE_CONFIG.landingPoint.iconAspectRatio` | `0.82` | `Sprite.scale.x = height * aspect` |
| Landing point icon anchor | `CABLE_CONFIG.landingPoint.anchorX / anchorY` | `0.52 / 0.276` | `Sprite.center`, aligns pin tip to landing point lat/lon |
| Landing point base scale | `CABLE_CONFIG.landingPoint.baseScale` | `12` | Matches compute center sprite height |
| Landing point radius offset | `CABLE_CONFIG.landingPoint.altitudeOffset` | `0.2` | Same surface height as cable lines, avoiding a floating marker |
| Landing point sprite height | local `LANDING_POINT_SPRITE_HEIGHT` | `3` | `THREE.Sprite` base height |
| Landing point reference FOV | local `LANDING_POINT_SIZE_REFERENCE_FOV` | `75` | Matches the current Earth camera FOV |
| Landing point scale minimum | local `LANDING_POINT_SIZE_SCALE_MIN` | `0.16` | Minimum multiplier after roughly 200% zoom, limiting high-zoom screen footprint; `3 * 0.16 = 0.48` |
| Landing point scale maximum | local `LANDING_POINT_SIZE_SCALE_MAX` | `3` | Maximum multiplier at far distance; current minimum zoom reaches roughly `2.50` |
| Landing point atlas size | local `LANDING_POINT_ATLAS_CELL_SIZE` | `128` | Canvas flat shaded sphere texture size |
| Landing point color | `CABLE_CONFIG.landingPoint.color` | `0xffaa00` | `SpriteMaterial.color` |
| Landing point opacity | `CABLE_CONFIG.landingPoint.opacity` | `1.0` | `SpriteMaterial.opacity` |
| Landing point renderOrder | `CABLE_CONFIG.landingPoint.renderOrder` | `4.5` | Aligns with compute center surface level |
| Landing point renderOrder | `CABLE_CONFIG.landingPoint.renderOrder` | `1` | Same level as cable lines; `depthTest: false` keeps the ball whole, while camera-to-center globe occlusion hides back-side points |
| Landing point dim brightness | `landingPointVisual.dimBrightness` | `0.62` | Dim state color multiplier |
| Related landing point opacity | `landingPointVisual.related.opacityBase / opacityPulse` | `0.8 / 0.2` | Highlight pulse |
| Dimmed landing point color | `landingPointVisual.dimmed.colorRGB` | `{ r: 180, g: 116, b: 28 }` | Dim state color; avoids dark base showing through as a dark hole |
| Dimmed landing point emissive | `landingPointVisual.dimmed.emissive` | `0x3a2200` | Dim state weak amber self-emission |
| Dimmed landing point opacity | `landingPointVisual.dimmed.opacity` | `0.78` | Dim state opacity; no longer uses low alpha blending with dark base |
## Satellites, Trails, and Footprints
@@ -168,18 +171,34 @@ This document records the material, color, opacity, line width, radius offset, a
| Satellite trail line width | `SATELLITE_CONFIG.trailLineWidth` | `3` | Ribbon shader uniform |
| Selected ring size | `SATELLITE_CONFIG.ringSize` | `0.07` | Hover / locked ring sprite |
| Satellite overlay renderOrder | `SATELLITE_CONFIG.overlayRenderOrder` | `12` | Locked ring / halo / orbit |
| Footprint renderOrder | local `GROUND_FOOTPRINT_RENDER_ORDER` | `3` | Footprint fill |
| Footprint renderOrder | local `GROUND_FOOTPRINT_RENDER_ORDER` | `3` | Starlink footprint fill and Iridium coverage ring; must stay above land / texture / terrain surface layers |
## AIS Vessels
| Name | Variable | Current Value | Location / Notes |
| --- | --- | --- | --- |
| Vessel radius offset | `VESSEL_CONFIG.altitudeOffset` | `0.2` | Normal marker position, close to the real terrain base layer |
| Vessel track radius offset | `VESSEL_CONFIG.track.altitudeOffset` | `0.2` | Selected vessel track line, aligned to the vessel marker radius; the frontend anchors the track endpoint to the current marker position |
| Vessel renderOrder | local `VESSEL_RENDER_ORDER` | `4.4` | Normal marker and interactive overlay |
| Vessel track renderOrder | `VESSEL_RENDER_ORDER - 0.1` | `4.3` | Below vessel markers |
| Vessel point pixel size | local `VESSEL_POINT_SIZE` | `34` | Shared size for normal markers and hover / locked overlays |
| Vessel texture canvas size | local `VESSEL_ATLAS_CELL_SIZE` | `128` | Canvas point texture |
| Course bucket count | local `VESSEL_COURSE_BINS` | `32` | Moving vessels are bucketed by COG to reduce draw calls while preserving direction |
| Vessel hover picking throttle | local `VESSEL_HOVER_PICK_INTERVAL_MS` | `100` | `main.js` hover picking |
| Vessel screen hit radius | local `VESSEL_POINTER_RADIUS_PX` | `22` | `main.js` screen-space picking |
## Compute Centers
| Name | Variable | Current Value | Location / Notes |
| --- | --- | --- | --- |
| Compute center radius offset | `COMPUTE_CENTER_CONFIG.altitudeOffset` | `0.48` | Marker position |
| Compute center base opacity | `COMPUTE_CENTER_CONFIG.marker.baseOpacity` | `0.88` | `SpriteMaterial.opacity` |
| Supercomputer marker scale | `COMPUTE_CENTER_CONFIG.marker.supercomputerScale` | `12` | Supercomputer marker |
| GPU cluster marker scale | `COMPUTE_CENTER_CONFIG.marker.gpuClusterScale` | `12` | GPU marker |
| Hover scale | `COMPUTE_CENTER_CONFIG.marker.hoverScale` | `1.16` | Hover state |
| Locked scale | `COMPUTE_CENTER_CONFIG.marker.lockedScale` | `1.22` | Locked state |
| Compute center point size | local `COMPUTE_CENTER_POINT_SIZE` | `36` | Shared Interactable base size for normal markers and hover / locked overlays |
| Compute center asset fit size | local `COMPUTE_CENTER_ICON_FIT_SIZE` | `60` | Maximum SVG asset draw size inside the `128x128` atlas canvas, controlled by `icon.fitSize` |
| Compute center base opacity | `COMPUTE_CENTER_CONFIG.marker.baseOpacity` | `0.88` | Normal `PointsMaterial.opacity` |
| Supercomputer marker scale | `COMPUTE_CENTER_CONFIG.marker.supercomputerScale` | `12` | Legacy Sprite scale; not directly used by the current Interactable path |
| GPU cluster marker scale | `COMPUTE_CENTER_CONFIG.marker.gpuClusterScale` | `12` | Legacy Sprite scale; not directly used by the current Interactable path |
| Hover scale | `COMPUTE_CENTER_CONFIG.marker.hoverScale` | `1.16` | Hover overlay size multiplier |
| Locked scale | `COMPUTE_CENTER_CONFIG.marker.lockedScale` | `1.22` | Locked overlay size multiplier, with pulse |
| Dimmed scale / opacity | `dimmedScale / dimmedOpacity` | `0.82 / 0.34` | Dim state |
| Supercomputer color | `COMPUTE_CENTER_CONFIG.colors.supercomputer` | `"#38bdf8"` | Marker texture |
| GPU cluster color | `COMPUTE_CENTER_CONFIG.colors.gpu_cluster` | `"#2dd4bf"` | Marker texture |
@@ -190,12 +209,16 @@ This document records the material, color, opacity, line width, radius offset, a
| Name | Variable | Current Value | Location / Notes |
| --- | --- | --- | --- |
| BGP event radius offset | `BGP_CONFIG.altitudeOffset` | `2.1` | Anomaly marker |
| BGP collector radius offset | `BGP_CONFIG.collectorAltitudeOffset` | `1.6` | Collector marker |
| Event base scale | `BGP_CONFIG.marker.eventBaseScale` | `6.2` | Anomaly sprite |
| Collector base scale | `BGP_CONFIG.marker.collectorBaseScale` | `7.4` | Collector plane |
| BGP event radius offset | `BGP_CONFIG.altitudeOffset` | `0.48` | BGP event Interactable marker |
| BGP collector radius offset | `BGP_CONFIG.collectorAltitudeOffset` | `0.2` | BGP collector Interactable marker, aligned with the vessel layer |
| BGP event point size | local `BGP_EVENT_POINT_SIZE` | `34` | Event Interactable base size, adjusted by severity through `getPointSizeMultiplier()` |
| BGP event symbol draw size | local `BGP_EVENT_SYMBOL_SIZE` | `60` | Event canvas symbol draw size inside the `128x128` atlas |
| BGP collector point size | local `BGP_COLLECTOR_POINT_SIZE` | `36` | Collector Interactable base size, adjusted by activity through `getPointSizeMultiplier()` |
| BGP collector asset fit size | local `BGP_COLLECTOR_ICON_FIT_SIZE` | `60` | Maximum `bgp-broadcast-pin.svg` draw size inside the atlas canvas |
| Event base scale | `BGP_CONFIG.marker.eventBaseScale` | `6.2` | Event ring anchor |
| Collector base scale | `BGP_CONFIG.marker.collectorBaseScale` | `7.4` | Collector halo / coverage animation anchor |
| Hover / dim scale | `hoverScale / dimmedScale` | `1.16 / 0.92` | Interaction states |
| Normal event opacity | `BGP_CONFIG.opacity.normal` | `0.78` | Anomaly sprite |
| Normal event opacity | `BGP_CONFIG.opacity.normal` | `0.78` | BGP event Interactable normal state |
| Hover opacity | `BGP_CONFIG.opacity.hover` | `1.0` | Hover state |
| Dimmed opacity | `BGP_CONFIG.opacity.dimmed` | `0.24` | Dim state |
| Collector opacity | `BGP_CONFIG.opacity.collector` | `0.62` | Collector state |

View File

@@ -103,7 +103,7 @@ Goals:
## Collector Configuration
`news_live_streams` does not need a separate new page; it reuses the existing data source configuration:
`news_live_streams` does not need a separate new page; it reuses Collector Settings under `/settings`:
- `endpoint`
- Channel directory JSON API URL

View File

@@ -6,8 +6,8 @@ Note: the layer control panel order and the registration / startup load order ar
| Order type | Current sequence | Notes |
| --- | --- | --- |
| Control panel order | Cables → Trails → Satellites → Compute Centers → BGP → Terrain → HD Texture → Cloud Layer → Borders → Grid | Controlled by `displayOrder`, sorted by operational relevance. |
| Registration / startup load order | Grid → Borders → HD Texture → Cloud Layer → Cables → Compute Centers → BGP → Satellites | Controlled by registration order and `startupPriority`, sorted surface-to-sky; Trails and Terrain are dependency/optional display layers and do not participate in normal startup data loading. |
| Control panel order | Cables → Trails → Satellites → Compute Centers → BGP → Terrain → HD Texture → Cloud Layer → Border Lines → Grid | Controlled by `displayOrder`, sorted by operational relevance. |
| Registration / startup load order | Grid → Border Lines / Land-Ocean Base → HD Texture → Cloud Layer → Cables → Compute Centers → BGP → Satellites | Controlled by registration order and `startupPriority`, sorted surface-to-sky; the startup queue reads persisted layer visibility first, skips normal layers explicitly saved as hidden, and HD Texture does not download the texture when disabled; Border Lines are the exception: the land-ocean base always preloads, while the persisted state only controls interactive border lines and hover; Trails and Terrain are dependency/optional display layers and do not participate in normal startup data loading. |
## Surface Layer Stack
@@ -26,7 +26,7 @@ Note: the layer control panel order and the registration / startup load order ar
| 2.2 | Country borders | `country-boundaries.js` | `lineAltitudeOffset` | Raycast disabled | Only needs to stay above HD texture. |
| 2.29 | Country border hover glow | `country-boundaries.js` | Hover radius + glow offset | `depthTest: false`, raycast disabled | Additive glow to reinforce border edge and terrain hover visibility. |
| 2.3 | Country border hover line | `country-boundaries.js` | `hoverAltitudeOffset` | `depthTest: false`, raycast disabled | Neon red-orange hover line; China and Taiwan share the same highlight group. |
| 3 | Satellite footprint fill | `satellites.js` | `GROUND_FOOTPRINT_RENDER_ORDER` | depth-tested, Group renderOrder stays 0 | Footprint above country borders, below compute centers and satellites. |
| 3 | Satellite footprint fill / Iridium coverage ring | `satellites.js`, `iridium-footprint-adapter.js` | `GROUND_FOOTPRINT_RENDER_ORDER` | depth-tested; Iridium adapter fill / ring use the same renderOrder | Footprint above land / texture / terrain and country borders, below compute centers and satellites. |
| 3-5 | BGP markers and overlays | `bgp.js` | Each marker's own renderOrder | BGP picking path | Preserves existing BGP visual level. |
| 4.5 | Compute centers | `compute-centers.js` | `COMPUTE_CENTER_RENDER_ORDER` | Compute center picking path | Surface facilities, below satellites. |
| 5 | Satellite background dot | `satellites.js` | Fixed renderOrder | Screen-space satellite picking | Below satellite dots. |
@@ -42,7 +42,7 @@ Note: the layer control panel order and the registration / startup load order ar
| HD texture on | Restores HD texture and the remembered terrain / day/night states. |
| Terrain on | Displayed above HD texture, but below country border hover, footprints, satellites, and other emphasis layers. |
| Cloud layer | Only controls cloud mesh visibility. |
| Country borders | Controls border line and hover line visibility; land/ocean base fill exists independently as the Earth base map. |
| Border Lines off | Hides only interactive border lines and hover, clearing hover state; the land/ocean base fill remains as the Earth base map. |
## Interaction Rules

View File

@@ -4,8 +4,8 @@ This document records the current product boundary, data rationale, and implemen
Related context:
- [earth-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/en/earth-frontend-context.md)
- [backend-collectors.md](/home/ray/dev/linkong/planet/docs/technical/en/backend-collectors.md)
- [Earth Frontend Context](/home/ray/dev/linkong/planet/docs/technical/en/earth-frontend-context.md)
- [Backend Collectors](/home/ray/dev/linkong/planet/docs/technical/en/backend-collectors.md)
- [backend/app/services/collectors/celestrak.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/celestrak.py)
- [frontend/public/earth/js/satellites.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/satellites.js)

View File

@@ -4,8 +4,8 @@ This document describes the current real structure of the console frontend. The
Related references:
- [rules.md](/home/ray/dev/linkong/planet/rules.md)
- [frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/en/frontend-layout-guidelines.md)
- [Project Rules](/home/ray/dev/linkong/planet/rules.md)
- [Frontend Layout Guidelines](/home/ray/dev/linkong/planet/docs/technical/en/frontend-layout-guidelines.md)
## Current Goal
@@ -263,7 +263,7 @@ These principles have been repeatedly validated in the project:
For detailed experience, see:
- [frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/en/frontend-layout-guidelines.md)
- [Frontend Layout Guidelines](/home/ray/dev/linkong/planet/docs/technical/en/frontend-layout-guidelines.md)
## Recommended Change Approach
@@ -290,4 +290,4 @@ Therefore:
For Earth-related structure, see:
- [earth-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/en/earth-frontend-context.md)
- [Earth Frontend Context](/home/ray/dev/linkong/planet/docs/technical/en/earth-frontend-context.md)

View File

@@ -7,7 +7,7 @@ This manual is for daily use, demos, development integration, and local operatio
- Console: admin backend (login required)
- Docs: public developer documentation and manual
For the shortest path to getting started, see [quickstart.md](/home/ray/dev/linkong/planet/docs/technical/en/quickstart.md).
For the shortest path to getting started, see [Quickstart](/home/ray/dev/linkong/planet/docs/technical/en/quickstart.md).
## Entry Overview
@@ -55,6 +55,30 @@ Parameters:
| `--allow-lan` | Enable LAN access |
| `--verbose` | Show more command output during execution |
### AI Provider Environment and Builds
AI Provider runtime configuration can live in `aiprovider/.env` or in matching variables in `~/.zshrc`. `planet.sh` reads simple `export AI_...=...` / `AI_...=...` lines and passes them to the container at startup.
Changing model, API key, or base URL does not rebuild the image. Restart only AI Provider to pick up runtime configuration changes:
```bash
./planet.sh restart -a
```
For complex shell expansion in `~/.zshrc`, opt in explicitly:
```bash
PLANET_LOAD_ZSHRC_ENV=source ./planet.sh start -a
```
To ignore `~/.zshrc` during troubleshooting:
```bash
PLANET_LOAD_ZSHRC_ENV=0 ./planet.sh start -a
```
The AI Provider Docker build context is intentionally limited to the files required by the service, and `uv sync` uses a BuildKit cache mount so dependency downloads are reused after the first build.
### Stop
```bash
@@ -178,7 +202,8 @@ Earth is used to observe in a single globe view:
- Satellites and orbital trails
- Submarine cables and landing points
- Compute centers
- Country borders, grid lines, HD texture, cloud layer, terrain
- AIS vessels
- Border lines, grid lines, HD texture, cloud layer, terrain
- Live news streams and situational news
- Search and focused object details
@@ -189,12 +214,13 @@ The right-side layer panel toggles visualization layers on or off.
Common layers include:
- Grid lines
- Country borders
- Border lines
- HD texture
- Atmospheric cloud layer
- Submarine cables
- Compute centers
- BGP observation
- AIS vessels
- Satellites
- Orbital trails
- Terrain
@@ -205,6 +231,21 @@ Some layers have dependencies:
- Trails require Satellites
- When HD texture is off, the globe shows the base map and edge glow effect
### Legend
The lower-left legend follows the currently focused or enabled layer.
Current legend modes include:
- Cables
- Satellites
- Border lines
- Compute centers
- BGP
- AIS vessels
AIS vessel legend entries are grouped by vessel type: cargo, tanker, passenger, fishing, military, anchored/slow, and other. Triangle markers represent moving vessels; dots represent anchored or slow vessels.
### Search
Earth search finds current globe objects, such as:
@@ -233,6 +274,25 @@ The settings panel contains:
These settings are stored in browser local storage. They revert to defaults if you switch browsers or clear site data.
### View Controls
Earth supports mouse, touchpad, and touchscreen interaction.
Common controls:
| Action | Result |
| --- | --- |
| Left-button drag | Rotates the globe |
| One-finger drag | Rotates the globe on touch devices |
| Mouse wheel | Zooms the view in or out |
| Two-finger pinch | Zooms the view on touch devices |
| Zoom buttons | Adjust zoom in fixed steps |
| Click the zoom percent | Resets to the default zoom |
When zooming, the top capsule briefly shows the current zoom level, for example `Zoom 180%`. This indicates view zoom only, not data loading progress. Loading status takes priority and will not be interrupted by zoom feedback.
Drag sensitivity adjusts automatically based on the current zoom. Around the default view it keeps the normal rotation feel; when zoomed in, dragging becomes progressively finer for inspecting a region, vessel, satellite, or BGP event; when zoomed out, dragging is slightly faster for global browsing.
### Cruise Mode
Cruise mode makes Earth automatically cycle through focus targets.
@@ -308,7 +368,7 @@ Common pages:
| --- | --- | --- |
| Dashboard | `/admin` | System overview |
| Earth | `/earth` | Opens the public Earth page |
| Data Sources | `/datasources` | Manage data sources and trigger collection |
| Data Sources | `/datasources` | View data sources and trigger collection |
| Collected Data | `/data` | View collected data |
| BGP Observation | `/bgp` | BGP situational data |
| System Alerts | `/alerts/system` | System-level alerts |
@@ -321,17 +381,21 @@ Common pages:
### Data Sources
`/datasources` shows and manages collection sources.
`/datasources` shows collection sources and triggers collection. It is now a data source directory that lists built-in and custom sources in one table.
Common operations:
- View data source status
- Trigger collection
- View recent collection tasks
- Adjust configuration
- Open the read-only detail drawer for endpoint, headers, runtime config, and built-in/custom source type
If a category of objects is missing on Earth, start here to confirm the data source is available.
The data source name opens an information drawer only. Endpoint, credentials, headers, and custom source configuration are maintained under `/settings` collector settings.
When collection tasks are running, the progress area shows a clickable `Collecting N` pill. Clicking it opens a modal with each running task's phase, progress, and processed count.
### Collected Data
`/data` shows the collected data table.
@@ -369,10 +433,50 @@ Current common uses:
- System settings
- TV live stream source configuration
- Data source configuration entry points
- Collector settings
- External integrations and AI Provider configuration
Available configuration depends on the current user's role.
#### Collector Settings
`/settings?tab=collector_credentials` is currently displayed as Collector Settings. It manages connection settings for all collectors, not only credentials.
Use it to:
1. Select a collector from the dropdown.
2. Review tags such as `Requires credentials`, module, enabled state, and `Unchecked` / `Available` / `Unavailable`.
3. Click the plug icon next to the selector to run a health check.
4. Edit endpoint, request headers, timeout, and retry settings.
5. Save the collector settings.
Free collectors are checked by requesting their endpoint directly. Credentialed collectors use their credential provider. If endpoint or credential fingerprint changes after the last successful validation, the collector must be checked again.
The system treats a collector as connected when the current configuration has either collected data successfully or passed the manual connection check.
#### BarentsWatch AIS Credentials
`BarentsWatch AIS` is a credentialed built-in collector. Its credential card appears above the basic configuration card.
Configured fields:
- `Client ID`
- `Client Secret`
- `Endpoint`
If a secret is already configured, the input shows a masked preview. Keeping that preview unchanged preserves the stored secret; entering a new value replaces it.
BarentsWatch AIS credentials can be read from:
1. Collector settings saved in the console.
2. Backend environment variables:
- `BARENTSWATCH_CLIENT_ID`
- `BARENTSWATCH_CLIENT_SECRET`
- historical spellings: `BARRENTSWATCH_CLIENT_ID`, `BARRENTSWATCH_CLIENT_SECRET`
3. matching `export` lines in `~/.zshrc`.
If connection fails, the page opens the credential guide. The guide can be regenerated through AI Provider or reset to the default guide. The default guide points users to the official BarentsWatch tutorial and emphasizes selecting `AIS - API`, not the regular `BarentsWatch - API`.
### System Logs
`/logs` views system logs. If the menu item is not visible, the current user likely lacks the required role.
@@ -481,9 +585,9 @@ When something goes wrong, follow this sequence:
## Related Docs
- [quickstart.md](/home/ray/dev/linkong/planet/docs/technical/en/quickstart.md)
- [frontend-admin-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/en/frontend-admin-frontend-context.md)
- [earth-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/en/earth-frontend-context.md)
- [earth-layer-style-reference.md](/home/ray/dev/linkong/planet/docs/technical/en/earth-layer-style-reference.md)
- [backend-system-service-control.md](/home/ray/dev/linkong/planet/docs/technical/en/backend-system-service-control.md)
- [backend-collectors.md](/home/ray/dev/linkong/planet/docs/technical/en/backend-collectors.md)
- [Quickstart](/home/ray/dev/linkong/planet/docs/technical/en/quickstart.md)
- [Admin Frontend Context](/home/ray/dev/linkong/planet/docs/technical/en/frontend-admin-frontend-context.md)
- [Earth Frontend Context](/home/ray/dev/linkong/planet/docs/technical/en/earth-frontend-context.md)
- [Earth Layer Style Reference](/home/ray/dev/linkong/planet/docs/technical/en/earth-layer-style-reference.md)
- [System Service Control](/home/ray/dev/linkong/planet/docs/technical/en/backend-system-service-control.md)
- [Backend Collectors](/home/ray/dev/linkong/planet/docs/technical/en/backend-collectors.md)

View File

@@ -24,6 +24,12 @@ This script checks and syncs common dependencies, and generates if missing:
- `aiprovider/.env`
- `frontend/.env.local`
Personal AI Provider configuration can also live in `~/.zshrc`. `planet.sh` reads simple `export AI_...=...` / `AI_...=...` lines and passes them to the AI Provider container. After changing model, key, or base URL, restart only AI Provider:
```bash
./planet.sh restart -a
```
## 1. Start Services
From the repository root:
@@ -73,6 +79,7 @@ Once in, verify:
- The globe renders correctly
- The right-side layer panel can toggle layers on/off
- Search can find cables, satellites, compute centers, BGP events
- Mouse drag, wheel zoom, and zoom percent feedback work correctly
- Settings panel can switch cruise mode, day/night mode, satellite display style
## 4. Open the Console
@@ -87,7 +94,7 @@ The console manages data sources, collected data, situational observation, alert
First-time inspection checklist:
- `/datasources`: data source configuration and collection status
- `/datasources`: data source directory and collection triggers; endpoint, headers, and credentials are configured under `/settings` collector settings
- `/data`: collected data
- `/bgp`: BGP situational view
- `/alerts/system`: system alerts
@@ -187,7 +194,7 @@ This shuts down the frontend, backend, AI Provider, PostgreSQL, and Redis.
## Next Steps
- Full usage guide: [manual.md](/home/ray/dev/linkong/planet/docs/technical/en/manual.md)
- Console structure: [frontend-admin-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/en/frontend-admin-frontend-context.md)
- Earth structure: [earth-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/en/earth-frontend-context.md)
- Backend collectors: [backend-collectors.md](/home/ray/dev/linkong/planet/docs/technical/en/backend-collectors.md)
- Full usage guide: [Planet Manual](/home/ray/dev/linkong/planet/docs/technical/en/manual.md)
- Console structure: [Admin Frontend Context](/home/ray/dev/linkong/planet/docs/technical/en/frontend-admin-frontend-context.md)
- Earth structure: [Earth Frontend Context](/home/ray/dev/linkong/planet/docs/technical/en/earth-frontend-context.md)
- Backend collectors: [Backend Collectors](/home/ray/dev/linkong/planet/docs/technical/en/backend-collectors.md)

View File

@@ -1,35 +1,37 @@
# Technical Docs
# 技术文档
这里放“当前实现和当前结构”的文档,重点回答:
- 现在代码是怎么组织的
- 当前入口在哪
- 状态和组件如何工作
- 后续改动应该沿着哪条实现边界继续走
适合放入这里的内容:
- Quickstart 和使用手册
- 快速开始和使用手册
- 前端上下文
- Earth 前端结构
- Earth 卫星 footprint 策略
- Earth 卫星覆盖策略
- Earth 渲染图层顺序
- Earth 图层样式属性索引
- 后端运行控制
- collector 现状
- 采集器现状
- 采集器设置与连接验证
- 采集格式约定
## 使用入口
- [quickstart.md](/home/ray/dev/linkong/planet/docs/technical/quickstart.md):从零启动 Planet 的最短路径
- [manual.md](/home/ray/dev/linkong/planet/docs/technical/manual.md):控制台、`planet.sh`、Earth 和 Docs 的完整使用手册
- [快速开始](/home/ray/dev/linkong/planet/docs/technical/zh/quickstart.md):从零启动 Planet 的最短路径
- [Planet 使用手册](/home/ray/dev/linkong/planet/docs/technical/zh/manual.md):控制台、`planet.sh`、Earth 和 Docs 的完整使用手册
- [数据源、采集器设置与连接验证](/home/ray/dev/linkong/planet/docs/technical/zh/datasource-collector-settings-connectivity.md)数据源目录、采集器设置、连接验证、BarentsWatch 凭证链路
- [Earth 可交互图标接入](/home/ray/dev/linkong/planet/docs/technical/zh/earth-interactable-usage.md)Earth 地表可交互图标 `Interactable` 的接口、生命周期和接入示例
不适合放入这里的内容:
- 尚未完成的 roadmap
- 尚未完成的路线图
- 未来迭代方案
- 大范围重构计划
这些应放入:
- [docs/plans/README.md](/home/ray/dev/linkong/planet/docs/plans/README.md)
- [计划文档索引](/home/ray/dev/linkong/planet/docs/plans/README.md)

View File

@@ -1,108 +1,108 @@
# AI Provider Guide
# AI Provider 指南
## Overview
## 概览
`aiprovider` is the model-adapter service for Planet.
`aiprovider` 是 Planet 的模型适配服务。
It isolates model-vendor details from the main backend so the rest of the system can call a stable business API:
它把模型厂商差异隔离在主后端之外,让系统其它部分可以调用稳定的业务 API
- Caller service -> `planet backend`
- 调用方服务 -> `planet backend`
- `planet backend` -> `aiprovider`
- `aiprovider` -> concrete model provider
- `aiprovider` -> 具体模型提供方
The recommended default is:
推荐默认方式:
- External and cross-service callers use `planet backend`
- Only infrastructure-grade internal jobs call `aiprovider` directly
- 外部调用方和跨服务调用方统一调用 `planet backend`
- 只有基础设施级内部任务才直接调用 `aiprovider`
## Responsibilities
## 职责边界
`backend` is responsible for:
`backend` 负责:
- authentication and authorization
- business-level request shaping
- stable `/api/v1/ai/...` endpoints
- internal service-to-service authentication toward `aiprovider`
- 身份认证和权限控制
- 业务层请求整理
- 稳定的 `/api/v1/ai/...` 接口
- 面向 `aiprovider` 的内部服务认证
`aiprovider` is responsible for:
`aiprovider` 负责:
- model protocol adaptation
- provider selection by `.env`
- timeout and lightweight retry
- request tracing via `X-Request-ID`
- 模型协议适配
- 基于 `.env` 选择 provider
- 超时和轻量重试
- 通过 `X-Request-ID` 串联请求追踪
This now follows an OpenClaw-like seam:
当前配置采用类似 OpenClaw 的拆分方式:
- `AI_PROVIDER` identifies the vendor or logical provider
- `AI_PROVIDER_API` identifies the wire adapter
- `AI_PROVIDER` 标识厂商或逻辑 provider
- `AI_PROVIDER_API` 标识实际请求协议适配器
That split makes MiniMax, Claude-compatible gateways, and self-hosted OpenAI-compatible services easier to model without overloading one config field.
这个拆分能更清楚地表达 MiniMaxClaude 兼容网关、自托管 OpenAI 兼容服务等情况,避免把所有含义塞进一个配置项。
## Supported Providers
## 支持的 Provider
`aiprovider` currently supports these provider identities:
`aiprovider` 当前支持以下 provider 标识:
- `openai`
- `anthropic`
- `minimax`
- `ollama`
Supported request adapters:
支持的请求适配器:
- `openai-completions`
- `anthropic-messages`
- `ollama-generate`
Backward-compatible aliases still accepted:
仍然兼容的历史别名:
- `openai_compatible`
- `anthropic_compatible`
- `claude_compatible`
Provider mapping:
推荐映射关系:
- `vLLM`, `LM Studio`, `One API`: `AI_PROVIDER=openai`, `AI_PROVIDER_API=openai-completions`
- `MiniMax`: `AI_PROVIDER=minimax`, `AI_PROVIDER_API=anthropic-messages`
- Claude-compatible gateways: `AI_PROVIDER=anthropic`, `AI_PROVIDER_API=anthropic-messages`
- `Ollama`: `AI_PROVIDER=ollama`, `AI_PROVIDER_API=ollama-generate`
- `vLLM``LM Studio``One API``AI_PROVIDER=openai``AI_PROVIDER_API=openai-completions`
- `MiniMax``AI_PROVIDER=minimax``AI_PROVIDER_API=anthropic-messages`
- Claude 兼容网关:`AI_PROVIDER=anthropic``AI_PROVIDER_API=anthropic-messages`
- `Ollama``AI_PROVIDER=ollama``AI_PROVIDER_API=ollama-generate`
## API Surfaces
## API
### Main backend API
### 主后端 API
Preferred stable entrypoints:
推荐使用的稳定入口:
- `GET /api/v1/ai/provider/status`
- `POST /api/v1/ai/situational-awareness/analyze`
Authentication:
认证方式:
- `Authorization: Bearer <jwt>`
Optional tracing header:
可选追踪头:
- `X-Request-ID: <caller-generated-id>`
The backend will propagate `X-Request-ID` to `aiprovider` and return the same header in the response.
后端会把 `X-Request-ID` 透传给 `aiprovider`,并在响应中返回同一个 header。
### AI provider internal API
### AI Provider 内部 API
Internal-only endpoints:
仅供内部调用的接口:
- `GET /v1/provider/status`
- `POST /v1/analyze`
Authentication:
认证方式:
- `X-Provider-Token: <shared-secret>`
Optional tracing header:
可选追踪头:
- `X-Request-ID: <caller-generated-id>`
## Request Example
## 请求示例
### Call through backend
### 通过后端调用
```bash
curl -X POST http://localhost:8000/api/v1/ai/situational-awareness/analyze \
@@ -127,7 +127,7 @@ curl -X POST http://localhost:8000/api/v1/ai/situational-awareness/analyze \
}'
```
### Call `aiprovider` directly
### 直接调用 `aiprovider`
```bash
curl -X POST http://localhost:8010/v1/analyze \
@@ -149,9 +149,9 @@ curl -X POST http://localhost:8010/v1/analyze \
}'
```
## Response Shape
## 响应结构
Both backend and `aiprovider` return the same payload shape:
后端和 `aiprovider` 返回相同的 payload 结构:
```json
{
@@ -166,15 +166,15 @@ Both backend and `aiprovider` return the same payload shape:
}
```
Both services also return:
两个服务都会返回:
- `X-Request-ID: <id>`
## Configuration
## 配置
### Backend
### 后端
Recommended backend `.env`:
推荐的后端 `.env`
```env
AI_PROVIDER_SERVICE_URL=http://localhost:8010
@@ -183,21 +183,21 @@ AI_PROVIDER_TIMEOUT_SECONDS=60
AI_PROVIDER_RETRY_ATTEMPTS=2
```
Reference file:
参考文件:
- [backend/.env.example](/home/ray/dev/linkong/planet/backend/.env.example)
### AI Provider
Reference file:
参考文件:
- [aiprovider/.env.example](/home/ray/dev/linkong/planet/aiprovider/.env.example)
Frontend local reference:
前端本地参考:
- [frontend/.env.example](/home/ray/dev/linkong/planet/frontend/.env.example)
Common settings:
通用配置:
```env
SERVICE_NAME=planet-ai-provider
@@ -208,7 +208,7 @@ AI_HTTP_RETRY_ATTEMPTS=2
AI_ANALYSIS_SYSTEM_PROMPT=你是态势感知分析助手。请基于输入的上下文、观测与约束,输出结构化、克制、可执行的分析。
```
### OpenAI-compatible example
### OpenAI 兼容示例
```env
AI_PROVIDER=openai
@@ -218,7 +218,7 @@ AI_API_KEY=local-key
AI_MODEL=your-local-model
```
### MiniMax CN example
### MiniMax 中国区示例
```env
AI_PROVIDER=minimax
@@ -230,13 +230,13 @@ AI_MAX_TOKENS=1200
AI_ANTHROPIC_VERSION=2023-06-01
```
MiniMax note:
MiniMax 说明:
- This follows the same Anthropic Messages request shape as the official MiniMax examples.
- For MiniMax, `aiprovider` now disables `thinking` by default unless the caller explicitly passes a `thinking` object.
- This mirrors OpenClaw's caution around MiniMax Anthropic-compatible behavior.
- 这里使用官方 MiniMax 示例中的 Anthropic Messages 请求结构。
- MiniMax`aiprovider` 默认不会开启 `thinking`,除非调用方显式传入 `thinking` 对象。
- 这个行为和 OpenClaw 对 MiniMax Anthropic 兼容接口的谨慎处理保持一致。
### Anthropic-compatible example
### Anthropic 兼容示例
```env
AI_PROVIDER=anthropic
@@ -248,7 +248,7 @@ AI_MAX_TOKENS=1200
AI_ANTHROPIC_VERSION=2023-06-01
```
### Ollama example
### Ollama 示例
```env
AI_PROVIDER=ollama
@@ -258,36 +258,36 @@ AI_API_KEY=
AI_MODEL=qwen2.5:7b
```
## Deployment Modes
## 部署模式
### Single machine
### 单机部署
Recommended local flow:
推荐的本地流程:
- `backend` on `localhost:8000`
- `aiprovider` on `localhost:8010`
- local model gateway on `localhost:11434` or another local port
- `backend` 运行在 `localhost:8000`
- `aiprovider` 运行在 `localhost:8010`
- 本地模型网关运行在 `localhost:11434` 或其它本地端口
Helpers already included:
仓库内已包含辅助入口:
- [planet.sh](/home/ray/dev/linkong/planet/planet.sh)
- [docker-compose.local-model.yml](/home/ray/dev/linkong/planet/docker-compose.local-model.yml)
### Multi-machine
### 多机部署
Example topology:
示例拓扑:
- app machine: `backend`
- AI gateway machine: `aiprovider`
- model machine: local model service or cloud proxy
- 应用机器:`backend`
- AI 网关机器:`aiprovider`
- 模型机器:本地模型服务或云代理
In that case, this becomes service-to-service HTTP RPC:
此时链路变成服务间 HTTP RPC
- caller -> backend
- backend -> `http://10.0.0.12:8010`
- `aiprovider` -> model endpoint
- `aiprovider` -> 模型端点
Recommended cross-machine backend config:
推荐的跨机器后端配置:
```env
AI_PROVIDER_SERVICE_URL=http://10.0.0.12:8010
@@ -296,38 +296,38 @@ AI_PROVIDER_TIMEOUT_SECONDS=60
AI_PROVIDER_RETRY_ATTEMPTS=2
```
Recommended operating rules:
推荐运行规则:
- keep `aiprovider` on a private network
- protect it with `X-Provider-Token` at minimum
- always send `X-Request-ID`
- keep callers on the backend API unless they are infrastructure jobs
- `aiprovider` 放在私有网络内
- 至少用 `X-Provider-Token` 保护它
- 始终发送 `X-Request-ID`
- 除基础设施任务外,调用方优先走后端 API
## Retry And Failure Behavior
## 重试和失败行为
`backend -> aiprovider`:
`backend -> aiprovider`
- retries lightweight network / 5xx failures
- returns `502` when the provider service is unavailable
- 对轻量网络错误和 5xx 失败进行重试
- provider 服务不可用时返回 `502`
`aiprovider -> model provider`:
`aiprovider -> model provider`
- retries lightweight network / 5xx failures
- returns `502` when the model provider is unavailable
- 对轻量网络错误和 5xx 失败进行重试
- 模型提供方不可用时返回 `502`
This is intentionally conservative. It avoids masking persistent errors while still absorbing short hiccups.
这个策略故意保持保守:它能吸收短暂抖动,但不会掩盖持续性错误。
## Operational Notes
## 运维说明
- `./planet.sh start` now starts `aiprovider` automatically
- `./planet.sh restart -a` restarts only `aiprovider`
- `./planet.sh log -a` tails `aiprovider` logs
- `./planet.sh health` reports `aiprovider` health
- `./planet.sh start` 会自动启动 `aiprovider`
- `./planet.sh restart -a` 只重启 `aiprovider`
- `./planet.sh log -a` 跟随查看 `aiprovider` 日志
- `./planet.sh health` 会报告 `aiprovider` 健康状态
## Recommended Calling Policy
## 推荐调用策略
- Frontend and application services: call `backend`
- Scheduled infra jobs and diagnostics: optionally call `aiprovider`
- Do not let multiple business services integrate model vendors independently
- 前端和应用服务:调用 `backend`
- 定时基础设施任务和诊断任务:可选直接调用 `aiprovider`
- 不要让多个业务服务分别接入模型厂商
That keeps provider switching centralized and avoids model-specific drift across the system.
这样可以集中管理 provider 切换,避免模型相关差异在系统里四处扩散。

View File

@@ -84,6 +84,8 @@ async def run(self, db):
| HuggingFace Spaces | space | Demo应用 | 1天 |
| PeeringDB | ixp/network/facility | 互联网交换点/网络/机房 | 1-2天 |
| TeleGeography | submarine_cable | 海底光缆信息 | 7天 |
| Space-Track TLE | satellite_tle | 卫星轨道 TLE 数据 | 依采集器配置 |
| BarentsWatch AIS | vessel | 船只位置、航速、航向、MMSI 等 AIS 数据 | 依采集器配置 |
## 四、数据格式 (统一存储到 CollectedData 表)
@@ -200,6 +202,30 @@ def start_scheduler():
**核心文件**: `backend/app/services/scheduler.py`
### 成功采集与连接状态
内置采集器成功采集后,调度器会记录当前有效配置已通过连接验证:
```python
if datasource.last_status == "success":
effective_candidate = await get_builtin_effective_candidate(db, datasource.source)
checksum, _ = await build_builtin_connectivity_checksum(...)
await save_connectivity_success(
db,
datasource.source,
checksum,
{"status_code": None},
connected_by="collection",
)
```
这个记录用于控制台“采集器设置”中的连接状态判断:如果当前配置和成功采集时的 checksum 一致,就视为已连接,不要求用户再手动点击连接按钮。只有 endpoint、请求头、基础配置或凭证指纹变化时才需要重新验证。
相关实现见:
- [datasource_connectivity.py](/home/ray/dev/linkong/planet/backend/app/services/datasource_connectivity.py)
- [数据源、采集器设置与连接验证](/home/ray/dev/linkong/planet/docs/technical/zh/datasource-collector-settings-connectivity.md)
## 八、相关代码文件
```
@@ -211,13 +237,86 @@ backend/app/services/collectors/
├── epoch_ai.py # Epoch AI采集器
├── huggingface.py # HuggingFace采集器
├── peeringdb.py # PeeringDB采集器
── telegeraphy.py # TeleGeography海底光缆采集器
── telegeraphy.py # TeleGeography海底光缆采集器
└── vessel_ais.py # BarentsWatch AIS 船只采集器
backend/app/models/
└── collected_data.py # 统一数据模型
```
## 九、数据使用场景
## 九、凭证型采集器
部分采集器需要外部服务凭证,例如:
| 采集器 | credential provider | 凭证来源 |
| --- | --- | --- |
| `barentswatch_vessels` | `barentswatch` | 控制台采集器设置、环境变量、`~/.zshrc` |
| `spacetrack_tle` | `spacetrack` | 环境变量、`~/.zshrc` |
### BarentsWatch AIS
BarentsWatch AIS 的凭证解析统一在:
- [barentswatch.py](/home/ray/dev/linkong/planet/backend/app/services/barentswatch.py)
`VesselAISCollector` 只负责采集和转换 AIS 数据,不再自己读取环境变量或拼 token 请求。它通过:
- `resolve_barentswatch_config()`
- `fetch_barentswatch_access_token()`
获取运行时配置。
解析优先级:
1. `DataSourceConfig.auth_config`
2. `DataSourceConfig.config`
3. 环境变量
4. `~/.zshrc`
支持变量:
```bash
export BARENTSWATCH_CLIENT_ID="..."
export BARENTSWATCH_CLIENT_SECRET="..."
```
并兼容历史拼写:
```bash
export BARRENTSWATCH_CLIENT_ID="..."
export BARRENTSWATCH_CLIENT_SECRET="..."
```
连接验证会先请求 `https://id.barentswatch.no/connect/token` 获取 `scope=ais` 的 access token再用 `Authorization: Bearer <token>` 请求 AIS endpoint。
## 十、采集器设置与连接验证
控制台的“采集器设置”页提供所有内置采集器的 endpoint、请求头、超时、重试和凭证配置。连接验证不是只看前端按钮状态而是由后端计算 checksum
- endpoint
- auth type
- headers
- config
- credential provider
- 凭证指纹
相关 API
```http
GET /api/v1/datasources/configs/all
POST /api/v1/datasources/configs/builtin/connection-status
POST /api/v1/datasources/configs/builtin/connect
POST /api/v1/settings/integrations/barentswatch/connect
GET /api/v1/settings/credential-guides/{provider}
POST /api/v1/settings/credential-guides/{provider}/generate
POST /api/v1/settings/credential-guides/{provider}/reset
```
更多细节见:
- [数据源、采集器设置与连接验证](/home/ray/dev/linkong/planet/docs/technical/zh/datasource-collector-settings-connectivity.md)
## 十一、数据使用场景
采集的数据最终会:
@@ -225,7 +324,7 @@ backend/app/models/
2. **态势分析** - 统计全球算力分布、增长趋势
3. **告警系统** - 检测重要节点变化
## 十、采集器注册机制
## 十、采集器注册机制
采集器在应用启动时自动注册:
@@ -247,7 +346,7 @@ collector_registry.register(TeleGeographyCableSystemCollector())
**核心文件**: `backend/app/services/collectors/registry.py`
## 十、触发采集
## 十、触发采集
### 方式一:定时触发
系统启动时APScheduler会自动根据各采集器的`frequency_hours`设置定时任务。

View File

@@ -0,0 +1,99 @@
# DataSources 列表接口性能优化
## 背景
`GET /api/v1/datasources` 是数据源管理页面的核心接口,响应慢会直接阻塞页面渲染。
## 优化前的查询链路
`_load_datasource_list_context` 按顺序执行以下查询:
| 序号 | 函数 | 查询内容 | 瓶颈 |
|------|------|---------|------|
| 1 | `_load_latest_running_tasks` | collection_tasks 窗口函数stale check 依赖此结果 | 必须串行 |
| 2 | `_load_latest_completed_tasks` | collection_tasks 窗口函数(最近完成任务) | 串行等待 |
| 3 | `_load_datasource_data_counts` | `COUNT(*) GROUP BY source` on collected_data | **最慢,全表扫描** |
| 4 | `_load_datasource_endpoint_overrides` | datasource_configs 简单 SELECT | 串行等待 |
## 第一阶段:并行化
将 2/3/4 三个互不依赖的查询改为 `asyncio.gather` + 独立 session 并行执行:
```python
async def _fetch_completed():
async with async_session_factory() as s:
return await _load_latest_completed_tasks(s, datasource_ids)
async def _fetch_counts():
async with async_session_factory() as s:
return await _load_datasource_data_counts(s, sources)
async def _fetch_overrides():
async with async_session_factory() as s:
return await _load_datasource_endpoint_overrides(s, sources)
completed_tasks, data_counts, endpoint_overrides = await asyncio.gather(
_fetch_completed(), _fetch_counts(), _fetch_overrides(),
)
```
> **注意**SQLAlchemy `AsyncSession` 不支持在同一 session 上并发,每个协程必须独立开 session。
## 第二阶段:删除重量级查询
### 删除 `_load_datasource_data_counts`
`data_count` 字段仅用于前端在"最近采集"列显示 `(0条)` 的边缘提示,不值得为此维持一次 `COUNT(*) GROUP BY` 全表扫描。
- 前端同步移除 `(0条)` 显示逻辑
- 移除 `BuiltInDataSource` 接口中的 `data_count` 字段
### 删除 `_load_latest_completed_tasks`
`last_status``last_run_at` 已由 collector 在任务完成时直接更新到 `DataSource` 模型字段,不需要再 JOIN collection_tasks 获取:
```python
# 优化前:需要查 completed_tasks
last_run_at = datasource.last_run_at or (last_task.completed_at if last_task else None)
last_status = datasource.last_status or (last_task.status if last_task else None)
# 优化后:直接读模型字段
last_run_at = datasource.last_run_at
last_status = datasource.last_status
```
同步移除 `last_records_processed` 字段(来源是 completed_tasks列表不显示此字段
## 优化后的查询链路
```
datasources SELECT → 主数据,必须
_load_latest_running_tasks → 必须(进行中状态 + stale check
_load_datasource_endpoint_overrides → 必须endpoint 覆盖,列表详情和采集器设置需要显示当前有效地址)
```
3 个查询(原来 5 个后两个顺序执行running tasks 先完成用于 stale checkendpoint overrides 轻量)。
## 前端 triggerDatasource 双调修复
`triggerDatasource` 中存在双重 `fetchData()` 调用:
```typescript
// 修复前
} else {
window.setTimeout(() => { fetchData() }, 800) // 无 task_id 时延迟刷
}
fetchData() // 总是立即刷 → 与上面的延迟刷重叠
// 修复后(二者互斥)
if (res.data.task_id) {
fetchData() // 有 task_id立即刷一次
} else {
window.setTimeout(fetchData, 800) // 无 task_id等 800ms 再刷一次
}
```
## 相关文件
- `backend/app/api/v1/datasources.py``_load_datasource_list_context``list_datasources`
- `frontend/src/pages/DataSources/DataSources.tsx``BuiltInDataSource` interface、`triggerDatasource`

View File

@@ -1,61 +1,57 @@
# System Service Control
# 系统服务控制
This document defines the fixed mapping between admin control-plane actions and
the existing `planet.sh` service-management commands.
本文定义后台控制面动作与现有 `planet.sh` 服务管理命令之间的固定映射。
The goal is to reuse the current operational script semantics without exposing
arbitrary shell execution to the frontend or API callers.
目标是在复用当前运维脚本语义的同时,不向前端或 API 调用方暴露任意 shell 执行能力。
## Scope
## 范围
- This mapping is for admin-side operational controls only.
- The control plane must submit a fixed action name, not a raw shell command.
- The backend is responsible for translating an allowed action into a fixed
`planet.sh` invocation.
- 这套映射只用于管理端运维控制。
- 控制面必须提交固定 action 名称,而不是原始 shell 命令。
- 后端负责把允许的 action 翻译成固定的 `planet.sh` 调用。
## Design Rules
## 设计规则
- Only whitelist actions may be executed.
- The frontend must never send arbitrary shell strings.
- The backend must build command arguments from a fixed mapping table.
- High-risk actions should be restricted to `super_admin`.
- Prefer partial restarts over full-stack restarts when UI continuity matters.
- 只允许执行白名单 action。
- 前端绝不能发送任意 shell 字符串。
- 后端必须从固定映射表构造命令参数。
- 高风险 action 应限制为 `super_admin`
- 在 UI 连续性重要时,优先局部重启,而不是全栈重启。
## Action Mapping
## Action 映射
| Action name | Intended use | `planet.sh` command | Notes |
| Action 名称 | 用途 | `planet.sh` 命令 | 备注 |
| --- | --- | --- | --- |
| `restart-backend` | Restart backend API only | `./planet.sh restart -b` | Recommended first implementation for UI-triggered restart flows. |
| `restart-database` | Restart PostgreSQL and Redis containers | `./planet.sh restart -d` | Useful when database/cache services need a controlled bounce without restarting the UI. |
| `restart-system` | Restart the whole application stack | `./planet.sh restart` | Frontend continuity breaks briefly; UI should switch to guided recovery mode. |
| `restart-frontend` | Restart frontend dev server only | `./planet.sh restart -f` | Use with caution; UI continuity is weaker than backend-only restart. |
| `restart-backend-port` | Restart backend on a specific port | `./planet.sh restart -b <port>` | Port must be backend-validated before execution. |
| `restart-frontend-port` | Restart frontend on a specific port | `./planet.sh restart -f <port>` | Port must be backend-validated before execution. |
| `health-check` | Read current service health | `./planet.sh health` | Safe read-only operational action. |
| `show-logs-backend` | Inspect backend logs | `./planet.sh log -b` | Best used for CLI/operator tooling, not normal Web UI streaming. |
| `show-logs-frontend` | Inspect frontend logs | `./planet.sh log -f` | Best used for CLI/operator tooling, not normal Web UI streaming. |
| `restart-backend` | 只重启后端 API | `./planet.sh restart -b` | 页面通常短暂失联后由 `/health` 轮询恢复。 |
| `restart-frontend` | 只重启前端开发服务器 | `./planet.sh restart -f` | 页面入口会短暂不可用UI 通过前端入口探测恢复后刷新。 |
| `restart-database` | 重启 PostgreSQL 和 Redis 容器 | `./planet.sh restart -d` | 适合数据库/缓存需要受控重启但不希望重启 UI 的场景。 |
| `restart-system` | 重启整个应用栈 | `./planet.sh restart` | 前端会短暂中断UI 应进入引导恢复模式。 |
| `restart-backend-port` | 在指定端口重启后端 | `./planet.sh restart -b <port>` | 执行前必须由后端校验端口。 |
| `restart-frontend-port` | 在指定端口重启前端 | `./planet.sh restart -f <port>` | 执行前必须由后端校验端口。 |
| `health-check` | 读取当前服务健康状态 | `./planet.sh health` | 安全的只读运维动作。 |
| `show-logs-backend` | 查看后端日志 | `./planet.sh log -b` | 更适合 CLI/运维工具,不建议作为普通 Web UI 日志流。 |
| `show-logs-frontend` | 查看前端日志 | `./planet.sh log -f` | 更适合 CLI/运维工具,不建议作为普通 Web UI 日志流。 |
## Not Exposed In UI By Default
## 默认不暴露到 UI 的能力
The following existing script capabilities should not be exposed directly in the
Web UI unless there is an explicit product need and an additional safety review:
除非有明确产品需求并经过额外安全评审,否则以下脚本能力不应直接暴露到 Web UI
- `./planet.sh restart`
- `./planet.sh start`
- `./planet.sh stop`
- `./planet.sh createuser`
- any future raw shell passthrough
- 任何未来的原始 shell 透传能力
Reason:
原因:
- full restart can break the current control session;
- stop/start have larger blast radius;
- user creation is not a service-control operation;
- raw shell passthrough creates unnecessary privilege risk.
- 全量重启可能打断当前控制会话;
- stop/start 影响面更大;
- 用户创建不是服务控制操作;
- 原始 shell 透传会引入不必要的权限风险。
## Recommended First-Phase UI Contract
## 第一阶段推荐 UI 契约
### Frontend action payload
### 前端 action payload
```json
{
@@ -63,7 +59,7 @@ Reason:
}
```
### Backend command resolution
### 后端命令解析
```text
restart-backend -> ["./planet.sh", "restart", "-b"]
@@ -73,19 +69,19 @@ restart-frontend -> ["./planet.sh", "restart", "-f"]
health-check -> ["./planet.sh", "health"]
```
## API Draft
## API 草案
### Primary Endpoint
### 主接口
- `POST /api/v1/system/restart-tasks`
Purpose:
用途:
- create a controlled restart task;
- resolve a whitelist action into a fixed `planet.sh` command;
- hand execution off to an external runner or detached subprocess.
- 创建受控重启任务;
- 将白名单 action 解析成固定 `planet.sh` 命令;
- 把执行交给外部 runner detached subprocess
### Request Body
### 请求体
```json
{
@@ -93,7 +89,7 @@ Purpose:
}
```
Optional future shape:
未来可选形态:
```json
{
@@ -102,7 +98,7 @@ Optional future shape:
}
```
### Response
### 响应
```json
{
@@ -114,11 +110,11 @@ Optional future shape:
}
```
### Task Query Endpoint
### 任务查询接口
- `GET /api/v1/system/restart-tasks/{task_id}`
Response shape:
响应结构:
```json
{
@@ -136,11 +132,11 @@ Response shape:
}
```
### Optional Log Endpoint
### 可选日志接口
- `GET /api/v1/system/restart-tasks/{task_id}/logs`
Suggested response:
建议响应:
```json
{
@@ -154,10 +150,9 @@ Suggested response:
}
```
This log endpoint is optional for phase one. The first version can work with
task state plus `/health` polling alone.
日志接口在第一阶段不是必需项。首版可以只依赖任务状态加 `/health` 轮询。
## Task State Model
## 任务状态模型
### Status
@@ -177,35 +172,32 @@ task state plus `/health` polling alone.
- `healthy`
- `failed`
### Interpretation
### 含义
- `status` is the high-level terminal or non-terminal state.
- `stage` is the operator-facing execution phase for the UI.
- `message` is the short human-readable line shown in the modal or full-screen
overlay.
- `status` 是高层终态/非终态状态。
- `stage` 是面向运维人员和 UI 的执行阶段。
- `message` 是 modal 或全屏遮罩中展示的短文本。
## Permission Model
## 权限模型
- `restart-backend` should require `super_admin`.
- Permission checks should follow the same role pattern already used in
[users.py](/home/ray/dev/linkong/planet/backend/app/api/v1/users.py).
- Frontend visibility may hide controls for non-`super_admin`, but backend must
still enforce authorization.
- `restart-backend` 应要求 `super_admin`
- 权限检查应沿用 [users.py](/home/ray/dev/linkong/planet/backend/app/api/v1/users.py) 中已有的角色模式。
- 前端可以对非 `super_admin` 隐藏控件,但后端必须继续强制鉴权。
## Storage Model
## 存储模型
Recommended first implementation:
推荐第一阶段实现:
- store restart task state in Redis;
- keep task lifetime short;
- keep recent logs as a bounded list.
- 将重启任务状态存入 Redis
- 任务生命周期保持较短;
- 最近日志用有界列表保存。
Suggested keys:
建议 key
- `system:restart_task:{task_id}`
- `system:restart_task:{task_id}:logs`
Suggested stored fields:
建议字段:
- `task_id`
- `action`
@@ -217,22 +209,21 @@ Suggested stored fields:
- `created_at`
- `updated_at`
## Execution Model
## 执行模型
The request-handling API process should not depend on itself surviving long
enough to stream the whole restart output.
处理请求的 API 进程不应依赖自身持续存活来流式输出完整重启日志。
Recommended execution flow:
推荐执行流程:
1. validate caller and action
2. create task state in Redis
3. resolve action to fixed `planet.sh` argv
4. spawn detached executor
5. return `task_id`
6. executor updates task state while restart is in progress
7. frontend polls health and/or task state until recovery
1. 校验调用方和 action
2. 在 Redis 中创建任务状态
3. action 解析为固定 `planet.sh` argv
4. 启动 detached executor
5. 返回 `task_id`
6. executor 在重启过程中更新任务状态
7. 前端轮询健康状态和/或任务状态,直到服务恢复
Recommended command resolution examples:
推荐命令解析示例:
```text
restart-backend -> ["./planet.sh", "restart", "-b"]
@@ -241,25 +232,25 @@ restart-backend-port -> ["./planet.sh", "restart", "-b", "<port>"]
health-check -> ["./planet.sh", "health"]
```
## Frontend Polling Flow
## 前端轮询流程
Recommended first-phase UX:
推荐第一阶段 UX
1. user clicks `重启后端`
2. confirmation modal explains temporary unavailability
3. frontend calls `POST /api/v1/system/restart-tasks`
4. UI enters blocking restart state
5. frontend polls `/health` every `1-2s`
6. temporary request failures are treated as expected
7. after `2-3` consecutive successful health checks, frontend reloads page
1. 用户点击 `重启后端`
2. 确认 modal 说明服务会短暂不可用
3. 前端调用 `POST /api/v1/system/restart-tasks`
4. UI 进入阻塞式重启状态
5. 前端每 `1-2s` 轮询 `/health`
6. 临时请求失败视为预期现象
7. 连续 `2-3` 次健康检查成功后,前端刷新页面
Optional richer polling:
可选增强轮询:
1. poll task status endpoint while backend is still reachable
2. switch to `/health` recovery polling after disconnect begins
3. refresh page after health recovery
1. 后端仍可达时轮询任务状态接口
2. 断连开始后切换为 `/health` 恢复轮询
3. 健康恢复后刷新页面
## Frontend State Machine
## 前端状态机
- `idle`
- `confirming`
@@ -270,7 +261,7 @@ Optional richer polling:
- `failed`
- `timeout`
Suggested UI messages:
建议 UI 文案:
- `已发送重启指令`
- `正在停止后端服务`
@@ -278,70 +269,65 @@ Suggested UI messages:
- `服务已恢复,正在刷新页面`
- `恢复超时,请手动检查服务状态`
## Phase-One Recommendation
## 当前 Dashboard 实现
Implement only the following in phase one:
Dashboard 当前已实现:
- `restart-backend`
- `super_admin` permission gate
- task creation endpoint
- Redis-backed task state
- frontend confirmation modal
- frontend `/health` polling
- automatic page reload after recovery
- `restart-frontend`
- `restart-ai-provider`
- `restart-database`
- `restart-system`
- `super_admin` 权限门禁
- 任务创建接口
- Redis 任务状态
- 前端确认 modal
- 后端 `/health` 轮询
- 前端入口轮询
- 恢复后自动刷新页面
Do not implement in phase one:
暂不实现:
- full `./planet.sh restart`
- raw shell command passthrough
- arbitrary service control
- full terminal stdout streaming
- multi-action concurrent restart queueing
- 原始 shell 命令透传
- 任意服务控制
- 完整终端 stdout 流式输出
- 多 action 并发重启队列
## Implementation Checklist
## 实现清单
### Backend
### 后端
1. add a dedicated system-control API module under `backend/app/api/v1/`
2. add a whitelist-based action resolver for `planet.sh`
3. store restart task state in Redis
4. add detached restart-runner script execution
5. expose:
1. `backend/app/api/v1/` 下新增专用系统控制 API 模块
2. 增加基于白名单的 `planet.sh` action 解析器
3. 将重启任务状态存入 Redis
4. 增加 detached restart-runner 脚本执行
5. 暴露:
- `POST /api/v1/system/restart-tasks`
- `GET /api/v1/system/restart-tasks/{task_id}`
- optional task log endpoint
6. enforce `super_admin` permission on all restart-task endpoints
- 可选任务日志接口
6. 对所有 restart-task 接口强制 `super_admin` 权限
### Frontend
### 前端
1. add a `重启后端` control on the dashboard for `super_admin`
2. show a confirmation modal before dispatch
3. after submission, switch modal into blocking restart state
4. poll `/health` until backend recovery is confirmed
5. auto-refresh page after consecutive successful health checks
6. show short stage-oriented logs instead of raw terminal streaming
1. dashboard `super_admin` 增加 `重启服务` 控件
2. 发送前展示确认 modal
3. 提交后将 modal 切换为阻塞式重启状态
4. 后端重启使用 `/health` 轮询确认恢复
5. 前端重启和完全重启使用前端入口探测确认恢复
6. 连续健康检查成功后自动刷新页面
7. 展示简短阶段日志,而不是原始终端流
### Operational Notes
### 运维说明
1. phase one should target backend-only restart
2. frontend restart should remain out of scope initially
3. command execution must always originate from repository root
4. only fixed action names may cross the API boundary
1. 优先使用局部重启,只有确实需要时才执行完全重启
2. 前端重启会打断当前页面入口,必须进入恢复等待状态
3. 命令执行必须始终从仓库根目录发起
4. API 边界只能传递固定 action 名称
## Validation Requirements
## 校验要求
- Reject any action not present in the whitelist.
- If a port-bearing action is added, validate the port as an integer in
`1..65535`.
- Resolve commands from the repository root so `planet.sh` runs with a stable
working directory.
- Record the requested action, operator identity, execution start time, and
result.
## Implementation Guidance
- For UI-triggered restart flows, prefer `restart-backend` first.
- Do not rely on the current API request process to stream full restart output
after it triggers its own restart.
- Use a task record plus polling/health-check recovery flow instead of raw
terminal streaming as the primary UX.
- 拒绝任何不在白名单中的 action。
- 如果增加带端口 action端口必须校验为 `1..65535` 的整数。
- 从仓库根目录解析命令,确保 `planet.sh` 的工作目录稳定。
- detached runner 使用 `zsh -ic` 执行白名单命令,确保 `~/.zshrc` 中的本地环境变量进入重启流程。
- 记录请求 action、操作者身份、执行开始时间和结果。

View File

@@ -0,0 +1,327 @@
# 采集器设置与连接验证
## 背景
控制台现在把“数据源目录”和“采集器配置”拆开:
- `/datasources`
- 展示所有数据源,包括内置和自定义。
- 点击名称只打开信息抽屉。
- 负责查看状态、触发采集和查看采集中任务。
- `/settings?tab=collector_credentials`
- 显示为“采集器设置”。
- 负责 endpoint、请求头、基础参数和凭证配置。
- 所有采集器都提供连接按钮,用于健康检查。
这样做是为了减少首次使用时的认知分裂:接口地址、请求头、凭证和自定义源配置都属于“采集器设置”,而不是散落在数据源列表和系统设置多个入口里。
## 用户侧规则
连接状态不是前端样式状态,而是由后端根据配置 checksum 和已验证记录判断。
一个内置采集器被视为“已连接”需要满足任一条件:
- 当前配置已经成功采集过数据。
- 当前配置点击过连接按钮,并且后端验证成功。
如果 endpoint、请求头、基础配置或凭证指纹相对上次验证成功时发生变化状态会回到“需要重新连接”。
## 前端入口
### 数据源目录
文件:
- [DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/DataSources/DataSources.tsx)
- [index.css](/home/ray/dev/linkong/planet/frontend/src/index.css)
当前行为:
- 内置数据源和自定义数据源合并为 `UnifiedDataSource` 列表。
- 表格只保留查看、采集和状态类操作。
- 名称点击打开只读抽屉。
- 抽屉中展示:
- 是否内置
- 是否启用
- 模块、优先级、频率
- endpoint
- 请求头
- 基础配置
- 是否需要凭证
- 有任务运行时,顶部进度区显示 `采集中 N` 可点击标签。
- 点击 `采集中 N` 打开任务列表弹窗,显示各任务进度。
`data-source-bulk-toolbar__running-pill` 是“采集中”标签的样式入口。它和其他状态标签同排,但通过 hover、箭头和蓝色描边表达可交互性。
### 采集器设置
文件:
- [Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Settings/Settings.tsx)
当前行为:
- `collector_credentials` tab 展示为“采集器设置”。
- 下拉框列出所有内置采集器。
- 下拉框右侧只有一个插头图标按钮,用于健康检查。
- 下拉框下方用状态标签展示:
- `需要凭证` / `无需凭证`
- 模块
- `启用` / `禁用`
- `未检查` / `可用` / `不可用`
- 是否覆盖 endpoint
- 需要凭证的采集器把凭证卡片放在基础配置上方。
- 不需要凭证的采集器只显示基础配置。
连接按钮使用内联 Tabler 风格插头图标,来源语义对应 `plug-connected`,避免继续使用刷新图标表达连接动作。
## 后端接口
### 数据源配置列表
```http
GET /api/v1/datasources/configs/all
```
返回 YAML 默认数据源和数据库覆盖配置的合并结果。该路由必须定义在 `/configs/{config_id}` 之前,否则 `all` 会被 FastAPI 当成路径参数并触发 422。
返回字段包括:
- `name`
- `default_url`
- `endpoint`
- `is_overridden`
- `is_active`
- `source_type`
- `auth_type`
- `headers`
- `config`
- `config_id`
- `description`
`config` 返回前会移除内部连接验证字段,避免前端把校验元数据当成用户配置展示。
### 内置采集器连接状态
```http
POST /api/v1/datasources/configs/builtin/connection-status
```
用途:
- 给定一份候选配置。
- 计算 checksum。
- 判断当前配置是否已经连接。
当前前端主要通过连接按钮即时检查,不强依赖这个接口,但它是后续保存按钮置灰、页面初始化状态恢复的后端依据。
### 内置采集器连接验证
```http
POST /api/v1/datasources/configs/builtin/connect
```
用途:
- 免费采集器直接请求 endpoint。
- 需要凭证的采集器走对应 credential provider。
- 验证成功后写入系统级连接记录。
成功返回中会带:
- `success`
- `connected`
- `checksum`
- `stage`
- `message`
- `response_time_ms`
- `credential_provider`
- `credential_source`
### BarentsWatch AIS 连接验证
```http
POST /api/v1/settings/integrations/barentswatch/connect
GET /api/v1/settings/integrations/barentswatch/connectivity
```
BarentsWatch 使用独立接口,是因为它需要在保存前验证草稿凭证:
- 使用草稿 `client_id` / `client_secret` 获取 token。
- 使用 token 请求 AIS endpoint。
- 连接成功后用草稿凭证指纹写入内置采集器连接记录。
## 连接校验服务
文件:
- [datasource_connectivity.py](/home/ray/dev/linkong/planet/backend/app/services/datasource_connectivity.py)
核心职责:
- 计算内置采集器配置 checksum。
- 读取环境变量和 `~/.zshrc` 中的凭证。
- 判断当前配置是否已连接。
- 执行 endpoint 健康检查。
- 保存连接成功记录。
### checksum 组成
checksum 包含:
- 采集器名称
- endpoint
- auth type
- headers
- 去掉内部校验字段后的 config
- credential provider
- 凭证指纹
凭证指纹使用凭证内容 hash不把明文凭证写入连接记录。
### 连接记录
连接成功记录写入 `SystemSetting`
```text
category = datasource_connectivity_validations
```
payload 以采集器 source 为 key
```json
{
"barentswatch_vessels": {
"checksum": "...",
"status": "success",
"validated_at": "2026-04-29T00:00:00+00:00",
"status_code": 200,
"credential_source": "datasource_config",
"connected_by": "connection_button"
}
}
```
`connected_by` 当前有两个来源:
- `connection_button`
- 用户手动点击连接按钮。
- `collection`
- 采集任务成功完成,系统自动记录当前有效配置已连通。
### 成功采集即连接
调度器在采集成功后会调用连接记录写入逻辑:
- [scheduler.py](/home/ray/dev/linkong/planet/backend/app/services/scheduler.py)
这样已有数据的采集器不会要求用户重复验证。只有当配置 checksum 变化时,才需要重新点击连接。
## BarentsWatch AIS 凭证链路
文件:
- [barentswatch.py](/home/ray/dev/linkong/planet/backend/app/services/barentswatch.py)
- [vessel_ais.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/vessel_ais.py)
解析优先级:
1. `DataSourceConfig.auth_config`
2. `DataSourceConfig.config`
3. 环境变量
4. `~/.zshrc`
支持的环境变量:
```bash
export BARENTSWATCH_CLIENT_ID="..."
export BARENTSWATCH_CLIENT_SECRET="..."
```
也兼容历史拼写:
```bash
export BARRENTSWATCH_CLIENT_ID="..."
export BARRENTSWATCH_CLIENT_SECRET="..."
```
Token 请求规则:
- Token URL`https://id.barentswatch.no/connect/token`
- `Content-Type`: `application/x-www-form-urlencoded`
- Body:
- `grant_type=client_credentials`
- `client_id`
- `client_secret`
- `scope=ais`
AIS 请求规则:
- Endpoint 默认:`https://live.ais.barentswatch.no/v1/latest/combined`
- Header`Authorization: Bearer <access_token>`
`VesselAISCollector` 不再自己读取环境变量,而是统一走 `resolve_barentswatch_config()``fetch_barentswatch_access_token()`,避免设置页、连接验证和采集器三套凭证逻辑分叉。
## 凭证教程
文件:
- [credential_guides.py](/home/ray/dev/linkong/planet/backend/app/services/credential_guides.py)
接口:
```http
GET /api/v1/settings/credential-guides/{provider}
POST /api/v1/settings/credential-guides/{provider}/generate
POST /api/v1/settings/credential-guides/{provider}/reset
```
当前支持:
- `barentswatch`
默认教程包含 BarentsWatch 官方 tutorial 地址:
```text
https://developer.barentswatch.no/docs/tutorial
```
如果用户点击“教程不好用”,后端会把默认 prompt 发给 AI Provider 生成新的中文教程,并保存到 `SystemSetting`
```text
category = collector_credential_guides
```
“重置”会删除自定义教程,恢复默认教程。
## 保存规则
内置采集器配置保存时会移除内部 `connectivity_validation` 字段,避免校验状态跟用户配置混在一起。
BarentsWatch `client_secret` 保存时有特殊处理:
- 输入框显示脱敏预览。
- 如果提交值仍等于脱敏预览,后端保留原 secret。
- 如果输入新值,才替换 secret。
- 不再提供单独“清除当前 secret”复选框。
## 测试覆盖
相关测试:
- [test_vessels.py](/home/ray/dev/linkong/planet/backend/tests/test_vessels.py)
新增覆盖:
- 能从 `~/.zshrc` 解析 BarentsWatch 凭证。
- 环境变量为空时,`resolve_barentswatch_config()` 能回退到 `~/.zshrc`
- 船只数据转换和 GeoJSON 输出保持兼容。
## 当前 Provider 覆盖
当前已经支持的凭证 provider
- `barentswatch`
- `spacetrack`
其他 `requires_credentials=true` 的采集器如果还没有 provider会返回“凭证链路尚未接入”前端显示 `不可用`

View File

@@ -1,31 +1,31 @@
# BGP Context
# BGP 态势上下文
## Current Goal
## 当前目标
The BGP module is being evolved from an anomaly-only demo into a layered observability pipeline:
BGP 模块正在从一个只展示异常的演示功能,演进为分层观测管线:
`raw observations -> enrichment -> detectors -> incidents -> console/Earth visualization`
The practical product goal is no longer just to "show incidents on the globe". The current product objective is:
实际产品目标已经不只是“在地球上显示事件”。当前目标是:
1. keep BGP visually present on Earth even when incident density is low
2. make incidents clearly feel like a higher-confidence layer than anomalies
3. show that the observation network is still active even when there are no active incidents
1. 即使 incident 密度很低,也让 BGP 在 Earth 上保持可见存在感
2. incident 明显比 anomaly 更像高置信度事件层
3. 即使没有活跃 incident也能表达观测网络仍在运行
In practice, that means Earth should behave like an observability surface, not only an incident map:
换句话说Earth 应该表现为观测面,而不只是事件地图:
- `collectors` show that observation is happening
- `activity` shows where routing state is currently active or noisy
- `incidents` become the highest-confidence focus layer
- `collectors` 表达观测正在发生
- `activity` 表达哪里的路由状态近期活跃或噪声较高
- `incidents` 成为最高置信度的聚焦层
## Current Backend Architecture
## 当前后端架构
### Data Layers
### 数据层
1. `BGPObservation`
- File: `backend/app/models/bgp_observation.py`
- Purpose: store normalized raw routing observations from live/history sources.
- Typical fields:
- 文件:`backend/app/models/bgp_observation.py`
- 用途:存储从实时/历史来源归一化后的原始路由观测。
- 典型字段:
- `source`
- `collector`
- `peer_asn`
@@ -42,80 +42,80 @@ In practice, that means Earth should behave like an observability surface, not o
- `ingest_batch_id`
2. `BGPAnomaly`
- File: `backend/app/models/bgp_anomaly.py`
- Purpose: hold atomic detector outputs.
- Current detector output types include:
- 文件:`backend/app/models/bgp_anomaly.py`
- 用途:保存原子级 detector 输出。
- 当前 detector 输出类型包括:
- `origin_change`
- `more_specific_burst`
- `mass_withdrawal`
3. `BGPIncident`
- File: `backend/app/models/bgp_incident.py`
- Purpose: aggregate atomic anomalies into incident-level objects for humans and the UI.
- 文件:`backend/app/models/bgp_incident.py`
- 用途:把原子 anomaly 聚合成人类和 UI 可消费的 incident 对象。
### Pipeline
### 管线
Main flow is currently anchored in:
主流程目前集中在:
- `backend/app/services/collectors/bgp_common.py`
- `backend/app/services/bgp_enrichment.py`
- `backend/app/services/bgp_detectors.py`
- `backend/app/services/bgp_incidents.py`
Operational flow:
运行流程:
1. collectors fetch raw BGP data
2. `normalize_bgp_event()` standardizes payloads
3. observations are persisted to `bgp_observations`
4. enrichment augments events with analysis context
5. detectors create `bgp_anomalies`
6. incident aggregation rolls anomalies up into `bgp_incidents`
1. 采集器抓取原始 BGP 数据
2. `normalize_bgp_event()` 规范化 payload
3. observation 写入 `bgp_observations`
4. enrichment 为事件补充分析上下文
5. detector 创建 `bgp_anomalies`
6. incident 聚合把 anomaly 汇总为 `bgp_incidents`
### Current Ingest Sources
### 当前接入来源
1. `RIPE RIS Live`
- Collector file: `backend/app/services/collectors/ris_live.py`
- Used for realtime observation flow.
- 采集器文件:`backend/app/services/collectors/ris_live.py`
- 用于实时观测流。
2. `CAIDA BGPStream Backfill`
- Collector file: `backend/app/services/collectors/bgpstream.py`
- Used as history/backfill entry point.
- 采集器文件:`backend/app/services/collectors/bgpstream.py`
- 用作历史/回填入口。
## Current Enrichment Status
## 当前 enrichment 状态
Implemented enrichment skeleton in:
已在以下文件实现 enrichment 骨架:
- `backend/app/services/bgp_enrichment.py`
Current enrichments:
当前 enrichment 内容:
- prefix family / prefix length
- supernet / more-specific derivation
- deduplicated AS path
- path prepending hints
- collector region info
- prefix baseline hints
- new-origin detection
- ASN organization profile from PeeringDB where available
- prefix scope / impacted region hints
- prefix geography source priority:
- `OpenGeoFeed` (override/high confidence)
- `IPtoASN` (country-range baseline)
- `NRO delegated stats` (registry-allocation fallback)
- supernet / more-specific 推导
- 去重 AS path
- path prepending 提示
- collector 区域信息
- prefix baseline 提示
- new-origin 检测
- 可用时从 PeeringDB 获取 ASN 组织画像
- prefix scope / 受影响区域提示
- prefix 地理来源优先级:
- `OpenGeoFeed`override,高置信)
- `IPtoASN`(国家范围 baseline
- `NRO delegated stats`registry allocation fallback
Current limitation:
当前限制:
- `RPKI` is still placeholder-only and returns `unknown`
- no real ROA validation source is integrated yet
- `inetnum` / `inet6num` whois fallback is still pending
- `RPKI` 仍只是占位,返回 `unknown`
- 尚未集成真实 ROA 校验来源
- `inetnum` / `inet6num` whois fallback 仍待实现
## Current API Surface
## 当前 API 面
Primary API file:
主 API 文件:
- `backend/app/api/v1/bgp.py`
Available endpoints:
可用接口:
- `/api/v1/bgp/events`
- `/api/v1/bgp/events/summary`
@@ -127,16 +127,16 @@ Available endpoints:
- `/api/v1/bgp/incidents/summary`
- `/api/v1/bgp/incidents/{id}`
Visualization GeoJSON endpoints:
可视化 GeoJSON 接口:
- `backend/app/api/v1/visualization.py`
- `/api/v1/visualization/geo/bgp-collectors`
- `/api/v1/visualization/geo/bgp-anomalies`
- `/api/v1/visualization/geo/bgp-incidents`
## Current Earth Behavior
## 当前 Earth 行为
Relevant files:
相关文件:
- `frontend/public/earth/js/bgp.js`
- `frontend/public/earth/js/main.js`
@@ -144,56 +144,56 @@ Relevant files:
- `frontend/public/earth/js/constants.js`
- `frontend/public/earth/index.html`
Current design:
当前设计:
1. Collectors are always shown when BGP is enabled.
2. Incident markers are now the primary Earth BGP markers.
3. If there are no incidents, Earth falls back to anomaly markers.
4. If there are no anomalies either, collectors still provide presence.
5. A dedicated `activity layer` now adds:
- per-collector recent 15-minute activity halos
- clustered regional activity hints derived from active collectors
6. Incident markers now use:
- symbol-driven event cores
- outward ring pulses
- reduced diffuse glow compared with older Earth builds
5. The right-side stats now show:
1. BGP 启用时始终显示 collectors。
2. Incident marker 现在是 Earth BGP 的主 marker
3. 如果没有 incidentEarth 回退显示 anomaly marker
4. 如果也没有 anomalycollector 仍然提供存在感。
5. 专用 `activity layer` 现在增加:
- 每个 collector 最近 15 分钟活动 halo
- 基于活跃 collector 推导的区域聚合活动提示
6. Incident marker 现在使用:
- 由符号驱动的事件核心
- 向外扩散的环形脉冲
- 相比旧版 Earth 更少的弥散 glow
7. 右侧统计现在显示:
- BGP events
- collector count
- BGP status summary
This is directionally correct, but still incomplete for low-event-density periods. Right now Earth can still feel too quiet when incidents are sparse because the system lacks a dedicated `activity layer` between raw observation and incident focus.
这个方向是对的,但在低事件密度时期仍不完整。当前 Earth 在 incident 稀疏时仍可能显得过于安静,因为系统还缺少位于原始观测和 incident 聚焦之间的专用 `activity layer`
Current BGP status strategy:
当前 BGP 状态策略:
- incidents present: show active incident count
- no incidents but anomalies present: show active anomaly count, plus active observation regions when available
- no incidents/anomalies but activity present: show `观测网络运行中`
- no incidents/anomalies but collectors present: show `观测网络运行中 · 当前未发现聚合级事件`
- no BGP data at all: show `暂无观测数据`
- incident:显示活跃 incident 数量
- incident 但有 anomaly显示活跃 anomaly 数量,并在可用时显示活跃观测区域
- incident/anomaly 但有 activity:显示 `观测网络运行中`
- incident/anomaly 但有 collectors:显示 `观测网络运行中 · 当前未发现聚合级事件`
- 完全无 BGP 数据:显示 `暂无观测数据`
Earth info-card strategy:
Earth info-card 策略:
- `bgp` card is now incident-centric in wording
- `bgp_collector` card shows collector location and current event count
- `bgp` 卡片文案以 incident 为中心
- `bgp_collector` 卡片显示 collector 位置和当前事件数
## Current Product Gap
## 当前产品缺口
The main product gap is not architecture correctness. It is low-density visualization strategy.
主要缺口不是架构正确性,而是低密度可视化策略。
Current reality:
当前事实:
- incident count is naturally much lower than anomaly count
- that is expected, because incidents are aggregated and de-noised
- but incident-first rendering makes the Earth view look too quiet unless there is another always-available activity layer
- incident 数量天然远低于 anomaly 数量
- 这是预期行为,因为 incident 是聚合和去噪后的结果
- incident-first 渲染会让 Earth 显得过于安静,除非有另一层始终可用的 activity layer
Implementation detail for the recommended `activity layer` is expanded in [bgp-region-aggregation-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-bgp-region-aggregation-plan.md).
推荐 `activity layer` 的实现细节在 [BGP 区域聚合计划](/home/ray/dev/linkong/planet/docs/plans/earth-bgp-region-aggregation-plan.md) 中展开。
So the immediate next milestone is:
因此最近的里程碑是:
`event map -> observability map`
That means Earth needs three simultaneously readable layers:
这意味着 Earth 需要三层同时可读:
1. `observation layer`
- collectors
@@ -205,108 +205,108 @@ That means Earth needs three simultaneously readable layers:
- regional activity scoring
- incident presence bonus
3. `incident layer`
- sparse but highly legible, high-confidence event objects
- symbol-driven markers
- outward ring pulse instead of broad diffuse glow
- 稀疏但高度清晰的高置信事件对象
- 符号化 marker
- 向外环形脉冲,而不是大面积弥散 glow
## Incident Visual Direction
## Incident 视觉方向
The Earth `incident` layer should not read like a large glowing patch. It should read like a compact, high-confidence event focus.
Earth `incident` 层不应该像一大片发光区域,而应该像紧凑、高置信度的事件焦点。
Design principles:
设计原则:
1. `incident` markers should use a strong primary symbol
- the symbol shape should carry type meaning where possible
- examples:
- `origin_change`: triangle-like warning marker
- `mass_withdrawal`: alert/exclamation-style marker
- `more_specific_burst`: split/radiating marker
1. `incident` marker 应使用强主符号
- 符号形状尽量承载类型含义
- 示例:
- `origin_change`:类似三角警告 marker
- `mass_withdrawal`:告警/感叹号风格 marker
- `more_specific_burst`:分裂/放射 marker
2. emphasis should come from outward ring pulses, not area flooding
- use a compact hot core
- use one or more expanding ring pulses
- avoid broad luminous blobs that make the event center feel vague
2. 强调应来自向外扩散的环形脉冲,而不是区域泛光
- 使用紧凑高亮核心
- 使用一个或多个扩张环形脉冲
- 避免让事件中心变得模糊的大面积亮斑
3. `collector` and `incident` must stay visually distinct
- collectors are observation infrastructure
- incidents are extracted event focus
- collector activity should stay quieter than incident pulse language
3. `collector` `incident` 必须保持视觉区别
- collector 是观测基础设施
- incident 是抽取后的事件焦点
- collector activity 应比 incident pulse 更安静
4. calm periods still need observability presence
- collectors and activity layers should keep the map alive
- once incidents appear, they should clearly dominate nearby BGP visuals
4. 平静期仍需要观测存在感
- collectors activity layer 应让地图保持活跃
- 一旦出现 incident,它们应明确压过附近 BGP 视觉元素
5. incident geography should become `prefix-centric`
- collectors should remain evidence sources, not the primary event location
- preferred geography priority:
5. incident 地理位置应转向 `prefix-centric`
- collector 应保持证据来源身份,而不是主要事件位置
- 推荐地理优先级:
- `prefix_geography`
- `prefix_scope`
- `ASN organization region`
- `collector centroid` as final fallback
- `prefix_scope` should remain an observation-derived scope hint
- a new `prefix_geography` layer should be introduced for actual prefix-centric placement
- `collector centroid` 作为最终 fallback
- `prefix_scope` 应保持为由观测推导出的范围提示
- 应新增真正面向 prefix 位置的 `prefix_geography`
Reference inspiration:
参考灵感:
- `World Monitor`
- sparse event symbols
- compact centers
- ring-like outward pulses
- stronger incident legibility than diffuse glow
- 稀疏事件符号
- 紧凑中心
- 类似环形的向外脉冲
- 比弥散 glow 更强的 incident 可读性
## Current Console Behavior
## 当前控制台行为
Relevant page:
相关页面:
- `frontend/src/pages/BGP/BGP.tsx`
Current BGP console page has three levels:
当前 BGP 控制台页面有三层:
1. observation summary
- total events
- collector count
- prefix count
1. 观测摘要
- 总事件数
- collector 数量
- prefix 数量
2. incident summary and incident table
2. incident 摘要和 incident 表格
3. anomaly detail table plus recent observation events
3. anomaly 详情表和最近 observation events
This means the BGP page still has useful signal even when there are zero anomalies.
这意味着即使 anomaly 为零BGP 页面仍有可用信号。
## Known Product/Engineering Boundaries
## 已知产品/工程边界
1. The current system is still closer to an event board than a full BGP sensing platform.
2. RIS coverage still needs to expand beyond narrow subscription scope.
3. BGPStream history is still not full MRT-to-prefix decoded analytics.
4. Collector geography still depends heavily on static RIPE RIS mappings.
5. Incident-to-cable/IXP/region association is still weak and early-stage.
6. Earth currently visualizes logical observation/impact structure, not true physical traffic paths.
1. 当前系统仍更接近事件看板,而不是完整 BGP sensing platform
2. RIS 覆盖范围仍需从较窄订阅范围继续扩展。
3. BGPStream 历史数据仍不是完整 MRT-to-prefix 解码分析。
4. Collector 地理位置仍高度依赖静态 RIPE RIS 映射。
5. Incident 与海缆、IXP、区域之间的关联仍较弱且处于早期阶段。
6. Earth 当前可视化的是逻辑观测/影响结构,而不是真实物理流量路径。
## Test Status
## 测试状态
BGP-specific tests live in:
BGP 专项测试位于:
- `backend/tests/test_bgp.py`
Verified status at this point:
当前已验证状态:
- `25 passed` for `backend/tests/test_bgp.py`
- `62 passed` for `backend/tests`
- `backend/tests/test_bgp.py``25 passed`
- `backend/tests``62 passed`
Covered areas include:
覆盖范围包括:
- normalization
- observation serialization
- enrichment
- detectors, including route leak candidate and path flap
- detectors,包括 route leak candidate path flap
- incident aggregation
- batch anomaly creation
- BGP events/incidents API
- summary endpoints
## Most Relevant Files
## 最相关文件
Backend:
后端:
- `backend/app/models/bgp_observation.py`
- `backend/app/models/bgp_anomaly.py`
@@ -318,7 +318,7 @@ Backend:
- `backend/app/api/v1/bgp.py`
- `backend/app/api/v1/visualization.py`
Frontend:
前端:
- `frontend/src/pages/BGP/BGP.tsx`
- `frontend/public/earth/js/bgp.js`
@@ -327,29 +327,29 @@ Frontend:
- `frontend/public/earth/js/constants.js`
- `frontend/public/earth/index.html`
## Recommended Next Steps
## 推荐下一步
### Next Backend / Detection Priority
### 后端 / 检测优先级
1. Integrate real RPKI validation data.
2. Expand realtime collector coverage and include withdrawals more broadly.
3. Continue refining route leak and path instability detectors with stronger heuristics.
1. 集成真实 RPKI 校验数据。
2. 扩展实时 collector 覆盖范围,并更广泛纳入 withdrawals。
3. 用更强启发式继续完善 route leak path instability detector
### Next Correlation / Storytelling Priority
### 关联 / 叙事优先级
4. Strengthen incident aggregation semantics and titles.
5. Add weak correlation from incidents to:
- cable corridors
- landing points
4. 强化 incident 聚合语义和标题。
5. 增加 incident 与以下对象的弱关联:
- 海缆走廊
- 登陆点
- IXPs
- other traffic anomaly sources
6. Refine Earth hover/click handoff between collectors and incidents.
- 其它流量异常来源
6. 优化 Earth collector incident 之间的 hover/click 交接。
### Next Visualization Priority
### 可视化优先级
7. Refine regional activity scoring so the activity layer is informative without becoming noisy.
8. Add more incident symbol types as new detectors land.
9. Add a real prefix geography source:
- `IPtoASN / IPtoCountry` as the first practical dataset
- `OpenGeoFeed` as a higher-quality override layer
- registry/whois only as fallback
7. 调整区域 activity scoring,让 activity layer 有信息量但不嘈杂。
8. 随着新 detector 落地,增加更多 incident 符号类型。
9. 增加真实 prefix geography 来源:
- `IPtoASN / IPtoCountry` 作为第一阶段可用数据集
- `OpenGeoFeed` 作为更高质量 override
- registry/whois 只作为 fallback

View File

@@ -1,11 +1,11 @@
# Earth Frontend Context
# Earth 前端结构
本文件描述当前 Earth 大屏前端的真实结构,重点是帮助后续继续改 HUD、图层、媒体面板、真实地形、BGP 可视化时,不再重复踩结构和状态同步上的坑。
相关规则建议一起参考:
- [rules.md](/home/ray/dev/linkong/planet/rules.md)
- [frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md)
- [项目规则](/home/ray/dev/linkong/planet/rules.md)
- [前端布局指南](/home/ray/dev/linkong/planet/docs/technical/zh/frontend-layout-guidelines.md)
## 当前目标
@@ -81,6 +81,13 @@ React 路由入口:
- status message
- tooltip / error / 清理逻辑
当前 status message 有两类短提示:
- 普通业务提示:通过 `showStatusMessage()` 入队显示。
- 手势提示:通过 `showGestureStatusMessage()` 直接短暂显示,用于缩放视角时的 `缩放 N%`
手势提示不会抢占 loading 状态。对应样式是 [hud.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/hud.css) 中的 `.earth-status-message.gesture`
### 5. 地球与地形
- [earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js)
@@ -98,6 +105,7 @@ React 路由入口:
- [cables.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/cables.js)
- [bgp.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp.js)
- [bgp-cruise-adapter.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp-cruise-adapter.js)
- [vessels.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/vessels.js)
- [news.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/news.js)
- [tv.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/tv.js)
- [layer-startup-tasks.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/layer-startup-tasks.js)
@@ -132,6 +140,10 @@ React 路由入口:
当前 BGP 巡航只是这套能力的一个调用方不应再把“按队列巡航”和“BGP 事件展示”混写在同一个状态机里。
新闻巡航摘要计划见:
- [Earth 新闻巡航摘要计划](/home/ray/dev/linkong/planet/docs/plans/earth-news-cruise-summary-plan.md)
## 当前样式分层
Earth 的 CSS 不是一份大样式表,而是分层管理:
@@ -232,6 +244,99 @@ Earth 图层按钮现在不应再只有“开/关”两态,而应支持:
这样后续新增会参与启动加载的图层时,顺序、模式和提示文案都在同一处定义,不需要再去 `main.js` 里补第二套常量。
### 船只图层与图例
AIS 船只图层入口:
- [vessels.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/vessels.js)
- [interactable.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/interactable.js)
船只图层当前负责:
- 请求 `/api/v1/visualization/geo/vessels`
- 将 BarentsWatch AIS GeoJSON 转为地球局部坐标 marker 数据
- 通过 `createInteractableLayer()` 注册 Interactable 图标层
- 用按航向分桶的 `THREE.Points` 批量渲染普通船只 marker
- 按船型映射颜色
- 根据航行/停泊状态绘制三角形或圆点纹理
- 用单点 `THREE.Points` overlay 承载 hover / locked glow
- 支持 hover、lock、轨迹加载和视觉聚焦
船只图层不再是“每艘船一个 `THREE.Sprite`”。原始 Sprite 方案在拖动地球时会把透明对象排序、draw call 和对象级 raycast 成本全部放到主交互路径上;即使 BarentsWatch 免费 AIS 当前只覆盖挪威周边,也会让地球拖动明显不跟手。
当前设计把普通船只拆成少量批次:
- moving / anchored 分开。
- moving 船只按 `VESSEL_COURSE_BINS` 做航向分桶。
- 每个批次是一组 `THREE.PointsMaterial`,位置和颜色写入 `BufferGeometry` attribute。
- 普通态不带 glowhover / locked 时才在相同点位叠加带 glow 的单点 overlay。
方向标准以 AIS `course / cog` 为准:从正北开始顺时针。普通态和交互态都通过同一套 canvas 旋转规则生成纹理,避免 hover 后箭头方向和原 marker 不一致。
船只 hover / click 也不再对渲染对象做 `raycaster.intersectObjects()``main.js` 只负责传入当前 Earth、camera、pointer 和命中半径,实际命中计算由 `interactable.js` 的图标层接口完成:
1. 拖动地球或惯性旋转时跳过 hover picking。
2. 对 hover picking 做轻量节流。
3. 只保留正面船只作为候选。
4. 将候选船只投影到屏幕坐标。
5.`VESSEL_POINTER_RADIUS_PX` 做像素距离命中,并取最近船只。
这样 picking 位置和用户看到的屏幕 marker 对齐,也避免 `Points` 自带 raycaster 在固定屏幕尺寸图标上的命中半径错位。
图例系统已经注册 `vessels` 模式:
- [legend.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/legend.js)
- [legend.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/legend.css)
`getVesselLegendItems()` 返回带 `shape` 的图例项:
- `shape: "vessel"`:三角形,表示航行船只。
- `shape: "dot"`:圆点,表示停泊或低速状态。
图例项颜色来自 `VESSEL_CONFIG.colors`,不要在 `legend.css` 里重新定义业务颜色。新增船型时,应优先改 `vessels.js``constants.js` 的船型映射,再同步图例项。
`interactable.js` 是后续地表图标类图层的共用入口。它当前已经承载船只、BGP 事件、BGP 观测站和算力中心图层的批量 `Points`、texture cache、hover / locked overlay、默认 glow、状态增量更新和屏幕空间 pickingBGP 事件的向外扩散圈、BGP 观测站的 halo / 覆盖扇形仍由 `bgp.js` 保留业务动画,但图标本体和 pointer 命中已经接入通用层。新增小型、中心对齐、可以参与深度测试的图标类元素时,应优先复用这个接口,而不是再次复制船只渲染逻辑。
登陆点是当前明确保留的例外:它曾接入 `Interactable`,但 pin 类 SVG 在地球边缘会被 `THREE.Points` 的深度测试裁切成碎片;关闭 depthTest 又会破坏背面遮挡语义。因此登陆点退回 `cables.js` 内的专用 `THREE.Sprite` 路径,并改为 canvas 生成的黄色扁平球纹理。它的 `altitudeOffset``renderOrder` 与海缆线一致避免漂在海缆之上Sprite 本体关闭 `depthTest` 保持球完整,背面可见性由 `isFacingCamera()` 的球体遮挡判断控制。
图标资源可以继续用 canvas draw也可以放到 `frontend/public/earth/assets/icons/` 后由 `Interactable` 预加载。asset 路径不会在每帧读取;图层加载阶段通过 `preloadAssets()` 只加载一次 SVG / 图片,之后按 `icon source + state + bucket + color` 生成 `CanvasTexture` 并复用。当前算力中心已经从 `assets/icons/compute-supercomputer.svg``assets/icons/compute-gpu-cluster.svg` 和备用 `assets/icons/compute-hdd-network.svg` 读取图标,再在 canvas 上叠加估算位置的 `?` badge。
asset 图标大小由 `Interactable``icon.fitSize` 控制。SVG / 图片文件应尽量保持原始 viewBox 和路径,不要为了在地球上显示成 60x60 而手写 `transform``drawAssetIcon()` 会把资源等比 contain 到指定尺寸并居中绘制到 atlas canvas。
`Interactable` 默认使用固定屏幕像素尺寸适合船只、BGP 事件、BGP 观测站、算力中心这类需要稳定识别的图标。如果某类图标需要跟随相机距离缩放,可以把 `sizeMode` 设为非 `"fixed"`,并用 `sizeScale.min / max / referenceFov` 控制缩放范围;单个 marker 的业务尺寸差异可以通过 `getPointSizeMultiplier()` 表达,例如 BGP 事件按严重级别调整点大小BGP 观测站按活跃度调整点大小。
`Interactable` 不再把图标本体额外抬离业务高度。`altitudeOffset` 就是 marker、hover glow、locked glow 和 picking 共同使用的地表高度;这样船只图标会继续贴着船只轨迹线,不会因为单独抬高显示位置而显得漂浮。后续如果要解决边缘 glow 裁切,应优先考虑 glow 纹理、overlay 尺寸或图层专属特效,而不是把通用图标层整体抬高。
跨 Interactable 的同坐标避让也在公共层处理。每个 marker 会保留 `icon_base_position` 作为业务原始位置;当多个 Interactable marker 归入同一个经纬度 key 时,公共层会把它们沿地表切平面排成小圈,并刷新已创建的 `THREE.Points` geometry。这样视觉位置和屏幕空间 picking 位置一致,不需要业务层再单独判断“算力中心和 BGP 事件重叠”这类场景。
接口细节、生命周期和接入示例见:
- [Earth 可交互图标接入](/home/ray/dev/linkong/planet/docs/technical/zh/earth-interactable-usage.md)
### 视角控制反馈
[controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js) 统一维护 Earth 缩放状态。滚轮缩放、缩放按钮和触屏双指捏合最终都会更新 `zoomLevel`,并通过 `showZoomStatusCapsule()` 显示当前缩放比例:
```javascript
showGestureStatusMessage(`缩放 ${Math.round(zoomLevel * 100)}%`, "info");
```
该提示每 90ms 最多更新一次,显示 760ms 后淡出。它是视角反馈,不是数据加载进度,也不应该写进图层 loading 状态。
[main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js) 只负责在双指捏合缩放时调用 `setZoomLevel()``showZoomStatusCapsule()`。鼠标滚轮与缩放按钮的胶囊提示应继续放在 `controls.js`,避免同一种缩放反馈散落在多个模块。
拖拽地球的旋转灵敏度会根据当前缩放连续衰减,而不是按某个缩放阈值分段:
```javascript
const scale = THREE.MathUtils.clamp(
Math.pow(zoom, -CONFIG.dragRotationZoomExponent),
CONFIG.dragRotationScaleMin,
CONFIG.dragRotationScaleMax,
);
```
调参入口在 [constants.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/constants.js)`dragRotationFactorBase` 控制基础速度,`dragRotationZoomExponent` 控制放大后的衰减曲线,`dragRotationScaleMin` / `dragRotationScaleMax` 控制上下限。
### `data-status-target`
图层按钮可以通过:
@@ -284,98 +389,23 @@ Earth 设置面板当前由 [controls.js](/home/ray/dev/linkong/planet/frontend/
1. 图层开关 loading 状态持续可见
2. 页面空闲时会预热 `ensureTerrainReady()`
也就是说,后续再继续优化 terrain 时,优先顺序应该是:
## 当前巡航链路
1. 先保证用户感知正确
2. 再压缩首次等待
3. 最后才做更激进的几何/瓦片优化
当前巡航边界:
## 当前高频风险点
### 1. 视觉状态和业务状态不同步
Earth 里最常见的 bug 不是“没渲染”,而是:
- 图层关了tooltip 还在
- 锁定对象隐藏了info card 还在
- legend 没跟图层切换
- loading 已结束,但按钮还像没开
后续改动必须优先检查状态同步。
### 2. HUD 布局问题先查结构,不要先打 CSS 补丁
Earth HUD 历史上反复出现:
- 面板只剩一条缝
- markdown 被裁掉
- tabs/iframe 被 `overflow: hidden` 吃掉
优先检查:
1. 谁负责高度
2. 谁负责滚动
3. 哪一层在裁剪
不要上来先加 `overflow: hidden` 或额外包装层。
### 3. Transitional path 必须收口
Earth 已经经历过多轮 HUD、toolbar、media panel 重构,所以最容易积累:
- 旧 helper
- 旧 class
- 旧 fallback 逻辑
- 已废弃变体
每次大功能完成后,都要做一次 cleanup pass。
### 4. 巡航与业务事件不要再深度耦合
当前正确边界应该是:
- 通用巡航层只知道:
- 通用巡航层包含:
- 当前目标
- 队列顺序
- 相机 focus
- 停留 / 隐藏 / 切换
- 业务模块只负责
- 业务模块提供
- 提供目标队列
- 提供 focus 坐标
- 提供卡片内容
- 提供高亮/图层副作用
如果以后再给海缆、卫星或新闻做巡航,不应复制一套新的 `main.js` 状态变量,而应复用
巡航模块的结构文件
- [cruise-sequencer.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/cruise-sequencer.js)
- [callout-connector.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/callout-connector.js)
- [bgp-cruise-adapter.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp-cruise-adapter.js) 这种业务适配层模式
## 当前推荐改动方式
如果后续继续改 Earth建议按这个顺序
1. 先确认改的是:
- Three.js 渲染层
- HUD 结构层
- 图层状态层
- 面板内容层
2. 如果涉及图层按钮,优先接入统一状态机
3. 如果涉及可见性切换,检查 tooltip / legend / info-card / lock 是否一起收口
4. 如果涉及面板布局,先查结构再动 CSS
## 当前与控制台前端的边界
Earth 前端和控制台前端不是同一套 UI 系统:
- 控制台前端React + Ant Design 工作台
- Earth 前端:`public/earth` 原生 HUD + Three.js 展示面
因此:
- Earth 不应该直接复用 Ant Table / AppLayout 语义
- 控制台也不应该照搬 Earth HUD 动画和玻璃层语言
控制台相关结构见:
- [admin-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/frontend-admin-frontend-context.md)
- [bgp-cruise-adapter.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp-cruise-adapter.js)

View File

@@ -0,0 +1,270 @@
# Earth Interactable 使用说明
`Interactable` 是 Earth 地表“图标类可交互元素”的通用渲染入口。它把船只图层验证过的模式抽成公共能力:普通态用批量 `THREE.Points`hover / locked 用少量 overlay拾取走屏幕空间命中图标资源统一转进 canvas texture并在公共层处理 glow、状态、尺寸、贴地渲染和同坐标避让。
当前已接入:
| 图层 | 业务文件 | 图标来源 | 补充动画 |
| --- | --- | --- | --- |
| AIS 船只 | `frontend/public/earth/js/vessels.js` | canvas draw航行三角形 / 停泊圆点 | 船只轨迹仍由业务层维护 |
| 算力中心 | `frontend/public/earth/js/compute-centers.js` | `assets/icons/compute-*.svg` | 估算位置 `?` badge 通过 `icon.afterDraw()` 叠加 |
| BGP 事件 | `frontend/public/earth/js/bgp.js` | canvas draw按事件类型绘制符号 | 向外扩散圈仍由 BGP 业务层维护 |
| BGP 观测站 | `frontend/public/earth/js/bgp.js` | `assets/icons/bgp-broadcast-pin.svg` | halo、活跃度 core、覆盖扇形和雷达扫掠仍由 BGP 业务层维护 |
登陆点曾尝试接入 Interactable但 pin 类 SVG 在地球边缘会被 `THREE.Points` 深度测试裁切成碎片;当前退回 `THREE.Sprite` 专用路径,并改为由 canvas 生成黄色扁平球纹理。旧 SVG 资产保留在 `assets/icons/` 目录中,但登陆点运行时不再依赖 SVG。
## 为什么需要 Interactable
之前每个地表图标图层都容易各写一套:
- icon texture 生成
- hover / locked 状态
- glow 样式
- picking 命中半径
- zoom 下的尺寸策略
- 同经纬度对象重叠避让
这些逻辑如果分散在业务文件里,视觉会漂移,后续调参也会变成逐图层修补。`Interactable` 的边界是:公共层负责“图标怎么在地球上稳定显示和被选中”,业务层负责“数据从哪里来、图标表达什么语义、详情卡展示什么、是否有额外动画”。
## 入口
```javascript
import { createInteractableLayer } from "./interactable.js";
```
核心调用形态:
```javascript
const layer = createInteractableLayer({
id: "example",
objectType: "example_object",
renderOrder: 4.4,
altitudeOffset: 0.2,
pointSize: 34,
icon: {
draw(context, options) {
// draw canvas icon
},
},
getPosition: (item) => ({
latitude: item.latitude,
longitude: item.longitude,
}),
getKind: (item) => item.kind || "default",
});
```
业务模块通常只暴露一层薄封装:
```javascript
export function getExampleMarkers() {
return layer.getMarkers();
}
export function getExamplePointerIntersections(options) {
return layer.getPointerIntersections(options);
}
export function setExampleMarkerState(marker, state = "normal") {
layer.setMarkerState(marker, state);
}
export function updateExampleVisualState(lockedObjectType, lockedObject, camera) {
layer.updateVisualState(lockedObjectType, lockedObject, camera);
}
```
## 配置参数
| 参数 | 默认值 | 说明 |
| --- | --- | --- |
| `id` | 必填 | 图层唯一标识,用于 group name、避让注册和 debug。 |
| `objectType` | `id` | marker 写入 `userData.type` 的业务类型,主交互层用它判断 locked 对象。 |
| `renderOrder` | `4` | 普通 points 和 hover / locked overlay 的基础渲染顺序。 |
| `altitudeOffset` | `0.2` | 业务高度,按 `CONFIG.earthRadius + altitudeOffset` 计算原始地表位置。 |
| `pointSize` | `32` | 基准屏幕像素尺寸。普通 points 和 overlay 都以它为基础。 |
| `sizeMode` | `"fixed"` | 默认固定屏幕尺寸;非 `"fixed"` 时会按相机距离做比例缩放。 |
| `sizeScale` | `{ referenceFov: 75, min: 0.12, max: 3 }` | `sizeMode !== "fixed"` 时的缩放范围。 |
| `atlasCellSize` | `128` | icon canvas texture 尺寸。 |
| `colors` | `{}` | 支持 `normal`、按 kind 的平铺 key以及 `byKind`。 |
| `opacity` | `{ normal: 0.88, dimmed: 0.26, hover: 0.98, locked: 1 }` | 各状态透明度。 |
| `stateScale` | `{ hover: 1, locked: 1, dimmed: 1 }` | 各状态尺寸倍率。 |
| `pulse` | `{}` | locked 态可选呼吸缩放,支持 `enabled``speed``amplitude`。 |
| `avoidance` | `{ enabled: true, precision: 4, radius: 1.1, step: 0.35 }` | 跨 Interactable 的同坐标避让配置。 |
| `icon` | 必填 | 图标来源,支持 canvas draw、SVG / 图片 asset、状态 asset、锚点和后处理。 |
| `getPosition(item)` | 必填 | 返回 `{ latitude, longitude }``THREE.Vector3`。 |
| `getKind(item)` | `item.type || "default"` | 返回业务类型,用于颜色和 texture 分桶。 |
| `getRotationBin(marker)` | `0` | 返回旋转分桶,例如船只按航向分 32 桶。 |
| `getBucketKey(marker)` | `String(getRotationBin(marker))` | 返回 texture / geometry 分桶 key。 |
| `getPointSizeMultiplier(marker)` | `1` | 单 marker 尺寸倍率。BGP 事件按严重级别、观测站按活跃度使用它。 |
| `getUserData(item)` | `item` | 写入 marker 的业务字段。 |
## Icon 配置
`icon.anchor` 可选,默认 `{ x: 0.5, y: 0.5 }`,表示纹理中心对齐 marker 坐标。它只适合小范围的视觉锚点偏移;如果图标主体很大、且需要在地球边缘完整显示,例如登陆点曾使用过的 pin 类图标,不应强行走 `THREE.Points + depthTest`,否则图标主体会被地球深度裁切。
### Canvas 图标
canvas 图标适合船只、BGP 事件这类需要按状态或旋转动态绘制的符号:
```javascript
const vesselIconLayer = createInteractableLayer({
id: "vessels",
objectType: "vessel",
pointSize: 34,
icon: {
draw(context, { marker, rotationBin = 0, glow = false, color = "#ffffff" }) {
if (!marker.userData.anchored) {
context.rotate((rotationBin / 32) * Math.PI * 2);
}
context.fillStyle = color;
context.shadowColor = color;
context.shadowBlur = glow ? 14 : 0;
context.beginPath();
context.moveTo(0, -37);
context.lineTo(28, 32);
context.lineTo(0, 17);
context.lineTo(-28, 32);
context.closePath();
context.fill();
},
},
getRotationBin: getCourseBin,
getBucketKey: (marker) => `${marker.userData.anchored ? "anchored" : "moving"}:${getCourseBin(marker)}`,
});
```
`icon.coordinates !== "canvas"` 时,`Interactable` 会先把 context 平移到 atlas 中心;船只这类自己使用中心坐标绘制的图标不需要声明 `coordinates`
### SVG / 图片 Asset 图标
asset 图标适合算力中心、BGP 观测站这类已有 SVG 的设施图标:
```javascript
const computeCenterIconLayer = createInteractableLayer({
id: "computeCenters",
objectType: "compute_center",
pointSize: 36,
atlasCellSize: 128,
icon: {
coordinates: "canvas",
colorable: false,
fitSize: 60,
glowBlur: 16,
getSource({ marker, item }) {
const siteType = marker?.userData?.site_type || item?.site_type || "gpu_cluster";
return COMPUTE_CENTER_ICON_SOURCES[siteType];
},
afterDraw(context, { marker, item }) {
if (marker?.userData?.is_estimated ?? item?.is_estimated) {
drawComputeCenterEstimatedBadge(context, true);
}
},
},
});
```
使用 asset 时有几个约定:
- SVG / 图片文件放在 `frontend/public/earth/assets/icons/`,以 `/earth/assets/icons/name.svg` 引用。
- 原始 SVG 应尽量保留标准 `viewBox` 和路径,不要为了显示大小写死 transform。
- 显示尺寸由 `icon.fitSize` 控制;它可以是数字、`{ width, height }`,也可以是函数。
- `icon.colorable !== false` 且提供状态颜色时,公共层会先把 asset 画到临时 canvas再用 `source-in` tint 成目标颜色。
- 多色图片或不希望被 tint 的 SVG 应设置 `colorable: false`
## 生命周期
常规加载流程:
```javascript
export async function loadExampleLayer(_scene, earth) {
clearExampleData(earth);
const markerData = await fetchExampleData();
await layer.preloadAssets(markerData);
layer.setData(markerData);
layer.attach(earth);
layer.setVisible(showExampleLayer);
return { totalCount: layer.getCount() };
}
```
各方法职责:
| 方法 | 说明 |
| --- | --- |
| `preloadAssets(items)` | 收集 normal / hover / locked 可能用到的 asset source并用浏览器 `Image` 预加载。canvas draw 图标可跳过。 |
| `setData(items)` | 清理旧 points生成 marker注册避让按 bucket 重建 `THREE.Points`。 |
| `attach(parent)` | 将图层 group 挂到 Earth root。 |
| `setVisible(next)` | 控制 group、points 和 overlay 可见性。 |
| `setMarkerState(marker, state)` | 设置 `normal` / `hover` 等状态并触发视觉状态失效。 |
| `updateVisualState(focusType, focusObject, camera)` | 更新普通态 opacity / size并刷新 hover / locked overlay。 |
| `getPointerIntersections(options)` | 屏幕空间拾取,返回按像素距离排序的命中结果。 |
| `clearData(parent)` | 注销避让、释放 geometry / material、清空 marker 并从 parent 移除 group。 |
## Picking 接入
`Interactable` 不依赖 Three.js 对 `Points` 的默认 raycast。主交互层只要把 Earth、camera、pointer 和命中半径传入:
```javascript
const intersects = getVesselPointerIntersections({
earth,
camera,
pointer,
radiusPx: 22,
width: window.innerWidth,
height: window.innerHeight,
});
```
公共层会做这些事:
1. 把相机位置转到 Earth local 坐标。
2. 跳过背面 marker。
3. 把 marker world position 投影到屏幕坐标。
4.`radiusPx` 做像素距离命中。
5. 返回最近的候选对象。
拖动地球、惯性旋转、hover 节流这些策略仍属于 `main.js`,因为它们和全局输入状态有关。
## 同坐标避让
避让默认开启,作用范围是所有通过 `createInteractableLayer()` 创建的图层。公共层会按经纬度或 `THREE.Vector3` 生成 `icon_avoidance_key`,同 key 的 marker 会沿地表切平面排成小圈。
关键点:
- `icon_base_position` 保留业务原始位置。
- 避让只改渲染位置和 picking 位置,不改业务经纬度。
- 单个 marker 回到原始位置时会直接使用 `altitudeOffset` 计算出的业务贴地位置。
- 多个 marker 同坐标时,第一圈用 `avoidance.radius`,后续每圈加 `avoidance.step`
如果某个业务图层需要严格压在原始点位,可以显式关闭:
```javascript
createInteractableLayer({
id: "strict-layer",
avoidance: { enabled: false },
});
```
## 业务动画边界
`Interactable` 当前只负责图标本体和通用 hover / locked overlay。复杂动画仍放在业务模块里但要跟随 Interactable marker 的位置:
- BGP 事件扩散圈由 `bgp.js` 创建独立 ring sprite并在每帧 `position.copy(marker.position)`
- BGP 观测站 halo、status core、coverage halo 和覆盖扇形由 `bgp.js` 管理,图标本体由 Interactable 管理。
- 船只轨迹线仍由 `vessels.js` 管理,因为它依赖点击后额外加载的轨迹数据。
这个边界能避免通用接口过早承载所有动画类型。后续如果多个图层复用同一类动画,再把它收进 Interactable 的 `animations` 扩展。
## 新图层接入清单
1. 在业务文件中准备 marker data并保留必要的业务字段。
2. 选择 icon 类型canvas draw、SVG / 图片 asset`getSource()` 动态选择。
3. 配置 `pointSize``icon.fitSize``colors``opacity``stateScale`
4. 如果需要业务尺寸差异,提供 `getPointSizeMultiplier()`
5. 如果有旋转,提供 `getRotationBin()` 和稳定的 `getBucketKey()`
6. 加载时先 `preloadAssets()`,再 `setData()``attach()``setVisible()`
7.`main.js` 接入 `getPointerIntersections()`,并复用现有 hover / locked 状态更新流程。
8. 在图层样式索引和渲染顺序文档中记录 altitude、renderOrder、pointSize 和动画层级。

View File

@@ -2,7 +2,7 @@
本文记录当前 Earth 前端各图层的材质、颜色、透明度、线宽、半径偏移和
`renderOrder` 等样式属性。层级关系请配合
[earth-render-layer-order.md](/home/ray/dev/linkong/planet/docs/technical/earth-render-layer-order.md)
[Earth 渲染图层顺序](/home/ray/dev/linkong/planet/docs/technical/zh/earth-render-layer-order.md)
查看。
## 命名约定
@@ -80,6 +80,8 @@
## 海陆基座与国界
海陆基座是 Earth 的底图资产随启动预加载图层面板里的“国界线”只控制普通国界线、hover 线和可交互 hover。
| 正式名称 | 变量名 | 当前值 | 使用位置 / 说明 |
| --- | --- | --- | --- |
| 国界数据路径 | `COUNTRY_BOUNDARY_CONFIG.dataPath` | `"/earth/data/countries-admin0.min.geojson"` | GeoJSON 输入 |
@@ -95,11 +97,11 @@
| 国界线颜色 | `COUNTRY_BOUNDARY_CONFIG.lineColor` | `0x7fc7ff` | 普通国界线 |
| 国界线透明度 | `COUNTRY_BOUNDARY_CONFIG.lineOpacity` | `0.58` | 普通国界线 opacity |
| 国界线 hover 时压暗透明度 | `COUNTRY_BOUNDARY_CONFIG.dimmedLineOpacity` | `0.18` | hover 时普通国界线 opacity |
| 国界线半径偏移 | `COUNTRY_BOUNDARY_CONFIG.lineAltitudeOffset` | `0.24` | 普通国界线半径 |
| 国界线半径偏移 | `COUNTRY_BOUNDARY_CONFIG.lineAltitudeOffset` | `0.115` | 普通国界线半径;略高于高清材质 `0.10`,低于地形基准 `0.16`,减少悬浮感 |
| 国界线 renderOrder | `COUNTRY_BOUNDARY_CONFIG.lineRenderOrder` | `2.2` | 普通国界线层级 |
| 国界 hover 颜色 | `COUNTRY_BOUNDARY_CONFIG.hoverLineColor` | `0xff3b1f` | 霓虹红橘 |
| 国界 hover 透明度 | `COUNTRY_BOUNDARY_CONFIG.hoverLineOpacity` | `1.0` | hover 实线 opacity |
| 国界 hover 半径偏移 | `COUNTRY_BOUNDARY_CONFIG.hoverAltitudeOffset` | `0.32` | hover 实线半径 |
| 国界 hover 半径偏移 | `COUNTRY_BOUNDARY_CONFIG.hoverAltitudeOffset` | `0.14` | hover 实线半径;贴近地表但高于普通国界线 |
| 国界 hover renderOrder | `COUNTRY_BOUNDARY_CONFIG.hoverLineRenderOrder` | `2.3` | hover 实线层级 |
| 国界 hover glow 透明度 | `COUNTRY_BOUNDARY_CONFIG.hoverGlowOpacity` | `0.38` | glow 线 opacity |
| 国界 hover glow 线宽 | `COUNTRY_BOUNDARY_CONFIG.hoverGlowLineWidth` | `3` | glow `LineBasicMaterial.linewidth` |
@@ -146,21 +148,18 @@
| 海缆线宽 | `CABLE_CONFIG.line.lineWidth` | `1` | `LineBasicMaterial.linewidth` |
| 海缆透明度 | `CABLE_CONFIG.line.opacity` | `1.0` | 海缆线 opacity |
| 海缆 renderOrder | `CABLE_CONFIG.line.renderOrder` | `1` | 海缆线层级 |
| 登陆点半径偏移 | `CABLE_CONFIG.landingPoint.altitudeOffset` | `0.48` | 对齐算力中心贴地表 marker 高度 |
| 登陆点 icon 贴图尺寸 | `CABLE_CONFIG.landingPoint.textureSize` | `256` | canvas 渲染 EPS 参考图的实心 map-pin中间圆孔透明镂空 |
| 登陆点 icon 宽高比 | `CABLE_CONFIG.landingPoint.iconAspectRatio` | `0.82` | `Sprite.scale.x = height * aspect` |
| 登陆点 icon 锚点 | `CABLE_CONFIG.landingPoint.anchorX / anchorY` | `0.52 / 0.276` | `Sprite.center`,将 pin 下端点对齐登陆点经纬度 |
| 登陆点基础缩放 | `CABLE_CONFIG.landingPoint.baseScale` | `12` | 对齐算力中心等地表 icon 的 sprite 高度 |
| 登陆点半径偏移 | `CABLE_CONFIG.landingPoint.altitudeOffset` | `0.2` | 与海缆线同层贴地,避免凌空 |
| 登陆点 sprite 高度 | local `LANDING_POINT_SPRITE_HEIGHT` | `3` | `THREE.Sprite` 基准高度 |
| 登陆点缩放参考 FOV | local `LANDING_POINT_SIZE_REFERENCE_FOV` | `75` | 与当前 Earth 相机 FOV 一致 |
| 登陆点缩放下限 | local `LANDING_POINT_SIZE_SCALE_MIN` | `0.16` | 地球放到 200% 之后的最小倍率,限制高倍 zoom 下的屏幕占比;`3 * 0.16 = 0.48` |
| 登陆点缩放上限 | local `LANDING_POINT_SIZE_SCALE_MAX` | `3` | 远距离时的最大倍率;当前最小缩放约只能到 `2.50` |
| 登陆点 atlas 尺寸 | local `LANDING_POINT_ATLAS_CELL_SIZE` | `128` | canvas 扁平立体球纹理尺寸 |
| 登陆点颜色 | `CABLE_CONFIG.landingPoint.color` | `0xffaa00` | `SpriteMaterial.color` |
| 登陆点 emissive | `CABLE_CONFIG.landingPoint.emissive` | `0x442200` | 兼容旧球体材质sprite 不使用 |
| 登陆点 emissive 强度 | `CABLE_CONFIG.landingPoint.emissiveIntensity` | `0.5` | 兼容旧球体材质sprite 不使用 |
| 登陆点透明度 | `CABLE_CONFIG.landingPoint.opacity` | `1.0` | `SpriteMaterial.opacity` |
| 登陆点 renderOrder | `CABLE_CONFIG.landingPoint.renderOrder` | `4.5` | 对齐算力中心地表设施层级 |
| 登陆点 renderOrder | `CABLE_CONFIG.landingPoint.renderOrder` | `1` | 与海缆线同层;`depthTest: false` 保持球体完整,背面通过相机到球心的球体遮挡判断隐藏 |
| 登陆点 dim 亮度系数 | `landingPointVisual.dimBrightness` | `0.62` | dim 状态颜色乘数 |
| 相关登陆点高亮 opacity | `landingPointVisual.related.opacityBase / opacityPulse` | `0.8 / 0.2` | 高亮脉冲 |
| 非相关登陆点颜色 | `landingPointVisual.dimmed.colorRGB` | `{ r: 180, g: 116, b: 28 }` | dim 状态颜色,避免黑色基座透出成暗洞 |
| 非相关登陆点 emissive | `landingPointVisual.dimmed.emissive` | `0x3a2200` | dim 状态弱琥珀自发光 |
| 非相关登陆点 emissive 强度 | `landingPointVisual.dimmed.emissiveIntensity` | `0.18` | dim 状态弱发光强度 |
| 非相关登陆点 opacity | `landingPointVisual.dimmed.opacity` | `0.78` | dim 状态透明度,不再用低 alpha 混黑底 |
## 卫星、轨迹和 footprint
@@ -183,19 +182,42 @@
| 卫星覆盖层 renderOrder | `SATELLITE_CONFIG.overlayRenderOrder` | `12` | locked ring / halo / orbit |
| 自发光选中点颜色 | inline default | `"#ffd25a"` | `showSelfGlowStyle()` |
| 自发光选中点透明度 | inline | `0.96` | locked dot material |
| footprint renderOrder | local `GROUND_FOOTPRINT_RENDER_ORDER` | `3` | footprint fill |
| footprint renderOrder | local `GROUND_FOOTPRINT_RENDER_ORDER` | `3` | Starlink footprint fill 和 Iridium coverage ring必须高于地表 land / texture / terrain 层 |
| footprint group renderOrder | inline | `0` | 避免 Group 排序盖过卫星点 |
## AIS 船只
| 正式名称 | 变量名 | 当前值 | 使用位置 / 说明 |
| --- | --- | --- | --- |
| 船只半径偏移 | `VESSEL_CONFIG.altitudeOffset` | `0.2` | 普通 marker 位置,贴近真实地形基础层 |
| 船只轨迹半径偏移 | `VESSEL_CONFIG.track.altitudeOffset` | `0.2` | 选中船只轨迹线,与船只 marker 同一半径;前端会把轨迹末端锚到当前 marker 位置 |
| 船只 renderOrder | local `VESSEL_RENDER_ORDER` | `4.4` | 普通 marker 和交互 overlay |
| 船只轨迹 renderOrder | `VESSEL_RENDER_ORDER - 0.1` | `4.3` | 低于船只 marker |
| 船只点像素尺寸 | local `VESSEL_POINT_SIZE` | `34` | 普通 marker 与 hover / locked overlay 共享尺寸 |
| 船只纹理画布尺寸 | local `VESSEL_ATLAS_CELL_SIZE` | `128` | canvas 点纹理 |
| 航向分桶数 | local `VESSEL_COURSE_BINS` | `32` | moving 船只按 COG 分桶,降低 draw call 同时保留方向 |
| 船只 hover 拾取节流 | local `VESSEL_HOVER_PICK_INTERVAL_MS` | `100` | `main.js` hover picking |
| 船只屏幕命中半径 | local `VESSEL_POINTER_RADIUS_PX` | `22` | `main.js` 屏幕空间 picking |
| 普通船只透明度 | `VESSEL_CONFIG.marker.baseOpacity` | `0.88` | 普通 `PointsMaterial.opacity` |
| dimmed 船只透明度 | `VESSEL_CONFIG.marker.dimmedOpacity` | `0.26` | 锁定某艘船后其他批次透明度 |
| hover 船只透明度 | inline | `0.98` | hover overlay |
| locked 船只透明度 | inline | `1` | locked overlay |
| 船型颜色 | `VESSEL_CONFIG.colors.*` | cargo / tanker / passenger / fishing / military / other | `PointsMaterial.vertexColors` 和 overlay texture |
AIS 船只普通态使用批量 `THREE.Points`,不是逐船 `THREE.Sprite`。航行船只保持三角形,停泊或低速船只保持圆点;普通态不带 glowhover / locked 时在同一屏幕尺寸上叠加带 glow 的单点 overlay。AIS 航向按 `course / cog` 从正北顺时针解释,普通态和交互态必须使用同一套 canvas 旋转规则。
## 算力中心
| 正式名称 | 变量名 | 当前值 | 使用位置 / 说明 |
| --- | --- | --- | --- |
| 算力中心半径偏移 | `COMPUTE_CENTER_CONFIG.altitudeOffset` | `0.48` | marker 位置 |
| 算力中心基础透明度 | `COMPUTE_CENTER_CONFIG.marker.baseOpacity` | `0.88` | `SpriteMaterial.opacity` |
| 超算 marker 缩放 | `COMPUTE_CENTER_CONFIG.marker.supercomputerScale` | `12` | 超算 marker |
| GPU 集群 marker 缩放 | `COMPUTE_CENTER_CONFIG.marker.gpuClusterScale` | `12` | GPU marker |
| hover 缩放 | `COMPUTE_CENTER_CONFIG.marker.hoverScale` | `1.16` | hover 状态 |
| locked 缩放 | `COMPUTE_CENTER_CONFIG.marker.lockedScale` | `1.22` | locked 状态 |
| 算力中心点像素尺寸 | local `COMPUTE_CENTER_POINT_SIZE` | `36` | `Interactable` 普通 marker 与 hover / locked overlay 共享基准尺寸 |
| 算力中心 asset fit size | local `COMPUTE_CENTER_ICON_FIT_SIZE` | `60` | SVG asset 在 `128x128` atlas canvas 内的最大绘制尺寸,由 `icon.fitSize` 控制 |
| 算力中心基础透明度 | `COMPUTE_CENTER_CONFIG.marker.baseOpacity` | `0.88` | 普通 `PointsMaterial.opacity` |
| 超算 marker 缩放 | `COMPUTE_CENTER_CONFIG.marker.supercomputerScale` | `12` | 旧 Sprite 缩放参数;当前 Interactable 路径不再直接使用 |
| GPU 集群 marker 缩放 | `COMPUTE_CENTER_CONFIG.marker.gpuClusterScale` | `12` | 旧 Sprite 缩放参数;当前 Interactable 路径不再直接使用 |
| hover 缩放 | `COMPUTE_CENTER_CONFIG.marker.hoverScale` | `1.16` | hover overlay 尺寸倍率 |
| locked 缩放 | `COMPUTE_CENTER_CONFIG.marker.lockedScale` | `1.22` | locked overlay 尺寸倍率,并叠加 pulse |
| dimmed 缩放 / 透明度 | `dimmedScale / dimmedOpacity` | `0.82 / 0.34` | dim 状态 |
| 超算颜色 | `COMPUTE_CENTER_CONFIG.colors.supercomputer` | `"#38bdf8"` | marker texture |
| GPU 集群颜色 | `COMPUTE_CENTER_CONFIG.colors.gpu_cluster` | `"#2dd4bf"` | marker texture |
@@ -206,12 +228,16 @@
| 正式名称 | 变量名 | 当前值 | 使用位置 / 说明 |
| --- | --- | --- | --- |
| BGP 事件半径偏移 | `BGP_CONFIG.altitudeOffset` | `2.1` | anomaly marker |
| BGP collector 半径偏移 | `BGP_CONFIG.collectorAltitudeOffset` | `1.6` | collector marker |
| 事件基础缩放 | `BGP_CONFIG.marker.eventBaseScale` | `6.2` | anomaly sprite |
| collector 基础缩放 | `BGP_CONFIG.marker.collectorBaseScale` | `7.4` | collector plane |
| BGP 事件半径偏移 | `BGP_CONFIG.altitudeOffset` | `0.48` | BGP 事件 Interactable marker |
| BGP collector 半径偏移 | `BGP_CONFIG.collectorAltitudeOffset` | `0.2` | BGP 观测站 Interactable marker与船只同层贴地 |
| BGP 事件点像素尺寸 | local `BGP_EVENT_POINT_SIZE` | `34` | 事件 icon 的 Interactable 基准尺寸,按严重级别通过 `getPointSizeMultiplier()` 调整 |
| BGP 事件符号绘制尺寸 | local `BGP_EVENT_SYMBOL_SIZE` | `60` | 事件 canvas 符号在 `128x128` atlas 中的绘制尺寸 |
| BGP collector 点像素尺寸 | local `BGP_COLLECTOR_POINT_SIZE` | `36` | 观测站 Interactable 基准尺寸,按活跃度通过 `getPointSizeMultiplier()` 调整 |
| BGP collector asset fit size | local `BGP_COLLECTOR_ICON_FIT_SIZE` | `60` | `bgp-broadcast-pin.svg` 在 atlas canvas 内的最大绘制尺寸 |
| 事件基础缩放 | `BGP_CONFIG.marker.eventBaseScale` | `6.2` | 事件扩散圈锚点 |
| collector 基础缩放 | `BGP_CONFIG.marker.collectorBaseScale` | `7.4` | 观测站 halo / 覆盖动画锚点 |
| hover / dim 缩放 | `hoverScale / dimmedScale` | `1.16 / 0.92` | 交互状态 |
| 普通事件透明度 | `BGP_CONFIG.opacity.normal` | `0.78` | anomaly sprite |
| 普通事件透明度 | `BGP_CONFIG.opacity.normal` | `0.78` | BGP 事件 Interactable 普通态 |
| hover 透明度 | `BGP_CONFIG.opacity.hover` | `1.0` | hover 状态 |
| dimmed 透明度 | `BGP_CONFIG.opacity.dimmed` | `0.24` | dim 状态 |
| collector 透明度 | `BGP_CONFIG.opacity.collector` | `0.62` | collector 状态 |
@@ -223,8 +249,8 @@
| region 色 | `BGP_CONFIG.regionColor` | `0x2dd4bf` | 区域覆盖 |
| BGP ring 缩放 | `BGP_CONFIG.ring.scaleA / scaleB` | `2.5 / 3.4` | anomaly ring |
| BGP ring 透明度 | `BGP_CONFIG.ring.opacity` | `0.5` | anomaly ring |
| collector marker renderOrder | inline | `3` | `marker.renderOrder` |
| anomaly marker renderOrder | inline | `5` normal, `7` active | `marker.renderOrder` |
| collector marker renderOrder | local `BGP_COLLECTOR_RENDER_ORDER` | `4.4` | 观测站主图标,与船只同层 |
| anomaly marker renderOrder | local `BGP_EVENT_RENDER_ORDER` | `4.5` | BGP 事件主图标,与算力中心同层 |
## 天体与星空

View File

@@ -1,4 +1,4 @@
# News Live Streams Collector Format
# 新闻直播采集格式
`news_live_streams` 采集器面向“频道目录 JSON”输入而不是直接抓网页。
@@ -103,7 +103,7 @@
## 采集器配置方式
`news_live_streams` 不需要单独新页面,直接复用现有数据源配置
`news_live_streams` 不需要单独新页面,直接复用控制台 `/settings` 的“采集器设置”
- `endpoint`
- 频道目录 JSON API 地址

View File

@@ -7,8 +7,8 @@
| 顺序类型 | 当前顺序 | 说明 |
| --- | --- | --- |
| 控制面板顺序 | 海缆 → 轨迹 → 卫星 → 算力中心 → BGP → 地形 → 高清材质 → 大气云图 → 国界 → 经纬线 | 由 `displayOrder` 控制,按操作关注度排列。 |
| 注册 / 启动加载顺序 | 经纬线 → 国界 → 高清材质 → 大气云图 → 海缆 → 算力中心 → BGP → 卫星 | 由注册顺序和 `startupPriority` 控制,按地表到天空排列;轨迹和地形是依赖/可选显示层,不参与常规启动数据加载。 |
| 控制面板顺序 | 海缆 → 轨迹 → 卫星 → 算力中心 → 船只 → BGP → 地形 → 高清材质 → 大气云图 → 国界线 → 经纬线 | 由 `displayOrder` 控制,按操作关注度排列。 |
| 注册 / 启动加载顺序 | 经纬线 → 国界线 / 海陆基座 → 高清材质 → 大气云图 → 海缆 → 算力中心 → 船只 → BGP → 卫星 | 由注册顺序和 `startupPriority` 控制,按地表到天空排列;启动队列会先读取保存的图层可见状态,明确关闭的普通图层不预加载,高清材质关闭时不下载贴图;国界线图层例外,海陆基座始终预加载,保存状态只控制可交互国界线和 hover轨迹和地形是依赖/可选显示层,不参与常规启动数据加载。 |
## 地表图层栈
@@ -21,15 +21,17 @@
| 0.86 | 海陆基座填充 | `country-boundaries.js` | `landAltitudeOffset`; 海洋 `#010609`,陆地 `#080f1b` | 禁用 raycast | 即使国界线关闭,基座地图仍保持可用。 |
| 0.96 | 高清 Earth 材质 | `earth.js` | `textureOverlayAltitudeOffset` | 可见时作为地表拾取目标 | 高清材质始终压过海陆基座填充。 |
| 1 | 大气辉光和云图 | `earth.js` | 大气 / 云层球 | 不走普通对象选择路径 | 云图由“大气云图”图层开关控制。 |
| 1 | 海缆 | `cables.js` | `CABLE_CONFIG.line.renderOrder` | 海缆拾取路径 | 保持现有海缆层级。 |
| 1 | 海缆 / 登陆点 | `cables.js` | 海缆线和登陆点都使用 `renderOrder = 1`;半径偏移都为 `0.2`;登陆点是专用 `THREE.Sprite` 黄色扁平球 | 海缆走海缆拾取路径;登陆点 `depthTest: false` 保持球体完整,并用相机到球心的球体遮挡判断避免背面穿透 | 登陆点和海缆同层贴地,避免地表设施层的凌空感。 |
| 1.2 | 真实地形 | `earth.js`, `terrain.js` | `TERRAIN_CONFIG.baseRadiusOffset` 加地形位移 | 禁用 raycast | 地形压过高清材质;高清材质关闭时临时隐藏,重新开启后恢复原状态。 |
| 2.05 | 经纬线 | `earth.js` | `CONFIG.earthRadius + 0.14` | 禁用 raycast | 低透明度显示在高清材质上。 |
| 2.2 | 国界线 | `country-boundaries.js` | `lineAltitudeOffset` | 禁用 raycast | 只保证压过高清材质。 |
| 2.2 | 国界线 | `country-boundaries.js` | `lineAltitudeOffset = 0.115` | `depthTest: true`,禁用 raycast | 略高于高清材质 `0.10`,低于地形基准 `0.16`,减少悬浮感;地形 `depthWrite: false`,所以地形开启时仍可见。 |
| 2.29 | 国界 hover 光晕 | `country-boundaries.js` | hover 半径加 glow 偏移 | `depthTest: false`,禁用 raycast | 用 additive 光晕增强交界边和地形开启时的 hover 可见性。 |
| 2.3 | 国界 hover 实线 | `country-boundaries.js` | `hoverAltitudeOffset` | `depthTest: false`,禁用 raycast | 霓虹红橘 hover 线;中国和中国(台湾)共享高亮组。 |
| 3 | 卫星 footprint 填充 | `satellites.js` | `GROUND_FOOTPRINT_RENDER_ORDER` | depth-testedGroup renderOrder 保持 0 | Footprint 在国界线之上,但在算力中心和卫星之下。 |
| 3-5 | BGP 标记和覆盖层 | `bgp.js` | 各 marker 自身 renderOrder | BGP 拾取路径 | 保持现有 BGP 视觉层级。 |
| 4.5 | 算力中心 | `compute-centers.js` | `COMPUTE_CENTER_RENDER_ORDER` | 算力中心拾取路径 | 地表设施,保持在卫星下方。 |
| 2.3 | 国界 hover 实线 | `country-boundaries.js` | `hoverAltitudeOffset = 0.14` | `depthTest: false`,禁用 raycast | 霓虹红橘 hover 线;中国和中国(台湾)共享高亮组。 |
| 3 | 卫星 footprint 填充 / Iridium coverage ring | `satellites.js`, `iridium-footprint-adapter.js` | `GROUND_FOOTPRINT_RENDER_ORDER` | depth-testedIridium adapter 的 fill / ring 也使用同一 renderOrder | Footprint 在 land / texture / terrain 和国界线之上,但在算力中心和卫星之下。 |
| 3-4.5 | BGP 观测站、事件扩散圈和事件 marker | `bgp.js`, `interactable.js` | BGP 观测站和事件 marker 均使用 `Interactable` 批量 `THREE.Points`;事件 marker 使用 `BGP_EVENT_RENDER_ORDER = 4.5`;观测站主图标使用 `BGP_COLLECTOR_RENDER_ORDER = 4.4``BGP_CONFIG.collectorAltitudeOffset = 0.2`;事件 overlay 进入 `bgp-event-overlay-layer`;观测站 halo 和覆盖扇形进入 `bgp-collector-radar-layer` | BGP 事件和观测站都通过 `Interactable` 屏幕空间 picking并参与同坐标避让 | BGP 观测站主图标与船只同层BGP 事件与算力中心同层;向外扩散圈、观测站雷达/覆盖动画继续由 BGP 业务逻辑驱动。 |
| 4.3 | AIS 船只轨迹线 | `vessels.js` | `VESSEL_RENDER_ORDER - 0.1``CONFIG.earthRadius + VESSEL_CONFIG.track.altitudeOffset` | 跟随船只显隐,不单独参与拾取 | 选中船只后显示最近轨迹,低于船只 marker。 |
| 4.4 | AIS 船只 marker | `vessels.js`, `interactable.js` | `VESSEL_RENDER_ORDER`;业务高度为 `CONFIG.earthRadius + VESSEL_CONFIG.altitudeOffset`;普通 marker 为分桶 `THREE.Points`hover / locked 为单点 `THREE.Points` overlay | `depthTest: true``main.js` 使用屏幕空间 picking只取正面 marker参与 Interactable 同坐标避让 | 航行船只用三角点纹理,停泊/低速用圆点;普通态无 glow交互态叠加同尺寸 glow低于算力中心 `4.5`。 |
| 4.5 | 算力中心 | `compute-centers.js`, `interactable.js` | 使用 `COMPUTE_CENTER_RENDER_ORDER` 并由 `Interactable` 绘制 | 通过 `Interactable` 屏幕空间 picking参与同坐标避让 | 地表设施层,保持在卫星下方。登陆点已下沉到海缆层。 |
| 5 | 卫星背景点 | `satellites.js` | 固定 renderOrder | 屏幕空间卫星拾取 | 位于卫星点下方。 |
| 6 | 卫星点 | `satellites.js` | 固定 renderOrder | 屏幕空间卫星拾取 | 卫星点压过 footprint 和算力中心。 |
| 12+ | 卫星锁定 ring、halo、预测轨道 | `satellites.js` | `SATELLITE_CONFIG.overlayRenderOrder` 及偏移 | 卫星覆盖层路径 | 用于选中 / 锁定卫星强调。 |
@@ -43,7 +45,7 @@
| 高清材质 on | 恢复高清材质,并恢复记住的地形 / 昼夜状态。 |
| 地形 on | 显示在高清材质之上,但低于国界 hover、footprint、卫星等强调层。 |
| 大气云图 | 只控制云图 mesh 显隐。 |
| 国界 | 控制国界线和 hover 线显隐;海陆基座填充独立存在,作为 Earth 基座地图使用。 |
| 国界线 off | 只隐藏可交互国界线和 hover,高亮状态会清除;海陆基座填充作为 Earth 底图保留。 |
## 交互规则
@@ -55,3 +57,4 @@
| 中国 / 台湾 hover | `CHN``TWN` 被归到同一个 hover 高亮组tooltip 仍显示鼠标实际命中的 feature。 |
| 地形 | 只作为视觉层参与,`terrain.raycast` 已禁用。 |
| 卫星 | 使用屏幕空间卫星拾取,避免 footprint 或地表层挡住卫星点击。 |
| 船只 | 不使用对象级 sprite raycast。`main.js` 会在拖动 / 惯性期间跳过 hover picking平时将正面船只投影到屏幕坐标用像素半径命中最近船只点击后可加载轨迹线。 |

View File

@@ -1,11 +1,11 @@
# Earth Satellite Footprint Policy
# Earth 卫星覆盖策略
本文件记录 Earth 卫星图层当前关于 `footprint` 的产品边界、资料依据和已落地实现,目标是避免把 Starlink 这套专用地表覆盖模型误用到其它星座上。
相关上下文:
- [earth-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/earth-frontend-context.md)
- [backend-collectors.md](/home/ray/dev/linkong/planet/docs/technical/backend-collectors.md)
- [Earth 前端结构](/home/ray/dev/linkong/planet/docs/technical/zh/earth-frontend-context.md)
- [数据采集系统](/home/ray/dev/linkong/planet/docs/technical/zh/backend-collectors.md)
- [backend/app/services/collectors/celestrak.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/celestrak.py)
- [frontend/public/earth/js/satellites.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/satellites.js)
@@ -188,11 +188,3 @@
- 非 Starlink 的能力判断属于“策略层 / 适配层”
- 不要把不同星座的覆盖模型再混写进同一套参数里
- `iridium-next` 已经切成独立 adapter应继续沿这条边界演进而不是给现有 Starlink bowtie 增加更多 if/else
## 后续建议
如果继续往前做,推荐顺序是:
1.`iridium-next` 新建独立 footprint adapter
2. 在 UI 上补一个只读提示,让用户知道当前卫星是否支持 footprint
3. 如果未来拿到 GEO beam contour / operator metadata再为 GEO 开 operator-specific footprint

View File

@@ -1,11 +1,11 @@
# Admin Frontend Context
# 控制台前端结构
本文件描述当前控制台前端的真实结构,目标是帮助后续页面开发、表格改造、布局治理和状态收口时快速找到正确入口。
相关规则建议一起参考:
- [rules.md](/home/ray/dev/linkong/planet/rules.md)
- [frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md)
- [项目规则](/home/ray/dev/linkong/planet/rules.md)
- [前端布局指南](/home/ray/dev/linkong/planet/docs/technical/zh/frontend-layout-guidelines.md)
## 当前目标
@@ -208,7 +208,7 @@
- 先通过 port/types 定义边界
- 再由 http/mock gateway 实现
## 当前页面分层建议
## 当前页面分层
### 1. 仪表盘和摘要型页面
@@ -237,6 +237,63 @@
- 不要让表格撑爆整页
- 新表格区域优先复用 `TableScrollRegion` / `ScrollbarOverlay`
### 数据源目录页
[DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/DataSources/DataSources.tsx) 当前不再承担配置编辑职责,而是数据源目录和采集操作页。
当前页面边界:
- 内置数据源和自定义数据源合并为 `UnifiedDataSource`
- 列表展示类型、状态、最近运行、采集进度和操作。
- 点击名称打开只读抽屉。
- 抽屉中明确显示“内置数据源”或“自定义数据源”。
- endpoint、headers、config 只展示,不在这里编辑。
- 需要凭证的采集器提示用户到“设置 -> 采集器设置”维护。
这个边界很重要:后续不要把自定义数据源编辑、内置 endpoint 覆盖或凭证表单再塞回 `/datasources`。这些配置入口统一放在 `/settings?tab=collector_credentials`
页面顶部的总进度区域新增 `采集中 N` 标签:
- 仅在存在运行中采集任务时显示。
- 样式定义在 [index.css](/home/ray/dev/linkong/planet/frontend/src/index.css) 的 `data-source-bulk-toolbar__running-pill`
- 点击后打开 `采集中任务` Modal。
- Modal 内展示每个运行任务的阶段、进度、已处理/总数。
这个标签和其他状态标签同排,但通过 hover、蓝色描边和箭头表示可交互不应改成普通 Tag。
### 采集器设置页
[Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Settings/Settings.tsx) 中的 `collector_credentials` tab 当前显示为“采集器设置”。
当前页面边界:
- 下拉框选择所有内置采集器。
- 下拉框右侧只有一个插头图标按钮,用于健康检查。
- 状态标签在选择器下方展示 `未检查` / `可用` / `不可用`
- 需要凭证的采集器将凭证卡片放在基础配置上方。
- 不需要凭证的采集器只显示基础配置endpoint、默认 endpoint、请求头、timeout、retry。
- `BarentsWatch AIS` 使用专用凭证表单。
连接图标使用内联 `PlugConnectIcon`,视觉语义来自 Tabler `plug-connected`。后续如果控制台重写图标体系,应迁移到 Tabler Icons而不是继续使用 Ant Design 刷新图标表达连接。
`Client Secret` 的表单语义:
- 已配置时,输入框显示脱敏 preview。
- 聚焦且当前值等于 preview 时清空,方便输入新 secret。
- 保存时如果值仍等于 preview提交空值表示保留原 secret。
- 不再提供单独的“清除当前 secret”复选框。
凭证教程 Modal
- `GET /api/v1/settings/credential-guides/{provider}` 读取教程。
- `POST /generate` 调用 AI Provider 重新生成教程。
- `POST /reset` 恢复默认教程。
- Modal 使用 `MarkdownRenderer` 渲染教程正文。
相关后端设计见:
- [数据源、采集器设置与连接验证](/home/ray/dev/linkong/planet/docs/technical/zh/datasource-collector-settings-connectivity.md)
### 3. 复杂工作区页面
例如:
@@ -263,31 +320,4 @@
详细经验见:
- [frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md)
## 当前推荐改动方式
如果后续继续改后台页面,建议按这个顺序:
1. 先确认页面属于摘要页、表格页还是复杂工作区
2. 先接入现有壳层和滚动语义
3. 优先复用共享滚动组件
4. 最后再改视觉和细节交互
不要先写局部 CSS 补丁,再回头补结构。
## 当前明显边界
控制台前端和 Earth 前端不是一套系统:
- 控制台前端是 React + Ant Design 工作台
- Earth 前端是 `public/earth` 下的独立原生 HUD 系统
因此:
- 不要把 Earth 的 HUD/动画/状态机直接挪进控制台
- 不要把控制台表格/滚动策略硬套到 Earth HUD
Earth 相关结构见:
- [earth-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/earth-frontend-context.md)
- [前端布局指南](/home/ray/dev/linkong/planet/docs/technical/zh/frontend-layout-guidelines.md)

View File

@@ -1,4 +1,4 @@
# Frontend Layout Guidelines
# 前端布局指南
本项目后台页面默认遵循“单屏工作区”布局规范。目标不是让页面永远不溢出,而是确保在常见桌面视口下:

View File

@@ -7,7 +7,7 @@
- 控制台:登录后的管理后台
- Docs公开开发文档与使用手册
快速启动路径见 [quickstart.md](/home/ray/dev/linkong/planet/docs/technical/quickstart.md)。
快速启动路径见 [快速开始](/home/ray/dev/linkong/planet/docs/technical/zh/quickstart.md)。
## 入口总览
@@ -55,6 +55,52 @@
| `--allow-lan` | 允许局域网访问 |
| `--verbose` | 在执行过程中显示更多命令输出 |
### AI Provider 环境变量和构建
AI Provider 的运行期配置可以放在两处:
| 位置 | 适合内容 | 说明 |
| --- | --- | --- |
| `aiprovider/.env` | 团队约定的本地默认配置 | Docker Compose 会作为 `env_file` 读取 |
| `~/.zshrc` | 个人机器上的 provider、模型、密钥和代理变量 | `planet.sh` 启动时会读取常见的 `AI_*``SERVICE_*``PYTHON_IMAGE``UV_IMAGE`、代理变量 |
推荐写法:
```bash
export AI_PROVIDER=minimax
export AI_PROVIDER_API=anthropic-messages
export AI_BASE_URL=https://api.example.com/anthropic
export AI_API_KEY=sk-change-me
export AI_MODEL=MiniMax-M2.7
export AI_PROVIDER_SERVICE_TOKEN=change_me
```
默认情况下,`planet.sh` 只静态解析 `~/.zshrc` 中简单的 `export KEY=value``KEY=value` 行,避免 shell 主题、插件或交互初始化拖慢启动。如果变量依赖复杂 shell 展开,可以显式启用 source 模式:
```bash
PLANET_LOAD_ZSHRC_ENV=source ./planet.sh start -a
```
如需完全忽略 `~/.zshrc`
```bash
PLANET_LOAD_ZSHRC_ENV=0 ./planet.sh start -a
```
AI Provider 镜像只在代码、Dockerfile、Compose 配置或相关 Python 依赖变化时重建。修改 `aiprovider/.env``~/.zshrc` 中的模型、密钥、Base URL 不会触发镜像重建;重启 AI Provider 即可让容器读取新配置:
```bash
./planet.sh restart -a
```
构建较慢时,优先判断当前卡在哪一层:
| 现象 | 常见原因 | 处理方式 |
| --- | --- | --- |
| `transferring context` 很大 | Docker build context 包含前端资源、PDF、数据目录等无关文件 | 当前仓库通过 `.dockerignore` 只发送 AI Provider 必需文件 |
| `uv sync` 下载依赖较慢 | 首次构建或缓存为空,网络访问 Python 包较慢 | 等待首次构建完成;后续会复用 BuildKit 的 uv 下载缓存 |
| 改密钥后仍显示旧配置 | 容器尚未重启 | 执行 `./planet.sh restart -a` |
### 停止
```bash
@@ -178,7 +224,7 @@ Earth 用于在一个地球视图中观察:
- 卫星和轨迹
- 海缆与登陆点
- 算力中心
- 国界、经纬线、高清材质、云图、地形
- 国界线、经纬线、高清材质、云图、地形
- 新闻直播和态势新闻
- 搜索和聚焦对象详情
@@ -189,13 +235,14 @@ Earth 用于在一个地球视图中观察:
常见图层包括:
- 经纬线
- 国界
- 国界线
- 高清材质
- 大气云图
- 海缆
- 算力中心
- BGP 观测
- 卫星
- AIS 船只
- 轨迹
- 地形
@@ -205,6 +252,31 @@ Earth 用于在一个地球视图中观察:
- 轨迹依赖卫星
- 高清材质关闭时,地球会显示基座地图和边缘识别效果
### 图例
左下角图例会跟随当前聚焦或启用的图层切换。
当前已覆盖:
- 海缆
- 卫星
- 国界线
- 算力中心
- BGP
- AIS 船只
AIS 船只图例按船型显示颜色:
- 货轮
- 油轮
- 客船
- 渔船
- 军舰
- 停泊/低速
- 其他船只
船只图例中的三角形对应地图上的航行船只标记,圆点对应停泊或低速状态。
### 搜索
Earth 搜索支持查找当前地球对象,例如:
@@ -233,6 +305,25 @@ Earth 搜索支持查找当前地球对象,例如:
这些设置会保存在浏览器本地存储中。换浏览器或清理站点数据后会恢复默认值。
### 视角控制
Earth 支持鼠标、触控板和触屏操作。
常用控制方式:
| 操作 | 作用 |
| --- | --- |
| 鼠标左键拖动 | 旋转地球 |
| 手指单指拖动 | 在触屏设备上旋转地球 |
| 鼠标滚轮 | 放大或缩小视角 |
| 双指捏合 | 在触屏设备上放大或缩小视角 |
| 缩放按钮 | 按固定步长调整缩放 |
| 点击缩放百分比 | 重置到默认缩放 |
缩放时,顶部胶囊会短暂显示当前缩放比例,例如 `缩放 180%`。这个提示只表示当前视角缩放,不代表数据加载进度;如果页面正在加载数据,加载提示优先显示,缩放提示不会打断加载状态。
拖动灵敏度会根据当前缩放自动调整。默认视角附近保持常规旋转速度;放大后拖动会逐步变细,适合检查某个区域、船只、卫星或 BGP 事件;缩小后拖动会略快,方便快速浏览全球态势。
### 巡航模式
巡航模式会让 Earth 自动轮播聚焦目标。
@@ -308,7 +399,7 @@ http://localhost:3000/admin
| --- | --- | --- |
| 仪表盘 | `/admin` | 系统概览 |
| Earth | `/earth` | 打开公开 Earth 页面 |
| 数据源 | `/datasources` | 管理数据源和触发采集 |
| 数据源 | `/datasources` | 查看数据源和触发采集 |
| 采集数据 | `/data` | 查看采集后的数据 |
| BGP 观测 | `/bgp` | 查看 BGP 专题数据 |
| 系统告警 | `/alerts/system` | 系统级告警 |
@@ -321,17 +412,21 @@ http://localhost:3000/admin
### 数据源
`/datasources` 用于查看和管理采集来源。
`/datasources` 用于查看采集来源和触发采集。当前页面是“数据源目录”,会把内置数据源和自定义数据源放在同一张列表里展示
常见操作:
- 查看数据源状态
- 触发采集
- 查看最近采集任务
- 调整配置项
- 打开详情抽屉查看 endpoint、请求头、基础配置和是否为内置数据源
如果 Earth 上某类对象缺失,通常先到这里确认数据源是否可用。
数据源列表中的名称点击后只打开信息抽屉,不再承担编辑入口。接口地址、凭证、请求头和自定义数据源配置统一到 `/settings` 的“采集器设置”里维护。
当有采集任务正在运行时,总体进度下方会出现 `采集中 N` 标签。这个标签和其他状态标签放在同一排,但带有可点击样式;点击后会弹出当前采集中任务列表,显示每个任务的阶段、进度和处理数量。
### 采集数据
`/data` 用于查看采集后的数据表。
@@ -369,10 +464,63 @@ http://localhost:3000/admin
- 系统设置
- 电视直播源配置
- 数据源相关配置入口
- 采集器设置
- 外部集成和 AI Provider 配置
具体可用配置取决于当前登录用户权限。
#### 采集器设置
`/settings?tab=collector_credentials` 当前显示为“采集器设置”。这里统一维护所有采集器的连接配置,而不是只维护凭证。
使用方式:
1. 在下拉框选择采集器。
2. 查看状态标签:
- `无需凭证` / `需要凭证`
- 所属模块
- `启用` / `禁用`
- `未检查` / `可用` / `不可用`
3. 点击下拉框右侧的插头图标执行健康检查。
4. 如果检查通过,状态会变为 `可用`
5. 修改 endpoint、请求头、超时或重试次数后保存。
对于免费且不需要凭证的采集器,连接检查会直接请求对应 endpoint。对于需要凭证的采集器连接检查会走对应凭证链路如果凭证或 endpoint 相比上次验证成功时发生变化,需要重新点击连接。
系统判断“已连接”的条件是:
- 当前配置已经成功采集过数据;或
- 当前配置已经点击过连接按钮并验证成功。
#### BarentsWatch AIS 凭证
`BarentsWatch AIS` 是需要凭证的内置采集器。选择该采集器后,凭证区域会显示在基础配置上方。
配置项:
- `Client ID`
- `Client Secret`
- `Endpoint`
如果已经配置过 secret输入框会显示脱敏预览。保存时如果保持这个脱敏预览不变系统会保留原 secret只有输入新的 secret 才会替换。
BarentsWatch AIS 支持从以下位置读取凭证:
1. 控制台采集器设置中保存的凭证。
2. 后端环境变量:
- `BARENTSWATCH_CLIENT_ID`
- `BARENTSWATCH_CLIENT_SECRET`
- 兼容历史拼写:`BARRENTSWATCH_CLIENT_ID``BARRENTSWATCH_CLIENT_SECRET`
3. `~/.zshrc` 中的同名 `export`
如果连接失败,页面会弹出凭证获取教程。教程支持:
- 查看默认教程。
- 点击“教程不好用”让 AI Provider 根据默认 prompt 重新生成教程。
- 点击“重置”恢复默认教程。
默认教程以 BarentsWatch 官方 tutorial 为准,并提醒 Live AIS 应选择 `AIS - API`,不是普通 `BarentsWatch - API`
### 系统日志
`/logs` 用于查看系统日志。若菜单中不可见,通常是当前用户角色没有权限。
@@ -397,7 +545,8 @@ http://localhost:3000/docs
当前公开内容来自:
```text
docs/technical/*.md
docs/technical/zh/*.md
docs/technical/en/*.md
```
Docs 支持:
@@ -480,9 +629,10 @@ source ~/.zshrc && bun run build
## 相关文档
- [quickstart.md](/home/ray/dev/linkong/planet/docs/technical/quickstart.md)
- [frontend-admin-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/frontend-admin-frontend-context.md)
- [earth-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/earth-frontend-context.md)
- [earth-layer-style-reference.md](/home/ray/dev/linkong/planet/docs/technical/earth-layer-style-reference.md)
- [backend-system-service-control.md](/home/ray/dev/linkong/planet/docs/technical/backend-system-service-control.md)
- [backend-collectors.md](/home/ray/dev/linkong/planet/docs/technical/backend-collectors.md)
- [快速开始](/home/ray/dev/linkong/planet/docs/technical/zh/quickstart.md)
- [控制台前端结构](/home/ray/dev/linkong/planet/docs/technical/zh/frontend-admin-frontend-context.md)
- [Earth 前端结构](/home/ray/dev/linkong/planet/docs/technical/zh/earth-frontend-context.md)
- [Earth 图层样式属性索引](/home/ray/dev/linkong/planet/docs/technical/zh/earth-layer-style-reference.md)
- [系统服务控制](/home/ray/dev/linkong/planet/docs/technical/zh/backend-system-service-control.md)
- [数据采集系统](/home/ray/dev/linkong/planet/docs/technical/zh/backend-collectors.md)
- [数据源、采集器设置与连接验证](/home/ray/dev/linkong/planet/docs/technical/zh/datasource-collector-settings-connectivity.md)

View File

@@ -0,0 +1,211 @@
# planet.sh 启动性能优化
## 背景
`planet.sh` 管理所有服务的启动/停止/重启。原有实现存在以下问题:
1. AI Provider 每次都重新构建(即使代码未变)
2. 杀端口速度极慢(最长等 45 秒)
3. 端口绑定检测用 Python 子进程(每次 ~300ms
4. 无参 `restart``restart -b` 行为不一致
## 问题一AI Provider 每次重建
### 根因
构建戳文件存放在 `/tmp/`WSL/Linux 重启后 `/tmp` 被清空,导致三个条件中的"戳文件非空"这一条始终不满足,进而判定需要重建:
```bash
# 三个条件必须同时成立才跳过重建
image_exists AND stamp_non_empty AND fingerprint_match
```
### 修复
将戳文件路径从 `/tmp/` 改到持久路径:
```bash
AI_PROVIDER_BUILD_STAMP_FILE="$HOME/.cache/planet/aiprovider_build.sha256"
```
写入时确保目录存在:
```bash
write_ai_provider_build_stamp() {
mkdir -p "$(dirname "$AI_PROVIDER_BUILD_STAMP_FILE")"
compute_ai_provider_build_fingerprint > "$AI_PROVIDER_BUILD_STAMP_FILE"
}
```
### fingerprint 计算提速
原实现对整个 `aiprovider/` 打 tar 包再算 SHA大目录下耗时可达数秒。改为 `find + stat`(只读文件元信息,不读内容):
```bash
compute_ai_provider_build_fingerprint() {
find aiprovider \
-type f \
! -path '*/__pycache__/*' \
! -name '.env' \
! -name '.env.*' \
! -name '*.pyc' \
! -name '*.pyo' \
| LC_ALL=C sort \
| xargs -r stat --format="%Y %s %n" 2>/dev/null
sha256sum docker-compose.yml docker-compose.simple.yml 2>/dev/null
python3 "$SCRIPT_DIR/scripts/compute_aiprovider_dependency_fingerprint.py" 2>/dev/null
}
```
速度提升约 10 倍大量小文件场景误报率相同mtime+size 变化 ≡ 文件被修改)。
`.env``.env.*` 被排除在 fingerprint 外。它们属于运行期配置,不应该因为修改模型、密钥或 Base URL 触发镜像重建。
### Docker build context 收敛
AI Provider 镜像只需要根目录的 `pyproject.toml``uv.lock``aiprovider/` 代码。仓库中还包含前端静态大图、PDF、历史数据和 Unreal 资料,如果 build context 使用整个仓库,`transferring context` 会浪费大量时间。
当前通过根目录 `.dockerignore` 收敛上下文:
```dockerignore
**
!pyproject.toml
!uv.lock
!aiprovider/
!aiprovider/**
aiprovider/.env
aiprovider/.env.*
!aiprovider/.env.example
```
Dockerfile 也从全仓复制改为只复制 AI Provider 代码:
```dockerfile
COPY pyproject.toml uv.lock /app/
RUN --mount=type=cache,target=/root/.cache/uv \
uv sync --frozen --no-dev
COPY aiprovider /app/aiprovider
```
`uv sync` 使用 BuildKit cache mount 后,首次构建仍可能受网络影响;后续构建会复用 `/root/.cache/uv`,依赖下载不再重复从零开始。
### 运行期配置来源
`planet.sh` 启动 AI Provider 前会生成临时 env-file并把它传给 Compose 或手动 `docker run` fallback。配置优先来自
1. `aiprovider/.env`
2. `~/.zshrc` 中简单的 `export AI_...=...``AI_...=...`
默认解析是静态的,只覆盖 AI Provider、镜像、代理相关变量避免执行交互 shell 初始化。如果确实需要复杂 shell 展开,可以显式启用:
```bash
PLANET_LOAD_ZSHRC_ENV=source ./planet.sh start -a
```
如果排查时需要忽略个人 shell 配置:
```bash
PLANET_LOAD_ZSHRC_ENV=0 ./planet.sh start -a
```
### 跳过重建的原理
fingerprint 一致时不执行 `docker compose build`,而是:
```bash
docker start planet_aiprovider # 启动已存在的容器,几秒内完成
```
`docker stop` 停容器,不删镜像;`cleanup_exit_containers` 删已退出容器,不删镜像。下次 `docker start` 会从现有镜像直接创建并启动容器。
## 问题二:杀端口速度慢
### 原因
`wait_for_port_release` 默认最多等 45 秒15 次 × 3 秒)。
### 修复
将后台进程清理场景的超时缩短至 3 秒TERM→1.5s→KILL→1.5s
```bash
PORT_RELEASE_ATTEMPTS=15
PORT_RELEASE_INTERVAL=0.2 # 每次等 0.2s,总计 3s
# cleanup_backend_processes / kill_port_if_requested
wait_for_port_release "$port" 15 0.2
```
`wait_for_port_release` 增加可选参数,允许不同场景使用不同超时:
```bash
wait_for_port_release() {
local port="$1"
local max_attempts="${2:-$PORT_RELEASE_ATTEMPTS}"
local interval="${3:-$PORT_RELEASE_INTERVAL}"
...
}
```
## 问题三:端口检测用 Python
### 原因
`can_bind_port``python3 -c "import socket..."` 检测端口,每次调用约 300ms。
### 修复
优先使用系统工具(~10msPython 作为兜底:
```bash
can_bind_port() {
local port="$1"
if command -v ss >/dev/null 2>&1; then
! ss -tlnH 2>/dev/null | awk '{print $4}' | grep -qE ":${port}$"
return
fi
if command -v lsof >/dev/null 2>&1; then
[ -z "$(lsof -tiTCP:"${port}" -sTCP:LISTEN 2>/dev/null)" ]
return
fi
python3 - "$port" <<'PY'
import sys, socket
p = int(sys.argv[1])
s = socket.socket()
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
try:
s.bind(("", p)); s.close(); sys.exit(0)
except OSError:
sys.exit(1)
PY
}
```
## 问题四restart 行为不一致
### 现象
- `restart -b`:停全部服务 → 检查 AI Provider fingerprint → 按需重建 → 启动
- `restart`(无参):停全部服务 → AI Provider 总是判定需要重建(因戳文件在 /tmp
### 修复
修复戳文件路径后,无参 `restart` 同样使用 `stop + start`fingerprint 检查正常生效,行为与 `restart -b` 完全一致。无需额外代码变更。
## 其他:移除不必要的 sleep
启动链路中两处 `sleep 3` 在实际已有健康检查覆盖的情况下多余,已移除:
- `start_backend_service`:数据库健康检查通过后的 `sleep 3`
- `restart_database_service`:重启后的等待 `sleep 3`
## 相关文件
- `planet.sh` — 全量修改
- `.dockerignore` — 收敛 AI Provider Docker build context
- `aiprovider/Dockerfile` — 只复制 AI Provider 代码,并为 `uv sync` 启用 BuildKit cache mount
- `docker-compose.yml` / `docker-compose.simple.yml` — 读取 `planet.sh` 生成的运行期 env-file
- `scripts/compute_aiprovider_dependency_fingerprint.py` — 依赖 fingerprint未改动

View File

@@ -1,6 +1,6 @@
# Quickstart
# 快速开始
这份 Quickstart 面向第一次启动 Planet 的开发者或演示操作者。目标是用最短路径把服务跑起来,并知道应该打开哪些入口。
这份快速开始面向第一次启动 Planet 的开发者或演示操作者。目标是用最短路径把服务跑起来,并知道应该打开哪些入口。
## 前置条件
@@ -24,6 +24,12 @@
- `aiprovider/.env`
- `frontend/.env.local`
AI Provider 的个人配置也可以放在 `~/.zshrc``planet.sh` 会读取简单的 `export AI_...=...``AI_...=...` 行,并在启动 AI Provider 时传给容器。修改模型、密钥或 Base URL 后,通常只需要重启 AI Provider
```bash
./planet.sh restart -a
```
## 1. 启动服务
在仓库根目录执行:
@@ -73,6 +79,7 @@ Earth 是公开页面,不需要登录。
- 地球正常显示
- 右侧图层控制可打开/关闭图层
- 搜索可以查找海缆、卫星、算力中心、BGP 事件
- 鼠标拖动、滚轮缩放和缩放百分比提示正常工作
- 设置面板可以切换巡航模式、日夜模式、卫星显示风格
## 4. 打开控制台
@@ -87,7 +94,7 @@ http://localhost:3000/admin
首次排查建议查看:
- `/datasources`:数据源配置和采集状态
- `/datasources`:数据源目录和采集触发;接口、请求头和凭证配置在 `/settings` 的“采集器设置”
- `/data`:已采集数据
- `/bgp`BGP 专题观测
- `/alerts/system`:系统告警
@@ -187,7 +194,7 @@ ss -ltnp | grep -E ':3000|:8000'
## 下一步
- 完整操作说明见 [manual.md](/home/ray/dev/linkong/planet/docs/technical/manual.md)
- 控制台结构见 [frontend-admin-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/frontend-admin-frontend-context.md)
- Earth 结构见 [earth-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/earth-frontend-context.md)
- 后端采集器见 [backend-collectors.md](/home/ray/dev/linkong/planet/docs/technical/backend-collectors.md)
- 完整操作说明见 [Planet 使用手册](/home/ray/dev/linkong/planet/docs/technical/zh/manual.md)
- 控制台结构见 [控制台前端结构](/home/ray/dev/linkong/planet/docs/technical/zh/frontend-admin-frontend-context.md)
- Earth 结构见 [Earth 前端结构](/home/ray/dev/linkong/planet/docs/technical/zh/earth-frontend-context.md)
- 后端采集器见 [数据采集系统](/home/ray/dev/linkong/planet/docs/technical/zh/backend-collectors.md)

View File

@@ -16,12 +16,23 @@
## Current Version
- `main` 当前主线历史推导到:`0.16.5`
- `dev` 当前开发分支历史推导到:`0.42.1`
- `dev` 当前开发分支历史推导到:`0.46.3`
## Timeline
| Version | Type | Branch | Commit | Summary |
| --- | --- | --- | --- | --- |
| `0.46.3` | bugfix | `dev` | `pending` | 优化 Starlink footprint 拖拽性能,避免旋转地球时重复重建覆盖网格,并恢复线缆点击呼吸动画 |
| `0.46.2` | bugfix | `dev` | `pending` | 修复 Earth 启动加载顺序、图层 localStorage 恢复、国界线底图语义、媒体面板、船只轨迹和 Iridium footprint 显示问题,并补充 AIS 聚合计划 |
| `0.46.1` | bugfix | `dev` | `pending` | 修复新增 Docs 技术文档未进前端白名单导致页面不可访问的问题,补齐英文文档并固化白名单/双语/裸文件标题检查 |
| `0.46.0` | feature | `dev` | `pending` | Earth 新增通用 Interactable 图标层统一船只、算力中心、BGP 事件/观测站交互图标,并优化登陆点与 toolbar 初始渲染 |
| `0.45.0` | feature | `dev` | `pending` | 新增采集任务 fetching 阶段量化进度,收敛 AI Provider 运行期环境注入和 Docker build context |
| `0.44.2` | bugfix | `dev` | `pending` | 补充 Earth 船只批量渲染、屏幕拾取、图层顺序、样式参考和性能计划状态文档 |
| `0.44.1` | bugfix | `dev` | `pending` | 优化 Earth 船只批量渲染性能,修复拖动卡顿、拾取错位、交互态方向/尺寸和地表压盖问题 |
| `0.44.0` | feature | `dev` | `pending` | 重构数据源目录与采集器设置,新增 BarentsWatch AIS 连接教程、Earth 船只/缩放体验优化和仪表盘前端重启 |
| `0.43.1` | bugfix | `dev` | `pending` | 修正全量 restart 后 AI Provider 启动提示语义,避免把预期未就绪描述成异常 |
| `0.43.0` | feature | `dev` | `pending` | 新增 Earth 船舶追踪、自定义数据源映射、外部集成配置中心、Markdown 渲染器增强,并整理规则/技能文档加载约束 |
| `0.42.2` | bugfix | `dev` | `pending` | Docs 中文模式补齐分组和文档标题翻译,并更新文档站品牌文案 |
| `0.42.1` | bugfix | `dev` | `pending` | 修正 release skill 的 feature 版本计算规则minor 进位时重置 patch 为 0 |
| `0.42.0` | feature | `dev` | `pending` | 新增公开 `/docs` 文档站、中英文技术/使用文档、搜索与主题切换,并补充公共组件复用和 Earth 无高清材质边缘提示 |
| `0.41.2` | improvement | `dev` | `pending` | 启动脚本新增 verbose 滚动日志与端口占用诊断Docker 构建支持镜像源覆盖,并修复 Earth 登陆点遮挡判断 |

View File

@@ -1,6 +1,6 @@
{
"name": "planet-frontend",
"version": "0.42.1",
"version": "0.46.3",
"private": true,
"packageManager": "bun@1",
"dependencies": {

View File

@@ -0,0 +1,3 @@
<svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" fill="#ffffff" viewBox="0 0 16 16">
<path d="M3.05 3.05a7 7 0 0 0 0 9.9.5.5 0 0 1-.707.707 8 8 0 0 1 0-11.314.5.5 0 0 1 .707.707zm2.122 2.122a4 4 0 0 0 0 5.656.5.5 0 1 1-.708.708 5 5 0 0 1 0-7.072.5.5 0 0 1 .708.708zm5.656-.708a.5.5 0 0 1 .708 0 5 5 0 0 1 0 7.072.5.5 0 1 1-.708-.708 4 4 0 0 0 0-5.656.5.5 0 0 1 0-.708zm2.122-2.12a.5.5 0 0 1 .707 0 8 8 0 0 1 0 11.313.5.5 0 0 1-.707-.707 7 7 0 0 0 0-9.9.5.5 0 0 1 0-.707zM6 8a2 2 0 1 1 2.5 1.937V15.5a.5.5 0 0 1-1 0V9.937A2 2 0 0 1 6 8z"/>
</svg>

After

Width:  |  Height:  |  Size: 562 B

View File

@@ -1,16 +1,6 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 128 128">
<!-- GPU cluster marker: database/cylinder stack icon. -->
<!-- Color: #2dd4bf (teal) per COMPUTE_CENTER_CONFIG.colors.gpu_cluster -->
<!-- States: normal, estimated (adds a "?" badge drawn separately at canvas level) -->
<!-- Outer cylinder: top ellipse cap + side rect + bottom half-ellipse -->
<!-- Inner groove ring: smaller cylinder shape overlaid at same color (subtle shape layering) -->
<g fill="#2dd4bf">
<rect x="46" y="46" width="36" height="28"/>
<ellipse cx="64" cy="46" rx="18" ry="8"/>
<path d="M 82,74 A 18,8 0 0,1 46,74 Z"/>
<rect x="52" y="58" width="24" height="6"/>
<ellipse cx="64" cy="58" rx="12" ry="4.5"/>
<path d="M 76,64 A 12,4.5 0 0,1 52,64 Z"/>
</g>
<svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" fill="#2dd4bf" viewBox="0 0 16 16">
<path fill-rule="evenodd" d="M3.904 1.777C4.978 1.289 6.427 1 8 1s3.022.289 4.096.777C13.125 2.245 14 2.993 14 4s-.875 1.755-1.904 2.223C11.022 6.711 9.573 7 8 7s-3.022-.289-4.096-.777C2.875 5.755 2 5.007 2 4s.875-1.755 1.904-2.223Z"/>
<path d="M2 6.161V7c0 1.007.875 1.755 1.904 2.223C4.978 9.71 6.427 10 8 10s3.022-.289 4.096-.777C13.125 8.755 14 8.007 14 7v-.839c-.457.432-1.004.751-1.49.972C11.278 7.693 9.682 8 8 8s-3.278-.307-4.51-.867c-.486-.22-1.033-.54-1.49-.972Z"/>
<path d="M2 9.161V10c0 1.007.875 1.755 1.904 2.223C4.978 12.711 6.427 13 8 13s3.022-.289 4.096-.777C13.125 11.755 14 11.007 14 10v-.839c-.457.432-1.004.751-1.49.972-1.232.56-2.828.867-4.51.867s-3.278-.307-4.51-.867c-.486-.22-1.033-.54-1.49-.972Z"/>
<path d="M2 12.161V13c0 1.007.875 1.755 1.904 2.223C4.978 15.711 6.427 16 8 16s3.022-.289 4.096-.777C13.125 14.755 14 14.007 14 13v-.839c-.457.432-1.004.751-1.49.972-1.232.56-2.828.867-4.51.867s-3.278-.307-4.51-.867c-.486-.22-1.033-.54-1.49-.972Z"/>
</svg>

Before

Width:  |  Height:  |  Size: 787 B

After

Width:  |  Height:  |  Size: 1.1 KiB

View File

@@ -0,0 +1,3 @@
<svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" fill="#f8fafc" viewBox="0 0 16 16">
<path d="M2 2a2 2 0 0 0-2 2v1a2 2 0 0 0 2 2h5.5v3A1.5 1.5 0 0 0 6 11.5H.5a.5.5 0 0 0 0 1H6A1.5 1.5 0 0 0 7.5 14h1a1.5 1.5 0 0 0 1.5-1.5h5.5a.5.5 0 0 0 0-1H10A1.5 1.5 0 0 0 8.5 10V7H14a2 2 0 0 0 2-2V4a2 2 0 0 0-2-2H2zm.5 3a.5.5 0 1 1 0-1 .5.5 0 0 1 0 1zm2 0a.5.5 0 1 1 0-1 .5.5 0 0 1 0 1z"/>
</svg>

After

Width:  |  Height:  |  Size: 399 B

View File

@@ -1,10 +1,3 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 128 128">
<!-- Supercomputer marker: flat-screen monitor with neck and base stand. -->
<!-- Color: #38bdf8 (sky-blue) per COMPUTE_CENTER_CONFIG.colors.supercomputer -->
<!-- States: normal, estimated (adds a "?" badge drawn separately at canvas level) -->
<g fill="#38bdf8">
<rect x="40" y="42" width="48" height="30" rx="7"/>
<rect x="58" y="74" width="12" height="8" rx="3"/>
<rect x="50" y="84" width="28" height="5" rx="2.5"/>
</g>
<svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" fill="#38bdf8" viewBox="0 0 16 16">
<path d="M1.5 0A1.5 1.5 0 0 0 0 1.5v7A1.5 1.5 0 0 0 1.5 10H6v1H1a1 1 0 0 0-1 1v3a1 1 0 0 0 1 1h14a1 1 0 0 0 1-1v-3a1 1 0 0 0-1-1h-5v-1h4.5A1.5 1.5 0 0 0 16 8.5v-7A1.5 1.5 0 0 0 14.5 0h-13Zm0 1h13a.5.5 0 0 1 .5.5v7a.5.5 0 0 1-.5.5h-13a.5.5 0 0 1-.5-.5v-7a.5.5 0 0 1 .5-.5ZM12 12.5a.5.5 0 1 1 1 0 .5.5 0 0 1-1 0Zm2 0a.5.5 0 1 1 1 0 .5.5 0 0 1-1 0ZM1.5 12h5a.5.5 0 0 1 0 1h-5a.5.5 0 0 1 0-1ZM1 14.25a.25.25 0 0 1 .25-.25h5.5a.25.25 0 1 1 0 .5h-5.5a.25.25 0 0 1-.25-.25Z"/>
</svg>

Before

Width:  |  Height:  |  Size: 520 B

After

Width:  |  Height:  |  Size: 578 B

View File

@@ -0,0 +1,3 @@
<svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" fill="currentColor" class="bi bi-geo-fill" viewBox="0 0 16 16">
<path fill-rule="evenodd" d="M4 4a4 4 0 1 1 4.5 3.969V13.5a.5.5 0 0 1-1 0V7.97A4 4 0 0 1 4 3.999zm2.493 8.574a.5.5 0 0 1-.411.575c-.712.118-1.28.295-1.655.493a1.319 1.319 0 0 0-.37.265.301.301 0 0 0-.057.09V14l.002.008a.147.147 0 0 0 .016.033.617.617 0 0 0 .145.15c.165.13.435.27.813.395.751.25 1.82.414 3.024.414s2.273-.163 3.024-.414c.378-.126.648-.265.813-.395a.619.619 0 0 0 .146-.15.148.148 0 0 0 .015-.033L12 14v-.004a.301.301 0 0 0-.057-.09 1.318 1.318 0 0 0-.37-.264c-.376-.198-.943-.375-1.655-.493a.5.5 0 1 1 .164-.986c.77.127 1.452.328 1.957.594C12.5 13 13 13.4 13 14c0 .426-.26.752-.544.977-.29.228-.68.413-1.116.558-.878.293-2.059.465-3.34.465-1.281 0-2.462-.172-3.34-.465-.436-.145-.826-.33-1.116-.558C3.26 14.752 3 14.426 3 14c0-.599.5-1 .961-1.243.505-.266 1.187-.467 1.957-.594a.5.5 0 0 1 .575.411z"/>
</svg>

After

Width:  |  Height:  |  Size: 953 B

View File

@@ -1668,6 +1668,12 @@ label.is-disabled.earth-mobile-settings-card {
opacity: 1;
}
.earth-status-message.gesture {
min-width: 0;
padding-right: calc(16px * var(--hud-scale));
color: #dcecff;
}
.earth-error-message {
top: calc(62px * var(--hud-scale));
z-index: 211;

View File

@@ -121,6 +121,19 @@
box-shadow: 0 0 4px currentColor;
}
.legend-dot--vessel {
width: 0;
height: 0;
border-radius: 0;
background: transparent !important;
border-left: calc(5px * var(--hud-scale)) solid transparent;
border-right: calc(5px * var(--hud-scale)) solid transparent;
border-bottom: calc(13px * var(--hud-scale)) solid currentColor;
color: inherit;
box-shadow: none;
transform: rotate(45deg);
}
.legend-label {
color: var(--hud-text);
font-size: calc(0.78rem * var(--hud-scale));

Some files were not shown because too many files have changed in this diff Show More