Compare commits

..

2 Commits

Author SHA1 Message Date
linkong
421234301a release: bump version to 0.47.0 2026-04-30 16:56:37 +08:00
linkong
f22079d33a release: bump version to 0.46.3 2026-04-30 14:46:19 +08:00
43 changed files with 2547 additions and 509 deletions

View File

@@ -1,232 +1,93 @@
---
description: 分析本次 git 变更,在 docs/technical/zh/ 中新建或更新对应的技术文档
argument-hint: 可选:指定要记录的主题,或留空自动从 git diff 推断
description: Create or update repository documentation from current code changes
argument-hint: Optional: topic to document, or leave empty to infer from git diff
allowed-tools: ["Read", "Edit", "Write", "Bash", "Glob", "Grep"]
---
# /docs — 技术文档写入工作流
# /docs — Documentation Workflow
## 目标
## Goal
根据当前 git 变更(或用户指定主题)在 `docs/technical/zh/` 中写入或更新技术文档,记录**为什么**这样做,而不只是记录做了什么。
Create or update documentation that explains why a change exists, how it behaves, and what maintainers need to know. Keep this command generic. Repository-specific coverage rules live in the repository and must be loaded separately.
## 执行步骤
## Repository Rules
### Step 1 — 理解变更范围
Before deciding scope, check whether the repository has a documentation rules file:
```bash
git diff HEAD --stat # 变更文件一览
git diff HEAD --name-only # 变更文件列表
git log --oneline -10 # 近期 commit 上下文
test -f docs/documentation-coverage-rules.md && sed -n '1,240p' docs/documentation-coverage-rules.md
```
`$ARGUMENTS` 指定了主题,优先聚焦该主题;否则从文件列表和 diff stat 推断变更主题。不要默认读取完整仓库 diff只对决定文档主题所需的文件读取 focused diff
If it exists, apply it as the project-specific coverage checklist. If it does not exist, continue with the generic workflow below.
## Workflow
### Step 1 — Understand The Change
```bash
git diff HEAD --stat
git diff HEAD --name-only
git log --oneline -10
rg --files docs
```
If `$ARGUMENTS` specifies a topic, focus on that topic. Otherwise infer the documentation topic from the changed files. Do not read the full repository diff by default; inspect focused files only:
```bash
git diff HEAD -- <path>
rg -n "class |def |function |export |router|@router|interface |type " <path>
```
### Step 2 — 确认文档范围
### Step 2 — Decide Scope
分析变更,判断:
- Prefer updating an existing relevant document over creating a duplicate.
- Use one document for one coherent topic.
- Split documents only when the change crosses meaningful domains.
- Keep filenames lowercase and hyphenated.
- Apply the repository-specific rules file before writing.
1. **应写几篇文档**:单一主题写一篇,跨领域变更可拆分(如后端性能优化 + 运维启动脚本分开写)
2. **是新建还是更新**:检查 `docs/technical/zh/` 中是否已有相关文档
3. **文档命名**:按 `领域-主题-副题.md` 格式,全小写,用连字符,如:
- `backend-datasources-api-performance.md`
- `ops-planet-sh-startup.md`
- `earth-bgp-context.md`
For ambiguous or large documentation changes, briefly state the intended doc plan before editing. For clear small changes, proceed directly.
```bash
ls docs/technical/zh/ # 查看现有文档
### Step 3 — Write
Explain:
- Background/problem: what was wrong or missing before.
- Core design decisions and rationale.
- Operational or user-facing impact.
- Relevant code paths, only when useful for future maintainers.
Style:
- Follow the repositorys existing language and heading conventions.
- Use fenced code blocks with language tags.
- Prefer tables for comparisons or parameter lists.
- Keep snippets concise and relevant.
### Step 4 — Verify
- Read the completed docs once for clarity and stale statements.
- Verify referenced paths exist with `test -e` or `rg --files`.
- Run applicable checks from `docs/documentation-coverage-rules.md`.
- Check Markdown links use readable user-facing titles unless repository rules allow otherwise.
### Step 5 — Report
Summarize changed docs and verification:
```md
Updated:
- path/to/doc.md — what changed
Verified:
- checks that passed
- checks that could not be run, if any
```
**先输出写作计划供用户确认**(若变更明确且范围小,可直接执行):
## Hard Constraints
```
文档计划:
新建docs/technical/zh/ops-planet-sh-startup.md — planet.sh 启动性能优化
更新docs/technical/zh/backend-datasources-api-performance.md — 补充并行化细节
```
### Step 2.5 — 覆盖范围检查
写文档前必须按变更类型检查配套文档,不要只更新一篇专题文档:
- 用户可见流程变化:更新 `docs/technical/zh/manual.md`,通常也更新 `docs/technical/zh/quickstart.md`
- `manual.md``quickstart.md` 这类用户手册存在英文版时,同步更新 `docs/technical/en/...`,至少避免英文版与中文版互相矛盾。
- 控制台页面职责、路由入口、表格/抽屉/设置页行为变化:更新 `docs/technical/zh/frontend-admin-frontend-context.md`
- Earth 前端行为、HUD、巡航、图层、图例、交互变化更新 `docs/technical/zh/earth-frontend-context.md`
- 新增 Earth 图层、调整 `renderOrder`、半径/高度偏移、深度策略、拾取策略、legend mode、图层面板顺序或启动加载顺序更新 `docs/technical/zh/earth-render-layer-order.md`
- Earth 图层视觉样式、颜色、图例符号语义变化:若影响样式索引,同步更新 `docs/technical/zh/earth-layer-style-reference.md`
- 采集器、数据源、凭证、设置页、连接检查、scheduler、后端 API 变化:更新相关后端文档,优先检查 `docs/technical/zh/backend-collectors.md` 和 datasource/settings 专题文档。
- 如果某个旧 plan 的假设已经被当前实现推翻,在对应 `docs/plans/*.md` 增加现状修正或更新该段,不要让计划文档继续给出相反方向。
- 新增 technical 文档后,如果需要被发现,更新 `docs/technical/zh/README.md`
- 如果 technical 文档需要在公开 Docs 页面显示,或从 technical README 链接进入,必须同步更新 `frontend/src/pages/Docs/docs-content.ts``DOCS_METADATA`。前端使用这份白名单,`docs/technical/{zh,en}/` 中存在 `.md` 文件并不会自动生成路由。
- 公开 technical 文档必须按同名文件维护中英文双语版本:`docs/technical/zh/<name>.md``docs/technical/en/<name>.md`。如果某篇文档刻意只保留单语,完成说明中必须明确写出原因。
- 对本次变更提取旧词做 stale search例如旧 tab 名、旧路由职责、旧认证假设、改名前 UI 文案:
```bash
rg -n "旧文案|旧路由职责|旧认证假设" docs/technical docs/plans
```
### Step 3 — 写文档
遵循以下原则:
**记录 WHY不只记录 WHAT**
- 好:`将戳文件从 /tmp 移到 ~/.cache/planet/,因为 WSL 重启后 /tmp 被清空`
- 差:`修改了 AI_PROVIDER_BUILD_STAMP_FILE 的值`
**必须包含的内容**
- 背景/问题:改动之前存在什么问题,为什么要改
- 核心设计决策及其理由
- 关键代码片段(用 diff 或 before/after 展示)
- 相关文件列表
**格式要求**
- 使用 `##``###` 分级,不要超过三级
- 代码块注明语言python / bash / typescript / sql
- 表格用于对比多个选项或列出参数
- 中文写作,技术术语保留英文原文
- `docs/technical/zh/` 中的文档不得用英文原文占位;如果存在 `docs/technical/en/` 对应文件,禁止逐字复制成中文文件
- 中文文档内部链接应指向 `docs/technical/zh/...`,除非明确引用英文专属文档
- 公开文档的 Markdown 链接显示文字应使用可读标题,不要直接暴露 `manual.md``earth-frontend-context.md` 这类裸文件名
**文档结构模板**
```markdown
# 标题(说明做了什么)
## 背景
为什么要做这个改动,改动前存在什么问题。
## 核心变更
### 子主题一
before/after 或决策说明 + 关键代码
### 子主题二
...
## 相关文件
- `path/to/file.py` — 简短说明
```
### Step 4 — 验证
- 读一遍写好的文档,确认逻辑清晰、代码片段无明显错误
-`rg --files``test -e` 确认文档中的文件路径在项目中真实存在,避免凭记忆判断:
- 检查中文文档没有误复制英文版:
```bash
python - <<'PY'
from pathlib import Path
same = []
for en in sorted(Path("docs/technical/en").glob("*.md")):
zh = Path("docs/technical/zh") / en.name
if zh.exists() and en.read_text() == zh.read_text():
same.append(en.name)
if same:
raise SystemExit("identical en/zh docs: " + ", ".join(same))
print("no identical en/zh docs")
PY
```
- 检查中文文档内部链接没有继续指向无语言目录:
```bash
rg -n "/home/ray/dev/linkong/planet/docs/technical/(?!zh|en)" docs/technical/zh --pcre2
```
- 检查公开文档链接已进入 Docs 前端白名单。凡是 `docs/technical/{zh,en}/README.md` 中链接到的 technical `.md`,都必须存在于 `DOCS_METADATA`
```bash
python - <<'PY'
import re
from pathlib import Path
metadata = Path("frontend/src/pages/Docs/docs-content.ts").read_text()
known = set(re.findall(r"'([^']+\.md)':\s*\{", metadata))
known.add("README.md")
missing = []
for readme in [Path("docs/technical/zh/README.md"), Path("docs/technical/en/README.md")]:
if not readme.exists():
continue
for href in re.findall(r"\]\(([^)]+\.md)\)", readme.read_text()):
path = Path(href)
if "docs/technical/" not in href:
continue
filename = path.name
if filename not in known:
missing.append(f"{readme}: {filename}")
if missing:
raise SystemExit("docs README links missing DOCS_METADATA: " + ", ".join(missing))
print("docs README links are whitelisted")
PY
```
- 检查公开文档双语同名文件齐备。除 `README.md` 外,所有白名单文档都应同时存在 zh/en 文件,除非本次说明中明确豁免:
```bash
python - <<'PY'
import re
from pathlib import Path
metadata = Path("frontend/src/pages/Docs/docs-content.ts").read_text()
filenames = sorted(set(re.findall(r"'([^']+\.md)':\s*\{", metadata)) - {"README.md"})
missing = []
for filename in filenames:
for lang in ("zh", "en"):
path = Path("docs/technical") / lang / filename
if not path.exists():
missing.append(str(path))
if missing:
raise SystemExit("missing bilingual docs: " + ", ".join(missing))
print("public docs have zh/en file pairs")
PY
```
- 检查公开文档里没有用裸 `.md` 文件名当链接标题。这个命令在 polished public docs 中应无输出:
```bash
rg -n "\[[^]]+\.md\]\(" docs/technical/zh docs/technical/en
```
```bash
# 对文档中提到的关键路径做快速验证
ls <mentioned_paths>
```
如需检查大量链接,优先用确定性提取:
```bash
rg -n "\]\(([^)]+)\)" docs/technical/zh/<doc>.md
```
### Step 5 — 完成确认
输出摘要:
```
✓ 新建docs/technical/zh/ops-planet-sh-startup.md约 xxx 字)
✓ 更新docs/technical/zh/backend-datasources-api-performance.md
```
## 注意事项
- 不要写流水账式的"改了 A、改了 B、改了 C",要写改动背后的约束和权衡
- 不要在文档中引用 PR 号、issue 号、或当前对话——这些会随时间失效
- 代码片段保持简洁,只保留说明问题的关键部分,省略无关样板代码
- 如果某个变更已有文档记录,优先在原文档中追加,而不是新建
- 公开 technical 文档没有注册 `DOCS_METADATA`Docs 页面不会显示;不要只创建 `.md` 文件就结束。
- 公开 technical 文档默认需要 zh/en 同名文件,不要只补一个语言版本。
- 链接可见文字使用文档标题或语义标题,不要使用裸文件名。
- 文档是给未来的开发者看的,假设读者熟悉项目但不了解这次改动的背景
- Do not leave placeholder docs.
- Do not duplicate bilingual files byte-for-byte.
- Do not reference PR numbers, issue numbers, or the current conversation unless explicitly requested.
- Do not write changelog-style lists without the reasoning and tradeoffs behind the change.
- Keep docs maintainable and concise.

View File

@@ -1,187 +1,72 @@
---
name: docs
description: Analyze current Planet repo changes and create or update technical documentation under docs/technical/zh. Use when the user asks to write docs, update technical docs, summarize implementation changes into documentation, or port the Claude docs-codex workflow into Codex.
description: Create or update repository documentation from current code changes. Use when the user asks to write docs, update docs, summarize implementation changes into docs, or check documentation coverage. Load repository-specific coverage rules from docs/documentation-coverage-rules.md when present.
---
# Docs
Use this skill when the user asks to create or update Planet technical documentation, especially under `docs/technical/zh/`.
Use this skill when the task is documentation work: creating, updating, checking, or summarizing docs for code or behavior changes.
## Goal
Write or update technical docs that explain why a change exists, not only what files changed.
Write documentation that explains why a change exists, how it behaves, and what maintainers need to know. Keep the skill generic; repository-specific rules belong in the repository, not in this skill.
Default target directory:
## Repository Rules
- `docs/technical/zh/`
Before deciding scope, check whether the repository has a documentation rules file:
```bash
test -f docs/documentation-coverage-rules.md && sed -n '1,240p' docs/documentation-coverage-rules.md
```
If it exists, apply it as the project-specific coverage checklist. If it does not exist, continue with the generic workflow below.
## Workflow
1. Gather change context:
1. Gather focused context:
```bash
git diff HEAD --stat
git diff HEAD --name-only
git log --oneline -10
ls docs/technical/zh/
rg --files docs
```
If the user gives a specific topic, focus on that topic. Otherwise infer the documentation topic from the file list and diff stat. Do **not** read the full repository diff by default; inspect focused diffs only for the files that define the doc topic:
If the user gives a topic, focus on that topic. Otherwise infer the doc topic from changed files. Avoid reading large full diffs by default; inspect focused files and symbols:
```bash
git diff HEAD -- <path>
rg -n "class |def |function |export |router|@router|interface |type " <path>
```
2. Decide document scope:
2. Decide scope:
- Use one document for one coherent topic.
- Split documents when the changes cross meaningful domains, such as backend performance and ops startup behavior.
- Prefer updating an existing relevant doc over creating a duplicate.
- Name new files as lowercase hyphenated `domain-topic-detail.md`, for example:
- `backend-datasources-api-performance.md`
- `ops-planet-sh-startup.md`
- `earth-bgp-context.md`
- Use one document for one coherent topic.
- Split documents only when changes cross meaningful domains.
- Keep filenames lowercase and hyphenated.
3. Apply the documentation coverage checklist before writing:
3. Write the doc:
- User-visible workflow changes must update `docs/technical/zh/manual.md` and usually `docs/technical/zh/quickstart.md`.
- If an English counterpart exists for user-facing docs such as `manual.md` or `quickstart.md`, update `docs/technical/en/...` enough that it does not contradict the Chinese source.
- Control console page responsibility changes must update `docs/technical/zh/frontend-admin-frontend-context.md`.
- Earth frontend behavior changes must update `docs/technical/zh/earth-frontend-context.md`.
- Earth layer additions, `renderOrder`, altitude/radius offsets, depth strategy, pointer picking, legend modes, or layer panel/startup ordering must update `docs/technical/zh/earth-render-layer-order.md`.
- Earth layer visual style or legend symbol/color semantics should also update `docs/technical/zh/earth-layer-style-reference.md` when that reference is affected.
- Collector, datasource, credential, settings, connectivity, scheduler, or API changes must update the relevant backend docs, especially `docs/technical/zh/backend-collectors.md` and any datasource/settings-specific doc.
- When a change turns an old plan assumption into current behavior, update the relevant `docs/plans/*.md` with a status note instead of leaving contradictory instructions.
- If adding a new technical document, add it to `docs/technical/zh/README.md` when it should be discoverable from the technical docs index.
- If a technical document should be visible in the public Docs page or linked from a technical README, register it in `frontend/src/pages/Docs/docs-content.ts` under `DOCS_METADATA`. The frontend uses this whitelist; files under `docs/technical/{zh,en}/` are not automatically routable.
- For every public technical doc, keep the bilingual file pair in sync by filename: `docs/technical/zh/<name>.md` and `docs/technical/en/<name>.md`. If the content is intentionally Chinese-only or English-only, state that intentionally in the final note.
- Search docs for stale terms introduced by the change, for example old tab names, old route responsibilities, obsolete auth assumptions, or renamed UI labels.
- Explain background/problem, design decisions, constraints, and operational impact.
- Keep code snippets short and directly relevant.
- List related files only when they help future maintainers navigate.
- Use the repositorys existing language, heading style, and naming conventions.
4. Write the doc in Chinese:
4. Verify:
- Write Chinese prose for `docs/technical/zh/`.
- Keep technical identifiers, API paths, config keys, code symbols, and standard product names in English where appropriate.
- Use `##` and `###` headings; avoid going deeper than three levels.
- Use fenced code blocks with language tags.
- Use tables when comparing options or listing parameters.
5. Required content:
- Background/problem: what was wrong before and why the change was needed.
- Core design decisions and rationale.
- Key code snippets, preferably before/after or focused excerpts.
- Related files and what each file contributes.
6. Verification:
- Read the completed doc and check that the reasoning is clear.
- Verify important referenced paths exist.
- Use `rg --files` or `test -e` for path existence instead of relying on memory.
- Run a quick duplicate-language check when editing bilingual docs:
```bash
python - <<'PY'
from pathlib import Path
same = []
for en in sorted(Path("docs/technical/en").glob("*.md")):
zh = Path("docs/technical/zh") / en.name
if zh.exists() and en.read_text() == zh.read_text():
same.append(en.name)
if same:
raise SystemExit("identical en/zh docs: " + ", ".join(same))
print("no identical en/zh docs")
PY
```
Also check that Chinese docs do not link to the old language-less technical docs path:
```bash
rg -n "/home/ray/dev/linkong/planet/docs/technical/(?!zh|en)" docs/technical/zh --pcre2
```
This command should return no matches.
Check that public docs are whitelisted in the frontend Docs registry. Any `.md` linked from `docs/technical/{zh,en}/README.md` and located under `docs/technical/{zh,en}/` must have a matching `DOCS_METADATA` key:
```bash
python - <<'PY'
import re
from pathlib import Path
metadata = Path("frontend/src/pages/Docs/docs-content.ts").read_text()
known = set(re.findall(r"'([^']+\.md)':\s*\{", metadata))
known.add("README.md")
missing = []
for readme in [Path("docs/technical/zh/README.md"), Path("docs/technical/en/README.md")]:
if not readme.exists():
continue
for href in re.findall(r"\]\(([^)]+\.md)\)", readme.read_text()):
path = Path(href)
if "docs/technical/" not in href:
continue
filename = path.name
if filename not in known:
missing.append(f"{readme}: {filename}")
if missing:
raise SystemExit("docs README links missing DOCS_METADATA: " + ", ".join(missing))
print("docs README links are whitelisted")
PY
```
Check bilingual parity for public docs. Every whitelisted document except `README.md` should exist in both language directories unless intentionally documented otherwise:
```bash
python - <<'PY'
import re
from pathlib import Path
metadata = Path("frontend/src/pages/Docs/docs-content.ts").read_text()
filenames = sorted(set(re.findall(r"'([^']+\.md)':\s*\{", metadata)) - {"README.md"})
missing = []
for filename in filenames:
for lang in ("zh", "en"):
path = Path("docs/technical") / lang / filename
if not path.exists():
missing.append(str(path))
if missing:
raise SystemExit("missing bilingual docs: " + ", ".join(missing))
print("public docs have zh/en file pairs")
PY
```
Check that Markdown links do not expose raw filenames as user-facing titles. This should return no matches for polished public docs:
```bash
rg -n "\[[^]]+\.md\]\(" docs/technical/zh docs/technical/en
```
If checking many links, prefer deterministic extraction:
```bash
rg -n "\]\(([^)]+)\)" docs/technical/zh/<doc>.md
```
Also run focused stale-term searches derived from the change, for example:
```bash
rg -n "old label|old route purpose|obsolete provider assumption" docs/technical docs/plans
```
- Read the completed doc once for clarity and stale statements.
- Verify important referenced paths exist with `test -e` or `rg --files`.
- Run repository-specific doc checks from `docs/documentation-coverage-rules.md` when present.
- For Markdown links, check that user-facing titles are readable and not raw filenames unless the repository rules allow it.
## Hard Constraints
- A file under `docs/technical/zh/` must not be an English source file copied as a placeholder.
- Do not leave a Chinese doc with only an English title and English first-screen content.
- When an English counterpart exists in `docs/technical/en/`, never duplicate it byte-for-byte into `docs/technical/zh/`.
- Internal links inside `docs/technical/zh/` should point to `docs/technical/zh/...` for Chinese docs, unless intentionally linking to an English-only file.
- Public technical documents must be registered in `frontend/src/pages/Docs/docs-content.ts` before considering them available in the Docs UI.
- Public technical documents should have both zh and en files with the same filename, unless intentionally exempted.
- Markdown link text in public docs should be a readable title, not a raw filename such as `manual.md`.
- Do not reference PR numbers, issue numbers, or the current conversation.
- Do not write changelog-style lists like "changed A, changed B, changed C" without the constraints and tradeoffs behind those changes.
- Keep code snippets concise and relevant.
- Do not leave placeholder docs or copied source text pretending to be documentation.
- Do not duplicate bilingual files byte-for-byte.
- Do not reference PR numbers, issue numbers, or the current conversation unless explicitly requested.
- Do not write changelog-style lists without the reasoning, constraints, and tradeoffs behind the change.
- Keep docs concise enough to maintain.
## Recommended Output
@@ -189,12 +74,9 @@ After editing, summarize:
```md
Updated:
- docs/technical/zh/example.md — what changed
- path/to/doc.md — what changed
Verified:
- no identical en/zh docs
- no language-less docs/technical links in zh docs
- public docs are registered in DOCS_METADATA
- public docs have zh/en file pairs
- no raw `.md` filenames as public link titles
- checks that passed
- checks that could not be run, if any
```

View File

@@ -26,6 +26,8 @@
- [ ] 重写控制台 UI逐步抛弃 Ant Design建立自有组件体系并统一采用 `tabler.io` / Tabler Icons 作为控制台主图标库
- [ ] 把 Earth 态势新闻源从 [earth_news.py](/home/ray/dev/linkong/planet/backend/app/services/earth_news.py) 的硬编码列表抽成可配置目录,优先保持当前“实时聚合”链路不变,只先解决新闻源不可配置的问题
- [ ] 为 Earth 态势新闻设计后续采集器化方案:明确新闻数据模型、去重策略、区域映射、过期清理和 Earth/AI 复用方式,再决定何时把新闻从实时抓取升级成正式 collector
- [ ] AIS v4开放船只多源聚合策略配置支持 source priority、字段级规则、freshness 窗口和高级保护开关;保存时校验未知字段、非法模式和危险动态字段锁定,并在聚合接口返回命中的配置版本
- [ ] AIS v5实现船舶资料 enrichment 与冲突治理,按 `mmsi + imo + name + callsign` 异步补充船型细分、AIS 大类、旗国、尺寸、建造年份、运营方和图片缓存;详情面板展示缓存资料和字段来源,不在实时 AIS 请求链路现场抓第三方页面
- [ ] 为 Earth 地球表面增加一层与基础纹理对齐的材质/纹理 overlay并在同层叠加国界轮廓参考线要求国界线与底图稳定对齐且 hover 到国家轮廓时能高亮当前国家,便于校准地表和增强交互
- [ ] 把 Earth 新闻接入通用巡航队列:按新闻发生地和时间排序生成巡航目标,巡航聚焦到新闻事件时显示对应新闻卡片,并保持实现边界为“通用巡航层 + 新闻业务适配层”,不要再把新闻逻辑直接耦合回 `main.js` 状态机
- [ ] 为未知位置的算力中心建立分层坐标补全链路:优先 `精确坐标 > 站点/园区命中 > 城市 > 州/省 > 国家内主要算力城市 > 国家质心`,并把每次回退的 `confidence / reason / precision` 明确写进统一 GeoJSON

View File

@@ -1 +1 @@
0.46.2
0.47.0

View File

@@ -5,7 +5,7 @@ from datetime import datetime
import base64
import json
import re
from fastapi import APIRouter, Depends, HTTPException, status
from fastapi import APIRouter, Depends, HTTPException
from sqlalchemy import select, func
from sqlalchemy.ext.asyncio import AsyncSession
from pydantic import BaseModel, Field
@@ -318,7 +318,7 @@ async def list_configs(
"""List all user-defined data source configurations"""
query = select(DataSourceConfig)
if active_only:
query = query.where(DataSourceConfig.is_active == True)
query = query.where(DataSourceConfig.is_active)
query = query.order_by(DataSourceConfig.created_at.desc())
result = await db.execute(query)
@@ -374,6 +374,11 @@ async def list_all_datasources(
"is_active": db_config.is_active if db_config else True,
"source_type": db_config.source_type if db_config else "http",
"auth_type": db_config.auth_type if db_config else "none",
"auth_configured": {
"api_key": bool((db_config.auth_config or {}).get("api_key"))
if db_config
else False,
},
"headers": db_config.headers if db_config else {},
"config": strip_connectivity_validation(db_config.config if db_config else {}),
"config_id": db_config.id if db_config else None,
@@ -464,6 +469,8 @@ async def update_config(
for field, value in update_data.items():
if field == "config":
value = strip_connectivity_validation(value)
if field == "auth_config" and value == {} and (config.auth_config or {}):
continue
setattr(config, field, value)
await db.commit()
@@ -601,6 +608,7 @@ async def connect_builtin_config(
config_data.headers,
config_data.config,
db,
config_data.auth_config,
)
if result.get("success") and result.get("checksum"):
validation = await save_connectivity_success(

View File

@@ -17,6 +17,7 @@ from app.models.datasource import DataSource
from app.models.datasource_config import DataSourceConfig
from app.models.system_setting import SystemSetting
from app.models.user import User
from app.models.vessel import AISSourceHealth
from app.services.barentswatch import (
BarentsWatchConfig,
check_barentswatch_config,
@@ -368,7 +369,12 @@ def format_frequency_label(minutes: int) -> str:
return f"{minutes}m"
def serialize_collector(datasource: DataSource) -> dict:
async def get_ais_source_health_by_source(db: AsyncSession) -> dict[str, dict]:
result = await db.execute(select(AISSourceHealth))
return {item.source: item.to_dict() for item in result.scalars().all()}
def serialize_collector(datasource: DataSource, ais_health_by_source: dict[str, dict] | None = None) -> dict:
defaults = DEFAULT_DATASOURCES.get(datasource.source, {})
return {
"id": datasource.id,
@@ -387,6 +393,7 @@ def serialize_collector(datasource: DataSource) -> dict:
"requires_credentials": bool(defaults.get("requires_credentials", False)),
"credential_provider": defaults.get("credential_provider"),
"credential_status": defaults.get("credential_status", "none"),
"ais_health": (ais_health_by_source or {}).get(datasource.source),
}
@@ -599,7 +606,8 @@ async def get_collector_settings(
):
result = await db.execute(select(DataSource).order_by(DataSource.module, DataSource.id))
datasources = result.scalars().all()
return {"collectors": [serialize_collector(datasource) for datasource in datasources]}
ais_health_by_source = await get_ais_source_health_by_source(db)
return {"collectors": [serialize_collector(datasource, ais_health_by_source) for datasource in datasources]}
@router.put("/collectors/{datasource_id}")
@@ -619,7 +627,8 @@ async def update_collector_settings(
await db.commit()
await db.refresh(datasource)
await sync_datasource_job(datasource.id)
return {"status": "updated", "collector": serialize_collector(datasource)}
ais_health_by_source = await get_ais_source_health_by_source(db)
return {"status": "updated", "collector": serialize_collector(datasource, ais_health_by_source)}
@router.get("")
@@ -633,12 +642,13 @@ async def get_all_settings(
db,
["system", "notifications", "security"],
)
ais_health_by_source = await get_ais_source_health_by_source(db)
return {
"system": setting_payloads["system"],
"notifications": setting_payloads["notifications"],
"security": setting_payloads["security"],
"tv": await get_tv_settings_payload(db),
"integrations": await serialize_external_integrations(db),
"collectors": [serialize_collector(datasource) for datasource in datasources],
"collectors": [serialize_collector(datasource, ais_health_by_source) for datasource in datasources],
"generated_at": to_iso8601_utc(datetime.now(UTC)),
}

View File

@@ -25,6 +25,14 @@ from app.services.bgp_collectors import build_bgp_collector_coverage
from app.services.cable_graph import build_graph_from_data, CableGraph, haversine_distance
from app.services.collectors.bgp_common import RIPE_RIS_COLLECTOR_COORDS
from app.services.persistent_logs import record_system_log
from app.services.vessel_ais_aggregation import (
build_field_conflict_candidates,
get_aggregated_vessel,
get_aggregated_vessel_track,
get_aggregated_vessels,
get_vessel_conflict_records,
get_vessel_raw_observations,
)
from app.core.logging import get_logger
router = APIRouter()
@@ -664,6 +672,54 @@ def convert_vessels_to_geojson(rows: List[Any]) -> Dict[str, Any]:
return {"type": "FeatureCollection", "features": features}
def convert_aggregated_vessels_to_geojson(vessels: List[dict[str, Any]]) -> Dict[str, Any]:
features = []
for vessel in vessels:
if vessel.get("lat") is None or vessel.get("lon") is None:
continue
source_summary = {}
for source, summary in (vessel.get("source_summary") or {}).items():
source_summary[source] = {
**summary,
"latest_observed_at": to_iso8601_utc(summary.get("latest_observed_at")),
}
props = {
"mmsi": vessel["mmsi"],
"name": vessel.get("name") or f"MMSI {vessel['mmsi']}",
"callsign": vessel.get("callsign"),
"imo": vessel.get("imo"),
"vessel_type": vessel.get("vessel_type"),
"vessel_type_name": vessel.get("vessel_type_name") or "Other",
"flag": vessel.get("flag"),
"length": vessel.get("length"),
"width": vessel.get("width"),
"draught": vessel.get("draught"),
"sog": vessel.get("sog"),
"cog": vessel.get("cog"),
"heading": vessel.get("heading"),
"nav_status": vessel.get("nav_status"),
"received_at": to_iso8601_utc(vessel.get("received_at")),
"field_sources": vessel.get("field_sources") or {},
"selected_reasons": vessel.get("selected_reasons") or {},
"source_summary": source_summary,
"quality_flags": vessel.get("quality_flags") or [],
"conflict_count": vessel.get("conflict_count", 0),
"data_type": "vessel",
}
features.append(
{
"type": "Feature",
"id": vessel["mmsi"],
"geometry": {
"type": "Point",
"coordinates": [vessel["lon"], vessel["lat"]],
},
"properties": props,
}
)
return {"type": "FeatureCollection", "features": features}
def _parse_bbox(value: Optional[str]) -> tuple[float, float, float, float] | None:
if not value:
return None
@@ -1411,10 +1467,37 @@ async def get_vessels_geojson(
None,
description="Comma-separated vessel types: cargo,tanker,passenger,fishing,military,other",
),
limit: int = Query(5000, ge=1, le=50000),
limit: Optional[int] = Query(
None,
ge=0,
description="Maximum vessel features to return. Omit or pass 0 for no limit.",
),
db: AsyncSession = Depends(get_db),
):
"""Return latest vessel positions as GeoJSON points."""
parsed_bbox = _parse_bbox(bbox)
aggregated_vessels = await get_aggregated_vessels(db, bbox=parsed_bbox, limit=limit)
if aggregated_vessels:
geojson = convert_aggregated_vessels_to_geojson(aggregated_vessels)
requested_types = {
item.strip().lower()
for item in (type or "").split(",")
if item.strip()
}
if requested_types:
geojson["features"] = [
feature
for feature in geojson.get("features", [])
if _matches_vessel_type(feature.get("properties", {}), requested_types)
]
features = geojson.get("features", [])
return {
**geojson,
"count": len(features),
"stats": _build_vessel_stats(features),
}
latest_times = (
select(
VesselPosition.mmsi.label("mmsi"),
@@ -1432,10 +1515,10 @@ async def get_vessels_geojson(
)
.outerjoin(VesselStatic, VesselStatic.mmsi == VesselPosition.mmsi)
.order_by(VesselPosition.received_at.desc())
.limit(limit)
)
if limit and limit > 0:
stmt = stmt.limit(limit)
parsed_bbox = _parse_bbox(bbox)
if parsed_bbox is not None:
lon_min, lat_min, lon_max, lat_max = parsed_bbox
stmt = stmt.where(
@@ -1470,6 +1553,15 @@ async def get_vessels_geojson(
@router.get("/vessels/{mmsi}")
async def get_vessel_detail(mmsi: int, db: AsyncSession = Depends(get_db)):
aggregated = await get_aggregated_vessel(db, mmsi)
if aggregated is not None:
return {
**aggregated,
"received_at": to_iso8601_utc(aggregated.get("received_at")),
"latitude": aggregated["lat"],
"longitude": aggregated["lon"],
}
latest_position_stmt = (
select(VesselPosition)
.where(VesselPosition.mmsi == mmsi)
@@ -1496,6 +1588,30 @@ async def get_vessel_track(
db: AsyncSession = Depends(get_db),
):
cutoff = datetime.now(UTC) - timedelta(hours=hours)
aggregated_points = await get_aggregated_vessel_track(db, mmsi, cutoff=cutoff)
if aggregated_points:
return {
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"geometry": {
"type": "LineString",
"coordinates": [[point["lon"], point["lat"]] for point in aggregated_points],
},
"properties": {
"mmsi": mmsi,
"hours": hours,
"point_count": len(aggregated_points),
"start_at": to_iso8601_utc(aggregated_points[0]["observed_at"]),
"end_at": to_iso8601_utc(aggregated_points[-1]["observed_at"]),
"point_sources": [point["source"] for point in aggregated_points],
},
}
],
"count": 1,
}
result = await db.execute(
select(VesselPosition)
.where(VesselPosition.mmsi == mmsi)
@@ -1532,6 +1648,37 @@ async def get_vessel_track(
}
@router.get("/vessels/{mmsi}/observations")
async def get_vessel_observations(
mmsi: int,
limit: int = Query(100, ge=1, le=500),
db: AsyncSession = Depends(get_db),
):
"""Return raw AIS observations for debugging source-level collector facts."""
observations = await get_vessel_raw_observations(db, mmsi, limit=limit)
return {
"mmsi": mmsi,
"count": len(observations),
"observations": [item.to_dict() for item in observations],
"conflict_candidates": build_field_conflict_candidates(observations),
}
@router.get("/vessels/{mmsi}/conflicts")
async def get_vessel_conflicts(mmsi: int, db: AsyncSession = Depends(get_db)):
"""Return recorded AIS conflicts plus current raw-observation candidates."""
records = await get_vessel_conflict_records(db, mmsi)
observations = await get_vessel_raw_observations(db, mmsi, limit=500)
return {
"mmsi": mmsi,
"count": len(records),
"conflicts": [item.to_dict() for item in records],
"candidates": build_field_conflict_candidates(observations),
}
@router.get("/geo/bgp-anomalies")
async def get_bgp_anomalies_geojson(
severity: Optional[str] = Query(None),

View File

@@ -1,7 +1,6 @@
import os
import yaml
from functools import lru_cache
from typing import Optional
COLLECTOR_URL_KEYS = {
@@ -32,6 +31,7 @@ COLLECTOR_URL_KEYS = {
"nro_delegated_prefix_geo": "nro.delegated_stats_url",
"news_live_streams": "news_live_streams.channels_url",
"barentswatch_vessels": "barentswatch_vessels.url",
"aisstream_vessels": "aisstream_vessels.url",
}
@@ -74,7 +74,7 @@ class DataSourcesConfig:
from app.models.datasource_config import DataSourceConfig
query = select(DataSourceConfig).where(
DataSourceConfig.name == collector_name, DataSourceConfig.is_active == True
DataSourceConfig.name == collector_name, DataSourceConfig.is_active
)
result = await db.execute(query)
db_config = result.scalar_one_or_none()

View File

@@ -98,3 +98,7 @@ news_live_streams:
barentswatch_vessels:
# BarentsWatch Live AIS latest combined endpoint. Requires an AIS bearer token.
url: "https://live.ais.barentswatch.no/v1/latest/combined"
aisstream_vessels:
# AISStream realtime WebSocket endpoint. Requires an AISStream API key.
url: "wss://stream.aisstream.io/v0/stream"

View File

@@ -245,6 +245,18 @@ DEFAULT_DATASOURCES = {
"credential_provider": "barentswatch",
"credential_status": "supported",
},
"aisstream_vessels": {
"id": 28,
"name": "AISStream Vessels",
"display_name": "AISStream 实时船舶",
"module": "L4",
"priority": "P1",
"frequency_minutes": 1,
"is_free": True,
"requires_credentials": True,
"credential_provider": "aisstream",
"credential_status": "supported",
},
}
ID_TO_COLLECTOR = {info["id"]: name for name, info in DEFAULT_DATASOURCES.items()}

View File

@@ -12,7 +12,7 @@ from app.models.system_setting import SystemSetting
from app.models.playground_session import PlaygroundSession
from app.models.playground_message import PlaygroundMessage
from app.models.system_log import SystemLog, AuditLog
from app.models.vessel import VesselPosition, VesselStatic
from app.models.vessel import AISConflictRecord, AISRawObservation, AISSourceHealth, VesselPosition, VesselStatic
from app.models.datasource_mapping import DataSourceMappingTemplate
__all__ = [
@@ -31,7 +31,12 @@ __all__ = [
"BGPObservation",
"SystemLog",
"AuditLog",
"PlaygroundSession",
"PlaygroundMessage",
"VesselPosition",
"VesselStatic",
"AISRawObservation",
"AISConflictRecord",
"AISSourceHealth",
"DataSourceMappingTemplate",
]

View File

@@ -1,6 +1,6 @@
"""Vessel AIS models for live maritime tracking."""
from sqlalchemy import BigInteger, Column, DateTime, Float, Index, Integer, SmallInteger, String
from sqlalchemy import BigInteger, Column, DateTime, Float, Index, Integer, JSON, SmallInteger, String
from sqlalchemy.sql import func
from app.core.time import to_iso8601_utc
@@ -73,3 +73,113 @@ class VesselPosition(Base):
"nav_status": self.nav_status,
"received_at": to_iso8601_utc(self.received_at),
}
class AISRawObservation(Base):
"""Source-level AIS fact before aggregation and conflict resolution."""
__tablename__ = "ais_raw_observations"
id = Column(Integer, primary_key=True, autoincrement=True)
target_schema = Column(String(64), nullable=False, default="vessel_ais", index=True)
source = Column(String(100), nullable=False, index=True)
entity_key = Column(String(64), nullable=False, index=True)
delivery_mode = Column(String(32), nullable=False, index=True)
transport = Column(String(32), nullable=False, index=True)
message_type = Column(String(64), nullable=True, index=True)
source_message_id = Column(String(128), nullable=True, index=True)
observation_hash = Column(String(64), nullable=False, unique=True, index=True)
observed_at = Column(DateTime(timezone=True), nullable=False, index=True)
collected_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now(), index=True)
normalized_payload = Column(JSON, default=dict)
raw_payload = Column(JSON, default=dict)
quality_flags = Column(JSON, default=list)
__table_args__ = (
Index("idx_ais_raw_entity_observed", "target_schema", "entity_key", "observed_at"),
Index("idx_ais_raw_source_entity", "source", "entity_key"),
)
def to_dict(self) -> dict:
return {
"id": self.id,
"target_schema": self.target_schema,
"source": self.source,
"entity_key": self.entity_key,
"delivery_mode": self.delivery_mode,
"transport": self.transport,
"message_type": self.message_type,
"source_message_id": self.source_message_id,
"observation_hash": self.observation_hash,
"observed_at": to_iso8601_utc(self.observed_at),
"collected_at": to_iso8601_utc(self.collected_at),
"normalized_payload": self.normalized_payload or {},
"raw_payload": self.raw_payload or {},
"quality_flags": self.quality_flags or [],
}
class AISConflictRecord(Base):
"""Recorded field-level disagreement between AIS sources."""
__tablename__ = "ais_conflict_records"
id = Column(Integer, primary_key=True, autoincrement=True)
target_schema = Column(String(64), nullable=False, default="vessel_ais", index=True)
entity_key = Column(String(64), nullable=False, index=True)
field = Column(String(64), nullable=False, index=True)
candidates = Column(JSON, default=dict)
selected_source = Column(String(100), nullable=True, index=True)
selected_value = Column(JSON, nullable=True)
selected_reason = Column(String(64), nullable=True, index=True)
resolved_by = Column(String(32), nullable=False, default="system", index=True)
status = Column(String(32), nullable=False, default="open", index=True)
created_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now(), index=True)
updated_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now())
__table_args__ = (
Index("idx_ais_conflict_entity_field", "target_schema", "entity_key", "field"),
)
def to_dict(self) -> dict:
return {
"id": self.id,
"target_schema": self.target_schema,
"entity_key": self.entity_key,
"field": self.field,
"candidates": self.candidates or {},
"selected_source": self.selected_source,
"selected_value": self.selected_value,
"selected_reason": self.selected_reason,
"resolved_by": self.resolved_by,
"status": self.status,
"created_at": to_iso8601_utc(self.created_at),
"updated_at": to_iso8601_utc(self.updated_at),
}
class AISSourceHealth(Base):
"""Runtime health signal for an AIS collector source."""
__tablename__ = "ais_source_health"
source = Column(String(100), primary_key=True)
connection_state = Column(String(32), nullable=False, default="disconnected", index=True)
last_seen_at = Column(DateTime(timezone=True), nullable=True, index=True)
last_success_at = Column(DateTime(timezone=True), nullable=True, index=True)
last_error = Column(String(500), nullable=True)
message_rate = Column(Float, nullable=True)
lag_seconds = Column(Float, nullable=True)
updated_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now(), index=True)
def to_dict(self) -> dict:
return {
"source": self.source,
"connection_state": self.connection_state,
"last_seen_at": to_iso8601_utc(self.last_seen_at),
"last_success_at": to_iso8601_utc(self.last_success_at),
"last_error": self.last_error,
"message_rate": self.message_rate,
"lag_seconds": self.lag_seconds,
"updated_at": to_iso8601_utc(self.updated_at),
}

View File

@@ -36,6 +36,7 @@ from app.services.collectors.iptoasn import IPtoASNPrefixGeoCollector
from app.services.collectors.opengeofeed import OpenGeoFeedPrefixGeoCollector
from app.services.collectors.nro_delegated import NRODelegatedPrefixGeoCollector
from app.services.collectors.news_live_streams import NewsLiveStreamsCollector
from app.services.collectors.aisstream import AISStreamCollector
from app.services.collectors.vessel_ais import VesselAISCollector
collector_registry.register(TOP500Collector())
@@ -65,3 +66,40 @@ collector_registry.register(OpenGeoFeedPrefixGeoCollector())
collector_registry.register(NRODelegatedPrefixGeoCollector())
collector_registry.register(NewsLiveStreamsCollector())
collector_registry.register(VesselAISCollector())
collector_registry.register(AISStreamCollector())
__all__ = [
"BaseCollector",
"HTTPCollector",
"IntervalCollector",
"collector_registry",
"CollectorRegistry",
"TOP500Collector",
"EpochAIGPUCollector",
"HuggingFaceModelCollector",
"HuggingFaceDatasetCollector",
"HuggingFaceSpacesCollector",
"PeeringDBIXPCollector",
"PeeringDBNetworkCollector",
"PeeringDBFacilityCollector",
"TeleGeographyCableCollector",
"TeleGeographyLandingPointCollector",
"TeleGeographyCableSystemCollector",
"CloudflareRadarDeviceCollector",
"CloudflareRadarTrafficCollector",
"CloudflareRadarTopASCollector",
"ArcGISCableCollector",
"FAOLandingPointCollector",
"ArcGISLandingPointCollector",
"ArcGISCableLandingRelationCollector",
"SpaceTrackTLECollector",
"CelesTrakTLECollector",
"RISLiveCollector",
"BGPStreamBackfillCollector",
"IPtoASNPrefixGeoCollector",
"OpenGeoFeedPrefixGeoCollector",
"NRODelegatedPrefixGeoCollector",
"NewsLiveStreamsCollector",
"VesselAISCollector",
"AISStreamCollector",
]

View File

@@ -0,0 +1,276 @@
"""AISStream WebSocket collector for realtime vessel AIS observations."""
from datetime import UTC, datetime
import asyncio
import json
import os
from typing import Any
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.data_sources import get_data_sources_config
from app.models.datasource_config import DataSourceConfig
from app.services.collectors.base import BaseCollector
from app.services.vessel_ais_aggregation import (
AISSTREAM_DELIVERY_MODE,
AISSTREAM_TRANSPORT,
record_vessel_ais_observation,
update_ais_source_health,
)
from app.services.vessel_types import normalize_vessel_type_name
DEFAULT_AISSTREAM_URL = "wss://stream.aisstream.io/v0/stream"
DEFAULT_BOUNDING_BOXES = [[[-90, -180], [90, 180]]]
DEFAULT_MESSAGE_TYPES = ["PositionReport", "ShipStaticData"]
class AISStreamCollector(BaseCollector):
"""Collect AISStream WebSocket messages into the raw AIS observation layer."""
name = "aisstream_vessels"
priority = "P1"
module = "L4"
frequency_hours = 1
data_type = "vessel_ais"
fail_on_empty = False
async def _load_datasource_config(self) -> DataSourceConfig | None:
if self._db_session is None:
return None
result = await self._db_session.execute(
select(DataSourceConfig)
.where(DataSourceConfig.name == self.name)
.where(DataSourceConfig.is_active.is_(True))
)
return result.scalar_one_or_none()
async def _get_effective_config(self) -> dict[str, Any]:
datasource_config = await self._load_datasource_config()
config = dict(datasource_config.config or {}) if datasource_config else {}
auth_config = dict(datasource_config.auth_config or {}) if datasource_config else {}
endpoint = (
(datasource_config.endpoint if datasource_config else None)
or self._resolved_url
or get_data_sources_config().get_yaml_url(self.name)
or DEFAULT_AISSTREAM_URL
)
api_key = (
auth_config.get("api_key")
or config.get("api_key")
or os.getenv("AISSTREAM_API_KEY")
)
return {
"endpoint": endpoint,
"api_key": api_key,
"bounding_boxes": config.get("bounding_boxes") or DEFAULT_BOUNDING_BOXES,
"message_types": config.get("message_types") or DEFAULT_MESSAGE_TYPES,
"max_messages": int(config.get("max_messages") or 500),
"receive_timeout_seconds": float(config.get("receive_timeout_seconds") or 30),
}
async def fetch(self) -> list[dict[str, Any]]:
config = await self._get_effective_config()
if not config["api_key"]:
raise RuntimeError("AISStream API key is not configured")
try:
import websockets
except ImportError as exc:
raise RuntimeError("Python package 'websockets' is required for AISStream") from exc
subscription = {
"APIKey": config["api_key"],
"BoundingBoxes": config["bounding_boxes"],
"FilterMessageTypes": config["message_types"],
}
messages: list[dict[str, Any]] = []
try:
async with websockets.connect(config["endpoint"]) as websocket:
await websocket.send(json.dumps(subscription))
while len(messages) < config["max_messages"]:
try:
raw_message = await asyncio.wait_for(
websocket.recv(),
timeout=config["receive_timeout_seconds"],
)
except TimeoutError:
break
payload = json.loads(raw_message)
if isinstance(payload, dict):
messages.append(payload)
except Exception as exc:
if self._db_session is not None:
await update_ais_source_health(
self._db_session,
source=self.name,
connection_state="disconnected",
last_error=f"{exc.__class__.__name__}: {exc}",
)
await self._db_session.commit()
raise
return messages
def transform(self, raw_data: list[dict[str, Any]]) -> list[dict[str, Any]]:
records = []
for item in raw_data:
record = self._normalize_message(item)
if record:
records.append(record)
return records
async def _save_data(
self,
db: AsyncSession,
data: list[dict[str, Any]],
task_id: int | None = None,
snapshot_id: int | None = None,
) -> int:
now = datetime.now(UTC)
records_added = 0
latest_observed_at = now
for index, item in enumerate(data):
observed_at = item.get("received_at") or now
observation = await record_vessel_ais_observation(
db,
source=self.name,
normalized_payload=item,
raw_payload=item.get("_raw_payload") or item,
delivery_mode=AISSTREAM_DELIVERY_MODE,
transport=AISSTREAM_TRANSPORT,
message_type=item.get("_message_type") or "PositionReport",
source_message_id=item.get("_source_message_id"),
observed_at=observed_at,
collected_at=now,
)
if observation is not None:
records_added += 1
if isinstance(observed_at, datetime) and observed_at > latest_observed_at:
latest_observed_at = observed_at
if (index + 1) % 1000 == 0:
await self.update_progress(index + 1, commit=True)
await update_ais_source_health(
db,
source=self.name,
connection_state="connected",
observed_count=len(data),
last_seen_at=latest_observed_at,
last_success_at=now if data else None,
lag_seconds=max((now - latest_observed_at).total_seconds(), 0),
)
await db.commit()
await self.update_progress(records_added, force=True)
return records_added
def _normalize_message(self, item: dict[str, Any]) -> dict[str, Any] | None:
message_type = str(item.get("MessageType") or item.get("message_type") or "")
metadata = item.get("MetaData") if isinstance(item.get("MetaData"), dict) else {}
message = item.get("Message") if isinstance(item.get("Message"), dict) else {}
body = message.get(message_type) if isinstance(message.get(message_type), dict) else message
if not isinstance(body, dict):
body = {}
mmsi = _as_int(_pick(metadata, "MMSI", "mmsi") or _pick(body, "MMSI", "mmsi"))
if mmsi is None:
return None
received_at = _parse_datetime(
_pick(metadata, "time_utc", "Time_UTC", "timestamp")
or _pick(body, "Timestamp", "timestamp", "time")
)
ship_name = _clean_text(
_pick(body, "Name", "ShipName", "name")
or _pick(metadata, "ShipName", "ship_name", "name")
)
record: dict[str, Any] = {
"mmsi": mmsi,
"received_at": received_at,
"_message_type": message_type or None,
"_source_message_id": item.get("MessageID") or item.get("message_id"),
"_raw_payload": item,
}
lat = _as_float(_pick(body, "Latitude", "lat", "latitude"))
lon = _as_float(_pick(body, "Longitude", "lon", "lng", "longitude"))
if lat is not None and lon is not None:
if not (-90 <= lat <= 90 and -180 <= lon <= 180):
return None
record.update(
{
"lat": lat,
"lon": lon,
"sog": _as_float(_pick(body, "Sog", "SOG", "speedOverGround")),
"cog": _as_float(_pick(body, "Cog", "COG", "courseOverGround")),
"heading": _as_int(_pick(body, "TrueHeading", "Heading", "heading")),
"nav_status": _as_int(_pick(body, "NavigationalStatus", "nav_status")),
}
)
vessel_type = _as_int(_pick(body, "Type", "ShipType", "vessel_type"))
record.update(
{
"name": ship_name,
"callsign": _pick(body, "CallSign", "callsign"),
"imo": _as_int(_pick(body, "ImoNumber", "IMO", "imo")),
"vessel_type": vessel_type,
"vessel_type_name": _pick(body, "TypeName", "ShipTypeName", "vessel_type_name")
or normalize_vessel_type_name(vessel_type),
"length": _as_float(_pick(body, "DimensionToBow", "Length", "length")),
"width": _as_float(_pick(body, "DimensionToPort", "Width", "width")),
}
)
return record
def _pick(item: dict[str, Any], *keys: str) -> Any:
for key in keys:
if key in item and item[key] not in (None, ""):
return item[key]
return None
def _clean_text(value: Any) -> str | None:
if value in (None, ""):
return None
text = str(value).strip()
return text or None
def _as_float(value: Any) -> float | None:
try:
if value in (None, ""):
return None
return float(value)
except (TypeError, ValueError):
return None
def _as_int(value: Any) -> int | None:
try:
if value in (None, ""):
return None
return int(float(value))
except (TypeError, ValueError):
return None
def _parse_datetime(value: Any) -> datetime | None:
if isinstance(value, datetime):
return value if value.tzinfo else value.replace(tzinfo=UTC)
if not value:
return None
if isinstance(value, (int, float)):
timestamp = float(value)
if timestamp > 10_000_000_000:
timestamp /= 1000
return datetime.fromtimestamp(timestamp, UTC)
if isinstance(value, str):
try:
parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
return parsed if parsed.tzinfo else parsed.replace(tzinfo=UTC)
except ValueError:
return None
return None

View File

@@ -4,7 +4,7 @@ from datetime import UTC, datetime, timedelta
from typing import Any
import httpx
from sqlalchemy import delete, select
from sqlalchemy import delete
from sqlalchemy.ext.asyncio import AsyncSession
from app.models.vessel import VesselPosition, VesselStatic
@@ -14,15 +14,13 @@ from app.services.barentswatch import (
resolve_barentswatch_config,
)
from app.services.collectors.base import BaseCollector
VESSEL_TYPE_NAMES = {
30: "Fishing",
35: "Military",
60: "Passenger",
70: "Cargo",
80: "Tanker",
}
from app.services.vessel_ais_aggregation import (
BARENTSWATCH_DELIVERY_MODE,
BARENTSWATCH_TRANSPORT,
record_vessel_ais_observation,
update_ais_source_health,
)
from app.services.vessel_types import normalize_vessel_type_name
class VesselAISCollector(BaseCollector):
@@ -92,6 +90,18 @@ class VesselAISCollector(BaseCollector):
records_added = 0
for index, item in enumerate(data):
observed_at = item.get("received_at") or now
await record_vessel_ais_observation(
db,
source=self.name,
normalized_payload=item,
raw_payload=item,
delivery_mode=BARENTSWATCH_DELIVERY_MODE,
transport=BARENTSWATCH_TRANSPORT,
observed_at=observed_at,
collected_at=now,
)
static = await db.get(VesselStatic, item["mmsi"])
if static is None:
static = VesselStatic(mmsi=item["mmsi"])
@@ -122,7 +132,7 @@ class VesselAISCollector(BaseCollector):
cog=item.get("cog"),
heading=item.get("heading"),
nav_status=item.get("nav_status"),
received_at=item.get("received_at") or now,
received_at=observed_at,
)
)
records_added += 1
@@ -130,6 +140,19 @@ class VesselAISCollector(BaseCollector):
if (index + 1) % 1000 == 0:
await self.update_progress(index + 1, commit=True)
latest_observed_at = max(
(item.get("received_at") for item in data if item.get("received_at")),
default=now,
)
await update_ais_source_health(
db,
source=self.name,
connection_state="connected",
observed_count=len(data),
last_seen_at=latest_observed_at,
last_success_at=now if data else None,
lag_seconds=max((now - latest_observed_at).total_seconds(), 0),
)
await db.execute(
delete(VesselPosition).where(VesselPosition.received_at < now - timedelta(hours=24))
)
@@ -156,7 +179,7 @@ class VesselAISCollector(BaseCollector):
vessel_type = _as_int(_pick(item, "vessel_type", "shipType", "ship_type", "ShipType"))
vessel_type_name = (
_pick(item, "vessel_type_name", "shipTypeName", "ship_type_name", "VesselTypeName")
or _vessel_type_name(vessel_type)
or normalize_vessel_type_name(vessel_type)
)
received_at = _parse_datetime(_pick(item, "received_at", "timestamp", "time", "msgtime"))
@@ -255,19 +278,3 @@ def _parse_datetime(value: Any) -> datetime | None:
except ValueError:
return None
return None
def _vessel_type_name(vessel_type: int | None) -> str:
if vessel_type is None:
return "Other"
if 70 <= vessel_type <= 79:
return "Cargo"
if 80 <= vessel_type <= 89:
return "Tanker"
if 60 <= vessel_type <= 69:
return "Passenger"
if vessel_type == 30:
return "Fishing"
if vessel_type == 35:
return "Military"
return VESSEL_TYPE_NAMES.get(vessel_type, "Other")

View File

@@ -66,9 +66,68 @@ BARENTSWATCH_DEFAULT_GUIDE = CredentialGuideDefault(
""",
)
AISSTREAM_DEFAULT_GUIDE = CredentialGuideDefault(
provider="aisstream",
title="AISStream API Key 获取教程",
prompt=(
"请生成一份中文教程,指导开发者获取 AISStream 的 API Key 并配置到 Planet。"
"教程要面向已经有本地开发环境的人,包含注册/登录 AISStream、获取 API Key、"
"理解免费额度和订阅范围、在 Planet 设置中心填写 API Key、配置 bounding boxes "
"和 message types、验证连接、常见失败排查。必须提醒用户以 AISStream 当前官网和"
"服务条款为准,不要编造具体页面按钮文案。"
),
markdown="""## AISStream API Key 获取
官方入口https://aisstream.io/
1. 打开 AISStream 官网,按当前页面指引注册或登录账号。
2. 在账号/API 管理页面创建或复制你的 API Key。
3. 先确认当前账号额度、使用条款和可订阅区域。实时 AIS 流量可能很大,不建议一开始订阅全球范围。
4. 回到 Planet 的 `设置 -> 采集器设置 -> AISStream 实时船舶`。
5. 在 `AISStream 凭证` 中填入 API Key。
6. Endpoint 通常保持默认:`wss://stream.aisstream.io/v0/stream`。
7. 按需配置 `Bounding Boxes JSON` 和 `消息类型`。
8. 点击连接测试,确认系统能读取凭证且 WebSocket endpoint 格式有效。
9. 保存采集器设置后再运行 `aisstream_vessels` collector。
### 推荐配置
默认消息类型:
```json
["PositionReport", "ShipStaticData"]
```
默认 Bounding Boxes 示例:
```json
[[[-90, -180], [90, 180]]]
```
这个示例表示全球范围。实际使用时建议先改成较小区域,降低消息量和处理压力。
### 请求规则
- Endpoint`wss://stream.aisstream.io/v0/stream`
- 传输方式WebSocket
- API Key 放在订阅 payload 中,不放在 HTTP header。
- Planet 会把 AISStream 标记为 `delivery_mode = realtime_stream`、`transport = websocket`。
- AISStream collector 只写入 AIS raw observations不直接覆盖最终船只展示表。
### 常见排查
- `未找到凭证`:确认 API Key 已保存到采集器设置,或设置了 `AISSTREAM_API_KEY` 环境变量 / `~/.zshrc`。
- `endpoint 必须是 ws:// 或 wss://`AISStream 是 WebSocket 流接口,不要填普通 `https://` API 地址。
- 采集量过大:缩小 `Bounding Boxes JSON`,减少 `message_types`,或降低单次最大消息数。
- 没有船只数据:确认订阅区域内确实有 AIS 活动,并检查 API Key 当前额度和权限。
- 连接中断实时流可能受网络和上游限流影响collector 会记录源健康状态供聚合服务回退。
""",
)
DEFAULT_CREDENTIAL_GUIDES = {
BARENTSWATCH_DEFAULT_GUIDE.provider: BARENTSWATCH_DEFAULT_GUIDE,
AISSTREAM_DEFAULT_GUIDE.provider: AISSTREAM_DEFAULT_GUIDE,
}

View File

@@ -26,6 +26,7 @@ from app.services.barentswatch import (
CONNECTIVITY_VALIDATION_KEY = "connectivity_validation"
CONNECTIVITY_STORE_CATEGORY = "datasource_connectivity_validations"
SUPPORTED_CREDENTIAL_PROVIDERS = {"barentswatch", "spacetrack", "aisstream"}
def _sha256_json(payload: Any) -> str:
@@ -43,6 +44,36 @@ def _resolve_spacetrack_credentials() -> tuple[str, str, str]:
return username, password, source or "missing"
async def _resolve_aisstream_api_key(
db=None,
credential_override: dict[str, str] | None = None,
) -> tuple[str, str]:
if credential_override and credential_override.get("api_key"):
return str(credential_override["api_key"]), "draft"
env_key = os.getenv("AISSTREAM_API_KEY")
zshrc_key = _read_zshrc_env().get("AISSTREAM_API_KEY")
if db is not None:
result = await db.execute(
select(DataSourceConfig)
.where(DataSourceConfig.name == "aisstream_vessels")
.where(DataSourceConfig.is_active.is_(True))
)
record = result.scalar_one_or_none()
if record:
auth_config = record.auth_config or {}
runtime_config = record.config or {}
api_key = auth_config.get("api_key") or runtime_config.get("api_key")
if api_key:
return str(api_key), "datasource_config"
if env_key:
return env_key, "environment"
if zshrc_key:
return zshrc_key, "~/.zshrc"
return "", "missing"
def strip_connectivity_validation(config: dict | None) -> dict:
cleaned = dict(config or {})
cleaned.pop(CONNECTIVITY_VALIDATION_KEY, None)
@@ -103,6 +134,10 @@ async def build_builtin_connectivity_checksum(
"password": password,
}
)
elif credential_provider == "aisstream":
api_key, credential_source = await _resolve_aisstream_api_key(db, credential_override)
has_credentials = bool(api_key)
credential_fingerprint = _sha256_json({"api_key": api_key})
elif defaults.get("requires_credentials"):
credential_source = str(credential_provider or "unsupported")
@@ -130,6 +165,7 @@ async def test_builtin_connectivity(
headers: dict | None,
config: dict | None,
db=None,
credential_override: dict[str, str] | None = None,
) -> dict[str, Any]:
defaults = DEFAULT_DATASOURCES.get(source)
if not defaults:
@@ -145,6 +181,7 @@ async def test_builtin_connectivity(
headers,
config,
db,
credential_override,
)
if credential_context["requires_credentials"] and not credential_context["has_credentials"]:
return {
@@ -155,10 +192,9 @@ async def test_builtin_connectivity(
"settings_tab": "collector_credentials",
**credential_context,
}
supported_credential_providers = {"barentswatch", "spacetrack"}
if (
credential_context["requires_credentials"]
and credential_context["credential_provider"] not in supported_credential_providers
and credential_context["credential_provider"] not in SUPPORTED_CREDENTIAL_PROVIDERS
):
return {
"success": False,
@@ -174,6 +210,23 @@ async def test_builtin_connectivity(
timeout = float(request_config.get("timeout") or 30)
request_endpoint = endpoint
if credential_context["credential_provider"] == "aisstream":
if not str(request_endpoint).startswith(("ws://", "wss://")):
return {
"success": False,
"checksum": checksum,
"stage": "endpoint",
"message": "AISStream endpoint 必须是 ws:// 或 wss:// WebSocket 地址。",
**credential_context,
}
return {
"success": True,
"checksum": checksum,
"stage": "credentials",
"message": "AISStream 凭证已配置WebSocket endpoint 格式有效。",
**credential_context,
}
try:
async with httpx.AsyncClient(timeout=timeout, follow_redirects=True) as client:
if credential_context["credential_provider"] == "barentswatch":

View File

@@ -0,0 +1,574 @@
"""AIS raw observation and aggregation support for vessel collectors."""
from datetime import UTC, datetime
from hashlib import sha256
import json
from typing import Any, Iterable
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.models.vessel import AISConflictRecord, AISRawObservation, AISSourceHealth
from app.services.vessel_types import normalize_vessel_type_name
VESSEL_AIS_SCHEMA = "vessel_ais"
BARENTSWATCH_DELIVERY_MODE = "polling"
BARENTSWATCH_TRANSPORT = "http"
AISSTREAM_DELIVERY_MODE = "realtime_stream"
AISSTREAM_TRANSPORT = "websocket"
DELIVERY_MODE_PRIORITY = {
"realtime_stream": 40,
"batch_stream": 30,
"polling": 20,
"snapshot": 10,
}
DYNAMIC_FIELDS = ("lat", "lon", "sog", "cog", "heading", "nav_status")
CONFLICT_FIELDS = (
"name",
"callsign",
"imo",
"flag",
"vessel_type",
"vessel_type_name",
"length",
"width",
"draught",
)
def _json_default(value: Any) -> Any:
if isinstance(value, datetime):
return value.astimezone(UTC).isoformat()
return str(value)
def _stable_payload(value: Any) -> str:
return json.dumps(value, sort_keys=True, separators=(",", ":"), default=_json_default)
def _jsonable(value: Any) -> Any:
if isinstance(value, datetime):
return value.astimezone(UTC).isoformat()
if isinstance(value, dict):
return {str(key): _jsonable(item) for key, item in value.items()}
if isinstance(value, list):
return [_jsonable(item) for item in value]
return value
def _coerce_datetime(value: Any) -> datetime | None:
if isinstance(value, datetime):
return value if value.tzinfo else value.replace(tzinfo=UTC)
if isinstance(value, (int, float)):
timestamp = float(value)
if timestamp > 10_000_000_000:
timestamp /= 1000
return datetime.fromtimestamp(timestamp, UTC)
if isinstance(value, str) and value:
try:
parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
return parsed if parsed.tzinfo else parsed.replace(tzinfo=UTC)
except ValueError:
return None
return None
def build_observation_hash(
*,
source: str,
entity_key: str,
message_type: str | None,
observed_at: datetime,
normalized_payload: dict[str, Any],
source_message_id: str | None = None,
) -> str:
"""Build a deterministic idempotency key for one source-level AIS observation."""
if source_message_id:
basis = {
"source": source,
"entity_key": entity_key,
"source_message_id": source_message_id,
}
else:
basis = {
"source": source,
"entity_key": entity_key,
"message_type": message_type,
"observed_at": observed_at.astimezone(UTC).isoformat(),
"payload": normalized_payload,
}
return sha256(_stable_payload(basis).encode("utf-8")).hexdigest()
def build_field_conflict_candidates(
observations: Iterable[AISRawObservation],
fields: Iterable[str] = CONFLICT_FIELDS,
) -> list[dict[str, Any]]:
"""Return current field disagreements from raw observations without mutating state."""
candidates_by_field: dict[str, dict[str, Any]] = {}
for observation in observations:
payload = observation.normalized_payload or {}
for field in fields:
value = payload.get(field)
if value in (None, ""):
continue
field_candidates = candidates_by_field.setdefault(field, {})
field_candidates[observation.source] = value
conflicts = []
for field, candidates in sorted(candidates_by_field.items()):
unique_values = {_stable_payload(value) for value in candidates.values()}
if len(unique_values) <= 1:
continue
conflicts.append(
{
"field": field,
"candidates": candidates,
"status": "candidate",
}
)
return conflicts
def _payload_value(payload: dict[str, Any], field: str) -> Any:
value = payload.get(field)
return None if value in (None, "") else value
def _clean_text(value: Any) -> str | None:
if value in (None, ""):
return None
text = str(value).strip()
return text or None
def _raw_metadata_value(observation: AISRawObservation, field: str) -> Any:
raw_payload = observation.raw_payload or {}
metadata = raw_payload.get("MetaData") if isinstance(raw_payload, dict) else None
if not isinstance(metadata, dict):
return None
if field == "name":
return _clean_text(metadata.get("ShipName") or metadata.get("ship_name") or metadata.get("name"))
return None
def _delivery_priority(observation: AISRawObservation) -> int:
return DELIVERY_MODE_PRIORITY.get(str(observation.delivery_mode or ""), 0)
def _has_valid_position(payload: dict[str, Any]) -> bool:
try:
lat = float(payload.get("lat"))
lon = float(payload.get("lon"))
except (TypeError, ValueError):
return False
return -90 <= lat <= 90 and -180 <= lon <= 180
def _is_future_observation(observation: AISRawObservation, now: datetime) -> bool:
return observation.observed_at > now
def _select_position_observation(
observations: list[AISRawObservation],
*,
now: datetime,
) -> tuple[AISRawObservation | None, list[str]]:
rejected_flags: list[str] = []
candidates = []
for observation in observations:
payload = observation.normalized_payload or {}
if not _has_valid_position(payload):
rejected_flags.append("invalid_position")
continue
if _is_future_observation(observation, now):
rejected_flags.append("future_timestamp")
continue
candidates.append(observation)
if not candidates:
return None, sorted(set(rejected_flags))
candidates.sort(
key=lambda item: (
item.observed_at,
_delivery_priority(item),
item.collected_at,
item.id or 0,
),
reverse=True,
)
return candidates[0], sorted(set(rejected_flags))
def _select_static_field(
observations: list[AISRawObservation],
field: str,
) -> tuple[Any, str | None, str | None]:
candidates = []
for observation in observations:
value = _payload_value(observation.normalized_payload or {}, field)
if value is None:
value = _raw_metadata_value(observation, field)
if value is None:
continue
candidates.append((observation, value))
if not candidates:
return None, None, None
candidates.sort(
key=lambda item: (
_delivery_priority(item[0]),
item[0].observed_at,
item[0].collected_at,
item[0].id or 0,
),
reverse=True,
)
selected_observation, selected_value = candidates[0]
unique_values = {_stable_payload(value) for _, value in candidates}
reason = "delivery_mode_priority" if len(unique_values) > 1 else "non_empty_priority"
return selected_value, selected_observation.source, reason
def _build_source_summary(observations: list[AISRawObservation]) -> dict[str, dict[str, Any]]:
summary: dict[str, dict[str, Any]] = {}
for observation in observations:
source_summary = summary.setdefault(
observation.source,
{
"observation_count": 0,
"latest_observed_at": None,
"delivery_mode": observation.delivery_mode,
"transport": observation.transport,
"message_types": [],
},
)
source_summary["observation_count"] += 1
latest_observed_at = source_summary["latest_observed_at"]
if latest_observed_at is None or observation.observed_at > latest_observed_at:
source_summary["latest_observed_at"] = observation.observed_at
if observation.message_type and observation.message_type not in source_summary["message_types"]:
source_summary["message_types"].append(observation.message_type)
return summary
def _build_aggregated_vessel(
entity_key: str,
observations: list[AISRawObservation],
*,
now: datetime,
) -> dict[str, Any] | None:
position_observation, rejected_flags = _select_position_observation(observations, now=now)
if position_observation is None:
return None
payload = position_observation.normalized_payload or {}
mmsi = int(entity_key)
result: dict[str, Any] = {
"mmsi": mmsi,
"lat": float(payload["lat"]),
"lon": float(payload["lon"]),
"received_at": position_observation.observed_at,
"field_sources": {},
"selected_reasons": {},
"source_summary": _build_source_summary(observations),
"quality_flags": sorted(
set((position_observation.quality_flags or []) + rejected_flags)
),
}
for field in DYNAMIC_FIELDS:
value = _payload_value(payload, field)
if field in ("lat", "lon") or value is not None:
result[field] = value
result["field_sources"][field] = position_observation.source
result["selected_reasons"][field] = "newest_observation"
for field in CONFLICT_FIELDS:
selected_value, selected_source, reason = _select_static_field(observations, field)
if selected_value is None:
continue
result[field] = selected_value
result["field_sources"][field] = selected_source
result["selected_reasons"][field] = reason
result["name"] = result.get("name") or f"MMSI {mmsi}"
result["vessel_type_name"] = result.get("vessel_type_name") or normalize_vessel_type_name(
result.get("vessel_type")
)
return result
async def _upsert_conflict_records(
db: AsyncSession,
entity_key: str,
observations: list[AISRawObservation],
aggregated: dict[str, Any],
) -> int:
conflicts = build_field_conflict_candidates(observations)
now = datetime.now(UTC)
for conflict in conflicts:
field = conflict["field"]
result = await db.execute(
select(AISConflictRecord)
.where(AISConflictRecord.target_schema == VESSEL_AIS_SCHEMA)
.where(AISConflictRecord.entity_key == entity_key)
.where(AISConflictRecord.field == field)
.limit(1)
)
record = result.scalar_one_or_none()
if record is None:
record = AISConflictRecord(
target_schema=VESSEL_AIS_SCHEMA,
entity_key=entity_key,
field=field,
)
db.add(record)
record.candidates = conflict["candidates"]
record.selected_source = (aggregated.get("field_sources") or {}).get(field)
record.selected_value = aggregated.get(field)
record.selected_reason = (aggregated.get("selected_reasons") or {}).get(field)
record.resolved_by = "system"
record.status = "open"
record.updated_at = now
return len(conflicts)
def _group_observations(observations: Iterable[AISRawObservation]) -> dict[str, list[AISRawObservation]]:
grouped: dict[str, list[AISRawObservation]] = {}
for observation in observations:
grouped.setdefault(str(observation.entity_key), []).append(observation)
return grouped
async def record_vessel_ais_observation(
db: AsyncSession,
*,
source: str,
normalized_payload: dict[str, Any],
raw_payload: dict[str, Any] | None = None,
delivery_mode: str,
transport: str,
message_type: str | None = "PositionReport",
source_message_id: str | None = None,
observed_at: datetime | None = None,
collected_at: datetime | None = None,
quality_flags: list[str] | None = None,
) -> AISRawObservation | None:
"""Insert one raw observation if the source-level fact has not already been stored."""
entity_key = str(normalized_payload["mmsi"])
collected_at = collected_at or datetime.now(UTC)
observed_at = (
_coerce_datetime(observed_at)
or _coerce_datetime(normalized_payload.get("received_at"))
or collected_at
)
normalized_json = _jsonable(normalized_payload)
raw_json = _jsonable(raw_payload or {})
observation_hash = build_observation_hash(
source=source,
entity_key=entity_key,
message_type=message_type,
observed_at=observed_at,
normalized_payload=normalized_json,
source_message_id=source_message_id,
)
existing_result = await db.execute(
select(AISRawObservation.id).where(AISRawObservation.observation_hash == observation_hash)
)
if existing_result.scalar_one_or_none() is not None:
return None
observation = AISRawObservation(
target_schema=VESSEL_AIS_SCHEMA,
source=source,
entity_key=entity_key,
delivery_mode=delivery_mode,
transport=transport,
message_type=message_type,
source_message_id=source_message_id,
observation_hash=observation_hash,
observed_at=observed_at,
collected_at=collected_at,
normalized_payload=normalized_json,
raw_payload=raw_json,
quality_flags=quality_flags or [],
)
db.add(observation)
return observation
async def aggregate_vessel_observations(
db: AsyncSession,
observations: Iterable[AISRawObservation],
*,
write_conflicts: bool = True,
) -> list[dict[str, Any]]:
now = datetime.now(UTC)
vessels = []
for entity_key, entity_observations in _group_observations(observations).items():
aggregated = _build_aggregated_vessel(entity_key, entity_observations, now=now)
if aggregated is None:
continue
if write_conflicts:
aggregated["conflict_count"] = await _upsert_conflict_records(
db,
entity_key,
entity_observations,
aggregated,
)
else:
aggregated["conflict_count"] = len(build_field_conflict_candidates(entity_observations))
vessels.append(aggregated)
vessels.sort(key=lambda item: item.get("received_at") or datetime.min.replace(tzinfo=UTC), reverse=True)
return vessels
async def get_aggregated_vessels(
db: AsyncSession,
*,
bbox: tuple[float, float, float, float] | None = None,
limit: int | None = None,
) -> list[dict[str, Any]]:
stmt = (
select(AISRawObservation)
.where(AISRawObservation.target_schema == VESSEL_AIS_SCHEMA)
.order_by(AISRawObservation.observed_at.desc(), AISRawObservation.id.desc())
)
if limit and limit > 0:
stmt = stmt.limit(max(limit * 20, limit))
result = await db.execute(stmt)
if not hasattr(result, "scalars"):
return []
vessels = await aggregate_vessel_observations(db, result.scalars().all())
if bbox is not None:
lon_min, lat_min, lon_max, lat_max = bbox
vessels = [
vessel
for vessel in vessels
if lon_min <= float(vessel["lon"]) <= lon_max
and lat_min <= float(vessel["lat"]) <= lat_max
]
if limit and limit > 0:
return vessels[:limit]
return vessels
async def get_aggregated_vessel(db: AsyncSession, mmsi: int) -> dict[str, Any] | None:
observations = await get_vessel_raw_observations(db, mmsi, limit=1000)
vessels = await aggregate_vessel_observations(db, observations)
return vessels[0] if vessels else None
async def get_aggregated_vessel_track(
db: AsyncSession,
mmsi: int,
*,
cutoff: datetime,
) -> list[dict[str, Any]]:
result = await db.execute(
select(AISRawObservation)
.where(AISRawObservation.target_schema == VESSEL_AIS_SCHEMA)
.where(AISRawObservation.entity_key == str(mmsi))
.where(AISRawObservation.observed_at >= cutoff)
.order_by(AISRawObservation.observed_at.asc(), AISRawObservation.id.asc())
)
if not hasattr(result, "scalars"):
return []
points: list[dict[str, Any]] = []
seen: set[tuple[str, float, float, str]] = set()
for observation in result.scalars().all():
payload = observation.normalized_payload or {}
if not _has_valid_position(payload):
continue
lat = float(payload["lat"])
lon = float(payload["lon"])
key = (
observation.observed_at.isoformat(),
round(lat, 5),
round(lon, 5),
observation.source,
)
if key in seen:
continue
seen.add(key)
points.append(
{
"lat": lat,
"lon": lon,
"observed_at": observation.observed_at,
"source": observation.source,
"selected_reason": "track_timeline",
"quality_flags": observation.quality_flags or [],
}
)
return points
async def update_ais_source_health(
db: AsyncSession,
*,
source: str,
connection_state: str,
observed_count: int = 0,
last_seen_at: datetime | None = None,
last_success_at: datetime | None = None,
last_error: str | None = None,
lag_seconds: float | None = None,
) -> AISSourceHealth:
"""Upsert the health row for an AIS source."""
now = datetime.now(UTC)
health = await db.get(AISSourceHealth, source)
if health is None:
health = AISSourceHealth(source=source)
db.add(health)
health.connection_state = connection_state
health.last_seen_at = last_seen_at or health.last_seen_at
health.last_success_at = last_success_at or health.last_success_at
health.last_error = last_error
health.message_rate = float(observed_count)
health.lag_seconds = lag_seconds
health.updated_at = now
return health
async def get_vessel_raw_observations(
db: AsyncSession,
mmsi: int,
*,
limit: int = 100,
) -> list[AISRawObservation]:
result = await db.execute(
select(AISRawObservation)
.where(AISRawObservation.target_schema == VESSEL_AIS_SCHEMA)
.where(AISRawObservation.entity_key == str(mmsi))
.order_by(AISRawObservation.observed_at.desc(), AISRawObservation.id.desc())
.limit(limit)
)
return list(result.scalars().all())
async def get_vessel_conflict_records(
db: AsyncSession,
mmsi: int,
) -> list[AISConflictRecord]:
result = await db.execute(
select(AISConflictRecord)
.where(AISConflictRecord.target_schema == VESSEL_AIS_SCHEMA)
.where(AISConflictRecord.entity_key == str(mmsi))
.order_by(AISConflictRecord.updated_at.desc(), AISConflictRecord.id.desc())
)
return list(result.scalars().all())

View File

@@ -0,0 +1,31 @@
"""Shared AIS vessel type helpers."""
from typing import Any
VESSEL_TYPE_NAMES = {
30: "Fishing",
35: "Military",
60: "Passenger",
70: "Cargo",
80: "Tanker",
}
def normalize_vessel_type_name(vessel_type: Any) -> str:
"""Map AIS numeric vessel type codes to display buckets."""
try:
type_code = int(float(vessel_type))
except (TypeError, ValueError):
return "Other"
if 70 <= type_code <= 79:
return "Cargo"
if 80 <= type_code <= 89:
return "Tanker"
if 60 <= type_code <= 69:
return "Passenger"
if type_code == 30:
return "Fishing"
if type_code == 35:
return "Military"
return VESSEL_TYPE_NAMES.get(type_code, "Other")

View File

@@ -1,11 +1,13 @@
"""Unit tests for data collectors"""
import pytest
from datetime import datetime
from unittest.mock import AsyncMock, MagicMock, patch
from unittest.mock import AsyncMock, patch
from app.core.datasource_defaults import DEFAULT_DATASOURCES
from app.services.credential_guides import DEFAULT_CREDENTIAL_GUIDES
from app.services.collectors.top500 import TOP500Collector
from app.services.collectors.base import BaseCollector, HTTPCollector
from app.services.collectors.registry import collector_registry
from app.services.datasource_connectivity import SUPPORTED_CREDENTIAL_PROVIDERS
from app.models.task import CollectionTask
@@ -145,3 +147,30 @@ class TestHTTPCollector:
assert hasattr(collector, "parse_response")
assert callable(collector.fetch)
assert callable(collector.parse_response)
def test_aisstream_collector_is_registered():
collector = collector_registry.get("aisstream_vessels")
assert collector is not None
assert collector.data_type == "vessel_ais"
def test_supported_credential_collectors_have_guides_and_connectivity_provider():
missing: list[str] = []
for source, info in DEFAULT_DATASOURCES.items():
if not info.get("requires_credentials"):
continue
if info.get("credential_status") != "supported":
continue
provider = info.get("credential_provider")
if not provider:
missing.append(f"{source}: missing credential_provider")
continue
if provider not in DEFAULT_CREDENTIAL_GUIDES:
missing.append(f"{source}: missing credential guide for {provider}")
if provider not in SUPPORTED_CREDENTIAL_PROVIDERS:
missing.append(f"{source}: missing connectivity provider for {provider}")
assert missing == []

View File

@@ -1,4 +1,5 @@
from datetime import datetime, timedelta, timezone
from unittest.mock import AsyncMock
import pytest
from httpx import ASGITransport, AsyncClient
@@ -6,9 +7,16 @@ from httpx import ASGITransport, AsyncClient
from app.api.v1.visualization import convert_vessels_to_geojson
from app.db.session import get_db
from app.main import app
from app.models.vessel import VesselPosition, VesselStatic
from app.models.vessel import AISRawObservation, VesselPosition, VesselStatic
from app.services import barentswatch
from app.services.collectors.aisstream import AISStreamCollector
from app.services.collectors.vessel_ais import VesselAISCollector
from app.services.vessel_ais_aggregation import (
aggregate_vessel_observations,
build_field_conflict_candidates,
build_observation_hash,
record_vessel_ais_observation,
)
def test_vessel_collector_transforms_barentswatch_like_records():
@@ -35,6 +43,328 @@ def test_vessel_collector_transforms_barentswatch_like_records():
assert records[0]["lat"] == pytest.approx(59.91)
def test_vessel_observation_hash_is_stable_for_same_payload():
observed_at = datetime(2026, 4, 30, 12, 0, tzinfo=timezone.utc)
payload = {
"mmsi": 257123000,
"lat": 59.91,
"lon": 10.73,
"received_at": observed_at,
}
first = build_observation_hash(
source="barentswatch_vessels",
entity_key="257123000",
message_type="PositionReport",
observed_at=observed_at,
normalized_payload=payload,
)
second = build_observation_hash(
source="barentswatch_vessels",
entity_key="257123000",
message_type="PositionReport",
observed_at=observed_at,
normalized_payload=dict(reversed(payload.items())),
)
assert first == second
assert len(first) == 64
@pytest.mark.asyncio
async def test_record_vessel_ais_observation_skips_existing_hash():
observed_at = datetime(2026, 4, 30, 12, 0, tzinfo=timezone.utc)
class _Result:
def scalar_one_or_none(self):
return 123
class _Session:
def __init__(self):
self.added = []
async def execute(self, _stmt):
return _Result()
def add(self, item):
self.added.append(item)
db = _Session()
observation = await record_vessel_ais_observation(
db,
source="barentswatch_vessels",
normalized_payload={
"mmsi": 257123000,
"lat": 59.91,
"lon": 10.73,
"received_at": observed_at,
},
delivery_mode="polling",
transport="http",
observed_at=observed_at.isoformat(),
)
assert observation is None
assert db.added == []
def test_build_field_conflict_candidates_from_raw_observations():
observations = [
AISRawObservation(
source="barentswatch_vessels",
normalized_payload={"name": "OSLO TRADER", "flag": "NO"},
),
AISRawObservation(
source="aisstream_vessels",
normalized_payload={"name": "OSLO TRADER II", "flag": "NO"},
),
]
conflicts = build_field_conflict_candidates(observations)
assert conflicts == [
{
"field": "name",
"candidates": {
"aisstream_vessels": "OSLO TRADER II",
"barentswatch_vessels": "OSLO TRADER",
},
"status": "candidate",
}
]
@pytest.mark.asyncio
async def test_aggregate_vessel_observations_prefers_realtime_and_records_conflict():
observed_at = datetime.now(timezone.utc) - timedelta(minutes=5)
class _Result:
def scalar_one_or_none(self):
return None
class _Session:
def __init__(self):
self.added = []
async def execute(self, _stmt):
return _Result()
def add(self, item):
self.added.append(item)
db = _Session()
observations = [
AISRawObservation(
id=1,
source="barentswatch_vessels",
entity_key="257123000",
delivery_mode="polling",
transport="http",
observed_at=observed_at,
collected_at=observed_at,
normalized_payload={
"mmsi": 257123000,
"name": "OSLO TRADER",
"lat": 59.91,
"lon": 10.73,
},
),
AISRawObservation(
id=2,
source="aisstream_vessels",
entity_key="257123000",
delivery_mode="realtime_stream",
transport="websocket",
observed_at=observed_at + timedelta(seconds=10),
collected_at=observed_at + timedelta(seconds=10),
normalized_payload={
"mmsi": 257123000,
"vessel_type": 79,
"lat": 59.92,
"lon": 10.74,
},
raw_payload={"MetaData": {"ShipName": "OSLO TRADER II "}},
),
]
vessels = await aggregate_vessel_observations(db, observations)
assert vessels[0]["lat"] == pytest.approx(59.92)
assert vessels[0]["field_sources"]["lat"] == "aisstream_vessels"
assert vessels[0]["name"] == "OSLO TRADER II"
assert vessels[0]["vessel_type_name"] == "Cargo"
assert vessels[0]["source_summary"]["aisstream_vessels"]["observation_count"] == 1
assert vessels[0]["source_summary"]["barentswatch_vessels"]["delivery_mode"] == "polling"
assert vessels[0]["conflict_count"] == 0
assert db.added == []
@pytest.mark.asyncio
async def test_vessel_collector_writes_raw_observations_without_changing_position_save(monkeypatch):
collector = VesselAISCollector()
collector.update_progress = AsyncMock()
record_observation = AsyncMock()
update_health = AsyncMock()
monkeypatch.setattr(
"app.services.collectors.vessel_ais.record_vessel_ais_observation",
record_observation,
)
monkeypatch.setattr(
"app.services.collectors.vessel_ais.update_ais_source_health",
update_health,
)
class _Session:
def __init__(self):
self.added = []
self.committed = False
async def get(self, *_args):
return None
def add(self, item):
self.added.append(item)
async def execute(self, _stmt):
return None
async def commit(self):
self.committed = True
db = _Session()
observed_at = datetime(2026, 4, 30, 12, 0, tzinfo=timezone.utc)
saved = await collector._save_data(
db,
[
{
"mmsi": 257123000,
"name": "OSLO TRADER",
"lat": 59.91,
"lon": 10.73,
"received_at": observed_at,
}
],
)
assert saved == 1
assert db.committed is True
assert any(isinstance(item, VesselStatic) for item in db.added)
assert any(isinstance(item, VesselPosition) for item in db.added)
record_observation.assert_awaited_once()
assert record_observation.await_args.kwargs["source"] == "barentswatch_vessels"
assert record_observation.await_args.kwargs["normalized_payload"]["mmsi"] == 257123000
update_health.assert_awaited_once()
def test_aisstream_collector_normalizes_position_report():
collector = AISStreamCollector()
records = collector.transform(
[
{
"MessageType": "PositionReport",
"MetaData": {
"MMSI": 257123000,
"ShipName": "OSLO TRADER ",
"time_utc": "2026-04-30T12:00:00Z",
},
"Message": {
"PositionReport": {
"Latitude": 59.91,
"Longitude": 10.73,
"Sog": 12.4,
"Cog": 214,
"TrueHeading": 215,
"NavigationalStatus": 0,
}
},
}
]
)
assert len(records) == 1
assert records[0]["mmsi"] == 257123000
assert records[0]["lat"] == pytest.approx(59.91)
assert records[0]["name"] == "OSLO TRADER"
assert records[0]["_message_type"] == "PositionReport"
def test_aisstream_collector_maps_ship_static_type_name():
collector = AISStreamCollector()
records = collector.transform(
[
{
"MessageType": "ShipStaticData",
"MetaData": {
"MMSI": 257123000,
"time_utc": "2026-04-30T12:00:00Z",
},
"Message": {
"ShipStaticData": {
"Name": "OSLO TRADER",
"Type": 79,
"CallSign": "LAAB",
}
},
}
]
)
assert len(records) == 1
assert records[0]["vessel_type"] == 79
assert records[0]["vessel_type_name"] == "Cargo"
@pytest.mark.asyncio
async def test_aisstream_collector_writes_only_raw_observations(monkeypatch):
collector = AISStreamCollector()
collector.update_progress = AsyncMock()
record_observation = AsyncMock(return_value=object())
update_health = AsyncMock()
monkeypatch.setattr(
"app.services.collectors.aisstream.record_vessel_ais_observation",
record_observation,
)
monkeypatch.setattr(
"app.services.collectors.aisstream.update_ais_source_health",
update_health,
)
class _Session:
def __init__(self):
self.added = []
self.committed = False
def add(self, item):
self.added.append(item)
async def commit(self):
self.committed = True
db = _Session()
saved = await collector._save_data(
db,
[
{
"mmsi": 257123000,
"lat": 59.91,
"lon": 10.73,
"received_at": datetime(2026, 4, 30, 12, 0, tzinfo=timezone.utc),
"_message_type": "PositionReport",
}
],
)
assert saved == 1
assert db.added == []
assert db.committed is True
record_observation.assert_awaited_once()
assert record_observation.await_args.kwargs["source"] == "aisstream_vessels"
update_health.assert_awaited_once()
def test_barentswatch_reads_credentials_from_zshrc(tmp_path):
zshrc = tmp_path / ".zshrc"
zshrc.write_text(
@@ -137,7 +467,7 @@ async def test_vessels_geojson_endpoint_filters_type_and_bbox():
async with AsyncClient(transport=transport, base_url="http://test") as client:
response = await client.get(
"/api/v1/visualization/geo/vessels",
params={"bbox": "0,50,20,70", "type": "cargo"},
params={"bbox": "0,50,20,70", "type": "cargo", "limit": 0},
)
assert response.status_code == 200

View File

@@ -8,6 +8,33 @@ This project follows the repository versioning rule:
- `improvement` -> `+0.0.1`bugfix + 小功能混合)
- `bugfix` -> `+0.0.1`
## [0.47.0] — 2026-04-30
Released: 2026-04-30
### ✨ Highlights
- 新增 AISStream WebSocket 船只采集器,并将 AIS 多源数据写入原始观测层,由聚合接口统一去重、合并和解释字段来源。
- 设置页新增 AISStream API Key、采集范围 preset、运行状态、连接验证和凭证教程入口让全球 AIS 采集链路可配置、可观察。
- Earth 船只图层默认不再限制 5000 艘,并统一 marker 颜色、详情卡、hover 和搜索结果的船型归一化显示。
### 🔧 Improvements
- 聚合接口新增 `field_sources``selected_reasons``source_summary``quality_flags` 和冲突记录调试接口,动态字段默认优先采用更新的实时流观测。
- AISStream 标准化支持 `MetaData.ShipName` 船名兜底,并将 AIS 数字船型映射为 Cargo / Tanker / Passenger / Fishing / Military。
- 将仓库 docs 技能改为通用文档工作流Planet 专属白名单、双语、裸文件标题和凭证教程规则迁移到 `docs/documentation-coverage-rules.md`
- 更新 AIS v4/v5 TODO 与计划文档,明确后续聚合策略配置、船舶资料 enrichment 和媒体缓存边界。
---
## [0.46.3] — 2026-04-30
Released: 2026-04-30
### 🐛 Fixes
- 优化 Starlink footprint 显示后的地球拖拽性能,避免旋转地球时每帧重建 footprint 大网格,同时保持现有视觉效果不变。
- 恢复点击线缆后的呼吸透明度动画,让 locked / hover 线缆重新使用既有 pulse 配置。
---
## [0.46.2] — 2026-04-30
Released: 2026-04-30

View File

@@ -0,0 +1,117 @@
# Documentation Coverage Rules
This file contains Planet-specific documentation coverage rules. Documentation skills and agents should read this file before deciding which docs to update. Keep tool-specific workflow in skills; keep product and repository rules here.
## Scope Rules
- User-visible workflow changes must update `docs/technical/zh/manual.md` and usually `docs/technical/zh/quickstart.md`.
- If an English counterpart exists for user-facing docs such as `manual.md` or `quickstart.md`, update `docs/technical/en/...` enough that it does not contradict the Chinese source.
- Control console page responsibility changes must update `docs/technical/zh/frontend-admin-frontend-context.md`.
- Earth frontend behavior changes must update `docs/technical/zh/earth-frontend-context.md`.
- Earth layer additions, `renderOrder`, altitude/radius offsets, depth strategy, pointer picking, legend modes, or layer panel/startup ordering must update `docs/technical/zh/earth-render-layer-order.md`.
- Earth layer visual style or legend symbol/color semantics should also update `docs/technical/zh/earth-layer-style-reference.md` when that reference is affected.
- Collector, datasource, credential, settings, connectivity, scheduler, or API changes must update the relevant backend docs, especially `docs/technical/zh/backend-collectors.md` and any datasource/settings-specific doc.
- When a change turns an old plan assumption into current behavior, update the relevant `docs/plans/*.md` with a status note instead of leaving contradictory instructions.
- Search docs for stale terms introduced by the change, for example old tab names, old route responsibilities, obsolete auth assumptions, or renamed UI labels.
## Public Docs Rules
- If adding a new technical document, add it to `docs/technical/zh/README.md` when it should be discoverable from the technical docs index.
- If a technical document should be visible in the public Docs page or linked from a technical README, register it in `frontend/src/pages/Docs/docs-content.ts` under `DOCS_METADATA`. Files under `docs/technical/{zh,en}/` are not automatically routable.
- For every public technical doc, keep the bilingual file pair in sync by filename: `docs/technical/zh/<name>.md` and `docs/technical/en/<name>.md`. If content is intentionally Chinese-only or English-only, state that intentionally in the final note.
- Public docs should use readable link text, not raw filenames such as `manual.md`.
## Credential Collector Rules
- Any built-in collector marked `requires_credentials: true` and `credential_status: supported` must have:
- a `credential_provider` in `backend/app/core/datasource_defaults.py`;
- a default credential guide in `backend/app/services/credential_guides.py`;
- a supported connectivity provider in `backend/app/services/datasource_connectivity.py`;
- settings UI guidance or a credential form in `frontend/src/pages/Settings/Settings.tsx`;
- a regression test that fails if the guide/provider is missing.
## Recommended Checks
Run the checks that match the affected docs.
### Duplicate Bilingual Docs
```bash
python - <<'PY'
from pathlib import Path
same = []
for en in sorted(Path("docs/technical/en").glob("*.md")):
zh = Path("docs/technical/zh") / en.name
if zh.exists() and en.read_text() == zh.read_text():
same.append(en.name)
if same:
raise SystemExit("identical en/zh docs: " + ", ".join(same))
print("no identical en/zh docs")
PY
```
### Language-Less Technical Links
```bash
rg -n "/home/ray/dev/linkong/planet/docs/technical/(?!zh|en)" docs/technical/zh --pcre2
```
This should return no matches.
### Public Docs Registry
```bash
python - <<'PY'
import re
from pathlib import Path
metadata = Path("frontend/src/pages/Docs/docs-content.ts").read_text()
known = set(re.findall(r"'([^']+\.md)':\s*\{", metadata))
known.add("README.md")
missing = []
for readme in [Path("docs/technical/zh/README.md"), Path("docs/technical/en/README.md")]:
if not readme.exists():
continue
for href in re.findall(r"\]\(([^)]+\.md)\)", readme.read_text()):
path = Path(href)
if "docs/technical/" not in href:
continue
filename = path.name
if filename not in known:
missing.append(f"{readme}: {filename}")
if missing:
raise SystemExit("docs README links missing DOCS_METADATA: " + ", ".join(missing))
print("docs README links are whitelisted")
PY
```
### Public Bilingual Pairs
```bash
python - <<'PY'
import re
from pathlib import Path
metadata = Path("frontend/src/pages/Docs/docs-content.ts").read_text()
filenames = sorted(set(re.findall(r"'([^']+\.md)':\s*\{", metadata)) - {"README.md"})
missing = []
for filename in filenames:
for lang in ("zh", "en"):
path = Path("docs/technical") / lang / filename
if not path.exists():
missing.append(str(path))
if missing:
raise SystemExit("missing bilingual docs: " + ", ".join(missing))
print("public docs have zh/en file pairs")
PY
```
### Raw Filename Link Titles
```bash
rg -n "\[[^]]+\.md\]\(" docs/technical/zh docs/technical/en
```
This should return no matches for polished public docs.

View File

@@ -1,6 +1,6 @@
# AIS 多源采集、冲突记录与聚合接口计划
**状态**:规划中
**状态**v0-v3 已实现v4+ 规划中
**创建日期**2026-04-30
**核心原则**:采集器只写原始观测;去重、合并、冲突解释放在聚合接口中完成
@@ -14,6 +14,8 @@
| 冲突处理 | 先记录冲突事实和当前选择原因,后续再开放用户规则配置 |
| 默认可信度 | 同类 AIS 数据源优先按 `delivery_mode` 评估:`realtime_stream` 优于 `batch_stream`,再优于 `polling``snapshot` |
| 过期保护 | 实时流源断流超过 freshness 窗口后,不能仅凭“实时源”身份压过更新的轮询数据 |
| 源健康状态 | 聚合时必须参考采集器健康状态,不能只看配置中的理论优先级 |
| 媒体富化 | 船只图片等媒体信息不进入 AIS 实时聚合主链路,后续单独做 enrichment |
## 背景
@@ -24,7 +26,7 @@
- WebSocket 或其他实时流通常更接近实时,但也可能断流或批量延迟。
- 如果每个 collector 自己做去重合并,规则会分散、不可审计,也很难让用户后续配置“某个字段信任哪个来源”。
因此 v1 不应让采集器直接覆盖最终船只表。更稳的方式是先保留观测事实,再由聚合接口统一给出当前展示视图。
因此第一阶段不应让采集器直接覆盖最终船只表。更稳的方式是先保留观测事实,再由聚合接口统一给出当前展示视图。
## 目标架构
@@ -53,11 +55,36 @@ flowchart LR
| `transport` | `websocket``sse``http``file` 等 |
| `observed_at` | 上游数据时间,优先使用 AIS 消息时间 |
| `collected_at` | 本系统接收或采集时间 |
| `source_message_id` | 上游消息 ID 或可推导 ID没有则为空 |
| `observation_hash` | 幂等去重指纹,用于防止同一来源重复写入同一条观测 |
| `normalized_payload` | 标准化后的 AIS JSON |
| `raw_payload` | 可选,保存原始或裁剪后的上游记录 |
| `quality_flags` | 观测级质量标记,例如 `stale``position_jump``future_timestamp` |
`delivery_mode``transport` 不应混为一谈。WebSocket 是传输方式streaming 是交付模式。聚合可信度主要看 `delivery_mode``transport` 只作为辅助信息。
原始观测层需要做存储级幂等去重,但这里的去重不是业务合并。推荐使用 `source + entity_key + message_type + observed_at + payload_hash` 或上游稳定消息 ID 作为唯一约束,避免 WebSocket 重连、HTTP 重试或批量回放导致同一事实重复入库。
### 源健康状态
每个采集器应维护独立的健康状态,供聚合服务读取:
| 字段 | 用途 |
|-----|------|
| `source` | 采集器标识 |
| `connection_state` | `connected``reconnecting``disconnected``disabled` 等 |
| `last_seen_at` | 最近收到上游消息或响应的时间 |
| `last_success_at` | 最近成功写入观测的时间 |
| `last_error` | 最近错误摘要 |
| `message_rate` | 最近窗口内的消息速率 |
| `lag_seconds` | 上游观测时间与本系统接收时间的延迟 |
聚合优先级不能只看 `source_priority`。例如 `aisstream_vessels` 默认优先于 `barentswatch_vessels`,但如果它处于 `disconnected``lag_seconds` 超过 freshness 窗口,则动态字段应回退到更新的可用来源。
### 身份键边界
v1 可以继续用 MMSI 作为 `entity_key`,因为它是 AIS 动态消息里最稳定、最容易获得的主键。但文档和模型都要为后续扩展留出口MMSI 可能复用、填错或缺少静态信息,后续身份解析应结合 `mmsi + imo + callsign + name + dimensions` 判断是否需要拆分或合并实体。
### 冲突记录层
聚合服务发现同一个实体、同一个字段存在多个非空不同值时,写入冲突记录。冲突记录不代表错误,只代表“有多个可用候选值”。
@@ -90,7 +117,8 @@ flowchart LR
| 动态位置 | `lat``lon``sog``cog``heading``nav_status` | 优先最新 `observed_at`,同时间再按来源优先级 |
| 静态身份 | `name``callsign``imo``flag` | 非空优先,再按字段策略或来源优先级 |
| 静态规格 | `vessel_type``vessel_type_name``length``width``draught` | 非空优先;冲突时记录候选值 |
| 元信息 | `field_sources``conflict_count``selected_reasons` | 聚合接口生成,便于调试和后续 UI 展示 |
| 轨迹点 | `track_points` | 按时间线合并;同一时间窗口内相近点去重;保留点级 `source` |
| 元信息 | `field_sources``conflict_count``selected_reasons``quality_flags` | 聚合接口生成,便于调试和后续 UI 展示 |
### 默认优先级
@@ -124,6 +152,27 @@ freshness:
如果 `aisstream_vessels` 最近 15 分钟没有该 MMSI 的新观测,而 BarentsWatch 轮询源有更新位置,则位置类字段应采用 BarentsWatch 的更新观测,并记录选择原因 `newest_observation``freshness_fallback`
### 异常位置保护
多源 AIS 接入后,聚合服务必须过滤或降权明显异常的位置观测:
- 经纬度必须在合法范围内。
- `observed_at` 不能明显来自未来。
- 同一 MMSI 短时间内跨越不合理距离时,标记 `position_jump`,默认不直接采用该点。
- 当异常点来自当前优先源时,应记录 `selected_reason = anomaly_rejected`,再回退到其他可用来源。
异常保护不应静默丢弃事实。原始观测仍应保留,聚合结果通过 `quality_flags` 和冲突记录解释为什么没有采用它。
### 轨迹聚合
轨迹接口不能简单拼接所有来源,否则前端会出现折返、抖动和重复点。默认规则:
-`observed_at` 排序,生成统一时间线。
- 同一来源的完全重复点通过 `observation_hash` 去重。
- 多来源在短时间窗口内上报的相近位置视为同一轨迹点,优先选择 freshness 和 source priority 更高的一条。
- 每个轨迹点保留 `source``selected_reason` 和必要的 `quality_flags`
- 对被判定为 `position_jump` 的点,默认不进入展示轨迹,但可通过调试参数查看。
## 聚合接口
现有展示接口应逐步改为消费聚合服务,而不是自己直接拼 `VesselPosition + VesselStatic`
@@ -155,6 +204,7 @@ GeoJSON properties 建议增加:
"lat": "newest_observation",
"vessel_type": "non_empty_priority"
},
"quality_flags": [],
"conflict_count": 2
}
```
@@ -231,15 +281,89 @@ aisstream_vessels:
- ShipStaticData
```
## 实施顺序
默认不建议直接订阅全球范围。AISStream 采集器应支持以下订阅策略:
1. 新增原始观测模型和冲突记录模型
2. 实现 AIS 聚合服务,先从现有 `vessel_position` / `vessel_static` 兼容读取,再逐步切换到原始观测层
3.`/geo/vessels``/vessels/{mmsi}` 改为走聚合服务
4. 改造 BarentsWatch 保存逻辑,让它写入原始观测,同时保留现有表作为兼容缓存
5. 实现 AISStream WebSocket collector
6. 接入系统设置中的聚合策略配置。
7. 做冲突治理 UI。
- 使用配置的固定 `bounding_boxes`
- 后续支持按 Earth 当前视口或关注区域动态调整订阅范围
- 支持限制 `message_types`,避免静态信息、位置报告和扩展消息全量涌入
- 断线后使用指数退避重连,并把连接状态写入源健康状态
- 重连后可能收到重复或回放消息,因此必须依赖原始观测层的幂等去重
### 媒体富化边界
VesselFinder 等服务里的船只图片不属于 AIS 实时数据本身。图片、船籍详情、公司信息等后续应作为独立 enrichment 链路:
- 通过 MMSI、IMO、船名等字段异步查询。
- 使用独立缓存和授权配置。
- 不阻塞 `vessel_ais` 实时观测入库。
- 聚合接口只暴露已经缓存好的媒体引用,不在请求链路中现场抓取。
## 版本拆分
计划按 5 个版本推进:
### v0 — 聚合基础设施(已实现)
目标是不改变前端展示行为,先把数据底座铺好。
1. 新增原始观测模型、冲突记录模型和源健康状态模型。
2. 为现有 BarentsWatch collector 写入原始观测,同时保留现有 `vessel_position` / `vessel_static` 兼容写入。
3. 实现存储级 `observation_hash` 幂等去重。
4. 补基础管理命令或调试接口,用于查看某个 MMSI 的原始观测和冲突候选。
### v1 — 聚合读接口(已实现)
目标是让展示接口开始消费聚合结果,但前端形状保持兼容。
1. 实现 AIS 聚合服务,先兼容读取现有表,再逐步切换到原始观测层。
2.`/geo/vessels``/vessels/{mmsi}` 改为走聚合服务。
3.`/vessels/{mmsi}/track` 改为走轨迹聚合逻辑。
4. 返回 `field_sources``selected_reasons``quality_flags``conflict_count`
5. 加入 freshness fallback 和异常位置保护。
### v2 — AISStream WebSocket collector已实现
目标是接入第二个真实 AIS 来源,并验证多源冲突和回退逻辑。
1. 实现 `aisstream_vessels` collector。
2. 支持 API key、订阅范围、消息类型、重连和限流配置。
3. 将 AISStream 写入原始观测层,不直接 upsert 最终展示表。
4. 接入源健康状态和 message rate 统计。
5. 提供 AISStream API Key 获取教程、设置页入口和连接验证支持。
6. 为重复消息、断流回退、WS 优先级写集成测试。
### v3 — AISStream 可用性与配置体验(已实现)
目标是让 AISStream 从“能采集”变成日常可观察、可调试、可配置的数据源。
1. 设置页展示 AISStream 运行状态:连接状态、最近收到、最近成功、本轮消息数、延迟和最近错误。
2. AISStream 设置页提供常用采集范围 preset并保留自定义 Bounding Boxes JSON。
3. 聚合结果返回 `source_summary`,展示每艘船的来源、观测数量、最新观测时间、传输模式和消息类型。
4. 保留 `field_sources``selected_reasons`,用于解释动态字段来自实时流、静态字段来自可用非空来源。
5. 船名标准化会读取 AISStream `MetaData.ShipName`;船型展示会从 `vessel_type_name` 和 AIS 数字 `vessel_type` 共同归一化,保证 marker 颜色、详情卡、hover 和搜索结果一致。
6. `/geo/vessels` 不再默认限制 5000 艘;不传 `limit` 或传 `limit=0` 表示全量返回,前端默认也不再二次裁剪到 5000。
### v4 — 策略配置
目标是开放系统级配置,但仍以安全默认值兜底。
1. 接入系统设置中的聚合策略配置。
2. 支持 source priority、字段级规则、freshness 窗口和高级保护开关。
3. 保存配置时校验未知字段、非法模式和危险动态字段锁定。
4. 聚合接口返回当前命中的配置版本,方便排查。
### v5 — 船舶资料 enrichment 与冲突治理
目标是把 AIS 实时流里不稳定或低频出现的静态信息,补成可缓存、可审计的船舶资料层,同时把冲突解释变成可操作能力。
1. 做冲突治理 UI。
2. 支持把人工选择沉淀成字段级规则。
3. 支持恢复默认策略。
4. 设计 `vessel_profile_enrichment`,按 `mmsi + imo + name + callsign` 异步补充船名、船型细分、AIS 大类、旗国、尺寸、建造年份、运营方等静态资料。
5. 设计 `vessel_media_enrichment`,异步补充船只图片和外部详情缓存。
6. enrichment 结果必须带 `source``fetched_at``expires_at``confidence` 和原始引用,不覆盖 AIS 原始观测。
7. 聚合接口只读取已缓存 enrichment请求链路不现场抓取第三方页面避免慢请求和授权风险。
8. 前端船只详情面板展示已缓存资料和媒体,并标注字段来源,不阻塞 AIS 实时链路。
## 测试计划
@@ -247,8 +371,12 @@ aisstream_vessels:
- 多来源同一 MMSI 的位置字段优先选择最新观测。
- 实时流和轮询源同时间冲突时,实时流优先。
- 实时流过期后,更新的轮询源可以接管动态字段。
- 实时流源健康状态异常时,动态字段可以回退到更新的可用来源。
- 静态字段不会被空值覆盖。
- 静态字段冲突会写入冲突记录。
- 明显异常位置不会进入默认展示轨迹,并会留下 `quality_flags`
- 同一时间窗口内多来源相近轨迹点只展示一个点。
- AISStream 重连或回放导致的重复消息不会重复进入聚合结果。
- 字段级配置可以覆盖默认来源优先级。
- 聚合接口在没有冲突表时仍可返回兼容 GeoJSON。

View File

@@ -124,7 +124,7 @@ CREATE UNIQUE INDEX ON vessel_latest(mmsi);
GET /api/v1/visualization/geo/vessels
?bbox=lon_min,lat_min,lon_max,lat_max # 视口裁剪
?type=cargo,tanker,passenger # 船型过滤
?limit=5000
?limit=0 # 可选;不传或 0 表示不裁剪数量
→ GeoJSON FeatureCollectionPoint
GET /api/v1/visualization/vessels/{mmsi} # 单船详情
@@ -163,6 +163,8 @@ GeoJSON Feature 格式:
- 后端 BarentsWatch collector 继续以 HTTP polling 方式采集
- AISStream 等实时源以独立 WebSocket collector 写入原始观测层
- 展示接口从聚合服务读取当前船只视图,而不是由单个 collector 决定最终展示值
- 前端默认不再给 `/geo/vessels``limit=5000``VESSEL_CONFIG.maxRenderedMarkers = 0` 表示不做前端数量裁剪;后续如性能不足再引入显式 LOD 上限
- marker 颜色、详情卡、hover 和搜索结果必须共享 `vessel_type_display` 船型归一化结果,避免 AIS 数字类型码已驱动颜色但卡片仍显示 `Other`
- 前端是否升级为 WebSocket delta push 是独立优化,不影响后端采集器可以使用 WebSocket 接上游实时源
---
@@ -196,8 +198,8 @@ GeoJSON Feature 格式:
| 相机距离 | 渲染策略 |
|---------|---------|
| > 400 | 仅渲染 top 1000 艘(按数据新鲜度 + 船型优先级) |
| 200400 | 渲染 top 5000 艘 |
| > 400 | 默认渲染当前接口返回的全部船只;如性能不足,再引入可配置 LOD 上限 |
| 200400 | 默认渲染当前接口返回的全部船只;如性能不足,再引入可配置 LOD 上限 |
| < 200 | 渲染当前视口 bbox 内全部船只 |
前端根据相机位置动态计算 bbox附加到 API 请求中。

View File

@@ -84,6 +84,8 @@ async def run(self, db):
| HuggingFace Spaces | space | Demo applications | 1 day |
| PeeringDB | ixp/network/facility | Internet exchange points / networks / facilities | 1-2 days |
| TeleGeography | submarine_cable | Submarine cable information | 7 days |
| BarentsWatch AIS | vessel | AIS vessel positions, speed, heading, MMSI, and related fields | Collector settings |
| AISStream Vessels | vessel_ais | AIS WebSocket realtime stream, written to the raw observation layer and displayed through aggregation | Collector settings |
## IV. Data Format (stored in CollectedData table)

View File

@@ -260,6 +260,30 @@ AIS request rules:
`VesselAISCollector` no longer reads environment variables directly. It goes through `resolve_barentswatch_config()` and `fetch_barentswatch_access_token()` so settings, connectivity validation, and collection do not fork into three credential flows.
## AISStream Collector Chain
Files:
- [aisstream.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/aisstream.py)
- [vessel_ais_aggregation.py](/home/ray/dev/linkong/planet/backend/app/services/vessel_ais_aggregation.py)
AISStream uses a WebSocket realtime stream. The collector writes only to the `ais_raw_observations` raw observation layer; it does not directly overwrite the final vessel display table. The aggregation API handles multi-source deduplication, field selection, and conflict records.
Configuration:
- `api_key`: stored in `DataSourceConfig.auth_config`, or provided through `AISSTREAM_API_KEY`.
- `endpoint`: defaults to `wss://stream.aisstream.io/v0/stream`.
- `message_types`: defaults to `PositionReport` and `ShipStaticData`.
- `bounding_boxes`: AISStream format is `[[[lat_min, lon_min], [lat_max, lon_max]]]`; the settings page provides global, Norway / North Sea, Europe coast, East Asia, and North America coast presets.
- `max_messages` and `receive_timeout_seconds`: control the batch-style WebSocket collection window.
Normalization:
- `PositionReport` mainly provides position, speed, course, heading, and navigation status.
- Vessel names can be filled from `MetaData.ShipName` even when the message body has no `name`.
- Vessel type usually comes from lower-frequency `ShipStaticData.Type`; the backend maps AIS numeric type codes to Cargo / Tanker / Passenger / Fishing / Military.
- If a vessel has not yet produced a static message, its aggregated type can still be `Other`; v5 vessel profile enrichment is planned to fill that gap.
## Credential Guide
File:
@@ -277,6 +301,7 @@ POST /api/v1/settings/credential-guides/{provider}/reset
Currently supported:
- `barentswatch`
- `aisstream`
The default guide includes the official BarentsWatch tutorial:

View File

@@ -96,6 +96,7 @@ Responsibilities:
- [satellites.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/satellites.js)
- [cables.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/cables.js)
- [vessels.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/vessels.js)
- [bgp.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp.js)
- [bgp-cruise-adapter.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp-cruise-adapter.js)
- [compute-centers.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/compute-centers.js)
@@ -108,6 +109,14 @@ Each module is responsible for its own:
- State tracking (loaded, visible, hover, locked)
- Self-cleanup (dispose on scene destroy)
### AIS Vessel Layer
The vessel layer fetches `/api/v1/visualization/geo/vessels` and renders the aggregated AIS GeoJSON through `createInteractableLayer()`. By default it does not send a `limit` parameter, and `VESSEL_CONFIG.maxRenderedMarkers = 0` means the frontend does not clip the result to 5000 vessels. A positive `options.limit` or positive `maxRenderedMarkers` can still be used as an explicit temporary cap.
Vessel color and vessel type text must use the same normalized classification. `vessels.js` derives `type` from both `vessel_type_name` and the AIS numeric `vessel_type` code; that `type` drives marker color. It also derives `vessel_type_display`, which `main.js` uses for the info card, hover summary, and search result subtitle. Do not make the info card read only the raw `vessel_type_name`, because AISStream can provide a numeric type while the raw name is still `Other`.
AISStream `PositionReport` messages commonly carry live position and `MetaData.ShipName`, while vessel type usually comes from lower-frequency `ShipStaticData.Type`. The backend normalizes `MetaData.ShipName` into the vessel name and maps numeric type codes into Cargo / Tanker / Passenger / Fishing / Military where available. Missing type detail should wait for a static AIS message or the planned vessel profile enrichment; the frontend should not invent a more specific type.
### 7. HUD Panels and Search
- [hud-panels.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/hud-panels.js)

View File

@@ -182,11 +182,14 @@ The land/ocean base is an Earth base-map asset and preloads at startup; the "Bor
| Vessel renderOrder | local `VESSEL_RENDER_ORDER` | `4.4` | Normal marker and interactive overlay |
| Vessel track renderOrder | `VESSEL_RENDER_ORDER - 0.1` | `4.3` | Below vessel markers |
| Vessel point pixel size | local `VESSEL_POINT_SIZE` | `34` | Shared size for normal markers and hover / locked overlays |
| Default vessel render cap | `VESSEL_CONFIG.maxRenderedMarkers` | `0` | `0` means the frontend does not clip by default; positive values send `limit` and clip markers |
| Vessel texture canvas size | local `VESSEL_ATLAS_CELL_SIZE` | `128` | Canvas point texture |
| Course bucket count | local `VESSEL_COURSE_BINS` | `32` | Moving vessels are bucketed by COG to reduce draw calls while preserving direction |
| Vessel hover picking throttle | local `VESSEL_HOVER_PICK_INTERVAL_MS` | `100` | `main.js` hover picking |
| Vessel screen hit radius | local `VESSEL_POINTER_RADIUS_PX` | `22` | `main.js` screen-space picking |
AIS vessel markers use batched `THREE.Points`, not one `THREE.Sprite` per vessel. Moving vessels stay triangular, anchored or slow vessels stay circular, and hover / locked states add a same-size glow overlay. Vessel type color and info-card type text must come from the same normalized result: `vessels.js` reads both backend `vessel_type_name` and AIS numeric `vessel_type`, derives the color-driving `type`, then exposes `vessel_type_display` for the info card, hover summary, and search results.
## Compute Centers
| Name | Variable | Current Value | Location / Notes |

View File

@@ -86,6 +86,7 @@ async def run(self, db):
| TeleGeography | submarine_cable | 海底光缆信息 | 7天 |
| Space-Track TLE | satellite_tle | 卫星轨道 TLE 数据 | 依采集器配置 |
| BarentsWatch AIS | vessel | 船只位置、航速、航向、MMSI 等 AIS 数据 | 依采集器配置 |
| AISStream Vessels | vessel_ais | AIS WebSocket 实时流,写入原始观测层并由聚合接口展示 | 依采集器配置 |
## 四、数据格式 (统一存储到 CollectedData 表)
@@ -238,7 +239,8 @@ backend/app/services/collectors/
├── huggingface.py # HuggingFace采集器
├── peeringdb.py # PeeringDB采集器
├── telegeraphy.py # TeleGeography海底光缆采集器
── vessel_ais.py # BarentsWatch AIS 船只采集器
── vessel_ais.py # BarentsWatch AIS 船只采集器
└── aisstream.py # AISStream WebSocket 船只采集器
backend/app/models/
└── collected_data.py # 统一数据模型
@@ -251,6 +253,7 @@ backend/app/models/
| 采集器 | credential provider | 凭证来源 |
| --- | --- | --- |
| `barentswatch_vessels` | `barentswatch` | 控制台采集器设置、环境变量、`~/.zshrc` |
| `aisstream_vessels` | `aisstream` | 控制台采集器设置、环境变量 |
| `spacetrack_tle` | `spacetrack` | 环境变量、`~/.zshrc` |
### BarentsWatch AIS

View File

@@ -262,6 +262,30 @@ AIS 请求规则:
`VesselAISCollector` 不再自己读取环境变量,而是统一走 `resolve_barentswatch_config()``fetch_barentswatch_access_token()`,避免设置页、连接验证和采集器三套凭证逻辑分叉。
## AISStream 采集器链路
文件:
- [aisstream.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/aisstream.py)
- [vessel_ais_aggregation.py](/home/ray/dev/linkong/planet/backend/app/services/vessel_ais_aggregation.py)
AISStream 使用 WebSocket 实时流,采集器只写入 `ais_raw_observations` 原始观测层,不直接覆盖最终船只展示表。聚合接口负责多源去重、字段选择和冲突记录。
配置项:
- `api_key`:保存在 `DataSourceConfig.auth_config`,也可用环境变量 `AISSTREAM_API_KEY`
- `endpoint`:默认 `wss://stream.aisstream.io/v0/stream`
- `message_types`:默认 `PositionReport``ShipStaticData`
- `bounding_boxes`AISStream 格式为 `[[[lat_min, lon_min], [lat_max, lon_max]]]`,设置页提供全球、挪威 / 北海、欧洲近海、东亚、北美东西海岸 preset。
- `max_messages``receive_timeout_seconds`:控制单次批次式 WebSocket 采集窗口。
标准化规则:
- `PositionReport` 主要提供位置、速度、航向和状态。
- 船名可以从 `MetaData.ShipName` 补入,即使消息体本身没有 `name`
- 船型通常来自低频 `ShipStaticData.Type`;后端会把 AIS 数字类型码映射为 Cargo / Tanker / Passenger / Fishing / Military。
- 如果某艘船尚未收到静态消息,聚合结果的船型仍可能是 `Other`,后续由 v5 船舶资料 enrichment 补齐。
## 凭证教程
文件:
@@ -279,6 +303,7 @@ POST /api/v1/settings/credential-guides/{provider}/reset
当前支持:
- `barentswatch`
- `aisstream`
默认教程包含 BarentsWatch 官方 tutorial 地址:

View File

@@ -254,10 +254,10 @@ AIS 船只图层入口:
船只图层当前负责:
- 请求 `/api/v1/visualization/geo/vessels`
- BarentsWatch AIS GeoJSON 转为地球局部坐标 marker 数据
-聚合后的 AIS GeoJSON 转为地球局部坐标 marker 数据;请求默认不传 `limit`,后端和前端都不再默认裁剪到 5000 艘
- 通过 `createInteractableLayer()` 注册 Interactable 图标层
- 用按航向分桶的 `THREE.Points` 批量渲染普通船只 marker
- 按船型映射颜色
- 按船型映射颜色`vessels.js` 会用 `vessel_type_name` 和 AIS `vessel_type` 数字共同归一化船型
- 根据航行/停泊状态绘制三角形或圆点纹理
- 用单点 `THREE.Points` overlay 承载 hover / locked glow
- 支持 hover、lock、轨迹加载和视觉聚焦
@@ -273,6 +273,10 @@ AIS 船只图层入口:
方向标准以 AIS `course / cog` 为准:从正北开始顺时针。普通态和交互态都通过同一套 canvas 旋转规则生成纹理,避免 hover 后箭头方向和原 marker 不一致。
船型展示也必须复用同一套归一化结果。`buildVesselMarkerData()` 会把后端的 `vessel_type_name` 和 AIS 数字类型码归一化为 `type`,用于 marker 颜色;同时生成 `vessel_type_display`供详情卡、hover 简述和搜索结果显示。不要让详情卡直接只读原始 `vessel_type_name`,否则会出现 marker 已按 Cargo/Tanker 等颜色显示、卡片仍写 `Other` 的不一致。
AISStream 的 `PositionReport` 常带实时位置和 `MetaData.ShipName`,但船型通常来自低频 `ShipStaticData.Type`。后端会把 `MetaData.ShipName` 补进船名,并将类型码映射为 Cargo / Tanker / Passenger / Fishing / Military仍缺失的船型需要等待静态 AIS 消息或后续船舶资料 enrichment不能在前端凭颜色之外的信息臆造细分类。
船只 hover / click 也不再对渲染对象做 `raycaster.intersectObjects()``main.js` 只负责传入当前 Earth、camera、pointer 和命中半径,实际命中计算由 `interactable.js` 的图标层接口完成:
1. 拖动地球或惯性旋转时跳过 hover picking。

View File

@@ -194,6 +194,7 @@
| 船只 renderOrder | local `VESSEL_RENDER_ORDER` | `4.4` | 普通 marker 和交互 overlay |
| 船只轨迹 renderOrder | `VESSEL_RENDER_ORDER - 0.1` | `4.3` | 低于船只 marker |
| 船只点像素尺寸 | local `VESSEL_POINT_SIZE` | `34` | 普通 marker 与 hover / locked overlay 共享尺寸 |
| 船只默认渲染上限 | `VESSEL_CONFIG.maxRenderedMarkers` | `0` | `0` 表示不在前端默认裁剪;正数才会给接口传 `limit` 并裁剪 marker |
| 船只纹理画布尺寸 | local `VESSEL_ATLAS_CELL_SIZE` | `128` | canvas 点纹理 |
| 航向分桶数 | local `VESSEL_COURSE_BINS` | `32` | moving 船只按 COG 分桶,降低 draw call 同时保留方向 |
| 船只 hover 拾取节流 | local `VESSEL_HOVER_PICK_INTERVAL_MS` | `100` | `main.js` hover picking |
@@ -206,6 +207,8 @@
AIS 船只普通态使用批量 `THREE.Points`,不是逐船 `THREE.Sprite`。航行船只保持三角形,停泊或低速船只保持圆点;普通态不带 glowhover / locked 时在同一屏幕尺寸上叠加带 glow 的单点 overlay。AIS 航向按 `course / cog` 从正北顺时针解释,普通态和交互态必须使用同一套 canvas 旋转规则。
船型颜色和详情卡船型文本必须来自同一套归一化结果:`vessels.js` 同时读取后端 `vessel_type_name` 和 AIS 数字 `vessel_type`,先得到颜色用的 `type`,再生成 `vessel_type_display` 给详情卡、hover 和搜索使用。
## 算力中心
| 正式名称 | 变量名 | 当前值 | 使用位置 / 说明 |

View File

@@ -16,12 +16,14 @@
## Current Version
- `main` 当前主线历史推导到:`0.16.5`
- `dev` 当前开发分支历史推导到:`0.46.2`
- `dev` 当前开发分支历史推导到:`0.47.0`
## Timeline
| Version | Type | Branch | Commit | Summary |
| --- | --- | --- | --- | --- |
| `0.47.0` | feature | `dev` | `pending` | 新增 AISStream WebSocket 船只采集器、多源 AIS 原始观测聚合、采集器状态配置、船型显示修正和文档规则解耦 |
| `0.46.3` | bugfix | `dev` | `pending` | 优化 Starlink footprint 拖拽性能,避免旋转地球时重复重建覆盖网格,并恢复线缆点击呼吸动画 |
| `0.46.2` | bugfix | `dev` | `pending` | 修复 Earth 启动加载顺序、图层 localStorage 恢复、国界线底图语义、媒体面板、船只轨迹和 Iridium footprint 显示问题,并补充 AIS 聚合计划 |
| `0.46.1` | bugfix | `dev` | `pending` | 修复新增 Docs 技术文档未进前端白名单导致页面不可访问的问题,补齐英文文档并固化白名单/双语/裸文件标题检查 |
| `0.46.0` | feature | `dev` | `pending` | Earth 新增通用 Interactable 图标层统一船只、算力中心、BGP 事件/观测站交互图标,并优化登陆点与 toolbar 初始渲染 |

View File

@@ -1,6 +1,6 @@
{
"name": "planet-frontend",
"version": "0.46.2",
"version": "0.47.0",
"private": true,
"packageManager": "bun@1",
"dependencies": {

View File

@@ -210,7 +210,7 @@ export const PATHS = {
export const VESSEL_CONFIG = {
altitudeOffset: 0.2,
maxRenderedMarkers: 5000,
maxRenderedMarkers: 0,
marker: {
baseScale: 7.5,
baseOpacity: 0.88,

View File

@@ -787,12 +787,13 @@ function formatVesselStatus(navStatus) {
function showVesselInfo(marker, coords) {
setLegendMode("vessels");
const vesselType = marker.userData?.vessel_type_display || marker.userData?.vessel_type_name || "-";
showInfoCard("vessel", {
name: marker.userData?.name || `MMSI ${marker.userData?.mmsi}`,
mmsi: marker.userData?.mmsi,
imo: marker.userData?.imo || "-",
flag: marker.userData?.flag || "-",
vessel_type: marker.userData?.vessel_type_name || "-",
vessel_type: vesselType,
speed: marker.userData?.sog ?? "-",
course: marker.userData?.cog ?? marker.userData?.heading ?? "-",
status: formatVesselStatus(marker.userData?.nav_status),
@@ -806,7 +807,8 @@ function showVesselInfo(marker, coords) {
function getVesselBriefHtml(marker) {
const name = marker.userData?.name || `MMSI ${marker.userData?.mmsi}`;
const speed = marker.userData?.sog ?? "-";
return `<strong>${name}</strong><br>${marker.userData?.vessel_type_name || "Vessel"} · ${speed} kn`;
const vesselType = marker.userData?.vessel_type_display || marker.userData?.vessel_type_name || "Vessel";
return `<strong>${name}</strong><br>${vesselType} · ${speed} kn`;
}
function getComputeCenterBriefHtml(marker) {
@@ -1390,6 +1392,7 @@ function resolveEarthSearchResults(query) {
marker.userData?.mmsi,
marker.userData?.imo,
marker.userData?.flag,
marker.userData?.vessel_type_display,
marker.userData?.vessel_type_name,
"船只 船舶 ais vessel ship maritime",
);
@@ -1401,7 +1404,7 @@ function resolveEarthSearchResults(query) {
typeLabel: "船只",
title: marker.userData?.name || `MMSI ${marker.userData?.mmsi}`,
subtitle: [
marker.userData?.vessel_type_name,
marker.userData?.vessel_type_display || marker.userData?.vessel_type_name,
marker.userData?.flag,
marker.userData?.sog !== undefined ? `${marker.userData.sog} kn` : null,
].filter(Boolean).join(" · ") || "AIS 船只",
@@ -2036,7 +2039,11 @@ function applyCableVisualState() {
switch (state) {
case CABLE_STATE.LOCKED:
case CABLE_STATE.HOVERED:
cable.material.opacity = 1;
cable.material.opacity = THREE.MathUtils.lerp(
CABLE_CONFIG.lockedOpacityMin,
CABLE_CONFIG.lockedOpacityMax,
pulse,
);
cable.material.color.setRGB(0.92, 0.98, 1.0);
break;
case CABLE_STATE.NORMAL:

View File

@@ -28,6 +28,7 @@ let lockedRingSprite = null;
let lockedDotSprite = null;
let lockedHaloMesh = null;
let lockedGroundFootprintMesh = null;
let lockedGroundFootprintFillMesh = null;
let lockedIridiumFootprintMesh = null;
let predictedOrbitLine = null;
let relatedSatelliteSprites = [];
@@ -159,6 +160,9 @@ const GROUND_FOOTPRINT_GAP_CENTER_MAX_RATIO = 0.82;
const GROUND_FOOTPRINT_GAP_WIDTH_CENTER_KM = 60;
const GROUND_FOOTPRINT_GAP_WIDTH_EDGE_KM = 120;
const GROUND_FOOTPRINT_GAP_LENGTH_RATIO = 1.08;
const GROUND_FOOTPRINT_REBUILD_DISTANCE = 0.001;
const GROUND_FOOTPRINT_REBUILD_DISTANCE_SQ =
GROUND_FOOTPRINT_REBUILD_DISTANCE * GROUND_FOOTPRINT_REBUILD_DISTANCE;
const scratchWorldSatellitePosition = new THREE.Vector3();
const scratchToCamera = new THREE.Vector3();
@@ -168,7 +172,9 @@ const scratchFootprintLateral = new THREE.Vector3();
const scratchFootprintReference = new THREE.Vector3();
const scratchFootprintVelocity = new THREE.Vector3();
const scratchFootprintTangent = new THREE.Vector3();
const scratchLastGroundFootprintPosition = new THREE.Vector3();
const satelliteSunDirection = new THREE.Vector3(1, 0.2, 0.4).normalize();
let hasGroundFootprintGeometry = false;
export let breathingPhase = 0;
@@ -1335,13 +1341,10 @@ export function setSatelliteCamera(camera) {
export function setSatelliteSunDirection(direction) {
if (!direction) return;
satelliteSunDirection.copy(direction).normalize();
if (lockedGroundFootprintMesh) {
const fillMesh = lockedGroundFootprintMesh.getObjectByName("footprint-fill");
if (fillMesh?.material?.uniforms?.uSunDirectionWorld) {
fillMesh.material.uniforms.uSunDirectionWorld.value.copy(
satelliteSunDirection,
);
}
if (lockedGroundFootprintFillMesh?.material?.uniforms?.uSunDirectionWorld) {
lockedGroundFootprintFillMesh.material.uniforms.uSunDirectionWorld.value.copy(
satelliteSunDirection,
);
}
}
@@ -1586,14 +1589,12 @@ function createGroundFootprintMaterial() {
},
vertexShader: `
varying vec3 vWorldPosition;
varying vec3 vWorldNormal;
varying vec2 vUv;
void main() {
vUv = uv;
vec4 worldPosition = modelMatrix * vec4(position, 1.0);
vWorldPosition = worldPosition.xyz;
vWorldNormal = normalize(mat3(modelMatrix) * normal);
gl_Position = projectionMatrix * modelViewMatrix * vec4(position, 1.0);
}
`,
@@ -1617,7 +1618,6 @@ function createGroundFootprintMaterial() {
uniform vec3 uSunDirectionWorld;
uniform float uDayVisibilityBoost;
varying vec3 vWorldPosition;
varying vec3 vWorldNormal;
varying vec2 vUv;
float bowtieHalfWidth(float xEast) {
@@ -1723,6 +1723,8 @@ function clearLockedSatelliteStyleVisuals() {
if (lockedGroundFootprintMesh) {
disposeObjectTree(lockedGroundFootprintMesh);
lockedGroundFootprintMesh = null;
lockedGroundFootprintFillMesh = null;
hasGroundFootprintGeometry = false;
}
if (lockedIridiumFootprintMesh) {
disposeIridiumFootprintAdapter(lockedIridiumFootprintMesh, earthObjRef);
@@ -1933,28 +1935,6 @@ function buildGroundFootprintGeometry(position) {
);
}
function toEastNorth(alongKm, crossKm) {
const offset = alongTrack
.clone()
.multiplyScalar(alongKm)
.addScaledVector(crossTrack, crossKm);
return {
xEast: offset.dot(east),
yNorth: offset.dot(north),
};
}
function fromEastNorth(xEast, yNorth) {
const offset = east
.clone()
.multiplyScalar(xEast)
.addScaledVector(north, yNorth);
return {
alongKm: offset.dot(alongTrack),
crossKm: offset.dot(crossTrack),
};
}
function bowtieHalfWidth(xEast) {
const t = THREE.MathUtils.clamp(
Math.abs(xEast) / Math.max(exclusionLengthKm, 1),
@@ -1969,6 +1949,7 @@ function buildGroundFootprintGeometry(position) {
}
const vertices = [];
const uvs = [];
const indices = [];
const indexMap = [];
@@ -2000,6 +1981,10 @@ function buildGroundFootprintGeometry(position) {
);
row.push(vertices.length / 3);
vertices.push(point.x, point.y, point.z);
uvs.push(
THREE.MathUtils.mapLinear(alongKm, -majorKm, majorKm, 0, 1),
THREE.MathUtils.mapLinear(crossKm, -minorKm, minorKm, 0, 1),
);
}
indexMap.push(row);
}
@@ -2021,29 +2006,8 @@ function buildGroundFootprintGeometry(position) {
"position",
new THREE.Float32BufferAttribute(vertices, 3),
);
const uvs = [];
for (let iy = 0; iy <= GROUND_FOOTPRINT_GRID_Y; iy += 1) {
const crossKm = THREE.MathUtils.lerp(
-minorKm,
minorKm,
iy / GROUND_FOOTPRINT_GRID_Y,
);
for (let ix = 0; ix <= GROUND_FOOTPRINT_GRID_X; ix += 1) {
const alongKm = THREE.MathUtils.lerp(
-majorKm,
majorKm,
ix / GROUND_FOOTPRINT_GRID_X,
);
if (!isInsideEllipse(alongKm, crossKm)) continue;
uvs.push(
THREE.MathUtils.mapLinear(alongKm, -majorKm, majorKm, 0, 1),
THREE.MathUtils.mapLinear(crossKm, -minorKm, minorKm, 0, 1),
);
}
}
fillGeometry.setAttribute("uv", new THREE.Float32BufferAttribute(uvs, 2));
fillGeometry.setIndex(indices);
fillGeometry.computeVertexNormals();
const basisToEastNorth = {
eastAlongDot: alongTrack.dot(east),
@@ -2064,10 +2028,18 @@ function buildGroundFootprintGeometry(position) {
function updateGroundFootprintTransform(position) {
if (!lockedGroundFootprintMesh || !position || !earthObjRef) return;
if (
hasGroundFootprintGeometry &&
scratchLastGroundFootprintPosition.distanceToSquared(position) <=
GROUND_FOOTPRINT_REBUILD_DISTANCE_SQ
) {
return;
}
const geometrySet = buildGroundFootprintGeometry(position);
if (!geometrySet) return;
const fillMesh = lockedGroundFootprintMesh.getObjectByName("footprint-fill");
const fillMesh = lockedGroundFootprintFillMesh;
if (fillMesh?.geometry) fillMesh.geometry.dispose();
@@ -2089,6 +2061,8 @@ function updateGroundFootprintTransform(position) {
fillMesh.material.uniforms.uNorthCrossDot.value =
geometrySet.basisToEastNorth.northCrossDot;
}
scratchLastGroundFootprintPosition.copy(position);
hasGroundFootprintGeometry = true;
}
}
@@ -2130,6 +2104,8 @@ function showGroundFootprintStyle(position) {
fill.name = "footprint-fill";
fill.renderOrder = GROUND_FOOTPRINT_RENDER_ORDER;
lockedGroundFootprintMesh.add(fill);
lockedGroundFootprintFillMesh = fill;
hasGroundFootprintGeometry = false;
earthObjRef.add(lockedGroundFootprintMesh);
updateGroundFootprintTransform(position);
}

View File

@@ -24,6 +24,17 @@ function normalizeVesselType(value, code) {
return "other";
}
function formatVesselTypeLabel(type, fallback = "") {
const rawFallback = String(fallback || "").trim();
const normalized = String(type || "").trim().toLowerCase();
if (normalized === "cargo") return "Cargo";
if (normalized === "tanker") return "Tanker";
if (normalized === "passenger") return "Passenger";
if (normalized === "fishing") return "Fishing";
if (normalized === "military") return "Military";
return rawFallback && rawFallback.toLowerCase() !== "other" ? rawFallback : "Other";
}
function drawVesselShape(context, anchored, glow, color = "#ffffff") {
context.fillStyle = color;
context.globalAlpha = anchored ? 0.55 : 0.96;
@@ -52,6 +63,7 @@ function buildVesselMarkerData(feature) {
if (!Number.isFinite(latitude) || !Number.isFinite(longitude)) return null;
const type = normalizeVesselType(props.vessel_type_name, props.vessel_type);
const vesselTypeLabel = formatVesselTypeLabel(type, props.vessel_type_name);
const navStatus = Number(props.nav_status);
const speed = Number(props.sog);
const anchored = navStatus === 1 || navStatus === 5 || (Number.isFinite(speed) && speed < 0.5);
@@ -61,6 +73,7 @@ function buildVesselMarkerData(feature) {
latitude,
longitude,
type,
vessel_type_display: vesselTypeLabel,
anchored,
course: Number(props.cog ?? props.heading ?? 0),
};
@@ -191,7 +204,10 @@ export function clearVesselData(earth) {
export async function loadVessels(_scene, earth, options = {}) {
const params = new URLSearchParams();
params.set("limit", String(options.limit || VESSEL_CONFIG.maxRenderedMarkers));
const requestedLimit = Number(options.limit ?? VESSEL_CONFIG.maxRenderedMarkers);
if (Number.isFinite(requestedLimit) && requestedLimit > 0) {
params.set("limit", String(requestedLimit));
}
const response = await fetch(`${PATHS.vesselsApi}?${params.toString()}`);
if (!response.ok) {
throw new Error(`Vessels HTTP ${response.status}`);
@@ -200,10 +216,12 @@ export async function loadVessels(_scene, earth, options = {}) {
const features = Array.isArray(payload?.features) ? payload.features : [];
clearVesselData(earth);
const markerData = features
let markerData = features
.map((feature) => buildVesselMarkerData(feature))
.filter(Boolean)
.slice(0, VESSEL_CONFIG.maxRenderedMarkers);
.filter(Boolean);
if (Number.isFinite(requestedLimit) && requestedLimit > 0) {
markerData = markerData.slice(0, requestedLimit);
}
vesselIconLayer.setData(markerData);
vesselIconLayer.attach(earth);

View File

@@ -26,6 +26,7 @@ import {
SyncOutlined,
} from '@ant-design/icons'
import axios, { type AxiosResponse } from 'axios'
import { useNavigate } from 'react-router-dom'
import AppLayout from '../../components/AppLayout/AppLayout'
import ScrollbarOverlay from '../../components/Scrollbar/ScrollbarOverlay'
@@ -224,6 +225,7 @@ function normalizeCustom(source: CustomDataSource): UnifiedDataSource {
function DataSources() {
const [messageApi, contextHolder] = message.useMessage()
const navigate = useNavigate()
const [modal, modalContextHolder] = Modal.useModal()
const [builtInSources, setBuiltInSources] = useState<BuiltInDataSource[]>([])
const [customSources, setCustomSources] = useState<CustomDataSource[]>([])
@@ -736,6 +738,15 @@ function DataSources() {
showIcon
message="需要采集器凭证"
description={viewingSource.credential_status === 'supported' ? '请在设置中心的采集器设置中维护该采集器凭证。' : '该采集器需要凭证,配置入口待接入。'}
action={viewingSource.credential_status === 'supported' ? (
<Button
size="small"
type="primary"
onClick={() => navigate(`/settings?tab=collector_credentials&collector=${encodeURIComponent(viewingSource.source)}`)}
>
</Button>
) : undefined}
/>
) : null}

View File

@@ -82,6 +82,18 @@ interface CollectorSettings {
requires_credentials?: boolean
credential_provider?: string | null
credential_status?: string
ais_health?: AISSourceHealth | null
}
interface AISSourceHealth {
source: string
connection_state: string
last_seen_at: string | null
last_success_at: string | null
last_error: string | null
message_rate: number | null
lag_seconds: number | null
updated_at: string | null
}
interface TVStreamSource {
@@ -166,12 +178,59 @@ interface CollectorConfigOption {
is_active: boolean
source_type: string
auth_type: string
auth_config?: Record<string, any>
auth_configured?: Record<string, boolean>
headers: Record<string, string>
config: Record<string, any>
config_id: number | null
description: string
}
const AISSTREAM_BBOX_PRESETS = [
{
value: 'global',
label: '全球',
boxes: [[[-90, -180], [90, 180]]],
},
{
value: 'norway_north_sea',
label: '挪威 / 北海',
boxes: [[[50, -8], [72, 32]]],
},
{
value: 'europe_coast',
label: '欧洲近海',
boxes: [[[35, -12], [72, 32]]],
},
{
value: 'east_asia',
label: '东亚',
boxes: [[[18, 105], [46, 146]]],
},
{
value: 'north_america_coasts',
label: '北美东西海岸',
boxes: [
[[24, -126], [50, -66]],
[[18, -98], [31, -80]],
],
},
]
const stringifyBoundingBoxes = (boxes: unknown) => JSON.stringify(boxes, null, 2)
const matchAisstreamBboxPreset = (boxes: unknown) => {
const serialized = JSON.stringify(boxes)
return AISSTREAM_BBOX_PRESETS.find((preset) => JSON.stringify(preset.boxes) === serialized)?.value || 'custom'
}
const formatLagSeconds = (value: number | null | undefined) => {
if (value == null) return '未知'
if (value < 60) return `${Math.round(value)}`
if (value < 3600) return `${Math.round(value / 60)} 分钟`
return `${Math.round(value / 3600)} 小时`
}
function PlugConnectIcon() {
return (
<svg width="1em" height="1em" viewBox="0 0 24 24" aria-hidden="true" focusable="false">
@@ -247,6 +306,7 @@ function SettingsPanel({
function Settings() {
const [searchParams, setSearchParams] = useSearchParams()
const requestedTab = searchParams.get('tab') || 'display'
const requestedCollector = searchParams.get('collector') || ''
const [loading, setLoading] = useState(true)
const [savingCollectorId, setSavingCollectorId] = useState<number | null>(null)
const [collectors, setCollectors] = useState<CollectorSettings[]>([])
@@ -285,6 +345,7 @@ function Settings() {
const selectedCollectorHealth = selectedCollector
? collectorHealthStatus[selectedCollector.source]
: undefined
const selectedAisRuntimeHealth = selectedCollector?.ais_health || null
const settingsTabKeys = new Set([
'display',
'notifications',
@@ -386,16 +447,31 @@ function Settings() {
useEffect(() => {
if (loading || !selectedCollectorConfig) return
const config = selectedCollectorConfig.config || {}
const boundingBoxes = config.bounding_boxes ?? [[[-90, -180], [90, 180]]]
collectorConfigForm.setFieldsValue({
endpoint: selectedCollectorConfig.endpoint,
auth_config: {
api_key: selectedCollectorConfig.auth_configured?.api_key ? '••••••••' : '',
},
headers: Object.entries(selectedCollectorConfig.headers || {}).map(([key, value]) => ({ key, value })),
config: {
timeout: selectedCollectorConfig.config?.timeout ?? 30,
retry: selectedCollectorConfig.config?.retry ?? 3,
timeout: config.timeout ?? 30,
retry: config.retry ?? 3,
max_messages: config.max_messages ?? 500,
receive_timeout_seconds: config.receive_timeout_seconds ?? 30,
message_types: config.message_types ?? ['PositionReport', 'ShipStaticData'],
bounding_box_preset: matchAisstreamBboxPreset(boundingBoxes),
bounding_boxes_json: stringifyBoundingBoxes(boundingBoxes),
},
})
}, [collectorConfigForm, loading, selectedCollectorConfig])
useEffect(() => {
if (!requestedCollector || !collectors.some((collector) => collector.source === requestedCollector)) return
setSelectedCollectorSource(requestedCollector)
}, [collectors, requestedCollector])
useEffect(() => {
const updateTableHeight = () => {
const regionHeight = collectorTableRegionRef.current?.offsetHeight || 0
@@ -440,6 +516,12 @@ function Settings() {
}, {})
)
const applyAisstreamBboxPreset = (presetValue: string) => {
const preset = AISSTREAM_BBOX_PRESETS.find((item) => item.value === presetValue)
if (!preset) return
collectorConfigForm.setFieldValue(['config', 'bounding_boxes_json'], stringifyBoundingBoxes(preset.boxes))
}
const saveCollector = async (collector: CollectorSettings) => {
try {
setSavingCollectorId(collector.id)
@@ -501,15 +583,39 @@ function Settings() {
return
}
const payload = {
const configValues = { ...(baseValues.config || {}) }
if (selectedCollector.source === 'aisstream_vessels') {
try {
configValues.bounding_boxes = JSON.parse(configValues.bounding_boxes_json || '[[[-90,-180],[90,180]]]')
} catch {
message.error('AISStream Bounding Boxes 必须是合法 JSON')
return
}
delete configValues.bounding_boxes_json
delete configValues.bounding_box_preset
}
const payload: Record<string, any> = {
name: selectedCollector.source,
description: `内置采集器覆盖配置:${selectedCollector.name}`,
source_type: 'http',
source_type: selectedCollector.source === 'aisstream_vessels' ? 'websocket' : 'http',
endpoint: baseValues.endpoint,
auth_type: 'none',
auth_config: {},
auth_type: selectedCollector.source === 'aisstream_vessels' ? 'api_key' : 'none',
headers,
config: baseValues.config || {},
config: configValues,
}
if (selectedCollector.source === 'aisstream_vessels') {
const apiKey = String(baseValues.auth_config?.api_key || '').trim()
if (apiKey && !apiKey.startsWith('••••')) {
payload.auth_config = {
api_key: apiKey,
in: 'payload',
}
} else if (!selectedCollectorConfig.config_id) {
payload.auth_config = {}
}
} else {
payload.auth_config = {}
}
if (selectedCollectorConfig.config_id) {
await axios.put(`/api/v1/datasources/configs/${selectedCollectorConfig.config_id}`, payload)
@@ -589,13 +695,16 @@ function Settings() {
try {
const values = collectorConfigForm.getFieldsValue(true)
setTestingCredentialProvider(selectedCollector.source)
const draftApiKey = String(values.auth_config?.api_key || '').trim()
const response = await axios.post('/api/v1/datasources/configs/builtin/connect', {
name: selectedCollector.source,
description: `内置采集器连接验证:${selectedCollector.name}`,
source_type: 'http',
source_type: selectedCollector.source === 'aisstream_vessels' ? 'websocket' : 'http',
endpoint: values.endpoint || selectedCollectorConfig?.endpoint || selectedCollectorConfig?.default_url || '',
auth_type: 'none',
auth_config: {},
auth_type: selectedCollector.source === 'aisstream_vessels' ? 'api_key' : 'none',
auth_config: selectedCollector.source === 'aisstream_vessels' && draftApiKey && !draftApiKey.startsWith('••••')
? { api_key: draftApiKey }
: {},
headers: headersListToMap(values.headers),
config: values.config || {},
})
@@ -611,6 +720,9 @@ function Settings() {
[selectedCollector.source]: { ok: false, message: response.data.message || '不可用' },
}))
message.error(response.data.message || '采集器健康检查失败')
if (selectedCollector.credential_provider) {
await loadCredentialGuide(selectedCollector.credential_provider, true)
}
}
} catch (error: unknown) {
const err = error as { response?: { data?: { detail?: string; message?: string } } }
@@ -620,6 +732,9 @@ function Settings() {
[selectedCollector.source]: { ok: false, message: errorMessage },
}))
message.error(errorMessage)
if (selectedCollector.credential_provider) {
await loadCredentialGuide(selectedCollector.credential_provider, true)
}
} finally {
setTestingCredentialProvider(null)
}
@@ -1383,6 +1498,13 @@ function Settings() {
) : (
<Tag></Tag>
)}
{selectedAisRuntimeHealth ? (
<Tooltip title={selectedAisRuntimeHealth.last_error || '采集器运行状态'}>
<Tag color={selectedAisRuntimeHealth.connection_state === 'connected' ? 'success' : 'default'}>
{selectedAisRuntimeHealth.connection_state}
</Tag>
</Tooltip>
) : null}
{selectedCollectorConfig?.is_overridden ? <Tag color="blue"> endpoint</Tag> : null}
</Space>
) : null}
@@ -1391,11 +1513,13 @@ function Settings() {
{selectedCollector?.requires_credentials && selectedCollector.source !== 'barentswatch_vessels' ? (
<Alert
showIcon
type="warning"
type={selectedCollector.source === 'aisstream_vessels' ? 'info' : 'warning'}
message="该采集器需要凭证"
description={selectedCollector.credential_status === 'supported'
? '该凭证类型已支持,但当前页面还没有专用表单。'
: '该凭证配置入口待接入。'}
description={selectedCollector.source === 'aisstream_vessels'
? '请在下方 AISStream 凭证中填写 API Key并保存采集器设置。'
: selectedCollector.credential_status === 'supported'
? '该凭证类型已支持,但当前页面还没有专用表单。'
: '该凭证配置入口待接入。'}
/>
) : null}
@@ -1435,6 +1559,87 @@ function Settings() {
</Card>
) : null}
{selectedCollector?.source === 'aisstream_vessels' ? (
<Card
size="small"
title={<Space><ApiOutlined />AISStream </Space>}
extra={(
<Space>
<Tooltip title="查看凭证获取教程">
<Button
size="small"
icon={<BookOutlined />}
onClick={() => { void loadCredentialGuide('aisstream', true) }}
/>
</Tooltip>
</Space>
)}
>
<Form form={collectorConfigForm} layout="vertical">
<Form.Item name={['auth_config', 'api_key']} label="API Key">
<Input.Password
autoComplete="new-password"
placeholder="输入 AISStream API Key"
/>
</Form.Item>
<Alert
showIcon
type="info"
message="AISStream 使用 WebSocket 实时流"
description="API Key 会保存在采集器覆盖配置中;保存后可用连接测试按钮验证凭证是否已配置。"
/>
</Form>
</Card>
) : null}
{selectedCollector?.source === 'aisstream_vessels' ? (
<Card size="small" title="AISStream 运行状态">
{selectedAisRuntimeHealth ? (
<div style={{ display: 'grid', gridTemplateColumns: 'repeat(2, minmax(160px, 1fr))', gap: 12 }}>
<div>
<Text type="secondary"></Text>
<div>
<Tag color={selectedAisRuntimeHealth.connection_state === 'connected' ? 'success' : 'default'}>
{selectedAisRuntimeHealth.connection_state}
</Tag>
</div>
</div>
<div>
<Text type="secondary"></Text>
<div>{selectedAisRuntimeHealth.message_rate ?? '未知'}</div>
</div>
<div>
<Text type="secondary"></Text>
<div>{formatDateTimeZhCN(selectedAisRuntimeHealth.last_seen_at)}</div>
</div>
<div>
<Text type="secondary"></Text>
<div>{formatDateTimeZhCN(selectedAisRuntimeHealth.last_success_at)}</div>
</div>
<div>
<Text type="secondary"></Text>
<div>{formatLagSeconds(selectedAisRuntimeHealth.lag_seconds)}</div>
</div>
<div>
<Text type="secondary"></Text>
<div>{formatDateTimeZhCN(selectedAisRuntimeHealth.updated_at)}</div>
</div>
</div>
) : (
<Alert showIcon type="warning" message="尚无运行状态" description="保存配置并触发一次 AISStream 采集后,这里会显示最近连接和消息统计。" />
)}
{selectedAisRuntimeHealth?.last_error ? (
<Alert
showIcon
type="error"
style={{ marginTop: 12 }}
message="最近错误"
description={selectedAisRuntimeHealth.last_error}
/>
) : null}
</Card>
) : null}
<Card size="small" title="基础配置">
<Form form={collectorConfigForm} layout="vertical">
<Form.Item name="endpoint" label="Endpoint" rules={[{ required: true, message: '请输入 Endpoint' }]}>
@@ -1471,6 +1676,36 @@ function Settings() {
<InputNumber min={0} max={10} style={{ width: '100%' }} />
</Form.Item>
</div>
{selectedCollector?.source === 'aisstream_vessels' ? (
<>
<div style={{ display: 'grid', gridTemplateColumns: '1fr 1fr', gap: '0 16px' }}>
<Form.Item name={['config', 'max_messages']} label="单次最大消息数">
<InputNumber min={1} max={10000} style={{ width: '100%' }} />
</Form.Item>
<Form.Item name={['config', 'receive_timeout_seconds']} label="接收超时(秒)">
<InputNumber min={1} max={300} style={{ width: '100%' }} />
</Form.Item>
</div>
<Form.Item name={['config', 'message_types']} label="消息类型">
<Select mode="tags" options={[
{ value: 'PositionReport', label: 'PositionReport' },
{ value: 'ShipStaticData', label: 'ShipStaticData' },
]} />
</Form.Item>
<Form.Item name={['config', 'bounding_box_preset']} label="采集范围">
<Select
options={[
...AISSTREAM_BBOX_PRESETS.map((preset) => ({ value: preset.value, label: preset.label })),
{ value: 'custom', label: '自定义 JSON' },
]}
onChange={applyAisstreamBboxPreset}
/>
</Form.Item>
<Form.Item name={['config', 'bounding_boxes_json']} label="Bounding Boxes JSON">
<Input.TextArea rows={4} />
</Form.Item>
</>
) : null}
</Form>
</Card>

View File

@@ -1,11 +1,12 @@
[project]
name = "planet"
version = "0.46.2"
version = "0.47.0"
description = "智能星球计划 - 态势感知系统"
requires-python = ">=3.14"
dependencies = [
"fastapi>=0.109.0",
"uvicorn[standard]>=0.27.0",
"websockets>=12.0",
"sqlalchemy[asyncio]>=2.0.25",
"asyncpg>=0.29.0",
"redis>=5.0.1",

4
uv.lock generated
View File

@@ -475,7 +475,7 @@ wheels = [
[[package]]
name = "planet"
version = "0.46.2"
version = "0.47.0"
source = { virtual = "." }
dependencies = [
{ name = "aiofiles" },
@@ -496,6 +496,7 @@ dependencies = [
{ name = "redis" },
{ name = "sqlalchemy", extra = ["asyncio"] },
{ name = "uvicorn", extra = ["standard"] },
{ name = "websockets" },
]
[package.dev-dependencies]
@@ -526,6 +527,7 @@ requires-dist = [
{ name = "redis", specifier = ">=5.0.1" },
{ name = "sqlalchemy", extras = ["asyncio"], specifier = ">=2.0.25" },
{ name = "uvicorn", extras = ["standard"], specifier = ">=0.27.0" },
{ name = "websockets", specifier = ">=12.0" },
]
[package.metadata.requires-dev]