Compare commits

...

8 Commits

Author SHA1 Message Date
linkong
bb9183b8a4 release: bump version to 0.48.0 2026-05-07 18:06:06 +08:00
linkong
421234301a release: bump version to 0.47.0 2026-04-30 16:56:37 +08:00
linkong
f22079d33a release: bump version to 0.46.3 2026-04-30 14:46:19 +08:00
linkong
9f737fdb89 release: bump version to 0.46.2 2026-04-30 14:30:12 +08:00
linkong
7418ce2fc1 release: bump version to 0.46.1 2026-04-30 09:41:08 +08:00
rayd1o
b1a5934b80 release: bump version to 0.46.0 2026-04-30 04:42:29 +08:00
rayd1o
ba54545ac7 release: bump version to 0.45.0 2026-04-29 23:43:54 +08:00
linkong
9dafbf4f6e release: bump version to 0.44.2 2026-04-29 18:11:37 +08:00
121 changed files with 11368 additions and 1966 deletions

View File

@@ -1,170 +1,93 @@
---
description: 分析本次 git 变更,在 docs/technical/zh/ 中新建或更新对应的技术文档
argument-hint: 可选:指定要记录的主题,或留空自动从 git diff 推断
description: Create or update repository documentation from current code changes
argument-hint: Optional: topic to document, or leave empty to infer from git diff
allowed-tools: ["Read", "Edit", "Write", "Bash", "Glob", "Grep"]
---
# /docs — 技术文档写入工作流
# /docs — Documentation Workflow
## 目标
## Goal
根据当前 git 变更(或用户指定主题)在 `docs/technical/zh/` 中写入或更新技术文档,记录**为什么**这样做,而不只是记录做了什么。
Create or update documentation that explains why a change exists, how it behaves, and what maintainers need to know. Keep this command generic. Repository-specific coverage rules live in the repository and must be loaded separately.
## 执行步骤
## Repository Rules
### Step 1 — 理解变更范围
Before deciding scope, check whether the repository has a documentation rules file:
```bash
git diff HEAD --stat # 变更文件一览
git diff HEAD --name-only # 变更文件列表
git log --oneline -10 # 近期 commit 上下文
test -f docs/documentation-coverage-rules.md && sed -n '1,240p' docs/documentation-coverage-rules.md
```
`$ARGUMENTS` 指定了主题,优先聚焦该主题;否则从文件列表和 diff stat 推断变更主题。不要默认读取完整仓库 diff只对决定文档主题所需的文件读取 focused diff
If it exists, apply it as the project-specific coverage checklist. If it does not exist, continue with the generic workflow below.
## Workflow
### Step 1 — Understand The Change
```bash
git diff HEAD --stat
git diff HEAD --name-only
git log --oneline -10
rg --files docs
```
If `$ARGUMENTS` specifies a topic, focus on that topic. Otherwise infer the documentation topic from the changed files. Do not read the full repository diff by default; inspect focused files only:
```bash
git diff HEAD -- <path>
rg -n "class |def |function |export |router|@router|interface |type " <path>
```
### Step 2 — 确认文档范围
### Step 2 — Decide Scope
分析变更,判断:
- Prefer updating an existing relevant document over creating a duplicate.
- Use one document for one coherent topic.
- Split documents only when the change crosses meaningful domains.
- Keep filenames lowercase and hyphenated.
- Apply the repository-specific rules file before writing.
1. **应写几篇文档**:单一主题写一篇,跨领域变更可拆分(如后端性能优化 + 运维启动脚本分开写)
2. **是新建还是更新**:检查 `docs/technical/zh/` 中是否已有相关文档
3. **文档命名**:按 `领域-主题-副题.md` 格式,全小写,用连字符,如:
- `backend-datasources-api-performance.md`
- `ops-planet-sh-startup.md`
- `earth-bgp-context.md`
For ambiguous or large documentation changes, briefly state the intended doc plan before editing. For clear small changes, proceed directly.
```bash
ls docs/technical/zh/ # 查看现有文档
### Step 3 — Write
Explain:
- Background/problem: what was wrong or missing before.
- Core design decisions and rationale.
- Operational or user-facing impact.
- Relevant code paths, only when useful for future maintainers.
Style:
- Follow the repositorys existing language and heading conventions.
- Use fenced code blocks with language tags.
- Prefer tables for comparisons or parameter lists.
- Keep snippets concise and relevant.
### Step 4 — Verify
- Read the completed docs once for clarity and stale statements.
- Verify referenced paths exist with `test -e` or `rg --files`.
- Run applicable checks from `docs/documentation-coverage-rules.md`.
- Check Markdown links use readable user-facing titles unless repository rules allow otherwise.
### Step 5 — Report
Summarize changed docs and verification:
```md
Updated:
- path/to/doc.md — what changed
Verified:
- checks that passed
- checks that could not be run, if any
```
**先输出写作计划供用户确认**(若变更明确且范围小,可直接执行):
## Hard Constraints
```
文档计划:
新建docs/technical/zh/ops-planet-sh-startup.md — planet.sh 启动性能优化
更新docs/technical/zh/backend-datasources-api-performance.md — 补充并行化细节
```
### Step 2.5 — 覆盖范围检查
写文档前必须按变更类型检查配套文档,不要只更新一篇专题文档:
- 用户可见流程变化:更新 `docs/technical/zh/manual.md`,通常也更新 `docs/technical/zh/quickstart.md`
- `manual.md``quickstart.md` 这类用户手册存在英文版时,同步更新 `docs/technical/en/...`,至少避免英文版与中文版互相矛盾。
- 控制台页面职责、路由入口、表格/抽屉/设置页行为变化:更新 `docs/technical/zh/frontend-admin-frontend-context.md`
- Earth 前端行为、HUD、巡航、图层、图例、交互变化更新 `docs/technical/zh/earth-frontend-context.md`
- 新增 Earth 图层、调整 `renderOrder`、半径/高度偏移、深度策略、拾取策略、legend mode、图层面板顺序或启动加载顺序更新 `docs/technical/zh/earth-render-layer-order.md`
- Earth 图层视觉样式、颜色、图例符号语义变化:若影响样式索引,同步更新 `docs/technical/zh/earth-layer-style-reference.md`
- 采集器、数据源、凭证、设置页、连接检查、scheduler、后端 API 变化:更新相关后端文档,优先检查 `docs/technical/zh/backend-collectors.md` 和 datasource/settings 专题文档。
- 如果某个旧 plan 的假设已经被当前实现推翻,在对应 `docs/plans/*.md` 增加现状修正或更新该段,不要让计划文档继续给出相反方向。
- 新增 technical 文档后,如果需要被发现,更新 `docs/technical/zh/README.md`
- 对本次变更提取旧词做 stale search例如旧 tab 名、旧路由职责、旧认证假设、改名前 UI 文案:
```bash
rg -n "旧文案|旧路由职责|旧认证假设" docs/technical docs/plans
```
### Step 3 — 写文档
遵循以下原则:
**记录 WHY不只记录 WHAT**
- 好:`将戳文件从 /tmp 移到 ~/.cache/planet/,因为 WSL 重启后 /tmp 被清空`
- 差:`修改了 AI_PROVIDER_BUILD_STAMP_FILE 的值`
**必须包含的内容**
- 背景/问题:改动之前存在什么问题,为什么要改
- 核心设计决策及其理由
- 关键代码片段(用 diff 或 before/after 展示)
- 相关文件列表
**格式要求**
- 使用 `##``###` 分级,不要超过三级
- 代码块注明语言python / bash / typescript / sql
- 表格用于对比多个选项或列出参数
- 中文写作,技术术语保留英文原文
- `docs/technical/zh/` 中的文档不得用英文原文占位;如果存在 `docs/technical/en/` 对应文件,禁止逐字复制成中文文件
- 中文文档内部链接应指向 `docs/technical/zh/...`,除非明确引用英文专属文档
**文档结构模板**
```markdown
# 标题(说明做了什么)
## 背景
为什么要做这个改动,改动前存在什么问题。
## 核心变更
### 子主题一
before/after 或决策说明 + 关键代码
### 子主题二
...
## 相关文件
- `path/to/file.py` — 简短说明
```
### Step 4 — 验证
- 读一遍写好的文档,确认逻辑清晰、代码片段无明显错误
-`rg --files``test -e` 确认文档中的文件路径在项目中真实存在,避免凭记忆判断:
- 检查中文文档没有误复制英文版:
```bash
python - <<'PY'
from pathlib import Path
same = []
for en in sorted(Path("docs/technical/en").glob("*.md")):
zh = Path("docs/technical/zh") / en.name
if zh.exists() and en.read_text() == zh.read_text():
same.append(en.name)
if same:
raise SystemExit("identical en/zh docs: " + ", ".join(same))
print("no identical en/zh docs")
PY
```
- 检查中文文档内部链接没有继续指向无语言目录:
```bash
rg -n "/home/ray/dev/linkong/planet/docs/technical/(?!zh|en)" docs/technical/zh --pcre2
```
```bash
# 对文档中提到的关键路径做快速验证
ls <mentioned_paths>
```
如需检查大量链接,优先用确定性提取:
```bash
rg -n "\]\(([^)]+)\)" docs/technical/zh/<doc>.md
```
### Step 5 — 完成确认
输出摘要:
```
✓ 新建docs/technical/zh/ops-planet-sh-startup.md约 xxx 字)
✓ 更新docs/technical/zh/backend-datasources-api-performance.md
```
## 注意事项
- 不要写流水账式的"改了 A、改了 B、改了 C",要写改动背后的约束和权衡
- 不要在文档中引用 PR 号、issue 号、或当前对话——这些会随时间失效
- 代码片段保持简洁,只保留说明问题的关键部分,省略无关样板代码
- 如果某个变更已有文档记录,优先在原文档中追加,而不是新建
- 文档是给未来的开发者看的,假设读者熟悉项目但不了解这次改动的背景
- Do not leave placeholder docs.
- Do not duplicate bilingual files byte-for-byte.
- Do not reference PR numbers, issue numbers, or the current conversation unless explicitly requested.
- Do not write changelog-style lists without the reasoning and tradeoffs behind the change.
- Keep docs maintainable and concise.

View File

@@ -1,126 +1,72 @@
---
name: docs
description: Analyze current Planet repo changes and create or update technical documentation under docs/technical/zh. Use when the user asks to write docs, update technical docs, summarize implementation changes into documentation, or port the Claude docs-codex workflow into Codex.
description: Create or update repository documentation from current code changes. Use when the user asks to write docs, update docs, summarize implementation changes into docs, or check documentation coverage. Load repository-specific coverage rules from docs/documentation-coverage-rules.md when present.
---
# Docs
Use this skill when the user asks to create or update Planet technical documentation, especially under `docs/technical/zh/`.
Use this skill when the task is documentation work: creating, updating, checking, or summarizing docs for code or behavior changes.
## Goal
Write or update technical docs that explain why a change exists, not only what files changed.
Write documentation that explains why a change exists, how it behaves, and what maintainers need to know. Keep the skill generic; repository-specific rules belong in the repository, not in this skill.
Default target directory:
## Repository Rules
- `docs/technical/zh/`
Before deciding scope, check whether the repository has a documentation rules file:
```bash
test -f docs/documentation-coverage-rules.md && sed -n '1,240p' docs/documentation-coverage-rules.md
```
If it exists, apply it as the project-specific coverage checklist. If it does not exist, continue with the generic workflow below.
## Workflow
1. Gather change context:
1. Gather focused context:
```bash
git diff HEAD --stat
git diff HEAD --name-only
git log --oneline -10
ls docs/technical/zh/
rg --files docs
```
If the user gives a specific topic, focus on that topic. Otherwise infer the documentation topic from the file list and diff stat. Do **not** read the full repository diff by default; inspect focused diffs only for the files that define the doc topic:
If the user gives a topic, focus on that topic. Otherwise infer the doc topic from changed files. Avoid reading large full diffs by default; inspect focused files and symbols:
```bash
git diff HEAD -- <path>
rg -n "class |def |function |export |router|@router|interface |type " <path>
```
2. Decide document scope:
2. Decide scope:
- Use one document for one coherent topic.
- Split documents when the changes cross meaningful domains, such as backend performance and ops startup behavior.
- Prefer updating an existing relevant doc over creating a duplicate.
- Name new files as lowercase hyphenated `domain-topic-detail.md`, for example:
- `backend-datasources-api-performance.md`
- `ops-planet-sh-startup.md`
- `earth-bgp-context.md`
- Use one document for one coherent topic.
- Split documents only when changes cross meaningful domains.
- Keep filenames lowercase and hyphenated.
3. Apply the documentation coverage checklist before writing:
3. Write the doc:
- User-visible workflow changes must update `docs/technical/zh/manual.md` and usually `docs/technical/zh/quickstart.md`.
- If an English counterpart exists for user-facing docs such as `manual.md` or `quickstart.md`, update `docs/technical/en/...` enough that it does not contradict the Chinese source.
- Control console page responsibility changes must update `docs/technical/zh/frontend-admin-frontend-context.md`.
- Earth frontend behavior changes must update `docs/technical/zh/earth-frontend-context.md`.
- Earth layer additions, `renderOrder`, altitude/radius offsets, depth strategy, pointer picking, legend modes, or layer panel/startup ordering must update `docs/technical/zh/earth-render-layer-order.md`.
- Earth layer visual style or legend symbol/color semantics should also update `docs/technical/zh/earth-layer-style-reference.md` when that reference is affected.
- Collector, datasource, credential, settings, connectivity, scheduler, or API changes must update the relevant backend docs, especially `docs/technical/zh/backend-collectors.md` and any datasource/settings-specific doc.
- When a change turns an old plan assumption into current behavior, update the relevant `docs/plans/*.md` with a status note instead of leaving contradictory instructions.
- If adding a new technical document, add it to `docs/technical/zh/README.md` when it should be discoverable from the technical docs index.
- Search docs for stale terms introduced by the change, for example old tab names, old route responsibilities, obsolete auth assumptions, or renamed UI labels.
- Explain background/problem, design decisions, constraints, and operational impact.
- Keep code snippets short and directly relevant.
- List related files only when they help future maintainers navigate.
- Use the repositorys existing language, heading style, and naming conventions.
4. Write the doc in Chinese:
4. Verify:
- Write Chinese prose for `docs/technical/zh/`.
- Keep technical identifiers, API paths, config keys, code symbols, and standard product names in English where appropriate.
- Use `##` and `###` headings; avoid going deeper than three levels.
- Use fenced code blocks with language tags.
- Use tables when comparing options or listing parameters.
5. Required content:
- Background/problem: what was wrong before and why the change was needed.
- Core design decisions and rationale.
- Key code snippets, preferably before/after or focused excerpts.
- Related files and what each file contributes.
6. Verification:
- Read the completed doc and check that the reasoning is clear.
- Verify important referenced paths exist.
- Use `rg --files` or `test -e` for path existence instead of relying on memory.
- Run a quick duplicate-language check when editing bilingual docs:
```bash
python - <<'PY'
from pathlib import Path
same = []
for en in sorted(Path("docs/technical/en").glob("*.md")):
zh = Path("docs/technical/zh") / en.name
if zh.exists() and en.read_text() == zh.read_text():
same.append(en.name)
if same:
raise SystemExit("identical en/zh docs: " + ", ".join(same))
print("no identical en/zh docs")
PY
```
Also check that Chinese docs do not link to the old language-less technical docs path:
```bash
rg -n "/home/ray/dev/linkong/planet/docs/technical/(?!zh|en)" docs/technical/zh --pcre2
```
This command should return no matches.
If checking many links, prefer deterministic extraction:
```bash
rg -n "\]\(([^)]+)\)" docs/technical/zh/<doc>.md
```
Also run focused stale-term searches derived from the change, for example:
```bash
rg -n "old label|old route purpose|obsolete provider assumption" docs/technical docs/plans
```
- Read the completed doc once for clarity and stale statements.
- Verify important referenced paths exist with `test -e` or `rg --files`.
- Run repository-specific doc checks from `docs/documentation-coverage-rules.md` when present.
- For Markdown links, check that user-facing titles are readable and not raw filenames unless the repository rules allow it.
## Hard Constraints
- A file under `docs/technical/zh/` must not be an English source file copied as a placeholder.
- Do not leave a Chinese doc with only an English title and English first-screen content.
- When an English counterpart exists in `docs/technical/en/`, never duplicate it byte-for-byte into `docs/technical/zh/`.
- Internal links inside `docs/technical/zh/` should point to `docs/technical/zh/...` for Chinese docs, unless intentionally linking to an English-only file.
- Do not reference PR numbers, issue numbers, or the current conversation.
- Do not write changelog-style lists like "changed A, changed B, changed C" without the constraints and tradeoffs behind those changes.
- Keep code snippets concise and relevant.
- Do not leave placeholder docs or copied source text pretending to be documentation.
- Do not duplicate bilingual files byte-for-byte.
- Do not reference PR numbers, issue numbers, or the current conversation unless explicitly requested.
- Do not write changelog-style lists without the reasoning, constraints, and tradeoffs behind the change.
- Keep docs concise enough to maintain.
## Recommended Output
@@ -128,9 +74,9 @@ After editing, summarize:
```md
Updated:
- docs/technical/zh/example.md — what changed
- path/to/doc.md — what changed
Verified:
- no identical en/zh docs
- no language-less docs/technical links in zh docs
- checks that passed
- checks that could not be run, if any
```

13
.dockerignore Normal file
View File

@@ -0,0 +1,13 @@
**
!pyproject.toml
!uv.lock
!aiprovider/
!aiprovider/**
aiprovider/.env
aiprovider/.env.*
!aiprovider/.env.example
**/__pycache__/
**/*.pyc
**/*.pyo

View File

@@ -26,6 +26,13 @@
- [ ] 重写控制台 UI逐步抛弃 Ant Design建立自有组件体系并统一采用 `tabler.io` / Tabler Icons 作为控制台主图标库
- [ ] 把 Earth 态势新闻源从 [earth_news.py](/home/ray/dev/linkong/planet/backend/app/services/earth_news.py) 的硬编码列表抽成可配置目录,优先保持当前“实时聚合”链路不变,只先解决新闻源不可配置的问题
- [ ] 为 Earth 态势新闻设计后续采集器化方案:明确新闻数据模型、去重策略、区域映射、过期清理和 Earth/AI 复用方式,再决定何时把新闻从实时抓取升级成正式 collector
- [ ] AIS v3.1:修复船只聚合完整性,`/geo/vessels` 合并 raw observation 聚合结果与 legacy `vessel_position + vessel_static` 最新结果,确保 BarentsWatch-only 船只不会因为 AISStream 子集存在而消失,并增加 raw/legacy/final unique MMSI 诊断统计
- [ ] AIS v3.2:把 AISStream 从收满 `max_messages` 后结束的批采集改成长连接 streaming service持续写入 raw observations通过内部 `/ws``vessels` channel 推送新船、位置和航向增量Earth 前端按 MMSI upsert marker
- [ ] AIS v3.3:修正 AISStream 采集页面状态语义,使用 connecting/streaming/reconnecting/stopped 与 indeterminate 状态展示运行时长、消息数、unique MMSI、message rate、最近消息和错误不再用一次性 REST 进度条表示长连接
- [ ] AIS v3.4修复船只身份字段和名称聚合MMSI/IMO/callsign 按字符串显示且不带千分位符;查询并列出所有仍以 MMSI 号码或 `MMSI <number>` 作为船名的记录标注来源、最近观测、message types 和缺失原因,并把这批 fallback-name 船只纳入名称聚合修复集合
- [ ] Earth Live Sync建立统一态势实时同步链路新增 `earth_summary` WS channel任意采集器成功后广播轻量 summary invalidation前端收到后重新拉 `/api/v1/visualization/geo/summary` 并更新 HUD同时为 BGP 增加 `bgp` WS channel使 BGP incidents/anomalies/collectors 在不刷新页面时也能 upsert 图层;卫星采集完成后触发 summary 刷新,必要时按 TLE 版本重新 hydrate 卫星数据
- [ ] AIS v4开放船只多源聚合策略配置支持 source priority、字段级规则、freshness 窗口和高级保护开关;保存时校验未知字段、非法模式和危险动态字段锁定,并在聚合接口返回命中的配置版本
- [ ] AIS v5实现船舶资料 enrichment 与冲突治理,按 `mmsi + imo + name + callsign` 异步补充船型细分、AIS 大类、旗国、尺寸、建造年份、运营方和图片缓存;详情面板展示缓存资料和字段来源,不在实时 AIS 请求链路现场抓第三方页面
- [ ] 为 Earth 地球表面增加一层与基础纹理对齐的材质/纹理 overlay并在同层叠加国界轮廓参考线要求国界线与底图稳定对齐且 hover 到国家轮廓时能高亮当前国家,便于校准地表和增强交互
- [ ] 把 Earth 新闻接入通用巡航队列:按新闻发生地和时间排序生成巡航目标,巡航聚焦到新闻事件时显示对应新闻卡片,并保持实现边界为“通用巡航层 + 新闻业务适配层”,不要再把新闻逻辑直接耦合回 `main.js` 状态机
- [ ] 为未知位置的算力中心建立分层坐标补全链路:优先 `精确坐标 > 站点/园区命中 > 城市 > 州/省 > 国家内主要算力城市 > 国家质心`,并把每次回退的 `confidence / reason / precision` 明确写进统一 GeoJSON

View File

@@ -1 +1 @@
0.44.1
0.48.0

View File

@@ -1,3 +1,5 @@
# syntax=docker/dockerfile:1.7
ARG PYTHON_IMAGE=python:3.14-slim
ARG UV_IMAGE=ghcr.io/astral-sh/uv:latest
@@ -18,9 +20,10 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
&& rm -rf /var/lib/apt/lists/*
COPY pyproject.toml uv.lock /app/
RUN uv sync --frozen --no-dev
RUN --mount=type=cache,target=/root/.cache/uv \
uv sync --frozen --no-dev
COPY . /app
COPY aiprovider /app/aiprovider
EXPOSE 8010

View File

@@ -12,6 +12,7 @@ from app.api.v1 import (
settings,
collected_data,
visualization,
vessel_aggregation,
bgp,
news,
system_control,
@@ -34,6 +35,11 @@ api_router.include_router(alerts.router, prefix="/alerts", tags=["alerts"])
api_router.include_router(settings.router, prefix="/settings", tags=["settings"])
api_router.include_router(system_control.router, prefix="/system", tags=["system"])
api_router.include_router(visualization.router, prefix="/visualization", tags=["visualization"])
api_router.include_router(
vessel_aggregation.router,
prefix="/vessel-aggregation",
tags=["vessel-aggregation"],
)
api_router.include_router(bgp.router, prefix="/bgp", tags=["bgp"])
api_router.include_router(tv.router, prefix="/tv", tags=["tv"])
api_router.include_router(news.router, prefix="/news", tags=["news"])

View File

@@ -5,8 +5,8 @@ from datetime import datetime
import base64
import json
import re
from fastapi import APIRouter, Depends, HTTPException, status
from sqlalchemy import select, func
from fastapi import APIRouter, Depends, HTTPException, Query
from sqlalchemy import delete, select, func
from sqlalchemy.ext.asyncio import AsyncSession
from pydantic import BaseModel, Field
import httpx
@@ -17,6 +17,8 @@ from app.db.session import get_db
from app.models.user import User
from app.models.datasource_config import DataSourceConfig
from app.models.datasource_mapping import DataSourceMappingTemplate
from app.models.collected_data import CollectedData
from app.models.vessel import AISRawObservation, AISSourceHealth
from app.core.security import get_current_user
from app.core.cache import cache
from app.core.time import to_iso8601_utc
@@ -26,10 +28,19 @@ from app.services.datasource_mapping import (
MappingError,
build_heuristic_mapping,
execute_mapping,
persist_mapped_records,
redact_for_llm,
stable_payload_hash,
)
from app.services.custom_datasource_runtime import (
CustomDatasourceRuntimeError,
fetch_rest_payload,
get_custom_stream_status,
run_mapped_rest_config,
run_mapped_websocket_config,
start_custom_stream,
stop_custom_stream,
test_websocket_config,
)
from app.services.datasource_connectivity import (
get_builtin_connection_status,
save_connectivity_success,
@@ -43,7 +54,7 @@ router = APIRouter()
class DataSourceConfigCreate(BaseModel):
name: str = Field(..., min_length=1, max_length=100)
description: Optional[str] = None
source_type: str = Field(..., description="http, api, database")
source_type: str = Field(..., description="rest, websocket, http, api, database")
endpoint: str = Field(..., max_length=500)
auth_type: str = Field(default="none", description="none, bearer, api_key, basic")
auth_config: dict = Field(default={})
@@ -219,6 +230,8 @@ def _build_query_params(auth_type: str, auth_config: dict, config: dict) -> dict
async def fetch_custom_sample_from_config(config: DataSourceConfig, limit_bytes: int) -> Any:
if str(config.source_type or "").lower() in {"websocket", "ws"}:
raise HTTPException(status_code=400, detail="WebSocket sources must use connection test or run-mapped stream.")
request_config = config.config or {}
method = str(request_config.get("method") or request_config.get("request_method") or "GET").upper()
if method not in {"GET", "POST"}:
@@ -318,7 +331,7 @@ async def list_configs(
"""List all user-defined data source configurations"""
query = select(DataSourceConfig)
if active_only:
query = query.where(DataSourceConfig.is_active == True)
query = query.where(DataSourceConfig.is_active)
query = query.order_by(DataSourceConfig.created_at.desc())
result = await db.execute(query)
@@ -374,6 +387,11 @@ async def list_all_datasources(
"is_active": db_config.is_active if db_config else True,
"source_type": db_config.source_type if db_config else "http",
"auth_type": db_config.auth_type if db_config else "none",
"auth_configured": {
"api_key": bool((db_config.auth_config or {}).get("api_key"))
if db_config
else False,
},
"headers": db_config.headers if db_config else {},
"config": strip_connectivity_validation(db_config.config if db_config else {}),
"config_id": db_config.id if db_config else None,
@@ -464,6 +482,8 @@ async def update_config(
for field, value in update_data.items():
if field == "config":
value = strip_connectivity_validation(value)
if field == "auth_config" and value == {} and (config.auth_config or {}):
continue
setattr(config, field, value)
await db.commit()
@@ -481,6 +501,8 @@ async def update_config(
@router.delete("/configs/{config_id}")
async def delete_config(
config_id: int,
delete_mappings: bool = Query(False),
delete_source_data: bool = Query(False),
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
@@ -491,12 +513,59 @@ async def delete_config(
if not config:
raise HTTPException(status_code=404, detail="Configuration not found")
deleted_mappings = 0
deleted_records = {
"collected_data": 0,
"ais_raw_observations": 0,
"ais_source_health": 0,
}
if delete_source_data:
collected_result = await db.execute(
delete(CollectedData).where(CollectedData.source == config.name)
)
raw_result = await db.execute(
delete(AISRawObservation).where(AISRawObservation.source == config.name)
)
health_result = await db.execute(
delete(AISSourceHealth).where(AISSourceHealth.source == config.name)
)
deleted_records = {
"collected_data": collected_result.rowcount or 0,
"ais_raw_observations": raw_result.rowcount or 0,
"ais_source_health": health_result.rowcount or 0,
}
if delete_mappings or delete_source_data:
mapping_result = await db.execute(
delete(DataSourceMappingTemplate).where(
DataSourceMappingTemplate.datasource_config_id == config_id
)
)
deleted_mappings = mapping_result.rowcount or 0
await db.delete(config)
await db.commit()
cache.delete_pattern("datasource_configs:*")
return {"message": "Configuration deleted successfully"}
if delete_source_data and (config.config or {}).get("target_schema") == "vessel_ais":
from app.core.websocket.broadcaster import broadcaster
await broadcaster.broadcast_custom(
"vessels",
{
"action": "reload",
"source": config.name,
"reason": "custom_source_deleted",
},
)
return {
"message": "Configuration deleted successfully",
"deleted_mappings": deleted_mappings,
"deleted_records": deleted_records,
}
@router.post("/configs/{config_id}/test")
@@ -513,6 +582,8 @@ async def test_config(
raise HTTPException(status_code=404, detail="Configuration not found")
try:
if str(config.source_type or "").lower() in {"websocket", "ws"}:
return await test_websocket_config(config)
result = await test_endpoint(
endpoint=config.endpoint,
auth_type=config.auth_type,
@@ -543,6 +614,18 @@ async def test_new_config(
):
"""Test a new data source configuration without saving"""
try:
if str(config_data.source_type or "").lower() in {"websocket", "ws"}:
config = DataSourceConfig(
name=config_data.name,
description=config_data.description,
source_type=config_data.source_type,
endpoint=config_data.endpoint,
auth_type=config_data.auth_type,
auth_config=config_data.auth_config,
headers=config_data.headers,
config=config_data.config,
)
return await test_websocket_config(config)
result = await test_endpoint(
endpoint=config_data.endpoint,
auth_type=config_data.auth_type,
@@ -601,6 +684,7 @@ async def connect_builtin_config(
config_data.headers,
config_data.config,
db,
config_data.auth_config,
)
if result.get("success") and result.get("checksum"):
validation = await save_connectivity_success(
@@ -867,6 +951,8 @@ async def update_datasource_mapping(
@router.post("/{config_id}/run-mapped")
async def run_mapped_datasource(
config_id: int,
background: bool = Query(False, description="For WebSocket sources, start a background stream task."),
debug_max_messages: int | None = Query(None, ge=1),
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
@@ -875,20 +961,24 @@ async def run_mapped_datasource(
if not datasource:
raise HTTPException(status_code=404, detail="Configuration not found")
result = await db.execute(
select(DataSourceMappingTemplate)
.where(DataSourceMappingTemplate.datasource_config_id == config_id)
.where(DataSourceMappingTemplate.is_active.is_(True))
.order_by(DataSourceMappingTemplate.version.desc())
.limit(1)
)
mapping = result.scalar_one_or_none()
if not mapping:
raise HTTPException(status_code=404, detail="No active mapping template found")
try:
sample = await fetch_custom_sample_from_config(datasource, 5_000_000)
mapped = execute_mapping(sample, mapping.mapping_json, mapping.target_schema)
if str(datasource.source_type or "").lower() in {"websocket", "ws"}:
if background and debug_max_messages is None:
started = start_custom_stream(config_id)
if not started:
raise HTTPException(status_code=409, detail="Custom WebSocket source is already running")
return {
"status": "started",
"datasource_config_id": config_id,
"stream": get_custom_stream_status(config_id),
}
return await run_mapped_websocket_config(
db,
datasource,
debug_max_messages=debug_max_messages,
)
return await run_mapped_rest_config(db, datasource)
except httpx.HTTPStatusError as exc:
raise HTTPException(
status_code=exc.response.status_code,
@@ -896,36 +986,26 @@ async def run_mapped_datasource(
) from exc
except httpx.HTTPError as exc:
raise HTTPException(status_code=502, detail=f"Datasource request failed: {exc}") from exc
except (MappingError, ValueError) as exc:
except (CustomDatasourceRuntimeError, MappingError, ValueError) as exc:
raise HTTPException(status_code=400, detail=f"Mapping failed: {exc}") from exc
if mapped["failed_count"] > 0:
return {
"status": "failed",
"datasource_config_id": config_id,
"mapping_id": mapping.id,
"mapping_version": mapping.version,
"target_schema": mapping.target_schema,
"mapped_count": mapped["mapped_count"],
"failed_count": mapped["failed_count"],
"errors": mapped["errors"][:20],
}
written_count = await persist_mapped_records(
db,
datasource_name=datasource.name,
datasource_config_id=datasource.id,
target_schema=mapping.target_schema,
records=mapped["records"],
mapping_version=mapping.version,
)
@router.post("/{config_id}/stop-mapped")
async def stop_mapped_datasource(
config_id: int,
current_user: User = Depends(get_current_user),
):
stopped = await stop_custom_stream(config_id)
return {
"status": "success",
"status": "stopped" if stopped else "not_running",
"datasource_config_id": config_id,
"mapping_id": mapping.id,
"mapping_version": mapping.version,
"target_schema": mapping.target_schema,
"fetched_count": mapped["total_items"],
"mapped_count": mapped["mapped_count"],
"written_count": written_count,
"stream": get_custom_stream_status(config_id),
}
@router.get("/{config_id}/stream-status")
async def get_mapped_stream_status(
config_id: int,
current_user: User = Depends(get_current_user),
):
return get_custom_stream_status(config_id)

View File

@@ -398,6 +398,11 @@ async def list_datasources(
"task_id": running_task.id if running_task else None,
"progress": running_task.progress if running_task else None,
"phase": running_task.phase if running_task else None,
"phase_progress": running_task.phase_progress if running_task else None,
"phase_message": running_task.phase_message if running_task else None,
"phase_current": running_task.phase_current if running_task else None,
"phase_total": running_task.phase_total if running_task else None,
"phase_unit": running_task.phase_unit if running_task else None,
"records_processed": running_task.records_processed if running_task else None,
"total_records": running_task.total_records if running_task else None,
}
@@ -626,6 +631,11 @@ async def trigger_datasource(
"message": "当前采集任务尚未完成,重新触发会丢失本次未完成进度。是否强制重新采集?",
"task_id": running_task.id,
"phase": running_task.phase,
"phase_progress": running_task.phase_progress,
"phase_message": running_task.phase_message,
"phase_current": running_task.phase_current,
"phase_total": running_task.phase_total,
"phase_unit": running_task.phase_unit,
"progress": running_task.progress,
"records_processed": running_task.records_processed,
"total_records": running_task.total_records,
@@ -709,13 +719,29 @@ async def get_task_status(
task = await get_running_task(db, datasource.id)
if not task:
return {"is_running": False, "task_id": None, "progress": None, "phase": None, "status": "idle"}
return {
"is_running": False,
"task_id": None,
"progress": None,
"phase": None,
"phase_progress": None,
"phase_message": None,
"phase_current": None,
"phase_total": None,
"phase_unit": None,
"status": "idle",
}
return {
"is_running": task.status == "running",
"task_id": task.id,
"progress": task.progress,
"phase": task.phase,
"phase_progress": task.phase_progress,
"phase_message": task.phase_message,
"phase_current": task.phase_current,
"phase_total": task.phase_total,
"phase_unit": task.phase_unit,
"records_processed": task.records_processed,
"total_records": task.total_records,
"status": task.status,

View File

@@ -17,6 +17,7 @@ from app.models.datasource import DataSource
from app.models.datasource_config import DataSourceConfig
from app.models.system_setting import SystemSetting
from app.models.user import User
from app.models.vessel import AISSourceHealth
from app.services.barentswatch import (
BarentsWatchConfig,
check_barentswatch_config,
@@ -368,7 +369,12 @@ def format_frequency_label(minutes: int) -> str:
return f"{minutes}m"
def serialize_collector(datasource: DataSource) -> dict:
async def get_ais_source_health_by_source(db: AsyncSession) -> dict[str, dict]:
result = await db.execute(select(AISSourceHealth))
return {item.source: item.to_dict() for item in result.scalars().all()}
def serialize_collector(datasource: DataSource, ais_health_by_source: dict[str, dict] | None = None) -> dict:
defaults = DEFAULT_DATASOURCES.get(datasource.source, {})
return {
"id": datasource.id,
@@ -387,6 +393,7 @@ def serialize_collector(datasource: DataSource) -> dict:
"requires_credentials": bool(defaults.get("requires_credentials", False)),
"credential_provider": defaults.get("credential_provider"),
"credential_status": defaults.get("credential_status", "none"),
"ais_health": (ais_health_by_source or {}).get(datasource.source),
}
@@ -599,7 +606,8 @@ async def get_collector_settings(
):
result = await db.execute(select(DataSource).order_by(DataSource.module, DataSource.id))
datasources = result.scalars().all()
return {"collectors": [serialize_collector(datasource) for datasource in datasources]}
ais_health_by_source = await get_ais_source_health_by_source(db)
return {"collectors": [serialize_collector(datasource, ais_health_by_source) for datasource in datasources]}
@router.put("/collectors/{datasource_id}")
@@ -619,7 +627,8 @@ async def update_collector_settings(
await db.commit()
await db.refresh(datasource)
await sync_datasource_job(datasource.id)
return {"status": "updated", "collector": serialize_collector(datasource)}
ais_health_by_source = await get_ais_source_health_by_source(db)
return {"status": "updated", "collector": serialize_collector(datasource, ais_health_by_source)}
@router.get("")
@@ -633,12 +642,13 @@ async def get_all_settings(
db,
["system", "notifications", "security"],
)
ais_health_by_source = await get_ais_source_health_by_source(db)
return {
"system": setting_payloads["system"],
"notifications": setting_payloads["notifications"],
"security": setting_payloads["security"],
"tv": await get_tv_settings_payload(db),
"integrations": await serialize_external_integrations(db),
"collectors": [serialize_collector(datasource) for datasource in datasources],
"collectors": [serialize_collector(datasource, ais_health_by_source) for datasource in datasources],
"generated_at": to_iso8601_utc(datetime.now(UTC)),
}

View File

@@ -27,7 +27,9 @@ async def list_tasks(
offset = (page - 1) * page_size
query = """
SELECT ct.id, ct.datasource_id, ds.name as datasource_name, ct.status,
ct.started_at, ct.completed_at, ct.records_processed, ct.error_message
ct.started_at, ct.completed_at, ct.records_processed, ct.error_message,
ct.phase, ct.phase_progress, ct.phase_message, ct.phase_current,
ct.phase_total, ct.phase_unit, ct.total_records, ct.progress
FROM collection_tasks ct
JOIN data_sources ds ON ct.datasource_id = ds.id
WHERE 1=1
@@ -66,6 +68,14 @@ async def list_tasks(
"completed_at": to_iso8601_utc(t[5]),
"records_processed": t[6],
"error_message": t[7],
"phase": t[8],
"phase_progress": t[9],
"phase_message": t[10],
"phase_current": t[11],
"phase_total": t[12],
"phase_unit": t[13],
"total_records": t[14],
"progress": t[15],
}
for t in tasks
],
@@ -81,7 +91,9 @@ async def get_task(
result = await db.execute(
text("""
SELECT ct.id, ct.datasource_id, ds.name as datasource_name, ct.status,
ct.started_at, ct.completed_at, ct.records_processed, ct.error_message
ct.started_at, ct.completed_at, ct.records_processed, ct.error_message,
ct.phase, ct.phase_progress, ct.phase_message, ct.phase_current,
ct.phase_total, ct.phase_unit, ct.total_records, ct.progress
FROM collection_tasks ct
JOIN data_sources ds ON ct.datasource_id = ds.id
WHERE ct.id = :id
@@ -105,6 +117,14 @@ async def get_task(
"completed_at": to_iso8601_utc(task[5]),
"records_processed": task[6],
"error_message": task[7],
"phase": task[8],
"phase_progress": task[9],
"phase_message": task[10],
"phase_current": task[11],
"phase_total": task[12],
"phase_unit": task[13],
"total_records": task[14],
"progress": task[15],
}

View File

@@ -0,0 +1,132 @@
"""v4 strategy + v5 conflict-promotion + enrichment APIs for vessel_ais."""
from __future__ import annotations
from typing import Any
from fastapi import APIRouter, Depends, HTTPException
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.security import get_current_user
from app.db.session import get_db
from app.models.user import User
from app.models.vessel import AISConflictRecord
from app.services.vessel_aggregation_strategy import (
StrategyValidationError,
load_strategy,
reset_strategy,
save_strategy,
)
from app.services.vessel_enrichment import (
get_vessel_enrichment_bundle,
upsert_vessel_media_enrichment,
upsert_vessel_profile_enrichment,
)
router = APIRouter()
@router.get("/strategy")
async def get_aggregation_strategy(db: AsyncSession = Depends(get_db)):
return await load_strategy(db)
@router.put("/strategy")
async def put_aggregation_strategy(
payload: dict[str, Any],
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
try:
return await save_strategy(db, payload)
except StrategyValidationError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
@router.delete("/strategy")
async def reset_aggregation_strategy(
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
return await reset_strategy(db)
@router.post("/conflicts/{mmsi}/{field}/promote-to-rule")
async def promote_conflict_to_rule(
mmsi: int,
field: str,
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
"""Lift the current conflict resolution into a persistent strategy rule."""
result = await db.execute(
select(AISConflictRecord)
.where(AISConflictRecord.target_schema == "vessel_ais")
.where(AISConflictRecord.entity_key == str(mmsi))
.where(AISConflictRecord.field == field)
.order_by(AISConflictRecord.updated_at.desc(), AISConflictRecord.id.desc())
.limit(1)
)
record = result.scalar_one_or_none()
if record is None or not record.selected_source:
raise HTTPException(status_code=404, detail="Conflict record with selected_source not found")
strategy = await load_strategy(db)
vessel_ais = dict(strategy.get("vessel_ais") or {})
field_rules = dict(vessel_ais.get("field_rules") or {})
field_rules[field] = {"mode": "source_priority", "source_priority": [record.selected_source]}
vessel_ais["field_rules"] = field_rules
incoming = {"version": int(strategy.get("version") or 0), "vessel_ais": vessel_ais}
try:
return await save_strategy(db, incoming)
except StrategyValidationError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
@router.delete("/conflicts/{mmsi}/{field}/promote-to-rule")
async def revert_conflict_rule(
mmsi: int,
field: str,
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
strategy = await load_strategy(db)
vessel_ais = dict(strategy.get("vessel_ais") or {})
field_rules = dict(vessel_ais.get("field_rules") or {})
if field in field_rules:
del field_rules[field]
vessel_ais["field_rules"] = field_rules
incoming = {"version": int(strategy.get("version") or 0), "vessel_ais": vessel_ais}
try:
return await save_strategy(db, incoming)
except StrategyValidationError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
@router.get("/enrichment/{mmsi}")
async def get_vessel_enrichment(mmsi: int, db: AsyncSession = Depends(get_db)):
return await get_vessel_enrichment_bundle(db, mmsi)
@router.put("/enrichment/{mmsi}/profile")
async def put_vessel_profile_enrichment(
mmsi: int,
payload: dict[str, Any],
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
return await upsert_vessel_profile_enrichment(db, mmsi=mmsi, payload=payload)
@router.put("/enrichment/{mmsi}/media")
async def put_vessel_media_enrichment(
mmsi: int,
payload: dict[str, Any],
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
return await upsert_vessel_media_enrichment(db, mmsi=mmsi, payload=payload)

View File

@@ -6,6 +6,7 @@ Returns GeoJSON format compatible with Three.js, CesiumJS, and Unreal Cesium.
from datetime import UTC, datetime, timedelta
import math
import re
import httpx
from fastapi import APIRouter, HTTPException, Depends, Query, Response
from sqlalchemy.ext.asyncio import AsyncSession
@@ -19,12 +20,22 @@ from app.core.time import to_iso8601_utc
from app.db.session import get_db
from app.models.bgp_anomaly import BGPAnomaly
from app.models.bgp_incident import BGPIncident
from app.models.bgp_observation import BGPObservation
from app.models.collected_data import CollectedData
from app.models.vessel import VesselPosition, VesselStatic
from app.models.vessel import AISSourceHealth, VesselPosition, VesselStatic
from app.services.bgp_collectors import build_bgp_collector_coverage
from app.services.cable_graph import build_graph_from_data, CableGraph, haversine_distance
from app.services.collectors.bgp_common import RIPE_RIS_COLLECTOR_COORDS
from app.services.persistent_logs import record_system_log
from app.services.vessel_ais_aggregation import (
build_field_conflict_candidates,
count_unique_raw_vessel_mmsi,
get_aggregated_vessel,
get_aggregated_vessel_track,
get_aggregated_vessels,
get_vessel_conflict_records,
get_vessel_raw_observations,
)
from app.core.logging import get_logger
router = APIRouter()
@@ -32,6 +43,7 @@ logger = get_logger(__name__, service="api")
TERRAIN_TILE_URL_TEMPLATE = (
"https://s3.amazonaws.com/elevation-tiles-prod/terrarium/{z}/{x}/{y}.png"
)
VESSEL_NAME_FALLBACK_PATTERN = re.compile(r"^mmsi\s*\d+$", re.IGNORECASE)
# ============== Converter Functions ==============
@@ -273,6 +285,120 @@ async def _load_current_collected_data(
return list(result.scalars().all())
async def _latest_task_id_for_source(
db: AsyncSession,
source: str,
*,
exclude_unknown_name: bool = False,
) -> int | None:
stmt = (
select(
CollectedData.task_id,
func.max(CollectedData.collected_at).label("latest_collected_at"),
func.max(CollectedData.id).label("latest_id"),
)
.where(CollectedData.source == source)
.where(CollectedData.task_id.isnot(None))
.group_by(CollectedData.task_id)
.order_by(func.max(CollectedData.collected_at).desc(), func.max(CollectedData.id).desc())
.limit(1)
)
if exclude_unknown_name:
stmt = stmt.where(CollectedData.name != "Unknown")
result = await db.execute(stmt)
row = result.first()
return int(row.task_id) if row and row.task_id is not None else None
async def _load_current_or_latest_task_data(
db: AsyncSession,
source: str,
*,
exclude_unknown_name: bool = False,
limit: Optional[int] = None,
) -> List[CollectedData]:
records = await _load_current_collected_data(
db,
source,
exclude_unknown_name=exclude_unknown_name,
limit=limit,
)
if records:
return records
latest_task_id = await _latest_task_id_for_source(
db,
source,
exclude_unknown_name=exclude_unknown_name,
)
if latest_task_id is None:
return []
stmt = (
select(CollectedData)
.where(CollectedData.source == source)
.where(CollectedData.task_id == latest_task_id)
.order_by(CollectedData.id.desc())
)
if exclude_unknown_name:
stmt = stmt.where(CollectedData.name != "Unknown")
if limit is not None:
stmt = stmt.limit(limit)
result = await db.execute(stmt)
return list(result.scalars().all())
async def _count_current_or_latest_task_data(
db: AsyncSession,
source: str,
*,
exclude_unknown_name: bool = False,
) -> int:
current_stmt = (
select(func.count(CollectedData.id))
.where(CollectedData.source == source)
.where(CollectedData.is_current.is_(True))
)
if exclude_unknown_name:
current_stmt = current_stmt.where(CollectedData.name != "Unknown")
current_result = await db.execute(current_stmt)
current_scalar = current_result.scalar()
if current_scalar is None and hasattr(current_result, "scalars"):
current_rows = current_result.scalars().all()
current_count = sum(
1
for row in current_rows
if getattr(row, "source", None) == source
and (not exclude_unknown_name or getattr(row, "name", None) != "Unknown")
)
else:
current_count = int(current_scalar or 0)
if current_count > 0:
return current_count
latest_task_id = await _latest_task_id_for_source(
db,
source,
exclude_unknown_name=exclude_unknown_name,
)
if latest_task_id is None:
return 0
latest_stmt = (
select(func.count(CollectedData.id))
.where(CollectedData.source == source)
.where(CollectedData.task_id == latest_task_id)
)
if exclude_unknown_name:
latest_stmt = latest_stmt.where(CollectedData.name != "Unknown")
latest_result = await db.execute(latest_stmt)
return int(latest_result.scalar() or 0)
async def _load_current_collected_data_by_sources(
db: AsyncSession,
sources: List[str],
@@ -628,14 +754,21 @@ VESSEL_TYPE_FILTERS = {
def convert_vessels_to_geojson(rows: List[Any]) -> Dict[str, Any]:
features = []
seen_mmsi: set[int] = set()
for position, static in rows:
if position.lat is None or position.lon is None:
continue
if position.mmsi in seen_mmsi:
continue
seen_mmsi.add(position.mmsi)
props = {
"mmsi": position.mmsi,
"mmsi_display": str(position.mmsi),
"name": getattr(static, "name", None) or f"MMSI {position.mmsi}",
"name_is_fallback": _is_vessel_name_fallback(getattr(static, "name", None), position.mmsi),
"callsign": getattr(static, "callsign", None),
"imo": getattr(static, "imo", None),
"imo_display": str(getattr(static, "imo")) if getattr(static, "imo", None) else None,
"vessel_type": getattr(static, "vessel_type", None),
"vessel_type_name": getattr(static, "vessel_type_name", None) or "Other",
"flag": getattr(static, "flag", None),
@@ -664,6 +797,58 @@ def convert_vessels_to_geojson(rows: List[Any]) -> Dict[str, Any]:
return {"type": "FeatureCollection", "features": features}
def convert_aggregated_vessels_to_geojson(vessels: List[dict[str, Any]]) -> Dict[str, Any]:
features = []
for vessel in vessels:
if vessel.get("lat") is None or vessel.get("lon") is None:
continue
source_summary = {}
for source, summary in (vessel.get("source_summary") or {}).items():
source_summary[source] = {
**summary,
"latest_observed_at": to_iso8601_utc(summary.get("latest_observed_at")),
}
props = {
"mmsi": vessel["mmsi"],
"mmsi_display": str(vessel["mmsi"]),
"name": vessel.get("name") or f"MMSI {vessel['mmsi']}",
"name_is_fallback": _is_vessel_name_fallback(vessel.get("name"), vessel["mmsi"]),
"callsign": vessel.get("callsign"),
"imo": vessel.get("imo"),
"imo_display": str(vessel.get("imo")) if vessel.get("imo") else None,
"vessel_type": vessel.get("vessel_type"),
"vessel_type_name": vessel.get("vessel_type_name") or "Other",
"flag": vessel.get("flag"),
"length": vessel.get("length"),
"width": vessel.get("width"),
"draught": vessel.get("draught"),
"sog": vessel.get("sog"),
"cog": vessel.get("cog"),
"heading": vessel.get("heading"),
"nav_status": vessel.get("nav_status"),
"received_at": to_iso8601_utc(vessel.get("received_at")),
"field_sources": vessel.get("field_sources") or {},
"selected_reasons": vessel.get("selected_reasons") or {},
"source_summary": source_summary,
"quality_flags": vessel.get("quality_flags") or [],
"conflict_count": vessel.get("conflict_count", 0),
"aggregation_strategy_version": vessel.get("aggregation_strategy_version", 0),
"data_type": "vessel",
}
features.append(
{
"type": "Feature",
"id": vessel["mmsi"],
"geometry": {
"type": "Point",
"coordinates": [vessel["lon"], vessel["lat"]],
},
"properties": props,
}
)
return {"type": "FeatureCollection", "features": features}
def _parse_bbox(value: Optional[str]) -> tuple[float, float, float, float] | None:
if not value:
return None
@@ -681,6 +866,24 @@ def _parse_bbox(value: Optional[str]) -> tuple[float, float, float, float] | Non
return lon_min, lat_min, lon_max, lat_max
def _is_vessel_name_fallback(name: Any, mmsi: Any) -> bool:
text = str(name or "").strip()
mmsi_text = str(mmsi or "").strip()
if not text:
return True
if mmsi_text and text == mmsi_text:
return True
return bool(VESSEL_NAME_FALLBACK_PATTERN.match(text))
def _requested_vessel_types(value: Optional[str]) -> set[str]:
return {
item.strip().lower()
for item in (value or "").split(",")
if item.strip()
}
def _matches_vessel_type(props: dict[str, Any], requested_types: set[str]) -> bool:
if not requested_types:
return True
@@ -691,6 +894,88 @@ def _matches_vessel_type(props: dict[str, Any], requested_types: set[str]) -> bo
return False
def _feature_mmsi_key(feature: dict[str, Any]) -> str | None:
props = feature.get("properties", {})
mmsi = props.get("mmsi") or feature.get("id")
if mmsi in (None, ""):
return None
return str(mmsi)
def _feature_in_bbox(feature: dict[str, Any], bbox: tuple[float, float, float, float] | None) -> bool:
if bbox is None:
return True
coordinates = feature.get("geometry", {}).get("coordinates") or []
if len(coordinates) < 2:
return False
try:
lon = float(coordinates[0])
lat = float(coordinates[1])
except (TypeError, ValueError):
return False
lon_min, lat_min, lon_max, lat_max = bbox
return lon_min <= lon <= lon_max and lat_min <= lat <= lat_max
def _filter_vessel_features(
features: list[dict[str, Any]],
*,
bbox: tuple[float, float, float, float] | None,
requested_types: set[str],
) -> list[dict[str, Any]]:
return [
feature
for feature in features
if _feature_in_bbox(feature, bbox)
and _matches_vessel_type(feature.get("properties", {}), requested_types)
]
def _merge_vessel_features(
raw_features: list[dict[str, Any]],
legacy_features: list[dict[str, Any]],
) -> tuple[list[dict[str, Any]], dict[str, Any]]:
"""Prefer aggregated raw observations as the canonical source of truth.
Legacy `vessel_position` rows only fill MMSIs that the unified pipeline does
not yet know about, so a vessel never appears twice when both BarentsWatch
and AISStream observe it. Once the legacy table drains, this branch becomes
a no-op.
"""
merged: list[dict[str, Any]] = []
seen: set[str] = set()
raw_keys: set[str] = set()
legacy_keys: set[str] = set()
for feature in raw_features:
key = _feature_mmsi_key(feature)
if key is None or key in seen:
continue
seen.add(key)
raw_keys.add(key)
merged.append(feature)
legacy_added = 0
for feature in legacy_features:
key = _feature_mmsi_key(feature)
if key is None:
continue
legacy_keys.add(key)
if key in seen:
continue
seen.add(key)
legacy_added += 1
merged.append(feature)
return merged, {
"raw_unique_mmsi": len(raw_keys),
"legacy_unique_mmsi": len(legacy_keys),
"legacy_backfilled_mmsi": legacy_added,
"final_unique_mmsi": len(seen),
}
def _build_vessel_stats(features: List[dict[str, Any]]) -> dict[str, Any]:
by_type: dict[str, int] = {}
underway = 0
@@ -1291,7 +1576,7 @@ async def get_satellites_geojson(
db: AsyncSession = Depends(get_db),
):
"""获取卫星 TLE GeoJSON 数据"""
records = await _load_current_collected_data(
records = await _load_current_or_latest_task_data(
db,
"celestrak_tle",
exclude_unknown_name=True,
@@ -1411,10 +1696,40 @@ async def get_vessels_geojson(
None,
description="Comma-separated vessel types: cargo,tanker,passenger,fishing,military,other",
),
limit: int = Query(5000, ge=1, le=50000),
limit: Optional[int] = Query(
None,
ge=0,
description="Maximum vessel features to return. Omit or pass 0 for no limit.",
),
db: AsyncSession = Depends(get_db),
):
"""Return latest vessel positions as GeoJSON points."""
parsed_bbox = _parse_bbox(bbox)
requested_types = _requested_vessel_types(type)
merged_features, diagnostics = await _load_merged_vessel_features(db)
features = _filter_vessel_features(
merged_features,
bbox=parsed_bbox,
requested_types=requested_types,
)
if limit and limit > 0:
features = features[:limit]
return {
"type": "FeatureCollection",
"features": features,
"count": len(features),
"stats": _build_vessel_stats(features),
"diagnostics": {
**diagnostics,
"filtered_count": len(features),
},
}
async def _load_merged_vessel_features(db: AsyncSession) -> tuple[list[dict[str, Any]], dict[str, Any]]:
aggregated_vessels = await get_aggregated_vessels(db)
raw_geojson = convert_aggregated_vessels_to_geojson(aggregated_vessels)
latest_times = (
select(
VesselPosition.mmsi.label("mmsi"),
@@ -1432,44 +1747,125 @@ async def get_vessels_geojson(
)
.outerjoin(VesselStatic, VesselStatic.mmsi == VesselPosition.mmsi)
.order_by(VesselPosition.received_at.desc())
.limit(limit)
)
parsed_bbox = _parse_bbox(bbox)
if parsed_bbox is not None:
lon_min, lat_min, lon_max, lat_max = parsed_bbox
stmt = stmt.where(
VesselPosition.lon >= lon_min,
VesselPosition.lon <= lon_max,
VesselPosition.lat >= lat_min,
VesselPosition.lat <= lat_max,
)
result = await db.execute(stmt)
rows = list(result.all())
geojson = convert_vessels_to_geojson(rows)
requested_types = {
item.strip().lower()
for item in (type or "").split(",")
if item.strip()
legacy_geojson = convert_vessels_to_geojson(rows)
merged_features, diagnostics = _merge_vessel_features(
raw_geojson.get("features", []),
legacy_geojson.get("features", []),
)
return merged_features, {
**diagnostics,
"raw_feature_count": len(raw_geojson.get("features", [])),
"legacy_feature_count": len(legacy_geojson.get("features", [])),
}
if requested_types:
geojson["features"] = [
feature
for feature in geojson.get("features", [])
if _matches_vessel_type(feature.get("properties", {}), requested_types)
]
features = geojson.get("features", [])
@router.get("/vessels/custom-supplements")
async def get_vessel_custom_supplements(db: AsyncSession = Depends(get_db)):
"""Group custom vessel_ais sources by their declared merge target for diagnostics."""
from app.models.datasource_config import DataSourceConfig
result = await db.execute(
select(DataSourceConfig.name, DataSourceConfig.config, DataSourceConfig.is_active)
.where(DataSourceConfig.config["target_schema"].as_string() == "vessel_ais")
)
grouped: dict[str, dict[str, Any]] = {}
for name, config, is_active in result.all():
config = config or {}
merge_target = str(config.get("merge_target_source") or "barentswatch_vessels")
bucket = grouped.setdefault(merge_target, {"merge_target": merge_target, "sources": []})
bucket["sources"].append({"name": name, "is_active": bool(is_active)})
return {"groups": list(grouped.values())}
@router.get("/vessels/name-fallbacks")
async def get_vessel_name_fallbacks(
limit: int = Query(500, ge=0, description="Maximum fallback-name vessels to return. 0 means no limit."),
db: AsyncSession = Depends(get_db),
):
"""Return vessels whose display name still falls back to MMSI."""
aggregated_vessels = await get_aggregated_vessels(db)
raw_geojson = convert_aggregated_vessels_to_geojson(aggregated_vessels)
latest_times = (
select(
VesselPosition.mmsi.label("mmsi"),
func.max(VesselPosition.received_at).label("received_at"),
)
.group_by(VesselPosition.mmsi)
.subquery()
)
result = await db.execute(
select(VesselPosition, VesselStatic)
.join(
latest_times,
(VesselPosition.mmsi == latest_times.c.mmsi)
& (VesselPosition.received_at == latest_times.c.received_at),
)
.outerjoin(VesselStatic, VesselStatic.mmsi == VesselPosition.mmsi)
.order_by(VesselPosition.received_at.desc())
)
legacy_geojson = convert_vessels_to_geojson(list(result.all()))
features, diagnostics = _merge_vessel_features(
raw_geojson.get("features", []),
legacy_geojson.get("features", []),
)
fallback_items = []
for feature in features:
props = feature.get("properties", {})
mmsi = props.get("mmsi")
name = props.get("name")
if not _is_vessel_name_fallback(name, mmsi):
continue
source_summary = props.get("source_summary") or {}
fallback_items.append(
{
"mmsi": str(mmsi),
"display_name": name or f"MMSI {mmsi}",
"reason": "missing_real_name",
"received_at": props.get("received_at"),
"sources": sorted(source_summary.keys()),
"source_summary": source_summary,
"message_types": sorted(
{
message_type
for summary in source_summary.values()
for message_type in (summary.get("message_types") or [])
}
),
"field_sources": props.get("field_sources") or {},
}
)
if limit and limit > 0:
fallback_items = fallback_items[:limit]
return {
**geojson,
"count": len(features),
"stats": _build_vessel_stats(features),
"count": len(fallback_items),
"items": fallback_items,
"diagnostics": diagnostics,
}
@router.get("/vessels/{mmsi}")
async def get_vessel_detail(mmsi: int, db: AsyncSession = Depends(get_db)):
from app.services.vessel_enrichment import get_vessel_enrichment_bundle
aggregated = await get_aggregated_vessel(db, mmsi)
enrichment = await get_vessel_enrichment_bundle(db, mmsi)
if aggregated is not None:
return {
**aggregated,
"received_at": to_iso8601_utc(aggregated.get("received_at")),
"latitude": aggregated["lat"],
"longitude": aggregated["lon"],
"enrichment": enrichment,
}
latest_position_stmt = (
select(VesselPosition)
.where(VesselPosition.mmsi == mmsi)
@@ -1486,6 +1882,7 @@ async def get_vessel_detail(mmsi: int, db: AsyncSession = Depends(get_db)):
**(geojson["features"][0]["properties"]),
"latitude": position.lat,
"longitude": position.lon,
"enrichment": enrichment,
}
@@ -1496,6 +1893,30 @@ async def get_vessel_track(
db: AsyncSession = Depends(get_db),
):
cutoff = datetime.now(UTC) - timedelta(hours=hours)
aggregated_points = await get_aggregated_vessel_track(db, mmsi, cutoff=cutoff)
if aggregated_points:
return {
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"geometry": {
"type": "LineString",
"coordinates": [[point["lon"], point["lat"]] for point in aggregated_points],
},
"properties": {
"mmsi": mmsi,
"hours": hours,
"point_count": len(aggregated_points),
"start_at": to_iso8601_utc(aggregated_points[0]["observed_at"]),
"end_at": to_iso8601_utc(aggregated_points[-1]["observed_at"]),
"point_sources": [point["source"] for point in aggregated_points],
},
}
],
"count": 1,
}
result = await db.execute(
select(VesselPosition)
.where(VesselPosition.mmsi == mmsi)
@@ -1532,6 +1953,37 @@ async def get_vessel_track(
}
@router.get("/vessels/{mmsi}/observations")
async def get_vessel_observations(
mmsi: int,
limit: int = Query(100, ge=1, le=500),
db: AsyncSession = Depends(get_db),
):
"""Return raw AIS observations for debugging source-level collector facts."""
observations = await get_vessel_raw_observations(db, mmsi, limit=limit)
return {
"mmsi": mmsi,
"count": len(observations),
"observations": [item.to_dict() for item in observations],
"conflict_candidates": build_field_conflict_candidates(observations),
}
@router.get("/vessels/{mmsi}/conflicts")
async def get_vessel_conflicts(mmsi: int, db: AsyncSession = Depends(get_db)):
"""Return recorded AIS conflicts plus current raw-observation candidates."""
records = await get_vessel_conflict_records(db, mmsi)
observations = await get_vessel_raw_observations(db, mmsi, limit=500)
return {
"mmsi": mmsi,
"count": len(records),
"conflicts": [item.to_dict() for item in records],
"candidates": build_field_conflict_candidates(observations),
}
@router.get("/geo/bgp-anomalies")
async def get_bgp_anomalies_geojson(
severity: Optional[str] = Query(None),
@@ -1590,31 +2042,16 @@ async def get_bgp_collectors_geojson(db: AsyncSession = Depends(get_db)):
@router.get("/geo/summary")
async def get_visualization_geo_summary(db: AsyncSession = Depends(get_db)):
"""Return lightweight Earth HUD counts without loading layer GeoJSON payloads."""
records_by_source = await _load_current_collected_data_by_sources(
cable_count = await _count_current_or_latest_task_data(db, "arcgis_cables")
landing_point_count = await _count_current_or_latest_task_data(db, "arcgis_landing_points")
satellite_count = await _count_current_or_latest_task_data(
db,
[
"arcgis_cables",
"arcgis_landing_points",
"celestrak_tle",
"top500",
"epoch_ai_gpu",
],
"celestrak_tle",
exclude_unknown_name=True,
)
cables = convert_cable_to_geojson(records_by_source.get("arcgis_cables", []))
landing_points = convert_landing_point_to_geojson(
records_by_source.get("arcgis_landing_points", []),
)
satellites = convert_satellite_to_geojson(
_filter_known_records(records_by_source.get("celestrak_tle", [])),
)
compute_centers = convert_compute_centers_to_geojson(
_filter_known_records(
records_by_source.get("top500", [])
+ records_by_source.get("epoch_ai_gpu", []),
),
)
compute_features = compute_centers.get("features", [])
supercomputer_count = await _count_current_or_latest_task_data(db, "top500")
gpu_cluster_count = await _count_current_or_latest_task_data(db, "epoch_ai_gpu")
compute_center_count = supercomputer_count + gpu_cluster_count
active_incident_result = await db.execute(
select(func.count(BGPIncident.id)).where(BGPIncident.status == "active"),
@@ -1624,35 +2061,56 @@ async def get_visualization_geo_summary(db: AsyncSession = Depends(get_db)):
)
active_incident_count = int(active_incident_result.scalar() or 0)
active_anomaly_count = int(active_anomaly_result.scalar() or 0)
bgp_collectors = await build_bgp_collector_coverage(
bgp_collector_result = await db.execute(
select(func.count(func.distinct(BGPObservation.collector)))
.where(BGPObservation.collector.isnot(None))
.where(func.length(func.btrim(BGPObservation.collector)) > 0)
.where(BGPObservation.source.in_(("ris_live_bgp", "bgpstream_bgp")))
)
bgp_collector_scalar = bgp_collector_result.scalar()
if bgp_collector_scalar is None:
bgp_collectors = await build_bgp_collector_coverage(
db,
source_filter=("ris_live_bgp", "bgpstream_bgp"),
)
bgp_collector_count = len(
[item for item in bgp_collectors if item.get("collector")]
)
else:
bgp_collector_count = int(bgp_collector_scalar or 0)
raw_unique_window_hours = 24
raw_unique_mmsi = await count_unique_raw_vessel_mmsi(
db,
source_filter=("ris_live_bgp", "bgpstream_bgp"),
observed_since=datetime.now(UTC) - timedelta(hours=raw_unique_window_hours),
)
vessel_count_result = await db.execute(
select(func.count(func.distinct(VesselPosition.mmsi))),
legacy_unique_result = await db.execute(
select(func.count(func.distinct(VesselPosition.mmsi)))
)
vessel_count = int(vessel_count_result.scalar() or 0)
legacy_unique_mmsi = int(legacy_unique_result.scalar() or 0)
vessel_count = max(raw_unique_mmsi, legacy_unique_mmsi)
aisstream_health = await db.get(AISSourceHealth, "aisstream_vessels")
return {
"generated_at": to_iso8601_utc(datetime.now(UTC)),
"stats": {
"cable_count": len(cables.get("features", [])),
"landing_point_count": len(landing_points.get("features", [])),
"satellite_count": len(satellites.get("features", [])),
"compute_center_count": len(compute_features),
"cable_count": cable_count,
"landing_point_count": landing_point_count,
"satellite_count": satellite_count,
"compute_center_count": compute_center_count,
"vessel_count": vessel_count,
"supercomputer_count": sum(
1 for feature in compute_features
if feature.get("properties", {}).get("site_type") == "supercomputer"
),
"gpu_cluster_count": sum(
1 for feature in compute_features
if feature.get("properties", {}).get("site_type") == "gpu_cluster"
),
"vessel_raw_unique_mmsi": raw_unique_mmsi,
"vessel_raw_unique_window_hours": raw_unique_window_hours,
"vessel_legacy_unique_mmsi": legacy_unique_mmsi,
"aisstream_connection_state": aisstream_health.connection_state if aisstream_health else None,
"aisstream_last_seen_at": to_iso8601_utc(aisstream_health.last_seen_at) if aisstream_health else None,
"aisstream_message_rate": aisstream_health.message_rate if aisstream_health else None,
"aisstream_lag_seconds": aisstream_health.lag_seconds if aisstream_health else None,
"supercomputer_count": supercomputer_count,
"gpu_cluster_count": gpu_cluster_count,
"bgp_event_count": active_incident_count or active_anomaly_count,
"bgp_incident_count": active_incident_count,
"bgp_anomaly_count": active_anomaly_count,
"bgp_collector_count": len([item for item in bgp_collectors if item.get("collector")]),
"bgp_collector_count": bgp_collector_count,
},
}

View File

@@ -40,16 +40,16 @@ async def authenticate_token(token: str) -> Optional[dict]:
@router.websocket("/ws")
async def websocket_endpoint(
websocket: WebSocket,
token: str = Query(...),
token: str | None = Query(None),
):
"""WebSocket endpoint for real-time data"""
logger.info_event(
"WebSocket connection attempt",
event="auth.websocket.connection_attempt",
context={"token_preview": f"{token[:8]}..."},
context={"token_preview": f"{token[:8]}..." if token else "anonymous"},
)
payload = await authenticate_token(token)
if payload is None:
payload = await authenticate_token(token) if token else None
if token and payload is None:
logger.warning_event(
"WebSocket authentication failed, closing connection",
event="auth.websocket.connection_rejected",
@@ -57,7 +57,17 @@ async def websocket_endpoint(
await websocket.close(code=4001)
return
user_id = str(payload.get("sub"))
is_anonymous = payload is None
user_id = str(payload.get("sub")) if payload else f"anonymous:{id(websocket)}"
supported_channels = ["vessels"] if is_anonymous else [
"gpu_clusters",
"submarine_cables",
"ixp_nodes",
"alerts",
"dashboard",
"datasource_tasks",
"vessels",
]
await manager.connect(websocket, user_id)
try:
@@ -68,14 +78,7 @@ async def websocket_endpoint(
"connection_id": f"conn_{user_id}",
"server_version": settings.VERSION,
"heartbeat_interval": 30,
"supported_channels": [
"gpu_clusters",
"submarine_cables",
"ixp_nodes",
"alerts",
"dashboard",
"datasource_tasks",
],
"supported_channels": supported_channels,
},
}
)
@@ -93,12 +96,24 @@ async def websocket_endpoint(
)
elif data.get("type") == "subscribe":
channels = data.get("data", {}).get("channels", [])
if is_anonymous:
channels = [channel for channel in channels if channel in supported_channels]
manager.subscribe(websocket, channels)
await websocket.send_json(
{
"type": "subscription_confirmed",
"data": {"action": "subscribe", "channels": channels},
}
)
elif data.get("type") == "unsubscribe":
channels = data.get("data", {}).get("channels", [])
manager.unsubscribe(websocket, channels)
await websocket.send_json(
{
"type": "subscription_confirmed",
"data": {"action": "unsubscribe", "channels": channels},
}
)
elif data.get("type") == "control_frame":
await websocket.send_json(
{"type": "control_acknowledged", "data": {"received": True}}

View File

@@ -1,7 +1,6 @@
import os
import yaml
from functools import lru_cache
from typing import Optional
COLLECTOR_URL_KEYS = {
@@ -32,6 +31,7 @@ COLLECTOR_URL_KEYS = {
"nro_delegated_prefix_geo": "nro.delegated_stats_url",
"news_live_streams": "news_live_streams.channels_url",
"barentswatch_vessels": "barentswatch_vessels.url",
"aisstream_vessels": "aisstream_vessels.url",
}
@@ -74,7 +74,7 @@ class DataSourcesConfig:
from app.models.datasource_config import DataSourceConfig
query = select(DataSourceConfig).where(
DataSourceConfig.name == collector_name, DataSourceConfig.is_active == True
DataSourceConfig.name == collector_name, DataSourceConfig.is_active
)
result = await db.execute(query)
db_config = result.scalar_one_or_none()

View File

@@ -98,3 +98,7 @@ news_live_streams:
barentswatch_vessels:
# BarentsWatch Live AIS latest combined endpoint. Requires an AIS bearer token.
url: "https://live.ais.barentswatch.no/v1/latest/combined"
aisstream_vessels:
# AISStream realtime WebSocket endpoint. Requires an AISStream API key.
url: "wss://stream.aisstream.io/v0/stream"

View File

@@ -245,6 +245,18 @@ DEFAULT_DATASOURCES = {
"credential_provider": "barentswatch",
"credential_status": "supported",
},
"aisstream_vessels": {
"id": 28,
"name": "AISStream Vessels",
"display_name": "AISStream 实时船舶",
"module": "L4",
"priority": "P1",
"frequency_minutes": 1,
"is_free": True,
"requires_credentials": True,
"credential_provider": "aisstream",
"credential_status": "supported",
},
}
ID_TO_COLLECTOR = {info["id"]: name for name, info in DEFAULT_DATASOURCES.items()}

View File

@@ -16,8 +16,11 @@ class VesselAISRecord(BaseModel):
sog: float | None = None
cog: float | None = Field(default=None, ge=0, le=360)
heading: int | None = Field(default=None, ge=0, le=511)
nav_status: int | None = None
name: str | None = None
callsign: str | None = None
vessel_type: str | int | None = None
vessel_type_name: str | None = None
received_at: datetime | None = None
@@ -104,8 +107,11 @@ TARGET_SCHEMAS: dict[str, TargetSchema] = {
TargetField("sog", "float", False, "对地航速,单位节", 12.4),
TargetField("cog", "float", False, "对地航向0-360 度", 184.5),
TargetField("heading", "integer", False, "船首向0-511", 186),
TargetField("nav_status", "integer", False, "导航状态码", 0),
TargetField("name", "string", False, "船名", "OSLO EXPRESS"),
TargetField("vessel_type", "string", False, "船型", "cargo"),
TargetField("callsign", "string", False, "呼号", "LAAB"),
TargetField("vessel_type", "string", False, "船型代码", 70),
TargetField("vessel_type_name", "string", False, "船型名称", "Cargo"),
TargetField("received_at", "datetime", False, "数据接收时间", "2026-04-28T00:00:00Z"),
),
),

View File

@@ -75,7 +75,7 @@ class DataBroadcaster:
"timestamp": to_iso8601_utc(datetime.now(UTC)),
"payload": data,
},
channel=channel if channel in manager.active_connections else "all",
channel=channel,
)
async def broadcast_datasource_task_update(self, data: Dict[str, Any]):

View File

@@ -1,9 +1,6 @@
"""WebSocket Connection Manager"""
import json
import asyncio
from typing import Dict, Set, Optional
from datetime import datetime
from fastapi import WebSocket
import redis.asyncio as redis
@@ -15,6 +12,8 @@ class ConnectionManager:
def __init__(self):
self.active_connections: Dict[str, Set[WebSocket]] = {} # user_id -> connections
self.channel_subscriptions: Dict[str, Set[WebSocket]] = {}
self.websocket_channels: Dict[WebSocket, Set[str]] = {}
self.redis_client: Optional[redis.Redis] = None
async def connect(self, websocket: WebSocket, user_id: str):
@@ -40,6 +39,39 @@ class ConnectionManager:
self.active_connections[user_id].discard(websocket)
if not self.active_connections[user_id]:
del self.active_connections[user_id]
self.unsubscribe_all(websocket)
def subscribe(self, websocket: WebSocket, channels: list[str]):
normalized_channels = {
str(channel).strip()
for channel in channels
if str(channel).strip()
}
if not normalized_channels:
return
socket_channels = self.websocket_channels.setdefault(websocket, set())
for channel in normalized_channels:
self.channel_subscriptions.setdefault(channel, set()).add(websocket)
socket_channels.add(channel)
def unsubscribe(self, websocket: WebSocket, channels: list[str]):
for channel in {str(channel).strip() for channel in channels if str(channel).strip()}:
subscribers = self.channel_subscriptions.get(channel)
if subscribers is not None:
subscribers.discard(websocket)
if not subscribers:
del self.channel_subscriptions[channel]
socket_channels = self.websocket_channels.get(websocket)
if socket_channels is not None:
socket_channels.discard(channel)
if not socket_channels:
del self.websocket_channels[websocket]
def unsubscribe_all(self, websocket: WebSocket):
channels = list(self.websocket_channels.get(websocket, set()))
if channels:
self.unsubscribe(websocket, channels)
async def send_personal_message(self, message: dict, user_id: str):
if user_id in self.active_connections:
@@ -54,13 +86,19 @@ class ConnectionManager:
for user_id in self.active_connections:
await self.send_personal_message(message, user_id)
else:
await self.send_personal_message(message, channel)
for connection in list(self.channel_subscriptions.get(channel, set())):
try:
await connection.send_json(message)
except Exception:
self.unsubscribe_all(connection)
async def close_all(self):
for user_id in self.active_connections:
for connection in self.active_connections[user_id]:
await connection.close()
self.active_connections.clear()
self.channel_subscriptions.clear()
self.websocket_channels.clear()
manager = ConnectionManager()

View File

@@ -111,6 +111,7 @@ async def init_db():
import app.models.playground_message # noqa: F401
import app.models.system_log # noqa: F401
import app.models.vessel # noqa: F401
import app.models.vessel_enrichment # noqa: F401
import app.models.datasource_mapping # noqa: F401
logger.warning_event(
@@ -146,7 +147,12 @@ async def init_db():
text(
"""
ALTER TABLE collection_tasks
ADD COLUMN IF NOT EXISTS phase VARCHAR(30) DEFAULT 'queued'
ADD COLUMN IF NOT EXISTS phase VARCHAR(30) DEFAULT 'queued',
ADD COLUMN IF NOT EXISTS phase_progress DOUBLE PRECISION,
ADD COLUMN IF NOT EXISTS phase_message VARCHAR(255),
ADD COLUMN IF NOT EXISTS phase_current BIGINT,
ADD COLUMN IF NOT EXISTS phase_total BIGINT,
ADD COLUMN IF NOT EXISTS phase_unit VARCHAR(30)
"""
)
)
@@ -158,6 +164,30 @@ async def init_db():
"""
)
)
await conn.execute(
text(
"""
CREATE INDEX IF NOT EXISTS idx_collected_data_source_current_id
ON collected_data (source, is_current, id)
"""
)
)
await conn.execute(
text(
"""
CREATE INDEX IF NOT EXISTS idx_collected_data_source_task_id
ON collected_data (source, task_id, id)
"""
)
)
await conn.execute(
text(
"""
CREATE INDEX IF NOT EXISTS idx_ais_raw_schema_observed_entity
ON ais_raw_observations (target_schema, observed_at, entity_key)
"""
)
)
await conn.execute(
text(
"""

View File

@@ -12,7 +12,7 @@ from app.models.system_setting import SystemSetting
from app.models.playground_session import PlaygroundSession
from app.models.playground_message import PlaygroundMessage
from app.models.system_log import SystemLog, AuditLog
from app.models.vessel import VesselPosition, VesselStatic
from app.models.vessel import AISConflictRecord, AISRawObservation, AISSourceHealth, VesselPosition, VesselStatic
from app.models.datasource_mapping import DataSourceMappingTemplate
__all__ = [
@@ -31,7 +31,12 @@ __all__ = [
"BGPObservation",
"SystemLog",
"AuditLog",
"PlaygroundSession",
"PlaygroundMessage",
"VesselPosition",
"VesselStatic",
"AISRawObservation",
"AISConflictRecord",
"AISSourceHealth",
"DataSourceMappingTemplate",
]

View File

@@ -48,6 +48,8 @@ class CollectedData(Base):
# Indexes for common queries
__table_args__ = (
Index("idx_collected_data_source_collected", "source", "collected_at"),
Index("idx_collected_data_source_current_id", "source", "is_current", "id"),
Index("idx_collected_data_source_task_id", "source", "task_id", "id"),
Index("idx_collected_data_source_type", "source", "data_type"),
Index("idx_collected_data_source_source_id", "source", "source_id"),
)

View File

@@ -1,6 +1,6 @@
"""Collection Task model"""
from sqlalchemy import Column, DateTime, Integer, String, Text, Float
from sqlalchemy import BigInteger, Column, DateTime, Integer, String, Text, Float
from sqlalchemy.sql import func
from app.db.session import Base
@@ -13,6 +13,11 @@ class CollectionTask(Base):
datasource_id = Column(Integer, nullable=False, index=True)
status = Column(String(20), nullable=False) # pending, running, success, failed, cancelled
phase = Column(String(30), default="queued")
phase_progress = Column(Float)
phase_message = Column(String(255))
phase_current = Column(BigInteger)
phase_total = Column(BigInteger)
phase_unit = Column(String(30))
started_at = Column(DateTime(timezone=True))
completed_at = Column(DateTime(timezone=True))
records_processed = Column(Integer, default=0)

View File

@@ -1,6 +1,6 @@
"""Vessel AIS models for live maritime tracking."""
from sqlalchemy import BigInteger, Column, DateTime, Float, Index, Integer, SmallInteger, String
from sqlalchemy import BigInteger, Column, DateTime, Float, Index, Integer, JSON, SmallInteger, String
from sqlalchemy.sql import func
from app.core.time import to_iso8601_utc
@@ -73,3 +73,114 @@ class VesselPosition(Base):
"nav_status": self.nav_status,
"received_at": to_iso8601_utc(self.received_at),
}
class AISRawObservation(Base):
"""Source-level AIS fact before aggregation and conflict resolution."""
__tablename__ = "ais_raw_observations"
id = Column(Integer, primary_key=True, autoincrement=True)
target_schema = Column(String(64), nullable=False, default="vessel_ais", index=True)
source = Column(String(100), nullable=False, index=True)
entity_key = Column(String(64), nullable=False, index=True)
delivery_mode = Column(String(32), nullable=False, index=True)
transport = Column(String(32), nullable=False, index=True)
message_type = Column(String(64), nullable=True, index=True)
source_message_id = Column(String(128), nullable=True, index=True)
observation_hash = Column(String(64), nullable=False, unique=True, index=True)
observed_at = Column(DateTime(timezone=True), nullable=False, index=True)
collected_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now(), index=True)
normalized_payload = Column(JSON, default=dict)
raw_payload = Column(JSON, default=dict)
quality_flags = Column(JSON, default=list)
__table_args__ = (
Index("idx_ais_raw_entity_observed", "target_schema", "entity_key", "observed_at"),
Index("idx_ais_raw_schema_observed_entity", "target_schema", "observed_at", "entity_key"),
Index("idx_ais_raw_source_entity", "source", "entity_key"),
)
def to_dict(self) -> dict:
return {
"id": self.id,
"target_schema": self.target_schema,
"source": self.source,
"entity_key": self.entity_key,
"delivery_mode": self.delivery_mode,
"transport": self.transport,
"message_type": self.message_type,
"source_message_id": self.source_message_id,
"observation_hash": self.observation_hash,
"observed_at": to_iso8601_utc(self.observed_at),
"collected_at": to_iso8601_utc(self.collected_at),
"normalized_payload": self.normalized_payload or {},
"raw_payload": self.raw_payload or {},
"quality_flags": self.quality_flags or [],
}
class AISConflictRecord(Base):
"""Recorded field-level disagreement between AIS sources."""
__tablename__ = "ais_conflict_records"
id = Column(Integer, primary_key=True, autoincrement=True)
target_schema = Column(String(64), nullable=False, default="vessel_ais", index=True)
entity_key = Column(String(64), nullable=False, index=True)
field = Column(String(64), nullable=False, index=True)
candidates = Column(JSON, default=dict)
selected_source = Column(String(100), nullable=True, index=True)
selected_value = Column(JSON, nullable=True)
selected_reason = Column(String(64), nullable=True, index=True)
resolved_by = Column(String(32), nullable=False, default="system", index=True)
status = Column(String(32), nullable=False, default="open", index=True)
created_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now(), index=True)
updated_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now())
__table_args__ = (
Index("idx_ais_conflict_entity_field", "target_schema", "entity_key", "field"),
)
def to_dict(self) -> dict:
return {
"id": self.id,
"target_schema": self.target_schema,
"entity_key": self.entity_key,
"field": self.field,
"candidates": self.candidates or {},
"selected_source": self.selected_source,
"selected_value": self.selected_value,
"selected_reason": self.selected_reason,
"resolved_by": self.resolved_by,
"status": self.status,
"created_at": to_iso8601_utc(self.created_at),
"updated_at": to_iso8601_utc(self.updated_at),
}
class AISSourceHealth(Base):
"""Runtime health signal for an AIS collector source."""
__tablename__ = "ais_source_health"
source = Column(String(100), primary_key=True)
connection_state = Column(String(32), nullable=False, default="disconnected", index=True)
last_seen_at = Column(DateTime(timezone=True), nullable=True, index=True)
last_success_at = Column(DateTime(timezone=True), nullable=True, index=True)
last_error = Column(String(500), nullable=True)
message_rate = Column(Float, nullable=True)
lag_seconds = Column(Float, nullable=True)
updated_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now(), index=True)
def to_dict(self) -> dict:
return {
"source": self.source,
"connection_state": self.connection_state,
"last_seen_at": to_iso8601_utc(self.last_seen_at),
"last_success_at": to_iso8601_utc(self.last_success_at),
"last_error": self.last_error,
"message_rate": self.message_rate,
"lag_seconds": self.lag_seconds,
"updated_at": to_iso8601_utc(self.updated_at),
}

View File

@@ -0,0 +1,63 @@
"""Vessel enrichment cache tables (v5).
Profile and media enrichment are stored separately so cache TTLs can differ
and so the conflict-resolution + display layers can read either independently.
"""
from sqlalchemy import BigInteger, Column, DateTime, Float, JSON, String
from sqlalchemy.sql import func
from app.core.time import to_iso8601_utc
from app.db.session import Base
class VesselProfileEnrichment(Base):
"""Cached static vessel profile (type, flag, dimensions, operator, etc.)."""
__tablename__ = "vessel_profile_enrichment"
mmsi = Column(BigInteger, primary_key=True)
source = Column(String(100), nullable=False, default="system")
payload = Column(JSON, nullable=False, default=dict)
fetched_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now())
expires_at = Column(DateTime(timezone=True), nullable=True)
confidence = Column(Float, nullable=True)
reference_url = Column(String(500), nullable=True)
updated_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now(), onupdate=func.now())
def to_dict(self) -> dict:
return {
"mmsi": self.mmsi,
"source": self.source,
"payload": self.payload or {},
"fetched_at": to_iso8601_utc(self.fetched_at),
"expires_at": to_iso8601_utc(self.expires_at),
"confidence": self.confidence,
"reference_url": self.reference_url,
}
class VesselMediaEnrichment(Base):
"""Cached vessel imagery / external detail references."""
__tablename__ = "vessel_media_enrichment"
mmsi = Column(BigInteger, primary_key=True)
source = Column(String(100), nullable=False, default="system")
payload = Column(JSON, nullable=False, default=dict)
fetched_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now())
expires_at = Column(DateTime(timezone=True), nullable=True)
confidence = Column(Float, nullable=True)
reference_url = Column(String(500), nullable=True)
updated_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now(), onupdate=func.now())
def to_dict(self) -> dict:
return {
"mmsi": self.mmsi,
"source": self.source,
"payload": self.payload or {},
"fetched_at": to_iso8601_utc(self.fetched_at),
"expires_at": to_iso8601_utc(self.expires_at),
"confidence": self.confidence,
"reference_url": self.reference_url,
}

View File

@@ -36,6 +36,7 @@ from app.services.collectors.iptoasn import IPtoASNPrefixGeoCollector
from app.services.collectors.opengeofeed import OpenGeoFeedPrefixGeoCollector
from app.services.collectors.nro_delegated import NRODelegatedPrefixGeoCollector
from app.services.collectors.news_live_streams import NewsLiveStreamsCollector
from app.services.collectors.aisstream import AISStreamCollector
from app.services.collectors.vessel_ais import VesselAISCollector
collector_registry.register(TOP500Collector())
@@ -65,3 +66,40 @@ collector_registry.register(OpenGeoFeedPrefixGeoCollector())
collector_registry.register(NRODelegatedPrefixGeoCollector())
collector_registry.register(NewsLiveStreamsCollector())
collector_registry.register(VesselAISCollector())
collector_registry.register(AISStreamCollector())
__all__ = [
"BaseCollector",
"HTTPCollector",
"IntervalCollector",
"collector_registry",
"CollectorRegistry",
"TOP500Collector",
"EpochAIGPUCollector",
"HuggingFaceModelCollector",
"HuggingFaceDatasetCollector",
"HuggingFaceSpacesCollector",
"PeeringDBIXPCollector",
"PeeringDBNetworkCollector",
"PeeringDBFacilityCollector",
"TeleGeographyCableCollector",
"TeleGeographyLandingPointCollector",
"TeleGeographyCableSystemCollector",
"CloudflareRadarDeviceCollector",
"CloudflareRadarTrafficCollector",
"CloudflareRadarTopASCollector",
"ArcGISCableCollector",
"FAOLandingPointCollector",
"ArcGISLandingPointCollector",
"ArcGISCableLandingRelationCollector",
"SpaceTrackTLECollector",
"CelesTrakTLECollector",
"RISLiveCollector",
"BGPStreamBackfillCollector",
"IPtoASNPrefixGeoCollector",
"OpenGeoFeedPrefixGeoCollector",
"NRODelegatedPrefixGeoCollector",
"NewsLiveStreamsCollector",
"VesselAISCollector",
"AISStreamCollector",
]

View File

@@ -0,0 +1,491 @@
"""AISStream WebSocket collector for realtime vessel AIS observations."""
from datetime import UTC, datetime
import asyncio
import json
import os
from typing import Any
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.data_sources import get_data_sources_config
from app.core.time import to_iso8601_utc
from app.core.websocket.broadcaster import broadcaster
from app.models.datasource_config import DataSourceConfig
from app.models.task import CollectionTask
from app.services.collectors.base import BaseCollector
from app.services.vessel_ais_aggregation import (
AISSTREAM_DELIVERY_MODE,
AISSTREAM_TRANSPORT,
record_vessel_ais_observation,
update_ais_source_health,
)
from app.services.vessel_types import normalize_vessel_type_name
DEFAULT_AISSTREAM_URL = "wss://stream.aisstream.io/v0/stream"
DEFAULT_BOUNDING_BOXES = [[[-90, -180], [90, 180]]]
DEFAULT_MESSAGE_TYPES = ["PositionReport", "ShipStaticData"]
class AISStreamCollector(BaseCollector):
"""Collect AISStream WebSocket messages into the raw AIS observation layer."""
name = "aisstream_vessels"
priority = "P1"
module = "L4"
frequency_hours = 1
data_type = "vessel_ais"
fail_on_empty = False
async def _load_datasource_config(self) -> DataSourceConfig | None:
if self._db_session is None:
return None
result = await self._db_session.execute(
select(DataSourceConfig)
.where(DataSourceConfig.name == self.name)
.where(DataSourceConfig.is_active.is_(True))
)
return result.scalar_one_or_none()
async def _get_effective_config(self) -> dict[str, Any]:
datasource_config = await self._load_datasource_config()
config = dict(datasource_config.config or {}) if datasource_config else {}
auth_config = dict(datasource_config.auth_config or {}) if datasource_config else {}
endpoint = (
(datasource_config.endpoint if datasource_config else None)
or self._resolved_url
or get_data_sources_config().get_yaml_url(self.name)
or DEFAULT_AISSTREAM_URL
)
api_key = (
auth_config.get("api_key")
or config.get("api_key")
or os.getenv("AISSTREAM_API_KEY")
)
return {
"endpoint": endpoint,
"api_key": api_key,
"bounding_boxes": config.get("bounding_boxes") or DEFAULT_BOUNDING_BOXES,
"message_types": config.get("message_types") or DEFAULT_MESSAGE_TYPES,
"max_messages": int(config.get("max_messages") or 500),
"streaming_enabled": config.get("streaming_enabled", True) is not False,
"streaming_commit_interval": int(config.get("streaming_commit_interval") or 1),
"streaming_max_messages": int(config.get("streaming_max_messages") or 0),
"reconnect_delay_seconds": float(config.get("reconnect_delay_seconds") or 5),
"receive_timeout_seconds": float(config.get("receive_timeout_seconds") or 30),
}
def _build_subscription(self, config: dict[str, Any]) -> dict[str, Any]:
return {
"APIKey": config["api_key"],
"BoundingBoxes": config["bounding_boxes"],
"FilterMessageTypes": config["message_types"],
}
async def fetch(self) -> list[dict[str, Any]]:
config = await self._get_effective_config()
if not config["api_key"]:
raise RuntimeError("AISStream API key is not configured")
try:
import websockets
except ImportError as exc:
raise RuntimeError("Python package 'websockets' is required for AISStream") from exc
subscription = self._build_subscription(config)
messages: list[dict[str, Any]] = []
try:
async with websockets.connect(config["endpoint"]) as websocket:
await websocket.send(json.dumps(subscription))
while len(messages) < config["max_messages"]:
try:
raw_message = await asyncio.wait_for(
websocket.recv(),
timeout=config["receive_timeout_seconds"],
)
except TimeoutError:
break
payload = json.loads(raw_message)
if isinstance(payload, dict):
messages.append(payload)
except Exception as exc:
if self._db_session is not None:
await update_ais_source_health(
self._db_session,
source=self.name,
connection_state="disconnected",
last_error=f"{exc.__class__.__name__}: {exc}",
)
await self._db_session.commit()
raise
return messages
async def run(self, db: AsyncSession) -> dict[str, Any]:
"""Run AISStream as a long-lived streaming collector by default."""
config = await self._get_effective_config()
if not config.get("streaming_enabled", True):
return await super().run(db)
if not config["api_key"]:
return {"status": "failed", "error": "AISStream API key is not configured"}
from app.services.collectors.registry import collector_registry
if not collector_registry.is_active(self.name):
return {"status": "skipped", "reason": "Collector is disabled"}
try:
import websockets
except ImportError as exc:
return {"status": "failed", "error": "Python package 'websockets' is required for AISStream"}
start_time = datetime.now(UTC)
task = CollectionTask(
datasource_id=getattr(self, "_datasource_id", 1),
status="running",
phase="connecting",
phase_message="正在连接 AISStream 实时流",
phase_unit="messages",
started_at=start_time,
)
db.add(task)
await db.commit()
self._current_task = task
self._db_session = db
self._last_broadcast_progress = None
await self.resolve_url(db)
await self._publish_task_update(force=True)
records_added = 0
messages_seen = 0
unique_mmsi: set[str] = set()
reconnect_delay = config["reconnect_delay_seconds"]
try:
while True:
config = await self._get_effective_config()
subscription = self._build_subscription(config)
try:
await update_ais_source_health(
db,
source=self.name,
connection_state="connecting",
)
await self.set_phase("connecting", message="正在连接 AISStream 实时流")
await db.commit()
async with websockets.connect(config["endpoint"]) as websocket:
await websocket.send(json.dumps(subscription))
await update_ais_source_health(
db,
source=self.name,
connection_state="connected",
last_success_at=datetime.now(UTC),
)
await self.set_phase(
"streaming",
message="正在接收 AISStream 实时消息",
reset_progress=False,
)
await db.commit()
while True:
try:
raw_message = await asyncio.wait_for(
websocket.recv(),
timeout=config["receive_timeout_seconds"],
)
except TimeoutError:
await update_ais_source_health(
db,
source=self.name,
connection_state="connected",
last_success_at=datetime.now(UTC),
)
await db.commit()
continue
payload = json.loads(raw_message)
if not isinstance(payload, dict):
continue
messages_seen += 1
record = self._normalize_message(payload)
if not record:
continue
unique_mmsi.add(str(record["mmsi"]))
created = await self._save_stream_record(db, record)
if created:
records_added += 1
task.records_processed = messages_seen
task.total_records = None
task.progress = None
task.phase = "streaming"
task.phase_message = "正在接收 AISStream 实时消息"
task.phase_current = messages_seen
task.phase_total = None
task.phase_unit = "messages"
await self._publish_task_update(force=True)
if config["streaming_max_messages"] and messages_seen >= config["streaming_max_messages"]:
task.status = "success"
task.phase = "stopped"
task.phase_message = "AISStream 测试流已停止"
task.completed_at = datetime.now(UTC)
await db.commit()
await self._publish_task_update(force=True)
return {
"status": "success",
"task_id": task.id,
"records_processed": records_added,
"messages_seen": messages_seen,
"unique_mmsi": len(unique_mmsi),
"execution_time_seconds": (datetime.now(UTC) - start_time).total_seconds(),
}
except asyncio.CancelledError:
raise
except Exception as exc:
await update_ais_source_health(
db,
source=self.name,
connection_state="reconnecting",
last_error=f"{exc.__class__.__name__}: {exc}",
)
task.phase = "reconnecting"
task.phase_message = "AISStream 连接中断,正在重连"
task.error_message = f"{exc.__class__.__name__}: {exc}"
await db.commit()
await self._publish_task_update(force=True)
await asyncio.sleep(reconnect_delay)
except asyncio.CancelledError:
task.status = "cancelled"
task.phase = "stopped"
task.phase_message = "AISStream 实时流已停止"
task.completed_at = datetime.now(UTC)
await update_ais_source_health(
db,
source=self.name,
connection_state="disconnected",
last_error=None,
)
await db.commit()
await self._publish_task_update(force=True)
raise
def transform(self, raw_data: list[dict[str, Any]]) -> list[dict[str, Any]]:
records = []
for item in raw_data:
record = self._normalize_message(item)
if record:
records.append(record)
return records
async def _save_data(
self,
db: AsyncSession,
data: list[dict[str, Any]],
task_id: int | None = None,
snapshot_id: int | None = None,
) -> int:
now = datetime.now(UTC)
records_added = 0
latest_observed_at = now
for index, item in enumerate(data):
observed_at = item.get("received_at") or now
observation = await record_vessel_ais_observation(
db,
source=self.name,
normalized_payload=item,
raw_payload=item.get("_raw_payload") or item,
delivery_mode=AISSTREAM_DELIVERY_MODE,
transport=AISSTREAM_TRANSPORT,
message_type=item.get("_message_type") or "PositionReport",
source_message_id=item.get("_source_message_id"),
observed_at=observed_at,
collected_at=now,
)
if observation is not None:
records_added += 1
if isinstance(observed_at, datetime) and observed_at > latest_observed_at:
latest_observed_at = observed_at
if (index + 1) % 1000 == 0:
await self.update_progress(index + 1, commit=True)
await update_ais_source_health(
db,
source=self.name,
connection_state="connected",
observed_count=len(data),
last_seen_at=latest_observed_at,
last_success_at=now if data else None,
lag_seconds=max((now - latest_observed_at).total_seconds(), 0),
)
await db.commit()
await self.update_progress(records_added, force=True)
return records_added
async def _save_stream_record(self, db: AsyncSession, item: dict[str, Any]) -> bool:
now = datetime.now(UTC)
observed_at = item.get("received_at") or now
observation = await record_vessel_ais_observation(
db,
source=self.name,
normalized_payload=item,
raw_payload=item.get("_raw_payload") or item,
delivery_mode=AISSTREAM_DELIVERY_MODE,
transport=AISSTREAM_TRANSPORT,
message_type=item.get("_message_type") or "PositionReport",
source_message_id=item.get("_source_message_id"),
observed_at=observed_at,
collected_at=now,
)
await update_ais_source_health(
db,
source=self.name,
connection_state="connected",
observed_count=1,
last_seen_at=observed_at if isinstance(observed_at, datetime) else now,
last_success_at=now,
lag_seconds=max((now - observed_at).total_seconds(), 0) if isinstance(observed_at, datetime) else None,
)
await db.commit()
await self._broadcast_vessel_delta(item, created=observation is not None)
return observation is not None
async def _broadcast_vessel_delta(self, item: dict[str, Any], *, created: bool) -> None:
await broadcaster.broadcast_custom(
"vessels",
{
"action": "upsert",
"source": self.name,
"created": created,
"vessels": [
{
"mmsi": item.get("mmsi"),
"mmsi_display": str(item.get("mmsi")) if item.get("mmsi") is not None else None,
"name": item.get("name"),
"lat": item.get("lat"),
"lon": item.get("lon"),
"sog": item.get("sog"),
"cog": item.get("cog"),
"heading": item.get("heading"),
"nav_status": item.get("nav_status"),
"vessel_type": item.get("vessel_type"),
"vessel_type_name": item.get("vessel_type_name"),
"received_at": to_iso8601_utc(item.get("received_at")),
}
],
},
)
def _normalize_message(self, item: dict[str, Any]) -> dict[str, Any] | None:
message_type = str(item.get("MessageType") or item.get("message_type") or "")
metadata = item.get("MetaData") if isinstance(item.get("MetaData"), dict) else {}
message = item.get("Message") if isinstance(item.get("Message"), dict) else {}
body = message.get(message_type) if isinstance(message.get(message_type), dict) else message
if not isinstance(body, dict):
body = {}
mmsi = _as_int(_pick(metadata, "MMSI", "mmsi") or _pick(body, "MMSI", "mmsi"))
if mmsi is None:
return None
received_at = _parse_datetime(
_pick(metadata, "time_utc", "Time_UTC", "timestamp")
or _pick(body, "Timestamp", "timestamp", "time")
)
ship_name = _clean_text(
_pick(body, "Name", "ShipName", "name")
or _pick(metadata, "ShipName", "ship_name", "name")
)
record: dict[str, Any] = {
"mmsi": mmsi,
"received_at": received_at,
"_message_type": message_type or None,
"_source_message_id": item.get("MessageID") or item.get("message_id"),
"_raw_payload": item,
}
lat = _as_float(_pick(body, "Latitude", "lat", "latitude"))
lon = _as_float(_pick(body, "Longitude", "lon", "lng", "longitude"))
if lat is not None and lon is not None:
if not (-90 <= lat <= 90 and -180 <= lon <= 180):
return None
record.update(
{
"lat": lat,
"lon": lon,
"sog": _as_float(_pick(body, "Sog", "SOG", "speedOverGround")),
"cog": _as_float(_pick(body, "Cog", "COG", "courseOverGround")),
"heading": _as_int(_pick(body, "TrueHeading", "Heading", "heading")),
"nav_status": _as_int(_pick(body, "NavigationalStatus", "nav_status")),
}
)
vessel_type = _as_int(_pick(body, "Type", "ShipType", "vessel_type"))
record.update(
{
"name": ship_name,
"callsign": _pick(body, "CallSign", "callsign"),
"imo": _as_int(_pick(body, "ImoNumber", "IMO", "imo")),
"vessel_type": vessel_type,
"vessel_type_name": _pick(body, "TypeName", "ShipTypeName", "vessel_type_name")
or normalize_vessel_type_name(vessel_type),
"length": _as_float(_pick(body, "DimensionToBow", "Length", "length")),
"width": _as_float(_pick(body, "DimensionToPort", "Width", "width")),
}
)
return record
def _pick(item: dict[str, Any], *keys: str) -> Any:
for key in keys:
if key in item and item[key] not in (None, ""):
return item[key]
return None
def _clean_text(value: Any) -> str | None:
if value in (None, ""):
return None
text = str(value).strip()
return text or None
def _as_float(value: Any) -> float | None:
try:
if value in (None, ""):
return None
return float(value)
except (TypeError, ValueError):
return None
def _as_int(value: Any) -> int | None:
try:
if value in (None, ""):
return None
return int(float(value))
except (TypeError, ValueError):
return None
def _parse_datetime(value: Any) -> datetime | None:
if isinstance(value, datetime):
return value if value.tzinfo else value.replace(tzinfo=UTC)
if not value:
return None
if isinstance(value, (int, float)):
timestamp = float(value)
if timestamp > 10_000_000_000:
timestamp /= 1000
return datetime.fromtimestamp(timestamp, UTC)
if isinstance(value, str):
try:
parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
return parsed if parsed.tzinfo else parsed.replace(tzinfo=UTC)
except ValueError:
return None
return None

View File

@@ -54,6 +54,11 @@ class BaseCollector(ABC):
"task_id": self._current_task.id,
"status": self._current_task.status,
"phase": self._current_task.phase,
"phase_progress": self._current_task.phase_progress,
"phase_message": self._current_task.phase_message,
"phase_current": self._current_task.phase_current,
"phase_total": self._current_task.phase_total,
"phase_unit": self._current_task.phase_unit,
"progress": progress,
"records_processed": self._current_task.records_processed,
"total_records": self._current_task.total_records,
@@ -80,12 +85,52 @@ class BaseCollector(ABC):
await self._publish_task_update(force=force)
async def set_phase(self, phase: str):
async def set_phase(self, phase: str, *, message: str | None = None, reset_progress: bool = True):
if self._current_task and self._db_session:
self._current_task.phase = phase
self._current_task.phase_message = message
if reset_progress:
self._current_task.phase_progress = None
self._current_task.phase_current = None
self._current_task.phase_total = None
self._current_task.phase_unit = None
await self._db_session.commit()
await self._publish_task_update(force=True)
async def update_phase_progress(
self,
*,
current: int | None = None,
total: int | None = None,
unit: str | None = None,
message: str | None = None,
progress: float | None = None,
commit: bool = False,
force: bool = False,
):
"""Update progress for the current phase without changing task totals."""
if not self._current_task or not self._db_session:
return
if progress is None and current is not None and total and total > 0:
progress = (current / total) * 100
if progress is not None:
self._current_task.phase_progress = max(0.0, min(float(progress), 100.0))
if current is not None:
self._current_task.phase_current = max(0, int(current))
if total is not None:
self._current_task.phase_total = max(0, int(total))
if unit is not None:
self._current_task.phase_unit = unit
if message is not None:
self._current_task.phase_message = message
if commit:
await self._db_session.commit()
await self._publish_task_update(force=force)
@abstractmethod
async def fetch(self) -> List[Dict[str, Any]]:
"""Fetch raw data from source"""
@@ -251,7 +296,7 @@ class BaseCollector(ABC):
await self._publish_task_update(force=True)
try:
await self.set_phase("fetching")
await self.set_phase("fetching", message="正在拉取原始数据")
raw_data = await self.fetch()
task.total_records = len(raw_data)
await db.commit()
@@ -260,15 +305,20 @@ class BaseCollector(ABC):
if self.fail_on_empty and not raw_data:
raise RuntimeError(f"Collector {self.name} returned no data")
await self.set_phase("transforming")
await self.set_phase("transforming", message="正在转换采集数据")
data = self.transform(raw_data)
snapshot_id = await self._create_snapshot(db, task_id, data, start_time)
await self.set_phase("saving")
await self.set_phase("saving", message="正在保存采集数据")
records_count = await self._save_data(db, data, task_id=task_id, snapshot_id=snapshot_id)
task.status = "success"
task.phase = "completed"
task.phase_progress = 100.0
task.phase_message = "采集完成"
task.phase_current = records_count
task.phase_total = records_count
task.phase_unit = "records"
task.records_processed = records_count
task.progress = 100.0
task.completed_at = datetime.now(UTC)
@@ -285,6 +335,7 @@ class BaseCollector(ABC):
await db.rollback()
task.status = "cancelled"
task.phase = "cancelled"
task.phase_message = "采集已取消"
task.error_message = "Collection cancelled by operator and rolled back"
task.completed_at = datetime.now(UTC)
if snapshot_id is not None:
@@ -301,6 +352,7 @@ class BaseCollector(ABC):
await db.rollback()
task.status = "failed"
task.phase = "failed"
task.phase_message = str(e)
task.error_message = str(e)
task.completed_at = datetime.now(UTC)
if snapshot_id is not None:

View File

@@ -108,6 +108,11 @@ class IPtoASNPrefixGeoCollector(BaseCollector):
self._current_task.total_records = total_expected
self._current_task.records_processed = 0
self._current_task.progress = 0.0
self._current_task.phase_progress = 0.0
self._current_task.phase_message = "正在下载 IPtoASN 数据"
self._current_task.phase_current = 0
self._current_task.phase_total = total_expected
self._current_task.phase_unit = "bytes"
await self._db_session.commit()
await self._publish_task_update(force=True)
@@ -135,7 +140,14 @@ class IPtoASNPrefixGeoCollector(BaseCollector):
return
last_emit["value"] = aggregated
last_emit["t"] = now
await self.update_progress(min(aggregated, total_expected), commit=True)
current = min(aggregated, total_expected)
await self.update_phase_progress(
current=current,
total=total_expected,
unit="bytes",
message="正在下载 IPtoASN 数据",
)
await self.update_progress(current, commit=True)
batches = await asyncio.gather(
*(
@@ -148,6 +160,12 @@ class IPtoASNPrefixGeoCollector(BaseCollector):
)
)
if total_expected > 0:
await self.update_phase_progress(
current=total_expected,
total=total_expected,
unit="bytes",
message="IPtoASN 数据下载完成",
)
await self.update_progress(total_expected, commit=True, force=True)
rows: list[dict[str, Any]] = []

View File

@@ -39,12 +39,23 @@ class NRODelegatedPrefixGeoCollector(BaseCollector):
self._current_task.total_records = total_expected
self._current_task.records_processed = 0
self._current_task.progress = 0.0
self._current_task.phase_progress = 0.0
self._current_task.phase_message = "正在下载 NRO delegated 数据"
self._current_task.phase_current = 0
self._current_task.phase_total = total_expected
self._current_task.phase_unit = "bytes"
await self._db_session.commit()
await self._publish_task_update(force=True)
async def on_progress(downloaded: int, total: int | None) -> None:
if not total or total <= 0:
return
await self.update_phase_progress(
current=min(downloaded, total),
total=total,
unit="bytes",
message="正在下载 NRO delegated 数据",
)
await self.update_progress(min(downloaded, total), commit=True)
body_path = await self._downloader.download_file(

View File

@@ -40,12 +40,23 @@ class OpenGeoFeedPrefixGeoCollector(BaseCollector):
self._current_task.total_records = total_expected
self._current_task.records_processed = 0
self._current_task.progress = 0.0
self._current_task.phase_progress = 0.0
self._current_task.phase_message = "正在下载 OpenGeoFeed 数据"
self._current_task.phase_current = 0
self._current_task.phase_total = total_expected
self._current_task.phase_unit = "bytes"
await self._db_session.commit()
await self._publish_task_update(force=True)
async def on_progress(downloaded: int, total: int | None) -> None:
if not total or total <= 0:
return
await self.update_phase_progress(
current=min(downloaded, total),
total=total,
unit="bytes",
message="正在下载 OpenGeoFeed 数据",
)
await self.update_progress(min(downloaded, total), commit=True)
body_path = await self._downloader.download_file(

View File

@@ -1,28 +1,26 @@
"""BarentsWatch AIS collector for vessel tracking."""
from datetime import UTC, datetime, timedelta
from datetime import UTC, datetime
from typing import Any
import httpx
from sqlalchemy import delete, select
from sqlalchemy.ext.asyncio import AsyncSession
from app.models.vessel import VesselPosition, VesselStatic
from app.core.time import to_iso8601_utc
from app.core.websocket.broadcaster import broadcaster
from app.services.barentswatch import (
BARENTSWATCH_LATEST_URL,
fetch_barentswatch_access_token,
resolve_barentswatch_config,
)
from app.services.collectors.base import BaseCollector
VESSEL_TYPE_NAMES = {
30: "Fishing",
35: "Military",
60: "Passenger",
70: "Cargo",
80: "Tanker",
}
from app.services.vessel_ais_aggregation import (
BARENTSWATCH_DELIVERY_MODE,
BARENTSWATCH_TRANSPORT,
record_vessel_ais_observation,
update_ais_source_health,
)
from app.services.vessel_types import normalize_vessel_type_name
class VesselAISCollector(BaseCollector):
@@ -92,51 +90,75 @@ class VesselAISCollector(BaseCollector):
records_added = 0
for index, item in enumerate(data):
static = await db.get(VesselStatic, item["mmsi"])
if static is None:
static = VesselStatic(mmsi=item["mmsi"])
db.add(static)
for field in (
"name",
"callsign",
"vessel_type",
"vessel_type_name",
"flag",
"length",
"width",
"draught",
"imo",
):
value = item.get(field)
if value not in (None, ""):
setattr(static, field, value)
static.updated_at = now
db.add(
VesselPosition(
mmsi=item["mmsi"],
lat=item["lat"],
lon=item["lon"],
sog=item.get("sog"),
cog=item.get("cog"),
heading=item.get("heading"),
nav_status=item.get("nav_status"),
received_at=item.get("received_at") or now,
)
observed_at = item.get("received_at") or now
await record_vessel_ais_observation(
db,
source=self.name,
normalized_payload=item,
raw_payload=item,
delivery_mode=BARENTSWATCH_DELIVERY_MODE,
transport=BARENTSWATCH_TRANSPORT,
observed_at=observed_at,
collected_at=now,
)
records_added += 1
if (index + 1) % 1000 == 0:
await self.update_progress(index + 1, commit=True)
await db.execute(
delete(VesselPosition).where(VesselPosition.received_at < now - timedelta(hours=24))
latest_observed_at = max(
(item.get("received_at") for item in data if item.get("received_at")),
default=now,
)
await update_ais_source_health(
db,
source=self.name,
connection_state="connected",
observed_count=len(data),
last_seen_at=latest_observed_at,
last_success_at=now if data else None,
lag_seconds=max((now - latest_observed_at).total_seconds(), 0),
)
await db.commit()
await self._broadcast_vessel_snapshot(data)
await self.update_progress(records_added, force=True)
return records_added
async def _broadcast_vessel_snapshot(self, data: list[dict[str, Any]]) -> None:
"""Push REST collector updates through the same realtime vessel channel."""
if not data:
return
batch_size = 500
for offset in range(0, len(data), batch_size):
batch = data[offset : offset + batch_size]
await broadcaster.broadcast_custom(
"vessels",
{
"action": "upsert",
"source": self.name,
"created": True,
"vessels": [
{
"mmsi": item.get("mmsi"),
"mmsi_display": str(item.get("mmsi")) if item.get("mmsi") is not None else None,
"name": item.get("name"),
"callsign": item.get("callsign"),
"lat": item.get("lat"),
"lon": item.get("lon"),
"sog": item.get("sog"),
"cog": item.get("cog"),
"heading": item.get("heading"),
"nav_status": item.get("nav_status"),
"vessel_type": item.get("vessel_type"),
"vessel_type_name": item.get("vessel_type_name"),
"received_at": to_iso8601_utc(item.get("received_at")),
}
for item in batch
],
},
)
def _normalize_record(self, item: dict[str, Any]) -> dict[str, Any] | None:
mmsi = _as_int(_pick(item, "mmsi", "MMSI", "Mmsi"))
lat = _as_float(_pick(item, "lat", "latitude", "Latitude"))
@@ -156,7 +178,7 @@ class VesselAISCollector(BaseCollector):
vessel_type = _as_int(_pick(item, "vessel_type", "shipType", "ship_type", "ShipType"))
vessel_type_name = (
_pick(item, "vessel_type_name", "shipTypeName", "ship_type_name", "VesselTypeName")
or _vessel_type_name(vessel_type)
or normalize_vessel_type_name(vessel_type)
)
received_at = _parse_datetime(_pick(item, "received_at", "timestamp", "time", "msgtime"))
@@ -255,19 +277,3 @@ def _parse_datetime(value: Any) -> datetime | None:
except ValueError:
return None
return None
def _vessel_type_name(vessel_type: int | None) -> str:
if vessel_type is None:
return "Other"
if 70 <= vessel_type <= 79:
return "Cargo"
if 80 <= vessel_type <= 89:
return "Tanker"
if 60 <= vessel_type <= 69:
return "Passenger"
if vessel_type == 30:
return "Fishing"
if vessel_type == 35:
return "Military"
return VESSEL_TYPE_NAMES.get(vessel_type, "Other")

View File

@@ -66,9 +66,68 @@ BARENTSWATCH_DEFAULT_GUIDE = CredentialGuideDefault(
""",
)
AISSTREAM_DEFAULT_GUIDE = CredentialGuideDefault(
provider="aisstream",
title="AISStream API Key 获取教程",
prompt=(
"请生成一份中文教程,指导开发者获取 AISStream 的 API Key 并配置到 Planet。"
"教程要面向已经有本地开发环境的人,包含注册/登录 AISStream、获取 API Key、"
"理解免费额度和订阅范围、在 Planet 设置中心填写 API Key、配置 bounding boxes "
"和 message types、验证连接、常见失败排查。必须提醒用户以 AISStream 当前官网和"
"服务条款为准,不要编造具体页面按钮文案。"
),
markdown="""## AISStream API Key 获取
官方入口https://aisstream.io/
1. 打开 AISStream 官网,按当前页面指引注册或登录账号。
2. 在账号/API 管理页面创建或复制你的 API Key。
3. 先确认当前账号额度、使用条款和可订阅区域。实时 AIS 流量可能很大,不建议一开始订阅全球范围。
4. 回到 Planet 的 `设置 -> 采集器设置 -> AISStream 实时船舶`。
5. 在 `AISStream 凭证` 中填入 API Key。
6. Endpoint 通常保持默认:`wss://stream.aisstream.io/v0/stream`。
7. 按需配置 `Bounding Boxes JSON` 和 `消息类型`。
8. 点击连接测试,确认系统能读取凭证且 WebSocket endpoint 格式有效。
9. 保存采集器设置后再运行 `aisstream_vessels` collector。
### 推荐配置
默认消息类型:
```json
["PositionReport", "ShipStaticData"]
```
默认 Bounding Boxes 示例:
```json
[[[-90, -180], [90, 180]]]
```
这个示例表示全球范围。实际使用时建议先改成较小区域,降低消息量和处理压力。
### 请求规则
- Endpoint`wss://stream.aisstream.io/v0/stream`
- 传输方式WebSocket
- API Key 放在订阅 payload 中,不放在 HTTP header。
- Planet 会把 AISStream 标记为 `delivery_mode = realtime_stream`、`transport = websocket`。
- AISStream collector 只写入 AIS raw observations不直接覆盖最终船只展示表。
### 常见排查
- `未找到凭证`:确认 API Key 已保存到采集器设置,或设置了 `AISSTREAM_API_KEY` 环境变量 / `~/.zshrc`。
- `endpoint 必须是 ws:// 或 wss://`AISStream 是 WebSocket 流接口,不要填普通 `https://` API 地址。
- 采集量过大:缩小 `Bounding Boxes JSON`,减少 `message_types`,或降低单次最大消息数。
- 没有船只数据:确认订阅区域内确实有 AIS 活动,并检查 API Key 当前额度和权限。
- 连接中断实时流可能受网络和上游限流影响collector 会记录源健康状态供聚合服务回退。
""",
)
DEFAULT_CREDENTIAL_GUIDES = {
BARENTSWATCH_DEFAULT_GUIDE.provider: BARENTSWATCH_DEFAULT_GUIDE,
AISSTREAM_DEFAULT_GUIDE.provider: AISSTREAM_DEFAULT_GUIDE,
}

View File

@@ -0,0 +1,391 @@
"""Runtime helpers for mapped custom data sources."""
from __future__ import annotations
import asyncio
import base64
import json
from datetime import UTC, datetime
from typing import Any
import httpx
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.target_schema_registry import TARGET_SCHEMAS
from app.db.session import async_session_factory
from app.models.datasource_config import DataSourceConfig
from app.models.datasource_mapping import DataSourceMappingTemplate
from app.services.datasource_mapping import (
MappingError,
execute_mapping,
extract_path,
persist_mapped_records,
)
DEFAULT_MAPPING_TEMPLATES: dict[str, dict[str, Any]] = {
"vessel_ais": {
"source": {"items_path": "$"},
"fields": {
"mmsi": {"path": "$.mmsi", "type": "integer"},
"name": {"path": "$.name", "type": "string", "default": None},
"lat": {"path": "$.lat", "type": "float"},
"lon": {"path": "$.lon", "type": "float"},
"sog": {"path": "$.sog", "type": "float", "default": None},
"cog": {"path": "$.cog", "type": "float", "default": None},
"heading": {"path": "$.heading", "type": "integer", "default": None},
"nav_status": {"path": "$.nav_status", "type": "integer", "default": None},
"callsign": {"path": "$.callsign", "type": "string", "default": None},
"vessel_type": {"path": "$.vessel_type", "type": "string", "default": None},
"vessel_type_name": {"path": "$.vessel_type_name", "type": "string", "default": None},
"received_at": {"path": "$.received_at", "type": "datetime", "default": None},
},
"meta": {"generated_by": "default_template", "requires_review": False},
},
}
RUNNING_CUSTOM_STREAM_TASKS: dict[int, asyncio.Task[Any]] = {}
class CustomDatasourceRuntimeError(RuntimeError):
"""Raised when a custom datasource cannot run."""
def build_request_headers(auth_type: str, auth_config: dict, headers: dict) -> dict[str, str]:
request_headers = {str(key): str(value) for key, value in (headers or {}).items()}
auth_type = str(auth_type or "none").lower()
auth_config = auth_config or {}
if auth_type == "bearer" and auth_config.get("token"):
request_headers["Authorization"] = f"Bearer {auth_config['token']}"
elif auth_type == "api_key" and auth_config.get("api_key"):
location = str(auth_config.get("in") or auth_config.get("location") or "header").lower()
if location != "query":
key_name = auth_config.get("key_name", "X-API-Key")
request_headers[str(key_name)] = str(auth_config["api_key"])
elif auth_type == "basic":
username = auth_config.get("username", "")
password = auth_config.get("password", "")
credentials = f"{username}:{password}"
encoded = base64.b64encode(credentials.encode()).decode()
request_headers["Authorization"] = f"Basic {encoded}"
return request_headers
def build_query_params(auth_type: str, auth_config: dict, config: dict) -> dict[str, Any]:
params: dict[str, Any] = {}
candidate = (config or {}).get("params") or (config or {}).get("query_params")
if isinstance(candidate, dict):
params.update(candidate)
auth_type = str(auth_type or "none").lower()
auth_config = auth_config or {}
if auth_type == "api_key" and auth_config.get("api_key"):
location = str(auth_config.get("in") or auth_config.get("location") or "header").lower()
if location == "query":
key_name = auth_config.get("key_name") or auth_config.get("param_name") or "api_key"
params[str(key_name)] = auth_config["api_key"]
return params
async def load_active_mapping(
db: AsyncSession,
datasource_config_id: int,
) -> DataSourceMappingTemplate:
result = await db.execute(
select(DataSourceMappingTemplate)
.where(DataSourceMappingTemplate.datasource_config_id == datasource_config_id)
.where(DataSourceMappingTemplate.is_active.is_(True))
.order_by(DataSourceMappingTemplate.version.desc())
.limit(1)
)
mapping = result.scalar_one_or_none()
if mapping is not None:
return mapping
datasource = await db.get(DataSourceConfig, datasource_config_id)
if datasource is None:
raise CustomDatasourceRuntimeError("Configuration not found")
target_schema = (datasource.config or {}).get("target_schema")
template_body = DEFAULT_MAPPING_TEMPLATES.get(str(target_schema or "")) if target_schema else None
if not template_body or target_schema not in TARGET_SCHEMAS:
raise CustomDatasourceRuntimeError(
"No active mapping template found and no default template available for this target schema"
)
mapping = DataSourceMappingTemplate(
datasource_config_id=datasource_config_id,
target_schema=str(target_schema),
mapping_json=template_body,
sample_payload_hash=None,
validation_status="valid",
version=1,
is_active=True,
)
db.add(mapping)
await db.commit()
await db.refresh(mapping)
return mapping
async def fetch_rest_payload(config: DataSourceConfig, limit_bytes: int) -> Any:
request_config = config.config or {}
method = str(request_config.get("method") or request_config.get("request_method") or "GET").upper()
if method not in {"GET", "POST"}:
raise CustomDatasourceRuntimeError("Only GET and POST sample requests are supported.")
headers = build_request_headers(config.auth_type, config.auth_config or {}, config.headers or {})
params = build_query_params(config.auth_type, config.auth_config or {}, request_config)
timeout = float(request_config.get("timeout", 30))
json_body = request_config.get("json_body")
if json_body is None and str(request_config.get("body_type") or "").lower() in {"json", ""}:
candidate = request_config.get("body")
if isinstance(candidate, (dict, list)):
json_body = candidate
async with httpx.AsyncClient(timeout=timeout, follow_redirects=True) as client:
response = await client.request(
method,
config.endpoint,
headers=headers,
params=params or None,
json=json_body,
)
response.raise_for_status()
content = response.content[:limit_bytes]
if "application/json" in response.headers.get("content-type", ""):
return json.loads(content.decode(response.encoding or "utf-8"))
return {"text": content.decode(response.encoding or "utf-8", errors="replace")}
async def run_mapped_rest_config(
db: AsyncSession,
datasource: DataSourceConfig,
) -> dict[str, Any]:
mapping = await load_active_mapping(db, datasource.id)
sample = await fetch_rest_payload(datasource, 5_000_000)
mapped = execute_mapping(sample, mapping.mapping_json, mapping.target_schema)
if mapped["failed_count"] > 0:
return {
"status": "failed",
"datasource_config_id": datasource.id,
"mapping_id": mapping.id,
"mapping_version": mapping.version,
"target_schema": mapping.target_schema,
"mapped_count": mapped["mapped_count"],
"failed_count": mapped["failed_count"],
"errors": mapped["errors"][:20],
}
request_config = datasource.config or {}
written_count = await persist_mapped_records(
db,
datasource_name=datasource.name,
datasource_config_id=datasource.id,
target_schema=mapping.target_schema,
records=mapped["records"],
mapping_version=mapping.version,
delivery_mode=request_config.get("delivery_mode") or "polling",
transport="http",
)
return {
"status": "success",
"datasource_config_id": datasource.id,
"mapping_id": mapping.id,
"mapping_version": mapping.version,
"target_schema": mapping.target_schema,
"fetched_count": mapped["total_items"],
"mapped_count": mapped["mapped_count"],
"written_count": written_count,
}
def _items_from_ws_message(payload: Any, config: dict) -> Any:
message_path = config.get("ws_message_path")
items_path = config.get("ws_items_path")
value = extract_path(payload, message_path) if message_path else payload
return extract_path(value, items_path) if items_path else value
async def _connect_websocket(endpoint: str, headers: dict[str, str]):
import websockets
try:
return await websockets.connect(endpoint, additional_headers=headers or None)
except TypeError:
return await websockets.connect(endpoint, extra_headers=headers or None)
async def test_websocket_config(config: DataSourceConfig) -> dict[str, Any]:
if not str(config.endpoint or "").startswith(("ws://", "wss://")):
raise CustomDatasourceRuntimeError("WebSocket datasource endpoint must start with ws:// or wss://")
runtime_config = config.config or {}
headers = build_request_headers(config.auth_type, config.auth_config or {}, config.headers or {})
receive_timeout = float(runtime_config.get("receive_timeout_seconds") or runtime_config.get("timeout") or 10)
async with await _connect_websocket(config.endpoint, headers) as websocket:
subscribe_message = runtime_config.get("ws_subscribe_message")
if isinstance(subscribe_message, (dict, list)):
await websocket.send(json.dumps(subscribe_message))
elif isinstance(subscribe_message, str) and subscribe_message.strip():
await websocket.send(subscribe_message)
raw_message = await asyncio.wait_for(websocket.recv(), timeout=receive_timeout)
return {
"success": True,
"message_preview": raw_message[:1000] if isinstance(raw_message, str) else str(raw_message)[:1000],
}
async def run_mapped_websocket_config(
db: AsyncSession,
datasource: DataSourceConfig,
*,
debug_max_messages: int | None = None,
use_config_debug_max_messages: bool = True,
) -> dict[str, Any]:
if not str(datasource.endpoint or "").startswith(("ws://", "wss://")):
raise CustomDatasourceRuntimeError("WebSocket datasource endpoint must start with ws:// or wss://")
mapping = await load_active_mapping(db, datasource.id)
runtime_config = datasource.config or {}
max_messages = debug_max_messages
if max_messages is None and use_config_debug_max_messages:
max_messages = runtime_config.get("debug_max_messages")
max_messages = int(max_messages) if max_messages else None
receive_timeout = float(runtime_config.get("receive_timeout_seconds") or runtime_config.get("timeout") or 30)
reconnect = bool(runtime_config.get("ws_reconnect", True))
reconnect_delay = float(runtime_config.get("reconnect_delay_seconds") or 3)
headers = build_request_headers(datasource.auth_type, datasource.auth_config or {}, datasource.headers or {})
messages_seen = 0
mapped_count = 0
failed_count = 0
written_count = 0
errors: list[dict[str, Any]] = []
started_at = datetime.now(UTC)
while True:
try:
async with await _connect_websocket(datasource.endpoint, headers) as websocket:
subscribe_message = runtime_config.get("ws_subscribe_message")
if isinstance(subscribe_message, (dict, list)):
await websocket.send(json.dumps(subscribe_message))
elif isinstance(subscribe_message, str) and subscribe_message.strip():
await websocket.send(subscribe_message)
while True:
raw_message = await asyncio.wait_for(websocket.recv(), timeout=receive_timeout)
messages_seen += 1
try:
payload = json.loads(raw_message)
except json.JSONDecodeError as exc:
failed_count += 1
errors.append({"message": "invalid_json", "error": str(exc)})
continue
extracted = _items_from_ws_message(payload, runtime_config)
try:
mapped = execute_mapping(extracted, mapping.mapping_json, mapping.target_schema)
except (MappingError, ValueError) as exc:
failed_count += 1
errors.append({"message": "mapping_failed", "error": str(exc)})
continue
mapped_count += mapped["mapped_count"]
failed_count += mapped["failed_count"]
if mapped["errors"]:
errors.extend(mapped["errors"][:5])
if mapped["records"]:
written_count += await persist_mapped_records(
db,
datasource_name=datasource.name,
datasource_config_id=datasource.id,
target_schema=mapping.target_schema,
records=mapped["records"],
mapping_version=mapping.version,
delivery_mode=runtime_config.get("delivery_mode") or "realtime_stream",
transport="websocket",
)
if max_messages and messages_seen >= max_messages:
return {
"status": "success",
"datasource_config_id": datasource.id,
"mapping_id": mapping.id,
"mapping_version": mapping.version,
"target_schema": mapping.target_schema,
"messages_seen": messages_seen,
"mapped_count": mapped_count,
"failed_count": failed_count,
"written_count": written_count,
"errors": errors[:20],
"execution_time_seconds": (datetime.now(UTC) - started_at).total_seconds(),
}
except asyncio.CancelledError:
raise
except Exception as exc:
failed_count += 1
errors.append({"message": "websocket_error", "error": f"{exc.__class__.__name__}: {exc}"})
if not reconnect or max_messages:
return {
"status": "failed" if written_count == 0 else "partial",
"datasource_config_id": datasource.id,
"mapping_id": mapping.id,
"mapping_version": mapping.version,
"target_schema": mapping.target_schema,
"messages_seen": messages_seen,
"mapped_count": mapped_count,
"failed_count": failed_count,
"written_count": written_count,
"errors": errors[:20],
}
await asyncio.sleep(reconnect_delay)
async def run_custom_stream_by_id(config_id: int) -> dict[str, Any]:
async with async_session_factory() as db:
datasource = await db.get(DataSourceConfig, config_id)
if not datasource:
raise CustomDatasourceRuntimeError("Configuration not found")
return await run_mapped_websocket_config(
db,
datasource,
use_config_debug_max_messages=False,
)
def start_custom_stream(config_id: int) -> bool:
existing = RUNNING_CUSTOM_STREAM_TASKS.get(config_id)
if existing is not None and not existing.done():
return False
task = asyncio.create_task(run_custom_stream_by_id(config_id), name=f"custom-stream:{config_id}")
RUNNING_CUSTOM_STREAM_TASKS[config_id] = task
def _cleanup(done_task: asyncio.Task[Any]) -> None:
if RUNNING_CUSTOM_STREAM_TASKS.get(config_id) is done_task:
RUNNING_CUSTOM_STREAM_TASKS.pop(config_id, None)
task.add_done_callback(_cleanup)
return True
async def stop_custom_stream(config_id: int) -> bool:
task = RUNNING_CUSTOM_STREAM_TASKS.get(config_id)
if task is None or task.done():
RUNNING_CUSTOM_STREAM_TASKS.pop(config_id, None)
return False
task.cancel()
try:
await task
except asyncio.CancelledError:
return True
return task.cancelled()
def get_custom_stream_status(config_id: int) -> dict[str, Any]:
task = RUNNING_CUSTOM_STREAM_TASKS.get(config_id)
return {
"config_id": config_id,
"running": bool(task and not task.done()),
"done": bool(task and task.done()),
}

View File

@@ -26,6 +26,7 @@ from app.services.barentswatch import (
CONNECTIVITY_VALIDATION_KEY = "connectivity_validation"
CONNECTIVITY_STORE_CATEGORY = "datasource_connectivity_validations"
SUPPORTED_CREDENTIAL_PROVIDERS = {"barentswatch", "spacetrack", "aisstream"}
def _sha256_json(payload: Any) -> str:
@@ -43,6 +44,36 @@ def _resolve_spacetrack_credentials() -> tuple[str, str, str]:
return username, password, source or "missing"
async def _resolve_aisstream_api_key(
db=None,
credential_override: dict[str, str] | None = None,
) -> tuple[str, str]:
if credential_override and credential_override.get("api_key"):
return str(credential_override["api_key"]), "draft"
env_key = os.getenv("AISSTREAM_API_KEY")
zshrc_key = _read_zshrc_env().get("AISSTREAM_API_KEY")
if db is not None:
result = await db.execute(
select(DataSourceConfig)
.where(DataSourceConfig.name == "aisstream_vessels")
.where(DataSourceConfig.is_active.is_(True))
)
record = result.scalar_one_or_none()
if record:
auth_config = record.auth_config or {}
runtime_config = record.config or {}
api_key = auth_config.get("api_key") or runtime_config.get("api_key")
if api_key:
return str(api_key), "datasource_config"
if env_key:
return env_key, "environment"
if zshrc_key:
return zshrc_key, "~/.zshrc"
return "", "missing"
def strip_connectivity_validation(config: dict | None) -> dict:
cleaned = dict(config or {})
cleaned.pop(CONNECTIVITY_VALIDATION_KEY, None)
@@ -103,6 +134,10 @@ async def build_builtin_connectivity_checksum(
"password": password,
}
)
elif credential_provider == "aisstream":
api_key, credential_source = await _resolve_aisstream_api_key(db, credential_override)
has_credentials = bool(api_key)
credential_fingerprint = _sha256_json({"api_key": api_key})
elif defaults.get("requires_credentials"):
credential_source = str(credential_provider or "unsupported")
@@ -130,6 +165,7 @@ async def test_builtin_connectivity(
headers: dict | None,
config: dict | None,
db=None,
credential_override: dict[str, str] | None = None,
) -> dict[str, Any]:
defaults = DEFAULT_DATASOURCES.get(source)
if not defaults:
@@ -145,6 +181,7 @@ async def test_builtin_connectivity(
headers,
config,
db,
credential_override,
)
if credential_context["requires_credentials"] and not credential_context["has_credentials"]:
return {
@@ -155,10 +192,9 @@ async def test_builtin_connectivity(
"settings_tab": "collector_credentials",
**credential_context,
}
supported_credential_providers = {"barentswatch", "spacetrack"}
if (
credential_context["requires_credentials"]
and credential_context["credential_provider"] not in supported_credential_providers
and credential_context["credential_provider"] not in SUPPORTED_CREDENTIAL_PROVIDERS
):
return {
"success": False,
@@ -174,6 +210,23 @@ async def test_builtin_connectivity(
timeout = float(request_config.get("timeout") or 30)
request_endpoint = endpoint
if credential_context["credential_provider"] == "aisstream":
if not str(request_endpoint).startswith(("ws://", "wss://")):
return {
"success": False,
"checksum": checksum,
"stage": "endpoint",
"message": "AISStream endpoint 必须是 ws:// 或 wss:// WebSocket 地址。",
**credential_context,
}
return {
"success": True,
"checksum": checksum,
"stage": "credentials",
"message": "AISStream 凭证已配置WebSocket endpoint 格式有效。",
**credential_context,
}
try:
async with httpx.AsyncClient(timeout=timeout, follow_redirects=True) as client:
if credential_context["credential_provider"] == "barentswatch":

View File

@@ -290,25 +290,77 @@ async def persist_mapped_records(
target_schema: str,
records: list[dict[str, Any]],
mapping_version: int,
delivery_mode: str | None = None,
transport: str | None = None,
) -> int:
"""Persist validated mapped records to the destination for a target schema."""
if target_schema == "vessel_ais":
from app.models.vessel import VesselPosition
from app.core.time import to_iso8601_utc
from app.core.websocket.broadcaster import broadcaster
from app.services.vessel_ais_aggregation import (
record_vessel_ais_observation,
update_ais_source_health,
)
now = datetime.now(UTC)
latest_observed_at = now
written_count = 0
for record in records:
db.add(
VesselPosition(
mmsi=record["mmsi"],
lat=record["lat"],
lon=record["lon"],
sog=record.get("sog"),
cog=record.get("cog"),
heading=record.get("heading"),
received_at=_parse_datetime(record.get("received_at")) or datetime.now(UTC),
)
observed_at = _parse_datetime(record.get("received_at")) or now
observation = await record_vessel_ais_observation(
db,
source=datasource_name,
normalized_payload=record,
raw_payload=record,
delivery_mode=delivery_mode or "polling",
transport=transport or "http",
message_type="PositionReport",
observed_at=observed_at,
collected_at=now,
)
if observation is not None:
written_count += 1
if observed_at > latest_observed_at:
latest_observed_at = observed_at
await update_ais_source_health(
db,
source=datasource_name,
connection_state="connected",
observed_count=len(records),
last_seen_at=latest_observed_at,
last_success_at=now if records else None,
lag_seconds=max((now - latest_observed_at).total_seconds(), 0),
)
await db.commit()
return len(records)
if records:
await broadcaster.broadcast_custom(
"vessels",
{
"action": "upsert",
"source": datasource_name,
"created": True,
"vessels": [
{
"mmsi": record.get("mmsi"),
"mmsi_display": str(record.get("mmsi")) if record.get("mmsi") is not None else None,
"name": record.get("name"),
"callsign": record.get("callsign"),
"lat": record.get("lat"),
"lon": record.get("lon"),
"sog": record.get("sog"),
"cog": record.get("cog"),
"heading": record.get("heading"),
"nav_status": record.get("nav_status"),
"vessel_type": record.get("vessel_type"),
"vessel_type_name": record.get("vessel_type_name"),
"received_at": to_iso8601_utc(_parse_datetime(record.get("received_at"))),
}
for record in records
],
},
)
return written_count
from app.models.collected_data import CollectedData

View File

@@ -0,0 +1,198 @@
"""Persistence + validation for the v4 vessel_ais aggregation strategy."""
from __future__ import annotations
from typing import Any
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.models.system_setting import SystemSetting
VESSEL_AGGREGATION_STRATEGY_CATEGORY = "vessel_aggregation_strategy"
DYNAMIC_FIELDS: tuple[str, ...] = ("lat", "lon", "sog", "cog", "heading", "nav_status")
STATIC_FIELDS: tuple[str, ...] = (
"name",
"callsign",
"imo",
"flag",
"vessel_type",
"vessel_type_name",
"length",
"width",
"draught",
)
ALLOWED_FIELDS: frozenset[str] = frozenset(DYNAMIC_FIELDS + STATIC_FIELDS)
ALLOWED_DYNAMIC_MODES: frozenset[str] = frozenset({"newest"})
ALLOWED_STATIC_MODES: frozenset[str] = frozenset({"source_priority", "non_empty", "newest", "locked"})
ALLOWED_LOCKED_DYNAMIC_MODES: frozenset[str] = frozenset({"newest", "source_priority", "locked"})
DEFAULT_STRATEGY: dict[str, Any] = {
"version": 1,
"vessel_ais": {
"source_priority": ["aisstream_vessels", "barentswatch_vessels"],
"field_rules": {},
"freshness": {
"realtime_stream_seconds": 900,
"polling_seconds": 3600,
},
"allow_dynamic_lock": False,
},
}
class StrategyValidationError(ValueError):
"""Raised when a saved strategy payload is malformed."""
def _coerce_str_list(value: Any, *, label: str) -> list[str]:
if value is None:
return []
if not isinstance(value, list):
raise StrategyValidationError(f"{label} must be a list of source names")
out: list[str] = []
for item in value:
if not isinstance(item, str) or not item.strip():
raise StrategyValidationError(f"{label} entries must be non-empty strings")
out.append(item.strip())
return out
def validate_strategy(payload: dict[str, Any]) -> dict[str, Any]:
"""Validate and normalize a strategy payload. Raise StrategyValidationError on issues."""
if not isinstance(payload, dict):
raise StrategyValidationError("strategy payload must be an object")
vessel_ais = payload.get("vessel_ais")
if not isinstance(vessel_ais, dict):
raise StrategyValidationError("strategy.vessel_ais is required and must be an object")
allow_dynamic_lock = bool(vessel_ais.get("allow_dynamic_lock", False))
source_priority = _coerce_str_list(
vessel_ais.get("source_priority"),
label="vessel_ais.source_priority",
)
raw_rules = vessel_ais.get("field_rules") or {}
if not isinstance(raw_rules, dict):
raise StrategyValidationError("vessel_ais.field_rules must be an object")
field_rules: dict[str, dict[str, Any]] = {}
for field, rule in raw_rules.items():
if field not in ALLOWED_FIELDS:
raise StrategyValidationError(f"unknown vessel_ais field: {field}")
if not isinstance(rule, dict):
raise StrategyValidationError(f"field_rules.{field} must be an object")
mode = str(rule.get("mode") or "").strip()
if not mode:
raise StrategyValidationError(f"field_rules.{field}.mode is required")
is_dynamic = field in DYNAMIC_FIELDS
if is_dynamic:
allowed_modes = ALLOWED_LOCKED_DYNAMIC_MODES if allow_dynamic_lock else ALLOWED_DYNAMIC_MODES
if mode not in allowed_modes:
if not allow_dynamic_lock:
raise StrategyValidationError(
f"field_rules.{field}.mode='{mode}' requires allow_dynamic_lock=true"
)
raise StrategyValidationError(
f"field_rules.{field}.mode must be one of {sorted(allowed_modes)}"
)
else:
if mode not in ALLOWED_STATIC_MODES:
raise StrategyValidationError(
f"field_rules.{field}.mode must be one of {sorted(ALLOWED_STATIC_MODES)}"
)
normalized_rule: dict[str, Any] = {"mode": mode}
rule_priority = rule.get("source_priority")
if rule_priority is not None:
normalized_rule["source_priority"] = _coerce_str_list(
rule_priority,
label=f"field_rules.{field}.source_priority",
)
if mode == "locked":
locked_source = rule.get("locked_source")
if not isinstance(locked_source, str) or not locked_source.strip():
raise StrategyValidationError(
f"field_rules.{field}.locked_source must be a non-empty string when mode=locked"
)
normalized_rule["locked_source"] = locked_source.strip()
field_rules[field] = normalized_rule
raw_freshness = vessel_ais.get("freshness") or {}
if not isinstance(raw_freshness, dict):
raise StrategyValidationError("vessel_ais.freshness must be an object")
freshness: dict[str, int] = {}
for key in ("realtime_stream_seconds", "polling_seconds"):
value = raw_freshness.get(key, DEFAULT_STRATEGY["vessel_ais"]["freshness"][key])
try:
seconds = int(value)
except (TypeError, ValueError) as exc:
raise StrategyValidationError(f"freshness.{key} must be an integer") from exc
if seconds < 0:
raise StrategyValidationError(f"freshness.{key} must be non-negative")
freshness[key] = seconds
return {
"version": int(payload.get("version") or 0) + 1,
"vessel_ais": {
"source_priority": source_priority,
"field_rules": field_rules,
"freshness": freshness,
"allow_dynamic_lock": allow_dynamic_lock,
},
}
async def _select_setting(db: AsyncSession) -> SystemSetting | None:
result = await db.execute(
select(SystemSetting).where(SystemSetting.category == VESSEL_AGGREGATION_STRATEGY_CATEGORY)
)
return result.scalar_one_or_none()
def _current_version(setting: SystemSetting | None) -> int:
if setting is None:
return 0
payload = setting.payload or {}
return int(payload.get("version") or 0)
async def load_strategy(db: AsyncSession) -> dict[str, Any]:
setting = await _select_setting(db)
if setting is None or not isinstance(setting.payload, dict):
return DEFAULT_STRATEGY
payload = setting.payload
if "vessel_ais" not in payload:
return DEFAULT_STRATEGY
return payload
async def save_strategy(db: AsyncSession, payload: dict[str, Any]) -> dict[str, Any]:
"""Validate + persist; bumps version automatically."""
existing = await _select_setting(db)
incoming = dict(payload)
incoming.setdefault("version", _current_version(existing))
validated = validate_strategy(incoming)
if existing is None:
existing = SystemSetting(category=VESSEL_AGGREGATION_STRATEGY_CATEGORY, payload=validated)
db.add(existing)
else:
existing.payload = validated
await db.commit()
return validated
async def reset_strategy(db: AsyncSession) -> dict[str, Any]:
existing = await _select_setting(db)
payload = {**DEFAULT_STRATEGY, "version": _current_version(existing) + 1}
if existing is None:
existing = SystemSetting(category=VESSEL_AGGREGATION_STRATEGY_CATEGORY, payload=payload)
db.add(existing)
else:
existing.payload = payload
await db.commit()
return payload

View File

@@ -0,0 +1,698 @@
"""AIS raw observation and aggregation support for vessel collectors."""
from datetime import UTC, datetime, timedelta
from hashlib import sha256
import json
from typing import Any, Iterable
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.models.vessel import AISConflictRecord, AISRawObservation, AISSourceHealth
from app.services.vessel_aggregation_strategy import (
DEFAULT_STRATEGY,
load_strategy,
)
from app.services.vessel_types import normalize_vessel_type_name
VESSEL_AIS_SCHEMA = "vessel_ais"
DEFAULT_AGGREGATION_WINDOW_HOURS = 24
BARENTSWATCH_DELIVERY_MODE = "polling"
BARENTSWATCH_TRANSPORT = "http"
AISSTREAM_DELIVERY_MODE = "realtime_stream"
AISSTREAM_TRANSPORT = "websocket"
DELIVERY_MODE_PRIORITY = {
"realtime_stream": 40,
"batch_stream": 30,
"polling": 20,
"snapshot": 10,
}
DYNAMIC_FIELDS = ("lat", "lon", "sog", "cog", "heading", "nav_status")
CONFLICT_FIELDS = (
"name",
"callsign",
"imo",
"flag",
"vessel_type",
"vessel_type_name",
"length",
"width",
"draught",
)
def _json_default(value: Any) -> Any:
if isinstance(value, datetime):
return value.astimezone(UTC).isoformat()
return str(value)
def _stable_payload(value: Any) -> str:
return json.dumps(value, sort_keys=True, separators=(",", ":"), default=_json_default)
def _jsonable(value: Any) -> Any:
if isinstance(value, datetime):
return value.astimezone(UTC).isoformat()
if isinstance(value, dict):
return {str(key): _jsonable(item) for key, item in value.items()}
if isinstance(value, list):
return [_jsonable(item) for item in value]
return value
def _coerce_datetime(value: Any) -> datetime | None:
if isinstance(value, datetime):
return value if value.tzinfo else value.replace(tzinfo=UTC)
if isinstance(value, (int, float)):
timestamp = float(value)
if timestamp > 10_000_000_000:
timestamp /= 1000
return datetime.fromtimestamp(timestamp, UTC)
if isinstance(value, str) and value:
try:
parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
return parsed if parsed.tzinfo else parsed.replace(tzinfo=UTC)
except ValueError:
return None
return None
def build_observation_hash(
*,
source: str,
entity_key: str,
message_type: str | None,
observed_at: datetime,
normalized_payload: dict[str, Any],
source_message_id: str | None = None,
) -> str:
"""Build a deterministic idempotency key for one source-level AIS observation."""
if source_message_id:
basis = {
"source": source,
"entity_key": entity_key,
"source_message_id": source_message_id,
}
else:
basis = {
"source": source,
"entity_key": entity_key,
"message_type": message_type,
"observed_at": observed_at.astimezone(UTC).isoformat(),
"payload": normalized_payload,
}
return sha256(_stable_payload(basis).encode("utf-8")).hexdigest()
def build_field_conflict_candidates(
observations: Iterable[AISRawObservation],
fields: Iterable[str] = CONFLICT_FIELDS,
) -> list[dict[str, Any]]:
"""Return current field disagreements from raw observations without mutating state."""
candidates_by_field: dict[str, dict[str, Any]] = {}
for observation in observations:
payload = observation.normalized_payload or {}
for field in fields:
value = payload.get(field)
if value in (None, ""):
continue
field_candidates = candidates_by_field.setdefault(field, {})
field_candidates[observation.source] = value
conflicts = []
for field, candidates in sorted(candidates_by_field.items()):
unique_values = {_stable_payload(value) for value in candidates.values()}
if len(unique_values) <= 1:
continue
conflicts.append(
{
"field": field,
"candidates": candidates,
"status": "candidate",
}
)
return conflicts
def _payload_value(payload: dict[str, Any], field: str) -> Any:
value = payload.get(field)
return None if value in (None, "") else value
def _clean_text(value: Any) -> str | None:
if value in (None, ""):
return None
text = str(value).strip()
return text or None
def _raw_metadata_value(observation: AISRawObservation, field: str) -> Any:
raw_payload = observation.raw_payload or {}
metadata = raw_payload.get("MetaData") if isinstance(raw_payload, dict) else None
if not isinstance(metadata, dict):
return None
if field == "name":
return _clean_text(metadata.get("ShipName") or metadata.get("ship_name") or metadata.get("name"))
return None
def _delivery_priority(observation: AISRawObservation) -> int:
return DELIVERY_MODE_PRIORITY.get(str(observation.delivery_mode or ""), 0)
def _has_valid_position(payload: dict[str, Any]) -> bool:
try:
lat = float(payload.get("lat"))
lon = float(payload.get("lon"))
except (TypeError, ValueError):
return False
return -90 <= lat <= 90 and -180 <= lon <= 180
def _is_future_observation(observation: AISRawObservation, now: datetime) -> bool:
return observation.observed_at > now
def _strategy_source_rank(
source: str,
strategy: dict[str, Any],
) -> int:
priority = (strategy.get("vessel_ais") or {}).get("source_priority") or []
if source in priority:
return len(priority) - priority.index(source)
return 0
def _is_stream_stale(
observation: AISRawObservation,
*,
now: datetime,
strategy: dict[str, Any],
) -> bool:
delivery_mode = str(observation.delivery_mode or "")
freshness = (strategy.get("vessel_ais") or {}).get("freshness") or {}
if delivery_mode == "realtime_stream":
window = int(freshness.get("realtime_stream_seconds", 0) or 0)
else:
window = int(freshness.get("polling_seconds", 0) or 0)
if window <= 0:
return False
return (now - observation.observed_at).total_seconds() > window
def _select_position_observation(
observations: list[AISRawObservation],
*,
now: datetime,
strategy: dict[str, Any] | None = None,
) -> tuple[AISRawObservation | None, list[str]]:
strategy = strategy or DEFAULT_STRATEGY
rejected_flags: list[str] = []
fresh_candidates: list[AISRawObservation] = []
stale_candidates: list[AISRawObservation] = []
for observation in observations:
payload = observation.normalized_payload or {}
if not _has_valid_position(payload):
rejected_flags.append("invalid_position")
continue
if _is_future_observation(observation, now):
rejected_flags.append("future_timestamp")
continue
if _is_stream_stale(observation, now=now, strategy=strategy):
stale_candidates.append(observation)
rejected_flags.append("freshness_fallback")
continue
fresh_candidates.append(observation)
candidates = fresh_candidates or stale_candidates
if not candidates:
return None, sorted(set(rejected_flags))
candidates.sort(
key=lambda item: (
item.observed_at,
_delivery_priority(item),
_strategy_source_rank(item.source, strategy),
item.collected_at,
item.id or 0,
),
reverse=True,
)
return candidates[0], sorted(set(rejected_flags))
def _select_static_field(
observations: list[AISRawObservation],
field: str,
strategy: dict[str, Any] | None = None,
) -> tuple[Any, str | None, str | None]:
strategy = strategy or DEFAULT_STRATEGY
candidates = []
for observation in observations:
value = _payload_value(observation.normalized_payload or {}, field)
if value is None:
value = _raw_metadata_value(observation, field)
if value is None:
continue
candidates.append((observation, value))
if not candidates:
return None, None, None
field_rules = (strategy.get("vessel_ais") or {}).get("field_rules") or {}
rule = field_rules.get(field) or {"mode": "source_priority"}
mode = rule.get("mode")
if mode == "locked":
locked_source = rule.get("locked_source")
for observation, value in candidates:
if observation.source == locked_source:
return value, observation.source, "locked"
if mode in ("source_priority", "locked"):
priority = rule.get("source_priority") or (strategy.get("vessel_ais") or {}).get("source_priority") or []
ranked = sorted(
candidates,
key=lambda item: (
priority.index(item[0].source) if item[0].source in priority else len(priority) + 1,
-_delivery_priority(item[0]),
-(item[0].observed_at.timestamp() if item[0].observed_at else 0),
),
)
observation, value = ranked[0]
return value, observation.source, "source_priority"
if mode == "newest":
ranked = sorted(
candidates,
key=lambda item: (item[0].observed_at, _delivery_priority(item[0]), item[0].id or 0),
reverse=True,
)
observation, value = ranked[0]
return value, observation.source, "newest_observation"
# default / non_empty: prefer delivery mode priority, then newest
candidates.sort(
key=lambda item: (
_delivery_priority(item[0]),
item[0].observed_at,
item[0].collected_at,
item[0].id or 0,
),
reverse=True,
)
selected_observation, selected_value = candidates[0]
unique_values = {_stable_payload(value) for _, value in candidates}
reason = "delivery_mode_priority" if len(unique_values) > 1 else "non_empty_priority"
return selected_value, selected_observation.source, reason
def _build_source_summary(observations: list[AISRawObservation]) -> dict[str, dict[str, Any]]:
summary: dict[str, dict[str, Any]] = {}
for observation in observations:
source_summary = summary.setdefault(
observation.source,
{
"observation_count": 0,
"latest_observed_at": None,
"delivery_mode": observation.delivery_mode,
"transport": observation.transport,
"message_types": [],
},
)
source_summary["observation_count"] += 1
latest_observed_at = source_summary["latest_observed_at"]
if latest_observed_at is None or observation.observed_at > latest_observed_at:
source_summary["latest_observed_at"] = observation.observed_at
if observation.message_type and observation.message_type not in source_summary["message_types"]:
source_summary["message_types"].append(observation.message_type)
return summary
def _build_aggregated_vessel(
entity_key: str,
observations: list[AISRawObservation],
*,
now: datetime,
strategy: dict[str, Any] | None = None,
) -> dict[str, Any] | None:
strategy = strategy or DEFAULT_STRATEGY
position_observation, rejected_flags = _select_position_observation(
observations, now=now, strategy=strategy
)
if position_observation is None:
return None
payload = position_observation.normalized_payload or {}
mmsi = int(entity_key)
result: dict[str, Any] = {
"mmsi": mmsi,
"lat": float(payload["lat"]),
"lon": float(payload["lon"]),
"received_at": position_observation.observed_at,
"field_sources": {},
"selected_reasons": {},
"source_summary": _build_source_summary(observations),
"quality_flags": sorted(
set((position_observation.quality_flags or []) + rejected_flags)
),
"aggregation_strategy_version": int(strategy.get("version") or 0),
}
for field in DYNAMIC_FIELDS:
value = _payload_value(payload, field)
if field in ("lat", "lon") or value is not None:
result[field] = value
result["field_sources"][field] = position_observation.source
result["selected_reasons"][field] = "newest_observation"
for field in CONFLICT_FIELDS:
selected_value, selected_source, reason = _select_static_field(
observations, field, strategy=strategy
)
if selected_value is None:
continue
result[field] = selected_value
result["field_sources"][field] = selected_source
result["selected_reasons"][field] = reason
result["name"] = result.get("name") or f"MMSI {mmsi}"
result["vessel_type_name"] = result.get("vessel_type_name") or normalize_vessel_type_name(
result.get("vessel_type")
)
return result
async def _upsert_conflict_records(
db: AsyncSession,
entity_key: str,
observations: list[AISRawObservation],
aggregated: dict[str, Any],
) -> int:
conflicts = build_field_conflict_candidates(observations)
now = datetime.now(UTC)
for conflict in conflicts:
field = conflict["field"]
result = await db.execute(
select(AISConflictRecord)
.where(AISConflictRecord.target_schema == VESSEL_AIS_SCHEMA)
.where(AISConflictRecord.entity_key == entity_key)
.where(AISConflictRecord.field == field)
.limit(1)
)
record = result.scalar_one_or_none()
if record is None:
record = AISConflictRecord(
target_schema=VESSEL_AIS_SCHEMA,
entity_key=entity_key,
field=field,
)
db.add(record)
record.candidates = conflict["candidates"]
record.selected_source = (aggregated.get("field_sources") or {}).get(field)
record.selected_value = aggregated.get(field)
record.selected_reason = (aggregated.get("selected_reasons") or {}).get(field)
record.resolved_by = "system"
record.status = "open"
record.updated_at = now
return len(conflicts)
def _group_observations(observations: Iterable[AISRawObservation]) -> dict[str, list[AISRawObservation]]:
grouped: dict[str, list[AISRawObservation]] = {}
for observation in observations:
grouped.setdefault(str(observation.entity_key), []).append(observation)
return grouped
async def record_vessel_ais_observation(
db: AsyncSession,
*,
source: str,
normalized_payload: dict[str, Any],
raw_payload: dict[str, Any] | None = None,
delivery_mode: str,
transport: str,
message_type: str | None = "PositionReport",
source_message_id: str | None = None,
observed_at: datetime | None = None,
collected_at: datetime | None = None,
quality_flags: list[str] | None = None,
) -> AISRawObservation | None:
"""Insert one raw observation if the source-level fact has not already been stored."""
entity_key = str(normalized_payload["mmsi"])
collected_at = collected_at or datetime.now(UTC)
observed_at = (
_coerce_datetime(observed_at)
or _coerce_datetime(normalized_payload.get("received_at"))
or collected_at
)
normalized_json = _jsonable(normalized_payload)
raw_json = _jsonable(raw_payload or {})
observation_hash = build_observation_hash(
source=source,
entity_key=entity_key,
message_type=message_type,
observed_at=observed_at,
normalized_payload=normalized_json,
source_message_id=source_message_id,
)
existing_result = await db.execute(
select(AISRawObservation.id).where(AISRawObservation.observation_hash == observation_hash)
)
if existing_result.scalar_one_or_none() is not None:
return None
observation = AISRawObservation(
target_schema=VESSEL_AIS_SCHEMA,
source=source,
entity_key=entity_key,
delivery_mode=delivery_mode,
transport=transport,
message_type=message_type,
source_message_id=source_message_id,
observation_hash=observation_hash,
observed_at=observed_at,
collected_at=collected_at,
normalized_payload=normalized_json,
raw_payload=raw_json,
quality_flags=quality_flags or [],
)
db.add(observation)
return observation
async def aggregate_vessel_observations(
db: AsyncSession,
observations: Iterable[AISRawObservation],
*,
write_conflicts: bool = False,
strategy: dict[str, Any] | None = None,
) -> list[dict[str, Any]]:
strategy = strategy if strategy is not None else await _safe_load_strategy(db)
now = datetime.now(UTC)
vessels = []
for entity_key, entity_observations in _group_observations(observations).items():
aggregated = _build_aggregated_vessel(
entity_key, entity_observations, now=now, strategy=strategy
)
if aggregated is None:
continue
if write_conflicts:
aggregated["conflict_count"] = await _upsert_conflict_records(
db,
entity_key,
entity_observations,
aggregated,
)
else:
aggregated["conflict_count"] = len(build_field_conflict_candidates(entity_observations))
vessels.append(aggregated)
vessels.sort(key=lambda item: item.get("received_at") or datetime.min.replace(tzinfo=UTC), reverse=True)
return vessels
async def _safe_load_strategy(db: AsyncSession) -> dict[str, Any]:
"""Tolerate fake test sessions where load_strategy may misbehave."""
try:
return await load_strategy(db)
except Exception:
return DEFAULT_STRATEGY
async def get_aggregated_vessels(
db: AsyncSession,
*,
bbox: tuple[float, float, float, float] | None = None,
limit: int | None = None,
observed_since: datetime | None = None,
) -> list[dict[str, Any]]:
observed_since = observed_since or (
datetime.now(UTC) - timedelta(hours=DEFAULT_AGGREGATION_WINDOW_HOURS)
)
stmt = (
select(AISRawObservation)
.where(AISRawObservation.target_schema == VESSEL_AIS_SCHEMA)
.where(AISRawObservation.observed_at >= observed_since)
.order_by(AISRawObservation.observed_at.desc(), AISRawObservation.id.desc())
)
if limit and limit > 0:
stmt = stmt.limit(max(limit * 20, limit))
result = await db.execute(stmt)
if not hasattr(result, "scalars"):
return []
vessels = await aggregate_vessel_observations(db, result.scalars().all())
if bbox is not None:
lon_min, lat_min, lon_max, lat_max = bbox
vessels = [
vessel
for vessel in vessels
if lon_min <= float(vessel["lon"]) <= lon_max
and lat_min <= float(vessel["lat"]) <= lat_max
]
if limit and limit > 0:
return vessels[:limit]
return vessels
async def get_aggregated_vessel(db: AsyncSession, mmsi: int) -> dict[str, Any] | None:
observations = await get_vessel_raw_observations(db, mmsi, limit=1000)
vessels = await aggregate_vessel_observations(db, observations)
return vessels[0] if vessels else None
async def get_aggregated_vessel_track(
db: AsyncSession,
mmsi: int,
*,
cutoff: datetime,
) -> list[dict[str, Any]]:
result = await db.execute(
select(AISRawObservation)
.where(AISRawObservation.target_schema == VESSEL_AIS_SCHEMA)
.where(AISRawObservation.entity_key == str(mmsi))
.where(AISRawObservation.observed_at >= cutoff)
.order_by(AISRawObservation.observed_at.asc(), AISRawObservation.id.asc())
)
if not hasattr(result, "scalars"):
return []
points: list[dict[str, Any]] = []
seen: set[tuple[str, float, float, str]] = set()
for observation in result.scalars().all():
payload = observation.normalized_payload or {}
if not _has_valid_position(payload):
continue
lat = float(payload["lat"])
lon = float(payload["lon"])
key = (
observation.observed_at.isoformat(),
round(lat, 5),
round(lon, 5),
observation.source,
)
if key in seen:
continue
seen.add(key)
points.append(
{
"lat": lat,
"lon": lon,
"observed_at": observation.observed_at,
"source": observation.source,
"selected_reason": "track_timeline",
"quality_flags": observation.quality_flags or [],
}
)
return points
async def update_ais_source_health(
db: AsyncSession,
*,
source: str,
connection_state: str,
observed_count: int = 0,
last_seen_at: datetime | None = None,
last_success_at: datetime | None = None,
last_error: str | None = None,
lag_seconds: float | None = None,
) -> AISSourceHealth:
"""Upsert the health row for an AIS source."""
now = datetime.now(UTC)
health = await db.get(AISSourceHealth, source)
if health is None:
health = AISSourceHealth(source=source)
db.add(health)
health.connection_state = connection_state
health.last_seen_at = last_seen_at or health.last_seen_at
health.last_success_at = last_success_at or health.last_success_at
health.last_error = last_error
health.message_rate = float(observed_count)
health.lag_seconds = lag_seconds
health.updated_at = now
return health
async def count_unique_raw_vessel_mmsi(
db: AsyncSession,
*,
observed_since: datetime | None = None,
) -> int:
"""Count unique raw vessel MMSI values for HUD counts; never aggregates."""
from sqlalchemy import func as sa_func
unique_mmsi_stmt = (
select(AISRawObservation.entity_key)
.where(AISRawObservation.target_schema == VESSEL_AIS_SCHEMA)
.distinct()
)
if observed_since is not None:
unique_mmsi_stmt = unique_mmsi_stmt.where(
AISRawObservation.observed_at >= observed_since,
)
result = await db.execute(
select(sa_func.count()).select_from(unique_mmsi_stmt.subquery()),
)
return int(result.scalar() or 0)
async def get_vessel_raw_observations(
db: AsyncSession,
mmsi: int,
*,
limit: int = 100,
) -> list[AISRawObservation]:
result = await db.execute(
select(AISRawObservation)
.where(AISRawObservation.target_schema == VESSEL_AIS_SCHEMA)
.where(AISRawObservation.entity_key == str(mmsi))
.order_by(AISRawObservation.observed_at.desc(), AISRawObservation.id.desc())
.limit(limit)
)
return list(result.scalars().all())
async def get_vessel_conflict_records(
db: AsyncSession,
mmsi: int,
) -> list[AISConflictRecord]:
result = await db.execute(
select(AISConflictRecord)
.where(AISConflictRecord.target_schema == VESSEL_AIS_SCHEMA)
.where(AISConflictRecord.entity_key == str(mmsi))
.order_by(AISConflictRecord.updated_at.desc(), AISConflictRecord.id.desc())
)
return list(result.scalars().all())

View File

@@ -0,0 +1,109 @@
"""v5 vessel enrichment service.
Read-only side: `get_vessel_enrichment_bundle` is the only path the
aggregation/detail endpoints use. It never reaches out to third parties; it
just returns whatever the upsert side has already cached. Expired rows are
filtered out so old data never leaks back into the live UI.
"""
from __future__ import annotations
from datetime import UTC, datetime
from typing import Any
from sqlalchemy.ext.asyncio import AsyncSession
from app.models.vessel_enrichment import VesselMediaEnrichment, VesselProfileEnrichment
def _coerce_datetime(value: Any) -> datetime | None:
if value in (None, ""):
return None
if isinstance(value, datetime):
return value if value.tzinfo else value.replace(tzinfo=UTC)
if isinstance(value, (int, float)):
ts = float(value)
if ts > 10_000_000_000:
ts /= 1000
return datetime.fromtimestamp(ts, UTC)
if isinstance(value, str):
try:
parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
return parsed if parsed.tzinfo else parsed.replace(tzinfo=UTC)
except ValueError:
return None
return None
def _build_payload(record, *, now: datetime) -> dict[str, Any] | None:
if record is None:
return None
expires_at = record.expires_at
if isinstance(expires_at, datetime):
if expires_at.tzinfo is None:
expires_at = expires_at.replace(tzinfo=UTC)
if expires_at < now:
return None
return record.to_dict()
async def get_vessel_enrichment_bundle(db: AsyncSession, mmsi: int) -> dict[str, Any]:
now = datetime.now(UTC)
profile = await db.get(VesselProfileEnrichment, mmsi)
media = await db.get(VesselMediaEnrichment, mmsi)
return {
"mmsi": mmsi,
"profile": _build_payload(profile, now=now),
"media": _build_payload(media, now=now),
}
async def upsert_vessel_profile_enrichment(
db: AsyncSession,
*,
mmsi: int,
payload: dict[str, Any],
) -> dict[str, Any]:
record = await db.get(VesselProfileEnrichment, mmsi)
if record is None:
record = VesselProfileEnrichment(mmsi=mmsi)
db.add(record)
return _apply_upsert(record, payload)
async def upsert_vessel_media_enrichment(
db: AsyncSession,
*,
mmsi: int,
payload: dict[str, Any],
) -> dict[str, Any]:
record = await db.get(VesselMediaEnrichment, mmsi)
if record is None:
record = VesselMediaEnrichment(mmsi=mmsi)
db.add(record)
return _apply_upsert(record, payload)
def _apply_upsert(record, payload: dict[str, Any]) -> dict[str, Any]:
if not isinstance(payload, dict):
raise ValueError("enrichment payload must be an object")
body = payload.get("payload")
if body is not None and not isinstance(body, dict):
raise ValueError("payload.payload must be an object")
if body is not None:
record.payload = body
if "source" in payload and isinstance(payload["source"], str) and payload["source"].strip():
record.source = payload["source"].strip()
fetched_at = _coerce_datetime(payload.get("fetched_at"))
record.fetched_at = fetched_at or datetime.now(UTC)
record.expires_at = _coerce_datetime(payload.get("expires_at"))
confidence = payload.get("confidence")
if confidence is not None:
try:
record.confidence = float(confidence)
except (TypeError, ValueError):
record.confidence = None
if "reference_url" in payload:
ref = payload.get("reference_url")
record.reference_url = str(ref) if ref else None
return record.to_dict()

View File

@@ -0,0 +1,31 @@
"""Shared AIS vessel type helpers."""
from typing import Any
VESSEL_TYPE_NAMES = {
30: "Fishing",
35: "Military",
60: "Passenger",
70: "Cargo",
80: "Tanker",
}
def normalize_vessel_type_name(vessel_type: Any) -> str:
"""Map AIS numeric vessel type codes to display buckets."""
try:
type_code = int(float(vessel_type))
except (TypeError, ValueError):
return "Other"
if 70 <= type_code <= 79:
return "Cargo"
if 80 <= type_code <= 89:
return "Tanker"
if 60 <= type_code <= 69:
return "Passenger"
if type_code == 30:
return "Fishing"
if type_code == 35:
return "Military"
return VESSEL_TYPE_NAMES.get(type_code, "Other")

View File

@@ -1,11 +1,14 @@
"""Unit tests for data collectors"""
import pytest
from datetime import datetime
from unittest.mock import AsyncMock, MagicMock, patch
from unittest.mock import AsyncMock, patch
from app.core.datasource_defaults import DEFAULT_DATASOURCES
from app.services.credential_guides import DEFAULT_CREDENTIAL_GUIDES
from app.services.collectors.top500 import TOP500Collector
from app.services.collectors.base import BaseCollector, HTTPCollector
from app.services.collectors.registry import collector_registry
from app.services.datasource_connectivity import SUPPORTED_CREDENTIAL_PROVIDERS
from app.models.task import CollectionTask
class TestBaseCollector:
@@ -19,6 +22,31 @@ class TestBaseCollector:
assert collector.module == "L1"
assert collector.frequency_hours == 4
@pytest.mark.asyncio
async def test_update_phase_progress_tracks_phase_fields(self, mock_db_session):
"""Test phase-level progress updates independently from record totals"""
collector = TOP500Collector()
task = CollectionTask(datasource_id=1, status="running", phase="fetching")
collector._current_task = task
collector._db_session = mock_db_session
with patch.object(collector, "_publish_task_update", new=AsyncMock()) as publish:
await collector.update_phase_progress(
current=512,
total=1024,
unit="bytes",
message="Downloading dataset",
commit=True,
)
assert task.phase_progress == 50.0
assert task.phase_current == 512
assert task.phase_total == 1024
assert task.phase_unit == "bytes"
assert task.phase_message == "Downloading dataset"
mock_db_session.commit.assert_awaited_once()
publish.assert_awaited_once()
class TestTOP500Collector:
"""Tests for TOP500Collector"""
@@ -119,3 +147,30 @@ class TestHTTPCollector:
assert hasattr(collector, "parse_response")
assert callable(collector.fetch)
assert callable(collector.parse_response)
def test_aisstream_collector_is_registered():
collector = collector_registry.get("aisstream_vessels")
assert collector is not None
assert collector.data_type == "vessel_ais"
def test_supported_credential_collectors_have_guides_and_connectivity_provider():
missing: list[str] = []
for source, info in DEFAULT_DATASOURCES.items():
if not info.get("requires_credentials"):
continue
if info.get("credential_status") != "supported":
continue
provider = info.get("credential_provider")
if not provider:
missing.append(f"{source}: missing credential_provider")
continue
if provider not in DEFAULT_CREDENTIAL_GUIDES:
missing.append(f"{source}: missing credential guide for {provider}")
if provider not in SUPPORTED_CREDENTIAL_PROVIDERS:
missing.append(f"{source}: missing connectivity provider for {provider}")
assert missing == []

View File

@@ -0,0 +1,149 @@
"""End-to-end integration test for the custom WebSocket datasource runner.
Boots an in-process WebSocket server that mimics the bun mock AIS server
(`scripts/mock-ais-ws-server.ts`) and runs the real
`run_mapped_websocket_config` against it. Catches regressions where the
runner stops connecting, fails to extract the configured message path,
or quietly drops mapped records before broadcasting.
"""
from __future__ import annotations
import asyncio
import json
from contextlib import asynccontextmanager
from datetime import UTC, datetime
from types import SimpleNamespace
from unittest.mock import AsyncMock
import pytest
import websockets
from app.models.datasource_config import DataSourceConfig
from app.services import custom_datasource_runtime
from app.services.custom_datasource_runtime import run_mapped_websocket_config
def _make_payload(seq: int) -> str:
return json.dumps(
{
"type": "vessel",
"sequence": seq,
"data": {
"mmsi": str(999_000_000 + seq),
"name": f"MOCK VESSEL {seq:03d}",
"lat": 36.20 + seq * 0.001,
"lon": 14.20 + seq * 0.001,
"sog": 12.0,
"cog": 90.0,
"heading": 90,
"vessel_type": 70,
"vessel_type_name": "Cargo",
"received_at": datetime.now(UTC).isoformat(),
},
}
)
@asynccontextmanager
async def _mock_ais_server(emit_count: int):
received_subscribe: list[str] = []
async def handler(ws):
try:
try:
msg = await asyncio.wait_for(ws.recv(), timeout=0.5)
received_subscribe.append(msg)
except (asyncio.TimeoutError, websockets.ConnectionClosed):
pass
for seq in range(1, emit_count + 1):
await ws.send(_make_payload(seq))
await asyncio.sleep(0.01)
# keep the socket open briefly so the runner observes the messages
await asyncio.sleep(0.05)
except websockets.ConnectionClosed:
return
async with websockets.serve(handler, "127.0.0.1", 0) as server:
port = next(iter(server.sockets)).getsockname()[1]
yield port, received_subscribe
@pytest.mark.asyncio
async def test_websocket_runner_streams_from_live_mock(monkeypatch):
mapping = SimpleNamespace(
id=11,
version=3,
target_schema="vessel_ais",
mapping_json={
"source": {"items_path": "$"},
"fields": {
"mmsi": {"path": "$.mmsi", "type": "integer"},
"lat": {"path": "$.lat", "type": "float"},
"lon": {"path": "$.lon", "type": "float"},
"name": {"path": "$.name", "type": "string"},
"vessel_type": {"path": "$.vessel_type", "type": "integer", "default": None},
"vessel_type_name": {"path": "$.vessel_type_name", "type": "string", "default": None},
"sog": {"path": "$.sog", "type": "float", "default": None},
"cog": {"path": "$.cog", "type": "float", "default": None},
"heading": {"path": "$.heading", "type": "integer", "default": None},
"received_at": {"path": "$.received_at", "type": "datetime"},
},
},
)
class FakeResult:
def scalar_one_or_none(self):
return mapping
class FakeDB:
async def execute(self, _stmt):
return FakeResult()
persist = AsyncMock(return_value=1)
monkeypatch.setattr(custom_datasource_runtime, "persist_mapped_records", persist)
async with _mock_ais_server(emit_count=3) as (port, received_subscribe):
result = await run_mapped_websocket_config(
FakeDB(),
DataSourceConfig(
id=99,
name="mock_ais_ws",
source_type="websocket",
endpoint=f"ws://127.0.0.1:{port}",
auth_type="none",
headers={},
config={
"ws_message_path": "$.data",
"ws_subscribe_message": {
"type": "subscribe",
"anchor": {"lat": 36.2, "lon": 14.2},
"spread_km": 50,
"rate_hz": 1,
},
"debug_max_messages": 2,
"delivery_mode": "realtime_stream",
"ws_reconnect": False,
},
),
use_config_debug_max_messages=True,
)
assert result["status"] == "success"
assert result["messages_seen"] == 2
assert result["written_count"] == 2
assert result["mapped_count"] == 2
assert result["target_schema"] == "vessel_ais"
# subscribe message must reach the server unchanged
assert received_subscribe, "runner did not forward ws_subscribe_message"
parsed = json.loads(received_subscribe[0])
assert parsed["type"] == "subscribe"
assert parsed["anchor"] == {"lat": 36.2, "lon": 14.2}
assert parsed["rate_hz"] == 1
# mapped records carry the real MMSIs from the mock stream
persisted_records = []
for call in persist.await_args_list:
persisted_records.extend(call.kwargs["records"])
assert {record["mmsi"] for record in persisted_records} == {999_000_001, 999_000_002}
assert all(record["vessel_type"] == 70 for record in persisted_records)
assert all(record["vessel_type_name"] == "Cargo" for record in persisted_records)

View File

@@ -1,13 +1,18 @@
from types import SimpleNamespace
import pytest
from unittest.mock import AsyncMock
from httpx import ASGITransport, AsyncClient
from app.api.v1.datasource_config import get_ai_provider_client
from app.core.websocket import broadcaster as broadcaster_module
from app.core.security import get_current_user
from app.core.target_schema_registry import get_target_schema, list_target_schemas
from app.main import app
from app.models.user import User
from app.models.datasource_config import DataSourceConfig
from app.services import custom_datasource_runtime
from app.services.custom_datasource_runtime import run_mapped_websocket_config
from app.services.datasource_mapping import execute_mapping, persist_mapped_records, redact_for_llm
@@ -106,6 +111,130 @@ async def test_persist_mapped_records_writes_generic_records():
assert db.added[0].extra_data["mapping_version"] == 3
@pytest.mark.asyncio
async def test_persist_mapped_vessel_records_writes_raw_and_broadcasts(monkeypatch):
record_observation = AsyncMock(return_value=object())
update_health = AsyncMock()
broadcast_custom = AsyncMock()
monkeypatch.setattr(
"app.services.vessel_ais_aggregation.record_vessel_ais_observation",
record_observation,
)
monkeypatch.setattr(
"app.services.vessel_ais_aggregation.update_ais_source_health",
update_health,
)
monkeypatch.setattr(broadcaster_module, "broadcast_custom", broadcast_custom)
class FakeDB:
def __init__(self):
self.committed = False
async def commit(self):
self.committed = True
db = FakeDB()
count = await persist_mapped_records(
db,
datasource_name="mock_ais_ws",
datasource_config_id=42,
target_schema="vessel_ais",
records=[
{
"mmsi": 999000001,
"lat": 31.2,
"lon": 121.4,
"name": "MOCK VESSEL 001",
"received_at": "2026-05-01T00:00:00Z",
}
],
mapping_version=1,
delivery_mode="realtime_stream",
transport="websocket",
)
assert count == 1
assert db.committed is True
record_observation.assert_awaited_once()
assert record_observation.await_args.kwargs["source"] == "mock_ais_ws"
assert record_observation.await_args.kwargs["delivery_mode"] == "realtime_stream"
assert record_observation.await_args.kwargs["transport"] == "websocket"
update_health.assert_awaited_once()
broadcast_custom.assert_awaited_once()
assert broadcast_custom.await_args.args[0] == "vessels"
assert broadcast_custom.await_args.args[1]["vessels"][0]["mmsi_display"] == "999000001"
@pytest.mark.asyncio
async def test_custom_websocket_runner_maps_and_persists_vessel_records(monkeypatch):
mapping = SimpleNamespace(
id=7,
version=2,
target_schema="vessel_ais",
mapping_json={
"source": {"items_path": "$"},
"fields": {
"mmsi": {"path": "$.mmsi", "type": "integer"},
"lat": {"path": "$.lat", "type": "float"},
"lon": {"path": "$.lon", "type": "float"},
"name": {"path": "$.name", "type": "string"},
"received_at": {"path": "$.received_at", "type": "datetime"},
},
},
)
class FakeResult:
def scalar_one_or_none(self):
return mapping
class FakeDB:
async def execute(self, _stmt):
return FakeResult()
class FakeWebSocket:
async def __aenter__(self):
return self
async def __aexit__(self, *_args):
return None
async def send(self, _message):
return None
async def recv(self):
return (
'{"type":"vessel","data":{"mmsi":"999000001","name":"MOCK VESSEL 001",'
'"lat":31.2,"lon":121.4,"received_at":"2026-05-01T00:00:00Z"}}'
)
persist = AsyncMock(return_value=1)
monkeypatch.setattr(custom_datasource_runtime, "_connect_websocket", AsyncMock(return_value=FakeWebSocket()))
monkeypatch.setattr(custom_datasource_runtime, "persist_mapped_records", persist)
result = await run_mapped_websocket_config(
FakeDB(),
DataSourceConfig(
id=42,
name="mock_ais_ws",
source_type="websocket",
endpoint="ws://localhost:8787/ais",
auth_type="none",
headers={},
config={"ws_message_path": "$.data", "debug_max_messages": 1},
),
)
assert result["status"] == "success"
assert result["messages_seen"] == 1
assert result["written_count"] == 1
persist.assert_awaited_once()
assert persist.await_args.kwargs["datasource_name"] == "mock_ais_ws"
assert persist.await_args.kwargs["records"][0]["mmsi"] == 999000001
assert persist.await_args.kwargs["delivery_mode"] == "realtime_stream"
assert persist.await_args.kwargs["transport"] == "websocket"
@pytest.mark.asyncio
async def test_mapping_preview_api_uses_deterministic_engine():
def override_get_current_user():

View File

@@ -121,6 +121,25 @@ class TestCollectionTaskModel:
)
assert task.records_processed == 100
def test_task_with_phase_progress(self):
"""Test collection task phase-level progress fields"""
task = CollectionTask(
datasource_id=1,
status="running",
phase="fetching",
phase_progress=42.5,
phase_message="Downloading dataset",
phase_current=1024,
phase_total=4096,
phase_unit="bytes",
)
assert task.phase == "fetching"
assert task.phase_progress == 42.5
assert task.phase_message == "Downloading dataset"
assert task.phase_current == 1024
assert task.phase_total == 4096
assert task.phase_unit == "bytes"
def test_task_error_message(self):
"""Test collection task with error message"""
task = CollectionTask(

View File

@@ -0,0 +1,161 @@
"""Tests for the v4 vessel_ais aggregation strategy."""
from datetime import datetime, timedelta, timezone
from unittest.mock import AsyncMock
import pytest
from app.models.vessel import AISRawObservation
from app.services.vessel_aggregation_strategy import (
DEFAULT_STRATEGY,
StrategyValidationError,
validate_strategy,
)
from app.services.vessel_ais_aggregation import aggregate_vessel_observations
def _obs(*, source: str, mmsi: int, observed_at: datetime, **payload) -> AISRawObservation:
payload = {"mmsi": mmsi, "lat": 50.0, "lon": 10.0, **payload}
delivery_mode = "realtime_stream" if source == "aisstream_vessels" else "polling"
transport = "websocket" if source == "aisstream_vessels" else "http"
return AISRawObservation(
target_schema="vessel_ais",
source=source,
entity_key=str(mmsi),
delivery_mode=delivery_mode,
transport=transport,
message_type="PositionReport",
observation_hash=f"{source}:{mmsi}:{observed_at.isoformat()}",
observed_at=observed_at,
collected_at=observed_at,
normalized_payload=payload,
raw_payload=payload,
quality_flags=[],
)
def test_validate_rejects_unknown_field():
with pytest.raises(StrategyValidationError, match="unknown vessel_ais field"):
validate_strategy({"vessel_ais": {"field_rules": {"definitely_not_a_field": {"mode": "newest"}}}})
def test_validate_rejects_dynamic_lock_without_flag():
with pytest.raises(StrategyValidationError, match="allow_dynamic_lock"):
validate_strategy(
{
"vessel_ais": {
"field_rules": {"lat": {"mode": "source_priority"}},
"allow_dynamic_lock": False,
}
}
)
def test_validate_allows_dynamic_lock_with_flag():
normalized = validate_strategy(
{
"version": 0,
"vessel_ais": {
"field_rules": {"lat": {"mode": "source_priority", "source_priority": ["barentswatch_vessels"]}},
"allow_dynamic_lock": True,
},
}
)
assert normalized["vessel_ais"]["field_rules"]["lat"]["mode"] == "source_priority"
assert normalized["version"] == 1
def test_validate_increments_version():
first = validate_strategy({"version": 5, "vessel_ais": {}})
assert first["version"] == 6
@pytest.mark.asyncio
async def test_strategy_field_rule_promotes_specific_source(monkeypatch):
now = datetime(2026, 5, 4, 12, 0, tzinfo=timezone.utc)
obs_a = _obs(
source="aisstream_vessels",
mmsi=257123000,
observed_at=now,
name="AISSTREAM ONE",
vessel_type_name="Cargo",
)
obs_b = _obs(
source="barentswatch_vessels",
mmsi=257123000,
observed_at=now - timedelta(seconds=1),
name="BARENTSWATCH ONE",
vessel_type_name="Cargo",
)
strategy = {
"version": 7,
"vessel_ais": {
"source_priority": [],
"field_rules": {
"name": {"mode": "source_priority", "source_priority": ["barentswatch_vessels", "aisstream_vessels"]},
},
"freshness": {"realtime_stream_seconds": 0, "polling_seconds": 0},
"allow_dynamic_lock": False,
},
}
db = AsyncMock()
vessels = await aggregate_vessel_observations(
db,
[obs_a, obs_b],
write_conflicts=False,
strategy=strategy,
)
assert len(vessels) == 1
vessel = vessels[0]
assert vessel["name"] == "BARENTSWATCH ONE"
assert vessel["field_sources"]["name"] == "barentswatch_vessels"
assert vessel["selected_reasons"]["name"] == "source_priority"
assert vessel["aggregation_strategy_version"] == 7
@pytest.mark.asyncio
async def test_strategy_freshness_falls_back_to_polling_when_realtime_stale():
now = datetime(2026, 5, 4, 12, 0, tzinfo=timezone.utc)
stale_realtime = _obs(
source="aisstream_vessels",
mmsi=257123000,
observed_at=now - timedelta(hours=1),
lat=58.0,
lon=10.0,
)
fresh_polling = _obs(
source="barentswatch_vessels",
mmsi=257123000,
observed_at=now - timedelta(seconds=30),
lat=60.0,
lon=11.0,
)
strategy = {
"version": 1,
"vessel_ais": {
"source_priority": ["aisstream_vessels", "barentswatch_vessels"],
"field_rules": {},
"freshness": {"realtime_stream_seconds": 900, "polling_seconds": 7200},
"allow_dynamic_lock": False,
},
}
db = AsyncMock()
vessels = await aggregate_vessel_observations(
db,
[stale_realtime, fresh_polling],
write_conflicts=False,
strategy=strategy,
)
assert vessels[0]["field_sources"]["lat"] == "barentswatch_vessels"
assert vessels[0]["lat"] == 60.0
def test_default_strategy_is_stable():
assert DEFAULT_STRATEGY["vessel_ais"]["allow_dynamic_lock"] is False
assert "freshness" in DEFAULT_STRATEGY["vessel_ais"]

View File

@@ -0,0 +1,155 @@
"""Tests for v5 enrichment + conflict promote-to-rule."""
from datetime import datetime, timedelta, timezone
from unittest.mock import AsyncMock
import pytest
from app.models.vessel import AISConflictRecord, AISRawObservation
from app.models.vessel_enrichment import VesselMediaEnrichment, VesselProfileEnrichment
from app.services.vessel_ais_aggregation import aggregate_vessel_observations
from app.services.vessel_enrichment import (
_apply_upsert,
get_vessel_enrichment_bundle,
)
class _StoreSession:
"""Minimal AsyncSession stand-in that tracks mmsi-keyed enrichment + a strategy."""
def __init__(self, *, profile=None, media=None, conflicts=None):
self.profile = profile
self.media = media
self.conflicts = list(conflicts or [])
self.added: list = []
self.committed = False
async def get(self, model, key):
if model is VesselProfileEnrichment:
return self.profile if self.profile and self.profile.mmsi == key else None
if model is VesselMediaEnrichment:
return self.media if self.media and self.media.mmsi == key else None
return None
@pytest.mark.asyncio
async def test_enrichment_bundle_filters_expired_records():
now = datetime.now(timezone.utc)
fresh = VesselProfileEnrichment(
mmsi=257123000,
source="local_cache",
payload={"vessel_subtype": "Container"},
fetched_at=now - timedelta(hours=1),
expires_at=now + timedelta(days=7),
confidence=0.9,
)
expired_media = VesselMediaEnrichment(
mmsi=257123000,
source="vesselfinder",
payload={"images": ["https://example.com/a.jpg"]},
fetched_at=now - timedelta(days=30),
expires_at=now - timedelta(days=1),
)
db = _StoreSession(profile=fresh, media=expired_media)
bundle = await get_vessel_enrichment_bundle(db, 257123000)
assert bundle["profile"]["payload"]["vessel_subtype"] == "Container"
assert bundle["media"] is None
def test_apply_upsert_preserves_payload_and_metadata():
record = VesselProfileEnrichment(mmsi=257123000)
out = _apply_upsert(
record,
{
"source": "vesselfinder",
"payload": {"vessel_subtype": "Container", "operator": "Maersk"},
"expires_at": "2026-12-31T00:00:00Z",
"confidence": 0.85,
"reference_url": "https://www.vesselfinder.com/vessels/257123000",
},
)
assert out["payload"]["operator"] == "Maersk"
assert out["confidence"] == 0.85
assert record.reference_url == "https://www.vesselfinder.com/vessels/257123000"
assert record.expires_at is not None
assert record.expires_at.year == 2026
def _obs(*, source: str, mmsi: int, observed_at, **payload) -> AISRawObservation:
payload = {"mmsi": mmsi, "lat": 60.0, "lon": 5.0, **payload}
delivery_mode = "realtime_stream" if source == "aisstream_vessels" else "polling"
transport = "websocket" if source == "aisstream_vessels" else "http"
return AISRawObservation(
target_schema="vessel_ais",
source=source,
entity_key=str(mmsi),
delivery_mode=delivery_mode,
transport=transport,
message_type="PositionReport",
observation_hash=f"{source}:{mmsi}:{observed_at.isoformat()}",
observed_at=observed_at,
collected_at=observed_at,
normalized_payload=payload,
raw_payload=payload,
quality_flags=[],
)
@pytest.mark.asyncio
async def test_promoted_rule_wins_during_aggregation():
"""Simulate the strategy that conflict-promote-to-rule writes."""
now = datetime.now(timezone.utc)
obs_a = _obs(
source="aisstream_vessels",
mmsi=257111000,
observed_at=now,
name="STREAM NAME",
vessel_type_name="Cargo",
)
obs_b = _obs(
source="barentswatch_vessels",
mmsi=257111000,
observed_at=now - timedelta(seconds=1),
name="REST NAME",
vessel_type_name="Cargo",
)
promoted_strategy = {
"version": 99,
"vessel_ais": {
"source_priority": [],
"field_rules": {
"name": {"mode": "source_priority", "source_priority": ["barentswatch_vessels"]}
},
"freshness": {"realtime_stream_seconds": 0, "polling_seconds": 0},
"allow_dynamic_lock": False,
},
}
db = AsyncMock()
vessels = await aggregate_vessel_observations(
db,
[obs_a, obs_b],
write_conflicts=False,
strategy=promoted_strategy,
)
assert vessels[0]["name"] == "REST NAME"
assert vessels[0]["selected_reasons"]["name"] == "source_priority"
assert vessels[0]["aggregation_strategy_version"] == 99
def test_conflict_record_holds_selected_source():
"""Sanity: the promote-to-rule API reads selected_source from this column."""
record = AISConflictRecord(
target_schema="vessel_ais",
entity_key="257111000",
field="name",
candidates={"a": "X", "b": "Y"},
selected_source="barentswatch_vessels",
selected_value="Y",
selected_reason="delivery_mode_priority",
)
serialized = record.to_dict()
assert serialized["selected_source"] == "barentswatch_vessels"
assert serialized["field"] == "name"

View File

@@ -1,14 +1,23 @@
from datetime import datetime, timedelta, timezone
from unittest.mock import AsyncMock
import pytest
from httpx import ASGITransport, AsyncClient
from app.api.v1 import visualization
from app.api.v1.visualization import convert_vessels_to_geojson
from app.db.session import get_db
from app.main import app
from app.models.vessel import VesselPosition, VesselStatic
from app.models.vessel import AISRawObservation, VesselPosition, VesselStatic
from app.services import barentswatch
from app.services.collectors.aisstream import AISStreamCollector
from app.services.collectors.vessel_ais import VesselAISCollector
from app.services.vessel_ais_aggregation import (
aggregate_vessel_observations,
build_field_conflict_candidates,
build_observation_hash,
record_vessel_ais_observation,
)
def test_vessel_collector_transforms_barentswatch_like_records():
@@ -35,6 +44,380 @@ def test_vessel_collector_transforms_barentswatch_like_records():
assert records[0]["lat"] == pytest.approx(59.91)
def test_vessel_observation_hash_is_stable_for_same_payload():
observed_at = datetime(2026, 4, 30, 12, 0, tzinfo=timezone.utc)
payload = {
"mmsi": 257123000,
"lat": 59.91,
"lon": 10.73,
"received_at": observed_at,
}
first = build_observation_hash(
source="barentswatch_vessels",
entity_key="257123000",
message_type="PositionReport",
observed_at=observed_at,
normalized_payload=payload,
)
second = build_observation_hash(
source="barentswatch_vessels",
entity_key="257123000",
message_type="PositionReport",
observed_at=observed_at,
normalized_payload=dict(reversed(payload.items())),
)
assert first == second
assert len(first) == 64
@pytest.mark.asyncio
async def test_record_vessel_ais_observation_skips_existing_hash():
observed_at = datetime(2026, 4, 30, 12, 0, tzinfo=timezone.utc)
class _Result:
def scalar_one_or_none(self):
return 123
class _Session:
def __init__(self):
self.added = []
async def execute(self, _stmt):
return _Result()
def add(self, item):
self.added.append(item)
db = _Session()
observation = await record_vessel_ais_observation(
db,
source="barentswatch_vessels",
normalized_payload={
"mmsi": 257123000,
"lat": 59.91,
"lon": 10.73,
"received_at": observed_at,
},
delivery_mode="polling",
transport="http",
observed_at=observed_at.isoformat(),
)
assert observation is None
assert db.added == []
def test_build_field_conflict_candidates_from_raw_observations():
observations = [
AISRawObservation(
source="barentswatch_vessels",
normalized_payload={"name": "OSLO TRADER", "flag": "NO"},
),
AISRawObservation(
source="aisstream_vessels",
normalized_payload={"name": "OSLO TRADER II", "flag": "NO"},
),
]
conflicts = build_field_conflict_candidates(observations)
assert conflicts == [
{
"field": "name",
"candidates": {
"aisstream_vessels": "OSLO TRADER II",
"barentswatch_vessels": "OSLO TRADER",
},
"status": "candidate",
}
]
@pytest.mark.asyncio
async def test_aggregate_vessel_observations_prefers_realtime_and_records_conflict():
observed_at = datetime.now(timezone.utc) - timedelta(minutes=5)
class _Result:
def scalar_one_or_none(self):
return None
class _Session:
def __init__(self):
self.added = []
async def execute(self, _stmt):
return _Result()
def add(self, item):
self.added.append(item)
db = _Session()
observations = [
AISRawObservation(
id=1,
source="barentswatch_vessels",
entity_key="257123000",
delivery_mode="polling",
transport="http",
observed_at=observed_at,
collected_at=observed_at,
normalized_payload={
"mmsi": 257123000,
"name": "OSLO TRADER",
"lat": 59.91,
"lon": 10.73,
},
),
AISRawObservation(
id=2,
source="aisstream_vessels",
entity_key="257123000",
delivery_mode="realtime_stream",
transport="websocket",
observed_at=observed_at + timedelta(seconds=10),
collected_at=observed_at + timedelta(seconds=10),
normalized_payload={
"mmsi": 257123000,
"vessel_type": 79,
"lat": 59.92,
"lon": 10.74,
},
raw_payload={"MetaData": {"ShipName": "OSLO TRADER II "}},
),
]
vessels = await aggregate_vessel_observations(db, observations)
assert vessels[0]["lat"] == pytest.approx(59.92)
assert vessels[0]["field_sources"]["lat"] == "aisstream_vessels"
assert vessels[0]["name"] == "OSLO TRADER II"
assert vessels[0]["vessel_type_name"] == "Cargo"
assert vessels[0]["source_summary"]["aisstream_vessels"]["observation_count"] == 1
assert vessels[0]["source_summary"]["barentswatch_vessels"]["delivery_mode"] == "polling"
assert vessels[0]["conflict_count"] == 0
assert db.added == []
@pytest.mark.asyncio
async def test_vessel_collector_writes_raw_observations_only(monkeypatch):
collector = VesselAISCollector()
collector.update_progress = AsyncMock()
record_observation = AsyncMock()
update_health = AsyncMock()
broadcast_custom = AsyncMock()
monkeypatch.setattr(
"app.services.collectors.vessel_ais.record_vessel_ais_observation",
record_observation,
)
monkeypatch.setattr(
"app.services.collectors.vessel_ais.update_ais_source_health",
update_health,
)
monkeypatch.setattr(
"app.services.collectors.vessel_ais.broadcaster.broadcast_custom",
broadcast_custom,
)
class _Session:
def __init__(self):
self.added = []
self.committed = False
async def get(self, *_args):
return None
def add(self, item):
self.added.append(item)
async def execute(self, _stmt):
return None
async def commit(self):
self.committed = True
db = _Session()
observed_at = datetime(2026, 4, 30, 12, 0, tzinfo=timezone.utc)
saved = await collector._save_data(
db,
[
{
"mmsi": 257123000,
"name": "OSLO TRADER",
"lat": 59.91,
"lon": 10.73,
"received_at": observed_at,
}
],
)
assert saved == 1
assert db.committed is True
# BarentsWatch must funnel through the unified AIS pipeline only — no legacy writes.
assert not any(isinstance(item, VesselStatic) for item in db.added)
assert not any(isinstance(item, VesselPosition) for item in db.added)
record_observation.assert_awaited_once()
assert record_observation.await_args.kwargs["source"] == "barentswatch_vessels"
assert record_observation.await_args.kwargs["normalized_payload"]["mmsi"] == 257123000
update_health.assert_awaited_once()
broadcast_custom.assert_awaited_once()
assert broadcast_custom.await_args.args[0] == "vessels"
assert broadcast_custom.await_args.args[1]["action"] == "upsert"
assert broadcast_custom.await_args.args[1]["vessels"][0]["mmsi_display"] == "257123000"
def test_aisstream_collector_normalizes_position_report():
collector = AISStreamCollector()
records = collector.transform(
[
{
"MessageType": "PositionReport",
"MetaData": {
"MMSI": 257123000,
"ShipName": "OSLO TRADER ",
"time_utc": "2026-04-30T12:00:00Z",
},
"Message": {
"PositionReport": {
"Latitude": 59.91,
"Longitude": 10.73,
"Sog": 12.4,
"Cog": 214,
"TrueHeading": 215,
"NavigationalStatus": 0,
}
},
}
]
)
assert len(records) == 1
assert records[0]["mmsi"] == 257123000
assert records[0]["lat"] == pytest.approx(59.91)
assert records[0]["name"] == "OSLO TRADER"
assert records[0]["_message_type"] == "PositionReport"
def test_aisstream_collector_maps_ship_static_type_name():
collector = AISStreamCollector()
records = collector.transform(
[
{
"MessageType": "ShipStaticData",
"MetaData": {
"MMSI": 257123000,
"time_utc": "2026-04-30T12:00:00Z",
},
"Message": {
"ShipStaticData": {
"Name": "OSLO TRADER",
"Type": 79,
"CallSign": "LAAB",
}
},
}
]
)
assert len(records) == 1
assert records[0]["vessel_type"] == 79
assert records[0]["vessel_type_name"] == "Cargo"
@pytest.mark.asyncio
async def test_aisstream_collector_writes_only_raw_observations(monkeypatch):
collector = AISStreamCollector()
collector.update_progress = AsyncMock()
record_observation = AsyncMock(return_value=object())
update_health = AsyncMock()
monkeypatch.setattr(
"app.services.collectors.aisstream.record_vessel_ais_observation",
record_observation,
)
monkeypatch.setattr(
"app.services.collectors.aisstream.update_ais_source_health",
update_health,
)
class _Session:
def __init__(self):
self.added = []
self.committed = False
def add(self, item):
self.added.append(item)
async def commit(self):
self.committed = True
db = _Session()
saved = await collector._save_data(
db,
[
{
"mmsi": 257123000,
"lat": 59.91,
"lon": 10.73,
"received_at": datetime(2026, 4, 30, 12, 0, tzinfo=timezone.utc),
"_message_type": "PositionReport",
}
],
)
assert saved == 1
assert db.added == []
assert db.committed is True
record_observation.assert_awaited_once()
assert record_observation.await_args.kwargs["source"] == "aisstream_vessels"
update_health.assert_awaited_once()
@pytest.mark.asyncio
async def test_aisstream_stream_record_broadcasts_vessel_delta(monkeypatch):
collector = AISStreamCollector()
record_observation = AsyncMock(return_value=object())
update_health = AsyncMock()
broadcast_custom = AsyncMock()
monkeypatch.setattr(
"app.services.collectors.aisstream.record_vessel_ais_observation",
record_observation,
)
monkeypatch.setattr(
"app.services.collectors.aisstream.update_ais_source_health",
update_health,
)
monkeypatch.setattr(
"app.services.collectors.aisstream.broadcaster.broadcast_custom",
broadcast_custom,
)
class _Session:
async def commit(self):
pass
created = await collector._save_stream_record(
_Session(),
{
"mmsi": 257123000,
"lat": 59.91,
"lon": 10.73,
"cog": 214,
"received_at": datetime(2026, 4, 30, 12, 0, tzinfo=timezone.utc),
},
)
assert created is True
record_observation.assert_awaited_once()
broadcast_custom.assert_awaited_once()
assert broadcast_custom.await_args.args[0] == "vessels"
assert broadcast_custom.await_args.args[1]["action"] == "upsert"
assert broadcast_custom.await_args.args[1]["vessels"][0]["mmsi_display"] == "257123000"
def test_barentswatch_reads_credentials_from_zshrc(tmp_path):
zshrc = tmp_path / ".zshrc"
zshrc.write_text(
@@ -106,6 +489,39 @@ def test_convert_vessels_to_geojson():
assert payload["features"][0]["properties"]["vessel_type_name"] == "Cargo"
def test_convert_vessels_to_geojson_dedupes_mmsi_rows():
first = VesselPosition(
mmsi=257123000,
lat=59.91,
lon=10.73,
received_at=datetime(2026, 4, 28, 1, 0, tzinfo=timezone.utc),
)
duplicate = VesselPosition(
mmsi=257123000,
lat=60.01,
lon=10.83,
received_at=datetime(2026, 4, 28, 1, 0, tzinfo=timezone.utc),
)
other = VesselPosition(
mmsi=257456000,
lat=60.3,
lon=5.3,
received_at=datetime(2026, 4, 28, 0, 59, tzinfo=timezone.utc),
)
payload = convert_vessels_to_geojson(
[
(first, VesselStatic(mmsi=257123000, name="OSLO TRADER")),
(duplicate, VesselStatic(mmsi=257123000, name="OSLO TRADER DUP")),
(other, VesselStatic(mmsi=257456000, name="BERGEN FERRY")),
]
)
mmsis = [feature["properties"]["mmsi"] for feature in payload["features"]]
assert mmsis == [257123000, 257456000]
assert payload["features"][0]["geometry"]["coordinates"] == [10.73, 59.91]
@pytest.mark.asyncio
async def test_vessels_geojson_endpoint_filters_type_and_bbox():
now = datetime(2026, 4, 28, 1, 0, tzinfo=timezone.utc)
@@ -137,7 +553,7 @@ async def test_vessels_geojson_endpoint_filters_type_and_bbox():
async with AsyncClient(transport=transport, base_url="http://test") as client:
response = await client.get(
"/api/v1/visualization/geo/vessels",
params={"bbox": "0,50,20,70", "type": "cargo"},
params={"bbox": "0,50,20,70", "type": "cargo", "limit": 0},
)
assert response.status_code == 200
@@ -147,3 +563,113 @@ async def test_vessels_geojson_endpoint_filters_type_and_bbox():
assert data["stats"]["by_type"]["Cargo"] == 1
finally:
app.dependency_overrides.clear()
@pytest.mark.asyncio
async def test_vessels_geojson_merges_raw_and_legacy_sources(monkeypatch):
now = datetime(2026, 4, 28, 1, 0, tzinfo=timezone.utc)
monkeypatch.setattr(
visualization,
"get_aggregated_vessels",
AsyncMock(
return_value=[
{
"mmsi": 1,
"lat": 59.9,
"lon": 10.7,
"received_at": now,
"name": "AISSTREAM SHIP",
"vessel_type_name": "Cargo",
"source_summary": {"aisstream_vessels": {"message_types": ["PositionReport"]}},
}
]
),
)
rows = [
(
VesselPosition(mmsi=1, lat=60.0, lon=10.8, received_at=now),
VesselStatic(mmsi=1, name="LEGACY DUP", vessel_type_name="Cargo"),
),
(
VesselPosition(mmsi=2, lat=60.3, lon=5.3, received_at=now),
VesselStatic(mmsi=2, name="BARENTSWATCH ONLY", vessel_type_name="Passenger"),
),
]
class _Result:
def all(self):
return rows
class _FakeSession:
async def execute(self, _query):
return _Result()
async def override_get_db():
yield _FakeSession()
app.dependency_overrides[get_db] = override_get_db
transport = ASGITransport(app=app)
try:
async with AsyncClient(transport=transport, base_url="http://test") as client:
response = await client.get("/api/v1/visualization/geo/vessels")
assert response.status_code == 200
data = response.json()
names = {feature["properties"]["mmsi"]: feature["properties"]["name"] for feature in data["features"]}
assert data["count"] == 2
assert names == {1: "AISSTREAM SHIP", 2: "BARENTSWATCH ONLY"}
assert data["diagnostics"]["legacy_backfilled_mmsi"] == 1
finally:
app.dependency_overrides.clear()
@pytest.mark.asyncio
async def test_vessel_name_fallbacks_reports_mmsi_display_names(monkeypatch):
now = datetime(2026, 4, 28, 1, 0, tzinfo=timezone.utc)
monkeypatch.setattr(
visualization,
"get_aggregated_vessels",
AsyncMock(
return_value=[
{
"mmsi": 257123000,
"lat": 59.9,
"lon": 10.7,
"received_at": now,
"name": "MMSI 257123000",
"vessel_type_name": "Other",
"source_summary": {
"aisstream_vessels": {
"latest_observed_at": now,
"message_types": ["PositionReport"],
}
},
}
]
),
)
class _Result:
def all(self):
return []
class _FakeSession:
async def execute(self, _query):
return _Result()
async def override_get_db():
yield _FakeSession()
app.dependency_overrides[get_db] = override_get_db
transport = ASGITransport(app=app)
try:
async with AsyncClient(transport=transport, base_url="http://test") as client:
response = await client.get("/api/v1/visualization/vessels/name-fallbacks")
assert response.status_code == 200
data = response.json()
assert data["count"] == 1
assert data["items"][0]["mmsi"] == "257123000"
assert data["items"][0]["message_types"] == ["PositionReport"]
finally:
app.dependency_overrides.clear()

View File

@@ -292,6 +292,9 @@ async def test_visualization_geo_summary_returns_counts(monkeypatch):
def scalar(self):
return self._scalar_value
def all(self):
return list(self._rows)
def scalars(self):
class _Scalars:
def __init__(self, rows):
@@ -304,13 +307,18 @@ async def test_visualization_geo_summary_returns_counts(monkeypatch):
class _FakeSession:
async def execute(self, query):
query_text = str(query)
query_text = str(query).lower()
if "bgp_incidents" in query_text:
return _ScalarResult(scalar_value=2)
if "bgp_anomalies" in query_text:
return _ScalarResult(scalar_value=3)
if "ais_raw_observations" in query_text or "vessel_position" in query_text:
return _ScalarResult(rows=[])
return _ScalarResult(rows=records)
async def get(self, *_args, **_kwargs):
return None
async def override_get_db():
yield _FakeSession()

View File

@@ -0,0 +1,46 @@
import pytest
from app.core.websocket.manager import ConnectionManager
class FakeWebSocket:
def __init__(self):
self.accepted = False
self.sent = []
self.closed = False
async def accept(self):
self.accepted = True
async def send_json(self, message):
self.sent.append(message)
async def close(self):
self.closed = True
@pytest.mark.asyncio
async def test_channel_subscribers_receive_channel_broadcasts():
manager = ConnectionManager()
socket = FakeWebSocket()
await manager.connect(socket, "user-1")
manager.subscribe(socket, ["dashboard"])
await manager.broadcast({"type": "data_frame", "channel": "dashboard"}, channel="dashboard")
assert socket.accepted is True
assert socket.sent == [{"type": "data_frame", "channel": "dashboard"}]
@pytest.mark.asyncio
async def test_disconnect_removes_channel_subscriptions():
manager = ConnectionManager()
socket = FakeWebSocket()
await manager.connect(socket, "user-1")
manager.subscribe(socket, ["dashboard"])
manager.disconnect(socket, "user-1")
await manager.broadcast({"type": "data_frame", "channel": "dashboard"}, channel="dashboard")
assert socket.sent == []
assert "dashboard" not in manager.channel_subscriptions

View File

@@ -86,6 +86,12 @@ CREATE TABLE collection_tasks (
id BIGSERIAL PRIMARY KEY,
datasource_id INTEGER NOT NULL REFERENCES data_sources(id) ON DELETE CASCADE,
status task_status NOT NULL DEFAULT 'pending',
phase VARCHAR(30) DEFAULT 'queued',
phase_progress FLOAT,
phase_message VARCHAR(255),
phase_current BIGINT,
phase_total BIGINT,
phase_unit VARCHAR(30),
started_at TIMESTAMP WITH TIME ZONE,
completed_at TIMESTAMP WITH TIME ZONE,
records_processed INTEGER DEFAULT 0,

View File

@@ -8,6 +8,9 @@ services:
args:
PYTHON_IMAGE: ${PYTHON_IMAGE:-python:3.14-slim}
UV_IMAGE: ${UV_IMAGE:-ghcr.io/astral-sh/uv:latest}
env_file:
- ./aiprovider/.env
- ${PLANET_AI_PROVIDER_RUNTIME_ENV_FILE:-./aiprovider/.env}
container_name: planet_aiprovider
ports:
- "8010:8010"

View File

@@ -10,6 +10,7 @@ services:
UV_IMAGE: ${UV_IMAGE:-ghcr.io/astral-sh/uv:latest}
env_file:
- ./aiprovider/.env
- ${PLANET_AI_PROVIDER_RUNTIME_ENV_FILE:-./aiprovider/.env}
container_name: planet_aiprovider
ports:
- "8010:8010"

View File

@@ -8,6 +8,118 @@ This project follows the repository versioning rule:
- `improvement` -> `+0.0.1`bugfix + 小功能混合)
- `bugfix` -> `+0.0.1`
## [0.48.0] — 2026-05-07
Released: 2026-05-07
### ✨ Highlights
- 自定义数据源新增 REST / WebSocket 映射运行时,并提供本地 AIS mock WebSocket用于实时船只 upsert 链路验证。
- AIS 原始观测、聚合策略、字段来源、冲突记录与船舶 enrichment 继续完善Earth 船只实时展示链路更接近生产数据形态。
- Earth 全球态势 summary 改为轻量 SQL 聚合,并在卫星 current 异常时回退到最近有效 TLE 批次,避免统计接口被大规模明细读取拖慢。
### 🔧 Improvements
- 修复 `/geo/summary``/geo/satellites` 在大表下加载慢或超时的问题,并补充 `collected_data` 与 AIS raw 相关索引。
- WebSocket 管理器支持匿名连接、频道订阅清理和更稳的连接生命周期测试,前端 WebSocket candidates / fallback 更可靠。
- `planet.sh` 强化端口释放、端口诊断和前端启动流程mock AIS server 提供 Bun 脚本入口。
---
## [0.47.0] — 2026-04-30
Released: 2026-04-30
### ✨ Highlights
- 新增 AISStream WebSocket 船只采集器,并将 AIS 多源数据写入原始观测层,由聚合接口统一去重、合并和解释字段来源。
- 设置页新增 AISStream API Key、采集范围 preset、运行状态、连接验证和凭证教程入口让全球 AIS 采集链路可配置、可观察。
- Earth 船只图层默认不再限制 5000 艘,并统一 marker 颜色、详情卡、hover 和搜索结果的船型归一化显示。
### 🔧 Improvements
- 聚合接口新增 `field_sources``selected_reasons``source_summary``quality_flags` 和冲突记录调试接口,动态字段默认优先采用更新的实时流观测。
- AISStream 标准化支持 `MetaData.ShipName` 船名兜底,并将 AIS 数字船型映射为 Cargo / Tanker / Passenger / Fishing / Military。
- 将仓库 docs 技能改为通用文档工作流Planet 专属白名单、双语、裸文件标题和凭证教程规则迁移到 `docs/documentation-coverage-rules.md`
- 更新 AIS v4/v5 TODO 与计划文档,明确后续聚合策略配置、船舶资料 enrichment 和媒体缓存边界。
---
## [0.46.3] — 2026-04-30
Released: 2026-04-30
### 🐛 Fixes
- 优化 Starlink footprint 显示后的地球拖拽性能,避免旋转地球时每帧重建 footprint 大网格,同时保持现有视觉效果不变。
- 恢复点击线缆后的呼吸透明度动画,让 locked / hover 线缆重新使用既有 pulse 配置。
---
## [0.46.2] — 2026-04-30
Released: 2026-04-30
### 🐛 Fixes
- 修复 Earth 启动时高清材质、云图和图层可见性绕过 `startupPriority` 的问题,统一由启动队列按文档顺序加载。
- 修复保存为关闭的高清材质/图层仍会先加载再关闭的问题,并保持海陆基座作为国界线图层的常驻底图。
- 修复搜索跳转会误关媒体面板、船只轨迹末端不贴合当前船只、Iridium footprint 被地表层遮挡等 Earth 交互问题。
### 📝 Documentation
- 更新 Earth 图层顺序、样式参考、使用手册和 AIS 聚合计划,补齐中英文说明与后续接入策略。
---
## [0.46.1] — 2026-04-30
Released: 2026-04-30
### 🐛 Fixes
- 修复新增 technical docs 文件存在但未进入 Docs 前端白名单时,侧栏不显示且 Markdown 链接无法解析到 `/docs/<slug>` 的问题。
- 补齐数据源/采集器连接验证与 Earth Interactable 使用说明的英文文档,保证公开 Docs 切换 EN 时同名页面可访问。
- 清理中英文 technical docs 中裸 `.md` 文件名链接标题,改为面向读者的语义标题。
### 📝 Documentation
- 将 Docs 前端白名单、公开文档双语配对、裸文件名链接标题三项检查写入 Claude 与 Codex 的 docs 技能流程。
---
## [0.46.0] — 2026-04-30
Released: 2026-04-30
### ✨ Highlights
- Earth 新增通用 Interactable 图标层船只、算力中心、BGP 事件与观测站统一使用批量 Points、屏幕拾取、状态 glow 和状态缩放。
- BGP 事件保留向外扩散圈,观测站保留雷达扫描层,并与 Interactable 主图标解耦到稳定的地表渲染层级。
- 登陆点回归黄色球形 Sprite贴近海缆层级并保持更稳定的地表显示和遮挡表现。
### 🔧 Improvements
- 新增 SVG asset 到 canvas texture 的 Interactable 资产加载路径,支持统一图标资源、缓存和可选染色。
- 同坐标 Interactable 自动做地表切向避让,降低重叠物件无法选择的问题。
- 优化 Earth toolbar 初始尺寸注入,避免首次显示原始尺寸后再跳到缩放尺寸。
- 补充 Interactable 计划、使用说明、图层顺序和 Earth 前端上下文文档。
- 修复船只 hover/locked 状态仅发光但放大反馈不明显的问题,将已有状态缩放接入通用图标层。
---
## [0.45.0] — 2026-04-29
### ✨ Highlights
- 采集任务新增阶段级量化进度,`fetching` 可展示百分比、阶段说明和字节下载量。
- AI Provider 启动链路支持从 `aiprovider/.env``~/.zshrc` 注入运行期配置,并避免密钥/模型变化触发镜像重建。
- AI Provider Docker build context 收敛到服务必需文件,`uv sync` 接入 BuildKit 缓存以减少重复下载。
### 🔧 Improvements
- IPtoASN、OpenGeoFeed、NRO delegated 下载型采集器接入真实字节进度上报。
- 数据源页、采集中任务弹窗和任务历史页展示阶段摘要,并在 tooltip 中保留完整进度细节。
- 调整 Earth 船只默认高度偏移,进一步贴近地表展示。
---
## [0.44.2] — 2026-04-29
### 📝 Documentation
- 补充 Earth 船只图层技术文档,记录分桶 `THREE.Points` 批量渲染、同尺寸交互 overlay 和屏幕空间 picking 的设计约束。
- 同步 Earth 渲染图层顺序和样式参考,明确 AIS 船只 renderOrder、depthTest、图标尺寸、航向分桶与 hover 命中半径。
- 更新船只渲染性能计划状态,标注 `0.44.1` 已落地的实现与后续全球 AIS / LOD 演进方向。
---
## [0.44.1] — 2026-04-29
### 🐛 Fixes

View File

@@ -0,0 +1,117 @@
# Documentation Coverage Rules
This file contains Planet-specific documentation coverage rules. Documentation skills and agents should read this file before deciding which docs to update. Keep tool-specific workflow in skills; keep product and repository rules here.
## Scope Rules
- User-visible workflow changes must update `docs/technical/zh/manual.md` and usually `docs/technical/zh/quickstart.md`.
- If an English counterpart exists for user-facing docs such as `manual.md` or `quickstart.md`, update `docs/technical/en/...` enough that it does not contradict the Chinese source.
- Control console page responsibility changes must update `docs/technical/zh/frontend-admin-frontend-context.md`.
- Earth frontend behavior changes must update `docs/technical/zh/earth-frontend-context.md`.
- Earth layer additions, `renderOrder`, altitude/radius offsets, depth strategy, pointer picking, legend modes, or layer panel/startup ordering must update `docs/technical/zh/earth-render-layer-order.md`.
- Earth layer visual style or legend symbol/color semantics should also update `docs/technical/zh/earth-layer-style-reference.md` when that reference is affected.
- Collector, datasource, credential, settings, connectivity, scheduler, or API changes must update the relevant backend docs, especially `docs/technical/zh/backend-collectors.md` and any datasource/settings-specific doc.
- When a change turns an old plan assumption into current behavior, update the relevant `docs/plans/*.md` with a status note instead of leaving contradictory instructions.
- Search docs for stale terms introduced by the change, for example old tab names, old route responsibilities, obsolete auth assumptions, or renamed UI labels.
## Public Docs Rules
- If adding a new technical document, add it to `docs/technical/zh/README.md` when it should be discoverable from the technical docs index.
- If a technical document should be visible in the public Docs page or linked from a technical README, register it in `frontend/src/pages/Docs/docs-content.ts` under `DOCS_METADATA`. Files under `docs/technical/{zh,en}/` are not automatically routable.
- For every public technical doc, keep the bilingual file pair in sync by filename: `docs/technical/zh/<name>.md` and `docs/technical/en/<name>.md`. If content is intentionally Chinese-only or English-only, state that intentionally in the final note.
- Public docs should use readable link text, not raw filenames such as `manual.md`.
## Credential Collector Rules
- Any built-in collector marked `requires_credentials: true` and `credential_status: supported` must have:
- a `credential_provider` in `backend/app/core/datasource_defaults.py`;
- a default credential guide in `backend/app/services/credential_guides.py`;
- a supported connectivity provider in `backend/app/services/datasource_connectivity.py`;
- settings UI guidance or a credential form in `frontend/src/pages/Settings/Settings.tsx`;
- a regression test that fails if the guide/provider is missing.
## Recommended Checks
Run the checks that match the affected docs.
### Duplicate Bilingual Docs
```bash
python - <<'PY'
from pathlib import Path
same = []
for en in sorted(Path("docs/technical/en").glob("*.md")):
zh = Path("docs/technical/zh") / en.name
if zh.exists() and en.read_text() == zh.read_text():
same.append(en.name)
if same:
raise SystemExit("identical en/zh docs: " + ", ".join(same))
print("no identical en/zh docs")
PY
```
### Language-Less Technical Links
```bash
rg -n "/home/ray/dev/linkong/planet/docs/technical/(?!zh|en)" docs/technical/zh --pcre2
```
This should return no matches.
### Public Docs Registry
```bash
python - <<'PY'
import re
from pathlib import Path
metadata = Path("frontend/src/pages/Docs/docs-content.ts").read_text()
known = set(re.findall(r"'([^']+\.md)':\s*\{", metadata))
known.add("README.md")
missing = []
for readme in [Path("docs/technical/zh/README.md"), Path("docs/technical/en/README.md")]:
if not readme.exists():
continue
for href in re.findall(r"\]\(([^)]+\.md)\)", readme.read_text()):
path = Path(href)
if "docs/technical/" not in href:
continue
filename = path.name
if filename not in known:
missing.append(f"{readme}: {filename}")
if missing:
raise SystemExit("docs README links missing DOCS_METADATA: " + ", ".join(missing))
print("docs README links are whitelisted")
PY
```
### Public Bilingual Pairs
```bash
python - <<'PY'
import re
from pathlib import Path
metadata = Path("frontend/src/pages/Docs/docs-content.ts").read_text()
filenames = sorted(set(re.findall(r"'([^']+\.md)':\s*\{", metadata)) - {"README.md"})
missing = []
for filename in filenames:
for lang in ("zh", "en"):
path = Path("docs/technical") / lang / filename
if not path.exists():
missing.append(str(path))
if missing:
raise SystemExit("missing bilingual docs: " + ", ".join(missing))
print("public docs have zh/en file pairs")
PY
```
### Raw Filename Link Titles
```bash
rg -n "\[[^]]+\.md\]\(" docs/technical/zh docs/technical/en
```
This should return no matches for polished public docs.

View File

@@ -26,6 +26,8 @@
- [earth-news-source-configuration-and-collector-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-news-source-configuration-and-collector-plan.md)
- [earth-news-cruise-summary-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-news-cruise-summary-plan.md)
- [earth-vessel-rendering-performance-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-vessel-rendering-performance-plan.md)
- [AIS 多源采集、冲突记录与聚合接口计划](/home/ray/dev/linkong/planet/docs/plans/earth-vessel-ais-aggregation-plan.md)
- [earth-interactable-layer-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-interactable-layer-plan.md)
- [frontend-public-docs-site-plan.md](/home/ray/dev/linkong/planet/docs/plans/frontend-public-docs-site-plan.md)
- [frontend-ai-playground-development-plan.md](/home/ray/dev/linkong/planet/docs/plans/frontend-ai-playground-development-plan.md)
- [ue5-mvp-fused-plan.md](/home/ray/dev/linkong/planet/docs/plans/ue5-mvp-fused-plan.md)

View File

@@ -0,0 +1,384 @@
# Custom Source Live Mock 计划
**状态**:实施中
**创建日期**2026-05-01
**任务名**`Custom Source Live Mock`
**核心目标**:把自定义源升级为同时支持 REST 与 WebSocket 的可映射采集入口,并提供本地 AIS mock WebSocket 服务,用于验证 Earth 船只实时新增与 upsert 链路。
## 背景
真实 AIS 接口变化频率不可控,无法稳定验证 Earth 页面“不刷新也能看到新船只”的实时链路。当前系统已经有自定义源基础设施:
- `datasource_configs` 保存 endpoint、auth、headers、config。
- `datasource_mapping_templates` 保存目标 schema 的确定性映射模板。
- `run-mapped` 支持保存后的自定义 REST 源通过 active mapping 写入目标数据。
但现有能力主要面向 REST sample 和批量 mapping缺少以下能力
- 自定义源不能明确选择 `REST``WebSocket` 采集模式。
- WebSocket 长连接、订阅消息、重连、消息路径提取还没有通用 runtime。
- `vessel_ais` 自定义数据写入后需要进入 AIS raw observation 和 `vessels` WS channel才能真实验证 Earth 实时 upsert。
- 删除自定义源时没有清晰的数据清理选项。
- 设置中心里“采集调度 / 凭证 / 自定义源”入口混杂,用户很难判断该在哪里配置。
## 已确认决策
| 项目 | 决策 |
|-----|------|
| 计划名称 | `Custom Source Live Mock` |
| 自定义源传输类型 | 支持 `REST``WebSocket` |
| 采集写入方式 | 先映射到目标 schema再由 destination handler 写入 |
| AIS mock 目标 | 优先打通 `vessel_ais`,验证 Earth 船只实时新增和同 MMSI upsert |
| mock 服务 runtime | 使用 `bun` 启动本地 mock WS 服务 |
| 凭证配置 | 支持 headers、bearer、api key、basic并保留 query/header API key 位置配置 |
| 删除策略 | 删除自定义源时允许选择是否删除该源写入的数据 |
| 合并语义 | 自定义源必须选择“合并到哪个内置数据”,作为内置源的补充数据进入同一聚合链路 |
| UI 方向 | 自定义源创建和维护放在“配置中心 > 采集器设置”的采集器下拉框内联入口;数据源页保留总览与运行控制 |
## 范围
### 本阶段要做
- 自定义源可选择 `REST``WebSocket`
- 自定义源支持请求头、凭证、query params、body、WS subscribe message。
- WebSocket 自定义源支持长连接、重连、消息解析、mapping、写入。
- `vessel_ais` 自定义源写入 AIS raw observations并广播 `vessels` channel。
- 提供 mock AIS WS 服务,持续发送新增 MMSI 和位置变更。
- 删除自定义源时提供“是否删除该源数据”的选项。
- 梳理设置中心信息架构,明确后续 UI 重构方向。
### 暂不做
- 不新增任意动态数据库表。
- 不允许用户提交可执行脚本作为 mapping。
- 不让 LLM 进入正式采集链路。
- 不把 mock 数据直接写 legacy `vessel_position`,优先写 AIS raw observations保持可追踪和可删除。
- 不在本阶段完成完整 `Earth Live Sync`,但要为后续 summary invalidation 留出 hook。
## 现状入口
| 能力 | 当前位置 |
|-----|----------|
| 自定义源配置模型 | `backend/app/models/datasource_config.py` |
| 自定义源 mapping 模型 | `backend/app/models/datasource_mapping.py` |
| 自定义源 API | `backend/app/api/v1/datasource_config.py` |
| 目标 schema registry | `backend/app/core/target_schema_registry.py` |
| mapping engine | `backend/app/services/datasource_mapping.py` |
| 数据源总览 UI | `frontend/src/pages/DataSources/DataSources.tsx` |
| 采集器设置 UI | `frontend/src/pages/Settings/Settings.tsx` |
## 目标架构
```mermaid
flowchart LR
A[Custom Source Config] --> B{source_type}
B -->|rest| C[Mapped REST Runner]
B -->|websocket| D[Mapped WS Runner]
C --> E[Mapping Engine]
D --> E
E --> F[Target Schema Validator]
F --> G{Destination Handler}
G -->|vessel_ais| H[AIS Raw Observations]
H --> I[AIS Aggregation]
H --> J[vessels WS Channel]
J --> K[Earth Vessel Upsert]
```
## 数据配置设计
短期可以继续复用 `DataSourceConfig`,避免大迁移。语义约定如下:
| 字段 | 用途 |
|-----|------|
| `name` | 自定义源唯一名称,例如 `mock_ais_ws` |
| `source_type` | `rest``websocket` |
| `endpoint` | `http(s)://...``ws(s)://...` |
| `auth_type` | `none``bearer``api_key``basic` |
| `auth_config` | token、api_key、key name、basic username/password 等 |
| `headers` | 静态请求头 |
| `config` | method、params、body、timeout、retry、WS 订阅消息、重连策略、消息路径等 |
建议 `config` 结构:
```json
{
"transport": "websocket",
"delivery_mode": "realtime_stream",
"merge_target_source": "barentswatch_vessels",
"target_schema": "vessel_ais",
"method": "GET",
"params": {},
"body": null,
"timeout": 30,
"retry": 3,
"ws_subscribe_message": {"type": "subscribe", "channel": "vessels"},
"ws_message_path": "$.data",
"ws_items_path": "$.vessels[*]",
"ws_reconnect": true,
"reconnect_delay_seconds": 3,
"debug_max_messages": null,
"delete_policy": "config_only"
}
```
## 后端实施计划
### Phase 1 — 自定义源类型与连接测试
- 允许 `source_type``rest``websocket`
- REST 连接测试保留现有 HTTP 请求逻辑。
- WebSocket 连接测试新增:
- 校验 endpoint 必须是 `ws://``wss://`
- 注入 headers 和 auth。
- 连接后可选发送 `ws_subscribe_message`
- 读取一条消息或超时返回诊断。
### Phase 2 — Mapped REST Runner 补齐
现有 `run-mapped` 继续作为 REST 一次性采集入口,补齐:
- `GET/POST` method。
- query params。
- JSON body。
- headers 和 auth 注入。
- sample limit 与响应大小限制。
- `vessel_ais` destination handler。
### Phase 3 — Mapped WebSocket Runner
新增通用 WebSocket runner读取 `DataSourceConfig + active mapping`
- 建立长连接。
- 发送可选订阅消息。
- 循环接收消息。
- JSON parse。
-`ws_message_path/ws_items_path` 提取 item 或 list。
- 使用 mapping engine 转换。
- 使用 target schema validator 校验。
- 调用 destination handler 写入。
- 更新采集任务状态:
- `connecting`
- `streaming`
- `reconnecting`
- `stopped`
- 维护运行指标:
- `messages_seen`
- `records_written`
- `unique_entities`
- `last_message_at`
- `last_error`
- 后台长连接不读取 `config.debug_max_messages`;该字段只用于显式的一次性调试运行,避免正式 WS 流被测试上限截断。
### Phase 4 — Destination Handler
为 target schema 建立明确写入处理器。
`vessel_ais` handler
- 写入 `AISRawObservation`
- `source = datasource.name`
- `delivery_mode` 来自 config默认 WS 为 `realtime_stream`、REST 为 `polling`
- `transport` 来自 `source_type`
- 生成幂等 observation hash。
- 更新 AIS source health。
- 广播 `vessels` channelpayload 使用当前 Earth 已支持的 upsert 格式。
`generic_records` handler
- 写入通用 collected data 或后续 generic store。
- 不直接进入 Earth。
### Phase 5 — 删除与数据清理
删除自定义源时新增清理策略:
| 选项 | 行为 |
|-----|------|
| 只删除配置 | 删除 `datasource_configs`,保留 mapping 和历史数据需要另行处理 |
| 删除配置和 mapping | 删除配置及对应 `datasource_mapping_templates` |
| 删除配置、mapping 和该源数据 | 同时删除该源写入的数据 |
数据删除范围:
- `collected_data.source == datasource.name`
- `ais_raw_observations.source == datasource.name`
- `ais_source_health.source == datasource.name`
不建议直接删除 legacy `vessel_position`,因为当前 legacy 表不带 source无法安全归因。自定义 AIS 源应优先只写 raw observations。
删除数据后应触发:
- `vessels` channel 的 reload/invalidation 事件,提示 Earth 重新拉船只聚合。
- 后续接入 `Earth Live Sync` 后,触发 `earth_summary` invalidation。
### Phase 6 — Mock AIS WebSocket 服务
新增脚本:
`scripts/mock-ais-ws-server.ts`
运行方式建议:
```bash
bun run mock:ais-ws
```
服务行为:
- 监听 `ws://localhost:8787/ais`
- 接受任意客户端连接。
- 可记录收到的 subscribe message。
- 每 1-2 秒发送一条 AIS-like JSON。
- 每隔 N 条生成新 MMSI验证船只数量增长。
- 已存在 MMSI 随时间改变 `lat/lon/cog/heading`,验证同 MMSI upsert。
- 支持固定 seed保证测试可复现。
示例 payload
```json
{
"type": "vessel",
"data": {
"mmsi": "999000001",
"name": "MOCK VESSEL 001",
"lat": 31.23,
"lon": 121.47,
"sog": 12.4,
"cog": 86,
"heading": 90,
"received_at": "2026-05-01T00:00:00Z"
}
}
```
## 前端实施计划
### 信息架构调整
自定义源不作为割裂的新入口,而是作为内置采集器的补充源,直接纳入“配置中心 > 采集器设置”的采集器选择器:
- 采集器下拉框同时展示内置采集器和自定义补充源。
- 下拉框右侧提供加号按钮,用于添加自定义源。
- 新建自定义源时必须选择“合并到内置数据”,例如合并到 `barentswatch_vessels`
- 选择自定义源后右侧基础配置区域沿用正常采集器配置形态支持连接测试、保存、endpoint、headers、auth、高级 JSON。
- 自定义源比内置源多一个“删除自定义源”按钮。
- 删除时弹出确认框,可勾选“同时删除该自定义源生成的所有数据”。
数据源页保留:
- 内置源总览。
- 内置源最近状态。
- 内置源手动触发。
- 不展示自定义源管理入口;自定义源创建、维护、删除统一在采集器设置中完成。
### 自定义源表单
新增或重构自定义源表单:
- 源名称。
- 类型:`REST` / `WebSocket`
- 合并到内置数据:必选,用于声明该源补充哪个内置数据域。
- endpoint。
- method/body/params仅 REST 显示。
- subscribe message/message path/items path仅 WS 显示。
- auth type。
- headers。
- target schema。
- sample/test 按钮。
- mapping assistant/preview。
- 保存并运行。
### 删除确认
删除自定义源时弹出确认:
- 默认只删除配置。
- 可勾选删除 mapping。
- 可勾选删除该源写入的数据。
- 显示将删除的数据范围和不可恢复提示。
## 验证方案
### Mock WS 验证路径
1. 启动 mock 服务:
```bash
bun run mock:ais-ws
```
2. 新建自定义源:
| 字段 | 值 |
|-----|----|
| name | `mock_ais_ws` |
| source_type | `websocket` |
| endpoint | `ws://localhost:8787/ais` |
| merge_target_source | `barentswatch_vessels` |
| target_schema | `vessel_ais` |
| ws_message_path | `$.data` |
3. 保存 active mapping
```json
{
"source": {
"items_path": "$"
},
"fields": {
"mmsi": {"path": "$.mmsi", "type": "integer"},
"name": {"path": "$.name", "type": "string"},
"lat": {"path": "$.lat", "type": "float"},
"lon": {"path": "$.lon", "type": "float"},
"sog": {"path": "$.sog", "type": "float", "default": null},
"cog": {"path": "$.cog", "type": "float", "default": null},
"heading": {"path": "$.heading", "type": "integer", "default": null},
"received_at": {"path": "$.received_at", "type": "datetime", "default": null}
}
}
```
4. 启动自定义源。
5. 打开 Earth 船只图层,不刷新页面观察:
- `vessels` WS channel 收到 `source = mock_ais_ws`
- HUD 船只数在新 MMSI 到达时增加。
- 地球出现 `MOCK VESSEL`
- 同 MMSI 后续消息更新位置和航向,不重复叠加。
### 自动化测试
后端测试:
- WebSocket 自定义源连接测试。
- WS message path 和 items path 提取。
- mapping 到 `vessel_ais`
- 写入 AIS raw observation。
- 广播 `vessels` channel。
- 删除自定义源时按策略删除 mapping 和源数据。
前端测试:
- REST/WS 表单条件显示。
- 删除确认选项。
- mock 源配置保存 payload。
- mapping preview 展示错误和成功记录。
## 风险与约束
- WebSocket 自定义源是长连接,不能沿用一次性 REST 进度条。
- 如果 mock 源写 legacy vessel 表,删除会变得不安全,因此先只写 raw observations。
- 自定义 WS 可能消息量很大,必须有 backpressure、日志限流和任务取消能力。
- 任意外部 WS 不能信任 payload必须经过 mapping 和 schema validation。
- headers/auth 不能进入 LLM mapping prompt。
## 交付顺序
1. Mock AIS WS 服务。
2. 后端自定义 WS runner。
3. `vessel_ais` destination handler 和 `vessels` broadcast。
4. 删除自定义源及数据清理。
5. 设置中心采集器下拉框内联自定义源 UI。
6. 配置中心信息架构重整。
7.`Earth Live Sync` 对接 summary invalidation。

View File

@@ -0,0 +1,313 @@
# Earth Interactable Layer Plan
## 背景
状态Phase 1 已经开始落地Phase 2 的 BGP 事件 / 观测站迁移和 Phase 3 的算力中心迁移也已完成。`frontend/public/earth/js/interactable.js` 已新增AIS 船只、BGP 事件、BGP 观测站和算力中心图层已经改为通过 `createInteractableLayer()` 使用通用批量 `Points`、hover / locked overlay、默认 glow、状态更新、asset icon 预加载、屏幕空间 picking、固定 / 距离缩放和跨 Interactable 同坐标避让。登陆点因 `THREE.Points` 边缘深度裁切和贴地层级要求,已退回专用 `THREE.Sprite` 黄色球路径,并与海缆同高度同 renderOrder。后续阶段聚焦把可复用的扩圈 / 雷达扇形动画正式沉淀成 `animations` 扩展。
当前实现说明和接入示例见:
- [earth-interactable-usage.md](/home/ray/dev/linkong/planet/docs/technical/zh/earth-interactable-usage.md)
当前 AIS 船只图层已经形成了一个适合作为基准的交互图标模式:
- 普通态使用批量 `THREE.Points` 渲染,避免每个对象一个 `Sprite` 带来的 draw call 和透明排序压力。
- hover / locked 态使用单点 overlay 叠加 glow不改变普通批次交互反馈清晰且成本低。
- moving / anchored 船只通过 canvas 点纹理表达不同形状moving 船只还按航向分桶。
- 拾取走屏幕空间命中,拖动和惯性期间跳过高频 hover picking。
- 图层高度贴近地表,仅保留很小的深度余量,避免“浮在表面层”的观感。
这个模式不应该只服务船只。后续 BGP 事件、BGP 观测站、算力中心、新闻事件、告警、地面传感器等都可能需要“图标类可交互元素”。如果每个图层继续各写一套 icon、glow、hover、locked、动画、picking 和图例逻辑,视觉会漂移,性能策略也会重复分叉。登陆点已经验证为例外:需要完整贴地且不被球面边缘裁切时,专用 Sprite 路径比通用 `Points` 更合适。
目标是把船只图层的成功做法抽象成一个通用接口:业务图层只描述“要画什么、在哪里、怎么交互”,底层统一负责批量渲染、默认 glow、状态 overlay、动画槽位、拾取和生命周期。
## 目标
1. 建立统一的 Earth 交互图标接口,作为未来地表图标类元素的默认入口。
2. 以 AIS 船只 glow 为默认 glow 视觉,其它图标默认沿用同一套 glow 质感。
3. 保留图标颜色、状态颜色、hover 放大、locked 强调、dimmed 聚焦、动画扩展等能力。
4. 支持 canvas / SVG / image icon不强行要求所有图标都可重着色。
5. 保持船只当前性能路线:批量绘制普通态,少量 overlay 处理交互态。
6. 给 BGP 事件扩圈、BGP 观测站雷达扇形等补充动画留出正式扩展点。
## 非目标
- 不在第一阶段重写所有 Earth 图层。
- 不把卫星、海缆、国家边界、真实地形这类非图标图层纳入同一个接口。
- 不为了抽象牺牲业务图标的差异表达例如船只航向、BGP 事件严重级别、观测站雷达扫掠。
- 不要求图片图标支持运行时重着色;图片图标只能通过预制多状态图片或 overlay tint 做有限表达。
## 核心设计
建议新增一个通用模块,例如:
```text
frontend/public/earth/js/interactable.js
```
它导出一个工厂或注册函数:
```js
createInteractableLayer({
id,
earth,
renderOrder,
altitudeOffset,
icon,
scale,
glow,
colors,
states,
animations,
picking,
data,
getPosition,
getKind,
getRotation,
getPayload,
});
```
业务模块仍保留自己的数据加载、图例、详情卡字段和业务语义。例如 `vessels.js` 负责 AIS 数据和船型映射,但 icon 渲染、hover overlay、locked overlay、默认 glow 和屏幕空间 picking 可以逐步迁入 `interactable.js`
## 参数草案
| 参数 | 类型 / 示例 | 默认值 | 说明 |
| --- | --- | --- | --- |
| `id` | `"vessels"` | 必填 | 图层唯一标识,用于 debug、picking、legend 和状态缓存。 |
| `earth` | `THREE.Object3D` | 必填 | 图层挂载目标,通常是 Earth root。 |
| `renderOrder` | `4.4` | `4` | 普通 icon 批次和 overlay 的基础渲染顺序。 |
| `altitudeOffset` | `0.2` | `0.2` | 图层高度,语义为 `CONFIG.earthRadius + altitudeOffset`。地表图标默认贴近真实地形基础层。 |
| `icon` | `{ type, source, draw, size, bins }` | 必填 | 图标来源。支持 canvas draw、SVG URL、image URL、内置 shape。 |
| `icon.fitSize` | `60``{ width: 60, height: 60 }` | `atlasCellSize` | asset 图标在 atlas canvas 内的最大绘制尺寸,默认居中等比 contain。SVG / 图片文件只负责原始形状,不需要为了显示大小手写 transform。 |
| `scale` | `{ base, min, max }` | `{ base: 1 }` | 基础缩放和距离稳定范围。当前船只可映射到 `VESSEL_POINT_SIZE` / `baseScale`。 |
| `sizeMode` | `"fixed" / "distance"` | `"fixed"` | 是否固定屏幕像素尺寸;非 fixed 时按相机到地表距离做比例缩放。 |
| `sizeScale` | `{ min, max, referenceFov }` | `{ min: 0.12, max: 3, referenceFov: 75 }` | `sizeMode !== "fixed"` 时的缩放限制和参考视角。 |
| `glow.enabled` | `true / false` | `true` | 是否启用默认 glow。默认 glow 以船只 hover / locked overlay 为基准。 |
| `glow.intensity` | `0.0 - 2.0` | `1` | glow 强度,内部映射到 canvas `shadowBlur`、opacity 或 shader uniform。 |
| `glow.colorMode` | `"state" / "icon" / "fixed"` | `"state"` | glow 颜色来源,默认跟随状态颜色。 |
| `hover.scale` | `1.0 - 2.0` | `1.18` | hover 放大倍率。当前船只保持同尺寸 glow overlay接口仍保留放大能力供其它图层使用。 |
| `hover.mode` | `"scale" / "glow-only" / "custom"` | `"scale"` | hover 反馈方式。船只可用 `"glow-only"`,其它图标默认放大。 |
| `colors.normal` | `"#4A90D9"` | icon 原色 | 普通态颜色。只有可上色 icon 生效。 |
| `colors.hover` | `"#7dd3fc"` | normal | hover 态颜色。 |
| `colors.locked` | `"#ffffff"` | hover | locked 态颜色。 |
| `colors.dimmed` | `"#9B9B9B"` | normal | 聚焦其它对象时的弱化颜色。 |
| `colors.byKind` | `{ cargo: "#4A90D9" }` | `{}` | 按业务类型着色如船型、BGP 严重级别。 |
| `colorable` | `true / false` | 由 icon 类型推断 | canvas shape 和 SVG mask 通常可上色;图片默认不可上色。 |
| `opacity` | `{ normal, hover, locked, dimmed }` | 船只当前值 | 各状态透明度。 |
| `rotation` | `{ enabled, bins, getAngle }` | disabled | 是否按角度分桶,例如船只按 COG 分 32 桶。 |
| `animations` | `IconAnimationSpec[]` | `[]` | 补充动画列表,例如扩圈、雷达扇形、脉冲、轨迹尾迹。 |
| `picking.radiusPx` | `22` | `20` | 屏幕空间命中半径。 |
| `picking.throttleMs` | `100` | `80` | hover picking 节流。 |
| `picking.skipWhileDragging` | `true` | `true` | 拖动和惯性期间跳过 hover picking。 |
| `zIndexPolicy` | `"surface-icon"` | `"surface-icon"` | 预设层级策略,避免每个业务图层手写高度和 renderOrder。 |
| `avoidance.enabled` | `true / false` | `true` | 是否参与跨 Interactable 的同坐标避让。默认开启,同一经纬度下的图标会沿地表切平面小幅排开,方便辨认和选择。 |
| `avoidance.radius` | `number` | `1.1` | 同坐标避让的第一圈半径,单位为地球本地坐标单位。 |
| `avoidance.precision` | `number` | `4` | 经纬度归并精度,默认约等于只处理几乎完全重叠的图标。 |
| `legend` | `{ label, color, shape }[]` | `[]` | 可选图例声明,业务层也可以继续自己导出。 |
| `metadata` | object | `{}` | 业务扩展数据,不参与渲染但参与 tooltip / info-card / search。 |
## Icon 规格
图标输入建议分三类:
```js
{
type: "canvas-shape",
size: 128,
draw(ctx, state) {
// draw triangle / dot / custom shape
},
}
```
```js
{
type: "svg-mask",
source: "/earth/assets/icons/bgp-event-dot.svg",
colorable: true,
}
```
```js
{
type: "image",
source: "/earth/assets/icons/vendor-logo.png",
colorable: false,
stateSources: {
hover: "/earth/assets/icons/vendor-logo-hover.png",
},
}
```
颜色策略:
- `canvas-shape` 默认可上色,适合船只、事件点、雷达站这类符号。
- `svg-mask` 如果能作为 mask 使用,则可上色;如果是完整多色 SVG则按图片处理。
- `image` 默认不可上色;需要状态变化时使用 `stateSources` 或额外 glow / ring。
## 默认 Glow 规范
默认 glow 以当前船只 overlay 为视觉基准:
- 普通态尽量不启用 glow保持地图干净。
- hover / locked 态叠加同位置 overlay。
- glow 颜色默认跟随状态颜色或业务类型颜色。
- glow blur 应该稳定,不随 camera zoom 夸张膨胀。
- 允许通过 `glow.intensity` 控制强度,但不要让业务图层各自发明完全不同的光晕语言。
建议内部把 glow 拆成两个层次:
1. `textureGlow`canvas texture 里的 `shadowBlur`,适合小图标 hover / locked。
2. `effectGlow`:额外 ring / halo / pulse适合告警、BGP 事件和锁定强调。
## 状态模型
通用状态至少包含:
| 状态 | 触发 | 默认表现 |
| --- | --- | --- |
| `normal` | 普通显示 | 批量 Points使用 normal 颜色和 opacity。 |
| `hover` | 指针悬停 | 默认放大并显示 glow船只可配置为同尺寸 glow-only。 |
| `locked` | 点击锁定 / 详情打开 | 强 glow、更高 opacity可选 ring 或 pulse。 |
| `dimmed` | 聚焦其它对象 | 降低 opacity保留上下文。 |
| `hidden` | 图层关闭或过滤 | 不参与绘制和 picking。 |
| `alert` | 业务告警 | 可叠加动画,不替代 locked 状态。 |
状态更新需要增量化:只在 hover 目标、locked 目标、过滤条件、数据版本或相机距离阈值变化时更新,不在每帧遍历全部 icon 写材质属性。
## 动画扩展
动画不直接塞进 icon 基础参数,而是作为 `animations` 列表注册。每个动画声明自己的 geometry / material / update 策略:
```js
{
type: "expanding-ring",
when: ["alert", "locked"],
color: "state",
radiusPx: [10, 42],
durationMs: 1400,
opacity: [0.8, 0],
}
```
```js
{
type: "radar-sweep",
when: ["normal", "hover", "locked"],
angleDeg: 72,
rotationMs: 2600,
opacity: 0.36,
}
```
首批建议内置动画:
| 动画 | 用例 | 说明 |
| --- | --- | --- |
| `pulse-ring` | locked、告警点 | 原地呼吸环,强调选中对象。 |
| `expanding-ring` | BGP 事件 | 向外扩散的事件波纹。 |
| `radar-sweep` | BGP 观测站 | 扇形扫描,可持续旋转。 |
| `orbiting-dot` | 数据流 / collector 活跃态 | 小点绕 icon 环绕,表达活动状态。 |
| `trail` | 移动目标 | 可选短尾迹,船只或飞机类目标使用。 |
动画必须支持批量或分组绘制,避免为每个对象创建独立的高频更新对象。只有 locked / hover / 少量 alert 对象可以使用单对象 overlay。
## 渲染策略
### 普通态
普通态优先使用分桶 `THREE.Points`
- 按 icon 类型、可上色策略、旋转分桶、纹理 key 分组。
- 每组一个 `BufferGeometry`,存 `position``color`、必要的 `payloadIndex`
- `PointsMaterial.sizeAttenuation = false`,保持屏幕尺寸稳定。
- `depthTest = true``depthWrite = false`,避免遮挡关系破坏地表。
### 交互态
hover / locked 使用少量 overlay
- overlay 复用 `THREE.Points` 单点对象或小型 ring mesh。
- overlay texture 从统一 cache 获取。
- overlay 更新只写当前 hover / locked 的 position、texture、opacity、size。
### 高密度升级
当某类图标超过分桶 Points 的舒适区,才考虑升级:
- `InstancedBufferGeometry` billboard。
- 自定义 shader 支持 per-instance rotation / scale / opacity。
- 视口 bbox / LOD / cluster。
这个升级不应该改变业务接口,只替换底层 renderer。
## Picking 策略
沿用船只当前方向:
- 默认屏幕空间 picking而不是 Three.js 对每个 Sprite / Points 做 raycast。
- 每个 icon 保留世界坐标和业务 payload。
- 每次 pointer move 将候选点投影到屏幕,按半径和深度判断命中。
- 拖动、惯性旋转、相机剧烈变化期间跳过 hover picking。
- click 时允许做一次更精确的 picking。
后续可以按图层或经纬度网格增加空间索引,减少候选点数量。
## 与现有图层的迁移路径
### Phase 1抽出船只基准能力
-`vessels.js` 提取 texture cache、canvas icon draw、overlay glow、分桶 Points 创建、状态增量更新。
- 保持 `vessels.js` 的公开 API 不变:`loadVessels()``toggleVessels()``getVesselMarkers()` 等继续可用。
- 新模块先只服务船只,确保视觉没有回退。
### Phase 2迁移 BGP 事件和观测站
- BGP 事件使用 `canvas-shape`,已接入 `Interactable`
- 严重级别映射到 `colors.byKind`,并通过通用 `getPointSizeMultiplier` 保留严重级别尺寸倍率。
- 当前扩圈效果保留在 BGP 业务动画中,并跟随 `Interactable` marker 位置更新。
- BGP 观测站主图标已接入 `Interactable`,活跃度映射到颜色和 `getPointSizeMultiplier`
- BGP 观测站 halo / 覆盖扇形继续由 BGP 业务动画表达扫描,并跟随 `Interactable` marker 位置更新。
### Phase 3迁移算力中心并评估登陆点
- 算力中心保留现有业务 icon但接入统一 hover / locked / glow。已完成
- 登陆点曾接入同一套 `Points` 渲染,但 pin 类 SVG 在地球边缘会被深度测试裁切;当前保留专用 `THREE.Sprite`,并使用 canvas 生成黄色扁平球,贴到海缆层级。
- TODO登陆点暂不迁移到完整 Interactable。后续若要统一交互接口优先考虑 Sprite-backed adapter只对齐 `getMarkers()``getPointerIntersections()``setMarkerState()``updateVisualState()` 等外观协议,不强行复用 `THREE.Points`、atlas 和跨图层避让。
- 检查图例、搜索和 info-card 是否只依赖业务 payload而不是依赖渲染对象类型。
### Phase 4形成 Earth 图标层规范
-`docs/technical/zh/earth-frontend-context.md` 记录当前实现入口。
-`docs/technical/zh/earth-layer-style-reference.md` 记录默认 glow、状态颜色、默认高度和动画参数。
-`docs/technical/zh/earth-render-layer-order.md` 记录 surface icon renderOrder 范围。
## 风险与约束
- 过早抽象可能让船只这种高质量基准被平均化,因此第一阶段必须以船只视觉不回退为验收标准。
- 图片 icon 不可上色,接口需要明确 `colorable = false` 的行为,避免业务层误以为颜色一定生效。
- 动画如果默认开启过多,会重新引入 overdraw 和每帧更新压力;默认只给 hover / locked 或少量 alert 使用。
- 地形开启时,贴地 icon 需要在高度、`depthTest``polygonOffset` 和 renderOrder 之间保持平衡。
- 统一 glow 不等于所有图标一模一样;业务可以调强度和颜色,但不应破坏整体视觉语言。
## 验收标准
1. 船只迁入通用接口后普通态、hover、locked、航向、颜色、轨迹和 picking 行为保持一致。
2. 新增一个 BGP 事件示例图层配置,不需要复制船只渲染代码即可得到 icon、glow、hover 和扩圈动画。
3. 新增一个 BGP 观测站示例图层配置,不需要自写独立动画循环即可得到雷达扇形。
4. 关闭图层后对应 icon、overlay、动画和 picking 全部停止。
5. 高密度数据下普通态仍走批量绘制hover / locked 只更新少量 overlay。
6. 文档同步说明默认高度、默认 glow、状态模型和动画扩展点。
## 相关文件
| 文件 | 当前角色 | 未来关系 |
| --- | --- | --- |
| `frontend/public/earth/js/vessels.js` | 船只基准实现,包含分桶 Points、hover / locked overlay、默认 glow 形态 | Phase 1 的抽象来源 |
| `frontend/public/earth/js/constants.js` | 保存船只高度、颜色、透明度、轨迹参数 | 后续可加入通用 surface icon 默认配置 |
| `frontend/public/earth/js/bgp.js` | BGP 事件和观测站视觉逻辑 | BGP 事件和观测站主图标已接入 Interactable扩圈、halo 和覆盖扇形仍保留业务动画 |
| `frontend/public/earth/js/compute-centers.js` | 算力中心 icon 和交互 | 已通过 Interactable 接入统一 Points、overlay、glow 和 picking |
| `frontend/public/earth/js/cables.js` | 登陆点 icon 和海缆线 | 登陆点当前使用专用 `THREE.Sprite` 黄色球,不再走 Interactable海缆线仍独立渲染 |
| `frontend/public/earth/js/main.js` | 当前集中处理 hover、click、locked 和 info-card 入口 | 后续需要接入通用 icon picking 结果 |
| `docs/technical/zh/earth-layer-style-reference.md` | 当前视觉参数参考 | 实现后同步默认 glow 和通用参数 |
| `docs/technical/zh/earth-render-layer-order.md` | 当前层级参考 | 实现后同步 surface icon 层级范围 |

View File

@@ -0,0 +1,478 @@
# AIS 多源采集、冲突记录与聚合接口计划
**状态**v0-v3 已实现v3.1-v3.4 为 v4/v5 前置稳定化任务v4 / v5 已落最小可用子集
**创建日期**2026-04-30
**核心原则**:采集器只写原始观测;去重、合并、冲突解释放在聚合接口中完成
## 已确认决策
| 项目 | 决策 |
|-----|------|
| AISStream 接入方式 | 单独实现 WebSocket 采集器,不塞进现有 BarentsWatch HTTP collector |
| 采集器职责 | 连接上游、标准化字段、写入原始观测,不直接决定最终展示值 |
| 去重合并位置 | 放在聚合服务和聚合 API 中,而不是散落在每个 collector 的保存逻辑里 |
| 冲突处理 | 先记录冲突事实和当前选择原因,后续再开放用户规则配置 |
| 默认可信度 | 同类 AIS 数据源优先按 `delivery_mode` 评估:`realtime_stream` 优于 `batch_stream`,再优于 `polling``snapshot` |
| 过期保护 | 实时流源断流超过 freshness 窗口后,不能仅凭“实时源”身份压过更新的轮询数据 |
| 源健康状态 | 聚合时必须参考采集器健康状态,不能只看配置中的理论优先级 |
| 媒体富化 | 船只图片等媒体信息不进入 AIS 实时聚合主链路,后续单独做 enrichment |
| v4/v5 顺序 | 在聚合完整性、AISStream 实时链路、采集状态语义和基础身份信息显示修好之前,不进入策略配置和 enrichment UI |
## 背景
当前 AIS 链路以 BarentsWatch 为主。它是 HTTP polling 模式,覆盖挪威附近海域,适合作为稳定的免费起点,但不适合承担全球实时船只数据的全部职责。后续接入 AISStream 后,会出现同一个 MMSI 被多个来源同时上报的情况:
- 位置、航速、航向可能在多个来源之间存在秒级差异。
- 船名、IMO、呼号、船型、尺寸等静态字段可能不完整甚至互相冲突。
- WebSocket 或其他实时流通常更接近实时,但也可能断流或批量延迟。
- 如果每个 collector 自己做去重合并,规则会分散、不可审计,也很难让用户后续配置“某个字段信任哪个来源”。
因此第一阶段不应让采集器直接覆盖最终船只表。更稳的方式是先保留观测事实,再由聚合接口统一给出当前展示视图。
## 目标架构
```mermaid
flowchart LR
A[BarentsWatch HTTP collector] --> D[AIS raw observations]
B[AISStream WebSocket collector] --> D
C[Custom mapped vessel_ais sources] --> D
D --> E[AIS aggregation service]
E --> F[Conflict records]
E --> G[GeoJSON vessels API]
E --> H[Vessel detail API]
I[Aggregation strategy config] --> E
```
### 原始观测层
原始观测层保存每个来源看到的事实。建议模型包含:
| 字段 | 用途 |
|-----|------|
| `target_schema` | 例如 `vessel_ais` |
| `source` | 例如 `barentswatch_vessels``aisstream_vessels` |
| `entity_key` | AIS 使用 MMSI |
| `delivery_mode` | `realtime_stream``batch_stream``polling``snapshot` |
| `transport` | `websocket``sse``http``file` 等 |
| `observed_at` | 上游数据时间,优先使用 AIS 消息时间 |
| `collected_at` | 本系统接收或采集时间 |
| `source_message_id` | 上游消息 ID 或可推导 ID没有则为空 |
| `observation_hash` | 幂等去重指纹,用于防止同一来源重复写入同一条观测 |
| `normalized_payload` | 标准化后的 AIS JSON |
| `raw_payload` | 可选,保存原始或裁剪后的上游记录 |
| `quality_flags` | 观测级质量标记,例如 `stale``position_jump``future_timestamp` |
`delivery_mode``transport` 不应混为一谈。WebSocket 是传输方式streaming 是交付模式。聚合可信度主要看 `delivery_mode``transport` 只作为辅助信息。
原始观测层需要做存储级幂等去重,但这里的去重不是业务合并。推荐使用 `source + entity_key + message_type + observed_at + payload_hash` 或上游稳定消息 ID 作为唯一约束,避免 WebSocket 重连、HTTP 重试或批量回放导致同一事实重复入库。
### 源健康状态
每个采集器应维护独立的健康状态,供聚合服务读取:
| 字段 | 用途 |
|-----|------|
| `source` | 采集器标识 |
| `connection_state` | `connected``reconnecting``disconnected``disabled` 等 |
| `last_seen_at` | 最近收到上游消息或响应的时间 |
| `last_success_at` | 最近成功写入观测的时间 |
| `last_error` | 最近错误摘要 |
| `message_rate` | 最近窗口内的消息速率 |
| `lag_seconds` | 上游观测时间与本系统接收时间的延迟 |
聚合优先级不能只看 `source_priority`。例如 `aisstream_vessels` 默认优先于 `barentswatch_vessels`,但如果它处于 `disconnected``lag_seconds` 超过 freshness 窗口,则动态字段应回退到更新的可用来源。
### 身份键边界
v1 可以继续用 MMSI 作为 `entity_key`,因为它是 AIS 动态消息里最稳定、最容易获得的主键。但文档和模型都要为后续扩展留出口MMSI 可能复用、填错或缺少静态信息,后续身份解析应结合 `mmsi + imo + callsign + name + dimensions` 判断是否需要拆分或合并实体。
### 冲突记录层
聚合服务发现同一个实体、同一个字段存在多个非空不同值时,写入冲突记录。冲突记录不代表错误,只代表“有多个可用候选值”。
```json
{
"target_schema": "vessel_ais",
"entity_key": "257123000",
"field": "name",
"candidates": {
"barentswatch_vessels": "OSLO TRADER",
"aisstream_vessels": "OSLO TRADER II"
},
"selected_source": "aisstream_vessels",
"selected_value": "OSLO TRADER II",
"selected_reason": "delivery_mode_priority",
"resolved_by": "system",
"status": "open"
}
```
第一阶段只需要记录冲突和当前选择原因,不需要做人工逐条确认。后续 UI 的目标也不是让用户处理每条冲突,而是把冲突沉淀成字段级规则。
## 聚合规则
### 字段分类
| 类型 | 字段 | 默认策略 |
|-----|------|----------|
| 动态位置 | `lat``lon``sog``cog``heading``nav_status` | 优先最新 `observed_at`,同时间再按来源优先级 |
| 静态身份 | `name``callsign``imo``flag` | 非空优先,再按字段策略或来源优先级 |
| 静态规格 | `vessel_type``vessel_type_name``length``width``draught` | 非空优先;冲突时记录候选值 |
| 轨迹点 | `track_points` | 按时间线合并;同一时间窗口内相近点去重;保留点级 `source` |
| 元信息 | `field_sources``conflict_count``selected_reasons``quality_flags` | 聚合接口生成,便于调试和后续 UI 展示 |
### 默认优先级
默认优先级应使用两个维度:
```yaml
delivery_mode_priority:
- realtime_stream
- batch_stream
- polling
- snapshot
transport_priority:
- websocket
- sse
- http
- file
```
`delivery_mode_priority` 是主判断。比如 AISStream 如果提供实时推送,应标记为 `realtime_stream + websocket`BarentsWatch 当前是 `polling + http`
### 断流保护
实时流不能永久凭身份占优。聚合时需要 freshness 窗口:
```yaml
freshness:
realtime_stream_seconds: 900
polling_seconds: 3600
```
如果 `aisstream_vessels` 最近 15 分钟没有该 MMSI 的新观测,而 BarentsWatch 轮询源有更新位置,则位置类字段应采用 BarentsWatch 的更新观测,并记录选择原因 `newest_observation``freshness_fallback`
### 异常位置保护
多源 AIS 接入后,聚合服务必须过滤或降权明显异常的位置观测:
- 经纬度必须在合法范围内。
- `observed_at` 不能明显来自未来。
- 同一 MMSI 短时间内跨越不合理距离时,标记 `position_jump`,默认不直接采用该点。
- 当异常点来自当前优先源时,应记录 `selected_reason = anomaly_rejected`,再回退到其他可用来源。
异常保护不应静默丢弃事实。原始观测仍应保留,聚合结果通过 `quality_flags` 和冲突记录解释为什么没有采用它。
### 轨迹聚合
轨迹接口不能简单拼接所有来源,否则前端会出现折返、抖动和重复点。默认规则:
-`observed_at` 排序,生成统一时间线。
- 同一来源的完全重复点通过 `observation_hash` 去重。
- 多来源在短时间窗口内上报的相近位置视为同一轨迹点,优先选择 freshness 和 source priority 更高的一条。
- 每个轨迹点保留 `source``selected_reason` 和必要的 `quality_flags`
- 对被判定为 `position_jump` 的点,默认不进入展示轨迹,但可通过调试参数查看。
## 聚合接口
现有展示接口应逐步改为消费聚合服务,而不是自己直接拼 `VesselPosition + VesselStatic`
```text
GET /api/v1/visualization/geo/vessels
GET /api/v1/visualization/vessels/{mmsi}
GET /api/v1/visualization/vessels/{mmsi}/track
GET /api/v1/visualization/vessels/{mmsi}/conflicts
```
GeoJSON properties 建议增加:
```json
{
"mmsi": 257123000,
"name": "OSLO TRADER",
"lat": 59.91,
"lon": 10.73,
"received_at": "2026-04-30T10:00:00Z",
"field_sources": {
"name": "aisstream_vessels",
"lat": "aisstream_vessels",
"lon": "aisstream_vessels",
"vessel_type": "barentswatch_vessels"
},
"selected_reasons": {
"name": "delivery_mode_priority",
"lat": "newest_observation",
"vessel_type": "non_empty_priority"
},
"quality_flags": [],
"conflict_count": 2
}
```
## 开放配置计划
### Phase 1 — 内置默认策略和只读解释
- 实现后端默认策略。
- 聚合接口返回 `field_sources``selected_reasons``conflict_count`
- 冲突记录可查询,但不允许用户修改。
- 保持现有前端船只图层接口形状基本兼容,新增字段只作为调试和后续 UI 输入。
### Phase 2 — 系统设置中的 JSON/YAML 策略配置
新增系统设置项,例如:
```yaml
collector_aggregation:
vessel_ais:
source_priority:
- aisstream_vessels
- barentswatch_vessels
field_rules:
name:
mode: source_priority
vessel_type:
mode: source_priority
source_priority:
- barentswatch_vessels
- aisstream_vessels
lat:
mode: newest
lon:
mode: newest
```
配置校验要求:
- 未知 source 只警告,不阻断保存,便于先配置后启用。
- 未知 field 必须拒绝,避免拼写错误悄悄失效。
- 动态位置字段默认不允许被固定来源永久锁死,除非显式开启高级选项。
- 空值不覆盖非空值是全局保护,不建议开放关闭。
### Phase 3 — 冲突治理 UI
基于冲突记录提供页面或 drawer
- 查看某个 MMSI 的冲突字段。
- 查看每个字段的候选来源和值。
- 查看当前选择原因。
- 将一次人工选择保存成字段规则,而不是只处理单条冲突。
- 支持恢复默认策略。
## AISStream 采集器计划
AISStream 采集器单独实现,建议命名为 `aisstream_vessels`。它的职责是:
- 维护 WebSocket 连接、订阅范围和重连。
- 将上游 AIS 消息标准化为 `vessel_ais` payload。
- 标记 `delivery_mode = realtime_stream``transport = websocket`
- 写入原始观测层。
- 不直接 upsert 最终展示数据。
配置应放入采集器设置,而不是硬编码:
```yaml
aisstream_vessels:
api_key: "${AISSTREAM_API_KEY}"
bounding_boxes:
- [[-180, -90], [180, 90]]
message_types:
- PositionReport
- ShipStaticData
```
默认不建议直接订阅全球范围。AISStream 采集器应支持以下订阅策略:
- 使用配置的固定 `bounding_boxes`
- 后续支持按 Earth 当前视口或关注区域动态调整订阅范围。
- 支持限制 `message_types`,避免静态信息、位置报告和扩展消息全量涌入。
- 断线后使用指数退避重连,并把连接状态写入源健康状态。
- 重连后可能收到重复或回放消息,因此必须依赖原始观测层的幂等去重。
### 媒体富化边界
VesselFinder 等服务里的船只图片不属于 AIS 实时数据本身。图片、船籍详情、公司信息等后续应作为独立 enrichment 链路:
- 通过 MMSI、IMO、船名等字段异步查询。
- 使用独立缓存和授权配置。
- 不阻塞 `vessel_ais` 实时观测入库。
- 聚合接口只暴露已经缓存好的媒体引用,不在请求链路中现场抓取。
## 版本拆分
计划先按 v0-v3 建立基础能力,再用 v3.1-v3.4 修复当前稳定性缺口,最后进入 v4/v5
### v0 — 聚合基础设施(已实现)
目标是不改变前端展示行为,先把数据底座铺好。
1. 新增原始观测模型、冲突记录模型和源健康状态模型。
2. 为现有 BarentsWatch collector 写入原始观测,同时保留现有 `vessel_position` / `vessel_static` 兼容写入。
3. 实现存储级 `observation_hash` 幂等去重。
4. 补基础管理命令或调试接口,用于查看某个 MMSI 的原始观测和冲突候选。
### v1 — 聚合读接口(已实现)
目标是让展示接口开始消费聚合结果,但前端形状保持兼容。
1. 实现 AIS 聚合服务,先兼容读取现有表,再逐步切换到原始观测层。
2.`/geo/vessels``/vessels/{mmsi}` 改为走聚合服务。
3.`/vessels/{mmsi}/track` 改为走轨迹聚合逻辑。
4. 返回 `field_sources``selected_reasons``quality_flags``conflict_count`
5. 加入 freshness fallback 和异常位置保护。
### v2 — AISStream WebSocket collector已实现
目标是接入第二个真实 AIS 来源,并验证多源冲突和回退逻辑。
1. 实现 `aisstream_vessels` collector。
2. 支持 API key、订阅范围、消息类型、重连和限流配置。
3. 将 AISStream 写入原始观测层,不直接 upsert 最终展示表。
4. 接入源健康状态和 message rate 统计。
5. 提供 AISStream API Key 获取教程、设置页入口和连接验证支持。
6. 为重复消息、断流回退、WS 优先级写集成测试。
### v3 — AISStream 可用性与配置体验(已实现)
目标是让 AISStream 从“能采集”变成日常可观察、可调试、可配置的数据源。
1. 设置页展示 AISStream 运行状态:连接状态、最近收到、最近成功、本轮消息数、延迟和最近错误。
2. AISStream 设置页提供常用采集范围 preset并保留自定义 Bounding Boxes JSON。
3. 聚合结果返回 `source_summary`,展示每艘船的来源、观测数量、最新观测时间、传输模式和消息类型。
4. 保留 `field_sources``selected_reasons`,用于解释动态字段来自实时流、静态字段来自可用非空来源。
5. 船名标准化会读取 AISStream `MetaData.ShipName`;船型展示会从 `vessel_type_name` 和 AIS 数字 `vessel_type` 共同归一化,保证 marker 颜色、详情卡、hover 和搜索结果一致。
6. `/geo/vessels` 不再默认限制 5000 艘;不传 `limit` 或传 `limit=0` 表示全量返回,前端默认也不再二次裁剪到 5000。
### v3.1 — 聚合完整性修复v4 前置)
目标是先保证“所有已采集到的船都能显示”BarentsWatch 不因为接入 AISStream 而被 raw observation 聚合结果遮蔽。
当前风险是 `/geo/vessels` 只要 raw observation 聚合返回非空,就直接使用 raw 聚合结果,不再补读兼容层 `vessel_position + vessel_static`。如果 raw observation 中只存在 AISStream 的几百艘船,或 BarentsWatch 历史数据没有完整回填到 raw 层,最终 Earth 就会只显示 AISStream 子集。
1. `/geo/vessels` 必须合并 raw observation 聚合结果和 legacy latest position 结果。
2. raw 与 legacy 同一 MMSI 同时存在时只显示一艘,优先使用 raw 聚合结果及其 `field_sources` / `selected_reasons`
3. raw 中不存在的 BarentsWatch-only MMSI 必须从 `vessel_position + vessel_static` 补齐。
4. `bbox``type``limit` 过滤必须作用在合并后的最终集合上;不传 `limit``limit=0` 仍表示全量返回。
5. 增加诊断统计,至少能看到 raw AISStream unique MMSI、raw BarentsWatch unique MMSI、legacy unique MMSI、final merged unique MMSI 和被 legacy 补齐的数量。
6. 为 raw 只有 AISStream 子集、legacy 有更多 BarentsWatch 船只的场景补回归测试。
### v3.2 — AISStream 真实时链路v4 前置)
目标是把 AISStream 从“一次 collector 收一批消息后结束”改成真正的 WebSocket 长连接实时数据源,并把实时变化推送到 Earth。
当前 `aisstream_vessels` 只在 collector `fetch()` 中连接 `wss://stream.aisstream.io/v0/stream`,默认收 `max_messages = 500` 条后结束。这不符合 WebSocket 流式数据源的运行语义,也不能保证新船、位置变化和航向变化实时出现在前端。
1. 为 AISStream 增加 streaming service / long-running runner不再依赖单次 `fetch -> transform -> save -> completed` 表达实时采集。
2. 外部 AISStream WebSocket 保持长连接,断线后指数退避重连,并持续更新 `AISSourceHealth`
3. 每条或小批量 AIS 消息标准化后写入 `ais_raw_observations`,按时间或数量短周期 commit避免长事务堆积。
4. 将新增船只、位置变化、航向变化和静态字段补充转换成 vessel delta。
5. 通过应用内部 `/ws``vessels` channel 广播 delta复用 `DataBroadcaster.broadcast_custom("vessels", payload)`
6. Earth 前端订阅 `vessels` channel`vessels.js` 支持按 MMSI upsert marker而不是每次全量 reload。
7. 船只改变航向时,前端必须更新 course bin / marker bucket避免 marker 方向滞后。
8. freshness 超时或 AISStream 健康异常时,动态字段可回退到 BarentsWatch 最新可用观测。
### v3.3 — Streaming 采集状态语义v4 前置)
目标是让采集页面正确表达 AISStream 这类长连接数据源,不再使用一次性 REST collector 的完成型进度条。
REST collector 的自然状态是 `fetch -> transform -> save -> progress 0..100 -> completed`。AISStream 的自然状态应是 `connecting -> streaming -> reconnecting -> stopped/failed`,没有固定总量,也不应在收到一批消息后显示“采集完成”。
1. AISStream 采集状态使用 indeterminate / streaming 状态,而不是百分比完成进度条。
2. 设置页运行状态卡展示连接状态、已运行时长、本轮消息数、新增观测数、unique MMSI、message rate、最近消息时间、延迟和最近错误。
3. `phase_message` 使用“正在接收 AISStream 实时消息”“重连中”“已停止”等长连接语义。
4. 停止、重连和配置变更要有明确操作入口;配置变化后必须安全重订阅。
5. 后端任务状态不能因为没有 `total_records` 就长期显示 `0%` 或误判失败。
6. WebSocket 健康状态和 collector task 状态要分离:上游短暂断线是 `reconnecting`,不是普通采集任务完成或失败。
### v3.4 — 船只身份字段和名称聚合修复v4 前置)
目标是把 MMSI、IMO、callsign 这类身份编号按字符串显示,并把仍然使用 MMSI 作为船名的记录视为信息聚合未完成,而不是正常船名。
1. 前端详情卡、hover、搜索结果和日志中的 `mmsi``imo``callsign` 必须作为 identifier 字段展示,禁止走 `toLocaleString()` 或数字千分位格式。
2. GeoJSON 可增加 `mmsi_display` / `imo_display` 等字符串字段,但前端仍必须对 identifier key 做兜底格式保护。
3. 聚合服务生成船名时,不能把 `MMSI 257123000` 当成真实 `name` 的成功结果;它只能作为 display fallback。
4. 增加诊断查询,列出所有当前仍以 MMSI 号码或 `MMSI <number>` 作为船只名称的记录,包括:
- `vessel_static.name` 为空或等于 MMSI fallback 的 MMSI
- raw observation 中没有任何非空 `name` / `MetaData.ShipName` / `ShipStaticData.Name` 的 MMSI
- 聚合结果最终 `name` 仍为 fallback 的 MMSI
- 每个 MMSI 的可用来源、最近观测时间、message types 和缺失原因。
5. 对这些 fallback-name 船只建立待修复集合,优先通过 AISStream `ShipStaticData`、BarentsWatch 静态字段和后续 enrichment 缓存补齐。
6. 船只详情面板需要区分“真实船名”和“显示兜底”:真实船名缺失时展示 `MMSI <id>` 可以继续作为标题,但字段来源应标注为 `fallback`,避免误以为聚合成功。
7. 为 MMSI 千分位格式、fallback-name 诊断和名称来源解释补回归测试。
### v4 — 策略配置v0 可用)
目标是开放系统级配置,但仍以安全默认值兜底。
已落地的最小子集:
1. 策略持久化在 `system_settings.category = 'vessel_aggregation_strategy'`,保存时自动版本递增。
2. `app/services/vessel_aggregation_strategy.py` 暴露 `load_strategy / save_strategy / reset_strategy / validate_strategy`,并维护 `DEFAULT_STRATEGY` 兜底。
3. 校验规则:
- 未知 `field_rules.<name>``400 unknown vessel_ais field`
- 未知 mode → `400 mode must be one of ...`
- 动态字段(`lat/lon/sog/cog/heading/nav_status`)使用非 `newest` mode 时必须显式 `allow_dynamic_lock=true`,否则拒绝;
- `freshness.realtime_stream_seconds` / `polling_seconds` 必须为非负整数;
- `mode=locked` 必须带非空 `locked_source`
4. 聚合服务 `vessel_ais_aggregation.py``_select_position_observation` 中按 `freshness` 把过期实时流降级到 stale 候选;在 `_select_static_field` 中按 `field_rules.mode = source_priority / locked / newest / non_empty` 选源。
5. 聚合输出每条 vessel 携带 `aggregation_strategy_version`,并在 `/geo/vessels` GeoJSON properties + `/vessels/{mmsi}` 详情中暴露。
6. API
- `GET /api/v1/vessel-aggregation/strategy`
- `PUT /api/v1/vessel-aggregation/strategy`(校验失败 400
- `DELETE /api/v1/vessel-aggregation/strategy`(恢复默认并 bump version
未做项(留给 v4 后续):
- 系统设置 UI 中的策略编辑器尚未做,目前直接调 API
- `transport_priority``quality_flags` 级别的策略尚未引入;
- `source_priority` 中的未知 source 不强校验,留给后续 warn-only 提示。
### v5 — 船舶资料 enrichment 与冲突治理v0 可用)
目标是把 AIS 实时流里不稳定或低频出现的静态信息,补成可缓存、可审计的船舶资料层,同时把冲突解释变成可操作能力。
已落地的最小子集:
1. 新增模型 `app/models/vessel_enrichment.py::VesselProfileEnrichment` + `VesselMediaEnrichment`:以 `mmsi` 为主键,记录 `source / payload / fetched_at / expires_at / confidence / reference_url`;通过 `Base.metadata.create_all``init_db` 中建表。
2. 服务 `app/services/vessel_enrichment.py` 提供 `upsert_vessel_profile_enrichment` / `upsert_vessel_media_enrichment` / `get_vessel_enrichment_bundle`;读路径只读缓存,过期记录(`expires_at < now`)直接过滤为 `None`,永不联网。
3. 聚合接口在 `/api/v1/visualization/vessels/{mmsi}` 响应中追加 `enrichment.profile``enrichment.media` 字段(含 `source / fetched_at / expires_at / confidence / reference_url`);命中失败时返回 `null`,不阻塞 AIS 实时链路。
4. 冲突治理 API
- `POST /api/v1/vessel-aggregation/conflicts/{mmsi}/{field}/promote-to-rule` 读取最近 `AISConflictRecord.selected_source`,写入 `field_rules[field] = {mode: source_priority, source_priority: [<source>]}` 并 bump version
- `DELETE` 对应路径移除该 field 的覆盖,恢复默认。
5. 前端 Earth `info-card.js` 渲染 `船舶资料` 区块profile.payload 标量字段平铺、媒体 `images` 数组缩略图、来源 / 更新时间 / 置信度元数据;缓存命中失败回退到 `资料缓存中`;常规字段在 `field_sources` 命中时附带来源 tag。
未做项(留给 v5 后续):
- 没有真正的异步 enrichment 抓取作业;当前依赖外部脚本/管理 API 写入缓存;
- 冲突治理 UI 还没接入设置中心,目前只暴露 API
- enrichment 命中状态尚未广播到 `vessels` channel详情面板首次打开时按需请求即可。
## 测试计划
- 同一来源同一 `mmsi + observed_at + lat + lon` 重复记录只聚合一次。
- 多来源同一 MMSI 的位置字段优先选择最新观测。
- 实时流和轮询源同时间冲突时,实时流优先。
- 实时流过期后,更新的轮询源可以接管动态字段。
- 实时流源健康状态异常时,动态字段可以回退到更新的可用来源。
- 静态字段不会被空值覆盖。
- 静态字段冲突会写入冲突记录。
- 明显异常位置不会进入默认展示轨迹,并会留下 `quality_flags`
- 同一时间窗口内多来源相近轨迹点只展示一个点。
- AISStream 重连或回放导致的重复消息不会重复进入聚合结果。
- raw observation 聚合结果和 legacy latest position 结果会按 MMSI 合并BarentsWatch-only 船只不会因为 AISStream 子集存在而消失。
- 不传 `limit` 或传 `limit=0` 时,`/geo/vessels` 全量返回合并后的船只集合。
- AISStream 长连接收到新船、位置变化和航向变化后,会通过内部 `/ws``vessels` channel 推送增量。
- AISStream streaming 状态不会显示成固定百分比完成进度条,也不会在收到一批消息后误报采集完成。
- `mmsi``imo``callsign` 等身份编号在前端不显示千分位符。
- 聚合结果中仍以 MMSI fallback 作为船名的记录可以被诊断查询完整列出,并带来源和缺失原因。
- 字段级配置可以覆盖默认来源优先级。
- 聚合接口在没有冲突表时仍可返回兼容 GeoJSON。
## 相关文件
- [实时船只监控系统计划](/home/ray/dev/linkong/planet/docs/plans/earth-vessel-tracking-plan.md)
- [自定义 API 数据源与 LLM 映射系统计划](/home/ray/dev/linkong/planet/docs/plans/datasource-custom-api-mapping-plan.md)
- [BarentsWatch AIS collector](/home/ray/dev/linkong/planet/backend/app/services/collectors/vessel_ais.py)
- [船只模型](/home/ray/dev/linkong/planet/backend/app/models/vessel.py)
- [可视化 API](/home/ray/dev/linkong/planet/backend/app/api/v1/visualization.py)

View File

@@ -1,5 +1,17 @@
# Earth Vessel Rendering Performance Plan
## 当前状态
该计划的前端核心部分已经在 `0.44.1` 落地,但最终实现不是原文设想的 `InstancedBufferGeometry` quad而是更稳的分桶 `THREE.Points` 方案:
- 普通船只按 moving / anchored 和 `VESSEL_COURSE_BINS` 航向分桶,使用 `PointsMaterial` 批量绘制。
- 航行船只仍是带方向的三角形,停泊或低速船只仍是圆点。
- hover / locked 不再放大成世界尺寸 Sprite而是在原点位叠加同尺寸单点 glow overlay。
- picking 改为屏幕空间命中,拖动和惯性期间跳过 hover picking。
- 普通态关闭 glow交互态才显示 glow降低 overdraw 并让默认地图更干净。
后续如果需要全球 AIS 或更高船只密度,再评估是否从分桶 `Points` 升级到真正 instanced quad 或视口 bbox / LOD。
## 背景
Earth 船只图层已经形成了一套较好的视觉语言:
@@ -8,10 +20,10 @@ Earth 船只图层已经形成了一套较好的视觉语言:
- 标记按航向旋转
- 停泊或低速船只使用圆点
- 不同船型使用不同颜色
- hover / locked 状态有放大、透明度和聚焦反馈
- hover / locked 状态有 glow、透明度和聚焦反馈
- 标记带有轻微 glow / soft edge和 Earth HUD 的观感一致
当前性能问题不应通过降级成普通 `Points` 来解决。目标是在保留现有观赏性的前提下,把底层从“每艘船一个 Sprite 对象”优化为批量绘制和轻量交互。
当前性能问题不应通过降级成无方向、无船型语义的普通小点来解决。目标是在保留现有观赏性的前提下,把底层从“每艘船一个 Sprite 对象”优化为批量绘制和轻量交互。
## 当前问题判断
@@ -106,11 +118,18 @@ Earth 船只图层已经形成了一套较好的视觉语言:
目标是减少屏幕空间重叠面积,而不是改变符号设计。
## Phase 3保留视觉的批量渲染
## Phase 3保留视觉的批量渲染(已落地为分桶 Points
正式方案是把每艘船的视觉从 `THREE.Sprite` 迁移为 instanced sprite batch。
原设想是把每艘船的视觉从 `THREE.Sprite` 迁移为 instanced sprite batch。实际落地时选择了更稳的分桶 `THREE.Points`
### 1. 使用 instanced quad
- 不依赖自定义 shader。
- 不依赖 `Points` 自带 raycaster。
- 用 canvas 纹理保留三角、圆点、船型颜色和航向。
- 用 hover / locked 单点 overlay 保留交互 glow。
如果未来全球 AIS 导致分桶 `Points` 仍不够,再升级到 instanced quad。
### 1. 原候选方案instanced quad
每艘船仍然显示为带贴图/软边的 billboard但底层使用
@@ -130,7 +149,7 @@ Earth 船只图层已经形成了一套较好的视觉语言:
这样 draw call 从“每艘船一个”变为“每类船只一个”。
### 2. per-instance attributes
### 2. 原候选方案:per-instance attributes
每个 instance 存:
@@ -142,9 +161,21 @@ Earth 船只图层已经形成了一套较好的视觉语言:
- state
- mmsi / data index
hover、locked、dimmed 通过更新少量 instance attribute 实现,不再逐个修改 material。
hover、locked、dimmed 通过更新少量 instance attribute 实现,不再逐个修改 material。
### 3. 复刻当前视觉
### 3. 当前落地方案:分桶 `THREE.Points`
当前实现按以下方式复刻视觉:
- moving 船只按 `VESSEL_COURSE_BINS` 做航向分桶。
- anchored / slow 船只使用圆点分桶。
- 每个分桶生成一组 `THREE.Points`,共享 `PointsMaterial` 和 canvas 点纹理。
- `VESSEL_CONFIG.colors` 仍通过 vertex colors 表示船型颜色。
- hover / locked 在原位置叠加同尺寸单点 overlay普通态不带 glow交互态才带 glow。
这样 draw call 从“每艘船一个”变为“每个形状 / 航向分桶一组”,同时避免自定义 shader 的兼容风险。
### 4. 复刻当前视觉
视觉上继续使用当前 canvas texture 或等效 shader
@@ -213,5 +244,6 @@ hover、locked、dimmed 通过更新少量 instance attribute 实现,不再逐
1. 先做 Phase 1快速恢复地球拖动手感。
2. 再做 Phase 2减少每帧 JS 写操作。
3. 最后做 Phase 3 和 Phase 4,把船只迁移到 instanced sprite batch
3. Phase 3 和 Phase 4 已按分桶 `THREE.Points` + 屏幕空间 picking 落地
4. Phase 5 等全球船只数据或数量压力出现后再推进。
5. 如果分桶 `THREE.Points` 达到瓶颈,再评估 instanced quad。

View File

@@ -12,7 +12,7 @@
| 船只规模 | BarentsWatch 阶段全部显示;全球数据接入后按需加船型过滤(默认 Cargo + Tanker + Passenger |
| 更新频率 | 准实时:前端 5 分钟轮询,后端 Collector 每分钟拉取写库 |
| 历史轨迹 | 保留(`vessel_position` 表保留 24h后期按需扩展 |
| 推送方式 | HTTP 轮询(不用 WebSocket换实时数据源后再评估升级 |
| 推送方式 | 前端展示仍可先用 HTTP 拉取聚合结果AISStream 等实时源应单独实现 WebSocket 采集器 |
---
@@ -36,13 +36,19 @@
- 字段mmsi, lat, lon, sog, cog, heading, nav_status, name, vessel_type, flag
- 刷新频率:数据约 3060s 更新一次,可随意轮询
### TODO付费数据源接入
### TODO多源 AIS 与实时流接入
- [ ] 接入 AISStream WebSocket 采集器,作为 BarentsWatch 覆盖不足的实时补充
- [ ] 将 BarentsWatch、AISStream、自定义 `vessel_ais` 映射源统一写入原始观测层
- [ ] 通过聚合接口做去重、字段合并、冲突记录和默认来源选择
- [ ] 开放字段级聚合策略配置,让用户决定不同字段优先信任哪个来源
- [ ] 评估 AISHub 订阅(全球覆盖,约 $30/月),接入全球实时流
- [ ] 评估 MarineTraffic API tier对比 AISHub 数据质量与成本
- [ ] 实现多数据源适配器,通过 `datasource_config` 切换
- [ ] 真实高频 AIS 稳定接入后,评估将 `vessel_position` 迁移为 TimescaleDB hypertable保留 Postgres 原生分区作为备选)
多源 AIS 的详细设计见 [AIS 多源采集、冲突记录与聚合接口计划](/home/ray/dev/linkong/planet/docs/plans/earth-vessel-ais-aggregation-plan.md)。
---
## 二、实施计划
@@ -118,7 +124,7 @@ CREATE UNIQUE INDEX ON vessel_latest(mmsi);
GET /api/v1/visualization/geo/vessels
?bbox=lon_min,lat_min,lon_max,lat_max # 视口裁剪
?type=cargo,tanker,passenger # 船型过滤
?limit=5000
?limit=0 # 可选;不传或 0 表示不裁剪数量
→ GeoJSON FeatureCollectionPoint
GET /api/v1/visualization/vessels/{mmsi} # 单船详情
@@ -151,12 +157,15 @@ GeoJSON Feature 格式:
#### 1.4 更新机制
**HTTP 轮询**(不使用 WebSocket
**前端聚合结果拉取 + 后端实时采集**
- 前端 `setInterval(fetchVessels, 5 * 60 * 1000)` 定期拉取最新快照
- 后端 Collector 每 60s 从 BarentsWatch 拉取并写库,`vessel_latest` 物化视图随时可查
- WebSocket 留给告警/事件驱动场景BGP、系统通知不混入周期性位置刷新
- 换用 AISHub / MarineTraffic 实时流后,届时再评估是否升级为 WebSocket delta push
- 后端 BarentsWatch collector 继续以 HTTP polling 方式采集
- AISStream 等实时源以独立 WebSocket collector 写入原始观测层
- 展示接口从聚合服务读取当前船只视图,而不是由单个 collector 决定最终展示值
- 前端默认不再给 `/geo/vessels``limit=5000``VESSEL_CONFIG.maxRenderedMarkers = 0` 表示不做前端数量裁剪;后续如性能不足再引入显式 LOD 上限
- marker 颜色、详情卡、hover 和搜索结果必须共享 `vessel_type_display` 船型归一化结果,避免 AIS 数字类型码已驱动颜色但卡片仍显示 `Other`
- 前端是否升级为 WebSocket delta push 是独立优化,不影响后端采集器可以使用 WebSocket 接上游实时源
---
@@ -189,8 +198,8 @@ GeoJSON Feature 格式:
| 相机距离 | 渲染策略 |
|---------|---------|
| > 400 | 仅渲染 top 1000 艘(按数据新鲜度 + 船型优先级) |
| 200400 | 渲染 top 5000 艘 |
| > 400 | 默认渲染当前接口返回的全部船只;如性能不足,再引入可配置 LOD 上限 |
| 200400 | 默认渲染当前接口返回的全部船只;如性能不足,再引入可配置 LOD 上限 |
| < 200 | 渲染当前视口 bbox 内全部船只 |
前端根据相机位置动态计算 bbox附加到 API 请求中。

View File

@@ -17,12 +17,16 @@ What belongs here:
- Earth layer style property index
- Backend runtime control
- Collector status
- Collector settings and connectivity validation
- Earth Interactable integration
- Collection format conventions
## Entry Points
- [quickstart.md](/home/ray/dev/linkong/planet/docs/technical/en/quickstart.md): The shortest path to getting Planet running from scratch
- [manual.md](/home/ray/dev/linkong/planet/docs/technical/en/manual.md): Complete usage guide for the console, `planet.sh`, Earth, and Docs
- [Quickstart](/home/ray/dev/linkong/planet/docs/technical/en/quickstart.md): The shortest path to getting Planet running from scratch
- [Planet Manual](/home/ray/dev/linkong/planet/docs/technical/en/manual.md): Complete usage guide for the console, `planet.sh`, Earth, and Docs
- [Collector Settings and Connectivity Validation](/home/ray/dev/linkong/planet/docs/technical/en/datasource-collector-settings-connectivity.md): Data source catalog, collector settings, connectivity validation, and BarentsWatch credentials
- [Earth Interactable Usage](/home/ray/dev/linkong/planet/docs/technical/en/earth-interactable-usage.md): API, lifecycle, and integration examples for Earth surface icon Interactable
What does not belong here:
@@ -32,4 +36,4 @@ What does not belong here:
Those belong in:
- [docs/plans/README.md](/home/ray/dev/linkong/planet/docs/plans/README.md)
- [Plans Index](/home/ray/dev/linkong/planet/docs/plans/README.md)

View File

@@ -84,6 +84,8 @@ async def run(self, db):
| HuggingFace Spaces | space | Demo applications | 1 day |
| PeeringDB | ixp/network/facility | Internet exchange points / networks / facilities | 1-2 days |
| TeleGeography | submarine_cable | Submarine cable information | 7 days |
| BarentsWatch AIS | vessel | AIS vessel positions, speed, heading, MMSI, and related fields | Collector settings |
| AISStream Vessels | vessel_ais | AIS WebSocket realtime stream, written to the raw observation layer and displayed through aggregation | Collector settings |
## IV. Data Format (stored in CollectedData table)

View File

@@ -0,0 +1,350 @@
# Collector Settings and Connectivity Validation
## Background
The console now separates the "data source catalog" from "collector configuration":
- `/datasources`
- Lists all data sources, including built-in and custom sources.
- Clicking a name only opens an information drawer.
- Focuses on status, manual collection, and running collection tasks.
- `/settings?tab=collector_credentials`
- Displays as "Collector Settings".
- Owns endpoint, headers, base parameters, and credentials.
- Every collector exposes a connection button for health checks.
This reduces first-use confusion: API endpoints, headers, credentials, and custom source configuration all belong to collector settings instead of being scattered across the data source list and system settings.
## User-Facing Rules
Connection state is not a frontend styling state. The backend derives it from the current configuration checksum and previously validated records.
A built-in collector is considered "connected" when either condition is true:
- The current configuration has successfully collected data.
- The user clicked the connection button for the current configuration and backend validation succeeded.
If endpoint, headers, base configuration, or credential fingerprint changes after the last successful validation, the state returns to "needs reconnection".
## Frontend Entry Points
### Data Source Catalog
Files:
- [DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/DataSources/DataSources.tsx)
- [index.css](/home/ray/dev/linkong/planet/frontend/src/index.css)
Current behavior:
- Built-in and custom data sources are merged into a `UnifiedDataSource` list.
- The table only keeps view, collect, and status actions.
- Clicking the name opens a read-only drawer.
- The drawer shows:
- Whether the source is built in
- Whether it is enabled
- Module, priority, and frequency
- Endpoint
- Headers
- Base configuration
- Whether credentials are required
- When tasks are running, the top progress area shows a clickable `Collecting N` pill.
- Clicking `Collecting N` opens a task list modal with per-task progress.
`data-source-bulk-toolbar__running-pill` is the styling entry point for the "Collecting" pill. It is aligned with other status tags, while hover treatment, arrow affordance, and blue outline indicate interactivity.
### Collector Settings
File:
- [Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Settings/Settings.tsx)
Current behavior:
- The `collector_credentials` tab is displayed as "Collector Settings".
- A select lists all built-in collectors.
- The only button beside the select is a plug icon for health checks.
- Status tags below the select show:
- `Credentials required` / `No credentials required`
- Module
- `Enabled` / `Disabled`
- `Unchecked` / `Available` / `Unavailable`
- Whether the endpoint is overridden
- Collectors that require credentials place the credential card above base configuration.
- Collectors without credentials only show base configuration.
The connection button uses an inline Tabler-style plug icon with `plug-connected` semantics, avoiding the older refresh icon for a connection action.
## Backend APIs
### Data Source Configuration List
```http
GET /api/v1/datasources/configs/all
```
Returns a merged view of YAML default data sources and database overrides. This route must be declared before `/configs/{config_id}`; otherwise FastAPI treats `all` as a path parameter and returns 422.
Returned fields include:
- `name`
- `default_url`
- `endpoint`
- `is_overridden`
- `is_active`
- `source_type`
- `auth_type`
- `headers`
- `config`
- `config_id`
- `description`
Before returning `config`, internal connectivity validation fields are removed so the frontend does not display validation metadata as user configuration.
### Built-In Collector Connection Status
```http
POST /api/v1/datasources/configs/builtin/connection-status
```
Purpose:
- Accept a candidate configuration.
- Compute its checksum.
- Determine whether the current configuration is already connected.
The current frontend mostly performs an immediate check through the connection button and does not strongly depend on this endpoint. It remains the backend basis for future save-button disabling and restoring initial page state.
### Built-In Collector Connectivity Validation
```http
POST /api/v1/datasources/configs/builtin/connect
```
Purpose:
- Free collectors request the endpoint directly.
- Credentialed collectors go through their credential provider.
- Successful validation writes a system-level connection record.
Successful responses include:
- `success`
- `connected`
- `checksum`
- `stage`
- `message`
- `response_time_ms`
- `credential_provider`
- `credential_source`
### BarentsWatch AIS Connectivity Validation
```http
POST /api/v1/settings/integrations/barentswatch/connect
GET /api/v1/settings/integrations/barentswatch/connectivity
```
BarentsWatch uses separate endpoints because draft credentials must be validated before saving:
- Use draft `client_id` / `client_secret` to fetch a token.
- Use that token to request the AIS endpoint.
- After success, write a built-in collector connection record using the draft credential fingerprint.
## Connectivity Validation Service
File:
- [datasource_connectivity.py](/home/ray/dev/linkong/planet/backend/app/services/datasource_connectivity.py)
Core responsibilities:
- Compute built-in collector configuration checksums.
- Read credentials from environment variables and `~/.zshrc`.
- Determine whether the current configuration is already connected.
- Run endpoint health checks.
- Save successful connection records.
### Checksum Inputs
The checksum includes:
- Collector name
- Endpoint
- Auth type
- Headers
- Config after removing internal validation fields
- Credential provider
- Credential fingerprint
The credential fingerprint is a hash of credential content. Plaintext credentials are not written into connection records.
### Connection Records
Successful connection records are written to `SystemSetting`:
```text
category = datasource_connectivity_validations
```
The payload uses collector source as the key:
```json
{
"barentswatch_vessels": {
"checksum": "...",
"status": "success",
"validated_at": "2026-04-29T00:00:00+00:00",
"status_code": 200,
"credential_source": "datasource_config",
"connected_by": "connection_button"
}
}
```
`connected_by` currently has two sources:
- `connection_button`: the user manually clicked the connection button.
- `collection`: a collection task completed successfully, so the system recorded the current effective configuration as connected.
### Successful Collection Means Connected
After a successful collection, the scheduler writes a connection record:
- [scheduler.py](/home/ray/dev/linkong/planet/backend/app/services/scheduler.py)
This prevents collectors that already have data from asking the user to validate again. Reconnection is only required when the configuration checksum changes.
## BarentsWatch AIS Credential Chain
Files:
- [barentswatch.py](/home/ray/dev/linkong/planet/backend/app/services/barentswatch.py)
- [vessel_ais.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/vessel_ais.py)
Resolution priority:
1. `DataSourceConfig.auth_config`
2. `DataSourceConfig.config`
3. Environment variables
4. `~/.zshrc`
Supported environment variables:
```bash
export BARENTSWATCH_CLIENT_ID="..."
export BARENTSWATCH_CLIENT_SECRET="..."
```
Historical misspellings are also supported:
```bash
export BARRENTSWATCH_CLIENT_ID="..."
export BARRENTSWATCH_CLIENT_SECRET="..."
```
Token request rules:
- Token URL: `https://id.barentswatch.no/connect/token`
- `Content-Type`: `application/x-www-form-urlencoded`
- Body:
- `grant_type=client_credentials`
- `client_id`
- `client_secret`
- `scope=ais`
AIS request rules:
- Default endpoint: `https://live.ais.barentswatch.no/v1/latest/combined`
- Header: `Authorization: Bearer <access_token>`
`VesselAISCollector` no longer reads environment variables directly. It goes through `resolve_barentswatch_config()` and `fetch_barentswatch_access_token()` so settings, connectivity validation, and collection do not fork into three credential flows.
## AISStream Collector Chain
Files:
- [aisstream.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/aisstream.py)
- [vessel_ais_aggregation.py](/home/ray/dev/linkong/planet/backend/app/services/vessel_ais_aggregation.py)
AISStream uses a WebSocket realtime stream. The collector writes only to the `ais_raw_observations` raw observation layer; it does not directly overwrite the final vessel display table. The aggregation API handles multi-source deduplication, field selection, and conflict records.
Configuration:
- `api_key`: stored in `DataSourceConfig.auth_config`, or provided through `AISSTREAM_API_KEY`.
- `endpoint`: defaults to `wss://stream.aisstream.io/v0/stream`.
- `message_types`: defaults to `PositionReport` and `ShipStaticData`.
- `bounding_boxes`: AISStream format is `[[[lat_min, lon_min], [lat_max, lon_max]]]`; the settings page provides global, Norway / North Sea, Europe coast, East Asia, and North America coast presets.
- `max_messages` and `receive_timeout_seconds`: control the batch-style WebSocket collection window.
Normalization:
- `PositionReport` mainly provides position, speed, course, heading, and navigation status.
- Vessel names can be filled from `MetaData.ShipName` even when the message body has no `name`.
- Vessel type usually comes from lower-frequency `ShipStaticData.Type`; the backend maps AIS numeric type codes to Cargo / Tanker / Passenger / Fishing / Military.
- If a vessel has not yet produced a static message, its aggregated type can still be `Other`; v5 vessel profile enrichment is planned to fill that gap.
## Credential Guide
File:
- [credential_guides.py](/home/ray/dev/linkong/planet/backend/app/services/credential_guides.py)
APIs:
```http
GET /api/v1/settings/credential-guides/{provider}
POST /api/v1/settings/credential-guides/{provider}/generate
POST /api/v1/settings/credential-guides/{provider}/reset
```
Currently supported:
- `barentswatch`
- `aisstream`
The default guide includes the official BarentsWatch tutorial:
```text
https://developer.barentswatch.no/docs/tutorial
```
If the user clicks that the tutorial is not useful, the backend sends the default prompt to AI Provider, generates a new Chinese tutorial, and saves it to `SystemSetting`:
```text
category = collector_credential_guides
```
Reset deletes the custom tutorial and restores the default guide.
## Save Rules
When built-in collector configuration is saved, the internal `connectivity_validation` field is removed so validation state does not mix with user configuration.
BarentsWatch `client_secret` has special handling:
- The input shows a masked preview.
- If the submitted value still matches the masked preview, the backend keeps the old secret.
- If a new value is submitted, the secret is replaced.
- The previous separate "clear current secret" checkbox is no longer provided.
## Test Coverage
Related tests:
- [test_vessels.py](/home/ray/dev/linkong/planet/backend/tests/test_vessels.py)
Added coverage:
- BarentsWatch credentials can be parsed from `~/.zshrc`.
- When environment variables are empty, `resolve_barentswatch_config()` can fall back to `~/.zshrc`.
- Vessel data conversion and GeoJSON output remain compatible.
## Current Provider Coverage
Credential providers currently supported:
- `barentswatch`
- `spacetrack`
Other collectors with `requires_credentials=true` return that their credential chain has not been wired yet, and the frontend shows `Unavailable`.

View File

@@ -187,7 +187,7 @@ Current reality:
- that is expected, because incidents are aggregated and de-noised
- but incident-first rendering makes the Earth view look too quiet unless there is another always-available activity layer
Implementation detail for the recommended `activity layer` is expanded in [bgp-region-aggregation-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-bgp-region-aggregation-plan.md).
Implementation detail for the recommended `activity layer` is expanded in the [BGP Region Aggregation Plan](/home/ray/dev/linkong/planet/docs/plans/earth-bgp-region-aggregation-plan.md).
So the immediate next milestone is:

View File

@@ -4,8 +4,8 @@ This document describes the current real structure of the Earth display frontend
Related references:
- [rules.md](/home/ray/dev/linkong/planet/rules.md)
- [frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/en/frontend-layout-guidelines.md)
- [Project Rules](/home/ray/dev/linkong/planet/rules.md)
- [Frontend Layout Guidelines](/home/ray/dev/linkong/planet/docs/technical/en/frontend-layout-guidelines.md)
## Current Goal
@@ -96,6 +96,7 @@ Responsibilities:
- [satellites.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/satellites.js)
- [cables.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/cables.js)
- [vessels.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/vessels.js)
- [bgp.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp.js)
- [bgp-cruise-adapter.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp-cruise-adapter.js)
- [compute-centers.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/compute-centers.js)
@@ -108,6 +109,14 @@ Each module is responsible for its own:
- State tracking (loaded, visible, hover, locked)
- Self-cleanup (dispose on scene destroy)
### AIS Vessel Layer
The vessel layer fetches `/api/v1/visualization/geo/vessels` and renders the aggregated AIS GeoJSON through `createInteractableLayer()`. By default it does not send a `limit` parameter, and `VESSEL_CONFIG.maxRenderedMarkers = 0` means the frontend does not clip the result to 5000 vessels. A positive `options.limit` or positive `maxRenderedMarkers` can still be used as an explicit temporary cap.
Vessel color and vessel type text must use the same normalized classification. `vessels.js` derives `type` from both `vessel_type_name` and the AIS numeric `vessel_type` code; that `type` drives marker color. It also derives `vessel_type_display`, which `main.js` uses for the info card, hover summary, and search result subtitle. Do not make the info card read only the raw `vessel_type_name`, because AISStream can provide a numeric type while the raw name is still `Other`.
AISStream `PositionReport` messages commonly carry live position and `MetaData.ShipName`, while vessel type usually comes from lower-frequency `ShipStaticData.Type`. The backend normalizes `MetaData.ShipName` into the vessel name and maps numeric type codes into Cargo / Tanker / Passenger / Fishing / Military where available. Missing type detail should wait for a static AIS message or the planned vessel profile enrichment; the frontend should not invent a more specific type.
### 7. HUD Panels and Search
- [hud-panels.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/hud-panels.js)
@@ -249,4 +258,4 @@ Therefore:
For console structure, see:
- [frontend-admin-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/en/frontend-admin-frontend-context.md)
- [Admin Frontend Context](/home/ray/dev/linkong/planet/docs/technical/en/frontend-admin-frontend-context.md)

View File

@@ -0,0 +1,270 @@
# Earth Interactable Usage
`Interactable` is the shared rendering entry point for icon-like interactive elements on the Earth surface. It extracts the pattern proven by the vessel layer into reusable behavior: normal state uses batched `THREE.Points`, hover and locked states use small overlays, picking uses screen-space hit testing, icon assets are normalized into canvas textures, and the shared layer handles glow, state, size, ground rendering, and same-coordinate avoidance.
Currently integrated layers:
| Layer | Business File | Icon Source | Extra Animation |
| --- | --- | --- | --- |
| AIS vessels | `frontend/public/earth/js/vessels.js` | canvas draw, moving triangle / anchored dot | Vessel tracks are still maintained by the business layer |
| Compute centers | `frontend/public/earth/js/compute-centers.js` | `assets/icons/compute-*.svg` | Estimated-location `?` badge is added through `icon.afterDraw()` |
| BGP events | `frontend/public/earth/js/bgp.js` | canvas draw, symbol by event type | Expanding rings are still maintained by the BGP business layer |
| BGP observers | `frontend/public/earth/js/bgp.js` | `assets/icons/bgp-broadcast-pin.svg` | Halo, activity core, coverage wedge, and radar sweep remain in the BGP business layer |
Landing sites were previously attempted on Interactable, but pin-style SVGs were fragmented by `THREE.Points` depth testing near the Earth edge. They now use a dedicated `THREE.Sprite` path with a yellow flat-sphere texture generated by canvas. The old SVG assets remain in `assets/icons/`, but landing sites no longer depend on SVG at runtime.
## Why Interactable Exists
Before this layer, each surface icon layer could easily reimplement its own version of:
- icon texture generation
- hover / locked state
- glow styling
- picking radius
- zoom-dependent size strategy
- overlap avoidance for identical coordinates
When this logic is scattered across business files, visual behavior drifts and later tuning becomes layer-by-layer repair. The boundary of `Interactable` is: the shared layer owns how icons remain stable on Earth and how they are selected; the business layer owns where data comes from, what the icon means, what detail cards show, and whether extra animation exists.
## Entry Point
```javascript
import { createInteractableLayer } from "./interactable.js";
```
Core call shape:
```javascript
const layer = createInteractableLayer({
id: "example",
objectType: "example_object",
renderOrder: 4.4,
altitudeOffset: 0.2,
pointSize: 34,
icon: {
draw(context, options) {
// draw canvas icon
},
},
getPosition: (item) => ({
latitude: item.latitude,
longitude: item.longitude,
}),
getKind: (item) => item.kind || "default",
});
```
Business modules usually expose only a thin wrapper:
```javascript
export function getExampleMarkers() {
return layer.getMarkers();
}
export function getExamplePointerIntersections(options) {
return layer.getPointerIntersections(options);
}
export function setExampleMarkerState(marker, state = "normal") {
layer.setMarkerState(marker, state);
}
export function updateExampleVisualState(lockedObjectType, lockedObject, camera) {
layer.updateVisualState(lockedObjectType, lockedObject, camera);
}
```
## Configuration
| Option | Default | Description |
| --- | --- | --- |
| `id` | required | Unique layer id used for group name, avoidance registration, and debug. |
| `objectType` | `id` | Business type written to `marker.userData.type`; the main interaction layer uses it to identify locked objects. |
| `renderOrder` | `4` | Base render order for normal points and hover / locked overlays. |
| `altitudeOffset` | `0.2` | Business altitude, used as `CONFIG.earthRadius + altitudeOffset` for the original surface position. |
| `pointSize` | `32` | Base screen pixel size used by both normal points and overlays. |
| `sizeMode` | `"fixed"` | Fixed screen size by default; non-`"fixed"` modes scale by camera distance. |
| `sizeScale` | `{ referenceFov: 75, min: 0.12, max: 3 }` | Scaling bounds when `sizeMode !== "fixed"`. |
| `atlasCellSize` | `128` | Canvas texture cell size for icons. |
| `colors` | `{}` | Supports `normal`, flattened kind keys, and `byKind`. |
| `opacity` | `{ normal: 0.88, dimmed: 0.26, hover: 0.98, locked: 1 }` | Opacity per state. |
| `stateScale` | `{ hover: 1, locked: 1, dimmed: 1 }` | Size multiplier per state. |
| `pulse` | `{}` | Optional locked-state breathing scale, with `enabled`, `speed`, and `amplitude`. |
| `avoidance` | `{ enabled: true, precision: 4, radius: 1.1, step: 0.35 }` | Same-coordinate avoidance across Interactable layers. |
| `icon` | required | Icon source, supporting canvas draw, SVG / image asset, state asset, anchor, and post-processing. |
| `getPosition(item)` | required | Returns `{ latitude, longitude }` or `THREE.Vector3`. |
| `getKind(item)` | `item.type || "default"` | Returns a business kind for color and texture buckets. |
| `getRotationBin(marker)` | `0` | Returns a rotation bucket, such as 32 heading buckets for vessels. |
| `getBucketKey(marker)` | `String(getRotationBin(marker))` | Returns a texture / geometry bucket key. |
| `getPointSizeMultiplier(marker)` | `1` | Per-marker size multiplier. BGP events use severity; observers use activity. |
| `getUserData(item)` | `item` | Business fields written onto the marker. |
## Icon Configuration
`icon.anchor` is optional and defaults to `{ x: 0.5, y: 0.5 }`, meaning the texture center aligns with the marker coordinate. It is only suitable for small visual anchor offsets. If the icon body is large and must remain fully visible at the Earth edge, such as the old landing-site pin, it should not be forced through `THREE.Points + depthTest`; the body will be clipped by Earth depth.
### Canvas Icons
Canvas icons fit vessels and BGP events where symbols need to be drawn dynamically by state or rotation:
```javascript
const vesselIconLayer = createInteractableLayer({
id: "vessels",
objectType: "vessel",
pointSize: 34,
icon: {
draw(context, { marker, rotationBin = 0, glow = false, color = "#ffffff" }) {
if (!marker.userData.anchored) {
context.rotate((rotationBin / 32) * Math.PI * 2);
}
context.fillStyle = color;
context.shadowColor = color;
context.shadowBlur = glow ? 14 : 0;
context.beginPath();
context.moveTo(0, -37);
context.lineTo(28, 32);
context.lineTo(0, 17);
context.lineTo(-28, 32);
context.closePath();
context.fill();
},
},
getRotationBin: getCourseBin,
getBucketKey: (marker) => `${marker.userData.anchored ? "anchored" : "moving"}:${getCourseBin(marker)}`,
});
```
When `icon.coordinates !== "canvas"`, `Interactable` translates the context to the atlas center first. Vessel-style icons that already draw around center coordinates do not need to declare `coordinates`.
### SVG / Image Asset Icons
Asset icons fit facilities such as compute centers and BGP observers:
```javascript
const computeCenterIconLayer = createInteractableLayer({
id: "computeCenters",
objectType: "compute_center",
pointSize: 36,
atlasCellSize: 128,
icon: {
coordinates: "canvas",
colorable: false,
fitSize: 60,
glowBlur: 16,
getSource({ marker, item }) {
const siteType = marker?.userData?.site_type || item?.site_type || "gpu_cluster";
return COMPUTE_CENTER_ICON_SOURCES[siteType];
},
afterDraw(context, { marker, item }) {
if (marker?.userData?.is_estimated ?? item?.is_estimated) {
drawComputeCenterEstimatedBadge(context, true);
}
},
},
});
```
Asset conventions:
- SVG / image files live in `frontend/public/earth/assets/icons/` and are referenced as `/earth/assets/icons/name.svg`.
- Original SVGs should keep a standard `viewBox` and paths; avoid hard-coding transform only for display size.
- Display size is controlled by `icon.fitSize`; it can be a number, `{ width, height }`, or a function.
- If `icon.colorable !== false` and state colors are provided, the shared layer first draws the asset to a temporary canvas and then tints it with `source-in`.
- Multicolor images or SVGs that should not be tinted must set `colorable: false`.
## Lifecycle
Typical load flow:
```javascript
export async function loadExampleLayer(_scene, earth) {
clearExampleData(earth);
const markerData = await fetchExampleData();
await layer.preloadAssets(markerData);
layer.setData(markerData);
layer.attach(earth);
layer.setVisible(showExampleLayer);
return { totalCount: layer.getCount() };
}
```
Method responsibilities:
| Method | Description |
| --- | --- |
| `preloadAssets(items)` | Collects asset sources that may be used by normal / hover / locked states and preloads them with browser `Image`. Canvas-drawn icons can skip this. |
| `setData(items)` | Clears old points, creates markers, registers avoidance, and rebuilds `THREE.Points` by bucket. |
| `attach(parent)` | Mounts the layer group onto the Earth root. |
| `setVisible(next)` | Controls visibility for the group, points, and overlays. |
| `setMarkerState(marker, state)` | Sets `normal` / `hover` and other states, then invalidates visual state. |
| `updateVisualState(focusType, focusObject, camera)` | Updates normal opacity / size and refreshes hover / locked overlays. |
| `getPointerIntersections(options)` | Runs screen-space picking and returns hits sorted by pixel distance. |
| `clearData(parent)` | Unregisters avoidance, disposes geometry / material, clears markers, and removes the group from the parent. |
## Picking Integration
`Interactable` does not depend on the default Three.js raycast for `Points`. The main interaction layer passes Earth, camera, pointer, and hit radius:
```javascript
const intersects = getVesselPointerIntersections({
earth,
camera,
pointer,
radiusPx: 22,
width: window.innerWidth,
height: window.innerHeight,
});
```
The shared layer:
1. Converts the camera position into Earth-local coordinates.
2. Skips markers on the back side.
3. Projects marker world position into screen coordinates.
4. Uses `radiusPx` for pixel-distance hits.
5. Returns the nearest candidate objects.
Earth dragging, inertia, and hover throttling still belong to `main.js` because they depend on global input state.
## Same-Coordinate Avoidance
Avoidance is enabled by default and applies to all layers created through `createInteractableLayer()`. The shared layer builds an `icon_avoidance_key` from latitude / longitude or `THREE.Vector3`, then arranges markers with the same key into a small circle along the surface tangent plane.
Key points:
- `icon_base_position` keeps the original business position.
- Avoidance only changes rendering and picking position. It does not change business latitude / longitude.
- When a single marker returns to its original position, it uses the business surface position computed from `altitudeOffset`.
- When multiple markers share coordinates, the first ring uses `avoidance.radius`; later rings add `avoidance.step`.
If a business layer must stay exactly on the original point, disable avoidance explicitly:
```javascript
createInteractableLayer({
id: "strict-layer",
avoidance: { enabled: false },
});
```
## Business Animation Boundary
`Interactable` currently owns only the icon body and common hover / locked overlays. Complex animations remain in business modules, but should follow the Interactable marker position:
- BGP event expanding rings are independent ring sprites created by `bgp.js`, updated every frame with `position.copy(marker.position)`.
- BGP observer halo, status core, coverage halo, and coverage wedge are managed by `bgp.js`; the icon body is managed by Interactable.
- Vessel tracks remain in `vessels.js` because they depend on track data loaded after a click.
This boundary avoids pushing every animation type into the shared interface too early. If multiple layers reuse the same animation type later, it can move into an Interactable `animations` extension.
## New Layer Checklist
1. Prepare marker data in the business file and keep required business fields.
2. Choose an icon type: canvas draw, SVG / image asset, or dynamic `getSource()`.
3. Configure `pointSize`, `icon.fitSize`, `colors`, `opacity`, and `stateScale`.
4. Provide `getPointSizeMultiplier()` if business-specific size variation is needed.
5. Provide `getRotationBin()` and a stable `getBucketKey()` if rotation exists.
6. During load, call `preloadAssets()` before `setData()`, `attach()`, and `setVisible()`.
7. Wire `getPointerIntersections()` in `main.js` and reuse the existing hover / locked state update flow.
8. Record altitude, `renderOrder`, `pointSize`, and animation ordering in the layer style index and render order documents.

View File

@@ -1,6 +1,6 @@
# Earth Layer Style Property Index
This document records the material, color, opacity, line width, radius offset, and `renderOrder` style properties of all Earth frontend layers. For layer ordering relationships, see [earth-render-layer-order.md](/home/ray/dev/linkong/planet/docs/technical/en/earth-render-layer-order.md).
This document records the material, color, opacity, line width, radius offset, and `renderOrder` style properties of all Earth frontend layers. For layer ordering relationships, see [Earth Render Layer Order](/home/ray/dev/linkong/planet/docs/technical/en/earth-render-layer-order.md).
## Naming Conventions
@@ -74,6 +74,8 @@ This document records the material, color, opacity, line width, radius offset, a
## Land/Ocean Base and Country Borders
The land/ocean base is an Earth base-map asset and preloads at startup; the "Border Lines" layer toggle only controls normal border lines, hover lines, and interactive hover.
| Name | Variable | Current Value | Location / Notes |
| --- | --- | --- | --- |
| Country border data path | `COUNTRY_BOUNDARY_CONFIG.dataPath` | `"/earth/data/countries-admin0.min.geojson"` | GeoJSON input |
@@ -89,11 +91,11 @@ This document records the material, color, opacity, line width, radius offset, a
| Border line color | `COUNTRY_BOUNDARY_CONFIG.lineColor` | `0x7fc7ff` | Normal border line |
| Border line opacity | `COUNTRY_BOUNDARY_CONFIG.lineOpacity` | `0.58` | Normal border line opacity |
| Border dimmed opacity on hover | `COUNTRY_BOUNDARY_CONFIG.dimmedLineOpacity` | `0.18` | Normal border opacity during hover |
| Border line radius offset | `COUNTRY_BOUNDARY_CONFIG.lineAltitudeOffset` | `0.24` | Normal border line radius |
| Border line radius offset | `COUNTRY_BOUNDARY_CONFIG.lineAltitudeOffset` | `0.115` | Normal border line radius; slightly above HD texture `0.10` and below terrain base `0.16` to reduce floating |
| Border line renderOrder | `COUNTRY_BOUNDARY_CONFIG.lineRenderOrder` | `2.2` | Normal border line level |
| Border hover color | `COUNTRY_BOUNDARY_CONFIG.hoverLineColor` | `0xff3b1f` | Neon red-orange |
| Border hover opacity | `COUNTRY_BOUNDARY_CONFIG.hoverLineOpacity` | `1.0` | Hover line opacity |
| Border hover radius offset | `COUNTRY_BOUNDARY_CONFIG.hoverAltitudeOffset` | `0.32` | Hover line radius |
| Border hover radius offset | `COUNTRY_BOUNDARY_CONFIG.hoverAltitudeOffset` | `0.14` | Hover line radius; close to the surface but above normal border lines |
| Border hover renderOrder | `COUNTRY_BOUNDARY_CONFIG.hoverLineRenderOrder` | `2.3` | Hover line level |
| Border hover glow opacity | `COUNTRY_BOUNDARY_CONFIG.hoverGlowOpacity` | `0.38` | Glow line opacity |
| Border hover glow line width | `COUNTRY_BOUNDARY_CONFIG.hoverGlowLineWidth` | `3` | Glow `LineBasicMaterial.linewidth` |
@@ -140,17 +142,18 @@ This document records the material, color, opacity, line width, radius offset, a
| Cable line width | `CABLE_CONFIG.line.lineWidth` | `1` | `LineBasicMaterial.linewidth` |
| Cable opacity | `CABLE_CONFIG.line.opacity` | `1.0` | Cable line opacity |
| Cable renderOrder | `CABLE_CONFIG.line.renderOrder` | `1` | Cable line level |
| Landing point radius offset | `CABLE_CONFIG.landingPoint.altitudeOffset` | `0.48` | Aligns with compute center marker height |
| Landing point icon texture size | `CABLE_CONFIG.landingPoint.textureSize` | `256` | Canvas size for solid map-pin icon |
| Landing point icon aspect ratio | `CABLE_CONFIG.landingPoint.iconAspectRatio` | `0.82` | `Sprite.scale.x = height * aspect` |
| Landing point icon anchor | `CABLE_CONFIG.landingPoint.anchorX / anchorY` | `0.52 / 0.276` | `Sprite.center`, aligns pin tip to landing point lat/lon |
| Landing point base scale | `CABLE_CONFIG.landingPoint.baseScale` | `12` | Matches compute center sprite height |
| Landing point radius offset | `CABLE_CONFIG.landingPoint.altitudeOffset` | `0.2` | Same surface height as cable lines, avoiding a floating marker |
| Landing point sprite height | local `LANDING_POINT_SPRITE_HEIGHT` | `3` | `THREE.Sprite` base height |
| Landing point reference FOV | local `LANDING_POINT_SIZE_REFERENCE_FOV` | `75` | Matches the current Earth camera FOV |
| Landing point scale minimum | local `LANDING_POINT_SIZE_SCALE_MIN` | `0.16` | Minimum multiplier after roughly 200% zoom, limiting high-zoom screen footprint; `3 * 0.16 = 0.48` |
| Landing point scale maximum | local `LANDING_POINT_SIZE_SCALE_MAX` | `3` | Maximum multiplier at far distance; current minimum zoom reaches roughly `2.50` |
| Landing point atlas size | local `LANDING_POINT_ATLAS_CELL_SIZE` | `128` | Canvas flat shaded sphere texture size |
| Landing point color | `CABLE_CONFIG.landingPoint.color` | `0xffaa00` | `SpriteMaterial.color` |
| Landing point opacity | `CABLE_CONFIG.landingPoint.opacity` | `1.0` | `SpriteMaterial.opacity` |
| Landing point renderOrder | `CABLE_CONFIG.landingPoint.renderOrder` | `4.5` | Aligns with compute center surface level |
| Landing point renderOrder | `CABLE_CONFIG.landingPoint.renderOrder` | `1` | Same level as cable lines; `depthTest: false` keeps the ball whole, while camera-to-center globe occlusion hides back-side points |
| Landing point dim brightness | `landingPointVisual.dimBrightness` | `0.62` | Dim state color multiplier |
| Related landing point opacity | `landingPointVisual.related.opacityBase / opacityPulse` | `0.8 / 0.2` | Highlight pulse |
| Dimmed landing point color | `landingPointVisual.dimmed.colorRGB` | `{ r: 180, g: 116, b: 28 }` | Dim state color; avoids dark base showing through as a dark hole |
| Dimmed landing point emissive | `landingPointVisual.dimmed.emissive` | `0x3a2200` | Dim state weak amber self-emission |
| Dimmed landing point opacity | `landingPointVisual.dimmed.opacity` | `0.78` | Dim state opacity; no longer uses low alpha blending with dark base |
## Satellites, Trails, and Footprints
@@ -168,18 +171,37 @@ This document records the material, color, opacity, line width, radius offset, a
| Satellite trail line width | `SATELLITE_CONFIG.trailLineWidth` | `3` | Ribbon shader uniform |
| Selected ring size | `SATELLITE_CONFIG.ringSize` | `0.07` | Hover / locked ring sprite |
| Satellite overlay renderOrder | `SATELLITE_CONFIG.overlayRenderOrder` | `12` | Locked ring / halo / orbit |
| Footprint renderOrder | local `GROUND_FOOTPRINT_RENDER_ORDER` | `3` | Footprint fill |
| Footprint renderOrder | local `GROUND_FOOTPRINT_RENDER_ORDER` | `3` | Starlink footprint fill and Iridium coverage ring; must stay above land / texture / terrain surface layers |
## AIS Vessels
| Name | Variable | Current Value | Location / Notes |
| --- | --- | --- | --- |
| Vessel radius offset | `VESSEL_CONFIG.altitudeOffset` | `0.2` | Normal marker position, close to the real terrain base layer |
| Vessel track radius offset | `VESSEL_CONFIG.track.altitudeOffset` | `0.2` | Selected vessel track line, aligned to the vessel marker radius; the frontend anchors the track endpoint to the current marker position |
| Vessel renderOrder | local `VESSEL_RENDER_ORDER` | `4.4` | Normal marker and interactive overlay |
| Vessel track renderOrder | `VESSEL_RENDER_ORDER - 0.1` | `4.3` | Below vessel markers |
| Vessel point pixel size | local `VESSEL_POINT_SIZE` | `34` | Shared size for normal markers and hover / locked overlays |
| Default vessel render cap | `VESSEL_CONFIG.maxRenderedMarkers` | `0` | `0` means the frontend does not clip by default; positive values send `limit` and clip markers |
| Vessel texture canvas size | local `VESSEL_ATLAS_CELL_SIZE` | `128` | Canvas point texture |
| Course bucket count | local `VESSEL_COURSE_BINS` | `32` | Moving vessels are bucketed by COG to reduce draw calls while preserving direction |
| Vessel hover picking throttle | local `VESSEL_HOVER_PICK_INTERVAL_MS` | `100` | `main.js` hover picking |
| Vessel screen hit radius | local `VESSEL_POINTER_RADIUS_PX` | `22` | `main.js` screen-space picking |
AIS vessel markers use batched `THREE.Points`, not one `THREE.Sprite` per vessel. Moving vessels stay triangular, anchored or slow vessels stay circular, and hover / locked states add a same-size glow overlay. Vessel type color and info-card type text must come from the same normalized result: `vessels.js` reads both backend `vessel_type_name` and AIS numeric `vessel_type`, derives the color-driving `type`, then exposes `vessel_type_display` for the info card, hover summary, and search results.
## Compute Centers
| Name | Variable | Current Value | Location / Notes |
| --- | --- | --- | --- |
| Compute center radius offset | `COMPUTE_CENTER_CONFIG.altitudeOffset` | `0.48` | Marker position |
| Compute center base opacity | `COMPUTE_CENTER_CONFIG.marker.baseOpacity` | `0.88` | `SpriteMaterial.opacity` |
| Supercomputer marker scale | `COMPUTE_CENTER_CONFIG.marker.supercomputerScale` | `12` | Supercomputer marker |
| GPU cluster marker scale | `COMPUTE_CENTER_CONFIG.marker.gpuClusterScale` | `12` | GPU marker |
| Hover scale | `COMPUTE_CENTER_CONFIG.marker.hoverScale` | `1.16` | Hover state |
| Locked scale | `COMPUTE_CENTER_CONFIG.marker.lockedScale` | `1.22` | Locked state |
| Compute center point size | local `COMPUTE_CENTER_POINT_SIZE` | `36` | Shared Interactable base size for normal markers and hover / locked overlays |
| Compute center asset fit size | local `COMPUTE_CENTER_ICON_FIT_SIZE` | `60` | Maximum SVG asset draw size inside the `128x128` atlas canvas, controlled by `icon.fitSize` |
| Compute center base opacity | `COMPUTE_CENTER_CONFIG.marker.baseOpacity` | `0.88` | Normal `PointsMaterial.opacity` |
| Supercomputer marker scale | `COMPUTE_CENTER_CONFIG.marker.supercomputerScale` | `12` | Legacy Sprite scale; not directly used by the current Interactable path |
| GPU cluster marker scale | `COMPUTE_CENTER_CONFIG.marker.gpuClusterScale` | `12` | Legacy Sprite scale; not directly used by the current Interactable path |
| Hover scale | `COMPUTE_CENTER_CONFIG.marker.hoverScale` | `1.16` | Hover overlay size multiplier |
| Locked scale | `COMPUTE_CENTER_CONFIG.marker.lockedScale` | `1.22` | Locked overlay size multiplier, with pulse |
| Dimmed scale / opacity | `dimmedScale / dimmedOpacity` | `0.82 / 0.34` | Dim state |
| Supercomputer color | `COMPUTE_CENTER_CONFIG.colors.supercomputer` | `"#38bdf8"` | Marker texture |
| GPU cluster color | `COMPUTE_CENTER_CONFIG.colors.gpu_cluster` | `"#2dd4bf"` | Marker texture |
@@ -190,12 +212,16 @@ This document records the material, color, opacity, line width, radius offset, a
| Name | Variable | Current Value | Location / Notes |
| --- | --- | --- | --- |
| BGP event radius offset | `BGP_CONFIG.altitudeOffset` | `2.1` | Anomaly marker |
| BGP collector radius offset | `BGP_CONFIG.collectorAltitudeOffset` | `1.6` | Collector marker |
| Event base scale | `BGP_CONFIG.marker.eventBaseScale` | `6.2` | Anomaly sprite |
| Collector base scale | `BGP_CONFIG.marker.collectorBaseScale` | `7.4` | Collector plane |
| BGP event radius offset | `BGP_CONFIG.altitudeOffset` | `0.48` | BGP event Interactable marker |
| BGP collector radius offset | `BGP_CONFIG.collectorAltitudeOffset` | `0.2` | BGP collector Interactable marker, aligned with the vessel layer |
| BGP event point size | local `BGP_EVENT_POINT_SIZE` | `34` | Event Interactable base size, adjusted by severity through `getPointSizeMultiplier()` |
| BGP event symbol draw size | local `BGP_EVENT_SYMBOL_SIZE` | `60` | Event canvas symbol draw size inside the `128x128` atlas |
| BGP collector point size | local `BGP_COLLECTOR_POINT_SIZE` | `36` | Collector Interactable base size, adjusted by activity through `getPointSizeMultiplier()` |
| BGP collector asset fit size | local `BGP_COLLECTOR_ICON_FIT_SIZE` | `60` | Maximum `bgp-broadcast-pin.svg` draw size inside the atlas canvas |
| Event base scale | `BGP_CONFIG.marker.eventBaseScale` | `6.2` | Event ring anchor |
| Collector base scale | `BGP_CONFIG.marker.collectorBaseScale` | `7.4` | Collector halo / coverage animation anchor |
| Hover / dim scale | `hoverScale / dimmedScale` | `1.16 / 0.92` | Interaction states |
| Normal event opacity | `BGP_CONFIG.opacity.normal` | `0.78` | Anomaly sprite |
| Normal event opacity | `BGP_CONFIG.opacity.normal` | `0.78` | BGP event Interactable normal state |
| Hover opacity | `BGP_CONFIG.opacity.hover` | `1.0` | Hover state |
| Dimmed opacity | `BGP_CONFIG.opacity.dimmed` | `0.24` | Dim state |
| Collector opacity | `BGP_CONFIG.opacity.collector` | `0.62` | Collector state |

View File

@@ -6,8 +6,8 @@ Note: the layer control panel order and the registration / startup load order ar
| Order type | Current sequence | Notes |
| --- | --- | --- |
| Control panel order | Cables → Trails → Satellites → Compute Centers → BGP → Terrain → HD Texture → Cloud Layer → Borders → Grid | Controlled by `displayOrder`, sorted by operational relevance. |
| Registration / startup load order | Grid → Borders → HD Texture → Cloud Layer → Cables → Compute Centers → BGP → Satellites | Controlled by registration order and `startupPriority`, sorted surface-to-sky; Trails and Terrain are dependency/optional display layers and do not participate in normal startup data loading. |
| Control panel order | Cables → Trails → Satellites → Compute Centers → BGP → Terrain → HD Texture → Cloud Layer → Border Lines → Grid | Controlled by `displayOrder`, sorted by operational relevance. |
| Registration / startup load order | Grid → Border Lines / Land-Ocean Base → HD Texture → Cloud Layer → Cables → Compute Centers → BGP → Satellites | Controlled by registration order and `startupPriority`, sorted surface-to-sky; the startup queue reads persisted layer visibility first, skips normal layers explicitly saved as hidden, and HD Texture does not download the texture when disabled; Border Lines are the exception: the land-ocean base always preloads, while the persisted state only controls interactive border lines and hover; Trails and Terrain are dependency/optional display layers and do not participate in normal startup data loading. |
## Surface Layer Stack
@@ -26,7 +26,7 @@ Note: the layer control panel order and the registration / startup load order ar
| 2.2 | Country borders | `country-boundaries.js` | `lineAltitudeOffset` | Raycast disabled | Only needs to stay above HD texture. |
| 2.29 | Country border hover glow | `country-boundaries.js` | Hover radius + glow offset | `depthTest: false`, raycast disabled | Additive glow to reinforce border edge and terrain hover visibility. |
| 2.3 | Country border hover line | `country-boundaries.js` | `hoverAltitudeOffset` | `depthTest: false`, raycast disabled | Neon red-orange hover line; China and Taiwan share the same highlight group. |
| 3 | Satellite footprint fill | `satellites.js` | `GROUND_FOOTPRINT_RENDER_ORDER` | depth-tested, Group renderOrder stays 0 | Footprint above country borders, below compute centers and satellites. |
| 3 | Satellite footprint fill / Iridium coverage ring | `satellites.js`, `iridium-footprint-adapter.js` | `GROUND_FOOTPRINT_RENDER_ORDER` | depth-tested; Iridium adapter fill / ring use the same renderOrder | Footprint above land / texture / terrain and country borders, below compute centers and satellites. |
| 3-5 | BGP markers and overlays | `bgp.js` | Each marker's own renderOrder | BGP picking path | Preserves existing BGP visual level. |
| 4.5 | Compute centers | `compute-centers.js` | `COMPUTE_CENTER_RENDER_ORDER` | Compute center picking path | Surface facilities, below satellites. |
| 5 | Satellite background dot | `satellites.js` | Fixed renderOrder | Screen-space satellite picking | Below satellite dots. |
@@ -42,7 +42,7 @@ Note: the layer control panel order and the registration / startup load order ar
| HD texture on | Restores HD texture and the remembered terrain / day/night states. |
| Terrain on | Displayed above HD texture, but below country border hover, footprints, satellites, and other emphasis layers. |
| Cloud layer | Only controls cloud mesh visibility. |
| Country borders | Controls border line and hover line visibility; land/ocean base fill exists independently as the Earth base map. |
| Border Lines off | Hides only interactive border lines and hover, clearing hover state; the land/ocean base fill remains as the Earth base map. |
## Interaction Rules

View File

@@ -4,8 +4,8 @@ This document records the current product boundary, data rationale, and implemen
Related context:
- [earth-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/en/earth-frontend-context.md)
- [backend-collectors.md](/home/ray/dev/linkong/planet/docs/technical/en/backend-collectors.md)
- [Earth Frontend Context](/home/ray/dev/linkong/planet/docs/technical/en/earth-frontend-context.md)
- [Backend Collectors](/home/ray/dev/linkong/planet/docs/technical/en/backend-collectors.md)
- [backend/app/services/collectors/celestrak.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/celestrak.py)
- [frontend/public/earth/js/satellites.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/satellites.js)

View File

@@ -4,8 +4,8 @@ This document describes the current real structure of the console frontend. The
Related references:
- [rules.md](/home/ray/dev/linkong/planet/rules.md)
- [frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/en/frontend-layout-guidelines.md)
- [Project Rules](/home/ray/dev/linkong/planet/rules.md)
- [Frontend Layout Guidelines](/home/ray/dev/linkong/planet/docs/technical/en/frontend-layout-guidelines.md)
## Current Goal
@@ -263,7 +263,7 @@ These principles have been repeatedly validated in the project:
For detailed experience, see:
- [frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/en/frontend-layout-guidelines.md)
- [Frontend Layout Guidelines](/home/ray/dev/linkong/planet/docs/technical/en/frontend-layout-guidelines.md)
## Recommended Change Approach
@@ -290,4 +290,4 @@ Therefore:
For Earth-related structure, see:
- [earth-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/en/earth-frontend-context.md)
- [Earth Frontend Context](/home/ray/dev/linkong/planet/docs/technical/en/earth-frontend-context.md)

View File

@@ -7,7 +7,7 @@ This manual is for daily use, demos, development integration, and local operatio
- Console: admin backend (login required)
- Docs: public developer documentation and manual
For the shortest path to getting started, see [quickstart.md](/home/ray/dev/linkong/planet/docs/technical/en/quickstart.md).
For the shortest path to getting started, see [Quickstart](/home/ray/dev/linkong/planet/docs/technical/en/quickstart.md).
## Entry Overview
@@ -55,6 +55,30 @@ Parameters:
| `--allow-lan` | Enable LAN access |
| `--verbose` | Show more command output during execution |
### AI Provider Environment and Builds
AI Provider runtime configuration can live in `aiprovider/.env` or in matching variables in `~/.zshrc`. `planet.sh` reads simple `export AI_...=...` / `AI_...=...` lines and passes them to the container at startup.
Changing model, API key, or base URL does not rebuild the image. Restart only AI Provider to pick up runtime configuration changes:
```bash
./planet.sh restart -a
```
For complex shell expansion in `~/.zshrc`, opt in explicitly:
```bash
PLANET_LOAD_ZSHRC_ENV=source ./planet.sh start -a
```
To ignore `~/.zshrc` during troubleshooting:
```bash
PLANET_LOAD_ZSHRC_ENV=0 ./planet.sh start -a
```
The AI Provider Docker build context is intentionally limited to the files required by the service, and `uv sync` uses a BuildKit cache mount so dependency downloads are reused after the first build.
### Stop
```bash
@@ -179,7 +203,7 @@ Earth is used to observe in a single globe view:
- Submarine cables and landing points
- Compute centers
- AIS vessels
- Country borders, grid lines, HD texture, cloud layer, terrain
- Border lines, grid lines, HD texture, cloud layer, terrain
- Live news streams and situational news
- Search and focused object details
@@ -190,7 +214,7 @@ The right-side layer panel toggles visualization layers on or off.
Common layers include:
- Grid lines
- Country borders
- Border lines
- HD texture
- Atmospheric cloud layer
- Submarine cables
@@ -215,7 +239,7 @@ Current legend modes include:
- Cables
- Satellites
- Country borders
- Border lines
- Compute centers
- BGP
- AIS vessels
@@ -561,9 +585,9 @@ When something goes wrong, follow this sequence:
## Related Docs
- [quickstart.md](/home/ray/dev/linkong/planet/docs/technical/en/quickstart.md)
- [frontend-admin-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/en/frontend-admin-frontend-context.md)
- [earth-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/en/earth-frontend-context.md)
- [earth-layer-style-reference.md](/home/ray/dev/linkong/planet/docs/technical/en/earth-layer-style-reference.md)
- [backend-system-service-control.md](/home/ray/dev/linkong/planet/docs/technical/en/backend-system-service-control.md)
- [backend-collectors.md](/home/ray/dev/linkong/planet/docs/technical/en/backend-collectors.md)
- [Quickstart](/home/ray/dev/linkong/planet/docs/technical/en/quickstart.md)
- [Admin Frontend Context](/home/ray/dev/linkong/planet/docs/technical/en/frontend-admin-frontend-context.md)
- [Earth Frontend Context](/home/ray/dev/linkong/planet/docs/technical/en/earth-frontend-context.md)
- [Earth Layer Style Reference](/home/ray/dev/linkong/planet/docs/technical/en/earth-layer-style-reference.md)
- [System Service Control](/home/ray/dev/linkong/planet/docs/technical/en/backend-system-service-control.md)
- [Backend Collectors](/home/ray/dev/linkong/planet/docs/technical/en/backend-collectors.md)

View File

@@ -24,6 +24,12 @@ This script checks and syncs common dependencies, and generates if missing:
- `aiprovider/.env`
- `frontend/.env.local`
Personal AI Provider configuration can also live in `~/.zshrc`. `planet.sh` reads simple `export AI_...=...` / `AI_...=...` lines and passes them to the AI Provider container. After changing model, key, or base URL, restart only AI Provider:
```bash
./planet.sh restart -a
```
## 1. Start Services
From the repository root:
@@ -188,7 +194,7 @@ This shuts down the frontend, backend, AI Provider, PostgreSQL, and Redis.
## Next Steps
- Full usage guide: [manual.md](/home/ray/dev/linkong/planet/docs/technical/en/manual.md)
- Console structure: [frontend-admin-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/en/frontend-admin-frontend-context.md)
- Earth structure: [earth-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/en/earth-frontend-context.md)
- Backend collectors: [backend-collectors.md](/home/ray/dev/linkong/planet/docs/technical/en/backend-collectors.md)
- Full usage guide: [Planet Manual](/home/ray/dev/linkong/planet/docs/technical/en/manual.md)
- Console structure: [Admin Frontend Context](/home/ray/dev/linkong/planet/docs/technical/en/frontend-admin-frontend-context.md)
- Earth structure: [Earth Frontend Context](/home/ray/dev/linkong/planet/docs/technical/en/earth-frontend-context.md)
- Backend collectors: [Backend Collectors](/home/ray/dev/linkong/planet/docs/technical/en/backend-collectors.md)

View File

@@ -21,9 +21,11 @@
## 使用入口
- [quickstart.md](/home/ray/dev/linkong/planet/docs/technical/zh/quickstart.md):从零启动 Planet 的最短路径
- [manual.md](/home/ray/dev/linkong/planet/docs/technical/zh/manual.md):控制台、`planet.sh`、Earth 和 Docs 的完整使用手册
- [datasource-collector-settings-connectivity.md](/home/ray/dev/linkong/planet/docs/technical/zh/datasource-collector-settings-connectivity.md)数据源目录、采集器设置、连接验证、BarentsWatch 凭证链路
- [快速开始](/home/ray/dev/linkong/planet/docs/technical/zh/quickstart.md):从零启动 Planet 的最短路径
- [Planet 使用手册](/home/ray/dev/linkong/planet/docs/technical/zh/manual.md):控制台、`planet.sh`、Earth 和 Docs 的完整使用手册
- [数据源、采集器设置与连接验证](/home/ray/dev/linkong/planet/docs/technical/zh/datasource-collector-settings-connectivity.md)数据源目录、采集器设置、连接验证、BarentsWatch 凭证链路
- [Earth 可交互图标接入](/home/ray/dev/linkong/planet/docs/technical/zh/earth-interactable-usage.md)Earth 地表可交互图标 `Interactable` 的接口、生命周期和接入示例
- [Earth 工具栏与浮层协同](/home/ray/dev/linkong/planet/docs/technical/zh/earth-toolbar-overlay-coordination.md):工具栏按钮与搜索 / 设置 / 新闻 / 图层浮层之间的关闭矩阵和接入规则
不适合放入这里的内容:
@@ -33,4 +35,4 @@
这些应放入:
- [docs/plans/README.md](/home/ray/dev/linkong/planet/docs/plans/README.md)
- [计划文档索引](/home/ray/dev/linkong/planet/docs/plans/README.md)

View File

@@ -86,6 +86,7 @@ async def run(self, db):
| TeleGeography | submarine_cable | 海底光缆信息 | 7天 |
| Space-Track TLE | satellite_tle | 卫星轨道 TLE 数据 | 依采集器配置 |
| BarentsWatch AIS | vessel | 船只位置、航速、航向、MMSI 等 AIS 数据 | 依采集器配置 |
| AISStream Vessels | vessel_ais | AIS WebSocket 实时流,写入原始观测层并由聚合接口展示 | 依采集器配置 |
## 四、数据格式 (统一存储到 CollectedData 表)
@@ -224,7 +225,7 @@ if datasource.last_status == "success":
相关实现见:
- [datasource_connectivity.py](/home/ray/dev/linkong/planet/backend/app/services/datasource_connectivity.py)
- [datasource-collector-settings-connectivity.md](/home/ray/dev/linkong/planet/docs/technical/zh/datasource-collector-settings-connectivity.md)
- [数据源、采集器设置与连接验证](/home/ray/dev/linkong/planet/docs/technical/zh/datasource-collector-settings-connectivity.md)
## 八、相关代码文件
@@ -238,7 +239,8 @@ backend/app/services/collectors/
├── huggingface.py # HuggingFace采集器
├── peeringdb.py # PeeringDB采集器
├── telegeraphy.py # TeleGeography海底光缆采集器
── vessel_ais.py # BarentsWatch AIS 船只采集器
── vessel_ais.py # BarentsWatch AIS 船只采集器
└── aisstream.py # AISStream WebSocket 船只采集器
backend/app/models/
└── collected_data.py # 统一数据模型
@@ -251,6 +253,7 @@ backend/app/models/
| 采集器 | credential provider | 凭证来源 |
| --- | --- | --- |
| `barentswatch_vessels` | `barentswatch` | 控制台采集器设置、环境变量、`~/.zshrc` |
| `aisstream_vessels` | `aisstream` | 控制台采集器设置、环境变量 |
| `spacetrack_tle` | `spacetrack` | 环境变量、`~/.zshrc` |
### BarentsWatch AIS
@@ -314,7 +317,7 @@ POST /api/v1/settings/credential-guides/{provider}/reset
更多细节见:
- [datasource-collector-settings-connectivity.md](/home/ray/dev/linkong/planet/docs/technical/zh/datasource-collector-settings-connectivity.md)
- [数据源、采集器设置与连接验证](/home/ray/dev/linkong/planet/docs/technical/zh/datasource-collector-settings-connectivity.md)
## 十一、数据使用场景

View File

@@ -262,6 +262,30 @@ AIS 请求规则:
`VesselAISCollector` 不再自己读取环境变量,而是统一走 `resolve_barentswatch_config()``fetch_barentswatch_access_token()`,避免设置页、连接验证和采集器三套凭证逻辑分叉。
## AISStream 采集器链路
文件:
- [aisstream.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/aisstream.py)
- [vessel_ais_aggregation.py](/home/ray/dev/linkong/planet/backend/app/services/vessel_ais_aggregation.py)
AISStream 使用 WebSocket 实时流,采集器只写入 `ais_raw_observations` 原始观测层,不直接覆盖最终船只展示表。聚合接口负责多源去重、字段选择和冲突记录。
配置项:
- `api_key`:保存在 `DataSourceConfig.auth_config`,也可用环境变量 `AISSTREAM_API_KEY`
- `endpoint`:默认 `wss://stream.aisstream.io/v0/stream`
- `message_types`:默认 `PositionReport``ShipStaticData`
- `bounding_boxes`AISStream 格式为 `[[[lat_min, lon_min], [lat_max, lon_max]]]`,设置页提供全球、挪威 / 北海、欧洲近海、东亚、北美东西海岸 preset。
- `max_messages``receive_timeout_seconds`:控制单次批次式 WebSocket 采集窗口。
标准化规则:
- `PositionReport` 主要提供位置、速度、航向和状态。
- 船名可以从 `MetaData.ShipName` 补入,即使消息体本身没有 `name`
- 船型通常来自低频 `ShipStaticData.Type`;后端会把 AIS 数字类型码映射为 Cargo / Tanker / Passenger / Fishing / Military。
- 如果某艘船尚未收到静态消息,聚合结果的船型仍可能是 `Other`,后续由 v5 船舶资料 enrichment 补齐。
## 凭证教程
文件:
@@ -279,6 +303,7 @@ POST /api/v1/settings/credential-guides/{provider}/reset
当前支持:
- `barentswatch`
- `aisstream`
默认教程包含 BarentsWatch 官方 tutorial 地址:

View File

@@ -187,7 +187,7 @@ Earth info-card 策略:
- 这是预期行为,因为 incident 是聚合和去噪后的结果
- 但 incident-first 渲染会让 Earth 显得过于安静,除非有另一层始终可用的 activity layer
推荐 `activity layer` 的实现细节在 [bgp-region-aggregation-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-bgp-region-aggregation-plan.md) 中展开。
推荐 `activity layer` 的实现细节在 [BGP 区域聚合计划](/home/ray/dev/linkong/planet/docs/plans/earth-bgp-region-aggregation-plan.md) 中展开。
因此最近的里程碑是:

View File

@@ -4,8 +4,8 @@
相关规则建议一起参考:
- [rules.md](/home/ray/dev/linkong/planet/rules.md)
- [frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/zh/frontend-layout-guidelines.md)
- [项目规则](/home/ray/dev/linkong/planet/rules.md)
- [前端布局指南](/home/ray/dev/linkong/planet/docs/technical/zh/frontend-layout-guidelines.md)
## 当前目标
@@ -142,7 +142,7 @@ React 路由入口:
新闻巡航摘要计划见:
- [earth-news-cruise-summary-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-news-cruise-summary-plan.md)
- [Earth 新闻巡航摘要计划](/home/ray/dev/linkong/planet/docs/plans/earth-news-cruise-summary-plan.md)
## 当前样式分层
@@ -249,15 +249,44 @@ Earth 图层按钮现在不应再只有“开/关”两态,而应支持:
AIS 船只图层入口:
- [vessels.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/vessels.js)
- [interactable.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/interactable.js)
船只图层当前负责:
- 请求 `/api/v1/visualization/geo/vessels`
- BarentsWatch AIS GeoJSON 转为 Three.js sprite
- 按船型映射颜色
-聚合后的 AIS GeoJSON 转为地球局部坐标 marker 数据;请求默认不传 `limit`,后端和前端都不再默认裁剪到 5000 艘
- 通过 `createInteractableLayer()` 注册 Interactable 图标层
- 用按航向分桶的 `THREE.Points` 批量渲染普通船只 marker
- 按船型映射颜色;`vessels.js` 会用 `vessel_type_name` 和 AIS `vessel_type` 数字共同归一化船型
- 根据航行/停泊状态绘制三角形或圆点纹理
- 用单点 `THREE.Points` overlay 承载 hover / locked glow
- 支持 hover、lock、轨迹加载和视觉聚焦
船只图层不再是“每艘船一个 `THREE.Sprite`”。原始 Sprite 方案在拖动地球时会把透明对象排序、draw call 和对象级 raycast 成本全部放到主交互路径上;即使 BarentsWatch 免费 AIS 当前只覆盖挪威周边,也会让地球拖动明显不跟手。
当前设计把普通船只拆成少量批次:
- moving / anchored 分开。
- moving 船只按 `VESSEL_COURSE_BINS` 做航向分桶。
- 每个批次是一组 `THREE.PointsMaterial`,位置和颜色写入 `BufferGeometry` attribute。
- 普通态不带 glowhover / locked 时才在相同点位叠加带 glow 的单点 overlay。
方向标准以 AIS `course / cog` 为准:从正北开始顺时针。普通态和交互态都通过同一套 canvas 旋转规则生成纹理,避免 hover 后箭头方向和原 marker 不一致。
船型展示也必须复用同一套归一化结果。`buildVesselMarkerData()` 会把后端的 `vessel_type_name` 和 AIS 数字类型码归一化为 `type`,用于 marker 颜色;同时生成 `vessel_type_display`供详情卡、hover 简述和搜索结果显示。不要让详情卡直接只读原始 `vessel_type_name`,否则会出现 marker 已按 Cargo/Tanker 等颜色显示、卡片仍写 `Other` 的不一致。
AISStream 的 `PositionReport` 常带实时位置和 `MetaData.ShipName`,但船型通常来自低频 `ShipStaticData.Type`。后端会把 `MetaData.ShipName` 补进船名,并将类型码映射为 Cargo / Tanker / Passenger / Fishing / Military仍缺失的船型需要等待静态 AIS 消息或后续船舶资料 enrichment不能在前端凭颜色之外的信息臆造细分类。
船只 hover / click 也不再对渲染对象做 `raycaster.intersectObjects()``main.js` 只负责传入当前 Earth、camera、pointer 和命中半径,实际命中计算由 `interactable.js` 的图标层接口完成:
1. 拖动地球或惯性旋转时跳过 hover picking。
2. 对 hover picking 做轻量节流。
3. 只保留正面船只作为候选。
4. 将候选船只投影到屏幕坐标。
5.`VESSEL_POINTER_RADIUS_PX` 做像素距离命中,并取最近船只。
这样 picking 位置和用户看到的屏幕 marker 对齐,也避免 `Points` 自带 raycaster 在固定屏幕尺寸图标上的命中半径错位。
图例系统已经注册 `vessels` 模式:
- [legend.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/legend.js)
@@ -270,6 +299,24 @@ AIS 船只图层入口:
图例项颜色来自 `VESSEL_CONFIG.colors`,不要在 `legend.css` 里重新定义业务颜色。新增船型时,应优先改 `vessels.js``constants.js` 的船型映射,再同步图例项。
`interactable.js` 是后续地表图标类图层的共用入口。它当前已经承载船只、BGP 事件、BGP 观测站和算力中心图层的批量 `Points`、texture cache、hover / locked overlay、默认 glow、状态增量更新和屏幕空间 pickingBGP 事件的向外扩散圈、BGP 观测站的 halo / 覆盖扇形仍由 `bgp.js` 保留业务动画,但图标本体和 pointer 命中已经接入通用层。新增小型、中心对齐、可以参与深度测试的图标类元素时,应优先复用这个接口,而不是再次复制船只渲染逻辑。
登陆点是当前明确保留的例外:它曾接入 `Interactable`,但 pin 类 SVG 在地球边缘会被 `THREE.Points` 的深度测试裁切成碎片;关闭 depthTest 又会破坏背面遮挡语义。因此登陆点退回 `cables.js` 内的专用 `THREE.Sprite` 路径,并改为 canvas 生成的黄色扁平球纹理。它的 `altitudeOffset``renderOrder` 与海缆线一致避免漂在海缆之上Sprite 本体关闭 `depthTest` 保持球完整,背面可见性由 `isFacingCamera()` 的球体遮挡判断控制。
图标资源可以继续用 canvas draw也可以放到 `frontend/public/earth/assets/icons/` 后由 `Interactable` 预加载。asset 路径不会在每帧读取;图层加载阶段通过 `preloadAssets()` 只加载一次 SVG / 图片,之后按 `icon source + state + bucket + color` 生成 `CanvasTexture` 并复用。当前算力中心已经从 `assets/icons/compute-supercomputer.svg``assets/icons/compute-gpu-cluster.svg` 和备用 `assets/icons/compute-hdd-network.svg` 读取图标,再在 canvas 上叠加估算位置的 `?` badge。
asset 图标大小由 `Interactable``icon.fitSize` 控制。SVG / 图片文件应尽量保持原始 viewBox 和路径,不要为了在地球上显示成 60x60 而手写 `transform``drawAssetIcon()` 会把资源等比 contain 到指定尺寸并居中绘制到 atlas canvas。
`Interactable` 默认使用固定屏幕像素尺寸适合船只、BGP 事件、BGP 观测站、算力中心这类需要稳定识别的图标。如果某类图标需要跟随相机距离缩放,可以把 `sizeMode` 设为非 `"fixed"`,并用 `sizeScale.min / max / referenceFov` 控制缩放范围;单个 marker 的业务尺寸差异可以通过 `getPointSizeMultiplier()` 表达,例如 BGP 事件按严重级别调整点大小BGP 观测站按活跃度调整点大小。
`Interactable` 不再把图标本体额外抬离业务高度。`altitudeOffset` 就是 marker、hover glow、locked glow 和 picking 共同使用的地表高度;这样船只图标会继续贴着船只轨迹线,不会因为单独抬高显示位置而显得漂浮。后续如果要解决边缘 glow 裁切,应优先考虑 glow 纹理、overlay 尺寸或图层专属特效,而不是把通用图标层整体抬高。
跨 Interactable 的同坐标避让也在公共层处理。每个 marker 会保留 `icon_base_position` 作为业务原始位置;当多个 Interactable marker 归入同一个经纬度 key 时,公共层会把它们沿地表切平面排成小圈,并刷新已创建的 `THREE.Points` geometry。这样视觉位置和屏幕空间 picking 位置一致,不需要业务层再单独判断“算力中心和 BGP 事件重叠”这类场景。
接口细节、生命周期和接入示例见:
- [Earth 可交互图标接入](/home/ray/dev/linkong/planet/docs/technical/zh/earth-interactable-usage.md)
### 视角控制反馈
[controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js) 统一维护 Earth 缩放状态。滚轮缩放、缩放按钮和触屏双指捏合最终都会更新 `zoomLevel`,并通过 `showZoomStatusCapsule()` 显示当前缩放比例:

View File

@@ -0,0 +1,270 @@
# Earth Interactable 使用说明
`Interactable` 是 Earth 地表“图标类可交互元素”的通用渲染入口。它把船只图层验证过的模式抽成公共能力:普通态用批量 `THREE.Points`hover / locked 用少量 overlay拾取走屏幕空间命中图标资源统一转进 canvas texture并在公共层处理 glow、状态、尺寸、贴地渲染和同坐标避让。
当前已接入:
| 图层 | 业务文件 | 图标来源 | 补充动画 |
| --- | --- | --- | --- |
| AIS 船只 | `frontend/public/earth/js/vessels.js` | canvas draw航行三角形 / 停泊圆点 | 船只轨迹仍由业务层维护 |
| 算力中心 | `frontend/public/earth/js/compute-centers.js` | `assets/icons/compute-*.svg` | 估算位置 `?` badge 通过 `icon.afterDraw()` 叠加 |
| BGP 事件 | `frontend/public/earth/js/bgp.js` | canvas draw按事件类型绘制符号 | 向外扩散圈仍由 BGP 业务层维护 |
| BGP 观测站 | `frontend/public/earth/js/bgp.js` | `assets/icons/bgp-broadcast-pin.svg` | halo、活跃度 core、覆盖扇形和雷达扫掠仍由 BGP 业务层维护 |
登陆点曾尝试接入 Interactable但 pin 类 SVG 在地球边缘会被 `THREE.Points` 深度测试裁切成碎片;当前退回 `THREE.Sprite` 专用路径,并改为由 canvas 生成黄色扁平球纹理。旧 SVG 资产保留在 `assets/icons/` 目录中,但登陆点运行时不再依赖 SVG。
## 为什么需要 Interactable
之前每个地表图标图层都容易各写一套:
- icon texture 生成
- hover / locked 状态
- glow 样式
- picking 命中半径
- zoom 下的尺寸策略
- 同经纬度对象重叠避让
这些逻辑如果分散在业务文件里,视觉会漂移,后续调参也会变成逐图层修补。`Interactable` 的边界是:公共层负责“图标怎么在地球上稳定显示和被选中”,业务层负责“数据从哪里来、图标表达什么语义、详情卡展示什么、是否有额外动画”。
## 入口
```javascript
import { createInteractableLayer } from "./interactable.js";
```
核心调用形态:
```javascript
const layer = createInteractableLayer({
id: "example",
objectType: "example_object",
renderOrder: 4.4,
altitudeOffset: 0.2,
pointSize: 34,
icon: {
draw(context, options) {
// draw canvas icon
},
},
getPosition: (item) => ({
latitude: item.latitude,
longitude: item.longitude,
}),
getKind: (item) => item.kind || "default",
});
```
业务模块通常只暴露一层薄封装:
```javascript
export function getExampleMarkers() {
return layer.getMarkers();
}
export function getExamplePointerIntersections(options) {
return layer.getPointerIntersections(options);
}
export function setExampleMarkerState(marker, state = "normal") {
layer.setMarkerState(marker, state);
}
export function updateExampleVisualState(lockedObjectType, lockedObject, camera) {
layer.updateVisualState(lockedObjectType, lockedObject, camera);
}
```
## 配置参数
| 参数 | 默认值 | 说明 |
| --- | --- | --- |
| `id` | 必填 | 图层唯一标识,用于 group name、避让注册和 debug。 |
| `objectType` | `id` | marker 写入 `userData.type` 的业务类型,主交互层用它判断 locked 对象。 |
| `renderOrder` | `4` | 普通 points 和 hover / locked overlay 的基础渲染顺序。 |
| `altitudeOffset` | `0.2` | 业务高度,按 `CONFIG.earthRadius + altitudeOffset` 计算原始地表位置。 |
| `pointSize` | `32` | 基准屏幕像素尺寸。普通 points 和 overlay 都以它为基础。 |
| `sizeMode` | `"fixed"` | 默认固定屏幕尺寸;非 `"fixed"` 时会按相机距离做比例缩放。 |
| `sizeScale` | `{ referenceFov: 75, min: 0.12, max: 3 }` | `sizeMode !== "fixed"` 时的缩放范围。 |
| `atlasCellSize` | `128` | icon canvas texture 尺寸。 |
| `colors` | `{}` | 支持 `normal`、按 kind 的平铺 key以及 `byKind`。 |
| `opacity` | `{ normal: 0.88, dimmed: 0.26, hover: 0.98, locked: 1 }` | 各状态透明度。 |
| `stateScale` | `{ hover: 1, locked: 1, dimmed: 1 }` | 各状态尺寸倍率。 |
| `pulse` | `{}` | locked 态可选呼吸缩放,支持 `enabled``speed``amplitude`。 |
| `avoidance` | `{ enabled: true, precision: 4, radius: 1.1, step: 0.35 }` | 跨 Interactable 的同坐标避让配置。 |
| `icon` | 必填 | 图标来源,支持 canvas draw、SVG / 图片 asset、状态 asset、锚点和后处理。 |
| `getPosition(item)` | 必填 | 返回 `{ latitude, longitude }``THREE.Vector3`。 |
| `getKind(item)` | `item.type || "default"` | 返回业务类型,用于颜色和 texture 分桶。 |
| `getRotationBin(marker)` | `0` | 返回旋转分桶,例如船只按航向分 32 桶。 |
| `getBucketKey(marker)` | `String(getRotationBin(marker))` | 返回 texture / geometry 分桶 key。 |
| `getPointSizeMultiplier(marker)` | `1` | 单 marker 尺寸倍率。BGP 事件按严重级别、观测站按活跃度使用它。 |
| `getUserData(item)` | `item` | 写入 marker 的业务字段。 |
## Icon 配置
`icon.anchor` 可选,默认 `{ x: 0.5, y: 0.5 }`,表示纹理中心对齐 marker 坐标。它只适合小范围的视觉锚点偏移;如果图标主体很大、且需要在地球边缘完整显示,例如登陆点曾使用过的 pin 类图标,不应强行走 `THREE.Points + depthTest`,否则图标主体会被地球深度裁切。
### Canvas 图标
canvas 图标适合船只、BGP 事件这类需要按状态或旋转动态绘制的符号:
```javascript
const vesselIconLayer = createInteractableLayer({
id: "vessels",
objectType: "vessel",
pointSize: 34,
icon: {
draw(context, { marker, rotationBin = 0, glow = false, color = "#ffffff" }) {
if (!marker.userData.anchored) {
context.rotate((rotationBin / 32) * Math.PI * 2);
}
context.fillStyle = color;
context.shadowColor = color;
context.shadowBlur = glow ? 14 : 0;
context.beginPath();
context.moveTo(0, -37);
context.lineTo(28, 32);
context.lineTo(0, 17);
context.lineTo(-28, 32);
context.closePath();
context.fill();
},
},
getRotationBin: getCourseBin,
getBucketKey: (marker) => `${marker.userData.anchored ? "anchored" : "moving"}:${getCourseBin(marker)}`,
});
```
`icon.coordinates !== "canvas"` 时,`Interactable` 会先把 context 平移到 atlas 中心;船只这类自己使用中心坐标绘制的图标不需要声明 `coordinates`
### SVG / 图片 Asset 图标
asset 图标适合算力中心、BGP 观测站这类已有 SVG 的设施图标:
```javascript
const computeCenterIconLayer = createInteractableLayer({
id: "computeCenters",
objectType: "compute_center",
pointSize: 36,
atlasCellSize: 128,
icon: {
coordinates: "canvas",
colorable: false,
fitSize: 60,
glowBlur: 16,
getSource({ marker, item }) {
const siteType = marker?.userData?.site_type || item?.site_type || "gpu_cluster";
return COMPUTE_CENTER_ICON_SOURCES[siteType];
},
afterDraw(context, { marker, item }) {
if (marker?.userData?.is_estimated ?? item?.is_estimated) {
drawComputeCenterEstimatedBadge(context, true);
}
},
},
});
```
使用 asset 时有几个约定:
- SVG / 图片文件放在 `frontend/public/earth/assets/icons/`,以 `/earth/assets/icons/name.svg` 引用。
- 原始 SVG 应尽量保留标准 `viewBox` 和路径,不要为了显示大小写死 transform。
- 显示尺寸由 `icon.fitSize` 控制;它可以是数字、`{ width, height }`,也可以是函数。
- `icon.colorable !== false` 且提供状态颜色时,公共层会先把 asset 画到临时 canvas再用 `source-in` tint 成目标颜色。
- 多色图片或不希望被 tint 的 SVG 应设置 `colorable: false`
## 生命周期
常规加载流程:
```javascript
export async function loadExampleLayer(_scene, earth) {
clearExampleData(earth);
const markerData = await fetchExampleData();
await layer.preloadAssets(markerData);
layer.setData(markerData);
layer.attach(earth);
layer.setVisible(showExampleLayer);
return { totalCount: layer.getCount() };
}
```
各方法职责:
| 方法 | 说明 |
| --- | --- |
| `preloadAssets(items)` | 收集 normal / hover / locked 可能用到的 asset source并用浏览器 `Image` 预加载。canvas draw 图标可跳过。 |
| `setData(items)` | 清理旧 points生成 marker注册避让按 bucket 重建 `THREE.Points`。 |
| `attach(parent)` | 将图层 group 挂到 Earth root。 |
| `setVisible(next)` | 控制 group、points 和 overlay 可见性。 |
| `setMarkerState(marker, state)` | 设置 `normal` / `hover` 等状态并触发视觉状态失效。 |
| `updateVisualState(focusType, focusObject, camera)` | 更新普通态 opacity / size并刷新 hover / locked overlay。 |
| `getPointerIntersections(options)` | 屏幕空间拾取,返回按像素距离排序的命中结果。 |
| `clearData(parent)` | 注销避让、释放 geometry / material、清空 marker 并从 parent 移除 group。 |
## Picking 接入
`Interactable` 不依赖 Three.js 对 `Points` 的默认 raycast。主交互层只要把 Earth、camera、pointer 和命中半径传入:
```javascript
const intersects = getVesselPointerIntersections({
earth,
camera,
pointer,
radiusPx: 22,
width: window.innerWidth,
height: window.innerHeight,
});
```
公共层会做这些事:
1. 把相机位置转到 Earth local 坐标。
2. 跳过背面 marker。
3. 把 marker world position 投影到屏幕坐标。
4.`radiusPx` 做像素距离命中。
5. 返回最近的候选对象。
拖动地球、惯性旋转、hover 节流这些策略仍属于 `main.js`,因为它们和全局输入状态有关。
## 同坐标避让
避让默认开启,作用范围是所有通过 `createInteractableLayer()` 创建的图层。公共层会按经纬度或 `THREE.Vector3` 生成 `icon_avoidance_key`,同 key 的 marker 会沿地表切平面排成小圈。
关键点:
- `icon_base_position` 保留业务原始位置。
- 避让只改渲染位置和 picking 位置,不改业务经纬度。
- 单个 marker 回到原始位置时会直接使用 `altitudeOffset` 计算出的业务贴地位置。
- 多个 marker 同坐标时,第一圈用 `avoidance.radius`,后续每圈加 `avoidance.step`
如果某个业务图层需要严格压在原始点位,可以显式关闭:
```javascript
createInteractableLayer({
id: "strict-layer",
avoidance: { enabled: false },
});
```
## 业务动画边界
`Interactable` 当前只负责图标本体和通用 hover / locked overlay。复杂动画仍放在业务模块里但要跟随 Interactable marker 的位置:
- BGP 事件扩散圈由 `bgp.js` 创建独立 ring sprite并在每帧 `position.copy(marker.position)`
- BGP 观测站 halo、status core、coverage halo 和覆盖扇形由 `bgp.js` 管理,图标本体由 Interactable 管理。
- 船只轨迹线仍由 `vessels.js` 管理,因为它依赖点击后额外加载的轨迹数据。
这个边界能避免通用接口过早承载所有动画类型。后续如果多个图层复用同一类动画,再把它收进 Interactable 的 `animations` 扩展。
## 新图层接入清单
1. 在业务文件中准备 marker data并保留必要的业务字段。
2. 选择 icon 类型canvas draw、SVG / 图片 asset`getSource()` 动态选择。
3. 配置 `pointSize``icon.fitSize``colors``opacity``stateScale`
4. 如果需要业务尺寸差异,提供 `getPointSizeMultiplier()`
5. 如果有旋转,提供 `getRotationBin()` 和稳定的 `getBucketKey()`
6. 加载时先 `preloadAssets()`,再 `setData()``attach()``setVisible()`
7.`main.js` 接入 `getPointerIntersections()`,并复用现有 hover / locked 状态更新流程。
8. 在图层样式索引和渲染顺序文档中记录 altitude、renderOrder、pointSize 和动画层级。

View File

@@ -2,7 +2,7 @@
本文记录当前 Earth 前端各图层的材质、颜色、透明度、线宽、半径偏移和
`renderOrder` 等样式属性。层级关系请配合
[earth-render-layer-order.md](/home/ray/dev/linkong/planet/docs/technical/zh/earth-render-layer-order.md)
[Earth 渲染图层顺序](/home/ray/dev/linkong/planet/docs/technical/zh/earth-render-layer-order.md)
查看。
## 命名约定
@@ -80,6 +80,8 @@
## 海陆基座与国界
海陆基座是 Earth 的底图资产随启动预加载图层面板里的“国界线”只控制普通国界线、hover 线和可交互 hover。
| 正式名称 | 变量名 | 当前值 | 使用位置 / 说明 |
| --- | --- | --- | --- |
| 国界数据路径 | `COUNTRY_BOUNDARY_CONFIG.dataPath` | `"/earth/data/countries-admin0.min.geojson"` | GeoJSON 输入 |
@@ -146,21 +148,18 @@
| 海缆线宽 | `CABLE_CONFIG.line.lineWidth` | `1` | `LineBasicMaterial.linewidth` |
| 海缆透明度 | `CABLE_CONFIG.line.opacity` | `1.0` | 海缆线 opacity |
| 海缆 renderOrder | `CABLE_CONFIG.line.renderOrder` | `1` | 海缆线层级 |
| 登陆点半径偏移 | `CABLE_CONFIG.landingPoint.altitudeOffset` | `0.48` | 对齐算力中心贴地表 marker 高度 |
| 登陆点 icon 贴图尺寸 | `CABLE_CONFIG.landingPoint.textureSize` | `256` | canvas 渲染 EPS 参考图的实心 map-pin中间圆孔透明镂空 |
| 登陆点 icon 宽高比 | `CABLE_CONFIG.landingPoint.iconAspectRatio` | `0.82` | `Sprite.scale.x = height * aspect` |
| 登陆点 icon 锚点 | `CABLE_CONFIG.landingPoint.anchorX / anchorY` | `0.52 / 0.276` | `Sprite.center`,将 pin 下端点对齐登陆点经纬度 |
| 登陆点基础缩放 | `CABLE_CONFIG.landingPoint.baseScale` | `12` | 对齐算力中心等地表 icon 的 sprite 高度 |
| 登陆点半径偏移 | `CABLE_CONFIG.landingPoint.altitudeOffset` | `0.2` | 与海缆线同层贴地,避免凌空 |
| 登陆点 sprite 高度 | local `LANDING_POINT_SPRITE_HEIGHT` | `3` | `THREE.Sprite` 基准高度 |
| 登陆点缩放参考 FOV | local `LANDING_POINT_SIZE_REFERENCE_FOV` | `75` | 与当前 Earth 相机 FOV 一致 |
| 登陆点缩放下限 | local `LANDING_POINT_SIZE_SCALE_MIN` | `0.16` | 地球放到 200% 之后的最小倍率,限制高倍 zoom 下的屏幕占比;`3 * 0.16 = 0.48` |
| 登陆点缩放上限 | local `LANDING_POINT_SIZE_SCALE_MAX` | `3` | 远距离时的最大倍率;当前最小缩放约只能到 `2.50` |
| 登陆点 atlas 尺寸 | local `LANDING_POINT_ATLAS_CELL_SIZE` | `128` | canvas 扁平立体球纹理尺寸 |
| 登陆点颜色 | `CABLE_CONFIG.landingPoint.color` | `0xffaa00` | `SpriteMaterial.color` |
| 登陆点 emissive | `CABLE_CONFIG.landingPoint.emissive` | `0x442200` | 兼容旧球体材质sprite 不使用 |
| 登陆点 emissive 强度 | `CABLE_CONFIG.landingPoint.emissiveIntensity` | `0.5` | 兼容旧球体材质sprite 不使用 |
| 登陆点透明度 | `CABLE_CONFIG.landingPoint.opacity` | `1.0` | `SpriteMaterial.opacity` |
| 登陆点 renderOrder | `CABLE_CONFIG.landingPoint.renderOrder` | `4.5` | 对齐算力中心地表设施层级 |
| 登陆点 renderOrder | `CABLE_CONFIG.landingPoint.renderOrder` | `1` | 与海缆线同层;`depthTest: false` 保持球体完整,背面通过相机到球心的球体遮挡判断隐藏 |
| 登陆点 dim 亮度系数 | `landingPointVisual.dimBrightness` | `0.62` | dim 状态颜色乘数 |
| 相关登陆点高亮 opacity | `landingPointVisual.related.opacityBase / opacityPulse` | `0.8 / 0.2` | 高亮脉冲 |
| 非相关登陆点颜色 | `landingPointVisual.dimmed.colorRGB` | `{ r: 180, g: 116, b: 28 }` | dim 状态颜色,避免黑色基座透出成暗洞 |
| 非相关登陆点 emissive | `landingPointVisual.dimmed.emissive` | `0x3a2200` | dim 状态弱琥珀自发光 |
| 非相关登陆点 emissive 强度 | `landingPointVisual.dimmed.emissiveIntensity` | `0.18` | dim 状态弱发光强度 |
| 非相关登陆点 opacity | `landingPointVisual.dimmed.opacity` | `0.78` | dim 状态透明度,不再用低 alpha 混黑底 |
## 卫星、轨迹和 footprint
@@ -183,19 +182,45 @@
| 卫星覆盖层 renderOrder | `SATELLITE_CONFIG.overlayRenderOrder` | `12` | locked ring / halo / orbit |
| 自发光选中点颜色 | inline default | `"#ffd25a"` | `showSelfGlowStyle()` |
| 自发光选中点透明度 | inline | `0.96` | locked dot material |
| footprint renderOrder | local `GROUND_FOOTPRINT_RENDER_ORDER` | `3` | footprint fill |
| footprint renderOrder | local `GROUND_FOOTPRINT_RENDER_ORDER` | `3` | Starlink footprint fill 和 Iridium coverage ring必须高于地表 land / texture / terrain 层 |
| footprint group renderOrder | inline | `0` | 避免 Group 排序盖过卫星点 |
## AIS 船只
| 正式名称 | 变量名 | 当前值 | 使用位置 / 说明 |
| --- | --- | --- | --- |
| 船只半径偏移 | `VESSEL_CONFIG.altitudeOffset` | `0.2` | 普通 marker 位置,贴近真实地形基础层 |
| 船只轨迹半径偏移 | `VESSEL_CONFIG.track.altitudeOffset` | `0.2` | 选中船只轨迹线,与船只 marker 同一半径;前端会把轨迹末端锚到当前 marker 位置 |
| 船只 renderOrder | local `VESSEL_RENDER_ORDER` | `4.4` | 普通 marker 和交互 overlay |
| 船只轨迹 renderOrder | `VESSEL_RENDER_ORDER - 0.1` | `4.3` | 低于船只 marker |
| 船只点像素尺寸 | local `VESSEL_POINT_SIZE` | `34` | 普通 marker 与 hover / locked overlay 共享尺寸 |
| 船只默认渲染上限 | `VESSEL_CONFIG.maxRenderedMarkers` | `0` | `0` 表示不在前端默认裁剪;正数才会给接口传 `limit` 并裁剪 marker |
| 船只纹理画布尺寸 | local `VESSEL_ATLAS_CELL_SIZE` | `128` | canvas 点纹理 |
| 航向分桶数 | local `VESSEL_COURSE_BINS` | `32` | moving 船只按 COG 分桶,降低 draw call 同时保留方向 |
| 船只 hover 拾取节流 | local `VESSEL_HOVER_PICK_INTERVAL_MS` | `100` | `main.js` hover picking |
| 船只屏幕命中半径 | local `VESSEL_POINTER_RADIUS_PX` | `22` | `main.js` 屏幕空间 picking |
| 普通船只透明度 | `VESSEL_CONFIG.marker.baseOpacity` | `0.88` | 普通 `PointsMaterial.opacity` |
| dimmed 船只透明度 | `VESSEL_CONFIG.marker.dimmedOpacity` | `0.26` | 锁定某艘船后其他批次透明度 |
| hover 船只透明度 | inline | `0.98` | hover overlay |
| locked 船只透明度 | inline | `1` | locked overlay |
| 船型颜色 | `VESSEL_CONFIG.colors.*` | cargo / tanker / passenger / fishing / military / other | `PointsMaterial.vertexColors` 和 overlay texture |
AIS 船只普通态使用批量 `THREE.Points`,不是逐船 `THREE.Sprite`。航行船只保持三角形,停泊或低速船只保持圆点;普通态不带 glowhover / locked 时在同一屏幕尺寸上叠加带 glow 的单点 overlay。AIS 航向按 `course / cog` 从正北顺时针解释,普通态和交互态必须使用同一套 canvas 旋转规则。
船型颜色和详情卡船型文本必须来自同一套归一化结果:`vessels.js` 同时读取后端 `vessel_type_name` 和 AIS 数字 `vessel_type`,先得到颜色用的 `type`,再生成 `vessel_type_display` 给详情卡、hover 和搜索使用。
## 算力中心
| 正式名称 | 变量名 | 当前值 | 使用位置 / 说明 |
| --- | --- | --- | --- |
| 算力中心半径偏移 | `COMPUTE_CENTER_CONFIG.altitudeOffset` | `0.48` | marker 位置 |
| 算力中心基础透明度 | `COMPUTE_CENTER_CONFIG.marker.baseOpacity` | `0.88` | `SpriteMaterial.opacity` |
| 超算 marker 缩放 | `COMPUTE_CENTER_CONFIG.marker.supercomputerScale` | `12` | 超算 marker |
| GPU 集群 marker 缩放 | `COMPUTE_CENTER_CONFIG.marker.gpuClusterScale` | `12` | GPU marker |
| hover 缩放 | `COMPUTE_CENTER_CONFIG.marker.hoverScale` | `1.16` | hover 状态 |
| locked 缩放 | `COMPUTE_CENTER_CONFIG.marker.lockedScale` | `1.22` | locked 状态 |
| 算力中心点像素尺寸 | local `COMPUTE_CENTER_POINT_SIZE` | `36` | `Interactable` 普通 marker 与 hover / locked overlay 共享基准尺寸 |
| 算力中心 asset fit size | local `COMPUTE_CENTER_ICON_FIT_SIZE` | `60` | SVG asset 在 `128x128` atlas canvas 内的最大绘制尺寸,由 `icon.fitSize` 控制 |
| 算力中心基础透明度 | `COMPUTE_CENTER_CONFIG.marker.baseOpacity` | `0.88` | 普通 `PointsMaterial.opacity` |
| 超算 marker 缩放 | `COMPUTE_CENTER_CONFIG.marker.supercomputerScale` | `12` | 旧 Sprite 缩放参数;当前 Interactable 路径不再直接使用 |
| GPU 集群 marker 缩放 | `COMPUTE_CENTER_CONFIG.marker.gpuClusterScale` | `12` | 旧 Sprite 缩放参数;当前 Interactable 路径不再直接使用 |
| hover 缩放 | `COMPUTE_CENTER_CONFIG.marker.hoverScale` | `1.16` | hover overlay 尺寸倍率 |
| locked 缩放 | `COMPUTE_CENTER_CONFIG.marker.lockedScale` | `1.22` | locked overlay 尺寸倍率,并叠加 pulse |
| dimmed 缩放 / 透明度 | `dimmedScale / dimmedOpacity` | `0.82 / 0.34` | dim 状态 |
| 超算颜色 | `COMPUTE_CENTER_CONFIG.colors.supercomputer` | `"#38bdf8"` | marker texture |
| GPU 集群颜色 | `COMPUTE_CENTER_CONFIG.colors.gpu_cluster` | `"#2dd4bf"` | marker texture |
@@ -206,12 +231,16 @@
| 正式名称 | 变量名 | 当前值 | 使用位置 / 说明 |
| --- | --- | --- | --- |
| BGP 事件半径偏移 | `BGP_CONFIG.altitudeOffset` | `2.1` | anomaly marker |
| BGP collector 半径偏移 | `BGP_CONFIG.collectorAltitudeOffset` | `1.6` | collector marker |
| 事件基础缩放 | `BGP_CONFIG.marker.eventBaseScale` | `6.2` | anomaly sprite |
| collector 基础缩放 | `BGP_CONFIG.marker.collectorBaseScale` | `7.4` | collector plane |
| BGP 事件半径偏移 | `BGP_CONFIG.altitudeOffset` | `0.48` | BGP 事件 Interactable marker |
| BGP collector 半径偏移 | `BGP_CONFIG.collectorAltitudeOffset` | `0.2` | BGP 观测站 Interactable marker与船只同层贴地 |
| BGP 事件点像素尺寸 | local `BGP_EVENT_POINT_SIZE` | `34` | 事件 icon 的 Interactable 基准尺寸,按严重级别通过 `getPointSizeMultiplier()` 调整 |
| BGP 事件符号绘制尺寸 | local `BGP_EVENT_SYMBOL_SIZE` | `60` | 事件 canvas 符号在 `128x128` atlas 中的绘制尺寸 |
| BGP collector 点像素尺寸 | local `BGP_COLLECTOR_POINT_SIZE` | `36` | 观测站 Interactable 基准尺寸,按活跃度通过 `getPointSizeMultiplier()` 调整 |
| BGP collector asset fit size | local `BGP_COLLECTOR_ICON_FIT_SIZE` | `60` | `bgp-broadcast-pin.svg` 在 atlas canvas 内的最大绘制尺寸 |
| 事件基础缩放 | `BGP_CONFIG.marker.eventBaseScale` | `6.2` | 事件扩散圈锚点 |
| collector 基础缩放 | `BGP_CONFIG.marker.collectorBaseScale` | `7.4` | 观测站 halo / 覆盖动画锚点 |
| hover / dim 缩放 | `hoverScale / dimmedScale` | `1.16 / 0.92` | 交互状态 |
| 普通事件透明度 | `BGP_CONFIG.opacity.normal` | `0.78` | anomaly sprite |
| 普通事件透明度 | `BGP_CONFIG.opacity.normal` | `0.78` | BGP 事件 Interactable 普通态 |
| hover 透明度 | `BGP_CONFIG.opacity.hover` | `1.0` | hover 状态 |
| dimmed 透明度 | `BGP_CONFIG.opacity.dimmed` | `0.24` | dim 状态 |
| collector 透明度 | `BGP_CONFIG.opacity.collector` | `0.62` | collector 状态 |
@@ -223,8 +252,8 @@
| region 色 | `BGP_CONFIG.regionColor` | `0x2dd4bf` | 区域覆盖 |
| BGP ring 缩放 | `BGP_CONFIG.ring.scaleA / scaleB` | `2.5 / 3.4` | anomaly ring |
| BGP ring 透明度 | `BGP_CONFIG.ring.opacity` | `0.5` | anomaly ring |
| collector marker renderOrder | inline | `3` | `marker.renderOrder` |
| anomaly marker renderOrder | inline | `5` normal, `7` active | `marker.renderOrder` |
| collector marker renderOrder | local `BGP_COLLECTOR_RENDER_ORDER` | `4.4` | 观测站主图标,与船只同层 |
| anomaly marker renderOrder | local `BGP_EVENT_RENDER_ORDER` | `4.5` | BGP 事件主图标,与算力中心同层 |
## 天体与星空

View File

@@ -7,8 +7,8 @@
| 顺序类型 | 当前顺序 | 说明 |
| --- | --- | --- |
| 控制面板顺序 | 海缆 → 轨迹 → 卫星 → 算力中心 → 船只 → BGP → 地形 → 高清材质 → 大气云图 → 国界 → 经纬线 | 由 `displayOrder` 控制,按操作关注度排列。 |
| 注册 / 启动加载顺序 | 经纬线 → 国界 → 高清材质 → 大气云图 → 海缆 → 算力中心 → 船只 → BGP → 卫星 | 由注册顺序和 `startupPriority` 控制,按地表到天空排列;船只和卫星默认关闭,只有可见时参与启动加载;轨迹和地形是依赖/可选显示层,不参与常规启动数据加载。 |
| 控制面板顺序 | 海缆 → 轨迹 → 卫星 → 算力中心 → 船只 → BGP → 地形 → 高清材质 → 大气云图 → 国界线 → 经纬线 | 由 `displayOrder` 控制,按操作关注度排列。 |
| 注册 / 启动加载顺序 | 经纬线 → 国界线 / 海陆基座 → 高清材质 → 大气云图 → 海缆 → 算力中心 → 船只 → BGP → 卫星 | 由注册顺序和 `startupPriority` 控制,按地表到天空排列;启动队列会先读取保存的图层可见状态,明确关闭的普通图层不预加载,高清材质关闭时不下载贴图;国界线图层例外,海陆基座始终预加载,保存状态只控制可交互国界线和 hover;轨迹和地形是依赖/可选显示层,不参与常规启动数据加载。 |
## 地表图层栈
@@ -21,17 +21,17 @@
| 0.86 | 海陆基座填充 | `country-boundaries.js` | `landAltitudeOffset`; 海洋 `#010609`,陆地 `#080f1b` | 禁用 raycast | 即使国界线关闭,基座地图仍保持可用。 |
| 0.96 | 高清 Earth 材质 | `earth.js` | `textureOverlayAltitudeOffset` | 可见时作为地表拾取目标 | 高清材质始终压过海陆基座填充。 |
| 1 | 大气辉光和云图 | `earth.js` | 大气 / 云层球 | 不走普通对象选择路径 | 云图由“大气云图”图层开关控制。 |
| 1 | 海缆 | `cables.js` | `CABLE_CONFIG.line.renderOrder` | 海缆拾取路径 | 保持现有海缆层级。 |
| 1 | 海缆 / 登陆点 | `cables.js` | 海缆线和登陆点都使用 `renderOrder = 1`;半径偏移都为 `0.2`;登陆点是专用 `THREE.Sprite` 黄色扁平球 | 海缆走海缆拾取路径;登陆点 `depthTest: false` 保持球体完整,并用相机到球心的球体遮挡判断避免背面穿透 | 登陆点和海缆同层贴地,避免地表设施层的凌空感。 |
| 1.2 | 真实地形 | `earth.js`, `terrain.js` | `TERRAIN_CONFIG.baseRadiusOffset` 加地形位移 | 禁用 raycast | 地形压过高清材质;高清材质关闭时临时隐藏,重新开启后恢复原状态。 |
| 2.05 | 经纬线 | `earth.js` | `CONFIG.earthRadius + 0.14` | 禁用 raycast | 低透明度显示在高清材质上。 |
| 2.2 | 国界线 | `country-boundaries.js` | `lineAltitudeOffset = 0.115` | `depthTest: true`,禁用 raycast | 略高于高清材质 `0.10`,低于地形基准 `0.16`,减少悬浮感;地形 `depthWrite: false`,所以地形开启时仍可见。 |
| 2.29 | 国界 hover 光晕 | `country-boundaries.js` | hover 半径加 glow 偏移 | `depthTest: false`,禁用 raycast | 用 additive 光晕增强交界边和地形开启时的 hover 可见性。 |
| 2.3 | 国界 hover 实线 | `country-boundaries.js` | `hoverAltitudeOffset = 0.14` | `depthTest: false`,禁用 raycast | 霓虹红橘 hover 线;中国和中国(台湾)共享高亮组。 |
| 3 | 卫星 footprint 填充 | `satellites.js` | `GROUND_FOOTPRINT_RENDER_ORDER` | depth-testedGroup renderOrder 保持 0 | Footprint 在国界线之上,但在算力中心和卫星之下。 |
| 3-5 | BGP 标记和覆盖层 | `bgp.js` | 各 marker 自身 renderOrder | BGP 拾取路径 | 保持现有 BGP 视觉层级。 |
| 3 | 卫星 footprint 填充 / Iridium coverage ring | `satellites.js`, `iridium-footprint-adapter.js` | `GROUND_FOOTPRINT_RENDER_ORDER` | depth-testedIridium adapter 的 fill / ring 也使用同一 renderOrder | Footprint 在 land / texture / terrain 和国界线之上,但在算力中心和卫星之下。 |
| 3-4.5 | BGP 观测站、事件扩散圈和事件 marker | `bgp.js`, `interactable.js` | BGP 观测站和事件 marker 均使用 `Interactable` 批量 `THREE.Points`;事件 marker 使用 `BGP_EVENT_RENDER_ORDER = 4.5`;观测站主图标使用 `BGP_COLLECTOR_RENDER_ORDER = 4.4``BGP_CONFIG.collectorAltitudeOffset = 0.2`;事件 overlay 进入 `bgp-event-overlay-layer`;观测站 halo 和覆盖扇形进入 `bgp-collector-radar-layer` | BGP 事件和观测站都通过 `Interactable` 屏幕空间 picking并参与同坐标避让 | BGP 观测站主图标与船只同层BGP 事件与算力中心同层;向外扩散圈、观测站雷达/覆盖动画继续由 BGP 业务逻辑驱动。 |
| 4.3 | AIS 船只轨迹线 | `vessels.js` | `VESSEL_RENDER_ORDER - 0.1``CONFIG.earthRadius + VESSEL_CONFIG.track.altitudeOffset` | 跟随船只显隐,不单独参与拾取 | 选中船只后显示最近轨迹,低于船只 marker。 |
| 4.4 | AIS 船只 marker | `vessels.js` | `VESSEL_RENDER_ORDER``CONFIG.earthRadius + VESSEL_CONFIG.altitudeOffset` | 船只拾取路径;只取正面 marker | 航行船只用三角 sprite,停泊/低速用圆点;低于算力中心 `4.5`。 |
| 4.5 | 算力中心 | `compute-centers.js` | `COMPUTE_CENTER_RENDER_ORDER` | 算力中心拾取路径 | 地表设施,保持在卫星下方。 |
| 4.4 | AIS 船只 marker | `vessels.js`, `interactable.js` | `VESSEL_RENDER_ORDER`业务高度为 `CONFIG.earthRadius + VESSEL_CONFIG.altitudeOffset`;普通 marker 为分桶 `THREE.Points`hover / locked 为单点 `THREE.Points` overlay | `depthTest: true``main.js` 使用屏幕空间 picking只取正面 marker参与 Interactable 同坐标避让 | 航行船只用三角点纹理,停泊/低速用圆点;普通态无 glow交互态叠加同尺寸 glow低于算力中心 `4.5`。 |
| 4.5 | 算力中心 | `compute-centers.js`, `interactable.js` | 使用 `COMPUTE_CENTER_RENDER_ORDER` 并由 `Interactable` 绘制 | 通过 `Interactable` 屏幕空间 picking参与同坐标避让 | 地表设施,保持在卫星下方。登陆点已下沉到海缆层。 |
| 5 | 卫星背景点 | `satellites.js` | 固定 renderOrder | 屏幕空间卫星拾取 | 位于卫星点下方。 |
| 6 | 卫星点 | `satellites.js` | 固定 renderOrder | 屏幕空间卫星拾取 | 卫星点压过 footprint 和算力中心。 |
| 12+ | 卫星锁定 ring、halo、预测轨道 | `satellites.js` | `SATELLITE_CONFIG.overlayRenderOrder` 及偏移 | 卫星覆盖层路径 | 用于选中 / 锁定卫星强调。 |
@@ -45,7 +45,7 @@
| 高清材质 on | 恢复高清材质,并恢复记住的地形 / 昼夜状态。 |
| 地形 on | 显示在高清材质之上,但低于国界 hover、footprint、卫星等强调层。 |
| 大气云图 | 只控制云图 mesh 显隐。 |
| 国界 | 控制国界线和 hover 线显隐;海陆基座填充独立存在,作为 Earth 基座地图使用。 |
| 国界线 off | 只隐藏可交互国界线和 hover,高亮状态会清除;海陆基座填充作为 Earth 底图保留。 |
## 交互规则
@@ -57,4 +57,4 @@
| 中国 / 台湾 hover | `CHN``TWN` 被归到同一个 hover 高亮组tooltip 仍显示鼠标实际命中的 feature。 |
| 地形 | 只作为视觉层参与,`terrain.raycast` 已禁用。 |
| 卫星 | 使用屏幕空间卫星拾取,避免 footprint 或地表层挡住卫星点击。 |
| 船只 | 使用 sprite marker 拾取,并在 `main.js` 中先过滤正面船只;点击后可加载轨迹线。 |
| 船只 | 使用对象级 sprite raycast。`main.js` 会在拖动 / 惯性期间跳过 hover picking平时将正面船只投影到屏幕坐标用像素半径命中最近船只;点击后可加载轨迹线。 |

View File

@@ -4,8 +4,8 @@
相关上下文:
- [earth-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/zh/earth-frontend-context.md)
- [backend-collectors.md](/home/ray/dev/linkong/planet/docs/technical/zh/backend-collectors.md)
- [Earth 前端结构](/home/ray/dev/linkong/planet/docs/technical/zh/earth-frontend-context.md)
- [数据采集系统](/home/ray/dev/linkong/planet/docs/technical/zh/backend-collectors.md)
- [backend/app/services/collectors/celestrak.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/celestrak.py)
- [frontend/public/earth/js/satellites.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/satellites.js)

View File

@@ -0,0 +1,94 @@
# Earth 工具栏与浮层协同
本文件描述 Earth 大屏右侧工具栏按钮,以及搜索面板、设置弹窗、新闻直播面板、图层面板这几个浮层之间当前的协同规则。改交互、加按钮、调整面板时按这个表对齐,避免出现「点 A 把不该关的 B 也关了」之类的协同冲突。
相关入口:
- [Earth 前端结构](/home/ray/dev/linkong/planet/docs/technical/zh/earth-frontend-context.md)
- [前端布局指南](/home/ray/dev/linkong/planet/docs/technical/zh/frontend-layout-guidelines.md)
## 工具栏按钮目录
工具栏在 [index.html](/home/ray/dev/linkong/planet/frontend/public/earth/index.html) 中以 `.earth-toolbar-btn` 标识,按钮列表:
| ID | 标题 | 类型 | 触发的浮层/动作 |
|----|------|------|------------------|
| `layer-action` | 图层 | 浮层切换 | HUD 面板 `layer-toggles`(桌面)/ 移动端抽屉 `layers` 卡 |
| `search-action` | 搜索 | 浮层切换 | 搜索面板(桌面)/ 移动端抽屉 `search` 卡 |
| `rotate-toggle` | 自动旋转 | 独立开关 | 不打开任何浮层 |
| `toggle-tv` | 新闻直播 | 浮层切换 | 媒体面板 `media-panel`(含 TV/News 两个 tab |
| `reload-data` | 重新加载数据 | 独立动作 | 不打开任何浮层 |
| `zoom-trigger` | 缩放控制 | 浮动菜单 | 缩放 floating menu |
| `settings-trigger` | 设置 | 浮层切换 | 设置弹窗(桌面)/ 移动端抽屉 `settings` 卡 |
| `reset-view` | 重置视角 | 独立动作 | 不打开任何浮层 |
| `layout-toggle` | 最大化布局 | 独立开关 | 不打开任何浮层 |
## 浮层协同的统一入口
[controls.js::closeTransientMobileOverlays](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js) 是「打开 X 时该关谁」的统一协调函数。
调用约定:每个会进入 fullscreen-style 浮层的开启路径调用 `closeTransientMobileOverlays({ except })`,告诉协调函数「除了我这一类,其他互斥浮层一律关掉」。
```js
closeTransientMobileOverlays({ except: "search" }); // 搜索打开
closeTransientMobileOverlays({ except: "settings" }); // 设置打开
closeTransientMobileOverlays({ except: "media" }); // 新闻直播打开
closeTransientMobileOverlays({ except: "layer-toggles" }); // 图层抽屉(移动端)
```
`except` 当前可取的值:`"search"``"settings"``"media"``"layer-toggles"`,或省略表示「全部关闭」。
## 关闭矩阵
下表描述「打开 X」时其它浮层的命运。`✓` = 关闭,`—` = 保留。
| 触发动作 → | 关搜索 | 关设置 | 关图层抽屉(移动端) | 关新闻/直播 |
|-----------|:------:|:------:|:--------------------:|:-----------:|
| 打开搜索 (`except: "search"`) | (自身)| ✓ | ✓ | — |
| 打开设置 (`except: "settings"`) | ✓ | (自身)| ✓ | — |
| 打开新闻/直播 (`except: "media"`) | ✓ | ✓ | ✓ | (自身)|
| 打开图层抽屉 (`except: "layer-toggles"`) | ✓ | ✓ | (自身)| ✓ |
| 全部关闭 (`except: null`) | ✓ | ✓ | ✓ | ✓ |
读法举例:
- 点工具栏「设置」,搜索面板和图层抽屉会被关掉,新闻/直播面板保持原状。
- 点工具栏「图层」(移动端打开 `layers` 抽屉),搜索 / 设置 / 新闻 全关。
- 点工具栏「新闻直播」,搜索 / 设置 / 图层抽屉全关,新闻面板自身切换为打开。
## 设计原则
下面是当前矩阵背后的几条不变量。新增浮层或调整规则时按它们对齐:
1. **`zoom-trigger` 等浮动菜单不属于浮层。** 它们走 `bindFloatingMenu`,由 `closeFloatingMenus()` 单独管理;任何浮层打开都会先调一次 `closeFloatingMenus()`
2. **桌面 `layer-toggles` 是常驻 HUD 面板,不是浮层。** `closeTransientMobileOverlays` 中只有 `activeMobileDrawerId === "layer-toggles"`(移动端抽屉态)才会被关掉。所以桌面打开搜索/设置/新闻不会动图层面板,符合「桌面屏幕大、可共存」的预期。
3. **新闻/直播面板独立于设置。** 用户切到设置改采集器时,常常想边看新闻边改配置,所以打开设置时不关新闻面板。这条是 2026-05 的协同补丁后建立的不变量;改设置打开路径时不要再去主动关 `media-panel`
4. **搜索和新闻面板视为「主信息浮层」,互相独立。** 搜索打开不关新闻、新闻打开不关搜索:两者面向不同任务(搜索定位 / 浏览态势新闻),允许同屏共存。如果未来 UX 上希望它们互斥,要在 `closeTransientMobileOverlays` 中**同时**改两边的规则,避免单边修改导致非对称的关闭逻辑。
5. **移动端抽屉是 fullscreen 级别的状态。** 一旦进入移动端抽屉,无论是 `layers` / `search` / `settings` 哪一类,都会通过 `setMobileDrawerState` 关闭其它浮层。这是 mobile 单一焦点 UX 的要求。
6. **`Escape` 键有固定的关闭顺序。** 见 [controls.js::setupKeyboardControls](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js):搜索 → 设置 → 移动端抽屉 → 浮动菜单 → 工具栏 hub → 锁定对象。新增浮层要决定它在这个顺序中的位置。
## 新加按钮 / 浮层时怎么接
按下面的清单走,规则就不会乱:
1. 按钮加在 [index.html](/home/ray/dev/linkong/planet/frontend/public/earth/index.html) 的 `.earth-toolbar` 容器里class 跟齐 `floating-btn liquid-glass-surface earth-toolbar-btn`
2. 决定它属于哪一类:
- **独立动作**reload / reset / rotate / layout直接 `bindListener`,不调任何 `closeTransientMobileOverlays`
- **浮动菜单**zoom 这种 dropdown`bindFloatingMenu`,不进协同矩阵。
- **互斥浮层**:进矩阵。
3. 互斥浮层要做两件事:
- 在打开路径调用 `closeTransientMobileOverlays({ except: "<your-key>" })`,让其他浮层主动让位。
-`closeTransientMobileOverlays` 函数体内补一条 `if (except !== "<your-key>" && isYourPanelVisible()) closeYourPanel();` 让别的浮层打开时关掉自己。
4. 如果新浮层和某个现有浮层(例如新闻面板)应当共存,参考第 3 条规则:在自己的关闭判断里 `&& except !== "<peer-key>"` 把对方排除掉。**不要**只单边改一处,否则关闭逻辑会非对称。
5. 新浮层应该有 `Escape` 关闭路径,加在 `setupKeyboardControls` 中合适的位置。
6. 移动端如果应进入抽屉态,使用 `setMobileDrawerState({ open: true, card: "<your-card>" })` 而不是直接 toggle 面板。
## 当前实现位置
- 协调入口:[controls.js::closeTransientMobileOverlays](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js)
- 设置浮层:[controls.js::openSettingsModal / closeSettingsModal](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js)
- 搜索浮层:[controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js)(导入自 search 模块)
- 新闻/直播浮层:[tv.js::setTVPanelVisible](/home/ray/dev/linkong/planet/frontend/public/earth/js/tv.js)、新闻 tab 在 [news.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/news.js)
- 图层抽屉(移动端):[controls.js::setMobileDrawerState](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js)
- 浮动菜单:[controls.js::bindFloatingMenu](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js)
- 工具栏 DOM[index.html](/home/ray/dev/linkong/planet/frontend/public/earth/index.html)

View File

@@ -4,8 +4,8 @@
相关规则建议一起参考:
- [rules.md](/home/ray/dev/linkong/planet/rules.md)
- [frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/zh/frontend-layout-guidelines.md)
- [项目规则](/home/ray/dev/linkong/planet/rules.md)
- [前端布局指南](/home/ray/dev/linkong/planet/docs/technical/zh/frontend-layout-guidelines.md)
## 当前目标
@@ -292,7 +292,7 @@
相关后端设计见:
- [datasource-collector-settings-connectivity.md](/home/ray/dev/linkong/planet/docs/technical/zh/datasource-collector-settings-connectivity.md)
- [数据源、采集器设置与连接验证](/home/ray/dev/linkong/planet/docs/technical/zh/datasource-collector-settings-connectivity.md)
### 3. 复杂工作区页面
@@ -320,4 +320,4 @@
详细经验见:
- [frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/zh/frontend-layout-guidelines.md)
- [前端布局指南](/home/ray/dev/linkong/planet/docs/technical/zh/frontend-layout-guidelines.md)

View File

@@ -7,7 +7,7 @@
- 控制台:登录后的管理后台
- Docs公开开发文档与使用手册
快速启动路径见 [quickstart.md](/home/ray/dev/linkong/planet/docs/technical/zh/quickstart.md)。
快速启动路径见 [快速开始](/home/ray/dev/linkong/planet/docs/technical/zh/quickstart.md)。
## 入口总览
@@ -55,6 +55,52 @@
| `--allow-lan` | 允许局域网访问 |
| `--verbose` | 在执行过程中显示更多命令输出 |
### AI Provider 环境变量和构建
AI Provider 的运行期配置可以放在两处:
| 位置 | 适合内容 | 说明 |
| --- | --- | --- |
| `aiprovider/.env` | 团队约定的本地默认配置 | Docker Compose 会作为 `env_file` 读取 |
| `~/.zshrc` | 个人机器上的 provider、模型、密钥和代理变量 | `planet.sh` 启动时会读取常见的 `AI_*``SERVICE_*``PYTHON_IMAGE``UV_IMAGE`、代理变量 |
推荐写法:
```bash
export AI_PROVIDER=minimax
export AI_PROVIDER_API=anthropic-messages
export AI_BASE_URL=https://api.example.com/anthropic
export AI_API_KEY=sk-change-me
export AI_MODEL=MiniMax-M2.7
export AI_PROVIDER_SERVICE_TOKEN=change_me
```
默认情况下,`planet.sh` 只静态解析 `~/.zshrc` 中简单的 `export KEY=value``KEY=value` 行,避免 shell 主题、插件或交互初始化拖慢启动。如果变量依赖复杂 shell 展开,可以显式启用 source 模式:
```bash
PLANET_LOAD_ZSHRC_ENV=source ./planet.sh start -a
```
如需完全忽略 `~/.zshrc`
```bash
PLANET_LOAD_ZSHRC_ENV=0 ./planet.sh start -a
```
AI Provider 镜像只在代码、Dockerfile、Compose 配置或相关 Python 依赖变化时重建。修改 `aiprovider/.env``~/.zshrc` 中的模型、密钥、Base URL 不会触发镜像重建;重启 AI Provider 即可让容器读取新配置:
```bash
./planet.sh restart -a
```
构建较慢时,优先判断当前卡在哪一层:
| 现象 | 常见原因 | 处理方式 |
| --- | --- | --- |
| `transferring context` 很大 | Docker build context 包含前端资源、PDF、数据目录等无关文件 | 当前仓库通过 `.dockerignore` 只发送 AI Provider 必需文件 |
| `uv sync` 下载依赖较慢 | 首次构建或缓存为空,网络访问 Python 包较慢 | 等待首次构建完成;后续会复用 BuildKit 的 uv 下载缓存 |
| 改密钥后仍显示旧配置 | 容器尚未重启 | 执行 `./planet.sh restart -a` |
### 停止
```bash
@@ -178,7 +224,7 @@ Earth 用于在一个地球视图中观察:
- 卫星和轨迹
- 海缆与登陆点
- 算力中心
- 国界、经纬线、高清材质、云图、地形
- 国界线、经纬线、高清材质、云图、地形
- 新闻直播和态势新闻
- 搜索和聚焦对象详情
@@ -189,7 +235,7 @@ Earth 用于在一个地球视图中观察:
常见图层包括:
- 经纬线
- 国界
- 国界线
- 高清材质
- 大气云图
- 海缆
@@ -214,7 +260,7 @@ Earth 用于在一个地球视图中观察:
- 海缆
- 卫星
- 国界
- 国界线
- 算力中心
- BGP
- AIS 船只
@@ -583,10 +629,10 @@ source ~/.zshrc && bun run build
## 相关文档
- [quickstart.md](/home/ray/dev/linkong/planet/docs/technical/zh/quickstart.md)
- [frontend-admin-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/zh/frontend-admin-frontend-context.md)
- [earth-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/zh/earth-frontend-context.md)
- [earth-layer-style-reference.md](/home/ray/dev/linkong/planet/docs/technical/zh/earth-layer-style-reference.md)
- [backend-system-service-control.md](/home/ray/dev/linkong/planet/docs/technical/zh/backend-system-service-control.md)
- [backend-collectors.md](/home/ray/dev/linkong/planet/docs/technical/zh/backend-collectors.md)
- [datasource-collector-settings-connectivity.md](/home/ray/dev/linkong/planet/docs/technical/zh/datasource-collector-settings-connectivity.md)
- [快速开始](/home/ray/dev/linkong/planet/docs/technical/zh/quickstart.md)
- [控制台前端结构](/home/ray/dev/linkong/planet/docs/technical/zh/frontend-admin-frontend-context.md)
- [Earth 前端结构](/home/ray/dev/linkong/planet/docs/technical/zh/earth-frontend-context.md)
- [Earth 图层样式属性索引](/home/ray/dev/linkong/planet/docs/technical/zh/earth-layer-style-reference.md)
- [系统服务控制](/home/ray/dev/linkong/planet/docs/technical/zh/backend-system-service-control.md)
- [数据采集系统](/home/ray/dev/linkong/planet/docs/technical/zh/backend-collectors.md)
- [数据源、采集器设置与连接验证](/home/ray/dev/linkong/planet/docs/technical/zh/datasource-collector-settings-connectivity.md)

View File

@@ -46,6 +46,8 @@ compute_ai_provider_build_fingerprint() {
find aiprovider \
-type f \
! -path '*/__pycache__/*' \
! -name '.env' \
! -name '.env.*' \
! -name '*.pyc' \
! -name '*.pyo' \
| LC_ALL=C sort \
@@ -57,6 +59,58 @@ compute_ai_provider_build_fingerprint() {
速度提升约 10 倍大量小文件场景误报率相同mtime+size 变化 ≡ 文件被修改)。
`.env``.env.*` 被排除在 fingerprint 外。它们属于运行期配置,不应该因为修改模型、密钥或 Base URL 触发镜像重建。
### Docker build context 收敛
AI Provider 镜像只需要根目录的 `pyproject.toml``uv.lock``aiprovider/` 代码。仓库中还包含前端静态大图、PDF、历史数据和 Unreal 资料,如果 build context 使用整个仓库,`transferring context` 会浪费大量时间。
当前通过根目录 `.dockerignore` 收敛上下文:
```dockerignore
**
!pyproject.toml
!uv.lock
!aiprovider/
!aiprovider/**
aiprovider/.env
aiprovider/.env.*
!aiprovider/.env.example
```
Dockerfile 也从全仓复制改为只复制 AI Provider 代码:
```dockerfile
COPY pyproject.toml uv.lock /app/
RUN --mount=type=cache,target=/root/.cache/uv \
uv sync --frozen --no-dev
COPY aiprovider /app/aiprovider
```
`uv sync` 使用 BuildKit cache mount 后,首次构建仍可能受网络影响;后续构建会复用 `/root/.cache/uv`,依赖下载不再重复从零开始。
### 运行期配置来源
`planet.sh` 启动 AI Provider 前会生成临时 env-file并把它传给 Compose 或手动 `docker run` fallback。配置优先来自
1. `aiprovider/.env`
2. `~/.zshrc` 中简单的 `export AI_...=...``AI_...=...`
默认解析是静态的,只覆盖 AI Provider、镜像、代理相关变量避免执行交互 shell 初始化。如果确实需要复杂 shell 展开,可以显式启用:
```bash
PLANET_LOAD_ZSHRC_ENV=source ./planet.sh start -a
```
如果排查时需要忽略个人 shell 配置:
```bash
PLANET_LOAD_ZSHRC_ENV=0 ./planet.sh start -a
```
### 跳过重建的原理
fingerprint 一致时不执行 `docker compose build`,而是:
@@ -151,4 +205,7 @@ PY
## 相关文件
- `planet.sh` — 全量修改
- `.dockerignore` — 收敛 AI Provider Docker build context
- `aiprovider/Dockerfile` — 只复制 AI Provider 代码,并为 `uv sync` 启用 BuildKit cache mount
- `docker-compose.yml` / `docker-compose.simple.yml` — 读取 `planet.sh` 生成的运行期 env-file
- `scripts/compute_aiprovider_dependency_fingerprint.py` — 依赖 fingerprint未改动

View File

@@ -24,6 +24,12 @@
- `aiprovider/.env`
- `frontend/.env.local`
AI Provider 的个人配置也可以放在 `~/.zshrc``planet.sh` 会读取简单的 `export AI_...=...``AI_...=...` 行,并在启动 AI Provider 时传给容器。修改模型、密钥或 Base URL 后,通常只需要重启 AI Provider
```bash
./planet.sh restart -a
```
## 1. 启动服务
在仓库根目录执行:
@@ -188,7 +194,7 @@ ss -ltnp | grep -E ':3000|:8000'
## 下一步
- 完整操作说明见 [manual.md](/home/ray/dev/linkong/planet/docs/technical/zh/manual.md)
- 控制台结构见 [frontend-admin-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/zh/frontend-admin-frontend-context.md)
- Earth 结构见 [earth-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/zh/earth-frontend-context.md)
- 后端采集器见 [backend-collectors.md](/home/ray/dev/linkong/planet/docs/technical/zh/backend-collectors.md)
- 完整操作说明见 [Planet 使用手册](/home/ray/dev/linkong/planet/docs/technical/zh/manual.md)
- 控制台结构见 [控制台前端结构](/home/ray/dev/linkong/planet/docs/technical/zh/frontend-admin-frontend-context.md)
- Earth 结构见 [Earth 前端结构](/home/ray/dev/linkong/planet/docs/technical/zh/earth-frontend-context.md)
- 后端采集器见 [数据采集系统](/home/ray/dev/linkong/planet/docs/technical/zh/backend-collectors.md)

View File

@@ -16,12 +16,20 @@
## Current Version
- `main` 当前主线历史推导到:`0.16.5`
- `dev` 当前开发分支历史推导到:`0.44.1`
- `dev` 当前开发分支历史推导到:`0.48.0`
## Timeline
| Version | Type | Branch | Commit | Summary |
| --- | --- | --- | --- | --- |
| `0.48.0` | feature | `dev` | `pending` | 新增自定义源 REST/WebSocket 实时 mock 链路,完善 AIS 多源聚合/船舶 enrichment并将 Earth 全球态势统计改为轻量 SQL 聚合 |
| `0.47.0` | feature | `dev` | `pending` | 新增 AISStream WebSocket 船只采集器、多源 AIS 原始观测聚合、采集器状态配置、船型显示修正和文档规则解耦 |
| `0.46.3` | bugfix | `dev` | `pending` | 优化 Starlink footprint 拖拽性能,避免旋转地球时重复重建覆盖网格,并恢复线缆点击呼吸动画 |
| `0.46.2` | bugfix | `dev` | `pending` | 修复 Earth 启动加载顺序、图层 localStorage 恢复、国界线底图语义、媒体面板、船只轨迹和 Iridium footprint 显示问题,并补充 AIS 聚合计划 |
| `0.46.1` | bugfix | `dev` | `pending` | 修复新增 Docs 技术文档未进前端白名单导致页面不可访问的问题,补齐英文文档并固化白名单/双语/裸文件标题检查 |
| `0.46.0` | feature | `dev` | `pending` | Earth 新增通用 Interactable 图标层统一船只、算力中心、BGP 事件/观测站交互图标,并优化登陆点与 toolbar 初始渲染 |
| `0.45.0` | feature | `dev` | `pending` | 新增采集任务 fetching 阶段量化进度,收敛 AI Provider 运行期环境注入和 Docker build context |
| `0.44.2` | bugfix | `dev` | `pending` | 补充 Earth 船只批量渲染、屏幕拾取、图层顺序、样式参考和性能计划状态文档 |
| `0.44.1` | bugfix | `dev` | `pending` | 优化 Earth 船只批量渲染性能,修复拖动卡顿、拾取错位、交互态方向/尺寸和地表压盖问题 |
| `0.44.0` | feature | `dev` | `pending` | 重构数据源目录与采集器设置,新增 BarentsWatch AIS 连接教程、Earth 船只/缩放体验优化和仪表盘前端重启 |
| `0.43.1` | bugfix | `dev` | `pending` | 修正全量 restart 后 AI Provider 启动提示语义,避免把预期未就绪描述成异常 |

View File

@@ -1,6 +1,6 @@
{
"name": "planet-frontend",
"version": "0.44.1",
"version": "0.48.0",
"private": true,
"packageManager": "bun@1",
"dependencies": {

View File

@@ -0,0 +1,3 @@
<svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" fill="#ffffff" viewBox="0 0 16 16">
<path d="M3.05 3.05a7 7 0 0 0 0 9.9.5.5 0 0 1-.707.707 8 8 0 0 1 0-11.314.5.5 0 0 1 .707.707zm2.122 2.122a4 4 0 0 0 0 5.656.5.5 0 1 1-.708.708 5 5 0 0 1 0-7.072.5.5 0 0 1 .708.708zm5.656-.708a.5.5 0 0 1 .708 0 5 5 0 0 1 0 7.072.5.5 0 1 1-.708-.708 4 4 0 0 0 0-5.656.5.5 0 0 1 0-.708zm2.122-2.12a.5.5 0 0 1 .707 0 8 8 0 0 1 0 11.313.5.5 0 0 1-.707-.707 7 7 0 0 0 0-9.9.5.5 0 0 1 0-.707zM6 8a2 2 0 1 1 2.5 1.937V15.5a.5.5 0 0 1-1 0V9.937A2 2 0 0 1 6 8z"/>
</svg>

After

Width:  |  Height:  |  Size: 562 B

View File

@@ -1,16 +1,6 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 128 128">
<!-- GPU cluster marker: database/cylinder stack icon. -->
<!-- Color: #2dd4bf (teal) per COMPUTE_CENTER_CONFIG.colors.gpu_cluster -->
<!-- States: normal, estimated (adds a "?" badge drawn separately at canvas level) -->
<!-- Outer cylinder: top ellipse cap + side rect + bottom half-ellipse -->
<!-- Inner groove ring: smaller cylinder shape overlaid at same color (subtle shape layering) -->
<g fill="#2dd4bf">
<rect x="46" y="46" width="36" height="28"/>
<ellipse cx="64" cy="46" rx="18" ry="8"/>
<path d="M 82,74 A 18,8 0 0,1 46,74 Z"/>
<rect x="52" y="58" width="24" height="6"/>
<ellipse cx="64" cy="58" rx="12" ry="4.5"/>
<path d="M 76,64 A 12,4.5 0 0,1 52,64 Z"/>
</g>
<svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" fill="#2dd4bf" viewBox="0 0 16 16">
<path fill-rule="evenodd" d="M3.904 1.777C4.978 1.289 6.427 1 8 1s3.022.289 4.096.777C13.125 2.245 14 2.993 14 4s-.875 1.755-1.904 2.223C11.022 6.711 9.573 7 8 7s-3.022-.289-4.096-.777C2.875 5.755 2 5.007 2 4s.875-1.755 1.904-2.223Z"/>
<path d="M2 6.161V7c0 1.007.875 1.755 1.904 2.223C4.978 9.71 6.427 10 8 10s3.022-.289 4.096-.777C13.125 8.755 14 8.007 14 7v-.839c-.457.432-1.004.751-1.49.972C11.278 7.693 9.682 8 8 8s-3.278-.307-4.51-.867c-.486-.22-1.033-.54-1.49-.972Z"/>
<path d="M2 9.161V10c0 1.007.875 1.755 1.904 2.223C4.978 12.711 6.427 13 8 13s3.022-.289 4.096-.777C13.125 11.755 14 11.007 14 10v-.839c-.457.432-1.004.751-1.49.972-1.232.56-2.828.867-4.51.867s-3.278-.307-4.51-.867c-.486-.22-1.033-.54-1.49-.972Z"/>
<path d="M2 12.161V13c0 1.007.875 1.755 1.904 2.223C4.978 15.711 6.427 16 8 16s3.022-.289 4.096-.777C13.125 14.755 14 14.007 14 13v-.839c-.457.432-1.004.751-1.49.972-1.232.56-2.828.867-4.51.867s-3.278-.307-4.51-.867c-.486-.22-1.033-.54-1.49-.972Z"/>
</svg>

Before

Width:  |  Height:  |  Size: 787 B

After

Width:  |  Height:  |  Size: 1.1 KiB

View File

@@ -0,0 +1,3 @@
<svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" fill="#f8fafc" viewBox="0 0 16 16">
<path d="M2 2a2 2 0 0 0-2 2v1a2 2 0 0 0 2 2h5.5v3A1.5 1.5 0 0 0 6 11.5H.5a.5.5 0 0 0 0 1H6A1.5 1.5 0 0 0 7.5 14h1a1.5 1.5 0 0 0 1.5-1.5h5.5a.5.5 0 0 0 0-1H10A1.5 1.5 0 0 0 8.5 10V7H14a2 2 0 0 0 2-2V4a2 2 0 0 0-2-2H2zm.5 3a.5.5 0 1 1 0-1 .5.5 0 0 1 0 1zm2 0a.5.5 0 1 1 0-1 .5.5 0 0 1 0 1z"/>
</svg>

After

Width:  |  Height:  |  Size: 399 B

View File

@@ -1,10 +1,3 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 128 128">
<!-- Supercomputer marker: flat-screen monitor with neck and base stand. -->
<!-- Color: #38bdf8 (sky-blue) per COMPUTE_CENTER_CONFIG.colors.supercomputer -->
<!-- States: normal, estimated (adds a "?" badge drawn separately at canvas level) -->
<g fill="#38bdf8">
<rect x="40" y="42" width="48" height="30" rx="7"/>
<rect x="58" y="74" width="12" height="8" rx="3"/>
<rect x="50" y="84" width="28" height="5" rx="2.5"/>
</g>
<svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" fill="#38bdf8" viewBox="0 0 16 16">
<path d="M1.5 0A1.5 1.5 0 0 0 0 1.5v7A1.5 1.5 0 0 0 1.5 10H6v1H1a1 1 0 0 0-1 1v3a1 1 0 0 0 1 1h14a1 1 0 0 0 1-1v-3a1 1 0 0 0-1-1h-5v-1h4.5A1.5 1.5 0 0 0 16 8.5v-7A1.5 1.5 0 0 0 14.5 0h-13Zm0 1h13a.5.5 0 0 1 .5.5v7a.5.5 0 0 1-.5.5h-13a.5.5 0 0 1-.5-.5v-7a.5.5 0 0 1 .5-.5ZM12 12.5a.5.5 0 1 1 1 0 .5.5 0 0 1-1 0Zm2 0a.5.5 0 1 1 1 0 .5.5 0 0 1-1 0ZM1.5 12h5a.5.5 0 0 1 0 1h-5a.5.5 0 0 1 0-1ZM1 14.25a.25.25 0 0 1 .25-.25h5.5a.25.25 0 1 1 0 .5h-5.5a.25.25 0 0 1-.25-.25Z"/>
</svg>

Before

Width:  |  Height:  |  Size: 520 B

After

Width:  |  Height:  |  Size: 578 B

View File

@@ -0,0 +1,3 @@
<svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" fill="currentColor" class="bi bi-geo-fill" viewBox="0 0 16 16">
<path fill-rule="evenodd" d="M4 4a4 4 0 1 1 4.5 3.969V13.5a.5.5 0 0 1-1 0V7.97A4 4 0 0 1 4 3.999zm2.493 8.574a.5.5 0 0 1-.411.575c-.712.118-1.28.295-1.655.493a1.319 1.319 0 0 0-.37.265.301.301 0 0 0-.057.09V14l.002.008a.147.147 0 0 0 .016.033.617.617 0 0 0 .145.15c.165.13.435.27.813.395.751.25 1.82.414 3.024.414s2.273-.163 3.024-.414c.378-.126.648-.265.813-.395a.619.619 0 0 0 .146-.15.148.148 0 0 0 .015-.033L12 14v-.004a.301.301 0 0 0-.057-.09 1.318 1.318 0 0 0-.37-.264c-.376-.198-.943-.375-1.655-.493a.5.5 0 1 1 .164-.986c.77.127 1.452.328 1.957.594C12.5 13 13 13.4 13 14c0 .426-.26.752-.544.977-.29.228-.68.413-1.116.558-.878.293-2.059.465-3.34.465-1.281 0-2.462-.172-3.34-.465-.436-.145-.826-.33-1.116-.558C3.26 14.752 3 14.426 3 14c0-.599.5-1 .961-1.243.505-.266 1.187-.467 1.957-.594a.5.5 0 0 1 .575.411z"/>
</svg>

After

Width:  |  Height:  |  Size: 953 B

View File

@@ -24,16 +24,16 @@
}
.earth-toolbar {
--toolbar-scale: 1;
--toolbar-orb-size: calc(46px * var(--toolbar-scale));
--toolbar-hub-size: calc(58px * var(--toolbar-scale));
--toolbar-arc-width: calc(420px * var(--toolbar-scale));
--toolbar-arc-height: calc(160px * var(--toolbar-scale));
--toolbar-inner-arc-width: calc(260px * var(--toolbar-scale));
--toolbar-inner-arc-height: calc(56px * var(--toolbar-scale));
--toolbar-scale: var(--initial-toolbar-scale, 1);
--toolbar-orb-size: var(--initial-toolbar-orb-size, calc(46px * var(--toolbar-scale)));
--toolbar-hub-size: var(--initial-toolbar-hub-size, calc(58px * var(--toolbar-scale)));
--toolbar-arc-width: var(--initial-toolbar-arc-width, calc(420px * var(--toolbar-scale)));
--toolbar-arc-height: var(--initial-toolbar-arc-height, calc(160px * var(--toolbar-scale)));
--toolbar-inner-arc-width: var(--initial-toolbar-inner-arc-width, calc(260px * var(--toolbar-scale)));
--toolbar-inner-arc-height: var(--initial-toolbar-inner-arc-height, calc(56px * var(--toolbar-scale)));
position: relative;
width: min(620px, calc(100vw - 40px));
height: calc(200px * var(--toolbar-scale));
height: var(--initial-toolbar-height, calc(200px * var(--toolbar-scale)));
display: flex;
align-items: center;
justify-content: center;
@@ -155,7 +155,8 @@
height: var(--toolbar-orb-size);
min-width: var(--toolbar-orb-size);
min-height: var(--toolbar-orb-size);
border-radius: 50%;
aspect-ratio: 1 / 1;
border-radius: 9999px;
overflow: hidden;
}
@@ -164,7 +165,8 @@
height: var(--toolbar-hub-size);
min-width: var(--toolbar-hub-size);
min-height: var(--toolbar-hub-size);
border-radius: 50%;
aspect-ratio: 1 / 1;
border-radius: 9999px;
overflow: hidden;
color: var(--hud-title);
}

View File

@@ -44,6 +44,87 @@
document.documentElement.style.setProperty("--hud-scale", clampedScale.toFixed(3));
})();
(function applyInitialToolbarLayout() {
var width = window.innerWidth;
var height = window.innerHeight;
var toolbarBaseWidth = 620;
var toolbarMinScale = 0.68;
var orbSizeBase = 46;
var hubSizeBase = 58;
var orbGapBase = 12;
var archSpanBase = 232;
var archRiseBase = 40;
var sidePaddingBase = 12;
var bottomClearanceBase = 34;
var extraHeightBase = 34;
var visibleOrbCount = 8;
var toolbarWidth = Math.min(toolbarBaseWidth, Math.max(0, width - 40));
var viewportScale = Math.min(width / 1920, height / 1080);
var toolbarScale = Math.max(
toolbarMinScale,
Math.min(1, Math.min(toolbarWidth / toolbarBaseWidth, viewportScale)),
);
var orbSize = orbSizeBase * toolbarScale;
var desiredGap = orbGapBase * toolbarScale;
var span = archSpanBase * toolbarScale;
var rise = archRiseBase * toolbarScale;
var minSpanForSpacing =
visibleOrbCount > 1
? (visibleOrbCount - 1) * (orbSize + desiredGap)
: orbSize;
var maxSpanByWidth =
toolbarWidth - orbSize - sidePaddingBase * 2 * toolbarScale;
if (minSpanForSpacing > maxSpanByWidth) {
toolbarScale = Math.max(
toolbarMinScale,
Math.min(toolbarScale, maxSpanByWidth / minSpanForSpacing),
);
orbSize = orbSizeBase * toolbarScale;
desiredGap = orbGapBase * toolbarScale;
span = archSpanBase * toolbarScale;
rise = archRiseBase * toolbarScale;
minSpanForSpacing =
visibleOrbCount > 1
? (visibleOrbCount - 1) * (orbSize + desiredGap)
: orbSize;
maxSpanByWidth =
toolbarWidth - orbSize - sidePaddingBase * 2 * toolbarScale;
}
span = Math.max(minSpanForSpacing, Math.min(maxSpanByWidth, span));
rise = Math.min(rise, span * 0.32);
var hubSize = hubSizeBase * toolbarScale;
var maxVerticalReach = rise + orbSize * 0.5;
var toolbarHeight =
maxVerticalReach +
hubSize +
bottomClearanceBase * toolbarScale +
extraHeightBase * toolbarScale;
var rootStyle = document.documentElement.style;
var roundedOrbSize = Math.round(orbSize);
var roundedHubSize = Math.round(hubSize);
rootStyle.setProperty("--initial-toolbar-scale", toolbarScale.toFixed(3));
rootStyle.setProperty("--initial-toolbar-orb-size", roundedOrbSize + "px");
rootStyle.setProperty("--initial-toolbar-hub-size", roundedHubSize + "px");
rootStyle.setProperty("--initial-toolbar-height", Math.ceil(toolbarHeight) + "px");
rootStyle.setProperty(
"--initial-toolbar-arc-width",
Math.ceil(span + orbSize + sidePaddingBase * 2 * toolbarScale) + "px",
);
rootStyle.setProperty(
"--initial-toolbar-arc-height",
Math.ceil(rise + orbSize * 0.95) + "px",
);
rootStyle.setProperty("--initial-toolbar-inner-arc-width", Math.ceil(span * 0.72) + "px");
rootStyle.setProperty(
"--initial-toolbar-inner-arc-height",
Math.ceil(hubSize * 0.8 + desiredGap * 0.5) + "px",
);
})();
</script>
<link rel="stylesheet" href="css/base.css">
<link rel="stylesheet" href="css/hud.css">
@@ -396,6 +477,10 @@
<span class="stats-footer-dot"></span>
<span id="bgp-status-summary" class="stats-footer-text" data-earth-stat="bgp-status-summary">暂无观测数据</span>
</div>
<div class="stats-footer">
<span class="stats-footer-dot"></span>
<span id="vessel-live-summary" class="stats-footer-text" data-earth-stat="vessel-live-summary">AISStream 未连接</span>
</div>
<!-- hidden elements kept for JS compatibility -->
<span id="terrain-status" data-earth-stat="terrain-status" hidden></span>
@@ -627,6 +712,7 @@
<div class="earth-mobile-situation-card">
<div class="earth-mobile-situation-card-title">BGP 状态</div>
<div id="mobile-bgp-status-summary" class="earth-mobile-situation-status" data-earth-stat="bgp-status-summary">暂无观测数据</div>
<div id="mobile-vessel-live-summary" class="earth-mobile-situation-status" data-earth-stat="vessel-live-summary">AISStream 未连接</div>
</div>
</div>
</section>

View File

@@ -1,10 +1,13 @@
import * as THREE from "three";
import { BGP_CONFIG, CONFIG, PATHS } from "./constants.js";
import { createInteractableLayer } from "./interactable.js";
import { getSurfaceMarkerCameraScale, latLonToVector3 } from "./utils.js";
const bgpGroup = new THREE.Group();
const bgpOverlayGroup = new THREE.Group();
const bgpEventOverlayGroup = new THREE.Group();
const bgpCollectorRadarGroup = new THREE.Group();
const collectorMarkers = [];
const anomalyMarkers = [];
const activeEventCountByCollector = new Map();
@@ -14,7 +17,6 @@ let totalAnomalyCount = 0;
let totalIncidentCount = 0;
let textureCache = null;
let eventRingTextureCache = null;
let collectorTextureCache = null;
const eventTextureCache = new Map();
let activeEventOverlay = null;
let activeCollectorOverlayContext = null;
@@ -22,17 +24,27 @@ const relativeTimeFormatter = new Intl.RelativeTimeFormat("zh-CN", {
numeric: "auto",
});
const collectorWorldPosition = new THREE.Vector3();
const collectorSurfaceNormal = new THREE.Vector3();
const collectorNorthPole = new THREE.Vector3(0, 1, 0);
const collectorFallbackForward = new THREE.Vector3(0, 0, 1);
const collectorNorthTangent = new THREE.Vector3();
const collectorEastTangent = new THREE.Vector3();
const collectorOrientationMatrix = new THREE.Matrix4();
const colorScratchA = new THREE.Color();
const colorScratchB = new THREE.Color();
const COLLECTOR_SCAN_SPEED_RAD = 0.00018;
const COLLECTOR_SCAN_REBUILD_MS = 80;
const MATERIAL_ACCESS_POINT_PATH = "M4.93 4.93A9.97 9.97 0 0 0 2 12c0 2.76 1.12 5.26 2.93 7.07l1.41-1.41A7.94 7.94 0 0 1 4 12c0-2.21.89-4.22 2.34-5.66zm14.14 0l-1.41 1.41A7.96 7.96 0 0 1 20 12c0 2.22-.89 4.22-2.34 5.66l1.41 1.41A9.97 9.97 0 0 0 22 12c0-2.76-1.12-5.26-2.93-7.07M7.76 7.76A5.98 5.98 0 0 0 6 12c0 1.65.67 3.15 1.76 4.24l1.41-1.41A4 4 0 0 1 8 12c0-1.11.45-2.11 1.17-2.83zm8.48 0l-1.41 1.41A4 4 0 0 1 16 12c0 1.11-.45 2.11-1.17 2.83l1.41 1.41A5.98 5.98 0 0 0 18 12c0-1.65-.67-3.15-1.76-4.24M12 10a2 2 0 0 0-2 2a2 2 0 0 0 2 2a2 2 0 0 0 2-2a2 2 0 0 0-2-2";
const BGP_EVENT_RENDER_ORDER = 4.5;
const BGP_EVENT_POINT_SIZE = 34;
const BGP_EVENT_SYMBOL_SIZE = 60;
const BGP_COLLECTOR_RENDER_ORDER = 4.4;
const BGP_COLLECTOR_ALTITUDE_OFFSET = BGP_CONFIG.collectorAltitudeOffset;
const BGP_COLLECTOR_POINT_SIZE = 36;
const BGP_COLLECTOR_ICON_FIT_SIZE = 60;
const BGP_COLLECTOR_ICON_SOURCE = "/earth/assets/icons/bgp-broadcast-pin.svg";
const BGP_COLLECTOR_HOVER_SCALE = 1.08;
const BGP_COLLECTOR_LOCKED_SCALE = 1.12;
const BGP_COLLECTOR_PULSE_AMPLITUDE = 0.14;
bgpOverlayGroup.name = "bgp-overlay-root";
bgpEventOverlayGroup.name = "bgp-event-overlay-layer";
bgpCollectorRadarGroup.name = "bgp-collector-radar-layer";
bgpOverlayGroup.add(bgpEventOverlayGroup);
bgpOverlayGroup.add(bgpCollectorRadarGroup);
function getMarkerTexture() {
if (textureCache) return textureCache;
@@ -85,48 +97,6 @@ function getEventRingTexture() {
return eventRingTextureCache;
}
function getCollectorTexture() {
if (collectorTextureCache) return collectorTextureCache;
const canvas = document.createElement("canvas");
canvas.width = 128;
canvas.height = 128;
const context = canvas.getContext("2d");
if (!context) {
collectorTextureCache = new THREE.Texture(canvas);
return collectorTextureCache;
}
context.clearRect(0, 0, 128, 128);
context.strokeStyle = BGP_CONFIG.collectorIcon.ringStroke;
context.lineWidth = BGP_CONFIG.collectorIcon.ringLineWidth;
context.beginPath();
context.arc(64, 64, BGP_CONFIG.collectorIcon.ringRadius, 0, Math.PI * 2);
context.stroke();
context.save();
context.translate(16, 16);
context.scale(4, 4);
const path = new Path2D(MATERIAL_ACCESS_POINT_PATH);
context.lineJoin = "round";
context.lineCap = "round";
context.lineWidth = BGP_CONFIG.collectorIcon.pathLineWidth;
context.strokeStyle = BGP_CONFIG.collectorIcon.pathStroke;
context.stroke(path);
context.fillStyle = BGP_CONFIG.collectorIcon.pathFill;
context.shadowBlur = 0;
context.fill(path);
context.fillStyle = BGP_CONFIG.collectorIcon.centerFill;
context.beginPath();
context.arc(12, 12, BGP_CONFIG.collectorIcon.centerRadius, 0, Math.PI * 2);
context.fill();
context.restore();
collectorTextureCache = new THREE.CanvasTexture(canvas);
return collectorTextureCache;
}
function getEventSymbolKind(anomalyType) {
const value = String(anomalyType || "").toLowerCase();
if (value.includes("origin")) return "triangle";
@@ -266,6 +236,169 @@ function getSeverityScale(severity) {
return BGP_CONFIG.severityScales[normalizeSeverity(severity)];
}
function severityColorHex(severity) {
return `#${getSeverityColor(severity).toString(16).padStart(6, "0")}`;
}
function colorNumberHex(colorNumber) {
return `#${Number(colorNumber || 0xffffff).toString(16).padStart(6, "0")}`;
}
function drawBGPEventIcon(context, { marker, color = "#ffffff", glow = false }) {
const kind = getEventSymbolKind(
marker?.userData?.incident_type || marker?.userData?.anomaly_type,
);
context.save();
context.fillStyle = color;
context.strokeStyle = color;
context.lineJoin = "round";
context.lineCap = "round";
context.shadowColor = color;
context.shadowBlur = glow ? 14 : 0;
const inset = (128 - BGP_EVENT_SYMBOL_SIZE) / 2;
context.translate(inset, inset);
context.scale(BGP_EVENT_SYMBOL_SIZE / 128, BGP_EVENT_SYMBOL_SIZE / 128);
if (kind === "triangle") {
drawTriangleSymbol(context);
} else if (kind === "exclamation") {
drawExclamationSymbol(context);
} else if (kind === "wave") {
drawWaveSymbol(context);
} else if (kind === "burst") {
drawBurstSymbol(context);
} else if (kind === "leak") {
drawLeakSymbol(context);
} else {
drawDotSymbol(context);
}
context.restore();
}
const bgpEventIconLayer = createInteractableLayer({
id: "bgp-events",
objectType: "bgp",
renderOrder: BGP_EVENT_RENDER_ORDER,
altitudeOffset: BGP_CONFIG.altitudeOffset,
pointSize: BGP_EVENT_POINT_SIZE,
colors: {
byKind: Object.fromEntries(
Object.keys(BGP_CONFIG.severityColors).map((severity) => [
severity,
severityColorHex(severity),
]),
),
normal: severityColorHex("medium"),
},
opacity: {
normal: BGP_CONFIG.opacity.normal,
hover: BGP_CONFIG.opacity.hover,
locked: BGP_CONFIG.opacity.lockedMax,
dimmed: BGP_CONFIG.opacity.dimmed,
},
stateScale: {
hover: BGP_CONFIG.marker.hoverScale,
locked: BGP_CONFIG.marker.hoverScale,
dimmed: BGP_CONFIG.marker.dimmedScale,
},
pulse: {
enabled: true,
speed: BGP_CONFIG.pulse.eventSpeed,
amplitude: BGP_CONFIG.pulse.lockedAmplitude,
},
icon: {
coordinates: "canvas",
draw: drawBGPEventIcon,
},
getPosition: (item) => ({
latitude: item.latitude,
longitude: item.longitude,
}),
getKind: (item) => normalizeSeverity(item.severity),
getBucketKey: (marker) => [
getEventSymbolKind(marker.userData?.incident_type || marker.userData?.anomaly_type),
normalizeSeverity(marker.userData?.severity),
].join(":"),
getPointSizeMultiplier: (marker) => getSeverityScale(marker.userData?.severity),
getUserData: (item) => ({
...item,
baseScale: BGP_CONFIG.marker.eventBaseScale * getSeverityScale(item.severity),
baseColor: getSeverityColor(item.severity),
pulseOffset: Math.random() * Math.PI * 2,
}),
});
const bgpCollectorIconLayer = createInteractableLayer({
id: "bgp-collectors",
objectType: "bgp_collector",
renderOrder: BGP_COLLECTOR_RENDER_ORDER,
altitudeOffset: BGP_COLLECTOR_ALTITUDE_OFFSET,
pointSize: BGP_COLLECTOR_POINT_SIZE,
colors: {
byKind: Object.fromEntries(
Object.entries(BGP_CONFIG.collectorHeatColors).map(([tier, color]) => [
tier,
colorNumberHex(color),
]),
),
normal: colorNumberHex(BGP_CONFIG.collectorColor),
},
opacity: {
normal: BGP_CONFIG.collectorIcon.idleOpacity,
hover: 0.98,
locked: 1,
dimmed: BGP_CONFIG.opacity.dimmed,
},
stateScale: {
hover: BGP_COLLECTOR_HOVER_SCALE,
locked: BGP_COLLECTOR_LOCKED_SCALE,
dimmed: BGP_CONFIG.marker.dimmedScale,
},
pulse: {
enabled: true,
speed: BGP_CONFIG.pulse.collectorSpeed,
amplitude: BGP_COLLECTOR_PULSE_AMPLITUDE,
},
icon: {
coordinates: "canvas",
fitSize: BGP_COLLECTOR_ICON_FIT_SIZE,
glowBlur: 16,
source: BGP_COLLECTOR_ICON_SOURCE,
},
getPosition: (item) => ({
latitude: item.displayLatitude,
longitude: item.displayLongitude,
}),
getKind: (item) => getCollectorActivityProfile(item).tier,
getBucketKey: (marker) => {
const scaleBoost = marker.userData?.activity?.scaleBoost ?? 1;
return `${marker.userData?.activity?.tier || "idle"}:${scaleBoost.toFixed(2)}`;
},
getPointSizeMultiplier: (marker) => marker.userData?.activity?.scaleBoost ?? 1,
getUserData: (item) => {
const activity = getCollectorActivityProfile(item);
const baseColor = activity.color;
const idleColor = blendHexColors(
BGP_CONFIG.collectorIcon.idleBaseColor,
baseColor,
BGP_CONFIG.collectorIcon.idleBlend,
);
return {
...item,
baseScale: BGP_CONFIG.marker.collectorBaseScale * activity.scaleBoost,
baseColor,
idleColor,
pulseOffset: Math.random() * Math.PI * 2,
anomaly_count: 0,
activity,
};
},
});
function clamp(value, min, max) {
return Math.min(max, Math.max(min, value));
}
@@ -281,7 +414,7 @@ function getCollectorDistanceScale(marker, camera) {
if (!marker || !camera || BGP_CONFIG.sizeStabilization?.enabled === false) return 1;
return getSurfaceMarkerCameraScale(camera, {
altitudeOffset: BGP_CONFIG.collectorAltitudeOffset,
altitudeOffset: BGP_COLLECTOR_ALTITUDE_OFFSET,
referenceFov: 75,
min: Number(BGP_CONFIG.sizeStabilization?.collectorMin ?? 0.6),
max: Number(BGP_CONFIG.sizeStabilization?.collectorMax ?? 1.9),
@@ -299,32 +432,6 @@ function getEventDistanceScale(marker, camera) {
});
}
function orientCollectorMarkerToSurface(marker, position) {
collectorSurfaceNormal.copy(position).normalize();
collectorNorthTangent
.copy(collectorNorthPole)
.projectOnPlane(collectorSurfaceNormal);
if (collectorNorthTangent.lengthSq() < 1e-6) {
collectorNorthTangent
.copy(collectorFallbackForward)
.projectOnPlane(collectorSurfaceNormal);
}
collectorNorthTangent.normalize();
collectorEastTangent
.copy(collectorNorthTangent)
.cross(collectorSurfaceNormal)
.normalize();
collectorOrientationMatrix.makeBasis(
collectorEastTangent,
collectorNorthTangent,
collectorSurfaceNormal,
);
marker.quaternion.setFromRotationMatrix(collectorOrientationMatrix);
}
function getCollectorActivityProfile(markerData) {
const recent24h = Number(markerData?.recent_24h_observation_count || 0);
const recent7d = Number(markerData?.recent_7d_observation_count || 0);
@@ -784,13 +891,26 @@ function clearMarkerArray(markers) {
const marker = markers.pop();
while (marker.children.length > 0) {
const child = marker.children.pop();
child.geometry?.dispose?.();
child.material?.dispose();
}
disposeCollectorEffectSprites(marker);
disposeEventRingSprite(marker.userData?.ringA);
disposeEventRingSprite(marker.userData?.ringB);
marker.material?.dispose();
bgpGroup.remove(marker);
marker.parent?.remove?.(marker);
}
}
function disposeAnomalyRingSprites() {
anomalyMarkers.forEach((marker) => {
disposeEventRingSprite(marker.userData?.ringA);
disposeEventRingSprite(marker.userData?.ringB);
delete marker.userData.ringA;
delete marker.userData.ringB;
});
}
function clearGroup(group) {
while (group.children.length > 0) {
const child = group.children[group.children.length - 1];
@@ -979,46 +1099,14 @@ function createRadialBoundaryPoints(
return points;
}
function createCollectorMarker(markerData) {
const activity = getCollectorActivityProfile(markerData);
const baseColor = activity.color;
const idleColor = blendHexColors(
BGP_CONFIG.collectorIcon.idleBaseColor,
baseColor,
BGP_CONFIG.collectorIcon.idleBlend,
);
const marker = new THREE.Mesh(
new THREE.PlaneGeometry(1, 1),
new THREE.MeshBasicMaterial({
map: getCollectorTexture(),
color: idleColor,
transparent: true,
opacity: BGP_CONFIG.collectorIcon.idleOpacity,
depthWrite: false,
depthTest: true,
side: THREE.DoubleSide,
}),
);
const position = latLonToVector3(
markerData.displayLatitude,
markerData.displayLongitude,
CONFIG.earthRadius + BGP_CONFIG.collectorAltitudeOffset,
);
marker.position.copy(position);
marker.scale.set(BGP_CONFIG.marker.collectorBaseScale * 0.88 * activity.scaleBoost, BGP_CONFIG.marker.collectorBaseScale * 1.08 * activity.scaleBoost, 1);
marker.renderOrder = 3;
marker.visible = showBGP;
orientCollectorMarkerToSurface(marker, position);
function attachCollectorEffectSprites(marker) {
const activity = marker.userData.activity;
const heatHalo = createOverlaySprite({
color: activity.color,
opacity: 0.0,
scale: activity.haloScale * 0.58,
});
heatHalo.renderOrder = 1;
marker.add(heatHalo);
const pulseHalo = createOverlaySprite({
color: activity.color,
@@ -1026,7 +1114,6 @@ function createCollectorMarker(markerData) {
scale: activity.pulseHaloScale * 0.48,
});
pulseHalo.renderOrder = 0;
marker.add(pulseHalo);
const statusCore = createOverlaySprite({
color: activity.color,
@@ -1037,9 +1124,7 @@ function createCollectorMarker(markerData) {
BGP_CONFIG.marker.collectorStatusCoreMinScale,
),
});
statusCore.position.set(0, 0, 0.02);
statusCore.renderOrder = 4;
marker.add(statusCore);
const coverageHalo = createOverlaySprite({
color: BGP_CONFIG.regionColor,
@@ -1048,65 +1133,52 @@ function createCollectorMarker(markerData) {
});
coverageHalo.renderOrder = 0;
coverageHalo.scale.set(activity.coverageHaloScale * 0.82, activity.coverageHaloScale * 0.56, 1);
marker.add(coverageHalo);
marker.userData = {
type: "bgp_collector",
state: "normal",
baseScale: BGP_CONFIG.marker.collectorBaseScale * activity.scaleBoost,
baseColor,
idleColor,
pulseOffset: Math.random() * Math.PI * 2,
anomaly_count: 0,
activity,
heatHalo,
pulseHalo,
statusCore,
coverageHalo,
...markerData,
};
marker.userData.heatHalo = heatHalo;
marker.userData.pulseHalo = pulseHalo;
marker.userData.statusCore = statusCore;
marker.userData.coverageHalo = coverageHalo;
collectorMarkers.push(marker);
bgpGroup.add(marker);
[heatHalo, pulseHalo, statusCore, coverageHalo].forEach((sprite) => {
sprite.position.copy(marker.position);
sprite.visible = showBGP;
bgpGroup.add(sprite);
});
}
function createAnomalyMarker(markerData) {
const sprite = new THREE.Sprite(
new THREE.SpriteMaterial({
map: getEventTexture(markerData.incident_type || markerData.anomaly_type),
color: getSeverityColor(markerData.severity),
transparent: true,
opacity: BGP_CONFIG.opacity.normal,
depthWrite: false,
depthTest: true,
blending: THREE.NormalBlending,
}),
);
function disposeCollectorEffectSprites(marker) {
[
marker?.userData?.heatHalo,
marker?.userData?.pulseHalo,
marker?.userData?.statusCore,
marker?.userData?.coverageHalo,
].forEach((sprite) => {
if (!sprite) return;
sprite.parent?.remove?.(sprite);
sprite.material?.dispose?.();
sprite.geometry?.dispose?.();
});
}
const position = latLonToVector3(
markerData.latitude,
markerData.longitude,
CONFIG.earthRadius + BGP_CONFIG.altitudeOffset,
);
async function setCollectorMarkers(markerData, earth) {
collectorMarkers.forEach(disposeCollectorEffectSprites);
collectorMarkers.length = 0;
await bgpCollectorIconLayer.preloadAssets(markerData);
bgpCollectorIconLayer.setData(markerData);
bgpCollectorIconLayer.attach(earth);
bgpCollectorIconLayer.setVisible(showBGP);
const baseScale = BGP_CONFIG.marker.eventBaseScale * getSeverityScale(markerData.severity);
sprite.position.copy(position);
sprite.scale.setScalar(baseScale);
sprite.renderOrder = 5;
sprite.visible = showBGP;
sprite.userData = {
type: "bgp",
state: "normal",
baseScale,
baseColor: getSeverityColor(markerData.severity),
pulseOffset: Math.random() * Math.PI * 2,
...markerData,
};
bgpCollectorIconLayer.getMarkers().forEach((marker) => {
attachCollectorEffectSprites(marker);
collectorMarkers.push(marker);
});
}
const ringA = new THREE.Sprite(
function createEventRingSprite(marker) {
const ring = new THREE.Sprite(
new THREE.SpriteMaterial({
map: getEventRingTexture(),
color: getSeverityColor(markerData.severity),
color: marker.userData.baseColor || getSeverityColor(marker.userData.severity),
transparent: true,
opacity: 0,
depthWrite: false,
@@ -1114,30 +1186,40 @@ function createAnomalyMarker(markerData) {
blending: THREE.AdditiveBlending,
}),
);
ringA.scale.setScalar(baseScale * BGP_CONFIG.ring.scaleA);
ringA.position.set(0, 0, -0.01);
sprite.add(ringA);
ring.position.copy(marker.position);
ring.renderOrder = BGP_EVENT_RENDER_ORDER - 0.05;
ring.visible = showBGP;
return ring;
}
const ringB = new THREE.Sprite(
new THREE.SpriteMaterial({
map: getEventRingTexture(),
color: getSeverityColor(markerData.severity),
transparent: true,
opacity: 0,
depthWrite: false,
depthTest: true,
blending: THREE.AdditiveBlending,
}),
);
ringB.scale.setScalar(baseScale * BGP_CONFIG.ring.scaleB);
ringB.position.set(0, 0, -0.02);
sprite.add(ringB);
function disposeEventRingSprite(ring) {
if (!ring) return;
ring.parent?.remove?.(ring);
ring.material?.dispose?.();
ring.geometry?.dispose?.();
}
sprite.userData.ringA = ringA;
sprite.userData.ringB = ringB;
function attachAnomalyRingSprites(marker) {
const ringA = createEventRingSprite(marker);
const ringB = createEventRingSprite(marker);
ringB.visible = false;
marker.userData.ringA = ringA;
marker.userData.ringB = ringB;
bgpGroup.add(ringA);
bgpGroup.add(ringB);
}
anomalyMarkers.push(sprite);
bgpGroup.add(sprite);
function setAnomalyMarkers(markerData, earth) {
disposeAnomalyRingSprites();
anomalyMarkers.length = 0;
bgpEventIconLayer.setData(markerData);
bgpEventIconLayer.attach(earth);
bgpEventIconLayer.setVisible(showBGP);
bgpEventIconLayer.getMarkers().forEach((marker) => {
attachAnomalyRingSprites(marker);
anomalyMarkers.push(marker);
});
}
function dedupeAnomalies(features) {
@@ -1304,17 +1386,18 @@ export async function loadBGPAnomalies(scene, earth) {
totalIncidentCount = selectedEventData.totalIncidentCount;
activeEventCountByCollector.clear();
spreadCollectorPositions(
const collectorMarkersData = spreadCollectorPositions(
collectorFeatures
.map(buildCollectorFeatureData)
.filter(Boolean),
).forEach(createCollectorMarker);
);
await setCollectorMarkers(collectorMarkersData, earth);
if (selectedEventData.mode === "incident") {
dedupeIncidents(selectedEventData.features).forEach(createAnomalyMarker);
} else {
dedupeAnomalies(selectedEventData.features).forEach(createAnomalyMarker);
}
const eventMarkers =
selectedEventData.mode === "incident"
? dedupeIncidents(selectedEventData.features)
: dedupeAnomalies(selectedEventData.features);
setAnomalyMarkers(eventMarkers, earth);
applyCollectorCounts();
if (!bgpGroup.parent) {
@@ -1360,7 +1443,6 @@ export function updateBGPVisualState(lockedObjectType, lockedObject, camera, cru
);
let scale = marker.userData.baseScale * getCollectorDistanceScale(marker, camera);
let opacity = BGP_CONFIG.collectorIcon.idleOpacity;
let haloOpacity = 0.0;
let pulseOpacity = 0.0;
let coverageOpacity = 0.0;
@@ -1374,14 +1456,12 @@ export function updateBGPVisualState(lockedObjectType, lockedObject, camera, cru
if (isLocked) {
scale *= 1.1 + 0.14 * pulse;
opacity = 0.96;
haloOpacity = 0.05;
pulseOpacity = 0.024;
coverageOpacity = 0.036;
markerColor = 0xcff2ff;
} else if (isHovered) {
scale *= 1.08;
opacity = 0.88;
haloOpacity = 0.03;
pulseOpacity = 0.014;
coverageOpacity = 0.02;
@@ -1392,7 +1472,6 @@ export function updateBGPVisualState(lockedObjectType, lockedObject, camera, cru
);
} else if (hasLockedLayer) {
scale *= 0.98;
opacity = BGP_CONFIG.collectorIcon.idleOpacity;
haloOpacity = 0.0;
pulseOpacity = 0.0;
coverageOpacity = 0.0;
@@ -1401,12 +1480,8 @@ export function updateBGPVisualState(lockedObjectType, lockedObject, camera, cru
scale *= 1 + 0.05 * pulse;
}
marker.scale.setScalar(scale);
marker.material.color.setHex(markerColor);
marker.material.opacity = opacity;
marker.visible = showBGP;
if (marker.userData.heatHalo) {
marker.userData.heatHalo.position.copy(marker.position);
marker.userData.heatHalo.material.opacity = haloOpacity;
marker.userData.heatHalo.material.color.setHex(marker.userData.baseColor || BGP_CONFIG.collectorColor);
marker.userData.heatHalo.scale.setScalar(
@@ -1414,6 +1489,7 @@ export function updateBGPVisualState(lockedObjectType, lockedObject, camera, cru
);
}
if (marker.userData.pulseHalo) {
marker.userData.pulseHalo.position.copy(marker.position);
marker.userData.pulseHalo.material.opacity = pulseOpacity;
marker.userData.pulseHalo.material.color.setHex(marker.userData.baseColor || BGP_CONFIG.collectorColor);
marker.userData.pulseHalo.scale.setScalar(
@@ -1421,6 +1497,7 @@ export function updateBGPVisualState(lockedObjectType, lockedObject, camera, cru
);
}
if (marker.userData.statusCore) {
marker.userData.statusCore.position.copy(marker.position);
marker.userData.statusCore.material.opacity =
isLocked ? 0.58 : isHovered ? 0.4 : hasLockedLayer ? 0.0 : 0.18;
marker.userData.statusCore.material.color.setHex(marker.userData.baseColor || BGP_CONFIG.collectorColor);
@@ -1433,6 +1510,7 @@ export function updateBGPVisualState(lockedObjectType, lockedObject, camera, cru
);
}
if (marker.userData.coverageHalo) {
marker.userData.coverageHalo.position.copy(marker.position);
marker.userData.coverageHalo.material.opacity = coverageOpacity;
marker.userData.coverageHalo.scale.set(
marker.userData.activity?.coverageHaloScale * 0.82 * (1 + pulse * 0.012),
@@ -1442,6 +1520,20 @@ export function updateBGPVisualState(lockedObjectType, lockedObject, camera, cru
}
});
const focusedCollector =
lockedObjectType === "bgp"
? collectorMarkers.find(
(marker) => marker.userData.collector === lockedObject?.userData?.collector,
)
: lockedObjectType === "bgp_collector"
? lockedObject
: null;
bgpCollectorIconLayer.updateVisualState(
focusedCollector ? "bgp_collector" : lockedObjectType,
focusedCollector || lockedObject,
camera,
);
anomalyMarkers.forEach((marker) => {
const isLocked = lockedObjectType === "bgp" && lockedObject === marker;
const isLinkedCollectorLocked =
@@ -1459,7 +1551,6 @@ export function updateBGPVisualState(lockedObjectType, lockedObject, camera, cru
const iconAnchorScale =
marker.userData.baseScale * getEventDistanceScale(marker, camera);
let scale = iconAnchorScale;
let opacity = BGP_CONFIG.opacity.normal;
let markerColor = marker.userData.baseColor || getSeverityColor(marker.userData.severity);
const isIncidentMarker = marker.userData.source === "bgp_incident";
let ringBaseOpacity = isIncidentMarker
@@ -1468,49 +1559,38 @@ export function updateBGPVisualState(lockedObjectType, lockedObject, camera, cru
if (isLocked || isLinkedCollectorLocked) {
scale *= 1 + BGP_CONFIG.pulse.lockedAmplitude * pulse;
opacity = 0.9 + 0.1 * pulse;
markerColor = 0xfff1a8;
ringBaseOpacity *= 1.2;
} else if (isCruise) {
scale *= 1 + BGP_CONFIG.pulse.lockedAmplitude * pulse;
opacity = 0.9 + 0.1 * pulse;
ringBaseOpacity *= 1.2;
} else if (isHovered) {
scale *= BGP_CONFIG.marker.hoverScale;
opacity = 0.9;
ringBaseOpacity *= 1.05;
} else if (isOtherLocked) {
scale *= BGP_CONFIG.marker.dimmedScale;
opacity = 0.22;
markerColor = 0x7d8ca3;
ringBaseOpacity = 0.02;
} else {
scale *= 1 + BGP_CONFIG.pulse.normalAmplitude * pulse;
opacity = isIncidentMarker ? 0.7 : 0.62;
}
marker.scale.setScalar(scale);
marker.material.color.setHex(markerColor);
marker.material.opacity = opacity;
marker.visible = showBGP;
marker.renderOrder = isActive ? 7 : 3;
const ringPhaseA = (now * BGP_CONFIG.ring.speed + marker.userData.pulseOffset) % 1;
const ringPhaseA = (now * BGP_CONFIG.ring.speed + marker.userData.pulseOffset) % 1;
const applyRingState = (ring, phase, maxScale) => {
if (!ring) return;
const progress = Math.max(0, Math.min(1, phase));
const minScale = 1.28;
const desiredWorldScale =
iconAnchorScale * (minScale + progress * (maxScale - minScale));
const parentScale = Math.max(scale, 0.0001);
const localRingScale = desiredWorldScale / parentScale;
scale * (minScale + progress * (maxScale - minScale));
const fadeIn = Math.max(0, Math.min(1, (progress - 0.08) / 0.14));
const fadeOut = 1 - progress;
const visibility = fadeIn * fadeOut;
ring.scale.setScalar(localRingScale);
ring.position.copy(marker.position);
ring.scale.setScalar(desiredWorldScale);
ring.material.color.setHex(markerColor);
ring.material.opacity = showBGP ? ringBaseOpacity * visibility : 0;
ring.visible = showBGP;
ring.renderOrder = isActive ? BGP_EVENT_RENDER_ORDER + 0.15 : BGP_EVENT_RENDER_ORDER - 0.05;
};
applyRingState(marker.userData.ringA, ringPhaseA, BGP_CONFIG.ring.scaleA);
@@ -1519,6 +1599,8 @@ export function updateBGPVisualState(lockedObjectType, lockedObject, camera, cru
marker.userData.ringB.visible = false;
}
});
bgpEventIconLayer.updateVisualState(lockedObjectType, lockedObject, camera);
}
export function setBGPMarkerState(marker, state = "normal") {
@@ -1526,15 +1608,19 @@ export function setBGPMarkerState(marker, state = "normal") {
if (marker.userData.type !== "bgp" && marker.userData.type !== "bgp_collector") {
return;
}
marker.userData.state = state;
if (marker.userData.type === "bgp") {
bgpEventIconLayer.setMarkerState(marker, state);
return;
}
bgpCollectorIconLayer.setMarkerState(marker, state);
}
export function clearBGPSelection() {
collectorMarkers.forEach((marker) => {
marker.userData.state = "normal";
bgpCollectorIconLayer.setMarkerState(marker, "normal");
});
anomalyMarkers.forEach((marker) => {
marker.userData.state = "normal";
bgpEventIconLayer.setMarkerState(marker, "normal");
});
clearBGPEventOverlay();
}
@@ -1542,6 +1628,8 @@ export function clearBGPSelection() {
export function clearBGPData(earth) {
clearMarkerArray(collectorMarkers);
clearMarkerArray(anomalyMarkers);
bgpCollectorIconLayer.clearData(earth);
bgpEventIconLayer.clearData(earth);
clearBGPEventOverlay();
activeEventCountByCollector.clear();
totalAnomalyCount = 0;
@@ -1559,11 +1647,21 @@ export function toggleBGP(show) {
showBGP = Boolean(show);
bgpGroup.visible = showBGP;
bgpOverlayGroup.visible = showBGP;
bgpCollectorIconLayer.setVisible(showBGP);
bgpEventIconLayer.setVisible(showBGP);
collectorMarkers.forEach((marker) => {
marker.visible = showBGP;
[
marker.userData.heatHalo,
marker.userData.pulseHalo,
marker.userData.statusCore,
marker.userData.coverageHalo,
].forEach((sprite) => {
if (sprite) sprite.visible = showBGP;
});
});
anomalyMarkers.forEach((marker) => {
marker.visible = showBGP;
marker.userData.ringA && (marker.userData.ringA.visible = showBGP);
marker.userData.ringB && (marker.userData.ringB.visible = false);
});
}
@@ -1579,6 +1677,14 @@ export function getBGPAnomalyMarkers() {
return anomalyMarkers;
}
export function getBGPAnomalyPointerIntersections(options = {}) {
return bgpEventIconLayer.getPointerIntersections(options);
}
export function getBGPCollectorPointerIntersections(options = {}) {
return bgpCollectorIconLayer.getPointerIntersections(options);
}
export function getBGPCollectorMarkers() {
return collectorMarkers;
}
@@ -1644,11 +1750,11 @@ export function showBGPEventOverlay(marker, earth) {
latLonToVector3(
region.latitude,
region.longitude,
CONFIG.earthRadius + BGP_CONFIG.collectorAltitudeOffset - 0.1,
CONFIG.earthRadius + BGP_COLLECTOR_ALTITUDE_OFFSET - 0.1,
),
);
halo.renderOrder = 2;
bgpOverlayGroup.add(halo);
bgpEventOverlayGroup.add(halo);
overlayItems.push(halo);
});
@@ -1687,11 +1793,11 @@ export function showBGPCollectorCoverageOverlay(marker, earth, options = {}) {
latLonToVector3(
marker.userData.displayLatitude ?? marker.userData.latitude,
marker.userData.displayLongitude ?? marker.userData.longitude,
CONFIG.earthRadius + BGP_CONFIG.collectorAltitudeOffset - 0.15,
CONFIG.earthRadius + BGP_COLLECTOR_ALTITUDE_OFFSET - 0.15,
),
);
halo.renderOrder = 2;
bgpOverlayGroup.add(halo);
bgpCollectorRadarGroup.add(halo);
const pulseHalo = createOverlaySprite({
color: BGP_CONFIG.collectorColor,
@@ -1700,7 +1806,7 @@ export function showBGPCollectorCoverageOverlay(marker, earth, options = {}) {
});
pulseHalo.position.copy(halo.position);
pulseHalo.renderOrder = 1;
bgpOverlayGroup.add(pulseHalo);
bgpCollectorRadarGroup.add(pulseHalo);
const innerRing = createOverlaySprite({
color: BGP_CONFIG.collectorColor,
opacity: 0.12,
@@ -1708,7 +1814,7 @@ export function showBGPCollectorCoverageOverlay(marker, earth, options = {}) {
});
innerRing.position.copy(halo.position);
innerRing.renderOrder = 3;
bgpOverlayGroup.add(innerRing);
bgpCollectorRadarGroup.add(innerRing);
const overlayItems = [halo, pulseHalo, innerRing];
const anchorLatitude = marker.userData.displayLatitude ?? marker.userData.latitude;
@@ -1723,8 +1829,8 @@ export function showBGPCollectorCoverageOverlay(marker, earth, options = {}) {
const startBearing = (sectorRotation - sectorHalfWidth) * (180 / Math.PI);
const endBearing = (sectorRotation + sectorHalfWidth) * (180 / Math.PI);
const coverageColor = marker.userData.baseColor || BGP_CONFIG.collectorColor;
const boundaryAltitude = CONFIG.earthRadius + BGP_CONFIG.collectorAltitudeOffset + 0.44;
const fillAltitude = CONFIG.earthRadius + BGP_CONFIG.collectorAltitudeOffset + 0.4;
const boundaryAltitude = CONFIG.earthRadius + BGP_COLLECTOR_ALTITUDE_OFFSET + 0.44;
const fillAltitude = CONFIG.earthRadius + BGP_COLLECTOR_ALTITUDE_OFFSET + 0.4;
const leftBoundaryPoints = createRadialBoundaryPoints(
anchorLatitude,
anchorLongitude,
@@ -1765,7 +1871,7 @@ export function showBGPCollectorCoverageOverlay(marker, earth, options = {}) {
0.12,
);
sectorFill.renderOrder = 2;
bgpOverlayGroup.add(sectorFill);
bgpCollectorRadarGroup.add(sectorFill);
overlayItems.push(sectorFill);
const outerArc = createCoverageBoundaryLine(
@@ -1774,7 +1880,7 @@ export function showBGPCollectorCoverageOverlay(marker, earth, options = {}) {
0.9,
);
outerArc.renderOrder = 3;
bgpOverlayGroup.add(outerArc);
bgpCollectorRadarGroup.add(outerArc);
overlayItems.push(outerArc);
const leftBoundary = createCoverageBoundaryLine(
@@ -1783,7 +1889,7 @@ export function showBGPCollectorCoverageOverlay(marker, earth, options = {}) {
0.76,
);
leftBoundary.renderOrder = 3;
bgpOverlayGroup.add(leftBoundary);
bgpCollectorRadarGroup.add(leftBoundary);
overlayItems.push(leftBoundary);
const rightBoundary = createCoverageBoundaryLine(
@@ -1792,7 +1898,7 @@ export function showBGPCollectorCoverageOverlay(marker, earth, options = {}) {
0.76,
);
rightBoundary.renderOrder = 3;
bgpOverlayGroup.add(rightBoundary);
bgpCollectorRadarGroup.add(rightBoundary);
overlayItems.push(rightBoundary);
activeEventOverlay = overlayItems;
@@ -1808,7 +1914,8 @@ export function showBGPCollectorCoverageOverlay(marker, earth, options = {}) {
export function clearBGPEventOverlay() {
activeEventOverlay = null;
activeCollectorOverlayContext = null;
clearGroup(bgpOverlayGroup);
clearGroup(bgpEventOverlayGroup);
clearGroup(bgpCollectorRadarGroup);
}
function updateCollectorOverlayScan(lockedObjectType, lockedObject) {

View File

@@ -20,80 +20,110 @@ export let lockedCable = null;
let cableIdMap = new Map();
let cableStates = new Map();
let cablesVisible = true;
let landingPointTexture = null;
const _lpEarthWorldPos = new THREE.Vector3();
const _lpWorldPos = new THREE.Vector3();
const _lpCameraRel = new THREE.Vector3();
const _lpPointRel = new THREE.Vector3();
const _lpCameraToPoint = new THREE.Vector3();
const LANDING_POINT_SPRITE_HEIGHT = 3;
const LANDING_POINT_SPRITE_ASPECT = 1;
const LANDING_POINT_SIZE_REFERENCE_FOV = 75;
const LANDING_POINT_SIZE_SCALE_MIN = 0.16;
const LANDING_POINT_SIZE_SCALE_MAX = 3;
const LANDING_POINT_ATLAS_CELL_SIZE = 128;
let landingPointTexture = null;
function createLandingPointTexture() {
const size = CABLE_CONFIG.landingPoint.textureSize;
const canvas = document.createElement("canvas");
canvas.width = size;
canvas.height = size;
const ctx = canvas.getContext("2d");
const iconPath = new Path2D(
[
"M400 704",
"C386 704 375 697 367 684",
"L173 378",
"C117 290 144 173 229 111",
"C278 75 337 57 400 57",
"C463 57 522 75 571 111",
"C656 173 683 290 627 378",
"L433 684",
"C425 697 414 704 400 704",
"Z",
].join(" "),
function getLandingPointPulse() {
return (
Math.sin(Date.now() * CABLE_CONFIG.landingPointVisual.pulseSpeed) + 1
) * 0.5;
}
function getLandingPointDimColor() {
const dimColor = CABLE_CONFIG.landingPointVisual.dimmed.colorRGB;
const brightness = CABLE_CONFIG.landingPointVisual.dimBrightness;
const color = new THREE.Color(
(dimColor.r * brightness) / 255,
(dimColor.g * brightness) / 255,
(dimColor.b * brightness) / 255,
);
return `#${color.getHexString()}`;
}
ctx.clearRect(0, 0, size, size);
ctx.save();
ctx.translate(size * 0.12, size * 0.02);
ctx.scale(size / 1000, size / 1000);
function createLandingPointBallTexture() {
const canvas = document.createElement("canvas");
canvas.width = LANDING_POINT_ATLAS_CELL_SIZE;
canvas.height = LANDING_POINT_ATLAS_CELL_SIZE;
const context = canvas.getContext("2d");
const center = LANDING_POINT_ATLAS_CELL_SIZE / 2;
const radius = 46;
ctx.fillStyle = "#ffffff";
ctx.fill(iconPath);
context.clearRect(0, 0, canvas.width, canvas.height);
ctx.globalCompositeOperation = "destination-out";
ctx.beginPath();
ctx.arc(400, 320, 86, 0, Math.PI * 2);
ctx.fill();
ctx.restore();
const shadow = context.createRadialGradient(
center - 16,
center - 18,
8,
center,
center,
radius,
);
shadow.addColorStop(0, "rgba(255, 255, 255, 1)");
shadow.addColorStop(0.48, "rgba(238, 238, 238, 0.98)");
shadow.addColorStop(0.82, "rgba(178, 178, 178, 0.94)");
shadow.addColorStop(1, "rgba(92, 92, 92, 0.88)");
context.beginPath();
context.arc(center, center, radius, 0, Math.PI * 2);
context.fillStyle = shadow;
context.fill();
context.beginPath();
context.ellipse(center - 14, center - 18, 14, 9, -0.45, 0, Math.PI * 2);
context.fillStyle = "rgba(255, 255, 255, 0.38)";
context.fill();
context.beginPath();
context.arc(center, center, radius - 1, 0, Math.PI * 2);
context.strokeStyle = "rgba(255, 255, 255, 0.24)";
context.lineWidth = 2;
context.stroke();
const texture = new THREE.CanvasTexture(canvas);
texture.colorSpace = THREE.SRGBColorSpace;
texture.generateMipmaps = false;
texture.minFilter = THREE.LinearFilter;
texture.magFilter = THREE.LinearFilter;
texture.needsUpdate = true;
return texture;
}
function getLandingPointTexture() {
if (!landingPointTexture) {
landingPointTexture = createLandingPointTexture();
}
async function getLandingPointTexture() {
if (landingPointTexture) return landingPointTexture;
landingPointTexture = createLandingPointBallTexture();
return landingPointTexture;
}
function clamp(value, min, max) {
return Math.min(max, Math.max(min, value));
}
function getLandingPointDistanceScale(point, camera) {
if (
!point ||
!camera ||
CABLE_CONFIG.landingPointSizeStabilization?.enabled === false
) return 1;
function getLandingPointDistanceScale(camera) {
if (!camera) return 1;
return getSurfaceMarkerCameraScale(camera, {
altitudeOffset: CABLE_CONFIG.landingPoint.altitudeOffset,
referenceFov: CABLE_CONFIG.landingPointSizeStabilization?.referenceFov || 75,
min: CABLE_CONFIG.landingPointSizeStabilization?.min ?? 0.12,
max: CABLE_CONFIG.landingPointSizeStabilization?.max ?? 3.0,
referenceFov: LANDING_POINT_SIZE_REFERENCE_FOV,
min: LANDING_POINT_SIZE_SCALE_MIN,
max: LANDING_POINT_SIZE_SCALE_MAX,
});
}
function setLandingPointScale(point, camera = null) {
const height = LANDING_POINT_SPRITE_HEIGHT * getLandingPointDistanceScale(camera);
point.scale.set(height * LANDING_POINT_SPRITE_ASPECT, height, 1);
}
function setLandingPointMaterialState(point, { color, opacity }) {
point.material.color.set(color);
point.material.opacity = opacity;
}
function disposeMaterial(material) {
if (!material) return;
@@ -122,22 +152,6 @@ function disposeObject(object, parent) {
}
}
function setLandingPointMaterialState(point, { color, opacity, emissive, emissiveIntensity }) {
point.material.color.set(color);
point.material.opacity = opacity;
if (point.material.emissive && emissive !== undefined) {
point.material.emissive.setHex(emissive);
}
if ("emissiveIntensity" in point.material && emissiveIntensity !== undefined) {
point.material.emissiveIntensity = emissiveIntensity;
}
}
function setLandingPointScale(point, heightScale) {
const aspect = CABLE_CONFIG.landingPoint.iconAspectRatio;
point.scale.set(heightScale * aspect, heightScale, 1);
}
function getCableColor(properties) {
if (properties.color) {
if (
@@ -426,7 +440,7 @@ export async function loadLandingPoints(scene, earthObj, options = {}) {
clearLandingPoints(earthObj);
let validCount = 0;
const markerTexture = await getLandingPointTexture();
for (const feature of data.features) {
if (!feature.geometry || !feature.geometry.coordinates) continue;
@@ -460,20 +474,18 @@ export async function loadLandingPoints(scene, earthObj, options = {}) {
const marker = new THREE.Sprite(
new THREE.SpriteMaterial({
map: getLandingPointTexture(),
map: markerTexture,
color: CABLE_CONFIG.landingPoint.color,
transparent: true,
opacity: CABLE_CONFIG.landingPoint.opacity,
depthTest: false,
depthWrite: false,
alphaTest: 0.01,
}),
);
marker.material.userData.sharedMap = true;
marker.renderOrder = CABLE_CONFIG.landingPoint.renderOrder;
marker.center.set(
CABLE_CONFIG.landingPoint.anchorX,
CABLE_CONFIG.landingPoint.anchorY,
);
marker.center.set(0.5, 0.5);
marker.position.copy(position);
marker.userData = {
type: "landingPoint",
@@ -481,15 +493,18 @@ export async function loadLandingPoints(scene, earthObj, options = {}) {
cableNames: properties.cable_names || [],
country: properties.country || "未知国家",
status: properties.status || "Unknown",
baseScale: CABLE_CONFIG.landingPoint.baseScale,
latitude: lat,
longitude: lon,
landing_visual_state: "normal",
};
setLandingPointScale(marker, CABLE_CONFIG.landingPoint.baseScale);
setLandingPointScale(marker);
earthObj.add(marker);
landingPoints.push(marker);
validCount++;
}
const validCount = landingPoints.length;
setEarthStatValue("landing-point-count", `${validCount}`);
if (!silent) {
@@ -619,10 +634,8 @@ function isFacingCamera(lp, camera) {
const distance = Math.sqrt(distanceSq);
_lpCameraToPoint.multiplyScalar(1 / distance);
// The pin sprite is rendered without depth testing so its full shape does
// not get sliced by the globe. Instead, hide it when the camera-to-anchor
// segment is occluded by a slightly inflated globe, matching the behavior of
// the BGP and compute-center markers near the limb.
// Sprite rendering keeps the full pin visible. Hide it when the anchor point
// is occluded by the globe so back-side landing points do not bleed through.
const occlusionRadius =
CONFIG.earthRadius + CABLE_CONFIG.landingPoint.altitudeOffset * 0.45;
const cameraProjection = _lpCameraRel.dot(_lpCameraToPoint);
@@ -638,75 +651,52 @@ function isFacingCamera(lp, camera) {
}
export function applyLandingPointVisualState(lockedCableName, dimAll = false, camera = null) {
const pulse =
(Math.sin(Date.now() * CABLE_CONFIG.landingPointVisual.pulseSpeed) + 1) * 0.5;
const brightness = CABLE_CONFIG.landingPointVisual.dimBrightness;
const relatedNames = Array.isArray(lockedCableName)
? lockedCableName.filter(Boolean)
: lockedCableName
? [lockedCableName]
: [];
landingPoints.forEach((lp) => {
lp.visible = cablesVisible && (camera ? isFacingCamera(lp, camera) : true);
const isVisible = cablesVisible && (camera ? isFacingCamera(lp, camera) : true);
const isRelated =
!dimAll &&
Array.isArray(lp.userData.cableNames) &&
lp.userData.cableNames.some((name) => relatedNames.includes(name));
lp.visible = isVisible;
setLandingPointScale(lp, camera);
if (isRelated) {
const pulse = getLandingPointPulse();
setLandingPointMaterialState(lp, {
color: 0xffd27a,
emissive: 0x7a4a00,
emissiveIntensity:
CABLE_CONFIG.landingPointVisual.related.emissiveIntensityBase +
0.2 +
pulse * (CABLE_CONFIG.landingPointVisual.related.emissiveIntensityPulse + 0.2),
opacity: Math.max(
0.92,
CABLE_CONFIG.landingPointVisual.related.opacityBase +
pulse * CABLE_CONFIG.landingPointVisual.related.opacityPulse,
),
});
const distanceScale = getLandingPointDistanceScale(lp, camera);
const baseScale = lp.userData?.baseScale || CABLE_CONFIG.landingPoint.baseScale;
setLandingPointScale(
lp,
(CABLE_CONFIG.landingPointVisual.related.scaleBase +
pulse * CABLE_CONFIG.landingPointVisual.related.scalePulse) *
baseScale *
distanceScale,
);
lp.userData.landing_visual_state = "related";
} else {
const dimColor = CABLE_CONFIG.landingPointVisual.dimmed.colorRGB;
const r = dimColor.r * brightness;
const g = dimColor.g * brightness;
const b = dimColor.b * brightness;
setLandingPointMaterialState(lp, {
color: new THREE.Color(r / 255, g / 255, b / 255),
emissive: CABLE_CONFIG.landingPointVisual.dimmed.emissive,
emissiveIntensity: CABLE_CONFIG.landingPointVisual.dimmed.emissiveIntensity,
color: getLandingPointDimColor(),
opacity: CABLE_CONFIG.landingPointVisual.dimmed.opacity,
});
const distanceScale = getLandingPointDistanceScale(lp, camera);
const baseScale = lp.userData?.baseScale || CABLE_CONFIG.landingPoint.baseScale;
setLandingPointScale(lp, baseScale * distanceScale);
lp.userData.landing_visual_state = isVisible ? "dimmed" : "hidden";
}
});
}
export function resetLandingPointVisualState(camera = null) {
landingPoints.forEach((lp) => {
lp.visible = cablesVisible && (camera ? isFacingCamera(lp, camera) : true);
const isVisible = cablesVisible && (camera ? isFacingCamera(lp, camera) : true);
lp.visible = isVisible;
setLandingPointScale(lp, camera);
setLandingPointMaterialState(lp, {
color: CABLE_CONFIG.landingPoint.color,
emissive: CABLE_CONFIG.landingPoint.emissive,
emissiveIntensity: CABLE_CONFIG.landingPoint.emissiveIntensity,
opacity: CABLE_CONFIG.landingPoint.opacity,
});
const distanceScale = getLandingPointDistanceScale(lp, camera);
const baseScale = lp.userData?.baseScale || CABLE_CONFIG.landingPoint.baseScale;
setLandingPointScale(lp, baseScale * distanceScale);
lp.userData.landing_visual_state = isVisible ? "normal" : "hidden";
});
}

View File

@@ -1,12 +1,15 @@
import * as THREE from "three";
import { COMPUTE_CENTER_CONFIG, PATHS } from "./constants.js";
import { createInteractableLayer } from "./interactable.js";
import { COMPUTE_CENTER_CONFIG, CONFIG, PATHS } from "./constants.js";
import { getSurfaceMarkerCameraScale, latLonToVector3 } from "./utils.js";
const computeCenterGroup = new THREE.Group();
const computeCenterMarkers = [];
const COMPUTE_CENTER_RENDER_ORDER = 4.5;
const textureCache = new Map();
const COMPUTE_CENTER_POINT_SIZE = 36;
const COMPUTE_CENTER_ICON_FIT_SIZE = 60;
const COMPUTE_CENTER_ATLAS_CELL_SIZE = 128;
const COMPUTE_CENTER_ICON_SOURCES = {
supercomputer: "/earth/assets/icons/compute-supercomputer.svg",
gpu_cluster: "/earth/assets/icons/compute-gpu-cluster.svg",
infrastructure: "/earth/assets/icons/compute-hdd-network.svg",
};
let showComputeCenters = true;
let supercomputerCount = 0;
let gpuClusterCount = 0;
@@ -69,71 +72,13 @@ function spreadComputeCenterPositions(markers) {
return markers;
}
function createMarkerTexture(siteType, isEstimated = false) {
const textureKey = `${siteType}:${isEstimated ? "estimated" : "precise"}`;
if (textureCache.has(textureKey)) {
return textureCache.get(textureKey);
}
const color =
COMPUTE_CENTER_CONFIG.colors[siteType] ||
COMPUTE_CENTER_CONFIG.colors.gpu_cluster;
const canvas = document.createElement("canvas");
canvas.width = 128;
canvas.height = 128;
const context = canvas.getContext("2d");
const centerX = 64;
const centerY = 64;
const baseFill = color;
function fillPath(draw, options = {}) {
const { fillStyle = color } = options;
context.save();
context.fillStyle = fillStyle;
context.beginPath();
draw();
context.fill();
context.restore();
}
context.clearRect(0, 0, 128, 128);
if (siteType === "supercomputer") {
fillPath(() => {
context.roundRect(40, 42, 48, 30, 7);
}, {
fillStyle: baseFill,
});
fillPath(() => {
context.roundRect(58, 74, 12, 8, 3);
context.roundRect(50, 84, 28, 5, 2.5);
}, {
fillStyle: baseFill,
});
} else {
fillPath(() => {
context.ellipse(centerX, 46, 18, 8, 0, 0, Math.PI * 2);
context.rect(46, 46, 36, 28);
context.ellipse(centerX, 74, 18, 8, 0, 0, Math.PI);
}, {
fillStyle: baseFill,
});
fillPath(() => {
context.ellipse(centerX, 58, 12, 4.5, 0, 0, Math.PI * 2);
context.rect(52, 58, 24, 6);
context.ellipse(centerX, 64, 12, 4.5, 0, 0, Math.PI);
}, {
fillStyle: baseFill,
});
}
function drawComputeCenterEstimatedBadge(context, isEstimated = false) {
if (isEstimated) {
fillPath(() => {
context.arc(94, 36, 12, 0, Math.PI * 2);
}, {
fillStyle: "rgba(15,23,42,0.92)",
});
context.save();
context.fillStyle = "rgba(15,23,42,0.92)";
context.beginPath();
context.arc(94, 36, 12, 0, Math.PI * 2);
context.fill();
context.fillStyle = "rgba(255,255,255,0.98)";
context.font = "bold 18px sans-serif";
context.textAlign = "center";
@@ -141,76 +86,74 @@ function createMarkerTexture(siteType, isEstimated = false) {
context.fillText("?", 94, 36);
context.restore();
}
const texture = new THREE.CanvasTexture(canvas);
texture.needsUpdate = true;
textureCache.set(textureKey, texture);
return texture;
}
function normalizeSiteType(siteType) {
return siteType === "supercomputer" ? "supercomputer" : "gpu_cluster";
}
function getBaseScale(siteType) {
return siteType === "supercomputer"
? COMPUTE_CENTER_CONFIG.marker.supercomputerScale
: COMPUTE_CENTER_CONFIG.marker.gpuClusterScale;
}
function getDistanceScale(marker, camera) {
if (!marker || !camera || COMPUTE_CENTER_CONFIG.sizeStabilization.enabled === false) {
return 1;
}
return getSurfaceMarkerCameraScale(camera, {
altitudeOffset: COMPUTE_CENTER_CONFIG.altitudeOffset,
referenceFov: 75,
min: COMPUTE_CENTER_CONFIG.sizeStabilization.min,
max: COMPUTE_CENTER_CONFIG.sizeStabilization.max,
});
}
function clearGroup(group) {
for (let index = group.children.length - 1; index >= 0; index -= 1) {
const child = group.children[index];
child.material?.dispose?.();
group.remove(child);
}
}
function createComputeCenterMarker(markerData) {
const siteType = markerData.site_type;
const material = new THREE.SpriteMaterial({
map: createMarkerTexture(siteType, Boolean(markerData.is_estimated)),
transparent: true,
depthWrite: false,
opacity: COMPUTE_CENTER_CONFIG.marker.baseOpacity,
});
const marker = new THREE.Sprite(material);
const baseScale = getBaseScale(siteType);
marker.position.copy(
latLonToVector3(
markerData.displayLatitude,
markerData.displayLongitude,
CONFIG.earthRadius + COMPUTE_CENTER_CONFIG.altitudeOffset,
),
);
marker.scale.setScalar(baseScale);
marker.renderOrder = COMPUTE_CENTER_RENDER_ORDER;
marker.visible = showComputeCenters;
marker.userData = {
...markerData,
site_type: siteType,
type: "compute_center",
baseScale,
state: "normal",
const computeCenterIconLayer = createInteractableLayer({
id: "computeCenters",
objectType: "compute_center",
renderOrder: COMPUTE_CENTER_RENDER_ORDER,
altitudeOffset: COMPUTE_CENTER_CONFIG.altitudeOffset,
pointSize: COMPUTE_CENTER_POINT_SIZE,
atlasCellSize: COMPUTE_CENTER_ATLAS_CELL_SIZE,
colors: {
byKind: COMPUTE_CENTER_CONFIG.colors,
normal: COMPUTE_CENTER_CONFIG.colors.gpu_cluster,
},
opacity: {
normal: COMPUTE_CENTER_CONFIG.marker.baseOpacity,
dimmed: COMPUTE_CENTER_CONFIG.marker.dimmedOpacity,
hover: 0.98,
locked: 1,
},
stateScale: {
hover: COMPUTE_CENTER_CONFIG.marker.hoverScale,
locked: COMPUTE_CENTER_CONFIG.marker.lockedScale,
dimmed: COMPUTE_CENTER_CONFIG.marker.dimmedScale,
},
pulse: {
enabled: true,
speed: COMPUTE_CENTER_CONFIG.marker.pulseSpeed,
amplitude: COMPUTE_CENTER_CONFIG.marker.pulseAmplitude,
},
icon: {
coordinates: "canvas",
colorable: false,
fitSize: COMPUTE_CENTER_ICON_FIT_SIZE,
glowBlur: 16,
getSource({ marker, item }) {
const siteType =
marker?.userData?.site_type || item?.site_type || "gpu_cluster";
return (
COMPUTE_CENTER_ICON_SOURCES[siteType] ||
COMPUTE_CENTER_ICON_SOURCES.infrastructure
);
},
afterDraw(context, { marker, item }) {
drawComputeCenterEstimatedBadge(
context,
Boolean(marker?.userData?.is_estimated ?? item?.is_estimated),
);
},
},
getPosition: (item) => ({
latitude: item.displayLatitude,
longitude: item.displayLongitude,
}),
getKind: (item) => item.site_type || "gpu_cluster",
getBucketKey: (marker) =>
[
marker.userData?.site_type || "gpu_cluster",
marker.userData?.is_estimated ? "estimated" : "precise",
].join(":"),
getUserData: (item) => ({
...item,
pulseOffset: Math.random() * Math.PI * 2,
};
computeCenterGroup.add(marker);
computeCenterMarkers.push(marker);
return marker;
}
}),
});
export function formatComputeCenterTypeLabel(siteType) {
return siteType === "supercomputer" ? "超算中心" : "GPU 集群";
@@ -252,11 +195,11 @@ export function getComputeCenterLegendItems() {
}
export function getComputeCenterMarkers() {
return computeCenterMarkers;
return computeCenterIconLayer.getMarkers();
}
export function getComputeCenterCount() {
return computeCenterMarkers.length;
return computeCenterIconLayer.getCount();
}
export function getComputeCenterSupercomputerCount() {
@@ -268,35 +211,27 @@ export function getComputeCenterGPUClusterCount() {
}
export function getComputeCenterStatusSummary() {
if (computeCenterMarkers.length === 0) return "暂无算力中心数据";
if (getComputeCenterCount() === 0) return "暂无算力中心数据";
return `${supercomputerCount} 台超算 / ${gpuClusterCount} 个 GPU 集群`;
}
export function setComputeCenterMarkerState(marker, state = "normal") {
if (!marker || marker.userData?.type !== "compute_center") return;
marker.userData.state = state;
computeCenterIconLayer.setMarkerState(marker, state);
}
export function clearComputeCenterSelection() {
computeCenterMarkers.forEach((marker) => setComputeCenterMarkerState(marker, "normal"));
getComputeCenterMarkers().forEach((marker) => setComputeCenterMarkerState(marker, "normal"));
}
export function clearComputeCenterData(earth) {
computeCenterMarkers.length = 0;
supercomputerCount = 0;
gpuClusterCount = 0;
clearGroup(computeCenterGroup);
if (earth && computeCenterGroup.parent === earth) {
earth.remove(computeCenterGroup);
}
computeCenterIconLayer.clearData(earth);
}
export function toggleComputeCenters(show) {
showComputeCenters = Boolean(show);
computeCenterGroup.visible = showComputeCenters;
computeCenterMarkers.forEach((marker) => {
marker.visible = showComputeCenters;
});
computeCenterIconLayer.setVisible(showComputeCenters);
}
export function getShowComputeCenters() {
@@ -313,64 +248,37 @@ export async function loadComputeCenters(_scene, earth) {
clearComputeCenterData(earth);
spreadComputeCenterPositions(
const markerData = spreadComputeCenterPositions(
features
.map((feature) => buildComputeCenterMarkerData(feature))
.filter(Boolean),
)
.slice(0, COMPUTE_CENTER_CONFIG.maxRenderedMarkers)
.forEach((markerData) => {
const marker = createComputeCenterMarker(markerData);
if (!marker) return;
if (marker.userData.site_type === "supercomputer") {
supercomputerCount += 1;
} else {
gpuClusterCount += 1;
}
});
.slice(0, COMPUTE_CENTER_CONFIG.maxRenderedMarkers);
if (earth && !computeCenterGroup.parent) {
earth.add(computeCenterGroup);
}
computeCenterGroup.visible = showComputeCenters;
markerData.forEach((item) => {
if (item.site_type === "supercomputer") {
supercomputerCount += 1;
} else {
gpuClusterCount += 1;
}
});
await computeCenterIconLayer.preloadAssets(markerData);
computeCenterIconLayer.setData(markerData);
computeCenterIconLayer.attach(earth);
computeCenterIconLayer.setVisible(showComputeCenters);
return {
totalCount: computeCenterMarkers.length,
totalCount: getComputeCenterCount(),
supercomputerCount,
gpuClusterCount,
summary: getComputeCenterStatusSummary(),
};
}
export function updateComputeCenterVisualState(lockedObjectType, lockedObject, camera) {
const hasFocus = lockedObjectType === "compute_center" && lockedObject;
const now = Date.now();
computeCenterMarkers.forEach((marker) => {
const isLocked = lockedObjectType === "compute_center" && lockedObject === marker;
const state = marker.userData?.state || "normal";
const pulse =
1 +
COMPUTE_CENTER_CONFIG.marker.pulseAmplitude *
Math.sin(now * COMPUTE_CENTER_CONFIG.marker.pulseSpeed + marker.userData.pulseOffset);
let opacity = COMPUTE_CENTER_CONFIG.marker.baseOpacity;
let scaleMultiplier = 1;
if (isLocked) {
opacity = 1;
scaleMultiplier = COMPUTE_CENTER_CONFIG.marker.lockedScale * pulse;
} else if (state === "hover") {
opacity = 0.98;
scaleMultiplier = COMPUTE_CENTER_CONFIG.marker.hoverScale;
} else if (hasFocus) {
opacity = COMPUTE_CENTER_CONFIG.marker.dimmedOpacity;
scaleMultiplier = COMPUTE_CENTER_CONFIG.marker.dimmedScale;
}
const distanceScale = getDistanceScale(marker, camera);
marker.material.opacity = showComputeCenters ? opacity : 0;
marker.scale.setScalar(marker.userData.baseScale * scaleMultiplier * distanceScale);
marker.visible = showComputeCenters;
});
export function getComputeCenterPointerIntersections(options) {
return computeCenterIconLayer.getPointerIntersections(options);
}
export function updateComputeCenterVisualState(lockedObjectType, lockedObject, camera) {
computeCenterIconLayer.updateVisualState(lockedObjectType, lockedObject, camera);
}

View File

@@ -209,8 +209,8 @@ export const PATHS = {
};
export const VESSEL_CONFIG = {
altitudeOffset: 0.56,
maxRenderedMarkers: 5000,
altitudeOffset: 0.2,
maxRenderedMarkers: 0,
marker: {
baseScale: 7.5,
baseOpacity: 0.88,
@@ -232,7 +232,7 @@ export const VESSEL_CONFIG = {
max: 2.4,
},
track: {
altitudeOffset: 0.7,
altitudeOffset: 0.2,
color: 0x7dd3fc,
opacity: 0.82,
},
@@ -293,39 +293,20 @@ export const CABLE_CONFIG = {
renderOrder: 1,
},
landingPoint: {
altitudeOffset: 0.48,
textureSize: 256,
iconAspectRatio: 0.82,
anchorX: 0.52,
anchorY: 0.276,
baseScale: 12,
altitudeOffset: 0.2,
color: 0xffaa00,
emissive: 0x442200,
emissiveIntensity: 0.5,
opacity: 1.0,
renderOrder: 4.5,
},
landingPointSizeStabilization: {
enabled: true,
referenceFov: 75,
min: 0.12,
max: 3.0,
renderOrder: 1,
},
landingPointVisual: {
pulseSpeed: 0.003,
dimBrightness: 0.62,
related: {
emissiveIntensityBase: 0.5,
emissiveIntensityPulse: 0.5,
opacityBase: 0.8,
opacityPulse: 0.2,
scaleBase: 1.2,
scalePulse: 0.3,
},
dimmed: {
colorRGB: { r: 180, g: 116, b: 28 },
emissive: 0x3a2200,
emissiveIntensity: 0.18,
opacity: 0.78,
},
},
@@ -364,8 +345,8 @@ export const SATELLITE_CONFIG = {
export const BGP_CONFIG = {
defaultFetchLimit: 200,
maxRenderedMarkers: 200,
altitudeOffset: 2.1,
collectorAltitudeOffset: 1.6,
altitudeOffset: 0.48,
collectorAltitudeOffset: 0.2,
marker: {
eventBaseScale: 6.2,
collectorBaseScale: 7.4,

Some files were not shown because too many files have changed in this diff Show More