Compare commits

...

10 Commits

Author SHA1 Message Date
rayd1o
65e6a96c0d release: bump version to 0.65.0
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
release / images (push) Has been cancelled
ci / delivery (push) Has been cancelled
ci / backend (pull_request) Has been cancelled
ci / frontend (pull_request) Has been cancelled
ci / delivery (pull_request) Has been cancelled
2026-05-21 05:41:49 +08:00
linkong
37e92e7572 release: bump version to 0.64.0
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-05-21 03:46:02 +08:00
linkong
a37d4b6289 fix: update default linkong password
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-05-21 02:20:45 +08:00
linkong
69789d7505 release: bump version to 0.63.1
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-05-21 02:14:11 +08:00
linkong
4f124121e7 fix: include admin next utils module
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-05-21 02:07:39 +08:00
linkong
085bdf9a80 release: bump version to 0.63.0
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-05-21 01:46:43 +08:00
linkong
fbca381512 release: bump version to 0.62.0
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-05-21 01:37:32 +08:00
rayd1o
5c65ee24d6 release: bump version to 0.61.0
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
release / images (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-05-18 02:37:19 +08:00
rayd1o
81970a1d05 release: bump version to 0.60.0
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-05-17 02:50:42 +08:00
rayd1o
9b913a3b83 release: bump version to 0.59.0
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-05-16 05:02:05 +08:00
208 changed files with 30923 additions and 6993 deletions

View File

@@ -73,6 +73,7 @@ Style:
- Use fenced code blocks with language tags.
- Prefer tables for comparisons or parameter lists.
- Keep snippets concise and relevant.
- For UI labels, chart labels, feature names, datasource names, and other terms that may become mixed Chinese/English copy, check `docs/technical/{zh,en}/naming-glossary.md` and use the documented display name. If a confusing term is missing, update the glossary in both languages as part of the docs change.
### Step 4 — Verify

View File

@@ -52,6 +52,7 @@ rg -n "class |def |function |export |router|@router|interface |type " <path>
- Keep code snippets short and directly relevant.
- List related files only when they help future maintainers navigate.
- Use the repositorys existing language, heading style, and naming conventions.
- For UI labels, chart labels, feature names, datasource names, and other terms that may become mixed Chinese/English copy, check `docs/technical/{zh,en}/naming-glossary.md` and use the documented display name. If a confusing term is missing, update the glossary in both languages as part of the docs change.
4. Verify:

1
.gitignore vendored
View File

@@ -8,6 +8,7 @@
.env
.env.local
.env.*.local
config/earth-boundary-sources.local.json
*.pem
*.key
*.crt

213
README.md
View File

@@ -8,68 +8,54 @@
## 系统架构
当前仓库的核心形态是“Web Earth 可视化 + React 运维台 + FastAPI 数据与 AI 编排后端 + 独立模型适配层”。物理大屏与 UE 客户端仍是长期方向,但不再作为本地开发和当前发布的必需运行单元。
```
┌─────────────────────────────────────────────────────────────────────────
物理大屏展示层
│ ┌─────────────────────────────────────────────────────────────────┐
│ │ 偏振片3D大屏 (2m×3m, 4K, 120Hz, 眼镜式)
│ │ ┌─────────────────────────────────────────────────────────┐
│ │ │ 虚幻引擎 UE5 客户端
│ │ │ ├── 3D地球渲染 (Cesium for UE)
│ │ ├── 算力点可视化 (GPU集群、智算中心) │ │
│ │ │ ├── 连接弧线 (光缆、路由、数据流向) │ │ │
│ │ ├── 粒子效果 (数据流动、告警提示) │ │ │
│ └── 自动巡航相机 + 交互控制 │ │ │
│ │ └─────────────────────────────────────────────────────────┘ │ │
└─────────────────────────────────────────────────────────────────┘
─────────────────────────────────────────────────────────────────────────┘
│ WebSocket (实时推送)
│ 120Hz 心跳 / 数据帧同步
┌─────────────────────────────────────────────────────────────────────────┐
数据中台服务层 (FastAPI)
─────────────────────────────────────────────────────────────────┐
│ │ API Gateway (Redis 限流) │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │
┌───────────────────┬────────────────────────────────────────────┐
│ 数据采集服务核心业务服务 运维管理服务
│ ┌─────────────┐ │ ┌─────────────────┐ │ ┌─────────────┐ │
│ │ 调度中心 │ │ │ WebSocket 服务 │ │ │ 用户管理 │ │
│ │ │ (Celery) │ │ │ (FastAPI) │ │ │ (JWT Auth) │ │ │
└─────────────┘ └─────────────────┘ │ └─────────────┘ │ │
┌─────────────┐ │ ┌─────────────────┐ │ ┌─────────────┐ │ │
│ 采集器池 │ │ │ 数据查询 API │ │ │ 数据源配置 │ │ │
│ │ │ (10+源) │ │ │ (REST) │ │ │ 监控告警 │ │ │
│ └─────────────┘ │ └─────────────────┘ │ └─────────────┘ │
│ │ ┌─────────────┐ │ ┌─────────────────┐ │ ┌─────────────┐ │ │
│ │ │ 消息队列 │ │ │ 态势分析引擎 │ │ │ 系统配置 │ │ │
│ │ │ (Kafka) │ │ │ (计算/聚合) │ │ │ 日志审计 │ │ │
│ │ └─────────────┘ │ └─────────────────┘ │ └─────────────┘ │ │
│ └───────────────────┴──────────────────────────┴──────────────────┘ │
└─────────────────────────────────────────────────────────────────────────┘
│ 内部 API 调用
┌─────────────────────────────────────────────────────────────────────────┐
│ Web管理端 (React Admin) │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ 登录页 │ 仪表盘 │ 用户管理 │ 数据源配置 │ 任务监控 │ 系统配置 │ │
│ └─────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────┘
│ PostgreSQL / Redis
┌─────────────────────────────────────────────────────────────────────────┐
│ 数据存储层 │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ PostgreSQL │ │ TimescaleDB │ │ Redis │ │ MinIO │ │
│ │ (用户/配置) │ │ (时序数据) │ │ (缓存/会话) │ │ (文件存储) │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │
└─────────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────┐
浏览器展示与运维层
│ ┌──────────────────────────────┐ ┌──────────────────────────────┐ │
│ │ Web Earth │ │ React 运维台 │ │
│ │ frontend/public/earth frontend/src │
│ │ Three.js 地球 / HUD / 新闻 │ 数据源 / 告警 / AI 设置
│ │ 国界精度 / 品牌内容配置 │ 提示词配置 / 用户与系统配置 │ │
└──────────────────────────────┘ └──────────────────────────────┘
└─────────────────────────────────────────────────────────────────────┘
│ REST / WebSocket
─────────────────────────────────────────────────────────────────────┐
FastAPI 业务与编排后端
│ ┌────────────────────┐ ┌────────────────────┐ ┌─────────────────┐ │
│ │ 数据 API 与认证 │ │ Earth 新闻增强 │ │ 告警与态势简报 │ │
│ │ JWT / 权限 / 审计 │ │ 位置推断 / 本地化 │ │ BGP / 告警研判 │ │
│ └────────────────────┘ └────────────────────┘ └─────────────────┘ │
┌────────────────────┐ ┌────────────────────┐ ┌─────────────────┐ │
│ │ 系统运行配置 │ │ 默认提示词注册表 │ │ 未来 Agent Runtime│ │
│ system_settings │ │ 代码发布 + DB 覆盖 │ │ 工具/证据/工作流
────────────────────┘ └────────────────────┘ └─────────────────
└─────────────────────────────────────────────────────────────────────┘
│ SQLAlchemy / Redis Stream │ 纯净 LLM 调用
┌──────────────────────────────┐ ┌──────────────────────────────┐
PostgreSQL / Redis │ aiprovider
用户、配置、采集结果、新闻 │ │ provider + protocol adapter
Stream、缓存、运行状态 │ │ OpenAI / MiniMax / Ollama 等
└──────────────────────────────┘ └──────────────────────────────┘
│ 采集器 / 外部数据源
┌─────────────────────────────────────────────────────────────────────┐
RSS 新闻、BGP 观测、公开数据源、后续 WebSearch/OCR/语音识别等工具
─────────────────────────────────────────────────────────────────────┘
```
架构边界:
- `backend` 负责业务语义、证据收集、提示词选择、AI 任务编排、权限和数据落库。
- `aiprovider` 只负责把纯净模型请求适配到不同供应商或协议,不内置具体业务提示词。
- 默认提示词随代码发布并保存在 `backend/app/ai_tasks/default_prompts.json`,运维台可在数据库中保存覆盖值,重置时回到当前代码版本的默认提示词。
- Earth 新闻保留英文原文,中文展示结果存入 `localizations`,前端默认展示 `zh-CN``display_title``display_summary` 和中文地域/状态文案。
- Earth LLM 指令、语音识别、多角色态势研判属于后续 Agent Runtime 方向,计划见 [docs/plans/agents-earth-command-runtime-plan.md](/home/ray/dev/linkong/planet/docs/plans/agents-earth-command-runtime-plan.md)。
## 四大核心要素
| 层级 | 要素 | 描述 |
@@ -87,11 +73,10 @@
|------|------|------|
| FastAPI | 0.109+ | Web 框架 |
| SQLAlchemy | 2.0+ | ORM |
| Alembic | - | 数据库迁移 |
| Celery | 5.3+ | 任务队列 |
| Redis | 7.0+ | 缓存/消息 |
| Kafka | 3.0+ | 事件流 |
| uv | - | Python 依赖与命令运行 |
| Redis | 7.0+ | 缓存、Stream 与运行协调 |
| PyJWT | - | 认证 |
| APScheduler / 后台任务 | - | 采集、增强与运行时任务 |
### 前端 (React Admin)
@@ -102,6 +87,7 @@
| Axios | HTTP 客户端 |
| Socket.io-client | WebSocket 客户端 |
| ECharts | 统计图表 |
| Three.js | Earth 3D 地球渲染 |
| Bun | 前端包管理与脚本运行 |
前端工程统一使用 Bun
@@ -110,22 +96,16 @@
- 运行脚本使用 `bun run <script>`
- 不使用 `npm``pnpm``yarn`
### 虚幻引擎客户端
### 大屏与 3D 展示方向
| 组件 | 版本 | 用途 |
|------|------|------|
| Unreal Engine 5 | 5.3+ | 3D 渲染引擎 |
| Cesium for Unreal | 1.5+ | 地理可视化 |
| Niagara | - | 粒子系统 |
当前发布优先使用浏览器 Web Earth。UE5 / Cesium for Unreal / Niagara 可作为后续物理大屏方向接入,但不是本地开发闭环的必需组件。
### 数据库
| 组件 | 用途 |
|------|------|
| PostgreSQL 15+ | 关系数据 |
| TimescaleDB | 时序数据扩展 |
| Redis 7+ | 缓存/会话 |
| MinIO | S3 兼容存储 |
| Redis 7+ | 缓存、Stream、运行状态 |
### 部署
@@ -152,9 +132,9 @@
| P0 | Epoch AI | 每小时 |
| P0 | Hugging Face | 每 2 小时 |
| P0 | GitHub | 每 4 小时 |
| P0 每日 |
| P0 | 海底光缆 / IXP / 卫星等基础设施数据 | 每日或按源刷新 |
| P0 | PeeringDB | 每 2 小时 |
| P1 | Cloudflare Radar | | TeleGeography | 每小时 |
| P1 | Cloudflare Radar / TeleGeography | 每小时 |
| P1 | CAIDA BGPStream | 每 15 分钟 |
## 项目结构
@@ -166,20 +146,18 @@
│ │ ├── core/ # 核心配置
│ │ ├── models/ # 数据模型
│ │ ├── schemas/ # Pydantic 模型
│ │ ├── services/ # 业务逻辑
│ │ └── tasks/ # Celery 任务
│ │ ├── services/ # 业务逻辑与 AI 任务编排
│ │ └── ai_tasks/ # 默认提示词与 AI 任务定义
│ └── tests/
├── aiprovider/ # 独立模型供应商适配层
├── frontend/ # React 管理后台
│ ├── src/
│ │ ├── components/ # 组件
│ │ ├── pages/ # 页面
│ │ ├── services/ # API 服务
│ │ └── store/ # 状态管理
── tests/
├── unreal/ # UE5 大屏客户端
│ ├── Content/
│ ├── Source/
│ └── Plugins/
── public/earth/ # Web Earth 静态应用
│ └── tests/ # 前端测试
├── data/ # 数据文件
├── docs/ # 文档
├── scripts/ # 脚本
@@ -191,10 +169,11 @@
## 快速启动
```bash
# 新机器首次初始化
./scripts/bootstrap-dev.sh
# 会自动安装/检查 uv、bun同步 Python/前端依赖
# 新机器或空项目首次初始化
./planet.sh init
# 会自动安装/检查 uv、bun同步 Python/前端依赖
# 会在缺少时生成 backend/.env、aiprovider/.env、frontend/.env.local
# 会启动 PostgreSQL/Redis并创建表、默认数据源和本地默认用户
# 启动前后端服务
./planet.sh start
@@ -210,6 +189,9 @@
# 查看服务状态
./planet.sh health
# 删除容器、卷、镜像和本地编译状态,执行前需要输入 Y 确认
./planet.sh destroy
```
前端命令约定:
@@ -364,14 +346,23 @@ DATABASE_RETRY_INTERVAL=10 \
- `FRONTEND_HEALTH_CHECK_ATTEMPTS` / `FRONTEND_HEALTH_CHECK_INTERVAL`: 控制前端 HTTP 可访问检查等待次数与间隔,默认 `10` 次、`2`
- `AI_PROVIDER_HEALTH_CHECK_ATTEMPTS` / `AI_PROVIDER_HEALTH_CHECK_INTERVAL`: 控制 `aiprovider` HTTP 健康检查等待次数与间隔,默认 `10` 次、`2`
## AI 接口预留
## AI 与智能体接口
项目现在采用“两层”设计:
项目现在采用“三段式”边界:
- 主后端暴露稳定业务接口: `GET /api/v1/ai/provider/status``POST /api/v1/ai/situational-awareness/analyze`
- 独立 `aiprovider` 服务负责适配具体模型供应商
- `backend`: 暴露业务接口,负责选择任务提示词、组织证据、调用工具、保存 AI 设置和结果。
- `aiprovider`: 暴露模型网关接口,只负责 provider / protocol 适配,不写入 BGP、新闻、告警等业务提示词。
- 模型供应商: OpenAI 兼容、MiniMax、Anthropic、Ollama 或其他兼容网关。
这样前端和业务代码不直接依赖 OpenAI、本地模型网关或其他订阅服务后续切换部署方式只需要调整环境变量
这样前端和业务代码不直接依赖某个模型供应商,后续增加 Agent Runtime、Earth 一键 LLM 指令、语音识别或多角色态势研判时,也可以把业务工作流放在后端,而不是污染模型适配层
当前已落地的 AI 配置能力:
- 运维台 AI 设置可维护 provider、模型、协议、超时、token 等运行配置。
- 运维台 AI 设置中的“提示词”页可选择不同功能入口,手动覆盖提示词,并一键重置到默认值。
- 默认提示词随代码发布,位于 [backend/app/ai_tasks/default_prompts.json](/home/ray/dev/linkong/planet/backend/app/ai_tasks/default_prompts.json)。
- 覆盖值保存在数据库运行配置中,升级代码后可继续保留现场配置,也可重置到新版本默认提示词。
- 态势摘要、告警研判、新闻本地化等入口应使用各自任务提示词;调用 `aiprovider` 时只传递当前任务所需的 `prompt` / `system_prompt`
主后端建议配置:
@@ -384,39 +375,33 @@ AI_PROVIDER_TIMEOUT_SECONDS=60
`aiprovider` 服务建议配置:
```env
AI_PROVIDER=openai_compatible
AI_BASE_URL=https://api.openai.com/v1
AI_PROVIDER=minimax
AI_PROVIDER_API=anthropic-messages
AI_BASE_URL=https://api.minimaxi.com/anthropic
AI_API_KEY=your_api_key
AI_MODEL=gpt-4o-mini
AI_MODEL=MiniMax-M2.7
AI_TIMEOUT_SECONDS=60
AI_PROVIDER_SERVICE_TOKEN=change_me
```
OpenAI 兼容场景推荐使用:
推荐映射关系:
- `AI_PROVIDER=openai_compatible`
- `vLLM` / `LM Studio` / `One API`: `AI_PROVIDER=openai` + `AI_PROVIDER_API=openai-completions`
- `MiniMax`: `AI_PROVIDER=minimax` + `AI_PROVIDER_API=anthropic-messages`
- Claude 兼容网关: `AI_PROVIDER=anthropic` + `AI_PROVIDER_API=anthropic-messages`
- `Ollama`: `AI_PROVIDER=ollama` + `AI_PROVIDER_API=ollama-generate`
Claude 兼容场景推荐使用:
- `AI_PROVIDER=anthropic`
- `AI_PROVIDER=anthropic_compatible`
- `AI_PROVIDER=claude_compatible`
Ollama 原生场景推荐使用:
- `AI_PROVIDER=ollama`
比如 MiniMax 或其他 Claude 兼容网关,可以这样配置:
比如 MiniMax 可以这样配置:
```env
AI_PROVIDER=claude_compatible
AI_BASE_URL=https://your-claude-compatible-endpoint.example.com
AI_PROVIDER=minimax
AI_PROVIDER_API=anthropic-messages
AI_BASE_URL=https://api.minimaxi.com/anthropic
AI_API_KEY=your_api_key
AI_MODEL=your-claude-compatible-model
AI_MODEL=MiniMax-M2.7
AI_TIMEOUT_SECONDS=60
AI_MAX_TOKENS=1200
AI_ANTHROPIC_VERSION=2023-06-01
AI_PROVIDER_SERVICE_TOKEN=change_me
```
如果你要本地直接起模型适配层,项目里已经补了模板:
@@ -424,12 +409,6 @@ AI_PROVIDER_SERVICE_TOKEN=change_me
- [aiprovider/.env.example](/home/ray/dev/linkong/planet/aiprovider/.env.example)
- [docker-compose.local-model.yml](/home/ray/dev/linkong/planet/docker-compose.local-model.yml)
推荐映射关系:
- `vLLM` / `LM Studio` / `One API`: `AI_PROVIDER=openai_compatible`
- `MiniMax` / Claude 兼容网关: `AI_PROVIDER=claude_compatible`
- `Ollama`: `AI_PROVIDER=ollama`
运行与调用补充:
- `./planet.sh start` 默认会启动 `aiprovider`
@@ -439,11 +418,13 @@ AI_PROVIDER_SERVICE_TOKEN=change_me
详细文档:
- [docs/technical/agents-aiprovider.md](/home/ray/dev/linkong/planet/docs/technical/agents-aiprovider.md)
- [docs/technical/zh/agents-aiprovider.md](/home/ray/dev/linkong/planet/docs/technical/zh/agents-aiprovider.md)
- [docs/technical/en/agents-aiprovider.md](/home/ray/dev/linkong/planet/docs/technical/en/agents-aiprovider.md)
- [aiprovider/README.md](/home/ray/dev/linkong/planet/aiprovider/README.md)
- [docs/technical/frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md)
- [docs/plans/frontend-ai-playground-development-plan.md](/home/ray/dev/linkong/planet/docs/plans/frontend-ai-playground-development-plan.md)
- [docs/plans/agents-situational-awareness-foundation-plan.md](/home/ray/dev/linkong/planet/docs/plans/agents-situational-awareness-foundation-plan.md)
- [docs/plans/agents-earth-command-runtime-plan.md](/home/ray/dev/linkong/planet/docs/plans/agents-earth-command-runtime-plan.md)
## 前端页面布局规范

11
TODO.md
View File

@@ -4,6 +4,9 @@ This file is the active backlog only. Completed history belongs in `docs/CHANGEL
## Earth
- [ ] Earth AI command entry: merge natural-language and speech-triggered LLM commands into the existing Earth search panel as described in [Agent Runtime, Earth LLM Command, And Speech Entry Plan](/home/ray/dev/linkong/planet/docs/plans/agents-earth-command-runtime-plan.md).
- [ ] Earth action executor: implement safe visualization actions for layer toggles, batch highlights, filters, focus, result panels, and clear-highlight behavior.
- [ ] Earth entity matching: support stable entity ids and batch matching for Beidou satellites, mainland China compute centers, BGP, news, vessels, and cables.
- [x] High-precision country boundary tile framework: implement the static vector tile builder, versioned seed output, frontend bbox tile loader, debounce, in-flight dedupe, and LRU cache described in [Earth High Precision Boundary Tiles Plan](/home/ray/dev/linkong/planet/docs/plans/earth-high-precision-boundary-tiles-plan.md).
- [x] Add the `pmtiles-mvt` frontend tile provider contract, MVT decoder dependencies, static PMTiles Nginx handling, collector artifact registration, production readiness check, and user operation docs for Earth boundaries.
- [x] Split Earth boundary ingestion into standard source collectors (`earth_admin0_boundaries`, `earth_coastline`, `earth_claim_lines`) plus the downstream `earth_boundary_tiles` PMTiles builder.
@@ -32,6 +35,14 @@ This file is the active backlog only. Completed history belongs in `docs/CHANGEL
## AI Provider And Agents
- [ ] Unified integration config schema: implement the shared low-code schema engine for datasource, AI Provider, Web Search, and OCR configuration described in [Integration Config Schema System Plan](/home/ray/dev/linkong/planet/docs/plans/integration-config-schema-system-plan.md).
- [ ] AI provider routing: finish the OpenClaw-style provider/model routing refactor described in [AI Provider OpenClaw-Style Routing Plan](/home/ray/dev/linkong/planet/docs/plans/ai-provider-openclaw-style-routing-plan.md), so model-specific transport rules live in provider metadata rather than runtime hardcoding.
- [ ] AI provider catalog: replace the temporary `model_provider_apis` bridge with structured `models_metadata`, discovery descriptors, and incremental model sync with stale marking.
- [ ] AI provider connectivity: keep the plug action as lightweight network/auth/model-directory validation only, and keep real generation tests inside Playground or explicit “trial run” actions.
- [ ] Agent runtime foundation: add auditable agent runs, steps, evidence, proposals, and the Agent operations UI described in [Agent Runtime, Earth LLM Command, And Speech Entry Plan](/home/ray/dev/linkong/planet/docs/plans/agents-earth-command-runtime-plan.md).
- [ ] Agent tool protocol: add backend JSON tool-call fallback, optional provider-native tool compatibility, tool whitelist validation, and policy-gated proposal application.
- [ ] Speech/ASR integration for agents: add provider-neutral transcription settings and API, defaulting to Whisper-compatible API providers while keeping text commands usable when ASR is unavailable.
- [ ] Earth voice wake: add device-local configurable wake-word preferences, microphone fallback states, and post-wake instruction upload for Earth commands.
- [ ] AI provider compatibility center: move provider/model compatibility rules into a JSON/YAML config read by runtime, instead of continuing to scatter provider-specific branches through Python code.
- [ ] Provider compatibility coverage: add explicit config for OpenAI, Anthropic, MiniMax, Ollama, Moonshot, DeepSeek, Qwen, GLM, Gemini, OpenRouter, vLLM, LM Studio, and One API.
- [ ] Compatibility schema: cover adapter type, base URL pattern, auth header, thinking/reasoning defaults, stream path, tool-call capability, multimodal capability, and provider-specific request patches.

View File

@@ -1 +1 @@
0.58.0
0.65.0

View File

@@ -17,9 +17,6 @@ class Settings(BaseSettings):
AI_HTTP_RETRY_ATTEMPTS: int = 2
AI_MAX_TOKENS: int = 1200
AI_ANTHROPIC_VERSION: str = "2023-06-01"
AI_ANALYSIS_SYSTEM_PROMPT: str = (
"你是态势感知分析助手。请基于输入的上下文、观测与约束,输出结构化、克制、可执行的分析。"
)
AI_PROVIDER_SERVICE_TOKEN: str = ""

View File

@@ -45,6 +45,7 @@ def get_provider_service(
x_ai_model: str | None = Header(default=None),
x_ai_max_tokens: str | None = Header(default=None),
x_ai_anthropic_version: str | None = Header(default=None),
x_ai_model_provider_apis: str | None = Header(default=None),
) -> ProviderService:
overrides = {
"provider": x_ai_provider,
@@ -53,6 +54,7 @@ def get_provider_service(
"api_key": x_ai_api_key,
"model": x_ai_model,
"anthropic_version": x_ai_anthropic_version,
"model_provider_apis": x_ai_model_provider_apis,
}
if x_ai_max_tokens:
overrides["max_tokens"] = x_ai_max_tokens

View File

@@ -1,6 +1,7 @@
from __future__ import annotations
import asyncio
import json
from typing import Any
import httpx
@@ -14,7 +15,6 @@ from aiprovider.schemas import (
SituationalAnalysisResponse,
)
def _normalize_provider(value: str) -> str:
return (value or "disabled").strip().lower()
@@ -62,7 +62,9 @@ class ProviderService:
self.anthropic_version = str(
overrides.get("anthropic_version") or settings.AI_ANTHROPIC_VERSION
)
self.system_prompt = settings.AI_ANALYSIS_SYSTEM_PROMPT
self.model_provider_apis = self._parse_model_provider_apis(
overrides.get("model_provider_apis")
)
def get_status(self) -> AIProviderStatusResponse:
enabled = self.provider != "disabled"
@@ -94,16 +96,23 @@ class ProviderService:
prompt = self._build_prompt(payload)
if self.provider_api == "openai-completions":
data = await self._request_openai_compatible(model, prompt)
provider_api = self._resolve_model_provider_api(model)
if provider_api == "openai-completions":
data = await self._request_openai_compatible(model, prompt, payload.system_prompt)
content = self._extract_openai_content(data)
content_blocks = self._extract_openai_blocks(data)
elif self.provider_api == "anthropic-messages":
data = await self._request_anthropic_messages(model, prompt, payload.thinking)
elif provider_api == "anthropic-messages":
data = await self._request_anthropic_messages(
model,
prompt,
payload.thinking,
payload.system_prompt,
)
content = self._extract_anthropic_content(data)
content_blocks = self._extract_anthropic_blocks(data)
elif self.provider_api == "ollama-generate":
data = await self._request_ollama(model, prompt)
elif provider_api == "ollama-generate":
data = await self._request_ollama(model, prompt, payload.system_prompt)
content = self._extract_ollama_content(data)
content_blocks = self._extract_ollama_blocks(data)
else:
@@ -128,6 +137,26 @@ class ProviderService:
def _requires_api_key(self) -> bool:
return self.provider_api != "ollama-generate"
def _resolve_model_provider_api(self, model: str) -> str:
return self.model_provider_apis.get(model) or self.provider_api
def _parse_model_provider_apis(self, value: Any) -> dict[str, str]:
if isinstance(value, dict):
raw = value
elif isinstance(value, str) and value.strip():
try:
parsed = json.loads(value)
except json.JSONDecodeError:
return {}
raw = parsed if isinstance(parsed, dict) else {}
else:
raw = {}
return {
str(model): _normalize_provider_api(str(provider_api))
for model, provider_api in raw.items()
if model and provider_api
}
def _build_prompt(self, payload: SituationalAnalysisRequest) -> str:
sections = [
f"任务标题:\n{payload.title}",
@@ -139,18 +168,26 @@ class ProviderService:
sections.append("约束条件:\n" + "\n".join(f"- {item}" for item in payload.constraints))
if payload.context:
sections.append(f"附加上下文:\n{payload.context}")
sections.append(
"请输出: 1) 态势摘要 2) 关键风险 3) 研判依据 4) 建议动作 5) 还缺少的数据。"
)
return "\n\n".join(sections)
async def _request_openai_compatible(self, model: str, prompt: str) -> dict[str, Any]:
def _resolve_system_prompt(self, system_prompt: str | None) -> str | None:
resolved = str(system_prompt or "").strip()
return resolved or None
async def _request_openai_compatible(
self,
model: str,
prompt: str,
system_prompt: str | None = None,
) -> dict[str, Any]:
messages = []
resolved_system_prompt = self._resolve_system_prompt(system_prompt)
if resolved_system_prompt:
messages.append({"role": "system", "content": resolved_system_prompt})
messages.append({"role": "user", "content": prompt})
request_body = {
"model": model,
"messages": [
{"role": "system", "content": self.system_prompt},
{"role": "user", "content": prompt},
],
"messages": messages,
"temperature": 0.2,
"max_tokens": self.max_tokens,
}
@@ -168,10 +205,10 @@ class ProviderService:
model: str,
prompt: str,
thinking: dict[str, Any] | None = None,
system_prompt: str | None = None,
) -> dict[str, Any]:
request_body = {
"model": model,
"system": self.system_prompt,
"messages": [
{
"role": "user",
@@ -186,6 +223,9 @@ class ProviderService:
"max_tokens": self.max_tokens,
"temperature": 0.2,
}
resolved_system_prompt = self._resolve_system_prompt(system_prompt)
if resolved_system_prompt:
request_body["system"] = resolved_system_prompt
resolved_thinking = self._resolve_anthropic_thinking(thinking)
if resolved_thinking:
request_body["thinking"] = resolved_thinking
@@ -219,20 +259,28 @@ class ProviderService:
model: str,
prompt: str,
thinking: dict[str, Any] | None = None,
system_prompt: str | None = None,
) -> dict[str, Any]:
return await self._request_anthropic_messages(model, prompt, thinking)
return await self._request_anthropic_messages(model, prompt, thinking, system_prompt)
async def _request_ollama(self, model: str, prompt: str) -> dict[str, Any]:
async def _request_ollama(
self,
model: str,
prompt: str,
system_prompt: str | None = None,
) -> dict[str, Any]:
request_body = {
"model": model,
"stream": False,
"system": self.system_prompt,
"prompt": prompt,
"options": {
"temperature": 0.2,
"num_predict": self.max_tokens,
},
}
resolved_system_prompt = self._resolve_system_prompt(system_prompt)
if resolved_system_prompt:
request_body["system"] = resolved_system_prompt
return await self._post(
path="/api/generate",
headers={
@@ -291,13 +339,19 @@ class ProviderService:
message = choices[0].get("message") or {}
content = message.get("content")
if isinstance(content, str):
return content
if content:
return content
reasoning_content = message.get("reasoning_content")
return reasoning_content if isinstance(reasoning_content, str) else ""
if isinstance(content, list):
return "".join(
item.get("text", "")
for item in content
if isinstance(item, dict)
)
reasoning_content = message.get("reasoning_content")
if isinstance(reasoning_content, str):
return reasoning_content
return ""
def _extract_openai_blocks(self, payload: dict[str, Any]) -> list[AIContentBlock]:
@@ -308,9 +362,14 @@ class ProviderService:
message = choices[0].get("message") or {}
content = message.get("content")
if isinstance(content, str):
return [AIContentBlock(type="text", text=content)]
blocks = [AIContentBlock(type="text", text=content)] if content else []
reasoning_content = message.get("reasoning_content")
if isinstance(reasoning_content, str) and reasoning_content:
blocks.append(AIContentBlock(type="thinking", thinking=reasoning_content))
return blocks
if not isinstance(content, list):
return []
reasoning_content = message.get("reasoning_content")
return [AIContentBlock(type="thinking", thinking=reasoning_content)] if isinstance(reasoning_content, str) and reasoning_content else []
blocks: list[AIContentBlock] = []
for item in content:
@@ -323,7 +382,11 @@ class ProviderService:
metadata={k: v for k, v in item.items() if k not in {"type", "text"}},
)
)
reasoning_content = message.get("reasoning_content")
if isinstance(reasoning_content, str) and reasoning_content:
blocks.append(AIContentBlock(type="thinking", thinking=reasoning_content))
return blocks
def _extract_anthropic_content(self, payload: dict[str, Any]) -> str:
content = payload.get("content")
if isinstance(content, str):

View File

@@ -13,10 +13,11 @@ class AIContentBlock(BaseModel):
class SituationalAnalysisRequest(BaseModel):
title: str = Field(..., min_length=1, max_length=200)
objective: str = Field(..., min_length=1, max_length=1000)
objective: str = Field(..., min_length=1, max_length=20000)
context: dict[str, Any] = Field(default_factory=dict)
observations: list[str] = Field(default_factory=list)
constraints: list[str] = Field(default_factory=list)
system_prompt: str | None = Field(default=None, max_length=8000)
preferred_model: str | None = Field(default=None, max_length=200)
thinking: dict[str, Any] | None = None

View File

@@ -0,0 +1,2 @@
"""AI task prompt registry and runtime helpers."""

View File

@@ -0,0 +1,74 @@
[
{
"key": "earth.news.enrich",
"label": "Earth 新闻汉化与定位",
"group": "Earth 新闻",
"version": "2026-05-16.2",
"system_prompt": "",
"prompt": "Return exactly one strict JSON object with a location object and a localizations object. Infer the most likely physical event location and produce a faithful Simplified Chinese title plus a one-sentence newswire-style Chinese summary based only on the supplied RSS headline, description, source, and date. The summary should read like a concise breaking-news lead, not a label, slogan, or keyword headline."
},
{
"key": "alerts.brief",
"label": "系统告警研判",
"group": "告警研判",
"version": "2026-05-16.1",
"system_prompt": "你是告警研判助手。请基于输入的告警事实、上下文与约束,输出结构化、克制、可执行的值班研判;明确区分事实、推断与建议,不要夸大证据不足的风险。",
"prompt": "基于当前告警总量、严重度、状态、数据源分布与最近告警摘录,生成一份面向值班人员的简明告警态势简报,突出待处理风险、告警集中点和优先动作。"
},
{
"key": "alerts.situational.brief",
"label": "跨模块态势告警研判",
"group": "告警研判",
"version": "2026-05-16.1",
"system_prompt": "你是告警研判助手。请基于输入的告警事实、上下文与约束,输出结构化、克制、可执行的值班研判;明确区分事实、推断与建议,不要夸大证据不足的风险。",
"prompt": "综合系统告警、BGP incidents、BGP anomalies 与近期 BGP AI 简报,生成一份面向值班人员的态势告警简报,指出当前最需要关注的风险域、跨模块联动迹象和优先动作。"
},
{
"key": "bgp.brief",
"label": "BGP 态势简报",
"group": "BGP",
"version": "2026-05-16.2",
"system_prompt": "你是 BGP 值班分析师。请直接输出面向值班人员的中文 Markdown 简报,只写最终研判内容;不要复述用户需求、提示词、写作计划、字段清单或“我将如何回答”。",
"prompt": "基于当前 BGP incidents、anomalies、原始观测事件、观测站覆盖与 prefix geography 证据,生成一份面向操作员的简明态势简报,突出区域热点、观测偏差、当前风险、证据和优先动作。"
},
{
"key": "location.factcheck.normalize",
"label": "位置事实核查结构化",
"group": "位置解析",
"version": "2026-05-16.1",
"system_prompt": "",
"prompt": "Convert the supplied location factcheck text into exactly one strict JSON object. Extract only facts present in the text or original query."
},
{
"key": "location.factcheck.resolve",
"label": "位置事实核查兜底",
"group": "位置解析",
"version": "2026-05-16.1",
"system_prompt": "",
"prompt": "Return exactly one JSON object for the most likely physical location. Use only fact-checkable public knowledge; return null fields rather than guessing when evidence is weak."
},
{
"key": "datasource.mapping",
"label": "数据源映射生成",
"group": "采集配置",
"version": "2026-05-16.1",
"system_prompt": "",
"prompt": "Return only JSON for a deterministic mapping DSL. The JSON must contain source.items_path and fields. Do not include prose or code."
},
{
"key": "credential.guide",
"label": "采集器凭据教程",
"group": "采集配置",
"version": "2026-05-16.1",
"system_prompt": "",
"prompt": "生成一份中文采集器凭据配置教程。只能根据 context.search_evidence 中的来源生成教程;如果证据不足,明确说明需要以官方页面为准。"
},
{
"key": "ai.connection_test",
"label": "AI Provider 连接测试",
"group": "运维测试",
"version": "2026-05-16.1",
"system_prompt": "",
"prompt": "Reply OK."
}
]

View File

@@ -0,0 +1,182 @@
from __future__ import annotations
from dataclasses import dataclass
from datetime import UTC, datetime
import json
from functools import lru_cache
from pathlib import Path
from typing import Any
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.models.system_setting import SystemSetting
AI_PROMPTS_CATEGORY = "ai_prompts"
DEFAULT_PROMPTS_PATH = Path(__file__).with_name("default_prompts.json")
@dataclass(frozen=True)
class AIPromptDefinition:
key: str
label: str
group: str
version: str
system_prompt: str
prompt: str
@dataclass(frozen=True)
class EffectiveAIPrompt:
key: str
label: str
group: str
version: str
default_system_prompt: str
default_prompt: str
system_prompt: str
prompt: str
is_custom: bool
updated_at: str | None = None
@lru_cache(maxsize=1)
def list_prompt_definitions() -> tuple[AIPromptDefinition, ...]:
raw_items = json.loads(DEFAULT_PROMPTS_PATH.read_text(encoding="utf-8"))
return tuple(
AIPromptDefinition(
key=str(item["key"]),
label=str(item["label"]),
group=str(item["group"]),
version=str(item["version"]),
system_prompt=str(item.get("system_prompt") or ""),
prompt=str(item.get("prompt") or ""),
)
for item in raw_items
)
def get_prompt_definition(task_key: str) -> AIPromptDefinition:
for definition in list_prompt_definitions():
if definition.key == task_key:
return definition
raise KeyError(task_key)
async def _get_prompt_setting(db: AsyncSession) -> SystemSetting | None:
result = await db.execute(
select(SystemSetting).where(SystemSetting.category == AI_PROMPTS_CATEGORY)
)
return result.scalar_one_or_none()
def _normalize_overrides(payload: dict[str, Any] | None) -> dict[str, dict[str, Any]]:
raw = (payload or {}).get("overrides")
if not isinstance(raw, dict):
return {}
return {
str(key): dict(value)
for key, value in raw.items()
if isinstance(value, dict)
}
async def get_prompt_overrides(db: AsyncSession) -> dict[str, dict[str, Any]]:
if not hasattr(db, "execute"):
return {}
setting = await _get_prompt_setting(db)
return _normalize_overrides(setting.payload if setting else None)
def _effective_prompt(
definition: AIPromptDefinition,
override: dict[str, Any] | None,
) -> EffectiveAIPrompt:
override = override or {}
custom_system = override.get("system_prompt")
custom_prompt = override.get("prompt")
has_custom_system = isinstance(custom_system, str)
has_custom_prompt = isinstance(custom_prompt, str)
return EffectiveAIPrompt(
key=definition.key,
label=definition.label,
group=definition.group,
version=definition.version,
default_system_prompt=definition.system_prompt,
default_prompt=definition.prompt,
system_prompt=custom_system if has_custom_system else definition.system_prompt,
prompt=custom_prompt if has_custom_prompt else definition.prompt,
is_custom=has_custom_system or has_custom_prompt,
updated_at=str(override.get("updated_at") or "") or None,
)
async def list_effective_prompts(db: AsyncSession) -> list[EffectiveAIPrompt]:
overrides = await get_prompt_overrides(db)
return [
_effective_prompt(definition, overrides.get(definition.key))
for definition in list_prompt_definitions()
]
async def get_effective_prompt(db: AsyncSession | None, task_key: str) -> EffectiveAIPrompt:
definition = get_prompt_definition(task_key)
if db is None:
return _effective_prompt(definition, None)
overrides = await get_prompt_overrides(db)
return _effective_prompt(definition, overrides.get(task_key))
async def save_prompt_override(
db: AsyncSession,
task_key: str,
*,
system_prompt: str,
prompt: str,
) -> EffectiveAIPrompt:
definition = get_prompt_definition(task_key)
setting = await _get_prompt_setting(db)
payload = dict(setting.payload or {}) if setting else {}
overrides = _normalize_overrides(payload)
overrides[definition.key] = {
"system_prompt": system_prompt,
"prompt": prompt,
"updated_at": datetime.now(UTC).isoformat().replace("+00:00", "Z"),
}
payload["overrides"] = overrides
if setting is None:
setting = SystemSetting(category=AI_PROMPTS_CATEGORY, payload=payload)
db.add(setting)
else:
setting.payload = payload
await db.commit()
return _effective_prompt(definition, overrides[definition.key])
async def reset_prompt_override(db: AsyncSession, task_key: str) -> EffectiveAIPrompt:
definition = get_prompt_definition(task_key)
setting = await _get_prompt_setting(db)
if setting is None:
return _effective_prompt(definition, None)
payload = dict(setting.payload or {})
overrides = _normalize_overrides(payload)
overrides.pop(definition.key, None)
payload["overrides"] = overrides
setting.payload = payload
await db.commit()
return _effective_prompt(definition, None)
def serialize_effective_prompt(prompt: EffectiveAIPrompt) -> dict[str, Any]:
return {
"key": prompt.key,
"label": prompt.label,
"group": prompt.group,
"version": prompt.version,
"default_system_prompt": prompt.default_system_prompt,
"default_prompt": prompt.default_prompt,
"system_prompt": prompt.system_prompt,
"prompt": prompt.prompt,
"is_custom": prompt.is_custom,
"updated_at": prompt.updated_at,
}

View File

@@ -6,6 +6,7 @@ from app.api.v1 import (
datasource_config,
datasources,
docs,
earth,
tasks,
dashboard,
alerts,
@@ -34,6 +35,7 @@ api_router.include_router(
api_router.include_router(datasources.router, prefix="/datasources", tags=["datasources"])
api_router.include_router(collected_data.router, prefix="/collected", tags=["collected-data"])
api_router.include_router(docs.router, prefix="/docs", tags=["docs"])
api_router.include_router(earth.router, prefix="/earth", tags=["earth"])
api_router.include_router(tasks.router, prefix="/tasks", tags=["tasks"])
api_router.include_router(dashboard.router, prefix="/dashboard", tags=["dashboard"])
api_router.include_router(alerts.router, prefix="/alerts", tags=["alerts"])

View File

@@ -341,6 +341,7 @@ async def collect_bgp_collector_location(
provider_client=provider_client,
query=query,
entity_type="bgp_collector",
db=db,
attempted_queries=attempted_queries,
search_evidence=search_result.evidence,
)

View File

@@ -102,7 +102,12 @@ def build_search_rank_sql(search: Optional[str]) -> str:
"""
def serialize_collected_row(row, source_name_map: dict[str, str] | None = None) -> dict:
def serialize_collected_row(
row,
source_name_map: dict[str, str] | None = None,
*,
include_metadata: bool = True,
) -> dict:
metadata = row[7]
source = row[1]
return {
@@ -120,7 +125,7 @@ def serialize_collected_row(row, source_name_map: dict[str, str] | None = None)
"longitude": get_metadata_field(metadata, "longitude"),
"value": get_metadata_field(metadata, "value"),
"unit": get_metadata_field(metadata, "unit"),
"metadata": metadata,
"metadata": metadata if include_metadata else None,
"cores": get_metadata_field(metadata, "cores"),
"rmax": get_metadata_field(metadata, "rmax"),
"rpeak": get_metadata_field(metadata, "rpeak"),
@@ -145,6 +150,7 @@ async def list_collected_data(
search: Optional[str] = Query(None, description="搜索名称"),
page: int = Query(1, ge=1, description="页码"),
page_size: int = Query(20, ge=1, le=100, description="每页数量"),
include_metadata: bool = Query(True, description="是否返回完整 metadata 字段"),
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
@@ -201,7 +207,7 @@ async def list_collected_data(
data = []
for row in rows:
data.append(serialize_collected_row(row[:11], source_name_map))
data.append(serialize_collected_row(row[:11], source_name_map, include_metadata=include_metadata))
return {
"total": total,

View File

@@ -5,7 +5,7 @@ from datetime import datetime
import base64
import json
import re
from fastapi import APIRouter, Depends, HTTPException, Query
from fastapi import APIRouter, Depends, HTTPException, Query, Request, status
from sqlalchemy import delete, select, func
from sqlalchemy.ext.asyncio import AsyncSession
from pydantic import BaseModel, Field
@@ -13,11 +13,6 @@ import httpx
from app.core.target_schema_registry import get_target_schema, list_target_schemas
from app.core.datasource_defaults import DEFAULT_DATASOURCES
from app.core.earth_boundary_defaults import (
EARTH_BOUNDARY_DEFAULT_SOURCES,
EARTH_BOUNDARY_SOURCE_MAPPING,
default_earth_boundary_config,
)
from app.db.session import get_db
from app.models.user import User
from app.models.datasource_config import DataSourceConfig
@@ -27,6 +22,7 @@ from app.models.vessel import AISRawObservation, AISSourceHealth
from app.core.security import get_current_user
from app.core.cache import cache
from app.core.time import to_iso8601_utc
from app.ai_tasks.prompts import get_effective_prompt
from app.schemas.ai import SituationalAnalysisRequest
from app.services.ai_client import AIProviderClient, get_ai_provider_client
from app.services.datasource_mapping import (
@@ -46,32 +42,80 @@ from app.services.custom_datasource_runtime import (
stop_custom_stream,
test_websocket_config,
)
DATASOURCE_MAPPING_PROMPT_KEY = "datasource.mapping"
from app.services.datasource_connectivity import (
_resolve_aisstream_api_key,
_resolve_spacetrack_credentials_with_override,
get_builtin_connection_status,
save_connectivity_success,
strip_connectivity_validation,
test_builtin_connectivity,
)
from app.services.barentswatch import resolve_barentswatch_config
from app.services.persistent_logs import record_audit_log
router = APIRouter()
SECRET_REVEAL_ROLES = {"admin", "super_admin"}
def _user_role_value(user: User) -> str:
role = getattr(user, "role", "")
return str(getattr(role, "value", role) or "").lower()
def _user_display_name(user: User) -> str:
return str(getattr(user, "username", None) or getattr(user, "email", None) or getattr(user, "id", ""))
async def _record_datasource_secret_reveal(
*,
current_user: User,
request: Request,
target_id: str,
result: str,
details: dict[str, Any],
) -> None:
await record_audit_log(
action="datasource_config.secret.reveal",
actor_id=getattr(current_user, "id", None),
actor_name=_user_display_name(current_user),
target_type="datasource_config_secret",
target_id=target_id,
result=result,
ip=request.client.host if request.client else None,
details=details,
)
async def _ensure_datasource_secret_reveal_allowed(
current_user: User,
request: Request,
target_id: str,
details: dict[str, Any],
) -> None:
if _user_role_value(current_user) in SECRET_REVEAL_ROLES:
return
await _record_datasource_secret_reveal(
current_user=current_user,
request=request,
target_id=target_id,
result="denied",
details={**details, "role": _user_role_value(current_user)},
)
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail="Only administrators can reveal datasource credentials",
)
def _default_builtin_config(name: str) -> dict[str, Any]:
if name in {"earth_admin0_boundaries", "earth_coastline", "earth_claim_lines"}:
return default_earth_boundary_config(name)
if name == "earth_boundary_tiles":
return {
"timeout": 30,
"retry": 0,
"input_mode": "latest_collected_earth_boundary_sources",
}
return {"timeout": 30, "retry": 3}
def _default_builtin_source_type(name: str) -> str:
if name == "aisstream_vessels":
return "websocket"
if name == "earth_boundary_tiles":
return "internal"
return "http"
@@ -400,11 +444,15 @@ async def list_all_datasources(
yaml_url = config.get_yaml_url(name)
db_config = db_configs.get(name)
default_config = _default_builtin_config(name)
default_url = EARTH_BOUNDARY_DEFAULT_SOURCES.get(name, {}).get("endpoint") or yaml_url
default_url = yaml_url
db_auth_config = db_config.auth_config or {} if db_config else {}
result.append(
{
"name": name,
"requires_credentials": bool(metadata.get("requires_credentials", False)),
"credential_provider": metadata.get("credential_provider"),
"credential_status": metadata.get("credential_status", "none"),
"default_url": default_url,
"endpoint": db_config.endpoint if db_config else default_url,
"is_overridden": db_config is not None and db_config.endpoint != yaml_url
@@ -413,10 +461,20 @@ async def list_all_datasources(
"is_active": db_config.is_active if db_config else True,
"source_type": db_config.source_type if db_config else _default_builtin_source_type(name),
"auth_type": db_config.auth_type if db_config else "none",
"auth_config": {
"client_id": db_auth_config.get("client_id") or "",
"username": db_auth_config.get("username") or "",
"key_name": db_auth_config.get("key_name") or db_auth_config.get("param_name") or "",
"param_name": db_auth_config.get("param_name") or db_auth_config.get("key_name") or "",
"location": db_auth_config.get("location") or db_auth_config.get("in") or "",
"in": db_auth_config.get("in") or db_auth_config.get("location") or "",
},
"auth_configured": {
"api_key": bool((db_config.auth_config or {}).get("api_key"))
if db_config
else False,
"api_key": bool(db_auth_config.get("api_key")),
"client_id": bool(db_auth_config.get("client_id")),
"client_secret": bool(db_auth_config.get("client_secret")),
"username": bool(db_auth_config.get("username")),
"password": bool(db_auth_config.get("password")),
},
"headers": db_config.headers if db_config else {},
"config": strip_connectivity_validation(db_config.config if db_config else default_config),
@@ -430,6 +488,96 @@ async def list_all_datasources(
return {"total": len(result), "data": result}
@router.get("/configs/secrets")
async def reveal_builtin_config_secrets(
request: Request,
name: str = Query(..., min_length=1),
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
"""Reveal configured built-in datasource credentials for admin editing."""
source = name.strip()
metadata = DEFAULT_DATASOURCES.get(source)
if not metadata or not metadata.get("requires_credentials"):
raise HTTPException(status_code=404, detail="Credentialed datasource config not found")
provider = str(metadata.get("credential_provider") or "")
target_id = f"datasource_config:{source}"
await _ensure_datasource_secret_reveal_allowed(
current_user,
request,
target_id,
{"source": source, "provider": provider},
)
result = await db.execute(select(DataSourceConfig).where(DataSourceConfig.name == source))
record = result.scalar_one_or_none()
auth_config = dict(record.auth_config or {}) if record else {}
payload: dict[str, Any] = {
"name": source,
"provider": provider,
}
details: dict[str, Any] = {"source": source, "provider": provider}
if provider == "barentswatch":
resolved = await resolve_barentswatch_config(db)
client_id = str(auth_config.get("client_id") or resolved.client_id or "")
client_secret = str(auth_config.get("client_secret") or resolved.client_secret or "")
source_label = "datasource_config" if auth_config.get("client_id") or auth_config.get("client_secret") else resolved.credential_source
payload.update(
{
"client_id": client_id,
"client_secret": client_secret,
"client_id_source": source_label if client_id else "missing",
"client_secret_source": source_label if client_secret else "missing",
}
)
details.update(
{
"client_id_configured": bool(client_id),
"client_secret_configured": bool(client_secret),
"credential_source": source_label,
}
)
elif provider == "aisstream":
api_key, api_key_source = await _resolve_aisstream_api_key(db)
payload.update({"api_key": api_key, "api_key_source": api_key_source})
details.update({"api_key_configured": bool(api_key), "api_key_source": api_key_source})
elif provider == "spacetrack":
if auth_config.get("username") or auth_config.get("password"):
username = str(auth_config.get("username") or "")
password = str(auth_config.get("password") or "")
credential_source = "datasource_config"
else:
username, password, credential_source = _resolve_spacetrack_credentials_with_override()
payload.update(
{
"username": username,
"password": password,
"username_source": credential_source if username else "missing",
"password_source": credential_source if password else "missing",
}
)
details.update(
{
"username_configured": bool(username),
"password_configured": bool(password),
"credential_source": credential_source,
}
)
else:
raise HTTPException(status_code=400, detail="Datasource credential provider is not supported")
await _record_datasource_secret_reveal(
current_user=current_user,
request=request,
target_id=target_id,
result="success",
details=details,
)
return payload
@router.get("/configs/{config_id}")
async def get_config(
config_id: int,
@@ -783,14 +931,12 @@ async def propose_datasource_mapping(
generated_by = "heuristic"
if payload.use_ai:
try:
prompt = await get_effective_prompt(db, DATASOURCE_MAPPING_PROMPT_KEY)
response = await ai_client.analyze(
SituationalAnalysisRequest(
title=f"Generate datasource mapping for {schema.key}",
objective=(
"Return only JSON for a deterministic mapping DSL. "
"The JSON must contain source.items_path and fields. "
"Do not include prose or code."
),
objective=prompt.prompt,
system_prompt=prompt.system_prompt or None,
context={
"target_schema": schema.to_dict(),
"sample_payload": redacted_sample,

View File

@@ -7,6 +7,7 @@ from pydantic import BaseModel, Field
from sqlalchemy import func, or_, select, text
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.cache import cache
from app.core.time import to_iso8601_utc
from app.core.security import get_current_user
from app.core.data_sources import get_data_sources_config
@@ -26,6 +27,7 @@ from app.services.scheduler import (
run_collector_now,
sync_datasource_job,
)
from app.services.earth_layer_cache import invalidate_earth_layer_cache_for_source
router = APIRouter()
STALE_RUNNING_TASK_TIMEOUT_MINUTES = 90
@@ -35,7 +37,6 @@ PRODUCT_SOURCE_KEYWORDS: tuple[tuple[str, tuple[str, ...]], ...] = (
("cables", ("cable", "landing", "telegeography", "arcgis", "fao")),
("satellites", ("tle", "satellite", "spacetrack", "celestrak")),
("bgp", ("bgp", "asn", "prefix_geo", "opengeofeed", "nro")),
("earth", ("earth_", "earth_boundary", "boundary", "country boundary", "coastline", "claim")),
("compute", ("top500", "gpu", "supercomputer", "compute")),
("ai", ("huggingface", "epoch_ai")),
("media", ("news", "tv", "live_stream")),
@@ -236,6 +237,8 @@ async def _load_datasource_endpoint_overrides(
async def _load_datasource_list_context(
db: AsyncSession,
datasources: list[DataSource],
*,
include_endpoint: bool = True,
) -> tuple[dict[int, CollectionTask], dict[int, CollectionTask], dict[str, str]]:
datasource_ids = [datasource.id for datasource in datasources]
sources = [datasource.source for datasource in datasources]
@@ -261,10 +264,65 @@ async def _load_datasource_list_context(
running_tasks = await _load_latest_running_tasks(db, datasource_ids)
latest_tasks = await _load_latest_tasks(db, datasource_ids)
endpoint_overrides = await _load_datasource_endpoint_overrides(db, sources)
endpoint_overrides = await _load_datasource_endpoint_overrides(db, sources) if include_endpoint else {}
return running_tasks, latest_tasks, endpoint_overrides
def serialize_datasource_row(
datasource: DataSource,
*,
running_tasks: dict[int, CollectionTask],
latest_tasks: dict[int, CollectionTask],
record_counts: dict[str, int],
endpoint_overrides: dict[str, str],
config,
include_endpoint: bool,
) -> dict:
running_task = running_tasks.get(datasource.id)
latest_task = latest_tasks.get(datasource.id)
display_task = running_task or latest_task
endpoint = None
if include_endpoint:
endpoint = endpoint_overrides.get(datasource.source) or config.get_yaml_url(datasource.source)
last_run_at = datasource.last_run_at or (latest_task.completed_at if latest_task else None)
last_status = datasource.last_status or (latest_task.status if latest_task else None)
collected_records = record_counts.get(datasource.source, 0)
row = {
"id": datasource.id,
"source": datasource.source,
"name": datasource.name,
**datasource_metadata(datasource.source),
"product": datasource_product_key(datasource),
"module": datasource.module,
"priority": datasource.priority,
"frequency": format_frequency_label(datasource.frequency_minutes),
"frequency_minutes": datasource.frequency_minutes,
"is_active": datasource.is_active,
"collector_class": datasource.collector_class,
"last_run": to_iso8601_utc(last_run_at),
"last_run_at": to_iso8601_utc(last_run_at),
"last_status": last_status,
"is_running": running_task is not None,
"task_id": display_task.id if display_task else None,
"progress": display_task.progress if display_task else None,
"phase": display_task.phase if display_task else None,
"phase_progress": display_task.phase_progress if display_task else None,
"phase_message": display_task.phase_message if display_task else None,
"phase_current": display_task.phase_current if display_task else None,
"phase_total": display_task.phase_total if display_task else None,
"phase_unit": display_task.phase_unit if display_task else None,
"records_processed": display_task.records_processed if display_task else None,
"total_records": display_task.total_records if display_task else None,
"error_message": display_task.error_message if display_task else None,
"collected_records": collected_records,
"has_collected_data": collected_records > 0,
}
if include_endpoint:
row["endpoint"] = endpoint
return row
def _apply_datasource_query_filters(
query,
*,
@@ -296,13 +354,14 @@ def _filter_datasources_in_memory(
datasources: list[DataSource],
*,
running_tasks: dict[int, CollectionTask],
latest_tasks: dict[int, CollectionTask],
latest_tasks: dict[int, CollectionTask] | None = None,
record_counts: dict[str, int],
product: Optional[str] = None,
run_status: Optional[str] = None,
collected: Optional[bool] = None,
credential_status: Optional[str] = None,
) -> list[DataSource]:
latest_tasks = latest_tasks or {}
filtered: list[DataSource] = []
for datasource in datasources:
record_count = record_counts.get(datasource.source, 0)
@@ -658,6 +717,7 @@ async def list_datasources(
collected: Optional[bool] = None,
credential_status: Optional[str] = None,
q: Optional[str] = None,
include_endpoint: bool = True,
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
@@ -676,7 +736,11 @@ async def list_datasources(
collector_list = []
config = get_data_sources_config()
running_tasks, latest_tasks, endpoint_overrides = await _load_datasource_list_context(db, datasources)
running_tasks, latest_tasks, endpoint_overrides = await _load_datasource_list_context(
db,
datasources,
include_endpoint=include_endpoint,
)
record_counts = await _load_collected_record_counts(db, [datasource.source for datasource in datasources])
datasources = _filter_datasources_in_memory(
datasources,
@@ -689,46 +753,16 @@ async def list_datasources(
credential_status=credential_status,
)
for datasource in datasources:
running_task = running_tasks.get(datasource.id)
latest_task = latest_tasks.get(datasource.id)
display_task = running_task or latest_task
endpoint = endpoint_overrides.get(datasource.source) or config.get_yaml_url(datasource.source)
last_run_at = datasource.last_run_at or (latest_task.completed_at if latest_task else None)
last_status = datasource.last_status or (latest_task.status if latest_task else None)
collected_records = record_counts.get(datasource.source, 0)
collector_list.append(
{
"id": datasource.id,
"source": datasource.source,
"name": datasource.name,
**datasource_metadata(datasource.source),
"product": datasource_product_key(datasource),
"module": datasource.module,
"priority": datasource.priority,
"frequency": format_frequency_label(datasource.frequency_minutes),
"frequency_minutes": datasource.frequency_minutes,
"is_active": datasource.is_active,
"collector_class": datasource.collector_class,
"endpoint": endpoint,
"last_run": to_iso8601_utc(last_run_at),
"last_run_at": to_iso8601_utc(last_run_at),
"last_status": last_status,
"is_running": running_task is not None,
"task_id": display_task.id if display_task else None,
"progress": display_task.progress if display_task else None,
"phase": display_task.phase if display_task else None,
"phase_progress": display_task.phase_progress if display_task else None,
"phase_message": display_task.phase_message if display_task else None,
"phase_current": display_task.phase_current if display_task else None,
"phase_total": display_task.phase_total if display_task else None,
"phase_unit": display_task.phase_unit if display_task else None,
"records_processed": display_task.records_processed if display_task else None,
"total_records": display_task.total_records if display_task else None,
"error_message": display_task.error_message if display_task else None,
"collected_records": collected_records,
"has_collected_data": collected_records > 0,
}
serialize_datasource_row(
datasource,
running_tasks=running_tasks,
latest_tasks=latest_tasks,
record_counts=record_counts,
endpoint_overrides=endpoint_overrides,
config=config,
include_endpoint=include_endpoint,
)
)
return {"total": len(collector_list), "data": collector_list}
@@ -785,6 +819,51 @@ async def trigger_datasource_batch(
return await _trigger_datasource_batch(db, datasources, force=payload.force)
@router.get("/snapshots")
async def list_datasource_snapshots(
source_id: Optional[str] = None,
current_only: Optional[bool] = None,
limit: int = Query(default=100, ge=1, le=500),
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
query = (
select(DataSnapshot, DataSource.name, DataSource.module)
.outerjoin(DataSource, DataSource.id == DataSnapshot.datasource_id)
.order_by(DataSnapshot.created_at.desc().nullslast(), DataSnapshot.id.desc())
.limit(limit)
)
if source_id:
query = query.where(DataSnapshot.source == source_id)
if current_only is not None:
query = query.where(DataSnapshot.is_current.is_(current_only))
result = await db.execute(query)
rows = []
for snapshot, datasource_name, datasource_module in result.all():
rows.append(
{
"id": snapshot.id,
"datasource_id": snapshot.datasource_id,
"datasource_name": datasource_name,
"module": datasource_module,
"task_id": snapshot.task_id,
"source": snapshot.source,
"snapshot_key": snapshot.snapshot_key,
"reference_date": to_iso8601_utc(snapshot.reference_date),
"started_at": to_iso8601_utc(snapshot.started_at),
"completed_at": to_iso8601_utc(snapshot.completed_at),
"record_count": snapshot.record_count,
"status": snapshot.status,
"is_current": snapshot.is_current,
"parent_snapshot_id": snapshot.parent_snapshot_id,
"summary": snapshot.summary or {},
"created_at": to_iso8601_utc(snapshot.created_at),
}
)
return {"total": len(rows), "data": rows}
@router.get("/{source_id}")
async def get_datasource(
source_id: str,
@@ -813,6 +892,37 @@ async def get_datasource(
}
@router.get("/{source_id}/row")
async def get_datasource_row(
source_id: str,
include_endpoint: bool = True,
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
datasource = await get_datasource_record(db, source_id)
if not datasource:
raise HTTPException(status_code=404, detail="Data source not found")
config = get_data_sources_config()
running_tasks, latest_tasks, endpoint_overrides = await _load_datasource_list_context(
db,
[datasource],
include_endpoint=include_endpoint,
)
record_counts = await _load_collected_record_counts(db, [datasource.source])
return {
"data": serialize_datasource_row(
datasource,
running_tasks=running_tasks,
latest_tasks=latest_tasks,
record_counts=record_counts,
endpoint_overrides=endpoint_overrides,
config=config,
include_endpoint=include_endpoint,
)
}
@router.post("/{source_id}/enable")
async def enable_datasource(
source_id: str,
@@ -960,6 +1070,29 @@ async def clear_datasource_data(
}
@router.delete("/{source_id}/cache")
async def clear_datasource_cache(
source_id: str,
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
datasource = await get_datasource_record(db, source_id)
if not datasource:
raise HTTPException(status_code=404, detail="Data source not found")
earth_deleted_count = invalidate_earth_layer_cache_for_source(datasource.source)
dashboard_deleted_count = int(cache.delete("dashboard:stats")) + int(cache.delete("dashboard:summary"))
deleted_count = earth_deleted_count + dashboard_deleted_count
return {
"status": "success",
"message": f"Cleared {deleted_count} cache keys for data source '{datasource.name}'",
"deleted_count": deleted_count,
"earth_layer_deleted_count": earth_deleted_count,
"dashboard_deleted_count": dashboard_deleted_count,
}
@router.get("/{source_id}/task-status")
async def get_task_status(
source_id: str,

417
backend/app/api/v1/earth.py Normal file
View File

@@ -0,0 +1,417 @@
"""Earth asset management APIs."""
from __future__ import annotations
from pathlib import Path
from typing import Any
from uuid import uuid4
from fastapi import APIRouter, Depends, File, HTTPException, Request, UploadFile, status
from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
from pydantic import BaseModel, Field
from sqlalchemy import delete, func, select, text
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.security import decode_token, get_current_user, redis_client
from app.db.session import get_db
from app.models.collected_data import CollectedData
from app.models.datasource import DataSource
from app.models.datasource_config import DataSourceConfig
from app.models.system_setting import SystemSetting
from app.models.user import User
from app.services.tv_streams import get_tv_settings_payload
from app.services.earth_boundaries import (
EarthBoundaryBuildError,
get_boundary_build_status,
get_boundary_status,
save_boundary_config,
start_boundary_build_job,
)
router = APIRouter()
optional_bearer = HTTPBearer(auto_error=False)
REPO_ROOT = Path(__file__).resolve().parents[4]
EARTH_BRAND_ASSET_DIR = REPO_ROOT / "data" / "earth-brand"
EARTH_BRAND_ASSET_URL_PREFIX = "/earth-brand-assets"
EARTH_BRAND_CATEGORY = "earth_brand"
EARTH_ABOUT_CATEGORY = "earth_about"
MAX_EARTH_BRAND_ASSET_BYTES = 3 * 1024 * 1024
ALLOWED_EARTH_BRAND_EXTENSIONS = {".png", ".jpg", ".jpeg", ".webp", ".svg"}
DEFAULT_EARTH_BRAND = {
"logo_src": "/earth/assets/brand/earth-logo.png",
"title_src": "/earth/assets/brand/title-zh.png",
"title_text": "智能星球计划",
"subtitle": "现实层宇宙全息感知系统",
"description": "卫星 · 海底光缆 · 算力基础设施",
"aria_label": "智能星球计划品牌标识",
"title_alt": "智能星球计划",
}
DEFAULT_EARTH_ABOUT = {
"logo_src": "/earth/assets/brand/lim-logo.png",
"kicker": "About",
"title": "智能星球计划",
"version": "v0.64.0",
"description": "面向临空场景下的智能媒体研究、全球态势感知与多源开放数据巡航,提供可视化观测、事件聚合与交互式探索能力。",
"meta": [
{"label": "出品方", "value": "浙江大学临空智能媒体研究院"},
{"label": "策划人", "value": "黄柳青"},
{"label": "产品兼开发者", "value": "钱坤、张鸽、齐鹏"},
],
}
class EarthBoundaryConfigPayload(BaseModel):
config: dict[str, Any] = Field(default_factory=dict)
class EarthBrandPayload(BaseModel):
logo_src: str = Field(default=DEFAULT_EARTH_BRAND["logo_src"], max_length=1000)
title_src: str = Field(default=DEFAULT_EARTH_BRAND["title_src"], max_length=1000)
title_text: str = Field(default=DEFAULT_EARTH_BRAND["title_text"], max_length=120)
subtitle: str = Field(default=DEFAULT_EARTH_BRAND["subtitle"], max_length=160)
description: str = Field(default=DEFAULT_EARTH_BRAND["description"], max_length=200)
aria_label: str = Field(default=DEFAULT_EARTH_BRAND["aria_label"], max_length=200)
title_alt: str = Field(default=DEFAULT_EARTH_BRAND["title_alt"], max_length=200)
class EarthAboutMetaItem(BaseModel):
label: str = Field(default="", max_length=80)
value: str = Field(default="", max_length=240)
class EarthAboutPayload(BaseModel):
logo_src: str = Field(default=DEFAULT_EARTH_ABOUT["logo_src"], max_length=1000)
kicker: str = Field(default=DEFAULT_EARTH_ABOUT["kicker"], max_length=80)
title: str = Field(default=DEFAULT_EARTH_ABOUT["title"], max_length=160)
version: str = Field(default=DEFAULT_EARTH_ABOUT["version"], max_length=80)
description: str = Field(default=DEFAULT_EARTH_ABOUT["description"], max_length=800)
meta: list[EarthAboutMetaItem] = Field(default_factory=list)
def _normalize_earth_brand_payload(payload: dict[str, Any] | None) -> dict[str, str]:
merged = DEFAULT_EARTH_BRAND.copy()
if payload:
for key in DEFAULT_EARTH_BRAND:
value = payload.get(key)
if value is not None:
merged[key] = str(value).strip()
if not merged["title_text"]:
merged["title_text"] = DEFAULT_EARTH_BRAND["title_text"]
if not merged["aria_label"]:
merged["aria_label"] = merged["title_text"]
if not merged["title_alt"]:
merged["title_alt"] = merged["title_text"]
return merged
def _normalize_earth_about_payload(payload: dict[str, Any] | None) -> dict[str, Any]:
merged: dict[str, Any] = {
key: value
for key, value in DEFAULT_EARTH_ABOUT.items()
if key != "meta"
}
raw_meta = DEFAULT_EARTH_ABOUT["meta"]
if payload:
for key in ("logo_src", "kicker", "title", "version", "description"):
value = payload.get(key)
if value is not None:
merged[key] = str(value).strip()
raw_meta = payload.get("meta") if isinstance(payload.get("meta"), list) else raw_meta
for key, default_value in DEFAULT_EARTH_ABOUT.items():
if key == "meta":
continue
if not merged.get(key):
merged[key] = default_value
normalized_meta: list[dict[str, str]] = []
for item in raw_meta:
if not isinstance(item, dict):
continue
label = str(item.get("label") or "").strip()
value = str(item.get("value") or "").strip()
if label or value:
normalized_meta.append({"label": label, "value": value})
if not normalized_meta:
normalized_meta = [dict(item) for item in DEFAULT_EARTH_ABOUT["meta"]]
merged["meta"] = normalized_meta
return merged
async def _get_earth_brand_record(db: AsyncSession) -> SystemSetting | None:
result = await db.execute(
select(SystemSetting).where(SystemSetting.category == EARTH_BRAND_CATEGORY)
)
return result.scalar_one_or_none()
async def _get_earth_brand_payload(db: AsyncSession) -> dict[str, Any]:
record = await _get_earth_brand_record(db)
return {
"brand": _normalize_earth_brand_payload(record.payload if record else None),
"is_default": record is None,
}
async def _get_earth_about_record(db: AsyncSession) -> SystemSetting | None:
result = await db.execute(
select(SystemSetting).where(SystemSetting.category == EARTH_ABOUT_CATEGORY)
)
return result.scalar_one_or_none()
async def _get_earth_about_payload(db: AsyncSession) -> dict[str, Any]:
record = await _get_earth_about_record(db)
return {
"about": _normalize_earth_about_payload(record.payload if record else None),
"is_default": record is None,
}
async def _get_optional_current_user(
credentials: HTTPAuthorizationCredentials | None = Depends(optional_bearer),
db: AsyncSession = Depends(get_db),
) -> User | None:
if credentials is None:
return None
token = credentials.credentials
if redis_client.sismember("blacklisted_tokens", token):
return None
payload = decode_token(token)
if payload is None or payload.get("type") != "access":
return None
user_id = payload.get("sub")
if user_id is None:
return None
result = await db.execute(
text(
"SELECT id, username, email, password_hash, role, is_active, gatekeeper_groups FROM users WHERE id = :id"
),
{"id": int(user_id)},
)
row = result.fetchone()
if row is None or not row[5]:
return None
user = User()
user.id = row[0]
user.username = row[1]
user.email = row[2]
user.password_hash = row[3]
user.role = row[4]
user.is_active = row[5]
user.gatekeeper_groups = row[6] or []
return user
@router.get("/brand")
async def get_earth_brand(db: AsyncSession = Depends(get_db)):
return await _get_earth_brand_payload(db)
@router.put("/brand")
async def update_earth_brand(
payload: EarthBrandPayload,
_current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
normalized = _normalize_earth_brand_payload(payload.model_dump())
record = await _get_earth_brand_record(db)
if record is None:
record = SystemSetting(category=EARTH_BRAND_CATEGORY, payload=normalized)
db.add(record)
else:
record.payload = normalized
await db.commit()
await db.refresh(record)
return {"status": "updated", "brand": _normalize_earth_brand_payload(record.payload), "is_default": False}
@router.delete("/brand")
@router.post("/brand/reset")
async def reset_earth_brand(
_current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
await db.execute(delete(SystemSetting).where(SystemSetting.category == EARTH_BRAND_CATEGORY))
await db.commit()
return {"status": "reset", "brand": DEFAULT_EARTH_BRAND.copy(), "is_default": True}
@router.post("/brand/assets")
async def upload_earth_brand_asset(
file: UploadFile = File(...),
_current_user: User = Depends(get_current_user),
):
original_name = file.filename or ""
extension = Path(original_name).suffix.lower()
if extension not in ALLOWED_EARTH_BRAND_EXTENSIONS:
raise HTTPException(
status_code=400,
detail={
"code": "unsupported_file_type",
"message": "Only png, jpg, jpeg, webp, and svg brand assets are supported.",
},
)
content = await file.read(MAX_EARTH_BRAND_ASSET_BYTES + 1)
if len(content) > MAX_EARTH_BRAND_ASSET_BYTES:
raise HTTPException(
status_code=400,
detail={
"code": "file_too_large",
"message": "Brand asset must be 3 MB or smaller.",
},
)
EARTH_BRAND_ASSET_DIR.mkdir(parents=True, exist_ok=True)
safe_name = f"{uuid4().hex}{extension}"
destination = EARTH_BRAND_ASSET_DIR / safe_name
destination.write_bytes(content)
asset_url = f"{EARTH_BRAND_ASSET_URL_PREFIX}/{safe_name}"
return {"url": asset_url, "filename": safe_name, "content_type": file.content_type}
@router.get("/about")
async def get_earth_about(db: AsyncSession = Depends(get_db)):
return await _get_earth_about_payload(db)
@router.put("/about")
async def update_earth_about(
payload: EarthAboutPayload,
_current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
normalized = _normalize_earth_about_payload(payload.model_dump())
record = await _get_earth_about_record(db)
if record is None:
record = SystemSetting(category=EARTH_ABOUT_CATEGORY, payload=normalized)
db.add(record)
else:
record.payload = normalized
await db.commit()
await db.refresh(record)
return {"status": "updated", "about": _normalize_earth_about_payload(record.payload), "is_default": False}
@router.delete("/about")
async def reset_earth_about(
_current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
await db.execute(delete(SystemSetting).where(SystemSetting.category == EARTH_ABOUT_CATEGORY))
await db.commit()
return {"status": "reset", "about": _normalize_earth_about_payload(None), "is_default": True}
@router.get("/oobe-status")
async def get_earth_oobe_status(
current_user: User | None = Depends(_get_optional_current_user),
db: AsyncSession = Depends(get_db),
):
current_count_result = await db.execute(
select(func.count(CollectedData.id)).where(CollectedData.is_current.is_(True))
)
current_record_count = int(current_count_result.scalar() or 0)
datasource_count_result = await db.execute(select(func.count(DataSource.id)))
datasource_count = int(datasource_count_result.scalar() or 0)
active_datasource_count_result = await db.execute(
select(func.count(DataSource.id)).where(DataSource.is_active.is_(True))
)
active_datasource_count = int(active_datasource_count_result.scalar() or 0)
config_result = await db.execute(select(func.count(DataSourceConfig.id)))
custom_config_count = int(config_result.scalar() or 0)
tv_payload = await get_tv_settings_payload(db)
tv_sources = tv_payload.get("sources") if isinstance(tv_payload, dict) else []
tv_source_count = len(tv_sources) if isinstance(tv_sources, list) else 0
boundary_status = get_boundary_status()
has_core_layers = bool(boundary_status.get("ready") or boundary_status.get("available") or boundary_status.get("status") in {"ready", "built", "ok"})
has_collected_data = current_record_count > 0
ready = has_collected_data
suggestions: list[str] = []
if not current_user:
suggestions.append("登录控制台")
if not has_collected_data:
suggestions.append("触发数据源采集")
if not custom_config_count:
suggestions.append("确认采集器配置")
if not has_core_layers:
suggestions.append("构建或启用 Earth 图层")
return {
"ready": ready,
"authenticated": current_user is not None,
"needs_login": current_user is None and not ready,
"has_collected_data": has_collected_data,
"has_tv_sources": tv_source_count > 0,
"has_core_layers": has_core_layers,
"current_record_count": current_record_count,
"datasource_count": datasource_count,
"active_datasource_count": active_datasource_count,
"custom_config_count": custom_config_count,
"tv_source_count": tv_source_count,
"suggestions": suggestions,
"login_url": "/login?next=/datasources",
"datasources_url": "/datasources",
"collection_url": "/collection-management",
}
@router.get("/boundaries/status")
async def get_earth_boundary_status():
return get_boundary_status()
def _is_loopback_request(request: Request) -> bool:
host = request.client.host if request.client else ""
return host in {"127.0.0.1", "::1", "localhost"} or host.startswith("127.")
def _require_local_or_user(request: Request, user: User | None) -> None:
if user is not None or _is_loopback_request(request):
return
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Authentication required outside localhost",
)
@router.put("/boundaries/config")
async def update_earth_boundary_config(
payload: EarthBoundaryConfigPayload,
_current_user: User = Depends(get_current_user),
):
try:
return save_boundary_config(payload.config)
except EarthBoundaryBuildError as exc:
raise HTTPException(
status_code=400,
detail={"code": exc.code, "message": str(exc), "details": exc.details},
) from exc
@router.post("/boundaries/build")
async def build_earth_boundary_assets(
request: Request,
current_user: User | None = Depends(_get_optional_current_user),
):
_require_local_or_user(request, current_user)
try:
return await start_boundary_build_job()
except EarthBoundaryBuildError as exc:
raise HTTPException(
status_code=400,
detail={"code": exc.code, "message": str(exc), "details": exc.details},
) from exc
@router.get("/boundaries/build/status")
async def get_earth_boundary_build_status():
return get_boundary_build_status()

View File

@@ -1,6 +1,6 @@
from typing import Any, Optional
from fastapi import APIRouter, Depends, HTTPException, Query
from fastapi import APIRouter, Depends, HTTPException, Query, Response
from sqlalchemy.ext.asyncio import AsyncSession
from app.api.v1.visualization import (
@@ -110,6 +110,7 @@ async def get_vessel_layer_snapshot(
vessel_type: Optional[str] = Query(None, alias="type"),
since_minutes: int = Query(60, ge=1, le=1440),
db: AsyncSession = Depends(get_db),
response: Response = None,
):
parsed_bbox = _parse_layer_bbox(bbox)
return await build_vessel_snapshot_response(
@@ -119,6 +120,7 @@ async def get_vessel_layer_snapshot(
limit=limit,
type_filter=vessel_type,
since_minutes=since_minutes,
response=response,
)

View File

@@ -4,7 +4,8 @@ import os
from pathlib import Path
from typing import Optional
from fastapi import APIRouter, Depends, HTTPException, Query
from fastapi import APIRouter, Depends, HTTPException, Query, Request, status
import httpx
from pydantic import BaseModel, EmailStr, Field
from dotenv import dotenv_values
from sqlalchemy import select
@@ -15,6 +16,13 @@ from app.core.time import to_iso8601_utc
from app.core.config import settings as app_settings
from app.core.data_sources import get_data_sources_config
from app.core.datasource_defaults import DEFAULT_DATASOURCES
from app.ai_tasks.prompts import (
get_effective_prompt,
list_effective_prompts,
reset_prompt_override,
save_prompt_override,
serialize_effective_prompt,
)
from app.db.session import get_db
from app.models.datasource import DataSource
from app.models.datasource_config import DataSourceConfig
@@ -57,9 +65,12 @@ from app.services.llm_provider_catalog import (
)
from app.services.scheduler import sync_datasource_job
from app.services.tv_streams import DEFAULT_TV_SETTINGS, get_tv_settings_payload, normalize_tv_settings
from app.services.persistent_logs import record_audit_log
router = APIRouter()
AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS = 5
AI_CONNECTION_TEST_PROMPT_KEY = "ai.connection_test"
SECRET_REVEAL_ROLES = {"admin", "super_admin"}
DEFAULT_SETTINGS = {
"system": {
@@ -122,6 +133,70 @@ DEFAULT_SETTINGS = {
}
def _user_role_value(user: User) -> str:
role = getattr(user, "role", "")
return role.value if hasattr(role, "value") else str(role or "")
def _user_display_name(user: User) -> str | None:
return getattr(user, "username", None) or getattr(user, "email", None)
def _request_client_ip(request: Request | None) -> str | None:
if request is None or request.client is None:
return None
return request.client.host
def _can_reveal_integration_secrets(user: User) -> bool:
return _user_role_value(user) in SECRET_REVEAL_ROLES
async def _record_integration_secret_reveal(
*,
current_user: User,
request: Request | None,
target_id: str,
result: str,
details: dict,
) -> None:
await record_audit_log(
action="settings.integration_secret.reveal",
actor_id=getattr(current_user, "id", None),
actor_name=_user_display_name(current_user),
target_type="integration_secret",
target_id=target_id,
result=result,
ip=_request_client_ip(request),
details=details,
)
async def _ensure_secret_reveal_allowed(
*,
current_user: User,
request: Request | None,
target_id: str,
details: dict | None = None,
) -> None:
if _can_reveal_integration_secrets(current_user):
return
await _record_integration_secret_reveal(
current_user=current_user,
request=request,
target_id=target_id,
result="denied",
details={
**(details or {}),
"role": _user_role_value(current_user),
},
)
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail="Only administrators can reveal integration secrets",
)
class SystemSettingsUpdate(BaseModel):
system_name: str = "智能星球"
refresh_interval: int = Field(default=60, ge=10, le=3600)
@@ -251,6 +326,11 @@ class OCRIntegrationUpdate(BaseModel):
output_format: str = Field(default="markdown", pattern="^(markdown|text|json)$")
class AIPromptUpdate(BaseModel):
system_prompt: str = Field(default="", max_length=8000)
prompt: str = Field(min_length=1, max_length=20000)
class ExternalIntegrationsUpdate(BaseModel):
ai_provider: AIProviderIntegrationUpdate
barentswatch: BarentsWatchIntegrationUpdate
@@ -340,9 +420,10 @@ def _get_provider_preset(provider: str) -> dict:
"provider": provider,
"provider_api": "openai-completions",
"base_url": "",
"model": "",
"models": [],
"api_key_env": "",
"model": "",
"models": [],
"model_provider_apis": {},
"api_key_env": "",
}
@@ -364,6 +445,9 @@ def _resolve_env_secret(*names: str) -> tuple[str, str]:
value = env_file_values.get(name)
if value:
return value, "env_file"
value = os.environ.get(name)
if value:
return value, "env"
return "", ""
@@ -406,9 +490,16 @@ def _provider_defaults(provider: str) -> dict:
1200 if preset.get("provider_api") == "anthropic-messages" else 4096
),
"anthropic_version": "2023-06-01",
"model_provider_apis": preset.get("model_provider_apis") or {},
}
def _selected_ai_env_provider() -> str:
env_file_values = _read_ai_provider_env_file()
provider = env_file_values.get("AI_PROVIDER") or os.environ.get("AI_PROVIDER") or "minimax"
return _normalize_provider_id(provider)
def _normalize_ai_provider_payload(ai_payload: dict | None) -> dict:
raw = dict(ai_payload or {})
default_provider = _normalize_provider_id(raw.get("default_provider") or raw.get("provider"))
@@ -427,6 +518,7 @@ def _normalize_ai_provider_payload(ai_payload: dict | None) -> dict:
"api_key",
"max_tokens",
"anthropic_version",
"model_provider_apis",
)
if raw.get(key) not in (None, "")
}
@@ -464,7 +556,12 @@ def _resolve_provider_api_key(provider: str, provider_config: dict) -> tuple[str
return str(saved_key), "runtime"
preset = _get_provider_preset(provider)
api_key_env = preset.get("api_key_env") or ""
return _resolve_env_secret(api_key_env, "AI_API_KEY")
value, source = _resolve_env_secret(api_key_env)
if value:
return value, source
if _normalize_provider_id(provider) == _selected_ai_env_provider():
return _resolve_env_secret("AI_API_KEY")
return "", ""
def _resolve_service_token(ai_payload: dict) -> tuple[str, str]:
@@ -487,12 +584,21 @@ def _is_secret_placeholder(value: Optional[str], current_preview: str = "") -> b
return True
if text == current_preview or text.startswith("••••"):
return True
if "-" in text:
_prefix, masked = text.split("-", 1)
if masked and all(char in {"*", "", " ", "\t"} for char in masked):
return True
return all(char in {"*", "", " ", "\t"} for char in text)
def _build_ai_provider_payload(current_payload: dict, update: AIProviderIntegrationUpdate) -> dict:
current_ai = _normalize_ai_provider_payload(current_payload.get("ai_provider") or {})
provider_id = _normalize_provider_id(update.default_provider or update.provider)
provider_id = _normalize_provider_id(update.provider)
default_provider = (
_normalize_provider_id(update.default_provider)
if update.default_provider is not None
else current_ai["default_provider"]
)
current_providers = {
provider: dict(config or {})
for provider, config in current_ai.get("providers", {}).items()
@@ -526,7 +632,7 @@ def _build_ai_provider_payload(current_payload: dict, update: AIProviderIntegrat
"service_url": update.service_url.strip()
or app_settings.AI_PROVIDER_SERVICE_URL,
"service_token": current_ai.get("service_token") or "",
"default_provider": provider_id,
"default_provider": default_provider,
"providers": current_providers,
"timeout_seconds": update.timeout_seconds,
"retry_attempts": update.retry_attempts,
@@ -560,6 +666,8 @@ def _runtime_config_from_ai_payload(ai_payload: dict) -> dict:
"api_key": api_key,
"max_tokens": int(provider_config.get("max_tokens") or 1200),
"anthropic_version": provider_config.get("anthropic_version") or "2023-06-01",
"model_provider_apis": provider_config.get("model_provider_apis") or {},
"preset_models": _get_provider_preset(default_provider).get("models") or [],
},
}
@@ -597,10 +705,12 @@ async def _validate_ai_provider_full_connection(ai_payload: dict) -> dict:
status_code=400,
detail="AI Provider 可访问,但当前 provider/model/key 未完整配置。",
)
prompt = await get_effective_prompt(None, AI_CONNECTION_TEST_PROMPT_KEY)
analysis_result = await client.analyze(
SituationalAnalysisRequest(
title="保存前完整连接测试",
objective="请用一句话回复连接可用。",
objective=prompt.prompt,
system_prompt=prompt.system_prompt or None,
observations=["这是保存 AI Provider 配置前的完整 LLM 调用测试。"],
constraints=["回复尽量简短。"],
)
@@ -612,6 +722,159 @@ async def _validate_ai_provider_full_connection(ai_payload: dict) -> dict:
}
def _join_provider_url(base_url: str, path: str) -> str:
return f"{base_url.rstrip('/')}/{path.lstrip('/')}"
def _extract_model_ids(payload: dict) -> list[str]:
data = payload.get("data") if isinstance(payload, dict) else None
if isinstance(data, list):
return [
str(item.get("id"))
for item in data
if isinstance(item, dict) and item.get("id")
]
models = payload.get("models") if isinstance(payload, dict) else None
if isinstance(models, list):
return [
str(item.get("name") or item.get("model") or item.get("id") or item)
for item in models
if item
]
return []
def _contains_model(model_ids: list[str], model: str) -> bool:
normalized_model = model.strip().lower()
return any(str(item).strip().lower() == normalized_model for item in model_ids)
async def _check_ai_provider_lightweight(llm_config: dict, timeout_seconds: int) -> dict:
provider = _normalize_provider_id(llm_config.get("provider") or "")
configured_api = str(llm_config.get("provider_api") or "").strip() or "openai-completions"
model = str(llm_config.get("model") or "").strip()
base_url = str(llm_config.get("base_url") or "").strip().rstrip("/")
api_key = str(llm_config.get("api_key") or "").strip()
provider_api = configured_api
model_provider_apis = llm_config.get("model_provider_apis")
if isinstance(model_provider_apis, dict):
provider_api = str(model_provider_apis.get(model) or provider_api)
preset_models = [
str(item)
for item in (llm_config.get("preset_models") or [])
if str(item).strip()
]
if not provider or not base_url or not model:
return {
"success": False,
"connected": False,
"message": "当前 provider/base_url/model 未完整配置。",
"mode": "lightweight_config",
}
if provider_api != "ollama-generate" and not api_key:
return {
"success": False,
"connected": False,
"message": "当前 provider 未配置 API Key。",
"mode": "lightweight_config",
}
if provider == "opencode-go":
url = _join_provider_url(base_url, "/models")
headers = {"Authorization": f"Bearer {api_key}"}
elif provider_api == "ollama-generate":
url = _join_provider_url(base_url, "/api/tags")
headers: dict[str, str] = {}
elif provider_api == "openai-completions":
url = _join_provider_url(base_url, "/models")
headers = {"Authorization": f"Bearer {api_key}"}
elif provider_api == "anthropic-messages":
url = _join_provider_url(base_url, "/models")
headers = {
"x-api-key": api_key,
"anthropic-version": str(llm_config.get("anthropic_version") or "2023-06-01"),
}
else:
return {
"success": False,
"connected": False,
"message": f"当前 provider_api 不支持轻量连通性测试: {provider_api}",
"mode": "lightweight_unsupported",
}
try:
async with httpx.AsyncClient(timeout=min(timeout_seconds, AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS)) as client:
response = await client.get(url, headers=headers)
response.raise_for_status()
payload = response.json()
except httpx.HTTPStatusError as exc:
detail = exc.response.text or exc.response.reason_phrase
if exc.response.status_code == 404 and _contains_model(preset_models, model):
return {
"success": True,
"connected": True,
"message": "轻量连通性测试通过;当前 provider 不提供可用的模型目录,已按内置模型预设确认。",
"mode": "lightweight_preset",
"provider": provider,
"provider_api": provider_api,
"model": model,
"url": url,
}
return {
"success": False,
"connected": False,
"message": f"轻量连通性测试失败: HTTP {exc.response.status_code} {detail}",
"mode": "lightweight_models",
"url": url,
}
except Exception as exc:
return {
"success": False,
"connected": False,
"message": f"轻量连通性测试失败: {exc}",
"mode": "lightweight_models",
"url": url,
}
model_ids = _extract_model_ids(payload)
if model_ids and not _contains_model(model_ids, model):
if _contains_model(preset_models, model):
return {
"success": True,
"connected": True,
"message": "轻量连通性测试通过provider 模型目录未返回当前别名,已按内置模型预设确认。",
"mode": "lightweight_models_with_preset_alias",
"provider": provider,
"provider_api": provider_api,
"model": model,
"models_count": len(model_ids),
"url": url,
}
return {
"success": False,
"connected": False,
"message": f"连接可用,但模型目录中没有当前模型: {model}",
"mode": "lightweight_models",
"provider": provider,
"model": model,
"models_count": len(model_ids),
"url": url,
}
return {
"success": True,
"connected": True,
"message": "轻量连通性测试通过",
"mode": "lightweight_models",
"provider": provider,
"provider_api": provider_api,
"model": model,
"models_count": len(model_ids),
"url": url,
}
def _web_search_provider_defaults(provider: str) -> dict:
return web_search_provider_defaults(provider).model_dump()
@@ -672,12 +935,21 @@ def _normalize_web_search_payload(web_search_payload: dict | None) -> dict:
}
def _resolve_web_search_api_key(provider: str, provider_config: dict) -> tuple[str, str]:
def _resolve_web_search_api_key(
provider: str,
provider_config: dict,
default_provider: str | None = None,
) -> tuple[str, str]:
saved_key = provider_config.get("api_key") or ""
if saved_key:
return str(saved_key), "runtime"
preset = get_web_search_provider_preset(provider)
return _resolve_web_search_env_secret(preset.get("api_key_env") or "", "WEB_SEARCH_API_KEY")
value, source = _resolve_web_search_env_secret(preset.get("api_key_env") or "")
if value:
return value, source
if normalize_web_search_provider(provider) == normalize_web_search_provider(default_provider or "tavily"):
return _resolve_web_search_env_secret("WEB_SEARCH_API_KEY")
return "", ""
def _build_web_search_payload(
@@ -687,13 +959,22 @@ def _build_web_search_payload(
current_web_search = _normalize_web_search_payload(current_payload.get("web_search") or {})
if update is None:
return current_web_search
provider_id = normalize_web_search_provider(update.default_provider or update.provider)
provider_id = normalize_web_search_provider(update.provider)
default_provider = (
normalize_web_search_provider(update.default_provider)
if update.default_provider is not None
else current_web_search["default_provider"]
)
current_providers = {
provider: dict(config or {})
for provider, config in current_web_search.get("providers", {}).items()
}
current_provider = current_providers.get(provider_id) or _web_search_provider_defaults(provider_id)
current_key, current_key_source = _resolve_web_search_api_key(provider_id, current_provider)
current_key, current_key_source = _resolve_web_search_api_key(
provider_id,
current_provider,
current_web_search["default_provider"],
)
current_key_preview = _mask_secret(current_key, current_key_source)["preview"]
provider_payload = {
**_web_search_provider_defaults(provider_id),
@@ -726,7 +1007,7 @@ def _build_web_search_payload(
current_providers[provider_id] = provider_payload
return {
"enabled": update.enabled,
"default_provider": provider_id,
"default_provider": default_provider,
"providers": current_providers,
}
@@ -735,12 +1016,12 @@ def _runtime_config_from_web_search_payload(web_search_payload: dict) -> WebSear
normalized = _normalize_web_search_payload(web_search_payload)
provider_id = normalized["default_provider"]
provider_config = normalized["providers"].get(provider_id) or _web_search_provider_defaults(provider_id)
api_key, _source = _resolve_web_search_api_key(provider_id, provider_config)
api_key, _source = _resolve_web_search_api_key(provider_id, provider_config, provider_id)
provider_models = {
provider: WebSearchProviderConfig(**{
**config,
"api_key": (
api_key if provider == provider_id else _resolve_web_search_api_key(provider, config)[0]
api_key if provider == provider_id else _resolve_web_search_api_key(provider, config, provider_id)[0]
),
})
for provider, config in normalized["providers"].items()
@@ -874,7 +1155,11 @@ async def serialize_external_integrations(db: AsyncSession) -> dict:
normalized_web_search["providers"].get(provider_id)
or _web_search_provider_defaults(provider_id)
)
api_key, api_key_source = _resolve_web_search_api_key(provider_id, provider_config)
api_key, api_key_source = _resolve_web_search_api_key(
provider_id,
provider_config,
normalized_web_search["default_provider"],
)
web_search_providers_payload[provider_id] = {
**{
key: value
@@ -961,10 +1246,7 @@ async def save_external_integrations_payload(
update: ExternalIntegrationsUpdate,
) -> dict:
current_payload = await get_setting_payload(db, "external_integrations")
current_ai_payload = _normalize_ai_provider_payload(current_payload.get("ai_provider") or {})
ai_payload = _build_ai_provider_payload(current_payload, update.ai_provider)
if _ai_provider_runtime_fingerprint(ai_payload) != _ai_provider_runtime_fingerprint(current_ai_payload):
await _validate_ai_provider_full_connection(ai_payload)
web_search_payload = _build_web_search_payload(current_payload, update.web_search)
ocr_payload = _build_ocr_payload(current_payload, update.ocr)
@@ -1212,6 +1494,47 @@ async def get_external_integrations(
return {"integrations": await serialize_external_integrations(db)}
@router.get("/ai-prompts")
async def get_ai_prompts(
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
prompts = await list_effective_prompts(db)
return {"data": [serialize_effective_prompt(prompt) for prompt in prompts]}
@router.put("/ai-prompts/{task_key}")
async def update_ai_prompt(
task_key: str,
payload: AIPromptUpdate,
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
try:
prompt = await save_prompt_override(
db,
task_key,
system_prompt=payload.system_prompt,
prompt=payload.prompt,
)
except KeyError:
raise HTTPException(status_code=404, detail="未知 AI 提示词任务") from None
return {"data": serialize_effective_prompt(prompt)}
@router.post("/ai-prompts/{task_key}/reset")
async def reset_ai_prompt(
task_key: str,
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
try:
prompt = await reset_prompt_override(db, task_key)
except KeyError:
raise HTTPException(status_code=404, detail="未知 AI 提示词任务") from None
return {"data": serialize_effective_prompt(prompt)}
@router.get("/integrations/barentswatch/connectivity")
async def get_barentswatch_connectivity(
current_user: User = Depends(get_current_user),
@@ -1271,6 +1594,10 @@ async def connect_ai_provider_integration(
db: AsyncSession = Depends(get_db),
):
current_payload = await get_setting_payload(db, "external_integrations")
# Connection testing should validate the provider being edited, not the
# currently saved default provider. This is a transient draft only and is
# intentionally not persisted.
payload = payload.model_copy(update={"default_provider": payload.provider})
draft_ai_payload = _build_ai_provider_payload(current_payload, payload)
runtime_config = _runtime_config_from_ai_payload(draft_ai_payload)
quick_llm_config = {
@@ -1294,22 +1621,16 @@ async def connect_ai_provider_integration(
"message": "AI Provider 可访问,但当前 provider/model/key 未完整配置。",
"status": status_result.model_dump(),
}
probe_result = await client.analyze(
SituationalAnalysisRequest(
title="快速连接测试",
objective="Reply OK.",
observations=[],
constraints=["Output only OK."],
)
lightweight_result = await _check_ai_provider_lightweight(
quick_llm_config,
timeout_seconds=min(
int(runtime_config["timeout_seconds"] or 60),
AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS,
),
)
return {
"success": True,
"connected": True,
"message": "连接测试通过",
**lightweight_result,
"status": status_result.model_dump(),
"provider": probe_result.provider,
"model": probe_result.model,
"mode": "quick_probe",
}
except HTTPException as exc:
return {
@@ -1327,16 +1648,38 @@ async def connect_ai_provider_integration(
@router.get("/integrations/ai-provider/secrets")
async def reveal_ai_provider_secrets(
request: Request,
provider: str = Query(default=""),
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
requested_provider = _normalize_provider_id(provider) if provider else "default"
await _ensure_secret_reveal_allowed(
current_user=current_user,
request=request,
target_id=f"ai_provider:{requested_provider}",
details={"kind": "ai_provider", "provider": requested_provider},
)
current_payload = await get_setting_payload(db, "external_integrations")
ai_payload = _normalize_ai_provider_payload(current_payload.get("ai_provider") or {})
provider_id = _normalize_provider_id(provider or ai_payload["default_provider"])
provider_config = ai_payload["providers"].get(provider_id) or _provider_defaults(provider_id)
api_key, api_key_source = _resolve_provider_api_key(provider_id, provider_config)
service_token, service_token_source = _resolve_service_token(ai_payload)
await _record_integration_secret_reveal(
current_user=current_user,
request=request,
target_id=f"ai_provider:{provider_id}",
result="success",
details={
"kind": "ai_provider",
"provider": provider_id,
"api_key_configured": bool(api_key),
"api_key_source": api_key_source,
"service_token_configured": bool(service_token),
"service_token_source": service_token_source,
},
)
return {
"provider": provider_id,
"api_key": api_key,
@@ -1355,10 +1698,18 @@ async def get_web_search_presets(
@router.get("/integrations/web-search/secrets")
async def reveal_web_search_secrets(
request: Request,
provider: str = Query(default=""),
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
requested_provider = normalize_web_search_provider(provider) if provider else "default"
await _ensure_secret_reveal_allowed(
current_user=current_user,
request=request,
target_id=f"web_search:{requested_provider}",
details={"kind": "web_search", "provider": requested_provider},
)
current_payload = await get_setting_payload(db, "external_integrations")
web_search_payload = _normalize_web_search_payload(current_payload.get("web_search") or {})
provider_id = normalize_web_search_provider(provider or web_search_payload["default_provider"])
@@ -1366,7 +1717,23 @@ async def reveal_web_search_secrets(
web_search_payload["providers"].get(provider_id)
or _web_search_provider_defaults(provider_id)
)
api_key, api_key_source = _resolve_web_search_api_key(provider_id, provider_config)
api_key, api_key_source = _resolve_web_search_api_key(
provider_id,
provider_config,
web_search_payload["default_provider"],
)
await _record_integration_secret_reveal(
current_user=current_user,
request=request,
target_id=f"web_search:{provider_id}",
result="success",
details={
"kind": "web_search",
"provider": provider_id,
"api_key_configured": bool(api_key),
"api_key_source": api_key_source,
},
)
return {
"provider": provider_id,
"api_key": api_key,
@@ -1376,12 +1743,31 @@ async def reveal_web_search_secrets(
@router.get("/integrations/ocr/secrets")
async def reveal_ocr_secrets(
request: Request,
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
await _ensure_secret_reveal_allowed(
current_user=current_user,
request=request,
target_id="ocr:default",
details={"kind": "ocr", "provider": "default"},
)
current_payload = await get_setting_payload(db, "external_integrations")
ocr_payload = _normalize_ocr_payload(current_payload.get("ocr") or {})
api_key, api_key_source = _resolve_ocr_api_key(ocr_payload)
await _record_integration_secret_reveal(
current_user=current_user,
request=request,
target_id=f"ocr:{ocr_payload['provider']}",
result="success",
details={
"kind": "ocr",
"provider": ocr_payload["provider"],
"api_key_configured": bool(api_key),
"api_key_source": api_key_source,
},
)
return {
"provider": ocr_payload["provider"],
"api_key": api_key,
@@ -1485,9 +1871,17 @@ async def get_ai_provider_presets(
async def refresh_ai_provider_preset(
provider: str,
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
try:
return {"data": await refresh_llm_provider_preset(provider)}
provider_id = _normalize_provider_id(provider)
api_key = None
if provider_id == "opencode-go":
current_payload = await get_setting_payload(db, "external_integrations")
ai_payload = _normalize_ai_provider_payload(current_payload.get("ai_provider") or {})
provider_config = ai_payload["providers"].get(provider_id) or _provider_defaults(provider_id)
api_key, _api_key_source = _resolve_provider_api_key(provider_id, provider_config)
return {"data": await refresh_llm_provider_preset(provider_id, api_key=api_key)}
except ValueError as exc:
raise HTTPException(status_code=404, detail=str(exc)) from exc
except Exception as exc:

View File

@@ -8,9 +8,13 @@ from datetime import datetime
from fastapi import APIRouter, Depends, HTTPException, Query, Request, status
from pydantic import BaseModel
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.config import ROOT_DIR
from app.core.security import get_current_user
from app.db.session import get_db
from app.models.system_log import AuditLog, SystemLog
from app.models.user import User
from app.services.persistent_logs import record_audit_log, record_system_log
from app.services.system_control import (
@@ -35,6 +39,7 @@ from app.services.system_logs import (
normalize_log_level,
read_log_snapshot,
)
from app.services.earth_layer_cache import earth_layer_cache
router = APIRouter()
@@ -112,6 +117,17 @@ class EarthClientLogEventResponse(BaseModel):
level: str
class EarthLayerCacheStatusResponse(BaseModel):
prefix: str
key_count: int
memory_bytes: int
layers: dict[str, dict[str, int]]
class EarthLayerCacheClearResponse(BaseModel):
deleted: int
def ensure_super_admin(current_user: User) -> None:
if not require_super_admin(current_user.role):
raise HTTPException(
@@ -132,6 +148,34 @@ def validate_log_date(raw_value: str | None, field_name: str) -> str | None:
) from exc
@router.get("/cache/earth-layers", response_model=EarthLayerCacheStatusResponse)
async def get_earth_layer_cache_status(
current_user: User = Depends(get_current_user),
):
ensure_super_admin(current_user)
try:
return earth_layer_cache.status()
except Exception as exc:
raise HTTPException(
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
detail=f"Unable to read Earth layer cache status: {exc}",
) from exc
@router.delete("/cache/earth-layers", response_model=EarthLayerCacheClearResponse)
async def clear_earth_layer_cache(
current_user: User = Depends(get_current_user),
):
ensure_super_admin(current_user)
try:
return {"deleted": earth_layer_cache.delete_pattern()}
except Exception as exc:
raise HTTPException(
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
detail=f"Unable to clear Earth layer cache: {exc}",
) from exc
@router.post("/restart-tasks", response_model=RestartTaskResponse)
async def create_restart_task(
payload: RestartTaskCreate,
@@ -270,7 +314,120 @@ async def get_system_log_sources(
current_user: User = Depends(get_current_user),
):
ensure_super_admin(current_user)
return {"items": list_log_sources()}
return {
"items": [
*list_log_sources(),
{
"source_id": "system-db",
"name": "系统事件",
"kind": "database",
"location": "table://system_logs",
"description": "后端持久化系统事件、AI 和采集器操作日志。",
"category": "database",
"status": "ok",
},
{
"source_id": "audit-db",
"name": "审计事件",
"kind": "database",
"location": "table://audit_logs",
"description": "管理员敏感操作和密钥 reveal 审计记录。",
"category": "audit",
"status": "ok",
},
]
}
async def read_database_log_snapshot(
source_id: str,
*,
limit: int,
level: str,
levels: str | None,
start_date: str | None,
end_date: str | None,
search: str | None,
db: AsyncSession,
) -> dict | None:
selected_levels = set(normalize_log_level(item) for item in (levels or level).split(",") if item.strip())
selected_levels.discard("all")
search_query = (search or "").strip().lower()
lines: list[str] = []
if source_id == "system-db":
query = select(SystemLog).order_by(SystemLog.occurred_at.desc().nullslast(), SystemLog.id.desc()).limit(limit * 5)
result = await db.execute(query)
records = result.scalars().all()
for record in records:
record_level = normalize_log_level(record.level)
if selected_levels and record_level not in selected_levels:
continue
occurred_at = record.occurred_at.date().isoformat() if record.occurred_at else ""
if start_date and occurred_at and occurred_at < start_date:
continue
if end_date and occurred_at and occurred_at > end_date:
continue
line = " ".join(
part
for part in [
record.occurred_at.isoformat() if record.occurred_at else "",
record_level.upper(),
record.source,
record.event or "",
record.message,
]
if part
)
if search_query and search_query not in line.lower():
continue
lines.append(line)
elif source_id == "audit-db":
query = select(AuditLog).order_by(AuditLog.occurred_at.desc().nullslast(), AuditLog.id.desc()).limit(limit * 5)
result = await db.execute(query)
records = result.scalars().all()
for record in records:
occurred_at = record.occurred_at.date().isoformat() if record.occurred_at else ""
if start_date and occurred_at and occurred_at < start_date:
continue
if end_date and occurred_at and occurred_at > end_date:
continue
line = " ".join(
part
for part in [
record.occurred_at.isoformat() if record.occurred_at else "",
"INFO",
record.action,
record.target_type or "",
record.target_id or "",
record.result or "",
]
if part
)
if search_query and search_query not in line.lower():
continue
lines.append(line)
else:
return None
lines = list(reversed(lines[:limit]))
return {
"source_id": source_id,
"name": "系统事件" if source_id == "system-db" else "审计事件",
"kind": "database",
"location": "table://system_logs" if source_id == "system-db" else "table://audit_logs",
"description": "数据库持久化日志",
"category": "database" if source_id == "system-db" else "audit",
"status": "ok" if lines else "empty",
"level": level,
"selected_levels": sorted(selected_levels),
"search_query": search or "",
"available_levels": ["all", "error", "warning", "info", "debug"],
"daily_markers": [],
"line_limit": limit,
"line_count": len(lines),
"lines": lines,
}
@router.get("/logs/{source_id}", response_model=SystemLogSnapshotResponse)
@@ -283,6 +440,7 @@ async def get_system_log_snapshot(
end_date: str | None = Query(None, description="Filter logs until this date (YYYY-MM-DD)"),
search: str | None = Query(None, description="Case-insensitive substring search"),
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
ensure_super_admin(current_user)
@@ -305,15 +463,26 @@ async def get_system_log_snapshot(
if normalized_start_date and normalized_end_date and normalized_start_date > normalized_end_date:
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="start_date must be earlier than or equal to end_date")
snapshot = read_log_snapshot(
snapshot = await read_database_log_snapshot(
source_id,
limit,
limit=limit,
level=level,
levels=levels,
start_date=normalized_start_date,
end_date=normalized_end_date,
search=search,
db=db,
)
if snapshot is None:
snapshot = read_log_snapshot(
source_id,
limit,
level=level,
levels=levels,
start_date=normalized_start_date,
end_date=normalized_end_date,
search=search,
)
if snapshot is None:
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Log source not found")
return snapshot

View File

@@ -2,7 +2,7 @@
from typing import Optional
from fastapi import APIRouter, Depends, HTTPException, Query
from fastapi import APIRouter, Depends, HTTPException, Query, Response
from sqlalchemy.ext.asyncio import AsyncSession
from app.api.v1.visualization import _parse_bbox, build_vessel_snapshot_response
@@ -23,6 +23,7 @@ async def get_vessel_snapshot(
limit: int = Query(1000, ge=1, le=MAX_SNAPSHOT_LIMIT),
since_minutes: int = Query(60, ge=1, le=1440),
db: AsyncSession = Depends(get_db),
response: Response = None,
):
if not bbox:
raise HTTPException(status_code=400, detail="bbox is required")
@@ -36,4 +37,5 @@ async def get_vessel_snapshot(
type_filter=type,
limit=limit,
since_minutes=since_minutes,
response=response,
)

View File

@@ -56,6 +56,13 @@ from app.services.vessel_ais_aggregation import (
get_vessel_raw_observations,
MAX_SNAPSHOT_LIMIT,
)
from app.services.earth_layer_cache import (
EarthLayerCachePolicy,
earth_layer_cache,
format_bbox_key,
get_or_build_layer_payload,
quantize_bbox,
)
from app.core.logging import get_logger
router = APIRouter()
@@ -69,6 +76,68 @@ TERRAIN_TILE_BATCH_CONCURRENCY = 16
_terrain_tile_cache: OrderedDict[tuple[int, int, int], tuple[bytes, str, dict[str, str]]] = OrderedDict()
VESSEL_NAME_FALLBACK_PATTERN = re.compile(r"^mmsi\s*\d+$", re.IGNORECASE)
VESSEL_SNAPSHOT_LEGACY_FALLBACK_ENABLED = True
SECONDS_PER_MINUTE = 60
BYTES_PER_MIB = 1024 * 1024
CABLE_CACHE_FRESH_SECONDS = 6 * 60 * SECONDS_PER_MINUTE
CABLE_CACHE_STALE_SECONDS = 24 * 60 * SECONDS_PER_MINUTE
SATELLITE_CACHE_FRESH_SECONDS = 15 * SECONDS_PER_MINUTE
SATELLITE_CACHE_STALE_SECONDS = 2 * 60 * SECONDS_PER_MINUTE
COMPUTE_CENTER_CACHE_FRESH_SECONDS = 10 * SECONDS_PER_MINUTE
COMPUTE_CENTER_CACHE_STALE_SECONDS = 60 * SECONDS_PER_MINUTE
BGP_CACHE_FRESH_SECONDS = 60
BGP_EVENT_CACHE_FRESH_SECONDS = 30
BGP_CACHE_STALE_SECONDS = 10 * SECONDS_PER_MINUTE
VESSEL_SNAPSHOT_CACHE_FRESH_SECONDS = 5
VESSEL_SNAPSHOT_CACHE_STALE_SECONDS = 30
CABLE_CACHE_POLICY = EarthLayerCachePolicy(
CABLE_CACHE_FRESH_SECONDS,
CABLE_CACHE_STALE_SECONDS,
max_features=6000,
max_bytes=10 * BYTES_PER_MIB,
)
LANDING_POINT_CACHE_POLICY = EarthLayerCachePolicy(
CABLE_CACHE_FRESH_SECONDS,
CABLE_CACHE_STALE_SECONDS,
max_features=6000,
max_bytes=8 * BYTES_PER_MIB,
)
SATELLITE_CACHE_POLICY = EarthLayerCachePolicy(
SATELLITE_CACHE_FRESH_SECONDS,
SATELLITE_CACHE_STALE_SECONDS,
max_features=8000,
max_bytes=10 * BYTES_PER_MIB,
)
COMPUTE_CENTER_CACHE_POLICY = EarthLayerCachePolicy(
COMPUTE_CENTER_CACHE_FRESH_SECONDS,
COMPUTE_CENTER_CACHE_STALE_SECONDS,
max_features=1000,
max_bytes=4 * BYTES_PER_MIB,
)
BGP_CACHE_POLICY = EarthLayerCachePolicy(
BGP_CACHE_FRESH_SECONDS,
BGP_CACHE_STALE_SECONDS,
max_features=1000,
max_bytes=3 * BYTES_PER_MIB,
)
BGP_EVENT_CACHE_POLICY = EarthLayerCachePolicy(
BGP_EVENT_CACHE_FRESH_SECONDS,
BGP_CACHE_STALE_SECONDS,
max_features=1000,
max_bytes=3 * BYTES_PER_MIB,
)
SUMMARY_CACHE_POLICY = EarthLayerCachePolicy(
BGP_EVENT_CACHE_FRESH_SECONDS,
BGP_CACHE_STALE_SECONDS,
max_features=0,
max_bytes=512 * 1024,
)
VESSEL_SNAPSHOT_CACHE_POLICY = EarthLayerCachePolicy(
VESSEL_SNAPSHOT_CACHE_FRESH_SECONDS,
VESSEL_SNAPSHOT_CACHE_STALE_SECONDS,
max_features=1500,
max_bytes=3 * BYTES_PER_MIB,
)
class TerrariumTileRequest(BaseModel):
@@ -1010,7 +1079,40 @@ async def build_vessel_snapshot_response(
type_filter: str | None,
limit: int | None,
since_minutes: int = 60,
response: Response | None = None,
use_cache: bool = True,
) -> dict[str, Any]:
if use_cache and bbox is not None:
safe_limit_for_key = _safe_vessel_limit(limit)
safe_since_for_key = min(max(int(since_minutes or 60), 1), 1440)
cache_key = earth_layer_cache.key(
"vessels-snapshot",
bbox=format_bbox_key(quantize_bbox(bbox)),
zoom=zoom or "none",
type=type_filter or "all",
limit=safe_limit_for_key,
since=safe_since_for_key,
)
async def build_uncached() -> dict[str, Any]:
return await build_vessel_snapshot_response(
db,
bbox=bbox,
zoom=zoom,
type_filter=type_filter,
limit=limit,
since_minutes=since_minutes,
response=None,
use_cache=False,
)
return await get_or_build_layer_payload(
key=cache_key,
policy=VESSEL_SNAPSHOT_CACHE_POLICY,
builder=build_uncached,
response=response,
)
requested_types = _requested_vessel_types(type_filter)
safe_limit = _safe_vessel_limit(limit)
safe_since_minutes = min(max(int(since_minutes or 60), 1), 1440)
@@ -1444,8 +1546,20 @@ def convert_bgp_incidents_to_geojson(
@router.get("/geo/cables")
async def get_cables_geojson(db: AsyncSession = Depends(get_db)):
async def get_cables_geojson(db: AsyncSession = Depends(get_db), response: Response = None):
"""获取海底电缆 GeoJSON 数据 (LineString)"""
async def build_payload() -> dict[str, Any]:
return await _build_cables_geojson(db)
return await get_or_build_layer_payload(
key=earth_layer_cache.key("cables"),
policy=CABLE_CACHE_POLICY,
builder=build_payload,
response=response,
)
async def _build_cables_geojson(db: AsyncSession) -> dict[str, Any]:
try:
records = await _load_current_collected_data(db, "arcgis_cables")
@@ -1478,7 +1592,19 @@ async def get_cables_geojson(db: AsyncSession = Depends(get_db)):
@router.get("/geo/landing-points")
async def get_landing_points_geojson(db: AsyncSession = Depends(get_db)):
async def get_landing_points_geojson(db: AsyncSession = Depends(get_db), response: Response = None):
async def build_payload() -> dict[str, Any]:
return await _build_landing_points_geojson(db)
return await get_or_build_layer_payload(
key=earth_layer_cache.key("landing-points"),
policy=LANDING_POINT_CACHE_POLICY,
builder=build_payload,
response=response,
)
async def _build_landing_points_geojson(db: AsyncSession) -> dict[str, Any]:
try:
records_by_source = await _load_current_collected_data_by_sources(
db,
@@ -1731,8 +1857,25 @@ async def get_satellites_geojson(
description="Maximum number of satellites to return. Omit for no limit.",
),
db: AsyncSession = Depends(get_db),
response: Response = None,
):
"""获取卫星 TLE GeoJSON 数据"""
async def build_payload() -> dict[str, Any]:
return await _build_satellites_geojson(limit=limit, db=db)
return await get_or_build_layer_payload(
key=earth_layer_cache.key("satellites", limit=limit or "all"),
policy=SATELLITE_CACHE_POLICY,
builder=build_payload,
response=response,
)
async def _build_satellites_geojson(
*,
limit: int | None,
db: AsyncSession,
) -> dict[str, Any]:
records = await _load_current_or_latest_task_data(
db,
"celestrak_tle",
@@ -1800,8 +1943,25 @@ async def get_gpu_clusters_geojson(
async def get_compute_centers_geojson(
limit: int = Query(200, ge=1, le=1000),
db: AsyncSession = Depends(get_db),
response: Response = None,
):
"""获取统一算力中心 GeoJSON 数据"""
async def build_payload() -> dict[str, Any]:
return await _build_compute_centers_geojson(limit=limit, db=db)
return await get_or_build_layer_payload(
key=earth_layer_cache.key("compute-centers", limit=limit),
policy=COMPUTE_CENTER_CACHE_POLICY,
builder=build_payload,
response=response,
)
async def _build_compute_centers_geojson(
*,
limit: int,
db: AsyncSession,
) -> dict[str, Any]:
records_by_source = await _load_current_collected_data_by_sources(
db,
["top500", "epoch_ai_gpu"],
@@ -1982,6 +2142,7 @@ async def collect_compute_center_location(
provider_client=provider_client,
query=query,
entity_type="compute_center",
db=db,
attempted_queries=attempted_queries,
search_evidence=search_result.evidence,
)
@@ -1998,6 +2159,24 @@ async def collect_compute_center_location(
llm_failure_reason = llm_result.failure_reason
if not candidates:
logger.warning_event(
"Compute center location collection returned no candidates",
event="visualization.compute_center.location_collect.completed",
context={
"source_id": source_id,
"record_id": record_id,
"name": name,
"success": False,
"llm_failure_reason": llm_failure_reason,
"attempted_queries": list(attempted_queries),
"context": {
"operator": operator,
"site": site,
"city": city,
"country": country,
},
},
)
return {
"source_id": source_id,
"record_id": record_id,
@@ -2019,13 +2198,34 @@ async def collect_compute_center_location(
},
}
best_candidate = candidates[0].to_dict()
logger.info_event(
"Compute center location collection returned candidates",
event="visualization.compute_center.location_collect.completed",
context={
"source_id": source_id,
"record_id": record_id,
"name": name,
"success": True,
"candidate_count": len(candidates),
"best_candidate": best_candidate,
"llm_failure_reason": llm_failure_reason,
"attempted_queries": list(attempted_queries),
"context": {
"operator": operator,
"site": site,
"city": city,
"country": country,
},
},
)
return {
"source_id": source_id,
"record_id": record_id,
"name": name,
"success": True,
"candidates": [candidate.to_dict() for candidate in candidates],
"best_candidate": candidates[0].to_dict(),
"best_candidate": best_candidate,
"attempted_queries": list(attempted_queries),
"context": {
"name": name,
@@ -2396,7 +2596,31 @@ async def get_bgp_anomalies_geojson(
status: Optional[str] = Query("active"),
limit: int = Query(200, ge=1, le=1000),
db: AsyncSession = Depends(get_db),
response: Response = None,
):
async def build_payload() -> dict[str, Any]:
return await _build_bgp_anomalies_geojson(
severity=severity,
status=status,
limit=limit,
db=db,
)
return await get_or_build_layer_payload(
key=earth_layer_cache.key("bgp-anomalies", severity=severity or "all", status=status or "all", limit=limit),
policy=BGP_EVENT_CACHE_POLICY,
builder=build_payload,
response=response,
)
async def _build_bgp_anomalies_geojson(
*,
severity: str | None,
status: str | None,
limit: int,
db: AsyncSession,
) -> dict[str, Any]:
stmt = select(BGPAnomaly).order_by(BGPAnomaly.created_at.desc()).limit(limit)
if severity:
stmt = stmt.where(BGPAnomaly.severity == severity)
@@ -2416,7 +2640,31 @@ async def get_bgp_incidents_geojson(
status: Optional[str] = Query("active"),
limit: int = Query(100, ge=1, le=500),
db: AsyncSession = Depends(get_db),
response: Response = None,
):
async def build_payload() -> dict[str, Any]:
return await _build_bgp_incidents_geojson(
severity=severity,
status=status,
limit=limit,
db=db,
)
return await get_or_build_layer_payload(
key=earth_layer_cache.key("bgp-incidents", severity=severity or "all", status=status or "all", limit=limit),
policy=BGP_EVENT_CACHE_POLICY,
builder=build_payload,
response=response,
)
async def _build_bgp_incidents_geojson(
*,
severity: str | None,
status: str | None,
limit: int,
db: AsyncSession,
) -> dict[str, Any]:
stmt = select(BGPIncident).order_by(BGPIncident.created_at.desc()).limit(limit)
if severity:
stmt = stmt.where(BGPIncident.severity == severity)
@@ -2431,7 +2679,19 @@ async def get_bgp_incidents_geojson(
@router.get("/geo/bgp-collectors")
async def get_bgp_collectors_geojson(db: AsyncSession = Depends(get_db)):
async def get_bgp_collectors_geojson(db: AsyncSession = Depends(get_db), response: Response = None):
async def build_payload() -> dict[str, Any]:
return await _build_bgp_collectors_geojson(db)
return await get_or_build_layer_payload(
key=earth_layer_cache.key("bgp-collectors"),
policy=BGP_CACHE_POLICY,
builder=build_payload,
response=response,
)
async def _build_bgp_collectors_geojson(db: AsyncSession) -> dict[str, Any]:
coverage = await build_bgp_collector_coverage(
db,
source_filter=("ris_live_bgp", "bgpstream_bgp"),
@@ -2446,8 +2706,20 @@ async def get_bgp_collectors_geojson(db: AsyncSession = Depends(get_db)):
@router.get("/geo/summary")
async def get_visualization_geo_summary(db: AsyncSession = Depends(get_db)):
async def get_visualization_geo_summary(db: AsyncSession = Depends(get_db), response: Response = None):
"""Return lightweight Earth HUD counts without loading layer GeoJSON payloads."""
async def build_payload() -> dict[str, Any]:
return await _build_visualization_geo_summary(db)
return await get_or_build_layer_payload(
key=earth_layer_cache.key("summary"),
policy=SUMMARY_CACHE_POLICY,
builder=build_payload,
response=response,
)
async def _build_visualization_geo_summary(db: AsyncSession) -> dict[str, Any]:
cable_count = await _count_current_or_latest_task_data(db, "arcgis_cables")
landing_point_count = await _count_current_or_latest_task_data(db, "arcgis_landing_points")
satellite_count = await _count_current_or_latest_task_data(

View File

@@ -14,6 +14,7 @@ from app.core.websocket.manager import manager
logger = get_logger(__name__, service="api")
router = APIRouter()
EARTH_UPDATES_CHANNEL = "earth_updates"
async def authenticate_token(token: str) -> Optional[dict]:
@@ -58,7 +59,7 @@ async def websocket_endpoint(
is_anonymous = payload is None
user_id = str(payload.get("sub")) if payload else f"anonymous:{id(websocket)}"
supported_channels = ["vessels", "earth_news"] if is_anonymous else [
supported_channels = ["vessels", "earth_news", EARTH_UPDATES_CHANNEL] if is_anonymous else [
"gpu_clusters",
"submarine_cables",
"ixp_nodes",
@@ -67,6 +68,7 @@ async def websocket_endpoint(
"datasource_tasks",
"vessels",
"earth_news",
EARTH_UPDATES_CHANNEL,
]
await manager.connect(websocket, user_id)

View File

@@ -257,46 +257,6 @@ DEFAULT_DATASOURCES = {
"credential_provider": "aisstream",
"credential_status": "supported",
},
"earth_admin0_boundaries": {
"id": 29,
"name": "Earth Admin-0 Boundaries",
"display_name": "Earth Admin-0 国界源",
"module": "L3",
"priority": "P1",
"frequency_minutes": 10080,
"is_free": True,
"requires_credentials": False,
},
"earth_coastline": {
"id": 30,
"name": "Earth Coastline",
"display_name": "Earth 海岸线源",
"module": "L3",
"priority": "P1",
"frequency_minutes": 10080,
"is_free": True,
"requires_credentials": False,
},
"earth_claim_lines": {
"id": 31,
"name": "Earth Claim Lines",
"display_name": "Earth 主张线源",
"module": "L3",
"priority": "P1",
"frequency_minutes": 10080,
"is_free": True,
"requires_credentials": False,
},
"earth_boundary_tiles": {
"id": 32,
"name": "Earth PMTiles Builder",
"display_name": "Earth PMTiles 构建器",
"module": "L3",
"priority": "P1",
"frequency_minutes": 10080,
"is_free": True,
"requires_credentials": False,
},
"media_news_archive": {
"id": 33,
"name": "Media News Archive",

View File

@@ -1,49 +0,0 @@
"""Default Earth boundary source configuration."""
from __future__ import annotations
from typing import Any
EARTH_BOUNDARY_SOURCE_MAPPING: dict[str, Any] = {
"source": {"items_path": "$.features[*]"},
"fields": {
"source_id": {"path": "$.properties.id", "type": "string"},
"name": {"path": "$.properties.name", "type": "string"},
"geometry": {"path": "$.geometry", "type": "object"},
"properties": {"path": "$.properties", "type": "object"},
},
}
EARTH_BOUNDARY_DEFAULT_SOURCES: dict[str, dict[str, Any]] = {
"earth_admin0_boundaries": {
"endpoint": "https://raw.githubusercontent.com/nvkelso/natural-earth-vector/master/geojson/ne_10m_admin_0_countries.geojson",
"source_kind": "admin0-boundaries",
"license": "Natural Earth public domain",
},
"earth_coastline": {
"endpoint": "https://raw.githubusercontent.com/nvkelso/natural-earth-vector/master/geojson/ne_10m_coastline.geojson",
"source_kind": "coastline",
"license": "Natural Earth public domain",
},
"earth_claim_lines": {
"endpoint": "https://www.arcgis.com/sharing/rest/content/items/faaa1908c3ab43f0823c6fde9f18389c/data",
"source_kind": "claim-lines",
"license": "CC BY 4.0; source item owner mapmakersami",
},
}
def default_earth_boundary_config(name: str) -> dict[str, Any]:
source = EARTH_BOUNDARY_DEFAULT_SOURCES.get(name)
if not source:
return {}
return {
"method": "GET",
"timeout": 120,
"retry": 3,
"target_schema": "earth_boundary_source",
"license": source["license"],
"mapping_json": EARTH_BOUNDARY_SOURCE_MAPPING,
}

View File

@@ -47,18 +47,6 @@ class GenericRecord(BaseModel):
return value
class EarthBoundarySourceRecord(BaseModel):
source_id: str | None = None
source_kind: str = Field(pattern="^(admin0-boundaries|coastline|claim-lines)$")
name: str | None = None
geometry: dict[str, Any] | None = None
properties: dict[str, Any] = Field(default_factory=dict)
artifact_path: str | None = None
sha256: str | None = None
feature_count: int | None = Field(default=None, ge=0)
license: str | None = None
@dataclass(frozen=True)
class TargetField:
name: str
@@ -155,24 +143,6 @@ TARGET_SCHEMAS: dict[str, TargetSchema] = {
TargetField("observed_at", "datetime", False, "观测时间", "2026-04-28T00:00:00Z"),
),
),
"earth_boundary_source": TargetSchema(
key="earth_boundary_source",
label="Earth 边界矢量源",
description="Admin-0 国界、海岸线、主张线等 Earth 边界源数据 artifact。",
destination="collected_data",
model=EarthBoundarySourceRecord,
fields=(
TargetField("source_id", "string", False, "来源侧 ID", "feature-1"),
TargetField("source_kind", "string", True, "边界源类型", "admin0-boundaries"),
TargetField("name", "string", False, "记录名称", "China"),
TargetField("geometry", "object", False, "GeoJSON geometry", {"type": "Polygon", "coordinates": []}),
TargetField("properties", "object", False, "GeoJSON properties", {"ISO_A3": "CHN"}),
TargetField("artifact_path", "string", False, "完整源数据本地 artifact 路径", "data/earth-boundary-sources/earth_admin0_boundaries/abc.geojson"),
TargetField("sha256", "string", False, "源数据 artifact SHA-256", "abc123"),
TargetField("feature_count", "integer", False, "源数据 feature 数", 1),
TargetField("license", "string", False, "源数据许可", "ODbL"),
),
),
}

View File

@@ -8,6 +8,8 @@ from app.core.time import to_iso8601_utc
from app.core.websocket.manager import manager
EARTH_UPDATES_CHANNEL = "earth_updates"
class DataBroadcaster:
"""Periodically broadcasts data to connected WebSocket clients"""
@@ -83,6 +85,10 @@ class DataBroadcaster:
channel=channel,
)
async def broadcast_earth_update(self, data: Dict[str, Any]):
"""Broadcast Earth visualization refresh hints to connected clients."""
await self.broadcast_custom(EARTH_UPDATES_CHANNEL, data)
def enqueue_vessel_update(self, data: Dict[str, Any]):
vessels = data.get("vessels") if isinstance(data, dict) else None
if not isinstance(vessels, list):

View File

@@ -1,6 +1,6 @@
from typing import AsyncGenerator
from sqlalchemy import text
from sqlalchemy import bindparam, text
from sqlalchemy.ext.asyncio import AsyncSession, create_async_engine, async_sessionmaker
from sqlalchemy.orm import declarative_base
@@ -72,6 +72,74 @@ async def seed_default_datasources(session: AsyncSession):
await session.commit()
LEGACY_EARTH_BOUNDARY_SOURCES = (
"earth_admin0_boundaries",
"earth_coastline",
"earth_claim_lines",
"earth_boundary_tiles",
)
LEGACY_EARTH_BOUNDARY_DATATYPES = (
"earth_boundary_source",
"earth_boundary_tiles",
)
LEGACY_EARTH_BOUNDARY_IDS = (29, 30, 31, 32)
async def purge_legacy_earth_boundary_datasources(session: AsyncSession) -> None:
source_names = tuple(LEGACY_EARTH_BOUNDARY_SOURCES)
source_ids = tuple(LEGACY_EARTH_BOUNDARY_IDS)
data_types = tuple(LEGACY_EARTH_BOUNDARY_DATATYPES)
await session.execute(
text(
"""
DELETE FROM datasource_mapping_templates
WHERE target_schema IN :data_types
OR datasource_config_id IN (
SELECT id FROM datasource_configs WHERE name IN :source_names
)
"""
).bindparams(bindparam("source_names", expanding=True), bindparam("data_types", expanding=True)),
{"source_names": list(source_names), "data_types": list(data_types)},
)
await session.execute(
text("DELETE FROM datasource_configs WHERE name IN :source_names").bindparams(
bindparam("source_names", expanding=True)
),
{"source_names": list(source_names)},
)
await session.execute(
text(
"""
DELETE FROM collected_data
WHERE source IN :source_names OR data_type IN :data_types
"""
).bindparams(bindparam("source_names", expanding=True), bindparam("data_types", expanding=True)),
{"source_names": list(source_names), "data_types": list(data_types)},
)
await session.execute(
text(
"""
DELETE FROM data_snapshots
WHERE source IN :source_names OR datasource_id IN :source_ids
"""
).bindparams(bindparam("source_names", expanding=True), bindparam("source_ids", expanding=True)),
{"source_names": list(source_names), "source_ids": list(source_ids)},
)
await session.execute(
text("DELETE FROM collection_tasks WHERE datasource_id IN :source_ids").bindparams(
bindparam("source_ids", expanding=True)
),
{"source_ids": list(source_ids)},
)
await session.execute(
text("DELETE FROM data_sources WHERE source IN :source_names OR id IN :source_ids").bindparams(
bindparam("source_names", expanding=True), bindparam("source_ids", expanding=True)
),
{"source_names": list(source_names), "source_ids": list(source_ids)},
)
await session.commit()
DEFAULT_LOGIN_USERS = (
{
"username": "admin",
@@ -82,7 +150,7 @@ DEFAULT_LOGIN_USERS = (
{
"username": "linkong",
"email": "linkong@planet.local",
"password": "12345678",
"password": "LK12345678",
"role": "super_admin",
},
)
@@ -203,6 +271,18 @@ async def init_db():
"""
)
)
await conn.execute(
text(
"""
ALTER TABLE earth_news_items
ADD COLUMN IF NOT EXISTS content_language VARCHAR(32) NOT NULL DEFAULT 'en',
ADD COLUMN IF NOT EXISTS localizations JSONB NOT NULL DEFAULT '{}'::jsonb,
ADD COLUMN IF NOT EXISTS enrichment_status VARCHAR(80) NOT NULL DEFAULT 'pending',
ADD COLUMN IF NOT EXISTS enrichment_error TEXT,
ADD COLUMN IF NOT EXISTS enriched_at TIMESTAMPTZ
"""
)
)
await conn.execute(
text(
"""
@@ -211,6 +291,22 @@ async def init_db():
"""
)
)
await conn.execute(
text(
"""
CREATE INDEX IF NOT EXISTS idx_earth_news_enrichment_status
ON earth_news_items (enrichment_status)
"""
)
)
await conn.execute(
text(
"""
CREATE INDEX IF NOT EXISTS idx_earth_news_enriched_at
ON earth_news_items (enriched_at)
"""
)
)
await conn.execute(
text(
"""
@@ -285,4 +381,5 @@ async def init_db():
await seed_default_bgp_collector_locations(session)
await seed_compute_center_locations_from_source_coords(session)
await seed_default_datasources(session)
await purge_legacy_earth_boundary_datasources(session)
await ensure_default_admin_user(session)

View File

@@ -1,8 +1,10 @@
from contextlib import asynccontextmanager
from pathlib import Path
from uuid import uuid4
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from fastapi.staticfiles import StaticFiles
from starlette.middleware.base import BaseHTTPMiddleware
from app.api.main import api_router
@@ -88,6 +90,14 @@ app.add_middleware(WebSocketCORSMiddleware)
app.include_router(api_router, prefix="/api/v1")
app.include_router(websocket.router)
EARTH_BRAND_ASSET_DIR = Path(__file__).resolve().parents[2] / "data" / "earth-brand"
EARTH_BRAND_ASSET_DIR.mkdir(parents=True, exist_ok=True)
app.mount(
"/earth-brand-assets",
StaticFiles(directory=str(EARTH_BRAND_ASSET_DIR)),
name="earth-brand-assets",
)
@app.get("/health")
async def health_check():

View File

@@ -10,6 +10,8 @@ class EarthNewsItem(Base):
id = Column(String(160), primary_key=True)
title = Column(String(500), nullable=False)
summary = Column(Text, nullable=False, default="")
content_language = Column(String(32), nullable=False, default="en")
localizations = Column(JSON, nullable=False, default=dict)
url = Column(Text, nullable=False)
source = Column(String(255), nullable=False, default="")
feed_name = Column(String(255), nullable=False, default="")
@@ -27,6 +29,9 @@ class EarthNewsItem(Base):
first_seen_at = Column(DateTime(timezone=True), server_default=func.now(), nullable=False)
last_seen_at = Column(DateTime(timezone=True), server_default=func.now(), nullable=False, index=True)
resolved_at = Column(DateTime(timezone=True), nullable=True, index=True)
enrichment_status = Column(String(80), nullable=False, default="pending", index=True)
enrichment_error = Column(Text, nullable=True)
enriched_at = Column(DateTime(timezone=True), nullable=True, index=True)
updated_at = Column(DateTime(timezone=True), server_default=func.now(), onupdate=func.now())
__table_args__ = (

View File

@@ -13,10 +13,11 @@ class AIContentBlock(BaseModel):
class SituationalAnalysisRequest(BaseModel):
title: str = Field(..., min_length=1, max_length=200)
objective: str = Field(..., min_length=1, max_length=1000)
objective: str = Field(..., min_length=1, max_length=20000)
context: dict[str, Any] = Field(default_factory=dict)
observations: list[str] = Field(default_factory=list)
constraints: list[str] = Field(default_factory=list)
system_prompt: str | None = Field(default=None, max_length=8000)
preferred_model: str | None = Field(default=None, max_length=200)
thinking: dict[str, Any] | None = None

View File

@@ -1,6 +1,7 @@
from __future__ import annotations
import asyncio
import json
import httpx
from fastapi import Depends, HTTPException, status
@@ -57,6 +58,9 @@ class AIProviderClient:
value = self.llm_config.get(key)
if value not in (None, ""):
headers[header_name] = str(value)
model_provider_apis = self.llm_config.get("model_provider_apis")
if isinstance(model_provider_apis, dict) and model_provider_apis:
headers["X-AI-Model-Provider-APIs"] = json.dumps(model_provider_apis)
return headers
async def get_status(self, request_id: str | None = None) -> AIProviderStatusResponse:

View File

@@ -8,6 +8,9 @@ from sqlalchemy.ext.asyncio import AsyncSession
from app.models.alert import Alert, AlertSeverity, AlertStatus
from app.schemas.ai import AlertBriefRequest, SituationalAnalysisRequest
from app.ai_tasks.prompts import get_effective_prompt
ALERT_BRIEF_PROMPT_KEY = "alerts.brief"
def _format_counter(counter: Counter[str], empty_text: str = "") -> str:
@@ -84,11 +87,13 @@ async def build_alert_brief_request(
"top_datasources": dict(datasource_counts.most_common(6)),
"top_active_datasources": dict(active_datasource_counts.most_common(5)),
}
prompt = await get_effective_prompt(db, ALERT_BRIEF_PROMPT_KEY)
return (
SituationalAnalysisRequest(
title="告警态势 AI 简报",
objective="基于当前告警总量、严重度、状态、数据源分布与最近告警摘录,生成一份面向值班人员的简明告警态势简报,突出待处理风险、告警集中点和优先动作。",
objective=prompt.prompt,
system_prompt=prompt.system_prompt or None,
observations=facts,
constraints=[
"明确区分事实、推断与建议。",

View File

@@ -11,9 +11,12 @@ from app.models.bgp_anomaly import BGPAnomaly
from app.models.bgp_incident import BGPIncident
from app.models.bgp_observation import BGPObservation
from app.schemas.ai import SituationalAnalysisRequest
from app.ai_tasks.prompts import get_effective_prompt
from app.services.bgp_collectors import build_bgp_collector_coverage
from app.services.bgp_enrichment import lookup_prefix_geography
BGP_BRIEF_PROMPT_KEY = "bgp.brief"
def _format_counter(counter: dict[str, int], empty_text: str = "") -> str:
if not counter:
@@ -243,12 +246,15 @@ async def build_bgp_brief_request(
for prefix, item in list(prefix_geographies.items())[:8]
},
}
prompt = await get_effective_prompt(db, BGP_BRIEF_PROMPT_KEY)
return SituationalAnalysisRequest(
title="BGP 态势 AI 简报",
objective="基于当前 BGP incidents、anomalies、原始观测事件、观测站覆盖与 prefix geography 证据,生成一份面向操作员的简明态势简报,突出区域热点、观测偏差、当前风险、证据和优先动作。",
objective=prompt.prompt,
system_prompt=prompt.system_prompt or None,
observations=observations_lines,
constraints=[
"直接输出中文 Markdown 简报正文,不要输出英文写作计划、提示词复述、字段说明或元评论。",
"明确区分事实、推断与建议。",
"优先指出需要立即关注的高严重度 incident 或异常模式。",
"需要单独指出哪些区域结论来自 prefix geography / affected regions哪些可能受 collector coverage 偏差影响。",

View File

@@ -39,12 +39,6 @@ from app.services.collectors.news_live_streams import NewsLiveStreamsCollector
from app.services.collectors.media_news_archive import MediaNewsArchiveCollector
from app.services.collectors.aisstream import AISStreamCollector
from app.services.collectors.vessel_ais import VesselAISCollector
from app.services.collectors.earth_boundaries import (
EarthAdmin0BoundaryCollector,
EarthBoundaryTileCollector,
EarthClaimLinesCollector,
EarthCoastlineCollector,
)
collector_registry.register(TOP500Collector())
collector_registry.register(EpochAIGPUCollector())
@@ -75,10 +69,6 @@ collector_registry.register(NewsLiveStreamsCollector())
collector_registry.register(MediaNewsArchiveCollector())
collector_registry.register(VesselAISCollector())
collector_registry.register(AISStreamCollector())
collector_registry.register(EarthAdmin0BoundaryCollector())
collector_registry.register(EarthCoastlineCollector())
collector_registry.register(EarthClaimLinesCollector())
collector_registry.register(EarthBoundaryTileCollector())
__all__ = [
"BaseCollector",
@@ -115,8 +105,4 @@ __all__ = [
"MediaNewsArchiveCollector",
"VesselAISCollector",
"AISStreamCollector",
"EarthAdmin0BoundaryCollector",
"EarthCoastlineCollector",
"EarthClaimLinesCollector",
"EarthBoundaryTileCollector",
]

View File

@@ -9,10 +9,37 @@ from sqlalchemy import select, text
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.collected_data_fields import build_dynamic_metadata, get_record_field
from app.core.config import settings
from app.core.countries import normalize_country
from app.core.time import to_iso8601_utc
from app.core.websocket.broadcaster import broadcaster
from app.services.earth_layer_cache import invalidate_earth_layer_cache_for_source
EARTH_UPDATE_LAYER_HINTS: dict[str, list[str]] = {
"ris_live_bgp": ["bgp"],
"bgpstream_bgp": ["bgp"],
"top500_supercomputers": ["computeCenters"],
"epoch_ai_gpu": ["computeCenters"],
"huggingface_models": ["computeCenters"],
"huggingface_datasets": ["computeCenters"],
"huggingface_spaces": ["computeCenters"],
"telegeography_cables": ["cables"],
"telegeography_landing_points": ["cables"],
"telegeography_cable_systems": ["cables"],
"arcgis_cables": ["cables"],
"fao_landing_points": ["cables"],
"arcgis_landing_points": ["cables"],
"arcgis_cable_landing_relations": ["cables"],
"spacetrack_tle": ["satellites"],
"celestrak_tle": ["satellites"],
"barentswatch_vessels": ["vessels"],
"aisstream_vessels": ["vessels"],
"news_live_streams": ["media"],
"media_news_archive": ["news"],
}
def get_earth_update_layers_for_source(source: str) -> list[str]:
return EARTH_UPDATE_LAYER_HINTS.get(source, [])
class BaseCollector(ABC):
@@ -69,6 +96,29 @@ class BaseCollector(ABC):
)
self._last_broadcast_progress = rounded_progress
async def _publish_earth_update(
self,
*,
action: str,
records_processed: int,
task_id: int | None = None,
) -> None:
layers = get_earth_update_layers_for_source(self.name)
if not layers:
return
await broadcaster.broadcast_earth_update(
{
"action": action,
"source": self.name,
"data_type": self.data_type,
"layers": layers,
"datasource_id": getattr(self, "_datasource_id", None),
"task_id": task_id,
"records_processed": records_processed,
"timestamp": to_iso8601_utc(datetime.now(UTC)),
}
)
async def update_progress(self, records_processed: int, *, commit: bool = False, force: bool = False):
"""Update task progress - call this during data processing"""
if self._current_task and self._db_session:
@@ -186,7 +236,7 @@ class BaseCollector(ABC):
result = await db.execute(
select(DataSnapshot)
.where(DataSnapshot.source == self.name, DataSnapshot.is_current == True)
.where(DataSnapshot.source == self.name, DataSnapshot.is_current.is_(True))
.order_by(DataSnapshot.completed_at.desc().nullslast(), DataSnapshot.id.desc())
.limit(1)
)
@@ -324,6 +374,11 @@ class BaseCollector(ABC):
task.completed_at = datetime.now(UTC)
await db.commit()
await self._publish_task_update(force=True)
await self._publish_earth_update(
action="collector_completed",
records_processed=records_count,
task_id=task_id,
)
return {
"status": "success",
@@ -405,7 +460,7 @@ class BaseCollector(ABC):
select(CollectedData)
.where(
CollectedData.source == self.name,
CollectedData.is_current == True,
CollectedData.is_current.is_(True),
)
.order_by(CollectedData.entity_key.asc(), CollectedData.collected_at.desc().nullslast(), CollectedData.id.desc())
)
@@ -531,6 +586,7 @@ class BaseCollector(ABC):
}
await db.commit()
invalidate_earth_layer_cache_for_source(self.name)
await self.update_progress(len(data), force=True)
return records_added

View File

@@ -19,6 +19,7 @@ from app.services.bgp_collector_locations import (
)
from app.services.bgp_event_locations import resolve_bgp_event_geo_dict
from app.services.bgp_incidents import create_bgp_incidents_for_anomalies
from app.services.earth_layer_cache import invalidate_earth_layer_cache_for_source
from app.services.bgp_detectors import (
detect_mass_withdrawal_anomalies,
detect_more_specific_burst_anomalies,
@@ -223,6 +224,8 @@ async def save_bgp_observations_for_batch(
if created:
await db.commit()
for source in {"ris_live_bgp", "bgpstream_bgp"}:
invalidate_earth_layer_cache_for_source(source)
return created

View File

@@ -1,578 +0,0 @@
"""Earth boundary source and static tile collector."""
from __future__ import annotations
import asyncio
import hashlib
import json
import shutil
import sys
from pathlib import Path
from types import SimpleNamespace
from typing import Any
import httpx
from sqlalchemy import select
from app.core.earth_boundary_defaults import EARTH_BOUNDARY_DEFAULT_SOURCES, default_earth_boundary_config
from app.models.collected_data import CollectedData
from app.models.datasource_config import DataSourceConfig
from app.services.collectors.base import BaseCollector
from app.services.custom_datasource_runtime import build_query_params, build_request_headers
REPO_ROOT = Path(__file__).resolve().parents[4]
SOURCE_OUTPUT_DIR = REPO_ROOT / "data/earth-boundary-sources"
SOURCE_MANIFEST_PATH = SOURCE_OUTPUT_DIR / "manifest.json"
BOUNDARY_OUTPUT_DIR = REPO_ROOT / "frontend/public/earth/data/boundaries/v1"
BOUNDARY_MANIFEST_PATH = BOUNDARY_OUTPUT_DIR / "manifest.json"
PMTILES_ARTIFACT_PATH = REPO_ROOT / "frontend/public/earth/data/boundaries/earth-boundaries-china-pov-v1.pmtiles"
POV_POLICY_PATH = REPO_ROOT / "config/earth-boundary-pov-policy.china-v1.json"
BUILD_CONFIG = {
"builder": "scripts/build_earth_boundary_china_pov_geojson.py",
"format": "geojson-high-precision",
"production_target": "geojson-high-precision",
}
EARTH_BOUNDARY_SOURCE_COLLECTORS = {
"earth_admin0_boundaries": "admin0-boundaries",
"earth_coastline": "coastline",
"earth_claim_lines": "claim-lines",
}
def _read_json(path: Path) -> dict[str, Any]:
if not path.exists():
return {}
with path.open("r", encoding="utf-8") as f:
return json.load(f)
def _directory_stats(path: Path) -> dict[str, int]:
if not path.exists():
return {"file_count": 0, "size_bytes": 0}
files = [item for item in path.rglob("*") if item.is_file()]
return {
"file_count": len(files),
"size_bytes": sum(item.stat().st_size for item in files),
}
def _stable_json_hash(payload: Any) -> str:
encoded = json.dumps(payload, ensure_ascii=False, sort_keys=True, separators=(",", ":"))
return hashlib.sha256(encoded.encode("utf-8")).hexdigest()
def _build_input_hash(source_manifest: dict[str, Any]) -> str:
return _stable_json_hash(
{
"source_manifest_schema": source_manifest.get("schema"),
"sources": [
{
"id": source.get("id"),
"sha256": source.get("sha256"),
"kind": source.get("kind"),
"pov": source.get("pov"),
}
for source in source_manifest.get("sources", [])
],
"pov_policy": source_manifest.get("povPolicy"),
"build_config": BUILD_CONFIG,
}
)
def _has_current_artifacts(boundary_manifest: dict[str, Any], build_input_hash: str) -> bool:
if not boundary_manifest:
return False
if boundary_manifest.get("buildInputHash") != build_input_hash:
return False
if not BOUNDARY_OUTPUT_DIR.exists():
return False
if boundary_manifest.get("tileProvider") == "pmtiles-mvt":
return PMTILES_ARTIFACT_PATH.exists()
return (BOUNDARY_OUTPUT_DIR / "base.geojson").exists() and (
BOUNDARY_OUTPUT_DIR / "hover-index.geojson"
).exists()
def _sha256_bytes(payload: bytes) -> str:
return hashlib.sha256(payload).hexdigest()
def _json_feature_count(payload: Any) -> int:
if isinstance(payload, dict) and isinstance(payload.get("features"), list):
return len(payload["features"])
if isinstance(payload, list):
return len(payload)
return 1 if payload else 0
def _json_sample_properties(payload: Any) -> dict[str, Any]:
feature = None
if isinstance(payload, dict) and isinstance(payload.get("features"), list) and payload["features"]:
feature = payload["features"][0]
elif isinstance(payload, list) and payload:
feature = payload[0]
elif isinstance(payload, dict):
feature = payload
if not isinstance(feature, dict):
return {}
props = feature.get("properties") if isinstance(feature.get("properties"), dict) else feature
return {str(key): value for key, value in list(props.items())[:20]}
def _artifact_extension(endpoint: str, content_type: str, payload: bytes) -> str:
suffix = Path(endpoint.split("?", 1)[0]).suffix.lower()
if suffix in {".json", ".geojson", ".zip", ".pbf"}:
return suffix
if "geo+json" in content_type or b'"FeatureCollection"' in payload[:4096]:
return ".geojson"
if "json" in content_type:
return ".json"
return ".dat"
async def _load_datasource_config(db, name: str) -> DataSourceConfig | None:
if db is None:
return None
result = await db.execute(
select(DataSourceConfig)
.where(DataSourceConfig.name == name)
.where(DataSourceConfig.is_active.is_(True))
.order_by(DataSourceConfig.id.desc())
.limit(1)
)
return result.scalar_one_or_none()
async def _latest_boundary_source_record(db, source_name: str) -> CollectedData | None:
result = await db.execute(
select(CollectedData)
.where(CollectedData.source == source_name)
.where(CollectedData.data_type == "earth_boundary_source")
.where(CollectedData.is_current.is_(True))
.order_by(CollectedData.collected_at.desc().nullslast(), CollectedData.id.desc())
.limit(1)
)
return result.scalar_one_or_none()
class EarthBoundarySourceCollector(BaseCollector):
priority = "P1"
module = "L3"
frequency_hours = 168
data_type = "earth_boundary_source"
fail_on_empty = True
source_kind = "unknown"
def _default_config(self):
source = EARTH_BOUNDARY_DEFAULT_SOURCES.get(self.name)
if not source:
return None
return SimpleNamespace(
name=self.name,
description=f"内置默认源:{self.name}",
endpoint=source["endpoint"],
source_type="http",
auth_type="none",
auth_config={},
headers={},
config=default_earth_boundary_config(self.name),
)
async def _download_payload(self, config: DataSourceConfig) -> tuple[bytes, str]:
request_config = config.config or {}
method = str(request_config.get("method") or request_config.get("request_method") or "GET").upper()
if method not in {"GET", "POST"}:
raise RuntimeError("Earth boundary source collectors support GET and POST only")
endpoint = str(config.endpoint or "").strip()
if not endpoint:
raise RuntimeError(
f"{self.name} requires an endpoint in Collector Settings before it can collect data"
)
if endpoint.startswith("file://") or Path(endpoint).expanduser().exists():
path = Path(endpoint.removeprefix("file://")).expanduser()
return path.read_bytes(), "application/octet-stream"
headers = build_request_headers(config.auth_type, config.auth_config or {}, config.headers or {})
params = build_query_params(config.auth_type, config.auth_config or {}, request_config)
timeout = float(request_config.get("timeout", 120))
json_body = request_config.get("json_body") or request_config.get("body")
async with httpx.AsyncClient(timeout=timeout, follow_redirects=True) as client:
response = await client.request(
method,
endpoint,
headers=headers,
params=params or None,
json=json_body if isinstance(json_body, (dict, list)) else None,
)
response.raise_for_status()
return response.content, response.headers.get("content-type", "")
async def fetch(self) -> list[dict[str, Any]]:
db = getattr(self, "_db_session", None)
config = await _load_datasource_config(db, self.name) or self._default_config()
if config is None:
raise RuntimeError(
f"{self.name} has no active Collector Settings config and no built-in default source."
)
await self.set_phase("fetching_source", message=f"正在下载 {self.source_kind} 源数据")
payload, content_type = await self._download_payload(config)
sha256 = _sha256_bytes(payload)
endpoint = str(config.endpoint or "")
extension = _artifact_extension(endpoint, content_type, payload)
source_dir = SOURCE_OUTPUT_DIR / self.name
source_dir.mkdir(parents=True, exist_ok=True)
artifact_path = source_dir / f"{sha256}{extension}"
artifact_path.write_bytes(payload)
parsed: Any = None
if extension in {".json", ".geojson"}:
parsed = json.loads(payload.decode("utf-8"))
feature_count = _json_feature_count(parsed)
if feature_count <= 0:
raise RuntimeError(f"{self.name} downloaded data but found no JSON/GeoJSON features")
config_body = config.config or {}
target_schema = str(config_body.get("target_schema") or "earth_boundary_source")
if target_schema != "earth_boundary_source":
raise RuntimeError(f"{self.name} target_schema must be earth_boundary_source")
await self.update_phase_progress(
current=1,
total=1,
unit="artifact",
message=f"已保存 {feature_count}{self.source_kind} feature",
progress=100,
commit=True,
force=True,
)
relative_artifact_path = str(artifact_path.relative_to(REPO_ROOT))
return [
{
"id": f"{self.source_kind}:{sha256}",
"name": config.description or self.name,
"description": f"Earth boundary source artifact collected from configured endpoint",
"source_kind": self.source_kind,
"source_id": sha256,
"value": feature_count,
"unit": "features",
"metadata": {
"target_schema": target_schema,
"source_kind": self.source_kind,
"endpoint": endpoint,
"method": str(config_body.get("method") or config_body.get("request_method") or "GET").upper(),
"artifact_path": relative_artifact_path,
"sha256": sha256,
"feature_count": feature_count,
"size_bytes": len(payload),
"content_type": content_type,
"license": config_body.get("license"),
"mapping_json": config_body.get("mapping_json"),
"sample_properties": _json_sample_properties(parsed),
},
}
]
class EarthAdmin0BoundaryCollector(EarthBoundarySourceCollector):
name = "earth_admin0_boundaries"
source_kind = "admin0-boundaries"
class EarthCoastlineCollector(EarthBoundarySourceCollector):
name = "earth_coastline"
source_kind = "coastline"
class EarthClaimLinesCollector(EarthBoundarySourceCollector):
name = "earth_claim_lines"
source_kind = "claim-lines"
class EarthBoundaryTileCollector(BaseCollector):
name = "earth_boundary_tiles"
priority = "P1"
module = "L3"
frequency_hours = 168
data_type = "earth_boundary_tiles"
fail_on_empty = False
async def _run_step(self, args: list[str], *, allow_failure: bool = False) -> dict[str, Any]:
process = await asyncio.create_subprocess_exec(
sys.executable,
*args,
cwd=REPO_ROOT,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
stdout_bytes, stderr_bytes = await process.communicate()
stdout = stdout_bytes.decode("utf-8", errors="replace").strip()
stderr = stderr_bytes.decode("utf-8", errors="replace").strip()
if process.returncode != 0 and not allow_failure:
raise RuntimeError(stderr or stdout or f"command failed: {' '.join(args)}")
payload: dict[str, Any] = {
"stdout": stdout,
"stderr": stderr,
"returncode": process.returncode,
}
last_line = stdout.splitlines()[-1:] or []
if last_line:
try:
payload["result"] = json.loads(last_line[0])
except json.JSONDecodeError:
payload["result"] = last_line[0]
return payload
async def fetch(self) -> list[dict[str, Any]]:
db = getattr(self, "_db_session", None)
if db is None:
raise RuntimeError("Earth PMTiles builder requires an active database session")
await self.set_phase("checking_sources", message="正在检查三类 Earth 边界源")
await self.update_phase_progress(
current=0,
total=3,
unit="steps",
message="读取 admin0 / coastline / claim-lines 最新采集结果",
progress=5,
commit=True,
force=True,
)
source_records: dict[str, CollectedData] = {}
missing_sources: list[str] = []
for source_name in EARTH_BOUNDARY_SOURCE_COLLECTORS:
record = await _latest_boundary_source_record(db, source_name)
if record is None:
missing_sources.append(source_name)
else:
source_records[source_name] = record
if missing_sources:
missing_text = ", ".join(missing_sources)
raise RuntimeError(f"未就绪:缺少 {missing_text}。不会更新 Earth 国界。")
sources = []
for source_name, record in source_records.items():
metadata = record.extra_data or {}
artifact_path = metadata.get("artifact_path")
if not artifact_path or not (REPO_ROOT / str(artifact_path)).exists():
missing_sources.append(f"{source_name}: artifact missing")
continue
sources.append(
{
"id": source_name,
"kind": metadata.get("source_kind") or EARTH_BOUNDARY_SOURCE_COLLECTORS[source_name],
"path": str(artifact_path),
"sha256": metadata.get("sha256"),
"featureCount": metadata.get("feature_count"),
"license": metadata.get("license"),
}
)
if missing_sources:
missing_text = ", ".join(missing_sources)
raise RuntimeError(f"未就绪:{missing_text}。不会更新 Earth 国界。")
source_manifest = {
"schema": "planet-earth-boundary-sources/v2",
"sources": sources,
"povPolicy": _read_json(POV_POLICY_PATH),
}
SOURCE_OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
with SOURCE_MANIFEST_PATH.open("w", encoding="utf-8") as f:
json.dump(source_manifest, f, ensure_ascii=False, indent=2)
f.write("\n")
build_input_hash = _build_input_hash(source_manifest)
await self.update_phase_progress(
current=1,
total=3,
unit="steps",
message="三类边界源已就绪",
progress=35,
commit=True,
force=True,
)
await self.set_phase("checking_readiness", message="正在检查 Earth 国界构建就绪状态")
missing_tools = [tool for tool in ("tippecanoe", "pmtiles") if shutil.which(tool) is None]
readiness_result = {
"returncode": 0,
"result": {
"ready": True,
"failures": [f"external tool not found in PATH: {tool}" for tool in missing_tools],
"sources": sources,
"artifact": str(PMTILES_ARTIFACT_PATH.relative_to(REPO_ROOT)),
"fallback_builder": "geojson-high-precision" if missing_tools else None,
},
}
await self.set_phase("building_tiles", message="正在检查 Earth 国界瓦片产物")
boundary_manifest = _read_json(BOUNDARY_MANIFEST_PATH)
build_skipped = _has_current_artifacts(boundary_manifest, build_input_hash)
if build_skipped:
build_result = {
"result": {
"status": "unchanged",
"reason": "source manifest and build config hash unchanged",
"buildInputHash": build_input_hash,
}
}
elif shutil.which("tippecanoe") and shutil.which("pmtiles"):
admin0 = next(source for source in sources if source["kind"] == "admin0-boundaries")
coastline = next(source for source in sources if source["kind"] == "coastline")
claim_lines = next(source for source in sources if source["kind"] == "claim-lines")
build_result = await self._run_step(
[
"scripts/build_earth_boundary_pmtiles.py",
"--admin0-source",
admin0["path"],
"--coastline-source",
coastline["path"],
"--claims-source",
claim_lines["path"],
"--output",
str(PMTILES_ARTIFACT_PATH.relative_to(REPO_ROOT)),
"--manifest",
str(BOUNDARY_MANIFEST_PATH.relative_to(REPO_ROOT)),
"--build-input-hash",
build_input_hash,
"--pov-policy",
str(POV_POLICY_PATH.relative_to(REPO_ROOT)),
]
)
boundary_manifest = _read_json(BOUNDARY_MANIFEST_PATH)
else:
admin0 = next(source for source in sources if source["kind"] == "admin0-boundaries")
coastline = next(source for source in sources if source["kind"] == "coastline")
claim_lines = next(source for source in sources if source["kind"] == "claim-lines")
build_result = await self._run_step(
[
"scripts/build_earth_boundary_china_pov_geojson.py",
"--admin0-source",
admin0["path"],
"--coastline-source",
coastline["path"],
"--claims-source",
claim_lines["path"],
"--output-dir",
str(BOUNDARY_OUTPUT_DIR.relative_to(REPO_ROOT)),
"--build-input-hash",
build_input_hash,
]
)
boundary_manifest = _read_json(BOUNDARY_MANIFEST_PATH)
boundary_stats = _directory_stats(BOUNDARY_OUTPUT_DIR)
pmtiles_exists = PMTILES_ARTIFACT_PATH.exists()
pmtiles_size = PMTILES_ARTIFACT_PATH.stat().st_size if pmtiles_exists else 0
await self.update_phase_progress(
current=2,
total=3,
unit="steps",
message=(
"边界源未变化,已跳过瓦片重建"
if build_skipped
else f"已生成 {boundary_manifest.get('tileProvider') or 'boundary'} 国界产物"
),
progress=80,
commit=True,
force=True,
)
await self.set_phase("indexing_artifacts", message="正在登记边界瓦片产物")
tile_counts = boundary_manifest.get("tiles", {}).get("countsByZoom", {})
records = [
{
"id": "source-manifest",
"name": "Earth boundary source manifest",
"description": "Offline source collection manifest for Earth boundary tiles",
"value": len(source_manifest.get("sources", [])),
"unit": "sources",
"metadata": {
"manifest_path": str(SOURCE_MANIFEST_PATH.relative_to(REPO_ROOT)),
"manifest": source_manifest,
"collector_result": {
"status": "loaded_from_collected_data",
"sources": [source["id"] for source in sources],
},
},
},
{
"id": "production-readiness",
"name": "Earth boundary production readiness",
"description": "Checks whether all source artifacts and boundary build tooling are available",
"value": 1 if readiness_result.get("returncode") == 0 else 0,
"unit": "ready",
"metadata": {
"result": readiness_result.get("result"),
"returncode": readiness_result.get("returncode"),
},
},
{
"id": "boundary-manifest",
"name": "Earth boundary tile manifest",
"description": "Versioned static vector tile manifest for Earth country boundaries",
"value": boundary_stats["file_count"],
"unit": "files",
"metadata": {
"manifest_path": str(BOUNDARY_MANIFEST_PATH.relative_to(REPO_ROOT)),
"output_dir": str(BOUNDARY_OUTPUT_DIR.relative_to(REPO_ROOT)),
"size_bytes": boundary_stats["size_bytes"],
"manifest": boundary_manifest,
"collector_result": build_result.get("result"),
"build_skipped": build_skipped,
"production_target": BUILD_CONFIG["production_target"],
"pmtiles_artifact": str(PMTILES_ARTIFACT_PATH.relative_to(REPO_ROOT)),
"pmtiles_exists": pmtiles_exists,
},
},
]
if pmtiles_exists:
records.append(
{
"id": "pmtiles-artifact",
"name": "Earth boundary PMTiles artifact",
"description": "Single-file PMTiles/MVT artifact for Earth boundaries",
"value": pmtiles_size,
"unit": "bytes",
"metadata": {
"path": str(PMTILES_ARTIFACT_PATH.relative_to(REPO_ROOT)),
"exists": pmtiles_exists,
"provider": "pmtiles-mvt",
},
}
)
for zoom, count in sorted(tile_counts.items(), key=lambda item: int(item[0])):
records.append(
{
"id": f"tile-z{zoom}",
"name": f"Earth boundary tiles z{zoom}",
"description": f"Generated Earth boundary tile count for zoom {zoom}",
"value": int(count),
"unit": "tiles",
"metadata": {
"zoom": int(zoom),
"tile_count": int(count),
"output_dir": str((BOUNDARY_OUTPUT_DIR / str(zoom)).relative_to(REPO_ROOT)),
},
}
)
await self.update_phase_progress(
current=3,
total=3,
unit="steps",
message="边界瓦片产物已登记",
progress=100,
commit=True,
force=True,
)
return records

View File

@@ -3,10 +3,13 @@
from __future__ import annotations
from dataclasses import dataclass
from copy import deepcopy
from typing import Any
from sqlalchemy import select
from sqlalchemy.orm.attributes import flag_modified
from app.ai_tasks.prompts import get_effective_prompt
from app.models.system_setting import SystemSetting
from app.schemas.ai import SituationalAnalysisRequest
from app.services.ai_client import AIProviderClient
@@ -15,6 +18,7 @@ from app.services.ai_tools.web_search import WebSearchClient, WebSearchError
CREDENTIAL_GUIDES_CATEGORY = "collector_credential_guides"
CREDENTIAL_GUIDE_PROMPT_KEY = "credential.guide"
@dataclass(frozen=True)
@@ -133,33 +137,57 @@ DEFAULT_CREDENTIAL_GUIDES = {
}
def _normalize_provider(provider: str) -> str:
return provider.strip().lower().replace(" ", "_")
def _credential_guide_default(provider: str) -> CredentialGuideDefault:
normalized = _normalize_provider(provider)
known = DEFAULT_CREDENTIAL_GUIDES.get(normalized)
if known is not None:
return known
title = f"{normalized or 'collector'} 凭证配置教程"
return CredentialGuideDefault(
provider=normalized,
title=title,
prompt=(
f"请生成一份中文教程,指导开发者为 Planet 采集器配置 {normalized} 凭证。"
"教程要面向已经有本地开发环境的人,包含官方入口或文档查找方式、"
"获取 API Key / Token / Client credentials 的通用步骤、在 Planet 采集器配置中"
"填写凭证字段、连接测试、保存、常见失败排查。不要编造具体页面按钮文案;"
"如果公开资料不足,必须明确提醒以 provider 官方文档和当前控制台页面为准。"
),
markdown="",
)
async def _get_guide_store(db) -> tuple[SystemSetting | None, dict[str, Any]]:
result = await db.execute(
select(SystemSetting).where(SystemSetting.category == CREDENTIAL_GUIDES_CATEGORY)
)
record = result.scalar_one_or_none()
payload = dict(record.payload or {}) if record and isinstance(record.payload, dict) else {}
payload = deepcopy(record.payload) if record and isinstance(record.payload, dict) else {}
return record, payload
async def get_credential_guide(db, provider: str) -> dict[str, Any]:
default = DEFAULT_CREDENTIAL_GUIDES.get(provider)
if default is None:
raise ValueError(f"Unsupported credential guide provider: {provider}")
provider = _normalize_provider(provider)
default = _credential_guide_default(provider)
_record, store = await _get_guide_store(db)
custom = store.get(provider) if isinstance(store.get(provider), dict) else None
has_default_markdown = bool(default.markdown.strip())
return {
"provider": provider,
"title": custom.get("title") if custom else default.title,
"markdown": custom.get("markdown") if custom else default.markdown,
"prompt": default.prompt,
"source": "ai" if custom else "default",
"source": "ai" if custom else "default" if has_default_markdown else "missing",
"sources": custom.get("sources", []) if custom else [],
"verification_status": (
custom.get("verification_status", "verified_with_search_evidence")
if custom
else "default_unverified"
else "default_unverified" if has_default_markdown else "missing"
),
"verification_error": custom.get("verification_error") if custom else None,
}
@@ -175,9 +203,8 @@ async def save_credential_guide(
verification_status: str = "verified_with_search_evidence",
verification_error: str | None = None,
) -> dict[str, Any]:
default = DEFAULT_CREDENTIAL_GUIDES.get(provider)
if default is None:
raise ValueError(f"Unsupported credential guide provider: {provider}")
provider = _normalize_provider(provider)
default = _credential_guide_default(provider)
record, store = await _get_guide_store(db)
store[provider] = {
@@ -190,21 +217,21 @@ async def save_credential_guide(
if record is None:
db.add(SystemSetting(category=CREDENTIAL_GUIDES_CATEGORY, payload=store))
else:
record.payload = store
record.payload = deepcopy(store)
flag_modified(record, "payload")
await db.commit()
return await get_credential_guide(db, provider)
async def reset_credential_guide(db, provider: str) -> dict[str, Any]:
default = DEFAULT_CREDENTIAL_GUIDES.get(provider)
if default is None:
raise ValueError(f"Unsupported credential guide provider: {provider}")
provider = _normalize_provider(provider)
record, store = await _get_guide_store(db)
if provider in store:
store.pop(provider, None)
if record is not None:
record.payload = store
record.payload = deepcopy(store)
flag_modified(record, "payload")
await db.commit()
return await get_credential_guide(db, provider)
@@ -215,9 +242,8 @@ async def generate_credential_guide(
ai_client: AIProviderClient,
web_search_client: WebSearchClient | None = None,
) -> dict[str, Any]:
default = DEFAULT_CREDENTIAL_GUIDES.get(provider)
if default is None:
raise ValueError(f"Unsupported credential guide provider: {provider}")
provider = _normalize_provider(provider)
default = _credential_guide_default(provider)
search_evidence: list[dict[str, Any]] = []
search_error: str | None = None
@@ -240,14 +266,12 @@ async def generate_credential_guide(
guide["sources"] = []
return guide
prompt = await get_effective_prompt(db, CREDENTIAL_GUIDE_PROMPT_KEY)
response = await ai_client.analyze(
SituationalAnalysisRequest(
title=f"Generate credential guide for {provider}",
objective=(
default.prompt
+ "\n只能根据 context.search_evidence 中的来源生成教程;"
+ "如果证据不足,明确说明需要以官方页面为准。"
),
objective=f"{default.prompt}\n{prompt.prompt}",
system_prompt=prompt.system_prompt or None,
context={
"provider": provider,
"current_default_guide": default.markdown,

View File

@@ -18,6 +18,7 @@ from app.models.datasource import DataSource
from app.models.datasource_config import DataSourceConfig
from app.models.system_setting import SystemSetting
from app.services.barentswatch import (
BarentsWatchConfig,
_read_zshrc_env,
fetch_barentswatch_access_token,
resolve_barentswatch_config,
@@ -44,6 +45,20 @@ def _resolve_spacetrack_credentials() -> tuple[str, str, str]:
return username, password, source or "missing"
def _resolve_spacetrack_credentials_with_override(
credential_override: dict[str, str] | None = None,
) -> tuple[str, str, str]:
if credential_override and (
credential_override.get("username") or credential_override.get("password")
):
return (
str(credential_override.get("username") or ""),
str(credential_override.get("password") or ""),
"draft",
)
return _resolve_spacetrack_credentials()
async def _resolve_aisstream_api_key(
db=None,
credential_override: dict[str, str] | None = None,
@@ -126,7 +141,9 @@ async def build_builtin_connectivity_checksum(
}
)
elif credential_provider == "spacetrack":
username, password, credential_source = _resolve_spacetrack_credentials()
username, password, credential_source = _resolve_spacetrack_credentials_with_override(
credential_override
)
has_credentials = bool(username and password)
credential_fingerprint = _sha256_json(
{
@@ -230,7 +247,16 @@ async def test_builtin_connectivity(
try:
async with httpx.AsyncClient(timeout=timeout, follow_redirects=True) as client:
if credential_context["credential_provider"] == "barentswatch":
barentswatch_config = await resolve_barentswatch_config(db)
if credential_override:
barentswatch_config = BarentsWatchConfig(
endpoint=str(request_endpoint or ""),
client_id=str(credential_override.get("client_id") or ""),
client_secret=str(credential_override.get("client_secret") or ""),
credential_source="draft",
endpoint_source="draft",
)
else:
barentswatch_config = await resolve_barentswatch_config(db)
token = await fetch_barentswatch_access_token(client, barentswatch_config)
if not token:
return {
@@ -243,7 +269,9 @@ async def test_builtin_connectivity(
}
request_headers["Authorization"] = f"Bearer {token}"
elif credential_context["credential_provider"] == "spacetrack":
username, password, _source = _resolve_spacetrack_credentials()
username, password, _source = _resolve_spacetrack_credentials_with_override(
credential_override
)
login_url = "https://www.space-track.org/ajaxauth/login"
login_response = await client.post(
login_url,

View File

@@ -301,52 +301,6 @@ async def persist_mapped_records(
transport: str | None = None,
) -> int:
"""Persist validated mapped records to the destination for a target schema."""
if target_schema == "earth_boundary_source":
from app.models.collected_data import CollectedData
now = datetime.now(UTC)
written_count = 0
for index, record in enumerate(records):
source_id = record.get("source_id") or record.get("sha256") or str(index)
entity_key = f"{datasource_name}:{source_id}"
previous_result = await db.execute(
select(CollectedData)
.where(CollectedData.source == datasource_name)
.where(CollectedData.entity_key == entity_key)
.where(CollectedData.is_current.is_(True))
.order_by(CollectedData.id.desc())
.limit(1)
)
previous = previous_result.scalar_one_or_none()
if previous is not None:
previous.is_current = False
db.add(
CollectedData(
source=datasource_name,
source_id=str(source_id),
entity_key=entity_key,
data_type=target_schema,
name=record.get("name") or str(source_id),
description=f"Earth boundary source artifact: {record.get('source_kind')}",
extra_data={
**record,
"datasource_config_id": datasource_config_id,
"mapping_version": mapping_version,
"delivery_mode": delivery_mode or "polling",
"transport": transport or "http",
},
collected_at=now,
is_valid=1,
is_current=True,
previous_record_id=previous.id if previous else None,
change_type="updated" if previous else "created",
change_summary={},
)
)
written_count += 1
await db.commit()
return written_count
if target_schema == "vessel_ais":
from app.core.time import to_iso8601_utc
from app.core.websocket.broadcaster import broadcaster

View File

@@ -47,6 +47,8 @@ DOCS_METADATA: tuple[DocsMetadata, ...] = (
DocsMetadata("frontend-admin-frontend-context.md", "frontend-admin-frontend-context", "docs_developer", "Frontend", 20, "控制台前端结构", "Admin Frontend Context"),
DocsMetadata("frontend-layout-guidelines.md", "frontend-layout-guidelines", "docs_developer", "Frontend", 21, "前端布局指南", "Frontend Layout Guidelines"),
DocsMetadata("docs-gatekeeper-development.md", "docs-gatekeeper-development", "docs_developer", "Frontend", 22, "Docs Gatekeeper 开发说明", "Docs Gatekeeper Development Guide"),
DocsMetadata("naming-glossary.md", "naming-glossary", "docs_developer", "Frontend", 23, "命名与术语对照", "Naming Glossary"),
DocsMetadata("tactile-ui-components.md", "tactile-ui-components", "docs_developer", "Frontend", 24, "Tactile UI 组件库", "Tactile UI Components"),
DocsMetadata("backend-collectors.md", "backend-collectors", "docs_developer", "Backend", 30, "数据采集系统", "Data Collectors"),
DocsMetadata("backend-system-service-control.md", "backend-system-service-control", "docs_admin", "Backend", 31, "系统服务控制", "System Service Control"),
DocsMetadata("datasource-collector-settings-connectivity.md", "datasource-collector-settings-connectivity", "docs_developer", "Backend", 32, "数据源、采集器设置与连接验证", "Datasource Collector Settings and Connectivity"),

View File

@@ -0,0 +1,671 @@
"""Earth boundary static asset service."""
from __future__ import annotations
import asyncio
import hashlib
import json
import shutil
import sys
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
from uuid import uuid4
import httpx
REPO_ROOT = Path(__file__).resolve().parents[3]
SOURCE_OUTPUT_DIR = REPO_ROOT / "data/earth-boundary-sources"
SOURCE_MANIFEST_PATH = SOURCE_OUTPUT_DIR / "manifest.json"
BUILD_RESULT_PATH = SOURCE_OUTPUT_DIR / "build-result.json"
BUILD_JOB_PATH = SOURCE_OUTPUT_DIR / "build-job.json"
BOUNDARY_OUTPUT_DIR = REPO_ROOT / "frontend/public/earth/data/boundaries/v1"
BOUNDARY_MANIFEST_PATH = BOUNDARY_OUTPUT_DIR / "manifest.json"
PMTILES_ARTIFACT_PATH = (
REPO_ROOT / "frontend/public/earth/data/boundaries/earth-boundaries-china-pov-v1.pmtiles"
)
LEGACY_GEOJSON_PATH = REPO_ROOT / "frontend/public/earth/data/countries-admin0.min.geojson"
POV_POLICY_PATH = REPO_ROOT / "config/earth-boundary-pov-policy.china-v1.json"
LOCAL_CONFIG_PATH = REPO_ROOT / "config/earth-boundary-sources.local.json"
EXAMPLE_CONFIG_PATH = REPO_ROOT / "config/earth-boundary-sources.example.json"
BOUNDARY_SOURCE_KINDS = {
"earth_admin0_boundaries": "admin0-boundaries",
"earth_coastline": "coastline",
"earth_claim_lines": "claim-lines",
}
DEFAULT_PUBLIC_BOUNDARY_SOURCES = {
"earth_admin0_boundaries": {
"displayName": "Natural Earth Admin-0 Countries",
"sourceKind": "admin0-boundaries",
"endpoint": "https://raw.githubusercontent.com/nvkelso/natural-earth-vector/master/geojson/ne_10m_admin_0_countries.geojson",
"method": "GET",
"headers": {},
"auth_type": "none",
"license": "Natural Earth public domain",
},
"earth_coastline": {
"displayName": "Natural Earth Coastline",
"sourceKind": "coastline",
"endpoint": "https://raw.githubusercontent.com/nvkelso/natural-earth-vector/master/geojson/ne_10m_coastline.geojson",
"method": "GET",
"headers": {},
"auth_type": "none",
"license": "Natural Earth public domain",
},
"earth_claim_lines": {
"displayName": "Natural Earth Disputed Boundaries",
"sourceKind": "claim-lines",
"endpoint": "https://raw.githubusercontent.com/nvkelso/natural-earth-vector/master/geojson/ne_10m_admin_0_boundary_lines_disputed_areas.geojson",
"method": "GET",
"headers": {},
"auth_type": "none",
"license": "Natural Earth public domain",
},
}
BUILD_CONFIG = {
"builder": "scripts/build_earth_boundary_pmtiles.py",
"format": "pmtiles+mvt",
"production_target": "pmtiles-mvt",
}
class EarthBoundaryBuildError(RuntimeError):
def __init__(self, message: str, *, code: str = "build_failed", details: Any = None) -> None:
super().__init__(message)
self.code = code
self.details = details
_build_job_lock = asyncio.Lock()
_build_task: asyncio.Task | None = None
_build_job_state: dict[str, Any] = {}
def _utc_now_iso() -> str:
return datetime.now(UTC).isoformat()
def _public_job_state() -> dict[str, Any]:
if _build_job_state:
return dict(_build_job_state)
return _read_json(BUILD_JOB_PATH)
def get_boundary_build_status() -> dict[str, Any]:
return {"job": _public_job_state()}
def _set_job_state(**updates: Any) -> dict[str, Any]:
global _build_job_state
current = dict(_build_job_state)
current.update(updates)
current["updated_at"] = _utc_now_iso()
_build_job_state = current
_write_json(BUILD_JOB_PATH, current)
return current
def _append_job_log(message: str) -> None:
logs = list(_build_job_state.get("logs") or [])
logs.append({"time": _utc_now_iso(), "message": message})
_set_job_state(logs=logs[-40:])
def _update_job_progress(progress: float, phase: str, message: str, **extra: Any) -> None:
bounded_progress = max(0, min(100, int(round(progress))))
_set_job_state(
status="running",
progress=bounded_progress,
phase=phase,
message=message,
**extra,
)
def _read_json(path: Path) -> dict[str, Any]:
if not path.exists():
return {}
with path.open("r", encoding="utf-8") as f:
payload = json.load(f)
return payload if isinstance(payload, dict) else {}
def _write_json(path: Path, payload: dict[str, Any]) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
with path.open("w", encoding="utf-8") as f:
json.dump(payload, f, ensure_ascii=False, indent=2)
f.write("\n")
def _sha256_bytes(payload: bytes) -> str:
return hashlib.sha256(payload).hexdigest()
def _stable_json_hash(payload: Any) -> str:
encoded = json.dumps(payload, ensure_ascii=False, sort_keys=True, separators=(",", ":"))
return hashlib.sha256(encoded.encode("utf-8")).hexdigest()
def _artifact_extension(endpoint: str, content_type: str, payload: bytes) -> str:
suffix = Path(endpoint.split("?", 1)[0]).suffix.lower()
if suffix in {".json", ".geojson", ".zip", ".pbf"}:
return suffix
if "geo+json" in content_type or b'"FeatureCollection"' in payload[:4096]:
return ".geojson"
if "json" in content_type:
return ".json"
return ".dat"
def _json_feature_count(payload: Any) -> int:
if isinstance(payload, dict) and isinstance(payload.get("features"), list):
return len(payload["features"])
if isinstance(payload, list):
return len(payload)
return 1 if payload else 0
def _directory_stats(path: Path) -> dict[str, int]:
if not path.exists():
return {"file_count": 0, "size_bytes": 0}
files = [item for item in path.rglob("*") if item.is_file()]
return {"file_count": len(files), "size_bytes": sum(item.stat().st_size for item in files)}
def _load_source_feature_collection(source: dict[str, Any]) -> dict[str, Any]:
path = REPO_ROOT / source["path"]
payload = _read_json(path)
features = payload.get("features") if isinstance(payload, dict) else None
return {
"type": "FeatureCollection",
"features": features if isinstance(features, list) else [],
}
def _write_high_precision_geojson_manifest(
sources: list[dict[str, Any]],
build_input_hash: str,
missing_tools: list[str],
) -> dict[str, Any]:
BOUNDARY_OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
admin0 = next(source for source in sources if source["kind"] == "admin0-boundaries")
coastline = next(source for source in sources if source["kind"] == "coastline")
claim_lines = next(source for source in sources if source["kind"] == "claim-lines")
admin0_payload = _load_source_feature_collection(admin0)
coastline_payload = _load_source_feature_collection(coastline)
claim_payload = _load_source_feature_collection(claim_lines)
for feature in coastline_payload["features"]:
props = feature.setdefault("properties", {})
if isinstance(props, dict):
props["PLANET_LAYER"] = "coastline"
base_payload = {
"type": "FeatureCollection",
"features": [*admin0_payload["features"], *coastline_payload["features"]],
}
base_path = BOUNDARY_OUTPUT_DIR / "earth-boundaries-high-precision-base.geojson"
hover_path = BOUNDARY_OUTPUT_DIR / "earth-boundaries-high-precision-hover.geojson"
claim_path = BOUNDARY_OUTPUT_DIR / "earth-boundaries-high-precision-claims.geojson"
_write_json(base_path, base_payload)
_write_json(hover_path, admin0_payload)
_write_json(claim_path, claim_payload)
manifest = {
"version": "natural-earth-v1",
"builtAt": _utc_now_iso(),
"tileProvider": "geojson-high-precision",
"format": "geojson-directory",
"buildInputHash": build_input_hash,
"base": base_path.name,
"hoverIndex": hover_path.name,
"claimLine": claim_path.name,
"sourceFeatureCount": {
"admin0": len(admin0_payload["features"]),
"coastline": len(coastline_payload["features"]),
"claimLines": len(claim_payload["features"]),
},
"pmtiles": None,
"missingTools": missing_tools,
}
_write_json(BOUNDARY_MANIFEST_PATH, manifest)
return manifest
def _relative(path: Path) -> str:
return str(path.relative_to(REPO_ROOT))
def load_boundary_config() -> tuple[dict[str, Any], str]:
if LOCAL_CONFIG_PATH.exists():
return _read_json(LOCAL_CONFIG_PATH), "local"
return _read_json(EXAMPLE_CONFIG_PATH), "example"
def save_boundary_config(payload: dict[str, Any]) -> dict[str, Any]:
if not isinstance(payload, dict):
raise EarthBoundaryBuildError("Earth boundary config must be a JSON object", code="invalid_config")
_write_json(LOCAL_CONFIG_PATH, payload)
return get_boundary_status()
def _source_configs(payload: dict[str, Any]) -> dict[str, Any]:
raw_sources = payload.get("collectorConfigs") or payload.get("sources") or {}
return raw_sources if isinstance(raw_sources, dict) else {}
def _is_placeholder_endpoint(endpoint: Any) -> bool:
value = str(endpoint or "").strip()
return not value or "example.com" in value
def _source_configs_with_defaults(payload: dict[str, Any]) -> dict[str, Any]:
raw_sources = _source_configs(payload)
merged: dict[str, Any] = {}
for source_key, default_config in DEFAULT_PUBLIC_BOUNDARY_SOURCES.items():
configured = raw_sources.get(source_key)
if not isinstance(configured, dict) or _is_placeholder_endpoint(configured.get("endpoint")):
merged[source_key] = dict(default_config)
else:
merged[source_key] = {**default_config, **configured}
for source_key, source_config in raw_sources.items():
if source_key not in merged:
merged[source_key] = source_config
return merged
def _build_input_hash(source_manifest: dict[str, Any]) -> str:
return _stable_json_hash(
{
"source_manifest_schema": source_manifest.get("schema"),
"sources": [
{
"id": source.get("id"),
"sha256": source.get("sha256"),
"kind": source.get("kind"),
}
for source in source_manifest.get("sources", [])
],
"pov_policy": source_manifest.get("povPolicy"),
"build_config": BUILD_CONFIG,
}
)
def _has_current_artifacts(boundary_manifest: dict[str, Any], build_input_hash: str) -> bool:
return (
bool(boundary_manifest)
and boundary_manifest.get("buildInputHash") == build_input_hash
and boundary_manifest.get("tileProvider") == "pmtiles-mvt"
and PMTILES_ARTIFACT_PATH.exists()
)
def get_boundary_status() -> dict[str, Any]:
config_payload, config_source = load_boundary_config()
effective_source_configs = _source_configs_with_defaults(config_payload)
boundary_manifest = _read_json(BOUNDARY_MANIFEST_PATH)
pmtiles_exists = PMTILES_ARTIFACT_PATH.exists()
manifest_exists = BOUNDARY_MANIFEST_PATH.exists()
high_precision_ready = (
manifest_exists
and (
(
boundary_manifest.get("tileProvider") == "pmtiles-mvt"
and pmtiles_exists
)
or boundary_manifest.get("tileProvider") == "geojson-high-precision"
)
)
legacy_exists = LEGACY_GEOJSON_PATH.exists()
provider = (
boundary_manifest.get("tileProvider")
if high_precision_ready
else "legacy-geojson" if legacy_exists else "missing"
)
return {
"provider": provider,
"high_precision_ready": high_precision_ready,
"fallback_available": legacy_exists,
"config_source": config_source,
"config_path": _relative(LOCAL_CONFIG_PATH),
"config_exists": LOCAL_CONFIG_PATH.exists(),
"config": config_payload,
"effective_default_sources": [
source_key
for source_key, source_config in effective_source_configs.items()
if source_key in DEFAULT_PUBLIC_BOUNDARY_SOURCES
and source_config.get("endpoint") == DEFAULT_PUBLIC_BOUNDARY_SOURCES[source_key]["endpoint"]
],
"manifest": {
"path": _relative(BOUNDARY_MANIFEST_PATH),
"exists": manifest_exists,
"tileProvider": boundary_manifest.get("tileProvider"),
"buildInputHash": boundary_manifest.get("buildInputHash"),
"builtAt": boundary_manifest.get("builtAt"),
},
"pmtiles": {
"path": _relative(PMTILES_ARTIFACT_PATH),
"exists": pmtiles_exists,
"size_bytes": PMTILES_ARTIFACT_PATH.stat().st_size if pmtiles_exists else 0,
},
"legacy": {
"path": _relative(LEGACY_GEOJSON_PATH),
"exists": legacy_exists,
"size_bytes": LEGACY_GEOJSON_PATH.stat().st_size if legacy_exists else 0,
},
"source_manifest": {
"path": _relative(SOURCE_MANIFEST_PATH),
"exists": SOURCE_MANIFEST_PATH.exists(),
},
"last_build": _read_json(BUILD_RESULT_PATH),
"current_job": _public_job_state(),
}
async def _download_source(
source_key: str,
source_config: dict[str, Any],
progress_callback: Any = None,
) -> dict[str, Any]:
endpoint = str(source_config.get("endpoint") or "").strip()
if _is_placeholder_endpoint(endpoint):
raise EarthBoundaryBuildError(
f"{source_key} endpoint is not configured",
code="source_not_configured",
details={"source": source_key},
)
method = str(source_config.get("method") or "GET").upper()
if method not in {"GET", "POST"}:
raise EarthBoundaryBuildError(
f"{source_key} method must be GET or POST",
code="invalid_config",
details={"source": source_key, "method": method},
)
if endpoint.startswith("file://") or Path(endpoint).expanduser().exists():
payload = Path(endpoint.removeprefix("file://")).expanduser().read_bytes()
content_type = "application/octet-stream"
if progress_callback:
progress_callback(1, len(payload), len(payload))
else:
timeout = float(source_config.get("timeout") or 120)
headers = source_config.get("headers") if isinstance(source_config.get("headers"), dict) else {}
async with httpx.AsyncClient(timeout=timeout, follow_redirects=True) as client:
async with client.stream(method, endpoint, headers=headers) as response:
response.raise_for_status()
content_type = response.headers.get("content-type", "")
total = int(response.headers.get("content-length") or 0)
chunks = []
downloaded = 0
async for chunk in response.aiter_bytes():
if not chunk:
continue
chunks.append(chunk)
downloaded += len(chunk)
if progress_callback:
progress_callback(
(downloaded / total) if total else None,
downloaded,
total,
)
payload = b"".join(chunks)
extension = _artifact_extension(endpoint, content_type, payload)
parsed: Any = None
if extension in {".json", ".geojson"}:
parsed = json.loads(payload.decode("utf-8"))
feature_count = _json_feature_count(parsed)
if feature_count <= 0:
raise EarthBoundaryBuildError(
f"{source_key} downloaded payload contains no features",
code="empty_source",
details={"source": source_key},
)
sha256 = _sha256_bytes(payload)
source_dir = SOURCE_OUTPUT_DIR / source_key
source_dir.mkdir(parents=True, exist_ok=True)
artifact_path = source_dir / f"{sha256}{extension}"
artifact_path.write_bytes(payload)
return {
"id": source_key,
"kind": source_config.get("sourceKind") or BOUNDARY_SOURCE_KINDS[source_key],
"path": _relative(artifact_path),
"sha256": sha256,
"featureCount": feature_count,
"license": source_config.get("license"),
}
async def _run_step(args: list[str]) -> dict[str, Any]:
process = await asyncio.create_subprocess_exec(
sys.executable,
*args,
cwd=REPO_ROOT,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
stdout_bytes, stderr_bytes = await process.communicate()
stdout = stdout_bytes.decode("utf-8", errors="replace").strip()
stderr = stderr_bytes.decode("utf-8", errors="replace").strip()
payload: dict[str, Any] = {"stdout": stdout, "stderr": stderr, "returncode": process.returncode}
last_line = stdout.splitlines()[-1:] or []
if last_line:
try:
payload["result"] = json.loads(last_line[0])
except json.JSONDecodeError:
payload["result"] = last_line[0]
if process.returncode != 0:
raise EarthBoundaryBuildError(
stderr or stdout or f"command failed: {' '.join(args)}",
code="build_command_failed",
details=payload,
)
return payload
async def build_boundary_assets(progress_callback: Any = None) -> dict[str, Any]:
config_payload, config_source = load_boundary_config()
source_configs = _source_configs_with_defaults(config_payload)
missing = [source for source in BOUNDARY_SOURCE_KINDS if source not in source_configs]
if missing:
raise EarthBoundaryBuildError(
f"Missing Earth boundary source configs: {', '.join(missing)}",
code="missing_sources",
details={"missing": missing},
)
sources = []
source_keys = list(BOUNDARY_SOURCE_KINDS)
for index, source_key in enumerate(source_keys):
source_config = source_configs[source_key]
if not isinstance(source_config, dict):
raise EarthBoundaryBuildError(
f"{source_key} config must be an object",
code="invalid_config",
details={"source": source_key},
)
source_start = 8 + index * 18
source_end = source_start + 18
if progress_callback:
progress_callback(source_start, "download", f"正在下载 {source_key}")
def report_download_progress(ratio: float | None, downloaded: int, total: int) -> None:
if not progress_callback:
return
if ratio is None:
progress_callback(source_start + 8, "download", f"{source_key} 已下载 {downloaded} bytes")
return
progress_callback(
source_start + (source_end - source_start) * ratio,
"download",
f"{source_key} 下载 {int(ratio * 100)}%",
downloaded_bytes=downloaded,
total_bytes=total,
)
sources.append(await _download_source(source_key, source_config, report_download_progress))
source_manifest = {
"schema": "planet-earth-boundary-sources/v2",
"sources": sources,
"povPolicy": _read_json(POV_POLICY_PATH),
}
if progress_callback:
progress_callback(65, "manifest", "正在写入边界源 manifest")
_write_json(SOURCE_MANIFEST_PATH, source_manifest)
build_input_hash = _build_input_hash(source_manifest)
boundary_manifest = _read_json(BOUNDARY_MANIFEST_PATH)
build_skipped = _has_current_artifacts(boundary_manifest, build_input_hash)
missing_tools = [tool for tool in ("tippecanoe", "pmtiles") if shutil.which(tool) is None]
if missing_tools and not build_skipped:
if progress_callback:
progress_callback(82, "build", "缺少 PMTiles 工具,正在生成 GeoJSON 高清包")
boundary_manifest = _write_high_precision_geojson_manifest(
sources,
build_input_hash,
missing_tools,
)
result = {
"status": "built_geojson_fallback",
"code": "missing_tools",
"missing_tools": missing_tools,
"sources": sources,
"boundary_manifest": _relative(BOUNDARY_MANIFEST_PATH),
"manifest": boundary_manifest,
}
_write_json(BUILD_RESULT_PATH, result)
if progress_callback:
progress_callback(96, "finalize", "GeoJSON 高清国界包已生成")
return {**get_boundary_status(), "build": result}
if build_skipped:
if progress_callback:
progress_callback(96, "unchanged", "高精国界已是最新")
build_result = {
"status": "unchanged",
"reason": "source manifest and build config hash unchanged",
"buildInputHash": build_input_hash,
}
else:
if progress_callback:
progress_callback(72, "build", "正在构建 PMTiles/MVT")
admin0 = next(source for source in sources if source["kind"] == "admin0-boundaries")
coastline = next(source for source in sources if source["kind"] == "coastline")
claim_lines = next(source for source in sources if source["kind"] == "claim-lines")
build_result = await _run_step(
[
"scripts/build_earth_boundary_pmtiles.py",
"--admin0-source",
admin0["path"],
"--coastline-source",
coastline["path"],
"--claims-source",
claim_lines["path"],
"--output",
_relative(PMTILES_ARTIFACT_PATH),
"--manifest",
_relative(BOUNDARY_MANIFEST_PATH),
"--build-input-hash",
build_input_hash,
"--pov-policy",
_relative(POV_POLICY_PATH),
]
)
if progress_callback:
progress_callback(95, "finalize", "正在校验构建产物")
boundary_manifest = _read_json(BOUNDARY_MANIFEST_PATH)
boundary_stats = _directory_stats(BOUNDARY_OUTPUT_DIR)
result = {
"status": "unchanged" if build_skipped else "built",
"sources": sources,
"source_manifest": _relative(SOURCE_MANIFEST_PATH),
"boundary_manifest": _relative(BOUNDARY_MANIFEST_PATH),
"pmtiles_artifact": _relative(PMTILES_ARTIFACT_PATH),
"pmtiles_exists": PMTILES_ARTIFACT_PATH.exists(),
"boundary_stats": boundary_stats,
"manifest": boundary_manifest,
"build_result": build_result,
}
_write_json(BUILD_RESULT_PATH, result)
return {**get_boundary_status(), "build": result}
async def _run_boundary_build_job(job_id: str) -> None:
def report(progress: float, phase: str, message: str, **extra: Any) -> None:
if _build_job_state.get("id") != job_id:
return
_update_job_progress(progress, phase, message, **extra)
try:
report(3, "prepare", "正在准备高精国界构建")
result = await build_boundary_assets(report)
_set_job_state(
id=job_id,
status="succeeded",
progress=100,
phase="complete",
message="高精国界构建完成",
finished_at=_utc_now_iso(),
result={
"provider": result.get("provider"),
"high_precision_ready": result.get("high_precision_ready"),
"pmtiles": result.get("pmtiles"),
"manifest": result.get("manifest"),
},
)
_append_job_log("高精国界构建完成")
except EarthBoundaryBuildError as exc:
_set_job_state(
id=job_id,
status="failed",
progress=_build_job_state.get("progress", 0),
phase="failed",
message=str(exc),
code=exc.code,
details=exc.details,
finished_at=_utc_now_iso(),
)
_append_job_log(str(exc))
except Exception as exc: # pragma: no cover - defensive guard for background task
_set_job_state(
id=job_id,
status="failed",
progress=_build_job_state.get("progress", 0),
phase="failed",
message=str(exc),
code="build_failed",
finished_at=_utc_now_iso(),
)
_append_job_log(str(exc))
async def start_boundary_build_job() -> dict[str, Any]:
global _build_task
async with _build_job_lock:
if _build_task and not _build_task.done():
return {"accepted": False, "job": _public_job_state()}
job_id = uuid4().hex
_set_job_state(
id=job_id,
status="queued",
progress=0,
phase="queued",
message="高精国界构建已加入队列",
logs=[],
started_at=_utc_now_iso(),
finished_at=None,
code=None,
details=None,
)
_append_job_log("高精国界构建已启动")
_build_task = asyncio.create_task(_run_boundary_build_job(job_id))
return {"accepted": True, "job": _public_job_state()}

View File

@@ -0,0 +1,408 @@
from __future__ import annotations
import asyncio
from collections.abc import Awaitable, Callable
from dataclasses import dataclass
import hashlib
import json
import time
from typing import Any
from fastapi import Response
from app.core.cache import _RedisClient
from app.core.config import settings
from app.core.logging import get_logger
logger = get_logger(__name__, service="earth_layer_cache")
EARTH_LAYER_CACHE_PREFIX = "earth:layer:v1"
EARTH_LAYER_LOCK_PREFIX = "earth:layer:lock:v1"
DEFAULT_LOCK_TTL_SECONDS = 10
DEFAULT_LOCK_WAIT_SECONDS = 0.2
DEFAULT_MAX_FEATURES = 5000
DEFAULT_MAX_BYTES = 5 * 1024 * 1024
DEFAULT_BBOX_PRECISION_DEGREES = 0.1
DEV_CACHE_KEY_HEADER = {"development", "dev", "test", "testing", "local"}
@dataclass(frozen=True)
class EarthLayerCachePolicy:
fresh_ttl_seconds: int
stale_ttl_seconds: int
max_features: int = DEFAULT_MAX_FEATURES
max_bytes: int = DEFAULT_MAX_BYTES
lock_ttl_seconds: int = DEFAULT_LOCK_TTL_SECONDS
lock_wait_seconds: float = DEFAULT_LOCK_WAIT_SECONDS
@dataclass(frozen=True)
class EarthLayerCacheResult:
payload: dict[str, Any]
state: str
key: str
features: int
bytes: int
class EarthLayerCache:
def __init__(self) -> None:
self._client = None
@property
def client(self):
if self._client is None:
self._client = _RedisClient.get_client()
return self._client
@staticmethod
def key(layer: str, **params: Any) -> str:
parts = [EARTH_LAYER_CACHE_PREFIX, _safe_key_part(layer)]
for name in sorted(params):
value = params[name]
if value is None:
value = "none"
parts.append(f"{_safe_key_part(name)}:{_safe_key_part(value)}")
return ":".join(parts)
@staticmethod
def stale_key(key: str) -> str:
return f"{key}:stale"
@staticmethod
def lock_key(key: str) -> str:
digest = hashlib.sha256(key.encode("utf-8")).hexdigest()[:32]
return f"{EARTH_LAYER_LOCK_PREFIX}:{digest}"
def get_json(self, key: str) -> dict[str, Any] | None:
raw = self.client.get(key)
if not raw:
return None
value = json.loads(raw)
return value if isinstance(value, dict) else None
def set_json(self, key: str, payload: dict[str, Any], ttl_seconds: int) -> None:
self.client.setex(key, ttl_seconds, json.dumps(payload, ensure_ascii=False, default=str))
def acquire_lock(self, key: str, ttl_seconds: int) -> bool:
return bool(self.client.set(self.lock_key(key), "1", nx=True, ex=ttl_seconds))
def release_lock(self, key: str) -> None:
try:
self.client.delete(self.lock_key(key))
except Exception:
pass
def delete_pattern(self, pattern: str = f"{EARTH_LAYER_CACHE_PREFIX}:*") -> int:
keys = list(self.client.scan_iter(match=pattern))
if not keys:
return 0
return int(self.client.delete(*keys))
def status(self) -> dict[str, Any]:
keys = list(self.client.scan_iter(match=f"{EARTH_LAYER_CACHE_PREFIX}:*"))
by_layer: dict[str, dict[str, Any]] = {}
total_memory = 0
for key in keys:
key_str = key.decode("utf-8") if isinstance(key, bytes) else str(key)
layer = _layer_from_key(key_str)
entry = by_layer.setdefault(layer, {"keys": 0, "stale_keys": 0, "memory_bytes": 0})
entry["keys"] += 1
if key_str.endswith(":stale"):
entry["stale_keys"] += 1
try:
memory = int(self.client.memory_usage(key) or 0)
except Exception:
memory = 0
entry["memory_bytes"] += memory
total_memory += memory
return {
"prefix": EARTH_LAYER_CACHE_PREFIX,
"key_count": len(keys),
"memory_bytes": total_memory,
"layers": by_layer,
}
earth_layer_cache = EarthLayerCache()
def quantize_bbox(
bbox: tuple[float, float, float, float],
*,
precision: float = DEFAULT_BBOX_PRECISION_DEGREES,
) -> tuple[float, float, float, float]:
return tuple(round(value / precision) * precision for value in bbox) # type: ignore[return-value]
def format_bbox_key(bbox: tuple[float, float, float, float]) -> str:
return ",".join(f"{value:.1f}" for value in bbox)
def apply_cache_headers(response: Response | None, result: EarthLayerCacheResult) -> None:
if response is None:
return
response.headers["X-Planet-Cache"] = result.state
response.headers["X-Planet-Cache-Features"] = str(result.features)
response.headers["X-Planet-Cache-Bytes"] = str(result.bytes)
env_name = str(getattr(settings, "ENVIRONMENT", "") or "development").lower()
if env_name in DEV_CACHE_KEY_HEADER:
response.headers["X-Planet-Cache-Key"] = result.key
async def get_or_build_layer_payload(
*,
key: str,
policy: EarthLayerCachePolicy,
builder: Callable[[], Awaitable[dict[str, Any]]],
response: Response | None = None,
) -> dict[str, Any]:
result = await resolve_layer_payload(key=key, policy=policy, builder=builder)
apply_cache_headers(response, result)
return result.payload
async def resolve_layer_payload(
*,
key: str,
policy: EarthLayerCachePolicy,
builder: Callable[[], Awaitable[dict[str, Any]]],
) -> EarthLayerCacheResult:
started = time.perf_counter()
try:
cached = earth_layer_cache.get_json(key)
if cached is not None:
return _result(cached, state="hit", key=key)
lock_acquired = earth_layer_cache.acquire_lock(key, policy.lock_ttl_seconds)
if lock_acquired:
try:
payload = await _build_budgeted_payload(builder, policy)
_write_fresh_and_stale(key, payload, policy)
_log_cache_event("refresh", key, payload, started)
return _result(payload, state="refresh", key=key)
except Exception as exc:
stale = _read_stale(key)
if stale is not None:
logger.warning_event(
"Earth layer cache builder failed; returning stale payload",
event="earth_layer_cache.stale_after_builder_error",
context={"key": key, "error": str(exc)},
)
return _result(stale, state="stale", key=key)
raise
finally:
earth_layer_cache.release_lock(key)
stale = _read_stale(key)
if stale is not None:
return _result(stale, state="stale", key=key)
await asyncio.sleep(policy.lock_wait_seconds)
cached_after_wait = earth_layer_cache.get_json(key)
if cached_after_wait is not None:
return _result(cached_after_wait, state="hit", key=key)
payload = await _build_budgeted_payload(builder, policy)
_log_cache_event("miss", key, payload, started)
return _result(payload, state="miss", key=key)
except Exception as exc:
try:
payload = await _build_budgeted_payload(builder, policy)
except Exception:
raise exc
logger.warning_event(
"Earth layer cache bypassed",
event="earth_layer_cache.bypass",
context={"key": key, "error": str(exc)},
)
return _result(payload, state="bypass", key=key)
def apply_payload_budget(payload: dict[str, Any], policy: EarthLayerCachePolicy) -> dict[str, Any]:
budgeted = _truncate_features(payload, policy.max_features, "feature_budget")
size = _payload_size(budgeted)
if size <= policy.max_bytes:
return budgeted
features = budgeted.get("features")
if not isinstance(features, list):
return _with_budget_diagnostics(
budgeted,
truncated=True,
reason="byte_budget",
bytes_before=size,
bytes_after=size,
)
low = 0
high = len(features)
best = []
best_size = _payload_size({**budgeted, "features": best})
while low <= high:
mid = (low + high) // 2
candidate_features = features[:mid]
candidate = _with_budget_diagnostics(
{**budgeted, "features": candidate_features},
truncated=mid < len(features),
reason="byte_budget",
bytes_before=size,
bytes_after=0,
)
candidate_size = _payload_size(candidate)
if candidate_size <= policy.max_bytes:
best = candidate_features
best_size = candidate_size
low = mid + 1
else:
high = mid - 1
return _with_budget_diagnostics(
{**budgeted, "features": best},
truncated=True,
reason="byte_budget",
bytes_before=size,
bytes_after=best_size,
)
def invalidate_earth_layer_cache_for_source(source: str) -> int:
source_key = str(source or "").strip()
patterns = {
"barentswatch_vessels": ["vessels*", "summary*"],
"aisstream_vessels": ["vessels*", "summary*"],
"telegeography_cables": ["cables*", "landing-points*", "summary*"],
"telegeography_landing": ["landing-points*", "summary*"],
"telegeography_landing_points": ["landing-points*", "summary*"],
"telegeography_systems": ["cables*", "summary*"],
"telegeography_cable_systems": ["cables*", "summary*"],
"arcgis_cables": ["cables*", "landing-points*", "summary*"],
"arcgis_landing_points": ["landing-points*", "summary*"],
"arcgis_cable_landing_relation": ["landing-points*", "summary*"],
"arcgis_cable_landing_relations": ["landing-points*", "summary*"],
"fao_landing_points": ["landing-points*", "summary*"],
"celestrak_tle": ["satellites*", "summary*"],
"spacetrack_tle": ["satellites*", "summary*"],
"top500": ["compute-centers*", "summary*"],
"top500_supercomputers": ["compute-centers*", "summary*"],
"epoch_ai_gpu": ["compute-centers*", "summary*"],
"huggingface_models": ["compute-centers*", "summary*"],
"huggingface_datasets": ["compute-centers*", "summary*"],
"huggingface_spaces": ["compute-centers*", "summary*"],
"ris_live_bgp": ["bgp*", "summary*"],
"bgpstream_bgp": ["bgp*", "summary*"],
"iptoasn_prefix_geo": ["bgp*", "summary*"],
"opengeofeed_prefix_geo": ["bgp*", "summary*"],
"nro_delegated_prefix_geo": ["bgp*", "summary*"],
}.get(source_key, [])
deleted = 0
for layer_pattern in patterns:
deleted += earth_layer_cache.delete_pattern(f"{EARTH_LAYER_CACHE_PREFIX}:{layer_pattern}")
return deleted
async def _build_budgeted_payload(
builder: Callable[[], Awaitable[dict[str, Any]]],
policy: EarthLayerCachePolicy,
) -> dict[str, Any]:
payload = await builder()
return apply_payload_budget(payload, policy)
def _write_fresh_and_stale(key: str, payload: dict[str, Any], policy: EarthLayerCachePolicy) -> None:
earth_layer_cache.set_json(key, payload, policy.fresh_ttl_seconds)
earth_layer_cache.set_json(earth_layer_cache.stale_key(key), payload, policy.stale_ttl_seconds)
def _read_stale(key: str) -> dict[str, Any] | None:
try:
return earth_layer_cache.get_json(earth_layer_cache.stale_key(key))
except Exception:
return None
def _truncate_features(payload: dict[str, Any], max_features: int, reason: str) -> dict[str, Any]:
features = payload.get("features")
if not isinstance(features, list) or len(features) <= max_features:
return payload
return _with_budget_diagnostics(
{**payload, "features": features[:max_features]},
truncated=True,
reason=reason,
original_feature_count=len(features),
)
def _with_budget_diagnostics(
payload: dict[str, Any],
*,
truncated: bool,
reason: str,
original_feature_count: int | None = None,
bytes_before: int | None = None,
bytes_after: int | None = None,
) -> dict[str, Any]:
diagnostics = dict(payload.get("diagnostics") or {})
diagnostics.update(
{
"truncated": bool(truncated or diagnostics.get("truncated")),
"limit_reason": reason,
}
)
if original_feature_count is not None:
diagnostics["original_feature_count"] = original_feature_count
if bytes_before is not None:
diagnostics["bytes_before_budget"] = bytes_before
if bytes_after is not None:
diagnostics["bytes_after_budget"] = bytes_after
return {**payload, "diagnostics": diagnostics}
def _result(payload: dict[str, Any], *, state: str, key: str) -> EarthLayerCacheResult:
return EarthLayerCacheResult(
payload=payload,
state=state,
key=key,
features=_feature_count(payload),
bytes=_payload_size(payload),
)
def _feature_count(payload: dict[str, Any]) -> int:
features = payload.get("features")
if isinstance(features, list):
return len(features)
count = payload.get("count")
return int(count) if isinstance(count, int) else 0
def _payload_size(payload: dict[str, Any]) -> int:
return len(json.dumps(payload, ensure_ascii=False, default=str).encode("utf-8"))
def _safe_key_part(value: Any) -> str:
raw = str(value).strip().lower()
return "".join(char if char.isalnum() or char in {"-", "_", ".", ","} else "_" for char in raw)[:160]
def _layer_from_key(key: str) -> str:
prefix = f"{EARTH_LAYER_CACHE_PREFIX}:"
if not key.startswith(prefix):
return "unknown"
remainder = key[len(prefix):]
return remainder.split(":", 1)[0]
def _log_cache_event(state: str, key: str, payload: dict[str, Any], started: float) -> None:
logger.info_event(
"Earth layer cache resolved",
event="earth_layer_cache.resolved",
context={
"state": state,
"key": key,
"features": _feature_count(payload),
"bytes": _payload_size(payload),
"duration_ms": round((time.perf_counter() - started) * 1000, 2),
},
)

View File

@@ -1,7 +1,7 @@
from __future__ import annotations
import asyncio
from dataclasses import dataclass
from dataclasses import dataclass, field
from datetime import UTC, datetime
from email.utils import parsedate_to_datetime
import hashlib
@@ -18,6 +18,7 @@ from bs4 import BeautifulSoup
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.countries import COUNTRY_VARIANTS_MAP, get_country_centroid, normalize_country
from app.ai_tasks.prompts import EffectiveAIPrompt, get_effective_prompt
from app.schemas.ai import SituationalAnalysisRequest
from app.services.ai_client import AIProviderClient
from app.services.location.resolvers.nominatim import build_default_nominatim_geocoder
@@ -31,6 +32,8 @@ STALE_CACHE_MAX_AGE_SECONDS = 60 * 45
RSS_SUPPLEMENT_MAX_AGE_SECONDS = STALE_CACHE_MAX_AGE_SECONDS
MAX_TARGET_INFERENCE_CONCURRENCY = 3
TARGET_INFERENCE_TIMEOUT_SECONDS = 6.0
DEFAULT_NEWS_LOCALE = "zh-CN"
NEWS_ENRICH_PROMPT_KEY = "earth.news.enrich"
@dataclass(frozen=True)
@@ -82,6 +85,11 @@ class ParsedNewsItem:
feed_region: str
homepage_url: str
published_at: datetime | None
content_language: str = "en"
localizations: dict[str, dict[str, str]] = field(default_factory=dict)
enrichment_status: str = "pending"
enrichment_error: str | None = None
enriched_at: datetime | None = None
target_location: NewsTargetLocation | None = None
target_resolution_stage: str = "unresolved"
target_ai_attempted: bool = False
@@ -365,6 +373,65 @@ def _first_json_object(text: str) -> dict[str, Any] | None:
return None
def _normalize_localizations(value: Any) -> dict[str, dict[str, str]]:
if not isinstance(value, dict):
return {}
normalized: dict[str, dict[str, str]] = {}
for locale, payload in value.items():
locale_key = _coerce_str(locale)
if not locale_key or not isinstance(payload, dict):
continue
title = _coerce_str(payload.get("title"))
summary = _coerce_str(payload.get("summary"))
entry: dict[str, str] = {}
if title:
entry["title"] = title
if summary:
entry["summary"] = summary
if entry:
normalized[locale_key] = entry
return normalized
def _get_locale_text(
item: ParsedNewsItem,
key: str,
*,
locale: str = DEFAULT_NEWS_LOCALE,
) -> str:
localized = item.localizations.get(locale)
if isinstance(localized, dict):
value = _coerce_str(localized.get(key))
if value:
return value
return ""
def _has_default_localization(item: ParsedNewsItem) -> bool:
localized = item.localizations.get(DEFAULT_NEWS_LOCALE)
if not isinstance(localized, dict):
return False
return bool(_coerce_str(localized.get("title")) and _coerce_str(localized.get("summary")))
def apply_enrichment_patch_to_item(
item: ParsedNewsItem,
patch: dict[str, Any],
) -> ParsedNewsItem:
item.location_patch = patch
if "content_language" in patch:
item.content_language = _coerce_str(patch.get("content_language")) or item.content_language
if "localizations" in patch:
item.localizations = _normalize_localizations(patch.get("localizations"))
if "enrichment_status" in patch:
item.enrichment_status = _coerce_str(patch.get("enrichment_status")) or item.enrichment_status
if "enrichment_error" in patch:
item.enrichment_error = _coerce_str(patch.get("enrichment_error"))
if "enriched_at" in patch:
item.enriched_at = _parse_datetime(_coerce_str(patch.get("enriched_at")))
return item
async def _geocode_target_location(query: str) -> dict[str, Any] | None:
return await asyncio.to_thread(_news_target_geocode, query)
@@ -513,42 +580,60 @@ async def _infer_news_target_location(
item: ParsedNewsItem,
*,
provider_client: AIProviderClient | None,
prompt: EffectiveAIPrompt | None = None,
) -> NewsTargetLocation | None:
target, _localizations = await _infer_news_enrichment(
item,
provider_client=provider_client,
prompt=prompt,
)
return target
async def _infer_news_enrichment(
item: ParsedNewsItem,
*,
provider_client: AIProviderClient | None,
prompt: EffectiveAIPrompt | None = None,
) -> tuple[NewsTargetLocation | None, dict[str, dict[str, str]]]:
text_hint = await _extract_target_location_from_text(item)
content_error: str | None = None
if text_hint is not None and text_hint.city:
item.target_resolution_stage = text_hint.source
item.target_ai_attempted = False
item.target_ai_status = "skipped_text_hint"
item.target_ai_error = None
item.target_debug_note = f"text hint matched {text_hint.label}"
return text_hint
localizations: dict[str, dict[str, str]] = {}
if provider_client is None:
item.target_resolution_stage = text_hint.source if text_hint is not None else "unresolved"
item.target_ai_attempted = False
item.target_ai_status = "unavailable"
item.target_ai_error = "AI provider is not configured or unavailable for earth-feed."
if text_hint is None or not text_hint.city:
item.target_resolution_stage = text_hint.source if text_hint is not None else "unresolved"
item.target_ai_attempted = False
item.target_ai_status = "unavailable"
item.target_ai_error = "AI provider is not configured or unavailable for earth-feed."
item.target_debug_note = (
f"text hint matched {text_hint.label}" if text_hint is not None else "no text location hint matched"
)
item.enrichment_status = "unavailable"
item.enrichment_error = "AI provider is not configured or unavailable for earth-feed."
return text_hint, localizations
if text_hint is None or not text_hint.city:
item.target_ai_attempted = True
item.target_ai_status = "attempted"
item.target_ai_error = None
item.target_debug_note = (
f"text hint matched {text_hint.label}" if text_hint is not None else "no text location hint matched"
)
return text_hint
item.target_ai_attempted = True
item.target_ai_status = "attempted"
item.target_ai_error = None
item.target_debug_note = (
f"text hint matched {text_hint.label}" if text_hint is not None else "no text location hint matched"
)
item.enrichment_status = "attempted"
item.enrichment_error = None
prompt = prompt or await get_effective_prompt(None, NEWS_ENRICH_PROMPT_KEY)
request = SituationalAnalysisRequest(
title="Infer likely event location for Earth news cruise",
objective=(
"Return exactly one strict JSON object for the most likely physical "
"location the news event is about. Prefer the host city when a state "
"visit, summit, meeting, attack, or disaster is clearly centered in a "
"known city. Fall back to the best-supported country only when a city "
"cannot be inferred."
),
title="Enrich Earth news item with event location and zh-CN content",
objective=prompt.prompt,
system_prompt=prompt.system_prompt or None,
context={
"news_item": {
"title": item.title,
@@ -564,17 +649,29 @@ async def _infer_news_target_location(
),
},
"required_json_schema": {
"country": "string|null",
"city": "string|null",
"matched_location_name": "string|null",
"latitude": "number|null",
"longitude": "number|null",
"confidence": "number from 0 to 1",
"reasoning_summary": "short string",
"location": {
"country": "string|null",
"city": "string|null",
"matched_location_name": "string|null",
"latitude": "number|null",
"longitude": "number|null",
"confidence": "number from 0 to 1",
"reasoning_summary": "short string",
},
"localizations": {
"zh-CN": {
"title": "faithful Simplified Chinese title",
"summary": "one-sentence newswire-style Simplified Chinese lead summary",
}
},
},
},
constraints=[
"Return only strict JSON. Do not wrap it in markdown.",
"For localizations, do not add facts that are absent from the RSS headline, description, source, or date.",
"Write zh-CN summary as one concise newswire-style sentence, like a breaking-news lead.",
"If the RSS description is thin, write a conservative one-sentence summary that says only what is supported.",
"Keep zh-CN summary factual, non-promotional, and avoid colon-heavy keyword labels.",
"Prefer the event location, not the newsroom or publisher headquarters.",
"When a country visit or summit is the clear topic but the city is omitted, use the most likely host city only if it is broadly public knowledge.",
"Use null for unknown fields instead of inventing details.",
@@ -584,40 +681,68 @@ async def _infer_news_target_location(
try:
response = await provider_client.analyze(request)
except Exception as exc:
item.target_resolution_stage = text_hint.source if text_hint is not None else "unresolved"
item.target_ai_status = "provider_error"
item.target_ai_error = str(exc)
return text_hint
if text_hint is None or not text_hint.city:
item.target_resolution_stage = text_hint.source if text_hint is not None else "unresolved"
item.target_ai_status = "provider_error"
item.target_ai_error = str(exc)
item.enrichment_status = "provider_error"
item.enrichment_error = str(exc)
return text_hint, localizations
payload = _first_json_object(response.content)
if not isinstance(payload, dict):
item.target_resolution_stage = text_hint.source if text_hint is not None else "unresolved"
item.target_ai_status = "parse_error"
item.target_ai_error = "AI response did not contain a parseable JSON object."
return text_hint
if text_hint is None or not text_hint.city:
item.target_resolution_stage = text_hint.source if text_hint is not None else "unresolved"
item.target_ai_status = "parse_error"
item.target_ai_error = "AI response did not contain a parseable JSON object."
item.enrichment_status = "parse_error"
item.enrichment_error = "AI response did not contain a parseable JSON object."
return text_hint, localizations
target = await _build_target_location_from_payload(payload)
localizations = _normalize_localizations(payload.get("localizations"))
if not localizations:
content_error = "AI returned no usable localizations."
location_payload = payload.get("location") if isinstance(payload.get("location"), dict) else payload
if text_hint is not None and text_hint.city:
target = text_hint
else:
target = await _build_target_location_from_payload(location_payload)
if target is None:
item.target_resolution_stage = text_hint.source if text_hint is not None else "unresolved"
item.target_ai_status = "no_result"
item.target_ai_error = "AI returned no usable target coordinates or geocodeable location."
return text_hint
if target.confidence is not None and target.confidence < 0.45:
target = text_hint
elif target.confidence is not None and target.confidence < 0.45:
item.target_resolution_stage = text_hint.source if text_hint is not None else "unresolved"
item.target_ai_status = "low_confidence"
item.target_ai_error = f"AI target confidence too low: {target.confidence:.2f}"
return text_hint
item.target_resolution_stage = target.source
item.target_ai_status = "success"
item.target_ai_error = None
item.target_debug_note = f"ai inferred {target.label}"
return target
target = text_hint
else:
item.target_resolution_stage = target.source
item.target_ai_status = "success"
item.target_ai_error = None
item.target_debug_note = f"ai inferred {target.label}"
item.localizations = localizations
if localizations and item.target_ai_status in {"success", "skipped_text_hint"}:
item.enrichment_status = "success"
item.enrichment_error = None
elif localizations:
item.enrichment_status = "content_only"
item.enrichment_error = item.target_ai_error
else:
item.enrichment_status = "location_only" if target is not None else "no_result"
item.enrichment_error = content_error or item.target_ai_error
item.enriched_at = datetime.now(UTC) if localizations else None
return target, localizations
async def _enrich_items_with_target_locations(
items: list[ParsedNewsItem],
*,
provider_client: AIProviderClient | None,
prompt: EffectiveAIPrompt | None = None,
) -> list[ParsedNewsItem]:
if not items:
return items
@@ -626,7 +751,11 @@ async def _enrich_items_with_target_locations(
async def enrich(item: ParsedNewsItem) -> ParsedNewsItem:
async with semaphore:
target = await _infer_news_target_location(item, provider_client=provider_client)
target = await _infer_news_target_location(
item,
provider_client=provider_client,
prompt=prompt,
)
item.target_location = target
return item
@@ -724,7 +853,7 @@ def _parse_feed_entries(xml_text: str, source: NewsFeedSource) -> list[ParsedNew
if not clean_title or not link:
continue
item_source = _normalize_source_name(clean_title, source.name)
item_source = source.name
display_title = clean_title
if source.source_type == "aggregated" and " - " in clean_title:
parts = clean_title.rsplit(" - ", 1)
@@ -783,6 +912,20 @@ def _serialize_target(target: NewsTargetLocation | None) -> dict[str, Any] | Non
}
def _serialize_enriched_at(value: datetime | None) -> str | None:
return value.isoformat().replace("+00:00", "Z") if value else None
def _content_patch(item: ParsedNewsItem) -> dict[str, Any]:
return {
"content_language": item.content_language,
"localizations": item.localizations,
"enrichment_status": item.enrichment_status,
"enrichment_error": item.enrichment_error,
"enriched_at": _serialize_enriched_at(item.enriched_at),
}
def build_anchor_location_patch(
item: ParsedNewsItem,
*,
@@ -798,6 +941,9 @@ def build_anchor_location_patch(
resolution_stage = item.target_resolution_stage
ai_status = item.target_ai_status
debug_note = item.target_debug_note
content_patch = _content_patch(item)
if queued and content_patch["enrichment_status"] == "pending":
content_patch["enrichment_status"] = "queued"
return {
"latitude": anchor.latitude,
"longitude": anchor.longitude,
@@ -814,6 +960,7 @@ def build_anchor_location_patch(
"target": None,
"anchor": _serialize_anchor(anchor),
},
**content_patch,
}
@@ -836,6 +983,7 @@ def build_target_location_patch(item: ParsedNewsItem, target: NewsTargetLocation
"target": _serialize_target(target),
"anchor": _serialize_anchor(anchor),
},
**_content_patch(item),
}
@@ -845,6 +993,11 @@ def build_target_location_job_payload(item: ParsedNewsItem) -> dict[str, Any]:
"id": item.id,
"title": item.title,
"summary": item.summary,
"content_language": item.content_language,
"localizations": item.localizations,
"enrichment_status": item.enrichment_status,
"enrichment_error": item.enrichment_error,
"enriched_at": _serialize_enriched_at(item.enriched_at),
"url": item.url,
"source": item.source,
"feed_name": item.feed_name,
@@ -859,6 +1012,11 @@ def parsed_news_item_from_job_payload(payload: dict[str, Any]) -> ParsedNewsItem
id=str(payload.get("id") or ""),
title=str(payload.get("title") or ""),
summary=str(payload.get("summary") or ""),
content_language=str(payload.get("content_language") or "en"),
localizations=_normalize_localizations(payload.get("localizations")),
enrichment_status=str(payload.get("enrichment_status") or "pending"),
enrichment_error=_coerce_str(payload.get("enrichment_error")),
enriched_at=_parse_datetime(_coerce_str(payload.get("enriched_at"))),
url=str(payload.get("url") or ""),
source=str(payload.get("source") or ""),
feed_name=str(payload.get("feed_name") or ""),
@@ -875,10 +1033,15 @@ def _serialize_item(item: ParsedNewsItem, *, active_region: str) -> dict[str, An
"id": item.id,
"title": item.title,
"summary": item.summary,
"content_language": item.content_language,
"localizations": item.localizations,
"display_title": _get_locale_text(item, "title"),
"display_summary": _get_locale_text(item, "summary"),
"url": item.url,
"source": item.source,
"feed_name": item.feed_name,
"region": item.feed_region,
"display_region": get_region_anchor(item.feed_region).label,
"homepage_url": item.homepage_url,
"published_at": published_at.isoformat().replace("+00:00", "Z") if published_at else None,
"latitude": location_patch["latitude"],
@@ -887,6 +1050,9 @@ def _serialize_item(item: ParsedNewsItem, *, active_region: str) -> dict[str, An
"location_source": location_patch["location_source"],
"verified": location_patch["verified"],
"location_meta": location_patch["location_meta"],
"enrichment_status": item.enrichment_status,
"enrichment_error": item.enrichment_error,
"enriched_at": _serialize_enriched_at(item.enriched_at),
"is_focus_match": item.feed_region == active_region,
}
@@ -911,6 +1077,7 @@ def _build_payload(
"lon": lon,
"region": active_region,
"label": profile.label,
"display_region": get_region_anchor(active_region).label,
"accent": profile.accent,
},
"sources": _serialize_sources(sources),
@@ -966,13 +1133,27 @@ async def _apply_cached_locations_and_enqueue(items: list[ParsedNewsItem]) -> li
get_cached_target_location_patch,
)
async def enqueue_item(item: ParsedNewsItem, *, force: bool = False) -> bool:
return await enqueue_target_location_job(build_target_location_job_payload(item), force=force)
async def apply_location(item: ParsedNewsItem) -> ParsedNewsItem:
cached_patch = await get_cached_target_location_patch(item.id)
if cached_patch:
item.location_patch = cached_patch
apply_enrichment_patch_to_item(item, cached_patch)
if not _has_default_localization(item):
queued = await enqueue_item(item, force=True)
if queued and item.enrichment_status in {
"pending",
"unavailable",
"provider_error",
"parse_error",
"no_result",
"location_only",
}:
item.enrichment_status = "queued"
return item
queued = await enqueue_target_location_job(build_target_location_job_payload(item))
queued = await enqueue_item(item)
item.location_patch = build_anchor_location_patch(
item,
queued=queued,
@@ -991,9 +1172,16 @@ async def _enqueue_unverified_locations(items: list[ParsedNewsItem]) -> None:
await asyncio.gather(
*(
enqueue_target_location_job(build_target_location_job_payload(item))
enqueue_target_location_job(
build_target_location_job_payload(item),
force=not _has_default_localization(item),
)
for item in items
if item.location_patch is None or item.location_patch.get("verified") is False
if (
item.location_patch is None
or item.location_patch.get("verified") is False
or not _has_default_localization(item)
)
)
)

View File

@@ -32,7 +32,7 @@ class NewsTargetLocationMessage:
class NewsTargetLocationQueue(Protocol):
async def enqueue(self, *, item_id: str, payload: dict[str, Any]) -> bool:
async def enqueue(self, *, item_id: str, payload: dict[str, Any], force: bool = False) -> bool:
...
async def consume_batch(
@@ -91,9 +91,11 @@ class RedisStreamsNewsTargetLocationQueue:
raise
self._group_ready = True
async def enqueue(self, *, item_id: str, payload: dict[str, Any]) -> bool:
async def enqueue(self, *, item_id: str, payload: dict[str, Any], force: bool = False) -> bool:
await self._ensure_group()
if await self.client.exists(_result_key(item_id)):
if force:
await self.client.delete(_result_key(item_id), _queued_key(item_id))
elif await self.client.exists(_result_key(item_id)):
return False
queued = await self.client.set(
_queued_key(item_id),
@@ -187,13 +189,13 @@ def get_news_target_location_queue() -> NewsTargetLocationQueue:
return RedisStreamsNewsTargetLocationQueue()
async def enqueue_target_location_job(payload: dict[str, Any]) -> bool:
async def enqueue_target_location_job(payload: dict[str, Any], *, force: bool = False) -> bool:
item_id = str(payload.get("id") or "")
if not item_id:
return False
try:
queue = get_news_target_location_queue()
return await queue.enqueue(item_id=item_id, payload=payload)
return await queue.enqueue(item_id=item_id, payload=payload, force=force)
except Exception as exc:
logger.warning_event(
"Failed to enqueue Earth news target location job",

View File

@@ -9,6 +9,7 @@ from sqlalchemy.ext.asyncio import AsyncSession
from app.models.earth_news import EarthNewsItem
from app.services.earth_news import (
ParsedNewsItem,
apply_enrichment_patch_to_item,
build_anchor_location_patch,
)
@@ -33,7 +34,7 @@ def _location_patch_from_record(record: EarthNewsItem) -> dict[str, Any]:
def record_to_parsed_news_item(record: EarthNewsItem) -> ParsedNewsItem:
return ParsedNewsItem(
item = ParsedNewsItem(
id=record.id,
title=record.title,
summary=record.summary or "",
@@ -43,8 +44,13 @@ def record_to_parsed_news_item(record: EarthNewsItem) -> ParsedNewsItem:
feed_region=record.region or "global",
homepage_url=record.homepage_url or "",
published_at=_coerce_datetime(record.published_at),
location_patch=_location_patch_from_record(record),
content_language=record.content_language or "en",
localizations=dict(record.localizations or {}),
enrichment_status=record.enrichment_status or "pending",
enrichment_error=record.enrichment_error,
enriched_at=_coerce_datetime(record.enriched_at),
)
return apply_enrichment_patch_to_item(item, _location_patch_from_record(record))
def _query_sort_key(active_region: str):
@@ -108,6 +114,8 @@ async def upsert_earth_news_items(db: AsyncSession, items: list[ParsedNewsItem])
id=item.id,
title=item.title,
summary=item.summary,
content_language=item.content_language,
localizations=dict(item.localizations or {}),
url=item.url,
source=item.source,
feed_name=item.feed_name,
@@ -122,6 +130,9 @@ async def upsert_earth_news_items(db: AsyncSession, items: list[ParsedNewsItem])
location_meta=patch["location_meta"],
first_seen_at=now,
last_seen_at=now,
enrichment_status=item.enrichment_status,
enrichment_error=item.enrichment_error,
enriched_at=item.enriched_at,
)
db.add(record)
changed += 1
@@ -136,6 +147,12 @@ async def upsert_earth_news_items(db: AsyncSession, items: list[ParsedNewsItem])
record.homepage_url = item.homepage_url
record.published_at = item.published_at
record.last_seen_at = now
if item.localizations:
record.content_language = item.content_language
record.localizations = dict(item.localizations or {})
record.enrichment_status = item.enrichment_status
record.enrichment_error = item.enrichment_error
record.enriched_at = item.enriched_at
changed += 1
await db.flush()
return changed
@@ -161,6 +178,45 @@ async def update_earth_news_item_location(
return True
async def update_earth_news_item_enrichment(
db: AsyncSession,
*,
item_id: str,
patch: dict[str, Any],
) -> bool:
record = await db.get(EarthNewsItem, item_id)
if record is None:
return False
if "latitude" in patch:
record.latitude = float(patch["latitude"])
record.longitude = float(patch["longitude"])
record.location_label = str(patch["location_label"])
record.location_source = str(patch["location_source"])
record.verified = bool(patch["verified"])
record.location_meta = dict(patch.get("location_meta") or {})
record.resolved_at = datetime.now(UTC) if record.verified else None
if "content_language" in patch:
record.content_language = str(patch.get("content_language") or "en")
if "localizations" in patch:
record.localizations = dict(patch.get("localizations") or {})
if "enrichment_status" in patch:
record.enrichment_status = str(patch.get("enrichment_status") or "pending")
if "enrichment_error" in patch:
record.enrichment_error = patch.get("enrichment_error")
if patch.get("enriched_at"):
try:
parsed_enriched_at = datetime.fromisoformat(
str(patch["enriched_at"]).replace("Z", "+00:00")
)
except ValueError:
parsed_enriched_at = datetime.now(UTC)
record.enriched_at = _coerce_datetime(parsed_enriched_at)
elif patch.get("localizations"):
record.enriched_at = datetime.now(UTC)
await db.flush()
return True
async def list_unverified_earth_news_items(
db: AsyncSession,
*,

View File

@@ -9,8 +9,10 @@ from app.core.logging import get_logger
from app.core.websocket.broadcaster import broadcaster
from app.db.session import async_session_factory
from app.services.ai_client import AIProviderClient
from app.ai_tasks.prompts import get_effective_prompt
from app.services.earth_news import (
_infer_news_target_location,
NEWS_ENRICH_PROMPT_KEY,
_infer_news_enrichment,
build_target_location_patch,
parsed_news_item_from_job_payload,
)
@@ -19,7 +21,7 @@ from app.services.earth_news_queue import (
get_news_target_location_queue,
save_target_location_patch,
)
from app.services.earth_news_store import update_earth_news_item_location
from app.services.earth_news_store import update_earth_news_item_enrichment as update_earth_news_item_location
logger = get_logger(__name__, service="earth_news")
@@ -59,8 +61,15 @@ async def process_target_location_message(
provider_client: AIProviderClient | None,
) -> dict[str, Any]:
item = parsed_news_item_from_job_payload(message.payload)
target = await _infer_news_target_location(item, provider_client=provider_client)
async with async_session_factory() as session:
prompt = await get_effective_prompt(session, NEWS_ENRICH_PROMPT_KEY)
target, localizations = await _infer_news_enrichment(
item,
provider_client=provider_client,
prompt=prompt,
)
item.target_location = target
item.localizations = localizations or item.localizations
patch = build_target_location_patch(item, target)
await save_target_location_patch(item.id, patch)
async with async_session_factory() as session:

View File

@@ -7,6 +7,26 @@ from typing import Any
import httpx
MODELS_DEV_URL = "https://models.dev/api.json"
OPENCODE_GO_MODELS_URL = "https://opencode.ai/zen/go/v1/models"
OPENCODE_GO_MODEL_PROVIDER_APIS = {
"minimax-m2.7": "anthropic-messages",
"minimax-m2.5": "anthropic-messages",
}
OPENCODE_GO_FALLBACK_MODELS = [
"minimax-m2.7",
"minimax-m2.5",
"kimi-k2.6",
"kimi-k2.5",
"glm-5.1",
"glm-5",
"deepseek-v4-pro",
"deepseek-v4-flash",
"qwen3.6-plus",
"qwen3.5-plus",
"mimo-v2.5-pro",
"mimo-v2.5",
]
FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
@@ -80,6 +100,17 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
"api_key_env": "OPENROUTER_API_KEY",
"source": "fallback",
},
"opencode-go": {
"provider": "opencode-go",
"label": "OpenCode Go",
"provider_api": "openai-completions",
"base_url": "https://opencode.ai/zen/go/v1",
"model": "glm-5.1",
"models": OPENCODE_GO_FALLBACK_MODELS,
"model_provider_apis": OPENCODE_GO_MODEL_PROVIDER_APIS,
"api_key_env": "OPENCODE_GO_API_KEY",
"source": "fallback",
},
"ollama": {
"provider": "ollama",
"label": "Ollama Local",
@@ -114,8 +145,43 @@ def get_fallback_llm_provider_preset(provider: str) -> dict[str, Any]:
return dict(FALLBACK_LLM_PROVIDER_PRESETS[key])
async def refresh_llm_provider_preset(provider: str) -> dict[str, Any]:
def _opencode_go_model_provider_apis(model_ids: list[str]) -> dict[str, str]:
return {
model_id: OPENCODE_GO_MODEL_PROVIDER_APIS.get(model_id, "openai-completions")
for model_id in model_ids
}
async def refresh_llm_provider_preset(provider: str, api_key: str | None = None) -> dict[str, Any]:
fallback = get_fallback_llm_provider_preset(provider)
if fallback["provider"] == "opencode-go":
headers = {"User-Agent": "Planet/1.0"}
if api_key:
headers["Authorization"] = f"Bearer {api_key}"
async with httpx.AsyncClient(timeout=15.0, follow_redirects=True) as client:
response = await client.get(
OPENCODE_GO_MODELS_URL,
headers=headers,
)
response.raise_for_status()
payload = response.json()
data = payload.get("data") if isinstance(payload, dict) else []
model_ids = [
str(item.get("id"))
for item in data
if isinstance(item, dict) and item.get("id")
][:120]
if not model_ids:
model_ids = fallback["models"]
return {
**fallback,
"model": fallback["model"] if fallback["model"] in model_ids else model_ids[0],
"models": model_ids,
"model_provider_apis": _opencode_go_model_provider_apis(model_ids),
"source": OPENCODE_GO_MODELS_URL,
}
models_dev_key = MODELS_DEV_PROVIDER_KEYS.get(fallback["provider"])
if not models_dev_key:
return fallback

View File

@@ -7,8 +7,12 @@ import re
from dataclasses import dataclass
from typing import Any, Iterable
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.logging import get_logger
from app.core.countries import COUNTRY_ENTRIES, normalize_country
from app.schemas.ai import SituationalAnalysisRequest
from app.ai_tasks.prompts import get_effective_prompt
from app.services.ai_client import AIProviderClient
from app.services.ai_tools.evidence_store import normalize_search_evidence
from app.services.ai_tools.web_search import WebSearchClient, WebSearchError
@@ -23,7 +27,12 @@ from app.services.location.text import (
VALID_LLM_PRECISIONS = {"precise", "site", "city"}
DEFAULT_MIN_CONFIDENCE = 0.55
LOCATION_NORMALIZE_PROMPT_KEY = "location.factcheck.normalize"
LOCATION_RESOLVE_PROMPT_KEY = "location.factcheck.resolve"
MODEL_CONFIDENCE_WEIGHT = 0.25
LOG_TEXT_LIMIT = 1200
LOG_EVIDENCE_LIMIT = 5
logger = get_logger(__name__, service="location")
_geocode_llm_city = build_default_nominatim_geocoder()
_LLM_LOCATION_NAME_KEYS = (
"matched_location_name",
@@ -92,6 +101,35 @@ class LocationEvidenceScore:
summary: str
def _truncate_log_text(value: Any, limit: int = LOG_TEXT_LIMIT) -> str:
text = coerce_str(value)
if len(text) <= limit:
return text
return f"{text[:limit]}"
def _summarize_search_evidence(evidence: list[dict[str, Any]] | None) -> list[dict[str, Any]]:
items: list[dict[str, Any]] = []
for item in (evidence or [])[:LOG_EVIDENCE_LIMIT]:
if not isinstance(item, dict):
continue
items.append(
{
"title": _truncate_log_text(item.get("title"), 180),
"source": _truncate_log_text(item.get("source") or item.get("name"), 120),
"url": _truncate_log_text(item.get("url"), 240),
"snippet": _truncate_log_text(
item.get("snippet")
or item.get("content")
or item.get("text")
or item.get("summary"),
360,
),
}
)
return items
def _first_json_object(text: str) -> dict[str, Any] | None:
stripped = text.strip()
if not stripped:
@@ -157,6 +195,64 @@ def _evidence_label(item: Any) -> str:
return coerce_str(item)
def _evidence_text(item: dict[str, Any]) -> str:
return " ".join(
coerce_str(item.get(key))
for key in ("title", "source", "name", "url", "snippet", "content", "text", "quote", "summary")
if coerce_str(item.get(key))
)
def _search_evidence_entity_match(item: dict[str, Any], query: LocationQuery) -> bool:
haystack = normalize_text(_evidence_text(item))
if not haystack:
return False
needles = [
coerce_str(query.name),
*[coerce_str(alias) for alias in query.aliases],
]
return any(normalize_text(needle) and normalize_text(needle) in haystack for needle in needles)
def _evidence_has_location_assertion(item: dict[str, Any], city: str) -> bool:
normalized_city = normalize_text(city)
text = normalize_text(_evidence_text(item))
if not normalized_city or normalized_city not in text:
return False
assertion_terms = (
"located",
"situated",
"built",
"hosted",
"deployed",
"installed",
"facility",
"campus",
"site",
"data center",
"datacenter",
"supercomputer center",
"位于",
"位於",
"坐落",
"建置",
"設置",
"设置",
)
return any(term in text for term in assertion_terms)
def _city_is_unsupported_name_hint(payload: dict[str, Any], query: LocationQuery, evidence_items: list[dict[str, Any]]) -> bool:
city = coerce_str(payload.get("city") or query.city)
if not city:
return False
normalized_city = normalize_text(city)
normalized_name = normalize_text(query.name)
if not normalized_city or not normalized_name or normalized_city not in normalized_name:
return False
return not any(_evidence_has_location_assertion(item, city) for item in evidence_items)
def _normalize_llm_precision(value: Any) -> str:
text = coerce_str(value).lower()
return LLM_PRECISION_ALIASES.get(text, text)
@@ -583,6 +679,7 @@ def _weak_evidence_penalty(
payload: dict[str, Any],
evidence_items: list[dict[str, Any]],
*,
query: LocationQuery,
entity_match: float,
geography_match: float,
conflict_penalty: float,
@@ -593,6 +690,8 @@ def _weak_evidence_penalty(
penalty += 0.20
if any(_truthy_evidence_field(item, "ambiguous") for item in evidence_items):
penalty += 0.15
if _city_is_unsupported_name_hint(payload, query, evidence_items):
penalty += 0.10
if conflict_penalty == 0.0 and entity_match > 0 and geography_match >= 0.20:
return min(penalty, 0.15)
return min(penalty, 0.30)
@@ -615,6 +714,7 @@ def _score_llm_location_payload(
weak_evidence_penalty = _weak_evidence_penalty(
payload,
evidence_items,
query=query,
entity_match=entity_match,
geography_match=geography_match,
conflict_penalty=conflict_penalty,
@@ -631,6 +731,8 @@ def _score_llm_location_payload(
- weak_evidence_penalty
)
score = min(max(score, 0.0), 1.0)
if _city_is_unsupported_name_hint(payload, query, evidence_items):
score = min(score, 0.54)
summary = (
f"combined={score:.2f}; model={model_confidence:.2f}; "
f"source={source_quality:.2f}; entity={entity_match:.2f}; "
@@ -842,15 +944,43 @@ async def collect_location_search_evidence(
) -> LocationSearchEvidenceResult:
search_query = _location_search_query(query, entity_type)
attempt = f"web_search:{entity_type}:{search_query}"
logger.info_event(
"Collecting location search evidence",
event="location.factcheck.web_search.start",
context={
"entity_type": entity_type,
"search_query": search_query,
"location_query": _query_context(query),
"max_results": max_results,
},
)
try:
evidence = await web_search_client.search(search_query, max_results=max_results)
except WebSearchError as exc:
logger.warning_event(
"Location search evidence failed",
event="location.factcheck.web_search.failed",
context={
"entity_type": entity_type,
"search_query": search_query,
"error": str(exc),
},
)
return LocationSearchEvidenceResult(
evidence=[],
attempted_queries=[attempt],
failure_reason=f"WebSearch location evidence failed: {exc}",
)
except Exception as exc:
logger.warning_event(
"Location search evidence unavailable",
event="location.factcheck.web_search.unavailable",
context={
"entity_type": entity_type,
"search_query": search_query,
"error": str(exc),
},
)
return LocationSearchEvidenceResult(
evidence=[],
attempted_queries=[attempt],
@@ -858,11 +988,29 @@ async def collect_location_search_evidence(
)
normalized = normalize_search_evidence(evidence, limit=max_results)
if not normalized:
logger.warning_event(
"Location search returned no usable evidence",
event="location.factcheck.web_search.empty",
context={
"entity_type": entity_type,
"search_query": search_query,
},
)
return LocationSearchEvidenceResult(
evidence=[],
attempted_queries=[attempt],
failure_reason="WebSearch returned no usable location evidence.",
)
logger.info_event(
"Collected location search evidence",
event="location.factcheck.web_search.result",
context={
"entity_type": entity_type,
"search_query": search_query,
"evidence_count": len(normalized),
"evidence": _summarize_search_evidence(normalized),
},
)
return LocationSearchEvidenceResult(
evidence=normalized,
attempted_queries=[attempt],
@@ -876,6 +1024,7 @@ async def _repair_location_payload_from_text(
raw_text: str,
query: LocationQuery,
entity_type: str,
db: AsyncSession | None = None,
) -> dict[str, Any] | None:
"""Second-pass structure repair for models that answer in prose.
@@ -884,12 +1033,11 @@ async def _repair_location_payload_from_text(
"""
if not coerce_str(raw_text):
return None
prompt = await get_effective_prompt(db, LOCATION_NORMALIZE_PROMPT_KEY)
request = SituationalAnalysisRequest(
title=f"Normalize location factcheck for {entity_type}",
objective=(
"Convert the supplied location factcheck text into exactly one strict "
"JSON object. Extract only facts present in the text or original query."
),
objective=prompt.prompt,
system_prompt=prompt.system_prompt or None,
context={
"entity_type": entity_type,
"location_query": _query_context(query),
@@ -929,6 +1077,7 @@ async def collect_llm_location_fallback_candidate(
provider_client: AIProviderClient,
query: LocationQuery,
entity_type: str,
db: AsyncSession | None = None,
attempted_queries: Iterable[str] = (),
search_evidence: list[dict[str, Any]] | None = None,
min_confidence: float = DEFAULT_MIN_CONFIDENCE,
@@ -941,18 +1090,25 @@ async def collect_llm_location_fallback_candidate(
"""
attempt = f"llm_factcheck:{entity_type}:{coerce_str(query.name) or 'unknown'}"
if search_evidence is not None and not search_evidence:
logger.warning_event(
"Skipping LLM location factcheck because search evidence is empty",
event="location.factcheck.llm.skipped_no_evidence",
context={
"entity_type": entity_type,
"attempt": attempt,
"location_query": _query_context(query),
},
)
return LocationLLMFallbackResult(
candidates=[],
attempted_queries=[attempt],
failure_reason="LLM location factcheck skipped: no WebSearch evidence.",
)
prompt = await get_effective_prompt(db, LOCATION_RESOLVE_PROMPT_KEY)
request = SituationalAnalysisRequest(
title=f"Location factcheck fallback for {entity_type}",
objective=(
"Return exactly one JSON object for the most likely physical location. "
"Use only fact-checkable public knowledge; return null fields rather "
"than guessing when evidence is weak."
),
objective=prompt.prompt,
system_prompt=prompt.system_prompt or None,
context={
"entity_type": entity_type,
"location_query": _query_context(query),
@@ -982,31 +1138,86 @@ async def collect_llm_location_fallback_candidate(
"Return evidence as objects when possible, including source, url, source_type, and entity_match.",
"Include source names or URLs in evidence when known. The backend will recompute the final confidence from model confidence plus evidence quality.",
"If search_evidence is provided, use only that evidence as factual support.",
"Do not treat a website footer, office address, publisher address, or contact address as the entity's physical location.",
"If the entity name contains a city name, do not choose that city unless evidence explicitly says the entity/facility/supercomputer is located, hosted, built, deployed, or installed there.",
"Prefer the facility/site if known; otherwise use the best supported city.",
],
)
logger.info_event(
"Sending location factcheck request to LLM",
event="location.factcheck.llm.request",
context={
"entity_type": entity_type,
"attempt": attempt,
"title": request.title,
"objective": request.objective,
"location_query": request.context.get("location_query"),
"observations": request.observations,
"constraints": request.constraints,
"search_evidence_count": len(search_evidence or []),
"search_evidence": _summarize_search_evidence(search_evidence),
},
)
try:
response = await provider_client.analyze(request)
except Exception as exc:
logger.warning_event(
"LLM location factcheck failed",
event="location.factcheck.llm.failed",
context={
"entity_type": entity_type,
"attempt": attempt,
"error": str(exc),
},
)
return LocationLLMFallbackResult(
candidates=[],
attempted_queries=[attempt],
failure_reason=f"LLM location factcheck failed: {exc}",
)
logger.info_event(
"Received location factcheck response from LLM",
event="location.factcheck.llm.response",
context={
"entity_type": entity_type,
"attempt": attempt,
"provider": response.provider,
"model": response.model,
"content": _truncate_log_text(response.content, 2000),
},
)
payload = _first_json_object(response.content)
if payload is None:
logger.warning_event(
"LLM location factcheck response was not strict JSON; attempting repair",
event="location.factcheck.llm.non_json",
context={
"entity_type": entity_type,
"attempt": attempt,
"content": _truncate_log_text(response.content, 1200),
},
)
payload = await _repair_location_payload_from_text(
provider_client=provider_client,
raw_text=response.content,
query=query,
entity_type=entity_type,
db=db,
)
if payload is None:
payload = _payload_from_free_text(response.content, query=query)
if payload is None:
if payload is None and entity_type != "compute_center":
payload = _payload_from_query_name_geocode(query)
if payload is None:
logger.warning_event(
"LLM location factcheck produced no parseable payload",
event="location.factcheck.llm.unparseable",
context={
"entity_type": entity_type,
"attempt": attempt,
},
)
return LocationLLMFallbackResult(
candidates=[],
attempted_queries=[attempt],
@@ -1027,7 +1238,7 @@ async def collect_llm_location_fallback_candidate(
"url": item.get("url"),
"text": item.get("snippet") or item.get("content"),
"source_type": "web_search",
"entity_match": True,
"entity_match": _search_evidence_entity_match(item, query),
}
for item in search_evidence
if isinstance(item, dict)
@@ -1049,6 +1260,18 @@ async def collect_llm_location_fallback_candidate(
if candidate is None:
if city_geocode_failure and rejection_reason == "missing, invalid, or zero latitude/longitude":
rejection_reason = f"{rejection_reason}; {city_geocode_failure}"
logger.warning_event(
"Rejected LLM location factcheck candidate",
event="location.factcheck.llm.rejected",
context={
"entity_type": entity_type,
"attempt": attempt,
"reason": rejection_reason,
"payload": payload,
"search_evidence_count": len(search_evidence or []),
"search_evidence": _summarize_search_evidence(search_evidence),
},
)
return LocationLLMFallbackResult(
candidates=[],
attempted_queries=[attempt],
@@ -1057,6 +1280,18 @@ async def collect_llm_location_fallback_candidate(
+ (f": {rejection_reason}." if rejection_reason else ".")
),
)
logger.info_event(
"Accepted LLM location factcheck candidate",
event="location.factcheck.llm.accepted",
context={
"entity_type": entity_type,
"attempt": attempt,
"candidate": candidate.to_dict(),
"payload": payload,
"search_evidence_count": len(search_evidence or []),
"search_evidence": _summarize_search_evidence(search_evidence),
},
)
return LocationLLMFallbackResult(
candidates=[candidate],
attempted_queries=[attempt],

View File

@@ -10,8 +10,11 @@ from app.models.alert import Alert, AlertSeverity, AlertStatus
from app.models.bgp_anomaly import BGPAnomaly
from app.models.bgp_incident import BGPIncident
from app.schemas.ai import SituationalAnalysisRequest
from app.ai_tasks.prompts import get_effective_prompt
from app.services.bgp_ai_brief_store import get_latest_bgp_brief_record
SITUATIONAL_ALERT_BRIEF_PROMPT_KEY = "alerts.situational.brief"
def _format_pairs(pairs: list[tuple[str, int]], empty_text: str = "") -> str:
if not pairs:
@@ -96,18 +99,24 @@ async def build_situational_alert_brief_request(
(str(item[0] or "未命名数据源"), item[1])
for item in alert_source_result.fetchall()
]
total_alerts = total_alerts_result.scalar() or 0
active_alerts = active_alerts_result.scalar() or 0
total_incidents = total_incidents_result.scalar() or 0
active_incidents = active_incidents_result.scalar() or 0
total_anomalies = total_anomalies_result.scalar() or 0
active_anomalies = active_anomalies_result.scalar() or 0
facts = [
(
f"系统告警侧:总告警 {total_alerts_result.scalar() or 0}active {active_alerts_result.scalar() or 0} 条;"
f"系统告警侧:总告警 {total_alerts}active {active_alerts} 条;"
f"活跃告警严重度分布为 {_format_pairs(active_alert_severities)}"
),
(
f"BGP态势侧累计 incidents {total_incidents_result.scalar() or 0}active incidents {active_incidents_result.scalar() or 0} 条;"
f"BGP态势侧累计 incidents {total_incidents}active incidents {active_incidents} 条;"
f"活跃 incidents 严重度分布为 {_format_pairs(active_bgp_severities)}"
),
(
f"BGP异常侧累计 anomalies {total_anomalies_result.scalar() or 0}active anomalies {active_anomalies_result.scalar() or 0} 条;"
f"BGP异常侧累计 anomalies {total_anomalies}active anomalies {active_anomalies} 条;"
f"活跃 anomaly 类型分布为 {_format_pairs(active_anomaly_types)}"
),
]
@@ -147,21 +156,23 @@ async def build_situational_alert_brief_request(
context = {
"source": "situational-alerts",
"active_system_alerts": active_alerts_result.scalar() or 0,
"active_system_alerts": active_alerts,
"active_system_alert_severities": dict(active_alert_severities),
"top_system_alert_sources": dict(active_alert_sources),
"active_bgp_incidents": active_incidents_result.scalar() or 0,
"active_bgp_incidents": active_incidents,
"active_bgp_incident_severities": dict(active_bgp_severities),
"active_bgp_anomalies": active_anomalies_result.scalar() or 0,
"active_bgp_anomalies": active_anomalies,
"active_bgp_anomaly_types": dict(active_anomaly_types),
"bgp_hot_regions": dict(bgp_region_counter.most_common(5)),
"latest_bgp_brief_id": latest_bgp_brief.id if latest_bgp_brief else None,
"latest_bgp_brief_generated_at": latest_bgp_brief.generated_at if latest_bgp_brief else None,
}
prompt = await get_effective_prompt(db, SITUATIONAL_ALERT_BRIEF_PROMPT_KEY)
request = SituationalAnalysisRequest(
title="态势告警 AI 简报",
objective="综合系统告警、BGP incidents、BGP anomalies 与近期 BGP AI 简报,生成一份面向值班人员的态势告警简报,指出当前最需要关注的风险域、跨模块联动迹象和优先动作。",
objective=prompt.prompt,
system_prompt=prompt.system_prompt or None,
observations=facts,
constraints=[
"明确区分事实、推断与建议。",

View File

@@ -1,6 +1,7 @@
from __future__ import annotations
import json
import os
import re
import shutil
import subprocess
@@ -86,6 +87,7 @@ class LogSource:
status: str = "ok"
buffer_key: str | None = None
container_name: str | None = None
fallback_locations: tuple[str, ...] = ()
@dataclass
@@ -104,22 +106,38 @@ class DailyLogMarker:
dominant_level: str
def _planet_state_dir() -> Path:
configured = os.getenv("PLANET_STATE_DIR")
if configured:
return Path(configured).expanduser()
xdg_state = os.getenv("XDG_STATE_HOME")
if xdg_state:
return Path(xdg_state).expanduser() / "planet"
return Path.home() / ".local" / "state" / "planet"
def _state_log_path(filename: str) -> str:
return str(_planet_state_dir() / filename)
LOG_SOURCES: dict[str, LogSource] = {
"backend": LogSource(
source_id="backend",
name="后端服务",
kind="file",
location="/tmp/planet_backend.log",
location=_state_log_path("backend.log"),
description="FastAPI 后端、调度器和采集任务共享日志。",
category="service",
fallback_locations=("/tmp/planet_backend.log",),
),
"frontend": LogSource(
source_id="frontend",
name="前端开发服务",
kind="file",
location="/tmp/planet_frontend.log",
location=_state_log_path("frontend.log"),
description="控制台与 Earth 前端开发服务输出。",
category="service",
fallback_locations=("/tmp/planet_frontend.log",),
),
"ai-provider": LogSource(
source_id="ai-provider",
@@ -164,9 +182,18 @@ def normalize_log_levels(level: str | None = None, levels: str | None = None) ->
return tuple(normalized_levels)
def resolve_file_log_path(source: LogSource) -> Path:
primary = Path(source.location).expanduser()
candidates = (primary, *(Path(item).expanduser() for item in source.fallback_locations))
for candidate in candidates:
if candidate.exists():
return candidate
return primary
def get_source_status(source: LogSource) -> str:
if source.kind == "file":
path = Path(source.location)
path = resolve_file_log_path(source)
if not path.exists():
return "missing"
return "ok" if path.stat().st_size > 0 else "empty"
@@ -190,7 +217,7 @@ def list_log_sources() -> list[dict[str, str]]:
"source_id": source.source_id,
"name": source.name,
"kind": source.kind,
"location": source.location,
"location": str(resolve_file_log_path(source)) if source.kind == "file" else source.location,
"description": source.description,
"category": source.category,
"status": get_source_status(source),
@@ -339,7 +366,7 @@ def build_buffer_entry(payload: dict[str, Any]) -> StructuredLogEntry:
def read_file_entries(source: LogSource, scan_limit: int) -> list[StructuredLogEntry]:
path = Path(source.location)
path = resolve_file_log_path(source)
if not path.exists():
return []
with path.open("r", encoding="utf-8", errors="replace") as handle:
@@ -511,7 +538,7 @@ def read_log_snapshot(
"source_id": source.source_id,
"name": source.name,
"kind": source.kind,
"location": source.location,
"location": str(resolve_file_log_path(source)) if source.kind == "file" else source.location,
"description": source.description,
"category": source.category,
"status": get_source_status(source),

View File

@@ -385,13 +385,18 @@ def build_public_tv_payload(
settings_payload: dict[str, Any],
collected_sources: list[dict[str, Any]],
) -> dict[str, Any]:
configured_by_id = {
source["id"]: source
for source in settings_payload["sources"]
if source.get("id")
}
configured_sources = [
source for source in settings_payload["sources"] if source["is_enabled"]
]
merged_by_id = {source["id"]: source for source in configured_sources}
for source in collected_sources:
if source["id"] in merged_by_id or not source["is_enabled"]:
if source["id"] in configured_by_id or not source["is_enabled"]:
continue
merged_by_id[source["id"]] = source

View File

@@ -18,7 +18,7 @@ async def create_admin():
if existing_user:
print("用户 linkong 已存在,更新密码...")
existing_user.set_password("12345678")
existing_user.set_password("LK12345678")
existing_user.role = "super_admin"
existing_user.email = "linkong@planet.local"
else:
@@ -26,7 +26,7 @@ async def create_admin():
user = User(
username="linkong",
email="linkong@planet.local",
password_hash=get_password_hash("12345678"),
password_hash=get_password_hash("LK12345678"),
role="super_admin",
is_active=True,
)

View File

@@ -19,7 +19,7 @@ DEFAULT_LOGIN_USERS = (
{
"username": "linkong",
"email": "linkong@planet.local",
"password": "12345678",
"password": "LK12345678",
"role": "super_admin",
},
)

View File

@@ -22,7 +22,7 @@ DEFAULT_LOGIN_USERS = (
{
"username": "linkong",
"email": "linkong@planet.local",
"password": "12345678",
"password": "LK12345678",
"role": "super_admin",
},
)

View File

@@ -5,10 +5,12 @@ from datetime import datetime
from unittest.mock import patch, AsyncMock
from httpx import AsyncClient, ASGITransport
from app.api.v1 import earth as earth_api
from app.main import app
from app.core.config import settings
from app.core.security import create_access_token
from app.db.session import get_db
from app.models.system_setting import SystemSetting
from app.models.user import User
from app.schemas.ai import (
AIProviderStatusResponse,
@@ -61,6 +63,148 @@ async def test_root_endpoint():
assert data["version"] == settings.VERSION
class _ScalarOneOrNoneResult:
def __init__(self, value=None):
self._value = value
def scalar_one_or_none(self):
return self._value
class _FakeEarthBrandSession:
def __init__(self, record=None):
self.record = record
self.added = None
self.deleted = False
self.committed = False
async def execute(self, statement):
if statement.__class__.__name__ == "Delete":
self.deleted = True
self.record = None
return _ScalarOneOrNoneResult(None)
return _ScalarOneOrNoneResult(self.record)
def add(self, record):
self.added = record
self.record = record
async def commit(self):
self.committed = True
async def refresh(self, _record):
return None
def _override_admin_user():
return User(id=1, username="testuser", email="test@example.com", role="admin", is_active=True)
@pytest.mark.asyncio
async def test_get_earth_brand_returns_static_defaults():
async def override_get_db():
yield _FakeEarthBrandSession()
app.dependency_overrides[get_db] = override_get_db
try:
transport = ASGITransport(app=app)
async with AsyncClient(transport=transport, base_url="http://test") as client:
response = await client.get("/api/v1/earth/brand")
assert response.status_code == 200
data = response.json()
assert data["is_default"] is True
assert data["brand"]["logo_src"] == "/earth/assets/brand/earth-logo.png"
assert data["brand"]["title_src"] == "/earth/assets/brand/title-zh.png"
finally:
app.dependency_overrides.clear()
@pytest.mark.asyncio
async def test_update_and_reset_earth_brand(auth_headers):
session = _FakeEarthBrandSession()
async def override_get_db():
yield session
app.dependency_overrides.update(
{
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: _override_admin_user,
get_db: override_get_db,
}
)
try:
transport = ASGITransport(app=app)
async with AsyncClient(transport=transport, base_url="http://test") as client:
update_response = await client.put(
"/api/v1/earth/brand",
headers=auth_headers,
json={
"logo_src": "/earth-brand-assets/custom.png",
"title_src": "",
"title_text": "Custom Earth",
"subtitle": "Custom subtitle",
"description": "Custom description",
"aria_label": "",
"title_alt": "",
},
)
reset_response = await client.delete("/api/v1/earth/brand", headers=auth_headers)
assert update_response.status_code == 200
updated = update_response.json()
assert updated["is_default"] is False
assert updated["brand"]["title_text"] == "Custom Earth"
assert updated["brand"]["aria_label"] == "Custom Earth"
assert isinstance(session.added, SystemSetting)
assert reset_response.status_code == 200
reset = reset_response.json()
assert reset["is_default"] is True
assert reset["brand"]["logo_src"] == "/earth/assets/brand/earth-logo.png"
finally:
app.dependency_overrides.clear()
@pytest.mark.asyncio
async def test_upload_earth_brand_asset_rejects_invalid_type(auth_headers):
app.dependency_overrides[
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user
] = _override_admin_user
try:
transport = ASGITransport(app=app)
async with AsyncClient(transport=transport, base_url="http://test") as client:
response = await client.post(
"/api/v1/earth/brand/assets",
headers=auth_headers,
files={"file": ("brand.txt", b"nope", "text/plain")},
)
assert response.status_code == 400
assert response.json()["detail"]["code"] == "unsupported_file_type"
finally:
app.dependency_overrides.clear()
@pytest.mark.asyncio
async def test_upload_earth_brand_asset_saves_file(auth_headers, tmp_path, monkeypatch):
monkeypatch.setattr(earth_api, "EARTH_BRAND_ASSET_DIR", tmp_path)
app.dependency_overrides[
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user
] = _override_admin_user
try:
transport = ASGITransport(app=app)
async with AsyncClient(transport=transport, base_url="http://test") as client:
response = await client.post(
"/api/v1/earth/brand/assets",
headers=auth_headers,
files={"file": ("brand.png", b"png-bytes", "image/png")},
)
assert response.status_code == 200
data = response.json()
assert data["url"].startswith("/earth-brand-assets/")
assert (tmp_path / data["filename"]).read_bytes() == b"png-bytes"
finally:
app.dependency_overrides.clear()
@pytest.mark.asyncio
async def test_dashboard_stats_without_auth():
"""Test dashboard stats requires authentication"""
@@ -528,6 +672,77 @@ async def test_ingest_earth_client_log_accepts_public_events():
app.dependency_overrides.clear()
@pytest.mark.asyncio
async def test_earth_layer_cache_status_requires_super_admin(auth_headers, monkeypatch):
def override_get_current_user():
return User(
id=1,
username="root",
email="root@example.com",
password_hash="hashed",
role="super_admin",
is_active=True,
)
monkeypatch.setattr(
"app.api.v1.system_control.earth_layer_cache.status",
lambda: {
"prefix": "earth:layer:v1",
"key_count": 2,
"memory_bytes": 42,
"layers": {"cables": {"keys": 2, "stale_keys": 1, "memory_bytes": 42}},
},
)
app.dependency_overrides = {
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
}
transport = ASGITransport(app=app)
try:
async with AsyncClient(transport=transport, base_url="http://test") as client:
response = await client.get("/api/v1/system/cache/earth-layers", headers=auth_headers)
assert response.status_code == 200
data = response.json()
assert data["prefix"] == "earth:layer:v1"
assert data["layers"]["cables"]["stale_keys"] == 1
finally:
app.dependency_overrides.clear()
@pytest.mark.asyncio
async def test_clear_earth_layer_cache_deletes_only_earth_layer_prefix(auth_headers, monkeypatch):
def override_get_current_user():
return User(
id=1,
username="root",
email="root@example.com",
password_hash="hashed",
role="super_admin",
is_active=True,
)
captured = {}
def fake_delete_pattern(pattern="earth:layer:v1:*"):
captured["pattern"] = pattern
return 3
monkeypatch.setattr("app.api.v1.system_control.earth_layer_cache.delete_pattern", fake_delete_pattern)
app.dependency_overrides = {
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
}
transport = ASGITransport(app=app)
try:
async with AsyncClient(transport=transport, base_url="http://test") as client:
response = await client.delete("/api/v1/system/cache/earth-layers", headers=auth_headers)
assert response.status_code == 200
assert response.json()["deleted"] == 3
assert captured["pattern"] == "earth:layer:v1:*"
finally:
app.dependency_overrides.clear()
@pytest.mark.asyncio
async def test_request_id_header_is_echoed_when_provided():
transport = ASGITransport(app=app)

View File

@@ -4,6 +4,7 @@ import pytest
from app.api.v1 import datasources as datasources_api
from app.models.datasource import DataSource
from app.services import earth_layer_cache as earth_cache
def make_datasource(
@@ -53,6 +54,71 @@ def test_filter_datasources_by_product_status_and_collected_state():
assert filtered == [vessels]
def test_serialize_datasource_row_can_skip_endpoint_resolution():
datasource = make_datasource(7, "arcgis_cables", last_status="success", module="L2")
class ExplodingConfig:
def get_yaml_url(self, _source):
raise AssertionError("endpoint resolution should be skipped")
row = datasources_api.serialize_datasource_row(
datasource,
running_tasks={},
latest_tasks={},
record_counts={"arcgis_cables": 343},
endpoint_overrides={},
config=ExplodingConfig(),
include_endpoint=False,
)
assert row["id"] == 7
assert row["source"] == "arcgis_cables"
assert row["module"] == "L2"
assert row["last_status"] == "success"
assert row["collected_records"] == 343
assert row["has_collected_data"] is True
assert "endpoint" not in row
def test_serialize_datasource_row_includes_endpoint_when_requested():
datasource = make_datasource(8, "arcgis_landing_points")
class Config:
def get_yaml_url(self, source):
return f"https://example.test/{source}"
row = datasources_api.serialize_datasource_row(
datasource,
running_tasks={},
latest_tasks={},
record_counts={},
endpoint_overrides={},
config=Config(),
include_endpoint=True,
)
assert row["endpoint"] == "https://example.test/arcgis_landing_points"
def test_invalidate_earth_layer_cache_for_source_covers_datasource_aliases(monkeypatch):
patterns: list[str] = []
def fake_delete_pattern(pattern: str) -> int:
patterns.append(pattern)
return 1
monkeypatch.setattr(earth_cache.earth_layer_cache, "delete_pattern", fake_delete_pattern)
deleted = earth_cache.invalidate_earth_layer_cache_for_source("telegeography_cables")
assert deleted == 3
assert patterns == [
"earth:layer:v1:cables*",
"earth:layer:v1:landing-points*",
"earth:layer:v1:summary*",
]
@pytest.mark.asyncio
async def test_trigger_datasource_batch_skips_disabled_and_frequency_window(monkeypatch):
now = datetime.now(timezone.utc)

View File

@@ -52,6 +52,19 @@ async def test_public_catalog_only_for_anonymous_user():
}
@pytest.mark.asyncio
async def test_developer_catalog_includes_frontend_reference_docs():
response = await get_json(
"/api/v1/docs/catalog",
make_user(role="viewer", groups=["docs_developer"]),
)
assert response.status_code == 200
zh_slugs = {item["slug"] for item in response.json()["items"] if item["lang"] == "zh"}
assert "naming-glossary" in zh_slugs
assert "tactile-ui-components" in zh_slugs
@pytest.mark.asyncio
async def test_anonymous_can_read_public_doc():
response = await get_json("/api/v1/docs/zh/quickstart")
@@ -83,10 +96,16 @@ async def test_developer_group_can_read_developer_but_not_admin_doc():
user = make_user(role="viewer", groups=["docs_developer"])
developer_response = await get_json("/api/v1/docs/zh/backend-collectors", user)
tactile_response = await get_json("/api/v1/docs/zh/tactile-ui-components", user)
glossary_response = await get_json("/api/v1/docs/zh/naming-glossary", user)
admin_response = await get_json("/api/v1/docs/zh/backend-system-service-control", user)
assert developer_response.status_code == 200
assert developer_response.json()["access"] == "docs_developer"
assert tactile_response.status_code == 200
assert tactile_response.json()["access"] == "docs_developer"
assert glossary_response.status_code == 200
assert glossary_response.json()["access"] == "docs_developer"
assert admin_response.status_code == 403

View File

@@ -0,0 +1,177 @@
import json
import pytest
from app.core.datasource_defaults import DEFAULT_DATASOURCES
from app.services.collectors.registry import collector_registry
from app.services import earth_boundaries
def write_geojson(path, name="Test"):
path.write_text(
json.dumps(
{
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"properties": {"name": name},
"geometry": {
"type": "Polygon",
"coordinates": [[[-1, -1], [1, -1], [1, 1], [-1, 1], [-1, -1]]],
},
}
],
}
),
encoding="utf-8",
)
def patch_paths(monkeypatch, tmp_path):
repo = tmp_path
source_dir = repo / "data/earth-boundary-sources"
boundary_dir = repo / "frontend/public/earth/data/boundaries/v1"
pmtiles = repo / "frontend/public/earth/data/boundaries/earth-boundaries-china-pov-v1.pmtiles"
legacy = repo / "frontend/public/earth/data/countries-admin0.min.geojson"
config = repo / "config/earth-boundary-sources.local.json"
example = repo / "config/earth-boundary-sources.example.json"
policy = repo / "config/earth-boundary-pov-policy.china-v1.json"
for path in (source_dir, boundary_dir, pmtiles.parent, legacy.parent, config.parent):
path.mkdir(parents=True, exist_ok=True)
policy.write_text('{"productionTileFormat":"pmtiles+mvt"}\n', encoding="utf-8")
example.write_text('{"collectorConfigs":{}}\n', encoding="utf-8")
monkeypatch.setattr(earth_boundaries, "REPO_ROOT", repo)
monkeypatch.setattr(earth_boundaries, "SOURCE_OUTPUT_DIR", source_dir)
monkeypatch.setattr(earth_boundaries, "SOURCE_MANIFEST_PATH", source_dir / "manifest.json")
monkeypatch.setattr(earth_boundaries, "BUILD_RESULT_PATH", source_dir / "build-result.json")
monkeypatch.setattr(earth_boundaries, "BUILD_JOB_PATH", source_dir / "build-job.json")
monkeypatch.setattr(earth_boundaries, "BOUNDARY_OUTPUT_DIR", boundary_dir)
monkeypatch.setattr(earth_boundaries, "BOUNDARY_MANIFEST_PATH", boundary_dir / "manifest.json")
monkeypatch.setattr(earth_boundaries, "PMTILES_ARTIFACT_PATH", pmtiles)
monkeypatch.setattr(earth_boundaries, "LEGACY_GEOJSON_PATH", legacy)
monkeypatch.setattr(earth_boundaries, "LOCAL_CONFIG_PATH", config)
monkeypatch.setattr(earth_boundaries, "EXAMPLE_CONFIG_PATH", example)
monkeypatch.setattr(earth_boundaries, "POV_POLICY_PATH", policy)
return {
"repo": repo,
"config": config,
"legacy": legacy,
"pmtiles": pmtiles,
"manifest": boundary_dir / "manifest.json",
}
def test_boundary_status_uses_legacy_provider_when_pmtiles_missing(monkeypatch, tmp_path):
paths = patch_paths(monkeypatch, tmp_path)
write_geojson(paths["legacy"])
status = earth_boundaries.get_boundary_status()
assert status["provider"] == "legacy-geojson"
assert status["fallback_available"] is True
assert status["high_precision_ready"] is False
def test_boundary_status_prefers_high_precision_when_manifest_and_pmtiles_exist(monkeypatch, tmp_path):
paths = patch_paths(monkeypatch, tmp_path)
write_geojson(paths["legacy"])
paths["pmtiles"].write_bytes(b"pmtiles")
paths["manifest"].write_text('{"tileProvider":"pmtiles-mvt"}\n', encoding="utf-8")
status = earth_boundaries.get_boundary_status()
assert status["provider"] == "pmtiles-mvt"
assert status["high_precision_ready"] is True
def test_save_boundary_config_writes_local_config(monkeypatch, tmp_path):
paths = patch_paths(monkeypatch, tmp_path)
payload = {"collectorConfigs": {"earth_admin0_boundaries": {"endpoint": "file:///tmp/a.geojson"}}}
status = earth_boundaries.save_boundary_config(payload)
assert paths["config"].exists()
assert status["config_source"] == "local"
assert status["config"] == payload
@pytest.mark.asyncio
async def test_build_reports_missing_tools_after_source_artifacts(monkeypatch, tmp_path):
paths = patch_paths(monkeypatch, tmp_path)
source_files = {}
for source in earth_boundaries.BOUNDARY_SOURCE_KINDS:
source_path = paths["repo"] / f"{source}.geojson"
write_geojson(source_path, name=source)
source_files[source] = source_path
paths["config"].write_text(
json.dumps(
{
"collectorConfigs": {
source: {
"sourceKind": kind,
"endpoint": str(source_files[source]),
"method": "GET",
}
for source, kind in earth_boundaries.BOUNDARY_SOURCE_KINDS.items()
}
}
),
encoding="utf-8",
)
monkeypatch.setattr(earth_boundaries.shutil, "which", lambda _tool: None)
progress_events = []
status = await earth_boundaries.build_boundary_assets(
lambda progress, phase, message, **_extra: progress_events.append((progress, phase, message))
)
assert status["provider"] == "geojson-high-precision"
assert status["high_precision_ready"] is True
assert (paths["repo"] / "data/earth-boundary-sources/manifest.json").exists()
assert paths["manifest"].exists()
assert any(phase == "download" for _progress, phase, _message in progress_events)
@pytest.mark.asyncio
async def test_start_boundary_build_job_records_geojson_fallback_success(monkeypatch, tmp_path):
paths = patch_paths(monkeypatch, tmp_path)
monkeypatch.setattr(earth_boundaries, "_build_task", None)
monkeypatch.setattr(earth_boundaries, "_build_job_state", {})
source_files = {}
for source in earth_boundaries.BOUNDARY_SOURCE_KINDS:
source_path = paths["repo"] / f"{source}.geojson"
write_geojson(source_path, name=source)
source_files[source] = source_path
paths["config"].write_text(
json.dumps(
{
"collectorConfigs": {
source: {
"sourceKind": kind,
"endpoint": str(source_files[source]),
"method": "GET",
}
for source, kind in earth_boundaries.BOUNDARY_SOURCE_KINDS.items()
}
}
),
encoding="utf-8",
)
monkeypatch.setattr(earth_boundaries.shutil, "which", lambda _tool: None)
response = await earth_boundaries.start_boundary_build_job()
await earth_boundaries._build_task
status = earth_boundaries.get_boundary_build_status()
assert response["accepted"] is True
assert status["job"]["status"] == "succeeded"
assert status["job"]["result"]["provider"] == "geojson-high-precision"
def test_earth_boundary_collectors_are_not_registered_as_datasources():
removed = set(earth_boundaries.BOUNDARY_SOURCE_KINDS) | {"earth_boundary_tiles"}
assert removed.isdisjoint(DEFAULT_DATASOURCES)
for source in removed:
assert collector_registry.get(source) is None

View File

@@ -0,0 +1,254 @@
from __future__ import annotations
import json
import pytest
from fastapi import Response
from app.api.v1 import visualization
from app.services.earth_layer_cache import (
EarthLayerCachePolicy,
apply_payload_budget,
earth_layer_cache,
format_bbox_key,
quantize_bbox,
resolve_layer_payload,
)
class FakeRedis:
def __init__(self, *, fail: bool = False) -> None:
self.store: dict[str, str] = {}
self.fail = fail
self.lock_claimed = False
def _maybe_fail(self) -> None:
if self.fail:
raise RuntimeError("redis unavailable")
def get(self, key: str):
self._maybe_fail()
return self.store.get(key)
def set(self, key: str, value: str, nx: bool = False, ex: int | None = None):
self._maybe_fail()
if nx and key in self.store:
return False
self.store[key] = value
return True
def setex(self, key: str, _seconds: int, value: str):
self._maybe_fail()
self.store[key] = value
return True
def delete(self, *keys: str):
self._maybe_fail()
deleted = 0
for key in keys:
deleted += 1 if self.store.pop(key, None) is not None else 0
return deleted
def scan_iter(self, match: str):
self._maybe_fail()
prefix = match.rstrip("*")
for key in list(self.store):
if key.startswith(prefix):
yield key
def memory_usage(self, key: str):
value = self.store.get(key, "")
return len(value.encode("utf-8"))
@pytest.fixture(autouse=True)
def fake_cache_client():
previous = earth_layer_cache._client
fake = FakeRedis()
earth_layer_cache._client = fake
try:
yield fake
finally:
earth_layer_cache._client = previous
def test_quantized_bbox_key_is_stable_for_small_movements():
first = format_bbox_key(quantize_bbox((10.01, 59.04, 10.96, 60.02)))
second = format_bbox_key(quantize_bbox((10.04, 59.01, 10.99, 60.04)))
assert first == second
assert first == "10.0,59.0,11.0,60.0"
def test_payload_budget_truncates_features():
payload = {
"type": "FeatureCollection",
"features": [{"id": index} for index in range(5)],
}
policy = EarthLayerCachePolicy(60, 120, max_features=2, max_bytes=1024)
result = apply_payload_budget(payload, policy)
assert len(result["features"]) == 2
assert result["diagnostics"]["truncated"] is True
assert result["diagnostics"]["limit_reason"] == "feature_budget"
assert result["diagnostics"]["original_feature_count"] == 5
@pytest.mark.asyncio
async def test_resolve_layer_payload_writes_fresh_and_stale(fake_cache_client):
calls = 0
async def builder():
nonlocal calls
calls += 1
return {"type": "FeatureCollection", "features": [{"id": "a"}]}
key = earth_layer_cache.key("satellites", limit="all")
policy = EarthLayerCachePolicy(60, 120)
first = await resolve_layer_payload(key=key, policy=policy, builder=builder)
second = await resolve_layer_payload(key=key, policy=policy, builder=builder)
assert first.state == "refresh"
assert second.state == "hit"
assert calls == 1
assert key in fake_cache_client.store
assert f"{key}:stale" in fake_cache_client.store
@pytest.mark.asyncio
async def test_resolve_layer_payload_returns_stale_when_builder_fails(fake_cache_client):
key = earth_layer_cache.key("bgp-incidents", status="active")
fake_cache_client.store[f"{key}:stale"] = json.dumps({"type": "FeatureCollection", "features": []})
async def builder():
raise RuntimeError("db exploded")
result = await resolve_layer_payload(
key=key,
policy=EarthLayerCachePolicy(60, 120),
builder=builder,
)
assert result.state == "stale"
assert result.payload["features"] == []
@pytest.mark.asyncio
async def test_resolve_layer_payload_uses_stale_during_lock_contention(fake_cache_client):
key = earth_layer_cache.key("cables")
fake_cache_client.store[earth_layer_cache.lock_key(key)] = "1"
fake_cache_client.store[f"{key}:stale"] = json.dumps(
{"type": "FeatureCollection", "features": [{"id": "stale-cable"}]}
)
calls = 0
async def builder():
nonlocal calls
calls += 1
return {"type": "FeatureCollection", "features": [{"id": "fresh-cable"}]}
result = await resolve_layer_payload(
key=key,
policy=EarthLayerCachePolicy(60, 120),
builder=builder,
)
assert result.state == "stale"
assert result.payload["features"][0]["id"] == "stale-cable"
assert calls == 0
@pytest.mark.asyncio
async def test_resolve_layer_payload_bypasses_redis_failure():
previous = earth_layer_cache._client
earth_layer_cache._client = FakeRedis(fail=True)
try:
async def builder():
return {"type": "FeatureCollection", "features": [{"id": "safe"}]}
result = await resolve_layer_payload(
key=earth_layer_cache.key("cables"),
policy=EarthLayerCachePolicy(60, 120),
builder=builder,
)
assert result.state == "bypass"
assert result.payload["features"][0]["id"] == "safe"
finally:
earth_layer_cache._client = previous
@pytest.mark.asyncio
async def test_visualization_endpoint_sets_cache_headers(fake_cache_client, monkeypatch):
calls = 0
async def fake_build_satellites_geojson(*, limit, db):
nonlocal calls
calls += 1
return {"type": "FeatureCollection", "features": [{"id": f"sat-{limit}"}], "count": 1}
monkeypatch.setattr(visualization, "_build_satellites_geojson", fake_build_satellites_geojson)
first_response = Response()
first = await visualization.get_satellites_geojson(limit=25, db=object(), response=first_response)
second_response = Response()
second = await visualization.get_satellites_geojson(limit=25, db=object(), response=second_response)
assert first == second
assert calls == 1
assert first_response.headers["X-Planet-Cache"] == "refresh"
assert second_response.headers["X-Planet-Cache"] == "hit"
@pytest.mark.asyncio
async def test_vessel_snapshot_uses_short_cache(fake_cache_client, monkeypatch):
calls = 0
async def fake_load_raw_vessel_snapshot_features(db, *, bbox, limit, observed_since):
nonlocal calls
calls += 1
return (
[
{
"type": "Feature",
"geometry": {"type": "Point", "coordinates": [10.1, 59.1]},
"properties": {"mmsi": 123, "vessel_type_name": "Cargo"},
}
],
{"raw_feature_count": 1},
)
monkeypatch.setattr(
visualization,
"_load_raw_vessel_snapshot_features",
fake_load_raw_vessel_snapshot_features,
)
first_response = Response()
first = await visualization.build_vessel_snapshot_response(
object(),
bbox=(10.01, 59.04, 10.96, 60.02),
zoom=12,
type_filter=None,
limit=1000,
since_minutes=60,
response=first_response,
)
second_response = Response()
second = await visualization.build_vessel_snapshot_response(
object(),
bbox=(10.04, 59.01, 10.99, 60.04),
zoom=12,
type_filter=None,
limit=1000,
since_minutes=60,
response=second_response,
)
assert first["count"] == 1
assert second == first
assert calls == 1
assert first_response.headers["X-Planet-Cache"] == "refresh"
assert second_response.headers["X-Planet-Cache"] == "hit"

View File

@@ -9,6 +9,7 @@ from app.services.earth_news import (
ParsedNewsItem,
_enrich_items_with_target_locations,
_extract_target_location_from_text,
_parse_feed_entries,
_serialize_item,
get_earth_news_payload,
)
@@ -109,6 +110,60 @@ def test_serialize_item_includes_inferred_target_location():
assert payload["location_meta"]["ai_error"] is None
def test_parse_plain_rss_uses_feed_name_as_source():
source = NewsFeedSource(
id="bbc-world",
name="BBC World",
region="global",
feed_url="https://feeds.bbci.co.uk/news/world/rss.xml",
homepage_url="https://www.bbc.com/news/world",
)
xml = """
<rss>
<channel>
<item>
<title>This may be the last time you hear my voice: Political executions surge in Iran since start of war</title>
<description>Story summary</description>
<link>https://www.bbc.com/news/example</link>
<pubDate>Fri, 15 May 2026 03:00:00 GMT</pubDate>
</item>
</channel>
</rss>
"""
items = _parse_feed_entries(xml, source)
assert items[0].title == "This may be the last time you hear my voice: Political executions surge in Iran since start of war"
assert items[0].source == "BBC World"
def test_parse_aggregated_rss_splits_publisher_from_title():
source = NewsFeedSource(
id="global-scan",
name="Global Monitor / World",
region="global",
feed_url="https://news.google.com/rss",
homepage_url="https://news.google.com/",
source_type="aggregated",
)
xml = """
<rss>
<channel>
<item>
<title>Example headline - Reuters</title>
<description>Story summary</description>
<link>https://news.google.com/example</link>
</item>
</channel>
</rss>
"""
items = _parse_feed_entries(xml, source)
assert items[0].title == "Example headline"
assert items[0].source == "Reuters"
@pytest.mark.asyncio
async def test_enrich_items_with_target_locations_uses_ai_and_geocode(monkeypatch):
item = ParsedNewsItem(
@@ -158,6 +213,56 @@ async def test_enrich_items_with_target_locations_uses_ai_and_geocode(monkeypatc
assert enriched[0].target_ai_error is None
@pytest.mark.asyncio
async def test_enrich_items_with_target_locations_adds_localizations(monkeypatch):
item = ParsedNewsItem(
id="global-scan:localized",
title="Global leaders meet to discuss energy security",
summary="Officials said the talks focused on supply chains and grid resilience.",
url="https://example.com/energy-security",
source="Example Source",
feed_name="Global Monitor / World",
feed_region="global",
homepage_url="https://example.com",
published_at=datetime(2026, 5, 14, 7, 13, 24, tzinfo=UTC),
)
async def fake_geocode(_query: str):
return {
"lat": "50.1109",
"lon": "8.6821",
"display_name": "Frankfurt am Main, Germany",
}
class FakeProviderClient:
async def analyze(self, _request):
class Response:
content = (
'{"location":{"country":"Germany","city":"Frankfurt",'
'"matched_location_name":"Frankfurt, Germany",'
'"latitude":null,"longitude":null,"confidence":0.77},'
'"localizations":{"zh-CN":{"title":"全球领导人讨论能源安全",'
'"summary":"官员表示,会谈聚焦供应链和电网韧性。"}}}'
)
return Response()
monkeypatch.setattr("app.services.earth_news._geocode_target_location", fake_geocode)
enriched = await _enrich_items_with_target_locations(
[item],
provider_client=FakeProviderClient(),
)
payload = _serialize_item(enriched[0], active_region="global")
assert payload["title"] == "Global leaders meet to discuss energy security"
assert payload["summary"] == "Officials said the talks focused on supply chains and grid resilience."
assert payload["localizations"]["zh-CN"]["title"] == "全球领导人讨论能源安全"
assert payload["display_title"] == "全球领导人讨论能源安全"
assert payload["display_summary"] == "官员表示,会谈聚焦供应链和电网韧性。"
assert payload["enrichment_status"] == "success"
@pytest.mark.asyncio
async def test_extract_target_location_from_text_uses_country_hint(monkeypatch):
item = ParsedNewsItem(
@@ -241,7 +346,7 @@ async def test_earth_news_payload_returns_anchor_items_and_enqueues_location_job
enqueued_payloads = []
async def fake_enqueue_target_location_job(payload):
async def fake_enqueue_target_location_job(payload, **_kwargs):
enqueued_payloads.append(payload)
return True
@@ -260,6 +365,8 @@ async def test_earth_news_payload_returns_anchor_items_and_enqueues_location_job
assert len(payload["items"]) == 1
assert payload["items"][0]["id"] == "test-feed:timeout"
assert payload["items"][0]["display_title"] == ""
assert payload["items"][0]["display_summary"] == ""
assert payload["items"][0]["latitude"] == 20.0
assert payload["items"][0]["longitude"] == 0.0
assert payload["items"][0]["location_source"] == "region_anchor"
@@ -359,7 +466,7 @@ async def test_earth_news_payload_initializes_empty_database_from_rss(monkeypatc
async def fake_list_earth_news_items(_db, *, active_region, limit):
return [item]
async def fake_enqueue_target_location_job(payload):
async def fake_enqueue_target_location_job(payload, **_kwargs):
enqueued.append(payload)
return True
@@ -415,7 +522,7 @@ async def test_earth_news_payload_supplements_stale_database_items(monkeypatch):
async def fake_list_earth_news_items(_db, *, active_region, limit):
return [old_item]
async def fake_enqueue_target_location_job(_payload):
async def fake_enqueue_target_location_job(_payload, **_kwargs):
return True
monkeypatch.setattr("app.services.earth_news_store.get_earth_news_freshness", fake_get_earth_news_freshness)
@@ -473,8 +580,11 @@ async def test_earth_news_payload_merges_cached_location_patch(monkeypatch):
async def fake_get_cached_target_location_patch(_item_id):
return cached_patch
async def fake_enqueue_target_location_job(_payload):
raise AssertionError("cached items should not be enqueued")
enqueued = []
async def fake_enqueue_target_location_job(payload, **_kwargs):
enqueued.append(payload)
return True
monkeypatch.setattr("app.services.earth_news.get_sources_for_region", lambda _region: [source])
monkeypatch.setattr("app.services.earth_news._fetch_source", fake_fetch_source)
@@ -493,6 +603,70 @@ async def test_earth_news_payload_merges_cached_location_patch(monkeypatch):
assert payload["items"][0]["longitude"] == 116.3912972
assert payload["items"][0]["verified"] is True
assert payload["items"][0]["location_source"] == "headline_location_hint"
assert enqueued[0]["id"] == "test-feed:cached"
@pytest.mark.asyncio
async def test_earth_news_payload_requeues_cached_failed_localization(monkeypatch):
source = NewsFeedSource(
id="test-feed",
name="Test Feed",
region="global",
homepage_url="https://example.com",
feed_url="https://example.com/rss.xml",
)
item = ParsedNewsItem(
id="test-feed:failed-localization",
title="Failed localization story",
summary="English source summary.",
url="https://example.com/failed-localization",
source="Test Feed",
feed_name="Test Feed",
feed_region="global",
homepage_url="https://example.com",
published_at=datetime(2026, 5, 15, 3, 0, tzinfo=UTC),
)
cached_patch = {
"latitude": 20.0,
"longitude": 0.0,
"location_label": "全球",
"location_source": "region_anchor",
"verified": False,
"location_meta": {"target": None, "anchor": {"region": "global"}},
"content_language": "en",
"localizations": {},
"enrichment_status": "parse_error",
"enrichment_error": "AI response did not contain a parseable JSON object.",
"enriched_at": None,
}
enqueued = []
async def fake_fetch_source(_client, feed_source):
return feed_source, [item], None
async def fake_get_cached_target_location_patch(_item_id):
return cached_patch
async def fake_enqueue_target_location_job(payload, **_kwargs):
enqueued.append(payload)
return True
monkeypatch.setattr("app.services.earth_news.get_sources_for_region", lambda _region: [source])
monkeypatch.setattr("app.services.earth_news._fetch_source", fake_fetch_source)
monkeypatch.setattr(
"app.services.earth_news_queue.get_cached_target_location_patch",
fake_get_cached_target_location_patch,
)
monkeypatch.setattr(
"app.services.earth_news_queue.enqueue_target_location_job",
fake_enqueue_target_location_job,
)
payload = await get_earth_news_payload(provider_client=None)
assert enqueued[0]["id"] == "test-feed:failed-localization"
assert payload["items"][0]["display_title"] == ""
assert payload["items"][0]["enrichment_status"] == "queued"
@pytest.mark.asyncio
@@ -598,6 +772,11 @@ async def test_media_news_archive_collector_maps_news_items(monkeypatch):
location_source="headline_location_hint",
verified=True,
location_meta={"target": {"country": "中国", "city": "Beijing"}},
content_language="en",
localizations={"zh-CN": {"title": "归档新闻", "summary": "归档概要"}},
enrichment_status="success",
enrichment_error=None,
enriched_at=datetime(2026, 5, 15, 3, 6, tzinfo=UTC),
first_seen_at=datetime(2026, 5, 15, 2, 0, tzinfo=UTC),
last_seen_at=datetime(2026, 5, 15, 3, 0, tzinfo=UTC),
resolved_at=datetime(2026, 5, 15, 3, 5, tzinfo=UTC),
@@ -619,3 +798,5 @@ async def test_media_news_archive_collector_maps_news_items(monkeypatch):
assert items[0]["city"] == "Beijing"
assert items[0]["latitude"] == 39.9057136
assert items[0]["metadata"]["verified"] is True
assert "localizations" not in items[0]["metadata"]
assert "enrichment_status" not in items[0]["metadata"]

View File

@@ -700,7 +700,7 @@ async def test_llm_location_fallback_repairs_non_json_answer(monkeypatch):
@pytest.mark.asyncio
async def test_llm_location_fallback_accepts_taipei_name_hint_with_weak_wording(monkeypatch):
async def test_llm_location_fallback_rejects_city_from_name_without_location_evidence(monkeypatch):
monkeypatch.setattr(
llm_fallback,
"_geocode_llm_city",
@@ -740,18 +740,60 @@ async def test_llm_location_fallback_accepts_taipei_name_hint_with_weak_wording(
entity_type="compute_center",
)
assert result.failure_reason is None
candidate = result.candidates[0]
assert candidate.city == "Taipei"
assert candidate.confidence >= 0.55
breakdown = candidate.suggested_registry_entry["llm_score_breakdown"]
assert breakdown["weak_evidence_penalty"] <= 0.15
assert breakdown["conflict_penalty"] == 0
assert breakdown["name_location_hint"] > 0
assert result.candidates == []
assert result.failure_reason is not None
assert "below minimum" in result.failure_reason
@pytest.mark.asyncio
async def test_llm_location_fallback_geocodes_city_from_entity_name_when_llm_unparseable(monkeypatch):
async def test_llm_location_fallback_accepts_explicit_facility_location_for_name_city_conflict(monkeypatch):
monkeypatch.setattr(
llm_fallback,
"_geocode_llm_city",
lambda query: {
"lat": "22.6048",
"lon": "120.3000",
"display_name": "Kaohsiung, Taiwan",
"address": {"city": "Kaohsiung", "country": "Taiwan"},
},
)
client = _FakeAIProviderClient(
json.dumps(
{
"latitude": None,
"longitude": None,
"precision": "city",
"confidence": 0.72,
"city": "Kaohsiung",
"country": "Taiwan",
"matched_location_name": "Kaohsiung, Taiwan",
"evidence": [
{
"source": "Taiwan News",
"source_type": "news",
"entity_match": True,
"text": "Nvidia's first AI supercomputer center, Taipei-1, is located in Kaohsiung.",
}
],
"reasoning_summary": "Explicit facility location evidence overrides the city-like system name.",
}
)
)
result = await collect_llm_location_fallback_candidate(
provider_client=client,
query=LocationQuery(name="TAIPEI-1", country="Taiwan"),
entity_type="compute_center",
)
assert result.failure_reason is None
candidate = result.candidates[0]
assert candidate.city == "Kaohsiung"
assert candidate.confidence >= 0.55
@pytest.mark.asyncio
async def test_llm_location_fallback_rejects_compute_center_city_from_entity_name_when_llm_unparseable(monkeypatch):
def _fake_geocode(query):
if query != "Taipei, 中国(台湾)":
return None
@@ -772,12 +814,9 @@ async def test_llm_location_fallback_geocodes_city_from_entity_name_when_llm_unp
)
assert client.calls == 2
assert result.failure_reason is None
candidate = result.candidates[0]
assert candidate.city == "Taipei"
assert candidate.latitude == pytest.approx(25.033)
assert candidate.longitude == pytest.approx(121.5654)
assert "Entity name city hint" in candidate.source_note
assert result.candidates == []
assert result.failure_reason is not None
assert "parseable city-level location fact" in result.failure_reason
@pytest.mark.asyncio

View File

@@ -0,0 +1,92 @@
from types import SimpleNamespace
import pytest
from app.ai_tasks.prompts import (
get_effective_prompt,
list_effective_prompts,
reset_prompt_override,
save_prompt_override,
)
class _ScalarResult:
def __init__(self, value):
self._value = value
def scalar_one_or_none(self):
return self._value
class _PromptSettingsDB:
def __init__(self, payload=None):
self.record = SimpleNamespace(category="ai_prompts", payload=payload) if payload is not None else None
self.added = None
self.commits = 0
async def execute(self, _statement):
return _ScalarResult(self.record)
def add(self, record):
self.record = record
self.added = record
async def commit(self):
self.commits += 1
@pytest.mark.asyncio
async def test_prompt_defaults_are_loaded_without_override():
db = _PromptSettingsDB()
prompt = await get_effective_prompt(db, "earth.news.enrich")
assert prompt.key == "earth.news.enrich"
assert prompt.is_custom is False
assert "strict JSON" in prompt.prompt
@pytest.mark.asyncio
async def test_prompt_override_save_and_reset():
db = _PromptSettingsDB()
saved = await save_prompt_override(
db,
"alerts.brief",
system_prompt="system custom",
prompt="prompt custom",
)
assert saved.is_custom is True
assert saved.system_prompt == "system custom"
assert saved.prompt == "prompt custom"
assert db.commits == 1
effective = await get_effective_prompt(db, "alerts.brief")
assert effective.prompt == "prompt custom"
reset = await reset_prompt_override(db, "alerts.brief")
assert reset.is_custom is False
assert reset.prompt != "prompt custom"
@pytest.mark.asyncio
async def test_prompt_list_marks_custom_items():
db = _PromptSettingsDB(
{
"overrides": {
"bgp.brief": {
"system_prompt": "",
"prompt": "custom bgp prompt",
"updated_at": "2026-05-16T00:00:00Z",
}
}
}
)
prompts = await list_effective_prompts(db)
by_key = {prompt.key: prompt for prompt in prompts}
assert by_key["bgp.brief"].is_custom is True
assert by_key["bgp.brief"].prompt == "custom bgp prompt"
assert by_key["earth.news.enrich"].is_custom is False

View File

@@ -5,21 +5,42 @@ import pytest
from app.api.v1 import settings as settings_api
from app.api.v1.settings import (
AIProviderIntegrationUpdate,
BarentsWatchIntegrationUpdate,
ExternalIntegrationsUpdate,
OCRIntegrationUpdate,
WebSearchIntegrationUpdate,
_build_ai_provider_payload,
_build_ocr_payload,
_can_reveal_integration_secrets,
_ensure_secret_reveal_allowed,
_mask_secret,
_normalize_ai_provider_payload,
_normalize_ocr_payload,
_record_integration_secret_reveal,
_resolve_provider_api_key,
get_runtime_ai_provider_config,
save_external_integrations_payload,
)
from app.services.llm_provider_catalog import get_fallback_llm_provider_preset
@pytest.fixture(autouse=True)
def isolated_ai_provider_env_file(monkeypatch, tmp_path):
env_file = tmp_path / ".env"
monkeypatch.setattr(settings_api, "AI_PROVIDER_ENV_FILE", env_file)
for name in (
"AI_PROVIDER",
"AI_API_KEY",
"MINIMAX_API_KEY",
"OPENAI_API_KEY",
"ANTHROPIC_API_KEY",
"DEEPSEEK_API_KEY",
"DASHSCOPE_API_KEY",
"MOONSHOT_API_KEY",
"OPENROUTER_API_KEY",
"OPENCODE_GO_API_KEY",
):
monkeypatch.delenv(name, raising=False)
return env_file
@@ -56,7 +77,7 @@ def test_provider_key_prefers_specific_env_file_key(isolated_ai_provider_env_fil
def test_provider_key_falls_back_to_generic_ai_api_key(isolated_ai_provider_env_file):
isolated_ai_provider_env_file.write_text(
"AI_API_KEY=generic-env-file-key\n",
"AI_PROVIDER=openai\nAI_API_KEY=generic-env-file-key\n",
encoding="utf-8",
)
@@ -66,11 +87,101 @@ def test_provider_key_falls_back_to_generic_ai_api_key(isolated_ai_provider_env_
assert source == "env_file"
def test_generic_ai_api_key_only_applies_to_selected_env_provider(isolated_ai_provider_env_file):
isolated_ai_provider_env_file.write_text(
"AI_PROVIDER=minimax\nAI_API_KEY=generic-env-file-key\n",
encoding="utf-8",
)
openai_value, openai_source = _resolve_provider_api_key("openai", {"api_key": ""})
minimax_value, minimax_source = _resolve_provider_api_key("minimax", {"api_key": ""})
assert openai_value == ""
assert openai_source == ""
assert minimax_value == "generic-env-file-key"
assert minimax_source == "env_file"
def test_opencode_go_provider_preset_is_openai_compatible():
preset = get_fallback_llm_provider_preset("opencode-go")
assert preset["label"] == "OpenCode Go"
assert preset["provider_api"] == "openai-completions"
assert preset["base_url"] == "https://opencode.ai/zen/go/v1"
assert preset["model"] == "glm-5.1"
assert "glm-5.1" in preset["models"]
assert "deepseek-v4-flash" in preset["models"]
assert preset["model_provider_apis"]["minimax-m2.7"] == "anthropic-messages"
assert preset["api_key_env"] == "OPENCODE_GO_API_KEY"
def test_mask_secret_without_prefix_is_fully_masked():
assert _mask_secret("plainsecret")["preview"] == "***********"
assert _mask_secret("sk-prefixed")["preview"] == "sk-********"
def test_secret_reveal_permission_is_admin_only():
assert _can_reveal_integration_secrets(SimpleNamespace(role="super_admin")) is True
assert _can_reveal_integration_secrets(SimpleNamespace(role="admin")) is True
assert _can_reveal_integration_secrets(SimpleNamespace(role="viewer")) is False
@pytest.mark.asyncio
async def test_secret_reveal_denial_is_audited(monkeypatch):
events = []
async def fake_record_audit_log(**payload):
events.append(payload)
monkeypatch.setattr(settings_api, "record_audit_log", fake_record_audit_log)
user = SimpleNamespace(id=7, username="viewer", email="viewer@example.test", role="viewer")
with pytest.raises(settings_api.HTTPException) as exc:
await _ensure_secret_reveal_allowed(
current_user=user,
request=None,
target_id="ai_provider:openai",
details={"kind": "ai_provider", "provider": "openai"},
)
assert exc.value.status_code == 403
assert events[0]["action"] == "settings.integration_secret.reveal"
assert events[0]["result"] == "denied"
assert events[0]["actor_id"] == 7
assert events[0]["target_id"] == "ai_provider:openai"
@pytest.mark.asyncio
async def test_secret_reveal_audit_does_not_store_plaintext(monkeypatch):
events = []
async def fake_record_audit_log(**payload):
events.append(payload)
monkeypatch.setattr(settings_api, "record_audit_log", fake_record_audit_log)
user = SimpleNamespace(id=1, username="admin", email="admin@example.test", role="admin")
await _record_integration_secret_reveal(
current_user=user,
request=None,
target_id="ai_provider:minimax",
result="success",
details={
"kind": "ai_provider",
"provider": "minimax",
"api_key_configured": True,
"api_key_source": "env_file",
"service_token_configured": True,
"service_token_source": "runtime",
},
)
serialized = str(events[0])
assert "secret-value" not in serialized
assert events[0]["details"]["api_key_source"] == "env_file"
assert events[0]["details"]["service_token_configured"] is True
def test_build_payload_updates_only_selected_provider_key():
current = {
"ai_provider": {
@@ -109,6 +220,30 @@ def test_build_payload_updates_only_selected_provider_key():
assert payload["providers"]["minimax"]["api_key"] == "minimax-old-key"
def test_build_payload_saves_provider_without_changing_default():
current = {
"ai_provider": {
"default_provider": "minimax",
"providers": {
"minimax": {"provider": "minimax", "api_key": "minimax-key"},
"openai": {"provider": "openai", "api_key": ""},
},
}
}
update = AIProviderIntegrationUpdate(
provider="openai",
provider_api="openai-completions",
base_url="https://api.openai.com/v1",
model="gpt-test",
api_key="openai-new-key",
)
payload = _build_ai_provider_payload(current, update)
assert payload["default_provider"] == "minimax"
assert payload["providers"]["openai"]["api_key"] == "openai-new-key"
def test_build_payload_keeps_saved_key_when_preview_submitted():
current = {
"ai_provider": {
@@ -207,3 +342,78 @@ async def test_runtime_config_uses_default_provider_specific_key(monkeypatch):
assert runtime_config["llm_config"]["provider"] == "minimax"
assert runtime_config["llm_config"]["api_key"] == "minimax-key"
assert runtime_config["llm_config"]["model"] == "MiniMax-test"
@pytest.mark.asyncio
async def test_saving_ai_provider_does_not_run_connection_validation(monkeypatch):
calls = {"validated": 0, "saved": 0}
async def fake_get_setting_payload(_db, category):
assert category == "external_integrations"
return {
"ai_provider": {
"default_provider": "minimax",
"providers": {
"minimax": {
"provider": "minimax",
"api_key": "minimax-key",
},
},
},
"web_search": {},
"ocr": {},
}
async def fake_validate(_payload):
calls["validated"] += 1
raise AssertionError("save should not run AI provider connection validation")
async def fake_save_setting_payload(_db, category, payload):
assert category == "external_integrations"
calls["saved"] += 1
return payload
async def fake_get_barentswatch_config_record(_db):
return SimpleNamespace(
endpoint="",
auth_config={},
)
async def fake_serialize_external_integrations(_db):
return {"ai_provider": {"default_provider": "minimax"}}
monkeypatch.setattr(settings_api, "get_setting_payload", fake_get_setting_payload)
monkeypatch.setattr(settings_api, "_validate_ai_provider_full_connection", fake_validate)
monkeypatch.setattr(settings_api, "save_setting_payload", fake_save_setting_payload)
monkeypatch.setattr(settings_api, "get_barentswatch_config_record", fake_get_barentswatch_config_record)
monkeypatch.setattr(settings_api, "serialize_external_integrations", fake_serialize_external_integrations)
update = ExternalIntegrationsUpdate(
ai_provider=AIProviderIntegrationUpdate(
provider="minimax",
default_provider="minimax",
provider_api="anthropic-messages",
base_url="https://api.minimaxi.com/anthropic",
model="MiniMax-M2.7",
api_key="sk-new-key",
),
barentswatch=BarentsWatchIntegrationUpdate(),
web_search=WebSearchIntegrationUpdate(),
ocr=OCRIntegrationUpdate(),
)
async def fake_commit():
return None
async def fake_refresh(_record):
return None
db = SimpleNamespace(
add=lambda _record: None,
commit=fake_commit,
refresh=fake_refresh,
)
await save_external_integrations_payload(db, update)
assert calls == {"validated": 0, "saved": 1}

View File

@@ -0,0 +1,71 @@
import pytest
from app.services.situational_alert_ai_brief import build_situational_alert_brief_request
class _SingleUseScalarResult:
def __init__(self, value=0, rows=None):
self.value = value
self.rows = rows or []
self.scalar_calls = 0
def scalar(self):
self.scalar_calls += 1
if self.scalar_calls > 1:
raise AssertionError("scalar result was consumed more than once")
return self.value
def fetchall(self):
return self.rows
def scalar_one_or_none(self):
return None
def scalars(self):
rows = self.rows
class _Scalars:
def all(self):
return rows
return _Scalars()
class _FakeBriefSession:
def __init__(self):
self._results = [
_SingleUseScalarResult(3),
_SingleUseScalarResult(2),
_SingleUseScalarResult(rows=[]),
_SingleUseScalarResult(rows=[]),
_SingleUseScalarResult(rows=[]),
_SingleUseScalarResult(4),
_SingleUseScalarResult(1),
_SingleUseScalarResult(rows=[]),
_SingleUseScalarResult(rows=[]),
_SingleUseScalarResult(5),
_SingleUseScalarResult(2),
_SingleUseScalarResult(rows=[]),
_SingleUseScalarResult(),
]
async def execute(self, _query):
return self._results.pop(0)
@pytest.mark.asyncio
async def test_situational_alert_brief_builder_reuses_counts_without_reconsuming_results(monkeypatch):
monkeypatch.setattr(
"app.services.situational_alert_ai_brief.get_latest_bgp_brief_record",
lambda: None,
)
request, facts, context = await build_situational_alert_brief_request(_FakeBriefSession())
assert request.title == "态势告警 AI 简报"
assert "总告警 3 条active 2 条" in facts[0]
assert "累计 incidents 4 条active incidents 1 条" in facts[1]
assert "累计 anomalies 5 条active anomalies 2 条" in facts[2]
assert context["active_system_alerts"] == 2
assert context["active_bgp_incidents"] == 1
assert context["active_bgp_anomalies"] == 2

View File

@@ -46,6 +46,21 @@ def test_web_search_key_prefers_provider_env(isolated_web_search_env_files):
assert source == "env_file"
def test_web_search_generic_key_only_applies_to_default_provider(isolated_web_search_env_files):
isolated_web_search_env_files.write_text(
"WEB_SEARCH_API_KEY=generic-search-key\n",
encoding="utf-8",
)
default_value, default_source = _resolve_web_search_api_key("tavily", {"api_key": ""}, "tavily")
other_value, other_source = _resolve_web_search_api_key("brave", {"api_key": ""}, "tavily")
assert default_value == "generic-search-key"
assert default_source == "env_file"
assert other_value == ""
assert other_source == ""
def test_build_web_search_payload_keeps_saved_key_when_preview_submitted():
current = {
"web_search": {
@@ -71,6 +86,35 @@ def test_build_web_search_payload_keeps_saved_key_when_preview_submitted():
assert payload["providers"]["tavily"]["api_key"] == "tvly-old-secret"
def test_build_web_search_payload_saves_provider_without_changing_default():
current = {
"web_search": {
"default_provider": "tavily",
"providers": {
"tavily": {
"provider": "tavily",
"api_key": "tvly-key",
},
"brave": {
"provider": "brave",
"api_key": "",
},
},
}
}
update = WebSearchIntegrationUpdate(
enabled=True,
provider="brave",
base_url="https://api.search.brave.com/res/v1/web/search",
api_key="brave-new-key",
)
payload = _build_web_search_payload(current, update)
assert payload["default_provider"] == "tavily"
assert payload["providers"]["brave"]["api_key"] == "brave-new-key"
@pytest.mark.asyncio
async def test_tavily_adapter_normalizes_results(monkeypatch):
config = WebSearchConfig(

View File

@@ -5,18 +5,17 @@
"povPolicyPath": "config/earth-boundary-pov-policy.china-v1.json",
"productionTileFormat": "pmtiles+mvt",
"debugTileFormat": "geojson-directory",
"description": "Example Collector Settings payloads for audited Earth boundary source ingestion. Replace every endpoint, license, and checksum note before production."
"description": "Default Earth boundary update sources. These public Natural Earth endpoints make local high-precision boundary download work out of the box; replace with audited internal sources for production if needed."
},
"collectorConfigs": {
"earth_admin0_boundaries": {
"displayName": "Earth Admin-0 国界源",
"sourceKind": "admin0-boundaries",
"endpoint": "https://example.com/admin0-boundaries.geojson",
"endpoint": "https://raw.githubusercontent.com/nvkelso/natural-earth-vector/master/geojson/ne_10m_admin_0_countries.geojson",
"method": "GET",
"headers": {},
"auth_type": "none",
"target_schema": "earth_boundary_source",
"license": "REPLACE_WITH_SOURCE_LICENSE",
"license": "Natural Earth public domain",
"mapping_json": {
"source": {
"items_path": "$.features[*]"
@@ -44,12 +43,11 @@
"earth_coastline": {
"displayName": "Earth 海岸线源",
"sourceKind": "coastline",
"endpoint": "https://example.com/coastline.geojson",
"endpoint": "https://raw.githubusercontent.com/nvkelso/natural-earth-vector/master/geojson/ne_10m_coastline.geojson",
"method": "GET",
"headers": {},
"auth_type": "none",
"target_schema": "earth_boundary_source",
"license": "REPLACE_WITH_SOURCE_LICENSE",
"license": "Natural Earth public domain",
"mapping_json": {
"source": {
"items_path": "$.features[*]"
@@ -77,12 +75,11 @@
"earth_claim_lines": {
"displayName": "Earth 主张线源",
"sourceKind": "claim-lines",
"endpoint": "https://example.com/claim-lines.geojson",
"endpoint": "https://raw.githubusercontent.com/nvkelso/natural-earth-vector/master/geojson/ne_10m_admin_0_boundary_lines_disputed_areas.geojson",
"method": "GET",
"headers": {},
"auth_type": "none",
"target_schema": "earth_boundary_source",
"license": "REPLACE_WITH_SOURCE_LICENSE",
"license": "Natural Earth public domain",
"mapping_json": {
"source": {
"items_path": "$.features[*]"
@@ -109,7 +106,7 @@
}
},
"notes": [
"Use the actual Collector Settings page as the source of truth for runtime endpoints.",
"The repository seed GeoJSON may be used only for smoke tests and cannot satisfy production high precision."
"Earth can download these sources directly from the toolbar settings when no local source override exists.",
"If tippecanoe/pmtiles are unavailable, the backend generates a GeoJSON high-precision package so the feature remains usable."
]
}

View File

@@ -8,6 +8,136 @@ This project follows the repository versioning rule:
- `improvement` -> `+0.0.1`bugfix + 小功能混合)
- `bugfix` -> `+0.0.1`
## [0.65.0] — 2026-05-21
Released: 2026-05-21
### Highlights
- 收敛 Admin Next 数据源触发入口:主按钮在未勾选时触发全部,勾选内置源后切换为“触发已选 N”并移除手填 ID 的批量触发弹窗。
- 优化数据源采集队列入口:右上角按钮常驻,空态显示队列图标,有任务时显示纯圆环进度,队列改为浮层避免挤压表格。
- 强化 `planet.sh destroy` 清理语义,销毁时先硬重置运行中的本地 Postgres `public` schema避免残留采集数据让 OOBE 误判 ready。
### Added / Fixed / Improved
- Admin Next 表格新增可选选择列,仅在 `/datasources` 内置源分区启用,支持当前可见行全选并在筛选、切分区或刷新时清空选择。
- 数据源批量触发复用 `/datasources/trigger-batch``source_ids`,成功后写入现有采集队列并清空勾选。
- `destroy` 补充清理 `planet-aiprovider:latest` 镜像以及 Python/Vite 等本地编译缓存,同时保留源码和 `.env`
- Docs Gatekeeper 与 Tactile UI 文档/样式继续补齐覆盖本轮按钮、队列、OOBE 和销毁流程说明。
---
## [0.64.0] — 2026-05-21
Released: 2026-05-21
### Highlights
- 新增 Earth 首次初始化 OOBE由后端真实采集状态决定是否显示避免 localStorage 清空后误弹,并提供桌面毛玻璃引导与移动端 bottom sheet。
- 数据源页新增下载列表式采集队列,把单源、批量和触发全部的任务进度统一展示,并支持失败重试与跳转详情。
- Earth 内容新增“关于”配置接口和后台 tabEarth 设置页 About 卡片改为运行时读取配置并带默认 fallback。
### Added / Fixed / Improved
- 新增 `/api/v1/earth/oobe-status``/api/v1/earth/about` GET/PUT/DELETE并让 Earth 前端加载 `about.js``oobe.js`
- Admin Next 数据源队列优先消费 `datasource_tasks` WebSocket断线时轮询 `/datasources/{id}/task-status`,刷新后只恢复后端仍在运行的真实任务。
- Admin Next 深色主题滑块补齐 Docs 同款 dark token侧栏主题控件在 dark 模式下不再保持浅色底座。
- 用户手册、快速开始、Earth 前端上下文和 Admin 前端上下文同步记录 OOBE、采集队列、About 配置与主题滑块行为。
---
## [0.63.1] — 2026-05-21
Released: 2026-05-21
### Highlights
- 修复新设备初始化后 Admin Next 无法打开的问题:补上被 `.gitignore``lib/` 规则误忽略的 Admin Next utility module。
### Added / Fixed / Improved
-`frontend/src/admin-next/lib/utils.ts` 纳入版本控制,恢复 `DashboardNext``AdminNextLayout` 等页面对 `cn``formatNumber` 的运行时依赖。
- 避免 Vite dev server 在新 clone 环境中因缺失源码模块而对 `/src/admin-next/*` 返回 500并导致动态导入 AdminNextRoutes 失败。
---
## [0.63.0] — 2026-05-21
Released: 2026-05-21
### Highlights
- 新增 `./planet.sh init` 首次初始化入口,将 uv/bun 依赖同步、env 模板补齐、数据库容器启动、建表 seed 和默认用户生成串成一条空项目引导路径。
- 新增 `./planet.sh destroy` 破坏性重置入口,带 CLI 确认保护,可清理 Planet 容器、卷、镜像和本地编译/运行状态,同时保留源码与 `.env` 配置。
- 改进 `planet.sh` 日志体验,只在带状态标签的输出行末尾追加时间戳,并让 `init` 在应用服务已运行时自动跳过重复初始化。
### Added / Fixed / Improved
- `init` 复用现有 Docker/依赖 helper支持重复执行时不覆盖 env、不清空数据并在完成后提示默认本地登录账号。
- `destroy` 停止本地服务后清理 Docker compose 状态、项目镜像、数据卷、`.venv``node_modules`、前端构建产物和 Planet state/cache。
- README 快速启动补充 `init``destroy` 命令说明,明确首次引导和重置路径。
---
## [0.62.0] — 2026-05-21
Released: 2026-05-21
### Highlights
- Admin Next 转正为正式后台入口,旧 AntD 控制台完整迁移到 `/legacy/admin/*` 作为回退和对照,并保留 `/admin-next/*` 兼容重定向。
- 新后台完成采集、AI Provider、Earth 内容、日志、BGP/告警、设置和认证链路的全量收口,新增 lazy tab loading、层级配置、移动端详情和 Markdown 文档渲染体验。
- 抽出 Tactile UI 按钮、开关、tooltip、滚动条和表格滚动组件为未来独立 npm 组件库打基础,同时补齐中英文开发文档。
### Added / Fixed / Improved
- 新增采集管理分层工作台,覆盖采集器配置、采集调度、采集历史/快照、凭证教程生成/重置、映射 propose/preview/create/activate 和真实连接/运行状态。
- 改进 AI Provider 与工具调用配置,修复 key fallback、脱敏显示、轻量连通性测试、默认 provider、OpenCode Go 路由和 Playground Markdown 输出。
- 更新 Earth 内容管理和 Earth 前台体验,支持品牌预览/上传/重置、TV 默认源与新增草稿、图层/新闻/3D 模型配置入口以及 live TV 预览。
- 强化后端数据源、系统日志、WebSocket、AI client、位置 LLM fallback、Earth layer cache 和 datasource connectivity并补充相关回归测试与运维脚本能力。
---
## [0.61.0] — 2026-05-18
Released: 2026-05-18
### Highlights
- 新增 Earth 图层 Redis 读穿缓存、防击穿锁、stale 兜底和 payload budget降低演示前重图层与船只 snapshot 对后端内存的冲击。
- 保持前端原 API 不变为海缆、登陆点、卫星、算力中心、BGP、summary 和船只 snapshot 增加透明缓存 header 可观测性。
- 新增 super admin Earth 图层缓存状态与清理接口,并在采集写入后按 source 主动失效相关缓存。
### Added / Fixed / Improved
- 新增 `earth:layer:v1:*` 缓存命名空间、fresh/stale 双 key、Redis 故障 bypass 和 OOM 防护诊断。
- 船只 snapshot 使用短 TTL、bbox 量化和响应预算,避免重复视窗请求和超大 payload 触发后端 OOM。
- 补充 Earth layer cache 计划文档与后端测试,覆盖 hit、refresh、stale、bypass、锁竞争、cache header 和运维清理。
---
## [0.60.0] — 2026-05-17
Released: 2026-05-17
### Highlights
- 新增 Earth 内容与国界运行体验:品牌内容配置、国界精度滑块、高精国界首次应用按钮和高精重载构建入口共同收口。
- 完成新闻中文展示链路收敛:滚动新闻改用一句话摘要,新闻本地化状态避免未完成内容进入滚动展示,英文原文继续保留在数据层。
- 重构 AI Provider 与提示词边界:`aiprovider` 保持纯净模型适配,业务提示词集中到后端 AI task 默认注册表并支持运维台覆盖和重置。
### Added / Fixed / Improved
- 新增 AI 设置提示词 tab按功能入口选择、编辑、保存和重置默认提示词并修复态势告警 / BGP 简报展示生成内容而非 prompt。
- 更新 Earth 新闻、HUD、巡航、桌面详情和移动详情的中文显示策略并优化新闻摘要提示词为新闻式一句话概要。
- 补充 Agent Runtime、Earth LLM 指令、语音识别和多角色态势感知的详细计划,并同步 README 架构说明、用户手册、FAQ 与开发者文档。
- 补齐相关后端测试,覆盖 Earth 品牌配置、AI 简报接口和态势告警 AI 简报生成链路。
---
## [0.59.0] — 2026-05-16
Released: 2026-05-16
### Highlights
- 将 Earth 国界从采集器体系迁移为 Earth 静态资产,恢复低精 GeoJSON fallback并新增 Earth 工具栏高精国界下载/构建进度与热应用。
- 重组后台“运维与配置”:新增 Earth 内容与采集管理二级入口,电视直播、国界精度、采集器、采集调度各归其位,未接入模块以占位页呈现。
- 新增 AI task prompt 覆盖管理,按稳定 task key 管理新闻汉化、告警研判、BGP 简报等业务提示词,避免全局 prompt 污染。
### Added / Fixed / Improved
- Earth 新闻锚点链路增加队列化 enrichment 状态、Redis Streams 后台精修和 WebSocket patch 语义,前端汉化/锚点策略更稳定。
- 国界 hover 与 interactable tooltip 解耦,鼠标位于国家 polygon 内时保持国界高亮,同时卫星/船只/BGP 等对象仍可显示自身信息。
- 新增 `/api/v1/earth/boundaries/*` 状态、配置、构建和进度接口,并在启动初始化中清理旧 boundary datasource/task/snapshot 历史入口。
- 补齐中英文用户手册、FAQ、quickstart、运维手册和开发者上下文文档明确用户 UI、运维操作和开发者稳定边界。
---
## [0.58.0] — 2026-05-15
Released: 2026-05-15

View File

@@ -35,7 +35,11 @@
- [Docs Gatekeeper 鉴权系统计划](/home/ray/dev/linkong/planet/docs/plans/docs-gatekeeper-auth-plan.md)
- [Location Resolver 共享管线计划](/home/ray/dev/linkong/planet/docs/plans/location-resolver-shared-pipeline-plan.md)
- [frontend-ai-playground-development-plan.md](/home/ray/dev/linkong/planet/docs/plans/frontend-ai-playground-development-plan.md)
- [AI Provider OpenClaw-Style Routing Plan](/home/ray/dev/linkong/planet/docs/plans/ai-provider-openclaw-style-routing-plan.md)
- [统一集成配置 Schema 系统计划](/home/ray/dev/linkong/planet/docs/plans/integration-config-schema-system-plan.md)
- [Lightweight Agent Orchestrator 与 WebSearch 证据层计划](/home/ray/dev/linkong/planet/docs/plans/agents-light-orchestrator-websearch-plan.md)
- [Admin Next Parity Checklist](/home/ray/dev/linkong/planet/docs/plans/admin-next-parity-checklist.md)
- [Admin Next Parity Audit Closeout](/home/ray/dev/linkong/planet/docs/plans/admin-next-parity-audit-closeout-plan.md)
- [ue5-mvp-fused-plan.md](/home/ray/dev/linkong/planet/docs/plans/ue5-mvp-fused-plan.md)
不适合放入这里的内容:

View File

@@ -0,0 +1,128 @@
# Admin Next 双轨全量迁移修正计划
## Summary
`/admin-next/*` 继续作为新版影子路由开发,旧 AntD 控制台必须完整保留作为生产回退和行为对照。只有新版完成全功能 parity 并通过验收后,才能逐个切换旧路由;删除旧页面和移除 AntD 必须作为最后独立步骤,并等待明确确认。
## Hard Rules
- 未完成 parity checklist 前,不删除旧页面、不移除 AntD、不切旧入口。
- “旧版能删除”只表示新版能力达到替代标准,不表示立即删除旧版。
- 新版不能只做看板旧版里可新增、编辑、删除、测试、触发、批量、轮询、WebSocket、权限控制、错误提示和确认弹窗的能力都必须迁移。
- `/admin-next/*` 可以重做交互和视觉但业务语义、API payload、权限和危险操作保护必须对齐旧版。
## Design Direction
- 风格采用“朴素工具化 + 轻微立体触感”:灰白/暗色低对比背景、清晰 1px 边框、轻微凸起/按下态、克制圆角、图标少量彩色。
- 禁止回到 soft-glass、hero、大发光、大渐变、大色块和装饰性卡片。
- 常规动作默认 icon-only + tooltip刷新、重启、退出、复制、查看、编辑、删除、关闭、设置。
- 强意图动作保留实心文字按钮:保存、创建、确认、执行、测试连接。
- 页面统一 `16px` 外边距、`16px` 主区块间距、`12px` 面板 header/body 间距;表格标题和表体不能贴边。
- 所有页面遵守一屏工作台:`PageHeader` 之外只允许一个主内容区域吃满剩余高度;详情区、表格区和 Playground 消息区内部滚动,不能被父级裁掉。
- 页内 tabs 必须按旧版用户心智组织,而不是按后端接口名拆散;接口状态只能作为分区内部信息。
## Implementation Plan
1. 恢复旧版安全网:恢复旧 AntD 页面、旧 `AppLayout`、旧 helper、AntD 依赖和旧路由;保留 `/admin-next/*` 影子路由。
2. 建立新版基础层:整理 admin-next token、按钮、tooltip、theme switch、sidebar account、`EntityTable``FormDialog``ConfirmDialog``DetailPanel`、移动端表格/卡片切换。
3. 拆掉通用看板页每个模块建立真实业务页面、API adapter、form/mutation、轮询或 WebSocket 逻辑,不再用万能 Resource table 聚合展示。
4. 按模块迁移:
- Dashboard统计、WebSocket、健康检查、重启任务、任务日志、恢复探测。
- DataSources内置源/自定义源/实时源、详情、stats、task-status、启停、触发、批量触发、清理数据。
- DataList列表、summary、sources/types/countries、搜索、筛选、分页、详情、导出、分布。
- Collection Managementconfigs CRUD、builtin connect/test、target schemas、mappings、run/stop、stream status。
- Settingssystem、notifications、security、SMTP、TV、integrations、collectors、测试连接、凭证指南。
- AIprovider、web search、OCR、prompt registry、playground 完整会话操作。
- Earth Contentbrand、upload、delete/reset、boundary config/build/status、TV/content。
- Logssources、level/date/search、刷新、详情、复制、tail/snapshot。
- BGPoverview、collectors、incidents、anomalies、events、AI briefs。
- Alerts系统告警、BGP 告警、态势告警,包含 ack/resolve/stats/brief。
- Userslist/create/edit/delete、role、Gatekeeper groups、权限显示。
- Auth登录、注册、验证邮箱、重发、找回、重置、logout。
5. 每完成一个模块,对照旧页面跑 checklist只有全部模块通过后才提出旧路由切换和旧版删除。
## Test Plan
- 恢复阶段:`cd frontend && /home/ray/.bun/bin/bun install && /home/ray/.bun/bin/bun run build`
- 旧路由检查:`/admin``/datasources``/settings``/ai``/logs` 必须继续打开旧版。
- 新路由检查:`/admin-next/*` 必须继续打开新版。
- 每模块 parity读写删、测试连接、触发、批量、轮询、WebSocket、权限、错误提示、确认弹窗、移动端和滚动。
- 视觉检查light/dark/system、低高度窗口、125%/150% 缩放、按钮/状态/badge 等高、无双滚动条。
## Parity Checklist
### Dashboard
- [x] `/dashboard/stats` 数据展示与旧版一致。
- [x] WebSocket 连接状态、健康状态、自动刷新语义一致。
- [x] 重启任务创建、轮询、日志查看、失败提示和恢复探测一致。
### DataSources
- [x] 内置源、自定义源、实时源分区清晰,不混表。
- [x] 详情、stats、task-status、enable/disable、trigger、trigger-batch、trigger-all、delete-data 与旧版一致。
- [x] 实时源 start/stop/restart、连接状态、任务状态与旧版一致。
### DataList
- [x] 列表、summary、sources/types/countries、搜索、筛选、分页参数与旧版一致。
- [x] 详情 metadata、分布摘要、JSON/CSV 导出与旧版一致。
- [x] 表格固定列、横向滚动、移动端卡片模式可用。
### Collection Management
- [x] configs/all、configs CRUD、builtin connect/test、custom sample 与旧版一致。
- [x] target schemas、mappings preview/list/update/activate 与旧版一致propose/create 已提供表单化入口。
- [x] run-mapped、stop-mapped、stream-status 与旧版一致。
### Settings
- [x] system、notifications、security、SMTP、TV、integrations、collectors 表单字段完整。
- [x] SMTP test、AI/Web/OCR secrets、connect tests、preset refresh 与旧版一致。
- [x] credential guides、cache 清理、危险操作确认一致。
### AI
- [x] Provider、Web Search、OCR、Prompt Registry 配置保存/重置完整。
- [x] Playground thread/session/messages/status/stop/resend/edit 完整。
- [x] provider refresh、connect test、secrets 读取、错误提示和 loading 状态完整。
- [x] 页内 tabs 按旧版心智恢复为模型供应商、工具调用、提示词、Playground。
### Earth Content
- [x] brand get/save/delete/reset/upload 完整。
- [x] boundary status/config/build/build-status 完整。
- [x] TV/content 配置与旧版 Settings 入口一致。
### Logs
- [x] sources、level/date/search、limit、刷新参数与旧版一致。
- [x] 日志正文、详情展开、复制、空态和错误态完整。
### BGP
- [x] overview、collectors、incidents、anomalies、events 的 summary/list/detail 完整。
- [x] AI brief list/detail/generate 完整。
- [x] collector collect-location、刷新、结果详情和错误提示完整。
### Alerts
- [x] 系统告警 list/detail/ack/resolve/stats/AI brief 完整。
- [x] BGP incidents/anomalies/brief 完整。
- [x] 态势告警 stats/list/brief 完整。
### Users
- [x] list/create/edit/delete、role、Gatekeeper groups 完整。
- [x] 权限显示、普通用户限制和 super admin 行为一致。
### Auth
- [x] login/register/verify/resend cooldown/forgot/reset/logout/me 完整。
- [x] 未验证邮箱跳转、错误提示、成功跳转和 token store 行为一致。
### Hierarchy Repair
- [x] AI 不再按接口/状态拍平成一张表;模型供应商恢复为 provider -> model/API/key 层级。
- [x] 工具调用恢复为 Web Search/OCR -> provider/API/高级参数层级。
- [x] 提示词恢复为 group -> prompt 入口层级,保存/重置只作用于当前入口。
- [x] Settings/Earth 配置型页面使用父级列表 + 右侧正式表单,长内容在一屏内内部滚动。
### Interaction Polish
- [x] 不会产生歧义的刷新、复制、详情、批量、状态、清理等动作优先改为 icon-only + tooltip/title。
- [x] 强意图动作保留实心文字按钮,例如保存、创建、触发全部、生成简报、确认执行。
- [x] 按钮字号、高度、图标容器和 hover 触感统一,接近侧栏深色模式滑块的轻微立体风格。
- [x] 表格与详情之间增加可拖动竖向 resize handle低高度和窄屏下不制造额外滚动条。
### Page Information Architecture
- [x] 页内 tab 默认使用中文BGP、AI、OCR、Web Search、Playground、Schema 等专有名词按可识别性保留。
- [x] 信息观测类页面使用列表 + 详情,例如 BGP、Alerts、Logs、DataSources。
- [x] 配置管理类页面使用分层结构,例如 AI、Settings、Earth Content、Collection Management。
- [x] 采集管理已从纯表格看板改为分层管理:采集器、映射模板、目标 Schema、运行状态先选父级再编辑或执行动作。

View File

@@ -0,0 +1,135 @@
# Admin Next Parity Audit Closeout
Last updated: 2026-05-21
## Status
Admin Next has been promoted to the official admin route family. The old AntD admin, old layout helpers, `antd`, and `@ant-design/icons` remain available under `/legacy/admin/*` as the rollback and comparison surface.
`/admin-next/*` is now compatibility-only: old test links redirect to the official route. Do not add new capabilities there as a separate entry point.
This closeout document is the final work log for the second parity audit. It records what has been fixed, what still requires manual verification, and what must not be treated as complete.
## Fixed In This Audit
- Restored the dual-track rule: old AntD pages and routes are not deleted and are not replaced before parity sign-off.
- Reworked management pages away from flat status tables toward business hierarchy:
- AI: `模型供应商 / 工具调用 / 提示词 / Playground`.
- Settings: system display, notification policy, security policy, SMTP only.
- Earth Content: brand, boundary precision, TV content.
- Collection: collector, collection schedule, history/snapshot.
- Repaired AI provider credential semantics:
- Provider status now comes from that provider's stored key or a runtime key that is explicitly scoped to that provider.
- Generic `.env` fallback no longer marks every provider as configured.
- Runtime fallback secrets stay masked by default; authorized administrators can reveal them through the console and each reveal must be audited without storing plaintext.
- Secret previews preserve the prefix before the first `-` when the backend preview exposes it, for example `sk-**********`.
- Connect-test buttons are icon-only plug actions and do not save configuration.
- Repaired key UI gaps:
- LLM API Key, proxy token, Web Search API Key, and OCR API Key use masked input with inline eye toggle.
- Save buttons use the disk icon.
- Default action is disabled when the item is already default.
- Repaired SMTP test flow:
- SMTP test has a recipient input.
- Request payload is `{ to, settings }`.
- Repaired Earth Content actions:
- Brand save, upload, delete/reset actions are available in the brand section.
- Brand preview renders with the same Earth left-top brand structure and starfield-style background.
- Boundary build is scoped to the boundary precision section, not the global page toolbar.
- TV default configuration is folded into item-level configuration; TV items support add/default/reset/delete semantics.
- TV default source now follows the same default-state semantics as AI Provider: if the response does not expose an explicit default, `cgtn-en` is treated as the runtime default; setting a default promotes that item visually and disables the redundant default action.
- Repaired Collection actions:
- Credential guide read/generate/reset entry points are available from collector-related panels.
- Create collector config uses field-first form controls with advanced JSON as a secondary path.
- Mapping propose/preview/create/activate flow is represented in the mapping workflow.
- Repaired shared UI rules:
- Management list names can wrap and remain readable instead of being squeezed by status tags.
- Status tags are fixed width and color-coded: default blue, configured green, unconfigured gray, error red.
- Configuration lists with a default item sort the default item to the top after save/default changes instead of leaving it in the previous alphabetical or API order.
- List footer actions are part of the scroll content; users see them after scrolling to the bottom.
- Footer icon buttons use the current tactile surface direction: external shadow first, without a separate inset/base-plate treatment.
- Detail panels use a dark neutral resize handle.
- Tooltip, dialog, modal backdrop, textarea scrollbar, custom scrollbar, mobile detail, and one-screen layout issues from the audit have been addressed in the shared admin-next layer.
- Multi-tab Admin Next pages now use lazy active-tab loading with local cache. Initial page load no longer requests every section endpoint; manual refresh and mutating actions refresh only the current section.
- Markdown-producing details, AI brief content, credential tutorials, and Playground assistant output use the shared Markdown renderer where the content is meant to be read as a document. Raw metadata remains available only where the page is explicitly showing original payloads.
- Added developer workflow scripts:
- `build:watch`
- `preview:auto`
- `bun run build` remains a production artifact build and does not reload an already-open dev page by itself.
## Manual Verification Status
The code has been updated. The list below records which areas are already manually checked and which still need route-level regression before replacement:
- AI:
- Latest manual pass marked provider status, secret reveal, and Playground behavior as checked.
- DataSources:
- Single trigger performs old task-status precheck, handles running-conflict confirmation, supports force recollect, and refreshes task status afterward.
- Batch trigger uses old `/datasources/trigger-batch` semantics for selected IDs, filters, and force.
- Builtin, custom, and realtime source actions match old enable/disable/start/stop/restart/clear-data behavior.
- 2026-05-20 implementation pass: Admin Next builtin datasource rows now expose task/collection status and metrics, and single trigger now performs task-status precheck with running-conflict force confirmation plus post-trigger task refresh.
- 2026-05-20 manual pass: builtin trigger, task-status precheck, force recollect, batch trigger, enable/disable, and clear-data were verified in Admin Next.
- Collection:
- Collector config create/update/delete, builtin connect, custom test/sample/run/stop/status, schedule save, history/snapshot, mapping propose/preview/create/activate, and credential guide read/generate/reset all work with real backend payloads.
- 2026-05-20 implementation pass: `采集历史 / 快照` now reads real datasource snapshots from `/datasources/snapshots` instead of config rows.
- 2026-05-20 implementation pass: default credential guide actions no longer call unsupported providers blindly; unsupported collectors show a clear empty state and supported providers use `barentswatch` / `aisstream`.
- Collection action buttons must follow the AI page's single-responsibility rule: navigation/link actions do not save, save actions do not connect/test, and connect/test actions do not persist drafts.
- 2026-05-20 manual pass: collector create/edit `auth_config`, mapping propose/preview/create/activate, custom test/sample/run/stop/status, credential guide read/generate/reset, and history/snapshot behavior were verified in Admin Next.
- 2026-05-20 follow-up: dead JSON modal create/mapping code was removed so Collection creation now goes through the list-bottom draft workflow only.
- Earth Content:
- Brand preview visually matches the Earth runtime left-top brand at the same structural level.
- Brand upload writes back `logo_src` or `title_src` correctly.
- 2026-05-20 implementation pass: Brand upload validates and displays accepted suffixes (`png`, `jpg`, `jpeg`, `webp`, `svg`) and supports drag-and-drop.
- Boundary status/config/build and TV add/save/default/delete/reset match old behavior.
- Logs:
- Sources, snapshot/tail, level/date/search, copy, empty/error states, and internal scrolling all work.
- 2026-05-20 implementation pass: Logs now include database-backed `system-db` and `audit-db` sources as a fallback when file/docker/buffer logs are empty or missing, and the UI auto-selects an available source. Manual regression must still verify snapshot/tail-like reading, filters, copy, and scrolling.
- BGP / Alerts:
- BGP overview, collectors, collect-location detail, incidents, anomalies, events, latest/detail/generate AI brief.
- System/BGP/situational alert detail, ack, resolve with note, refresh after mutation, and AI brief details.
- Users/Auth:
- Users CRUD, role, Gatekeeper groups.
- 2026-05-20 implementation pass: User edit select dropdowns now use an opaque high-z-index menu surface, and Gatekeeper checkbox groups use an explicit surface background. Manual visual regression is still required.
- Login, register, verify email, resend cooldown, forgot/reset password.
- Mobile:
- 375px, 430px, and 768px widths default to list/group first.
- Detail opens only after tap/click and has a visible back action.
- Long forms, JSON, logs, and Playground messages scroll internally without body-level double scrollbars.
## Explicit Non-Production Items
The old AntD page itself treated these Earth content tabs as placeholder-level capabilities. Admin Next must show a visible “backend capability pending” empty state and must not mix in unrelated data:
- `basemap`
- `layer_resources`
- `models_3d`
- `news_anchor_strategy`
If backend endpoints are later added, these items must be promoted into `docs/plans/admin-next-parity-checklist.md` with concrete API and UI acceptance criteria.
## Final Gate
After the route promotion, the final gate is no longer “switch old routes.” The remaining gate is “keep legacy available until the promoted routes are stable enough to remove AntD.”
1. Run `cd frontend && /home/ray/.bun/bin/bun run build`.
2. Run the static checks:
- `rg "map: \\(\\) => \\[\\]|暂不支持保存|placeholder" frontend/src/admin-next`
- `rg "ShadowPage|FeatureConsole|GlassPanel|InspectorDrawer" frontend/src/admin-next`
3. Manually verify every official route listed in `docs/plans/admin-next-parity-checklist.md`.
4. Confirm `/legacy/admin/*` still opens old AntD pages during the validation window.
5. Delete old AntD pages and remove AntD dependencies only as a separate final cleanup task after explicit confirmation.
## Current Build Verification
Last successful local build during this closeout, after the 2026-05-20 follow-up fixes:
```bash
cd frontend && /home/ray/.bun/bin/bun run build
```
Result: passed.
Backend syntax check also passed:
```bash
/home/ray/.local/bin/uv run python -m py_compile backend/app/api/v1/datasources.py backend/app/api/v1/datasource_config.py backend/app/api/v1/system_control.py
```

View File

@@ -0,0 +1,43 @@
# Admin Next Parity Checklist
Last updated: 2026-05-21
This checklist is the hard gate for deleting the old AntD admin. Admin Next now owns the official admin routes; old AntD pages stay available under `/legacy/admin/*` until every production capability below is verified.
## Route Gate
- `/admin` Dashboard: stats, health probe, WebSocket status, restart task, restart logs, recovery probe.
- `/datasources`: builtin sources, custom sources, realtime sources, detail/stat/task-status, trigger, batch trigger, enable/disable, clear data, realtime start/stop/restart, custom source test/sample/run/stop/status link.
- `/data`: list, summary, source/type/country filters, search, pagination, detail metadata, export, distribution chart.
- `/collection-management`: collector config, mapping templates, target schemas, collection schedule, history/snapshot, create config, propose/preview/create/activate mapping, builtin connect, run/stop/stream-status, credential guide read/generate/reset.
- `/settings`: system display, notifications, security, SMTP get/save/test with recipient.
- `/ai`: model providers, tool calling, prompts, key reveal, provider refresh, connect tests, OCR/WebSearch secrets, Playground thread/chat/edit/resend/stop.
- `/earth-content`: brand preview as Earth renders it, upload/save/delete/reset, boundary status/build/config, TV source add/save/default/delete/reset.
- `/logs`: sources, filters, snapshot, empty/error states, copy.
- `/bgp`: overview, collectors, collect-location, incidents, anomalies, events, AI brief list/latest/detail/generate.
- `/alerts/system`, `/alerts/bgp`, `/alerts/situational`: list/stat/detail, ack, resolve with note, AI brief generation.
- `/users`: list/create/edit/delete, role, Gatekeeper permission groups.
- Auth pages: login, register, verify email, resend cooldown, forgot/reset password.
Compatibility routes under `/admin-next/*` should redirect to these official paths and are not a separate validation target.
## Verified Manual Passes
- 2026-05-20 `/admin-next/collection-management`: user verified collector create/edit payloads, mapping propose/preview/create/activate, custom collector test/sample/run/stop/status, credential guide read/generate/reset, and collection history/snapshot behavior after the draft-form and Time Capsule updates.
- 2026-05-20 `/admin-next/datasources`: user verified builtin source trigger, task-status precheck, force recollect, batch trigger, enable/disable, and clear-data behavior.
## Known Non-Production Tabs
The old AntD page also marked these Earth content tabs as placeholder-level capabilities. Admin Next must show them as “backend capability pending” and must not mix in unrelated data:
- `basemap`
- `layer_resources`
- `models_3d`
- `news_anchor_strategy`
## Replacement Rules
- Keep `/legacy/admin/*` available until the matching route gate above is manually verified after promotion.
- Do not delete old AntD pages, old layout helpers, `antd`, or `@ant-design/icons` until final explicit confirmation.
- Do not use fake rows, unrelated endpoint data, or empty adapters for a real old capability.
- Any backend-missing capability must be recorded here with the missing endpoint and visible UI empty state.

View File

@@ -0,0 +1,217 @@
# Admin Next Soft Glass Goal-Driven Plan
## Goal
Rebuild `/admin-next/*` into a modern soft-glass / light-neumorphic console while keeping the legacy Ant Design admin routes available for comparison. The new console must use visual references only as design anchors, not as imported templates, and must be implemented as Planet-owned reusable components and page patterns.
The redesign must cover desktop and mobile. Data display, icon semantics, table readability, and scroll behavior are first-class acceptance criteria.
## Criteria For Success
- This plan exists at `docs/plans/admin-next-soft-glass-goal-driven-plan.md`.
- `/admin-next/*` has real pages for every route; route usage of `ShadowPage` is removed.
- Admin Next supports `system`, `light`, and `dark` theme modes using the same persistence and system-theme idea as Docs.
- The visual language reads as soft-glass / light-neumorphic instead of an AntD reskin: translucent panels, fine borders, subtle glow, cool backgrounds, restrained accent colors, crisp icons, and tactile controls.
- Mobile is explicitly designed: mobile navigation, filter sheets/chips, card-list data views, full-screen detail sheets, and bottom action bars instead of squeezed desktop tables.
- Tables do not show double scrollbars. Long fields are inspectable, copyable, or expandable; important columns stay readable.
- Existing custom scrollbar components remain the default scroll experience.
- `cd frontend && bun run build` passes.
- Static checks can prove no route-level placeholder remains.
## Reference Strategy
Use references to calibrate feel, not to take over code:
- User-provided soft-glass / light-neumorphic dashboard images define the desired mood.
- Shadcn / Tailwind / glassmorphism admin templates may be inspected for shadow, spacing, panel, dark-mode, and app-shell ideas.
- Do not import a complete template, router, auth layer, state layer, or business page.
- Planet-specific interactions such as datasource tasks, BGP briefs, AI settings, log streams, Earth resources, and custom scrollbars are implemented in this repo.
## Source-Of-Truth Mining
Design content and page boundaries from existing repo history and docs:
- `docs/CHANGELOG.md`
- `docs/plans/*`
- `docs/technical/*frontend*`
- `TODO.md`
Important current boundaries:
- `DataSources` is a datasource directory and runtime-control page, not a configuration editor.
- `Collection Management` owns collector endpoint, credential, custom source, mapping, and runtime configuration.
- `Earth Content` owns TV livestreams, brand assets, boundary precision, basemap, and Earth display resources.
- `AI` owns provider settings, tools, prompts, and Playground.
- `BGP` is an observability and evidence workspace, not just tables.
- `Alerts` is a duty/analysis workspace for system, BGP, and situational risk.
- `Logs` should become a usable log workbench with filtering and structured detail.
- `Settings` should keep platform settings only.
## Reusable Architecture
Recommended structure:
- `admin-next/design`: theme tokens, theme mode, status colors, icon semantics, chart themes, scrollbar themes.
- `admin-next/components`: primitive UI such as button, input, select, tabs, dialog, drawer, badge, status pill, icon badge.
- `admin-next/patterns`: page patterns such as `PageFrame`, `GlassPanel`, `MetricCard`, `CommandBar`, `FilterRail`, `InspectorDrawer`, `DataWorkspace`, `SettingsWorkspace`, `ScrollableRegion`, `TableRegion`, and `MobileSheet`.
- `admin-next/features`: business modules such as datasources, data-list, bgp, alerts, ai, logs, settings, earth-content, collection-management, users.
- `admin-next/services`: feature API gateways and hooks. Pages should not scatter URL construction.
- `admin-next/routes`: route manifest used by navigation, command search, breadcrumbs, and legacy links.
## Interaction Rules
- Row click opens an inspector; frequent row actions live in the inspector header.
- Bulk selection opens a floating command bar.
- Common filters stay visible; advanced filters open a sheet/drawer.
- Detail, JSON, log context, and AI brief use inspector or reading panels instead of stacked modal dialogs.
- Settings are split into `FormSection` blocks with local save, dirty state, connection test, and reset.
- Dangerous actions use a confirm sheet.
- Loading, empty, error, and retry states are shared through reusable patterns.
## Mobile Rules
- Do not squeeze desktop layout into mobile.
- Use mobile navigation and sheets instead of a permanent desktop sidebar.
- Filters collapse into a filter sheet; active filters remain visible as chips.
- Data workspaces default to card-list mode on mobile and allow explicit table mode for dense comparison.
- Details open in full-screen mobile sheets.
- Batch actions use a sticky bottom command bar.
- Long logs and JSON use full-screen reading/editing surfaces with custom scrollbars.
## Table And Data Display Rules
- Tables must prioritize complete data display:
- key columns get stable widths and priority;
- long fields have peek/expand or inspector detail;
- IDs, URLs, metadata, and errors are copyable;
- complex fields are rendered fully in inspector.
- Desktop tables use one `TableRegion` scroll container.
- Mobile defaults to card-list display and supports dense table mode.
- Native scrollbar visuals are hidden where custom scrollbars apply.
## Icon Rules
- Use `lucide-react`.
- Define icon semantics by category:
- navigation icon;
- entity icon;
- status icon;
- action icon;
- severity icon.
- Icons must not be the only semantic carrier unless the icon is universally obvious; use labels or tooltips.
- Status icons pair with `StatusPill`.
- Colors come from tokens, not ad hoc hard-coded values.
## Page Design
### Dashboard
- System health, realtime connection, datasource activity, task trend, alert summary, recent events, and quick commands.
- Restart flow uses a confirm sheet and log console.
- Mobile uses horizontal metric cards and collapsible event/command panels.
### DataSources
- Datasource directory and runtime control.
- Desktop: overview rail, datasource table/list, inspector.
- Mobile: datasource cards with status, latest task, trigger/toggle actions.
- Batch trigger uses command bar.
- Endpoint/config/task details are shown fully in inspector.
### DataList
- Collection result browsing, search, source/type filters, paging, distribution chart, and detail inspector.
- Fix double scrollbars.
- Mobile defaults to cards and allows table mode.
### BGP
- Views: collectors, incidents, anomalies, events, AI brief.
- Emphasize severity, region, ASN/prefix, collector coverage, and evidence.
- AI brief shows facts, judgment, and evidence gaps.
### Alerts
- Real pages for system, BGP, and situational alerts.
- Shared alert workspace with stats, severity/status filters, list, and inspector.
- AI brief appears in inspector/mobile sheet.
### AI
- Provider, tools, prompts, and Playground.
- Provider and tool settings use local save and connection tests.
- Prompt registry is grouped by task.
- Mobile Playground uses step tabs rather than cramped columns.
### Logs
- Log workbench with source, level, date, search, refresh, copy, and structured detail.
- Desktop uses a terminal-like log stream.
- Mobile rows expand or open a full-screen reading sheet.
### Users
- Search, role filter, create/edit/delete, Gatekeeper groups.
- Edit/create in drawer/sheet.
- Gatekeeper groups use checkbox chips.
### Settings
- Platform settings only: system, notification, security, SMTP.
- Each section saves independently.
### Earth Content
- TV livestreams, brand assets, boundary precision, basemap, and layer resources.
- Resource previews, upload state, and boundary build status are emphasized.
### Collection Management
- Collector config, custom sources, mapping, runtime control.
- Custom source creation uses a step sheet.
- Advanced JSON is isolated in a collapsible editor.
## Milestones
1. Save this plan and establish goal-driven criteria.
2. Build design/theme/pattern foundations.
3. Remove route-level placeholders and create real pages for all `/admin-next/*` routes.
4. Redesign layout, theme, mobile shell, scroll behavior, and table behavior.
5. Fill core workspaces with real API calls and reusable patterns.
6. Verify build, no `ShadowPage` route usage, and core route availability.
## Verification
Run:
```bash
cd frontend && bun run build
rg "ShadowPage" frontend/src/admin-next
rg "axios\\." frontend/src/admin-next/pages
rg "overflow: auto|overflow-y: auto" frontend/src/admin-next
```
Manual route checks:
- `/admin-next`
- `/admin-next/datasources`
- `/admin-next/data`
- `/admin-next/bgp`
- `/admin-next/alerts/system`
- `/admin-next/alerts/bgp`
- `/admin-next/alerts/situational`
- `/admin-next/ai`
- `/admin-next/logs`
- `/admin-next/users`
- `/admin-next/settings`
- `/admin-next/earth-content`
- `/admin-next/collection-management`
Manual viewport checks:
- desktop;
- mobile width;
- low height;
- 125% / 150% browser zoom;
- light / dark / system theme modes.

View File

@@ -0,0 +1,810 @@
# Agent Runtime, Earth LLM Command, And Speech Entry Plan
## Summary
Build an auditable backend Agent Runtime and upgrade the existing Earth search panel into a combined search, AI command, and voice wake entry. The first version is a runtime foundation, not the full multi-role simulation product yet.
Typical user goals:
- In Earth, type "高亮所有北斗卫星" and have the system open the satellite layer and highlight matching Beidou satellites.
- Type "中国大陆的算力中心" and have the system open the compute-center layer, match mainland China compute centers, and highlight them.
- When microphone permission is granted, use a configurable wake word, then speak an Earth command.
- Save every Earth AI command as an agent run so operators can review the original input, speech transcription, tool steps, entity matches, final action plan, and frontend execution result.
Core boundaries:
- `aiprovider` remains the model gateway. It must not own business tools, database access, Earth actions, or agent policy.
- The backend owns agent orchestration, tools, evidence storage, permission policy, and proposal application.
- Earth v1 executes visualization actions only. It does not mutate business data.
- Speech recognition uses a provider-neutral ASR API first, defaulting to OpenAI/Whisper-compatible transcription APIs, with local `whisper.cpp` style providers as later adapters.
## Architecture
### Backend Agent Runtime
Recommended module shape:
```text
backend/app/models/
agent_run.py
agent_step.py
agent_evidence.py
agent_proposal.py
backend/app/schemas/
agents.py
speech.py
backend/app/services/agents/
runtime.py
orchestrator.py
tool_protocol.py
tool_registry.py
policy.py
proposals.py
earth_command.py
entity_query.py
speech.py
backend/app/api/v1/
agents.py
```
Database conventions should follow the current project style:
- Use SQLAlchemy models.
- Import new models from `init_db()`.
- Let `Base.metadata.create_all` create tables.
- Add required indexes with `CREATE INDEX IF NOT EXISTS`.
- Do not introduce Alembic for this feature.
### aiprovider Boundary
`aiprovider` should continue to provide model transport only:
- Keep the existing `/v1/analyze` endpoint.
- If schemas are extended, only pass through model/provider request fields and normalize responses.
- Do not add WebSearch, database queries, Earth entity lookup, or business action execution inside `aiprovider`.
- If a provider does not support native tool calling, the backend must use JSON tool-call fallback.
### Agent Tool Protocol
The first version should support two protocols:
- Default path: backend JSON tool-call loop.
- Optional path: provider-native tools when the configured provider supports them.
- Fallback path: if provider-native tools are unavailable or unstable, automatically use JSON tool-call.
Example JSON tool call:
```json
{
"type": "tool_call",
"tool": "earth.find_entities",
"arguments": {
"domain": "satellites",
"filters": {
"constellation": "beidou"
},
"limit": 500
}
}
```
Example final response:
```json
{
"type": "final",
"summary": "已找到并高亮北斗卫星。",
"result": {
"actions": []
}
}
```
Hard limits:
- One run may execute at most 8 tool steps by default.
- One tool call times out after 30 seconds by default.
- Only registered whitelist tools may run.
- Tool arguments must pass Pydantic validation.
- Illegal tools, invalid arguments, and denied actions must be recorded as step errors.
- LLM output may not directly write business state. Writes are either proposals or backend-executed policy-approved actions.
## Data Model
### AgentRun
Suggested fields:
- `id`: integer primary key.
- `public_id`: public string id.
- `run_type`: `earth_command`, `situational_awareness`, `config_proposal`, or `diagnostic`.
- `status`: `queued`, `running`, `waiting_approval`, `completed`, `failed`, or `stopped`.
- `title`: short display title.
- `objective`: text objective.
- `input`: JSONB original input, including text and audio metadata.
- `context`: JSONB run context.
- `result_markdown`: final human-readable output.
- `result_json`: structured output, including Earth action plans.
- `provider`: nullable provider id.
- `model`: nullable model id.
- `request_id`: nullable propagated request id.
- `created_by`: user id.
- `created_at`, `updated_at`, `completed_at`.
- `error`: nullable text error.
### AgentStep
Suggested fields:
- `id`.
- `run_id`.
- `step_index`.
- `step_type`: `llm`, `tool`, `policy`, `action`, or `transcription`.
- `status`: `pending`, `running`, `completed`, `failed`, or `skipped`.
- `name`: step name, for example `earth.find_entities`.
- `input`: JSONB.
- `output`: JSONB.
- `error`: nullable text.
- `started_at`, `completed_at`.
- `duration_ms`.
### AgentEvidence
Suggested fields:
- `id`.
- `run_id`.
- `step_id`: nullable.
- `evidence_type`: `internal_record`, `web_search`, `web_fetch`, `entity_match`, or `transcription`.
- `source`: source id.
- `title`: display title.
- `url`: nullable source URL.
- `content`: text evidence content.
- `content_hash`: nullable hash.
- `metadata`: JSONB.
- `retrieved_at`.
### AgentProposal
Suggested fields:
- `id`.
- `run_id`.
- `proposal_type`: `datasource_config`, `ai_prompt`, or `external_integration`.
- `status`: `pending`, `approved`, `applied`, `rejected`, or `failed`.
- `risk_level`: `low`, `medium`, or `high`.
- `target`: JSONB target descriptor.
- `before_payload`: JSONB.
- `after_payload`: JSONB.
- `rationale`: text.
- `policy_result`: JSONB.
- `applied_by`: nullable user id.
- `applied_at`: nullable timestamp.
- `error`: nullable text.
## Public APIs
### Agent Runs
Add:
```text
POST /api/v1/agents/runs
GET /api/v1/agents/runs
GET /api/v1/agents/runs/{run_id}
POST /api/v1/agents/runs/{run_id}/stop
POST /api/v1/agents/proposals/{proposal_id}/apply
```
### Earth Command
Add:
```text
POST /api/v1/agents/earth/command
```
Request shape:
```json
{
"input_text": "高亮所有北斗卫星",
"source": "text",
"transcription_id": null,
"client_context": {
"visible_layers": ["satellites"],
"locale": "zh-CN",
"viewport": {
"is_mobile": false
}
}
}
```
Response shape:
```json
{
"run_id": "agent_xxx",
"summary": "已找到并高亮北斗卫星。",
"actions": [
{
"type": "show_layer",
"layer": "satellites",
"enabled": true
},
{
"type": "highlight_entities",
"domain": "satellites",
"entity_ids": ["satellite:norad:12345"],
"style": {
"color": "#7dd3fc"
}
},
{
"type": "open_result_panel",
"title": "北斗卫星",
"items": []
}
],
"matched_entities": [],
"confidence": 0.86,
"missing_data": []
}
```
### Speech / ASR
Add:
```text
POST /api/v1/agents/speech/transcriptions
```
Request should use multipart form data:
- `file`: audio blob.
- `language`: default `zh`.
- `provider`: optional provider override.
- `source`: default `earth_command`.
Response shape:
```json
{
"text": "高亮所有北斗卫星",
"provider": "openai_compatible",
"model": "whisper-1",
"duration_ms": 1234,
"confidence": null,
"metadata": {}
}
```
## Earth Action Plan
### Allowed Action Types
The first version may only return:
```text
show_layer
highlight_entities
filter_entities
focus_view
open_result_panel
clear_highlight
```
### Allowed Domains
The first version supports:
```text
satellites
compute_centers
bgp
news
vessels
cables
```
### EarthAction Shape
Example:
```json
{
"type": "highlight_entities",
"domain": "satellites",
"entity_ids": ["satellite:norad:12345"],
"style": {
"color": "#7dd3fc",
"mode": "glow"
},
"reason": "用户要求高亮北斗卫星"
}
```
### Safety Rules
The backend must validate action plans before returning them:
- `type` must be in the allowed action list.
- `domain` must be in the allowed domain list.
- `entity_ids` must come from backend or current Earth candidate data. The model may not invent ids.
- One highlight action should include at most 500 entities by default. If more match, return a truncation note in `missing_data` or `summary`.
- `focus_view` must include valid coordinates, a valid region, or a matched entity.
- No action may return executable JavaScript, arbitrary CSS, or arbitrary URLs to fetch.
## Earth Entity Query
Add a backend entity query service used by both deterministic resolvers and LLM tools.
### Satellites
Sources:
- `/api/v1/visualization/geo/satellites`.
- Current TLE collected data.
- GeoJSON feature properties.
Filters:
- `constellation`: `beidou`, `gps`, `galileo`, `glonass`, `starlink`, `iridium`, `geo`, `leo`.
- Name contains.
- NORAD id.
- Country/operator when present in data.
- Orbital class when inferable from existing fields.
Beidou matching:
- Prefer `constellation == beidou`.
- Then match names containing `BEIDOU`, `BDS`, `BEIDOU-`, or `北斗`.
- Stable entity id format should be `satellite:norad:{norad_id}` when possible, with fallback `satellite:index:{index}`.
### Compute Centers
Sources:
- `/api/v1/visualization/geo/compute-centers`.
- Unified TOP500 and Epoch AI GPU GeoJSON properties.
Filters:
- Country/region.
- `site_type`: `supercomputer` or `gpu_cluster`.
- Source: `top500` or `epoch_ai_gpu`.
- Name contains.
- `needs_confirmation`.
- `location_precision`.
Mainland China matching:
- Match country values such as `China`, `中国`, or `People's Republic of China`.
- Exclude obvious non-mainland records when fields identify Hong Kong, Macau, or Taiwan.
- If records do not expose enough region detail to separate mainland China from Hong Kong, Macau, or Taiwan, return a `missing_data` note and conservatively match `country=China`.
### BGP, News, Vessels, And Cables
First-version basic support:
- BGP: severity, status, region, collector, prefix, ASN.
- News: region, source, localized title, localized summary.
- Vessels: vessel type, country/area, name, status.
- Cables: cable name, landing point, country/region.
## Earth Frontend Integration
### Search Panel Merge
Reuse the existing Earth search panel:
- Default behavior remains normal local search.
- Add an AI command state for natural-language commands.
- Add a "use AI" command button.
- Add a microphone button.
- Show running status, result summary, and clear-highlight action.
- Visually distinguish ordinary search results from AI action results.
Natural-language routing:
- If the user clicks the AI command button, always call the AI command endpoint.
- If input contains action words such as `高亮`, `显示`, `找出`, `聚焦`, `打开`, `筛选`, `隐藏`, or `清除`, suggest AI command mode.
- Short ordinary keywords continue to use local search.
### Earth Action Executor
Add:
```text
frontend/public/earth/js/earth-command-actions.js
```
Responsibilities:
- Receive backend `EarthActionPlan`.
- Toggle required layers.
- Apply highlights and filters.
- Focus view when requested.
- Open or update the result panel.
- Clear the previous AI command highlight.
Execution order:
1. Clear previous AI command highlights unless the plan is explicitly additive.
2. Show required layers.
3. Apply filters and highlights.
4. Focus the view if requested.
5. Open the result panel.
6. Record the execution summary in UI state.
### Satellite Highlight
Existing `highlightRelatedSatellites(indices, color)` can be reused, but Earth command support needs an entity-id-to-index map.
Add:
- Satellite id map built from current satellite data.
- Highlight by `entity_ids`.
- Clear AI command satellite highlights without clearing manual locked selection.
- Auto-enable the satellite layer before highlighting.
### Compute Center Highlight
Existing compute center markers support marker state, but need a dedicated batch AI highlight state.
Add:
- Batch set compute center markers to AI highlighted state.
- Keep AI highlight compatible with hover and locked states.
- Clear AI command compute highlights without clearing manual locked selection.
- Auto-enable the compute-center layer before highlighting.
### Result Panel
First version can show results inside the merged search panel:
- Title, for example `北斗卫星`.
- Count, for example `已高亮 32 个对象`.
- List with the first 20 entities.
- Actions: clear highlight, rerun, view Agent Run.
## Voice And Wake Word
### ASR Configuration
Add Speech/ASR under the AI settings tool tab.
Fields:
- `enabled`.
- `provider`.
- `base_url`.
- `api_key`.
- `model`.
- `language`.
- `timeout_seconds`.
- `max_audio_size_mb`.
First-version providers:
- `openai_whisper`.
- `openai_compatible`.
- `local_whisper`.
Recommended defaults:
- `provider`: `openai_compatible`.
- `model`: `whisper-1` or user-configured equivalent.
- `language`: `zh`.
Secret handling:
- Use the existing masked secret pattern.
- Do not allow agent proposals to write API keys.
- Support environment fallback such as `ASR_API_KEY` and `OPENAI_API_KEY`.
### Wake Word
The wake word is configurable.
Implementation rules:
- Treat wake word as a browser-local preference in v1.
- Store it in `localStorage`.
- Default suggestion: `小星球`.
- Provide a wake-word setting inside the Earth search/command panel.
- Do not listen until the user explicitly enables voice wake.
- Do not upload audio before wake.
- After wake, record one instruction audio segment and upload it to backend ASR.
Fallbacks:
- Browser does not support continuous local recognition: fall back to click-to-record.
- Microphone permission denied: show `麦克风权限未开启,仍可输入文字指令`.
- ASR not configured: show `语音识别未配置`; text commands remain available.
- User can disable wake listening and keep manual microphone recording.
### Wake Word Technical Choice
Do not require local Whisper in v1.
Recommended path:
- Use browser Web Speech API for local wake-word detection when available.
- Fall back to click-to-record when unavailable.
- Upload only the post-wake instruction audio to backend ASR.
- Add local `whisper.cpp` streaming wake-word or ASR provider later.
## Prompt Registry
Add default prompt keys:
```text
agents.runtime.system
agents.earth.command
agents.situational.assessment
agents.config.proposal
agents.roles.network
agents.roles.bgp
agents.roles.platform_ops
agents.roles.business_impact
```
`agents.earth.command` must require:
- Strict JSON output only.
- Use only provided candidate entities.
- Do not invent objects.
- Do not return arbitrary code.
- Do not modify business data.
- Return `clarification_needed` when intent is unclear.
- Return action plans matching the schema.
- Use Chinese summary for Chinese user input.
## Policy
### Tool Policy
- The LLM can request tools, but the backend decides whether to execute.
- Every tool must declare:
- name.
- description.
- input schema.
- output schema.
- permission.
- side-effect level.
- First-version side-effect levels:
- `read`: can run directly.
- `proposal`: can only generate proposals.
- `write`: can only execute during proposal apply.
### Proposal Policy
First version supports configuration proposal application with role-based automatic/manual gating:
- `super_admin` may enable automatic application for low-risk proposals.
- Normal admins must manually confirm.
- High-risk proposals always require manual confirmation.
- Proposals containing secret fields are rejected.
- Proposals failing schema validation are rejected.
- Before and after payloads must be saved.
- Failed applications must save errors.
Low-risk scope:
- Datasource endpoint/config non-secret fields.
- AI prompt overrides.
- External integration non-secret fields.
Out of scope for v1:
- Alert acknowledge/resolve.
- Data deletion.
- User permission changes.
- Authentication configuration changes.
- Database schema changes by agent.
- Mutation of original Earth collected data.
## Agent Operations UI
Add an `Agent` page under `运维与配置`.
Run list:
- Status.
- Type.
- Title.
- Creator.
- Model.
- Time.
- Duration.
Run detail:
- Input.
- Status timeline.
- LLM steps.
- Tool steps.
- Evidence.
- Final result.
- Proposals.
Proposal apply:
- Before/after diff.
- Risk level.
- Policy result.
- Apply button.
- Reject button.
Earth command run detail:
- Original text or speech transcription.
- Matched entities.
- Action plan.
- Link back to Earth or copy run id.
## Frontend Types
Add or extend:
```ts
interface AgentRun {}
interface AgentStep {}
interface AgentEvidence {}
interface AgentProposal {}
interface EarthCommandRequest {}
interface EarthActionPlan {}
interface EarthAction {}
interface SpeechTranscriptionResponse {}
```
Earth action executor API:
```js
executeEarthActionPlan(plan, context)
clearEarthCommandHighlights()
getEarthCommandExecutionState()
```
## Testing
### Backend Tests
Use `uv`.
Suggested files:
```text
backend/tests/test_agents_runtime.py
backend/tests/test_agent_tool_protocol.py
backend/tests/test_agent_policy.py
backend/tests/test_earth_agent_command.py
backend/tests/test_speech_transcription.py
```
Coverage:
- Create run.
- Persist ordered run steps.
- Parse valid JSON tool calls.
- Reject invalid JSON tool calls without executing tools.
- Reject unregistered tools.
- Reject invalid tool arguments.
- Pass through provider-native tools fields when configured.
- Fall back to JSON tool calls when native tools are unavailable.
- When WebSearch is not configured, record `missing_data` and do not fail Earth command.
- "高亮所有北斗卫星" returns a satellite highlight action.
- "中国大陆的算力中心" returns a compute-center highlight action.
- Mainland China matching excludes Hong Kong, Macau, and Taiwan when fields permit it.
- Field granularity limitations are returned in `missing_data`.
- ASR not configured returns a clear error.
- ASR provider success returns transcription.
- Normal admins cannot auto-apply proposals.
- `super_admin` can apply low-risk proposals.
- Secret-field proposals are rejected.
### Frontend Tests
Use `bun`.
Required build check:
```bash
cd frontend && bun run build
```
Suggested Earth JS tests:
- Action executor opens the satellite layer.
- Action executor highlights satellite entity ids.
- Action executor highlights compute-center entity ids.
- Clear highlight does not clear manual locked selection.
- Search panel ordinary search still uses local search.
- AI command button calls `/api/v1/agents/earth/command`.
- ASR missing configuration and microphone permission denial show clear UI feedback.
### Manual Acceptance
1. Type `高亮所有北斗卫星`.
- Satellite layer opens.
- Beidou satellites are highlighted.
- Panel shows count and summary.
- Agent run is reviewable.
2. Type `显示中国大陆的算力中心`.
- Compute-center layer opens.
- Mainland China related compute centers are highlighted.
- If data cannot separate mainland China from Hong Kong/Macau/Taiwan, the UI shows a missing-data note.
3. Click microphone.
- Permission denial is clear.
- Permission grant allows recording.
- Configured ASR transcribes and executes the command.
- If ASR is not configured, text input still works.
4. Enable voice wake.
- Wake word is configurable.
- Audio is not uploaded before wake.
- After wake, one instruction audio segment is uploaded.
- Wake listening can be disabled.
5. Open Agent operations UI.
- Earth command run is listed.
- Transcription step is visible when speech was used.
- Entity query step is visible.
- Final action plan is visible.
- User can return to Earth or copy the run id.
## Documentation
When implemented, update:
```text
docs/technical/zh/agents-aiprovider.md
docs/technical/en/agents-aiprovider.md
docs/technical/zh/manual.md
docs/technical/en/manual.md
docs/technical/zh/earth-frontend-context.md
docs/technical/en/earth-frontend-context.md
```
Document:
- `aiprovider` and backend agent boundaries.
- Earth AI command entry.
- Speech/ASR configuration.
- Wake-word privacy behavior.
- Agent run/evidence/proposal review.
- V1 capability boundary: Earth visualization actions only, no business data mutation.
## Implementation Order
1. Add backend Agent data models, `init_db()` imports, and indexes.
2. Add Agent schemas and run CRUD API.
3. Add tool registry, JSON tool-call parser, and policy skeleton.
4. Add Earth entity query service.
5. Add `agents.earth.command` prompt and backend command endpoint.
6. Add frontend Earth action executor.
7. Merge AI command entry into Earth search panel.
8. Add satellite and compute-center batch highlight support.
9. Add Speech/ASR settings and transcription API.
10. Add Earth microphone recording, wake-word local setting, and fallback behavior.
11. Add Agent operations UI.
12. Add proposal apply policy and low-risk configuration application.
13. Add tests and documentation updates.
## Assumptions
- Earth LLM v1 only executes visualization actions.
- Earth commands create agent runs but do not automatically create configuration proposals.
- Configuration proposals are applied only from the Agent operations UI.
- Wake word is a device-local preference in v1 and is stored in `localStorage`.
- ASR is API-first and Whisper-compatible by default; local Whisper is a later provider.
- Multi-role simulation only reserves schemas and prompt keys in v1.
- Palantir-style situational workflows should grow from the shared evidence, entity, action, and assessment model instead of a separate isolated system.

View File

@@ -0,0 +1,52 @@
# AI Prompt Settings and Task Registry Plan
## Summary
Add an AI prompt settings tab under the operations AI settings page. Operators can select a business AI task from a dropdown, edit its prompt, save the override, and reset it back to the shipped default. Runtime LLM calls must resolve prompts through a task registry instead of embedding large prompt blocks at each call site.
Default prompts are shipped as versioned resource data, not scattered business-code literals. Business services reference stable task keys, and the runtime resolves the effective prompt from the database override first, then the shipped default resource.
## Key Changes
- Add a backend task prompt registry with stable keys, labels, groups, versions, default system prompts, and default task prompts.
- Store operator overrides in the existing `SystemSetting` table under an `ai_prompts` category. Store only custom overrides; defaults remain in the versioned prompt resource.
- Add settings APIs:
- `GET /api/v1/settings/ai-prompts`
- `PUT /api/v1/settings/ai-prompts/{task_key}`
- `POST /api/v1/settings/ai-prompts/{task_key}/reset`
- Migrate business LLM entrypoints to resolve prompts by task key. `aiprovider` remains a pure model adapter and does not inject business prompts.
- Add a “提示词” tab to `/ai`. The tab shows a grouped task dropdown, current/default prompt status, editable prompt fields, save, and reset-to-default controls.
## Initial Tasks
- `earth.news.enrich` — Earth news localization and location enrichment.
- `alerts.brief` — system alert AI brief.
- `alerts.situational.brief` — situational alert AI brief.
- `bgp.brief` — BGP AI brief.
- `location.factcheck.normalize` — location factcheck normalization.
- `location.factcheck.resolve` — location factcheck fallback resolution.
- `datasource.mapping` — datasource mapping DSL generation.
- `credential.guide` — credential guide generation.
- `ai.connection_test` — AI provider connection test.
Playground and public free-form analyze endpoints stay caller-controlled and are not shown in the prompt settings dropdown.
## Test Plan
- Backend uses `uv`:
- `uv run pytest tests/test_settings_ai_prompts.py`
- `uv run pytest tests/test_earth_news.py`
- `uv run pytest tests/test_api.py`
- Frontend uses `bun`:
- `cd frontend && bun run build`
- Manual checks:
- Prompt dropdown switches task content correctly.
- Save persists an override and runtime calls use it.
- Reset deletes the override and restores the shipped default.
- Alert prompts do not leak into news, BGP, datasource, or location tasks.
## Assumptions
- This iteration does not add prompt history, approval workflows, A/B testing, or per-user prompt variants.
- Strict JSON tasks may fail validation if an operator edits away the output contract; existing task-specific failure and retry behavior remains responsible for recovery.
- The UI stays Chinese-only in this iteration.

View File

@@ -0,0 +1,182 @@
# AI Provider OpenClaw-Style Routing Plan
Last updated: 2026-05-20
## Summary
Planet 的 AI Provider 路由要从“运行时识别特殊 provider / 特殊模型”收敛到 OpenClaw 风格的配置驱动模型:模型引用、协议、鉴权、轻量探测、真实调用和模型级例外都由 provider catalog / preset / runtime metadata 描述,运行时只解释这些元数据,不再散落 `if provider == ... and model == ...` 这类硬编码。
这份计划覆盖 Admin Next 的 Provider 配置体验、backend settings API、`aiprovider` 适配服务和未来模型目录同步方式。目标是让 OpenCode Go、MiniMax、DeepSeek、OpenAI-compatible、Anthropic-compatible、Ollama、OpenRouter / One API 类代理都能用同一套规则扩展。
## Background
当前实现已经完成了两步临时修正:
- OpenCode Go 模型目录不再使用普通 Zen free 列表,而是使用 `https://opencode.ai/zen/go/v1/models`
- `minimax-m2.7` / `minimax-m2.5` 的协议例外已从 `aiprovider` 运行逻辑移到 `model_provider_apis` 元数据中。
但整体还没有完全达到 OpenClaw 式结构。OpenClaw 的关键思想是:
- 模型引用使用 `provider/model`,由 provider 前缀确定 runtime provider。
- provider 插件或 catalog 拥有 `normalizeModelId``normalizeTransport``normalizeConfig``prepareRuntimeAuth``createStreamFn` 等 provider 行为。
- 主推理循环不认识具体模型名,只使用解析后的 provider config、transport 和 request adapter。
- 上游网关能自己路由时,尽量透传 provider routing metadata不在本地复制上游逻辑。
Planet 不需要完整复制 OpenClaw 插件系统,但需要学习它的边界划分。
## Design Principles
- Provider catalog 是路由事实来源runtime 不是。
- 模型级协议例外必须是 metadata例如 `model_provider_apis`,不能是 Python set / if 分支。
- 轻量连通性测试只验证网络、鉴权和模型目录,不发真实 prompt。
- 真实模型调用只发生在 Playground、AI brief、分析任务等明确需要生成的路径。
- 保存配置不自动设为默认,不自动触发连接测试;保存、设默认、测试三种按钮职责分离。
- 目录刷新使用增量合并语义:发现新模型,标记旧模型 stale不直接删除用户选择或自定义模型。
- 如果 provider 不提供可靠 `/models`,可以用内置 preset 确认已知模型,但 UI 必须说清楚这是 preset confirmation不是假装 provider 返回了目录。
## Target Data Model
Provider preset / runtime config 应逐步收敛为类似结构:
```json
{
"provider": "opencode-go",
"label": "OpenCode Go",
"default_transport": "openai-completions",
"base_url": "https://opencode.ai/zen/go/v1",
"auth": {
"type": "bearer",
"api_key_env": "OPENCODE_GO_API_KEY"
},
"models": [
{
"id": "glm-5.1",
"label": "GLM 5.1",
"transport": "openai-completions",
"context_window": null,
"capabilities": ["text"]
},
{
"id": "minimax-m2.7",
"label": "MiniMax M2.7",
"transport": "anthropic-messages",
"capabilities": ["text", "reasoning"]
}
],
"discovery": {
"type": "openai-models",
"url": "https://opencode.ai/zen/go/v1/models",
"auth": "provider-api-key"
}
}
```
Runtime 选择规则:
1. 解析 provider。
2. 解析 model。
3.`models[].transport` 找模型级 transport。
4. 若没有模型级 transport使用 provider `default_transport`
5. 将解析结果传给 `aiprovider`
6. `aiprovider` 只按 `transport` 组装请求,不认识 provider 专属模型名。
## Implementation Plan
### Phase 1: Stabilize Current Metadata Path
- Keep `model_provider_apis` as the immediate compatibility bridge.
- Ensure `_provider_defaults()` includes provider metadata such as `model_provider_apis`.
- Ensure `_runtime_config_from_ai_payload()` sends the resolved metadata through `AIProviderClient`.
- Ensure `AIProviderClient` forwards metadata to `aiprovider` with a structured header.
- Ensure `aiprovider.ProviderService` reads model metadata and resolves `provider_api = model_provider_apis[model] ?? provider_api`.
- Add tests proving `aiprovider` does not contain provider/model-specific literals for routing decisions.
### Phase 2: Replace `model_provider_apis` With Structured Model Catalog
- Extend `backend/app/services/llm_provider_catalog.py` preset shape with `models_metadata`.
- Preserve old `models` as a compatibility list for the UI.
- Add helpers:
- `get_provider_model_metadata(provider, model)`
- `resolve_provider_transport(provider_config, model)`
- `merge_discovered_models(existing, discovered)`
- Return both `models` and `models_metadata` from refresh endpoints.
- Admin Next should render model labels, capabilities and transport hints from metadata.
### Phase 3: Provider Discovery And Incremental Sync
- Add provider discovery descriptors:
- OpenAI-compatible `/models`
- Anthropic-compatible no-models / preset-confirmed path
- Ollama `/api/tags`
- OpenCode Go `/zen/go/v1/models`
- OpenRouter / One API passthrough model discovery
- Add incremental merge behavior:
- New discovered model: add.
- Existing discovered model: update `last_seen_at`, metadata.
- Missing discovered model: mark `stale`, do not delete.
- User custom model: keep unless explicitly removed.
- Surface discovery source in Admin Next: `实时发现 / 内置预设 / 用户自定义 / 已过期`.
### Phase 4: Transport Adapters
- Replace provider-specific request decisions with adapter descriptors:
- `openai-completions`
- `anthropic-messages`
- `ollama-generate`
- future `openai-responses`
- future `gemini-generate-content`
- Each adapter owns:
- path
- auth header format
- request body transform
- response text extraction
- reasoning/thinking block extraction
- models endpoint strategy
- `ProviderService.analyze()` should select adapter by resolved transport and call the adapter.
### Phase 5: Admin Next UX
- Model provider page should show:
- provider status tag
- default model tag
- source tag: env / runtime / preset / discovered
- model list with transport/capability hint
- separate buttons for save, set default, refresh model catalog, lightweight test
- The connect plug button remains lightweight.
- Full generation test lives only in Playground or a clearly named “试运行” action.
- If lightweight test falls back to preset confirmation, toast must say so explicitly.
## TODO
- [ ] Add `models_metadata` to provider presets and refresh responses.
- [ ] Add runtime resolver helper for provider/model transport selection.
- [ ] Remove any remaining provider/model-specific literals from `aiprovider` runtime routing.
- [ ] Add tests that `opencode-go/minimax-m2.7` resolves through metadata, not through runtime hardcode.
- [ ] Add tests for lightweight connectivity:
- [ ] 401 / 403 fail as auth error.
- [ ] 404 with known preset model passes as preset-confirmed.
- [ ] `/models` missing alias passes only when preset contains the alias.
- [ ] unknown model fails.
- [ ] Add discovery descriptors for OpenCode Go, OpenAI-compatible, Anthropic-compatible, Ollama, OpenRouter / One API.
- [ ] Add incremental model catalog merge semantics with stale marking.
- [ ] Update Admin Next model list to show model source, transport and capability.
- [ ] Keep save / set default / lightweight test / full test as separate actions.
- [ ] Document the final provider catalog schema in technical docs after implementation.
## Current Acceptance Criteria
- No runtime routing branch may depend on concrete model names like `minimax-m2.7`.
- OpenCode Go model refresh must not use the ordinary Zen free-model endpoint.
- Lightweight connect must not call `analyze()` or consume generation quota.
- Saving a provider must not automatically set it as default.
- Provider UI must distinguish configured key, fallback key, preset model and live-discovered model.
## Related Files
- `backend/app/services/llm_provider_catalog.py`
- `backend/app/api/v1/settings.py`
- `backend/app/services/ai_client.py`
- `aiprovider/main.py`
- `aiprovider/provider_service.py`
- `frontend/src/admin-next/pages/PlainResourcePages.tsx`
- `docs/plans/admin-next-parity-audit-closeout-plan.md`

View File

@@ -31,7 +31,7 @@
3. **登录与找回密码** — 登录页、忘记密码流程
4. **账户设置** — 修改密码、修改邮箱(需重新验证)、查看权限组、登出
5. **Console 总览** — 左侧菜单结构、各路由用途
6. **配置数据采集器**`/settings?tab=collector_credentials`:选择 collector、连接测试、保存凭证BarentsWatch / AISStream 两个典型例子
6. **配置数据采集器**`/collection-management?tab=collector_credentials`:选择 collector、连接测试、保存凭证BarentsWatch / AISStream 两个典型例子
7. **配置 AI 凭证**`/ai?tab=providers`:默认 provider、模型、Base URL、API Key、本地代理工具 tabWebSearch、OCR
8. **系统设置**`/settings` 其他子 tab系统设置、电视直播源、SMTP 邮件)
9. **用户管理(管理员)**`/users`创建、删除、改角色、Gatekeeper 权限组
@@ -48,7 +48,7 @@
- 打开管理员给你的 URL
- 注册账号 + 邮箱验证
- 登录后第一次做什么(建议先到 `/settings?tab=collector_credentials` 配一个 collector再到 `/ai` 配模型)
- 登录后第一次做什么(建议先到 `/collection-management?tab=collector_credentials` 配一个 collector再到 `/ai` 配模型)
- 看 Earth
部署/开发的 quickstart 内容并入 `ops-runbook.md` 的"首次部署"小节,**不**再单独出 `ops-quickstart.md`,避免新增维护点。
@@ -57,7 +57,7 @@
抽自现 manual.md重新组织
1. 首次启动 — `./planet.sh start`、默认账号(`admin/admin123``linkong/12345678`,引用 `b15d097b` 引入的 `DEFAULT_LOGIN_USERS`
1. 首次启动 — `./planet.sh start`、默认账号(`admin/admin123``linkong/LK12345678`,引用 `b15d097b` 引入的 `DEFAULT_LOGIN_USERS`
2. 启停与按模块重启 — `start/stop/restart``-b -f -a -d`
3. 健康检查 — `./planet.sh health`
4. 日志 — `./planet.sh log``-f -b -a`,日志文件路径

View File

@@ -2,7 +2,11 @@
## Status
Current implementation:
Superseded status:
This plan originally treated boundaries as collector-managed source records. The current implementation has moved country boundaries out of the datasource / collector lifecycle. Boundaries are now Earth static rendering assets managed by `Operations and Configuration -> Earth Content -> Boundary Precision` and `/api/v1/earth/boundaries/*`. The bundled low-precision GeoJSON is the default fallback, and high precision is an opt-in local PMTiles build.
Historical implementation notes below are retained only as context and must not be used as the current architecture:
- Three standard source collectors now handle real source ingestion: `earth_admin0_boundaries`, `earth_coastline`, and `earth_claim_lines`.
- Each source collector reads endpoint / headers / auth / `target_schema=earth_boundary_source` from Collector Settings, downloads the configured payload, stores the full artifact under `data/earth-boundary-sources/<collector>/<sha256>.*`, and writes a hash / feature-count / artifact-path record to `CollectedData`.
@@ -21,11 +25,11 @@ Still required before claiming true one-to-one high precision:
The Earth boundary layer should use one static PMTiles archive containing MVT tiles instead of thousands of loose GeoJSON files. The artifact is POV-specific: `earth-boundaries-china-pov-v1.pmtiles` has China POV baked in during offline source preparation, and the browser never patches political boundaries at runtime.
Production must serve a single PMTiles artifact through static hosting and HTTP range requests. Missing PMTiles is a hard boundary-layer error, not a silent low-precision fallback.
Production should serve a single PMTiles artifact through static hosting and HTTP range requests. In development or on machines that have not opted into high precision, missing PMTiles falls back to the bundled low-precision GeoJSON so the Earth base remains usable.
## Key Implementation Rules
- Source inputs must be auditable. OSM admin boundaries, coastline packages, and claim-line endpoints are configured through Collector Settings using the `earth_boundary_source` target schema; `config/earth-boundary-sources.example.json` remains the versioned example template.
- Source inputs must be auditable. OSM admin boundaries, coastline packages, and claim-line endpoints are configured through Earth Content boundary precision settings; `config/earth-boundary-sources.example.json` remains the versioned example template.
- China POV geometry is applied before tiling:
- Zangnan and Aksai Chin are unioned into China and subtracted from India.
- Taiwan/Penghu, Diaoyu Dao and affiliated islands, Chiwei Yu, and South China Sea islands are China hover/country features.
@@ -36,9 +40,9 @@ Production must serve a single PMTiles artifact through static hosting and HTTP
- `boundary_disputed_internal`
- `coastline`
- `claim_line`
- The frontend provider is selected from the boundary manifest:
- The frontend provider is selected from local high-precision preference plus the boundary manifest:
- `tileProvider: "pmtiles-mvt"` reads the PMTiles artifact.
- Any other provider, missing manifest, or missing PMTiles artifact is treated as an error.
- Missing high-precision preference, missing manifest, or missing PMTiles artifact falls back to low-precision GeoJSON.
- Redis is not part of v1. Static PMTiles plus browser/CDN range caching is the default performance model.
## Cleanup And Documentation
@@ -46,11 +50,11 @@ Production must serve a single PMTiles artifact through static hosting and HTTP
- Do not commit generated loose tiles under `frontend/public/earth/data/boundaries/` or source downloads under `data/earth-boundary-sources/`.
- Remove stale generated debug data before production builds; regenerate it only when smoke testing the debug path.
- Keep the high-level plan, backend collector docs, layer style docs, and ops runbook aligned whenever the provider contract changes.
- After implementation changes, provide user-facing operation steps covering source configuration, source collection, artifact build/deploy, page verification, and fallback troubleshooting.
- After implementation changes, provide user-facing operation steps covering source configuration, artifact build/deploy, page verification, and fallback troubleshooting.
## Verification
- The PMTiles builder fails as not ready when any of `earth_admin0_boundaries`, `earth_coastline`, or `earth_claim_lines` has not produced a current artifact record.
- The Earth boundary build API reports missing source configuration or missing tools clearly, without creating datasource collection records.
- The PMTiles builder fails as not ready when source artifacts exist but `tippecanoe` / `pmtiles` are missing.
- Running the PMTiles builder twice returns `unchanged` on the second run when inputs are stable.
- `git add . --dry-run` does not include generated loose boundary tiles or source downloads.
@@ -58,10 +62,10 @@ Production must serve a single PMTiles artifact through static hosting and HTTP
- Manual Earth checks confirm:
- PMTiles range requests are issued only for visible tiles.
- Boundary toggle, hover tooltip, and country highlight still work.
- PMTiles failure reports a boundary-layer error; the frontend must not draw low-precision legacy boundaries.
- PMTiles failure reports a high-precision boundary error; machines without high-precision enabled continue drawing low-precision fallback boundaries.
## Assumptions
- "One-to-one" means source-faithful to the selected audited vector source, not hand-tuned to a screenshot.
- The China POV artifact is static and versioned; no runtime region-based POV switching is planned.
- The removed repository seed file cannot be used as a runtime fallback for country boundaries.
- The repository low-precision seed file is retained as the runtime fallback for country boundaries.

View File

@@ -0,0 +1,71 @@
# Earth 图层 Redis 缓存与 OOM 防护完整计划
## Summary
目标不是单纯“加缓存”,而是把 Earth 图层读路径改成可控、可观测、可降级的缓存架构,避免演示前高并发、重图层、船只数据膨胀再次把后端打到 OOM 无限重启。
前端继续请求原 APIresponse body 保持兼容。后端新增 Redis 读穿缓存、防击穿锁、stale 兜底、payload budget、主动失效、观测 header 和日志。
更新架构图:
```mermaid
flowchart TD
Earth["Earth 前端<br/>原 API 不变"] --> API["FastAPI Visualization / Layers APIs"]
API --> Guard["Request Guard<br/>limit clamp / bbox required / payload budget"]
Guard --> Cache["Layer Cache Adapter<br/>key / TTL / lock / stale"]
Cache -->|fresh hit| Redis["Redis<br/>earth:layer:v1:*<br/>fresh + stale payloads"]
Cache -->|miss or refresh| Builder["Layer Builders<br/>DB query + GeoJSON conversion"]
Builder --> DB["PostgreSQL / Timescale<br/>authoritative data"]
Builder --> Budget["Response Budget Check<br/>feature cap / byte cap / diagnostics"]
Budget --> Cache
Cache --> API
API --> Earth
Collectors["Collectors / Data writes"] --> DB
Collectors --> Invalidate["Source-scoped invalidation"]
Invalidate --> Redis
Cache --> Metrics["Structured logs / headers<br/>hit miss stale bypass refresh<br/>bytes features duration"]
```
## Implementation Changes
- Add an Earth layer cache adapter that owns Redis keys, TTLs, stale fallback, single-flight locks, JSON serialization, response headers, and graceful Redis bypass.
- Use `earth:layer:v1:{layer}:{params}` for fresh cache, `earth:layer:v1:{layer}:{params}:stale` for stale fallback, and `earth:layer:lock:v1:{hash}` for rebuild locks.
- Cache policy:
- `cables`, `landing-points`: fresh `6h`, stale `24h`
- `satellites`: fresh `15m`, stale `2h`
- `compute-centers`: fresh `10m`, stale `1h`
- `bgp-collectors`, `bgp-anomalies`, `bgp-incidents`, `geo/summary`: fresh `30-60s`, stale `10m`
- `vessels snapshot`: fresh `5s`, stale `30s`, with bbox rounded to `0.1` degrees and key including `zoom/type/limit/since_minutes`
- Prevent cache stampedes with `SET NX EX` locks. The lock holder refreshes; other requests prefer stale, wait briefly, then fall back to the guarded DB path.
- Enforce payload budgets on every cached layer: maximum features, maximum serialized bytes, and diagnostics when truncation happens.
- Keep vessel snapshot viewport-first: require bbox, clamp low-zoom limits, never build an unbounded all-vessel GeoJSON for Earth startup.
- Add cache observability headers: `X-Planet-Cache`, `X-Planet-Cache-Features`, `X-Planet-Cache-Bytes`, and development-only `X-Planet-Cache-Key`.
- Add super-admin system endpoints for Earth layer cache status and clearing.
## Public Interfaces
- Frontend request URLs stay unchanged.
- Response bodies stay compatible.
- New optional response headers report cache state.
- New system endpoints:
- `GET /api/v1/system/cache/earth-layers`
- `DELETE /api/v1/system/cache/earth-layers`
- Redis key contract: `earth:layer:v1:*`. Existing news keys remain `earth_news:target_location:*`.
## Test Plan
- Unit-test key generation, bbox rounding, payload budget truncation, Redis miss/hit, stale fallback, Redis bypass, and single-flight lock behavior.
- API-test repeated requests for cache headers, super-admin cache status/clear endpoints, and vessel snapshot bbox/limit safeguards.
- Regression-test existing layer guard behavior and vessel type forwarding.
- Verify Redis outage does not break Earth API responses.
- Verify large vessel requests return bounded payload diagnostics instead of exhausting memory.
## Assumptions
- PostgreSQL remains the authoritative data source; Redis is disposable read-through cache.
- First phase does not change frontend rendering. If Three.js rendering becomes the bottleneck, that is a separate frontend performance task.
- When safety conflicts with completeness, vessel responses prefer bounded/truncated data plus diagnostics over risking backend OOM.
- GeoJSON schema changes should bump the Redis key version from `v1` to `v2`.

View File

@@ -0,0 +1,349 @@
# 统一集成配置 Schema 系统计划
Last updated: 2026-05-20
## Summary
Planet 的采集器、AI Provider 和工具调用配置需要从“页面各自硬编码字段”收敛到同一套低代码 schema 系统。系统负责两件事:
- 用后台可编辑 schema 生成配置表单。
- 按字段 target 把表单值组装成后端运行时需要的请求、凭证和 JSON 配置。
这套 schema 不替代 `target_schema_registry``target_schema_registry` 继续负责采集结果映射和校验;本计划中的 `integration_config_schemas` 负责“怎么配置一个集成”。
## Current Problems
- Admin Next 的采集器配置曾把不同凭证形态压成通用 `api_key`,导致 `barentswatch_vessels` 这种 OAuth client credentials 丢失 `client_id` 字段。
- AI Provider、Web Search、OCR 和 DataSource 配置各自维护表单字段、secret 处理和 payload 组装逻辑,重复且容易漂移。
- 新增字段时需要改前端字段列表和保存逻辑,无法做到后台配置化扩展。
## Target Model
新增统一 registry`integration_config_schemas`,存储在 `SystemSetting.payload`
Registry 包含:
- `fragments`:可复用字段片段,例如 endpoint、API Key、OAuth Client、HTTP 请求、WebSocket 订阅、AI Provider 基础字段、工具超时字段。
- `auth_schemas`:可复用认证编排,例如 API Key、Bearer Token、Basic、OAuth2 Client Credentials、OAuth2 Authorization Code、Session Cookie Login。
- `schemas`:具体配置对象使用的 schema例如 `datasource:barentswatch_vessels``ai_provider:minimax``tool:web_search:tavily`
- `defaults`:每类集成的默认 schema例如 `datasource``ai_provider``tool`
Schema 必须声明 `kind`
- `datasource`
- `ai_provider`
- `tool`
字段定义统一使用:
```json
{
"key": "client_secret",
"label": "Client Secret",
"type": "secret",
"secret": true,
"target": "auth_config.client_secret",
"required": true,
"wide": true
}
```
字段 target 支持写入:
- DataSource`root.*``auth_config.*``headers.*``config.*`
- AI Provider`ai_provider.*``ai_provider.providers.{provider}.*`
- Tool`web_search.*``web_search.providers.{provider}.*``ocr.*`
## Auth Schema Model
认证必须成为 schema 系统的一等能力,不能再把所有凭证强行压成 `api_key`。每个配置 schema 可以引用一个 `auth_schema`,也可以内联声明认证编排。
Auth schema 描述:
- 凭证字段:哪些字段是 secret、是否必填、写入哪个 target。
- 凭证来源DB、env fallback、运行时草稿、用户授权回调。
- 预认证请求例如登录接口、token endpoint、OAuth callback。
- 凭证注入方式header、query、form、JSON body、cookie jar、WebSocket subscription payload。
- reveal 策略:管理员可 reveal 并写 audit log无 DB/env 值时显示空。
- 测试策略:连接测试必须使用当前表单草稿优先,再 fallback 到已保存/env。
v1 需要支持的认证类型:
- `none`:无认证。
- `api_key`API Key 写入 header/query/form/body。
- `bearer_token`Bearer token header。
- `basic`username/password支持直接 Basic header 或 provider 特定登录。
- `oauth2_client_credentials`client_id/client_secret 换 access_token。
- `oauth2_authorization_code`:第三方登录授权,包含 authorize URL、callback、token exchange、refresh。
- `session_cookie_login`:用户名密码登录后保存 cookie jar再访问数据接口。
- `custom_auth_preflight`:无法归类时,用声明式 preflight 请求生成后续请求上下文。
Auth schema 示例:
```json
{
"key": "spacetrack_session",
"type": "session_cookie_login",
"fields": [
{
"key": "username",
"label": "Username",
"target": "auth_config.username",
"required": true
},
{
"key": "password",
"label": "Password",
"type": "secret",
"secret": true,
"target": "auth_config.password",
"required": true
}
],
"preflight": {
"method": "POST",
"url": "https://www.space-track.org/ajaxauth/login",
"body_type": "form",
"body": {
"identity": "{{auth_config.username}}",
"password": "{{auth_config.password}}"
},
"success": {
"type": "cookie"
}
},
"inject": {
"type": "cookie_jar"
}
}
```
OAuth Authorization Code 示例:
```json
{
"key": "github_oauth",
"type": "oauth2_authorization_code",
"fields": [
{ "key": "client_id", "target": "auth_config.client_id", "required": true },
{ "key": "client_secret", "type": "secret", "secret": true, "target": "auth_config.client_secret", "required": true },
{ "key": "scopes", "type": "tags", "target": "auth_config.scopes" }
],
"authorization": {
"url": "https://github.com/login/oauth/authorize",
"client_id": "{{auth_config.client_id}}",
"scopes": "{{auth_config.scopes}}",
"redirect_uri": "{{system.callback_base_url}}/api/v1/integrations/oauth/github/callback"
},
"token": {
"method": "POST",
"url": "https://github.com/login/oauth/access_token",
"body_type": "form",
"body": {
"client_id": "{{auth_config.client_id}}",
"client_secret": "{{auth_config.client_secret}}",
"code": "{{oauth.code}}",
"redirect_uri": "{{oauth.redirect_uri}}"
}
},
"inject": {
"type": "bearer_header",
"token_path": "access_token"
}
}
```
## Request And Runtime Assembly
这套系统本质是面向 Planet 集成的低代码 Postman但目标不是临时发请求而是沉淀成可保存、可测试、可调度、可审计的运行配置。
Schema 支持描述:
- HTTP method`GET``POST`
- endpoint
- headers
- query params
- JSON body / form body
- auth schema / auth config / preflight auth flow
- WebSocket endpoint 和 subscription payload
- AI Provider 的 `provider_api``base_url``model``api_key``service_token`
- Tool 的 provider、base_url、api_key、timeout 和工具专属参数
请求执行顺序:
1. 从 schema 字段 target 组装 root/auth_config/headers/config。
2. 如果存在 auth schema先解析凭证来源和草稿覆盖。
3. 需要 preflight 时执行认证请求,例如 token exchange 或 login。
4. 把认证结果注入正式请求,例如 bearer header、cookie jar、query token。
5. 执行连接测试、采样、采集、AI Provider connect 或 tool connect。
## Default Schemas
### DataSource
默认采集器:
- endpoint
- method
- headers JSON
- query/body/config JSON
- API Key auth
- advanced JSON
`barentswatch_vessels`
- endpoint
- Client ID -> `auth_config.client_id`
- Client Secret -> `auth_config.client_secret`
- fixed `auth_type = oauth_client`
`aisstream_vessels`
- WebSocket endpoint
- API Key -> `auth_config.api_key`
- subscription / bounding boxes config
`spacetrack_tle`
- API Base URL / endpoint
- Username -> `auth_config.username`
- Password -> `auth_config.password`
- auth schema = `session_cookie_login`
- login endpoint = `https://www.space-track.org/ajaxauth/login`
- login body fields`identity` / `password`
- run/test request uses returned session cookie
### AI Provider
默认字段:
- provider
- provider_api
- base_url
- model
- api_key
- max_tokens
- anthropic_version
- service_url
- service_token
- timeout_seconds
- retry_attempts
- model_provider_apis
Provider presets supply initial defaults, but the editable schema controls which fields appear and where values are saved.
AI Provider auth variants
- OpenAI-compatible providers`api_key` or `bearer_token`
- Local/sidecar services`none``service_token` or custom header。
- OAuth-backed providers`oauth2_authorization_code`,适用于需要用户授权登录的 provider。
- CLI/session-backed tools such as Codex优先作为 `tool` 或本机 runner 集成;如果作为 provider必须显式声明会话来源、权限边界、不可多用户复用的限制。
### Tool
`web_search`
- enabled
- provider / default_provider
- base_url
- api_key
- max_results
- timeout_seconds
- endpoint_path
- search_depth
- engine
- include_answer / include_raw_content / include_text
- search_path / scrape_path / scrape_formats
`ocr`
- enabled
- provider
- base_url
- api_key
- model
- languages
- timeout_seconds
- max_file_size_mb
- output_format
Tool auth variants
- GitHub PAT`bearer_token`
- GitHub OAuth App`oauth2_authorization_code`,适合用户授权登录和代表用户访问。
- GitHub App`app_installation`,需要 app id/private key/installation id并通过 schema 声明 installation token exchange。
- Browser/session tools必须显式标记为 `session_local_only`,不能作为后台多用户稳定凭证。
## API Plan
- `GET /api/v1/integration-config-schemas`
- Return the full registry.
- `PUT /api/v1/integration-config-schemas`
- Save the registry. Admin only.
- `POST /api/v1/integration-config-schemas/validate`
- Validate full registry or one schema.
- `POST /api/v1/integration-config-schemas/auth/test`
- Test auth schema with draft credentials without saving.
- `GET /api/v1/integration-config-schemas/auth/secrets`
- Reveal stored/env-backed secret fields for admins; write audit log.
- `POST /api/v1/integrations/oauth/{provider}/start`
- Start OAuth Authorization Code flow.
- `GET /api/v1/integrations/oauth/{provider}/callback`
- Complete OAuth callback and store token material according to schema.
- `GET /api/v1/datasources/configs/all`
- Add `form_schema` to each row.
- `GET /api/v1/settings/integrations`
- Add `form_schema` for AI Provider and tools.
Existing save APIs remain compatible:
- DataSource saves to `DataSourceConfig`.
- AI Provider, Web Search and OCR save to `external_integrations`.
Secret fields never return plaintext through list/config endpoints. They return configured state and masked preview only.
Secret reveal endpoints return plaintext only on explicit administrator action and must log target, actor, source, result and timestamp. List/config endpoints must never leak secret plaintext.
## Frontend Plan
Admin Next extracts a reusable `SchemaForm`:
- Render fields from `form_schema.fields`.
- Build payload by writing values to each field `target`.
- Preserve masked secret semantics: unchanged masked values do not overwrite stored secrets.
- Reveal secrets through a schema-aware reveal action; if neither DB nor env has a value, show an empty editable input.
- Support text, secret, number, boolean, select, textarea, JSON and tags controls.
- Support auth controls for API key, username/password, OAuth connect/disconnect, session cookie login and custom preflight status.
- Validate schema before saving registry changes.
Pages migrated in v1:
- Collection Management / collector configs.
- AI / Provider configuration.
- AI / Tools configuration for Web Search and OCR.
Each detail page gets a schema editing action for admins. The editor saves registry JSON after validation.
## Test Plan
- Default registry initializes with datasource, AI Provider and tool schemas.
- Validation rejects duplicate keys, illegal targets, illegal field types and secret plaintext defaults.
- `barentswatch_vessels` renders and saves `client_id` / `client_secret`.
- `spacetrack_tle` renders username/password, tests with draft credentials, and does not require API Key.
- `aisstream_vessels` renders API Key and WebSocket subscription fields.
- Session cookie login auth executes preflight before sample/run and uses the resulting cookie jar.
- OAuth Authorization Code schema can start callback flow, store token metadata and inject bearer token.
- Secret reveal returns DB value, env fallback or empty value according to source, and writes audit log.
- Connection tests always prefer current draft credentials over saved/env credentials.
- AI Provider renders and saves `provider_api`, `base_url`, `model`, `api_key` and `service_token`.
- AI Provider and tool schemas can reuse the same auth schema primitives as DataSource.
- Web Search and OCR render and save provider-specific tool fields.
- Adding a schema field in the registry makes it appear in Admin Next without frontend code changes.
- Existing connection tests, datasource sampling, datasource run, AI Provider connect/reveal/refresh, and Web Search connect keep working.
## Assumptions
- `target_schema_registry` remains separate because it describes collected result shape, not configuration forms.
- The first implementation stores schema registry in `SystemSetting`; no new database table is required.
- Old AntD Settings pages stay compatible but are not migrated in v1.
- Tool scope in v1 is Web Search and OCR.

View File

@@ -30,8 +30,10 @@ What belongs here:
- [Collector Settings and Connectivity Validation](/home/ray/dev/linkong/planet/docs/technical/en/datasource-collector-settings-connectivity.md): Data source catalog, collector settings, connectivity validation, and BarentsWatch credentials
- [Shared Location Resolution Pipeline Development Guide](/home/ray/dev/linkong/planet/docs/technical/en/location-pipeline-development.md): Backend location resolver / pipeline interfaces, registries, and extension points
- [Docs Gatekeeper Development Guide](/home/ray/dev/linkong/planet/docs/technical/en/docs-gatekeeper-development.md): Backend Docs catalog, Markdown content loading, and Gatekeeper permission groups
- [Naming Glossary](/home/ray/dev/linkong/planet/docs/technical/en/naming-glossary.md): English/Chinese term mapping for the console, Earth, backend, and docs
- [Earth Interactable Usage](/home/ray/dev/linkong/planet/docs/technical/en/earth-interactable-usage.md): API, lifecycle, and integration examples for Earth surface icon Interactable
- [Earth Toolbar and Overlay Coordination](/home/ray/dev/linkong/planet/docs/technical/en/earth-toolbar-overlay-coordination.md): Closing matrix and integration rules for toolbar buttons, search, settings, news, and layer overlays
- [Tactile UI Components](/home/ray/dev/linkong/planet/docs/technical/en/tactile-ui-components.md): Portable button, switch, tooltip, and scrollbar APIs, theme tokens, and migration rules
What does not belong here:

View File

@@ -95,9 +95,37 @@ The AI settings page uses:
- `POST /api/v1/settings/integrations/ai-provider/connect`
- `GET /api/v1/settings/integrations/ai-provider/secrets`
- `GET /api/v1/settings/integrations/ai-provider/presets`
- `GET /api/v1/settings/ai-prompts`
- `PUT /api/v1/settings/ai-prompts/{task_key}`
- `POST /api/v1/settings/ai-prompts/{task_key}/reset`
These endpoints require an authenticated user. The `secrets` endpoint is only used when the settings page reveals a key or token; hiding the field restores the masked preview.
Admin Next keeps the AI page aligned with the legacy information architecture:
- `Model Providers`
- Manages provider, wire adapter, default model, LLM API key, proxy URL, proxy token, model refresh, set-as-default, and lightweight connectivity testing.
- `Tool Calling`
- Manages tools such as WebSearch and OCR. Each tool first selects a provider, then edits that provider's API, key, and advanced parameters.
- `Prompts`
- Edits system/user prompts by prompt group and task key. Save and reset only affect the current task.
- `Playground`
- Runs real conversations with the active provider and prompt configuration. AI responses are rendered as Markdown.
Save, set-as-default, and connectivity testing are separate responsibilities: save only persists the form, set-as-default only changes the active provider/tool, and connectivity testing only validates the current draft. It must not implicitly save or switch defaults.
The `ai-prompts` endpoints back the Prompts tab in AI settings. Shipped defaults come from versioned backend resources, while business code references stable task keys. The API stores only operator overrides. Resetting a prompt removes the override and falls back to the current shipped default.
### Prompt Boundary
`aiprovider` is a pure model adapter and does not inject a global business system prompt. News localization, alert briefing, BGP briefing, location factcheck, datasource mapping, and credential guide generation each resolve their own effective prompt by task key. Alert-analysis system prompts are only sent by alert-related tasks and do not leak into other LLM calls.
### Agent And Tool Boundary
Agent workflows belong in the `backend`, not in `aiprovider`. Future Earth LLM commands, situational awareness, multi-role simulation, WebSearch, database queries, evidence storage, and configuration proposal application should be orchestrated by the backend Agent Runtime. `aiprovider` should receive model-ready requests from the backend and return normalized model responses.
If a provider supports native tool calling, `aiprovider` may pass through protocol fields and normalize response blocks, but tool whitelists, argument validation, permission policy, run records, and write approvals must stay in the backend. When a provider does not support native tools, the backend uses JSON tool-call fallback; business tools should not move into `aiprovider` for a provider-specific shortcut.
### AI provider internal API
Internal-only endpoints:
@@ -244,6 +272,34 @@ Each provider has its own key slot. Resolution order is:
`.env` is only a fallback. After the settings page saves successfully, or after the connection test succeeds, PostgreSQL becomes the global default source.
Admin Next must compute key status per provider or tool:
- If the database has a key for the current provider/tool, show `configured`.
- If the database has no key but the fallback provider, model, or tool matches the current item, show the fallback masked preview.
- If neither database nor matching fallback exists, show `not configured`; a generic `.env` key for another provider must not make this item appear configured.
- Masking keeps the prefix before the first `-`, for example `sk-********`. Plaintext reveal is only available inside the authorized configuration page.
Tool keys follow the same rule. WebSearch and OCR must match the current tool and provider before they can use fallback credentials.
### Lightweight Connectivity Testing
The Admin Next plug button performs a lightweight connectivity check and does not save configuration. Common API-platform practice is two-tiered:
- Check a provider catalog or low-cost endpoint to validate base URL, authentication, and model reachability.
- Send full model requests only when the user explicitly runs Playground or a business task.
Connectivity results should be explicit:
- `ok`: authentication, route, and model catalog are usable.
- `warning`: service is reachable, but the current model is missing from the catalog or capability metadata is incomplete.
- `error`: authentication, network, protocol, or model lookup failed.
Toast titles must match the result; failures must not be titled as a successful connection.
### OpenCode Go Routing Model
Subscription channels such as OpenCode Go should not be handled by hard-coded frontend model sets. Prefer provider catalog or backend capability discovery that records per-model capabilities such as `chat_completions`, `anthropic_messages`, `models_endpoint`, and whether a subscription key is required. The frontend should display capabilities; the backend should map provider, base URL, model, and adapter into the real request.
#### Settings Page Behavior
- The Provider select controls the global default provider.
@@ -287,7 +343,6 @@ SERVICE_VERSION=0.1.0
AI_PROVIDER_SERVICE_TOKEN=change_me
AI_TIMEOUT_SECONDS=60
AI_HTTP_RETRY_ATTEMPTS=2
AI_ANALYSIS_SYSTEM_PROMPT=你是态势感知分析助手。请基于输入的上下文、观测与约束,输出结构化、克制、可执行的分析。
```
Optional provider-specific keys:

View File

@@ -86,19 +86,23 @@ async def run(self, db):
| TeleGeography | submarine_cable | Submarine cable information | 7 days |
| BarentsWatch AIS | vessel | AIS vessel positions, speed, heading, MMSI, and related fields | Collector settings |
| AISStream Vessels | vessel_ais | AIS WebSocket realtime stream, written to the raw observation layer and displayed through aggregation | Collector settings |
| Earth Admin-0 Boundaries | earth_admin0_boundaries | Downloads the configured country-boundary source, saves an artifact, and writes an `earth_boundary_source` manifest record | Collector settings |
| Earth Coastline | earth_coastline | Downloads the configured coastline source, saves an artifact, and writes an `earth_boundary_source` manifest record | Collector settings |
| Earth Claim Lines | earth_claim_lines | Downloads the configured claim-line source, saves an artifact, and writes an `earth_boundary_source` manifest record | Collector settings |
| Earth PMTiles Builder | earth_boundary_tiles | Reads the three Earth boundary source records and builds / registers the PMTiles artifact | Collector settings |
AIS vessel collectors use a different persistence path from regular `CollectedData` collectors. BarentsWatch, AISStream, and custom `vessel_ais` sources write into the AIS raw observation layer first, then the aggregation service merges those observations into the GeoJSON and detail payloads used by the Earth vessel layer. This preserves source, transport, field conflicts, and observation time instead of letting one realtime source overwrite the final display table.
Earth boundaries are now split into three real source collectors plus one downstream builder. `earth_admin0_boundaries`, `earth_coastline`, and `earth_claim_lines` read endpoint, headers, auth, and `config.target_schema=earth_boundary_source` from Collector Settings. Triggering them requests the configured endpoint, writes the full response to `data/earth-boundary-sources/<collector>/<sha256>.*`, and stores sha256, feature count, license, artifact path, sample properties, and mapping metadata in `CollectedData`.
`earth_boundary_tiles` no longer means source-data collection. It reads the latest successful records from those three source collectors; if any source is missing, the task fails as "not ready" and does not register "4 high-precision tile" records. Once all sources exist, it uses `tippecanoe` / `pmtiles` to build `frontend/public/earth/data/boundaries/earth-boundaries-china-pov-v1.pmtiles`; if those tools are missing, the task fails with the missing-tool message. There is no legacy low-precision fallback for country boundaries.
Earth boundaries are no longer data collectors. They are Earth static rendering assets: the Earth Assets settings panel owns source configuration, and `/api/v1/earth/boundaries/*` builds `frontend/public/earth/data/boundaries/earth-boundaries-china-pov-v1.pmtiles`. When no high-precision PMTiles artifact is available locally, the frontend uses the bundled low-precision GeoJSON fallback and does not write boundary records to `CollectedData`.
TOP500 and Epoch AI compute sources do not always provide usable coordinates. The unified Earth compute-center endpoint uses only valid source-provided coordinates or `compute_center_locations` dimension-table coordinates during the main map startup path; records without coordinates are returned as `unresolved` instead of being rendered from a local registry, country centroid, or guessed city. When users manually collect candidates, the backend queries ROR and Nominatim/OpenStreetMap from source fields; accepted candidates are saved into `compute_center_locations` and rendered from that table on the next layer refresh.
Admin Next collection management follows the business hierarchy instead of flattening every endpoint into one table:
- `Collectors`: endpoint, authentication, headers, base parameters, enabled state, and credential guides.
- `Collection Schedule`: scheduler state and task controls.
- `Collection History / Snapshots`: history grouped by collector, with a detail-side snapshot selector for versions.
Snapshot lists should not show every snapshot of the same collector as separate top-level records. The top-level list selects a collector; the detail area switches between time versions.
Credential guides are maintained by `backend/app/services/credential_guides.py`. The console uses read / generate / reset actions to load or create Markdown instructions. The frontend should render the guide Markdown for operators, not expose generation prompts or raw metadata.
## IV. Data Format (stored in CollectedData table)
```python
@@ -306,7 +310,7 @@ State semantics:
- `reconnecting`: upstream or network interruption; the collector records `AISSourceHealth` and waits before reconnecting.
- `stopped` / `cancelled`: stopped by a test limit or user action.
AISStream connectivity validation reads the saved collector configuration, environment variables, and `AISSTREAM_API_KEY` in `~/.zshrc` through `datasource_connectivity.py`. For actual collection, the most reliable path is saving the API key in `Settings -> Collector Settings -> AISStream Vessels`; if the key only lives in `~/.zshrc`, confirm that the backend process inherited it.
AISStream connectivity validation reads the saved collector configuration, environment variables, and `AISSTREAM_API_KEY` in `~/.zshrc` through `datasource_connectivity.py`. For actual collection, the most reliable path is saving the API key in `Collection Management -> Collectors -> AISStream Vessels`; if the key only lives in `~/.zshrc`, confirm that the backend process inherited it.
The console manages AISStream from `/datasources -> Realtime Streams`, not from the normal finite collection progress bar. The realtime stream API aggregates runtime state, health, configuration preview, and raw observation counters:
@@ -374,9 +378,9 @@ GET /api/v1/layers/bgp/collectors?bbox=lon_min,lat_min,lon_max,lat_max&zoom=12&l
`/api/v1/data-products/*` is for aggregate panels and keeps a global statistics scope independent of the map bbox. `/api/v1/layers/*` is for map rendering, requires `bbox` and `zoom`, defaults to `limit=1000`, and caps `limit` at `5000`; low zoom falls back to a smaller response cap and reports `degraded`, `truncated`, `limit_clamped`, and `stats_scope=viewport` in `diagnostics`. Non-vessel layers currently reuse the existing GeoJSON converters before the guard layer; future product-specific queries can push bbox filtering deeper.
## X. Collector Settings And Connectivity Validation
## X. Collectors And Connectivity Validation
The console "Collector Settings" page owns endpoint, headers, timeouts, retries, and credentials for all built-in collectors. Connectivity is derived by the backend checksum rather than by frontend button styling:
The console "Collectors" page owns endpoint, headers, timeouts, retries, and credentials for all built-in collectors. Connectivity is derived by the backend checksum rather than by frontend button styling:
- endpoint
- auth type
@@ -397,7 +401,7 @@ POST /api/v1/settings/credential-guides/{provider}/generate
POST /api/v1/settings/credential-guides/{provider}/reset
```
See [Collector Settings and Connectivity Validation](/home/ray/dev/linkong/planet/docs/technical/en/datasource-collector-settings-connectivity.md) for the full flow.
See [Collectors and Connectivity Validation](/home/ray/dev/linkong/planet/docs/technical/en/datasource-collector-settings-connectivity.md) for the full flow.
## XI. Data Usage

View File

@@ -8,12 +8,12 @@ The console now separates the "data source catalog" from "collector configuratio
- Lists all data sources, including built-in and custom sources.
- Clicking a name only opens an information drawer.
- Focuses on status, manual collection, and running collection tasks.
- `/settings?tab=collector_credentials`
- Displays as "Collector Settings".
- `/collection-management?tab=collector_credentials`
- Displays as "Collectors".
- Owns endpoint, headers, base parameters, and credentials.
- Every collector exposes a connection button for health checks.
This reduces first-use confusion: API endpoints, headers, credentials, and custom source configuration all belong to collector settings instead of being scattered across the data source list and system settings.
This reduces first-use confusion: API endpoints, headers, credentials, and custom source configuration all belong to Collectors instead of being scattered across the data source list and system settings.
## User-Facing Rules
@@ -32,8 +32,8 @@ If endpoint, headers, base configuration, or credential fingerprint changes afte
Files:
- [DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/DataSources/DataSources.tsx)
- [index.css](/home/ray/dev/linkong/planet/frontend/src/index.css)
- [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/pages/PlainResourcePages.tsx)
- [AdminNextRoutes.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/AdminNextRoutes.tsx)
Current behavior:
@@ -53,30 +53,35 @@ Current behavior:
`data-source-bulk-toolbar__running-pill` is the styling entry point for the "Collecting" pill. It is aligned with other status tags, while hover treatment, arrow affordance, and blue outline indicate interactivity.
### Collector Settings
### Collection Management
File:
- [Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Settings/Settings.tsx)
- [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/pages/PlainResourcePages.tsx)
Current behavior:
- The `collector_credentials` tab is displayed as "Collector Settings".
- A select lists built-in collectors and supports maintaining custom supplemental sources that merge into built-in data.
- The only button beside the select is a plug icon for health checks.
- Status tags below the select show:
- `Credentials required` / `No credentials required`
- Module
- `Enabled` / `Disabled`
- `Unchecked` / `Available` / `Unavailable`
- Whether the endpoint is overridden
- Collectors that require credentials place the credential card above base configuration.
- Collectors without credentials only show base configuration.
- `/collection-management` follows the legacy hierarchy: `Collectors`, `Collection Schedule`, and `Collection History / Snapshots`.
- `Collectors` is the configuration page. The left list shows collector configs; the right form edits endpoint, authentication, headers, collection parameters, and enabled state.
- New collectors and target schemas use draft detail pages instead of transparent JSON modals; save persists the draft, while cancel destroys it.
- The connection button only tests connectivity and does not save. The save button only persists the form.
- Credential guides open in a draggable Markdown modal. When no guide exists, the modal still opens and offers a generate action. Generation shows a centered waiting state; reset confirmation must render above the guide modal.
- `Collection History / Snapshots` groups by collector. The list must not repeat every snapshot for the same datasource; detail view switches versions through a Time Capsule / Time Machine style selector.
- The AISStream collector uses WebSocket semantics: connecting, streaming, reconnecting, or stopped. It does not use a fixed completion percentage.
- Custom source editing lives in collector settings. The data source catalog keeps overview, run controls, and read-only drawers.
The connection button uses an inline Tabler-style plug icon with `plug-connected` semantics, avoiding the older refresh icon for a connection action.
### Mapping Templates and Target Schemas
Mapping remains part of collection management, but not the main collector tab:
- `Mapping Templates` owns sample payload, AI propose, preview, create/update, and activate.
- `Target Schemas` maintains writable target structures.
- `run-mapped`, `stop-mapped`, and `stream-status` run mapped custom collectors.
The new UI should keep these paths form-first. Only advanced fields should collapse into JSON. Do not flatten templates, schemas, runtime status, and collector configuration into one table.
## Backend APIs
### Data Source Configuration List
@@ -140,6 +145,26 @@ Successful responses include:
- `credential_provider`
- `credential_source`
### Credential Guides
```http
GET /api/v1/datasources/credential-guides/{provider}
POST /api/v1/datasources/credential-guides/{provider}/generate
POST /api/v1/datasources/credential-guides/{provider}/reset
```
Purpose:
- Read the Markdown credential guide for a provider.
- Generate a guide from collector metadata when none exists.
- Reset back to the backend default guide.
Frontend rules:
- Render Markdown, not backend prompts or metadata.
- Generate/reset actions belong inside the guide modal, not in the collector configuration toolbar.
- Missing guides still open a modal so the user can generate one from there.
### BarentsWatch AIS Connectivity Validation
```http
@@ -307,7 +332,7 @@ Files:
Custom sources are supplemental inputs for existing target schemas, not isolated data islands. The most complete target today is `vessel_ais`: a custom REST or WebSocket source is mapped deterministically, written into AIS raw observations, and then pushed to Earth through the `vessels` WebSocket channel.
Earth high-precision boundaries use the same target-schema mechanism. `earth_boundary_source` receives mapped records for `earth_admin0_boundaries`, `earth_coastline`, and `earth_claim_lines`; full GeoJSON / JSON payloads are stored as artifacts, while the database only keeps source kind, sha256, feature count, license, artifact path, and sample properties so large geometries do not land in a single row.
Earth high-precision boundaries no longer use custom-source target schemas. Boundaries are Earth static assets: the Earth Assets settings panel saves local source configuration and triggers PMTiles builds without writing records to `CollectedData`.
### Configuration Semantics
@@ -318,7 +343,7 @@ Important fields:
- `auth_type`: `none`, `bearer`, `api_key`, or `basic`.
- `headers`: static request headers.
- `auth_config`: token, API key, or basic username/password; API keys can be sent by header or query.
- `config.target_schema`: for example `vessel_ais` or `earth_boundary_source`.
- `config.target_schema`: for example `vessel_ais`, `geo_points`, or `generic_records`.
- `config.delivery_mode`: REST defaults to `polling`; WebSocket defaults to `realtime_stream`.
- `config.merge_target_source`: records which built-in source this custom source supplements, such as `barentswatch_vessels`.

View File

@@ -66,10 +66,10 @@ DocsMetadata(
)
```
When adding a public technical doc:
When adding a technical doc that should appear in the Docs page:
- Add both Chinese and English Markdown files.
- Add filename, slug, access, group, order, and titles to server `DOCS_METADATA`.
- Add filename, slug, access, group, order, and titles to server `DOCS_METADATA`. The backend catalog endpoint is authoritative; frontend metadata alone does not publish a document into `/docs` navigation.
- Add matching metadata to frontend [docs-content.ts](/home/ray/dev/linkong/planet/frontend/src/pages/Docs/docs-content.ts) so navigation titles and sorting stay aligned.
- Update `docs/technical/zh/README.md` and `docs/technical/en/README.md` when the document should be discoverable from the README.

View File

@@ -71,6 +71,10 @@ Responsibilities:
This is currently the most critical UI control entry point for the Earth frontend.
Earth settings are now grouped by `data-settings-tab` and `data-settings-tab-panel`. Desktop and mobile share the same category semantics: Runtime, Display, Panels, Motion, Shortcuts, and System. When adding a setting, first choose its category, then add the DOM, persistence field, and restore logic; do not keep growing one long undifferentiated panel.
Shortcut configuration is a device-local preference owned by `controls.js`: read, capture, enable/disable, and reset all stay in the Earth frontend. It should not be written to backend user settings and should not affect other browsers. New shortcuts must provide a default key, display label, disabled/enabled state, and reset path instead of being hard-coded only in a keydown handler.
### 4. UI and Status Messages
- [ui.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/ui.js)
@@ -145,6 +149,22 @@ Each module is responsible for its own:
`tv.js` owns the live / aggregation-news tabs inside `media-panel`. Toolbar open and tab-switch actions write back through `earth:tv-visibility-change` and `earth:tv-tab-change`: panel visibility remains viewport-scoped at `views.<scope>.panelVisibility.media-panel`, while the active tab is stored at `shared.mediaPanelActiveTab`. Refreshing the page therefore restores the user's last live/news state. Temporary hides from `closeTransientMobileOverlays()` carry `persist:false` and do not overwrite the preference.
`brand.js` manages Earth HUD brand resources. Static assets provide the default brand; runtime overrides come from `/api/v1/earth/brand`, and uploaded images are served from `/earth-brand-assets/...`. The frontend must treat logo/title images and text fallback separately: if an image fails, show the text title; if text fields are empty, rely on backend defaults so the HUD brand area never renders blank. The console Earth Content page owns saving and resetting brand configuration; the Earth frontend only consumes it.
`about.js` manages the About card inside Earth settings. Frontend defaults remain as a fallback, while runtime content is loaded from `/api/v1/earth/about`. If the request fails or fields are missing, the renderer must fall back per field so the settings page never renders an empty card. Admin Next exposes an Earth Content `About` tab; saving uses `PUT /api/v1/earth/about`, and restoring defaults uses `DELETE /api/v1/earth/about`.
`oobe.js` manages the first-run Earth initialization guide. OOBE visibility must be driven by `/api/v1/earth/oobe-status` and its `ready` field, not by `localStorage`. `localStorage` may only store a short-lived "skip on this browser" flag; if the backend reports `ready: true`, logout, cleared browser storage, or a different browser must not show OOBE again. Desktop uses a dark starfield scrim and glass startup panel, while mobile uses a bottom sheet and respects `prefers-reduced-motion`.
The Admin Next Earth Content page must preserve runtime semantics:
- `Brand`: brand preview should use the same dark starfield background, size, spacing, logo/title rendering, and text fallback as the Earth HUD top-left brand block, not a generic form preview.
- `About`: configures the About card in Earth settings, including logo, kicker, title, version, description, and metadata items. Earth runtime reads `/earth/about` and falls back to defaults on failure.
- `Boundary Precision`: build boundary, refresh status, and restore defaults belong inside this section, not in the global page toolbar.
- `TV`: the list distinguishes built-in, collected, and custom sources. Card state represents enabled, disabled, draft, or error. Built-in sources cannot be deleted; collected and custom sources can. A new live source only enters draft state after the plus button is clicked; save persists it into the list, while cancel destroys the draft.
- `Basemap`, `Layer Resources`, `3D Models`, and `News Anchor Strategy`: if backend capability is not available yet, the console should show an explicit pending state instead of mixing those items into TV or brand configuration.
TV preview should reuse the Earth runtime live-card structure and state labels as closely as possible so built-in markers, live loading state, stream source, region, and language match what users see on Earth.
The compute-center layer row has a notification badge for GeoJSON `unresolved` records. The badge means "no trustworthy coordinates, cannot render on the globe"; it is different from the `?` marker drawn on already positioned but unconfirmed compute centers. Clicking the badge opens a fixed info card beside the layer panel. Row-level `采集` fetches candidates only. Header-level `一键采用` processes the queue top-to-bottom, saves the highest-confidence valid candidate, removes successful rows, renumbers the list, and dispatches `earth:compute-center-unresolved-count-change` so the badge updates immediately. When the batch ends, `earth:compute-center-location-saved` refreshes the real layer.
Location candidate state in the details card is cached in [info-card.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/info-card.js) by `entityType:entityId`. If the user closes the details card or unresolved queue and reopens the same compute center / BGP collector, previously collected candidates and status text are restored. Header-level `一键采用` prefers cached candidates, avoiding repeated online geocoding or LLM factcheck calls. After a location is saved, that entity's candidate list is cleared to a "refreshing layer" status so stale candidates do not keep misleading the user.
@@ -216,6 +236,8 @@ The real satellite altitude preference is persisted by `controls.js`, while the
SGP4 propagation returns an inertial-frame position, so it must not be drawn directly as Earth-fixed longitude / latitude. `satellites.js` uses `gstime` to convert ECI/TEME positions to ECF, then maps that result into the same Three.js axes as `latLonToVector3()`. Satellite dots and short trails use Earth-fixed coordinates for each sample time, representing the object's current position relative to the globe surface. The locked predicted orbit uses the `gstime` from the lock moment for the whole future orbit, projecting the inertial orbit plane onto the current globe pose; that keeps the line closed and keeps the visual orbit inclination aligned with the details card. Fallback predicted orbits must also use a real RAAN + inclination orbital-plane formula, not treat inclination as a constant latitude.
Boundary precision is stored separately by `country-boundaries.js` under `planet.earth.boundaries.highPrecisionEnabled`. When high precision is off, Earth keeps using the bundled low-precision `countries-admin0.min.geojson` fallback even if high-precision manifest/PMTiles files exist locally. When high precision is on but the artifact is missing, the Earth toolbar settings call `/api/v1/earth/boundaries/build` and poll progress. After success, `reloadCountryBoundaries()` hot-swaps the boundary layer without refreshing the page. Boundary hover is independent from interactable hover: a country polygon remains highlighted whenever the surface coordinate is inside it, while the tooltip can still prioritize a satellite, vessel, BGP marker, or other interactable.
## Current Terrain Pipeline
1. `terrain.js` creates a sphere geometry with enough segments

View File

@@ -78,9 +78,10 @@ The land/ocean base is an Earth base-map asset and preloads at startup; the "Bor
| Name | Variable | Current Value | Location / Notes |
| --- | --- | --- | --- |
| Boundary tile manifest | `COUNTRY_BOUNDARY_CONFIG.tileManifestPath` | `"/earth/data/boundaries/v1/manifest.json"` | Required production PMTiles manifest; missing manifest is an error |
| Boundary tile provider | `COUNTRY_BOUNDARY_CONFIG.tileProvider` | `"pmtiles-mvt"` | Only PMTiles/MVT is accepted for country boundaries |
| Boundary tile manifest | `COUNTRY_BOUNDARY_CONFIG.tileManifestPath` | `"/earth/data/boundaries/v1/manifest.json"` | High-precision PMTiles manifest; missing manifest uses the low-precision fallback |
| Boundary tile provider | `COUNTRY_BOUNDARY_CONFIG.tileProvider` | `"auto"` | Prefer high-precision PMTiles/MVT, then fall back to legacy GeoJSON |
| PMTiles artifact path | `COUNTRY_BOUNDARY_CONFIG.pmtilesPath` | `"/earth/data/boundaries/earth-boundaries-china-pov-v1.pmtiles"` | Production single-file PMTiles/MVT artifact |
| Low-precision fallback | `COUNTRY_BOUNDARY_CONFIG.legacyFallbackPath` | `"/earth/data/countries-admin0.min.geojson"` | Default land/ocean base and hover data when no high-precision boundary asset has been built locally |
| MVT layer names | `COUNTRY_BOUNDARY_CONFIG.mvtLayerNames` | `boundary_admin0 / boundary_disputed_internal / coastline / claim_line` | Fixed layer names decoded by the PMTiles provider |
| Boundary tile base path | `COUNTRY_BOUNDARY_CONFIG.tileBasePath` | `"/earth/data/boundaries/v1/"` | PMTiles manifest base path |
| Boundary tile zoom thresholds | `COUNTRY_BOUNDARY_CONFIG.tileZoomThresholds` | `1.6 -> z5`, `2.8 -> z6`, `3.4 -> z7`, `4.0 -> z8`, `4.6 -> z9`, `5.2 -> z10` | Production PMTiles zoom selection |

View File

@@ -103,7 +103,7 @@ Goals:
## Collector Configuration
`news_live_streams` does not need a separate new page; it reuses Collector Settings under `/settings`:
`news_live_streams` does not need a separate new page; it reuses Collectors under `/collection-management`:
- `endpoint`
- Channel directory JSON API URL

View File

@@ -294,7 +294,7 @@ If only AI Provider is unhealthy, restart just that service:
### Connectivity validation passes, but collection cannot read credentials. Why?
Connectivity validation can read saved console settings, environment variables, and some credentials from `~/.zshrc`. For actual collection, prefer saving credentials in Settings -> Collector Settings, especially for AISStream's long-lived WebSocket collector.
Connectivity validation can read saved console settings, environment variables, and some credentials from `~/.zshrc`. For actual collection, prefer saving credentials in Collection Management -> Collectors, especially for AISStream's long-lived WebSocket collector.
If `AISSTREAM_API_KEY` only lives in `~/.zshrc`, confirm the backend process actually inherited it. Otherwise validation may pass while the collector runtime has no key.
@@ -308,7 +308,7 @@ export BARENTSWATCH_CLIENT_ID="..."
export BARENTSWATCH_CLIENT_SECRET="..."
```
For stable operation, save credentials in Collector Settings so connectivity validation, collection jobs, and Earth realtime aggregation use the same configuration.
For stable operation, save credentials in Collectors so connectivity validation, collection jobs, and Earth realtime aggregation use the same configuration.
## Docs / Permissions
@@ -323,12 +323,29 @@ Docs visibility is controlled by Gatekeeper groups:
## Earth Common Tasks
### Why does Earth say the boundary endpoint is not configured, or only show low precision boundaries?
Country boundaries have moved out of the collector system. They are no longer generated by datasource collection tasks. The low-precision boundary file is bundled with the frontend and is the expected fallback when no local high-precision PMTiles artifact exists.
There are two high-precision entry points:
- Earth page settings gear -> Boundary Precision: switching to High Precision starts the first background download/build, shows percentage progress, and applies the result automatically.
- Console `Operations and Configuration -> Earth Content -> Boundary Precision`: use this to inspect provider, manifest, PMTiles, fallback state, edit source JSON, or rebuild manually.
If the UI says the update source is incomplete, save the source configuration from `Earth Content -> Boundary Precision`. The private local config is written to `config/earth-boundary-sources.local.json`; do not commit it. Falling back to low precision is normal when no high-precision artifact has been built.
### Why did collecting a location candidate not write anything?
Collecting and saving are two separate actions. Candidates can be previewed on Earth first. A candidate is written only after clicking Save or using the unresolved list's one-click adopt flow.
Compute-center saves write to `compute_center_locations` and refresh the layer. Records with no candidate stay in the unresolved list; Planet does not fabricate a location from a country center or hard-coded hint.
### Why did the Earth logo or title not return to the default after I edited it?
Earth brand assets are managed from `Operations and Configuration -> Earth Content -> Brand Assets`. Uploaded images are stored as Earth brand asset URLs. If text fields such as title or ARIA label are cleared, Planet falls back to default text so the HUD never renders an empty brand.
Use `Reset Brand Assets` to restore the shipped logo, title image, and copy. Refreshing Earth does not delete the saved runtime brand configuration.
### Why are satellites no longer on one sphere?
Earth enables "Real Satellite Altitude" by default. Satellite positions still come from TLE/SGP4, but altitude is compressed for display: LEO satellites stay close to the globe, while higher-orbit satellites render farther out without leaving the normal view. The maximum display offset is `25`, about one quarter of the current globe radius; this is a readability compromise that separates GEO / MEO / LEO without drawing real kilometers to scale. This setting also affects satellite trails and the predicted orbit shown after locking a satellite.

View File

@@ -22,7 +22,7 @@ Main entry point:
- [App.tsx](/home/ray/dev/linkong/planet/frontend/src/App.tsx)
Current admin-related routes:
Admin Next now owns the official admin routes:
- `/admin`
- `/users`
@@ -33,17 +33,49 @@ Current admin-related routes:
- `/alerts/situational`
- `/bgp`
- `/ai`
- `/earth-content`
- `/collection-management`
- `/settings`
These routes render [AdminNextRoutes.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/AdminNextRoutes.tsx). Page metadata and menu entries come from [manifest.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/routes/manifest.tsx). `/admin-next/*` remains only as a compatibility entry and redirects to the official route; it is no longer a parallel primary entry.
The old AntD console remains available under `/legacy/admin/*` for comparison and rollback:
- `/legacy/admin`
- `/legacy/admin/datasources`
- `/legacy/admin/data`
- `/legacy/admin/collection-management`
- `/legacy/admin/earth-content`
- `/legacy/admin/ai`
- `/legacy/admin/logs`
- `/legacy/admin/settings`
- `/legacy/admin/users`
- `/legacy/admin/bgp`
- `/legacy/admin/alerts/*`
Legacy pages, `AppLayout`, `antd`, and `@ant-design/icons` stay in place during the legacy validation window. Do not remove them before Admin Next parity is accepted.
`/earth` is a standalone display page and is not part of the console shell.
## Current Page Shell
The console shared shell is at:
The official admin shell is at:
- [AdminNextLayout.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/components/layout/AdminNextLayout.tsx)
Responsibilities:
- Left navigation, grouped collapse, and mobile drawer
- Current account, version, logout, and theme switching
- Top search, breadcrumbs, and page shortcuts
- Single-screen content-area height closure
- Coordination for Admin Next internal scrolling, tables, detail panels, and mobile detail views
The old AntD legacy shell remains at:
- [AppLayout.tsx](/home/ray/dev/linkong/planet/frontend/src/components/AppLayout/AppLayout.tsx)
Responsibilities:
Legacy responsibilities:
- Left-side navigation
- Collapse and expand
@@ -64,7 +96,40 @@ Current structure:
</Layout>
```
Future console pages should adapt to this shell rather than redefining full-page height semantics.
Future official admin pages should adapt to `AdminNextLayout` and Admin Next page patterns rather than adding new capability to the old `AppLayout`. Only `/legacy/admin/*` maintenance should change the old shell.
## Admin Next Section Loading
Multi-tab pages are currently coordinated by [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/pages/PlainResourcePages.tsx), which hosts the current management and information workbench patterns. Section loading follows these rules:
- Initial page load requests only the active tab; it does not prefetch every tab endpoint.
- Switching tabs lazily loads that tab. Loaded tabs stay cached in local `states`, so returning to a tab reuses the previous data.
- Explicit actions such as refresh, save, test, upload, and credential-guide generation refresh only the current section instead of fanning out to unrelated sections.
- Datasource directory filter changes clear the built-in-source section cache and reload the current section with the new filters.
- Top summary metrics count loaded sections only, so unopened tabs are not reported as failed interfaces.
This keeps Earth, AI, collection management, and other multi-section pages from flooding backend APIs on cold start while preserving a fast cached tab-switching experience. Full health checks should use backend health endpoints or explicit refresh flows rather than relying on page initialization to touch every business endpoint.
## Datasource Collection Queue
The Admin Next datasource page routes single-source trigger, table-selected trigger, and trigger-all into a browser-download-list style collection queue:
- Queue state is managed by [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/pages/PlainResourcePages.tsx). It is a current-session visibility layer and does not fake task history in `localStorage`.
- Progress first consumes the `/ws` `datasource_tasks` channel. If the socket is unavailable or stale, the page polls `/api/v1/datasources/{id}/task-status`.
- Trigger responses immediately insert `triggered`, `skipped`, and `failed` items. After a refresh, the queue restores only real backend rows that are still `running`, `pending`, or `queued`.
- The `Built-in Sources` section uses the table selection column for selected-source triggering. With no rows selected, the primary button is `Trigger All`; after selection, the same button becomes `Trigger Selected N`, replacing the old manual-ID batch button.
- The expanded queue no longer lives in the page content flow, so trigger-all cannot squeeze the table and detail panel. The top-right actions area uses the existing `Button` styling; the empty state shows a `ListChecks` icon, and active queues show only a pure circular total-progress indicator. Clicking it opens a floating panel grouped by running, failed, completed, and skipped.
- Queue rows can jump to the datasource detail panel, and failed rows can retry. The detail panel's task summary only reports the selected source's latest task; it does not save configuration.
This queue is a user-perception layer. Backend task status remains the only source of truth for running, completion, failure, and skipped decisions.
## Admin Next Theme Slider
The Admin Next sidebar theme switcher still reuses shared [SegmentedControl.tsx](/home/ray/dev/linkong/planet/frontend/src/components/SegmentedControl/SegmentedControl.tsx), while [styles.css](/home/ray/dev/linkong/planet/frontend/src/admin-next/styles.css) overrides the segment variables by `data-theme`:
- Light mode uses `--d-segment-bg: #eef3f9`, a white slider, and a light external shadow.
- Dark mode uses the same dark base, `#202938` slider, and dark external shadow semantics as Docs.
- Product code only hides the text label and keeps icon + tooltip behavior; it should not recreate the private slider DOM.
## Current Shared Components
@@ -72,7 +137,8 @@ Future console pages should adapt to this shell rather than redefining full-page
File:
- [Scrollbar.tsx](/home/ray/dev/linkong/planet/frontend/src/components/Scrollbar/Scrollbar.tsx)
- [Scrollbar.tsx](/home/ray/dev/linkong/planet/frontend/src/components/tactile-ui/Scrollbar.tsx)
- [Tactile UI Components](/home/ray/dev/linkong/planet/docs/technical/en/tactile-ui-components.md)
Purpose:
@@ -89,7 +155,7 @@ Current constraint:
File:
- [ScrollbarOverlay.tsx](/home/ray/dev/linkong/planet/frontend/src/components/Scrollbar/ScrollbarOverlay.tsx)
- [ScrollbarOverlay.tsx](/home/ray/dev/linkong/planet/frontend/src/components/tactile-ui/ScrollbarOverlay.tsx)
Purpose:
@@ -98,25 +164,42 @@ Purpose:
Current usage:
- Data sources
- Collected data
- User management
- Settings page
- Alerts page
- BGP page
- Admin Next data sources, collected data, collection management, logs, alerts, and BGP pages
- Old AntD legacy pages continue using shared scrolling behavior through compatibility wrappers
### 3. `TableScrollRegion`
File:
- [TableScrollRegion.tsx](/home/ray/dev/linkong/planet/frontend/src/components/Scrollbar/TableScrollRegion.tsx)
- [TableScrollRegion.tsx](/home/ray/dev/linkong/planet/frontend/src/components/tactile-ui/TableScrollRegion.tsx)
Purpose:
- Provides a unified wrapper for table scroll areas
- New table pages should reuse this rather than repeating the "table area + overlay scrollbar" boilerplate
### 4. `SegmentedControl`
### 4. `TactileButton` / `TactileSwitch` / `ControlGroup`
Files:
- [Button.tsx](/home/ray/dev/linkong/planet/frontend/src/components/tactile-ui/Button.tsx)
- [Switch.tsx](/home/ray/dev/linkong/planet/frontend/src/components/tactile-ui/Switch.tsx)
- [ControlGroup.tsx](/home/ray/dev/linkong/planet/frontend/src/components/tactile-ui/ControlGroup.tsx)
Purpose:
- Admin Next global tool buttons and detail-panel toolbars
- Icon-only ordinary actions with tooltips
- Strong-intent actions such as save, create, confirm, delete, and stop
- Compact switches aligned with the Docs theme slider
Current constraints:
- Neutral buttons default to a white tactile surface with external shadow
- Unambiguous actions prefer icon + tooltip; strong-intent actions like save may keep text
- Feature pages should customize through props and CSS variables, not by rewriting core button CSS
### 5. `SegmentedControl`
Files:
@@ -144,7 +227,7 @@ Current constraints:
- Prefer CSS variable overrides for colors instead of hard-coding theme colors in feature components
- Best for a small set of mutually exclusive choices; do not use it as a long list, navigation menu, or select replacement
### 5. `MarkdownRenderer`
### 6. `MarkdownRenderer`
File:
@@ -163,7 +246,7 @@ Current constraints:
- Internal document links should be converted to `/docs/:slug` through `transformLink`
- Heading anchors are injected through `getHeadingId`, keeping route state outside the renderer
### 6. `ConnectionTestInput`
### 7. `ConnectionTestInput`
File:
@@ -181,7 +264,7 @@ Current constraints:
- Disabled integrations must grey out both the input and its connection-test action
- The component only combines the input and action; callers still own form state, loading, disabled state, and the request itself
### 7. `TableActions`
### 8. `TableActions`
File:
@@ -277,6 +360,42 @@ Constraints:
- Do not let tables blow out the full page
- New table areas should reuse `TableScrollRegion` / `ScrollbarOverlay`
### Datasource Directory Page
[DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/DataSources/DataSources.tsx) is the datasource directory and collection operation page. It should not grow back into a configuration editor.
Current page boundary:
- Built-in and custom sources are merged as `UnifiedDataSource`.
- The list shows type, state, last run, collection progress, and actions.
- Clicking a name opens a read-only drawer.
- Endpoint, headers, and config are displayed here, not edited.
- Credential-bearing collectors point users to `Collection Management -> Collectors`.
Keep this boundary: do not put custom datasource editing, built-in endpoint overrides, or credential forms back into `/datasources`. Those configuration entry points live at `/collection-management?tab=collector_credentials`.
### Collectors Page
[Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Settings/Settings.tsx) has three route modes: `/settings` for System Settings, `/earth-content` for Earth Content, and `/collection-management` for Collection Management. The `collector_credentials` tab is shown as `Collectors` under `/collection-management`.
Current boundary:
- The dropdown selects built-in collectors.
- The plug icon beside the dropdown runs the health check.
- Credential-bearing collectors place credential forms above base config.
- Free collectors show endpoint, default endpoint, headers, timeout, and retry.
- BarentsWatch AIS keeps its dedicated credential form.
### Earth Content Page
`/earth-content` reuses the same single-screen tab container from [Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Settings/Settings.tsx), but its ownership is separate from System Settings:
- `TV Livestream` owns the Earth media-panel source configuration.
- `Boundary Precision` owns the Earth static boundary asset state: provider, low-precision fallback, high-precision manifest/PMTiles, source JSON, and build action.
- `Base Map`, `Layer Resources`, `3D Assets`, and `News Anchor Strategy` are placeholders only. They show module status and do not invent fake APIs or fake data.
Do not add Earth experience resources or collection-lifecycle tabs back into `/settings`; collection belongs to `/collection-management`, and Earth display resources belong to `/earth-content`.
### 3. Complex Workspace Pages
Examples:

View File

@@ -76,15 +76,17 @@ The console at `http://localhost:3000/admin` is built with React + Ant Design. T
| BGP Alerts | `/alerts/bgp` | BGP-related alerts |
| Situational Alerts | `/alerts/situational` | Situational analysis alerts |
| AI | `/ai` | Model providers, tools, testbench |
| Earth Content | `/earth-content` | TV livestreams, boundary precision, base-map and layer-resource entry points |
| Collection Management | `/collection-management` | Collectors, scheduling, collection history entry points |
| Logs | `/logs` | Usually visible only to super admin |
| Users | `/users` | Create/delete users, change roles/groups |
| Settings | `/settings` | System, SMTP, TV, collectors |
| System Settings | `/settings` | Display, notification, security, SMTP |
Menu items hide automatically when you lack permission. If a menu is missing, check your role and Gatekeeper groups.
## Configure Data Collectors
`/settings?tab=collector_credentials` is the "Collector Settings" page. It manages connection configuration for every collector, not just credentials.
`/collection-management?tab=collector_credentials` is the "Collectors" page. It manages connection configuration for every collector, not just credentials. Legacy `/settings?tab=collector_credentials` redirects here; the datasource directory remains at `/datasources`.
Steps:
@@ -125,7 +127,7 @@ The default guide follows the BarentsWatch official tutorial and reminds you to
Steps:
1. Open `/settings?tab=collector_credentials` and select `AISStream Realtime Vessels : aisstream_vessels`
1. Open `/collection-management?tab=collector_credentials` and select `AISStream Realtime Vessels : aisstream_vessels`
2. Fill the AISStream API Key
3. Keep the default endpoint `wss://stream.aisstream.io/v0/stream`
4. Click the plug icon to test; confirm it reports `Reachable`
@@ -139,10 +141,11 @@ Steps:
## Configure AI Credentials
`/ai?tab=providers` is the AI management entry. Two key sub-tabs:
`/ai?tab=providers` is the AI management entry. Three key sub-tabs:
- `Model Providers`: default LLM provider, model, base URL, API key, local `aiprovider` proxy, connection test
- `Tools`: a dropdown for specific tools — currently WebSearch and OCR
- `Prompts`: a task dropdown for news localization, alert analysis, BGP briefs, and other LLM tasks. Operators can edit the prompt or reset it to the default
### Model Providers
@@ -163,6 +166,10 @@ The plug icon at the end of the Base URL input runs a connection test. A passing
- **WebSearch**: provider, API key, base URL, max results, timeout, advanced provider parameters. While disabled, all fields except the enable switch are greyed out
- **OCR**: provider, base URL, API key, model/engine, recognition languages, timeout, max file size, output format
### Prompts
After selecting a task, the page shows the effective prompt, whether it is customized, the shipped default version, and a reset button. Saving affects only that task. Reset restores the default prompt from the current release package. Business facts, context, and output schemas are still assembled by the backend for each task.
The legacy link `/settings?tab=ai` redirects to `/ai?tab=providers`.
## System Settings
@@ -173,8 +180,30 @@ The legacy link `/settings?tab=ai` redirects to `/ai?tab=providers`.
- **Notifications**: alert email switch, recipient, critical/warning/daily summary
- **Security**: session timeout, max login attempts, password policy
- **SMTP Email**: outgoing email used by registration and password reset (visible to `admin` / `super_admin` only)
- **TV Livestream**: TV source management
- **AI / WebSearch / OCR**: see above
TV livestreams and boundary precision moved to `/earth-content`; collectors and scheduling moved to `/collection-management`; AI Provider / WebSearch / OCR live at `/ai`.
### Earth Content
`/earth-content` is under the console's Operations and Configuration group and owns resources used by the Earth frontend:
- **Brand Assets**: manages the logo, title image, title text, subtitle, and description used by the Earth HUD. Uploaded images are saved as Earth brand assets and read by the Earth page immediately.
- **About**: manages the About card shown in Earth settings, including logo, kicker, title, version, description, and metadata.
- **TV Livestream**: manages sources shown in the Earth media panel.
- **Boundary Precision**: shows the current provider, low-precision fallback, high-precision PMTiles/manifest status, local source JSON, and manual build action.
- **Base Map**, **Layer Resources**, **3D Assets**, and **News Anchor Strategy**: placeholder tabs for future configuration. They do not display fake data.
The Earth page settings gear also includes Boundary Precision. Switching to High Precision starts a local background download/build, like a game update package, when no high-precision asset exists yet. Progress is shown as a percentage, and the result applies automatically after success without a page reload. Switching back to Low Precision only changes the local display preference.
If the backend decides Earth is not initialized yet, the first visit to `/earth` shows a glassy startup guide that points the user to sign in and collect data. This is based on backend data state; once the system has collected data, clearing browser storage does not make the guide reappear.
### Collection Management
`/collection-management` is also under Operations and Configuration and owns the collection lifecycle:
- **Collectors**: endpoint, headers, credentials, timeout, retry, and connection checks.
- **Collection Scheduling**: the existing scheduling configuration.
- **Collection History / Snapshots**: groups snapshots by datasource and lets the detail panel switch versions through a Time Capsule selector.
### SMTP Email Settings
@@ -204,7 +233,7 @@ To let a regular user read developer or operations docs, add `docs_developer` or
## Data Exploration
- `/datasources`: source directory. The `Collection Tasks` tab is for one-shot, scheduled, and finite collectors; it can be filtered by product domain, layer/module, enabled state, last run status, whether collected records exist, and search text. Selecting rows triggers only those sources; with no selected rows, `Collect current filter` triggers the filtered scope. The `Realtime Streams` tab is for AISStream / WebSocket long connections and shows connection health, stored totals, time-window counters, and Start / Stop / Reconnect actions. Clicking a name opens an info drawer showing endpoint, headers, base config, and built-in flag; endpoint/credentials editing happens at `/settings -> Collector Settings`. The `Collecting N` tag under the overall progress can be clicked to expand the current collection task list
- `/datasources`: source directory. The `Built-in Sources` tab can be filtered by product domain, layer/module, enabled state, last run status, whether collected records exist, and search text. With no rows selected, the primary button shows `Trigger All`; after selecting rows, it becomes `Trigger Selected N` and submits only those sources. The top-right queue button shows a queue icon when empty and a pure circular total-progress indicator while tasks exist; it opens a floating panel grouped by running, completed, failed, and skipped. Failed rows can retry, and completed rows can jump to detail. `Realtime Sources` is for AISStream / WebSocket long connections and shows connection health, stored totals, time-window counters, and Start / Stop / Reconnect actions. Endpoint/credential/header editing happens at `/collection-management -> Collectors`.
- `/data`: collected data table — used to verify "did data arrive", "is the freshness right", "does a source emit valid records"
- `/bgp`: BGP detail page with list + detail + analysis; complements the BGP layer on Earth
- `/alerts/system`, `/alerts/bgp`, `/alerts/situational`: system, BGP, and situational alerts
@@ -255,7 +284,7 @@ Candidates preview on Earth directly. Saving a compute-center candidate writes t
### Settings
The settings panel covers: rotate / cruise / motion mode, cruise modules (BGP/news/compute centers/vessels/cables/satellites), view (satellite display style, hover tooltip, satellite idle breathing, real satellite altitude, track display, compact dots, day-night mode, panel toggles), motion debug mode / input source / skeleton-only, default globe size, terrain opacity, reset.
The settings panel is grouped into Runtime, Display, Panels, Motion, Shortcuts, and System. It covers rotate / cruise / motion mode, cruise modules (BGP/news/compute centers/vessels/cables/satellites), view (satellite display style, hover tooltip, satellite idle breathing, real satellite altitude, track display, compact dots, day-night mode, panel toggles), motion debug mode / input source / skeleton-only, shortcut enablement and remapping, default globe size, terrain opacity, reset.
"Real Satellite Altitude" is enabled by default: satellite positions use a compressed display height based on TLE/SGP4 orbital altitude. LEO satellites remain close to the globe, while high-orbit satellites render farther out without leaving the normal view. The high-orbit display height is capped at about one quarter of the globe radius, so GEO / MEO objects remain visually separated from LEO without spreading trails and selection targets too far apart. Turning it off restores the legacy same-sphere satellite display. "Track Display" controls satellite trail visibility; trails are unavailable while the satellite layer is hidden.
@@ -263,6 +292,8 @@ The settings panel covers: rotate / cruise / motion mode, cruise modules (BGP/ne
These settings live in browser local storage; switching browsers or clearing site data resets them.
Shortcuts also live in browser local storage. Use the Shortcuts category to disable an individual shortcut, capture a new key, or restore the default binding for the current browser only.
### View Controls
| Action | Effect |

View File

@@ -0,0 +1,99 @@
# Naming Glossary
This document standardizes terms used across the Planet console, Earth, backend APIs, and documentation. When adding Chinese UI labels, chart labels, or translated documentation, prefer the Chinese display names listed here to avoid unnecessary mixed Chinese/English copy.
## Usage Rules
- Chinese UI should use the Chinese display name.
- API fields, database fields, code identifiers, and external brand names keep their original English or snake_case form.
- On first mention in long-form docs, use “Chinese Display Name (English Term)” when clarification helps; subsequent mentions can use the Chinese display name.
- Industry abbreviations such as BGP, AI, API, URL, ID, TLE, ASN, GPU, and AIS may remain English.
- External service brands keep their official English names, such as PeeringDB, Space-Track, BarentsWatch, AISStream, and GitHub.
## Core Product Terms
| English / Key | Chinese Display Name | Usage |
| --- | --- | --- |
| Planet | Planet | Product name |
| Admin Next | 新控制台 | New admin console |
| Earth | Earth | Visualization product name |
| datasource | 数据源 | APIs, lists, filters |
| collector | 采集器 | Collection jobs and credential configuration |
| collected data | 采集数据 | Data list and statistics |
| snapshot | 快照 | Collection history |
| mapping | 映射 | Field mapping |
| target schema | 目标 Schema | Mapping target structure |
| credential | 凭证 | Secrets, account passwords, OAuth material |
| secret reveal | 密钥 reveal | Explicit administrator secret reveal action |
| AI Provider | AI Provider | Service name |
| tool | 工具 | Web Search, OCR, and similar integrations |
| Playground | Playground | Interactive debugging entry |
## Data Types
| data_type | Chinese Display Name | Meaning |
| --- | --- | --- |
| `supercomputer` | 超算 | TOP500 supercomputer record |
| `gpu_cluster` | GPU 集群 | Epoch AI GPU cluster |
| `model` | 模型 | Hugging Face model |
| `dataset` | 数据集 | Hugging Face dataset |
| `space` | 空间 | Hugging Face Space |
| `ixp` | 互联网交换点 | PeeringDB IXP |
| `network` | 网络 | PeeringDB network |
| `facility` | 设施 | PeeringDB facility |
| `submarine_cable` | 海底光缆 | Submarine cable route |
| `landing_point` | 登陆点 | Cable landing point |
| `cable_system` | 海缆系统 | TeleGeography cable system |
| `cable_landing_relation` | 海缆登陆关系 | Cable-to-landing-point relation |
| `satellite_tle` | 卫星轨道根数 | CelesTrak / Space-Track TLE |
| `bgp_rib` | BGP 路由表 | BGP RIB snapshot |
| `bgp_update` | BGP 更新 | BGP update event |
| `prefix_geography` | 前缀地理位置 | IP prefix geography |
| `vessel_ais` | AIS 船舶 | AIS vessel observation |
| `vessel` | 船舶 | Aggregated vessel object |
| `news_live_stream` | 新闻直播源 | News live stream source |
| `news_item` | 新闻条目 | Archived news item |
| `device_stats` | 设备统计 | Cloudflare Radar device statistics |
| `traffic_stats` | 流量统计 | Cloudflare Radar traffic statistics |
| `as_stats` | 自治系统统计 | Cloudflare Radar AS statistics |
| `compute_center` | 算力中心 | Aggregated Earth compute-center object |
| `generic` | 通用数据 | Generic collector output |
| `generic_records` | 通用记录 | Generic mapping output |
## Datasources And External Services
| English / Key | Chinese Display Name | Notes |
| --- | --- | --- |
| TOP500 | TOP500 超算榜单 | Keep TOP500 as the external list name |
| Epoch AI | Epoch AI GPU 集群 | Keep Epoch AI as the service name |
| Hugging Face | Hugging Face | Official brand name |
| PeeringDB IXP | PeeringDB 交换中心 | IXP can also be explained as 互联网交换点 |
| PeeringDB Networks | PeeringDB 网络 | Network entity |
| PeeringDB Facilities | PeeringDB 设施 | Facility entity |
| Submarine Cables | 海底光缆 | Prefer Chinese in charts and lists |
| Cable Landing Points | 海缆登陆点 | Use 登陆点 consistently |
| CelesTrak TLE | CelesTrak 轨道根数 | TLE may remain abbreviated |
| Space-Track TLE | Space-Track 轨道根数 | Keep Space-Track as brand name |
| RIPE RIS Live BGP | RIPE RIS Live BGP | Keep service name and BGP abbreviation |
| IPtoASN Prefix Geography | IPtoASN 前缀地理位置 | Keep IPtoASN as dataset name |
| OpenGeoFeed Prefix Geography | OpenGeoFeed 前缀地理位置 | Keep OpenGeoFeed as brand name |
| NRO Delegated Prefix Geography | NRO Delegated 前缀地理位置 | Keep NRO Delegated as dataset name |
| BarentsWatch AIS Vessels | BarentsWatch AIS 船舶 | Keep BarentsWatch / AIS |
| AISStream Vessels | AISStream 船舶 | Keep AISStream as brand name |
## UI Actions And Statuses
| English / Key | Chinese Display Name | Usage |
| --- | --- | --- |
| success | 成功 | Status badge |
| failed | 失败 | Status badge |
| running | 运行中 | Task status |
| completed | 已完成 | Task status |
| cancelled | 已取消 | Task status |
| enabled | 已启用 | Toggle state |
| disabled | 已停用 | Toggle state |
| configured | 已配置 | Credential / integration state |
| unconfigured | 未配置 | Credential / integration state |
| refresh | 刷新 | Button |
| copy | 复制 | Button |
| reveal | reveal | Keep English for audited secret reveal semantics |

View File

@@ -21,7 +21,7 @@ First startup seeds two default accounts (see `DEFAULT_LOGIN_USERS` in `backend/
| Username | Password | Role |
| --- | --- | --- |
| `admin` | `admin123` | `super_admin` |
| `linkong` | `12345678` | `super_admin` |
| `linkong` | `LK12345678` | `super_admin` |
Both seed accounts are created with `email_verified = TRUE` and can log into the console immediately. Any other account must either go through the public registration flow described in the Manual, or be created via `./planet.sh createuser`.
@@ -61,6 +61,22 @@ Per-module restart:
Per-module restart is preferred during development to avoid interrupting unrelated services.
## Destructive Reset
```bash
./planet.sh destroy
```
`destroy` returns a local development environment to a near-empty project state. It requires typing `Y` before it runs; source files and existing `.env` files are preserved.
Cleanup order and boundaries:
- If `planet_postgres` is running, the script first clears the `public` schema in `planet_db`. This prevents old `collected_data.is_current = true` rows from making Earth OOBE report `ready=true` if Docker volume removal later fails.
- Docker cleanup targets resources whose Compose project is `planet`, plus the explicit volumes `planet_postgres_data`, `planet_redis_data`, `postgres_data`, and `redis_data`; do not delete unlabeled volumes by a broad `planet_*` pattern, because another local project could own them.
- Local build state removes `.venv`, frontend `node_modules` / `dist`, Planet state/cache, and scattered Python / Vite cache directories.
After the reset, run `./planet.sh init` again to recreate tables and default seed data. Old collected records are not restored, and Earth OOBE is evaluated from the backend's real collection state on the next visit.
## Health Check
```bash
@@ -87,8 +103,8 @@ Recent logs:
Follow:
```bash
./planet.sh log -f # frontend: /tmp/planet_frontend.log
./planet.sh log -b # backend: /tmp/planet_backend.log
./planet.sh log -f # frontend: ~/.local/state/planet/frontend.log
./planet.sh log -b # backend: ~/.local/state/planet/backend.log
./planet.sh log -a # AI Provider: planet_aiprovider container logs
```
@@ -234,6 +250,20 @@ Validate the frontend build:
source ~/.zshrc && bun run build
```
Use `bun run dev` during development; Vite HMR refreshes the browser after source saves. `bun run build` only writes the `dist` artifact and does not refresh an already-open dev page.
To inspect the production bundle with automatic reload after successful builds:
```bash
bun run preview:auto
```
To watch sources and rebuild continuously without starting the preview server:
```bash
bun run build:watch
```
Backend dependencies are managed with uv:
```bash
@@ -243,13 +273,12 @@ uv run pytest backend/tests/test_otp_service.py
## Earth Boundary PMTiles Operations
1. In Collector Settings, configure endpoints, headers/auth, `config.target_schema=earth_boundary_source`, license, and mapping for `Earth Admin-0 Boundaries`, `Earth Coastline`, and `Earth Claim Lines`.
2. In the data-source console, collect those three sources first. Each successful source writes the full artifact to `data/earth-boundary-sources/<collector>/<sha256>.*` and stores sha256, feature count, artifact path, and sample properties.
3. After all three sources succeed, collect `Earth PMTiles Builder`. If any source is missing, it fails as "not ready" and does not update Earth boundaries.
4. The builder requires `tippecanoe` and `pmtiles` on PATH. Missing tools fail the task with a clear message.
5. A successful production build outputs `frontend/public/earth/data/boundaries/earth-boundaries-china-pov-v1.pmtiles` and its manifest.
6. After deployment, open Earth, enable "Border Lines", and inspect China's southeast coast, Taiwan, Hainan, the South China Sea, Zangnan, Kosovo, and Gaza for hover behavior and boundary policy.
7. If PMTiles loading fails, Earth reports a boundary-layer error and does not draw legacy low-precision borders. Troubleshoot in this order: browser Network range requests for PMTiles, manifest `tileProvider: "pmtiles-mvt"`, Nginx static serving for `.pmtiles`, and artifact path / sha256 consistency with the manifest.
1. In the console, open `Operations and Configuration -> Earth Content -> Boundary Precision` to save boundary source configuration. The local config is written to `config/earth-boundary-sources.local.json`; do not commit it.
2. Click "Build high precision boundaries", or switch the Earth toolbar settings gear to High Precision for the first build. The backend downloads the three source packages to `data/earth-boundary-sources/`, writes the source manifest, and invokes the PMTiles build script.
3. The builder requires `tippecanoe` and `pmtiles` on PATH. Missing tools return a clear API error and do not write data-source collection records.
4. A successful production build outputs `frontend/public/earth/data/boundaries/earth-boundaries-china-pov-v1.pmtiles` and its manifest.
5. After deployment, open Earth, enable "Border Lines", and inspect China's southeast coast, Taiwan, Hainan, the South China Sea, Zangnan, Kosovo, and Gaza for hover behavior and boundary policy.
6. If no high-precision manifest/PMTiles exists locally, Earth uses the bundled `frontend/public/earth/data/countries-admin0.min.geojson` fallback. If high-precision assets exist but tile requests fail, troubleshoot PMTiles range requests, manifest provider, Nginx `.pmtiles` static serving, and sha256 consistency.
## Related Docs

View File

@@ -11,7 +11,7 @@ Open the URL your administrator gave you, e.g. `http://planet.example.com`. A lo
Entry points are split in two:
- Public: `/earth` (3D situational view), `/docs` (public documentation)
- Login required: `/admin` (console), `/ai` (AI), `/settings` (system configuration)
- Login required: `/admin` (console), `/ai` (AI), `/settings` (system settings), `/earth-content` (Earth content), `/collection-management` (collection management)
## 2. Register
@@ -32,9 +32,9 @@ The default role is `viewer`: you can sign in but only see public pages. For col
After landing on the `/admin` dashboard, here's a recommended walk-through:
1. `/settings?tab=collector_credentials`: pick a collector and click the plug icon to test connectivity. Free collectors (e.g. open BGP) usually work right away; credential-bearing ones like `AISStream` or `BarentsWatch` need an API key / client secret first
1. `/collection-management?tab=collector_credentials`: pick a collector and click the plug icon to test connectivity. Free collectors (e.g. open BGP) usually work right away; credential-bearing ones like `AISStream` or `BarentsWatch` need an API key / client secret first
2. `/ai?tab=providers`: fill an LLM provider (e.g. `minimax` / `openai`), model, base URL, API key, and click the plug at the end of the base URL to test. WebSearch / OCR tools are optional
3. `/datasources` or `/data`: check whether collectors have produced data. Use `/datasources -> Collection Tasks` for finite collectors, and `/datasources -> Realtime Streams` for AISStream / WebSocket health and counters
3. `/datasources` or `/data`: check whether collectors have produced data. Use `/datasources -> Built-in Sources` for finite collectors: with no rows selected, click `Trigger All`; after selecting rows, the primary button becomes `Trigger Selected N`. The top-right queue button shows progress. Use `/datasources -> Realtime Sources` for AISStream / WebSocket health and counters
4. `/alerts/system`: verify system alerts look right
5. `/users` (super_admin only): open accounts for teammates or adjust their groups
@@ -42,6 +42,8 @@ After landing on the `/admin` dashboard, here's a recommended walk-through:
Visit `/earth`. This is a public page — no login required.
If the system has no collected data yet, Earth shows an initialization guide that points you to sign in and trigger collection. This guide is backend-state driven, so clearing browser storage does not make it appear once the system is ready.
Once in, verify:
- The globe renders, and the right-side layer panel can toggle layers

View File

@@ -0,0 +1,223 @@
# Tactile UI Components
Tactile UI is Planet's portable React control layer. It was extracted from the Admin Next button, switch, scrollbar, and tooltip work, but the components themselves do not depend on Admin Next, AntD, Radix, Tailwind, or `an-*` classes. The immediate goal is stable in-repo usage; the structure is intentionally close to something that can later be published as an npm package.
## Design Goals
- **Light tactile feel**: controls default to a white or themed surface, thin borders, and external shadow, matching the subtle depth of the Docs theme slider rather than large colored blocks or glow.
- **Portable styling**: classes use the `tui-*` prefix and styles live in `frontend/src/components/tactile-ui/styles.css`.
- **Low dependency surface**: components assume React and React DOM. Preset icons currently use `lucide-react`, and callers may also pass custom React nodes.
- **Theme friendly**: CSS variables expose the default styling surface. Planet pages adapt the library through theme variables and props.
- **Clear semantics**: unambiguous actions should prefer icon-only buttons with tooltips; strong-intent actions such as save, confirm, create, and run can keep text.
## Import
In this repository:
```tsx
import { TactileButton, TactileSwitch, ControlGroup } from '@/components/tactile-ui'
import '@/components/tactile-ui/styles.css'
```
After a future package extraction:
```tsx
import { TactileButton, TactileSwitch } from '@planet/tactile-ui'
import '@planet/tactile-ui/styles.css'
```
## Theme Tokens
Core tokens are exposed as `--tui-*` CSS variables. Product themes should override variables instead of rewriting internal component classes.
```css
:root {
--tui-surface: #ffffff;
--tui-surface-raised-hover: #f8fbff;
--tui-border-soft: #d6dfeb;
--tui-border-hover: #b8c6d9;
--tui-text: #0f172a;
--tui-primary: #2563eb;
--tui-danger: #dc2626;
}
[data-theme='dark'] {
--tui-surface: #111827;
--tui-text: #e5edf8;
}
```
Most controls also accept a `tactile` prop for local width, height, radius, background, border, and shadow overrides. Use local overrides for small special cases; use CSS variables for product-wide styling.
## Portals and Dark Theme
`TactileTooltip` and the Admin Next `Dialog`, `Select`, and toast controls that use Tactile UI may render through a portal attached to `document.body`. Those nodes are not descendants of `.admin-next-theme-root[data-theme='dark']`, so dark tokens cannot rely only on an ancestor selector inside the Admin Next root.
The Admin Next theme provider mirrors the active theme to `body[data-admin-next-theme]`. Shared styles need to support both selector paths:
```css
[data-theme='dark'] .tui-button,
body[data-admin-next-theme='dark'] .tui-button {
--tui-surface: #172033;
--tui-text: #e5edf8;
}
```
When adding a portal-based control, first check whether it renders into body. If it does, add a `body[data-admin-next-theme='dark']` branch in that component's style entry, or reuse the already covered `--tui-*` / `--an-*` tokens. Avoid hard-coding a one-off dark modal style, because the same contrast problem can reappear in dropdowns, tooltips, toasts, and confirmation dialogs.
## `TactileButton`
The button component covers regular buttons, icon buttons, strong-intent buttons, and link-like buttons.
Common props:
| Prop | Description |
| --- | --- |
| `variant` | `neutral`, `primary`, `danger`, `subtle`, or `ghost` |
| `size` | `sm`, `md`, `lg`, or `icon` |
| `shape` | `square` or `pill` |
| `icon` | Preset icon name or a custom React node |
| `iconOnly` | Fixed-size icon button; provide `tooltip` or `aria-label` |
| `tooltip` | Rendered through a portal and offset away from the cursor |
| `loading` | Disables the button and exposes `aria-disabled` |
| `tactile` | Overrides size, radius, shadow, background, and dark-mode variables |
```tsx
<TactileButton icon="refresh" iconOnly tooltip="Refresh" />
<TactileButton variant="primary" icon="save">Save</TactileButton>
<TactileButton variant="danger" icon="delete" iconOnly tooltip="Delete" />
```
`variant="neutral"` defaults to a white tactile button. Colored buttons should still keep the same height and external shadow instead of relying on page-specific CSS overrides.
Colored button borders must not use the exact fill color. `primary`, `danger`, and future colored variants should use a lighter border from the same hue, such as `color-mix(in srgb, var(--tui-danger) 64%, white)`. The border still reads as part of the button color, but its visual weight is lower than the fill surface, so red or blue buttons do not look one outline larger than neutral buttons. Hover states should brighten rather than darken: mix a little white into the current fill color, and keep the hover border lighter than the hover fill.
## Icon Presets
Preset icons are maintained in `tactileIconPresets`. Feature pages should call icons by semantic name so actions remain consistent across the console.
Common semantics:
| Name | Use |
| --- | --- |
| `refresh` | Refresh data |
| `save` | Save |
| `delete` | Delete |
| `trigger` / `collect` | Trigger collection |
| `start` / `play` | Start |
| `stop` | Stop |
| `connect` / `test` | Connectivity check |
| `guide` | Credential guide |
| `generate` | AI generation |
| `reset` | Restore defaults |
| `detail` | View details |
| `copy` | Copy |
| `upload` | Upload |
Custom icons are also supported:
```tsx
<TactileButton icon={<MyIcon aria-hidden="true" />} iconOnly tooltip="Custom action" />
```
## `TactileSwitch`
`TactileSwitch` is for binary settings. It is not a large iOS-style switch; it matches the compact tactile slider used by Docs.
```tsx
<TactileSwitch
checked={enabled}
onCheckedChange={setEnabled}
label="Enable WebSearch"
tooltip="Enable WebSearch"
/>
```
The label defaults to tooltip content. If visible text is needed, render it in the surrounding layout rather than inside the switch.
## `ControlGroup`
`ControlGroup` arranges action buttons with consistent spacing and alignment. It has no grey base by default; use `withBase` or `baseTactile` only when a base surface is intentional.
```tsx
<ControlGroup align="end" gap={8}>
<TactileButton icon="refresh" iconOnly tooltip="Refresh" />
<TactileButton variant="primary" icon="save">Save</TactileButton>
</ControlGroup>
```
In list footers, the group should appear at the end of the scroll content, not float in the middle of the list. In detail toolbars, size and shadow should come from the shared component defaults.
## `TactileTooltip`
Tooltips render to `document.body` via `createPortal`, support `top`, `right`, `bottom`, and `left`, and perform simple collision correction. The default position is offset down and to the right from the trigger so it does not sit under the cursor.
```tsx
<TactileTooltip label="Refresh" anchor={buttonRef.current} open={open} side="bottom" />
```
Application code usually does not need to use it directly because `TactileButton` and `TactileSwitch` include tooltip handling.
## `Scrollbar`
`Scrollbar` wraps ordinary scrollable content. Native scrolling remains on the viewport; Tactile UI only renders the overlay track and thumb.
```tsx
<Scrollbar axis="both" minThumbSize={28} autoHide>
<LongContent />
</Scrollbar>
```
Common props:
| Prop | Description |
| --- | --- |
| `axis` | `x`, `y`, or `both` |
| `autoHide` | Fade/soften when not hovered or dragged |
| `alwaysVisible` | Keep scrollbars visible |
| `trackSize` / `thumbSize` | Track and thumb dimensions |
| `inset` / `radius` | Edge inset and radius |
| `thumbColor` / `trackColor` | Local color overrides |
| `viewportRef` | External real scroll node |
Do not force overlay scrollbars onto textareas. Text selection and the resize grip need native behavior. If a consistent textarea look is needed, use a dedicated resizable textarea style that preserves the native grip hit area.
## `ScrollbarOverlay`
`ScrollbarOverlay` is for areas that already own a scroll node, such as a third-party table or custom viewport. It does not create a new scroll container; it observes the target and paints overlay scrollbars.
```tsx
<div ref={hostRef}>
<ThirdPartyTable />
<ScrollbarOverlay containerRef={hostRef} targetSelector=".tui-scroll-target" />
</div>
```
## `TableScrollRegion`
`TableScrollRegion` is a convenience wrapper for table scroll areas. The default target selector is `.tui-scroll-target`; Admin Next passes its table viewport selector explicitly so package code does not contain product-specific names.
```tsx
<TableScrollRegion targetSelector=".table-viewport">
<EntityTable />
</TableScrollRegion>
```
## Accessibility
- `iconOnly` buttons must provide `tooltip` or `aria-label`.
- `TactileSwitch` uses `role="switch"` and `aria-checked`.
- Tooltip text is explanatory only; state must still be represented by text, badges, or `aria-*` attributes.
- Disabled and loading states set `aria-disabled`; real `button` elements also receive `disabled`.
## Admin Next Migration Rules
Admin Next should use Tactile UI for global tool buttons, detail-panel toolbars, and list footer actions:
- Unambiguous actions: `TactileButton iconOnly tooltip`
- Save/create/confirm: `TactileButton variant="primary"`, usually with text
- Delete/stop: `TactileButton variant="danger"`, with text only when risk or ambiguity requires it
- Switches: `TactileSwitch`
- Long lists, tables, logs, and Markdown: `Scrollbar` / `ScrollbarOverlay` / `TableScrollRegion`
The boundary is deliberate: Tactile UI owns touch feel, size, shadow, tooltip, and scrollbar behavior; feature pages own APIs, state machines, permissions, and copy.

View File

@@ -28,8 +28,10 @@
- [数据源、采集器设置与连接验证](/home/ray/dev/linkong/planet/docs/technical/zh/datasource-collector-settings-connectivity.md)数据源目录、采集器设置、连接验证、BarentsWatch 凭证链路
- [通用位置估算管线开发说明](/home/ray/dev/linkong/planet/docs/technical/zh/location-pipeline-development.md):后端 location resolver / pipeline 的接口、注册表和扩展方式
- [Docs Gatekeeper 开发说明](/home/ray/dev/linkong/planet/docs/technical/zh/docs-gatekeeper-development.md):后端 Docs 目录、正文读取和 Gatekeeper 权限组实现
- [命名与术语对照](/home/ray/dev/linkong/planet/docs/technical/zh/naming-glossary.md)控制台、Earth、后端和文档常见名词的中英对照
- [Earth 可交互图标接入](/home/ray/dev/linkong/planet/docs/technical/zh/earth-interactable-usage.md)Earth 地表可交互图标 `Interactable` 的接口、生命周期和接入示例
- [Earth 工具栏与浮层协同](/home/ray/dev/linkong/planet/docs/technical/zh/earth-toolbar-overlay-coordination.md):工具栏按钮与搜索 / 设置 / 新闻 / 图层浮层之间的关闭矩阵和接入规则
- [Tactile UI 组件库](/home/ray/dev/linkong/planet/docs/technical/zh/tactile-ui-components.md)可移植按钮、开关、tooltip 和滚动条组件的 API、主题 token 与迁移约定
不适合放入这里的内容:

View File

@@ -95,9 +95,37 @@ AI 配置页使用的接口:
- `POST /api/v1/settings/integrations/ai-provider/connect`
- `GET /api/v1/settings/integrations/ai-provider/secrets`
- `GET /api/v1/settings/integrations/ai-provider/presets`
- `GET /api/v1/settings/ai-prompts`
- `PUT /api/v1/settings/ai-prompts/{task_key}`
- `POST /api/v1/settings/ai-prompts/{task_key}/reset`
这些接口都需要用户登录。`secrets` 接口只用于配置页点击显示 key/token 时取回明文,隐藏时前端恢复为脱敏预览。
Admin Next 的 AI 页面按旧版信息架构组织为:
- `模型供应商`
- 管理 provider、协议适配、默认模型、LLM API Key、代理地址、代理 token、模型列表刷新、设为默认和轻量连通性测试。
- `工具调用`
- 管理 WebSearch、OCR 等工具。工具内部先选 provider再编辑该 provider 的 API、key 和高级参数。
- `提示词`
- 按 prompt 分组和 task key 编辑系统提示词 / 用户提示词;保存和重置只作用于当前 task。
- `Playground`
- 使用当前默认 provider 和 prompt 配置发起真实会话AI 回复使用 Markdown 渲染。
保存、设为默认和连通性测试是三个独立职责:保存只持久化表单,设为默认只切换默认 provider/tool连通性测试只验证当前草稿是否可用不应隐式保存或切换默认项。
`ai-prompts` 接口用于运维配置页的“提示词”Tab。默认提示词来自后端随发布包携带的版本化资源业务代码只引用稳定 task key接口只保存运维覆盖值。重置时删除覆盖值并恢复当前发布包中的缺省提示词。
### 提示词边界
`aiprovider` 是纯模型适配器,不注入通用业务 system prompt。新闻汉化、告警研判、BGP 简报、位置 factcheck、数据源映射和凭据教程等入口各自通过 task key 解析有效提示词。告警研判 prompt 只会在告警相关 task 中作为 system prompt 传入,不会污染其它 LLM 调用。
### Agent 与工具边界
Agent 工作流属于 `backend`,不属于 `aiprovider`。后续 Earth LLM 指令、态势感知、多角色模拟、WebSearch、数据库查询、证据存储和配置提案应用都应由后端 Agent Runtime 编排;`aiprovider` 只接收后端整理好的模型请求并返回规范化响应。
如果某个 provider 支持原生 tool calling`aiprovider` 可以透传协议字段并规范化响应块但工具白名单、参数校验、权限策略、运行记录和写入审批仍必须留在后端。provider 不支持原生 tools 时,后端使用 JSON tool-call fallback不应为了某个模型厂商把业务工具下沉到 `aiprovider`
### AI Provider 内部 API
仅供内部调用的接口:
@@ -244,6 +272,34 @@ AI 配置仍保存在 PostgreSQL不写入 JSON 文件。核心结构如下:
`.env` 只是兜底。配置页保存或测试连接成功后PostgreSQL 中的配置会成为全局默认。
Admin Next 的密钥状态必须按 provider / tool 精确判断:
- 数据库中当前 provider/tool 有密钥时,显示为“已配置”。
- 数据库没有密钥,但 fallback provider、model 或 tool 与当前项匹配时,可以显示 fallback 的脱敏预览。
- 数据库没有密钥且 fallback 不匹配当前项时,显示为“未配置”,不能把其它 provider 的 `.env` 通用 key 当成当前项已配置。
- 脱敏规则保留 key 第一个 `-` 之前的前缀,例如 `sk-********`;明文显示只在有权限的配置页面内按需触发。
工具 key 使用同样规则。WebSearch、OCR 这类工具必须先匹配当前工具和 provider再决定能否使用 fallback。
### 轻量连通性测试
Admin Next 的插头按钮走轻量连通性测试,不承担保存职责。业界常见做法是分两层:
- 快速检查 provider 目录或低成本 endpoint确认 base URL、鉴权和当前模型是否可达。
- 只有在用户明确运行 Playground 或业务任务时才发完整模型请求。
因此,连通性测试应尽量使用低成本请求,并返回明确状态:
- `ok`: 鉴权、路由和模型目录可用。
- `warning`: 服务可达,但当前模型不在目录或能力声明不完整。
- `error`: 鉴权失败、网络失败、协议错误或模型不可用。
错误 toast 标题必须和结果一致,不能在失败时显示“连通性正常”。
### OpenCode Go 路由模型
OpenCode Go 这类订阅通道不要靠前端硬编码模型集合判断协议。推荐在 provider catalog 或后端能力发现中记录每个模型的协议能力,例如 `chat_completions``anthropic_messages``models_endpoint` 和是否需要订阅 key。前端只展示能力结果后端负责把 provider、base URL、model 和协议适配映射为实际请求。
#### 配置页行为
- Provider 下拉框决定当前默认 provider。
@@ -287,7 +343,6 @@ SERVICE_VERSION=0.1.0
AI_PROVIDER_SERVICE_TOKEN=change_me
AI_TIMEOUT_SECONDS=60
AI_HTTP_RETRY_ATTEMPTS=2
AI_ANALYSIS_SYSTEM_PROMPT=你是态势感知分析助手。请基于输入的上下文、观测与约束,输出结构化、克制、可执行的分析。
```
可选 provider 专属 key

Some files were not shown because too many files have changed in this diff Show More