Compare commits
13 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
69789d7505 | ||
|
|
4f124121e7 | ||
|
|
085bdf9a80 | ||
|
|
fbca381512 | ||
|
|
5c65ee24d6 | ||
|
|
81970a1d05 | ||
|
|
9b913a3b83 | ||
|
|
93eb41a9f7 | ||
|
|
dd176a6ae6 | ||
|
|
f14ff6ec0f | ||
|
|
39854b9983 | ||
|
|
3b4347c87d | ||
|
|
d9efd98d26 |
@@ -73,6 +73,7 @@ Style:
|
||||
- Use fenced code blocks with language tags.
|
||||
- Prefer tables for comparisons or parameter lists.
|
||||
- Keep snippets concise and relevant.
|
||||
- For UI labels, chart labels, feature names, datasource names, and other terms that may become mixed Chinese/English copy, check `docs/technical/{zh,en}/naming-glossary.md` and use the documented display name. If a confusing term is missing, update the glossary in both languages as part of the docs change.
|
||||
|
||||
### Step 4 — Verify
|
||||
|
||||
|
||||
@@ -52,6 +52,7 @@ rg -n "class |def |function |export |router|@router|interface |type " <path>
|
||||
- Keep code snippets short and directly relevant.
|
||||
- List related files only when they help future maintainers navigate.
|
||||
- Use the repository’s existing language, heading style, and naming conventions.
|
||||
- For UI labels, chart labels, feature names, datasource names, and other terms that may become mixed Chinese/English copy, check `docs/technical/{zh,en}/naming-glossary.md` and use the documented display name. If a confusing term is missing, update the glossary in both languages as part of the docs change.
|
||||
|
||||
4. Verify:
|
||||
|
||||
|
||||
@@ -2,9 +2,14 @@
|
||||
|
||||
!pyproject.toml
|
||||
!uv.lock
|
||||
!VERSION
|
||||
!backend/
|
||||
!backend/**
|
||||
!aiprovider/
|
||||
!aiprovider/**
|
||||
|
||||
backend/.env
|
||||
backend/.env.*
|
||||
aiprovider/.env
|
||||
aiprovider/.env.*
|
||||
!aiprovider/.env.example
|
||||
|
||||
64
.gitea/workflows/ci.yaml
Normal file
64
.gitea/workflows/ci.yaml
Normal file
@@ -0,0 +1,64 @@
|
||||
name: ci
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- dev
|
||||
- main
|
||||
pull_request:
|
||||
|
||||
env:
|
||||
REGISTRY: gitea.rclaw.top
|
||||
IMAGE_NAMESPACE: linkong/planet
|
||||
|
||||
jobs:
|
||||
backend:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Install uv
|
||||
run: curl -LsSf https://astral.sh/uv/install.sh | sh
|
||||
- name: Sync Python dependencies
|
||||
run: ~/.local/bin/uv sync --group dev
|
||||
- name: Run backend smoke tests
|
||||
working-directory: backend
|
||||
run: PYTHONPATH=. "$GITHUB_WORKSPACE/.venv/bin/python" -m pytest -s tests/test_api.py tests/test_realtime_sources.py -q
|
||||
|
||||
frontend:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Install Bun
|
||||
run: curl -fsSL https://bun.sh/install | bash
|
||||
- name: Build frontend
|
||||
working-directory: frontend
|
||||
run: |
|
||||
~/.bun/bin/bun install --frozen-lockfile
|
||||
~/.bun/bin/bun run build
|
||||
|
||||
delivery:
|
||||
runs-on: ubuntu-latest
|
||||
needs:
|
||||
- backend
|
||||
- frontend
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Install Helm
|
||||
run: |
|
||||
mkdir -p "$HOME/.local/bin"
|
||||
curl -fsSL https://get.helm.sh/helm-v3.15.4-linux-amd64.tar.gz -o /tmp/helm.tar.gz
|
||||
tar -xzf /tmp/helm.tar.gz -C /tmp
|
||||
mv /tmp/linux-amd64/helm "$HOME/.local/bin/helm"
|
||||
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
|
||||
- name: Docker build smoke
|
||||
run: |
|
||||
docker build -t "$REGISTRY/$IMAGE_NAMESPACE/frontend:${GITHUB_SHA}" ./frontend
|
||||
docker build -t "$REGISTRY/$IMAGE_NAMESPACE/backend:${GITHUB_SHA}" -f backend/Dockerfile .
|
||||
docker build -t "$REGISTRY/$IMAGE_NAMESPACE/aiprovider:${GITHUB_SHA}" -f aiprovider/Dockerfile .
|
||||
- name: Helm template smoke
|
||||
run: |
|
||||
helm lint deploy/helm/planet
|
||||
helm template planet-staging deploy/helm/planet \
|
||||
--namespace planet-staging \
|
||||
-f deploy/helm/planet/values.single-node.yaml \
|
||||
--set image.tag="${GITHUB_SHA}" >/tmp/planet-rendered.yaml
|
||||
67
.gitea/workflows/deploy-staging.yaml
Normal file
67
.gitea/workflows/deploy-staging.yaml
Normal file
@@ -0,0 +1,67 @@
|
||||
name: deploy-staging
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
push:
|
||||
branches:
|
||||
- main
|
||||
|
||||
env:
|
||||
REGISTRY: gitea.rclaw.top
|
||||
IMAGE_NAMESPACE: linkong/planet
|
||||
RELEASE_NAME: planet-staging
|
||||
NAMESPACE: planet-staging
|
||||
|
||||
jobs:
|
||||
deploy:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Install deploy tools
|
||||
run: |
|
||||
mkdir -p "$HOME/.local/bin"
|
||||
curl -fsSL https://get.helm.sh/helm-v3.15.4-linux-amd64.tar.gz -o /tmp/helm.tar.gz
|
||||
tar -xzf /tmp/helm.tar.gz -C /tmp
|
||||
mv /tmp/linux-amd64/helm "$HOME/.local/bin/helm"
|
||||
curl -fsSL https://dl.k8s.io/release/v1.30.5/bin/linux/amd64/kubectl -o /tmp/kubectl
|
||||
install -m 0755 /tmp/kubectl "$HOME/.local/bin/kubectl"
|
||||
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
|
||||
- name: Configure kubeconfig
|
||||
run: |
|
||||
mkdir -p "$HOME/.kube"
|
||||
printf "%s" "${{ secrets.KUBE_CONFIG_STAGING }}" | base64 -d > "$HOME/.kube/config"
|
||||
chmod 600 "$HOME/.kube/config"
|
||||
- name: Deploy Helm release
|
||||
run: |
|
||||
kubectl create namespace "$NAMESPACE" --dry-run=client -o yaml | kubectl apply -f -
|
||||
helm upgrade --install "$RELEASE_NAME" deploy/helm/planet \
|
||||
--namespace "$NAMESPACE" \
|
||||
-f deploy/helm/planet/values.single-node.yaml \
|
||||
--set global.imageRegistry="$REGISTRY" \
|
||||
--set global.imageNamespace="$IMAGE_NAMESPACE" \
|
||||
--set image.tag="${GITHUB_SHA}"
|
||||
- name: Wait for rollout
|
||||
run: |
|
||||
kubectl rollout status deployment/planet-frontend -n "$NAMESPACE" --timeout=180s
|
||||
kubectl rollout status deployment/planet-backend -n "$NAMESPACE" --timeout=180s
|
||||
kubectl rollout status deployment/planet-aiprovider -n "$NAMESPACE" --timeout=180s
|
||||
- name: Smoke test services
|
||||
run: |
|
||||
kubectl run planet-smoke-${GITHUB_RUN_NUMBER} \
|
||||
--rm -i --restart=Never \
|
||||
--namespace "$NAMESPACE" \
|
||||
--image=curlimages/curl:8.11.1 \
|
||||
--command -- sh -c '
|
||||
set -eu
|
||||
curl -fsS http://planet-frontend:3000/ >/dev/null
|
||||
curl -fsS http://planet-frontend:3000/health >/dev/null
|
||||
curl -fsS http://planet-frontend:3000/api/health >/dev/null
|
||||
curl -fsS http://planet-backend:8000/health >/dev/null
|
||||
curl -fsS http://planet-aiprovider:8010/health >/dev/null
|
||||
'
|
||||
- name: Collect diagnostics on failure
|
||||
if: failure()
|
||||
run: |
|
||||
kubectl get all -n "$NAMESPACE" -o wide || true
|
||||
kubectl describe pods -n "$NAMESPACE" || true
|
||||
kubectl logs -n "$NAMESPACE" -l app.kubernetes.io/instance="$RELEASE_NAME" --all-containers --tail=200 || true
|
||||
58
.gitea/workflows/release.yaml
Normal file
58
.gitea/workflows/release.yaml
Normal file
@@ -0,0 +1,58 @@
|
||||
name: release
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- main
|
||||
tags:
|
||||
- "v*"
|
||||
workflow_dispatch:
|
||||
|
||||
env:
|
||||
REGISTRY: gitea.rclaw.top
|
||||
IMAGE_NAMESPACE: linkong/planet
|
||||
|
||||
jobs:
|
||||
images:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Resolve image tags
|
||||
id: meta
|
||||
run: |
|
||||
echo "sha_tag=${GITHUB_SHA}" >> "$GITHUB_OUTPUT"
|
||||
if printf "%s" "${GITHUB_REF}" | grep -q '^refs/tags/v'; then
|
||||
echo "release_tag=${GITHUB_REF_NAME}" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo "release_tag=" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
- name: Login to registry
|
||||
run: echo "${{ secrets.REGISTRY_PASSWORD }}" | docker login "$REGISTRY" -u "${{ secrets.REGISTRY_USER }}" --password-stdin
|
||||
- name: Build and push images
|
||||
run: |
|
||||
for service in frontend backend aiprovider; do
|
||||
case "$service" in
|
||||
frontend)
|
||||
dockerfile="./frontend/Dockerfile"
|
||||
context="./frontend"
|
||||
;;
|
||||
backend)
|
||||
dockerfile="backend/Dockerfile"
|
||||
context="."
|
||||
;;
|
||||
aiprovider)
|
||||
dockerfile="aiprovider/Dockerfile"
|
||||
context="."
|
||||
;;
|
||||
esac
|
||||
|
||||
image="$REGISTRY/$IMAGE_NAMESPACE/$service:${{ steps.meta.outputs.sha_tag }}"
|
||||
docker build -t "$image" -f "$dockerfile" "$context"
|
||||
docker push "$image"
|
||||
|
||||
if [ -n "${{ steps.meta.outputs.release_tag }}" ]; then
|
||||
release_image="$REGISTRY/$IMAGE_NAMESPACE/$service:${{ steps.meta.outputs.release_tag }}"
|
||||
docker tag "$image" "$release_image"
|
||||
docker push "$release_image"
|
||||
fi
|
||||
done
|
||||
7
.gitignore
vendored
7
.gitignore
vendored
@@ -8,6 +8,7 @@
|
||||
.env
|
||||
.env.local
|
||||
.env.*.local
|
||||
config/earth-boundary-sources.local.json
|
||||
*.pem
|
||||
*.key
|
||||
*.crt
|
||||
@@ -150,3 +151,9 @@ temp/
|
||||
# Runtime Data
|
||||
# ----------------------
|
||||
data/ai/bgp-briefs/
|
||||
data/earth-boundary-sources/
|
||||
|
||||
# Generated Earth boundary tile artifacts. Keep source configs and builders in
|
||||
# Git; publish PMTiles/MVT artifacts through release/deploy storage instead of
|
||||
# committing thousands of generated tile files.
|
||||
frontend/public/earth/data/boundaries/
|
||||
|
||||
262
README.md
262
README.md
@@ -8,68 +8,54 @@
|
||||
|
||||
## 系统架构
|
||||
|
||||
当前仓库的核心形态是“Web Earth 可视化 + React 运维台 + FastAPI 数据与 AI 编排后端 + 独立模型适配层”。物理大屏与 UE 客户端仍是长期方向,但不再作为本地开发和当前发布的必需运行单元。
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────┐
|
||||
│ 物理大屏展示层 │
|
||||
│ ┌─────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ 偏振片3D大屏 (2m×3m, 4K, 120Hz, 眼镜式) │ │
|
||||
│ │ ┌─────────────────────────────────────────────────────────┐ │ │
|
||||
│ │ │ 虚幻引擎 UE5 客户端 │ │ │
|
||||
│ │ │ ├── 3D地球渲染 (Cesium for UE) │ │ │
|
||||
│ │ │ ├── 算力点可视化 (GPU集群、智算中心) │ │ │
|
||||
│ │ │ ├── 连接弧线 (光缆、路由、数据流向) │ │ │
|
||||
│ │ │ ├── 粒子效果 (数据流动、告警提示) │ │ │
|
||||
│ │ │ └── 自动巡航相机 + 交互控制 │ │ │
|
||||
│ │ └─────────────────────────────────────────────────────────┘ │ │
|
||||
│ └─────────────────────────────────────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────────────┘
|
||||
▲
|
||||
│ WebSocket (实时推送)
|
||||
│ 120Hz 心跳 / 数据帧同步
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────────────┐
|
||||
│ 数据中台服务层 (FastAPI) │
|
||||
│ ┌─────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ API Gateway (Redis 限流) │ │
|
||||
│ └─────────────────────────────────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌───────────────────┬──────────────────────────┬──────────────────┐ │
|
||||
│ │ 数据采集服务 │ 核心业务服务 │ 运维管理服务 │ │
|
||||
│ │ ┌─────────────┐ │ ┌─────────────────┐ │ ┌─────────────┐ │ │
|
||||
│ │ │ 调度中心 │ │ │ WebSocket 服务 │ │ │ 用户管理 │ │ │
|
||||
│ │ │ (Celery) │ │ │ (FastAPI) │ │ │ (JWT Auth) │ │ │
|
||||
│ │ └─────────────┘ │ └─────────────────┘ │ └─────────────┘ │ │
|
||||
│ │ ┌─────────────┐ │ ┌─────────────────┐ │ ┌─────────────┐ │ │
|
||||
│ │ │ 采集器池 │ │ │ 数据查询 API │ │ │ 数据源配置 │ │ │
|
||||
│ │ │ (10+源) │ │ │ (REST) │ │ │ 监控告警 │ │ │
|
||||
│ │ └─────────────┘ │ └─────────────────┘ │ └─────────────┘ │ │
|
||||
│ │ ┌─────────────┐ │ ┌─────────────────┐ │ ┌─────────────┐ │ │
|
||||
│ │ │ 消息队列 │ │ │ 态势分析引擎 │ │ │ 系统配置 │ │ │
|
||||
│ │ │ (Kafka) │ │ │ (计算/聚合) │ │ │ 日志审计 │ │ │
|
||||
│ │ └─────────────┘ │ └─────────────────┘ │ └─────────────┘ │ │
|
||||
│ └───────────────────┴──────────────────────────┴──────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────────────┘
|
||||
▲
|
||||
│ 内部 API 调用
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────────────┐
|
||||
│ Web管理端 (React Admin) │
|
||||
│ ┌─────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ 登录页 │ 仪表盘 │ 用户管理 │ 数据源配置 │ 任务监控 │ 系统配置 │ │
|
||||
│ └─────────────────────────────────────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────────────┘
|
||||
▲
|
||||
│ PostgreSQL / Redis
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────────────┐
|
||||
│ 数据存储层 │
|
||||
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
|
||||
│ │ PostgreSQL │ │ TimescaleDB │ │ Redis │ │ MinIO │ │
|
||||
│ │ (用户/配置) │ │ (时序数据) │ │ (缓存/会话) │ │ (文件存储) │ │
|
||||
│ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────────────┘
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ 浏览器展示与运维层 │
|
||||
│ ┌──────────────────────────────┐ ┌──────────────────────────────┐ │
|
||||
│ │ Web Earth │ │ React 运维台 │ │
|
||||
│ │ frontend/public/earth │ │ frontend/src │ │
|
||||
│ │ Three.js 地球 / HUD / 新闻 │ │ 数据源 / 告警 / AI 设置 │ │
|
||||
│ │ 国界精度 / 品牌内容配置 │ │ 提示词配置 / 用户与系统配置 │ │
|
||||
│ └──────────────────────────────┘ └──────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
│ REST / WebSocket
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ FastAPI 业务与编排后端 │
|
||||
│ ┌────────────────────┐ ┌────────────────────┐ ┌─────────────────┐ │
|
||||
│ │ 数据 API 与认证 │ │ Earth 新闻增强 │ │ 告警与态势简报 │ │
|
||||
│ │ JWT / 权限 / 审计 │ │ 位置推断 / 本地化 │ │ BGP / 告警研判 │ │
|
||||
│ └────────────────────┘ └────────────────────┘ └─────────────────┘ │
|
||||
│ ┌────────────────────┐ ┌────────────────────┐ ┌─────────────────┐ │
|
||||
│ │ 系统运行配置 │ │ 默认提示词注册表 │ │ 未来 Agent Runtime│ │
|
||||
│ │ system_settings │ │ 代码发布 + DB 覆盖 │ │ 工具/证据/工作流 │ │
|
||||
│ └────────────────────┘ └────────────────────┘ └─────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
│ SQLAlchemy / Redis Stream │ 纯净 LLM 调用
|
||||
▼ ▼
|
||||
┌──────────────────────────────┐ ┌──────────────────────────────┐
|
||||
│ PostgreSQL / Redis │ │ aiprovider │
|
||||
│ 用户、配置、采集结果、新闻 │ │ provider + protocol adapter │
|
||||
│ Stream、缓存、运行状态 │ │ OpenAI / MiniMax / Ollama 等 │
|
||||
└──────────────────────────────┘ └──────────────────────────────┘
|
||||
▲
|
||||
│ 采集器 / 外部数据源
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ RSS 新闻、BGP 观测、公开数据源、后续 WebSearch/OCR/语音识别等工具 │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
架构边界:
|
||||
|
||||
- `backend` 负责业务语义、证据收集、提示词选择、AI 任务编排、权限和数据落库。
|
||||
- `aiprovider` 只负责把纯净模型请求适配到不同供应商或协议,不内置具体业务提示词。
|
||||
- 默认提示词随代码发布并保存在 `backend/app/ai_tasks/default_prompts.json`,运维台可在数据库中保存覆盖值,重置时回到当前代码版本的默认提示词。
|
||||
- Earth 新闻保留英文原文,中文展示结果存入 `localizations`,前端默认展示 `zh-CN` 的 `display_title`、`display_summary` 和中文地域/状态文案。
|
||||
- Earth LLM 指令、语音识别、多角色态势研判属于后续 Agent Runtime 方向,计划见 [docs/plans/agents-earth-command-runtime-plan.md](/home/ray/dev/linkong/planet/docs/plans/agents-earth-command-runtime-plan.md)。
|
||||
|
||||
## 四大核心要素
|
||||
|
||||
| 层级 | 要素 | 描述 |
|
||||
@@ -87,11 +73,10 @@
|
||||
|------|------|------|
|
||||
| FastAPI | 0.109+ | Web 框架 |
|
||||
| SQLAlchemy | 2.0+ | ORM |
|
||||
| Alembic | - | 数据库迁移 |
|
||||
| Celery | 5.3+ | 任务队列 |
|
||||
| Redis | 7.0+ | 缓存/消息 |
|
||||
| Kafka | 3.0+ | 事件流 |
|
||||
| uv | - | Python 依赖与命令运行 |
|
||||
| Redis | 7.0+ | 缓存、Stream 与运行协调 |
|
||||
| PyJWT | - | 认证 |
|
||||
| APScheduler / 后台任务 | - | 采集、增强与运行时任务 |
|
||||
|
||||
### 前端 (React Admin)
|
||||
|
||||
@@ -102,6 +87,7 @@
|
||||
| Axios | HTTP 客户端 |
|
||||
| Socket.io-client | WebSocket 客户端 |
|
||||
| ECharts | 统计图表 |
|
||||
| Three.js | Earth 3D 地球渲染 |
|
||||
| Bun | 前端包管理与脚本运行 |
|
||||
|
||||
前端工程统一使用 Bun:
|
||||
@@ -110,22 +96,16 @@
|
||||
- 运行脚本使用 `bun run <script>`
|
||||
- 不使用 `npm`、`pnpm`、`yarn`
|
||||
|
||||
### 虚幻引擎客户端
|
||||
### 大屏与 3D 展示方向
|
||||
|
||||
| 组件 | 版本 | 用途 |
|
||||
|------|------|------|
|
||||
| Unreal Engine 5 | 5.3+ | 3D 渲染引擎 |
|
||||
| Cesium for Unreal | 1.5+ | 地理可视化 |
|
||||
| Niagara | - | 粒子系统 |
|
||||
当前发布优先使用浏览器 Web Earth。UE5 / Cesium for Unreal / Niagara 可作为后续物理大屏方向接入,但不是本地开发闭环的必需组件。
|
||||
|
||||
### 数据库
|
||||
|
||||
| 组件 | 用途 |
|
||||
|------|------|
|
||||
| PostgreSQL 15+ | 关系数据 |
|
||||
| TimescaleDB | 时序数据扩展 |
|
||||
| Redis 7+ | 缓存/会话 |
|
||||
| MinIO | S3 兼容存储 |
|
||||
| Redis 7+ | 缓存、Stream、运行状态 |
|
||||
|
||||
### 部署
|
||||
|
||||
@@ -152,9 +132,9 @@
|
||||
| P0 | Epoch AI | 每小时 |
|
||||
| P0 | Hugging Face | 每 2 小时 |
|
||||
| P0 | GitHub | 每 4 小时 |
|
||||
| P0 每日 |
|
||||
| P0 | 海底光缆 / IXP / 卫星等基础设施数据 | 每日或按源刷新 |
|
||||
| P0 | PeeringDB | 每 2 小时 |
|
||||
| P1 | Cloudflare Radar | | TeleGeography | 每小时 |
|
||||
| P1 | Cloudflare Radar / TeleGeography | 每小时 |
|
||||
| P1 | CAIDA BGPStream | 每 15 分钟 |
|
||||
|
||||
## 项目结构
|
||||
@@ -166,20 +146,18 @@
|
||||
│ │ ├── core/ # 核心配置
|
||||
│ │ ├── models/ # 数据模型
|
||||
│ │ ├── schemas/ # Pydantic 模型
|
||||
│ │ ├── services/ # 业务逻辑
|
||||
│ │ └── tasks/ # Celery 任务
|
||||
│ │ ├── services/ # 业务逻辑与 AI 任务编排
|
||||
│ │ └── ai_tasks/ # 默认提示词与 AI 任务定义
|
||||
│ └── tests/
|
||||
├── aiprovider/ # 独立模型供应商适配层
|
||||
├── frontend/ # React 管理后台
|
||||
│ ├── src/
|
||||
│ │ ├── components/ # 组件
|
||||
│ │ ├── pages/ # 页面
|
||||
│ │ ├── services/ # API 服务
|
||||
│ │ └── store/ # 状态管理
|
||||
│ └── tests/
|
||||
├── unreal/ # UE5 大屏客户端
|
||||
│ ├── Content/
|
||||
│ ├── Source/
|
||||
│ └── Plugins/
|
||||
│ ├── public/earth/ # Web Earth 静态应用
|
||||
│ └── tests/ # 前端测试
|
||||
├── data/ # 数据文件
|
||||
├── docs/ # 文档
|
||||
├── scripts/ # 脚本
|
||||
@@ -191,10 +169,11 @@
|
||||
## 快速启动
|
||||
|
||||
```bash
|
||||
# 新机器首次初始化
|
||||
./scripts/bootstrap-dev.sh
|
||||
# 会自动安装/检查 uv、bun,并同步 Python/前端依赖
|
||||
# 新机器或空项目首次初始化
|
||||
./planet.sh init
|
||||
# 会自动安装/检查 uv、bun,同步 Python/前端依赖
|
||||
# 会在缺少时生成 backend/.env、aiprovider/.env、frontend/.env.local
|
||||
# 会启动 PostgreSQL/Redis,并创建表、默认数据源和本地默认用户
|
||||
|
||||
# 启动前后端服务
|
||||
./planet.sh start
|
||||
@@ -210,6 +189,9 @@
|
||||
|
||||
# 查看服务状态
|
||||
./planet.sh health
|
||||
|
||||
# 删除容器、卷、镜像和本地编译状态,执行前需要输入 Y 确认
|
||||
./planet.sh destroy
|
||||
```
|
||||
|
||||
前端命令约定:
|
||||
@@ -244,7 +226,7 @@ bun run build
|
||||
./planet.sh start --allow-lan
|
||||
```
|
||||
|
||||
这会让前端监听 `0.0.0.0:3000`,后端监听 `0.0.0.0:8000`。
|
||||
这会让前端监听 `0.0.0.0:3000`,后端监听 `0.0.0.0:8000`,AI Provider 通过 Docker 发布到 `0.0.0.0:8010`。启动前脚本会检查这三个端口;如果 WSL/Linux 侧无法释放端口,并检测到 Windows 侧 listener 或旧 `portproxy`,会请求管理员 PowerShell 清理。
|
||||
|
||||
### 2. 先确认 WSL 内部服务正常
|
||||
|
||||
@@ -253,14 +235,16 @@ bun run build
|
||||
```bash
|
||||
curl http://localhost:3000
|
||||
curl http://localhost:8000/health
|
||||
ss -ltnp | grep -E ':3000|:8000'
|
||||
curl http://localhost:8010/health
|
||||
ss -ltnp | grep -E ':3000|:8000|:8010'
|
||||
```
|
||||
|
||||
预期:
|
||||
|
||||
- `3000` 返回前端 HTML
|
||||
- `8000/health` 返回健康检查 JSON
|
||||
- `ss` 中能看到 `0.0.0.0:3000` 和 `0.0.0.0:8000`
|
||||
- `8010/health` 返回 AI Provider 健康检查 JSON
|
||||
- `ss` 中能看到 `0.0.0.0:3000`、`0.0.0.0:8000` 和 `0.0.0.0:8010`,或 Docker 已发布 `8010`
|
||||
|
||||
如果这一步不通,先不要继续做 Windows 转发。
|
||||
|
||||
@@ -271,42 +255,31 @@ ss -ltnp | grep -E ':3000|:8000'
|
||||
```powershell
|
||||
curl http://localhost:3000
|
||||
curl http://localhost:8000/health
|
||||
curl http://localhost:8010/health
|
||||
```
|
||||
|
||||
在常见的 WSL2 开发环境下,Windows 通常可以直接通过 `localhost` 访问 WSL 中的服务。
|
||||
|
||||
### 4. 如果需要让局域网设备访问,再做 Windows 端口转发
|
||||
### 4. 如果需要让局域网设备访问,清理端口和防火墙
|
||||
|
||||
注意:下面的命令必须在“以管理员身份运行”的 PowerShell 中执行。
|
||||
`./planet.sh start --allow-lan` 不再启动额外的 Windows 端口转发进程。它直接让开发服务对 `3000` / `8000` / `8010` 开放,并在启动前尝试释放这些端口。端口被 Windows 侧 listener 或旧 `portproxy` 占用时,脚本会请求一次管理员 PowerShell 清理。
|
||||
|
||||
先把 Windows 对外网卡上的 `3000` / `8000` 转发到 Windows 本机 `127.0.0.1`:
|
||||
如果以前手动配置过持久 `portproxy`,若自动请求被取消,可以手动清理,避免 `iphlpsvc` 继续占用端口:
|
||||
|
||||
```powershell
|
||||
netsh interface portproxy delete v4tov4 listenaddress=0.0.0.0 listenport=3000
|
||||
netsh interface portproxy delete v4tov4 listenaddress=0.0.0.0 listenport=8000
|
||||
|
||||
netsh interface portproxy add v4tov4 listenaddress=0.0.0.0 listenport=3000 connectaddress=127.0.0.1 connectport=3000
|
||||
netsh interface portproxy add v4tov4 listenaddress=0.0.0.0 listenport=8000 connectaddress=127.0.0.1 connectport=8000
|
||||
netsh interface portproxy delete v4tov4 listenaddress=0.0.0.0 listenport=8010
|
||||
```
|
||||
|
||||
再放行 Windows 防火墙:
|
||||
脚本会检测 Windows 防火墙是否已放行 `3000` / `8000` / `8010`。如果缺少规则,会触发一次 Windows UAC 管理员 PowerShell 请求来自动创建。若自动请求被取消,也可以手动执行:
|
||||
|
||||
```powershell
|
||||
New-NetFirewallRule -DisplayName "WSL Planet 3000" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 3000
|
||||
New-NetFirewallRule -DisplayName "WSL Planet 8000" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 8000
|
||||
New-NetFirewallRule -DisplayName "WSL Planet 8010" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 8010
|
||||
```
|
||||
|
||||
检查转发规则是否生效:
|
||||
|
||||
```powershell
|
||||
netsh interface portproxy show all
|
||||
```
|
||||
|
||||
预期能看到:
|
||||
|
||||
- `0.0.0.0:3000 -> 127.0.0.1:3000`
|
||||
- `0.0.0.0:8000 -> 127.0.0.1:8000`
|
||||
|
||||
### 5. 查 Windows 局域网 IP,并让其他设备访问
|
||||
|
||||
在 Windows PowerShell 中执行:
|
||||
@@ -321,6 +294,8 @@ ipconfig
|
||||
|
||||
- `http://<Windows局域网IP>:3000/earth`
|
||||
- `http://<Windows局域网IP>:3000/admin`
|
||||
- `http://<Windows局域网IP>:8000/health`
|
||||
- `http://<Windows局域网IP>:8010/health`
|
||||
|
||||
例如:
|
||||
|
||||
@@ -329,7 +304,7 @@ ipconfig
|
||||
### 6. 常见现象与判断
|
||||
|
||||
- WSL 中 `curl localhost:3000` 能通,但 Windows 访问 `WSL 的局域网 IP:3000` 不通:这是正常现象之一,优先验证 Windows 的 `localhost:3000`
|
||||
- Windows `localhost:3000` 能通,但局域网设备访问 `Windows 局域网 IP:3000` 不通:通常缺少 `portproxy` 或防火墙放行
|
||||
- Windows `localhost:3000` 能通,但局域网设备访问 `Windows 局域网 IP:3000` 不通:通常是 Windows 防火墙、网络配置或旧 `portproxy` 残留
|
||||
- `whoami /groups` 中 `S-1-5-32-544` 显示 `deny only`:说明当前 PowerShell 不是提权管理员窗口
|
||||
|
||||
### 7. 本项目一次性验证顺序
|
||||
@@ -338,10 +313,12 @@ ipconfig
|
||||
|
||||
1. WSL 中执行 `curl http://localhost:3000`
|
||||
2. WSL 中执行 `curl http://localhost:8000/health`
|
||||
3. Windows 中执行 `curl http://localhost:3000`
|
||||
4. Windows 中执行 `curl http://localhost:8000/health`
|
||||
5. 管理员 PowerShell 配置 `portproxy` 和防火墙
|
||||
6. 用手机或其他电脑访问 `http://<Windows局域网IP>:3000/earth`
|
||||
3. WSL 中执行 `curl http://localhost:8010/health`
|
||||
4. Windows 中执行 `curl http://localhost:3000`
|
||||
5. Windows 中执行 `curl http://localhost:8000/health`
|
||||
6. Windows 中执行 `curl http://localhost:8010/health`
|
||||
7. 按脚本提示完成 Windows 防火墙或端口清理 UAC 请求
|
||||
8. 用手机或其他电脑访问 Windows 对外端口,例如 `http://<Windows局域网IP>:3000/earth`
|
||||
|
||||
## 启动容错参数
|
||||
|
||||
@@ -365,18 +342,27 @@ DATABASE_RETRY_INTERVAL=10 \
|
||||
- `AI_PROVIDER_START_MAX_RETRIES` / `AI_PROVIDER_RETRY_INTERVAL`: 控制 `aiprovider` 的构建/启动与容器重启自愈,默认 `3` 次、`5` 秒
|
||||
- `BACKEND_MAX_RETRIES`: 控制后端进程启动重试次数,默认 `3`
|
||||
- `FRONTEND_MAX_RETRIES`: 控制前端 dev server 启动重试次数,默认 `3`
|
||||
- `BACKEND_HEALTH_CHECK_ATTEMPTS` / `BACKEND_HEALTH_CHECK_INTERVAL`: 控制后端 HTTP 健康检查等待次数与间隔,默认 `10` 次、`2` 秒
|
||||
- `BACKEND_HEALTH_CHECK_ATTEMPTS` / `BACKEND_HEALTH_CHECK_INTERVAL`: 控制后端 HTTP 健康检查等待次数与间隔,默认 `60` 次、`2` 秒
|
||||
- `FRONTEND_HEALTH_CHECK_ATTEMPTS` / `FRONTEND_HEALTH_CHECK_INTERVAL`: 控制前端 HTTP 可访问检查等待次数与间隔,默认 `10` 次、`2` 秒
|
||||
- `AI_PROVIDER_HEALTH_CHECK_ATTEMPTS` / `AI_PROVIDER_HEALTH_CHECK_INTERVAL`: 控制 `aiprovider` HTTP 健康检查等待次数与间隔,默认 `10` 次、`2` 秒
|
||||
|
||||
## AI 接口预留
|
||||
## AI 与智能体接口
|
||||
|
||||
项目现在采用“两层”设计:
|
||||
项目现在采用“三段式”边界:
|
||||
|
||||
- 主后端暴露稳定业务接口: `GET /api/v1/ai/provider/status`、`POST /api/v1/ai/situational-awareness/analyze`
|
||||
- 独立 `aiprovider` 服务负责适配具体模型供应商
|
||||
- `backend`: 暴露业务接口,负责选择任务提示词、组织证据、调用工具、保存 AI 设置和结果。
|
||||
- `aiprovider`: 暴露模型网关接口,只负责 provider / protocol 适配,不写入 BGP、新闻、告警等业务提示词。
|
||||
- 模型供应商: OpenAI 兼容、MiniMax、Anthropic、Ollama 或其他兼容网关。
|
||||
|
||||
这样前端和业务代码不直接依赖 OpenAI、本地模型网关或其他订阅服务,后续切换部署方式只需要调整环境变量。
|
||||
这样前端和业务代码不直接依赖某个模型供应商,后续增加 Agent Runtime、Earth 一键 LLM 指令、语音识别或多角色态势研判时,也可以把业务工作流放在后端,而不是污染模型适配层。
|
||||
|
||||
当前已落地的 AI 配置能力:
|
||||
|
||||
- 运维台 AI 设置可维护 provider、模型、协议、超时、token 等运行配置。
|
||||
- 运维台 AI 设置中的“提示词”页可选择不同功能入口,手动覆盖提示词,并一键重置到默认值。
|
||||
- 默认提示词随代码发布,位于 [backend/app/ai_tasks/default_prompts.json](/home/ray/dev/linkong/planet/backend/app/ai_tasks/default_prompts.json)。
|
||||
- 覆盖值保存在数据库运行配置中,升级代码后可继续保留现场配置,也可重置到新版本默认提示词。
|
||||
- 态势摘要、告警研判、新闻本地化等入口应使用各自任务提示词;调用 `aiprovider` 时只传递当前任务所需的 `prompt` / `system_prompt`。
|
||||
|
||||
主后端建议配置:
|
||||
|
||||
@@ -389,39 +375,33 @@ AI_PROVIDER_TIMEOUT_SECONDS=60
|
||||
`aiprovider` 服务建议配置:
|
||||
|
||||
```env
|
||||
AI_PROVIDER=openai_compatible
|
||||
AI_BASE_URL=https://api.openai.com/v1
|
||||
AI_PROVIDER=minimax
|
||||
AI_PROVIDER_API=anthropic-messages
|
||||
AI_BASE_URL=https://api.minimaxi.com/anthropic
|
||||
AI_API_KEY=your_api_key
|
||||
AI_MODEL=gpt-4o-mini
|
||||
AI_MODEL=MiniMax-M2.7
|
||||
AI_TIMEOUT_SECONDS=60
|
||||
AI_PROVIDER_SERVICE_TOKEN=change_me
|
||||
```
|
||||
|
||||
OpenAI 兼容场景推荐使用:
|
||||
推荐映射关系:
|
||||
|
||||
- `AI_PROVIDER=openai_compatible`
|
||||
- `vLLM` / `LM Studio` / `One API`: `AI_PROVIDER=openai` + `AI_PROVIDER_API=openai-completions`
|
||||
- `MiniMax`: `AI_PROVIDER=minimax` + `AI_PROVIDER_API=anthropic-messages`
|
||||
- Claude 兼容网关: `AI_PROVIDER=anthropic` + `AI_PROVIDER_API=anthropic-messages`
|
||||
- `Ollama`: `AI_PROVIDER=ollama` + `AI_PROVIDER_API=ollama-generate`
|
||||
|
||||
Claude 兼容场景推荐使用:
|
||||
|
||||
- `AI_PROVIDER=anthropic`
|
||||
- `AI_PROVIDER=anthropic_compatible`
|
||||
- `AI_PROVIDER=claude_compatible`
|
||||
|
||||
Ollama 原生场景推荐使用:
|
||||
|
||||
- `AI_PROVIDER=ollama`
|
||||
|
||||
比如 MiniMax 或其他 Claude 兼容网关,可以这样配置:
|
||||
比如 MiniMax 可以这样配置:
|
||||
|
||||
```env
|
||||
AI_PROVIDER=claude_compatible
|
||||
AI_BASE_URL=https://your-claude-compatible-endpoint.example.com
|
||||
AI_PROVIDER=minimax
|
||||
AI_PROVIDER_API=anthropic-messages
|
||||
AI_BASE_URL=https://api.minimaxi.com/anthropic
|
||||
AI_API_KEY=your_api_key
|
||||
AI_MODEL=your-claude-compatible-model
|
||||
AI_MODEL=MiniMax-M2.7
|
||||
AI_TIMEOUT_SECONDS=60
|
||||
AI_MAX_TOKENS=1200
|
||||
AI_ANTHROPIC_VERSION=2023-06-01
|
||||
AI_PROVIDER_SERVICE_TOKEN=change_me
|
||||
```
|
||||
|
||||
如果你要本地直接起模型适配层,项目里已经补了模板:
|
||||
@@ -429,12 +409,6 @@ AI_PROVIDER_SERVICE_TOKEN=change_me
|
||||
- [aiprovider/.env.example](/home/ray/dev/linkong/planet/aiprovider/.env.example)
|
||||
- [docker-compose.local-model.yml](/home/ray/dev/linkong/planet/docker-compose.local-model.yml)
|
||||
|
||||
推荐映射关系:
|
||||
|
||||
- `vLLM` / `LM Studio` / `One API`: `AI_PROVIDER=openai_compatible`
|
||||
- `MiniMax` / Claude 兼容网关: `AI_PROVIDER=claude_compatible`
|
||||
- `Ollama`: `AI_PROVIDER=ollama`
|
||||
|
||||
运行与调用补充:
|
||||
|
||||
- `./planet.sh start` 默认会启动 `aiprovider`
|
||||
@@ -444,11 +418,13 @@ AI_PROVIDER_SERVICE_TOKEN=change_me
|
||||
|
||||
详细文档:
|
||||
|
||||
- [docs/technical/agents-aiprovider.md](/home/ray/dev/linkong/planet/docs/technical/agents-aiprovider.md)
|
||||
- [docs/technical/zh/agents-aiprovider.md](/home/ray/dev/linkong/planet/docs/technical/zh/agents-aiprovider.md)
|
||||
- [docs/technical/en/agents-aiprovider.md](/home/ray/dev/linkong/planet/docs/technical/en/agents-aiprovider.md)
|
||||
- [aiprovider/README.md](/home/ray/dev/linkong/planet/aiprovider/README.md)
|
||||
- [docs/technical/frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md)
|
||||
- [docs/plans/frontend-ai-playground-development-plan.md](/home/ray/dev/linkong/planet/docs/plans/frontend-ai-playground-development-plan.md)
|
||||
- [docs/plans/agents-situational-awareness-foundation-plan.md](/home/ray/dev/linkong/planet/docs/plans/agents-situational-awareness-foundation-plan.md)
|
||||
- [docs/plans/agents-earth-command-runtime-plan.md](/home/ray/dev/linkong/planet/docs/plans/agents-earth-command-runtime-plan.md)
|
||||
|
||||
## 前端页面布局规范
|
||||
|
||||
|
||||
128
TODO.md
128
TODO.md
@@ -1,45 +1,87 @@
|
||||
# TODO
|
||||
|
||||
- [x] 把 BGP 观测站和异常点的 `hover/click` 手感再磨细一点
|
||||
- [x] 开始做 BGP 异常和海缆/区域的关联展示
|
||||
- [x] 做 Earth 侧的 `BGP activity layer`,让低 incident 密度时地图仍然有持续可感知的观测存在感
|
||||
- [x] 给 Earth BGP 补三层状态表达:`平稳观测态 / 局部波动态 / 事件活跃态`
|
||||
- [x] 把“当前无活跃事件”改造成“观测网络仍在运行、当前未发现聚合级事件”的状态表达
|
||||
- [x] 做 collector / region 近 15 分钟 activity score 聚合接口或动态聚合逻辑
|
||||
- [x] 把 Earth 的 BGP incident 改成 `紧凑事件核 + 向外扩张环形 pulse`,替换当前大面积 glow
|
||||
- [x] 为 BGP incident 建立符号系统:按事件类型用不同 marker,而不是都用同一种亮点
|
||||
- [x] 把 incident 地理定位从 `collector-centric` 改成 `prefix-centric`,优先使用 `prefix_geography`,其次 `prefix_scope`,再次 ASN 区域,最后才回退到观测区域质心
|
||||
- [x] 新增 `prefix_geography` 数据层,不再把 `prefix_scope` 当成 prefix 地理归属本身
|
||||
- [x] 接入 `IPtoASN / IPtoCountry` 作为 prefix-centric geography 的主数据源
|
||||
- [x] 接入 `OpenGeoFeed` 作为 prefix geography 的高质量覆盖/override 数据源
|
||||
- [x] 把 RIR delegated 设计成 prefix geography 的 fallback,而不是主来源
|
||||
- [ ] 为 `aiprovider` 建立 `provider -> api adapter -> compat policy` 的配置中心,优先落成 `json` 或 `yaml` 文件,运行时按 `provider/model` 读取兼容设置,而不是把专项兼容继续散落在 Python 分支里
|
||||
- [ ] 为市面上主流 AI 服务补专项兼容配置并固化到配置文件中,至少覆盖 `OpenAI / Anthropic / MiniMax / Ollama / Moonshot / DeepSeek / Qwen / GLM / Gemini / OpenRouter / vLLM / LM Studio / One API`
|
||||
- [ ] 在兼容配置中补齐可声明项:`api adapter`、`base_url pattern`、`auth header`、`thinking default`、`reasoning block mapping`、`stream path`、`tool-call capability`、`multimodal capability`、`provider-specific request patch`
|
||||
- [ ] 接入 `inetnum` / `inet6num` whois 作为比 RIR 更细粒度的后备层
|
||||
- [x] 在 activity layer 之后继续补 `route leak` 和 `path instability / flap` detector
|
||||
- [ ] 对 [frontend/public/earth/js/bgp.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp.js) 做按职责拆分的小重构,拆成 data / markers / overlays / animation,降低后续维护复杂度
|
||||
- [ ] 可选优化(非必做):将 BGP incident/collector 标点改为 HTML marker(参考 worldmonitor 的 `htmlElementsData` 思路),实现近乎固定屏幕尺寸与更高密度可点击性
|
||||
- [ ] 保持 Earth 当前这批纯个人偏好设置继续走本地持久化:`旋转模式`、HUD 面板显示/隐藏、`地形透明度` 暂不升级到后端系统设置,避免把设备级偏好过早做成全局配置
|
||||
- [ ] 如果后续明确需要“账号级同步 Earth 偏好”,再单独设计 `Earth user preferences`:优先按用户维度而不是全局系统设置保存,并规划 `localStorage -> backend` 的平滑迁移策略
|
||||
- [ ] 为 Planet / Earth 补一个可用的日志查看系统:先明确前后端/AI Provider/采集任务的日志入口、最近日志聚合、筛选与 tail 能力,再决定是先做脚本级统一入口还是控制台内置日志面板
|
||||
- [ ] 重写控制台 UI,逐步抛弃 Ant Design,建立自有组件体系,并统一采用 `tabler.io` / Tabler Icons 作为控制台主图标库
|
||||
- [ ] 把 Earth 态势新闻源从 [earth_news.py](/home/ray/dev/linkong/planet/backend/app/services/earth_news.py) 的硬编码列表抽成可配置目录,优先保持当前“实时聚合”链路不变,只先解决新闻源不可配置的问题
|
||||
- [ ] 为 Earth 态势新闻设计后续采集器化方案:明确新闻数据模型、去重策略、区域映射、过期清理和 Earth/AI 复用方式,再决定何时把新闻从实时抓取升级成正式 collector
|
||||
- [ ] AIS v3.1:修复船只聚合完整性,`/geo/vessels` 合并 raw observation 聚合结果与 legacy `vessel_position + vessel_static` 最新结果,确保 BarentsWatch-only 船只不会因为 AISStream 子集存在而消失,并增加 raw/legacy/final unique MMSI 诊断统计
|
||||
- [ ] AIS v3.2:把 AISStream 从收满 `max_messages` 后结束的批采集改成长连接 streaming service,持续写入 raw observations,通过内部 `/ws` 的 `vessels` channel 推送新船、位置和航向增量,Earth 前端按 MMSI upsert marker
|
||||
- [ ] AIS v3.3:修正 AISStream 采集页面状态语义,使用 connecting/streaming/reconnecting/stopped 与 indeterminate 状态,展示运行时长、消息数、unique MMSI、message rate、最近消息和错误,不再用一次性 REST 进度条表示长连接
|
||||
- [ ] AIS v3.4:修复船只身份字段和名称聚合,MMSI/IMO/callsign 按字符串显示且不带千分位符;查询并列出所有仍以 MMSI 号码或 `MMSI <number>` 作为船名的记录,标注来源、最近观测、message types 和缺失原因,并把这批 fallback-name 船只纳入名称聚合修复集合
|
||||
- [ ] Earth Live Sync:建立统一态势实时同步链路,新增 `earth_summary` WS channel,任意采集器成功后广播轻量 summary invalidation,前端收到后重新拉 `/api/v1/visualization/geo/summary` 并更新 HUD;同时为 BGP 增加 `bgp` WS channel,使 BGP incidents/anomalies/collectors 在不刷新页面时也能 upsert 图层;卫星采集完成后触发 summary 刷新,必要时按 TLE 版本重新 hydrate 卫星数据
|
||||
- [ ] AIS v4:开放船只多源聚合策略配置,支持 source priority、字段级规则、freshness 窗口和高级保护开关;保存时校验未知字段、非法模式和危险动态字段锁定,并在聚合接口返回命中的配置版本
|
||||
- [ ] AIS v5:实现船舶资料 enrichment 与冲突治理,按 `mmsi + imo + name + callsign` 异步补充船型细分、AIS 大类、旗国、尺寸、建造年份、运营方和图片缓存;详情面板展示缓存资料和字段来源,不在实时 AIS 请求链路现场抓第三方页面
|
||||
- [ ] 为 Earth 地球表面增加一层与基础纹理对齐的材质/纹理 overlay,并在同层叠加国界轮廓参考线;要求国界线与底图稳定对齐,且 hover 到国家轮廓时能高亮当前国家,便于校准地表和增强交互
|
||||
- [ ] 把 Earth 新闻接入通用巡航队列:按新闻发生地和时间排序生成巡航目标,巡航聚焦到新闻事件时显示对应新闻卡片,并保持实现边界为“通用巡航层 + 新闻业务适配层”,不要再把新闻逻辑直接耦合回 `main.js` 状态机
|
||||
- [ ] 为未知位置的算力中心建立分层坐标补全链路:优先 `精确坐标 > 站点/园区命中 > 城市 > 州/省 > 国家内主要算力城市 > 国家质心`,并把每次回退的 `confidence / reason / precision` 明确写进统一 GeoJSON
|
||||
- [ ] 为算力中心补一份可维护的本地位置注册表,例如 `canonical_name / aliases / operator / country / region / city / lat / lon / confidence / source_note`,避免把地点知识长期硬编码在 `visualization.py`
|
||||
- [ ] 增强 `epoch_ai_gpu` 和相关算力采集器的源页面解析:即使公开 API 不给坐标,也继续尝试从详情页、HTML、内嵌 JSON、schema.org、OpenGraph、脚本变量和 PDF/新闻稿链接里抽地点线索
|
||||
- [ ] 为未知位置算力中心增加外部富化策略评估:可选接入公开知识源或搜索兜底,只抓“站点名/园区名/城市名”级别线索,不直接抓经纬度结论,并把结果作为候选证据而不是真值
|
||||
- [ ] 为算力中心建立 `operator / cluster name / facility alias` 归一化层,先解决 `xAI / Colossus / Memphis`、`OpenAI / Stargate`、`CoreWeave`、`Lambda`、`Crusoe` 这类同一对象多种写法导致的地点匹配失败
|
||||
- [ ] 为估算位置增加更细的视觉和产品表达:除了问号角标,还要支持 tooltip/详情中的“估算依据”“精度级别”“最后核验时间”,并允许在设置中单独开关“仅看精确位置”
|
||||
- [ ] 为国家级估算点设计更合理的落点策略:优先落在“该国主要算力/数据中心城市候选集”而不是几何质心,必要时同国多节点做稳定散列分配,避免大量节点堆在荒漠或海上
|
||||
- [ ] 为未知位置算力中心建立人工校验工作流:支持导出待核验清单、记录人工确认结果,并把人工确认反哺到位置注册表,逐步减少问号点比例
|
||||
This file is the active backlog only. Completed history belongs in `docs/CHANGELOG.md`; detailed designs belong in `docs/plans/`.
|
||||
|
||||
## Earth
|
||||
|
||||
- [ ] Earth AI command entry: merge natural-language and speech-triggered LLM commands into the existing Earth search panel as described in [Agent Runtime, Earth LLM Command, And Speech Entry Plan](/home/ray/dev/linkong/planet/docs/plans/agents-earth-command-runtime-plan.md).
|
||||
- [ ] Earth action executor: implement safe visualization actions for layer toggles, batch highlights, filters, focus, result panels, and clear-highlight behavior.
|
||||
- [ ] Earth entity matching: support stable entity ids and batch matching for Beidou satellites, mainland China compute centers, BGP, news, vessels, and cables.
|
||||
- [x] High-precision country boundary tile framework: implement the static vector tile builder, versioned seed output, frontend bbox tile loader, debounce, in-flight dedupe, and LRU cache described in [Earth High Precision Boundary Tiles Plan](/home/ray/dev/linkong/planet/docs/plans/earth-high-precision-boundary-tiles-plan.md).
|
||||
- [x] Add the `pmtiles-mvt` frontend tile provider contract, MVT decoder dependencies, static PMTiles Nginx handling, collector artifact registration, production readiness check, and user operation docs for Earth boundaries.
|
||||
- [x] Split Earth boundary ingestion into standard source collectors (`earth_admin0_boundaries`, `earth_coastline`, `earth_claim_lines`) plus the downstream `earth_boundary_tiles` PMTiles builder.
|
||||
- [ ] Replace debug GeoJSON boundary tiles with the real `earth-boundaries-china-pov-v1.pmtiles` production artifact after audited admin-0 / coastline / claim-line sources and the PMTiles toolchain are available.
|
||||
- [ ] Import authoritative China POV / coastline / claim-line source packages through the three standard Earth boundary source collectors, then rebuild a versioned PMTiles artifact so highest zoom `8-10` preserves trusted source geometry instead of seed data.
|
||||
- [ ] Earth boundary data: acquire or generate auditable China POV geometry for Zangnan, Aksai Chin, Taiwan/Penghu, Diaoyu Dao and affiliated islands, Chiwei Yu, South China Sea islands, Kosovo, Gaza, and the official dashed maritime claim line before implementing final visual changes.
|
||||
- [ ] Earth high-resolution basemap tiles: implement the viewport-loaded imagery layer described in [Earth High Resolution Basemap Tiles Plan](/home/ray/dev/linkong/planet/docs/plans/earth-high-resolution-basemap-tiles-plan.md), using high-precision coastline as the alignment reference instead of replacing the globe with one huge texture.
|
||||
- [ ] Presentation controller ownership: replace the singleton card fallback in [presentation-controller.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/presentation-controller.js) with a presentation/card token check before BGP/News migrate onto the shared controller, so connectors only attach to their owning card.
|
||||
- [ ] BGP frontend maintainability: split [bgp.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp.js) by responsibility into data loading, marker rendering, overlays, and animation once the current interaction behavior is stable.
|
||||
- [ ] Optional BGP marker experiment: evaluate HTML markers for BGP incident/collector points if WebGL marker density or fixed screen-size clickability becomes a real blocker.
|
||||
- [ ] Earth news cruise: connect Earth news to the generic cruise queue via a news adapter rather than coupling news-specific sequencing into `main.js`.
|
||||
|
||||
## Compute Centers And Location
|
||||
|
||||
- [ ] Unknown compute-center locations: continue reducing unresolved records through the shared location pipeline, with confidence, precision, reason, and verification date preserved in GeoJSON/details.
|
||||
- [ ] Compute-center registry: keep expanding the local canonical location registry with `canonical_name`, aliases, operator, country/region/city, coordinates, confidence, and source notes.
|
||||
- [ ] Compute-center enrichment: improve source-page parsing for Epoch AI and related collectors by extracting location clues from detail pages, embedded JSON, schema.org, OpenGraph, script variables, PDFs, and press releases.
|
||||
- [ ] Compute-center identity normalization: normalize operator / cluster / facility aliases such as `xAI / Colossus / Memphis`, `OpenAI / Stargate`, `CoreWeave`, `Lambda`, and `Crusoe`.
|
||||
- [ ] Compute-center manual review: add an export/review/import workflow for unresolved or estimated locations and feed confirmed results back into the registry.
|
||||
|
||||
## AIS / Vessels
|
||||
|
||||
- [ ] AIS aggregation strategy v4: expose source priority, field-level merge rules, freshness windows, and protected dynamic-field rules in configuration, with validation and strategy version returned by vessel APIs.
|
||||
- [ ] AIS vessel enrichment v5: add asynchronous vessel profile enrichment for ship type detail, AIS class, flag, dimensions, build year, operator, and cached media. Do not fetch third-party pages in the realtime AIS request path.
|
||||
- [ ] AIS identity cleanup: continue identifying vessels whose display name is only `MMSI <number>` and backfill names from AISStream static messages, BarentsWatch static fields, or enrichment cache.
|
||||
|
||||
## AI Provider And Agents
|
||||
|
||||
- [ ] Unified integration config schema: implement the shared low-code schema engine for datasource, AI Provider, Web Search, and OCR configuration described in [Integration Config Schema System Plan](/home/ray/dev/linkong/planet/docs/plans/integration-config-schema-system-plan.md).
|
||||
- [ ] AI provider routing: finish the OpenClaw-style provider/model routing refactor described in [AI Provider OpenClaw-Style Routing Plan](/home/ray/dev/linkong/planet/docs/plans/ai-provider-openclaw-style-routing-plan.md), so model-specific transport rules live in provider metadata rather than runtime hardcoding.
|
||||
- [ ] AI provider catalog: replace the temporary `model_provider_apis` bridge with structured `models_metadata`, discovery descriptors, and incremental model sync with stale marking.
|
||||
- [ ] AI provider connectivity: keep the plug action as lightweight network/auth/model-directory validation only, and keep real generation tests inside Playground or explicit “trial run” actions.
|
||||
- [ ] Agent runtime foundation: add auditable agent runs, steps, evidence, proposals, and the Agent operations UI described in [Agent Runtime, Earth LLM Command, And Speech Entry Plan](/home/ray/dev/linkong/planet/docs/plans/agents-earth-command-runtime-plan.md).
|
||||
- [ ] Agent tool protocol: add backend JSON tool-call fallback, optional provider-native tool compatibility, tool whitelist validation, and policy-gated proposal application.
|
||||
- [ ] Speech/ASR integration for agents: add provider-neutral transcription settings and API, defaulting to Whisper-compatible API providers while keeping text commands usable when ASR is unavailable.
|
||||
- [ ] Earth voice wake: add device-local configurable wake-word preferences, microphone fallback states, and post-wake instruction upload for Earth commands.
|
||||
- [ ] AI provider compatibility center: move provider/model compatibility rules into a JSON/YAML config read by runtime, instead of continuing to scatter provider-specific branches through Python code.
|
||||
- [ ] Provider compatibility coverage: add explicit config for OpenAI, Anthropic, MiniMax, Ollama, Moonshot, DeepSeek, Qwen, GLM, Gemini, OpenRouter, vLLM, LM Studio, and One API.
|
||||
- [ ] Compatibility schema: cover adapter type, base URL pattern, auth header, thinking/reasoning defaults, stream path, tool-call capability, multimodal capability, and provider-specific request patches.
|
||||
- [ ] BGP geography fallback: evaluate `inetnum` / `inet6num` whois as a finer fallback layer after `prefix_geography`, `OpenGeoFeed`, and RIR delegated data.
|
||||
|
||||
## Platform
|
||||
|
||||
- [ ] Earth preferences scope: keep current device-local Earth preferences in `localStorage`; only design backend user preferences if account-level synchronization becomes a real product requirement.
|
||||
- [ ] System logs: finish a usable Planet log viewing flow that covers backend, frontend, AI Provider, and collector/task logs, with filtering and tailing.
|
||||
- [ ] Console UI modernization: gradually replace Ant Design with Planet-owned components and a consistent Tabler Icons based icon system.
|
||||
- [ ] Earth live sync: design a unified realtime invalidation path for summary/BGP/satellite updates if polling and current WebSocket channels become insufficient.
|
||||
|
||||
## Archive
|
||||
|
||||
Archived items stay here so old context is not lost. Completed items remain checked; obsolete, invalid, or superseded items stay unchecked and include the reason.
|
||||
|
||||
### Completed
|
||||
|
||||
- [x] Refined BGP observer and anomaly `hover/click` feel.
|
||||
- [x] Added BGP anomaly relationship display with cables / regions.
|
||||
- [x] Added the Earth BGP activity layer so the map still feels alive when incident density is low.
|
||||
- [x] Added BGP state expression for stable observation, local fluctuation, and active incident states.
|
||||
- [x] Reframed "no active incident" as "observation network is running; no aggregate incident detected".
|
||||
- [x] Added collector / region recent activity scoring.
|
||||
- [x] Replaced oversized BGP incident glow with compact incident core plus outward pulse rings.
|
||||
- [x] Added BGP incident symbol types instead of using one generic bright marker.
|
||||
- [x] Switched BGP incident geography from collector-centric to prefix-centric priority.
|
||||
- [x] Added `prefix_geography` as a separate data layer instead of treating `prefix_scope` as prefix geography.
|
||||
- [x] Added IPtoASN / IPtoCountry as the main prefix-centric geography source.
|
||||
- [x] Added OpenGeoFeed as a high-quality prefix geography override source.
|
||||
- [x] Made RIR delegated data a prefix geography fallback rather than the primary source.
|
||||
- [x] Added route leak and path instability / flap detectors after the activity layer work.
|
||||
|
||||
### Obsolete Or Superseded
|
||||
|
||||
- [ ] AIS v3.1 old `/geo/vessels` full-merge requirement. Superseded by `/api/v1/vessels/snapshot`, controlled legacy fallback, and diagnostics in the AIS aggregation plan.
|
||||
- [ ] AIS v3.2 old framing of AISStream as a batch collector that needed conversion. Superseded by the implemented long-lived AISStream collector and realtime stream UI.
|
||||
- [ ] AIS v3.3 old one-shot REST progress semantics for AISStream. Superseded by realtime stream status handling.
|
||||
- [ ] AIS v3.4 broad identity cleanup wording. Folded into the active AIS identity cleanup and v5 enrichment tasks.
|
||||
- [ ] Earth surface material overlay for boundary calibration. Superseded by the high-precision boundary tile plan; future work must use source-faithful boundary/coastline data rather than overlay calibration against the coarse base map.
|
||||
- [ ] Hardcoded Earth news source extraction as a standalone task. Superseded by the broader Earth news source configuration and collector plans.
|
||||
- [ ] Country-level compute-center fallback placement as a standalone task. Superseded by the shared location pipeline and registry/manual-review backlog.
|
||||
|
||||
@@ -27,4 +27,7 @@ COPY aiprovider /app/aiprovider
|
||||
|
||||
EXPOSE 8010
|
||||
|
||||
CMD ["uv", "run", "--frozen", "--no-dev", "--project", "/app", "python", "-m", "uvicorn", "aiprovider.main:app", "--host", "0.0.0.0", "--port", "8010", "--reload"]
|
||||
HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
|
||||
CMD curl -fsS http://127.0.0.1:8010/health >/dev/null || exit 1
|
||||
|
||||
CMD ["uv", "run", "--frozen", "--no-dev", "--project", "/app", "python", "-m", "uvicorn", "aiprovider.main:app", "--host", "0.0.0.0", "--port", "8010"]
|
||||
|
||||
@@ -17,9 +17,6 @@ class Settings(BaseSettings):
|
||||
AI_HTTP_RETRY_ATTEMPTS: int = 2
|
||||
AI_MAX_TOKENS: int = 1200
|
||||
AI_ANTHROPIC_VERSION: str = "2023-06-01"
|
||||
AI_ANALYSIS_SYSTEM_PROMPT: str = (
|
||||
"你是态势感知分析助手。请基于输入的上下文、观测与约束,输出结构化、克制、可执行的分析。"
|
||||
)
|
||||
|
||||
AI_PROVIDER_SERVICE_TOKEN: str = ""
|
||||
|
||||
|
||||
@@ -45,6 +45,7 @@ def get_provider_service(
|
||||
x_ai_model: str | None = Header(default=None),
|
||||
x_ai_max_tokens: str | None = Header(default=None),
|
||||
x_ai_anthropic_version: str | None = Header(default=None),
|
||||
x_ai_model_provider_apis: str | None = Header(default=None),
|
||||
) -> ProviderService:
|
||||
overrides = {
|
||||
"provider": x_ai_provider,
|
||||
@@ -53,6 +54,7 @@ def get_provider_service(
|
||||
"api_key": x_ai_api_key,
|
||||
"model": x_ai_model,
|
||||
"anthropic_version": x_ai_anthropic_version,
|
||||
"model_provider_apis": x_ai_model_provider_apis,
|
||||
}
|
||||
if x_ai_max_tokens:
|
||||
overrides["max_tokens"] = x_ai_max_tokens
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
@@ -14,7 +15,6 @@ from aiprovider.schemas import (
|
||||
SituationalAnalysisResponse,
|
||||
)
|
||||
|
||||
|
||||
def _normalize_provider(value: str) -> str:
|
||||
return (value or "disabled").strip().lower()
|
||||
|
||||
@@ -62,7 +62,9 @@ class ProviderService:
|
||||
self.anthropic_version = str(
|
||||
overrides.get("anthropic_version") or settings.AI_ANTHROPIC_VERSION
|
||||
)
|
||||
self.system_prompt = settings.AI_ANALYSIS_SYSTEM_PROMPT
|
||||
self.model_provider_apis = self._parse_model_provider_apis(
|
||||
overrides.get("model_provider_apis")
|
||||
)
|
||||
|
||||
def get_status(self) -> AIProviderStatusResponse:
|
||||
enabled = self.provider != "disabled"
|
||||
@@ -94,16 +96,23 @@ class ProviderService:
|
||||
|
||||
prompt = self._build_prompt(payload)
|
||||
|
||||
if self.provider_api == "openai-completions":
|
||||
data = await self._request_openai_compatible(model, prompt)
|
||||
provider_api = self._resolve_model_provider_api(model)
|
||||
|
||||
if provider_api == "openai-completions":
|
||||
data = await self._request_openai_compatible(model, prompt, payload.system_prompt)
|
||||
content = self._extract_openai_content(data)
|
||||
content_blocks = self._extract_openai_blocks(data)
|
||||
elif self.provider_api == "anthropic-messages":
|
||||
data = await self._request_anthropic_messages(model, prompt, payload.thinking)
|
||||
elif provider_api == "anthropic-messages":
|
||||
data = await self._request_anthropic_messages(
|
||||
model,
|
||||
prompt,
|
||||
payload.thinking,
|
||||
payload.system_prompt,
|
||||
)
|
||||
content = self._extract_anthropic_content(data)
|
||||
content_blocks = self._extract_anthropic_blocks(data)
|
||||
elif self.provider_api == "ollama-generate":
|
||||
data = await self._request_ollama(model, prompt)
|
||||
elif provider_api == "ollama-generate":
|
||||
data = await self._request_ollama(model, prompt, payload.system_prompt)
|
||||
content = self._extract_ollama_content(data)
|
||||
content_blocks = self._extract_ollama_blocks(data)
|
||||
else:
|
||||
@@ -128,6 +137,26 @@ class ProviderService:
|
||||
def _requires_api_key(self) -> bool:
|
||||
return self.provider_api != "ollama-generate"
|
||||
|
||||
def _resolve_model_provider_api(self, model: str) -> str:
|
||||
return self.model_provider_apis.get(model) or self.provider_api
|
||||
|
||||
def _parse_model_provider_apis(self, value: Any) -> dict[str, str]:
|
||||
if isinstance(value, dict):
|
||||
raw = value
|
||||
elif isinstance(value, str) and value.strip():
|
||||
try:
|
||||
parsed = json.loads(value)
|
||||
except json.JSONDecodeError:
|
||||
return {}
|
||||
raw = parsed if isinstance(parsed, dict) else {}
|
||||
else:
|
||||
raw = {}
|
||||
return {
|
||||
str(model): _normalize_provider_api(str(provider_api))
|
||||
for model, provider_api in raw.items()
|
||||
if model and provider_api
|
||||
}
|
||||
|
||||
def _build_prompt(self, payload: SituationalAnalysisRequest) -> str:
|
||||
sections = [
|
||||
f"任务标题:\n{payload.title}",
|
||||
@@ -139,19 +168,28 @@ class ProviderService:
|
||||
sections.append("约束条件:\n" + "\n".join(f"- {item}" for item in payload.constraints))
|
||||
if payload.context:
|
||||
sections.append(f"附加上下文:\n{payload.context}")
|
||||
sections.append(
|
||||
"请输出: 1) 态势摘要 2) 关键风险 3) 研判依据 4) 建议动作 5) 还缺少的数据。"
|
||||
)
|
||||
return "\n\n".join(sections)
|
||||
|
||||
async def _request_openai_compatible(self, model: str, prompt: str) -> dict[str, Any]:
|
||||
def _resolve_system_prompt(self, system_prompt: str | None) -> str | None:
|
||||
resolved = str(system_prompt or "").strip()
|
||||
return resolved or None
|
||||
|
||||
async def _request_openai_compatible(
|
||||
self,
|
||||
model: str,
|
||||
prompt: str,
|
||||
system_prompt: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
messages = []
|
||||
resolved_system_prompt = self._resolve_system_prompt(system_prompt)
|
||||
if resolved_system_prompt:
|
||||
messages.append({"role": "system", "content": resolved_system_prompt})
|
||||
messages.append({"role": "user", "content": prompt})
|
||||
request_body = {
|
||||
"model": model,
|
||||
"messages": [
|
||||
{"role": "system", "content": self.system_prompt},
|
||||
{"role": "user", "content": prompt},
|
||||
],
|
||||
"messages": messages,
|
||||
"temperature": 0.2,
|
||||
"max_tokens": self.max_tokens,
|
||||
}
|
||||
return await self._post(
|
||||
path="/chat/completions",
|
||||
@@ -167,10 +205,10 @@ class ProviderService:
|
||||
model: str,
|
||||
prompt: str,
|
||||
thinking: dict[str, Any] | None = None,
|
||||
system_prompt: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
request_body = {
|
||||
"model": model,
|
||||
"system": self.system_prompt,
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
@@ -185,6 +223,9 @@ class ProviderService:
|
||||
"max_tokens": self.max_tokens,
|
||||
"temperature": 0.2,
|
||||
}
|
||||
resolved_system_prompt = self._resolve_system_prompt(system_prompt)
|
||||
if resolved_system_prompt:
|
||||
request_body["system"] = resolved_system_prompt
|
||||
resolved_thinking = self._resolve_anthropic_thinking(thinking)
|
||||
if resolved_thinking:
|
||||
request_body["thinking"] = resolved_thinking
|
||||
@@ -218,19 +259,28 @@ class ProviderService:
|
||||
model: str,
|
||||
prompt: str,
|
||||
thinking: dict[str, Any] | None = None,
|
||||
system_prompt: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
return await self._request_anthropic_messages(model, prompt, thinking)
|
||||
return await self._request_anthropic_messages(model, prompt, thinking, system_prompt)
|
||||
|
||||
async def _request_ollama(self, model: str, prompt: str) -> dict[str, Any]:
|
||||
async def _request_ollama(
|
||||
self,
|
||||
model: str,
|
||||
prompt: str,
|
||||
system_prompt: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
request_body = {
|
||||
"model": model,
|
||||
"stream": False,
|
||||
"system": self.system_prompt,
|
||||
"prompt": prompt,
|
||||
"options": {
|
||||
"temperature": 0.2,
|
||||
"num_predict": self.max_tokens,
|
||||
},
|
||||
}
|
||||
resolved_system_prompt = self._resolve_system_prompt(system_prompt)
|
||||
if resolved_system_prompt:
|
||||
request_body["system"] = resolved_system_prompt
|
||||
return await self._post(
|
||||
path="/api/generate",
|
||||
headers={
|
||||
@@ -289,13 +339,19 @@ class ProviderService:
|
||||
message = choices[0].get("message") or {}
|
||||
content = message.get("content")
|
||||
if isinstance(content, str):
|
||||
return content
|
||||
if content:
|
||||
return content
|
||||
reasoning_content = message.get("reasoning_content")
|
||||
return reasoning_content if isinstance(reasoning_content, str) else ""
|
||||
if isinstance(content, list):
|
||||
return "".join(
|
||||
item.get("text", "")
|
||||
for item in content
|
||||
if isinstance(item, dict)
|
||||
)
|
||||
reasoning_content = message.get("reasoning_content")
|
||||
if isinstance(reasoning_content, str):
|
||||
return reasoning_content
|
||||
return ""
|
||||
|
||||
def _extract_openai_blocks(self, payload: dict[str, Any]) -> list[AIContentBlock]:
|
||||
@@ -306,9 +362,14 @@ class ProviderService:
|
||||
message = choices[0].get("message") or {}
|
||||
content = message.get("content")
|
||||
if isinstance(content, str):
|
||||
return [AIContentBlock(type="text", text=content)]
|
||||
blocks = [AIContentBlock(type="text", text=content)] if content else []
|
||||
reasoning_content = message.get("reasoning_content")
|
||||
if isinstance(reasoning_content, str) and reasoning_content:
|
||||
blocks.append(AIContentBlock(type="thinking", thinking=reasoning_content))
|
||||
return blocks
|
||||
if not isinstance(content, list):
|
||||
return []
|
||||
reasoning_content = message.get("reasoning_content")
|
||||
return [AIContentBlock(type="thinking", thinking=reasoning_content)] if isinstance(reasoning_content, str) and reasoning_content else []
|
||||
|
||||
blocks: list[AIContentBlock] = []
|
||||
for item in content:
|
||||
@@ -321,7 +382,11 @@ class ProviderService:
|
||||
metadata={k: v for k, v in item.items() if k not in {"type", "text"}},
|
||||
)
|
||||
)
|
||||
reasoning_content = message.get("reasoning_content")
|
||||
if isinstance(reasoning_content, str) and reasoning_content:
|
||||
blocks.append(AIContentBlock(type="thinking", thinking=reasoning_content))
|
||||
return blocks
|
||||
|
||||
def _extract_anthropic_content(self, payload: dict[str, Any]) -> str:
|
||||
content = payload.get("content")
|
||||
if isinstance(content, str):
|
||||
|
||||
@@ -13,10 +13,11 @@ class AIContentBlock(BaseModel):
|
||||
|
||||
class SituationalAnalysisRequest(BaseModel):
|
||||
title: str = Field(..., min_length=1, max_length=200)
|
||||
objective: str = Field(..., min_length=1, max_length=1000)
|
||||
objective: str = Field(..., min_length=1, max_length=20000)
|
||||
context: dict[str, Any] = Field(default_factory=dict)
|
||||
observations: list[str] = Field(default_factory=list)
|
||||
constraints: list[str] = Field(default_factory=list)
|
||||
system_prompt: str | None = Field(default=None, max_length=8000)
|
||||
preferred_model: str | None = Field(default=None, max_length=200)
|
||||
thinking: dict[str, Any] | None = None
|
||||
|
||||
|
||||
@@ -12,6 +12,7 @@ ENV PYTHONDONTWRITEBYTECODE=1
|
||||
ENV PYTHONUNBUFFERED=1
|
||||
ENV UV_COMPILE_BYTECODE=1
|
||||
ENV UV_LINK_MODE=copy
|
||||
ENV PYTHONPATH=/app/backend
|
||||
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
curl \
|
||||
@@ -25,4 +26,7 @@ COPY VERSION /app/VERSION
|
||||
|
||||
EXPOSE 8000
|
||||
|
||||
CMD ["uv", "run", "--frozen", "--no-dev", "--project", "/app", "python", "-m", "uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000", "--reload"]
|
||||
HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
|
||||
CMD curl -fsS http://127.0.0.1:8000/health >/dev/null || exit 1
|
||||
|
||||
CMD ["uv", "run", "--frozen", "--no-dev", "--project", "/app", "python", "-m", "uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
|
||||
2
backend/app/ai_tasks/__init__.py
Normal file
2
backend/app/ai_tasks/__init__.py
Normal file
@@ -0,0 +1,2 @@
|
||||
"""AI task prompt registry and runtime helpers."""
|
||||
|
||||
74
backend/app/ai_tasks/default_prompts.json
Normal file
74
backend/app/ai_tasks/default_prompts.json
Normal file
@@ -0,0 +1,74 @@
|
||||
[
|
||||
{
|
||||
"key": "earth.news.enrich",
|
||||
"label": "Earth 新闻汉化与定位",
|
||||
"group": "Earth 新闻",
|
||||
"version": "2026-05-16.2",
|
||||
"system_prompt": "",
|
||||
"prompt": "Return exactly one strict JSON object with a location object and a localizations object. Infer the most likely physical event location and produce a faithful Simplified Chinese title plus a one-sentence newswire-style Chinese summary based only on the supplied RSS headline, description, source, and date. The summary should read like a concise breaking-news lead, not a label, slogan, or keyword headline."
|
||||
},
|
||||
{
|
||||
"key": "alerts.brief",
|
||||
"label": "系统告警研判",
|
||||
"group": "告警研判",
|
||||
"version": "2026-05-16.1",
|
||||
"system_prompt": "你是告警研判助手。请基于输入的告警事实、上下文与约束,输出结构化、克制、可执行的值班研判;明确区分事实、推断与建议,不要夸大证据不足的风险。",
|
||||
"prompt": "基于当前告警总量、严重度、状态、数据源分布与最近告警摘录,生成一份面向值班人员的简明告警态势简报,突出待处理风险、告警集中点和优先动作。"
|
||||
},
|
||||
{
|
||||
"key": "alerts.situational.brief",
|
||||
"label": "跨模块态势告警研判",
|
||||
"group": "告警研判",
|
||||
"version": "2026-05-16.1",
|
||||
"system_prompt": "你是告警研判助手。请基于输入的告警事实、上下文与约束,输出结构化、克制、可执行的值班研判;明确区分事实、推断与建议,不要夸大证据不足的风险。",
|
||||
"prompt": "综合系统告警、BGP incidents、BGP anomalies 与近期 BGP AI 简报,生成一份面向值班人员的态势告警简报,指出当前最需要关注的风险域、跨模块联动迹象和优先动作。"
|
||||
},
|
||||
{
|
||||
"key": "bgp.brief",
|
||||
"label": "BGP 态势简报",
|
||||
"group": "BGP",
|
||||
"version": "2026-05-16.2",
|
||||
"system_prompt": "你是 BGP 值班分析师。请直接输出面向值班人员的中文 Markdown 简报,只写最终研判内容;不要复述用户需求、提示词、写作计划、字段清单或“我将如何回答”。",
|
||||
"prompt": "基于当前 BGP incidents、anomalies、原始观测事件、观测站覆盖与 prefix geography 证据,生成一份面向操作员的简明态势简报,突出区域热点、观测偏差、当前风险、证据和优先动作。"
|
||||
},
|
||||
{
|
||||
"key": "location.factcheck.normalize",
|
||||
"label": "位置事实核查结构化",
|
||||
"group": "位置解析",
|
||||
"version": "2026-05-16.1",
|
||||
"system_prompt": "",
|
||||
"prompt": "Convert the supplied location factcheck text into exactly one strict JSON object. Extract only facts present in the text or original query."
|
||||
},
|
||||
{
|
||||
"key": "location.factcheck.resolve",
|
||||
"label": "位置事实核查兜底",
|
||||
"group": "位置解析",
|
||||
"version": "2026-05-16.1",
|
||||
"system_prompt": "",
|
||||
"prompt": "Return exactly one JSON object for the most likely physical location. Use only fact-checkable public knowledge; return null fields rather than guessing when evidence is weak."
|
||||
},
|
||||
{
|
||||
"key": "datasource.mapping",
|
||||
"label": "数据源映射生成",
|
||||
"group": "采集配置",
|
||||
"version": "2026-05-16.1",
|
||||
"system_prompt": "",
|
||||
"prompt": "Return only JSON for a deterministic mapping DSL. The JSON must contain source.items_path and fields. Do not include prose or code."
|
||||
},
|
||||
{
|
||||
"key": "credential.guide",
|
||||
"label": "采集器凭据教程",
|
||||
"group": "采集配置",
|
||||
"version": "2026-05-16.1",
|
||||
"system_prompt": "",
|
||||
"prompt": "生成一份中文采集器凭据配置教程。只能根据 context.search_evidence 中的来源生成教程;如果证据不足,明确说明需要以官方页面为准。"
|
||||
},
|
||||
{
|
||||
"key": "ai.connection_test",
|
||||
"label": "AI Provider 连接测试",
|
||||
"group": "运维测试",
|
||||
"version": "2026-05-16.1",
|
||||
"system_prompt": "",
|
||||
"prompt": "Reply OK."
|
||||
}
|
||||
]
|
||||
182
backend/app/ai_tasks/prompts.py
Normal file
182
backend/app/ai_tasks/prompts.py
Normal file
@@ -0,0 +1,182 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
import json
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.models.system_setting import SystemSetting
|
||||
|
||||
AI_PROMPTS_CATEGORY = "ai_prompts"
|
||||
DEFAULT_PROMPTS_PATH = Path(__file__).with_name("default_prompts.json")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AIPromptDefinition:
|
||||
key: str
|
||||
label: str
|
||||
group: str
|
||||
version: str
|
||||
system_prompt: str
|
||||
prompt: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class EffectiveAIPrompt:
|
||||
key: str
|
||||
label: str
|
||||
group: str
|
||||
version: str
|
||||
default_system_prompt: str
|
||||
default_prompt: str
|
||||
system_prompt: str
|
||||
prompt: str
|
||||
is_custom: bool
|
||||
updated_at: str | None = None
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def list_prompt_definitions() -> tuple[AIPromptDefinition, ...]:
|
||||
raw_items = json.loads(DEFAULT_PROMPTS_PATH.read_text(encoding="utf-8"))
|
||||
return tuple(
|
||||
AIPromptDefinition(
|
||||
key=str(item["key"]),
|
||||
label=str(item["label"]),
|
||||
group=str(item["group"]),
|
||||
version=str(item["version"]),
|
||||
system_prompt=str(item.get("system_prompt") or ""),
|
||||
prompt=str(item.get("prompt") or ""),
|
||||
)
|
||||
for item in raw_items
|
||||
)
|
||||
|
||||
|
||||
def get_prompt_definition(task_key: str) -> AIPromptDefinition:
|
||||
for definition in list_prompt_definitions():
|
||||
if definition.key == task_key:
|
||||
return definition
|
||||
raise KeyError(task_key)
|
||||
|
||||
|
||||
async def _get_prompt_setting(db: AsyncSession) -> SystemSetting | None:
|
||||
result = await db.execute(
|
||||
select(SystemSetting).where(SystemSetting.category == AI_PROMPTS_CATEGORY)
|
||||
)
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
|
||||
def _normalize_overrides(payload: dict[str, Any] | None) -> dict[str, dict[str, Any]]:
|
||||
raw = (payload or {}).get("overrides")
|
||||
if not isinstance(raw, dict):
|
||||
return {}
|
||||
return {
|
||||
str(key): dict(value)
|
||||
for key, value in raw.items()
|
||||
if isinstance(value, dict)
|
||||
}
|
||||
|
||||
|
||||
async def get_prompt_overrides(db: AsyncSession) -> dict[str, dict[str, Any]]:
|
||||
if not hasattr(db, "execute"):
|
||||
return {}
|
||||
setting = await _get_prompt_setting(db)
|
||||
return _normalize_overrides(setting.payload if setting else None)
|
||||
|
||||
|
||||
def _effective_prompt(
|
||||
definition: AIPromptDefinition,
|
||||
override: dict[str, Any] | None,
|
||||
) -> EffectiveAIPrompt:
|
||||
override = override or {}
|
||||
custom_system = override.get("system_prompt")
|
||||
custom_prompt = override.get("prompt")
|
||||
has_custom_system = isinstance(custom_system, str)
|
||||
has_custom_prompt = isinstance(custom_prompt, str)
|
||||
return EffectiveAIPrompt(
|
||||
key=definition.key,
|
||||
label=definition.label,
|
||||
group=definition.group,
|
||||
version=definition.version,
|
||||
default_system_prompt=definition.system_prompt,
|
||||
default_prompt=definition.prompt,
|
||||
system_prompt=custom_system if has_custom_system else definition.system_prompt,
|
||||
prompt=custom_prompt if has_custom_prompt else definition.prompt,
|
||||
is_custom=has_custom_system or has_custom_prompt,
|
||||
updated_at=str(override.get("updated_at") or "") or None,
|
||||
)
|
||||
|
||||
|
||||
async def list_effective_prompts(db: AsyncSession) -> list[EffectiveAIPrompt]:
|
||||
overrides = await get_prompt_overrides(db)
|
||||
return [
|
||||
_effective_prompt(definition, overrides.get(definition.key))
|
||||
for definition in list_prompt_definitions()
|
||||
]
|
||||
|
||||
|
||||
async def get_effective_prompt(db: AsyncSession | None, task_key: str) -> EffectiveAIPrompt:
|
||||
definition = get_prompt_definition(task_key)
|
||||
if db is None:
|
||||
return _effective_prompt(definition, None)
|
||||
overrides = await get_prompt_overrides(db)
|
||||
return _effective_prompt(definition, overrides.get(task_key))
|
||||
|
||||
|
||||
async def save_prompt_override(
|
||||
db: AsyncSession,
|
||||
task_key: str,
|
||||
*,
|
||||
system_prompt: str,
|
||||
prompt: str,
|
||||
) -> EffectiveAIPrompt:
|
||||
definition = get_prompt_definition(task_key)
|
||||
setting = await _get_prompt_setting(db)
|
||||
payload = dict(setting.payload or {}) if setting else {}
|
||||
overrides = _normalize_overrides(payload)
|
||||
overrides[definition.key] = {
|
||||
"system_prompt": system_prompt,
|
||||
"prompt": prompt,
|
||||
"updated_at": datetime.now(UTC).isoformat().replace("+00:00", "Z"),
|
||||
}
|
||||
payload["overrides"] = overrides
|
||||
if setting is None:
|
||||
setting = SystemSetting(category=AI_PROMPTS_CATEGORY, payload=payload)
|
||||
db.add(setting)
|
||||
else:
|
||||
setting.payload = payload
|
||||
await db.commit()
|
||||
return _effective_prompt(definition, overrides[definition.key])
|
||||
|
||||
|
||||
async def reset_prompt_override(db: AsyncSession, task_key: str) -> EffectiveAIPrompt:
|
||||
definition = get_prompt_definition(task_key)
|
||||
setting = await _get_prompt_setting(db)
|
||||
if setting is None:
|
||||
return _effective_prompt(definition, None)
|
||||
payload = dict(setting.payload or {})
|
||||
overrides = _normalize_overrides(payload)
|
||||
overrides.pop(definition.key, None)
|
||||
payload["overrides"] = overrides
|
||||
setting.payload = payload
|
||||
await db.commit()
|
||||
return _effective_prompt(definition, None)
|
||||
|
||||
|
||||
def serialize_effective_prompt(prompt: EffectiveAIPrompt) -> dict[str, Any]:
|
||||
return {
|
||||
"key": prompt.key,
|
||||
"label": prompt.label,
|
||||
"group": prompt.group,
|
||||
"version": prompt.version,
|
||||
"default_system_prompt": prompt.default_system_prompt,
|
||||
"default_prompt": prompt.default_prompt,
|
||||
"system_prompt": prompt.system_prompt,
|
||||
"prompt": prompt.prompt,
|
||||
"is_custom": prompt.is_custom,
|
||||
"updated_at": prompt.updated_at,
|
||||
}
|
||||
@@ -6,6 +6,7 @@ from app.api.v1 import (
|
||||
datasource_config,
|
||||
datasources,
|
||||
docs,
|
||||
earth,
|
||||
tasks,
|
||||
dashboard,
|
||||
alerts,
|
||||
@@ -18,6 +19,7 @@ from app.api.v1 import (
|
||||
vessels,
|
||||
bgp,
|
||||
news,
|
||||
realtime_sources,
|
||||
system_control,
|
||||
tv,
|
||||
)
|
||||
@@ -33,6 +35,7 @@ api_router.include_router(
|
||||
api_router.include_router(datasources.router, prefix="/datasources", tags=["datasources"])
|
||||
api_router.include_router(collected_data.router, prefix="/collected", tags=["collected-data"])
|
||||
api_router.include_router(docs.router, prefix="/docs", tags=["docs"])
|
||||
api_router.include_router(earth.router, prefix="/earth", tags=["earth"])
|
||||
api_router.include_router(tasks.router, prefix="/tasks", tags=["tasks"])
|
||||
api_router.include_router(dashboard.router, prefix="/dashboard", tags=["dashboard"])
|
||||
api_router.include_router(alerts.router, prefix="/alerts", tags=["alerts"])
|
||||
@@ -50,3 +53,4 @@ api_router.include_router(vessels.router, prefix="/vessels", tags=["vessels"])
|
||||
api_router.include_router(bgp.router, prefix="/bgp", tags=["bgp"])
|
||||
api_router.include_router(tv.router, prefix="/tv", tags=["tv"])
|
||||
api_router.include_router(news.router, prefix="/news", tags=["news"])
|
||||
api_router.include_router(realtime_sources.router, prefix="/realtime-sources", tags=["realtime-sources"])
|
||||
|
||||
@@ -341,6 +341,7 @@ async def collect_bgp_collector_location(
|
||||
provider_client=provider_client,
|
||||
query=query,
|
||||
entity_type="bgp_collector",
|
||||
db=db,
|
||||
attempted_queries=attempted_queries,
|
||||
search_evidence=search_result.evidence,
|
||||
)
|
||||
|
||||
@@ -102,7 +102,12 @@ def build_search_rank_sql(search: Optional[str]) -> str:
|
||||
"""
|
||||
|
||||
|
||||
def serialize_collected_row(row, source_name_map: dict[str, str] | None = None) -> dict:
|
||||
def serialize_collected_row(
|
||||
row,
|
||||
source_name_map: dict[str, str] | None = None,
|
||||
*,
|
||||
include_metadata: bool = True,
|
||||
) -> dict:
|
||||
metadata = row[7]
|
||||
source = row[1]
|
||||
return {
|
||||
@@ -120,7 +125,7 @@ def serialize_collected_row(row, source_name_map: dict[str, str] | None = None)
|
||||
"longitude": get_metadata_field(metadata, "longitude"),
|
||||
"value": get_metadata_field(metadata, "value"),
|
||||
"unit": get_metadata_field(metadata, "unit"),
|
||||
"metadata": metadata,
|
||||
"metadata": metadata if include_metadata else None,
|
||||
"cores": get_metadata_field(metadata, "cores"),
|
||||
"rmax": get_metadata_field(metadata, "rmax"),
|
||||
"rpeak": get_metadata_field(metadata, "rpeak"),
|
||||
@@ -145,6 +150,7 @@ async def list_collected_data(
|
||||
search: Optional[str] = Query(None, description="搜索名称"),
|
||||
page: int = Query(1, ge=1, description="页码"),
|
||||
page_size: int = Query(20, ge=1, le=100, description="每页数量"),
|
||||
include_metadata: bool = Query(True, description="是否返回完整 metadata 字段"),
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
@@ -201,7 +207,7 @@ async def list_collected_data(
|
||||
|
||||
data = []
|
||||
for row in rows:
|
||||
data.append(serialize_collected_row(row[:11], source_name_map))
|
||||
data.append(serialize_collected_row(row[:11], source_name_map, include_metadata=include_metadata))
|
||||
|
||||
return {
|
||||
"total": total,
|
||||
|
||||
@@ -5,7 +5,7 @@ from datetime import datetime
|
||||
import base64
|
||||
import json
|
||||
import re
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Request, status
|
||||
from sqlalchemy import delete, select, func
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from pydantic import BaseModel, Field
|
||||
@@ -22,6 +22,7 @@ from app.models.vessel import AISRawObservation, AISSourceHealth
|
||||
from app.core.security import get_current_user
|
||||
from app.core.cache import cache
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.ai_tasks.prompts import get_effective_prompt
|
||||
from app.schemas.ai import SituationalAnalysisRequest
|
||||
from app.services.ai_client import AIProviderClient, get_ai_provider_client
|
||||
from app.services.datasource_mapping import (
|
||||
@@ -41,15 +42,82 @@ from app.services.custom_datasource_runtime import (
|
||||
stop_custom_stream,
|
||||
test_websocket_config,
|
||||
)
|
||||
|
||||
DATASOURCE_MAPPING_PROMPT_KEY = "datasource.mapping"
|
||||
from app.services.datasource_connectivity import (
|
||||
_resolve_aisstream_api_key,
|
||||
_resolve_spacetrack_credentials_with_override,
|
||||
get_builtin_connection_status,
|
||||
save_connectivity_success,
|
||||
strip_connectivity_validation,
|
||||
test_builtin_connectivity,
|
||||
)
|
||||
from app.services.barentswatch import resolve_barentswatch_config
|
||||
from app.services.persistent_logs import record_audit_log
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
SECRET_REVEAL_ROLES = {"admin", "super_admin"}
|
||||
|
||||
|
||||
def _user_role_value(user: User) -> str:
|
||||
role = getattr(user, "role", "")
|
||||
return str(getattr(role, "value", role) or "").lower()
|
||||
|
||||
|
||||
def _user_display_name(user: User) -> str:
|
||||
return str(getattr(user, "username", None) or getattr(user, "email", None) or getattr(user, "id", ""))
|
||||
|
||||
|
||||
async def _record_datasource_secret_reveal(
|
||||
*,
|
||||
current_user: User,
|
||||
request: Request,
|
||||
target_id: str,
|
||||
result: str,
|
||||
details: dict[str, Any],
|
||||
) -> None:
|
||||
await record_audit_log(
|
||||
action="datasource_config.secret.reveal",
|
||||
actor_id=getattr(current_user, "id", None),
|
||||
actor_name=_user_display_name(current_user),
|
||||
target_type="datasource_config_secret",
|
||||
target_id=target_id,
|
||||
result=result,
|
||||
ip=request.client.host if request.client else None,
|
||||
details=details,
|
||||
)
|
||||
|
||||
|
||||
async def _ensure_datasource_secret_reveal_allowed(
|
||||
current_user: User,
|
||||
request: Request,
|
||||
target_id: str,
|
||||
details: dict[str, Any],
|
||||
) -> None:
|
||||
if _user_role_value(current_user) in SECRET_REVEAL_ROLES:
|
||||
return
|
||||
await _record_datasource_secret_reveal(
|
||||
current_user=current_user,
|
||||
request=request,
|
||||
target_id=target_id,
|
||||
result="denied",
|
||||
details={**details, "role": _user_role_value(current_user)},
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_403_FORBIDDEN,
|
||||
detail="Only administrators can reveal datasource credentials",
|
||||
)
|
||||
|
||||
def _default_builtin_config(name: str) -> dict[str, Any]:
|
||||
return {"timeout": 30, "retry": 3}
|
||||
|
||||
|
||||
def _default_builtin_source_type(name: str) -> str:
|
||||
if name == "aisstream_vessels":
|
||||
return "websocket"
|
||||
return "http"
|
||||
|
||||
|
||||
class DataSourceConfigCreate(BaseModel):
|
||||
name: str = Field(..., min_length=1, max_length=100)
|
||||
@@ -364,7 +432,7 @@ async def list_all_datasources(
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""List all data sources: YAML defaults + DB overrides"""
|
||||
from app.core.data_sources import COLLECTOR_URL_KEYS, get_data_sources_config
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
|
||||
config = get_data_sources_config()
|
||||
|
||||
@@ -372,38 +440,144 @@ async def list_all_datasources(
|
||||
db_configs = {c.name: c for c in db_query.scalars().all()}
|
||||
|
||||
result = []
|
||||
for name, yaml_key in COLLECTOR_URL_KEYS.items():
|
||||
for name, metadata in DEFAULT_DATASOURCES.items():
|
||||
yaml_url = config.get_yaml_url(name)
|
||||
db_config = db_configs.get(name)
|
||||
default_config = _default_builtin_config(name)
|
||||
default_url = yaml_url
|
||||
db_auth_config = db_config.auth_config or {} if db_config else {}
|
||||
|
||||
result.append(
|
||||
{
|
||||
"name": name,
|
||||
"default_url": yaml_url,
|
||||
"endpoint": db_config.endpoint if db_config else yaml_url,
|
||||
"requires_credentials": bool(metadata.get("requires_credentials", False)),
|
||||
"credential_provider": metadata.get("credential_provider"),
|
||||
"credential_status": metadata.get("credential_status", "none"),
|
||||
"default_url": default_url,
|
||||
"endpoint": db_config.endpoint if db_config else default_url,
|
||||
"is_overridden": db_config is not None and db_config.endpoint != yaml_url
|
||||
if yaml_url
|
||||
if default_url
|
||||
else db_config is not None,
|
||||
"is_active": db_config.is_active if db_config else True,
|
||||
"source_type": db_config.source_type if db_config else "http",
|
||||
"source_type": db_config.source_type if db_config else _default_builtin_source_type(name),
|
||||
"auth_type": db_config.auth_type if db_config else "none",
|
||||
"auth_config": {
|
||||
"client_id": db_auth_config.get("client_id") or "",
|
||||
"username": db_auth_config.get("username") or "",
|
||||
"key_name": db_auth_config.get("key_name") or db_auth_config.get("param_name") or "",
|
||||
"param_name": db_auth_config.get("param_name") or db_auth_config.get("key_name") or "",
|
||||
"location": db_auth_config.get("location") or db_auth_config.get("in") or "",
|
||||
"in": db_auth_config.get("in") or db_auth_config.get("location") or "",
|
||||
},
|
||||
"auth_configured": {
|
||||
"api_key": bool((db_config.auth_config or {}).get("api_key"))
|
||||
if db_config
|
||||
else False,
|
||||
"api_key": bool(db_auth_config.get("api_key")),
|
||||
"client_id": bool(db_auth_config.get("client_id")),
|
||||
"client_secret": bool(db_auth_config.get("client_secret")),
|
||||
"username": bool(db_auth_config.get("username")),
|
||||
"password": bool(db_auth_config.get("password")),
|
||||
},
|
||||
"headers": db_config.headers if db_config else {},
|
||||
"config": strip_connectivity_validation(db_config.config if db_config else {}),
|
||||
"config": strip_connectivity_validation(db_config.config if db_config else default_config),
|
||||
"config_id": db_config.id if db_config else None,
|
||||
"description": db_config.description
|
||||
if db_config
|
||||
else f"Data source from YAML: {yaml_key}",
|
||||
else f"内置采集器默认配置:{metadata.get('display_name') or metadata.get('name') or name}",
|
||||
}
|
||||
)
|
||||
|
||||
return {"total": len(result), "data": result}
|
||||
|
||||
|
||||
@router.get("/configs/secrets")
|
||||
async def reveal_builtin_config_secrets(
|
||||
request: Request,
|
||||
name: str = Query(..., min_length=1),
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""Reveal configured built-in datasource credentials for admin editing."""
|
||||
source = name.strip()
|
||||
metadata = DEFAULT_DATASOURCES.get(source)
|
||||
if not metadata or not metadata.get("requires_credentials"):
|
||||
raise HTTPException(status_code=404, detail="Credentialed datasource config not found")
|
||||
|
||||
provider = str(metadata.get("credential_provider") or "")
|
||||
target_id = f"datasource_config:{source}"
|
||||
await _ensure_datasource_secret_reveal_allowed(
|
||||
current_user,
|
||||
request,
|
||||
target_id,
|
||||
{"source": source, "provider": provider},
|
||||
)
|
||||
|
||||
result = await db.execute(select(DataSourceConfig).where(DataSourceConfig.name == source))
|
||||
record = result.scalar_one_or_none()
|
||||
auth_config = dict(record.auth_config or {}) if record else {}
|
||||
payload: dict[str, Any] = {
|
||||
"name": source,
|
||||
"provider": provider,
|
||||
}
|
||||
details: dict[str, Any] = {"source": source, "provider": provider}
|
||||
|
||||
if provider == "barentswatch":
|
||||
resolved = await resolve_barentswatch_config(db)
|
||||
client_id = str(auth_config.get("client_id") or resolved.client_id or "")
|
||||
client_secret = str(auth_config.get("client_secret") or resolved.client_secret or "")
|
||||
source_label = "datasource_config" if auth_config.get("client_id") or auth_config.get("client_secret") else resolved.credential_source
|
||||
payload.update(
|
||||
{
|
||||
"client_id": client_id,
|
||||
"client_secret": client_secret,
|
||||
"client_id_source": source_label if client_id else "missing",
|
||||
"client_secret_source": source_label if client_secret else "missing",
|
||||
}
|
||||
)
|
||||
details.update(
|
||||
{
|
||||
"client_id_configured": bool(client_id),
|
||||
"client_secret_configured": bool(client_secret),
|
||||
"credential_source": source_label,
|
||||
}
|
||||
)
|
||||
elif provider == "aisstream":
|
||||
api_key, api_key_source = await _resolve_aisstream_api_key(db)
|
||||
payload.update({"api_key": api_key, "api_key_source": api_key_source})
|
||||
details.update({"api_key_configured": bool(api_key), "api_key_source": api_key_source})
|
||||
elif provider == "spacetrack":
|
||||
if auth_config.get("username") or auth_config.get("password"):
|
||||
username = str(auth_config.get("username") or "")
|
||||
password = str(auth_config.get("password") or "")
|
||||
credential_source = "datasource_config"
|
||||
else:
|
||||
username, password, credential_source = _resolve_spacetrack_credentials_with_override()
|
||||
payload.update(
|
||||
{
|
||||
"username": username,
|
||||
"password": password,
|
||||
"username_source": credential_source if username else "missing",
|
||||
"password_source": credential_source if password else "missing",
|
||||
}
|
||||
)
|
||||
details.update(
|
||||
{
|
||||
"username_configured": bool(username),
|
||||
"password_configured": bool(password),
|
||||
"credential_source": credential_source,
|
||||
}
|
||||
)
|
||||
else:
|
||||
raise HTTPException(status_code=400, detail="Datasource credential provider is not supported")
|
||||
|
||||
await _record_datasource_secret_reveal(
|
||||
current_user=current_user,
|
||||
request=request,
|
||||
target_id=target_id,
|
||||
result="success",
|
||||
details=details,
|
||||
)
|
||||
return payload
|
||||
|
||||
|
||||
@router.get("/configs/{config_id}")
|
||||
async def get_config(
|
||||
config_id: int,
|
||||
@@ -757,14 +931,12 @@ async def propose_datasource_mapping(
|
||||
generated_by = "heuristic"
|
||||
if payload.use_ai:
|
||||
try:
|
||||
prompt = await get_effective_prompt(db, DATASOURCE_MAPPING_PROMPT_KEY)
|
||||
response = await ai_client.analyze(
|
||||
SituationalAnalysisRequest(
|
||||
title=f"Generate datasource mapping for {schema.key}",
|
||||
objective=(
|
||||
"Return only JSON for a deterministic mapping DSL. "
|
||||
"The JSON must contain source.items_path and fields. "
|
||||
"Do not include prose or code."
|
||||
),
|
||||
objective=prompt.prompt,
|
||||
system_prompt=prompt.system_prompt or None,
|
||||
context={
|
||||
"target_schema": schema.to_dict(),
|
||||
"sample_payload": redacted_sample,
|
||||
|
||||
@@ -7,6 +7,7 @@ from pydantic import BaseModel, Field
|
||||
from sqlalchemy import func, or_, select, text
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.cache import cache
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.core.security import get_current_user
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
@@ -18,12 +19,15 @@ from app.models.datasource import DataSource
|
||||
from app.models.datasource_config import DataSourceConfig
|
||||
from app.models.task import CollectionTask
|
||||
from app.models.user import User
|
||||
from app.models.vessel import AISRawObservation
|
||||
from app.services.vessel_ais_aggregation import VESSEL_AIS_SCHEMA
|
||||
from app.services.scheduler import (
|
||||
cancel_running_collector_now,
|
||||
get_latest_task_id_for_datasource,
|
||||
run_collector_now,
|
||||
sync_datasource_job,
|
||||
)
|
||||
from app.services.earth_layer_cache import invalidate_earth_layer_cache_for_source
|
||||
|
||||
router = APIRouter()
|
||||
STALE_RUNNING_TASK_TIMEOUT_MINUTES = 90
|
||||
@@ -148,6 +152,33 @@ async def _load_latest_task_ids(
|
||||
return {datasource_id: task_id for datasource_id, task_id in result.all()}
|
||||
|
||||
|
||||
async def _load_latest_tasks(
|
||||
db: AsyncSession,
|
||||
datasource_ids: list[int],
|
||||
) -> dict[int, CollectionTask]:
|
||||
if not datasource_ids:
|
||||
return {}
|
||||
|
||||
ranked_tasks = (
|
||||
select(
|
||||
CollectionTask.id.label("task_id"),
|
||||
CollectionTask.datasource_id.label("datasource_id"),
|
||||
func.row_number().over(
|
||||
partition_by=CollectionTask.datasource_id,
|
||||
order_by=(CollectionTask.created_at.desc().nullslast(), CollectionTask.id.desc()),
|
||||
).label("row_num"),
|
||||
)
|
||||
.where(CollectionTask.datasource_id.in_(datasource_ids))
|
||||
.subquery()
|
||||
)
|
||||
result = await db.execute(
|
||||
select(CollectionTask)
|
||||
.join(ranked_tasks, CollectionTask.id == ranked_tasks.c.task_id)
|
||||
.where(ranked_tasks.c.row_num == 1)
|
||||
)
|
||||
return {task.datasource_id: task for task in result.scalars().all()}
|
||||
|
||||
|
||||
async def _load_collected_record_counts(
|
||||
db: AsyncSession,
|
||||
sources: list[str],
|
||||
@@ -161,7 +192,26 @@ async def _load_collected_record_counts(
|
||||
.where(CollectedData.is_current.is_(True))
|
||||
.group_by(CollectedData.source)
|
||||
)
|
||||
return {source: int(count or 0) for source, count in result.all()}
|
||||
counts = {source: int(count or 0) for source, count in result.all()}
|
||||
|
||||
vessel_sources = [
|
||||
source
|
||||
for source in sources
|
||||
if datasource_metadata(source)["credential_provider"] in {"aisstream", "barentswatch"}
|
||||
or "vessel" in source
|
||||
or "ais" in source
|
||||
]
|
||||
if vessel_sources:
|
||||
raw_result = await db.execute(
|
||||
select(AISRawObservation.source, func.count(AISRawObservation.id))
|
||||
.where(AISRawObservation.target_schema == VESSEL_AIS_SCHEMA)
|
||||
.where(AISRawObservation.source.in_(vessel_sources))
|
||||
.group_by(AISRawObservation.source)
|
||||
)
|
||||
for source, count in raw_result.all():
|
||||
counts[source] = max(counts.get(source, 0), int(count or 0))
|
||||
|
||||
return counts
|
||||
|
||||
|
||||
async def _load_datasource_endpoint_overrides(
|
||||
@@ -187,7 +237,9 @@ async def _load_datasource_endpoint_overrides(
|
||||
async def _load_datasource_list_context(
|
||||
db: AsyncSession,
|
||||
datasources: list[DataSource],
|
||||
) -> tuple[dict[int, CollectionTask], dict[str, str]]:
|
||||
*,
|
||||
include_endpoint: bool = True,
|
||||
) -> tuple[dict[int, CollectionTask], dict[int, CollectionTask], dict[str, str]]:
|
||||
datasource_ids = [datasource.id for datasource in datasources]
|
||||
sources = [datasource.source for datasource in datasources]
|
||||
|
||||
@@ -211,8 +263,64 @@ async def _load_datasource_list_context(
|
||||
if stale_datasource_ids:
|
||||
running_tasks = await _load_latest_running_tasks(db, datasource_ids)
|
||||
|
||||
endpoint_overrides = await _load_datasource_endpoint_overrides(db, sources)
|
||||
return running_tasks, endpoint_overrides
|
||||
latest_tasks = await _load_latest_tasks(db, datasource_ids)
|
||||
endpoint_overrides = await _load_datasource_endpoint_overrides(db, sources) if include_endpoint else {}
|
||||
return running_tasks, latest_tasks, endpoint_overrides
|
||||
|
||||
|
||||
def serialize_datasource_row(
|
||||
datasource: DataSource,
|
||||
*,
|
||||
running_tasks: dict[int, CollectionTask],
|
||||
latest_tasks: dict[int, CollectionTask],
|
||||
record_counts: dict[str, int],
|
||||
endpoint_overrides: dict[str, str],
|
||||
config,
|
||||
include_endpoint: bool,
|
||||
) -> dict:
|
||||
running_task = running_tasks.get(datasource.id)
|
||||
latest_task = latest_tasks.get(datasource.id)
|
||||
display_task = running_task or latest_task
|
||||
endpoint = None
|
||||
if include_endpoint:
|
||||
endpoint = endpoint_overrides.get(datasource.source) or config.get_yaml_url(datasource.source)
|
||||
last_run_at = datasource.last_run_at or (latest_task.completed_at if latest_task else None)
|
||||
last_status = datasource.last_status or (latest_task.status if latest_task else None)
|
||||
collected_records = record_counts.get(datasource.source, 0)
|
||||
|
||||
row = {
|
||||
"id": datasource.id,
|
||||
"source": datasource.source,
|
||||
"name": datasource.name,
|
||||
**datasource_metadata(datasource.source),
|
||||
"product": datasource_product_key(datasource),
|
||||
"module": datasource.module,
|
||||
"priority": datasource.priority,
|
||||
"frequency": format_frequency_label(datasource.frequency_minutes),
|
||||
"frequency_minutes": datasource.frequency_minutes,
|
||||
"is_active": datasource.is_active,
|
||||
"collector_class": datasource.collector_class,
|
||||
"last_run": to_iso8601_utc(last_run_at),
|
||||
"last_run_at": to_iso8601_utc(last_run_at),
|
||||
"last_status": last_status,
|
||||
"is_running": running_task is not None,
|
||||
"task_id": display_task.id if display_task else None,
|
||||
"progress": display_task.progress if display_task else None,
|
||||
"phase": display_task.phase if display_task else None,
|
||||
"phase_progress": display_task.phase_progress if display_task else None,
|
||||
"phase_message": display_task.phase_message if display_task else None,
|
||||
"phase_current": display_task.phase_current if display_task else None,
|
||||
"phase_total": display_task.phase_total if display_task else None,
|
||||
"phase_unit": display_task.phase_unit if display_task else None,
|
||||
"records_processed": display_task.records_processed if display_task else None,
|
||||
"total_records": display_task.total_records if display_task else None,
|
||||
"error_message": display_task.error_message if display_task else None,
|
||||
"collected_records": collected_records,
|
||||
"has_collected_data": collected_records > 0,
|
||||
}
|
||||
if include_endpoint:
|
||||
row["endpoint"] = endpoint
|
||||
return row
|
||||
|
||||
|
||||
def _apply_datasource_query_filters(
|
||||
@@ -230,11 +338,6 @@ def _apply_datasource_query_filters(
|
||||
query = query.where(DataSource.is_active == is_active)
|
||||
if priority:
|
||||
query = query.where(DataSource.priority == priority)
|
||||
if run_status and run_status not in {"running", "collected", "uncollected"}:
|
||||
if run_status == "not_run":
|
||||
query = query.where(DataSource.last_status.is_(None))
|
||||
else:
|
||||
query = query.where(DataSource.last_status == run_status)
|
||||
if q:
|
||||
like_value = f"%{q.strip()}%"
|
||||
query = query.where(
|
||||
@@ -251,15 +354,25 @@ def _filter_datasources_in_memory(
|
||||
datasources: list[DataSource],
|
||||
*,
|
||||
running_tasks: dict[int, CollectionTask],
|
||||
latest_tasks: dict[int, CollectionTask] | None = None,
|
||||
record_counts: dict[str, int],
|
||||
product: Optional[str] = None,
|
||||
run_status: Optional[str] = None,
|
||||
collected: Optional[bool] = None,
|
||||
credential_status: Optional[str] = None,
|
||||
) -> list[DataSource]:
|
||||
latest_tasks = latest_tasks or {}
|
||||
filtered: list[DataSource] = []
|
||||
for datasource in datasources:
|
||||
record_count = record_counts.get(datasource.source, 0)
|
||||
latest_task = latest_tasks.get(datasource.id)
|
||||
effective_status = (
|
||||
"running"
|
||||
if datasource.id in running_tasks
|
||||
else latest_task.status
|
||||
if latest_task is not None
|
||||
else datasource.last_status
|
||||
)
|
||||
if product and datasource_product_key(datasource) != product:
|
||||
continue
|
||||
if collected is not None and (record_count > 0) != collected:
|
||||
@@ -270,6 +383,10 @@ def _filter_datasources_in_memory(
|
||||
continue
|
||||
if run_status == "running" and datasource.id not in running_tasks:
|
||||
continue
|
||||
if run_status == "not_run" and effective_status is not None:
|
||||
continue
|
||||
if run_status not in {None, "running", "not_run", "collected", "uncollected"} and effective_status != run_status:
|
||||
continue
|
||||
if run_status == "collected" and record_count <= 0:
|
||||
continue
|
||||
if run_status == "uncollected" and record_count > 0:
|
||||
@@ -600,6 +717,7 @@ async def list_datasources(
|
||||
collected: Optional[bool] = None,
|
||||
credential_status: Optional[str] = None,
|
||||
q: Optional[str] = None,
|
||||
include_endpoint: bool = True,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
@@ -618,11 +736,16 @@ async def list_datasources(
|
||||
|
||||
collector_list = []
|
||||
config = get_data_sources_config()
|
||||
running_tasks, endpoint_overrides = await _load_datasource_list_context(db, datasources)
|
||||
running_tasks, latest_tasks, endpoint_overrides = await _load_datasource_list_context(
|
||||
db,
|
||||
datasources,
|
||||
include_endpoint=include_endpoint,
|
||||
)
|
||||
record_counts = await _load_collected_record_counts(db, [datasource.source for datasource in datasources])
|
||||
datasources = _filter_datasources_in_memory(
|
||||
datasources,
|
||||
running_tasks=running_tasks,
|
||||
latest_tasks=latest_tasks,
|
||||
record_counts=record_counts,
|
||||
product=product,
|
||||
run_status=run_status,
|
||||
@@ -630,43 +753,16 @@ async def list_datasources(
|
||||
credential_status=credential_status,
|
||||
)
|
||||
for datasource in datasources:
|
||||
running_task = running_tasks.get(datasource.id)
|
||||
endpoint = endpoint_overrides.get(datasource.source) or config.get_yaml_url(datasource.source)
|
||||
last_run_at = datasource.last_run_at
|
||||
last_status = datasource.last_status
|
||||
collected_records = record_counts.get(datasource.source, 0)
|
||||
|
||||
collector_list.append(
|
||||
{
|
||||
"id": datasource.id,
|
||||
"source": datasource.source,
|
||||
"name": datasource.name,
|
||||
**datasource_metadata(datasource.source),
|
||||
"product": datasource_product_key(datasource),
|
||||
"module": datasource.module,
|
||||
"priority": datasource.priority,
|
||||
"frequency": format_frequency_label(datasource.frequency_minutes),
|
||||
"frequency_minutes": datasource.frequency_minutes,
|
||||
"is_active": datasource.is_active,
|
||||
"collector_class": datasource.collector_class,
|
||||
"endpoint": endpoint,
|
||||
"last_run": to_iso8601_utc(last_run_at),
|
||||
"last_run_at": to_iso8601_utc(last_run_at),
|
||||
"last_status": last_status,
|
||||
"is_running": running_task is not None,
|
||||
"task_id": running_task.id if running_task else None,
|
||||
"progress": running_task.progress if running_task else None,
|
||||
"phase": running_task.phase if running_task else None,
|
||||
"phase_progress": running_task.phase_progress if running_task else None,
|
||||
"phase_message": running_task.phase_message if running_task else None,
|
||||
"phase_current": running_task.phase_current if running_task else None,
|
||||
"phase_total": running_task.phase_total if running_task else None,
|
||||
"phase_unit": running_task.phase_unit if running_task else None,
|
||||
"records_processed": running_task.records_processed if running_task else None,
|
||||
"total_records": running_task.total_records if running_task else None,
|
||||
"collected_records": collected_records,
|
||||
"has_collected_data": collected_records > 0,
|
||||
}
|
||||
serialize_datasource_row(
|
||||
datasource,
|
||||
running_tasks=running_tasks,
|
||||
latest_tasks=latest_tasks,
|
||||
record_counts=record_counts,
|
||||
endpoint_overrides=endpoint_overrides,
|
||||
config=config,
|
||||
include_endpoint=include_endpoint,
|
||||
)
|
||||
)
|
||||
|
||||
return {"total": len(collector_list), "data": collector_list}
|
||||
@@ -708,11 +804,12 @@ async def trigger_datasource_batch(
|
||||
|
||||
result = await db.execute(query)
|
||||
datasources = result.scalars().all()
|
||||
running_tasks, _ = await _load_datasource_list_context(db, datasources)
|
||||
running_tasks, latest_tasks, _ = await _load_datasource_list_context(db, datasources)
|
||||
record_counts = await _load_collected_record_counts(db, [datasource.source for datasource in datasources])
|
||||
datasources = _filter_datasources_in_memory(
|
||||
datasources,
|
||||
running_tasks=running_tasks,
|
||||
latest_tasks=latest_tasks,
|
||||
record_counts=record_counts,
|
||||
product=None if payload.source_ids else payload.product,
|
||||
run_status=None if payload.source_ids else payload.run_status,
|
||||
@@ -722,6 +819,51 @@ async def trigger_datasource_batch(
|
||||
return await _trigger_datasource_batch(db, datasources, force=payload.force)
|
||||
|
||||
|
||||
@router.get("/snapshots")
|
||||
async def list_datasource_snapshots(
|
||||
source_id: Optional[str] = None,
|
||||
current_only: Optional[bool] = None,
|
||||
limit: int = Query(default=100, ge=1, le=500),
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
query = (
|
||||
select(DataSnapshot, DataSource.name, DataSource.module)
|
||||
.outerjoin(DataSource, DataSource.id == DataSnapshot.datasource_id)
|
||||
.order_by(DataSnapshot.created_at.desc().nullslast(), DataSnapshot.id.desc())
|
||||
.limit(limit)
|
||||
)
|
||||
if source_id:
|
||||
query = query.where(DataSnapshot.source == source_id)
|
||||
if current_only is not None:
|
||||
query = query.where(DataSnapshot.is_current.is_(current_only))
|
||||
|
||||
result = await db.execute(query)
|
||||
rows = []
|
||||
for snapshot, datasource_name, datasource_module in result.all():
|
||||
rows.append(
|
||||
{
|
||||
"id": snapshot.id,
|
||||
"datasource_id": snapshot.datasource_id,
|
||||
"datasource_name": datasource_name,
|
||||
"module": datasource_module,
|
||||
"task_id": snapshot.task_id,
|
||||
"source": snapshot.source,
|
||||
"snapshot_key": snapshot.snapshot_key,
|
||||
"reference_date": to_iso8601_utc(snapshot.reference_date),
|
||||
"started_at": to_iso8601_utc(snapshot.started_at),
|
||||
"completed_at": to_iso8601_utc(snapshot.completed_at),
|
||||
"record_count": snapshot.record_count,
|
||||
"status": snapshot.status,
|
||||
"is_current": snapshot.is_current,
|
||||
"parent_snapshot_id": snapshot.parent_snapshot_id,
|
||||
"summary": snapshot.summary or {},
|
||||
"created_at": to_iso8601_utc(snapshot.created_at),
|
||||
}
|
||||
)
|
||||
return {"total": len(rows), "data": rows}
|
||||
|
||||
|
||||
@router.get("/{source_id}")
|
||||
async def get_datasource(
|
||||
source_id: str,
|
||||
@@ -750,6 +892,37 @@ async def get_datasource(
|
||||
}
|
||||
|
||||
|
||||
@router.get("/{source_id}/row")
|
||||
async def get_datasource_row(
|
||||
source_id: str,
|
||||
include_endpoint: bool = True,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
datasource = await get_datasource_record(db, source_id)
|
||||
if not datasource:
|
||||
raise HTTPException(status_code=404, detail="Data source not found")
|
||||
|
||||
config = get_data_sources_config()
|
||||
running_tasks, latest_tasks, endpoint_overrides = await _load_datasource_list_context(
|
||||
db,
|
||||
[datasource],
|
||||
include_endpoint=include_endpoint,
|
||||
)
|
||||
record_counts = await _load_collected_record_counts(db, [datasource.source])
|
||||
return {
|
||||
"data": serialize_datasource_row(
|
||||
datasource,
|
||||
running_tasks=running_tasks,
|
||||
latest_tasks=latest_tasks,
|
||||
record_counts=record_counts,
|
||||
endpoint_overrides=endpoint_overrides,
|
||||
config=config,
|
||||
include_endpoint=include_endpoint,
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
@router.post("/{source_id}/enable")
|
||||
async def enable_datasource(
|
||||
source_id: str,
|
||||
@@ -897,6 +1070,29 @@ async def clear_datasource_data(
|
||||
}
|
||||
|
||||
|
||||
@router.delete("/{source_id}/cache")
|
||||
async def clear_datasource_cache(
|
||||
source_id: str,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
datasource = await get_datasource_record(db, source_id)
|
||||
if not datasource:
|
||||
raise HTTPException(status_code=404, detail="Data source not found")
|
||||
|
||||
earth_deleted_count = invalidate_earth_layer_cache_for_source(datasource.source)
|
||||
dashboard_deleted_count = int(cache.delete("dashboard:stats")) + int(cache.delete("dashboard:summary"))
|
||||
deleted_count = earth_deleted_count + dashboard_deleted_count
|
||||
|
||||
return {
|
||||
"status": "success",
|
||||
"message": f"Cleared {deleted_count} cache keys for data source '{datasource.name}'",
|
||||
"deleted_count": deleted_count,
|
||||
"earth_layer_deleted_count": earth_deleted_count,
|
||||
"dashboard_deleted_count": dashboard_deleted_count,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/{source_id}/task-status")
|
||||
async def get_task_status(
|
||||
source_id: str,
|
||||
@@ -914,6 +1110,14 @@ async def get_task_status(
|
||||
raise HTTPException(status_code=404, detail="Task not found")
|
||||
else:
|
||||
task = await get_running_task(db, datasource.id)
|
||||
if task is None:
|
||||
result = await db.execute(
|
||||
select(CollectionTask)
|
||||
.where(CollectionTask.datasource_id == datasource.id)
|
||||
.order_by(CollectionTask.created_at.desc().nullslast(), CollectionTask.id.desc())
|
||||
.limit(1)
|
||||
)
|
||||
task = result.scalar_one_or_none()
|
||||
|
||||
if not task:
|
||||
return {
|
||||
@@ -942,4 +1146,5 @@ async def get_task_status(
|
||||
"records_processed": task.records_processed,
|
||||
"total_records": task.total_records,
|
||||
"status": task.status,
|
||||
"error_message": task.error_message,
|
||||
}
|
||||
|
||||
247
backend/app/api/v1/earth.py
Normal file
247
backend/app/api/v1/earth.py
Normal file
@@ -0,0 +1,247 @@
|
||||
"""Earth asset management APIs."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from uuid import uuid4
|
||||
|
||||
from fastapi import APIRouter, Depends, File, HTTPException, Request, UploadFile, status
|
||||
from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
|
||||
from pydantic import BaseModel, Field
|
||||
from sqlalchemy import delete, select, text
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.security import decode_token, get_current_user, redis_client
|
||||
from app.db.session import get_db
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.models.user import User
|
||||
from app.services.earth_boundaries import (
|
||||
EarthBoundaryBuildError,
|
||||
get_boundary_build_status,
|
||||
get_boundary_status,
|
||||
save_boundary_config,
|
||||
start_boundary_build_job,
|
||||
)
|
||||
|
||||
|
||||
router = APIRouter()
|
||||
optional_bearer = HTTPBearer(auto_error=False)
|
||||
REPO_ROOT = Path(__file__).resolve().parents[4]
|
||||
EARTH_BRAND_ASSET_DIR = REPO_ROOT / "data" / "earth-brand"
|
||||
EARTH_BRAND_ASSET_URL_PREFIX = "/earth-brand-assets"
|
||||
EARTH_BRAND_CATEGORY = "earth_brand"
|
||||
MAX_EARTH_BRAND_ASSET_BYTES = 3 * 1024 * 1024
|
||||
ALLOWED_EARTH_BRAND_EXTENSIONS = {".png", ".jpg", ".jpeg", ".webp", ".svg"}
|
||||
|
||||
DEFAULT_EARTH_BRAND = {
|
||||
"logo_src": "/earth/assets/brand/earth-logo.png",
|
||||
"title_src": "/earth/assets/brand/title-zh.png",
|
||||
"title_text": "智能星球计划",
|
||||
"subtitle": "现实层宇宙全息感知系统",
|
||||
"description": "卫星 · 海底光缆 · 算力基础设施",
|
||||
"aria_label": "智能星球计划品牌标识",
|
||||
"title_alt": "智能星球计划",
|
||||
}
|
||||
|
||||
|
||||
class EarthBoundaryConfigPayload(BaseModel):
|
||||
config: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class EarthBrandPayload(BaseModel):
|
||||
logo_src: str = Field(default=DEFAULT_EARTH_BRAND["logo_src"], max_length=1000)
|
||||
title_src: str = Field(default=DEFAULT_EARTH_BRAND["title_src"], max_length=1000)
|
||||
title_text: str = Field(default=DEFAULT_EARTH_BRAND["title_text"], max_length=120)
|
||||
subtitle: str = Field(default=DEFAULT_EARTH_BRAND["subtitle"], max_length=160)
|
||||
description: str = Field(default=DEFAULT_EARTH_BRAND["description"], max_length=200)
|
||||
aria_label: str = Field(default=DEFAULT_EARTH_BRAND["aria_label"], max_length=200)
|
||||
title_alt: str = Field(default=DEFAULT_EARTH_BRAND["title_alt"], max_length=200)
|
||||
|
||||
|
||||
def _normalize_earth_brand_payload(payload: dict[str, Any] | None) -> dict[str, str]:
|
||||
merged = DEFAULT_EARTH_BRAND.copy()
|
||||
if payload:
|
||||
for key in DEFAULT_EARTH_BRAND:
|
||||
value = payload.get(key)
|
||||
if value is not None:
|
||||
merged[key] = str(value).strip()
|
||||
|
||||
if not merged["title_text"]:
|
||||
merged["title_text"] = DEFAULT_EARTH_BRAND["title_text"]
|
||||
if not merged["aria_label"]:
|
||||
merged["aria_label"] = merged["title_text"]
|
||||
if not merged["title_alt"]:
|
||||
merged["title_alt"] = merged["title_text"]
|
||||
return merged
|
||||
|
||||
|
||||
async def _get_earth_brand_record(db: AsyncSession) -> SystemSetting | None:
|
||||
result = await db.execute(
|
||||
select(SystemSetting).where(SystemSetting.category == EARTH_BRAND_CATEGORY)
|
||||
)
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
|
||||
async def _get_earth_brand_payload(db: AsyncSession) -> dict[str, Any]:
|
||||
record = await _get_earth_brand_record(db)
|
||||
return {
|
||||
"brand": _normalize_earth_brand_payload(record.payload if record else None),
|
||||
"is_default": record is None,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/brand")
|
||||
async def get_earth_brand(db: AsyncSession = Depends(get_db)):
|
||||
return await _get_earth_brand_payload(db)
|
||||
|
||||
|
||||
@router.put("/brand")
|
||||
async def update_earth_brand(
|
||||
payload: EarthBrandPayload,
|
||||
_current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
normalized = _normalize_earth_brand_payload(payload.model_dump())
|
||||
record = await _get_earth_brand_record(db)
|
||||
if record is None:
|
||||
record = SystemSetting(category=EARTH_BRAND_CATEGORY, payload=normalized)
|
||||
db.add(record)
|
||||
else:
|
||||
record.payload = normalized
|
||||
await db.commit()
|
||||
await db.refresh(record)
|
||||
return {"status": "updated", "brand": _normalize_earth_brand_payload(record.payload), "is_default": False}
|
||||
|
||||
|
||||
@router.delete("/brand")
|
||||
@router.post("/brand/reset")
|
||||
async def reset_earth_brand(
|
||||
_current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
await db.execute(delete(SystemSetting).where(SystemSetting.category == EARTH_BRAND_CATEGORY))
|
||||
await db.commit()
|
||||
return {"status": "reset", "brand": DEFAULT_EARTH_BRAND.copy(), "is_default": True}
|
||||
|
||||
|
||||
@router.post("/brand/assets")
|
||||
async def upload_earth_brand_asset(
|
||||
file: UploadFile = File(...),
|
||||
_current_user: User = Depends(get_current_user),
|
||||
):
|
||||
original_name = file.filename or ""
|
||||
extension = Path(original_name).suffix.lower()
|
||||
if extension not in ALLOWED_EARTH_BRAND_EXTENSIONS:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail={
|
||||
"code": "unsupported_file_type",
|
||||
"message": "Only png, jpg, jpeg, webp, and svg brand assets are supported.",
|
||||
},
|
||||
)
|
||||
|
||||
content = await file.read(MAX_EARTH_BRAND_ASSET_BYTES + 1)
|
||||
if len(content) > MAX_EARTH_BRAND_ASSET_BYTES:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail={
|
||||
"code": "file_too_large",
|
||||
"message": "Brand asset must be 3 MB or smaller.",
|
||||
},
|
||||
)
|
||||
|
||||
EARTH_BRAND_ASSET_DIR.mkdir(parents=True, exist_ok=True)
|
||||
safe_name = f"{uuid4().hex}{extension}"
|
||||
destination = EARTH_BRAND_ASSET_DIR / safe_name
|
||||
destination.write_bytes(content)
|
||||
asset_url = f"{EARTH_BRAND_ASSET_URL_PREFIX}/{safe_name}"
|
||||
return {"url": asset_url, "filename": safe_name, "content_type": file.content_type}
|
||||
|
||||
|
||||
@router.get("/boundaries/status")
|
||||
async def get_earth_boundary_status():
|
||||
return get_boundary_status()
|
||||
|
||||
|
||||
async def _get_optional_current_user(
|
||||
credentials: HTTPAuthorizationCredentials | None = Depends(optional_bearer),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
) -> User | None:
|
||||
if credentials is None:
|
||||
return None
|
||||
token = credentials.credentials
|
||||
if redis_client.sismember("blacklisted_tokens", token):
|
||||
return None
|
||||
payload = decode_token(token)
|
||||
if payload is None or payload.get("type") != "access":
|
||||
return None
|
||||
user_id = payload.get("sub")
|
||||
if user_id is None:
|
||||
return None
|
||||
result = await db.execute(
|
||||
text(
|
||||
"SELECT id, username, email, password_hash, role, is_active, gatekeeper_groups FROM users WHERE id = :id"
|
||||
),
|
||||
{"id": int(user_id)},
|
||||
)
|
||||
row = result.fetchone()
|
||||
if row is None or not row[5]:
|
||||
return None
|
||||
user = User()
|
||||
user.id = row[0]
|
||||
user.username = row[1]
|
||||
user.email = row[2]
|
||||
user.password_hash = row[3]
|
||||
user.role = row[4]
|
||||
user.is_active = row[5]
|
||||
user.gatekeeper_groups = row[6] or []
|
||||
return user
|
||||
|
||||
|
||||
def _is_loopback_request(request: Request) -> bool:
|
||||
host = request.client.host if request.client else ""
|
||||
return host in {"127.0.0.1", "::1", "localhost"} or host.startswith("127.")
|
||||
|
||||
|
||||
def _require_local_or_user(request: Request, user: User | None) -> None:
|
||||
if user is not None or _is_loopback_request(request):
|
||||
return
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="Authentication required outside localhost",
|
||||
)
|
||||
|
||||
|
||||
@router.put("/boundaries/config")
|
||||
async def update_earth_boundary_config(
|
||||
payload: EarthBoundaryConfigPayload,
|
||||
_current_user: User = Depends(get_current_user),
|
||||
):
|
||||
try:
|
||||
return save_boundary_config(payload.config)
|
||||
except EarthBoundaryBuildError as exc:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail={"code": exc.code, "message": str(exc), "details": exc.details},
|
||||
) from exc
|
||||
|
||||
|
||||
@router.post("/boundaries/build")
|
||||
async def build_earth_boundary_assets(
|
||||
request: Request,
|
||||
current_user: User | None = Depends(_get_optional_current_user),
|
||||
):
|
||||
_require_local_or_user(request, current_user)
|
||||
try:
|
||||
return await start_boundary_build_job()
|
||||
except EarthBoundaryBuildError as exc:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail={"code": exc.code, "message": str(exc), "details": exc.details},
|
||||
) from exc
|
||||
|
||||
|
||||
@router.get("/boundaries/build/status")
|
||||
async def get_earth_boundary_build_status():
|
||||
return get_boundary_build_status()
|
||||
@@ -1,6 +1,6 @@
|
||||
from typing import Any, Optional
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Response
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.api.v1.visualization import (
|
||||
@@ -110,6 +110,7 @@ async def get_vessel_layer_snapshot(
|
||||
vessel_type: Optional[str] = Query(None, alias="type"),
|
||||
since_minutes: int = Query(60, ge=1, le=1440),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
response: Response = None,
|
||||
):
|
||||
parsed_bbox = _parse_layer_bbox(bbox)
|
||||
return await build_vessel_snapshot_response(
|
||||
@@ -117,8 +118,9 @@ async def get_vessel_layer_snapshot(
|
||||
bbox=parsed_bbox,
|
||||
zoom=zoom,
|
||||
limit=limit,
|
||||
vessel_type=vessel_type,
|
||||
type_filter=vessel_type,
|
||||
since_minutes=since_minutes,
|
||||
response=response,
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
from fastapi import APIRouter, Query
|
||||
from fastapi import APIRouter, Depends, Query
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.db.session import get_db
|
||||
from app.services.earth_news import get_earth_news_payload
|
||||
|
||||
router = APIRouter()
|
||||
@@ -9,5 +11,6 @@ router = APIRouter()
|
||||
async def get_earth_feed(
|
||||
lat: float | None = Query(None, description="Current Earth view center latitude"),
|
||||
lon: float | None = Query(None, description="Current Earth view center longitude"),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await get_earth_news_payload(lat=lat, lon=lon)
|
||||
return await get_earth_news_payload(lat=lat, lon=lon, db=db)
|
||||
|
||||
280
backend/app/api/v1/realtime_sources.py
Normal file
280
backend/app/api/v1/realtime_sources.py
Normal file
@@ -0,0 +1,280 @@
|
||||
"""Realtime datasource operations and runtime statistics."""
|
||||
|
||||
from datetime import UTC, datetime, timedelta
|
||||
import os
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from sqlalchemy import distinct, func, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
from app.core.security import get_current_user
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.db.session import get_db
|
||||
from app.models.datasource import DataSource
|
||||
from app.models.datasource_config import DataSourceConfig
|
||||
from app.models.user import User
|
||||
from app.models.vessel import AISRawObservation, AISSourceHealth
|
||||
from app.services.custom_datasource_runtime import (
|
||||
get_custom_stream_status,
|
||||
start_custom_stream,
|
||||
stop_custom_stream,
|
||||
)
|
||||
from app.services.scheduler import (
|
||||
cancel_running_collector_now,
|
||||
is_collector_running,
|
||||
run_collector_now,
|
||||
)
|
||||
from app.services.vessel_ais_aggregation import VESSEL_AIS_SCHEMA, update_ais_source_health
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
BUILTIN_REALTIME_SOURCES = {"aisstream_vessels"}
|
||||
REALTIME_SOURCE_TYPES = {"websocket", "ws"}
|
||||
|
||||
|
||||
def _is_realtime_config(config: DataSourceConfig) -> bool:
|
||||
return str(config.source_type or "").lower() in REALTIME_SOURCE_TYPES
|
||||
|
||||
|
||||
def _safe_config_dict(value: Any) -> dict[str, Any]:
|
||||
return value if isinstance(value, dict) else {}
|
||||
|
||||
|
||||
def _credential_configured(config: DataSourceConfig | None) -> bool:
|
||||
if config is not None:
|
||||
auth_config = _safe_config_dict(config.auth_config)
|
||||
config_payload = _safe_config_dict(config.config)
|
||||
if auth_config.get("api_key") or config_payload.get("api_key"):
|
||||
return True
|
||||
return bool(os.getenv("AISSTREAM_API_KEY"))
|
||||
|
||||
|
||||
async def _load_realtime_stats(db: AsyncSession, source: str) -> dict[str, Any]:
|
||||
now = datetime.now(UTC)
|
||||
observed_24h = now - timedelta(hours=24)
|
||||
observed_1h = now - timedelta(hours=1)
|
||||
payload_mmsi = AISRawObservation.entity_key
|
||||
|
||||
result = await db.execute(
|
||||
select(
|
||||
func.count(AISRawObservation.id).label("total_observations"),
|
||||
func.count(AISRawObservation.id)
|
||||
.filter(AISRawObservation.observed_at >= observed_24h)
|
||||
.label("observations_24h"),
|
||||
func.count(AISRawObservation.id)
|
||||
.filter(AISRawObservation.observed_at >= observed_1h)
|
||||
.label("observations_1h"),
|
||||
func.count(distinct(payload_mmsi)).label("unique_mmsi_total"),
|
||||
func.count(distinct(payload_mmsi))
|
||||
.filter(AISRawObservation.observed_at >= observed_24h)
|
||||
.label("unique_mmsi_24h"),
|
||||
func.max(AISRawObservation.observed_at).label("latest_observed_at"),
|
||||
func.max(AISRawObservation.collected_at).label("latest_collected_at"),
|
||||
)
|
||||
.where(AISRawObservation.target_schema == VESSEL_AIS_SCHEMA)
|
||||
.where(AISRawObservation.source == source)
|
||||
)
|
||||
row = result.mappings().one()
|
||||
return {
|
||||
"total_observations": int(row["total_observations"] or 0),
|
||||
"observations_24h": int(row["observations_24h"] or 0),
|
||||
"observations_1h": int(row["observations_1h"] or 0),
|
||||
"unique_mmsi_total": int(row["unique_mmsi_total"] or 0),
|
||||
"unique_mmsi_24h": int(row["unique_mmsi_24h"] or 0),
|
||||
"latest_observed_at": to_iso8601_utc(row["latest_observed_at"]),
|
||||
"latest_collected_at": to_iso8601_utc(row["latest_collected_at"]),
|
||||
}
|
||||
|
||||
|
||||
def _runtime_status_for_builtin(source: str) -> dict[str, Any]:
|
||||
running = is_collector_running(source)
|
||||
return {
|
||||
"running": running,
|
||||
"done": False,
|
||||
"runtime": "collector",
|
||||
}
|
||||
|
||||
|
||||
def _runtime_status_for_custom(config_id: int) -> dict[str, Any]:
|
||||
status = get_custom_stream_status(config_id)
|
||||
return {
|
||||
"running": bool(status.get("running")),
|
||||
"done": bool(status.get("done")),
|
||||
"runtime": "custom_stream",
|
||||
}
|
||||
|
||||
|
||||
async def _serialize_builtin_aisstream(
|
||||
db: AsyncSession,
|
||||
datasource: DataSource,
|
||||
config: DataSourceConfig | None,
|
||||
) -> dict[str, Any]:
|
||||
health = await db.get(AISSourceHealth, datasource.source)
|
||||
config_payload = _safe_config_dict(config.config if config else {})
|
||||
endpoint = (
|
||||
(config.endpoint if config else None)
|
||||
or get_data_sources_config().get_yaml_url(datasource.source)
|
||||
)
|
||||
return {
|
||||
"source": datasource.source,
|
||||
"name": datasource.name,
|
||||
"display_name": "AISStream 实时船舶",
|
||||
"kind": "builtin",
|
||||
"source_type": "websocket",
|
||||
"endpoint": endpoint,
|
||||
"is_active": bool(datasource.is_active),
|
||||
"credential_configured": _credential_configured(config),
|
||||
"message_types": config_payload.get("message_types") or ["PositionReport", "ShipStaticData"],
|
||||
"bounding_boxes": config_payload.get("bounding_boxes") or [[[-90, -180], [90, 180]]],
|
||||
"config": config_payload,
|
||||
"runtime": _runtime_status_for_builtin(datasource.source),
|
||||
"health": health.to_dict() if health else None,
|
||||
"stats": await _load_realtime_stats(db, datasource.source),
|
||||
}
|
||||
|
||||
|
||||
async def _serialize_custom_stream(
|
||||
db: AsyncSession,
|
||||
config: DataSourceConfig,
|
||||
) -> dict[str, Any]:
|
||||
health = await db.get(AISSourceHealth, config.name)
|
||||
config_payload = _safe_config_dict(config.config)
|
||||
return {
|
||||
"source": config.name,
|
||||
"name": config.name,
|
||||
"display_name": config.description or config.name,
|
||||
"kind": "custom",
|
||||
"config_id": config.id,
|
||||
"source_type": config.source_type,
|
||||
"endpoint": config.endpoint,
|
||||
"is_active": bool(config.is_active),
|
||||
"credential_configured": config.auth_type == "none" or bool(_safe_config_dict(config.auth_config)),
|
||||
"message_types": config_payload.get("message_types") or [],
|
||||
"bounding_boxes": config_payload.get("bounding_boxes") or [],
|
||||
"config": config_payload,
|
||||
"runtime": _runtime_status_for_custom(config.id),
|
||||
"health": health.to_dict() if health else None,
|
||||
"stats": await _load_realtime_stats(db, config.name),
|
||||
}
|
||||
|
||||
|
||||
async def _load_builtin_aisstream(db: AsyncSession) -> tuple[DataSource | None, DataSourceConfig | None]:
|
||||
result = await db.execute(select(DataSource).where(DataSource.source == "aisstream_vessels"))
|
||||
datasource = result.scalar_one_or_none()
|
||||
config_result = await db.execute(
|
||||
select(DataSourceConfig)
|
||||
.where(DataSourceConfig.name == "aisstream_vessels")
|
||||
.where(DataSourceConfig.is_active.is_(True))
|
||||
.order_by(DataSourceConfig.id.desc())
|
||||
.limit(1)
|
||||
)
|
||||
return datasource, config_result.scalar_one_or_none()
|
||||
|
||||
|
||||
async def _load_custom_realtime_config(db: AsyncSession, source: str) -> DataSourceConfig | None:
|
||||
result = await db.execute(
|
||||
select(DataSourceConfig)
|
||||
.where(DataSourceConfig.name == source)
|
||||
.order_by(DataSourceConfig.id.desc())
|
||||
.limit(1)
|
||||
)
|
||||
config = result.scalar_one_or_none()
|
||||
return config if config is not None and _is_realtime_config(config) else None
|
||||
|
||||
|
||||
@router.get("")
|
||||
async def list_realtime_sources(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
sources: list[dict[str, Any]] = []
|
||||
datasource, builtin_config = await _load_builtin_aisstream(db)
|
||||
if datasource is not None:
|
||||
sources.append(await _serialize_builtin_aisstream(db, datasource, builtin_config))
|
||||
|
||||
custom_result = await db.execute(
|
||||
select(DataSourceConfig)
|
||||
.where(func.lower(DataSourceConfig.source_type).in_(REALTIME_SOURCE_TYPES))
|
||||
.order_by(DataSourceConfig.name)
|
||||
)
|
||||
for config in custom_result.scalars().all():
|
||||
if config.name in BUILTIN_REALTIME_SOURCES:
|
||||
continue
|
||||
sources.append(await _serialize_custom_stream(db, config))
|
||||
|
||||
return {"total": len(sources), "data": sources}
|
||||
|
||||
|
||||
async def _ensure_builtin_startable(db: AsyncSession) -> DataSourceConfig | None:
|
||||
datasource, config = await _load_builtin_aisstream(db)
|
||||
if datasource is None:
|
||||
raise HTTPException(status_code=404, detail="Realtime source not found")
|
||||
if not datasource.is_active:
|
||||
raise HTTPException(status_code=400, detail="Realtime source is disabled")
|
||||
if not _credential_configured(config):
|
||||
raise HTTPException(status_code=400, detail="AISStream API key is not configured")
|
||||
return config
|
||||
|
||||
|
||||
@router.post("/{source}/start")
|
||||
async def start_realtime_source(
|
||||
source: str,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
if source == "aisstream_vessels":
|
||||
await _ensure_builtin_startable(db)
|
||||
if is_collector_running(source):
|
||||
return {"status": "already_running", "source": source, "runtime": _runtime_status_for_builtin(source)}
|
||||
if not run_collector_now(source):
|
||||
raise HTTPException(status_code=409, detail="Realtime source could not be started")
|
||||
return {"status": "started", "source": source, "runtime": _runtime_status_for_builtin(source)}
|
||||
|
||||
config = await _load_custom_realtime_config(db, source)
|
||||
if config is None:
|
||||
raise HTTPException(status_code=404, detail="Realtime source not found")
|
||||
if not config.is_active:
|
||||
raise HTTPException(status_code=400, detail="Realtime source is disabled")
|
||||
started = start_custom_stream(config.id)
|
||||
return {
|
||||
"status": "started" if started else "already_running",
|
||||
"source": source,
|
||||
"runtime": _runtime_status_for_custom(config.id),
|
||||
}
|
||||
|
||||
|
||||
@router.post("/{source}/stop")
|
||||
async def stop_realtime_source(
|
||||
source: str,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
if source == "aisstream_vessels":
|
||||
stopped = await cancel_running_collector_now(source)
|
||||
await update_ais_source_health(db, source=source, connection_state="disconnected", last_error=None)
|
||||
await db.commit()
|
||||
return {"status": "stopped" if stopped else "not_running", "source": source, "runtime": _runtime_status_for_builtin(source)}
|
||||
|
||||
config = await _load_custom_realtime_config(db, source)
|
||||
if config is None:
|
||||
raise HTTPException(status_code=404, detail="Realtime source not found")
|
||||
stopped = await stop_custom_stream(config.id)
|
||||
await update_ais_source_health(db, source=source, connection_state="disconnected", last_error=None)
|
||||
await db.commit()
|
||||
return {
|
||||
"status": "stopped" if stopped else "not_running",
|
||||
"source": source,
|
||||
"runtime": _runtime_status_for_custom(config.id),
|
||||
}
|
||||
|
||||
|
||||
@router.post("/{source}/restart")
|
||||
async def restart_realtime_source(
|
||||
source: str,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
await stop_realtime_source(source, current_user=current_user, db=db)
|
||||
return await start_realtime_source(source, current_user=current_user, db=db)
|
||||
@@ -4,7 +4,8 @@ import os
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Request, status
|
||||
import httpx
|
||||
from pydantic import BaseModel, EmailStr, Field
|
||||
from dotenv import dotenv_values
|
||||
from sqlalchemy import select
|
||||
@@ -15,6 +16,13 @@ from app.core.time import to_iso8601_utc
|
||||
from app.core.config import settings as app_settings
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
from app.core.datasource_defaults import DEFAULT_DATASOURCES
|
||||
from app.ai_tasks.prompts import (
|
||||
get_effective_prompt,
|
||||
list_effective_prompts,
|
||||
reset_prompt_override,
|
||||
save_prompt_override,
|
||||
serialize_effective_prompt,
|
||||
)
|
||||
from app.db.session import get_db
|
||||
from app.models.datasource import DataSource
|
||||
from app.models.datasource_config import DataSourceConfig
|
||||
@@ -57,8 +65,12 @@ from app.services.llm_provider_catalog import (
|
||||
)
|
||||
from app.services.scheduler import sync_datasource_job
|
||||
from app.services.tv_streams import DEFAULT_TV_SETTINGS, get_tv_settings_payload, normalize_tv_settings
|
||||
from app.services.persistent_logs import record_audit_log
|
||||
|
||||
router = APIRouter()
|
||||
AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS = 5
|
||||
AI_CONNECTION_TEST_PROMPT_KEY = "ai.connection_test"
|
||||
SECRET_REVEAL_ROLES = {"admin", "super_admin"}
|
||||
|
||||
DEFAULT_SETTINGS = {
|
||||
"system": {
|
||||
@@ -121,6 +133,70 @@ DEFAULT_SETTINGS = {
|
||||
}
|
||||
|
||||
|
||||
def _user_role_value(user: User) -> str:
|
||||
role = getattr(user, "role", "")
|
||||
return role.value if hasattr(role, "value") else str(role or "")
|
||||
|
||||
|
||||
def _user_display_name(user: User) -> str | None:
|
||||
return getattr(user, "username", None) or getattr(user, "email", None)
|
||||
|
||||
|
||||
def _request_client_ip(request: Request | None) -> str | None:
|
||||
if request is None or request.client is None:
|
||||
return None
|
||||
return request.client.host
|
||||
|
||||
|
||||
def _can_reveal_integration_secrets(user: User) -> bool:
|
||||
return _user_role_value(user) in SECRET_REVEAL_ROLES
|
||||
|
||||
|
||||
async def _record_integration_secret_reveal(
|
||||
*,
|
||||
current_user: User,
|
||||
request: Request | None,
|
||||
target_id: str,
|
||||
result: str,
|
||||
details: dict,
|
||||
) -> None:
|
||||
await record_audit_log(
|
||||
action="settings.integration_secret.reveal",
|
||||
actor_id=getattr(current_user, "id", None),
|
||||
actor_name=_user_display_name(current_user),
|
||||
target_type="integration_secret",
|
||||
target_id=target_id,
|
||||
result=result,
|
||||
ip=_request_client_ip(request),
|
||||
details=details,
|
||||
)
|
||||
|
||||
|
||||
async def _ensure_secret_reveal_allowed(
|
||||
*,
|
||||
current_user: User,
|
||||
request: Request | None,
|
||||
target_id: str,
|
||||
details: dict | None = None,
|
||||
) -> None:
|
||||
if _can_reveal_integration_secrets(current_user):
|
||||
return
|
||||
await _record_integration_secret_reveal(
|
||||
current_user=current_user,
|
||||
request=request,
|
||||
target_id=target_id,
|
||||
result="denied",
|
||||
details={
|
||||
**(details or {}),
|
||||
"role": _user_role_value(current_user),
|
||||
},
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_403_FORBIDDEN,
|
||||
detail="Only administrators can reveal integration secrets",
|
||||
)
|
||||
|
||||
|
||||
class SystemSettingsUpdate(BaseModel):
|
||||
system_name: str = "智能星球"
|
||||
refresh_interval: int = Field(default=60, ge=10, le=3600)
|
||||
@@ -250,6 +326,11 @@ class OCRIntegrationUpdate(BaseModel):
|
||||
output_format: str = Field(default="markdown", pattern="^(markdown|text|json)$")
|
||||
|
||||
|
||||
class AIPromptUpdate(BaseModel):
|
||||
system_prompt: str = Field(default="", max_length=8000)
|
||||
prompt: str = Field(min_length=1, max_length=20000)
|
||||
|
||||
|
||||
class ExternalIntegrationsUpdate(BaseModel):
|
||||
ai_provider: AIProviderIntegrationUpdate
|
||||
barentswatch: BarentsWatchIntegrationUpdate
|
||||
@@ -339,9 +420,10 @@ def _get_provider_preset(provider: str) -> dict:
|
||||
"provider": provider,
|
||||
"provider_api": "openai-completions",
|
||||
"base_url": "",
|
||||
"model": "",
|
||||
"models": [],
|
||||
"api_key_env": "",
|
||||
"model": "",
|
||||
"models": [],
|
||||
"model_provider_apis": {},
|
||||
"api_key_env": "",
|
||||
}
|
||||
|
||||
|
||||
@@ -363,6 +445,9 @@ def _resolve_env_secret(*names: str) -> tuple[str, str]:
|
||||
value = env_file_values.get(name)
|
||||
if value:
|
||||
return value, "env_file"
|
||||
value = os.environ.get(name)
|
||||
if value:
|
||||
return value, "env"
|
||||
return "", ""
|
||||
|
||||
|
||||
@@ -405,9 +490,16 @@ def _provider_defaults(provider: str) -> dict:
|
||||
1200 if preset.get("provider_api") == "anthropic-messages" else 4096
|
||||
),
|
||||
"anthropic_version": "2023-06-01",
|
||||
"model_provider_apis": preset.get("model_provider_apis") or {},
|
||||
}
|
||||
|
||||
|
||||
def _selected_ai_env_provider() -> str:
|
||||
env_file_values = _read_ai_provider_env_file()
|
||||
provider = env_file_values.get("AI_PROVIDER") or os.environ.get("AI_PROVIDER") or "minimax"
|
||||
return _normalize_provider_id(provider)
|
||||
|
||||
|
||||
def _normalize_ai_provider_payload(ai_payload: dict | None) -> dict:
|
||||
raw = dict(ai_payload or {})
|
||||
default_provider = _normalize_provider_id(raw.get("default_provider") or raw.get("provider"))
|
||||
@@ -426,6 +518,7 @@ def _normalize_ai_provider_payload(ai_payload: dict | None) -> dict:
|
||||
"api_key",
|
||||
"max_tokens",
|
||||
"anthropic_version",
|
||||
"model_provider_apis",
|
||||
)
|
||||
if raw.get(key) not in (None, "")
|
||||
}
|
||||
@@ -463,7 +556,12 @@ def _resolve_provider_api_key(provider: str, provider_config: dict) -> tuple[str
|
||||
return str(saved_key), "runtime"
|
||||
preset = _get_provider_preset(provider)
|
||||
api_key_env = preset.get("api_key_env") or ""
|
||||
return _resolve_env_secret(api_key_env, "AI_API_KEY")
|
||||
value, source = _resolve_env_secret(api_key_env)
|
||||
if value:
|
||||
return value, source
|
||||
if _normalize_provider_id(provider) == _selected_ai_env_provider():
|
||||
return _resolve_env_secret("AI_API_KEY")
|
||||
return "", ""
|
||||
|
||||
|
||||
def _resolve_service_token(ai_payload: dict) -> tuple[str, str]:
|
||||
@@ -484,12 +582,23 @@ def _is_secret_placeholder(value: Optional[str], current_preview: str = "") -> b
|
||||
text = str(value).strip()
|
||||
if not text:
|
||||
return True
|
||||
return text == current_preview or text.startswith("••••") or "*" in text
|
||||
if text == current_preview or text.startswith("••••"):
|
||||
return True
|
||||
if "-" in text:
|
||||
_prefix, masked = text.split("-", 1)
|
||||
if masked and all(char in {"*", "•", " ", "\t"} for char in masked):
|
||||
return True
|
||||
return all(char in {"*", "•", " ", "\t"} for char in text)
|
||||
|
||||
|
||||
def _build_ai_provider_payload(current_payload: dict, update: AIProviderIntegrationUpdate) -> dict:
|
||||
current_ai = _normalize_ai_provider_payload(current_payload.get("ai_provider") or {})
|
||||
provider_id = _normalize_provider_id(update.default_provider or update.provider)
|
||||
provider_id = _normalize_provider_id(update.provider)
|
||||
default_provider = (
|
||||
_normalize_provider_id(update.default_provider)
|
||||
if update.default_provider is not None
|
||||
else current_ai["default_provider"]
|
||||
)
|
||||
current_providers = {
|
||||
provider: dict(config or {})
|
||||
for provider, config in current_ai.get("providers", {}).items()
|
||||
@@ -523,7 +632,7 @@ def _build_ai_provider_payload(current_payload: dict, update: AIProviderIntegrat
|
||||
"service_url": update.service_url.strip()
|
||||
or app_settings.AI_PROVIDER_SERVICE_URL,
|
||||
"service_token": current_ai.get("service_token") or "",
|
||||
"default_provider": provider_id,
|
||||
"default_provider": default_provider,
|
||||
"providers": current_providers,
|
||||
"timeout_seconds": update.timeout_seconds,
|
||||
"retry_attempts": update.retry_attempts,
|
||||
@@ -557,10 +666,215 @@ def _runtime_config_from_ai_payload(ai_payload: dict) -> dict:
|
||||
"api_key": api_key,
|
||||
"max_tokens": int(provider_config.get("max_tokens") or 1200),
|
||||
"anthropic_version": provider_config.get("anthropic_version") or "2023-06-01",
|
||||
"model_provider_apis": provider_config.get("model_provider_apis") or {},
|
||||
"preset_models": _get_provider_preset(default_provider).get("models") or [],
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def _ai_provider_runtime_fingerprint(ai_payload: dict) -> dict:
|
||||
runtime_config = _runtime_config_from_ai_payload(ai_payload)
|
||||
llm_config = runtime_config.get("llm_config") or {}
|
||||
return {
|
||||
"service_url": runtime_config.get("service_url") or "",
|
||||
"service_token": runtime_config.get("service_token") or "",
|
||||
"timeout_seconds": int(runtime_config.get("timeout_seconds") or 0),
|
||||
"retry_attempts": int(runtime_config.get("retry_attempts") or 0),
|
||||
"provider": llm_config.get("provider") or "",
|
||||
"provider_api": llm_config.get("provider_api") or "",
|
||||
"base_url": llm_config.get("base_url") or "",
|
||||
"model": llm_config.get("model") or "",
|
||||
"api_key": llm_config.get("api_key") or "",
|
||||
"max_tokens": int(llm_config.get("max_tokens") or 0),
|
||||
"anthropic_version": llm_config.get("anthropic_version") or "",
|
||||
}
|
||||
|
||||
|
||||
async def _validate_ai_provider_full_connection(ai_payload: dict) -> dict:
|
||||
runtime_config = _runtime_config_from_ai_payload(ai_payload)
|
||||
client = AIProviderClient(
|
||||
service_url=runtime_config["service_url"],
|
||||
service_token=runtime_config["service_token"],
|
||||
timeout=runtime_config["timeout_seconds"],
|
||||
retry_attempts=runtime_config["retry_attempts"],
|
||||
llm_config=runtime_config.get("llm_config") or {},
|
||||
)
|
||||
status_result = await client.get_status()
|
||||
if not status_result.configured:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail="AI Provider 可访问,但当前 provider/model/key 未完整配置。",
|
||||
)
|
||||
prompt = await get_effective_prompt(None, AI_CONNECTION_TEST_PROMPT_KEY)
|
||||
analysis_result = await client.analyze(
|
||||
SituationalAnalysisRequest(
|
||||
title="保存前完整连接测试",
|
||||
objective=prompt.prompt,
|
||||
system_prompt=prompt.system_prompt or None,
|
||||
observations=["这是保存 AI Provider 配置前的完整 LLM 调用测试。"],
|
||||
constraints=["回复尽量简短。"],
|
||||
)
|
||||
)
|
||||
return {
|
||||
"status": status_result.model_dump(),
|
||||
"provider": analysis_result.provider,
|
||||
"model": analysis_result.model,
|
||||
}
|
||||
|
||||
|
||||
def _join_provider_url(base_url: str, path: str) -> str:
|
||||
return f"{base_url.rstrip('/')}/{path.lstrip('/')}"
|
||||
|
||||
|
||||
def _extract_model_ids(payload: dict) -> list[str]:
|
||||
data = payload.get("data") if isinstance(payload, dict) else None
|
||||
if isinstance(data, list):
|
||||
return [
|
||||
str(item.get("id"))
|
||||
for item in data
|
||||
if isinstance(item, dict) and item.get("id")
|
||||
]
|
||||
models = payload.get("models") if isinstance(payload, dict) else None
|
||||
if isinstance(models, list):
|
||||
return [
|
||||
str(item.get("name") or item.get("model") or item.get("id") or item)
|
||||
for item in models
|
||||
if item
|
||||
]
|
||||
return []
|
||||
|
||||
|
||||
def _contains_model(model_ids: list[str], model: str) -> bool:
|
||||
normalized_model = model.strip().lower()
|
||||
return any(str(item).strip().lower() == normalized_model for item in model_ids)
|
||||
|
||||
|
||||
async def _check_ai_provider_lightweight(llm_config: dict, timeout_seconds: int) -> dict:
|
||||
provider = _normalize_provider_id(llm_config.get("provider") or "")
|
||||
configured_api = str(llm_config.get("provider_api") or "").strip() or "openai-completions"
|
||||
model = str(llm_config.get("model") or "").strip()
|
||||
base_url = str(llm_config.get("base_url") or "").strip().rstrip("/")
|
||||
api_key = str(llm_config.get("api_key") or "").strip()
|
||||
provider_api = configured_api
|
||||
model_provider_apis = llm_config.get("model_provider_apis")
|
||||
if isinstance(model_provider_apis, dict):
|
||||
provider_api = str(model_provider_apis.get(model) or provider_api)
|
||||
preset_models = [
|
||||
str(item)
|
||||
for item in (llm_config.get("preset_models") or [])
|
||||
if str(item).strip()
|
||||
]
|
||||
|
||||
if not provider or not base_url or not model:
|
||||
return {
|
||||
"success": False,
|
||||
"connected": False,
|
||||
"message": "当前 provider/base_url/model 未完整配置。",
|
||||
"mode": "lightweight_config",
|
||||
}
|
||||
if provider_api != "ollama-generate" and not api_key:
|
||||
return {
|
||||
"success": False,
|
||||
"connected": False,
|
||||
"message": "当前 provider 未配置 API Key。",
|
||||
"mode": "lightweight_config",
|
||||
}
|
||||
|
||||
if provider == "opencode-go":
|
||||
url = _join_provider_url(base_url, "/models")
|
||||
headers = {"Authorization": f"Bearer {api_key}"}
|
||||
elif provider_api == "ollama-generate":
|
||||
url = _join_provider_url(base_url, "/api/tags")
|
||||
headers: dict[str, str] = {}
|
||||
elif provider_api == "openai-completions":
|
||||
url = _join_provider_url(base_url, "/models")
|
||||
headers = {"Authorization": f"Bearer {api_key}"}
|
||||
elif provider_api == "anthropic-messages":
|
||||
url = _join_provider_url(base_url, "/models")
|
||||
headers = {
|
||||
"x-api-key": api_key,
|
||||
"anthropic-version": str(llm_config.get("anthropic_version") or "2023-06-01"),
|
||||
}
|
||||
else:
|
||||
return {
|
||||
"success": False,
|
||||
"connected": False,
|
||||
"message": f"当前 provider_api 不支持轻量连通性测试: {provider_api}",
|
||||
"mode": "lightweight_unsupported",
|
||||
}
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=min(timeout_seconds, AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS)) as client:
|
||||
response = await client.get(url, headers=headers)
|
||||
response.raise_for_status()
|
||||
payload = response.json()
|
||||
except httpx.HTTPStatusError as exc:
|
||||
detail = exc.response.text or exc.response.reason_phrase
|
||||
if exc.response.status_code == 404 and _contains_model(preset_models, model):
|
||||
return {
|
||||
"success": True,
|
||||
"connected": True,
|
||||
"message": "轻量连通性测试通过;当前 provider 不提供可用的模型目录,已按内置模型预设确认。",
|
||||
"mode": "lightweight_preset",
|
||||
"provider": provider,
|
||||
"provider_api": provider_api,
|
||||
"model": model,
|
||||
"url": url,
|
||||
}
|
||||
return {
|
||||
"success": False,
|
||||
"connected": False,
|
||||
"message": f"轻量连通性测试失败: HTTP {exc.response.status_code} {detail}",
|
||||
"mode": "lightweight_models",
|
||||
"url": url,
|
||||
}
|
||||
except Exception as exc:
|
||||
return {
|
||||
"success": False,
|
||||
"connected": False,
|
||||
"message": f"轻量连通性测试失败: {exc}",
|
||||
"mode": "lightweight_models",
|
||||
"url": url,
|
||||
}
|
||||
|
||||
model_ids = _extract_model_ids(payload)
|
||||
if model_ids and not _contains_model(model_ids, model):
|
||||
if _contains_model(preset_models, model):
|
||||
return {
|
||||
"success": True,
|
||||
"connected": True,
|
||||
"message": "轻量连通性测试通过;provider 模型目录未返回当前别名,已按内置模型预设确认。",
|
||||
"mode": "lightweight_models_with_preset_alias",
|
||||
"provider": provider,
|
||||
"provider_api": provider_api,
|
||||
"model": model,
|
||||
"models_count": len(model_ids),
|
||||
"url": url,
|
||||
}
|
||||
return {
|
||||
"success": False,
|
||||
"connected": False,
|
||||
"message": f"连接可用,但模型目录中没有当前模型: {model}",
|
||||
"mode": "lightweight_models",
|
||||
"provider": provider,
|
||||
"model": model,
|
||||
"models_count": len(model_ids),
|
||||
"url": url,
|
||||
}
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"connected": True,
|
||||
"message": "轻量连通性测试通过",
|
||||
"mode": "lightweight_models",
|
||||
"provider": provider,
|
||||
"provider_api": provider_api,
|
||||
"model": model,
|
||||
"models_count": len(model_ids),
|
||||
"url": url,
|
||||
}
|
||||
|
||||
|
||||
def _web_search_provider_defaults(provider: str) -> dict:
|
||||
return web_search_provider_defaults(provider).model_dump()
|
||||
|
||||
@@ -621,12 +935,21 @@ def _normalize_web_search_payload(web_search_payload: dict | None) -> dict:
|
||||
}
|
||||
|
||||
|
||||
def _resolve_web_search_api_key(provider: str, provider_config: dict) -> tuple[str, str]:
|
||||
def _resolve_web_search_api_key(
|
||||
provider: str,
|
||||
provider_config: dict,
|
||||
default_provider: str | None = None,
|
||||
) -> tuple[str, str]:
|
||||
saved_key = provider_config.get("api_key") or ""
|
||||
if saved_key:
|
||||
return str(saved_key), "runtime"
|
||||
preset = get_web_search_provider_preset(provider)
|
||||
return _resolve_web_search_env_secret(preset.get("api_key_env") or "", "WEB_SEARCH_API_KEY")
|
||||
value, source = _resolve_web_search_env_secret(preset.get("api_key_env") or "")
|
||||
if value:
|
||||
return value, source
|
||||
if normalize_web_search_provider(provider) == normalize_web_search_provider(default_provider or "tavily"):
|
||||
return _resolve_web_search_env_secret("WEB_SEARCH_API_KEY")
|
||||
return "", ""
|
||||
|
||||
|
||||
def _build_web_search_payload(
|
||||
@@ -636,13 +959,22 @@ def _build_web_search_payload(
|
||||
current_web_search = _normalize_web_search_payload(current_payload.get("web_search") or {})
|
||||
if update is None:
|
||||
return current_web_search
|
||||
provider_id = normalize_web_search_provider(update.default_provider or update.provider)
|
||||
provider_id = normalize_web_search_provider(update.provider)
|
||||
default_provider = (
|
||||
normalize_web_search_provider(update.default_provider)
|
||||
if update.default_provider is not None
|
||||
else current_web_search["default_provider"]
|
||||
)
|
||||
current_providers = {
|
||||
provider: dict(config or {})
|
||||
for provider, config in current_web_search.get("providers", {}).items()
|
||||
}
|
||||
current_provider = current_providers.get(provider_id) or _web_search_provider_defaults(provider_id)
|
||||
current_key, current_key_source = _resolve_web_search_api_key(provider_id, current_provider)
|
||||
current_key, current_key_source = _resolve_web_search_api_key(
|
||||
provider_id,
|
||||
current_provider,
|
||||
current_web_search["default_provider"],
|
||||
)
|
||||
current_key_preview = _mask_secret(current_key, current_key_source)["preview"]
|
||||
provider_payload = {
|
||||
**_web_search_provider_defaults(provider_id),
|
||||
@@ -675,7 +1007,7 @@ def _build_web_search_payload(
|
||||
current_providers[provider_id] = provider_payload
|
||||
return {
|
||||
"enabled": update.enabled,
|
||||
"default_provider": provider_id,
|
||||
"default_provider": default_provider,
|
||||
"providers": current_providers,
|
||||
}
|
||||
|
||||
@@ -684,12 +1016,12 @@ def _runtime_config_from_web_search_payload(web_search_payload: dict) -> WebSear
|
||||
normalized = _normalize_web_search_payload(web_search_payload)
|
||||
provider_id = normalized["default_provider"]
|
||||
provider_config = normalized["providers"].get(provider_id) or _web_search_provider_defaults(provider_id)
|
||||
api_key, _source = _resolve_web_search_api_key(provider_id, provider_config)
|
||||
api_key, _source = _resolve_web_search_api_key(provider_id, provider_config, provider_id)
|
||||
provider_models = {
|
||||
provider: WebSearchProviderConfig(**{
|
||||
**config,
|
||||
"api_key": (
|
||||
api_key if provider == provider_id else _resolve_web_search_api_key(provider, config)[0]
|
||||
api_key if provider == provider_id else _resolve_web_search_api_key(provider, config, provider_id)[0]
|
||||
),
|
||||
})
|
||||
for provider, config in normalized["providers"].items()
|
||||
@@ -823,7 +1155,11 @@ async def serialize_external_integrations(db: AsyncSession) -> dict:
|
||||
normalized_web_search["providers"].get(provider_id)
|
||||
or _web_search_provider_defaults(provider_id)
|
||||
)
|
||||
api_key, api_key_source = _resolve_web_search_api_key(provider_id, provider_config)
|
||||
api_key, api_key_source = _resolve_web_search_api_key(
|
||||
provider_id,
|
||||
provider_config,
|
||||
normalized_web_search["default_provider"],
|
||||
)
|
||||
web_search_providers_payload[provider_id] = {
|
||||
**{
|
||||
key: value
|
||||
@@ -1158,6 +1494,47 @@ async def get_external_integrations(
|
||||
return {"integrations": await serialize_external_integrations(db)}
|
||||
|
||||
|
||||
@router.get("/ai-prompts")
|
||||
async def get_ai_prompts(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
prompts = await list_effective_prompts(db)
|
||||
return {"data": [serialize_effective_prompt(prompt) for prompt in prompts]}
|
||||
|
||||
|
||||
@router.put("/ai-prompts/{task_key}")
|
||||
async def update_ai_prompt(
|
||||
task_key: str,
|
||||
payload: AIPromptUpdate,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
try:
|
||||
prompt = await save_prompt_override(
|
||||
db,
|
||||
task_key,
|
||||
system_prompt=payload.system_prompt,
|
||||
prompt=payload.prompt,
|
||||
)
|
||||
except KeyError:
|
||||
raise HTTPException(status_code=404, detail="未知 AI 提示词任务") from None
|
||||
return {"data": serialize_effective_prompt(prompt)}
|
||||
|
||||
|
||||
@router.post("/ai-prompts/{task_key}/reset")
|
||||
async def reset_ai_prompt(
|
||||
task_key: str,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
try:
|
||||
prompt = await reset_prompt_override(db, task_key)
|
||||
except KeyError:
|
||||
raise HTTPException(status_code=404, detail="未知 AI 提示词任务") from None
|
||||
return {"data": serialize_effective_prompt(prompt)}
|
||||
|
||||
|
||||
@router.get("/integrations/barentswatch/connectivity")
|
||||
async def get_barentswatch_connectivity(
|
||||
current_user: User = Depends(get_current_user),
|
||||
@@ -1217,14 +1594,22 @@ async def connect_ai_provider_integration(
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
current_payload = await get_setting_payload(db, "external_integrations")
|
||||
# Connection testing should validate the provider being edited, not the
|
||||
# currently saved default provider. This is a transient draft only and is
|
||||
# intentionally not persisted.
|
||||
payload = payload.model_copy(update={"default_provider": payload.provider})
|
||||
draft_ai_payload = _build_ai_provider_payload(current_payload, payload)
|
||||
runtime_config = _runtime_config_from_ai_payload(draft_ai_payload)
|
||||
quick_llm_config = {
|
||||
**(runtime_config.get("llm_config") or {}),
|
||||
"max_tokens": 1,
|
||||
}
|
||||
client = AIProviderClient(
|
||||
service_url=runtime_config["service_url"],
|
||||
service_token=runtime_config["service_token"],
|
||||
timeout=runtime_config["timeout_seconds"],
|
||||
retry_attempts=runtime_config["retry_attempts"],
|
||||
llm_config=runtime_config.get("llm_config") or {},
|
||||
timeout=min(int(runtime_config["timeout_seconds"] or 60), AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS),
|
||||
retry_attempts=1,
|
||||
llm_config=quick_llm_config,
|
||||
)
|
||||
|
||||
try:
|
||||
@@ -1236,29 +1621,16 @@ async def connect_ai_provider_integration(
|
||||
"message": "AI Provider 可访问,但当前 provider/model/key 未完整配置。",
|
||||
"status": status_result.model_dump(),
|
||||
}
|
||||
analysis_result = await client.analyze(
|
||||
SituationalAnalysisRequest(
|
||||
title="连接测试",
|
||||
objective="请用一句话回复连接可用。",
|
||||
observations=["这是配置中心发起的 LLM 连接测试。"],
|
||||
constraints=["回复尽量简短。"],
|
||||
)
|
||||
)
|
||||
current_web_search = _normalize_web_search_payload(current_payload.get("web_search") or {})
|
||||
current_ocr = _normalize_ocr_payload(current_payload.get("ocr") or {})
|
||||
await save_setting_payload(
|
||||
db,
|
||||
"external_integrations",
|
||||
{"ai_provider": draft_ai_payload, "web_search": current_web_search, "ocr": current_ocr},
|
||||
lightweight_result = await _check_ai_provider_lightweight(
|
||||
quick_llm_config,
|
||||
timeout_seconds=min(
|
||||
int(runtime_config["timeout_seconds"] or 60),
|
||||
AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS,
|
||||
),
|
||||
)
|
||||
return {
|
||||
"success": True,
|
||||
"connected": True,
|
||||
"message": "AI Provider 连接成功,已保存为全局默认配置。",
|
||||
**lightweight_result,
|
||||
"status": status_result.model_dump(),
|
||||
"provider": analysis_result.provider,
|
||||
"model": analysis_result.model,
|
||||
"integrations": await serialize_external_integrations(db),
|
||||
}
|
||||
except HTTPException as exc:
|
||||
return {
|
||||
@@ -1276,16 +1648,38 @@ async def connect_ai_provider_integration(
|
||||
|
||||
@router.get("/integrations/ai-provider/secrets")
|
||||
async def reveal_ai_provider_secrets(
|
||||
request: Request,
|
||||
provider: str = Query(default=""),
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
requested_provider = _normalize_provider_id(provider) if provider else "default"
|
||||
await _ensure_secret_reveal_allowed(
|
||||
current_user=current_user,
|
||||
request=request,
|
||||
target_id=f"ai_provider:{requested_provider}",
|
||||
details={"kind": "ai_provider", "provider": requested_provider},
|
||||
)
|
||||
current_payload = await get_setting_payload(db, "external_integrations")
|
||||
ai_payload = _normalize_ai_provider_payload(current_payload.get("ai_provider") or {})
|
||||
provider_id = _normalize_provider_id(provider or ai_payload["default_provider"])
|
||||
provider_config = ai_payload["providers"].get(provider_id) or _provider_defaults(provider_id)
|
||||
api_key, api_key_source = _resolve_provider_api_key(provider_id, provider_config)
|
||||
service_token, service_token_source = _resolve_service_token(ai_payload)
|
||||
await _record_integration_secret_reveal(
|
||||
current_user=current_user,
|
||||
request=request,
|
||||
target_id=f"ai_provider:{provider_id}",
|
||||
result="success",
|
||||
details={
|
||||
"kind": "ai_provider",
|
||||
"provider": provider_id,
|
||||
"api_key_configured": bool(api_key),
|
||||
"api_key_source": api_key_source,
|
||||
"service_token_configured": bool(service_token),
|
||||
"service_token_source": service_token_source,
|
||||
},
|
||||
)
|
||||
return {
|
||||
"provider": provider_id,
|
||||
"api_key": api_key,
|
||||
@@ -1304,10 +1698,18 @@ async def get_web_search_presets(
|
||||
|
||||
@router.get("/integrations/web-search/secrets")
|
||||
async def reveal_web_search_secrets(
|
||||
request: Request,
|
||||
provider: str = Query(default=""),
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
requested_provider = normalize_web_search_provider(provider) if provider else "default"
|
||||
await _ensure_secret_reveal_allowed(
|
||||
current_user=current_user,
|
||||
request=request,
|
||||
target_id=f"web_search:{requested_provider}",
|
||||
details={"kind": "web_search", "provider": requested_provider},
|
||||
)
|
||||
current_payload = await get_setting_payload(db, "external_integrations")
|
||||
web_search_payload = _normalize_web_search_payload(current_payload.get("web_search") or {})
|
||||
provider_id = normalize_web_search_provider(provider or web_search_payload["default_provider"])
|
||||
@@ -1315,7 +1717,23 @@ async def reveal_web_search_secrets(
|
||||
web_search_payload["providers"].get(provider_id)
|
||||
or _web_search_provider_defaults(provider_id)
|
||||
)
|
||||
api_key, api_key_source = _resolve_web_search_api_key(provider_id, provider_config)
|
||||
api_key, api_key_source = _resolve_web_search_api_key(
|
||||
provider_id,
|
||||
provider_config,
|
||||
web_search_payload["default_provider"],
|
||||
)
|
||||
await _record_integration_secret_reveal(
|
||||
current_user=current_user,
|
||||
request=request,
|
||||
target_id=f"web_search:{provider_id}",
|
||||
result="success",
|
||||
details={
|
||||
"kind": "web_search",
|
||||
"provider": provider_id,
|
||||
"api_key_configured": bool(api_key),
|
||||
"api_key_source": api_key_source,
|
||||
},
|
||||
)
|
||||
return {
|
||||
"provider": provider_id,
|
||||
"api_key": api_key,
|
||||
@@ -1325,12 +1743,31 @@ async def reveal_web_search_secrets(
|
||||
|
||||
@router.get("/integrations/ocr/secrets")
|
||||
async def reveal_ocr_secrets(
|
||||
request: Request,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
await _ensure_secret_reveal_allowed(
|
||||
current_user=current_user,
|
||||
request=request,
|
||||
target_id="ocr:default",
|
||||
details={"kind": "ocr", "provider": "default"},
|
||||
)
|
||||
current_payload = await get_setting_payload(db, "external_integrations")
|
||||
ocr_payload = _normalize_ocr_payload(current_payload.get("ocr") or {})
|
||||
api_key, api_key_source = _resolve_ocr_api_key(ocr_payload)
|
||||
await _record_integration_secret_reveal(
|
||||
current_user=current_user,
|
||||
request=request,
|
||||
target_id=f"ocr:{ocr_payload['provider']}",
|
||||
result="success",
|
||||
details={
|
||||
"kind": "ocr",
|
||||
"provider": ocr_payload["provider"],
|
||||
"api_key_configured": bool(api_key),
|
||||
"api_key_source": api_key_source,
|
||||
},
|
||||
)
|
||||
return {
|
||||
"provider": ocr_payload["provider"],
|
||||
"api_key": api_key,
|
||||
@@ -1434,9 +1871,17 @@ async def get_ai_provider_presets(
|
||||
async def refresh_ai_provider_preset(
|
||||
provider: str,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
try:
|
||||
return {"data": await refresh_llm_provider_preset(provider)}
|
||||
provider_id = _normalize_provider_id(provider)
|
||||
api_key = None
|
||||
if provider_id == "opencode-go":
|
||||
current_payload = await get_setting_payload(db, "external_integrations")
|
||||
ai_payload = _normalize_ai_provider_payload(current_payload.get("ai_provider") or {})
|
||||
provider_config = ai_payload["providers"].get(provider_id) or _provider_defaults(provider_id)
|
||||
api_key, _api_key_source = _resolve_provider_api_key(provider_id, provider_config)
|
||||
return {"data": await refresh_llm_provider_preset(provider_id, api_key=api_key)}
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=404, detail=str(exc)) from exc
|
||||
except Exception as exc:
|
||||
|
||||
@@ -8,9 +8,13 @@ from datetime import datetime
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Request, status
|
||||
from pydantic import BaseModel
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.config import ROOT_DIR
|
||||
from app.core.security import get_current_user
|
||||
from app.db.session import get_db
|
||||
from app.models.system_log import AuditLog, SystemLog
|
||||
from app.models.user import User
|
||||
from app.services.persistent_logs import record_audit_log, record_system_log
|
||||
from app.services.system_control import (
|
||||
@@ -35,6 +39,7 @@ from app.services.system_logs import (
|
||||
normalize_log_level,
|
||||
read_log_snapshot,
|
||||
)
|
||||
from app.services.earth_layer_cache import earth_layer_cache
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
@@ -112,6 +117,17 @@ class EarthClientLogEventResponse(BaseModel):
|
||||
level: str
|
||||
|
||||
|
||||
class EarthLayerCacheStatusResponse(BaseModel):
|
||||
prefix: str
|
||||
key_count: int
|
||||
memory_bytes: int
|
||||
layers: dict[str, dict[str, int]]
|
||||
|
||||
|
||||
class EarthLayerCacheClearResponse(BaseModel):
|
||||
deleted: int
|
||||
|
||||
|
||||
def ensure_super_admin(current_user: User) -> None:
|
||||
if not require_super_admin(current_user.role):
|
||||
raise HTTPException(
|
||||
@@ -132,6 +148,34 @@ def validate_log_date(raw_value: str | None, field_name: str) -> str | None:
|
||||
) from exc
|
||||
|
||||
|
||||
@router.get("/cache/earth-layers", response_model=EarthLayerCacheStatusResponse)
|
||||
async def get_earth_layer_cache_status(
|
||||
current_user: User = Depends(get_current_user),
|
||||
):
|
||||
ensure_super_admin(current_user)
|
||||
try:
|
||||
return earth_layer_cache.status()
|
||||
except Exception as exc:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
|
||||
detail=f"Unable to read Earth layer cache status: {exc}",
|
||||
) from exc
|
||||
|
||||
|
||||
@router.delete("/cache/earth-layers", response_model=EarthLayerCacheClearResponse)
|
||||
async def clear_earth_layer_cache(
|
||||
current_user: User = Depends(get_current_user),
|
||||
):
|
||||
ensure_super_admin(current_user)
|
||||
try:
|
||||
return {"deleted": earth_layer_cache.delete_pattern()}
|
||||
except Exception as exc:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
|
||||
detail=f"Unable to clear Earth layer cache: {exc}",
|
||||
) from exc
|
||||
|
||||
|
||||
@router.post("/restart-tasks", response_model=RestartTaskResponse)
|
||||
async def create_restart_task(
|
||||
payload: RestartTaskCreate,
|
||||
@@ -270,7 +314,120 @@ async def get_system_log_sources(
|
||||
current_user: User = Depends(get_current_user),
|
||||
):
|
||||
ensure_super_admin(current_user)
|
||||
return {"items": list_log_sources()}
|
||||
return {
|
||||
"items": [
|
||||
*list_log_sources(),
|
||||
{
|
||||
"source_id": "system-db",
|
||||
"name": "系统事件",
|
||||
"kind": "database",
|
||||
"location": "table://system_logs",
|
||||
"description": "后端持久化系统事件、AI 和采集器操作日志。",
|
||||
"category": "database",
|
||||
"status": "ok",
|
||||
},
|
||||
{
|
||||
"source_id": "audit-db",
|
||||
"name": "审计事件",
|
||||
"kind": "database",
|
||||
"location": "table://audit_logs",
|
||||
"description": "管理员敏感操作和密钥 reveal 审计记录。",
|
||||
"category": "audit",
|
||||
"status": "ok",
|
||||
},
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
async def read_database_log_snapshot(
|
||||
source_id: str,
|
||||
*,
|
||||
limit: int,
|
||||
level: str,
|
||||
levels: str | None,
|
||||
start_date: str | None,
|
||||
end_date: str | None,
|
||||
search: str | None,
|
||||
db: AsyncSession,
|
||||
) -> dict | None:
|
||||
selected_levels = set(normalize_log_level(item) for item in (levels or level).split(",") if item.strip())
|
||||
selected_levels.discard("all")
|
||||
search_query = (search or "").strip().lower()
|
||||
lines: list[str] = []
|
||||
|
||||
if source_id == "system-db":
|
||||
query = select(SystemLog).order_by(SystemLog.occurred_at.desc().nullslast(), SystemLog.id.desc()).limit(limit * 5)
|
||||
result = await db.execute(query)
|
||||
records = result.scalars().all()
|
||||
for record in records:
|
||||
record_level = normalize_log_level(record.level)
|
||||
if selected_levels and record_level not in selected_levels:
|
||||
continue
|
||||
occurred_at = record.occurred_at.date().isoformat() if record.occurred_at else ""
|
||||
if start_date and occurred_at and occurred_at < start_date:
|
||||
continue
|
||||
if end_date and occurred_at and occurred_at > end_date:
|
||||
continue
|
||||
line = " ".join(
|
||||
part
|
||||
for part in [
|
||||
record.occurred_at.isoformat() if record.occurred_at else "",
|
||||
record_level.upper(),
|
||||
record.source,
|
||||
record.event or "",
|
||||
record.message,
|
||||
]
|
||||
if part
|
||||
)
|
||||
if search_query and search_query not in line.lower():
|
||||
continue
|
||||
lines.append(line)
|
||||
elif source_id == "audit-db":
|
||||
query = select(AuditLog).order_by(AuditLog.occurred_at.desc().nullslast(), AuditLog.id.desc()).limit(limit * 5)
|
||||
result = await db.execute(query)
|
||||
records = result.scalars().all()
|
||||
for record in records:
|
||||
occurred_at = record.occurred_at.date().isoformat() if record.occurred_at else ""
|
||||
if start_date and occurred_at and occurred_at < start_date:
|
||||
continue
|
||||
if end_date and occurred_at and occurred_at > end_date:
|
||||
continue
|
||||
line = " ".join(
|
||||
part
|
||||
for part in [
|
||||
record.occurred_at.isoformat() if record.occurred_at else "",
|
||||
"INFO",
|
||||
record.action,
|
||||
record.target_type or "",
|
||||
record.target_id or "",
|
||||
record.result or "",
|
||||
]
|
||||
if part
|
||||
)
|
||||
if search_query and search_query not in line.lower():
|
||||
continue
|
||||
lines.append(line)
|
||||
else:
|
||||
return None
|
||||
|
||||
lines = list(reversed(lines[:limit]))
|
||||
return {
|
||||
"source_id": source_id,
|
||||
"name": "系统事件" if source_id == "system-db" else "审计事件",
|
||||
"kind": "database",
|
||||
"location": "table://system_logs" if source_id == "system-db" else "table://audit_logs",
|
||||
"description": "数据库持久化日志",
|
||||
"category": "database" if source_id == "system-db" else "audit",
|
||||
"status": "ok" if lines else "empty",
|
||||
"level": level,
|
||||
"selected_levels": sorted(selected_levels),
|
||||
"search_query": search or "",
|
||||
"available_levels": ["all", "error", "warning", "info", "debug"],
|
||||
"daily_markers": [],
|
||||
"line_limit": limit,
|
||||
"line_count": len(lines),
|
||||
"lines": lines,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/logs/{source_id}", response_model=SystemLogSnapshotResponse)
|
||||
@@ -283,6 +440,7 @@ async def get_system_log_snapshot(
|
||||
end_date: str | None = Query(None, description="Filter logs until this date (YYYY-MM-DD)"),
|
||||
search: str | None = Query(None, description="Case-insensitive substring search"),
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
ensure_super_admin(current_user)
|
||||
|
||||
@@ -305,15 +463,26 @@ async def get_system_log_snapshot(
|
||||
if normalized_start_date and normalized_end_date and normalized_start_date > normalized_end_date:
|
||||
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="start_date must be earlier than or equal to end_date")
|
||||
|
||||
snapshot = read_log_snapshot(
|
||||
snapshot = await read_database_log_snapshot(
|
||||
source_id,
|
||||
limit,
|
||||
limit=limit,
|
||||
level=level,
|
||||
levels=levels,
|
||||
start_date=normalized_start_date,
|
||||
end_date=normalized_end_date,
|
||||
search=search,
|
||||
db=db,
|
||||
)
|
||||
if snapshot is None:
|
||||
snapshot = read_log_snapshot(
|
||||
source_id,
|
||||
limit,
|
||||
level=level,
|
||||
levels=levels,
|
||||
start_date=normalized_start_date,
|
||||
end_date=normalized_end_date,
|
||||
search=search,
|
||||
)
|
||||
if snapshot is None:
|
||||
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Log source not found")
|
||||
return snapshot
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Response
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.api.v1.visualization import _parse_bbox, build_vessel_snapshot_response
|
||||
@@ -23,6 +23,7 @@ async def get_vessel_snapshot(
|
||||
limit: int = Query(1000, ge=1, le=MAX_SNAPSHOT_LIMIT),
|
||||
since_minutes: int = Query(60, ge=1, le=1440),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
response: Response = None,
|
||||
):
|
||||
if not bbox:
|
||||
raise HTTPException(status_code=400, detail="bbox is required")
|
||||
@@ -36,4 +37,5 @@ async def get_vessel_snapshot(
|
||||
type_filter=type,
|
||||
limit=limit,
|
||||
since_minutes=since_minutes,
|
||||
response=response,
|
||||
)
|
||||
|
||||
@@ -56,6 +56,13 @@ from app.services.vessel_ais_aggregation import (
|
||||
get_vessel_raw_observations,
|
||||
MAX_SNAPSHOT_LIMIT,
|
||||
)
|
||||
from app.services.earth_layer_cache import (
|
||||
EarthLayerCachePolicy,
|
||||
earth_layer_cache,
|
||||
format_bbox_key,
|
||||
get_or_build_layer_payload,
|
||||
quantize_bbox,
|
||||
)
|
||||
from app.core.logging import get_logger
|
||||
|
||||
router = APIRouter()
|
||||
@@ -68,6 +75,69 @@ TERRAIN_TILE_BATCH_MAX_ITEMS = 128
|
||||
TERRAIN_TILE_BATCH_CONCURRENCY = 16
|
||||
_terrain_tile_cache: OrderedDict[tuple[int, int, int], tuple[bytes, str, dict[str, str]]] = OrderedDict()
|
||||
VESSEL_NAME_FALLBACK_PATTERN = re.compile(r"^mmsi\s*\d+$", re.IGNORECASE)
|
||||
VESSEL_SNAPSHOT_LEGACY_FALLBACK_ENABLED = True
|
||||
SECONDS_PER_MINUTE = 60
|
||||
BYTES_PER_MIB = 1024 * 1024
|
||||
CABLE_CACHE_FRESH_SECONDS = 6 * 60 * SECONDS_PER_MINUTE
|
||||
CABLE_CACHE_STALE_SECONDS = 24 * 60 * SECONDS_PER_MINUTE
|
||||
SATELLITE_CACHE_FRESH_SECONDS = 15 * SECONDS_PER_MINUTE
|
||||
SATELLITE_CACHE_STALE_SECONDS = 2 * 60 * SECONDS_PER_MINUTE
|
||||
COMPUTE_CENTER_CACHE_FRESH_SECONDS = 10 * SECONDS_PER_MINUTE
|
||||
COMPUTE_CENTER_CACHE_STALE_SECONDS = 60 * SECONDS_PER_MINUTE
|
||||
BGP_CACHE_FRESH_SECONDS = 60
|
||||
BGP_EVENT_CACHE_FRESH_SECONDS = 30
|
||||
BGP_CACHE_STALE_SECONDS = 10 * SECONDS_PER_MINUTE
|
||||
VESSEL_SNAPSHOT_CACHE_FRESH_SECONDS = 5
|
||||
VESSEL_SNAPSHOT_CACHE_STALE_SECONDS = 30
|
||||
|
||||
CABLE_CACHE_POLICY = EarthLayerCachePolicy(
|
||||
CABLE_CACHE_FRESH_SECONDS,
|
||||
CABLE_CACHE_STALE_SECONDS,
|
||||
max_features=6000,
|
||||
max_bytes=10 * BYTES_PER_MIB,
|
||||
)
|
||||
LANDING_POINT_CACHE_POLICY = EarthLayerCachePolicy(
|
||||
CABLE_CACHE_FRESH_SECONDS,
|
||||
CABLE_CACHE_STALE_SECONDS,
|
||||
max_features=6000,
|
||||
max_bytes=8 * BYTES_PER_MIB,
|
||||
)
|
||||
SATELLITE_CACHE_POLICY = EarthLayerCachePolicy(
|
||||
SATELLITE_CACHE_FRESH_SECONDS,
|
||||
SATELLITE_CACHE_STALE_SECONDS,
|
||||
max_features=8000,
|
||||
max_bytes=10 * BYTES_PER_MIB,
|
||||
)
|
||||
COMPUTE_CENTER_CACHE_POLICY = EarthLayerCachePolicy(
|
||||
COMPUTE_CENTER_CACHE_FRESH_SECONDS,
|
||||
COMPUTE_CENTER_CACHE_STALE_SECONDS,
|
||||
max_features=1000,
|
||||
max_bytes=4 * BYTES_PER_MIB,
|
||||
)
|
||||
BGP_CACHE_POLICY = EarthLayerCachePolicy(
|
||||
BGP_CACHE_FRESH_SECONDS,
|
||||
BGP_CACHE_STALE_SECONDS,
|
||||
max_features=1000,
|
||||
max_bytes=3 * BYTES_PER_MIB,
|
||||
)
|
||||
BGP_EVENT_CACHE_POLICY = EarthLayerCachePolicy(
|
||||
BGP_EVENT_CACHE_FRESH_SECONDS,
|
||||
BGP_CACHE_STALE_SECONDS,
|
||||
max_features=1000,
|
||||
max_bytes=3 * BYTES_PER_MIB,
|
||||
)
|
||||
SUMMARY_CACHE_POLICY = EarthLayerCachePolicy(
|
||||
BGP_EVENT_CACHE_FRESH_SECONDS,
|
||||
BGP_CACHE_STALE_SECONDS,
|
||||
max_features=0,
|
||||
max_bytes=512 * 1024,
|
||||
)
|
||||
VESSEL_SNAPSHOT_CACHE_POLICY = EarthLayerCachePolicy(
|
||||
VESSEL_SNAPSHOT_CACHE_FRESH_SECONDS,
|
||||
VESSEL_SNAPSHOT_CACHE_STALE_SECONDS,
|
||||
max_features=1500,
|
||||
max_bytes=3 * BYTES_PER_MIB,
|
||||
)
|
||||
|
||||
|
||||
class TerrariumTileRequest(BaseModel):
|
||||
@@ -941,6 +1011,39 @@ def _merge_vessel_features(
|
||||
}
|
||||
|
||||
|
||||
async def _load_legacy_vessel_snapshot_features(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
bbox: tuple[float, float, float, float] | None,
|
||||
limit: int,
|
||||
) -> list[dict[str, Any]]:
|
||||
latest_positions = select(
|
||||
VesselPosition.mmsi.label("mmsi"),
|
||||
func.max(VesselPosition.received_at).label("received_at"),
|
||||
)
|
||||
if bbox is not None:
|
||||
lon_min, lat_min, lon_max, lat_max = bbox
|
||||
latest_positions = latest_positions.where(VesselPosition.lon >= lon_min)
|
||||
latest_positions = latest_positions.where(VesselPosition.lon <= lon_max)
|
||||
latest_positions = latest_positions.where(VesselPosition.lat >= lat_min)
|
||||
latest_positions = latest_positions.where(VesselPosition.lat <= lat_max)
|
||||
|
||||
latest_positions = latest_positions.group_by(VesselPosition.mmsi).subquery()
|
||||
result = await db.execute(
|
||||
select(VesselPosition, VesselStatic)
|
||||
.join(
|
||||
latest_positions,
|
||||
(VesselPosition.mmsi == latest_positions.c.mmsi)
|
||||
& (VesselPosition.received_at == latest_positions.c.received_at),
|
||||
)
|
||||
.outerjoin(VesselStatic, VesselStatic.mmsi == VesselPosition.mmsi)
|
||||
.order_by(VesselPosition.received_at.desc())
|
||||
.limit(limit)
|
||||
)
|
||||
legacy_geojson = convert_vessels_to_geojson(list(result.all()))
|
||||
return legacy_geojson.get("features", [])[:limit]
|
||||
|
||||
|
||||
def _build_vessel_stats(features: List[dict[str, Any]]) -> dict[str, Any]:
|
||||
by_type: dict[str, int] = {}
|
||||
underway = 0
|
||||
@@ -976,7 +1079,40 @@ async def build_vessel_snapshot_response(
|
||||
type_filter: str | None,
|
||||
limit: int | None,
|
||||
since_minutes: int = 60,
|
||||
response: Response | None = None,
|
||||
use_cache: bool = True,
|
||||
) -> dict[str, Any]:
|
||||
if use_cache and bbox is not None:
|
||||
safe_limit_for_key = _safe_vessel_limit(limit)
|
||||
safe_since_for_key = min(max(int(since_minutes or 60), 1), 1440)
|
||||
cache_key = earth_layer_cache.key(
|
||||
"vessels-snapshot",
|
||||
bbox=format_bbox_key(quantize_bbox(bbox)),
|
||||
zoom=zoom or "none",
|
||||
type=type_filter or "all",
|
||||
limit=safe_limit_for_key,
|
||||
since=safe_since_for_key,
|
||||
)
|
||||
|
||||
async def build_uncached() -> dict[str, Any]:
|
||||
return await build_vessel_snapshot_response(
|
||||
db,
|
||||
bbox=bbox,
|
||||
zoom=zoom,
|
||||
type_filter=type_filter,
|
||||
limit=limit,
|
||||
since_minutes=since_minutes,
|
||||
response=None,
|
||||
use_cache=False,
|
||||
)
|
||||
|
||||
return await get_or_build_layer_payload(
|
||||
key=cache_key,
|
||||
policy=VESSEL_SNAPSHOT_CACHE_POLICY,
|
||||
builder=build_uncached,
|
||||
response=response,
|
||||
)
|
||||
|
||||
requested_types = _requested_vessel_types(type_filter)
|
||||
safe_limit = _safe_vessel_limit(limit)
|
||||
safe_since_minutes = min(max(int(since_minutes or 60), 1), 1440)
|
||||
@@ -1410,8 +1546,20 @@ def convert_bgp_incidents_to_geojson(
|
||||
|
||||
|
||||
@router.get("/geo/cables")
|
||||
async def get_cables_geojson(db: AsyncSession = Depends(get_db)):
|
||||
async def get_cables_geojson(db: AsyncSession = Depends(get_db), response: Response = None):
|
||||
"""获取海底电缆 GeoJSON 数据 (LineString)"""
|
||||
async def build_payload() -> dict[str, Any]:
|
||||
return await _build_cables_geojson(db)
|
||||
|
||||
return await get_or_build_layer_payload(
|
||||
key=earth_layer_cache.key("cables"),
|
||||
policy=CABLE_CACHE_POLICY,
|
||||
builder=build_payload,
|
||||
response=response,
|
||||
)
|
||||
|
||||
|
||||
async def _build_cables_geojson(db: AsyncSession) -> dict[str, Any]:
|
||||
try:
|
||||
records = await _load_current_collected_data(db, "arcgis_cables")
|
||||
|
||||
@@ -1444,7 +1592,19 @@ async def get_cables_geojson(db: AsyncSession = Depends(get_db)):
|
||||
|
||||
|
||||
@router.get("/geo/landing-points")
|
||||
async def get_landing_points_geojson(db: AsyncSession = Depends(get_db)):
|
||||
async def get_landing_points_geojson(db: AsyncSession = Depends(get_db), response: Response = None):
|
||||
async def build_payload() -> dict[str, Any]:
|
||||
return await _build_landing_points_geojson(db)
|
||||
|
||||
return await get_or_build_layer_payload(
|
||||
key=earth_layer_cache.key("landing-points"),
|
||||
policy=LANDING_POINT_CACHE_POLICY,
|
||||
builder=build_payload,
|
||||
response=response,
|
||||
)
|
||||
|
||||
|
||||
async def _build_landing_points_geojson(db: AsyncSession) -> dict[str, Any]:
|
||||
try:
|
||||
records_by_source = await _load_current_collected_data_by_sources(
|
||||
db,
|
||||
@@ -1697,8 +1857,25 @@ async def get_satellites_geojson(
|
||||
description="Maximum number of satellites to return. Omit for no limit.",
|
||||
),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
response: Response = None,
|
||||
):
|
||||
"""获取卫星 TLE GeoJSON 数据"""
|
||||
async def build_payload() -> dict[str, Any]:
|
||||
return await _build_satellites_geojson(limit=limit, db=db)
|
||||
|
||||
return await get_or_build_layer_payload(
|
||||
key=earth_layer_cache.key("satellites", limit=limit or "all"),
|
||||
policy=SATELLITE_CACHE_POLICY,
|
||||
builder=build_payload,
|
||||
response=response,
|
||||
)
|
||||
|
||||
|
||||
async def _build_satellites_geojson(
|
||||
*,
|
||||
limit: int | None,
|
||||
db: AsyncSession,
|
||||
) -> dict[str, Any]:
|
||||
records = await _load_current_or_latest_task_data(
|
||||
db,
|
||||
"celestrak_tle",
|
||||
@@ -1766,8 +1943,25 @@ async def get_gpu_clusters_geojson(
|
||||
async def get_compute_centers_geojson(
|
||||
limit: int = Query(200, ge=1, le=1000),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
response: Response = None,
|
||||
):
|
||||
"""获取统一算力中心 GeoJSON 数据"""
|
||||
async def build_payload() -> dict[str, Any]:
|
||||
return await _build_compute_centers_geojson(limit=limit, db=db)
|
||||
|
||||
return await get_or_build_layer_payload(
|
||||
key=earth_layer_cache.key("compute-centers", limit=limit),
|
||||
policy=COMPUTE_CENTER_CACHE_POLICY,
|
||||
builder=build_payload,
|
||||
response=response,
|
||||
)
|
||||
|
||||
|
||||
async def _build_compute_centers_geojson(
|
||||
*,
|
||||
limit: int,
|
||||
db: AsyncSession,
|
||||
) -> dict[str, Any]:
|
||||
records_by_source = await _load_current_collected_data_by_sources(
|
||||
db,
|
||||
["top500", "epoch_ai_gpu"],
|
||||
@@ -1948,6 +2142,7 @@ async def collect_compute_center_location(
|
||||
provider_client=provider_client,
|
||||
query=query,
|
||||
entity_type="compute_center",
|
||||
db=db,
|
||||
attempted_queries=attempted_queries,
|
||||
search_evidence=search_result.evidence,
|
||||
)
|
||||
@@ -1964,6 +2159,24 @@ async def collect_compute_center_location(
|
||||
llm_failure_reason = llm_result.failure_reason
|
||||
|
||||
if not candidates:
|
||||
logger.warning_event(
|
||||
"Compute center location collection returned no candidates",
|
||||
event="visualization.compute_center.location_collect.completed",
|
||||
context={
|
||||
"source_id": source_id,
|
||||
"record_id": record_id,
|
||||
"name": name,
|
||||
"success": False,
|
||||
"llm_failure_reason": llm_failure_reason,
|
||||
"attempted_queries": list(attempted_queries),
|
||||
"context": {
|
||||
"operator": operator,
|
||||
"site": site,
|
||||
"city": city,
|
||||
"country": country,
|
||||
},
|
||||
},
|
||||
)
|
||||
return {
|
||||
"source_id": source_id,
|
||||
"record_id": record_id,
|
||||
@@ -1985,13 +2198,34 @@ async def collect_compute_center_location(
|
||||
},
|
||||
}
|
||||
|
||||
best_candidate = candidates[0].to_dict()
|
||||
logger.info_event(
|
||||
"Compute center location collection returned candidates",
|
||||
event="visualization.compute_center.location_collect.completed",
|
||||
context={
|
||||
"source_id": source_id,
|
||||
"record_id": record_id,
|
||||
"name": name,
|
||||
"success": True,
|
||||
"candidate_count": len(candidates),
|
||||
"best_candidate": best_candidate,
|
||||
"llm_failure_reason": llm_failure_reason,
|
||||
"attempted_queries": list(attempted_queries),
|
||||
"context": {
|
||||
"operator": operator,
|
||||
"site": site,
|
||||
"city": city,
|
||||
"country": country,
|
||||
},
|
||||
},
|
||||
)
|
||||
return {
|
||||
"source_id": source_id,
|
||||
"record_id": record_id,
|
||||
"name": name,
|
||||
"success": True,
|
||||
"candidates": [candidate.to_dict() for candidate in candidates],
|
||||
"best_candidate": candidates[0].to_dict(),
|
||||
"best_candidate": best_candidate,
|
||||
"attempted_queries": list(attempted_queries),
|
||||
"context": {
|
||||
"name": name,
|
||||
@@ -2073,33 +2307,6 @@ async def _load_compute_center_record(db: AsyncSession, source_id: str) -> Colle
|
||||
return result.scalars().first()
|
||||
|
||||
|
||||
@router.get("/geo/vessels")
|
||||
async def get_vessels_geojson(
|
||||
bbox: Optional[str] = Query(
|
||||
None,
|
||||
description="Viewport bbox as lon_min,lat_min,lon_max,lat_max",
|
||||
),
|
||||
type: Optional[str] = Query(
|
||||
None,
|
||||
description="Comma-separated vessel types: cargo,tanker,passenger,fishing,military,other",
|
||||
),
|
||||
limit: Optional[int] = Query(
|
||||
None,
|
||||
ge=0,
|
||||
description="Maximum vessel features to return. Omit or pass 0 for no limit.",
|
||||
),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""Legacy vessel endpoint removed in favor of /api/v1/vessels/snapshot."""
|
||||
raise HTTPException(
|
||||
status_code=410,
|
||||
detail=(
|
||||
"Legacy vessel GeoJSON endpoint has been removed. "
|
||||
"Use /api/v1/vessels/snapshot with bbox, zoom, and limit."
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
async def _load_raw_vessel_snapshot_features(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
@@ -2121,18 +2328,38 @@ async def _load_raw_vessel_snapshot_features(
|
||||
observed_since=observed_since,
|
||||
)
|
||||
raw_geojson = convert_aggregated_vessels_to_geojson(aggregated_vessels)
|
||||
features = raw_geojson.get("features", [])
|
||||
raw_features = raw_geojson.get("features", [])
|
||||
features = raw_features
|
||||
legacy_features: list[dict[str, Any]] = []
|
||||
legacy_fallback_used = False
|
||||
if not raw_features and VESSEL_SNAPSHOT_LEGACY_FALLBACK_ENABLED:
|
||||
legacy_features = await _load_legacy_vessel_snapshot_features(
|
||||
db,
|
||||
bbox=bbox,
|
||||
limit=limit,
|
||||
)
|
||||
features, _merge_diagnostics = _merge_vessel_features(raw_features, legacy_features)
|
||||
legacy_fallback_used = bool(legacy_features)
|
||||
|
||||
return features, {
|
||||
"raw_feature_count": len(features),
|
||||
"raw_feature_count": len(raw_features),
|
||||
"raw_unique_mmsi": len(
|
||||
{
|
||||
key
|
||||
for key in (_feature_mmsi_key(feature) for feature in features)
|
||||
for key in (_feature_mmsi_key(feature) for feature in raw_features)
|
||||
if key is not None
|
||||
}
|
||||
),
|
||||
"legacy_feature_count": 0,
|
||||
"legacy_backfilled_mmsi": 0,
|
||||
"legacy_feature_count": len(legacy_features),
|
||||
"legacy_backfilled_mmsi": len(
|
||||
{
|
||||
key
|
||||
for key in (_feature_mmsi_key(feature) for feature in legacy_features)
|
||||
if key is not None
|
||||
}
|
||||
),
|
||||
"legacy_fallback_enabled": VESSEL_SNAPSHOT_LEGACY_FALLBACK_ENABLED,
|
||||
"legacy_fallback_used": legacy_fallback_used,
|
||||
"final_unique_mmsi": len(
|
||||
{
|
||||
key
|
||||
@@ -2369,7 +2596,31 @@ async def get_bgp_anomalies_geojson(
|
||||
status: Optional[str] = Query("active"),
|
||||
limit: int = Query(200, ge=1, le=1000),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
response: Response = None,
|
||||
):
|
||||
async def build_payload() -> dict[str, Any]:
|
||||
return await _build_bgp_anomalies_geojson(
|
||||
severity=severity,
|
||||
status=status,
|
||||
limit=limit,
|
||||
db=db,
|
||||
)
|
||||
|
||||
return await get_or_build_layer_payload(
|
||||
key=earth_layer_cache.key("bgp-anomalies", severity=severity or "all", status=status or "all", limit=limit),
|
||||
policy=BGP_EVENT_CACHE_POLICY,
|
||||
builder=build_payload,
|
||||
response=response,
|
||||
)
|
||||
|
||||
|
||||
async def _build_bgp_anomalies_geojson(
|
||||
*,
|
||||
severity: str | None,
|
||||
status: str | None,
|
||||
limit: int,
|
||||
db: AsyncSession,
|
||||
) -> dict[str, Any]:
|
||||
stmt = select(BGPAnomaly).order_by(BGPAnomaly.created_at.desc()).limit(limit)
|
||||
if severity:
|
||||
stmt = stmt.where(BGPAnomaly.severity == severity)
|
||||
@@ -2389,7 +2640,31 @@ async def get_bgp_incidents_geojson(
|
||||
status: Optional[str] = Query("active"),
|
||||
limit: int = Query(100, ge=1, le=500),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
response: Response = None,
|
||||
):
|
||||
async def build_payload() -> dict[str, Any]:
|
||||
return await _build_bgp_incidents_geojson(
|
||||
severity=severity,
|
||||
status=status,
|
||||
limit=limit,
|
||||
db=db,
|
||||
)
|
||||
|
||||
return await get_or_build_layer_payload(
|
||||
key=earth_layer_cache.key("bgp-incidents", severity=severity or "all", status=status or "all", limit=limit),
|
||||
policy=BGP_EVENT_CACHE_POLICY,
|
||||
builder=build_payload,
|
||||
response=response,
|
||||
)
|
||||
|
||||
|
||||
async def _build_bgp_incidents_geojson(
|
||||
*,
|
||||
severity: str | None,
|
||||
status: str | None,
|
||||
limit: int,
|
||||
db: AsyncSession,
|
||||
) -> dict[str, Any]:
|
||||
stmt = select(BGPIncident).order_by(BGPIncident.created_at.desc()).limit(limit)
|
||||
if severity:
|
||||
stmt = stmt.where(BGPIncident.severity == severity)
|
||||
@@ -2404,7 +2679,19 @@ async def get_bgp_incidents_geojson(
|
||||
|
||||
|
||||
@router.get("/geo/bgp-collectors")
|
||||
async def get_bgp_collectors_geojson(db: AsyncSession = Depends(get_db)):
|
||||
async def get_bgp_collectors_geojson(db: AsyncSession = Depends(get_db), response: Response = None):
|
||||
async def build_payload() -> dict[str, Any]:
|
||||
return await _build_bgp_collectors_geojson(db)
|
||||
|
||||
return await get_or_build_layer_payload(
|
||||
key=earth_layer_cache.key("bgp-collectors"),
|
||||
policy=BGP_CACHE_POLICY,
|
||||
builder=build_payload,
|
||||
response=response,
|
||||
)
|
||||
|
||||
|
||||
async def _build_bgp_collectors_geojson(db: AsyncSession) -> dict[str, Any]:
|
||||
coverage = await build_bgp_collector_coverage(
|
||||
db,
|
||||
source_filter=("ris_live_bgp", "bgpstream_bgp"),
|
||||
@@ -2419,8 +2706,20 @@ async def get_bgp_collectors_geojson(db: AsyncSession = Depends(get_db)):
|
||||
|
||||
|
||||
@router.get("/geo/summary")
|
||||
async def get_visualization_geo_summary(db: AsyncSession = Depends(get_db)):
|
||||
async def get_visualization_geo_summary(db: AsyncSession = Depends(get_db), response: Response = None):
|
||||
"""Return lightweight Earth HUD counts without loading layer GeoJSON payloads."""
|
||||
async def build_payload() -> dict[str, Any]:
|
||||
return await _build_visualization_geo_summary(db)
|
||||
|
||||
return await get_or_build_layer_payload(
|
||||
key=earth_layer_cache.key("summary"),
|
||||
policy=SUMMARY_CACHE_POLICY,
|
||||
builder=build_payload,
|
||||
response=response,
|
||||
)
|
||||
|
||||
|
||||
async def _build_visualization_geo_summary(db: AsyncSession) -> dict[str, Any]:
|
||||
cable_count = await _count_current_or_latest_task_data(db, "arcgis_cables")
|
||||
landing_point_count = await _count_current_or_latest_task_data(db, "arcgis_landing_points")
|
||||
satellite_count = await _count_current_or_latest_task_data(
|
||||
@@ -2466,7 +2765,12 @@ async def get_visualization_geo_summary(db: AsyncSession = Depends(get_db)):
|
||||
select(func.count(func.distinct(VesselPosition.mmsi)))
|
||||
)
|
||||
legacy_unique_mmsi = int(legacy_unique_result.scalar() or 0)
|
||||
vessel_count = max(raw_unique_mmsi, legacy_unique_mmsi)
|
||||
legacy_fallback_active = (
|
||||
VESSEL_SNAPSHOT_LEGACY_FALLBACK_ENABLED
|
||||
and raw_unique_mmsi == 0
|
||||
and legacy_unique_mmsi > 0
|
||||
)
|
||||
vessel_count = legacy_unique_mmsi if legacy_fallback_active else raw_unique_mmsi
|
||||
aisstream_health = await db.get(AISSourceHealth, "aisstream_vessels")
|
||||
|
||||
return {
|
||||
@@ -2477,6 +2781,8 @@ async def get_visualization_geo_summary(db: AsyncSession = Depends(get_db)):
|
||||
"satellite_count": satellite_count,
|
||||
"compute_center_count": compute_center_count,
|
||||
"vessel_count": vessel_count,
|
||||
"vessel_count_source": "legacy_fallback" if legacy_fallback_active else "raw_recent",
|
||||
"vessel_legacy_fallback_enabled": VESSEL_SNAPSHOT_LEGACY_FALLBACK_ENABLED,
|
||||
"vessel_raw_unique_mmsi": raw_unique_mmsi,
|
||||
"vessel_raw_unique_window_hours": raw_unique_window_hours,
|
||||
"vessel_legacy_unique_mmsi": legacy_unique_mmsi,
|
||||
|
||||
@@ -14,6 +14,7 @@ from app.core.websocket.manager import manager
|
||||
|
||||
logger = get_logger(__name__, service="api")
|
||||
router = APIRouter()
|
||||
EARTH_UPDATES_CHANNEL = "earth_updates"
|
||||
|
||||
|
||||
async def authenticate_token(token: str) -> Optional[dict]:
|
||||
@@ -58,7 +59,7 @@ async def websocket_endpoint(
|
||||
|
||||
is_anonymous = payload is None
|
||||
user_id = str(payload.get("sub")) if payload else f"anonymous:{id(websocket)}"
|
||||
supported_channels = ["vessels"] if is_anonymous else [
|
||||
supported_channels = ["vessels", "earth_news", EARTH_UPDATES_CHANNEL] if is_anonymous else [
|
||||
"gpu_clusters",
|
||||
"submarine_cables",
|
||||
"ixp_nodes",
|
||||
@@ -66,6 +67,8 @@ async def websocket_endpoint(
|
||||
"dashboard",
|
||||
"datasource_tasks",
|
||||
"vessels",
|
||||
"earth_news",
|
||||
EARTH_UPDATES_CHANNEL,
|
||||
]
|
||||
await manager.connect(websocket, user_id)
|
||||
|
||||
|
||||
@@ -257,6 +257,16 @@ DEFAULT_DATASOURCES = {
|
||||
"credential_provider": "aisstream",
|
||||
"credential_status": "supported",
|
||||
},
|
||||
"media_news_archive": {
|
||||
"id": 33,
|
||||
"name": "Media News Archive",
|
||||
"display_name": "媒体新闻归档",
|
||||
"module": "L4",
|
||||
"priority": "P2",
|
||||
"frequency_minutes": 720,
|
||||
"is_free": True,
|
||||
"requires_credentials": False,
|
||||
},
|
||||
}
|
||||
|
||||
ID_TO_COLLECTOR = {info["id"]: name for name, info in DEFAULT_DATASOURCES.items()}
|
||||
|
||||
@@ -8,6 +8,8 @@ from app.core.time import to_iso8601_utc
|
||||
from app.core.websocket.manager import manager
|
||||
|
||||
|
||||
EARTH_UPDATES_CHANNEL = "earth_updates"
|
||||
|
||||
|
||||
class DataBroadcaster:
|
||||
"""Periodically broadcasts data to connected WebSocket clients"""
|
||||
@@ -83,6 +85,10 @@ class DataBroadcaster:
|
||||
channel=channel,
|
||||
)
|
||||
|
||||
async def broadcast_earth_update(self, data: Dict[str, Any]):
|
||||
"""Broadcast Earth visualization refresh hints to connected clients."""
|
||||
await self.broadcast_custom(EARTH_UPDATES_CHANNEL, data)
|
||||
|
||||
def enqueue_vessel_update(self, data: Dict[str, Any]):
|
||||
vessels = data.get("vessels") if isinstance(data, dict) else None
|
||||
if not isinstance(vessels, list):
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
from typing import AsyncGenerator
|
||||
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy import bindparam, text
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, create_async_engine, async_sessionmaker
|
||||
from sqlalchemy.orm import declarative_base
|
||||
|
||||
@@ -72,6 +72,74 @@ async def seed_default_datasources(session: AsyncSession):
|
||||
await session.commit()
|
||||
|
||||
|
||||
LEGACY_EARTH_BOUNDARY_SOURCES = (
|
||||
"earth_admin0_boundaries",
|
||||
"earth_coastline",
|
||||
"earth_claim_lines",
|
||||
"earth_boundary_tiles",
|
||||
)
|
||||
LEGACY_EARTH_BOUNDARY_DATATYPES = (
|
||||
"earth_boundary_source",
|
||||
"earth_boundary_tiles",
|
||||
)
|
||||
LEGACY_EARTH_BOUNDARY_IDS = (29, 30, 31, 32)
|
||||
|
||||
|
||||
async def purge_legacy_earth_boundary_datasources(session: AsyncSession) -> None:
|
||||
source_names = tuple(LEGACY_EARTH_BOUNDARY_SOURCES)
|
||||
source_ids = tuple(LEGACY_EARTH_BOUNDARY_IDS)
|
||||
data_types = tuple(LEGACY_EARTH_BOUNDARY_DATATYPES)
|
||||
await session.execute(
|
||||
text(
|
||||
"""
|
||||
DELETE FROM datasource_mapping_templates
|
||||
WHERE target_schema IN :data_types
|
||||
OR datasource_config_id IN (
|
||||
SELECT id FROM datasource_configs WHERE name IN :source_names
|
||||
)
|
||||
"""
|
||||
).bindparams(bindparam("source_names", expanding=True), bindparam("data_types", expanding=True)),
|
||||
{"source_names": list(source_names), "data_types": list(data_types)},
|
||||
)
|
||||
await session.execute(
|
||||
text("DELETE FROM datasource_configs WHERE name IN :source_names").bindparams(
|
||||
bindparam("source_names", expanding=True)
|
||||
),
|
||||
{"source_names": list(source_names)},
|
||||
)
|
||||
await session.execute(
|
||||
text(
|
||||
"""
|
||||
DELETE FROM collected_data
|
||||
WHERE source IN :source_names OR data_type IN :data_types
|
||||
"""
|
||||
).bindparams(bindparam("source_names", expanding=True), bindparam("data_types", expanding=True)),
|
||||
{"source_names": list(source_names), "data_types": list(data_types)},
|
||||
)
|
||||
await session.execute(
|
||||
text(
|
||||
"""
|
||||
DELETE FROM data_snapshots
|
||||
WHERE source IN :source_names OR datasource_id IN :source_ids
|
||||
"""
|
||||
).bindparams(bindparam("source_names", expanding=True), bindparam("source_ids", expanding=True)),
|
||||
{"source_names": list(source_names), "source_ids": list(source_ids)},
|
||||
)
|
||||
await session.execute(
|
||||
text("DELETE FROM collection_tasks WHERE datasource_id IN :source_ids").bindparams(
|
||||
bindparam("source_ids", expanding=True)
|
||||
),
|
||||
{"source_ids": list(source_ids)},
|
||||
)
|
||||
await session.execute(
|
||||
text("DELETE FROM data_sources WHERE source IN :source_names OR id IN :source_ids").bindparams(
|
||||
bindparam("source_names", expanding=True), bindparam("source_ids", expanding=True)
|
||||
),
|
||||
{"source_names": list(source_names), "source_ids": list(source_ids)},
|
||||
)
|
||||
await session.commit()
|
||||
|
||||
|
||||
DEFAULT_LOGIN_USERS = (
|
||||
{
|
||||
"username": "admin",
|
||||
@@ -134,6 +202,7 @@ async def init_db():
|
||||
import app.models.vessel # noqa: F401
|
||||
import app.models.vessel_enrichment # noqa: F401
|
||||
import app.models.datasource_mapping # noqa: F401
|
||||
import app.models.earth_news # noqa: F401
|
||||
|
||||
logger.warning_event(
|
||||
"Database pool settings active",
|
||||
@@ -202,6 +271,18 @@ async def init_db():
|
||||
"""
|
||||
)
|
||||
)
|
||||
await conn.execute(
|
||||
text(
|
||||
"""
|
||||
ALTER TABLE earth_news_items
|
||||
ADD COLUMN IF NOT EXISTS content_language VARCHAR(32) NOT NULL DEFAULT 'en',
|
||||
ADD COLUMN IF NOT EXISTS localizations JSONB NOT NULL DEFAULT '{}'::jsonb,
|
||||
ADD COLUMN IF NOT EXISTS enrichment_status VARCHAR(80) NOT NULL DEFAULT 'pending',
|
||||
ADD COLUMN IF NOT EXISTS enrichment_error TEXT,
|
||||
ADD COLUMN IF NOT EXISTS enriched_at TIMESTAMPTZ
|
||||
"""
|
||||
)
|
||||
)
|
||||
await conn.execute(
|
||||
text(
|
||||
"""
|
||||
@@ -210,6 +291,22 @@ async def init_db():
|
||||
"""
|
||||
)
|
||||
)
|
||||
await conn.execute(
|
||||
text(
|
||||
"""
|
||||
CREATE INDEX IF NOT EXISTS idx_earth_news_enrichment_status
|
||||
ON earth_news_items (enrichment_status)
|
||||
"""
|
||||
)
|
||||
)
|
||||
await conn.execute(
|
||||
text(
|
||||
"""
|
||||
CREATE INDEX IF NOT EXISTS idx_earth_news_enriched_at
|
||||
ON earth_news_items (enriched_at)
|
||||
"""
|
||||
)
|
||||
)
|
||||
await conn.execute(
|
||||
text(
|
||||
"""
|
||||
@@ -284,4 +381,5 @@ async def init_db():
|
||||
await seed_default_bgp_collector_locations(session)
|
||||
await seed_compute_center_locations_from_source_coords(session)
|
||||
await seed_default_datasources(session)
|
||||
await purge_legacy_earth_boundary_datasources(session)
|
||||
await ensure_default_admin_user(session)
|
||||
|
||||
@@ -1,8 +1,10 @@
|
||||
from contextlib import asynccontextmanager
|
||||
from pathlib import Path
|
||||
from uuid import uuid4
|
||||
|
||||
from fastapi import FastAPI
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
from fastapi.staticfiles import StaticFiles
|
||||
from starlette.middleware.base import BaseHTTPMiddleware
|
||||
|
||||
from app.api.main import api_router
|
||||
@@ -18,6 +20,10 @@ from app.services.scheduler import (
|
||||
stop_scheduler,
|
||||
sync_scheduler_with_datasources,
|
||||
)
|
||||
from app.services.earth_news_worker import (
|
||||
start_earth_news_target_worker,
|
||||
stop_earth_news_target_worker,
|
||||
)
|
||||
|
||||
|
||||
configure_logging()
|
||||
@@ -53,7 +59,9 @@ async def lifespan(app: FastAPI):
|
||||
start_scheduler()
|
||||
await sync_scheduler_with_datasources()
|
||||
broadcaster.start()
|
||||
start_earth_news_target_worker()
|
||||
yield
|
||||
await stop_earth_news_target_worker()
|
||||
broadcaster.stop()
|
||||
stop_scheduler()
|
||||
|
||||
@@ -82,6 +90,14 @@ app.add_middleware(WebSocketCORSMiddleware)
|
||||
app.include_router(api_router, prefix="/api/v1")
|
||||
app.include_router(websocket.router)
|
||||
|
||||
EARTH_BRAND_ASSET_DIR = Path(__file__).resolve().parents[2] / "data" / "earth-brand"
|
||||
EARTH_BRAND_ASSET_DIR.mkdir(parents=True, exist_ok=True)
|
||||
app.mount(
|
||||
"/earth-brand-assets",
|
||||
StaticFiles(directory=str(EARTH_BRAND_ASSET_DIR)),
|
||||
name="earth-brand-assets",
|
||||
)
|
||||
|
||||
|
||||
@app.get("/health")
|
||||
async def health_check():
|
||||
|
||||
@@ -16,6 +16,7 @@ from app.models.playground_message import PlaygroundMessage
|
||||
from app.models.system_log import SystemLog, AuditLog
|
||||
from app.models.vessel import AISConflictRecord, AISRawObservation, AISSourceHealth, VesselPosition, VesselStatic
|
||||
from app.models.datasource_mapping import DataSourceMappingTemplate
|
||||
from app.models.earth_news import EarthNewsItem
|
||||
|
||||
__all__ = [
|
||||
"User",
|
||||
@@ -43,4 +44,5 @@ __all__ = [
|
||||
"AISConflictRecord",
|
||||
"AISSourceHealth",
|
||||
"DataSourceMappingTemplate",
|
||||
"EarthNewsItem",
|
||||
]
|
||||
|
||||
40
backend/app/models/earth_news.py
Normal file
40
backend/app/models/earth_news.py
Normal file
@@ -0,0 +1,40 @@
|
||||
from sqlalchemy import Boolean, Column, DateTime, Float, Index, JSON, String, Text
|
||||
from sqlalchemy.sql import func
|
||||
|
||||
from app.db.session import Base
|
||||
|
||||
|
||||
class EarthNewsItem(Base):
|
||||
__tablename__ = "earth_news_items"
|
||||
|
||||
id = Column(String(160), primary_key=True)
|
||||
title = Column(String(500), nullable=False)
|
||||
summary = Column(Text, nullable=False, default="")
|
||||
content_language = Column(String(32), nullable=False, default="en")
|
||||
localizations = Column(JSON, nullable=False, default=dict)
|
||||
url = Column(Text, nullable=False)
|
||||
source = Column(String(255), nullable=False, default="")
|
||||
feed_name = Column(String(255), nullable=False, default="")
|
||||
region = Column(String(80), nullable=False, index=True)
|
||||
homepage_url = Column(Text, nullable=False, default="")
|
||||
published_at = Column(DateTime(timezone=True), nullable=True, index=True)
|
||||
|
||||
latitude = Column(Float, nullable=False)
|
||||
longitude = Column(Float, nullable=False)
|
||||
location_label = Column(String(255), nullable=False)
|
||||
location_source = Column(String(80), nullable=False, default="region_anchor")
|
||||
verified = Column(Boolean, nullable=False, default=False, index=True)
|
||||
location_meta = Column(JSON, nullable=False, default=dict)
|
||||
|
||||
first_seen_at = Column(DateTime(timezone=True), server_default=func.now(), nullable=False)
|
||||
last_seen_at = Column(DateTime(timezone=True), server_default=func.now(), nullable=False, index=True)
|
||||
resolved_at = Column(DateTime(timezone=True), nullable=True, index=True)
|
||||
enrichment_status = Column(String(80), nullable=False, default="pending", index=True)
|
||||
enrichment_error = Column(Text, nullable=True)
|
||||
enriched_at = Column(DateTime(timezone=True), nullable=True, index=True)
|
||||
updated_at = Column(DateTime(timezone=True), server_default=func.now(), onupdate=func.now())
|
||||
|
||||
__table_args__ = (
|
||||
Index("idx_earth_news_region_published", "region", "published_at"),
|
||||
Index("idx_earth_news_region_seen", "region", "last_seen_at"),
|
||||
)
|
||||
@@ -13,10 +13,11 @@ class AIContentBlock(BaseModel):
|
||||
|
||||
class SituationalAnalysisRequest(BaseModel):
|
||||
title: str = Field(..., min_length=1, max_length=200)
|
||||
objective: str = Field(..., min_length=1, max_length=1000)
|
||||
objective: str = Field(..., min_length=1, max_length=20000)
|
||||
context: dict[str, Any] = Field(default_factory=dict)
|
||||
observations: list[str] = Field(default_factory=list)
|
||||
constraints: list[str] = Field(default_factory=list)
|
||||
system_prompt: str | None = Field(default=None, max_length=8000)
|
||||
preferred_model: str | None = Field(default=None, max_length=200)
|
||||
thinking: dict[str, Any] | None = None
|
||||
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
|
||||
import httpx
|
||||
from fastapi import Depends, HTTPException, status
|
||||
@@ -57,6 +58,9 @@ class AIProviderClient:
|
||||
value = self.llm_config.get(key)
|
||||
if value not in (None, ""):
|
||||
headers[header_name] = str(value)
|
||||
model_provider_apis = self.llm_config.get("model_provider_apis")
|
||||
if isinstance(model_provider_apis, dict) and model_provider_apis:
|
||||
headers["X-AI-Model-Provider-APIs"] = json.dumps(model_provider_apis)
|
||||
return headers
|
||||
|
||||
async def get_status(self, request_id: str | None = None) -> AIProviderStatusResponse:
|
||||
|
||||
@@ -8,6 +8,9 @@ from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.models.alert import Alert, AlertSeverity, AlertStatus
|
||||
from app.schemas.ai import AlertBriefRequest, SituationalAnalysisRequest
|
||||
from app.ai_tasks.prompts import get_effective_prompt
|
||||
|
||||
ALERT_BRIEF_PROMPT_KEY = "alerts.brief"
|
||||
|
||||
|
||||
def _format_counter(counter: Counter[str], empty_text: str = "无") -> str:
|
||||
@@ -84,11 +87,13 @@ async def build_alert_brief_request(
|
||||
"top_datasources": dict(datasource_counts.most_common(6)),
|
||||
"top_active_datasources": dict(active_datasource_counts.most_common(5)),
|
||||
}
|
||||
prompt = await get_effective_prompt(db, ALERT_BRIEF_PROMPT_KEY)
|
||||
|
||||
return (
|
||||
SituationalAnalysisRequest(
|
||||
title="告警态势 AI 简报",
|
||||
objective="基于当前告警总量、严重度、状态、数据源分布与最近告警摘录,生成一份面向值班人员的简明告警态势简报,突出待处理风险、告警集中点和优先动作。",
|
||||
objective=prompt.prompt,
|
||||
system_prompt=prompt.system_prompt or None,
|
||||
observations=facts,
|
||||
constraints=[
|
||||
"明确区分事实、推断与建议。",
|
||||
|
||||
@@ -11,9 +11,12 @@ from app.models.bgp_anomaly import BGPAnomaly
|
||||
from app.models.bgp_incident import BGPIncident
|
||||
from app.models.bgp_observation import BGPObservation
|
||||
from app.schemas.ai import SituationalAnalysisRequest
|
||||
from app.ai_tasks.prompts import get_effective_prompt
|
||||
from app.services.bgp_collectors import build_bgp_collector_coverage
|
||||
from app.services.bgp_enrichment import lookup_prefix_geography
|
||||
|
||||
BGP_BRIEF_PROMPT_KEY = "bgp.brief"
|
||||
|
||||
|
||||
def _format_counter(counter: dict[str, int], empty_text: str = "无") -> str:
|
||||
if not counter:
|
||||
@@ -243,12 +246,15 @@ async def build_bgp_brief_request(
|
||||
for prefix, item in list(prefix_geographies.items())[:8]
|
||||
},
|
||||
}
|
||||
prompt = await get_effective_prompt(db, BGP_BRIEF_PROMPT_KEY)
|
||||
|
||||
return SituationalAnalysisRequest(
|
||||
title="BGP 态势 AI 简报",
|
||||
objective="基于当前 BGP incidents、anomalies、原始观测事件、观测站覆盖与 prefix geography 证据,生成一份面向操作员的简明态势简报,突出区域热点、观测偏差、当前风险、证据和优先动作。",
|
||||
objective=prompt.prompt,
|
||||
system_prompt=prompt.system_prompt or None,
|
||||
observations=observations_lines,
|
||||
constraints=[
|
||||
"直接输出中文 Markdown 简报正文,不要输出英文写作计划、提示词复述、字段说明或元评论。",
|
||||
"明确区分事实、推断与建议。",
|
||||
"优先指出需要立即关注的高严重度 incident 或异常模式。",
|
||||
"需要单独指出哪些区域结论来自 prefix geography / affected regions,哪些可能受 collector coverage 偏差影响。",
|
||||
|
||||
@@ -36,6 +36,7 @@ from app.services.collectors.iptoasn import IPtoASNPrefixGeoCollector
|
||||
from app.services.collectors.opengeofeed import OpenGeoFeedPrefixGeoCollector
|
||||
from app.services.collectors.nro_delegated import NRODelegatedPrefixGeoCollector
|
||||
from app.services.collectors.news_live_streams import NewsLiveStreamsCollector
|
||||
from app.services.collectors.media_news_archive import MediaNewsArchiveCollector
|
||||
from app.services.collectors.aisstream import AISStreamCollector
|
||||
from app.services.collectors.vessel_ais import VesselAISCollector
|
||||
|
||||
@@ -65,6 +66,7 @@ collector_registry.register(IPtoASNPrefixGeoCollector())
|
||||
collector_registry.register(OpenGeoFeedPrefixGeoCollector())
|
||||
collector_registry.register(NRODelegatedPrefixGeoCollector())
|
||||
collector_registry.register(NewsLiveStreamsCollector())
|
||||
collector_registry.register(MediaNewsArchiveCollector())
|
||||
collector_registry.register(VesselAISCollector())
|
||||
collector_registry.register(AISStreamCollector())
|
||||
|
||||
@@ -100,6 +102,7 @@ __all__ = [
|
||||
"OpenGeoFeedPrefixGeoCollector",
|
||||
"NRODelegatedPrefixGeoCollector",
|
||||
"NewsLiveStreamsCollector",
|
||||
"MediaNewsArchiveCollector",
|
||||
"VesselAISCollector",
|
||||
"AISStreamCollector",
|
||||
]
|
||||
|
||||
@@ -9,10 +9,37 @@ from sqlalchemy import select, text
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.collected_data_fields import build_dynamic_metadata, get_record_field
|
||||
from app.core.config import settings
|
||||
from app.core.countries import normalize_country
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.core.websocket.broadcaster import broadcaster
|
||||
from app.services.earth_layer_cache import invalidate_earth_layer_cache_for_source
|
||||
|
||||
EARTH_UPDATE_LAYER_HINTS: dict[str, list[str]] = {
|
||||
"ris_live_bgp": ["bgp"],
|
||||
"bgpstream_bgp": ["bgp"],
|
||||
"top500_supercomputers": ["computeCenters"],
|
||||
"epoch_ai_gpu": ["computeCenters"],
|
||||
"huggingface_models": ["computeCenters"],
|
||||
"huggingface_datasets": ["computeCenters"],
|
||||
"huggingface_spaces": ["computeCenters"],
|
||||
"telegeography_cables": ["cables"],
|
||||
"telegeography_landing_points": ["cables"],
|
||||
"telegeography_cable_systems": ["cables"],
|
||||
"arcgis_cables": ["cables"],
|
||||
"fao_landing_points": ["cables"],
|
||||
"arcgis_landing_points": ["cables"],
|
||||
"arcgis_cable_landing_relations": ["cables"],
|
||||
"spacetrack_tle": ["satellites"],
|
||||
"celestrak_tle": ["satellites"],
|
||||
"barentswatch_vessels": ["vessels"],
|
||||
"aisstream_vessels": ["vessels"],
|
||||
"news_live_streams": ["media"],
|
||||
"media_news_archive": ["news"],
|
||||
}
|
||||
|
||||
|
||||
def get_earth_update_layers_for_source(source: str) -> list[str]:
|
||||
return EARTH_UPDATE_LAYER_HINTS.get(source, [])
|
||||
|
||||
|
||||
class BaseCollector(ABC):
|
||||
@@ -69,6 +96,29 @@ class BaseCollector(ABC):
|
||||
)
|
||||
self._last_broadcast_progress = rounded_progress
|
||||
|
||||
async def _publish_earth_update(
|
||||
self,
|
||||
*,
|
||||
action: str,
|
||||
records_processed: int,
|
||||
task_id: int | None = None,
|
||||
) -> None:
|
||||
layers = get_earth_update_layers_for_source(self.name)
|
||||
if not layers:
|
||||
return
|
||||
await broadcaster.broadcast_earth_update(
|
||||
{
|
||||
"action": action,
|
||||
"source": self.name,
|
||||
"data_type": self.data_type,
|
||||
"layers": layers,
|
||||
"datasource_id": getattr(self, "_datasource_id", None),
|
||||
"task_id": task_id,
|
||||
"records_processed": records_processed,
|
||||
"timestamp": to_iso8601_utc(datetime.now(UTC)),
|
||||
}
|
||||
)
|
||||
|
||||
async def update_progress(self, records_processed: int, *, commit: bool = False, force: bool = False):
|
||||
"""Update task progress - call this during data processing"""
|
||||
if self._current_task and self._db_session:
|
||||
@@ -186,7 +236,7 @@ class BaseCollector(ABC):
|
||||
|
||||
result = await db.execute(
|
||||
select(DataSnapshot)
|
||||
.where(DataSnapshot.source == self.name, DataSnapshot.is_current == True)
|
||||
.where(DataSnapshot.source == self.name, DataSnapshot.is_current.is_(True))
|
||||
.order_by(DataSnapshot.completed_at.desc().nullslast(), DataSnapshot.id.desc())
|
||||
.limit(1)
|
||||
)
|
||||
@@ -324,6 +374,11 @@ class BaseCollector(ABC):
|
||||
task.completed_at = datetime.now(UTC)
|
||||
await db.commit()
|
||||
await self._publish_task_update(force=True)
|
||||
await self._publish_earth_update(
|
||||
action="collector_completed",
|
||||
records_processed=records_count,
|
||||
task_id=task_id,
|
||||
)
|
||||
|
||||
return {
|
||||
"status": "success",
|
||||
@@ -405,7 +460,7 @@ class BaseCollector(ABC):
|
||||
select(CollectedData)
|
||||
.where(
|
||||
CollectedData.source == self.name,
|
||||
CollectedData.is_current == True,
|
||||
CollectedData.is_current.is_(True),
|
||||
)
|
||||
.order_by(CollectedData.entity_key.asc(), CollectedData.collected_at.desc().nullslast(), CollectedData.id.desc())
|
||||
)
|
||||
@@ -531,6 +586,7 @@ class BaseCollector(ABC):
|
||||
}
|
||||
|
||||
await db.commit()
|
||||
invalidate_earth_layer_cache_for_source(self.name)
|
||||
await self.update_progress(len(data), force=True)
|
||||
return records_added
|
||||
|
||||
|
||||
@@ -19,6 +19,7 @@ from app.services.bgp_collector_locations import (
|
||||
)
|
||||
from app.services.bgp_event_locations import resolve_bgp_event_geo_dict
|
||||
from app.services.bgp_incidents import create_bgp_incidents_for_anomalies
|
||||
from app.services.earth_layer_cache import invalidate_earth_layer_cache_for_source
|
||||
from app.services.bgp_detectors import (
|
||||
detect_mass_withdrawal_anomalies,
|
||||
detect_more_specific_burst_anomalies,
|
||||
@@ -223,6 +224,8 @@ async def save_bgp_observations_for_batch(
|
||||
|
||||
if created:
|
||||
await db.commit()
|
||||
for source in {"ris_live_bgp", "bgpstream_bgp"}:
|
||||
invalidate_earth_layer_cache_for_source(source)
|
||||
|
||||
return created
|
||||
|
||||
|
||||
57
backend/app/services/collectors/media_news_archive.py
Normal file
57
backend/app/services/collectors/media_news_archive.py
Normal file
@@ -0,0 +1,57 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from app.services.collectors.base import BaseCollector
|
||||
from app.services.earth_news_store import list_all_earth_news_records
|
||||
|
||||
|
||||
class MediaNewsArchiveCollector(BaseCollector):
|
||||
name = "media_news_archive"
|
||||
priority = "P2"
|
||||
module = "L4"
|
||||
frequency_hours = 12
|
||||
data_type = "news_item"
|
||||
fail_on_empty = False
|
||||
|
||||
async def fetch(self) -> list[dict[str, Any]]:
|
||||
if not self._db_session:
|
||||
return []
|
||||
|
||||
records = await list_all_earth_news_records(self._db_session)
|
||||
items: list[dict[str, Any]] = []
|
||||
for record in records:
|
||||
location_meta = dict(record.location_meta or {})
|
||||
target = location_meta.get("target") if isinstance(location_meta.get("target"), dict) else {}
|
||||
country = target.get("country")
|
||||
city = target.get("city")
|
||||
items.append(
|
||||
{
|
||||
"id": record.id,
|
||||
"source_id": record.id,
|
||||
"name": record.title,
|
||||
"title": record.title,
|
||||
"description": record.summary,
|
||||
"country": country,
|
||||
"city": city,
|
||||
"latitude": record.latitude,
|
||||
"longitude": record.longitude,
|
||||
"reference_date": record.published_at,
|
||||
"metadata": {
|
||||
"url": record.url,
|
||||
"source": record.source,
|
||||
"feed_name": record.feed_name,
|
||||
"region": record.region,
|
||||
"homepage_url": record.homepage_url,
|
||||
"published_at": record.published_at.isoformat() if record.published_at else None,
|
||||
"location_label": record.location_label,
|
||||
"location_source": record.location_source,
|
||||
"verified": record.verified,
|
||||
"location_meta": location_meta,
|
||||
"first_seen_at": record.first_seen_at.isoformat() if record.first_seen_at else None,
|
||||
"last_seen_at": record.last_seen_at.isoformat() if record.last_seen_at else None,
|
||||
"resolved_at": record.resolved_at.isoformat() if record.resolved_at else None,
|
||||
},
|
||||
}
|
||||
)
|
||||
return items
|
||||
@@ -3,10 +3,13 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from copy import deepcopy
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm.attributes import flag_modified
|
||||
|
||||
from app.ai_tasks.prompts import get_effective_prompt
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.schemas.ai import SituationalAnalysisRequest
|
||||
from app.services.ai_client import AIProviderClient
|
||||
@@ -15,6 +18,7 @@ from app.services.ai_tools.web_search import WebSearchClient, WebSearchError
|
||||
|
||||
|
||||
CREDENTIAL_GUIDES_CATEGORY = "collector_credential_guides"
|
||||
CREDENTIAL_GUIDE_PROMPT_KEY = "credential.guide"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
@@ -133,33 +137,57 @@ DEFAULT_CREDENTIAL_GUIDES = {
|
||||
}
|
||||
|
||||
|
||||
def _normalize_provider(provider: str) -> str:
|
||||
return provider.strip().lower().replace(" ", "_")
|
||||
|
||||
|
||||
def _credential_guide_default(provider: str) -> CredentialGuideDefault:
|
||||
normalized = _normalize_provider(provider)
|
||||
known = DEFAULT_CREDENTIAL_GUIDES.get(normalized)
|
||||
if known is not None:
|
||||
return known
|
||||
title = f"{normalized or 'collector'} 凭证配置教程"
|
||||
return CredentialGuideDefault(
|
||||
provider=normalized,
|
||||
title=title,
|
||||
prompt=(
|
||||
f"请生成一份中文教程,指导开发者为 Planet 采集器配置 {normalized} 凭证。"
|
||||
"教程要面向已经有本地开发环境的人,包含官方入口或文档查找方式、"
|
||||
"获取 API Key / Token / Client credentials 的通用步骤、在 Planet 采集器配置中"
|
||||
"填写凭证字段、连接测试、保存、常见失败排查。不要编造具体页面按钮文案;"
|
||||
"如果公开资料不足,必须明确提醒以 provider 官方文档和当前控制台页面为准。"
|
||||
),
|
||||
markdown="",
|
||||
)
|
||||
|
||||
|
||||
async def _get_guide_store(db) -> tuple[SystemSetting | None, dict[str, Any]]:
|
||||
result = await db.execute(
|
||||
select(SystemSetting).where(SystemSetting.category == CREDENTIAL_GUIDES_CATEGORY)
|
||||
)
|
||||
record = result.scalar_one_or_none()
|
||||
payload = dict(record.payload or {}) if record and isinstance(record.payload, dict) else {}
|
||||
payload = deepcopy(record.payload) if record and isinstance(record.payload, dict) else {}
|
||||
return record, payload
|
||||
|
||||
|
||||
async def get_credential_guide(db, provider: str) -> dict[str, Any]:
|
||||
default = DEFAULT_CREDENTIAL_GUIDES.get(provider)
|
||||
if default is None:
|
||||
raise ValueError(f"Unsupported credential guide provider: {provider}")
|
||||
provider = _normalize_provider(provider)
|
||||
default = _credential_guide_default(provider)
|
||||
|
||||
_record, store = await _get_guide_store(db)
|
||||
custom = store.get(provider) if isinstance(store.get(provider), dict) else None
|
||||
has_default_markdown = bool(default.markdown.strip())
|
||||
return {
|
||||
"provider": provider,
|
||||
"title": custom.get("title") if custom else default.title,
|
||||
"markdown": custom.get("markdown") if custom else default.markdown,
|
||||
"prompt": default.prompt,
|
||||
"source": "ai" if custom else "default",
|
||||
"source": "ai" if custom else "default" if has_default_markdown else "missing",
|
||||
"sources": custom.get("sources", []) if custom else [],
|
||||
"verification_status": (
|
||||
custom.get("verification_status", "verified_with_search_evidence")
|
||||
if custom
|
||||
else "default_unverified"
|
||||
else "default_unverified" if has_default_markdown else "missing"
|
||||
),
|
||||
"verification_error": custom.get("verification_error") if custom else None,
|
||||
}
|
||||
@@ -175,9 +203,8 @@ async def save_credential_guide(
|
||||
verification_status: str = "verified_with_search_evidence",
|
||||
verification_error: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
default = DEFAULT_CREDENTIAL_GUIDES.get(provider)
|
||||
if default is None:
|
||||
raise ValueError(f"Unsupported credential guide provider: {provider}")
|
||||
provider = _normalize_provider(provider)
|
||||
default = _credential_guide_default(provider)
|
||||
|
||||
record, store = await _get_guide_store(db)
|
||||
store[provider] = {
|
||||
@@ -190,21 +217,21 @@ async def save_credential_guide(
|
||||
if record is None:
|
||||
db.add(SystemSetting(category=CREDENTIAL_GUIDES_CATEGORY, payload=store))
|
||||
else:
|
||||
record.payload = store
|
||||
record.payload = deepcopy(store)
|
||||
flag_modified(record, "payload")
|
||||
await db.commit()
|
||||
return await get_credential_guide(db, provider)
|
||||
|
||||
|
||||
async def reset_credential_guide(db, provider: str) -> dict[str, Any]:
|
||||
default = DEFAULT_CREDENTIAL_GUIDES.get(provider)
|
||||
if default is None:
|
||||
raise ValueError(f"Unsupported credential guide provider: {provider}")
|
||||
provider = _normalize_provider(provider)
|
||||
|
||||
record, store = await _get_guide_store(db)
|
||||
if provider in store:
|
||||
store.pop(provider, None)
|
||||
if record is not None:
|
||||
record.payload = store
|
||||
record.payload = deepcopy(store)
|
||||
flag_modified(record, "payload")
|
||||
await db.commit()
|
||||
return await get_credential_guide(db, provider)
|
||||
|
||||
@@ -215,9 +242,8 @@ async def generate_credential_guide(
|
||||
ai_client: AIProviderClient,
|
||||
web_search_client: WebSearchClient | None = None,
|
||||
) -> dict[str, Any]:
|
||||
default = DEFAULT_CREDENTIAL_GUIDES.get(provider)
|
||||
if default is None:
|
||||
raise ValueError(f"Unsupported credential guide provider: {provider}")
|
||||
provider = _normalize_provider(provider)
|
||||
default = _credential_guide_default(provider)
|
||||
|
||||
search_evidence: list[dict[str, Any]] = []
|
||||
search_error: str | None = None
|
||||
@@ -240,14 +266,12 @@ async def generate_credential_guide(
|
||||
guide["sources"] = []
|
||||
return guide
|
||||
|
||||
prompt = await get_effective_prompt(db, CREDENTIAL_GUIDE_PROMPT_KEY)
|
||||
response = await ai_client.analyze(
|
||||
SituationalAnalysisRequest(
|
||||
title=f"Generate credential guide for {provider}",
|
||||
objective=(
|
||||
default.prompt
|
||||
+ "\n只能根据 context.search_evidence 中的来源生成教程;"
|
||||
+ "如果证据不足,明确说明需要以官方页面为准。"
|
||||
),
|
||||
objective=f"{default.prompt}\n{prompt.prompt}",
|
||||
system_prompt=prompt.system_prompt or None,
|
||||
context={
|
||||
"provider": provider,
|
||||
"current_default_guide": default.markdown,
|
||||
|
||||
@@ -18,6 +18,7 @@ from app.models.datasource import DataSource
|
||||
from app.models.datasource_config import DataSourceConfig
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.services.barentswatch import (
|
||||
BarentsWatchConfig,
|
||||
_read_zshrc_env,
|
||||
fetch_barentswatch_access_token,
|
||||
resolve_barentswatch_config,
|
||||
@@ -44,6 +45,20 @@ def _resolve_spacetrack_credentials() -> tuple[str, str, str]:
|
||||
return username, password, source or "missing"
|
||||
|
||||
|
||||
def _resolve_spacetrack_credentials_with_override(
|
||||
credential_override: dict[str, str] | None = None,
|
||||
) -> tuple[str, str, str]:
|
||||
if credential_override and (
|
||||
credential_override.get("username") or credential_override.get("password")
|
||||
):
|
||||
return (
|
||||
str(credential_override.get("username") or ""),
|
||||
str(credential_override.get("password") or ""),
|
||||
"draft",
|
||||
)
|
||||
return _resolve_spacetrack_credentials()
|
||||
|
||||
|
||||
async def _resolve_aisstream_api_key(
|
||||
db=None,
|
||||
credential_override: dict[str, str] | None = None,
|
||||
@@ -126,7 +141,9 @@ async def build_builtin_connectivity_checksum(
|
||||
}
|
||||
)
|
||||
elif credential_provider == "spacetrack":
|
||||
username, password, credential_source = _resolve_spacetrack_credentials()
|
||||
username, password, credential_source = _resolve_spacetrack_credentials_with_override(
|
||||
credential_override
|
||||
)
|
||||
has_credentials = bool(username and password)
|
||||
credential_fingerprint = _sha256_json(
|
||||
{
|
||||
@@ -230,7 +247,16 @@ async def test_builtin_connectivity(
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=timeout, follow_redirects=True) as client:
|
||||
if credential_context["credential_provider"] == "barentswatch":
|
||||
barentswatch_config = await resolve_barentswatch_config(db)
|
||||
if credential_override:
|
||||
barentswatch_config = BarentsWatchConfig(
|
||||
endpoint=str(request_endpoint or ""),
|
||||
client_id=str(credential_override.get("client_id") or ""),
|
||||
client_secret=str(credential_override.get("client_secret") or ""),
|
||||
credential_source="draft",
|
||||
endpoint_source="draft",
|
||||
)
|
||||
else:
|
||||
barentswatch_config = await resolve_barentswatch_config(db)
|
||||
token = await fetch_barentswatch_access_token(client, barentswatch_config)
|
||||
if not token:
|
||||
return {
|
||||
@@ -243,7 +269,9 @@ async def test_builtin_connectivity(
|
||||
}
|
||||
request_headers["Authorization"] = f"Bearer {token}"
|
||||
elif credential_context["credential_provider"] == "spacetrack":
|
||||
username, password, _source = _resolve_spacetrack_credentials()
|
||||
username, password, _source = _resolve_spacetrack_credentials_with_override(
|
||||
credential_override
|
||||
)
|
||||
login_url = "https://www.space-track.org/ajaxauth/login"
|
||||
login_response = await client.post(
|
||||
login_url,
|
||||
|
||||
@@ -8,6 +8,7 @@ import re
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.target_schema_registry import TargetSchema, get_target_schema
|
||||
@@ -254,6 +255,12 @@ def _best_field_match(field_name: str, candidates: list[str]) -> str | None:
|
||||
"lat": ("lat", "latitude", "y"),
|
||||
"lon": ("lon", "lng", "longitude", "x"),
|
||||
"mmsi": ("mmsi",),
|
||||
"geometry": ("geometry", "geom"),
|
||||
"properties": ("properties", "props"),
|
||||
"source_kind": ("source_kind", "kind", "type"),
|
||||
"feature_count": ("feature_count", "features_count", "count"),
|
||||
"artifact_path": ("artifact_path", "path", "file"),
|
||||
"sha256": ("sha256", "hash", "checksum"),
|
||||
"sog": ("sog", "speed", "speedOverGround"),
|
||||
"cog": ("cog", "course", "courseOverGround"),
|
||||
"received_at": ("received_at", "timestamp", "time", "updated_at"),
|
||||
|
||||
671
backend/app/services/earth_boundaries.py
Normal file
671
backend/app/services/earth_boundaries.py
Normal file
@@ -0,0 +1,671 @@
|
||||
"""Earth boundary static asset service."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import hashlib
|
||||
import json
|
||||
import shutil
|
||||
import sys
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from uuid import uuid4
|
||||
|
||||
import httpx
|
||||
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[3]
|
||||
SOURCE_OUTPUT_DIR = REPO_ROOT / "data/earth-boundary-sources"
|
||||
SOURCE_MANIFEST_PATH = SOURCE_OUTPUT_DIR / "manifest.json"
|
||||
BUILD_RESULT_PATH = SOURCE_OUTPUT_DIR / "build-result.json"
|
||||
BUILD_JOB_PATH = SOURCE_OUTPUT_DIR / "build-job.json"
|
||||
BOUNDARY_OUTPUT_DIR = REPO_ROOT / "frontend/public/earth/data/boundaries/v1"
|
||||
BOUNDARY_MANIFEST_PATH = BOUNDARY_OUTPUT_DIR / "manifest.json"
|
||||
PMTILES_ARTIFACT_PATH = (
|
||||
REPO_ROOT / "frontend/public/earth/data/boundaries/earth-boundaries-china-pov-v1.pmtiles"
|
||||
)
|
||||
LEGACY_GEOJSON_PATH = REPO_ROOT / "frontend/public/earth/data/countries-admin0.min.geojson"
|
||||
POV_POLICY_PATH = REPO_ROOT / "config/earth-boundary-pov-policy.china-v1.json"
|
||||
LOCAL_CONFIG_PATH = REPO_ROOT / "config/earth-boundary-sources.local.json"
|
||||
EXAMPLE_CONFIG_PATH = REPO_ROOT / "config/earth-boundary-sources.example.json"
|
||||
|
||||
BOUNDARY_SOURCE_KINDS = {
|
||||
"earth_admin0_boundaries": "admin0-boundaries",
|
||||
"earth_coastline": "coastline",
|
||||
"earth_claim_lines": "claim-lines",
|
||||
}
|
||||
|
||||
DEFAULT_PUBLIC_BOUNDARY_SOURCES = {
|
||||
"earth_admin0_boundaries": {
|
||||
"displayName": "Natural Earth Admin-0 Countries",
|
||||
"sourceKind": "admin0-boundaries",
|
||||
"endpoint": "https://raw.githubusercontent.com/nvkelso/natural-earth-vector/master/geojson/ne_10m_admin_0_countries.geojson",
|
||||
"method": "GET",
|
||||
"headers": {},
|
||||
"auth_type": "none",
|
||||
"license": "Natural Earth public domain",
|
||||
},
|
||||
"earth_coastline": {
|
||||
"displayName": "Natural Earth Coastline",
|
||||
"sourceKind": "coastline",
|
||||
"endpoint": "https://raw.githubusercontent.com/nvkelso/natural-earth-vector/master/geojson/ne_10m_coastline.geojson",
|
||||
"method": "GET",
|
||||
"headers": {},
|
||||
"auth_type": "none",
|
||||
"license": "Natural Earth public domain",
|
||||
},
|
||||
"earth_claim_lines": {
|
||||
"displayName": "Natural Earth Disputed Boundaries",
|
||||
"sourceKind": "claim-lines",
|
||||
"endpoint": "https://raw.githubusercontent.com/nvkelso/natural-earth-vector/master/geojson/ne_10m_admin_0_boundary_lines_disputed_areas.geojson",
|
||||
"method": "GET",
|
||||
"headers": {},
|
||||
"auth_type": "none",
|
||||
"license": "Natural Earth public domain",
|
||||
},
|
||||
}
|
||||
|
||||
BUILD_CONFIG = {
|
||||
"builder": "scripts/build_earth_boundary_pmtiles.py",
|
||||
"format": "pmtiles+mvt",
|
||||
"production_target": "pmtiles-mvt",
|
||||
}
|
||||
|
||||
|
||||
class EarthBoundaryBuildError(RuntimeError):
|
||||
def __init__(self, message: str, *, code: str = "build_failed", details: Any = None) -> None:
|
||||
super().__init__(message)
|
||||
self.code = code
|
||||
self.details = details
|
||||
|
||||
|
||||
_build_job_lock = asyncio.Lock()
|
||||
_build_task: asyncio.Task | None = None
|
||||
_build_job_state: dict[str, Any] = {}
|
||||
|
||||
|
||||
def _utc_now_iso() -> str:
|
||||
return datetime.now(UTC).isoformat()
|
||||
|
||||
|
||||
def _public_job_state() -> dict[str, Any]:
|
||||
if _build_job_state:
|
||||
return dict(_build_job_state)
|
||||
return _read_json(BUILD_JOB_PATH)
|
||||
|
||||
|
||||
def get_boundary_build_status() -> dict[str, Any]:
|
||||
return {"job": _public_job_state()}
|
||||
|
||||
|
||||
def _set_job_state(**updates: Any) -> dict[str, Any]:
|
||||
global _build_job_state
|
||||
current = dict(_build_job_state)
|
||||
current.update(updates)
|
||||
current["updated_at"] = _utc_now_iso()
|
||||
_build_job_state = current
|
||||
_write_json(BUILD_JOB_PATH, current)
|
||||
return current
|
||||
|
||||
|
||||
def _append_job_log(message: str) -> None:
|
||||
logs = list(_build_job_state.get("logs") or [])
|
||||
logs.append({"time": _utc_now_iso(), "message": message})
|
||||
_set_job_state(logs=logs[-40:])
|
||||
|
||||
|
||||
def _update_job_progress(progress: float, phase: str, message: str, **extra: Any) -> None:
|
||||
bounded_progress = max(0, min(100, int(round(progress))))
|
||||
_set_job_state(
|
||||
status="running",
|
||||
progress=bounded_progress,
|
||||
phase=phase,
|
||||
message=message,
|
||||
**extra,
|
||||
)
|
||||
|
||||
|
||||
def _read_json(path: Path) -> dict[str, Any]:
|
||||
if not path.exists():
|
||||
return {}
|
||||
with path.open("r", encoding="utf-8") as f:
|
||||
payload = json.load(f)
|
||||
return payload if isinstance(payload, dict) else {}
|
||||
|
||||
|
||||
def _write_json(path: Path, payload: dict[str, Any]) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with path.open("w", encoding="utf-8") as f:
|
||||
json.dump(payload, f, ensure_ascii=False, indent=2)
|
||||
f.write("\n")
|
||||
|
||||
|
||||
def _sha256_bytes(payload: bytes) -> str:
|
||||
return hashlib.sha256(payload).hexdigest()
|
||||
|
||||
|
||||
def _stable_json_hash(payload: Any) -> str:
|
||||
encoded = json.dumps(payload, ensure_ascii=False, sort_keys=True, separators=(",", ":"))
|
||||
return hashlib.sha256(encoded.encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
def _artifact_extension(endpoint: str, content_type: str, payload: bytes) -> str:
|
||||
suffix = Path(endpoint.split("?", 1)[0]).suffix.lower()
|
||||
if suffix in {".json", ".geojson", ".zip", ".pbf"}:
|
||||
return suffix
|
||||
if "geo+json" in content_type or b'"FeatureCollection"' in payload[:4096]:
|
||||
return ".geojson"
|
||||
if "json" in content_type:
|
||||
return ".json"
|
||||
return ".dat"
|
||||
|
||||
|
||||
def _json_feature_count(payload: Any) -> int:
|
||||
if isinstance(payload, dict) and isinstance(payload.get("features"), list):
|
||||
return len(payload["features"])
|
||||
if isinstance(payload, list):
|
||||
return len(payload)
|
||||
return 1 if payload else 0
|
||||
|
||||
|
||||
def _directory_stats(path: Path) -> dict[str, int]:
|
||||
if not path.exists():
|
||||
return {"file_count": 0, "size_bytes": 0}
|
||||
files = [item for item in path.rglob("*") if item.is_file()]
|
||||
return {"file_count": len(files), "size_bytes": sum(item.stat().st_size for item in files)}
|
||||
|
||||
|
||||
def _load_source_feature_collection(source: dict[str, Any]) -> dict[str, Any]:
|
||||
path = REPO_ROOT / source["path"]
|
||||
payload = _read_json(path)
|
||||
features = payload.get("features") if isinstance(payload, dict) else None
|
||||
return {
|
||||
"type": "FeatureCollection",
|
||||
"features": features if isinstance(features, list) else [],
|
||||
}
|
||||
|
||||
|
||||
def _write_high_precision_geojson_manifest(
|
||||
sources: list[dict[str, Any]],
|
||||
build_input_hash: str,
|
||||
missing_tools: list[str],
|
||||
) -> dict[str, Any]:
|
||||
BOUNDARY_OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
|
||||
admin0 = next(source for source in sources if source["kind"] == "admin0-boundaries")
|
||||
coastline = next(source for source in sources if source["kind"] == "coastline")
|
||||
claim_lines = next(source for source in sources if source["kind"] == "claim-lines")
|
||||
|
||||
admin0_payload = _load_source_feature_collection(admin0)
|
||||
coastline_payload = _load_source_feature_collection(coastline)
|
||||
claim_payload = _load_source_feature_collection(claim_lines)
|
||||
for feature in coastline_payload["features"]:
|
||||
props = feature.setdefault("properties", {})
|
||||
if isinstance(props, dict):
|
||||
props["PLANET_LAYER"] = "coastline"
|
||||
|
||||
base_payload = {
|
||||
"type": "FeatureCollection",
|
||||
"features": [*admin0_payload["features"], *coastline_payload["features"]],
|
||||
}
|
||||
base_path = BOUNDARY_OUTPUT_DIR / "earth-boundaries-high-precision-base.geojson"
|
||||
hover_path = BOUNDARY_OUTPUT_DIR / "earth-boundaries-high-precision-hover.geojson"
|
||||
claim_path = BOUNDARY_OUTPUT_DIR / "earth-boundaries-high-precision-claims.geojson"
|
||||
_write_json(base_path, base_payload)
|
||||
_write_json(hover_path, admin0_payload)
|
||||
_write_json(claim_path, claim_payload)
|
||||
|
||||
manifest = {
|
||||
"version": "natural-earth-v1",
|
||||
"builtAt": _utc_now_iso(),
|
||||
"tileProvider": "geojson-high-precision",
|
||||
"format": "geojson-directory",
|
||||
"buildInputHash": build_input_hash,
|
||||
"base": base_path.name,
|
||||
"hoverIndex": hover_path.name,
|
||||
"claimLine": claim_path.name,
|
||||
"sourceFeatureCount": {
|
||||
"admin0": len(admin0_payload["features"]),
|
||||
"coastline": len(coastline_payload["features"]),
|
||||
"claimLines": len(claim_payload["features"]),
|
||||
},
|
||||
"pmtiles": None,
|
||||
"missingTools": missing_tools,
|
||||
}
|
||||
_write_json(BOUNDARY_MANIFEST_PATH, manifest)
|
||||
return manifest
|
||||
|
||||
|
||||
def _relative(path: Path) -> str:
|
||||
return str(path.relative_to(REPO_ROOT))
|
||||
|
||||
|
||||
def load_boundary_config() -> tuple[dict[str, Any], str]:
|
||||
if LOCAL_CONFIG_PATH.exists():
|
||||
return _read_json(LOCAL_CONFIG_PATH), "local"
|
||||
return _read_json(EXAMPLE_CONFIG_PATH), "example"
|
||||
|
||||
|
||||
def save_boundary_config(payload: dict[str, Any]) -> dict[str, Any]:
|
||||
if not isinstance(payload, dict):
|
||||
raise EarthBoundaryBuildError("Earth boundary config must be a JSON object", code="invalid_config")
|
||||
_write_json(LOCAL_CONFIG_PATH, payload)
|
||||
return get_boundary_status()
|
||||
|
||||
|
||||
def _source_configs(payload: dict[str, Any]) -> dict[str, Any]:
|
||||
raw_sources = payload.get("collectorConfigs") or payload.get("sources") or {}
|
||||
return raw_sources if isinstance(raw_sources, dict) else {}
|
||||
|
||||
|
||||
def _is_placeholder_endpoint(endpoint: Any) -> bool:
|
||||
value = str(endpoint or "").strip()
|
||||
return not value or "example.com" in value
|
||||
|
||||
|
||||
def _source_configs_with_defaults(payload: dict[str, Any]) -> dict[str, Any]:
|
||||
raw_sources = _source_configs(payload)
|
||||
merged: dict[str, Any] = {}
|
||||
for source_key, default_config in DEFAULT_PUBLIC_BOUNDARY_SOURCES.items():
|
||||
configured = raw_sources.get(source_key)
|
||||
if not isinstance(configured, dict) or _is_placeholder_endpoint(configured.get("endpoint")):
|
||||
merged[source_key] = dict(default_config)
|
||||
else:
|
||||
merged[source_key] = {**default_config, **configured}
|
||||
for source_key, source_config in raw_sources.items():
|
||||
if source_key not in merged:
|
||||
merged[source_key] = source_config
|
||||
return merged
|
||||
|
||||
|
||||
def _build_input_hash(source_manifest: dict[str, Any]) -> str:
|
||||
return _stable_json_hash(
|
||||
{
|
||||
"source_manifest_schema": source_manifest.get("schema"),
|
||||
"sources": [
|
||||
{
|
||||
"id": source.get("id"),
|
||||
"sha256": source.get("sha256"),
|
||||
"kind": source.get("kind"),
|
||||
}
|
||||
for source in source_manifest.get("sources", [])
|
||||
],
|
||||
"pov_policy": source_manifest.get("povPolicy"),
|
||||
"build_config": BUILD_CONFIG,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def _has_current_artifacts(boundary_manifest: dict[str, Any], build_input_hash: str) -> bool:
|
||||
return (
|
||||
bool(boundary_manifest)
|
||||
and boundary_manifest.get("buildInputHash") == build_input_hash
|
||||
and boundary_manifest.get("tileProvider") == "pmtiles-mvt"
|
||||
and PMTILES_ARTIFACT_PATH.exists()
|
||||
)
|
||||
|
||||
|
||||
def get_boundary_status() -> dict[str, Any]:
|
||||
config_payload, config_source = load_boundary_config()
|
||||
effective_source_configs = _source_configs_with_defaults(config_payload)
|
||||
boundary_manifest = _read_json(BOUNDARY_MANIFEST_PATH)
|
||||
pmtiles_exists = PMTILES_ARTIFACT_PATH.exists()
|
||||
manifest_exists = BOUNDARY_MANIFEST_PATH.exists()
|
||||
high_precision_ready = (
|
||||
manifest_exists
|
||||
and (
|
||||
(
|
||||
boundary_manifest.get("tileProvider") == "pmtiles-mvt"
|
||||
and pmtiles_exists
|
||||
)
|
||||
or boundary_manifest.get("tileProvider") == "geojson-high-precision"
|
||||
)
|
||||
)
|
||||
legacy_exists = LEGACY_GEOJSON_PATH.exists()
|
||||
provider = (
|
||||
boundary_manifest.get("tileProvider")
|
||||
if high_precision_ready
|
||||
else "legacy-geojson" if legacy_exists else "missing"
|
||||
)
|
||||
return {
|
||||
"provider": provider,
|
||||
"high_precision_ready": high_precision_ready,
|
||||
"fallback_available": legacy_exists,
|
||||
"config_source": config_source,
|
||||
"config_path": _relative(LOCAL_CONFIG_PATH),
|
||||
"config_exists": LOCAL_CONFIG_PATH.exists(),
|
||||
"config": config_payload,
|
||||
"effective_default_sources": [
|
||||
source_key
|
||||
for source_key, source_config in effective_source_configs.items()
|
||||
if source_key in DEFAULT_PUBLIC_BOUNDARY_SOURCES
|
||||
and source_config.get("endpoint") == DEFAULT_PUBLIC_BOUNDARY_SOURCES[source_key]["endpoint"]
|
||||
],
|
||||
"manifest": {
|
||||
"path": _relative(BOUNDARY_MANIFEST_PATH),
|
||||
"exists": manifest_exists,
|
||||
"tileProvider": boundary_manifest.get("tileProvider"),
|
||||
"buildInputHash": boundary_manifest.get("buildInputHash"),
|
||||
"builtAt": boundary_manifest.get("builtAt"),
|
||||
},
|
||||
"pmtiles": {
|
||||
"path": _relative(PMTILES_ARTIFACT_PATH),
|
||||
"exists": pmtiles_exists,
|
||||
"size_bytes": PMTILES_ARTIFACT_PATH.stat().st_size if pmtiles_exists else 0,
|
||||
},
|
||||
"legacy": {
|
||||
"path": _relative(LEGACY_GEOJSON_PATH),
|
||||
"exists": legacy_exists,
|
||||
"size_bytes": LEGACY_GEOJSON_PATH.stat().st_size if legacy_exists else 0,
|
||||
},
|
||||
"source_manifest": {
|
||||
"path": _relative(SOURCE_MANIFEST_PATH),
|
||||
"exists": SOURCE_MANIFEST_PATH.exists(),
|
||||
},
|
||||
"last_build": _read_json(BUILD_RESULT_PATH),
|
||||
"current_job": _public_job_state(),
|
||||
}
|
||||
|
||||
|
||||
async def _download_source(
|
||||
source_key: str,
|
||||
source_config: dict[str, Any],
|
||||
progress_callback: Any = None,
|
||||
) -> dict[str, Any]:
|
||||
endpoint = str(source_config.get("endpoint") or "").strip()
|
||||
if _is_placeholder_endpoint(endpoint):
|
||||
raise EarthBoundaryBuildError(
|
||||
f"{source_key} endpoint is not configured",
|
||||
code="source_not_configured",
|
||||
details={"source": source_key},
|
||||
)
|
||||
method = str(source_config.get("method") or "GET").upper()
|
||||
if method not in {"GET", "POST"}:
|
||||
raise EarthBoundaryBuildError(
|
||||
f"{source_key} method must be GET or POST",
|
||||
code="invalid_config",
|
||||
details={"source": source_key, "method": method},
|
||||
)
|
||||
|
||||
if endpoint.startswith("file://") or Path(endpoint).expanduser().exists():
|
||||
payload = Path(endpoint.removeprefix("file://")).expanduser().read_bytes()
|
||||
content_type = "application/octet-stream"
|
||||
if progress_callback:
|
||||
progress_callback(1, len(payload), len(payload))
|
||||
else:
|
||||
timeout = float(source_config.get("timeout") or 120)
|
||||
headers = source_config.get("headers") if isinstance(source_config.get("headers"), dict) else {}
|
||||
async with httpx.AsyncClient(timeout=timeout, follow_redirects=True) as client:
|
||||
async with client.stream(method, endpoint, headers=headers) as response:
|
||||
response.raise_for_status()
|
||||
content_type = response.headers.get("content-type", "")
|
||||
total = int(response.headers.get("content-length") or 0)
|
||||
chunks = []
|
||||
downloaded = 0
|
||||
async for chunk in response.aiter_bytes():
|
||||
if not chunk:
|
||||
continue
|
||||
chunks.append(chunk)
|
||||
downloaded += len(chunk)
|
||||
if progress_callback:
|
||||
progress_callback(
|
||||
(downloaded / total) if total else None,
|
||||
downloaded,
|
||||
total,
|
||||
)
|
||||
payload = b"".join(chunks)
|
||||
|
||||
extension = _artifact_extension(endpoint, content_type, payload)
|
||||
parsed: Any = None
|
||||
if extension in {".json", ".geojson"}:
|
||||
parsed = json.loads(payload.decode("utf-8"))
|
||||
feature_count = _json_feature_count(parsed)
|
||||
if feature_count <= 0:
|
||||
raise EarthBoundaryBuildError(
|
||||
f"{source_key} downloaded payload contains no features",
|
||||
code="empty_source",
|
||||
details={"source": source_key},
|
||||
)
|
||||
|
||||
sha256 = _sha256_bytes(payload)
|
||||
source_dir = SOURCE_OUTPUT_DIR / source_key
|
||||
source_dir.mkdir(parents=True, exist_ok=True)
|
||||
artifact_path = source_dir / f"{sha256}{extension}"
|
||||
artifact_path.write_bytes(payload)
|
||||
return {
|
||||
"id": source_key,
|
||||
"kind": source_config.get("sourceKind") or BOUNDARY_SOURCE_KINDS[source_key],
|
||||
"path": _relative(artifact_path),
|
||||
"sha256": sha256,
|
||||
"featureCount": feature_count,
|
||||
"license": source_config.get("license"),
|
||||
}
|
||||
|
||||
|
||||
async def _run_step(args: list[str]) -> dict[str, Any]:
|
||||
process = await asyncio.create_subprocess_exec(
|
||||
sys.executable,
|
||||
*args,
|
||||
cwd=REPO_ROOT,
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
stdout_bytes, stderr_bytes = await process.communicate()
|
||||
stdout = stdout_bytes.decode("utf-8", errors="replace").strip()
|
||||
stderr = stderr_bytes.decode("utf-8", errors="replace").strip()
|
||||
payload: dict[str, Any] = {"stdout": stdout, "stderr": stderr, "returncode": process.returncode}
|
||||
last_line = stdout.splitlines()[-1:] or []
|
||||
if last_line:
|
||||
try:
|
||||
payload["result"] = json.loads(last_line[0])
|
||||
except json.JSONDecodeError:
|
||||
payload["result"] = last_line[0]
|
||||
if process.returncode != 0:
|
||||
raise EarthBoundaryBuildError(
|
||||
stderr or stdout or f"command failed: {' '.join(args)}",
|
||||
code="build_command_failed",
|
||||
details=payload,
|
||||
)
|
||||
return payload
|
||||
|
||||
|
||||
async def build_boundary_assets(progress_callback: Any = None) -> dict[str, Any]:
|
||||
config_payload, config_source = load_boundary_config()
|
||||
|
||||
source_configs = _source_configs_with_defaults(config_payload)
|
||||
missing = [source for source in BOUNDARY_SOURCE_KINDS if source not in source_configs]
|
||||
if missing:
|
||||
raise EarthBoundaryBuildError(
|
||||
f"Missing Earth boundary source configs: {', '.join(missing)}",
|
||||
code="missing_sources",
|
||||
details={"missing": missing},
|
||||
)
|
||||
|
||||
sources = []
|
||||
source_keys = list(BOUNDARY_SOURCE_KINDS)
|
||||
for index, source_key in enumerate(source_keys):
|
||||
source_config = source_configs[source_key]
|
||||
if not isinstance(source_config, dict):
|
||||
raise EarthBoundaryBuildError(
|
||||
f"{source_key} config must be an object",
|
||||
code="invalid_config",
|
||||
details={"source": source_key},
|
||||
)
|
||||
source_start = 8 + index * 18
|
||||
source_end = source_start + 18
|
||||
if progress_callback:
|
||||
progress_callback(source_start, "download", f"正在下载 {source_key}")
|
||||
|
||||
def report_download_progress(ratio: float | None, downloaded: int, total: int) -> None:
|
||||
if not progress_callback:
|
||||
return
|
||||
if ratio is None:
|
||||
progress_callback(source_start + 8, "download", f"{source_key} 已下载 {downloaded} bytes")
|
||||
return
|
||||
progress_callback(
|
||||
source_start + (source_end - source_start) * ratio,
|
||||
"download",
|
||||
f"{source_key} 下载 {int(ratio * 100)}%",
|
||||
downloaded_bytes=downloaded,
|
||||
total_bytes=total,
|
||||
)
|
||||
|
||||
sources.append(await _download_source(source_key, source_config, report_download_progress))
|
||||
|
||||
source_manifest = {
|
||||
"schema": "planet-earth-boundary-sources/v2",
|
||||
"sources": sources,
|
||||
"povPolicy": _read_json(POV_POLICY_PATH),
|
||||
}
|
||||
if progress_callback:
|
||||
progress_callback(65, "manifest", "正在写入边界源 manifest")
|
||||
_write_json(SOURCE_MANIFEST_PATH, source_manifest)
|
||||
build_input_hash = _build_input_hash(source_manifest)
|
||||
|
||||
boundary_manifest = _read_json(BOUNDARY_MANIFEST_PATH)
|
||||
build_skipped = _has_current_artifacts(boundary_manifest, build_input_hash)
|
||||
missing_tools = [tool for tool in ("tippecanoe", "pmtiles") if shutil.which(tool) is None]
|
||||
if missing_tools and not build_skipped:
|
||||
if progress_callback:
|
||||
progress_callback(82, "build", "缺少 PMTiles 工具,正在生成 GeoJSON 高清包")
|
||||
boundary_manifest = _write_high_precision_geojson_manifest(
|
||||
sources,
|
||||
build_input_hash,
|
||||
missing_tools,
|
||||
)
|
||||
result = {
|
||||
"status": "built_geojson_fallback",
|
||||
"code": "missing_tools",
|
||||
"missing_tools": missing_tools,
|
||||
"sources": sources,
|
||||
"boundary_manifest": _relative(BOUNDARY_MANIFEST_PATH),
|
||||
"manifest": boundary_manifest,
|
||||
}
|
||||
_write_json(BUILD_RESULT_PATH, result)
|
||||
if progress_callback:
|
||||
progress_callback(96, "finalize", "GeoJSON 高清国界包已生成")
|
||||
return {**get_boundary_status(), "build": result}
|
||||
|
||||
if build_skipped:
|
||||
if progress_callback:
|
||||
progress_callback(96, "unchanged", "高精国界已是最新")
|
||||
build_result = {
|
||||
"status": "unchanged",
|
||||
"reason": "source manifest and build config hash unchanged",
|
||||
"buildInputHash": build_input_hash,
|
||||
}
|
||||
else:
|
||||
if progress_callback:
|
||||
progress_callback(72, "build", "正在构建 PMTiles/MVT")
|
||||
admin0 = next(source for source in sources if source["kind"] == "admin0-boundaries")
|
||||
coastline = next(source for source in sources if source["kind"] == "coastline")
|
||||
claim_lines = next(source for source in sources if source["kind"] == "claim-lines")
|
||||
build_result = await _run_step(
|
||||
[
|
||||
"scripts/build_earth_boundary_pmtiles.py",
|
||||
"--admin0-source",
|
||||
admin0["path"],
|
||||
"--coastline-source",
|
||||
coastline["path"],
|
||||
"--claims-source",
|
||||
claim_lines["path"],
|
||||
"--output",
|
||||
_relative(PMTILES_ARTIFACT_PATH),
|
||||
"--manifest",
|
||||
_relative(BOUNDARY_MANIFEST_PATH),
|
||||
"--build-input-hash",
|
||||
build_input_hash,
|
||||
"--pov-policy",
|
||||
_relative(POV_POLICY_PATH),
|
||||
]
|
||||
)
|
||||
if progress_callback:
|
||||
progress_callback(95, "finalize", "正在校验构建产物")
|
||||
|
||||
boundary_manifest = _read_json(BOUNDARY_MANIFEST_PATH)
|
||||
boundary_stats = _directory_stats(BOUNDARY_OUTPUT_DIR)
|
||||
result = {
|
||||
"status": "unchanged" if build_skipped else "built",
|
||||
"sources": sources,
|
||||
"source_manifest": _relative(SOURCE_MANIFEST_PATH),
|
||||
"boundary_manifest": _relative(BOUNDARY_MANIFEST_PATH),
|
||||
"pmtiles_artifact": _relative(PMTILES_ARTIFACT_PATH),
|
||||
"pmtiles_exists": PMTILES_ARTIFACT_PATH.exists(),
|
||||
"boundary_stats": boundary_stats,
|
||||
"manifest": boundary_manifest,
|
||||
"build_result": build_result,
|
||||
}
|
||||
_write_json(BUILD_RESULT_PATH, result)
|
||||
return {**get_boundary_status(), "build": result}
|
||||
|
||||
|
||||
async def _run_boundary_build_job(job_id: str) -> None:
|
||||
def report(progress: float, phase: str, message: str, **extra: Any) -> None:
|
||||
if _build_job_state.get("id") != job_id:
|
||||
return
|
||||
_update_job_progress(progress, phase, message, **extra)
|
||||
|
||||
try:
|
||||
report(3, "prepare", "正在准备高精国界构建")
|
||||
result = await build_boundary_assets(report)
|
||||
_set_job_state(
|
||||
id=job_id,
|
||||
status="succeeded",
|
||||
progress=100,
|
||||
phase="complete",
|
||||
message="高精国界构建完成",
|
||||
finished_at=_utc_now_iso(),
|
||||
result={
|
||||
"provider": result.get("provider"),
|
||||
"high_precision_ready": result.get("high_precision_ready"),
|
||||
"pmtiles": result.get("pmtiles"),
|
||||
"manifest": result.get("manifest"),
|
||||
},
|
||||
)
|
||||
_append_job_log("高精国界构建完成")
|
||||
except EarthBoundaryBuildError as exc:
|
||||
_set_job_state(
|
||||
id=job_id,
|
||||
status="failed",
|
||||
progress=_build_job_state.get("progress", 0),
|
||||
phase="failed",
|
||||
message=str(exc),
|
||||
code=exc.code,
|
||||
details=exc.details,
|
||||
finished_at=_utc_now_iso(),
|
||||
)
|
||||
_append_job_log(str(exc))
|
||||
except Exception as exc: # pragma: no cover - defensive guard for background task
|
||||
_set_job_state(
|
||||
id=job_id,
|
||||
status="failed",
|
||||
progress=_build_job_state.get("progress", 0),
|
||||
phase="failed",
|
||||
message=str(exc),
|
||||
code="build_failed",
|
||||
finished_at=_utc_now_iso(),
|
||||
)
|
||||
_append_job_log(str(exc))
|
||||
|
||||
|
||||
async def start_boundary_build_job() -> dict[str, Any]:
|
||||
global _build_task
|
||||
async with _build_job_lock:
|
||||
if _build_task and not _build_task.done():
|
||||
return {"accepted": False, "job": _public_job_state()}
|
||||
job_id = uuid4().hex
|
||||
_set_job_state(
|
||||
id=job_id,
|
||||
status="queued",
|
||||
progress=0,
|
||||
phase="queued",
|
||||
message="高精国界构建已加入队列",
|
||||
logs=[],
|
||||
started_at=_utc_now_iso(),
|
||||
finished_at=None,
|
||||
code=None,
|
||||
details=None,
|
||||
)
|
||||
_append_job_log("高精国界构建已启动")
|
||||
_build_task = asyncio.create_task(_run_boundary_build_job(job_id))
|
||||
return {"accepted": True, "job": _public_job_state()}
|
||||
408
backend/app/services/earth_layer_cache.py
Normal file
408
backend/app/services/earth_layer_cache.py
Normal file
@@ -0,0 +1,408 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from collections.abc import Awaitable, Callable
|
||||
from dataclasses import dataclass
|
||||
import hashlib
|
||||
import json
|
||||
import time
|
||||
from typing import Any
|
||||
|
||||
from fastapi import Response
|
||||
|
||||
from app.core.cache import _RedisClient
|
||||
from app.core.config import settings
|
||||
from app.core.logging import get_logger
|
||||
|
||||
|
||||
logger = get_logger(__name__, service="earth_layer_cache")
|
||||
|
||||
EARTH_LAYER_CACHE_PREFIX = "earth:layer:v1"
|
||||
EARTH_LAYER_LOCK_PREFIX = "earth:layer:lock:v1"
|
||||
DEFAULT_LOCK_TTL_SECONDS = 10
|
||||
DEFAULT_LOCK_WAIT_SECONDS = 0.2
|
||||
DEFAULT_MAX_FEATURES = 5000
|
||||
DEFAULT_MAX_BYTES = 5 * 1024 * 1024
|
||||
DEFAULT_BBOX_PRECISION_DEGREES = 0.1
|
||||
DEV_CACHE_KEY_HEADER = {"development", "dev", "test", "testing", "local"}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class EarthLayerCachePolicy:
|
||||
fresh_ttl_seconds: int
|
||||
stale_ttl_seconds: int
|
||||
max_features: int = DEFAULT_MAX_FEATURES
|
||||
max_bytes: int = DEFAULT_MAX_BYTES
|
||||
lock_ttl_seconds: int = DEFAULT_LOCK_TTL_SECONDS
|
||||
lock_wait_seconds: float = DEFAULT_LOCK_WAIT_SECONDS
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class EarthLayerCacheResult:
|
||||
payload: dict[str, Any]
|
||||
state: str
|
||||
key: str
|
||||
features: int
|
||||
bytes: int
|
||||
|
||||
|
||||
class EarthLayerCache:
|
||||
def __init__(self) -> None:
|
||||
self._client = None
|
||||
|
||||
@property
|
||||
def client(self):
|
||||
if self._client is None:
|
||||
self._client = _RedisClient.get_client()
|
||||
return self._client
|
||||
|
||||
@staticmethod
|
||||
def key(layer: str, **params: Any) -> str:
|
||||
parts = [EARTH_LAYER_CACHE_PREFIX, _safe_key_part(layer)]
|
||||
for name in sorted(params):
|
||||
value = params[name]
|
||||
if value is None:
|
||||
value = "none"
|
||||
parts.append(f"{_safe_key_part(name)}:{_safe_key_part(value)}")
|
||||
return ":".join(parts)
|
||||
|
||||
@staticmethod
|
||||
def stale_key(key: str) -> str:
|
||||
return f"{key}:stale"
|
||||
|
||||
@staticmethod
|
||||
def lock_key(key: str) -> str:
|
||||
digest = hashlib.sha256(key.encode("utf-8")).hexdigest()[:32]
|
||||
return f"{EARTH_LAYER_LOCK_PREFIX}:{digest}"
|
||||
|
||||
def get_json(self, key: str) -> dict[str, Any] | None:
|
||||
raw = self.client.get(key)
|
||||
if not raw:
|
||||
return None
|
||||
value = json.loads(raw)
|
||||
return value if isinstance(value, dict) else None
|
||||
|
||||
def set_json(self, key: str, payload: dict[str, Any], ttl_seconds: int) -> None:
|
||||
self.client.setex(key, ttl_seconds, json.dumps(payload, ensure_ascii=False, default=str))
|
||||
|
||||
def acquire_lock(self, key: str, ttl_seconds: int) -> bool:
|
||||
return bool(self.client.set(self.lock_key(key), "1", nx=True, ex=ttl_seconds))
|
||||
|
||||
def release_lock(self, key: str) -> None:
|
||||
try:
|
||||
self.client.delete(self.lock_key(key))
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
def delete_pattern(self, pattern: str = f"{EARTH_LAYER_CACHE_PREFIX}:*") -> int:
|
||||
keys = list(self.client.scan_iter(match=pattern))
|
||||
if not keys:
|
||||
return 0
|
||||
return int(self.client.delete(*keys))
|
||||
|
||||
def status(self) -> dict[str, Any]:
|
||||
keys = list(self.client.scan_iter(match=f"{EARTH_LAYER_CACHE_PREFIX}:*"))
|
||||
by_layer: dict[str, dict[str, Any]] = {}
|
||||
total_memory = 0
|
||||
for key in keys:
|
||||
key_str = key.decode("utf-8") if isinstance(key, bytes) else str(key)
|
||||
layer = _layer_from_key(key_str)
|
||||
entry = by_layer.setdefault(layer, {"keys": 0, "stale_keys": 0, "memory_bytes": 0})
|
||||
entry["keys"] += 1
|
||||
if key_str.endswith(":stale"):
|
||||
entry["stale_keys"] += 1
|
||||
try:
|
||||
memory = int(self.client.memory_usage(key) or 0)
|
||||
except Exception:
|
||||
memory = 0
|
||||
entry["memory_bytes"] += memory
|
||||
total_memory += memory
|
||||
return {
|
||||
"prefix": EARTH_LAYER_CACHE_PREFIX,
|
||||
"key_count": len(keys),
|
||||
"memory_bytes": total_memory,
|
||||
"layers": by_layer,
|
||||
}
|
||||
|
||||
|
||||
earth_layer_cache = EarthLayerCache()
|
||||
|
||||
|
||||
def quantize_bbox(
|
||||
bbox: tuple[float, float, float, float],
|
||||
*,
|
||||
precision: float = DEFAULT_BBOX_PRECISION_DEGREES,
|
||||
) -> tuple[float, float, float, float]:
|
||||
return tuple(round(value / precision) * precision for value in bbox) # type: ignore[return-value]
|
||||
|
||||
|
||||
def format_bbox_key(bbox: tuple[float, float, float, float]) -> str:
|
||||
return ",".join(f"{value:.1f}" for value in bbox)
|
||||
|
||||
|
||||
def apply_cache_headers(response: Response | None, result: EarthLayerCacheResult) -> None:
|
||||
if response is None:
|
||||
return
|
||||
response.headers["X-Planet-Cache"] = result.state
|
||||
response.headers["X-Planet-Cache-Features"] = str(result.features)
|
||||
response.headers["X-Planet-Cache-Bytes"] = str(result.bytes)
|
||||
env_name = str(getattr(settings, "ENVIRONMENT", "") or "development").lower()
|
||||
if env_name in DEV_CACHE_KEY_HEADER:
|
||||
response.headers["X-Planet-Cache-Key"] = result.key
|
||||
|
||||
|
||||
async def get_or_build_layer_payload(
|
||||
*,
|
||||
key: str,
|
||||
policy: EarthLayerCachePolicy,
|
||||
builder: Callable[[], Awaitable[dict[str, Any]]],
|
||||
response: Response | None = None,
|
||||
) -> dict[str, Any]:
|
||||
result = await resolve_layer_payload(key=key, policy=policy, builder=builder)
|
||||
apply_cache_headers(response, result)
|
||||
return result.payload
|
||||
|
||||
|
||||
async def resolve_layer_payload(
|
||||
*,
|
||||
key: str,
|
||||
policy: EarthLayerCachePolicy,
|
||||
builder: Callable[[], Awaitable[dict[str, Any]]],
|
||||
) -> EarthLayerCacheResult:
|
||||
started = time.perf_counter()
|
||||
try:
|
||||
cached = earth_layer_cache.get_json(key)
|
||||
if cached is not None:
|
||||
return _result(cached, state="hit", key=key)
|
||||
|
||||
lock_acquired = earth_layer_cache.acquire_lock(key, policy.lock_ttl_seconds)
|
||||
if lock_acquired:
|
||||
try:
|
||||
payload = await _build_budgeted_payload(builder, policy)
|
||||
_write_fresh_and_stale(key, payload, policy)
|
||||
_log_cache_event("refresh", key, payload, started)
|
||||
return _result(payload, state="refresh", key=key)
|
||||
except Exception as exc:
|
||||
stale = _read_stale(key)
|
||||
if stale is not None:
|
||||
logger.warning_event(
|
||||
"Earth layer cache builder failed; returning stale payload",
|
||||
event="earth_layer_cache.stale_after_builder_error",
|
||||
context={"key": key, "error": str(exc)},
|
||||
)
|
||||
return _result(stale, state="stale", key=key)
|
||||
raise
|
||||
finally:
|
||||
earth_layer_cache.release_lock(key)
|
||||
|
||||
stale = _read_stale(key)
|
||||
if stale is not None:
|
||||
return _result(stale, state="stale", key=key)
|
||||
|
||||
await asyncio.sleep(policy.lock_wait_seconds)
|
||||
cached_after_wait = earth_layer_cache.get_json(key)
|
||||
if cached_after_wait is not None:
|
||||
return _result(cached_after_wait, state="hit", key=key)
|
||||
|
||||
payload = await _build_budgeted_payload(builder, policy)
|
||||
_log_cache_event("miss", key, payload, started)
|
||||
return _result(payload, state="miss", key=key)
|
||||
except Exception as exc:
|
||||
try:
|
||||
payload = await _build_budgeted_payload(builder, policy)
|
||||
except Exception:
|
||||
raise exc
|
||||
logger.warning_event(
|
||||
"Earth layer cache bypassed",
|
||||
event="earth_layer_cache.bypass",
|
||||
context={"key": key, "error": str(exc)},
|
||||
)
|
||||
return _result(payload, state="bypass", key=key)
|
||||
|
||||
|
||||
def apply_payload_budget(payload: dict[str, Any], policy: EarthLayerCachePolicy) -> dict[str, Any]:
|
||||
budgeted = _truncate_features(payload, policy.max_features, "feature_budget")
|
||||
size = _payload_size(budgeted)
|
||||
if size <= policy.max_bytes:
|
||||
return budgeted
|
||||
|
||||
features = budgeted.get("features")
|
||||
if not isinstance(features, list):
|
||||
return _with_budget_diagnostics(
|
||||
budgeted,
|
||||
truncated=True,
|
||||
reason="byte_budget",
|
||||
bytes_before=size,
|
||||
bytes_after=size,
|
||||
)
|
||||
|
||||
low = 0
|
||||
high = len(features)
|
||||
best = []
|
||||
best_size = _payload_size({**budgeted, "features": best})
|
||||
while low <= high:
|
||||
mid = (low + high) // 2
|
||||
candidate_features = features[:mid]
|
||||
candidate = _with_budget_diagnostics(
|
||||
{**budgeted, "features": candidate_features},
|
||||
truncated=mid < len(features),
|
||||
reason="byte_budget",
|
||||
bytes_before=size,
|
||||
bytes_after=0,
|
||||
)
|
||||
candidate_size = _payload_size(candidate)
|
||||
if candidate_size <= policy.max_bytes:
|
||||
best = candidate_features
|
||||
best_size = candidate_size
|
||||
low = mid + 1
|
||||
else:
|
||||
high = mid - 1
|
||||
|
||||
return _with_budget_diagnostics(
|
||||
{**budgeted, "features": best},
|
||||
truncated=True,
|
||||
reason="byte_budget",
|
||||
bytes_before=size,
|
||||
bytes_after=best_size,
|
||||
)
|
||||
|
||||
|
||||
def invalidate_earth_layer_cache_for_source(source: str) -> int:
|
||||
source_key = str(source or "").strip()
|
||||
patterns = {
|
||||
"barentswatch_vessels": ["vessels*", "summary*"],
|
||||
"aisstream_vessels": ["vessels*", "summary*"],
|
||||
"telegeography_cables": ["cables*", "landing-points*", "summary*"],
|
||||
"telegeography_landing": ["landing-points*", "summary*"],
|
||||
"telegeography_landing_points": ["landing-points*", "summary*"],
|
||||
"telegeography_systems": ["cables*", "summary*"],
|
||||
"telegeography_cable_systems": ["cables*", "summary*"],
|
||||
"arcgis_cables": ["cables*", "landing-points*", "summary*"],
|
||||
"arcgis_landing_points": ["landing-points*", "summary*"],
|
||||
"arcgis_cable_landing_relation": ["landing-points*", "summary*"],
|
||||
"arcgis_cable_landing_relations": ["landing-points*", "summary*"],
|
||||
"fao_landing_points": ["landing-points*", "summary*"],
|
||||
"celestrak_tle": ["satellites*", "summary*"],
|
||||
"spacetrack_tle": ["satellites*", "summary*"],
|
||||
"top500": ["compute-centers*", "summary*"],
|
||||
"top500_supercomputers": ["compute-centers*", "summary*"],
|
||||
"epoch_ai_gpu": ["compute-centers*", "summary*"],
|
||||
"huggingface_models": ["compute-centers*", "summary*"],
|
||||
"huggingface_datasets": ["compute-centers*", "summary*"],
|
||||
"huggingface_spaces": ["compute-centers*", "summary*"],
|
||||
"ris_live_bgp": ["bgp*", "summary*"],
|
||||
"bgpstream_bgp": ["bgp*", "summary*"],
|
||||
"iptoasn_prefix_geo": ["bgp*", "summary*"],
|
||||
"opengeofeed_prefix_geo": ["bgp*", "summary*"],
|
||||
"nro_delegated_prefix_geo": ["bgp*", "summary*"],
|
||||
}.get(source_key, [])
|
||||
deleted = 0
|
||||
for layer_pattern in patterns:
|
||||
deleted += earth_layer_cache.delete_pattern(f"{EARTH_LAYER_CACHE_PREFIX}:{layer_pattern}")
|
||||
return deleted
|
||||
|
||||
|
||||
async def _build_budgeted_payload(
|
||||
builder: Callable[[], Awaitable[dict[str, Any]]],
|
||||
policy: EarthLayerCachePolicy,
|
||||
) -> dict[str, Any]:
|
||||
payload = await builder()
|
||||
return apply_payload_budget(payload, policy)
|
||||
|
||||
|
||||
def _write_fresh_and_stale(key: str, payload: dict[str, Any], policy: EarthLayerCachePolicy) -> None:
|
||||
earth_layer_cache.set_json(key, payload, policy.fresh_ttl_seconds)
|
||||
earth_layer_cache.set_json(earth_layer_cache.stale_key(key), payload, policy.stale_ttl_seconds)
|
||||
|
||||
|
||||
def _read_stale(key: str) -> dict[str, Any] | None:
|
||||
try:
|
||||
return earth_layer_cache.get_json(earth_layer_cache.stale_key(key))
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def _truncate_features(payload: dict[str, Any], max_features: int, reason: str) -> dict[str, Any]:
|
||||
features = payload.get("features")
|
||||
if not isinstance(features, list) or len(features) <= max_features:
|
||||
return payload
|
||||
return _with_budget_diagnostics(
|
||||
{**payload, "features": features[:max_features]},
|
||||
truncated=True,
|
||||
reason=reason,
|
||||
original_feature_count=len(features),
|
||||
)
|
||||
|
||||
|
||||
def _with_budget_diagnostics(
|
||||
payload: dict[str, Any],
|
||||
*,
|
||||
truncated: bool,
|
||||
reason: str,
|
||||
original_feature_count: int | None = None,
|
||||
bytes_before: int | None = None,
|
||||
bytes_after: int | None = None,
|
||||
) -> dict[str, Any]:
|
||||
diagnostics = dict(payload.get("diagnostics") or {})
|
||||
diagnostics.update(
|
||||
{
|
||||
"truncated": bool(truncated or diagnostics.get("truncated")),
|
||||
"limit_reason": reason,
|
||||
}
|
||||
)
|
||||
if original_feature_count is not None:
|
||||
diagnostics["original_feature_count"] = original_feature_count
|
||||
if bytes_before is not None:
|
||||
diagnostics["bytes_before_budget"] = bytes_before
|
||||
if bytes_after is not None:
|
||||
diagnostics["bytes_after_budget"] = bytes_after
|
||||
return {**payload, "diagnostics": diagnostics}
|
||||
|
||||
|
||||
def _result(payload: dict[str, Any], *, state: str, key: str) -> EarthLayerCacheResult:
|
||||
return EarthLayerCacheResult(
|
||||
payload=payload,
|
||||
state=state,
|
||||
key=key,
|
||||
features=_feature_count(payload),
|
||||
bytes=_payload_size(payload),
|
||||
)
|
||||
|
||||
|
||||
def _feature_count(payload: dict[str, Any]) -> int:
|
||||
features = payload.get("features")
|
||||
if isinstance(features, list):
|
||||
return len(features)
|
||||
count = payload.get("count")
|
||||
return int(count) if isinstance(count, int) else 0
|
||||
|
||||
|
||||
def _payload_size(payload: dict[str, Any]) -> int:
|
||||
return len(json.dumps(payload, ensure_ascii=False, default=str).encode("utf-8"))
|
||||
|
||||
|
||||
def _safe_key_part(value: Any) -> str:
|
||||
raw = str(value).strip().lower()
|
||||
return "".join(char if char.isalnum() or char in {"-", "_", ".", ","} else "_" for char in raw)[:160]
|
||||
|
||||
|
||||
def _layer_from_key(key: str) -> str:
|
||||
prefix = f"{EARTH_LAYER_CACHE_PREFIX}:"
|
||||
if not key.startswith(prefix):
|
||||
return "unknown"
|
||||
remainder = key[len(prefix):]
|
||||
return remainder.split(":", 1)[0]
|
||||
|
||||
|
||||
def _log_cache_event(state: str, key: str, payload: dict[str, Any], started: float) -> None:
|
||||
logger.info_event(
|
||||
"Earth layer cache resolved",
|
||||
event="earth_layer_cache.resolved",
|
||||
context={
|
||||
"state": state,
|
||||
"key": key,
|
||||
"features": _feature_count(payload),
|
||||
"bytes": _payload_size(payload),
|
||||
"duration_ms": round((time.perf_counter() - started) * 1000, 2),
|
||||
},
|
||||
)
|
||||
@@ -1,11 +1,13 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from dataclasses import dataclass
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import UTC, datetime
|
||||
from email.utils import parsedate_to_datetime
|
||||
import hashlib
|
||||
import html
|
||||
import json
|
||||
import math
|
||||
import re
|
||||
from typing import Any
|
||||
from urllib.parse import quote
|
||||
@@ -13,6 +15,13 @@ import xml.etree.ElementTree as ET
|
||||
|
||||
import httpx
|
||||
from bs4 import BeautifulSoup
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.countries import COUNTRY_VARIANTS_MAP, get_country_centroid, normalize_country
|
||||
from app.ai_tasks.prompts import EffectiveAIPrompt, get_effective_prompt
|
||||
from app.schemas.ai import SituationalAnalysisRequest
|
||||
from app.services.ai_client import AIProviderClient
|
||||
from app.services.location.resolvers.nominatim import build_default_nominatim_geocoder
|
||||
|
||||
|
||||
USER_AGENT = "PlanetEarthNewsBoard/1.0 (+https://planet.local)"
|
||||
@@ -20,6 +29,11 @@ REQUEST_TIMEOUT = 12.0
|
||||
MAX_ITEMS_PER_SOURCE = 6
|
||||
MAX_ITEMS_TOTAL = 12
|
||||
STALE_CACHE_MAX_AGE_SECONDS = 60 * 45
|
||||
RSS_SUPPLEMENT_MAX_AGE_SECONDS = STALE_CACHE_MAX_AGE_SECONDS
|
||||
MAX_TARGET_INFERENCE_CONCURRENCY = 3
|
||||
TARGET_INFERENCE_TIMEOUT_SECONDS = 6.0
|
||||
DEFAULT_NEWS_LOCALE = "zh-CN"
|
||||
NEWS_ENRICH_PROMPT_KEY = "earth.news.enrich"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
@@ -49,6 +63,17 @@ class NewsFeedSource:
|
||||
priority: int = 100
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class NewsTargetLocation:
|
||||
latitude: float
|
||||
longitude: float
|
||||
label: str
|
||||
source: str
|
||||
confidence: float | None = None
|
||||
country: str | None = None
|
||||
city: str | None = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class ParsedNewsItem:
|
||||
id: str
|
||||
@@ -60,6 +85,18 @@ class ParsedNewsItem:
|
||||
feed_region: str
|
||||
homepage_url: str
|
||||
published_at: datetime | None
|
||||
content_language: str = "en"
|
||||
localizations: dict[str, dict[str, str]] = field(default_factory=dict)
|
||||
enrichment_status: str = "pending"
|
||||
enrichment_error: str | None = None
|
||||
enriched_at: datetime | None = None
|
||||
target_location: NewsTargetLocation | None = None
|
||||
target_resolution_stage: str = "unresolved"
|
||||
target_ai_attempted: bool = False
|
||||
target_ai_status: str = "not_attempted"
|
||||
target_ai_error: str | None = None
|
||||
target_debug_note: str | None = None
|
||||
location_patch: dict[str, Any] | None = None
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -236,6 +273,17 @@ NEWS_FEED_SOURCES: tuple[NewsFeedSource, ...] = (
|
||||
|
||||
|
||||
_REGION_CACHE: dict[str, CachedRegionFeed] = {}
|
||||
_news_target_geocode = build_default_nominatim_geocoder(user_agent=USER_AGENT)
|
||||
_CITY_HINTS: tuple[dict[str, str | None], ...] = (
|
||||
{"name": "Beijing", "country": "中国"},
|
||||
{"name": "Havana", "country": "古巴"},
|
||||
{"name": "Kyiv", "country": "乌克兰"},
|
||||
{"name": "Bangkok", "country": "泰国"},
|
||||
{"name": "Tehran", "country": "伊朗"},
|
||||
{"name": "Moscow", "country": "俄罗斯"},
|
||||
{"name": "Taipei", "country": "中国(台湾)"},
|
||||
{"name": "Hong Kong", "country": "中国(香港)"},
|
||||
)
|
||||
|
||||
|
||||
def determine_focus_region(lat: float | None, lon: float | None) -> str:
|
||||
@@ -258,6 +306,462 @@ def get_region_anchor(region: str) -> RegionAnchor:
|
||||
return REGION_ANCHORS.get(region, REGION_ANCHORS["global"])
|
||||
|
||||
|
||||
def _coerce_str(value: Any) -> str | None:
|
||||
if value is None:
|
||||
return None
|
||||
if not isinstance(value, str):
|
||||
value = str(value)
|
||||
cleaned = re.sub(r"\s+", " ", value).strip()
|
||||
return cleaned or None
|
||||
|
||||
|
||||
def _contains_location_alias(text: str, alias: str) -> bool:
|
||||
normalized_alias = _coerce_str(alias)
|
||||
if not normalized_alias:
|
||||
return False
|
||||
if re.search(r"[A-Za-z]", normalized_alias):
|
||||
pattern = r"(?<![A-Za-z])" + re.escape(normalized_alias) + r"(?![A-Za-z])"
|
||||
return re.search(pattern, text, flags=re.IGNORECASE) is not None
|
||||
return normalized_alias in text
|
||||
|
||||
|
||||
def _iter_searchable_country_variants(
|
||||
canonical: str,
|
||||
variants: list[str],
|
||||
) -> tuple[str, ...]:
|
||||
searchable: list[str] = []
|
||||
seen: set[str] = set()
|
||||
for variant in (canonical, *variants):
|
||||
normalized = _coerce_str(variant)
|
||||
if not normalized:
|
||||
continue
|
||||
if re.fullmatch(r"[A-Z]{2,3}", normalized):
|
||||
continue
|
||||
if len(normalized) <= 2:
|
||||
continue
|
||||
key = normalized.casefold()
|
||||
if key in seen:
|
||||
continue
|
||||
seen.add(key)
|
||||
searchable.append(normalized)
|
||||
return tuple(searchable)
|
||||
|
||||
|
||||
def _coerce_float(value: Any) -> float | None:
|
||||
try:
|
||||
parsed = float(value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
if not math.isfinite(parsed):
|
||||
return None
|
||||
return parsed
|
||||
|
||||
|
||||
def _first_json_object(text: str) -> dict[str, Any] | None:
|
||||
if not text:
|
||||
return None
|
||||
decoder = json.JSONDecoder()
|
||||
for index, char in enumerate(text):
|
||||
if char != "{":
|
||||
continue
|
||||
try:
|
||||
payload, _ = decoder.raw_decode(text[index:])
|
||||
except ValueError:
|
||||
continue
|
||||
if isinstance(payload, dict):
|
||||
return payload
|
||||
return None
|
||||
|
||||
|
||||
def _normalize_localizations(value: Any) -> dict[str, dict[str, str]]:
|
||||
if not isinstance(value, dict):
|
||||
return {}
|
||||
normalized: dict[str, dict[str, str]] = {}
|
||||
for locale, payload in value.items():
|
||||
locale_key = _coerce_str(locale)
|
||||
if not locale_key or not isinstance(payload, dict):
|
||||
continue
|
||||
title = _coerce_str(payload.get("title"))
|
||||
summary = _coerce_str(payload.get("summary"))
|
||||
entry: dict[str, str] = {}
|
||||
if title:
|
||||
entry["title"] = title
|
||||
if summary:
|
||||
entry["summary"] = summary
|
||||
if entry:
|
||||
normalized[locale_key] = entry
|
||||
return normalized
|
||||
|
||||
|
||||
def _get_locale_text(
|
||||
item: ParsedNewsItem,
|
||||
key: str,
|
||||
*,
|
||||
locale: str = DEFAULT_NEWS_LOCALE,
|
||||
) -> str:
|
||||
localized = item.localizations.get(locale)
|
||||
if isinstance(localized, dict):
|
||||
value = _coerce_str(localized.get(key))
|
||||
if value:
|
||||
return value
|
||||
return ""
|
||||
|
||||
|
||||
def _has_default_localization(item: ParsedNewsItem) -> bool:
|
||||
localized = item.localizations.get(DEFAULT_NEWS_LOCALE)
|
||||
if not isinstance(localized, dict):
|
||||
return False
|
||||
return bool(_coerce_str(localized.get("title")) and _coerce_str(localized.get("summary")))
|
||||
|
||||
|
||||
def apply_enrichment_patch_to_item(
|
||||
item: ParsedNewsItem,
|
||||
patch: dict[str, Any],
|
||||
) -> ParsedNewsItem:
|
||||
item.location_patch = patch
|
||||
if "content_language" in patch:
|
||||
item.content_language = _coerce_str(patch.get("content_language")) or item.content_language
|
||||
if "localizations" in patch:
|
||||
item.localizations = _normalize_localizations(patch.get("localizations"))
|
||||
if "enrichment_status" in patch:
|
||||
item.enrichment_status = _coerce_str(patch.get("enrichment_status")) or item.enrichment_status
|
||||
if "enrichment_error" in patch:
|
||||
item.enrichment_error = _coerce_str(patch.get("enrichment_error"))
|
||||
if "enriched_at" in patch:
|
||||
item.enriched_at = _parse_datetime(_coerce_str(patch.get("enriched_at")))
|
||||
return item
|
||||
|
||||
|
||||
async def _geocode_target_location(query: str) -> dict[str, Any] | None:
|
||||
return await asyncio.to_thread(_news_target_geocode, query)
|
||||
|
||||
|
||||
async def _build_target_location_from_payload(
|
||||
payload: dict[str, Any],
|
||||
) -> NewsTargetLocation | None:
|
||||
country = normalize_country(payload.get("country"))
|
||||
city = _coerce_str(payload.get("city"))
|
||||
matched_location_name = _coerce_str(payload.get("matched_location_name"))
|
||||
confidence = _coerce_float(payload.get("confidence"))
|
||||
if confidence is not None:
|
||||
confidence = max(0.0, min(confidence, 1.0))
|
||||
|
||||
latitude = _coerce_float(payload.get("latitude"))
|
||||
longitude = _coerce_float(payload.get("longitude"))
|
||||
if latitude not in (None, 0.0) and longitude not in (None, 0.0):
|
||||
label = matched_location_name or ", ".join(part for part in (city, country) if part) or "关联位置"
|
||||
return NewsTargetLocation(
|
||||
latitude=latitude,
|
||||
longitude=longitude,
|
||||
label=label,
|
||||
source="ai_inferred_target",
|
||||
confidence=confidence,
|
||||
country=country,
|
||||
city=city,
|
||||
)
|
||||
|
||||
geocode_queries: list[str] = []
|
||||
for value in (
|
||||
", ".join(part for part in (city, country) if part),
|
||||
matched_location_name,
|
||||
city,
|
||||
country,
|
||||
):
|
||||
normalized = _coerce_str(value)
|
||||
if normalized and normalized not in geocode_queries:
|
||||
geocode_queries.append(normalized)
|
||||
|
||||
for query in geocode_queries:
|
||||
try:
|
||||
result = await _geocode_target_location(query)
|
||||
except Exception:
|
||||
continue
|
||||
if not isinstance(result, dict):
|
||||
continue
|
||||
latitude = _coerce_float(result.get("lat"))
|
||||
longitude = _coerce_float(result.get("lon"))
|
||||
if latitude in (None, 0.0) or longitude in (None, 0.0):
|
||||
continue
|
||||
label = (
|
||||
_coerce_str(result.get("display_name"))
|
||||
or matched_location_name
|
||||
or ", ".join(part for part in (city, country) if part)
|
||||
or query
|
||||
)
|
||||
return NewsTargetLocation(
|
||||
latitude=latitude,
|
||||
longitude=longitude,
|
||||
label=label,
|
||||
source="ai_inferred_target",
|
||||
confidence=confidence,
|
||||
country=country,
|
||||
city=city,
|
||||
)
|
||||
|
||||
centroid = get_country_centroid(country)
|
||||
if centroid:
|
||||
label = matched_location_name or city or country or "关联位置"
|
||||
return NewsTargetLocation(
|
||||
latitude=centroid["latitude"],
|
||||
longitude=centroid["longitude"],
|
||||
label=label,
|
||||
source="ai_inferred_target",
|
||||
confidence=confidence,
|
||||
country=country,
|
||||
city=city,
|
||||
)
|
||||
|
||||
return None
|
||||
|
||||
|
||||
async def _extract_target_location_from_text(item: ParsedNewsItem) -> NewsTargetLocation | None:
|
||||
combined_text = " ".join(part for part in (item.title, item.summary) if part).strip()
|
||||
if not combined_text:
|
||||
return None
|
||||
|
||||
for hint in _CITY_HINTS:
|
||||
city_name = _coerce_str(hint.get("name"))
|
||||
if not city_name or not _contains_location_alias(combined_text, city_name):
|
||||
continue
|
||||
country = normalize_country(hint.get("country"))
|
||||
geocode_query = ", ".join(part for part in (city_name, country) if part)
|
||||
try:
|
||||
result = await _geocode_target_location(geocode_query)
|
||||
except Exception:
|
||||
result = None
|
||||
if isinstance(result, dict):
|
||||
latitude = _coerce_float(result.get("lat"))
|
||||
longitude = _coerce_float(result.get("lon"))
|
||||
if latitude not in (None, 0.0) and longitude not in (None, 0.0):
|
||||
return NewsTargetLocation(
|
||||
latitude=latitude,
|
||||
longitude=longitude,
|
||||
label=_coerce_str(result.get("display_name")) or geocode_query,
|
||||
source="headline_location_hint",
|
||||
confidence=0.78,
|
||||
country=country,
|
||||
city=city_name,
|
||||
)
|
||||
centroid = get_country_centroid(country)
|
||||
if centroid:
|
||||
return NewsTargetLocation(
|
||||
latitude=centroid["latitude"],
|
||||
longitude=centroid["longitude"],
|
||||
label=geocode_query,
|
||||
source="headline_location_hint",
|
||||
confidence=0.68,
|
||||
country=country,
|
||||
city=city_name,
|
||||
)
|
||||
|
||||
for canonical, variants in COUNTRY_VARIANTS_MAP.items():
|
||||
if not get_country_centroid(canonical):
|
||||
continue
|
||||
searchable_variants = _iter_searchable_country_variants(canonical, variants)
|
||||
if not any(_contains_location_alias(combined_text, variant) for variant in searchable_variants):
|
||||
continue
|
||||
centroid = get_country_centroid(canonical)
|
||||
if not centroid:
|
||||
continue
|
||||
return NewsTargetLocation(
|
||||
latitude=centroid["latitude"],
|
||||
longitude=centroid["longitude"],
|
||||
label=canonical,
|
||||
source="headline_country_hint",
|
||||
confidence=0.62,
|
||||
country=canonical,
|
||||
city=None,
|
||||
)
|
||||
|
||||
return None
|
||||
|
||||
|
||||
async def _infer_news_target_location(
|
||||
item: ParsedNewsItem,
|
||||
*,
|
||||
provider_client: AIProviderClient | None,
|
||||
prompt: EffectiveAIPrompt | None = None,
|
||||
) -> NewsTargetLocation | None:
|
||||
target, _localizations = await _infer_news_enrichment(
|
||||
item,
|
||||
provider_client=provider_client,
|
||||
prompt=prompt,
|
||||
)
|
||||
return target
|
||||
|
||||
|
||||
async def _infer_news_enrichment(
|
||||
item: ParsedNewsItem,
|
||||
*,
|
||||
provider_client: AIProviderClient | None,
|
||||
prompt: EffectiveAIPrompt | None = None,
|
||||
) -> tuple[NewsTargetLocation | None, dict[str, dict[str, str]]]:
|
||||
text_hint = await _extract_target_location_from_text(item)
|
||||
content_error: str | None = None
|
||||
if text_hint is not None and text_hint.city:
|
||||
item.target_resolution_stage = text_hint.source
|
||||
item.target_ai_attempted = False
|
||||
item.target_ai_status = "skipped_text_hint"
|
||||
item.target_ai_error = None
|
||||
item.target_debug_note = f"text hint matched {text_hint.label}"
|
||||
localizations: dict[str, dict[str, str]] = {}
|
||||
|
||||
if provider_client is None:
|
||||
if text_hint is None or not text_hint.city:
|
||||
item.target_resolution_stage = text_hint.source if text_hint is not None else "unresolved"
|
||||
item.target_ai_attempted = False
|
||||
item.target_ai_status = "unavailable"
|
||||
item.target_ai_error = "AI provider is not configured or unavailable for earth-feed."
|
||||
item.target_debug_note = (
|
||||
f"text hint matched {text_hint.label}" if text_hint is not None else "no text location hint matched"
|
||||
)
|
||||
item.enrichment_status = "unavailable"
|
||||
item.enrichment_error = "AI provider is not configured or unavailable for earth-feed."
|
||||
return text_hint, localizations
|
||||
|
||||
if text_hint is None or not text_hint.city:
|
||||
item.target_ai_attempted = True
|
||||
item.target_ai_status = "attempted"
|
||||
item.target_ai_error = None
|
||||
item.target_debug_note = (
|
||||
f"text hint matched {text_hint.label}" if text_hint is not None else "no text location hint matched"
|
||||
)
|
||||
item.enrichment_status = "attempted"
|
||||
item.enrichment_error = None
|
||||
|
||||
prompt = prompt or await get_effective_prompt(None, NEWS_ENRICH_PROMPT_KEY)
|
||||
request = SituationalAnalysisRequest(
|
||||
title="Enrich Earth news item with event location and zh-CN content",
|
||||
objective=prompt.prompt,
|
||||
system_prompt=prompt.system_prompt or None,
|
||||
context={
|
||||
"news_item": {
|
||||
"title": item.title,
|
||||
"summary": item.summary,
|
||||
"source": item.source,
|
||||
"feed_name": item.feed_name,
|
||||
"feed_region": item.feed_region,
|
||||
"url": item.url,
|
||||
"published_at": (
|
||||
item.published_at.isoformat().replace("+00:00", "Z")
|
||||
if item.published_at
|
||||
else None
|
||||
),
|
||||
},
|
||||
"required_json_schema": {
|
||||
"location": {
|
||||
"country": "string|null",
|
||||
"city": "string|null",
|
||||
"matched_location_name": "string|null",
|
||||
"latitude": "number|null",
|
||||
"longitude": "number|null",
|
||||
"confidence": "number from 0 to 1",
|
||||
"reasoning_summary": "short string",
|
||||
},
|
||||
"localizations": {
|
||||
"zh-CN": {
|
||||
"title": "faithful Simplified Chinese title",
|
||||
"summary": "one-sentence newswire-style Simplified Chinese lead summary",
|
||||
}
|
||||
},
|
||||
},
|
||||
},
|
||||
constraints=[
|
||||
"Return only strict JSON. Do not wrap it in markdown.",
|
||||
"For localizations, do not add facts that are absent from the RSS headline, description, source, or date.",
|
||||
"Write zh-CN summary as one concise newswire-style sentence, like a breaking-news lead.",
|
||||
"If the RSS description is thin, write a conservative one-sentence summary that says only what is supported.",
|
||||
"Keep zh-CN summary factual, non-promotional, and avoid colon-heavy keyword labels.",
|
||||
"Prefer the event location, not the newsroom or publisher headquarters.",
|
||||
"When a country visit or summit is the clear topic but the city is omitted, use the most likely host city only if it is broadly public knowledge.",
|
||||
"Use null for unknown fields instead of inventing details.",
|
||||
"Calibrate confidence conservatively: 0.75+ only when the city is strongly supported, 0.55-0.74 for country-level or likely city inference, below 0.55 when weak.",
|
||||
],
|
||||
)
|
||||
try:
|
||||
response = await provider_client.analyze(request)
|
||||
except Exception as exc:
|
||||
if text_hint is None or not text_hint.city:
|
||||
item.target_resolution_stage = text_hint.source if text_hint is not None else "unresolved"
|
||||
item.target_ai_status = "provider_error"
|
||||
item.target_ai_error = str(exc)
|
||||
item.enrichment_status = "provider_error"
|
||||
item.enrichment_error = str(exc)
|
||||
return text_hint, localizations
|
||||
|
||||
payload = _first_json_object(response.content)
|
||||
if not isinstance(payload, dict):
|
||||
if text_hint is None or not text_hint.city:
|
||||
item.target_resolution_stage = text_hint.source if text_hint is not None else "unresolved"
|
||||
item.target_ai_status = "parse_error"
|
||||
item.target_ai_error = "AI response did not contain a parseable JSON object."
|
||||
item.enrichment_status = "parse_error"
|
||||
item.enrichment_error = "AI response did not contain a parseable JSON object."
|
||||
return text_hint, localizations
|
||||
|
||||
localizations = _normalize_localizations(payload.get("localizations"))
|
||||
if not localizations:
|
||||
content_error = "AI returned no usable localizations."
|
||||
|
||||
location_payload = payload.get("location") if isinstance(payload.get("location"), dict) else payload
|
||||
if text_hint is not None and text_hint.city:
|
||||
target = text_hint
|
||||
else:
|
||||
target = await _build_target_location_from_payload(location_payload)
|
||||
if target is None:
|
||||
item.target_resolution_stage = text_hint.source if text_hint is not None else "unresolved"
|
||||
item.target_ai_status = "no_result"
|
||||
item.target_ai_error = "AI returned no usable target coordinates or geocodeable location."
|
||||
target = text_hint
|
||||
elif target.confidence is not None and target.confidence < 0.45:
|
||||
item.target_resolution_stage = text_hint.source if text_hint is not None else "unresolved"
|
||||
item.target_ai_status = "low_confidence"
|
||||
item.target_ai_error = f"AI target confidence too low: {target.confidence:.2f}"
|
||||
target = text_hint
|
||||
else:
|
||||
item.target_resolution_stage = target.source
|
||||
item.target_ai_status = "success"
|
||||
item.target_ai_error = None
|
||||
item.target_debug_note = f"ai inferred {target.label}"
|
||||
|
||||
item.localizations = localizations
|
||||
if localizations and item.target_ai_status in {"success", "skipped_text_hint"}:
|
||||
item.enrichment_status = "success"
|
||||
item.enrichment_error = None
|
||||
elif localizations:
|
||||
item.enrichment_status = "content_only"
|
||||
item.enrichment_error = item.target_ai_error
|
||||
else:
|
||||
item.enrichment_status = "location_only" if target is not None else "no_result"
|
||||
item.enrichment_error = content_error or item.target_ai_error
|
||||
item.enriched_at = datetime.now(UTC) if localizations else None
|
||||
return target, localizations
|
||||
|
||||
|
||||
async def _enrich_items_with_target_locations(
|
||||
items: list[ParsedNewsItem],
|
||||
*,
|
||||
provider_client: AIProviderClient | None,
|
||||
prompt: EffectiveAIPrompt | None = None,
|
||||
) -> list[ParsedNewsItem]:
|
||||
if not items:
|
||||
return items
|
||||
|
||||
semaphore = asyncio.Semaphore(MAX_TARGET_INFERENCE_CONCURRENCY)
|
||||
|
||||
async def enrich(item: ParsedNewsItem) -> ParsedNewsItem:
|
||||
async with semaphore:
|
||||
target = await _infer_news_target_location(
|
||||
item,
|
||||
provider_client=provider_client,
|
||||
prompt=prompt,
|
||||
)
|
||||
item.target_location = target
|
||||
return item
|
||||
|
||||
return list(await asyncio.gather(*(enrich(item) for item in items)))
|
||||
|
||||
|
||||
def get_sources_for_region(region: str) -> list[NewsFeedSource]:
|
||||
return sorted(
|
||||
[source for source in NEWS_FEED_SOURCES if source.region in {"global", region}],
|
||||
@@ -349,7 +853,7 @@ def _parse_feed_entries(xml_text: str, source: NewsFeedSource) -> list[ParsedNew
|
||||
if not clean_title or not link:
|
||||
continue
|
||||
|
||||
item_source = _normalize_source_name(clean_title, source.name)
|
||||
item_source = source.name
|
||||
display_title = clean_title
|
||||
if source.source_type == "aggregated" and " - " in clean_title:
|
||||
parts = clean_title.rsplit(" - ", 1)
|
||||
@@ -385,23 +889,170 @@ def _serialize_sources(sources: list[NewsFeedSource]) -> list[dict[str, Any]]:
|
||||
]
|
||||
|
||||
|
||||
def _serialize_item(item: ParsedNewsItem, *, active_region: str) -> dict[str, Any]:
|
||||
published_at = item.published_at
|
||||
def _serialize_anchor(anchor: RegionAnchor) -> dict[str, Any]:
|
||||
return {
|
||||
"region": anchor.region,
|
||||
"label": anchor.label,
|
||||
"latitude": anchor.latitude,
|
||||
"longitude": anchor.longitude,
|
||||
}
|
||||
|
||||
|
||||
def _serialize_target(target: NewsTargetLocation | None) -> dict[str, Any] | None:
|
||||
if target is None:
|
||||
return None
|
||||
return {
|
||||
"latitude": target.latitude,
|
||||
"longitude": target.longitude,
|
||||
"label": target.label,
|
||||
"source": target.source,
|
||||
"confidence": target.confidence,
|
||||
"country": target.country,
|
||||
"city": target.city,
|
||||
}
|
||||
|
||||
|
||||
def _serialize_enriched_at(value: datetime | None) -> str | None:
|
||||
return value.isoformat().replace("+00:00", "Z") if value else None
|
||||
|
||||
|
||||
def _content_patch(item: ParsedNewsItem) -> dict[str, Any]:
|
||||
return {
|
||||
"content_language": item.content_language,
|
||||
"localizations": item.localizations,
|
||||
"enrichment_status": item.enrichment_status,
|
||||
"enrichment_error": item.enrichment_error,
|
||||
"enriched_at": _serialize_enriched_at(item.enriched_at),
|
||||
}
|
||||
|
||||
|
||||
def build_anchor_location_patch(
|
||||
item: ParsedNewsItem,
|
||||
*,
|
||||
queued: bool = False,
|
||||
queue_available: bool | None = None,
|
||||
) -> dict[str, Any]:
|
||||
anchor = get_region_anchor(item.feed_region)
|
||||
if queued:
|
||||
resolution_stage = "queued"
|
||||
ai_status = "queued"
|
||||
debug_note = "queued for async target location inference"
|
||||
else:
|
||||
resolution_stage = item.target_resolution_stage
|
||||
ai_status = item.target_ai_status
|
||||
debug_note = item.target_debug_note
|
||||
content_patch = _content_patch(item)
|
||||
if queued and content_patch["enrichment_status"] == "pending":
|
||||
content_patch["enrichment_status"] = "queued"
|
||||
return {
|
||||
"latitude": anchor.latitude,
|
||||
"longitude": anchor.longitude,
|
||||
"location_label": anchor.label,
|
||||
"location_source": "region_anchor",
|
||||
"verified": False,
|
||||
"location_meta": {
|
||||
"resolution_stage": resolution_stage,
|
||||
"ai_attempted": item.target_ai_attempted,
|
||||
"ai_status": ai_status,
|
||||
"ai_error": item.target_ai_error,
|
||||
"debug_note": debug_note,
|
||||
"queue_available": queue_available,
|
||||
"target": None,
|
||||
"anchor": _serialize_anchor(anchor),
|
||||
},
|
||||
**content_patch,
|
||||
}
|
||||
|
||||
|
||||
def build_target_location_patch(item: ParsedNewsItem, target: NewsTargetLocation | None) -> dict[str, Any]:
|
||||
if target is None:
|
||||
return build_anchor_location_patch(item)
|
||||
anchor = get_region_anchor(item.feed_region)
|
||||
return {
|
||||
"latitude": target.latitude,
|
||||
"longitude": target.longitude,
|
||||
"location_label": target.label,
|
||||
"location_source": target.source,
|
||||
"verified": True,
|
||||
"location_meta": {
|
||||
"resolution_stage": item.target_resolution_stage,
|
||||
"ai_attempted": item.target_ai_attempted,
|
||||
"ai_status": item.target_ai_status,
|
||||
"ai_error": item.target_ai_error,
|
||||
"debug_note": item.target_debug_note,
|
||||
"target": _serialize_target(target),
|
||||
"anchor": _serialize_anchor(anchor),
|
||||
},
|
||||
**_content_patch(item),
|
||||
}
|
||||
|
||||
|
||||
def build_target_location_job_payload(item: ParsedNewsItem) -> dict[str, Any]:
|
||||
published_at = item.published_at
|
||||
return {
|
||||
"id": item.id,
|
||||
"title": item.title,
|
||||
"summary": item.summary,
|
||||
"content_language": item.content_language,
|
||||
"localizations": item.localizations,
|
||||
"enrichment_status": item.enrichment_status,
|
||||
"enrichment_error": item.enrichment_error,
|
||||
"enriched_at": _serialize_enriched_at(item.enriched_at),
|
||||
"url": item.url,
|
||||
"source": item.source,
|
||||
"feed_name": item.feed_name,
|
||||
"feed_region": item.feed_region,
|
||||
"homepage_url": item.homepage_url,
|
||||
"published_at": published_at.isoformat().replace("+00:00", "Z") if published_at else None,
|
||||
}
|
||||
|
||||
|
||||
def parsed_news_item_from_job_payload(payload: dict[str, Any]) -> ParsedNewsItem:
|
||||
return ParsedNewsItem(
|
||||
id=str(payload.get("id") or ""),
|
||||
title=str(payload.get("title") or ""),
|
||||
summary=str(payload.get("summary") or ""),
|
||||
content_language=str(payload.get("content_language") or "en"),
|
||||
localizations=_normalize_localizations(payload.get("localizations")),
|
||||
enrichment_status=str(payload.get("enrichment_status") or "pending"),
|
||||
enrichment_error=_coerce_str(payload.get("enrichment_error")),
|
||||
enriched_at=_parse_datetime(_coerce_str(payload.get("enriched_at"))),
|
||||
url=str(payload.get("url") or ""),
|
||||
source=str(payload.get("source") or ""),
|
||||
feed_name=str(payload.get("feed_name") or ""),
|
||||
feed_region=str(payload.get("feed_region") or "global"),
|
||||
homepage_url=str(payload.get("homepage_url") or ""),
|
||||
published_at=_parse_datetime(_coerce_str(payload.get("published_at"))),
|
||||
)
|
||||
|
||||
|
||||
def _serialize_item(item: ParsedNewsItem, *, active_region: str) -> dict[str, Any]:
|
||||
published_at = item.published_at
|
||||
location_patch = item.location_patch or build_target_location_patch(item, item.target_location)
|
||||
return {
|
||||
"id": item.id,
|
||||
"title": item.title,
|
||||
"summary": item.summary,
|
||||
"content_language": item.content_language,
|
||||
"localizations": item.localizations,
|
||||
"display_title": _get_locale_text(item, "title"),
|
||||
"display_summary": _get_locale_text(item, "summary"),
|
||||
"url": item.url,
|
||||
"source": item.source,
|
||||
"feed_name": item.feed_name,
|
||||
"region": item.feed_region,
|
||||
"display_region": get_region_anchor(item.feed_region).label,
|
||||
"homepage_url": item.homepage_url,
|
||||
"published_at": published_at.isoformat().replace("+00:00", "Z") if published_at else None,
|
||||
"latitude": anchor.latitude,
|
||||
"longitude": anchor.longitude,
|
||||
"location_label": anchor.label,
|
||||
"location_inferred": True,
|
||||
"latitude": location_patch["latitude"],
|
||||
"longitude": location_patch["longitude"],
|
||||
"location_label": location_patch["location_label"],
|
||||
"location_source": location_patch["location_source"],
|
||||
"verified": location_patch["verified"],
|
||||
"location_meta": location_patch["location_meta"],
|
||||
"enrichment_status": item.enrichment_status,
|
||||
"enrichment_error": item.enrichment_error,
|
||||
"enriched_at": _serialize_enriched_at(item.enriched_at),
|
||||
"is_focus_match": item.feed_region == active_region,
|
||||
}
|
||||
|
||||
@@ -426,6 +1077,7 @@ def _build_payload(
|
||||
"lon": lon,
|
||||
"region": active_region,
|
||||
"label": profile.label,
|
||||
"display_region": get_region_anchor(active_region).label,
|
||||
"accent": profile.accent,
|
||||
},
|
||||
"sources": _serialize_sources(sources),
|
||||
@@ -472,6 +1124,68 @@ def _store_region_cache(region: str, *, items: list[ParsedNewsItem], sources: li
|
||||
)
|
||||
|
||||
|
||||
async def _apply_cached_locations_and_enqueue(items: list[ParsedNewsItem]) -> list[ParsedNewsItem]:
|
||||
if not items:
|
||||
return items
|
||||
|
||||
from app.services.earth_news_queue import (
|
||||
enqueue_target_location_job,
|
||||
get_cached_target_location_patch,
|
||||
)
|
||||
|
||||
async def enqueue_item(item: ParsedNewsItem, *, force: bool = False) -> bool:
|
||||
return await enqueue_target_location_job(build_target_location_job_payload(item), force=force)
|
||||
|
||||
async def apply_location(item: ParsedNewsItem) -> ParsedNewsItem:
|
||||
cached_patch = await get_cached_target_location_patch(item.id)
|
||||
if cached_patch:
|
||||
apply_enrichment_patch_to_item(item, cached_patch)
|
||||
if not _has_default_localization(item):
|
||||
queued = await enqueue_item(item, force=True)
|
||||
if queued and item.enrichment_status in {
|
||||
"pending",
|
||||
"unavailable",
|
||||
"provider_error",
|
||||
"parse_error",
|
||||
"no_result",
|
||||
"location_only",
|
||||
}:
|
||||
item.enrichment_status = "queued"
|
||||
return item
|
||||
|
||||
queued = await enqueue_item(item)
|
||||
item.location_patch = build_anchor_location_patch(
|
||||
item,
|
||||
queued=queued,
|
||||
queue_available=queued,
|
||||
)
|
||||
return item
|
||||
|
||||
return list(await asyncio.gather(*(apply_location(item) for item in items)))
|
||||
|
||||
|
||||
async def _enqueue_unverified_locations(items: list[ParsedNewsItem]) -> None:
|
||||
if not items:
|
||||
return
|
||||
|
||||
from app.services.earth_news_queue import enqueue_target_location_job
|
||||
|
||||
await asyncio.gather(
|
||||
*(
|
||||
enqueue_target_location_job(
|
||||
build_target_location_job_payload(item),
|
||||
force=not _has_default_localization(item),
|
||||
)
|
||||
for item in items
|
||||
if (
|
||||
item.location_patch is None
|
||||
or item.location_patch.get("verified") is False
|
||||
or not _has_default_localization(item)
|
||||
)
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
async def _fetch_source(
|
||||
client: httpx.AsyncClient,
|
||||
source: NewsFeedSource,
|
||||
@@ -484,11 +1198,10 @@ async def _fetch_source(
|
||||
return source, [], str(exc)
|
||||
|
||||
|
||||
async def get_earth_news_payload(lat: float | None = None, lon: float | None = None) -> dict[str, Any]:
|
||||
active_region = determine_focus_region(lat, lon)
|
||||
sources = get_sources_for_region(active_region)
|
||||
async def _fetch_rss_items_for_sources(
|
||||
sources: list[NewsFeedSource],
|
||||
) -> tuple[list[ParsedNewsItem], list[str]]:
|
||||
errors: list[str] = []
|
||||
|
||||
async with httpx.AsyncClient(
|
||||
timeout=REQUEST_TIMEOUT,
|
||||
follow_redirects=True,
|
||||
@@ -502,9 +1215,29 @@ async def get_earth_news_payload(lat: float | None = None, lon: float | None = N
|
||||
errors.append(f"{source.name}: {error}")
|
||||
continue
|
||||
fetched_items.extend(items)
|
||||
return fetched_items, errors
|
||||
|
||||
|
||||
def _needs_rss_supplement(*, item_count: int, newest_at: datetime | None) -> bool:
|
||||
if item_count < MAX_ITEMS_TOTAL:
|
||||
return True
|
||||
if newest_at is None:
|
||||
return True
|
||||
age_seconds = (datetime.now(UTC) - newest_at).total_seconds()
|
||||
return age_seconds > RSS_SUPPLEMENT_MAX_AGE_SECONDS
|
||||
|
||||
|
||||
async def _get_earth_news_payload_from_rss_only(
|
||||
*,
|
||||
lat: float | None,
|
||||
lon: float | None,
|
||||
active_region: str,
|
||||
sources: list[NewsFeedSource],
|
||||
) -> dict[str, Any]:
|
||||
fetched_items, errors = await _fetch_rss_items_for_sources(sources)
|
||||
ranked_items = _rank_and_trim_items(fetched_items, active_region=active_region)
|
||||
if ranked_items:
|
||||
ranked_items = await _apply_cached_locations_and_enqueue(ranked_items)
|
||||
_store_region_cache(active_region, items=ranked_items, sources=sources)
|
||||
return _build_payload(
|
||||
lat=lat,
|
||||
@@ -518,6 +1251,7 @@ async def get_earth_news_payload(lat: float | None = None, lon: float | None = N
|
||||
|
||||
cached = _get_cached_region_feed(active_region)
|
||||
if cached:
|
||||
cached.items = await _apply_cached_locations_and_enqueue(cached.items)
|
||||
return _build_payload(
|
||||
lat=lat,
|
||||
lon=lon,
|
||||
@@ -538,3 +1272,55 @@ async def get_earth_news_payload(lat: float | None = None, lon: float | None = N
|
||||
errors=errors,
|
||||
stale=False,
|
||||
)
|
||||
|
||||
|
||||
async def get_earth_news_payload(
|
||||
lat: float | None = None,
|
||||
lon: float | None = None,
|
||||
*,
|
||||
provider_client: AIProviderClient | None = None,
|
||||
db: AsyncSession | None = None,
|
||||
) -> dict[str, Any]:
|
||||
del provider_client
|
||||
active_region = determine_focus_region(lat, lon)
|
||||
sources = get_sources_for_region(active_region)
|
||||
|
||||
if db is None:
|
||||
return await _get_earth_news_payload_from_rss_only(
|
||||
lat=lat,
|
||||
lon=lon,
|
||||
active_region=active_region,
|
||||
sources=sources,
|
||||
)
|
||||
|
||||
from app.services.earth_news_store import (
|
||||
get_earth_news_freshness,
|
||||
list_earth_news_items,
|
||||
upsert_earth_news_items,
|
||||
)
|
||||
|
||||
errors: list[str] = []
|
||||
item_count, newest_at = await get_earth_news_freshness(db, active_region=active_region)
|
||||
should_supplement = _needs_rss_supplement(item_count=item_count, newest_at=newest_at)
|
||||
if should_supplement:
|
||||
fetched_items, errors = await _fetch_rss_items_for_sources(sources)
|
||||
ranked_fetched_items = _rank_and_trim_items(fetched_items, active_region=active_region)
|
||||
await upsert_earth_news_items(db, ranked_fetched_items)
|
||||
|
||||
items = await list_earth_news_items(
|
||||
db,
|
||||
active_region=active_region,
|
||||
limit=MAX_ITEMS_TOTAL,
|
||||
)
|
||||
await _enqueue_unverified_locations(items)
|
||||
stale = bool(errors and items)
|
||||
|
||||
return _build_payload(
|
||||
lat=lat,
|
||||
lon=lon,
|
||||
active_region=active_region,
|
||||
items=items,
|
||||
sources=sources,
|
||||
errors=errors,
|
||||
stale=stale,
|
||||
)
|
||||
|
||||
234
backend/app/services/earth_news_queue.py
Normal file
234
backend/app/services/earth_news_queue.py
Normal file
@@ -0,0 +1,234 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
import json
|
||||
from typing import Any, Protocol
|
||||
|
||||
import redis.asyncio as redis
|
||||
from redis.exceptions import ResponseError
|
||||
|
||||
from app.core.config import settings
|
||||
from app.core.logging import get_logger
|
||||
|
||||
|
||||
logger = get_logger(__name__, service="earth_news")
|
||||
|
||||
TARGET_LOCATION_STREAM = "earth_news:target_location:jobs"
|
||||
TARGET_LOCATION_GROUP = "earth_news_target_location"
|
||||
TARGET_LOCATION_DEAD_LETTER_STREAM = "earth_news:target_location:dead"
|
||||
TARGET_LOCATION_RESULT_TTL_SECONDS = 60 * 60 * 12
|
||||
TARGET_LOCATION_JOB_DEDUP_TTL_SECONDS = 60 * 60 * 6
|
||||
TARGET_LOCATION_MAX_ATTEMPTS = 3
|
||||
|
||||
_redis_client: redis.Redis | None = None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class NewsTargetLocationMessage:
|
||||
message_id: str
|
||||
item_id: str
|
||||
payload: dict[str, Any]
|
||||
attempts: int = 0
|
||||
|
||||
|
||||
class NewsTargetLocationQueue(Protocol):
|
||||
async def enqueue(self, *, item_id: str, payload: dict[str, Any], force: bool = False) -> bool:
|
||||
...
|
||||
|
||||
async def consume_batch(
|
||||
self,
|
||||
*,
|
||||
consumer_name: str,
|
||||
count: int,
|
||||
block_ms: int,
|
||||
) -> list[NewsTargetLocationMessage]:
|
||||
...
|
||||
|
||||
async def ack(self, message_id: str) -> None:
|
||||
...
|
||||
|
||||
async def retry_or_dead_letter(
|
||||
self,
|
||||
message: NewsTargetLocationMessage,
|
||||
*,
|
||||
error: str,
|
||||
) -> None:
|
||||
...
|
||||
|
||||
|
||||
def _get_redis_client() -> redis.Redis:
|
||||
global _redis_client
|
||||
if _redis_client is None:
|
||||
_redis_client = redis.from_url(settings.REDIS_URL, decode_responses=True)
|
||||
return _redis_client
|
||||
|
||||
|
||||
def _result_key(item_id: str) -> str:
|
||||
return f"earth_news:target_location:result:{item_id}"
|
||||
|
||||
|
||||
def _queued_key(item_id: str) -> str:
|
||||
return f"earth_news:target_location:queued:{item_id}"
|
||||
|
||||
|
||||
class RedisStreamsNewsTargetLocationQueue:
|
||||
def __init__(self, client: redis.Redis | None = None) -> None:
|
||||
self.client = client or _get_redis_client()
|
||||
self._group_ready = False
|
||||
|
||||
async def _ensure_group(self) -> None:
|
||||
if self._group_ready:
|
||||
return
|
||||
try:
|
||||
await self.client.xgroup_create(
|
||||
TARGET_LOCATION_STREAM,
|
||||
TARGET_LOCATION_GROUP,
|
||||
id="0",
|
||||
mkstream=True,
|
||||
)
|
||||
except ResponseError as exc:
|
||||
if "BUSYGROUP" not in str(exc):
|
||||
raise
|
||||
self._group_ready = True
|
||||
|
||||
async def enqueue(self, *, item_id: str, payload: dict[str, Any], force: bool = False) -> bool:
|
||||
await self._ensure_group()
|
||||
if force:
|
||||
await self.client.delete(_result_key(item_id), _queued_key(item_id))
|
||||
elif await self.client.exists(_result_key(item_id)):
|
||||
return False
|
||||
queued = await self.client.set(
|
||||
_queued_key(item_id),
|
||||
"1",
|
||||
nx=True,
|
||||
ex=TARGET_LOCATION_JOB_DEDUP_TTL_SECONDS,
|
||||
)
|
||||
if not queued:
|
||||
return bool(await self.client.exists(_queued_key(item_id)))
|
||||
await self.client.xadd(
|
||||
TARGET_LOCATION_STREAM,
|
||||
{
|
||||
"item_id": item_id,
|
||||
"attempts": "0",
|
||||
"payload": json.dumps(payload, ensure_ascii=False),
|
||||
},
|
||||
)
|
||||
return True
|
||||
|
||||
async def consume_batch(
|
||||
self,
|
||||
*,
|
||||
consumer_name: str,
|
||||
count: int,
|
||||
block_ms: int,
|
||||
) -> list[NewsTargetLocationMessage]:
|
||||
await self._ensure_group()
|
||||
streams = await self.client.xreadgroup(
|
||||
TARGET_LOCATION_GROUP,
|
||||
consumer_name,
|
||||
{TARGET_LOCATION_STREAM: ">"},
|
||||
count=count,
|
||||
block=block_ms,
|
||||
)
|
||||
messages: list[NewsTargetLocationMessage] = []
|
||||
for _stream_name, stream_messages in streams:
|
||||
for message_id, fields in stream_messages:
|
||||
raw_payload = fields.get("payload")
|
||||
item_id = fields.get("item_id")
|
||||
if not raw_payload or not item_id:
|
||||
await self.ack(message_id)
|
||||
continue
|
||||
try:
|
||||
payload = json.loads(raw_payload)
|
||||
except json.JSONDecodeError:
|
||||
await self.ack(message_id)
|
||||
continue
|
||||
attempts = int(fields.get("attempts") or 0)
|
||||
messages.append(
|
||||
NewsTargetLocationMessage(
|
||||
message_id=message_id,
|
||||
item_id=item_id,
|
||||
payload=payload,
|
||||
attempts=attempts,
|
||||
)
|
||||
)
|
||||
return messages
|
||||
|
||||
async def ack(self, message_id: str) -> None:
|
||||
await self.client.xack(TARGET_LOCATION_STREAM, TARGET_LOCATION_GROUP, message_id)
|
||||
|
||||
async def retry_or_dead_letter(
|
||||
self,
|
||||
message: NewsTargetLocationMessage,
|
||||
*,
|
||||
error: str,
|
||||
) -> None:
|
||||
await self.ack(message.message_id)
|
||||
if message.attempts + 1 >= TARGET_LOCATION_MAX_ATTEMPTS:
|
||||
await self.client.xadd(
|
||||
TARGET_LOCATION_DEAD_LETTER_STREAM,
|
||||
{
|
||||
"item_id": message.item_id,
|
||||
"attempts": str(message.attempts + 1),
|
||||
"error": error,
|
||||
"payload": json.dumps(message.payload, ensure_ascii=False),
|
||||
},
|
||||
)
|
||||
return
|
||||
await self.client.xadd(
|
||||
TARGET_LOCATION_STREAM,
|
||||
{
|
||||
"item_id": message.item_id,
|
||||
"attempts": str(message.attempts + 1),
|
||||
"payload": json.dumps(message.payload, ensure_ascii=False),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def get_news_target_location_queue() -> NewsTargetLocationQueue:
|
||||
return RedisStreamsNewsTargetLocationQueue()
|
||||
|
||||
|
||||
async def enqueue_target_location_job(payload: dict[str, Any], *, force: bool = False) -> bool:
|
||||
item_id = str(payload.get("id") or "")
|
||||
if not item_id:
|
||||
return False
|
||||
try:
|
||||
queue = get_news_target_location_queue()
|
||||
return await queue.enqueue(item_id=item_id, payload=payload, force=force)
|
||||
except Exception as exc:
|
||||
logger.warning_event(
|
||||
"Failed to enqueue Earth news target location job",
|
||||
event="earth_news.target_location.enqueue_failed",
|
||||
context={"item_id": item_id, "error": str(exc)},
|
||||
)
|
||||
return False
|
||||
|
||||
|
||||
async def get_cached_target_location_patch(item_id: str) -> dict[str, Any] | None:
|
||||
try:
|
||||
raw_value = await _get_redis_client().get(_result_key(item_id))
|
||||
except Exception as exc:
|
||||
logger.warning_event(
|
||||
"Failed to read Earth news target location cache",
|
||||
event="earth_news.target_location.cache_read_failed",
|
||||
context={"item_id": item_id, "error": str(exc)},
|
||||
)
|
||||
return None
|
||||
if not raw_value:
|
||||
return None
|
||||
try:
|
||||
value = json.loads(raw_value)
|
||||
except json.JSONDecodeError:
|
||||
return None
|
||||
return value if isinstance(value, dict) else None
|
||||
|
||||
|
||||
async def save_target_location_patch(item_id: str, patch: dict[str, Any]) -> None:
|
||||
client = _get_redis_client()
|
||||
await client.setex(
|
||||
_result_key(item_id),
|
||||
TARGET_LOCATION_RESULT_TTL_SECONDS,
|
||||
json.dumps(patch, ensure_ascii=False),
|
||||
)
|
||||
await client.delete(_queued_key(item_id))
|
||||
244
backend/app/services/earth_news_store.py
Normal file
244
backend/app/services/earth_news_store.py
Normal file
@@ -0,0 +1,244 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.models.earth_news import EarthNewsItem
|
||||
from app.services.earth_news import (
|
||||
ParsedNewsItem,
|
||||
apply_enrichment_patch_to_item,
|
||||
build_anchor_location_patch,
|
||||
)
|
||||
|
||||
|
||||
def _coerce_datetime(value: datetime | None) -> datetime | None:
|
||||
if value is None:
|
||||
return None
|
||||
if value.tzinfo is None:
|
||||
return value.replace(tzinfo=UTC)
|
||||
return value.astimezone(UTC)
|
||||
|
||||
|
||||
def _location_patch_from_record(record: EarthNewsItem) -> dict[str, Any]:
|
||||
return {
|
||||
"latitude": record.latitude,
|
||||
"longitude": record.longitude,
|
||||
"location_label": record.location_label,
|
||||
"location_source": record.location_source,
|
||||
"verified": record.verified,
|
||||
"location_meta": dict(record.location_meta or {}),
|
||||
}
|
||||
|
||||
|
||||
def record_to_parsed_news_item(record: EarthNewsItem) -> ParsedNewsItem:
|
||||
item = ParsedNewsItem(
|
||||
id=record.id,
|
||||
title=record.title,
|
||||
summary=record.summary or "",
|
||||
url=record.url,
|
||||
source=record.source or "",
|
||||
feed_name=record.feed_name or "",
|
||||
feed_region=record.region or "global",
|
||||
homepage_url=record.homepage_url or "",
|
||||
published_at=_coerce_datetime(record.published_at),
|
||||
content_language=record.content_language or "en",
|
||||
localizations=dict(record.localizations or {}),
|
||||
enrichment_status=record.enrichment_status or "pending",
|
||||
enrichment_error=record.enrichment_error,
|
||||
enriched_at=_coerce_datetime(record.enriched_at),
|
||||
)
|
||||
return apply_enrichment_patch_to_item(item, _location_patch_from_record(record))
|
||||
|
||||
|
||||
def _query_sort_key(active_region: str):
|
||||
return (
|
||||
EarthNewsItem.region != active_region,
|
||||
EarthNewsItem.published_at.is_(None),
|
||||
EarthNewsItem.published_at.desc().nullslast(),
|
||||
EarthNewsItem.feed_name.asc(),
|
||||
)
|
||||
|
||||
|
||||
async def list_earth_news_items(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
active_region: str,
|
||||
limit: int,
|
||||
) -> list[ParsedNewsItem]:
|
||||
regions = {"global", active_region}
|
||||
result = await db.execute(
|
||||
select(EarthNewsItem)
|
||||
.where(EarthNewsItem.region.in_(regions))
|
||||
.order_by(*_query_sort_key(active_region))
|
||||
.limit(limit)
|
||||
)
|
||||
return [record_to_parsed_news_item(record) for record in result.scalars().all()]
|
||||
|
||||
|
||||
async def get_earth_news_freshness(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
active_region: str,
|
||||
) -> tuple[int, datetime | None]:
|
||||
regions = {"global", active_region}
|
||||
result = await db.execute(
|
||||
select(
|
||||
func.count(EarthNewsItem.id),
|
||||
func.max(func.coalesce(EarthNewsItem.published_at, EarthNewsItem.last_seen_at)),
|
||||
).where(EarthNewsItem.region.in_(regions))
|
||||
)
|
||||
count, newest = result.one()
|
||||
item_count = int(count or 0)
|
||||
if item_count == 0:
|
||||
return 0, None
|
||||
return item_count, _coerce_datetime(newest)
|
||||
|
||||
|
||||
async def upsert_earth_news_items(db: AsyncSession, items: list[ParsedNewsItem]) -> int:
|
||||
if not items:
|
||||
return 0
|
||||
now = datetime.now(UTC)
|
||||
existing_result = await db.execute(
|
||||
select(EarthNewsItem).where(EarthNewsItem.id.in_([item.id for item in items]))
|
||||
)
|
||||
existing = {record.id: record for record in existing_result.scalars().all()}
|
||||
changed = 0
|
||||
for item in items:
|
||||
record = existing.get(item.id)
|
||||
if record is None:
|
||||
patch = build_anchor_location_patch(item)
|
||||
record = EarthNewsItem(
|
||||
id=item.id,
|
||||
title=item.title,
|
||||
summary=item.summary,
|
||||
content_language=item.content_language,
|
||||
localizations=dict(item.localizations or {}),
|
||||
url=item.url,
|
||||
source=item.source,
|
||||
feed_name=item.feed_name,
|
||||
region=item.feed_region,
|
||||
homepage_url=item.homepage_url,
|
||||
published_at=item.published_at,
|
||||
latitude=patch["latitude"],
|
||||
longitude=patch["longitude"],
|
||||
location_label=patch["location_label"],
|
||||
location_source=patch["location_source"],
|
||||
verified=patch["verified"],
|
||||
location_meta=patch["location_meta"],
|
||||
first_seen_at=now,
|
||||
last_seen_at=now,
|
||||
enrichment_status=item.enrichment_status,
|
||||
enrichment_error=item.enrichment_error,
|
||||
enriched_at=item.enriched_at,
|
||||
)
|
||||
db.add(record)
|
||||
changed += 1
|
||||
continue
|
||||
|
||||
record.title = item.title
|
||||
record.summary = item.summary
|
||||
record.url = item.url
|
||||
record.source = item.source
|
||||
record.feed_name = item.feed_name
|
||||
record.region = item.feed_region
|
||||
record.homepage_url = item.homepage_url
|
||||
record.published_at = item.published_at
|
||||
record.last_seen_at = now
|
||||
if item.localizations:
|
||||
record.content_language = item.content_language
|
||||
record.localizations = dict(item.localizations or {})
|
||||
record.enrichment_status = item.enrichment_status
|
||||
record.enrichment_error = item.enrichment_error
|
||||
record.enriched_at = item.enriched_at
|
||||
changed += 1
|
||||
await db.flush()
|
||||
return changed
|
||||
|
||||
|
||||
async def update_earth_news_item_location(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
item_id: str,
|
||||
patch: dict[str, Any],
|
||||
) -> bool:
|
||||
record = await db.get(EarthNewsItem, item_id)
|
||||
if record is None:
|
||||
return False
|
||||
record.latitude = float(patch["latitude"])
|
||||
record.longitude = float(patch["longitude"])
|
||||
record.location_label = str(patch["location_label"])
|
||||
record.location_source = str(patch["location_source"])
|
||||
record.verified = bool(patch["verified"])
|
||||
record.location_meta = dict(patch.get("location_meta") or {})
|
||||
record.resolved_at = datetime.now(UTC) if record.verified else None
|
||||
await db.flush()
|
||||
return True
|
||||
|
||||
|
||||
async def update_earth_news_item_enrichment(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
item_id: str,
|
||||
patch: dict[str, Any],
|
||||
) -> bool:
|
||||
record = await db.get(EarthNewsItem, item_id)
|
||||
if record is None:
|
||||
return False
|
||||
if "latitude" in patch:
|
||||
record.latitude = float(patch["latitude"])
|
||||
record.longitude = float(patch["longitude"])
|
||||
record.location_label = str(patch["location_label"])
|
||||
record.location_source = str(patch["location_source"])
|
||||
record.verified = bool(patch["verified"])
|
||||
record.location_meta = dict(patch.get("location_meta") or {})
|
||||
record.resolved_at = datetime.now(UTC) if record.verified else None
|
||||
if "content_language" in patch:
|
||||
record.content_language = str(patch.get("content_language") or "en")
|
||||
if "localizations" in patch:
|
||||
record.localizations = dict(patch.get("localizations") or {})
|
||||
if "enrichment_status" in patch:
|
||||
record.enrichment_status = str(patch.get("enrichment_status") or "pending")
|
||||
if "enrichment_error" in patch:
|
||||
record.enrichment_error = patch.get("enrichment_error")
|
||||
if patch.get("enriched_at"):
|
||||
try:
|
||||
parsed_enriched_at = datetime.fromisoformat(
|
||||
str(patch["enriched_at"]).replace("Z", "+00:00")
|
||||
)
|
||||
except ValueError:
|
||||
parsed_enriched_at = datetime.now(UTC)
|
||||
record.enriched_at = _coerce_datetime(parsed_enriched_at)
|
||||
elif patch.get("localizations"):
|
||||
record.enriched_at = datetime.now(UTC)
|
||||
await db.flush()
|
||||
return True
|
||||
|
||||
|
||||
async def list_unverified_earth_news_items(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
active_region: str,
|
||||
limit: int,
|
||||
) -> list[ParsedNewsItem]:
|
||||
regions = {"global", active_region}
|
||||
result = await db.execute(
|
||||
select(EarthNewsItem)
|
||||
.where(EarthNewsItem.region.in_(regions))
|
||||
.where(EarthNewsItem.verified.is_(False))
|
||||
.order_by(*_query_sort_key(active_region))
|
||||
.limit(limit)
|
||||
)
|
||||
return [record_to_parsed_news_item(record) for record in result.scalars().all()]
|
||||
|
||||
|
||||
async def list_all_earth_news_records(db: AsyncSession) -> list[EarthNewsItem]:
|
||||
result = await db.execute(
|
||||
select(EarthNewsItem).order_by(
|
||||
EarthNewsItem.published_at.desc().nullslast(),
|
||||
EarthNewsItem.last_seen_at.desc(),
|
||||
)
|
||||
)
|
||||
return list(result.scalars().all())
|
||||
142
backend/app/services/earth_news_worker.py
Normal file
142
backend/app/services/earth_news_worker.py
Normal file
@@ -0,0 +1,142 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from contextlib import suppress
|
||||
from socket import gethostname
|
||||
from typing import Any
|
||||
|
||||
from app.core.logging import get_logger
|
||||
from app.core.websocket.broadcaster import broadcaster
|
||||
from app.db.session import async_session_factory
|
||||
from app.services.ai_client import AIProviderClient
|
||||
from app.ai_tasks.prompts import get_effective_prompt
|
||||
from app.services.earth_news import (
|
||||
NEWS_ENRICH_PROMPT_KEY,
|
||||
_infer_news_enrichment,
|
||||
build_target_location_patch,
|
||||
parsed_news_item_from_job_payload,
|
||||
)
|
||||
from app.services.earth_news_queue import (
|
||||
NewsTargetLocationMessage,
|
||||
get_news_target_location_queue,
|
||||
save_target_location_patch,
|
||||
)
|
||||
from app.services.earth_news_store import update_earth_news_item_enrichment as update_earth_news_item_location
|
||||
|
||||
|
||||
logger = get_logger(__name__, service="earth_news")
|
||||
|
||||
WORKER_BATCH_SIZE = 4
|
||||
WORKER_BLOCK_MS = 5000
|
||||
WORKER_BACKOFF_SECONDS = 5.0
|
||||
|
||||
_worker_task: asyncio.Task | None = None
|
||||
|
||||
|
||||
async def _build_provider_client() -> AIProviderClient | None:
|
||||
try:
|
||||
from app.api.v1.settings import get_runtime_ai_provider_config
|
||||
|
||||
async with async_session_factory() as session:
|
||||
runtime_config = await get_runtime_ai_provider_config(session)
|
||||
return AIProviderClient(
|
||||
service_url=runtime_config["service_url"],
|
||||
service_token=runtime_config["service_token"],
|
||||
timeout=runtime_config["timeout_seconds"],
|
||||
retry_attempts=runtime_config["retry_attempts"],
|
||||
llm_config=runtime_config.get("llm_config") or {},
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.warning_event(
|
||||
"Failed to build Earth news AI provider client",
|
||||
event="earth_news.target_location.provider_unavailable",
|
||||
context={"error": str(exc)},
|
||||
)
|
||||
return None
|
||||
|
||||
|
||||
async def process_target_location_message(
|
||||
message: NewsTargetLocationMessage,
|
||||
*,
|
||||
provider_client: AIProviderClient | None,
|
||||
) -> dict[str, Any]:
|
||||
item = parsed_news_item_from_job_payload(message.payload)
|
||||
async with async_session_factory() as session:
|
||||
prompt = await get_effective_prompt(session, NEWS_ENRICH_PROMPT_KEY)
|
||||
target, localizations = await _infer_news_enrichment(
|
||||
item,
|
||||
provider_client=provider_client,
|
||||
prompt=prompt,
|
||||
)
|
||||
item.target_location = target
|
||||
item.localizations = localizations or item.localizations
|
||||
patch = build_target_location_patch(item, target)
|
||||
await save_target_location_patch(item.id, patch)
|
||||
async with async_session_factory() as session:
|
||||
await update_earth_news_item_location(session, item_id=item.id, patch=patch)
|
||||
await session.commit()
|
||||
await broadcaster.broadcast_custom(
|
||||
"earth_news",
|
||||
{
|
||||
"item_id": item.id,
|
||||
"patch": patch,
|
||||
},
|
||||
)
|
||||
return patch
|
||||
|
||||
|
||||
async def _run_target_location_worker() -> None:
|
||||
consumer_name = f"{gethostname()}:{id(asyncio.current_task())}"
|
||||
queue = get_news_target_location_queue()
|
||||
while True:
|
||||
try:
|
||||
messages = await queue.consume_batch(
|
||||
consumer_name=consumer_name,
|
||||
count=WORKER_BATCH_SIZE,
|
||||
block_ms=WORKER_BLOCK_MS,
|
||||
)
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception as exc:
|
||||
logger.warning_event(
|
||||
"Earth news target location worker queue read failed",
|
||||
event="earth_news.target_location.worker_read_failed",
|
||||
context={"error": str(exc)},
|
||||
)
|
||||
await asyncio.sleep(WORKER_BACKOFF_SECONDS)
|
||||
continue
|
||||
|
||||
if not messages:
|
||||
continue
|
||||
provider_client = await _build_provider_client()
|
||||
for message in messages:
|
||||
try:
|
||||
await process_target_location_message(message, provider_client=provider_client)
|
||||
await queue.ack(message.message_id)
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception as exc:
|
||||
logger.warning_event(
|
||||
"Earth news target location worker job failed",
|
||||
event="earth_news.target_location.worker_job_failed",
|
||||
context={"item_id": message.item_id, "error": str(exc)},
|
||||
)
|
||||
with suppress(Exception):
|
||||
await queue.retry_or_dead_letter(message, error=str(exc))
|
||||
|
||||
|
||||
def start_earth_news_target_worker() -> None:
|
||||
global _worker_task
|
||||
if _worker_task is None or _worker_task.done():
|
||||
_worker_task = asyncio.create_task(_run_target_location_worker())
|
||||
|
||||
|
||||
async def stop_earth_news_target_worker() -> None:
|
||||
global _worker_task
|
||||
task = _worker_task
|
||||
if task is None:
|
||||
return
|
||||
task.cancel()
|
||||
with suppress(asyncio.CancelledError):
|
||||
await task
|
||||
_worker_task = None
|
||||
@@ -7,6 +7,26 @@ from typing import Any
|
||||
import httpx
|
||||
|
||||
MODELS_DEV_URL = "https://models.dev/api.json"
|
||||
OPENCODE_GO_MODELS_URL = "https://opencode.ai/zen/go/v1/models"
|
||||
|
||||
OPENCODE_GO_MODEL_PROVIDER_APIS = {
|
||||
"minimax-m2.7": "anthropic-messages",
|
||||
"minimax-m2.5": "anthropic-messages",
|
||||
}
|
||||
OPENCODE_GO_FALLBACK_MODELS = [
|
||||
"minimax-m2.7",
|
||||
"minimax-m2.5",
|
||||
"kimi-k2.6",
|
||||
"kimi-k2.5",
|
||||
"glm-5.1",
|
||||
"glm-5",
|
||||
"deepseek-v4-pro",
|
||||
"deepseek-v4-flash",
|
||||
"qwen3.6-plus",
|
||||
"qwen3.5-plus",
|
||||
"mimo-v2.5-pro",
|
||||
"mimo-v2.5",
|
||||
]
|
||||
|
||||
|
||||
FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
|
||||
@@ -80,6 +100,17 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
|
||||
"api_key_env": "OPENROUTER_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
"opencode-go": {
|
||||
"provider": "opencode-go",
|
||||
"label": "OpenCode Go",
|
||||
"provider_api": "openai-completions",
|
||||
"base_url": "https://opencode.ai/zen/go/v1",
|
||||
"model": "glm-5.1",
|
||||
"models": OPENCODE_GO_FALLBACK_MODELS,
|
||||
"model_provider_apis": OPENCODE_GO_MODEL_PROVIDER_APIS,
|
||||
"api_key_env": "OPENCODE_GO_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
"ollama": {
|
||||
"provider": "ollama",
|
||||
"label": "Ollama Local",
|
||||
@@ -114,8 +145,43 @@ def get_fallback_llm_provider_preset(provider: str) -> dict[str, Any]:
|
||||
return dict(FALLBACK_LLM_PROVIDER_PRESETS[key])
|
||||
|
||||
|
||||
async def refresh_llm_provider_preset(provider: str) -> dict[str, Any]:
|
||||
def _opencode_go_model_provider_apis(model_ids: list[str]) -> dict[str, str]:
|
||||
return {
|
||||
model_id: OPENCODE_GO_MODEL_PROVIDER_APIS.get(model_id, "openai-completions")
|
||||
for model_id in model_ids
|
||||
}
|
||||
|
||||
|
||||
async def refresh_llm_provider_preset(provider: str, api_key: str | None = None) -> dict[str, Any]:
|
||||
fallback = get_fallback_llm_provider_preset(provider)
|
||||
if fallback["provider"] == "opencode-go":
|
||||
headers = {"User-Agent": "Planet/1.0"}
|
||||
if api_key:
|
||||
headers["Authorization"] = f"Bearer {api_key}"
|
||||
async with httpx.AsyncClient(timeout=15.0, follow_redirects=True) as client:
|
||||
response = await client.get(
|
||||
OPENCODE_GO_MODELS_URL,
|
||||
headers=headers,
|
||||
)
|
||||
response.raise_for_status()
|
||||
payload = response.json()
|
||||
|
||||
data = payload.get("data") if isinstance(payload, dict) else []
|
||||
model_ids = [
|
||||
str(item.get("id"))
|
||||
for item in data
|
||||
if isinstance(item, dict) and item.get("id")
|
||||
][:120]
|
||||
if not model_ids:
|
||||
model_ids = fallback["models"]
|
||||
return {
|
||||
**fallback,
|
||||
"model": fallback["model"] if fallback["model"] in model_ids else model_ids[0],
|
||||
"models": model_ids,
|
||||
"model_provider_apis": _opencode_go_model_provider_apis(model_ids),
|
||||
"source": OPENCODE_GO_MODELS_URL,
|
||||
}
|
||||
|
||||
models_dev_key = MODELS_DEV_PROVIDER_KEYS.get(fallback["provider"])
|
||||
if not models_dev_key:
|
||||
return fallback
|
||||
|
||||
@@ -7,8 +7,12 @@ import re
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Iterable
|
||||
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.logging import get_logger
|
||||
from app.core.countries import COUNTRY_ENTRIES, normalize_country
|
||||
from app.schemas.ai import SituationalAnalysisRequest
|
||||
from app.ai_tasks.prompts import get_effective_prompt
|
||||
from app.services.ai_client import AIProviderClient
|
||||
from app.services.ai_tools.evidence_store import normalize_search_evidence
|
||||
from app.services.ai_tools.web_search import WebSearchClient, WebSearchError
|
||||
@@ -23,7 +27,12 @@ from app.services.location.text import (
|
||||
|
||||
VALID_LLM_PRECISIONS = {"precise", "site", "city"}
|
||||
DEFAULT_MIN_CONFIDENCE = 0.55
|
||||
LOCATION_NORMALIZE_PROMPT_KEY = "location.factcheck.normalize"
|
||||
LOCATION_RESOLVE_PROMPT_KEY = "location.factcheck.resolve"
|
||||
MODEL_CONFIDENCE_WEIGHT = 0.25
|
||||
LOG_TEXT_LIMIT = 1200
|
||||
LOG_EVIDENCE_LIMIT = 5
|
||||
logger = get_logger(__name__, service="location")
|
||||
_geocode_llm_city = build_default_nominatim_geocoder()
|
||||
_LLM_LOCATION_NAME_KEYS = (
|
||||
"matched_location_name",
|
||||
@@ -92,6 +101,35 @@ class LocationEvidenceScore:
|
||||
summary: str
|
||||
|
||||
|
||||
def _truncate_log_text(value: Any, limit: int = LOG_TEXT_LIMIT) -> str:
|
||||
text = coerce_str(value)
|
||||
if len(text) <= limit:
|
||||
return text
|
||||
return f"{text[:limit]}…"
|
||||
|
||||
|
||||
def _summarize_search_evidence(evidence: list[dict[str, Any]] | None) -> list[dict[str, Any]]:
|
||||
items: list[dict[str, Any]] = []
|
||||
for item in (evidence or [])[:LOG_EVIDENCE_LIMIT]:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
items.append(
|
||||
{
|
||||
"title": _truncate_log_text(item.get("title"), 180),
|
||||
"source": _truncate_log_text(item.get("source") or item.get("name"), 120),
|
||||
"url": _truncate_log_text(item.get("url"), 240),
|
||||
"snippet": _truncate_log_text(
|
||||
item.get("snippet")
|
||||
or item.get("content")
|
||||
or item.get("text")
|
||||
or item.get("summary"),
|
||||
360,
|
||||
),
|
||||
}
|
||||
)
|
||||
return items
|
||||
|
||||
|
||||
def _first_json_object(text: str) -> dict[str, Any] | None:
|
||||
stripped = text.strip()
|
||||
if not stripped:
|
||||
@@ -157,6 +195,64 @@ def _evidence_label(item: Any) -> str:
|
||||
return coerce_str(item)
|
||||
|
||||
|
||||
def _evidence_text(item: dict[str, Any]) -> str:
|
||||
return " ".join(
|
||||
coerce_str(item.get(key))
|
||||
for key in ("title", "source", "name", "url", "snippet", "content", "text", "quote", "summary")
|
||||
if coerce_str(item.get(key))
|
||||
)
|
||||
|
||||
|
||||
def _search_evidence_entity_match(item: dict[str, Any], query: LocationQuery) -> bool:
|
||||
haystack = normalize_text(_evidence_text(item))
|
||||
if not haystack:
|
||||
return False
|
||||
needles = [
|
||||
coerce_str(query.name),
|
||||
*[coerce_str(alias) for alias in query.aliases],
|
||||
]
|
||||
return any(normalize_text(needle) and normalize_text(needle) in haystack for needle in needles)
|
||||
|
||||
|
||||
def _evidence_has_location_assertion(item: dict[str, Any], city: str) -> bool:
|
||||
normalized_city = normalize_text(city)
|
||||
text = normalize_text(_evidence_text(item))
|
||||
if not normalized_city or normalized_city not in text:
|
||||
return False
|
||||
assertion_terms = (
|
||||
"located",
|
||||
"situated",
|
||||
"built",
|
||||
"hosted",
|
||||
"deployed",
|
||||
"installed",
|
||||
"facility",
|
||||
"campus",
|
||||
"site",
|
||||
"data center",
|
||||
"datacenter",
|
||||
"supercomputer center",
|
||||
"位于",
|
||||
"位於",
|
||||
"坐落",
|
||||
"建置",
|
||||
"設置",
|
||||
"设置",
|
||||
)
|
||||
return any(term in text for term in assertion_terms)
|
||||
|
||||
|
||||
def _city_is_unsupported_name_hint(payload: dict[str, Any], query: LocationQuery, evidence_items: list[dict[str, Any]]) -> bool:
|
||||
city = coerce_str(payload.get("city") or query.city)
|
||||
if not city:
|
||||
return False
|
||||
normalized_city = normalize_text(city)
|
||||
normalized_name = normalize_text(query.name)
|
||||
if not normalized_city or not normalized_name or normalized_city not in normalized_name:
|
||||
return False
|
||||
return not any(_evidence_has_location_assertion(item, city) for item in evidence_items)
|
||||
|
||||
|
||||
def _normalize_llm_precision(value: Any) -> str:
|
||||
text = coerce_str(value).lower()
|
||||
return LLM_PRECISION_ALIASES.get(text, text)
|
||||
@@ -583,6 +679,7 @@ def _weak_evidence_penalty(
|
||||
payload: dict[str, Any],
|
||||
evidence_items: list[dict[str, Any]],
|
||||
*,
|
||||
query: LocationQuery,
|
||||
entity_match: float,
|
||||
geography_match: float,
|
||||
conflict_penalty: float,
|
||||
@@ -593,6 +690,8 @@ def _weak_evidence_penalty(
|
||||
penalty += 0.20
|
||||
if any(_truthy_evidence_field(item, "ambiguous") for item in evidence_items):
|
||||
penalty += 0.15
|
||||
if _city_is_unsupported_name_hint(payload, query, evidence_items):
|
||||
penalty += 0.10
|
||||
if conflict_penalty == 0.0 and entity_match > 0 and geography_match >= 0.20:
|
||||
return min(penalty, 0.15)
|
||||
return min(penalty, 0.30)
|
||||
@@ -615,6 +714,7 @@ def _score_llm_location_payload(
|
||||
weak_evidence_penalty = _weak_evidence_penalty(
|
||||
payload,
|
||||
evidence_items,
|
||||
query=query,
|
||||
entity_match=entity_match,
|
||||
geography_match=geography_match,
|
||||
conflict_penalty=conflict_penalty,
|
||||
@@ -631,6 +731,8 @@ def _score_llm_location_payload(
|
||||
- weak_evidence_penalty
|
||||
)
|
||||
score = min(max(score, 0.0), 1.0)
|
||||
if _city_is_unsupported_name_hint(payload, query, evidence_items):
|
||||
score = min(score, 0.54)
|
||||
summary = (
|
||||
f"combined={score:.2f}; model={model_confidence:.2f}; "
|
||||
f"source={source_quality:.2f}; entity={entity_match:.2f}; "
|
||||
@@ -842,15 +944,43 @@ async def collect_location_search_evidence(
|
||||
) -> LocationSearchEvidenceResult:
|
||||
search_query = _location_search_query(query, entity_type)
|
||||
attempt = f"web_search:{entity_type}:{search_query}"
|
||||
logger.info_event(
|
||||
"Collecting location search evidence",
|
||||
event="location.factcheck.web_search.start",
|
||||
context={
|
||||
"entity_type": entity_type,
|
||||
"search_query": search_query,
|
||||
"location_query": _query_context(query),
|
||||
"max_results": max_results,
|
||||
},
|
||||
)
|
||||
try:
|
||||
evidence = await web_search_client.search(search_query, max_results=max_results)
|
||||
except WebSearchError as exc:
|
||||
logger.warning_event(
|
||||
"Location search evidence failed",
|
||||
event="location.factcheck.web_search.failed",
|
||||
context={
|
||||
"entity_type": entity_type,
|
||||
"search_query": search_query,
|
||||
"error": str(exc),
|
||||
},
|
||||
)
|
||||
return LocationSearchEvidenceResult(
|
||||
evidence=[],
|
||||
attempted_queries=[attempt],
|
||||
failure_reason=f"WebSearch location evidence failed: {exc}",
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.warning_event(
|
||||
"Location search evidence unavailable",
|
||||
event="location.factcheck.web_search.unavailable",
|
||||
context={
|
||||
"entity_type": entity_type,
|
||||
"search_query": search_query,
|
||||
"error": str(exc),
|
||||
},
|
||||
)
|
||||
return LocationSearchEvidenceResult(
|
||||
evidence=[],
|
||||
attempted_queries=[attempt],
|
||||
@@ -858,11 +988,29 @@ async def collect_location_search_evidence(
|
||||
)
|
||||
normalized = normalize_search_evidence(evidence, limit=max_results)
|
||||
if not normalized:
|
||||
logger.warning_event(
|
||||
"Location search returned no usable evidence",
|
||||
event="location.factcheck.web_search.empty",
|
||||
context={
|
||||
"entity_type": entity_type,
|
||||
"search_query": search_query,
|
||||
},
|
||||
)
|
||||
return LocationSearchEvidenceResult(
|
||||
evidence=[],
|
||||
attempted_queries=[attempt],
|
||||
failure_reason="WebSearch returned no usable location evidence.",
|
||||
)
|
||||
logger.info_event(
|
||||
"Collected location search evidence",
|
||||
event="location.factcheck.web_search.result",
|
||||
context={
|
||||
"entity_type": entity_type,
|
||||
"search_query": search_query,
|
||||
"evidence_count": len(normalized),
|
||||
"evidence": _summarize_search_evidence(normalized),
|
||||
},
|
||||
)
|
||||
return LocationSearchEvidenceResult(
|
||||
evidence=normalized,
|
||||
attempted_queries=[attempt],
|
||||
@@ -876,6 +1024,7 @@ async def _repair_location_payload_from_text(
|
||||
raw_text: str,
|
||||
query: LocationQuery,
|
||||
entity_type: str,
|
||||
db: AsyncSession | None = None,
|
||||
) -> dict[str, Any] | None:
|
||||
"""Second-pass structure repair for models that answer in prose.
|
||||
|
||||
@@ -884,12 +1033,11 @@ async def _repair_location_payload_from_text(
|
||||
"""
|
||||
if not coerce_str(raw_text):
|
||||
return None
|
||||
prompt = await get_effective_prompt(db, LOCATION_NORMALIZE_PROMPT_KEY)
|
||||
request = SituationalAnalysisRequest(
|
||||
title=f"Normalize location factcheck for {entity_type}",
|
||||
objective=(
|
||||
"Convert the supplied location factcheck text into exactly one strict "
|
||||
"JSON object. Extract only facts present in the text or original query."
|
||||
),
|
||||
objective=prompt.prompt,
|
||||
system_prompt=prompt.system_prompt or None,
|
||||
context={
|
||||
"entity_type": entity_type,
|
||||
"location_query": _query_context(query),
|
||||
@@ -929,6 +1077,7 @@ async def collect_llm_location_fallback_candidate(
|
||||
provider_client: AIProviderClient,
|
||||
query: LocationQuery,
|
||||
entity_type: str,
|
||||
db: AsyncSession | None = None,
|
||||
attempted_queries: Iterable[str] = (),
|
||||
search_evidence: list[dict[str, Any]] | None = None,
|
||||
min_confidence: float = DEFAULT_MIN_CONFIDENCE,
|
||||
@@ -941,18 +1090,25 @@ async def collect_llm_location_fallback_candidate(
|
||||
"""
|
||||
attempt = f"llm_factcheck:{entity_type}:{coerce_str(query.name) or 'unknown'}"
|
||||
if search_evidence is not None and not search_evidence:
|
||||
logger.warning_event(
|
||||
"Skipping LLM location factcheck because search evidence is empty",
|
||||
event="location.factcheck.llm.skipped_no_evidence",
|
||||
context={
|
||||
"entity_type": entity_type,
|
||||
"attempt": attempt,
|
||||
"location_query": _query_context(query),
|
||||
},
|
||||
)
|
||||
return LocationLLMFallbackResult(
|
||||
candidates=[],
|
||||
attempted_queries=[attempt],
|
||||
failure_reason="LLM location factcheck skipped: no WebSearch evidence.",
|
||||
)
|
||||
prompt = await get_effective_prompt(db, LOCATION_RESOLVE_PROMPT_KEY)
|
||||
request = SituationalAnalysisRequest(
|
||||
title=f"Location factcheck fallback for {entity_type}",
|
||||
objective=(
|
||||
"Return exactly one JSON object for the most likely physical location. "
|
||||
"Use only fact-checkable public knowledge; return null fields rather "
|
||||
"than guessing when evidence is weak."
|
||||
),
|
||||
objective=prompt.prompt,
|
||||
system_prompt=prompt.system_prompt or None,
|
||||
context={
|
||||
"entity_type": entity_type,
|
||||
"location_query": _query_context(query),
|
||||
@@ -982,31 +1138,86 @@ async def collect_llm_location_fallback_candidate(
|
||||
"Return evidence as objects when possible, including source, url, source_type, and entity_match.",
|
||||
"Include source names or URLs in evidence when known. The backend will recompute the final confidence from model confidence plus evidence quality.",
|
||||
"If search_evidence is provided, use only that evidence as factual support.",
|
||||
"Do not treat a website footer, office address, publisher address, or contact address as the entity's physical location.",
|
||||
"If the entity name contains a city name, do not choose that city unless evidence explicitly says the entity/facility/supercomputer is located, hosted, built, deployed, or installed there.",
|
||||
"Prefer the facility/site if known; otherwise use the best supported city.",
|
||||
],
|
||||
)
|
||||
logger.info_event(
|
||||
"Sending location factcheck request to LLM",
|
||||
event="location.factcheck.llm.request",
|
||||
context={
|
||||
"entity_type": entity_type,
|
||||
"attempt": attempt,
|
||||
"title": request.title,
|
||||
"objective": request.objective,
|
||||
"location_query": request.context.get("location_query"),
|
||||
"observations": request.observations,
|
||||
"constraints": request.constraints,
|
||||
"search_evidence_count": len(search_evidence or []),
|
||||
"search_evidence": _summarize_search_evidence(search_evidence),
|
||||
},
|
||||
)
|
||||
try:
|
||||
response = await provider_client.analyze(request)
|
||||
except Exception as exc:
|
||||
logger.warning_event(
|
||||
"LLM location factcheck failed",
|
||||
event="location.factcheck.llm.failed",
|
||||
context={
|
||||
"entity_type": entity_type,
|
||||
"attempt": attempt,
|
||||
"error": str(exc),
|
||||
},
|
||||
)
|
||||
return LocationLLMFallbackResult(
|
||||
candidates=[],
|
||||
attempted_queries=[attempt],
|
||||
failure_reason=f"LLM location factcheck failed: {exc}",
|
||||
)
|
||||
|
||||
logger.info_event(
|
||||
"Received location factcheck response from LLM",
|
||||
event="location.factcheck.llm.response",
|
||||
context={
|
||||
"entity_type": entity_type,
|
||||
"attempt": attempt,
|
||||
"provider": response.provider,
|
||||
"model": response.model,
|
||||
"content": _truncate_log_text(response.content, 2000),
|
||||
},
|
||||
)
|
||||
payload = _first_json_object(response.content)
|
||||
if payload is None:
|
||||
logger.warning_event(
|
||||
"LLM location factcheck response was not strict JSON; attempting repair",
|
||||
event="location.factcheck.llm.non_json",
|
||||
context={
|
||||
"entity_type": entity_type,
|
||||
"attempt": attempt,
|
||||
"content": _truncate_log_text(response.content, 1200),
|
||||
},
|
||||
)
|
||||
payload = await _repair_location_payload_from_text(
|
||||
provider_client=provider_client,
|
||||
raw_text=response.content,
|
||||
query=query,
|
||||
entity_type=entity_type,
|
||||
db=db,
|
||||
)
|
||||
if payload is None:
|
||||
payload = _payload_from_free_text(response.content, query=query)
|
||||
if payload is None:
|
||||
if payload is None and entity_type != "compute_center":
|
||||
payload = _payload_from_query_name_geocode(query)
|
||||
if payload is None:
|
||||
logger.warning_event(
|
||||
"LLM location factcheck produced no parseable payload",
|
||||
event="location.factcheck.llm.unparseable",
|
||||
context={
|
||||
"entity_type": entity_type,
|
||||
"attempt": attempt,
|
||||
},
|
||||
)
|
||||
return LocationLLMFallbackResult(
|
||||
candidates=[],
|
||||
attempted_queries=[attempt],
|
||||
@@ -1027,7 +1238,7 @@ async def collect_llm_location_fallback_candidate(
|
||||
"url": item.get("url"),
|
||||
"text": item.get("snippet") or item.get("content"),
|
||||
"source_type": "web_search",
|
||||
"entity_match": True,
|
||||
"entity_match": _search_evidence_entity_match(item, query),
|
||||
}
|
||||
for item in search_evidence
|
||||
if isinstance(item, dict)
|
||||
@@ -1049,6 +1260,18 @@ async def collect_llm_location_fallback_candidate(
|
||||
if candidate is None:
|
||||
if city_geocode_failure and rejection_reason == "missing, invalid, or zero latitude/longitude":
|
||||
rejection_reason = f"{rejection_reason}; {city_geocode_failure}"
|
||||
logger.warning_event(
|
||||
"Rejected LLM location factcheck candidate",
|
||||
event="location.factcheck.llm.rejected",
|
||||
context={
|
||||
"entity_type": entity_type,
|
||||
"attempt": attempt,
|
||||
"reason": rejection_reason,
|
||||
"payload": payload,
|
||||
"search_evidence_count": len(search_evidence or []),
|
||||
"search_evidence": _summarize_search_evidence(search_evidence),
|
||||
},
|
||||
)
|
||||
return LocationLLMFallbackResult(
|
||||
candidates=[],
|
||||
attempted_queries=[attempt],
|
||||
@@ -1057,6 +1280,18 @@ async def collect_llm_location_fallback_candidate(
|
||||
+ (f": {rejection_reason}." if rejection_reason else ".")
|
||||
),
|
||||
)
|
||||
logger.info_event(
|
||||
"Accepted LLM location factcheck candidate",
|
||||
event="location.factcheck.llm.accepted",
|
||||
context={
|
||||
"entity_type": entity_type,
|
||||
"attempt": attempt,
|
||||
"candidate": candidate.to_dict(),
|
||||
"payload": payload,
|
||||
"search_evidence_count": len(search_evidence or []),
|
||||
"search_evidence": _summarize_search_evidence(search_evidence),
|
||||
},
|
||||
)
|
||||
return LocationLLMFallbackResult(
|
||||
candidates=[candidate],
|
||||
attempted_queries=[attempt],
|
||||
|
||||
@@ -175,19 +175,29 @@ async def run_collector_task(collector_name: str):
|
||||
)
|
||||
|
||||
try:
|
||||
collector._datasource_id = datasource.id
|
||||
datasource_id = datasource.id
|
||||
datasource_source = datasource.source
|
||||
collector._datasource_id = datasource_id
|
||||
logger.info_event(
|
||||
"Running collector",
|
||||
event="collector.run.started",
|
||||
context={"collector_name": collector_name, "datasource_id": datasource.id},
|
||||
context={"collector_name": collector_name, "datasource_id": datasource_id},
|
||||
)
|
||||
task_result = await collector.run(db)
|
||||
datasource = await db.get(DataSource, datasource_id)
|
||||
if datasource is None:
|
||||
logger.error_event(
|
||||
"Datasource disappeared after collector run",
|
||||
event="collector.run.datasource_missing_after_run",
|
||||
context={"collector_name": collector_name, "datasource_id": datasource_id},
|
||||
)
|
||||
return
|
||||
datasource.last_run_at = datetime.now(UTC)
|
||||
datasource.last_status = task_result.get("status")
|
||||
if datasource.last_status == "success":
|
||||
effective_candidate = await get_builtin_effective_candidate(db, datasource.source)
|
||||
effective_candidate = await get_builtin_effective_candidate(db, datasource_source)
|
||||
checksum, _credential_context = await build_builtin_connectivity_checksum(
|
||||
datasource.source,
|
||||
datasource_source,
|
||||
effective_candidate["endpoint"],
|
||||
effective_candidate["auth_type"],
|
||||
effective_candidate["headers"],
|
||||
@@ -196,7 +206,7 @@ async def run_collector_task(collector_name: str):
|
||||
)
|
||||
await save_connectivity_success(
|
||||
db,
|
||||
datasource.source,
|
||||
datasource_source,
|
||||
checksum,
|
||||
{"status_code": None},
|
||||
connected_by="collection",
|
||||
@@ -205,9 +215,11 @@ async def run_collector_task(collector_name: str):
|
||||
logger.info_event(
|
||||
"Collector completed",
|
||||
event="collector.run.completed",
|
||||
context={"collector_name": collector_name, "datasource_id": datasource.id, "result": task_result},
|
||||
context={"collector_name": collector_name, "datasource_id": datasource_id, "result": task_result},
|
||||
)
|
||||
except asyncio.CancelledError:
|
||||
await db.rollback()
|
||||
datasource = await db.get(DataSource, datasource_id)
|
||||
datasource.last_run_at = datetime.now(UTC)
|
||||
datasource.last_status = "cancelled"
|
||||
await db.commit()
|
||||
@@ -218,6 +230,8 @@ async def run_collector_task(collector_name: str):
|
||||
)
|
||||
raise
|
||||
except Exception as exc:
|
||||
await db.rollback()
|
||||
datasource = await db.get(DataSource, datasource_id)
|
||||
datasource.last_run_at = datetime.now(UTC)
|
||||
datasource.last_status = "failed"
|
||||
await db.commit()
|
||||
@@ -374,6 +388,11 @@ def run_collector_now(collector_name: str) -> bool:
|
||||
return False
|
||||
|
||||
|
||||
def is_collector_running(collector_name: str) -> bool:
|
||||
task = get_running_collector_task(collector_name)
|
||||
return bool(task is not None and not task.done())
|
||||
|
||||
|
||||
async def cancel_running_collector_now(collector_name: str) -> bool:
|
||||
task = get_running_collector_task(collector_name)
|
||||
if task is None or task.done():
|
||||
|
||||
@@ -10,8 +10,11 @@ from app.models.alert import Alert, AlertSeverity, AlertStatus
|
||||
from app.models.bgp_anomaly import BGPAnomaly
|
||||
from app.models.bgp_incident import BGPIncident
|
||||
from app.schemas.ai import SituationalAnalysisRequest
|
||||
from app.ai_tasks.prompts import get_effective_prompt
|
||||
from app.services.bgp_ai_brief_store import get_latest_bgp_brief_record
|
||||
|
||||
SITUATIONAL_ALERT_BRIEF_PROMPT_KEY = "alerts.situational.brief"
|
||||
|
||||
|
||||
def _format_pairs(pairs: list[tuple[str, int]], empty_text: str = "无") -> str:
|
||||
if not pairs:
|
||||
@@ -96,18 +99,24 @@ async def build_situational_alert_brief_request(
|
||||
(str(item[0] or "未命名数据源"), item[1])
|
||||
for item in alert_source_result.fetchall()
|
||||
]
|
||||
total_alerts = total_alerts_result.scalar() or 0
|
||||
active_alerts = active_alerts_result.scalar() or 0
|
||||
total_incidents = total_incidents_result.scalar() or 0
|
||||
active_incidents = active_incidents_result.scalar() or 0
|
||||
total_anomalies = total_anomalies_result.scalar() or 0
|
||||
active_anomalies = active_anomalies_result.scalar() or 0
|
||||
|
||||
facts = [
|
||||
(
|
||||
f"系统告警侧:总告警 {total_alerts_result.scalar() or 0} 条,active {active_alerts_result.scalar() or 0} 条;"
|
||||
f"系统告警侧:总告警 {total_alerts} 条,active {active_alerts} 条;"
|
||||
f"活跃告警严重度分布为 {_format_pairs(active_alert_severities)}。"
|
||||
),
|
||||
(
|
||||
f"BGP态势侧:累计 incidents {total_incidents_result.scalar() or 0} 条,active incidents {active_incidents_result.scalar() or 0} 条;"
|
||||
f"BGP态势侧:累计 incidents {total_incidents} 条,active incidents {active_incidents} 条;"
|
||||
f"活跃 incidents 严重度分布为 {_format_pairs(active_bgp_severities)}。"
|
||||
),
|
||||
(
|
||||
f"BGP异常侧:累计 anomalies {total_anomalies_result.scalar() or 0} 条,active anomalies {active_anomalies_result.scalar() or 0} 条;"
|
||||
f"BGP异常侧:累计 anomalies {total_anomalies} 条,active anomalies {active_anomalies} 条;"
|
||||
f"活跃 anomaly 类型分布为 {_format_pairs(active_anomaly_types)}。"
|
||||
),
|
||||
]
|
||||
@@ -147,21 +156,23 @@ async def build_situational_alert_brief_request(
|
||||
|
||||
context = {
|
||||
"source": "situational-alerts",
|
||||
"active_system_alerts": active_alerts_result.scalar() or 0,
|
||||
"active_system_alerts": active_alerts,
|
||||
"active_system_alert_severities": dict(active_alert_severities),
|
||||
"top_system_alert_sources": dict(active_alert_sources),
|
||||
"active_bgp_incidents": active_incidents_result.scalar() or 0,
|
||||
"active_bgp_incidents": active_incidents,
|
||||
"active_bgp_incident_severities": dict(active_bgp_severities),
|
||||
"active_bgp_anomalies": active_anomalies_result.scalar() or 0,
|
||||
"active_bgp_anomalies": active_anomalies,
|
||||
"active_bgp_anomaly_types": dict(active_anomaly_types),
|
||||
"bgp_hot_regions": dict(bgp_region_counter.most_common(5)),
|
||||
"latest_bgp_brief_id": latest_bgp_brief.id if latest_bgp_brief else None,
|
||||
"latest_bgp_brief_generated_at": latest_bgp_brief.generated_at if latest_bgp_brief else None,
|
||||
}
|
||||
prompt = await get_effective_prompt(db, SITUATIONAL_ALERT_BRIEF_PROMPT_KEY)
|
||||
|
||||
request = SituationalAnalysisRequest(
|
||||
title="态势告警 AI 简报",
|
||||
objective="综合系统告警、BGP incidents、BGP anomalies 与近期 BGP AI 简报,生成一份面向值班人员的态势告警简报,指出当前最需要关注的风险域、跨模块联动迹象和优先动作。",
|
||||
objective=prompt.prompt,
|
||||
system_prompt=prompt.system_prompt or None,
|
||||
observations=facts,
|
||||
constraints=[
|
||||
"明确区分事实、推断与建议。",
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
@@ -86,6 +87,7 @@ class LogSource:
|
||||
status: str = "ok"
|
||||
buffer_key: str | None = None
|
||||
container_name: str | None = None
|
||||
fallback_locations: tuple[str, ...] = ()
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -104,22 +106,38 @@ class DailyLogMarker:
|
||||
dominant_level: str
|
||||
|
||||
|
||||
def _planet_state_dir() -> Path:
|
||||
configured = os.getenv("PLANET_STATE_DIR")
|
||||
if configured:
|
||||
return Path(configured).expanduser()
|
||||
xdg_state = os.getenv("XDG_STATE_HOME")
|
||||
if xdg_state:
|
||||
return Path(xdg_state).expanduser() / "planet"
|
||||
return Path.home() / ".local" / "state" / "planet"
|
||||
|
||||
|
||||
def _state_log_path(filename: str) -> str:
|
||||
return str(_planet_state_dir() / filename)
|
||||
|
||||
|
||||
LOG_SOURCES: dict[str, LogSource] = {
|
||||
"backend": LogSource(
|
||||
source_id="backend",
|
||||
name="后端服务",
|
||||
kind="file",
|
||||
location="/tmp/planet_backend.log",
|
||||
location=_state_log_path("backend.log"),
|
||||
description="FastAPI 后端、调度器和采集任务共享日志。",
|
||||
category="service",
|
||||
fallback_locations=("/tmp/planet_backend.log",),
|
||||
),
|
||||
"frontend": LogSource(
|
||||
source_id="frontend",
|
||||
name="前端开发服务",
|
||||
kind="file",
|
||||
location="/tmp/planet_frontend.log",
|
||||
location=_state_log_path("frontend.log"),
|
||||
description="控制台与 Earth 前端开发服务输出。",
|
||||
category="service",
|
||||
fallback_locations=("/tmp/planet_frontend.log",),
|
||||
),
|
||||
"ai-provider": LogSource(
|
||||
source_id="ai-provider",
|
||||
@@ -164,9 +182,18 @@ def normalize_log_levels(level: str | None = None, levels: str | None = None) ->
|
||||
return tuple(normalized_levels)
|
||||
|
||||
|
||||
def resolve_file_log_path(source: LogSource) -> Path:
|
||||
primary = Path(source.location).expanduser()
|
||||
candidates = (primary, *(Path(item).expanduser() for item in source.fallback_locations))
|
||||
for candidate in candidates:
|
||||
if candidate.exists():
|
||||
return candidate
|
||||
return primary
|
||||
|
||||
|
||||
def get_source_status(source: LogSource) -> str:
|
||||
if source.kind == "file":
|
||||
path = Path(source.location)
|
||||
path = resolve_file_log_path(source)
|
||||
if not path.exists():
|
||||
return "missing"
|
||||
return "ok" if path.stat().st_size > 0 else "empty"
|
||||
@@ -190,7 +217,7 @@ def list_log_sources() -> list[dict[str, str]]:
|
||||
"source_id": source.source_id,
|
||||
"name": source.name,
|
||||
"kind": source.kind,
|
||||
"location": source.location,
|
||||
"location": str(resolve_file_log_path(source)) if source.kind == "file" else source.location,
|
||||
"description": source.description,
|
||||
"category": source.category,
|
||||
"status": get_source_status(source),
|
||||
@@ -339,7 +366,7 @@ def build_buffer_entry(payload: dict[str, Any]) -> StructuredLogEntry:
|
||||
|
||||
|
||||
def read_file_entries(source: LogSource, scan_limit: int) -> list[StructuredLogEntry]:
|
||||
path = Path(source.location)
|
||||
path = resolve_file_log_path(source)
|
||||
if not path.exists():
|
||||
return []
|
||||
with path.open("r", encoding="utf-8", errors="replace") as handle:
|
||||
@@ -511,7 +538,7 @@ def read_log_snapshot(
|
||||
"source_id": source.source_id,
|
||||
"name": source.name,
|
||||
"kind": source.kind,
|
||||
"location": source.location,
|
||||
"location": str(resolve_file_log_path(source)) if source.kind == "file" else source.location,
|
||||
"description": source.description,
|
||||
"category": source.category,
|
||||
"status": get_source_status(source),
|
||||
|
||||
@@ -385,13 +385,18 @@ def build_public_tv_payload(
|
||||
settings_payload: dict[str, Any],
|
||||
collected_sources: list[dict[str, Any]],
|
||||
) -> dict[str, Any]:
|
||||
configured_by_id = {
|
||||
source["id"]: source
|
||||
for source in settings_payload["sources"]
|
||||
if source.get("id")
|
||||
}
|
||||
configured_sources = [
|
||||
source for source in settings_payload["sources"] if source["is_enabled"]
|
||||
]
|
||||
|
||||
merged_by_id = {source["id"]: source for source in configured_sources}
|
||||
for source in collected_sources:
|
||||
if source["id"] in merged_by_id or not source["is_enabled"]:
|
||||
if source["id"] in configured_by_id or not source["is_enabled"]:
|
||||
continue
|
||||
merged_by_id[source["id"]] = source
|
||||
|
||||
|
||||
@@ -5,10 +5,12 @@ from datetime import datetime
|
||||
from unittest.mock import patch, AsyncMock
|
||||
from httpx import AsyncClient, ASGITransport
|
||||
|
||||
from app.api.v1 import earth as earth_api
|
||||
from app.main import app
|
||||
from app.core.config import settings
|
||||
from app.core.security import create_access_token
|
||||
from app.db.session import get_db
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.models.user import User
|
||||
from app.schemas.ai import (
|
||||
AIProviderStatusResponse,
|
||||
@@ -18,6 +20,17 @@ from app.schemas.ai import (
|
||||
)
|
||||
|
||||
|
||||
class _FakeRedisClient:
|
||||
def sismember(self, *_args, **_kwargs):
|
||||
return False
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def fake_token_blacklist(monkeypatch):
|
||||
"""Keep API auth tests independent from an external Redis service."""
|
||||
monkeypatch.setattr("app.core.security.redis_client", _FakeRedisClient())
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def auth_headers():
|
||||
"""Create authentication headers"""
|
||||
@@ -50,6 +63,148 @@ async def test_root_endpoint():
|
||||
assert data["version"] == settings.VERSION
|
||||
|
||||
|
||||
class _ScalarOneOrNoneResult:
|
||||
def __init__(self, value=None):
|
||||
self._value = value
|
||||
|
||||
def scalar_one_or_none(self):
|
||||
return self._value
|
||||
|
||||
|
||||
class _FakeEarthBrandSession:
|
||||
def __init__(self, record=None):
|
||||
self.record = record
|
||||
self.added = None
|
||||
self.deleted = False
|
||||
self.committed = False
|
||||
|
||||
async def execute(self, statement):
|
||||
if statement.__class__.__name__ == "Delete":
|
||||
self.deleted = True
|
||||
self.record = None
|
||||
return _ScalarOneOrNoneResult(None)
|
||||
return _ScalarOneOrNoneResult(self.record)
|
||||
|
||||
def add(self, record):
|
||||
self.added = record
|
||||
self.record = record
|
||||
|
||||
async def commit(self):
|
||||
self.committed = True
|
||||
|
||||
async def refresh(self, _record):
|
||||
return None
|
||||
|
||||
|
||||
def _override_admin_user():
|
||||
return User(id=1, username="testuser", email="test@example.com", role="admin", is_active=True)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_earth_brand_returns_static_defaults():
|
||||
async def override_get_db():
|
||||
yield _FakeEarthBrandSession()
|
||||
|
||||
app.dependency_overrides[get_db] = override_get_db
|
||||
try:
|
||||
transport = ASGITransport(app=app)
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/api/v1/earth/brand")
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["is_default"] is True
|
||||
assert data["brand"]["logo_src"] == "/earth/assets/brand/earth-logo.png"
|
||||
assert data["brand"]["title_src"] == "/earth/assets/brand/title-zh.png"
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_update_and_reset_earth_brand(auth_headers):
|
||||
session = _FakeEarthBrandSession()
|
||||
|
||||
async def override_get_db():
|
||||
yield session
|
||||
|
||||
app.dependency_overrides.update(
|
||||
{
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: _override_admin_user,
|
||||
get_db: override_get_db,
|
||||
}
|
||||
)
|
||||
try:
|
||||
transport = ASGITransport(app=app)
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
update_response = await client.put(
|
||||
"/api/v1/earth/brand",
|
||||
headers=auth_headers,
|
||||
json={
|
||||
"logo_src": "/earth-brand-assets/custom.png",
|
||||
"title_src": "",
|
||||
"title_text": "Custom Earth",
|
||||
"subtitle": "Custom subtitle",
|
||||
"description": "Custom description",
|
||||
"aria_label": "",
|
||||
"title_alt": "",
|
||||
},
|
||||
)
|
||||
reset_response = await client.delete("/api/v1/earth/brand", headers=auth_headers)
|
||||
|
||||
assert update_response.status_code == 200
|
||||
updated = update_response.json()
|
||||
assert updated["is_default"] is False
|
||||
assert updated["brand"]["title_text"] == "Custom Earth"
|
||||
assert updated["brand"]["aria_label"] == "Custom Earth"
|
||||
assert isinstance(session.added, SystemSetting)
|
||||
assert reset_response.status_code == 200
|
||||
reset = reset_response.json()
|
||||
assert reset["is_default"] is True
|
||||
assert reset["brand"]["logo_src"] == "/earth/assets/brand/earth-logo.png"
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_upload_earth_brand_asset_rejects_invalid_type(auth_headers):
|
||||
app.dependency_overrides[
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user
|
||||
] = _override_admin_user
|
||||
try:
|
||||
transport = ASGITransport(app=app)
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.post(
|
||||
"/api/v1/earth/brand/assets",
|
||||
headers=auth_headers,
|
||||
files={"file": ("brand.txt", b"nope", "text/plain")},
|
||||
)
|
||||
assert response.status_code == 400
|
||||
assert response.json()["detail"]["code"] == "unsupported_file_type"
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_upload_earth_brand_asset_saves_file(auth_headers, tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(earth_api, "EARTH_BRAND_ASSET_DIR", tmp_path)
|
||||
app.dependency_overrides[
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user
|
||||
] = _override_admin_user
|
||||
try:
|
||||
transport = ASGITransport(app=app)
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.post(
|
||||
"/api/v1/earth/brand/assets",
|
||||
headers=auth_headers,
|
||||
files={"file": ("brand.png", b"png-bytes", "image/png")},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["url"].startswith("/earth-brand-assets/")
|
||||
assert (tmp_path / data["filename"]).read_bytes() == b"png-bytes"
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_dashboard_stats_without_auth():
|
||||
"""Test dashboard stats requires authentication"""
|
||||
@@ -62,20 +217,43 @@ async def test_dashboard_stats_without_auth():
|
||||
@pytest.mark.asyncio
|
||||
async def test_dashboard_stats_with_auth(auth_headers):
|
||||
"""Test dashboard stats with authentication"""
|
||||
with patch("app.api.v1.dashboard.cache.get", return_value=None):
|
||||
with patch("app.api.v1.dashboard.cache.set", return_value=True):
|
||||
with patch("app.db.session.get_db") as mock_get_db:
|
||||
mock_session = AsyncMock()
|
||||
mock_result = AsyncMock()
|
||||
mock_result.scalar.return_value = 0
|
||||
mock_result.fetchall.return_value = []
|
||||
mock_session.execute.return_value = mock_result
|
||||
class _StatsResult:
|
||||
def __init__(self, row):
|
||||
self._row = row
|
||||
|
||||
async def mock_db_context():
|
||||
yield mock_session
|
||||
def one(self):
|
||||
return self._row
|
||||
|
||||
mock_get_db.return_value = mock_db_context()
|
||||
class _FakeStatsSession:
|
||||
def __init__(self):
|
||||
self._rows = [
|
||||
type("DatasourceStats", (), {"custom_count": 0, "custom_active": 0})(),
|
||||
type("TaskStats", (), {"tasks_today": 0, "success_tasks": 0})(),
|
||||
type(
|
||||
"AlertStats",
|
||||
(),
|
||||
{"critical_alerts": 0, "warning_alerts": 0, "info_alerts": 0},
|
||||
)(),
|
||||
]
|
||||
|
||||
async def execute(self, _query):
|
||||
return _StatsResult(self._rows.pop(0))
|
||||
|
||||
def override_get_current_user():
|
||||
return User(id=1, username="testuser", email="test@example.com", role="admin", is_active=True)
|
||||
|
||||
async def override_get_db():
|
||||
yield _FakeStatsSession()
|
||||
|
||||
app.dependency_overrides.update(
|
||||
{
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
get_db: override_get_db,
|
||||
}
|
||||
)
|
||||
try:
|
||||
with patch("app.api.v1.dashboard.cache.get", return_value=None):
|
||||
with patch("app.api.v1.dashboard.cache.set", return_value=True):
|
||||
transport = ASGITransport(app=app)
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get(
|
||||
@@ -85,6 +263,8 @@ async def test_dashboard_stats_with_auth(auth_headers):
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert "total_datasources" in data
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@@ -492,6 +672,77 @@ async def test_ingest_earth_client_log_accepts_public_events():
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_earth_layer_cache_status_requires_super_admin(auth_headers, monkeypatch):
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="root",
|
||||
email="root@example.com",
|
||||
password_hash="hashed",
|
||||
role="super_admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
monkeypatch.setattr(
|
||||
"app.api.v1.system_control.earth_layer_cache.status",
|
||||
lambda: {
|
||||
"prefix": "earth:layer:v1",
|
||||
"key_count": 2,
|
||||
"memory_bytes": 42,
|
||||
"layers": {"cables": {"keys": 2, "stale_keys": 1, "memory_bytes": 42}},
|
||||
},
|
||||
)
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/api/v1/system/cache/earth-layers", headers=auth_headers)
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["prefix"] == "earth:layer:v1"
|
||||
assert data["layers"]["cables"]["stale_keys"] == 1
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_clear_earth_layer_cache_deletes_only_earth_layer_prefix(auth_headers, monkeypatch):
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="root",
|
||||
email="root@example.com",
|
||||
password_hash="hashed",
|
||||
role="super_admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
captured = {}
|
||||
|
||||
def fake_delete_pattern(pattern="earth:layer:v1:*"):
|
||||
captured["pattern"] = pattern
|
||||
return 3
|
||||
|
||||
monkeypatch.setattr("app.api.v1.system_control.earth_layer_cache.delete_pattern", fake_delete_pattern)
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.delete("/api/v1/system/cache/earth-layers", headers=auth_headers)
|
||||
assert response.status_code == 200
|
||||
assert response.json()["deleted"] == 3
|
||||
assert captured["pattern"] == "earth:layer:v1:*"
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_request_id_header_is_echoed_when_provided():
|
||||
transport = ASGITransport(app=app)
|
||||
|
||||
@@ -4,6 +4,7 @@ import pytest
|
||||
|
||||
from app.api.v1 import datasources as datasources_api
|
||||
from app.models.datasource import DataSource
|
||||
from app.services import earth_layer_cache as earth_cache
|
||||
|
||||
|
||||
def make_datasource(
|
||||
@@ -53,6 +54,71 @@ def test_filter_datasources_by_product_status_and_collected_state():
|
||||
assert filtered == [vessels]
|
||||
|
||||
|
||||
def test_serialize_datasource_row_can_skip_endpoint_resolution():
|
||||
datasource = make_datasource(7, "arcgis_cables", last_status="success", module="L2")
|
||||
|
||||
class ExplodingConfig:
|
||||
def get_yaml_url(self, _source):
|
||||
raise AssertionError("endpoint resolution should be skipped")
|
||||
|
||||
row = datasources_api.serialize_datasource_row(
|
||||
datasource,
|
||||
running_tasks={},
|
||||
latest_tasks={},
|
||||
record_counts={"arcgis_cables": 343},
|
||||
endpoint_overrides={},
|
||||
config=ExplodingConfig(),
|
||||
include_endpoint=False,
|
||||
)
|
||||
|
||||
assert row["id"] == 7
|
||||
assert row["source"] == "arcgis_cables"
|
||||
assert row["module"] == "L2"
|
||||
assert row["last_status"] == "success"
|
||||
assert row["collected_records"] == 343
|
||||
assert row["has_collected_data"] is True
|
||||
assert "endpoint" not in row
|
||||
|
||||
|
||||
def test_serialize_datasource_row_includes_endpoint_when_requested():
|
||||
datasource = make_datasource(8, "arcgis_landing_points")
|
||||
|
||||
class Config:
|
||||
def get_yaml_url(self, source):
|
||||
return f"https://example.test/{source}"
|
||||
|
||||
row = datasources_api.serialize_datasource_row(
|
||||
datasource,
|
||||
running_tasks={},
|
||||
latest_tasks={},
|
||||
record_counts={},
|
||||
endpoint_overrides={},
|
||||
config=Config(),
|
||||
include_endpoint=True,
|
||||
)
|
||||
|
||||
assert row["endpoint"] == "https://example.test/arcgis_landing_points"
|
||||
|
||||
|
||||
def test_invalidate_earth_layer_cache_for_source_covers_datasource_aliases(monkeypatch):
|
||||
patterns: list[str] = []
|
||||
|
||||
def fake_delete_pattern(pattern: str) -> int:
|
||||
patterns.append(pattern)
|
||||
return 1
|
||||
|
||||
monkeypatch.setattr(earth_cache.earth_layer_cache, "delete_pattern", fake_delete_pattern)
|
||||
|
||||
deleted = earth_cache.invalidate_earth_layer_cache_for_source("telegeography_cables")
|
||||
|
||||
assert deleted == 3
|
||||
assert patterns == [
|
||||
"earth:layer:v1:cables*",
|
||||
"earth:layer:v1:landing-points*",
|
||||
"earth:layer:v1:summary*",
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_trigger_datasource_batch_skips_disabled_and_frequency_window(monkeypatch):
|
||||
now = datetime.now(timezone.utc)
|
||||
|
||||
177
backend/tests/test_earth_boundaries.py
Normal file
177
backend/tests/test_earth_boundaries.py
Normal file
@@ -0,0 +1,177 @@
|
||||
import json
|
||||
|
||||
import pytest
|
||||
|
||||
from app.core.datasource_defaults import DEFAULT_DATASOURCES
|
||||
from app.services.collectors.registry import collector_registry
|
||||
from app.services import earth_boundaries
|
||||
|
||||
|
||||
def write_geojson(path, name="Test"):
|
||||
path.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"type": "FeatureCollection",
|
||||
"features": [
|
||||
{
|
||||
"type": "Feature",
|
||||
"properties": {"name": name},
|
||||
"geometry": {
|
||||
"type": "Polygon",
|
||||
"coordinates": [[[-1, -1], [1, -1], [1, 1], [-1, 1], [-1, -1]]],
|
||||
},
|
||||
}
|
||||
],
|
||||
}
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
|
||||
def patch_paths(monkeypatch, tmp_path):
|
||||
repo = tmp_path
|
||||
source_dir = repo / "data/earth-boundary-sources"
|
||||
boundary_dir = repo / "frontend/public/earth/data/boundaries/v1"
|
||||
pmtiles = repo / "frontend/public/earth/data/boundaries/earth-boundaries-china-pov-v1.pmtiles"
|
||||
legacy = repo / "frontend/public/earth/data/countries-admin0.min.geojson"
|
||||
config = repo / "config/earth-boundary-sources.local.json"
|
||||
example = repo / "config/earth-boundary-sources.example.json"
|
||||
policy = repo / "config/earth-boundary-pov-policy.china-v1.json"
|
||||
for path in (source_dir, boundary_dir, pmtiles.parent, legacy.parent, config.parent):
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
policy.write_text('{"productionTileFormat":"pmtiles+mvt"}\n', encoding="utf-8")
|
||||
example.write_text('{"collectorConfigs":{}}\n', encoding="utf-8")
|
||||
monkeypatch.setattr(earth_boundaries, "REPO_ROOT", repo)
|
||||
monkeypatch.setattr(earth_boundaries, "SOURCE_OUTPUT_DIR", source_dir)
|
||||
monkeypatch.setattr(earth_boundaries, "SOURCE_MANIFEST_PATH", source_dir / "manifest.json")
|
||||
monkeypatch.setattr(earth_boundaries, "BUILD_RESULT_PATH", source_dir / "build-result.json")
|
||||
monkeypatch.setattr(earth_boundaries, "BUILD_JOB_PATH", source_dir / "build-job.json")
|
||||
monkeypatch.setattr(earth_boundaries, "BOUNDARY_OUTPUT_DIR", boundary_dir)
|
||||
monkeypatch.setattr(earth_boundaries, "BOUNDARY_MANIFEST_PATH", boundary_dir / "manifest.json")
|
||||
monkeypatch.setattr(earth_boundaries, "PMTILES_ARTIFACT_PATH", pmtiles)
|
||||
monkeypatch.setattr(earth_boundaries, "LEGACY_GEOJSON_PATH", legacy)
|
||||
monkeypatch.setattr(earth_boundaries, "LOCAL_CONFIG_PATH", config)
|
||||
monkeypatch.setattr(earth_boundaries, "EXAMPLE_CONFIG_PATH", example)
|
||||
monkeypatch.setattr(earth_boundaries, "POV_POLICY_PATH", policy)
|
||||
return {
|
||||
"repo": repo,
|
||||
"config": config,
|
||||
"legacy": legacy,
|
||||
"pmtiles": pmtiles,
|
||||
"manifest": boundary_dir / "manifest.json",
|
||||
}
|
||||
|
||||
|
||||
def test_boundary_status_uses_legacy_provider_when_pmtiles_missing(monkeypatch, tmp_path):
|
||||
paths = patch_paths(monkeypatch, tmp_path)
|
||||
write_geojson(paths["legacy"])
|
||||
|
||||
status = earth_boundaries.get_boundary_status()
|
||||
|
||||
assert status["provider"] == "legacy-geojson"
|
||||
assert status["fallback_available"] is True
|
||||
assert status["high_precision_ready"] is False
|
||||
|
||||
|
||||
def test_boundary_status_prefers_high_precision_when_manifest_and_pmtiles_exist(monkeypatch, tmp_path):
|
||||
paths = patch_paths(monkeypatch, tmp_path)
|
||||
write_geojson(paths["legacy"])
|
||||
paths["pmtiles"].write_bytes(b"pmtiles")
|
||||
paths["manifest"].write_text('{"tileProvider":"pmtiles-mvt"}\n', encoding="utf-8")
|
||||
|
||||
status = earth_boundaries.get_boundary_status()
|
||||
|
||||
assert status["provider"] == "pmtiles-mvt"
|
||||
assert status["high_precision_ready"] is True
|
||||
|
||||
|
||||
def test_save_boundary_config_writes_local_config(monkeypatch, tmp_path):
|
||||
paths = patch_paths(monkeypatch, tmp_path)
|
||||
payload = {"collectorConfigs": {"earth_admin0_boundaries": {"endpoint": "file:///tmp/a.geojson"}}}
|
||||
|
||||
status = earth_boundaries.save_boundary_config(payload)
|
||||
|
||||
assert paths["config"].exists()
|
||||
assert status["config_source"] == "local"
|
||||
assert status["config"] == payload
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_build_reports_missing_tools_after_source_artifacts(monkeypatch, tmp_path):
|
||||
paths = patch_paths(monkeypatch, tmp_path)
|
||||
source_files = {}
|
||||
for source in earth_boundaries.BOUNDARY_SOURCE_KINDS:
|
||||
source_path = paths["repo"] / f"{source}.geojson"
|
||||
write_geojson(source_path, name=source)
|
||||
source_files[source] = source_path
|
||||
paths["config"].write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"collectorConfigs": {
|
||||
source: {
|
||||
"sourceKind": kind,
|
||||
"endpoint": str(source_files[source]),
|
||||
"method": "GET",
|
||||
}
|
||||
for source, kind in earth_boundaries.BOUNDARY_SOURCE_KINDS.items()
|
||||
}
|
||||
}
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
monkeypatch.setattr(earth_boundaries.shutil, "which", lambda _tool: None)
|
||||
progress_events = []
|
||||
|
||||
status = await earth_boundaries.build_boundary_assets(
|
||||
lambda progress, phase, message, **_extra: progress_events.append((progress, phase, message))
|
||||
)
|
||||
|
||||
assert status["provider"] == "geojson-high-precision"
|
||||
assert status["high_precision_ready"] is True
|
||||
assert (paths["repo"] / "data/earth-boundary-sources/manifest.json").exists()
|
||||
assert paths["manifest"].exists()
|
||||
assert any(phase == "download" for _progress, phase, _message in progress_events)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_start_boundary_build_job_records_geojson_fallback_success(monkeypatch, tmp_path):
|
||||
paths = patch_paths(monkeypatch, tmp_path)
|
||||
monkeypatch.setattr(earth_boundaries, "_build_task", None)
|
||||
monkeypatch.setattr(earth_boundaries, "_build_job_state", {})
|
||||
source_files = {}
|
||||
for source in earth_boundaries.BOUNDARY_SOURCE_KINDS:
|
||||
source_path = paths["repo"] / f"{source}.geojson"
|
||||
write_geojson(source_path, name=source)
|
||||
source_files[source] = source_path
|
||||
paths["config"].write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"collectorConfigs": {
|
||||
source: {
|
||||
"sourceKind": kind,
|
||||
"endpoint": str(source_files[source]),
|
||||
"method": "GET",
|
||||
}
|
||||
for source, kind in earth_boundaries.BOUNDARY_SOURCE_KINDS.items()
|
||||
}
|
||||
}
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
monkeypatch.setattr(earth_boundaries.shutil, "which", lambda _tool: None)
|
||||
|
||||
response = await earth_boundaries.start_boundary_build_job()
|
||||
await earth_boundaries._build_task
|
||||
status = earth_boundaries.get_boundary_build_status()
|
||||
|
||||
assert response["accepted"] is True
|
||||
assert status["job"]["status"] == "succeeded"
|
||||
assert status["job"]["result"]["provider"] == "geojson-high-precision"
|
||||
|
||||
|
||||
def test_earth_boundary_collectors_are_not_registered_as_datasources():
|
||||
removed = set(earth_boundaries.BOUNDARY_SOURCE_KINDS) | {"earth_boundary_tiles"}
|
||||
|
||||
assert removed.isdisjoint(DEFAULT_DATASOURCES)
|
||||
for source in removed:
|
||||
assert collector_registry.get(source) is None
|
||||
254
backend/tests/test_earth_layer_cache.py
Normal file
254
backend/tests/test_earth_layer_cache.py
Normal file
@@ -0,0 +1,254 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
|
||||
import pytest
|
||||
from fastapi import Response
|
||||
|
||||
from app.api.v1 import visualization
|
||||
from app.services.earth_layer_cache import (
|
||||
EarthLayerCachePolicy,
|
||||
apply_payload_budget,
|
||||
earth_layer_cache,
|
||||
format_bbox_key,
|
||||
quantize_bbox,
|
||||
resolve_layer_payload,
|
||||
)
|
||||
|
||||
|
||||
class FakeRedis:
|
||||
def __init__(self, *, fail: bool = False) -> None:
|
||||
self.store: dict[str, str] = {}
|
||||
self.fail = fail
|
||||
self.lock_claimed = False
|
||||
|
||||
def _maybe_fail(self) -> None:
|
||||
if self.fail:
|
||||
raise RuntimeError("redis unavailable")
|
||||
|
||||
def get(self, key: str):
|
||||
self._maybe_fail()
|
||||
return self.store.get(key)
|
||||
|
||||
def set(self, key: str, value: str, nx: bool = False, ex: int | None = None):
|
||||
self._maybe_fail()
|
||||
if nx and key in self.store:
|
||||
return False
|
||||
self.store[key] = value
|
||||
return True
|
||||
|
||||
def setex(self, key: str, _seconds: int, value: str):
|
||||
self._maybe_fail()
|
||||
self.store[key] = value
|
||||
return True
|
||||
|
||||
def delete(self, *keys: str):
|
||||
self._maybe_fail()
|
||||
deleted = 0
|
||||
for key in keys:
|
||||
deleted += 1 if self.store.pop(key, None) is not None else 0
|
||||
return deleted
|
||||
|
||||
def scan_iter(self, match: str):
|
||||
self._maybe_fail()
|
||||
prefix = match.rstrip("*")
|
||||
for key in list(self.store):
|
||||
if key.startswith(prefix):
|
||||
yield key
|
||||
|
||||
def memory_usage(self, key: str):
|
||||
value = self.store.get(key, "")
|
||||
return len(value.encode("utf-8"))
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def fake_cache_client():
|
||||
previous = earth_layer_cache._client
|
||||
fake = FakeRedis()
|
||||
earth_layer_cache._client = fake
|
||||
try:
|
||||
yield fake
|
||||
finally:
|
||||
earth_layer_cache._client = previous
|
||||
|
||||
|
||||
def test_quantized_bbox_key_is_stable_for_small_movements():
|
||||
first = format_bbox_key(quantize_bbox((10.01, 59.04, 10.96, 60.02)))
|
||||
second = format_bbox_key(quantize_bbox((10.04, 59.01, 10.99, 60.04)))
|
||||
|
||||
assert first == second
|
||||
assert first == "10.0,59.0,11.0,60.0"
|
||||
|
||||
|
||||
def test_payload_budget_truncates_features():
|
||||
payload = {
|
||||
"type": "FeatureCollection",
|
||||
"features": [{"id": index} for index in range(5)],
|
||||
}
|
||||
policy = EarthLayerCachePolicy(60, 120, max_features=2, max_bytes=1024)
|
||||
|
||||
result = apply_payload_budget(payload, policy)
|
||||
|
||||
assert len(result["features"]) == 2
|
||||
assert result["diagnostics"]["truncated"] is True
|
||||
assert result["diagnostics"]["limit_reason"] == "feature_budget"
|
||||
assert result["diagnostics"]["original_feature_count"] == 5
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_resolve_layer_payload_writes_fresh_and_stale(fake_cache_client):
|
||||
calls = 0
|
||||
|
||||
async def builder():
|
||||
nonlocal calls
|
||||
calls += 1
|
||||
return {"type": "FeatureCollection", "features": [{"id": "a"}]}
|
||||
|
||||
key = earth_layer_cache.key("satellites", limit="all")
|
||||
policy = EarthLayerCachePolicy(60, 120)
|
||||
|
||||
first = await resolve_layer_payload(key=key, policy=policy, builder=builder)
|
||||
second = await resolve_layer_payload(key=key, policy=policy, builder=builder)
|
||||
|
||||
assert first.state == "refresh"
|
||||
assert second.state == "hit"
|
||||
assert calls == 1
|
||||
assert key in fake_cache_client.store
|
||||
assert f"{key}:stale" in fake_cache_client.store
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_resolve_layer_payload_returns_stale_when_builder_fails(fake_cache_client):
|
||||
key = earth_layer_cache.key("bgp-incidents", status="active")
|
||||
fake_cache_client.store[f"{key}:stale"] = json.dumps({"type": "FeatureCollection", "features": []})
|
||||
|
||||
async def builder():
|
||||
raise RuntimeError("db exploded")
|
||||
|
||||
result = await resolve_layer_payload(
|
||||
key=key,
|
||||
policy=EarthLayerCachePolicy(60, 120),
|
||||
builder=builder,
|
||||
)
|
||||
|
||||
assert result.state == "stale"
|
||||
assert result.payload["features"] == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_resolve_layer_payload_uses_stale_during_lock_contention(fake_cache_client):
|
||||
key = earth_layer_cache.key("cables")
|
||||
fake_cache_client.store[earth_layer_cache.lock_key(key)] = "1"
|
||||
fake_cache_client.store[f"{key}:stale"] = json.dumps(
|
||||
{"type": "FeatureCollection", "features": [{"id": "stale-cable"}]}
|
||||
)
|
||||
calls = 0
|
||||
|
||||
async def builder():
|
||||
nonlocal calls
|
||||
calls += 1
|
||||
return {"type": "FeatureCollection", "features": [{"id": "fresh-cable"}]}
|
||||
|
||||
result = await resolve_layer_payload(
|
||||
key=key,
|
||||
policy=EarthLayerCachePolicy(60, 120),
|
||||
builder=builder,
|
||||
)
|
||||
|
||||
assert result.state == "stale"
|
||||
assert result.payload["features"][0]["id"] == "stale-cable"
|
||||
assert calls == 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_resolve_layer_payload_bypasses_redis_failure():
|
||||
previous = earth_layer_cache._client
|
||||
earth_layer_cache._client = FakeRedis(fail=True)
|
||||
try:
|
||||
async def builder():
|
||||
return {"type": "FeatureCollection", "features": [{"id": "safe"}]}
|
||||
|
||||
result = await resolve_layer_payload(
|
||||
key=earth_layer_cache.key("cables"),
|
||||
policy=EarthLayerCachePolicy(60, 120),
|
||||
builder=builder,
|
||||
)
|
||||
|
||||
assert result.state == "bypass"
|
||||
assert result.payload["features"][0]["id"] == "safe"
|
||||
finally:
|
||||
earth_layer_cache._client = previous
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_visualization_endpoint_sets_cache_headers(fake_cache_client, monkeypatch):
|
||||
calls = 0
|
||||
|
||||
async def fake_build_satellites_geojson(*, limit, db):
|
||||
nonlocal calls
|
||||
calls += 1
|
||||
return {"type": "FeatureCollection", "features": [{"id": f"sat-{limit}"}], "count": 1}
|
||||
|
||||
monkeypatch.setattr(visualization, "_build_satellites_geojson", fake_build_satellites_geojson)
|
||||
|
||||
first_response = Response()
|
||||
first = await visualization.get_satellites_geojson(limit=25, db=object(), response=first_response)
|
||||
second_response = Response()
|
||||
second = await visualization.get_satellites_geojson(limit=25, db=object(), response=second_response)
|
||||
|
||||
assert first == second
|
||||
assert calls == 1
|
||||
assert first_response.headers["X-Planet-Cache"] == "refresh"
|
||||
assert second_response.headers["X-Planet-Cache"] == "hit"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_vessel_snapshot_uses_short_cache(fake_cache_client, monkeypatch):
|
||||
calls = 0
|
||||
|
||||
async def fake_load_raw_vessel_snapshot_features(db, *, bbox, limit, observed_since):
|
||||
nonlocal calls
|
||||
calls += 1
|
||||
return (
|
||||
[
|
||||
{
|
||||
"type": "Feature",
|
||||
"geometry": {"type": "Point", "coordinates": [10.1, 59.1]},
|
||||
"properties": {"mmsi": 123, "vessel_type_name": "Cargo"},
|
||||
}
|
||||
],
|
||||
{"raw_feature_count": 1},
|
||||
)
|
||||
|
||||
monkeypatch.setattr(
|
||||
visualization,
|
||||
"_load_raw_vessel_snapshot_features",
|
||||
fake_load_raw_vessel_snapshot_features,
|
||||
)
|
||||
|
||||
first_response = Response()
|
||||
first = await visualization.build_vessel_snapshot_response(
|
||||
object(),
|
||||
bbox=(10.01, 59.04, 10.96, 60.02),
|
||||
zoom=12,
|
||||
type_filter=None,
|
||||
limit=1000,
|
||||
since_minutes=60,
|
||||
response=first_response,
|
||||
)
|
||||
second_response = Response()
|
||||
second = await visualization.build_vessel_snapshot_response(
|
||||
object(),
|
||||
bbox=(10.04, 59.01, 10.99, 60.04),
|
||||
zoom=12,
|
||||
type_filter=None,
|
||||
limit=1000,
|
||||
since_minutes=60,
|
||||
response=second_response,
|
||||
)
|
||||
|
||||
assert first["count"] == 1
|
||||
assert second == first
|
||||
assert calls == 1
|
||||
assert first_response.headers["X-Planet-Cache"] == "refresh"
|
||||
assert second_response.headers["X-Planet-Cache"] == "hit"
|
||||
@@ -1,6 +1,21 @@
|
||||
from datetime import UTC, datetime
|
||||
from types import SimpleNamespace
|
||||
|
||||
from app.services.earth_news import ParsedNewsItem, _serialize_item
|
||||
import pytest
|
||||
|
||||
from app.services.earth_news import (
|
||||
NewsFeedSource,
|
||||
NewsTargetLocation,
|
||||
ParsedNewsItem,
|
||||
_enrich_items_with_target_locations,
|
||||
_extract_target_location_from_text,
|
||||
_parse_feed_entries,
|
||||
_serialize_item,
|
||||
get_earth_news_payload,
|
||||
)
|
||||
from app.services.earth_news_queue import NewsTargetLocationMessage
|
||||
from app.services.earth_news_worker import process_target_location_message
|
||||
from app.services.collectors.media_news_archive import MediaNewsArchiveCollector
|
||||
|
||||
|
||||
def test_serialize_item_includes_region_anchor_for_cruise():
|
||||
@@ -21,7 +36,10 @@ def test_serialize_item_includes_region_anchor_for_cruise():
|
||||
assert payload["latitude"] == 1.3521
|
||||
assert payload["longitude"] == 103.8198
|
||||
assert payload["location_label"] == "亚太"
|
||||
assert payload["location_inferred"] is True
|
||||
assert payload["location_source"] == "region_anchor"
|
||||
assert payload["verified"] is False
|
||||
assert payload["location_meta"]["target"] is None
|
||||
assert payload["location_meta"]["anchor"]["region"] == "asia-pacific"
|
||||
assert payload["is_focus_match"] is True
|
||||
assert payload["published_at"] == "2026-04-23T02:30:00Z"
|
||||
|
||||
@@ -44,6 +62,741 @@ def test_serialize_item_falls_back_to_global_anchor():
|
||||
assert payload["latitude"] == 20.0
|
||||
assert payload["longitude"] == 0.0
|
||||
assert payload["location_label"] == "全球"
|
||||
assert payload["location_inferred"] is True
|
||||
assert payload["location_source"] == "region_anchor"
|
||||
assert payload["verified"] is False
|
||||
assert payload["location_meta"]["anchor"]["region"] == "global"
|
||||
assert payload["is_focus_match"] is False
|
||||
assert payload["published_at"] is None
|
||||
|
||||
|
||||
def test_serialize_item_includes_inferred_target_location():
|
||||
item = ParsedNewsItem(
|
||||
id="bbc-world:f55310fb667b",
|
||||
title="Watch: What happened on day one of Trump's China visit?",
|
||||
summary=(
|
||||
"China welcomed US President Donald Trump with cheering children "
|
||||
"and a troop parade."
|
||||
),
|
||||
url="https://example.com/china-visit",
|
||||
source="BBC World",
|
||||
feed_name="BBC World",
|
||||
feed_region="global",
|
||||
homepage_url="https://example.com",
|
||||
published_at=datetime(2026, 5, 14, 7, 13, 24, tzinfo=UTC),
|
||||
target_location=NewsTargetLocation(
|
||||
latitude=39.9042,
|
||||
longitude=116.4074,
|
||||
label="Beijing, China",
|
||||
source="ai_inferred_target",
|
||||
confidence=0.88,
|
||||
country="中国",
|
||||
city="Beijing",
|
||||
),
|
||||
)
|
||||
|
||||
payload = _serialize_item(item, active_region="global")
|
||||
|
||||
assert payload["latitude"] == 39.9042
|
||||
assert payload["longitude"] == 116.4074
|
||||
assert payload["location_label"] == "Beijing, China"
|
||||
assert payload["location_source"] == "ai_inferred_target"
|
||||
assert payload["verified"] is True
|
||||
assert payload["location_meta"]["target"]["confidence"] == 0.88
|
||||
assert payload["location_meta"]["target"]["country"] == "中国"
|
||||
assert payload["location_meta"]["target"]["city"] == "Beijing"
|
||||
assert payload["location_meta"]["resolution_stage"] == "unresolved"
|
||||
assert payload["location_meta"]["ai_attempted"] is False
|
||||
assert payload["location_meta"]["ai_status"] == "not_attempted"
|
||||
assert payload["location_meta"]["ai_error"] is None
|
||||
|
||||
|
||||
def test_parse_plain_rss_uses_feed_name_as_source():
|
||||
source = NewsFeedSource(
|
||||
id="bbc-world",
|
||||
name="BBC World",
|
||||
region="global",
|
||||
feed_url="https://feeds.bbci.co.uk/news/world/rss.xml",
|
||||
homepage_url="https://www.bbc.com/news/world",
|
||||
)
|
||||
xml = """
|
||||
<rss>
|
||||
<channel>
|
||||
<item>
|
||||
<title>This may be the last time you hear my voice: Political executions surge in Iran since start of war</title>
|
||||
<description>Story summary</description>
|
||||
<link>https://www.bbc.com/news/example</link>
|
||||
<pubDate>Fri, 15 May 2026 03:00:00 GMT</pubDate>
|
||||
</item>
|
||||
</channel>
|
||||
</rss>
|
||||
"""
|
||||
|
||||
items = _parse_feed_entries(xml, source)
|
||||
|
||||
assert items[0].title == "This may be the last time you hear my voice: Political executions surge in Iran since start of war"
|
||||
assert items[0].source == "BBC World"
|
||||
|
||||
|
||||
def test_parse_aggregated_rss_splits_publisher_from_title():
|
||||
source = NewsFeedSource(
|
||||
id="global-scan",
|
||||
name="Global Monitor / World",
|
||||
region="global",
|
||||
feed_url="https://news.google.com/rss",
|
||||
homepage_url="https://news.google.com/",
|
||||
source_type="aggregated",
|
||||
)
|
||||
xml = """
|
||||
<rss>
|
||||
<channel>
|
||||
<item>
|
||||
<title>Example headline - Reuters</title>
|
||||
<description>Story summary</description>
|
||||
<link>https://news.google.com/example</link>
|
||||
</item>
|
||||
</channel>
|
||||
</rss>
|
||||
"""
|
||||
|
||||
items = _parse_feed_entries(xml, source)
|
||||
|
||||
assert items[0].title == "Example headline"
|
||||
assert items[0].source == "Reuters"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_enrich_items_with_target_locations_uses_ai_and_geocode(monkeypatch):
|
||||
item = ParsedNewsItem(
|
||||
id="bbc-world:f55310fb667b",
|
||||
title="Watch: What happened on day one of Trump's China visit?",
|
||||
summary="China welcomed US President Donald Trump before a long meeting with Xi Jinping.",
|
||||
url="https://example.com/china-visit",
|
||||
source="BBC World",
|
||||
feed_name="BBC World",
|
||||
feed_region="global",
|
||||
homepage_url="https://example.com",
|
||||
published_at=datetime(2026, 5, 14, 7, 13, 24, tzinfo=UTC),
|
||||
)
|
||||
|
||||
async def fake_geocode(_query: str):
|
||||
return {
|
||||
"lat": "39.9042",
|
||||
"lon": "116.4074",
|
||||
"display_name": "Beijing, China",
|
||||
}
|
||||
|
||||
class FakeProviderClient:
|
||||
async def analyze(self, _request):
|
||||
class Response:
|
||||
content = (
|
||||
'{"country":"China","city":"Beijing","matched_location_name":"Beijing, China",'
|
||||
'"latitude":null,"longitude":null,"confidence":0.88}'
|
||||
)
|
||||
|
||||
return Response()
|
||||
|
||||
monkeypatch.setattr("app.services.earth_news._geocode_target_location", fake_geocode)
|
||||
|
||||
enriched = await _enrich_items_with_target_locations(
|
||||
[item],
|
||||
provider_client=FakeProviderClient(),
|
||||
)
|
||||
|
||||
assert len(enriched) == 1
|
||||
assert enriched[0].target_location is not None
|
||||
assert enriched[0].target_location.latitude == 39.9042
|
||||
assert enriched[0].target_location.longitude == 116.4074
|
||||
assert enriched[0].target_location.label == "Beijing, China"
|
||||
assert enriched[0].target_resolution_stage == "ai_inferred_target"
|
||||
assert enriched[0].target_ai_attempted is True
|
||||
assert enriched[0].target_ai_status == "success"
|
||||
assert enriched[0].target_ai_error is None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_enrich_items_with_target_locations_adds_localizations(monkeypatch):
|
||||
item = ParsedNewsItem(
|
||||
id="global-scan:localized",
|
||||
title="Global leaders meet to discuss energy security",
|
||||
summary="Officials said the talks focused on supply chains and grid resilience.",
|
||||
url="https://example.com/energy-security",
|
||||
source="Example Source",
|
||||
feed_name="Global Monitor / World",
|
||||
feed_region="global",
|
||||
homepage_url="https://example.com",
|
||||
published_at=datetime(2026, 5, 14, 7, 13, 24, tzinfo=UTC),
|
||||
)
|
||||
|
||||
async def fake_geocode(_query: str):
|
||||
return {
|
||||
"lat": "50.1109",
|
||||
"lon": "8.6821",
|
||||
"display_name": "Frankfurt am Main, Germany",
|
||||
}
|
||||
|
||||
class FakeProviderClient:
|
||||
async def analyze(self, _request):
|
||||
class Response:
|
||||
content = (
|
||||
'{"location":{"country":"Germany","city":"Frankfurt",'
|
||||
'"matched_location_name":"Frankfurt, Germany",'
|
||||
'"latitude":null,"longitude":null,"confidence":0.77},'
|
||||
'"localizations":{"zh-CN":{"title":"全球领导人讨论能源安全",'
|
||||
'"summary":"官员表示,会谈聚焦供应链和电网韧性。"}}}'
|
||||
)
|
||||
|
||||
return Response()
|
||||
|
||||
monkeypatch.setattr("app.services.earth_news._geocode_target_location", fake_geocode)
|
||||
|
||||
enriched = await _enrich_items_with_target_locations(
|
||||
[item],
|
||||
provider_client=FakeProviderClient(),
|
||||
)
|
||||
payload = _serialize_item(enriched[0], active_region="global")
|
||||
|
||||
assert payload["title"] == "Global leaders meet to discuss energy security"
|
||||
assert payload["summary"] == "Officials said the talks focused on supply chains and grid resilience."
|
||||
assert payload["localizations"]["zh-CN"]["title"] == "全球领导人讨论能源安全"
|
||||
assert payload["display_title"] == "全球领导人讨论能源安全"
|
||||
assert payload["display_summary"] == "官员表示,会谈聚焦供应链和电网韧性。"
|
||||
assert payload["enrichment_status"] == "success"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_extract_target_location_from_text_uses_country_hint(monkeypatch):
|
||||
item = ParsedNewsItem(
|
||||
id="bbc-world:country-hint",
|
||||
title="Giant new dinosaur identified from fossils in Thailand",
|
||||
summary="The nagatitan is the largest dinosaur found in South-East Asia.",
|
||||
url="https://example.com/thailand-dinosaur",
|
||||
source="BBC World",
|
||||
feed_name="BBC World",
|
||||
feed_region="global",
|
||||
homepage_url="https://example.com",
|
||||
published_at=datetime(2026, 5, 14, 17, 28, 56, tzinfo=UTC),
|
||||
)
|
||||
|
||||
target = await _extract_target_location_from_text(item)
|
||||
|
||||
assert target is not None
|
||||
assert target.country == "泰国"
|
||||
assert target.latitude == 15.87
|
||||
assert target.longitude == 100.9925
|
||||
assert target.source == "headline_country_hint"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_enrich_items_with_target_locations_records_ai_provider_error():
|
||||
item = ParsedNewsItem(
|
||||
id="global-scan:no-hint",
|
||||
title="The New Geopolitics of Power: Whoever Controls Electrons Wins the Decade",
|
||||
summary="A broad analysis of industrial policy and energy systems.",
|
||||
url="https://example.com/geopolitics-power",
|
||||
source="Example Source",
|
||||
feed_name="Global Monitor / World",
|
||||
feed_region="global",
|
||||
homepage_url="https://example.com",
|
||||
published_at=datetime(2026, 5, 14, 17, 3, 10, tzinfo=UTC),
|
||||
)
|
||||
|
||||
class FailingProviderClient:
|
||||
async def analyze(self, _request):
|
||||
raise RuntimeError("upstream ai timeout")
|
||||
|
||||
enriched = await _enrich_items_with_target_locations(
|
||||
[item],
|
||||
provider_client=FailingProviderClient(),
|
||||
)
|
||||
|
||||
assert len(enriched) == 1
|
||||
assert enriched[0].target_location is None
|
||||
assert enriched[0].target_resolution_stage == "unresolved"
|
||||
assert enriched[0].target_ai_attempted is True
|
||||
assert enriched[0].target_ai_status == "provider_error"
|
||||
assert enriched[0].target_ai_error == "upstream ai timeout"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_earth_news_payload_returns_anchor_items_and_enqueues_location_job(monkeypatch):
|
||||
source = NewsFeedSource(
|
||||
id="test-feed",
|
||||
name="Test Feed",
|
||||
region="global",
|
||||
homepage_url="https://example.com",
|
||||
feed_url="https://example.com/rss.xml",
|
||||
)
|
||||
item = ParsedNewsItem(
|
||||
id="test-feed:timeout",
|
||||
title="Example story",
|
||||
summary="Example summary",
|
||||
url="https://example.com/story",
|
||||
source="Test Feed",
|
||||
feed_name="Test Feed",
|
||||
feed_region="global",
|
||||
homepage_url="https://example.com",
|
||||
published_at=datetime(2026, 5, 15, 3, 0, tzinfo=UTC),
|
||||
)
|
||||
|
||||
async def fake_fetch_source(_client, feed_source):
|
||||
return feed_source, [item], None
|
||||
|
||||
async def fake_get_cached_target_location_patch(_item_id):
|
||||
return None
|
||||
|
||||
enqueued_payloads = []
|
||||
|
||||
async def fake_enqueue_target_location_job(payload, **_kwargs):
|
||||
enqueued_payloads.append(payload)
|
||||
return True
|
||||
|
||||
monkeypatch.setattr("app.services.earth_news.get_sources_for_region", lambda _region: [source])
|
||||
monkeypatch.setattr("app.services.earth_news._fetch_source", fake_fetch_source)
|
||||
monkeypatch.setattr(
|
||||
"app.services.earth_news_queue.get_cached_target_location_patch",
|
||||
fake_get_cached_target_location_patch,
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"app.services.earth_news_queue.enqueue_target_location_job",
|
||||
fake_enqueue_target_location_job,
|
||||
)
|
||||
|
||||
payload = await get_earth_news_payload(provider_client=None)
|
||||
|
||||
assert len(payload["items"]) == 1
|
||||
assert payload["items"][0]["id"] == "test-feed:timeout"
|
||||
assert payload["items"][0]["display_title"] == ""
|
||||
assert payload["items"][0]["display_summary"] == ""
|
||||
assert payload["items"][0]["latitude"] == 20.0
|
||||
assert payload["items"][0]["longitude"] == 0.0
|
||||
assert payload["items"][0]["location_source"] == "region_anchor"
|
||||
assert payload["items"][0]["verified"] is False
|
||||
assert payload["items"][0]["location_meta"]["ai_status"] == "queued"
|
||||
assert enqueued_payloads[0]["id"] == "test-feed:timeout"
|
||||
assert payload["errors"] == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_earth_news_payload_uses_fresh_database_items_without_rss(monkeypatch):
|
||||
db = object()
|
||||
item = ParsedNewsItem(
|
||||
id="db:fresh",
|
||||
title="Fresh database story",
|
||||
summary="Stored summary",
|
||||
url="https://example.com/fresh",
|
||||
source="Stored Source",
|
||||
feed_name="Stored Feed",
|
||||
feed_region="global",
|
||||
homepage_url="https://example.com",
|
||||
published_at=datetime(2026, 5, 15, 3, 0, tzinfo=UTC),
|
||||
)
|
||||
item.location_patch = {
|
||||
"latitude": 39.9057136,
|
||||
"longitude": 116.3912972,
|
||||
"location_label": "北京市, 中国",
|
||||
"location_source": "headline_location_hint",
|
||||
"verified": True,
|
||||
"location_meta": {"target": {"city": "Beijing"}, "anchor": {"region": "global"}},
|
||||
}
|
||||
|
||||
async def fake_get_earth_news_freshness(_db, *, active_region):
|
||||
return 12, datetime.now(UTC)
|
||||
|
||||
async def fake_list_earth_news_items(_db, *, active_region, limit):
|
||||
assert limit == 12
|
||||
return [item]
|
||||
|
||||
async def fail_fetch(_sources):
|
||||
raise AssertionError("fresh database items should not fetch RSS")
|
||||
|
||||
monkeypatch.setattr("app.services.earth_news_store.get_earth_news_freshness", fake_get_earth_news_freshness)
|
||||
monkeypatch.setattr("app.services.earth_news_store.list_earth_news_items", fake_list_earth_news_items)
|
||||
monkeypatch.setattr("app.services.earth_news._fetch_rss_items_for_sources", fail_fetch)
|
||||
|
||||
payload = await get_earth_news_payload(db=db)
|
||||
|
||||
assert payload["items"][0]["id"] == "db:fresh"
|
||||
assert payload["items"][0]["verified"] is True
|
||||
assert payload["items"][0]["latitude"] == 39.9057136
|
||||
assert payload["stale"] is False
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_earth_news_payload_initializes_empty_database_from_rss(monkeypatch):
|
||||
db = object()
|
||||
source = NewsFeedSource(
|
||||
id="test-feed",
|
||||
name="Test Feed",
|
||||
region="global",
|
||||
homepage_url="https://example.com",
|
||||
feed_url="https://example.com/rss.xml",
|
||||
)
|
||||
item = ParsedNewsItem(
|
||||
id="test-feed:init",
|
||||
title="Initial RSS story",
|
||||
summary="Initial summary",
|
||||
url="https://example.com/init",
|
||||
source="Test Feed",
|
||||
feed_name="Test Feed",
|
||||
feed_region="global",
|
||||
homepage_url="https://example.com",
|
||||
published_at=datetime(2026, 5, 15, 3, 0, tzinfo=UTC),
|
||||
)
|
||||
item.location_patch = {
|
||||
"latitude": 20.0,
|
||||
"longitude": 0.0,
|
||||
"location_label": "全球",
|
||||
"location_source": "region_anchor",
|
||||
"verified": False,
|
||||
"location_meta": {"target": None, "anchor": {"region": "global"}},
|
||||
}
|
||||
upserted = []
|
||||
enqueued = []
|
||||
|
||||
async def fake_get_earth_news_freshness(_db, *, active_region):
|
||||
return 0, None
|
||||
|
||||
async def fake_fetch_rss_items_for_sources(_sources):
|
||||
return [item], []
|
||||
|
||||
async def fake_upsert_earth_news_items(_db, items):
|
||||
upserted.extend(items)
|
||||
return len(items)
|
||||
|
||||
async def fake_list_earth_news_items(_db, *, active_region, limit):
|
||||
return [item]
|
||||
|
||||
async def fake_enqueue_target_location_job(payload, **_kwargs):
|
||||
enqueued.append(payload)
|
||||
return True
|
||||
|
||||
monkeypatch.setattr("app.services.earth_news.get_sources_for_region", lambda _region: [source])
|
||||
monkeypatch.setattr("app.services.earth_news_store.get_earth_news_freshness", fake_get_earth_news_freshness)
|
||||
monkeypatch.setattr("app.services.earth_news._fetch_rss_items_for_sources", fake_fetch_rss_items_for_sources)
|
||||
monkeypatch.setattr("app.services.earth_news_store.upsert_earth_news_items", fake_upsert_earth_news_items)
|
||||
monkeypatch.setattr("app.services.earth_news_store.list_earth_news_items", fake_list_earth_news_items)
|
||||
monkeypatch.setattr("app.services.earth_news_queue.enqueue_target_location_job", fake_enqueue_target_location_job)
|
||||
|
||||
payload = await get_earth_news_payload(db=db)
|
||||
|
||||
assert upserted[0].id == "test-feed:init"
|
||||
assert payload["items"][0]["id"] == "test-feed:init"
|
||||
assert payload["items"][0]["verified"] is False
|
||||
assert enqueued[0]["id"] == "test-feed:init"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_earth_news_payload_supplements_stale_database_items(monkeypatch):
|
||||
db = object()
|
||||
old_item = ParsedNewsItem(
|
||||
id="db:old",
|
||||
title="Old story",
|
||||
summary="Old summary",
|
||||
url="https://example.com/old",
|
||||
source="Stored Source",
|
||||
feed_name="Stored Feed",
|
||||
feed_region="global",
|
||||
homepage_url="https://example.com",
|
||||
published_at=datetime(2026, 5, 14, 3, 0, tzinfo=UTC),
|
||||
)
|
||||
old_item.location_patch = {
|
||||
"latitude": 20.0,
|
||||
"longitude": 0.0,
|
||||
"location_label": "全球",
|
||||
"location_source": "region_anchor",
|
||||
"verified": False,
|
||||
"location_meta": {"target": None, "anchor": {"region": "global"}},
|
||||
}
|
||||
fetched = []
|
||||
|
||||
async def fake_get_earth_news_freshness(_db, *, active_region):
|
||||
return 12, datetime(2026, 5, 14, 3, 0, tzinfo=UTC)
|
||||
|
||||
async def fake_fetch_rss_items_for_sources(_sources):
|
||||
fetched.append(True)
|
||||
return [old_item], []
|
||||
|
||||
async def fake_upsert_earth_news_items(_db, items):
|
||||
return len(items)
|
||||
|
||||
async def fake_list_earth_news_items(_db, *, active_region, limit):
|
||||
return [old_item]
|
||||
|
||||
async def fake_enqueue_target_location_job(_payload, **_kwargs):
|
||||
return True
|
||||
|
||||
monkeypatch.setattr("app.services.earth_news_store.get_earth_news_freshness", fake_get_earth_news_freshness)
|
||||
monkeypatch.setattr("app.services.earth_news._fetch_rss_items_for_sources", fake_fetch_rss_items_for_sources)
|
||||
monkeypatch.setattr("app.services.earth_news_store.upsert_earth_news_items", fake_upsert_earth_news_items)
|
||||
monkeypatch.setattr("app.services.earth_news_store.list_earth_news_items", fake_list_earth_news_items)
|
||||
monkeypatch.setattr("app.services.earth_news_queue.enqueue_target_location_job", fake_enqueue_target_location_job)
|
||||
|
||||
payload = await get_earth_news_payload(db=db)
|
||||
|
||||
assert fetched == [True]
|
||||
assert payload["items"][0]["id"] == "db:old"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_earth_news_payload_merges_cached_location_patch(monkeypatch):
|
||||
source = NewsFeedSource(
|
||||
id="test-feed",
|
||||
name="Test Feed",
|
||||
region="global",
|
||||
homepage_url="https://example.com",
|
||||
feed_url="https://example.com/rss.xml",
|
||||
)
|
||||
item = ParsedNewsItem(
|
||||
id="test-feed:cached",
|
||||
title="Cached story",
|
||||
summary="Cached summary",
|
||||
url="https://example.com/cached",
|
||||
source="Test Feed",
|
||||
feed_name="Test Feed",
|
||||
feed_region="global",
|
||||
homepage_url="https://example.com",
|
||||
published_at=datetime(2026, 5, 15, 3, 0, tzinfo=UTC),
|
||||
)
|
||||
cached_patch = {
|
||||
"latitude": 39.9057136,
|
||||
"longitude": 116.3912972,
|
||||
"location_label": "北京市, 中国",
|
||||
"location_source": "headline_location_hint",
|
||||
"verified": True,
|
||||
"location_meta": {
|
||||
"resolution_stage": "headline_location_hint",
|
||||
"ai_attempted": False,
|
||||
"ai_status": "skipped_text_hint",
|
||||
"ai_error": None,
|
||||
"debug_note": "text hint matched 北京市, 中国",
|
||||
"target": {"city": "Beijing"},
|
||||
"anchor": {"region": "global"},
|
||||
},
|
||||
}
|
||||
|
||||
async def fake_fetch_source(_client, feed_source):
|
||||
return feed_source, [item], None
|
||||
|
||||
async def fake_get_cached_target_location_patch(_item_id):
|
||||
return cached_patch
|
||||
|
||||
enqueued = []
|
||||
|
||||
async def fake_enqueue_target_location_job(payload, **_kwargs):
|
||||
enqueued.append(payload)
|
||||
return True
|
||||
|
||||
monkeypatch.setattr("app.services.earth_news.get_sources_for_region", lambda _region: [source])
|
||||
monkeypatch.setattr("app.services.earth_news._fetch_source", fake_fetch_source)
|
||||
monkeypatch.setattr(
|
||||
"app.services.earth_news_queue.get_cached_target_location_patch",
|
||||
fake_get_cached_target_location_patch,
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"app.services.earth_news_queue.enqueue_target_location_job",
|
||||
fake_enqueue_target_location_job,
|
||||
)
|
||||
|
||||
payload = await get_earth_news_payload(provider_client=None)
|
||||
|
||||
assert payload["items"][0]["latitude"] == 39.9057136
|
||||
assert payload["items"][0]["longitude"] == 116.3912972
|
||||
assert payload["items"][0]["verified"] is True
|
||||
assert payload["items"][0]["location_source"] == "headline_location_hint"
|
||||
assert enqueued[0]["id"] == "test-feed:cached"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_earth_news_payload_requeues_cached_failed_localization(monkeypatch):
|
||||
source = NewsFeedSource(
|
||||
id="test-feed",
|
||||
name="Test Feed",
|
||||
region="global",
|
||||
homepage_url="https://example.com",
|
||||
feed_url="https://example.com/rss.xml",
|
||||
)
|
||||
item = ParsedNewsItem(
|
||||
id="test-feed:failed-localization",
|
||||
title="Failed localization story",
|
||||
summary="English source summary.",
|
||||
url="https://example.com/failed-localization",
|
||||
source="Test Feed",
|
||||
feed_name="Test Feed",
|
||||
feed_region="global",
|
||||
homepage_url="https://example.com",
|
||||
published_at=datetime(2026, 5, 15, 3, 0, tzinfo=UTC),
|
||||
)
|
||||
cached_patch = {
|
||||
"latitude": 20.0,
|
||||
"longitude": 0.0,
|
||||
"location_label": "全球",
|
||||
"location_source": "region_anchor",
|
||||
"verified": False,
|
||||
"location_meta": {"target": None, "anchor": {"region": "global"}},
|
||||
"content_language": "en",
|
||||
"localizations": {},
|
||||
"enrichment_status": "parse_error",
|
||||
"enrichment_error": "AI response did not contain a parseable JSON object.",
|
||||
"enriched_at": None,
|
||||
}
|
||||
enqueued = []
|
||||
|
||||
async def fake_fetch_source(_client, feed_source):
|
||||
return feed_source, [item], None
|
||||
|
||||
async def fake_get_cached_target_location_patch(_item_id):
|
||||
return cached_patch
|
||||
|
||||
async def fake_enqueue_target_location_job(payload, **_kwargs):
|
||||
enqueued.append(payload)
|
||||
return True
|
||||
|
||||
monkeypatch.setattr("app.services.earth_news.get_sources_for_region", lambda _region: [source])
|
||||
monkeypatch.setattr("app.services.earth_news._fetch_source", fake_fetch_source)
|
||||
monkeypatch.setattr(
|
||||
"app.services.earth_news_queue.get_cached_target_location_patch",
|
||||
fake_get_cached_target_location_patch,
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"app.services.earth_news_queue.enqueue_target_location_job",
|
||||
fake_enqueue_target_location_job,
|
||||
)
|
||||
|
||||
payload = await get_earth_news_payload(provider_client=None)
|
||||
|
||||
assert enqueued[0]["id"] == "test-feed:failed-localization"
|
||||
assert payload["items"][0]["display_title"] == ""
|
||||
assert payload["items"][0]["enrichment_status"] == "queued"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_worker_processes_target_location_message_and_returns_patch(monkeypatch):
|
||||
message = NewsTargetLocationMessage(
|
||||
message_id="1-0",
|
||||
item_id="bbc-world:worker",
|
||||
payload={
|
||||
"id": "bbc-world:worker",
|
||||
"title": "Ukraine rescuers pull dead from rubble of Kyiv flats",
|
||||
"summary": "Massive Russian drone and missile attacks in Ukraine's capital.",
|
||||
"url": "https://example.com/kyiv",
|
||||
"source": "BBC World",
|
||||
"feed_name": "BBC World",
|
||||
"feed_region": "global",
|
||||
"homepage_url": "https://www.bbc.com/news/world",
|
||||
"published_at": "2026-05-14T13:16:32Z",
|
||||
},
|
||||
)
|
||||
|
||||
async def fake_geocode(_query: str):
|
||||
return {
|
||||
"lat": "50.4500336",
|
||||
"lon": "30.5241361",
|
||||
"display_name": "Київ, Україна",
|
||||
}
|
||||
|
||||
saved = {}
|
||||
broadcasted = {}
|
||||
|
||||
async def fake_save_target_location_patch(item_id, patch):
|
||||
saved["item_id"] = item_id
|
||||
saved["patch"] = patch
|
||||
|
||||
async def fake_update_earth_news_item_location(_session, *, item_id, patch):
|
||||
saved["db_item_id"] = item_id
|
||||
saved["db_patch"] = patch
|
||||
return True
|
||||
|
||||
async def fake_broadcast_custom(channel, data):
|
||||
broadcasted["channel"] = channel
|
||||
broadcasted["data"] = data
|
||||
|
||||
class FakeSession:
|
||||
async def commit(self):
|
||||
saved["committed"] = True
|
||||
|
||||
class FakeSessionFactory:
|
||||
async def __aenter__(self):
|
||||
return FakeSession()
|
||||
|
||||
async def __aexit__(self, exc_type, exc, tb):
|
||||
return False
|
||||
|
||||
monkeypatch.setattr("app.services.earth_news._geocode_target_location", fake_geocode)
|
||||
monkeypatch.setattr(
|
||||
"app.services.earth_news_worker.save_target_location_patch",
|
||||
fake_save_target_location_patch,
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"app.services.earth_news_worker.update_earth_news_item_location",
|
||||
fake_update_earth_news_item_location,
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"app.services.earth_news_worker.async_session_factory",
|
||||
lambda: FakeSessionFactory(),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"app.services.earth_news_worker.broadcaster.broadcast_custom",
|
||||
fake_broadcast_custom,
|
||||
)
|
||||
|
||||
patch = await process_target_location_message(message, provider_client=None)
|
||||
|
||||
assert patch["latitude"] == 50.4500336
|
||||
assert patch["longitude"] == 30.5241361
|
||||
assert patch["location_source"] == "headline_location_hint"
|
||||
assert patch["verified"] is True
|
||||
assert saved["item_id"] == "bbc-world:worker"
|
||||
assert saved["db_item_id"] == "bbc-world:worker"
|
||||
assert saved["committed"] is True
|
||||
assert broadcasted["channel"] == "earth_news"
|
||||
assert broadcasted["data"]["item_id"] == "bbc-world:worker"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_media_news_archive_collector_maps_news_items(monkeypatch):
|
||||
collector = MediaNewsArchiveCollector()
|
||||
collector._db_session = object()
|
||||
record = SimpleNamespace(
|
||||
id="bbc-world:archive",
|
||||
title="Archived news",
|
||||
summary="Archived summary",
|
||||
url="https://example.com/archive",
|
||||
source="BBC World",
|
||||
feed_name="BBC World",
|
||||
region="global",
|
||||
homepage_url="https://www.bbc.com/news/world",
|
||||
published_at=datetime(2026, 5, 15, 3, 0, tzinfo=UTC),
|
||||
latitude=39.9057136,
|
||||
longitude=116.3912972,
|
||||
location_label="北京市, 中国",
|
||||
location_source="headline_location_hint",
|
||||
verified=True,
|
||||
location_meta={"target": {"country": "中国", "city": "Beijing"}},
|
||||
content_language="en",
|
||||
localizations={"zh-CN": {"title": "归档新闻", "summary": "归档概要"}},
|
||||
enrichment_status="success",
|
||||
enrichment_error=None,
|
||||
enriched_at=datetime(2026, 5, 15, 3, 6, tzinfo=UTC),
|
||||
first_seen_at=datetime(2026, 5, 15, 2, 0, tzinfo=UTC),
|
||||
last_seen_at=datetime(2026, 5, 15, 3, 0, tzinfo=UTC),
|
||||
resolved_at=datetime(2026, 5, 15, 3, 5, tzinfo=UTC),
|
||||
)
|
||||
|
||||
async def fake_list_all_earth_news_records(_db):
|
||||
return [record]
|
||||
|
||||
monkeypatch.setattr(
|
||||
"app.services.collectors.media_news_archive.list_all_earth_news_records",
|
||||
fake_list_all_earth_news_records,
|
||||
)
|
||||
|
||||
items = await collector.fetch()
|
||||
|
||||
assert items[0]["source_id"] == "bbc-world:archive"
|
||||
assert collector.data_type == "news_item"
|
||||
assert items[0]["country"] == "中国"
|
||||
assert items[0]["city"] == "Beijing"
|
||||
assert items[0]["latitude"] == 39.9057136
|
||||
assert items[0]["metadata"]["verified"] is True
|
||||
assert "localizations" not in items[0]["metadata"]
|
||||
assert "enrichment_status" not in items[0]["metadata"]
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
from fastapi import HTTPException
|
||||
import pytest
|
||||
|
||||
from app.api.v1 import layers
|
||||
|
||||
@@ -42,3 +43,28 @@ def test_layer_guard_filters_bbox_and_clamps_low_zoom_limit():
|
||||
assert result["diagnostics"]["limit"] == layers.LOW_ZOOM_FEATURE_LIMIT
|
||||
assert result["diagnostics"]["limit_clamped"] is True
|
||||
assert result["diagnostics"]["degraded"] is True
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_vessel_layer_snapshot_passes_type_filter(monkeypatch):
|
||||
captured = {}
|
||||
|
||||
async def fake_build_vessel_snapshot_response(db, **kwargs):
|
||||
captured.update(kwargs)
|
||||
return {"type": "FeatureCollection", "features": []}
|
||||
|
||||
monkeypatch.setattr(layers, "build_vessel_snapshot_response", fake_build_vessel_snapshot_response)
|
||||
|
||||
result = await layers.get_vessel_layer_snapshot(
|
||||
bbox="10,59,11,60",
|
||||
zoom=12,
|
||||
limit=1000,
|
||||
vessel_type="cargo",
|
||||
since_minutes=30,
|
||||
db=object(),
|
||||
)
|
||||
|
||||
assert result["features"] == []
|
||||
assert captured["bbox"] == (10.0, 59.0, 11.0, 60.0)
|
||||
assert captured["type_filter"] == "cargo"
|
||||
assert "vessel_type" not in captured
|
||||
|
||||
@@ -700,7 +700,7 @@ async def test_llm_location_fallback_repairs_non_json_answer(monkeypatch):
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_llm_location_fallback_accepts_taipei_name_hint_with_weak_wording(monkeypatch):
|
||||
async def test_llm_location_fallback_rejects_city_from_name_without_location_evidence(monkeypatch):
|
||||
monkeypatch.setattr(
|
||||
llm_fallback,
|
||||
"_geocode_llm_city",
|
||||
@@ -740,18 +740,60 @@ async def test_llm_location_fallback_accepts_taipei_name_hint_with_weak_wording(
|
||||
entity_type="compute_center",
|
||||
)
|
||||
|
||||
assert result.failure_reason is None
|
||||
candidate = result.candidates[0]
|
||||
assert candidate.city == "Taipei"
|
||||
assert candidate.confidence >= 0.55
|
||||
breakdown = candidate.suggested_registry_entry["llm_score_breakdown"]
|
||||
assert breakdown["weak_evidence_penalty"] <= 0.15
|
||||
assert breakdown["conflict_penalty"] == 0
|
||||
assert breakdown["name_location_hint"] > 0
|
||||
assert result.candidates == []
|
||||
assert result.failure_reason is not None
|
||||
assert "below minimum" in result.failure_reason
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_llm_location_fallback_geocodes_city_from_entity_name_when_llm_unparseable(monkeypatch):
|
||||
async def test_llm_location_fallback_accepts_explicit_facility_location_for_name_city_conflict(monkeypatch):
|
||||
monkeypatch.setattr(
|
||||
llm_fallback,
|
||||
"_geocode_llm_city",
|
||||
lambda query: {
|
||||
"lat": "22.6048",
|
||||
"lon": "120.3000",
|
||||
"display_name": "Kaohsiung, Taiwan",
|
||||
"address": {"city": "Kaohsiung", "country": "Taiwan"},
|
||||
},
|
||||
)
|
||||
client = _FakeAIProviderClient(
|
||||
json.dumps(
|
||||
{
|
||||
"latitude": None,
|
||||
"longitude": None,
|
||||
"precision": "city",
|
||||
"confidence": 0.72,
|
||||
"city": "Kaohsiung",
|
||||
"country": "Taiwan",
|
||||
"matched_location_name": "Kaohsiung, Taiwan",
|
||||
"evidence": [
|
||||
{
|
||||
"source": "Taiwan News",
|
||||
"source_type": "news",
|
||||
"entity_match": True,
|
||||
"text": "Nvidia's first AI supercomputer center, Taipei-1, is located in Kaohsiung.",
|
||||
}
|
||||
],
|
||||
"reasoning_summary": "Explicit facility location evidence overrides the city-like system name.",
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
result = await collect_llm_location_fallback_candidate(
|
||||
provider_client=client,
|
||||
query=LocationQuery(name="TAIPEI-1", country="Taiwan"),
|
||||
entity_type="compute_center",
|
||||
)
|
||||
|
||||
assert result.failure_reason is None
|
||||
candidate = result.candidates[0]
|
||||
assert candidate.city == "Kaohsiung"
|
||||
assert candidate.confidence >= 0.55
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_llm_location_fallback_rejects_compute_center_city_from_entity_name_when_llm_unparseable(monkeypatch):
|
||||
def _fake_geocode(query):
|
||||
if query != "Taipei, 中国(台湾)":
|
||||
return None
|
||||
@@ -772,12 +814,9 @@ async def test_llm_location_fallback_geocodes_city_from_entity_name_when_llm_unp
|
||||
)
|
||||
|
||||
assert client.calls == 2
|
||||
assert result.failure_reason is None
|
||||
candidate = result.candidates[0]
|
||||
assert candidate.city == "Taipei"
|
||||
assert candidate.latitude == pytest.approx(25.033)
|
||||
assert candidate.longitude == pytest.approx(121.5654)
|
||||
assert "Entity name city hint" in candidate.source_note
|
||||
assert result.candidates == []
|
||||
assert result.failure_reason is not None
|
||||
assert "parseable city-level location fact" in result.failure_reason
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
|
||||
101
backend/tests/test_realtime_sources.py
Normal file
101
backend/tests/test_realtime_sources.py
Normal file
@@ -0,0 +1,101 @@
|
||||
from datetime import UTC, datetime
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
import pytest
|
||||
from fastapi import HTTPException
|
||||
|
||||
from app.api.v1 import realtime_sources
|
||||
from app.models.datasource import DataSource
|
||||
from app.models.datasource_config import DataSourceConfig
|
||||
from app.models.vessel import AISSourceHealth
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_serialize_builtin_aisstream_includes_health_config_and_stats(monkeypatch):
|
||||
datasource = DataSource(
|
||||
id=28,
|
||||
name="AISStream Vessels",
|
||||
source="aisstream_vessels",
|
||||
module="L4",
|
||||
priority="P1",
|
||||
collector_class="aisstream_vessels",
|
||||
is_active=True,
|
||||
)
|
||||
config = DataSourceConfig(
|
||||
id=3,
|
||||
name="aisstream_vessels",
|
||||
source_type="websocket",
|
||||
endpoint="wss://stream.aisstream.io/v0/stream",
|
||||
auth_type="api_key",
|
||||
auth_config={"api_key": "test-key"},
|
||||
config={
|
||||
"message_types": ["PositionReport"],
|
||||
"bounding_boxes": [[[-10, 50], [35, 75]]],
|
||||
},
|
||||
is_active=True,
|
||||
)
|
||||
health = AISSourceHealth(
|
||||
source="aisstream_vessels",
|
||||
connection_state="connected",
|
||||
last_seen_at=datetime(2026, 5, 13, 1, 0, tzinfo=UTC),
|
||||
)
|
||||
|
||||
class _Session:
|
||||
async def get(self, _model, key):
|
||||
assert key == "aisstream_vessels"
|
||||
return health
|
||||
|
||||
monkeypatch.setattr(
|
||||
realtime_sources,
|
||||
"_load_realtime_stats",
|
||||
AsyncMock(
|
||||
return_value={
|
||||
"total_observations": 10,
|
||||
"observations_24h": 4,
|
||||
"observations_1h": 1,
|
||||
"unique_mmsi_total": 8,
|
||||
"unique_mmsi_24h": 3,
|
||||
"latest_observed_at": "2026-05-13T01:00:00Z",
|
||||
"latest_collected_at": "2026-05-13T01:00:01Z",
|
||||
}
|
||||
),
|
||||
)
|
||||
monkeypatch.setattr(realtime_sources, "is_collector_running", lambda source: False)
|
||||
|
||||
payload = await realtime_sources._serialize_builtin_aisstream(_Session(), datasource, config)
|
||||
|
||||
assert payload["source"] == "aisstream_vessels"
|
||||
assert payload["kind"] == "builtin"
|
||||
assert payload["credential_configured"] is True
|
||||
assert payload["message_types"] == ["PositionReport"]
|
||||
assert payload["runtime"]["running"] is False
|
||||
assert payload["health"]["connection_state"] == "connected"
|
||||
assert payload["stats"]["total_observations"] == 10
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_start_builtin_realtime_source_rejects_disabled(monkeypatch):
|
||||
datasource = DataSource(
|
||||
id=28,
|
||||
name="AISStream Vessels",
|
||||
source="aisstream_vessels",
|
||||
module="L4",
|
||||
priority="P1",
|
||||
collector_class="aisstream_vessels",
|
||||
is_active=False,
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
realtime_sources,
|
||||
"_load_builtin_aisstream",
|
||||
AsyncMock(return_value=(datasource, None)),
|
||||
)
|
||||
|
||||
with pytest.raises(HTTPException) as excinfo:
|
||||
await realtime_sources.start_realtime_source(
|
||||
"aisstream_vessels",
|
||||
current_user=object(),
|
||||
db=object(),
|
||||
)
|
||||
|
||||
assert excinfo.value.status_code == 400
|
||||
assert "disabled" in excinfo.value.detail
|
||||
92
backend/tests/test_settings_ai_prompts.py
Normal file
92
backend/tests/test_settings_ai_prompts.py
Normal file
@@ -0,0 +1,92 @@
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
|
||||
from app.ai_tasks.prompts import (
|
||||
get_effective_prompt,
|
||||
list_effective_prompts,
|
||||
reset_prompt_override,
|
||||
save_prompt_override,
|
||||
)
|
||||
|
||||
|
||||
class _ScalarResult:
|
||||
def __init__(self, value):
|
||||
self._value = value
|
||||
|
||||
def scalar_one_or_none(self):
|
||||
return self._value
|
||||
|
||||
|
||||
class _PromptSettingsDB:
|
||||
def __init__(self, payload=None):
|
||||
self.record = SimpleNamespace(category="ai_prompts", payload=payload) if payload is not None else None
|
||||
self.added = None
|
||||
self.commits = 0
|
||||
|
||||
async def execute(self, _statement):
|
||||
return _ScalarResult(self.record)
|
||||
|
||||
def add(self, record):
|
||||
self.record = record
|
||||
self.added = record
|
||||
|
||||
async def commit(self):
|
||||
self.commits += 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_prompt_defaults_are_loaded_without_override():
|
||||
db = _PromptSettingsDB()
|
||||
|
||||
prompt = await get_effective_prompt(db, "earth.news.enrich")
|
||||
|
||||
assert prompt.key == "earth.news.enrich"
|
||||
assert prompt.is_custom is False
|
||||
assert "strict JSON" in prompt.prompt
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_prompt_override_save_and_reset():
|
||||
db = _PromptSettingsDB()
|
||||
|
||||
saved = await save_prompt_override(
|
||||
db,
|
||||
"alerts.brief",
|
||||
system_prompt="system custom",
|
||||
prompt="prompt custom",
|
||||
)
|
||||
|
||||
assert saved.is_custom is True
|
||||
assert saved.system_prompt == "system custom"
|
||||
assert saved.prompt == "prompt custom"
|
||||
assert db.commits == 1
|
||||
|
||||
effective = await get_effective_prompt(db, "alerts.brief")
|
||||
assert effective.prompt == "prompt custom"
|
||||
|
||||
reset = await reset_prompt_override(db, "alerts.brief")
|
||||
assert reset.is_custom is False
|
||||
assert reset.prompt != "prompt custom"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_prompt_list_marks_custom_items():
|
||||
db = _PromptSettingsDB(
|
||||
{
|
||||
"overrides": {
|
||||
"bgp.brief": {
|
||||
"system_prompt": "",
|
||||
"prompt": "custom bgp prompt",
|
||||
"updated_at": "2026-05-16T00:00:00Z",
|
||||
}
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
prompts = await list_effective_prompts(db)
|
||||
by_key = {prompt.key: prompt for prompt in prompts}
|
||||
|
||||
assert by_key["bgp.brief"].is_custom is True
|
||||
assert by_key["bgp.brief"].prompt == "custom bgp prompt"
|
||||
assert by_key["earth.news.enrich"].is_custom is False
|
||||
@@ -5,21 +5,42 @@ import pytest
|
||||
from app.api.v1 import settings as settings_api
|
||||
from app.api.v1.settings import (
|
||||
AIProviderIntegrationUpdate,
|
||||
BarentsWatchIntegrationUpdate,
|
||||
ExternalIntegrationsUpdate,
|
||||
OCRIntegrationUpdate,
|
||||
WebSearchIntegrationUpdate,
|
||||
_build_ai_provider_payload,
|
||||
_build_ocr_payload,
|
||||
_can_reveal_integration_secrets,
|
||||
_ensure_secret_reveal_allowed,
|
||||
_mask_secret,
|
||||
_normalize_ai_provider_payload,
|
||||
_normalize_ocr_payload,
|
||||
_record_integration_secret_reveal,
|
||||
_resolve_provider_api_key,
|
||||
get_runtime_ai_provider_config,
|
||||
save_external_integrations_payload,
|
||||
)
|
||||
from app.services.llm_provider_catalog import get_fallback_llm_provider_preset
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def isolated_ai_provider_env_file(monkeypatch, tmp_path):
|
||||
env_file = tmp_path / ".env"
|
||||
monkeypatch.setattr(settings_api, "AI_PROVIDER_ENV_FILE", env_file)
|
||||
for name in (
|
||||
"AI_PROVIDER",
|
||||
"AI_API_KEY",
|
||||
"MINIMAX_API_KEY",
|
||||
"OPENAI_API_KEY",
|
||||
"ANTHROPIC_API_KEY",
|
||||
"DEEPSEEK_API_KEY",
|
||||
"DASHSCOPE_API_KEY",
|
||||
"MOONSHOT_API_KEY",
|
||||
"OPENROUTER_API_KEY",
|
||||
"OPENCODE_GO_API_KEY",
|
||||
):
|
||||
monkeypatch.delenv(name, raising=False)
|
||||
return env_file
|
||||
|
||||
|
||||
@@ -56,7 +77,7 @@ def test_provider_key_prefers_specific_env_file_key(isolated_ai_provider_env_fil
|
||||
|
||||
def test_provider_key_falls_back_to_generic_ai_api_key(isolated_ai_provider_env_file):
|
||||
isolated_ai_provider_env_file.write_text(
|
||||
"AI_API_KEY=generic-env-file-key\n",
|
||||
"AI_PROVIDER=openai\nAI_API_KEY=generic-env-file-key\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
@@ -66,11 +87,101 @@ def test_provider_key_falls_back_to_generic_ai_api_key(isolated_ai_provider_env_
|
||||
assert source == "env_file"
|
||||
|
||||
|
||||
def test_generic_ai_api_key_only_applies_to_selected_env_provider(isolated_ai_provider_env_file):
|
||||
isolated_ai_provider_env_file.write_text(
|
||||
"AI_PROVIDER=minimax\nAI_API_KEY=generic-env-file-key\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
openai_value, openai_source = _resolve_provider_api_key("openai", {"api_key": ""})
|
||||
minimax_value, minimax_source = _resolve_provider_api_key("minimax", {"api_key": ""})
|
||||
|
||||
assert openai_value == ""
|
||||
assert openai_source == ""
|
||||
assert minimax_value == "generic-env-file-key"
|
||||
assert minimax_source == "env_file"
|
||||
|
||||
|
||||
def test_opencode_go_provider_preset_is_openai_compatible():
|
||||
preset = get_fallback_llm_provider_preset("opencode-go")
|
||||
|
||||
assert preset["label"] == "OpenCode Go"
|
||||
assert preset["provider_api"] == "openai-completions"
|
||||
assert preset["base_url"] == "https://opencode.ai/zen/go/v1"
|
||||
assert preset["model"] == "glm-5.1"
|
||||
assert "glm-5.1" in preset["models"]
|
||||
assert "deepseek-v4-flash" in preset["models"]
|
||||
assert preset["model_provider_apis"]["minimax-m2.7"] == "anthropic-messages"
|
||||
assert preset["api_key_env"] == "OPENCODE_GO_API_KEY"
|
||||
|
||||
|
||||
def test_mask_secret_without_prefix_is_fully_masked():
|
||||
assert _mask_secret("plainsecret")["preview"] == "***********"
|
||||
assert _mask_secret("sk-prefixed")["preview"] == "sk-********"
|
||||
|
||||
|
||||
def test_secret_reveal_permission_is_admin_only():
|
||||
assert _can_reveal_integration_secrets(SimpleNamespace(role="super_admin")) is True
|
||||
assert _can_reveal_integration_secrets(SimpleNamespace(role="admin")) is True
|
||||
assert _can_reveal_integration_secrets(SimpleNamespace(role="viewer")) is False
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_secret_reveal_denial_is_audited(monkeypatch):
|
||||
events = []
|
||||
|
||||
async def fake_record_audit_log(**payload):
|
||||
events.append(payload)
|
||||
|
||||
monkeypatch.setattr(settings_api, "record_audit_log", fake_record_audit_log)
|
||||
user = SimpleNamespace(id=7, username="viewer", email="viewer@example.test", role="viewer")
|
||||
|
||||
with pytest.raises(settings_api.HTTPException) as exc:
|
||||
await _ensure_secret_reveal_allowed(
|
||||
current_user=user,
|
||||
request=None,
|
||||
target_id="ai_provider:openai",
|
||||
details={"kind": "ai_provider", "provider": "openai"},
|
||||
)
|
||||
|
||||
assert exc.value.status_code == 403
|
||||
assert events[0]["action"] == "settings.integration_secret.reveal"
|
||||
assert events[0]["result"] == "denied"
|
||||
assert events[0]["actor_id"] == 7
|
||||
assert events[0]["target_id"] == "ai_provider:openai"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_secret_reveal_audit_does_not_store_plaintext(monkeypatch):
|
||||
events = []
|
||||
|
||||
async def fake_record_audit_log(**payload):
|
||||
events.append(payload)
|
||||
|
||||
monkeypatch.setattr(settings_api, "record_audit_log", fake_record_audit_log)
|
||||
user = SimpleNamespace(id=1, username="admin", email="admin@example.test", role="admin")
|
||||
|
||||
await _record_integration_secret_reveal(
|
||||
current_user=user,
|
||||
request=None,
|
||||
target_id="ai_provider:minimax",
|
||||
result="success",
|
||||
details={
|
||||
"kind": "ai_provider",
|
||||
"provider": "minimax",
|
||||
"api_key_configured": True,
|
||||
"api_key_source": "env_file",
|
||||
"service_token_configured": True,
|
||||
"service_token_source": "runtime",
|
||||
},
|
||||
)
|
||||
|
||||
serialized = str(events[0])
|
||||
assert "secret-value" not in serialized
|
||||
assert events[0]["details"]["api_key_source"] == "env_file"
|
||||
assert events[0]["details"]["service_token_configured"] is True
|
||||
|
||||
|
||||
def test_build_payload_updates_only_selected_provider_key():
|
||||
current = {
|
||||
"ai_provider": {
|
||||
@@ -109,6 +220,30 @@ def test_build_payload_updates_only_selected_provider_key():
|
||||
assert payload["providers"]["minimax"]["api_key"] == "minimax-old-key"
|
||||
|
||||
|
||||
def test_build_payload_saves_provider_without_changing_default():
|
||||
current = {
|
||||
"ai_provider": {
|
||||
"default_provider": "minimax",
|
||||
"providers": {
|
||||
"minimax": {"provider": "minimax", "api_key": "minimax-key"},
|
||||
"openai": {"provider": "openai", "api_key": ""},
|
||||
},
|
||||
}
|
||||
}
|
||||
update = AIProviderIntegrationUpdate(
|
||||
provider="openai",
|
||||
provider_api="openai-completions",
|
||||
base_url="https://api.openai.com/v1",
|
||||
model="gpt-test",
|
||||
api_key="openai-new-key",
|
||||
)
|
||||
|
||||
payload = _build_ai_provider_payload(current, update)
|
||||
|
||||
assert payload["default_provider"] == "minimax"
|
||||
assert payload["providers"]["openai"]["api_key"] == "openai-new-key"
|
||||
|
||||
|
||||
def test_build_payload_keeps_saved_key_when_preview_submitted():
|
||||
current = {
|
||||
"ai_provider": {
|
||||
@@ -207,3 +342,78 @@ async def test_runtime_config_uses_default_provider_specific_key(monkeypatch):
|
||||
assert runtime_config["llm_config"]["provider"] == "minimax"
|
||||
assert runtime_config["llm_config"]["api_key"] == "minimax-key"
|
||||
assert runtime_config["llm_config"]["model"] == "MiniMax-test"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_saving_ai_provider_does_not_run_connection_validation(monkeypatch):
|
||||
calls = {"validated": 0, "saved": 0}
|
||||
|
||||
async def fake_get_setting_payload(_db, category):
|
||||
assert category == "external_integrations"
|
||||
return {
|
||||
"ai_provider": {
|
||||
"default_provider": "minimax",
|
||||
"providers": {
|
||||
"minimax": {
|
||||
"provider": "minimax",
|
||||
"api_key": "minimax-key",
|
||||
},
|
||||
},
|
||||
},
|
||||
"web_search": {},
|
||||
"ocr": {},
|
||||
}
|
||||
|
||||
async def fake_validate(_payload):
|
||||
calls["validated"] += 1
|
||||
raise AssertionError("save should not run AI provider connection validation")
|
||||
|
||||
async def fake_save_setting_payload(_db, category, payload):
|
||||
assert category == "external_integrations"
|
||||
calls["saved"] += 1
|
||||
return payload
|
||||
|
||||
async def fake_get_barentswatch_config_record(_db):
|
||||
return SimpleNamespace(
|
||||
endpoint="",
|
||||
auth_config={},
|
||||
)
|
||||
|
||||
async def fake_serialize_external_integrations(_db):
|
||||
return {"ai_provider": {"default_provider": "minimax"}}
|
||||
|
||||
monkeypatch.setattr(settings_api, "get_setting_payload", fake_get_setting_payload)
|
||||
monkeypatch.setattr(settings_api, "_validate_ai_provider_full_connection", fake_validate)
|
||||
monkeypatch.setattr(settings_api, "save_setting_payload", fake_save_setting_payload)
|
||||
monkeypatch.setattr(settings_api, "get_barentswatch_config_record", fake_get_barentswatch_config_record)
|
||||
monkeypatch.setattr(settings_api, "serialize_external_integrations", fake_serialize_external_integrations)
|
||||
|
||||
update = ExternalIntegrationsUpdate(
|
||||
ai_provider=AIProviderIntegrationUpdate(
|
||||
provider="minimax",
|
||||
default_provider="minimax",
|
||||
provider_api="anthropic-messages",
|
||||
base_url="https://api.minimaxi.com/anthropic",
|
||||
model="MiniMax-M2.7",
|
||||
api_key="sk-new-key",
|
||||
),
|
||||
barentswatch=BarentsWatchIntegrationUpdate(),
|
||||
web_search=WebSearchIntegrationUpdate(),
|
||||
ocr=OCRIntegrationUpdate(),
|
||||
)
|
||||
|
||||
async def fake_commit():
|
||||
return None
|
||||
|
||||
async def fake_refresh(_record):
|
||||
return None
|
||||
|
||||
db = SimpleNamespace(
|
||||
add=lambda _record: None,
|
||||
commit=fake_commit,
|
||||
refresh=fake_refresh,
|
||||
)
|
||||
|
||||
await save_external_integrations_payload(db, update)
|
||||
|
||||
assert calls == {"validated": 0, "saved": 1}
|
||||
|
||||
12
backend/tests/test_settings_secret_placeholder.py
Normal file
12
backend/tests/test_settings_secret_placeholder.py
Normal file
@@ -0,0 +1,12 @@
|
||||
from app.api.v1.settings import _is_secret_placeholder
|
||||
|
||||
|
||||
def test_secret_placeholder_treats_masked_values_as_placeholder():
|
||||
assert _is_secret_placeholder("••••••••", "••••1234") is True
|
||||
assert _is_secret_placeholder("********", "") is True
|
||||
assert _is_secret_placeholder(" * * * ", "") is True
|
||||
|
||||
|
||||
def test_secret_placeholder_preserves_real_keys_with_asterisks():
|
||||
assert _is_secret_placeholder("sk-live-*real-key*", "") is False
|
||||
assert _is_secret_placeholder("token_with*embedded*star", "") is False
|
||||
71
backend/tests/test_situational_alert_ai_brief.py
Normal file
71
backend/tests/test_situational_alert_ai_brief.py
Normal file
@@ -0,0 +1,71 @@
|
||||
import pytest
|
||||
|
||||
from app.services.situational_alert_ai_brief import build_situational_alert_brief_request
|
||||
|
||||
|
||||
class _SingleUseScalarResult:
|
||||
def __init__(self, value=0, rows=None):
|
||||
self.value = value
|
||||
self.rows = rows or []
|
||||
self.scalar_calls = 0
|
||||
|
||||
def scalar(self):
|
||||
self.scalar_calls += 1
|
||||
if self.scalar_calls > 1:
|
||||
raise AssertionError("scalar result was consumed more than once")
|
||||
return self.value
|
||||
|
||||
def fetchall(self):
|
||||
return self.rows
|
||||
|
||||
def scalar_one_or_none(self):
|
||||
return None
|
||||
|
||||
def scalars(self):
|
||||
rows = self.rows
|
||||
|
||||
class _Scalars:
|
||||
def all(self):
|
||||
return rows
|
||||
|
||||
return _Scalars()
|
||||
|
||||
|
||||
class _FakeBriefSession:
|
||||
def __init__(self):
|
||||
self._results = [
|
||||
_SingleUseScalarResult(3),
|
||||
_SingleUseScalarResult(2),
|
||||
_SingleUseScalarResult(rows=[]),
|
||||
_SingleUseScalarResult(rows=[]),
|
||||
_SingleUseScalarResult(rows=[]),
|
||||
_SingleUseScalarResult(4),
|
||||
_SingleUseScalarResult(1),
|
||||
_SingleUseScalarResult(rows=[]),
|
||||
_SingleUseScalarResult(rows=[]),
|
||||
_SingleUseScalarResult(5),
|
||||
_SingleUseScalarResult(2),
|
||||
_SingleUseScalarResult(rows=[]),
|
||||
_SingleUseScalarResult(),
|
||||
]
|
||||
|
||||
async def execute(self, _query):
|
||||
return self._results.pop(0)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_situational_alert_brief_builder_reuses_counts_without_reconsuming_results(monkeypatch):
|
||||
monkeypatch.setattr(
|
||||
"app.services.situational_alert_ai_brief.get_latest_bgp_brief_record",
|
||||
lambda: None,
|
||||
)
|
||||
|
||||
request, facts, context = await build_situational_alert_brief_request(_FakeBriefSession())
|
||||
|
||||
assert request.title == "态势告警 AI 简报"
|
||||
assert "总告警 3 条,active 2 条" in facts[0]
|
||||
assert "累计 incidents 4 条,active incidents 1 条" in facts[1]
|
||||
assert "累计 anomalies 5 条,active anomalies 2 条" in facts[2]
|
||||
assert context["active_system_alerts"] == 2
|
||||
assert context["active_bgp_incidents"] == 1
|
||||
assert context["active_bgp_anomalies"] == 2
|
||||
@@ -574,13 +574,12 @@ async def test_vessel_snapshot_filters_type_and_bbox(monkeypatch):
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_legacy_vessels_geojson_endpoint_is_gone():
|
||||
async def test_legacy_vessels_geojson_route_is_not_registered():
|
||||
transport = ASGITransport(app=app)
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/api/v1/visualization/geo/vessels")
|
||||
|
||||
assert response.status_code == 410
|
||||
assert "/api/v1/vessels/snapshot" in response.json()["detail"]
|
||||
assert response.status_code == 404
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@@ -669,15 +668,49 @@ async def test_vessel_snapshot_filters_bbox_and_caps_limit(monkeypatch):
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_legacy_vessels_geojson_rejects_even_with_bbox():
|
||||
transport = ASGITransport(app=app)
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get(
|
||||
"/api/v1/visualization/geo/vessels",
|
||||
params={"bbox": "10,59,11,60", "type": "cargo", "limit": 1000},
|
||||
)
|
||||
async def test_vessel_snapshot_uses_legacy_fallback_when_raw_window_is_empty(monkeypatch):
|
||||
now = datetime(2026, 4, 28, 1, 0, tzinfo=timezone.utc)
|
||||
monkeypatch.setattr(
|
||||
visualization,
|
||||
"get_aggregated_vessels_snapshot",
|
||||
AsyncMock(return_value=[]),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
visualization,
|
||||
"_load_legacy_vessel_snapshot_features",
|
||||
AsyncMock(
|
||||
return_value=[
|
||||
{
|
||||
"type": "Feature",
|
||||
"id": 257123000,
|
||||
"geometry": {"type": "Point", "coordinates": [10.73, 59.91]},
|
||||
"properties": {
|
||||
"mmsi": 257123000,
|
||||
"name": "OSLO TRADER",
|
||||
"vessel_type": 70,
|
||||
"vessel_type_name": "Cargo",
|
||||
"received_at": now.isoformat(),
|
||||
},
|
||||
}
|
||||
]
|
||||
),
|
||||
)
|
||||
|
||||
assert response.status_code == 410
|
||||
result = await visualization.build_vessel_snapshot_response(
|
||||
object(),
|
||||
bbox=(10.0, 59.0, 11.0, 60.0),
|
||||
zoom=12,
|
||||
type_filter=None,
|
||||
limit=1000,
|
||||
since_minutes=60,
|
||||
)
|
||||
|
||||
assert result["count"] == 1
|
||||
assert result["features"][0]["properties"]["name"] == "OSLO TRADER"
|
||||
assert result["diagnostics"]["raw_feature_count"] == 0
|
||||
assert result["diagnostics"]["legacy_feature_count"] == 1
|
||||
assert result["diagnostics"]["legacy_backfilled_mmsi"] == 1
|
||||
assert result["diagnostics"]["legacy_fallback_used"] is True
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
|
||||
@@ -46,6 +46,21 @@ def test_web_search_key_prefers_provider_env(isolated_web_search_env_files):
|
||||
assert source == "env_file"
|
||||
|
||||
|
||||
def test_web_search_generic_key_only_applies_to_default_provider(isolated_web_search_env_files):
|
||||
isolated_web_search_env_files.write_text(
|
||||
"WEB_SEARCH_API_KEY=generic-search-key\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
default_value, default_source = _resolve_web_search_api_key("tavily", {"api_key": ""}, "tavily")
|
||||
other_value, other_source = _resolve_web_search_api_key("brave", {"api_key": ""}, "tavily")
|
||||
|
||||
assert default_value == "generic-search-key"
|
||||
assert default_source == "env_file"
|
||||
assert other_value == ""
|
||||
assert other_source == ""
|
||||
|
||||
|
||||
def test_build_web_search_payload_keeps_saved_key_when_preview_submitted():
|
||||
current = {
|
||||
"web_search": {
|
||||
@@ -71,6 +86,35 @@ def test_build_web_search_payload_keeps_saved_key_when_preview_submitted():
|
||||
assert payload["providers"]["tavily"]["api_key"] == "tvly-old-secret"
|
||||
|
||||
|
||||
def test_build_web_search_payload_saves_provider_without_changing_default():
|
||||
current = {
|
||||
"web_search": {
|
||||
"default_provider": "tavily",
|
||||
"providers": {
|
||||
"tavily": {
|
||||
"provider": "tavily",
|
||||
"api_key": "tvly-key",
|
||||
},
|
||||
"brave": {
|
||||
"provider": "brave",
|
||||
"api_key": "",
|
||||
},
|
||||
},
|
||||
}
|
||||
}
|
||||
update = WebSearchIntegrationUpdate(
|
||||
enabled=True,
|
||||
provider="brave",
|
||||
base_url="https://api.search.brave.com/res/v1/web/search",
|
||||
api_key="brave-new-key",
|
||||
)
|
||||
|
||||
payload = _build_web_search_payload(current, update)
|
||||
|
||||
assert payload["default_provider"] == "tavily"
|
||||
assert payload["providers"]["brave"]["api_key"] == "brave-new-key"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_tavily_adapter_normalizes_results(monkeypatch):
|
||||
config = WebSearchConfig(
|
||||
|
||||
92
config/earth-boundary-pov-policy.china-v1.json
Normal file
92
config/earth-boundary-pov-policy.china-v1.json
Normal file
@@ -0,0 +1,92 @@
|
||||
{
|
||||
"schema": "planet-earth-boundary-pov-policy/v1",
|
||||
"profile": "china-pov-v1",
|
||||
"description": "Product boundary policy for the China POV Earth boundary build. This file declares intent only; geometry must come from audited source packages and be applied offline before PMTiles/MVT generation.",
|
||||
"defaultCountryHandling": "source-admin0-with-reviewed-overrides",
|
||||
"rules": [
|
||||
{
|
||||
"id": "china-zangnan",
|
||||
"name": "Zangnan / South Tibet",
|
||||
"action": "union_to_country",
|
||||
"targetIsoA3": "CHN",
|
||||
"subtractFromIsoA3": ["IND"],
|
||||
"hoverIsoA3": "CHN",
|
||||
"labelPolicy": "show_country_only"
|
||||
},
|
||||
{
|
||||
"id": "china-aksai-chin",
|
||||
"name": "Aksai Chin",
|
||||
"action": "union_to_country",
|
||||
"targetIsoA3": "CHN",
|
||||
"subtractFromIsoA3": ["IND"],
|
||||
"hoverIsoA3": "CHN",
|
||||
"labelPolicy": "show_country_only"
|
||||
},
|
||||
{
|
||||
"id": "china-taiwan-penghu",
|
||||
"name": "Taiwan and Penghu",
|
||||
"action": "union_to_country",
|
||||
"targetIsoA3": "CHN",
|
||||
"hoverIsoA3": "CHN",
|
||||
"labelPolicy": "show_country_only"
|
||||
},
|
||||
{
|
||||
"id": "china-diaoyu-chiwei",
|
||||
"name": "Diaoyu Dao, affiliated islands, and Chiwei Yu",
|
||||
"action": "union_to_country",
|
||||
"targetIsoA3": "CHN",
|
||||
"hoverIsoA3": "CHN",
|
||||
"labelPolicy": "show_country_only"
|
||||
},
|
||||
{
|
||||
"id": "china-south-china-sea-islands",
|
||||
"name": "Dongsha, Xisha, Zhongsha, Nansha, Huangyan Dao, Zengmu Ansha and related islands/reefs",
|
||||
"action": "union_to_country",
|
||||
"targetIsoA3": "CHN",
|
||||
"hoverIsoA3": "CHN",
|
||||
"labelPolicy": "show_country_only"
|
||||
},
|
||||
{
|
||||
"id": "china-maritime-claim-line",
|
||||
"name": "South China Sea dashed maritime claim line",
|
||||
"action": "render_claim_line",
|
||||
"targetIsoA3": "CHN",
|
||||
"geometryRole": "claim_line_only",
|
||||
"landPolygonEffect": "none"
|
||||
},
|
||||
{
|
||||
"id": "kosovo",
|
||||
"name": "Kosovo",
|
||||
"action": "render_as_disputed_with_parent",
|
||||
"parentIsoA3": "SRB",
|
||||
"hoverIsoA3": "SRB",
|
||||
"boundaryStyle": "disputed_internal",
|
||||
"labelPolicy": "show_parent_country"
|
||||
},
|
||||
{
|
||||
"id": "gaza",
|
||||
"name": "Gaza Strip",
|
||||
"action": "render_as_region_of_country",
|
||||
"targetIsoA3": "PSE",
|
||||
"hoverIsoA3": "PSE",
|
||||
"boundaryStyle": "admin_or_disputed",
|
||||
"labelPolicy": "show_country_only"
|
||||
}
|
||||
],
|
||||
"sourceRequirements": {
|
||||
"geometryMustBeAudited": true,
|
||||
"noHandDrawnClaimLines": true,
|
||||
"noFrontendRuntimePovPatch": true,
|
||||
"artifactIsolation": "one PMTiles/MVT artifact per POV profile"
|
||||
},
|
||||
"officialPositionNotes": [
|
||||
{
|
||||
"id": "kosovo",
|
||||
"note": "China has emphasized respect for Serbia's sovereignty and territorial integrity and the framework of UNSC Resolution 1244."
|
||||
},
|
||||
{
|
||||
"id": "gaza",
|
||||
"note": "China supports the two-state solution and an independent State of Palestine based on the 1967 borders with East Jerusalem as its capital; Gaza governance should follow Palestinians governing Palestine."
|
||||
}
|
||||
]
|
||||
}
|
||||
112
config/earth-boundary-sources.example.json
Normal file
112
config/earth-boundary-sources.example.json
Normal file
@@ -0,0 +1,112 @@
|
||||
{
|
||||
"policy": {
|
||||
"runtimeFetch": false,
|
||||
"profile": "china-pov-v1",
|
||||
"povPolicyPath": "config/earth-boundary-pov-policy.china-v1.json",
|
||||
"productionTileFormat": "pmtiles+mvt",
|
||||
"debugTileFormat": "geojson-directory",
|
||||
"description": "Default Earth boundary update sources. These public Natural Earth endpoints make local high-precision boundary download work out of the box; replace with audited internal sources for production if needed."
|
||||
},
|
||||
"collectorConfigs": {
|
||||
"earth_admin0_boundaries": {
|
||||
"displayName": "Earth Admin-0 国界源",
|
||||
"sourceKind": "admin0-boundaries",
|
||||
"endpoint": "https://raw.githubusercontent.com/nvkelso/natural-earth-vector/master/geojson/ne_10m_admin_0_countries.geojson",
|
||||
"method": "GET",
|
||||
"headers": {},
|
||||
"auth_type": "none",
|
||||
"license": "Natural Earth public domain",
|
||||
"mapping_json": {
|
||||
"source": {
|
||||
"items_path": "$.features[*]"
|
||||
},
|
||||
"fields": {
|
||||
"source_id": {
|
||||
"path": "$.properties.id",
|
||||
"type": "string"
|
||||
},
|
||||
"name": {
|
||||
"path": "$.properties.name",
|
||||
"type": "string"
|
||||
},
|
||||
"geometry": {
|
||||
"path": "$.geometry",
|
||||
"type": "object"
|
||||
},
|
||||
"properties": {
|
||||
"path": "$.properties",
|
||||
"type": "object"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"earth_coastline": {
|
||||
"displayName": "Earth 海岸线源",
|
||||
"sourceKind": "coastline",
|
||||
"endpoint": "https://raw.githubusercontent.com/nvkelso/natural-earth-vector/master/geojson/ne_10m_coastline.geojson",
|
||||
"method": "GET",
|
||||
"headers": {},
|
||||
"auth_type": "none",
|
||||
"license": "Natural Earth public domain",
|
||||
"mapping_json": {
|
||||
"source": {
|
||||
"items_path": "$.features[*]"
|
||||
},
|
||||
"fields": {
|
||||
"source_id": {
|
||||
"path": "$.properties.id",
|
||||
"type": "string"
|
||||
},
|
||||
"name": {
|
||||
"path": "$.properties.name",
|
||||
"type": "string"
|
||||
},
|
||||
"geometry": {
|
||||
"path": "$.geometry",
|
||||
"type": "object"
|
||||
},
|
||||
"properties": {
|
||||
"path": "$.properties",
|
||||
"type": "object"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"earth_claim_lines": {
|
||||
"displayName": "Earth 主张线源",
|
||||
"sourceKind": "claim-lines",
|
||||
"endpoint": "https://raw.githubusercontent.com/nvkelso/natural-earth-vector/master/geojson/ne_10m_admin_0_boundary_lines_disputed_areas.geojson",
|
||||
"method": "GET",
|
||||
"headers": {},
|
||||
"auth_type": "none",
|
||||
"license": "Natural Earth public domain",
|
||||
"mapping_json": {
|
||||
"source": {
|
||||
"items_path": "$.features[*]"
|
||||
},
|
||||
"fields": {
|
||||
"source_id": {
|
||||
"path": "$.properties.id",
|
||||
"type": "string"
|
||||
},
|
||||
"name": {
|
||||
"path": "$.properties.name",
|
||||
"type": "string"
|
||||
},
|
||||
"geometry": {
|
||||
"path": "$.geometry",
|
||||
"type": "object"
|
||||
},
|
||||
"properties": {
|
||||
"path": "$.properties",
|
||||
"type": "object"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"notes": [
|
||||
"Earth can download these sources directly from the toolbar settings when no local source override exists.",
|
||||
"If tippecanoe/pmtiles are unavailable, the backend generates a GeoJSON high-precision package so the feature remains usable."
|
||||
]
|
||||
}
|
||||
6
deploy/helm/planet/Chart.yaml
Normal file
6
deploy/helm/planet/Chart.yaml
Normal file
@@ -0,0 +1,6 @@
|
||||
apiVersion: v2
|
||||
name: planet
|
||||
description: Planet situational awareness platform
|
||||
type: application
|
||||
version: 0.1.0
|
||||
appVersion: "0.52.0"
|
||||
33
deploy/helm/planet/templates/_helpers.tpl
Normal file
33
deploy/helm/planet/templates/_helpers.tpl
Normal file
@@ -0,0 +1,33 @@
|
||||
{{- define "planet.name" -}}
|
||||
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}}
|
||||
{{- end -}}
|
||||
|
||||
{{- define "planet.fullname" -}}
|
||||
{{- if .Values.fullnameOverride -}}
|
||||
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}}
|
||||
{{- else -}}
|
||||
{{- $name := include "planet.name" . -}}
|
||||
{{- if contains $name .Release.Name -}}
|
||||
{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
|
||||
{{- else -}}
|
||||
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
|
||||
{{- end -}}
|
||||
{{- end -}}
|
||||
{{- end -}}
|
||||
|
||||
{{- define "planet.labels" -}}
|
||||
app.kubernetes.io/name: {{ include "planet.name" . }}
|
||||
app.kubernetes.io/instance: {{ .Release.Name }}
|
||||
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
|
||||
app.kubernetes.io/managed-by: {{ .Release.Service }}
|
||||
helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
|
||||
{{- end -}}
|
||||
|
||||
{{- define "planet.selectorLabels" -}}
|
||||
app.kubernetes.io/name: {{ include "planet.name" . }}
|
||||
app.kubernetes.io/instance: {{ .Release.Name }}
|
||||
{{- end -}}
|
||||
|
||||
{{- define "planet.image" -}}
|
||||
{{- printf "%s/%s/%s:%s" .root.Values.global.imageRegistry .root.Values.global.imageNamespace .repository .root.Values.image.tag -}}
|
||||
{{- end -}}
|
||||
66
deploy/helm/planet/templates/aiprovider.yaml
Normal file
66
deploy/helm/planet/templates/aiprovider.yaml
Normal file
@@ -0,0 +1,66 @@
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: {{ include "planet.fullname" . }}-aiprovider
|
||||
labels:
|
||||
{{- include "planet.labels" . | nindent 4 }}
|
||||
app.kubernetes.io/component: aiprovider
|
||||
spec:
|
||||
replicas: {{ .Values.aiprovider.replicaCount }}
|
||||
selector:
|
||||
matchLabels:
|
||||
{{- include "planet.selectorLabels" . | nindent 6 }}
|
||||
app.kubernetes.io/component: aiprovider
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
{{- include "planet.selectorLabels" . | nindent 8 }}
|
||||
app.kubernetes.io/component: aiprovider
|
||||
spec:
|
||||
{{- with .Values.global.imagePullSecrets }}
|
||||
imagePullSecrets:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
containers:
|
||||
- name: aiprovider
|
||||
image: {{ include "planet.image" (dict "root" . "repository" .Values.aiprovider.image.repository) | quote }}
|
||||
imagePullPolicy: {{ .Values.image.pullPolicy }}
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 8010
|
||||
envFrom:
|
||||
- configMapRef:
|
||||
name: {{ include "planet.fullname" . }}-config
|
||||
- secretRef:
|
||||
name: {{ include "planet.fullname" . }}-secrets
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 10
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 30
|
||||
periodSeconds: 20
|
||||
resources:
|
||||
{{- toYaml .Values.aiprovider.resources | nindent 12 }}
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: {{ include "planet.fullname" . }}-aiprovider
|
||||
labels:
|
||||
{{- include "planet.labels" . | nindent 4 }}
|
||||
app.kubernetes.io/component: aiprovider
|
||||
spec:
|
||||
type: ClusterIP
|
||||
ports:
|
||||
- name: http
|
||||
port: {{ .Values.aiprovider.service.port }}
|
||||
targetPort: http
|
||||
selector:
|
||||
{{- include "planet.selectorLabels" . | nindent 4 }}
|
||||
app.kubernetes.io/component: aiprovider
|
||||
66
deploy/helm/planet/templates/backend.yaml
Normal file
66
deploy/helm/planet/templates/backend.yaml
Normal file
@@ -0,0 +1,66 @@
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: {{ include "planet.fullname" . }}-backend
|
||||
labels:
|
||||
{{- include "planet.labels" . | nindent 4 }}
|
||||
app.kubernetes.io/component: backend
|
||||
spec:
|
||||
replicas: {{ .Values.backend.replicaCount }}
|
||||
selector:
|
||||
matchLabels:
|
||||
{{- include "planet.selectorLabels" . | nindent 6 }}
|
||||
app.kubernetes.io/component: backend
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
{{- include "planet.selectorLabels" . | nindent 8 }}
|
||||
app.kubernetes.io/component: backend
|
||||
spec:
|
||||
{{- with .Values.global.imagePullSecrets }}
|
||||
imagePullSecrets:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
containers:
|
||||
- name: backend
|
||||
image: {{ include "planet.image" (dict "root" . "repository" .Values.backend.image.repository) | quote }}
|
||||
imagePullPolicy: {{ .Values.image.pullPolicy }}
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 8000
|
||||
envFrom:
|
||||
- configMapRef:
|
||||
name: {{ include "planet.fullname" . }}-config
|
||||
- secretRef:
|
||||
name: {{ include "planet.fullname" . }}-secrets
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 10
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 30
|
||||
periodSeconds: 20
|
||||
resources:
|
||||
{{- toYaml .Values.backend.resources | nindent 12 }}
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: {{ include "planet.fullname" . }}-backend
|
||||
labels:
|
||||
{{- include "planet.labels" . | nindent 4 }}
|
||||
app.kubernetes.io/component: backend
|
||||
spec:
|
||||
type: ClusterIP
|
||||
ports:
|
||||
- name: http
|
||||
port: {{ .Values.backend.service.port }}
|
||||
targetPort: http
|
||||
selector:
|
||||
{{- include "planet.selectorLabels" . | nindent 4 }}
|
||||
app.kubernetes.io/component: backend
|
||||
27
deploy/helm/planet/templates/configmap.yaml
Normal file
27
deploy/helm/planet/templates/configmap.yaml
Normal file
@@ -0,0 +1,27 @@
|
||||
{{- $postgresHost := .Values.postgresql.external.host -}}
|
||||
{{- if .Values.postgresql.internal.enabled -}}
|
||||
{{- $postgresHost = printf "%s-postgresql" (include "planet.fullname" .) -}}
|
||||
{{- end -}}
|
||||
{{- $redisHost := .Values.redis.external.host -}}
|
||||
{{- if .Values.redis.internal.enabled -}}
|
||||
{{- $redisHost = printf "%s-redis" (include "planet.fullname" .) -}}
|
||||
{{- end -}}
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: {{ include "planet.fullname" . }}-config
|
||||
labels:
|
||||
{{- include "planet.labels" . | nindent 4 }}
|
||||
data:
|
||||
DATABASE_URL: "postgresql+asyncpg://{{ .Values.postgresql.external.username }}:{{ .Values.postgresql.external.password }}@{{ $postgresHost }}:{{ .Values.postgresql.external.port }}/{{ .Values.postgresql.external.database }}"
|
||||
REDIS_SERVER: {{ $redisHost | quote }}
|
||||
REDIS_PORT: {{ .Values.redis.external.port | quote }}
|
||||
REDIS_DB: {{ .Values.redis.external.db | quote }}
|
||||
AI_BASE_URL: "http://{{ include "planet.fullname" . }}-aiprovider:{{ .Values.aiprovider.service.port }}"
|
||||
AI_PROVIDER_API: "http://{{ include "planet.fullname" . }}-aiprovider:{{ .Values.aiprovider.service.port }}"
|
||||
{{- range $key, $value := .Values.backend.env }}
|
||||
{{ $key }}: {{ $value | quote }}
|
||||
{{- end }}
|
||||
{{- range $key, $value := .Values.aiprovider.env }}
|
||||
{{ $key }}: {{ $value | quote }}
|
||||
{{- end }}
|
||||
61
deploy/helm/planet/templates/frontend.yaml
Normal file
61
deploy/helm/planet/templates/frontend.yaml
Normal file
@@ -0,0 +1,61 @@
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: {{ include "planet.fullname" . }}-frontend
|
||||
labels:
|
||||
{{- include "planet.labels" . | nindent 4 }}
|
||||
app.kubernetes.io/component: frontend
|
||||
spec:
|
||||
replicas: {{ .Values.frontend.replicaCount }}
|
||||
selector:
|
||||
matchLabels:
|
||||
{{- include "planet.selectorLabels" . | nindent 6 }}
|
||||
app.kubernetes.io/component: frontend
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
{{- include "planet.selectorLabels" . | nindent 8 }}
|
||||
app.kubernetes.io/component: frontend
|
||||
spec:
|
||||
{{- with .Values.global.imagePullSecrets }}
|
||||
imagePullSecrets:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
containers:
|
||||
- name: frontend
|
||||
image: {{ include "planet.image" (dict "root" . "repository" .Values.frontend.image.repository) | quote }}
|
||||
imagePullPolicy: {{ .Values.image.pullPolicy }}
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 3000
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 10
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 20
|
||||
periodSeconds: 20
|
||||
resources:
|
||||
{{- toYaml .Values.frontend.resources | nindent 12 }}
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: {{ include "planet.fullname" . }}-frontend
|
||||
labels:
|
||||
{{- include "planet.labels" . | nindent 4 }}
|
||||
app.kubernetes.io/component: frontend
|
||||
spec:
|
||||
type: ClusterIP
|
||||
ports:
|
||||
- name: http
|
||||
port: {{ .Values.frontend.service.port }}
|
||||
targetPort: http
|
||||
selector:
|
||||
{{- include "planet.selectorLabels" . | nindent 4 }}
|
||||
app.kubernetes.io/component: frontend
|
||||
29
deploy/helm/planet/templates/ingress.yaml
Normal file
29
deploy/helm/planet/templates/ingress.yaml
Normal file
@@ -0,0 +1,29 @@
|
||||
{{- if .Values.frontend.ingress.enabled }}
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: {{ include "planet.fullname" . }}
|
||||
labels:
|
||||
{{- include "planet.labels" . | nindent 4 }}
|
||||
{{- with .Values.frontend.ingress.annotations }}
|
||||
annotations:
|
||||
{{- toYaml . | nindent 4 }}
|
||||
{{- end }}
|
||||
spec:
|
||||
ingressClassName: {{ .Values.frontend.ingress.className | quote }}
|
||||
{{- with .Values.frontend.ingress.tls }}
|
||||
tls:
|
||||
{{- toYaml . | nindent 4 }}
|
||||
{{- end }}
|
||||
rules:
|
||||
- host: {{ .Values.frontend.ingress.host | quote }}
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: {{ include "planet.fullname" . }}-frontend
|
||||
port:
|
||||
name: http
|
||||
{{- end }}
|
||||
14
deploy/helm/planet/templates/secrets.yaml
Normal file
14
deploy/helm/planet/templates/secrets.yaml
Normal file
@@ -0,0 +1,14 @@
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: {{ include "planet.fullname" . }}-secrets
|
||||
labels:
|
||||
{{- include "planet.labels" . | nindent 4 }}
|
||||
type: Opaque
|
||||
stringData:
|
||||
{{- range $key, $value := .Values.backend.secretEnv }}
|
||||
{{ $key }}: {{ $value | quote }}
|
||||
{{- end }}
|
||||
{{- range $key, $value := .Values.aiprovider.secretEnv }}
|
||||
{{ $key }}: {{ $value | quote }}
|
||||
{{- end }}
|
||||
116
deploy/helm/planet/templates/single-node-deps.yaml
Normal file
116
deploy/helm/planet/templates/single-node-deps.yaml
Normal file
@@ -0,0 +1,116 @@
|
||||
{{- if .Values.postgresql.internal.enabled }}
|
||||
apiVersion: apps/v1
|
||||
kind: StatefulSet
|
||||
metadata:
|
||||
name: {{ include "planet.fullname" . }}-postgresql
|
||||
labels:
|
||||
{{- include "planet.labels" . | nindent 4 }}
|
||||
app.kubernetes.io/component: postgresql
|
||||
spec:
|
||||
serviceName: {{ include "planet.fullname" . }}-postgresql
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
{{- include "planet.selectorLabels" . | nindent 6 }}
|
||||
app.kubernetes.io/component: postgresql
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
{{- include "planet.selectorLabels" . | nindent 8 }}
|
||||
app.kubernetes.io/component: postgresql
|
||||
spec:
|
||||
containers:
|
||||
- name: postgresql
|
||||
image: postgres:15
|
||||
ports:
|
||||
- name: postgres
|
||||
containerPort: 5432
|
||||
env:
|
||||
- name: POSTGRES_USER
|
||||
value: {{ .Values.postgresql.external.username | quote }}
|
||||
- name: POSTGRES_PASSWORD
|
||||
value: {{ .Values.postgresql.external.password | quote }}
|
||||
- name: POSTGRES_DB
|
||||
value: {{ .Values.postgresql.external.database | quote }}
|
||||
readinessProbe:
|
||||
exec:
|
||||
command: ["pg_isready", "-U", {{ .Values.postgresql.external.username | quote }}]
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 10
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /var/lib/postgresql/data
|
||||
volumeClaimTemplates:
|
||||
- metadata:
|
||||
name: data
|
||||
spec:
|
||||
accessModes: ["ReadWriteOnce"]
|
||||
resources:
|
||||
requests:
|
||||
storage: 8Gi
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: {{ include "planet.fullname" . }}-postgresql
|
||||
labels:
|
||||
{{- include "planet.labels" . | nindent 4 }}
|
||||
app.kubernetes.io/component: postgresql
|
||||
spec:
|
||||
ports:
|
||||
- name: postgres
|
||||
port: 5432
|
||||
targetPort: postgres
|
||||
selector:
|
||||
{{- include "planet.selectorLabels" . | nindent 4 }}
|
||||
app.kubernetes.io/component: postgresql
|
||||
{{- end }}
|
||||
{{- if .Values.redis.internal.enabled }}
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: {{ include "planet.fullname" . }}-redis
|
||||
labels:
|
||||
{{- include "planet.labels" . | nindent 4 }}
|
||||
app.kubernetes.io/component: redis
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
{{- include "planet.selectorLabels" . | nindent 6 }}
|
||||
app.kubernetes.io/component: redis
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
{{- include "planet.selectorLabels" . | nindent 8 }}
|
||||
app.kubernetes.io/component: redis
|
||||
spec:
|
||||
containers:
|
||||
- name: redis
|
||||
image: redis:7-alpine
|
||||
ports:
|
||||
- name: redis
|
||||
containerPort: 6379
|
||||
readinessProbe:
|
||||
exec:
|
||||
command: ["redis-cli", "ping"]
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 10
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: {{ include "planet.fullname" . }}-redis
|
||||
labels:
|
||||
{{- include "planet.labels" . | nindent 4 }}
|
||||
app.kubernetes.io/component: redis
|
||||
spec:
|
||||
ports:
|
||||
- name: redis
|
||||
port: 6379
|
||||
targetPort: redis
|
||||
selector:
|
||||
{{- include "planet.selectorLabels" . | nindent 4 }}
|
||||
app.kubernetes.io/component: redis
|
||||
{{- end }}
|
||||
29
deploy/helm/planet/values.single-node.yaml
Normal file
29
deploy/helm/planet/values.single-node.yaml
Normal file
@@ -0,0 +1,29 @@
|
||||
global:
|
||||
imageRegistry: gitea.rclaw.top
|
||||
imageNamespace: linkong/planet
|
||||
|
||||
image:
|
||||
tag: latest
|
||||
|
||||
frontend:
|
||||
ingress:
|
||||
enabled: true
|
||||
host: planet.local
|
||||
|
||||
postgresql:
|
||||
internal:
|
||||
enabled: true
|
||||
external:
|
||||
host: planet-postgresql
|
||||
port: 5432
|
||||
database: planet_db
|
||||
username: postgres
|
||||
password: postgres
|
||||
|
||||
redis:
|
||||
internal:
|
||||
enabled: true
|
||||
external:
|
||||
host: planet-redis
|
||||
port: 6379
|
||||
db: 0
|
||||
73
deploy/helm/planet/values.yaml
Normal file
73
deploy/helm/planet/values.yaml
Normal file
@@ -0,0 +1,73 @@
|
||||
global:
|
||||
imageRegistry: gitea.rclaw.top
|
||||
imageNamespace: linkong/planet
|
||||
imagePullSecrets: []
|
||||
|
||||
fullnameOverride: planet
|
||||
|
||||
image:
|
||||
tag: latest
|
||||
pullPolicy: IfNotPresent
|
||||
|
||||
frontend:
|
||||
replicaCount: 1
|
||||
image:
|
||||
repository: frontend
|
||||
service:
|
||||
port: 3000
|
||||
ingress:
|
||||
enabled: true
|
||||
className: nginx
|
||||
host: planet.example.com
|
||||
annotations: {}
|
||||
tls: []
|
||||
resources: {}
|
||||
|
||||
backend:
|
||||
replicaCount: 1
|
||||
image:
|
||||
repository: backend
|
||||
service:
|
||||
port: 8000
|
||||
env:
|
||||
PROJECT_NAME: Planet
|
||||
CORS_ORIGINS: '["*"]'
|
||||
secretEnv:
|
||||
SECRET_KEY: change-me
|
||||
resources: {}
|
||||
|
||||
aiprovider:
|
||||
replicaCount: 1
|
||||
image:
|
||||
repository: aiprovider
|
||||
service:
|
||||
port: 8010
|
||||
env:
|
||||
AI_PROVIDER: openai
|
||||
AI_MODEL: gpt-4o-mini
|
||||
secretEnv:
|
||||
AI_API_KEY: ""
|
||||
resources: {}
|
||||
|
||||
postgresql:
|
||||
internal:
|
||||
enabled: false
|
||||
external:
|
||||
host: postgres.example.com
|
||||
port: 5432
|
||||
database: planet_db
|
||||
username: postgres
|
||||
password: postgres
|
||||
|
||||
redis:
|
||||
internal:
|
||||
enabled: false
|
||||
external:
|
||||
host: redis.example.com
|
||||
port: 6379
|
||||
db: 0
|
||||
|
||||
smoke:
|
||||
frontendPath: /
|
||||
backendHealthPath: /health
|
||||
aiProviderHealthPath: /health
|
||||
@@ -8,6 +8,200 @@ This project follows the repository versioning rule:
|
||||
- `improvement` -> `+0.0.1`(bugfix + 小功能混合)
|
||||
- `bugfix` -> `+0.0.1`
|
||||
|
||||
## [0.63.1] — 2026-05-21
|
||||
|
||||
Released: 2026-05-21
|
||||
|
||||
### Highlights
|
||||
- 修复新设备初始化后 Admin Next 无法打开的问题:补上被 `.gitignore` 的 `lib/` 规则误忽略的 Admin Next utility module。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 将 `frontend/src/admin-next/lib/utils.ts` 纳入版本控制,恢复 `DashboardNext`、`AdminNextLayout` 等页面对 `cn` 和 `formatNumber` 的运行时依赖。
|
||||
- 避免 Vite dev server 在新 clone 环境中因缺失源码模块而对 `/src/admin-next/*` 返回 500,并导致动态导入 AdminNextRoutes 失败。
|
||||
|
||||
---
|
||||
|
||||
## [0.63.0] — 2026-05-21
|
||||
|
||||
Released: 2026-05-21
|
||||
|
||||
### Highlights
|
||||
- 新增 `./planet.sh init` 首次初始化入口,将 uv/bun 依赖同步、env 模板补齐、数据库容器启动、建表 seed 和默认用户生成串成一条空项目引导路径。
|
||||
- 新增 `./planet.sh destroy` 破坏性重置入口,带 CLI 确认保护,可清理 Planet 容器、卷、镜像和本地编译/运行状态,同时保留源码与 `.env` 配置。
|
||||
- 改进 `planet.sh` 日志体验,只在带状态标签的输出行末尾追加时间戳,并让 `init` 在应用服务已运行时自动跳过重复初始化。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- `init` 复用现有 Docker/依赖 helper,支持重复执行时不覆盖 env、不清空数据,并在完成后提示默认本地登录账号。
|
||||
- `destroy` 停止本地服务后清理 Docker compose 状态、项目镜像、数据卷、`.venv`、`node_modules`、前端构建产物和 Planet state/cache。
|
||||
- README 快速启动补充 `init` 和 `destroy` 命令说明,明确首次引导和重置路径。
|
||||
|
||||
---
|
||||
|
||||
## [0.62.0] — 2026-05-21
|
||||
|
||||
Released: 2026-05-21
|
||||
|
||||
### Highlights
|
||||
- Admin Next 转正为正式后台入口,旧 AntD 控制台完整迁移到 `/legacy/admin/*` 作为回退和对照,并保留 `/admin-next/*` 兼容重定向。
|
||||
- 新后台完成采集、AI Provider、Earth 内容、日志、BGP/告警、设置和认证链路的全量收口,新增 lazy tab loading、层级配置、移动端详情和 Markdown 文档渲染体验。
|
||||
- 抽出 Tactile UI 按钮、开关、tooltip、滚动条和表格滚动组件,为未来独立 npm 组件库打基础,同时补齐中英文开发文档。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 新增采集管理分层工作台,覆盖采集器配置、采集调度、采集历史/快照、凭证教程生成/重置、映射 propose/preview/create/activate 和真实连接/运行状态。
|
||||
- 改进 AI Provider 与工具调用配置,修复 key fallback、脱敏显示、轻量连通性测试、默认 provider、OpenCode Go 路由和 Playground Markdown 输出。
|
||||
- 更新 Earth 内容管理和 Earth 前台体验,支持品牌预览/上传/重置、TV 默认源与新增草稿、图层/新闻/3D 模型配置入口以及 live TV 预览。
|
||||
- 强化后端数据源、系统日志、WebSocket、AI client、位置 LLM fallback、Earth layer cache 和 datasource connectivity,并补充相关回归测试与运维脚本能力。
|
||||
|
||||
---
|
||||
|
||||
## [0.61.0] — 2026-05-18
|
||||
|
||||
Released: 2026-05-18
|
||||
|
||||
### Highlights
|
||||
- 新增 Earth 图层 Redis 读穿缓存、防击穿锁、stale 兜底和 payload budget,降低演示前重图层与船只 snapshot 对后端内存的冲击。
|
||||
- 保持前端原 API 不变,为海缆、登陆点、卫星、算力中心、BGP、summary 和船只 snapshot 增加透明缓存 header 可观测性。
|
||||
- 新增 super admin Earth 图层缓存状态与清理接口,并在采集写入后按 source 主动失效相关缓存。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 新增 `earth:layer:v1:*` 缓存命名空间、fresh/stale 双 key、Redis 故障 bypass 和 OOM 防护诊断。
|
||||
- 船只 snapshot 使用短 TTL、bbox 量化和响应预算,避免重复视窗请求和超大 payload 触发后端 OOM。
|
||||
- 补充 Earth layer cache 计划文档与后端测试,覆盖 hit、refresh、stale、bypass、锁竞争、cache header 和运维清理。
|
||||
|
||||
---
|
||||
|
||||
## [0.60.0] — 2026-05-17
|
||||
|
||||
Released: 2026-05-17
|
||||
|
||||
### Highlights
|
||||
- 新增 Earth 内容与国界运行体验:品牌内容配置、国界精度滑块、高精国界首次应用按钮和高精重载构建入口共同收口。
|
||||
- 完成新闻中文展示链路收敛:滚动新闻改用一句话摘要,新闻本地化状态避免未完成内容进入滚动展示,英文原文继续保留在数据层。
|
||||
- 重构 AI Provider 与提示词边界:`aiprovider` 保持纯净模型适配,业务提示词集中到后端 AI task 默认注册表并支持运维台覆盖和重置。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 新增 AI 设置提示词 tab,按功能入口选择、编辑、保存和重置默认提示词,并修复态势告警 / BGP 简报展示生成内容而非 prompt。
|
||||
- 更新 Earth 新闻、HUD、巡航、桌面详情和移动详情的中文显示策略,并优化新闻摘要提示词为新闻式一句话概要。
|
||||
- 补充 Agent Runtime、Earth LLM 指令、语音识别和多角色态势感知的详细计划,并同步 README 架构说明、用户手册、FAQ 与开发者文档。
|
||||
- 补齐相关后端测试,覆盖 Earth 品牌配置、AI 简报接口和态势告警 AI 简报生成链路。
|
||||
|
||||
---
|
||||
|
||||
## [0.59.0] — 2026-05-16
|
||||
|
||||
Released: 2026-05-16
|
||||
|
||||
### Highlights
|
||||
- 将 Earth 国界从采集器体系迁移为 Earth 静态资产,恢复低精 GeoJSON fallback,并新增 Earth 工具栏高精国界下载/构建进度与热应用。
|
||||
- 重组后台“运维与配置”:新增 Earth 内容与采集管理二级入口,电视直播、国界精度、采集器、采集调度各归其位,未接入模块以占位页呈现。
|
||||
- 新增 AI task prompt 覆盖管理,按稳定 task key 管理新闻汉化、告警研判、BGP 简报等业务提示词,避免全局 prompt 污染。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- Earth 新闻锚点链路增加队列化 enrichment 状态、Redis Streams 后台精修和 WebSocket patch 语义,前端汉化/锚点策略更稳定。
|
||||
- 国界 hover 与 interactable tooltip 解耦,鼠标位于国家 polygon 内时保持国界高亮,同时卫星/船只/BGP 等对象仍可显示自身信息。
|
||||
- 新增 `/api/v1/earth/boundaries/*` 状态、配置、构建和进度接口,并在启动初始化中清理旧 boundary datasource/task/snapshot 历史入口。
|
||||
- 补齐中英文用户手册、FAQ、quickstart、运维手册和开发者上下文文档,明确用户 UI、运维操作和开发者稳定边界。
|
||||
|
||||
---
|
||||
|
||||
## [0.58.0] — 2026-05-15
|
||||
|
||||
Released: 2026-05-15
|
||||
|
||||
### Highlights
|
||||
- 新增 Earth 高精度国界 PMTiles/MVT 前端链路,移除旧低精度 GeoJSON 国界兜底,国界缺失时显式报错。
|
||||
- 将 Earth 边界数据拆成 Admin-0、coastline、claim-lines 三个标准源采集器,并把 `earth_boundary_tiles` 收口为下游 PMTiles 构建器。
|
||||
- 修复 Earth 远距缩放下海陆基座与高清贴图 z-fighting 导致的雪花/黑块闪烁,并记录地表多层 shell 的深度间距规则。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 新增 `earth_boundary_source` 目标 schema、China POV policy 配置、PMTiles readiness/build 脚本和 collector artifact 登记流程。
|
||||
- Earth 新闻巡航改为优先使用可缓存的目标地点解析队列,并补充媒体新闻归档采集器与回归测试。
|
||||
- 调整数据源列表任务状态展示,让 Earth 采集器失败/未就绪状态可见,不再表现为“未执行”。
|
||||
- 更新中英文采集器、数据源设置、Earth 图层顺序、运维 runbook、FAQ、规则和计划文档。
|
||||
|
||||
---
|
||||
|
||||
## [0.57.0] — 2026-05-14
|
||||
|
||||
Released: 2026-05-14
|
||||
|
||||
### Highlights
|
||||
- 新增 WSL `--allow-lan` 临时 Windows relay,保持本机 `localhost:3000` / `localhost:8000` 不变,同时用 Windows 局域网 IP 暴露相同端口。
|
||||
- 启动脚本会检测旧 `netsh interface portproxy` 冲突并请求管理员 PowerShell 清理,避免 `svchost.exe / iphlpsvc` 持久占用 3000/8000。
|
||||
- 修复 Vite CJS Node API deprecated warning,将前端配置迁移到 ESM,并让脚本按实际后端端口注入代理目标。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 新增 [scripts/windows-lan-relay.ps1](/home/ray/dev/linkong/planet/scripts/windows-lan-relay.ps1),在 Windows 侧启动随 WSL 服务健康状态自动退出的 TCP relay。
|
||||
- `planet.sh --allow-lan` 仅在 WSL + PowerShell 可用时启用 Windows relay,并自动检查 Windows 防火墙规则;非 WSL 环境保持原有路径。
|
||||
- 更新中英文 README、FAQ 和运维文档,说明旧 portproxy 清理、UAC 防火墙授权、同端口 localhost/LAN 访问和故障恢复方式。
|
||||
|
||||
---
|
||||
|
||||
## [0.56.0] — 2026-05-13
|
||||
|
||||
Released: 2026-05-13
|
||||
|
||||
### Highlights
|
||||
- 修复 Earth 卫星 SGP4 坐标口径,当前点/短尾迹使用地固坐标,锁定预测轨道使用固定地球姿态下的闭合惯性轨道。
|
||||
- 优化真实高度压缩显示上限,将高轨显示控制在地球半径外约四分之一,保持 GEO/MEO/LEO 分层同时避免轨迹过远。
|
||||
- 统一 BGP 光晕与图标色调,并更新超算中心建筑图标和 Earth 新闻/HUD 面板体验。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 修正卫星详情卡近地点/远地点高度计算,避免把轨道半径误显示为离地高度。
|
||||
- 调整 BGP event / collector halo 的浅色派生规则,使红色事件、橙色活跃观测站和蓝色 idle 观测站保持各自色相。
|
||||
- 补充中英文用户手册、FAQ、Earth frontend context、render order、layer style reference 和相关计划文档。
|
||||
|
||||
---
|
||||
|
||||
## [0.55.0] — 2026-05-13
|
||||
|
||||
Released: 2026-05-13
|
||||
|
||||
### Highlights
|
||||
- 新增 Earth 卫星真实高度显示开关,默认按 TLE/SGP4 真实轨道高度压缩分层,关闭后恢复旧版同层球面。
|
||||
- 将卫星轨迹显示保留在设置面板,并让轨迹、预测轨道和锁定视觉跟随真实高度开关即时刷新。
|
||||
- 修复 Earth 设置面板 switch 圆点垂直居中问题,并同步更新用户手册、FAQ 和 Earth 开发者文档。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 新增 `satelliteRealAltitudeEnabled` 本地偏好和真实高度压缩参数,保留 TLE 缺失/传播失败时的 fallback 固定高度。
|
||||
- 更新卫星渲染高度、轨迹、预测轨道和设置持久化链路,避免同一轨迹混入两套高度模型。
|
||||
- 补充中英文 manual、quickstart、FAQ、Earth frontend context、render order 和 layer style reference。
|
||||
|
||||
---
|
||||
|
||||
## [0.54.0] — 2026-05-13
|
||||
|
||||
Released: 2026-05-13
|
||||
|
||||
### Highlights
|
||||
- 新增 Earth 轨迹显示设置,将卫星轨迹从图层列表迁移到桌面与移动设置面板,便于独立控制轨迹线显示。
|
||||
- 优化卫星静止呼吸与低缩放圆点闪烁参数,让卫星态势视觉节奏更柔和、层次更清晰。
|
||||
- 更新算力中心图标资产,统一 Earth 交互图标在高分辨率显示下的视觉规格。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 新增轨迹显示偏好持久化,并在卫星图层不可用时同步禁用轨迹设置入口。
|
||||
- 调整卫星呼吸、透明度和 twinkle 参数,降低闪烁频率并增强可读性。
|
||||
- 将轨迹控制状态同步到桌面设置、移动设置和图层卡片刷新流程。
|
||||
|
||||
---
|
||||
|
||||
## [0.53.0] — 2026-05-13
|
||||
|
||||
Released: 2026-05-13
|
||||
|
||||
### Highlights
|
||||
- 新增正式交付基线:Gitea Actions CI、镜像发布、staging 自动部署,以及 Kubernetes Helm chart。
|
||||
- 前端生产镜像改为 `vite build` 静态产物 + nginx 托管,后端与 AI Provider 镜像移除开发 reload 并加入容器健康检查。
|
||||
- 明确 `planet.sh` 只作为开发启动入口,生产环境交给 Kubernetes Service、Ingress、readiness/liveness probe 和 rollout 管理。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 新增 frontend/backend/aiprovider 镜像 build smoke、Helm lint/template、staging rollout 与 HTTP smoke test 流水线。
|
||||
- 新增 Helm values、single-node 演示依赖、ConfigMap/Secret 引用、ClusterIP 服务和 frontend Ingress。
|
||||
- 更新运维文档与交付计划,保留 Vite 生产构建路线,不新增 Webpack 双构建链,Electron 暂不进入主线。
|
||||
- 继续收口启动脚本风险:状态目录、健康检查端口默认值、PID 校验、端口诊断和开发环境跨平台边界说明。
|
||||
|
||||
---
|
||||
|
||||
## [0.52.0] — 2026-05-12
|
||||
|
||||
Released: 2026-05-12
|
||||
|
||||
@@ -19,7 +19,7 @@
|
||||
- [earth-mobile-drawer-ui-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-mobile-drawer-ui-plan.md)
|
||||
- [earth-compute-center-bgp-style-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-compute-center-bgp-style-plan.md)
|
||||
- [earth-renderer-architecture-separation-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-renderer-architecture-separation-plan.md)
|
||||
- [earth-country-boundary-overlay-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-country-boundary-overlay-plan.md)
|
||||
- [Earth 高精度国界静态瓦片计划](/home/ray/dev/linkong/planet/docs/plans/earth-high-precision-boundary-tiles-plan.md)
|
||||
- [earth-predicted-orbit-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-predicted-orbit-plan.md)
|
||||
- [earth-webgl-instancing-satellites-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-webgl-instancing-satellites-plan.md)
|
||||
- [earth-real-terrain-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-real-terrain-plan.md)
|
||||
@@ -35,7 +35,11 @@
|
||||
- [Docs Gatekeeper 鉴权系统计划](/home/ray/dev/linkong/planet/docs/plans/docs-gatekeeper-auth-plan.md)
|
||||
- [Location Resolver 共享管线计划](/home/ray/dev/linkong/planet/docs/plans/location-resolver-shared-pipeline-plan.md)
|
||||
- [frontend-ai-playground-development-plan.md](/home/ray/dev/linkong/planet/docs/plans/frontend-ai-playground-development-plan.md)
|
||||
- [AI Provider OpenClaw-Style Routing Plan](/home/ray/dev/linkong/planet/docs/plans/ai-provider-openclaw-style-routing-plan.md)
|
||||
- [统一集成配置 Schema 系统计划](/home/ray/dev/linkong/planet/docs/plans/integration-config-schema-system-plan.md)
|
||||
- [Lightweight Agent Orchestrator 与 WebSearch 证据层计划](/home/ray/dev/linkong/planet/docs/plans/agents-light-orchestrator-websearch-plan.md)
|
||||
- [Admin Next Parity Checklist](/home/ray/dev/linkong/planet/docs/plans/admin-next-parity-checklist.md)
|
||||
- [Admin Next Parity Audit Closeout](/home/ray/dev/linkong/planet/docs/plans/admin-next-parity-audit-closeout-plan.md)
|
||||
- [ue5-mvp-fused-plan.md](/home/ray/dev/linkong/planet/docs/plans/ue5-mvp-fused-plan.md)
|
||||
|
||||
不适合放入这里的内容:
|
||||
|
||||
128
docs/plans/admin-next-dual-track-full-migration-plan.md
Normal file
128
docs/plans/admin-next-dual-track-full-migration-plan.md
Normal file
@@ -0,0 +1,128 @@
|
||||
# Admin Next 双轨全量迁移修正计划
|
||||
|
||||
## Summary
|
||||
|
||||
`/admin-next/*` 继续作为新版影子路由开发,旧 AntD 控制台必须完整保留作为生产回退和行为对照。只有新版完成全功能 parity 并通过验收后,才能逐个切换旧路由;删除旧页面和移除 AntD 必须作为最后独立步骤,并等待明确确认。
|
||||
|
||||
## Hard Rules
|
||||
|
||||
- 未完成 parity checklist 前,不删除旧页面、不移除 AntD、不切旧入口。
|
||||
- “旧版能删除”只表示新版能力达到替代标准,不表示立即删除旧版。
|
||||
- 新版不能只做看板;旧版里可新增、编辑、删除、测试、触发、批量、轮询、WebSocket、权限控制、错误提示和确认弹窗的能力,都必须迁移。
|
||||
- `/admin-next/*` 可以重做交互和视觉,但业务语义、API payload、权限和危险操作保护必须对齐旧版。
|
||||
|
||||
## Design Direction
|
||||
|
||||
- 风格采用“朴素工具化 + 轻微立体触感”:灰白/暗色低对比背景、清晰 1px 边框、轻微凸起/按下态、克制圆角、图标少量彩色。
|
||||
- 禁止回到 soft-glass、hero、大发光、大渐变、大色块和装饰性卡片。
|
||||
- 常规动作默认 icon-only + tooltip:刷新、重启、退出、复制、查看、编辑、删除、关闭、设置。
|
||||
- 强意图动作保留实心文字按钮:保存、创建、确认、执行、测试连接。
|
||||
- 页面统一 `16px` 外边距、`16px` 主区块间距、`12px` 面板 header/body 间距;表格标题和表体不能贴边。
|
||||
- 所有页面遵守一屏工作台:`PageHeader` 之外只允许一个主内容区域吃满剩余高度;详情区、表格区和 Playground 消息区内部滚动,不能被父级裁掉。
|
||||
- 页内 tabs 必须按旧版用户心智组织,而不是按后端接口名拆散;接口状态只能作为分区内部信息。
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
1. 恢复旧版安全网:恢复旧 AntD 页面、旧 `AppLayout`、旧 helper、AntD 依赖和旧路由;保留 `/admin-next/*` 影子路由。
|
||||
2. 建立新版基础层:整理 admin-next token、按钮、tooltip、theme switch、sidebar account、`EntityTable`、`FormDialog`、`ConfirmDialog`、`DetailPanel`、移动端表格/卡片切换。
|
||||
3. 拆掉通用看板页:每个模块建立真实业务页面、API adapter、form/mutation、轮询或 WebSocket 逻辑,不再用万能 Resource table 聚合展示。
|
||||
4. 按模块迁移:
|
||||
- Dashboard:统计、WebSocket、健康检查、重启任务、任务日志、恢复探测。
|
||||
- DataSources:内置源/自定义源/实时源、详情、stats、task-status、启停、触发、批量触发、清理数据。
|
||||
- DataList:列表、summary、sources/types/countries、搜索、筛选、分页、详情、导出、分布。
|
||||
- Collection Management:configs CRUD、builtin connect/test、target schemas、mappings、run/stop、stream status。
|
||||
- Settings:system、notifications、security、SMTP、TV、integrations、collectors、测试连接、凭证指南。
|
||||
- AI:provider、web search、OCR、prompt registry、playground 完整会话操作。
|
||||
- Earth Content:brand、upload、delete/reset、boundary config/build/status、TV/content。
|
||||
- Logs:sources、level/date/search、刷新、详情、复制、tail/snapshot。
|
||||
- BGP:overview、collectors、incidents、anomalies、events、AI briefs。
|
||||
- Alerts:系统告警、BGP 告警、态势告警,包含 ack/resolve/stats/brief。
|
||||
- Users:list/create/edit/delete、role、Gatekeeper groups、权限显示。
|
||||
- Auth:登录、注册、验证邮箱、重发、找回、重置、logout。
|
||||
5. 每完成一个模块,对照旧页面跑 checklist;只有全部模块通过后,才提出旧路由切换和旧版删除。
|
||||
|
||||
## Test Plan
|
||||
|
||||
- 恢复阶段:`cd frontend && /home/ray/.bun/bin/bun install && /home/ray/.bun/bin/bun run build`。
|
||||
- 旧路由检查:`/admin`、`/datasources`、`/settings`、`/ai`、`/logs` 必须继续打开旧版。
|
||||
- 新路由检查:`/admin-next/*` 必须继续打开新版。
|
||||
- 每模块 parity:读写删、测试连接、触发、批量、轮询、WebSocket、权限、错误提示、确认弹窗、移动端和滚动。
|
||||
- 视觉检查:light/dark/system、低高度窗口、125%/150% 缩放、按钮/状态/badge 等高、无双滚动条。
|
||||
|
||||
## Parity Checklist
|
||||
|
||||
### Dashboard
|
||||
- [x] `/dashboard/stats` 数据展示与旧版一致。
|
||||
- [x] WebSocket 连接状态、健康状态、自动刷新语义一致。
|
||||
- [x] 重启任务创建、轮询、日志查看、失败提示和恢复探测一致。
|
||||
|
||||
### DataSources
|
||||
- [x] 内置源、自定义源、实时源分区清晰,不混表。
|
||||
- [x] 详情、stats、task-status、enable/disable、trigger、trigger-batch、trigger-all、delete-data 与旧版一致。
|
||||
- [x] 实时源 start/stop/restart、连接状态、任务状态与旧版一致。
|
||||
|
||||
### DataList
|
||||
- [x] 列表、summary、sources/types/countries、搜索、筛选、分页参数与旧版一致。
|
||||
- [x] 详情 metadata、分布摘要、JSON/CSV 导出与旧版一致。
|
||||
- [x] 表格固定列、横向滚动、移动端卡片模式可用。
|
||||
|
||||
### Collection Management
|
||||
- [x] configs/all、configs CRUD、builtin connect/test、custom sample 与旧版一致。
|
||||
- [x] target schemas、mappings preview/list/update/activate 与旧版一致;propose/create 已提供表单化入口。
|
||||
- [x] run-mapped、stop-mapped、stream-status 与旧版一致。
|
||||
|
||||
### Settings
|
||||
- [x] system、notifications、security、SMTP、TV、integrations、collectors 表单字段完整。
|
||||
- [x] SMTP test、AI/Web/OCR secrets、connect tests、preset refresh 与旧版一致。
|
||||
- [x] credential guides、cache 清理、危险操作确认一致。
|
||||
|
||||
### AI
|
||||
- [x] Provider、Web Search、OCR、Prompt Registry 配置保存/重置完整。
|
||||
- [x] Playground thread/session/messages/status/stop/resend/edit 完整。
|
||||
- [x] provider refresh、connect test、secrets 读取、错误提示和 loading 状态完整。
|
||||
- [x] 页内 tabs 按旧版心智恢复为模型供应商、工具调用、提示词、Playground。
|
||||
|
||||
### Earth Content
|
||||
- [x] brand get/save/delete/reset/upload 完整。
|
||||
- [x] boundary status/config/build/build-status 完整。
|
||||
- [x] TV/content 配置与旧版 Settings 入口一致。
|
||||
|
||||
### Logs
|
||||
- [x] sources、level/date/search、limit、刷新参数与旧版一致。
|
||||
- [x] 日志正文、详情展开、复制、空态和错误态完整。
|
||||
|
||||
### BGP
|
||||
- [x] overview、collectors、incidents、anomalies、events 的 summary/list/detail 完整。
|
||||
- [x] AI brief list/detail/generate 完整。
|
||||
- [x] collector collect-location、刷新、结果详情和错误提示完整。
|
||||
|
||||
### Alerts
|
||||
- [x] 系统告警 list/detail/ack/resolve/stats/AI brief 完整。
|
||||
- [x] BGP incidents/anomalies/brief 完整。
|
||||
- [x] 态势告警 stats/list/brief 完整。
|
||||
|
||||
### Users
|
||||
- [x] list/create/edit/delete、role、Gatekeeper groups 完整。
|
||||
- [x] 权限显示、普通用户限制和 super admin 行为一致。
|
||||
|
||||
### Auth
|
||||
- [x] login/register/verify/resend cooldown/forgot/reset/logout/me 完整。
|
||||
- [x] 未验证邮箱跳转、错误提示、成功跳转和 token store 行为一致。
|
||||
|
||||
### Hierarchy Repair
|
||||
- [x] AI 不再按接口/状态拍平成一张表;模型供应商恢复为 provider -> model/API/key 层级。
|
||||
- [x] 工具调用恢复为 Web Search/OCR -> provider/API/高级参数层级。
|
||||
- [x] 提示词恢复为 group -> prompt 入口层级,保存/重置只作用于当前入口。
|
||||
- [x] Settings/Earth 配置型页面使用父级列表 + 右侧正式表单,长内容在一屏内内部滚动。
|
||||
|
||||
### Interaction Polish
|
||||
- [x] 不会产生歧义的刷新、复制、详情、批量、状态、清理等动作优先改为 icon-only + tooltip/title。
|
||||
- [x] 强意图动作保留实心文字按钮,例如保存、创建、触发全部、生成简报、确认执行。
|
||||
- [x] 按钮字号、高度、图标容器和 hover 触感统一,接近侧栏深色模式滑块的轻微立体风格。
|
||||
- [x] 表格与详情之间增加可拖动竖向 resize handle,低高度和窄屏下不制造额外滚动条。
|
||||
|
||||
### Page Information Architecture
|
||||
- [x] 页内 tab 默认使用中文;BGP、AI、OCR、Web Search、Playground、Schema 等专有名词按可识别性保留。
|
||||
- [x] 信息观测类页面使用列表 + 详情,例如 BGP、Alerts、Logs、DataSources。
|
||||
- [x] 配置管理类页面使用分层结构,例如 AI、Settings、Earth Content、Collection Management。
|
||||
- [x] 采集管理已从纯表格看板改为分层管理:采集器、映射模板、目标 Schema、运行状态先选父级,再编辑或执行动作。
|
||||
135
docs/plans/admin-next-parity-audit-closeout-plan.md
Normal file
135
docs/plans/admin-next-parity-audit-closeout-plan.md
Normal file
@@ -0,0 +1,135 @@
|
||||
# Admin Next Parity Audit Closeout
|
||||
|
||||
Last updated: 2026-05-21
|
||||
|
||||
## Status
|
||||
|
||||
Admin Next has been promoted to the official admin route family. The old AntD admin, old layout helpers, `antd`, and `@ant-design/icons` remain available under `/legacy/admin/*` as the rollback and comparison surface.
|
||||
|
||||
`/admin-next/*` is now compatibility-only: old test links redirect to the official route. Do not add new capabilities there as a separate entry point.
|
||||
|
||||
This closeout document is the final work log for the second parity audit. It records what has been fixed, what still requires manual verification, and what must not be treated as complete.
|
||||
|
||||
## Fixed In This Audit
|
||||
|
||||
- Restored the dual-track rule: old AntD pages and routes are not deleted and are not replaced before parity sign-off.
|
||||
- Reworked management pages away from flat status tables toward business hierarchy:
|
||||
- AI: `模型供应商 / 工具调用 / 提示词 / Playground`.
|
||||
- Settings: system display, notification policy, security policy, SMTP only.
|
||||
- Earth Content: brand, boundary precision, TV content.
|
||||
- Collection: collector, collection schedule, history/snapshot.
|
||||
- Repaired AI provider credential semantics:
|
||||
- Provider status now comes from that provider's stored key or a runtime key that is explicitly scoped to that provider.
|
||||
- Generic `.env` fallback no longer marks every provider as configured.
|
||||
- Runtime fallback secrets stay masked by default; authorized administrators can reveal them through the console and each reveal must be audited without storing plaintext.
|
||||
- Secret previews preserve the prefix before the first `-` when the backend preview exposes it, for example `sk-**********`.
|
||||
- Connect-test buttons are icon-only plug actions and do not save configuration.
|
||||
- Repaired key UI gaps:
|
||||
- LLM API Key, proxy token, Web Search API Key, and OCR API Key use masked input with inline eye toggle.
|
||||
- Save buttons use the disk icon.
|
||||
- Default action is disabled when the item is already default.
|
||||
- Repaired SMTP test flow:
|
||||
- SMTP test has a recipient input.
|
||||
- Request payload is `{ to, settings }`.
|
||||
- Repaired Earth Content actions:
|
||||
- Brand save, upload, delete/reset actions are available in the brand section.
|
||||
- Brand preview renders with the same Earth left-top brand structure and starfield-style background.
|
||||
- Boundary build is scoped to the boundary precision section, not the global page toolbar.
|
||||
- TV default configuration is folded into item-level configuration; TV items support add/default/reset/delete semantics.
|
||||
- TV default source now follows the same default-state semantics as AI Provider: if the response does not expose an explicit default, `cgtn-en` is treated as the runtime default; setting a default promotes that item visually and disables the redundant default action.
|
||||
- Repaired Collection actions:
|
||||
- Credential guide read/generate/reset entry points are available from collector-related panels.
|
||||
- Create collector config uses field-first form controls with advanced JSON as a secondary path.
|
||||
- Mapping propose/preview/create/activate flow is represented in the mapping workflow.
|
||||
- Repaired shared UI rules:
|
||||
- Management list names can wrap and remain readable instead of being squeezed by status tags.
|
||||
- Status tags are fixed width and color-coded: default blue, configured green, unconfigured gray, error red.
|
||||
- Configuration lists with a default item sort the default item to the top after save/default changes instead of leaving it in the previous alphabetical or API order.
|
||||
- List footer actions are part of the scroll content; users see them after scrolling to the bottom.
|
||||
- Footer icon buttons use the current tactile surface direction: external shadow first, without a separate inset/base-plate treatment.
|
||||
- Detail panels use a dark neutral resize handle.
|
||||
- Tooltip, dialog, modal backdrop, textarea scrollbar, custom scrollbar, mobile detail, and one-screen layout issues from the audit have been addressed in the shared admin-next layer.
|
||||
- Multi-tab Admin Next pages now use lazy active-tab loading with local cache. Initial page load no longer requests every section endpoint; manual refresh and mutating actions refresh only the current section.
|
||||
- Markdown-producing details, AI brief content, credential tutorials, and Playground assistant output use the shared Markdown renderer where the content is meant to be read as a document. Raw metadata remains available only where the page is explicitly showing original payloads.
|
||||
- Added developer workflow scripts:
|
||||
- `build:watch`
|
||||
- `preview:auto`
|
||||
- `bun run build` remains a production artifact build and does not reload an already-open dev page by itself.
|
||||
|
||||
## Manual Verification Status
|
||||
|
||||
The code has been updated. The list below records which areas are already manually checked and which still need route-level regression before replacement:
|
||||
|
||||
- AI:
|
||||
- Latest manual pass marked provider status, secret reveal, and Playground behavior as checked.
|
||||
- DataSources:
|
||||
- Single trigger performs old task-status precheck, handles running-conflict confirmation, supports force recollect, and refreshes task status afterward.
|
||||
- Batch trigger uses old `/datasources/trigger-batch` semantics for selected IDs, filters, and force.
|
||||
- Builtin, custom, and realtime source actions match old enable/disable/start/stop/restart/clear-data behavior.
|
||||
- 2026-05-20 implementation pass: Admin Next builtin datasource rows now expose task/collection status and metrics, and single trigger now performs task-status precheck with running-conflict force confirmation plus post-trigger task refresh.
|
||||
- 2026-05-20 manual pass: builtin trigger, task-status precheck, force recollect, batch trigger, enable/disable, and clear-data were verified in Admin Next.
|
||||
- Collection:
|
||||
- Collector config create/update/delete, builtin connect, custom test/sample/run/stop/status, schedule save, history/snapshot, mapping propose/preview/create/activate, and credential guide read/generate/reset all work with real backend payloads.
|
||||
- 2026-05-20 implementation pass: `采集历史 / 快照` now reads real datasource snapshots from `/datasources/snapshots` instead of config rows.
|
||||
- 2026-05-20 implementation pass: default credential guide actions no longer call unsupported providers blindly; unsupported collectors show a clear empty state and supported providers use `barentswatch` / `aisstream`.
|
||||
- Collection action buttons must follow the AI page's single-responsibility rule: navigation/link actions do not save, save actions do not connect/test, and connect/test actions do not persist drafts.
|
||||
- 2026-05-20 manual pass: collector create/edit `auth_config`, mapping propose/preview/create/activate, custom test/sample/run/stop/status, credential guide read/generate/reset, and history/snapshot behavior were verified in Admin Next.
|
||||
- 2026-05-20 follow-up: dead JSON modal create/mapping code was removed so Collection creation now goes through the list-bottom draft workflow only.
|
||||
- Earth Content:
|
||||
- Brand preview visually matches the Earth runtime left-top brand at the same structural level.
|
||||
- Brand upload writes back `logo_src` or `title_src` correctly.
|
||||
- 2026-05-20 implementation pass: Brand upload validates and displays accepted suffixes (`png`, `jpg`, `jpeg`, `webp`, `svg`) and supports drag-and-drop.
|
||||
- Boundary status/config/build and TV add/save/default/delete/reset match old behavior.
|
||||
- Logs:
|
||||
- Sources, snapshot/tail, level/date/search, copy, empty/error states, and internal scrolling all work.
|
||||
- 2026-05-20 implementation pass: Logs now include database-backed `system-db` and `audit-db` sources as a fallback when file/docker/buffer logs are empty or missing, and the UI auto-selects an available source. Manual regression must still verify snapshot/tail-like reading, filters, copy, and scrolling.
|
||||
- BGP / Alerts:
|
||||
- BGP overview, collectors, collect-location detail, incidents, anomalies, events, latest/detail/generate AI brief.
|
||||
- System/BGP/situational alert detail, ack, resolve with note, refresh after mutation, and AI brief details.
|
||||
- Users/Auth:
|
||||
- Users CRUD, role, Gatekeeper groups.
|
||||
- 2026-05-20 implementation pass: User edit select dropdowns now use an opaque high-z-index menu surface, and Gatekeeper checkbox groups use an explicit surface background. Manual visual regression is still required.
|
||||
- Login, register, verify email, resend cooldown, forgot/reset password.
|
||||
- Mobile:
|
||||
- 375px, 430px, and 768px widths default to list/group first.
|
||||
- Detail opens only after tap/click and has a visible back action.
|
||||
- Long forms, JSON, logs, and Playground messages scroll internally without body-level double scrollbars.
|
||||
|
||||
## Explicit Non-Production Items
|
||||
|
||||
The old AntD page itself treated these Earth content tabs as placeholder-level capabilities. Admin Next must show a visible “backend capability pending” empty state and must not mix in unrelated data:
|
||||
|
||||
- `basemap`
|
||||
- `layer_resources`
|
||||
- `models_3d`
|
||||
- `news_anchor_strategy`
|
||||
|
||||
If backend endpoints are later added, these items must be promoted into `docs/plans/admin-next-parity-checklist.md` with concrete API and UI acceptance criteria.
|
||||
|
||||
## Final Gate
|
||||
|
||||
After the route promotion, the final gate is no longer “switch old routes.” The remaining gate is “keep legacy available until the promoted routes are stable enough to remove AntD.”
|
||||
|
||||
1. Run `cd frontend && /home/ray/.bun/bin/bun run build`.
|
||||
2. Run the static checks:
|
||||
- `rg "map: \\(\\) => \\[\\]|暂不支持保存|placeholder" frontend/src/admin-next`
|
||||
- `rg "ShadowPage|FeatureConsole|GlassPanel|InspectorDrawer" frontend/src/admin-next`
|
||||
3. Manually verify every official route listed in `docs/plans/admin-next-parity-checklist.md`.
|
||||
4. Confirm `/legacy/admin/*` still opens old AntD pages during the validation window.
|
||||
5. Delete old AntD pages and remove AntD dependencies only as a separate final cleanup task after explicit confirmation.
|
||||
|
||||
## Current Build Verification
|
||||
|
||||
Last successful local build during this closeout, after the 2026-05-20 follow-up fixes:
|
||||
|
||||
```bash
|
||||
cd frontend && /home/ray/.bun/bin/bun run build
|
||||
```
|
||||
|
||||
Result: passed.
|
||||
|
||||
Backend syntax check also passed:
|
||||
|
||||
```bash
|
||||
/home/ray/.local/bin/uv run python -m py_compile backend/app/api/v1/datasources.py backend/app/api/v1/datasource_config.py backend/app/api/v1/system_control.py
|
||||
```
|
||||
43
docs/plans/admin-next-parity-checklist.md
Normal file
43
docs/plans/admin-next-parity-checklist.md
Normal file
@@ -0,0 +1,43 @@
|
||||
# Admin Next Parity Checklist
|
||||
|
||||
Last updated: 2026-05-21
|
||||
|
||||
This checklist is the hard gate for deleting the old AntD admin. Admin Next now owns the official admin routes; old AntD pages stay available under `/legacy/admin/*` until every production capability below is verified.
|
||||
|
||||
## Route Gate
|
||||
|
||||
- `/admin` Dashboard: stats, health probe, WebSocket status, restart task, restart logs, recovery probe.
|
||||
- `/datasources`: builtin sources, custom sources, realtime sources, detail/stat/task-status, trigger, batch trigger, enable/disable, clear data, realtime start/stop/restart, custom source test/sample/run/stop/status link.
|
||||
- `/data`: list, summary, source/type/country filters, search, pagination, detail metadata, export, distribution chart.
|
||||
- `/collection-management`: collector config, mapping templates, target schemas, collection schedule, history/snapshot, create config, propose/preview/create/activate mapping, builtin connect, run/stop/stream-status, credential guide read/generate/reset.
|
||||
- `/settings`: system display, notifications, security, SMTP get/save/test with recipient.
|
||||
- `/ai`: model providers, tool calling, prompts, key reveal, provider refresh, connect tests, OCR/WebSearch secrets, Playground thread/chat/edit/resend/stop.
|
||||
- `/earth-content`: brand preview as Earth renders it, upload/save/delete/reset, boundary status/build/config, TV source add/save/default/delete/reset.
|
||||
- `/logs`: sources, filters, snapshot, empty/error states, copy.
|
||||
- `/bgp`: overview, collectors, collect-location, incidents, anomalies, events, AI brief list/latest/detail/generate.
|
||||
- `/alerts/system`, `/alerts/bgp`, `/alerts/situational`: list/stat/detail, ack, resolve with note, AI brief generation.
|
||||
- `/users`: list/create/edit/delete, role, Gatekeeper permission groups.
|
||||
- Auth pages: login, register, verify email, resend cooldown, forgot/reset password.
|
||||
|
||||
Compatibility routes under `/admin-next/*` should redirect to these official paths and are not a separate validation target.
|
||||
|
||||
## Verified Manual Passes
|
||||
|
||||
- 2026-05-20 `/admin-next/collection-management`: user verified collector create/edit payloads, mapping propose/preview/create/activate, custom collector test/sample/run/stop/status, credential guide read/generate/reset, and collection history/snapshot behavior after the draft-form and Time Capsule updates.
|
||||
- 2026-05-20 `/admin-next/datasources`: user verified builtin source trigger, task-status precheck, force recollect, batch trigger, enable/disable, and clear-data behavior.
|
||||
|
||||
## Known Non-Production Tabs
|
||||
|
||||
The old AntD page also marked these Earth content tabs as placeholder-level capabilities. Admin Next must show them as “backend capability pending” and must not mix in unrelated data:
|
||||
|
||||
- `basemap`
|
||||
- `layer_resources`
|
||||
- `models_3d`
|
||||
- `news_anchor_strategy`
|
||||
|
||||
## Replacement Rules
|
||||
|
||||
- Keep `/legacy/admin/*` available until the matching route gate above is manually verified after promotion.
|
||||
- Do not delete old AntD pages, old layout helpers, `antd`, or `@ant-design/icons` until final explicit confirmation.
|
||||
- Do not use fake rows, unrelated endpoint data, or empty adapters for a real old capability.
|
||||
- Any backend-missing capability must be recorded here with the missing endpoint and visible UI empty state.
|
||||
217
docs/plans/admin-next-soft-glass-goal-driven-plan.md
Normal file
217
docs/plans/admin-next-soft-glass-goal-driven-plan.md
Normal file
@@ -0,0 +1,217 @@
|
||||
# Admin Next Soft Glass Goal-Driven Plan
|
||||
|
||||
## Goal
|
||||
|
||||
Rebuild `/admin-next/*` into a modern soft-glass / light-neumorphic console while keeping the legacy Ant Design admin routes available for comparison. The new console must use visual references only as design anchors, not as imported templates, and must be implemented as Planet-owned reusable components and page patterns.
|
||||
|
||||
The redesign must cover desktop and mobile. Data display, icon semantics, table readability, and scroll behavior are first-class acceptance criteria.
|
||||
|
||||
## Criteria For Success
|
||||
|
||||
- This plan exists at `docs/plans/admin-next-soft-glass-goal-driven-plan.md`.
|
||||
- `/admin-next/*` has real pages for every route; route usage of `ShadowPage` is removed.
|
||||
- Admin Next supports `system`, `light`, and `dark` theme modes using the same persistence and system-theme idea as Docs.
|
||||
- The visual language reads as soft-glass / light-neumorphic instead of an AntD reskin: translucent panels, fine borders, subtle glow, cool backgrounds, restrained accent colors, crisp icons, and tactile controls.
|
||||
- Mobile is explicitly designed: mobile navigation, filter sheets/chips, card-list data views, full-screen detail sheets, and bottom action bars instead of squeezed desktop tables.
|
||||
- Tables do not show double scrollbars. Long fields are inspectable, copyable, or expandable; important columns stay readable.
|
||||
- Existing custom scrollbar components remain the default scroll experience.
|
||||
- `cd frontend && bun run build` passes.
|
||||
- Static checks can prove no route-level placeholder remains.
|
||||
|
||||
## Reference Strategy
|
||||
|
||||
Use references to calibrate feel, not to take over code:
|
||||
|
||||
- User-provided soft-glass / light-neumorphic dashboard images define the desired mood.
|
||||
- Shadcn / Tailwind / glassmorphism admin templates may be inspected for shadow, spacing, panel, dark-mode, and app-shell ideas.
|
||||
- Do not import a complete template, router, auth layer, state layer, or business page.
|
||||
- Planet-specific interactions such as datasource tasks, BGP briefs, AI settings, log streams, Earth resources, and custom scrollbars are implemented in this repo.
|
||||
|
||||
## Source-Of-Truth Mining
|
||||
|
||||
Design content and page boundaries from existing repo history and docs:
|
||||
|
||||
- `docs/CHANGELOG.md`
|
||||
- `docs/plans/*`
|
||||
- `docs/technical/*frontend*`
|
||||
- `TODO.md`
|
||||
|
||||
Important current boundaries:
|
||||
|
||||
- `DataSources` is a datasource directory and runtime-control page, not a configuration editor.
|
||||
- `Collection Management` owns collector endpoint, credential, custom source, mapping, and runtime configuration.
|
||||
- `Earth Content` owns TV livestreams, brand assets, boundary precision, basemap, and Earth display resources.
|
||||
- `AI` owns provider settings, tools, prompts, and Playground.
|
||||
- `BGP` is an observability and evidence workspace, not just tables.
|
||||
- `Alerts` is a duty/analysis workspace for system, BGP, and situational risk.
|
||||
- `Logs` should become a usable log workbench with filtering and structured detail.
|
||||
- `Settings` should keep platform settings only.
|
||||
|
||||
## Reusable Architecture
|
||||
|
||||
Recommended structure:
|
||||
|
||||
- `admin-next/design`: theme tokens, theme mode, status colors, icon semantics, chart themes, scrollbar themes.
|
||||
- `admin-next/components`: primitive UI such as button, input, select, tabs, dialog, drawer, badge, status pill, icon badge.
|
||||
- `admin-next/patterns`: page patterns such as `PageFrame`, `GlassPanel`, `MetricCard`, `CommandBar`, `FilterRail`, `InspectorDrawer`, `DataWorkspace`, `SettingsWorkspace`, `ScrollableRegion`, `TableRegion`, and `MobileSheet`.
|
||||
- `admin-next/features`: business modules such as datasources, data-list, bgp, alerts, ai, logs, settings, earth-content, collection-management, users.
|
||||
- `admin-next/services`: feature API gateways and hooks. Pages should not scatter URL construction.
|
||||
- `admin-next/routes`: route manifest used by navigation, command search, breadcrumbs, and legacy links.
|
||||
|
||||
## Interaction Rules
|
||||
|
||||
- Row click opens an inspector; frequent row actions live in the inspector header.
|
||||
- Bulk selection opens a floating command bar.
|
||||
- Common filters stay visible; advanced filters open a sheet/drawer.
|
||||
- Detail, JSON, log context, and AI brief use inspector or reading panels instead of stacked modal dialogs.
|
||||
- Settings are split into `FormSection` blocks with local save, dirty state, connection test, and reset.
|
||||
- Dangerous actions use a confirm sheet.
|
||||
- Loading, empty, error, and retry states are shared through reusable patterns.
|
||||
|
||||
## Mobile Rules
|
||||
|
||||
- Do not squeeze desktop layout into mobile.
|
||||
- Use mobile navigation and sheets instead of a permanent desktop sidebar.
|
||||
- Filters collapse into a filter sheet; active filters remain visible as chips.
|
||||
- Data workspaces default to card-list mode on mobile and allow explicit table mode for dense comparison.
|
||||
- Details open in full-screen mobile sheets.
|
||||
- Batch actions use a sticky bottom command bar.
|
||||
- Long logs and JSON use full-screen reading/editing surfaces with custom scrollbars.
|
||||
|
||||
## Table And Data Display Rules
|
||||
|
||||
- Tables must prioritize complete data display:
|
||||
- key columns get stable widths and priority;
|
||||
- long fields have peek/expand or inspector detail;
|
||||
- IDs, URLs, metadata, and errors are copyable;
|
||||
- complex fields are rendered fully in inspector.
|
||||
- Desktop tables use one `TableRegion` scroll container.
|
||||
- Mobile defaults to card-list display and supports dense table mode.
|
||||
- Native scrollbar visuals are hidden where custom scrollbars apply.
|
||||
|
||||
## Icon Rules
|
||||
|
||||
- Use `lucide-react`.
|
||||
- Define icon semantics by category:
|
||||
- navigation icon;
|
||||
- entity icon;
|
||||
- status icon;
|
||||
- action icon;
|
||||
- severity icon.
|
||||
- Icons must not be the only semantic carrier unless the icon is universally obvious; use labels or tooltips.
|
||||
- Status icons pair with `StatusPill`.
|
||||
- Colors come from tokens, not ad hoc hard-coded values.
|
||||
|
||||
## Page Design
|
||||
|
||||
### Dashboard
|
||||
|
||||
- System health, realtime connection, datasource activity, task trend, alert summary, recent events, and quick commands.
|
||||
- Restart flow uses a confirm sheet and log console.
|
||||
- Mobile uses horizontal metric cards and collapsible event/command panels.
|
||||
|
||||
### DataSources
|
||||
|
||||
- Datasource directory and runtime control.
|
||||
- Desktop: overview rail, datasource table/list, inspector.
|
||||
- Mobile: datasource cards with status, latest task, trigger/toggle actions.
|
||||
- Batch trigger uses command bar.
|
||||
- Endpoint/config/task details are shown fully in inspector.
|
||||
|
||||
### DataList
|
||||
|
||||
- Collection result browsing, search, source/type filters, paging, distribution chart, and detail inspector.
|
||||
- Fix double scrollbars.
|
||||
- Mobile defaults to cards and allows table mode.
|
||||
|
||||
### BGP
|
||||
|
||||
- Views: collectors, incidents, anomalies, events, AI brief.
|
||||
- Emphasize severity, region, ASN/prefix, collector coverage, and evidence.
|
||||
- AI brief shows facts, judgment, and evidence gaps.
|
||||
|
||||
### Alerts
|
||||
|
||||
- Real pages for system, BGP, and situational alerts.
|
||||
- Shared alert workspace with stats, severity/status filters, list, and inspector.
|
||||
- AI brief appears in inspector/mobile sheet.
|
||||
|
||||
### AI
|
||||
|
||||
- Provider, tools, prompts, and Playground.
|
||||
- Provider and tool settings use local save and connection tests.
|
||||
- Prompt registry is grouped by task.
|
||||
- Mobile Playground uses step tabs rather than cramped columns.
|
||||
|
||||
### Logs
|
||||
|
||||
- Log workbench with source, level, date, search, refresh, copy, and structured detail.
|
||||
- Desktop uses a terminal-like log stream.
|
||||
- Mobile rows expand or open a full-screen reading sheet.
|
||||
|
||||
### Users
|
||||
|
||||
- Search, role filter, create/edit/delete, Gatekeeper groups.
|
||||
- Edit/create in drawer/sheet.
|
||||
- Gatekeeper groups use checkbox chips.
|
||||
|
||||
### Settings
|
||||
|
||||
- Platform settings only: system, notification, security, SMTP.
|
||||
- Each section saves independently.
|
||||
|
||||
### Earth Content
|
||||
|
||||
- TV livestreams, brand assets, boundary precision, basemap, and layer resources.
|
||||
- Resource previews, upload state, and boundary build status are emphasized.
|
||||
|
||||
### Collection Management
|
||||
|
||||
- Collector config, custom sources, mapping, runtime control.
|
||||
- Custom source creation uses a step sheet.
|
||||
- Advanced JSON is isolated in a collapsible editor.
|
||||
|
||||
## Milestones
|
||||
|
||||
1. Save this plan and establish goal-driven criteria.
|
||||
2. Build design/theme/pattern foundations.
|
||||
3. Remove route-level placeholders and create real pages for all `/admin-next/*` routes.
|
||||
4. Redesign layout, theme, mobile shell, scroll behavior, and table behavior.
|
||||
5. Fill core workspaces with real API calls and reusable patterns.
|
||||
6. Verify build, no `ShadowPage` route usage, and core route availability.
|
||||
|
||||
## Verification
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
cd frontend && bun run build
|
||||
rg "ShadowPage" frontend/src/admin-next
|
||||
rg "axios\\." frontend/src/admin-next/pages
|
||||
rg "overflow: auto|overflow-y: auto" frontend/src/admin-next
|
||||
```
|
||||
|
||||
Manual route checks:
|
||||
|
||||
- `/admin-next`
|
||||
- `/admin-next/datasources`
|
||||
- `/admin-next/data`
|
||||
- `/admin-next/bgp`
|
||||
- `/admin-next/alerts/system`
|
||||
- `/admin-next/alerts/bgp`
|
||||
- `/admin-next/alerts/situational`
|
||||
- `/admin-next/ai`
|
||||
- `/admin-next/logs`
|
||||
- `/admin-next/users`
|
||||
- `/admin-next/settings`
|
||||
- `/admin-next/earth-content`
|
||||
- `/admin-next/collection-management`
|
||||
|
||||
Manual viewport checks:
|
||||
|
||||
- desktop;
|
||||
- mobile width;
|
||||
- low height;
|
||||
- 125% / 150% browser zoom;
|
||||
- light / dark / system theme modes.
|
||||
|
||||
810
docs/plans/agents-earth-command-runtime-plan.md
Normal file
810
docs/plans/agents-earth-command-runtime-plan.md
Normal file
@@ -0,0 +1,810 @@
|
||||
# Agent Runtime, Earth LLM Command, And Speech Entry Plan
|
||||
|
||||
## Summary
|
||||
|
||||
Build an auditable backend Agent Runtime and upgrade the existing Earth search panel into a combined search, AI command, and voice wake entry. The first version is a runtime foundation, not the full multi-role simulation product yet.
|
||||
|
||||
Typical user goals:
|
||||
|
||||
- In Earth, type "高亮所有北斗卫星" and have the system open the satellite layer and highlight matching Beidou satellites.
|
||||
- Type "中国大陆的算力中心" and have the system open the compute-center layer, match mainland China compute centers, and highlight them.
|
||||
- When microphone permission is granted, use a configurable wake word, then speak an Earth command.
|
||||
- Save every Earth AI command as an agent run so operators can review the original input, speech transcription, tool steps, entity matches, final action plan, and frontend execution result.
|
||||
|
||||
Core boundaries:
|
||||
|
||||
- `aiprovider` remains the model gateway. It must not own business tools, database access, Earth actions, or agent policy.
|
||||
- The backend owns agent orchestration, tools, evidence storage, permission policy, and proposal application.
|
||||
- Earth v1 executes visualization actions only. It does not mutate business data.
|
||||
- Speech recognition uses a provider-neutral ASR API first, defaulting to OpenAI/Whisper-compatible transcription APIs, with local `whisper.cpp` style providers as later adapters.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Backend Agent Runtime
|
||||
|
||||
Recommended module shape:
|
||||
|
||||
```text
|
||||
backend/app/models/
|
||||
agent_run.py
|
||||
agent_step.py
|
||||
agent_evidence.py
|
||||
agent_proposal.py
|
||||
|
||||
backend/app/schemas/
|
||||
agents.py
|
||||
speech.py
|
||||
|
||||
backend/app/services/agents/
|
||||
runtime.py
|
||||
orchestrator.py
|
||||
tool_protocol.py
|
||||
tool_registry.py
|
||||
policy.py
|
||||
proposals.py
|
||||
earth_command.py
|
||||
entity_query.py
|
||||
speech.py
|
||||
|
||||
backend/app/api/v1/
|
||||
agents.py
|
||||
```
|
||||
|
||||
Database conventions should follow the current project style:
|
||||
|
||||
- Use SQLAlchemy models.
|
||||
- Import new models from `init_db()`.
|
||||
- Let `Base.metadata.create_all` create tables.
|
||||
- Add required indexes with `CREATE INDEX IF NOT EXISTS`.
|
||||
- Do not introduce Alembic for this feature.
|
||||
|
||||
### aiprovider Boundary
|
||||
|
||||
`aiprovider` should continue to provide model transport only:
|
||||
|
||||
- Keep the existing `/v1/analyze` endpoint.
|
||||
- If schemas are extended, only pass through model/provider request fields and normalize responses.
|
||||
- Do not add WebSearch, database queries, Earth entity lookup, or business action execution inside `aiprovider`.
|
||||
- If a provider does not support native tool calling, the backend must use JSON tool-call fallback.
|
||||
|
||||
### Agent Tool Protocol
|
||||
|
||||
The first version should support two protocols:
|
||||
|
||||
- Default path: backend JSON tool-call loop.
|
||||
- Optional path: provider-native tools when the configured provider supports them.
|
||||
- Fallback path: if provider-native tools are unavailable or unstable, automatically use JSON tool-call.
|
||||
|
||||
Example JSON tool call:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "tool_call",
|
||||
"tool": "earth.find_entities",
|
||||
"arguments": {
|
||||
"domain": "satellites",
|
||||
"filters": {
|
||||
"constellation": "beidou"
|
||||
},
|
||||
"limit": 500
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Example final response:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "final",
|
||||
"summary": "已找到并高亮北斗卫星。",
|
||||
"result": {
|
||||
"actions": []
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Hard limits:
|
||||
|
||||
- One run may execute at most 8 tool steps by default.
|
||||
- One tool call times out after 30 seconds by default.
|
||||
- Only registered whitelist tools may run.
|
||||
- Tool arguments must pass Pydantic validation.
|
||||
- Illegal tools, invalid arguments, and denied actions must be recorded as step errors.
|
||||
- LLM output may not directly write business state. Writes are either proposals or backend-executed policy-approved actions.
|
||||
|
||||
## Data Model
|
||||
|
||||
### AgentRun
|
||||
|
||||
Suggested fields:
|
||||
|
||||
- `id`: integer primary key.
|
||||
- `public_id`: public string id.
|
||||
- `run_type`: `earth_command`, `situational_awareness`, `config_proposal`, or `diagnostic`.
|
||||
- `status`: `queued`, `running`, `waiting_approval`, `completed`, `failed`, or `stopped`.
|
||||
- `title`: short display title.
|
||||
- `objective`: text objective.
|
||||
- `input`: JSONB original input, including text and audio metadata.
|
||||
- `context`: JSONB run context.
|
||||
- `result_markdown`: final human-readable output.
|
||||
- `result_json`: structured output, including Earth action plans.
|
||||
- `provider`: nullable provider id.
|
||||
- `model`: nullable model id.
|
||||
- `request_id`: nullable propagated request id.
|
||||
- `created_by`: user id.
|
||||
- `created_at`, `updated_at`, `completed_at`.
|
||||
- `error`: nullable text error.
|
||||
|
||||
### AgentStep
|
||||
|
||||
Suggested fields:
|
||||
|
||||
- `id`.
|
||||
- `run_id`.
|
||||
- `step_index`.
|
||||
- `step_type`: `llm`, `tool`, `policy`, `action`, or `transcription`.
|
||||
- `status`: `pending`, `running`, `completed`, `failed`, or `skipped`.
|
||||
- `name`: step name, for example `earth.find_entities`.
|
||||
- `input`: JSONB.
|
||||
- `output`: JSONB.
|
||||
- `error`: nullable text.
|
||||
- `started_at`, `completed_at`.
|
||||
- `duration_ms`.
|
||||
|
||||
### AgentEvidence
|
||||
|
||||
Suggested fields:
|
||||
|
||||
- `id`.
|
||||
- `run_id`.
|
||||
- `step_id`: nullable.
|
||||
- `evidence_type`: `internal_record`, `web_search`, `web_fetch`, `entity_match`, or `transcription`.
|
||||
- `source`: source id.
|
||||
- `title`: display title.
|
||||
- `url`: nullable source URL.
|
||||
- `content`: text evidence content.
|
||||
- `content_hash`: nullable hash.
|
||||
- `metadata`: JSONB.
|
||||
- `retrieved_at`.
|
||||
|
||||
### AgentProposal
|
||||
|
||||
Suggested fields:
|
||||
|
||||
- `id`.
|
||||
- `run_id`.
|
||||
- `proposal_type`: `datasource_config`, `ai_prompt`, or `external_integration`.
|
||||
- `status`: `pending`, `approved`, `applied`, `rejected`, or `failed`.
|
||||
- `risk_level`: `low`, `medium`, or `high`.
|
||||
- `target`: JSONB target descriptor.
|
||||
- `before_payload`: JSONB.
|
||||
- `after_payload`: JSONB.
|
||||
- `rationale`: text.
|
||||
- `policy_result`: JSONB.
|
||||
- `applied_by`: nullable user id.
|
||||
- `applied_at`: nullable timestamp.
|
||||
- `error`: nullable text.
|
||||
|
||||
## Public APIs
|
||||
|
||||
### Agent Runs
|
||||
|
||||
Add:
|
||||
|
||||
```text
|
||||
POST /api/v1/agents/runs
|
||||
GET /api/v1/agents/runs
|
||||
GET /api/v1/agents/runs/{run_id}
|
||||
POST /api/v1/agents/runs/{run_id}/stop
|
||||
POST /api/v1/agents/proposals/{proposal_id}/apply
|
||||
```
|
||||
|
||||
### Earth Command
|
||||
|
||||
Add:
|
||||
|
||||
```text
|
||||
POST /api/v1/agents/earth/command
|
||||
```
|
||||
|
||||
Request shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"input_text": "高亮所有北斗卫星",
|
||||
"source": "text",
|
||||
"transcription_id": null,
|
||||
"client_context": {
|
||||
"visible_layers": ["satellites"],
|
||||
"locale": "zh-CN",
|
||||
"viewport": {
|
||||
"is_mobile": false
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Response shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"run_id": "agent_xxx",
|
||||
"summary": "已找到并高亮北斗卫星。",
|
||||
"actions": [
|
||||
{
|
||||
"type": "show_layer",
|
||||
"layer": "satellites",
|
||||
"enabled": true
|
||||
},
|
||||
{
|
||||
"type": "highlight_entities",
|
||||
"domain": "satellites",
|
||||
"entity_ids": ["satellite:norad:12345"],
|
||||
"style": {
|
||||
"color": "#7dd3fc"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "open_result_panel",
|
||||
"title": "北斗卫星",
|
||||
"items": []
|
||||
}
|
||||
],
|
||||
"matched_entities": [],
|
||||
"confidence": 0.86,
|
||||
"missing_data": []
|
||||
}
|
||||
```
|
||||
|
||||
### Speech / ASR
|
||||
|
||||
Add:
|
||||
|
||||
```text
|
||||
POST /api/v1/agents/speech/transcriptions
|
||||
```
|
||||
|
||||
Request should use multipart form data:
|
||||
|
||||
- `file`: audio blob.
|
||||
- `language`: default `zh`.
|
||||
- `provider`: optional provider override.
|
||||
- `source`: default `earth_command`.
|
||||
|
||||
Response shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"text": "高亮所有北斗卫星",
|
||||
"provider": "openai_compatible",
|
||||
"model": "whisper-1",
|
||||
"duration_ms": 1234,
|
||||
"confidence": null,
|
||||
"metadata": {}
|
||||
}
|
||||
```
|
||||
|
||||
## Earth Action Plan
|
||||
|
||||
### Allowed Action Types
|
||||
|
||||
The first version may only return:
|
||||
|
||||
```text
|
||||
show_layer
|
||||
highlight_entities
|
||||
filter_entities
|
||||
focus_view
|
||||
open_result_panel
|
||||
clear_highlight
|
||||
```
|
||||
|
||||
### Allowed Domains
|
||||
|
||||
The first version supports:
|
||||
|
||||
```text
|
||||
satellites
|
||||
compute_centers
|
||||
bgp
|
||||
news
|
||||
vessels
|
||||
cables
|
||||
```
|
||||
|
||||
### EarthAction Shape
|
||||
|
||||
Example:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "highlight_entities",
|
||||
"domain": "satellites",
|
||||
"entity_ids": ["satellite:norad:12345"],
|
||||
"style": {
|
||||
"color": "#7dd3fc",
|
||||
"mode": "glow"
|
||||
},
|
||||
"reason": "用户要求高亮北斗卫星"
|
||||
}
|
||||
```
|
||||
|
||||
### Safety Rules
|
||||
|
||||
The backend must validate action plans before returning them:
|
||||
|
||||
- `type` must be in the allowed action list.
|
||||
- `domain` must be in the allowed domain list.
|
||||
- `entity_ids` must come from backend or current Earth candidate data. The model may not invent ids.
|
||||
- One highlight action should include at most 500 entities by default. If more match, return a truncation note in `missing_data` or `summary`.
|
||||
- `focus_view` must include valid coordinates, a valid region, or a matched entity.
|
||||
- No action may return executable JavaScript, arbitrary CSS, or arbitrary URLs to fetch.
|
||||
|
||||
## Earth Entity Query
|
||||
|
||||
Add a backend entity query service used by both deterministic resolvers and LLM tools.
|
||||
|
||||
### Satellites
|
||||
|
||||
Sources:
|
||||
|
||||
- `/api/v1/visualization/geo/satellites`.
|
||||
- Current TLE collected data.
|
||||
- GeoJSON feature properties.
|
||||
|
||||
Filters:
|
||||
|
||||
- `constellation`: `beidou`, `gps`, `galileo`, `glonass`, `starlink`, `iridium`, `geo`, `leo`.
|
||||
- Name contains.
|
||||
- NORAD id.
|
||||
- Country/operator when present in data.
|
||||
- Orbital class when inferable from existing fields.
|
||||
|
||||
Beidou matching:
|
||||
|
||||
- Prefer `constellation == beidou`.
|
||||
- Then match names containing `BEIDOU`, `BDS`, `BEIDOU-`, or `北斗`.
|
||||
- Stable entity id format should be `satellite:norad:{norad_id}` when possible, with fallback `satellite:index:{index}`.
|
||||
|
||||
### Compute Centers
|
||||
|
||||
Sources:
|
||||
|
||||
- `/api/v1/visualization/geo/compute-centers`.
|
||||
- Unified TOP500 and Epoch AI GPU GeoJSON properties.
|
||||
|
||||
Filters:
|
||||
|
||||
- Country/region.
|
||||
- `site_type`: `supercomputer` or `gpu_cluster`.
|
||||
- Source: `top500` or `epoch_ai_gpu`.
|
||||
- Name contains.
|
||||
- `needs_confirmation`.
|
||||
- `location_precision`.
|
||||
|
||||
Mainland China matching:
|
||||
|
||||
- Match country values such as `China`, `中国`, or `People's Republic of China`.
|
||||
- Exclude obvious non-mainland records when fields identify Hong Kong, Macau, or Taiwan.
|
||||
- If records do not expose enough region detail to separate mainland China from Hong Kong, Macau, or Taiwan, return a `missing_data` note and conservatively match `country=China`.
|
||||
|
||||
### BGP, News, Vessels, And Cables
|
||||
|
||||
First-version basic support:
|
||||
|
||||
- BGP: severity, status, region, collector, prefix, ASN.
|
||||
- News: region, source, localized title, localized summary.
|
||||
- Vessels: vessel type, country/area, name, status.
|
||||
- Cables: cable name, landing point, country/region.
|
||||
|
||||
## Earth Frontend Integration
|
||||
|
||||
### Search Panel Merge
|
||||
|
||||
Reuse the existing Earth search panel:
|
||||
|
||||
- Default behavior remains normal local search.
|
||||
- Add an AI command state for natural-language commands.
|
||||
- Add a "use AI" command button.
|
||||
- Add a microphone button.
|
||||
- Show running status, result summary, and clear-highlight action.
|
||||
- Visually distinguish ordinary search results from AI action results.
|
||||
|
||||
Natural-language routing:
|
||||
|
||||
- If the user clicks the AI command button, always call the AI command endpoint.
|
||||
- If input contains action words such as `高亮`, `显示`, `找出`, `聚焦`, `打开`, `筛选`, `隐藏`, or `清除`, suggest AI command mode.
|
||||
- Short ordinary keywords continue to use local search.
|
||||
|
||||
### Earth Action Executor
|
||||
|
||||
Add:
|
||||
|
||||
```text
|
||||
frontend/public/earth/js/earth-command-actions.js
|
||||
```
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- Receive backend `EarthActionPlan`.
|
||||
- Toggle required layers.
|
||||
- Apply highlights and filters.
|
||||
- Focus view when requested.
|
||||
- Open or update the result panel.
|
||||
- Clear the previous AI command highlight.
|
||||
|
||||
Execution order:
|
||||
|
||||
1. Clear previous AI command highlights unless the plan is explicitly additive.
|
||||
2. Show required layers.
|
||||
3. Apply filters and highlights.
|
||||
4. Focus the view if requested.
|
||||
5. Open the result panel.
|
||||
6. Record the execution summary in UI state.
|
||||
|
||||
### Satellite Highlight
|
||||
|
||||
Existing `highlightRelatedSatellites(indices, color)` can be reused, but Earth command support needs an entity-id-to-index map.
|
||||
|
||||
Add:
|
||||
|
||||
- Satellite id map built from current satellite data.
|
||||
- Highlight by `entity_ids`.
|
||||
- Clear AI command satellite highlights without clearing manual locked selection.
|
||||
- Auto-enable the satellite layer before highlighting.
|
||||
|
||||
### Compute Center Highlight
|
||||
|
||||
Existing compute center markers support marker state, but need a dedicated batch AI highlight state.
|
||||
|
||||
Add:
|
||||
|
||||
- Batch set compute center markers to AI highlighted state.
|
||||
- Keep AI highlight compatible with hover and locked states.
|
||||
- Clear AI command compute highlights without clearing manual locked selection.
|
||||
- Auto-enable the compute-center layer before highlighting.
|
||||
|
||||
### Result Panel
|
||||
|
||||
First version can show results inside the merged search panel:
|
||||
|
||||
- Title, for example `北斗卫星`.
|
||||
- Count, for example `已高亮 32 个对象`.
|
||||
- List with the first 20 entities.
|
||||
- Actions: clear highlight, rerun, view Agent Run.
|
||||
|
||||
## Voice And Wake Word
|
||||
|
||||
### ASR Configuration
|
||||
|
||||
Add Speech/ASR under the AI settings tool tab.
|
||||
|
||||
Fields:
|
||||
|
||||
- `enabled`.
|
||||
- `provider`.
|
||||
- `base_url`.
|
||||
- `api_key`.
|
||||
- `model`.
|
||||
- `language`.
|
||||
- `timeout_seconds`.
|
||||
- `max_audio_size_mb`.
|
||||
|
||||
First-version providers:
|
||||
|
||||
- `openai_whisper`.
|
||||
- `openai_compatible`.
|
||||
- `local_whisper`.
|
||||
|
||||
Recommended defaults:
|
||||
|
||||
- `provider`: `openai_compatible`.
|
||||
- `model`: `whisper-1` or user-configured equivalent.
|
||||
- `language`: `zh`.
|
||||
|
||||
Secret handling:
|
||||
|
||||
- Use the existing masked secret pattern.
|
||||
- Do not allow agent proposals to write API keys.
|
||||
- Support environment fallback such as `ASR_API_KEY` and `OPENAI_API_KEY`.
|
||||
|
||||
### Wake Word
|
||||
|
||||
The wake word is configurable.
|
||||
|
||||
Implementation rules:
|
||||
|
||||
- Treat wake word as a browser-local preference in v1.
|
||||
- Store it in `localStorage`.
|
||||
- Default suggestion: `小星球`.
|
||||
- Provide a wake-word setting inside the Earth search/command panel.
|
||||
- Do not listen until the user explicitly enables voice wake.
|
||||
- Do not upload audio before wake.
|
||||
- After wake, record one instruction audio segment and upload it to backend ASR.
|
||||
|
||||
Fallbacks:
|
||||
|
||||
- Browser does not support continuous local recognition: fall back to click-to-record.
|
||||
- Microphone permission denied: show `麦克风权限未开启,仍可输入文字指令`.
|
||||
- ASR not configured: show `语音识别未配置`; text commands remain available.
|
||||
- User can disable wake listening and keep manual microphone recording.
|
||||
|
||||
### Wake Word Technical Choice
|
||||
|
||||
Do not require local Whisper in v1.
|
||||
|
||||
Recommended path:
|
||||
|
||||
- Use browser Web Speech API for local wake-word detection when available.
|
||||
- Fall back to click-to-record when unavailable.
|
||||
- Upload only the post-wake instruction audio to backend ASR.
|
||||
- Add local `whisper.cpp` streaming wake-word or ASR provider later.
|
||||
|
||||
## Prompt Registry
|
||||
|
||||
Add default prompt keys:
|
||||
|
||||
```text
|
||||
agents.runtime.system
|
||||
agents.earth.command
|
||||
agents.situational.assessment
|
||||
agents.config.proposal
|
||||
agents.roles.network
|
||||
agents.roles.bgp
|
||||
agents.roles.platform_ops
|
||||
agents.roles.business_impact
|
||||
```
|
||||
|
||||
`agents.earth.command` must require:
|
||||
|
||||
- Strict JSON output only.
|
||||
- Use only provided candidate entities.
|
||||
- Do not invent objects.
|
||||
- Do not return arbitrary code.
|
||||
- Do not modify business data.
|
||||
- Return `clarification_needed` when intent is unclear.
|
||||
- Return action plans matching the schema.
|
||||
- Use Chinese summary for Chinese user input.
|
||||
|
||||
## Policy
|
||||
|
||||
### Tool Policy
|
||||
|
||||
- The LLM can request tools, but the backend decides whether to execute.
|
||||
- Every tool must declare:
|
||||
- name.
|
||||
- description.
|
||||
- input schema.
|
||||
- output schema.
|
||||
- permission.
|
||||
- side-effect level.
|
||||
- First-version side-effect levels:
|
||||
- `read`: can run directly.
|
||||
- `proposal`: can only generate proposals.
|
||||
- `write`: can only execute during proposal apply.
|
||||
|
||||
### Proposal Policy
|
||||
|
||||
First version supports configuration proposal application with role-based automatic/manual gating:
|
||||
|
||||
- `super_admin` may enable automatic application for low-risk proposals.
|
||||
- Normal admins must manually confirm.
|
||||
- High-risk proposals always require manual confirmation.
|
||||
- Proposals containing secret fields are rejected.
|
||||
- Proposals failing schema validation are rejected.
|
||||
- Before and after payloads must be saved.
|
||||
- Failed applications must save errors.
|
||||
|
||||
Low-risk scope:
|
||||
|
||||
- Datasource endpoint/config non-secret fields.
|
||||
- AI prompt overrides.
|
||||
- External integration non-secret fields.
|
||||
|
||||
Out of scope for v1:
|
||||
|
||||
- Alert acknowledge/resolve.
|
||||
- Data deletion.
|
||||
- User permission changes.
|
||||
- Authentication configuration changes.
|
||||
- Database schema changes by agent.
|
||||
- Mutation of original Earth collected data.
|
||||
|
||||
## Agent Operations UI
|
||||
|
||||
Add an `Agent` page under `运维与配置`.
|
||||
|
||||
Run list:
|
||||
|
||||
- Status.
|
||||
- Type.
|
||||
- Title.
|
||||
- Creator.
|
||||
- Model.
|
||||
- Time.
|
||||
- Duration.
|
||||
|
||||
Run detail:
|
||||
|
||||
- Input.
|
||||
- Status timeline.
|
||||
- LLM steps.
|
||||
- Tool steps.
|
||||
- Evidence.
|
||||
- Final result.
|
||||
- Proposals.
|
||||
|
||||
Proposal apply:
|
||||
|
||||
- Before/after diff.
|
||||
- Risk level.
|
||||
- Policy result.
|
||||
- Apply button.
|
||||
- Reject button.
|
||||
|
||||
Earth command run detail:
|
||||
|
||||
- Original text or speech transcription.
|
||||
- Matched entities.
|
||||
- Action plan.
|
||||
- Link back to Earth or copy run id.
|
||||
|
||||
## Frontend Types
|
||||
|
||||
Add or extend:
|
||||
|
||||
```ts
|
||||
interface AgentRun {}
|
||||
interface AgentStep {}
|
||||
interface AgentEvidence {}
|
||||
interface AgentProposal {}
|
||||
interface EarthCommandRequest {}
|
||||
interface EarthActionPlan {}
|
||||
interface EarthAction {}
|
||||
interface SpeechTranscriptionResponse {}
|
||||
```
|
||||
|
||||
Earth action executor API:
|
||||
|
||||
```js
|
||||
executeEarthActionPlan(plan, context)
|
||||
clearEarthCommandHighlights()
|
||||
getEarthCommandExecutionState()
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
### Backend Tests
|
||||
|
||||
Use `uv`.
|
||||
|
||||
Suggested files:
|
||||
|
||||
```text
|
||||
backend/tests/test_agents_runtime.py
|
||||
backend/tests/test_agent_tool_protocol.py
|
||||
backend/tests/test_agent_policy.py
|
||||
backend/tests/test_earth_agent_command.py
|
||||
backend/tests/test_speech_transcription.py
|
||||
```
|
||||
|
||||
Coverage:
|
||||
|
||||
- Create run.
|
||||
- Persist ordered run steps.
|
||||
- Parse valid JSON tool calls.
|
||||
- Reject invalid JSON tool calls without executing tools.
|
||||
- Reject unregistered tools.
|
||||
- Reject invalid tool arguments.
|
||||
- Pass through provider-native tools fields when configured.
|
||||
- Fall back to JSON tool calls when native tools are unavailable.
|
||||
- When WebSearch is not configured, record `missing_data` and do not fail Earth command.
|
||||
- "高亮所有北斗卫星" returns a satellite highlight action.
|
||||
- "中国大陆的算力中心" returns a compute-center highlight action.
|
||||
- Mainland China matching excludes Hong Kong, Macau, and Taiwan when fields permit it.
|
||||
- Field granularity limitations are returned in `missing_data`.
|
||||
- ASR not configured returns a clear error.
|
||||
- ASR provider success returns transcription.
|
||||
- Normal admins cannot auto-apply proposals.
|
||||
- `super_admin` can apply low-risk proposals.
|
||||
- Secret-field proposals are rejected.
|
||||
|
||||
### Frontend Tests
|
||||
|
||||
Use `bun`.
|
||||
|
||||
Required build check:
|
||||
|
||||
```bash
|
||||
cd frontend && bun run build
|
||||
```
|
||||
|
||||
Suggested Earth JS tests:
|
||||
|
||||
- Action executor opens the satellite layer.
|
||||
- Action executor highlights satellite entity ids.
|
||||
- Action executor highlights compute-center entity ids.
|
||||
- Clear highlight does not clear manual locked selection.
|
||||
- Search panel ordinary search still uses local search.
|
||||
- AI command button calls `/api/v1/agents/earth/command`.
|
||||
- ASR missing configuration and microphone permission denial show clear UI feedback.
|
||||
|
||||
### Manual Acceptance
|
||||
|
||||
1. Type `高亮所有北斗卫星`.
|
||||
- Satellite layer opens.
|
||||
- Beidou satellites are highlighted.
|
||||
- Panel shows count and summary.
|
||||
- Agent run is reviewable.
|
||||
|
||||
2. Type `显示中国大陆的算力中心`.
|
||||
- Compute-center layer opens.
|
||||
- Mainland China related compute centers are highlighted.
|
||||
- If data cannot separate mainland China from Hong Kong/Macau/Taiwan, the UI shows a missing-data note.
|
||||
|
||||
3. Click microphone.
|
||||
- Permission denial is clear.
|
||||
- Permission grant allows recording.
|
||||
- Configured ASR transcribes and executes the command.
|
||||
- If ASR is not configured, text input still works.
|
||||
|
||||
4. Enable voice wake.
|
||||
- Wake word is configurable.
|
||||
- Audio is not uploaded before wake.
|
||||
- After wake, one instruction audio segment is uploaded.
|
||||
- Wake listening can be disabled.
|
||||
|
||||
5. Open Agent operations UI.
|
||||
- Earth command run is listed.
|
||||
- Transcription step is visible when speech was used.
|
||||
- Entity query step is visible.
|
||||
- Final action plan is visible.
|
||||
- User can return to Earth or copy the run id.
|
||||
|
||||
## Documentation
|
||||
|
||||
When implemented, update:
|
||||
|
||||
```text
|
||||
docs/technical/zh/agents-aiprovider.md
|
||||
docs/technical/en/agents-aiprovider.md
|
||||
docs/technical/zh/manual.md
|
||||
docs/technical/en/manual.md
|
||||
docs/technical/zh/earth-frontend-context.md
|
||||
docs/technical/en/earth-frontend-context.md
|
||||
```
|
||||
|
||||
Document:
|
||||
|
||||
- `aiprovider` and backend agent boundaries.
|
||||
- Earth AI command entry.
|
||||
- Speech/ASR configuration.
|
||||
- Wake-word privacy behavior.
|
||||
- Agent run/evidence/proposal review.
|
||||
- V1 capability boundary: Earth visualization actions only, no business data mutation.
|
||||
|
||||
## Implementation Order
|
||||
|
||||
1. Add backend Agent data models, `init_db()` imports, and indexes.
|
||||
2. Add Agent schemas and run CRUD API.
|
||||
3. Add tool registry, JSON tool-call parser, and policy skeleton.
|
||||
4. Add Earth entity query service.
|
||||
5. Add `agents.earth.command` prompt and backend command endpoint.
|
||||
6. Add frontend Earth action executor.
|
||||
7. Merge AI command entry into Earth search panel.
|
||||
8. Add satellite and compute-center batch highlight support.
|
||||
9. Add Speech/ASR settings and transcription API.
|
||||
10. Add Earth microphone recording, wake-word local setting, and fallback behavior.
|
||||
11. Add Agent operations UI.
|
||||
12. Add proposal apply policy and low-risk configuration application.
|
||||
13. Add tests and documentation updates.
|
||||
|
||||
## Assumptions
|
||||
|
||||
- Earth LLM v1 only executes visualization actions.
|
||||
- Earth commands create agent runs but do not automatically create configuration proposals.
|
||||
- Configuration proposals are applied only from the Agent operations UI.
|
||||
- Wake word is a device-local preference in v1 and is stored in `localStorage`.
|
||||
- ASR is API-first and Whisper-compatible by default; local Whisper is a later provider.
|
||||
- Multi-role simulation only reserves schemas and prompt keys in v1.
|
||||
- Palantir-style situational workflows should grow from the shared evidence, entity, action, and assessment model instead of a separate isolated system.
|
||||
52
docs/plans/ai-prompt-settings-task-registry-plan.md
Normal file
52
docs/plans/ai-prompt-settings-task-registry-plan.md
Normal file
@@ -0,0 +1,52 @@
|
||||
# AI Prompt Settings and Task Registry Plan
|
||||
|
||||
## Summary
|
||||
|
||||
Add an AI prompt settings tab under the operations AI settings page. Operators can select a business AI task from a dropdown, edit its prompt, save the override, and reset it back to the shipped default. Runtime LLM calls must resolve prompts through a task registry instead of embedding large prompt blocks at each call site.
|
||||
|
||||
Default prompts are shipped as versioned resource data, not scattered business-code literals. Business services reference stable task keys, and the runtime resolves the effective prompt from the database override first, then the shipped default resource.
|
||||
|
||||
## Key Changes
|
||||
|
||||
- Add a backend task prompt registry with stable keys, labels, groups, versions, default system prompts, and default task prompts.
|
||||
- Store operator overrides in the existing `SystemSetting` table under an `ai_prompts` category. Store only custom overrides; defaults remain in the versioned prompt resource.
|
||||
- Add settings APIs:
|
||||
- `GET /api/v1/settings/ai-prompts`
|
||||
- `PUT /api/v1/settings/ai-prompts/{task_key}`
|
||||
- `POST /api/v1/settings/ai-prompts/{task_key}/reset`
|
||||
- Migrate business LLM entrypoints to resolve prompts by task key. `aiprovider` remains a pure model adapter and does not inject business prompts.
|
||||
- Add a “提示词” tab to `/ai`. The tab shows a grouped task dropdown, current/default prompt status, editable prompt fields, save, and reset-to-default controls.
|
||||
|
||||
## Initial Tasks
|
||||
|
||||
- `earth.news.enrich` — Earth news localization and location enrichment.
|
||||
- `alerts.brief` — system alert AI brief.
|
||||
- `alerts.situational.brief` — situational alert AI brief.
|
||||
- `bgp.brief` — BGP AI brief.
|
||||
- `location.factcheck.normalize` — location factcheck normalization.
|
||||
- `location.factcheck.resolve` — location factcheck fallback resolution.
|
||||
- `datasource.mapping` — datasource mapping DSL generation.
|
||||
- `credential.guide` — credential guide generation.
|
||||
- `ai.connection_test` — AI provider connection test.
|
||||
|
||||
Playground and public free-form analyze endpoints stay caller-controlled and are not shown in the prompt settings dropdown.
|
||||
|
||||
## Test Plan
|
||||
|
||||
- Backend uses `uv`:
|
||||
- `uv run pytest tests/test_settings_ai_prompts.py`
|
||||
- `uv run pytest tests/test_earth_news.py`
|
||||
- `uv run pytest tests/test_api.py`
|
||||
- Frontend uses `bun`:
|
||||
- `cd frontend && bun run build`
|
||||
- Manual checks:
|
||||
- Prompt dropdown switches task content correctly.
|
||||
- Save persists an override and runtime calls use it.
|
||||
- Reset deletes the override and restores the shipped default.
|
||||
- Alert prompts do not leak into news, BGP, datasource, or location tasks.
|
||||
|
||||
## Assumptions
|
||||
|
||||
- This iteration does not add prompt history, approval workflows, A/B testing, or per-user prompt variants.
|
||||
- Strict JSON tasks may fail validation if an operator edits away the output contract; existing task-specific failure and retry behavior remains responsible for recovery.
|
||||
- The UI stays Chinese-only in this iteration.
|
||||
182
docs/plans/ai-provider-openclaw-style-routing-plan.md
Normal file
182
docs/plans/ai-provider-openclaw-style-routing-plan.md
Normal file
@@ -0,0 +1,182 @@
|
||||
# AI Provider OpenClaw-Style Routing Plan
|
||||
|
||||
Last updated: 2026-05-20
|
||||
|
||||
## Summary
|
||||
|
||||
Planet 的 AI Provider 路由要从“运行时识别特殊 provider / 特殊模型”收敛到 OpenClaw 风格的配置驱动模型:模型引用、协议、鉴权、轻量探测、真实调用和模型级例外都由 provider catalog / preset / runtime metadata 描述,运行时只解释这些元数据,不再散落 `if provider == ... and model == ...` 这类硬编码。
|
||||
|
||||
这份计划覆盖 Admin Next 的 Provider 配置体验、backend settings API、`aiprovider` 适配服务和未来模型目录同步方式。目标是让 OpenCode Go、MiniMax、DeepSeek、OpenAI-compatible、Anthropic-compatible、Ollama、OpenRouter / One API 类代理都能用同一套规则扩展。
|
||||
|
||||
## Background
|
||||
|
||||
当前实现已经完成了两步临时修正:
|
||||
|
||||
- OpenCode Go 模型目录不再使用普通 Zen free 列表,而是使用 `https://opencode.ai/zen/go/v1/models`。
|
||||
- `minimax-m2.7` / `minimax-m2.5` 的协议例外已从 `aiprovider` 运行逻辑移到 `model_provider_apis` 元数据中。
|
||||
|
||||
但整体还没有完全达到 OpenClaw 式结构。OpenClaw 的关键思想是:
|
||||
|
||||
- 模型引用使用 `provider/model`,由 provider 前缀确定 runtime provider。
|
||||
- provider 插件或 catalog 拥有 `normalizeModelId`、`normalizeTransport`、`normalizeConfig`、`prepareRuntimeAuth`、`createStreamFn` 等 provider 行为。
|
||||
- 主推理循环不认识具体模型名,只使用解析后的 provider config、transport 和 request adapter。
|
||||
- 上游网关能自己路由时,尽量透传 provider routing metadata,不在本地复制上游逻辑。
|
||||
|
||||
Planet 不需要完整复制 OpenClaw 插件系统,但需要学习它的边界划分。
|
||||
|
||||
## Design Principles
|
||||
|
||||
- Provider catalog 是路由事实来源,runtime 不是。
|
||||
- 模型级协议例外必须是 metadata,例如 `model_provider_apis`,不能是 Python set / if 分支。
|
||||
- 轻量连通性测试只验证网络、鉴权和模型目录,不发真实 prompt。
|
||||
- 真实模型调用只发生在 Playground、AI brief、分析任务等明确需要生成的路径。
|
||||
- 保存配置不自动设为默认,不自动触发连接测试;保存、设默认、测试三种按钮职责分离。
|
||||
- 目录刷新使用增量合并语义:发现新模型,标记旧模型 stale,不直接删除用户选择或自定义模型。
|
||||
- 如果 provider 不提供可靠 `/models`,可以用内置 preset 确认已知模型,但 UI 必须说清楚这是 preset confirmation,不是假装 provider 返回了目录。
|
||||
|
||||
## Target Data Model
|
||||
|
||||
Provider preset / runtime config 应逐步收敛为类似结构:
|
||||
|
||||
```json
|
||||
{
|
||||
"provider": "opencode-go",
|
||||
"label": "OpenCode Go",
|
||||
"default_transport": "openai-completions",
|
||||
"base_url": "https://opencode.ai/zen/go/v1",
|
||||
"auth": {
|
||||
"type": "bearer",
|
||||
"api_key_env": "OPENCODE_GO_API_KEY"
|
||||
},
|
||||
"models": [
|
||||
{
|
||||
"id": "glm-5.1",
|
||||
"label": "GLM 5.1",
|
||||
"transport": "openai-completions",
|
||||
"context_window": null,
|
||||
"capabilities": ["text"]
|
||||
},
|
||||
{
|
||||
"id": "minimax-m2.7",
|
||||
"label": "MiniMax M2.7",
|
||||
"transport": "anthropic-messages",
|
||||
"capabilities": ["text", "reasoning"]
|
||||
}
|
||||
],
|
||||
"discovery": {
|
||||
"type": "openai-models",
|
||||
"url": "https://opencode.ai/zen/go/v1/models",
|
||||
"auth": "provider-api-key"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Runtime 选择规则:
|
||||
|
||||
1. 解析 provider。
|
||||
2. 解析 model。
|
||||
3. 从 `models[].transport` 找模型级 transport。
|
||||
4. 若没有模型级 transport,使用 provider `default_transport`。
|
||||
5. 将解析结果传给 `aiprovider`。
|
||||
6. `aiprovider` 只按 `transport` 组装请求,不认识 provider 专属模型名。
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
### Phase 1: Stabilize Current Metadata Path
|
||||
|
||||
- Keep `model_provider_apis` as the immediate compatibility bridge.
|
||||
- Ensure `_provider_defaults()` includes provider metadata such as `model_provider_apis`.
|
||||
- Ensure `_runtime_config_from_ai_payload()` sends the resolved metadata through `AIProviderClient`.
|
||||
- Ensure `AIProviderClient` forwards metadata to `aiprovider` with a structured header.
|
||||
- Ensure `aiprovider.ProviderService` reads model metadata and resolves `provider_api = model_provider_apis[model] ?? provider_api`.
|
||||
- Add tests proving `aiprovider` does not contain provider/model-specific literals for routing decisions.
|
||||
|
||||
### Phase 2: Replace `model_provider_apis` With Structured Model Catalog
|
||||
|
||||
- Extend `backend/app/services/llm_provider_catalog.py` preset shape with `models_metadata`.
|
||||
- Preserve old `models` as a compatibility list for the UI.
|
||||
- Add helpers:
|
||||
- `get_provider_model_metadata(provider, model)`
|
||||
- `resolve_provider_transport(provider_config, model)`
|
||||
- `merge_discovered_models(existing, discovered)`
|
||||
- Return both `models` and `models_metadata` from refresh endpoints.
|
||||
- Admin Next should render model labels, capabilities and transport hints from metadata.
|
||||
|
||||
### Phase 3: Provider Discovery And Incremental Sync
|
||||
|
||||
- Add provider discovery descriptors:
|
||||
- OpenAI-compatible `/models`
|
||||
- Anthropic-compatible no-models / preset-confirmed path
|
||||
- Ollama `/api/tags`
|
||||
- OpenCode Go `/zen/go/v1/models`
|
||||
- OpenRouter / One API passthrough model discovery
|
||||
- Add incremental merge behavior:
|
||||
- New discovered model: add.
|
||||
- Existing discovered model: update `last_seen_at`, metadata.
|
||||
- Missing discovered model: mark `stale`, do not delete.
|
||||
- User custom model: keep unless explicitly removed.
|
||||
- Surface discovery source in Admin Next: `实时发现 / 内置预设 / 用户自定义 / 已过期`.
|
||||
|
||||
### Phase 4: Transport Adapters
|
||||
|
||||
- Replace provider-specific request decisions with adapter descriptors:
|
||||
- `openai-completions`
|
||||
- `anthropic-messages`
|
||||
- `ollama-generate`
|
||||
- future `openai-responses`
|
||||
- future `gemini-generate-content`
|
||||
- Each adapter owns:
|
||||
- path
|
||||
- auth header format
|
||||
- request body transform
|
||||
- response text extraction
|
||||
- reasoning/thinking block extraction
|
||||
- models endpoint strategy
|
||||
- `ProviderService.analyze()` should select adapter by resolved transport and call the adapter.
|
||||
|
||||
### Phase 5: Admin Next UX
|
||||
|
||||
- Model provider page should show:
|
||||
- provider status tag
|
||||
- default model tag
|
||||
- source tag: env / runtime / preset / discovered
|
||||
- model list with transport/capability hint
|
||||
- separate buttons for save, set default, refresh model catalog, lightweight test
|
||||
- The connect plug button remains lightweight.
|
||||
- Full generation test lives only in Playground or a clearly named “试运行” action.
|
||||
- If lightweight test falls back to preset confirmation, toast must say so explicitly.
|
||||
|
||||
## TODO
|
||||
|
||||
- [ ] Add `models_metadata` to provider presets and refresh responses.
|
||||
- [ ] Add runtime resolver helper for provider/model transport selection.
|
||||
- [ ] Remove any remaining provider/model-specific literals from `aiprovider` runtime routing.
|
||||
- [ ] Add tests that `opencode-go/minimax-m2.7` resolves through metadata, not through runtime hardcode.
|
||||
- [ ] Add tests for lightweight connectivity:
|
||||
- [ ] 401 / 403 fail as auth error.
|
||||
- [ ] 404 with known preset model passes as preset-confirmed.
|
||||
- [ ] `/models` missing alias passes only when preset contains the alias.
|
||||
- [ ] unknown model fails.
|
||||
- [ ] Add discovery descriptors for OpenCode Go, OpenAI-compatible, Anthropic-compatible, Ollama, OpenRouter / One API.
|
||||
- [ ] Add incremental model catalog merge semantics with stale marking.
|
||||
- [ ] Update Admin Next model list to show model source, transport and capability.
|
||||
- [ ] Keep save / set default / lightweight test / full test as separate actions.
|
||||
- [ ] Document the final provider catalog schema in technical docs after implementation.
|
||||
|
||||
## Current Acceptance Criteria
|
||||
|
||||
- No runtime routing branch may depend on concrete model names like `minimax-m2.7`.
|
||||
- OpenCode Go model refresh must not use the ordinary Zen free-model endpoint.
|
||||
- Lightweight connect must not call `analyze()` or consume generation quota.
|
||||
- Saving a provider must not automatically set it as default.
|
||||
- Provider UI must distinguish configured key, fallback key, preset model and live-discovered model.
|
||||
|
||||
## Related Files
|
||||
|
||||
- `backend/app/services/llm_provider_catalog.py`
|
||||
- `backend/app/api/v1/settings.py`
|
||||
- `backend/app/services/ai_client.py`
|
||||
- `aiprovider/main.py`
|
||||
- `aiprovider/provider_service.py`
|
||||
- `frontend/src/admin-next/pages/PlainResourcePages.tsx`
|
||||
- `docs/plans/admin-next-parity-audit-closeout-plan.md`
|
||||
@@ -13,7 +13,7 @@
|
||||
| 采集时是否调用 LLM | 不调用;采集链路必须确定性、可审计、可复现 |
|
||||
| 自定义数据如何进入 Earth | 必须映射到已支持的目标 schema,或先进入通用数据沉淀 |
|
||||
| 外部凭证放置位置 | Settings / 外部集成统一管理 provider token;DataSources 引用 provider profile |
|
||||
| TimescaleDB | 放入 TODO;高频时序数据稳定后再评估迁移 |
|
||||
| TimescaleDB | 高频时序数据稳定后再评估迁移 |
|
||||
|
||||
---
|
||||
|
||||
@@ -57,7 +57,7 @@ flowchart LR
|
||||
| schema | 用途 | Earth 可视化 |
|
||||
|-------|------|-------------|
|
||||
| `vessel_ais` | 船只 AIS 位置、航速、航向、MMSI 等 | 进入船舶图层 |
|
||||
| `geo_points` | 通用点位数据,包含经纬度、名称、类型、时间 | 进入通用 geo layer(TODO) |
|
||||
| `geo_points` | 通用点位数据,包含经纬度、名称、类型、时间 | 未来进入通用 geo layer |
|
||||
| `news_events` | 新闻/事件类数据,带时间、地点、摘要、来源 | 复用新闻/事件链路 |
|
||||
| `compute_centers` | 算力中心、机房、数据中心数据 | 复用算力中心图层 |
|
||||
| `generic_records` | 未知结构化数据沉淀 | 不直接展示 |
|
||||
@@ -294,7 +294,7 @@ PostgreSQL 可以承载当前规模的采集、关系查询、JSONB 沉淀和基
|
||||
- 查询模式还没稳定。
|
||||
- 需要快速迭代 schema 与 mapping。
|
||||
|
||||
### TODO:TimescaleDB
|
||||
### TimescaleDB 后续评估
|
||||
|
||||
以下条件满足后,再评估 TimescaleDB:
|
||||
|
||||
@@ -324,7 +324,7 @@ PostgreSQL 可以承载当前规模的采集、关系查询、JSONB 沉淀和基
|
||||
- Settings 中保存 provider credentials。
|
||||
- API 返回配置时必须 mask secret。
|
||||
- LLM prompt 只能包含脱敏 sample 和 schema 说明。
|
||||
- 后续 TODO:引入字段级加密或 KMS。
|
||||
- 后续可引入字段级加密或 KMS。
|
||||
|
||||
### Mapping 治理
|
||||
|
||||
|
||||
@@ -31,7 +31,7 @@
|
||||
3. **登录与找回密码** — 登录页、忘记密码流程
|
||||
4. **账户设置** — 修改密码、修改邮箱(需重新验证)、查看权限组、登出
|
||||
5. **Console 总览** — 左侧菜单结构、各路由用途
|
||||
6. **配置数据采集器** — `/settings?tab=collector_credentials`:选择 collector、连接测试、保存凭证;BarentsWatch / AISStream 两个典型例子
|
||||
6. **配置数据采集器** — `/collection-management?tab=collector_credentials`:选择 collector、连接测试、保存凭证;BarentsWatch / AISStream 两个典型例子
|
||||
7. **配置 AI 凭证** — `/ai?tab=providers`:默认 provider、模型、Base URL、API Key、本地代理;工具 tab(WebSearch、OCR)
|
||||
8. **系统设置** — `/settings` 其他子 tab(系统设置、电视直播源、SMTP 邮件)
|
||||
9. **用户管理(管理员)** — `/users`:创建、删除、改角色、Gatekeeper 权限组
|
||||
@@ -48,7 +48,7 @@
|
||||
|
||||
- 打开管理员给你的 URL
|
||||
- 注册账号 + 邮箱验证
|
||||
- 登录后第一次做什么(建议先到 `/settings?tab=collector_credentials` 配一个 collector,再到 `/ai` 配模型)
|
||||
- 登录后第一次做什么(建议先到 `/collection-management?tab=collector_credentials` 配一个 collector,再到 `/ai` 配模型)
|
||||
- 看 Earth
|
||||
|
||||
部署/开发的 quickstart 内容并入 `ops-runbook.md` 的"首次部署"小节,**不**再单独出 `ops-quickstart.md`,避免新增维护点。
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user