Compare commits
24 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
9523f2bb83 | ||
|
|
9dc3b61fa5 | ||
|
|
698863da68 | ||
|
|
4c0b49962c | ||
|
|
ffb3134d93 | ||
|
|
e8742b5d3c | ||
|
|
28a7132eec | ||
|
|
dd41588527 | ||
|
|
45604a1ebb | ||
|
|
06c122bade | ||
|
|
96e9b15fda | ||
|
|
a8713e58fd | ||
|
|
05377bf5c6 | ||
|
|
67548d8278 | ||
|
|
4b8feeb87e | ||
|
|
a2025dbfe0 | ||
|
|
003b316d42 | ||
|
|
0c40764c58 | ||
|
|
5bfc790dec | ||
|
|
60eefb19c9 | ||
|
|
83a10a6c34 | ||
|
|
cee1996809 | ||
|
|
58671e7bc3 | ||
|
|
a54fcdbeed |
@@ -3,8 +3,23 @@
|
||||
!pyproject.toml
|
||||
!uv.lock
|
||||
!VERSION
|
||||
!frontend/
|
||||
!frontend/public/
|
||||
!frontend/public/earth/
|
||||
!frontend/public/earth/assets/
|
||||
!frontend/public/earth/assets/cards/
|
||||
!frontend/public/earth/assets/cards/manifest.json
|
||||
!backend/
|
||||
!backend/**
|
||||
!config/
|
||||
!config/earth-boundary-pov-policy.china-v1.json
|
||||
!scripts/
|
||||
!scripts/build_earth_boundary_china_pov_geojson.py
|
||||
!scripts/build_earth_boundary_pmtiles.py
|
||||
!scripts/earth_boundary_policy.py
|
||||
!resources/
|
||||
!resources/earth/
|
||||
!resources/earth/boundary-pov-geometry.china-v1.json
|
||||
!aiprovider/
|
||||
!aiprovider/**
|
||||
|
||||
|
||||
5
.gitignore
vendored
5
.gitignore
vendored
@@ -160,3 +160,8 @@ data/earth-boundary-sources/
|
||||
# Git; publish PMTiles/MVT artifacts through release/deploy storage instead of
|
||||
# committing thousands of generated tile files.
|
||||
frontend/public/earth/data/boundaries/
|
||||
|
||||
# Offline comparison originals; runtime 8K maps and provenance remain versioned.
|
||||
resources/earth/candidates/nasa-blue-marble/*.jpg
|
||||
resources/earth/candidates/natural-earth-3/*.jpg
|
||||
*:Zone.Identifier
|
||||
|
||||
@@ -113,6 +113,9 @@ behavior and document the compatibility note in `docs/harness-audit.md` or
|
||||
## Operating Mode
|
||||
|
||||
- Default to acting directly when the user gives a clear task.
|
||||
- For authorized multi-agent work, follow the Coding Agent Model Strategy in
|
||||
`rules.md` workflow: Sol for implementation, Luna for bounded supporting work,
|
||||
and Astra for complex reasoning; the main agent owns final verification.
|
||||
- Ask before acting only when the missing decision is risky, cannot be
|
||||
discovered from repository context, and no conservative assumption is safe.
|
||||
- Read relevant files before editing.
|
||||
|
||||
@@ -17,6 +17,7 @@ are the source of detail for specific subsystems.
|
||||
| `deploy/helm/planet/` | Helm chart for staging/deployment smoke paths | CI runs helm lint/template when delivery checks are available. |
|
||||
| `.gitea/workflows/` | CI, release image build, staging deploy workflows | This repository uses Gitea workflow files, not `.github/workflows/`. |
|
||||
| `planet.sh` | Main local lifecycle script | Owns init/start/restart/stop/health/log/createuser/destroy. |
|
||||
| `planet.cmd` and `scripts/windows/` | Portable Windows entry, bound desktop launcher generation, and WSL restart bridge | Delegates service lifecycle to `planet.sh`; Windows and isolated WSL regression tests live under `scripts/harness/`. |
|
||||
|
||||
## Runtime Entry Points
|
||||
|
||||
|
||||
@@ -168,10 +168,12 @@
|
||||
|
||||
## 快速启动
|
||||
|
||||
入口需要先具备 `zsh`、`curl` 和可访问的软件源。Ubuntu / Ubuntu WSL 上,`init` 会自动检测并补装 Docker Engine、Compose v2 和 Buildx,启动 Docker 服务并配置当前用户的访问权限;需要系统权限时会提示输入 sudo 密码。其他系统请先准备可用的 Docker 环境。
|
||||
|
||||
```bash
|
||||
# 新机器或空项目首次初始化
|
||||
./planet.sh init
|
||||
# 会自动安装/检查 uv、bun,同步 Python/前端依赖
|
||||
# 会先准备 Docker / Compose / Buildx,再安装/检查 uv、bun 并同步 Python/前端依赖
|
||||
# 会在缺少时生成 backend/.env、aiprovider/.env、frontend/.env.local
|
||||
# 会启动 PostgreSQL/Redis,并创建表、默认数据源和本地默认用户
|
||||
|
||||
@@ -322,7 +324,7 @@ ipconfig
|
||||
|
||||
## 启动容错参数
|
||||
|
||||
`planet.sh` 现在为依赖安装、数据库、AI Provider 启动加入了有限次重试,并会在数据库与 `aiprovider` 启动后额外等待 Docker healthcheck。
|
||||
`planet.sh` 为依赖安装、数据库、AI Provider 启动提供有限次重试。数据库先检查容器健康,再验证后端实际连接;`aiprovider` 直接以宿主机 `/health` 就绪为准。后端进程退出或应用初始化失败时立即停止等待,避免重复消耗健康检查预算。
|
||||
|
||||
可通过环境变量临时调整:
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# syntax=docker/dockerfile:1.7
|
||||
# Use BuildKit's bundled frontend to avoid a separate Docker Hub fetch.
|
||||
|
||||
ARG PYTHON_IMAGE=python:3.14-slim
|
||||
ARG UV_IMAGE=ghcr.io/astral-sh/uv:latest
|
||||
@@ -7,9 +7,6 @@ ARG AI_PROVIDER_BUILD_FINGERPRINT=unknown
|
||||
FROM ${UV_IMAGE} AS uv
|
||||
FROM ${PYTHON_IMAGE}
|
||||
|
||||
ARG AI_PROVIDER_BUILD_FINGERPRINT
|
||||
LABEL planet.aiprovider.build-fingerprint="${AI_PROVIDER_BUILD_FINGERPRINT}"
|
||||
|
||||
COPY --from=uv /uv /uvx /bin/
|
||||
|
||||
WORKDIR /app
|
||||
@@ -28,13 +25,16 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
COPY pyproject.toml uv.lock /app/
|
||||
RUN --mount=type=cache,target=/root/.cache/uv \
|
||||
--mount=type=secret,id=planet_uv_config,target=/root/.config/uv/uv.toml,required=false \
|
||||
uv sync --frozen --no-dev
|
||||
uv sync --frozen --only-group aiprovider
|
||||
|
||||
COPY aiprovider /app/aiprovider
|
||||
|
||||
ARG AI_PROVIDER_BUILD_FINGERPRINT
|
||||
LABEL planet.aiprovider.build-fingerprint="${AI_PROVIDER_BUILD_FINGERPRINT}"
|
||||
|
||||
EXPOSE 8010
|
||||
|
||||
HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
|
||||
CMD curl -fsS http://127.0.0.1:8010/health >/dev/null || exit 1
|
||||
|
||||
CMD ["uv", "run", "--frozen", "--no-dev", "--project", "/app", "python", "-m", "uvicorn", "aiprovider.main:app", "--host", "0.0.0.0", "--port", "8010"]
|
||||
CMD ["/app/.venv/bin/python", "-m", "uvicorn", "aiprovider.main:app", "--host", "0.0.0.0", "--port", "8010"]
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
|
||||
完整使用说明见:
|
||||
|
||||
- [docs/agents/aiprovider.md](/home/ray/dev/linkong/planet/docs/agents/aiprovider.md)
|
||||
- [AI Provider 指南](../docs/technical/zh/agents-aiprovider.md)
|
||||
|
||||
当前支持:
|
||||
|
||||
@@ -15,6 +15,7 @@
|
||||
- `AI_PROVIDER=ollama`
|
||||
- request adapter:
|
||||
- `AI_PROVIDER_API=openai-completions`
|
||||
- `AI_PROVIDER_API=openai-responses`
|
||||
- `AI_PROVIDER_API=anthropic-messages`
|
||||
- `AI_PROVIDER_API=ollama-generate`
|
||||
|
||||
|
||||
@@ -3,6 +3,7 @@ from __future__ import annotations
|
||||
import asyncio
|
||||
import json
|
||||
from typing import Any
|
||||
from uuid import NAMESPACE_URL, uuid4, uuid5
|
||||
|
||||
import httpx
|
||||
from fastapi import HTTPException, status
|
||||
@@ -65,6 +66,7 @@ class ProviderService:
|
||||
self.model_provider_apis = self._parse_model_provider_apis(
|
||||
overrides.get("model_provider_apis")
|
||||
)
|
||||
self.session_id = str(uuid4())
|
||||
|
||||
def get_status(self) -> AIProviderStatusResponse:
|
||||
enabled = self.provider != "disabled"
|
||||
@@ -95,6 +97,8 @@ class ProviderService:
|
||||
)
|
||||
|
||||
prompt = self._build_prompt(payload)
|
||||
if payload.context.get("session_id") is not None:
|
||||
self.session_id = str(uuid5(NAMESPACE_URL, f"planet:{payload.context['session_id']}"))
|
||||
|
||||
provider_api = self._resolve_model_provider_api(model)
|
||||
|
||||
@@ -102,6 +106,10 @@ class ProviderService:
|
||||
data = await self._request_openai_compatible(model, prompt, payload.system_prompt)
|
||||
content = self._extract_openai_content(data)
|
||||
content_blocks = self._extract_openai_blocks(data)
|
||||
elif provider_api == "openai-responses":
|
||||
data = await self._request_openai_responses(model, prompt, payload.system_prompt)
|
||||
content_blocks = self._extract_responses_blocks(data)
|
||||
content = "".join(block.text for block in content_blocks if block.text)
|
||||
elif provider_api == "anthropic-messages":
|
||||
data = await self._request_anthropic_messages(
|
||||
model,
|
||||
@@ -200,6 +208,39 @@ class ProviderService:
|
||||
request_body=request_body,
|
||||
)
|
||||
|
||||
async def _request_openai_responses(
|
||||
self, model: str, prompt: str, system_prompt: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
request_body: dict[str, Any] = {
|
||||
"model": model, "input": prompt, "max_output_tokens": self.max_tokens, "store": False,
|
||||
}
|
||||
resolved_system_prompt = self._resolve_system_prompt(system_prompt)
|
||||
if resolved_system_prompt:
|
||||
request_body["instructions"] = resolved_system_prompt
|
||||
return await self._post(
|
||||
path="/responses",
|
||||
headers={"Authorization": f"Bearer {self.api_key}", "Content-Type": "application/json"},
|
||||
request_body=request_body,
|
||||
)
|
||||
|
||||
def _extract_responses_blocks(self, payload: dict[str, Any]) -> list[AIContentBlock]:
|
||||
blocks: list[AIContentBlock] = []
|
||||
for item in payload.get("output") or []:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
if item.get("type") == "message":
|
||||
for part in item.get("content") or []:
|
||||
if not isinstance(part, dict):
|
||||
continue
|
||||
text = part.get("text") or part.get("refusal")
|
||||
if isinstance(text, str) and text:
|
||||
blocks.append(AIContentBlock(type="text", text=text))
|
||||
elif item.get("type") == "reasoning":
|
||||
for part in item.get("summary") or []:
|
||||
if isinstance(part, dict) and isinstance(part.get("text"), str):
|
||||
blocks.append(AIContentBlock(type="thinking", thinking=part["text"]))
|
||||
return blocks
|
||||
|
||||
async def _request_anthropic_messages(
|
||||
self,
|
||||
model: str,
|
||||
@@ -226,7 +267,7 @@ class ProviderService:
|
||||
resolved_system_prompt = self._resolve_system_prompt(system_prompt)
|
||||
if resolved_system_prompt:
|
||||
request_body["system"] = resolved_system_prompt
|
||||
resolved_thinking = self._resolve_anthropic_thinking(thinking)
|
||||
resolved_thinking = self._resolve_anthropic_thinking(thinking, model)
|
||||
if resolved_thinking:
|
||||
request_body["thinking"] = resolved_thinking
|
||||
if self.provider == "minimax" and self.base_url.endswith("/anthropic"):
|
||||
@@ -243,8 +284,12 @@ class ProviderService:
|
||||
request_body=request_body,
|
||||
)
|
||||
|
||||
def _resolve_anthropic_thinking(self, thinking: dict[str, Any] | None) -> dict[str, Any] | None:
|
||||
def _resolve_anthropic_thinking(
|
||||
self, thinking: dict[str, Any] | None, model: str,
|
||||
) -> dict[str, Any] | None:
|
||||
if thinking:
|
||||
if model.casefold() == "minimax-m3" and thinking.get("type") == "enabled":
|
||||
return {"type": "adaptive"}
|
||||
return thinking
|
||||
|
||||
# OpenClaw treats MiniMax's Anthropic-compatible path specially:
|
||||
@@ -295,6 +340,9 @@ class ProviderService:
|
||||
headers: dict[str, str],
|
||||
request_body: dict[str, Any],
|
||||
) -> dict[str, Any]:
|
||||
headers = {"User-Agent": "Planet/1.0", **headers}
|
||||
if self.provider == "opencode-go":
|
||||
headers["x-opencode-session"] = self.session_id
|
||||
last_error: Exception | None = None
|
||||
for attempt in range(1, self.http_retry_attempts + 1):
|
||||
try:
|
||||
|
||||
@@ -28,6 +28,12 @@ RUN --mount=type=cache,target=/root/.cache/uv \
|
||||
uv sync --frozen --no-dev
|
||||
|
||||
COPY backend /app/backend
|
||||
COPY config/earth-boundary-pov-policy.china-v1.json /app/config/earth-boundary-pov-policy.china-v1.json
|
||||
COPY scripts/build_earth_boundary_china_pov_geojson.py /app/scripts/build_earth_boundary_china_pov_geojson.py
|
||||
COPY scripts/build_earth_boundary_pmtiles.py /app/scripts/build_earth_boundary_pmtiles.py
|
||||
COPY scripts/earth_boundary_policy.py /app/scripts/earth_boundary_policy.py
|
||||
COPY resources/earth/boundary-pov-geometry.china-v1.json /app/resources/earth/boundary-pov-geometry.china-v1.json
|
||||
COPY frontend/public/earth/assets/cards/manifest.json /app/frontend/public/earth/assets/cards/manifest.json
|
||||
COPY VERSION /app/VERSION
|
||||
|
||||
EXPOSE 8000
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
from app.api.v1 import card_images
|
||||
from fastapi import APIRouter
|
||||
from app.api.v1 import (
|
||||
ai,
|
||||
@@ -12,6 +13,7 @@ from app.api.v1 import (
|
||||
alerts,
|
||||
settings,
|
||||
collected_data,
|
||||
compute_intelligence,
|
||||
data_products,
|
||||
layers,
|
||||
visualization,
|
||||
@@ -43,6 +45,7 @@ api_router.include_router(alerts.router, prefix="/alerts", tags=["alerts"])
|
||||
api_router.include_router(settings.router, prefix="/settings", tags=["settings"])
|
||||
api_router.include_router(system_control.router, prefix="/system", tags=["system"])
|
||||
api_router.include_router(data_products.router, prefix="/data-products", tags=["data-products"])
|
||||
api_router.include_router(compute_intelligence.router, prefix="/data-products/compute", tags=["compute"])
|
||||
api_router.include_router(layers.router, prefix="/layers", tags=["layers"])
|
||||
api_router.include_router(visualization.router, prefix="/visualization", tags=["visualization"])
|
||||
api_router.include_router(
|
||||
@@ -56,3 +59,5 @@ api_router.include_router(tv.router, prefix="/tv", tags=["tv"])
|
||||
api_router.include_router(news.router, prefix="/news", tags=["news"])
|
||||
api_router.include_router(interactables.router, prefix="/interactables", tags=["interactables"])
|
||||
api_router.include_router(realtime_sources.router, prefix="/realtime-sources", tags=["realtime-sources"])
|
||||
|
||||
api_router.include_router(card_images.router, prefix="/earth/card-images", tags=["card-images"])
|
||||
|
||||
108
backend/app/api/v1/card_images.py
Normal file
108
backend/app/api/v1/card_images.py
Normal file
@@ -0,0 +1,108 @@
|
||||
"""Card-image configuration, upload, catalog selection and public resolution."""
|
||||
|
||||
from fastapi import APIRouter, Depends, File, HTTPException, Query, Response, UploadFile
|
||||
from fastapi.responses import FileResponse
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from starlette.concurrency import run_in_threadpool
|
||||
|
||||
from app.core.enums import UserRole
|
||||
from app.core.security import get_current_user
|
||||
from app.db.session import get_db
|
||||
from app.models.user import User
|
||||
from app.schemas.card_images import (
|
||||
CardImageConfiguration,
|
||||
CardImagePreview,
|
||||
CardImagePublish,
|
||||
EntityType,
|
||||
MatchScope,
|
||||
)
|
||||
from app.services.card_image_configuration import (
|
||||
publish_configuration,
|
||||
read_configuration,
|
||||
resolve_image,
|
||||
)
|
||||
from app.services.card_image_files import MAX_UPLOAD_BYTES, store_image, uploaded_path
|
||||
from app.services.card_image_targets import get_target, search_targets
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
async def image_admin(user: User = Depends(get_current_user)) -> User:
|
||||
if user.role not in {UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value}:
|
||||
raise HTTPException(403, "Administrator access required")
|
||||
return user
|
||||
|
||||
|
||||
@router.get("/configuration")
|
||||
async def get_configuration(
|
||||
response: Response, _actor: User = Depends(image_admin), db: AsyncSession = Depends(get_db)
|
||||
):
|
||||
response.headers["Cache-Control"] = "no-store"
|
||||
return await read_configuration(db)
|
||||
|
||||
|
||||
@router.put("/configuration")
|
||||
async def save_configuration(
|
||||
payload: CardImagePublish,
|
||||
actor: User = Depends(image_admin),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await publish_configuration(db, payload.revision, payload.config, actor)
|
||||
|
||||
|
||||
@router.post("/upload")
|
||||
async def upload_image(file: UploadFile = File(...), _actor: User = Depends(image_admin)):
|
||||
content = await file.read(MAX_UPLOAD_BYTES + 1)
|
||||
await file.close()
|
||||
return await run_in_threadpool(store_image, content, (file.filename or "Image")[:200])
|
||||
|
||||
|
||||
@router.get("/files/{filename}")
|
||||
async def image_file(filename: str):
|
||||
path = uploaded_path(filename)
|
||||
if not path.is_file():
|
||||
raise HTTPException(404, "Image not found")
|
||||
return FileResponse(
|
||||
path,
|
||||
headers={
|
||||
"Cache-Control": "public, max-age=31536000, immutable",
|
||||
"X-Content-Type-Options": "nosniff",
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@router.get("/targets")
|
||||
async def targets(
|
||||
entity_type: EntityType,
|
||||
scope: MatchScope = "object",
|
||||
q: str = Query(default="", max_length=100),
|
||||
limit: int = Query(default=25, ge=1, le=50),
|
||||
_actor: User = Depends(image_admin),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return {"items": await search_targets(db, entity_type, scope, q.strip(), limit)}
|
||||
|
||||
|
||||
@router.get("/resolve")
|
||||
async def resolve(
|
||||
response: Response,
|
||||
entity_type: EntityType,
|
||||
entity_id: str = Query(min_length=1, max_length=300),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
response.headers["Cache-Control"] = "no-store"
|
||||
envelope = await read_configuration(db)
|
||||
config = CardImageConfiguration.model_validate(envelope["config"])
|
||||
context = await get_target(db, entity_type, entity_id)
|
||||
result = resolve_image(config, context)
|
||||
return {**result, "revision": envelope["revision"], "entity": context}
|
||||
|
||||
|
||||
@router.post("/preview")
|
||||
async def preview(
|
||||
payload: CardImagePreview,
|
||||
_actor: User = Depends(image_admin),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
context = await get_target(db, payload.entity_type, payload.entity_id)
|
||||
return {**resolve_image(payload.config, context), "entity": context}
|
||||
235
backend/app/api/v1/compute_intelligence.py
Normal file
235
backend/app/api/v1/compute_intelligence.py
Normal file
@@ -0,0 +1,235 @@
|
||||
"""Compute observations, administrator configuration and independent facility samples."""
|
||||
|
||||
from datetime import UTC, datetime
|
||||
from typing import Literal
|
||||
|
||||
from app.core.compute_types import COMPUTE_SITE_TYPES, compute_record_filter, compute_site_type_expression
|
||||
from app.api.v1.visualization import convert_compute_centers_to_geojson
|
||||
from app.core.earth_geography_policy import display_country
|
||||
from app.core.enums import UserRole
|
||||
from app.core.security import get_current_user
|
||||
from app.db.session import get_db
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.models.data_snapshot import DataSnapshot
|
||||
from app.models.user import User
|
||||
from app.schemas.compute_configuration import (
|
||||
ComputeConfiguration,
|
||||
ComputeConfigurationReset,
|
||||
ComputeConfigurationWrite,
|
||||
)
|
||||
from app.services.compute_catalog import (
|
||||
PARTICIPANTS,
|
||||
participant_members,
|
||||
source_country_names,
|
||||
source_country_participants,
|
||||
)
|
||||
from app.services.compute_center_locations import refresh_compute_center_location_cache
|
||||
from app.services.compute_configuration import (
|
||||
DEFAULT_CONFIGURATION,
|
||||
configuration_response,
|
||||
publish_configuration,
|
||||
read_configuration,
|
||||
)
|
||||
from app.services.compute_history import compute_history
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Response
|
||||
from sqlalchemy import and_, case, func, or_, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
def selected_countries(countries: str) -> list[str]:
|
||||
codes = list(dict.fromkeys(code.strip().upper() for code in countries.split(",")))
|
||||
if not codes or len(codes) > 12 or any(code not in PARTICIPANTS for code in codes):
|
||||
raise HTTPException(422, "Select 1–12 participants from the compute catalog")
|
||||
members = participant_members(codes)
|
||||
flattened = [country for participant in members.values() for country in participant]
|
||||
if len(flattened) != len(set(flattened)):
|
||||
raise HTTPException(422, "Comparison participants cannot have overlapping country members")
|
||||
return codes
|
||||
|
||||
|
||||
@router.get("/series")
|
||||
async def get_series(
|
||||
response: Response,
|
||||
countries: str = "CN,US",
|
||||
mode: Literal["global_share", "bilateral_share", "absolute"] = "global_share",
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
codes = selected_countries(countries)
|
||||
envelope = await read_configuration(db)
|
||||
response.headers["Cache-Control"] = "no-store"
|
||||
result = compute_history(codes, mode, ComputeConfiguration.model_validate(envelope["config"]))
|
||||
result.update(
|
||||
version=envelope["revision"],
|
||||
provenance="bundled_reports" if envelope["is_default"] else "admin_curated_reports",
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
async def configuration_admin(user: User = Depends(get_current_user)) -> User:
|
||||
if user.role not in {UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value}:
|
||||
raise HTTPException(403, "Administrator access required")
|
||||
return user
|
||||
|
||||
|
||||
@router.get("/configuration")
|
||||
async def get_configuration(
|
||||
response: Response,
|
||||
_actor: User = Depends(configuration_admin),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
response.headers["Cache-Control"] = "no-store"
|
||||
return configuration_response(await read_configuration(db))
|
||||
|
||||
|
||||
@router.put("/configuration")
|
||||
async def update_configuration(
|
||||
payload: ComputeConfigurationWrite,
|
||||
actor: User = Depends(configuration_admin),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
envelope = await publish_configuration(
|
||||
db, revision=payload.revision, config=payload.config, actor=actor,
|
||||
)
|
||||
return configuration_response(envelope)
|
||||
|
||||
|
||||
@router.post("/configuration/reset")
|
||||
async def reset_configuration(
|
||||
payload: ComputeConfigurationReset,
|
||||
actor: User = Depends(configuration_admin),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
envelope = await publish_configuration(
|
||||
db, revision=payload.revision, config=DEFAULT_CONFIGURATION, actor=actor, reset=True,
|
||||
)
|
||||
return configuration_response(envelope)
|
||||
|
||||
|
||||
async def sample_query(db: AsyncSession, countries: list[str], year: int | None):
|
||||
now = datetime.now(UTC)
|
||||
cutoff = min(datetime(year + 1, 1, 1, tzinfo=UTC), now) if year else now
|
||||
ranked = (
|
||||
select(
|
||||
DataSnapshot.id,
|
||||
func.row_number()
|
||||
.over(
|
||||
partition_by=DataSnapshot.source,
|
||||
order_by=(DataSnapshot.completed_at.desc(), DataSnapshot.id.desc()),
|
||||
)
|
||||
.label("rank"),
|
||||
)
|
||||
.where(
|
||||
DataSnapshot.source.in_(select(CollectedData.source).where(compute_record_filter(CollectedData)).distinct()),
|
||||
DataSnapshot.status == "success",
|
||||
DataSnapshot.completed_at < cutoff,
|
||||
)
|
||||
.subquery()
|
||||
)
|
||||
snapshots = select(ranked.c.id).where(ranked.c.rank == 1)
|
||||
infrastructure = compute_site_type_expression(CollectedData).in_(COMPUTE_SITE_TYPES[2:])
|
||||
# Old collector rows have ambiguous units/identity and must not become
|
||||
# national evidence merely because a new UI can display them.
|
||||
query = select(CollectedData).where(
|
||||
compute_record_filter(CollectedData),
|
||||
CollectedData.deleted_at.is_(None),
|
||||
CollectedData.snapshot_id.in_(snapshots),
|
||||
CollectedData.is_valid == 1,
|
||||
or_(CollectedData.extra_data["compute_contract"].as_string() == "v1", infrastructure),
|
||||
or_(
|
||||
infrastructure,
|
||||
CollectedData.source == "top500",
|
||||
and_(
|
||||
CollectedData.extra_data["status"].as_string() == "Existing",
|
||||
func.coalesce(CollectedData.extra_data["superseded_by"].as_string(), "") == "",
|
||||
CollectedData.extra_data["possible_duplicate"].as_boolean().is_not(True),
|
||||
),
|
||||
),
|
||||
CollectedData.extra_data["country"].as_string().in_(source_country_names(countries)),
|
||||
)
|
||||
return query
|
||||
|
||||
|
||||
@router.get("/facilities")
|
||||
async def get_facilities(
|
||||
countries: str = "CN,US",
|
||||
year: int | None = Query(None, ge=2000, le=2100),
|
||||
limit: int = Query(1000, ge=1, le=2000),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
codes = selected_countries(countries)
|
||||
query = await sample_query(db, codes, year)
|
||||
country = CollectedData.extra_data["country"].as_string()
|
||||
source_participants = source_country_participants(codes)
|
||||
# Canonical source subdivisions retain their raw label on each record but
|
||||
# share one owner in the comparison, using the same policy as the globe.
|
||||
owner_cases = [
|
||||
(country == name, display_country(name))
|
||||
for name in source_country_names(codes)
|
||||
if display_country(name) != name
|
||||
]
|
||||
display = case(*owner_cases, else_=country) if owner_cases else country
|
||||
participant_cases = [(country == name, code) for name, code in source_participants.items()]
|
||||
participant = case(*participant_cases, else_=None)
|
||||
counts = (
|
||||
(
|
||||
await db.execute(
|
||||
query.with_only_columns(
|
||||
display.label("country"),
|
||||
participant.label("participant_code"),
|
||||
CollectedData.source,
|
||||
func.count().label("count"),
|
||||
).group_by(display, participant, CollectedData.source)
|
||||
)
|
||||
)
|
||||
.mappings()
|
||||
.all()
|
||||
)
|
||||
participant_counts = (
|
||||
(
|
||||
await db.execute(
|
||||
query.with_only_columns(
|
||||
participant.label("participant_code"),
|
||||
CollectedData.source,
|
||||
func.count().label("count"),
|
||||
).group_by(participant, CollectedData.source)
|
||||
)
|
||||
)
|
||||
.mappings()
|
||||
.all()
|
||||
)
|
||||
records = (
|
||||
(await db.execute(query.order_by(CollectedData.name, CollectedData.id).limit(limit)))
|
||||
.scalars()
|
||||
.all()
|
||||
)
|
||||
if year:
|
||||
await refresh_compute_center_location_cache(db, as_of=datetime(year + 1, 1, 1, tzinfo=UTC))
|
||||
else:
|
||||
await refresh_compute_center_location_cache(db)
|
||||
payload = convert_compute_centers_to_geojson(records)
|
||||
for feature in payload["features"]:
|
||||
feature["properties"]["historical_snapshot"] = year is not None
|
||||
properties = feature["properties"]
|
||||
properties["source_country"] = properties.get("country")
|
||||
properties["country"] = display_country(properties.get("country"))
|
||||
total = sum(row["count"] for row in counts)
|
||||
payload.update(
|
||||
{
|
||||
"counts": [dict(row) for row in counts],
|
||||
"participant_counts": [dict(row) for row in participant_counts],
|
||||
"participants": [PARTICIPANTS[code] for code in codes],
|
||||
"total_count": total,
|
||||
"returned_count": len(records),
|
||||
"truncated": total > limit,
|
||||
"reference_year": year,
|
||||
"scope": "public_source_records",
|
||||
"history_basis": (
|
||||
"snapshots_collected_by_year_end" if year else "latest_successful_snapshot"
|
||||
),
|
||||
"coverage": "available" if total else "no_verified_snapshot",
|
||||
"cross_source_deduplicated": False,
|
||||
}
|
||||
)
|
||||
return payload
|
||||
@@ -52,7 +52,7 @@ PRODUCT_DEFINITIONS: dict[str, dict] = {
|
||||
"name": "算力",
|
||||
"sources": ["top500", "epoch_ai_gpu"],
|
||||
"primary_stat_key": "compute_center_count",
|
||||
"stat_keys": ["compute_center_count", "supercomputer_count", "gpu_cluster_count"],
|
||||
"stat_keys": ["compute_center_count", "supercomputer_count", "gpu_cluster_count", "data_center_count", "cloud_center_count", "edge_node_count"],
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@@ -3,9 +3,9 @@ from datetime import UTC, datetime
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Request, status
|
||||
import httpx
|
||||
from pydantic import BaseModel, EmailStr, Field
|
||||
from dotenv import dotenv_values
|
||||
from sqlalchemy import select
|
||||
@@ -65,6 +65,7 @@ from app.services.llm_provider_catalog import (
|
||||
list_fallback_llm_provider_presets,
|
||||
refresh_llm_provider_preset,
|
||||
)
|
||||
from app.services.llm_model_catalog import catalog_error_message, fetch_model_catalog
|
||||
from app.services.scheduler import sync_datasource_job
|
||||
from app.services.tv_streams import DEFAULT_TV_SETTINGS, get_tv_settings_payload, normalize_tv_settings
|
||||
from app.services.persistent_logs import record_audit_log
|
||||
@@ -75,6 +76,7 @@ logger = get_logger(__name__, service="api")
|
||||
AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS = 5
|
||||
AI_CONNECTION_TEST_PROMPT_KEY = "ai.connection_test"
|
||||
SECRET_REVEAL_ROLES = {UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value}
|
||||
LLM_PROVIDER_PRESET_CATEGORY_PREFIX = "llm_provider_preset:"
|
||||
|
||||
DEFAULT_SETTINGS = {
|
||||
"system": {
|
||||
@@ -345,7 +347,13 @@ class ExternalIntegrationsUpdate(BaseModel):
|
||||
|
||||
|
||||
def merge_with_defaults(category: str, payload: Optional[dict]) -> dict:
|
||||
merged = deepcopy(DEFAULT_SETTINGS[category])
|
||||
if category.startswith(LLM_PROVIDER_PRESET_CATEGORY_PREFIX):
|
||||
defaults = get_fallback_llm_provider_preset(
|
||||
category.removeprefix(LLM_PROVIDER_PRESET_CATEGORY_PREFIX)
|
||||
)
|
||||
else:
|
||||
defaults = DEFAULT_SETTINGS[category]
|
||||
merged = deepcopy(defaults)
|
||||
if payload:
|
||||
merged.update(payload)
|
||||
return merged
|
||||
@@ -382,6 +390,7 @@ async def get_setting_payload(db: AsyncSession, category: str) -> dict:
|
||||
|
||||
|
||||
async def save_setting_payload(db: AsyncSession, category: str, payload: dict) -> dict:
|
||||
merged = merge_with_defaults(category, payload)
|
||||
record = await get_setting_record(db, category)
|
||||
if record is None:
|
||||
record = SystemSetting(category=category, payload=payload)
|
||||
@@ -391,7 +400,7 @@ async def save_setting_payload(db: AsyncSession, category: str, payload: dict) -
|
||||
|
||||
await db.commit()
|
||||
await db.refresh(record)
|
||||
return merge_with_defaults(category, record.payload)
|
||||
return merged
|
||||
|
||||
|
||||
AI_PROVIDER_ENV_FILE = Path(__file__).resolve().parents[4] / "aiprovider" / ".env"
|
||||
@@ -655,6 +664,12 @@ def _runtime_config_from_ai_payload(ai_payload: dict) -> dict:
|
||||
normalized_ai["providers"].get(default_provider) or _provider_defaults(default_provider)
|
||||
)
|
||||
api_key, _api_key_source = _resolve_provider_api_key(default_provider, provider_config)
|
||||
model_provider_apis = {
|
||||
**(provider_config.get("model_provider_apis") or {}),
|
||||
**(_get_provider_preset(default_provider).get("model_provider_apis") or {}),
|
||||
}
|
||||
if default_provider == "openai" and urlsplit(provider_config.get("base_url") or "").hostname != "api.openai.com":
|
||||
model_provider_apis = {}
|
||||
return {
|
||||
"service_url": normalized_ai.get("service_url") or app_settings.AI_PROVIDER_SERVICE_URL,
|
||||
"service_token": _resolve_service_token(normalized_ai)[0],
|
||||
@@ -672,8 +687,7 @@ def _runtime_config_from_ai_payload(ai_payload: dict) -> dict:
|
||||
"api_key": api_key,
|
||||
"max_tokens": int(provider_config.get("max_tokens") or 1200),
|
||||
"anthropic_version": provider_config.get("anthropic_version") or "2023-06-01",
|
||||
"model_provider_apis": provider_config.get("model_provider_apis") or {},
|
||||
"preset_models": _get_provider_preset(default_provider).get("models") or [],
|
||||
"model_provider_apis": model_provider_apis,
|
||||
},
|
||||
}
|
||||
|
||||
@@ -753,159 +767,35 @@ async def _validate_ai_provider_full_connection(ai_payload: dict) -> dict:
|
||||
}
|
||||
|
||||
|
||||
def _join_provider_url(base_url: str, path: str) -> str:
|
||||
return f"{base_url.rstrip('/')}/{path.lstrip('/')}"
|
||||
|
||||
|
||||
def _extract_model_ids(payload: dict) -> list[str]:
|
||||
data = payload.get("data") if isinstance(payload, dict) else None
|
||||
if isinstance(data, list):
|
||||
return [
|
||||
str(item.get("id"))
|
||||
for item in data
|
||||
if isinstance(item, dict) and item.get("id")
|
||||
]
|
||||
models = payload.get("models") if isinstance(payload, dict) else None
|
||||
if isinstance(models, list):
|
||||
return [
|
||||
str(item.get("name") or item.get("model") or item.get("id") or item)
|
||||
for item in models
|
||||
if item
|
||||
]
|
||||
return []
|
||||
|
||||
|
||||
def _contains_model(model_ids: list[str], model: str) -> bool:
|
||||
normalized_model = model.strip().lower()
|
||||
return any(str(item).strip().lower() == normalized_model for item in model_ids)
|
||||
|
||||
|
||||
async def _check_ai_provider_lightweight(llm_config: dict, timeout_seconds: int) -> dict:
|
||||
provider = _normalize_provider_id(llm_config.get("provider") or "")
|
||||
configured_api = (
|
||||
str(llm_config.get("provider_api") or "").strip()
|
||||
or ProviderApi.OPENAI_COMPLETIONS.value
|
||||
)
|
||||
provider_api = str(llm_config.get("provider_api") or ProviderApi.OPENAI_COMPLETIONS.value)
|
||||
model = str(llm_config.get("model") or "").strip()
|
||||
base_url = str(llm_config.get("base_url") or "").strip().rstrip("/")
|
||||
base_url = str(llm_config.get("base_url") or "").strip()
|
||||
api_key = str(llm_config.get("api_key") or "").strip()
|
||||
provider_api = configured_api
|
||||
model_provider_apis = llm_config.get("model_provider_apis")
|
||||
if isinstance(model_provider_apis, dict):
|
||||
provider_api = str(model_provider_apis.get(model) or provider_api)
|
||||
preset_models = [
|
||||
str(item)
|
||||
for item in (llm_config.get("preset_models") or [])
|
||||
if str(item).strip()
|
||||
]
|
||||
|
||||
if not provider or not base_url or not model:
|
||||
return {
|
||||
"success": False,
|
||||
"connected": False,
|
||||
"message": "当前 provider/base_url/model 未完整配置。",
|
||||
"mode": "lightweight_config",
|
||||
}
|
||||
result = {
|
||||
"success": False, "connected": False, "mode": "lightweight_models",
|
||||
"provider": provider, "model": model,
|
||||
}
|
||||
if not base_url or not model:
|
||||
return {**result, "message": "当前 provider/base_url/model 未完整配置。"}
|
||||
if provider_api != ProviderApi.OLLAMA_GENERATE.value and not api_key:
|
||||
return {
|
||||
"success": False,
|
||||
"connected": False,
|
||||
"message": "当前 provider 未配置 API Key。",
|
||||
"mode": "lightweight_config",
|
||||
}
|
||||
|
||||
if provider == "opencode-go":
|
||||
url = _join_provider_url(base_url, "/models")
|
||||
headers = {"Authorization": f"Bearer {api_key}"}
|
||||
elif provider_api == ProviderApi.OLLAMA_GENERATE.value:
|
||||
url = _join_provider_url(base_url, "/api/tags")
|
||||
headers: dict[str, str] = {}
|
||||
elif provider_api == ProviderApi.OPENAI_COMPLETIONS.value:
|
||||
url = _join_provider_url(base_url, "/models")
|
||||
headers = {"Authorization": f"Bearer {api_key}"}
|
||||
elif provider_api == ProviderApi.ANTHROPIC_MESSAGES.value:
|
||||
url = _join_provider_url(base_url, "/models")
|
||||
headers = {
|
||||
"x-api-key": api_key,
|
||||
"anthropic-version": str(llm_config.get("anthropic_version") or "2023-06-01"),
|
||||
}
|
||||
else:
|
||||
return {
|
||||
"success": False,
|
||||
"connected": False,
|
||||
"message": f"当前 provider_api 不支持轻量连通性测试: {provider_api}",
|
||||
"mode": "lightweight_unsupported",
|
||||
}
|
||||
|
||||
return {**result, "message": "当前 provider 未配置 API Key。"}
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=min(timeout_seconds, AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS)) as client:
|
||||
response = await client.get(url, headers=headers)
|
||||
response.raise_for_status()
|
||||
payload = response.json()
|
||||
except httpx.HTTPStatusError as exc:
|
||||
detail = exc.response.text or exc.response.reason_phrase
|
||||
if exc.response.status_code == 404 and _contains_model(preset_models, model):
|
||||
return {
|
||||
"success": True,
|
||||
"connected": True,
|
||||
"message": "轻量连通性测试通过;当前 provider 不提供可用的模型目录,已按内置模型预设确认。",
|
||||
"mode": "lightweight_preset",
|
||||
"provider": provider,
|
||||
"provider_api": provider_api,
|
||||
"model": model,
|
||||
"url": url,
|
||||
}
|
||||
return {
|
||||
"success": False,
|
||||
"connected": False,
|
||||
"message": f"轻量连通性测试失败: HTTP {exc.response.status_code} {detail}",
|
||||
"mode": "lightweight_models",
|
||||
"url": url,
|
||||
}
|
||||
catalog = await fetch_model_catalog(
|
||||
provider, base_url, provider_api, api_key,
|
||||
str(llm_config.get("anthropic_version") or "2023-06-01"), timeout_seconds,
|
||||
)
|
||||
except Exception as exc:
|
||||
return {
|
||||
"success": False,
|
||||
"connected": False,
|
||||
"message": f"轻量连通性测试失败: {exc}",
|
||||
"mode": "lightweight_models",
|
||||
"url": url,
|
||||
}
|
||||
|
||||
model_ids = _extract_model_ids(payload)
|
||||
if model_ids and not _contains_model(model_ids, model):
|
||||
if _contains_model(preset_models, model):
|
||||
return {
|
||||
"success": True,
|
||||
"connected": True,
|
||||
"message": "轻量连通性测试通过;provider 模型目录未返回当前别名,已按内置模型预设确认。",
|
||||
"mode": "lightweight_models_with_preset_alias",
|
||||
"provider": provider,
|
||||
"provider_api": provider_api,
|
||||
"model": model,
|
||||
"models_count": len(model_ids),
|
||||
"url": url,
|
||||
}
|
||||
return {
|
||||
"success": False,
|
||||
"connected": False,
|
||||
"message": f"连接可用,但模型目录中没有当前模型: {model}",
|
||||
"mode": "lightweight_models",
|
||||
"provider": provider,
|
||||
"model": model,
|
||||
"models_count": len(model_ids),
|
||||
"url": url,
|
||||
}
|
||||
|
||||
return {**result, "message": catalog_error_message(exc)}
|
||||
result.update({"url": catalog.url, "models_count": len(catalog.models)})
|
||||
if model.casefold() not in {item.casefold() for item in catalog.models}:
|
||||
return {**result, "message": f"模型目录查询成功,但当前模型不在目录中:{model}"}
|
||||
model_apis = llm_config.get("model_provider_apis") or {}
|
||||
return {
|
||||
"success": True,
|
||||
"connected": True,
|
||||
"message": "轻量连通性测试通过",
|
||||
"mode": "lightweight_models",
|
||||
"provider": provider,
|
||||
"provider_api": provider_api,
|
||||
"model": model,
|
||||
"models_count": len(model_ids),
|
||||
"url": url,
|
||||
**result, "success": True, "connected": True,
|
||||
"provider_api": model_apis.get(model) or provider_api,
|
||||
"message": "模型目录查询成功,当前模型已找到;尚未执行生成调用。",
|
||||
}
|
||||
|
||||
|
||||
@@ -1689,7 +1579,8 @@ async def connect_ai_provider_integration(
|
||||
)
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="settings.ai_provider.connect.success",
|
||||
event=("settings.ai_provider.connect.success" if lightweight_result["success"]
|
||||
else "settings.ai_provider.connect.failed"),
|
||||
message="AI provider connection test completed",
|
||||
category="ai",
|
||||
service="api",
|
||||
@@ -1699,7 +1590,7 @@ async def connect_ai_provider_integration(
|
||||
"provider": payload.provider,
|
||||
"model": payload.model,
|
||||
"configured": True,
|
||||
"lightweight_status": lightweight_result.get("status"),
|
||||
"connected": lightweight_result["connected"],
|
||||
},
|
||||
)
|
||||
return {
|
||||
@@ -2044,8 +1935,18 @@ async def reset_provider_credential_guide(
|
||||
@router.get("/integrations/ai-provider/presets")
|
||||
async def get_ai_provider_presets(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return {"data": list_fallback_llm_provider_presets()}
|
||||
presets = list_fallback_llm_provider_presets()
|
||||
saved = await get_setting_payloads(
|
||||
db, [f"{LLM_PROVIDER_PRESET_CATEGORY_PREFIX}{preset['provider']}" for preset in presets]
|
||||
)
|
||||
return {
|
||||
"data": [
|
||||
{**preset, **saved[f"{LLM_PROVIDER_PRESET_CATEGORY_PREFIX}{preset['provider']}"]}
|
||||
for preset in presets
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
@router.post("/integrations/ai-provider/presets/{provider}/refresh")
|
||||
@@ -2053,22 +1954,52 @@ async def refresh_ai_provider_preset(
|
||||
provider: str,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
payload: AIProviderIntegrationUpdate | None = None,
|
||||
):
|
||||
try:
|
||||
provider_id = _normalize_provider_id(provider)
|
||||
api_key = None
|
||||
if provider_id == "opencode-go":
|
||||
current_payload = await get_setting_payload(db, "external_integrations")
|
||||
ai_payload = _normalize_ai_provider_payload(current_payload.get("ai_provider") or {})
|
||||
provider_config = ai_payload["providers"].get(provider_id) or _provider_defaults(provider_id)
|
||||
api_key, _api_key_source = _resolve_provider_api_key(provider_id, provider_config)
|
||||
return {"data": await refresh_llm_provider_preset(provider_id, api_key=api_key)}
|
||||
get_fallback_llm_provider_preset(provider_id)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=404, detail=str(exc)) from exc
|
||||
|
||||
current_payload = await get_setting_payload(db, "external_integrations")
|
||||
if payload is not None:
|
||||
if _normalize_provider_id(payload.provider) != provider_id:
|
||||
raise HTTPException(status_code=400, detail="刷新供应商与表单供应商不一致。")
|
||||
ai_payload = _build_ai_provider_payload(current_payload, payload)
|
||||
else:
|
||||
ai_payload = _normalize_ai_provider_payload(current_payload.get("ai_provider") or {})
|
||||
provider_config = ai_payload["providers"].get(provider_id) or _provider_defaults(provider_id)
|
||||
api_key, _api_key_source = _resolve_provider_api_key(provider_id, provider_config)
|
||||
if not api_key and provider_id not in {"ollama", "opencode-go", "openrouter"}:
|
||||
raise HTTPException(
|
||||
status_code=400, detail="请先配置当前供应商的 API Key,再刷新模型列表。",
|
||||
)
|
||||
try:
|
||||
refreshed = await refresh_llm_provider_preset(
|
||||
provider_id, api_key=api_key, base_url=provider_config.get("base_url"),
|
||||
provider_api=provider_config.get("provider_api"),
|
||||
anthropic_version=provider_config.get("anthropic_version") or "2023-06-01",
|
||||
)
|
||||
except Exception as exc:
|
||||
fallback = get_fallback_llm_provider_preset(provider)
|
||||
fallback["refresh_error"] = str(exc)
|
||||
return {"data": fallback}
|
||||
logger.warning(
|
||||
"LLM provider catalog refresh failed",
|
||||
extra={
|
||||
"event": "settings.ai_provider.catalog.failed",
|
||||
"context": {
|
||||
"provider": provider_id,
|
||||
"error_type": type(exc).__name__,
|
||||
},
|
||||
},
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=502,
|
||||
detail=f"{catalog_error_message(exc)} 已保留上次模型列表。",
|
||||
) from exc
|
||||
|
||||
refreshed["refreshed_at"] = to_iso8601_utc(datetime.now(UTC))
|
||||
await save_setting_payload(db, f"{LLM_PROVIDER_PRESET_CATEGORY_PREFIX}{provider_id}", refreshed)
|
||||
return {"data": refreshed}
|
||||
|
||||
|
||||
@router.put("/integrations")
|
||||
|
||||
@@ -7,6 +7,7 @@ from fastapi.responses import Response
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.db.session import get_db
|
||||
from app.services.tv_catalog import get_tv_catalog_page
|
||||
from app.services.tv_streams import get_public_tv_payload, is_allowed_tv_proxy_url
|
||||
|
||||
router = APIRouter()
|
||||
@@ -34,9 +35,13 @@ def _should_strip_hls_metadata_line(line: str) -> bool:
|
||||
|
||||
@router.get("/streams")
|
||||
async def list_public_tv_streams(
|
||||
offset: int = Query(0, ge=0),
|
||||
limit: int = Query(50, ge=1, le=100),
|
||||
q: str = Query("", max_length=200),
|
||||
selected_id: str | None = Query(None, max_length=200),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await get_public_tv_payload(db)
|
||||
return await get_tv_catalog_page(db, offset=offset, limit=limit, q=q, selected_id=selected_id)
|
||||
|
||||
|
||||
@router.get("/proxy")
|
||||
|
||||
@@ -17,11 +17,13 @@ from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from sqlalchemy import select, func
|
||||
from typing import List, Dict, Any, Optional
|
||||
|
||||
from app.core.compute_types import COMPUTE_SITE_TYPES, COMPUTE_SOURCES, compute_site_type, compute_record_filter, compute_site_type_expression
|
||||
from app.core.collected_data_fields import get_record_field
|
||||
from app.core.enums import BGPStatus
|
||||
from app.core.satellite_tle import build_tle_lines_from_elements
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.db.session import get_db
|
||||
from app.core.satellite_presentation import satellite_constellation_group as _normalize_satellite_constellation_group
|
||||
from app.models.bgp_anomaly import BGPAnomaly
|
||||
from app.models.bgp_incident import BGPIncident
|
||||
from app.models.bgp_observation import BGPObservation
|
||||
@@ -337,22 +339,6 @@ def convert_satellite_to_geojson(records: List[CollectedData]) -> Dict[str, Any]
|
||||
return {"type": "FeatureCollection", "features": features}
|
||||
|
||||
|
||||
def _normalize_satellite_constellation_group(
|
||||
raw_group: Any,
|
||||
name: Optional[str],
|
||||
) -> Optional[str]:
|
||||
normalized_group = str(raw_group or "").strip().lower()
|
||||
if normalized_group:
|
||||
return normalized_group
|
||||
|
||||
normalized_name = str(name or "").strip().upper()
|
||||
if normalized_name.startswith("STARLINK"):
|
||||
return "starlink"
|
||||
if normalized_name.startswith("IRIDIUM"):
|
||||
return "iridium-next"
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def _get_satellite_footprint_policy(constellation_group: Optional[str]) -> str:
|
||||
if constellation_group == "starlink":
|
||||
@@ -505,16 +491,19 @@ async def _count_current_or_latest_task_data(
|
||||
async def _load_current_collected_data_by_sources(
|
||||
db: AsyncSession,
|
||||
sources: List[str],
|
||||
*, include_compute_types: bool = False,
|
||||
) -> Dict[str, List[CollectedData]]:
|
||||
if not sources:
|
||||
return {}
|
||||
|
||||
stmt = (
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source.in_(sources))
|
||||
.where(compute_record_filter(CollectedData) if include_compute_types else CollectedData.source.in_(sources))
|
||||
.where(CollectedData.is_current.is_(True))
|
||||
.order_by(CollectedData.source.asc(), CollectedData.id.desc())
|
||||
)
|
||||
if include_compute_types:
|
||||
stmt = stmt.where(CollectedData.deleted_at.is_(None))
|
||||
result = await db.execute(stmt)
|
||||
|
||||
grouped_records: Dict[str, List[CollectedData]] = {source: [] for source in sources}
|
||||
@@ -644,7 +633,9 @@ def _normalize_capacity_band(capacity_value: Optional[float], capacity_unit: str
|
||||
return "unknown"
|
||||
|
||||
unit = str(capacity_unit or "").strip().lower()
|
||||
if unit in {"pflop/s", "pflops", "pflop"}:
|
||||
if unit in {"flop/s", "flops"}:
|
||||
normalized_tflops = capacity_value / 1e12
|
||||
elif unit in {"pflop/s", "pflops", "pflop"}:
|
||||
normalized_tflops = capacity_value * 1000
|
||||
elif unit in {"gflop/s", "gflops", "gflop"}:
|
||||
normalized_tflops = capacity_value
|
||||
@@ -672,15 +663,15 @@ def convert_compute_centers_to_geojson(records: List[CollectedData]) -> Dict[str
|
||||
"""
|
||||
features: List[Dict[str, Any]] = []
|
||||
unresolved: List[Dict[str, Any]] = []
|
||||
withheld_count = 0
|
||||
|
||||
for record in records:
|
||||
metadata = record.extra_data or {}
|
||||
if metadata.get("compute_contract") == "v1" and metadata.get("anonymous"):
|
||||
withheld_count += 1
|
||||
continue
|
||||
result = resolve_compute_center_location_full(record, metadata)
|
||||
site_type = (
|
||||
"supercomputer"
|
||||
if record.source == "top500" or record.data_type == "supercomputer"
|
||||
else "gpu_cluster"
|
||||
)
|
||||
site_type = compute_site_type(record)
|
||||
|
||||
if not result.is_resolved:
|
||||
diagnostic = result.diagnostic or ResolutionDiagnostic(
|
||||
@@ -706,7 +697,10 @@ def convert_compute_centers_to_geojson(records: List[CollectedData]) -> Dict[str
|
||||
latitude = location.latitude
|
||||
longitude = location.longitude
|
||||
|
||||
if site_type == "supercomputer":
|
||||
if metadata.get("compute_contract") == "v1":
|
||||
capacity_value = _parse_float(get_record_field(record, "value"))
|
||||
capacity_unit = str(get_record_field(record, "unit") or "FLOP/s")
|
||||
elif site_type == "supercomputer":
|
||||
capacity_value = _parse_float(get_record_field(record, "rmax"))
|
||||
capacity_unit = "GFlops"
|
||||
else:
|
||||
@@ -767,7 +761,7 @@ def convert_compute_centers_to_geojson(records: List[CollectedData]) -> Dict[str
|
||||
}
|
||||
)
|
||||
|
||||
return {"type": "FeatureCollection", "features": features, "unresolved": unresolved}
|
||||
return {"type": "FeatureCollection", "features": features, "unresolved": unresolved, "withheld_count": withheld_count}
|
||||
|
||||
|
||||
VESSEL_TYPE_FILTERS = {
|
||||
@@ -1121,6 +1115,7 @@ async def build_vessel_snapshot_response(
|
||||
safe_limit = _safe_vessel_limit(limit)
|
||||
safe_since_minutes = min(max(int(since_minutes or 60), 1), 1440)
|
||||
observed_since = datetime.now(UTC) - timedelta(minutes=safe_since_minutes)
|
||||
snapshot_started_at = datetime.now(UTC)
|
||||
features, diagnostics = await _load_raw_vessel_snapshot_features(
|
||||
db,
|
||||
bbox=bbox,
|
||||
@@ -1137,6 +1132,7 @@ async def build_vessel_snapshot_response(
|
||||
"features": features,
|
||||
"count": len(features),
|
||||
"stats": _build_vessel_stats(features),
|
||||
"generated_at": to_iso8601_utc(snapshot_started_at),
|
||||
"diagnostics": {
|
||||
**diagnostics,
|
||||
"filtered_count": len(features),
|
||||
@@ -1953,9 +1949,10 @@ async def _build_compute_centers_geojson(
|
||||
records_by_source = await _load_current_collected_data_by_sources(
|
||||
db,
|
||||
["top500", "epoch_ai_gpu"],
|
||||
include_compute_types=True,
|
||||
)
|
||||
records = _filter_known_records(
|
||||
records_by_source.get("top500", []) + records_by_source.get("epoch_ai_gpu", []),
|
||||
[record for group in records_by_source.values() for record in group],
|
||||
)
|
||||
if limit is not None:
|
||||
records = records[:limit]
|
||||
@@ -1970,6 +1967,7 @@ async def _build_compute_centers_geojson(
|
||||
"total": 0,
|
||||
"supercomputers": 0,
|
||||
"gpu_clusters": 0,
|
||||
"by_type": dict.fromkeys(COMPUTE_SITE_TYPES, 0),
|
||||
"unresolved": 0,
|
||||
},
|
||||
}
|
||||
@@ -2012,6 +2010,7 @@ async def _build_compute_centers_geojson(
|
||||
"count": len(sanitized_features),
|
||||
"stats": {
|
||||
"total": len(sanitized_features),
|
||||
"by_type": {kind: sum(feature["properties"]["site_type"] == kind for feature in sanitized_features) for kind in COMPUTE_SITE_TYPES},
|
||||
"supercomputers": sum(
|
||||
1 for feature in sanitized_features
|
||||
if feature.get("properties", {}).get("site_type") == "supercomputer"
|
||||
@@ -2328,7 +2327,8 @@ async def _load_compute_center_record(db: AsyncSession, source_id: str) -> Colle
|
||||
stmt = (
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source_id == source_id)
|
||||
.where(CollectedData.source.in_(["top500", "epoch_ai_gpu"]))
|
||||
.where(compute_record_filter(CollectedData))
|
||||
.where(CollectedData.deleted_at.is_(None))
|
||||
.order_by(CollectedData.is_current.desc(), CollectedData.id.desc())
|
||||
.limit(1)
|
||||
)
|
||||
@@ -2727,6 +2727,25 @@ async def get_visualization_geo_summary(db: AsyncSession = Depends(get_db), resp
|
||||
)
|
||||
|
||||
|
||||
async def _compute_facility_counts(db: AsyncSession) -> dict[str, int]:
|
||||
kind = compute_site_type_expression(CollectedData)
|
||||
rows = (await db.execute(select(CollectedData.source, kind, func.count(CollectedData.id))
|
||||
.where(compute_record_filter(CollectedData), CollectedData.is_current.is_(True),
|
||||
CollectedData.deleted_at.is_(None))
|
||||
.group_by(CollectedData.source, kind))).all()
|
||||
counts = dict.fromkeys(COMPUTE_SITE_TYPES, 0)
|
||||
present_sources = set()
|
||||
for source, site_type, count in rows:
|
||||
present_sources.add(source)
|
||||
counts[site_type] += int(count or 0)
|
||||
# Retain the existing last-task fallback for the two collected catalogs.
|
||||
for source in COMPUTE_SOURCES:
|
||||
if source not in present_sources:
|
||||
key = "supercomputer" if source == "top500" else "gpu_cluster"
|
||||
counts[key] += await _count_current_or_latest_task_data(db, source)
|
||||
return counts
|
||||
|
||||
|
||||
async def _build_visualization_geo_summary(db: AsyncSession) -> dict[str, Any]:
|
||||
cable_count = await _count_current_or_latest_task_data(db, "arcgis_cables")
|
||||
landing_point_count = await _count_current_or_latest_task_data(db, "arcgis_landing_points")
|
||||
@@ -2735,9 +2754,10 @@ async def _build_visualization_geo_summary(db: AsyncSession) -> dict[str, Any]:
|
||||
"celestrak_tle",
|
||||
exclude_unknown_name=True,
|
||||
)
|
||||
supercomputer_count = await _count_current_or_latest_task_data(db, "top500")
|
||||
gpu_cluster_count = await _count_current_or_latest_task_data(db, "epoch_ai_gpu")
|
||||
compute_center_count = supercomputer_count + gpu_cluster_count
|
||||
compute_counts = await _compute_facility_counts(db)
|
||||
supercomputer_count = compute_counts["supercomputer"]
|
||||
gpu_cluster_count = compute_counts["gpu_cluster"]
|
||||
compute_center_count = sum(compute_counts.values())
|
||||
|
||||
active_incident_result = await db.execute(
|
||||
select(func.count(BGPIncident.id)).where(BGPIncident.status == BGPStatus.ACTIVE.value),
|
||||
@@ -2792,6 +2812,9 @@ async def _build_visualization_geo_summary(db: AsyncSession) -> dict[str, Any]:
|
||||
"aisstream_lag_seconds": aisstream_health.lag_seconds if aisstream_health else None,
|
||||
"supercomputer_count": supercomputer_count,
|
||||
"gpu_cluster_count": gpu_cluster_count,
|
||||
"data_center_count": compute_counts["data_center"],
|
||||
"cloud_center_count": compute_counts["cloud_center"],
|
||||
"edge_node_count": compute_counts["edge_node"],
|
||||
"bgp_event_count": active_incident_count or active_anomaly_count,
|
||||
"bgp_incident_count": active_incident_count,
|
||||
"bgp_anomaly_count": active_anomaly_count,
|
||||
|
||||
@@ -161,7 +161,7 @@ async def websocket_endpoint(
|
||||
if is_anonymous:
|
||||
channels = [channel for channel in channels if channel in supported_channels]
|
||||
vessel_subscription = None
|
||||
if "vessels" in channels and "bbox" in payload_data:
|
||||
if "vessels" in channels:
|
||||
try:
|
||||
vessel_subscription = manager.subscribe_vessels(websocket, payload_data)
|
||||
except ValueError as exc:
|
||||
|
||||
35
backend/app/core/compute_types.py
Normal file
35
backend/app/core/compute_types.py
Normal file
@@ -0,0 +1,35 @@
|
||||
"""Compute facility classification shared by the map and image resolver."""
|
||||
|
||||
from sqlalchemy import case, or_
|
||||
|
||||
COMPUTE_SITE_TYPES = ("supercomputer", "gpu_cluster", "data_center", "cloud_center", "edge_node")
|
||||
COMPUTE_SOURCES = ("top500", "epoch_ai_gpu")
|
||||
|
||||
|
||||
def compute_site_type(record) -> str:
|
||||
metadata = record.extra_data or {}
|
||||
for value in (metadata.get("site_type"), record.data_type):
|
||||
if value in COMPUTE_SITE_TYPES:
|
||||
return value
|
||||
return {"top500": "supercomputer", "epoch_ai_gpu": "gpu_cluster"}.get(
|
||||
record.source, "data_center"
|
||||
)
|
||||
|
||||
|
||||
def compute_site_type_expression(record):
|
||||
declared = record.extra_data["site_type"].as_string()
|
||||
return case(
|
||||
(declared.in_(COMPUTE_SITE_TYPES), declared),
|
||||
(record.data_type.in_(COMPUTE_SITE_TYPES), record.data_type),
|
||||
(record.source == "top500", "supercomputer"),
|
||||
(record.source == "epoch_ai_gpu", "gpu_cluster"),
|
||||
else_="data_center",
|
||||
)
|
||||
|
||||
|
||||
def compute_record_filter(record):
|
||||
return or_(
|
||||
record.source.in_(COMPUTE_SOURCES),
|
||||
record.data_type.in_(COMPUTE_SITE_TYPES),
|
||||
record.extra_data["site_type"].as_string().in_(COMPUTE_SITE_TYPES),
|
||||
)
|
||||
57
backend/app/core/earth_geography_policy.py
Normal file
57
backend/app/core/earth_geography_policy.py
Normal file
@@ -0,0 +1,57 @@
|
||||
"""Country display ownership for Earth comparisons and cards.
|
||||
|
||||
Keep source-country evidence separate; use this projection only for display
|
||||
grouping so regional source records cannot become competing national keys.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
|
||||
from app.core.countries import COUNTRY_ENTRIES, normalize_country
|
||||
|
||||
|
||||
_DISPLAY_ISO_A3 = {
|
||||
canonical: next(
|
||||
(alias for alias in aliases if len(alias) == 3 and alias.isascii() and alias.isupper()),
|
||||
None,
|
||||
)
|
||||
for canonical, aliases in COUNTRY_ENTRIES
|
||||
}
|
||||
_COUNTRY_BY_ISO_A3 = {code: country for country, code in _DISPLAY_ISO_A3.items() if code}
|
||||
_POLICY_PATH = Path(__file__).resolve().parents[3] / "config/earth-boundary-pov-policy.china-v1.json"
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def _policy_country_by_code() -> dict[str, str]:
|
||||
try:
|
||||
content = _POLICY_PATH.read_text(encoding="utf-8")
|
||||
except FileNotFoundError as error:
|
||||
raise FileNotFoundError(f"EARTH_DISPLAY_POLICY_MISSING: {_POLICY_PATH}") from error
|
||||
policy = json.loads(content)
|
||||
if policy.get("profile") != "china-pov-v1":
|
||||
raise ValueError("unsupported Earth display-policy profile")
|
||||
mappings = {}
|
||||
for rule in policy["rules"]:
|
||||
target = rule.get("targetIsoA3") or rule.get("parentIsoA3")
|
||||
if target not in _COUNTRY_BY_ISO_A3:
|
||||
continue
|
||||
for code in rule.get("sourceIsoA3", []):
|
||||
mappings[code] = _COUNTRY_BY_ISO_A3[target]
|
||||
return mappings
|
||||
|
||||
|
||||
def display_country(value: str | None) -> str | None:
|
||||
country = normalize_country(value)
|
||||
code = _DISPLAY_ISO_A3.get(country) if country else str(value or "").strip().upper()
|
||||
owner = _policy_country_by_code().get(code)
|
||||
if owner:
|
||||
return owner
|
||||
return country
|
||||
|
||||
|
||||
def display_iso_a3(value: str | None) -> str | None:
|
||||
country = display_country(value)
|
||||
return _DISPLAY_ISO_A3.get(country) if country else None
|
||||
@@ -138,6 +138,7 @@ class DatasourceRunStatus(StrEnum):
|
||||
class ProviderApi(StrEnum):
|
||||
ANTHROPIC_MESSAGES = "anthropic-messages"
|
||||
OPENAI_COMPLETIONS = "openai-completions"
|
||||
OPENAI_RESPONSES = "openai-responses"
|
||||
OLLAMA_GENERATE = "ollama-generate"
|
||||
|
||||
|
||||
|
||||
13
backend/app/core/satellite_presentation.py
Normal file
13
backend/app/core/satellite_presentation.py
Normal file
@@ -0,0 +1,13 @@
|
||||
"""Shared source-group normalization; never infer hardware generations."""
|
||||
|
||||
|
||||
def satellite_constellation_group(raw_group: object, name: str | None) -> str | None:
|
||||
group = str(raw_group or "").strip().lower()
|
||||
if group:
|
||||
return group
|
||||
normalized_name = str(name or "").strip().upper()
|
||||
if normalized_name.startswith("STARLINK"):
|
||||
return "starlink"
|
||||
if normalized_name.startswith("IRIDIUM"):
|
||||
return "iridium-next"
|
||||
return None
|
||||
@@ -6,7 +6,7 @@ import redis
|
||||
from fastapi import Depends, HTTPException, status
|
||||
from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
|
||||
from jose import JWTError, jwt
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.config import settings
|
||||
@@ -81,6 +81,21 @@ def decode_token(token: str) -> Optional[dict]:
|
||||
return None
|
||||
|
||||
|
||||
async def _load_active_user(user_id: str, db: AsyncSession) -> User:
|
||||
try:
|
||||
parsed_id = int(user_id)
|
||||
except (TypeError, ValueError):
|
||||
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Invalid token")
|
||||
result = await db.execute(select(User).where(User.id == parsed_id))
|
||||
user = result.scalar_one_or_none()
|
||||
if user is None or not user.is_active:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="User not found or inactive",
|
||||
)
|
||||
return user
|
||||
|
||||
|
||||
async def get_current_user(
|
||||
credentials: HTTPAuthorizationCredentials = Depends(oauth2_scheme),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
@@ -103,27 +118,7 @@ async def get_current_user(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="Invalid token",
|
||||
)
|
||||
result = await db.execute(
|
||||
text(
|
||||
"SELECT id, username, email, password_hash, role, is_active, gatekeeper_groups FROM users WHERE id = :id"
|
||||
),
|
||||
{"id": int(user_id)},
|
||||
)
|
||||
row = result.fetchone()
|
||||
if row is None or not row[5]:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="User not found or inactive",
|
||||
)
|
||||
user = User()
|
||||
user.id = row[0]
|
||||
user.username = row[1]
|
||||
user.email = row[2]
|
||||
user.password_hash = row[3]
|
||||
user.role = row[4]
|
||||
user.is_active = row[5]
|
||||
user.gatekeeper_groups = row[6] or []
|
||||
return user
|
||||
return await _load_active_user(user_id, db)
|
||||
|
||||
|
||||
async def get_current_user_refresh(
|
||||
@@ -143,27 +138,7 @@ async def get_current_user_refresh(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="Invalid token",
|
||||
)
|
||||
result = await db.execute(
|
||||
text(
|
||||
"SELECT id, username, email, password_hash, role, is_active, gatekeeper_groups FROM users WHERE id = :id"
|
||||
),
|
||||
{"id": int(user_id)},
|
||||
)
|
||||
row = result.fetchone()
|
||||
if row is None or not row[5]:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="User not found or inactive",
|
||||
)
|
||||
user = User()
|
||||
user.id = row[0]
|
||||
user.username = row[1]
|
||||
user.email = row[2]
|
||||
user.password_hash = row[3]
|
||||
user.role = row[4]
|
||||
user.is_active = row[5]
|
||||
user.gatekeeper_groups = row[6] or []
|
||||
return user
|
||||
return await _load_active_user(user_id, db)
|
||||
|
||||
|
||||
def blacklist_token(token: str) -> None:
|
||||
|
||||
@@ -4,11 +4,14 @@ import asyncio
|
||||
from datetime import UTC, datetime
|
||||
from typing import Dict, Any
|
||||
|
||||
from app.core.logging import get_logger
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.core.websocket.manager import manager
|
||||
|
||||
|
||||
EARTH_UPDATES_CHANNEL = "earth_updates"
|
||||
VESSEL_STATE_QUERY_BATCH_SIZE = 1000
|
||||
logger = get_logger(__name__, service="websocket")
|
||||
|
||||
|
||||
class DataBroadcaster:
|
||||
@@ -114,16 +117,22 @@ class DataBroadcaster:
|
||||
return
|
||||
pending = self._pending_vessel_updates
|
||||
self._pending_vessel_updates = {}
|
||||
vessels = []
|
||||
for item in pending.values():
|
||||
vessel = dict(item)
|
||||
source = vessel.pop("_source", None)
|
||||
action = vessel.pop("_action", "upsert")
|
||||
created = vessel.pop("_created", None)
|
||||
vessel["source"] = source
|
||||
vessel["action"] = action
|
||||
vessel["created"] = created
|
||||
vessels.append(vessel)
|
||||
try:
|
||||
vessels = await self._load_current_vessel_updates(list(pending))
|
||||
except Exception:
|
||||
# Preserve newer updates that arrived during the failed database read.
|
||||
self._pending_vessel_updates = {**pending, **self._pending_vessel_updates}
|
||||
raise
|
||||
if not vessels:
|
||||
return
|
||||
vessels = [
|
||||
{
|
||||
**vessel,
|
||||
"action": vessel.get("action", "upsert"),
|
||||
"created": pending.get(str(vessel["mmsi"]), {}).get("_created"),
|
||||
}
|
||||
for vessel in vessels
|
||||
]
|
||||
await manager.broadcast_vessels(
|
||||
{
|
||||
"action": "upsert",
|
||||
@@ -133,12 +142,31 @@ class DataBroadcaster:
|
||||
}
|
||||
)
|
||||
|
||||
async def _load_current_vessel_updates(self, keys: list[str]) -> list[Dict[str, Any]]:
|
||||
from app.db.session import async_session_factory
|
||||
from app.services.vessel_ais_aggregation import get_current_vessels_by_mmsi
|
||||
|
||||
mmsis = [int(key) for key in keys if key.isdigit()]
|
||||
vessels = []
|
||||
async with async_session_factory() as db:
|
||||
for offset in range(0, len(mmsis), VESSEL_STATE_QUERY_BATCH_SIZE):
|
||||
vessels.extend(await get_current_vessels_by_mmsi(
|
||||
db, mmsis[offset:offset + VESSEL_STATE_QUERY_BATCH_SIZE]
|
||||
))
|
||||
present = {int(vessel["mmsi"]) for vessel in vessels}
|
||||
vessels.extend({"mmsi": mmsi, "action": "remove"} for mmsi in mmsis if mmsi not in present)
|
||||
return vessels
|
||||
|
||||
async def broadcast_vessels_periodically(self):
|
||||
while self.running:
|
||||
try:
|
||||
await self.flush_vessel_updates()
|
||||
except Exception:
|
||||
pass
|
||||
except Exception as exc:
|
||||
logger.exception_event(
|
||||
"Failed to flush vessel updates",
|
||||
event="vessels.broadcast.failed",
|
||||
context={"error": str(exc)},
|
||||
)
|
||||
await asyncio.sleep(self._vessel_flush_interval)
|
||||
|
||||
async def broadcast_datasource_task_update(self, data: Dict[str, Any]):
|
||||
|
||||
@@ -63,6 +63,8 @@ class ConnectionManager:
|
||||
|
||||
def unsubscribe(self, websocket: WebSocket, channels: list[str]):
|
||||
for channel in {str(channel).strip() for channel in channels if str(channel).strip()}:
|
||||
if channel == "vessels":
|
||||
self.vessel_subscriptions.pop(websocket, None)
|
||||
subscribers = self.channel_subscriptions.get(channel)
|
||||
if subscribers is not None:
|
||||
subscribers.discard(websocket)
|
||||
@@ -88,7 +90,8 @@ class ConnectionManager:
|
||||
return subscription
|
||||
|
||||
def _normalize_vessel_subscription(self, config: dict[str, Any]) -> dict[str, Any]:
|
||||
bbox = config.get("bbox")
|
||||
global_scope = config.get("scope") == "global"
|
||||
bbox = [-180, -90, 180, 90] if global_scope else config.get("bbox")
|
||||
if not isinstance(bbox, (list, tuple)) or len(bbox) != 4:
|
||||
raise ValueError("vessels subscription requires bbox=[lon_min,lat_min,lon_max,lat_max]")
|
||||
try:
|
||||
@@ -103,7 +106,7 @@ class ConnectionManager:
|
||||
raise ValueError("bbox longitude values must be between -180 and 180")
|
||||
if not (-90 <= lat_min <= 90 and -90 <= lat_max <= 90):
|
||||
raise ValueError("bbox latitude values must be between -90 and 90")
|
||||
if (lon_max - lon_min) * (lat_max - lat_min) > MAX_VESSEL_BBOX_AREA:
|
||||
if not global_scope and (lon_max - lon_min) * (lat_max - lat_min) > MAX_VESSEL_BBOX_AREA:
|
||||
raise ValueError("bbox is too large; zoom in or request a smaller viewport")
|
||||
|
||||
zoom = int(config.get("zoom") or 1)
|
||||
@@ -116,10 +119,11 @@ class ConnectionManager:
|
||||
if str(item).strip()
|
||||
}
|
||||
return {
|
||||
"scope": "global" if global_scope else "viewport",
|
||||
"bbox": (lon_min, lat_min, lon_max, lat_max),
|
||||
"zoom": zoom,
|
||||
"limit": limit,
|
||||
"type": vessel_types,
|
||||
"type": sorted(vessel_types),
|
||||
"last_sent_at": None,
|
||||
}
|
||||
|
||||
@@ -152,7 +156,9 @@ class ConnectionManager:
|
||||
vessel
|
||||
for vessel in vessels
|
||||
if self._vessel_matches_subscription(vessel, subscription)
|
||||
][: min(subscription["limit"], MAX_VESSEL_WS_MESSAGE_ITEMS)]
|
||||
]
|
||||
if subscription.get("scope") != "global":
|
||||
matched = matched[:subscription["limit"]]
|
||||
if not matched:
|
||||
continue
|
||||
subscription["last_sent_at"] = datetime.now(UTC)
|
||||
@@ -162,16 +168,24 @@ class ConnectionManager:
|
||||
"timestamp": subscription["last_sent_at"].isoformat(),
|
||||
"payload": {
|
||||
**data,
|
||||
"vessels": matched,
|
||||
"vessels": [],
|
||||
"subscription": {
|
||||
"bbox": list(subscription["bbox"]),
|
||||
"zoom": subscription["zoom"],
|
||||
"limit": subscription["limit"],
|
||||
"scope": subscription.get("scope", "viewport"),
|
||||
},
|
||||
},
|
||||
}
|
||||
try:
|
||||
await connection.send_json(message)
|
||||
for offset in range(0, len(matched), MAX_VESSEL_WS_MESSAGE_ITEMS):
|
||||
await connection.send_json({
|
||||
**message,
|
||||
"payload": {
|
||||
**message["payload"],
|
||||
"vessels": matched[offset:offset + MAX_VESSEL_WS_MESSAGE_ITEMS],
|
||||
},
|
||||
})
|
||||
except Exception:
|
||||
self.unsubscribe_all(connection)
|
||||
|
||||
@@ -180,6 +194,8 @@ class ConnectionManager:
|
||||
vessel: dict[str, Any],
|
||||
subscription: dict[str, Any],
|
||||
) -> bool:
|
||||
if vessel.get("action") == "remove" and subscription.get("scope") == "global":
|
||||
return True
|
||||
try:
|
||||
lon = float(vessel.get("lon"))
|
||||
lat = float(vessel.get("lat"))
|
||||
|
||||
128
backend/app/schemas/card_images.py
Normal file
128
backend/app/schemas/card_images.py
Normal file
@@ -0,0 +1,128 @@
|
||||
"""Typed, validated configuration for optional object-card images."""
|
||||
|
||||
from typing import Literal, Self
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
|
||||
|
||||
EntityType = Literal["satellite", "compute_center", "cable", "landing_point", "vessel", "bgp", "bgp_collector", "earth_interactable"]
|
||||
MatchScope = Literal["object", "model", "group", "category"]
|
||||
|
||||
|
||||
def safe_url(value: str, *, local: bool = False) -> str:
|
||||
if not value:
|
||||
return value
|
||||
if any(ord(char) < 32 for char in value) or "\\" in value:
|
||||
raise ValueError("Invalid URL")
|
||||
parts = urlsplit(value)
|
||||
if local and value.startswith(("/earth/assets/cards/", "/api/v1/earth/card-images/files/")):
|
||||
if (
|
||||
".." not in parts.path
|
||||
and "%" not in parts.path
|
||||
and not parts.netloc
|
||||
and not parts.query
|
||||
and not parts.fragment
|
||||
):
|
||||
return value
|
||||
if (
|
||||
parts.scheme in {"http", "https"}
|
||||
and parts.hostname
|
||||
and not parts.username
|
||||
and not parts.password
|
||||
):
|
||||
return value
|
||||
raise ValueError("Use an HTTP(S) URL without credentials or a managed image path")
|
||||
|
||||
|
||||
class CardImageAsset(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid", str_strip_whitespace=True)
|
||||
id: str = Field(min_length=1, max_length=100, pattern=r"^[a-zA-Z0-9_-]+$")
|
||||
title: str = Field(min_length=1, max_length=200)
|
||||
url: str = Field(min_length=1, max_length=2000)
|
||||
variants: dict[str, str] = Field(default_factory=dict, max_length=4)
|
||||
width: int = Field(default=0, ge=0, le=20000)
|
||||
height: int = Field(default=0, ge=0, le=20000)
|
||||
credit: str = Field(default="", max_length=500)
|
||||
source_url: str = Field(default="", max_length=2000)
|
||||
license: str = Field(default="", max_length=300)
|
||||
license_url: str = Field(default="", max_length=2000)
|
||||
kind: Literal["photo", "model_photo", "render", "animation_still"] = "photo"
|
||||
rights_status: Literal["available", "permission_pending"] = "available"
|
||||
enabled: bool = True
|
||||
focus_x: float = Field(default=50, ge=0, le=100, allow_inf_nan=False)
|
||||
focus_y: float = Field(default=50, ge=0, le=100, allow_inf_nan=False)
|
||||
|
||||
@field_validator("url")
|
||||
@classmethod
|
||||
def validate_image_url(cls, value: str) -> str:
|
||||
return safe_url(value, local=True)
|
||||
|
||||
@field_validator("source_url", "license_url")
|
||||
@classmethod
|
||||
def validate_source_url(cls, value: str) -> str:
|
||||
return safe_url(value)
|
||||
|
||||
@field_validator("variants")
|
||||
@classmethod
|
||||
def validate_variants(cls, value: dict[str, str]) -> dict[str, str]:
|
||||
for width, url in value.items():
|
||||
if width not in {"640", "1280"}:
|
||||
raise ValueError("Unsupported image variant")
|
||||
safe_url(url, local=True)
|
||||
return value
|
||||
|
||||
|
||||
class CardImageRule(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid", str_strip_whitespace=True)
|
||||
id: str = Field(min_length=1, max_length=100, pattern=r"^[a-zA-Z0-9_-]+$")
|
||||
entity_type: EntityType
|
||||
scope: MatchScope
|
||||
value: str = Field(min_length=1, max_length=300)
|
||||
label: str = Field(default="", max_length=500)
|
||||
asset_id: str | None = Field(default=None, max_length=100)
|
||||
suppress: bool = False
|
||||
enabled: bool = True
|
||||
|
||||
@model_validator(mode="after")
|
||||
def validate_action(self) -> Self:
|
||||
if self.scope != "object":
|
||||
self.value = self.value.casefold()
|
||||
if self.suppress and (self.scope != "object" or self.asset_id):
|
||||
raise ValueError("Only an object rule may suppress its image, without an asset")
|
||||
if not self.suppress and not self.asset_id:
|
||||
raise ValueError("Choose an image or suppress this object")
|
||||
return self
|
||||
|
||||
def target(self) -> tuple[str, str, str]:
|
||||
return self.entity_type, self.scope, self.value
|
||||
|
||||
|
||||
class CardImageConfiguration(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
assets: list[CardImageAsset] = Field(default_factory=list, max_length=1000)
|
||||
rules: list[CardImageRule] = Field(default_factory=list, max_length=3000)
|
||||
|
||||
@model_validator(mode="after")
|
||||
def validate_relations(self) -> Self:
|
||||
assets = {asset.id for asset in self.assets}
|
||||
if len(assets) != len(self.assets) or len({rule.id for rule in self.rules}) != len(
|
||||
self.rules
|
||||
):
|
||||
raise ValueError("Image and rule IDs must be unique")
|
||||
targets = [rule.target() for rule in self.rules]
|
||||
if len(set(targets)) != len(targets):
|
||||
raise ValueError("Each target may have only one rule")
|
||||
if any(rule.asset_id and rule.asset_id not in assets for rule in self.rules):
|
||||
raise ValueError("Remove image associations before deleting the image")
|
||||
return self
|
||||
|
||||
|
||||
class CardImagePublish(BaseModel):
|
||||
revision: str = Field(max_length=200)
|
||||
config: CardImageConfiguration
|
||||
|
||||
|
||||
class CardImagePreview(BaseModel):
|
||||
entity_type: EntityType
|
||||
entity_id: str = Field(min_length=1, max_length=300)
|
||||
config: CardImageConfiguration
|
||||
94
backend/app/schemas/compute_configuration.py
Normal file
94
backend/app/schemas/compute_configuration.py
Normal file
@@ -0,0 +1,94 @@
|
||||
"""Validated, source-backed observations for the compute comparison cards."""
|
||||
|
||||
from datetime import date
|
||||
from typing import Annotated, Literal, Self
|
||||
|
||||
from app.services.compute_catalog import COUNTRY_PARTICIPANTS
|
||||
from pydantic import BaseModel, ConfigDict, Field, HttpUrl, field_validator, model_validator
|
||||
|
||||
Year = Annotated[int, Field(strict=True, ge=2000, le=2100)]
|
||||
Value = Annotated[float, Field(strict=True, allow_inf_nan=False, ge=0, le=1e12)]
|
||||
Identifier = Annotated[str, Field(min_length=1, max_length=80, pattern=r"^[a-zA-Z0-9_-]+$")]
|
||||
Text = Annotated[str, Field(min_length=1, max_length=500)]
|
||||
|
||||
|
||||
class ConfigurationModel(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid", str_strip_whitespace=True)
|
||||
|
||||
|
||||
class ComputeSource(ConfigurationModel):
|
||||
id: Identifier
|
||||
title: Text
|
||||
url: HttpUrl
|
||||
publisher: Text
|
||||
report_year: Year
|
||||
published_at: date | None = None
|
||||
locator: Text
|
||||
|
||||
@field_validator("url")
|
||||
@classmethod
|
||||
def public_reference(cls, value: HttpUrl) -> HttpUrl:
|
||||
if value.username or value.password or len(str(value)) > 2000:
|
||||
raise ValueError("Use a report URL without credentials (maximum 2000 characters)")
|
||||
return value
|
||||
|
||||
|
||||
class ComputeObservation(ConfigurationModel):
|
||||
year: Year
|
||||
metric: Literal["global_share", "absolute"]
|
||||
source_id: Identifier
|
||||
values: dict[str, Value] = Field(min_length=1, max_length=250)
|
||||
|
||||
@model_validator(mode="after")
|
||||
def comparable_values(self) -> Self:
|
||||
if any(code not in COUNTRY_PARTICIPANTS for code in self.values):
|
||||
raise ValueError(
|
||||
"Use countries from the canonical compute catalog, not regional totals"
|
||||
)
|
||||
if self.metric == "global_share" and sum(self.values.values()) > 100.000001:
|
||||
raise ValueError("Global shares for one year must total at most 100 percent")
|
||||
return self
|
||||
|
||||
|
||||
class InnovationObservation(ConfigurationModel):
|
||||
year: Year
|
||||
value: Annotated[float, Field(strict=True, allow_inf_nan=False, ge=-100, le=100)]
|
||||
source_id: Identifier
|
||||
|
||||
|
||||
class ComputeConfiguration(ConfigurationModel):
|
||||
reviewed_at: date
|
||||
sources: list[ComputeSource] = Field(max_length=200)
|
||||
observations: list[ComputeObservation] = Field(max_length=202)
|
||||
innovation: list[InnovationObservation] = Field(max_length=101)
|
||||
|
||||
@model_validator(mode="after")
|
||||
def consistent_evidence(self) -> Self:
|
||||
ids = [source.id for source in self.sources]
|
||||
if len(ids) != len(set(ids)):
|
||||
raise ValueError("Report source IDs must be unique")
|
||||
keys = [(row.metric, row.year) for row in self.observations]
|
||||
if len(keys) != len(set(keys)):
|
||||
raise ValueError("Each metric/year may have only one observation")
|
||||
years = [row.year for row in self.innovation]
|
||||
if len(years) != len(set(years)):
|
||||
raise ValueError("Innovation observation years must be unique")
|
||||
sources = {source.id: source for source in self.sources}
|
||||
for row in [*self.observations, *self.innovation]:
|
||||
source = sources.get(row.source_id)
|
||||
if source is None:
|
||||
raise ValueError(f"Missing report source: {row.source_id}")
|
||||
if row.year > source.report_year:
|
||||
raise ValueError("Observation year must not be after its report year")
|
||||
if self.reviewed_at > date.today():
|
||||
raise ValueError("Review date must not be in the future")
|
||||
return self
|
||||
|
||||
|
||||
class ComputeConfigurationWrite(ConfigurationModel):
|
||||
revision: str = Field(min_length=1, max_length=80)
|
||||
config: ComputeConfiguration
|
||||
|
||||
|
||||
class ComputeConfigurationReset(ConfigurationModel):
|
||||
revision: str = Field(min_length=1, max_length=80)
|
||||
@@ -36,6 +36,9 @@ class UserInDB(UserBase):
|
||||
|
||||
|
||||
class UserResponse(UserBase):
|
||||
# Bootstrap accounts have reserved .local email addresses. Validate email
|
||||
# on writes, while allowing existing local identities to be serialized.
|
||||
email: str
|
||||
id: int
|
||||
role: str
|
||||
gatekeeper_groups: list[str] = Field(default_factory=list)
|
||||
|
||||
174
backend/app/services/card_image_configuration.py
Normal file
174
backend/app/services/card_image_configuration.py
Normal file
@@ -0,0 +1,174 @@
|
||||
"""Revision-checked image settings; new bundled assets never replace user edits."""
|
||||
|
||||
from uuid import uuid4
|
||||
|
||||
from fastapi import HTTPException
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.dialects.postgresql import insert
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.request_context import get_request_id
|
||||
from app.models.system_log import AuditLog
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.models.user import User
|
||||
from app.schemas.card_images import CardImageConfiguration
|
||||
from app.services.card_image_defaults import (
|
||||
bundled_configuration,
|
||||
previous_bundled_assets,
|
||||
previous_bundled_rules,
|
||||
)
|
||||
|
||||
CATEGORY = "earth_card_images"
|
||||
|
||||
|
||||
def effective_envelope(payload: dict | None) -> dict:
|
||||
defaults, version = bundled_configuration()
|
||||
config = (
|
||||
CardImageConfiguration.model_validate(payload["config"])
|
||||
if payload
|
||||
else defaults.model_copy(deep=True)
|
||||
)
|
||||
if payload:
|
||||
previous = previous_bundled_assets()
|
||||
latest = {asset.id: asset for asset in defaults.assets}
|
||||
for index, asset in enumerate(config.assets):
|
||||
old = next((item for item in previous.get(asset.id, []) if item.url == asset.url), None)
|
||||
# Replace the old bundled source while preserving fields the user
|
||||
# changed (crop, title, enabled state, etc.). Custom URLs are untouched.
|
||||
if old is not None and asset.url == old.url and asset.id in latest:
|
||||
overrides = {
|
||||
key: value
|
||||
for key, value in asset.model_dump().items()
|
||||
if value != getattr(old, key)
|
||||
}
|
||||
config.assets[index] = latest[asset.id].model_copy(update=overrides, deep=True)
|
||||
known_assets = set(payload.get("known_assets", []))
|
||||
asset_ids = {asset.id for asset in config.assets}
|
||||
config.assets.extend(
|
||||
asset.model_copy(deep=True)
|
||||
for asset in defaults.assets
|
||||
if asset.id not in known_assets and asset.id not in asset_ids
|
||||
)
|
||||
prior_rules = previous_bundled_rules()
|
||||
latest_rules = {rule.id: rule for rule in defaults.rules}
|
||||
for index, rule in enumerate(config.rules):
|
||||
old_rule = prior_rules.get(rule.id)
|
||||
if (
|
||||
old_rule
|
||||
and rule.id in latest_rules
|
||||
and rule.target() == old_rule.target()
|
||||
and rule.asset_id == old_rule.asset_id
|
||||
):
|
||||
overrides = {
|
||||
key: value
|
||||
for key, value in rule.model_dump().items()
|
||||
if value != getattr(old_rule, key)
|
||||
}
|
||||
config.rules[index] = latest_rules[rule.id].model_copy(update=overrides, deep=True)
|
||||
known_rules = set(payload.get("known_rules", []))
|
||||
rule_ids = {rule.id for rule in config.rules}
|
||||
targets = {rule.target() for rule in config.rules}
|
||||
available = {asset.id for asset in config.assets}
|
||||
config.rules.extend(
|
||||
rule.model_copy(deep=True)
|
||||
for rule in defaults.rules
|
||||
if rule.id not in known_rules
|
||||
and rule.id not in rule_ids
|
||||
and rule.target() not in targets
|
||||
and rule.asset_id in available
|
||||
)
|
||||
return {
|
||||
"config": config.model_dump(mode="json"),
|
||||
"revision": f"{version}:{payload['revision'] if payload else 'default'}",
|
||||
"is_default": payload is None,
|
||||
}
|
||||
|
||||
|
||||
async def read_configuration(db: AsyncSession) -> dict:
|
||||
record = (
|
||||
await db.execute(select(SystemSetting).where(SystemSetting.category == CATEGORY))
|
||||
).scalar_one_or_none()
|
||||
return effective_envelope(record.payload if record else None)
|
||||
|
||||
|
||||
async def publish_configuration(
|
||||
db: AsyncSession, revision: str, config: CardImageConfiguration, actor: User
|
||||
) -> dict:
|
||||
defaults, _ = bundled_configuration()
|
||||
pending = {asset.id for asset in defaults.assets if asset.rights_status == "permission_pending"}
|
||||
if any(
|
||||
asset.id in pending and asset.rights_status != "permission_pending"
|
||||
for asset in config.assets
|
||||
):
|
||||
raise HTTPException(422, "Bundled image permission is still unverified")
|
||||
initial = {
|
||||
"config": defaults.model_dump(mode="json"),
|
||||
"revision": "default",
|
||||
"known_assets": [],
|
||||
"known_rules": [],
|
||||
}
|
||||
await db.execute(
|
||||
insert(SystemSetting)
|
||||
.values(category=CATEGORY, payload=initial)
|
||||
.on_conflict_do_nothing(index_elements=[SystemSetting.category])
|
||||
)
|
||||
record = (
|
||||
await db.execute(
|
||||
select(SystemSetting).where(SystemSetting.category == CATEGORY).with_for_update()
|
||||
)
|
||||
).scalar_one()
|
||||
if effective_envelope(record.payload)["revision"] != revision:
|
||||
await db.rollback()
|
||||
raise HTTPException(409, "Configuration changed. Reload before saving your edits.")
|
||||
before = record.payload
|
||||
record.payload = {
|
||||
"config": config.model_dump(mode="json"),
|
||||
"revision": uuid4().hex,
|
||||
"known_assets": [asset.id for asset in defaults.assets],
|
||||
"known_rules": [rule.id for rule in defaults.rules],
|
||||
}
|
||||
db.add(
|
||||
AuditLog(
|
||||
actor_id=actor.id,
|
||||
actor_name=actor.username,
|
||||
action="card_images.update",
|
||||
target_type="system_setting",
|
||||
target_id=CATEGORY,
|
||||
result="success",
|
||||
request_id=get_request_id(),
|
||||
details={"before": before, "after": record.payload},
|
||||
)
|
||||
)
|
||||
await db.commit()
|
||||
return effective_envelope(record.payload)
|
||||
|
||||
|
||||
def resolve_image(config: CardImageConfiguration, context: dict[str, str]) -> dict:
|
||||
assets = {asset.id: asset for asset in config.assets}
|
||||
for scope in ("object", "model", "group", "category"):
|
||||
value = context.get("entity_id" if scope == "object" else scope)
|
||||
values = [value] if value else []
|
||||
if scope == "category" and value != "*":
|
||||
values.append("*")
|
||||
for value in values:
|
||||
value = value if scope == "object" else value.casefold()
|
||||
rule = next(
|
||||
(
|
||||
rule
|
||||
for rule in config.rules
|
||||
if rule.enabled
|
||||
and rule.entity_type == context["entity_type"]
|
||||
and rule.scope == scope
|
||||
and rule.value == value
|
||||
),
|
||||
None,
|
||||
)
|
||||
if not rule:
|
||||
continue
|
||||
match = {"scope": scope, "value": value, "rule_id": rule.id}
|
||||
if rule.suppress:
|
||||
return {"image": None, "match": match, "suppressed": True}
|
||||
asset = assets.get(rule.asset_id)
|
||||
if asset and asset.enabled:
|
||||
return {"image": asset.model_dump(mode="json"), "match": match, "suppressed": False}
|
||||
return {"image": None, "match": None, "suppressed": False}
|
||||
108
backend/app/services/card_image_defaults.py
Normal file
108
backend/app/services/card_image_defaults.py
Normal file
@@ -0,0 +1,108 @@
|
||||
"""Read immutable bundled assets without inserting machine-specific database rows."""
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
|
||||
from app.schemas.card_images import CardImageAsset, CardImageConfiguration, CardImageRule
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[3]
|
||||
MANIFEST_PATH = REPO_ROOT / "frontend/public/earth/assets/cards/manifest.json"
|
||||
|
||||
|
||||
def _asset_from_manifest(item: dict) -> CardImageAsset:
|
||||
pending = item.get("redistribution_status") == "permission_not_verified"
|
||||
return CardImageAsset(
|
||||
id=item["id"],
|
||||
title=item["label"],
|
||||
url=item["url"],
|
||||
width=item["width"],
|
||||
height=item["height"],
|
||||
credit=item["credit"],
|
||||
source_url=item.get("source_page") or "",
|
||||
license=item["license"],
|
||||
license_url=item.get("license_url") or "",
|
||||
kind=item["kind"],
|
||||
rights_status="permission_pending" if pending else "available",
|
||||
enabled=item.get("enabled", not pending),
|
||||
focus_x=item.get("focus_x", 50),
|
||||
focus_y=item.get("focus_y", 50),
|
||||
variants=item.get("variants", {}),
|
||||
)
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def previous_bundled_assets() -> dict[str, list[CardImageAsset]]:
|
||||
manifest = json.loads(MANIFEST_PATH.read_bytes())
|
||||
return {
|
||||
key: [CardImageAsset.model_validate(item) for item in value]
|
||||
for key, value in manifest.get("previous_defaults", {}).items()
|
||||
}
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def previous_bundled_rules() -> dict[str, CardImageRule]:
|
||||
manifest = json.loads(MANIFEST_PATH.read_bytes())
|
||||
return {
|
||||
key: CardImageRule.model_validate(value)
|
||||
for key, value in manifest.get("previous_rules", {}).items()
|
||||
}
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def bundled_configuration() -> tuple[CardImageConfiguration, str]:
|
||||
content = MANIFEST_PATH.read_bytes()
|
||||
manifest = json.loads(content)
|
||||
assets, rules = [], []
|
||||
for item in manifest["assets"]:
|
||||
asset = _asset_from_manifest(item)
|
||||
assets.append(asset)
|
||||
if not asset.enabled:
|
||||
continue
|
||||
hints = item.get("binding_hints", {})
|
||||
for index, target in enumerate(hints.get("defaults", [])):
|
||||
rules.append(
|
||||
CardImageRule(
|
||||
id=target.get("id", f"builtin-{asset.id}-default-{index}"),
|
||||
entity_type=target["entity_type"],
|
||||
scope=target["scope"],
|
||||
value=target["value"],
|
||||
label=asset.title,
|
||||
asset_id=asset.id,
|
||||
)
|
||||
)
|
||||
for index, entity_id in enumerate(hints.get("entity_ids", [])):
|
||||
rules.append(
|
||||
CardImageRule(
|
||||
id=f"builtin-{asset.id}-entity-{index}",
|
||||
entity_type=item["entity_type"],
|
||||
scope="object",
|
||||
value=entity_id,
|
||||
label=asset.title,
|
||||
asset_id=asset.id,
|
||||
)
|
||||
)
|
||||
for norad_id in hints.get("norad_ids", []):
|
||||
rules.append(
|
||||
CardImageRule(
|
||||
id=f"builtin-{asset.id}-{norad_id}",
|
||||
entity_type="satellite",
|
||||
scope="object",
|
||||
value=str(norad_id),
|
||||
label=f"{asset.title} · NORAD {norad_id}",
|
||||
asset_id=asset.id,
|
||||
)
|
||||
)
|
||||
if group := hints.get("constellation_group"):
|
||||
rules.append(
|
||||
CardImageRule(
|
||||
id=f"builtin-{asset.id}-group",
|
||||
entity_type="satellite",
|
||||
scope="group",
|
||||
value=group,
|
||||
label=asset.title,
|
||||
asset_id=asset.id,
|
||||
)
|
||||
)
|
||||
return CardImageConfiguration(assets=assets, rules=rules), hashlib.sha256(content).hexdigest()
|
||||
95
backend/app/services/card_image_files.py
Normal file
95
backend/app/services/card_image_files.py
Normal file
@@ -0,0 +1,95 @@
|
||||
"""Validated raster uploads and content-addressed, bounded card-size derivatives."""
|
||||
|
||||
import hashlib
|
||||
import io
|
||||
import re
|
||||
import warnings
|
||||
from uuid import uuid4
|
||||
from pathlib import Path
|
||||
import os
|
||||
|
||||
from fastapi import HTTPException
|
||||
from PIL import Image, ImageOps, UnidentifiedImageError
|
||||
|
||||
from app.schemas.card_images import CardImageAsset
|
||||
from app.services.card_image_defaults import REPO_ROOT
|
||||
|
||||
ASSET_DIR = Path(
|
||||
os.environ.get("PLANET_CARD_IMAGE_DIR", str(REPO_ROOT / "data/earth-card-images"))
|
||||
).resolve()
|
||||
MAX_UPLOAD_BYTES = 12 * 1024 * 1024
|
||||
MAX_PIXELS = 32_000_000
|
||||
FILE_PATTERN = re.compile(r"^[a-f0-9]{64}-(?:original\.(?:jpg|png|webp)|(?:640|1280)\.webp)$")
|
||||
FORMATS = {"JPEG": "jpg", "PNG": "png", "WEBP": "webp"}
|
||||
|
||||
|
||||
def uploaded_path(filename: str) -> Path:
|
||||
if not FILE_PATTERN.fullmatch(filename):
|
||||
raise HTTPException(404, "Image not found")
|
||||
return ASSET_DIR / filename
|
||||
|
||||
|
||||
def store_image(content: bytes, title: str) -> CardImageAsset:
|
||||
if len(content) > MAX_UPLOAD_BYTES:
|
||||
raise HTTPException(413, "Image must be 12 MiB or smaller")
|
||||
try:
|
||||
with warnings.catch_warnings():
|
||||
warnings.simplefilter("error", Image.DecompressionBombWarning)
|
||||
image = Image.open(io.BytesIO(content))
|
||||
extension = FORMATS.get(image.format)
|
||||
if (
|
||||
not extension
|
||||
or image.width * image.height > MAX_PIXELS
|
||||
or getattr(image, "is_animated", False)
|
||||
):
|
||||
raise ValueError("Unsupported or oversized raster")
|
||||
image.load()
|
||||
image = ImageOps.exif_transpose(image).convert(
|
||||
"RGBA" if "A" in image.getbands() or "transparency" in image.info else "RGB"
|
||||
)
|
||||
except (
|
||||
UnidentifiedImageError,
|
||||
OSError,
|
||||
ValueError,
|
||||
Image.DecompressionBombError,
|
||||
Image.DecompressionBombWarning,
|
||||
) as error:
|
||||
raise HTTPException(
|
||||
422, "Use a valid, still JPEG, PNG or WebP image below 32 megapixels"
|
||||
) from error
|
||||
digest = hashlib.sha256(content).hexdigest()
|
||||
ASSET_DIR.mkdir(parents=True, exist_ok=True)
|
||||
original = f"{digest}-original.{extension}"
|
||||
original_path = uploaded_path(original)
|
||||
if not original_path.exists():
|
||||
temporary = original_path.with_suffix(f".{uuid4().hex}.tmp")
|
||||
temporary.write_bytes(content)
|
||||
temporary.replace(original_path)
|
||||
variants = {}
|
||||
for size, content in render_variants(image).items():
|
||||
filename = f"{digest}-{size}.webp"
|
||||
path = uploaded_path(filename)
|
||||
if not path.exists():
|
||||
temporary = path.with_suffix(f".{uuid4().hex}.tmp")
|
||||
temporary.write_bytes(content)
|
||||
temporary.replace(path)
|
||||
variants[size] = f"/api/v1/earth/card-images/files/{filename}"
|
||||
return CardImageAsset(
|
||||
id=f"upload-{digest[:24]}",
|
||||
title=title or "Image",
|
||||
url=f"/api/v1/earth/card-images/files/{original}",
|
||||
variants=variants,
|
||||
width=image.width,
|
||||
height=image.height,
|
||||
)
|
||||
|
||||
|
||||
def render_variants(image: Image.Image) -> dict[str, bytes]:
|
||||
result = {}
|
||||
for size in (640, 1280):
|
||||
resized = image.copy()
|
||||
resized.thumbnail((size, size), Image.Resampling.LANCZOS)
|
||||
buffer = io.BytesIO()
|
||||
resized.save(buffer, format="WEBP", quality=93, method=4)
|
||||
result[str(size)] = buffer.getvalue()
|
||||
return result
|
||||
180
backend/app/services/card_image_targets.py
Normal file
180
backend/app/services/card_image_targets.py
Normal file
@@ -0,0 +1,180 @@
|
||||
"""Bounded catalog queries and type defaults for every Earth object card."""
|
||||
|
||||
from sqlalchemy import String, case, cast, func, literal, or_, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.compute_types import (
|
||||
COMPUTE_SITE_TYPES,
|
||||
compute_site_type_expression,
|
||||
compute_record_filter,
|
||||
)
|
||||
from app.models.bgp_incident import BGPIncident
|
||||
from app.models.bgp_collector_location import BGPCollectorLocation
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.models.earth_interactable import EarthInteractable
|
||||
from app.models.vessel import VesselCurrentState
|
||||
from app.schemas.card_images import EntityType, MatchScope
|
||||
|
||||
SOURCES = {
|
||||
"satellite": ("celestrak_tle", "spacetrack_tle"),
|
||||
"compute_center": ("top500", "epoch_ai_gpu"),
|
||||
"cable": ("arcgis_cables",),
|
||||
"landing_point": ("arcgis_landing_points",),
|
||||
}
|
||||
# Named constellation families, not inferred hardware models or generations.
|
||||
SATELLITE_GROUP_PATTERNS = (
|
||||
("STARLINK%", "starlink"),
|
||||
("IRIDIUM%", "iridium-next"),
|
||||
("BEIDOU-3%", "beidou-3"),
|
||||
("BEIDOU%", "beidou"),
|
||||
("QIANFAN%", "qianfan"),
|
||||
("ONEWEB%", "oneweb"),
|
||||
("KUIPER%", "kuiper"),
|
||||
)
|
||||
|
||||
|
||||
def collected_query(entity_type: EntityType):
|
||||
record, metadata = CollectedData, CollectedData.extra_data
|
||||
model = func.coalesce(
|
||||
metadata["satellite_model" if entity_type == "satellite" else "model"].as_string(), ""
|
||||
)
|
||||
category = (
|
||||
compute_site_type_expression(record)
|
||||
if entity_type == "compute_center"
|
||||
else literal(entity_type)
|
||||
)
|
||||
if entity_type == "satellite":
|
||||
group = func.coalesce(
|
||||
func.nullif(func.lower(metadata["constellation_group"].as_string()), ""),
|
||||
case(
|
||||
*[
|
||||
(func.upper(record.name).like(pattern), value)
|
||||
for pattern, value in SATELLITE_GROUP_PATTERNS
|
||||
],
|
||||
else_="",
|
||||
),
|
||||
)
|
||||
entity_id = metadata["norad_cat_id"].as_string()
|
||||
else:
|
||||
group = func.coalesce(
|
||||
metadata["operator"].as_string(), metadata["organization"].as_string(), ""
|
||||
)
|
||||
entity_id = record.source + ":" + record.source_id
|
||||
return select(
|
||||
record.name.label("name"),
|
||||
cast(entity_id, String).label("entity_id"),
|
||||
model.label("model"),
|
||||
group.label("group"),
|
||||
category.label("category"),
|
||||
).where(
|
||||
(
|
||||
compute_record_filter(record)
|
||||
if entity_type == "compute_center"
|
||||
else record.source.in_(SOURCES[entity_type])
|
||||
),
|
||||
record.is_current.is_(True),
|
||||
record.deleted_at.is_(None),
|
||||
metadata["anonymous"].as_boolean().is_not(True),
|
||||
)
|
||||
|
||||
|
||||
def target_query(entity_type: EntityType):
|
||||
if entity_type in SOURCES:
|
||||
return collected_query(entity_type)
|
||||
if entity_type == "vessel":
|
||||
model = VesselCurrentState
|
||||
identity, name, group, category = model.mmsi, model.name, model.flag, model.vessel_type_name
|
||||
conditions = []
|
||||
elif entity_type == "bgp":
|
||||
model = BGPIncident
|
||||
identity, name, group, category = (
|
||||
model.incident_key,
|
||||
model.title,
|
||||
model.source,
|
||||
model.incident_type,
|
||||
)
|
||||
conditions = []
|
||||
elif entity_type == "bgp_collector":
|
||||
model = BGPCollectorLocation
|
||||
identity, name, group, category = (
|
||||
model.collector_id,
|
||||
model.collector_id,
|
||||
model.operator,
|
||||
literal("bgp_collector"),
|
||||
)
|
||||
conditions = []
|
||||
else:
|
||||
model = EarthInteractable
|
||||
identity, name, group, category = model.id, model.label, model.layer, model.kind
|
||||
conditions = [model.is_deleted.is_(False)]
|
||||
return select(
|
||||
name.label("name"),
|
||||
cast(identity, String).label("entity_id"),
|
||||
literal("").label("model"),
|
||||
func.coalesce(group, "").label("group"),
|
||||
func.coalesce(category, entity_type).label("category"),
|
||||
).where(*conditions)
|
||||
|
||||
|
||||
def category_context(entity_type: EntityType) -> dict[str, str]:
|
||||
return {
|
||||
"entity_type": entity_type,
|
||||
"entity_id": "",
|
||||
"name": "",
|
||||
"model": "",
|
||||
"group": "",
|
||||
"category": entity_type,
|
||||
}
|
||||
|
||||
|
||||
async def get_target(db: AsyncSession, entity_type: EntityType, entity_id: str) -> dict[str, str]:
|
||||
if entity_type == "compute_center" and entity_id.startswith("__category__:"):
|
||||
category = entity_id.partition(":")[2]
|
||||
if category in COMPUTE_SITE_TYPES:
|
||||
return {**category_context(entity_type), "category": category}
|
||||
if entity_id == "__category__":
|
||||
return category_context(entity_type)
|
||||
if entity_type in SOURCES and entity_type != "satellite":
|
||||
source, separator, source_id = entity_id.partition(":")
|
||||
if not separator or not source or not source_id:
|
||||
return category_context(entity_type)
|
||||
# Match the indexed identity columns before projecting JSON metadata.
|
||||
statement = collected_query(entity_type).where(
|
||||
CollectedData.source == source, CollectedData.source_id == source_id
|
||||
)
|
||||
else:
|
||||
catalog = target_query(entity_type).subquery()
|
||||
statement = select(catalog).where(catalog.c.entity_id == entity_id)
|
||||
row = (await db.execute(statement.limit(1))).mappings().first()
|
||||
if not row:
|
||||
# Old map snapshots can outlive a catalog refresh. They may use the type
|
||||
# illustration, but must not claim an unverified exact-object identity.
|
||||
return category_context(entity_type)
|
||||
context = {key: str(value or "") for key, value in row.items()}
|
||||
return {**context, "entity_type": entity_type}
|
||||
|
||||
|
||||
async def search_targets(
|
||||
db: AsyncSession, entity_type: EntityType, scope: MatchScope, query: str, limit: int
|
||||
) -> list[dict[str, str]]:
|
||||
catalog = target_query(entity_type).subquery()
|
||||
value = catalog.c["entity_id" if scope == "object" else scope]
|
||||
label = catalog.c.name if scope == "object" else value
|
||||
statement = select(value.label("value"), label.label("label")).where(
|
||||
value.is_not(None), value != ""
|
||||
)
|
||||
if query:
|
||||
escaped = query.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
|
||||
statement = statement.where(
|
||||
or_(value.ilike(f"%{escaped}%", escape="\\"), label.ilike(f"%{escaped}%", escape="\\"))
|
||||
)
|
||||
rows = (await db.execute(statement.distinct().order_by(label, value).limit(limit))).mappings()
|
||||
items = [{"value": row["value"], "label": row["label"] or row["value"]} for row in rows]
|
||||
if scope == "category" and not query:
|
||||
known = {item["value"] for item in items}
|
||||
if entity_type == "compute_center":
|
||||
items.extend(
|
||||
{"value": kind, "label": kind} for kind in COMPUTE_SITE_TYPES if kind not in known
|
||||
)
|
||||
items.insert(0, {"value": "*", "label": "*"})
|
||||
return items
|
||||
@@ -31,6 +31,7 @@ class BaseCollector(ABC):
|
||||
frequency_hours: int = 4
|
||||
data_type: str = "generic"
|
||||
fail_on_empty: bool = False
|
||||
preserve_snapshot_on_failure: bool = False
|
||||
|
||||
def __init__(self):
|
||||
self._current_task = None
|
||||
@@ -361,6 +362,8 @@ class BaseCollector(ABC):
|
||||
context={"task_id": task_id, "raw_count": len(raw_data)},
|
||||
)
|
||||
data = self.transform(raw_data)
|
||||
if self.fail_on_empty and not data:
|
||||
raise RuntimeError(f"Collector {self.name} produced no valid records")
|
||||
await self._log_collection_event(
|
||||
"collector.phase.transforming.success",
|
||||
"Collector transform phase completed",
|
||||
@@ -456,6 +459,10 @@ class BaseCollector(ABC):
|
||||
raise
|
||||
except Exception as e:
|
||||
await db.rollback()
|
||||
if snapshot_id is not None and self.preserve_snapshot_on_failure:
|
||||
await self._rollback_incomplete_run(
|
||||
db, task_id=task_id, snapshot_id=snapshot_id, reason=str(e)
|
||||
)
|
||||
task.status = JobStatus.FAILED.value
|
||||
task.phase = "failed"
|
||||
task.phase_message = str(e)
|
||||
@@ -465,6 +472,8 @@ class BaseCollector(ABC):
|
||||
snapshot = await db.get(DataSnapshot, snapshot_id)
|
||||
if snapshot:
|
||||
snapshot.status = SnapshotStatus.FAILED.value
|
||||
if self.preserve_snapshot_on_failure:
|
||||
snapshot.is_current = False
|
||||
snapshot.completed_at = datetime.now(UTC)
|
||||
snapshot.summary = {"error": str(e)}
|
||||
await db.commit()
|
||||
|
||||
@@ -18,7 +18,6 @@ from app.services.business_logs import emit_business_log, exception_context
|
||||
from app.services.collectors.base import BaseCollector
|
||||
from app.services.collectors.downloads import DownloadHTTPStatusError, ResumableFileDownloader
|
||||
|
||||
|
||||
logger = get_logger(__name__, service="collector")
|
||||
ACTIVE_GROUP = "active"
|
||||
FALLBACK_GROUPS = (
|
||||
@@ -39,6 +38,7 @@ FALLBACK_GROUPS = (
|
||||
)
|
||||
FETCH_RETRY_ATTEMPTS = 3
|
||||
FETCH_RETRY_BASE_DELAY_SECONDS = 0.8
|
||||
CELESTRAK_CACHE_TTL_SECONDS = 2 * 60 * 60
|
||||
CELESTRAK_NOT_UPDATED_MARKER = "GP data has not updated since your last successful"
|
||||
|
||||
|
||||
@@ -51,6 +51,8 @@ class CelesTrakTLECollector(BaseCollector):
|
||||
_downloader = ResumableFileDownloader(
|
||||
cache_namespace="celestrak",
|
||||
default_accept="application/json",
|
||||
preflight_head=False,
|
||||
cache_ttl_seconds=CELESTRAK_CACHE_TTL_SECONDS,
|
||||
)
|
||||
|
||||
@property
|
||||
@@ -195,7 +197,9 @@ class CelesTrakTLECollector(BaseCollector):
|
||||
if not is_final_attempt:
|
||||
await asyncio.sleep(FETCH_RETRY_BASE_DELAY_SECONDS * attempt)
|
||||
|
||||
raise RuntimeError(f"CelesTrak active satellite download failed after retries: {last_error}")
|
||||
raise RuntimeError(
|
||||
f"CelesTrak active satellite download failed after retries: {last_error}"
|
||||
)
|
||||
|
||||
async def _fetch_fallback_groups(
|
||||
self,
|
||||
@@ -226,25 +230,25 @@ class CelesTrakTLECollector(BaseCollector):
|
||||
try:
|
||||
for group in FALLBACK_GROUPS:
|
||||
group_url = self._group_url(group)
|
||||
cached_path = self._downloader.get_cached_file(
|
||||
group_url,
|
||||
".json",
|
||||
validate_existing=self._validate_json_file,
|
||||
)
|
||||
if cached_path is not None:
|
||||
body_path = cached_path
|
||||
else:
|
||||
try:
|
||||
body_path = await self._downloader.download_file(
|
||||
client,
|
||||
group_url,
|
||||
extension=".json",
|
||||
accept="application/json",
|
||||
validate_existing=self._validate_json_file,
|
||||
)
|
||||
except DownloadHTTPStatusError as exc:
|
||||
if not self._is_not_updated_response(exc):
|
||||
raise RuntimeError(f"CelesTrak fallback group '{group}' download failed: {exc}") from exc
|
||||
try:
|
||||
body_path = await self._downloader.download_file(
|
||||
client,
|
||||
group_url,
|
||||
extension=".json",
|
||||
accept="application/json",
|
||||
validate_existing=self._validate_json_file,
|
||||
)
|
||||
except DownloadHTTPStatusError as exc:
|
||||
if not self._is_not_updated_response(exc):
|
||||
raise RuntimeError(
|
||||
f"CelesTrak fallback group '{group}' download failed: {exc}"
|
||||
) from exc
|
||||
body_path = self._downloader.get_cached_file(
|
||||
group_url,
|
||||
".json",
|
||||
validate_existing=self._validate_json_file,
|
||||
)
|
||||
if body_path is None:
|
||||
raise RuntimeError(
|
||||
f"CelesTrak fallback group '{group}' has not updated and no local cached copy is available"
|
||||
) from exc
|
||||
@@ -348,13 +352,13 @@ class CelesTrakTLECollector(BaseCollector):
|
||||
commit=True,
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _validate_json_file(path: Path) -> bool:
|
||||
@classmethod
|
||||
def _validate_json_file(cls, path: Path) -> bool:
|
||||
try:
|
||||
data = json.loads(path.read_text(encoding="utf-8"))
|
||||
except (OSError, json.JSONDecodeError, UnicodeDecodeError):
|
||||
cls._load_active_payload(path)
|
||||
except RuntimeError:
|
||||
return False
|
||||
return isinstance(data, list)
|
||||
return True
|
||||
|
||||
async def _log_parse_failure(self, exc: Exception) -> None:
|
||||
await emit_business_log(
|
||||
@@ -375,7 +379,8 @@ class CelesTrakTLECollector(BaseCollector):
|
||||
),
|
||||
)
|
||||
|
||||
def _load_active_payload(self, path: Path) -> List[Dict[str, Any]]:
|
||||
@staticmethod
|
||||
def _load_active_payload(path: Path) -> List[Dict[str, Any]]:
|
||||
try:
|
||||
raw = json.loads(path.read_text(encoding="utf-8"))
|
||||
except (OSError, json.JSONDecodeError, UnicodeDecodeError) as exc:
|
||||
@@ -391,7 +396,9 @@ class CelesTrakTLECollector(BaseCollector):
|
||||
else:
|
||||
invalid_count += 1
|
||||
if invalid_count:
|
||||
raise RuntimeError(f"CelesTrak active payload contains {invalid_count} invalid record(s)")
|
||||
raise RuntimeError(
|
||||
f"CelesTrak active payload contains {invalid_count} invalid record(s)"
|
||||
)
|
||||
if not records:
|
||||
raise RuntimeError("CelesTrak active payload contains no satellite records")
|
||||
return records
|
||||
|
||||
82
backend/app/services/collectors/compute_base.py
Normal file
82
backend/app/services/collectors/compute_base.py
Normal file
@@ -0,0 +1,82 @@
|
||||
"""Quality gates shared by the two compute sources."""
|
||||
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.countries import normalize_country
|
||||
from app.models.compute_center_location import ComputeCenterLocationRecord
|
||||
from app.models.data_snapshot import DataSnapshot
|
||||
from app.services.collectors.base import BaseCollector
|
||||
|
||||
|
||||
class ComputeCollector(BaseCollector):
|
||||
fail_on_empty = True
|
||||
preserve_snapshot_on_failure = True
|
||||
|
||||
async def _create_snapshot(
|
||||
self, db: AsyncSession, task_id: int, data: list[dict], started_at: datetime
|
||||
) -> int:
|
||||
previous_count = await db.scalar(
|
||||
select(DataSnapshot.record_count)
|
||||
.where(
|
||||
DataSnapshot.source == self.name,
|
||||
DataSnapshot.is_current.is_(True),
|
||||
DataSnapshot.status == "success",
|
||||
)
|
||||
.order_by(DataSnapshot.id.desc())
|
||||
.limit(1)
|
||||
)
|
||||
if previous_count and len(data) < previous_count * 0.6:
|
||||
raise ValueError("Compute source lost over 40% of rows; previous snapshot retained")
|
||||
return await super()._create_snapshot(db, task_id, data, started_at)
|
||||
|
||||
async def _save_data(self, db: AsyncSession, data: list[dict], **kwargs) -> int:
|
||||
if self.name == "top500":
|
||||
await self._copy_verified_locations(db, data)
|
||||
return await super()._save_data(db, data, **kwargs)
|
||||
|
||||
async def _copy_verified_locations(self, db: AsyncSession, data: list[dict]) -> None:
|
||||
"""Keep historical location keys; copy only unambiguous exact identities.
|
||||
|
||||
Never migrate by rank: the machine occupying rank 1 changes over time.
|
||||
"""
|
||||
names = [item["name"] for item in data]
|
||||
locations = (
|
||||
(
|
||||
await db.execute(
|
||||
select(ComputeCenterLocationRecord).where(
|
||||
ComputeCenterLocationRecord.source == self.name,
|
||||
ComputeCenterLocationRecord.name.in_(names),
|
||||
)
|
||||
)
|
||||
)
|
||||
.scalars()
|
||||
.all()
|
||||
)
|
||||
existing = {location.source_id for location in locations}
|
||||
for item in data:
|
||||
if item["source_id"] in existing:
|
||||
continue
|
||||
country = normalize_country(item["country"])
|
||||
if not country:
|
||||
continue
|
||||
matches = [
|
||||
location
|
||||
for location in locations
|
||||
if location.name == item["name"]
|
||||
and normalize_country(location.country) == country
|
||||
and (not location.site or location.site == item.get("metadata", {}).get("site"))
|
||||
and not location.source_id.startswith("top500_system_")
|
||||
]
|
||||
if len(matches) != 1:
|
||||
continue
|
||||
old = matches[0]
|
||||
values = {
|
||||
column.name: getattr(old, column.name)
|
||||
for column in old.__table__.columns
|
||||
if column.name not in {"id", "source_id", "created_at", "updated_at"}
|
||||
}
|
||||
db.add(ComputeCenterLocationRecord(**values, source_id=item["source_id"]))
|
||||
existing.add(item["source_id"])
|
||||
@@ -12,7 +12,6 @@ from typing import Any, Awaitable, Callable
|
||||
|
||||
import httpx
|
||||
|
||||
|
||||
ProgressCallback = Callable[[int, int | None], Awaitable[None]]
|
||||
ValidateCallback = Callable[[Path], bool]
|
||||
|
||||
@@ -52,10 +51,14 @@ class ResumableFileDownloader:
|
||||
user_agent: str = "Planet-Intelligence-System/1.0 (Python/collector)",
|
||||
default_accept: str = "*/*",
|
||||
cache_root: Path | None = None,
|
||||
preflight_head: bool = True,
|
||||
cache_ttl_seconds: float = 0,
|
||||
) -> None:
|
||||
self._cache_dir = (cache_root or default_download_cache_root()) / cache_namespace
|
||||
self._user_agent = user_agent
|
||||
self._default_accept = default_accept
|
||||
self._preflight_head = preflight_head
|
||||
self._cache_ttl_seconds = cache_ttl_seconds
|
||||
|
||||
@staticmethod
|
||||
def _cache_key(url: str) -> str:
|
||||
@@ -93,8 +96,9 @@ class ResumableFileDownloader:
|
||||
if not meta_path.exists():
|
||||
return {}
|
||||
try:
|
||||
return json.loads(meta_path.read_text(encoding="utf-8"))
|
||||
except (json.JSONDecodeError, OSError):
|
||||
data = json.loads(meta_path.read_text(encoding="utf-8"))
|
||||
return data if isinstance(data, dict) else {}
|
||||
except json.JSONDecodeError, OSError:
|
||||
return {}
|
||||
|
||||
@staticmethod
|
||||
@@ -111,20 +115,25 @@ class ResumableFileDownloader:
|
||||
return last_modified == str(meta.get("last_modified") or "").strip()
|
||||
return True
|
||||
|
||||
@staticmethod
|
||||
def _response_info(response: httpx.Response) -> dict[str, Any]:
|
||||
# httpx yields decoded bytes; encoded Content-Length cannot validate them.
|
||||
raw_length = response.headers.get("content-length")
|
||||
encoded = response.headers.get("content-encoding", "identity") != "identity"
|
||||
return {
|
||||
"etag": response.headers.get("etag"),
|
||||
"last_modified": response.headers.get("last-modified"),
|
||||
"content_length": int(raw_length) if raw_length and not encoded else None,
|
||||
"accept_ranges": (response.headers.get("accept-ranges") or "").lower(),
|
||||
}
|
||||
|
||||
async def fetch_remote_info(self, client: httpx.AsyncClient, url: str) -> dict[str, Any]:
|
||||
try:
|
||||
response = await client.head(url)
|
||||
if response.status_code >= 400:
|
||||
return {}
|
||||
content_length_raw = response.headers.get("content-length")
|
||||
content_length = int(content_length_raw) if content_length_raw else None
|
||||
return {
|
||||
"etag": response.headers.get("etag"),
|
||||
"last_modified": response.headers.get("last-modified"),
|
||||
"content_length": content_length,
|
||||
"accept_ranges": (response.headers.get("accept-ranges") or "").lower(),
|
||||
}
|
||||
except (httpx.HTTPError, ValueError):
|
||||
return self._response_info(response)
|
||||
except httpx.HTTPError, ValueError:
|
||||
return {}
|
||||
|
||||
async def download_file(
|
||||
@@ -139,14 +148,24 @@ class ResumableFileDownloader:
|
||||
) -> Path:
|
||||
self._cache_dir.mkdir(parents=True, exist_ok=True)
|
||||
final_path, part_path, meta_path = self._cache_paths(url, extension)
|
||||
cached_path = self.get_cached_file(url, extension, validate_existing=validate_existing)
|
||||
if cached_path and 0 <= time.time() - cached_path.stat().st_mtime < self._cache_ttl_seconds:
|
||||
size = cached_path.stat().st_size
|
||||
if progress_callback:
|
||||
await progress_callback(size, size)
|
||||
return cached_path
|
||||
meta = self._load_meta(meta_path)
|
||||
remote = await self.fetch_remote_info(client, url)
|
||||
remote = await self.fetch_remote_info(client, url) if self._preflight_head else {}
|
||||
expected_size = remote.get("content_length")
|
||||
|
||||
if final_path.exists():
|
||||
local_size = final_path.stat().st_size
|
||||
size_match = expected_size is None or local_size == expected_size
|
||||
if self._validators_match(meta, remote) and size_match:
|
||||
if (
|
||||
(remote.get("etag") or remote.get("last_modified"))
|
||||
and self._validators_match(meta, remote)
|
||||
and size_match
|
||||
):
|
||||
if validate_existing and not validate_existing(final_path):
|
||||
final_path.unlink(missing_ok=True)
|
||||
else:
|
||||
@@ -181,9 +200,13 @@ class ResumableFileDownloader:
|
||||
headers["If-Range"] = str(remote.get("last_modified"))
|
||||
|
||||
async with client.stream("GET", url, headers=headers) as response:
|
||||
if response.status_code == 304 and final_path.exists():
|
||||
if progress_callback and expected_size and expected_size > 0:
|
||||
await progress_callback(expected_size, expected_size)
|
||||
if response.status_code == 304:
|
||||
if cached_path is None:
|
||||
raise RuntimeError(f"Server returned 304 without a valid local cache for {url}")
|
||||
final_path.touch()
|
||||
size = final_path.stat().st_size
|
||||
if progress_callback:
|
||||
await progress_callback(size, size)
|
||||
return final_path
|
||||
if response.status_code >= 400:
|
||||
body = (await response.aread()).decode("utf-8", errors="replace")
|
||||
@@ -194,12 +217,16 @@ class ResumableFileDownloader:
|
||||
else:
|
||||
mode = "wb"
|
||||
resume_from = 0
|
||||
remote = self._response_info(response)
|
||||
expected_size = remote.get("content_length")
|
||||
|
||||
downloaded = resume_from
|
||||
last_emit_bytes = 0
|
||||
last_emit_time = time.monotonic()
|
||||
min_emit_bytes = (
|
||||
max(expected_size // 150, 512 * 1024) if expected_size and expected_size > 0 else 1024 * 1024
|
||||
max(expected_size // 150, 512 * 1024)
|
||||
if expected_size and expected_size > 0
|
||||
else 1024 * 1024
|
||||
)
|
||||
|
||||
with part_path.open(mode) as f:
|
||||
@@ -230,6 +257,10 @@ class ResumableFileDownloader:
|
||||
f"Resumable download incomplete for {url}: expected={expected_size}, got={final_size}"
|
||||
)
|
||||
|
||||
# Never replace the last good payload with an HTML error or invalid JSON.
|
||||
if validate_existing and not validate_existing(part_path):
|
||||
part_path.unlink(missing_ok=True)
|
||||
raise RuntimeError(f"Downloaded file validation failed for {url}")
|
||||
part_path.replace(final_path)
|
||||
self._save_meta(
|
||||
meta_path,
|
||||
@@ -242,9 +273,6 @@ class ResumableFileDownloader:
|
||||
},
|
||||
)
|
||||
|
||||
if validate_existing and not validate_existing(final_path):
|
||||
raise RuntimeError(f"Downloaded file validation failed for {url}")
|
||||
|
||||
if progress_callback and expected_size and expected_size > 0:
|
||||
await progress_callback(expected_size, expected_size)
|
||||
|
||||
|
||||
@@ -1,119 +1,135 @@
|
||||
"""Epoch AI GPU Clusters Collector
|
||||
|
||||
Collects data from Epoch AI GPU clusters tracking.
|
||||
https://epoch.ai/data/gpu-clusters
|
||||
"""
|
||||
"""Epoch's official CSV; preserve precision, source dates and anonymous records."""
|
||||
|
||||
import csv
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
import math
|
||||
import re
|
||||
from typing import Dict, Any, List
|
||||
from datetime import UTC, datetime
|
||||
from bs4 import BeautifulSoup
|
||||
from urllib.parse import urlparse
|
||||
|
||||
import httpx
|
||||
|
||||
from app.services.collectors.base import BaseCollector
|
||||
from app.core.countries import normalize_country
|
||||
from app.services.collectors.compute_base import ComputeCollector
|
||||
from app.services.epoch_location import assess_epoch_location
|
||||
|
||||
EPOCH_CSV_URL = "https://epoch.ai/data/gpu_clusters.csv"
|
||||
|
||||
|
||||
|
||||
class EpochAIGPUCollector(BaseCollector):
|
||||
class EpochAIGPUCollector(ComputeCollector):
|
||||
name = "epoch_ai_gpu"
|
||||
priority = "P0"
|
||||
module = "L1"
|
||||
frequency_hours = 6
|
||||
data_type = "gpu_cluster"
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
"""Fetch Epoch AI GPU clusters data from webpage"""
|
||||
url = self._resolved_url or ""
|
||||
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
async def fetch(self) -> list[dict]:
|
||||
configured = self._resolved_url or EPOCH_CSV_URL
|
||||
parsed = urlparse(configured)
|
||||
url = (
|
||||
EPOCH_CSV_URL
|
||||
if parsed.hostname == "epoch.ai" and not parsed.path.endswith(".csv")
|
||||
else configured
|
||||
)
|
||||
async with httpx.AsyncClient(timeout=60, follow_redirects=True) as client:
|
||||
response = await client.get(url)
|
||||
response.raise_for_status()
|
||||
return self.parse_response(response.text)
|
||||
return self.parse_response(response.text)
|
||||
|
||||
def parse_response(self, html: str) -> List[Dict[str, Any]]:
|
||||
"""Parse Epoch AI webpage to extract GPU cluster data"""
|
||||
data = []
|
||||
soup = BeautifulSoup(html, "html.parser")
|
||||
def parse_response(self, text: str) -> list[dict]:
|
||||
reader = csv.DictReader(io.StringIO(text.lstrip("\ufeff")))
|
||||
required = {"Name", "Country", "Status", "First Operational Date", "16-bit OP/s (log)"}
|
||||
if not required.issubset(reader.fieldnames or []):
|
||||
raise ValueError(
|
||||
"Epoch CSV schema changed or HTML returned; previous snapshot retained"
|
||||
)
|
||||
entries = []
|
||||
seen = set()
|
||||
anonymous_counts = {}
|
||||
for row in reader:
|
||||
name = (row.get("Name") or "").strip()
|
||||
country = normalize_country(row.get("Country"))
|
||||
if not name:
|
||||
raise ValueError("Epoch row has missing identity")
|
||||
source_id = f"epoch_{re.sub(r'[^a-zA-Z0-9]', '_', name.lower())}"
|
||||
anonymous = "anonym" in name.lower()
|
||||
if anonymous:
|
||||
# Public anonymous rows have no facility identity. Preserve
|
||||
# multiplicity within snapshots, without guessing names.
|
||||
digest = hashlib.sha256(json.dumps(row, sort_keys=True).encode()).hexdigest()[:24]
|
||||
anonymous_counts[digest] = anonymous_counts.get(digest, 0) + 1
|
||||
source_id = f"epoch_anonymous_{digest}_{anonymous_counts[digest]}"
|
||||
if source_id in seen:
|
||||
raise ValueError("Epoch identity collision; previous snapshot retained")
|
||||
seen.add(source_id)
|
||||
raw_date = row.get("First Operational Date", "")
|
||||
reference_date = raw_date if re.fullmatch(r"\d{4}-\d{2}-\d{2}", raw_date) else None
|
||||
value = None
|
||||
raw_value = row.get("16-bit OP/s (log)", "")
|
||||
if raw_value:
|
||||
exponent = float(raw_value)
|
||||
if not math.isfinite(exponent) or not 0 <= exponent <= 30:
|
||||
raise ValueError("Epoch performance exponent out of bounds")
|
||||
value = 10**exponent
|
||||
anonymous = anonymous or (country == "中国" and not row.get("Location"))
|
||||
location = row.get("Location")
|
||||
location_assessment = assess_epoch_location(
|
||||
location,
|
||||
country=country,
|
||||
raw_country=row.get("Country"),
|
||||
)
|
||||
entries.append(
|
||||
{
|
||||
"source_id": source_id,
|
||||
"name": name,
|
||||
"country": country,
|
||||
"value": value,
|
||||
"unit": "FLOP/s",
|
||||
"reference_date": reference_date,
|
||||
"latitude": None if anonymous else self.coordinate(row.get("latitude"), 90),
|
||||
"longitude": None if anonymous else self.coordinate(row.get("longitude"), 180),
|
||||
"metadata": {
|
||||
"compute_contract": "v1",
|
||||
"precision": "FP16",
|
||||
"measurement": "theoretical",
|
||||
"source_url": EPOCH_CSV_URL,
|
||||
"raw_reference_date": raw_date,
|
||||
"status": row.get("Status"),
|
||||
"certainty": row.get("Certainty"),
|
||||
"operator": row.get("Owner"),
|
||||
"gpu_type": row.get("Chip type (primary)"),
|
||||
"gpu_count": row.get("Chip quantity (primary)"),
|
||||
"location": location,
|
||||
"location_precision": location_assessment.precision,
|
||||
"location_precision_reason": location_assessment.reason,
|
||||
"raw_country": row.get("Country"),
|
||||
"raw_performance": raw_value,
|
||||
"performance_encoding": "log10",
|
||||
"builds_upon": row.get("Builds Upon"),
|
||||
"superseded_by": row.get("Superseded by"),
|
||||
"possible_duplicate": row.get("Possible Duplicate", "").lower() == "true",
|
||||
"include_in_standard_analysis": row.get(
|
||||
"Include in Standard Analysis", ""
|
||||
).lower()
|
||||
== "true",
|
||||
"rounded": country == "中国",
|
||||
"anonymous": anonymous,
|
||||
"identity_scope": "snapshot_record" if anonymous else "named_cluster",
|
||||
"country_unresolved": country is None,
|
||||
"sources": [
|
||||
row[f"Source {i}"] for i in range(1, 6) if row.get(f"Source {i}")
|
||||
],
|
||||
},
|
||||
}
|
||||
)
|
||||
return entries
|
||||
|
||||
# Try to find data table on the page
|
||||
tables = soup.find_all("table")
|
||||
for table in tables:
|
||||
rows = table.find_all("tr")
|
||||
for row in rows[1:]: # Skip header
|
||||
cells = row.find_all(["td", "th"])
|
||||
if len(cells) >= 5:
|
||||
try:
|
||||
cluster_name = cells[0].get_text(strip=True)
|
||||
if not cluster_name or cluster_name in ["Cluster", "System", "Name"]:
|
||||
continue
|
||||
|
||||
location_cell = cells[1].get_text(strip=True) if len(cells) > 1 else ""
|
||||
country, city = self._parse_location(location_cell)
|
||||
|
||||
perf_cell = cells[2].get_text(strip=True) if len(cells) > 2 else ""
|
||||
|
||||
entry = {
|
||||
"source_id": f"epoch_{re.sub(r'[^a-zA-Z0-9]', '_', cluster_name.lower())}",
|
||||
"name": cluster_name,
|
||||
"country": country,
|
||||
"city": city,
|
||||
"latitude": "",
|
||||
"longitude": "",
|
||||
"value": self._parse_performance(perf_cell),
|
||||
"unit": "TFlop/s",
|
||||
"metadata": {
|
||||
"raw_data": perf_cell,
|
||||
},
|
||||
"reference_date": datetime.now(UTC).strftime("%Y-%m-%d"),
|
||||
}
|
||||
data.append(entry)
|
||||
except (ValueError, IndexError, AttributeError):
|
||||
continue
|
||||
|
||||
# If no table found, return sample data
|
||||
if not data:
|
||||
data = self._get_sample_data()
|
||||
|
||||
return data
|
||||
|
||||
def _parse_location(self, location: str) -> tuple:
|
||||
"""Parse location string into country and city"""
|
||||
if not location:
|
||||
return "", ""
|
||||
if "," in location:
|
||||
parts = location.rsplit(",", 1)
|
||||
city = parts[0].strip()
|
||||
country = parts[1].strip() if len(parts) > 1 else ""
|
||||
return country, city
|
||||
return location, ""
|
||||
|
||||
def _parse_performance(self, perf: str) -> str:
|
||||
"""Parse performance string to extract value"""
|
||||
if not perf:
|
||||
return "0"
|
||||
match = re.search(r"([\d,.]+)\s*(TFlop/s|PFlop/s|GFlop/s)?", perf, re.I)
|
||||
if match:
|
||||
return match.group(1).replace(",", "")
|
||||
match = re.search(r"([\d,.]+)", perf)
|
||||
if match:
|
||||
return match.group(1).replace(",", "")
|
||||
return "0"
|
||||
|
||||
def _get_sample_data(self) -> List[Dict[str, Any]]:
|
||||
"""Return sample data for testing when scraping fails"""
|
||||
return [
|
||||
{
|
||||
"source_id": "epoch_sample_1",
|
||||
"name": "Sample GPU Cluster",
|
||||
"country": "United States",
|
||||
"city": "San Francisco, CA",
|
||||
"latitude": "",
|
||||
"longitude": "",
|
||||
"value": "1000",
|
||||
"unit": "TFlop/s",
|
||||
"metadata": {
|
||||
"note": "Sample data - Epoch AI page structure may vary",
|
||||
},
|
||||
"reference_date": datetime.now(UTC).strftime("%Y-%m-%d"),
|
||||
},
|
||||
]
|
||||
@staticmethod
|
||||
def coordinate(value: str | None, bound: int) -> float | None:
|
||||
if not value:
|
||||
return None
|
||||
number = float(value)
|
||||
if not math.isfinite(number) or abs(number) > bound:
|
||||
raise ValueError("Epoch coordinate out of bounds")
|
||||
return number
|
||||
|
||||
@@ -35,7 +35,7 @@ class NewsLiveStreamsCollector(BaseCollector):
|
||||
DEFAULT_IPTV_ORG_LOGOS_URL = "https://iptv-org.github.io/api/logos.json"
|
||||
DEFAULT_IPTV_ORG_NEWS_CATEGORIES = ("news", "business", "weather")
|
||||
DEFAULT_IPTV_ORG_EXCLUDE_CATEGORIES = ("music", "sports", "kids", "entertainment")
|
||||
DEFAULT_IPTV_ORG_MAX_SOURCES = 120
|
||||
DEFAULT_IPTV_ORG_MAX_SOURCES = 0 # Zero keeps the complete matching channel catalog.
|
||||
|
||||
async def fetch(self) -> list[dict[str, Any]]:
|
||||
request_url = (self._resolved_url or "").strip()
|
||||
@@ -445,7 +445,7 @@ class NewsLiveStreamsCollector(BaseCollector):
|
||||
"reference_date": datetime.now(UTC).isoformat(),
|
||||
}
|
||||
)
|
||||
if len(normalized) >= max_sources:
|
||||
if max_sources > 0 and len(normalized) >= max_sources:
|
||||
break
|
||||
|
||||
return normalized
|
||||
|
||||
@@ -5,16 +5,18 @@ https://top500.org/lists/top500/
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import math
|
||||
import re
|
||||
from urllib.parse import urljoin
|
||||
from typing import Dict, Any, List
|
||||
from bs4 import BeautifulSoup
|
||||
import httpx
|
||||
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
from app.services.collectors.base import BaseCollector
|
||||
from app.services.collectors.compute_base import ComputeCollector
|
||||
|
||||
|
||||
class TOP500Collector(BaseCollector):
|
||||
class TOP500Collector(ComputeCollector):
|
||||
name = "top500"
|
||||
priority = "P0"
|
||||
module = "L1"
|
||||
@@ -28,7 +30,7 @@ class TOP500Collector(BaseCollector):
|
||||
async with httpx.AsyncClient(timeout=60.0, follow_redirects=True) as client:
|
||||
response = await client.get(url)
|
||||
response.raise_for_status()
|
||||
entries = self.parse_response(response.text)
|
||||
entries = self.parse_response(response.text, str(response.url))
|
||||
|
||||
semaphore = asyncio.Semaphore(8)
|
||||
|
||||
@@ -55,7 +57,7 @@ class TOP500Collector(BaseCollector):
|
||||
system_name = link.get_text(" ", strip=True) if link else system_cell.get_text(" ", strip=True)
|
||||
detail_url = ""
|
||||
if link and link.get("href"):
|
||||
detail_url = f"{str(top500_base_url).rstrip('/')}{link.get('href')}"
|
||||
detail_url = urljoin(str(top500_base_url), link.get("href"))
|
||||
|
||||
manufacturer = ""
|
||||
if link and link.next_sibling:
|
||||
@@ -127,10 +129,14 @@ class TOP500Collector(BaseCollector):
|
||||
|
||||
return detail_map
|
||||
|
||||
def parse_response(self, html: str) -> List[Dict[str, Any]]:
|
||||
def parse_response(self, html: str, source_url: str = "") -> List[Dict[str, Any]]:
|
||||
"""Parse TOP500 HTML response"""
|
||||
data = []
|
||||
soup = BeautifulSoup(html, "html.parser")
|
||||
canonical = soup.find("link", rel="canonical")
|
||||
date_url = source_url or (canonical.get("href", "") if canonical else "") or self._resolved_url or ""
|
||||
date_match = re.search(r"/lists/top500/(\d{4})/(06|11)/", date_url)
|
||||
reference_date = f"{date_match[1]}-{date_match[2]}-01" if date_match else None
|
||||
|
||||
# Find the table with TOP500 data
|
||||
table = None
|
||||
@@ -152,6 +158,9 @@ class TOP500Collector(BaseCollector):
|
||||
table = soup.find("table", {"class": "top500-table"}) or soup.find("table", {"id": "top500"})
|
||||
|
||||
if table:
|
||||
headers = [cell.get_text(" ", strip=True) for cell in table.select("tr")[0].find_all(["th", "td"])]
|
||||
if not reference_date:
|
||||
raise ValueError("TOP500 list date missing; previous snapshot retained")
|
||||
rows = table.select("tr")
|
||||
for row in rows[1:]: # Skip header row
|
||||
cells = row.find_all(["td", "th"])
|
||||
@@ -176,25 +185,32 @@ class TOP500Collector(BaseCollector):
|
||||
cores = cells[2].get_text(strip=True).replace(",", "")
|
||||
|
||||
rmax_text = cells[3].get_text(strip=True)
|
||||
rmax = self._parse_performance(rmax_text)
|
||||
rmax = self._parse_performance(rmax_text, headers[3])
|
||||
|
||||
rpeak_text = cells[4].get_text(strip=True)
|
||||
rpeak = self._parse_performance(rpeak_text)
|
||||
rpeak = self._parse_performance(rpeak_text, headers[4])
|
||||
|
||||
power = ""
|
||||
if len(cells) >= 6:
|
||||
power = cells[5].get_text(strip=True).replace(",", "")
|
||||
|
||||
identity = re.search(r"/system/(\d+)", detail_url)
|
||||
if not identity:
|
||||
raise ValueError("TOP500 stable system ID missing")
|
||||
entry = {
|
||||
"source_id": f"top500_{rank}",
|
||||
"source_id": f"top500_system_{identity[1]}",
|
||||
"name": system_name,
|
||||
"country": country,
|
||||
"city": city,
|
||||
"latitude": 0.0,
|
||||
"longitude": 0.0,
|
||||
"latitude": None,
|
||||
"longitude": None,
|
||||
"value": str(rmax),
|
||||
"unit": "PFlop/s",
|
||||
"unit": "FLOP/s",
|
||||
"metadata": {
|
||||
"compute_contract": "v1",
|
||||
"precision": "FP64", "measurement": "LINPACK Rmax",
|
||||
"source_url": detail_url, "list_url": date_url,
|
||||
"status": "listed", "rmax_flops": rmax, "rpeak_flops": rpeak,
|
||||
"rank": rank,
|
||||
"cores": cores,
|
||||
"rmax": rmax_text,
|
||||
@@ -204,15 +220,11 @@ class TOP500Collector(BaseCollector):
|
||||
"site": site,
|
||||
},
|
||||
"_detail_url": detail_url,
|
||||
"reference_date": "2025-11-01",
|
||||
"reference_date": reference_date,
|
||||
}
|
||||
data.append(entry)
|
||||
except (ValueError, IndexError, AttributeError) as e:
|
||||
continue
|
||||
|
||||
# If scraping failed, return sample data for testing
|
||||
if not data:
|
||||
data = self._get_sample_data()
|
||||
raise ValueError("Invalid TOP500 row; previous snapshot retained") from e
|
||||
|
||||
return data
|
||||
|
||||
@@ -227,9 +239,14 @@ class TOP500Collector(BaseCollector):
|
||||
return 0.0
|
||||
return 0.0
|
||||
|
||||
def _parse_performance(self, text: str) -> float:
|
||||
def _parse_performance(self, text: str, header: str = "") -> float:
|
||||
"""Parse performance value from text (handles E, P, T suffixes)"""
|
||||
text = text.strip().upper()
|
||||
text = text.strip().upper().replace(",", "")
|
||||
if not re.search(r"[EPTGMK]F", text):
|
||||
unit = re.search(r"\b([EPTGMK])FLOP/?S", header.upper())
|
||||
if not unit:
|
||||
raise ValueError("TOP500 performance unit missing")
|
||||
text = f"{text} {unit[1]}FLOP/S"
|
||||
multipliers = {
|
||||
"E": 1e18,
|
||||
"P": 1e15,
|
||||
@@ -249,73 +266,9 @@ class TOP500Collector(BaseCollector):
|
||||
|
||||
# Try simple float parsing
|
||||
try:
|
||||
return float(text.replace(",", ""))
|
||||
value = float(text.replace(",", ""))
|
||||
if not math.isfinite(value) or value < 0:
|
||||
raise ValueError("Invalid TOP500 performance")
|
||||
return value
|
||||
except ValueError:
|
||||
return 0.0
|
||||
|
||||
def _get_sample_data(self) -> List[Dict[str, Any]]:
|
||||
"""Return sample data for testing when scraping fails"""
|
||||
return [
|
||||
{
|
||||
"source_id": "top500_1",
|
||||
"name": "El Capitan - HPE Cray EX255a, AMD 4th Gen EPYC 24C 1.8GHz, AMD Instinct MI300A",
|
||||
"country": "United States",
|
||||
"city": "Livermore, CA",
|
||||
"latitude": 37.6819,
|
||||
"longitude": -121.7681,
|
||||
"value": "1742.00",
|
||||
"unit": "PFlop/s",
|
||||
"metadata": {
|
||||
"rank": 1,
|
||||
"cores": "11039616",
|
||||
"rmax": "1742.00",
|
||||
"rpeak": "2746.38",
|
||||
"power": "29581",
|
||||
"manufacturer": "HPE",
|
||||
"site": "DOE/NNSA/LLNL",
|
||||
"processor": "AMD 4th Gen EPYC 24C 1.8GHz",
|
||||
"interconnect": "Slingshot-11",
|
||||
"installation_year": "2025",
|
||||
},
|
||||
"reference_date": "2025-11-01",
|
||||
},
|
||||
{
|
||||
"source_id": "top500_2",
|
||||
"name": "Frontier - HPE Cray EX235a, AMD Optimized 3rd Generation EPYC 64C 2GHz, AMD Instinct MI250X",
|
||||
"country": "United States",
|
||||
"city": "Oak Ridge, TN",
|
||||
"latitude": 36.0107,
|
||||
"longitude": -84.2663,
|
||||
"value": "1353.00",
|
||||
"unit": "PFlop/s",
|
||||
"metadata": {
|
||||
"rank": 2,
|
||||
"cores": "9066176",
|
||||
"rmax": "1353.00",
|
||||
"rpeak": "2055.72",
|
||||
"power": "24607",
|
||||
"manufacturer": "HPE",
|
||||
"site": "DOE/SC/Oak Ridge National Laboratory",
|
||||
},
|
||||
"reference_date": "2025-11-01",
|
||||
},
|
||||
{
|
||||
"source_id": "top500_3",
|
||||
"name": "Aurora - HPE Cray EX - Intel Exascale Compute Blade, Xeon CPU Max 9470 52C 2.4GHz, Intel Data Center GPU Max",
|
||||
"country": "United States",
|
||||
"city": "Argonne, IL",
|
||||
"latitude": 41.3784,
|
||||
"longitude": -87.8600,
|
||||
"value": "1012.00",
|
||||
"unit": "PFlop/s",
|
||||
"metadata": {
|
||||
"rank": 3,
|
||||
"cores": "9264128",
|
||||
"rmax": "1012.00",
|
||||
"rpeak": "1980.01",
|
||||
"power": "38698",
|
||||
"manufacturer": "Intel",
|
||||
},
|
||||
"reference_date": "2025-11-01",
|
||||
},
|
||||
]
|
||||
raise ValueError("Invalid TOP500 performance") from None
|
||||
|
||||
200
backend/app/services/compute_catalog.py
Normal file
200
backend/app/services/compute_catalog.py
Normal file
@@ -0,0 +1,200 @@
|
||||
"""Canonical country and mutually exclusive regional compute participants."""
|
||||
|
||||
import re
|
||||
from collections.abc import Iterable
|
||||
|
||||
from app.core.countries import COUNTRY_CENTROIDS, COUNTRY_ENTRIES
|
||||
from app.core.earth_geography_policy import display_country
|
||||
|
||||
PALETTE = ("#91baff", "#f0c88a", "#89d2c2", "#c5abdf", "#eda9b3", "#b6c99a")
|
||||
DEFAULT_PARTICIPANTS = ("CN", "US", "EUROPE", "APAC", "OTHER")
|
||||
|
||||
# Explicit membership prevents a regional total from double-counting a country.
|
||||
# APAC deliberately excludes China.
|
||||
COMPARISON_GROUPS = (
|
||||
("CN", "中国", "China", ("CN",), {"latitude": 35.8617, "longitude": 104.1954}),
|
||||
("US", "美国", "United States", ("US",), {"latitude": 39.8283, "longitude": -98.5795}),
|
||||
(
|
||||
"EUROPE",
|
||||
"欧洲",
|
||||
"Europe",
|
||||
(
|
||||
"GB",
|
||||
"DE",
|
||||
"FR",
|
||||
"CH",
|
||||
"ES",
|
||||
"FI",
|
||||
"HU",
|
||||
"IE",
|
||||
"IT",
|
||||
"LU",
|
||||
"MD",
|
||||
"NL",
|
||||
"PL",
|
||||
"RO",
|
||||
"RU",
|
||||
"UA",
|
||||
),
|
||||
{"latitude": 54.0, "longitude": 20.0},
|
||||
),
|
||||
(
|
||||
"APAC",
|
||||
"亚太(不含中国)",
|
||||
"Asia Pacific (excluding China)",
|
||||
("JP", "KR", "IN", "AU", "AF", "ID", "NP", "PH", "PK", "SG", "TH"),
|
||||
{"latitude": 36.0, "longitude": 130.5},
|
||||
),
|
||||
(
|
||||
"OTHER",
|
||||
"其他地区",
|
||||
"Other",
|
||||
(
|
||||
"CA",
|
||||
"AO",
|
||||
"EG",
|
||||
"KE",
|
||||
"MZ",
|
||||
"NG",
|
||||
"UG",
|
||||
"ZA",
|
||||
"BR",
|
||||
"CO",
|
||||
"CR",
|
||||
"DO",
|
||||
"MX",
|
||||
"AE",
|
||||
"SY",
|
||||
"TR",
|
||||
),
|
||||
{"latitude": 12.0, "longitude": 10.0},
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _country_code(canonical: str, aliases: list[str]) -> str | None:
|
||||
code = next((alias for alias in aliases if re.fullmatch(r"[A-Z]{2}", alias)), None)
|
||||
return "GB" if canonical == "英国" else code
|
||||
|
||||
|
||||
def build_country_catalog() -> list[dict]:
|
||||
"""Build the policy-projected canonical country drilldown catalog."""
|
||||
countries = []
|
||||
for canonical, aliases in COUNTRY_ENTRIES:
|
||||
if display_country(canonical) != canonical or canonical not in COUNTRY_CENTROIDS:
|
||||
continue
|
||||
code = _country_code(canonical, aliases)
|
||||
if code:
|
||||
countries.append(
|
||||
{
|
||||
"code": code,
|
||||
"name": canonical,
|
||||
"name_en": aliases[0],
|
||||
"kind": "country",
|
||||
"members": [code],
|
||||
"anchor": dict(COUNTRY_CENTROIDS[canonical]),
|
||||
"anchor_kind": "regional_label",
|
||||
}
|
||||
)
|
||||
return countries
|
||||
|
||||
|
||||
COUNTRY_CATALOG = build_country_catalog()
|
||||
COUNTRIES = {entry["code"]: entry["name"] for entry in COUNTRY_CATALOG}
|
||||
COUNTRY_PARTICIPANTS = {entry["code"]: entry for entry in COUNTRY_CATALOG}
|
||||
|
||||
|
||||
def _validate_group_members() -> None:
|
||||
members = [
|
||||
member for _code, _name, _name_en, codes, _anchor in COMPARISON_GROUPS for member in codes
|
||||
]
|
||||
if len(members) != len(set(members)) or set(members) != set(COUNTRIES):
|
||||
raise ValueError("compute comparison groups must cover country catalog exactly once")
|
||||
|
||||
|
||||
_validate_group_members()
|
||||
GROUP_BY_MEMBER = {
|
||||
member: code
|
||||
for code, _name, _name_en, members, _anchor in COMPARISON_GROUPS
|
||||
for member in members
|
||||
}
|
||||
|
||||
|
||||
def build_catalog() -> list[dict]:
|
||||
"""Return the five default participants before country drilldown entries."""
|
||||
catalog: list[dict] = []
|
||||
for index, (code, name, name_en, members, anchor) in enumerate(COMPARISON_GROUPS):
|
||||
entry = (
|
||||
dict(COUNTRY_PARTICIPANTS[code])
|
||||
if code in COUNTRY_PARTICIPANTS
|
||||
else {
|
||||
"code": code,
|
||||
"name": name,
|
||||
"name_en": name_en,
|
||||
"kind": "region",
|
||||
"members": list(members),
|
||||
"anchor": dict(anchor),
|
||||
"anchor_kind": "regional_label",
|
||||
}
|
||||
)
|
||||
entry.update(
|
||||
{
|
||||
"members": list(members),
|
||||
"parent_group": name,
|
||||
"parent_group_code": code,
|
||||
"order": index,
|
||||
"default_selected": True,
|
||||
"color": PALETTE[index % len(PALETTE)],
|
||||
}
|
||||
)
|
||||
catalog.append(entry)
|
||||
|
||||
for entry in COUNTRY_CATALOG:
|
||||
if entry["code"] in DEFAULT_PARTICIPANTS:
|
||||
continue
|
||||
country = dict(entry)
|
||||
group_code = GROUP_BY_MEMBER[country["code"]]
|
||||
group_name = next(group[1] for group in COMPARISON_GROUPS if group[0] == group_code)
|
||||
country.update(
|
||||
{
|
||||
"parent_group": group_name,
|
||||
"parent_group_code": group_code,
|
||||
"order": len(catalog),
|
||||
"default_selected": False,
|
||||
"color": PALETTE[len(catalog) % len(PALETTE)],
|
||||
}
|
||||
)
|
||||
catalog.append(country)
|
||||
return catalog
|
||||
|
||||
|
||||
CATALOG = build_catalog()
|
||||
PARTICIPANTS = {entry["code"]: entry for entry in CATALOG}
|
||||
|
||||
|
||||
def participant_members(codes: Iterable[str]) -> dict[str, tuple[str, ...]]:
|
||||
"""Resolve country or region participant codes to canonical country codes."""
|
||||
return {code: tuple(PARTICIPANTS[code]["members"]) for code in codes}
|
||||
|
||||
|
||||
def source_country_names(codes: Iterable[str]) -> list[str]:
|
||||
"""Return unique canonical source labels, respecting China and Serbia ownership."""
|
||||
selected_members = {
|
||||
member for members in participant_members(codes).values() for member in members
|
||||
}
|
||||
selected_names = {COUNTRIES[code] for code in selected_members}
|
||||
return [name for name, _aliases in COUNTRY_ENTRIES if display_country(name) in selected_names]
|
||||
|
||||
|
||||
def source_country_participants(codes: Iterable[str]) -> dict[str, str]:
|
||||
"""Map stored source-country labels to their requested participant code."""
|
||||
owner_by_country = {
|
||||
member: participant
|
||||
for participant, members in participant_members(codes).items()
|
||||
for member in members
|
||||
}
|
||||
code_by_name = {name: code for code, name in COUNTRIES.items()}
|
||||
return {
|
||||
name: owner_by_country[code_by_name[display_country(name)]]
|
||||
for name in source_country_names(codes)
|
||||
}
|
||||
@@ -3,7 +3,12 @@
|
||||
This module is a thin domain wrapper that wires up
|
||||
:mod:`app.services.location` for compute centers:
|
||||
|
||||
SourceCoordinates
|
||||
Confirmed stored location -> trustworthy source coordinates -> exact
|
||||
official facility registration -> source city/site coordinates.
|
||||
|
||||
Epoch AI coordinates are admitted only at the grain asserted by its official
|
||||
``Location`` column. Country/region/empty locations remain unresolved even
|
||||
when the CSV happens to contain a latitude/longitude pair.
|
||||
|
||||
The online Nominatim step is intentionally reserved for the user-triggered
|
||||
``collect-location`` flow. The regular GeoJSON endpoint runs during Earth
|
||||
@@ -28,9 +33,12 @@ import httpx
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.compute_types import compute_record_filter
|
||||
from app.core.collected_data_fields import get_record_field
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.models.compute_center_location import ComputeCenterLocationRecord
|
||||
from app.services.compute_center_verified_sites import get_verified_compute_site
|
||||
from app.services.epoch_location import assess_epoch_location
|
||||
|
||||
from app.services.location import (
|
||||
LocationCandidate,
|
||||
@@ -166,8 +174,13 @@ def set_compute_center_location_cache(
|
||||
|
||||
async def refresh_compute_center_location_cache(
|
||||
session: AsyncSession,
|
||||
*,
|
||||
as_of: datetime | None = None,
|
||||
) -> dict[str, dict[str, Any]]:
|
||||
result = await session.execute(select(ComputeCenterLocationRecord))
|
||||
query = select(ComputeCenterLocationRecord)
|
||||
if as_of is not None:
|
||||
query = query.where(ComputeCenterLocationRecord.verified_at < as_of)
|
||||
result = await session.execute(query)
|
||||
records = result.scalars().all()
|
||||
cache = {}
|
||||
for record in records:
|
||||
@@ -350,6 +363,58 @@ class ROROrganizationResolver:
|
||||
)
|
||||
|
||||
|
||||
class ComputeCenterSourceCoordinatesResolver:
|
||||
"""Use source coordinates only at the grain asserted by the source text."""
|
||||
|
||||
name = "source_coordinates"
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._default = SourceCoordinatesResolver()
|
||||
|
||||
def resolve(self, query: LocationQuery) -> ResolverOutput:
|
||||
extra = query.extra or {}
|
||||
if extra.get("source") != "epoch_ai_gpu":
|
||||
return self._default.resolve(query)
|
||||
if query.source_latitude in (None, 0.0) or query.source_longitude in (None, 0.0):
|
||||
return ResolverOutput()
|
||||
assessment = assess_epoch_location(
|
||||
extra.get("source_location"),
|
||||
country=query.country,
|
||||
raw_country=extra.get("source_raw_country"),
|
||||
)
|
||||
if not assessment.is_renderable:
|
||||
return ResolverOutput(
|
||||
attempted_queries=(f"source_coordinates:{assessment.precision}",)
|
||||
)
|
||||
output = self._default.resolve(query)
|
||||
if not output.candidates:
|
||||
return output
|
||||
candidate = output.candidates[0]
|
||||
source_location = extra.get("source_location")
|
||||
source_location_text = (
|
||||
source_location.strip() if isinstance(source_location, str) else ""
|
||||
)
|
||||
source_note = assessment.reason
|
||||
if source_location_text:
|
||||
source_note = f"{source_note}; Epoch Location: {source_location_text}"
|
||||
return ResolverOutput(
|
||||
candidates=(
|
||||
LocationCandidate(
|
||||
**{
|
||||
**candidate.__dict__,
|
||||
"precision": assessment.precision,
|
||||
"confidence": 1.0 if assessment.precision == "precise" else 0.9,
|
||||
"source_note": source_note,
|
||||
"needs_confirmation": assessment.precision != "precise",
|
||||
"matched_location_name": source_location_text
|
||||
or candidate.matched_location_name,
|
||||
}
|
||||
),
|
||||
),
|
||||
attempted_queries=output.attempted_queries,
|
||||
)
|
||||
|
||||
|
||||
class StoredComputeCenterLocationResolver:
|
||||
"""Resolve a compute center through the DB-backed current-location cache."""
|
||||
|
||||
@@ -367,13 +432,26 @@ class StoredComputeCenterLocationResolver:
|
||||
longitude = parse_float(stored.get("longitude"))
|
||||
if latitude in (None, 0.0) or longitude in (None, 0.0):
|
||||
return ResolverOutput()
|
||||
precision = stored.get("precision") or "city"
|
||||
if (
|
||||
extra.get("source") == "epoch_ai_gpu"
|
||||
and stored.get("location_source") == "source_coordinates"
|
||||
):
|
||||
assessment = assess_epoch_location(
|
||||
extra.get("source_location"),
|
||||
country=query.country,
|
||||
raw_country=extra.get("source_raw_country"),
|
||||
)
|
||||
if not assessment.is_renderable:
|
||||
return ResolverOutput()
|
||||
precision = assessment.precision
|
||||
return ResolverOutput(
|
||||
candidates=(
|
||||
LocationCandidate(
|
||||
latitude=latitude,
|
||||
longitude=longitude,
|
||||
display_name=stored.get("name") or query.name or "Compute center",
|
||||
precision=stored.get("precision") or "city",
|
||||
precision=precision,
|
||||
confidence=float(stored.get("confidence") or 0.85),
|
||||
query=f"stored_compute_center_location::{stored.get('source')}:{stored.get('source_id')}",
|
||||
source=self.name,
|
||||
@@ -386,6 +464,60 @@ class StoredComputeCenterLocationResolver:
|
||||
matched_location_name=stored.get("site") or stored.get("name") or query.name,
|
||||
location_verified_at=stored.get("verified_at"),
|
||||
suggested_registry_entry=None,
|
||||
raw_payload={
|
||||
"stored_location_source": stored.get("location_source"),
|
||||
"verification_status": stored.get("verification_status"),
|
||||
},
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
class VerifiedComputeSiteResolver:
|
||||
"""Resolve an exact source identity to an evidence-backed facility site."""
|
||||
|
||||
name = "official_facility_registry"
|
||||
|
||||
def resolve(self, query: LocationQuery) -> ResolverOutput:
|
||||
extra = query.extra or {}
|
||||
source = coerce_str(extra.get("source"))
|
||||
source_id = coerce_str(extra.get("source_id"))
|
||||
if not source or not source_id:
|
||||
return ResolverOutput()
|
||||
site = get_verified_compute_site(source, source_id)
|
||||
if not site:
|
||||
return ResolverOutput()
|
||||
source_urls = [coerce_str(url) for url in site.get("sources", []) if coerce_str(url)]
|
||||
source_note = coerce_str(site.get("source_note"))
|
||||
if source_urls:
|
||||
source_note = f"{source_note} Sources: {'; '.join(source_urls)}".strip()
|
||||
return ResolverOutput(
|
||||
candidates=(
|
||||
LocationCandidate(
|
||||
latitude=float(site["latitude"]),
|
||||
longitude=float(site["longitude"]),
|
||||
display_name=coerce_str(site.get("matched_location_name"))
|
||||
or query.name
|
||||
or "Compute center",
|
||||
precision=coerce_str(site.get("precision")) or "site",
|
||||
confidence=1.0,
|
||||
query=f"official_facility_registry::{source}:{source_id}",
|
||||
source=self.name,
|
||||
source_note=source_note or None,
|
||||
matched_fields=("source", "source_id"),
|
||||
needs_confirmation=False,
|
||||
city=coerce_str(site.get("city")) or None,
|
||||
region=None,
|
||||
country=coerce_str(site.get("country")) or query.country,
|
||||
matched_location_name=coerce_str(site.get("matched_location_name"))
|
||||
or None,
|
||||
location_verified_at=coerce_str(site.get("verified_at")) or None,
|
||||
suggested_registry_entry=None,
|
||||
raw_payload={
|
||||
"address": site.get("address"),
|
||||
"coordinate_basis": site.get("coordinate_basis"),
|
||||
"sources": source_urls,
|
||||
},
|
||||
),
|
||||
)
|
||||
)
|
||||
@@ -400,7 +532,7 @@ def _short_system_name(name: Any) -> str:
|
||||
return head or text
|
||||
|
||||
|
||||
def _record_context(record: Any, metadata: dict[str, Any]) -> dict[str, str]:
|
||||
def _record_context(record: Any, metadata: dict[str, Any]) -> dict[str, Any]:
|
||||
name = coerce_str(getattr(record, "name", None))
|
||||
return {
|
||||
"source": coerce_str(getattr(record, "source", None)),
|
||||
@@ -417,11 +549,13 @@ def _record_context(record: Any, metadata: dict[str, Any]) -> dict[str, str]:
|
||||
or metadata.get("manufacturer")
|
||||
),
|
||||
"organization": coerce_str(metadata.get("organization")),
|
||||
"source_location": metadata.get("location"),
|
||||
"source_raw_country": metadata.get("raw_country"),
|
||||
}
|
||||
|
||||
|
||||
def _context_to_query(
|
||||
context: dict[str, str],
|
||||
context: dict[str, Any],
|
||||
*,
|
||||
source_lat: float | None = None,
|
||||
source_lon: float | None = None,
|
||||
@@ -444,6 +578,8 @@ def _context_to_query(
|
||||
"site": context.get("site") or "",
|
||||
"operator": context.get("operator") or "",
|
||||
"organization": context.get("organization") or "",
|
||||
"source_location": context.get("source_location"),
|
||||
"source_raw_country": context.get("source_raw_country"),
|
||||
},
|
||||
)
|
||||
|
||||
@@ -497,11 +633,36 @@ def _compute_center_query_plan(
|
||||
return plan
|
||||
|
||||
|
||||
def _compute_center_candidate_sort_key(candidate: LocationCandidate) -> tuple[int, int, float]:
|
||||
stored_origin = (candidate.raw_payload or {}).get("stored_location_source")
|
||||
stored_confirmed = (
|
||||
candidate.source == "stored_compute_center_location"
|
||||
and not candidate.needs_confirmation
|
||||
and stored_origin != "source_coordinates"
|
||||
)
|
||||
if stored_confirmed:
|
||||
source_rank = 0
|
||||
elif candidate.source == "source_coordinates" and candidate.precision == "precise":
|
||||
source_rank = 1
|
||||
elif candidate.source == "official_facility_registry":
|
||||
source_rank = 2
|
||||
elif candidate.source == "source_coordinates":
|
||||
source_rank = 3
|
||||
elif candidate.source == "stored_compute_center_location":
|
||||
source_rank = 4
|
||||
else:
|
||||
source_rank = 5
|
||||
precision_rank = {"precise": 0, "site": 1, "city": 2}.get(candidate.precision, 9)
|
||||
return (source_rank, precision_rank, -float(candidate.confidence or 0))
|
||||
|
||||
|
||||
COMPUTE_CENTER_PIPELINE = LocationPipeline(
|
||||
[
|
||||
SourceCoordinatesResolver(),
|
||||
ComputeCenterSourceCoordinatesResolver(),
|
||||
StoredComputeCenterLocationResolver(),
|
||||
VerifiedComputeSiteResolver(),
|
||||
],
|
||||
sort_key=_compute_center_candidate_sort_key,
|
||||
failure_reason=(
|
||||
"Could not resolve to city-level coordinates from source coords"
|
||||
" or stored compute-center location."
|
||||
@@ -510,7 +671,8 @@ COMPUTE_CENTER_PIPELINE = LocationPipeline(
|
||||
|
||||
COMPUTE_CENTER_COLLECTION_PIPELINE = LocationPipeline(
|
||||
[
|
||||
SourceCoordinatesResolver(),
|
||||
ComputeCenterSourceCoordinatesResolver(),
|
||||
VerifiedComputeSiteResolver(),
|
||||
ROROrganizationResolver(),
|
||||
NominatimResolver(
|
||||
query_plan_builder=_compute_center_query_plan,
|
||||
@@ -518,6 +680,7 @@ COMPUTE_CENTER_COLLECTION_PIPELINE = LocationPipeline(
|
||||
geocoder=lambda q: _geocode_online(q),
|
||||
),
|
||||
],
|
||||
sort_key=_compute_center_candidate_sort_key,
|
||||
failure_reason=(
|
||||
"Could not resolve to city-level coordinates from source coords"
|
||||
", ROR organization lookup, or online geocoding."
|
||||
@@ -531,6 +694,7 @@ COMPUTE_CENTER_COLLECTION_PIPELINE = LocationPipeline(
|
||||
_GEOGRAPHY_MODE_BY_SOURCE = {
|
||||
"source_coordinates": "source_coordinates",
|
||||
"stored_compute_center_location": "stored_compute_center_location",
|
||||
"official_facility_registry": "official_facility_registry",
|
||||
"ror_organization_registry": "ror_organization",
|
||||
"nominatim_online_geocode": "online_geocode",
|
||||
}
|
||||
@@ -539,7 +703,7 @@ _GEOGRAPHY_MODE_BY_SOURCE = {
|
||||
def _candidate_to_location(
|
||||
candidate: LocationCandidate,
|
||||
*,
|
||||
context: dict[str, str],
|
||||
context: dict[str, Any],
|
||||
) -> ComputeCenterLocation:
|
||||
geography_mode = _GEOGRAPHY_MODE_BY_SOURCE.get(candidate.source, "online_geocode")
|
||||
is_estimated = candidate.needs_confirmation or candidate.source.startswith(
|
||||
@@ -594,7 +758,7 @@ def _candidate_to_location(
|
||||
|
||||
def _diagnostic_for(
|
||||
record: Any,
|
||||
context: dict[str, str],
|
||||
context: dict[str, Any],
|
||||
*,
|
||||
failure_reason: str,
|
||||
attempted_queries: tuple[str, ...] = (),
|
||||
@@ -738,7 +902,7 @@ def build_compute_center_location_query(
|
||||
organization: str | None = None,
|
||||
) -> LocationQuery:
|
||||
name_value = coerce_str(name)
|
||||
context: dict[str, str] = {
|
||||
context: dict[str, Any] = {
|
||||
"source": coerce_str(source),
|
||||
"source_id": coerce_str(source_id),
|
||||
"name": name_value,
|
||||
@@ -767,7 +931,7 @@ async def seed_compute_center_locations_from_source_coords(
|
||||
"""Seed stored compute-center locations only from real source coordinates."""
|
||||
stmt = (
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source.in_(["top500", "epoch_ai_gpu"]))
|
||||
.where(compute_record_filter(CollectedData))
|
||||
.where(CollectedData.is_current.is_(True))
|
||||
)
|
||||
result = await session.execute(stmt)
|
||||
@@ -783,6 +947,23 @@ async def seed_compute_center_locations_from_source_coords(
|
||||
longitude = parse_float(get_record_field(record, "longitude"))
|
||||
if latitude in (None, 0.0) or longitude in (None, 0.0):
|
||||
continue
|
||||
metadata = record.extra_data or {}
|
||||
precision = "precise"
|
||||
confidence = 1.0
|
||||
needs_confirmation = False
|
||||
source_note = "Seeded from source-provided compute-center coordinates"
|
||||
if source_value == "epoch_ai_gpu":
|
||||
assessment = assess_epoch_location(
|
||||
metadata.get("location"),
|
||||
country=get_record_field(record, "country"),
|
||||
raw_country=metadata.get("raw_country"),
|
||||
)
|
||||
if not assessment.is_renderable:
|
||||
continue
|
||||
precision = assessment.precision
|
||||
confidence = 1.0 if precision == "precise" else 0.9
|
||||
needs_confirmation = precision != "precise"
|
||||
source_note = assessment.reason
|
||||
existing = await session.scalar(
|
||||
select(ComputeCenterLocationRecord)
|
||||
.where(ComputeCenterLocationRecord.source == source_value)
|
||||
@@ -790,7 +971,6 @@ async def seed_compute_center_locations_from_source_coords(
|
||||
)
|
||||
if existing:
|
||||
continue
|
||||
metadata = record.extra_data or {}
|
||||
session.add(
|
||||
ComputeCenterLocationRecord(
|
||||
source=source_value,
|
||||
@@ -802,16 +982,16 @@ async def seed_compute_center_locations_from_source_coords(
|
||||
country=coerce_str(get_record_field(record, "country")) or None,
|
||||
latitude=latitude,
|
||||
longitude=longitude,
|
||||
precision="precise",
|
||||
confidence=1.0,
|
||||
precision=precision,
|
||||
confidence=confidence,
|
||||
location_source="source_coordinates",
|
||||
source_note="Seeded from source-provided compute-center coordinates",
|
||||
source_note=source_note,
|
||||
raw_payload={
|
||||
"record_id": getattr(record, "id", None),
|
||||
"source": source_value,
|
||||
"source_id": source_id,
|
||||
},
|
||||
needs_confirmation=False,
|
||||
needs_confirmation=needs_confirmation,
|
||||
verification_status="source_provided",
|
||||
verified_at=None,
|
||||
)
|
||||
|
||||
44
backend/app/services/compute_center_verified_sites.json
Normal file
44
backend/app/services/compute_center_verified_sites.json
Normal file
@@ -0,0 +1,44 @@
|
||||
{
|
||||
"version": 1,
|
||||
"sites": [
|
||||
{
|
||||
"source": "epoch_ai_gpu",
|
||||
"source_ids": ["epoch_jamstec_zettascaler_2_0_gyoukou", "epoch_jamstec_zettascaler_2_2_gyoukou"],
|
||||
"latitude": 35.380924,
|
||||
"longitude": 139.625008,
|
||||
"precision": "site",
|
||||
"location_source": "official_facility_registry",
|
||||
"country": "日本",
|
||||
"city": "Yokohama",
|
||||
"matched_location_name": "JAMSTEC Yokohama Institute for Earth Sciences",
|
||||
"address": "3173-25 Showa-machi, Kanazawa-ku, Yokohama, Kanagawa, Japan",
|
||||
"source_note": "Gyoukou deployment confirmed by JAMSTEC's 2017 release. Campus-level anchor from its official access map center, not a surveyed machine-room or building coordinate.",
|
||||
"verified_at": "2026-09-21T17:06:31Z",
|
||||
"coordinate_basis": "official_access_map_center",
|
||||
"sources": [
|
||||
"https://www.jamstec.go.jp/j/about/press_release/archive/2017/20171114.pdf",
|
||||
"https://www.jamstec.go.jp/e/about/access/yokohama.html"
|
||||
]
|
||||
},
|
||||
{
|
||||
"source": "epoch_ai_gpu",
|
||||
"source_ids": ["epoch_argonne_nl_sophia__formerly_theta_"],
|
||||
"latitude": 41.718109383287235,
|
||||
"longitude": -87.984498648496,
|
||||
"precision": "site",
|
||||
"location_source": "official_facility_registry",
|
||||
"country": "美国",
|
||||
"city": "Argonne",
|
||||
"matched_location_name": "Argonne Leadership Computing Facility",
|
||||
"address": "9700 South Cass Avenue, Building 240, Argonne, IL 60439, USA",
|
||||
"source_note": "Sophia is an ALCF system. Campus-level anchor from ALCF's official visitor map center; the published address identifies Building 240 but the coordinate is not a surveyed building or rack location.",
|
||||
"verified_at": "2026-09-21T17:06:31Z",
|
||||
"coordinate_basis": "official_access_map_center",
|
||||
"sources": [
|
||||
"https://www.alcf.anl.gov/systems/sophia",
|
||||
"https://docs.alcf.anl.gov/sophia/",
|
||||
"https://www.alcf.anl.gov/about/visiting-alcf"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
26
backend/app/services/compute_center_verified_sites.py
Normal file
26
backend/app/services/compute_center_verified_sites.py
Normal file
@@ -0,0 +1,26 @@
|
||||
"""Evidence-backed facility anchors; never match an operator's headquarters by name."""
|
||||
|
||||
import json
|
||||
from copy import deepcopy
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def _site_index() -> dict[tuple[str, str], dict]:
|
||||
payload = json.loads(Path(__file__).with_suffix(".json").read_text(encoding="utf-8"))
|
||||
index = {}
|
||||
for site in payload["sites"]:
|
||||
if not (-90 <= site["latitude"] <= 90 and -180 <= site["longitude"] <= 180):
|
||||
raise ValueError("Invalid verified compute facility coordinate")
|
||||
for source_id in site["source_ids"]:
|
||||
key = (site["source"], source_id)
|
||||
if key in index:
|
||||
raise ValueError("Duplicate verified compute facility identity")
|
||||
index[key] = site
|
||||
return index
|
||||
|
||||
|
||||
def get_verified_compute_site(source: str, source_id: str) -> dict | None:
|
||||
site = _site_index().get((source, source_id))
|
||||
return deepcopy(site) if site else None
|
||||
99
backend/app/services/compute_configuration.py
Normal file
99
backend/app/services/compute_configuration.py
Normal file
@@ -0,0 +1,99 @@
|
||||
"""Database overrides with bundled defaults and atomic, revision-checked publishing."""
|
||||
|
||||
import hashlib
|
||||
from pathlib import Path
|
||||
from uuid import uuid4
|
||||
|
||||
from app.core.request_context import get_request_id
|
||||
from app.models.system_log import AuditLog
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.models.user import User
|
||||
from app.schemas.compute_configuration import ComputeConfiguration
|
||||
from app.services.compute_catalog import COUNTRY_CATALOG
|
||||
from fastapi import HTTPException
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.dialects.postgresql import insert
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
CATEGORY = "compute_comparison"
|
||||
DEFAULT_PATH = Path(__file__).with_name("compute_history_defaults.json")
|
||||
DEFAULT_CONFIGURATION = ComputeConfiguration.model_validate_json(DEFAULT_PATH.read_text())
|
||||
DEFAULT_REVISION = hashlib.sha256(DEFAULT_PATH.read_bytes()).hexdigest()
|
||||
|
||||
|
||||
def default_envelope() -> dict:
|
||||
return {
|
||||
"config": DEFAULT_CONFIGURATION.model_dump(mode="json"),
|
||||
"revision": DEFAULT_REVISION,
|
||||
"is_default": True,
|
||||
}
|
||||
|
||||
|
||||
async def read_configuration(db: AsyncSession) -> dict:
|
||||
record = (
|
||||
await db.execute(select(SystemSetting).where(SystemSetting.category == CATEGORY))
|
||||
).scalar_one_or_none()
|
||||
return dict(record.payload) if record else default_envelope()
|
||||
|
||||
|
||||
def configuration_response(envelope: dict) -> dict:
|
||||
return {
|
||||
**envelope,
|
||||
"countries": [
|
||||
{key: country[key] for key in ("code", "name", "name_en")}
|
||||
for country in COUNTRY_CATALOG
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
async def publish_configuration(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
revision: str,
|
||||
config: ComputeConfiguration,
|
||||
actor: User,
|
||||
reset: bool = False,
|
||||
) -> dict:
|
||||
# The unique category plus row lock also serializes concurrent first writes.
|
||||
await db.execute(
|
||||
insert(SystemSetting)
|
||||
.values(category=CATEGORY, payload=default_envelope())
|
||||
.on_conflict_do_nothing(index_elements=[SystemSetting.category])
|
||||
)
|
||||
record = (
|
||||
await db.execute(
|
||||
select(SystemSetting).where(SystemSetting.category == CATEGORY).with_for_update()
|
||||
)
|
||||
).scalar_one()
|
||||
if record.payload["revision"] != revision:
|
||||
await db.rollback()
|
||||
raise HTTPException(409, "Configuration changed. Reload before saving your edits.")
|
||||
envelope = (
|
||||
default_envelope()
|
||||
if reset
|
||||
else {
|
||||
"config": config.model_dump(mode="json"),
|
||||
"revision": uuid4().hex,
|
||||
"is_default": reset,
|
||||
}
|
||||
)
|
||||
before = record.payload
|
||||
if reset:
|
||||
await db.delete(record)
|
||||
else:
|
||||
record.payload = envelope
|
||||
# Audit and configuration commit together: a published edit must be attributable.
|
||||
db.add(
|
||||
AuditLog(
|
||||
actor_id=actor.id,
|
||||
actor_name=actor.username,
|
||||
action="compute_comparison.reset" if reset else "compute_comparison.update",
|
||||
target_type="system_setting",
|
||||
target_id=CATEGORY,
|
||||
result="success",
|
||||
request_id=get_request_id(),
|
||||
details={"before": before, "after": envelope},
|
||||
)
|
||||
)
|
||||
await db.commit()
|
||||
return envelope
|
||||
122
backend/app/services/compute_history.py
Normal file
122
backend/app/services/compute_history.py
Normal file
@@ -0,0 +1,122 @@
|
||||
"""Versioned, source-backed annual observations for compute comparisons."""
|
||||
|
||||
from copy import deepcopy
|
||||
from datetime import date
|
||||
|
||||
from app.schemas.compute_configuration import ComputeConfiguration
|
||||
from app.services.compute_catalog import (
|
||||
CATALOG,
|
||||
DEFAULT_PARTICIPANTS,
|
||||
PARTICIPANTS,
|
||||
participant_members,
|
||||
)
|
||||
from app.services.compute_configuration import DEFAULT_CONFIGURATION
|
||||
|
||||
|
||||
def _participant_value(
|
||||
members: tuple[str, ...], observations: dict[str, float | int], mode: str
|
||||
) -> tuple[float | int | None, dict]:
|
||||
covered = [member for member in members if member in observations]
|
||||
missing = [member for member in members if member not in observations]
|
||||
coverage = {
|
||||
"status": "complete" if not missing else "incomplete",
|
||||
"required_members": list(members),
|
||||
"covered_members": covered,
|
||||
"missing_members": missing,
|
||||
}
|
||||
if missing:
|
||||
return None, coverage
|
||||
if mode == "bilateral_share":
|
||||
if (
|
||||
not set(members).issubset({"CN", "US"})
|
||||
or not {"CN", "US"}.issubset(observations)
|
||||
or observations["US"] + observations["CN"] <= 0
|
||||
):
|
||||
coverage.update(
|
||||
{
|
||||
"status": "incompatible",
|
||||
"reason": "bilateral_share_requires_CN_and_US_observations",
|
||||
}
|
||||
)
|
||||
return None, coverage
|
||||
return (
|
||||
round(
|
||||
sum(observations[member] for member in members)
|
||||
/ (observations["US"] + observations["CN"])
|
||||
* 100,
|
||||
2,
|
||||
),
|
||||
coverage,
|
||||
)
|
||||
return sum(observations[member] for member in members), coverage
|
||||
|
||||
|
||||
def compute_history(
|
||||
countries: list[str],
|
||||
mode: str = "global_share",
|
||||
config: ComputeConfiguration | None = None,
|
||||
) -> dict:
|
||||
"""Aggregate only fully covered, same-year compatible observations."""
|
||||
config = config if config is not None else DEFAULT_CONFIGURATION
|
||||
metric = "global_share" if mode == "bilateral_share" else mode
|
||||
annual = {row.year: row for row in config.observations if row.metric == metric}
|
||||
gaps = {row.year: row for row in config.innovation}
|
||||
observed_years = [row.year for row in config.observations] + list(gaps)
|
||||
years = tuple(
|
||||
range(
|
||||
min(observed_years, default=date.today().year),
|
||||
max([date.today().year, *observed_years]) + 1,
|
||||
)
|
||||
)
|
||||
members_by_participant = participant_members(countries)
|
||||
rows = []
|
||||
for year in years:
|
||||
record = annual.get(year)
|
||||
observations = record.values if record else {}
|
||||
values, coverage = {}, {}
|
||||
for code, members in members_by_participant.items():
|
||||
values[code], coverage[code] = _participant_value(members, observations, mode)
|
||||
evidence_id = record.source_id if record else None
|
||||
rows.append(
|
||||
{"year": year, "values": values, "coverage": coverage, "evidence_id": evidence_id}
|
||||
)
|
||||
|
||||
selected_members = {member for members in members_by_participant.values() for member in members}
|
||||
has_innovation_scope = {"CN", "US"}.issubset(selected_members)
|
||||
return {
|
||||
"version": f"compute-history-{config.reviewed_at}",
|
||||
"provenance": "curated_reports",
|
||||
"reviewed_at": config.reviewed_at.isoformat(),
|
||||
"time_basis": "retrospective",
|
||||
"years": years,
|
||||
"mode": mode,
|
||||
"unit": "EFlop/s (FP32)" if mode == "absolute" else "%",
|
||||
"denominator": {"global_share": "global", "bilateral_share": "US+CN", "absolute": None}[
|
||||
mode
|
||||
],
|
||||
"catalog": deepcopy(CATALOG),
|
||||
"default_participants": list(DEFAULT_PARTICIPANTS),
|
||||
"countries": [deepcopy(PARTICIPANTS[code]) for code in countries],
|
||||
"compute": rows,
|
||||
"innovation": [
|
||||
{
|
||||
"year": year,
|
||||
"value": gaps[year].value if year in gaps and has_innovation_scope else None,
|
||||
"evidence_id": gaps[year].source_id
|
||||
if year in gaps and has_innovation_scope
|
||||
else None,
|
||||
}
|
||||
for year in years
|
||||
],
|
||||
"evidence": {
|
||||
source.id: source.model_dump(mode="json", exclude={"id"}) for source in config.sources
|
||||
},
|
||||
"innovation_scope": ["US", "CN"],
|
||||
"limitations": [
|
||||
"公开设施样本不代表全国总算力;FP64、FP16 与 FP32 不可直接相加。",
|
||||
"区域值只在全部成员有同年、同指标、可比较观测时汇总;缺失成员时返回 null。",
|
||||
"全球份额不从中美或其他局部成员反推剩余地区。",
|
||||
"年度资料按已配置报告引用展示,观测年与报告年份分列;跨年方法可比性仍需核验。",
|
||||
"创新分差为美国减中国;只有分差,不反推单国得分,不证明算力与创新的因果关系。",
|
||||
],
|
||||
}
|
||||
125
backend/app/services/compute_history_defaults.json
Normal file
125
backend/app/services/compute_history_defaults.json
Normal file
@@ -0,0 +1,125 @@
|
||||
{
|
||||
"reviewed_at": "2026-09-22",
|
||||
"sources": [
|
||||
{
|
||||
"id": "caict-2021",
|
||||
"title": "中国算力发展指数白皮书(2021)",
|
||||
"url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202109/P020210918521091309950.pdf",
|
||||
"publisher": "中国信息通信研究院",
|
||||
"report_year": 2021,
|
||||
"published_at": null,
|
||||
"locator": "正文 11 页图 4(2020 年全球份额)"
|
||||
},
|
||||
{
|
||||
"id": "caict-2022",
|
||||
"title": "中国算力发展指数白皮书(2022)",
|
||||
"url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202211/P020221105727522653499.pdf",
|
||||
"publisher": "中国信息通信研究院",
|
||||
"report_year": 2022,
|
||||
"published_at": null,
|
||||
"locator": "正文 13 页图 5(2021 年全球份额)"
|
||||
},
|
||||
{
|
||||
"id": "caict-2023",
|
||||
"title": "中国算力发展指数白皮书(2023)",
|
||||
"url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202309/P020240326630458153765.pdf",
|
||||
"publisher": "中国信息通信研究院",
|
||||
"report_year": 2023,
|
||||
"published_at": null,
|
||||
"locator": "正文 13 页图 5(2022 年全球份额);16 页(中国总规模)"
|
||||
},
|
||||
{
|
||||
"id": "caict-2024",
|
||||
"title": "先进计算暨算力发展指数蓝皮书(2024年)",
|
||||
"url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202501/P020250117503621662777.pdf",
|
||||
"publisher": "中国信息通信研究院",
|
||||
"report_year": 2024,
|
||||
"published_at": null,
|
||||
"locator": "正文 10 页图 3(2023 年全球份额);12 页(中国总规模)"
|
||||
},
|
||||
{
|
||||
"id": "innovation-2026",
|
||||
"title": "全球人工智能创新指数报告 2026:科技日报报道",
|
||||
"url": "https://finance.sina.com.cn/tech/roll/2026-07-17/doc-iniickaz8224252.shtml",
|
||||
"publisher": "科技日报(新浪转载)",
|
||||
"report_year": 2026,
|
||||
"published_at": "2026-07-17",
|
||||
"locator": "2023–2025 年美国减中国的指数分差"
|
||||
}
|
||||
],
|
||||
"observations": [
|
||||
{
|
||||
"year": 2020,
|
||||
"metric": "global_share",
|
||||
"source_id": "caict-2021",
|
||||
"values": {
|
||||
"US": 36,
|
||||
"CN": 31,
|
||||
"JP": 6
|
||||
}
|
||||
},
|
||||
{
|
||||
"year": 2021,
|
||||
"metric": "global_share",
|
||||
"source_id": "caict-2022",
|
||||
"values": {
|
||||
"US": 34,
|
||||
"CN": 33,
|
||||
"JP": 5
|
||||
}
|
||||
},
|
||||
{
|
||||
"year": 2022,
|
||||
"metric": "global_share",
|
||||
"source_id": "caict-2023",
|
||||
"values": {
|
||||
"US": 34,
|
||||
"CN": 33,
|
||||
"JP": 4
|
||||
}
|
||||
},
|
||||
{
|
||||
"year": 2023,
|
||||
"metric": "global_share",
|
||||
"source_id": "caict-2024",
|
||||
"values": {
|
||||
"US": 41,
|
||||
"CN": 31,
|
||||
"JP": 4
|
||||
}
|
||||
},
|
||||
{
|
||||
"year": 2022,
|
||||
"metric": "absolute",
|
||||
"source_id": "caict-2023",
|
||||
"values": {
|
||||
"CN": 302
|
||||
}
|
||||
},
|
||||
{
|
||||
"year": 2023,
|
||||
"metric": "absolute",
|
||||
"source_id": "caict-2024",
|
||||
"values": {
|
||||
"CN": 435
|
||||
}
|
||||
}
|
||||
],
|
||||
"innovation": [
|
||||
{
|
||||
"year": 2023,
|
||||
"value": 22.02,
|
||||
"source_id": "innovation-2026"
|
||||
},
|
||||
{
|
||||
"year": 2024,
|
||||
"value": 19.96,
|
||||
"source_id": "innovation-2026"
|
||||
},
|
||||
{
|
||||
"year": 2025,
|
||||
"value": 17.95,
|
||||
"source_id": "innovation-2026"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -6,8 +6,10 @@ import asyncio
|
||||
import hashlib
|
||||
import json
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from datetime import UTC, datetime
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from uuid import uuid4
|
||||
@@ -25,8 +27,11 @@ BOUNDARY_MANIFEST_PATH = BOUNDARY_OUTPUT_DIR / "manifest.json"
|
||||
PMTILES_ARTIFACT_PATH = (
|
||||
REPO_ROOT / "frontend/public/earth/data/boundaries/earth-boundaries-china-pov-v1.pmtiles"
|
||||
)
|
||||
LEGACY_GEOJSON_PATH = REPO_ROOT / "frontend/public/earth/data/countries-admin0.min.geojson"
|
||||
POLICY_FALLBACK_PATH = (
|
||||
REPO_ROOT / "frontend/public/earth/data/countries-admin0.china-pov-v1.geojson"
|
||||
)
|
||||
POV_POLICY_PATH = REPO_ROOT / "config/earth-boundary-pov-policy.china-v1.json"
|
||||
POV_GEOMETRY_PATH = REPO_ROOT / "resources/earth/boundary-pov-geometry.china-v1.json"
|
||||
LOCAL_CONFIG_PATH = REPO_ROOT / "config/earth-boundary-sources.local.json"
|
||||
EXAMPLE_CONFIG_PATH = REPO_ROOT / "config/earth-boundary-sources.example.json"
|
||||
|
||||
@@ -150,6 +155,21 @@ def _stable_json_hash(payload: Any) -> str:
|
||||
return hashlib.sha256(encoded.encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
@lru_cache(maxsize=16)
|
||||
def _read_versioned_asset_cached(
|
||||
path: Path, mtime_ns: int, size_bytes: int
|
||||
) -> tuple[dict[str, Any], str]:
|
||||
payload = _read_json(path)
|
||||
return payload, _stable_json_hash(payload)
|
||||
|
||||
|
||||
def _versioned_asset(path: Path) -> tuple[dict[str, Any], str]:
|
||||
if not path.exists():
|
||||
return {}, ""
|
||||
stat = path.stat()
|
||||
return _read_versioned_asset_cached(path, stat.st_mtime_ns, stat.st_size)
|
||||
|
||||
|
||||
def _artifact_extension(endpoint: str, content_type: str, payload: bytes) -> str:
|
||||
suffix = Path(endpoint.split("?", 1)[0]).suffix.lower()
|
||||
if suffix in {".json", ".geojson", ".zip", ".pbf"}:
|
||||
@@ -176,62 +196,37 @@ def _directory_stats(path: Path) -> dict[str, int]:
|
||||
return {"file_count": len(files), "size_bytes": sum(item.stat().st_size for item in files)}
|
||||
|
||||
|
||||
def _load_source_feature_collection(source: dict[str, Any]) -> dict[str, Any]:
|
||||
path = REPO_ROOT / source["path"]
|
||||
payload = _read_json(path)
|
||||
features = payload.get("features") if isinstance(payload, dict) else None
|
||||
return {
|
||||
"type": "FeatureCollection",
|
||||
"features": features if isinstance(features, list) else [],
|
||||
}
|
||||
|
||||
|
||||
def _write_high_precision_geojson_manifest(
|
||||
sources: list[dict[str, Any]],
|
||||
build_input_hash: str,
|
||||
missing_tools: list[str],
|
||||
) -> dict[str, Any]:
|
||||
BOUNDARY_OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
|
||||
admin0 = next(source for source in sources if source["kind"] == "admin0-boundaries")
|
||||
coastline = next(source for source in sources if source["kind"] == "coastline")
|
||||
claim_lines = next(source for source in sources if source["kind"] == "claim-lines")
|
||||
|
||||
admin0_payload = _load_source_feature_collection(admin0)
|
||||
coastline_payload = _load_source_feature_collection(coastline)
|
||||
claim_payload = _load_source_feature_collection(claim_lines)
|
||||
for feature in coastline_payload["features"]:
|
||||
props = feature.setdefault("properties", {})
|
||||
if isinstance(props, dict):
|
||||
props["PLANET_LAYER"] = "coastline"
|
||||
|
||||
base_payload = {
|
||||
"type": "FeatureCollection",
|
||||
"features": [*admin0_payload["features"], *coastline_payload["features"]],
|
||||
}
|
||||
base_path = BOUNDARY_OUTPUT_DIR / "earth-boundaries-high-precision-base.geojson"
|
||||
hover_path = BOUNDARY_OUTPUT_DIR / "earth-boundaries-high-precision-hover.geojson"
|
||||
claim_path = BOUNDARY_OUTPUT_DIR / "earth-boundaries-high-precision-claims.geojson"
|
||||
_write_json(base_path, base_payload)
|
||||
_write_json(hover_path, admin0_payload)
|
||||
_write_json(claim_path, claim_payload)
|
||||
|
||||
manifest = {
|
||||
"version": "natural-earth-v1",
|
||||
"builtAt": _utc_now_iso(),
|
||||
"tileProvider": "geojson-high-precision",
|
||||
"format": "geojson-directory",
|
||||
"buildInputHash": build_input_hash,
|
||||
"base": base_path.name,
|
||||
"hoverIndex": hover_path.name,
|
||||
"claimLine": claim_path.name,
|
||||
"sourceFeatureCount": {
|
||||
"admin0": len(admin0_payload["features"]),
|
||||
"coastline": len(coastline_payload["features"]),
|
||||
"claimLines": len(claim_payload["features"]),
|
||||
},
|
||||
"pmtiles": None,
|
||||
"missingTools": missing_tools,
|
||||
}
|
||||
result = subprocess.run(
|
||||
[
|
||||
sys.executable,
|
||||
str(Path(__file__).resolve().parents[3] / "scripts/build_earth_boundary_china_pov_geojson.py"),
|
||||
"--admin0-source", admin0["path"],
|
||||
"--coastline-source", coastline["path"],
|
||||
"--claims-source", claim_lines["path"],
|
||||
"--output-dir", _relative(BOUNDARY_OUTPUT_DIR),
|
||||
"--build-input-hash", build_input_hash,
|
||||
"--pov-policy", _relative(POV_POLICY_PATH),
|
||||
"--pov-geometry", str(POV_GEOMETRY_PATH),
|
||||
],
|
||||
cwd=REPO_ROOT,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
if result.returncode:
|
||||
raise EarthBoundaryBuildError(
|
||||
result.stderr.strip() or "Earth boundary policy build failed",
|
||||
code="policy_build_failed",
|
||||
)
|
||||
manifest = _read_json(BOUNDARY_MANIFEST_PATH)
|
||||
manifest["missingTools"] = missing_tools
|
||||
_write_json(BOUNDARY_MANIFEST_PATH, manifest)
|
||||
return manifest
|
||||
|
||||
@@ -291,6 +286,7 @@ def _build_input_hash(source_manifest: dict[str, Any]) -> str:
|
||||
for source in source_manifest.get("sources", [])
|
||||
],
|
||||
"pov_policy": source_manifest.get("povPolicy"),
|
||||
"pov_geometry_sha256": _versioned_asset(POV_GEOMETRY_PATH)[1],
|
||||
"build_config": BUILD_CONFIG,
|
||||
}
|
||||
)
|
||||
@@ -301,7 +297,47 @@ def _has_current_artifacts(boundary_manifest: dict[str, Any], build_input_hash:
|
||||
bool(boundary_manifest)
|
||||
and boundary_manifest.get("buildInputHash") == build_input_hash
|
||||
and boundary_manifest.get("tileProvider") == "pmtiles-mvt"
|
||||
and _policy_verified(boundary_manifest)
|
||||
and PMTILES_ARTIFACT_PATH.exists()
|
||||
and _pmtiles_hover_exists(boundary_manifest)
|
||||
)
|
||||
|
||||
|
||||
def _policy_verified(manifest: dict[str, Any]) -> bool:
|
||||
policy, policy_hash = _versioned_asset(POV_POLICY_PATH)
|
||||
geometry, geometry_hash = _versioned_asset(POV_GEOMETRY_PATH)
|
||||
coverage = manifest.get("povPolicy") or {}
|
||||
return (
|
||||
policy.get("profile") == "china-pov-v1"
|
||||
and coverage.get("profile") == policy["profile"]
|
||||
and coverage.get("sha256") == policy_hash
|
||||
and coverage.get("reviewedGeometrySourceCommit")
|
||||
== geometry.get("sourceCommit")
|
||||
and coverage.get("geometrySha256") == geometry_hash
|
||||
and coverage.get("applied") is True
|
||||
and {"china-taiwan-penghu", "china-hong-kong", "china-macao", "kosovo"}
|
||||
<= set(coverage.get("appliedRuleIds") or [])
|
||||
)
|
||||
|
||||
|
||||
def _geojson_artifacts_exist(manifest: dict[str, Any]) -> bool:
|
||||
paths = [manifest.get("base"), manifest.get("hoverIndex")]
|
||||
if manifest.get("chinaClaims", {}).get("available"):
|
||||
paths.append(manifest.get("claimLine"))
|
||||
for asset in paths:
|
||||
if not isinstance(asset, str) or Path(asset).name != asset:
|
||||
return False
|
||||
if not (BOUNDARY_OUTPUT_DIR / asset).is_file():
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _pmtiles_hover_exists(manifest: dict[str, Any]) -> bool:
|
||||
hover = manifest.get("hoverIndex")
|
||||
return (
|
||||
isinstance(hover, str)
|
||||
and Path(hover).name == hover
|
||||
and (BOUNDARY_OUTPUT_DIR / hover).is_file()
|
||||
)
|
||||
|
||||
|
||||
@@ -313,24 +349,34 @@ def get_boundary_status() -> dict[str, Any]:
|
||||
manifest_exists = BOUNDARY_MANIFEST_PATH.exists()
|
||||
high_precision_ready = (
|
||||
manifest_exists
|
||||
and _policy_verified(boundary_manifest)
|
||||
and (
|
||||
(
|
||||
boundary_manifest.get("tileProvider") == "pmtiles-mvt"
|
||||
and pmtiles_exists
|
||||
and _pmtiles_hover_exists(boundary_manifest)
|
||||
)
|
||||
or (
|
||||
boundary_manifest.get("tileProvider") == "geojson-high-precision"
|
||||
and _geojson_artifacts_exist(boundary_manifest)
|
||||
)
|
||||
or boundary_manifest.get("tileProvider") == "geojson-high-precision"
|
||||
)
|
||||
)
|
||||
legacy_exists = LEGACY_GEOJSON_PATH.exists()
|
||||
legacy_exists = POLICY_FALLBACK_PATH.exists() and _policy_verified(
|
||||
_versioned_asset(POLICY_FALLBACK_PATH)[0]
|
||||
)
|
||||
provider = (
|
||||
boundary_manifest.get("tileProvider")
|
||||
if high_precision_ready
|
||||
else "legacy-geojson" if legacy_exists else "missing"
|
||||
else "policy-geojson-fallback" if legacy_exists else "missing"
|
||||
)
|
||||
return {
|
||||
"provider": provider,
|
||||
"high_precision_ready": high_precision_ready,
|
||||
"fallback_available": legacy_exists,
|
||||
"policy_publishable": bool(
|
||||
high_precision_ready and boundary_manifest.get("povPolicy", {}).get("publicationReady")
|
||||
),
|
||||
"config_source": config_source,
|
||||
"config_path": _relative(LOCAL_CONFIG_PATH),
|
||||
"config_exists": LOCAL_CONFIG_PATH.exists(),
|
||||
@@ -347,6 +393,8 @@ def get_boundary_status() -> dict[str, Any]:
|
||||
"tileProvider": boundary_manifest.get("tileProvider"),
|
||||
"buildInputHash": boundary_manifest.get("buildInputHash"),
|
||||
"builtAt": boundary_manifest.get("builtAt"),
|
||||
"povPolicy": boundary_manifest.get("povPolicy"),
|
||||
"publicationReady": boundary_manifest.get("povPolicy", {}).get("publicationReady", False),
|
||||
},
|
||||
"pmtiles": {
|
||||
"path": _relative(PMTILES_ARTIFACT_PATH),
|
||||
@@ -354,9 +402,9 @@ def get_boundary_status() -> dict[str, Any]:
|
||||
"size_bytes": PMTILES_ARTIFACT_PATH.stat().st_size if pmtiles_exists else 0,
|
||||
},
|
||||
"legacy": {
|
||||
"path": _relative(LEGACY_GEOJSON_PATH),
|
||||
"path": _relative(POLICY_FALLBACK_PATH),
|
||||
"exists": legacy_exists,
|
||||
"size_bytes": LEGACY_GEOJSON_PATH.stat().st_size if legacy_exists else 0,
|
||||
"size_bytes": POLICY_FALLBACK_PATH.stat().st_size if legacy_exists else 0,
|
||||
},
|
||||
"source_manifest": {
|
||||
"path": _relative(SOURCE_MANIFEST_PATH),
|
||||
@@ -577,6 +625,8 @@ async def build_boundary_assets(progress_callback: Any = None) -> dict[str, Any]
|
||||
build_input_hash,
|
||||
"--pov-policy",
|
||||
_relative(POV_POLICY_PATH),
|
||||
"--pov-geometry",
|
||||
str(POV_GEOMETRY_PATH),
|
||||
]
|
||||
)
|
||||
if progress_callback:
|
||||
|
||||
@@ -56,6 +56,8 @@ def build_earth_update_from_db_payload(payload: dict[str, Any]) -> dict[str, Any
|
||||
source_has_adapter = bool(get_earth_update_layers_for_source(source))
|
||||
effective_source = source if source_has_adapter else (table_name if table_name else source)
|
||||
operation = payload.get("operation")
|
||||
if "vessels" in layers and operation in {"DELETE", "TRUNCATE"}:
|
||||
refresh_strategy = "reload"
|
||||
update: dict[str, Any] = {
|
||||
"event": "earth.layer.changed",
|
||||
"action": "database_changed",
|
||||
@@ -113,6 +115,8 @@ class PendingEarthDbChange:
|
||||
operation = payload.get("operation")
|
||||
if operation:
|
||||
self.operations.add(str(operation))
|
||||
if "vessels" in self.layers and operation in {"DELETE", "TRUNCATE"}:
|
||||
self.refresh_strategy = "reload"
|
||||
entity_keys = payload.get("entity_keys")
|
||||
if not isinstance(entity_keys, list):
|
||||
entity_key = payload.get("entity_key")
|
||||
@@ -176,6 +180,17 @@ class EarthDbChangeDispatcher:
|
||||
if not update:
|
||||
return False
|
||||
|
||||
if payload.get("table") == "vessel_current_state" and payload.get("operation") == "DELETE":
|
||||
from app.core.websocket.broadcaster import broadcaster
|
||||
|
||||
keys = payload.get("entity_keys")
|
||||
if not isinstance(keys, list):
|
||||
keys = [payload.get("entity_key")]
|
||||
broadcaster.enqueue_vessel_update({
|
||||
"action": "remove",
|
||||
"vessels": [{"mmsi": key} for key in keys if key is not None],
|
||||
})
|
||||
|
||||
source = update["source"]
|
||||
pending = self._pending.get(source)
|
||||
if pending is None:
|
||||
|
||||
@@ -25,6 +25,7 @@ EARTH_LAYER_ADAPTERS: tuple[EarthLayerAdapter, ...] = (
|
||||
layers=("vessels",),
|
||||
cache_patterns=("vessels*", "summary*"),
|
||||
derived_models=("ais_raw_observations", "ais_conflict_records", "ais_source_health"),
|
||||
refresh_strategy="delta",
|
||||
),
|
||||
EarthLayerAdapter(
|
||||
sources=frozenset(
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import codecs
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from email.utils import parsedate_to_datetime
|
||||
@@ -1800,6 +1801,30 @@ def _extract_item_text(element: ET.Element, *names: str) -> str:
|
||||
return ""
|
||||
|
||||
|
||||
def _decode_feed_response(response: httpx.Response) -> str:
|
||||
content = response.content
|
||||
for marker, encoding in (
|
||||
(codecs.BOM_UTF32_LE, "utf-32"),
|
||||
(codecs.BOM_UTF32_BE, "utf-32"),
|
||||
(codecs.BOM_UTF16_LE, "utf-16"),
|
||||
(codecs.BOM_UTF16_BE, "utf-16"),
|
||||
(codecs.BOM_UTF8, "utf-8-sig"),
|
||||
):
|
||||
if content.startswith(marker):
|
||||
return content.decode(encoding)
|
||||
|
||||
declaration = re.match(
|
||||
br"\s*<\?xml\s+[^?]*\bencoding\s*=\s*['\"]([^'\"]+)['\"]",
|
||||
content,
|
||||
re.IGNORECASE,
|
||||
)
|
||||
encoding = response.charset_encoding or (
|
||||
declaration.group(1).decode("ascii") if declaration else "utf-8"
|
||||
)
|
||||
# Decode strictly before parsing: response.text silently replaces non-UTF-8 bytes.
|
||||
return content.decode(encoding)
|
||||
|
||||
|
||||
def _parse_feed_entries(
|
||||
xml_text: str,
|
||||
source: NewsFeedSource,
|
||||
@@ -2453,7 +2478,9 @@ async def _fetch_single_feed_url(
|
||||
feed=feed,
|
||||
)
|
||||
try:
|
||||
items = _parse_feed_entries(response.text, source, feed=feed, config_payload=config_payload)
|
||||
items = _parse_feed_entries(
|
||||
_decode_feed_response(response), source, feed=feed, config_payload=config_payload
|
||||
)
|
||||
except Exception as exc:
|
||||
error, status = _format_source_fetch_error(
|
||||
status_code=status_code,
|
||||
@@ -2461,7 +2488,7 @@ async def _fetch_single_feed_url(
|
||||
body=response.text,
|
||||
parsed_count=0,
|
||||
)
|
||||
if error is None:
|
||||
if status == "empty":
|
||||
error = str(exc)
|
||||
status = "format_error"
|
||||
return source, [], error, _source_health_result(
|
||||
|
||||
198
backend/app/services/epoch_location.py
Normal file
198
backend/app/services/epoch_location.py
Normal file
@@ -0,0 +1,198 @@
|
||||
"""Interpret the geographic grain of Epoch AI's official ``Location`` field."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
import math
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
from app.core.countries import normalize_country
|
||||
|
||||
|
||||
_COUNTRY_LABEL_OVERRIDES = {
|
||||
"korea republic of": "韩国",
|
||||
}
|
||||
_REGION_ONLY_LABELS = {
|
||||
"europe",
|
||||
"iowa",
|
||||
"maryland",
|
||||
"new mexico",
|
||||
"texas",
|
||||
"the nordics",
|
||||
"virginia",
|
||||
"washington",
|
||||
}
|
||||
_UNLOCATABLE_LABELS = {'"on prem" for tesla', "on prem for tesla"}
|
||||
_HEMISPHERE_COORDINATE_PATTERN = re.compile(
|
||||
r"(?P<degrees>\d{1,3})(?:[°º]|\s)\s*(?P<minutes>\d{1,2})(?:['′]|\s)"
|
||||
r"(?P<seconds>\d{1,2}(?:\.\d+)?)?[\"″]?\s*(?P<hemisphere>[NSEW])",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_LABELED_COORDINATE_PAIR_PATTERN = re.compile(
|
||||
r"\b(?:lat|latitude)\b\s*[:=]?\s*(?P<latitude>-?\d{1,2}(?:\.\d+)?)"
|
||||
r".{0,40}\b(?:lon|lng|longitude)\b\s*[:=]?\s*"
|
||||
r"(?P<longitude>-?\d{1,3}(?:\.\d+)?)",
|
||||
re.IGNORECASE | re.DOTALL,
|
||||
)
|
||||
_DECIMAL_COORDINATE_PAIR_PATTERN = re.compile(
|
||||
r"^\s*[\[(]?\s*(-?\d{1,2}\.\d+)\s*[,;]\s*"
|
||||
r"(-?\d{1,3}\.\d+)\s*[\])]?\s*$"
|
||||
)
|
||||
_NUMBERED_STREET_PATTERN = re.compile(
|
||||
r"(?:\b(?:no\.?\s*)?\d+[a-z]?\s+(?:[\w.'’\-]+\s+){0,6}"
|
||||
r"(?:road|rd|street|st|avenue|ave|boulevard|blvd|drive|dr|lane|ln|"
|
||||
r"highway|hwy|way|parkway|pkwy|chome)\b|"
|
||||
r"\b(?:via|rue|calle|plaça|strasse|straße)\b[^,\n]{0,80},?\s*\d)",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_POSTAL_CODE_PATTERN = re.compile(
|
||||
r"(?:\b\d{5}(?:-\d{4})?\b|\b\d{3}-\d{4}\b|"
|
||||
r"\b[A-Z]{1,2}\d[A-Z\d]?\s*\d[A-Z]{2}\b)",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_SITE_CONTEXT_PATTERN = re.compile(
|
||||
r"\b(?:campus|building|cluster|data\s+cent(?:er|re)|facility|institute|"
|
||||
r"laborator(?:y|ies)|science\s+park|supercomputing\s+cent(?:er|re))\b",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_COORDINATE_CUE_PATTERN = re.compile(
|
||||
r"(?:\b(?:lat|latitude|lon|lng|longitude)\b|[°º].*[NSEW])",
|
||||
re.IGNORECASE | re.DOTALL,
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class EpochLocationAssessment:
|
||||
precision: str
|
||||
reason: str
|
||||
|
||||
@property
|
||||
def is_renderable(self) -> bool:
|
||||
return self.precision in {"precise", "site", "city"}
|
||||
|
||||
|
||||
def _normalized_label(value: Any) -> str:
|
||||
return re.sub(r"[^a-z0-9]+", " ", str(value or "").casefold()).strip()
|
||||
|
||||
|
||||
def _country_from_label(value: Any) -> str | None:
|
||||
return normalize_country(value) or _COUNTRY_LABEL_OVERRIDES.get(
|
||||
_normalized_label(value)
|
||||
)
|
||||
|
||||
|
||||
def _coordinate(mapping: dict[str, Any], keys: tuple[str, ...], bound: float) -> float | None:
|
||||
raw = next((mapping.get(key) for key in keys if mapping.get(key) not in (None, "")), None)
|
||||
if isinstance(raw, bool):
|
||||
return None
|
||||
try:
|
||||
number = float(raw)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
return number if math.isfinite(number) and abs(number) <= bound else None
|
||||
|
||||
|
||||
def _has_explicit_coordinate_pair(text: str) -> bool:
|
||||
hemispheres = {}
|
||||
for match in _HEMISPHERE_COORDINATE_PATTERN.finditer(text):
|
||||
hemisphere = match.group("hemisphere").upper()
|
||||
degrees = int(match.group("degrees"))
|
||||
minutes = int(match.group("minutes"))
|
||||
seconds = float(match.group("seconds") or 0)
|
||||
bound = 90 if hemisphere in {"N", "S"} else 180
|
||||
if degrees <= bound and minutes < 60 and seconds < 60:
|
||||
hemispheres[hemisphere] = degrees
|
||||
if ({"N", "S"} & hemispheres.keys()) and ({"E", "W"} & hemispheres.keys()):
|
||||
return True
|
||||
labeled_pair = _LABELED_COORDINATE_PAIR_PATTERN.search(text)
|
||||
if labeled_pair:
|
||||
latitude = float(labeled_pair.group("latitude"))
|
||||
longitude = float(labeled_pair.group("longitude"))
|
||||
return abs(latitude) <= 90 and abs(longitude) <= 180
|
||||
decimal_pair = _DECIMAL_COORDINATE_PAIR_PATTERN.match(text)
|
||||
if not decimal_pair:
|
||||
return False
|
||||
latitude, longitude = map(float, decimal_pair.groups())
|
||||
return abs(latitude) <= 90 and abs(longitude) <= 180
|
||||
|
||||
|
||||
def _has_address_evidence(text: str) -> bool:
|
||||
if _NUMBERED_STREET_PATTERN.search(text):
|
||||
return True
|
||||
return bool(
|
||||
_POSTAL_CODE_PATTERN.search(text)
|
||||
and ("\n" in text or text.count(",") >= 2)
|
||||
and _SITE_CONTEXT_PATTERN.search(text)
|
||||
)
|
||||
|
||||
|
||||
def assess_epoch_location(
|
||||
location: Any,
|
||||
*,
|
||||
country: Any = None,
|
||||
raw_country: Any = None,
|
||||
) -> EpochLocationAssessment:
|
||||
"""Classify source coordinates without upgrading their source text.
|
||||
|
||||
Epoch's CSV supplies coordinates even when ``Location`` contains only a
|
||||
country. Those coordinates are useful source evidence, but are not a
|
||||
facility point and must not enter the renderable location pipeline.
|
||||
"""
|
||||
if isinstance(location, dict):
|
||||
latitude = _coordinate(location, ("lat", "latitude"), 90)
|
||||
longitude = _coordinate(location, ("lon", "lng", "longitude"), 180)
|
||||
has_coordinates = latitude is not None and longitude is not None
|
||||
return EpochLocationAssessment(
|
||||
"precise" if has_coordinates else "unknown",
|
||||
"complete explicit coordinate object"
|
||||
if has_coordinates
|
||||
else "coordinate object lacks a complete valid latitude/longitude pair",
|
||||
)
|
||||
|
||||
text = str(location or "").strip()
|
||||
if not text:
|
||||
return EpochLocationAssessment("unknown", "Epoch Location is empty")
|
||||
|
||||
normalized = _normalized_label(text)
|
||||
normalized_country = normalize_country(country)
|
||||
source_country = normalize_country(raw_country)
|
||||
location_country = _country_from_label(text)
|
||||
if location_country or (
|
||||
raw_country and normalized == _normalized_label(raw_country)
|
||||
):
|
||||
matched = location_country or source_country or normalized_country
|
||||
return EpochLocationAssessment(
|
||||
"country",
|
||||
f"Epoch Location identifies country only: {matched or text}",
|
||||
)
|
||||
if normalized in _REGION_ONLY_LABELS:
|
||||
return EpochLocationAssessment(
|
||||
"region", f"Epoch Location identifies region/state only: {text}"
|
||||
)
|
||||
comma_parts = [part.strip() for part in text.split(",") if part.strip()]
|
||||
if (
|
||||
len(comma_parts) == 2
|
||||
and _normalized_label(comma_parts[0]) in _REGION_ONLY_LABELS
|
||||
and _country_from_label(comma_parts[1])
|
||||
):
|
||||
return EpochLocationAssessment(
|
||||
"region", f"Epoch Location identifies region/state only: {text}"
|
||||
)
|
||||
if normalized in _UNLOCATABLE_LABELS:
|
||||
return EpochLocationAssessment("unknown", "Epoch Location has no geographic place")
|
||||
if normalized.isdigit():
|
||||
return EpochLocationAssessment("unknown", "Epoch Location is only a numeric label")
|
||||
if _has_explicit_coordinate_pair(text):
|
||||
return EpochLocationAssessment(
|
||||
"precise", "Epoch Location contains a complete valid coordinate pair"
|
||||
)
|
||||
if _COORDINATE_CUE_PATTERN.search(text):
|
||||
return EpochLocationAssessment(
|
||||
"unknown", "Epoch Location contains an incomplete or invalid coordinate pair"
|
||||
)
|
||||
if _has_address_evidence(text):
|
||||
return EpochLocationAssessment("precise", "Epoch Location contains an address")
|
||||
if "\n" in text or _SITE_CONTEXT_PATTERN.search(text):
|
||||
return EpochLocationAssessment("site", "Epoch Location names a facility or campus")
|
||||
return EpochLocationAssessment("city", "Epoch Location identifies a locality")
|
||||
186
backend/app/services/llm_model_catalog.py
Normal file
186
backend/app/services/llm_model_catalog.py
Normal file
@@ -0,0 +1,186 @@
|
||||
"""Authenticated model discovery shared by refresh and connection checks."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from urllib.parse import urlsplit, urlunsplit
|
||||
|
||||
import httpx
|
||||
|
||||
CATALOG_TIMEOUT_SECONDS = 30
|
||||
CATALOG_MAX_PAGES = 100
|
||||
CATALOG_PAGE_SIZE = 100
|
||||
CATALOG_REQUEST_ATTEMPTS = 2
|
||||
CATALOG_RETRY_DELAY_SECONDS = 0.2
|
||||
SUPPORTED_PROVIDER_APIS = {
|
||||
"openai-completions",
|
||||
"openai-responses",
|
||||
"anthropic-messages",
|
||||
"ollama-generate",
|
||||
}
|
||||
|
||||
|
||||
class LLMProviderCatalogError(RuntimeError):
|
||||
"""A safe, user-facing catalog failure with no upstream response or credentials."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ModelCatalog:
|
||||
url: str
|
||||
models: list[str]
|
||||
|
||||
|
||||
def model_catalog_url(provider: str, base_url: str, provider_api: str) -> str:
|
||||
parts = urlsplit(base_url.strip())
|
||||
if parts.scheme not in {"http", "https"} or not parts.hostname:
|
||||
raise LLMProviderCatalogError("请填写有效的 HTTP(S) 模型基础地址。")
|
||||
if parts.username or parts.password or parts.query or parts.fragment:
|
||||
raise LLMProviderCatalogError("模型基础地址不能包含账号、密码、查询参数或片段。")
|
||||
if provider_api not in SUPPORTED_PROVIDER_APIS:
|
||||
raise LLMProviderCatalogError("当前接口协议不支持模型目录查询。")
|
||||
path = parts.path.rstrip("/")
|
||||
if provider_api == "ollama-generate":
|
||||
path = path.removesuffix("/api").removesuffix("/v1") + "/api/tags"
|
||||
elif provider == "alibaba" and parts.hostname.endswith(".aliyuncs.com"):
|
||||
path = "/api/v1/models"
|
||||
else:
|
||||
if not path or (provider_api == "anthropic-messages" and path.endswith("/anthropic")):
|
||||
path += "/v1"
|
||||
path += "/models"
|
||||
return urlunsplit((parts.scheme, parts.netloc, path, "", ""))
|
||||
|
||||
|
||||
def catalog_error_message(exc: Exception) -> str:
|
||||
if isinstance(exc, LLMProviderCatalogError):
|
||||
return str(exc)
|
||||
if isinstance(exc, (httpx.TimeoutException, TimeoutError)):
|
||||
return "模型目录请求超时,请检查网络后重试。"
|
||||
if isinstance(exc, httpx.HTTPStatusError):
|
||||
code = exc.response.status_code
|
||||
messages = {
|
||||
401: "API Key 验证失败,请检查当前供应商的凭证。",
|
||||
403: "当前 API Key 无权访问该模型目录,请检查账号权限和服务地域。",
|
||||
404: "模型目录接口不存在,请检查基础地址、地域和接口协议。",
|
||||
429: "供应商请求限流,请稍后重试。",
|
||||
}
|
||||
return messages.get(code, f"供应商模型目录返回 HTTP {code},请稍后重试。")
|
||||
if isinstance(exc, httpx.RequestError):
|
||||
return "无法连接模型目录,请检查基础地址和网络。"
|
||||
return "模型目录响应无效,请稍后重试。"
|
||||
|
||||
|
||||
def _model_rows(payload: object) -> tuple[list[dict[str, object]], dict[str, object]]:
|
||||
if not isinstance(payload, dict):
|
||||
raise LLMProviderCatalogError("供应商返回了无效的模型目录。")
|
||||
envelope = payload.get("output", payload)
|
||||
if not isinstance(envelope, dict):
|
||||
raise LLMProviderCatalogError("供应商返回了无效的模型目录。")
|
||||
rows = envelope.get("data", envelope.get("models"))
|
||||
if not isinstance(rows, list):
|
||||
raise LLMProviderCatalogError("供应商响应中没有模型列表。")
|
||||
if any(not isinstance(row, dict) for row in rows):
|
||||
raise LLMProviderCatalogError("供应商返回了无效的模型条目。")
|
||||
return rows, envelope
|
||||
|
||||
|
||||
def _model_id(row: dict[str, object]) -> str:
|
||||
value = row.get("id") or row.get("model") or row.get("name")
|
||||
if not isinstance(value, str) or not value.strip():
|
||||
raise LLMProviderCatalogError("供应商返回了缺少 ID 的模型条目。")
|
||||
return value.strip()
|
||||
|
||||
|
||||
def _model_date(row: dict[str, object]) -> float:
|
||||
value = row.get("created_at") or row.get("published_time") or row.get("created")
|
||||
if isinstance(value, (int, float)):
|
||||
return float(value)
|
||||
if isinstance(value, str):
|
||||
try:
|
||||
return datetime.fromisoformat(value.replace("Z", "+00:00")).timestamp()
|
||||
except ValueError:
|
||||
pass
|
||||
return 0
|
||||
|
||||
|
||||
async def _get_catalog_page(
|
||||
client: httpx.AsyncClient,
|
||||
url: str,
|
||||
headers: dict[str, str],
|
||||
params: dict[str, str | int],
|
||||
) -> object:
|
||||
for attempt in range(CATALOG_REQUEST_ATTEMPTS):
|
||||
try:
|
||||
response = await client.get(url, headers=headers, params=params)
|
||||
response.raise_for_status()
|
||||
return response.json()
|
||||
except httpx.HTTPStatusError as exc:
|
||||
if attempt or exc.response.status_code not in {502, 503, 504}:
|
||||
raise
|
||||
except httpx.TransportError:
|
||||
if attempt:
|
||||
raise
|
||||
await asyncio.sleep(CATALOG_RETRY_DELAY_SECONDS)
|
||||
raise LLMProviderCatalogError("模型目录请求失败。")
|
||||
|
||||
|
||||
async def fetch_model_catalog(
|
||||
provider: str,
|
||||
base_url: str,
|
||||
provider_api: str,
|
||||
api_key: str = "",
|
||||
anthropic_version: str = "2023-06-01",
|
||||
timeout_seconds: int = CATALOG_TIMEOUT_SECONDS,
|
||||
) -> ModelCatalog:
|
||||
url = model_catalog_url(provider, base_url, provider_api)
|
||||
public_catalog = provider in {"opencode-go", "openrouter"}
|
||||
if not api_key and provider_api != "ollama-generate" and not public_catalog:
|
||||
raise LLMProviderCatalogError("请先配置当前供应商的 API Key,再刷新模型列表。")
|
||||
headers = {"User-Agent": "Planet/1.0", "Accept": "application/json"}
|
||||
if provider_api == "anthropic-messages" and provider not in {
|
||||
"opencode-go",
|
||||
"openrouter",
|
||||
"alibaba",
|
||||
"moonshotai",
|
||||
}:
|
||||
headers.update({"x-api-key": api_key, "anthropic-version": anthropic_version})
|
||||
elif api_key:
|
||||
headers["Authorization"] = f"Bearer {api_key}"
|
||||
native_dashscope = provider == "alibaba" and urlsplit(url).hostname.endswith(".aliyuncs.com")
|
||||
params: dict[str, str | int] = {}
|
||||
if native_dashscope:
|
||||
params = {"page_no": 1, "page_size": CATALOG_PAGE_SIZE, "capabilities": "TG"}
|
||||
rows_by_id: dict[str, dict[str, object]] = {}
|
||||
timeout = max(1, min(timeout_seconds, CATALOG_TIMEOUT_SECONDS))
|
||||
# Bound the complete pagination/retry cycle, not just each individual request.
|
||||
async with asyncio.timeout(timeout), httpx.AsyncClient(timeout=timeout) as client:
|
||||
for page in range(CATALOG_MAX_PAGES):
|
||||
payload = await _get_catalog_page(client, url, headers, params)
|
||||
rows, envelope = _model_rows(payload)
|
||||
previous_count = len(rows_by_id)
|
||||
for row in rows:
|
||||
rows_by_id[_model_id(row)] = row
|
||||
has_more = envelope.get("has_more") is True
|
||||
if native_dashscope:
|
||||
total = envelope.get("total")
|
||||
if not isinstance(total, int) or total < 0:
|
||||
raise LLMProviderCatalogError("供应商返回了无效的模型目录分页信息。")
|
||||
has_more = len(rows_by_id) < total
|
||||
params["page_no"] = page + 2
|
||||
elif has_more:
|
||||
cursor = envelope.get("last_id")
|
||||
if not isinstance(cursor, str) or not cursor or cursor == params.get("after_id"):
|
||||
raise LLMProviderCatalogError("供应商返回了无效的模型目录分页信息。")
|
||||
params["after_id"] = cursor
|
||||
if not has_more:
|
||||
models = sorted(
|
||||
rows_by_id, key=lambda key: _model_date(rows_by_id[key]), reverse=True
|
||||
)
|
||||
# An empty Ollama catalog is valid: no models have been installed yet.
|
||||
if not models and provider_api != "ollama-generate":
|
||||
raise LLMProviderCatalogError("供应商返回了空模型目录,已保留上次模型列表。")
|
||||
return ModelCatalog(url=url, models=models)
|
||||
if len(rows_by_id) == previous_count:
|
||||
raise LLMProviderCatalogError("供应商模型目录分页没有进展,已保留上次模型列表。")
|
||||
raise LLMProviderCatalogError("供应商模型目录分页超过限制,已保留上次模型列表。")
|
||||
@@ -3,17 +3,30 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
import httpx
|
||||
|
||||
MODELS_DEV_URL = "https://models.dev/api.json"
|
||||
OPENCODE_GO_MODELS_URL = "https://opencode.ai/zen/go/v1/models"
|
||||
from app.services.llm_model_catalog import fetch_model_catalog
|
||||
|
||||
OPENCODE_GO_MODEL_PROVIDER_APIS = {
|
||||
"minimax-m3": "anthropic-messages",
|
||||
"qwen3.8-max": "anthropic-messages",
|
||||
"qwen3.8-flash": "anthropic-messages",
|
||||
"qwen3.7-max": "anthropic-messages",
|
||||
"qwen3.7-plus": "anthropic-messages",
|
||||
"qwen3.6-plus": "anthropic-messages",
|
||||
"grok-4.6": "openai-responses",
|
||||
"gpt-5.6-luna": "openai-responses",
|
||||
"muse-spark-1.3-contributor": "openai-responses",
|
||||
"muse-spark-1.2-contributor": "openai-responses",
|
||||
"minimax-m2.7": "anthropic-messages",
|
||||
"minimax-m2.5": "anthropic-messages",
|
||||
}
|
||||
OPENCODE_GO_FALLBACK_MODELS = [
|
||||
"minimax-m3",
|
||||
"kimi-k3",
|
||||
"glm-5.3",
|
||||
"qwen3.8-max",
|
||||
"gpt-5.6-luna",
|
||||
"minimax-m2.7",
|
||||
"minimax-m2.5",
|
||||
"kimi-k2.6",
|
||||
@@ -29,24 +42,47 @@ OPENCODE_GO_FALLBACK_MODELS = [
|
||||
]
|
||||
|
||||
|
||||
OPENAI_MODEL_PROVIDER_APIS = {
|
||||
model: "openai-responses"
|
||||
for model in [
|
||||
"gpt-6-astra",
|
||||
"gpt-5.6-sol",
|
||||
"gpt-5.6-terra",
|
||||
"gpt-5.6-luna",
|
||||
"gpt-5.1",
|
||||
"gpt-5.1-codex",
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
|
||||
"minimax": {
|
||||
"provider": "minimax",
|
||||
"label": "MiniMax",
|
||||
"provider_api": "anthropic-messages",
|
||||
"base_url": "https://api.minimaxi.com/anthropic",
|
||||
"model": "MiniMax-M2.7",
|
||||
"models": ["MiniMax-M2.7", "MiniMax-M2.7-highspeed", "MiniMax-M2.5", "MiniMax-M2"],
|
||||
"model": "MiniMax-M3",
|
||||
"models": [
|
||||
"MiniMax-M3",
|
||||
"MiniMax-M2.7",
|
||||
"MiniMax-M2.7-highspeed",
|
||||
"MiniMax-M2.5",
|
||||
"MiniMax-M2.5-highspeed",
|
||||
"MiniMax-M2.1",
|
||||
"MiniMax-M2.1-highspeed",
|
||||
"MiniMax-M2",
|
||||
],
|
||||
"api_key_env": "MINIMAX_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
"openai": {
|
||||
"provider": "openai",
|
||||
"label": "OpenAI",
|
||||
"provider_api": "openai-completions",
|
||||
"provider_api": "openai-responses",
|
||||
"base_url": "https://api.openai.com/v1",
|
||||
"model": "gpt-5.1",
|
||||
"models": ["gpt-5.1", "gpt-5.1-codex", "gpt-4.1", "gpt-4o"],
|
||||
"model": "gpt-6-astra",
|
||||
"models": ["gpt-6-astra", "gpt-5.6-sol", "gpt-5.6-terra", "gpt-5.6-luna", "gpt-4.1"],
|
||||
"model_provider_apis": OPENAI_MODEL_PROVIDER_APIS,
|
||||
"api_key_env": "OPENAI_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
@@ -55,8 +91,8 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
|
||||
"label": "Anthropic",
|
||||
"provider_api": "anthropic-messages",
|
||||
"base_url": "https://api.anthropic.com/v1",
|
||||
"model": "claude-sonnet-4-6",
|
||||
"models": ["claude-sonnet-4-6", "claude-opus-4-5", "claude-3-5-haiku-20241022"],
|
||||
"model": "claude-opus-5",
|
||||
"models": ["claude-opus-5", "claude-sonnet-4-6"],
|
||||
"api_key_env": "ANTHROPIC_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
@@ -65,8 +101,8 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
|
||||
"label": "DeepSeek",
|
||||
"provider_api": "openai-completions",
|
||||
"base_url": "https://api.deepseek.com/v1",
|
||||
"model": "deepseek-chat",
|
||||
"models": ["deepseek-chat", "deepseek-reasoner"],
|
||||
"model": "deepseek-flash",
|
||||
"models": ["deepseek-flash", "deepseek-v4-pro"],
|
||||
"api_key_env": "DEEPSEEK_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
@@ -75,8 +111,8 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
|
||||
"label": "Alibaba Qwen / DashScope",
|
||||
"provider_api": "openai-completions",
|
||||
"base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1",
|
||||
"model": "qwen3-max",
|
||||
"models": ["qwen3-max", "qwen3.5-plus", "qwen-max", "qwen-plus"],
|
||||
"model": "qwen3.8-max",
|
||||
"models": ["qwen3.8-max", "qwen3-max", "qwen-plus"],
|
||||
"api_key_env": "DASHSCOPE_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
@@ -85,8 +121,8 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
|
||||
"label": "Moonshot AI / Kimi",
|
||||
"provider_api": "openai-completions",
|
||||
"base_url": "https://api.moonshot.ai/v1",
|
||||
"model": "kimi-k2.5",
|
||||
"models": ["kimi-k2.5", "kimi-k2-thinking", "kimi-k2-turbo-preview"],
|
||||
"model": "kimi-k3",
|
||||
"models": ["kimi-k3", "kimi-k2.5"],
|
||||
"api_key_env": "MOONSHOT_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
@@ -123,16 +159,6 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
|
||||
},
|
||||
}
|
||||
|
||||
MODELS_DEV_PROVIDER_KEYS = {
|
||||
"minimax": "minimax",
|
||||
"openai": "openai",
|
||||
"anthropic": "anthropic",
|
||||
"deepseek": "deepseek",
|
||||
"alibaba": "alibaba",
|
||||
"moonshotai": "moonshotai",
|
||||
"openrouter": "openrouter",
|
||||
}
|
||||
|
||||
|
||||
def list_fallback_llm_provider_presets() -> list[dict[str, Any]]:
|
||||
return [dict(value) for value in FALLBACK_LLM_PROVIDER_PRESETS.values()]
|
||||
@@ -152,65 +178,40 @@ def _opencode_go_model_provider_apis(model_ids: list[str]) -> dict[str, str]:
|
||||
}
|
||||
|
||||
|
||||
async def refresh_llm_provider_preset(provider: str, api_key: str | None = None) -> dict[str, Any]:
|
||||
async def refresh_llm_provider_preset(
|
||||
provider: str,
|
||||
api_key: str | None = None,
|
||||
*,
|
||||
base_url: str | None = None,
|
||||
provider_api: str | None = None,
|
||||
anthropic_version: str = "2023-06-01",
|
||||
) -> dict[str, Any]:
|
||||
fallback = get_fallback_llm_provider_preset(provider)
|
||||
if fallback["provider"] == "opencode-go":
|
||||
headers = {"User-Agent": "Planet/1.0"}
|
||||
if api_key:
|
||||
headers["Authorization"] = f"Bearer {api_key}"
|
||||
async with httpx.AsyncClient(timeout=15.0, follow_redirects=True) as client:
|
||||
response = await client.get(
|
||||
OPENCODE_GO_MODELS_URL,
|
||||
headers=headers,
|
||||
)
|
||||
response.raise_for_status()
|
||||
payload = response.json()
|
||||
|
||||
data = payload.get("data") if isinstance(payload, dict) else []
|
||||
model_ids = [
|
||||
str(item.get("id"))
|
||||
for item in data
|
||||
if isinstance(item, dict) and item.get("id")
|
||||
][:120]
|
||||
if not model_ids:
|
||||
model_ids = fallback["models"]
|
||||
return {
|
||||
**fallback,
|
||||
"model": fallback["model"] if fallback["model"] in model_ids else model_ids[0],
|
||||
"models": model_ids,
|
||||
"model_provider_apis": _opencode_go_model_provider_apis(model_ids),
|
||||
"source": OPENCODE_GO_MODELS_URL,
|
||||
}
|
||||
|
||||
models_dev_key = MODELS_DEV_PROVIDER_KEYS.get(fallback["provider"])
|
||||
if not models_dev_key:
|
||||
return fallback
|
||||
|
||||
async with httpx.AsyncClient(timeout=15.0, follow_redirects=True) as client:
|
||||
response = await client.get(
|
||||
MODELS_DEV_URL,
|
||||
headers={"User-Agent": "Planet/1.0"},
|
||||
)
|
||||
response.raise_for_status()
|
||||
catalog = response.json()
|
||||
|
||||
upstream = catalog.get(models_dev_key)
|
||||
if not isinstance(upstream, dict):
|
||||
return fallback
|
||||
|
||||
upstream_models = upstream.get("models") if isinstance(upstream.get("models"), dict) else {}
|
||||
model_ids = list(upstream_models.keys())[:80]
|
||||
base_url = upstream.get("api") or fallback["base_url"]
|
||||
if fallback["provider"] == "deepseek" and base_url == "https://api.deepseek.com":
|
||||
base_url = "https://api.deepseek.com/v1"
|
||||
|
||||
refreshed = {
|
||||
resolved_base_url = base_url or fallback["base_url"]
|
||||
resolved_api = provider_api or fallback["provider_api"]
|
||||
catalog = await fetch_model_catalog(
|
||||
fallback["provider"],
|
||||
resolved_base_url,
|
||||
resolved_api,
|
||||
api_key or "",
|
||||
anthropic_version,
|
||||
)
|
||||
model_provider_apis = (
|
||||
_opencode_go_model_provider_apis(catalog.models)
|
||||
if fallback["provider"] == "opencode-go"
|
||||
else fallback.get("model_provider_apis", {})
|
||||
)
|
||||
if (
|
||||
fallback["provider"] == "openai"
|
||||
and urlsplit(resolved_base_url).hostname != "api.openai.com"
|
||||
):
|
||||
model_provider_apis = {}
|
||||
return {
|
||||
**fallback,
|
||||
"label": upstream.get("name") or fallback["label"],
|
||||
"base_url": base_url,
|
||||
"model": model_ids[0] if model_ids else fallback["model"],
|
||||
"models": model_ids or fallback["models"],
|
||||
"api_key_env": (upstream.get("env") or [fallback["api_key_env"]])[0],
|
||||
"source": MODELS_DEV_URL,
|
||||
"base_url": resolved_base_url,
|
||||
"provider_api": resolved_api,
|
||||
"model": catalog.models[0] if catalog.models else "",
|
||||
"models": catalog.models,
|
||||
"model_provider_apis": model_provider_apis,
|
||||
"source": catalog.url,
|
||||
}
|
||||
return refreshed
|
||||
|
||||
@@ -635,6 +635,7 @@ async def _run_assistant_message(
|
||||
constraints=_collect_constraints(payload.constraints),
|
||||
context={
|
||||
"source": "playground",
|
||||
"session_id": session_id,
|
||||
"preset": payload.selected_preset_key,
|
||||
"conversation_history": conversation_history,
|
||||
"history_size": len(conversation_history),
|
||||
|
||||
148
backend/app/services/tv_catalog.py
Normal file
148
backend/app/services/tv_catalog.py
Normal file
@@ -0,0 +1,148 @@
|
||||
"""Search and paginate the public live TV catalog at the database boundary."""
|
||||
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import Select, func, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.services.tv_streams import (
|
||||
TV_LIVE_SOURCE_COLLECTOR,
|
||||
TV_LIVE_SOURCE_DATA_TYPE,
|
||||
_build_collected_tv_source,
|
||||
build_public_tv_payload,
|
||||
get_tv_settings_payload,
|
||||
)
|
||||
|
||||
|
||||
def _source_key():
|
||||
return func.coalesce(
|
||||
func.nullif(CollectedData.extra_data["id"].as_string(), ""),
|
||||
func.nullif(CollectedData.source_id, ""),
|
||||
CollectedData.entity_key,
|
||||
)
|
||||
|
||||
|
||||
def _collected_catalog_query(configured_ids: list[str]) -> Select[tuple[CollectedData]]:
|
||||
metadata = CollectedData.extra_data
|
||||
source_id = _source_key()
|
||||
enabled = func.lower(func.trim(func.coalesce(metadata["is_enabled"].as_string(), "true")))
|
||||
ranked = (
|
||||
select(
|
||||
CollectedData.id,
|
||||
func.row_number()
|
||||
.over(
|
||||
partition_by=source_id,
|
||||
order_by=CollectedData.id.desc(),
|
||||
)
|
||||
.label("source_rank"),
|
||||
)
|
||||
.where(
|
||||
CollectedData.source == TV_LIVE_SOURCE_COLLECTOR,
|
||||
CollectedData.data_type == TV_LIVE_SOURCE_DATA_TYPE,
|
||||
CollectedData.is_current.is_(True),
|
||||
CollectedData.is_valid == 1,
|
||||
CollectedData.deleted_at.is_(None),
|
||||
enabled.notin_(("false", "0", "no", "off")),
|
||||
source_id.notin_(configured_ids),
|
||||
)
|
||||
.subquery()
|
||||
)
|
||||
return (
|
||||
select(CollectedData)
|
||||
.join(ranked, ranked.c.id == CollectedData.id)
|
||||
.where(ranked.c.source_rank == 1)
|
||||
)
|
||||
|
||||
|
||||
def _filter_catalog_query(
|
||||
query: Select[tuple[CollectedData]], terms: list[str]
|
||||
) -> Select[tuple[CollectedData]]:
|
||||
metadata = CollectedData.extra_data
|
||||
searchable = func.lower(
|
||||
func.concat_ws(
|
||||
" ",
|
||||
CollectedData.name,
|
||||
CollectedData.title,
|
||||
CollectedData.source_id,
|
||||
metadata["name"].as_string(),
|
||||
metadata["provider"].as_string(),
|
||||
metadata["region"].as_string(),
|
||||
metadata["country"].as_string(),
|
||||
metadata["language"].as_string(),
|
||||
)
|
||||
)
|
||||
for term in terms:
|
||||
query = query.where(searchable.contains(term, autoescape=True))
|
||||
return query
|
||||
|
||||
|
||||
def _matches_source(source: dict[str, Any], terms: list[str]) -> bool:
|
||||
searchable = " ".join(
|
||||
str(source.get(key) or "") for key in ("id", "name", "provider", "region", "language")
|
||||
).lower()
|
||||
return all(term in searchable for term in terms)
|
||||
|
||||
|
||||
async def get_tv_catalog_page(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
offset: int = 0,
|
||||
limit: int = 50,
|
||||
q: str = "",
|
||||
selected_id: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
settings = await get_tv_settings_payload(db)
|
||||
payload = build_public_tv_payload(settings, [])
|
||||
configured = payload["sources"]
|
||||
query = _collected_catalog_query([source["id"] for source in settings["sources"]])
|
||||
if selected_id:
|
||||
selected = next((source for source in configured if source["id"] == selected_id), None)
|
||||
if selected is None:
|
||||
record = await db.scalar(query.where(_source_key() == selected_id).limit(1))
|
||||
selected = _build_collected_tv_source(record, 0) if record else None
|
||||
if selected:
|
||||
payload["selected_source"] = selected
|
||||
summary = (
|
||||
await db.execute(
|
||||
select(func.count(), func.max(CollectedData.collected_at))
|
||||
.select_from(CollectedData)
|
||||
.where(CollectedData.id.in_(query.with_only_columns(CollectedData.id)))
|
||||
)
|
||||
).one()
|
||||
total_collected, latest_update = summary
|
||||
terms = q.lower().split()
|
||||
matched_configured = [source for source in configured if _matches_source(source, terms)]
|
||||
filtered_query = _filter_catalog_query(query, terms)
|
||||
matched_collected = (
|
||||
await db.scalar(select(func.count()).select_from(filtered_query.subquery()))
|
||||
if terms
|
||||
else total_collected
|
||||
)
|
||||
sources = matched_configured[offset : offset + limit]
|
||||
remaining = limit - len(sources)
|
||||
if remaining:
|
||||
rows = await db.scalars(
|
||||
filtered_query.order_by(func.lower(CollectedData.name), CollectedData.id)
|
||||
.offset(max(0, offset - len(matched_configured)))
|
||||
.limit(remaining)
|
||||
)
|
||||
sources.extend(
|
||||
_build_collected_tv_source(record, index) for index, record in enumerate(rows)
|
||||
)
|
||||
total = len(matched_configured) + matched_collected
|
||||
next_offset = offset + len(sources)
|
||||
return {
|
||||
**payload,
|
||||
"sources": sources,
|
||||
"source_count": len(configured) + total_collected,
|
||||
"latest_updated_at": (
|
||||
to_iso8601_utc(latest_update) if latest_update else payload["latest_updated_at"]
|
||||
),
|
||||
"total": total,
|
||||
"offset": offset,
|
||||
"limit": limit,
|
||||
"has_more": next_offset < total,
|
||||
"next_offset": next_offset if next_offset < total else None,
|
||||
}
|
||||
@@ -11,10 +11,16 @@ from app.core.time import to_iso8601_utc
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.models.system_setting import SystemSetting
|
||||
|
||||
DEFAULT_TV_SOURCE_ID = "cgtn-en"
|
||||
DEFAULT_TV_SOURCE_ID = "aljazeera-mubasher"
|
||||
TV_SETTINGS_CATEGORY = "tv"
|
||||
TV_LIVE_SOURCE_COLLECTOR = "news_live_streams"
|
||||
TV_LIVE_SOURCE_DATA_TYPE = "news_live_stream"
|
||||
CCTV4_HOMEPAGE = "https://tv.cctv.com/live/cctv4/"
|
||||
CCTV4_STREAM = "https://h5cul1yar48um3t.wcetv.com/hls/cctv4.m3u8"
|
||||
UNSUPPORTED_CCTV4_STREAM = (
|
||||
"https://ldocctvwbcdtxy.liveplay.myqcloud.com/ldocctvwbcd/cdrmldcctv4_1_td.m3u8"
|
||||
)
|
||||
CCTV4_PLAYBACK_NOTE = "CCTV-4 美洲版,使用 WCETV HLS 直播源;节目编排可能与亚洲版不同。"
|
||||
|
||||
DEFAULT_TV_SETTINGS = {
|
||||
"default_source_id": DEFAULT_TV_SOURCE_ID,
|
||||
@@ -22,20 +28,21 @@ DEFAULT_TV_SETTINGS = {
|
||||
"sources": [
|
||||
{
|
||||
"id": "cctv4",
|
||||
"name": "CCTV-4 中文国际",
|
||||
"name": "CCTV-4 中文国际(美洲版)",
|
||||
"provider": "CCTV",
|
||||
"region": "China",
|
||||
"language": "zh-CN",
|
||||
"source_type": "hls",
|
||||
"embed_url": "https://tv.cctv.com/live/cctv4/",
|
||||
"stream_url": "https://ldocctvwbcdtxy.liveplay.myqcloud.com/ldocctvwbcd/cdrmldcctv4_1_td.m3u8",
|
||||
"homepage_url": "https://tv.cctv.com/live/cctv4/",
|
||||
"embed_url": "",
|
||||
"stream_url": CCTV4_STREAM,
|
||||
"use_proxy": False,
|
||||
"homepage_url": CCTV4_HOMEPAGE,
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": True,
|
||||
"sort_order": 10,
|
||||
"collector_source": None,
|
||||
"notes": "默认兜底新闻直播源。优先尝试 CCTV-4 官方 HLS 播放流,若直播放失败则回退到央视官网直播页。",
|
||||
"notes": CCTV4_PLAYBACK_NOTE,
|
||||
},
|
||||
{
|
||||
"id": "reuters-tv",
|
||||
@@ -237,6 +244,17 @@ def _clean_int(value: Any, *, default: int) -> int:
|
||||
|
||||
def normalize_tv_source(source: dict[str, Any] | None, *, index: int = 0) -> dict[str, Any]:
|
||||
payload = dict(source or {})
|
||||
# Retire only the formerly bundled CDRM URL, including saved copies of that default.
|
||||
# HTTP 200 and valid TS framing do not mean this stream is decodable by hls.js.
|
||||
if _clean_text(payload.get("id")) == "cctv4" and _clean_url(
|
||||
payload.get("stream_url")
|
||||
) == UNSUPPORTED_CCTV4_STREAM:
|
||||
payload.update(
|
||||
name="CCTV-4 中文国际(美洲版)",
|
||||
source_type="hls", stream_url=CCTV4_STREAM, embed_url="",
|
||||
use_proxy=False,
|
||||
homepage_url=CCTV4_HOMEPAGE, notes=CCTV4_PLAYBACK_NOTE,
|
||||
)
|
||||
source_id = _clean_text(payload.get("id")) or f"tv-source-{index + 1}"
|
||||
source_type = _clean_text(payload.get("source_type")).lower()
|
||||
youtube_video_id = _clean_text(payload.get("youtube_video_id"))
|
||||
@@ -259,6 +277,7 @@ def normalize_tv_source(source: dict[str, Any] | None, *, index: int = 0) -> dic
|
||||
"source_type": source_type,
|
||||
"embed_url": _clean_url(payload.get("embed_url")),
|
||||
"stream_url": _clean_url(payload.get("stream_url")),
|
||||
"use_proxy": _clean_bool(payload.get("use_proxy"), default=True),
|
||||
"homepage_url": _clean_url(payload.get("homepage_url")),
|
||||
"poster_url": _clean_url(payload.get("poster_url")),
|
||||
"youtube_video_id": youtube_video_id,
|
||||
@@ -300,8 +319,12 @@ def normalize_tv_settings(payload: dict[str, Any] | None) -> dict[str, Any]:
|
||||
]
|
||||
|
||||
if not any(source["id"] == DEFAULT_TV_SOURCE_ID for source in normalized_sources):
|
||||
default_source = next(
|
||||
source for source in DEFAULT_TV_SETTINGS["sources"]
|
||||
if source["id"] == DEFAULT_TV_SOURCE_ID
|
||||
)
|
||||
normalized_sources.append(
|
||||
normalize_tv_source(DEFAULT_TV_SETTINGS["sources"][0], index=len(normalized_sources))
|
||||
normalize_tv_source(default_source, index=len(normalized_sources))
|
||||
)
|
||||
|
||||
default_source_exists = any(
|
||||
|
||||
@@ -614,6 +614,18 @@ async def get_current_vessels_snapshot(
|
||||
return [item.to_dict() for item in result.scalars().all()]
|
||||
|
||||
|
||||
async def get_current_vessels_by_mmsi(
|
||||
db: AsyncSession, mmsis: list[int]
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Read canonical render state for the vessels changed by a stream flush."""
|
||||
if not mmsis:
|
||||
return []
|
||||
result = await db.execute(
|
||||
select(VesselCurrentState).where(VesselCurrentState.mmsi.in_(mmsis))
|
||||
)
|
||||
return [_jsonable(item.to_dict()) for item in result.scalars().all()]
|
||||
|
||||
|
||||
async def aggregate_vessel_observations(
|
||||
db: AsyncSession,
|
||||
observations: Iterable[AISRawObservation],
|
||||
|
||||
72
backend/tests/test_auth_identity.py
Normal file
72
backend/tests/test_auth_identity.py
Normal file
@@ -0,0 +1,72 @@
|
||||
from datetime import UTC, datetime
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from fastapi import FastAPI
|
||||
from pydantic import ValidationError
|
||||
|
||||
from app.api.v1.auth import router
|
||||
from app.core import security
|
||||
from app.db.session import DEFAULT_LOGIN_USERS, get_db
|
||||
from app.models.user import User
|
||||
from app.schemas.user import UserRegister
|
||||
|
||||
|
||||
async def identity_request(monkeypatch, user, token=None):
|
||||
result = MagicMock()
|
||||
result.scalar_one_or_none.return_value = user
|
||||
db = AsyncMock()
|
||||
db.execute.return_value = result
|
||||
monkeypatch.setattr(security.redis_client, "sismember", lambda *_: False)
|
||||
app = FastAPI()
|
||||
app.include_router(router, prefix="/auth")
|
||||
|
||||
async def override_db():
|
||||
yield db
|
||||
|
||||
app.dependency_overrides[get_db] = override_db
|
||||
async with httpx.AsyncClient(
|
||||
transport=httpx.ASGITransport(app=app), base_url="http://test"
|
||||
) as client:
|
||||
return await client.get(
|
||||
"/auth/me",
|
||||
headers={"Authorization": "Bearer " + (token or security.create_access_token({"sub": 1}))},
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("default", DEFAULT_LOGIN_USERS)
|
||||
async def test_default_identity_preserves_profile_fields(monkeypatch, default):
|
||||
created = datetime(2026, 9, 21, tzinfo=UTC)
|
||||
user = User(
|
||||
id=1, username=default["username"], email=default["email"],
|
||||
role=default["role"], password_hash="not-exposed", is_active=True,
|
||||
email_verified=True, gatekeeper_groups=["ops"], created_at=created,
|
||||
)
|
||||
response = await identity_request(monkeypatch, user)
|
||||
assert response.status_code == 200
|
||||
payload = response.json()
|
||||
assert payload["email"] == default["email"]
|
||||
assert payload["email_verified"] is True
|
||||
assert datetime.fromisoformat(payload["created_at"].replace("Z", "+00:00")) == created
|
||||
assert payload["gatekeeper_groups"] == ["ops"]
|
||||
assert "password_hash" not in payload
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("user", [None, User(id=1, is_active=False)])
|
||||
async def test_missing_or_inactive_identity_is_rejected(monkeypatch, user):
|
||||
response = await identity_request(monkeypatch, user)
|
||||
assert response.status_code == 401
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_refresh_token_is_not_an_access_identity(monkeypatch):
|
||||
response = await identity_request(monkeypatch, None, security.create_refresh_token({"sub": 1}))
|
||||
assert response.status_code == 401
|
||||
|
||||
|
||||
def test_registration_still_validates_email():
|
||||
with pytest.raises(ValidationError):
|
||||
UserRegister(username="newuser", email="newuser@planet.local", password="example-password")
|
||||
333
backend/tests/test_card_images.py
Normal file
333
backend/tests/test_card_images.py
Normal file
@@ -0,0 +1,333 @@
|
||||
import io
|
||||
from copy import deepcopy
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import pytest
|
||||
from fastapi import FastAPI, HTTPException
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
from PIL import Image
|
||||
from pydantic import ValidationError
|
||||
|
||||
from app.api.v1.card_images import router
|
||||
from app.core.security import get_current_user
|
||||
from app.db.session import get_db
|
||||
from app.schemas.card_images import CardImageAsset, CardImageConfiguration, CardImageRule
|
||||
from app.services.card_image_configuration import (
|
||||
effective_envelope,
|
||||
publish_configuration,
|
||||
resolve_image,
|
||||
)
|
||||
from app.services.card_image_defaults import bundled_configuration
|
||||
from app.services import card_image_files
|
||||
|
||||
|
||||
def configuration():
|
||||
assets = [
|
||||
CardImageAsset(id=key, title=key, url=f"/earth/assets/cards/{key}.jpg")
|
||||
for key in ["exact", "model", "group", "category"]
|
||||
]
|
||||
rules = [
|
||||
CardImageRule(id=key, entity_type="satellite", scope=scope, value=value, asset_id=key)
|
||||
for key, scope, value in [
|
||||
("exact", "object", "123"),
|
||||
("model", "model", "m1"),
|
||||
("group", "group", "starlink"),
|
||||
("category", "category", "satellite"),
|
||||
]
|
||||
]
|
||||
return CardImageConfiguration(assets=assets, rules=rules)
|
||||
|
||||
|
||||
def test_precedence_disabled_fallback_and_explicit_no_image():
|
||||
config = configuration()
|
||||
context = dict(
|
||||
entity_type="satellite", entity_id="123", model="M1", group="starlink", category="satellite"
|
||||
)
|
||||
for expected in ["exact", "model", "group", "category"]:
|
||||
assert resolve_image(config, context)["image"]["id"] == expected
|
||||
next(asset for asset in config.assets if asset.id == expected).enabled = False
|
||||
assert resolve_image(config, context)["image"] is None
|
||||
config = configuration()
|
||||
config.rules[0].asset_id = None
|
||||
config.rules[0].suppress = True
|
||||
result = resolve_image(config, context)
|
||||
assert result["suppressed"] and result["image"] is None
|
||||
|
||||
|
||||
def test_bundled_series_images_retain_truthful_source_metadata():
|
||||
config, _ = bundled_configuration()
|
||||
result = resolve_image(config, dict(entity_type="satellite", entity_id="123", group="starlink"))
|
||||
assert result["image"]["id"] == "starlink-solar-array"
|
||||
pending = next(asset for asset in config.assets if asset.id == "qianfan-animation-still")
|
||||
assert pending.rights_status == "permission_pending"
|
||||
assert (
|
||||
resolve_image(config, dict(entity_type="satellite", group="qianfan"))["image"]["id"]
|
||||
== pending.id
|
||||
)
|
||||
assert pending.enabled
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"url",
|
||||
[
|
||||
"javascript:alert(1)",
|
||||
"//evil.example/image.jpg",
|
||||
"/earth/assets/cards/../secret",
|
||||
"https://user:pass@example.org/a.jpg",
|
||||
],
|
||||
)
|
||||
def test_unsafe_urls_rejected(url):
|
||||
with pytest.raises(ValidationError):
|
||||
CardImageAsset(id="test", title="test", url=url)
|
||||
|
||||
|
||||
def test_duplicate_targets_and_dangling_assets_rejected():
|
||||
data = configuration().model_dump()
|
||||
duplicate = {**data["rules"][0], "id": "another"}
|
||||
data["rules"].append(duplicate)
|
||||
with pytest.raises(ValidationError):
|
||||
CardImageConfiguration.model_validate(data)
|
||||
data = configuration().model_dump()
|
||||
data["assets"] = []
|
||||
with pytest.raises(ValidationError):
|
||||
CardImageConfiguration.model_validate(data)
|
||||
|
||||
|
||||
def test_new_defaults_do_not_undo_saved_removals_or_manual_edits():
|
||||
defaults, _ = bundled_configuration()
|
||||
payload = {
|
||||
"revision": "custom",
|
||||
"config": {"assets": [], "rules": []},
|
||||
"known_assets": [asset.id for asset in defaults.assets],
|
||||
"known_rules": [rule.id for rule in defaults.rules],
|
||||
}
|
||||
assert effective_envelope(payload)["config"] == {"assets": [], "rules": []}
|
||||
payload["known_assets"] = []
|
||||
assert len(effective_envelope(payload)["config"]["assets"]) == len(defaults.assets)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("format", ["JPEG", "PNG", "WEBP"])
|
||||
def test_upload_verifies_raster_and_creates_reusable_variants(tmp_path, monkeypatch, format):
|
||||
monkeypatch.setattr(card_image_files, "ASSET_DIR", tmp_path)
|
||||
output = io.BytesIO()
|
||||
Image.new("RGB", (1600, 900), "blue").save(output, format)
|
||||
asset = card_image_files.store_image(output.getvalue(), "Facility")
|
||||
assert (asset.width, asset.height) == (1600, 900)
|
||||
assert len(list(tmp_path.iterdir())) == 3
|
||||
for size, url in asset.variants.items():
|
||||
with Image.open(tmp_path / url.rsplit("/", 1)[1]) as image:
|
||||
assert max(image.size) == int(size)
|
||||
assert card_image_files.store_image(output.getvalue(), "Again").id == asset.id
|
||||
with pytest.raises(HTTPException):
|
||||
card_image_files.store_image(b"<svg></svg>", "bad")
|
||||
with pytest.raises(HTTPException):
|
||||
card_image_files.uploaded_path("../private")
|
||||
|
||||
|
||||
def fake_db(payload):
|
||||
db = AsyncMock()
|
||||
db.add = MagicMock()
|
||||
record = SimpleNamespace(payload=payload)
|
||||
result = MagicMock()
|
||||
result.scalar_one.return_value = record
|
||||
result.scalar_one_or_none.return_value = record
|
||||
db.execute.return_value = result
|
||||
return db, record
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_publish_revision_and_audit():
|
||||
defaults, _ = bundled_configuration()
|
||||
payload = {
|
||||
"revision": "default",
|
||||
"config": defaults.model_dump(mode="json"),
|
||||
"known_assets": [],
|
||||
"known_rules": [],
|
||||
}
|
||||
db, record = fake_db(payload)
|
||||
revision = effective_envelope(payload)["revision"]
|
||||
result = await publish_configuration(
|
||||
db, revision, defaults, SimpleNamespace(id=1, username="admin")
|
||||
)
|
||||
assert result["revision"] != revision
|
||||
assert db.add.call_args.args[0].action == "card_images.update"
|
||||
before = deepcopy(record.payload)
|
||||
with pytest.raises(HTTPException) as error:
|
||||
await publish_configuration(db, revision, defaults, SimpleNamespace(id=1, username="admin"))
|
||||
assert error.value.status_code == 409 and record.payload == before
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_configuration_requires_admin_and_validates_http_payload():
|
||||
app = FastAPI()
|
||||
app.include_router(router)
|
||||
defaults, _ = bundled_configuration()
|
||||
db, _ = fake_db(
|
||||
{
|
||||
"revision": "default",
|
||||
"config": defaults.model_dump(mode="json"),
|
||||
"known_assets": [],
|
||||
"known_rules": [],
|
||||
}
|
||||
)
|
||||
app.dependency_overrides[get_db] = lambda: db
|
||||
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
|
||||
assert (await client.get("/configuration")).status_code == 401
|
||||
app.dependency_overrides[get_current_user] = lambda: SimpleNamespace(role="viewer")
|
||||
assert (await client.get("/configuration")).status_code == 403
|
||||
app.dependency_overrides[get_current_user] = lambda: SimpleNamespace(
|
||||
role="admin", id=1, username="admin"
|
||||
)
|
||||
assert (await client.get("/configuration")).status_code == 200
|
||||
assert (
|
||||
await client.put("/configuration", json={"revision": "x", "config": {"rules": [{}]}})
|
||||
).status_code == 422
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_multipart_upload_and_public_file_delivery(tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(card_image_files, "ASSET_DIR", tmp_path)
|
||||
app = FastAPI()
|
||||
app.include_router(router)
|
||||
app.dependency_overrides[get_current_user] = lambda: SimpleNamespace(
|
||||
role="admin", id=1, username="admin"
|
||||
)
|
||||
buffer = io.BytesIO()
|
||||
Image.new("RGB", (800, 500), "blue").save(buffer, "JPEG")
|
||||
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
|
||||
response = await client.post(
|
||||
"/upload", files={"file": ("facility.jpg", buffer.getvalue(), "image/jpeg")}
|
||||
)
|
||||
assert response.status_code == 200
|
||||
asset = response.json()
|
||||
filename = asset["variants"]["640"].rsplit("/", 1)[1]
|
||||
image = await client.get(f"/files/{filename}")
|
||||
assert image.status_code == 200 and image.headers["content-type"] == "image/webp"
|
||||
assert "immutable" in image.headers["cache-control"]
|
||||
assert (
|
||||
await client.post("/upload", files={"file": ("bad.jpg", b"not an image", "image/jpeg")})
|
||||
).status_code == 422
|
||||
app.dependency_overrides[get_current_user] = lambda: SimpleNamespace(role="viewer")
|
||||
assert (
|
||||
await client.post(
|
||||
"/upload", files={"file": ("facility.jpg", buffer.getvalue(), "image/jpeg")}
|
||||
)
|
||||
).status_code == 403
|
||||
|
||||
|
||||
def test_all_object_types_have_a_default_without_overriding_exact_suppression():
|
||||
from typing import get_args
|
||||
from app.schemas.card_images import EntityType
|
||||
|
||||
config, _ = bundled_configuration()
|
||||
for entity_type in get_args(EntityType):
|
||||
result = resolve_image(config, dict(entity_type=entity_type, category="unlisted"))
|
||||
assert result["image"] and result["match"]["value"] == "*"
|
||||
config = config.model_copy(deep=True)
|
||||
config.rules.append(
|
||||
CardImageRule(id="off", entity_type="satellite", scope="object", value="123", suppress=True)
|
||||
)
|
||||
assert resolve_image(config, dict(entity_type="satellite", entity_id="123"))["suppressed"]
|
||||
|
||||
|
||||
def test_upgrade_replaces_old_bundled_source_but_preserves_custom_assets():
|
||||
from app.services.card_image_defaults import previous_bundled_assets
|
||||
|
||||
old = previous_bundled_assets()["starlink-solar-array"][0]
|
||||
payload = {
|
||||
"revision": "old",
|
||||
"config": {"assets": [old.model_dump()], "rules": []},
|
||||
"known_assets": [old.id],
|
||||
"known_rules": [],
|
||||
}
|
||||
|
||||
def starlink():
|
||||
return next(a for a in effective_envelope(payload)["config"]["assets"] if a["id"] == old.id)
|
||||
|
||||
assert starlink()["url"].endswith("starlink-orbit.webp")
|
||||
payload["config"]["assets"][0]["focus_x"] = 20
|
||||
assert starlink()["focus_x"] == 20
|
||||
assert starlink()["url"].endswith("starlink-orbit.webp")
|
||||
payload["config"]["assets"][0]["enabled"] = False
|
||||
assert starlink()["enabled"] is False
|
||||
payload["config"]["assets"][0]["url"] = "https://example.org/custom.jpg"
|
||||
assert starlink()["url"] == "https://example.org/custom.jpg"
|
||||
|
||||
|
||||
def test_compute_categories_and_bgp_get_separate_user_images():
|
||||
config, _ = bundled_configuration()
|
||||
for kind, filename in [
|
||||
("supercomputer", "supercomputer.png"),
|
||||
("gpu_cluster", "gpu-cluster.jpg"),
|
||||
("data_center", "data-center.png"),
|
||||
("cloud_center", "cloud-center.png"),
|
||||
("edge_node", "edge-node.png"),
|
||||
]:
|
||||
result = resolve_image(config, dict(entity_type="compute_center", category=kind))
|
||||
assert result["image"]["url"].endswith(filename)
|
||||
for kind in ("bgp", "bgp_collector"):
|
||||
result = resolve_image(config, dict(entity_type=kind, category=kind))
|
||||
assert result["image"]["url"].endswith(
|
||||
"bgp-event.jpg" if kind == "bgp" else "bgp-collector.png"
|
||||
)
|
||||
|
||||
|
||||
def test_saved_collector_default_migrates_without_changing_user_bindings():
|
||||
from app.services.card_image_defaults import previous_bundled_rules, previous_bundled_assets
|
||||
|
||||
old_rule = previous_bundled_rules()["builtin-network-switches-default-1"]
|
||||
old_asset = previous_bundled_assets()["network-switches"][0]
|
||||
payload = {
|
||||
"revision": "old",
|
||||
"config": {"assets": [old_asset.model_dump()], "rules": [old_rule.model_dump()]},
|
||||
"known_assets": [old_asset.id],
|
||||
"known_rules": [old_rule.id],
|
||||
}
|
||||
result = CardImageConfiguration.model_validate(effective_envelope(payload)["config"])
|
||||
assert resolve_image(result, dict(entity_type="bgp_collector", category="bgp_collector"))[
|
||||
"image"
|
||||
]["url"].endswith("bgp-collector.png")
|
||||
payload["config"]["rules"][0]["enabled"] = False
|
||||
result = effective_envelope(payload)["config"]
|
||||
assert not next(rule for rule in result["rules"] if rule["id"] == old_rule.id)["enabled"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_compute_lookup_filters_indexed_identity_and_keeps_actual_category():
|
||||
from app.services.card_image_targets import get_target
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
db = AsyncMock()
|
||||
result = MagicMock()
|
||||
result.mappings.return_value.first.return_value = {
|
||||
"entity_id": "epoch_ai_gpu:site",
|
||||
"name": "Site",
|
||||
"category": "cloud_center",
|
||||
"group": "Operator",
|
||||
"model": "Model",
|
||||
}
|
||||
db.execute.return_value = result
|
||||
context = await get_target(db, "compute_center", "epoch_ai_gpu:site")
|
||||
query = db.execute.call_args.args[0]
|
||||
sql = str(query.compile(dialect=postgresql.dialect(), compile_kwargs={"literal_binds": True}))
|
||||
assert "collected_data.source = 'epoch_ai_gpu'" in sql
|
||||
assert "collected_data.source_id = 'site'" in sql
|
||||
assert context["category"] == "cloud_center" and context["model"] == "Model"
|
||||
|
||||
|
||||
def test_old_gpu_preview_urls_are_replaced_with_latest_variants():
|
||||
from app.services.card_image_defaults import previous_bundled_assets
|
||||
|
||||
old = previous_bundled_assets()["compute-gpu-cluster"][0]
|
||||
payload = {
|
||||
"revision": "old",
|
||||
"config": {"assets": [old.model_dump()], "rules": []},
|
||||
"known_assets": [old.id],
|
||||
"known_rules": [],
|
||||
}
|
||||
result = effective_envelope(payload)["config"]
|
||||
upgraded = next(a for a in result["assets"] if a["id"] == old.id)
|
||||
latest = next(a for a in bundled_configuration()[0].assets if a.id == old.id)
|
||||
assert upgraded["variants"] == latest.variants
|
||||
assert upgraded["url"] == latest.url
|
||||
218
backend/tests/test_collector_downloads.py
Normal file
218
backend/tests/test_collector_downloads.py
Normal file
@@ -0,0 +1,218 @@
|
||||
"""Request-level regressions for quota-limited satellite downloads."""
|
||||
|
||||
import gzip
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from app.services.collectors.celestrak import CelesTrakTLECollector, CELESTRAK_CACHE_TTL_SECONDS
|
||||
from app.services.collectors.downloads import ResumableFileDownloader
|
||||
|
||||
URL = "https://celestrak.example/gp.php?GROUP=active&FORMAT=json"
|
||||
|
||||
|
||||
def setup_collector(monkeypatch, tmp_path, handler):
|
||||
collector = CelesTrakTLECollector()
|
||||
collector._resolved_url = "https://celestrak.example/gp.php"
|
||||
monkeypatch.setattr(collector._downloader, "_cache_dir", tmp_path)
|
||||
monkeypatch.setattr(collector, "update_phase_progress", AsyncMock())
|
||||
monkeypatch.setattr("app.services.collectors.celestrak.emit_business_log", AsyncMock())
|
||||
monkeypatch.setattr("app.services.collectors.celestrak.asyncio.sleep", AsyncMock())
|
||||
original_client = httpx.AsyncClient
|
||||
monkeypatch.setattr(
|
||||
"app.services.collectors.celestrak.httpx.AsyncClient",
|
||||
lambda **kwargs: original_client(transport=httpx.MockTransport(handler), **kwargs),
|
||||
)
|
||||
return collector
|
||||
|
||||
|
||||
def expire(path):
|
||||
old = time.time() - CELESTRAK_CACHE_TTL_SECONDS - 1
|
||||
os.utime(path, (old, old))
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_celestrak_uses_one_get_and_revalidates_after_cache_expiry(monkeypatch, tmp_path):
|
||||
requests = []
|
||||
|
||||
def handler(request):
|
||||
requests.append(request)
|
||||
assert request.method == "GET", "HEAD consumes this upstream's download allowance"
|
||||
return httpx.Response(
|
||||
200, json=[{"NORAD_CAT_ID": len(requests)}], headers={"etag": f'"v{len(requests)}"'}
|
||||
)
|
||||
|
||||
collector = setup_collector(monkeypatch, tmp_path, handler)
|
||||
assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 1
|
||||
assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 1
|
||||
assert len(requests) == 1
|
||||
path = collector._downloader.cached_file_path(URL, ".json")
|
||||
expire(path)
|
||||
assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 2
|
||||
assert requests[1].headers["if-none-match"] == '"v1"'
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_not_updated_reuses_valid_cache_without_retry_storm(monkeypatch, tmp_path):
|
||||
requests = []
|
||||
|
||||
def handler(request):
|
||||
requests.append(request.method)
|
||||
return httpx.Response(
|
||||
403, text="GP data has not updated since your last successful download"
|
||||
)
|
||||
|
||||
collector = setup_collector(monkeypatch, tmp_path, handler)
|
||||
path = collector._downloader.cached_file_path(URL, ".json")
|
||||
path.write_text('[{"NORAD_CAT_ID":42}]')
|
||||
expire(path)
|
||||
assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 42
|
||||
assert requests == ["GET"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_bad_download_does_not_replace_last_good_cache(monkeypatch, tmp_path):
|
||||
collector = setup_collector(monkeypatch, tmp_path, lambda request: httpx.Response(200, json=[]))
|
||||
path = collector._downloader.cached_file_path(URL, ".json")
|
||||
path.write_text('[{"NORAD_CAT_ID":42}]')
|
||||
expire(path)
|
||||
with pytest.raises(RuntimeError, match="validation failed"):
|
||||
await collector.fetch()
|
||||
assert json.loads(path.read_text()) == [{"NORAD_CAT_ID": 42}]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_corrupt_cache_is_not_reused_or_conditionally_requested(monkeypatch, tmp_path):
|
||||
requests = []
|
||||
|
||||
def handler(request):
|
||||
requests.append(request)
|
||||
return httpx.Response(200, json=[{"NORAD_CAT_ID": 7}])
|
||||
|
||||
collector = setup_collector(monkeypatch, tmp_path, handler)
|
||||
path, _, meta = collector._downloader._cache_paths(URL, ".json")
|
||||
path.write_text("[]")
|
||||
meta.write_text('{"etag":"old"}')
|
||||
assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 7
|
||||
assert "if-none-match" not in requests[0].headers
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_conditional_304_keeps_payload_and_refreshes_cache_window(tmp_path):
|
||||
downloader = ResumableFileDownloader(
|
||||
cache_namespace="test", cache_root=tmp_path, preflight_head=False, cache_ttl_seconds=7200
|
||||
)
|
||||
path, _, meta = downloader._cache_paths(URL, ".json")
|
||||
path.parent.mkdir(parents=True)
|
||||
path.write_text('[{"NORAD_CAT_ID":42}]')
|
||||
meta.write_text('{"etag":"v1"}')
|
||||
expire(path)
|
||||
requests = []
|
||||
|
||||
def handler(request):
|
||||
requests.append(request)
|
||||
return httpx.Response(304)
|
||||
|
||||
async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as client:
|
||||
for _ in range(2):
|
||||
await downloader.download_file(
|
||||
client,
|
||||
URL,
|
||||
extension=".json",
|
||||
validate_existing=CelesTrakTLECollector._validate_json_file,
|
||||
)
|
||||
assert len(requests) == 1
|
||||
assert requests[0].headers["if-none-match"] == "v1"
|
||||
assert json.loads(path.read_text())[0]["NORAD_CAT_ID"] == 42
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_compressed_get_length_uses_decoded_payload(tmp_path):
|
||||
raw = json.dumps([{"NORAD_CAT_ID": 42}]).encode()
|
||||
body = gzip.compress(raw)
|
||||
downloader = ResumableFileDownloader(
|
||||
cache_namespace="test", cache_root=tmp_path, preflight_head=False
|
||||
)
|
||||
async with httpx.AsyncClient(
|
||||
transport=httpx.MockTransport(
|
||||
lambda request: httpx.Response(
|
||||
200,
|
||||
content=body,
|
||||
headers={"content-encoding": "gzip", "content-length": str(len(body))},
|
||||
)
|
||||
)
|
||||
) as client:
|
||||
path = await downloader.download_file(
|
||||
client,
|
||||
URL,
|
||||
extension=".json",
|
||||
validate_existing=CelesTrakTLECollector._validate_json_file,
|
||||
)
|
||||
assert path.read_bytes() == raw
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_other_collectors_keep_head_and_range_resume(tmp_path):
|
||||
downloader = ResumableFileDownloader(cache_namespace="test", cache_root=tmp_path)
|
||||
final, part, meta = downloader._cache_paths(URL, ".bin")
|
||||
part.parent.mkdir(parents=True)
|
||||
part.write_bytes(b"abc")
|
||||
meta.write_text('{"etag":"v1"}')
|
||||
requests = []
|
||||
|
||||
def handler(request):
|
||||
requests.append(request)
|
||||
if request.method == "HEAD":
|
||||
return httpx.Response(
|
||||
200, headers={"etag": "v1", "accept-ranges": "bytes", "content-length": "6"}
|
||||
)
|
||||
assert request.headers["range"] == "bytes=3-"
|
||||
return httpx.Response(206, content=b"def", headers={"content-range": "bytes 3-5/6"})
|
||||
|
||||
async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as client:
|
||||
await downloader.download_file(client, URL, extension=".bin")
|
||||
assert [r.method for r in requests] == ["HEAD", "GET"]
|
||||
assert final.read_bytes() == b"abcdef"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_fallback_group_refreshes_expired_cache(monkeypatch, tmp_path):
|
||||
requests = []
|
||||
|
||||
def handler(request):
|
||||
requests.append(request)
|
||||
if request.url.params["GROUP"] == "active":
|
||||
return httpx.Response(
|
||||
403, text="GP data has not updated since your last successful download"
|
||||
)
|
||||
return httpx.Response(200, json=[{"NORAD_CAT_ID": 99}])
|
||||
|
||||
collector = setup_collector(monkeypatch, tmp_path, handler)
|
||||
monkeypatch.setattr("app.services.collectors.celestrak.FALLBACK_GROUPS", ("starlink",))
|
||||
path = collector._downloader.cached_file_path(collector._group_url("starlink"), ".json")
|
||||
path.write_text('[{"NORAD_CAT_ID":42}]')
|
||||
expire(path)
|
||||
assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 99
|
||||
assert [r.method for r in requests] == ["GET", "GET"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_generic_cache_without_validators_must_refresh(tmp_path):
|
||||
downloader = ResumableFileDownloader(cache_namespace="test", cache_root=tmp_path)
|
||||
path = downloader.cached_file_path(URL, ".json")
|
||||
path.parent.mkdir(parents=True)
|
||||
path.write_text("[]")
|
||||
requests = []
|
||||
|
||||
def handler(request):
|
||||
requests.append(request.method)
|
||||
return httpx.Response(200, content=b"[1]" if request.method == "GET" else b"")
|
||||
|
||||
async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as client:
|
||||
await downloader.download_file(client, URL, extension=".json")
|
||||
assert requests == ["HEAD", "GET"]
|
||||
assert path.read_text() == "[1]"
|
||||
@@ -79,7 +79,8 @@ class TestTOP500Collector:
|
||||
"""Test parsing empty response"""
|
||||
collector = TOP500Collector()
|
||||
result = collector.parse_response("<html><body><table></table></body></html>")
|
||||
assert len(result) > 0
|
||||
assert result == []
|
||||
assert collector.fail_on_empty
|
||||
|
||||
def test_parse_response_single_item(self):
|
||||
"""Test parsing single item response"""
|
||||
@@ -97,9 +98,9 @@ class TestTOP500Collector:
|
||||
</tr>
|
||||
</table>
|
||||
"""
|
||||
result = collector.parse_response(response)
|
||||
result = collector.parse_response(response, "https://top500.org/lists/top500/2025/11/")
|
||||
assert len(result) == 1
|
||||
assert result[0]["source_id"] == "top500_1"
|
||||
assert result[0]["source_id"] == "top500_system_1"
|
||||
assert result[0]["name"] == "Test Supercomputer"
|
||||
assert result[0]["country"] == "USA"
|
||||
assert result[0]["metadata"]["rank"] == 1
|
||||
@@ -113,7 +114,7 @@ class TestTOP500Collector:
|
||||
<tr><th>Rank</th><th>System</th><th>Cores</th><th>Rmax</th><th>Rpeak</th><th>Power</th></tr>
|
||||
<tr>
|
||||
<td>1</td>
|
||||
<td>Valid\nVendor\nSite\nUSA</td>
|
||||
<td><a href="/system/2/">Valid</a>\nVendor\nSite\nUSA</td>
|
||||
<td>1000</td>
|
||||
<td>10 PFLOP/s</td>
|
||||
<td>12 PFLOP/s</td>
|
||||
@@ -129,7 +130,7 @@ class TestTOP500Collector:
|
||||
</tr>
|
||||
</table>
|
||||
"""
|
||||
result = collector.parse_response(response)
|
||||
result = collector.parse_response(response, "https://top500.org/lists/top500/2025/11/")
|
||||
assert len(result) == 1
|
||||
assert result[0]["name"] == "Valid"
|
||||
|
||||
|
||||
222
backend/tests/test_compute_configuration.py
Normal file
222
backend/tests/test_compute_configuration.py
Normal file
@@ -0,0 +1,222 @@
|
||||
from copy import deepcopy
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import pytest
|
||||
from app.api.v1.compute_intelligence import configuration_admin, get_series, router
|
||||
from app.core.security import get_current_user
|
||||
from app.db.session import get_db
|
||||
from app.models.system_log import AuditLog
|
||||
from app.schemas.compute_configuration import ComputeConfiguration
|
||||
from app.services.compute_configuration import (
|
||||
DEFAULT_CONFIGURATION,
|
||||
default_envelope,
|
||||
publish_configuration,
|
||||
read_configuration,
|
||||
)
|
||||
from app.services.compute_history import compute_history
|
||||
from fastapi import FastAPI, HTTPException, Response
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
from pydantic import ValidationError
|
||||
|
||||
|
||||
def default_config():
|
||||
return DEFAULT_CONFIGURATION.model_dump(mode="json")
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"mutation",
|
||||
[
|
||||
lambda value: value["observations"][0]["values"].update(CN=101),
|
||||
lambda value: value["observations"][0]["values"].update(CN=-1),
|
||||
lambda value: value["observations"][0]["values"].update(CN=float("nan")),
|
||||
lambda value: value["observations"][0]["values"].update(CN=True),
|
||||
lambda value: value["observations"][0]["values"].update(EUROPE=10),
|
||||
lambda value: value["observations"][0].update(source_id="unknown"),
|
||||
lambda value: value["observations"][0].update(year=2026),
|
||||
lambda value: value["observations"].append(value["observations"][0]),
|
||||
lambda value: value["sources"].append(value["sources"][0]),
|
||||
lambda value: value["innovation"].append(value["innovation"][0]),
|
||||
lambda value: value["sources"][0].update(url="javascript:alert(1)"),
|
||||
lambda value: value["sources"][0].update(url="https://user:pass@example.com/report"),
|
||||
],
|
||||
)
|
||||
def test_rejects_invalid_or_ambiguous_evidence(mutation):
|
||||
payload = default_config()
|
||||
mutation(payload)
|
||||
with pytest.raises(ValidationError):
|
||||
ComputeConfiguration.model_validate(payload)
|
||||
|
||||
|
||||
def test_arbitrary_report_updates_values_sources_and_years_without_code_changes():
|
||||
payload = default_config()
|
||||
payload["sources"].append(
|
||||
{
|
||||
"id": "new-report",
|
||||
"title": "New report",
|
||||
"url": "https://example.com/report",
|
||||
"publisher": "Research group",
|
||||
"report_year": 2026,
|
||||
"locator": "Figure 2",
|
||||
}
|
||||
)
|
||||
payload["observations"].append(
|
||||
{
|
||||
"year": 2025,
|
||||
"metric": "global_share",
|
||||
"source_id": "new-report",
|
||||
"values": {"CN": 25, "US": 50, "JP": 0},
|
||||
}
|
||||
)
|
||||
config = ComputeConfiguration.model_validate(payload)
|
||||
result = compute_history(["CN", "US", "JP", "EUROPE"], config=config)
|
||||
row = next(row for row in result["compute"] if row["year"] == 2025)
|
||||
assert row["values"] == {"CN": 25, "US": 50, "JP": 0, "EUROPE": None}
|
||||
assert result["evidence"][row["evidence_id"]]["title"] == "New report"
|
||||
assert DEFAULT_CONFIGURATION.model_dump(mode="json") == default_config()
|
||||
|
||||
|
||||
def test_zero_bilateral_denominator_is_missing_and_empty_override_stays_empty():
|
||||
payload = default_config()
|
||||
payload["observations"][0]["values"] = {"CN": 0, "US": 0}
|
||||
result = compute_history(
|
||||
["CN", "US"], "bilateral_share", ComputeConfiguration.model_validate(payload)
|
||||
)
|
||||
assert result["compute"][0]["values"] == {"CN": None, "US": None}
|
||||
payload.update(sources=[], observations=[], innovation=[])
|
||||
result = compute_history(["CN"], config=ComputeConfiguration.model_validate(payload))
|
||||
assert all(row["values"]["CN"] is None for row in result["compute"])
|
||||
assert result["evidence"] == {}
|
||||
|
||||
|
||||
def fake_db(record=None):
|
||||
db = AsyncMock()
|
||||
db.add = MagicMock()
|
||||
result = MagicMock()
|
||||
result.scalar_one_or_none.return_value = record
|
||||
result.scalar_one.return_value = record
|
||||
db.execute.return_value = result
|
||||
return db
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_fresh_install_uses_defaults_without_writing_database():
|
||||
db = fake_db()
|
||||
assert await read_configuration(db) == default_envelope()
|
||||
db.add.assert_not_called()
|
||||
db.commit.assert_not_awaited()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_series_reads_persisted_override_and_disables_http_cache():
|
||||
envelope = default_envelope()
|
||||
envelope["is_default"] = False
|
||||
envelope["revision"] = "custom"
|
||||
envelope["config"]["observations"][3]["values"]["CN"] = 30
|
||||
response = Response()
|
||||
data = await get_series(
|
||||
response, "CN,US", "global_share", fake_db(SimpleNamespace(payload=envelope))
|
||||
)
|
||||
assert data["compute"][3]["values"]["CN"] == 30
|
||||
assert data["version"] == "custom"
|
||||
assert data["provenance"] == "admin_curated_reports"
|
||||
assert response.headers["Cache-Control"] == "no-store"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_publish_and_audit_are_one_transaction_and_conflicts_do_not_overwrite():
|
||||
record = SimpleNamespace(payload=default_envelope())
|
||||
db = fake_db(record)
|
||||
actor = SimpleNamespace(id=1, username="admin")
|
||||
old = deepcopy(record.payload)
|
||||
result = await publish_configuration(
|
||||
db,
|
||||
revision=old["revision"],
|
||||
config=DEFAULT_CONFIGURATION,
|
||||
actor=actor,
|
||||
)
|
||||
assert result["revision"] != old["revision"]
|
||||
audit = db.add.call_args.args[0]
|
||||
assert isinstance(audit, AuditLog)
|
||||
assert audit.details == {"before": old, "after": result}
|
||||
assert audit.actor_id == actor.id
|
||||
db.commit.assert_awaited_once()
|
||||
with pytest.raises(HTTPException) as failure:
|
||||
await publish_configuration(
|
||||
db, revision=old["revision"], config=DEFAULT_CONFIGURATION, actor=actor
|
||||
)
|
||||
assert failure.value.status_code == 409
|
||||
assert record.payload == result
|
||||
db.rollback.assert_awaited_once()
|
||||
db.commit.assert_awaited_once()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_reset_changes_revision_and_restores_bundled_data():
|
||||
record = SimpleNamespace(
|
||||
payload={**default_envelope(), "revision": "custom", "is_default": False}
|
||||
)
|
||||
result = await publish_configuration(
|
||||
fake_db(record),
|
||||
revision="custom",
|
||||
config=DEFAULT_CONFIGURATION,
|
||||
actor=SimpleNamespace(id=1, username="admin"),
|
||||
reset=True,
|
||||
)
|
||||
assert result["is_default"] is True
|
||||
assert result["config"] == default_config()
|
||||
assert result["revision"] != "custom"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("role", ["user", "viewer", "operator"])
|
||||
async def test_non_admin_cannot_configure(role):
|
||||
with pytest.raises(HTTPException) as failure:
|
||||
await configuration_admin(SimpleNamespace(role=role))
|
||||
assert failure.value.status_code == 403
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_http_configuration_auth_validation_and_public_series():
|
||||
app = FastAPI()
|
||||
app.include_router(router)
|
||||
record = SimpleNamespace(payload=default_envelope())
|
||||
db = fake_db(record)
|
||||
app.dependency_overrides[get_db] = lambda: db
|
||||
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
|
||||
assert (await client.get("/configuration")).status_code == 401
|
||||
assert (await client.get("/series")).status_code == 200
|
||||
app.dependency_overrides[get_current_user] = lambda: SimpleNamespace(
|
||||
id=1,
|
||||
username="admin",
|
||||
role="admin",
|
||||
)
|
||||
loaded = (await client.get("/configuration")).json()
|
||||
invalid = deepcopy(loaded["config"])
|
||||
invalid["observations"][0]["values"]["CN"] = 200
|
||||
response = await client.put(
|
||||
"/configuration",
|
||||
json={
|
||||
"revision": loaded["revision"],
|
||||
"config": invalid,
|
||||
},
|
||||
)
|
||||
assert response.status_code == 422
|
||||
db.commit.assert_not_awaited()
|
||||
response = await client.put(
|
||||
"/configuration",
|
||||
json={
|
||||
"revision": loaded["revision"],
|
||||
"config": loaded["config"],
|
||||
},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
assert response.json()["revision"] != loaded["revision"]
|
||||
assert (
|
||||
await client.post(
|
||||
"/configuration/reset",
|
||||
json={
|
||||
"revision": loaded["revision"],
|
||||
},
|
||||
)
|
||||
).status_code == 409
|
||||
268
backend/tests/test_compute_intelligence.py
Normal file
268
backend/tests/test_compute_intelligence.py
Normal file
@@ -0,0 +1,268 @@
|
||||
import csv
|
||||
import io
|
||||
from datetime import UTC, datetime
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import pytest
|
||||
from fastapi import HTTPException
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
from app.api.v1.compute_intelligence import sample_query, selected_countries
|
||||
from app.services.collectors.epoch_ai import EpochAIGPUCollector
|
||||
from app.services.collectors.top500 import TOP500Collector
|
||||
from app.services.compute_history import compute_history
|
||||
|
||||
|
||||
def epoch_csv(**overrides):
|
||||
row = {
|
||||
"Name": "Cluster Alpha",
|
||||
"Country": "United States of America",
|
||||
"Status": "Existing",
|
||||
"First Operational Date": "2023-05-02",
|
||||
"16-bit OP/s (log)": "18",
|
||||
"Location": "3231 Riverport Rd, Memphis, TN 38109",
|
||||
"latitude": "35",
|
||||
"longitude": "-90",
|
||||
**overrides,
|
||||
}
|
||||
output = io.StringIO()
|
||||
writer = csv.DictWriter(output, fieldnames=list(row))
|
||||
writer.writeheader()
|
||||
writer.writerow(row)
|
||||
return output.getvalue()
|
||||
|
||||
|
||||
def test_epoch_csv_preserves_precision_date_and_missing_values():
|
||||
collector = EpochAIGPUCollector()
|
||||
row = collector.parse_response(epoch_csv())[0]
|
||||
assert row["value"] == 1e18
|
||||
assert row["unit"] == "FLOP/s"
|
||||
assert row["metadata"]["precision"] == "FP16"
|
||||
assert row["metadata"]["location_precision"] == "precise"
|
||||
assert row["metadata"]["raw_country"] == "United States of America"
|
||||
assert row["reference_date"] == "2023-05-02"
|
||||
missing = collector.parse_response(
|
||||
epoch_csv(**{"16-bit OP/s (log)": "", "First Operational Date": ""})
|
||||
)[0]
|
||||
assert missing["value"] is None
|
||||
assert missing["reference_date"] is None
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("location", "country", "expected"),
|
||||
[
|
||||
("United States of America", "United States of America", "country"),
|
||||
("Japan", "Japan", "country"),
|
||||
("Korea (Republic of)", "South Korea", "country"),
|
||||
("Texas", "United States of America", "region"),
|
||||
("Iowa, USA", "United States of America", "region"),
|
||||
("Seoul", "South Korea", "city"),
|
||||
("Paris, France", "France", "city"),
|
||||
("Skolkovo Innovation Center\nMoscow Oblast\nRussia", "Russia", "site"),
|
||||
("Cluster 2, Texas", "United States of America", "site"),
|
||||
("Campus 7, Tokyo", "Japan", "site"),
|
||||
("Building 240, Argonne", "United States of America", "site"),
|
||||
("7", "United States of America", "unknown"),
|
||||
("Jülich Supercomputing Centre\n52428 Jülich, Germany", "Germany", "precise"),
|
||||
("35.380924,139.625008", "Japan", "precise"),
|
||||
('33°24\'28.7"N 112°21\'54.3"W', "United States of America", "precise"),
|
||||
("latitude: 35.0 longitude: 181.0", "United States of America", "unknown"),
|
||||
('99°99\'00"N 200°00\'00"W', "United States of America", "unknown"),
|
||||
("3231 Riverport Rd, Memphis, TN 38109", "United States of America", "precise"),
|
||||
],
|
||||
)
|
||||
def test_epoch_location_precision_follows_location_field(location, country, expected):
|
||||
row = EpochAIGPUCollector().parse_response(
|
||||
epoch_csv(Location=location, Country=country)
|
||||
)[0]
|
||||
|
||||
assert row["metadata"]["location_precision"] == expected
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("location", "expected"),
|
||||
[
|
||||
({"lat": 35.0, "lng": 139.0}, "precise"),
|
||||
({"latitude": 35.0, "longitude": 181.0}, "unknown"),
|
||||
({"lat": 35.0}, "unknown"),
|
||||
({"lng": 139.0}, "unknown"),
|
||||
({"lat": "invalid", "lng": 139.0}, "unknown"),
|
||||
],
|
||||
)
|
||||
def test_epoch_coordinate_object_requires_complete_valid_pair(location, expected):
|
||||
from app.services.epoch_location import assess_epoch_location
|
||||
|
||||
assert assess_epoch_location(location).precision == expected
|
||||
|
||||
|
||||
def test_epoch_anonymous_rows_are_not_geocoded():
|
||||
row = EpochAIGPUCollector().parse_response(epoch_csv(Name="Anonymized 1", Country="China"))[0]
|
||||
assert row["latitude"] is None and row["longitude"] is None
|
||||
assert row["metadata"]["anonymous"] and row["metadata"]["rounded"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"payload",
|
||||
["<html>blocked</html>", "Name,Country\nA,US", epoch_csv(**{"16-bit OP/s (log)": "nan"})],
|
||||
)
|
||||
def test_epoch_invalid_payload_fails_closed(payload):
|
||||
with pytest.raises(ValueError):
|
||||
EpochAIGPUCollector().parse_response(payload)
|
||||
|
||||
|
||||
def test_epoch_duplicate_identity_rejected():
|
||||
payload = epoch_csv()
|
||||
with pytest.raises(ValueError, match="identity collision"):
|
||||
EpochAIGPUCollector().parse_response(payload + payload.splitlines()[1] + "\n")
|
||||
|
||||
|
||||
def test_anonymous_source_rows_preserve_multiplicity_without_claiming_identity():
|
||||
payload = epoch_csv(Name="Anonymized Chinese System", Country="China")
|
||||
rows = EpochAIGPUCollector().parse_response(payload + payload.splitlines()[1] + "\n")
|
||||
assert len(rows) == 2
|
||||
assert rows[0]["source_id"] != rows[1]["source_id"]
|
||||
assert all(row["metadata"]["identity_scope"] == "snapshot_record" for row in rows)
|
||||
|
||||
|
||||
def test_top500_units_and_stable_identity():
|
||||
collector = TOP500Collector()
|
||||
html = '<table id="top500"><tr><th>Rank</th><th>System</th><th>Cores</th><th>Rmax [TFlop/s]</th><th>Rpeak [PFlop/s]</th><th>Power</th></tr><tr><td>{rank}</td><td><a href="/system/123/">Test</a><br>Lab<br>USA</td><td>10</td><td>1,500</td><td>2</td><td>1</td></tr></table>'
|
||||
first = collector.parse_response(
|
||||
html.format(rank=1), "https://top500.org/lists/top500/2025/11/"
|
||||
)[0]
|
||||
second = collector.parse_response(
|
||||
html.format(rank=2), "https://top500.org/lists/top500/2026/06/"
|
||||
)[0]
|
||||
assert first["source_id"] == second["source_id"] == "top500_system_123"
|
||||
assert float(first["value"]) == 1.5e15
|
||||
assert first["metadata"]["rpeak_flops"] == 2e15
|
||||
assert first["reference_date"] == "2025-11-01"
|
||||
assert second["reference_date"] == "2026-06-01"
|
||||
with pytest.raises(ValueError, match="date missing"):
|
||||
collector.parse_response(html.format(rank=1))
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_large_source_loss_does_not_create_snapshot():
|
||||
db = AsyncMock()
|
||||
db.scalar.return_value = 500
|
||||
db.add = MagicMock()
|
||||
with pytest.raises(ValueError, match="40%"):
|
||||
await TOP500Collector()._create_snapshot(db, 1, [{}] * 2, datetime.now(UTC))
|
||||
db.add.assert_not_called()
|
||||
db.commit.assert_not_awaited()
|
||||
|
||||
|
||||
def test_history_missing_years_and_denominators():
|
||||
global_data = compute_history(["CN", "US"])
|
||||
assert global_data["compute"][3]["values"] == {"CN": 31, "US": 41}
|
||||
assert global_data["compute"][-1]["values"] == {"CN": None, "US": None}
|
||||
bilateral = compute_history(["CN", "US"], "bilateral_share")
|
||||
assert sum(bilateral["compute"][3]["values"].values()) == 100
|
||||
assert bilateral["denominator"] == "US+CN"
|
||||
absolute = compute_history(["US", "CN"], "absolute")
|
||||
assert absolute["compute"][3]["values"] == {"US": None, "CN": 435}
|
||||
assert global_data["innovation"][-2]["value"] == 17.95
|
||||
assert global_data["innovation"][-1]["value"] is None
|
||||
assert global_data["evidence"]["innovation-2026"]["published_at"] == "2026-07-17"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_snapshot_query_uses_observation_cutoff_not_location():
|
||||
query = await sample_query(AsyncMock(), ["CN"], 2023)
|
||||
sql = str(query.compile(dialect=postgresql.dialect(), compile_kwargs={"literal_binds": True}))
|
||||
assert "2024-01-01" in sql
|
||||
assert "row_number() OVER (PARTITION BY" in sql
|
||||
assert "compute_contract" in sql and "中国" in sql
|
||||
assert "latitude" not in sql and "longitude" not in sql
|
||||
|
||||
|
||||
def test_country_input_is_bounded_and_deduplicated():
|
||||
assert selected_countries("CN,CN,US") == ["CN", "US"]
|
||||
assert selected_countries("CN,US,EUROPE,APAC,OTHER")[-1] == "OTHER"
|
||||
with pytest.raises(HTTPException):
|
||||
selected_countries("invalid")
|
||||
with pytest.raises(HTTPException, match="overlapping"):
|
||||
selected_countries("EUROPE,DE")
|
||||
|
||||
|
||||
def test_catalog_supports_global_participants_without_inventing_history():
|
||||
from app.services.compute_catalog import (
|
||||
CATALOG,
|
||||
COMPARISON_GROUPS,
|
||||
COUNTRIES,
|
||||
source_country_names,
|
||||
source_country_participants,
|
||||
)
|
||||
|
||||
assert {"CN", "US", "JP", "KR", "IN", "DE", "FR", "GB", "AU", "BR", "ZA"}.issubset(
|
||||
{entry["code"] for entry in CATALOG}
|
||||
)
|
||||
assert not {"TW", "HK", "MO", "XK"}.intersection(entry["code"] for entry in CATALOG)
|
||||
owners = source_country_names(["CN"])
|
||||
assert {"中国", "中国(台湾)", "中国(香港)", "中国(澳门)"}.issubset(owners)
|
||||
assert source_country_participants(["CN"])["中国(台湾)"] == "CN"
|
||||
assert [entry["code"] for entry in CATALOG[:5]] == ["CN", "US", "EUROPE", "APAC", "OTHER"]
|
||||
assert [entry["kind"] for entry in CATALOG[:5]] == [
|
||||
"country",
|
||||
"country",
|
||||
"region",
|
||||
"region",
|
||||
"region",
|
||||
]
|
||||
group_members = [
|
||||
member
|
||||
for _code, _name, _name_en, members, _anchor in COMPARISON_GROUPS
|
||||
for member in members
|
||||
]
|
||||
assert len(group_members) == len(set(group_members))
|
||||
assert set(group_members) == set(COUNTRIES)
|
||||
data = compute_history(["JP", "KR"])
|
||||
assert [row["values"]["JP"] for row in data["compute"][:4]] == [6, 5, 4, 4]
|
||||
assert all(row["values"]["KR"] is None for row in data["compute"])
|
||||
assert all(row["value"] is None for row in data["innovation"])
|
||||
assert len(data["catalog"]) > 30
|
||||
|
||||
|
||||
def test_region_history_requires_complete_same_year_member_coverage():
|
||||
data = compute_history(["CN", "US", "EUROPE", "APAC", "OTHER"])
|
||||
row_2023 = next(row for row in data["compute"] if row["year"] == 2023)
|
||||
assert row_2023["values"]["CN"] == 31
|
||||
assert row_2023["values"]["US"] == 41
|
||||
assert row_2023["values"]["EUROPE"] is None
|
||||
assert row_2023["values"]["APAC"] is None
|
||||
assert row_2023["values"]["OTHER"] is None
|
||||
assert row_2023["coverage"]["EUROPE"]["status"] == "incomplete"
|
||||
assert "DE" in row_2023["coverage"]["EUROPE"]["missing_members"]
|
||||
assert data["default_participants"] == ["CN", "US", "EUROPE", "APAC", "OTHER"]
|
||||
|
||||
|
||||
def test_anonymous_records_never_enter_the_location_repair_queue():
|
||||
from app.api.v1.visualization import convert_compute_centers_to_geojson
|
||||
from app.models.collected_data import CollectedData
|
||||
|
||||
record = CollectedData(
|
||||
source="epoch_ai_gpu",
|
||||
source_id="anonymous",
|
||||
name="Anonymous",
|
||||
extra_data={"compute_contract": "v1", "anonymous": True},
|
||||
)
|
||||
result = convert_compute_centers_to_geojson([record])
|
||||
assert result["features"] == []
|
||||
assert result["unresolved"] == []
|
||||
assert result["withheld_count"] == 1
|
||||
|
||||
|
||||
def test_absolute_report_year_mapping_and_bilateral_denominator_ignore_other_countries():
|
||||
absolute = compute_history(["CN"], "absolute")
|
||||
observed = {row["year"]: row for row in absolute["compute"]}
|
||||
assert observed[2022]["values"]["CN"] == 302
|
||||
assert observed[2022]["evidence_id"] == "caict-2023"
|
||||
assert observed[2023]["evidence_id"] == "caict-2024"
|
||||
bilateral = compute_history(["CN", "US", "JP"], "bilateral_share")
|
||||
values = next(row["values"] for row in bilateral["compute"] if row["year"] == 2023)
|
||||
assert values == {"CN": 43.06, "US": 56.94, "JP": None}
|
||||
for row in absolute["compute"]:
|
||||
if row["evidence_id"]:
|
||||
report = absolute["evidence"][row["evidence_id"]]
|
||||
assert report["url"].startswith("https://www.caict.ac.cn/")
|
||||
18
backend/tests/test_compute_verified_sites.py
Normal file
18
backend/tests/test_compute_verified_sites.py
Normal file
@@ -0,0 +1,18 @@
|
||||
from app.services.compute_center_verified_sites import get_verified_compute_site
|
||||
|
||||
|
||||
def test_registry_requires_exact_facility_identity_not_operator_name():
|
||||
assert get_verified_compute_site("epoch_ai_gpu", "NVIDIA") is None
|
||||
assert get_verified_compute_site("top500", "epoch_argonne_nl_sophia__formerly_theta_") is None
|
||||
|
||||
|
||||
def test_official_campus_anchors_preserve_evidence_and_site_level_precision():
|
||||
site = get_verified_compute_site("epoch_ai_gpu", "epoch_jamstec_zettascaler_2_2_gyoukou")
|
||||
assert site["city"] == "Yokohama"
|
||||
assert site["precision"] == "site"
|
||||
assert site["coordinate_basis"] == "official_access_map_center"
|
||||
assert site["sources"]
|
||||
site["latitude"] = 0
|
||||
assert get_verified_compute_site(
|
||||
"epoch_ai_gpu", "epoch_jamstec_zettascaler_2_2_gyoukou"
|
||||
)["latitude"] != 0
|
||||
@@ -1,4 +1,5 @@
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
@@ -8,6 +9,10 @@ from app.services import earth_boundaries
|
||||
|
||||
|
||||
def write_geojson(path, name="Test"):
|
||||
names = [name]
|
||||
if name == "earth_admin0_boundaries":
|
||||
names = ["China", "India", "Japan", "Palestine", "Taiwan", "Hong Kong", "Macao", "Serbia", "Kosovo"]
|
||||
codes = {"China": "CHN", "India": "IND", "Japan": "JPN", "Palestine": "PSE", "Taiwan": "TWN", "Hong Kong": "HKG", "Macao": "MAC", "Serbia": "SRB", "Kosovo": "KOS"}
|
||||
path.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
@@ -15,12 +20,17 @@ def write_geojson(path, name="Test"):
|
||||
"features": [
|
||||
{
|
||||
"type": "Feature",
|
||||
"properties": {"name": name},
|
||||
"properties": {
|
||||
"NAME": f"source renamed {item}" if item in {"Taiwan", "Kosovo"} else item,
|
||||
"ISO_A3": "-99" if item == "Kosovo" else codes.get(item, "TEST"),
|
||||
"ADM0_A3": "XKS" if item == "Kosovo" else codes.get(item, "TEST"),
|
||||
},
|
||||
"geometry": {
|
||||
"type": "Polygon",
|
||||
"coordinates": [[[-1, -1], [1, -1], [1, 1], [-1, 1], [-1, -1]]],
|
||||
},
|
||||
}
|
||||
for item in names
|
||||
],
|
||||
}
|
||||
),
|
||||
@@ -37,9 +47,14 @@ def patch_paths(monkeypatch, tmp_path):
|
||||
config = repo / "config/earth-boundary-sources.local.json"
|
||||
example = repo / "config/earth-boundary-sources.example.json"
|
||||
policy = repo / "config/earth-boundary-pov-policy.china-v1.json"
|
||||
geometry = repo / "resources/earth/boundary-pov-geometry.china-v1.json"
|
||||
for path in (source_dir, boundary_dir, pmtiles.parent, legacy.parent, config.parent):
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
policy.write_text('{"productionTileFormat":"pmtiles+mvt"}\n', encoding="utf-8")
|
||||
source_policy = Path(__file__).resolve().parents[2] / "config/earth-boundary-pov-policy.china-v1.json"
|
||||
policy.write_text(source_policy.read_text(encoding="utf-8"), encoding="utf-8")
|
||||
geometry.parent.mkdir(parents=True, exist_ok=True)
|
||||
source_geometry = Path(__file__).resolve().parents[2] / "resources/earth/boundary-pov-geometry.china-v1.json"
|
||||
geometry.write_bytes(source_geometry.read_bytes())
|
||||
example.write_text('{"collectorConfigs":{}}\n', encoding="utf-8")
|
||||
monkeypatch.setattr(earth_boundaries, "REPO_ROOT", repo)
|
||||
monkeypatch.setattr(earth_boundaries, "SOURCE_OUTPUT_DIR", source_dir)
|
||||
@@ -49,10 +64,11 @@ def patch_paths(monkeypatch, tmp_path):
|
||||
monkeypatch.setattr(earth_boundaries, "BOUNDARY_OUTPUT_DIR", boundary_dir)
|
||||
monkeypatch.setattr(earth_boundaries, "BOUNDARY_MANIFEST_PATH", boundary_dir / "manifest.json")
|
||||
monkeypatch.setattr(earth_boundaries, "PMTILES_ARTIFACT_PATH", pmtiles)
|
||||
monkeypatch.setattr(earth_boundaries, "LEGACY_GEOJSON_PATH", legacy)
|
||||
monkeypatch.setattr(earth_boundaries, "POLICY_FALLBACK_PATH", legacy)
|
||||
monkeypatch.setattr(earth_boundaries, "LOCAL_CONFIG_PATH", config)
|
||||
monkeypatch.setattr(earth_boundaries, "EXAMPLE_CONFIG_PATH", example)
|
||||
monkeypatch.setattr(earth_boundaries, "POV_POLICY_PATH", policy)
|
||||
monkeypatch.setattr(earth_boundaries, "POV_GEOMETRY_PATH", geometry)
|
||||
return {
|
||||
"repo": repo,
|
||||
"config": config,
|
||||
@@ -62,27 +78,73 @@ def patch_paths(monkeypatch, tmp_path):
|
||||
}
|
||||
|
||||
|
||||
def write_fallback(path, policy_path, geometry_path):
|
||||
policy = earth_boundaries._read_json(policy_path)
|
||||
geometry = earth_boundaries._read_json(geometry_path)
|
||||
coverage = {
|
||||
"profile": "china-pov-v1",
|
||||
"sha256": earth_boundaries._stable_json_hash(policy),
|
||||
"geometrySha256": earth_boundaries._stable_json_hash(geometry),
|
||||
"reviewedGeometrySourceCommit": geometry["sourceCommit"],
|
||||
"applied": True,
|
||||
"appliedRuleIds": ["china-taiwan-penghu", "china-hong-kong", "china-macao", "kosovo"],
|
||||
}
|
||||
path.write_text(json.dumps({"type": "FeatureCollection", "features": [], "povPolicy": coverage}), encoding="utf-8")
|
||||
|
||||
|
||||
def test_boundary_status_uses_legacy_provider_when_pmtiles_missing(monkeypatch, tmp_path):
|
||||
paths = patch_paths(monkeypatch, tmp_path)
|
||||
write_fallback(paths["legacy"], earth_boundaries.POV_POLICY_PATH, earth_boundaries.POV_GEOMETRY_PATH)
|
||||
|
||||
status = earth_boundaries.get_boundary_status()
|
||||
|
||||
assert status["provider"] == "policy-geojson-fallback"
|
||||
assert status["fallback_available"] is True
|
||||
assert status["high_precision_ready"] is False
|
||||
|
||||
|
||||
def test_boundary_status_rejects_unverified_fallback(monkeypatch, tmp_path):
|
||||
paths = patch_paths(monkeypatch, tmp_path)
|
||||
write_geojson(paths["legacy"])
|
||||
|
||||
status = earth_boundaries.get_boundary_status()
|
||||
|
||||
assert status["provider"] == "legacy-geojson"
|
||||
assert status["fallback_available"] is True
|
||||
assert status["high_precision_ready"] is False
|
||||
assert status["provider"] == "missing"
|
||||
assert status["fallback_available"] is False
|
||||
|
||||
|
||||
def test_boundary_status_prefers_high_precision_when_manifest_and_pmtiles_exist(monkeypatch, tmp_path):
|
||||
paths = patch_paths(monkeypatch, tmp_path)
|
||||
write_geojson(paths["legacy"])
|
||||
write_fallback(paths["legacy"], earth_boundaries.POV_POLICY_PATH, earth_boundaries.POV_GEOMETRY_PATH)
|
||||
paths["pmtiles"].write_bytes(b"pmtiles")
|
||||
paths["manifest"].write_text('{"tileProvider":"pmtiles-mvt"}\n', encoding="utf-8")
|
||||
(paths["manifest"].parent / "hover-index.geojson").write_text("{}", encoding="utf-8")
|
||||
coverage = {
|
||||
"profile": "china-pov-v1",
|
||||
"sha256": earth_boundaries._stable_json_hash(earth_boundaries._read_json(earth_boundaries.POV_POLICY_PATH)),
|
||||
"applied": True,
|
||||
"appliedRuleIds": ["china-taiwan-penghu", "china-hong-kong", "china-macao", "kosovo"],
|
||||
"reviewedGeometrySourceCommit": earth_boundaries._read_json(earth_boundaries.POV_GEOMETRY_PATH)["sourceCommit"],
|
||||
"geometrySha256": earth_boundaries._stable_json_hash(earth_boundaries._read_json(earth_boundaries.POV_GEOMETRY_PATH)),
|
||||
}
|
||||
paths["manifest"].write_text(json.dumps({"tileProvider": "pmtiles-mvt", "hoverIndex": "hover-index.geojson", "povPolicy": coverage}), encoding="utf-8")
|
||||
|
||||
status = earth_boundaries.get_boundary_status()
|
||||
|
||||
assert status["provider"] == "pmtiles-mvt"
|
||||
assert status["high_precision_ready"] is True
|
||||
assert status["policy_publishable"] is False
|
||||
|
||||
|
||||
def test_boundary_status_rejects_unverified_high_precision(monkeypatch, tmp_path):
|
||||
paths = patch_paths(monkeypatch, tmp_path)
|
||||
write_fallback(paths["legacy"], earth_boundaries.POV_POLICY_PATH, earth_boundaries.POV_GEOMETRY_PATH)
|
||||
paths["pmtiles"].write_bytes(b"pmtiles")
|
||||
paths["manifest"].write_text('{"tileProvider":"pmtiles-mvt","povPolicy":{"applied":false}}', encoding="utf-8")
|
||||
|
||||
status = earth_boundaries.get_boundary_status()
|
||||
|
||||
assert status["provider"] == "policy-geojson-fallback"
|
||||
assert status["high_precision_ready"] is False
|
||||
|
||||
|
||||
def test_save_boundary_config_writes_local_config(monkeypatch, tmp_path):
|
||||
@@ -130,6 +192,22 @@ async def test_build_reports_missing_tools_after_source_artifacts(monkeypatch, t
|
||||
assert status["high_precision_ready"] is True
|
||||
assert (paths["repo"] / "data/earth-boundary-sources/manifest.json").exists()
|
||||
assert paths["manifest"].exists()
|
||||
manifest = json.loads(paths["manifest"].read_text(encoding="utf-8"))
|
||||
assert manifest["povPolicy"]["applied"] is True
|
||||
assert manifest["povPolicy"]["coverage"] == "partial"
|
||||
assert manifest["povPolicy"]["publicationReady"] is False
|
||||
assert "china-south-china-sea-islands" in manifest["povPolicy"]["unresolvedRuleIds"]
|
||||
hover = json.loads((paths["manifest"].parent / manifest["hoverIndex"]).read_text(encoding="utf-8"))
|
||||
by_rule = {
|
||||
feature["properties"].get("PLANET_POLICY_RULE"): feature
|
||||
for feature in hover["features"]
|
||||
}
|
||||
assert by_rule["china-hong-kong"]["properties"]["ISO_A3"] == "CHN"
|
||||
assert by_rule["china-macao"]["properties"]["ISO_A3"] == "CHN"
|
||||
assert all(
|
||||
feature["properties"].get("ADM0_A3") not in {"TWN", "KOS", "XKS"}
|
||||
for feature in hover["features"]
|
||||
)
|
||||
assert any(phase == "download" for _progress, phase, _message in progress_events)
|
||||
|
||||
|
||||
@@ -175,3 +253,23 @@ def test_earth_boundary_collectors_are_not_registered_as_datasources():
|
||||
assert removed.isdisjoint(DEFAULT_DATASOURCES)
|
||||
for source in removed:
|
||||
assert collector_registry.get(source) is None
|
||||
|
||||
|
||||
def test_static_policy_fallback_contains_reviewed_jeju_polygon():
|
||||
repo = Path(__file__).resolve().parents[2]
|
||||
path = repo / "frontend/public/earth/data/countries-admin0.china-pov-v1.geojson"
|
||||
payload = json.loads(path.read_text(encoding="utf-8"))
|
||||
matches = [
|
||||
feature for feature in payload["features"]
|
||||
if feature["properties"].get("PLANET_GEOMETRY_SUPPLEMENT") == "jeju-island"
|
||||
]
|
||||
|
||||
assert len(matches) == 1
|
||||
jeju = matches[0]
|
||||
assert jeju["properties"]["ISO_A3"] == "KOR"
|
||||
assert jeju["properties"]["PLANET_SOURCE_NE_ID"] == 1159320985
|
||||
assert payload["povPolicy"]["geometrySupplement"]["jejuComponent"] == 3
|
||||
ring = jeju["geometry"]["coordinates"][0]
|
||||
assert len(ring) == 61
|
||||
assert min(point[0] for point in ring) < 126.53 < max(point[0] for point in ring)
|
||||
assert min(point[1] for point in ring) < 33.36 < max(point[1] for point in ring)
|
||||
|
||||
@@ -63,6 +63,21 @@ def test_build_earth_update_maps_derived_tables_to_layers():
|
||||
assert vessel_update is not None
|
||||
assert vessel_update["source"] == "vessel_position"
|
||||
assert vessel_update["layers"] == ["vessels"]
|
||||
assert vessel_update["refresh_strategy"] == "reload"
|
||||
|
||||
|
||||
def test_vessel_stream_changes_do_not_request_full_layer_rebuilds():
|
||||
from app.services.earth_db_change_listener import PendingEarthDbChange
|
||||
|
||||
for source in ("aisstream_vessels", "barentswatch_vessels"):
|
||||
update = build_earth_update_from_db_payload({
|
||||
"source": source, "table": "vessel_current_state", "operation": "UPDATE",
|
||||
})
|
||||
assert update["refresh_strategy"] == "delta"
|
||||
pending = PendingEarthDbChange(source=source, layers=["vessels"], refresh_strategy="delta")
|
||||
pending.add({"operation": "UPDATE"})
|
||||
pending.add({"operation": "DELETE"})
|
||||
assert pending.refresh_strategy == "reload"
|
||||
|
||||
|
||||
def test_build_earth_update_maps_interactable_delete_to_delta():
|
||||
|
||||
41
backend/tests/test_earth_geography_policy.py
Normal file
41
backend/tests/test_earth_geography_policy.py
Normal file
@@ -0,0 +1,41 @@
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from app.core.earth_geography_policy import display_country, display_iso_a3
|
||||
|
||||
|
||||
def test_missing_deployment_policy_has_an_actionable_error(monkeypatch, tmp_path):
|
||||
from app.core import earth_geography_policy as policy
|
||||
|
||||
policy._policy_country_by_code.cache_clear()
|
||||
monkeypatch.setattr(policy, "_POLICY_PATH", tmp_path / "missing-policy.json")
|
||||
try:
|
||||
with pytest.raises(FileNotFoundError, match="EARTH_DISPLAY_POLICY_MISSING"):
|
||||
policy.display_country("China")
|
||||
finally:
|
||||
policy._policy_country_by_code.cache_clear()
|
||||
|
||||
|
||||
def test_display_country_groups_china_regions_and_kosovo() -> None:
|
||||
for source in ("Taiwan", "TWN", "Hong Kong", "HKG", "Macao", "MAC", "中国(台湾)"):
|
||||
assert display_country(source) == "中国"
|
||||
assert display_iso_a3(source) == "CHN"
|
||||
for source in ("Kosovo", "XK", "SRB"):
|
||||
assert display_country(source) == "塞尔维亚"
|
||||
assert display_iso_a3(source) == "SRB"
|
||||
|
||||
|
||||
def test_display_country_keeps_other_country_and_rejects_unknown() -> None:
|
||||
assert display_country("Japan") == "日本"
|
||||
assert display_iso_a3("Japan") == "JPN"
|
||||
assert display_country("unexpected non-country") is None
|
||||
|
||||
|
||||
def test_country_projection_agrees_with_versioned_boundary_rules() -> None:
|
||||
path = Path(__file__).resolve().parents[2] / "config/earth-boundary-pov-policy.china-v1.json"
|
||||
policy = json.loads(path.read_text(encoding="utf-8"))
|
||||
for rule in policy["rules"]:
|
||||
for code in rule.get("sourceIsoA3", []):
|
||||
assert display_iso_a3(code) == rule.get("targetIsoA3", rule.get("parentIsoA3"))
|
||||
86
backend/tests/test_earth_news_encoding.py
Normal file
86
backend/tests/test_earth_news_encoding.py
Normal file
@@ -0,0 +1,86 @@
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from app.services.earth_news import (
|
||||
NewsFeedEndpoint,
|
||||
NewsFeedSource,
|
||||
ParsedNewsItem,
|
||||
_fetch_single_feed_url,
|
||||
_serialize_item,
|
||||
)
|
||||
|
||||
|
||||
async def fetch_feed(
|
||||
content: bytes, content_type: str = "text/xml"
|
||||
) -> tuple[NewsFeedSource, list[ParsedNewsItem], str | None, dict[str, Any]]:
|
||||
source = NewsFeedSource(
|
||||
id="ebrun",
|
||||
name="亿邦动力",
|
||||
region="asia-pacific",
|
||||
feed_url="https://example.com/rss.xml",
|
||||
homepage_url="https://example.com/",
|
||||
)
|
||||
feed = NewsFeedEndpoint(id="b2c", name="B2C", url=source.feed_url)
|
||||
transport = httpx.MockTransport(
|
||||
lambda request: httpx.Response(
|
||||
200, content=content, headers={"content-type": content_type}
|
||||
)
|
||||
)
|
||||
async with httpx.AsyncClient(transport=transport) as client:
|
||||
return await _fetch_single_feed_url(client, source, feed)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
("encoding", "declaration", "content_type"),
|
||||
[
|
||||
("gbk", 'encoding="GBK"', "text/xml"),
|
||||
("gb2312", "encoding='GB2312'", "application/rss+xml"),
|
||||
("gb18030", 'encoding="GB18030"', "text/xml"),
|
||||
("gbk", "", "text/xml; charset=gbk"),
|
||||
("utf-8", 'encoding="UTF-8"', "application/rss+xml"),
|
||||
("utf-8", "", "text/xml"),
|
||||
("utf-8-sig", 'encoding="UTF-8"', "text/xml"),
|
||||
("utf-16", 'encoding="UTF-16"', "text/xml"),
|
||||
("utf-32", 'encoding="UTF-32"', "text/xml"),
|
||||
],
|
||||
)
|
||||
async def test_feed_encoding_preserves_chinese_in_display(
|
||||
encoding: str, declaration: str, content_type: str
|
||||
):
|
||||
title = "618电商战报:零售额增长266%"
|
||||
summary = "亿邦动力:跨境电商订单增长,物流效率提升。"
|
||||
xml = f'''<?xml version="1.0" {declaration}?>
|
||||
<rss version="2.0"><channel><item>
|
||||
<title><![CDATA[{title}]]></title>
|
||||
<description><![CDATA[<p>{summary}</p>]]></description>
|
||||
<link>https://example.com/story</link>
|
||||
</item></channel></rss>'''
|
||||
|
||||
_, items, error, health = await fetch_feed(xml.encode(encoding), content_type)
|
||||
|
||||
assert error is None
|
||||
assert health["ok"] is True
|
||||
assert len(items) == 1
|
||||
assert items[0].title == title
|
||||
assert items[0].summary == summary
|
||||
payload = _serialize_item(items[0], active_region="asia-pacific", locale="zh-CN")
|
||||
assert payload["display_title"] == title
|
||||
assert payload["display_summary"] == summary
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("encoding", ["UTF-8", "unknown-encoding"])
|
||||
async def test_invalid_feed_encoding_is_reported_instead_of_storing_garbled_text(encoding: str):
|
||||
xml = f'''<?xml version="1.0" encoding="{encoding}"?>
|
||||
<rss><channel><item><title>电商新闻</title>
|
||||
<link>https://example.com/story</link></item></channel></rss>'''
|
||||
|
||||
_, items, error, health = await fetch_feed(xml.encode("gbk"))
|
||||
|
||||
assert items == []
|
||||
assert error
|
||||
assert health["ok"] is False
|
||||
assert health["status"] == "format_error"
|
||||
219
backend/tests/test_llm_provider_catalog.py
Normal file
219
backend/tests/test_llm_provider_catalog.py
Normal file
@@ -0,0 +1,219 @@
|
||||
from copy import deepcopy
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from app.services import llm_model_catalog as discovery
|
||||
from app.services import llm_provider_catalog as catalog
|
||||
|
||||
REAL_CLIENT = httpx.AsyncClient
|
||||
|
||||
|
||||
def mock_http(monkeypatch, handler):
|
||||
transport = httpx.MockTransport(handler)
|
||||
monkeypatch.setattr(
|
||||
discovery.httpx, "AsyncClient", lambda **kw: REAL_CLIENT(transport=transport, **kw)
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
"provider,path,auth",
|
||||
[
|
||||
("minimax", "/anthropic/v1/models", "x-api-key"),
|
||||
("anthropic", "/v1/models", "x-api-key"),
|
||||
("openai", "/v1/models", "authorization"),
|
||||
("deepseek", "/v1/models", "authorization"),
|
||||
("alibaba", "/api/v1/models", "authorization"),
|
||||
("moonshotai", "/v1/models", "authorization"),
|
||||
("openrouter", "/api/v1/models", "authorization"),
|
||||
("opencode-go", "/zen/go/v1/models", "authorization"),
|
||||
("ollama", "/api/tags", "authorization"),
|
||||
],
|
||||
)
|
||||
async def test_official_catalog_requests(monkeypatch, provider, path, auth):
|
||||
preset = catalog.get_fallback_llm_provider_preset(provider)
|
||||
|
||||
def handle(request):
|
||||
assert request.url.path == path
|
||||
assert request.url.host == httpx.URL(preset["base_url"]).host
|
||||
assert request.headers[auth] == ("test-key" if auth == "x-api-key" else "Bearer test-key")
|
||||
if provider == "alibaba":
|
||||
assert request.url.params["capabilities"] == "TG"
|
||||
return httpx.Response(
|
||||
200, json={"output": {"total": 1, "models": [{"model": "latest"}]}}
|
||||
)
|
||||
if provider == "ollama":
|
||||
return httpx.Response(200, json={"models": [{"name": "latest:7b"}]})
|
||||
return httpx.Response(200, json={"data": [{"id": "latest"}]})
|
||||
|
||||
mock_http(monkeypatch, handle)
|
||||
result = await catalog.refresh_llm_provider_preset(provider, api_key="test-key")
|
||||
assert result["models"] == (["latest:7b"] if provider == "ollama" else ["latest"])
|
||||
assert result["base_url"] == preset["base_url"]
|
||||
assert "test-key" not in str(result)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"provider,base,api,path",
|
||||
[
|
||||
(
|
||||
"minimax",
|
||||
"https://api.minimax.io/anthropic/v1/",
|
||||
"anthropic-messages",
|
||||
"/anthropic/v1/models",
|
||||
),
|
||||
("anthropic", "https://api.anthropic.com", "anthropic-messages", "/v1/models"),
|
||||
(
|
||||
"alibaba",
|
||||
"https://workspace.cn-beijing.maas.aliyuncs.com/compatible-mode/v1",
|
||||
"openai-completions",
|
||||
"/api/v1/models",
|
||||
),
|
||||
(
|
||||
"alibaba",
|
||||
"https://dashscope-intl.aliyuncs.com/compatible-mode/v1",
|
||||
"openai-completions",
|
||||
"/api/v1/models",
|
||||
),
|
||||
("alibaba", "https://custom.test/gateway/v1", "openai-completions", "/gateway/v1/models"),
|
||||
("moonshotai", "https://api.moonshot.cn/v1", "openai-completions", "/v1/models"),
|
||||
("ollama", "http://localhost:11434/api", "ollama-generate", "/api/tags"),
|
||||
],
|
||||
)
|
||||
def test_urls_preserve_region_and_gateway(provider, base, api, path):
|
||||
url = httpx.URL(discovery.model_catalog_url(provider, base, api))
|
||||
assert url.host == httpx.URL(base).host
|
||||
assert url.path == path
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_anthropic_pagination_and_release_order(monkeypatch):
|
||||
seen = []
|
||||
|
||||
def handle(request):
|
||||
seen.append(request.url.params.get("after_id"))
|
||||
if len(seen) == 1:
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"data": [{"id": "old", "created_at": "2025-01-01T00:00:00Z"}],
|
||||
"has_more": True,
|
||||
"last_id": "old",
|
||||
},
|
||||
)
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={"data": [{"id": "new", "created_at": "2026-06-01T00:00:00Z"}], "has_more": False},
|
||||
)
|
||||
|
||||
mock_http(monkeypatch, handle)
|
||||
result = await catalog.refresh_llm_provider_preset("anthropic", "test-key")
|
||||
assert seen == [None, "old"]
|
||||
assert result["models"] == ["new", "old"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_dashscope_pagination(monkeypatch):
|
||||
def handle(request):
|
||||
page = int(request.url.params["page_no"])
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"output": {
|
||||
"total": 2,
|
||||
"models": [
|
||||
{"model": f"page-{page}", "published_time": f"2026-06-0{page} 00:00:00"}
|
||||
],
|
||||
}
|
||||
},
|
||||
)
|
||||
|
||||
mock_http(monkeypatch, handle)
|
||||
result = await catalog.refresh_llm_provider_preset("alibaba", "test-key")
|
||||
assert result["models"] == ["page-2", "page-1"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
"payload",
|
||||
[
|
||||
{},
|
||||
{"data": []},
|
||||
{"data": [None]},
|
||||
{"data": [{}]},
|
||||
{"data": [{"id": "same"}], "has_more": True, "last_id": "same"},
|
||||
],
|
||||
)
|
||||
async def test_invalid_or_incomplete_catalog_fails(monkeypatch, payload):
|
||||
mock_http(monkeypatch, lambda request: httpx.Response(200, json=payload))
|
||||
with pytest.raises(discovery.LLMProviderCatalogError):
|
||||
await catalog.refresh_llm_provider_preset("minimax", "test-key")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_all_models_retained_and_defaults_unchanged(monkeypatch):
|
||||
before = deepcopy(catalog.FALLBACK_LLM_PROVIDER_PRESETS)
|
||||
rows = [{"id": f"model-{i}", "created": i} for i in range(140)]
|
||||
mock_http(monkeypatch, lambda request: httpx.Response(200, json={"data": rows}))
|
||||
result = await catalog.refresh_llm_provider_preset("openai", "test-key")
|
||||
assert len(result["models"]) == 140
|
||||
assert result["models"][0] == "model-139"
|
||||
assert catalog.FALLBACK_LLM_PROVIDER_PRESETS == before
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_empty_ollama_catalog_is_valid(monkeypatch):
|
||||
mock_http(monkeypatch, lambda request: httpx.Response(200, json={"models": []}))
|
||||
result = await catalog.refresh_llm_provider_preset("ollama")
|
||||
assert result["models"] == []
|
||||
assert result["model"] == ""
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_opencode_documented_protocols(monkeypatch):
|
||||
models = [
|
||||
"minimax-m3",
|
||||
"qwen3.8-max",
|
||||
"gpt-5.6-luna",
|
||||
"grok-4.6",
|
||||
"muse-spark-1.3-contributor",
|
||||
"kimi-k3",
|
||||
]
|
||||
mock_http(
|
||||
monkeypatch,
|
||||
lambda request: httpx.Response(200, json={"data": [{"id": model} for model in models]}),
|
||||
)
|
||||
result = await catalog.refresh_llm_provider_preset("opencode-go")
|
||||
assert list(result["model_provider_apis"].values()) == [
|
||||
"anthropic-messages",
|
||||
"anthropic-messages",
|
||||
"openai-responses",
|
||||
"openai-responses",
|
||||
"openai-responses",
|
||||
"openai-completions",
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_retry_transient_error_only(monkeypatch):
|
||||
calls = []
|
||||
|
||||
def handle(request):
|
||||
calls.append(request)
|
||||
return httpx.Response(503 if len(calls) == 1 else 200, json={"data": [{"id": "latest"}]})
|
||||
|
||||
mock_http(monkeypatch, handle)
|
||||
await catalog.refresh_llm_provider_preset("minimax", "test-key")
|
||||
assert len(calls) == 2
|
||||
calls.clear()
|
||||
|
||||
def unauthorized(request):
|
||||
calls.append(request)
|
||||
return httpx.Response(401, text="secret-upstream-text")
|
||||
|
||||
mock_http(monkeypatch, unauthorized)
|
||||
with pytest.raises(httpx.HTTPStatusError) as error:
|
||||
await catalog.refresh_llm_provider_preset("minimax", "test-key")
|
||||
assert len(calls) == 1
|
||||
assert "secret-upstream-text" not in discovery.catalog_error_message(error.value)
|
||||
165
backend/tests/test_llm_provider_settings.py
Normal file
165
backend/tests/test_llm_provider_settings.py
Normal file
@@ -0,0 +1,165 @@
|
||||
from copy import deepcopy
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from sqlalchemy import create_engine, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.api.v1 import settings as api
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.services import llm_model_catalog as discovery
|
||||
from app.services.llm_provider_catalog import list_fallback_llm_provider_presets
|
||||
|
||||
REAL_CLIENT = httpx.AsyncClient
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def settings_db(monkeypatch, tmp_path):
|
||||
# Real SQL reads, commits and reloads; do not mock the settings helpers.
|
||||
engine = create_engine("sqlite://")
|
||||
SystemSetting.__table__.create(engine)
|
||||
session = Session(engine)
|
||||
db = SimpleNamespace(
|
||||
execute=AsyncMock(side_effect=session.execute),
|
||||
add=session.add,
|
||||
commit=AsyncMock(side_effect=session.commit),
|
||||
refresh=AsyncMock(side_effect=session.refresh),
|
||||
)
|
||||
monkeypatch.setattr(api, "AI_PROVIDER_ENV_FILE", tmp_path / "missing.env")
|
||||
monkeypatch.setattr(api, "_resolve_env_secret", lambda *names: ("", ""))
|
||||
yield db
|
||||
session.close()
|
||||
engine.dispose()
|
||||
|
||||
|
||||
def use_upstream(monkeypatch, handler):
|
||||
transport = httpx.MockTransport(handler)
|
||||
monkeypatch.setattr(
|
||||
discovery.httpx, "AsyncClient", lambda **kw: REAL_CLIENT(transport=transport, **kw)
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_fresh_database_lists_all_builtin_presets(settings_db):
|
||||
result = await api.get_ai_provider_presets(current_user=None, db=settings_db)
|
||||
assert len(result["data"]) == 9
|
||||
assert all(row["models"] for row in result["data"])
|
||||
assert "MiniMax-M3" in result["data"][0]["models"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_refresh_commits_reloads_and_preserves_runtime_and_draft(settings_db, monkeypatch):
|
||||
runtime = {
|
||||
"ai_provider": {
|
||||
"default_provider": "minimax",
|
||||
"service_token": "private-service-token",
|
||||
"providers": {
|
||||
"minimax": {
|
||||
"base_url": "https://saved.test/anthropic",
|
||||
"model": "old",
|
||||
"api_key": "saved-secret",
|
||||
}
|
||||
},
|
||||
}
|
||||
}
|
||||
await api.save_setting_payload(settings_db, "external_integrations", deepcopy(runtime))
|
||||
before = await api.get_setting_payload(settings_db, "external_integrations")
|
||||
seen = []
|
||||
|
||||
def handle(request):
|
||||
seen.append(request)
|
||||
assert str(request.url) == "https://draft.test/anthropic/v1/models"
|
||||
assert request.headers["x-api-key"] == "draft-secret"
|
||||
return httpx.Response(200, json={"data": [{"id": "MiniMax-M3"}, {"id": "old"}]})
|
||||
|
||||
use_upstream(monkeypatch, handle)
|
||||
draft = api.AIProviderIntegrationUpdate(
|
||||
provider="minimax",
|
||||
base_url="https://draft.test/anthropic",
|
||||
model="unsaved-model",
|
||||
api_key="draft-secret",
|
||||
service_token="unsaved-service-token",
|
||||
)
|
||||
refreshed = await api.refresh_ai_provider_preset("minimax", None, settings_db, draft)
|
||||
result = await api.get_ai_provider_presets(None, settings_db)
|
||||
preset = next(row for row in result["data"] if row["provider"] == "minimax")
|
||||
assert len(seen) == 1
|
||||
assert preset["models"] == ["MiniMax-M3", "old"]
|
||||
assert preset["refreshed_at"] == refreshed["data"]["refreshed_at"]
|
||||
assert await api.get_setting_payload(settings_db, "external_integrations") == before
|
||||
assert "secret" not in str(result)
|
||||
assert "private-service-token" not in str(result)
|
||||
assert draft.model == "unsaved-model"
|
||||
stored = await settings_db.execute(
|
||||
select(SystemSetting).where(SystemSetting.category == "llm_provider_preset:minimax")
|
||||
)
|
||||
assert stored.scalar_one().payload["models"] == ["MiniMax-M3", "old"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_failed_refresh_preserves_stored_catalog_and_sanitizes_errors(
|
||||
settings_db, monkeypatch
|
||||
):
|
||||
await api.save_setting_payload(
|
||||
settings_db, "llm_provider_preset:openrouter", {"models": ["saved-model"]}
|
||||
)
|
||||
use_upstream(monkeypatch, lambda request: httpx.Response(401, text="private-upstream-secret"))
|
||||
commits = settings_db.commit.await_count
|
||||
with pytest.raises(api.HTTPException) as error:
|
||||
await api.refresh_ai_provider_preset("openrouter", None, settings_db)
|
||||
assert error.value.status_code == 502
|
||||
assert "private-upstream-secret" not in error.value.detail
|
||||
assert settings_db.commit.await_count == commits
|
||||
assert (await api.get_setting_payload(settings_db, "llm_provider_preset:openrouter"))[
|
||||
"models"
|
||||
] == ["saved-model"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_missing_key_is_actionable_and_does_not_request_upstream(settings_db, monkeypatch):
|
||||
def fail_request(request):
|
||||
pytest.fail("must not query a private catalog without credentials")
|
||||
|
||||
use_upstream(monkeypatch, fail_request)
|
||||
with pytest.raises(api.HTTPException) as error:
|
||||
await api.refresh_ai_provider_preset("minimax", None, settings_db)
|
||||
assert error.value.status_code == 400
|
||||
assert "API Key" in error.value.detail
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
"preset", list_fallback_llm_provider_presets(), ids=lambda p: p["provider"]
|
||||
)
|
||||
async def test_404_never_passes_by_builtin_model_name(monkeypatch, preset):
|
||||
use_upstream(monkeypatch, lambda request: httpx.Response(404, text="private-upstream-secret"))
|
||||
result = await api._check_ai_provider_lightweight(
|
||||
{**preset, "api_key": "invalid-key", "preset_models": preset["models"]}, 5
|
||||
)
|
||||
assert result["success"] is False
|
||||
assert result["connected"] is False
|
||||
assert "private-upstream-secret" not in result["message"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("payload", [{}, {"data": []}, {"data": [{"id": "different-model"}]}])
|
||||
async def test_invalid_or_missing_models_never_pass(monkeypatch, payload):
|
||||
use_upstream(monkeypatch, lambda request: httpx.Response(200, json=payload))
|
||||
preset = list_fallback_llm_provider_presets()[0]
|
||||
result = await api._check_ai_provider_lightweight({**preset, "api_key": "test-key"}, 5)
|
||||
assert result["success"] is False
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_new_model_does_not_need_a_builtin_whitelist(monkeypatch):
|
||||
use_upstream(
|
||||
monkeypatch, lambda request: httpx.Response(200, json={"data": [{"id": "future-model"}]})
|
||||
)
|
||||
preset = list_fallback_llm_provider_presets()[0]
|
||||
result = await api._check_ai_provider_lightweight(
|
||||
{**preset, "model": "future-model", "api_key": "test-key"}, 5
|
||||
)
|
||||
assert result["success"] is True
|
||||
assert result["url"].endswith("/anthropic/v1/models")
|
||||
141
backend/tests/test_provider_protocols.py
Normal file
141
backend/tests/test_provider_protocols.py
Normal file
@@ -0,0 +1,141 @@
|
||||
from pathlib import Path
|
||||
import sys
|
||||
from unittest.mock import patch
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||
|
||||
with patch(
|
||||
"pydantic_settings.sources.providers.dotenv.DotEnvSettingsSource._read_env_files",
|
||||
return_value={},
|
||||
):
|
||||
from aiprovider.provider_service import ProviderService
|
||||
from aiprovider.schemas import SituationalAnalysisRequest
|
||||
from app.api.v1.settings import _runtime_config_from_ai_payload
|
||||
|
||||
|
||||
def test_custom_openai_gateway_keeps_its_configured_protocol():
|
||||
config = _runtime_config_from_ai_payload(
|
||||
{
|
||||
"default_provider": "openai",
|
||||
"providers": {
|
||||
"openai": {
|
||||
"base_url": "https://gateway.test/v1",
|
||||
"provider_api": "openai-completions",
|
||||
"model": "gpt-6-astra",
|
||||
},
|
||||
},
|
||||
}
|
||||
)["llm_config"]
|
||||
service = ProviderService({**config, "api_key": "test-key"})
|
||||
assert service._resolve_model_provider_api("gpt-6-astra") == "openai-completions"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
"model,path",
|
||||
[("minimax-m3", "/messages"), ("qwen3.8-max", "/messages"), ("gpt-5.6-luna", "/responses")],
|
||||
)
|
||||
async def test_runtime_model_routes_use_current_protocols(monkeypatch, model, path):
|
||||
llm = _runtime_config_from_ai_payload(
|
||||
{
|
||||
"default_provider": "opencode-go",
|
||||
"providers": {
|
||||
"opencode-go": {
|
||||
"model": model,
|
||||
"api_key": "test-key",
|
||||
"model_provider_apis": {model: "openai-completions"},
|
||||
}
|
||||
},
|
||||
}
|
||||
)["llm_config"]
|
||||
service = ProviderService(llm)
|
||||
seen = []
|
||||
|
||||
async def post(**request):
|
||||
seen.append(request)
|
||||
assert request["path"] == path
|
||||
assert request["request_body"]["model"] == model
|
||||
if path == "/responses":
|
||||
assert request["request_body"]["store"] is False
|
||||
assert "max_tokens" not in request["request_body"]
|
||||
return {
|
||||
"output": [
|
||||
{"type": "reasoning", "summary": [{"text": "reason"}]},
|
||||
{"type": "message", "content": [{"type": "output_text", "text": "OK"}]},
|
||||
]
|
||||
}
|
||||
return {"content": [{"type": "text", "text": "OK"}]}
|
||||
|
||||
monkeypatch.setattr(service, "_post", post)
|
||||
result = await service.analyze(SituationalAnalysisRequest(title="test", objective="reply OK"))
|
||||
assert len(seen) == 1
|
||||
assert result.content == "OK"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_minimax_messages_url_accepts_both_documented_base_forms(monkeypatch):
|
||||
seen = []
|
||||
client_type = httpx.AsyncClient
|
||||
|
||||
def handle(request):
|
||||
seen.append(str(request.url))
|
||||
return httpx.Response(200, json={"content": [{"type": "text", "text": "OK"}]})
|
||||
|
||||
transport = httpx.MockTransport(handle)
|
||||
monkeypatch.setattr(httpx, "AsyncClient", lambda **kw: client_type(transport=transport, **kw))
|
||||
for base in ["https://api.minimaxi.com/anthropic", "https://api.minimaxi.com/anthropic/v1"]:
|
||||
service = ProviderService(
|
||||
{
|
||||
"provider": "minimax",
|
||||
"provider_api": "anthropic-messages",
|
||||
"base_url": base,
|
||||
"api_key": "test-key",
|
||||
"model": "MiniMax-M3",
|
||||
}
|
||||
)
|
||||
result = await service.analyze(
|
||||
SituationalAnalysisRequest(title="test", objective="reply OK")
|
||||
)
|
||||
assert result.content == "OK"
|
||||
assert seen == ["https://api.minimaxi.com/anthropic/v1/messages"] * 2
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_m3_playground_thinking_and_opencode_session_headers(monkeypatch):
|
||||
seen = []
|
||||
client_type = httpx.AsyncClient
|
||||
|
||||
def handle(request):
|
||||
import json
|
||||
|
||||
seen.append(request)
|
||||
assert json.loads(request.content)["thinking"] == {"type": "adaptive"}
|
||||
assert request.headers["user-agent"] == "Planet/1.0"
|
||||
return httpx.Response(200, json={"content": [{"type": "text", "text": "OK"}]})
|
||||
|
||||
transport = httpx.MockTransport(handle)
|
||||
monkeypatch.setattr(httpx, "AsyncClient", lambda **kw: client_type(transport=transport, **kw))
|
||||
for session_id in ["conversation-one", "conversation-one", "conversation-two"]:
|
||||
service = ProviderService(
|
||||
{
|
||||
"provider": "opencode-go",
|
||||
"provider_api": "anthropic-messages",
|
||||
"base_url": "https://opencode.ai/zen/go/v1",
|
||||
"api_key": "test-key",
|
||||
"model": "minimax-m3",
|
||||
}
|
||||
)
|
||||
await service.analyze(
|
||||
SituationalAnalysisRequest(
|
||||
title="test",
|
||||
objective="reply OK",
|
||||
thinking={"type": "enabled"},
|
||||
context={"session_id": session_id},
|
||||
)
|
||||
)
|
||||
session_headers = [request.headers["x-opencode-session"] for request in seen]
|
||||
assert session_headers[0] == session_headers[1]
|
||||
assert session_headers[0] != session_headers[2]
|
||||
160
backend/tests/test_tv_catalog.py
Normal file
160
backend/tests/test_tv_catalog.py
Normal file
@@ -0,0 +1,160 @@
|
||||
from datetime import UTC, datetime
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import create_engine, event
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.db.session import Base
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.models.data_snapshot import DataSnapshot
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.models.task import CollectionTask
|
||||
from app.services.collectors.news_live_streams import NewsLiveStreamsCollector
|
||||
from app.services.tv_catalog import get_tv_catalog_page
|
||||
from app.services.tv_streams import normalize_tv_settings
|
||||
|
||||
|
||||
class CatalogSession:
|
||||
"""Execute the real catalog queries against an isolated SQLite database."""
|
||||
|
||||
def __init__(self, session):
|
||||
self.session = session
|
||||
|
||||
async def execute(self, query):
|
||||
return self.session.execute(query)
|
||||
|
||||
async def scalar(self, query):
|
||||
return self.session.scalar(query)
|
||||
|
||||
async def scalars(self, query):
|
||||
return self.session.scalars(query)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def catalog_db():
|
||||
engine = create_engine("sqlite:///:memory:")
|
||||
|
||||
@event.listens_for(engine, "connect")
|
||||
def register_functions(connection, _record):
|
||||
connection.create_function(
|
||||
"concat_ws",
|
||||
-1,
|
||||
lambda sep, *args: sep.join(str(arg) for arg in args if arg is not None),
|
||||
)
|
||||
|
||||
Base.metadata.create_all(
|
||||
engine,
|
||||
tables=[
|
||||
CollectionTask.__table__,
|
||||
DataSnapshot.__table__,
|
||||
CollectedData.__table__,
|
||||
SystemSetting.__table__,
|
||||
],
|
||||
)
|
||||
with Session(engine) as session:
|
||||
for index in range(135):
|
||||
session.add(
|
||||
CollectedData(
|
||||
source="news_live_streams",
|
||||
source_id=f"channel-{index:03}",
|
||||
data_type="news_live_stream",
|
||||
name=f"Channel {index:03}",
|
||||
collected_at=datetime(2026, 9, 13, tzinfo=UTC),
|
||||
is_current=True,
|
||||
is_valid=1,
|
||||
extra_data={
|
||||
"stream_url": "https://example.invalid/live.m3u8",
|
||||
"region": "Canada",
|
||||
},
|
||||
)
|
||||
)
|
||||
session.flush()
|
||||
yield CatalogSession(session)
|
||||
engine.dispose()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_pages_include_entire_catalog_without_overlap(catalog_db):
|
||||
first = await get_tv_catalog_page(catalog_db, limit=50)
|
||||
second = await get_tv_catalog_page(catalog_db, offset=first["next_offset"], limit=50)
|
||||
third = await get_tv_catalog_page(catalog_db, offset=second["next_offset"], limit=50)
|
||||
ids = [source["id"] for page in (first, second, third) for source in page["sources"]]
|
||||
assert [len(page["sources"]) for page in (first, second, third)] == [50, 50, 45]
|
||||
assert len(set(ids)) == first["source_count"] == 145
|
||||
assert ids[-1] == "channel-134"
|
||||
assert third["next_offset"] is None and not third["has_more"]
|
||||
beyond = await get_tv_catalog_page(catalog_db, offset=200)
|
||||
assert beyond["sources"] == [] and not beyond["has_more"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_search_finds_later_pages_and_treats_wildcards_literally(catalog_db):
|
||||
payload = await get_tv_catalog_page(catalog_db, q="CANADA 134")
|
||||
assert [source["id"] for source in payload["sources"]] == ["channel-134"]
|
||||
assert payload["total"] == 1 and payload["source_count"] == 145
|
||||
assert (await get_tv_catalog_page(catalog_db, q="%"))["total"] == 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_selection_survives_refresh_when_outside_first_page(catalog_db):
|
||||
payload = await get_tv_catalog_page(catalog_db, selected_id="channel-134")
|
||||
assert payload["selected_source"]["id"] == "channel-134"
|
||||
assert "channel-134" not in [source["id"] for source in payload["sources"]]
|
||||
assert payload["default_source_id"] == "aljazeera-mubasher"
|
||||
default = (await get_tv_catalog_page(catalog_db, selected_id="removed"))["selected_source"]
|
||||
assert default["id"] == "aljazeera-mubasher" and default["source_type"] == "hls"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_catalog_hides_inactive_records_and_deduplicates_ids(catalog_db):
|
||||
for name, values in [
|
||||
("Disabled", {"extra_data": {"is_enabled": False}}),
|
||||
("Historical", {"is_current": False}),
|
||||
("Invalid", {"is_valid": 0}),
|
||||
("Deleted", {"deleted_at": datetime.now(UTC)}),
|
||||
("Replacement", {"source_id": "channel-134"}),
|
||||
]:
|
||||
record = dict(
|
||||
source="news_live_streams",
|
||||
source_id=name,
|
||||
name=name,
|
||||
data_type="news_live_stream",
|
||||
is_current=True,
|
||||
is_valid=1,
|
||||
)
|
||||
catalog_db.session.add(CollectedData(**{**record, **values}))
|
||||
catalog_db.session.flush()
|
||||
payload = await get_tv_catalog_page(catalog_db, q="replacement")
|
||||
assert payload["source_count"] == 145
|
||||
assert [source["id"] for source in payload["sources"]] == ["channel-134"]
|
||||
|
||||
|
||||
def test_missing_builtin_default_adds_aljazeera_without_losing_custom_source():
|
||||
settings = normalize_tv_settings({"sources": [{"id": "custom", "name": "Custom"}]})
|
||||
assert settings["default_source_id"] == "aljazeera-mubasher"
|
||||
assert {source["id"] for source in settings["sources"]} == {"custom", "aljazeera-mubasher"}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
"config, expected", [({}, 135), ({"max_sources": 0}, 135), ({"max_sources": 7}, 7)]
|
||||
)
|
||||
async def test_collector_keeps_all_matching_channels_unless_explicitly_limited(
|
||||
monkeypatch, config, expected
|
||||
):
|
||||
collector = NewsLiveStreamsCollector()
|
||||
channels = [
|
||||
{"id": f"channel-{i}", "name": f"Channel {i}", "categories": ["news"]} for i in range(135)
|
||||
]
|
||||
channels.append({"id": "sport", "name": "Sports", "categories": ["sports"]})
|
||||
streams = [
|
||||
{"channel": channel["id"], "url": "https://example.invalid/live.m3u8"}
|
||||
for channel in channels
|
||||
]
|
||||
monkeypatch.setattr(
|
||||
collector, "_gather_iptv_org_payloads", AsyncMock(return_value=(channels, streams, []))
|
||||
)
|
||||
records = await collector._fetch_iptv_org("https://example.invalid/channels.json", config)
|
||||
assert len(records) == expected
|
||||
assert all(record["source_id"] != "sport" for record in records)
|
||||
37
backend/tests/test_tv_playback.py
Normal file
37
backend/tests/test_tv_playback.py
Normal file
@@ -0,0 +1,37 @@
|
||||
from app.services.tv_streams import (
|
||||
CCTV4_HOMEPAGE,
|
||||
CCTV4_STREAM,
|
||||
UNSUPPORTED_CCTV4_STREAM,
|
||||
normalize_tv_settings,
|
||||
normalize_tv_source,
|
||||
)
|
||||
|
||||
|
||||
def test_default_keeps_aljazeera_and_does_not_offer_cdrm_as_plain_hls():
|
||||
settings = normalize_tv_settings(None)
|
||||
assert settings["default_source_id"] == "aljazeera-mubasher"
|
||||
cctv = next(source for source in settings["sources"] if source["id"] == "cctv4")
|
||||
assert cctv["source_type"] == "hls"
|
||||
assert cctv["stream_url"] == CCTV4_STREAM
|
||||
assert "美洲版" in cctv["name"]
|
||||
assert cctv["use_proxy"] is False
|
||||
assert cctv["homepage_url"] == CCTV4_HOMEPAGE
|
||||
|
||||
|
||||
def test_saved_legacy_default_is_corrected_without_mutating_input():
|
||||
saved = {"id": "cctv4", "source_type": "hls", "stream_url": UNSUPPORTED_CCTV4_STREAM}
|
||||
source = normalize_tv_source(saved)
|
||||
assert source["source_type"] == "hls"
|
||||
assert source["stream_url"] == CCTV4_STREAM
|
||||
assert source["use_proxy"] is False
|
||||
assert saved["stream_url"] == UNSUPPORTED_CCTV4_STREAM
|
||||
|
||||
|
||||
def test_custom_cctv_stream_and_fallback_preference_are_preserved():
|
||||
custom = {"id": "cctv4", "source_type": "hls", "stream_url": "https://example.com/licensed.m3u8"}
|
||||
settings = normalize_tv_settings({"auto_fallback": False, "sources": [custom]})
|
||||
source = next(source for source in settings["sources"] if source["id"] == "cctv4")
|
||||
assert source["stream_url"] == custom["stream_url"]
|
||||
assert source["source_type"] == "hls"
|
||||
assert source["use_proxy"] is True
|
||||
assert settings["auto_fallback"] is False
|
||||
@@ -76,6 +76,7 @@ def test_convert_compute_centers_to_geojson_unifies_sources():
|
||||
longitude=-90.05,
|
||||
metadata={
|
||||
"organization": "xAI",
|
||||
"location": "3231 Riverport Rd, Memphis, TN 38109",
|
||||
"gpu_type": "H100",
|
||||
"gpu_count": 100000,
|
||||
"value": "20000",
|
||||
@@ -168,6 +169,241 @@ def test_compute_center_source_coordinates_win_over_stored_location():
|
||||
compute_center_locations.set_compute_center_location_cache({})
|
||||
|
||||
|
||||
def test_epoch_country_level_coordinates_are_unresolved():
|
||||
record = _build_record(
|
||||
record_id=33,
|
||||
source="epoch_ai_gpu",
|
||||
data_type="gpu_cluster",
|
||||
name="Meta 100k",
|
||||
country="United States",
|
||||
city="",
|
||||
latitude=39.381266,
|
||||
longitude=-97.922211,
|
||||
metadata={
|
||||
"location": "United States of America",
|
||||
"raw_country": "United States of America",
|
||||
},
|
||||
)
|
||||
|
||||
payload = convert_compute_centers_to_geojson([record])
|
||||
|
||||
assert payload["features"] == []
|
||||
assert len(payload["unresolved"]) == 1
|
||||
assert payload["unresolved"][0]["source_id"] == "epoch_ai_gpu-33"
|
||||
assert "source_coordinates:country" in payload["unresolved"][0]["attempted_queries"]
|
||||
|
||||
|
||||
def test_epoch_city_coordinates_are_not_labeled_precise():
|
||||
record = _build_record(
|
||||
record_id=34,
|
||||
source="epoch_ai_gpu",
|
||||
data_type="gpu_cluster",
|
||||
name="Seoul Cluster",
|
||||
country="South Korea",
|
||||
city="",
|
||||
latitude=37.5665,
|
||||
longitude=126.978,
|
||||
metadata={"location": "Seoul", "raw_country": "South Korea"},
|
||||
)
|
||||
|
||||
result = compute_center_locations.resolve_compute_center_location_full(
|
||||
record, record.extra_data
|
||||
)
|
||||
|
||||
assert result.is_resolved
|
||||
assert result.location.location_precision == "city"
|
||||
assert result.location.needs_confirmation is True
|
||||
assert result.location.is_estimated is True
|
||||
assert result.location.city is None
|
||||
assert result.location.matched_location_name == "Seoul"
|
||||
assert "Epoch Location: Seoul" in result.location.location_source_note
|
||||
|
||||
|
||||
def test_epoch_city_geojson_preserves_source_location_without_guessing_city():
|
||||
record = _build_record(
|
||||
record_id=37,
|
||||
source="epoch_ai_gpu",
|
||||
data_type="gpu_cluster",
|
||||
name="Petrobras Fênix",
|
||||
country="Brazil",
|
||||
city="",
|
||||
latitude=-22.909534,
|
||||
longitude=-43.209934,
|
||||
metadata={
|
||||
"location": "Rio de Janeiro, Brazil",
|
||||
"raw_country": "Brazil",
|
||||
},
|
||||
)
|
||||
|
||||
payload = convert_compute_centers_to_geojson([record])
|
||||
|
||||
feature = payload["features"][0]
|
||||
assert feature["properties"]["location_precision"] == "city"
|
||||
assert feature["properties"]["matched_location_name"] == "Rio de Janeiro, Brazil"
|
||||
assert "Epoch Location: Rio de Janeiro, Brazil" in feature["properties"][
|
||||
"location_source_note"
|
||||
]
|
||||
|
||||
|
||||
def test_confirmed_stored_location_beats_epoch_precise_coordinates():
|
||||
compute_center_locations.set_compute_center_location_cache({
|
||||
"epoch_ai_gpu:epoch_ai_gpu-35": {
|
||||
"source": "epoch_ai_gpu",
|
||||
"source_id": "epoch_ai_gpu-35",
|
||||
"name": "Confirmed Site",
|
||||
"latitude": 35.0,
|
||||
"longitude": 139.0,
|
||||
"precision": "site",
|
||||
"confidence": 1.0,
|
||||
"location_source": "official_facility_registry",
|
||||
"needs_confirmation": False,
|
||||
}
|
||||
})
|
||||
try:
|
||||
record = _build_record(
|
||||
record_id=35,
|
||||
source="epoch_ai_gpu",
|
||||
data_type="gpu_cluster",
|
||||
name="Japan Cluster",
|
||||
country="Japan",
|
||||
city="Tokyo",
|
||||
latitude=35.68882,
|
||||
longitude=139.692526,
|
||||
metadata={
|
||||
"location": "1-1 Verified Street, Tokyo 100-0001, Japan",
|
||||
"raw_country": "Japan",
|
||||
},
|
||||
)
|
||||
|
||||
result = compute_center_locations.resolve_compute_center_location_full(
|
||||
record, record.extra_data
|
||||
)
|
||||
|
||||
assert result.location.location_source == "stored_compute_center_location"
|
||||
assert result.location.location_precision == "site"
|
||||
assert (result.location.latitude, result.location.longitude) == (35.0, 139.0)
|
||||
finally:
|
||||
compute_center_locations.set_compute_center_location_cache({})
|
||||
|
||||
|
||||
def test_legacy_seeded_country_coordinate_is_ignored_on_read():
|
||||
compute_center_locations.set_compute_center_location_cache({
|
||||
"epoch_ai_gpu:epoch_ai_gpu-36": {
|
||||
"source": "epoch_ai_gpu",
|
||||
"source_id": "epoch_ai_gpu-36",
|
||||
"name": "Legacy Country Point",
|
||||
"latitude": 36.386493,
|
||||
"longitude": 138.59223,
|
||||
"precision": "precise",
|
||||
"confidence": 1.0,
|
||||
"location_source": "source_coordinates",
|
||||
"needs_confirmation": False,
|
||||
}
|
||||
})
|
||||
try:
|
||||
record = _build_record(
|
||||
record_id=36,
|
||||
source="epoch_ai_gpu",
|
||||
data_type="gpu_cluster",
|
||||
name="Legacy Japan Cluster",
|
||||
country="Japan",
|
||||
city="",
|
||||
latitude=36.386493,
|
||||
longitude=138.59223,
|
||||
metadata={"location": "Japan", "raw_country": "Japan"},
|
||||
)
|
||||
|
||||
result = compute_center_locations.resolve_compute_center_location_full(
|
||||
record, record.extra_data
|
||||
)
|
||||
|
||||
assert result.location is None
|
||||
finally:
|
||||
compute_center_locations.set_compute_center_location_cache({})
|
||||
|
||||
|
||||
def test_verified_sites_replace_epoch_country_center_coordinates():
|
||||
specifications = [
|
||||
(
|
||||
"epoch_jamstec_zettascaler_2_0_gyoukou",
|
||||
"Gyoukou 2.0",
|
||||
"Japan",
|
||||
36.386493,
|
||||
138.59223,
|
||||
[139.625008, 35.380924],
|
||||
),
|
||||
(
|
||||
"epoch_jamstec_zettascaler_2_2_gyoukou",
|
||||
"Gyoukou 2.2",
|
||||
"Japan",
|
||||
36.386493,
|
||||
138.59223,
|
||||
[139.625008, 35.380924],
|
||||
),
|
||||
(
|
||||
"epoch_argonne_nl_sophia__formerly_theta_",
|
||||
"Sophia",
|
||||
"United States of America",
|
||||
39.381266,
|
||||
-97.922211,
|
||||
[-87.984498648496, 41.718109383287235],
|
||||
),
|
||||
]
|
||||
records = []
|
||||
for index, (source_id, name, country, latitude, longitude, _expected) in enumerate(
|
||||
specifications, start=40
|
||||
):
|
||||
record = _build_record(
|
||||
record_id=index,
|
||||
source="epoch_ai_gpu",
|
||||
data_type="gpu_cluster",
|
||||
name=name,
|
||||
country=country,
|
||||
city="",
|
||||
latitude=latitude,
|
||||
longitude=longitude,
|
||||
metadata={"location": country, "raw_country": country},
|
||||
)
|
||||
record.source_id = source_id
|
||||
records.append(record)
|
||||
|
||||
payload = convert_compute_centers_to_geojson(records)
|
||||
|
||||
assert payload["unresolved"] == []
|
||||
assert len(payload["features"]) == 3
|
||||
for feature, specification in zip(payload["features"], specifications):
|
||||
assert feature["geometry"]["coordinates"] == specification[-1]
|
||||
assert feature["properties"]["location_precision"] == "site"
|
||||
assert feature["properties"]["location_source"] == "official_facility_registry"
|
||||
assert "https://" in feature["properties"]["location_source_note"]
|
||||
|
||||
|
||||
def test_verified_site_beats_epoch_city_level_source_coordinates():
|
||||
record = _build_record(
|
||||
record_id=44,
|
||||
source="epoch_ai_gpu",
|
||||
data_type="gpu_cluster",
|
||||
name="Gyoukou 2.2",
|
||||
country="Japan",
|
||||
city="Tokyo",
|
||||
latitude=35.68882,
|
||||
longitude=139.692526,
|
||||
metadata={"location": "Tokyo, Japan", "raw_country": "Japan"},
|
||||
)
|
||||
record.source_id = "epoch_jamstec_zettascaler_2_2_gyoukou"
|
||||
|
||||
result = compute_center_locations.resolve_compute_center_location_full(
|
||||
record, record.extra_data
|
||||
)
|
||||
|
||||
assert result.location.location_source == "official_facility_registry"
|
||||
assert result.location.location_precision == "site"
|
||||
assert (result.location.latitude, result.location.longitude) == (
|
||||
35.380924,
|
||||
139.625008,
|
||||
)
|
||||
|
||||
|
||||
def test_compute_center_geojson_uses_stored_location_when_source_coords_missing():
|
||||
compute_center_locations.set_compute_center_location_cache({
|
||||
"epoch_ai_gpu:epoch_ai_gpu-32": {
|
||||
@@ -505,6 +741,7 @@ def test_collect_location_candidates_failure_returns_attempted_queries(monkeypat
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_collect_compute_center_location_skips_llm_when_candidates_exist(monkeypatch):
|
||||
monkeypatch.setattr(visualization_api, "_compute_center_location_web_search_capability", AsyncMock(return_value={"enabled": True}))
|
||||
candidate = compute_center_locations.LocationCandidate(
|
||||
latitude=45.764,
|
||||
longitude=4.8357,
|
||||
@@ -548,6 +785,7 @@ async def test_collect_compute_center_location_skips_llm_when_candidates_exist(m
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_collect_compute_center_location_uses_llm_when_candidates_empty(monkeypatch):
|
||||
monkeypatch.setattr(visualization_api, "_compute_center_location_web_search_capability", AsyncMock(return_value={"enabled": True}))
|
||||
llm_candidate = compute_center_locations.LocationCandidate(
|
||||
latitude=45.764,
|
||||
longitude=4.8357,
|
||||
@@ -615,7 +853,11 @@ async def test_collect_compute_center_location_uses_llm_when_candidates_empty(mo
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_compute_centers_geojson_endpoint_returns_stats():
|
||||
async def test_compute_centers_geojson_endpoint_returns_stats(monkeypatch):
|
||||
async def build_without_shared_cache(**kwargs):
|
||||
return await kwargs["builder"]()
|
||||
|
||||
monkeypatch.setattr(visualization_api, "get_or_build_layer_payload", build_without_shared_cache)
|
||||
records = [
|
||||
_build_record(
|
||||
record_id=1,
|
||||
@@ -637,7 +879,11 @@ async def test_compute_centers_geojson_endpoint_returns_stats():
|
||||
city="Memphis",
|
||||
latitude=35.15,
|
||||
longitude=-90.05,
|
||||
metadata={"value": "20000", "unit": "TFlop/s"},
|
||||
metadata={
|
||||
"location": "3231 Riverport Rd, Memphis, TN 38109",
|
||||
"value": "20000",
|
||||
"unit": "TFlop/s",
|
||||
},
|
||||
),
|
||||
]
|
||||
|
||||
@@ -680,6 +926,11 @@ async def test_compute_centers_geojson_endpoint_returns_stats():
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_visualization_geo_summary_returns_counts(monkeypatch):
|
||||
async def build_without_shared_cache(**kwargs):
|
||||
return await kwargs["builder"]()
|
||||
|
||||
# Exercise the fixture DB, never the running development server's Redis payload.
|
||||
monkeypatch.setattr(visualization_api, "get_or_build_layer_payload", build_without_shared_cache)
|
||||
records = [
|
||||
_build_record(
|
||||
record_id=1,
|
||||
@@ -741,7 +992,11 @@ async def test_visualization_geo_summary_returns_counts(monkeypatch):
|
||||
city="Memphis",
|
||||
latitude=35.15,
|
||||
longitude=-90.05,
|
||||
metadata={"value": "20000", "unit": "TFlop/s"},
|
||||
metadata={
|
||||
"location": "3231 Riverport Rd, Memphis, TN 38109",
|
||||
"value": "20000",
|
||||
"unit": "TFlop/s",
|
||||
},
|
||||
),
|
||||
]
|
||||
|
||||
@@ -769,6 +1024,8 @@ async def test_visualization_geo_summary_returns_counts(monkeypatch):
|
||||
class _FakeSession:
|
||||
async def execute(self, query):
|
||||
query_text = str(query).lower()
|
||||
if "group by collected_data.source" in query_text:
|
||||
return _ScalarResult(rows=[("top500", "supercomputer", 1), ("epoch_ai_gpu", "gpu_cluster", 1)])
|
||||
if "bgp_incidents" in query_text:
|
||||
return _ScalarResult(scalar_value=2)
|
||||
if "bgp_anomalies" in query_text:
|
||||
@@ -839,6 +1096,7 @@ async def test_empty_cable_layers_return_empty_feature_collections():
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_collect_location_endpoint_returns_candidates_for_known_record(monkeypatch):
|
||||
monkeypatch.setattr(visualization_api, "_compute_center_location_web_search_capability", AsyncMock(return_value={"enabled": True}))
|
||||
def _fake_ror(query):
|
||||
assert query == "Oak Ridge National Laboratory"
|
||||
return {
|
||||
@@ -926,6 +1184,7 @@ async def test_collect_location_endpoint_returns_candidates_for_known_record(mon
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_collect_location_endpoint_returns_failure_reason(monkeypatch):
|
||||
monkeypatch.setattr(visualization_api, "_compute_center_location_web_search_capability", AsyncMock(return_value={"enabled": True}))
|
||||
monkeypatch.setattr(compute_center_locations, "_lookup_ror_organization", lambda _query: None)
|
||||
monkeypatch.setattr(compute_center_locations, "_geocode_online", lambda _query: None)
|
||||
|
||||
@@ -1136,3 +1395,19 @@ def test_repository_has_no_forbidden_precision_tokens():
|
||||
assert "FORBIDDEN_PRECISIONS" in text, (
|
||||
f"{token} appears in {target} outside the FORBIDDEN_PRECISIONS guard"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("kind", ["data_center", "cloud_center", "edge_node"])
|
||||
def test_infrastructure_type_is_preserved_in_geojson_and_image_classification(kind):
|
||||
from app.core.compute_types import compute_site_type
|
||||
record = _build_record(record_id=900, source="facility_catalog", data_type=kind,
|
||||
name="Verified facility", country="Japan", city="Tokyo", latitude=35.68, longitude=139.76,
|
||||
metadata={"location_precision": "site"})
|
||||
result = convert_compute_centers_to_geojson([record])
|
||||
assert compute_site_type(record) == kind
|
||||
assert result["features"][0]["properties"]["site_type"] == kind
|
||||
record.source = "epoch_ai_gpu"
|
||||
record.data_type = "gpu_cluster"
|
||||
record.extra_data = {**record.extra_data, "site_type": kind, "location": "Tokyo, Japan"}
|
||||
assert compute_site_type(record) == kind
|
||||
assert convert_compute_centers_to_geojson([record])["features"][0]["properties"]["site_type"] == kind
|
||||
|
||||
@@ -1,5 +1,9 @@
|
||||
import pytest
|
||||
import importlib
|
||||
import json
|
||||
|
||||
import pytest
|
||||
from fastapi import FastAPI
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from app.core.websocket.manager import ConnectionManager
|
||||
from app.core.websocket.broadcaster import DataBroadcaster
|
||||
@@ -113,6 +117,11 @@ async def test_vessel_broadcaster_keeps_latest_update_per_mmsi(monkeypatch):
|
||||
broadcaster_module = importlib.import_module("app.core.websocket.broadcaster")
|
||||
monkeypatch.setattr(broadcaster_module.manager, "broadcast_vessels", fake_broadcast_vessels)
|
||||
broadcaster = DataBroadcaster()
|
||||
async def load_current(keys):
|
||||
assert keys == ["1"]
|
||||
return [{"mmsi": 1, "lat": 60.0, "lon": 11.0, "source": "barentswatch_vessels"}]
|
||||
|
||||
monkeypatch.setattr(broadcaster, "_load_current_vessel_updates", load_current)
|
||||
broadcaster.enqueue_vessel_update(
|
||||
{
|
||||
"source": "aisstream_vessels",
|
||||
@@ -129,10 +138,67 @@ async def test_vessel_broadcaster_keeps_latest_update_per_mmsi(monkeypatch):
|
||||
assert sent[0]["vessels"] == [
|
||||
{
|
||||
"mmsi": 1,
|
||||
"lat": 59.1,
|
||||
"lon": 10.1,
|
||||
"source": "aisstream_vessels",
|
||||
"lat": 60.0,
|
||||
"lon": 11.0,
|
||||
"source": "barentswatch_vessels",
|
||||
"action": "upsert",
|
||||
"created": None,
|
||||
}
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_global_vessel_subscription_delivers_every_item_in_bounded_frames():
|
||||
manager = ConnectionManager()
|
||||
socket = FakeWebSocket()
|
||||
config = manager.subscribe_vessels(socket, {"scope": "global", "zoom": 4})
|
||||
json.dumps(config)
|
||||
vessels = [{"mmsi": index, "lat": 60, "lon": 10} for index in range(2501)]
|
||||
await manager.broadcast_vessels({"vessels": vessels})
|
||||
|
||||
assert [len(frame["payload"]["vessels"]) for frame in socket.sent] == [1000, 1000, 501]
|
||||
assert [item for frame in socket.sent for item in frame["payload"]["vessels"]] == vessels
|
||||
manager.unsubscribe(socket, ["vessels"])
|
||||
await manager.broadcast_vessels({"vessels": vessels})
|
||||
assert len(socket.sent) == 3
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_global_vessel_removal_does_not_require_coordinates():
|
||||
manager = ConnectionManager()
|
||||
socket = FakeWebSocket()
|
||||
manager.subscribe_vessels(socket, {"scope": "global", "zoom": 4})
|
||||
await manager.broadcast_vessels({"vessels": [{"mmsi": 123, "action": "remove"}]})
|
||||
assert socket.sent[0]["payload"]["vessels"] == [{"mmsi": 123, "action": "remove"}]
|
||||
|
||||
|
||||
def test_anonymous_earth_can_confirm_global_vessel_subscription(monkeypatch):
|
||||
websocket_module = importlib.import_module("app.api.v1.websocket")
|
||||
monkeypatch.setattr(websocket_module, "manager", ConnectionManager())
|
||||
app = FastAPI()
|
||||
app.include_router(websocket_module.router)
|
||||
with TestClient(app) as client, client.websocket_connect("/ws") as socket:
|
||||
assert socket.receive_json()["type"] == "connection_established"
|
||||
socket.send_json({
|
||||
"type": "subscribe",
|
||||
"data": {"channels": ["earth_updates", "vessels"], "scope": "global", "zoom": 4},
|
||||
})
|
||||
response = socket.receive_json()
|
||||
assert response["type"] == "subscription_confirmed"
|
||||
assert response["data"]["vessels"]["scope"] == "global"
|
||||
assert response["data"]["vessels"]["type"] == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_vessel_flush_retries_without_overwriting_newer_queued_updates(monkeypatch):
|
||||
broadcaster = DataBroadcaster()
|
||||
broadcaster.enqueue_vessel_update({"vessels": [{"mmsi": 1, "lat": 59, "lon": 10}]})
|
||||
|
||||
async def fail_read(_keys):
|
||||
broadcaster.enqueue_vessel_update({"vessels": [{"mmsi": 1, "lat": 61, "lon": 12}]})
|
||||
raise RuntimeError("database unavailable")
|
||||
|
||||
monkeypatch.setattr(broadcaster, "_load_current_vessel_updates", fail_read)
|
||||
with pytest.raises(RuntimeError, match="database unavailable"):
|
||||
await broadcaster.flush_vessel_updates()
|
||||
assert broadcaster._pending_vessel_updates["1"]["lat"] == 61
|
||||
|
||||
@@ -25,6 +25,25 @@
|
||||
{
|
||||
"id": "china-taiwan-penghu",
|
||||
"name": "Taiwan and Penghu",
|
||||
"sourceIsoA3": ["TWN"],
|
||||
"action": "union_to_country",
|
||||
"targetIsoA3": "CHN",
|
||||
"hoverIsoA3": "CHN",
|
||||
"labelPolicy": "show_country_only"
|
||||
},
|
||||
{
|
||||
"id": "china-hong-kong",
|
||||
"name": "Hong Kong",
|
||||
"sourceIsoA3": ["HKG"],
|
||||
"action": "union_to_country",
|
||||
"targetIsoA3": "CHN",
|
||||
"hoverIsoA3": "CHN",
|
||||
"labelPolicy": "show_country_only"
|
||||
},
|
||||
{
|
||||
"id": "china-macao",
|
||||
"name": "Macao",
|
||||
"sourceIsoA3": ["MAC"],
|
||||
"action": "union_to_country",
|
||||
"targetIsoA3": "CHN",
|
||||
"hoverIsoA3": "CHN",
|
||||
@@ -57,6 +76,7 @@
|
||||
{
|
||||
"id": "kosovo",
|
||||
"name": "Kosovo",
|
||||
"sourceIsoA3": ["KOS", "XKS"],
|
||||
"action": "render_as_disputed_with_parent",
|
||||
"parentIsoA3": "SRB",
|
||||
"hoverIsoA3": "SRB",
|
||||
|
||||
@@ -1,5 +1,3 @@
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
aiprovider:
|
||||
image: ${AI_PROVIDER_IMAGE_NAME:-planet-aiprovider:latest}
|
||||
@@ -32,7 +30,7 @@ services:
|
||||
POSTGRES_PASSWORD: postgres
|
||||
POSTGRES_DB: planet_db
|
||||
ports:
|
||||
- "5432:5432"
|
||||
- "127.0.0.1:${PLANET_POSTGRES_PORT:-5432}:5432"
|
||||
volumes:
|
||||
- postgres_data:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
@@ -45,7 +43,7 @@ services:
|
||||
image: redis:7-alpine
|
||||
container_name: planet_redis
|
||||
ports:
|
||||
- "6379:6379"
|
||||
- "127.0.0.1:${PLANET_REDIS_PORT:-6379}:6379"
|
||||
volumes:
|
||||
- redis_data:/data
|
||||
healthcheck:
|
||||
|
||||
@@ -8,6 +8,296 @@ This project follows the repository versioning rule:
|
||||
- `improvement` -> `+0.0.1`(bugfix + 小功能混合)
|
||||
- `bugfix` -> `+0.0.1`
|
||||
|
||||
## [0.76.8] — 2026-10-02
|
||||
|
||||
Released: 2026-10-02
|
||||
|
||||
### Highlights
|
||||
- 修复展开新闻面板时后台 ticker / realtime 更新导致卡片重建和 hover 抖动的问题。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 面板浏览期间冻结卡片 DOM,后台更新只写入最新 payload。
|
||||
- 关闭面板时一次性应用待处理新闻,保持滚动位置和 hover 稳定。
|
||||
- 展开面板时暂停隐藏 ticker,手动刷新仍可立即更新面板内容。
|
||||
|
||||
---
|
||||
|
||||
## [0.76.7] — 2026-10-02
|
||||
|
||||
Released: 2026-10-02
|
||||
|
||||
### Highlights
|
||||
- 统一 breaking 新闻卡片的 hover 交互,让通用 hover 行为叠加卡片自身等级颜色。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 红、橙、黄 breaking 卡片共用同一套 hover 提亮、边框增强和浮起逻辑。
|
||||
- 等级颜色通过卡片变量注入,避免为每种颜色复制 hover 样式。
|
||||
|
||||
---
|
||||
|
||||
## [0.76.6] — 2026-10-02
|
||||
|
||||
Released: 2026-10-02
|
||||
|
||||
### Highlights
|
||||
- 扩展 breaking ticker 的全域扫光效果,并拉开新闻卡片与上方状态内容的间距。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- ticker 反射从局部光斑改为覆盖整条横向区域的扫光/闪烁带。
|
||||
- 桌面与移动新闻列表首卡预留顶部呼吸空间,保持内容区层次。
|
||||
|
||||
---
|
||||
|
||||
## [0.76.5] — 2026-10-02
|
||||
|
||||
Released: 2026-10-02
|
||||
|
||||
### Highlights
|
||||
- 增加 breaking 新闻的玻璃反射式警灯脉冲,并修复新闻卡片内容被 flex 压缩的问题。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- ticker、卡片边框和卡片反射统一使用同一 breaking 等级颜色。
|
||||
- breaking ticker 使用局部红/橙高光脉冲,卡片使用反射带扫过;面板底色保持蓝色。
|
||||
- 新闻卡片固定非收缩布局并预留最小内容高度,避免只剩媒体名和时间。
|
||||
- 同步中英文 Earth 前端开发文档,记录视觉状态与 reduced-motion 行为。
|
||||
|
||||
---
|
||||
|
||||
## [0.76.4] — 2026-10-01
|
||||
|
||||
Released: 2026-10-01
|
||||
|
||||
### Highlights
|
||||
- 修正 breaking 新闻状态显示:ticker 仅在当前滚动到 breaking 新闻时变红,展开面板始终保持蓝色。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 选中、焦点、关闭动画与 breaking 数据状态彻底分离,关闭新闻栏不再留下橙色边线。
|
||||
- breaking 新闻按过期时间重新计算,单条新闻卡片继续保留自身 breaking 红色样式。
|
||||
- 增加 ticker breaking 状态单测和浏览器关闭/重开验证。
|
||||
|
||||
---
|
||||
|
||||
## [0.76.3] — 2026-10-01
|
||||
|
||||
Released: 2026-10-01
|
||||
|
||||
### Highlights
|
||||
- 修复新闻栏关闭后残留橙色边线的问题,明确 breaking、展开面板和新闻选择的状态边界。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- ticker 保持中性,breaking 等级只作用于展开新闻面板整体背景和阴影。
|
||||
- 关闭新闻面板时清理选中故事和过期焦点状态,breaking 按新闻过期时间重新计算。
|
||||
- 增加 breaking 状态单测,保持新闻滚动、选择和面板动画行为不变。
|
||||
|
||||
---
|
||||
|
||||
## [0.76.2] — 2026-10-01
|
||||
|
||||
Released: 2026-10-01
|
||||
|
||||
### Highlights
|
||||
- 优化光缆密集区的卫星点拾取,并将点状交互物件优先级泛化到其他密集图层。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 屏幕命中先统一比较卫星、BGP、算力和其他点状对象的距离,海缆线作为兜底候选,保护穿插在密集线缆中的小面积对象。
|
||||
- hover 与 click 共用同一候选排序,避免鼠标提示和最终选中对象不一致。
|
||||
- 补充密集区域拾取回归测试,清理中间态代码并同步渲染层说明。
|
||||
|
||||
---
|
||||
|
||||
## [0.76.1] — 2026-10-01
|
||||
|
||||
Released: 2026-10-01
|
||||
|
||||
### Highlights
|
||||
- 修复选中 BGP 事件静态光晕在地表边缘被切断或陷入地表的问题。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 抽取通用整体锚点遮挡能力,BGP 事件保持原有图标样式并复用圆角图标的前后显隐规则。
|
||||
- 选中事件光晕与事件图标共享地表锚点、前景层级和前后显隐路径,清理中间态代码与资源。
|
||||
- 补充中英文渲染层说明、回归测试和浏览器验证记录。
|
||||
|
||||
---
|
||||
|
||||
## [0.76.0] — 2026-09-30
|
||||
|
||||
Released: 2026-09-30
|
||||
|
||||
### Highlights
|
||||
- 新增详情卡默认配图与后台配置,缺图时保持完整数据卡片,并清理已替换的旧资源。
|
||||
- 算力中心补齐超算、GPU 集群、数据中心、云计算中心和边缘计算节点类型及配图。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 统一地表物件身份、拾取优先级、连线目标、聚合展示和选中置顶;跨图层重叠时焦点对象保持在前景。
|
||||
- 详情卡保留全量显示,统一高度、字段顺序、拖拽落位和新闻展开交互;图片解析改用按对象索引的快速路径。
|
||||
- 合并卫星、轨迹、海缆和聚合渲染的重复计算,保留完整数据与现有交互;同步中英文 Earth、后台与运维文档。
|
||||
|
||||
---
|
||||
|
||||
## [0.75.4] — 2026-09-29
|
||||
|
||||
Released: 2026-09-29
|
||||
|
||||
### Highlights
|
||||
- 图例统一承载分类筛选,完善卫星微点与性能、圆形菜单 DPI 适配及星空颜色输出,清理废弃代码与旧方案。
|
||||
|
||||
### Improved / Fixed
|
||||
- 海缆、卫星、算力、船只、BGP 和国界图例直接切换显示;同步聚类、拾取、实时更新、选中卡片和覆盖状态,移除图层中的独立筛选入口。
|
||||
- 卫星使用单批硬边 Points,按参考校准远近点径与类别色,保留平滑短拖尾;400% 起显示真实标签,并加入可持久化 FPS 与滚动平均统计。
|
||||
- 缓存轨道插值半径、合并投影误差小于 0.2px 的冗余拖尾段、仅上传活跃缓冲;覆盖网格复用并在 GPU 投影,保留完整数据与网格精度。
|
||||
- 修复扇形菜单及缩放桥接的 DPI/hover 状态、星空局部 sRGB 输出、加载时菱形纬圈、地表标记侧面遮挡与登陆点海缆对齐;保留船只独立箭头并统一电视方角。
|
||||
- RSS/Atom 严格识别 BOM、HTTP/XML 编码,避免中文新闻乱码;删除旧随机星点、背景点、无效图例接口、未引用绘制分支及已完成的旧计划,更新双语说明。
|
||||
|
||||
---
|
||||
|
||||
## [0.75.3] — 2026-09-27
|
||||
|
||||
Released: 2026-09-27
|
||||
|
||||
### Highlights
|
||||
- 升级地球材质、地图标记与卫星展示,修复拖拽、采集缓存和任务队列,清理废弃渲染路径。
|
||||
|
||||
### Improved / Fixed
|
||||
- 高清地表采用海陆分离 Shader、可读夜侧与单层柔和大气;保留四张底图及真实昼夜,移除旧厚光壳和无用高光配置。
|
||||
- 卫星在真实轨道采样之间平滑移动,类别色在昼夜保持一致,蓝色覆盖与拾取位置同步;预测轨道开放并加入方向流光,消除强制闭合尖角。
|
||||
- 地表设施统一渐变徽标,BGP 保留不规则扁平例外;修复低缩放圆点开关,电视面板与播放器恢复方角。
|
||||
- 鼠标与单指拖出球体仍持续旋转,保留惯性并在极大倍率下限制灵敏度;修复欧拉角换支导致的反向和普通横拖侧歪。
|
||||
- CelesTrak 改为直接 GET、限时有效缓存和校验后原子替换;采集队列按 task ID 保留独立历史,防止失败任务被新任务或迟到消息重新激活。
|
||||
|
||||
---
|
||||
|
||||
## [0.75.2] — 2026-09-25
|
||||
|
||||
Released: 2026-09-25
|
||||
|
||||
### Highlights
|
||||
- 修复 Windows 启动器固定使用旧版 PowerShell 的问题,优先在 Windows Terminal 新窗口中运行 PowerShell 7。
|
||||
|
||||
### Fixed
|
||||
- 仓库与桌面 CMD 引导优先使用 `pwsh.exe`,缺少新版 PowerShell 或 Terminal 时保留可用环境回退。
|
||||
- 已以管理员运行的入口同样进入首选终端,子窗口校验权限后直接衔接 WSL,避免重复提权和递归开窗。
|
||||
- 补齐终端选择、含空格路径、提权分支回归和中英文说明,保持 WSL 重启参数及默认端口不变。
|
||||
|
||||
---
|
||||
|
||||
## [0.75.1] — 2026-09-25
|
||||
|
||||
Released: 2026-09-25
|
||||
|
||||
### Highlights
|
||||
- Windows 启动器从当前项目位置识别 WSL 发行版与目录,首次生成本机桌面 CMD,后续沿用标准重启流程。
|
||||
- 算力对比数据可由后台维护;小球恢复惯性拖动,大幅放大后保留球面跟手,修复移出球面后的反向旋转。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 桌面入口兼容两种 WSL UNC 路径和 Windows 本地磁盘映射,保留原有文件,支持只安装检查;默认端口仍为 3000 / 8000 / 8010。
|
||||
- 算力报告来源、年度观测与创新分差使用数据库配置,提供数值校验、版本冲突保护、事务审计、恢复默认和地球自动刷新。
|
||||
- 根据相机视锥在按下时固定拖动模式;小球使用旧版位移平滑与惯性,大球不再用球外最近点生成反向旋转。
|
||||
- 更新开发 agent 的 Sol / Luna / Astra 分工规则,补齐中英文运维说明、启动故障编号与隔离回归测试。
|
||||
|
||||
---
|
||||
|
||||
## [0.75.0] — 2026-09-23
|
||||
|
||||
Released: 2026-09-23
|
||||
|
||||
### Highlights
|
||||
- 新增算力与资源区域比较、可拖动的地理卡片和移动端原位展开;以算力占比为主,保留趋势、覆盖与来源信息。
|
||||
- 四张 8K 底图可切换,原始底图设为默认并列于首位;优化高清首帧加载,修复最大缩放穿透和高倍拖动不跟手。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 统一物件卡片的先绘线后显示时序与靠边布局;地球明显转向后成组调整区域卡片,减少交叉并保留阅读时间。
|
||||
- 合并算力区域目录及历史口径,校验设施坐标粒度与公开站点,避免以样本数冒充算力占比或将近似坐标规则排列。
|
||||
- 地表贴图异步解码与按需替换,启动模块打包、数据并行加载;复用共享下拉菜单、HUD 样式与移动端交互。
|
||||
- 国界贴合地表并贯穿固定地理显示规则;修复直播恢复、已选频道回退、本地账户身份响应与相关交互回归。
|
||||
- 补齐后端镜像所需规则、审核几何和构建脚本,记录高精国界重建要求,更新中英文使用及运维文档。
|
||||
|
||||
---
|
||||
|
||||
## [0.74.7] — 2026-09-17
|
||||
|
||||
Released: 2026-09-17
|
||||
|
||||
### Highlights
|
||||
- 启动进度统一显示在状态副行,成功清除、失败保留具体错误并退出,减少演示前的静默等待和重复报错。
|
||||
- 镜像仓库网络故障支持一次临时备用源回退,校验具体镜像 manifest 并锁定 digest,不改写全局镜像源。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 为公开 Python、uv、PostgreSQL 和 Redis 镜像提供 DaoCloud 回退及可配置前缀;认证、证书、限流和 Dockerfile 依赖安装错误不误触发回退。
|
||||
- 数据库启动前识别其他项目的端口占用,按后端实际配置同步本机映射,保留命名数据卷;修复停止后 TIME_WAIT 被误判为监听占用。
|
||||
- Docker 命令增加有界等待、私有日志和进程组清理;失败副行保留实际原因,避免外层汇总及清理通知覆盖。
|
||||
- 修复 Python 依赖清华源回退前过早冻结错误状态,移除 Compose 失败后的旧手动启动分支。
|
||||
- 同步中英文排障手册,补齐镜像回退、非网络错误、digest 校验、端口复用及失败退出回归,并验证实际备用镜像构建与项目启动。
|
||||
|
||||
---
|
||||
|
||||
## [0.74.6] — 2026-09-16
|
||||
|
||||
Released: 2026-09-16
|
||||
|
||||
### Highlights
|
||||
- AI Provider 构建前自动检测主机代理和直连路径,减少终端网络正常但 Docker 镜像拉取超时的问题。
|
||||
- 运维错误原因与脚本诊断共用一份对照表,统一错误编号、原因和处理建议。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 已有 Compose v2 时不再因构建或启动失败回退 v1;Dockerfile 改用 BuildKit 内置解析器,减少一次外部镜像下载。
|
||||
- 新增 --no-build 启动选项,明确复用本地 AI Provider 镜像,缺少镜像时报错且不改写构建指纹。
|
||||
- 自动代理检测尊重 NO_PROXY,仅修改 Planet 管理的本地 Docker 配置;更新前备份和校验,必要时重启并恢复原有容器,失败时回滚。
|
||||
- 中英文运维表覆盖网络、证书、权限、限流和启动故障;未知错误明确保留未归类状态,并要求新确认原因补表与回归用例。
|
||||
- 修复详细日志模式丢失构建失败退出码的问题,补齐代理切换、Compose 选择、错误分类及文档一致性验证,并收录算力与资源态势实施计划。
|
||||
|
||||
---
|
||||
|
||||
## [0.74.5] — 2026-09-13
|
||||
|
||||
Released: 2026-09-13
|
||||
|
||||
### Highlights
|
||||
- 恢复模型供应商预设的加载与持久化,模型刷新和轻量连接检查统一使用供应商官方目录接口。
|
||||
- 修复 MiniMax M3 思考模式和 OpenCode Go 模型协议路由,补齐 Responses 适配。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 修复 `llm_provider_preset:*` 动态设置分类触发 KeyError 和列表 500,使用真实数据库读写回归验证保存与重载。
|
||||
- 统一九家供应商的目录 URL、鉴权、响应解析与分页,保留服务地域和自定义地址;明确处理限流、空目录和失败,移除 404 白名单误判成功。
|
||||
- 模型刷新读取表单草稿地址和凭证,保留默认模型、未保存修改及失败前目录;连接提示区分目录可用与实际生成成功。
|
||||
- 增加 Responses 请求与响应解析、M3 adaptive 思考适配和 OpenCode 会话标识;已知模型选择同步协议,自定义 OpenAI 网关保留手动协议。
|
||||
- 同步中英文接口说明、操作手册与快速开始,并覆盖模型目录失败、草稿保留和协议适配回归。
|
||||
|
||||
---
|
||||
|
||||
## [0.74.4] — 2026-09-13
|
||||
|
||||
Released: 2026-09-13
|
||||
|
||||
### Highlights
|
||||
- 保留 Earth 全量对象与交互,优化海缆、登陆点和卫星绘制,并让 AIS / BarentsWatch 船只通过确认状态增量更新。
|
||||
- 新闻直播支持完整频道目录搜索和无限滚动;模型目录刷新可保存结果,启动流程减少不必要的依赖安装与等待。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 海缆和登陆点合批绘制,卫星 SGP4 计算移入 Worker、呼吸动画移入 GPU,并缩小动态文本的翻译扫描范围。
|
||||
- 船舶全局订阅按 MMSI 合并、拆包和原位更新,保留选择状态,并处理删除、重连及旧快照覆盖。
|
||||
- 新闻直播取消默认 120 项采集截断,增加数据库分页、完整目录搜索和固定计数栏,默认源改为半岛电视台 HLS。
|
||||
- 算力中心定位队列在当前 Earth 页面内独立于详情面板继续运行;模型刷新保存新目录并保留当前模型、凭证草稿与失败前目录。
|
||||
- AI Provider 镜像使用独立依赖组,容器直接运行已安装环境;启动提前验证数据库、及时识别后端失败,并保留可复用容器。
|
||||
|
||||
---
|
||||
|
||||
## [0.74.3] — 2026-09-13
|
||||
|
||||
Released: 2026-09-13
|
||||
|
||||
### Highlights
|
||||
- Ubuntu / WSL 新机器初始化会自动检测并准备 Docker Engine、Compose v2、Buildx 和当前用户权限,减少手工安装步骤。
|
||||
- 数据库初始化先核对容器端口与后端真实连接,连接和认证通过后才创建表和默认数据。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 区分 Docker CLI 缺失、服务未安装、daemon 不可用及 socket 权限不足,修正未安装 Docker 时误提示启动 socket 的诊断。
|
||||
- 自动补齐缺失的 Docker 依赖并启动本地服务,以原用户身份刷新 Docker 组权限;保留参数和 PATH,不依赖 sg。
|
||||
- 通过 Compose 同步已有 PostgreSQL / Redis 容器配置,保留端口冲突等具体错误;端口映射异常时最多保留数据卷重建一次 PostgreSQL。
|
||||
- 新增后端数据库只读连接检查,对认证、库名和网络失败给出不含密码或完整连接串的诊断。
|
||||
- 将 Docker 与数据库启动隔离回归测试接入快速检查,并同步 README、harness 和中英文运维说明。
|
||||
|
||||
---
|
||||
|
||||
## [0.74.2] — 2026-07-01
|
||||
|
||||
Released: 2026-07-01
|
||||
|
||||
@@ -79,6 +79,16 @@ git diff --unified=0 HEAD -- <path>
|
||||
|
||||
## Harness Commands
|
||||
|
||||
Windows desktop launcher tests run separately on Windows with
|
||||
`powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\harness\test_windows_launcher.ps1`.
|
||||
They use temporary desktop folders and mocked restart commands; no UAC prompt
|
||||
or real service restart is triggered. The WSL bridge regression suite,
|
||||
`scripts/harness/test_windows_restart.py`, is included in quick-check and uses
|
||||
disposable command substitutes. For a real entry-point check without restarting,
|
||||
run the repository `planet.cmd --install-only`, then the generated desktop
|
||||
launcher with the same flag. Installation and mocked checks do not prove a live
|
||||
elevated restart; report that separately when it has not been exercised.
|
||||
|
||||
| Tier | Command | What It Does |
|
||||
| --- | --- | --- |
|
||||
| Doctor | `scripts/harness/doctor.sh` | Checks required files, required tools, optional delivery tools, and forbidden frontend lockfiles. |
|
||||
@@ -86,7 +96,7 @@ git diff --unified=0 HEAD -- <path>
|
||||
| Backend Rules | `scripts/harness/backend-rules-check.sh` | Checks backend app Python for direct `print()`, `breakpoint()`, and `pdb.set_trace()` debug calls so service code uses structured logging. |
|
||||
| Frontend Rules | `scripts/harness/frontend-rules-check.sh` | Checks Bun-only scripts, admin route manifest coherence, literal internal route links, admin search route targets, frontend debug output, native button safety, icon-button accessibility, no nested Cards, no AntD/Space layout primitives, ConnectionTestInput usage, admin/docs shell height-chain sizing, same-category style owner warnings, viewport-scaled font sizes, zero letter spacing, and high-signal UI rule warnings. |
|
||||
| Docs Consistency | `scripts/harness/docs-consistency-check.sh` | Checks frontend Docs metadata against backend Gatekeeper metadata, public Docs registration, full technical-doc bilingual file pairs, public doc links, readable link titles, language-scoped technical links, README/project-context admin stack drift, supported credential collector contracts, manual console route coverage against the actual admin manifest, documented UI route drift, documented `?section=` deep-link validity against the actual admin section config in technical docs and active plan docs, and the harness rules-coverage notes. |
|
||||
| Quick | `scripts/harness/quick-check.sh` | Runs doctor, whitespace diff check, shell syntax checks, security scan, backend/frontend/doc consistency checks, and CI backend smoke tests. |
|
||||
| Quick | `scripts/harness/quick-check.sh` | Runs doctor, whitespace diff check, shell syntax checks, isolated Docker bootstrap/proxy and startup tests, error-catalog consistency tests, security scan, backend/frontend/doc consistency checks, and CI backend smoke tests. |
|
||||
| Full | `scripts/harness/validate.sh` | Runs quick check, frontend Bun install/build, Playwright route smoke, optional Helm checks, and opt-in Docker image smoke builds. |
|
||||
|
||||
Docker image smoke builds are expensive and are off by default:
|
||||
@@ -220,9 +230,26 @@ Optional for delivery smoke:
|
||||
- Docker daemon for image builds
|
||||
- Helm for chart lint/template checks
|
||||
|
||||
If a required local tool is missing, do not install system software
|
||||
automatically. Report the gap and point to `./planet.sh init` or
|
||||
`scripts/bootstrap-dev.sh` as the existing bootstrap path.
|
||||
For routine harness validation, do not install missing system software
|
||||
automatically. Report the gap and point to the explicit bootstrap entry points.
|
||||
`./planet.sh init` can install missing Docker Engine, Compose v2, and Buildx on
|
||||
Ubuntu / Ubuntu WSL, start the local service, and configure Docker group access.
|
||||
This bootstrap behavior is intentional; do not invoke it merely to make harness
|
||||
checks pass. `scripts/bootstrap-dev.sh` only prepares application dependencies.
|
||||
|
||||
Docker bootstrap regression checks use isolated command stubs and never install
|
||||
packages or modify the host daemon:
|
||||
|
||||
```bash
|
||||
uv run --frozen --project . python scripts/harness/test_docker_bootstrap.py
|
||||
uv run --frozen --project . python scripts/harness/test_database_startup.py
|
||||
```
|
||||
|
||||
Database startup regressions also run in quick-check. They cover Compose
|
||||
reconciliation of existing containers, visible startup errors, published-port
|
||||
checks, bounded recreation that preserves volumes, and the backend connection
|
||||
gate before schema initialization. Their command stubs and driver mocks do not
|
||||
modify the host Docker environment.
|
||||
|
||||
## What Agents Must Not Change Automatically
|
||||
|
||||
|
||||
@@ -22,12 +22,14 @@
|
||||
|
||||
当前重点入口:
|
||||
|
||||
- [开发 Agent 模型分配与 Token 控制计划](development-agent-model-allocation-plan.md)
|
||||
- [Earth 地理资源、空间浮动卡片与性能实施计划](earth-spatial-presentation-and-geographic-assets-plan.md)
|
||||
- [算力与资源态势:展示、采集与 AI 迭代实施计划](compute-resource-intelligence-plan.md)
|
||||
- [控制台 i18n 接入计划](/home/ray/dev/linkong/planet/docs/plans/admin-console-i18n-plan.md)
|
||||
- [Earth Mobile Drawer UI Plan](/home/ray/dev/linkong/planet/docs/plans/earth-mobile-drawer-ui-plan.md)
|
||||
- [Earth Compute Center BGP Style Plan](/home/ray/dev/linkong/planet/docs/plans/earth-compute-center-bgp-style-plan.md)
|
||||
- [Earth Renderer Architecture Separation Plan](/home/ray/dev/linkong/planet/docs/plans/earth-renderer-architecture-separation-plan.md)
|
||||
- [Earth 高精度国界静态瓦片计划](/home/ray/dev/linkong/planet/docs/plans/earth-high-precision-boundary-tiles-plan.md)
|
||||
- [Earth Predicted Orbit Plan](/home/ray/dev/linkong/planet/docs/plans/earth-predicted-orbit-plan.md)
|
||||
- [Earth WebGL Instancing Satellites Plan](/home/ray/dev/linkong/planet/docs/plans/earth-webgl-instancing-satellites-plan.md)
|
||||
- [Earth Real Terrain Plan](/home/ray/dev/linkong/planet/docs/plans/earth-real-terrain-plan.md)
|
||||
- [Earth News Source Configuration And Collector Plan](/home/ray/dev/linkong/planet/docs/plans/earth-news-source-configuration-and-collector-plan.md)
|
||||
|
||||
602
docs/plans/compute-resource-intelligence-plan.md
Normal file
602
docs/plans/compute-resource-intelligence-plan.md
Normal file
@@ -0,0 +1,602 @@
|
||||
# 算力与资源态势:展示、采集与 AI 迭代实施计划
|
||||
|
||||
状态:首版空间卡片原型实现中(2026-09-21)。已接入算力专题入口、45 个参与方目录、年份/口径选择、区域数据卡与来源、成功快照设施查询,以及 Epoch/TOP500 的首批质量修复。旧固定双图布局已移除。历史年度数据使用受控版本导入;各国完整指标覆盖、卡片图表、全榜获取、跨源设施合并、持久化指标管理、变化归因与 A4 AI 解读尚未完成,不视为全部 A1—A3 已验收。后续章节保留完整设计目标,当前实现以技术文档为准。2026-09-14 完成的口径核查仍是内容依据。面向产品、开发、数据研究与运维人员,作为此专题后续实现的设计依据。
|
||||
|
||||
## 0. 当前执行方向与范围
|
||||
|
||||
2026-09-21 展示方向修订:UI、地球材质、政治展示规则、地理资源对齐及性能门槛以[Earth 地理资源与空间卡片计划](earth-spatial-presentation-and-geographic-assets-plan.md)为准。下文固定右侧国家面板、底部横贯双图等布局不再作为实施目标。采用从区域锚点引出连接线、卡片围绕地球自适应浮动的方向,先验证空间交互再完善卡片内容。数据来源、口径与证据要求继续有效。
|
||||
|
||||
平台长期定位是“信息展示 + 决策辅助”。当前实施重点是把全球算力对比作为第一个成熟专题做清楚:地图显示资源在哪里,图表说明规模和相对位置怎样变化,证据说明数字从哪里来,AI 帮助阅读变化。
|
||||
|
||||
采用“专题先交付,公共能力按复用需要沉淀”的顺序。第 7—9 节的完整 AI 迭代、模型实验与通用持久化是后续路线,不是首版展示的前置条件;第 6 节总分保持实验性,不能为了首屏有大数字而生成不完整排名。
|
||||
|
||||
| 当前要做 | 当前完成形态 | 后续才扩展 |
|
||||
| --- | --- | --- |
|
||||
| 算力数据可信 | 修复既有采集,国家/设施/精度/年份不混用 | 大规模自动来源发现、完整证据图谱 |
|
||||
| 全球分布与国家对比 | 现有算力中心地球 + 可比参与方 + 明确样本覆盖 | 制造、矿产、电力和贸易的跨主题关系 |
|
||||
| 历史展示 | 两张核对后的图、时间选择、已知事件轨道 | 复杂预测、不确定性传播与可训练参数 |
|
||||
| 证据与解释 | 来源、参考年、关键字段;基于数据快照的简短 AI 说明 | 自动补证、模型候选、回测、影子运行与提升 |
|
||||
|
||||
首版验收路径:打开算力专题 → 看全球公开设施分布 → 选中国和美国 → 同时看算力与创新历史 → 点一个变化或设施 → 查看出处及解释。此路径稳定后,才扩大评价维度。
|
||||
|
||||
## 1. 要交付什么
|
||||
|
||||
在智能星球现有算力中心图层上增加“算力与资源态势”模式,并在控制台提供对应的数据管理和模型实验工作区。核心问题是:各方拥有什么资源、资源如何形成可用能力、相对位置如何变化、变化有何证据,以及未来什么约束可能改变格局。
|
||||
|
||||
首批比较中国和美国;数据结构从第一天支持多方。随后纳入欧洲、日本、韩国、印度及其他数据合格的参与方。地区汇总与成员国不得重复计入全球分母。所有国家和地区字段使用现有 `normalize_country()` 和规范字典,不以网页原文或 AI 判断覆盖项目标签。
|
||||
|
||||
交付三个相互连接的产品能力:
|
||||
|
||||
| 能力 | 用户获得的结果 | 首版边界 |
|
||||
| --- | --- | --- |
|
||||
| 历史观测 | 算力绝对规模、全球份额、创新指数与分差、设施时间线 | 使用可验证观测;缺失年份可见,不插值冒充事实 |
|
||||
| 变化解释 | 哪些设施、资源、数据修订造成指标变化,每项贡献和证据 | 数学贡献可复算;原因解释区分事实、假设与未解问题 |
|
||||
| 模型改进 | AI 发现缺口、补证、提出改进,经过验证产生新版本 | 固定工作流先行;不能由大模型直接决定国家分数 |
|
||||
|
||||
第一版先交付可信的事实与回放,不以总分完整作为上线前提。完整创新指数分项、有效算力参数和项目交付数据不足时,继续展示已完成的观测能力,综合评分显示“数据不足”,不生成示意国家排名。
|
||||
|
||||
不在第一版承诺全球所有设施的完整清单、精确的国家真实算力、每日变化的国家创新能力、可验证的国家综合实力真值,或自动预测某年必然追平。
|
||||
|
||||
## 2. 当前项目基础与必须补齐的边界
|
||||
|
||||
以下结论来自当前文件检查,未对生产数据库、凭证可用性或线上采集结果做完整审计。
|
||||
|
||||
| 已有入口 | 已确认能力 | 本专题需要补齐 |
|
||||
| --- | --- | --- |
|
||||
| [采集基类](../../backend/app/services/collectors/base.py)、[数据作业](../../backend/app/services/data_jobs.py) | 采集、转换、任务状态、快照、取消与重试基础 | 文档型数据源、质量门槛、领域规范化、失败保留正式快照 |
|
||||
| [Epoch 采集器](../../backend/app/services/collectors/epoch_ai.py) | GPU 集群采集入口 | 改读公开 CSV;去掉解析失败返回示例记录;单位、精度、状态和时间字段校正 |
|
||||
| [TOP500 采集器](../../backend/app/services/collectors/top500.py) | 榜单与详情解析 | 去掉示例回退、固定参考日期、排名作为身份;按当期表头解析单位 |
|
||||
| [采集记录](../../backend/app/models/collected_data.py)、[快照](../../backend/app/models/data_snapshot.py) | `entity_key`、快照关联、前一记录和变化摘要 | 原始证据版本、发表时间、有效时间、指标注册和可重放的观测集合 |
|
||||
| [算力位置服务](../../backend/app/services/compute_center_locations.py)、[GeoJSON](../../backend/app/api/v1/visualization.py) | 设施定位、位置置信信息、未定位记录 | 国家统计与地图可定位集合分离;匿名设施不能猜坐标 |
|
||||
| [算力中心渲染](../../frontend/public/earth/js/compute-centers.js)、[Earth 壳](../../frontend/src/pages/Earth/Earth.tsx) | Three.js 交互层;Earth 由 iframe 加载独立页面 | 历史时点、国家比较、双图、关系层和解释面板 |
|
||||
| [图层适配器](../../backend/app/services/earth_layer_adapters.py)、[数据库监听](../../backend/app/services/earth_db_change_listener.py) | 通过 PostgreSQL outbox 驱动缓存与 `earth_updates` | 正式指标快照发布事件;草稿与实验结果不改变默认地球 |
|
||||
| [AI Client](../../backend/app/services/ai_client.py)、[提示词注册](../../backend/app/ai_tasks/prompts.py) | 全局 provider 配置、模型调用、任务提示词及运维覆盖 | 算力专题的固定工作流、结构化输出验证、预算与审计 |
|
||||
| [搜索工具](../../backend/app/services/ai_tools/web_search.py)、[网页取证](../../backend/app/services/ai_tools/web_fetch.py) | 搜索和基础 HTML 正文提取 | PDF/表格提取、引用位置、下载时大小限制、URL 与重定向的访问限制 |
|
||||
| [证据辅助函数](../../backend/app/services/ai_tools/evidence_store.py) | 摘要、去重和内容哈希规范化 | 该文件不是持久化证据库,需补建不可变原文存储和证据记录 |
|
||||
|
||||
沿用[数据作业与 Outbox 架构](../technical/zh/data-job-earth-sync-architecture.md)及[轻量 Agent 编排计划](agents-light-orchestrator-websearch-plan.md):PostgreSQL 负责可靠账本,Redis 负责缓存;不另起 Kafka、Celery 或另一个独立调度真相源。`aiprovider` 保持协议适配,业务取证、估计、工具权限和模型发布全部由后端负责。
|
||||
|
||||
当前 `JobType` 只有采集、清理和 Earth 刷新。新增分析、实验工作流需要显式扩展任务类型、worker 分发和取消策略,不能把现有采集队列描述成已经支持通用 Agent。
|
||||
|
||||
## 3. 页面与地球怎样展示
|
||||
|
||||
### 3.1 默认工作区
|
||||
|
||||
保留 `/earth` 主入口,在现有算力中心工具中增加专题模式,避免增加一个与地球无关联的新大屏。进入后使用以下概念布局;这是规划线框,不是现有页面截图。
|
||||
|
||||
```text
|
||||
┌ 时间/参考期 · 中国/美国/其他参与方 · 指标口径 · 观测/情景 ┐
|
||||
│ │
|
||||
│ 3D 地球:设施、资源、已证实关系 国家对比面板 │
|
||||
│ 绝对量/份额 │
|
||||
│ 变化贡献 │
|
||||
│ 证据与缺口 │
|
||||
├────────────── 历史时间轴 / 事件轨道 ────────────────────┤
|
||||
│ 算力规模与份额图 综合创新指数与分差图 │
|
||||
└───────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
桌面默认同时显示两张历史图,用相同国家配色,但各自保留单位和实际观测年份。图表高度优先保证刻度、来源标识与数据点可读,不能通过压缩字体维持布局。较窄屏幕把图表移入可切换的详情区,国家和设施详情使用抽屉;不叠加多个遮挡地图的浮层。
|
||||
|
||||
沿用项目一屏高度链和主题。桌面在 1366×768、1920×1080,移动端在约 390×844 验证,另测 125%/150% 浏览器缩放。页面根与中间 flex 容器分别正确设置高度和 `min-height: 0`,仅详情、长表和证据正文内部滚动。
|
||||
|
||||
### 3.2 视图职责
|
||||
|
||||
| 视图 | 主内容 | 点击或切换后的行为 |
|
||||
| --- | --- | --- |
|
||||
| 国家总览 | 可比算力绝对量、份额、子分、实际参考期;总分合格才出现 | 选中参与方,定位地球并联动两图;不重设全局评分尺度 |
|
||||
| 算力趋势 | 绝对规模、全球份额、双边份额三种独立视图 | 同一来源口径内切换;不把百分比轴和 FLOP/s 轴混合 |
|
||||
| 创新趋势 | 各方指数;只有分差时只显示中美分差 | 切换年度,查看分项与报告版本,不能反推缺失的单国得分 |
|
||||
| 变化账单 | 新投产、扩建、退役、效率变化、资料补录、方法修订 | 点选贡献定位资产;逐步展示输入、计算和证据 |
|
||||
| 设施详情 | 设备、精度、投产状态、功率、运营方、地点精度、历史版本 | 展开相关公告、设备来源及不确定字段 |
|
||||
| 资源关系 | 已证实的芯片供给、所有权、电力接入或网络连接 | 选一类关系再显示,不能让所有关系默认铺满地球 |
|
||||
| 情景工作区 | 并网推迟、交付变化、效率假设及结果区间 | 独立 scenario ID;退出恢复正式快照 |
|
||||
| 控制台研究区 | 数据缺口、待核验证据、采集任务、模型实验和版本 | 使用已有采集与 AI 设置入口;Playground 仍用于调试 |
|
||||
|
||||
### 3.3 两张图进入产品的明确规则
|
||||
|
||||
总算力图以核对后的报告观测为起点:2020 年美国/中国为 36%/31%,2021 年 34%/33%,2022 年 34%/33%,2023 年 41%/31%。原始来源和报告版本见第 15 节。2024—2025 年暂不填值;2026 年 42%/33% 只保存为待核实主张,不能接到正式历史曲线上。
|
||||
|
||||
创新图使用报道明确披露的 2023、2024、2025 年分差 22.02、19.96、17.95 分。报告发表于 2026 年,必须区分报告年份与参考年份。完整分项、年度得分和方法修订仍需采集。图中是综合创新指数,不是纯算力或纯产出。
|
||||
|
||||
原始主张、报告估计、已验证记录和未来情景用文字标签区分。年度点之间的视觉连线只辅助阅读,不进入计算;不能把连线插值写成月度实测。正式指标没有新观测时保留参考期并提示新鲜度,不让刷新网页造成“国家实力实时跳动”。
|
||||
|
||||
### 3.3a 当前数据合同(2026-09-22)
|
||||
|
||||
默认主指标是全球计算设备总规模,拆分基础计算、智能计算和超级计算,观察期为 2020—2023。CAICT 报告核查目前使用官方 PDF 索引文本;本地 PDF 下载未成功,因此不宣称已肉眼核验 PDF 页面。绝对量合同记录 2022 年 302(证据版本 `caict2023`)和 2023 年 435(证据版本 `caict2024`);2024 年报告正式书名为《先进计算暨算力发展指数蓝皮书(2024年)》。官方 URL 已替换为四个镜像来源。
|
||||
|
||||
日本全球份额观测为 2020/2021/2022/2023 = 6%/5%/4%/4%。产品分组 `APAC` 与 `OTHER` 不能直接套用报告中的 Japan 或“其他”;`EUROPE` 成员集合尚未与报告口径对齐,仍返回缺失。双边份额分母固定为 US + China,新加入日本不改变分母。主要指标缺失时可显示最近有报告的年份(当前可到 2023),而不是留空为 2026;用户明确选择 2026 时仍显示缺失,不回填最近值。不可比性尚未确认,不展示未经确认的同比变化;覆盖数据保留在辅助区并带 count。
|
||||
|
||||
数据 API 审计字段为 `reviewed_at=2026-09-22`,版本与该日期一致。独立 Epoch 份额模式尚未实现,用户对该问法的选择待后续确认;当前合同不提前新增该模式或声称完成 dataset 计算。
|
||||
|
||||
默认“历史回顾”允许查看最新证据重述后的过去;“当时可知”只使用在截止时点已经发表且可证明可获得的资料。每张图同时展示参考期与知识截止时间。已选 2025 年而指标最新只到 2023 年时,必须显示“最新观测:2023”,或在严格同年模式显示缺失。
|
||||
|
||||
### 3.4 地球编码与交互
|
||||
|
||||
- 节点面积按同口径能力编码;跨度过大时可切换对数档位并明示图例。国家颜色只用于身份,不能把颜色本身当作领先或落后。
|
||||
- 已投产用实心,建设或情景用空心/虚线;置信不足使用独立标记,不能把低置信透明度误读为小算力。
|
||||
- 来源只到国家或区域时,保留在国家/区域统计和未定位列表;不画在首都,不把行政区域中心当作设施坐标。
|
||||
- 国家总量与可见设施合计可以不同,分别显示覆盖范围。不得只统计有坐标的节点,也不得把宏观缺口按现有节点比例摊分。
|
||||
- Epoch 当前公开版本对中国部分集群匿名化并取整;匿名记录只能按允许的粒度使用。新增官方公告可作为独立证据,不能依据匿名数值猜出设施身份或制造地点。
|
||||
- 供应、所有权和网络关系采用不同图例,显示关系来源与有效期;商业供应关系不等同于实际运输路线。
|
||||
- 全国可汇总吞吐与最大单集群能力分别展示;地理分散设备不能合并成一个训练集群。
|
||||
- 时间回放使用稳定实体 ID 更新已有节点。隐藏图层、变更时点或情景时同步清理 hover、locked、tooltip 和 selection;请求过期结果不得覆盖新时点。
|
||||
- 使用现有 Three.js Interactable、图标和实例化能力。新增壳层必须遵循现有深度顺序,在远距约 50% 缩放检查闪烁;不另建悬浮球面掩盖深度问题。
|
||||
|
||||
## 4. 数据需要补充采集什么
|
||||
|
||||
### 4.1 P0:决定首版可信度的数据
|
||||
|
||||
“已核对入口”只表示资料或文档可查,不表示接口凭证、完整历史和生产连通性已经验证。以下频率是 Planet 建议检查频率,不是来源承诺的更新频率。
|
||||
|
||||
| ID / 优先级 | 数据与来源 | 必需字段/粒度 | 采集方式与频率 | 当前缺口与采用条件 |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| D01 / P0 | 信通院历年算力报告 | 国家×参考年×口径;绝对规模、全球总量/份额、单位、精度、估算方法、版本 | 已取得报告先受控导入;月度检查新版;PDF 表格提取后核验 | 已核对 2020—2023 份额及部分绝对量;补 2024 以后和其他国家;跨版本方法改变时断开序列 |
|
||||
| D02 / P0 | Epoch GPU Clusters | 集群/版本;设备数、型号、16/8/32-bit 性能、所在地、状态、投产/退役时间、功率、扩建关系 | 官方公开 CSV,每日条件抓取,按内容哈希去重 | 改造现有 HTML 解析;处理匿名、取整、缺坐标和覆盖偏差;不能声称完整全国清单 |
|
||||
| D03 / P0 | TOP500 / 可选 Green500 | 稳定系统 ID×榜单期;Rmax/Rpeak、FP64、功率、机构、名次、列表日期 | 每周检查新榜单;按发布期导入历史与详情 | 修复 rank 作为 source ID、固定日期和单位问题;榜单消失不等于退役;不当成 AI 总算力 |
|
||||
| D04 / P0 | 全球 AI 创新指数原报告及发布材料 | 国家×参考年×指标;单国总分、分项、权重、标准化、成员范围、方法版本 | 文档导入+月度更新检查 | 当前只具备部分新闻转述分差;优先获得分项和年度可比性证据;不能用差值合成各国历史得分 |
|
||||
| D05 / P0 | 已有位置维表与公开设施公告 | 实体别名、运营方、国家、地点、精度、采用依据、有效期 | 复用位置管线,新增/变化时触发 | 全国汇总独立于定位成败;不把同址不同分期误合并,也不把扩建当第二个全量集群 |
|
||||
|
||||
首版最低闭环:D01 和 D04 的已验证历史可先支撑双图,D02/D03/D05 支撑设施地图;两者不互相伪装为同一覆盖范围。若 D04 分项不可取得,创新总分仍作为独立观测,O 分与综合 S 不发布。
|
||||
|
||||
首版可以先受控导入已核实的年度数据,不必先完成整套 PDF 自动解析和报告发现。D04 原报告分项继续作为补数任务,不阻塞已经核对的创新分差图;原始摘录、出处、年份和未核实范围必须随数据保存。
|
||||
|
||||
### 4.2 P1/P2:有效能力、增长与资源关系
|
||||
|
||||
| ID / 优先级 | 补充数据与候选来源 | 粒度与核心字段 | 方式/频率 | 对模型的作用及限制 |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| D06 / P1 | Epoch 芯片销售、所有权、硬件数据;厂商规格 | 芯片型号×时间;规格、销售/交付、持有主体、数量、性能换算、估计区间 | 官方数据文件与规格文档;周查/季度快照 | 校验硬件供给;设计者、所有者、实际所在地与使用权分开;不能把出货和已部署重复加总 |
|
||||
| D07 / P1 | MLPerf Training / Inference、可复现厂商测试 | 配置×任务×软件版本;模型、质量门槛、batch、上下文、时延、吞吐、能耗、可用状态 | 官方结果导入;月查新版本 | 校准特定任务的性能系数;不同 benchmark 版本/质量/负载不直连;实验室成绩不等同生产利用率 |
|
||||
| D08 / P1 | IEA、电力统计机构、EIA、并网与运营方资料 | 国家/地区/项目×时点;发电、用电、接入 MW、PUE、并网日期、可用负荷 | 国家统计月/季;报告年;项目公告日查 | 电力作为设施约束;国家发电量、用电 TWh 不能直接转换机房可用 MW;中国与美国地方数据可得性不同 |
|
||||
| D09 / P1 | 项目运营方、许可部门、官方采购和工程进展 | 项目×阶段;公告、开工、设备交付、并网、验收、部分投产、撤销、金额与范围 | 已注册来源每日检查,重要项目每周核验 | 建立投产与延期标签;最终失败项目也保留,避免只收集成功项目;单纯新闻未更新不判定失败 |
|
||||
| D10 / P1 | 公司投资者关系、SEC EDGAR、境内及其他市场法定披露 | 法人/分部×财季;CapEx、已付款/指引、租赁、合同、地域/业务归属 | 官方 API/报告;日查新披露,按季归档 | 用于资金与项目约束;总 CapEx 不是 AI 投资;合并报表、融资租赁、设备与机房成本须防重复 |
|
||||
| D11 / P1 | 创新报告分项、公开模型评测、OpenAlex;HF 作为辅助 | 机构/国家×年/月;研发成果、质量归一化产出、可比模型能力、应用指标 | 原报告年;评测周;研究元数据月 | 重建不含重复资源项的 O;合作成果使用明确分摊;不从作者姓名推断国籍;下载量不代表实际使用或产业产出 |
|
||||
| D12 / P2 | 晶圆制造、先进封装、HBM、关键设备的正式披露 | 工厂/供应方×工艺/产品×季度;已投产能力、良率范围、交付、客户与依赖 | 公司/监管原件;月查、季度快照 | 建立瓶颈和替代关系;晶圆数不能无依据换成 GPU 数;总行业份额不能归给某集群 |
|
||||
| D13 / P2 | USGS、各国统计、UN Comtrade | 商品×国家×年/月;产量、储量、加工量、贸易流、HS 版本和单位 | 年度报告、月/年贸易数据 | 扩展矿产与资源依赖;储量、开采和加工分开;宽泛服务器税号不能反推高端芯片数量 |
|
||||
| D14 / P2 | 既有海缆、PeeringDB、BGP 与已证实连接资料 | 设施/网络/关系×有效期;容量声明、连接、依赖、故障 | 复用现有产品,按其真实更新节奏 | 提供连通性和韧性背景;BGP 可见性不是带宽,海缆容量不是集群内部互联性能 |
|
||||
|
||||
EIA、SEC、Epoch、MLCommons、IEA、USGS 已核对资料入口;UN Comtrade 门户可见,但本次未验证可调用 API,接入要单独完成认证、额度、字段和历史测试。OpenAlex 需在接入时按最新访问规则配置,不假定无限制匿名访问。所有付费、登录和受限数据只列为可选增强,不作为首版隐藏前提。
|
||||
|
||||
### 4.3 首批采购与补数优先顺序
|
||||
|
||||
1. 先完成现有 Epoch/TOP500 的真实性与时间口径修复,避免新增来源扩大错误。
|
||||
2. 导入信通院历史、创新分差和原始出处;追索创新原报告分项及最新版总算力数据。
|
||||
3. 建立中美重点设施及项目阶段台账,并记录未知量。重点设施用于高影响解释,不冒充全国抽样无偏估计。
|
||||
4. 用官方规格与 MLPerf 建立少量明确工作负载的换算表;其余硬件维持理论值或范围。
|
||||
5. 补项目级并网、交付、资金与退役记录,积累可校准的真实标签。
|
||||
6. 最后扩展制造、HBM、矿产与多方关系;每个新主题必须有对应指标和展示消费者,不只收集而不使用。
|
||||
|
||||
## 5. 采集到正式数据的流水线
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A[注册的数据源] --> B[原文与内容哈希]
|
||||
B --> C[确定性解析或 AI 提取候选]
|
||||
C --> D[单位/时间/实体/引用校验]
|
||||
D --> E[版本化有效观测]
|
||||
D --> F[冲突与待核实队列]
|
||||
E --> G[确定性指标计算]
|
||||
G --> H[正式结果快照]
|
||||
H --> I[Outbox 与缓存更新]
|
||||
I --> J[地球/双图/变化账单]
|
||||
```
|
||||
|
||||
### 5.1 数据源合同
|
||||
|
||||
每个数据源保存来源机构、域名与下载入口、文档类型、凭证配置引用、检查频率、预期粒度、单位与精度、更新时间语义、空结果语义、历史范围、使用许可、维护负责人和解析器版本。授权文件或数据保存在运行数据存储,不提交到仓库;公开 UI 只展示允许公开的摘录或链接。
|
||||
|
||||
API/CSV 优先;HTML/PDF 采用固定解析;必须用 AI 时先产生候选,核验后保存确定性映射。扫描件 OCR 必须保留页号、表格区域和原图,数值不能只靠模型复述。下载器在流式读取时限制实际字节量,不能下载完整文件后才截断;按域名限速,遵守来源规则。
|
||||
|
||||
### 5.2 事实键与时间
|
||||
|
||||
观测键至少包括实体、指标、单位、精度、工作负载、地理范围、归属口径、参考期及来源版本。历史至少区分:
|
||||
|
||||
| 时间 | 含义 |
|
||||
| --- | --- |
|
||||
| `reference_start/end` 或 `effective_at` | 数据描述的年度、季度或设施事件发生时间 |
|
||||
| `published_at` | 资料何时首次公开;不知道就保持未知 |
|
||||
| `retrieved_at` | Planet 何时取得该版本 |
|
||||
| `recorded_at` / `superseded_at` | 系统何时采用、替换该观测 |
|
||||
|
||||
抓取日期不代替投产日期;报告年份不代替参考年份;机构估计、实测、公告、模型估计和人工情景分别标识。只有年份时保留年份精度,不能伪造 1 月 1 日投产;回放按时间范围或“当年”事件显示。
|
||||
|
||||
### 5.3 合并与质量门槛
|
||||
|
||||
- 稳定身份优先使用官方系统 ID、项目 ID、法人 ID 和可信来源链接。TOP500 排名只作为观测属性。自动实体合并必须可撤销,保存别名和合并理由。
|
||||
- 同一集群的全量扩建公告、一期/二期、重复转载与另一个数据集的同一设备只计一次。芯片交付和机房投产分别保留,不能合并为两个已投产资产。
|
||||
- 国家级统计、企业所有权统计、设施地理统计是不同视图。国家公司在海外租用算力不得同时作为两国境内部署能力;全球成员集合固定且版本化。
|
||||
- 转换单位前先核验浮点精度、稀疏/稠密、峰值/实测和时间量纲。FLOP 与 FLOP/s、TWh 与 MW 均不得互换。
|
||||
- 解析为空、字段突然缺失、总量异常下降或源结构改变,进入失败/隔离状态,保留上次正式结果。缺榜、缺行或匿名化不直接触发实体退役。
|
||||
- 多条转载归到同一 `source_family`;媒体数量不构成多份独立证据。冲突先检查口径,再保存多个主张及采用规则,不能简单取平均。
|
||||
- 缺坐标不影响已知国家统计;未知国家不强行分配。数据覆盖率可描述已登记实体或必填字段的完整率;无法知道总体时不声称“覆盖全球百分之多少”。
|
||||
- 一位有效数字等取整值保留取整范围;这是观测精度范围,不冒充统计置信区间。
|
||||
|
||||
首批历史导入先双人或规则加人工抽查核验高影响记录,保存一套回归样本。后续同版本确定性解析通过固定门槛可自动采用;新来源、新语义或无法解释的大幅变化进入候选队列。
|
||||
|
||||
## 6. 数学模型如何工作
|
||||
|
||||
### 6.1 分清事实、估计和评价
|
||||
|
||||
模型分三层:观测层保存来源说了什么;估计层回答可用能力、交付时间和不确定性;评价层按明确价值取向组合指标。国家综合实力没有一个可直接取得的标签真值,因此不能声称“AI 找到了客观最优的国家评分权重”。
|
||||
|
||||
保留 v2 的模型草案:
|
||||
|
||||
\[
|
||||
S_{c,t}=0.40H_{c,t}+0.30O_{c,t}+0.20G_{c,t}+0.10R_{c,t}
|
||||
\]
|
||||
|
||||
H 是资源能力,O 是不重复的创新与应用表现,G 是扩张潜力,R 是韧性;四项均为 0—100 的版本化子分。S 称为“综合态势分”,包含未来潜力,不能称作已投产算力、胜率或国家实力百分比。权重是默认设计参数,需通过敏感性分析和评价目的评审后冻结。
|
||||
|
||||
### 6.2 资源能力 H 与两类算力模式
|
||||
|
||||
总算力模式使用 D01 的同口径绝对规模;AI 训练模式使用明确工作负载的有效能力;推理模式另选固定模型、质量、上下文与时延要求。不同模式不共用没有物理依据的换算,也不比较彼此总分。
|
||||
|
||||
\[
|
||||
C^{eff}_{c,t,w}=\sum_{j\in deployed(c,t)} C^{peak}_{j,t,p}\,u_{j,t,w}
|
||||
\]
|
||||
|
||||
`p` 是固定精度,`w` 是参考工作负载;`u` 由可比测试、已知运行约束或经校准的估计支持。没有依据时只发布理论能力,或者明示区间;不能统一假设某国芯片只有另一个国家的固定折扣。
|
||||
|
||||
内存、互联、软件和电力会影响同一个有效利用系数,不得分别随意乘一串折扣重复惩罚。若已知场地供电上限,可用场地总功率除以 PUE 约束 IT 负荷,再按同配置效率估计可运行容量;已在利用系数中计入的限电不再扣一次。总吞吐与可用于单次大训练的最大连续集群另列。
|
||||
|
||||
国家宏观估计和公开集群样本并列校验,不直接把二者相加。若 H 只能覆盖已观测设施,分数标题和参与方排名必须明确限定为这个样本,不外推为全国真实能力。
|
||||
|
||||
### 6.3 创新与应用 O
|
||||
|
||||
完整创新指数 I 单独显示。取得 D04 分项后,建立指标归属表,逐一标注资源、扩张、韧性或创新;只把不重叠的研发质量、模型/科研成果与应用表现用于 O。不是简单执行“创新总分减去算力分”,因为不同分项的权重和标准化可能不同。
|
||||
|
||||
备用指标可来自 D11,但新增指标意味着新定义和新版本,不能偷偷替换原报告。论文数量须去重、处理合作分摊及引用年龄;模型结果要冻结评测版本;应用指标明确平台样本。专利、下载、融资和模型排行榜均不是彼此的等价替代。
|
||||
|
||||
\[
|
||||
D^I_t=I_{US,t}-I_{CN,t},\qquad v^I_t=(D^I_{t-1}-D^I_t)/\Delta years
|
||||
\]
|
||||
|
||||
2023—2025 年披露分差缩小 4.07 分,约为起点分差的 18.5%;不能解释为中国能力提高 18.5%,也不能据此推断算法效率提升。只有差值不能恢复各国绝对得分。跨年方法不一致时分段显示,不能外推追平时间。
|
||||
|
||||
### 6.4 扩张潜力 G
|
||||
|
||||
\[
|
||||
G^{raw}_{c,t,h}=\sum_{j\in pipeline(c,t)}P(T_j\leq t+h\mid X_{j,\leq t})\,\Delta C^{eff}_j
|
||||
\]
|
||||
|
||||
默认 `h=12 个月`。X 只含当时可知的项目状态、设备交付、并网、建设和资金证据。分期投产拆分未交付增量;已投产部分进入 H 后从剩余 G 扣除。公告金额不直接转成已投产能力。
|
||||
|
||||
初期采用明确的保守/基准/乐观情景,没有历史标签就不提供伪精确概率。标签积累后可比较阶段条件概率、校准的逻辑回归或生存分析;处理尚未到期项目的右删失、取消与部分交付,不能把所有未完成记录标成失败。
|
||||
|
||||
### 6.5 韧性 R
|
||||
|
||||
\[
|
||||
R_{c,t}=100\sum_s q_s\,\min(1,C^{eff}_{c,t,s}/C^{eff}_{c,t,base})
|
||||
\]
|
||||
|
||||
s 是对各方一致定义的冲击情景,例如特定供应类别中断或并网延迟;q 为公开、固定的情景权重,除非有概率依据,否则不称为发生概率。基准能力为零或未知时不计算此比值。
|
||||
|
||||
关系图用于识别依赖、替代和共同故障源。没有替代来源证据就标为未知;政策公告通常改变未来供应或可用性假设,不自动删除境内现有芯片存量。
|
||||
|
||||
### 6.6 标准化与发布条件
|
||||
|
||||
数量型正向指标可采用固定基期映射:
|
||||
|
||||
\[
|
||||
N(x;a,b)=100\,clip\left(\frac{\ln(1+x/a)}{\ln(1+b/a)},0,1\right),\quad a,b>0
|
||||
\]
|
||||
|
||||
a、b 与 x 单位一致,来自有覆盖说明的固定参考面板,存入模型版本。成本型、强度型和比例型指标各自定义映射。不得按每日领先国家重定标,也不能因加入一个国家导致全部历史分数被静默重写。
|
||||
|
||||
关键输入缺失时 `S=null`,保留可用子分,不填零或自动重分配权重。O、G 或 R 没有可比输入时综合分不上线;原始图表和证据浏览仍可上线。样本少、匿名取整或来源偏差反映在不确定性和适用范围中,不直接扣国家实力分。
|
||||
|
||||
不确定性计算分开保存观测精度、参数估计和情景假设。只有有依据的输入分布才运行 Monte Carlo;共享芯片规格、同一公告、同一供应链的误差必须相关采样。无法量化的覆盖偏差直接披露,不能靠窄置信区间掩盖。排名用“无法区分”处理不稳健结果,不展示过多小数制造精确感。
|
||||
|
||||
### 6.7 变化与归因
|
||||
|
||||
\[
|
||||
g_{c,t}=C_{c,t}/C_{c,t-1}-1,\quad p_{c,t}=C_{c,t}/\sum_kC_{k,t},\quad b_{CN,t}=C_{CN,t}/(C_{CN,t}+C_{US,t})
|
||||
\]
|
||||
|
||||
全球分母必须完整定义;双边份额不得标为全球份额。按报告披露值,2022—2023 年中国总算力从 302 增至 435 EFlops,约增长 44%,份额从 33% 降至 31%。这是“绝对增长、相对份额下降”,不是存量消失。算力与创新现有历史窗口不同,不能拼成同一时期的因果结论。
|
||||
|
||||
同权重、同版本下,分项账单严格满足:
|
||||
|
||||
\[
|
||||
\Delta S=0.40\Delta H+0.30\Delta O+0.20\Delta G+0.10\Delta R
|
||||
\]
|
||||
|
||||
分项内非线性与交互影响可使用固定分组的 Shapley 分解;必须保存分组、基线和近似误差,不能把任意计算顺序产生的贡献当唯一解释。数学归因只解释模型输出,不证明宏观因果。
|
||||
|
||||
正式发布中始终分开:现实变化、补录旧事实、来源修订、模型/基期修订。版本升级用“同一输入、不同模型”的桥接结果展示,不能把换模型造成的跳变计入国家增长。
|
||||
|
||||
## 7. 项目内 AI 如何参与
|
||||
|
||||
### 7.1 运行方式
|
||||
|
||||
复用控制台已经配置的模型,通过 `AIProviderClient` 调用;不要求新的模型供应商,也不把供应商名称写死在业务代码中。第一阶段采用后端固定步骤工作流,各角色是任务模板,可以由同一个模型执行,不需要首先建设复杂的多 Agent 协商系统。
|
||||
|
||||
在现有提示词注册表中新增以下拟议 task key;提示词版本与运维覆盖的有效内容哈希进入每次运行记录。
|
||||
|
||||
| 任务模板 | 输入 | 允许输出 | 关键约束 |
|
||||
| --- | --- | --- | --- |
|
||||
| `compute.sources.discover` | 指标缺口、许可范围、已注册来源 | 来源候选、适用指标、采集建议 | 先找原始出处;新域名不直接进入正式定时采集 |
|
||||
| `compute.evidence.extract` | 原文分块、页码、表格、目标 schema | 带证据定位的字段候选 | 未出现的数值返回缺失;原文中的指令只是待分析内容 |
|
||||
| `compute.evidence.review` | 候选、对照记录、单位与实体规则 | 冲突类型、支持/反驳证据、待核验项 | 第二个 AI 的同意不是独立事实证明 |
|
||||
| `compute.gaps.prioritize` | 缺失、来源健康、参数敏感性、采集成本 | 下一步补证清单 | 优先可能影响结论且可查证的缺口,不以国家倾向排序 |
|
||||
| `compute.change.explain` | 已计算的贡献账单、事实版本 | 有引用的可读解释 | 不能修改计算结果;事实、估计、假设分别表达 |
|
||||
| `compute.model.propose` | 误差报告、数据质量问题、模型配置 | 有边界的参数/映射/新特征提案 | 必须说明可证伪假设、预期改善、影响范围和回退方案 |
|
||||
| `compute.model.review` | 候选方案、独立评估结果、影响报告 | 发布建议与限制 | 无权改变测试标签、门槛、保留集或自己批准自己 |
|
||||
|
||||
结构化结果由后端 Pydantic schema 校验。当前 AI 返回以文本为主,需要新增结果解析与受限重试;JSON 不合格、引用不匹配、单位冲突时运行失败或保留候选,不能将自然语言当作正式数据。
|
||||
|
||||
### 7.2 工具与权限
|
||||
|
||||
允许工具包括:参数化只读指标查询、已注册来源搜索/抓取、文档段落读取、规则化单位转换、实体候选查询、固定估计器运行、创建缺口任务、保存提案、提交实验任务。
|
||||
|
||||
AI 不直接写正式分数、覆盖原始证据、执行任意 SQL/Python/Shell、修改发布门槛或扩大自己的工具权限。生成的采集映射必须通过回放样本后才保存;新增可执行采集器代码属于普通开发和发布流程,不在生产 Agent 内 `eval`。
|
||||
|
||||
网页抓取需校验协议、主机、解析后的目标地址及每次重定向,阻止访问内网、回环和元数据端点;使用专门外部抓取边界,不能把内部业务 API 和外部取证 URL 混用。AI 只得到所需公开证据或经授权的业务摘要,凭证由后端设置解析,不进入提示词、引用或日志。
|
||||
|
||||
### 7.3 用户能看见的 AI 结果
|
||||
|
||||
每份分析展示三部分:已验证事实、模型解释、仍缺哪些证据。每条数值和关键结论可展开来源。用户可以标记“实体合并错误”“单位/年份错误”“证据不支持”“假设不合理”,形成结构化反馈;收藏、点赞和是否喜欢国家排名不能成为事实标签。
|
||||
|
||||
“AI 发现数据不足”可以是成功结果。模型不可用时,确定性采集、计算、双图和地球继续工作,解释面板保留上次结果并标注版本与时间。
|
||||
|
||||
## 8. 数学模型如何自我迭代
|
||||
|
||||
### 8.1 自动迭代分成三类
|
||||
|
||||
| 层次 | 可以怎样改进 | 自动程度 |
|
||||
| --- | --- | --- |
|
||||
| 证据和数据 | 补充来源、修正字段、发现重复和过期、改进解析映射 | 已核准来源与已验证规则可自动运行;新语义及高影响冲突先进入候选 |
|
||||
| 可验证的估计参数 | 项目延期分布、特定硬件/任务效率、记录错误率 | 通过冻结评估、影子运行和预先配置的边界后,可以自动发布小范围参数更新 |
|
||||
| 评价定义 | H/O/G/R 权重、指标含义、基期、国家范围和价值取向 | 默认生成提案,由模型负责人确认版本;不是机器从不存在的“国家实力真值”中自动学习 |
|
||||
|
||||
此处是产品建议的默认发布策略,不是在本次计划工作中申请执行权限。未来可配置自动发布的参数白名单,但不能用“开启自动迭代”笼统授权所有定义变化。
|
||||
|
||||
### 8.2 迭代闭环
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[新事实/人工纠错/成熟预测结果] --> B[确定性误差与质量评估]
|
||||
B --> C[AI 提出可证伪改进]
|
||||
C --> D[生成候选配置与锁定实验]
|
||||
D --> E[按时间回测及国家分组评估]
|
||||
E --> F{是否通过预设门槛}
|
||||
F -->|否| G[记录失败原因并保留现行版本]
|
||||
F -->|是| H[影子运行:结果不改变正式排名]
|
||||
H --> I[按发布策略评审或受限自动提升]
|
||||
I --> J[发布不可变版本与变动桥接]
|
||||
J --> K[监测退化并可原子回滚]
|
||||
K --> B
|
||||
```
|
||||
|
||||
每次提案只改变一种主要因素,或者明确作为一个联合实验;不能同时换来源、标签、权重和评估方法后宣称单个参数有效。失败提案也保存,避免只展示成功实验。
|
||||
|
||||
提案记录至少包括:问题、证据 ID、基线版本、拟议配置差异、适用实体/任务、可检验目标、锁定数据截止时间、主要评估指标、停止条件和回滚版本。可以提出“将并网阶段纳入交付预测”,不能只提出“让某国得分更合理”。
|
||||
|
||||
### 8.3 真实反馈标签从哪里来
|
||||
|
||||
| 估计对象 | 可观察标签 | 评估方式 |
|
||||
| --- | --- | --- |
|
||||
| 文档字段提取 | 人工核对原文的数值、单位、年份、实体和引用位置 | 字段精确率/召回率、关键错误、引用支持率,按语言/文档类型分组 |
|
||||
| 实体合并 | 官方 ID、项目连续记录或人工确认的同一/不同实体 | 错误合并率和遗漏合并率;高影响错误单列 |
|
||||
| 项目按期投产 | 后续正式验收/投产/取消记录,保留观测截止 | Brier score、可靠性图;右删失与失访不能计作失败 |
|
||||
| 新增容量 | 未来实际确认的设备与容量,具有同口径 | 绝对误差、对数误差;同时报告项目等权与容量加权结果 |
|
||||
| 任务性能系数 | 未参与拟合的可复现实测配置 | 相同任务质量下的预测误差与区间覆盖 |
|
||||
| 区间预测 | 后续实际值是否落入范围及区间宽度 | 覆盖率与区间评分一起评价,避免靠无限放宽范围“提高准确率” |
|
||||
|
||||
不使用 AI 自己生成的摘要作事实标签,不用现行总分拟合下一版总分,不把另一个复合指数直接当国家实力真值。完整创新指数可做外部一致性讨论,但若其包含输入指标或与 O 重叠,就不能作为独立验证集。
|
||||
|
||||
### 8.4 回测与数据泄漏控制
|
||||
|
||||
预测训练、验证和保留集按真实可获得时间滚动切分,同一项目的后续阶段、同一公告转载和同一扩建链必须归组,避免分散进训练和测试。每个特征都要求 `published_at <= forecast_cutoff`;仅今天抓到而无法证明当时可得的资料,不参加“当时可知”回测。
|
||||
|
||||
新模型先与简单基线比较:维持上次状态、沿用公告日期、同阶段历史交付中位数。不得只与一个刻意弱的旧模型比较。参数、标准化锚点和来源质量估计只从训练窗口取得。
|
||||
|
||||
大模型的训练记忆可能知道后来结果,所以历史回测中的预测器必须是只读取冻结特征的确定性估计器;不能让 LLM 凭记忆补旧事实。提案生成与真实保留集隔离,实际前瞻影子运行仍是必要证据。
|
||||
|
||||
按国家、硬件代际、项目规模、来源语言分别评估,并按项目/运营方/时间块进行重采样,不能把高度相关的重复记录当独立样本。小样本报告区间与不足,不为满足仪表盘而宣称显著改进。
|
||||
|
||||
### 8.5 建议发布门槛
|
||||
|
||||
以下是实施阶段的初始门槛草案,需在 M0 用试点数据冻结。它们是最低操作条件,不保证统计有效性,AI 无权自行降低。
|
||||
|
||||
| 门槛 | 初始建议 |
|
||||
| --- | --- |
|
||||
| 硬正确性 | 单位、时间穿越、重复计数、采集失败保留正式值等必测案例全部通过 |
|
||||
| 提取评估 | 至少 200 条人工核验字段、覆盖中英文及主要文档类型;报告精确率/召回率及区间;关键数值/单位错误不得进入自动采用集 |
|
||||
| 预测参数自动提升资格 | 至少 50 个有可验证结果的独立项目、覆盖至少 3 个滚动窗口;每个主要参与方至少 10 个,不足则继续影子运行或人工评估 |
|
||||
| 改善幅度 | 预先指定主要损失相对现行版和简单基线至少改善 5%;按依赖结构重采样的损失差区间须支持改善 |
|
||||
| 分组保护 | 主要参与方损失退化不超过预先约定范围,初始建议 2%;不能以平均改善掩盖一个国家明显恶化 |
|
||||
| 区间质量 | 不仅检查覆盖,还检查宽度与评分;对无法量化的来源覆盖偏差单列说明 |
|
||||
| 影子观察 | 至少连续 4 周稳定运行,并达到所需成熟标签数;12 个月预测不因观察满 4 周就算验证完成 |
|
||||
| 试验节制 | 初始每月最多 3 个主要候选,锁定真实保留集;多次搜索造成的选择偏差要通过新的前瞻样本再次检查 |
|
||||
|
||||
标签暂时不足时,系统可以自动补数、解释、积累预测与结果,但不发布“自我学习后的更准模型”。这不会阻塞第一版事实产品上线。
|
||||
|
||||
### 8.6 一个完整迭代例子
|
||||
|
||||
以下是流程示例,不是已发生的项目事实或评估成绩。
|
||||
|
||||
系统发现一批已订购芯片的项目未如期投产;确定性评估记录预测误差。AI 查询证据后提出:现有交付模型只使用芯片到货时间,遗漏电力接入阶段。它提交一个新增“并网是否完成”特征的候选。
|
||||
|
||||
后端按项目原始发表时间构造训练数据,用固定估计器拟合;锁定保留集检验总体和中美分组误差。即便论文或第二个 AI 都认为这个想法合理,只要结果未通过门槛就不发布。通过后先影子运行;正式发布时展示受影响的 G 分、预计容量和模型版本变化,已投产 H 不因推测而下降。
|
||||
|
||||
若后续发现并网字段提取错误导致退化,回滚活动模型指针,保存新证据和失败实验。不能删除失败记录或改写过去已发布的预测。
|
||||
|
||||
## 9. 后端数据与实现结构
|
||||
|
||||
首版采用最小领域实现:沿用 `CollectedData`、`DataSnapshot` 和位置维表,新增经过 schema 校验的国家指标记录与聚合服务。数值口径、参考期、发表时间、来源链接、原文哈希和证据定位可先作为受约束的元数据保存;必要的原件存放在许可允许的运行数据目录。只有现有表确实无法表达查询或历史约束时才增加专用表。
|
||||
|
||||
先冻结面向专题的 API 契约,地球与图表只消费该契约。将来拆出指标观测、证据、实验等表时,通过服务层迁移,不让前端依赖临时 JSON 或物理表结构。首版不以第 9.1 节全部表建完作为开工条件。
|
||||
|
||||
### 9.1 拟议持久化对象
|
||||
|
||||
下表为待新增的逻辑对象,可按最终数据库设计合并有限的小表;不要与现有采集、位置和任务状态形成并行真相源。
|
||||
|
||||
| 对象 | 保存内容 | 关联和约束 |
|
||||
| --- | --- | --- |
|
||||
| `evidence_documents` | 原文存储引用、原始/提取哈希、来源、发表/抓取时间、许可、提取版本 | 内容不可变;全文不在高频列表接口返回 |
|
||||
| `metric_definitions` | 粒度、单位、精度、工作负载、归属、方向、适用范围和版本 | 一个稳定 metric ID;模型版本引用具体定义版本 |
|
||||
| `metric_observations` | 数值/范围、参考期、知识时间、状态、来源版本、取整和缺失原因 | 链接原采集记录;追加式修订,不复制整份原始 JSON |
|
||||
| `observation_evidence` | 观测与证据的支持/反驳关系、页号、表格/段落、短摘录 | 多对多;来源家族避免转载重复投票 |
|
||||
| `compute_entities` / `resource_relations` | 稳定实体、别名、分期关系、所有权/供应/连接、有效期 | 复用现有位置维表;不另存一套会漂移的正式坐标 |
|
||||
| `model_versions` | 指标集合、权重、锚点、参数、代码版本、情景与发布状态 | 不可变版本;活动指针独立且原子切换 |
|
||||
| `model_runs` / `model_results` | 输入清单哈希、版本、截止时点、随机种子、环境、国家结果、分项和不确定性 | 结果可重放;实验、情景与正式运行分开 |
|
||||
| `model_proposals` / `model_evaluations` | 配置差异、假设、冻结评估方案、损失、分组结果、决定与回退目标 | 发布必须引用通过的评估,保存失败实验 |
|
||||
| `agent_runs` | task key、提示词有效哈希、模型、工具调用、引用、预算、结构化输出、终态 | 链接现有任务账本,Agent 自身不拥有另一套调度状态 |
|
||||
|
||||
重放保留输入观测 ID/版本清单或不可变清单文件,不能只保存一段 SQL 然后查询已被更改的数据。记录计算实现版本、依赖环境和随机种子;LLM 输出保存为证据,不要求重新调用模型才能复算数值。
|
||||
|
||||
历史删除与保留策略需要显式区分:普通缓存可清理;被正式模型版本引用的观测及许可允许保存的证据受保留保护。来源撤回或许可变更时标记可访问状态并保留允许保留的哈希和元数据,不强行公开原文。
|
||||
|
||||
### 9.2 待新增模块与现有边界
|
||||
|
||||
| 拟议模块 | 职责 |
|
||||
| --- | --- |
|
||||
| `backend/app/services/compute_intelligence/observations.py` | 规范化、有效观测选择和数据缺口 |
|
||||
| `.../metrics.py`、`.../scoring.py` | 纯数值计算、单位/指标注册、标准化和总分 |
|
||||
| `.../forecasting.py`、`.../evaluation.py` | 固定估计器、时间切分、基线、分组指标与实验 |
|
||||
| `.../workflows.py`、`.../proposals.py` | 固定 AI 工作流、结构化候选、预算和发布规则 |
|
||||
| `.../publication.py` | 正式快照、变动桥接、活动版本和 outbox |
|
||||
| `backend/app/api/v1/compute_intelligence.py` | 参数化只读查询和后台任务入口 |
|
||||
| `frontend/public/earth/js/compute-intelligence.js` | 专题 UI 状态、图表与现有图层联动,消费后端计算结果 |
|
||||
| 控制台专题页与现有采集/AI 设置页签 | 证据核验、缺口、模型实验、版本回看 |
|
||||
|
||||
具体文件在实现时保持单一职责,优先复用已有服务,不为目录结构创建空文件。新来源继续注册到现有采集器体系及 datasource 配置;新提示词放入现有 `default_prompts.json`,不散落在 renderer 或 `aiprovider`。
|
||||
|
||||
### 9.3 拟议 API
|
||||
|
||||
下列接口尚不存在,名称在 M0 与当前路由约定核对后冻结。
|
||||
|
||||
| 方法与路径(统一前缀 `/api/v1/compute-intelligence`) | 返回内容 |
|
||||
| --- | --- |
|
||||
| `GET /overview` | 所选参与方、口径、参考期与知识截止下的结果、适用范围及覆盖说明 |
|
||||
| `GET /series` | 指定指标的有界历史序列,含缺失、版本、实际观测时间与来源引用 |
|
||||
| `GET /entities/{id}` | 设施、分期、资源关系和历史观测 |
|
||||
| `GET /changes` | 数据或模型贡献账单,数据库侧分页 |
|
||||
| `GET /evidence/{id}` | 根据权限和许可返回证据摘要、定位与允许展示的内容 |
|
||||
| `GET /models`、`GET /runs/{id}` | 模型版本、输入清单、计算结果与评估摘要 |
|
||||
| `POST /recompute`、`POST /agents/run` | 建立任务并立即返回任务 ID,允许取消和查看进度 |
|
||||
| `POST /proposals/{id}/evaluate` | 创建冻结数据与方案的评估任务 |
|
||||
| `POST /models/{id}/promote`、`POST /models/{id}/rollback` | 按角色及发布策略原子切换活动版本,记录理由和影响 |
|
||||
|
||||
查询参数固定包含 `metric_profile`、`entities`、`reference_at`、`knowledge_cutoff`、`model_version` 和可选 `scenario_id`。前端不接收任意表达式或 SQL,不自行补算另一套总分。
|
||||
|
||||
### 9.4 发布与性能
|
||||
|
||||
后台完整计算并通过质量门槛后,在事务中发布结果快照和活动指针,通过既有 outbox 唤醒刷新。新 adapter 显式声明国家面板/历史图的刷新范围;设施变化才触发相应 `computeCenters` 更新。草稿证据、失败实验和影子运行不改变默认地球。
|
||||
|
||||
缓存键包含口径、实体集合、参考期、知识截止、数据清单哈希、模型与情景版本。WebSocket 只通知结果版本与必要增量,客户端拒绝较旧响应;不能每次国家 hover 都重算模型或执行 LLM。
|
||||
|
||||
聚合、过滤、排序和分页在数据库完成;预计算国家年度/月度可用序列,地图按分辨率聚合节点。时间滑动去抖并复用已取快照,播放使用相邻状态差量。建议试点验收目标为缓存查询 P95 小于 1 秒、切换已缓存时点小于 500 毫秒,声明基准机器、数据规模和网络条件;未实测前不作为已达到性能。
|
||||
|
||||
## 10. 刷新、成本和失败处理
|
||||
|
||||
| 工作 | 建议节奏 | 触发条件与成本控制 |
|
||||
| --- | --- | --- |
|
||||
| CSV/API 检查 | 日或来源允许的更慢频率 | ETag/Last-Modified/哈希去重,无内容变化不调用 AI |
|
||||
| 年报与指数发现 | 月度,发布季可提高 | 找到新版本再下载和提取,不把月检写成月度指标 |
|
||||
| 重点项目证据 | 日查、周复核 | 去重后仅高影响变化进入提取队列 |
|
||||
| 指标重算 | 有效观测或模型版本改变时 | 只重算受影响实体/时点/指标依赖 |
|
||||
| AI 解释 | 有意义的结果变化或用户请求 | 相同输入哈希复用,不因页面打开重复生成 |
|
||||
| 数据缺口排序 | 每周 | 敏感性、覆盖风险、可获得性和预计采集成本共同决定 |
|
||||
| 参数候选实验 | 月度或成熟标签达到门槛 | 限制候选数量;无足够标签不强制产出升级 |
|
||||
|
||||
试点默认每工作流最多 3 次搜索、8 个页面抓取、4 次 LLM 调用、1 次结构化修复重试,最多同时运行 2 个 AI 工作流;均为可配置预算,不是第三方接口能力承诺。按全局模型配置记录 token/费用,设置日预算和单任务截止时间,超额进入延后或停止状态,不无限自递归。
|
||||
|
||||
失败处理:采集失败保留上次正式值和新鲜度;AI 不可用仍提供确定性结果;引用失效保留许可允许的原件和哈希;长期资料缺口显示缺失并降低结论覆盖范围;新源冲突进入待核验;模型退化恢复前一活动版本。不能用重试或回滚删除已提交有效证据。
|
||||
|
||||
## 11. 分期实施与依赖
|
||||
|
||||
工作量是规划估算,按一名熟悉项目的全栈开发者、持续可用的数据核验支持和现有环境可用计算;不是交付承诺。开发时间不包括等待付费授权、原始报告或 12 个月预测结果成熟的时间。
|
||||
|
||||
### 11.1 当前优先交付:展示专题
|
||||
|
||||
| 顺序 | 修改方向 | 验收内容 | 估计开发工作日 |
|
||||
| --- | --- | --- | ---: |
|
||||
| A1 修复数据与最小合同 | Epoch CSV、TOP500 日期/身份/单位、空结果保护;保存历史指标和来源 | 能得到真实当前快照与核验后的年度序列,不要求宏观指标全到最新年 | 2—3 |
|
||||
| A2 国家对比与历史接口 | 国家汇总、公开样本与宏观统计分离、双图数据、证据摘要 | 绝对量/份额/创新分差有各自定义;缺失与时点可见 | 2—3 |
|
||||
| A3 地球与图表展示 | 专题模式、国家面板、双图、时间联动、设施/证据详情 | 完成第 0 节完整浏览路径,并通过桌面/移动/缩放验证 | 3—5 |
|
||||
| A4 可选 AI 阅读辅助 | 复用现有 client 和任务提示词,针对冻结快照生成带引用说明 | 事实与推断分开;可关闭;失败不影响地球与图表 | 1—2 |
|
||||
|
||||
A1—A3 是当前必须交付的闭环,约 7—11 个开发工作日;A4 可在闭环稳定后增加。实际数据源不通、现有采集回归问题或主题布局复杂度会改变估算。实现时首先验证 A1,而不是先开发通用 Agent 框架或完整综合评分。
|
||||
|
||||
### 11.2 后续平台能力路线
|
||||
|
||||
下表是完整能力建设的工作包估算,与 A 路线存在复用和重叠,不应重复相加。A 路线完成后先盘点已具备的 M0—M2 能力,只实施剩余部分。
|
||||
|
||||
| 阶段 | 工作包 | 交付物与通过条件 | 估计开发工作日 |
|
||||
| --- | --- | --- | ---: |
|
||||
| M0 合同与试点 | 固定口径、成员、来源许可、历史时间语义、角色权限、质量/评估门槛 | 指标注册表、数据源接入清单、黄金核验样本;所有未知明确登记 | 3—5 |
|
||||
| M1 可信数据 | 修复 Epoch/TOP500;证据与指标观测;导入两组历史;稳定身份与失败保护 | 可查询、可追溯、可重放的历史 API;无示例数据进入正式结果 | 7—10 |
|
||||
| M2 可见产品 | 地球专题、双图、国家对比、变化账单、证据抽屉和控制台缺口页 | 中美历史回放闭环;移动/缩放和旧图层回归通过;综合分可保持未就绪 | 7—10 |
|
||||
| M3 资源与潜力 | 芯片、实测、项目、电力/资金台账,场景与分项估计 | H 与可用 G/R 子项具备来源;O 仅在分项齐备后发布;总分门槛满足才开放 | 8—12 |
|
||||
| M4 AI 工作流 | 固定提取/核验/补证/解释流程、反馈和预算;扩展可靠任务类型 | AI 只能产生有引用候选与解释,失败不会污染正式数据 | 7—10 |
|
||||
| M5 模型迭代 | 真实标签、时间回测、候选评估、影子运行、发布桥接与回滚 | 能证明某个可观察估计目标改善;不要求每轮必有新模型 | 8—12 |
|
||||
|
||||
从零完整建设上述广义平台能力约 40—59 个开发工作日,包含比 A 路线更完整的证据和实验治理;它不是当前展示专题的工期。影子期至少四周,但不成熟的预测标签会延长自动提升资格等待期。
|
||||
|
||||
依赖顺序:M0 → M1 → M2;M3 的项目台账应尽早开始积累标签;M4 依赖 M1 的可信证据结构;M5 依赖 M3 的标签和 M4 的审计工作流。多方、矿产和复杂供应链在 M2 之后按数据质量逐个扩展,不等全部主题齐备再上线。
|
||||
|
||||
## 12. 验收与测试矩阵
|
||||
|
||||
| 范围 | 必测案例 | 通过标准 |
|
||||
| --- | --- | --- |
|
||||
| 来源真实性 | HTML 改版、空 CSV、404、部分下载、示例字符串 | 正式数据不被示例或空结果替换;错误原因可见 |
|
||||
| 单位与精度 | T/P/E 换算,FP64/FP16/FP8,稀疏/稠密,MW/TWh,FLOP/FLOP/s | 不同口径拒绝合并;可比转换有可复算结果 |
|
||||
| 时间与版本 | 固定抓取日误作投产日、2026 报告回填 2023、未知发布日期 | 严格截止查询无未来信息;历史回顾明确重述 |
|
||||
| 实体与去重 | 排名换位、扩建、名称变化、匿名化、重复转载、跨源同集群 | 身份稳定;不重复计数;匿名或缺榜不自动退役 |
|
||||
| 统计与地理 | 缺坐标、未知国家、欧盟与成员国、跨境所有权 | 国家统计不依赖坐标;分母不重叠;归属视图不混用 |
|
||||
| 两张历史图 | 已核验年份值、缺失年份、2026 待核实主张 | 只展示有证据的观测;不能伪造平直历史或默认预测 |
|
||||
| 综合评分 | 缺 O/G/R、权重和不为 1、标准化越界、国家集合变化 | 非法配置拒绝;缺失总分为 null;固定版本不静默漂移 |
|
||||
| 变化账单 | 新投产、补录、源修订、换模型 | 分项贡献之和满足计算容差;类型和版本差异可解释 |
|
||||
| AI 输出 | 格式错误、虚构引用、原文含指令、模型超时、重复任务 | 输出隔离;工具边界有效;任务幂等并能取消 |
|
||||
| 模型评估 | 标签穿越、同项目跨集、过拟合小样本、分组退化 | 不通过门槛不发布;保留失败实验 |
|
||||
| 发布与回滚 | 两个并发发布、计算中断、旧 WS/HTTP 响应 | 原子版本;半成品不可见;客户端不回滚到旧时点 |
|
||||
| 地球渲染 | 远距/近距、时间播放、层隐藏、节点锁定、场景退出 | 无闪烁和悬空 tooltip;实例复用,不每帧重新建纹理 |
|
||||
| UI | 桌面、移动、125%/150% 缩放、键盘、无数据/低置信/加载状态 | 一屏工作区无意外全局滚动;关键标签不靠颜色区分 |
|
||||
|
||||
实现阶段按变更执行后端精确测试、`scripts/harness/quick-check.sh`、Bun 前端构建和渲染 smoke;地球改变不能只凭构建通过。保持现有公共页面、鉴权、管理路由、安全导航及其他地球图层的回归证据。性能阈值用记录过的基准规模验证,不能把开发空数据测试称为生产验收。
|
||||
|
||||
## 13. 文档、发布与完成定义
|
||||
|
||||
每阶段 PR 和部署使用已有项目流程,功能旗标分别控制专题 UI、估计分项、AI 工作流与自动参数提升。先开放读取与事实浏览,再启用模型实验;关闭新旗标应回到原有算力图层而不丢历史数据。
|
||||
|
||||
发生真实用户工作流变化时更新中英文手册与快速入门;采集、AI、作业与位置变化更新相应后端技术文档;控制台、Earth、图层顺序和样式变化更新对应 context 与规范。新增公共技术文档需要同步中英文和 Docs 注册;本文件是内部中文计划,不注册为已实现的公共手册。
|
||||
|
||||
当前展示专题完成定义:用户完成第 0 节浏览路径,可以从国家指标变化回到出处和所用计算版本;重新计算得到同一数值;知道哪些资料尚缺。两张图应和国家、设施、时间、证据联动,不能只是贴在地球旁的静态图片。
|
||||
|
||||
后续决策辅助与模型迭代完成定义:情景的假设与结果可复算;AI 的一次成功改进有独立验证目标、评估记录和可回滚版本。只有一段 AI 分析或未经校验的总分,不代表完成决策辅助能力。
|
||||
|
||||
## 14. 风险与预先决定的退路
|
||||
|
||||
| 风险 | 产品和实施处理 |
|
||||
| --- | --- |
|
||||
| 无法取得创新原报告分项 | 保留完整指数独立观测;O 和总分不发布,不伪造拆分 |
|
||||
| 中国或其他地区公开设施更少 | 显示宏观统计与设施样本两个覆盖视图,扩充本地语言原始来源,不把保密或匿名化当能力下降 |
|
||||
| 硬件真实利用率不可知 | 理论量先行,明确任务和区间;不把实验室 benchmark 当生产实测 |
|
||||
| 来源改版、匿名化或许可证变化 | 版本化下载、质量隔离、许可范围内保存证据,必要时停更单一来源 |
|
||||
| 资金、电力、芯片相互重复计分 | 固定指标归属与约束链;不把投入、存量、产出和事件各加一次 |
|
||||
| 训练标签不足或结果多年后才成熟 | 先提供情景与证据改进,延后参数自动提升资格 |
|
||||
| 模型迎合既定国家排名 | 预先定义目标和门槛,评价权重单独评审,不以用户喜好或另一个 AI 打分训练 |
|
||||
| 已有采集/定位状态被新服务覆盖 | 复用当前真相源,使用明确字段所有者、发布快照和回归测试 |
|
||||
|
||||
## 15. 来源与相关文档
|
||||
|
||||
以下链接支持数据口径、来源可得性和架构选择;除明确标注的历史值外,本计划的字段、频率、权重、工期和发布门槛均是拟议设计。
|
||||
|
||||
- [信通院 2021 年版算力白皮书(镜像)](https://pdf.dfcfw.com/pdf/H3_AP202109271518811781_1.pdf):2020 年总算力份额。
|
||||
- [信通院 2022 年版白皮书(镜像)](https://13115299.s21i.faiusr.com/61/1/ABUIABA9GAAgvq2fmwYozLrrpwU.pdf):正文第 13—14 页,2021 年份额与口径。
|
||||
- [信通院 2023 年版白皮书(镜像)](https://www.ahchanye.com/wp-content/uploads/2023/09/2023092815342684.pdf):正文第 13 页的 2022 年份额,以及 302 EFlops 总规模。
|
||||
- [信通院 2024 年版蓝皮书(镜像)](https://pdf.dfcfw.com/pdf/H3_AP202502051642799257_1.pdf):正文第 10 页的 2023 年份额及后续规模说明。
|
||||
- [科技日报关于创新指数报告的报道(新浪转载)](https://finance.sina.com.cn/tech/roll/2026-07-17/doc-iniickaz8224252.shtml):分差与综合指标维度;未替代原报告分项。
|
||||
- [2026 年 42%/33% 候选出处](https://www.mornai.cn/news/gpu/ai-computing-power-never-sleeps/):待核实商业文章,不作为正式统计。
|
||||
- [Epoch GPU 集群公开数据](https://epoch.ai/data/gpu-clusters)、[字段与下载说明](https://epoch.ai/data/gpu-clusters-documentation):CSV、地理字段、覆盖与公开版本限制。
|
||||
- [Epoch 芯片所有权](https://epoch.ai/data/ai-chip-owners):所有权不等于使用权或所在地,交付与投产有时间差。
|
||||
- [MLPerf Training](https://mlcommons.org/benchmarks/training/)、[MLPerf Inference Datacenter](https://mlcommons.org/benchmarks/inference-datacenter/):固定任务、质量及测试配置。
|
||||
- [IEA Energy and AI](https://www.iea.org/reports/energy-and-ai)、[EIA 开放数据](https://www.eia.gov/opendata/):能源资料入口,不能直接替代设施并网证据。
|
||||
- [SEC EDGAR API 说明](https://www.sec.gov/search-filings/edgar-application-programming-interfaces):公司提交与财务数据访问;需注意财季与标签口径。
|
||||
- [USGS 矿产概要](https://www.usgs.gov/centers/national-minerals-information-center/mineral-commodity-summaries)、[UN Comtrade 门户](https://comtradeplus.un.org/):矿产与贸易候选数据源。
|
||||
- [OpenAlex 帮助与数据说明](https://help.openalex.org/):研究元数据候选来源;接入时核验访问与归属规则。
|
||||
- [业务架构与数据流转](../technical/zh/platform-data-flows.md)、[后端采集器](../technical/zh/backend-collectors.md)、[AI Provider 边界](../technical/zh/agents-aiprovider.md)。
|
||||
- [智能星球前端上下文](../technical/zh/earth-frontend-context.md)、[渲染图层顺序](../technical/zh/earth-render-layer-order.md)、[图层视觉规范](../technical/zh/earth-layer-style-reference.md)。
|
||||
- [轻量 Agent 编排](agents-light-orchestrator-websearch-plan.md)、[态势感知基础计划](agents-situational-awareness-foundation-plan.md):复用总体方向,不重复建设 provider 内业务编排。
|
||||
154
docs/plans/development-agent-model-allocation-plan.md
Normal file
154
docs/plans/development-agent-model-allocation-plan.md
Normal file
@@ -0,0 +1,154 @@
|
||||
# Planet 开发 Agent 模型分配与 Token 控制计划
|
||||
|
||||
日期:2026-09-21。状态:已用于本轮地理与空间卡片开发;未改变全局或项目默认模型配置。适用于已授权的开发任务,不代替用户对任务范围的决定。
|
||||
|
||||
目标:在固定政治展示规则、现有 UI 逻辑和性能门槛下,降低完成一个合格任务的总成本与返工次数。不是让所有工作都使用低价模型,也不是让最强模型从头包办。
|
||||
|
||||
当前重点任务见 [Earth 地理资源与空间卡片计划](earth-spatial-presentation-and-geographic-assets-plan.md)。该计划的功能与验收要求优先;本文件只决定如何组织开发。
|
||||
|
||||
## 1. 模型角色
|
||||
|
||||
模型定位参考 OpenAI Docs;下表的具体分工是针对 Planet 的工程建议,需要结合实际一次通过率和返工记录调整,不是模型能力保证。
|
||||
|
||||
| 模型 | 精确模型 ID | 默认推理强度 | 分配的工作 | 不作为默认用途 |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| Astra | `gpt-6-astra` | `high` | 跨系统难题、空间展示方案、复杂坐标/深度/性能诊断、关键设计与独立审查 | 日常文件搜索、机械编辑、反复运行已确定命令 |
|
||||
| Sol | `gpt-5.6-sol` | `medium`;复杂实现用 `high` | 执行主控、核心代码、共享组件整合、Three.js 状态与生命周期、最终集成 | 逐条整理大批重复记录、重复复述整个项目背景 |
|
||||
| Terra | `gpt-5.6-terra` | `medium` | 接口已明确的常规实现、局部 UI 修复、数据处理脚本、测试编写、浏览器回归 | 独自决定空间交互方向、坐标模型或跨模块架构 |
|
||||
| Luna | `gpt-5.6-luna` | `low`;明确的小修改可用 `medium` | 定向检索、清单比对、日志归类、说明文档初稿、固定格式整理 | 核心渲染、数据迁移、几何归属修改、最终审美或性能放行 |
|
||||
|
||||
日常推荐组合是 Sol 主控加一个 Terra worker;Luna 和 Astra 按需进入,不要求每个阶段凑齐四种模型。已经完成的资源准备不因采用这份方案重新做一遍。
|
||||
|
||||
当前会话的主模型不会因为写入计划自动变成 Sol。真正执行时,主会话模型由实际任务设置决定,子 Agent 在派发时明确指定模型和推理强度,并核对返回的运行配置;不能把角色名称当作已经切换模型的证据。
|
||||
|
||||
## 2. 本项目任务分配表
|
||||
|
||||
| 工作包 | 主执行 | 独立验证/审查 | 交付和升级条件 |
|
||||
| --- | --- | --- | --- |
|
||||
| 现有窗口、连线、巡航、滚动条入口盘点 | Terra | Sol 检查引用与遗漏 | 文件/符号清单、现有行为证据;Luna 可辅助定向搜索,无需读取全仓库 |
|
||||
| 贴图归档、尺寸、哈希、配准元数据检查 | 直接运行脚本;脚本需新增时 Terra | Sol 抽查实际输出 | S0 已完成,后续只做有变化的核对,不委托模型重复计算哈希 |
|
||||
| 固定政治规则的技术合同与几何覆盖边界 | Sol | Astra 审查规则是否贯穿各数据路径 | 权属口径已由项目确定,任何模型均不得重新决定;标识不明确先核对,不能猜测几何 |
|
||||
| 贴图、海岸线、拾取错位定位 | Sol `high` | Astra 在根因不清或涉及多个坐标层时介入 | 用明确经纬点复现,区分错误数据、回退、投影、UV 与半像素偏移 |
|
||||
| 规则已明确的映射、来源清单、局部转换脚本 | Terra | Sol;关键几何结果由 Astra 抽查 | 只落实批准合同;发现新的几何语义不得自行扩大规则 |
|
||||
| 多区域空间卡片的第一版运动设计 | Astra `high` | Sol 检查能否复用当前实现 | 明确锚点、投影、避让、换侧、背面及阅读稳定性;产出小合同,不产出整套 UI 大改 |
|
||||
| 2 张卡片的连接、弹性移动、生命周期原型 | Sol `high` | Terra 实测;Astra 看关键动效证据 | 优先实现最小完整循环;使用参考图和实际运动证据,不用文字自评代替 |
|
||||
| 扩至 4/6 卡、增删区域和事件整合 | Sol | Terra 跑交互/竞态测试 | 复用原型合同;`main.js` 等共享入口只由指定 owner 写入 |
|
||||
| 共享窗口行为、标准字体、移动抽屉及深色滚动条 | Terra | Sol 看渲染与旧面板回归 | 修改公共滚动所有者,验证图层等所有相关页;涉及共享状态重构时升级给 Sol |
|
||||
| 地球柔光、材质层次、节点与背景关系 | Sol | Astra 对照 UE/PPT/海报审查 | 第一轮只改少量确定参数并给实际截图;性能退化交 Sol 诊断,难题交 Astra |
|
||||
| 卡片内指标、普通图表、加载/缺失/错误状态 | Terra | Sol | 数据合同先冻结;不复活固定全屏大面板,不编造缺失年份 |
|
||||
| 国家/区域接口和前后端联动 | Terra;跨模块变更由 Sol | Sol | 不硬编码中美为唯一对象;范围筛选、比较对象和组合去重必须一致 |
|
||||
| 性能基准工具和回归测试 | Terra | Sol 检查方法;Astra 审查疑难退化 | 固定设备、数据与相机操作;日志含帧时间和内存证据,不能只报平均 FPS |
|
||||
| 单位、日期、缓存、旧请求覆盖等复杂数据故障 | Sol | Astra 仅处理跨系统未解问题 | 先给最小复现和失败测试,避免多个模型同时试不同修复 |
|
||||
| 中英文手册、资源清单、计划状态同步 | Luna 初稿;技术性较强时 Terra | Sol 对照实际 diff | 已实现与待实施分开;Luna 不能自行宣告功能完成 |
|
||||
| 最终集成与验收 | Sol | Astra 对关键风险做一次集中审查 | 相关测试、渲染、政治规则及性能证据齐全;普通修订不逐项再次走 Astra |
|
||||
|
||||
## 3. 顺序与并行安排
|
||||
|
||||
### 第一轮:证据与边界
|
||||
|
||||
Sol 读取现有计划、当前工作区差异和必要规则,整理短任务清单。Terra 盘点实际运行的数据提供者、组件和性能基线;独立且批量的文档/引用整理才交 Luna。先读已有检查结果,避免重新下载贴图、重新研究 World Monitor 或重复跑已经通过的检查。
|
||||
|
||||
Sol 将确定的组件接口、规则约束、实际截图与未解问题整理成一个短包,Astra 只审查高返工风险的空间方案和坐标问题。Astra 返回决定、理由和验证点,不再从零扫描整个仓库。
|
||||
|
||||
### 第二轮:最小空间原型与独立 UI 修复
|
||||
|
||||
Sol 实现 2 张卡片的锚点、连线和有阻尼移动;Terra 可以同时处理公共滚动条或写回归测试,前提是文件所有权不冲突。先完成最小可演示运动,再扩展内容和卡片数量。若两者都要改 `controls.js`、`hud.css` 等共享文件,则按顺序交接,不并行写入。
|
||||
|
||||
Astra 对原型的实际截图/录屏做一次聚焦审查,检查是否符合“从地球四周延伸”的方向。空间原型尚未确认时,不展开完整指标卡片和全量国家 UI,以免重演方向错误后大规模返工。
|
||||
|
||||
### 第三轮:内容、全球扩展与性能
|
||||
|
||||
合同稳定后,Terra 接入指标与状态,Sol 负责与渲染/生命周期的整合。性能测试由脚本执行,Terra 整理差异;可复现的退化先交 Sol。只有不能局部解释的 GPU/CPU、坐标、异步问题才交 Astra。
|
||||
|
||||
### 第四轮:一次集成验收
|
||||
|
||||
Sol 整合并运行所需检查;Luna 根据最终 diff 更新文档初稿。Astra 集中审查关键几何、政治规则完整性、卡片运动与性能结论。已通过范围不反复重测,除非新修改影响该范围。
|
||||
|
||||
## 4. 并发上限与文件所有权
|
||||
|
||||
当前环境总并发槽位为 4,含主 Agent;执行时仍以实际可用额度为准。默认使用主 Agent 加 1 个 worker。只有存在两项以上真正独立的工作,且主 Agent 同时有有用工作可做时,才扩大到主 Agent 加 2 个 worker;第 4 个槽位只给临时审查或独立批处理。
|
||||
|
||||
每项任务指定可写文件范围。`main.js`、`controls.js`、共享 HUD/InfoCard 与跨层状态接口同一时间只有一个写入 owner。其他 Agent 只读检查或提交补丁建议,由 owner 集成。不得并行启动多个浏览器重测相同场景,也不得共享有冲突的登录/本地存储测试状态。
|
||||
|
||||
浏览器验证优先使用当前可用的 Browser 控制能力,包括统一 CUA 入口。不能仅因技能目录中缺少单独的 Browser 条目就判定工具不存在;应继续检查已暴露工具并多次重试连接。只有多次确认 Browser 不可用后,才考虑独立 Playwright,并说明具体阻塞。
|
||||
|
||||
Agent 完成有边界的任务后停止。只有同一任务需要补充验证时才续用;不能让 worker 在空闲时自动寻找新功能或自行扩大范围。创建用户侧新任务不作为内部拆工方式。
|
||||
|
||||
## 5. 节省 Token 的具体做法
|
||||
|
||||
1. 派发窄任务,不默认继承整个历史对话。使用简短任务卡加源文件路径;必要历史按最小范围提供。短上下文不能省略政治规则、性能要求和相关视觉证据。
|
||||
2. 主控保存一份简洁任务账本:当前合同、文件 owner、已完成证据、失败原因、下一步。各 Agent 引用同一事实,不各写一份长计划。
|
||||
3. 先 `rg` 定位,再读函数和局部 diff;不重复读取巨大 JSON、整仓日志或全部历史文档。大数据留在文件里,由脚本给出数量、哈希、差异和异常样例。
|
||||
4. 图片、截图和录屏用实际文件查看。不能为了省 token 把视觉任务变成只读文件名或文字概括。
|
||||
5. 相同来源只研究一次,保存来源、日期和关键结论;只有数据变化、结论不足或任务扩大时重查。
|
||||
6. 子任务首次任务卡建议控制在约 500–1500 token,结果摘要约 300–800 token;这是上下文组织目标,不是截断必要证据的硬上限。
|
||||
7. 返回“改了什么、哪些文件、验证证据、剩余阻塞”即可。完整日志和截图放到文件,用路径引用;失败时返回相关片段,不整段倒入主控上下文。
|
||||
8. 哈希、解压、下载、构建、测试和统计交给确定性工具。不要为了运行一条已知命令另开一个 Agent。
|
||||
9. 不重复安排多个模型完整实现同一功能。关键风险做一次独立审查,普通修改由测试与 owner 检查。
|
||||
10. 默认不开 `max` / `ultra` 推理。提高模型或推理强度只针对具体难题,并记录升级理由。
|
||||
|
||||
## 6. 升级与停止条件
|
||||
|
||||
| 情况 | 处理 |
|
||||
| --- | --- |
|
||||
| Luna 遇到不明确需求、需要理解跨文件状态或开始猜测行为 | 交 Terra;若涉及核心状态/几何,直接交 Sol |
|
||||
| Terra 的任务超出明确合同,或同一实质失败在一次有针对性的修复后仍未解决 | 给 Sol 最小复现、尝试记录和失败证据,不继续盲试 |
|
||||
| Sol 遇到跨坐标层、性能来源不明或与现有交互语义冲突的问题 | 给 Astra 聚焦诊断任务;可以由 Astra 完成必要关键补丁,不限于口头审查 |
|
||||
| 已知一开始就是复杂的空间布局或高风险几何设计 | 直接用 Sol/Astra,不为了经过低模型梯级而增加一次无效尝试 |
|
||||
| 网络、凭证、工具缺失、原始资料不存在 | 不通过升模型解决环境问题;检查环境或记录缺口,继续独立工作 |
|
||||
| 失败来自缺失用户决定或无法识别的地名/视觉目标 | 提出具体问题,不让多个模型猜不同版本 |
|
||||
|
||||
默认允许 worker 一次有证据的修复;是否继续由主控根据新信息决定。超时或 token 消耗增加但没有新增证据时停止扩展,整理已知结果后重新分配,不能无限自递归。
|
||||
|
||||
## 7. 审查门槛
|
||||
|
||||
Astra 重点进入三个位置:最初高风险空间/几何合同、首次运动原型、最终关键风险验收;中间疑难问题按需增加。它不需要审阅每一处 CSS、每一段翻译或每个测试日志。
|
||||
|
||||
Sol 对各 worker 的交付负责。不能以“另一模型说通过”代替运行结果。任何模型都无权改变项目政治口径、降低性能门槛、把未知数据填成确定值,或自行部署和发版。
|
||||
|
||||
涉及政治几何和性能结论,使用与实现过程分离的检查证据;普通文档格式或固定映射使用针对性检查即可。视觉方向仍由已确认的用户要求和参考图决定,Astra 的审美判断不替代用户目标。
|
||||
|
||||
## 8. 单任务派发模板
|
||||
|
||||
```text
|
||||
任务:一个可独立验收的目标
|
||||
模型与推理:明确 model ID + reasoning effort
|
||||
必要约束:相关 rules 模块、政治口径、性能、共享组件、视觉参考
|
||||
输入:最少文件/符号、当前合同、已有验证结果和图片路径
|
||||
可写范围:明确文件;共享文件由指定 owner 修改
|
||||
验收:必须出现的行为、必要测试/渲染/性能证据
|
||||
停止/升级:哪些条件不能继续猜测或扩展
|
||||
返回:结果摘要、改动文件、检查结果、证据路径、遗留问题
|
||||
```
|
||||
|
||||
派发有意使用较小上下文时,必须把不可省略的约束写进任务卡。执行任务的 Agent 自行按规则加载相应模块,不因主控已读过就假定子 Agent 也知道。
|
||||
|
||||
## 9. 如何验证分配确实更高效
|
||||
|
||||
每个工作包记录模型、推理强度、输入/输出 token(环境可提供时)、耗时、工具耗时、修复轮数、一次验收是否通过和最终证据。若读不到准确 token,就标记未知,不用消息长度伪造 token 统计。
|
||||
|
||||
比较每个已验收任务的总消耗,包含主控、worker、审查和返工,而不是仅比较某次调用。先完成 3–5 个有代表性的工作包,再调整默认路由;这个数量是试点取样建议,不代表统计显著性或保证节省比例。
|
||||
|
||||
API 单价、订阅额度和任务级成本不是同一个指标。本计划不预先承诺固定节省百分比;模型可用性和实际费用以执行时配置为准。低成本模型导致反复修复时应提前升级;高能力模型能明显减少整体返工时可以直接采用。
|
||||
|
||||
## 10. 官方依据与执行边界
|
||||
|
||||
OpenAI Docs 将 Astra 定位为最具能力、适合困难端到端任务的模型,Sol 面向复杂专业工作,Terra 平衡能力与成本,Luna 面向成本敏感任务;它们并未规定 Planet 各文件应由哪个模型处理。本计划按任务风险给出项目级分工。
|
||||
|
||||
- [OpenAI 模型说明](https://developers.openai.com/api/docs/models)
|
||||
- [Codex 模型选择](https://learn.chatgpt.com/docs/models)
|
||||
- [子 Agent 配置与模型选择](https://learn.chatgpt.com/docs/agent-configuration/subagents)
|
||||
|
||||
只在用户已授权相应开发与委派、且运行环境支持时执行这套拆工。仅制定本计划不会切换当前会话模型、修改默认配置或自动启动 Agent。执行时若某个精确模型不可用,先说明并选择可用替代,不把替代模型冒充原分配。
|
||||
|
||||
## 11. 本轮实际执行账本
|
||||
|
||||
| 工作包 | 实际分配 | 结果与升级 |
|
||||
| --- | --- | --- |
|
||||
| 地理政策、源证据、几何构建与回退 | Sol `high`,`geography_policy` | 版本化规则与校验已实现;如实保留未核验几何,未用模型猜测填补 |
|
||||
| 空间连接线、卡片生命周期与主入口 | Sol `high`,`spatial_callouts` | 完成原型;六卡避让多次渲染仍有冲突后,交主控接手约束求解,worker 停止同文件写入 |
|
||||
| 同机性能基线、对照工具、公共滚动条 | Terra `medium`,`performance_baseline` | 保存桌面/模拟手机各 5 次基线;共享滚动容器修复和对照工具已完成 |
|
||||
| 复杂避让、全球目录/API、视觉参数、集成验证 | 主会话 | 主控整合并审查;性能脚本的场景同步多次失败后也由主控接手修复。独占布局文件,浏览器/构建按顺序执行,避免测试相互干扰 |
|
||||
| Luna 辅助 | 未启用 | 没有值得额外传递上下文的独立机械任务,不为凑齐模型而派工 |
|
||||
|
||||
本轮精确 token 与费用不可得,记为未知;不报告未经测量的节省比例。空间避让存在返工,不能把功能实现或子任务数等同于效率提升。性能问题应在首次稳定复现后采集 CPU/布局证据,再修改热点;本轮最终定位到共享 SVG 长度读取造成的同步布局,单纯调整避让参数不足以解释该成本。性能、运动观感及地理缺项以功能计划的验收状态为准。
|
||||
@@ -1,715 +0,0 @@
|
||||
# Earth 天球背景与日月位置实施方案
|
||||
|
||||
## 目标
|
||||
|
||||
为 Earth 大屏增加一套真正可用的天文背景层,覆盖三件事:
|
||||
|
||||
1. 用真实天球背景替换当前随机星点
|
||||
2. 在当前时间下显示太阳与月亮的相对位置
|
||||
3. 让太阳方向同时驱动地球受光,形成更可信的昼夜关系
|
||||
|
||||
本方案优先追求:
|
||||
|
||||
- 与当前 Three.js Earth 架构兼容
|
||||
- 风险可控
|
||||
- 先落地一版真实感明显提升的 V1
|
||||
- 为后续更严格的天文参考系升级预留余地
|
||||
|
||||
## 当前现状
|
||||
|
||||
当前 Earth 的基础条件已经具备:
|
||||
|
||||
- 地球、云层、地形、网格都基于 Three.js,主渲染入口在 [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js)
|
||||
- 地球实体创建在 [frontend/public/earth/js/earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js)
|
||||
- 当前所谓“宇宙背景”只是 `createStars()` 生成的随机星点,不是真实星图
|
||||
- Earth 已有倾角常量 `EARTH_CONFIG.tiltRad`,位于 [frontend/public/earth/js/constants.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/constants.js)
|
||||
- 主循环 `animate()` 已稳定运行,可在其中接入天体更新逻辑
|
||||
|
||||
这意味着:
|
||||
|
||||
- 不需要重写 Earth
|
||||
- 可以在现有 scene/world 层新增一个 celestial layer
|
||||
- 第一阶段不必拆 Earth / satellite / cable 的参考系
|
||||
|
||||
## 总体策略
|
||||
|
||||
采用“两层现实”设计:
|
||||
|
||||
### 1. 世界层(world-space celestial layer)
|
||||
|
||||
用于放置:
|
||||
|
||||
- 天球背景
|
||||
- 太阳
|
||||
- 月亮
|
||||
- 太阳光方向
|
||||
|
||||
这些对象不挂在 `earthObj` 上,而是直接放在 `scene` 中。
|
||||
|
||||
### 2. 地球层(earth-fixed layer)
|
||||
|
||||
继续保持当前结构:
|
||||
|
||||
- 海缆
|
||||
- 登陆点
|
||||
- 卫星点与轨迹
|
||||
- BGP 覆盖
|
||||
- 地球纹理、云层、地形
|
||||
|
||||
这些对象继续挂在 `earthObj` 下,不打断现有交互。
|
||||
|
||||
## 为什么先这样做
|
||||
|
||||
当前用户交互是“拖动地球本体”,而不是“移动相机绕惯性系观测”。
|
||||
如果现在直接做严格惯性参考系改造,会同时影响:
|
||||
|
||||
- `earthObj.rotation`
|
||||
- 卫星轨迹与锁定逻辑
|
||||
- 海缆与登陆点附着关系
|
||||
- resetView / autoRotate / hover / click 等交互链路
|
||||
|
||||
所以第一阶段只做:
|
||||
|
||||
- 真正的天空
|
||||
- 真正的日月方向
|
||||
- 不碰现有 Earth 附着对象的语义
|
||||
|
||||
## 推荐技术选型
|
||||
|
||||
### 天文计算库
|
||||
|
||||
推荐:
|
||||
|
||||
- [Astronomy Engine](https://github.com/cosinekitty/astronomy)
|
||||
|
||||
原因:
|
||||
|
||||
- 有 JavaScript 版本
|
||||
- 支持 Sun / Moon 的矢量与坐标变换
|
||||
- 精度、可扩展性都比轻量太阳高度角库更适合本项目
|
||||
- 后续若要加行星、月相、黄道、赤道网,也能继续沿用
|
||||
|
||||
不作为主选的库:
|
||||
|
||||
- [SunCalc](https://github.com/mourner/suncalc)
|
||||
|
||||
原因:
|
||||
|
||||
- 更偏本地观察者视角的太阳/月亮高度角
|
||||
- 用于“地面日出日落”很好
|
||||
- 但不如 Astronomy Engine 适合做真实天球与后续空间参考系扩展
|
||||
|
||||
### Three.js 表现层
|
||||
|
||||
推荐组合:
|
||||
|
||||
- 天球:内翻球壳 + 星图纹理
|
||||
- 太阳:`THREE.Sprite`
|
||||
- 月亮:`THREE.Sprite` 或小型 `THREE.Mesh`
|
||||
- 太阳光:`THREE.DirectionalLight`
|
||||
|
||||
参考:
|
||||
|
||||
- [Three.js SpriteMaterial](https://threejs.org/docs/pages/SpriteMaterial.html)
|
||||
|
||||
## 天球背景资源与星体数据来源
|
||||
|
||||
为避免把“视觉背景”和“可计算天体位置”混为一谈,本方案明确分成两类资源:
|
||||
|
||||
### 1. 背景资源:全天星图贴图
|
||||
|
||||
用于 Phase 1 的“真实天空背景”。
|
||||
|
||||
推荐优先来源:
|
||||
|
||||
- NASA SVS 的 Tycho 全天星图
|
||||
- [The Tycho Catalog Skymap - Version 2.0](https://svs.gsfc.nasa.gov/3572/)
|
||||
- NASA Deep Star Maps 2020
|
||||
- SatelliteMap.space 在 credits 中明确提到其使用了 `NASA Deep Star Maps 2020 - High-resolution star field (1.7 billion stars from Gaia DR2)` 作为星空视觉资源
|
||||
- 这说明行业内成熟实现并不一定直接渲染全部星表点,而很可能先使用一张高质量官方深空星图作为背景层
|
||||
- 如需后续替换,也可评估 ESA / Gaia 的全天 sky map 资源
|
||||
- [Gaia DR3 stories](https://www.cosmos.esa.int/web/gaia/dr3-stories)
|
||||
|
||||
建议要求:
|
||||
|
||||
- 使用官方来源或官方衍生可复用资源
|
||||
- 等距矩形投影(equirectangular)
|
||||
- 坐标定义尽量明确为赤道坐标展开
|
||||
- 分辨率建议至少 `4k`
|
||||
- 颜色不要过亮,避免压过 Earth HUD 前景
|
||||
- 尽量优先选择官方天文机构已经生产好的深空图,而不是自行拼接低质量星空纹理
|
||||
|
||||
建议本地资源目录:
|
||||
|
||||
- `frontend/public/earth/assets/celestial/starmap_equatorial_4k.jpg`
|
||||
|
||||
### 2. 位置数据:星表与天体计算
|
||||
|
||||
用于 Phase 2+ 的“位置正确的星体”。
|
||||
|
||||
推荐来源分两层:
|
||||
|
||||
- 太阳、月亮位置
|
||||
- 使用 [Astronomy Engine](https://github.com/cosinekitty/astronomy)
|
||||
- 恒星位置
|
||||
- 第一优先:Hipparcos / Tycho
|
||||
- [Hipparcos overview](https://www.cosmos.esa.int/web/Hipparcos)
|
||||
- [Hipparcos catalogues](https://www.cosmos.esa.int/web/hipparcos/catalogues)
|
||||
- 第二优先:Gaia
|
||||
- [Gaia DR3 stories](https://www.cosmos.esa.int/web/gaia/dr3-stories)
|
||||
|
||||
建议策略:
|
||||
|
||||
- V1:背景球壳只用全天星图,不立即生成全量恒星点
|
||||
- V2:只挑选亮星(例如星等 `< 5.5`)生成恒星点层
|
||||
- V3:如果确实需要更丰富的星场,再逐步扩展到更深星等
|
||||
|
||||
这样做的原因:
|
||||
|
||||
- 背景球壳负责“天球真实感”
|
||||
- 亮星点负责“位置正确、可后续标注和高亮”
|
||||
- 不需要一开始就处理数十万甚至数百万颗星
|
||||
|
||||
### 3. 对外部成熟实现的参考结论
|
||||
|
||||
`SatelliteMap.space` 的公开 credits 提供了一个很有价值的参考样板:
|
||||
|
||||
- 图形渲染使用 `TWGL.js`
|
||||
- 天文计算使用 `Skyfield` 与 `Astronomia`
|
||||
- 星空/天球视觉资源使用 `NASA Deep Star Maps 2020`
|
||||
|
||||
这给本项目的启发是:
|
||||
|
||||
- “真实感强的天球背景”完全可以先依赖官方高质量深空图
|
||||
- “位置正确的动态天体”则应依赖单独的天文计算链路
|
||||
- 没有必要在第一版就直接渲染完整星表
|
||||
|
||||
因此本项目推荐继续坚持两层拆分:
|
||||
|
||||
- 背景层:官方深空图 / 全天星图
|
||||
- 计算层:太阳、月亮与后续亮星点
|
||||
|
||||
## 如何保证星体位置正确
|
||||
|
||||
位置正确不是只看“图看起来像”,而是要统一参考系和转换链路。
|
||||
|
||||
### 1. 统一坐标基准
|
||||
|
||||
本方案推荐统一使用:
|
||||
|
||||
- `J2000` 赤道坐标系作为恒星位置基准
|
||||
|
||||
原因:
|
||||
|
||||
- Hipparcos / Tycho 资料和大量天文可视化都容易映射到该基准
|
||||
- 太阳、月亮也可以通过 Astronomy Engine 转到同一坐标系
|
||||
- 这样背景、恒星点、太阳、月亮就能共用一套 sky orientation
|
||||
|
||||
### 2. 背景贴图与点位必须使用同一展开逻辑
|
||||
|
||||
如果背景球壳使用赤道坐标全天图,那么:
|
||||
|
||||
- 亮星点也必须按赤道坐标贴到同一球面方向
|
||||
- 太阳/月亮 sprite 也必须按赤道坐标转换后落到同一 world-space
|
||||
|
||||
否则会出现:
|
||||
|
||||
- 背景银河带是对的
|
||||
- 但太阳/月亮或亮星点飘到不匹配的位置
|
||||
|
||||
### 3. RA / Dec 到 Three.js 坐标的落点方式
|
||||
|
||||
亮星点和日月方向最终都要转成单位球面向量。
|
||||
|
||||
概念步骤:
|
||||
|
||||
1. 读取赤经 `RA`
|
||||
2. 读取赤纬 `Dec`
|
||||
3. 转成弧度
|
||||
4. 映射到单位球面向量
|
||||
5. 再根据 Three.js 当前世界坐标定义做轴向映射
|
||||
|
||||
参考公式:
|
||||
|
||||
```text
|
||||
x = cos(dec) * cos(ra)
|
||||
y = sin(dec)
|
||||
z = cos(dec) * sin(ra)
|
||||
```
|
||||
|
||||
实际接入 Three.js 时,需要做一次项目内坐标轴校准:
|
||||
|
||||
- 验证 `RA = 0h`
|
||||
- 验证 `RA = 6h`
|
||||
- 验证北天极
|
||||
- 验证银河带主方向
|
||||
|
||||
然后确定最终的:
|
||||
|
||||
- `x/y/z` 对应 Three.js 哪个轴
|
||||
- 是否需要 `z` 取反
|
||||
- 是否需要整体再做一个固定 `rotation`
|
||||
|
||||
建议把这层显式封装在:
|
||||
|
||||
```js
|
||||
function equatorialToWorldVector(raRad, decRad)
|
||||
```
|
||||
|
||||
不要把轴映射散落在不同模块里。
|
||||
|
||||
### 4. 背景球壳与恒星点的关系
|
||||
|
||||
推荐最终组合:
|
||||
|
||||
- 背景层:全天星图球壳
|
||||
- 点位层:亮星点
|
||||
- 动态层:太阳 / 月亮
|
||||
|
||||
这样有三个好处:
|
||||
|
||||
- 背景层提供密集真实的天空纹理
|
||||
- 亮星点提供位置正确、可扩展的标注基础
|
||||
- 太阳/月亮提供与时间相关的真实动态对象
|
||||
|
||||
## 数据与资源建议清单
|
||||
|
||||
### 推荐首批引入资源
|
||||
|
||||
1. 全天星图
|
||||
- 来源:NASA Tycho all-sky map
|
||||
- 用途:背景球壳纹理
|
||||
|
||||
2. 月亮纹理
|
||||
- 用途:Phase 4 月相表现
|
||||
- 路径建议:
|
||||
- `frontend/public/earth/assets/celestial/moon_albedo_2k.jpg`
|
||||
|
||||
3. 太阳 glow 贴图
|
||||
- 用途:太阳 sprite halo
|
||||
- 路径建议:
|
||||
- `frontend/public/earth/assets/celestial/sun_glow.png`
|
||||
|
||||
### 推荐首批数据文件
|
||||
|
||||
如果要上亮星层,建议新增一个预处理后的轻量数据文件:
|
||||
|
||||
- `frontend/public/earth/assets/celestial/bright-stars.json`
|
||||
|
||||
建议字段:
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"id": 32349,
|
||||
"name": "Sirius",
|
||||
"raDeg": 101.2875,
|
||||
"decDeg": -16.7161,
|
||||
"mag": -1.46,
|
||||
"colorIndex": 0.00
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
建议不要在浏览器里直接吞原始 Gaia 大表,而是先离线裁剪成:
|
||||
|
||||
- 只保留亮星
|
||||
- 只保留渲染必需字段
|
||||
- JSON 或二进制轻量格式
|
||||
|
||||
## 资源与数据实施路线
|
||||
|
||||
### 路线 A:先做可用版本(推荐)
|
||||
|
||||
1. 引入 NASA Tycho 全天图
|
||||
- 或评估替换为更接近 SatelliteMap.space 路线的 `NASA Deep Star Maps 2020`
|
||||
2. 实现背景球壳
|
||||
3. 用 Astronomy Engine 计算太阳/月亮方向
|
||||
4. 暂不做亮星点
|
||||
|
||||
优点:
|
||||
|
||||
- 最快见效
|
||||
- 风险最低
|
||||
- 就能明显提升天球真实感
|
||||
|
||||
### 路线 B:在 A 基础上增强
|
||||
|
||||
1. 离线生成 `bright-stars.json`
|
||||
2. 浏览器端渲染亮星点
|
||||
3. 后续可加:
|
||||
- 星座线
|
||||
- 亮星名称
|
||||
- 特定星体高亮
|
||||
|
||||
优点:
|
||||
|
||||
- 背景真实感和“位置正确的可交互星体”同时兼顾
|
||||
|
||||
## 代码模块建议细化
|
||||
|
||||
### 新增模块
|
||||
|
||||
- `frontend/public/earth/js/celestial.js`
|
||||
- 管理天球背景
|
||||
- 管理太阳/月亮
|
||||
- 管理亮星层(后续)
|
||||
|
||||
- `frontend/public/earth/js/celestial-data.js`
|
||||
- 资源路径
|
||||
- 星图方向配置
|
||||
- 亮星数据加载(后续)
|
||||
|
||||
### 建议函数设计
|
||||
|
||||
```js
|
||||
export function initCelestialLayer(scene)
|
||||
export function updateCelestialLayer(date)
|
||||
export function setCelestialVisibility(visible)
|
||||
export function disposeCelestialLayer()
|
||||
|
||||
function loadStarMapTexture()
|
||||
function createSkySphere(texture)
|
||||
function createSunSprite()
|
||||
function createMoonSprite()
|
||||
function getSunEquatorialPosition(date)
|
||||
function getMoonEquatorialPosition(date)
|
||||
function equatorialToWorldVector(raRad, decRad)
|
||||
```
|
||||
|
||||
### 推荐后续预处理脚本
|
||||
|
||||
如要引入亮星层,建议单独做离线脚本:
|
||||
|
||||
- `scripts/build_bright_stars.py`
|
||||
|
||||
职责:
|
||||
|
||||
- 从 Hipparcos / Tycho 源数据读取
|
||||
- 过滤亮星
|
||||
- 生成 `bright-stars.json`
|
||||
|
||||
这样浏览器端只消费轻量结果,不承担大表解析成本。
|
||||
|
||||
## 分阶段实施
|
||||
|
||||
## Phase 1:真实天球背景
|
||||
|
||||
### 目标
|
||||
|
||||
用真实全天星图替换当前随机星点背景。
|
||||
|
||||
### 做法
|
||||
|
||||
1. 新增一张全天星图纹理
|
||||
|
||||
建议路径:
|
||||
|
||||
- `frontend/public/earth/assets/celestial/starmap_equatorial_4k.jpg`
|
||||
|
||||
纹理要求:
|
||||
|
||||
- 等距矩形投影
|
||||
- 赤经/赤纬坐标展开
|
||||
- 无地平线、无地景遮挡
|
||||
- 尽量深色、弱干扰,适合大屏 HUD 叠加
|
||||
|
||||
2. 新增天球球壳
|
||||
|
||||
新增模块:
|
||||
|
||||
- [frontend/public/earth/js/celestial.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/celestial.js)
|
||||
|
||||
建议接口:
|
||||
|
||||
```js
|
||||
export function initCelestialLayer(scene)
|
||||
export function updateCelestialLayer(date, camera, earth)
|
||||
export function disposeCelestialLayer()
|
||||
```
|
||||
|
||||
3. 实现一个大半径内翻球体
|
||||
|
||||
建议参数:
|
||||
|
||||
- 半径:`600 ~ 900`
|
||||
- 材质:`MeshBasicMaterial`
|
||||
- `side: THREE.BackSide`
|
||||
- 不受场景光照影响
|
||||
- 始终围绕场景中心
|
||||
|
||||
### 验收标准
|
||||
|
||||
- 初始加载后背景不再是随机星点
|
||||
- 旋转地球时,背景保持为稳定天球而不是跟地球一起转
|
||||
- 不明显干扰海缆/卫星/BGP 的前景识别
|
||||
|
||||
## Phase 2:太阳与月亮真实位置
|
||||
|
||||
### 目标
|
||||
|
||||
在当前 UTC 时间下,计算太阳与月亮在天球中的方向,并显示出来。
|
||||
|
||||
### 做法
|
||||
|
||||
1. 在 `celestial.js` 内封装天体位置计算
|
||||
|
||||
建议函数:
|
||||
|
||||
```js
|
||||
function getSunDirection(date)
|
||||
function getMoonDirection(date)
|
||||
```
|
||||
|
||||
输出统一为 world-space `THREE.Vector3`
|
||||
|
||||
2. 太阳显示
|
||||
|
||||
- 一个暖色发光 sprite
|
||||
- 比月亮更大、更亮
|
||||
- 可选添加柔和 halo
|
||||
|
||||
3. 月亮显示
|
||||
|
||||
- 一个较小 sprite 或 sphere
|
||||
- 灰白偏冷色
|
||||
- 后续 Phase 3 再做月相
|
||||
|
||||
4. 更新频率
|
||||
|
||||
不要每帧重新做完整天文计算,建议:
|
||||
|
||||
- 每 30 秒或 60 秒重算一次真实位置
|
||||
- 渲染帧内做平滑过渡
|
||||
|
||||
### 验收标准
|
||||
|
||||
- 页面可见太阳与月亮两个对象
|
||||
- 时间变化时位置会更新
|
||||
- 日月不会跟随地球局部旋转而错误附着
|
||||
|
||||
## Phase 3:太阳驱动地球受光
|
||||
|
||||
### 目标
|
||||
|
||||
让地球光照方向与太阳方向一致,不再使用写死的固定主光。
|
||||
|
||||
### 做法
|
||||
|
||||
1. 替换或接管当前主定向光
|
||||
|
||||
当前 [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js) 中 `addLights()` 里用了固定方向的 `DirectionalLight`。
|
||||
|
||||
建议改为:
|
||||
|
||||
- 保留环境补光
|
||||
- 主太阳光方向由 `sunDirection` 决定
|
||||
|
||||
2. 太阳光参数建议
|
||||
|
||||
- `DirectionalLight` 颜色偏暖白
|
||||
- 强度略高于当前主光
|
||||
- 保留一个弱背光作为氛围补偿,避免背面过死黑
|
||||
|
||||
3. 先不做物理级大气散射
|
||||
|
||||
第一版只要求:
|
||||
|
||||
- 亮面与暗面方向真实
|
||||
- 云层和大气仍保持当前风格
|
||||
|
||||
### 验收标准
|
||||
|
||||
- 地球明暗面会随太阳方向改变
|
||||
- 太阳 sprite 和地球亮面方向一致
|
||||
- 不破坏现有海缆、卫星、BGP 的可见性
|
||||
|
||||
## Phase 4:月相与天文细节增强
|
||||
|
||||
### 目标
|
||||
|
||||
在日月真实位置基础上增加更强的“天文可信度”。
|
||||
|
||||
### 可选项
|
||||
|
||||
1. 月相
|
||||
|
||||
- 根据日月夹角计算 illuminated fraction
|
||||
- 用月相纹理或 shader 表达盈亏
|
||||
|
||||
2. 赤道/黄道辅助线
|
||||
|
||||
- 可作为开发调试层,不默认显示
|
||||
|
||||
3. 太阳 terminator 增强
|
||||
|
||||
- 给地球夜面加入更自然的 night tint
|
||||
- 未来可叠加城市夜光纹理
|
||||
|
||||
4. 天文时间入口
|
||||
|
||||
- 设置中加入“当前时刻 / 指定时刻 / 加速时间”模式
|
||||
|
||||
### 验收标准
|
||||
|
||||
- 月亮不再只是一个静态圆点
|
||||
- 后续扩展行星或观测模式时无需推倒重来
|
||||
|
||||
## Phase 5:严格参考系升级(可选,不作为 V1 必做)
|
||||
|
||||
### 目标
|
||||
|
||||
把 Earth 从“用户旋转球体”升级为“真实地球姿态 + 用户观察姿态”的双层模型。
|
||||
|
||||
### 需要处理的问题
|
||||
|
||||
- 地球自转角与 UTC 的一致性
|
||||
- 赤道坐标系、地固坐标系、相机交互层分离
|
||||
- 卫星轨道显示与 Earth 旋转同步关系
|
||||
- resetView 和 autoRotate 的语义重定
|
||||
|
||||
### 风险
|
||||
|
||||
这一步会影响:
|
||||
|
||||
- [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js)
|
||||
- [frontend/public/earth/js/satellites.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/satellites.js)
|
||||
- [frontend/public/earth/js/cables.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/cables.js)
|
||||
- [frontend/public/earth/js/controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js)
|
||||
|
||||
因此不建议与 V1 同时推进。
|
||||
|
||||
## 代码改造清单
|
||||
|
||||
## 1. 新增文件
|
||||
|
||||
- [frontend/public/earth/js/celestial.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/celestial.js)
|
||||
|
||||
职责:
|
||||
|
||||
- 管理天球背景、太阳、月亮
|
||||
- 对外暴露 init/update/dispose
|
||||
|
||||
## 2. 修改 `constants.js`
|
||||
|
||||
文件:
|
||||
|
||||
- [frontend/public/earth/js/constants.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/constants.js)
|
||||
|
||||
新增:
|
||||
|
||||
```js
|
||||
export const CELESTIAL_CONFIG = {
|
||||
sphereRadius: 800,
|
||||
updateIntervalMs: 60000,
|
||||
sunSpriteScale: 28,
|
||||
moonSpriteScale: 16,
|
||||
sunLightIntensity: 1.25,
|
||||
ambientIntensity: 0.28,
|
||||
backLightIntensity: 0.18,
|
||||
};
|
||||
```
|
||||
|
||||
## 3. 修改 `main.js`
|
||||
|
||||
文件:
|
||||
|
||||
- [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js)
|
||||
|
||||
主要改动:
|
||||
|
||||
1. `init()` 中:
|
||||
- 初始化 celestial layer
|
||||
2. `addLights()` 中:
|
||||
- 把固定太阳光改成可更新的 celestial sun light
|
||||
3. `animate()` 中:
|
||||
- 每帧调 `updateCelestialLayer()`
|
||||
4. `destroy()` 中:
|
||||
- 清理 celestial 资源
|
||||
|
||||
## 4. 修改 `earth.js`
|
||||
|
||||
文件:
|
||||
|
||||
- [frontend/public/earth/js/earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js)
|
||||
|
||||
主要改动:
|
||||
|
||||
- `createStars()` 逐步退役
|
||||
- 第一阶段可先保留作为 fallback
|
||||
- 当真实星图加载成功后,不再显示随机星点
|
||||
|
||||
## 5. 新增资源
|
||||
|
||||
目录建议:
|
||||
|
||||
- `frontend/public/earth/assets/celestial/`
|
||||
|
||||
建议至少包含:
|
||||
|
||||
- `starmap_equatorial_4k.jpg`
|
||||
- `sun_glow.png`
|
||||
- `moon_albedo_2k.jpg`
|
||||
|
||||
## 数据流设计
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A["main.js:init()"] --> B["initCelestialLayer(scene)"]
|
||||
B --> C["创建天球球壳"]
|
||||
B --> D["创建太阳 sprite + 主定向光"]
|
||||
B --> E["创建月亮 sprite"]
|
||||
|
||||
F["animate()"] --> G["updateCelestialLayer(now, camera, earth)"]
|
||||
G --> H["Astronomy Engine 计算 Sun/Moon 方向"]
|
||||
H --> I["更新 sun sprite / moon sprite 位置"]
|
||||
H --> J["更新太阳 DirectionalLight 方向"]
|
||||
J --> K["地球昼夜方向变化"]
|
||||
```
|
||||
|
||||
## 风险与注意事项
|
||||
|
||||
### 1. 星图投影方向容易反
|
||||
|
||||
这会表现为:
|
||||
|
||||
- 星图左右镜像
|
||||
- 赤经方向颠倒
|
||||
- 日月位置和背景对不上
|
||||
|
||||
建议:
|
||||
|
||||
- 先做一个开发调试模式
|
||||
- 显示赤经/赤纬参考点,快速校正纹理朝向
|
||||
|
||||
### 2. 不要让天球跟随 Earth 旋转
|
||||
|
||||
天球背景和日月必须属于 scene/world,而不是 `earthObj`。
|
||||
|
||||
### 3. 不要每帧做重型天文计算
|
||||
|
||||
真实位置更新应节流,否则会浪费 CPU。
|
||||
|
||||
### 4. 月亮先求“方向正确”,再求“月相精致”
|
||||
|
||||
月相属于第二步优化,不应阻塞 V1 上线。
|
||||
|
||||
## 推荐实施顺序
|
||||
|
||||
1. 新建 `celestial.js`
|
||||
2. 用星图球壳替换随机星点
|
||||
3. 接入 Astronomy Engine
|
||||
4. 加太阳/月亮 sprite
|
||||
5. 用太阳方向驱动主光
|
||||
6. 再决定要不要做月相和更严格参考系
|
||||
|
||||
## 最终建议
|
||||
|
||||
对于当前 Planet Earth,最稳妥的方案是:
|
||||
|
||||
- 先做真实天球背景
|
||||
- 再做真实太阳/月亮方向
|
||||
- 再让太阳驱动地球受光
|
||||
- 暂时不做 Earth 参考系重构
|
||||
|
||||
这样可以在不破坏现有 Earth 交互和图层系统的前提下,显著提升空间感、真实感和演示说服力。
|
||||
@@ -1,106 +0,0 @@
|
||||
# Earth Predicted Orbit Plan
|
||||
|
||||
> Source note: this plan absorbs useful ideas from a sisyphus-created draft formerly stored at `.sisyphus/plans/predicted-orbit.md`.
|
||||
|
||||
## Goal
|
||||
|
||||
在 Earth 中锁定卫星时,显示“预测轨道”而不是只有历史尾迹:
|
||||
|
||||
- 从当前时刻开始
|
||||
- 绕地球一圈
|
||||
- 当前点最亮
|
||||
- 向后沿轨道逐步衰减
|
||||
|
||||
## Current State
|
||||
|
||||
当前已经有:
|
||||
|
||||
- 卫星历史轨迹
|
||||
- 锁定卫星
|
||||
- 轨道高亮与相关联动
|
||||
|
||||
但“预测轨道”仍然不是一套稳定、可验证的单独功能计划。
|
||||
|
||||
## Why It Is Valuable
|
||||
|
||||
预测轨道可以明显提升:
|
||||
|
||||
- 锁定卫星后的空间可读性
|
||||
- 轨道类型辨识
|
||||
- 演示解释力
|
||||
|
||||
相比短历史尾迹,预测轨道更符合用户对“这颗卫星接下来会怎么走”的预期。
|
||||
|
||||
## Scope
|
||||
|
||||
### Phase 1
|
||||
|
||||
- 锁定卫星时显示一整圈预测轨道
|
||||
- 解锁时隐藏
|
||||
- 不替代现有普通轨迹系统
|
||||
|
||||
### Phase 2
|
||||
|
||||
- 根据轨道类型调整采样率
|
||||
- GEO / MEO / LEO 不同密度
|
||||
- 进一步减少 fallback 轨迹的比例
|
||||
|
||||
## Implementation Direction
|
||||
|
||||
### 1. Orbit period
|
||||
|
||||
基于 `meanMotion` 估算轨道周期。
|
||||
|
||||
### 2. Predicted samples
|
||||
|
||||
以固定采样步长从 `now -> now + period` 推算轨迹点。
|
||||
|
||||
### 3. Render object lifecycle
|
||||
|
||||
预测轨道应是一个独立渲染对象:
|
||||
|
||||
- show
|
||||
- update
|
||||
- hide
|
||||
- dispose
|
||||
|
||||
### 4. Visual semantics
|
||||
|
||||
预测轨道不应与普通尾迹混淆:
|
||||
|
||||
- 更稳定
|
||||
- 更完整
|
||||
- 透明度沿轨道衰减
|
||||
- 当前点附近更亮
|
||||
|
||||
## Known Risks
|
||||
|
||||
### 1. TLE propagation gaps
|
||||
|
||||
部分卫星可能出现 SGP4 计算不足,需要 fallback。
|
||||
|
||||
### 2. Multiple orbit lines
|
||||
|
||||
必须确保:
|
||||
|
||||
- 锁定切换前先清旧轨道
|
||||
- 页面隐藏/销毁时清理
|
||||
|
||||
### 3. Performance
|
||||
|
||||
GEO 轨道点数高,采样率需要按轨道类型分层。
|
||||
|
||||
## Acceptance
|
||||
|
||||
1. 锁定单颗卫星时只显示一条预测轨道
|
||||
2. 解锁后轨道立即清除
|
||||
3. 不同轨道类型下点数可控
|
||||
4. 页面切换回来不会闪出旧轨道残留
|
||||
|
||||
## Satellite Footprint Follow-Up Items
|
||||
|
||||
从技术文档迁出的 footprint 后续项,作为卫星覆盖能力的计划 backlog:
|
||||
|
||||
1. 为 `iridium-next` 新建独立 footprint adapter。
|
||||
2. 在 UI 上补一个只读提示,让用户知道当前卫星是否支持 footprint。
|
||||
3. 如果未来拿到 GEO beam contour / operator metadata,再为 GEO 开 operator-specific footprint。
|
||||
@@ -0,0 +1,237 @@
|
||||
# Earth 地理资源与空间卡片展示实施计划
|
||||
|
||||
日期:2026-09-21。状态:本地可运行原型。贴图已替换并归档旧图;固定规则、部分核验几何、全球参与方目录、空间卡片、共享移动滚动条与轻量材质调整已实现。完整政治几何、逐段海岸对齐、视觉认可及性能门槛尚未全部验收,不代表可发布。
|
||||
|
||||
本计划以已确认的展示方向为准,替代算力专题旧方案中的固定右侧栏、横贯底部双图和隐藏其他面板的布局。空间卡片与算力目录已有本地可运行原型,但仍需完成最终 QA;算力数据采集、指标证据和长期模型设计继续参见[算力与资源态势计划](compute-resource-intelligence-plan.md)。
|
||||
|
||||
## 1. 目标与工作边界
|
||||
|
||||
地球是空间信息的主体。打开区域对比后,多个区域分别引出连接线,牵引周围的浮动信息卡。地球转动时,地理锚点跟随球体,卡片保持正向可读,通过平滑、有阻尼的运动调整位置并互相避让,形成信息从地球四周延伸出来的感觉。
|
||||
|
||||
地球质感参考 UE 3D 截图在易拉宝与 PPT 中的表现:球面有层次、边缘与明暗交界柔和,背景退后,数据与球体构成同一空间。保持当前操作流畅度是硬验收条件。
|
||||
|
||||
已授权实施范围包括资源、空间卡片、共享 UI 与性能验证。继续使用现有 Three.js 引擎和资源加载机制。资源版本与实际转换参数以[地球资源说明](../../resources/earth/README.md)和清单为准。
|
||||
|
||||
### 当前实现与未完成项
|
||||
|
||||
本轮已实现卡片运动、配置控制、内容承载及中性地球材质调整。配置窗口与卡片显示开关已分离,关闭配置不再停用卡片。卡片默认常显基础信息:36px 主数值、年份/指标/口径说明、设施样本与 Epoch 来源、趋势首末年份及单位、证据入口和区域成员数;长列表仅在需要时展开完整图表。桌面卡片约宽 268px,高度随内容自适应,国家约 214px、区域约 223px;展开卡片支持用户调整至最高 480px,位置/尺寸可保存并恢复自动布局。手机保留同等摘要信息。高清 overlay 使用白色 MeshBasicMaterial、opacity 1、toneMapped:false;r128/r160 按 active API 使用对应 sRGB 读写路径。没有新增 GPU draw pass 或高分辨率纹理,但新增并预加载 45 个本地 SVG 旗帜资源。
|
||||
|
||||
旧版底边换侧的连续跳动已定位并由路径规划与稳定换侧策略处理。新实现由 `spatial-callout-controller.js` 管理,布局采用事件驱动并保留有效旧矩形;投影连线持续跟随地球,另一侧候选需持续 1800ms 才换侧,12 秒阅读间隔用于抑制不必要的替换。悬停、焦点和详情阅读会锁定卡片,用户拖动复用 `hud-drag.js`,现有 controls 也消费该共享模块。多卡布局通过实际空闲矩形安排最终位置,优先避开已有功能窗;局部 UI QA 已完成,全球性能与最终发布回归仍按后续门槛验证。下方性能数字和旧尺寸的回归记录属于此前原型,不能直接作为当前大卡片的性能验收结论。
|
||||
|
||||
- 底图默认“原始底图”(菜单首项),显示菜单可切换并保存;Natural Earth I 保留,NASA 21K 仅作源归档。运行时统一 8K,按需只保留一张 GPU 纹理;成功后释放旧图,失败保留旧图,过期响应不能覆盖当前图源,不新增渲染层。星空逐字节恢复 commit `83a10a6c`,不再校准密度和颜色。高清 overlay 使用白色 MeshBasicMaterial;r128/r160 按 active API 选择对应 sRGB 读写。没有新增长驻高分辨率纹理、后处理或 GPU draw pass;算力卡另新增并预加载 45 个本地 SVG 旗帜资源。Natural Earth 原图偏浅,不把不存在的细节写成恢复结果。
|
||||
- 地理政策由版本化清单贯穿规范编码、后台聚合、高精几何及低精回退。中国、台湾、港澳、科索沃等已有回归;已核验几何及济州岛补全见资源说明。高精技术就绪与政策可发布分开:`high_precision_ready=true`,`policy_publishable=false`。东沙、中沙、曾母暗沙点注记及经核验海上主张线仍有缺项。
|
||||
- 规范目录覆盖 45 个国家编码,界面分组为中国、美国、欧洲、不含中国的亚太和其他;分组不当作统计合计。当前空间原型最多同时显示 6 张,后端最多接受 12 个参与方。没有历史观测的地区返回 `null` 历史与覆盖信息;设施样本数永远不作为计算出的国家总量。
|
||||
- 桌面卡片复用共享 HUD 和连接线,保留稳定 DOM 身份;选择器和来源窗使用共享拖动。背面卡片保留内容并隐去连接线。旋转通过原动画循环更新,投影连线持续更新;布局按事件分帧处理并保留有效旧矩形。`main.js` 的 `isDragging`、`activeTouchPoints`、`hasActiveGlobeInertia` 接入运动模式:手动/惯性快速模式以 72ms/12px 为响应预算,在当前帧完成受限布局,停止后一次 settle;slow auto 保留 1.8 秒换侧和 12 秒阅读保持,用户拖动放置优先。
|
||||
- 手机复用公共抽屉与深色滚动所有者,使用适合手机阅读的字号;目前是抽屉内紧凑卡片列表,尚未实现手机环绕空间卡片。桌面原型没有强制隐藏新闻、媒体或统计窗口。
|
||||
- 2/4/6 卡片、背面投影、共享窗口避让、饱和纵列与共享连线动画已有 11 项单元测试。浏览器已检查 1366×768、1920×1080、390×844 及 125%/150% 对应的有效 CSS 视口;单元测试不替代运动观感验收。
|
||||
|
||||
### 本轮空间卡片 UI 定向验收(2026-09-21)
|
||||
|
||||
本计划保留此前的历史验证记录;它们不代表当前地图源切换或全球 Earth 验收已经完成。当前实现说明只记录所需资源生命周期与 UI 约束,Browser/全球性能结论仍按后续门槛单独复核。
|
||||
|
||||
以上是本轮卡片 UI 与局部资产的定向证据。全球地理政策、逐段海岸对齐、完整 Earth 渲染性能、真实设备性能及发布级回归仍按本计划后续门槛单独验收。
|
||||
|
||||
## 2. 优先级与不可退让的要求
|
||||
|
||||
| 优先级 | 要求 | 执行原则 |
|
||||
| --- | --- | --- |
|
||||
| P0 | 固定政治展示规则 | 外部数据、更新、回退及 AI 均不能覆盖项目规则 |
|
||||
| P0 | 性能不退化 | 在同设备、同数据、同场景下验证;可复现的退化不能用视觉提升抵消 |
|
||||
| P1 | 地理对应一致 | 贴图定位、海岸线、区域归属、拾取和卡片锚点一致 |
|
||||
| P1 | 地球关联的空间卡片 | 使用现有连接线与展示机制扩展,不恢复固定大屏布局 |
|
||||
| P1 | 共享 UI 与移动端一致 | 共用字体、标题栏、按钮、滚动容器及窗口行为 |
|
||||
| P2 | 卡片内图表和内容完善 | 先验收空间关系与运动,再充实图表和解释 |
|
||||
|
||||
## 3. 固定政治展示规则
|
||||
|
||||
项目展示采用固定中国视角:中国相关领土及争议区域按项目既定中国视角处理,台湾按中国处理;科索沃按塞尔维亚处理。该口径先于数据源选择,不由数据源默认分类、模型输出或换源结果决定。
|
||||
|
||||
实施时建立单一、版本化的规则清单与几何修正包。名称映射只是其中一部分,必须覆盖国家编码、Polygon/MultiPolygon 归属、边界绘制、区域填色、鼠标拾取、高亮、统计及卡片归属。处理后的结果才可以成为运行资源;未经规则校验的新来源不得替换正式资源,旧资源回退也必须通过同一规则。
|
||||
|
||||
规则清单应包含:稳定规则 ID、目标区域的明确标识、目标归属、显示名称、几何来源、合并/裁切操作、版本及回归样例。不能只依赖模糊地名字符串替换。语音中未能确认的具体岛礁名称必须在修正清单中明确定位,不能猜测名字后直接移动多边形。
|
||||
|
||||
海岸线、陆地行政区域和海上主张线是不同几何层。项目口径在各层一致执行;海上主张线不能作为陆地面直接填充,也不能代替具体岛礁的几何和归属规则。
|
||||
|
||||
必须具备的失败测试:外部源恢复原始分类、别名变化、同区域不同编码、几何修改后拾取仍用旧数据、低精度回退覆盖规则。任何一项失败,都不发布该资源版本。
|
||||
|
||||
## 4. 已查明的现状
|
||||
|
||||
### 4.1 贴图与国界
|
||||
|
||||
- 原运行贴图为 `frontend/public/earth/assets/8k_earth_daymap.jpg`,8192×4096;代码直接加载 JPG,未读取该文件配套的地理定位元数据。
|
||||
- `country-boundaries.js` 使用独立 GeoJSON/PMTiles 几何绘线与拾取;高精度是一个选择状态,存在退回低精度的路径,按钮名称不等于实际正在使用的文件。
|
||||
- 本机此前检查到 `natural-earth-v1`、`geojson-high-precision` 构建清单,以及独立 base、hover、claim 文件;该构建不能直接等同于完整应用了项目政治几何规则。实施前需要重新读取实际运行版本。
|
||||
- 已提供的两张问题截图中,一张出现海岸折线与影像陆地错位;另一张有可见岛屿但没有对应线框。截图只能证明现象,尚不能断言根因一定是 UV、投影、低精度回退或缺岛。
|
||||
- 精度检查必须同时检查“看得到轮廓”和“能选中对应区域”,二者不能各用一套不一致的几何。
|
||||
|
||||
### 4.2 World Monitor 对照
|
||||
|
||||
本机 World Monitor `463aec0f` 使用独立地表图片与 `countries.geojson`;经纬度归属由点落入多边形的计算决定,二维底图和自带的交互区域也分开加载。
|
||||
|
||||
本次文件统计:World Monitor 的中国、日本、韩国分别为 200、60、14 个坐标点;Planet 的低精度分别为 240、65、19;本机高精度文件分别为 14,118、6,952、3,060。点数不等于测绘准确度,但表明不能靠直接换用 World Monitor 的简化国家文件解决近景海岸问题。它的原始归属处理也不能代替本项目规则。
|
||||
|
||||
### 4.3 已有共享机制
|
||||
|
||||
| 入口 | 当前作用 | 计划复用方式 |
|
||||
| --- | --- | --- |
|
||||
| `frontend/public/earth/js/callout-connector.js` | 连接线、元素锚点与路径 | 扩展为多个稳定区域锚点与卡片配对 |
|
||||
| `frontend/public/earth/js/presentation-controller.js` | 展示生命周期与连接器更新 | 在单项展示的基础上设计受控的多项展示,不破坏现有模式 |
|
||||
| `frontend/public/earth/js/cruise-sequencer.js` | 巡航时序 | 保留当前巡航行为,区域对比明确自己的生命周期 |
|
||||
| `frontend/public/earth/js/info-card.js` | 公用详情与移动详情 | 抽出/复用内容渲染能力,不复制成另一套卡片组件 |
|
||||
| `frontend/public/earth/js/hud-panels.js`、`controls.js` | 面板收起、显示、拖动与约束 | 工具窗继续遵循这些规则;明确 createHUDPanel 本身不包办所有拖动/缩放 |
|
||||
| `frontend/public/earth/css/hud.css`、`info-panel.css` | 公用外观、字号、滚动与移动抽屉 | 去掉专题私设的大字号、独立控件和白色滚动条 |
|
||||
|
||||
## 5. 地理资源:先采用配套贴图的简单方案
|
||||
|
||||
### 5.1 选择与准备
|
||||
|
||||
默认运行底图为“原始底图”(菜单首项),显示菜单可切换并保存底图源;Natural Earth I 保留可用。NASA 21K 仅保留为源归档,运行包统一使用 8192×4096 的单张 GPU 贴图。官方源版本与许可信息仍以资源清单为准,不能把归档源误写成运行资源。
|
||||
|
||||
采用不带政治标注的物理地表图。按需只保留一张运行 GPU 贴图;新源成功加载后释放旧图,失败保留旧图,过期响应不得覆盖当前图源。源包保留 TIFF、世界文件和说明,运行资源统一为 8192×4096 的 Web 图。转换只做尺寸与格式适配,不用生成式图像重绘海岸、岛屿或国界,不随意裁切、旋转或拉伸经纬范围;底图切换不新增渲染层。
|
||||
|
||||
下载的原始大文件保存在本地 `downloads/earth/`,不纳入前端发布包。可追溯清单、配准信息与旧贴图归档放在 `resources/earth/`;运行图放在现有 `frontend/public/earth/assets/`。旧图必须先按字节完整归档并核对 SHA256,之后才能替换运行路径。
|
||||
|
||||
### 5.2 对齐验证
|
||||
|
||||
1. 读取 TIFF/TFW 的像元大小、原点、行列方向、地理范围和 CRS;同时记录 JPEG 派生后的像元中心定位,避免半像素偏移。
|
||||
2. 明确运行中加载的贴图、边界文件、资源版本与是否发生回退。
|
||||
3. 使用同一投影和球面经纬转换叠加成套矢量,检查赤道、零经线、日期变更线和高纬区域。
|
||||
4. 在截图对应海岸、近海岛屿、台湾周边和其他不同经度的典型海岸逐点验证,区分整体偏移、局部概化和缺失几何。
|
||||
5. 政治修正后的同一份区域几何用于标注、高亮和拾取。所有需要烘焙的线也从这份数据导出,不能在图片和矢量上分别手工描线。
|
||||
6. 小岛、岛礁或海岸不满足要求时,先补充对应区域数据并重新配准;仅当这条简化路线确实不够用时,再考虑局部细节或分级资源。
|
||||
|
||||
成套资源声明配准不等于本项目已经逐像素验收。不能承诺卫星影像每一处潮线都与概化海岸线完全一致;验收需保存具体偏差和缺失清单,不能靠强行拉伸边界迎合图片。
|
||||
|
||||
### 5.3 常态渲染与交互
|
||||
|
||||
优先预合成常态显示所需的边界,以减少额外线段绘制。需要独立国界开关时,评估复用现有覆盖层或同定位的透明边界层;不默认再常驻一张 8K 纹理。矢量数据继续承担区域判定和选中高亮,不在每帧重新解释整份 GeoJSON。
|
||||
|
||||
本阶段不以替换 Three.js、引入新 GIS 平台或建设全球在线瓦片服务作为前置条件。
|
||||
|
||||
## 6. 整体视觉方向
|
||||
|
||||
参照资产:`Planet-Handoff-20260916` 中的完整易拉宝效果稿和卡片展示 PPT,尤其 PPT 内来自 UE 3D 的原始星球画面。海报中的线点含概念叠加,不能拿来充当精确地理数据。
|
||||
|
||||
- 地球保持正圆与正确地理位置,视觉上成为主体;不把纹理精细度等同于画面品质。
|
||||
- 球面采用柔和、连续的明暗层次,边缘有薄而自然的大气过渡;地理细节仍然可读。
|
||||
- 降低背景星点对比与密度造成的干扰,让背景退后;变化以参考图与实际渲染对照决定。
|
||||
- 数据节点与连接线有轻重层级,避免强霓虹、油亮塑料感、浓重太空雾和粗硬轮廓。
|
||||
- 优先在现有材质、光照与透明度范围内调整;新增后处理、辉光或高分辨率透明层须先证明成本。
|
||||
- 用同一位置、距离、光照条件对比前后效果,近景与远景都检查边缘、深度冲突和细节。
|
||||
|
||||
## 7. 区域卡片:从地球四周延伸出来
|
||||
|
||||
### 7.1 行为模型
|
||||
|
||||
一个区域对应稳定的地理锚点、连接线和卡片 ID。锚点随球体旋转;卡片处在屏幕空间,保持正向、稳定大小和内容身份。转动时变化的是屏幕位置与线的路径,不重建卡片、不交换身份、不清空正在阅读的内容。
|
||||
|
||||
卡片根据区域投影分布到球体周围,保留疏密和空间关系,不固定平均分成八个方向,不横排占满屏幕。使用有限的避让和有阻尼的缓动,最终能够安定;不持续抖动或为了追踪锚点逐帧跳位。
|
||||
|
||||
### 7.2 布局与生命周期
|
||||
|
||||
- 沿用现有连线的起点/终点接口和动画语言,先验证 2 张、4 张、6 张卡片;6 张是原型测试档位,不是已确定的产品上限。
|
||||
- 保持地域锚点与区域绑定;算法可以移动卡片,但不能移动真实地理坐标来换取好看的布局。
|
||||
- 优先避让其他卡片和已有工具窗,不自动隐藏新闻、直播、统计、图层面板。
|
||||
- 对卡片换侧设置稳定区间,避免锚点越过某条中心线就立即左右互换。
|
||||
- 背面区域不画成穿过地球的前景锚点。建议保留卡片身份,用边缘提示表达背面方位;完整隐藏、收束或边缘提示的最终表现用短动画原型比较后决定。
|
||||
- 用户正在操作的卡片短暂锁定其屏幕位置;是否允许长期固定/拖动偏移,先以现有交互能力做原型验证,不增加未经确认的复杂模式。
|
||||
- 增删区域、切换年份、退出对比、隐藏相关图层和请求失败都要有明确的清理与恢复逻辑;旧响应不能复活已关闭卡片。
|
||||
- 区域名称锚点用于对比展示;不能把区域中心点画成设施位置。
|
||||
|
||||
### 7.3 卡片内容与参与方
|
||||
|
||||
先验证空间展示,再完善图表。首轮卡片可以只包含名称、一个有来源的关键指标、参考期及来源入口;固定侧栏和横贯底部双图的旧设计不再继续扩展,普通浮窗加分页的方案也不是最终展示方向。
|
||||
|
||||
参与方必须支持全球扩展,不能在接口和 UI 中永久硬编码成中美两项。日本、韩国等可以作为独立对象;亚太、欧洲可用于选择菜单的分组或区域筛选。具体默认组合与区域汇总口径还需在完整 UI 原型中确定,不把早期讨论的菜单建议当成已确认设计。组合区域与成员国不能重复计入同一分母。
|
||||
|
||||
缺少同口径历史数据的参与方仍可展示公开设施覆盖及明确的缺失状态,不伪造总量、得分或排名。全国总算力、公开 AI 集群样本、TOP500 FP64 和创新指数继续分开。2026 年未核验值不得加入正式历史。A4 AI 解读不阻塞空间展示验收。
|
||||
|
||||
## 8. 共享窗口与移动端
|
||||
|
||||
设置、来源、图层等操作窗口继续复用已有标题栏、按钮、拖动和按能力启用的缩放逻辑。不得用单独字号和固定网格重新构建一套视觉体系。数据更新和语言切换应保留窗口位置、尺寸、选中对象和滚动进度。
|
||||
|
||||
移动端复用原底部抽屉与详情展示。区域卡片数量受可用空间控制,保证地球与主要交互可见;可收束为可切换的关联卡片,但具体排列需要实机原型确认。不能把桌面全部卡片原尺寸堆到手机上。
|
||||
|
||||
白色滚动条问题从 `.earth-mobile-drawer-content` 及实际滚动所有者核对,统一复用 Earth 的深色滚动条样式与滚动容器。覆盖图层、搜索、态势、算力、新闻和设置页;底部导航保持固定,避免嵌套重复滚动,不仅对算力页补一条局部 CSS。
|
||||
|
||||
## 9. 性能基线与硬验收
|
||||
|
||||
### 9.1 基线
|
||||
|
||||
资源替换前的画面可通过归档旧贴图复现。保存设备、浏览器、GPU、视口、DPR、图层开关、数据量、相机路径与资源版本。在相同条件下重复测试,至少记录 5 次稳定运行,并区分冷启动、热启动和稳定旋转。
|
||||
|
||||
观察:首个可操作画面时间、首个有效地球画面时间、帧时间 p50/p95/p99、长帧/长任务、交互延迟、JS 堆与纹理内存估算、draw calls、纹理/对象数、网络字节及请求次数。只比较平均 FPS 不足以证明没有卡顿。
|
||||
|
||||
### 9.2 实现预算
|
||||
|
||||
- 运行贴图先维持 8192×4096。相同格式下纹理像素量不增加;RGBA8 粗估基础约 128 MiB,完整 mipmap 约 171 MiB,不能把 JPEG 文件变小当成显存同比下降。
|
||||
- 保持已有 mesh、图标实例与纹理缓存;旋转不重新解码图片或构造国家几何。
|
||||
- 地理投影和轻量位置更新与渲染帧同步;碰撞布局只在锚点明显变化、卡片变化或窗口尺寸变化时重算,不让每张卡片各起一套动画循环。
|
||||
- 卡片 DOM 保持稳定,图表仅在数据/尺寸变化时更新;避免每帧测量布局后又写布局造成同步回流。
|
||||
- 网络、缓存、监听器、纹理及卡片数量均有边界,关闭/重开多轮不得持续增长。
|
||||
- 默认不增加新的高分辨率常驻覆盖层、昂贵后处理或全部区域的最高精度常驻数据。
|
||||
|
||||
### 9.3 判定
|
||||
|
||||
同条件下不能出现可复现的流畅度、加载、交互或资源占用退化。测量噪声必须记录并重复验证,不能事先允许固定比例退化来降低要求。超出预算先调整效果、卡片密度或实现,再复测。桌面与移动分别验收;没有真实手机数据时只能说明模拟检查通过,不能宣布手机性能达标。
|
||||
|
||||
### 9.4 内容卡扩容前的原型验证记录(2026-09-21)
|
||||
|
||||
同机 Headless Edge、桌面 1440×900 / 手机尺寸模拟 390×844、DPR 1、自动旋转,每种尺寸 5 次;预热 1 秒、帧时间采样 2 秒。该短采样用于定位,不能代替真实设备长时交互验收。原始基线标识 `2026-09-20T18-30-56-184Z`,最新对照标识 `2026-09-20T21-07-40-495Z`(UTC);完整 JSON 留在本机临时验证目录 `planet-earth-performance-baseline`。
|
||||
|
||||
| 项目 | 原基线 | 本轮结果 | 判断 |
|
||||
| --- | --- | --- | --- |
|
||||
| 桌面无卡 p95 帧时间,5 次中位数 | 7.09 ms | 7.10 ms | 接近;不能据此宣称完整性能通过 |
|
||||
| 手机尺寸模拟无卡 p95 | 7.00 ms | 7.20 ms | 未做真实手机测量 |
|
||||
| 桌面 2 / 4 / 6 卡 p95,5 次中位数 | 无同场景旧实现对照 | 7.10 / 7.10 / 14.00 ms | 六卡比本轮无卡场景有明确成本,仍需优化 |
|
||||
| 六卡采样窗口长任务 | — | 5 次均为 0 | 没有 50 ms 长任务不等于没有掉帧 |
|
||||
| 桌面 DOMContentLoaded 中位数 | 1.262 s | 1.627 s | 已移除 Google Fonts 的约 10 秒阻塞;仍不宣称加载性能与基线完全持平 |
|
||||
| 桌面基础 JS 堆中位数 | 约 50.2 MiB | 约 53.2 MiB | 不是显存;未替代纹理/绘制统计 |
|
||||
|
||||
每轮另做 5 次专题开关并强制 GC 检查,关闭后的保留堆约 25 MB,首末差约 0.28–0.33 MB;这不构成长期无泄漏证明。浏览器单独核对关闭后的卡片悬停/焦点监听器为 0,来源窗与拖出容器的选择器均能关闭。
|
||||
|
||||
最终 Bun 构建、前端规则检查、文档一致性检查与 `git diff --check` 通过。既有前端 smoke(使用其 API mocks)通过 321 项路由/交互检查,包括公共页面、鉴权、管理导航、移动布局和管理页缩放;Earth 空间卡片另用真实本地接口检查,不能混同两种证据。
|
||||
|
||||
布局求解已分帧执行、减少重复候选评分、优先前景区域,在合格布局找到后停止重复试排;工具栏按可见按钮而非透明容器避让。CPU 采样发现共享 SVG 的 `getTotalLength()` 同步布局热点,已改为按已知折线点计算长度,保留原绘线动画。11 项布局/投影/连线测试通过。1366×768 浏览器中 4/6 卡、旋转后、稳定后及工具栏展开时未发现与卡片、选择器或现有 HUD 的矩形交叠;1920×1080、1093×614、911×512 同样完成六卡边界检查,后两项是浏览器缩放的等效 CSS 视口,不冒充真实设备缩放测试。46 项相关后端测试通过,高精与回退的台湾、藏南、阿克赛钦、科索沃、济州岛浏览器拾取均符合规则。官方几何缺项、完整海岸对齐、六卡性能与真实手机仍未通过最终验收。
|
||||
|
||||
## 10. 分阶段实施与交付门槛
|
||||
|
||||
开发模型分工、文件所有权、并行限制和上下文控制按[开发 Agent 模型分配计划](development-agent-model-allocation-plan.md)执行:Sol 主控与核心实现,Terra 承担明确子任务,Luna 处理有边界的辅助工作,Astra 负责高风险设计及疑难审查。模型分工不改变本计划的政治规则、视觉方向和性能验收门槛。
|
||||
|
||||
| 阶段 | 工作 | 必须交付的证据 |
|
||||
| --- | --- | --- |
|
||||
| S0 资源准备 | 新贴图、配准信息、旧图归档、来源清单 | 新旧尺寸/哈希、解码与球面渲染检查;本轮范围 |
|
||||
| S1 规则与配准 | 固定政治清单、实际提供者诊断、配套边界对齐 | 规则测试、截图问题点对照、回退测试;不能只报高精度开关已打开 |
|
||||
| S2 空间运动原型 | 复用连接器,多区域锚点、卡片稳定身份与避让 | 转动/缩放/换侧/背面/退出短录屏,2/4/6 卡档位及性能差值 |
|
||||
| S3 视觉与组件 | 柔光质感、背景层次、共享字体与移动滚动 | UE/PPT/海报参照对照、多个视角、桌面/手机及缩放截图 |
|
||||
| S4 内容联动 | 多参与方、真实指标与来源、时间/设施联动 | 可操作路径、缺失/错误状态、来源与口径验证 |
|
||||
| S5 回归与发布准备 | 性能复测、旧图层/巡航/工具窗回归、同步文档 | 完整验收记录;需要发布时另走仓库发布流程 |
|
||||
|
||||
S1 未通过,不能把新图片称为边界问题已解决。S2 的空间交互未确认,不能直接投入完整图表和全部国家卡片。性能或政治规则任何一项未通过,后续阶段不能据此宣布完成。
|
||||
|
||||
## 11. 验收清单
|
||||
|
||||
- 政治:标签、几何、填色、拾取、统计、卡片与回退源口径一致;科索沃等明确规则有专门用例。
|
||||
- 地理:截图所指海岸和小岛、台湾周边、日期变更线、高纬度;记录偏差而非只肉眼说“差不多”。
|
||||
- 视觉:球体不变形,边缘柔和但地理清楚,卡片从区域延伸且不压满地球;远距无闪烁和黑块。
|
||||
- 交互:转动、缩放、选中、背面、卡片增删、触摸、键盘、语言切换、退出、窗口大小变化及旧响应竞态。
|
||||
- 共存:新闻、媒体、统计、图层及原详情卡保持可用;不通过强制隐藏制造布局空间。
|
||||
- 移动:公共抽屉无白色原生滚动条,底部导航固定,只有预期内容区滚动,文字和操作不被截断。
|
||||
- 响应式:1366×768、1920×1080、约 390×844,以及 125%/150% 缩放;模拟与实机结果分别记录。
|
||||
- 回归:公共页面、鉴权、管理路由、导航/搜索/标签页、既有 BGP/新闻/动捕巡航与旧图层。
|
||||
- 性能:与归档基线同条件比较,保存帧时间、内存、网络和交互证据;循环打开关闭验证资源释放。
|
||||
- 文档:更新实际变更的中英文手册、Earth context、图层规则和资源清单;计划与已实现行为不能混写。
|
||||
|
||||
## 12. 来源、恢复与完成定义
|
||||
|
||||
- [Natural Earth 栅格与矢量配准说明](https://www.naturalearthdata.com/downloads/10m-raster-data/)
|
||||
- [Natural Earth I 资源页面](https://www.naturalearthdata.com/downloads/10m-raster-data/10m-natural-earth-1/)
|
||||
- [Natural Earth 使用条款](https://www.naturalearthdata.com/about/terms-of-use/)
|
||||
- [现有 Presentation 解耦计划](earth-presentation-decoupled-architecture-plan.md)
|
||||
- [原高精度边界计划及历史说明](earth-high-precision-boundary-tiles-plan.md)
|
||||
- [地球前端上下文](../technical/zh/earth-frontend-context.md)
|
||||
- [渲染层次规范](../technical/zh/earth-render-layer-order.md)
|
||||
|
||||
旧贴图恢复路径和 SHA256 见资源清单。先核对归档哈希,再将归档副本复制回原运行资源路径并重建预览;不以回退图片为由撤销其他用户改动。新资源和旧归档均不得无记录覆盖。
|
||||
|
||||
最终完成意味着:展示规则固定且经验证,地图与区域选择对应,多个卡片确实由地球锚点牵引并稳定运动,操作组件和手机滚动保持一致,实际性能不低于基线。单张效果图、构建通过或“高精度”名称都不能单独证明完成。
|
||||
@@ -2,6 +2,8 @@
|
||||
|
||||
## 当前状态
|
||||
|
||||
最新实现继续保留分桶 `Points` 与全局数据范围,并已接通 `vessels` 全局实时订阅:后端读取确认状态后按 MMSI 拆包推送,前端原位修改缓冲;快照用于首次加载、重连和删除校准。旧快照不能覆盖较新的更新或删除,旋转与缩放仍不触发视口请求。下方 Sprite 问题分析和阶段方案保留为历史背景,当前约束以[地球前端上下文](../technical/zh/earth-frontend-context.md)为准。
|
||||
|
||||
该计划的前端核心部分已经在 `0.44.1` 落地,但最终实现不是原文设想的 `InstancedBufferGeometry` quad,而是更稳的分桶 `THREE.Points` 方案:
|
||||
|
||||
- 普通船只按 moving / anchored 和 `VESSEL_COURSE_BINS` 航向分桶,使用 `PointsMaterial` 批量绘制。
|
||||
|
||||
@@ -51,6 +51,7 @@ That split makes MiniMax, Claude-compatible gateways, and self-hosted OpenAI-com
|
||||
Supported request adapters:
|
||||
|
||||
- `openai-completions`
|
||||
- `openai-responses`
|
||||
- `anthropic-messages`
|
||||
- `ollama-generate`
|
||||
|
||||
@@ -95,12 +96,37 @@ The AI settings page uses:
|
||||
- `POST /api/v1/settings/integrations/ai-provider/connect`
|
||||
- `GET /api/v1/settings/integrations/ai-provider/secrets`
|
||||
- `GET /api/v1/settings/integrations/ai-provider/presets`
|
||||
- `POST /api/v1/settings/integrations/ai-provider/presets/{provider}/refresh`
|
||||
- `GET /api/v1/settings/ai-prompts`
|
||||
- `PUT /api/v1/settings/ai-prompts/{task_key}`
|
||||
- `POST /api/v1/settings/ai-prompts/{task_key}/reset`
|
||||
|
||||
These endpoints require an authenticated user. The `secrets` endpoint is only used when the settings page reveals a key or token; hiding the field restores the masked preview.
|
||||
|
||||
Model refresh and lightweight checks share `backend/app/services/llm_model_catalog.py` and query provider endpoints directly, without models.dev. Refresh uses the draft base URL, protocol and credentials, or saved configuration when no draft is supplied. Regional hosts and custom gateway paths are preserved. Missing required credentials return 400; upstream failures return safe errors and retain the previous catalog.
|
||||
|
||||
Successful catalogs and `refreshed_at` are stored under `llm_provider_preset:<provider>` in `system_settings`. Dynamic categories use the provider preset as their defaults and are validated before committing. Listing prefers saved catalogs and otherwise shows bundled suggestions. Refresh never modifies the active model, protocol, URL or credentials in `external_integrations`. The frontend retains the draft and previously loaded models when listing fails, with an explicit error message.
|
||||
|
||||
### Model Catalog Endpoints
|
||||
|
||||
These endpoints were checked against official documentation on 2026-09-13. The table lists discovery URLs; the form still takes the generation base URL. Credentials must belong to the configured service region.
|
||||
|
||||
| Provider | Model list GET endpoint | Authentication and parsing |
|
||||
| --- | --- | --- |
|
||||
| [MiniMax](https://platform.minimax.io/docs/api-reference/models/anthropic/list-models) | `https://api.minimaxi.com/anthropic/v1/models`; international: `api.minimax.io` | `x-api-key`; `data[].id`; `has_more/last_id` pagination |
|
||||
| [OpenAI](https://developers.openai.com/api/reference/resources/models/methods/list) | `https://api.openai.com/v1/models` | Bearer; `data[].id` |
|
||||
| [Anthropic](https://platform.claude.com/docs/en/api/models/list) | `https://api.anthropic.com/v1/models` | `x-api-key`, `anthropic-version`; paginate with `after_id` |
|
||||
| [DeepSeek](https://api-docs.deepseek.com/api/list-models) | `https://api.deepseek.com/v1/models` | Bearer; `data[].id` |
|
||||
| [Alibaba Model Studio](https://help.aliyun.com/zh/model-studio/list-models) | `/api/v1/models` on the configured regional host | Bearer; `output.models[].model`; `page_no/page_size/output.total`; filter `capabilities=TG` |
|
||||
| [Moonshot / Kimi](https://platform.kimi.ai/docs/api/list-models) | `https://api.moonshot.ai/v1/models`; China: `api.moonshot.cn` | Bearer; `data[].id` |
|
||||
| [OpenRouter](https://openrouter.ai/docs/api/api-reference/models/list-all-models-and-their-properties) | `https://openrouter.ai/api/v1/models` | Public catalog; Bearer when supplied; `data[].id` |
|
||||
| [OpenCode Go](https://opencode.ai/docs/go/#models) | `https://opencode.ai/zen/go/v1/models` | Public catalog; Bearer when supplied; `data[].id` |
|
||||
| [Ollama](https://docs.ollama.com/api/tags) | `/api/tags` on the configured server | Local servers need no key; `models[].model/name`; empty means no installed models |
|
||||
|
||||
New Model Studio endpoints for Beijing, Tokyo, Frankfurt and Virginia require the actual workspace host, such as `<WorkspaceId>.cn-beijing.maas.aliyuncs.com`. Singapore uses `dashscope-intl.aliyuncs.com`; Hong Kong uses `cn-hongkong.dashscope.aliyuncs.com`. Discovery changes only the path, never guesses a workspace or switches credential regions. If a legacy Beijing host no longer accepts the account, update the base URL from the provider console.
|
||||
|
||||
Pagination and retries have a 30-second overall deadline. Network failures and 502/503/504 allow up to two attempts; authentication failures are not retried. Models are sorted by supplied creation/release dates, preserving upstream order on ties, without arbitrary list truncation. Public catalog membership does not establish account-specific generation permission.
|
||||
|
||||
Admin keeps the AI page aligned with the legacy information architecture:
|
||||
|
||||
- `Model Providers`
|
||||
@@ -283,22 +309,11 @@ Tool keys follow the same rule. WebSearch and OCR must match the current tool an
|
||||
|
||||
### Lightweight Connectivity Testing
|
||||
|
||||
The Admin plug button performs a lightweight connectivity check and does not save configuration. Common API-platform practice is two-tiered:
|
||||
|
||||
- Check a provider catalog or low-cost endpoint to validate base URL, authentication, and model reachability.
|
||||
- Send full model requests only when the user explicitly runs Playground or a business task.
|
||||
|
||||
Connectivity results should be explicit:
|
||||
|
||||
- `ok`: authentication, route, and model catalog are usable.
|
||||
- `warning`: service is reachable, but the current model is missing from the catalog or capability metadata is incomplete.
|
||||
- `error`: authentication, network, protocol, or model lookup failed.
|
||||
|
||||
Toast titles must match the result; failures must not be titled as a successful connection.
|
||||
The Admin plug button checks proxy configuration and queries the current provider catalog. It returns `success=true` only when discovery succeeds and contains the selected model. A 404, authentication error, invalid response or missing model never becomes success based on bundled suggestions. The message distinguishes discovery from generation; Playground or a business task verifies actual generation. Testing never saves the draft or switches the default provider.
|
||||
|
||||
### OpenCode Go Routing Model
|
||||
|
||||
Subscription channels such as OpenCode Go should not be handled by hard-coded frontend model sets. Prefer provider catalog or backend capability discovery that records per-model capabilities such as `chat_completions`, `anthropic_messages`, `models_endpoint`, and whether a subscription key is required. The frontend should display capabilities; the backend should map provider, base URL, model, and adapter into the real request.
|
||||
OpenCode Go protocol mappings are owned by the backend catalog: MiniMax M3/M2.7/M2.5 and the verified Qwen3.6/3.7/3.8 models use Anthropic Messages; GPT-5.6 Luna, Grok 4.6 and Muse Spark Contributor use Responses; other supported models use Chat Completions. The official `/models` response currently supplies IDs only, so new models still require checking the documented endpoint table. Selecting a model updates its protocol in the draft, and backend mappings supersede known stale mappings. Responses uses `input`, `max_output_tokens` and `store=false`, and parses text and reasoning summaries. OpenCode calls carry an application User-Agent and a stable Playground conversation identifier. Generic `thinking=enabled` is translated to MiniMax M3’s `adaptive` format.
|
||||
|
||||
#### Settings Page Behavior
|
||||
|
||||
|
||||
@@ -99,7 +99,7 @@ AIS vessel collectors use a different persistence path from regular `CollectedDa
|
||||
|
||||
Earth boundaries are no longer data collectors. They are Earth static rendering assets: the Earth Assets settings panel owns source configuration, and `/api/v1/earth/boundaries/*` builds `frontend/public/earth/data/boundaries/earth-boundaries-china-pov-v1.pmtiles`. When no high-precision PMTiles artifact is available locally, the frontend uses the bundled low-precision GeoJSON fallback and does not write boundary records to `CollectedData`.
|
||||
|
||||
TOP500 and Epoch AI compute sources do not always provide usable coordinates. The unified Earth compute-center endpoint uses only valid source-provided coordinates or `compute_center_locations` dimension-table coordinates during the main map startup path; records without coordinates are returned as `unresolved` instead of being rendered from a local registry, country centroid, or guessed city. When users manually collect candidates, the backend queries ROR and Nominatim/OpenStreetMap from source fields; accepted candidates are saved into `compute_center_locations` and rendered from that table on the next layer refresh.
|
||||
TOP500 and Epoch AI compute sources do not always provide usable coordinates. Epoch's official CSV `Location` may contain only a country or be empty even when a row has latitude/longitude; it is never treated as a precise site without the required source semantics. The unified Earth compute-center endpoint uses valid source-provided coordinates or `compute_center_locations` dimension-table coordinates during the main map startup path; records without renderable coordinates remain in the complete `unresolved` list instead of being rendered from a local registry, country centroid, or guessed headquarters. `precise`/`site` keep source coordinates; city-level and confirmation-required points may receive only a deterministic display offset keyed by `source` + `source_id`, with the original coordinate untouched. When users manually collect candidates, the backend queries ROR and Nominatim/OpenStreetMap from source fields; accepted candidates are saved into `compute_center_locations` and rendered from that table on the next layer refresh.
|
||||
|
||||
Admin collection management follows the business hierarchy instead of flattening every endpoint into one table:
|
||||
|
||||
@@ -111,6 +111,12 @@ Snapshot lists should not show every snapshot of the same collector as separate
|
||||
|
||||
Credential guides are maintained by `backend/app/services/credential_guides.py`. The console uses read / generate / reset actions to load or create Markdown instructions. The frontend should render the guide Markdown for operators, not expose generation prompts or raw metadata.
|
||||
|
||||
### News Live Stream Catalog
|
||||
|
||||
The IPTV-org adapter in `news_live_streams` retains its news-category filters. Its default `max_sources` is `0`, meaning all matching channels; an explicit positive value still limits collection. Existing configurations retaining the old `120` limit must change it to `0` and collect again to populate the complete catalog.
|
||||
|
||||
`GET /api/v1/tv/streams` uses database pagination in `tv_catalog.py`: `offset` defaults to `0`, `limit` defaults to `50` with a maximum of `100`, and whitespace-separated `q` terms match channel names, providers, regions, and languages. Built-in and configured sources come first. Collected sources are deduplicated by channel ID, exclude configured overrides, and are filtered, counted, sorted, and paginated in SQL. The response's `total` counts matches, while `source_count` counts the complete available catalog; use `has_more` and `next_offset` for subsequent pages. `selected_id` can also return the selected channel outside the current page without consuming its quota. `tv_streams.py` remains the owner of default and fallback sources.
|
||||
|
||||
## IV. Data Format (stored in CollectedData table)
|
||||
|
||||
```python
|
||||
@@ -352,7 +358,11 @@ GET /api/v1/visualization/vessels/{mmsi}/conflicts
|
||||
|
||||
`/api/v1/vessels/snapshot` requires `bbox` and `zoom`, and caps `limit` at `5000`. The Earth frontend uses a global bbox for current state and does not refetch on camera viewport changes. The endpoint reads `vessel_current_state` and reports `diagnostics.source = "vessel_current_state"`. The old `/api/v1/visualization/geo/vessels` route has been removed.
|
||||
|
||||
High-frequency AIS updates must not become per-delta full-layer rebuilds. If Earth uses the `/ws` `vessels` channel, it should send low-frequency reload/dirty hints and let the frontend merge snapshot refreshes. Tracks and conflicts still read historical facts through the single-vessel APIs.
|
||||
AISStream and BarentsWatch share the `/ws` `vessels` delta channel. Earth subscribes with `scope: "global"` on its existing WebSocket, without changing subscriptions as the camera moves. The backend coalesces notifications by MMSI for one second, then reads confirmed `vessel_current_state` rows. Frames contain at most 1000 items; further frames carry the remaining vessels rather than truncating them. Raw source messages must not overwrite confirmed client positions. Deleting current-state rows also produces MMSI-based remove notifications.
|
||||
|
||||
The frontend coalesces short bursts per MMSI and uses `Interactable.updateItems()` to update position and color buffers in place. Heading-bucket changes touch only affected buckets, growing capacity when needed. Existing marker identities, selections, and materials survive. Initial entry, reconnects, deletion hints, and the minute reconciliation still use snapshots without first clearing the layer. A bounded snapshot must not treat truncated vessels as deleted, and older responses must not overwrite newer stream updates or removals received while the request was in flight. Snapshots include query-start `generated_at`; it is compared with stream frame time to prevent stale cached snapshots from rolling back new vessels or removals.
|
||||
|
||||
Ordinary vessel writes use the `delta` strategy on `earth_updates`; while the dedicated channel is connected, they no longer request whole-layer reloads. Deletion or disconnected reconciliation uses `reload` while reusing existing objects. Hiding the layer unsubscribes and clears queued changes. Tracks, conflicts, and audit data continue through the single-vessel historical APIs.
|
||||
|
||||
### Layer APIs And Global Stats
|
||||
|
||||
@@ -450,3 +460,27 @@ POST /api/v1/datasources/trigger-batch
|
||||
The request body may pass `source_ids` for selected rows. Without `source_ids`, the backend filters by `product`, `module`, `is_active`, `run_status`, `collected`, `credential_status`, and `q`. The endpoint skips disabled sources, sources already running without `force`, and sources still inside their frequency window, then returns `triggered`, `skipped`, and `failed` groups.
|
||||
|
||||
**Core file**: `backend/app/api/v1/datasources.py`
|
||||
|
||||
## Compute source contracts and topic APIs
|
||||
|
||||
Epoch uses its official public CSV, retaining theoretical FP16 performance, original dates, status, anonymization/rounding flags, and upgrade relationships. Anonymous rows use content fingerprints plus equal-row ordinals to preserve snapshot multiplicity, without claiming persistent facility identity or assigning coordinates. Unknown countries remain flagged and are excluded from selected-country counts. TOP500 uses system-detail IDs, reads the edition from the list URL, and converts header/cell units into FLOP/s with an explicit FP64 LINPACK basis. Public pages may expose only part of an edition; this does not establish full TOP500 coverage.
|
||||
|
||||
Neither collector returns sample data on failure. Empty fetches or transformed results, schema errors, and losses exceeding 40% of the previous snapshot fail while retaining prior data. Incomplete writes are rolled back. TOP500 location migration copies only a unique exact name and canonical-country match to the new system ID, retaining the historical key; rank is never a migration key.
|
||||
|
||||
`compute_history_defaults.json` ships the original report data. `compute_configuration.py` reads a complete override from the `compute_comparison` category in `system_settings`; absent records use bundled defaults without migrations or manual initialization. `compute_history.py` performs same-year aggregation. Reference periods and publication dates remain distinct, and unknown publication days remain unspecified. `/api/v1/data-products/compute/series` reads configuration on each request with `Cache-Control: no-store`, offering global share, fixed US + China bilateral share and FP32 absolute observations. Innovation gaps remain independent. An empty override stays empty instead of silently restoring observations.
|
||||
|
||||
Under the same prefix, `GET/PUT /configuration` and `POST /configuration/reset` require admin/super_admin. Writes accept `{revision, config}` and reset accepts `revision`. Pydantic validates canonical countries, finite values, share totals, duplicate metric/year pairs and report references; each metric/year has one report basis. PostgreSQL category uniqueness and row locking serialize first writes and updates; stale revisions return 409. Configuration and before/after audit records commit in one transaction. Reset removes the override so subsequent reads use the current bundled defaults. `version` is a configuration revision, while `reviewed_at` is the supplied review date. Admin-curated data is not automatically labeled official. Source URLs are references, not scrape instructions.
|
||||
|
||||
`/api/v1/data-products/compute/facilities` reads successful snapshots with `compute_contract=v1`, applying country filters and per-source counts in SQL. Epoch excludes planned, decommissioned, superseded, and suspected duplicate records; TOP500 means list observations. Cross-source deduplication remains pending. Historical cutoffs use collection time, never pretending a recently fetched old source was available earlier. Geographic results are bounded and report truncation; total record counts are independent of this limit. Official site helpers are keyed by exact `source` + `source_id`; their access-map centers are site-level references only and retain their official URL in `source_note`.
|
||||
|
||||
|
||||
### CelesTrak download windows and caching
|
||||
|
||||
Both active and fallback-group downloads skip HEAD preflight and send GET directly: a preflight may consume the upstream download allowance before the body request. Valid complete cache entries are reused for two hours, then refreshed with conditional GET when validators exist, or ordinary GET otherwise. A 304 reuses validated cache; only the explicit “GP data has not updated” 403 permits a valid-cache fallback. Other 403 errors remain failures. Fallback groups also respect cache age rather than reusing old files indefinitely.
|
||||
|
||||
A nonempty GP record list is validated in `.part` before atomically replacing the cache; failed downloads preserve the previous valid payload. Length comes from the GET response, and compressed wire length is not compared with decoded bytes. Other collectors retain HEAD metadata and Range resume support. Restoring a raw cache does not populate the database: trigger a new collection task afterward.
|
||||
|
||||
|
||||
### Earth news encoding
|
||||
|
||||
Before parsing RSS/Atom, `earth_news.py` selects encoding from the BOM, HTTP charset, XML encoding declaration, then UTF-8, and decodes strictly. GBK/GB2312/GB18030 feeds no longer silently replace Chinese bytes with corrupted text. Unknown encodings or invalid bytes produce `format_error` instead of damaged titles. `test_earth_news_encoding.py` covers titles, summaries, display serialization and failure status.
|
||||
|
||||
@@ -70,7 +70,10 @@ The unified event model is `earth.layer.changed`:
|
||||
| --- | --- |
|
||||
| `clear_then_reload` | Clear local frontend layer objects first, then force a refetch. Prefer this for deletes. |
|
||||
| `reload` | Keep old objects until fresh data returns. Use it for location, metadata, or non-destructive updates. |
|
||||
| `delta` | Used only for `earth_interactables`; upsert or remove objects by id. |
|
||||
| `delta` | `earth_interactables` upserts/removes by id; ordinary vessel writes update confirmed state by MMSI through the dedicated `vessels` channel without clearing the layer. |
|
||||
|
||||
Vessel deletion still emits a `reload` reconciliation hint; individual `vessel_current_state` deletions also enter the vessel remove channel. Source notifications identify changed MMSIs, while transmitted values come from current state. Global subscriptions use `scope: "global"`; message-size limits split frames instead of discarding remaining vessels.
|
||||
|
||||
|
||||
APIs must return HTTP 200 with an empty collection for real zero-data states; 5xx is reserved for real endpoint failures. After a delete event, if refetch fails, the frontend should keep the cleared state and show a lightweight error instead of restoring stale objects.
|
||||
|
||||
|
||||
@@ -321,7 +321,7 @@ The new vessel list entry point is no longer the legacy `/api/v1/visualization/g
|
||||
GET /api/v1/vessels/snapshot?bbox=-180,-85.05112878,180,85.05112878&zoom=12&limit=3000
|
||||
```
|
||||
|
||||
That endpoint reads `vessel_current_state`, returning the latest point per MMSI inside the freshness window. Raw `ais_raw_observations` remain available for tracks, audit, and situational analysis, but the display endpoint no longer scans and aggregates history on the fly. Earth sends a global bbox rather than the current camera viewport. If the `/ws` `vessels` channel is connected, it should act as a reload/dirty hint for merged refreshes, not as a per-AIS-delta full-layer rebuild path.
|
||||
That endpoint reads `vessel_current_state`, returning the latest point per MMSI inside the freshness window. Raw `ais_raw_observations` remain available for tracks, audit, and situational analysis, but the display endpoint no longer scans and aggregates history on the fly. Earth sends a global bbox rather than the current camera viewport. Realtime updates use the global `/ws` `vessels` subscription to deliver canonical state by MMSI and update existing markers in place. Larger updates are split across frames without dropping vessels. Initial load, reconnection, and deletion reconciliation still use snapshots; see [Collector Architecture](backend-collectors.md).
|
||||
|
||||
## Custom REST / WebSocket Mapping Runtime
|
||||
|
||||
|
||||
@@ -1,5 +1,21 @@
|
||||
# Earth Frontend Context
|
||||
|
||||
## HD surface and atmosphere materials
|
||||
|
||||
The visible HD surface uses a single-draw ShaderMaterial in `earth-surface-material.js`, retaining the R+0.48 mesh, depth policy and surface pick target. The existing controller still switches all four 8K maps, updates the sampler and disposes the previous texture only after success. Normals, view direction and astronomical sunlight share world coordinates. Land is matte; water has grazing-angle cool reflection without a white specular lobe. The existing geographic land/ocean texture owns the optical mask independently of border visibility, with a conservative chroma fallback until loaded and reflection suppression over ice. Original JPEGs are neither repainted nor downsampled.
|
||||
|
||||
Color textures are tagged sRGB with mipmaps. The r128 surface shader decodes sRGB; r160 uses hardware texture decoding. Both combine light in linear space and encode once. A local filmic curve affects only the surface instead of enabling global ACES exposure on the sky and data layers. The terminator spans normal dot products [-0.075, 0.16]; night shading has its own geographic texture ghosting and cool ambient branch. A 0.55 power of the locally tone-mapped texture retains shadow variation, followed by separate muted land and deep-indigo ocean tints and low ambient floors. Day/night mixing remains linear with one final sRGB output. Day exposure and reflections are unchanged; no synthetic city lights are added. One R×1.025 BackSide/Additive shell replaces the three rim shells, with depthWrite=false and depthTest=true; inner scattering is part of the surface pass. Independently switchable clouds use matte day/night shading at renderOrder 1.1. The sky uses the existing `starmap_deep_8k.jpg` without a tint overlay.
|
||||
|
||||
## Configurable compute observations
|
||||
|
||||
Compute cards read annual observations from `/api/v1/data-products/compute/series`; the admin Compute comparison section is the maintenance entry point. `compute-topic.js` fetches with no-store, listens for same-origin `planet-compute-configuration` notifications, and checks every 60 seconds while visible and enabled, plus when the page becomes visible. Observation refresh does not reload facility layers. `destroy()` releases the timer, channel and visibility listener. Country membership, full regional coverage and missing-value semantics remain backend-owned.
|
||||
|
||||
## Surface texture provenance
|
||||
|
||||
On 2026-09-21, `assets/8k_earth_daymap.jpg` was replaced with an 8192×4096 Web derivative of the paired Natural Earth I raster. Its loading path and rendering code remain unchanged. The source is a 16200×8100 EPSG:4326 image; the previous JPEG is preserved byte-for-byte under `resources/earth/archive/2026-09-21/`. Source/license records, the catalogue versus embedded-version discrepancy, SHA256 hashes, JPEG parameters, and recalculated world-file metadata live in `resources/earth/natural-earth-1/`. Large originals remain in Git-ignored `downloads/earth/`, outside the frontend release bundle.
|
||||
|
||||
|
||||
Basemaps and TV channels share `hud-select-menu.js` for the HUD popover, list, selection mark, keyboard navigation, and positioning; `tv-source-menu.js` only adapts channel copy. Basemaps disable search, pagination, and counts. The trigger owner identifies popover clicks as internal to the mobile drawer.
|
||||
This document describes the current real structure of the Earth display frontend. The focus is on helping future changes to the HUD, layers, media panel, real terrain, and BGP visualization avoid repeating past structural and state-sync pitfalls.
|
||||
|
||||
Related references:
|
||||
@@ -159,6 +175,14 @@ Each module is responsible for its own:
|
||||
|
||||
`tv.js` owns the live / aggregation-news tabs inside `media-panel`. Toolbar open and tab-switch actions write back through `earth:tv-visibility-change` and `earth:tv-tab-change`: panel visibility remains viewport-scoped at `views.<scope>.panelVisibility.media-panel`, while the active tab is stored at `shared.mediaPanelActiveTab`. Refreshing the page therefore restores the user's last live/news state. Temporary hides from `closeTransientMobileOverlays()` carry `persist:false` and do not overwrite the preference.
|
||||
|
||||
`tv-source-menu.js` reuses HUD and legend-list styling in a popover with search, a scrolling list, and a fixed count footer. `tv.js` requests 50 entries from `/api/v1/tv/streams` and caches channel details; search runs against the complete backend catalog, and pagination uses `next_offset`. Cancellation and a request generation counter prevent stale results from replacing a newer search. Catalog refresh uses `selected_id` to restore a channel outside the first page.
|
||||
|
||||
`tv-playback-policy.js` owns live recovery decisions: all non-fatal HLS events remain with the player, while fatal network/media failures receive at most three recovery attempts. Switching requires automatic fallback to be enabled, no explicit user channel selection, and an embeddable backup. Async callbacks in `tv.js` verify player identity and channel; native video errors no longer bypass HLS recovery. `tv_streams.py` replaces the old bundled CCTV-4 CDRM URL, including saved copies, with WCETV's America-edition HLS stream, names the edition explicitly, and preserves custom streams. An HTTP success does not establish video decodability.
|
||||
|
||||
Sources with verified CORS support may explicitly set `use_proxy:false` for direct browser playback without segment forwarding. Other sources continue to use the TV proxy by default. The bundled WCETV source uses direct playback.
|
||||
|
||||
Source information reuses `createHUDPanel` and its existing 0.22-second CSS transition. The shared controller measures content into `--hud-body-content-height`, avoiding the non-interpolable change from `max-height: none` to zero. ResizeObserver updates content measurements and compensates the top position of upward-expanding dragged panels during animation, keeping their bottom edge fixed without disabling the body transition. The shell's top transition is paused during compensation to prevent drift from double interpolation.
|
||||
|
||||
`brand.js` manages Earth HUD brand resources. Static assets provide the default brand; runtime overrides come from `/api/v1/earth/brand`, and uploaded images are served from `/earth-brand-assets/...`. The frontend must treat logo/title images and text fallback separately: if an image fails, show the text title; if text fields are empty, rely on backend defaults so the HUD brand area never renders blank. The console Earth Content page owns saving and resetting brand configuration; the Earth frontend only consumes it.
|
||||
|
||||
`about.js` manages the About card inside Earth settings. Frontend defaults remain as a fallback, while runtime content is loaded from `/api/v1/earth/about`. If the request fails or fields are missing, the renderer must fall back per field so the settings page never renders an empty card. Admin exposes an Earth Content `About` tab; saving uses `PUT /api/v1/earth/about`, and restoring defaults uses `DELETE /api/v1/earth/about`.
|
||||
@@ -179,6 +203,8 @@ The compute-center layer row has a notification badge for GeoJSON `unresolved` r
|
||||
|
||||
Location candidate state in the details card is cached in [info-card.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/info-card.js) by `entityType:entityId`. If the user closes the details card or unresolved queue and reopens the same compute center / BGP collector, previously collected candidates and status text are restored. Header-level `一键采用` prefers cached candidates, avoiding repeated online geocoding or LLM factcheck calls. After a location is saved, that entity's candidate list is cleared to a "refreshing layer" status so stale candidates do not keep misleading the user.
|
||||
|
||||
`runUnresolvedComputeCenterBatch()` owns a queue of entity contexts rather than panel DOM. `locationCollectStateCache` retains candidates, progress, and save results so rebuilt cards can hydrate the current state. `earth:compute-center-location-batch-change` updates the badge, `savedCandidate` excludes completed entries, and a ✓ entry remains after completion. Queue state lives only in the current Earth page: refresh, close, or route navigation interrupts unfinished work, while saved coordinates remain authoritative in the backend.
|
||||
|
||||
The `预览 / 保存` buttons on each candidate row use a single delegated `click` handler per candidate root (the `[data-collect-cache-key]` block in the details card, or `[data-unresolved-item]` in the unresolved queue), guarded by a `data-candidate-actions-bound` flag so it cannot be double-bound. Direct `pointerup` / `click` listeners on individual buttons and overlapping delegated handlers were removed. Candidate objects are no longer JSON-stringified into an HTML attribute and parsed back; buttons only carry `data-candidate-index`, and the handler resolves the candidate object from a module-level `Map` keyed by cache-key. This removes the entire class of failures caused by HTML entity escaping of `&` / `<` / `"` in candidate fields. Clicking `预览` dispatches `earth:preview-location-candidate`; `main.js`'s `previewLocationCandidate()` calls `showComputeCenterLocationPreview()`, which attaches a hollow breathing-ring sprite pair at the candidate coordinates (visually mirroring the BGP event ring) and focuses the camera on the candidate. Previewing another candidate replaces the ring; saving clears it and `spawnSavedComputeCenterLocation()` immediately spawns the formal compute-center interactable. Note that `main.js` has no module-level `earth` variable — every location-save / preview handler must call `const earth = getEarth();` first, otherwise the event handler throws a `ReferenceError` that the surrounding `.catch` swallows, producing the failure mode where the button "does nothing".
|
||||
|
||||
The `earth:compute-center-location-saved` reconciliation pipeline is deliberately silent on background-refresh failures. `spawnComputeCenterAfterLocationSave()` already presents the success toast and locked state; `refreshComputeCentersAfterLocationSave()` only reloads backend data when the scene is ready and no longer emits its own `已保存` toast. `handleComputeCenterLocationSaved()` runs refresh in the background after a successful spawn; only when spawn returns `null` (scene not ready) or throws does refresh take over the success toast. A refresh error is only `console.warn`'d — it must never surface as a `保存失败` message, because the save itself succeeded and the refresh is a follow-up sync.
|
||||
@@ -191,7 +217,11 @@ The backend snapshot endpoint still requires `bbox`, but the Earth runtime treat
|
||||
|
||||
Vessel markers are rendered through `createInteractableLayer()` as batched `THREE.Points`, with `cluster` and `avoidance` explicitly disabled. Dense waterways may overlap. Dragging or inertial rotation skips hover picking; normal hover uses screen-space nearest-point picking. Do not reconnect vessels to dynamic screen clustering or per-frame Points rebuilds, because those interaction costs are what make a 3000-marker layer feel heavy.
|
||||
|
||||
If the `/ws` `vessels` channel is used by Earth, it should be a low-frequency reload/dirty hint only. Do not create multiple viewport subscriptions, and do not turn every AIS delta into a full layer rebuild.
|
||||
AISStream and BarentsWatch share the `/ws` `vessels` delta channel. Earth subscribes with `scope: "global"` on its existing WebSocket, without changing subscriptions as the camera moves. The backend coalesces notifications by MMSI for one second, then reads confirmed `vessel_current_state` rows. Frames contain at most 1000 items; further frames carry the remaining vessels rather than truncating them. Raw source messages must not overwrite confirmed client positions. Deleting current-state rows also produces MMSI-based remove notifications.
|
||||
|
||||
The frontend coalesces short bursts per MMSI and uses `Interactable.updateItems()` to update position and color buffers in place. Heading-bucket changes touch only affected buckets, growing capacity when needed. Existing marker identities, selections, and materials survive. Initial entry, reconnects, deletion hints, and the minute reconciliation still use snapshots without first clearing the layer. A bounded snapshot must not treat truncated vessels as deleted, and older responses must not overwrite newer stream updates or removals received while the request was in flight. Snapshots include query-start `generated_at`; it is compared with stream frame time to prevent stale cached snapshots from rolling back new vessels or removals.
|
||||
|
||||
Ordinary vessel writes use the `delta` strategy on `earth_updates`; while the dedicated channel is connected, they no longer request whole-layer reloads. Deletion or disconnected reconciliation uses `reload` while reusing existing objects. Hiding the layer unsubscribes and clears queued changes. Tracks, conflicts, and audit data continue through the single-vessel historical APIs.
|
||||
|
||||
The legacy `/api/v1/visualization/geo/vessels` route has been removed. Frontend code should keep using `PATHS.vesselsApi` and can verify the current-state path through `diagnostics.source == "vessel_current_state"`.
|
||||
|
||||
@@ -221,6 +251,14 @@ The cruise sequencer handles generic logic: current target, queue order, camera
|
||||
|
||||
All material, layer, satellite, BGP, cable, terrain, celestial, and other style parameters are maintained here. Do not scatter magic numbers in module files.
|
||||
|
||||
## Render hot paths
|
||||
|
||||
`cable-batches.js` batches the complete cable and landing-point sets. `cables.js` still owns original business objects, picking, occlusion, and selection state. Do not make these interaction proxies draw individually again or return a render batch as the detail-card business object. In dense cable areas, screen-distance picking compares satellites, BGP, compute and other point objects before using cable lines as the fallback; hover and click share the same ordering so small objects threaded through a line bundle remain selectable.
|
||||
|
||||
Satellite breathing runs in vertex shaders. `satellite-position-worker.js` moves full SGP4 position and initial trail calculations off the main thread; the main thread applies snapshots to interaction coordinates and render buffers. `satellite-propagation.js` supplies the same mathematics to the Worker, predicted orbits, and synchronous fallback. Reload, clear, and altitude changes must terminate prior work so stale snapshots cannot resurrect cleared layers.
|
||||
|
||||
The `i18n.js` MutationObserver translates only added subtrees, changed text, or changed attributes. Local clocks, status labels, and download progress must not synchronize all locale controls. Global locale synchronization belongs to initialization, locale changes, or subtrees containing new locale controls.
|
||||
|
||||
## Current Style Layers
|
||||
|
||||
CSS files in `frontend/public/earth/css/` each correspond to a specific component scope. Do not write global Earth styles into `base.css` unless they genuinely apply to everything.
|
||||
@@ -252,7 +290,7 @@ The surface hover tooltip preference is persisted by `controls.js` as `shared.su
|
||||
|
||||
The real satellite altitude preference is persisted by `controls.js`, while the rendering state lives in `satellites.js`. When enabled, the real radius from SGP4 is compressed logarithmically into the current Earth visual radius range. When disabled, satellite dots, trails, and predicted orbits all return to the legacy same-sphere display. Toggling this setting must refresh satellite positions and clear trail buffers so a trail never mixes both height models. `maxRealAltitudeOffset = 25` is a visual cap tuned for the current camera and `earthRadius = 100`: GEO / MEO remain clearly higher than LEO, but the highest orbits stay within about 25% beyond the globe radius so selection targets, red trails, and the globe do not feel disconnected.
|
||||
|
||||
SGP4 propagation returns an inertial-frame position, so it must not be drawn directly as Earth-fixed longitude / latitude. `satellites.js` uses `gstime` to convert ECI/TEME positions to ECF, then maps that result into the same Three.js axes as `latLonToVector3()`. Satellite dots and short trails use Earth-fixed coordinates for each sample time, representing the object's current position relative to the globe surface. The locked predicted orbit uses the `gstime` from the lock moment for the whole future orbit, projecting the inertial orbit plane onto the current globe pose; that keeps the line closed and keeps the visual orbit inclination aligned with the details card. Fallback predicted orbits must also use a real RAAN + inclination orbital-plane formula, not treat inclination as a constant latitude.
|
||||
SGP4 propagation returns an inertial-frame position, so it must not be drawn directly as Earth-fixed longitude / latitude. `satellites.js` uses `gstime` to convert ECI/TEME positions to ECF, then maps that result into the same Three.js axes as `latLonToVector3()`. Satellite dots and short trails use Earth-fixed coordinates for each sample time, representing the object's current position relative to the globe surface. The locked predicted orbit uses the `gstime` from the lock moment for an open full-period prediction curve, projecting the inertial orbit plane onto the current globe pose. No first vertex or closing chord is appended; arc-length highlights travel toward increasing prediction time, and visual inclination stays aligned with the details card. Fallback predicted orbits must also use a real RAAN + inclination orbital-plane formula, not treat inclination as a constant latitude.
|
||||
|
||||
Boundary precision is stored separately by `country-boundaries.js` under `planet.earth.boundaries.highPrecisionEnabled`. When high precision is off, Earth keeps using the bundled low-precision `countries-admin0.min.geojson` fallback even if high-precision manifest/PMTiles files exist locally. When high precision is on but the artifact is missing, the Earth toolbar settings call `/api/v1/earth/boundaries/build` and poll progress. After success, `reloadCountryBoundaries()` hot-swaps the boundary layer without refreshing the page. Boundary hover is independent from interactable hover: a country polygon remains highlighted whenever the surface coordinate is inside it, while the tooltip can still prioritize a satellite, vessel, BGP marker, or other interactable.
|
||||
|
||||
@@ -333,7 +371,9 @@ If future cable, satellite, or news cruise is added, do not copy a new set of `m
|
||||
|
||||
[controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js) owns the Earth zoom state, and every zoom entry point must ultimately call `setZoomLevel()` to write the camera distance. Do not write `camera.position.z` from other modules, or the zoom percentage, drag sensitivity, and Interactable clustering thresholds will diverge again.
|
||||
|
||||
Interactable clustering is selected per layer through `cluster.strategy`. `stable-spherical` uses discrete zoom bands and local 3D bucket clustering, so BGP, compute centers, and Earth interactables do not regroup while the globe rotates inside the same band. `dynamic-screen` keeps the projection-based behavior for high-frequency realtime layers such as vessels, and `none` disables clustering. Stable cluster dots stay rigidly aligned to their 3D centroid projection and do not participate in 2D avoidance. See [Earth Interactable Clustering](/home/ray/dev/linkong/planet/docs/technical/en/earth-interactable-clustering.md) for strategy configuration and tuning.
|
||||
`globe-navigation.js` retains the safe zoom-distance mapping. Mouse and single-touch drags uniformly use legacy screen deltas, 0.18 smoothing and release inertia, including movement outside the globe silhouette; ray/sphere intersection no longer gates dragging. Normal zoom retains legacy sensitivity. Once the globe exceeds the viewport, a smooth surface-distance/viewport cap limits excessive rotation at extreme zoom. Horizontal motion uses geographic yaw, vertical input changes tilt, and quaternion math prevents multi-revolution reversal. Pinch transitions and pointer capture remain intact.
|
||||
|
||||
Surface markers retain category icons and fixed geographic cluster membership. Numbered solid circles expand through a 320ms tween above 250% zoom; zoom never regroups members, and 500% has no aggregate badges. Vessels retain independent heading arrows without clustering. See [Earth Interactable Clustering](earth-interactable-clustering.md).
|
||||
|
||||
Wheel input has two paths. Traditional mouse wheels keep the 10% step and short animation, using `wheelZoomTarget` as the logical base for continuous wheel input. Trackpads and high-precision wheels use the pixel delta for continuous zoom and call `setZoomLevel()` directly instead of passing through the 10% stepped animation. The trackpad path also filters a short-window, old-direction residual delta after a real direction change so inertia tails do not pull a just-reversed zoom back in the previous direction.
|
||||
|
||||
@@ -369,3 +409,186 @@ Therefore:
|
||||
For console structure, see:
|
||||
|
||||
- [Admin Frontend Context](/home/ray/dev/linkong/planet/docs/technical/en/frontend-admin-frontend-context.md)
|
||||
|
||||
## Spatial compute cards
|
||||
|
||||
`compute-topic.js` owns periods, measures, participants, and evidence. `compute_catalog.py` supplies canonical names, picker groups, and regional label anchors for 45 currently registered participants. Unknown historical values remain missing. APIs are no longer limited to CN/US. Group labels never become overlapping statistical denominators, and regional coordinates must not become facility locations.
|
||||
|
||||
Earth's existing Roboto Condensed face is now served locally from `assets/fonts/`, with its source and license in that directory and the shared declaration in `css/fonts.css`. The remote Google Fonts stylesheet no longer blocks the page; font family and weights are preserved.
|
||||
|
||||
`spatial-callouts.js` provides geographic projection and shared scopes; `spatial-callout-controller.js` owns stable peripheral reading slots. `callout-dock-layout.js` treats the central observation area as a hard obstacle, retains valid placements first, then places new or affected cards. Rotation, front/rear changes, connector crossings and length do not request reflow. Only card-list changes, deliberate expand/drag actions, viewport changes and HUD-space changes do. Limited space preserves full cards and permits peripheral overlap, without automatic folding, grouping or font reduction. settleCardDrop validates the released position: valid drops stay, while out-of-bounds or central drops settle to a nearby valid position. Peripheral overlap remains allowed. Persistence records the committed destination, and hovering cannot pause a correction inside the center. Restored positions use the same validation; manual reset controls and handling are removed. Starting a drag raises that card above all regional cards and retains its priority until another card is dragged. Compute and news cards share stacking levels; configuration and evidence panels remain above them.
|
||||
|
||||
Object presentation is shared through `PresentationController` and the new object-callout anchor/layout path. A click on a BGP, compute, vessel, satellite, or cable object draws the connector before placing the adjacent card. The source anchor begins at the actual click or original region label, then follows a visible highlight geometry sample as rotation approaches occlusion. The stable `layoutReferencePoint` is derived from the actual connector exit direction, so `packDirectionalEdgeCallouts` keeps cards on the corresponding screen edge and leaves the globe's center open; a card may move inward only when that edge lane is unavailable, and it may never cover its own connector exit point. The bounded joint assignment considers up to 24 beam states and 32 candidates, avoids existing HUD rectangles, then allows at most three safe exchanges to reduce directional crossings; the previous solver remains a fallback and the work is not a full relayout every frame. A slow source-direction gate also checks whether the actual card target reverses more than 90 degrees from the source before committing a change. Manual drag and hover locks take priority. The source point itself remains a per-frame visual connection target; fixed objects track live positions while cards remain draggable. Rear-side objects retain a globe-outline direction cue, a subdued connector, and a hollow source point; these are orientation guides rather than geographic points. Country candidates are capped at 96 with a hard limit of 128 and cached, so GeoJSON is not scanned every frame. Compute cards keep their original point independent from dynamic source-point movement, and the shared card endpoint slides along the obstacle perimeter. Cable presentation prefers the true ray-hit local point, then samples highlighted geometry by `cableId`. Mobile keeps the existing mini popup and opens drawer details. A stale asynchronous presentation must never dismiss a newer selection.
|
||||
|
||||
`configVisible` and `cardsEnabled` are separate. X/Escape closes only configuration; disabling the switch releases connectors and restores the previous map query/layer state. Configuration and source windows bind HUD typography directly so reparenting cannot enlarge their text. The main card presents the selected compute metric as a large share percentage or an absolute value with unit, observation year, historical context, and source entry. Missing comparable data is explicit; no empty trend chart is rendered without observations. Facility records and coverage remain complete in the expanded auxiliary area, where sample counts are evidence context rather than a compute value or global coverage rate. Long lists open only when needed. Mobile preserves the same single metric meaning in its accordion. Local position/size persistence and reset-to-auto remain available. Stable DOM, dimensions, and reading state survive updates, and stale responses cannot re-enable cards.
|
||||
|
||||
Globe dragging and inertia only pause card motion progress; they never replace committed destinations. Connector anchors follow geographic projection independently of card placement.
|
||||
|
||||
The shared `CalloutConnector` calculates polyline length from its points instead of reading `getTotalLength()` after each SVG write, avoiding repeated synchronous layout during motion. Initial draw animation is retained, so existing cruise presentations receive the same optimization.
|
||||
|
||||
On mobile, compute uses a compact regional overview in the existing drawer. Rows are an in-place accordion: tapping the same header collapses it, tapping another collapses the previous row, and at most one expanded card is shown. The expanded card keeps trends, region members, facilities, media, and sources; there is no region subpage, while Settings remains a separate on-demand page. The shared HUD collapsible panel animation is reused and respects reduced motion. Summary loading remains active when map cards are off; enabling cards shares the map loading event and cancels stale requests. Drawer handlers use `composedPath()` so inner HTML replacement cannot collapse the active row. Facility names remain in a scroll region capped at 240px, with left-aligned wrapping rows and at least 42px touch height.
|
||||
|
||||
`/series` retains source-backed annual observations; the default primary metric is global compute-device scale split into basic, intelligent, and supercomputer capacity for 2020–2023. Reviewed absolute observations are 302 for 2022 (`caict2023`) and 435 for 2023 (`caict2024`); Japan's global share is 6%/5%/4%/4% for 2020–2023. Product groups APAC excluding China and Other do not map directly to report Japan or “other”; Europe remains `null` until membership aligns with the report. Bilateral share keeps a fixed US + China denominator. Missing primary values may show the latest reported year (currently 2023), while an explicit 2026 selection stays missing. Unconfirmed comparability does not produce year-over-year change. `/facilities` filters and counts unique canonical country codes in successful collection snapshots; coverage counts remain auxiliary, and are not capacity or a global coverage rate. `compute_catalog.py` groups China, US, Europe, APAC excluding China, and Other, covering 45 canonical country codes; these display groups never become statistical denominators. `earth_geography_policy.py` shares display ownership with map policy, grouping Taiwan/Hong Kong/Macao source subdivisions under China while retaining `source_country`. API review metadata is `reviewed_at=2026-09-22` with the matching version date. An independent Epoch share mode is not implemented. Innovation differences are supplied only when both CN and US are selected.
|
||||
|
||||
### Boundary surface attachment
|
||||
|
||||
Normal borders, coastlines, claim lines, highlights and landing points use `earth-surface.js` to share the HD sphere UV-triangle surface. Borders draw after the texture but before landing points. Highlight lines are always transparent-queue objects with depth testing, preventing through-globe display or overpainting markers. Geometry is computed and cached at construction, not reprojected per frame. See the [render-layer specification](earth-render-layer-order.md).
|
||||
|
||||
### News ticker edge fading and breaking state machine
|
||||
|
||||
A separate pseudo-element paints the ticker background and horizontal rules, fading to full transparency outside the text bounds. The shell has no rectangular backdrop filter, box shadow, or text mask. The region label sizes to its content and the scrolling text follows directly without fading.
|
||||
|
||||
The previous breaking rule was global shell promotion: if the payload contained any unexpired breaking story, the highest level (`critical > breaking > watch`) was applied to the ticker or expanded panel. That made a closed panel, or a ticker currently showing a normal story, retain an orange/red state. `breaking_expires_at` eventually cleared the state, while selection, opening and closing did not change the underlying breaking data.
|
||||
|
||||
The current state machine keeps data, ticker visibility, panel lifecycle and selection separate:
|
||||
|
||||
- `getNewsBreakingState()` derives the data state from `items` and `cruise_items`, ignores expired stories, and ranks `critical > breaking > watch`. The expiry timestamp only schedules the next recomputation.
|
||||
- A ticker viewport watcher runs every 120 ms and chooses the item with the greatest viewport overlap, breaking ties by distance to the viewport center. Only when that current item has a non-`none` level does the ticker switch its horizontal rules to `has-breaking-*`; it returns to neutral as soon as the current item is not breaking.
|
||||
- The panel lifecycle is `closed → opening → open → closing → closed`. Opening and closing control morphing, visibility and ticker hiding; they do not mutate the breaking data. During `opening/open`, the current card DOM stays stable while incoming payloads are deferred and applied once on close, so realtime updates cannot rebuild a hovered card.
|
||||
- `selectedCruiseStoryId` is independent of breaking severity. Selection, focus, hover and close only change the selected story and reading position; close also clears the selection and filter popover.
|
||||
- The expanded panel always keeps its blue glass surface. Breaking colors remain on the matching story cards, so opening the panel never turns the entire glass panel red; the ticker can still show a red signal when the currently scrolling story is breaking.
|
||||
|
||||
The glass-reflection pulse is implemented: when the current ticker story enters a breaking state, its `::after` emits a low-frequency, limited-amplitude full-width horizontal red/amber sweep and pulse; the matching breaking card gets a single glass reflection sweep through its own `::after`. The panel base remains blue and there is no permanently flashing full border. `prefers-reduced-motion` disables the animation and keeps a low-intensity static cue. This visual sub-state stays attached to the current breaking story and matching card, independent of whether the panel is open.
|
||||
|
||||
Compute cards directly reuse the shared `.hud-panel` background, border, shadow, and glass treatment. Summary, essentials, and detail areas add no separate black surface. Their headers share inset spacing with the layer panel through `.hud-panel__header--inset` / `.layer-panel-header`; primary numbers remain prominent and essential information stays visible.
|
||||
|
||||
|
||||
### First frame and spatial card entrance
|
||||
|
||||
`earth-runtime-plugin.mjs` bundles local Earth modules into content-hashed entries and shared chunks for both development and production. The existing import map still pins third-party versions; relative assets and workers retain the `/earth/js/` base. `earth-preload.js` requests the selected full-size map early. `earth-texture-image.js` decodes an ImageBitmap asynchronously and transfers that same image to Three.js without resampling or changing texture color/sampling settings. Replaced and destroyed textures release their bitmaps. The full-resolution surface precedes independent concurrent data layers; data refresh retains it. Versioned boundary geometry, picking indexes and claim lines use HTTP caching and parallel requests. The live player engine is imported on demand; media and realtime connections start after the surface frame.
|
||||
|
||||
The root attributes `data-earth-first-frame-ms` and `data-earth-surface-ready-ms` record navigation-to-render timings. Measure cold and warm caches separately; local warm results are not a guarantee for all first-visit networks.
|
||||
|
||||
Positioned cards stay hidden while the shared connector draws for 420 ms, then fade in using the HUD 180 ms opacity timing. Targets remain fixed during drawing; subsequent motion does not replay entrance. Reduced-motion preferences skip this animation.
|
||||
|
||||
Object clicks, search focus and gesture confirmation share `PresentationController`. Cable selection updates selection/legend state without independently opening the card. InfoCard reveal callbacks are versioned so staging or hiding invalidates older callbacks. Connector completion gates card reveal; cancelled cruises no longer hide a newer selection. Object details reuse the regional-card edge candidates and collision scoring, with dimensions measured while staged. Selecting a mobile search result collapses the search drawer so the focused object, connector and compact card remain visible.
|
||||
|
||||
### Satellite particles and smooth rendering
|
||||
|
||||
Satellites use one `THREE.Points` batch with `PointsMaterial`: a 32×32 white disc texture of radius 0.42, `alphaTest=0.5`, and `sizeAttenuation=false`. The dark backdrop batch and simulated sphere lighting are removed. `satellite-display.js` calibrates against satellitemap screenshot pixels: point squares are 2.4px at zooms up to 150% and 8px at 500%, with a smoothstep transition. The disc fills 84% of the square, yielding visible cores of about 2px and 6–7px respectively. Blue/orange/green/purple/rose use the reference palette directly without the prototype's extra exposure. Point sizes retain the DPR-cap-2 adjustment without reducing globe resolution. Ordinary breathing defaults off but remains an opt-in setting. Existing hover/selected rings, halos, open predicted orbits and forward-flow animation remain intact.
|
||||
|
||||
Real GP/TLE propagation stays in the Worker at 250ms intervals. `satellite-motion.js` caches sample radii and interpolates radius-preserving positions each frame, shared by points, picking, trail tips, selection rings and coverage. `satellite-trail-geometry.js` merges redundant ribbon segments only when the projected error remains below 0.2 pixels across supported zooms. All satellites and 10 history samples are retained; curved or near-camera trails keep their original segments. Only active buffer ranges are uploaded. Trail width is capped at 60% of the point size and tapers toward the tail, avoiding a thick fragmented rim at distant views. `satellite-footprint-geometry.js` retains the full 260×170 coverage grid, created once. The vertex shader evaluates the same surface projection, while local basis and extent uniforms update each frame instead of rebuilding tens of thousands of vertices. Starlink/Iridium footprints share blue `#3399ff`; sunlight changes opacity only, with real coverage models preserved.
|
||||
|
||||
`SATELLITE_CATEGORIES` owns real inclination categories and the blue/orange/green/purple/rose/gray palette. Selecting a satellite automatically switches to its legend. Legend categories directly control points, trails, labels and picking; hiding a selected category clears selection and coverage. Layers and settings have no separate filter entry. Explicit search selection re-enables its category. `satellite-labels.js` shows actual names and models (NORAD identifiers when unavailable) at the configured zoom threshold (**400%** by default, adjustable from **250% to 500%**). The slider below Close-up labels sets a persisted desktop/mobile count limit from 5 to 36 (default 36), immediately rescanning candidates on change. Actual visible counts still depend on available positions and occlusion. It checks occlusion and overlap in the central 65% of the view, scans candidates every 160ms and follows active labels each frame.
|
||||
|
||||
Settings → Display → Show FPS persists through `shared.showFps`, defaulting off. `frame-stats.js` displays actual frame-interval statistics beside the brand, updating once per second; its native tooltip contains mean interval, P95, main-thread CPU time, draw calls and an approximately 30-second rolling mean FPS (total frames / total duration, not an average of per-second rates). Slow foreground frames remain in the sample; visibility changes reset the sampling window. A single high-FPS sample is not a sustained-performance guarantee. Performance acceptance uses an actual canvas of at least 1920×1080, samples at least 30 seconds and targets a mean of 60 FPS while allowing occasional low frames. Compare distant and 400% views with matching viewport, complete data, layers and playback state, and verify category hiding, search selection and FPS persistence after reload.
|
||||
|
||||
The boundary loading tint, land fill, base and HD layers share the `earth-surface.js` geometry factory. Tint and fill share R+0.32, fixing pre-texture diamond rings without changing boundary/HD controls. Intersection and far-zoom depth margins are covered by `earth-surface.test.js`; see [render layer order](earth-render-layer-order.md).
|
||||
|
||||
Cables and landing points share HD-surface coordinates through `cable-surface.js`. `cable-batches.js` controls related halos with per-instance emphasis cleared on deselection. `interactable.js` uses `surface-marker-occlusion.js` for whole-badge visibility and picking near the limb. BGP change events retain their original icons; collectors use activity-colored broadcast badges. Sky material uses the original brightness (multiplier 1.0).
|
||||
|
||||
`toolbar-hover.js` owns the zoom submenu and radial toolbar as one state machine. Magnifier hover opens zoom; trigger, transparent gap bridge and vertical controls form a continuous keep-open region. Leaving first runs the submenu's 220ms exit, then checks the latest pointer location and pinned state before collapsing the parent. Remaining in the radial wake region or pinning the parent closes only zoom; reentry cancels stale timers. CSS hover/focus-within no longer owns visibility independently. Zoom clicks retain parent pinning, with keyboard and touch entry preserved.
|
||||
|
||||
The radial menu inherits global `--hud-scale` (1920×1080 reference, clamped to 0.7–1) instead of fixing its scale at 1. Buttons, icons, fan radii, bottom spacing and the zoom submenu scale together. Base fan coordinates are scaled in CSS transforms; hover wake bounds use the same coordinate scale to prevent hit-region drift under DPI changes. Scaling changes geometry only, preserving animation timing and pinned/hover state transitions.
|
||||
|
||||
### Sky color output
|
||||
|
||||
The star map is decoded as sRGB. `celestial.js` uses shared `preserveTextureOutput()` to encode only the sky material back to display sRGB. The Three.js r128 renderer retains its existing global linear output so Earth, satellites and other layers remain unchanged. `skyBrightness=1.0` retains source brightness; missing output encoding displays decoded dark values directly and cannot be corrected by raising the multiplier alone. Validate with the same map, camera and orientation, comparing rendered pixels and the 1.0/1.5 response instead of checking asset loading only.
|
||||
|
||||
|
||||
### Direct legend filtering
|
||||
|
||||
The current interactive object continues to determine the legend mode; enabling a layer with its existing switch also selects its legend, without adding a filter entry. Cables filter by route name, compute facilities by type, vessels by type and anchored state, and BGP by severity and collector activity. Coverage and impacted-region entries control their respective effects. The geographic legend separately controls lines, land fill and ocean fill without changing the shared land mask data. The obsolete BGP connections/hubs entry is removed.
|
||||
|
||||
`legend-filters.js` owns independent per-mode session filters. `legend.js` renders shared desktop/mobile toggle buttons with `aria-pressed`, preserving keyboard focus when refreshed. `interactable.js` filters draw batches, cluster candidates and picking while retaining complete source data, counts and object identities. Live vessel type changes, data reload and layer toggles respect filters. Hiding a selected item clears its card and overlays; explicit search selection reveals its category. Vessel track requests are versioned so late responses cannot resurrect a hidden track.
|
||||
|
||||
### Stable cards and display preferences
|
||||
|
||||
`callout-rest.js` retains a 450ms interaction-settle gate and subtle floating capped at 1.5px. Hover, keyboard reading, dragging and reduced-motion pause the float. Decorative offsets never enter persistence or placement solving, and the central boundary remains protected.
|
||||
|
||||
Display settings add `shared.satelliteLabelsEnabled` (true by default), `shared.satelliteLabelMinZoom` (2.5–5, default 4) and `shared.skyBrightness` (0–2, default 1). Desktop/mobile controls, reload and reset share runtime state. Sky brightness multiplies the sky material in linear space with its existing local sRGB output, leaving Earth exposure unchanged. Compute legend counts reflect located types. Chip and stacked SVG glyphs remain mapped to `site_type`; a GPU-only dataset does not invent other types or coordinates.
|
||||
|
||||
All destinations and legal routes are decided before presentation changes. `callout-dock-motion.js` uses a 240–420ms tween; `callout-routing.js` only plans static obstacle avoidance. Pauses retain progress and destination. Layout and movement never change collapse state. If a full-card route is unavailable, use its final destination directly instead of temporarily folding the card. Continuous scoring, staged yielding, central fallback and committing intermediate coordinates have been removed.
|
||||
|
||||
|
||||
### Cluster badges and Spiderfy
|
||||
|
||||
Ordinary surface markers retain GPU batches. Each aggregate uses the same projected children in `cluster-presentation.js` through collapse, expansion and the final resting state. A 26px solid numbered circle represents the group, with an amber ring for mixed categories. Membership changes only with data or filters, never through intermediate groups while zooming.
|
||||
|
||||
`cluster-motion.js` drives a 320ms cubic tween after visible icons are prepared and decoded, with encoded textures cached. Children interpolate from the group center to their real projections; `marker-stack.js` owns final display offsets. Animation and resting endpoints retain the same DOM nodes, with continuous reversal.
|
||||
|
||||
The UI's `globeZoomForDistance()` expands groups above 250%, with return hysteresis and no aggregate badges at 500%. Only co-located or extremely close geographic anchors (about 0.00015 degrees) share a compact cross-layer stack. Screen overlap and coarse city buckets never merge different locations. Each icon retains its own real projection as its baseline, with the entire stack bounded to 24px per axis and small groups exposing approximately 8–12px edges. Stable IDs define resting order; selecting a rear item brings it forward, with projected children at `z-index: 1000` and unclustered locked overlays at shared `renderOrder >= 100`, so focus stays above overlapping layers; clearing focus restores its slot. Ordinary expansion ends exactly at real projected positions; coordinates and marker positions remain unchanged. Data refresh removes retired member instances and synchronizes GPU ownership. Satellites and AIS vessels stay separate.
|
||||
|
||||
### Regional news cards
|
||||
|
||||
`news-region-cards.js` reuses `renderNewsCardContent` from `info-card.js` and the cruise adapter's news presentation mapping. Cruise and regional cards share the news template, 📰 icon and gold header. Regional metadata is reduced to source, category and time. Each card owns an independent cancellable `typewriter.js` instance, while an invisible sizing layer reserves the full summary height. Summaries are not truncated. A fixed content area with internal scrolling keeps new headlines and typing from changing placement. Automatic stowing pauses typing and rotation; expanding resumes, and the seven-second reading interval begins after typing completes.
|
||||
|
||||
`news-region-model.js` owns four isolated queues and remaining reading time, using `CRUISE_CONFIG.dwellMs` (seven seconds). It supports per-card pause, navigation, hidden-page resume and current-ID preservation across refreshes. Existing `getCruiseNewsItems()` data updates through `earth:news-payload-updated`, retaining source/category/content-readiness filtering. No new collection requests or global/other-region backfill are introduced.
|
||||
|
||||
News and compute submit scopes through `createCalloutGroups` to one `createSpatialCallouts` controller, sharing destination planning, drag locks, size changes and obstacle handling. Resize state is looked up by element identity; news layout keys use `news:<region>` to avoid compute-key collisions. Existing regional connector styling and routing remain unchanged, as do normal object-detail connectors.
|
||||
|
||||
The opt-in switch persists under `planet.earth.news-region-cards.v1` and lives in the existing news panel. Mobile moves the same nodes into the news drawer and removes their map connectors. Destruction releases cards, animations, observers and subscriptions. Verification covers region isolation, pause/resume, current-story retention, scope removal, simultaneous 1080p layout, mobile and returning to desktop.
|
||||
|
||||
The news cruise adapter completes `presentItem` only after typing finishes, so `CruiseSequencer` dwell never cuts off long text. The wait checks sequence ownership every frame and exits on manual interruption. Regional-news toggles reflect `aria-pressed` with a highlighted background and border. Compute headers retain country flags and use small representative emoji for regional groups.
|
||||
|
||||
### Card DPI and frame-stage boundaries
|
||||
|
||||
Compute, regional news and normal/cruise detail cards share `--hud-detail-card-width` and the same `info-panel.css` typography, metadata and summary rules. Compute cards, charts, icons, spacing and controls use root `--hud-scale`; manually resized inline pixel dimensions are normalized to a scale expression. SVG viewBox units scale with the chart viewport rather than receiving a second multiplier. Europe uses a small EU monogram with the title's font metrics, avoiding Windows flag-emoji fallback alignment.
|
||||
|
||||
The configuration menu retains its independent 340px width and original height limit rather than inheriting detail-card geometry scaling. Compute headers keep the shared HUD specification; larger primary values, charts and content spacing support the common width. Regional motion is owned by the JS tween: CSS position transitions and forced transform compositing are removed. Final left/top values align to device pixels, including decorative float, and connectors use that same displayed position. Regional cards use peripheral stowing when full content does not fit.
|
||||
|
||||
Global selections are heterogeneous: satellites are GeoJSON objects, cables are lines, and only surface interaction markers carry `userData.icon_layer_id`. `markerPresentationId` generates stack identity only for that marker contract, returning null for other selections instead of dereferencing missing fields during the frame loop.
|
||||
|
||||
`frame-stages.js` isolates independent update stages so a failed card, label or layer cannot skip other stages and scene rendering. It reports the stage/error, retries at one-second intervals or immediately after selection changes, and resumes normally on success. Fixing the cause and observing no runtime errors in normal scenarios remain acceptance requirements; isolation does not replace the fix.
|
||||
|
||||
All object-selection confirmations use `showSelectionStatus(name)` and “Selected: name”, including direct clicks, cluster children, search and motion confirmation. `confirmInteractionCandidate` and its feedback option are input-neutral. The cable layer updates selection/legend state while the interaction entry emits one notification. Obsolete lock/location/motion-confirmation message keys are removed; actual motion-mode changes and gesture guidance retain their mode label.
|
||||
|
||||
### Detail-card information cleanup
|
||||
|
||||
Normal details retain identity, business properties, timestamps and useful reliability context. Satellite cards omit the three rendering diagnostics (coverage capability, current presentation and coverage model) without changing coverage controls. News details omit feed identity, source type and fetch channel, retaining publisher, category, region, publication time and summary. Facility cards omit geocoding provenance, confidence scores, reasoning, matched names and verification timestamps while retaining location precision and verification status. Identical GPU model/platform values appear once. Internal source status is omitted; compute precision/measurement and original-source access remain.
|
||||
|
||||
Collector cards omit the redundant activity summary while retaining observation counts, event times and coverage. BGP event confidence remains relevant to alert reliability. Cables omit a single-coordinate row and custom points omit internal record IDs. Empty vessel enrichment no longer displays a caching message or enrichment confidence scores; identity, navigation, available profile data and source attribution remain. Desktop and mobile share field definitions. Underlying data and location-correction operations remain intact.
|
||||
|
||||
### Performance with multiple cards and the full satellite set
|
||||
|
||||
Regional news and compute cards retain their shared dark surfaces, borders, shadows and text while avoiding per-card backdrop filtering of the changing WebGL/video scene and SVG edge distortion. `contain: layout style` limits layout propagation. Pixel-aligned positioning, card count, content, typing and float timing remain intact.
|
||||
|
||||
Each `CalloutConnector` SVG covers its path plus node/shadow padding rather than the entire viewport; path coordinates remain in screen space. Geographic anchors cache unchanged local coordinates, invalidate on coordinate changes, and still update world projection and front/rear visibility each frame. Obstacle tests compare rectangle coordinates directly without allocating bounds for every candidate pair. HUD obstacle measurement precedes that frame's card-style writes.
|
||||
|
||||
Satellite interpolation copies exact endpoint samples and retains radius-preserving interpolation between them. Trail compaction only adds an early rejection for high-altitude trails that already fail the existing error bound. Repeated identical hover/selection indices avoid full alpha-buffer rewrites. Static cable and landing-point local matrices are cached; landing-point scale changes explicitly rebuild their matrix while parent-world transforms propagate normally. No satellites, trail samples or rendering pixels are removed.
|
||||
|
||||
For development diagnostics, append `?profileFrames=1` to the Earth URL. `data-earth-frame-profile` reports per-stage CPU mean, maximum and count every second; profiling is disabled normally. `data-earth-gpu-renderer` identifies the GPU. Performance evidence must include actual canvas pixel dimensions, fully visible card count, satellite visibility, zoom, video state and 30-second average FPS. Pending cards or smaller viewports do not establish a 1080p full-scene result. CPU stage durations are not GPU timings.
|
||||
|
||||
## Detail-card images
|
||||
|
||||
`card-images.js` integrates optional imagery into the existing `showInfoCard` path used by selection and cruise, with one presenter implementation for desktop and mobile. It requests `/api/v1/earth/card-images/resolve` on selection using NORAD IDs or source-qualified object IDs. The server derives models, groups and categories from current catalog data. A bounded 100-identity, 30-second cache avoids repeated lookups; a same-origin storage revision event refreshes open cards after an admin save. No image work runs in the animation loop.
|
||||
|
||||
The shared `card-images.css` places imagery behind the rightmost 68% of the existing body. Horizontal and bottom gradients intersect using standard/WebKit mask composition. The image uses 0.85 opacity with `saturate(1.2) contrast(1.1)`; its containing layer fades to 1, without an additional dark scrim or group opacity reduction. Text, headers, DPI sizing, scrolling, dragging and connectors retain their own layout. Decode precedes the fade; reduced motion skips animation. Missing/broken images produce a data-only card. Selection tokens guard both resolution and decoding, so late results cannot replace newer selections or reopen a closed card. The header source link exposes attribution and license information.
|
||||
|
||||
`card_image_configuration.py` stores assets and assignments in a separate system setting, using revision checks, row locks and audit transactions. Collected records are never edited. Resolution prefers object, model, group and category in that order; explicit per-object suppression stops fallback. Disabled assets cannot resolve; permission metadata remains truthful and separate from the enabled flag. Schemas reject duplicate targets and dangling references. Built-in assets come from the same manifest copied into the backend container, without seeding the database. Known built-in IDs preserve edits/removals while merging newly shipped entries; previous_defaults replaces old bundled URLs while preserving fields changed from the prior default and all custom image URLs.
|
||||
|
||||
Pillow validates still JPEG/PNG/WebP uploads and generates 640/1280-bound WebP derivatives, preserving original files. Content-addressed files are written atomically and served through a read-only public endpoint; management, upload, catalog search and preview require administrators. See the operations runbook for storage/backup. The Frontier photo is facility-specific; Starlink remains a series illustration. Qianfan series imagery is enabled as requested, while its permission metadata remains unverified.
|
||||
|
||||
Category `*` covers satellites, compute facilities, cables, landing stations, vessels, BGP incidents, BGP collectors and custom points, after specific category rules. Stable keys are MMSI, incident_key, collector_id and the custom-point ID. Missing identities in old snapshots allow only a type illustration. Rebind a collected object if its upstream source_id changes.
|
||||
|
||||
|
||||
### Five compute facility types and default imagery
|
||||
|
||||
Supported types are `supercomputer`, `gpu_cluster`, `data_center`, `cloud_center` and `edge_node`. `compute-site-types.js` centralizes frontend titles, labels, counts and fallback; backend `compute_types.py` shares classification between GeoJSON and image lookup. An explicit supported metadata.site_type wins, followed by data_type and known source defaults. Unrecognized facilities use a generic data-center presentation instead of becoming GPU clusters. No facilities are fabricated or existing records reclassified by name.
|
||||
|
||||
Each type supports legend filtering, marker icons, desktop/mobile details and image identity. New infrastructure cards use identity, operator, capacity, power and location fields without GPU counts or TOP500 ranks. The map accepts known sources or records declaring a supported compute type. Regional facilities retain snapshot/country constraints; infrastructure records do not require a GPU measurement contract and do not imply a global compute share. Built-in collectors remain TOP500 and Epoch AI; adding types does not create records.
|
||||
|
||||
Seven user-supplied images provide defaults for the five facility types, BGP collectors and BGP events. All five categories are available in image management before data exists. Exact assignments retain priority. Manifest previous_defaults / previous_rules migrate old bundled sources and unchanged collector assignments while preserving custom crop, enabled state and specific bindings.
|
||||
|
||||
|
||||
### Frame-work reductions with unchanged presentation
|
||||
|
||||
Cluster presentation reuses each parent's world matrix and local camera position within one update, refreshing on every frame. Fully collapsed members without child DOM still participate in hidden-state reconciliation, but skip invisible endpoint projections. Unchanged count and ARIA text no longer cause repeated DOM writes.
|
||||
|
||||
Static point batches refresh on data, selection, filtering, distance scaling or explicit invalidation; selected overlays continue pulsing every frame. Parsed colors cache immutable strings/numbers only, never mutable caller-owned Color objects. Satellite dot/halo overlays update only the required ancestor transform rather than traversing all Earth descendants twice.
|
||||
|
||||
Trail sampling combines the radius bound and initial error scan, preserving the existing threshold and subdivision indices. No satellites, trajectory samples, card content or resolution are removed. Deterministic comparison covered 10,000 histories with identical indices. Local Bun microbenchmarks (not whole-page FPS): 480 cluster members took about 1.19→0.90 ms collapsed and 4.47→2.86 ms expanded; scanning 16,621 trails took about 11.50→8.94 ms. End-to-end FPS requires a separate 30-second foreground comparison at identical resolution and scene/card/video settings.
|
||||
|
||||
Regional news collapse buttons explicitly invoke the shared HUD controller. Their bodies use hud-panel__body--collapsible and aria-expanded follows the actual state. Only user clicks collapse cards; layout and performance code never hide their contents automatically.
|
||||
|
||||
Floating object details and regional news share `--hud-detail-card-max-height`: 390px times HUD scale, capped at viewport height minus 24px. A min-height:0 flex chain keeps the header fixed and scrolls only the image-backed body. Short cards keep their natural height. BGP incidents prioritize type/severity/status/impact/time/summary; collectors prioritize identity/status/recent activity; facilities prioritize name/capacity/hardware, with location verification and provenance later. Desktop and mobile field order share CARD_CONFIG.
|
||||
|
||||
|
||||
### Detail identity and connector consistency
|
||||
|
||||
BGP collector indexes use collector names. Shared marker indexes omit empty IDs and refuse cross-refresh replacement for ambiguous duplicates; a clicked marker still in the current layer retains its own reference. Stack presentation shares icon_item_id with the index. BGP incidents prefer incident_key, while raw anomalies use source plus ID to prevent numeric collisions across feeds.
|
||||
|
||||
Satellite detail connectors identify the selected NORAD and relocate its index when the catalog array changes. Selection highlights rebind to that NORAD after refresh; a removed target clears selection. Uninitialized zero positions are not valid anchors. Cable anchors remain confined to the selected cable geometry, landing points use the clicked marker, and regional news/compute cards retain their regional anchors.
|
||||
|
||||
|
||||
When compute image configuration has no model/group rules, resolution uses a bounded fast path: the stable source prefix supplies the supercomputer/GPU category, then object and category rules are checked without scanning the collected catalog. Adding model or group rules automatically uses the full catalog resolver. Frontend identities, collected data and other object types are unchanged.
|
||||
|
||||
@@ -1,84 +1,35 @@
|
||||
# Intelligent Planet Interactable Clustering
|
||||
# Earth interactable clustering
|
||||
|
||||
Earth interactable icons are managed by `createInteractableLayer`. Rendering still uses Three.js `Points`, and picking, hover, locked state, tooltips, and cruise focus still depend on marker `userData`; the clustering strategy only decides which markers are represented by a cluster dot.
|
||||
`interactable.js` owns fixed geographic membership, `cluster-motion.js` owns timed tweening, and `cluster-presentation.js` owns circular badges, projected children, picking and Spiderfy. Independent markers remain GPU Points. Each clustered child uses the same DOM icon throughout its animation and at the final position.
|
||||
|
||||
## Strategies
|
||||
## Membership and zoom
|
||||
|
||||
`cluster.strategy` supports three modes:
|
||||
Spherical buckets use real `icon_base_position` anchors. Defaults are distance 15 in Earth units, at least two members and at most 14 per group. Data, filter and visibility changes invalidate membership; rotation and zoom do not. Badges stay at the spherical centroid without screen avoidance.
|
||||
|
||||
- `stable-spherical`: stable spherical clustering. It clusters by local 3D positions on the globe and caches topology by zoom band. Rotation and small zoom changes within the same band only update projection and size. Use it for semi-static layers such as BGP, compute centers, and Earth interactables.
|
||||
- `dynamic-screen`: dynamic screen-space clustering. This keeps the previous projection-based behavior and evaluates visible relationships per frame. Use it for high-frequency realtime layers such as vessels, or as a fallback.
|
||||
- `none`: no clustering. Every marker is shown independently. Use it for low-count layers or precision-first views.
|
||||
Use `cluster: { enabled: true }` or `cluster: false`. Options are `distance`, `minCount`, `maxMarkersPerDot`, `disableAboveZoom` (2.5) and `expansionHysteresis` (0.08). Zoom uses the UI's `globeZoomForDistance()` mapping: expand above 250%, collapse below 230%, and show no aggregate badges at 500%. Satellites and AIS vessels remain separate.
|
||||
|
||||
`cluster: false` is equivalent to `strategy: "none"`. If a layer enables clustering without declaring a strategy, it keeps the compatible `dynamic-screen` behavior.
|
||||
## Motion and Spiderfy
|
||||
|
||||
## Stable Spherical
|
||||
Both directions use a 320ms easeInOutCubic tween, complete after wheel input stops, and reverse continuously. Children interpolate directly from the centroid to their real local positions and retain the same DOM nodes at the endpoint. Because this chord can temporarily enter the sphere, occlusion uses a radial surface anchor while projection retains the interpolated position; intermediate frames remain visible.
|
||||
|
||||
`stable-spherical` moves cluster identity from screen distance to globe distance:
|
||||
Only co-located or extremely close geographic anchors (about 0.00015 degrees) share a compact cross-layer stack. Screen overlap and coarse city buckets never merge different locations. Each icon retains its own real projection as its baseline, with the entire stack bounded to 24px per axis and small groups exposing approximately 8–12px edges. Stable IDs define resting order; selecting a rear item brings it forward, and clearing focus restores its slot. Ordinary expansion ends exactly at real projected positions; coordinates and marker positions remain unchanged.
|
||||
|
||||
- Each marker uses `icon_base_position` as its geographic anchor.
|
||||
- By default, zoom levels above `2.5` force clustering off and show every marker as its original icon.
|
||||
- The current zoom maps to a discrete band; rotation and small zoom changes inside the same band do not recompute topology.
|
||||
- Clusters are recomputed only when the band, data revision, visibility, or filter state changes.
|
||||
- A cluster centroid is computed from member 3D positions and normalized back to the globe shell, so the cluster dot stays rigidly aligned to its geographic center.
|
||||
- Cluster dots do not participate in 2D avoidance, so screen-space repulsion cannot push them away from their real geographic projection.
|
||||
- Band changes use a small hysteresis margin so zooming at a boundary does not repeatedly bounce between two bands.
|
||||
- Newly created marker and cluster dots run a short scale + opacity ease. This is only a rendering transition; it does not change marker coordinates, picking objects, or locked state.
|
||||
## Interaction and BGP
|
||||
|
||||
The stable strategy uses spherical bucket/hash neighbor lookup and must not use an all-pairs loop. Most frames only pay projection and material-size cost; topology cost is paid only on band or data changes.
|
||||
Clicking a badge expands it and pauses rotation. Legs have 14px transparent hit areas; hover highlights the matching leg and child, and either click selects the same business object. The canvas container accepts wheel input over the canvas and projected markers while excluding HUD scrolling.
|
||||
|
||||
## Configuration
|
||||
Projected textures bake the original category tint. BGP events retain severity colors. Badges containing BGP and expanded BGP event children use an expanding/fading halo based on `BGP_CONFIG.ring`; replaced GPU effects are hidden to avoid duplicate rings at old anchors. Reduced-motion preference disables the pulse.
|
||||
|
||||
```js
|
||||
const computeCenterIconLayer = createInteractableLayer({
|
||||
id: "computeCenters",
|
||||
// ...
|
||||
avoidance: SURFACE_AVOIDANCE_PROFILES.city,
|
||||
cluster: {
|
||||
strategy: "stable-spherical",
|
||||
minCount: 2,
|
||||
maxMarkersPerDot: 14,
|
||||
transitionMs: 220,
|
||||
bandHysteresis: 0.08,
|
||||
disableAboveZoom: 2.5,
|
||||
bands: [
|
||||
{ key: "far", maxZoom: 1.7, distance: 15 },
|
||||
{ key: "mid", maxZoom: 2.6, distance: 8 },
|
||||
{ key: "near", maxZoom: 3.5, distance: 4 },
|
||||
{ key: "detail", maxZoom: Infinity, distance: 0 },
|
||||
],
|
||||
},
|
||||
});
|
||||
```
|
||||
## Removed paths
|
||||
|
||||
Realtime layers can stay dynamic:
|
||||
The low-zoom compact-dot desktop/mobile controls, persistence field, textures, rendering/picking branches and translations are removed. Dynamic screen clustering, discrete zoom bands, GPU aggregate dots and the old creation-time fade/scale transition are also removed. There is no legacy strategy fallback.
|
||||
|
||||
```js
|
||||
const vesselIconLayer = createInteractableLayer({
|
||||
id: "vessels",
|
||||
// ...
|
||||
cluster: {
|
||||
strategy: "dynamic-screen",
|
||||
enabled: true,
|
||||
maxMarkersPerDot: 10,
|
||||
},
|
||||
});
|
||||
```
|
||||
## Regression coverage
|
||||
|
||||
## Tuning
|
||||
Tests cover 320ms completion, reversal continuity, visible intermediate frames, retained endpoint nodes, 500% expansion, immutable source coordinates, filtering cleanup, wheel target routing, linked leg hover/click and projected BGP category colors: `cluster-motion.test.js`, `cluster-presentation.test.js`, `globe-navigation.test.js`, `marker-badges.test.js`.
|
||||
|
||||
- `distance` is the 3D globe-distance threshold and uses the same unit as `CONFIG.earthRadius`. Larger values cluster more aggressively.
|
||||
- The farthest band usually uses a larger `distance`; the nearest detail band usually uses `0` to split clusters into original icons.
|
||||
- `bandHysteresis` controls band-boundary stickiness. Too little can flicker near thresholds; too much can make band changes feel late.
|
||||
- `transitionMs` controls cluster split/merge easing. Keep it around 160-260ms; longer durations can make realtime layers feel sluggish.
|
||||
- `disableAboveZoom` controls the precision-view threshold. It defaults to `2.5`; above that zoom, no cluster dots are generated. Set it to `false` to disable the hard threshold.
|
||||
- High-frequency realtime layers should prefer `dynamic-screen` so frequent data changes do not trigger stable topology recomputation.
|
||||
- If a layer behaves poorly, switch it back to `dynamic-screen` or use `cluster: false`.
|
||||
## Refresh and first-frame preparation
|
||||
|
||||
## Acceptance Checks
|
||||
A reused cluster ID does not imply reused Object3D instances: stale children are reconciled against current instances. Ownership changes immediately rebuild the GPU batch, and topology invalidation clears its old signature.
|
||||
|
||||
- Rotating the globe inside one zoom band should not cause clusters to flicker or regroup.
|
||||
- Cluster dots should stay aligned with the globe-surface centroid and should not be pushed by avoidance.
|
||||
- Zooming into the detail band should restore the layer's original icon textures and click behavior.
|
||||
- Zoom levels above 250% should not show cluster dots.
|
||||
- Realtime layers such as vessels should reflect incoming updates immediately.
|
||||
Expansion prepares visible icons and waits for image decoding before starting its 320ms clock. Encoded URLs are reused per texture, and invisible rear members are not instantiated. Regression tests cover repeated refresh/zoom, preparation slower than the tween duration, selection/restoration and connector anchors.
|
||||
|
||||
@@ -1,5 +1,17 @@
|
||||
# Earth Layer Style Property Index
|
||||
|
||||
`EARTH_MATERIAL_CONFIG.surface.night` centralizes night texture contrast (detailGamma=0.55) and separate land/ocean tint and ambient values. It only contributes to the night branch, keeps day exposure unchanged, and reuses the geographic mask without another texture or draw call.
|
||||
|
||||
## Modern object presentation
|
||||
|
||||
- Surface: `EARTH_MATERIAL_CONFIG.surface` owns exposure=0.9, landSaturation=0.9, oceanAbsorption=0.36, oceanDeepColor=[0.004,0.012,0.032], night retention and Fresnel parameters. `atmosphere` owns the single shell's radius, color and strength. Base and terrain specular highlights are disabled.
|
||||
- Cables blend a cool normal tint with their category color, consistently with the legend. Selected pulses follow cumulative route length with 6 repetitions and speed 4.5 in the existing batch draw. Drawing no longer writes depth; original Line pick objects, cable IDs and state logic remain intact.
|
||||
- Landing points use the 128×128 monochrome `marker-landing-point.svg` micro-core instead of a glossy procedural ball; Sprite picking and billboard size logic remain intact.
|
||||
- Compute markers use canonical SVG inner glyphs over shared gradient plates, pale rims, top highlights and shadows. GPU/AI uses purple stacked layers, supercomputers purple chips, and infrastructure teal server racks. Estimated badges and picking remain intact.
|
||||
- Satellites share vivid real-inclination category colors with the legend and predicted orbits. Normal markers are prototype hard discs without outlines or sphere shading; breathing defaults off. Coverage stays blue and sunlight only affects opacity.
|
||||
|
||||
|
||||
|
||||
This document records the material, color, opacity, line width, radius offset, and `renderOrder` style properties of all Earth frontend layers. For layer ordering relationships, see [Earth Render Layer Order](/home/ray/dev/linkong/planet/docs/technical/en/earth-render-layer-order.md).
|
||||
|
||||
## Naming Conventions
|
||||
@@ -20,15 +32,11 @@ This document records the material, color, opacity, line width, radius offset, a
|
||||
| Earth base radius | `CONFIG.earthRadius` | `100` | `earth.js:createEarth()` |
|
||||
| Earth base color | `EARTH_MATERIAL_CONFIG.color` | `0x010609` | `MeshPhongMaterial.color` |
|
||||
| Earth base emissive | `EARTH_MATERIAL_CONFIG.emissive` | `0x010609` | `MeshPhongMaterial.emissive` |
|
||||
| Earth base specular | `EARTH_MATERIAL_CONFIG.specular` | `0x1a2d45` | `MeshPhongMaterial.specular` |
|
||||
| Earth base shininess | `EARTH_MATERIAL_CONFIG.shininess` | `12` | `MeshPhongMaterial.shininess` |
|
||||
| Earth base specular | `EARTH_MATERIAL_CONFIG.specular` | `0x000000` | `MeshPhongMaterial.specular` |
|
||||
| Earth base shininess | `EARTH_MATERIAL_CONFIG.shininess` | `0` | `MeshPhongMaterial.shininess` |
|
||||
| Earth base opacity | `EARTH_MATERIAL_CONFIG.opacity` | `1` | `MeshPhongMaterial.opacity` |
|
||||
| HD texture radius offset | `EARTH_MATERIAL_CONFIG.textureOverlayAltitudeOffset` | `0.48` | Standalone HD texture sphere radius; must keep enough depth separation from the land/ocean base and Earth base sphere to avoid far-zoom z-fighting |
|
||||
| HD texture opacity | `EARTH_MATERIAL_CONFIG.textureOverlayOpacity` | `0.88` | HD texture `MeshPhongMaterial.opacity` |
|
||||
| HD texture renderOrder | `EARTH_MATERIAL_CONFIG.textureOverlayRenderOrder` | `0.96` | `_earthTextureOverlay.renderOrder` |
|
||||
| HD texture specular | `EARTH_MATERIAL_CONFIG.textureOverlaySpecular` | `0x05080d` | Reduces specular highlight in direct-light areas to avoid blown-out texture |
|
||||
| HD texture shininess | `EARTH_MATERIAL_CONFIG.textureOverlayShininess` | `4` | Reduces specular concentration |
|
||||
| HD texture color multiplier | inline | `0xffffff` | `_earthTextureOverlayMaterial.color` |
|
||||
|
||||
## Earth Occluder and Day/Night
|
||||
|
||||
@@ -36,7 +44,7 @@ This document records the material, color, opacity, line width, radius offset, a
|
||||
| --- | --- | --- | --- |
|
||||
| Occluder radius factor | `EARTH_MATERIAL_CONFIG.occluderRadiusFactor` | `0.999` | Depth occluder sphere radius |
|
||||
| Occluder segments | `EARTH_MATERIAL_CONFIG.occluderSegments` | `48` | Occluder geometry segments |
|
||||
| Occluder renderOrder | inline | `-1` | `occluder.renderOrder` |
|
||||
| Occluder renderOrder | inline | `0.5` | `occluder.renderOrder` |
|
||||
| Day/night sun direction | `EARTH_MATERIAL_CONFIG.dayNight.sunDirection` | `{ x: 1, y: 0.2, z: 0.4 }` | Custom day/night shader |
|
||||
| Night-side minimum brightness | `EARTH_MATERIAL_CONFIG.dayNight.nightFloor` | `0.24` | Shader uniform |
|
||||
| Day-side boost | `EARTH_MATERIAL_CONFIG.dayNight.dayBoost` | `1.12` | Shader uniform |
|
||||
@@ -50,27 +58,12 @@ This document records the material, color, opacity, line width, radius offset, a
|
||||
|
||||
| Name | Variable | Current Value | Location / Notes |
|
||||
| --- | --- | --- | --- |
|
||||
| Inner atmosphere radius factor | `EARTH_MATERIAL_CONFIG.atmosInnerRadiusFactor` | `1.01` | `atmosInnerGeo` |
|
||||
| Inner atmosphere segments | `EARTH_MATERIAL_CONFIG.atmosInnerSegments` | `64` | `atmosInnerGeo` |
|
||||
| Inner atmosphere color | `EARTH_MATERIAL_CONFIG.atmosInnerColor` | `[0.25, 0.62, 1.0]` | Shader RGB |
|
||||
| Inner atmosphere rim power | `EARTH_MATERIAL_CONFIG.atmosInnerRimPower` | `3.2` | Shader rim falloff |
|
||||
| Inner atmosphere intensity | `EARTH_MATERIAL_CONFIG.atmosInnerIntensity` | `0.18` | Shader alpha multiplier |
|
||||
| Outer atmosphere radius factor | `EARTH_MATERIAL_CONFIG.atmosOuterRadiusFactor` | `1.016` | `atmosOuterGeo` |
|
||||
| Outer atmosphere segments | `EARTH_MATERIAL_CONFIG.atmosOuterSegments` | `48` | `atmosOuterGeo` |
|
||||
| Outer atmosphere color | `EARTH_MATERIAL_CONFIG.atmosOuterColor` | `[0.18, 0.45, 0.9]` | Shader RGB |
|
||||
| Outer atmosphere rim power | `EARTH_MATERIAL_CONFIG.atmosOuterRimPower` | `5.0` | Shader rim falloff |
|
||||
| Outer atmosphere intensity | `EARTH_MATERIAL_CONFIG.atmosOuterIntensity` | `0.02` | Shader alpha multiplier |
|
||||
| Atmosphere blending | inline | `THREE.AdditiveBlending` | `ShaderMaterial.blending` |
|
||||
| Atmosphere renderOrder | inline | `1` | `atmosInner/Outer.renderOrder` |
|
||||
| No-HD-texture rim glow color | `EARTH_MATERIAL_CONFIG.rimGlowColor` | `[0.35, 0.65, 1.0]` | Fresnel shell RGB when HD texture is hidden or unavailable |
|
||||
| No-HD-texture rim glow power | `EARTH_MATERIAL_CONFIG.rimGlowPower` | `3.8` | Shader rim falloff; higher = narrower edge |
|
||||
| No-HD-texture rim glow intensity | `EARTH_MATERIAL_CONFIG.rimGlowIntensity` | `0.28` | Shader alpha multiplier |
|
||||
| No-HD-texture rim glow segments | `EARTH_MATERIAL_CONFIG.rimGlowSegments` | `64` | `earth-rim-glow` geometry segments |
|
||||
| Cloud layer radius offset | `CLOUD_LAYER_CONFIG.radiusOffset` | `3` | Cloud sphere radius |
|
||||
| Cloud layer segments | `CLOUD_LAYER_CONFIG.widthSegments / heightSegments` | `64 / 64` | Cloud sphere geometry |
|
||||
| Cloud layer opacity | `CLOUD_LAYER_CONFIG.opacity` | `0.15` | `MeshPhongMaterial.opacity` |
|
||||
| Cloud layer opacity | `CLOUD_LAYER_CONFIG.opacity` | `0.15` | `MeshBasicMaterial.opacity` |
|
||||
| Cloud texture | `CLOUD_LAYER_CONFIG.textureUrl` | `"./assets/earth_clouds_1024.png"` | Cloud texture map |
|
||||
| Cloud blending | inline | `THREE.AdditiveBlending` | `MeshPhongMaterial.blending` |
|
||||
| Cloud blending | inline | `THREE.AdditiveBlending` | `MeshBasicMaterial.blending` |
|
||||
|
||||
## Land/Ocean Base and Country Borders
|
||||
|
||||
@@ -94,7 +87,7 @@ The land/ocean base is an Earth base-map asset and preloads at startup; the "Bor
|
||||
| Land/ocean base renderOrder | `COUNTRY_BOUNDARY_CONFIG.landRenderOrder` | `0.86` | `country-land-ocean.renderOrder` |
|
||||
| Land/ocean mask size | `landMaskWidth / landMaskHeight` | `2048 / 1024` | Canvas / DataTexture size |
|
||||
| Country tint color | `COUNTRY_BOUNDARY_CONFIG.tintColor` | `0x0b1830` | Tint when HD texture is off |
|
||||
| Country tint radius offset | `COUNTRY_BOUNDARY_CONFIG.tintAltitudeOffset` | `0.04` | `country-tint` radius |
|
||||
| Country tint radius offset | `COUNTRY_BOUNDARY_CONFIG.tintAltitudeOffset` | `0.32` | `country-tint` radius |
|
||||
| Country tint renderOrder | `COUNTRY_BOUNDARY_CONFIG.tintRenderOrder` | `0.2` | `country-tint.renderOrder` |
|
||||
| Border line color | `COUNTRY_BOUNDARY_CONFIG.lineColor` | `0x7fc7ff` | Normal border line |
|
||||
| Border line opacity | `COUNTRY_BOUNDARY_CONFIG.lineOpacity` | `0.58` | Normal border line opacity |
|
||||
@@ -123,8 +116,8 @@ The land/ocean base is an Earth base-map asset and preloads at startup; the "Bor
|
||||
| Terrain opacity | `TERRAIN_CONFIG.opacity` | `0.68` | `MeshPhongMaterial.opacity` |
|
||||
| Terrain color | `TERRAIN_CONFIG.color` | `0x8aa884` | `MeshPhongMaterial.color` |
|
||||
| Terrain emissive | `TERRAIN_CONFIG.emissive` | `0x030704` | Reduces self-emission to preserve terrain shading |
|
||||
| Terrain specular | `TERRAIN_CONFIG.specular` | `0x344438` | Gives terrain local sheen without boosting HD texture brightness |
|
||||
| Terrain shininess | `TERRAIN_CONFIG.shininess` | `16` | Tightens terrain highlight |
|
||||
| Terrain specular | `TERRAIN_CONFIG.specular` | `0x000000` | Matte terrain; no white specular lobe |
|
||||
| Terrain shininess | `TERRAIN_CONFIG.shininess` | `0` | Specular disabled |
|
||||
| Terrain renderOrder | inline | `1.2` | `terrain.renderOrder` |
|
||||
| Terrain polygonOffset | inline | `factor -1`, `units -1` | Reduces z-fighting near sphere surface |
|
||||
|
||||
@@ -143,14 +136,16 @@ The land/ocean base is an Earth base-map asset and preloads at startup; the "Bor
|
||||
|
||||
## Submarine Cables and Landing Points
|
||||
|
||||
`cable-batches.js` batches cable lines and landing points separately. The Sprite material and size parameters below remain owned by picking and selection proxies in `cables.js`; their color, opacity, scale, and visibility update the style texture or instance attributes. Drawing uses `LineSegments` and instanced billboards with the existing textures, colors, render order, and globe occlusion rules.
|
||||
|
||||
| Name | Variable | Current Value | Location / Notes |
|
||||
| --- | --- | --- | --- |
|
||||
| Default cable color | `CABLE_COLORS.default` | `0xffff44` | Used when no data color available |
|
||||
| Cable radius offset | `CABLE_CONFIG.line.altitudeOffset` | `0.2` | Cable line radius |
|
||||
| Cable radius offset | `CABLE_CONFIG.line.altitudeOffset` | `0.48` | Cable line radius |
|
||||
| Cable line width | `CABLE_CONFIG.line.lineWidth` | `1` | `LineBasicMaterial.linewidth` |
|
||||
| Cable opacity | `CABLE_CONFIG.line.opacity` | `1.0` | Cable line opacity |
|
||||
| Cable renderOrder | `CABLE_CONFIG.line.renderOrder` | `1` | Cable line level |
|
||||
| Landing point radius offset | `CABLE_CONFIG.landingPoint.altitudeOffset` | `0.2` | Same surface height as cable lines, avoiding a floating marker |
|
||||
| Landing point radius offset | `CABLE_CONFIG.landingPoint.altitudeOffset` | `0.48` | Same surface height as cable lines, avoiding a floating marker |
|
||||
| Landing point sprite height | local `LANDING_POINT_SPRITE_HEIGHT` | `3` | `THREE.Sprite` base height |
|
||||
| Landing point reference FOV | local `LANDING_POINT_SIZE_REFERENCE_FOV` | `75` | Matches the current Earth camera FOV |
|
||||
| Landing point scale minimum | local `LANDING_POINT_SIZE_SCALE_MIN` | `0.16` | Minimum multiplier after roughly 200% zoom, limiting high-zoom screen footprint; `3 * 0.16 = 0.48` |
|
||||
@@ -160,7 +155,7 @@ The land/ocean base is an Earth base-map asset and preloads at startup; the "Bor
|
||||
| Landing point opacity | `CABLE_CONFIG.landingPoint.opacity` | `1.0` | `SpriteMaterial.opacity` |
|
||||
| Landing point renderOrder | `CABLE_CONFIG.landingPoint.renderOrder` | `1` | Same level as cable lines; `depthTest: false` keeps the ball whole, while camera-to-center globe occlusion hides back-side points |
|
||||
| Landing point dim brightness | `landingPointVisual.dimBrightness` | `0.62` | Dim state color multiplier |
|
||||
| Related landing point opacity | `landingPointVisual.related.opacityBase / opacityPulse` | `0.8 / 0.2` | Highlight pulse |
|
||||
| Related landing halo extent | `landingPointVisual.related.haloScale / haloFalloff` | `1.65 / 8` | Expands only the halo canvas; core size and color stay fixed |
|
||||
| Dimmed landing point color | `landingPointVisual.dimmed.colorRGB` | `{ r: 180, g: 116, b: 28 }` | Dim state color; avoids dark base showing through as a dark hole |
|
||||
| Dimmed landing point opacity | `landingPointVisual.dimmed.opacity` | `0.78` | Dim state opacity; no longer uses low alpha blending with dark base |
|
||||
|
||||
@@ -172,14 +167,11 @@ The land/ocean base is an Earth base-map asset and preloads at startup; the "Bor
|
||||
| Satellite altitude compression scale | `SATELLITE_CONFIG.altitudeCompressionKm` | `1200` | TLE/SGP4 altitude compression; larger values soften high-orbit separation |
|
||||
| Satellite real-altitude cap | `SATELLITE_CONFIG.maxDisplayAltitudeKm` | `40000` | Altitude clamp for the compressed display mapping, covering GEO-range altitude |
|
||||
| Satellite real-altitude display offset range | `minRealAltitudeOffset / maxRealAltitudeOffset` | `4 / 25` | Compressed offset added to `CONFIG.earthRadius`; `25` is about one quarter of the current globe radius `100`, keeping high-orbit separation visible without pushing GEO/MEO trails too far away |
|
||||
| Satellite dot base pixel size | `SATELLITE_CONFIG.dotBaseSize` | `2.8` | Point shader size |
|
||||
| Satellite backdrop dot scale | `SATELLITE_CONFIG.dotBackdropScale` | `1.28` | Backdrop dot size |
|
||||
| Satellite dot opacity range | `dotOpacityMin / dotOpacityMax` | `0.42 / 1.0` | Breathing animation |
|
||||
| Satellite dot breathing speed | `SATELLITE_CONFIG.dotBreathingSpeed` | `0.12` | Dot opacity animation |
|
||||
| Satellite backdrop renderOrder | inline | `5` | `satelliteBackdropPoints.renderOrder` |
|
||||
| Satellite dot base pixel size | `SATELLITE_CONFIG.dotBaseSize` | `2.2` | Legacy interaction hit-radius basis; not visual size |
|
||||
| Satellite dot opacity range | `dotOpacityMin / dotOpacityMax` | `0.72 / 1.0` | Selected-point breathing; ordinary breathing defaults off |
|
||||
| Satellite dot renderOrder | inline | `6` | `satellitePoints.renderOrder` |
|
||||
| Satellite trail length | `SATELLITE_CONFIG.trailLength` | `10` | Trail buffer |
|
||||
| Satellite trail line width | `SATELLITE_CONFIG.trailLineWidth` | `3` | Ribbon shader uniform |
|
||||
| Satellite trail line width | `SATELLITE_CONFIG.trailLineWidth` | `3` (cap) | Also capped at 60% of point size with tapered short tails |
|
||||
| Selected ring size | `SATELLITE_CONFIG.ringSize` | `0.07` | Hover / locked ring sprite |
|
||||
| Satellite overlay renderOrder | `SATELLITE_CONFIG.overlayRenderOrder` | `12` | Locked ring / halo / orbit |
|
||||
| Footprint renderOrder | local `GROUND_FOOTPRINT_RENDER_ORDER` | `3` | Starlink footprint fill and Iridium coverage ring; must stay above land / texture / terrain surface layers |
|
||||
@@ -214,8 +206,8 @@ AIS vessel markers use batched `THREE.Points`, not one `THREE.Sprite` per vessel
|
||||
| Hover scale | `COMPUTE_CENTER_CONFIG.marker.hoverScale` | `1.16` | Hover overlay size multiplier |
|
||||
| Locked scale | `COMPUTE_CENTER_CONFIG.marker.lockedScale` | `1.22` | Locked overlay size multiplier, with pulse |
|
||||
| Dimmed scale / opacity | `dimmedScale / dimmedOpacity` | `0.82 / 0.34` | Dim state |
|
||||
| Supercomputer color | `COMPUTE_CENTER_CONFIG.colors.supercomputer` | `"#38bdf8"` | Marker texture |
|
||||
| GPU cluster color | `COMPUTE_CENTER_CONFIG.colors.gpu_cluster` | `"#2dd4bf"` | Marker texture |
|
||||
| Supercomputer color | `COMPUTE_CENTER_CONFIG.colors.supercomputer` | `"#a855f7"` | Marker texture |
|
||||
| GPU cluster color | `COMPUTE_CENTER_CONFIG.colors.gpu_cluster` | `"#a855f7"` | Marker texture |
|
||||
| Linked color | `COMPUTE_CENTER_CONFIG.colors.linked` | `"#f8fafc"` | Linked state |
|
||||
| Compute center renderOrder | local `COMPUTE_CENTER_RENDER_ORDER` | `4.5` | Surface facility below satellites |
|
||||
|
||||
@@ -257,7 +249,54 @@ AIS vessel markers use batched `THREE.Points`, not one `THREE.Sprite` per vessel
|
||||
| Moon halo scale | `CELESTIAL_CONFIG.moonHaloScale` | `62` | Moon halo |
|
||||
| Sun light color / intensity | `sunLightColor / sunLightIntensity` | `0xfff4df / 1.02` | Scene light |
|
||||
| Back light color / intensity | `backLightColor / backLightIntensity` | `0x2b4c78 / 0.3` | Scene light |
|
||||
| Star count | `STARFIELD_CONFIG.count` | `8000` | `createStars()` |
|
||||
| Star radius range | `minRadius + radiusJitter` | `800 + 200` | Random distribution |
|
||||
| Star color | `STARFIELD_CONFIG.color` | `0xffffff` | `PointsMaterial.color` |
|
||||
| Star size | `STARFIELD_CONFIG.size` | `0.5` | `PointsMaterial.size` |
|
||||
|
||||
|
||||
## Badge taxonomy and palette
|
||||
|
||||
| Type | Primary color | Inner glyph |
|
||||
| --- | --- | --- |
|
||||
| Datacenter/infrastructure | Teal `#14b8a6` → `#0d9488`, rim `#5eead4` | Server rack |
|
||||
| GPU/AI | Purple `#a855f7` → `#7e22ce`, rim `#d8b4fe` | Stacked compute layers |
|
||||
| Supercomputer | Same purple family | Chip |
|
||||
| Cable landing point | Amber `#f59e0b`, pale-gold `#fef3c7` rim | Glowing circular core |
|
||||
| BGP collector | Blue and existing activity heat colors | Rounded badge containing the broadcast-tower glyph |
|
||||
| BGP event | Low/medium/high/critical: blue/amber/orange/red | Irregular flat event symbols, no badge plate |
|
||||
| Vessel | Cargo blue, tanker orange, passenger teal, fishing amber, military indigo, other silver | Heading arrow; anchored state retained |
|
||||
| Custom marker | Place green, note yellow, alert red, default blue | Pin, note sheet, warning triangle |
|
||||
|
||||
BGP change events retain irregular flat symbols as the exception; collectors use rounded badges; other finite surface objects share cached badge textures without a new DOM node or draw object per marker. Vessels retain heading buckets and vertex colors rather than multiplying buckets by vessel type. Satellites use distinct category colors and hard discs without an outline; orbit and footprint calculations stay intact. The low-zoom dot control, persisted field and rendering/picking branches are removed.
|
||||
|
||||
|
||||
### Stable satellite palette
|
||||
|
||||
Category colors follow [satellitemap.space](https://satellitemap.space/) while retaining this project's real inclination classes: 0–30° blue `#3399ff`, 30–60° orange `#ff801a`, 60–90° green `#00e680`, 90–120° purple `#cc33ff`, 120–180° rose `#ff3366`, unknown gray `#94a3b8`. A satellite keeps its hue across day/night; predicted orbits no longer use an unrelated yellow-purple gradient.
|
||||
|
||||
All coverage uses `#3399ff`, opacity 0.34 and day-side opacity multiplier 1.65. Normal points have no dark outline; one `PointsMaterial` batch uses alphaTest=0.5. `satellitePixelSize()` smoothly varies visual size from 2.4 to 8.0.
|
||||
|
||||
### Selected orbit and panel outlines
|
||||
|
||||
Prediction covers the full period without truncating or fading the endpoints and without an appended closing vertex. Highlights travel forward along cumulative arc length in the same Line draw. TV and shared HUD shells use square corners, including the player clip, without changing panel size or dragging. Temporary frame-statistics instrumentation and its QA-only DOM attributes are removed from release code.
|
||||
|
||||
### Landing points, limb badges and subtle sky brightness
|
||||
|
||||
Cables and landing points share R+0.48 and renderOrder=1. `cable-surface.js` generates both on the actual surface triangles, removing ideal-sphere versus surface endpoint parallax. Normal points are yellow discs with a pale ring and no persistent halo. Only related points get a subtle pulsing halo in the existing batch shader; unrelated points dim, and deselection restores the normal yellow state.
|
||||
|
||||
Rounded badges use anchor occlusion through `surface-marker-occlusion.js`: the GPU keeps a visible anchor's whole badge and hides it when Earth occludes the anchor. Screen-space picking shares the same check. Only badge layers replace per-pixel depth clipping; BGP change events keep their flat symbols and existing depth behavior. Collectors retain activity colors inside rounded plates. Normal, hover, locked and cluster badges share the path.
|
||||
|
||||
Only the sky texture material receives `CELESTIAL_CONFIG.skyBrightness = 1.0`; Earth exposure, celestial lighting and business-layer brightness stay unchanged.
|
||||
|
||||
Vessels are independent moving objects: their pre-badge heading arrows, anchored dots and hover/selected glow are restored. They do not use the fixed-facility rounded plates or whole-badge occlusion path.
|
||||
|
||||
Normal and related landing points retain the same orange core and yellow rim. Related state adds a wider soft pulsing halo without changing core color, opacity or size. Deselecting clears the halo; unrelated points retain the dim state.
|
||||
|
||||
|
||||
### Sky color output
|
||||
|
||||
The star map is decoded as sRGB. `celestial.js` uses shared `preserveTextureOutput()` to encode only the sky material back to display sRGB. The Three.js r128 renderer retains its existing global linear output so Earth, satellites and other layers remain unchanged. `skyBrightness=1.0` retains source brightness; missing output encoding displays decoded dark values directly and cannot be corrected by raising the multiplier alone. Validate with the same map, camera and orientation, comparing rendered pixels and the 1.0/1.5 response instead of checking asset loading only.
|
||||
|
||||
Satellite hover rings use the hovered satellite inclination category color, matching the point and legend. Changing the hovered object resolves its color again; position updates preserve it.
|
||||
|
||||
All map cards use the root HUD DPI scale. Regional and cruise news share header icon, colors, typography and padding; the regional variant only simplifies metadata and adds independent rotation controls. Compute metrics, trend charts and title icons no longer use unscaled fixed pixels. Europe uses a compact EU monogram; other region emoji follow the same scale.
|
||||
|
||||
Compute markers use five canonical SVGs: compute-supercomputer.svg, compute-gpu-cluster.svg, compute-hdd-network.svg, compute-cloud.svg and compute-edge.svg. They retain the compute layer purple badge and existing selection behavior.
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
# Earth Render Layer Order
|
||||
|
||||
The compute topic reuses the compute-center Interactable icons, depth behavior, and layer order. Country/year changes replace that layer's data and clear prior selection. Its HUD is screen UI excluded from Earth picking in `main.js`. Regional cards are tethered in screen space; rear anchors do not draw connectors through the globe. Facility details reuse the existing card and mobile details drawer. The visible HD surface uses a single-draw ShaderMaterial in `earth-surface-material.js`, retaining the R+0.48 mesh, depth policy and surface pick target. The existing controller still switches all four 8K maps, updates the sampler and disposes the previous texture only after success. Normals, view direction and astronomical sunlight share world coordinates. Land is matte; water has grazing-angle cool reflection without a white specular lobe. The existing geographic land/ocean texture owns the optical mask independently of border visibility, with a conservative chroma fallback until loaded and reflection suppression over ice. Original JPEGs are neither repainted nor downsampled. Color textures are tagged sRGB with mipmaps. The r128 surface shader decodes sRGB; r160 uses hardware texture decoding. Both combine light in linear space and encode once. A local filmic curve affects only the surface instead of enabling global ACES exposure on the sky and data layers. The terminator spans normal dot products [-0.12, 0.28]; night attenuation retains 0.12–0.32 linear illumination after tone mapping to avoid crushing source shadows twice. One R×1.025 BackSide/Additive shell replaces the three rim shells, with depthWrite=false and depthTest=true; inner scattering is part of the surface pass. Independently switchable clouds use matte day/night shading at renderOrder 1.1. The sky uses the existing `starmap_deep_8k.jpg` without a tint overlay.
|
||||
|
||||
This document records the current Earth renderer's layer order and the intent of each layer. When adjusting `renderOrder`, radius offsets, depth strategy, or pointer interaction, update this document accordingly.
|
||||
|
||||
Note: the layer control panel order and the registration / startup load order are two separate semantics.
|
||||
@@ -14,28 +16,35 @@ Note: the layer control panel order and the registration / startup load order ar
|
||||
| Order | Layer | Source | Render / Radius Strategy | Depth / Interaction Strategy | Notes |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| -1000 | Celestial background mesh | `celestial.js` | Background sphere | Not part of surface picking | Behind all Earth content. |
|
||||
| -1 | Earth occluder sphere | `earth.js` | Invisible inner sphere | Writes depth buffer | Occludes objects behind the Earth. |
|
||||
| 0.5 | Earth occluder sphere | `earth.js` | Invisible inner sphere | Writes depth buffer | Occludes objects behind the Earth. |
|
||||
| 0 | Earth base sphere | `earth.js` | `CONFIG.earthRadius` | Surface picking fallback target | Dark base; still visible when all optional map layers are off. |
|
||||
| 0.2 | Country dark tint | `country-boundaries.js` | `tintAltitudeOffset` | Raycast disabled | Used when HD texture is off. |
|
||||
| 0.86 | Land/ocean base fill | `country-boundaries.js` | `landAltitudeOffset = 0.32`; ocean `#010609`, land `#080f1b` | Raycast disabled | Base map remains usable even when country borders are off; radius is separated from the base sphere to avoid far-zoom z-fighting. |
|
||||
| 0.96 | HD Earth texture | `earth.js` | `textureOverlayAltitudeOffset = EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET = 0.48` | Surface picking target when visible | HD texture always overlays the land/ocean base fill; radius must stay above the land/ocean base and far enough from the base sphere. |
|
||||
| 1 | Atmospheric glow and clouds | `earth.js` | Atmosphere / cloud spheres | Not in normal object selection path | Cloud layer controlled by the "Cloud Layer" toggle. |
|
||||
| 1 | Submarine cables | `cables.js` | `CABLE_CONFIG.line.renderOrder` | Cable picking path | Preserves existing cable layer level. |
|
||||
| 1 / 1.1 | Atmosphere / clouds | `earth.js` | One atmosphere shell at R×1.025; clouds at R+3 | Atmosphere uses BackSide, depthWrite=false, depthTest=true; cloud toggle retained | Inner scattering shares the surface pass; no fullscreen post-processing. |
|
||||
| 1 | Submarine cables / landing points | `cables.js`, `cable-batches.js` | All cable segments share one `LineSegments`; all landing points use instanced billboards; render order remains `1`; cables and landing-point anchors share the HD surface offset `0.48` and the same triangle-surface coordinates | Original `Line` / `Sprite` objects retain per-item picking and selection state but no longer draw individually; landing-point sphere occlusion remains, with `depthTest: false` on the batch | Style textures and instance attributes preserve color, pulse, size, visibility, and click behavior. |
|
||||
| 1.2 | Real terrain | `earth.js`, `terrain.js` | `TERRAIN_CONFIG.baseRadiusOffset` plus terrain displacement | Raycast disabled | Terrain overlays HD texture; temporarily hidden when HD texture is off, restores to prior state when re-enabled. |
|
||||
| 2.05 | Grid lines | `earth.js` | `CONFIG.earthRadius + 0.14` | Raycast disabled | Low-opacity lines over HD texture. |
|
||||
| 2.2 | Country borders | `country-boundaries.js` | `lineAltitudeOffset = EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET = 0.48`; claim lines have no extra lift | Raycast disabled | Line geometry still has its own `renderOrder`, but it shares the exact same radius as the HD texture shell to avoid parallax while the globe rotates. |
|
||||
| 2.29 | Country border hover glow | `country-boundaries.js` | Hover radius + glow offset | `depthTest: false`, raycast disabled | Additive glow to reinforce border edge and terrain hover visibility. |
|
||||
| 2.3 | Country border hover line | `country-boundaries.js` | `hoverAltitudeOffset = EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET = 0.48` | `depthTest: false`, raycast disabled | Neon red-orange hover line; aligned with the normal borders and HD texture shell to avoid ghosting or floating; China and Taiwan share the same highlight group. |
|
||||
| 0.97–0.99 | Country borders | `country-boundaries.js` | `lineAltitudeOffset = EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET = 0.48`; claim lines have no extra lift | Raycast disabled | Line geometry still has its own `renderOrder`, but it shares the exact same radius as the HD texture shell to avoid parallax while the globe rotates. |
|
||||
| 0.98 | Country border hover glow | `country-boundaries.js` | Same surface mapping, no extra lift | `depthTest: true`, transparent render queue, raycast disabled | Additive glow to reinforce border edge and terrain hover visibility. |
|
||||
| 0.99 | Country border hover line | `country-boundaries.js` | `hoverAltitudeOffset = EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET = 0.48` | `depthTest: true`, transparent render queue, raycast disabled | Neon red-orange hover line; aligned with the normal borders and HD texture shell to avoid ghosting or floating; China and Taiwan share the same highlight group. |
|
||||
| 3 | Satellite footprint fill / Iridium coverage ring | `satellites.js`, `iridium-footprint-adapter.js` | `GROUND_FOOTPRINT_RENDER_ORDER` | depth-tested; Iridium adapter fill / ring use the same renderOrder | Footprint above land / texture / terrain and country borders, below compute centers and satellites. |
|
||||
| 3-4.5 | BGP collectors, event rings, and event markers | `bgp.js`, `interactable.js` | Collector markers use `BGP_COLLECTOR_RENDER_ORDER = 4.4`; event markers use `BGP_EVENT_RENDER_ORDER = 4.5`; overlays remain in BGP-owned groups | Screen-space Interactable picking; BGP collectors/events may use collision avoidance where appropriate | BGP collectors share the surface-facility band with vessels; BGP events share the compute-center band. |
|
||||
| 4.3 | AIS vessel track lines | `vessels.js` | `VESSEL_RENDER_ORDER - 0.1`; `CONFIG.earthRadius + VESSEL_CONFIG.track.altitudeOffset` | Follows vessel selection, not independently picked | Recent track for the selected vessel, below vessel markers. |
|
||||
| 4.4 | AIS vessel markers | `vessels.js`, `interactable.js` | `VESSEL_RENDER_ORDER`; business radius `CONFIG.earthRadius + VESSEL_CONFIG.altitudeOffset`; normal markers are bucketed `THREE.Points`, hover / locked states are single-point overlays | `depthTest: true`; `main.js` uses screen-space picking for front-facing markers; `cluster: false` and `avoidance: false` | Moving vessels use triangular point textures, anchored / slow vessels use dots. Dense waterways may overlap and do not participate in dynamic screen clustering, avoiding Points rebuilds during rotation. |
|
||||
| 4.5 | Compute centers | `compute-centers.js`, `interactable.js` | `COMPUTE_CENTER_RENDER_ORDER` | Screen-space Interactable picking, with surface-icon collision handling | Surface facilities, below satellites. |
|
||||
| 5 | Satellite background dot | `satellites.js` | Fixed renderOrder; by default TLE/SGP4 altitude is compressed to `CONFIG.earthRadius + 4..25`; with real altitude disabled or propagation failed, uses `fallbackAltitudeOffset = 8` | Screen-space satellite picking | Below satellite dots. |
|
||||
| 6 | Satellite dots | `satellites.js` | Same compressed / fallback height as satellite backdrop dots | Screen-space satellite picking | Satellite dots above footprints and compute centers. |
|
||||
| 12+ | Satellite locked ring, halo, predicted orbit | `satellites.js` | `SATELLITE_CONFIG.overlayRenderOrder` and offsets; predicted orbit follows the same real-altitude toggle and fixes the lock-time globe pose to draw a closed inertial orbit; returns to same-sphere mode when real altitude is disabled | Satellite overlay path | Used for selected/locked satellite emphasis. |
|
||||
| 6 | Satellite dots | `satellites.js` | SGP4 altitude compressed to R+4..25; fallback R+8 | Screen-space satellite picking | Satellite dots above footprints and compute centers. |
|
||||
| 12+ | Satellite locked ring, halo, predicted orbit | `satellites.js` | `SATELLITE_CONFIG.overlayRenderOrder` and offsets; predicted orbit follows the same real-altitude toggle and fixes the lock-time globe pose to draw an open full-period inertial curve with forward-flow highlights; returns to same-sphere mode when real altitude is disabled | Satellite overlay path | Used for selected/locked satellite emphasis. |
|
||||
| 98-100 | Sun / moon halo and sprite | `celestial.js` | Fixed renderOrder | Celestial picking disabled | Foreground celestial sprites. |
|
||||
|
||||
## Full-set rendering and updates
|
||||
|
||||
- Batching changes GPU submission, not the number of satellites, cable segments, or landing points. It does not filter data by the visible hemisphere. Every adjacent cable vertex pair remains present, and picking still returns the original business object.
|
||||
- Normal satellites use one hard-disc `Points` batch without a backdrop batch. Optional breathing remains in the vertex shader, defaulting off; category filters and hover/locked masking share the same visibility state.
|
||||
- `satellite-position-worker.js` uses the same Three.js / SGP4 versions as the main thread. `satellite-propagation.js` owns shared orbit, display-altitude, and fallback calculations. Full positions and initial trail samples use transferable arrays; only one calculation is in flight. Reload, clear, and altitude-mode changes terminate the previous Worker.
|
||||
- Worker startup failure or `SATELLITE_CONFIG.workerStartupTimeoutMs` expiry falls back to the shared synchronous calculation. Counts, APIs, the existing update interval, and trail length are unchanged.
|
||||
- Verify full draw ranges, per-item picking, locked overlays, trails, rear-side occlusion, visibility toggles, and resource release after clearing. Frame-time comparisons require identical data, view, and resolution.
|
||||
|
||||
## Toggle Behavior
|
||||
|
||||
| Toggle | Behavior |
|
||||
@@ -65,7 +74,35 @@ Maintenance rules:
|
||||
| --- | --- |
|
||||
| Earth coordinate hover | When HD texture is visible, uses the HD texture overlay as the surface picking target; otherwise uses the Earth base sphere. |
|
||||
| Country border hover | Converts surface pick coordinates to lat/lon, then uses GeoJSON point-in-polygon; the border hover line itself does not receive raycasts. |
|
||||
| Country border hover visual | On hover, dims normal border lines and draws no-depth-test glow and solid lines. |
|
||||
| Country border hover visual | On hover, dims normal border lines and draws depth-tested glow and solid lines below landing points. |
|
||||
| China / Taiwan hover | `CHN` and `TWN` are grouped in the same hover highlight group; the tooltip still shows the actually-hit feature. |
|
||||
| Terrain | Acts as a visual layer only; `terrain.raycast` is disabled. |
|
||||
| Satellites | Uses screen-space satellite picking to prevent footprints or surface layers from blocking satellite clicks. |
|
||||
|
||||
### Surface attachment and landing point order
|
||||
|
||||
`earth-surface.js` projects geographic vertices onto the high-resolution SphereGeometry UV triangles at geometry construction time. Boundaries, coastlines, claim lines, hover outlines and landing points share this surface mapping. Long edges are subdivided using the surface grid spacing. Landing-point anchors now use the texture shell offset of 0.48; cable line offsets remain 0.2. The draw order is texture 0.96, boundaries/highlights 0.97–0.99, then landing points 1. Hover outlines always use the transparent queue with depth testing, so they do not draw through the globe or over landing markers. Geography is unchanged, and cached geometry is not rebuilt during rotation.
|
||||
|
||||
|
||||
### Startup loading versus drawing order
|
||||
|
||||
The selected full-resolution surface loads before independent data requests; boundary construction no longer blocks its first frame. This does not change renderOrder, depth behavior, geometry precision or image resolution. Data refresh retains the surface, with texture disposal limited to successful replacement or scene teardown.
|
||||
|
||||
### Loading-state boundary diamond regression
|
||||
|
||||
`v0.58.0` moved land fill to R+0.32 and HD imagery to R+0.48, but left the tint placeholder at R+0.04 with a 64×32 mesh. Its chords intersected the 128×128 base, producing latitude-ring diamonds until loaded imagery covered them. The old geometry penetrated the base by approximately 0.153 at tested samples.
|
||||
|
||||
Base, tint, land fill and HD surface now use `createEarthSurfaceGeometry()` and one UV triangulation. Tint and land share R+0.32; neither writes depth, so their existing renderOrder lets loaded fill replace the tint. HD remains R+0.48. Occluder, terrain, clouds, picking and geographic display policy are unchanged. Do not mask intersecting geometry by changing renderOrder alone, disabling depth tests or hiding the loading globe.
|
||||
|
||||
`earth-surface.test.js` ray-tests the actual loading tint and verifies at least two 24-bit depth steps of separation at 50%, 100% and 500%. Browser checks cover delayed textures, boundary toggles, rotation/zoom, and land/HD completion.
|
||||
|
||||
### Surface marker alignment and limb visibility
|
||||
|
||||
Cables and landing points share R+0.48, the `earthSurfacePoint` triangle projection and renderOrder=1. Related landing halos use an emphasis attribute in the existing batch, adding no Sprite or draw call. Rounded badges use GPU anchor ray/sphere occlusion: draw the entire billboard when its anchor is visible, otherwise clip it entirely. CPU picking uses the same test to reject rear anchors. Only `icon.badge` layers (including compute and collectors; vessels independently use heading arrows) use this path. The selected BGP event's static overlay halo uses the event icon's R+0.48 anchor, foreground order and whole-anchor front/rear visibility; the existing animated BGP event ring remains a separate effect with its original behavior.
|
||||
|
||||
|
||||
The `.map-clusters` projection layer uses z-index 8 below HUD and whole-marker occlusion from real surface anchors. Ordinary points retain their Three.js layers; transitioning members render only once in the projection layer. Spider legs use widened transparent hit areas; hover highlights the matching line and child, and both clicks resolve the same business object. Wheel input over projected markers and legs continues to zoom the globe.
|
||||
|
||||
Co-located markers share a compact projection stack across categories; screen collisions never merge different locations. A selected child uses projection `z-index: 1000`, while the locked overlay for an unclustered marker uses the shared foreground `renderOrder >= 100`, so focus stays on top across layers; clearing focus restores stable ID order. The aggregate badge stays below a selected child during its opening tween. `icon_presentation_screen` lets detail connectors follow display offsets while markers retain their real anchors. Ownership changes immediately rebuild the corresponding GPU batch.
|
||||
|
||||
Regional news and compute cards share the `#earth-region-callouts` spatial root (z-index 48), with independent data and reading state. News connectors use representative regional anchors, unchanged by story rotation. One controller excludes all owned cards when measuring HUD obstacles and tracks size changes by element. Mobile news cards live in the drawer without map connectors.
|
||||
|
||||
@@ -1,5 +1,9 @@
|
||||
# Admin Frontend Context
|
||||
|
||||
## Compute comparison configuration
|
||||
|
||||
The Earth content `compute_comparison` section mounts `ComputeComparisonSettings.tsx`. Structured forms maintain report sources, annual country observations and innovation gaps using the shared Panel, Input, Select, Button and confirmation dialog; operators do not edit JSON. Drafts survive section switches, full-page unload warns about unsaved work, and discarding a draft or restoring defaults requires confirmation. Management data loads on activation. Publishing sends the backend revision to prevent overwriting concurrent edits, and failures preserve the draft. The content area scrolls internally while the heading and publish controls stay in the first viewport. `compute-comparison-model.ts` owns types and same-origin update notifications; `compute-comparison.css` only owns this section's layout.
|
||||
|
||||
This document describes the current real structure of the console frontend. The goal is to help future page development, table refactoring, layout governance, and state consolidation quickly find the right entry points.
|
||||
|
||||
Related references:
|
||||
@@ -71,6 +75,8 @@ This keeps Earth, AI, collection management, and other multi-section pages from
|
||||
|
||||
## Datasource Collection Queue
|
||||
|
||||
`datasource-task-queue.ts` reconciles attempts by task ID. Retrying a source adds a new attempt and preserves failed/completed/cancelled history; source matching is allowed only for active placeholders without an ID. Late events cannot reactivate terminal attempts, and older task polls cannot overwrite the latest datasource task. Task status uses `ListChecks`; refresh retains `RefreshCw`.
|
||||
|
||||
The Admin datasource page routes single-source trigger, table-selected trigger, and trigger-all into a browser-download-list style collection queue:
|
||||
|
||||
- Queue state is managed by [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx). It is a current-session visibility layer and does not fake task history in `localStorage`.
|
||||
@@ -389,7 +395,7 @@ Current boundary:
|
||||
- `TV Livestream` owns the Earth media-panel source configuration.
|
||||
- `Branding` owns Earth HUD brand assets. `Logo URL` and `Title Image URL` use inline upload controls inside the fields; the upload buttons keep the primary `TactileButton` style, and dropping an image onto the matching field shows a low-saturation drag reaction instead of the old global asset-picker toolbar.
|
||||
- `Boundary Precision` owns the Earth static boundary asset state: provider, low-precision fallback, high-precision manifest/PMTiles, source JSON, and build action.
|
||||
- `Base Map`, `Layer Resources`, `3D Assets`, and `News Anchor Strategy` are placeholders only. They show module status and do not invent fake APIs or fake data.
|
||||
- `Base Map`, `3D Assets`, and `News Anchor Strategy` are placeholders only. They show module status and do not invent fake APIs or fake data.
|
||||
|
||||
Do not add Earth experience resources or collection-lifecycle tabs back into `/settings`; collection belongs to `/collection-management`, and Earth display resources belong to `/earth-content`.
|
||||
|
||||
@@ -448,3 +454,11 @@ Therefore:
|
||||
For Earth-related structure, see:
|
||||
|
||||
- [Earth Frontend Context](/home/ray/dev/linkong/planet/docs/technical/en/earth-frontend-context.md)
|
||||
|
||||
## Detail-card image workspace
|
||||
|
||||
`CardImageSettings.tsx` replaces the `layer_resources` placeholder in Earth Content, retaining the shared page frame, section navigation, controls and confirmation dialogs. Its persistent component preserves drafts across section changes. List and editor own their scrolling with a constrained height chain; narrow layouts stack these regions. `CardImagePreview.tsx` reuses the actual Earth mask stylesheet.
|
||||
|
||||
The image library supports upload/URL entry, attribution, licensing, enabled state and crop focus. Referenced assets cannot be deleted until assignments are removed. Assignments use database-backed object/model/group/category choices and support edit, disable, removal and explicit per-object suppression. Draft preview calls the same resolver used after publication. Only Save publishes; conflicts retain drafts, and the editing fieldset is disabled during save/upload. Uploading alone does not publish an asset.
|
||||
|
||||
Under `/earth/card-images`, configuration GET/PUT, upload POST, targets GET and preview POST require administrators. Resolve and content-addressed files are public read-only routes. Configuration writes use revision checks and audit transactions.
|
||||
|
||||
@@ -16,6 +16,10 @@ The refactored rules:
|
||||
- Earth renders only city-level or better locations.
|
||||
- Local JSON registries are not runtime candidate sources for compute centers or BGP collectors; persisted location facts live in database dimension tables.
|
||||
|
||||
For compute-center display, `precise` and `site` candidates keep their source coordinates. `city`, `estimated`, and `needs_confirmation` records that share a coordinate are separated only in `displayLatLon`: a stable hash of `source` and `source_id` selects an irregular cloud direction and a radius in the 0.4–6 km range. The display offset is deterministic, is neither a circle, grid, nor spiral, and never uses a database row ID or list order. Source coordinates are never written back. Tooltips and detail cards label these points “City-level approximate location (dispersed display)”.
|
||||
|
||||
The resolver priority is confirmation cache, source precise coordinate, official registered site, source city/site, then unconfirmed cache. A city-level approximation must never be presented as `precise` or as a guessed headquarters. Official registration helpers are keyed by exact `source` + `source_id`; the current entries cover Gyoukou 2.0/2.2 at JAMSTEC Yokohama and Sophia ALCF (three records across two campuses). Their official access-map centers are site points only, not rack/building survey points, and each candidate keeps the official URL in `source_note`.
|
||||
|
||||
## Core Interfaces
|
||||
|
||||
```python
|
||||
@@ -51,7 +55,7 @@ class LocationResolver(Protocol):
|
||||
def resolve(self, query: LocationQuery) -> ResolverOutput: ...
|
||||
```
|
||||
|
||||
`LocationPipeline.collect_candidates()` returns sorted candidates plus `attempted_queries`; `resolve_best()` returns the best candidate with diagnostics. The default sort key ranks source, precision, and confidence, then deduplicates candidates with the same source and rounded coordinates.
|
||||
`LocationPipeline.collect_candidates()` returns sorted candidates plus `attempted_queries`; `resolve_best()` returns the best candidate with diagnostics. The default sort key ranks source, precision, and confidence, then deduplicates candidates with the same source and rounded coordinates. Legacy records are reclassified through the same path as `country`, `region`, or `unknown`; non-renderable records remain in the complete `unresolved` list.
|
||||
|
||||
## Built-In Resolvers
|
||||
|
||||
|
||||
@@ -123,6 +123,9 @@ The default guide follows the BarentsWatch official tutorial and reminds you to
|
||||
|
||||
### AISStream Realtime Vessels
|
||||
|
||||
Once the vessel layer is open, subsequent AISStream and BarentsWatch positions update automatically. Selected vessels stay selected during updates, and reconnecting automatically reconciles the display without repeatedly toggling the layer.
|
||||
|
||||
|
||||
`AISStream Realtime Vessels` is the global AIS WebSocket collector. A passing connection test only confirms API key + endpoint format. Actual global vessel data requires the backend `aisstream_vessels` collector to stay connected and write to `ais_raw_observations`.
|
||||
|
||||
Steps:
|
||||
@@ -160,7 +163,11 @@ Providers and models accept presets or arbitrary custom IDs. Common fields:
|
||||
- Max Tokens, Anthropic Version: keep defaults if unsure
|
||||
- Timeout / Retry: timeout and retry attempts
|
||||
|
||||
The plug icon at the end of the Base URL input runs a connection test. A passing test echoes the model's short reply.
|
||||
The plug icon at the end of the Base URL input runs a lightweight connection check: it checks the proxy and model catalog, then confirms the selected model is listed. Use Playground afterward to verify generation access and the model's reply; the catalog check itself does not generate a response.
|
||||
|
||||
Use the refresh icon at the top right to update the available models. A successful refresh saves the catalog for later visits; catalogs with release dates show newer models first. Refresh preserves the current model, credentials, URLs, and unsaved edits. Select a model and click Save to change the model used by the application. A failed refresh displays an error and keeps the previous catalog.
|
||||
|
||||
Refresh uses the base URL, protocol, and API key currently entered in the form, so you can fill new configuration and refresh before saving. Credentialed providers require a key for the matching service region; an empty Ollama catalog means no models are installed. Selecting a model with a known protocol also updates the protocol field. Custom OpenAI gateways retain the manually selected protocol.
|
||||
|
||||
### Tools
|
||||
|
||||
@@ -197,7 +204,7 @@ TV livestreams and boundary precision moved to `/earth-content`; collectors and
|
||||
- **TV Livestream**: manages sources shown in the Earth media panel.
|
||||
- **News Content**: browses news grouped by RSS source and manual group. RSS items remain read-only; manual groups support create, JSON import, edit, delete, and reprocess.
|
||||
- **Boundary Precision**: shows the current provider, low-precision fallback, high-precision PMTiles/manifest status, local source JSON, and manual build action.
|
||||
- **Base Map**, **Layer Resources**, **3D Assets**, and **News Anchor Strategy**: placeholder tabs for future configuration. They do not display fake data.
|
||||
- **Base Map**, **3D Assets**, and **News Anchor Strategy**: placeholder tabs for future configuration. They do not display fake data.
|
||||
|
||||
The Earth page settings gear also includes Boundary Precision. Switching to High Precision starts a local background download/build, like a game update package, when no high-precision asset exists yet. Progress is shown as a percentage, and the result applies automatically after success without a page reload. Switching back to Low Precision only changes the local display preference.
|
||||
|
||||
@@ -262,6 +269,8 @@ Earth at `http://localhost:3000/earth` is the public 3D situational page; no log
|
||||
|
||||
A single globe view of: BGP events and observations, satellites and tracks, cables and landing points, compute centers, country boundaries / graticules / high-res tiles / clouds / terrain, news live streams and situational news, search and focus details.
|
||||
|
||||
At 500% zoom the viewpoint remains outside the surface. Mouse or single-finger dragging keeps the grabbed surface under the pointer, with auto-rotation suspended while held. Release retains inertia; pausing before release stops it. After a pinch, the remaining finger can continue dragging immediately.
|
||||
|
||||
### Layer Control
|
||||
|
||||
The right-side layer panel toggles layers. Common layers: graticule, country boundaries, high-res tiles, atmospheric clouds, cables, compute centers, BGP, satellites, AIS vessels, terrain. Satellite tracks are no longer listed as a standalone layer; they are controlled from Settings.
|
||||
@@ -272,21 +281,45 @@ Dependencies:
|
||||
- Tracks depend on satellites
|
||||
- With high-res tiles disabled, the globe shows the base map with edge highlighting
|
||||
|
||||
The base-map selector defaults to Original basemap, listed first. The display menu can switch among the available map sources and saves the selection locally. The existing Natural Earth I source and an “Original basemap” option remain available; the archived option is a byte-identical copy of the September 21, 2026 archive; NASA 21K is source archive material only, while runtime delivery stays at one 8K texture. Loading replaces the previous GPU texture only after success, releases the old texture then, keeps it on failure, and ignores stale responses. The switch does not add a rendering layer.
|
||||
|
||||
### Legend
|
||||
|
||||
The bottom-left legend follows the focused or enabled layer. Covered: cables, satellites, country boundaries, compute centers, BGP, AIS vessels.
|
||||
|
||||
AIS vessel legend colors by type: cargo, tanker, passenger, fishing, military, moored/slow, other. Triangles indicate moving vessels; dots indicate moored or slow targets.
|
||||
|
||||
Click legend entries to hide or restore their route, category or visual effect; hidden entries appear dimmed. Each layer retains its filter state for the current page. Selecting another object or enabling a layer switches to its legend without a separate filter button. Hiding a selected object dismisses its details and coverage; explicitly selecting a hidden object through search reveals its category. Filtering does not change total statistics. Vessel-type and anchored/slow filters compose; geographic fill controls are visible when the base map is shown.
|
||||
|
||||
Satellite names and NORAD identifiers appear in a limited set at 400% zoom and above, with point size growing smoothly with zoom. Settings → Display → Show FPS displays actual frame rate beside the brand; hover for an approximately 30-second average. The toggle persists. The radial menu and zoom submenu follow interface DPI scaling; moving from the magnifier across the gap to the plus/minus controls keeps the submenu open.
|
||||
|
||||
### Search
|
||||
|
||||
Search finds cables, landing points, satellites, compute centers, BGP events, BGP observers. Results jump to and focus the object.
|
||||
|
||||
Click a BGP, compute, vessel, satellite, or cable object on the globe to draw a connector from the object before revealing a nearby detail card. The card follows the source direction toward the nearest usable screen edge, leaving breathing room around the globe; it never covers its own connector exit point. As the globe rotates, the source moves smoothly toward a visible part of the highlight. Fully hidden objects retain an outline direction cue with a lighter line and hollow source point. Cards stay readable and remain draggable; their connection points adjust along the edge automatically. Mobile shows a compact popup first, with full details available in the drawer. Dropdown menus and tool lists retain their existing opening behavior.
|
||||
|
||||
### Four regional news cards
|
||||
|
||||
Open the news panel from the top ticker, then choose “Show regional news cards” in its header. Four cards cover the Americas, Europe, Middle East & Africa, and Asia-Pacific. They remain visible after the panel closes, and the switch is remembered. Its background and border highlight while enabled; click again to turn it off.
|
||||
|
||||
Each card rotates only displayable stories from its own region after its typewriter summary finishes and a seven-second reading interval elapses. Hovering, keyboard interaction, dragging or collapsing pauses that card while other regions continue. Controls provide previous/next, pause/resume and a source link. Refresh preserves the current story where possible. Empty regions stay empty instead of borrowing unrelated stories.
|
||||
|
||||
Cards reuse the cruise-news gold header and 📰 icon, preserve full summaries and simplify source metadata, with the compute cards' dragging, collapsing and automatic placement. News and compute cards share one placement plan. Connectors identify representative regional anchors, not precise event locations. On mobile, enable “Region cards” from the news page and read them in the drawer.
|
||||
|
||||
### Choosing a News Live Stream
|
||||
|
||||
Open the live tab in the media panel and click the current channel to open the search menu. Search covers the complete catalog. The list loads 50 channels at a time and appends another page when you scroll to the bottom. A fixed footer shows the loaded and matching channel counts; a failed request can be retried below the list. The default source is Al Jazeera Mubasher, using its HLS playback URL. Other channels remain subject to availability of their playback service.
|
||||
|
||||
Buffering and recoverable playback errors do not switch channels. A manually selected channel remains selected for the current page session; refresh to retry after failure. An automatically started channel may switch only after fatal recovery is exhausted, automatic fallback is enabled, and a playable backup exists. The bundled CCTV-4 source uses WCETV's directly playable America-edition HLS stream. The edition is identified in the channel name because its schedule may differ from the Asia edition. The old bundled CDRM URL is no longer passed to the generic decoder; other custom CCTV-4 URLs remain unchanged.
|
||||
|
||||
Source information briefly expands when a channel loads, then collapses with the existing HUD animation. Manual toggles use the same component. Expanded height is measured from actual content; panels dragged to the bottom retain their bottom anchor while animating upward.
|
||||
|
||||
### Coordinate Candidate Collection
|
||||
|
||||
Compute center and BGP observer detail cards support automatic coordinate-candidate collection. Click the object then use "Collect Coordinate Candidates" or "Re-collect Coordinates". The backend assembles candidates from source coordinates, public-org registry APIs, and online geocoders. When regular sources have no candidate, the current default AI Provider runs one LLM factcheck fallback. BGP observers' stored coordinates only fill query context; they are not returned as candidates.
|
||||
|
||||
Candidates preview on Earth directly. Saving a compute-center candidate writes to the `compute_center_locations` dimension table and refreshes the layer immediately. The notification badge at the top-left of the compute-center layer shows the unresolved count; clicking it opens the queue, supports single collection, or "Adopt All" to save the top-confidence candidates from top to bottom. Records without candidates stay in the queue rather than being faked to country centroids.
|
||||
Candidates preview on Earth directly. Saving a compute-center candidate writes to the `compute_center_locations` dimension table and refreshes the layer immediately. Precise and site candidates keep their source coordinates. City-level, estimated, and confirmation-required records that overlap are dispersed only for display using a stable source + source ID placement; the source coordinate is never changed. Their tooltip and detail card say “City-level approximate location (dispersed display)”. The notification badge at the top-left of the compute-center layer shows the unresolved count; clicking it opens the queue, supports single collection, or "Adopt All" to save the top-confidence candidates from top to bottom. Records without candidates stay in the complete queue rather than being faked to country centroids or headquarters.
|
||||
|
||||
Recommended single-object flow:
|
||||
|
||||
@@ -298,9 +331,11 @@ Recommended single-object flow:
|
||||
|
||||
Adopt All is for batch processing the compute-center unresolved queue. It starts from the top and adopts the highest-confidence candidate. Records without factual support remain in the queue. When WebSearch is disabled, single locate and Adopt All are disabled because location validation depends on factual lookup.
|
||||
|
||||
The batch continues within the current Earth page when you close the candidate panel, inspect another object, or switch browser tabs. Reopening the queue restores progress and results. Saved entries are not collected again, and a ✓ entry beside the layer remains available after completion. Refreshing, closing, or navigating away from Earth interrupts unfinished work; coordinates already saved remain stored.
|
||||
|
||||
### Settings
|
||||
|
||||
The settings panel is grouped into Runtime, Display, Panels, Motion, Shortcuts, and System. It covers rotate / cruise / motion mode, cruise modules (BGP/news/compute centers/vessels/cables/satellites), view (satellite display style, hover tooltip, satellite idle breathing, real satellite altitude, track display, compact dots, day-night mode, panel toggles), motion debug mode / input source / skeleton-only / recognized-gesture whitelist, shortcut enablement and remapping, default globe size, terrain opacity, reset.
|
||||
The settings panel is grouped into Runtime, Display, Panels, Motion, Shortcuts, and System. It covers rotate / cruise / motion mode, cruise modules (BGP/news/compute centers/vessels/cables/satellites), view (satellite display style, hover tooltip, satellite idle breathing, real satellite altitude, track display, day-night mode, panel toggles), motion debug mode / input source / skeleton-only / recognized-gesture whitelist, shortcut enablement and remapping, default globe size, terrain opacity, reset.
|
||||
|
||||
News categories use the same chip selector as Cruise Modules. They only filter the news panel and news cruise items in the current browser; they do not affect layers, TV, data points, basemap, boundaries, collector jobs, or admin news-source configuration.
|
||||
|
||||
@@ -386,3 +421,51 @@ Docs supports: category navigation, Markdown rendering, tables and code blocks,
|
||||
- [Quickstart](/home/ray/dev/linkong/planet/docs/technical/en/quickstart.md)
|
||||
- [FAQ](/home/ray/dev/linkong/planet/docs/technical/en/faq.md)
|
||||
- [Intelligent Planet Ops Runbook](/home/ray/dev/linkong/planet/docs/technical/en/ops-runbook.md)
|
||||
|
||||
## Compute intelligence
|
||||
|
||||
Administrators maintain card data in **Earth content → Compute comparison**. Add a report title, URL, publisher, report year and page/figure locator under Sources, then add country values by observation year, metric and source under Observations. Save and publish applies the complete configuration. Global shares use the world denominator; absolute values use FP32 EFlop/s. Bilateral shares are derived from same-year US and China observations, and regions require every member. Do not mix FP16/FP64 capacity, regional estimates or facility counts. Omit missing countries; enter zero only when the source explicitly reports zero.
|
||||
|
||||
Innovation gap observations remain independent US-minus-China values. Referenced reports cannot be removed until their observations are removed or reassigned. Duplicate metric/year rows, missing sources and global shares totaling over 100% are rejected. Concurrent edits return a conflict while preserving the local draft. Restore defaults requires confirmation and restores the bundled report data. Publishing and restoring are audited. Same-origin Earth pages receive a refresh notification; other visible pages check once per minute and hidden pages check when visible again. Reports must still be reviewed manually; configuring a URL does not scrape a report.
|
||||
|
||||
Open the analysis icon beside Compute Centers, then enable Show region cards. Connectors draw to their target positions before the cards fade in; subsequent globe motion does not replay entrance. Closing the configuration window or pressing Escape leaves the cards enabled; the analysis icon reopens configuration, while the switch disables the cards. Anchors rotate with Earth while cards remain readable. Dragging or expanding grants reading time; after a substantial change in geographic direction, cards can move together when that materially reduces crossing or detours. Active interaction prevents movement, while manual positions remain preferences rather than permanent pins. Rear regions retain their state without a connector drawn through the globe. Existing layer, news, and media windows remain available.
|
||||
|
||||
The picker groups the canonical comparison catalog as China, US, Europe, APAC excluding China, and Other; the groups are display buckets, not statistical denominators. It offers period and metric selection. Bilateral share still uses US + China as its fixed denominator; unavailable measures stay missing for other participants. Regional anchors are label positions, never facility locations. Source buttons open report evidence and facility buttons reuse the existing detail card. Mobile uses a compact regional overview in the drawer. Each row is an accordion: tapping the same header collapses it, tapping another collapses the old row, and at most one long card is open. The expanded card keeps trends, region members, facilities, media, and sources; there is no region subpage. Settings remain a separate on-demand page.
|
||||
|
||||
The default national metric is global compute-device scale, covering basic, intelligent, and supercomputer capacity with observations from 2020–2023. The reviewed absolute observations are 302 for 2022 (`caict2023` evidence) and 435 for 2023 (`caict2024` evidence). Japan's global-share observations are 6%/5%/4%/4% for 2020/2021/2022/2023. Product groups APAC and OTHER do not mean the report's Japan or “other” categories, and EUROPE remains missing until its membership matches the report. Bilateral share always uses US + China as its denominator. A missing primary value may show the latest reported year (currently 2023); explicitly selecting 2026 still shows missing and does not backfill. Unconfirmed comparability does not produce year-over-year change. National report metrics remain separate from public facility samples; coverage counts remain auxiliary evidence, never computed capacity or a global coverage rate. TOP500 and Epoch may overlap, so source-record counts are not a global census. The bundled reports were reviewed on 2026-09-22; published overrides use their maintained review date and observations. An independent Epoch share mode is not implemented.
|
||||
|
||||
Select up to six compute participants sharing peripheral space with four regional news cards. Full cards use the same DPI-scaled 340px base width and retain business metrics, years, trends and sources; missing values remain explicit. Cards stay near screen edges while globe rotation moves only connectors. When space is limited, full cards remain visible with some peripheral overlap. On release, valid drops retain the new position. Out-of-bounds or central drops settle nearby, without returning to the pre-drag slot. Peripheral overlap is allowed. Cards have no manual reset action; positions and sizes remain persistent. News rotation and typing do not repeatedly push neighboring cards; long content scrolls internally.
|
||||
|
||||
When the map cards switch is off, the overview summary still loads. Turning it on shares the map loading event; a newer request cancels the previous one. Drawer event handling follows `composedPath()` so replacing inner HTML cannot accidentally collapse the active row. The shared HUD collapsible panel animation is reused and respects reduced-motion preferences. Facility names remain in a scroll region capped at 240px; rows are left-aligned, wrap long names, and retain at least 42px of touch height.
|
||||
|
||||
## Surface and object appearance
|
||||
|
||||
The real surface retains all four map choices. Day/night shading follows the sun, with uniform illumination when disabled. Selected cables show traveling pulses and landing points use amber micro-markers. Facilities retain category symbols, with numbered solid circles for aggregates. Satellites retain self-glow/ground-coverage modes, trails and their existing orbital motion. BGP severity, vessel heading, related highlights, detail cards and mobile drawers retain their established interactions.
|
||||
|
||||
### Satellite orbits and collection attempts
|
||||
|
||||
Selecting a satellite shows an open predicted arc with highlights moving in the travel direction. Satellites, trails and footprints remain synchronized. Coverage stays blue across day/night; particle and orbit category colors match the legend. Dragging continues outside the globe silhouette, with reduced sensitivity at extreme zoom. The TV panel has square corners like other cards.
|
||||
|
||||
Retriggering a datasource creates a new task while older failures remain in the failed group. Task Status uses a checklist icon, distinct from Refresh. CelesTrak can reuse valid recent download cache; each new collection attempt still gets its own task and updates collected data.
|
||||
|
||||
|
||||
Display → Satellites → Close-up labels defaults on; turning it off hides satellite map names while retaining detail cards. The zoom-threshold slider below the switch ranges from 250% to 500% (default 400%); labels appear at or above the selected zoom and hide below it. The count slider sets a saved limit of 5–36 labels (default 36); occlusion or limited space may result in fewer visible labels. Sky brightness ranges from 0–200%, defaulting to 100%. Both choices persist. Compute region cards wait for drag/globe motion and new-card placement to settle before necessary avoidance. Otherwise they float subtly around stable positions and pause during interaction. Compute legend category counts describe currently located markers.
|
||||
|
||||
|
||||
Surface aggregates appear as solid numbered circles. Click to expand members; only co-located or extremely close geographic markers form compact stacks at their location, with exposed clickable edges. Ordinary markers stay at their real projected positions. Selecting a rear icon brings it forward and opens its details; clearing focus restores its original position. Ordinary groups split above 250%; no aggregate badge remains at 500%, and co-located markers expand directly.
|
||||
|
||||
Object detail cards show identity, business properties and update times without rendering diagnostics, collection channels or coordinate-matching internals. Facility location precision and verification status, BGP alert confidence, and news publisher, publication time and summary remain visible.
|
||||
|
||||
Detail popups share the news-card height cap. Headers remain fixed while long bodies scroll; key event information and device specifications precede location verification, provenance and related objects.
|
||||
|
||||
### Detail-card images
|
||||
|
||||
Open Earth Content → Layer resources → Detail card images. Upload a still JPEG, PNG or WebP, or enter an image URL. Set its title, source, credit, license and crop focus; the preview uses the actual feathered card background.
|
||||
|
||||
In Assignments, select satellites, compute facilities, cables, landing stations, vessels, BGP incidents, BGP collectors and custom points, then an object/model/group/category. Search selects current catalog objects and stable identifiers; missing model data does not produce invented choices. Apply an assignment to the draft, then Save to publish. Object rules take precedence over model, group and category rules. “No image for this object” suppresses all fallback images for that object.
|
||||
|
||||
Verify an object to see its draft match before publishing. Disabled assignments allow more general rules to match. Remove assignments before deleting an image. Conflicting saves preserve your draft.
|
||||
|
||||
Earth fades available imagery behind the existing detail data. Missing or failed images leave a complete data-only card. The header image icon links to the source and exposes attribution/license in its tooltip. Every map object type has a bundled default: exact and series images take precedence over explicitly labeled type illustrations. Starlink, BeiDou-3 and Qianfan are assigned; Qianfan retains its unverified permission metadata. Fresh installs need no setup. Upgrades merge missing defaults after services restart while preserving custom assignments and explicit suppression.
|
||||
|
||||
Compute legends and details support supercomputers, GPU clusters, data centers, cloud computing centers and edge nodes. Manage their defaults under Compute facilities → Category in image management. Types without collected records show a zero count. BGP collectors and incidents have separate default imagery.
|
||||
|
||||
@@ -60,6 +60,7 @@ This document standardizes terms used across Intelligent Planet, the console, ba
|
||||
| `traffic_stats` | 流量统计 | Cloudflare Radar traffic statistics |
|
||||
| `as_stats` | 自治系统统计 | Cloudflare Radar AS statistics |
|
||||
| `compute_center` | 算力中心 | Aggregated Earth compute-center object |
|
||||
| `compute_intelligence` | 算力与资源态势 | Compute intelligence: national observations, public samples, history, and sources |
|
||||
| `generic` | 通用数据 | Generic collector output |
|
||||
| `generic_records` | 通用记录 | Generic mapping output |
|
||||
|
||||
|
||||
@@ -1,6 +1,24 @@
|
||||
# `planet.sh` Startup Performance Optimization
|
||||
|
||||
## Background
|
||||
## Windows desktop launcher
|
||||
|
||||
The launcher prefers a new Windows Terminal window and explicitly runs PowerShell 7 (`pwsh.exe`), independently of the default Terminal profile. Without PowerShell 7 it falls back to Windows PowerShell; without Terminal it starts the available PowerShell directly. The CMD bootstrap also prefers PowerShell 7. Ordinary entries request elevation; entries already running as administrator still dispatch to the preferred terminal without another elevation request. The child validates its privileges and enters WSL directly, preventing recursive windows. The existing post-restart WSL shell is retained.
|
||||
|
||||
On first use, open the actual project folder in Windows Explorer and double-click the repository's `planet.cmd`. It derives the WSL distribution and Linux project directory from its own location, creates `Planet.cmd` on the current user's actual Windows desktop, then follows the normal restart flow. Both `\\wsl$\distribution\...` and `\\wsl.localhost\distribution\...` work. Local Windows-drive checkouts use the default WSL distribution and `wslpath`. Windows resolves the desktop location, including redirected desktops such as OneDrive; WSL, zsh and project dependencies must already be installed.
|
||||
|
||||
Subsequent desktop launches use the bound target, request the existing administrator elevation and call `planet.sh restart --allow-lan`. No test ports are embedded: defaults remain frontend 3000, backend 8000 and AI Provider 8010, subject to explicit `DEFAULT_*_PORT` environment overrides. Successful restarts open Earth using the actual `ports.env` values and retain the WSL shell; failures display their error without opening a success page. `scripts/windows/planet-launcher.ps1` owns path resolution, desktop installation and elevation; `scripts/windows/restart.zsh` only bridges to the existing lifecycle script.
|
||||
|
||||
Use `planet.cmd --install-only` on Windows to generate or validate a launcher without restarting services. Re-running the repository entry updates the same target's launcher. Other projects and unmanaged files are preserved by choosing a target-specific filename. After moving a checkout, regenerate from its new location; old launchers never guess another directory. Desktop launchers require the scripts in the checkout and are not standalone installation packages.
|
||||
|
||||
Do not copy the raw repository `planet.cmd` to the desktop by itself: first installation needs the adjacent `scripts/windows/` directory. On another computer, generate a new launcher from that computer's checkout instead of reusing an old desktop copy. Server paths such as `/home/linkong/planet` and personal workstation paths both work without machine-specific Git edits. Run under the Windows account that owns the WSL distribution; elevating under a different account may expose a different set of registered distributions. Installation itself does not require administrator privileges; elevation is requested for the actual restart.
|
||||
|
||||
Run Windows regression checks with `powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\harness\test_windows_launcher.ps1`. They cover target resolution, special characters, generated launchers and name collisions. `scripts/harness/test_windows_restart.py` uses disposable command substitutes to check restart arguments, browser opening after success and shell retention after failure. The latter runs in quick-check without starting real services.
|
||||
|
||||
## Startup background
|
||||
|
||||
Use `zsh ./planet.sh start --non-motion-agent` for daily startup; use `init` when preparing a new environment. When investigating latency, distinguish initial dependency downloads, container readiness, and application initialization using the stage timestamps.
|
||||
|
||||
Before preparing the AI Provider image, startup verifies the backend's actual database connection and recreates a missing port mapping once while preserving the volume. A terminated backend process or Uvicorn initialization, ASGI loading, import, or syntax failure stops waiting and identical retries immediately. Normally slow initialization keeps its existing timeout budget. AI Provider readiness probes the host `/health` endpoint directly, without waiting for Docker's first scheduled health check.
|
||||
|
||||
`planet.sh` manages start, stop, restart, health checks, and logs for all local services. The previous implementation had several startup issues:
|
||||
|
||||
@@ -37,27 +55,9 @@ write_ai_provider_build_stamp() {
|
||||
}
|
||||
```
|
||||
|
||||
### Faster Fingerprint
|
||||
### Fingerprint Scope
|
||||
|
||||
The previous implementation tarred the whole `aiprovider/` directory before hashing, which could take seconds in large trees. The new version uses `find + stat` and reads only file metadata:
|
||||
|
||||
```bash
|
||||
compute_ai_provider_build_fingerprint() {
|
||||
find aiprovider \
|
||||
-type f \
|
||||
! -path '*/__pycache__/*' \
|
||||
! -name '.env' \
|
||||
! -name '.env.*' \
|
||||
! -name '*.pyc' \
|
||||
! -name '*.pyo' \
|
||||
| LC_ALL=C sort \
|
||||
| xargs -r stat --format="%Y %s %n" 2>/dev/null
|
||||
sha256sum docker-compose.yml docker-compose.simple.yml 2>/dev/null
|
||||
python3 "$SCRIPT_DIR/scripts/compute_aiprovider_dependency_fingerprint.py" 2>/dev/null
|
||||
}
|
||||
```
|
||||
|
||||
This is roughly 10 times faster for many-small-file workloads while preserving the same practical rebuild signal. `.env` and `.env.*` are excluded because runtime model, key, and Base URL changes should not force an image rebuild.
|
||||
The fingerprint hashes file contents from `aiprovider/`, the Dockerfile, the root manifest and lockfile, and the provider dependency information. It does not traverse frontend assets or downloaded data. `.env` and `.env.*` are excluded because they are runtime configuration. The Dockerfile applies its fingerprint label after dependency installation so a changed build marker alone does not invalidate dependency layers.
|
||||
|
||||
### Docker Build Context
|
||||
|
||||
@@ -83,16 +83,18 @@ The Dockerfile copies only AI Provider inputs:
|
||||
```dockerfile
|
||||
COPY pyproject.toml uv.lock /app/
|
||||
RUN --mount=type=cache,target=/root/.cache/uv \
|
||||
uv sync --frozen --no-dev
|
||||
uv sync --frozen --only-group aiprovider
|
||||
|
||||
COPY aiprovider /app/aiprovider
|
||||
```
|
||||
|
||||
`uv sync` uses a BuildKit cache mount. The first build may still depend on network speed, but later builds reuse `/root/.cache/uv`.
|
||||
|
||||
The `aiprovider` dependency group in the root `pyproject.toml` uses the same `uv.lock` and installs only the API, HTTP client, settings, and ASGI runtime dependencies. The image excludes backend collectors and OpenCV / MediaPipe motion dependencies. The build fingerprint label comes after dependency installation and code copying, so a fingerprint change alone does not invalidate dependency layers. The container starts the installed `.venv/bin/python` directly, without runtime dependency synchronization. Dependency changes must update this group and the lockfile and validate image imports and `/health`.
|
||||
|
||||
### Runtime Configuration
|
||||
|
||||
Before starting AI Provider, `planet.sh` generates a current-user runtime env-file and passes it to Compose or the manual `docker run` fallback. The default path is `${XDG_STATE_HOME:-$HOME/.local/state}/planet/aiprovider_runtime.env`. Configuration priority:
|
||||
Before starting AI Provider, `planet.sh` generates a current-user runtime env-file and passes it to Compose; a failed Compose operation stops with its error instead of falling back to a manual launch. The default path is `${XDG_STATE_HOME:-$HOME/.local/state}/planet/aiprovider_runtime.env`. Configuration priority:
|
||||
|
||||
1. `aiprovider/.env`
|
||||
2. simple `export AI_...=...` or `AI_...=...` lines from `~/.zshrc`
|
||||
@@ -117,7 +119,7 @@ When the fingerprint matches, the script skips `docker compose build` and starts
|
||||
docker start planet_aiprovider
|
||||
```
|
||||
|
||||
`docker stop` stops the container without deleting the image. `cleanup_exit_containers` removes exited containers but not images, so the next `docker start` can reuse the existing image.
|
||||
`docker stop` stops the container without deleting the image. `start` and `restart` retain stopped containers instead of scanning and deleting all exited containers on the host. An unchanged AI Provider can be reused, while Compose still reconciles database configuration. Existing recreation paths remain responsible for image or configuration changes.
|
||||
|
||||
## Issue 2: Slow Port Cleanup
|
||||
|
||||
|
||||
@@ -2,6 +2,48 @@
|
||||
|
||||
This runbook is for deployment, on-call, and maintenance engineers. End-user UI flows live in the [Intelligent Planet Manual](/home/ray/dev/linkong/planet/docs/technical/en/manual.md); this document only covers shell, Docker, logs, environment variables, and troubleshooting.
|
||||
|
||||
## Docker Initialization and Access
|
||||
|
||||
Initialize a new machine before starting the application services:
|
||||
|
||||
```bash
|
||||
zsh ./planet.sh init --non-motion-agent && zsh ./planet.sh start --non-motion-agent
|
||||
```
|
||||
|
||||
The entry point still requires `zsh`, `curl`, and reachable package repositories. Before synchronizing Python and frontend dependencies, `init` prepares Docker:
|
||||
|
||||
- Reuse working Docker, Compose v2, and Buildx (at least 0.17.0).
|
||||
- On Ubuntu / Ubuntu WSL, use apt to install the missing parts of `docker.io`, `docker-compose-v2`, and `docker-buildx`. When Docker CE CLI is already installed, use the configured Docker CE repository and corresponding plugin packages to keep the package family consistent.
|
||||
- If the local daemon is unavailable, check that `docker.service` exists, then enable and start it. WSL must have systemd enabled; unavailable service management produces an explicit Docker preparation error.
|
||||
- If the current user cannot read and write the Docker socket, check for `usermod`, install its `passwd` package when needed, and add the user to the `docker` group. This group grants privileged control of the local Docker engine. The script uses `sudo` to refresh group access as the original user and continue the original command with its arguments preserved. It does not depend on `sg` or run application processes as root.
|
||||
|
||||
When elevation is needed, sudo authentication runs in the foreground. Missing sudo for an unprivileged user, authentication failure, repository errors, or insufficient versions after installation stop initialization with a specific error.
|
||||
|
||||
Subsequent `planet.sh start` and other service commands in the same old terminal also refresh Docker group membership when it has been granted but is not yet active. Open a new Ubuntu session to use `docker` directly in the terminal.
|
||||
|
||||
When Docker Desktop is present but its WSL integration is unavailable, the script asks the operator to start Desktop and enable WSL Integration for the distribution. Unreachable remote or rootless endpoints produce a diagnostic for that environment; neither case installs a second local engine automatically. Automatic installation on other operating systems is not currently supported.
|
||||
|
||||
`planet.sh` calls `scripts/lib/docker-bootstrap.zsh` for preparation. Missing CLI, missing service units, socket permissions, and stopped daemons receive separate diagnostics. Advice to start `docker.socket` is shown only after confirming that the unit exists. Verify the result with:
|
||||
|
||||
```bash
|
||||
docker info
|
||||
docker compose version
|
||||
docker buildx version
|
||||
```
|
||||
|
||||
## Database Initialization and Connection Checks
|
||||
|
||||
`init` and `start` reconcile PostgreSQL / Redis containers through Compose, including port configuration on existing containers. A plain `docker start` cannot apply configuration changes. Compose failures retain their specific errors, such as an occupied port, instead of falling back to an old container and reporting success.
|
||||
|
||||
The container's `pg_isready` check only establishes that the server accepts connections; it does not validate the host backend's address and credentials. Once containers are healthy, both initialization and backend startup run `scripts/check_database_connection.py` using the backend's effective `DATABASE_URL`. It checks the local PostgreSQL published port and executes a read-only `SELECT 1`. Startup performs this check before preparing the AI Provider image and stops immediately on failure. Initialization only creates tables and seed data after the check passes.
|
||||
|
||||
- If the actual local port mapping is still missing or mismatched, the script recreates PostgreSQL once from Compose while preserving its data volume, then checks again. A second failure stops initialization.
|
||||
- Authentication, database-name, and network failures stop before schema changes. Diagnostics show the host, port, and database name without passwords, full connection strings, or raw driver exceptions.
|
||||
- A process-level `DATABASE_URL` overrides `backend/.env`. Changing `POSTGRES_PASSWORD` alone updates neither the connection string nor the password stored in an existing data volume. Existing environment files are retained and their effective configuration must be checked.
|
||||
- Explicit external databases do not require a local container mapping. Host networking also does not require published ports. Both still require the real connection check.
|
||||
|
||||
For `port is already allocated` or `address already in use`, inspect `docker ps` port information and `ss -ltnp '( sport = :5432 )'`. With WSL mirrored networking, also inspect Windows listeners. Initialization does not kill other database services to acquire a port, delete data volumes, or reset passwords.
|
||||
|
||||
## First Startup
|
||||
|
||||
```bash
|
||||
@@ -25,6 +67,8 @@ First startup seeds two default accounts (see `DEFAULT_LOGIN_USERS` in `backend/
|
||||
|
||||
Both seed accounts are created with `email_verified = TRUE` and can log into the console immediately. Any other account must either go through the public registration flow described in the Manual, or be created via `./planet.sh createuser`.
|
||||
|
||||
An isolated preview database does not receive default accounts when application lifespan is skipped. For a default-account 401, first verify the actual database and account existence. Run the existing `ensure_default_admin_user` against the confirmed database only when needed; it does not reset existing passwords. Verify both `/api/v1/auth/login` and authenticated `/api/v1/auth/me`: identity loading preserves full fields such as `email_verified` and `created_at`, and responses accept existing bootstrap `.local` addresses while registration and email updates stay strict. The AUTH catalog markers below require manually confirmed evidence; generic 401/500 responses alone do not establish these causes.
|
||||
|
||||
Specify custom ports:
|
||||
|
||||
```bash
|
||||
@@ -232,6 +276,15 @@ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple/"
|
||||
default = true
|
||||
```
|
||||
|
||||
Start and restart use fingerprint-based build detection by default. To explicitly reuse a local image for one command, add --no-build:
|
||||
|
||||
```bash
|
||||
./planet.sh start --no-build
|
||||
./planet.sh restart -a --no-build
|
||||
```
|
||||
|
||||
This option affects only AI Provider and keeps other startup checks enabled. A missing local image is an error, and the old image is never stamped as containing current code. When Compose v2 is available, builds, startup and build-capability checks use v2 only and preserve its failure. Compose v1 is considered only when v2 is unavailable.
|
||||
|
||||
Diagnose slow builds:
|
||||
|
||||
| Symptom | Common cause | Fix |
|
||||
@@ -249,6 +302,102 @@ When SMTP is unset, `POST /api/v1/auth/register` returns `503 EMAIL_PROVIDER_NOT
|
||||
|
||||
One-time codes are stored in Redis under `otp:{purpose}:{email}` with a 600-second TTL. The key is invalidated after 5 invalid attempts. Resend cooldown is 60 seconds, enforced via `otp_rate:{purpose}:{email}`.
|
||||
|
||||
## Error Cause Catalog
|
||||
|
||||
By default, planet.sh retains one specific error beneath the main status. With --verbose, log_error reads its diagnostic code, cause and remedy from the Chinese table. AI Provider build failures also inspect the full build log. Rows are matched in order using case-insensitive literal fragments separated by semicolons; specific errors precede summaries. Codes are diagnostic identifiers, not process exit codes; failure still returns a nonzero status. Matching fragments below intentionally retain the Chinese shell messages.
|
||||
|
||||
For each newly confirmed cause, update both language tables with a stable code, distinguishing evidence and a regression case before integrating it into the script. Unverified failures remain P_UNKNOWN; the script must not append guessed causes to documentation. scripts/lib/error-diagnostics.zsh reads the table. Do not put vertical bars in cells. scripts/harness/test_error_diagnostics.py checks classification, bilingual codes and runtime wording.
|
||||
|
||||
<!-- planet-error-catalog:start -->
|
||||
| Error code | Matching fragments (semicolon-separated) | Cause / established failure | Remedy |
|
||||
| --- | --- | --- | --- |
|
||||
| P_CELESTRAK_NOT_UPDATED | GP data has not updated since your last successful | CelesTrak rejected a repeated download within an update window. HEAD 200 followed by GET 403 can mean preflight consumed the allowance, not an unreachable egress route. | Use the collector without HEAD preflight and preserve validated raw cache. Without valid cache, wait for the next GP update or configure Space-Track instead of retrying repeatedly. |
|
||||
| P_WINDOWS_LAUNCHER_TARGET_MISSING | P_WINDOWS_LAUNCHER_TARGET_MISSING | The bound WSL project is unavailable or lacks launcher scripts, commonly after moving a checkout, losing access to a distribution or copying another machine's desktop launcher. | Run the repository planet.cmd from the current checkout to regenerate this machine's desktop launcher and verify the WSL distribution; do not hardcode another machine's path. |
|
||||
| P_PROXY_EXTERNAL | PLANET_PROXY_EXTERNAL | Docker proxy settings belong to another configuration source or have been edited manually. | Inspect daemon.json, systemd proxy settings and the Planet ownership record before changing ownership; existing settings are preserved. |
|
||||
| P_AUTH_BOOTSTRAP_MISSING | DEFAULT_LOGIN_USER_MISSING | The active backend database is confirmed to lack default accounts, often in an isolated preview that skipped initialization. | Verify database ownership and run the existing account initializer without resetting existing passwords or clearing data. |
|
||||
| P_AUTH_PROFILE_INVALID | AUTH_IDENTITY_RESPONSE_INVALID | Login succeeds but the identity response lacks required fields or incorrectly rejects a stored local email address. | Load the complete persisted identity, separate response serialization from registration validation, and run default-account identity regressions. |
|
||||
| P_EARTH_POLICY_MISSING | EARTH_DISPLAY_POLICY_MISSING | The backend image or deployment directory omitted the shared geography policy required when compute routes import. | Include config/earth-boundary-pov-policy.china-v1.json, the boundary builders and reviewed geometry through both Dockerfile and .dockerignore; rebuild the image without resetting the database. |
|
||||
| P_PROXY_CHANGED | PLANET_PROXY_CONFIG_CHANGED | Docker proxy settings changed after detection. | Wait for concurrent configuration work to finish and retry without overwriting it. |
|
||||
| P_PROXY_ROLLBACK | PLANET_PROXY_ROLLBACK_FAILED | Docker or previously running containers could not be restored after a failed proxy update. | Inspect daemon and container logs; the original configuration was restored. Use daemon.json.planet-proxy.bak for manual recovery if necessary. |
|
||||
| P_PROXY_CONFIG | PLANET_PROXY_CONFIG_FAILED;Docker 构建代理检测失败;Docker 构建代理更新失败 | Automatic proxy detection, validation or service update failed. | Check Python 3, curl, Docker and sudo access; updates attempt rollback and proxy credentials are excluded from error output. |
|
||||
| P_PROXY_NO_ROUTE | PLANET_PROXY_NO_ROUTE | No configured host proxy could reach the required build registries, and direct probes also failed. | Restore a working proxy or repair direct networking, DNS and registry addresses; disabling builds does not repair connectivity. |
|
||||
| P_REGISTRY_RATE_LIMIT | 429 Too Many Requests;toomanyrequests;pull rate limit | The registry responded but imposed a request-rate or image-pull quota limit; consult the response for the specific limit. | Avoid repeated retries and follow upstream retry guidance. For anonymous pull quotas, check docker login identity and allowance; do not mistake throttling for an unreachable proxy. |
|
||||
| P_DNS | no such host;temporary failure in name resolution;could not resolve host | Name resolution failed; the log alone does not identify the DNS configuration or upstream fault. | Check DNS and proxy resolution in the Docker environment; compare shell and daemon resolution. |
|
||||
| P_TLS_CERT | x509:;certificate verify failed;certificate signed by unknown authority | TLS certificate validation failed. | Check time, certificate chain and proxy CA; install trusted certificates without disabling verification. |
|
||||
| P_PROXY_AUTH | proxy authentication required;407 proxy | The proxy requires authentication and rejected the request. | Check daemon proxy credentials and permissions; keep credentials out of the repository and logs. |
|
||||
| P_NETWORK_TIMEOUT | i/o timeout;tls handshake timeout;context deadline exceeded;deadlineexceeded | A connection or TLS handshake timed out; the log alone cannot distinguish proxy, DNS and IPv6 faults. | Compare direct and proxied requests; check daemon proxy settings, DNS and IPv6 routes. Shell proxy settings do not configure the daemon. |
|
||||
| P_CONNECTION_REFUSED | connection refused | The destination refused the connection; its listener, address or port may be wrong. | Identify whether the target is a proxy, database or registry, then check its listener and service state. |
|
||||
| P_REGISTRY_AUTH | pull access denied;unauthorized:;insufficient_scope;denied: requested access | Registry access was denied or the current identity cannot pull the image. | Verify the image name, repository permissions and docker login identity. |
|
||||
| P_IMAGE_TAG | manifest unknown;manifest not found | The registry cannot find the requested manifest or tag. | Check PYTHON_IMAGE, UV_IMAGE and other tags, including architecture support. |
|
||||
| P_DISK_FULL | no space left on device | Disk space or inodes are exhausted. | Check df -h, df -i and docker system df; remove only confirmed disposable data, never database volumes as a troubleshooting shortcut. |
|
||||
| P_DOCKER_SOCKET | permission denied while trying to connect;刷新组权限后仍无法访问 Docker socket | The current user cannot access the Docker socket. | Check socket ownership and docker group membership; run ./planet.sh init for permissions and open a new terminal. |
|
||||
| P_DOCKER_SERVICE | 没有可用的 docker.service;Docker Engine 启动失败;cannot connect to the docker daemon;无法连接 daemon | Docker is not ready or the client cannot reach the current endpoint. | Check systemctl status docker, docker context ls and journalctl -u docker.service; connection failure does not prove Docker is absent. |
|
||||
| P_DOCKER_DESKTOP | 检测到 Docker Desktop,但当前 WSL | Docker Desktop or its WSL integration is unavailable. | Start Docker Desktop and enable WSL Integration for this distribution. |
|
||||
| P_DOCKER_ENDPOINT | 当前 Docker 使用其他 context 或远程/rootless endpoint | The selected remote or rootless Docker endpoint is unavailable. | Check docker context ls, DOCKER_HOST and the target service; do not replace it with a local engine automatically. |
|
||||
| P_BUILDX | 未检测到 docker buildx;buildx 0.17;buildx >=;buildx v;当前 docker compose 不支持 build;安装后 Docker CLI、Compose v2 | Docker build plugins are missing, too old or unable to provide build support. | Check docker buildx version and docker compose version; install or upgrade the plugins required by the project. |
|
||||
| P_COMPOSE_MISSING | 未检测到可用的 Docker Compose | No usable Compose command was found. | Install the Compose v2 plugin and verify docker compose version; a v2 operation failure must not trigger v1 fallback. |
|
||||
| P_LOCAL_IMAGE_MISSING | 已指定 --no-build,但本地没有 AI Provider 镜像 | Builds were disabled but the AI Provider image is not present locally. | Build or import the image before using --no-build; the option never builds automatically. |
|
||||
| P_SUDO | 缺少 sudo | The privilege escalation tool required for dependency setup is unavailable. | Ask an administrator to install and authorize sudo, or provision the dependencies in advance. |
|
||||
| P_UNSUPPORTED_OS | Docker 自动安装目前支持;未识别系统包管理器 | The current system is outside the automatic installation support scope. | Install dependencies through supported system procedures and retry. |
|
||||
| P_LOCKFILE_CHANGED | 修改了 uv.lock | Dependency preparation unexpectedly changed the lockfile. | Check manifest and lockfile consistency; use frozen installation without implicit lockfile updates. |
|
||||
| P_DEPENDENCIES | 安装失败;安装后仍不可用;安装完成后仍未找到;未找到 .venv/bin/python;自动安装后仍无法解析运行时;未找到 Vite Bun 入口;缺少 mediapipe/opencv-python;仍无法导入 mediapipe/opencv-python;需要 openssl | A required dependency failed to install, is missing or is unavailable in the active environment. | Inspect the installer log, network, package sources and PATH; use Bun for frontend and the project uv environment for Python. |
|
||||
| P_ARGUMENT | 未知参数;非法端口;需要端口号;需要逗号分隔;--motion-agent-mode 需要;--motion-agent-wsl-usbipd-busid 需要;用法: ./planet.sh | The command or an argument does not match the supported format. | Check ./planet.sh usage and correct the arguments before retrying. |
|
||||
| P_DB_PORT_OCCUPIED | PLANET_DB_PORT_OCCUPIED | Another container or host service occupies the Planet database or Redis port. | Identify the owner with docker ps; update the DATABASE_URL port or REDIS_PORT in backend/.env (process REDIS_URL takes precedence). planet.sh synchronizes Compose mappings without stopping other projects or deleting volumes. |
|
||||
| P_PORT | 地址已被占用;端口仍不可用;清理失败,请检查占用进程;port is already allocated;address already in use | The requested port is occupied or cannot be bound in the host environment. | Check ss and Windows Get-NetTCPConnection; identify the owner before changing ports or stopping the service. |
|
||||
| P_CAMERA | live 模式缺少可用摄像头;未找到可打开并能读帧的摄像头 | Motion Agent cannot capture frames from a usable camera. | Check hardware, permissions and WSL USB forwarding; use --non-motion-agent or explicit dry-run when live capture is not needed. |
|
||||
| P_DB_CONNECTION | 后端数据库连接检查失败 | The backend database connection or published-port check failed. | Inspect the probe output and verify DATABASE_URL, credentials, database name and port; container health alone is insufficient. |
|
||||
| P_DB_START | 数据库启动失败;数据库重启失败;PostgreSQL 启动失败;数据库初始化失败 | Database startup, health checks or initialization did not complete. | Inspect PostgreSQL and Redis logs and the specific database error; do not troubleshoot by deleting volumes. |
|
||||
| P_BACKEND_START | 后端进程已退出;后端启动失败 | The backend exited or did not pass its health check. | Use ./planet.sh log -b and resolve import, configuration, database or application initialization errors first. |
|
||||
| P_AI_START | AI Provider 启动失败 | AI Provider did not start or pass its health check. | Use ./planet.sh log -a and check runtime configuration, ports and container exit details. |
|
||||
| P_FRONTEND_START | 前端启动失败 | The frontend did not pass its startup health check. | Use ./planet.sh log -f and check Bun, dependencies, the Vite entry point and ports. |
|
||||
| P_MOTION_START | Motion Agent 启动失败 | Motion Agent failed to start. | Use ./planet.sh log -m and check dependencies, cameras and input mode. |
|
||||
| P_ACCOUNT_INPUT | 用户名不能为空;密码不能为空;密码长度不能少于;两次输入的密码不一致 | User creation input failed validation. | Supply a username and matching passwords that satisfy the minimum length. |
|
||||
| P_HTTP_HEALTH | 不可访问: | The specified HTTP endpoint failed its access check. | Check the URL, listener, firewall and local or LAN routing; check certificate trust for HTTPS. |
|
||||
| P_DOCKER_MIRROR_FAILED | PLANET_DOCKER_MIRROR_FAILED | Mirror manifest validation or image pulling failed, so fallback could not complete. | Inspect the specific registry error, mirror availability, allowlist, architecture and digest. A reachable homepage is insufficient. Configure PLANET_DOCKER_MIRROR_PREFIX or disable fallback while repairing the origin. |
|
||||
| P_DOCKER_COMMAND_TIMEOUT | PLANET_DOCKER_COMMAND_TIMEOUT | The current Docker command exceeded the script deadline; timeout alone does not establish a proxy, network or local engine fault. | Locate the stalled stage using its label, latest output and log. For silent commands, inspect that Docker command and daemon logs; increase the deadline only after confirming progress. |
|
||||
| P_COMPOSE_FAILED | Docker Compose 执行失败;docker-compose v1 执行失败 | The selected Compose command failed without a more specific classified cause. | Inspect the preceding original error; repair an installed Compose v2 instead of installing v1 as a fallback. |
|
||||
| P_BUILD_FAILED | AI Provider 镜像构建失败;failed to solve | Image build failed without evidence matching a known specific cause. | Inspect the earliest specific error in aiprovider_build.log and add the confirmed cause and a regression case to this catalog. |
|
||||
| P_UNKNOWN | — | Unclassified; the available evidence does not establish the cause. | Retain the error and command; after verifying the root cause, update both language catalogs and add a regression case. |
|
||||
<!-- planet-error-catalog:end -->
|
||||
|
||||
### Docker Progress and Deadlines
|
||||
|
||||
Docker probes, Compose startup, database connection checks and AI Provider builds update the secondary line beneath the existing main status. There are no extra start/completion records, version numbers or log paths. Success clears the secondary line. Silent waits retain the current step and update elapsed time. Failure leaves one specific error and exits; later summaries and cleanup notices do not overwrite it. Failed Compose operations do not retry; health checks after a successful container launch retain their existing budgets.
|
||||
|
||||
`PLANET_DOCKER_PROBE_TIMEOUT`, `PLANET_COMPOSE_TIMEOUT` and `PLANET_DOCKER_BUILD_TIMEOUT` default to 15, 180 and 900 seconds respectively. Each accepts positive seconds and bounds one command. Timeout terminates that CLI process group; the daemon may already have accepted a request, so inspect actual container state. A Compose probe timeout does not fall back to v1.
|
||||
|
||||
Detailed logs remain under `${PLANET_STATE_DIR:-${XDG_STATE_HOME:-$HOME/.local/state}/planet}`: `compose.*.log`, `aiprovider_build.log` and `database-check.*.log`. Logs use mode 600 and mask proxy usernames/passwords in URLs. `scripts/docker_command.py` owns recording, timing and process cleanup; `planet.sh` owns terminal rendering. Invoking system `docker compose` directly does not load this wrapper.
|
||||
|
||||
### Database Troubleshooting Before Demos
|
||||
|
||||
A confirmed failure occurs when another project's PostgreSQL and Redis occupy `5432` and `6379`. Planet containers may appear healthy without published host ports, and the backend can accidentally reach the other database and fail authentication. `scripts/check_database_connection.py --ports` identifies conflicts before startup and stops instead of repeatedly recreating containers.
|
||||
|
||||
For coexistence, change the port in `backend/.env`'s `DATABASE_URL` to an available port such as `15432`, and `REDIS_PORT` to an available port such as `16379`. Preserve credentials and the database name. If the process exports `REDIS_URL`, update it too. `planet.sh` derives `PLANET_POSTGRES_PORT` and `PLANET_REDIS_PORT` from effective backend settings and preserves named volumes when recreating port mappings. Direct Compose invocations must supply these variables explicitly. Database ports bind only to `127.0.0.1`.
|
||||
|
||||
Handle other errors according to evidence: authentication failures require checking the target and existing volume credentials; changing `POSTGRES_PASSWORD` does not reset an existing password. Missing published ports without a conflict allow one recreation preserving volumes. DNS, TLS, proxy authentication and registry rate limits have separate catalog remedies. Slow downloads or dependency installation are not failures; unchanged code and dependencies should reuse the image on subsequent starts.
|
||||
|
||||
### Temporary Docker Image Mirrors
|
||||
|
||||
The Tsinghua [Docker CE repository](https://mirrors.tuna.tsinghua.edu.cn/help/docker-ce/) distributes Docker installation packages, not Docker Hub images. Image fallback defaults to the [DaoCloud public image mirror](https://github.com/DaoCloud/public-image-mirror). Tsinghua PyPI fallback remains specific to local Python dependencies.
|
||||
|
||||
Build preparation tries the origin through proxy and direct routes, then the mirror route when neither works. A network failure during an actual registry request can also trigger one fallback after a successful origin probe. Authentication, certificate, rate-limit, port and Dockerfile package-installation failures do not switch image sources. Exhausted routes leave the specific error in the secondary line and exit. Original failures remain in adjacent `*.primary.log` files.
|
||||
|
||||
The specific mirror image must pass manifest inspection; execution pins its returned sha256 digest. An explicitly requested digest must match. Base images are overridden only through this build's arguments; original configuration, fingerprint and output image name remain unchanged. Database fallback pulls only missing images for requested services, tags them with the original local names, then retries the original Compose operation. Global registry-mirrors, source configuration and named volumes remain unchanged.
|
||||
|
||||
`PLANET_DOCKER_MIRROR_FALLBACK=0` disables fallback. `PLANET_DOCKER_MIRROR_PREFIX` defaults to `m.daocloud.io` and accepts a trusted registry/path prefix. `PLANET_DOCKER_MIRROR_TIMEOUT` defaults to 120 seconds for base-image verification. Only Planet's public Python, uv, PostgreSQL and Redis repositories are mapped; private/custom images are not sent to public mirrors. Public services have allowlists, quotas and cache delays, so mutable tags can lag. Use digests when strict version identity matters and build/cache images before demos.
|
||||
|
||||
`scripts/docker_mirror.py` owns mapping, manifest checks and database pulls; `scripts/lib/docker-mirror.zsh` owns network-error selection and temporary arguments. Both reuse the existing progress/deadline runner. Regressions cover private repositories, digest mismatches, non-network errors, a single fallback and preservation of original configuration.
|
||||
|
||||
### Docker Daemon Proxy and Build Networking
|
||||
|
||||
Shell HTTP_PROXY / HTTPS_PROXY settings do not automatically configure a running Docker daemon. When the shell can reach a registry through its proxy but Docker pulls time out, compare direct requests, proxied requests and daemon settings. HTTP 401, 403 and 429 establish a registry response, not pull authorization or remaining quota. Verify authentication and rate limits with the actual build; these responses must not cause a reachable proxy to be disabled.
|
||||
|
||||
Before an actual AI Provider build, the script reads HTTPS_PROXY, HTTP_PROXY and ALL_PROXY, including lowercase forms. It validates HTTP/HTTPS candidates against the registries selected by PYTHON_IMAGE and UV_IMAGE, respecting NO_PROXY. A working proxy is configured for the local Linux Docker Engine. Missing or unusable proxies cause a direct probe and removal of stale Planet-managed proxy settings. If neither origin route works, the mirror route is checked; P_PROXY_NO_ROUTE stops the build only when that also fails. A host without a proxy and a daemon already using direct access receives no proxy configuration. Fingerprint cache hits and --no-build skip probing. This is not a background monitor: proxy availability is checked on the next actual build.
|
||||
|
||||
scripts/docker_proxy.py manages the proxies section of /etc/docker/daemon.json and a root-only ownership record at /etc/docker/planet-proxy-state.json. Administrator settings, systemd proxy settings and manually edited proxies are not overwritten. Unchanged settings need neither elevation nor a restart. Updates use the existing sudo flow, preserve unrelated Docker settings, save daemon.json.planet-proxy.bak, validate configuration, restart Docker and start previously running containers. Failures trigger rollback; check container health after recovery. Credentials travel through environment variables or restricted files and are excluded from logs. Docker Desktop, remote and rootless daemons retain their own settings without local daemon.json changes. Proxy addresses come from the environment, never a machine-specific port in the repository.
|
||||
|
||||
The AI Provider Dockerfile uses the bundled BuildKit frontend to avoid a separate docker/dockerfile image fetch. Python and uv base images and package downloads still require network access. --no-build explicitly reuses a local image; it does not repair build networking. Build errors are recorded in ${XDG_STATE_HOME:-$HOME/.local/state}/planet/aiprovider_build.log. Verify restored services with ./planet.sh health.
|
||||
|
||||
## Troubleshooting Order
|
||||
|
||||
```bash
|
||||
@@ -330,3 +479,19 @@ uv run pytest backend/tests/test_otp_service.py
|
||||
- [System Service Control](/home/ray/dev/linkong/planet/docs/technical/en/backend-system-service-control.md)
|
||||
- [Docker + Compose + Buildx Upgrade](/home/ray/dev/linkong/planet/docs/technical/en/ops-docker-compose-buildx-upgrade.md)
|
||||
- [Data Collectors](/home/ray/dev/linkong/planet/docs/technical/en/backend-collectors.md)
|
||||
|
||||
## Shipping the selectable Earth basemaps
|
||||
|
||||
These changes from 0.74.7 add no database migration or dependency-lock update. Restart the existing Bun development services; static/container deployments must rebuild the frontend and relevant images rather than keep an old dist after pulling. New modules, four runtime maps, flags, policy fallback geometry and the shared policy must be delivered together. Committing only already-tracked files omits required new assets.
|
||||
|
||||
The old `frontend/public/earth/assets/8k_earth_daymap.jpg` path remains but now contains Natural Earth I. The exact 0.74.7 image is retained at `frontend/public/earth/assets/maps/archived-original-20260921-8k.jpg`, exposed as Original basemap. Pull/start does not delete the archive. Settings without a map-source field use the Original basemap default. `resources/earth/candidates` contains provenance/reference originals and is not a frontend runtime dependency.
|
||||
|
||||
High-precision boundary outputs live in a Git-ignored generated directory. Pulling does not deliver them automatically. Outputs lacking the required new policy metadata are rejected in favor of the shipped policy fallback; regenerate and publish the static artifacts through the existing boundary workflow to retain high precision. This fallback does not reduce basemap resolution.
|
||||
|
||||
Run `PLANET_HARNESS_DOCKER_SMOKE=1 .venv/bin/python scripts/harness/test_backend_runtime_assets.py` to exercise the actual Docker context and production COPY rules, verify the shared policy/geometry and builder imports, and exclude environment files. This probe does not start the full application stack.
|
||||
|
||||
## Detail-card image storage
|
||||
|
||||
Bundled originals and their manifest live under `frontend/public/earth/assets/cards/`, with card-sized derivatives in `previews/`. After changing source assets, run `uv run --frozen --project . python scripts/prepare_card_images.py`, verify source hashes and permissions, then commit originals, manifest and generated WebP files. Preview generation follows the manifest enabled flag; permission records are retained unchanged. Backend images copy the same manifest while frontend builds distribute the files; deploy both images from the same revision.
|
||||
|
||||
Uploads default to the repository-root `data/earth-card-images/`; `PLANET_CARD_IMAGE_DIR` can select an absolute directory. Production containers need a persistent mount, shared across backend replicas. Back up that directory together with the PostgreSQL `earth_card_images` system setting. Database backups alone cannot restore uploaded bytes. Original and 640/1280 WebP files use content-addressed URLs under `/api/v1/earth/card-images/files/`, already covered by the API proxy. Removing configuration does not immediately delete physical files while older clients may still use them.
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user