Compare commits

...

7 Commits

Author SHA1 Message Date
d15a9d488a release: bump version to 0.66.3
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-05-26 17:26:19 +08:00
rayd1o
eb4c4b7904 release: bump version to 0.66.2
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-05-26 08:45:33 +08:00
rayd1o
887fec972e release: bump version to 0.66.1
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-05-26 04:38:18 +08:00
rayd1o
5bf5c73ca0 release: bump version to 0.66.0
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-05-26 03:41:47 +08:00
e65267fe21 release: bump version to 0.65.2
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
release / images (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-05-22 21:37:38 +08:00
ae982e51cd release: bump version to 0.65.1
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-05-22 20:42:03 +08:00
rayd1o
65e6a96c0d release: bump version to 0.65.0
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
release / images (push) Has been cancelled
ci / delivery (push) Has been cancelled
ci / backend (pull_request) Has been cancelled
ci / frontend (pull_request) Has been cancelled
ci / delivery (pull_request) Has been cancelled
2026-05-21 05:41:49 +08:00
190 changed files with 10082 additions and 13444 deletions

2
.gitignore vendored
View File

@@ -29,6 +29,8 @@ downloads/
eggs/
.eggs/
lib/
!frontend/src/admin/lib/
!frontend/src/admin/lib/**
lib64/
parts/
sdist/

View File

@@ -7,9 +7,6 @@ This file is the active backlog only. Completed history belongs in `docs/CHANGEL
- [ ] Earth AI command entry: merge natural-language and speech-triggered LLM commands into the existing Earth search panel as described in [Agent Runtime, Earth LLM Command, And Speech Entry Plan](/home/ray/dev/linkong/planet/docs/plans/agents-earth-command-runtime-plan.md).
- [ ] Earth action executor: implement safe visualization actions for layer toggles, batch highlights, filters, focus, result panels, and clear-highlight behavior.
- [ ] Earth entity matching: support stable entity ids and batch matching for Beidou satellites, mainland China compute centers, BGP, news, vessels, and cables.
- [x] High-precision country boundary tile framework: implement the static vector tile builder, versioned seed output, frontend bbox tile loader, debounce, in-flight dedupe, and LRU cache described in [Earth High Precision Boundary Tiles Plan](/home/ray/dev/linkong/planet/docs/plans/earth-high-precision-boundary-tiles-plan.md).
- [x] Add the `pmtiles-mvt` frontend tile provider contract, MVT decoder dependencies, static PMTiles Nginx handling, collector artifact registration, production readiness check, and user operation docs for Earth boundaries.
- [x] Split Earth boundary ingestion into standard source collectors (`earth_admin0_boundaries`, `earth_coastline`, `earth_claim_lines`) plus the downstream `earth_boundary_tiles` PMTiles builder.
- [ ] Replace debug GeoJSON boundary tiles with the real `earth-boundaries-china-pov-v1.pmtiles` production artifact after audited admin-0 / coastline / claim-line sources and the PMTiles toolchain are available.
- [ ] Import authoritative China POV / coastline / claim-line source packages through the three standard Earth boundary source collectors, then rebuild a versioned PMTiles artifact so highest zoom `8-10` preserves trusted source geometry instead of seed data.
- [ ] Earth boundary data: acquire or generate auditable China POV geometry for Zangnan, Aksai Chin, Taiwan/Penghu, Diaoyu Dao and affiliated islands, Chiwei Yu, South China Sea islands, Kosovo, Gaza, and the official dashed maritime claim line before implementing final visual changes.
@@ -61,6 +58,9 @@ Archived items stay here so old context is not lost. Completed items remain chec
### Completed
- [x] Implemented the high-precision country boundary tile framework from [Earth High Precision Boundary Tiles Plan](/home/ray/dev/linkong/planet/docs/plans/earth-high-precision-boundary-tiles-plan.md): static vector tile builder, versioned seed output, frontend bbox tile loader, debounce, in-flight dedupe, and LRU cache.
- [x] Added the `pmtiles-mvt` frontend tile provider contract, MVT decoder dependencies, static PMTiles Nginx handling, collector artifact registration, production readiness check, and user operation docs for Earth boundaries.
- [x] Split Earth boundary ingestion into standard source collectors (`earth_admin0_boundaries`, `earth_coastline`, `earth_claim_lines`) plus the downstream `earth_boundary_tiles` PMTiles builder.
- [x] Refined BGP observer and anomaly `hover/click` feel.
- [x] Added BGP anomaly relationship display with cables / regions.
- [x] Added the Earth BGP activity layer so the map still feels alive when incident density is low.

View File

@@ -1 +1 @@
0.64.0
0.66.3

View File

@@ -2,10 +2,14 @@
ARG PYTHON_IMAGE=python:3.14-slim
ARG UV_IMAGE=ghcr.io/astral-sh/uv:latest
ARG AI_PROVIDER_BUILD_FINGERPRINT=unknown
FROM ${UV_IMAGE} AS uv
FROM ${PYTHON_IMAGE}
ARG AI_PROVIDER_BUILD_FINGERPRINT
LABEL planet.aiprovider.build-fingerprint="${AI_PROVIDER_BUILD_FINGERPRINT}"
COPY --from=uv /uv /uvx /bin/
WORKDIR /app
@@ -15,12 +19,15 @@ ENV PYTHONUNBUFFERED=1
ENV UV_COMPILE_BYTECODE=1
ENV UV_LINK_MODE=copy
RUN mkdir -p /root/.config/uv
RUN apt-get update && apt-get install -y --no-install-recommends \
curl \
&& rm -rf /var/lib/apt/lists/*
COPY pyproject.toml uv.lock /app/
RUN --mount=type=cache,target=/root/.cache/uv \
--mount=type=secret,id=planet_uv_config,target=/root/.config/uv/uv.toml,required=false \
uv sync --frozen --no-dev
COPY aiprovider /app/aiprovider

View File

@@ -1,3 +1,5 @@
# syntax=docker/dockerfile:1.7
ARG PYTHON_IMAGE=python:3.14-slim
ARG UV_IMAGE=ghcr.io/astral-sh/uv:latest
@@ -14,12 +16,16 @@ ENV UV_COMPILE_BYTECODE=1
ENV UV_LINK_MODE=copy
ENV PYTHONPATH=/app/backend
RUN mkdir -p /root/.config/uv
RUN apt-get update && apt-get install -y --no-install-recommends \
curl \
&& rm -rf /var/lib/apt/lists/*
COPY pyproject.toml uv.lock /app/
RUN uv sync --frozen --no-dev
RUN --mount=type=cache,target=/root/.cache/uv \
--mount=type=secret,id=planet_uv_config,target=/root/.config/uv/uv.toml,required=false \
uv sync --frozen --no-dev
COPY backend /app/backend
COPY VERSION /app/VERSION

View File

@@ -19,6 +19,7 @@ from app.api.v1 import (
vessels,
bgp,
news,
interactables,
realtime_sources,
system_control,
tv,
@@ -53,4 +54,5 @@ api_router.include_router(vessels.router, prefix="/vessels", tags=["vessels"])
api_router.include_router(bgp.router, prefix="/bgp", tags=["bgp"])
api_router.include_router(tv.router, prefix="/tv", tags=["tv"])
api_router.include_router(news.router, prefix="/news", tags=["news"])
api_router.include_router(interactables.router, prefix="/interactables", tags=["interactables"])
api_router.include_router(realtime_sources.router, prefix="/realtime-sources", tags=["realtime-sources"])

View File

@@ -3,6 +3,7 @@ from uuid import uuid4
from fastapi import APIRouter, Depends, HTTPException, Request, Response
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.logging import get_logger
from app.core.security import get_current_user
from app.db.session import get_db
from app.models.user import User
@@ -47,8 +48,10 @@ from app.services.playground_chat_service import (
stop_message,
)
from app.services.situational_alert_ai_brief import build_situational_alert_brief_request
from app.services.business_logs import emit_business_log, exception_context
router = APIRouter()
logger = get_logger(__name__, service="api")
@router.get("/provider/status", response_model=AIProviderStatusResponse)
@@ -122,6 +125,16 @@ async def create_playground_message(
provider_client: AIProviderClient = Depends(get_ai_provider_client),
db: AsyncSession = Depends(get_db),
):
await emit_business_log(
logger,
event="ai.playground.message.create",
message="Playground message creation requested",
category="ai",
service="api",
module=__name__,
user_id=current_user.id,
context={"session_key": payload.session_key, "preset": payload.selected_preset_key},
)
return await create_turn(
db,
user_id=current_user.id,
@@ -136,6 +149,16 @@ async def stop_playground_message(
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
await emit_business_log(
logger,
event="ai.playground.message.stop",
message="Playground message stop requested",
category="ai",
service="api",
module=__name__,
user_id=current_user.id,
context={"session_key": payload.session_key, "message_id": payload.message_id},
)
return await stop_message(
db,
user_id=current_user.id,
@@ -150,6 +173,16 @@ async def resend_playground_message(
provider_client: AIProviderClient = Depends(get_ai_provider_client),
db: AsyncSession = Depends(get_db),
):
await emit_business_log(
logger,
event="ai.playground.message.resend",
message="Playground message resend requested",
category="ai",
service="api",
module=__name__,
user_id=current_user.id,
context={"session_key": payload.session_key, "user_message_id": payload.user_message_id},
)
return await resend_turn(
db,
user_id=current_user.id,
@@ -214,16 +247,70 @@ async def analyze_bgp_brief(
anomaly_limit=payload.anomaly_limit,
collector_limit=payload.collector_limit,
)
await emit_business_log(
logger,
event="ai.brief.bgp.facts_collected",
message="BGP brief facts collected",
category="ai",
service="api",
module=__name__,
request_id=request_id,
user_id=current_user.id,
context={
"incident_limit": payload.incident_limit,
"anomaly_limit": payload.anomaly_limit,
"collector_limit": payload.collector_limit,
"fact_count": len(facts or []),
},
)
brief_request.preferred_model = payload.preferred_model
brief_request.thinking = payload.thinking
analysis = await provider_client.analyze(brief_request, request_id=request_id)
return save_bgp_brief_record(
analysis,
await emit_business_log(
logger,
event="ai.brief.bgp.start",
message="BGP brief AI analysis started",
category="ai",
service="api",
module=__name__,
request_id=request_id,
facts=facts,
context=context,
user_id=current_user.id,
context={"preferred_model": payload.preferred_model},
)
try:
analysis = await provider_client.analyze(brief_request, request_id=request_id)
record = save_bgp_brief_record(
analysis,
request_id=request_id,
facts=facts,
context=context,
)
await emit_business_log(
logger,
event="ai.brief.bgp.completed",
message="BGP brief AI analysis saved",
category="ai",
service="api",
module=__name__,
request_id=request_id,
user_id=current_user.id,
context={"provider": analysis.provider, "model": analysis.model, "brief_id": record.id},
)
return record
except Exception as exc:
await emit_business_log(
logger,
event="ai.brief.bgp.failed",
message="BGP brief AI analysis failed",
category="ai",
level="error",
service="api",
module=__name__,
request_id=request_id,
user_id=current_user.id,
context=exception_context(exc, {"preferred_model": payload.preferred_model}),
)
raise
@router.post("/alerts/brief", response_model=AlertBriefResponse)
@@ -242,17 +329,65 @@ async def analyze_alert_brief(
db,
alert_limit=payload.alert_limit,
)
await emit_business_log(
logger,
event="ai.brief.alerts.facts_collected",
message="Alert brief facts collected",
category="ai",
service="api",
module=__name__,
request_id=request_id,
user_id=current_user.id,
context={"alert_limit": payload.alert_limit, "fact_count": len(facts or [])},
)
brief_request.preferred_model = payload.preferred_model
brief_request.thinking = payload.thinking
analysis = await provider_client.analyze(brief_request, request_id=request_id)
return AlertBriefResponse(
**analysis.model_dump(),
title=brief_request.title,
objective=brief_request.objective,
facts=facts,
context=context,
await emit_business_log(
logger,
event="ai.brief.alerts.start",
message="Alert brief AI analysis started",
category="ai",
service="api",
module=__name__,
request_id=request_id,
user_id=current_user.id,
context={"preferred_model": payload.preferred_model},
)
try:
analysis = await provider_client.analyze(brief_request, request_id=request_id)
await emit_business_log(
logger,
event="ai.brief.alerts.completed",
message="Alert brief AI analysis completed",
category="ai",
service="api",
module=__name__,
request_id=request_id,
user_id=current_user.id,
context={"provider": analysis.provider, "model": analysis.model},
)
return AlertBriefResponse(
**analysis.model_dump(),
title=brief_request.title,
objective=brief_request.objective,
facts=facts,
context=context,
)
except Exception as exc:
await emit_business_log(
logger,
event="ai.brief.alerts.failed",
message="Alert brief AI analysis failed",
category="ai",
level="error",
service="api",
module=__name__,
request_id=request_id,
user_id=current_user.id,
context=exception_context(exc, {"preferred_model": payload.preferred_model}),
)
raise
@router.post("/situational-alerts/brief", response_model=SituationalAlertBriefResponse)
@@ -268,14 +403,62 @@ async def analyze_situational_alert_brief(
response.headers["X-Request-ID"] = request_id
brief_request, facts, context = await build_situational_alert_brief_request(db)
await emit_business_log(
logger,
event="ai.brief.situational_alerts.facts_collected",
message="Situational alert brief facts collected",
category="ai",
service="api",
module=__name__,
request_id=request_id,
user_id=current_user.id,
context={"fact_count": len(facts or [])},
)
brief_request.preferred_model = payload.preferred_model
brief_request.thinking = payload.thinking
analysis = await provider_client.analyze(brief_request, request_id=request_id)
return SituationalAlertBriefResponse(
**analysis.model_dump(),
title=brief_request.title,
objective=brief_request.objective,
facts=facts,
context=context,
await emit_business_log(
logger,
event="ai.brief.situational_alerts.start",
message="Situational alert brief AI analysis started",
category="ai",
service="api",
module=__name__,
request_id=request_id,
user_id=current_user.id,
context={"preferred_model": payload.preferred_model},
)
try:
analysis = await provider_client.analyze(brief_request, request_id=request_id)
await emit_business_log(
logger,
event="ai.brief.situational_alerts.completed",
message="Situational alert brief AI analysis completed",
category="ai",
service="api",
module=__name__,
request_id=request_id,
user_id=current_user.id,
context={"provider": analysis.provider, "model": analysis.model},
)
return SituationalAlertBriefResponse(
**analysis.model_dump(),
title=brief_request.title,
objective=brief_request.objective,
facts=facts,
context=context,
)
except Exception as exc:
await emit_business_log(
logger,
event="ai.brief.situational_alerts.failed",
message="Situational alert brief AI analysis failed",
category="ai",
level="error",
service="api",
module=__name__,
request_id=request_id,
user_id=current_user.id,
context=exception_context(exc, {"preferred_model": payload.preferred_model}),
)
raise

View File

@@ -1,4 +1,3 @@
import asyncio
from datetime import datetime, timedelta, timezone
from typing import Optional
@@ -7,7 +6,7 @@ from pydantic import BaseModel, Field
from sqlalchemy import func, or_, select, text
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.cache import cache
from app.core.logging import get_logger
from app.core.time import to_iso8601_utc
from app.core.security import get_current_user
from app.core.data_sources import get_data_sources_config
@@ -22,16 +21,26 @@ from app.models.user import User
from app.models.vessel import AISRawObservation
from app.services.vessel_ais_aggregation import VESSEL_AIS_SCHEMA
from app.services.scheduler import (
cancel_running_collector_now,
get_latest_task_id_for_datasource,
run_collector_now,
sync_datasource_job,
)
from app.services.earth_layer_cache import invalidate_earth_layer_cache_for_source
from app.services.data_jobs import (
JOB_STATUS_CANCELLING,
JOB_STATUS_QUEUED,
JOB_STATUS_RUNNING,
JOB_TYPE_CLEAR_CACHE,
JOB_TYPE_CLEAR_DATA,
JOB_TYPE_COLLECT,
enqueue_datasource_job,
get_active_datasource_job,
request_cancel_datasource_task,
)
from app.services.business_logs import emit_business_log
router = APIRouter()
logger = get_logger(__name__, service="api")
STALE_RUNNING_TASK_TIMEOUT_MINUTES = 90
PRODUCT_SOURCE_KEYWORDS: tuple[tuple[str, tuple[str, ...]], ...] = (
("vessels", ("vessel", "ais")),
("cables", ("cable", "landing", "telegeography", "arcgis", "fao")),
@@ -115,7 +124,7 @@ async def _load_latest_running_tasks(
_task_rank_column(CollectionTask.started_at),
)
.where(CollectionTask.datasource_id.in_(datasource_ids))
.where(CollectionTask.status == "running")
.where(CollectionTask.status.in_((JOB_STATUS_QUEUED, JOB_STATUS_RUNNING, JOB_STATUS_CANCELLING)))
.subquery()
)
result = await db.execute(
@@ -126,32 +135,6 @@ async def _load_latest_running_tasks(
return {task.datasource_id: task for task in result.scalars().all()}
async def _load_latest_task_ids(
db: AsyncSession,
datasource_ids: list[int],
) -> dict[int, int]:
if not datasource_ids:
return {}
ranked_tasks = (
select(
CollectionTask.id.label("task_id"),
CollectionTask.datasource_id.label("datasource_id"),
func.row_number().over(
partition_by=CollectionTask.datasource_id,
order_by=CollectionTask.id.desc(),
).label("row_num"),
)
.where(CollectionTask.datasource_id.in_(datasource_ids))
.subquery()
)
result = await db.execute(
select(ranked_tasks.c.datasource_id, ranked_tasks.c.task_id)
.where(ranked_tasks.c.row_num == 1)
)
return {datasource_id: task_id for datasource_id, task_id in result.all()}
async def _load_latest_tasks(
db: AsyncSession,
datasource_ids: list[int],
@@ -303,8 +286,11 @@ def serialize_datasource_row(
"last_run": to_iso8601_utc(last_run_at),
"last_run_at": to_iso8601_utc(last_run_at),
"last_status": last_status,
"is_running": running_task is not None,
"is_running": running_task is not None and running_task.task_type == JOB_TYPE_COLLECT,
"is_task_active": running_task is not None,
"task_status": running_task.status if running_task else None,
"task_id": display_task.id if display_task else None,
"task_type": display_task.task_type if display_task else None,
"progress": display_task.progress if display_task else None,
"phase": display_task.phase if display_task else None,
"phase_progress": display_task.phase_progress if display_task else None,
@@ -400,9 +386,25 @@ async def _trigger_datasource_batch(
datasources: list[DataSource],
*,
force: bool,
actor_id: int | None = None,
trigger_kind: str = "batch",
) -> dict:
await emit_business_log(
logger,
event=f"collector.trigger.{trigger_kind}.start",
message="Datasource batch trigger started",
category="collector",
service="api",
module=__name__,
user_id=actor_id,
context={
"trigger_kind": trigger_kind,
"force": force,
"requested_count": len(datasources),
},
)
if not datasources:
return {
result = {
"status": "noop",
"message": "No matching data sources to trigger",
"force": force,
@@ -410,8 +412,18 @@ async def _trigger_datasource_batch(
"skipped": [],
"failed": [],
}
await emit_business_log(
logger,
event=f"collector.trigger.{trigger_kind}.completed",
message="Datasource batch trigger completed with no matching sources",
category="collector",
service="api",
module=__name__,
user_id=actor_id,
context={"trigger_kind": trigger_kind, "force": force, "status": "noop", "triggered_count": 0},
)
return result
previous_task_ids: dict[int, Optional[int]] = {}
triggered_sources: list[dict] = []
skipped_sources: list[dict] = []
failed_sources: list[dict] = []
@@ -446,9 +458,11 @@ async def _trigger_datasource_batch(
}
)
continue
cancelled = await cancel_running_collector_now(datasource.source)
if not cancelled:
await rollback_orphaned_running_task(db, datasource, running_task)
await request_cancel_datasource_task(
db,
running_task,
reason="superseded_by_forced_collection",
)
if not force and not is_due_for_collection(datasource, now):
skipped_sources.append(
@@ -465,57 +479,51 @@ async def _trigger_datasource_batch(
)
continue
previous_task_ids[datasource.id] = None
success = run_collector_now(datasource.source)
if not success:
failed_sources.append(
{
"id": datasource.id,
"source": datasource.source,
"name": datasource.name,
"reason": "trigger_failed",
}
)
continue
task = await enqueue_datasource_job(
db,
datasource,
JOB_TYPE_COLLECT,
payload={"force": force, "trigger": "batch"},
)
triggered_sources.append(
{
"id": datasource.id,
"source": datasource.source,
"name": datasource.name,
"task_id": None,
"task_id": task.id,
}
)
latest_task_ids = await _load_latest_task_ids(
db,
[datasource.id for datasource in datasources],
)
for datasource_id in previous_task_ids:
previous_task_ids[datasource_id] = latest_task_ids.get(datasource_id)
for _ in range(20):
await asyncio.sleep(0.1)
pending = [item for item in triggered_sources if item["task_id"] is None]
if not pending:
break
latest_task_ids = await _load_latest_task_ids(
db,
[item["id"] for item in pending],
)
for item in pending:
task_id = latest_task_ids.get(item["id"])
if task_id is not None and task_id != previous_task_ids.get(item["id"]):
item["task_id"] = task_id
return {
"status": "triggered" if triggered_sources else "partial",
"message": f"Triggered {len(triggered_sources)} data sources",
result = {
"status": "queued" if triggered_sources else "partial",
"message": f"Queued {len(triggered_sources)} data source jobs",
"force": force,
"triggered": triggered_sources,
"skipped": skipped_sources,
"failed": failed_sources,
}
await emit_business_log(
logger,
event=f"collector.trigger.{trigger_kind}.completed",
message="Datasource batch trigger completed",
category="collector",
service="api",
module=__name__,
user_id=actor_id,
context={
"trigger_kind": trigger_kind,
"force": force,
"status": result["status"],
"requested_count": len(datasources),
"triggered_count": len(triggered_sources),
"skipped_count": len(skipped_sources),
"failed_count": len(failed_sources),
"triggered_sources": [item["source"] for item in triggered_sources],
"skipped_reasons": [item["reason"] for item in skipped_sources],
},
)
return result
async def get_datasource_record(db: AsyncSession, source_id: str) -> Optional[DataSource]:
@@ -780,7 +788,13 @@ async def trigger_all_datasources(
.order_by(DataSource.module, DataSource.id)
)
datasources = result.scalars().all()
return await _trigger_datasource_batch(db, datasources, force=force)
return await _trigger_datasource_batch(
db,
datasources,
force=force,
actor_id=current_user.id,
trigger_kind="all",
)
@router.post("/trigger-batch")
@@ -816,7 +830,13 @@ async def trigger_datasource_batch(
collected=None if payload.source_ids else payload.collected,
credential_status=None if payload.source_ids else payload.credential_status,
)
return await _trigger_datasource_batch(db, datasources, force=payload.force)
return await _trigger_datasource_batch(
db,
datasources,
force=payload.force,
actor_id=current_user.id,
trigger_kind="batch",
)
@router.get("/snapshots")
@@ -992,8 +1012,24 @@ async def trigger_datasource(
if not datasource.is_active:
raise HTTPException(status_code=400, detail="Data source is disabled")
running_task = await get_running_task(db, datasource.id)
running_task = await get_active_datasource_job(db, datasource.id, task_types=(JOB_TYPE_COLLECT,))
if running_task is not None and not force:
await emit_business_log(
logger,
event="collector.trigger.single.skipped_already_running",
message="Datasource trigger skipped because a task is already running",
category="collector",
level="warning",
service="api",
module=__name__,
user_id=current_user.id,
context={
"collector_name": datasource.source,
"datasource_id": datasource.id,
"task_id": running_task.id,
"status": "skipped",
},
)
raise HTTPException(
status_code=409,
detail={
@@ -1013,31 +1049,42 @@ async def trigger_datasource(
)
if running_task is not None and force:
cancelled = await cancel_running_collector_now(datasource.source)
if not cancelled:
await rollback_orphaned_running_task(db, datasource, running_task)
await request_cancel_datasource_task(
db,
running_task,
reason="superseded_by_forced_collection",
)
previous_task_id = await get_latest_task_id_for_datasource(datasource.id)
success = run_collector_now(datasource.source)
if not success:
raise HTTPException(status_code=500, detail=f"Failed to trigger collector '{datasource.source}'")
task_id = None
for _ in range(20):
await asyncio.sleep(0.1)
task_id = await get_latest_task_id_for_datasource(datasource.id)
if task_id is not None and task_id != previous_task_id:
break
if task_id == previous_task_id:
task_id = None
task = await enqueue_datasource_job(
db,
datasource,
JOB_TYPE_COLLECT,
payload={"force": force, "trigger": "single"},
)
await emit_business_log(
logger,
event="collector.trigger.single.completed",
message="Datasource trigger queued",
category="collector",
service="api",
module=__name__,
user_id=current_user.id,
context={
"collector_name": datasource.source,
"datasource_id": datasource.id,
"task_id": task.id,
"force": force,
"status": "queued",
},
)
return {
"status": "triggered",
"status": "queued",
"source_id": datasource.id,
"task_id": task_id,
"task_id": task.id,
"collector_name": datasource.source,
"force": force,
"message": f"Collector '{datasource.source}' has been triggered",
"message": f"Collector '{datasource.source}' has been queued",
}
@@ -1051,22 +1098,30 @@ async def clear_datasource_data(
if not datasource:
raise HTTPException(status_code=404, detail="Data source not found")
result = await db.execute(
select(func.count(CollectedData.id)).where(CollectedData.source == datasource.source)
active_task = await get_active_datasource_job(db, datasource.id)
if active_task is not None:
raise HTTPException(
status_code=409,
detail={
"reason": "datasource_job_in_progress",
"message": "当前数据源已有任务在执行,请等待完成或先取消任务。",
"task_id": active_task.id,
"task_type": active_task.task_type,
"status": active_task.status,
},
)
task = await enqueue_datasource_job(
db,
datasource,
JOB_TYPE_CLEAR_DATA,
payload={"source": datasource.source},
)
count = result.scalar() or 0
if count == 0:
return {"status": "success", "message": "No data to clear", "deleted_count": 0}
delete_query = CollectedData.__table__.delete().where(CollectedData.source == datasource.source)
await db.execute(delete_query)
await db.commit()
return {
"status": "success",
"message": f"Cleared {count} records for data source '{datasource.name}'",
"deleted_count": count,
"status": "queued",
"message": f"Queued data clearing for data source '{datasource.name}'",
"task_id": task.id,
"deleted_count": None,
}
@@ -1080,16 +1135,44 @@ async def clear_datasource_cache(
if not datasource:
raise HTTPException(status_code=404, detail="Data source not found")
earth_deleted_count = invalidate_earth_layer_cache_for_source(datasource.source)
dashboard_deleted_count = int(cache.delete("dashboard:stats")) + int(cache.delete("dashboard:summary"))
deleted_count = earth_deleted_count + dashboard_deleted_count
task = await enqueue_datasource_job(
db,
datasource,
JOB_TYPE_CLEAR_CACHE,
payload={"source": datasource.source},
dedupe_key=f"clear_cache:{datasource.source}",
)
return {
"status": "success",
"message": f"Cleared {deleted_count} cache keys for data source '{datasource.name}'",
"deleted_count": deleted_count,
"earth_layer_deleted_count": earth_deleted_count,
"dashboard_deleted_count": dashboard_deleted_count,
"status": "queued",
"message": f"Queued cache clearing for data source '{datasource.name}'",
"task_id": task.id,
"deleted_count": None,
}
@router.post("/{source_id}/tasks/{task_id}/cancel")
async def cancel_datasource_task(
source_id: str,
task_id: int,
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
datasource = await get_datasource_record(db, source_id)
if not datasource:
raise HTTPException(status_code=404, detail="Data source not found")
task = await db.get(CollectionTask, task_id)
if not task or task.datasource_id != datasource.id:
raise HTTPException(status_code=404, detail="Task not found")
task = await request_cancel_datasource_task(db, task)
return {
"status": "cancelled" if task.completed_at else "cancelling",
"task_id": task.id,
"task_type": task.task_type,
"phase": task.phase,
"requested_cancel_at": to_iso8601_utc(task.requested_cancel_at),
}
@@ -1109,7 +1192,7 @@ async def get_task_status(
if not task or task.datasource_id != datasource.id:
raise HTTPException(status_code=404, detail="Task not found")
else:
task = await get_running_task(db, datasource.id)
task = await get_active_datasource_job(db, datasource.id)
if task is None:
result = await db.execute(
select(CollectionTask)
@@ -1134,8 +1217,12 @@ async def get_task_status(
}
return {
"is_running": task.status == "running",
"is_running": task.status in {JOB_STATUS_QUEUED, JOB_STATUS_RUNNING, JOB_STATUS_CANCELLING}
and task.task_type == JOB_TYPE_COLLECT,
"is_task_active": task.status in {JOB_STATUS_QUEUED, JOB_STATUS_RUNNING, JOB_STATUS_CANCELLING},
"task_id": task.id,
"task_type": task.task_type,
"task_status": task.status,
"progress": task.progress,
"phase": task.phase,
"phase_progress": task.phase_progress,
@@ -1146,5 +1233,6 @@ async def get_task_status(
"records_processed": task.records_processed,
"total_records": task.total_records,
"status": task.status,
"requested_cancel_at": to_iso8601_utc(task.requested_cancel_at),
"error_message": task.error_message,
}

View File

@@ -12,6 +12,7 @@ from pydantic import BaseModel, Field
from sqlalchemy import delete, func, select, text
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.config import settings as app_settings
from app.core.security import decode_token, get_current_user, redis_client
from app.db.session import get_db
from app.models.collected_data import CollectedData
@@ -36,9 +37,16 @@ EARTH_BRAND_ASSET_DIR = REPO_ROOT / "data" / "earth-brand"
EARTH_BRAND_ASSET_URL_PREFIX = "/earth-brand-assets"
EARTH_BRAND_CATEGORY = "earth_brand"
EARTH_ABOUT_CATEGORY = "earth_about"
SYSTEM_SETTINGS_CATEGORY = "system"
MAX_EARTH_BRAND_ASSET_BYTES = 3 * 1024 * 1024
ALLOWED_EARTH_BRAND_EXTENSIONS = {".png", ".jpg", ".jpeg", ".webp", ".svg"}
def _app_version_label() -> str:
version = str(app_settings.VERSION or "").strip() or "0.0.0"
return version if version.startswith("v") else f"v{version}"
DEFAULT_EARTH_BRAND = {
"logo_src": "/earth/assets/brand/earth-logo.png",
"title_src": "/earth/assets/brand/title-zh.png",
@@ -53,14 +61,15 @@ DEFAULT_EARTH_ABOUT = {
"logo_src": "/earth/assets/brand/lim-logo.png",
"kicker": "About",
"title": "智能星球计划",
"version": "v0.64.0",
"version": _app_version_label(),
"description": "面向临空场景下的智能媒体研究、全球态势感知与多源开放数据巡航,提供可视化观测、事件聚合与交互式探索能力。",
"meta": [
{"label": "出品方", "value": "浙江大学临空智能媒体研究院"},
{"label": "策划人", "value": "黄柳青"},
{"label": "策划人", "value": "方兴东、黄柳青"},
{"label": "产品兼开发者", "value": "钱坤、张鸽、齐鹏"},
],
}
EARTH_ABOUT_LEGACY_PLANNER_VALUE = "黄柳青"
class EarthBoundaryConfigPayload(BaseModel):
@@ -116,11 +125,12 @@ def _normalize_earth_about_payload(payload: dict[str, Any] | None) -> dict[str,
}
raw_meta = DEFAULT_EARTH_ABOUT["meta"]
if payload:
for key in ("logo_src", "kicker", "title", "version", "description"):
for key in ("logo_src", "kicker", "title", "description"):
value = payload.get(key)
if value is not None:
merged[key] = str(value).strip()
raw_meta = payload.get("meta") if isinstance(payload.get("meta"), list) else raw_meta
merged["version"] = _app_version_label()
for key, default_value in DEFAULT_EARTH_ABOUT.items():
if key == "meta":
@@ -134,6 +144,8 @@ def _normalize_earth_about_payload(payload: dict[str, Any] | None) -> dict[str,
continue
label = str(item.get("label") or "").strip()
value = str(item.get("value") or "").strip()
if label == "策划人" and value == EARTH_ABOUT_LEGACY_PLANNER_VALUE:
value = "方兴东、黄柳青"
if label or value:
normalized_meta.append({"label": label, "value": value})
if not normalized_meta:
@@ -142,6 +154,10 @@ def _normalize_earth_about_payload(payload: dict[str, Any] | None) -> dict[str,
return merged
def _is_demo_mode_enabled(payload: Any) -> bool:
return bool(payload.get("demo_mode")) if isinstance(payload, dict) else False
async def _get_earth_brand_record(db: AsyncSession) -> SystemSetting | None:
result = await db.execute(
select(SystemSetting).where(SystemSetting.category == EARTH_BRAND_CATEGORY)
@@ -317,6 +333,11 @@ async def get_earth_oobe_status(
select(func.count(CollectedData.id)).where(CollectedData.is_current.is_(True))
)
current_record_count = int(current_count_result.scalar() or 0)
system_result = await db.execute(
select(SystemSetting).where(SystemSetting.category == SYSTEM_SETTINGS_CATEGORY)
)
system_record = system_result.scalar_one_or_none()
demo_mode = _is_demo_mode_enabled(system_record.payload if system_record else None)
datasource_count_result = await db.execute(select(func.count(DataSource.id)))
datasource_count = int(datasource_count_result.scalar() or 0)
@@ -337,6 +358,8 @@ async def get_earth_oobe_status(
ready = has_collected_data
suggestions: list[str] = []
if demo_mode:
suggestions.append("演示模式已开启")
if not current_user:
suggestions.append("登录控制台")
if not has_collected_data:
@@ -348,8 +371,9 @@ async def get_earth_oobe_status(
return {
"ready": ready,
"demo_mode": demo_mode,
"authenticated": current_user is not None,
"needs_login": current_user is None and not ready,
"needs_login": current_user is None and not ready and not demo_mode,
"has_collected_data": has_collected_data,
"has_tv_sources": tv_source_count > 0,
"has_core_layers": has_core_layers,

View File

@@ -0,0 +1,190 @@
"""CRUD APIs for persistent Earth interactables."""
from __future__ import annotations
from datetime import UTC, datetime
from typing import Any
from fastapi import APIRouter, Depends, HTTPException, Query, Response, status
from pydantic import BaseModel, Field, field_validator
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.security import get_current_user
from app.db.session import get_db
from app.models.earth_interactable import EarthInteractable
from app.models.user import User
from app.services.earth_interactables import (
build_interactable_event,
interactables_to_geojson,
invalidate_interactable_cache,
list_interactables,
normalize_interactable_id,
publish_interactable_event,
serialize_interactable,
)
from app.services.earth_layer_cache import (
EarthLayerCachePolicy,
earth_layer_cache,
get_or_build_layer_payload,
)
router = APIRouter()
INTERACTABLE_CACHE_POLICY = EarthLayerCachePolicy(
fresh_ttl_seconds=60,
stale_ttl_seconds=10 * 60,
max_features=5000,
)
class InteractableCreate(BaseModel):
id: str | None = Field(default=None, max_length=160)
layer: str = Field(default="default", min_length=1, max_length=80)
kind: str = Field(default="default", min_length=1, max_length=80)
label: str = Field(default="", max_length=255)
description: str = Field(default="", max_length=4000)
latitude: float = Field(ge=-90, le=90)
longitude: float = Field(ge=-180, le=180)
altitude: float | None = None
properties: dict[str, Any] = Field(default_factory=dict)
@field_validator("layer", "kind")
@classmethod
def normalize_key(cls, value: str) -> str:
normalized = str(value or "").strip()
if not normalized:
raise ValueError("must not be empty")
return normalized
class InteractableUpdate(BaseModel):
layer: str | None = Field(default=None, min_length=1, max_length=80)
kind: str | None = Field(default=None, min_length=1, max_length=80)
label: str | None = Field(default=None, max_length=255)
description: str | None = Field(default=None, max_length=4000)
latitude: float | None = Field(default=None, ge=-90, le=90)
longitude: float | None = Field(default=None, ge=-180, le=180)
altitude: float | None = None
properties: dict[str, Any] | None = None
@router.get("")
async def get_interactables(
response: Response,
layer: str | None = Query(default=None),
include_deleted: bool = Query(default=False),
db: AsyncSession = Depends(get_db),
):
items = await list_interactables(db, layer=layer, include_deleted=include_deleted)
response.headers["X-Planet-Interactables-Count"] = str(len(items))
return {"items": [serialize_interactable(item) for item in items]}
@router.get("/geojson")
async def get_interactables_geojson(
response: Response,
layer: str | None = Query(default=None),
db: AsyncSession = Depends(get_db),
):
async def build_payload() -> dict[str, Any]:
items = await list_interactables(db, layer=layer)
return interactables_to_geojson(items)
payload = await get_or_build_layer_payload(
key=earth_layer_cache.key("interactables", layer=layer or "all"),
policy=INTERACTABLE_CACHE_POLICY,
builder=build_payload,
response=response,
)
response.headers["X-Planet-Interactables-Count"] = str(len(payload.get("features") or []))
return payload
@router.post("", status_code=status.HTTP_201_CREATED)
async def create_interactable(
payload: InteractableCreate,
_current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
record_id = normalize_interactable_id(payload.id)
existing = await db.get(EarthInteractable, record_id)
if existing and not existing.is_deleted:
raise HTTPException(status_code=409, detail="Interactable already exists")
if existing is None:
record = EarthInteractable(id=record_id)
db.add(record)
else:
record = existing
record.is_deleted = False
record.deleted_at = None
record.revision += 1
record.layer = payload.layer
record.kind = payload.kind
record.label = payload.label
record.description = payload.description
record.latitude = payload.latitude
record.longitude = payload.longitude
record.altitude = payload.altitude
record.properties = payload.properties
await db.commit()
await db.refresh(record)
invalidate_interactable_cache(record.layer)
await publish_interactable_event("created", record)
return {"item": serialize_interactable(record)}
@router.get("/{interactable_id}")
async def get_interactable(interactable_id: str, db: AsyncSession = Depends(get_db)):
record = await db.get(EarthInteractable, interactable_id)
if record is None or record.is_deleted:
raise HTTPException(status_code=404, detail="Interactable not found")
return {"item": serialize_interactable(record)}
@router.patch("/{interactable_id}")
async def update_interactable(
interactable_id: str,
payload: InteractableUpdate,
_current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
record = await db.get(EarthInteractable, interactable_id)
if record is None or record.is_deleted:
raise HTTPException(status_code=404, detail="Interactable not found")
previous_layer = record.layer
patch = payload.model_dump(exclude_unset=True)
for key, value in patch.items():
setattr(record, key, value)
record.revision += 1
await db.commit()
await db.refresh(record)
invalidate_interactable_cache(previous_layer)
if record.layer != previous_layer:
invalidate_interactable_cache(record.layer)
await publish_interactable_event("updated", record)
return {"item": serialize_interactable(record)}
@router.delete("/{interactable_id}")
async def delete_interactable(
interactable_id: str,
_current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
record = await db.get(EarthInteractable, interactable_id)
if record is None or record.is_deleted:
raise HTTPException(status_code=404, detail="Interactable not found")
record.is_deleted = True
record.deleted_at = datetime.now(UTC)
record.revision += 1
await db.commit()
await db.refresh(record)
invalidate_interactable_cache(record.layer)
await publish_interactable_event("deleted", record)
event = build_interactable_event(action="deleted", record=record, include_item=True)
return {"deleted": True, "event": event}

View File

@@ -11,6 +11,7 @@ from dotenv import dotenv_values
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.logging import get_logger
from app.core.security import get_current_user
from app.core.time import to_iso8601_utc
from app.core.config import settings as app_settings
@@ -66,8 +67,10 @@ from app.services.llm_provider_catalog import (
from app.services.scheduler import sync_datasource_job
from app.services.tv_streams import DEFAULT_TV_SETTINGS, get_tv_settings_payload, normalize_tv_settings
from app.services.persistent_logs import record_audit_log
from app.services.business_logs import emit_business_log, exception_context
router = APIRouter()
logger = get_logger(__name__, service="api")
AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS = 5
AI_CONNECTION_TEST_PROMPT_KEY = "ai.connection_test"
SECRET_REVEAL_ROLES = {"admin", "super_admin"}
@@ -79,6 +82,7 @@ DEFAULT_SETTINGS = {
"auto_refresh": True,
"data_retention_days": 30,
"max_concurrent_tasks": 5,
"demo_mode": False,
},
"notifications": {
"email_enabled": False,
@@ -203,6 +207,7 @@ class SystemSettingsUpdate(BaseModel):
auto_refresh: bool = True
data_retention_days: int = Field(default=30, ge=1, le=3650)
max_concurrent_tasks: int = Field(default=5, ge=1, le=50)
demo_mode: bool = False
class NotificationSettingsUpdate(BaseModel):
@@ -699,6 +704,18 @@ async def _validate_ai_provider_full_connection(ai_payload: dict) -> dict:
retry_attempts=runtime_config["retry_attempts"],
llm_config=runtime_config.get("llm_config") or {},
)
await emit_business_log(
logger,
event="settings.ai_provider.full_connection.start",
message="AI provider full connection validation started",
category="ai",
service="api",
module=__name__,
context={
"provider": runtime_config.get("llm_config", {}).get("provider"),
"model": runtime_config.get("llm_config", {}).get("model"),
},
)
status_result = await client.get_status()
if not status_result.configured:
raise HTTPException(
@@ -715,6 +732,19 @@ async def _validate_ai_provider_full_connection(ai_payload: dict) -> dict:
constraints=["回复尽量简短。"],
)
)
await emit_business_log(
logger,
event="settings.ai_provider.full_connection.success",
message="AI provider full connection validation completed",
category="ai",
service="api",
module=__name__,
context={
"provider": analysis_result.provider,
"model": analysis_result.model,
"configured": status_result.configured,
},
)
return {
"status": status_result.model_dump(),
"provider": analysis_result.provider,
@@ -1612,9 +1642,34 @@ async def connect_ai_provider_integration(
llm_config=quick_llm_config,
)
await emit_business_log(
logger,
event="settings.ai_provider.connect.start",
message="AI provider connection test started",
category="ai",
service="api",
module=__name__,
user_id=current_user.id,
context={
"provider": payload.provider,
"model": payload.model,
"timeout_seconds": min(int(runtime_config["timeout_seconds"] or 60), AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS),
},
)
try:
status_result = await client.get_status()
if not status_result.configured:
await emit_business_log(
logger,
event="settings.ai_provider.connect.failed",
message="AI provider connection test failed because provider is incomplete",
category="ai",
level="warning",
service="api",
module=__name__,
user_id=current_user.id,
context={"provider": payload.provider, "model": payload.model, "configured": False},
)
return {
"success": False,
"connected": False,
@@ -1628,6 +1683,21 @@ async def connect_ai_provider_integration(
AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS,
),
)
await emit_business_log(
logger,
event="settings.ai_provider.connect.success",
message="AI provider connection test completed",
category="ai",
service="api",
module=__name__,
user_id=current_user.id,
context={
"provider": payload.provider,
"model": payload.model,
"configured": True,
"lightweight_status": lightweight_result.get("status"),
},
)
return {
**lightweight_result,
"status": status_result.model_dump(),
@@ -1639,6 +1709,17 @@ async def connect_ai_provider_integration(
"message": str(exc.detail),
}
except Exception as exc:
await emit_business_log(
logger,
event="settings.ai_provider.connect.failed",
message="AI provider connection test failed",
category="ai",
level="error",
service="api",
module=__name__,
user_id=current_user.id,
context=exception_context(exc, {"provider": payload.provider, "model": payload.model}),
)
return {
"success": False,
"connected": False,
@@ -1786,8 +1867,28 @@ async def connect_web_search_integration(
runtime_config = _runtime_config_from_web_search_payload(draft_web_search_payload)
client = WebSearchClient(runtime_config)
await emit_business_log(
logger,
event="settings.web_search.connect.start",
message="WebSearch connection test started",
category="ai_tool",
service="api",
module=__name__,
user_id=current_user.id,
context={"provider": runtime_config.default_provider},
)
try:
results = await client.test_connection()
await emit_business_log(
logger,
event="settings.web_search.connect.success",
message="WebSearch connection test completed",
category="ai_tool",
service="api",
module=__name__,
user_id=current_user.id,
context={"provider": runtime_config.default_provider, "result_count": len(results)},
)
return {
"success": True,
"connected": True,
@@ -1796,18 +1897,51 @@ async def connect_web_search_integration(
"results": [item.model_dump(mode="json") for item in results[:3]],
}
except WebSearchConfigurationError as exc:
await emit_business_log(
logger,
event="settings.web_search.connect.failed",
message="WebSearch connection test failed because configuration is incomplete",
category="ai_tool",
level="warning",
service="api",
module=__name__,
user_id=current_user.id,
context=exception_context(exc, {"provider": runtime_config.default_provider}),
)
return {
"success": False,
"connected": False,
"message": str(exc),
}
except WebSearchError as exc:
await emit_business_log(
logger,
event="settings.web_search.connect.failed",
message="WebSearch connection test failed",
category="ai_tool",
level="error",
service="api",
module=__name__,
user_id=current_user.id,
context=exception_context(exc, {"provider": runtime_config.default_provider}),
)
return {
"success": False,
"connected": False,
"message": str(exc),
}
except Exception as exc:
await emit_business_log(
logger,
event="settings.web_search.connect.failed",
message="WebSearch connection test failed",
category="ai_tool",
level="error",
service="api",
module=__name__,
user_id=current_user.id,
context=exception_context(exc, {"provider": runtime_config.default_provider}),
)
return {
"success": False,
"connected": False,
@@ -1835,17 +1969,60 @@ async def generate_provider_credential_guide(
ai_client: AIProviderClient = Depends(get_ai_provider_client),
):
try:
await emit_business_log(
logger,
event="settings.credential_guide.generate.start",
message="Credential guide generation started",
category="ai",
service="api",
module=__name__,
user_id=current_user.id,
context={"provider": provider},
)
web_search_client = await get_web_search_client(db)
return {
"guide": await generate_credential_guide(
guide = await generate_credential_guide(
db,
provider,
ai_client,
web_search_client,
)
}
await emit_business_log(
logger,
event="settings.credential_guide.generate.success",
message="Credential guide generation completed",
category="ai",
service="api",
module=__name__,
user_id=current_user.id,
context={"provider": provider},
)
return {"guide": guide}
except ValueError as exc:
await emit_business_log(
logger,
event="settings.credential_guide.generate.failed",
message="Credential guide generation failed",
category="ai",
level="warning",
service="api",
module=__name__,
user_id=current_user.id,
context=exception_context(exc, {"provider": provider}),
)
raise HTTPException(status_code=404, detail=str(exc)) from exc
except Exception as exc:
await emit_business_log(
logger,
event="settings.credential_guide.generate.failed",
message="Credential guide generation failed",
category="ai",
level="error",
service="api",
module=__name__,
user_id=current_user.id,
context=exception_context(exc, {"provider": provider}),
)
raise
@router.post("/credential-guides/{provider}/reset")

View File

@@ -1,6 +1,7 @@
from __future__ import annotations
import os
import json
import subprocess
import sys
@@ -44,6 +45,42 @@ from app.services.earth_layer_cache import earth_layer_cache
router = APIRouter()
def _compact_log_context(context: dict | None) -> str:
if not context:
return ""
allowed = {
key: value
for key, value in (context or {}).items()
if key
in {
"status",
"duration_ms",
"provider",
"model",
"result_provider",
"result_model",
"collector_name",
"datasource_id",
"task_id",
"snapshot_id",
"raw_count",
"transformed_count",
"saved_count",
"created",
"updated",
"unchanged",
"deleted",
"result_count",
"status_code",
"error_type",
"error",
}
}
if not allowed:
return ""
return json.dumps(allowed, ensure_ascii=False, sort_keys=True)
class RestartTaskCreate(BaseModel):
action: str
@@ -374,8 +411,11 @@ async def read_database_log_snapshot(
record.occurred_at.isoformat() if record.occurred_at else "",
record_level.upper(),
record.source,
record.category or "",
record.event or "",
f"request_id={record.request_id}" if record.request_id else "",
record.message,
_compact_log_context(record.context),
]
if part
)

View File

@@ -38,7 +38,7 @@ from app.services.compute_center_locations import (
upsert_compute_center_location,
)
from app.services.ai_client import get_ai_provider_client
from app.api.v1.settings import get_web_search_client
from app.api.v1.settings import get_runtime_web_search_config, get_web_search_client
from app.services.collectors.bgp_common import RIPE_RIS_COLLECTOR_COORDS
from app.services.location.llm_fallback import (
collect_llm_location_fallback_candidate,
@@ -105,8 +105,8 @@ LANDING_POINT_CACHE_POLICY = EarthLayerCachePolicy(
SATELLITE_CACHE_POLICY = EarthLayerCachePolicy(
SATELLITE_CACHE_FRESH_SECONDS,
SATELLITE_CACHE_STALE_SECONDS,
max_features=8000,
max_bytes=10 * BYTES_PER_MIB,
max_features=25000,
max_bytes=32 * BYTES_PER_MIB,
)
COMPUTE_CENTER_CACHE_POLICY = EarthLayerCachePolicy(
COMPUTE_CENTER_CACHE_FRESH_SECONDS,
@@ -1563,15 +1563,7 @@ async def _build_cables_geojson(db: AsyncSession) -> dict[str, Any]:
try:
records = await _load_current_collected_data(db, "arcgis_cables")
if not records:
raise HTTPException(
status_code=404,
detail="No cable data found. Please run the arcgis_cables collector first.",
)
return convert_cable_to_geojson(records)
except HTTPException:
raise
except Exception as e:
logger.exception_event(
"Failed to build cables GeoJSON response",
@@ -1625,16 +1617,8 @@ async def _build_landing_points_geojson(db: AsyncSession) -> dict[str, Any]:
relation_records,
cable_records,
)
if not records:
raise HTTPException(
status_code=404,
detail="No landing point data found. Please run the arcgis_landing_points collector first.",
)
return convert_landing_point_to_geojson(records, city_to_cable_ids_map, cable_id_to_name_map)
except HTTPException:
raise
except Exception as e:
logger.exception_event(
"Failed to build landing points GeoJSON response",
@@ -2071,6 +2055,44 @@ class SaveComputeCenterLocationRequest(BaseModel):
model_config = {"populate_by_name": True}
async def _compute_center_location_web_search_capability(db: AsyncSession) -> Dict[str, Any]:
try:
config = await get_runtime_web_search_config(db)
except Exception as exc:
return {
"enabled": False,
"provider": None,
"reason": f"WebSearch 配置读取失败:{exc}",
}
provider_config = config.active_provider_config
has_api_key = bool((provider_config.api_key or "").strip())
if not config.enabled:
return {
"enabled": False,
"provider": config.default_provider,
"reason": "WebSearch 未开启,无法进行事实核查定位。",
}
if not has_api_key:
return {
"enabled": False,
"provider": config.default_provider,
"reason": f"WebSearch Provider {config.default_provider} 未配置 API Key。",
}
return {
"enabled": True,
"provider": config.default_provider,
"reason": "",
}
@router.get("/compute-centers/location-capability")
async def get_compute_center_location_capability(
db: AsyncSession = Depends(get_db),
):
"""Return whether fact-checked compute-center location collection can run."""
return await _compute_center_location_web_search_capability(db)
@router.post("/compute-centers/{source_id}/collect-location")
async def collect_compute_center_location(
source_id: str,
@@ -2089,6 +2111,9 @@ async def collect_compute_center_location(
"""
if not source_id or not source_id.strip():
raise HTTPException(status_code=400, detail="source_id is required")
capability = await _compute_center_location_web_search_capability(db)
if not capability.get("enabled"):
raise HTTPException(status_code=409, detail=capability)
record = await _load_compute_center_record(db, source_id)
name = payload.name or (record.name if record else None)
@@ -2696,6 +2721,8 @@ async def _build_bgp_collectors_geojson(db: AsyncSession) -> dict[str, Any]:
db,
source_filter=("ris_live_bgp", "bgpstream_bgp"),
)
if not any(int(item.get("observation_count") or 0) > 0 for item in coverage):
return {"type": "FeatureCollection", "features": [], "count": 0}
coverage_by_collector = {
item["collector"]: item
for item in coverage

View File

@@ -150,7 +150,7 @@ class DataBroadcaster:
"timestamp": to_iso8601_utc(datetime.now(UTC)),
"payload": data,
},
channel="all",
channel="datasource_tasks",
)
def start(self):

View File

@@ -203,6 +203,7 @@ async def init_db():
import app.models.vessel_enrichment # noqa: F401
import app.models.datasource_mapping # noqa: F401
import app.models.earth_news # noqa: F401
import app.models.earth_interactable # noqa: F401
logger.warning_event(
"Database pool settings active",
@@ -258,6 +259,406 @@ async def init_db():
"""
)
)
await conn.execute(
text(
"""
CREATE TABLE IF NOT EXISTS earth_data_change_events (
id BIGSERIAL PRIMARY KEY,
table_name VARCHAR(128) NOT NULL,
operation VARCHAR(16) NOT NULL,
source VARCHAR(128),
entity_key VARCHAR(255),
payload JSONB NOT NULL,
occurred_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
consumed_at TIMESTAMPTZ
)
"""
)
)
await conn.execute(
text(
"""
CREATE INDEX IF NOT EXISTS idx_earth_data_change_events_unconsumed
ON earth_data_change_events (consumed_at, id)
WHERE consumed_at IS NULL
"""
)
)
await conn.execute(
text(
"""
CREATE OR REPLACE FUNCTION planet_emit_earth_data_changed_statement(
change_table TEXT,
change_operation TEXT,
change_source TEXT,
source_record_count INTEGER,
source_entity_keys TEXT[]
)
RETURNS VOID AS $$
DECLARE
change_event_id BIGINT;
change_payload JSONB;
BEGIN
change_payload := jsonb_build_object(
'event', 'earth.layer.changed',
'table', change_table,
'operation', change_operation,
'source', change_source,
'entity_key', NULL,
'entity_keys', COALESCE(to_jsonb(source_entity_keys), '[]'::jsonb),
'records_processed', COALESCE(source_record_count, 0),
'occurred_at', NOW()
);
INSERT INTO earth_data_change_events (
table_name,
operation,
source,
entity_key,
payload,
occurred_at
) VALUES (
change_table,
change_operation,
change_source,
NULL,
change_payload,
NOW()
)
RETURNING id INTO change_event_id;
change_payload := change_payload || jsonb_build_object(
'event_id', change_event_id
);
UPDATE earth_data_change_events
SET payload = change_payload
WHERE id = change_event_id;
PERFORM pg_notify(
'planet_earth_data_changes',
change_payload::text
);
END;
$$ LANGUAGE plpgsql;
"""
)
)
await conn.execute(
text(
"""
CREATE OR REPLACE FUNCTION planet_emit_collected_data_changed_statement(
change_operation TEXT,
change_source TEXT,
source_record_count INTEGER,
source_entity_keys TEXT[]
)
RETURNS VOID AS $$
BEGIN
PERFORM planet_emit_earth_data_changed_statement(
'collected_data',
change_operation,
change_source,
source_record_count,
source_entity_keys
);
END;
$$ LANGUAGE plpgsql;
"""
)
)
await conn.execute(
text(
"""
CREATE OR REPLACE FUNCTION planet_notify_earth_table_changed_statement()
RETURNS trigger AS $$
DECLARE
change_source TEXT;
source_record_count INTEGER;
source_entity_keys TEXT[];
BEGIN
IF TG_OP = 'INSERT' THEN
FOR change_source IN
SELECT DISTINCT COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME)
FROM (SELECT to_jsonb(t) AS row_data FROM new_rows AS t) changed_rows
LOOP
SELECT
COUNT(*),
ARRAY(
SELECT DISTINCT COALESCE(
NULLIF(row_data->>'entity_key', ''),
NULLIF(row_data->>'source_id', ''),
NULLIF(row_data->>'incident_key', ''),
NULLIF(row_data->>'id', ''),
NULLIF(row_data->>'mmsi', '')
)
FROM (SELECT to_jsonb(t) AS row_data FROM new_rows AS t) rows_for_keys
WHERE COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) = change_source
LIMIT 20
)
INTO source_record_count, source_entity_keys
FROM (SELECT to_jsonb(t) AS row_data FROM new_rows AS t) rows_for_count
WHERE COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) = change_source;
PERFORM planet_emit_earth_data_changed_statement(
TG_TABLE_NAME,
TG_OP,
change_source,
source_record_count,
source_entity_keys
);
END LOOP;
ELSIF TG_OP = 'DELETE' THEN
FOR change_source IN
SELECT DISTINCT COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME)
FROM (SELECT to_jsonb(t) AS row_data FROM old_rows AS t) changed_rows
LOOP
SELECT
COUNT(*),
ARRAY(
SELECT DISTINCT COALESCE(
NULLIF(row_data->>'entity_key', ''),
NULLIF(row_data->>'source_id', ''),
NULLIF(row_data->>'incident_key', ''),
NULLIF(row_data->>'id', ''),
NULLIF(row_data->>'mmsi', '')
)
FROM (SELECT to_jsonb(t) AS row_data FROM old_rows AS t) rows_for_keys
WHERE COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) = change_source
LIMIT 20
)
INTO source_record_count, source_entity_keys
FROM (SELECT to_jsonb(t) AS row_data FROM old_rows AS t) rows_for_count
WHERE COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) = change_source;
PERFORM planet_emit_earth_data_changed_statement(
TG_TABLE_NAME,
TG_OP,
change_source,
source_record_count,
source_entity_keys
);
END LOOP;
ELSIF TG_OP = 'UPDATE' THEN
FOR change_source IN
SELECT DISTINCT COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME)
FROM (
SELECT to_jsonb(t) AS row_data FROM new_rows AS t
UNION ALL
SELECT to_jsonb(t) AS row_data FROM old_rows AS t
) changed_rows
LOOP
SELECT
COUNT(*),
ARRAY(
SELECT DISTINCT COALESCE(
NULLIF(row_data->>'entity_key', ''),
NULLIF(row_data->>'source_id', ''),
NULLIF(row_data->>'incident_key', ''),
NULLIF(row_data->>'id', ''),
NULLIF(row_data->>'mmsi', '')
)
FROM (
SELECT to_jsonb(t) AS row_data FROM new_rows AS t
UNION ALL
SELECT to_jsonb(t) AS row_data FROM old_rows AS t
) rows_for_keys
WHERE COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) = change_source
LIMIT 20
)
INTO source_record_count, source_entity_keys
FROM (
SELECT to_jsonb(t) AS row_data FROM new_rows AS t
UNION ALL
SELECT to_jsonb(t) AS row_data FROM old_rows AS t
) rows_for_count
WHERE COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) = change_source;
PERFORM planet_emit_earth_data_changed_statement(
TG_TABLE_NAME,
TG_OP,
change_source,
source_record_count,
source_entity_keys
);
END LOOP;
END IF;
RETURN NULL;
END;
$$ LANGUAGE plpgsql;
"""
)
)
await conn.execute(
text(
"""
CREATE OR REPLACE FUNCTION planet_notify_collected_data_changed_statement()
RETURNS trigger AS $$
DECLARE
change_source TEXT;
source_record_count INTEGER;
source_entity_keys TEXT[];
BEGIN
IF TG_OP = 'INSERT' THEN
FOR change_source IN
SELECT DISTINCT source FROM new_rows WHERE source IS NOT NULL
LOOP
SELECT
COUNT(*),
ARRAY(
SELECT DISTINCT COALESCE(entity_key, source_id, id::text)
FROM new_rows
WHERE source = change_source
LIMIT 20
)
INTO source_record_count, source_entity_keys
FROM new_rows
WHERE source = change_source;
PERFORM planet_emit_collected_data_changed_statement(
TG_OP,
change_source,
source_record_count,
source_entity_keys
);
END LOOP;
ELSIF TG_OP = 'DELETE' THEN
FOR change_source IN
SELECT DISTINCT source FROM old_rows WHERE source IS NOT NULL
LOOP
SELECT
COUNT(*),
ARRAY(
SELECT DISTINCT COALESCE(entity_key, source_id, id::text)
FROM old_rows
WHERE source = change_source
LIMIT 20
)
INTO source_record_count, source_entity_keys
FROM old_rows
WHERE source = change_source;
PERFORM planet_emit_collected_data_changed_statement(
TG_OP,
change_source,
source_record_count,
source_entity_keys
);
END LOOP;
ELSIF TG_OP = 'UPDATE' THEN
FOR change_source IN
SELECT DISTINCT source FROM (
SELECT source FROM new_rows
UNION
SELECT source FROM old_rows
) changed_sources
WHERE source IS NOT NULL
LOOP
SELECT
COUNT(*),
ARRAY(
SELECT DISTINCT COALESCE(entity_key, source_id, id::text)
FROM (
SELECT id, source_id, entity_key, source FROM new_rows
UNION ALL
SELECT id, source_id, entity_key, source FROM old_rows
) changed_rows
WHERE source = change_source
LIMIT 20
)
INTO source_record_count, source_entity_keys
FROM (
SELECT id, source_id, entity_key, source FROM new_rows
UNION ALL
SELECT id, source_id, entity_key, source FROM old_rows
) changed_rows
WHERE source = change_source;
PERFORM planet_emit_collected_data_changed_statement(
TG_OP,
change_source,
source_record_count,
source_entity_keys
);
END LOOP;
END IF;
RETURN NULL;
END;
$$ LANGUAGE plpgsql;
"""
)
)
for statement in (
"DROP TRIGGER IF EXISTS tr_planet_collected_data_changed ON collected_data",
"DROP TRIGGER IF EXISTS tr_planet_collected_data_changed_insert ON collected_data",
"DROP TRIGGER IF EXISTS tr_planet_collected_data_changed_update ON collected_data",
"DROP TRIGGER IF EXISTS tr_planet_collected_data_changed_delete ON collected_data",
"DROP FUNCTION IF EXISTS planet_notify_collected_data_changed()",
"""
CREATE TRIGGER tr_planet_collected_data_changed_insert
AFTER INSERT ON collected_data
REFERENCING NEW TABLE AS new_rows
FOR EACH STATEMENT
EXECUTE FUNCTION planet_notify_collected_data_changed_statement()
""",
"""
CREATE TRIGGER tr_planet_collected_data_changed_update
AFTER UPDATE ON collected_data
REFERENCING OLD TABLE AS old_rows NEW TABLE AS new_rows
FOR EACH STATEMENT
EXECUTE FUNCTION planet_notify_collected_data_changed_statement()
""",
"""
CREATE TRIGGER tr_planet_collected_data_changed_delete
AFTER DELETE ON collected_data
REFERENCING OLD TABLE AS old_rows
FOR EACH STATEMENT
EXECUTE FUNCTION planet_notify_collected_data_changed_statement()
""",
):
await conn.execute(text(statement))
for table_name in (
"bgp_observations",
"bgp_anomalies",
"bgp_incidents",
"bgp_collector_locations",
"vessel_static",
"vessel_position",
"ais_raw_observations",
"ais_source_health",
"compute_center_locations",
"earth_interactables",
"earth_news_items",
):
for statement in (
f"DROP TRIGGER IF EXISTS tr_planet_{table_name}_changed_insert ON {table_name}",
f"DROP TRIGGER IF EXISTS tr_planet_{table_name}_changed_update ON {table_name}",
f"DROP TRIGGER IF EXISTS tr_planet_{table_name}_changed_delete ON {table_name}",
f"""
CREATE TRIGGER tr_planet_{table_name}_changed_insert
AFTER INSERT ON {table_name}
REFERENCING NEW TABLE AS new_rows
FOR EACH STATEMENT
EXECUTE FUNCTION planet_notify_earth_table_changed_statement()
""",
f"""
CREATE TRIGGER tr_planet_{table_name}_changed_update
AFTER UPDATE ON {table_name}
REFERENCING OLD TABLE AS old_rows NEW TABLE AS new_rows
FOR EACH STATEMENT
EXECUTE FUNCTION planet_notify_earth_table_changed_statement()
""",
f"""
CREATE TRIGGER tr_planet_{table_name}_changed_delete
AFTER DELETE ON {table_name}
REFERENCING OLD TABLE AS old_rows
FOR EACH STATEMENT
EXECUTE FUNCTION planet_notify_earth_table_changed_statement()
""",
):
await conn.execute(text(statement))
await conn.execute(
text(
"""
@@ -267,7 +668,16 @@ async def init_db():
ADD COLUMN IF NOT EXISTS phase_message VARCHAR(255),
ADD COLUMN IF NOT EXISTS phase_current BIGINT,
ADD COLUMN IF NOT EXISTS phase_total BIGINT,
ADD COLUMN IF NOT EXISTS phase_unit VARCHAR(30)
ADD COLUMN IF NOT EXISTS phase_unit VARCHAR(30),
ADD COLUMN IF NOT EXISTS source VARCHAR(100),
ADD COLUMN IF NOT EXISTS task_type VARCHAR(30) NOT NULL DEFAULT 'collect',
ADD COLUMN IF NOT EXISTS payload JSONB NOT NULL DEFAULT '{}'::jsonb,
ADD COLUMN IF NOT EXISTS rollback_policy VARCHAR(40) NOT NULL DEFAULT 'keep_committed_batches',
ADD COLUMN IF NOT EXISTS dedupe_key VARCHAR(180),
ADD COLUMN IF NOT EXISTS worker_id VARCHAR(120),
ADD COLUMN IF NOT EXISTS locked_at TIMESTAMPTZ,
ADD COLUMN IF NOT EXISTS requested_cancel_at TIMESTAMPTZ,
ADD COLUMN IF NOT EXISTS cancel_reason TEXT
"""
)
)
@@ -283,6 +693,17 @@ async def init_db():
"""
)
)
await conn.execute(
text(
"""
ALTER TABLE earth_interactables
ADD COLUMN IF NOT EXISTS altitude DOUBLE PRECISION,
ADD COLUMN IF NOT EXISTS revision INTEGER NOT NULL DEFAULT 1,
ADD COLUMN IF NOT EXISTS is_deleted BOOLEAN NOT NULL DEFAULT FALSE,
ADD COLUMN IF NOT EXISTS deleted_at TIMESTAMPTZ
"""
)
)
await conn.execute(
text(
"""
@@ -307,6 +728,48 @@ async def init_db():
"""
)
)
await conn.execute(
text(
"""
CREATE INDEX IF NOT EXISTS idx_earth_interactables_layer_deleted
ON earth_interactables (layer, is_deleted)
"""
)
)
await conn.execute(
text(
"""
CREATE INDEX IF NOT EXISTS idx_earth_interactables_updated_at
ON earth_interactables (updated_at)
"""
)
)
await conn.execute(
text(
"""
CREATE INDEX IF NOT EXISTS idx_collection_tasks_source_status
ON collection_tasks (source, status)
"""
)
)
await conn.execute(
text(
"""
CREATE INDEX IF NOT EXISTS idx_collection_tasks_queue
ON collection_tasks (status, created_at, id)
WHERE status = 'queued'
"""
)
)
await conn.execute(
text(
"""
CREATE INDEX IF NOT EXISTS idx_collection_tasks_dedupe
ON collection_tasks (dedupe_key)
WHERE dedupe_key IS NOT NULL
"""
)
)
await conn.execute(
text(
"""

View File

@@ -24,6 +24,11 @@ from app.services.earth_news_worker import (
start_earth_news_target_worker,
stop_earth_news_target_worker,
)
from app.services.earth_db_change_listener import (
start_earth_db_change_listener,
stop_earth_db_change_listener,
)
from app.services.data_jobs import start_data_job_worker, stop_data_job_worker
configure_logging()
@@ -59,9 +64,13 @@ async def lifespan(app: FastAPI):
start_scheduler()
await sync_scheduler_with_datasources()
broadcaster.start()
start_data_job_worker()
start_earth_db_change_listener()
start_earth_news_target_worker()
yield
await stop_earth_news_target_worker()
await stop_earth_db_change_listener()
await stop_data_job_worker()
broadcaster.stop()
stop_scheduler()

View File

@@ -17,6 +17,7 @@ from app.models.system_log import SystemLog, AuditLog
from app.models.vessel import AISConflictRecord, AISRawObservation, AISSourceHealth, VesselPosition, VesselStatic
from app.models.datasource_mapping import DataSourceMappingTemplate
from app.models.earth_news import EarthNewsItem
from app.models.earth_interactable import EarthInteractable
__all__ = [
"User",
@@ -45,4 +46,5 @@ __all__ = [
"AISSourceHealth",
"DataSourceMappingTemplate",
"EarthNewsItem",
"EarthInteractable",
]

View File

@@ -0,0 +1,30 @@
"""Persistent Earth interactable objects."""
from sqlalchemy import Boolean, Column, DateTime, Float, Index, Integer, JSON, String, Text
from sqlalchemy.sql import func
from app.db.session import Base
class EarthInteractable(Base):
__tablename__ = "earth_interactables"
id = Column(String(160), primary_key=True)
layer = Column(String(80), nullable=False, default="interactables", index=True)
kind = Column(String(80), nullable=False, default="default", index=True)
label = Column(String(255), nullable=False, default="")
description = Column(Text, nullable=False, default="")
latitude = Column(Float, nullable=False)
longitude = Column(Float, nullable=False)
altitude = Column(Float, nullable=True)
revision = Column(Integer, nullable=False, default=1)
properties = Column(JSON, nullable=False, default=dict)
is_deleted = Column(Boolean, nullable=False, default=False, index=True)
created_at = Column(DateTime(timezone=True), server_default=func.now(), nullable=False)
updated_at = Column(DateTime(timezone=True), server_default=func.now(), onupdate=func.now())
deleted_at = Column(DateTime(timezone=True), nullable=True, index=True)
__table_args__ = (
Index("idx_earth_interactables_layer_deleted", "layer", "is_deleted"),
Index("idx_earth_interactables_updated_at", "updated_at"),
)

View File

@@ -1,6 +1,6 @@
"""Collection Task model"""
"""Datasource job model."""
from sqlalchemy import BigInteger, Column, DateTime, Integer, String, Text, Float
from sqlalchemy import BigInteger, Column, DateTime, Float, Integer, JSON, String, Text
from sqlalchemy.sql import func
from app.db.session import Base
@@ -11,7 +11,9 @@ class CollectionTask(Base):
id = Column(Integer, primary_key=True, autoincrement=True)
datasource_id = Column(Integer, nullable=False, index=True)
status = Column(String(20), nullable=False) # pending, running, success, failed, cancelled
source = Column(String(100), nullable=True, index=True)
task_type = Column(String(30), nullable=False, default="collect", index=True)
status = Column(String(20), nullable=False) # queued, running, cancelling, success, failed, cancelled
phase = Column(String(30), default="queued")
phase_progress = Column(Float)
phase_message = Column(String(255))
@@ -24,6 +26,13 @@ class CollectionTask(Base):
total_records = Column(Integer, default=0) # Total records to process
progress = Column(Float, default=0.0) # Progress percentage (0-100)
error_message = Column(Text)
payload = Column(JSON, default=dict)
rollback_policy = Column(String(40), nullable=False, default="keep_committed_batches")
dedupe_key = Column(String(180), nullable=True, index=True)
worker_id = Column(String(120), nullable=True, index=True)
locked_at = Column(DateTime(timezone=True), nullable=True, index=True)
requested_cancel_at = Column(DateTime(timezone=True), nullable=True)
cancel_reason = Column(Text)
created_at = Column(DateTime(timezone=True), server_default=func.now())
def __repr__(self):

View File

@@ -2,18 +2,24 @@ from __future__ import annotations
import asyncio
import json
from time import perf_counter
import httpx
from fastapi import Depends, HTTPException, status
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.config import settings
from app.core.logging import get_logger
from app.db.session import get_db
from app.schemas.ai import (
AIProviderStatusResponse,
SituationalAnalysisRequest,
SituationalAnalysisResponse,
)
from app.services.business_logs import emit_business_log, exception_context
logger = get_logger(__name__, service="ai")
class AIProviderClient:
@@ -64,7 +70,19 @@ class AIProviderClient:
return headers
async def get_status(self, request_id: str | None = None) -> AIProviderStatusResponse:
context = self._base_log_context(operation="status")
if not self.service_url:
await emit_business_log(
logger,
event="ai.provider.status.failed",
message="AI provider status skipped because service URL is not configured",
category="ai",
level="warning",
service="ai",
module=__name__,
request_id=request_id,
context={**context, "status": "unconfigured"},
)
return AIProviderStatusResponse(
provider="unconfigured",
enabled=False,
@@ -73,27 +91,133 @@ class AIProviderClient:
base_url=None,
)
data = await self._request("GET", "/v1/provider/status", request_id=request_id)
return AIProviderStatusResponse.model_validate(data)
started_at = perf_counter()
await emit_business_log(
logger,
event="ai.provider.status.start",
message="AI provider status request started",
category="ai",
service="ai",
module=__name__,
request_id=request_id,
context=context,
)
try:
data = await self._request("GET", "/v1/provider/status", request_id=request_id, operation="status")
result = AIProviderStatusResponse.model_validate(data)
await emit_business_log(
logger,
event="ai.provider.status.success",
message="AI provider status request completed",
category="ai",
service="ai",
module=__name__,
request_id=request_id,
context={
**context,
"status": "success",
"duration_ms": self._duration_ms(started_at),
"result_provider": result.provider,
"result_model": result.model,
"configured": result.configured,
"enabled": result.enabled,
},
)
return result
except Exception as exc:
await emit_business_log(
logger,
event="ai.provider.status.failed",
message="AI provider status request failed",
category="ai",
level="error",
service="ai",
module=__name__,
request_id=request_id,
context=exception_context(exc, {**context, "status": "failed", "duration_ms": self._duration_ms(started_at)}),
)
raise
async def analyze(
self,
payload: SituationalAnalysisRequest,
request_id: str | None = None,
) -> SituationalAnalysisResponse:
context = self._base_log_context(
operation="analyze",
preferred_model=payload.preferred_model,
input_summary=self._summarize_analysis_payload(payload),
)
if not self.service_url:
await emit_business_log(
logger,
event="ai.provider.analyze.failed",
message="AI provider analyze skipped because service URL is not configured",
category="ai",
level="warning",
service="ai",
module=__name__,
request_id=request_id,
context={**context, "status": "unconfigured"},
)
raise HTTPException(
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
detail="AI provider service URL is not configured.",
)
data = await self._request(
"POST",
"/v1/analyze",
json=payload.model_dump(),
started_at = perf_counter()
await emit_business_log(
logger,
event="ai.provider.analyze.start",
message="AI provider analyze request started",
category="ai",
service="ai",
module=__name__,
request_id=request_id,
context=context,
)
return SituationalAnalysisResponse.model_validate(data)
try:
data = await self._request(
"POST",
"/v1/analyze",
json=payload.model_dump(),
request_id=request_id,
operation="analyze",
payload_summary=context["input_summary"],
)
result = SituationalAnalysisResponse.model_validate(data)
await emit_business_log(
logger,
event="ai.provider.analyze.success",
message="AI provider analyze request completed",
category="ai",
service="ai",
module=__name__,
request_id=request_id,
context={
**context,
"status": "success",
"duration_ms": self._duration_ms(started_at),
"result_provider": result.provider,
"result_model": result.model,
"content_block_count": len(result.content_blocks or []),
"thinking_block_count": len(result.thinking_blocks or []),
},
)
return result
except Exception as exc:
await emit_business_log(
logger,
event="ai.provider.analyze.failed",
message="AI provider analyze request failed",
category="ai",
level="error",
service="ai",
module=__name__,
request_id=request_id,
context=exception_context(exc, {**context, "status": "failed", "duration_ms": self._duration_ms(started_at)}),
)
raise
async def _request(
self,
@@ -101,9 +225,12 @@ class AIProviderClient:
path: str,
json: dict | None = None,
request_id: str | None = None,
operation: str = "request",
payload_summary: dict | None = None,
) -> dict:
last_error: Exception | None = None
for attempt in range(1, self.retry_attempts + 1):
attempt_started_at = perf_counter()
try:
async with httpx.AsyncClient(timeout=self.timeout) as client:
response = await client.request(
@@ -117,6 +244,15 @@ class AIProviderClient:
except httpx.HTTPStatusError as exc:
last_error = exc
if attempt < self.retry_attempts and exc.response.status_code >= 500:
await self._log_retry(
operation=operation,
request_id=request_id,
attempt=attempt,
status_code=exc.response.status_code,
duration_ms=self._duration_ms(attempt_started_at),
error=exc,
payload_summary=payload_summary,
)
await asyncio.sleep(0.3 * attempt)
continue
detail = exc.response.text or "AI provider service returned an error"
@@ -127,6 +263,14 @@ class AIProviderClient:
except httpx.HTTPError as exc:
last_error = exc
if attempt < self.retry_attempts:
await self._log_retry(
operation=operation,
request_id=request_id,
attempt=attempt,
duration_ms=self._duration_ms(attempt_started_at),
error=exc,
payload_summary=payload_summary,
)
await asyncio.sleep(0.3 * attempt)
continue
raise HTTPException(
@@ -139,6 +283,71 @@ class AIProviderClient:
detail=f"AI provider service request failed: {last_error}",
)
def _base_log_context(self, **extra: object) -> dict:
llm_provider_apis = self.llm_config.get("model_provider_apis")
return {
"provider": self.llm_config.get("provider") or "",
"provider_api": self.llm_config.get("provider_api") or "",
"model": self.llm_config.get("model") or "",
"base_url_configured": bool(self.llm_config.get("base_url")),
"service_url_configured": bool(self.service_url),
"timeout_seconds": self.timeout,
"retry_attempts": self.retry_attempts,
"model_provider_api_count": len(llm_provider_apis or {}) if isinstance(llm_provider_apis, dict) else 0,
**extra,
}
@staticmethod
def _duration_ms(started_at: float) -> int:
return int((perf_counter() - started_at) * 1000)
@staticmethod
def _summarize_analysis_payload(payload: SituationalAnalysisRequest) -> dict:
context = payload.context if isinstance(payload.context, dict) else {}
thinking = payload.thinking if isinstance(payload.thinking, dict) else payload.thinking
return {
"title_length": len(payload.title or ""),
"objective_length": len(payload.objective or ""),
"observation_count": len(payload.observations or []),
"constraint_count": len(payload.constraints or []),
"has_system_prompt": bool(payload.system_prompt),
"thinking_enabled": bool(thinking),
"context_keys": sorted(str(key) for key in context.keys()),
}
async def _log_retry(
self,
*,
operation: str,
request_id: str | None,
attempt: int,
duration_ms: int,
error: BaseException,
status_code: int | None = None,
payload_summary: dict | None = None,
) -> None:
await emit_business_log(
logger,
event=f"ai.provider.{operation}.retry",
message="AI provider request will retry",
category="ai",
level="warning",
service="ai",
module=__name__,
request_id=request_id,
context=exception_context(
error,
{
**self._base_log_context(operation=operation),
"attempt": attempt,
"next_attempt": attempt + 1,
"status_code": status_code,
"duration_ms": duration_ms,
"input_summary": payload_summary,
},
),
)
async def get_ai_provider_client(db: AsyncSession = Depends(get_db)) -> AIProviderClient:
from app.api.v1.settings import get_runtime_ai_provider_config

View File

@@ -1,11 +1,18 @@
from __future__ import annotations
import hashlib
from time import perf_counter
from urllib.parse import urlparse
import httpx
from bs4 import BeautifulSoup
from app.core.logging import get_logger
from app.services.ai_tools.schemas import FetchedEvidence
from app.services.business_logs import emit_business_log, exception_context
logger = get_logger(__name__, service="ai_tool")
class WebFetchError(RuntimeError):
@@ -29,8 +36,33 @@ async def fetch_url_evidence(
timeout_seconds: int = 20,
max_bytes: int = 1_500_000,
) -> FetchedEvidence:
started_at = perf_counter()
if not url:
await emit_business_log(
logger,
event="ai_tool.web_fetch.failed",
message="WebFetch failed because URL is empty",
category="ai_tool",
level="warning",
service="ai_tool",
module=__name__,
context={"reason": "empty_url"},
)
raise WebFetchError("url is required")
request_host = urlparse(url).netloc
await emit_business_log(
logger,
event="ai_tool.web_fetch.start",
message="WebFetch request started",
category="ai_tool",
service="ai_tool",
module=__name__,
context={
"url_host": request_host,
"timeout_seconds": timeout_seconds,
"max_bytes": max_bytes,
},
)
try:
async with httpx.AsyncClient(
timeout=timeout_seconds,
@@ -41,10 +73,45 @@ async def fetch_url_evidence(
response.raise_for_status()
content = response.content[:max_bytes]
except httpx.HTTPError as exc:
await emit_business_log(
logger,
event="ai_tool.web_fetch.failed",
message="WebFetch request failed",
category="ai_tool",
level="error",
service="ai_tool",
module=__name__,
context=exception_context(
exc,
{
"url_host": request_host,
"status": "failed",
"duration_ms": int((perf_counter() - started_at) * 1000),
},
),
)
raise WebFetchError(f"failed to fetch page: {exc}") from exc
title, text = _extract_title_and_text(content.decode(response.encoding or "utf-8", errors="ignore"))
content_hash = hashlib.sha256(text.encode("utf-8")).hexdigest()
await emit_business_log(
logger,
event="ai_tool.web_fetch.success",
message="WebFetch request completed",
category="ai_tool",
service="ai_tool",
module=__name__,
context={
"url_host": request_host,
"final_url_host": urlparse(str(response.url)).netloc,
"status": "success",
"status_code": response.status_code,
"bytes_read": len(content),
"content_hash": content_hash,
"duration_ms": int((perf_counter() - started_at) * 1000),
"extractor": "beautifulsoup_basic",
},
)
return FetchedEvidence(
url=url,
final_url=str(response.url),
@@ -53,4 +120,3 @@ async def fetch_url_evidence(
content_hash=content_hash,
extractor="beautifulsoup_basic",
)

View File

@@ -1,11 +1,18 @@
from __future__ import annotations
from copy import deepcopy
import hashlib
from time import perf_counter
from typing import Any
import httpx
from app.core.logging import get_logger
from app.services.ai_tools.schemas import SearchEvidence, WebSearchConfig, WebSearchProviderConfig
from app.services.business_logs import emit_business_log, exception_context
logger = get_logger(__name__, service="ai_tool")
WEB_SEARCH_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
@@ -120,29 +127,108 @@ class WebSearchClient:
domains: list[str] | None = None,
freshness_days: int | None = None,
) -> list[SearchEvidence]:
started_at = perf_counter()
if not self.config.enabled:
await emit_business_log(
logger,
event="ai_tool.web_search.unavailable",
message="WebSearch skipped because integration is disabled",
category="ai_tool",
level="warning",
service="ai_tool",
module=__name__,
context={"provider": self.config.default_provider, "reason": "disabled"},
)
raise WebSearchConfigurationError("WebSearch is disabled.")
provider_config = self.config.active_provider_config
provider = normalize_web_search_provider(provider_config.provider)
if provider != "searxng" and not provider_config.api_key:
await emit_business_log(
logger,
event="ai_tool.web_search.unavailable",
message="WebSearch skipped because API key is not configured",
category="ai_tool",
level="warning",
service="ai_tool",
module=__name__,
context={"provider": provider, "reason": "missing_api_key"},
)
raise WebSearchConfigurationError(f"{provider} API key is not configured.")
query = " ".join(str(query or "").split())
if not query:
await emit_business_log(
logger,
event="ai_tool.web_search.failed",
message="WebSearch failed because query is empty",
category="ai_tool",
level="warning",
service="ai_tool",
module=__name__,
context={"provider": provider, "reason": "empty_query"},
)
raise WebSearchConfigurationError("search query is required.")
limit = max_results or provider_config.max_results
if provider == "tavily":
return await self._search_tavily(provider_config, query, limit, domains, freshness_days)
if provider == "brave":
return await self._search_brave(provider_config, query, limit, domains)
if provider == "serpapi":
return await self._search_serpapi(provider_config, query, limit)
if provider == "exa":
return await self._search_exa(provider_config, query, limit, domains)
if provider == "firecrawl":
return await self._search_firecrawl(provider_config, query, limit)
if provider == "searxng":
return await self._search_searxng(provider_config, query, limit, domains)
raise WebSearchConfigurationError(f"Unsupported web search provider: {provider}")
context = {
"provider": provider,
"query_hash": hashlib.sha256(query.encode("utf-8")).hexdigest(),
"query_length": len(query),
"max_results": limit,
"domain_count": len(domains or []),
"freshness_days": freshness_days,
}
await emit_business_log(
logger,
event="ai_tool.web_search.start",
message="WebSearch request started",
category="ai_tool",
service="ai_tool",
module=__name__,
context=context,
)
try:
if provider == "tavily":
results = await self._search_tavily(provider_config, query, limit, domains, freshness_days)
elif provider == "brave":
results = await self._search_brave(provider_config, query, limit, domains)
elif provider == "serpapi":
results = await self._search_serpapi(provider_config, query, limit)
elif provider == "exa":
results = await self._search_exa(provider_config, query, limit, domains)
elif provider == "firecrawl":
results = await self._search_firecrawl(provider_config, query, limit)
elif provider == "searxng":
results = await self._search_searxng(provider_config, query, limit, domains)
else:
raise WebSearchConfigurationError(f"Unsupported web search provider: {provider}")
event = "ai_tool.web_search.success" if results else "ai_tool.web_search.empty"
await emit_business_log(
logger,
event=event,
message="WebSearch request completed" if results else "WebSearch returned no results",
category="ai_tool",
level="info" if results else "warning",
service="ai_tool",
module=__name__,
context={
**context,
"status": "success" if results else "empty",
"result_count": len(results),
"duration_ms": int((perf_counter() - started_at) * 1000),
},
)
return results
except Exception as exc:
await emit_business_log(
logger,
event="ai_tool.web_search.failed",
message="WebSearch request failed",
category="ai_tool",
level="error",
service="ai_tool",
module=__name__,
context=exception_context(exc, {**context, "status": "failed", "duration_ms": int((perf_counter() - started_at) * 1000)}),
)
raise
async def test_connection(self) -> list[SearchEvidence]:
return await self.search("Planet WebSearch connectivity test", max_results=1)
@@ -388,4 +474,3 @@ def _float_or_none(value: Any) -> float | None:
return float(value)
except (TypeError, ValueError):
return None

View File

@@ -0,0 +1,117 @@
from __future__ import annotations
import asyncio
from collections.abc import Mapping
from typing import Any
from app.core.logging import PlanetLoggerAdapter, sanitize_log_value
from app.core.request_context import get_request_id
from app.services.persistent_logs import record_system_log
LEVEL_METHODS = {
"debug": "debug_event",
"info": "info_event",
"warning": "warning_event",
"error": "error_event",
}
def normalize_business_level(level: str | None) -> str:
normalized = str(level or "info").strip().lower()
if normalized in {"warn", "warning"}:
return "warning"
if normalized in {"err", "error", "critical", "fatal"}:
return "error"
if normalized == "debug":
return "debug"
return "info"
def build_business_context(
context: Mapping[str, Any] | None = None,
**fields: Any,
) -> dict[str, Any]:
payload = dict(context or {})
for key, value in fields.items():
if value is not None:
payload[key] = value
return sanitize_log_value(payload)
async def emit_business_log(
logger: PlanetLoggerAdapter,
*,
event: str,
message: str,
category: str,
level: str = "info",
source: str = "backend",
service: str | None = None,
module: str | None = None,
request_id: str | None = None,
user_id: int | None = None,
context: Mapping[str, Any] | None = None,
) -> None:
normalized_level = normalize_business_level(level)
safe_context = build_business_context(context)
log_method = getattr(logger, LEVEL_METHODS[normalized_level])
log_method(message, event=event, context=safe_context)
await record_system_log(
source=source,
level=normalized_level,
message=message,
service=service,
module=module,
event=event,
request_id=request_id or get_request_id(),
user_id=user_id,
category=category,
context=safe_context,
)
def emit_business_log_background(
logger: PlanetLoggerAdapter,
*,
event: str,
message: str,
category: str,
level: str = "info",
source: str = "backend",
service: str | None = None,
module: str | None = None,
request_id: str | None = None,
user_id: int | None = None,
context: Mapping[str, Any] | None = None,
) -> None:
normalized_level = normalize_business_level(level)
safe_context = build_business_context(context)
log_method = getattr(logger, LEVEL_METHODS[normalized_level])
log_method(message, event=event, context=safe_context)
try:
loop = asyncio.get_running_loop()
except RuntimeError:
return
loop.create_task(
record_system_log(
source=source,
level=normalized_level,
message=message,
service=service,
module=module,
event=event,
request_id=request_id or get_request_id(),
user_id=user_id,
category=category,
context=safe_context,
)
)
def exception_context(exc: BaseException, context: Mapping[str, Any] | None = None) -> dict[str, Any]:
return build_business_context(
context,
error_type=type(exc).__name__,
error=str(exc),
)

View File

@@ -4,42 +4,22 @@ import asyncio
from abc import ABC, abstractmethod
from typing import Dict, List, Any, Optional
from datetime import UTC, datetime
from time import perf_counter
from urllib.parse import urlparse
import httpx
from sqlalchemy import select, text
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.collected_data_fields import build_dynamic_metadata, get_record_field
from app.core.countries import normalize_country
from app.core.logging import get_logger
from app.core.time import to_iso8601_utc
from app.core.websocket.broadcaster import broadcaster
from app.services.earth_layer_cache import invalidate_earth_layer_cache_for_source
EARTH_UPDATE_LAYER_HINTS: dict[str, list[str]] = {
"ris_live_bgp": ["bgp"],
"bgpstream_bgp": ["bgp"],
"top500_supercomputers": ["computeCenters"],
"epoch_ai_gpu": ["computeCenters"],
"huggingface_models": ["computeCenters"],
"huggingface_datasets": ["computeCenters"],
"huggingface_spaces": ["computeCenters"],
"telegeography_cables": ["cables"],
"telegeography_landing_points": ["cables"],
"telegeography_cable_systems": ["cables"],
"arcgis_cables": ["cables"],
"fao_landing_points": ["cables"],
"arcgis_landing_points": ["cables"],
"arcgis_cable_landing_relations": ["cables"],
"spacetrack_tle": ["satellites"],
"celestrak_tle": ["satellites"],
"barentswatch_vessels": ["vessels"],
"aisstream_vessels": ["vessels"],
"news_live_streams": ["media"],
"media_news_archive": ["news"],
}
from app.services.business_logs import emit_business_log, exception_context
from app.services.earth_layer_adapters import get_earth_update_layers_for_source
def get_earth_update_layers_for_source(source: str) -> list[str]:
return EARTH_UPDATE_LAYER_HINTS.get(source, [])
logger = get_logger(__name__, service="collector")
class BaseCollector(ABC):
@@ -58,6 +38,7 @@ class BaseCollector(ABC):
self._datasource_id = 1
self._resolved_url: Optional[str] = None
self._last_broadcast_progress: Optional[int] = None
self._last_save_summary: dict[str, int] = {}
async def resolve_url(self, db: AsyncSession) -> None:
from app.core.data_sources import get_data_sources_config
@@ -96,29 +77,6 @@ class BaseCollector(ABC):
)
self._last_broadcast_progress = rounded_progress
async def _publish_earth_update(
self,
*,
action: str,
records_processed: int,
task_id: int | None = None,
) -> None:
layers = get_earth_update_layers_for_source(self.name)
if not layers:
return
await broadcaster.broadcast_earth_update(
{
"action": action,
"source": self.name,
"data_type": self.data_type,
"layers": layers,
"datasource_id": getattr(self, "_datasource_id", None),
"task_id": task_id,
"records_processed": records_processed,
"timestamp": to_iso8601_utc(datetime.now(UTC)),
}
)
async def update_progress(self, records_processed: int, *, commit: bool = False, force: bool = False):
"""Update task progress - call this during data processing"""
if self._current_task and self._db_session:
@@ -322,19 +280,39 @@ class BaseCollector(ABC):
from app.models.data_snapshot import DataSnapshot
start_time = datetime.now(UTC)
started_at = perf_counter()
datasource_id = getattr(self, "_datasource_id", 1)
snapshot_id: Optional[int] = None
if not collector_registry.is_active(self.name):
await self._log_collection_event(
"collector.run.skipped_disabled",
"Collector skipped because it is disabled",
level="info",
context={"status": "skipped", "reason": "disabled"},
)
return {"status": "skipped", "reason": "Collector is disabled"}
task = CollectionTask(
datasource_id=datasource_id,
status="running",
phase="queued",
started_at=start_time,
)
db.add(task)
task = self._current_task if isinstance(self._current_task, CollectionTask) else None
if task is None:
task = CollectionTask(
datasource_id=datasource_id,
source=self.name,
task_type="collect",
status="running",
phase="queued",
started_at=start_time,
)
db.add(task)
else:
task.datasource_id = datasource_id
task.source = task.source or self.name
task.task_type = task.task_type or "collect"
task.status = "running"
task.phase = "queued"
task.started_at = task.started_at or start_time
task.completed_at = None
task.error_message = None
await db.commit()
task_id = task.id
@@ -344,23 +322,76 @@ class BaseCollector(ABC):
await self.resolve_url(db)
await self._publish_task_update(force=True)
await self._log_collection_event(
"collector.run.started",
"Collector run started",
context={"status": "running", "task_id": task_id},
)
try:
phase_started_at = perf_counter()
await self.set_phase("fetching", message="正在拉取原始数据")
await self._log_collection_event(
"collector.phase.fetching.start",
"Collector fetch phase started",
context={"task_id": task_id, "snapshot_id": snapshot_id},
)
raw_data = await self.fetch()
task.total_records = len(raw_data)
await db.commit()
await self._publish_task_update(force=True)
await self._log_collection_event(
"collector.phase.fetching.success",
"Collector fetch phase completed",
context={
"task_id": task_id,
"raw_count": len(raw_data),
"duration_ms": self._duration_ms(phase_started_at),
},
)
if self.fail_on_empty and not raw_data:
raise RuntimeError(f"Collector {self.name} returned no data")
phase_started_at = perf_counter()
await self.set_phase("transforming", message="正在转换采集数据")
await self._log_collection_event(
"collector.phase.transforming.start",
"Collector transform phase started",
context={"task_id": task_id, "raw_count": len(raw_data)},
)
data = self.transform(raw_data)
await self._log_collection_event(
"collector.phase.transforming.success",
"Collector transform phase completed",
context={
"task_id": task_id,
"raw_count": len(raw_data),
"transformed_count": len(data),
"duration_ms": self._duration_ms(phase_started_at),
},
)
snapshot_id = await self._create_snapshot(db, task_id, data, start_time)
phase_started_at = perf_counter()
await self.set_phase("saving", message="正在保存采集数据")
await self._log_collection_event(
"collector.phase.saving.start",
"Collector save phase started",
context={"task_id": task_id, "snapshot_id": snapshot_id, "transformed_count": len(data)},
)
records_count = await self._save_data(db, data, task_id=task_id, snapshot_id=snapshot_id)
await self._log_collection_event(
"collector.phase.saving.success",
"Collector save phase completed",
context={
"task_id": task_id,
"snapshot_id": snapshot_id,
"saved_count": records_count,
**self._last_save_summary,
"duration_ms": self._duration_ms(phase_started_at),
},
)
task.status = "success"
task.phase = "completed"
@@ -374,10 +405,19 @@ class BaseCollector(ABC):
task.completed_at = datetime.now(UTC)
await db.commit()
await self._publish_task_update(force=True)
await self._publish_earth_update(
action="collector_completed",
records_processed=records_count,
task_id=task_id,
await self._log_collection_event(
"collector.run.completed",
"Collector run completed",
context={
"status": "success",
"task_id": task_id,
"snapshot_id": snapshot_id,
"raw_count": len(raw_data),
"transformed_count": len(data),
"saved_count": records_count,
**self._last_save_summary,
"duration_ms": self._duration_ms(started_at),
},
)
return {
@@ -402,6 +442,17 @@ class BaseCollector(ABC):
)
await db.commit()
await self._publish_task_update(force=True)
await self._log_collection_event(
"collector.run.cancelled",
"Collector run cancelled",
level="warning",
context={
"status": "cancelled",
"task_id": task_id,
"snapshot_id": snapshot_id,
"duration_ms": self._duration_ms(started_at),
},
)
raise
except Exception as e:
await db.rollback()
@@ -418,6 +469,20 @@ class BaseCollector(ABC):
snapshot.summary = {"error": str(e)}
await db.commit()
await self._publish_task_update(force=True)
await self._log_collection_event(
"collector.run.failed",
"Collector run failed",
level="error",
context=exception_context(
e,
{
"status": "failed",
"task_id": task_id,
"snapshot_id": snapshot_id,
"duration_ms": self._duration_ms(started_at),
},
),
)
return {
"status": "failed",
@@ -438,6 +503,7 @@ class BaseCollector(ABC):
from app.models.data_snapshot import DataSnapshot
if not data:
self._last_save_summary = {"created": 0, "updated": 0, "unchanged": 0, "deleted": 0}
if snapshot_id is not None:
snapshot = await db.get(DataSnapshot, snapshot_id)
if snapshot:
@@ -584,12 +650,51 @@ class BaseCollector(ABC):
"unchanged": unchanged_count,
"deleted": len(deleted_keys),
}
self._last_save_summary = {
"created": created_count,
"updated": updated_count,
"unchanged": unchanged_count,
"deleted": len(deleted_keys),
}
else:
self._last_save_summary = {
"created": created_count,
"updated": updated_count,
"unchanged": unchanged_count,
"deleted": 0,
}
await db.commit()
invalidate_earth_layer_cache_for_source(self.name)
await self.update_progress(len(data), force=True)
return records_added
@staticmethod
def _duration_ms(started_at: float) -> int:
return int((perf_counter() - started_at) * 1000)
async def _log_collection_event(
self,
event: str,
message: str,
*,
level: str = "info",
context: Dict[str, Any] | None = None,
) -> None:
await emit_business_log(
logger,
event=event,
message=message,
category="collector",
level=level,
service="collector",
module=__name__,
context={
"collector_name": self.name,
"datasource_id": getattr(self, "_datasource_id", None),
**(context or {}),
},
)
async def save(self, db: AsyncSession, data: List[Dict[str, Any]]) -> int:
"""Save data to database (legacy method, use _save_data instead)"""
return await self._save_data(db, data)
@@ -602,10 +707,65 @@ class HTTPCollector(BaseCollector):
headers: Dict[str, str] = {}
async def fetch(self) -> List[Dict[str, Any]]:
started_at = perf_counter()
request_host = urlparse(self.base_url).netloc
await emit_business_log(
logger,
event="collector.http.fetch.start",
message="Collector HTTP request started",
category="collector",
service="collector",
module=__name__,
context={
"collector_name": self.name,
"datasource_id": getattr(self, "_datasource_id", None),
"url_host": request_host,
},
)
async with httpx.AsyncClient(timeout=60.0) as client:
response = await client.get(self.base_url, headers=self.headers)
response.raise_for_status()
return self.parse_response(response.json())
try:
response = await client.get(self.base_url, headers=self.headers)
response.raise_for_status()
payload = response.json()
parsed = self.parse_response(payload)
await emit_business_log(
logger,
event="collector.http.fetch.success",
message="Collector HTTP request completed",
category="collector",
service="collector",
module=__name__,
context={
"collector_name": self.name,
"datasource_id": getattr(self, "_datasource_id", None),
"url_host": request_host,
"status_code": response.status_code,
"response_bytes": len(response.content or b""),
"parsed_count": len(parsed),
"duration_ms": BaseCollector._duration_ms(started_at),
},
)
return parsed
except Exception as exc:
await emit_business_log(
logger,
event="collector.http.fetch.failed",
message="Collector HTTP request failed",
category="collector",
level="error",
service="collector",
module=__name__,
context=exception_context(
exc,
{
"collector_name": self.name,
"datasource_id": getattr(self, "_datasource_id", None),
"url_host": request_host,
"duration_ms": BaseCollector._duration_ms(started_at),
},
),
)
raise
@abstractmethod
def parse_response(self, response: Dict[str, Any]) -> List[Dict[str, Any]]:

View File

@@ -1,15 +1,39 @@
"""CelesTrak TLE Collector
"""CelesTrak TLE Collector.
Collects satellite TLE (Two-Line Element) data from CelesTrak.org.
Free, no authentication required.
Collects the full active satellite GP element set from CelesTrak.
"""
import asyncio
import json
from typing import Dict, Any, List
from pathlib import Path
from time import perf_counter
from typing import Any, Dict, List
from urllib.parse import urlencode, urlparse
import httpx
from app.core.logging import get_logger
from app.core.satellite_tle import build_tle_lines_from_elements
from app.services.business_logs import emit_business_log, exception_context
from app.services.collectors.base import BaseCollector
from app.services.collectors.downloads import DownloadHTTPStatusError, ResumableFileDownloader
logger = get_logger(__name__, service="collector")
ACTIVE_GROUP = "active"
FALLBACK_GROUPS = (
"starlink",
"gps-ops",
"galileo",
"glonass",
"beidou",
"leo",
"geo",
"iridium-next",
)
FETCH_RETRY_ATTEMPTS = 3
FETCH_RETRY_BASE_DELAY_SECONDS = 0.8
CELESTRAK_NOT_UPDATED_MARKER = "GP data has not updated since your last successful"
class CelesTrakTLECollector(BaseCollector):
@@ -18,55 +42,359 @@ class CelesTrakTLECollector(BaseCollector):
module = "L3"
frequency_hours = 24
data_type = "satellite_tle"
_downloader = ResumableFileDownloader(
cache_namespace="celestrak",
default_accept="application/json",
)
@property
def base_url(self) -> str:
return self._resolved_url or ""
def _active_url(self) -> str:
return self._group_url(ACTIVE_GROUP)
def _group_url(self, group: str) -> str:
if not self.base_url:
raise RuntimeError("CelesTrak base URL is not configured")
return f"{self.base_url}?{urlencode({'GROUP': group, 'FORMAT': 'json'})}"
async def fetch(self) -> List[Dict[str, Any]]:
satellite_groups = [
"starlink",
"gps-ops",
"galileo",
"glonass",
"beidou",
"leo",
"geo",
"iridium-next",
]
url = self._active_url()
last_error: Exception | None = None
all_satellites = []
async with httpx.AsyncClient(timeout=120.0) as client:
for group in satellite_groups:
async with httpx.AsyncClient(timeout=180.0, follow_redirects=True) as client:
for attempt in range(1, FETCH_RETRY_ATTEMPTS + 1):
started_at = perf_counter()
try:
url = f"{self.base_url}?GROUP={group}&FORMAT=json"
response = await client.get(url)
await emit_business_log(
logger,
event="collector.celestrak.download.start",
message="CelesTrak active satellite download started",
category="collector",
service="collector",
module=__name__,
context={
"collector_name": self.name,
"datasource_id": getattr(self, "_datasource_id", None),
"group": ACTIVE_GROUP,
"attempt": attempt,
"url_host": urlparse(url).netloc,
},
)
body_path = await self._downloader.download_file(
client,
url,
extension=".json",
accept="application/json",
progress_callback=self._report_download_progress,
validate_existing=self._validate_json_file,
)
data = await self._load_downloaded_payload(body_path, url)
await emit_business_log(
logger,
event="collector.celestrak.download.success",
message="CelesTrak active satellite download completed",
category="collector",
service="collector",
module=__name__,
context={
"collector_name": self.name,
"datasource_id": getattr(self, "_datasource_id", None),
"group": ACTIVE_GROUP,
"attempt": attempt,
"record_count": len(data),
"duration_ms": self._duration_ms(started_at),
},
)
return data
except DownloadHTTPStatusError as exc:
if self._is_not_updated_response(exc):
cached_path = self._downloader.get_cached_file(
url,
".json",
validate_existing=self._validate_json_file,
)
if cached_path is not None:
data = await self._load_downloaded_payload(cached_path, url)
await emit_business_log(
logger,
event="collector.celestrak.download.cached_not_updated",
message="CelesTrak active satellite data has not changed; using cached download",
category="collector",
level="warning",
service="collector",
module=__name__,
context={
"collector_name": self.name,
"datasource_id": getattr(self, "_datasource_id", None),
"group": ACTIVE_GROUP,
"attempt": attempt,
"record_count": len(data),
"duration_ms": self._duration_ms(started_at),
},
)
return data
await emit_business_log(
logger,
event="collector.celestrak.download.not_updated_no_cache",
message="CelesTrak active satellite data has not changed; trying fallback groups",
category="collector",
level="warning",
service="collector",
module=__name__,
context=exception_context(
exc,
{
"collector_name": self.name,
"datasource_id": getattr(self, "_datasource_id", None),
"group": ACTIVE_GROUP,
"attempt": attempt,
"duration_ms": self._duration_ms(started_at),
},
),
)
return await self._fetch_fallback_groups(client, active_error=exc)
raise
except Exception as exc:
last_error = exc
is_final_attempt = attempt >= FETCH_RETRY_ATTEMPTS
await emit_business_log(
logger,
event=(
"collector.celestrak.download.failed"
if is_final_attempt
else "collector.celestrak.download.retry"
),
message=(
"CelesTrak active satellite download failed"
if is_final_attempt
else "CelesTrak active satellite download will retry"
),
category="collector",
level="error" if is_final_attempt else "warning",
service="collector",
module=__name__,
context=exception_context(
exc,
{
"collector_name": self.name,
"datasource_id": getattr(self, "_datasource_id", None),
"group": ACTIVE_GROUP,
"attempt": attempt,
"duration_ms": self._duration_ms(started_at),
},
),
)
if not is_final_attempt:
await asyncio.sleep(FETCH_RETRY_BASE_DELAY_SECONDS * attempt)
if response.status_code == 200:
data = response.json()
if isinstance(data, list):
for item in data:
if isinstance(item, dict):
item["_celestrak_group"] = group
all_satellites.extend(data)
print(f"CelesTrak: Fetched {len(data)} satellites from group '{group}'")
except Exception as e:
print(f"CelesTrak: Error fetching group '{group}': {e}")
raise RuntimeError(f"CelesTrak active satellite download failed after retries: {last_error}")
if not all_satellites:
return self._get_sample_data()
async def _fetch_fallback_groups(
self,
client: httpx.AsyncClient,
*,
active_error: DownloadHTTPStatusError,
) -> List[Dict[str, Any]]:
started_at = perf_counter()
records_by_norad: dict[str, Dict[str, Any]] = {}
group_counts: dict[str, int] = {}
print(f"CelesTrak: Total satellites fetched: {len(all_satellites)}")
await emit_business_log(
logger,
event="collector.celestrak.fallback_groups.start",
message="CelesTrak fallback group download started",
category="collector",
level="warning",
service="collector",
module=__name__,
context={
"collector_name": self.name,
"datasource_id": getattr(self, "_datasource_id", None),
"groups": list(FALLBACK_GROUPS),
"reason": "active_not_updated_without_cache",
},
)
# Return raw data - base.run() will call transform()
return all_satellites
try:
for group in FALLBACK_GROUPS:
group_url = self._group_url(group)
try:
body_path = await self._downloader.download_file(
client,
group_url,
extension=".json",
accept="application/json",
validate_existing=self._validate_json_file,
)
except DownloadHTTPStatusError as exc:
if not self._is_not_updated_response(exc):
raise RuntimeError(f"CelesTrak fallback group '{group}' download failed: {exc}") from exc
cached_path = self._downloader.get_cached_file(
group_url,
".json",
validate_existing=self._validate_json_file,
)
if cached_path is None:
raise RuntimeError(
f"CelesTrak fallback group '{group}' has not updated and no local cached copy is available"
) from exc
body_path = cached_path
group_records = await self._load_downloaded_payload(
body_path,
group_url,
query_group=group,
constellation_group=group,
)
group_counts[group] = len(group_records)
for item in group_records:
norad_cat_id = item.get("NORAD_CAT_ID")
if norad_cat_id is None:
continue
records_by_norad.setdefault(str(norad_cat_id), item)
except Exception as exc:
await emit_business_log(
logger,
event="collector.celestrak.fallback_groups.failed",
message="CelesTrak fallback group download failed",
category="collector",
level="error",
service="collector",
module=__name__,
context=exception_context(
exc,
{
"collector_name": self.name,
"datasource_id": getattr(self, "_datasource_id", None),
"groups": list(FALLBACK_GROUPS),
"completed_groups": list(group_counts),
"duration_ms": self._duration_ms(started_at),
},
),
)
raise RuntimeError(
"CelesTrak active data has not updated since this network's last successful download, "
"no active cache is available, and fallback group mode failed. Wait until CelesTrak "
"publishes the next GP update, restore the Planet download cache, or use Space-Track."
) from active_error
records = list(records_by_norad.values())
if not records:
raise RuntimeError("CelesTrak fallback group mode produced no satellite records")
await emit_business_log(
logger,
event="collector.celestrak.fallback_groups.success",
message="CelesTrak fallback group download completed",
category="collector",
level="warning",
service="collector",
module=__name__,
context={
"collector_name": self.name,
"datasource_id": getattr(self, "_datasource_id", None),
"groups": list(FALLBACK_GROUPS),
"group_counts": group_counts,
"record_count": len(records),
"duration_ms": self._duration_ms(started_at),
},
)
return records
async def _load_downloaded_payload(
self,
body_path: Path,
url: str,
*,
query_group: str = ACTIVE_GROUP,
constellation_group: str | None = None,
) -> List[Dict[str, Any]]:
try:
data = self._load_active_payload(body_path)
except RuntimeError as exc:
await self._log_parse_failure(exc)
raise
for item in data:
item["_celestrak_query_group"] = query_group
item["_celestrak_source_url"] = url
if constellation_group:
item["_celestrak_group"] = constellation_group
return data
@staticmethod
def _is_not_updated_response(exc: DownloadHTTPStatusError) -> bool:
return exc.status_code == 403 and CELESTRAK_NOT_UPDATED_MARKER in exc.body
@staticmethod
def _duration_ms(started_at: float) -> int:
return int((perf_counter() - started_at) * 1000)
async def _report_download_progress(self, downloaded: int, total: int | None) -> None:
if total and total > 0:
await self.update_phase_progress(
current=min(downloaded, total),
total=total,
unit="bytes",
message=f"正在下载 CelesTrak active 卫星数据 {downloaded}/{total} bytes",
commit=True,
)
@staticmethod
def _validate_json_file(path: Path) -> bool:
try:
data = json.loads(path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError, UnicodeDecodeError):
return False
return isinstance(data, list)
async def _log_parse_failure(self, exc: Exception) -> None:
await emit_business_log(
logger,
event="collector.celestrak.parse.failed",
message="CelesTrak active satellite JSON parsing failed",
category="collector",
level="error",
service="collector",
module=__name__,
context=exception_context(
exc,
{
"collector_name": self.name,
"datasource_id": getattr(self, "_datasource_id", None),
"group": ACTIVE_GROUP,
},
),
)
def _load_active_payload(self, path: Path) -> List[Dict[str, Any]]:
try:
raw = json.loads(path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError, UnicodeDecodeError) as exc:
raise RuntimeError(f"CelesTrak active payload is not valid JSON: {exc}") from exc
if not isinstance(raw, list):
raise RuntimeError("CelesTrak active payload is not a JSON array")
records: List[Dict[str, Any]] = []
invalid_count = 0
for item in raw:
if isinstance(item, dict) and item.get("NORAD_CAT_ID") is not None:
records.append(item)
else:
invalid_count += 1
if invalid_count:
raise RuntimeError(f"CelesTrak active payload contains {invalid_count} invalid record(s)")
if not records:
raise RuntimeError("CelesTrak active payload contains no satellite records")
return records
def transform(self, raw_data: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
transformed = []
for item in raw_data:
norad_cat_id = item.get("NORAD_CAT_ID")
tle_line1, tle_line2 = build_tle_lines_from_elements(
norad_cat_id=item.get("NORAD_CAT_ID"),
norad_cat_id=norad_cat_id,
epoch=item.get("EPOCH"),
inclination=item.get("INCLINATION"),
raan=item.get("RA_OF_ASC_NODE"),
@@ -75,14 +403,18 @@ class CelesTrakTLECollector(BaseCollector):
mean_anomaly=item.get("MEAN_ANOMALY"),
mean_motion=item.get("MEAN_MOTION"),
)
constellation_group = self._infer_constellation_group(item)
transformed.append(
{
"source_id": str(norad_cat_id),
"name": item.get("OBJECT_NAME", "Unknown"),
"reference_date": item.get("EPOCH", ""),
"metadata": {
"constellation_group": item.get("_celestrak_group"),
"norad_cat_id": item.get("NORAD_CAT_ID"),
"constellation_group": constellation_group,
"celestrak_query_group": item.get("_celestrak_query_group") or ACTIVE_GROUP,
"celestrak_source_url": item.get("_celestrak_source_url"),
"norad_cat_id": norad_cat_id,
"international_designator": item.get("OBJECT_ID"),
"epoch": item.get("EPOCH"),
"mean_motion": item.get("MEAN_MOTION"),
@@ -105,6 +437,19 @@ class CelesTrakTLECollector(BaseCollector):
)
return transformed
@staticmethod
def _infer_constellation_group(item: Dict[str, Any]) -> str | None:
explicit_group = str(item.get("_celestrak_group") or "").strip().lower()
if explicit_group and explicit_group != ACTIVE_GROUP:
return explicit_group
name = str(item.get("OBJECT_NAME") or "").strip().upper()
if name.startswith("STARLINK"):
return "starlink"
if name.startswith("IRIDIUM"):
return "iridium-next"
return None
def _get_sample_data(self) -> List[Dict[str, Any]]:
return [
{

View File

@@ -4,7 +4,7 @@ from __future__ import annotations
import hashlib
import json
import tempfile
import os
import time
from datetime import UTC, datetime
from pathlib import Path
@@ -17,6 +17,31 @@ ProgressCallback = Callable[[int, int | None], Awaitable[None]]
ValidateCallback = Callable[[Path], bool]
class DownloadHTTPStatusError(RuntimeError):
"""HTTP status error that keeps the upstream response body for caller-specific handling."""
def __init__(self, *, url: str, status_code: int, body: str) -> None:
self.url = url
self.status_code = status_code
self.body = body
preview = body.strip().replace("\r", " ").replace("\n", " ")[:240]
suffix = f": {preview}" if preview else ""
super().__init__(f"HTTP {status_code} while downloading {url}{suffix}")
def default_download_cache_root() -> Path:
configured = os.getenv("PLANET_DOWNLOAD_CACHE_DIR")
if configured:
return Path(configured).expanduser()
planet_cache = os.getenv("PLANET_CACHE_DIR")
if planet_cache:
return Path(planet_cache).expanduser() / "downloads"
xdg_cache = os.getenv("XDG_CACHE_HOME")
if xdg_cache:
return Path(xdg_cache).expanduser() / "planet" / "downloads"
return Path.home() / ".cache" / "planet" / "downloads"
class ResumableFileDownloader:
"""Download files with cache validators and byte-range resume support."""
@@ -26,8 +51,9 @@ class ResumableFileDownloader:
cache_namespace: str,
user_agent: str = "Planet-Intelligence-System/1.0 (Python/collector)",
default_accept: str = "*/*",
cache_root: Path | None = None,
) -> None:
self._cache_dir = Path(tempfile.gettempdir()) / "planet-download-cache" / cache_namespace
self._cache_dir = (cache_root or default_download_cache_root()) / cache_namespace
self._user_agent = user_agent
self._default_accept = default_accept
@@ -43,6 +69,25 @@ class ResumableFileDownloader:
meta_path = self._cache_dir / f"{key}.meta.json"
return final_path, part_path, meta_path
def cached_file_path(self, url: str, extension: str) -> Path:
final_path, _, _ = self._cache_paths(url, extension)
return final_path
def get_cached_file(
self,
url: str,
extension: str,
*,
validate_existing: ValidateCallback | None = None,
) -> Path | None:
final_path = self.cached_file_path(url, extension)
if not final_path.exists():
return None
if validate_existing and not validate_existing(final_path):
final_path.unlink(missing_ok=True)
return None
return final_path
@staticmethod
def _load_meta(meta_path: Path) -> dict[str, Any]:
if not meta_path.exists():
@@ -140,7 +185,9 @@ class ResumableFileDownloader:
if progress_callback and expected_size and expected_size > 0:
await progress_callback(expected_size, expected_size)
return final_path
response.raise_for_status()
if response.status_code >= 400:
body = (await response.aread()).decode("utf-8", errors="replace")
raise DownloadHTTPStatusError(url=url, status_code=response.status_code, body=body)
if response.status_code == 206 and resume_from > 0:
mode = "ab"

View File

@@ -163,32 +163,64 @@ class TeleGeographyLandingPointCollector(BaseCollector):
data_type = "landing_point"
async def fetch(self) -> List[Dict[str, Any]]:
"""Fetch landing point data from GitHub mirror"""
url = self._resolved_url or ""
"""Fetch landing point data, falling back when the old mirror disappears."""
config = get_data_sources_config()
sources = [
self._resolved_url or "",
str(config.get_yaml_value("telegeography.landing_point_url") or ""),
str(config.get_yaml_value("arcgis.landing_point_url") or ""),
]
async with httpx.AsyncClient(timeout=60.0) as client:
response = await client.get(url)
response.raise_for_status()
return self.parse_response(response.json())
last_error: Exception | None = None
async with httpx.AsyncClient(timeout=60.0, follow_redirects=True) as client:
for url in dict.fromkeys(source for source in sources if source):
try:
params = (
{"where": "1=1", "outFields": "*", "returnGeometry": "true", "f": "geojson"}
if "FeatureServer" in url or url.endswith("/query")
else None
)
response = await client.get(url, params=params)
response.raise_for_status()
records = self.parse_response(response.json())
if records:
return records
except Exception as exc:
last_error = exc
continue
def parse_response(self, data: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
if last_error:
raise last_error
return self._get_sample_data()
def parse_response(self, data: List[Dict[str, Any]] | Dict[str, Any]) -> List[Dict[str, Any]]:
"""Parse landing point data"""
result = []
items = data.get("features", []) if isinstance(data, dict) else data
for item in data:
for item in items:
props = item.get("properties", {}) if isinstance(item, dict) else {}
geometry = item.get("geometry", {}) if isinstance(item, dict) else {}
source = props or item
coords = geometry.get("coordinates", []) if isinstance(geometry, dict) else []
longitude = coords[0] if len(coords) > 0 else source.get("longitude")
latitude = coords[1] if len(coords) > 1 else source.get("latitude")
source_id = source.get("id") or source.get("OBJECTID") or source.get("city_id") or ""
try:
entry = {
"source_id": f"telegeo_lp_{item.get('id', '')}",
"name": item.get("name", "Unknown"),
"country": item.get("country", "Unknown"),
"city": item.get("city", item.get("name", "")),
"latitude": str(item.get("latitude", "")),
"longitude": str(item.get("longitude", "")),
"source_id": f"telegeo_lp_{source_id}",
"name": source.get("name", source.get("Name", "Unknown")),
"country": source.get("country", "Unknown"),
"city": source.get("city", source.get("Name", source.get("name", ""))),
"latitude": str(latitude or ""),
"longitude": str(longitude or ""),
"value": "",
"unit": "",
"metadata": {
"cable_count": len(item.get("cables", [])),
"url": item.get("url"),
"cable_count": len(source.get("cables", [])),
"url": source.get("url"),
"objectid": source.get("OBJECTID"),
"city_id": source.get("city_id"),
},
"reference_date": datetime.now(UTC).strftime("%Y-%m-%d"),
}

View File

@@ -11,7 +11,7 @@ startup, so it must stay local and deterministic.
For the full design and the reason behind the abstraction (compute centers,
BGP collectors, BGP events, and future entities all share one pipeline),
see ``docs/plans/location-resolver-shared-pipeline-plan.md``.
see ``docs/technical/zh/location-pipeline-development.md``.
The ``ComputeCenterLocation`` dataclass and the public function signatures are
preserved verbatim so existing callers and tests do not need to change.

View File

@@ -0,0 +1,597 @@
"""Kafka-ready datasource job queue backed by PostgreSQL for v1."""
from __future__ import annotations
import asyncio
from datetime import UTC, datetime, timedelta
from uuid import uuid4
from typing import Any
from sqlalchemy import bindparam, select, text
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.cache import cache
from app.core.config import settings
from app.core.logging import get_logger
from app.core.time import to_iso8601_utc
from app.core.websocket.broadcaster import broadcaster
from app.db.session import async_session_factory
from app.models.collected_data import CollectedData
from app.models.data_snapshot import DataSnapshot
from app.models.datasource import DataSource
from app.models.task import CollectionTask
from app.services.collectors.registry import collector_registry
from app.services.datasource_connectivity import (
build_builtin_connectivity_checksum,
get_builtin_effective_candidate,
save_connectivity_success,
)
from app.services.earth_layer_adapters import (
clear_derived_datasource_data,
get_earth_refresh_strategy_for_change,
get_earth_update_layers_for_source,
)
from app.services.earth_layer_cache import invalidate_earth_layer_cache_for_source
from app.services.scheduler import sync_datasource_job
logger = get_logger(__name__)
JOB_TYPE_COLLECT = "collect"
JOB_TYPE_CLEAR_DATA = "clear_data"
JOB_TYPE_CLEAR_CACHE = "clear_cache"
JOB_TYPE_EARTH_REFRESH = "earth_refresh"
JOB_STATUS_QUEUED = "queued"
JOB_STATUS_RUNNING = "running"
JOB_STATUS_CANCELLING = "cancelling"
JOB_STATUS_SUCCESS = "success"
JOB_STATUS_FAILED = "failed"
JOB_STATUS_CANCELLED = "cancelled"
ACTIVE_JOB_STATUSES = (JOB_STATUS_QUEUED, JOB_STATUS_RUNNING, JOB_STATUS_CANCELLING)
TERMINAL_JOB_STATUSES = (JOB_STATUS_SUCCESS, JOB_STATUS_FAILED, JOB_STATUS_CANCELLED)
DATA_WRITE_JOB_TYPES = (JOB_TYPE_COLLECT, JOB_TYPE_CLEAR_DATA, JOB_TYPE_CLEAR_CACHE)
SOURCE_LOCK_JOB_STATUSES = (JOB_STATUS_RUNNING, JOB_STATUS_CANCELLING)
QUEUE_POLL_SECONDS = 0.35
JOB_STALE_LOCK_MINUTES = 90
DEFAULT_WORKER_CONCURRENCY = 2
RUNNING_DATA_JOB_TASKS: dict[int, asyncio.Task[Any]] = {}
def _utcnow() -> datetime:
return datetime.now(UTC)
def _job_worker_id() -> str:
return f"{settings.PROJECT_NAME}:data-job-worker:{uuid4().hex[:8]}"
def is_terminal_job_status(status: str | None) -> bool:
return status in TERMINAL_JOB_STATUSES
async def enqueue_datasource_job(
db: AsyncSession,
datasource: DataSource,
task_type: str,
*,
payload: dict[str, Any] | None = None,
rollback_policy: str = "keep_committed_batches",
dedupe_key: str | None = None,
) -> CollectionTask:
if dedupe_key:
existing = await _get_active_job_by_dedupe_key(db, dedupe_key)
if existing is not None:
return existing
task = CollectionTask(
datasource_id=datasource.id,
source=datasource.source,
task_type=task_type,
status=JOB_STATUS_QUEUED,
phase="queued",
phase_message="任务已进入队列",
payload=payload or {},
rollback_policy=rollback_policy,
dedupe_key=dedupe_key,
)
db.add(task)
await db.commit()
await db.refresh(task)
await _broadcast_task_update(task)
return task
async def enqueue_earth_refresh_job(
db: AsyncSession,
*,
source: str,
payload: dict[str, Any] | None = None,
) -> CollectionTask | None:
layers = list((payload or {}).get("layers") or get_earth_update_layers_for_source(source))
if not layers:
return None
datasource = await _get_or_create_virtual_datasource(db, source)
refresh_payload = {
"source": source,
"layers": layers,
"refresh_strategy": (payload or {}).get("refresh_strategy")
or get_earth_refresh_strategy_for_change((payload or {}).get("table"), source)
or "clear_then_reload",
**(payload or {}),
}
return await enqueue_datasource_job(
db,
datasource,
JOB_TYPE_EARTH_REFRESH,
payload=refresh_payload,
dedupe_key=f"earth_refresh:{source}",
)
async def enqueue_earth_refresh_from_update(payload: dict[str, Any]) -> None:
source = str(payload.get("source") or "").strip()
if not source:
return
async with async_session_factory() as db:
await enqueue_earth_refresh_job(db, source=source, payload=payload)
async def request_cancel_datasource_task(
db: AsyncSession,
task: CollectionTask,
*,
reason: str = "cancelled_by_operator",
) -> CollectionTask:
if is_terminal_job_status(task.status):
return task
running_task = RUNNING_DATA_JOB_TASKS.get(task.id)
if task.status == JOB_STATUS_QUEUED or (
running_task is None
and (
task.status == JOB_STATUS_CANCELLING
or (task.status == JOB_STATUS_RUNNING and task.task_type != JOB_TYPE_COLLECT)
)
):
return await _cancel_task_without_runner(db, task, reason=reason)
task.status = JOB_STATUS_CANCELLING
task.phase = JOB_STATUS_CANCELLING
task.phase_message = "正在停止任务"
task.requested_cancel_at = _utcnow()
task.cancel_reason = reason
await db.commit()
await db.refresh(task)
if running_task is not None and not running_task.done():
running_task.cancel()
await _broadcast_task_update(task)
return task
async def _cancel_task_without_runner(
db: AsyncSession,
task: CollectionTask,
*,
reason: str,
) -> CollectionTask:
if task.task_type == JOB_TYPE_COLLECT:
await db.execute(CollectedData.__table__.delete().where(CollectedData.task_id == task.id))
snapshot_result = await db.execute(select(DataSnapshot).where(DataSnapshot.task_id == task.id))
for snapshot in snapshot_result.scalars().all():
snapshot.status = JOB_STATUS_CANCELLED
snapshot.completed_at = _utcnow()
snapshot.error_message = "Cancelled after operator stop request; no active worker handle remained"
datasource = await db.get(DataSource, task.datasource_id)
if datasource is not None:
datasource.last_status = JOB_STATUS_CANCELLED
task.status = JOB_STATUS_CANCELLED
task.phase = JOB_STATUS_CANCELLED
task.phase_message = "任务已停止"
task.completed_at = _utcnow()
task.requested_cancel_at = task.requested_cancel_at or _utcnow()
task.cancel_reason = reason
await db.commit()
await db.refresh(task)
await _broadcast_task_update(task)
return task
async def get_active_datasource_job(
db: AsyncSession,
datasource_id: int,
*,
task_types: tuple[str, ...] = DATA_WRITE_JOB_TYPES,
) -> CollectionTask | None:
result = await db.execute(
select(CollectionTask)
.where(CollectionTask.datasource_id == datasource_id)
.where(CollectionTask.task_type.in_(task_types))
.where(CollectionTask.status.in_(ACTIVE_JOB_STATUSES))
.order_by(CollectionTask.created_at.desc().nullslast(), CollectionTask.id.desc())
.limit(1)
)
return result.scalar_one_or_none()
async def _get_active_job_by_dedupe_key(db: AsyncSession, dedupe_key: str) -> CollectionTask | None:
result = await db.execute(
select(CollectionTask)
.where(CollectionTask.dedupe_key == dedupe_key)
.where(CollectionTask.status.in_(ACTIVE_JOB_STATUSES))
.order_by(CollectionTask.created_at.desc().nullslast(), CollectionTask.id.desc())
.limit(1)
)
return result.scalar_one_or_none()
async def _get_or_create_virtual_datasource(db: AsyncSession, source: str) -> DataSource:
result = await db.execute(select(DataSource).where(DataSource.source == source))
datasource = result.scalar_one_or_none()
if datasource is not None:
return datasource
datasource = DataSource(
name=f"Earth refresh: {source}",
source=source,
module="SYS",
collector_class="EarthRefreshJob",
is_active=True,
)
db.add(datasource)
await db.commit()
await db.refresh(datasource)
return datasource
async def _broadcast_task_update(task: CollectionTask) -> None:
await broadcaster.broadcast_datasource_task_update(
{
"datasource_id": task.datasource_id,
"collector_name": task.source,
"task_id": task.id,
"task_type": task.task_type,
"status": task.status,
"phase": task.phase,
"phase_progress": task.phase_progress,
"phase_message": task.phase_message,
"phase_current": task.phase_current,
"phase_total": task.phase_total,
"phase_unit": task.phase_unit,
"progress": task.progress,
"records_processed": task.records_processed,
"total_records": task.total_records,
"started_at": to_iso8601_utc(task.started_at),
"completed_at": to_iso8601_utc(task.completed_at),
"requested_cancel_at": to_iso8601_utc(task.requested_cancel_at),
"error_message": task.error_message,
}
)
class DataJobWorker:
def __init__(self, *, concurrency: int = DEFAULT_WORKER_CONCURRENCY) -> None:
self.worker_id = _job_worker_id()
self.concurrency = max(1, concurrency)
self._task: asyncio.Task[None] | None = None
self._stop_event: asyncio.Event | None = None
self._running: set[asyncio.Task[Any]] = set()
def start(self) -> None:
if self._task and not self._task.done():
return
self._stop_event = asyncio.Event()
self._task = asyncio.create_task(self._run(), name="data-job-worker")
async def stop(self) -> None:
if self._stop_event:
self._stop_event.set()
for task in list(self._running):
task.cancel()
if self._task:
await asyncio.gather(self._task, return_exceptions=True)
if self._running:
await asyncio.gather(*self._running, return_exceptions=True)
async def _run(self) -> None:
assert self._stop_event is not None
await self._recover_stale_running_jobs()
while not self._stop_event.is_set():
self._running = {task for task in self._running if not task.done()}
if len(self._running) >= self.concurrency:
await asyncio.sleep(QUEUE_POLL_SECONDS)
continue
task_id = await self._claim_next_job()
if task_id is None:
await asyncio.sleep(QUEUE_POLL_SECONDS)
continue
runner = asyncio.create_task(self._run_claimed_job(task_id), name=f"data-job:{task_id}")
self._running.add(runner)
async def _recover_stale_running_jobs(self) -> None:
cutoff = _utcnow() - timedelta(minutes=JOB_STALE_LOCK_MINUTES)
async with async_session_factory() as db:
result = await db.execute(
select(CollectionTask)
.where(CollectionTask.status.in_((JOB_STATUS_RUNNING, JOB_STATUS_CANCELLING)))
.where(CollectionTask.locked_at.is_not(None))
.where(CollectionTask.locked_at < cutoff)
)
stale_jobs = list(result.scalars().all())
for job in stale_jobs:
job.status = JOB_STATUS_FAILED
job.phase = JOB_STATUS_FAILED
job.completed_at = _utcnow()
job.error_message = "Marked failed after stale data job lock timeout"
if stale_jobs:
await db.commit()
async def _claim_next_job(self) -> int | None:
async with async_session_factory() as db:
row = await db.execute(
text(
"""
SELECT queued.id
FROM collection_tasks AS queued
WHERE queued.status = :queued_status
AND NOT EXISTS (
SELECT 1
FROM collection_tasks AS active
WHERE active.source = queued.source
AND active.id <> queued.id
AND active.status IN :active_statuses
)
ORDER BY queued.created_at ASC NULLS FIRST, queued.id ASC
LIMIT 1
FOR UPDATE SKIP LOCKED
"""
).bindparams(bindparam("active_statuses", expanding=True)),
{
"queued_status": JOB_STATUS_QUEUED,
"active_statuses": SOURCE_LOCK_JOB_STATUSES,
},
)
task_id = row.scalar_one_or_none()
if task_id is None:
return None
task = await db.get(CollectionTask, int(task_id))
if task is None:
return None
task.status = JOB_STATUS_RUNNING
task.phase = "starting"
task.phase_message = "任务开始执行"
task.started_at = task.started_at or _utcnow()
task.worker_id = self.worker_id
task.locked_at = _utcnow()
await db.commit()
await _broadcast_task_update(task)
return int(task_id)
async def _run_claimed_job(self, task_id: int) -> None:
current_task = asyncio.current_task()
if current_task is not None:
RUNNING_DATA_JOB_TASKS[task_id] = current_task
try:
async with async_session_factory() as db:
task = await db.get(CollectionTask, task_id)
if task is None:
return
await self._execute_job(db, task)
except asyncio.CancelledError:
async with async_session_factory() as db:
task = await db.get(CollectionTask, task_id)
if task is not None and not is_terminal_job_status(task.status):
task.status = JOB_STATUS_CANCELLED
task.phase = JOB_STATUS_CANCELLED
task.phase_message = "任务已停止"
task.completed_at = _utcnow()
await db.commit()
await _broadcast_task_update(task)
raise
except Exception as exc:
logger.exception_event(
"Data job failed",
event="data_jobs.job_failed",
context={"task_id": task_id, "error": str(exc)},
)
async with async_session_factory() as db:
task = await db.get(CollectionTask, task_id)
if task is not None:
task.status = JOB_STATUS_FAILED
task.phase = JOB_STATUS_FAILED
task.phase_message = str(exc)
task.error_message = str(exc)
task.completed_at = _utcnow()
await db.commit()
await _broadcast_task_update(task)
finally:
RUNNING_DATA_JOB_TASKS.pop(task_id, None)
async def _execute_job(self, db: AsyncSession, task: CollectionTask) -> None:
if task.task_type == JOB_TYPE_COLLECT:
await _run_collect_job(db, task)
elif task.task_type == JOB_TYPE_CLEAR_DATA:
await _run_clear_data_job(db, task)
elif task.task_type == JOB_TYPE_CLEAR_CACHE:
await _run_clear_cache_job(db, task)
elif task.task_type == JOB_TYPE_EARTH_REFRESH:
await _run_earth_refresh_job(db, task)
else:
raise RuntimeError(f"Unsupported data job type: {task.task_type}")
async def _run_collect_job(db: AsyncSession, task: CollectionTask) -> None:
datasource = await db.get(DataSource, task.datasource_id)
if datasource is None:
raise RuntimeError("Data source not found")
collector = collector_registry.get(datasource.source)
if collector is None:
raise RuntimeError(f"Collector '{datasource.source}' not found")
if not datasource.is_active:
raise RuntimeError("Data source is disabled")
collector._datasource_id = datasource.id
collector._current_task = task
collector._db_session = db
result = await collector.run(db)
datasource.last_run_at = _utcnow()
datasource.last_status = result.get("status")
if datasource.last_status == JOB_STATUS_SUCCESS:
effective_candidate = await get_builtin_effective_candidate(db, datasource.source)
checksum, _credential_context = await build_builtin_connectivity_checksum(
datasource.source,
effective_candidate["endpoint"],
effective_candidate["auth_type"],
effective_candidate["headers"],
effective_candidate["config"],
db,
)
await save_connectivity_success(
db,
datasource.source,
checksum,
{"status_code": None},
connected_by="collection",
)
await db.commit()
await sync_datasource_job(datasource.id)
async def _run_clear_data_job(db: AsyncSession, task: CollectionTask) -> None:
source = str(task.source or (task.payload or {}).get("source") or "").strip()
if not source:
raise RuntimeError("Clear data job has no source")
task.phase = "clearing_data"
task.phase_message = "正在删除数据库数据"
await db.commit()
await _broadcast_task_update(task)
count_result = await db.execute(
select(CollectedData.id).where(CollectedData.source == source)
)
collected_ids = [row[0] for row in count_result.all()]
derived_deleted_counts = await clear_derived_datasource_data(db, source)
if collected_ids:
await db.execute(CollectedData.__table__.delete().where(CollectedData.id.in_(collected_ids)))
deleted_count = len(collected_ids)
derived_deleted_count = sum(derived_deleted_counts.values())
task.records_processed = deleted_count + derived_deleted_count
task.total_records = task.records_processed
task.progress = 100.0
task.phase_progress = 100.0
task.phase_current = task.records_processed
task.phase_total = task.records_processed
task.phase_unit = "records"
task.payload = {
**(task.payload or {}),
"deleted_count": deleted_count,
"derived_deleted_count": derived_deleted_count,
"derived_deleted_counts": derived_deleted_counts,
}
task.status = JOB_STATUS_SUCCESS
task.phase = "completed"
task.phase_message = "数据库数据已清理"
task.completed_at = _utcnow()
await db.commit()
await _broadcast_task_update(task)
async def _run_clear_cache_job(db: AsyncSession, task: CollectionTask) -> None:
source = str(task.source or (task.payload or {}).get("source") or "").strip()
if not source:
raise RuntimeError("Clear cache job has no source")
earth_deleted_count = invalidate_earth_layer_cache_for_source(source)
dashboard_deleted_count = int(cache.delete("dashboard:stats")) + int(cache.delete("dashboard:summary"))
deleted_count = earth_deleted_count + dashboard_deleted_count
task.records_processed = deleted_count
task.total_records = deleted_count
task.progress = 100.0
task.phase_progress = 100.0
task.phase = "completed"
task.phase_message = "缓存已清理"
task.payload = {
**(task.payload or {}),
"earth_layer_deleted_count": earth_deleted_count,
"dashboard_deleted_count": dashboard_deleted_count,
}
task.status = JOB_STATUS_SUCCESS
task.completed_at = _utcnow()
await db.commit()
await _broadcast_task_update(task)
await enqueue_earth_refresh_job(db, source=source, payload={"operation": "CACHE_INVALIDATED"})
async def _run_earth_refresh_job(db: AsyncSession, task: CollectionTask) -> None:
payload = task.payload or {}
source = str(payload.get("source") or task.source or "").strip()
layers = list(payload.get("layers") or get_earth_update_layers_for_source(source))
if not source or not layers:
task.status = JOB_STATUS_SUCCESS
task.phase = "completed"
task.phase_message = "没有需要刷新的 Earth 图层"
task.completed_at = _utcnow()
await db.commit()
await _broadcast_task_update(task)
return
deleted_cache_entries = invalidate_earth_layer_cache_for_source(source)
update_payload = {
"event": "earth.layer.changed",
"action": "database_changed",
"source": source,
"table": payload.get("table"),
"data_type": source,
"layers": layers,
"refresh_strategy": payload.get("refresh_strategy") or "clear_then_reload",
"records_processed": payload.get("records_processed", 0),
"operations": payload.get("operations") or [payload.get("operation") or "CHANGE"],
"operation": payload.get("operation"),
"cache_entries_invalidated": deleted_cache_entries,
"timestamp": to_iso8601_utc(_utcnow()),
}
if payload.get("entity") == "interactable":
update_payload.update(
{
"entity": "interactable",
"action": payload.get("action") or "changed",
"ids": payload.get("ids") or payload.get("entity_keys") or [],
"item": payload.get("item"),
}
)
await broadcaster.broadcast_earth_update(update_payload)
task.records_processed = int(payload.get("records_processed") or 0)
task.progress = 100.0
task.phase_progress = 100.0
task.phase = "completed"
task.phase_message = "Earth 图层刷新通知已发送"
task.status = JOB_STATUS_SUCCESS
task.completed_at = _utcnow()
task.payload = {**payload, "cache_entries_invalidated": deleted_cache_entries}
await db.commit()
await _broadcast_task_update(task)
_worker = DataJobWorker()
def start_data_job_worker() -> None:
_worker.start()
async def stop_data_job_worker() -> None:
await _worker.stop()

View File

@@ -32,28 +32,31 @@ class DocsMetadata:
DOCS_METADATA: tuple[DocsMetadata, ...] = (
DocsMetadata(DOCS_README_FILENAME, DEFAULT_DOCS_SLUG, "public", "Overview", 0, "技术文档", "Technical Docs"),
DocsMetadata("quickstart.md", "quickstart", "public", "Manual", 1, "快速开始", "Quickstart"),
DocsMetadata("manual.md", "manual", "public", "Manual", 2, "Planet 使用手册", "Planet Manual"),
DocsMetadata("manual.md", "manual", "public", "Manual", 1, "智能星球使用手册", "Intelligent Planet Manual"),
DocsMetadata("quickstart.md", "quickstart", "public", "Manual", 2, "快速开始", "Quickstart"),
DocsMetadata("faq.md", "faq", "public", "Manual", 3, "常见问题", "FAQ"),
DocsMetadata("location-pipeline-user.md", "location-pipeline-user", "public", "Manual", 4, "Earth 位置候选采集使用手册", "Earth Location Candidate Collection User Guide"),
DocsMetadata("earth-frontend-context.md", "earth-frontend-context", "docs_developer", "Earth", 10, "Earth 前端结构", "Earth Frontend Context"),
DocsMetadata("earth-layer-style-reference.md", "earth-layer-style-reference", "docs_developer", "Earth", 11, "Earth 图层样式属性索引", "Earth Layer Style Reference"),
DocsMetadata("earth-render-layer-order.md", "earth-render-layer-order", "docs_developer", "Earth", 12, "Earth 渲染图层顺序", "Earth Render Layer Order"),
DocsMetadata("earth-satellite-footprint-policy.md", "earth-satellite-footprint-policy", "docs_developer", "Earth", 13, "Earth 卫星覆盖策略", "Earth Satellite Footprint Policy"),
DocsMetadata("platform-data-flows.md", "platform-data-flows", "docs_developer", "Architecture", 5, "业务架构与数据流转", "Business Architecture and Data Flows"),
DocsMetadata("naming-glossary.md", "naming-glossary", "docs_developer", "Architecture", 6, "命名与术语对照", "Naming Glossary"),
DocsMetadata("earth-frontend-context.md", "earth-frontend-context", "docs_developer", "Earth", 10, "智能星球前端结构", "Intelligent Planet Frontend Context"),
DocsMetadata("earth-layer-style-reference.md", "earth-layer-style-reference", "docs_developer", "Earth", 11, "智能星球图层样式属性索引", "Intelligent Planet Layer Style Reference"),
DocsMetadata("earth-render-layer-order.md", "earth-render-layer-order", "docs_developer", "Earth", 12, "智能星球渲染图层顺序", "Intelligent Planet Render Layer Order"),
DocsMetadata("earth-satellite-footprint-policy.md", "earth-satellite-footprint-policy", "docs_developer", "Earth", 13, "智能星球卫星覆盖策略", "Intelligent Planet Satellite Footprint Policy"),
DocsMetadata("earth-bgp-context.md", "earth-bgp-context", "docs_developer", "Earth", 14, "BGP 态势上下文", "BGP Context"),
DocsMetadata("earth-news-live-streams-collector-format.md", "earth-news-live-streams-collector-format", "docs_developer", "Earth", 15, "新闻直播采集格式", "News Live Streams Collector Format"),
DocsMetadata("earth-interactable-usage.md", "earth-interactable-usage", "docs_developer", "Earth", 16, "Earth 可交互图标接入", "Earth Interactable Usage"),
DocsMetadata("earth-toolbar-overlay-coordination.md", "earth-toolbar-overlay-coordination", "docs_developer", "Earth", 17, "Earth 工具栏与浮层协同", "Earth Toolbar and Overlay Coordination"),
DocsMetadata("earth-interactable-usage.md", "earth-interactable-usage", "docs_developer", "Earth", 16, "智能星球可交互图标接入", "Intelligent Planet Interactable Usage"),
DocsMetadata("earth-toolbar-overlay-coordination.md", "earth-toolbar-overlay-coordination", "docs_developer", "Earth", 17, "智能星球工具栏与浮层协同", "Intelligent Planet Toolbar and Overlay Coordination"),
DocsMetadata("frontend-admin-frontend-context.md", "frontend-admin-frontend-context", "docs_developer", "Frontend", 20, "控制台前端结构", "Admin Frontend Context"),
DocsMetadata("frontend-layout-guidelines.md", "frontend-layout-guidelines", "docs_developer", "Frontend", 21, "前端布局指南", "Frontend Layout Guidelines"),
DocsMetadata("docs-gatekeeper-development.md", "docs-gatekeeper-development", "docs_developer", "Frontend", 22, "Docs Gatekeeper 开发说明", "Docs Gatekeeper Development Guide"),
DocsMetadata("tactile-ui-components.md", "tactile-ui-components", "docs_developer", "Frontend", 24, "Tactile UI 组件库", "Tactile UI Components"),
DocsMetadata("backend-collectors.md", "backend-collectors", "docs_developer", "Backend", 30, "数据采集系统", "Data Collectors"),
DocsMetadata("backend-system-service-control.md", "backend-system-service-control", "docs_admin", "Backend", 31, "系统服务控制", "System Service Control"),
DocsMetadata("datasource-collector-settings-connectivity.md", "datasource-collector-settings-connectivity", "docs_developer", "Backend", 32, "数据源、采集器设置与连接验证", "Datasource Collector Settings and Connectivity"),
DocsMetadata("backend-datasources-api-performance.md", "backend-datasources-api-performance", "docs_developer", "Backend", 33, "数据源 API 性能", "Datasource API Performance"),
DocsMetadata("location-pipeline-development.md", "location-pipeline-development", "docs_developer", "Backend", 34, "通用位置估算管线开发说明", "Shared Location Resolution Pipeline Development Guide"),
DocsMetadata("data-job-earth-sync-architecture.md", "data-job-earth-sync-architecture", "docs_developer", "Backend", 34, "数据作业与 Outbox 技术架构", "Data Jobs and Outbox Architecture"),
DocsMetadata("location-pipeline-development.md", "location-pipeline-development", "docs_developer", "Backend", 35, "通用位置估算管线开发说明", "Shared Location Resolution Pipeline Development Guide"),
DocsMetadata("earth-news-live-streams-collector-format.md", "earth-news-live-streams-collector-format", "docs_developer", "Backend", 36, "新闻直播采集格式", "News Live Streams Collector Format"),
DocsMetadata("docs-gatekeeper-development.md", "docs-gatekeeper-development", "docs_developer", "Backend", 37, "Docs Gatekeeper 开发说明", "Docs Gatekeeper Development Guide"),
DocsMetadata("agents-aiprovider.md", "agents-aiprovider", "docs_developer", "Agents", 40, "AI Provider 指南", "AI Provider Guide"),
DocsMetadata("ops-runbook.md", "ops-runbook", "docs_admin", "Ops", 49, "Planet 运维手册", "Planet Ops Runbook"),
DocsMetadata("ops-runbook.md", "ops-runbook", "docs_admin", "Ops", 49, "智能星球运维手册", "Intelligent Planet Ops Runbook"),
DocsMetadata("ops-docker-compose-buildx-upgrade.md", "ops-docker-compose-buildx-upgrade", "docs_admin", "Ops", 50, "Docker + Compose + Buildx 升级", "Docker + Compose + Buildx Upgrade"),
DocsMetadata("ops-planet-sh-startup.md", "ops-planet-sh-startup", "docs_admin", "Ops", 51, "planet.sh 启动机制", "planet.sh Startup"),
)

View File

@@ -0,0 +1,482 @@
"""PostgreSQL LISTEN/NOTIFY bridge for Earth layer refresh events."""
from __future__ import annotations
import asyncio
import json
from collections import deque
from dataclasses import dataclass, field
from datetime import UTC, datetime
from time import monotonic
from typing import Any, Awaitable, Callable
import asyncpg
from app.core.config import settings
from app.core.logging import get_logger
from app.core.time import to_iso8601_utc
from app.core.websocket.broadcaster import broadcaster
from app.services.earth_layer_adapters import (
get_earth_refresh_strategy_for_change,
get_earth_update_layers_for_change,
get_earth_update_layers_for_source,
)
from app.services.earth_layer_cache import invalidate_earth_layer_cache_for_source
logger = get_logger(__name__)
EARTH_DATA_CHANGES_CHANNEL = "planet_earth_data_changes"
DEFAULT_DEBOUNCE_SECONDS = 0.25
DEFAULT_MAX_WAIT_SECONDS = 1.5
DELETE_FAST_FLUSH_SECONDS = 0.05
LISTEN_KEEPALIVE_SECONDS = 5.0
OUTBOX_POLL_LIMIT = 5000
MAX_ENTITY_KEY_SAMPLES = 20
MAX_SEEN_EVENT_IDS = 20000
BroadcastFn = Callable[[dict[str, Any]], Awaitable[None]]
InvalidateFn = Callable[[str], int]
def normalize_asyncpg_dsn(dsn: str) -> str:
"""Convert SQLAlchemy asyncpg URLs into asyncpg-compatible URLs."""
return dsn.replace("postgresql+asyncpg://", "postgresql://", 1)
def build_earth_update_from_db_payload(payload: dict[str, Any]) -> dict[str, Any] | None:
table = payload.get("table")
source = str(payload.get("source") or "").strip()
table_name = str(table or "").strip()
if not source and not table_name:
return None
layers = get_earth_update_layers_for_change(table_name, source)
if not layers:
return None
refresh_strategy = get_earth_refresh_strategy_for_change(table_name, source) or "clear_then_reload"
source_has_adapter = bool(get_earth_update_layers_for_source(source))
effective_source = source if source_has_adapter else (table_name if table_name else source)
operation = payload.get("operation")
update: dict[str, Any] = {
"event": "earth.layer.changed",
"action": "database_changed",
"source": effective_source,
"original_source": source or None,
"table": table_name or None,
"data_type": effective_source,
"layers": layers,
"refresh_strategy": refresh_strategy,
"operation": operation,
"entity_key": payload.get("entity_key"),
"timestamp": to_iso8601_utc(datetime.now(UTC)),
}
if table_name == "earth_interactables" and refresh_strategy == "delta":
ids = payload.get("entity_keys")
if not isinstance(ids, list):
ids = [payload.get("entity_key")] if payload.get("entity_key") else []
update.update(
{
"entity": "interactable",
"action": "deleted" if operation == "DELETE" else "changed",
"ids": [str(item) for item in ids if item],
"item": None,
}
)
return {
**update,
}
@dataclass
class PendingEarthDbChange:
source: str
layers: list[str]
table: str | None = None
refresh_strategy: str = "clear_then_reload"
entity: str | None = None
action: str = "database_changed"
records_processed: int = 0
operations: set[str] = field(default_factory=set)
entity_keys: list[str] = field(default_factory=list)
first_occurred_at: str | None = None
last_occurred_at: str | None = None
first_seen_monotonic: float = field(default_factory=monotonic)
last_seen_monotonic: float = field(default_factory=monotonic)
def add(self, payload: dict[str, Any]) -> None:
self.last_seen_monotonic = monotonic()
records_processed = payload.get("records_processed", 1)
try:
records_processed = int(records_processed)
except (TypeError, ValueError):
records_processed = 1
self.records_processed += max(records_processed, 1)
operation = payload.get("operation")
if operation:
self.operations.add(str(operation))
entity_keys = payload.get("entity_keys")
if not isinstance(entity_keys, list):
entity_key = payload.get("entity_key")
entity_keys = [entity_key] if entity_key else []
for entity_key in entity_keys:
if entity_key and len(self.entity_keys) < MAX_ENTITY_KEY_SAMPLES:
self.entity_keys.append(str(entity_key))
occurred_at = payload.get("occurred_at")
if occurred_at:
occurred_at = str(occurred_at)
self.first_occurred_at = self.first_occurred_at or occurred_at
self.last_occurred_at = occurred_at
class EarthDbChangeDispatcher:
"""Debounces database notifications and broadcasts Earth refresh hints."""
def __init__(
self,
*,
broadcast_earth_update: BroadcastFn | None = None,
invalidate_cache: InvalidateFn | None = None,
debounce_seconds: float = DEFAULT_DEBOUNCE_SECONDS,
max_wait_seconds: float = DEFAULT_MAX_WAIT_SECONDS,
) -> None:
self._broadcast_earth_update = broadcast_earth_update or broadcaster.broadcast_earth_update
self._invalidate_cache = invalidate_cache or invalidate_earth_layer_cache_for_source
self._debounce_seconds = debounce_seconds
self._max_wait_seconds = max(max_wait_seconds, debounce_seconds)
self._pending: dict[str, PendingEarthDbChange] = {}
self._flush_tasks: dict[str, asyncio.Task[None]] = {}
self._seen_event_ids: set[int] = set()
self._seen_event_order: deque[int] = deque()
def handle_notification(self, payload_text: str) -> bool:
try:
payload = json.loads(payload_text)
except json.JSONDecodeError:
logger.warning_event(
"Ignoring malformed Earth database change notification",
event="earth.db_changes.notification_malformed",
)
return False
if not isinstance(payload, dict):
return False
return self.handle_payload(payload)
def handle_payload(self, payload: dict[str, Any]) -> bool:
event_id = payload.get("event_id")
if event_id is not None:
try:
normalized_event_id = int(event_id)
except (TypeError, ValueError):
normalized_event_id = None
if normalized_event_id is not None:
if normalized_event_id in self._seen_event_ids:
return False
self._remember_event_id(normalized_event_id)
update = build_earth_update_from_db_payload(payload)
if not update:
return False
source = update["source"]
pending = self._pending.get(source)
if pending is None:
pending = PendingEarthDbChange(
source=source,
layers=list(update["layers"]),
table=update.get("table"),
refresh_strategy=str(update.get("refresh_strategy") or "clear_then_reload"),
entity=update.get("entity"),
action=str(update.get("action") or "database_changed"),
)
self._pending[source] = pending
pending.add(payload)
task = self._flush_tasks.pop(source, None)
if task and not task.done():
task.cancel()
self._flush_tasks[source] = asyncio.create_task(
self._flush_later(source, delay_seconds=self._next_flush_delay(pending))
)
return True
def _next_flush_delay(self, pending: PendingEarthDbChange) -> float:
if "DELETE" in pending.operations and pending.refresh_strategy == "clear_then_reload":
return DELETE_FAST_FLUSH_SECONDS
elapsed = max(0.0, monotonic() - pending.first_seen_monotonic)
remaining = self._max_wait_seconds - elapsed
if remaining <= 0:
return 0.0
return min(self._debounce_seconds, remaining)
def _remember_event_id(self, event_id: int) -> None:
self._seen_event_ids.add(event_id)
self._seen_event_order.append(event_id)
while len(self._seen_event_order) > MAX_SEEN_EVENT_IDS:
expired_event_id = self._seen_event_order.popleft()
self._seen_event_ids.discard(expired_event_id)
async def _flush_later(self, source: str, *, delay_seconds: float) -> None:
try:
if delay_seconds > 0:
await asyncio.sleep(delay_seconds)
await self.flush_source(source)
except asyncio.CancelledError:
raise
except Exception as exc:
logger.exception_event(
"Failed to broadcast debounced Earth database change",
event="earth.db_changes.flush_failed",
context={"source": source, "error": str(exc)},
)
finally:
current = self._flush_tasks.get(source)
if current is asyncio.current_task():
self._flush_tasks.pop(source, None)
async def flush_source(self, source: str) -> None:
pending = self._pending.get(source)
if pending is None:
return
flushed_at = datetime.now(UTC)
deleted_cache_entries = self._invalidate_cache(source)
operations = sorted(pending.operations)
payload: dict[str, Any] = {
"event": "earth.layer.changed",
"action": "database_changed",
"source": source,
"table": pending.table,
"data_type": source,
"layers": pending.layers,
"refresh_strategy": pending.refresh_strategy,
"records_processed": pending.records_processed,
"operations": operations,
"operation": operations[-1] if len(operations) == 1 else None,
"entity_keys": pending.entity_keys,
"entity_key_sample_size": len(pending.entity_keys),
"cache_entries_invalidated": deleted_cache_entries,
"first_occurred_at": pending.first_occurred_at,
"last_occurred_at": pending.last_occurred_at,
"debounce_ms": int((monotonic() - pending.first_seen_monotonic) * 1000),
"total_latency_ms": self._total_latency_ms(pending, flushed_at),
"timestamp": to_iso8601_utc(flushed_at),
}
if pending.entity == "interactable":
payload.update(
{
"entity": "interactable",
"action": "deleted" if "DELETE" in pending.operations else "changed",
"ids": pending.entity_keys,
"item": None,
}
)
await self._broadcast_earth_update(payload)
self._pending.pop(source, None)
logger.info_event(
"Broadcasted Earth database change",
event="earth.db_changes.broadcasted",
context={
"source": source,
"layers": pending.layers,
"records_processed": pending.records_processed,
"cache_entries_invalidated": deleted_cache_entries,
"debounce_ms": int((monotonic() - pending.first_seen_monotonic) * 1000),
"total_latency_ms": payload["total_latency_ms"],
},
)
@staticmethod
def _total_latency_ms(pending: PendingEarthDbChange, flushed_at: datetime) -> int | None:
occurred_at = pending.first_occurred_at
if not occurred_at:
return None
try:
normalized = occurred_at.replace("Z", "+00:00")
occurred = datetime.fromisoformat(normalized)
if occurred.tzinfo is None:
occurred = occurred.replace(tzinfo=UTC)
return max(0, int((flushed_at - occurred.astimezone(UTC)).total_seconds() * 1000))
except ValueError:
return None
async def flush_all(self) -> None:
sources = list(self._pending)
for source in sources:
task = self._flush_tasks.pop(source, None)
if task and not task.done():
task.cancel()
await self.flush_source(source)
async def stop(self) -> None:
tasks = [task for task in self._flush_tasks.values() if not task.done()]
for task in tasks:
task.cancel()
if tasks:
await asyncio.gather(*tasks, return_exceptions=True)
self._flush_tasks.clear()
await self.flush_all()
class EarthDbChangeListener:
def __init__(
self,
*,
dsn: str,
dispatcher: EarthDbChangeDispatcher,
channel: str = EARTH_DATA_CHANGES_CHANNEL,
) -> None:
self._dsn = normalize_asyncpg_dsn(dsn)
self._dispatcher = dispatcher
self._channel = channel
self._task: asyncio.Task[None] | None = None
self._stop_event: asyncio.Event | None = None
self._connection: asyncpg.Connection | None = None
self._loop: asyncio.AbstractEventLoop | None = None
def start(self) -> None:
if self._task and not self._task.done():
return
self._loop = asyncio.get_running_loop()
self._stop_event = asyncio.Event()
self._task = asyncio.create_task(self._run())
async def stop(self) -> None:
if self._stop_event:
self._stop_event.set()
if self._connection:
await self._connection.close()
if self._task:
await asyncio.gather(self._task, return_exceptions=True)
await self._dispatcher.stop()
async def _run(self) -> None:
backoff_seconds = 1.0
assert self._stop_event is not None
while not self._stop_event.is_set():
try:
self._connection = await asyncpg.connect(self._dsn)
await self._connection.add_listener(self._channel, self._on_notification)
logger.info_event(
"Earth database change listener connected",
event="earth.db_changes.connected",
context={"channel": self._channel},
)
backoff_seconds = 1.0
while not self._stop_event.is_set():
try:
await asyncio.wait_for(
self._stop_event.wait(),
timeout=LISTEN_KEEPALIVE_SECONDS,
)
except asyncio.TimeoutError:
await self._poll_outbox()
except asyncio.CancelledError:
raise
except Exception as exc:
logger.exception_event(
"Earth database change listener failed",
event="earth.db_changes.listener_failed",
context={"channel": self._channel, "error": str(exc)},
)
try:
await asyncio.wait_for(self._stop_event.wait(), timeout=backoff_seconds)
except asyncio.TimeoutError:
pass
backoff_seconds = min(backoff_seconds * 2, 30.0)
finally:
if self._connection:
try:
await self._connection.remove_listener(self._channel, self._on_notification)
except Exception:
pass
try:
await self._connection.close()
except Exception:
pass
self._connection = None
async def _poll_outbox(self) -> None:
if self._connection is None:
return
rows = await self._connection.fetch(
"""
SELECT id, payload
FROM earth_data_change_events
WHERE consumed_at IS NULL
ORDER BY id
LIMIT $1
""",
OUTBOX_POLL_LIMIT,
)
accepted_count = 0
consumed_ids: list[int] = []
for row in rows:
payload = row["payload"]
if isinstance(payload, str):
try:
payload = json.loads(payload)
except json.JSONDecodeError:
consumed_ids.append(int(row["id"]))
continue
if isinstance(payload, dict):
if self._dispatcher.handle_payload(payload):
accepted_count += 1
consumed_ids.append(int(row["id"]))
else:
consumed_ids.append(int(row["id"]))
if consumed_ids:
await self._dispatcher.flush_all()
if consumed_ids:
await self._connection.execute(
"""
UPDATE earth_data_change_events
SET consumed_at = NOW()
WHERE id = ANY($1::bigint[])
AND consumed_at IS NULL
""",
consumed_ids,
)
if rows:
logger.info_event(
"Polled Earth database change outbox",
event="earth.db_changes.outbox_polled",
context={"events": len(rows), "accepted": accepted_count},
)
def _on_notification(
self,
_connection: asyncpg.Connection,
_pid: int,
_channel: str,
payload: str,
) -> None:
if self._loop and self._loop.is_running():
self._loop.call_soon_threadsafe(self._dispatcher.handle_notification, payload)
return
self._dispatcher.handle_notification(payload)
_dispatcher = EarthDbChangeDispatcher(
broadcast_earth_update=broadcaster.broadcast_earth_update,
invalidate_cache=invalidate_earth_layer_cache_for_source,
)
_listener: EarthDbChangeListener | None = None
def start_earth_db_change_listener() -> None:
global _listener
if _listener is not None:
return
_listener = EarthDbChangeListener(dsn=settings.DATABASE_URL, dispatcher=_dispatcher)
_listener.start()
async def stop_earth_db_change_listener() -> None:
global _listener
if _listener is None:
await _dispatcher.stop()
return
listener = _listener
_listener = None
await listener.stop()

View File

@@ -0,0 +1,113 @@
from __future__ import annotations
from datetime import UTC, datetime
from typing import Any
from uuid import uuid4
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.time import to_iso8601_utc
from app.core.websocket.broadcaster import broadcaster
from app.models.earth_interactable import EarthInteractable
from app.services.earth_layer_cache import EARTH_LAYER_CACHE_PREFIX, earth_layer_cache
INTERACTABLE_ENTITY = "interactable"
INTERACTABLE_LAYER = "interactables"
def normalize_interactable_id(value: str | None = None) -> str:
raw = str(value or "").strip()
return raw or f"interactable-{uuid4().hex}"
def serialize_interactable(record: EarthInteractable) -> dict[str, Any]:
return {
"id": record.id,
"layer": record.layer,
"kind": record.kind,
"label": record.label,
"description": record.description,
"latitude": record.latitude,
"longitude": record.longitude,
"altitude": record.altitude,
"revision": record.revision,
"properties": record.properties or {},
"is_deleted": bool(record.is_deleted),
"created_at": to_iso8601_utc(record.created_at),
"updated_at": to_iso8601_utc(record.updated_at),
"deleted_at": to_iso8601_utc(record.deleted_at),
}
def interactables_to_geojson(items: list[EarthInteractable]) -> dict[str, Any]:
return {
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"id": item.id,
"geometry": {
"type": "Point",
"coordinates": [item.longitude, item.latitude],
},
"properties": serialize_interactable(item),
}
for item in items
if not item.is_deleted
],
}
def invalidate_interactable_cache(layer: str | None = None) -> int:
layer_key = str(layer or "*").strip() or "*"
deleted = earth_layer_cache.delete_pattern(
f"{EARTH_LAYER_CACHE_PREFIX}:interactables:layer:{layer_key}*"
)
if layer_key != "all":
deleted += earth_layer_cache.delete_pattern(
f"{EARTH_LAYER_CACHE_PREFIX}:interactables:layer:all*"
)
return deleted
def build_interactable_event(
*,
action: str,
record: EarthInteractable,
include_item: bool = True,
) -> dict[str, Any]:
item = serialize_interactable(record)
return {
"entity": INTERACTABLE_ENTITY,
"action": action,
"layer": record.layer,
"layers": [INTERACTABLE_LAYER],
"ids": [record.id],
"revision": record.revision,
"changed_at": item["deleted_at"] or item["updated_at"] or to_iso8601_utc(datetime.now(UTC)),
"item": item if include_item else None,
"source": "earth_interactables",
}
async def publish_interactable_event(action: str, record: EarthInteractable, *, include_item: bool = True) -> None:
await broadcaster.broadcast_earth_update(
build_interactable_event(action=action, record=record, include_item=include_item)
)
async def list_interactables(
db: AsyncSession,
*,
layer: str | None = None,
include_deleted: bool = False,
) -> list[EarthInteractable]:
stmt = select(EarthInteractable)
if layer:
stmt = stmt.where(EarthInteractable.layer == layer)
if not include_deleted:
stmt = stmt.where(EarthInteractable.is_deleted.is_(False))
stmt = stmt.order_by(EarthInteractable.updated_at.desc(), EarthInteractable.id.asc())
result = await db.execute(stmt)
return list(result.scalars().all())

View File

@@ -0,0 +1,179 @@
"""Earth layer adapter registry for datasource-backed refresh behavior."""
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Any
from sqlalchemy.ext.asyncio import AsyncSession
@dataclass(frozen=True)
class EarthLayerAdapter:
sources: frozenset[str]
layers: tuple[str, ...]
cache_patterns: tuple[str, ...]
tables: frozenset[str] = field(default_factory=frozenset)
derived_models: tuple[str, ...] = field(default_factory=tuple)
refresh_strategy: str = "clear_then_reload"
EARTH_LAYER_ADAPTERS: tuple[EarthLayerAdapter, ...] = (
EarthLayerAdapter(
sources=frozenset({"barentswatch_vessels", "aisstream_vessels", "vessel_static", "vessel_position", "ais_raw_observations", "ais_source_health"}),
tables=frozenset({"vessel_static", "vessel_position", "ais_raw_observations", "ais_source_health"}),
layers=("vessels",),
cache_patterns=("vessels*", "summary*"),
),
EarthLayerAdapter(
sources=frozenset(
{
"telegeography_cables",
"telegeography_landing",
"telegeography_landing_points",
"telegeography_systems",
"telegeography_cable_systems",
"arcgis_cables",
"arcgis_landing_points",
"arcgis_cable_landing_relation",
"arcgis_cable_landing_relations",
"fao_landing_points",
}
),
tables=frozenset({"collected_data"}),
layers=("cables",),
cache_patterns=("cables*", "landing-points*", "summary*"),
),
EarthLayerAdapter(
sources=frozenset({"celestrak_tle", "spacetrack_tle"}),
tables=frozenset({"collected_data"}),
layers=("satellites",),
cache_patterns=("satellites*", "summary*"),
),
EarthLayerAdapter(
sources=frozenset(
{
"top500",
"top500_supercomputers",
"epoch_ai_gpu",
"huggingface_models",
"huggingface_datasets",
"huggingface_spaces",
"compute_center_locations",
}
),
tables=frozenset({"compute_center_locations"}),
layers=("computeCenters",),
cache_patterns=("compute-centers*", "summary*"),
refresh_strategy="reload",
),
EarthLayerAdapter(
sources=frozenset(
{
"ris_live_bgp",
"bgpstream_bgp",
"iptoasn_prefix_geo",
"opengeofeed_prefix_geo",
"nro_delegated_prefix_geo",
"bgp_observations",
"bgp_anomalies",
"bgp_incidents",
"bgp_collector_locations",
}
),
tables=frozenset({"bgp_observations", "bgp_anomalies", "bgp_incidents", "bgp_collector_locations"}),
layers=("bgp",),
cache_patterns=("bgp*", "summary*"),
derived_models=("bgp_observations", "bgp_anomalies", "bgp_incidents"),
),
EarthLayerAdapter(
sources=frozenset({"news_live_streams"}),
tables=frozenset({"collected_data"}),
layers=("media",),
cache_patterns=("summary*",),
refresh_strategy="reload",
),
EarthLayerAdapter(
sources=frozenset({"media_news_archive", "earth_news_items"}),
tables=frozenset({"earth_news_items"}),
layers=("news",),
cache_patterns=("summary*",),
refresh_strategy="reload",
),
EarthLayerAdapter(
sources=frozenset({"earth_interactables"}),
tables=frozenset({"earth_interactables"}),
layers=("interactables",),
cache_patterns=("interactables*", "summary*"),
refresh_strategy="delta",
),
)
_ADAPTERS_BY_SOURCE = {
source: adapter
for adapter in EARTH_LAYER_ADAPTERS
for source in adapter.sources
}
_ADAPTERS_BY_TABLE = {
table: adapter
for adapter in EARTH_LAYER_ADAPTERS
for table in adapter.tables
}
def get_earth_layer_adapter_for_source(source: str | None) -> EarthLayerAdapter | None:
return _ADAPTERS_BY_SOURCE.get(str(source or "").strip())
def get_earth_layer_adapter_for_change(table: str | None, source: str | None) -> EarthLayerAdapter | None:
table_key = str(table or "").strip()
source_key = str(source or "").strip()
if table_key and table_key != "collected_data":
adapter = _ADAPTERS_BY_TABLE.get(table_key)
if adapter is not None:
return adapter
return get_earth_layer_adapter_for_source(source_key)
def get_earth_update_layers_for_source(source: str | None) -> list[str]:
adapter = get_earth_layer_adapter_for_source(source)
return list(adapter.layers) if adapter else []
def get_earth_update_layers_for_change(table: str | None, source: str | None) -> list[str]:
adapter = get_earth_layer_adapter_for_change(table, source)
return list(adapter.layers) if adapter else []
def get_earth_refresh_strategy_for_change(table: str | None, source: str | None) -> str | None:
adapter = get_earth_layer_adapter_for_change(table, source)
return adapter.refresh_strategy if adapter else None
def get_earth_cache_patterns_for_source(source: str | None) -> list[str]:
adapter = get_earth_layer_adapter_for_source(source)
return list(adapter.cache_patterns) if adapter else []
async def clear_derived_datasource_data(db: AsyncSession, source: str) -> dict[str, int]:
adapter = get_earth_layer_adapter_for_source(source)
if adapter is None or not adapter.derived_models:
return {}
from app.models.bgp_anomaly import BGPAnomaly
from app.models.bgp_incident import BGPIncident
from app.models.bgp_observation import BGPObservation
model_by_key: dict[str, Any] = {
"bgp_observations": BGPObservation,
"bgp_anomalies": BGPAnomaly,
"bgp_incidents": BGPIncident,
}
deleted_counts: dict[str, int] = {}
for key in adapter.derived_models:
model = model_by_key.get(key)
if model is None:
continue
result = await db.execute(model.__table__.delete().where(model.source == source))
deleted_counts[key] = int(result.rowcount or 0)
return deleted_counts

View File

@@ -268,34 +268,10 @@ def apply_payload_budget(payload: dict[str, Any], policy: EarthLayerCachePolicy)
def invalidate_earth_layer_cache_for_source(source: str) -> int:
from app.services.earth_layer_adapters import get_earth_cache_patterns_for_source
source_key = str(source or "").strip()
patterns = {
"barentswatch_vessels": ["vessels*", "summary*"],
"aisstream_vessels": ["vessels*", "summary*"],
"telegeography_cables": ["cables*", "landing-points*", "summary*"],
"telegeography_landing": ["landing-points*", "summary*"],
"telegeography_landing_points": ["landing-points*", "summary*"],
"telegeography_systems": ["cables*", "summary*"],
"telegeography_cable_systems": ["cables*", "summary*"],
"arcgis_cables": ["cables*", "landing-points*", "summary*"],
"arcgis_landing_points": ["landing-points*", "summary*"],
"arcgis_cable_landing_relation": ["landing-points*", "summary*"],
"arcgis_cable_landing_relations": ["landing-points*", "summary*"],
"fao_landing_points": ["landing-points*", "summary*"],
"celestrak_tle": ["satellites*", "summary*"],
"spacetrack_tle": ["satellites*", "summary*"],
"top500": ["compute-centers*", "summary*"],
"top500_supercomputers": ["compute-centers*", "summary*"],
"epoch_ai_gpu": ["compute-centers*", "summary*"],
"huggingface_models": ["compute-centers*", "summary*"],
"huggingface_datasets": ["compute-centers*", "summary*"],
"huggingface_spaces": ["compute-centers*", "summary*"],
"ris_live_bgp": ["bgp*", "summary*"],
"bgpstream_bgp": ["bgp*", "summary*"],
"iptoasn_prefix_geo": ["bgp*", "summary*"],
"opengeofeed_prefix_geo": ["bgp*", "summary*"],
"nro_delegated_prefix_geo": ["bgp*", "summary*"],
}.get(source_key, [])
patterns = get_earth_cache_patterns_for_source(source_key)
deleted = 0
for layer_pattern in patterns:
deleted += earth_layer_cache.delete_pattern(f"{EARTH_LAYER_CACHE_PREFIX}:{layer_pattern}")

View File

@@ -1063,6 +1063,7 @@ def _build_payload(
lon: float | None,
active_region: str,
items: list[ParsedNewsItem],
cruise_items: list[ParsedNewsItem] | None = None,
sources: list[NewsFeedSource],
errors: list[str],
stale: bool,
@@ -1082,6 +1083,10 @@ def _build_payload(
},
"sources": _serialize_sources(sources),
"items": [_serialize_item(item, active_region=active_region) for item in items],
"cruise_items": [
_serialize_item(item, active_region=active_region)
for item in (cruise_items if cruise_items is not None else items)
],
"errors": errors,
"stale": stale,
}
@@ -1295,6 +1300,7 @@ async def get_earth_news_payload(
from app.services.earth_news_store import (
get_earth_news_freshness,
list_earth_news_cruise_items,
list_earth_news_items,
upsert_earth_news_items,
)
@@ -1312,6 +1318,13 @@ async def get_earth_news_payload(
active_region=active_region,
limit=MAX_ITEMS_TOTAL,
)
if hasattr(db, "execute"):
cruise_items = await list_earth_news_cruise_items(
db,
limit=MAX_ITEMS_TOTAL * len(REGION_ANCHORS),
)
else:
cruise_items = items
await _enqueue_unverified_locations(items)
stale = bool(errors and items)
@@ -1320,6 +1333,7 @@ async def get_earth_news_payload(
lon=lon,
active_region=active_region,
items=items,
cruise_items=cruise_items,
sources=sources,
errors=errors,
stale=stale,

View File

@@ -78,6 +78,24 @@ async def list_earth_news_items(
return [record_to_parsed_news_item(record) for record in result.scalars().all()]
async def list_earth_news_cruise_items(
db: AsyncSession,
*,
limit: int,
) -> list[ParsedNewsItem]:
result = await db.execute(
select(EarthNewsItem)
.order_by(
EarthNewsItem.region.asc(),
EarthNewsItem.published_at.desc().nullslast(),
EarthNewsItem.last_seen_at.desc(),
EarthNewsItem.feed_name.asc(),
)
.limit(limit)
)
return [record_to_parsed_news_item(record) for record in result.scalars().all()]
async def get_earth_news_freshness(
db: AsyncSession,
*,

View File

@@ -10,6 +10,7 @@ from fastapi import HTTPException, status
from sqlalchemy import func, select, update
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.logging import get_logger
from app.db.session import async_session_factory
from app.models.playground_message import PlaygroundMessage
from app.models.playground_session import PlaygroundSession
@@ -27,9 +28,11 @@ from app.schemas.ai import (
SituationalAnalysisRequest,
)
from app.services.ai_client import AIProviderClient
from app.services.business_logs import emit_business_log, exception_context
from app.services.playground_session_store import _to_response as session_to_response
from app.services.playground_session_store import upsert_playground_session
logger = get_logger(__name__, service="ai")
STREAM_CHUNK_SIZE = 24
STREAM_INTERVAL_SECONDS = 0.08
THINKING_PREVIEW_SECONDS = 2.6
@@ -624,7 +627,42 @@ async def _run_assistant_message(
thinking={"type": "enabled"},
)
await emit_business_log(
logger,
event="ai.playground.run.start",
message="Playground AI run started",
category="ai",
service="ai",
module=__name__,
request_id=request_id,
user_id=user_id,
context={
"session_id": session_id,
"session_key": session_key,
"user_message_id": user_message_id,
"assistant_message_id": assistant_message_id,
"preset": payload.selected_preset_key,
},
)
analysis = await provider_client.analyze(request_payload, request_id=request_id)
await emit_business_log(
logger,
event="ai.playground.run.success",
message="Playground AI run completed",
category="ai",
service="ai",
module=__name__,
request_id=request_id,
user_id=user_id,
context={
"session_id": session_id,
"session_key": session_key,
"provider": analysis.provider,
"model": analysis.model,
"content_block_count": len(analysis.content_blocks or []),
"thinking_block_count": len(analysis.thinking_blocks or []),
},
)
async with async_session_factory() as db:
assistant_message = await _mark_message_state(
@@ -704,6 +742,23 @@ async def _run_assistant_message(
await db.flush()
await db.commit()
except asyncio.CancelledError:
await emit_business_log(
logger,
event="ai.playground.run.cancelled",
message="Playground AI run cancelled",
category="ai",
level="warning",
service="ai",
module=__name__,
request_id=request_id,
user_id=user_id,
context={
"session_id": session_id,
"session_key": session_key,
"assistant_message_id": assistant_message_id,
"duration_ms": round((perf_counter() - started_at) * 1000),
},
)
async with async_session_factory() as db:
result = await db.execute(select(PlaygroundMessage).where(PlaygroundMessage.id == assistant_message_id))
message = result.scalar_one_or_none()
@@ -715,6 +770,26 @@ async def _run_assistant_message(
await db.commit()
raise
except Exception as exc:
await emit_business_log(
logger,
event="ai.playground.run.failed",
message="Playground AI run failed",
category="ai",
level="error",
service="ai",
module=__name__,
request_id=request_id,
user_id=user_id,
context=exception_context(
exc,
{
"session_id": session_id,
"session_key": session_key,
"assistant_message_id": assistant_message_id,
"duration_ms": round((perf_counter() - started_at) * 1000),
},
),
)
error_message = _format_run_exception(exc)
async with async_session_factory() as db:
result = await db.execute(select(PlaygroundMessage).where(PlaygroundMessage.id == assistant_message_id))

View File

@@ -13,6 +13,7 @@ from app.db.session import async_session_factory
from app.core.time import to_iso8601_utc
from app.models.datasource import DataSource
from app.models.task import CollectionTask
from app.services.business_logs import emit_business_log, emit_business_log_background, exception_context
from app.services.collectors.registry import collector_registry
from app.services.datasource_connectivity import (
build_builtin_connectivity_checksum,
@@ -124,6 +125,15 @@ async def run_collector_task(collector_name: str):
event="collector.run.skipped_disabled",
context={"collector_name": collector_name},
)
await emit_business_log(
logger,
event="collector.run.skipped_disabled",
message="Skipping disabled collector",
category="collector",
service="scheduler",
module=__name__,
context={"collector_name": collector_name, "datasource_id": datasource.id, "status": "skipped"},
)
return
running_result = await db.execute(
@@ -152,6 +162,21 @@ async def run_collector_task(collector_name: str):
event="collector.run.skipped_already_running",
context={"collector_name": collector_name, "task_id": existing_running.id},
)
await emit_business_log(
logger,
event="collector.run.skipped_already_running",
message="Skipping collector trigger because task is already running",
category="collector",
level="warning",
service="scheduler",
module=__name__,
context={
"collector_name": collector_name,
"datasource_id": datasource.id,
"task_id": existing_running.id,
"status": "skipped",
},
)
return
existing_error = (existing_running.error_message or "").strip()
@@ -173,6 +198,21 @@ async def run_collector_task(collector_name: str):
event="collector.run.stale_task_failed",
context={"collector_name": collector_name, "task_id": existing_running.id},
)
await emit_business_log(
logger,
event="collector.run.stale_task_failed",
message="Marked stale running task as failed before rerun",
category="collector",
level="warning",
service="scheduler",
module=__name__,
context={
"collector_name": collector_name,
"datasource_id": datasource.id,
"task_id": existing_running.id,
"status": "failed",
},
)
try:
datasource_id = datasource.id
@@ -183,6 +223,15 @@ async def run_collector_task(collector_name: str):
event="collector.run.started",
context={"collector_name": collector_name, "datasource_id": datasource_id},
)
await emit_business_log(
logger,
event="collector.run.scheduled_started",
message="Scheduler started collector run",
category="collector",
service="scheduler",
module=__name__,
context={"collector_name": collector_name, "datasource_id": datasource_id, "status": "running"},
)
task_result = await collector.run(db)
datasource = await db.get(DataSource, datasource_id)
if datasource is None:
@@ -217,6 +266,20 @@ async def run_collector_task(collector_name: str):
event="collector.run.completed",
context={"collector_name": collector_name, "datasource_id": datasource_id, "result": task_result},
)
await emit_business_log(
logger,
event="collector.run.scheduled_completed",
message="Scheduler completed collector run",
category="collector",
service="scheduler",
module=__name__,
context={
"collector_name": collector_name,
"datasource_id": datasource_id,
"status": task_result.get("status"),
"result": task_result,
},
)
except asyncio.CancelledError:
await db.rollback()
datasource = await db.get(DataSource, datasource_id)
@@ -228,6 +291,16 @@ async def run_collector_task(collector_name: str):
event="collector.run.cancelled",
context={"collector_name": collector_name, "datasource_id": datasource.id},
)
await emit_business_log(
logger,
event="collector.run.cancelled",
message="Collector cancelled by operator",
category="collector",
level="warning",
service="scheduler",
module=__name__,
context={"collector_name": collector_name, "datasource_id": datasource.id, "status": "cancelled"},
)
raise
except Exception as exc:
await db.rollback()
@@ -240,6 +313,19 @@ async def run_collector_task(collector_name: str):
event="collector.run.failed",
context={"collector_name": collector_name, "datasource_id": datasource.id, "error": str(exc)},
)
await emit_business_log(
logger,
event="collector.run.failed",
message="Collector failed",
category="collector",
level="error",
service="scheduler",
module=__name__,
context=exception_context(
exc,
{"collector_name": collector_name, "datasource_id": datasource.id, "status": "failed"},
),
)
async def cleanup_stale_running_tasks(max_age_hours: int = 2) -> int:
@@ -361,6 +447,16 @@ def run_collector_now(collector_name: str) -> bool:
event="collector.trigger.skipped_already_running",
context={"collector_name": collector_name},
)
emit_business_log_background(
logger,
event="collector.trigger.skipped_already_running",
message="Collector is already running in-memory; skipping duplicate trigger",
category="collector",
level="warning",
service="scheduler",
module=__name__,
context={"collector_name": collector_name, "status": "skipped"},
)
return False
try:
@@ -378,6 +474,15 @@ def run_collector_now(collector_name: str) -> bool:
event="collector.trigger.started",
context={"collector_name": collector_name},
)
emit_business_log_background(
logger,
event="collector.trigger.started",
message="Triggered collector",
category="collector",
service="scheduler",
module=__name__,
context={"collector_name": collector_name, "status": "queued"},
)
return True
except Exception as exc:
logger.error_event(
@@ -385,6 +490,16 @@ def run_collector_now(collector_name: str) -> bool:
event="collector.trigger.failed",
context={"collector_name": collector_name, "error": str(exc)},
)
emit_business_log_background(
logger,
event="collector.trigger.failed",
message="Failed to trigger collector",
category="collector",
level="error",
service="scheduler",
module=__name__,
context=exception_context(exc, {"collector_name": collector_name, "status": "failed"}),
)
return False

View File

@@ -0,0 +1,128 @@
import pytest
from fastapi import HTTPException
from app.schemas.ai import SituationalAnalysisRequest
from app.services.ai_tools import web_search as web_search_module
from app.services.ai_tools.schemas import SearchEvidence, WebSearchConfig, WebSearchProviderConfig
from app.services.ai_tools.web_search import WebSearchClient
from app.services import ai_client as ai_client_module
from app.services.ai_client import AIProviderClient
@pytest.mark.asyncio
async def test_ai_client_analyze_logs_summary_without_prompt(monkeypatch):
events = []
async def fake_emit_business_log(_logger, **payload):
events.append(payload)
async def fake_request(self, method, path, json=None, request_id=None, operation="request", payload_summary=None):
return {
"provider": "test-provider",
"model": "test-model",
"content": "ok",
"content_blocks": [],
"text_blocks": ["ok"],
"thinking_blocks": [],
"raw_response": {},
}
monkeypatch.setattr(ai_client_module, "emit_business_log", fake_emit_business_log)
monkeypatch.setattr(AIProviderClient, "_request", fake_request)
client = AIProviderClient(
service_url="http://provider.test",
llm_config={"provider": "openai", "provider_api": "openai-completions", "model": "gpt-test", "api_key": "sk-secret"},
)
result = await client.analyze(
SituationalAnalysisRequest(
title="Sensitive title",
objective="Do not store this full prompt",
observations=["secret observation"],
constraints=["secret constraint"],
context={"source": "test", "private": "value"},
),
request_id="req-ai-test",
)
assert result.model == "test-model"
assert [event["event"] for event in events] == [
"ai.provider.analyze.start",
"ai.provider.analyze.success",
]
serialized = str(events)
assert "Do not store this full prompt" not in serialized
assert "secret observation" not in serialized
assert "sk-secret" not in serialized
start_context = events[0]["context"]
assert start_context["model"] == "gpt-test"
assert start_context["input_summary"]["objective_length"] == len("Do not store this full prompt")
assert start_context["input_summary"]["observation_count"] == 1
assert start_context["input_summary"]["context_keys"] == ["private", "source"]
@pytest.mark.asyncio
async def test_ai_client_analyze_logs_failure(monkeypatch):
events = []
async def fake_emit_business_log(_logger, **payload):
events.append(payload)
async def fake_request(self, method, path, json=None, request_id=None, operation="request", payload_summary=None):
raise HTTPException(status_code=502, detail="provider failed")
monkeypatch.setattr(ai_client_module, "emit_business_log", fake_emit_business_log)
monkeypatch.setattr(AIProviderClient, "_request", fake_request)
client = AIProviderClient(service_url="http://provider.test", llm_config={"provider": "openai", "model": "gpt-test"})
with pytest.raises(HTTPException):
await client.analyze(
SituationalAnalysisRequest(title="T", objective="O", observations=["one"]),
request_id="req-ai-fail",
)
assert events[-1]["event"] == "ai.provider.analyze.failed"
assert events[-1]["level"] == "error"
assert events[-1]["context"]["error_type"] == "HTTPException"
@pytest.mark.asyncio
async def test_web_search_logs_query_hash_without_query(monkeypatch):
events = []
async def fake_emit_business_log(_logger, **payload):
events.append(payload)
async def fake_search_tavily(self, config, query, max_results, domains, freshness_days):
return [
SearchEvidence(
title="Example",
url="https://example.test",
snippet="result",
source_provider="tavily",
)
]
monkeypatch.setattr(web_search_module, "emit_business_log", fake_emit_business_log)
monkeypatch.setattr(WebSearchClient, "_search_tavily", fake_search_tavily)
client = WebSearchClient(
WebSearchConfig(
enabled=True,
default_provider="tavily",
providers={"tavily": WebSearchProviderConfig(provider="tavily", api_key="secret-key")},
)
)
results = await client.search("secret query text", max_results=1)
assert len(results) == 1
assert [event["event"] for event in events] == [
"ai_tool.web_search.start",
"ai_tool.web_search.success",
]
serialized = str(events)
assert "secret query text" not in serialized
assert "secret-key" not in serialized
assert events[0]["context"]["query_length"] == len("secret query text")
assert events[1]["context"]["result_count"] == 1

View File

@@ -1,9 +1,13 @@
"""Unit tests for data collectors"""
import json
import pytest
from unittest.mock import AsyncMock, patch
from app.core.datasource_defaults import DEFAULT_DATASOURCES
from app.services.collectors.celestrak import CelesTrakTLECollector
from app.services.collectors.downloads import DownloadHTTPStatusError, ResumableFileDownloader
from app.services.credential_guides import DEFAULT_CREDENTIAL_GUIDES
from app.services.collectors.top500 import TOP500Collector
from app.services.collectors.registry import collector_registry
@@ -149,6 +153,172 @@ class TestHTTPCollector:
assert callable(collector.parse_response)
class TestCelesTrakTLECollector:
def test_transform_uses_norad_as_source_id_and_preserves_starlink_group(self):
collector = CelesTrakTLECollector()
result = collector.transform([
{
"NORAD_CAT_ID": 44720,
"OBJECT_NAME": "STARLINK-1000",
"OBJECT_ID": "2019-029AZ",
"EPOCH": "2026-03-13T00:00:00Z",
"MEAN_MOTION": 15.79234567,
"ECCENTRICITY": 0.0001234,
"INCLINATION": 53.0,
"RA_OF_ASC_NODE": 10.0,
"ARG_OF_PERICENTER": 20.0,
"MEAN_ANOMALY": 30.0,
"_celestrak_query_group": "active",
"_celestrak_source_url": "https://celestrak.example/gp.php?GROUP=active&FORMAT=json",
}
])
assert result[0]["source_id"] == "44720"
assert result[0]["metadata"]["constellation_group"] == "starlink"
assert result[0]["metadata"]["celestrak_query_group"] == "active"
assert result[0]["metadata"]["norad_cat_id"] == 44720
assert result[0]["metadata"]["tle_line1"]
assert result[0]["metadata"]["tle_line2"]
def test_load_active_payload_rejects_invalid_records(self, tmp_path):
collector = CelesTrakTLECollector()
payload_path = tmp_path / "active.json"
payload_path.write_text(json.dumps([{"OBJECT_NAME": "missing norad"}]), encoding="utf-8")
with pytest.raises(RuntimeError, match="invalid record"):
collector._load_active_payload(payload_path)
def test_load_active_payload_accepts_complete_array(self, tmp_path):
collector = CelesTrakTLECollector()
payload_path = tmp_path / "active.json"
payload_path.write_text(
json.dumps([{"NORAD_CAT_ID": 25544, "OBJECT_NAME": "ISS (ZARYA)"}]),
encoding="utf-8",
)
records = collector._load_active_payload(payload_path)
assert records == [{"NORAD_CAT_ID": 25544, "OBJECT_NAME": "ISS (ZARYA)"}]
@pytest.mark.asyncio
async def test_fetch_retries_and_raises_instead_of_returning_partial_data(self, monkeypatch, tmp_path):
collector = CelesTrakTLECollector()
collector._resolved_url = "https://celestrak.example/NORAD/elements/gp.php"
attempts = 0
async def fake_download_file(*args, **kwargs):
nonlocal attempts
attempts += 1
raise RuntimeError("network interrupted")
async def fake_emit_business_log(*args, **kwargs):
return None
monkeypatch.setattr(collector._downloader, "download_file", fake_download_file)
monkeypatch.setattr("app.services.collectors.celestrak.emit_business_log", fake_emit_business_log)
monkeypatch.setattr("app.services.collectors.celestrak.asyncio.sleep", AsyncMock())
with pytest.raises(RuntimeError, match="failed after retries"):
await collector.fetch()
assert attempts == 3
@pytest.mark.asyncio
async def test_fetch_uses_cache_when_celestrak_reports_not_updated(self, monkeypatch, tmp_path):
collector = CelesTrakTLECollector()
collector._resolved_url = "https://celestrak.example/NORAD/elements/gp.php"
collector._downloader = ResumableFileDownloader(cache_namespace="celestrak-test", cache_root=tmp_path)
url = collector._active_url()
cached_path = collector._downloader.cached_file_path(url, ".json")
cached_path.parent.mkdir(parents=True, exist_ok=True)
cached_path.write_text(
json.dumps([{"NORAD_CAT_ID": 25544, "OBJECT_NAME": "ISS (ZARYA)"}]),
encoding="utf-8",
)
async def fake_download_file(*args, **kwargs):
raise DownloadHTTPStatusError(
url=url,
status_code=403,
body="GP data has not updated since your last successful download of GROUP=active.",
)
async def fake_emit_business_log(*args, **kwargs):
return None
monkeypatch.setattr(collector._downloader, "download_file", fake_download_file)
monkeypatch.setattr("app.services.collectors.celestrak.emit_business_log", fake_emit_business_log)
records = await collector.fetch()
assert records[0]["NORAD_CAT_ID"] == 25544
assert records[0]["_celestrak_query_group"] == "active"
@pytest.mark.asyncio
async def test_fetch_not_updated_without_cache_does_not_retry(self, monkeypatch, tmp_path):
collector = CelesTrakTLECollector()
collector._resolved_url = "https://celestrak.example/NORAD/elements/gp.php"
collector._downloader = ResumableFileDownloader(cache_namespace="celestrak-test", cache_root=tmp_path)
attempts = 0
async def fake_download_file(*args, **kwargs):
nonlocal attempts
attempts += 1
raise DownloadHTTPStatusError(
url=collector._active_url(),
status_code=403,
body="GP data has not updated since your last successful download of GROUP=active.",
)
async def fake_emit_business_log(*args, **kwargs):
return None
monkeypatch.setattr(collector._downloader, "download_file", fake_download_file)
monkeypatch.setattr("app.services.collectors.celestrak.FALLBACK_GROUPS", ("starlink",))
monkeypatch.setattr("app.services.collectors.celestrak.emit_business_log", fake_emit_business_log)
monkeypatch.setattr("app.services.collectors.celestrak.asyncio.sleep", AsyncMock())
with pytest.raises(RuntimeError, match="fallback group mode failed"):
await collector.fetch()
assert attempts == 2
@pytest.mark.asyncio
async def test_fetch_falls_back_to_all_groups_when_active_not_updated_without_cache(self, monkeypatch, tmp_path):
collector = CelesTrakTLECollector()
collector._resolved_url = "https://celestrak.example/NORAD/elements/gp.php"
collector._downloader = ResumableFileDownloader(cache_namespace="celestrak-test", cache_root=tmp_path)
payload_by_group = {
"starlink": [{"NORAD_CAT_ID": 100, "OBJECT_NAME": "STARLINK-100"}],
"gps-ops": [{"NORAD_CAT_ID": 200, "OBJECT_NAME": "GPS BIIR-2"}],
}
async def fake_download_file(_client, url, **_kwargs):
if "GROUP=active" in url:
raise DownloadHTTPStatusError(
url=url,
status_code=403,
body="GP data has not updated since your last successful download of GROUP=active.",
)
group = "starlink" if "GROUP=starlink" in url else "gps-ops"
path = tmp_path / f"{group}.json"
path.write_text(json.dumps(payload_by_group[group]), encoding="utf-8")
return path
async def fake_emit_business_log(*args, **kwargs):
return None
monkeypatch.setattr(collector._downloader, "download_file", fake_download_file)
monkeypatch.setattr("app.services.collectors.celestrak.FALLBACK_GROUPS", tuple(payload_by_group))
monkeypatch.setattr("app.services.collectors.celestrak.emit_business_log", fake_emit_business_log)
records = await collector.fetch()
assert [item["NORAD_CAT_ID"] for item in records] == [100, 200]
assert records[0]["_celestrak_query_group"] == "starlink"
assert records[1]["_celestrak_group"] == "gps-ops"
def test_aisstream_collector_is_registered():
collector = collector_registry.get("aisstream_vessels")

View File

@@ -1,9 +1,11 @@
import asyncio
import inspect
from datetime import datetime, timedelta, timezone
import pytest
from app.api.v1 import datasources as datasources_api
from app.models.datasource import DataSource
from app.models.task import CollectionTask
from app.services import data_jobs
from app.services import earth_layer_cache as earth_cache
@@ -100,6 +102,71 @@ def test_serialize_datasource_row_includes_endpoint_when_requested():
assert row["endpoint"] == "https://example.test/arcgis_landing_points"
def test_serialize_datasource_row_separates_collector_running_from_delete_task():
datasource = make_datasource(9, "top500", last_status="success")
task = CollectionTask(
id=44,
datasource_id=9,
source="top500",
task_type="clear_data",
status="running",
phase="clearing_data",
)
row = datasources_api.serialize_datasource_row(
datasource,
running_tasks={9: task},
latest_tasks={9: task},
record_counts={"top500": 500},
endpoint_overrides={},
config=object(),
include_endpoint=False,
)
assert row["is_task_active"] is True
assert row["is_running"] is False
assert row["task_type"] == "clear_data"
assert row["task_status"] == "running"
def test_cancel_queued_delete_task_finishes_immediately(monkeypatch):
task = CollectionTask(
id=45,
datasource_id=9,
source="top500",
task_type="clear_data",
status="queued",
phase="queued",
)
class FakeDb:
async def commit(self):
return None
async def refresh(self, _task):
return None
async def fake_broadcast(_task):
return None
monkeypatch.setattr(data_jobs, "_broadcast_task_update", fake_broadcast)
async def run():
return await data_jobs.request_cancel_datasource_task(FakeDb(), task)
result = asyncio.run(run())
assert result.status == "cancelled"
assert result.phase == "cancelled"
assert result.completed_at is not None
def test_clear_data_job_relies_on_db_outbox_instead_of_extra_earth_refresh_task():
source = inspect.getsource(data_jobs._run_clear_data_job)
assert "enqueue_earth_refresh_job" not in source
def test_invalidate_earth_layer_cache_for_source_covers_datasource_aliases(monkeypatch):
patterns: list[str] = []
@@ -119,34 +186,36 @@ def test_invalidate_earth_layer_cache_for_source_covers_datasource_aliases(monke
]
@pytest.mark.asyncio
async def test_trigger_datasource_batch_skips_disabled_and_frequency_window(monkeypatch):
now = datetime.now(timezone.utc)
disabled = make_datasource(1, "aisstream_vessels", is_active=False)
not_due = make_datasource(2, "telegeography_cables", last_run_at=now, frequency_minutes=120)
due = make_datasource(3, "ris_live_bgp", last_run_at=now - timedelta(hours=2))
triggered_sources: list[str] = []
def test_trigger_datasource_batch_skips_disabled_and_frequency_window(monkeypatch):
async def run():
now = datetime.now(timezone.utc)
disabled = make_datasource(1, "aisstream_vessels", is_active=False)
not_due = make_datasource(2, "telegeography_cables", last_run_at=now, frequency_minutes=120)
due = make_datasource(3, "ris_live_bgp", last_run_at=now - timedelta(hours=2))
queued_sources: list[str] = []
async def fake_running_tasks(_db, _ids):
return {}
async def fake_running_tasks(_db, _ids):
return {}
async def fake_latest_task_ids(_db, _ids):
return {}
async def fake_enqueue(_db, datasource, task_type, **_kwargs):
queued_sources.append(datasource.source)
return CollectionTask(id=100 + datasource.id, datasource_id=datasource.id, source=datasource.source, task_type=task_type, status="queued")
monkeypatch.setattr(datasources_api, "_load_latest_running_tasks", fake_running_tasks)
monkeypatch.setattr(datasources_api, "_load_latest_task_ids", fake_latest_task_ids)
monkeypatch.setattr(
datasources_api,
"run_collector_now",
lambda source: triggered_sources.append(source) or True,
)
monkeypatch.setattr(datasources_api, "_load_latest_running_tasks", fake_running_tasks)
monkeypatch.setattr(
datasources_api,
"enqueue_datasource_job",
fake_enqueue,
)
result = await datasources_api._trigger_datasource_batch(
object(),
[disabled, not_due, due],
force=False,
)
result = await datasources_api._trigger_datasource_batch(
object(),
[disabled, not_due, due],
force=False,
)
assert [item["source"] for item in result["triggered"]] == ["ris_live_bgp"]
assert {item["reason"] for item in result["skipped"]} == {"disabled", "within_frequency_window"}
assert triggered_sources == ["ris_live_bgp"]
assert [item["source"] for item in result["triggered"]] == ["ris_live_bgp"]
assert {item["reason"] for item in result["skipped"]} == {"disabled", "within_frequency_window"}
assert queued_sources == ["ris_live_bgp"]
asyncio.run(run())

View File

@@ -1,11 +1,15 @@
"""Docs Gatekeeper API tests."""
import re
from pathlib import Path
import pytest
from httpx import ASGITransport, AsyncClient
from app.api.v1 import docs as docs_api
from app.main import app
from app.models.user import User
from app.services.docs_gatekeeper import DOCS_METADATA
def make_user(role: str = "viewer", groups: list[str] | None = None) -> User:
@@ -43,22 +47,45 @@ async def test_public_catalog_only_for_anonymous_user():
assert response.status_code == 200
items = response.json()["items"]
assert {item["access"] for item in items} == {"public"}
assert {item["slug"] for item in items if item["lang"] == "zh"} == {
zh_items = [item for item in items if item["lang"] == "zh"]
assert [item["slug"] for item in zh_items] == [
"overview",
"quickstart",
"manual",
"quickstart",
"faq",
"location-pipeline-user",
}
]
@pytest.mark.asyncio
async def test_developer_catalog_includes_architecture_and_frontend_reference_docs():
response = await get_json(
"/api/v1/docs/catalog",
make_user(role="viewer", groups=["docs_developer"]),
)
assert response.status_code == 200
zh_slugs = {item["slug"] for item in response.json()["items"] if item["lang"] == "zh"}
zh_items = [item for item in response.json()["items"] if item["lang"] == "zh"]
assert [item["group"] for item in zh_items[:4]] == ["Overview", "Manual", "Manual", "Manual"]
assert zh_items[4]["group"] == "Architecture"
assert "platform-data-flows" in zh_slugs
assert "naming-glossary" in zh_slugs
assert "tactile-ui-components" in zh_slugs
@pytest.mark.asyncio
async def test_anonymous_can_read_public_doc():
response = await get_json("/api/v1/docs/zh/quickstart")
manual_response = await get_json("/api/v1/docs/zh/manual")
overview_response = await get_json("/api/v1/docs/zh/overview")
assert response.status_code == 200
assert response.json()["access"] == "public"
assert "快速开始" in response.json()["markdown"]
assert manual_response.status_code == 200
assert manual_response.json()["access"] == "public"
assert overview_response.status_code == 200
assert overview_response.json()["access"] == "public"
@pytest.mark.asyncio
@@ -83,10 +110,16 @@ async def test_developer_group_can_read_developer_but_not_admin_doc():
user = make_user(role="viewer", groups=["docs_developer"])
developer_response = await get_json("/api/v1/docs/zh/backend-collectors", user)
tactile_response = await get_json("/api/v1/docs/zh/tactile-ui-components", user)
glossary_response = await get_json("/api/v1/docs/zh/naming-glossary", user)
admin_response = await get_json("/api/v1/docs/zh/backend-system-service-control", user)
assert developer_response.status_code == 200
assert developer_response.json()["access"] == "docs_developer"
assert tactile_response.status_code == 200
assert tactile_response.json()["access"] == "docs_developer"
assert glossary_response.status_code == 200
assert glossary_response.json()["access"] == "docs_developer"
assert admin_response.status_code == 403
@@ -114,3 +147,29 @@ async def test_unknown_language_slug_and_path_traversal_do_not_read_files():
assert bad_lang.status_code == 404
assert bad_slug.status_code == 404
assert traversal.status_code == 404
def test_public_docs_markdown_links_do_not_create_missing_docs_routes():
repo_root = Path(__file__).resolve().parents[2]
technical_root = repo_root / "docs" / "technical"
registered_filenames = {entry.filename for entry in DOCS_METADATA}
problems: list[str] = []
for markdown_path in sorted(technical_root.glob("*/*.md")):
markdown = markdown_path.read_text(encoding="utf-8")
for match in re.finditer(r"\[([^\]]+)]\(([^)]+\.md(?:#[^)]+)?)\)", markdown):
label, href = match.group(1), match.group(2)
if href.startswith(("http://", "https://", "mailto:")):
continue
href_without_hash = href.split("#", 1)[0].replace("\\", "/")
filename = Path(href_without_hash).name
if "/docs/technical/" in href_without_hash:
if filename not in registered_filenames:
problems.append(f"{markdown_path.relative_to(repo_root)} links unregistered public doc {href!r} ({label})")
continue
if href_without_hash.endswith(".md"):
problems.append(f"{markdown_path.relative_to(repo_root)} links non-public markdown {href!r} ({label})")
assert problems == []

View File

@@ -0,0 +1,454 @@
import asyncio
import json
from app.services.earth_db_change_listener import (
EarthDbChangeDispatcher,
EarthDbChangeListener,
build_earth_update_from_db_payload,
normalize_asyncpg_dsn,
)
def test_normalize_asyncpg_dsn_strips_sqlalchemy_driver():
assert (
normalize_asyncpg_dsn("postgresql+asyncpg://postgres:postgres@localhost:5432/planet_db")
== "postgresql://postgres:postgres@localhost:5432/planet_db"
)
def test_build_earth_update_maps_known_sources_to_layers():
satellite_update = build_earth_update_from_db_payload(
{"source": "celestrak_tle", "operation": "DELETE", "entity_key": "25544"}
)
cable_update = build_earth_update_from_db_payload(
{"source": "arcgis_cables", "operation": "UPDATE", "entity_key": "cable-1"}
)
compute_update = build_earth_update_from_db_payload(
{"source": "top500", "operation": "DELETE", "entity_key": None}
)
landing_update = build_earth_update_from_db_payload(
{"source": "telegeography_landing", "operation": "DELETE", "entity_key": None}
)
assert satellite_update is not None
assert satellite_update["layers"] == ["satellites"]
assert cable_update is not None
assert cable_update["layers"] == ["cables"]
assert compute_update is not None
assert compute_update["layers"] == ["computeCenters"]
assert landing_update is not None
assert landing_update["layers"] == ["cables"]
assert landing_update["refresh_strategy"] == "clear_then_reload"
def test_build_earth_update_maps_derived_tables_to_layers():
bgp_update = build_earth_update_from_db_payload(
{"table": "bgp_anomalies", "source": "ris_live_bgp", "operation": "DELETE", "entity_key": "a1"}
)
compute_update = build_earth_update_from_db_payload(
{"table": "compute_center_locations", "source": "top500", "operation": "UPDATE", "entity_key": "top500_1"}
)
vessel_update = build_earth_update_from_db_payload(
{"table": "vessel_position", "operation": "DELETE", "entity_key": "123456789"}
)
assert bgp_update is not None
assert bgp_update["source"] == "ris_live_bgp"
assert bgp_update["table"] == "bgp_anomalies"
assert bgp_update["layers"] == ["bgp"]
assert bgp_update["refresh_strategy"] == "clear_then_reload"
assert compute_update is not None
assert compute_update["layers"] == ["computeCenters"]
assert compute_update["refresh_strategy"] == "reload"
assert vessel_update is not None
assert vessel_update["source"] == "vessel_position"
assert vessel_update["layers"] == ["vessels"]
def test_build_earth_update_maps_interactable_delete_to_delta():
update = build_earth_update_from_db_payload(
{
"table": "earth_interactables",
"operation": "DELETE",
"entity_keys": ["note-1"],
"records_processed": 1,
}
)
assert update is not None
assert update["entity"] == "interactable"
assert update["action"] == "deleted"
assert update["ids"] == ["note-1"]
assert update["layers"] == ["interactables"]
assert update["refresh_strategy"] == "delta"
def test_build_earth_update_ignores_unmapped_sources():
assert build_earth_update_from_db_payload({"source": "not_for_earth"}) is None
def test_dispatcher_debounces_same_source_notifications():
async def run():
broadcasts = []
invalidated = []
async def broadcast(payload):
broadcasts.append(payload)
def invalidate(source):
invalidated.append(source)
return 2
dispatcher = EarthDbChangeDispatcher(
broadcast_earth_update=broadcast,
invalidate_cache=invalidate,
debounce_seconds=10,
)
dispatcher.handle_payload(
{
"table": "collected_data",
"operation": "INSERT",
"source": "arcgis_cables",
"entity_key": "cable-1",
"occurred_at": "2026-05-22T00:00:00Z",
}
)
dispatcher.handle_payload(
{
"table": "collected_data",
"operation": "UPDATE",
"source": "arcgis_cables",
"entity_key": "cable-2",
"occurred_at": "2026-05-22T00:00:01Z",
}
)
await dispatcher.flush_all()
assert invalidated == ["arcgis_cables"]
assert len(broadcasts) == 1
assert broadcasts[0]["action"] == "database_changed"
assert broadcasts[0]["source"] == "arcgis_cables"
assert broadcasts[0]["layers"] == ["cables"]
assert broadcasts[0]["records_processed"] == 2
assert broadcasts[0]["operations"] == ["INSERT", "UPDATE"]
assert broadcasts[0]["entity_keys"] == ["cable-1", "cable-2"]
assert broadcasts[0]["cache_entries_invalidated"] == 2
asyncio.run(run())
def test_dispatcher_handles_delete_notification_as_earth_update():
async def run():
broadcasts = []
async def broadcast(payload):
broadcasts.append(payload)
dispatcher = EarthDbChangeDispatcher(
broadcast_earth_update=broadcast,
invalidate_cache=lambda source: 1,
debounce_seconds=10,
)
accepted = dispatcher.handle_notification(
json.dumps(
{
"table": "collected_data",
"operation": "DELETE",
"source": "celestrak_tle",
"entity_key": "sat:25544",
"occurred_at": "2026-05-22T00:00:00Z",
}
)
)
await dispatcher.flush_all()
assert accepted is True
assert len(broadcasts) == 1
assert broadcasts[0]["action"] == "database_changed"
assert broadcasts[0]["source"] == "celestrak_tle"
assert broadcasts[0]["layers"] == ["satellites"]
assert broadcasts[0]["operation"] == "DELETE"
assert broadcasts[0]["refresh_strategy"] == "clear_then_reload"
asyncio.run(run())
def test_dispatcher_debounces_bgp_derived_table_events_by_source():
async def run():
broadcasts = []
async def broadcast(payload):
broadcasts.append(payload)
dispatcher = EarthDbChangeDispatcher(
broadcast_earth_update=broadcast,
invalidate_cache=lambda source: 0,
debounce_seconds=10,
)
dispatcher.handle_payload(
{
"event_id": 200,
"table": "bgp_observations",
"operation": "DELETE",
"source": "ris_live_bgp",
"records_processed": 2,
}
)
dispatcher.handle_payload(
{
"event_id": 201,
"table": "bgp_anomalies",
"operation": "DELETE",
"source": "ris_live_bgp",
"records_processed": 3,
}
)
await dispatcher.flush_all()
assert len(broadcasts) == 1
assert broadcasts[0]["source"] == "ris_live_bgp"
assert broadcasts[0]["layers"] == ["bgp"]
assert broadcasts[0]["records_processed"] == 5
assert broadcasts[0]["refresh_strategy"] == "clear_then_reload"
asyncio.run(run())
def test_dispatcher_deduplicates_notify_and_outbox_by_event_id():
async def run():
broadcasts = []
async def broadcast(payload):
broadcasts.append(payload)
dispatcher = EarthDbChangeDispatcher(
broadcast_earth_update=broadcast,
invalidate_cache=lambda source: 0,
debounce_seconds=10,
)
payload = {
"event_id": 42,
"table": "collected_data",
"operation": "INSERT",
"source": "celestrak_tle",
"entity_key": "sat:42",
"occurred_at": "2026-05-23T00:00:00Z",
}
assert dispatcher.handle_payload(payload) is True
assert dispatcher.handle_payload(dict(payload)) is False
await dispatcher.flush_all()
assert len(broadcasts) == 1
assert broadcasts[0]["records_processed"] == 1
assert broadcasts[0]["entity_keys"] == ["sat:42"]
asyncio.run(run())
def test_dispatcher_flushes_continuous_events_at_max_wait():
async def run():
broadcasts = []
sleeps = []
async def broadcast(payload):
broadcasts.append(payload)
dispatcher = EarthDbChangeDispatcher(
broadcast_earth_update=broadcast,
invalidate_cache=lambda source: 0,
debounce_seconds=10,
max_wait_seconds=0.01,
)
dispatcher.handle_payload(
{
"event_id": 300,
"table": "collected_data",
"operation": "INSERT",
"source": "arcgis_cables",
"records_processed": 1,
}
)
await asyncio.sleep(0.02)
dispatcher.handle_payload(
{
"event_id": 301,
"table": "collected_data",
"operation": "UPDATE",
"source": "arcgis_cables",
"records_processed": 1,
}
)
await asyncio.sleep(0)
await dispatcher.flush_all()
assert len(broadcasts) == 1
assert broadcasts[0]["records_processed"] == 2
assert broadcasts[0]["debounce_ms"] >= 0
asyncio.run(run())
def test_dispatcher_fast_flushes_delete_clear_then_reload():
async def run():
broadcasts = []
async def broadcast(payload):
broadcasts.append(payload)
dispatcher = EarthDbChangeDispatcher(
broadcast_earth_update=broadcast,
invalidate_cache=lambda source: 0,
debounce_seconds=10,
max_wait_seconds=10,
)
dispatcher.handle_payload(
{
"event_id": 310,
"table": "collected_data",
"operation": "DELETE",
"source": "celestrak_tle",
"records_processed": 10,
}
)
await asyncio.sleep(0.08)
assert len(broadcasts) == 1
assert broadcasts[0]["source"] == "celestrak_tle"
assert broadcasts[0]["refresh_strategy"] == "clear_then_reload"
asyncio.run(run())
def test_outbox_rows_are_consumed_after_successful_flush():
async def run():
class FakeConnection:
def __init__(self):
self.consumed_ids = []
async def fetch(self, _query, _limit):
return [
{
"id": 501,
"payload": {
"event_id": 501,
"table": "collected_data",
"operation": "DELETE",
"source": "celestrak_tle",
"records_processed": 1,
},
}
]
async def execute(self, _query, consumed_ids):
self.consumed_ids.extend(consumed_ids)
broadcasts = []
async def broadcast(payload):
broadcasts.append(payload)
dispatcher = EarthDbChangeDispatcher(
broadcast_earth_update=broadcast,
invalidate_cache=lambda source: 0,
debounce_seconds=10,
)
listener = EarthDbChangeListener(
dsn="postgresql://example/db",
dispatcher=dispatcher,
)
connection = FakeConnection()
listener._connection = connection
await listener._poll_outbox()
assert connection.consumed_ids == [501]
assert len(broadcasts) == 1
asyncio.run(run())
def test_outbox_rows_remain_unconsumed_when_flush_fails():
async def run():
class FakeConnection:
def __init__(self):
self.consumed_ids = []
async def fetch(self, _query, _limit):
return [
{
"id": 601,
"payload": {
"event_id": 601,
"table": "collected_data",
"operation": "DELETE",
"source": "celestrak_tle",
"records_processed": 1,
},
}
]
async def execute(self, _query, consumed_ids):
self.consumed_ids.extend(consumed_ids)
async def broadcast(_payload):
raise RuntimeError("ws down")
dispatcher = EarthDbChangeDispatcher(
broadcast_earth_update=broadcast,
invalidate_cache=lambda source: 0,
debounce_seconds=10,
)
listener = EarthDbChangeListener(
dsn="postgresql://example/db",
dispatcher=dispatcher,
)
connection = FakeConnection()
listener._connection = connection
try:
await listener._poll_outbox()
except RuntimeError:
pass
assert connection.consumed_ids == []
asyncio.run(run())
def test_dispatcher_uses_aggregated_statement_record_count():
async def run():
broadcasts = []
async def broadcast(payload):
broadcasts.append(payload)
dispatcher = EarthDbChangeDispatcher(
broadcast_earth_update=broadcast,
invalidate_cache=lambda source: 0,
debounce_seconds=10,
)
dispatcher.handle_payload(
{
"event_id": 99,
"table": "collected_data",
"operation": "DELETE",
"source": "top500",
"records_processed": 100,
"entity_keys": ["top500:1", "top500:2"],
"occurred_at": "2026-05-23T00:00:00Z",
}
)
await dispatcher.flush_all()
assert len(broadcasts) == 1
assert broadcasts[0]["source"] == "top500"
assert broadcasts[0]["layers"] == ["computeCenters"]
assert broadcasts[0]["records_processed"] == 100
assert broadcasts[0]["entity_keys"] == ["top500:1", "top500:2"]
asyncio.run(run())

View File

@@ -0,0 +1,79 @@
from datetime import UTC, datetime
from app.models.earth_interactable import EarthInteractable
from app.services import earth_interactables
def make_interactable(**overrides):
values = {
"id": "poi-1",
"layer": "places",
"kind": "note",
"label": "Test POI",
"description": "A point on Earth",
"latitude": 30.25,
"longitude": 120.15,
"altitude": None,
"revision": 3,
"properties": {"owner": "test"},
"is_deleted": False,
"created_at": datetime(2026, 5, 22, 1, 0, tzinfo=UTC),
"updated_at": datetime(2026, 5, 22, 1, 5, tzinfo=UTC),
"deleted_at": None,
}
values.update(overrides)
return EarthInteractable(**values)
def test_interactable_event_uses_object_delta_contract():
record = make_interactable()
event = earth_interactables.build_interactable_event(
action="updated",
record=record,
)
assert event["entity"] == "interactable"
assert event["action"] == "updated"
assert event["layer"] == "places"
assert event["layers"] == ["interactables"]
assert event["ids"] == ["poi-1"]
assert event["revision"] == 3
assert event["item"]["latitude"] == 30.25
assert event["item"]["properties"] == {"owner": "test"}
def test_interactable_geojson_omits_deleted_records():
active = make_interactable(id="active")
deleted = make_interactable(id="deleted", is_deleted=True)
payload = earth_interactables.interactables_to_geojson([active, deleted])
assert payload["type"] == "FeatureCollection"
assert [feature["id"] for feature in payload["features"]] == ["active"]
assert payload["features"][0]["geometry"] == {
"type": "Point",
"coordinates": [120.15, 30.25],
}
def test_interactable_cache_invalidation_clears_layer_and_all(monkeypatch):
patterns = []
def fake_delete_pattern(pattern):
patterns.append(pattern)
return 1
monkeypatch.setattr(
earth_interactables.earth_layer_cache,
"delete_pattern",
fake_delete_pattern,
)
deleted = earth_interactables.invalidate_interactable_cache("places")
assert deleted == 2
assert patterns == [
"earth:layer:v1:interactables:layer:places*",
"earth:layer:v1:interactables:layer:all*",
]

View File

@@ -421,6 +421,62 @@ async def test_earth_news_payload_uses_fresh_database_items_without_rss(monkeypa
assert payload["stale"] is False
@pytest.mark.asyncio
async def test_earth_news_payload_keeps_current_items_and_all_cruise_items(monkeypatch):
class FakeDb:
execute = object()
current_item = ParsedNewsItem(
id="db:current",
title="Current region story",
summary="Current summary",
url="https://example.com/current",
source="Stored Source",
feed_name="Stored Feed",
feed_region="americas",
homepage_url="https://example.com",
published_at=datetime(2026, 5, 15, 3, 0, tzinfo=UTC),
)
cruise_item = ParsedNewsItem(
id="db:apac",
title="APAC story",
summary="APAC summary",
url="https://example.com/apac",
source="Stored Source",
feed_name="Stored Feed",
feed_region="asia-pacific",
homepage_url="https://example.com",
published_at=datetime(2026, 5, 15, 4, 0, tzinfo=UTC),
)
async def fake_get_earth_news_freshness(_db, *, active_region):
return 12, datetime.now(UTC)
async def fake_list_earth_news_items(_db, *, active_region, limit):
return [current_item]
async def fake_list_earth_news_cruise_items(_db, *, limit):
return [current_item, cruise_item]
async def fake_enqueue_target_location_job(_payload, **_kwargs):
return True
async def fail_fetch(_sources):
raise AssertionError("fresh database items should not fetch RSS")
monkeypatch.setattr("app.services.earth_news_store.get_earth_news_freshness", fake_get_earth_news_freshness)
monkeypatch.setattr("app.services.earth_news_store.list_earth_news_items", fake_list_earth_news_items)
monkeypatch.setattr("app.services.earth_news_store.list_earth_news_cruise_items", fake_list_earth_news_cruise_items)
monkeypatch.setattr("app.services.earth_news_queue.enqueue_target_location_job", fake_enqueue_target_location_job)
monkeypatch.setattr("app.services.earth_news._fetch_rss_items_for_sources", fail_fetch)
payload = await get_earth_news_payload(lat=35.0, lon=-100.0, db=FakeDb())
assert [item["id"] for item in payload["items"]] == ["db:current"]
assert [item["id"] for item in payload["cruise_items"]] == ["db:current", "db:apac"]
assert payload["cruise_items"][1]["region"] == "asia-pacific"
@pytest.mark.asyncio
async def test_earth_news_payload_initializes_empty_database_from_rss(monkeypatch):
db = object()

View File

@@ -4,8 +4,11 @@ import logging
from io import StringIO
import pytest
from app.core.logging import PlanetContextFilter, PlanetFormatter, get_logger
from app.core.request_context import set_request_id
from app.services import business_logs
def _capture_output(callback):
@@ -76,3 +79,43 @@ def test_structured_logger_redacts_sensitive_text_and_context():
assert "hunter2" not in output
assert "[REDACTED]" in output
assert '"safe": "visible"' in output
def test_business_context_redacts_nested_sensitive_values():
context = business_logs.build_business_context(
{
"provider": "openai",
"api_key": "sk-secret",
"nested": {
"token": "plain-token",
"safe": "visible",
},
}
)
assert context["api_key"] == "[REDACTED]"
assert context["nested"]["token"] == "[REDACTED]"
assert context["nested"]["safe"] == "visible"
@pytest.mark.asyncio
async def test_emit_business_log_persists_sanitized_system_event(monkeypatch):
events = []
async def fake_record_system_log(**payload):
events.append(payload)
monkeypatch.setattr(business_logs, "record_system_log", fake_record_system_log)
await business_logs.emit_business_log(
get_logger("tests.business"),
event="ai.provider.analyze.success",
message="AI request completed",
category="ai",
context={"model": "gpt-test", "api_key": "sk-secret"},
)
assert events[0]["event"] == "ai.provider.analyze.success"
assert events[0]["category"] == "ai"
assert events[0]["context"]["model"] == "gpt-test"
assert events[0]["context"]["api_key"] == "[REDACTED]"

View File

@@ -816,6 +816,27 @@ async def test_visualization_geo_summary_returns_counts(monkeypatch):
app.dependency_overrides.clear()
@pytest.mark.asyncio
async def test_empty_cable_layers_return_empty_feature_collections():
class _ScalarResult:
def scalars(self):
class _Scalars:
def all(self):
return []
return _Scalars()
class _FakeSession:
async def execute(self, _query):
return _ScalarResult()
cables = await visualization_api._build_cables_geojson(_FakeSession())
landing_points = await visualization_api._build_landing_points_geojson(_FakeSession())
assert cables == {"type": "FeatureCollection", "features": []}
assert landing_points == {"type": "FeatureCollection", "features": []}
@pytest.mark.asyncio
async def test_collect_location_endpoint_returns_candidates_for_known_record(monkeypatch):
def _fake_ror(query):

View File

@@ -15,12 +15,16 @@ services:
retries: 10
aiprovider:
image: ${AI_PROVIDER_IMAGE_NAME:-planet-aiprovider:latest}
build:
context: .
dockerfile: aiprovider/Dockerfile
args:
PYTHON_IMAGE: ${PYTHON_IMAGE:-python:3.14-slim}
UV_IMAGE: ${UV_IMAGE:-ghcr.io/astral-sh/uv:latest}
AI_PROVIDER_BUILD_FINGERPRINT: ${AI_PROVIDER_BUILD_FINGERPRINT:-unknown}
secrets:
- planet_uv_config
container_name: planet_aiprovider
ports:
- "8010:8010"
@@ -43,3 +47,7 @@ services:
volumes:
ollama_data:
secrets:
planet_uv_config:
file: ${PLANET_UV_CONFIG_FILE:-/dev/null}

View File

@@ -2,12 +2,16 @@ version: '3.8'
services:
aiprovider:
image: ${AI_PROVIDER_IMAGE_NAME:-planet-aiprovider:latest}
build:
context: .
dockerfile: aiprovider/Dockerfile
args:
PYTHON_IMAGE: ${PYTHON_IMAGE:-python:3.14-slim}
UV_IMAGE: ${UV_IMAGE:-ghcr.io/astral-sh/uv:latest}
AI_PROVIDER_BUILD_FINGERPRINT: ${AI_PROVIDER_BUILD_FINGERPRINT:-unknown}
secrets:
- planet_uv_config
env_file:
- ./aiprovider/.env
- ${PLANET_AI_PROVIDER_RUNTIME_ENV_FILE:-./aiprovider/.env}
@@ -43,3 +47,7 @@ services:
volumes:
postgres_data:
redis_data:
secrets:
planet_uv_config:
file: ${PLANET_UV_CONFIG_FILE:-/dev/null}

View File

@@ -2,12 +2,16 @@ version: '3.8'
services:
aiprovider:
image: ${AI_PROVIDER_IMAGE_NAME:-planet-aiprovider:latest}
build:
context: .
dockerfile: aiprovider/Dockerfile
args:
PYTHON_IMAGE: ${PYTHON_IMAGE:-python:3.14-slim}
UV_IMAGE: ${UV_IMAGE:-ghcr.io/astral-sh/uv:latest}
AI_PROVIDER_BUILD_FINGERPRINT: ${AI_PROVIDER_BUILD_FINGERPRINT:-unknown}
secrets:
- planet_uv_config
env_file:
- ./aiprovider/.env
- ${PLANET_AI_PROVIDER_RUNTIME_ENV_FILE:-./aiprovider/.env}
@@ -53,3 +57,7 @@ services:
volumes:
postgres_data:
redis_data:
secrets:
planet_uv_config:
file: ${PLANET_UV_CONFIG_FILE:-/dev/null}

View File

@@ -8,6 +8,119 @@ This project follows the repository versioning rule:
- `improvement` -> `+0.0.1`bugfix + 小功能混合)
- `bugfix` -> `+0.0.1`
## [0.66.3] — 2026-05-26
Released: 2026-05-26
### Highlights
- 修复控制台 Admin 动态导入时缺失 `../lib/utils` 导致 Vite 返回 500 的问题。
- 补齐前端依赖安装状态,确保 Markdown Mermaid 渲染器可解析 `mermaid` 包。
### Added / Fixed / Improved
- 新增 Admin 本地 `cn``formatNumber` 工具模块恢复布局、UI 组件和 Dashboard 的共享工具引用。
- 放开 `.gitignore``frontend/src/admin/lib` 的源码例外,避免 utility module 再次被全局 `lib/` 规则漏提交。
- 重启前端开发服务并验证 `/admin` 无 Vite overlay 和 console error。
---
## [0.66.2] — 2026-05-26
Released: 2026-05-26
### Highlights
- 统一中文界面和公开文档中的产品命名:`Earth` 显示为“智能星球”,`Admin` 显示为“控制台”,`Docs` 显示为“文档”。
- 修复文档手册、Docs catalog、控制台入口和智能星球设置中残留的中英混排标题。
### Added / Fixed / Improved
- 更新控制台侧边栏、智能星球系统设置、移动端抽屉和登录页品牌文案。
- 同步前后端 Docs metadata、中文技术文档标题、使用手册、快速开始、术语表和运维手册中的产品命名。
- 补充清理智能星球图层缓存、品牌配置、展示缓存等控制台 toast / dialog 文案。
---
## [0.66.1] — 2026-05-26
Released: 2026-05-26
### Highlights
- 修复 CelesTrak active 更新窗口内清库后无法恢复的问题,新增持久原始下载缓存和完整 fallback group mode。
- 避免数据源任务 WebSocket 与轮询同时完成时重复弹出采集失败 toast。
- `planet.sh destroy` 保留上游原始下载缓存,数据库清空后仍可用缓存重灌。
### Added / Fixed / Improved
- CelesTrak 403 `GP data has not updated` 会先复用 active 缓存;无 active 缓存时完整拉取 `starlink/gps-ops/galileo/glonass/beidou/leo/geo/iridium-next`,任一 group 缺失则整体失败,不保存 partial。
- 下载器缓存从 `/tmp` 迁到 `$PLANET_CACHE_DIR/downloads`,并保留 HTTP 错误响应正文以支持上游限频语义判断。
- 补充 CelesTrak cache/fallback 回归测试和中英文运维/卫星策略文档。
---
## [0.66.0] — 2026-05-26
Released: 2026-05-26
### Highlights
- 将 Admin 正式化为唯一控制台入口,移除旧 AntD 后台、Admin Next 路由痕迹和相关依赖。
- 引入 PostgreSQL 数据作业队列、Earth outbox 同步和可交互对象管线,让采集、清理和 Earth 刷新进入可追踪异步链路。
- 强化 AI、AI 工具、设置连接测试和采集任务的结构化业务日志,关键事件可在系统日志中检索。
- CelesTrak 轨道根数采集改为完整 `active` 目录下载、续传和重试,避免部分分组失败时保存不完整卫星数据。
### Added / Fixed / Improved
- Admin 数据源页修复内置源调度启停按钮,使用 `is_active` 判断 `/enable``/disable`,避免采集状态误导调度开关。
- Earth 新增通用 interactables 层和平台数据流文档,支持对象级 delta 同步与后续小图层扩展。
- `planet.sh init/start/destroy` 补齐 uv 镜像回退、Bun unzip 依赖、HTTPS/LAN 跳转协议和 OOBE 清理语义。
- Markdown code block、数据源队列、演示模式、About 版本展示、TV 链接协议和文档索引同步完成。
---
## [0.65.2] — 2026-05-22
Released: 2026-05-22
### Highlights
- Stabilize AI Provider rebuild detection so unchanged source no longer rebuilds just because file timestamps or local cache state changed.
- Keep `uv.lock` on the official registry while still allowing local and Docker builds to use the user's `uv.toml`.
- Prevent local startup and bootstrap commands from rewriting `uv.lock` when a user-level mirror is configured.
### Added / Fixed / Improved
- AI Provider fingerprints now use content hashes, base image inputs, dependency metadata, and an image label instead of file mtimes.
- `planet.sh` and `scripts/bootstrap-dev.sh` now use `uv sync --frozen`, and backend startup uses `uv run --frozen`.
- Docker Compose passes the AI Provider fingerprint into the image build so future starts can inspect the image label directly.
---
## [0.65.1] — 2026-05-22
Released: 2026-05-22
### Highlights
- 修复 `planet.sh` 与 Docker Compose 对 AI Provider 镜像名不一致导致的重复构建问题。
- 让本地 `uv` 与 Docker build 统一使用用户机器上的 `uv.toml`,避免把镜像源写入 `uv.lock`
- 通过 BuildKit secret 向容器构建传入 uv 配置,保留用户源选择且不把配置写入镜像层。
### Added / Fixed / Improved
- `planet.sh` 自动解析 `UV_CONFIG_FILE`、仓库 `uv.toml` 与用户级 `~/.config/uv/uv.toml`,并清理会覆盖配置的 `UV_INDEX_URL` 环境变量。
- AI Provider 和后端 Dockerfile 在 `uv sync` 阶段挂载 uv 配置 secret并复用 BuildKit uv cache。
- Docker Compose 三套配置统一声明 `planet-aiprovider:latest` 镜像名和 `planet_uv_config` build secret。
---
## [0.65.0] — 2026-05-21
Released: 2026-05-21
### Highlights
- 收敛 Admin Next 数据源触发入口:主按钮在未勾选时触发全部,勾选内置源后切换为“触发已选 N”并移除手填 ID 的批量触发弹窗。
- 优化数据源采集队列入口:右上角按钮常驻,空态显示队列图标,有任务时显示纯圆环进度,队列改为浮层避免挤压表格。
- 强化 `planet.sh destroy` 清理语义,销毁时先硬重置运行中的本地 Postgres `public` schema避免残留采集数据让 OOBE 误判 ready。
### Added / Fixed / Improved
- Admin Next 表格新增可选选择列,仅在 `/datasources` 内置源分区启用,支持当前可见行全选并在筛选、切分区或刷新时清空选择。
- 数据源批量触发复用 `/datasources/trigger-batch``source_ids`,成功后写入现有采集队列并清空勾选。
- `destroy` 补充清理 `planet-aiprovider:latest` 镜像以及 Python/Vite 等本地编译缓存,同时保留源码和 `.env`
- Docs Gatekeeper 与 Tactile UI 文档/样式继续补齐覆盖本轮按钮、队列、OOBE 和销毁流程说明。
---
## [0.64.0] — 2026-05-21
Released: 2026-05-21

View File

@@ -16,7 +16,24 @@
- 已被正式实现替代、继续放在 `docs/` 根目录会误导后续开发的计划,归档
- 仍然指导未来开发、尚未完成或仍有明确执行价值的文档,继续保留在 `docs/`
当前替代入口:
- 业务和数据产品链路见 [业务架构与数据流转](/home/ray/dev/linkong/planet/docs/technical/zh/platform-data-flows.md)。
- 当前后端作业、outbox 和 Earth 同步实现见 [数据作业与 Outbox 技术架构](/home/ray/dev/linkong/planet/docs/technical/zh/data-job-earth-sync-architecture.md)。
- 用户操作流程见 [Planet 使用手册](/home/ray/dev/linkong/planet/docs/technical/zh/manual.md)。
- 当前代码结构见 [技术文档索引](/home/ray/dev/linkong/planet/docs/technical/zh/README.md)。
补充说明:
- 一部分归档文档来自外部或临时工作流草案,例如 sisyphus 生成的初稿
- 这类文档如果有可用内容,应先吸收到 `docs/plans/``docs/technical/`,再归档保留来源记录
## 近期归档
- [Docs Gatekeeper 鉴权系统计划](/home/ray/dev/linkong/planet/docs/deprecated/docs-gatekeeper-auth-plan.md):已落地,当前实现见技术文档。
- [Location Resolver 共享管线计划](/home/ray/dev/linkong/planet/docs/deprecated/location-resolver-shared-pipeline-plan.md):已落地,当前实现见技术文档。
- [Earth Surface Hover Info Plan](/home/ray/dev/linkong/planet/docs/deprecated/earth-surface-hover-info-plan.md):已实现,保留为历史记录。
- [Admin Next Dual Track Full Migration Plan](/home/ray/dev/linkong/planet/docs/deprecated/admin-next-dual-track-full-migration-plan.md):已被当前 `frontend/src/admin/` 控制台结构替代。
- [Admin Next Soft Glass Goal Driven Plan](/home/ray/dev/linkong/planet/docs/deprecated/admin-next-soft-glass-goal-driven-plan.md):已被当前 `frontend/src/admin/` 控制台结构替代。
- [Admin Next Parity Checklist](/home/ray/dev/linkong/planet/docs/deprecated/admin-next-parity-checklist.md):旧 `/admin-next/*` 验收资料,保留为迁移历史。
- [Admin Next Parity Audit Closeout](/home/ray/dev/linkong/planet/docs/deprecated/admin-next-parity-audit-closeout-plan.md):旧 `/admin-next/*` 审计资料,保留为迁移历史。

View File

@@ -104,7 +104,7 @@ The old AntD page itself treated these Earth content tabs as placeholder-level c
- `models_3d`
- `news_anchor_strategy`
If backend endpoints are later added, these items must be promoted into `docs/plans/admin-next-parity-checklist.md` with concrete API and UI acceptance criteria.
If backend endpoints are later added, these items must be promoted into a new active plan under `docs/plans/` with concrete API and UI acceptance criteria.
## Final Gate
@@ -114,7 +114,7 @@ After the route promotion, the final gate is no longer “switch old routes.”
2. Run the static checks:
- `rg "map: \\(\\) => \\[\\]|暂不支持保存|placeholder" frontend/src/admin-next`
- `rg "ShadowPage|FeatureConsole|GlassPanel|InspectorDrawer" frontend/src/admin-next`
3. Manually verify every official route listed in `docs/plans/admin-next-parity-checklist.md`.
3. Manually verify every official route listed in `docs/deprecated/admin-next-parity-checklist.md`.
4. Confirm `/legacy/admin/*` still opens old AntD pages during the validation window.
5. Delete old AntD pages and remove AntD dependencies only as a separate final cleanup task after explicit confirmation.

View File

@@ -8,7 +8,7 @@ The redesign must cover desktop and mobile. Data display, icon semantics, table
## Criteria For Success
- This plan exists at `docs/plans/admin-next-soft-glass-goal-driven-plan.md`.
- This archived plan exists at `docs/deprecated/admin-next-soft-glass-goal-driven-plan.md`.
- `/admin-next/*` has real pages for every route; route usage of `ShadowPage` is removed.
- Admin Next supports `system`, `light`, and `dark` theme modes using the same persistence and system-theme idea as Docs.
- The visual language reads as soft-glass / light-neumorphic instead of an AntD reskin: translucent panels, fine borders, subtle glow, cool backgrounds, restrained accent colors, crisp icons, and tactile controls.
@@ -214,4 +214,3 @@ Manual viewport checks:
- low height;
- 125% / 150% browser zoom;
- light / dark / system theme modes.

View File

@@ -1,6 +1,6 @@
# Location Resolver Shared Pipeline Plan
**状态**:已实现,当前用户流程见 [Earth 位置候选采集使用手册](/home/ray/dev/linkong/planet/docs/technical/zh/location-pipeline-user.md),开发接口见 [通用位置估算管线开发说明](/home/ray/dev/linkong/planet/docs/technical/zh/location-pipeline-development.md)。
**状态**:已实现,当前用户流程见 [Planet 使用手册](/home/ray/dev/linkong/planet/docs/technical/zh/manual.md) 的 Earth 位置候选采集章节,开发接口见 [通用位置估算管线开发说明](/home/ray/dev/linkong/planet/docs/technical/zh/location-pipeline-development.md)。
## Goal

View File

@@ -37,7 +37,7 @@ If the same action has both a UI and a CLI path (e.g. user creation), describe t
- a `credential_provider` in `backend/app/core/datasource_defaults.py`;
- a default credential guide in `backend/app/services/credential_guides.py`;
- a supported connectivity provider in `backend/app/services/datasource_connectivity.py`;
- settings UI guidance or a credential form in `frontend/src/pages/Settings/Settings.tsx`;
- settings UI guidance or a credential form in `frontend/src/admin/pages/PlainResourcePages.tsx`;
- a regression test that fails if the guide/provider is missing.
## Recommended Checks

View File

@@ -7,6 +7,12 @@
- 分几期做
- 当前差距和下一步是什么
当前实现、业务架构和数据链路不放在这里。它们分别进入:
- [业务架构与数据流转](/home/ray/dev/linkong/planet/docs/technical/zh/platform-data-flows.md)
- [数据作业与 Outbox 技术架构](/home/ray/dev/linkong/planet/docs/technical/zh/data-job-earth-sync-architecture.md)
- [技术文档索引](/home/ray/dev/linkong/planet/docs/technical/zh/README.md)
适合放入这里的内容:
- Earth / BGP / 地形 / 天球实施方案
@@ -16,38 +22,40 @@
当前重点入口:
- [earth-mobile-drawer-ui-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-mobile-drawer-ui-plan.md)
- [earth-compute-center-bgp-style-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-compute-center-bgp-style-plan.md)
- [earth-renderer-architecture-separation-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-renderer-architecture-separation-plan.md)
- [Earth Mobile Drawer UI Plan](/home/ray/dev/linkong/planet/docs/plans/earth-mobile-drawer-ui-plan.md)
- [Earth Compute Center BGP Style Plan](/home/ray/dev/linkong/planet/docs/plans/earth-compute-center-bgp-style-plan.md)
- [Earth Renderer Architecture Separation Plan](/home/ray/dev/linkong/planet/docs/plans/earth-renderer-architecture-separation-plan.md)
- [Earth 高精度国界静态瓦片计划](/home/ray/dev/linkong/planet/docs/plans/earth-high-precision-boundary-tiles-plan.md)
- [earth-predicted-orbit-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-predicted-orbit-plan.md)
- [earth-webgl-instancing-satellites-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-webgl-instancing-satellites-plan.md)
- [earth-real-terrain-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-real-terrain-plan.md)
- [earth-news-source-configuration-and-collector-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-news-source-configuration-and-collector-plan.md)
- [earth-news-cruise-summary-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-news-cruise-summary-plan.md)
- [Earth Predicted Orbit Plan](/home/ray/dev/linkong/planet/docs/plans/earth-predicted-orbit-plan.md)
- [Earth WebGL Instancing Satellites Plan](/home/ray/dev/linkong/planet/docs/plans/earth-webgl-instancing-satellites-plan.md)
- [Earth Real Terrain Plan](/home/ray/dev/linkong/planet/docs/plans/earth-real-terrain-plan.md)
- [Earth News Source Configuration And Collector Plan](/home/ray/dev/linkong/planet/docs/plans/earth-news-source-configuration-and-collector-plan.md)
- [Earth News Cruise Summary Plan](/home/ray/dev/linkong/planet/docs/plans/earth-news-cruise-summary-plan.md)
- [Earth 动作捕捉手势控制计划](/home/ray/dev/linkong/planet/docs/plans/earth-motion-capture-gesture-control-plan.md)
- [Earth 动捕交互语义 V2 计划](/home/ray/dev/linkong/planet/docs/plans/earth-motion-gesture-interaction-v2-plan.md)
- [Earth Presentation 解耦架构计划](/home/ray/dev/linkong/planet/docs/plans/earth-presentation-decoupled-architecture-plan.md)
- [earth-vessel-rendering-performance-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-vessel-rendering-performance-plan.md)
- [Earth Vessel Rendering Performance Plan](/home/ray/dev/linkong/planet/docs/plans/earth-vessel-rendering-performance-plan.md)
- [AIS 多源采集、冲突记录与聚合接口计划](/home/ray/dev/linkong/planet/docs/plans/earth-vessel-ais-aggregation-plan.md)
- [earth-interactable-layer-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-interactable-layer-plan.md)
- [frontend-public-docs-site-plan.md](/home/ray/dev/linkong/planet/docs/plans/frontend-public-docs-site-plan.md)
- [Docs Gatekeeper 鉴权系统计划](/home/ray/dev/linkong/planet/docs/plans/docs-gatekeeper-auth-plan.md)
- [Location Resolver 共享管线计划](/home/ray/dev/linkong/planet/docs/plans/location-resolver-shared-pipeline-plan.md)
- [frontend-ai-playground-development-plan.md](/home/ray/dev/linkong/planet/docs/plans/frontend-ai-playground-development-plan.md)
- [Earth Interactable Layer Plan](/home/ray/dev/linkong/planet/docs/plans/earth-interactable-layer-plan.md)
- [Frontend Public Docs Site Plan](/home/ray/dev/linkong/planet/docs/plans/frontend-public-docs-site-plan.md)
- [Frontend AI Playground Development Plan](/home/ray/dev/linkong/planet/docs/plans/frontend-ai-playground-development-plan.md)
- [AI Provider OpenClaw-Style Routing Plan](/home/ray/dev/linkong/planet/docs/plans/ai-provider-openclaw-style-routing-plan.md)
- [统一集成配置 Schema 系统计划](/home/ray/dev/linkong/planet/docs/plans/integration-config-schema-system-plan.md)
- [Lightweight Agent Orchestrator 与 WebSearch 证据层计划](/home/ray/dev/linkong/planet/docs/plans/agents-light-orchestrator-websearch-plan.md)
- [Admin Next Parity Checklist](/home/ray/dev/linkong/planet/docs/plans/admin-next-parity-checklist.md)
- [Admin Next Parity Audit Closeout](/home/ray/dev/linkong/planet/docs/plans/admin-next-parity-audit-closeout-plan.md)
- [ue5-mvp-fused-plan.md](/home/ray/dev/linkong/planet/docs/plans/ue5-mvp-fused-plan.md)
- [UE5 MVP Fused Plan](/home/ray/dev/linkong/planet/docs/plans/ue5-mvp-fused-plan.md)
历史计划入口:
- 已完成、已替代或只作为历史决策背景保留的文档,统一放在 [Deprecated Docs](/home/ray/dev/linkong/planet/docs/deprecated/README.md)。
不适合放入这里的内容:
- 当前代码结构说明
- 组件现状和实现入口
- 已经落地的技术上下文说明
- 已经成为当前行为的“架构说明”
这些应放入:
- [docs/technical/README.md](/home/ray/dev/linkong/planet/docs/technical/README.md)
- [技术文档索引](/home/ray/dev/linkong/planet/docs/technical/zh/README.md)
- [Technical Docs Index](/home/ray/dev/linkong/planet/docs/technical/en/README.md)

View File

@@ -178,5 +178,5 @@ Runtime 选择规则:
- `backend/app/services/ai_client.py`
- `aiprovider/main.py`
- `aiprovider/provider_service.py`
- `frontend/src/admin-next/pages/PlainResourcePages.tsx`
- `docs/plans/admin-next-parity-audit-closeout-plan.md`
- `frontend/src/admin/pages/PlainResourcePages.tsx`
- `docs/deprecated/admin-next-parity-audit-closeout-plan.md`

View File

@@ -1,46 +1,52 @@
# Technical Docs
This directory holds "current implementation and current structure" documentation, focusing on:
This is the current Intelligent Planet documentation entry point. Docs are organized by reader path: start with business architecture to understand data products, then move into user manuals or implementation references.
- How the code is organized right now
- Where the current entry points are
- How state and components work
- Which implementation boundaries future changes should follow
## Business Architecture
What belongs here:
- [Business Architecture and Data Flows](/home/ray/dev/linkong/planet/docs/technical/en/platform-data-flows.md): purpose, collection flow, fact tables, derived tables, cache, and WebSocket broadcast path for each Earth data product
- [Naming Glossary](/home/ray/dev/linkong/planet/docs/technical/en/naming-glossary.md): English/Chinese terms used across the console, Earth, backend, and docs
- Quickstart and user manual
- Frontend context
- Earth frontend structure
- Earth satellite footprint policy
- Earth render layer order
- Earth layer style property index
- Backend runtime control
- Collector status
- Collector settings and connectivity validation
- Earth Interactable integration
- Collection format conventions
## Manual
## Entry Points
- [Quickstart](/home/ray/dev/linkong/planet/docs/technical/en/quickstart.md): shortest path to getting Intelligent Planet running from scratch
- [Intelligent Planet Manual](/home/ray/dev/linkong/planet/docs/technical/en/manual.md): user workflows for the console, Earth, Docs, and common features
- [FAQ](/home/ray/dev/linkong/planet/docs/technical/en/faq.md): troubleshooting for Windows / WSL, ports, dependencies, motion capture, credentials, and Docs permissions
- [Quickstart](/home/ray/dev/linkong/planet/docs/technical/en/quickstart.md): The shortest path to getting Planet running from scratch
- [Planet Manual](/home/ray/dev/linkong/planet/docs/technical/en/manual.md): Complete usage guide for the console, `planet.sh`, Earth, and Docs
- [FAQ](/home/ray/dev/linkong/planet/docs/technical/en/faq.md): Central troubleshooting entry for Windows / WSL, ports, dependencies, motion capture, credentials, and Docs permissions
- [Earth Location Candidate Collection User Guide](/home/ray/dev/linkong/planet/docs/technical/en/location-pipeline-user.md): Collect and preview coordinate candidates for compute centers and BGP collectors on Earth
- [Collector Settings and Connectivity Validation](/home/ray/dev/linkong/planet/docs/technical/en/datasource-collector-settings-connectivity.md): Data source catalog, collector settings, connectivity validation, and BarentsWatch credentials
- [Shared Location Resolution Pipeline Development Guide](/home/ray/dev/linkong/planet/docs/technical/en/location-pipeline-development.md): Backend location resolver / pipeline interfaces, registries, and extension points
- [Docs Gatekeeper Development Guide](/home/ray/dev/linkong/planet/docs/technical/en/docs-gatekeeper-development.md): Backend Docs catalog, Markdown content loading, and Gatekeeper permission groups
- [Naming Glossary](/home/ray/dev/linkong/planet/docs/technical/en/naming-glossary.md): English/Chinese term mapping for the console, Earth, backend, and docs
- [Earth Interactable Usage](/home/ray/dev/linkong/planet/docs/technical/en/earth-interactable-usage.md): API, lifecycle, and integration examples for Earth surface icon Interactable
- [Earth Toolbar and Overlay Coordination](/home/ray/dev/linkong/planet/docs/technical/en/earth-toolbar-overlay-coordination.md): Closing matrix and integration rules for toolbar buttons, search, settings, news, and layer overlays
- [Tactile UI Components](/home/ray/dev/linkong/planet/docs/technical/en/tactile-ui-components.md): Portable button, switch, tooltip, and scrollbar APIs, theme tokens, and migration rules
## Earth Implementation
What does not belong here:
- [Earth Frontend Context](/home/ray/dev/linkong/planet/docs/technical/en/earth-frontend-context.md): Earth modules, state, WebSocket refresh, and layer lifecycle
- [Earth Layer Style Reference](/home/ray/dev/linkong/planet/docs/technical/en/earth-layer-style-reference.md): layer colors, symbols, materials, and visual parameters
- [Earth Render Layer Order](/home/ray/dev/linkong/planet/docs/technical/en/earth-render-layer-order.md): renderOrder, depth strategy, picking, and collision avoidance
- [Earth Satellite Footprint Policy](/home/ray/dev/linkong/planet/docs/technical/en/earth-satellite-footprint-policy.md): satellite footprint display boundaries and strategy
- [BGP Context](/home/ray/dev/linkong/planet/docs/technical/en/earth-bgp-context.md): BGP rendering, aggregation, and collector implementation in Earth
- [Earth Interactable Usage](/home/ray/dev/linkong/planet/docs/technical/en/earth-interactable-usage.md): `Interactable` API, lifecycle, and integration examples
- [Earth Toolbar and Overlay Coordination](/home/ray/dev/linkong/planet/docs/technical/en/earth-toolbar-overlay-coordination.md): close matrix for toolbar buttons, search, settings, news, and layer overlays
- Incomplete roadmaps
- Future iteration plans
- Large-scale refactor proposals
## Frontend Implementation
Those belong in:
- [Admin Frontend Context](/home/ray/dev/linkong/planet/docs/technical/en/frontend-admin-frontend-context.md): console page structure, state boundaries, and lazy loading
- [Frontend Layout Guidelines](/home/ray/dev/linkong/planet/docs/technical/en/frontend-layout-guidelines.md): layout, tables, panels, and responsive constraints
- [Tactile UI Components](/home/ray/dev/linkong/planet/docs/technical/en/tactile-ui-components.md): button, switch, tooltip, scrollbar APIs, and theme tokens
- [Plans Index](/home/ray/dev/linkong/planet/docs/plans/README.md)
## Backend Implementation
- [Data Collectors](/home/ray/dev/linkong/planet/docs/technical/en/backend-collectors.md): collectors, task types, save layer, and status updates
- [System Service Control](/home/ray/dev/linkong/planet/docs/technical/en/backend-system-service-control.md): backend service control and system operation APIs
- [Datasource Collector Settings and Connectivity](/home/ray/dev/linkong/planet/docs/technical/en/datasource-collector-settings-connectivity.md): datasource catalog, collector settings, connectivity validation, and credentials
- [Datasource API Performance](/home/ray/dev/linkong/planet/docs/technical/en/backend-datasources-api-performance.md): DataSources list API performance and caching
- [Data Jobs and Outbox Architecture](/home/ray/dev/linkong/planet/docs/technical/en/data-job-earth-sync-architecture.md): PostgreSQL job queue, outbox, listener, and Kafka / Spark evolution boundaries
- [Shared Location Resolution Pipeline Development Guide](/home/ray/dev/linkong/planet/docs/technical/en/location-pipeline-development.md): location resolver / pipeline interfaces, registries, and extension points
- [News Live Streams Collector Format](/home/ray/dev/linkong/planet/docs/technical/en/earth-news-live-streams-collector-format.md): news, live stream, and media collection payload conventions
- [Docs Gatekeeper Development Guide](/home/ray/dev/linkong/planet/docs/technical/en/docs-gatekeeper-development.md): backend Docs catalog, Markdown content loading, and Gatekeeper permission groups
## Agents and Operations
- [AI Provider Guide](/home/ray/dev/linkong/planet/docs/technical/en/agents-aiprovider.md): model provider adapters, task prompts, and invocation boundaries
- [Intelligent Planet Ops Runbook](/home/ray/dev/linkong/planet/docs/technical/en/ops-runbook.md): deployment, startup, troubleshooting, and sensitive operations
- [Docker + Compose + Buildx Upgrade](/home/ray/dev/linkong/planet/docs/technical/en/ops-docker-compose-buildx-upgrade.md): Docker toolchain upgrade steps
- [planet.sh Startup](/home/ray/dev/linkong/planet/docs/technical/en/ops-planet-sh-startup.md): startup script, health checks, and performance optimization
## Plans and History
Incomplete roadmaps, large refactor proposals, and future plans live in the repository path `docs/plans/README.md`. Completed, outdated, or replaced designs live in `docs/deprecated/README.md`.

View File

@@ -101,7 +101,7 @@ The AI settings page uses:
These endpoints require an authenticated user. The `secrets` endpoint is only used when the settings page reveals a key or token; hiding the field restores the masked preview.
Admin Next keeps the AI page aligned with the legacy information architecture:
Admin keeps the AI page aligned with the legacy information architecture:
- `Model Providers`
- Manages provider, wire adapter, default model, LLM API key, proxy URL, proxy token, model refresh, set-as-default, and lightweight connectivity testing.
@@ -272,7 +272,7 @@ Each provider has its own key slot. Resolution order is:
`.env` is only a fallback. After the settings page saves successfully, or after the connection test succeeds, PostgreSQL becomes the global default source.
Admin Next must compute key status per provider or tool:
Admin must compute key status per provider or tool:
- If the database has a key for the current provider/tool, show `configured`.
- If the database has no key but the fallback provider, model, or tool matches the current item, show the fallback masked preview.
@@ -283,7 +283,7 @@ Tool keys follow the same rule. WebSearch and OCR must match the current tool an
### Lightweight Connectivity Testing
The Admin Next plug button performs a lightweight connectivity check and does not save configuration. Common API-platform practice is two-tiered:
The Admin plug button performs a lightweight connectivity check and does not save configuration. Common API-platform practice is two-tiered:
- Check a provider catalog or low-cost endpoint to validate base URL, authentication, and model reachability.
- Send full model requests only when the user explicitly runs Playground or a business task.

View File

@@ -73,6 +73,8 @@ async def run(self, db):
**Core file**: `backend/app/services/collectors/base.py`
Manual trigger, data clearing, and cache clearing now enter the PostgreSQL data job queue. `collection_tasks` remains the task ledger. Collectors only own `fetch -> transform -> save`; the `data_jobs.py` worker claims `collect` / `clear_data` / `clear_cache` / `earth_refresh` jobs and writes progress back. Earth layer refresh relationships live in `earth_layer_adapters.py`; do not hand-code cache invalidation or WebSocket broadcasts inside individual collectors or buttons.
## III. Collector List
| Collector | Data type | Content | Frequency |
@@ -93,7 +95,7 @@ Earth boundaries are no longer data collectors. They are Earth static rendering
TOP500 and Epoch AI compute sources do not always provide usable coordinates. The unified Earth compute-center endpoint uses only valid source-provided coordinates or `compute_center_locations` dimension-table coordinates during the main map startup path; records without coordinates are returned as `unresolved` instead of being rendered from a local registry, country centroid, or guessed city. When users manually collect candidates, the backend queries ROR and Nominatim/OpenStreetMap from source fields; accepted candidates are saved into `compute_center_locations` and rendered from that table on the next layer refresh.
Admin Next collection management follows the business hierarchy instead of flattening every endpoint into one table:
Admin collection management follows the business hierarchy instead of flattening every endpoint into one table:
- `Collectors`: endpoint, authentication, headers, base parameters, enabled state, and credential guides.
- `Collection Schedule`: scheduler state and task controls.

View File

@@ -96,4 +96,4 @@ if (res.data.task_id) {
## Related Files
- [datasources.py](/home/ray/dev/linkong/planet/backend/app/api/v1/datasources.py): `_load_datasource_list_context`, `list_datasources`
- [DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/DataSources/DataSources.tsx): `BuiltInDataSource`, `triggerDatasource`
- [DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx): `BuiltInDataSource`, `triggerDatasource`

View File

@@ -0,0 +1,141 @@
# Data Jobs and Outbox Architecture
This document records the technical boundary for Planet v1 data jobs, database outbox, and Earth refresh. For each data product's business purpose and end-to-end flow, see [Business Architecture and Data Flows](/home/ray/dev/linkong/planet/docs/technical/en/platform-data-flows.md).
## Architecture Boundary
- PostgreSQL is the durable v1 job ledger and outbox. Kafka, Celery, and RQ are intentionally not part of v1.
- `collection_tasks` records collection, data clearing, cache clearing, and non-database Earth refresh jobs.
- `earth_data_change_events` records fact-table or derived-table changes and is the reliable source for Earth sync.
- `LISTEN/NOTIFY` is only the low-latency wakeup path; the listener still polls unconsumed outbox rows.
- Redis is mainly cache, auth helper, OTP / rate limit, temporary logs, and WebSocket support. It is not the durable queue.
## Database Change Sync
```mermaid
flowchart LR
Write["Fact or derived table write"] --> Trigger["PostgreSQL trigger"]
Trigger --> Outbox["earth_data_change_events"]
Trigger --> Notify["planet_earth_data_changes"]
Outbox --> Listener["earth_db_change_listener"]
Notify --> Listener
Listener --> Adapter["earth_layer_adapters"]
Adapter --> Cache["cache invalidation"]
Cache --> WS["earth_updates"]
```
1. Collection, clearing, location resolution, or projection jobs write fact or derived tables.
2. Statement triggers write outbox rows for `INSERT` / `UPDATE` / `DELETE`.
3. The listener wakes through notify or finds pending rows through polling.
4. The listener maps `table + source` to Earth layers, refresh strategy, and cache patterns through `earth_layer_adapters.py`.
5. The listener merges short-window same-layer events, invalidates cache, and broadcasts `earth_updates`.
6. The outbox row is marked consumed only after successful broadcast; failed rows stay retryable.
DB changes no longer create default `earth_refresh` jobs, so they are not blocked by long same-source collection or clearing jobs. `earth_refresh` remains for manual cache clearing and non-DB refresh hints.
## Data Job Queue
`collection_tasks` is the unified job ledger. Workers claim `queued` jobs with PostgreSQL `FOR UPDATE SKIP LOCKED`; write jobs for the same `source` run serially, while different sources may run in parallel.
| task_type | Purpose |
| --- | --- |
| `collect` | Run a built-in datasource collector |
| `clear_data` | Delete collected rows and declared derived rows for the source |
| `clear_cache` | Delete Earth / dashboard cache for the source |
| `earth_refresh` | Invalidate Earth layer cache and broadcast a refresh hint for non-DB changes |
API handlers only create jobs and return `task_id`. Execution, progress, cancellation, and terminal state are written back by workers and pushed to the frontend through the `datasource_tasks` channel.
Cancellation means “keep committed batches”: clicking stop marks the job as `cancelling` and cancels the in-memory coroutine. Already committed batches remain; unfinished batches follow the collector or cleanup rollback path.
## Earth Sync Event Model
The unified event model is `earth.layer.changed`:
```json
{
"event": "earth.layer.changed",
"action": "database_changed",
"source": "celestrak_tle",
"table": "collected_data",
"operation": "DELETE",
"layers": ["satellites"],
"refresh_strategy": "clear_then_reload",
"records_processed": 11125,
"occurred_at": "2026-05-25T10:20:30Z"
}
```
| strategy | Purpose |
| --- | --- |
| `clear_then_reload` | Clear local frontend layer objects first, then force a refetch. Prefer this for deletes. |
| `reload` | Keep old objects until fresh data returns. Use it for location, metadata, or non-destructive updates. |
| `delta` | Used only for `earth_interactables`; upsert or remove objects by id. |
APIs must return HTTP 200 with an empty collection for real zero-data states; 5xx is reserved for real endpoint failures. After a delete event, if refetch fails, the frontend should keep the cleared state and show a lightweight error instead of restoring stale objects.
## Layer Adapter Contract
`earth_layer_adapters.py` is the single registry for sources, derived tables, Earth layers, cache patterns, and refresh strategy. New layers should be added through an adapter entry, not through one-off button handlers, collector branches, or frontend special cases.
Each adapter must declare:
- Which source or table feeds which Earth layer.
- Which Earth cache key patterns must be invalidated.
- Which owned derived tables must be removed during `clear_data`.
- The default refresh strategy for that layer.
When a source is cleared, the `clear_data` job first deletes `collected_data.source = <source>`, then deletes adapter-owned derived rows. Direct derived-table edits also trigger the outbox, so background jobs, admin APIs, and SQL repair scripts reach Earth as long as they mutate fact or derived tables.
## Operations and Troubleshooting
Check whether outbox rows are piling up:
```sql
SELECT id, table_name, operation, source, occurred_at
FROM earth_data_change_events
WHERE consumed_at IS NULL
ORDER BY id
LIMIT 20;
```
Check that triggers exist:
```sql
SELECT tgname, tgrelid::regclass
FROM pg_trigger
WHERE tgname LIKE 'tr_planet_%_changed_%'
ORDER BY 2, 1;
```
Useful log events:
- `earth.db_changes.connected`: the listener connected to PostgreSQL and started listening.
- `earth.db_changes.outbox_polled`: polling found unconsumed outbox rows.
- `earth.db_changes.broadcasted`: an Earth refresh broadcast was produced.
- `data_job.started` / `data_job.completed`: job execution state.
If Earth does not update, check in order: fact table changed, outbox was consumed, adapter covers the `source/table`, the listener is online, frontend WebSocket is connected, and the visualization API returns HTTP 200 with either an empty collection or fresh data.
## Kafka-ready Boundaries
Business code avoids depending on a concrete queue implementation by preserving these boundaries:
- `JobQueue`: submit, claim, cancel, and complete data jobs.
- `DataChangeBus`: publish database fact changes.
- `EarthLayerAdapterRegistry`: declare source, layer, cache, and derived-data relationships.
Kafka becomes appropriate when:
- Several independent services must consume the same data-change stream.
- AIS, BGP, or sensor streams become sustained high-throughput inputs.
- Consumer groups, replay, and service decoupling are required.
Spark becomes appropriate when:
- Historical data reaches tens or hundreds of millions of rows and PostgreSQL aggregation becomes expensive.
- Cross-source, long-window, spatiotemporal analysis is needed.
- Raw data lands in Parquet / Iceberg / Delta and the system starts producing offline derived data products.
For second-level continuous stream processing, evaluate Flink first. Spark is a better fit for batch or micro-batch analytics.

View File

@@ -32,8 +32,8 @@ If endpoint, headers, base configuration, or credential fingerprint changes afte
Files:
- [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/pages/PlainResourcePages.tsx)
- [AdminNextRoutes.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/AdminNextRoutes.tsx)
- [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
- [AdminRoutes.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/AdminRoutes.tsx)
Current behavior:
@@ -57,7 +57,7 @@ Current behavior:
File:
- [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/pages/PlainResourcePages.tsx)
- [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
Current behavior:

View File

@@ -2,7 +2,7 @@
Docs Gatekeeper moves `/docs` from "bundle all Markdown into the frontend" to "return catalog and content from the backend according to permissions." Its goal is to keep public manuals, user docs, developer docs, and admin/ops docs in one searchable Docs page while making every protected Markdown body pass through a server-side whitelist and authorization check.
For the user workflow, see the Docs section in [Planet Manual](/home/ray/dev/linkong/planet/docs/technical/en/manual.md).
For the user workflow, see the Docs section in [Intelligent Planet Manual](/home/ray/dev/linkong/planet/docs/technical/en/manual.md).
## Authorization Model
@@ -61,15 +61,15 @@ DocsMetadata(
"public",
"Manual",
2,
"Planet 使用手册",
"Planet Manual",
"智能星球使用手册",
"Intelligent Planet Manual",
)
```
When adding a public technical doc:
When adding a technical doc that should appear in the Docs page:
- Add both Chinese and English Markdown files.
- Add filename, slug, access, group, order, and titles to server `DOCS_METADATA`.
- Add filename, slug, access, group, order, and titles to server `DOCS_METADATA`. The backend catalog endpoint is authoritative; frontend metadata alone does not publish a document into `/docs` navigation.
- Add matching metadata to frontend [docs-content.ts](/home/ray/dev/linkong/planet/frontend/src/pages/Docs/docs-content.ts) so navigation titles and sorting stay aligned.
- Update `docs/technical/zh/README.md` and `docs/technical/en/README.md` when the document should be discoverable from the README.
@@ -83,7 +83,7 @@ The user API:
- Validates group names on update: only `docs_user`, `docs_developer`, and `docs_admin` are accepted.
- Allows only `super_admin` to modify Gatekeeper groups.
Frontend [Users.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Users/Users.tsx) displays group tags and provides a multi-select in the edit form. Non-`super_admin` users see the field disabled, and submission removes `gatekeeper_groups` before sending.
Frontend [Users.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/Users.tsx) displays group tags and provides a multi-select in the edit form. Non-`super_admin` users see the field disabled, and submission removes `gatekeeper_groups` before sending.
## Frontend Docs Loading

View File

@@ -188,7 +188,7 @@ Current reality:
- that is expected, because incidents are aggregated and de-noised
- but incident-first rendering makes the Earth view look too quiet unless there is another always-available activity layer
Implementation detail for the recommended `activity layer` is expanded in the [BGP Region Aggregation Plan](/home/ray/dev/linkong/planet/docs/plans/earth-bgp-region-aggregation-plan.md).
Implementation detail for the recommended `activity layer` is kept in the repository path `docs/plans/earth-bgp-region-aggregation-plan.md`.
So the immediate next milestone is:
@@ -259,7 +259,7 @@ Reference inspiration:
Relevant page:
- `frontend/src/pages/BGP/BGP.tsx`
- `frontend/src/admin/pages/PlainResourcePages.tsx`
Current BGP console page has three levels:
@@ -321,7 +321,7 @@ Backend:
Frontend:
- `frontend/src/pages/BGP/BGP.tsx`
- `frontend/src/admin/pages/PlainResourcePages.tsx`
- `frontend/public/earth/js/bgp.js`
- `frontend/public/earth/js/main.js`
- `frontend/public/earth/js/info-card.js`

View File

@@ -4,7 +4,7 @@ This document describes the current real structure of the Earth display frontend
Related references:
- [Project Rules](/home/ray/dev/linkong/planet/rules.md)
- Repository root `rules.md`
- [Frontend Layout Guidelines](/home/ray/dev/linkong/planet/docs/technical/en/frontend-layout-guidelines.md)
## Current Goal
@@ -56,6 +56,8 @@ Responsibilities:
- Layer module integration
- Earth-level state synchronization
Earth treats `/ws` `earth_updates` as refresh hints only; real data is fetched again from `/api/v1/visualization/...`. Backend database-driven refresh now has the listener clear cache and broadcast directly instead of going through the default `earth_refresh` job queue. After `database_changed`, the frontend applies the per-layer `clear_then_reload`, `reload`, or `delta` strategy. `clear_then_reload` must clear Three.js objects before a no-store refetch, and summary is only a consistency check, not a reason to skip a layer reload when the count is `0`. See [Data Jobs and Outbox Architecture](/home/ray/dev/linkong/planet/docs/technical/en/data-job-earth-sync-architecture.md) for the technical pipeline and [Business Architecture and Data Flows](/home/ray/dev/linkong/planet/docs/technical/en/platform-data-flows.md) for business data flows.
### 3. Earth Control Layer
- [controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js)
@@ -151,11 +153,11 @@ Each module is responsible for its own:
`brand.js` manages Earth HUD brand resources. Static assets provide the default brand; runtime overrides come from `/api/v1/earth/brand`, and uploaded images are served from `/earth-brand-assets/...`. The frontend must treat logo/title images and text fallback separately: if an image fails, show the text title; if text fields are empty, rely on backend defaults so the HUD brand area never renders blank. The console Earth Content page owns saving and resetting brand configuration; the Earth frontend only consumes it.
`about.js` manages the About card inside Earth settings. Frontend defaults remain as a fallback, while runtime content is loaded from `/api/v1/earth/about`. If the request fails or fields are missing, the renderer must fall back per field so the settings page never renders an empty card. Admin Next exposes an Earth Content `About` tab; saving uses `PUT /api/v1/earth/about`, and restoring defaults uses `DELETE /api/v1/earth/about`.
`about.js` manages the About card inside Earth settings. Frontend defaults remain as a fallback, while runtime content is loaded from `/api/v1/earth/about`. If the request fails or fields are missing, the renderer must fall back per field so the settings page never renders an empty card. Admin exposes an Earth Content `About` tab; saving uses `PUT /api/v1/earth/about`, and restoring defaults uses `DELETE /api/v1/earth/about`.
`oobe.js` manages the first-run Earth initialization guide. OOBE visibility must be driven by `/api/v1/earth/oobe-status` and its `ready` field, not by `localStorage`. `localStorage` may only store a short-lived "skip on this browser" flag; if the backend reports `ready: true`, logout, cleared browser storage, or a different browser must not show OOBE again. Desktop uses a dark starfield scrim and glass startup panel, while mobile uses a bottom sheet and respects `prefers-reduced-motion`.
The Admin Next Earth Content page must preserve runtime semantics:
The Admin Earth Content page must preserve runtime semantics:
- `Brand`: brand preview should use the same dark starfield background, size, spacing, logo/title rendering, and text fallback as the Earth HUD top-left brand block, not a generic form preview.
- `About`: configures the About card in Earth settings, including logo, kicker, title, version, description, and metadata items. Earth runtime reads `/earth/about` and falls back to defaults on failure.

View File

@@ -17,26 +17,21 @@ Related context:
## Current Local Categories
Current CelesTrak satellite groups in [backend/app/services/collectors/celestrak.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/celestrak.py) include:
The CelesTrak collector now prefers the complete active satellite catalog from `GROUP=active&FORMAT=json`. This prevents one failed CelesTrak group request from being saved as a successful but incomplete batch. The collector only proceeds when the downloaded JSON is a parseable array and records include `NORAD_CAT_ID`; network, resume, or parsing failures are retried, and final failure preserves the previous current dataset.
- `starlink`
- `gps-ops`
- `galileo`
- `glonass`
- `beidou`
- `leo`
- `geo`
- `iridium-next`
CelesTrak applies a repeat-download window to large groups such as `active`. To make database resets recoverable, Planet stores raw downloads under `$PLANET_CACHE_DIR/downloads`. When `active` returns the CelesTrak "GP data has not updated" HTTP 403:
Non-Starlink categories:
- If an `active` cache exists, the collector reuses it to repopulate the database.
- If no `active` cache exists, the collector switches to fallback group mode and downloads `starlink`, `gps-ops`, `galileo`, `glonass`, `beidou`, `leo`, `geo`, and `iridium-next`.
- Fallback group mode requires every group to succeed or have a reusable cache; if any group is missing, the whole collection fails and no partial dataset is saved.
- `gps-ops`
- `galileo`
- `glonass`
- `beidou`
- `leo`
- `geo`
- `iridium-next`
The collector still provides `metadata.constellation_group` to the frontend, but the value now comes from executable inference:
- `OBJECT_NAME` starting with `STARLINK` is marked as `starlink`
- `OBJECT_NAME` starting with `IRIDIUM` is marked as `iridium-next`
- Other active satellites are not forced into the old CelesTrak small-group labels, because a broad source group is not an exact constellation
The product policy therefore still discusses GNSS/RNSS, GEO, generic LEO, and Iridium NEXT semantics. However, only fallback group mode stores the old `gps-ops`, `galileo`, `glonass`, `beidou`, `leo`, or `geo` labels; the active primary path does not force every satellite into those labels.
## Research Conclusions
@@ -136,7 +131,8 @@ This implementation only does the minimum executable version and does not change
1. Backend passes constellation group and footprint policy hint to the frontend
- CelesTrak collector stores `GROUP` in `metadata.constellation_group`
- CelesTrak collector stores the source query in `metadata.celestrak_query_group = active`
- `metadata.constellation_group` only stores inferred business constellations such as `starlink` and `iridium-next`
- Visualization API outputs:
- `properties.constellation_group`
- `properties.footprint_policy`

View File

@@ -4,7 +4,7 @@ This document describes the current real structure of the console frontend. The
Related references:
- [Project Rules](/home/ray/dev/linkong/planet/rules.md)
- Repository root `rules.md`
- [Frontend Layout Guidelines](/home/ray/dev/linkong/planet/docs/technical/en/frontend-layout-guidelines.md)
## Current Goal
@@ -22,7 +22,7 @@ Main entry point:
- [App.tsx](/home/ray/dev/linkong/planet/frontend/src/App.tsx)
Admin Next now owns the official admin routes:
Admin now owns the official admin routes:
- `/admin`
- `/users`
@@ -37,23 +37,7 @@ Admin Next now owns the official admin routes:
- `/collection-management`
- `/settings`
These routes render [AdminNextRoutes.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/AdminNextRoutes.tsx). Page metadata and menu entries come from [manifest.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/routes/manifest.tsx). `/admin-next/*` remains only as a compatibility entry and redirects to the official route; it is no longer a parallel primary entry.
The old AntD console remains available under `/legacy/admin/*` for comparison and rollback:
- `/legacy/admin`
- `/legacy/admin/datasources`
- `/legacy/admin/data`
- `/legacy/admin/collection-management`
- `/legacy/admin/earth-content`
- `/legacy/admin/ai`
- `/legacy/admin/logs`
- `/legacy/admin/settings`
- `/legacy/admin/users`
- `/legacy/admin/bgp`
- `/legacy/admin/alerts/*`
Legacy pages, `AppLayout`, `antd`, and `@ant-design/icons` stay in place during the legacy validation window. Do not remove them before Admin Next parity is accepted.
These routes render [AdminRoutes.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/AdminRoutes.tsx). Page metadata and menu entries come from [manifest.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/routes/manifest.tsx). Admin is the only console entry point; there is no parallel console or rollback route.
`/earth` is a standalone display page and is not part of the console shell.
@@ -61,7 +45,7 @@ Legacy pages, `AppLayout`, `antd`, and `@ant-design/icons` stay in place during
The official admin shell is at:
- [AdminNextLayout.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/components/layout/AdminNextLayout.tsx)
- [AdminLayout.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/components/layout/AdminLayout.tsx)
Responsibilities:
@@ -69,38 +53,13 @@ Responsibilities:
- Current account, version, logout, and theme switching
- Top search, breadcrumbs, and page shortcuts
- Single-screen content-area height closure
- Coordination for Admin Next internal scrolling, tables, detail panels, and mobile detail views
- Coordination for Admin internal scrolling, tables, detail panels, and mobile detail views
The old AntD legacy shell remains at:
Future official admin pages should adapt to `AdminLayout` and Admin page patterns; do not reintroduce a parallel admin shell.
- [AppLayout.tsx](/home/ray/dev/linkong/planet/frontend/src/components/AppLayout/AppLayout.tsx)
## Admin Section Loading
Legacy responsibilities:
- Left-side navigation
- Collapse and expand
- Current account / version information
- Content area height closure
- Site-wide unified sidebar scrollbar
Current structure:
```tsx
<Layout className="dashboard-layout">
<Sider className="dashboard-sider">...</Sider>
<Layout>
<Content className="dashboard-content">
<div className="dashboard-content-inner">{children}</div>
</Content>
</Layout>
</Layout>
```
Future official admin pages should adapt to `AdminNextLayout` and Admin Next page patterns rather than adding new capability to the old `AppLayout`. Only `/legacy/admin/*` maintenance should change the old shell.
## Admin Next Section Loading
Multi-tab pages are currently coordinated by [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/pages/PlainResourcePages.tsx), which hosts the current management and information workbench patterns. Section loading follows these rules:
Multi-tab pages are currently coordinated by [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx), which hosts the current management and information workbench patterns. Section loading follows these rules:
- Initial page load requests only the active tab; it does not prefetch every tab endpoint.
- Switching tabs lazily loads that tab. Loaded tabs stay cached in local `states`, so returning to a tab reuses the previous data.
@@ -112,19 +71,22 @@ This keeps Earth, AI, collection management, and other multi-section pages from
## Datasource Collection Queue
The Admin Next datasource page routes single-source trigger, batch trigger, and trigger-all into a browser-download-list style collection queue:
The Admin datasource page routes single-source trigger, table-selected trigger, and trigger-all into a browser-download-list style collection queue:
- Queue state is managed by [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/pages/PlainResourcePages.tsx). It is a current-session visibility layer and does not fake task history in `localStorage`.
- Queue state is managed by [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx). It is a current-session visibility layer and does not fake task history in `localStorage`.
- Progress first consumes the `/ws` `datasource_tasks` channel. If the socket is unavailable or stale, the page polls `/api/v1/datasources/{id}/task-status`.
- Trigger responses immediately insert `triggered`, `skipped`, and `failed` items. After a refresh, the queue restores only real backend rows that are still `running`, `pending`, or `queued`.
- The page shows a compact progress bar and counts above the table. The "view queue" action opens a bottom panel grouped by running, failed, completed, and skipped.
- The `Built-in Sources` section uses the table selection column for selected-source triggering. With no rows selected, the primary button is `Trigger All`; after selection, the same button becomes `Trigger Selected N`, replacing the old manual-ID batch button.
- The expanded queue no longer lives in the page content flow, so trigger-all cannot squeeze the table and detail panel. The top-right actions area uses the existing `Button` styling; the empty state shows a `ListChecks` icon, and active queues show only a pure circular total-progress indicator. Clicking it opens a floating panel grouped by running, failed, completed, and skipped.
- Queue rows can jump to the datasource detail panel, and failed rows can retry. The detail panel's task summary only reports the selected source's latest task; it does not save configuration.
- While a single-source collection is running, the action button becomes `Stop Collection`; unfinished queue rows expose a cancel action on the right. Cancellation calls `/api/v1/datasources/{source_id}/tasks/{task_id}/cancel`; the backend keeps committed batches and rolls back unfinished work.
- Clearing database data and clearing display cache enter the same queue. The frontend shows task state and does not assume the API completed synchronously.
This queue is a user-perception layer. Backend task status remains the only source of truth for running, completion, failure, and skipped decisions.
## Admin Next Theme Slider
## Admin Theme Slider
The Admin Next sidebar theme switcher still reuses shared [SegmentedControl.tsx](/home/ray/dev/linkong/planet/frontend/src/components/SegmentedControl/SegmentedControl.tsx), while [styles.css](/home/ray/dev/linkong/planet/frontend/src/admin-next/styles.css) overrides the segment variables by `data-theme`:
The Admin sidebar theme switcher still reuses shared [SegmentedControl.tsx](/home/ray/dev/linkong/planet/frontend/src/components/SegmentedControl/SegmentedControl.tsx), while [styles.css](/home/ray/dev/linkong/planet/frontend/src/admin/styles.css) overrides the segment variables by `data-theme`:
- Light mode uses `--d-segment-bg: #eef3f9`, a white slider, and a light external shadow.
- Dark mode uses the same dark base, `#202938` slider, and dark external shadow semantics as Docs.
@@ -163,7 +125,7 @@ Purpose:
Current usage:
- Admin Next data sources, collected data, collection management, logs, alerts, and BGP pages
- Admin data sources, collected data, collection management, logs, alerts, and BGP pages
- Old AntD legacy pages continue using shared scrolling behavior through compatibility wrappers
### 3. `TableScrollRegion`
@@ -187,7 +149,7 @@ Files:
Purpose:
- Admin Next global tool buttons and detail-panel toolbars
- Admin global tool buttons and detail-panel toolbars
- Icon-only ordinary actions with tooltips
- Strong-intent actions such as save, create, confirm, delete, and stop
- Compact switches aligned with the Docs theme slider
@@ -245,39 +207,19 @@ Current constraints:
- Internal document links should be converted to `/docs/:slug` through `transformLink`
- Heading anchors are injected through `getHeadingId`, keeping route state outside the renderer
### 7. `ConnectionTestInput`
### 7. Admin UI Primitives
File:
Files:
- [ConnectionTestInput.tsx](/home/ray/dev/linkong/planet/frontend/src/components/ConnectionTestInput/ConnectionTestInput.tsx)
- [button.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/components/ui/button.tsx)
- [dialog.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/components/ui/dialog.tsx)
- [switch.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/components/ui/switch.tsx)
Purpose:
Use:
- Console form fields that combine an endpoint/Base URL value with a connection check
- Connection-test entry points for AI Provider and WebSearch
- Future collector configuration fields should reuse it when the test action belongs inside the input
Current constraints:
- The input suffix shows a single plug/connector icon, not an adjacent text button
- Disabled integrations must grey out both the input and its connection-test action
- The component only combines the input and action; callers still own form state, loading, disabled state, and the request itself
### 8. `TableActions`
File:
- [TableActions.tsx](/home/ray/dev/linkong/planet/frontend/src/components/TableActions/TableActions.tsx)
Purpose:
- Shared action entry for table operation columns
- Shows inline actions when expanded
- Uses a more-actions dropdown when collapsed
Companion export:
- `actionCellProps`: for action-column `onCell`, preventing action buttons from being ellipsized or wrapped
- Global tool buttons, detail actions, confirmation dialogs, and binary settings.
- Aligned with Tactile UI tokens so Admin controls keep consistent size, hover, disabled, and dark-mode behavior.
- Row actions should prefer icon buttons plus tooltip/title; do not reintroduce a separate action-menu component.
## Current State Sources
@@ -300,7 +242,7 @@ Responsibilities:
File:
- [AISettings.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/AISettings/AISettings.tsx)
- [AISettings.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
Responsibilities:
@@ -336,7 +278,7 @@ Constraints:
Example:
- [Dashboard.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Dashboard/Dashboard.tsx)
- [Dashboard.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/Dashboard.tsx)
Priority goals:
@@ -348,10 +290,10 @@ Priority goals:
Examples:
- [DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/DataSources/DataSources.tsx)
- [DataList.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/DataList/DataList.tsx)
- [Users.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Users/Users.tsx)
- [Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Settings/Settings.tsx)
- [DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
- [DataList.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/DataList.tsx)
- [Users.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/Users.tsx)
- [Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
Constraints:
@@ -361,7 +303,7 @@ Constraints:
### Datasource Directory Page
[DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/DataSources/DataSources.tsx) is the datasource directory and collection operation page. It should not grow back into a configuration editor.
[DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx) is the datasource directory and collection operation page. It should not grow back into a configuration editor.
Current page boundary:
@@ -375,7 +317,9 @@ Keep this boundary: do not put custom datasource editing, built-in endpoint over
### Collectors Page
[Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Settings/Settings.tsx) has three route modes: `/settings` for System Settings, `/earth-content` for Earth Content, and `/collection-management` for Collection Management. The `collector_credentials` tab is shown as `Collectors` under `/collection-management`.
[Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx) has three route modes: `/settings` for System Settings, `/earth-content` for Earth Content, and `/collection-management` for Collection Management. The `collector_credentials` tab is shown as `Collectors` under `/collection-management`.
The `System Display` section under `/settings` includes the `Demo Mode` switch. When enabled, Earth OOBE ignores existing current collected data and the local `browse first` temporary skip state, then opens the initialization guide directly. This switch is only for demos and acceptance checks; it does not change datasources, collection queues, or Earth content resources.
Current boundary:
@@ -387,7 +331,7 @@ Current boundary:
### Earth Content Page
`/earth-content` reuses the same single-screen tab container from [Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Settings/Settings.tsx), but its ownership is separate from System Settings:
`/earth-content` reuses the same single-screen tab container from [Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx), but its ownership is separate from System Settings:
- `TV Livestream` owns the Earth media-panel source configuration.
- `Boundary Precision` owns the Earth static boundary asset state: provider, low-precision fallback, high-precision manifest/PMTiles, source JSON, and build action.
@@ -399,8 +343,8 @@ Do not add Earth experience resources or collection-lifecycle tabs back into `/s
Examples:
- [BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx)
- [Playground.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Playground/Playground.tsx)
- [BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
- [Playground.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
Constraints:

View File

@@ -8,7 +8,8 @@ Admin pages in this project default to a "single-screen workspace" layout standa
Current recommended reference implementations:
- [frontend/src/pages/BGP/BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx)
- [frontend/src/admin/pages/PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
- [frontend/src/admin/pages/DataList.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/DataList.tsx)
- [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css)
## Core Principles
@@ -23,15 +24,15 @@ Admin pages default to:
Recommended structure:
```tsx
<AppLayout>
<AdminLayout>
<div className="page-shell">
<div className="page-shell__header">...</div>
<div className="page-shell__body">...</div>
</div>
</AppLayout>
</AdminLayout>
```
Total page height should be bounded within the `AppLayout` content area, not allowed to grow naturally downward without limit.
Total page height should be bounded within the `AdminLayout` content area, not allowed to grow naturally downward without limit.
### 2. Scrolling Should Happen Inside Modules

View File

@@ -2,7 +2,7 @@
`backend/app/services/location/` is the shared abstraction for any "given a record, decide its lat/lon" workflow. Compute centers, BGP collectors, and BGP events now run on this pipeline. Future entities such as satellite ground stations, user-claimed points, and IXP facilities should plug in here instead of creating another geocoding path.
For the user workflow, see [Earth Location Candidate Collection User Guide](/home/ray/dev/linkong/planet/docs/technical/en/location-pipeline-user.md).
For the user workflow, see the Earth coordinate-candidate section in [Intelligent Planet Manual](/home/ray/dev/linkong/planet/docs/technical/en/manual.md).
## Design Goals

View File

@@ -1,8 +1,8 @@
# Planet Manual
# Intelligent Planet Manual
This manual is for Planet end users. Starting from the browser, it covers account registration, login, configuring collectors, configuring AI, using Earth and the console, and reading the docs site. Every action happens in a browser.
This manual is for Intelligent Planet end users. Starting from the browser, it covers account registration, login, configuring collectors, configuring AI, using Intelligent Planet and the console, and reading the docs site. Every action happens in a browser.
If you are responsible for deployment or on-call duty, read the [Planet Ops Runbook](/home/ray/dev/linkong/planet/docs/technical/en/ops-runbook.md) instead — it covers shell commands, log paths, and CLI fallbacks for user creation.
If you are responsible for deployment or on-call duty, read the [Intelligent Planet Ops Runbook](/home/ray/dev/linkong/planet/docs/technical/en/ops-runbook.md) instead — it covers shell commands, log paths, and CLI fallbacks for user creation.
## Entry Overview
@@ -233,7 +233,7 @@ To let a regular user read developer or operations docs, add `docs_developer` or
## Data Exploration
- `/datasources`: source directory. The `Built-in Sources` tab can be filtered by product domain, layer/module, enabled state, last run status, whether collected records exist, and search text. Selecting rows triggers only those sources; `Trigger All` enters a browser-download-list style collection queue. The queue bar shows total progress plus running, completed, failed, and skipped counts; `View Queue` opens the bottom panel, failed rows can retry, and completed rows can jump to detail. `Realtime Sources` is for AISStream / WebSocket long connections and shows connection health, stored totals, time-window counters, and Start / Stop / Reconnect actions. Endpoint/credential/header editing happens at `/collection-management -> Collectors`.
- `/datasources`: source directory. The `Built-in Sources` tab can be filtered by product domain, layer/module, enabled state, last run status, whether collected records exist, and search text. With no rows selected, the primary button shows `Trigger All`; after selecting rows, it becomes `Trigger Selected N` and submits only those sources. The top-right queue button shows a queue icon when empty and a pure circular total-progress indicator while tasks exist; it opens a floating panel grouped by running, completed, failed, and skipped. Failed rows can retry, and completed rows can jump to detail. `Realtime Sources` is for AISStream / WebSocket long connections and shows connection health, stored totals, time-window counters, and Start / Stop / Reconnect actions. Endpoint/credential/header editing happens at `/collection-management -> Collectors`.
- `/data`: collected data table — used to verify "did data arrive", "is the freshness right", "does a source emit valid records"
- `/bgp`: BGP detail page with list + detail + analysis; complements the BGP layer on Earth
- `/alerts/system`, `/alerts/bgp`, `/alerts/situational`: system, BGP, and situational alerts
@@ -280,7 +280,17 @@ Search finds cables, landing points, satellites, compute centers, BGP events, BG
Compute center and BGP observer detail cards support automatic coordinate-candidate collection. Click the object then use "Collect Coordinate Candidates" or "Re-collect Coordinates". The backend assembles candidates from source coordinates, public-org registry APIs, and online geocoders. When regular sources have no candidate, the current default AI Provider runs one LLM factcheck fallback. BGP observers' stored coordinates only fill query context; they are not returned as candidates.
Candidates preview on Earth directly. Saving a compute-center candidate writes to the `compute_center_locations` dimension table and refreshes the layer immediately. The notification badge at the top-left of the compute-center layer shows the unresolved count; clicking it opens the queue, supports single collection, or "Adopt All" to save the top-confidence candidates from top to bottom. Records without candidates stay in the queue rather than being faked to country centroids. See [Earth Location Candidate Collection User Guide](/home/ray/dev/linkong/planet/docs/technical/en/location-pipeline-user.md).
Candidates preview on Earth directly. Saving a compute-center candidate writes to the `compute_center_locations` dimension table and refreshes the layer immediately. The notification badge at the top-left of the compute-center layer shows the unresolved count; clicking it opens the queue, supports single collection, or "Adopt All" to save the top-confidence candidates from top to bottom. Records without candidates stay in the queue rather than being faked to country centroids.
Recommended single-object flow:
1. Open a compute center or BGP observer detail card.
2. Click "Collect Coordinate Candidates".
3. Wait for candidates; entries that depend on WebSearch / AI factcheck show collection state.
4. Preview candidate positions on Earth.
5. Save the candidate when it is credible; closing the card does not lose the current task state.
Adopt All is for batch processing the compute-center unresolved queue. It starts from the top and adopts the highest-confidence candidate. Records without factual support remain in the queue. When WebSearch is disabled, single locate and Adopt All are disabled because location validation depends on factual lookup.
### Settings
@@ -351,7 +361,7 @@ Mobile uses a drawer layout: layer control moves into a drawer; search/settings/
Docs at `http://localhost:3000/docs` are served by the backend with access control, not bundled into the frontend build.
Anonymous visitors see only `public` docs: README, Quickstart, Manual, FAQ, Earth Location Candidate Collection User Guide. Authenticated users with Gatekeeper groups see more:
Anonymous visitors see only `public` docs: README, Quickstart, Manual, and FAQ. Authenticated users with Gatekeeper groups see more:
- `docs_user`: end-user operational docs
- `docs_developer`: Earth, frontend, backend, collectors, AI Provider development docs
@@ -365,5 +375,4 @@ Docs supports: category navigation, Markdown rendering, tables and code blocks,
- [Quickstart](/home/ray/dev/linkong/planet/docs/technical/en/quickstart.md)
- [FAQ](/home/ray/dev/linkong/planet/docs/technical/en/faq.md)
- [Earth Location Candidate Collection User Guide](/home/ray/dev/linkong/planet/docs/technical/en/location-pipeline-user.md)
- [Planet Ops Runbook](/home/ray/dev/linkong/planet/docs/technical/en/ops-runbook.md)
- [Intelligent Planet Ops Runbook](/home/ray/dev/linkong/planet/docs/technical/en/ops-runbook.md)

View File

@@ -1,6 +1,6 @@
# Naming Glossary
This document standardizes terms used across the Planet console, Earth, backend APIs, and documentation. When adding Chinese UI labels, chart labels, or translated documentation, prefer the Chinese display names listed here to avoid unnecessary mixed Chinese/English copy.
This document standardizes terms used across Intelligent Planet, the console, backend APIs, and documentation. When adding Chinese UI labels, chart labels, or translated documentation, prefer the Chinese display names listed here to avoid unnecessary mixed Chinese/English copy.
## Usage Rules
@@ -14,9 +14,9 @@ This document standardizes terms used across the Planet console, Earth, backend
| English / Key | Chinese Display Name | Usage |
| --- | --- | --- |
| Planet | Planet | Product name |
| Admin Next | 控制台 | New admin console |
| Earth | Earth | Visualization product name |
| Planet | 智能星球 | Product name and primary experience entry |
| Admin | 控制台 | Admin console context |
| Earth | 智能星球 | Visualization product name and `/earth` entry |
| datasource | 数据源 | APIs, lists, filters |
| collector | 采集器 | Collection jobs and credential configuration |
| collected data | 采集数据 | Data list and statistics |

View File

@@ -1,6 +1,6 @@
# Planet Ops Runbook
# Intelligent Planet Ops Runbook
This runbook is for deployment, on-call, and maintenance engineers. End-user UI flows live in the [Planet Manual](/home/ray/dev/linkong/planet/docs/technical/en/manual.md); this document only covers shell, Docker, logs, environment variables, and troubleshooting.
This runbook is for deployment, on-call, and maintenance engineers. End-user UI flows live in the [Intelligent Planet Manual](/home/ray/dev/linkong/planet/docs/technical/en/manual.md); this document only covers shell, Docker, logs, environment variables, and troubleshooting.
## First Startup
@@ -61,6 +61,22 @@ Per-module restart:
Per-module restart is preferred during development to avoid interrupting unrelated services.
## Destructive Reset
```bash
./planet.sh destroy
```
`destroy` returns a local development environment to a near-empty project state. It requires typing `Y` before it runs; source files and existing `.env` files are preserved.
Cleanup order and boundaries:
- If `planet_postgres` is running, the script first clears the `public` schema in `planet_db`. This prevents old `collected_data.is_current = true` rows from making Earth OOBE report `ready=true` if Docker volume removal later fails.
- Docker cleanup targets resources whose Compose project is `planet`, plus the explicit volumes `planet_postgres_data`, `planet_redis_data`, `postgres_data`, and `redis_data`; do not delete unlabeled volumes by a broad `planet_*` pattern, because another local project could own them.
- Local build state removes `.venv`, frontend `node_modules` / `dist`, Planet state, and scattered Python / Vite cache directories. `$PLANET_CACHE_DIR/downloads` is preserved so upstream raw downloads such as CelesTrak can survive database resets and local rebuild cleanup.
After the reset, run `./planet.sh init` again to recreate tables and default seed data. Old collected records are not restored, and Earth OOBE is evaluated from the backend's real collection state on the next visit. When CelesTrak later returns its "GP data has not updated" HTTP 403, the backend first reuses the preserved download cache to repopulate the database; if no cache exists, wait for the next CelesTrak update window or use Space-Track as a fallback.
## Health Check
```bash
@@ -120,6 +136,20 @@ Useful for:
- Demoing Earth from a phone or tablet
- Other LAN machines reaching the same dev instance
On Windows, the repository-root `planet.cmd` can be used as a one-click entrypoint. It requests Administrator privileges, enters the `Ubuntu` WSL distribution at `/home/linkong/planet`, runs `./planet.sh restart --allow-lan`, opens `http://localhost:3000/earth` after a successful restart, and leaves the terminal inside a WSL shell for log inspection. If the local WSL distribution name or checkout path differs, adjust the `wsl.exe -d ... --cd ...` arguments in `planet.cmd` first.
On a new Windows machine, check the WSL generation first:
```powershell
wsl -l -v
```
Planet development should use WSL2. WSL1 has different networking, filesystem, and process behavior, and can surface as Bun package-manager commands returning only `An unknown error occurred (Unexpected)`, unstable port release, or LAN behavior that does not match the script's assumptions. Convert the distribution if it still runs as WSL1:
```powershell
wsl --set-version Ubuntu 2
```
`--allow-lan` directly exposes the frontend, backend, and AI Provider from the development machine: frontend `3000`, backend `8000`, and AI Provider `8010`. Before startup, the script checks all three ports. If WSL/Linux cannot release a port and a Windows-side listener or stale `portproxy` rule owns it, the script requests Administrator PowerShell cleanup. When Planet runs in WSL, Windows can usually reach it through `localhost`; other LAN machines reaching the Windows LAN IP still need Windows Firewall allow rules.
Diagnose in this order:
@@ -184,13 +214,32 @@ The AI Provider image only rebuilds when code, Dockerfile, Compose config, or Py
./planet.sh restart -a
```
Rebuild detection is based on a content fingerprint rather than only file mtimes. `planet.sh` hashes the `aiprovider/` files, `aiprovider/Dockerfile`, `pyproject.toml`, `uv.lock`, `PYTHON_IMAGE`, `UV_IMAGE`, and the dependency fingerprint into `AI_PROVIDER_BUILD_FINGERPRINT`; Docker writes it into the image label `planet.aiprovider.build-fingerprint`. If the existing `planet-aiprovider:latest` image has a matching label, the script skips rebuild and refreshes the local stamp. Older images without the label fall back to the state/cache stamp.
Docker builds use `uv sync --frozen`. To make container builds reuse the host uv mirror configuration, the script resolves the first config file in this order and mounts it into the build as a BuildKit secret at `/root/.config/uv/uv.toml`:
1. The current `UV_CONFIG_FILE`
2. Repository-root `uv.toml`
3. `${XDG_CONFIG_HOME:-~/.config}/uv/uv.toml`
4. `~/.uv/uv.toml`
If none exists, the script creates an empty state file for the secret so Compose does not fail on a missing file. Before Docker build it unsets `UV_DEFAULT_INDEX`, `UV_INDEX_URL`, and `UV_EXTRA_INDEX_URL`, keeping the build tied to the explicit `UV_CONFIG_FILE`. For a temporary Tsinghua mirror, place this in repository-root `uv.toml`:
```toml
[[index]]
name = "tsinghua"
url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple/"
default = true
```
Diagnose slow builds:
| Symptom | Common cause | Fix |
| --- | --- | --- |
| Large `transferring context` | build context includes unrelated frontend / data files | `.dockerignore` ships only required files |
| `uv sync` is slow | first build or cold cache | wait for the first build; later runs reuse BuildKit cache |
| `uv sync --frozen` is slow | first build, cold cache, or missing uv mirror config | wait for the first build; later runs reuse BuildKit cache; configure `uv.toml` when needed |
| Old keys still in effect after edit | container not restarted | `./planet.sh restart -a` |
| Code changed but the image did not rebuild | fingerprint still matches the image label | Confirm the change is under `aiprovider/`, Dockerfile, or Python dependency inputs; delete `planet-aiprovider:latest` and retry if needed |
## SMTP Email (Required for Public Registration)
@@ -217,6 +266,15 @@ One-time codes are stored in Redis under `otp:{purpose}:{email}` with a 600-seco
## Development Command Conventions
Backend setup and script initialization use the lockfile:
```bash
uv python install 3.14
uv sync --frozen --group dev
```
`--frozen` rejects implicit `uv.lock` rewrites, which is the desired behavior on new machines, CI, and Docker builds. Dependency upgrades should explicitly update `pyproject.toml` / `uv.lock` on a development machine and commit the lockfile.
Frontend must use Bun:
```bash

View File

@@ -0,0 +1,160 @@
# Business Architecture and Data Flows
This document is the business entry point for Intelligent Planet data products. It explains why each Earth data type exists, where it is collected from, which fact or derived tables it uses, and how cache invalidation plus WebSocket hints reach Earth. Frontend, backend, and Earth docs should focus on implementation details; start here when you need the cross-system data flow.
## Overview
Intelligent Planet data moves through three stages:
1. **Collect and normalize**: built-in collectors, admin actions, or the location pipeline write PostgreSQL. Generic raw output lands in `collected_data`; layer-ready projections land in derived tables.
2. **Project and broadcast**: database triggers write fact changes to the `earth_data_change_events` outbox and wake the backend listener with `LISTEN/NOTIFY`. The listener maps the change through layer adapters, invalidates cache, and broadcasts `earth_updates`.
3. **Reload and render in Earth**: Earth receives layer-level refresh hints, applies `clear_then_reload`, `reload`, or `delta`, then refetches `/api/v1/visualization/...`.
```mermaid
flowchart LR
Source["External source / admin action"] --> Collector["Collector or data job"]
Collector --> Facts["collected_data"]
Collector --> Derived["Derived tables"]
Facts --> Outbox["earth_data_change_events"]
Derived --> Outbox
Outbox --> Listener["DB change listener"]
Listener --> Cache["Earth cache invalidation"]
Listener --> WS["earth_updates"]
WS --> Earth["Earth layer reload"]
Earth --> API["Visualization APIs"]
```
`LISTEN/NOTIFY` is only the low-latency wakeup path. The reliable source is the outbox. A real zero-data state is valid and APIs should return HTTP 200 with an empty collection; only actual endpoint failures should return 5xx.
## Data Products
```mermaid
flowchart TB
Satellites["Satellite TLE"] --> SatelliteLayer["satellites layer"]
Cables["Cables + landing points"] --> CableLayer["cables layer"]
Compute["TOP500 / AI GPU / HF"] --> ComputeLocations["compute_center_locations"]
ComputeLocations --> ComputeLayer["computeCenters layer"]
BgpRaw["RIS Live / BGPStream / Prefix"] --> BgpDerived["bgp_observations / anomalies / incidents"]
BgpDerived --> BgpLayer["bgp layer"]
VesselRaw["AIS / BarentsWatch"] --> VesselDerived["vessel_static / vessel_position"]
VesselDerived --> VesselLayer["vessels layer"]
Interactables["earth_interactables"] --> InteractableLayer["interactables layer"]
NewsRaw["RSS / Live / News"] --> NewsItems["earth_news_items"]
NewsItems --> NewsLayer["news / media layer"]
```
| Data product | Business use | Fact sources | Derived / dimension tables | Earth layer | Strategy |
| --- | --- | --- | --- | --- | --- |
| Satellites | In-orbit objects, tracks, coverage, and cruise targets | `celestrak_tle`, `spacetrack_tle` | No long-lived projection; APIs convert TLE on demand | `satellites` | `clear_then_reload` |
| Cables and landing points | Submarine connectivity, landing cities, cable detail, and search | `arcgis_cables`, `arcgis_landing_points`, TeleGeography / FAO landing sources | Cable relation and landing aggregation data | `cables` | `clear_then_reload` |
| Compute centers | TOP500, AI GPU, model platform facilities, and location-completion state | `top500`, `epoch_ai_gpu`, HuggingFace sources | `compute_center_locations` | `computeCenters` | `reload` |
| BGP context | Collectors, anomalies, incidents, route events, and regional context | `ris_live_bgp`, `bgpstream_bgp`, prefix geography sources | `bgp_observations`, `bgp_anomalies`, `bgp_incidents`, `bgp_collector_locations` | `bgp` | `clear_then_reload` |
| Vessels | AIS vessels, positions, tracks, legend, and source health | AIS sources, `barentswatch_vessels` | `vessel_static`, `vessel_position`, `ais_raw_observations`, `ais_source_health` | `vessels` | `clear_then_reload` |
| Interactables | Generic surface icons, manual objects, and future small layers | `earth_interactables` | None | `interactables` | `delta` |
| News and media | Earth news, live streams, cruise summaries, and situation content | news sources | `earth_news_items` | `news` / `media` | `reload` |
## Satellites
Satellite data powers online satellite markers, tracks, footprint policy, and cruise lists. CelesTrak or Space-Track collectors fetch TLE and write `collected_data`. The visualization API converts TLE into current positions and trails on request.
- **Collection entry**: CelesTrak TLE, Space-Track TLE.
- **Fact table**: `collected_data.source IN ('celestrak_tle', 'spacetrack_tle')`.
- **API**: satellite visualization API reads TLE and builds the Earth payload.
- **Delete semantics**: deleting the source broadcasts `satellites` with `clear_then_reload`; the frontend clears satellite markers and trails before refetching. With no TLE, the API returns an empty list.
- **Common failure**: summary is zero but Earth still shows satellites. Usually WS did not fire, the adapter did not cover the source, or the frontend did not clear existing Three.js objects.
## Cables and Landing Points
Cables and landing points show global network connectivity, landing cities, cable details, and search targets. Cable lines and landing points are one business layer; deleting either side must refresh `cables`.
- **Collection entry**: ArcGIS cables, ArcGIS landing points, and TeleGeography / FAO landing related sources.
- **Fact table**: cable and landing point sources in `collected_data`.
- **Derived data**: cable relation tables, landing point aggregation, interface cache.
- **API**: cable visualization API returns cables, landing points, and relation data.
- **Delete semantics**: deleting cable or landing source clears owned projections and broadcasts `cables` with `clear_then_reload`. Empty data returns HTTP 200, not 404.
- **Common failure**: cables disappear for a second and return. Usually a collection replacement or cache refresh window reuses old cache; check duplicate broadcasts and cache patterns.
## Compute Centers
Compute centers represent supercomputers, AI GPU sites, model-platform facilities, and unresolved-location state. Raw sources often only provide organization, country, site, or fuzzy location, so Earth rendering depends on `compute_center_locations`.
- **Collection entry**: TOP500, Epoch AI GPU, HuggingFace related sources.
- **Fact table**: compute sources in `collected_data`.
- **Dimension table**: `compute_center_locations` stores adopted coordinate candidates.
- **API**: compute center visualization API merges raw records with location rows.
- **Delete semantics**: deleting TOP500 or similar sources refreshes `computeCenters`; changing the location table also refreshes the layer. The default is `reload`, because location updates do not always need an immediate clear.
- **Common failure**: collection finished but Earth count did not change. Check visualization cache, location rows, and adapter coverage.
## BGP
BGP data shows route collectors, anomalies, incidents, diffusion rings, and situation summaries. Earth does not render raw BGP rows directly; it renders projected observations, anomalies, and incidents. This is the most common path where deleting raw data can leave old objects visible.
```mermaid
sequenceDiagram
participant UI as Admin delete button
participant Job as clear_data job
participant DB as PostgreSQL
participant Outbox as earth_data_change_events
participant Listener as DB change listener
participant Earth as Earth frontend
UI->>Job: Submit delete for ris_live_bgp / bgpstream_bgp
Job->>DB: Delete collected_data raw rows
Job->>DB: Delete BGP-owned derived tables
DB->>Outbox: trigger writes bgp layer changed
DB-->>Listener: LISTEN/NOTIFY wakeup
Listener->>Listener: merge events and clear bgp cache
Listener-->>Earth: broadcast earth_updates clear_then_reload
Earth->>Earth: clear BGP objects
Earth->>DB: refetch projections through visualization API
```
- **Collection entry**: RIPE RIS Live BGP, CAIDA BGPStream Backfill, IPtoASN / OpenGeoFeed / NRO prefix geography.
- **Fact table**: BGP and prefix sources in `collected_data`.
- **Derived tables**: `bgp_observations`, `bgp_anomalies`, `bgp_incidents`, `bgp_collector_locations`.
- **API**: BGP visualization API reads projections and combines stored or candidate locations.
- **Delete semantics**: deleting RIS Live or BGPStream raw sources must also clear BGP-owned projections and broadcast `bgp` with `clear_then_reload`. Direct derived-table deletes also write outbox events.
- **Common failure**: collectors or incidents remain after raw rows are removed. Check derived tables first, not only `collected_data`.
## Vessels
Vessel data shows AIS vessels, navigation state, vessel-type legend, and source health. Earth rendering uses position snapshots and static vessel information; it should not render every raw AIS observation.
- **Collection entry**: AIS sources, BarentsWatch vessels.
- **Fact table**: `collected_data` or AIS raw observation tables.
- **Derived tables**: `vessel_static`, `vessel_position`, `ais_raw_observations`, `ais_source_health`.
- **API**: vessel visualization API returns current vessel markers and detail fields.
- **Delete semantics**: deleting any vessel source lets owned projections broadcast `vessels` with `clear_then_reload`.
- **Common failure**: the count panel changes but vessels remain. Summary and layer data are separate; Earth should clear objects based on layer updates.
## Interactables
`earth_interactables` is the generic surface-icon capability for manual objects, extensions, and future small layers. Unlike most layers, it keeps object-level delta.
- **Fact table**: `earth_interactables`.
- **API**: interactable API and Earth generic icon interface.
- **Strategy**: upsert for create/update and removeItem for delete; no full layer reload.
- **Delete semantics**: delete payloads must include a stable id so the frontend can remove the object.
- **Common failure**: an object cannot be removed. Usually the stable id is missing, the delta handler did not match the object type, or an older duplicate render path still exists.
## News and Media
News and media support the Earth news ticker, live stream panel, news cruise, and situation summaries. They are content refresh paths rather than stable geographic object layers, so they default to `reload`.
- **Collection entry**: RSS, live streams, news sources.
- **Fact table**: news source rows in `collected_data`.
- **Derived table**: `earth_news_items`.
- **API**: news, live stream, and media content APIs.
- **Delete semantics**: deleting news sources or `earth_news_items` broadcasts `news` / `media` reload; empty responses hide the corresponding content.
- **Common failure**: the live panel shows stale content. Usually the media component ignored the layer update or the content API cache was not invalidated.
## Adding a New Layer
Add a new Earth data product in this order:
1. Define the business purpose and Earth layer name.
2. Identify fact sources, fact tables, and derived tables.
3. Register source/table coverage, cache patterns, owned derived cleanup, and default strategy in the backend layer adapter.
4. Ensure the visualization API returns HTTP 200 with an empty collection for zero data.
5. Keep the Earth frontend driven only by layer and strategy, not database table names.
6. Update this document with the data product flow, then document implementation details in the relevant frontend, backend, or Earth technical doc.

View File

@@ -1,8 +1,8 @@
# Quickstart
This quickstart is for Planet end users who just received an access URL and need the shortest path from "open the browser" to "first useful configuration done". Every action happens in the browser.
This quickstart is for Intelligent Planet end users who just received an access URL and need the shortest path from "open the browser" to "first useful configuration done". Every action happens in the browser.
If you are responsible for deployment or operations, read the [Planet Ops Runbook](/home/ray/dev/linkong/planet/docs/technical/en/ops-runbook.md) instead.
If you are responsible for deployment or operations, read the [Intelligent Planet Ops Runbook](/home/ray/dev/linkong/planet/docs/technical/en/ops-runbook.md) instead.
## 1. Open the URL
@@ -34,7 +34,7 @@ After landing on the `/admin` dashboard, here's a recommended walk-through:
1. `/collection-management?tab=collector_credentials`: pick a collector and click the plug icon to test connectivity. Free collectors (e.g. open BGP) usually work right away; credential-bearing ones like `AISStream` or `BarentsWatch` need an API key / client secret first
2. `/ai?tab=providers`: fill an LLM provider (e.g. `minimax` / `openai`), model, base URL, API key, and click the plug at the end of the base URL to test. WebSearch / OCR tools are optional
3. `/datasources` or `/data`: check whether collectors have produced data. Use `/datasources -> Built-in Sources` for finite collectors and open the collection queue after triggering; use `/datasources -> Realtime Sources` for AISStream / WebSocket health and counters
3. `/datasources` or `/data`: check whether collectors have produced data. Use `/datasources -> Built-in Sources` for finite collectors: with no rows selected, click `Trigger All`; after selecting rows, the primary button becomes `Trigger Selected N`. The top-right queue button shows progress. Use `/datasources -> Realtime Sources` for AISStream / WebSocket health and counters
4. `/alerts/system`: verify system alerts look right
5. `/users` (super_admin only): open accounts for teammates or adjust their groups
@@ -62,7 +62,7 @@ Open `/forgot-password`, enter your email, receive a code, then enter the code p
## Next Steps
- Full UI walkthrough: [Planet Manual](/home/ray/dev/linkong/planet/docs/technical/en/manual.md)
- Full UI walkthrough: [Intelligent Planet Manual](/home/ray/dev/linkong/planet/docs/technical/en/manual.md)
- Troubleshooting and configuration questions: [FAQ](/home/ray/dev/linkong/planet/docs/technical/en/faq.md)
- Detailed Earth coordinate candidate flow: [Earth Location Candidate Collection User Guide](/home/ray/dev/linkong/planet/docs/technical/en/location-pipeline-user.md)
- Deployment / operations commands: [Planet Ops Runbook](/home/ray/dev/linkong/planet/docs/technical/en/ops-runbook.md)
- Detailed Earth coordinate candidate flow: see the Earth section in [Intelligent Planet Manual](/home/ray/dev/linkong/planet/docs/technical/en/manual.md)
- Deployment / operations commands: [Intelligent Planet Ops Runbook](/home/ray/dev/linkong/planet/docs/technical/en/ops-runbook.md)

View File

@@ -1,6 +1,6 @@
# Tactile UI Components
Tactile UI is Planet's portable React control layer. It was extracted from the Admin Next button, switch, scrollbar, and tooltip work, but the components themselves do not depend on Admin Next, AntD, Radix, Tailwind, or `an-*` classes. The immediate goal is stable in-repo usage; the structure is intentionally close to something that can later be published as an npm package.
Tactile UI is Planet's portable React control layer. It was extracted from the Admin button, switch, scrollbar, and tooltip work, but the components themselves do not depend on Admin, AntD, Radix, Tailwind, or `an-*` classes. The immediate goal is stable in-repo usage; the structure is intentionally close to something that can later be published as an npm package.
## Design Goals
@@ -49,6 +49,22 @@ Core tokens are exposed as `--tui-*` CSS variables. Product themes should overri
Most controls also accept a `tactile` prop for local width, height, radius, background, border, and shadow overrides. Use local overrides for small special cases; use CSS variables for product-wide styling.
## Portals and Dark Theme
`TactileTooltip` and the Admin `Dialog`, `Select`, and toast controls that use Tactile UI may render through a portal attached to `document.body`. Those nodes are not descendants of `.admin-theme-root[data-theme='dark']`, so dark tokens cannot rely only on an ancestor selector inside the Admin root.
The Admin theme provider mirrors the active theme to `body[data-admin-theme]`. Shared styles need to support both selector paths:
```css
[data-theme='dark'] .tui-button,
body[data-admin-theme='dark'] .tui-button {
--tui-surface: #172033;
--tui-text: #e5edf8;
}
```
When adding a portal-based control, first check whether it renders into body. If it does, add a `body[data-admin-theme='dark']` branch in that component's style entry, or reuse the already covered `--tui-*` / `--an-*` tokens. Avoid hard-coding a one-off dark modal style, because the same contrast problem can reappear in dropdowns, tooltips, toasts, and confirmation dialogs.
## `TactileButton`
The button component covers regular buttons, icon buttons, strong-intent buttons, and link-like buttons.
@@ -74,6 +90,8 @@ Common props:
`variant="neutral"` defaults to a white tactile button. Colored buttons should still keep the same height and external shadow instead of relying on page-specific CSS overrides.
Colored button borders must not use the exact fill color. `primary`, `danger`, and future colored variants should use a lighter border from the same hue, such as `color-mix(in srgb, var(--tui-danger) 64%, white)`. The border still reads as part of the button color, but its visual weight is lower than the fill surface, so red or blue buttons do not look one outline larger than neutral buttons. Hover states should brighten rather than darken: mix a little white into the current fill color, and keep the hover border lighter than the hover fill.
## Icon Presets
Preset icons are maintained in `tactileIconPresets`. Feature pages should call icons by semantic name so actions remain consistent across the console.
@@ -177,7 +195,7 @@ Do not force overlay scrollbars onto textareas. Text selection and the resize gr
## `TableScrollRegion`
`TableScrollRegion` is a convenience wrapper for table scroll areas. The default target selector is `.tui-scroll-target`; Admin Next passes its table viewport selector explicitly so package code does not contain product-specific names.
`TableScrollRegion` is a convenience wrapper for table scroll areas. The default target selector is `.tui-scroll-target`; Admin passes its table viewport selector explicitly so package code does not contain product-specific names.
```tsx
<TableScrollRegion targetSelector=".table-viewport">
@@ -192,9 +210,9 @@ Do not force overlay scrollbars onto textareas. Text selection and the resize gr
- Tooltip text is explanatory only; state must still be represented by text, badges, or `aria-*` attributes.
- Disabled and loading states set `aria-disabled`; real `button` elements also receive `disabled`.
## Admin Next Migration Rules
## Admin Migration Rules
Admin Next should use Tactile UI for global tool buttons, detail-panel toolbars, and list footer actions:
Admin should use Tactile UI for global tool buttons, detail-panel toolbars, and list footer actions:
- Unambiguous actions: `TactileButton iconOnly tooltip`
- Save/create/confirm: `TactileButton variant="primary"`, usually with text

View File

@@ -1,44 +1,52 @@
# 技术文档
这里放“当前实现和当前结构”的文档,重点回答:
这里是智能星球当前文档入口。文档按读者和问题类型分层:先看业务架构理解数据产品,再进入使用手册或技术实现文档。
- 现在代码是怎么组织的
- 当前入口在哪
- 状态和组件如何工作
## 业务架构
适合放入这里的内容:
- [业务架构与数据流转](/home/ray/dev/linkong/planet/docs/technical/zh/platform-data-flows.md):每类智能星球数据的用途、采集链路、事实表、派生表、缓存和 WebSocket 广播链路
- [命名与术语对照](/home/ray/dev/linkong/planet/docs/technical/zh/naming-glossary.md):智能星球、控制台、后端和文档常见名词的中英对照
- 快速开始和使用手册
- 前端上下文
- Earth 前端结构
- Earth 卫星覆盖策略
- Earth 渲染图层顺序
- Earth 图层样式属性索引
- 后端运行控制
- 采集器现状
- 采集器设置与连接验证
- 采集格式约定
## 使用手册
## 使用入口
- [快速开始](/home/ray/dev/linkong/planet/docs/technical/zh/quickstart.md):从零启动智能星球的最短路径
- [智能星球使用手册](/home/ray/dev/linkong/planet/docs/technical/zh/manual.md):控制台、智能星球、文档和常用功能的用户操作说明
- [常见问题](/home/ray/dev/linkong/planet/docs/technical/zh/faq.md)Windows / WSL、端口、依赖、动捕、凭证和文档权限排障
- [快速开始](/home/ray/dev/linkong/planet/docs/technical/zh/quickstart.md):从零启动 Planet 的最短路径
- [Planet 使用手册](/home/ray/dev/linkong/planet/docs/technical/zh/manual.md):控制台、`planet.sh`、Earth 和 Docs 的完整使用手册
- [常见问题](/home/ray/dev/linkong/planet/docs/technical/zh/faq.md)Windows / WSL、端口、依赖、动捕、凭证和 Docs 权限的集中排障入口
- [Earth 位置候选采集使用手册](/home/ray/dev/linkong/planet/docs/technical/zh/location-pipeline-user.md):在 Earth 上为算力中心和 BGP 观测站采集、预览坐标候选
- [数据源、采集器设置与连接验证](/home/ray/dev/linkong/planet/docs/technical/zh/datasource-collector-settings-connectivity.md)数据源目录、采集器设置、连接验证、BarentsWatch 凭证链路
- [通用位置估算管线开发说明](/home/ray/dev/linkong/planet/docs/technical/zh/location-pipeline-development.md):后端 location resolver / pipeline 的接口、注册表和扩展方式
- [Docs Gatekeeper 开发说明](/home/ray/dev/linkong/planet/docs/technical/zh/docs-gatekeeper-development.md):后端 Docs 目录、正文读取和 Gatekeeper 权限组实现
- [命名与术语对照](/home/ray/dev/linkong/planet/docs/technical/zh/naming-glossary.md)控制台、Earth、后端和文档常见名词的中英对照
- [Earth 可交互图标接入](/home/ray/dev/linkong/planet/docs/technical/zh/earth-interactable-usage.md)Earth 地表可交互图标 `Interactable` 的接口、生命周期和接入示例
- [Earth 工具栏与浮层协同](/home/ray/dev/linkong/planet/docs/technical/zh/earth-toolbar-overlay-coordination.md):工具栏按钮与搜索 / 设置 / 新闻 / 图层浮层之间的关闭矩阵和接入规则
- [Tactile UI 组件库](/home/ray/dev/linkong/planet/docs/technical/zh/tactile-ui-components.md)可移植按钮、开关、tooltip 和滚动条组件的 API、主题 token 与迁移约定
## 智能星球技术实现
不适合放入这里的内容:
- [智能星球前端结构](/home/ray/dev/linkong/planet/docs/technical/zh/earth-frontend-context.md)智能星球页面模块、状态、WebSocket 刷新和图层生命周期
- [智能星球图层样式属性索引](/home/ray/dev/linkong/planet/docs/technical/zh/earth-layer-style-reference.md):图层颜色、符号、材质和视觉参数
- [智能星球渲染图层顺序](/home/ray/dev/linkong/planet/docs/technical/zh/earth-render-layer-order.md)renderOrder、深度策略、拾取和同坐标避让
- [智能星球卫星覆盖策略](/home/ray/dev/linkong/planet/docs/technical/zh/earth-satellite-footprint-policy.md):卫星 footprint 的显示边界和策略
- [BGP 态势上下文](/home/ray/dev/linkong/planet/docs/technical/zh/earth-bgp-context.md)BGP 在智能星球中的渲染、聚合和观测站实现
- [智能星球可交互图标接入](/home/ray/dev/linkong/planet/docs/technical/zh/earth-interactable-usage.md)`Interactable` 的接口、生命周期和接入示例
- [智能星球工具栏与浮层协同](/home/ray/dev/linkong/planet/docs/technical/zh/earth-toolbar-overlay-coordination.md):工具栏按钮与搜索、设置、新闻、图层浮层的关闭矩阵
- 尚未完成的路线图
- 未来迭代方案
- 大范围重构计划
## 前端技术实现
这些应放入:
- [控制台前端结构](/home/ray/dev/linkong/planet/docs/technical/zh/frontend-admin-frontend-context.md):控制台页面结构、状态边界和懒加载策略
- [前端布局指南](/home/ray/dev/linkong/planet/docs/technical/zh/frontend-layout-guidelines.md):布局、表格、面板和响应式约束
- [Tactile UI 组件库](/home/ray/dev/linkong/planet/docs/technical/zh/tactile-ui-components.md)按钮、开关、tooltip、滚动条组件 API 和主题 token
- [计划文档索引](/home/ray/dev/linkong/planet/docs/plans/README.md)
## 后端技术实现
- [数据采集系统](/home/ray/dev/linkong/planet/docs/technical/zh/backend-collectors.md):采集器、任务类型、保存层和状态回写
- [系统服务控制](/home/ray/dev/linkong/planet/docs/technical/zh/backend-system-service-control.md):后端服务控制和系统操作接口
- [数据源、采集器设置与连接验证](/home/ray/dev/linkong/planet/docs/technical/zh/datasource-collector-settings-connectivity.md):数据源目录、采集器设置、连接验证和凭证链路
- [数据源 API 性能](/home/ray/dev/linkong/planet/docs/technical/zh/backend-datasources-api-performance.md)DataSources 列表接口性能和缓存策略
- [数据作业与 Outbox 技术架构](/home/ray/dev/linkong/planet/docs/technical/zh/data-job-earth-sync-architecture.md)PostgreSQL 作业队列、outbox、listener 和 Kafka / Spark 演进边界
- [通用位置估算管线开发说明](/home/ray/dev/linkong/planet/docs/technical/zh/location-pipeline-development.md)location resolver / pipeline 的接口、注册表和扩展方式
- [新闻直播采集格式](/home/ray/dev/linkong/planet/docs/technical/zh/earth-news-live-streams-collector-format.md):新闻、直播和媒体采集 payload 约定
- [Docs Gatekeeper 开发说明](/home/ray/dev/linkong/planet/docs/technical/zh/docs-gatekeeper-development.md):后端文档目录、正文读取和 Gatekeeper 权限组实现
## 智能体与运维
- [AI Provider 指南](/home/ray/dev/linkong/planet/docs/technical/zh/agents-aiprovider.md):模型供应商适配、任务 prompt 和调用边界
- [智能星球运维手册](/home/ray/dev/linkong/planet/docs/technical/zh/ops-runbook.md):部署、启动、排障和敏感操作
- [Docker + Compose + Buildx 升级](/home/ray/dev/linkong/planet/docs/technical/zh/ops-docker-compose-buildx-upgrade.md)Docker 工具链升级步骤
- [planet.sh 启动机制](/home/ray/dev/linkong/planet/docs/technical/zh/ops-planet-sh-startup.md):启动脚本、健康检查和性能优化
## 计划与历史
尚未完成的路线图、大范围重构方案和未来计划放在仓库路径 `docs/plans/README.md`。已完成、过时或被新实现替代的方案放在 `docs/deprecated/README.md`

View File

@@ -101,7 +101,7 @@ AI 配置页使用的接口:
这些接口都需要用户登录。`secrets` 接口只用于配置页点击显示 key/token 时取回明文,隐藏时前端恢复为脱敏预览。
Admin Next 的 AI 页面按旧版信息架构组织为:
Admin 的 AI 页面按业务信息架构组织为:
- `模型供应商`
- 管理 provider、协议适配、默认模型、LLM API Key、代理地址、代理 token、模型列表刷新、设为默认和轻量连通性测试。
@@ -272,7 +272,7 @@ AI 配置仍保存在 PostgreSQL不写入 JSON 文件。核心结构如下:
`.env` 只是兜底。配置页保存或测试连接成功后PostgreSQL 中的配置会成为全局默认。
Admin Next 的密钥状态必须按 provider / tool 精确判断:
Admin 的密钥状态必须按 provider / tool 精确判断:
- 数据库中当前 provider/tool 有密钥时,显示为“已配置”。
- 数据库没有密钥,但 fallback provider、model 或 tool 与当前项匹配时,可以显示 fallback 的脱敏预览。
@@ -283,7 +283,7 @@ Admin Next 的密钥状态必须按 provider / tool 精确判断:
### 轻量连通性测试
Admin Next 的插头按钮走轻量连通性测试,不承担保存职责。业界常见做法是分两层:
Admin 的插头按钮走轻量连通性测试,不承担保存职责。业界常见做法是分两层:
- 快速检查 provider 目录或低成本 endpoint确认 base URL、鉴权和当前模型是否可达。
- 只有在用户明确运行 Playground 或业务任务时才发完整模型请求。

View File

@@ -73,6 +73,8 @@ async def run(self, db):
**核心文件**: `backend/app/services/collectors/base.py`
手动触发、删除数据、清理缓存现在统一进入 PostgreSQL 数据作业队列,任务账本仍是 `collection_tasks`。采集器只负责 `fetch -> transform -> save`,由 `data_jobs.py` worker 领取 `collect` / `clear_data` / `clear_cache` / `earth_refresh` 任务并回写进度。Earth 图层刷新关系集中在 `earth_layer_adapters.py`,不要再在单个采集器或按钮里手写缓存失效和 WebSocket 广播。
## 三、采集器列表
| 采集器 | 数据类型 | 数据内容 | 采集频率 |
@@ -90,7 +92,7 @@ async def run(self, db):
AIS 船只类采集器和其它 `CollectedData` 采集器的落库路径不同。BarentsWatch、AISStream 和自定义 `vessel_ais` 源都会进入 AIS 原始观测层,随后由聚合服务合并成 Earth 船只图层使用的 GeoJSON 和详情数据。这样做可以保留来源、传输方式、字段冲突和观测时间,避免某个实时源直接覆盖最终展示表。
Earth 国界不再属于采集器体系。它是 Earth 静态渲染资产,由控制台 `运维与配置 -> Earth 内容 -> 国界精度` 维护源配置,并由 `/api/v1/earth/boundaries/*` 构建 `frontend/public/earth/data/boundaries/earth-boundaries-china-pov-v1.pmtiles`。本地没有高精 PMTiles 时,前端会使用仓库内置的低精度 GeoJSON 作为 fallback不会向 `CollectedData` 写入国界记录。
智能星球国界不再属于采集器体系。它是智能星球静态渲染资产,由控制台 `运维与配置 -> 智能星球内容 -> 国界精度` 维护源配置,并由 `/api/v1/earth/boundaries/*` 构建 `frontend/public/earth/data/boundaries/earth-boundaries-china-pov-v1.pmtiles`。本地没有高精 PMTiles 时,前端会使用仓库内置的低精度 GeoJSON 作为 fallback不会向 `CollectedData` 写入国界记录。
TOP500 和 Epoch AI 算力数据的公开源不总是提供可用经纬度。Earth 统一算力中心接口在主地图启动链路中只使用源数据自带坐标或 `compute_center_locations` 维表坐标;缺少坐标的记录会进入 `unresolved`,不会通过本地注册表、国家质心或猜测城市自动渲染。用户手动采集候选时,后端会用源字段调用 ROR 组织注册 API 和 Nominatim/OpenStreetMap 在线搜索;候选经前端保存后写入 `compute_center_locations`,后续地图刷新再从维表渲染。
@@ -230,7 +232,7 @@ if datasource.last_status == "success":
### 采集管理与快照
Admin Next 的采集管理入口按业务层级组织:
Admin 的采集管理入口按业务层级组织:
- `采集器`:配置 endpoint、认证方式、请求头、基础参数、启用状态和凭证教程。
- `采集调度`:查看和调整调度状态,触发、停止或刷新采集任务。

View File

@@ -96,4 +96,4 @@ if (res.data.task_id) {
## 相关文件
- `backend/app/api/v1/datasources.py``_load_datasource_list_context``list_datasources`
- `frontend/src/pages/DataSources/DataSources.tsx``BuiltInDataSource` interface、`triggerDatasource`
- `frontend/src/admin/pages/PlainResourcePages.tsx``BuiltInDataSource` interface、`triggerDatasource`

View File

@@ -0,0 +1,140 @@
# 数据作业与 Outbox 技术架构
本文记录 Planet v1 的任务队列、数据库 outbox 和 Earth 刷新技术边界。每类数据产品的业务用途和端到端链路见 [业务架构与数据流转](/home/ray/dev/linkong/planet/docs/technical/zh/platform-data-flows.md)。
## 架构边界
- PostgreSQL 是 v1 的可靠任务账本和 outbox不引入 Kafka、Celery 或 RQ。
- `collection_tasks` 记录采集、删除、清缓存和非数据库触发的 Earth refresh 任务。
- `earth_data_change_events` 记录事实表或派生表变化,是 Earth 同步的可靠来源。
- `LISTEN/NOTIFY` 只做低延迟唤醒listener 仍会轮询未消费 outbox。
- Redis 主要用于缓存、认证辅助、OTP / rate limit、临时日志和 WebSocket 辅助,不是可靠队列。
## 数据库变化同步
```mermaid
flowchart LR
Write["Fact or derived table write"] --> Trigger["PostgreSQL trigger"]
Trigger --> Outbox["earth_data_change_events"]
Trigger --> Notify["planet_earth_data_changes"]
Outbox --> Listener["earth_db_change_listener"]
Notify --> Listener
Listener --> Adapter["earth_layer_adapters"]
Adapter --> Cache["cache invalidation"]
Cache --> WS["earth_updates"]
```
1. 采集、删除、定位或派生任务写入事实表或派生表。
2. statement trigger 为 `INSERT` / `UPDATE` / `DELETE` 写入 outbox。
3. listener 被 notify 唤醒,或通过轮询发现未消费事件。
4. listener 使用 `earth_layer_adapters.py``table + source` 映射成 Earth layer、刷新策略和 cache pattern。
5. listener 短窗口合并同 layer 事件,失效缓存并广播 `earth_updates`
6. 广播成功后标记 outbox consumed失败时保留待重试。
DB 变化不再默认创建 `earth_refresh` 任务,因此不会被同 source 的长采集或删除任务阻塞。`earth_refresh` 只保留给手动清缓存和非 DB 变化刷新提示。
## 数据作业队列
`collection_tasks` 是统一 job ledger。worker 使用 PostgreSQL `FOR UPDATE SKIP LOCKED` 领取 `queued` 任务;同一 `source` 的写任务串行,不同 source 可并行。
| task_type | 作用 |
| --- | --- |
| `collect` | 执行内置 datasource 采集 |
| `clear_data` | 删除该 source 的采集数据和声明过的派生数据 |
| `clear_cache` | 删除该 source 对应的 Earth / dashboard 缓存 |
| `earth_refresh` | 非 DB 变化场景下失效智能星球图层缓存并广播刷新提示 |
接口只创建任务并返回 `task_id`。任务执行、进度、取消和终态由 worker 写回 `collection_tasks`,并通过 `datasource_tasks` channel 通知前端。
取消语义是“保留已提交批次”:点击停止后,后端把任务标记为 `cancelling` 并取消内存中的执行协程;已经提交的批次保留,未完成批次按采集器或清理任务的回滚逻辑处理。
## Earth 同步事件模型
统一事件模型是 `earth.layer.changed`
```json
{
"event": "earth.layer.changed",
"action": "database_changed",
"source": "celestrak_tle",
"table": "collected_data",
"operation": "DELETE",
"layers": ["satellites"],
"refresh_strategy": "clear_then_reload",
"records_processed": 11125,
"occurred_at": "2026-05-25T10:20:30Z"
}
```
| strategy | 用途 |
| --- | --- |
| `clear_then_reload` | 先清前端本地图层对象,再强制重拉接口。删除数据时优先使用。 |
| `reload` | 保留旧对象直到新数据返回,适合定位、元数据或非破坏性更新。 |
| `delta` | 只用于 `earth_interactables`,按 id upsert 或 remove。 |
接口在真实 0 数据时必须返回 200 和空集合;只有真实接口异常才返回 5xx。前端收到删除事件后如果重拉失败应保持已清空状态并显示轻量错误不恢复旧对象。
## Layer Adapter 约定
`earth_layer_adapters.py` 是 source、派生表、Earth layer、缓存和刷新策略的唯一注册表。新图层只应新增 adapter不应在按钮 handler、采集器或前端分支里手写同步逻辑。
Adapter 必须声明:
- source 或 table 由哪个 Earth layer 消费。
- 需要清理哪些 Earth cache key pattern。
- `clear_data` 删除 source 时是否需要同时删除 owned 派生表。
- 该 layer 的默认刷新策略。
删除 source 时,`clear_data` 作业先删除 `collected_data.source = <source>`,再根据 adapter 删除 owned 派生表。直接修改派生表也会触发 outbox所以后台任务、管理接口和 SQL 修复脚本只要落到事实表或派生表Earth 都能感知变化。
## 运维排障
检查 outbox 是否堆积:
```sql
SELECT id, table_name, operation, source, occurred_at
FROM earth_data_change_events
WHERE consumed_at IS NULL
ORDER BY id
LIMIT 20;
```
检查触发器是否存在:
```sql
SELECT tgname, tgrelid::regclass
FROM pg_trigger
WHERE tgname LIKE 'tr_planet_%_changed_%'
ORDER BY 2, 1;
```
常用日志事件:
- `earth.db_changes.connected`listener 已连接 PostgreSQL 并开始监听。
- `earth.db_changes.outbox_polled`:轮询到了未消费 outbox。
- `earth.db_changes.broadcasted`:已产生 Earth 刷新广播。
- `data_job.started` / `data_job.completed`:任务执行状态。
如果 Earth 没更新按顺序检查事实表是否变化、outbox 是否消费、adapter 是否覆盖对应 `source/table`、listener 是否在线、前端 WebSocket 是否连接、visualization 接口是否返回 200 空集合或新数据。
## Kafka-ready 边界
业务代码不直接依赖具体队列实现,而是通过这些边界组织:
- `JobQueue`:提交、领取、取消、完成数据作业。
- `DataChangeBus`:发布数据库事实变化。
- `EarthLayerAdapterRegistry`:声明 source、layer、缓存和派生数据关系。
需要 Kafka 的信号:
- 多个独立服务需要消费同一批数据变化。
- AIS、BGP 或传感器流达到持续高吞吐。
- 需要 consumer group、事件回放、跨服务解耦。
需要 Spark 的信号:
- 历史数据到千万或亿级PostgreSQL 聚合开始吃力。
- 需要跨源、长时间窗口、空间时间关联分析。
- 原始数据进入 Parquet / Iceberg / Delta 等湖仓,并开始生产离线派生数据产品。
若目标是秒级连续流计算,优先评估 FlinkSpark 更适合批量或微批分析。

View File

@@ -32,8 +32,8 @@
文件:
- [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/pages/PlainResourcePages.tsx)
- [AdminNextRoutes.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/AdminNextRoutes.tsx)
- [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
- [AdminRoutes.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/AdminRoutes.tsx)
当前行为:
@@ -57,11 +57,11 @@
文件:
- [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/pages/PlainResourcePages.tsx)
- [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
当前行为:
- `/collection-management`旧版层级收敛为 `采集器``采集调度``采集历史 / 快照`
- `/collection-management`业务层级收敛为 `采集器``采集调度``采集历史 / 快照`
- `采集器` 是配置页,左侧列表展示采集器配置;右侧表单编辑 endpoint、认证、请求头、采集参数和启用状态。
- 新增采集器和目标 Schema 使用草稿详情页,不再用透明 JSON 弹窗;保存后才固化到列表,取消会销毁草稿。
- 连接按钮只做连通性测试,不保存配置;保存按钮只持久化表单。
@@ -80,7 +80,7 @@
- `目标 Schema` 维护可写入目标结构。
- `run-mapped``stop-mapped``stream-status` 负责运行映射后的自定义采集器。
这些入口在新版中需要表单化,只有高级字段才折叠为 JSON。不要把模板、Schema、运行状态和采集器配置拍平成同一张表。
这些入口在 Admin 中需要表单化,只有高级字段才折叠为 JSON。不要把模板、Schema、运行状态和采集器配置拍平成同一张表。
## 后端接口
@@ -334,7 +334,7 @@ GET /api/v1/vessels/snapshot?bbox=lon_min,lat_min,lon_max,lat_max&zoom=12&limit=
自定义源现在不是独立的新数据孤岛,而是作为内置数据源的补充源写入目标 schema。当前最完整的目标是 `vessel_ais`:自定义 REST 或 WebSocket 源经过确定性 mapping 后写入 AIS raw observations再通过 `vessels` WebSocket channel 推送给 Earth。
Earth 高精度边界不再使用自定义源目标 schema。国界是 Earth 静态资产,由控制台 `运维与配置 -> Earth 内容 -> 国界精度` 保存本机源配置并触发 PMTiles 构建,不写入 `CollectedData`
智能星球高精度边界不再使用自定义源目标 schema。国界是智能星球静态资产,由控制台 `运维与配置 -> 智能星球内容 -> 国界精度` 保存本机源配置并触发 PMTiles 构建,不写入 `CollectedData`
### 配置语义

View File

@@ -2,7 +2,7 @@
Docs Gatekeeper 把 `/docs` 从“前端构建时打包所有 Markdown”改成“后端按权限返回目录和正文”。它的目标是让公开使用手册、用户文档、开发文档和管理/运维文档在同一个 Docs 页面内可检索,但正文读取必须经过服务端白名单和用户权限检查。
用户侧说明见 [Planet 使用手册](/home/ray/dev/linkong/planet/docs/technical/zh/manual.md) 的 Docs 章节。
用户侧说明见 [智能星球使用手册](/home/ray/dev/linkong/planet/docs/technical/zh/manual.md) 的文档章节。
## 鉴权模型
@@ -16,7 +16,7 @@ Docs 使用两层权限:
| 组 | 用途 |
| --- | --- |
| `docs_user` | 用户操作类文档 |
| `docs_developer` | Earth、前端、后端、采集器和 AI Provider 开发文档 |
| `docs_developer` | 智能星球、前端、后端、采集器和 AI Provider 开发文档 |
| `docs_admin` | 服务控制、运维、环境变量和敏感操作文档 |
继承规则:
@@ -24,7 +24,7 @@ Docs 使用两层权限:
- 未登录用户只能读 `public`
- `docs_developer` 隐含 `docs_user`
- `docs_admin` 隐含 `docs_developer``docs_user`
- `admin``super_admin` 默认拥有全部 Docs 权限。
- `admin``super_admin` 默认拥有全部文档权限。
## 后端入口
@@ -61,15 +61,15 @@ DocsMetadata(
"public",
"Manual",
2,
"Planet 使用手册",
"Planet Manual",
"智能星球使用手册",
"Intelligent Planet Manual",
)
```
新增公开文档时,需要同步:
新增可在 Docs 页面展示的技术文档时,需要同步:
- 新增中英文 Markdown 文件。
- 在服务端 `DOCS_METADATA` 添加 filename、slug、access、group、order、标题。
- 在服务端 `DOCS_METADATA` 添加 filename、slug、access、group、order、标题。后端目录接口以这里为准,只改前端 metadata 不会让文档出现在 `/docs` 导航中。
- 在前端 [docs-content.ts](/home/ray/dev/linkong/planet/frontend/src/pages/Docs/docs-content.ts) 添加同名 metadata保持导航标题和排序一致。
- 如果需要从 README 发现,更新 `docs/technical/zh/README.md``docs/technical/en/README.md`
@@ -83,7 +83,7 @@ DocsMetadata(
- 更新用户时校验组名只能是 `docs_user``docs_developer``docs_admin`
- 只有 `super_admin` 能修改 Gatekeeper 权限组。
前端 [Users.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Users/Users.tsx) 展示权限组标签,并在编辑表单中提供多选框。非 `super_admin` 打开的表单会禁用该字段,并在提交前移除 `gatekeeper_groups`
前端 [Users.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/Users.tsx) 展示权限组标签,并在编辑表单中提供多选框。非 `super_admin` 打开的表单会禁用该字段,并在提交前移除 `gatekeeper_groups`
## 前端 Docs 加载

View File

@@ -188,7 +188,7 @@ Earth info-card 策略:
- 这是预期行为,因为 incident 是聚合和去噪后的结果
- 但 incident-first 渲染会让 Earth 显得过于安静,除非有另一层始终可用的 activity layer
推荐 `activity layer` 的实现细节在 [BGP 区域聚合计划](/home/ray/dev/linkong/planet/docs/plans/earth-bgp-region-aggregation-plan.md) 中展开
推荐 `activity layer`未来实现细节保存在仓库路径 `docs/plans/earth-bgp-region-aggregation-plan.md`
因此最近的里程碑是:
@@ -259,7 +259,7 @@ Earth 的 `incident` 层不应该像一大片发光区域,而应该像紧凑
相关页面:
- `frontend/src/pages/BGP/BGP.tsx`
- `frontend/src/admin/pages/PlainResourcePages.tsx`
当前 BGP 控制台页面有三层:
@@ -321,7 +321,7 @@ BGP 专项测试位于:
前端:
- `frontend/src/pages/BGP/BGP.tsx`
- `frontend/src/admin/pages/PlainResourcePages.tsx`
- `frontend/public/earth/js/bgp.js`
- `frontend/public/earth/js/main.js`
- `frontend/public/earth/js/info-card.js`

View File

@@ -1,10 +1,10 @@
# Earth 前端结构
# 智能星球前端结构
本文件描述当前 Earth 大屏前端的真实结构,重点是帮助后续继续改 HUD、图层、媒体面板、真实地形、BGP 可视化时,不再重复踩结构和状态同步上的坑。
本文件描述当前智能星球前端的真实结构,重点是帮助后续继续改 HUD、图层、媒体面板、真实地形、BGP 可视化时,不再重复踩结构和状态同步上的坑。
相关规则建议一起参考:
- [项目规则](/home/ray/dev/linkong/planet/rules.md)
- 仓库根目录 `rules.md`
- [前端布局指南](/home/ray/dev/linkong/planet/docs/technical/zh/frontend-layout-guidelines.md)
## 当前目标
@@ -56,6 +56,8 @@ React 路由入口:
- 各图层集成
- Earth 级别状态同步
Earth 收到 `/ws``earth_updates` 时只把它当作刷新提示,真实数据仍通过 `/api/v1/visualization/...` 接口重新 GET。数据库驱动的刷新由后端 listener 直接清理缓存再广播,不再默认经过 `earth_refresh` 作业队列;前端收到 `database_changed` 后会按 layer 读取 `clear_then_reload``reload``delta` 策略。`clear_then_reload` 必须先清 Three.js 对象再 no-store 重拉summary 只做一致性校验,不能用 `0` 作为跳过图层重拉的理由。技术链路见 [数据作业与 Outbox 技术架构](/home/ray/dev/linkong/planet/docs/technical/zh/data-job-earth-sync-architecture.md),业务数据流见 [业务架构与数据流转](/home/ray/dev/linkong/planet/docs/technical/zh/platform-data-flows.md)。
### 3. 地球控制层
- [controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js)
@@ -136,7 +138,7 @@ Browser Camera provider 的手势识别管线在 [motion-browser-provider.js](/h
- terrain tile 拉取、解码、位移、着色
- 海陆基座与国界底图的整球 overlay
Earth 地表是多层近似同心球,不是单一 mesh。`earth.js` 的基座球、高清材质 overlay、云层/大气,以及 `country-boundaries.js` 的海陆基座都需要明确半径间距。远距视图下 GPU 深度精度会下降,相邻 shell 过近会 z-fighting表现为黑色闪烁块或雪花。当前稳定策略是让海陆基座使用 `landAltitudeOffset = 0.32`,高清材质使用 `textureOverlayAltitudeOffset = 0.48`;后续新增或调整整球地表 overlay 时,必须同步检查 [Earth 渲染图层顺序](/home/ray/dev/linkong/planet/docs/technical/zh/earth-render-layer-order.md),并在 50% 缩放视图验证。
智能星球地表是多层近似同心球,不是单一 mesh。`earth.js` 的基座球、高清材质 overlay、云层/大气,以及 `country-boundaries.js` 的海陆基座都需要明确半径间距。远距视图下 GPU 深度精度会下降,相邻 shell 过近会 z-fighting表现为黑色闪烁块或雪花。当前稳定策略是让海陆基座使用 `landAltitudeOffset = 0.32`,高清材质使用 `textureOverlayAltitudeOffset = 0.48`;后续新增或调整整球地表 overlay 时,必须同步检查 [智能星球渲染图层顺序](/home/ray/dev/linkong/planet/docs/technical/zh/earth-render-layer-order.md),并在 50% 缩放视图验证。
### 7. 图层模块
@@ -160,13 +162,13 @@ Earth 地表是多层近似同心球,不是单一 mesh。`earth.js` 的基座
`tv.js` 管理 `media-panel` 里的直播 / 态势新闻 tab。toolbar 打开或切换 TV/新闻时,会通过 `earth:tv-visibility-change``earth:tv-tab-change` 回写 Earth 设置:面板可见性仍按 desktop/mobile viewport 存在 `views.<scope>.panelVisibility.media-panel`,当前 tab 存在 `shared.mediaPanelActiveTab`,因此刷新页面后能恢复用户上次打开的直播或新闻状态。`closeTransientMobileOverlays()` 这类临时收起会带 `persist:false`,不会覆盖用户偏好。
`brand.js` 管理 Earth HUD 品牌资源。默认品牌来自静态资源,运行时覆盖值来自 `/api/v1/earth/brand`,上传的图片通过 `/earth-brand-assets/...` 读取。前端必须把 logo/title 图片和文本 fallback 分开处理:图片加载失败时显示文本标题,文本字段为空时使用后端默认值,避免 HUD 品牌区空白。控制台的 Earth 内容页负责保存和重置品牌配置,Earth 前端只消费结果。
`brand.js` 管理智能星球 HUD 品牌资源。默认品牌来自静态资源,运行时覆盖值来自 `/api/v1/earth/brand`,上传的图片通过 `/earth-brand-assets/...` 读取。前端必须把 logo/title 图片和文本 fallback 分开处理:图片加载失败时显示文本标题,文本字段为空时使用后端默认值,避免 HUD 品牌区空白。控制台的智能星球内容页负责保存和重置品牌配置,智能星球前端只消费结果。
`about.js` 管理 Earth 设置里的“关于”卡片。默认内容仍保留在前端作为兜底,运行时优先读取 `/api/v1/earth/about`。接口失败或字段缺失时必须回退默认值,避免设置页出现空白。Admin Next 的 Earth 内容页提供“关于”tab保存走 `PUT /api/v1/earth/about`,恢复默认走 `DELETE /api/v1/earth/about`
`about.js` 管理智能星球设置里的“关于”卡片。默认内容仍保留在前端作为兜底,运行时优先读取 `/api/v1/earth/about`。接口失败或字段缺失时必须回退默认值,避免设置页出现空白。控制台的智能星球内容页提供“关于”tab保存走 `PUT /api/v1/earth/about`,恢复默认走 `DELETE /api/v1/earth/about`
`oobe.js` 管理 Earth 首次初始化引导。是否显示 OOBE 必须由 `/api/v1/earth/oobe-status``ready` 字段决定,不能依赖 `localStorage` 判断系统是否初始化。`localStorage` 只允许记录“本浏览器暂时跳过”的短时状态;如果后端已经认为 `ready: true`,退出登录、清空本地缓存或换浏览器都不应再次弹出 OOBE。桌面端使用深色星空遮罩和毛玻璃启动面板移动端改为底部 sheet并尊重 `prefers-reduced-motion`
Admin Next 的 Earth 内容页必须按运行时语义组织这些配置:
控制台的智能星球内容页必须按运行时语义组织这些配置:
- `品牌标识`:品牌预览应使用与 Earth HUD 左上角一致的深色星空背景、尺寸、间距、logo/title 渲染和文本 fallback而不是普通表单预览。
- `关于`:配置 Earth 设置里的 About 卡片,包括 logo、眉标、标题、版本、描述和元信息条目Earth 运行时从 `/earth/about` 读取,失败时回退默认内容。
@@ -197,9 +199,7 @@ TV 预览需要尽量复用 Earth 运行时的直播卡片结构和状态标签
当前 BGP 巡航只是这套能力的一个调用方不应再把“按队列巡航”和“BGP 事件展示”混写在同一个状态机里。
新闻巡航摘要计划见:
- [Earth 新闻巡航摘要计划](/home/ray/dev/linkong/planet/docs/plans/earth-news-cruise-summary-plan.md)
新闻巡航摘要的未来计划保存在仓库路径 `docs/plans/earth-news-cruise-summary-plan.md`,不作为公开 Docs 页面入口。
## 当前样式分层
@@ -385,7 +385,7 @@ asset 图标大小由 `Interactable` 的 `icon.fitSize` 控制。SVG / 图片文
接口细节、生命周期和接入示例见:
- [Earth 可交互图标接入](/home/ray/dev/linkong/planet/docs/technical/zh/earth-interactable-usage.md)
- [智能星球可交互图标接入](/home/ray/dev/linkong/planet/docs/technical/zh/earth-interactable-usage.md)
### 视角控制反馈
@@ -457,7 +457,7 @@ Earth 设置面板当前由 [controls.js](/home/ray/dev/linkong/planet/frontend/
SGP4 传播输出是惯性系位置,不能直接当成 Earth 的经纬度固定坐标使用。`satellites.js` 会用当前时间的 `gstime` 把 ECI/TEME 位置转换到 ECF再映射到 `latLonToVector3()` 使用的 Three.js 坐标轴。卫星点和短尾迹使用随采样时间变化的地固坐标,表示相对当前地球表面的实际位置;锁定后的预测轨道线使用锁定时刻固定的 `gstime`把未来一圈惯性轨道投到当前地球姿态上显示因此会闭合并且轨道面倾角应与详情卡一致。fallback 预测轨道也必须使用真正的 RAAN + inclination 轨道平面公式,不能把 inclination 当成恒定纬度。
国界精度偏好独立存储在 `country-boundaries.js``planet.earth.boundaries.highPrecisionEnabled`。未开启高精时,即使本机已经有高精 manifest/PMTiles也继续加载低精 `countries-admin0.min.geojson` fallback开启高精但高精产物缺失时Earth 工具栏设置会调用 `/api/v1/earth/boundaries/build` 启动后台构建并轮询进度。构建成功后调用 `reloadCountryBoundaries()` 热切换,不再刷新整个页面。国界 hover 与 tooltip 解耦:只要地表坐标落在国界 polygon 内就保持高亮如果鼠标同时命中卫星、船只、BGP 等 interactabletooltip 显示 interactable 信息,但国界高亮不应闪烁。
国界精度偏好独立存储在 `country-boundaries.js``planet.earth.boundaries.highPrecisionEnabled`。未开启高精时,即使本机已经有高精 manifest/PMTiles也继续加载低精 `countries-admin0.min.geojson` fallback开启高精但高精产物缺失时智能星球工具栏设置会调用 `/api/v1/earth/boundaries/build` 启动后台构建并轮询进度。构建成功后调用 `reloadCountryBoundaries()` 热切换,不再刷新整个页面。国界 hover 与 tooltip 解耦:只要地表坐标落在国界 polygon 内就保持高亮如果鼠标同时命中卫星、船只、BGP 等 interactabletooltip 显示 interactable 信息,但国界高亮不应闪烁。
## 当前地形链路

View File

@@ -1,6 +1,6 @@
# Earth Interactable 使用说明
# 智能星球 Interactable 使用说明
`Interactable` Earth 地表“图标类可交互元素”的通用渲染入口。它把船只图层验证过的模式抽成公共能力:普通态用批量 `THREE.Points`hover / locked 用少量 overlay拾取走屏幕空间命中图标资源统一转进 canvas texture并在公共层处理 glow、状态、尺寸、贴地渲染和同坐标避让。
`Interactable`智能星球地表“图标类可交互元素”的通用渲染入口。它把船只图层验证过的模式抽成公共能力:普通态用批量 `THREE.Points`hover / locked 用少量 overlay拾取走屏幕空间命中图标资源统一转进 canvas texture并在公共层处理 glow、状态、尺寸、贴地渲染和同坐标避让。
当前已接入:

View File

@@ -1,8 +1,8 @@
# Earth 图层样式属性索引
# 智能星球图层样式属性索引
本文记录当前 Earth 前端各图层的材质、颜色、透明度、线宽、半径偏移和
`renderOrder` 等样式属性。层级关系请配合
[Earth 渲染图层顺序](/home/ray/dev/linkong/planet/docs/technical/zh/earth-render-layer-order.md)
[智能星球渲染图层顺序](/home/ray/dev/linkong/planet/docs/technical/zh/earth-render-layer-order.md)
查看。
## 命名约定
@@ -20,12 +20,12 @@
| 正式名称 | 变量名 | 当前值 | 使用位置 / 说明 |
| --- | --- | --- | --- |
| Earth 基座半径 | `CONFIG.earthRadius` | `100` | `earth.js:createEarth()` |
| Earth 基座颜色 | `EARTH_MATERIAL_CONFIG.color` | `0x010609` | `MeshPhongMaterial.color` |
| Earth 基座 emissive | `EARTH_MATERIAL_CONFIG.emissive` | `0x010609` | `MeshPhongMaterial.emissive` |
| Earth 基座 specular | `EARTH_MATERIAL_CONFIG.specular` | `0x1a2d45` | `MeshPhongMaterial.specular` |
| Earth 基座 shininess | `EARTH_MATERIAL_CONFIG.shininess` | `12` | `MeshPhongMaterial.shininess` |
| Earth 基座 opacity | `EARTH_MATERIAL_CONFIG.opacity` | `1` | `MeshPhongMaterial.opacity` |
| 智能星球基座半径 | `CONFIG.earthRadius` | `100` | `earth.js:createEarth()` |
| 智能星球基座颜色 | `EARTH_MATERIAL_CONFIG.color` | `0x010609` | `MeshPhongMaterial.color` |
| 智能星球基座 emissive | `EARTH_MATERIAL_CONFIG.emissive` | `0x010609` | `MeshPhongMaterial.emissive` |
| 智能星球基座 specular | `EARTH_MATERIAL_CONFIG.specular` | `0x1a2d45` | `MeshPhongMaterial.specular` |
| 智能星球基座 shininess | `EARTH_MATERIAL_CONFIG.shininess` | `12` | `MeshPhongMaterial.shininess` |
| 智能星球基座 opacity | `EARTH_MATERIAL_CONFIG.opacity` | `1` | `MeshPhongMaterial.opacity` |
| 高清材质半径偏移 | `EARTH_MATERIAL_CONFIG.textureOverlayAltitudeOffset` | `0.48` | 独立高清材质球半径;必须与海陆基座和地球基座保持足够深度间距,避免远距 z-fighting |
| 高清材质透明度 | `EARTH_MATERIAL_CONFIG.textureOverlayOpacity` | `0.88` | 高清材质 `MeshPhongMaterial.opacity` |
| 高清材质 renderOrder | `EARTH_MATERIAL_CONFIG.textureOverlayRenderOrder` | `0.96` | `_earthTextureOverlay.renderOrder` |

View File

@@ -1,4 +1,4 @@
# Earth 渲染图层顺序
# 智能星球渲染图层顺序
本文记录当前 Earth 渲染器的图层顺序和每层意图。后续调整
`renderOrder`、半径偏移、深度策略或指针交互时,需要同步更新这里。
@@ -14,9 +14,9 @@
| 顺序 | 图层 | 来源 | 渲染 / 半径策略 | 深度 / 交互策略 | 备注 |
| --- | --- | --- | --- | --- | --- |
| -1000 | 天体背景 mesh | `celestial.js` | 背景球 | 不参与地表拾取 | 位于所有 Earth 内容之后。 |
| -1 | Earth 遮挡球 | `earth.js` | 地球内侧不可见球 | 写入深度 | 遮挡地球背面的对象。 |
| 0 | Earth 基座球 | `earth.js` | `CONFIG.earthRadius` | 地表拾取兜底目标 | 深色基座,所有可选地图层关闭时仍可见。 |
| -1000 | 天体背景 mesh | `celestial.js` | 背景球 | 不参与地表拾取 | 位于所有智能星球内容之后。 |
| -1 | 智能星球遮挡球 | `earth.js` | 地球内侧不可见球 | 写入深度 | 遮挡地球背面的对象。 |
| 0 | 智能星球基座球 | `earth.js` | `CONFIG.earthRadius` | 地表拾取兜底目标 | 深色基座,所有可选地图层关闭时仍可见。 |
| 0.2 | 国界暗色 tint | `country-boundaries.js` | `tintAltitudeOffset` | 禁用 raycast | 高清材质关闭时使用。 |
| 0.86 | 海陆基座填充 | `country-boundaries.js` | `landAltitudeOffset = 0.32`; 海洋 `#010609`,陆地 `#080f1b` | 禁用 raycast | 即使国界线关闭,基座地图仍保持可用;半径与基座球拉开以避免远距 z-fighting。 |
| 0.96 | 高清 Earth 材质 | `earth.js` | `textureOverlayAltitudeOffset = 0.48` | 可见时作为地表拾取目标 | 高清材质始终压过海陆基座填充;半径必须高于海陆基座并与基座球保持足够间距。 |
@@ -63,7 +63,7 @@ Earth 的地表不是单一 mesh而是多层近似同心球基座球、海
| 交互 | 当前规则 |
| --- | --- |
| Earth 坐标 hover | 高清材质可见时使用高清材质 overlay 作为地表拾取目标,否则使用 Earth 基座球。 |
| 智能星球坐标 hover | 高清材质可见时使用高清材质 overlay 作为地表拾取目标,否则使用智能星球基座球。 |
| 国界 hover | 先把地表拾取坐标转成经纬度,再用 GeoJSON 点面判断;国界 hover 线本身不接收 raycast。 |
| 国界 hover 视觉 | hover 时压暗普通国界线,并绘制无深度测试的光晕和实线。 |
| 中国 / 台湾 hover | `CHN``TWN` 被归到同一个 hover 高亮组tooltip 仍显示鼠标实际命中的 feature。 |

View File

@@ -1,10 +1,10 @@
# Earth 卫星覆盖策略
# 智能星球卫星覆盖策略
本文件记录 Earth 卫星图层当前关于 `footprint` 的产品边界、资料依据和已落地实现,目标是避免把 Starlink 这套专用地表覆盖模型误用到其它星座上。
相关上下文:
- [Earth 前端结构](/home/ray/dev/linkong/planet/docs/technical/zh/earth-frontend-context.md)
- [智能星球前端结构](/home/ray/dev/linkong/planet/docs/technical/zh/earth-frontend-context.md)
- [数据采集系统](/home/ray/dev/linkong/planet/docs/technical/zh/backend-collectors.md)
- [backend/app/services/collectors/celestrak.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/celestrak.py)
- [frontend/public/earth/js/satellites.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/satellites.js)
@@ -17,26 +17,21 @@
## 本地实际类别
当前 CelesTrak 卫星分组在 [backend/app/services/collectors/celestrak.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/celestrak.py) 中包括:
当前 CelesTrak 采集器优先从 `GROUP=active&FORMAT=json` 拉取完整活跃卫星目录。这样可以避免某个 CelesTrak 分组请求失败时仍把不完整结果保存为成功批次。采集器只在完整 JSON 数组可解析、且记录含 `NORAD_CAT_ID` 时进入转换和保存;网络、续传或解析失败会重试,最终失败时保留上一批 current 数据。
- `starlink`
- `gps-ops`
- `galileo`
- `glonass`
- `beidou`
- `leo`
- `geo`
- `iridium-next`
CelesTrak 对 `active` 这类大 group 有每次 GP 数据更新窗口内的重复下载限制。为避免清空数据库后无法立即恢复Planet 会把原始下载文件保存在 `$PLANET_CACHE_DIR/downloads`。当 `active` 返回“本轮数据未更新”的 403 时:
其中非 Starlink 类别是:
- 如果 `active` 缓存存在,直接用缓存重新写入数据库。
- 如果 `active` 缓存不存在,才切换到 fallback group mode`starlink``gps-ops``galileo``glonass``beidou``leo``geo``iridium-next` 全部下载并合并。
- fallback group mode 要求所有 group 都成功或有缓存可复用;任意 group 缺失都会整体失败,不保存 partial。
- `gps-ops`
- `galileo`
- `glonass`
- `beidou`
- `leo`
- `geo`
- `iridium-next`
采集结果仍会给前端提供 `metadata.constellation_group`,但该字段现在来自可执行推断:
- `OBJECT_NAME``STARLINK` 开头时标记为 `starlink`
- `OBJECT_NAME``IRIDIUM` 开头时标记为 `iridium-next`
- 其它活跃卫星不强行归入旧 CelesTrak 小分组,避免把泛化类别当成精确星座
因此,非 Starlink 类别在产品策略中仍包括 GNSS/RNSS、GEO、generic LEO 和 Iridium NEXT 等语义;但只有 fallback group mode 会保存旧的 `gps-ops``galileo``glonass``beidou``leo``geo` 分组标签active 主路径不会强行给所有卫星补这类标签。
## 资料结论
@@ -136,7 +131,8 @@
1. 后端把星座分组和 footprint 策略提示透给前端
- CelesTrak collector 会把 `GROUP` 记入 `metadata.constellation_group`
- CelesTrak collector 会把原始查询来源记入 `metadata.celestrak_query_group = active`
- `metadata.constellation_group` 只保存可推断的业务星座,例如 `starlink``iridium-next`
- Visualization API 会输出:
- `properties.constellation_group`
- `properties.footprint_policy`

View File

@@ -1,10 +1,10 @@
# Earth 工具栏与浮层协同
# 智能星球工具栏与浮层协同
本文件描述 Earth 大屏右侧工具栏按钮,以及搜索面板、设置弹窗、新闻直播面板、图层面板这几个浮层之间当前的协同规则。改交互、加按钮、调整面板时按这个表对齐,避免出现「点 A 把不该关的 B 也关了」之类的协同冲突。
相关入口:
- [Earth 前端结构](/home/ray/dev/linkong/planet/docs/technical/zh/earth-frontend-context.md)
- [智能星球前端结构](/home/ray/dev/linkong/planet/docs/technical/zh/earth-frontend-context.md)
- [前端布局指南](/home/ray/dev/linkong/planet/docs/technical/zh/frontend-layout-guidelines.md)
## 工具栏按钮目录

View File

@@ -1,6 +1,6 @@
# 常见问题
这页集中收录本地启动、Windows / WSL、依赖、动捕、凭证和 Docs 权限相关的常见排障路径。更完整的背景说明仍在对应专题文档中,这里只保留最常用的判断顺序和命令。
这页集中收录本地启动、Windows / WSL、依赖、动捕、凭证和文档权限相关的常见排障路径。更完整的背景说明仍在对应专题文档中,这里只保留最常用的判断顺序和命令。
## 启动与端口
@@ -321,7 +321,7 @@ Docs 按 Gatekeeper 权限组控制可见性:
- 快速开始、使用手册、FAQ 等基础文档公开可见。
- 开发文档通常需要 `docs_developer`
- 运维和服务控制文档通常需要 `docs_admin`
- `admin``super_admin` 默认具备 Docs 权限;普通用户需要在控制台“用户管理”中分配权限组。
- `admin``super_admin` 默认具备文档权限;普通用户需要在控制台“用户管理”中分配权限组。
## Earth 常见操作
@@ -332,9 +332,9 @@ Docs 按 Gatekeeper 权限组控制可见性:
要启用高精国界,有两条入口:
- Earth 页面齿轮设置里的“国界精度”:切到“高精”会启动首次后台下载/构建,并显示百分比,完成后自动应用。
- 控制台 `运维与配置 -> Earth 内容 -> 国界精度`:适合查看 provider、manifest、PMTiles、fallback 状态,编辑源配置 JSON或手动重建。
- 控制台 `运维与配置 -> 智能星球内容 -> 国界精度`:适合查看 provider、manifest、PMTiles、fallback 状态,编辑源配置 JSON或手动重建。
如果看到“更新源未配置完整”,先到 `Earth 内容 -> 国界精度` 保存源配置;本机私有配置写入 `config/earth-boundary-sources.local.json`,不要提交到仓库。没有高精产物时,使用低精 fallback 是正常行为。
如果看到“更新源未配置完整”,先到 `智能星球内容 -> 国界精度` 保存源配置;本机私有配置写入 `config/earth-boundary-sources.local.json`,不要提交到仓库。没有高精产物时,使用低精 fallback 是正常行为。
### Earth 位置候选采集后没有写入怎么办?
@@ -342,9 +342,9 @@ Docs 按 Gatekeeper 权限组控制可见性:
算力中心候选保存后会写入 `compute_center_locations`。没有可用候选的记录会保留在待定位列表中,系统不会用国家中心点或硬编码 hint 伪造位置。
### Earth 品牌 logo 或标题改完后为什么没恢复默认?
### 智能星球品牌 logo 或标题改完后为什么没恢复默认?
Earth 品牌资源在控制台 `运维与配置 -> Earth 内容 -> 品牌资源` 中维护。上传图片后页面会使用返回的 Earth 品牌资产地址如果只是清空标题、ARIA 文案等文本字段,系统会回退到默认标题,避免出现空白品牌。
智能星球品牌资源在控制台 `运维与配置 -> 智能星球内容 -> 品牌资源` 中维护。上传图片后页面会使用返回的智能星球品牌资产地址如果只是清空标题、ARIA 文案等文本字段,系统会回退到默认标题,避免出现空白品牌。
要恢复发布包自带的默认 logo、标题图和文案使用“重置品牌资源”。只刷新 Earth 页面不会删除已经保存的运行时品牌配置。

View File

@@ -4,7 +4,7 @@
相关规则建议一起参考:
- [项目规则](/home/ray/dev/linkong/planet/rules.md)
- 仓库根目录 `rules.md`
- [前端布局指南](/home/ray/dev/linkong/planet/docs/technical/zh/frontend-layout-guidelines.md)
## 当前目标
@@ -22,7 +22,7 @@
- [App.tsx](/home/ray/dev/linkong/planet/frontend/src/App.tsx)
当前正式后台路由已经由 Admin Next 接管:
当前正式后台路由已经由控制台接管:
- `/admin`
- `/users`
@@ -37,23 +37,7 @@
- `/collection-management`
- `/settings`
这些路径渲染 [AdminNextRoutes.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/AdminNextRoutes.tsx),页面清单和菜单元信息来自 [manifest.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/routes/manifest.tsx)。`/admin-next/*` 仅作为兼容入口存在,会重定向到上面的正式路径,不再作为并行主入口
旧 AntD 控制台保留在 `/legacy/admin/*`,用于对照和回退:
- `/legacy/admin`
- `/legacy/admin/datasources`
- `/legacy/admin/data`
- `/legacy/admin/collection-management`
- `/legacy/admin/earth-content`
- `/legacy/admin/ai`
- `/legacy/admin/logs`
- `/legacy/admin/settings`
- `/legacy/admin/users`
- `/legacy/admin/bgp`
- `/legacy/admin/alerts/*`
旧页面、`AppLayout``antd``@ant-design/icons` 在 legacy 验收期继续保留。不要在新版 parity 验收前删除这些文件或依赖。
这些路径渲染 [AdminRoutes.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/AdminRoutes.tsx),页面清单和菜单元信息来自 [manifest.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/routes/manifest.tsx)。控制台是唯一后台入口,不再维护并行控制台或回退路由
`/earth` 是独立展示页,不属于控制台骨架。
@@ -61,7 +45,7 @@
正式后台公共壳层在:
- [AdminNextLayout.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/components/layout/AdminNextLayout.tsx)
- [AdminLayout.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/components/layout/AdminLayout.tsx)
职责:
@@ -69,38 +53,13 @@
- 当前账号、版本、退出登录和主题切换
- 顶部搜索、面包屑和页面快捷入口
- 内容区单屏高度闭合
- Admin Next 内部滚动、表格、详情面板和移动端详情视图协调
- 控制台内部滚动、表格、详情面板和移动端详情视图协调
旧 AntD legacy 壳层仍在:
后续正式控制台页面应适配 `AdminLayout` 和控制台页面模式;不要重新引入并行后台壳层。
- [AppLayout.tsx](/home/ray/dev/linkong/planet/frontend/src/components/AppLayout/AppLayout.tsx)
## 控制台分区加载策略
legacy 职责
- 左侧导航
- 折叠与展开
- 当前账号/版本信息
- 内容区高度闭合
- 全站统一侧边栏滚动条
当前结构是:
```tsx
<Layout className="dashboard-layout">
<Sider className="dashboard-sider">...</Sider>
<Layout>
<Content className="dashboard-content">
<div className="dashboard-content-inner">{children}</div>
</Content>
</Layout>
</Layout>
```
后续正式控制台页面应优先适配 `AdminNextLayout` 和 Admin Next 页面模式,而不是继续往旧 `AppLayout` 增加新能力。只有维护 `/legacy/admin/*` 时才应修改旧壳层。
## Admin Next 分区加载策略
多 tab 页面由 [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/pages/PlainResourcePages.tsx) 统一承载当前的管理型和信息型工作台。分区加载规则是:
多 tab 页面由 [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx) 统一承载当前的管理型和信息型工作台。分区加载规则是
- 初次进入页面只请求当前 active tab不预先拉取所有 tab 的接口。
- 用户切换 tab 时懒加载该 tab已经加载过的 tab 保留在本地 `states` 缓存中,切回时直接复用。
@@ -112,19 +71,22 @@ legacy 职责:
## 数据源采集队列
Admin Next 的数据源页把单源触发、批量触发和触发全部统一接入浏览器下载列表式采集队列:
Admin 的数据源页把单源触发、表格勾选触发和触发全部统一接入浏览器下载列表式采集队列:
- 队列状态由 [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/pages/PlainResourcePages.tsx) 管理,只保存当前会话中的可见任务,不用 `localStorage` 伪造历史。
- 队列状态由 [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx) 管理,只保存当前会话中的可见任务,不用 `localStorage` 伪造历史。
- 任务进度优先消费 `/ws``datasource_tasks` channel如果 WebSocket 未连接或没有及时返回,则轮询 `/api/v1/datasources/{id}/task-status`
- 触发接口返回的 `triggered``skipped``failed` 会立即进入队列;刷新页面后只根据后端当前仍在 `running/pending/queued` 的数据源恢复队列。
- 页面上方只显示紧凑进度条和数量摘要;点击“查看队列”展开底部面板,按运行中、失败、完成、跳过分组
- `内置源` 分区通过表格选择列收敛批量触发。没有勾选时主按钮是“触发全部”;勾选后同一个主按钮变成“触发已选 N”不再提供手填 ID 的独立批量按钮
- 页面内容流不再承载展开队列,避免全量触发后挤压列表和详情面板。右上角 actions 区的队列按钮沿用现有 `Button` 样式;空态使用 `ListChecks` 图标,有任务时只显示纯圆环总进度。点击后打开浮层,按运行中、失败、完成、跳过分组。
- 队列项可以跳转到对应数据源详情,失败项可以重试。详情页内的“采集任务”摘要只展示当前数据源最近任务,不承担保存配置职责。
- 运行中的单源采集按钮显示为“停止采集”;队列中未完成项右侧提供取消按钮。取消调用 `/api/v1/datasources/{source_id}/tasks/{task_id}/cancel`,后端语义是保留已提交批次并回滚未完成批次。
- 删除数据库数据和清理展示缓存也会进入同一队列,前端只展示任务状态,不假定接口同步完成。
这个队列是用户感知层,不替代后端调度状态。后端仍然是任务是否运行、完成、失败或跳过的唯一事实来源。
## Admin Next 主题滑块
## 控制台主题滑块
Admin Next 侧栏底部主题切换继续复用共享 [SegmentedControl.tsx](/home/ray/dev/linkong/planet/frontend/src/components/SegmentedControl/SegmentedControl.tsx),但主题变量在 [styles.css](/home/ray/dev/linkong/planet/frontend/src/admin-next/styles.css) 内跟随 `data-theme` 覆盖:
Admin 侧栏底部主题切换继续复用共享 [SegmentedControl.tsx](/home/ray/dev/linkong/planet/frontend/src/components/SegmentedControl/SegmentedControl.tsx),但主题变量在 [styles.css](/home/ray/dev/linkong/planet/frontend/src/admin/styles.css) 内跟随 `data-theme` 覆盖:
- light 下使用 `--d-segment-bg: #eef3f9`、白色 slider 和轻投影。
- dark 下使用与 Docs 一致的深色底座、`#202938` slider 和深色外投影。
@@ -163,8 +125,7 @@ Admin Next 侧栏底部主题切换继续复用共享 [SegmentedControl.tsx](/ho
当前使用场景:
- Admin Next 数据源、采集数据、采集管理、日志、告警和 BGP 页面
- 旧 AntD legacy 页面通过兼容封装继续使用共享滚动能力
- 控制台数据源、采集数据、采集管理、日志、告警和 BGP 页面
### 3. `TableScrollRegion`
@@ -187,10 +148,10 @@ Admin Next 侧栏底部主题切换继续复用共享 [SegmentedControl.tsx](/ho
用途:
- Admin Next 全局工具按钮和详情页工具按钮
- 控制台全局工具按钮和详情页工具按钮
- icon-only + tooltip 的普通操作
- 保存、创建、确认、删除、停止等强意图操作
- Docs 主题滑块一致的紧凑开关
-文档主题滑块一致的紧凑开关
当前约束:
@@ -237,7 +198,7 @@ Admin Next 侧栏底部主题切换继续复用共享 [SegmentedControl.tsx](/ho
- 渲染 `/docs` 的 Markdown 正文
- 支持标题、列表、引用、代码块、表格和基础行内格式
- 代码块和表格内部复用 `Scrollbar`,避免横向内容撑爆文档页
- Docs 正文由后端 `/api/v1/docs/...` 按 Gatekeeper 权限返回;前端只渲染当前用户可见内容
- 文档正文由后端 `/api/v1/docs/...` 按 Gatekeeper 权限返回;前端只渲染当前用户可见内容
当前约束:
@@ -245,39 +206,19 @@ Admin Next 侧栏底部主题切换继续复用共享 [SegmentedControl.tsx](/ho
- 文档内部链接应通过 `transformLink` 转成 `/docs/:slug`
- 标题锚点由 `getHeadingId` 注入,避免渲染器自己理解路由状态
### 7. `ConnectionTestInput`
### 7. 控制台 UI primitives
文件:
- [ConnectionTestInput.tsx](/home/ray/dev/linkong/planet/frontend/src/components/ConnectionTestInput/ConnectionTestInput.tsx)
- [button.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/components/ui/button.tsx)
- [dialog.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/components/ui/dialog.tsx)
- [switch.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/components/ui/switch.tsx)
用途:
- Endpoint、Base URL 这类“输入值 + 连接验证”的控制台表单项
- AI Provider 和 WebSearch 的连接测试入口
- 后续采集器配置如果把连接测试放进输入框,也应复用它
当前约束:
- 输入框末端只显示一个插头/连接器图标,不再并排放“测试连接”文字按钮
- 禁用的集成能力必须同时置灰输入框和连接测试按钮
- 组件只负责输入框与测试入口组合不保存业务状态调用方仍负责表单值、loading、disabled 和连接请求
### 8. `TableActions`
文件:
- [TableActions.tsx](/home/ray/dev/linkong/planet/frontend/src/components/TableActions/TableActions.tsx)
用途:
- 表格操作列的统一操作入口
- 展开状态下直接展示按钮
- 收起状态下用更多菜单承载操作
配套导出:
- `actionCellProps`:用于操作列 `onCell`,防止操作按钮被省略号截断或换行
- 全局工具按钮、详情页动作、确认弹窗和二元设置。
- 与 Tactile UI token 对齐保持控制台内部控件尺寸、hover、disabled 和 dark mode 一致。
- 表格行内动作优先使用 icon button + tooltip/title不重新引入独立操作菜单组件。
## 当前状态来源
@@ -300,7 +241,7 @@ Admin Next 侧栏底部主题切换继续复用共享 [SegmentedControl.tsx](/ho
文件:
- [AISettings.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/AISettings/AISettings.tsx)
- [AISettings.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
职责:
@@ -336,7 +277,7 @@ Admin Next 侧栏底部主题切换继续复用共享 [SegmentedControl.tsx](/ho
例如:
- [Dashboard.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Dashboard/Dashboard.tsx)
- [Dashboard.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/Dashboard.tsx)
优先目标:
@@ -348,10 +289,10 @@ Admin Next 侧栏底部主题切换继续复用共享 [SegmentedControl.tsx](/ho
例如:
- [DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/DataSources/DataSources.tsx)
- [DataList.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/DataList/DataList.tsx)
- [Users.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Users/Users.tsx)
- [Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Settings/Settings.tsx)
- [DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
- [DataList.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/DataList.tsx)
- [Users.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/Users.tsx)
- [Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
约束:
@@ -361,7 +302,7 @@ Admin Next 侧栏底部主题切换继续复用共享 [SegmentedControl.tsx](/ho
### 数据源目录页
[DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/DataSources/DataSources.tsx) 当前不再承担配置编辑职责,而是数据源目录和采集操作页。
[DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx) 当前不再承担配置编辑职责,而是数据源目录和采集操作页。
当前页面边界:
@@ -385,7 +326,9 @@ Admin Next 侧栏底部主题切换继续复用共享 [SegmentedControl.tsx](/ho
### 采集器设置页
[Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Settings/Settings.tsx) 会按路由进入三种模式:`/settings` 是系统设置,`/earth-content` Earth 内容,`/collection-management` 是采集管理。`collector_credentials` tab 当前在 `/collection-management` 下显示为“采集器”。
[Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx) 会按路由进入三种模式:`/settings` 是系统设置,`/earth-content`智能星球内容,`/collection-management` 是采集管理。`collector_credentials` tab 当前在 `/collection-management` 下显示为“采集器”。
`/settings` 的“系统显示”分区包含 `演示模式` 开关。开启后,智能星球的 OOBE 会忽略“已有当前采集数据”和本地“先浏览”临时跳过状态,直接展示初始化引导;该开关仅用于演示/验收流程,不改变数据源、采集队列或智能星球内容资源配置。
当前页面边界:
@@ -416,9 +359,9 @@ Admin Next 侧栏底部主题切换继续复用共享 [SegmentedControl.tsx](/ho
- [数据源、采集器设置与连接验证](/home/ray/dev/linkong/planet/docs/technical/zh/datasource-collector-settings-connectivity.md)
### Earth 内容页
### 智能星球内容页
`/earth-content` 复用 [Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Settings/Settings.tsx) 的单屏 tab 容器,但页面责任与系统设置分离:
`/earth-content` 复用 [Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx) 的单屏 tab 容器,但页面责任与系统设置分离:
- `电视直播` 迁移原直播源配置,继续管理 Earth 媒体面板内容源。
- `国界精度` 管理 Earth 静态国界资产provider 状态、低精 fallback、高精 manifest/PMTiles、源配置 JSON 和构建动作。
@@ -430,8 +373,8 @@ Admin Next 侧栏底部主题切换继续复用共享 [SegmentedControl.tsx](/ho
例如:
- [BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx)
- [Playground.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Playground/Playground.tsx)
- [BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
- [Playground.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
约束:

View File

@@ -8,7 +8,8 @@
当前推荐参考实现:
- [frontend/src/pages/BGP/BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx)
- [frontend/src/admin/pages/PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
- [frontend/src/admin/pages/DataList.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/DataList.tsx)
- [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css)
## 核心原则
@@ -23,15 +24,15 @@
推荐结构:
```tsx
<AppLayout>
<AdminLayout>
<div className="page-shell">
<div className="page-shell__header">...</div>
<div className="page-shell__body">...</div>
</div>
</AppLayout>
</AdminLayout>
```
页面总高度应被限制在 `AppLayout` 内容区内,而不是继续让整个页面自然向下增长。
页面总高度应被限制在 `AdminLayout` 内容区内,而不是继续让整个页面自然向下增长。
### 2. 滚动优先发生在模块内部

View File

@@ -2,7 +2,7 @@
`backend/app/services/location/` 是所有“给定一条记录,决定它的 lat/lon”业务的共享抽象。算力中心、BGP 观测站、BGP 事件目前都跑在这条管线上。未来需要位置估算的实体例如卫星地面站、用户认领点位、IXP 设施,也应接入这里,而不是各自再写地理解析逻辑。
用户侧流程见 [Earth 位置候选采集使用手册](/home/ray/dev/linkong/planet/docs/technical/zh/location-pipeline-user.md)
用户侧流程见 [智能星球使用手册](/home/ray/dev/linkong/planet/docs/technical/zh/manual.md) 的 Earth 位置候选采集章节
## 设计目标

View File

@@ -1,15 +1,15 @@
# Planet 使用手册
# 智能星球使用手册
这份手册面向 Planet 的最终用户。从打开浏览器开始,覆盖注册账号、登录、配置数据采集器、配置 AI、使用 Earth 和控制台、阅读文档站。所有操作都在浏览器里完成。
这份手册面向智能星球的最终用户。从打开浏览器开始,覆盖注册账号、登录、配置数据采集器、配置 AI、使用智能星球和控制台、阅读文档站。所有操作都在浏览器里完成。
如果你是负责部署或值班的运维,请改读 [Planet 运维手册](/home/ray/dev/linkong/planet/docs/technical/zh/ops-runbook.md),里面是 shell 命令、日志位置、SMTP 兜底创建用户等内容。
如果你是负责部署或值班的运维,请改读 [智能星球运维手册](/home/ray/dev/linkong/planet/docs/technical/zh/ops-runbook.md),里面是 shell 命令、日志位置、SMTP 兜底创建用户等内容。
## 入口总览
| 名称 | 地址 | 是否需要登录 | 说明 |
| --- | --- | --- | --- |
| Earth | `http://<域名>/earth` | 否 | 公开 3D 地球态势页面 |
| Docs | `http://<域名>/docs` | 部分需要 | 公共文档免登录,开发/运维文档按 Gatekeeper 权限组开放 |
| 智能星球 | `http://<域名>/earth` | 否 | 公开 3D 态势页面 |
| 文档 | `http://<域名>/docs` | 部分需要 | 公共文档免登录,开发/运维文档按 Gatekeeper 权限组开放 |
| 注册 / 登录 / 找回密码 | `/register``/login``/forgot-password` | 否 | 自助开通和恢复账号 |
| 控制台 | `http://<域名>/admin` | 是 | 数据、采集器、告警、AI、用户、设置 |
| AI | `http://<域名>/ai` | 是 | 模型供应商、工具和测试台 |
@@ -68,7 +68,7 @@
| 页面 | 路由 | 用途 |
| --- | --- | --- |
| 仪表盘 | `/admin` | 系统概览 |
| Earth | `/earth` | 跳到公开 Earth 页面 |
| 智能星球 | `/earth` | 跳到公开智能星球页面 |
| 数据源 | `/datasources` | 数据源目录、触发采集 |
| 采集数据 | `/data` | 已落库的数据 |
| BGP 观测 | `/bgp` | BGP 专题观测 |
@@ -76,7 +76,7 @@
| BGP 告警 | `/alerts/bgp` | BGP 相关告警 |
| 态势告警 | `/alerts/situational` | 态势研判告警 |
| AI | `/ai` | 模型供应商、工具、测试台 |
| Earth 内容 | `/earth-content` | 电视直播、国界精度、底图和图层资源入口 |
| 智能星球内容 | `/earth-content` | 电视直播、国界精度、底图和图层资源入口 |
| 采集管理 | `/collection-management` | 采集器、采集调度、采集历史入口 |
| 系统日志 | `/logs` | 通常仅 super admin 可见 |
| 用户管理 | `/users` | 创建/删除/改角色/调权限组 |
@@ -186,19 +186,19 @@ Base URL 输入框尾端的插头图标会触发连接测试。测试通过会
电视直播和国界精度已经移到 `/earth-content`,采集器和采集调度已经移到 `/collection-management`AI Provider / WebSearch / OCR 在 `/ai`
### Earth 内容
### 智能星球内容
`/earth-content` 位于控制台“运维与配置”下,面向 Earth 前端体验资源:
`/earth-content` 位于控制台“运维与配置”下,面向智能星球前端体验资源:
- **品牌资源**:维护 Earth HUD 使用的 logo、标题图、标题文本、副标题和描述上传的图片会保存为 Earth 品牌资产并立即供 Earth 页面读取。
- **关于**:维护 Earth 设置面板里的关于卡片,包括 logo、眉标、标题、版本、描述和元信息。
- **电视直播**:维护 Earth 媒体面板里的直播源。
- **品牌资源**:维护智能星球 HUD 使用的 logo、标题图、标题文本、副标题和描述上传的图片会保存为智能星球品牌资产并立即供智能星球页面读取。
- **关于**:维护智能星球设置面板里的关于卡片,包括 logo、眉标、标题、版本、描述和元信息。
- **电视直播**:维护智能星球媒体面板里的直播源。
- **国界精度**:查看当前国界 provider、低精 fallback、高精 PMTiles/manifest 状态,编辑本机源配置并手动构建。
- **地球底图**、**图层资源**、**三维素材**、**新闻锚点策略**:目前是待接入占位页,不展示假数据。
Earth 页面工具栏齿轮中也有“国界精度”。切到“高精”时,如果本机尚未构建高精资产,会像游戏更新包一样启动后台下载/构建并显示百分比;构建成功后自动应用,无需刷新。切回“低精”只切换本机显示偏好,不重新下载。
智能星球页面工具栏齿轮中也有“国界精度”。切到“高精”时,如果本机尚未构建高精资产,会像游戏更新包一样启动后台下载/构建并显示百分比;构建成功后自动应用,无需刷新。切回“低精”只切换本机显示偏好,不重新下载。
如果后端判断 Earth 尚未初始化,首次进入 `/earth` 会出现毛玻璃引导,提示登录控制台并采集数据。这个判断来自后端真实数据状态;如果系统已经有已采集数据,清空浏览器缓存也不会重新弹出。
如果后端判断智能星球尚未初始化,首次进入 `/earth` 会出现毛玻璃引导,提示登录控制台并采集数据。这个判断来自后端真实数据状态;如果系统已经有已采集数据,清空浏览器缓存也不会重新弹出。
### 采集管理
@@ -229,16 +229,16 @@ Earth 页面工具栏齿轮中也有“国界精度”。切到“高精”时
- 查看用户列表(用户名、邮箱、角色、是否激活、邮箱是否已验证)
- 创建用户:与公开注册等价,但跳过邮箱验证(管理员认账)
- 修改角色:`viewer` / `operator` / `admin` / `super_admin`
- 调整 Gatekeeper 权限组:`docs_user` / `docs_developer` / `docs_admin`,影响 Docs 站可见文档范围
- 调整 Gatekeeper 权限组:`docs_user` / `docs_developer` / `docs_admin`,影响文档站可见文档范围
- 禁用 / 启用账号
要让普通用户能看开发或运维文档,进 `/users` 给他加 `docs_developer``docs_admin`
## 数据探索
- `/datasources`:数据源目录。`内置源` 支持按产品域、层级、启用状态、最近执行状态、是否已有采集数据和关键词筛选;勾选多行后可批量触发,`触发全部` 会进入类似浏览器下载列表的采集队列。队列上方显示总进度、运行中、完成、失败和跳过数量,`查看队列` 可展开底部面板,失败项可重试,完成项可跳到详情。`实时源` 面向 AISStream / WebSocket 长连接,展示连接健康、累计入库、时间窗统计和启动 / 停止 / 重连操作。接口、凭证、请求头的编辑统一在 `/collection-management` 的"采集器"。
- `/datasources`:数据源目录。`内置源` 支持按产品域、层级、启用状态、最近执行状态、是否已有采集数据和关键词筛选;未勾选时主按钮显示“触发全部”,勾选多行后会变成“触发已选 N”并只提交所选数据源。右上角队列按钮空态显示队列图标有任务时显示纯圆环总进度点击后打开队列浮层按运行中、完成、失败和跳过分组,失败项可重试,完成项可跳到详情。`实时源` 面向 AISStream / WebSocket 长连接,展示连接健康、累计入库、时间窗统计和启动 / 停止 / 重连操作。接口、凭证、请求头的编辑统一在 `/collection-management` 的"采集器"。
- `/data`:采集后数据表,适合排查"数据是否已经进入系统"、"更新时间是否符合预期"、"某个数据源是否产出有效记录"
- `/bgp`BGP 专题页面,列表 + 详情 + 研判,与 Earth 的 BGP 图层互补
- `/bgp`BGP 专题页面,列表 + 详情 + 研判,与智能星球的 BGP 图层互补
- `/alerts/system``/alerts/bgp``/alerts/situational`系统、BGP、态势告警
## AI 测试台
@@ -251,9 +251,9 @@ Earth 页面工具栏齿轮中也有“国界精度”。切到“高精”时
旧链接 `/playground` 会跳到这里。
## Earth 公开页面
## 智能星球公开页面
Earth `http://localhost:3000/earth` 是公开 3D 态势页面不需要登录。React 路由中的 `/earth` 用 iframe 承载独立前端(位于 `frontend/public/earth/`)。
智能星球 `http://localhost:3000/earth` 是公开 3D 态势页面不需要登录。React 路由中的 `/earth` 用 iframe 承载独立前端(位于 `frontend/public/earth/`)。
### 主要用途
@@ -283,7 +283,17 @@ AIS 船只图例按船型显示颜色:货轮、油轮、客船、渔船、军
算力中心和 BGP 观测站详情卡支持自动采集坐标候选。点击对象后用"自动采集坐标候选"或"重新自动采集坐标"按钮,后端会从源坐标、开放组织注册 API 和在线地理编码中整理候选;常规来源没有候选时使用当前默认 AI Provider 做 LLM factcheck 兜底。BGP 观测站的已存储位置只用于补齐查询上下文,不会作为候选直接返回。
候选可以直接在 Earth 预览。算力中心候选点击"保存"后写入 `compute_center_locations` 维表并刷新图层。算力中心图层左上角的通知气泡显示无法渲染的待定位数量;点击查看列表,单条采集候选,或用"一键采用"从上到下保存最高置信候选。没有可用候选的记录会留在列表中,不会被国家中心点或硬编码 hint 伪造位置。详细流程见 [Earth 位置候选采集使用手册](/home/ray/dev/linkong/planet/docs/technical/zh/location-pipeline-user.md)。
候选可以直接在智能星球预览。算力中心候选点击"保存"后写入 `compute_center_locations` 维表并刷新图层。算力中心图层左上角的通知气泡显示无法渲染的待定位数量;点击查看列表,单条采集候选,或用"一键采用"从上到下保存最高置信候选。没有可用候选的记录会留在列表中,不会被国家中心点或硬编码 hint 伪造位置。
单个对象的推荐流程:
1. 打开算力中心或 BGP 观测站详情卡。
2. 点击"自动采集坐标候选"。
3. 等待候选列表返回;有 WebSearch / AI factcheck 依赖的候选会显示采集中状态。
4. 在智能星球上预览候选位置。
5. 确认可用候选后点击"保存";不确定时关闭卡片不会丢失当前任务状态。
一键定位用于批量处理算力中心待定位队列。它会从列表顶部开始采用最高置信候选;仍没有事实依据的记录会保留在队列中。未开启 WebSearch 时,单个定位和一键定位会置灰,因为位置核验依赖事实查询。
### 设置
@@ -314,12 +324,12 @@ AIS 船只图例按船型显示颜色:货轮、油轮、客船、渔船、军
### 动作捕捉控制
Earth 预留了动作捕捉控制入口。实时链路两种输入源:
智能星球预留了动作捕捉控制入口。实时链路两种输入源:
- **浏览器摄像头**(默认):直接用网页 `getUserMedia` 在本机浏览器识别;无需安装应用,但页面必须运行在 HTTPS 或 localhost且需允许浏览器摄像头权限
- **Motion Agent**:摄像头/RTSP/HTTP → 本地 Agent → 本地 WebSocket → Earth 页面用于双摄、USB index、手机/网络摄像头流
- **Motion Agent**:摄像头/RTSP/HTTP → 本地 Agent → 本地 WebSocket → 智能星球页面用于双摄、USB index、手机/网络摄像头流
打开方式:设置中开启"动捕调试模式",或加 URL 参数 `?motion=1` 打开 Earth 动捕连接。Motion Agent 默认地址 `ws://127.0.0.1:8765/ws/gestures`,可用 `motionAgent` URL 参数覆盖。也可以直接 `?motion=1&motionProvider=browser``?motion=1&motionProvider=agent`
打开方式:设置中开启"动捕调试模式",或加 URL 参数 `?motion=1` 打开智能星球动捕连接。Motion Agent 默认地址 `ws://127.0.0.1:8765/ws/gestures`,可用 `motionAgent` URL 参数覆盖。也可以直接 `?motion=1&motionProvider=browser``?motion=1&motionProvider=agent`
两种模式都不会把摄像头帧或实时手势发到云端,也不会复用新闻/RSS 聚合接口。
@@ -337,7 +347,7 @@ Earth 预留了动作捕捉控制入口。实时链路两种输入源:
### 巡航模式
巡航模式让 Earth 自动轮播聚焦目标。当前巡航模块BGP、新闻、算力中心、船只、海缆、卫星。适合演示、监控大屏或无人值守。
巡航模式让智能星球自动轮播聚焦目标。当前巡航模块BGP、新闻、算力中心、船只、海缆、卫星。适合演示、监控大屏或无人值守。
### 移动端
@@ -345,28 +355,27 @@ Earth 预留了动作捕捉控制入口。实时链路两种输入源:
### 常见问题
- **Earth 打不开**:先确认前端服务是否在线;如果端口不是 `3000`,使用启动输出的实际端口
- **智能星球打不开**:先确认前端服务是否在线;如果端口不是 `3000`,使用启动输出的实际端口
- **图层没有数据**:进 `/datasources` 看数据源状态、是否已采集和最近执行结果,再到 `/data``/bgp` 看是否有记录
- **卫星 / BGP / 海缆加载慢**:这些图层依赖后端接口和外部数据源,首次加载需要等启动任务完成
- **卫星看起来不在同一层**:这是默认的真实高度压缩显示。想回到旧版同层球面,可在设置中关闭“真实卫星高度”
## Docs 文档站
## 文档站
文档站 `http://localhost:3000/docs` 由后端按权限读取,不再把全部 Markdown 直接打进前端构建产物。
未登录访客默认只能看到 `public` 文档:首页、快速开始、使用手册、常见问题、Earth 位置候选采集使用手册。登录用户被分配 Gatekeeper 权限组后可以看到更多技术文档:
未登录访客默认只能看到 `public` 文档:首页、快速开始、使用手册、常见问题。登录用户被分配 Gatekeeper 权限组后可以看到更多技术文档:
- `docs_user`:用户操作类文档
- `docs_developer`Earth、前端、后端、采集器和 AI Provider 等开发文档
- `docs_developer`智能星球、前端、后端、采集器和 AI Provider 等开发文档
- `docs_admin`:服务控制、运维、环境变量和敏感操作文档(包括运维手册)
`admin` 默认拥有 `docs_admin``super_admin` 拥有全部 Docs 权限。Gatekeeper 权限组在"用户管理"中配置。
`admin` 默认拥有 `docs_admin``super_admin` 拥有全部文档权限。Gatekeeper 权限组在"用户管理"中配置。
Docs 支持分类导航、Markdown 渲染、表格和代码块、文档内目录、对当前可见文档搜索、technical 文档间内部链接跳转。
文档站支持分类导航、Markdown 渲染、表格和代码块、文档内目录、对当前可见文档搜索、technical 文档间内部链接跳转。
## 相关文档
- [快速开始](/home/ray/dev/linkong/planet/docs/technical/zh/quickstart.md)
- [常见问题](/home/ray/dev/linkong/planet/docs/technical/zh/faq.md)
- [Earth 位置候选采集使用手册](/home/ray/dev/linkong/planet/docs/technical/zh/location-pipeline-user.md)
- [Planet 运维手册](/home/ray/dev/linkong/planet/docs/technical/zh/ops-runbook.md)
- [智能星球运维手册](/home/ray/dev/linkong/planet/docs/technical/zh/ops-runbook.md)

Some files were not shown because too many files have changed in this diff Show More