Compare commits

...

12 Commits

Author SHA1 Message Date
d15a9d488a release: bump version to 0.66.3
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-05-26 17:26:19 +08:00
rayd1o
eb4c4b7904 release: bump version to 0.66.2
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-05-26 08:45:33 +08:00
rayd1o
887fec972e release: bump version to 0.66.1
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-05-26 04:38:18 +08:00
rayd1o
5bf5c73ca0 release: bump version to 0.66.0
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-05-26 03:41:47 +08:00
e65267fe21 release: bump version to 0.65.2
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
release / images (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-05-22 21:37:38 +08:00
ae982e51cd release: bump version to 0.65.1
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-05-22 20:42:03 +08:00
rayd1o
65e6a96c0d release: bump version to 0.65.0
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
release / images (push) Has been cancelled
ci / delivery (push) Has been cancelled
ci / backend (pull_request) Has been cancelled
ci / frontend (pull_request) Has been cancelled
ci / delivery (pull_request) Has been cancelled
2026-05-21 05:41:49 +08:00
linkong
37e92e7572 release: bump version to 0.64.0
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-05-21 03:46:02 +08:00
linkong
a37d4b6289 fix: update default linkong password
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-05-21 02:20:45 +08:00
linkong
69789d7505 release: bump version to 0.63.1
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-05-21 02:14:11 +08:00
linkong
4f124121e7 fix: include admin next utils module
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-05-21 02:07:39 +08:00
linkong
085bdf9a80 release: bump version to 0.63.0
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-05-21 01:46:43 +08:00
197 changed files with 11574 additions and 13409 deletions

2
.gitignore vendored
View File

@@ -29,6 +29,8 @@ downloads/
eggs/
.eggs/
lib/
!frontend/src/admin/lib/
!frontend/src/admin/lib/**
lib64/
parts/
sdist/

View File

@@ -189,6 +189,9 @@
# 查看服务状态
./planet.sh health
# 删除容器、卷、镜像和本地编译状态,执行前需要输入 Y 确认
./planet.sh destroy
```
前端命令约定:

View File

@@ -7,9 +7,6 @@ This file is the active backlog only. Completed history belongs in `docs/CHANGEL
- [ ] Earth AI command entry: merge natural-language and speech-triggered LLM commands into the existing Earth search panel as described in [Agent Runtime, Earth LLM Command, And Speech Entry Plan](/home/ray/dev/linkong/planet/docs/plans/agents-earth-command-runtime-plan.md).
- [ ] Earth action executor: implement safe visualization actions for layer toggles, batch highlights, filters, focus, result panels, and clear-highlight behavior.
- [ ] Earth entity matching: support stable entity ids and batch matching for Beidou satellites, mainland China compute centers, BGP, news, vessels, and cables.
- [x] High-precision country boundary tile framework: implement the static vector tile builder, versioned seed output, frontend bbox tile loader, debounce, in-flight dedupe, and LRU cache described in [Earth High Precision Boundary Tiles Plan](/home/ray/dev/linkong/planet/docs/plans/earth-high-precision-boundary-tiles-plan.md).
- [x] Add the `pmtiles-mvt` frontend tile provider contract, MVT decoder dependencies, static PMTiles Nginx handling, collector artifact registration, production readiness check, and user operation docs for Earth boundaries.
- [x] Split Earth boundary ingestion into standard source collectors (`earth_admin0_boundaries`, `earth_coastline`, `earth_claim_lines`) plus the downstream `earth_boundary_tiles` PMTiles builder.
- [ ] Replace debug GeoJSON boundary tiles with the real `earth-boundaries-china-pov-v1.pmtiles` production artifact after audited admin-0 / coastline / claim-line sources and the PMTiles toolchain are available.
- [ ] Import authoritative China POV / coastline / claim-line source packages through the three standard Earth boundary source collectors, then rebuild a versioned PMTiles artifact so highest zoom `8-10` preserves trusted source geometry instead of seed data.
- [ ] Earth boundary data: acquire or generate auditable China POV geometry for Zangnan, Aksai Chin, Taiwan/Penghu, Diaoyu Dao and affiliated islands, Chiwei Yu, South China Sea islands, Kosovo, Gaza, and the official dashed maritime claim line before implementing final visual changes.
@@ -61,6 +58,9 @@ Archived items stay here so old context is not lost. Completed items remain chec
### Completed
- [x] Implemented the high-precision country boundary tile framework from [Earth High Precision Boundary Tiles Plan](/home/ray/dev/linkong/planet/docs/plans/earth-high-precision-boundary-tiles-plan.md): static vector tile builder, versioned seed output, frontend bbox tile loader, debounce, in-flight dedupe, and LRU cache.
- [x] Added the `pmtiles-mvt` frontend tile provider contract, MVT decoder dependencies, static PMTiles Nginx handling, collector artifact registration, production readiness check, and user operation docs for Earth boundaries.
- [x] Split Earth boundary ingestion into standard source collectors (`earth_admin0_boundaries`, `earth_coastline`, `earth_claim_lines`) plus the downstream `earth_boundary_tiles` PMTiles builder.
- [x] Refined BGP observer and anomaly `hover/click` feel.
- [x] Added BGP anomaly relationship display with cables / regions.
- [x] Added the Earth BGP activity layer so the map still feels alive when incident density is low.

View File

@@ -1 +1 @@
0.62.0
0.66.3

View File

@@ -2,10 +2,14 @@
ARG PYTHON_IMAGE=python:3.14-slim
ARG UV_IMAGE=ghcr.io/astral-sh/uv:latest
ARG AI_PROVIDER_BUILD_FINGERPRINT=unknown
FROM ${UV_IMAGE} AS uv
FROM ${PYTHON_IMAGE}
ARG AI_PROVIDER_BUILD_FINGERPRINT
LABEL planet.aiprovider.build-fingerprint="${AI_PROVIDER_BUILD_FINGERPRINT}"
COPY --from=uv /uv /uvx /bin/
WORKDIR /app
@@ -15,12 +19,15 @@ ENV PYTHONUNBUFFERED=1
ENV UV_COMPILE_BYTECODE=1
ENV UV_LINK_MODE=copy
RUN mkdir -p /root/.config/uv
RUN apt-get update && apt-get install -y --no-install-recommends \
curl \
&& rm -rf /var/lib/apt/lists/*
COPY pyproject.toml uv.lock /app/
RUN --mount=type=cache,target=/root/.cache/uv \
--mount=type=secret,id=planet_uv_config,target=/root/.config/uv/uv.toml,required=false \
uv sync --frozen --no-dev
COPY aiprovider /app/aiprovider

View File

@@ -1,3 +1,5 @@
# syntax=docker/dockerfile:1.7
ARG PYTHON_IMAGE=python:3.14-slim
ARG UV_IMAGE=ghcr.io/astral-sh/uv:latest
@@ -14,12 +16,16 @@ ENV UV_COMPILE_BYTECODE=1
ENV UV_LINK_MODE=copy
ENV PYTHONPATH=/app/backend
RUN mkdir -p /root/.config/uv
RUN apt-get update && apt-get install -y --no-install-recommends \
curl \
&& rm -rf /var/lib/apt/lists/*
COPY pyproject.toml uv.lock /app/
RUN uv sync --frozen --no-dev
RUN --mount=type=cache,target=/root/.cache/uv \
--mount=type=secret,id=planet_uv_config,target=/root/.config/uv/uv.toml,required=false \
uv sync --frozen --no-dev
COPY backend /app/backend
COPY VERSION /app/VERSION

View File

@@ -19,6 +19,7 @@ from app.api.v1 import (
vessels,
bgp,
news,
interactables,
realtime_sources,
system_control,
tv,
@@ -53,4 +54,5 @@ api_router.include_router(vessels.router, prefix="/vessels", tags=["vessels"])
api_router.include_router(bgp.router, prefix="/bgp", tags=["bgp"])
api_router.include_router(tv.router, prefix="/tv", tags=["tv"])
api_router.include_router(news.router, prefix="/news", tags=["news"])
api_router.include_router(interactables.router, prefix="/interactables", tags=["interactables"])
api_router.include_router(realtime_sources.router, prefix="/realtime-sources", tags=["realtime-sources"])

View File

@@ -3,6 +3,7 @@ from uuid import uuid4
from fastapi import APIRouter, Depends, HTTPException, Request, Response
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.logging import get_logger
from app.core.security import get_current_user
from app.db.session import get_db
from app.models.user import User
@@ -47,8 +48,10 @@ from app.services.playground_chat_service import (
stop_message,
)
from app.services.situational_alert_ai_brief import build_situational_alert_brief_request
from app.services.business_logs import emit_business_log, exception_context
router = APIRouter()
logger = get_logger(__name__, service="api")
@router.get("/provider/status", response_model=AIProviderStatusResponse)
@@ -122,6 +125,16 @@ async def create_playground_message(
provider_client: AIProviderClient = Depends(get_ai_provider_client),
db: AsyncSession = Depends(get_db),
):
await emit_business_log(
logger,
event="ai.playground.message.create",
message="Playground message creation requested",
category="ai",
service="api",
module=__name__,
user_id=current_user.id,
context={"session_key": payload.session_key, "preset": payload.selected_preset_key},
)
return await create_turn(
db,
user_id=current_user.id,
@@ -136,6 +149,16 @@ async def stop_playground_message(
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
await emit_business_log(
logger,
event="ai.playground.message.stop",
message="Playground message stop requested",
category="ai",
service="api",
module=__name__,
user_id=current_user.id,
context={"session_key": payload.session_key, "message_id": payload.message_id},
)
return await stop_message(
db,
user_id=current_user.id,
@@ -150,6 +173,16 @@ async def resend_playground_message(
provider_client: AIProviderClient = Depends(get_ai_provider_client),
db: AsyncSession = Depends(get_db),
):
await emit_business_log(
logger,
event="ai.playground.message.resend",
message="Playground message resend requested",
category="ai",
service="api",
module=__name__,
user_id=current_user.id,
context={"session_key": payload.session_key, "user_message_id": payload.user_message_id},
)
return await resend_turn(
db,
user_id=current_user.id,
@@ -214,16 +247,70 @@ async def analyze_bgp_brief(
anomaly_limit=payload.anomaly_limit,
collector_limit=payload.collector_limit,
)
await emit_business_log(
logger,
event="ai.brief.bgp.facts_collected",
message="BGP brief facts collected",
category="ai",
service="api",
module=__name__,
request_id=request_id,
user_id=current_user.id,
context={
"incident_limit": payload.incident_limit,
"anomaly_limit": payload.anomaly_limit,
"collector_limit": payload.collector_limit,
"fact_count": len(facts or []),
},
)
brief_request.preferred_model = payload.preferred_model
brief_request.thinking = payload.thinking
analysis = await provider_client.analyze(brief_request, request_id=request_id)
return save_bgp_brief_record(
analysis,
await emit_business_log(
logger,
event="ai.brief.bgp.start",
message="BGP brief AI analysis started",
category="ai",
service="api",
module=__name__,
request_id=request_id,
facts=facts,
context=context,
user_id=current_user.id,
context={"preferred_model": payload.preferred_model},
)
try:
analysis = await provider_client.analyze(brief_request, request_id=request_id)
record = save_bgp_brief_record(
analysis,
request_id=request_id,
facts=facts,
context=context,
)
await emit_business_log(
logger,
event="ai.brief.bgp.completed",
message="BGP brief AI analysis saved",
category="ai",
service="api",
module=__name__,
request_id=request_id,
user_id=current_user.id,
context={"provider": analysis.provider, "model": analysis.model, "brief_id": record.id},
)
return record
except Exception as exc:
await emit_business_log(
logger,
event="ai.brief.bgp.failed",
message="BGP brief AI analysis failed",
category="ai",
level="error",
service="api",
module=__name__,
request_id=request_id,
user_id=current_user.id,
context=exception_context(exc, {"preferred_model": payload.preferred_model}),
)
raise
@router.post("/alerts/brief", response_model=AlertBriefResponse)
@@ -242,17 +329,65 @@ async def analyze_alert_brief(
db,
alert_limit=payload.alert_limit,
)
await emit_business_log(
logger,
event="ai.brief.alerts.facts_collected",
message="Alert brief facts collected",
category="ai",
service="api",
module=__name__,
request_id=request_id,
user_id=current_user.id,
context={"alert_limit": payload.alert_limit, "fact_count": len(facts or [])},
)
brief_request.preferred_model = payload.preferred_model
brief_request.thinking = payload.thinking
analysis = await provider_client.analyze(brief_request, request_id=request_id)
return AlertBriefResponse(
**analysis.model_dump(),
title=brief_request.title,
objective=brief_request.objective,
facts=facts,
context=context,
await emit_business_log(
logger,
event="ai.brief.alerts.start",
message="Alert brief AI analysis started",
category="ai",
service="api",
module=__name__,
request_id=request_id,
user_id=current_user.id,
context={"preferred_model": payload.preferred_model},
)
try:
analysis = await provider_client.analyze(brief_request, request_id=request_id)
await emit_business_log(
logger,
event="ai.brief.alerts.completed",
message="Alert brief AI analysis completed",
category="ai",
service="api",
module=__name__,
request_id=request_id,
user_id=current_user.id,
context={"provider": analysis.provider, "model": analysis.model},
)
return AlertBriefResponse(
**analysis.model_dump(),
title=brief_request.title,
objective=brief_request.objective,
facts=facts,
context=context,
)
except Exception as exc:
await emit_business_log(
logger,
event="ai.brief.alerts.failed",
message="Alert brief AI analysis failed",
category="ai",
level="error",
service="api",
module=__name__,
request_id=request_id,
user_id=current_user.id,
context=exception_context(exc, {"preferred_model": payload.preferred_model}),
)
raise
@router.post("/situational-alerts/brief", response_model=SituationalAlertBriefResponse)
@@ -268,14 +403,62 @@ async def analyze_situational_alert_brief(
response.headers["X-Request-ID"] = request_id
brief_request, facts, context = await build_situational_alert_brief_request(db)
await emit_business_log(
logger,
event="ai.brief.situational_alerts.facts_collected",
message="Situational alert brief facts collected",
category="ai",
service="api",
module=__name__,
request_id=request_id,
user_id=current_user.id,
context={"fact_count": len(facts or [])},
)
brief_request.preferred_model = payload.preferred_model
brief_request.thinking = payload.thinking
analysis = await provider_client.analyze(brief_request, request_id=request_id)
return SituationalAlertBriefResponse(
**analysis.model_dump(),
title=brief_request.title,
objective=brief_request.objective,
facts=facts,
context=context,
await emit_business_log(
logger,
event="ai.brief.situational_alerts.start",
message="Situational alert brief AI analysis started",
category="ai",
service="api",
module=__name__,
request_id=request_id,
user_id=current_user.id,
context={"preferred_model": payload.preferred_model},
)
try:
analysis = await provider_client.analyze(brief_request, request_id=request_id)
await emit_business_log(
logger,
event="ai.brief.situational_alerts.completed",
message="Situational alert brief AI analysis completed",
category="ai",
service="api",
module=__name__,
request_id=request_id,
user_id=current_user.id,
context={"provider": analysis.provider, "model": analysis.model},
)
return SituationalAlertBriefResponse(
**analysis.model_dump(),
title=brief_request.title,
objective=brief_request.objective,
facts=facts,
context=context,
)
except Exception as exc:
await emit_business_log(
logger,
event="ai.brief.situational_alerts.failed",
message="Situational alert brief AI analysis failed",
category="ai",
level="error",
service="api",
module=__name__,
request_id=request_id,
user_id=current_user.id,
context=exception_context(exc, {"preferred_model": payload.preferred_model}),
)
raise

View File

@@ -1,4 +1,3 @@
import asyncio
from datetime import datetime, timedelta, timezone
from typing import Optional
@@ -7,7 +6,7 @@ from pydantic import BaseModel, Field
from sqlalchemy import func, or_, select, text
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.cache import cache
from app.core.logging import get_logger
from app.core.time import to_iso8601_utc
from app.core.security import get_current_user
from app.core.data_sources import get_data_sources_config
@@ -22,16 +21,26 @@ from app.models.user import User
from app.models.vessel import AISRawObservation
from app.services.vessel_ais_aggregation import VESSEL_AIS_SCHEMA
from app.services.scheduler import (
cancel_running_collector_now,
get_latest_task_id_for_datasource,
run_collector_now,
sync_datasource_job,
)
from app.services.earth_layer_cache import invalidate_earth_layer_cache_for_source
from app.services.data_jobs import (
JOB_STATUS_CANCELLING,
JOB_STATUS_QUEUED,
JOB_STATUS_RUNNING,
JOB_TYPE_CLEAR_CACHE,
JOB_TYPE_CLEAR_DATA,
JOB_TYPE_COLLECT,
enqueue_datasource_job,
get_active_datasource_job,
request_cancel_datasource_task,
)
from app.services.business_logs import emit_business_log
router = APIRouter()
logger = get_logger(__name__, service="api")
STALE_RUNNING_TASK_TIMEOUT_MINUTES = 90
PRODUCT_SOURCE_KEYWORDS: tuple[tuple[str, tuple[str, ...]], ...] = (
("vessels", ("vessel", "ais")),
("cables", ("cable", "landing", "telegeography", "arcgis", "fao")),
@@ -115,7 +124,7 @@ async def _load_latest_running_tasks(
_task_rank_column(CollectionTask.started_at),
)
.where(CollectionTask.datasource_id.in_(datasource_ids))
.where(CollectionTask.status == "running")
.where(CollectionTask.status.in_((JOB_STATUS_QUEUED, JOB_STATUS_RUNNING, JOB_STATUS_CANCELLING)))
.subquery()
)
result = await db.execute(
@@ -126,32 +135,6 @@ async def _load_latest_running_tasks(
return {task.datasource_id: task for task in result.scalars().all()}
async def _load_latest_task_ids(
db: AsyncSession,
datasource_ids: list[int],
) -> dict[int, int]:
if not datasource_ids:
return {}
ranked_tasks = (
select(
CollectionTask.id.label("task_id"),
CollectionTask.datasource_id.label("datasource_id"),
func.row_number().over(
partition_by=CollectionTask.datasource_id,
order_by=CollectionTask.id.desc(),
).label("row_num"),
)
.where(CollectionTask.datasource_id.in_(datasource_ids))
.subquery()
)
result = await db.execute(
select(ranked_tasks.c.datasource_id, ranked_tasks.c.task_id)
.where(ranked_tasks.c.row_num == 1)
)
return {datasource_id: task_id for datasource_id, task_id in result.all()}
async def _load_latest_tasks(
db: AsyncSession,
datasource_ids: list[int],
@@ -303,8 +286,11 @@ def serialize_datasource_row(
"last_run": to_iso8601_utc(last_run_at),
"last_run_at": to_iso8601_utc(last_run_at),
"last_status": last_status,
"is_running": running_task is not None,
"is_running": running_task is not None and running_task.task_type == JOB_TYPE_COLLECT,
"is_task_active": running_task is not None,
"task_status": running_task.status if running_task else None,
"task_id": display_task.id if display_task else None,
"task_type": display_task.task_type if display_task else None,
"progress": display_task.progress if display_task else None,
"phase": display_task.phase if display_task else None,
"phase_progress": display_task.phase_progress if display_task else None,
@@ -400,9 +386,25 @@ async def _trigger_datasource_batch(
datasources: list[DataSource],
*,
force: bool,
actor_id: int | None = None,
trigger_kind: str = "batch",
) -> dict:
await emit_business_log(
logger,
event=f"collector.trigger.{trigger_kind}.start",
message="Datasource batch trigger started",
category="collector",
service="api",
module=__name__,
user_id=actor_id,
context={
"trigger_kind": trigger_kind,
"force": force,
"requested_count": len(datasources),
},
)
if not datasources:
return {
result = {
"status": "noop",
"message": "No matching data sources to trigger",
"force": force,
@@ -410,8 +412,18 @@ async def _trigger_datasource_batch(
"skipped": [],
"failed": [],
}
await emit_business_log(
logger,
event=f"collector.trigger.{trigger_kind}.completed",
message="Datasource batch trigger completed with no matching sources",
category="collector",
service="api",
module=__name__,
user_id=actor_id,
context={"trigger_kind": trigger_kind, "force": force, "status": "noop", "triggered_count": 0},
)
return result
previous_task_ids: dict[int, Optional[int]] = {}
triggered_sources: list[dict] = []
skipped_sources: list[dict] = []
failed_sources: list[dict] = []
@@ -446,9 +458,11 @@ async def _trigger_datasource_batch(
}
)
continue
cancelled = await cancel_running_collector_now(datasource.source)
if not cancelled:
await rollback_orphaned_running_task(db, datasource, running_task)
await request_cancel_datasource_task(
db,
running_task,
reason="superseded_by_forced_collection",
)
if not force and not is_due_for_collection(datasource, now):
skipped_sources.append(
@@ -465,57 +479,51 @@ async def _trigger_datasource_batch(
)
continue
previous_task_ids[datasource.id] = None
success = run_collector_now(datasource.source)
if not success:
failed_sources.append(
{
"id": datasource.id,
"source": datasource.source,
"name": datasource.name,
"reason": "trigger_failed",
}
)
continue
task = await enqueue_datasource_job(
db,
datasource,
JOB_TYPE_COLLECT,
payload={"force": force, "trigger": "batch"},
)
triggered_sources.append(
{
"id": datasource.id,
"source": datasource.source,
"name": datasource.name,
"task_id": None,
"task_id": task.id,
}
)
latest_task_ids = await _load_latest_task_ids(
db,
[datasource.id for datasource in datasources],
)
for datasource_id in previous_task_ids:
previous_task_ids[datasource_id] = latest_task_ids.get(datasource_id)
for _ in range(20):
await asyncio.sleep(0.1)
pending = [item for item in triggered_sources if item["task_id"] is None]
if not pending:
break
latest_task_ids = await _load_latest_task_ids(
db,
[item["id"] for item in pending],
)
for item in pending:
task_id = latest_task_ids.get(item["id"])
if task_id is not None and task_id != previous_task_ids.get(item["id"]):
item["task_id"] = task_id
return {
"status": "triggered" if triggered_sources else "partial",
"message": f"Triggered {len(triggered_sources)} data sources",
result = {
"status": "queued" if triggered_sources else "partial",
"message": f"Queued {len(triggered_sources)} data source jobs",
"force": force,
"triggered": triggered_sources,
"skipped": skipped_sources,
"failed": failed_sources,
}
await emit_business_log(
logger,
event=f"collector.trigger.{trigger_kind}.completed",
message="Datasource batch trigger completed",
category="collector",
service="api",
module=__name__,
user_id=actor_id,
context={
"trigger_kind": trigger_kind,
"force": force,
"status": result["status"],
"requested_count": len(datasources),
"triggered_count": len(triggered_sources),
"skipped_count": len(skipped_sources),
"failed_count": len(failed_sources),
"triggered_sources": [item["source"] for item in triggered_sources],
"skipped_reasons": [item["reason"] for item in skipped_sources],
},
)
return result
async def get_datasource_record(db: AsyncSession, source_id: str) -> Optional[DataSource]:
@@ -780,7 +788,13 @@ async def trigger_all_datasources(
.order_by(DataSource.module, DataSource.id)
)
datasources = result.scalars().all()
return await _trigger_datasource_batch(db, datasources, force=force)
return await _trigger_datasource_batch(
db,
datasources,
force=force,
actor_id=current_user.id,
trigger_kind="all",
)
@router.post("/trigger-batch")
@@ -816,7 +830,13 @@ async def trigger_datasource_batch(
collected=None if payload.source_ids else payload.collected,
credential_status=None if payload.source_ids else payload.credential_status,
)
return await _trigger_datasource_batch(db, datasources, force=payload.force)
return await _trigger_datasource_batch(
db,
datasources,
force=payload.force,
actor_id=current_user.id,
trigger_kind="batch",
)
@router.get("/snapshots")
@@ -992,8 +1012,24 @@ async def trigger_datasource(
if not datasource.is_active:
raise HTTPException(status_code=400, detail="Data source is disabled")
running_task = await get_running_task(db, datasource.id)
running_task = await get_active_datasource_job(db, datasource.id, task_types=(JOB_TYPE_COLLECT,))
if running_task is not None and not force:
await emit_business_log(
logger,
event="collector.trigger.single.skipped_already_running",
message="Datasource trigger skipped because a task is already running",
category="collector",
level="warning",
service="api",
module=__name__,
user_id=current_user.id,
context={
"collector_name": datasource.source,
"datasource_id": datasource.id,
"task_id": running_task.id,
"status": "skipped",
},
)
raise HTTPException(
status_code=409,
detail={
@@ -1013,31 +1049,42 @@ async def trigger_datasource(
)
if running_task is not None and force:
cancelled = await cancel_running_collector_now(datasource.source)
if not cancelled:
await rollback_orphaned_running_task(db, datasource, running_task)
await request_cancel_datasource_task(
db,
running_task,
reason="superseded_by_forced_collection",
)
previous_task_id = await get_latest_task_id_for_datasource(datasource.id)
success = run_collector_now(datasource.source)
if not success:
raise HTTPException(status_code=500, detail=f"Failed to trigger collector '{datasource.source}'")
task_id = None
for _ in range(20):
await asyncio.sleep(0.1)
task_id = await get_latest_task_id_for_datasource(datasource.id)
if task_id is not None and task_id != previous_task_id:
break
if task_id == previous_task_id:
task_id = None
task = await enqueue_datasource_job(
db,
datasource,
JOB_TYPE_COLLECT,
payload={"force": force, "trigger": "single"},
)
await emit_business_log(
logger,
event="collector.trigger.single.completed",
message="Datasource trigger queued",
category="collector",
service="api",
module=__name__,
user_id=current_user.id,
context={
"collector_name": datasource.source,
"datasource_id": datasource.id,
"task_id": task.id,
"force": force,
"status": "queued",
},
)
return {
"status": "triggered",
"status": "queued",
"source_id": datasource.id,
"task_id": task_id,
"task_id": task.id,
"collector_name": datasource.source,
"force": force,
"message": f"Collector '{datasource.source}' has been triggered",
"message": f"Collector '{datasource.source}' has been queued",
}
@@ -1051,22 +1098,30 @@ async def clear_datasource_data(
if not datasource:
raise HTTPException(status_code=404, detail="Data source not found")
result = await db.execute(
select(func.count(CollectedData.id)).where(CollectedData.source == datasource.source)
active_task = await get_active_datasource_job(db, datasource.id)
if active_task is not None:
raise HTTPException(
status_code=409,
detail={
"reason": "datasource_job_in_progress",
"message": "当前数据源已有任务在执行,请等待完成或先取消任务。",
"task_id": active_task.id,
"task_type": active_task.task_type,
"status": active_task.status,
},
)
task = await enqueue_datasource_job(
db,
datasource,
JOB_TYPE_CLEAR_DATA,
payload={"source": datasource.source},
)
count = result.scalar() or 0
if count == 0:
return {"status": "success", "message": "No data to clear", "deleted_count": 0}
delete_query = CollectedData.__table__.delete().where(CollectedData.source == datasource.source)
await db.execute(delete_query)
await db.commit()
return {
"status": "success",
"message": f"Cleared {count} records for data source '{datasource.name}'",
"deleted_count": count,
"status": "queued",
"message": f"Queued data clearing for data source '{datasource.name}'",
"task_id": task.id,
"deleted_count": None,
}
@@ -1080,16 +1135,44 @@ async def clear_datasource_cache(
if not datasource:
raise HTTPException(status_code=404, detail="Data source not found")
earth_deleted_count = invalidate_earth_layer_cache_for_source(datasource.source)
dashboard_deleted_count = int(cache.delete("dashboard:stats")) + int(cache.delete("dashboard:summary"))
deleted_count = earth_deleted_count + dashboard_deleted_count
task = await enqueue_datasource_job(
db,
datasource,
JOB_TYPE_CLEAR_CACHE,
payload={"source": datasource.source},
dedupe_key=f"clear_cache:{datasource.source}",
)
return {
"status": "success",
"message": f"Cleared {deleted_count} cache keys for data source '{datasource.name}'",
"deleted_count": deleted_count,
"earth_layer_deleted_count": earth_deleted_count,
"dashboard_deleted_count": dashboard_deleted_count,
"status": "queued",
"message": f"Queued cache clearing for data source '{datasource.name}'",
"task_id": task.id,
"deleted_count": None,
}
@router.post("/{source_id}/tasks/{task_id}/cancel")
async def cancel_datasource_task(
source_id: str,
task_id: int,
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
datasource = await get_datasource_record(db, source_id)
if not datasource:
raise HTTPException(status_code=404, detail="Data source not found")
task = await db.get(CollectionTask, task_id)
if not task or task.datasource_id != datasource.id:
raise HTTPException(status_code=404, detail="Task not found")
task = await request_cancel_datasource_task(db, task)
return {
"status": "cancelled" if task.completed_at else "cancelling",
"task_id": task.id,
"task_type": task.task_type,
"phase": task.phase,
"requested_cancel_at": to_iso8601_utc(task.requested_cancel_at),
}
@@ -1109,7 +1192,7 @@ async def get_task_status(
if not task or task.datasource_id != datasource.id:
raise HTTPException(status_code=404, detail="Task not found")
else:
task = await get_running_task(db, datasource.id)
task = await get_active_datasource_job(db, datasource.id)
if task is None:
result = await db.execute(
select(CollectionTask)
@@ -1134,8 +1217,12 @@ async def get_task_status(
}
return {
"is_running": task.status == "running",
"is_running": task.status in {JOB_STATUS_QUEUED, JOB_STATUS_RUNNING, JOB_STATUS_CANCELLING}
and task.task_type == JOB_TYPE_COLLECT,
"is_task_active": task.status in {JOB_STATUS_QUEUED, JOB_STATUS_RUNNING, JOB_STATUS_CANCELLING},
"task_id": task.id,
"task_type": task.task_type,
"task_status": task.status,
"progress": task.progress,
"phase": task.phase,
"phase_progress": task.phase_progress,
@@ -1146,5 +1233,6 @@ async def get_task_status(
"records_processed": task.records_processed,
"total_records": task.total_records,
"status": task.status,
"requested_cancel_at": to_iso8601_utc(task.requested_cancel_at),
"error_message": task.error_message,
}

View File

@@ -9,13 +9,18 @@ from uuid import uuid4
from fastapi import APIRouter, Depends, File, HTTPException, Request, UploadFile, status
from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
from pydantic import BaseModel, Field
from sqlalchemy import delete, select, text
from sqlalchemy import delete, func, select, text
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.config import settings as app_settings
from app.core.security import decode_token, get_current_user, redis_client
from app.db.session import get_db
from app.models.collected_data import CollectedData
from app.models.datasource import DataSource
from app.models.datasource_config import DataSourceConfig
from app.models.system_setting import SystemSetting
from app.models.user import User
from app.services.tv_streams import get_tv_settings_payload
from app.services.earth_boundaries import (
EarthBoundaryBuildError,
get_boundary_build_status,
@@ -31,9 +36,17 @@ REPO_ROOT = Path(__file__).resolve().parents[4]
EARTH_BRAND_ASSET_DIR = REPO_ROOT / "data" / "earth-brand"
EARTH_BRAND_ASSET_URL_PREFIX = "/earth-brand-assets"
EARTH_BRAND_CATEGORY = "earth_brand"
EARTH_ABOUT_CATEGORY = "earth_about"
SYSTEM_SETTINGS_CATEGORY = "system"
MAX_EARTH_BRAND_ASSET_BYTES = 3 * 1024 * 1024
ALLOWED_EARTH_BRAND_EXTENSIONS = {".png", ".jpg", ".jpeg", ".webp", ".svg"}
def _app_version_label() -> str:
version = str(app_settings.VERSION or "").strip() or "0.0.0"
return version if version.startswith("v") else f"v{version}"
DEFAULT_EARTH_BRAND = {
"logo_src": "/earth/assets/brand/earth-logo.png",
"title_src": "/earth/assets/brand/title-zh.png",
@@ -44,6 +57,20 @@ DEFAULT_EARTH_BRAND = {
"title_alt": "智能星球计划",
}
DEFAULT_EARTH_ABOUT = {
"logo_src": "/earth/assets/brand/lim-logo.png",
"kicker": "About",
"title": "智能星球计划",
"version": _app_version_label(),
"description": "面向临空场景下的智能媒体研究、全球态势感知与多源开放数据巡航,提供可视化观测、事件聚合与交互式探索能力。",
"meta": [
{"label": "出品方", "value": "浙江大学临空智能媒体研究院"},
{"label": "策划人", "value": "方兴东、黄柳青"},
{"label": "产品兼开发者", "value": "钱坤、张鸽、齐鹏"},
],
}
EARTH_ABOUT_LEGACY_PLANNER_VALUE = "黄柳青"
class EarthBoundaryConfigPayload(BaseModel):
config: dict[str, Any] = Field(default_factory=dict)
@@ -59,6 +86,20 @@ class EarthBrandPayload(BaseModel):
title_alt: str = Field(default=DEFAULT_EARTH_BRAND["title_alt"], max_length=200)
class EarthAboutMetaItem(BaseModel):
label: str = Field(default="", max_length=80)
value: str = Field(default="", max_length=240)
class EarthAboutPayload(BaseModel):
logo_src: str = Field(default=DEFAULT_EARTH_ABOUT["logo_src"], max_length=1000)
kicker: str = Field(default=DEFAULT_EARTH_ABOUT["kicker"], max_length=80)
title: str = Field(default=DEFAULT_EARTH_ABOUT["title"], max_length=160)
version: str = Field(default=DEFAULT_EARTH_ABOUT["version"], max_length=80)
description: str = Field(default=DEFAULT_EARTH_ABOUT["description"], max_length=800)
meta: list[EarthAboutMetaItem] = Field(default_factory=list)
def _normalize_earth_brand_payload(payload: dict[str, Any] | None) -> dict[str, str]:
merged = DEFAULT_EARTH_BRAND.copy()
if payload:
@@ -76,6 +117,47 @@ def _normalize_earth_brand_payload(payload: dict[str, Any] | None) -> dict[str,
return merged
def _normalize_earth_about_payload(payload: dict[str, Any] | None) -> dict[str, Any]:
merged: dict[str, Any] = {
key: value
for key, value in DEFAULT_EARTH_ABOUT.items()
if key != "meta"
}
raw_meta = DEFAULT_EARTH_ABOUT["meta"]
if payload:
for key in ("logo_src", "kicker", "title", "description"):
value = payload.get(key)
if value is not None:
merged[key] = str(value).strip()
raw_meta = payload.get("meta") if isinstance(payload.get("meta"), list) else raw_meta
merged["version"] = _app_version_label()
for key, default_value in DEFAULT_EARTH_ABOUT.items():
if key == "meta":
continue
if not merged.get(key):
merged[key] = default_value
normalized_meta: list[dict[str, str]] = []
for item in raw_meta:
if not isinstance(item, dict):
continue
label = str(item.get("label") or "").strip()
value = str(item.get("value") or "").strip()
if label == "策划人" and value == EARTH_ABOUT_LEGACY_PLANNER_VALUE:
value = "方兴东、黄柳青"
if label or value:
normalized_meta.append({"label": label, "value": value})
if not normalized_meta:
normalized_meta = [dict(item) for item in DEFAULT_EARTH_ABOUT["meta"]]
merged["meta"] = normalized_meta
return merged
def _is_demo_mode_enabled(payload: Any) -> bool:
return bool(payload.get("demo_mode")) if isinstance(payload, dict) else False
async def _get_earth_brand_record(db: AsyncSession) -> SystemSetting | None:
result = await db.execute(
select(SystemSetting).where(SystemSetting.category == EARTH_BRAND_CATEGORY)
@@ -91,6 +173,56 @@ async def _get_earth_brand_payload(db: AsyncSession) -> dict[str, Any]:
}
async def _get_earth_about_record(db: AsyncSession) -> SystemSetting | None:
result = await db.execute(
select(SystemSetting).where(SystemSetting.category == EARTH_ABOUT_CATEGORY)
)
return result.scalar_one_or_none()
async def _get_earth_about_payload(db: AsyncSession) -> dict[str, Any]:
record = await _get_earth_about_record(db)
return {
"about": _normalize_earth_about_payload(record.payload if record else None),
"is_default": record is None,
}
async def _get_optional_current_user(
credentials: HTTPAuthorizationCredentials | None = Depends(optional_bearer),
db: AsyncSession = Depends(get_db),
) -> User | None:
if credentials is None:
return None
token = credentials.credentials
if redis_client.sismember("blacklisted_tokens", token):
return None
payload = decode_token(token)
if payload is None or payload.get("type") != "access":
return None
user_id = payload.get("sub")
if user_id is None:
return None
result = await db.execute(
text(
"SELECT id, username, email, password_hash, role, is_active, gatekeeper_groups FROM users WHERE id = :id"
),
{"id": int(user_id)},
)
row = result.fetchone()
if row is None or not row[5]:
return None
user = User()
user.id = row[0]
user.username = row[1]
user.email = row[2]
user.password_hash = row[3]
user.role = row[4]
user.is_active = row[5]
user.gatekeeper_groups = row[6] or []
return user
@router.get("/brand")
async def get_earth_brand(db: AsyncSession = Depends(get_db)):
return await _get_earth_brand_payload(db)
@@ -159,46 +291,108 @@ async def upload_earth_brand_asset(
return {"url": asset_url, "filename": safe_name, "content_type": file.content_type}
@router.get("/about")
async def get_earth_about(db: AsyncSession = Depends(get_db)):
return await _get_earth_about_payload(db)
@router.put("/about")
async def update_earth_about(
payload: EarthAboutPayload,
_current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
normalized = _normalize_earth_about_payload(payload.model_dump())
record = await _get_earth_about_record(db)
if record is None:
record = SystemSetting(category=EARTH_ABOUT_CATEGORY, payload=normalized)
db.add(record)
else:
record.payload = normalized
await db.commit()
await db.refresh(record)
return {"status": "updated", "about": _normalize_earth_about_payload(record.payload), "is_default": False}
@router.delete("/about")
async def reset_earth_about(
_current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
await db.execute(delete(SystemSetting).where(SystemSetting.category == EARTH_ABOUT_CATEGORY))
await db.commit()
return {"status": "reset", "about": _normalize_earth_about_payload(None), "is_default": True}
@router.get("/oobe-status")
async def get_earth_oobe_status(
current_user: User | None = Depends(_get_optional_current_user),
db: AsyncSession = Depends(get_db),
):
current_count_result = await db.execute(
select(func.count(CollectedData.id)).where(CollectedData.is_current.is_(True))
)
current_record_count = int(current_count_result.scalar() or 0)
system_result = await db.execute(
select(SystemSetting).where(SystemSetting.category == SYSTEM_SETTINGS_CATEGORY)
)
system_record = system_result.scalar_one_or_none()
demo_mode = _is_demo_mode_enabled(system_record.payload if system_record else None)
datasource_count_result = await db.execute(select(func.count(DataSource.id)))
datasource_count = int(datasource_count_result.scalar() or 0)
active_datasource_count_result = await db.execute(
select(func.count(DataSource.id)).where(DataSource.is_active.is_(True))
)
active_datasource_count = int(active_datasource_count_result.scalar() or 0)
config_result = await db.execute(select(func.count(DataSourceConfig.id)))
custom_config_count = int(config_result.scalar() or 0)
tv_payload = await get_tv_settings_payload(db)
tv_sources = tv_payload.get("sources") if isinstance(tv_payload, dict) else []
tv_source_count = len(tv_sources) if isinstance(tv_sources, list) else 0
boundary_status = get_boundary_status()
has_core_layers = bool(boundary_status.get("ready") or boundary_status.get("available") or boundary_status.get("status") in {"ready", "built", "ok"})
has_collected_data = current_record_count > 0
ready = has_collected_data
suggestions: list[str] = []
if demo_mode:
suggestions.append("演示模式已开启")
if not current_user:
suggestions.append("登录控制台")
if not has_collected_data:
suggestions.append("触发数据源采集")
if not custom_config_count:
suggestions.append("确认采集器配置")
if not has_core_layers:
suggestions.append("构建或启用 Earth 图层")
return {
"ready": ready,
"demo_mode": demo_mode,
"authenticated": current_user is not None,
"needs_login": current_user is None and not ready and not demo_mode,
"has_collected_data": has_collected_data,
"has_tv_sources": tv_source_count > 0,
"has_core_layers": has_core_layers,
"current_record_count": current_record_count,
"datasource_count": datasource_count,
"active_datasource_count": active_datasource_count,
"custom_config_count": custom_config_count,
"tv_source_count": tv_source_count,
"suggestions": suggestions,
"login_url": "/login?next=/datasources",
"datasources_url": "/datasources",
"collection_url": "/collection-management",
}
@router.get("/boundaries/status")
async def get_earth_boundary_status():
return get_boundary_status()
async def _get_optional_current_user(
credentials: HTTPAuthorizationCredentials | None = Depends(optional_bearer),
db: AsyncSession = Depends(get_db),
) -> User | None:
if credentials is None:
return None
token = credentials.credentials
if redis_client.sismember("blacklisted_tokens", token):
return None
payload = decode_token(token)
if payload is None or payload.get("type") != "access":
return None
user_id = payload.get("sub")
if user_id is None:
return None
result = await db.execute(
text(
"SELECT id, username, email, password_hash, role, is_active, gatekeeper_groups FROM users WHERE id = :id"
),
{"id": int(user_id)},
)
row = result.fetchone()
if row is None or not row[5]:
return None
user = User()
user.id = row[0]
user.username = row[1]
user.email = row[2]
user.password_hash = row[3]
user.role = row[4]
user.is_active = row[5]
user.gatekeeper_groups = row[6] or []
return user
def _is_loopback_request(request: Request) -> bool:
host = request.client.host if request.client else ""
return host in {"127.0.0.1", "::1", "localhost"} or host.startswith("127.")

View File

@@ -0,0 +1,190 @@
"""CRUD APIs for persistent Earth interactables."""
from __future__ import annotations
from datetime import UTC, datetime
from typing import Any
from fastapi import APIRouter, Depends, HTTPException, Query, Response, status
from pydantic import BaseModel, Field, field_validator
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.security import get_current_user
from app.db.session import get_db
from app.models.earth_interactable import EarthInteractable
from app.models.user import User
from app.services.earth_interactables import (
build_interactable_event,
interactables_to_geojson,
invalidate_interactable_cache,
list_interactables,
normalize_interactable_id,
publish_interactable_event,
serialize_interactable,
)
from app.services.earth_layer_cache import (
EarthLayerCachePolicy,
earth_layer_cache,
get_or_build_layer_payload,
)
router = APIRouter()
INTERACTABLE_CACHE_POLICY = EarthLayerCachePolicy(
fresh_ttl_seconds=60,
stale_ttl_seconds=10 * 60,
max_features=5000,
)
class InteractableCreate(BaseModel):
id: str | None = Field(default=None, max_length=160)
layer: str = Field(default="default", min_length=1, max_length=80)
kind: str = Field(default="default", min_length=1, max_length=80)
label: str = Field(default="", max_length=255)
description: str = Field(default="", max_length=4000)
latitude: float = Field(ge=-90, le=90)
longitude: float = Field(ge=-180, le=180)
altitude: float | None = None
properties: dict[str, Any] = Field(default_factory=dict)
@field_validator("layer", "kind")
@classmethod
def normalize_key(cls, value: str) -> str:
normalized = str(value or "").strip()
if not normalized:
raise ValueError("must not be empty")
return normalized
class InteractableUpdate(BaseModel):
layer: str | None = Field(default=None, min_length=1, max_length=80)
kind: str | None = Field(default=None, min_length=1, max_length=80)
label: str | None = Field(default=None, max_length=255)
description: str | None = Field(default=None, max_length=4000)
latitude: float | None = Field(default=None, ge=-90, le=90)
longitude: float | None = Field(default=None, ge=-180, le=180)
altitude: float | None = None
properties: dict[str, Any] | None = None
@router.get("")
async def get_interactables(
response: Response,
layer: str | None = Query(default=None),
include_deleted: bool = Query(default=False),
db: AsyncSession = Depends(get_db),
):
items = await list_interactables(db, layer=layer, include_deleted=include_deleted)
response.headers["X-Planet-Interactables-Count"] = str(len(items))
return {"items": [serialize_interactable(item) for item in items]}
@router.get("/geojson")
async def get_interactables_geojson(
response: Response,
layer: str | None = Query(default=None),
db: AsyncSession = Depends(get_db),
):
async def build_payload() -> dict[str, Any]:
items = await list_interactables(db, layer=layer)
return interactables_to_geojson(items)
payload = await get_or_build_layer_payload(
key=earth_layer_cache.key("interactables", layer=layer or "all"),
policy=INTERACTABLE_CACHE_POLICY,
builder=build_payload,
response=response,
)
response.headers["X-Planet-Interactables-Count"] = str(len(payload.get("features") or []))
return payload
@router.post("", status_code=status.HTTP_201_CREATED)
async def create_interactable(
payload: InteractableCreate,
_current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
record_id = normalize_interactable_id(payload.id)
existing = await db.get(EarthInteractable, record_id)
if existing and not existing.is_deleted:
raise HTTPException(status_code=409, detail="Interactable already exists")
if existing is None:
record = EarthInteractable(id=record_id)
db.add(record)
else:
record = existing
record.is_deleted = False
record.deleted_at = None
record.revision += 1
record.layer = payload.layer
record.kind = payload.kind
record.label = payload.label
record.description = payload.description
record.latitude = payload.latitude
record.longitude = payload.longitude
record.altitude = payload.altitude
record.properties = payload.properties
await db.commit()
await db.refresh(record)
invalidate_interactable_cache(record.layer)
await publish_interactable_event("created", record)
return {"item": serialize_interactable(record)}
@router.get("/{interactable_id}")
async def get_interactable(interactable_id: str, db: AsyncSession = Depends(get_db)):
record = await db.get(EarthInteractable, interactable_id)
if record is None or record.is_deleted:
raise HTTPException(status_code=404, detail="Interactable not found")
return {"item": serialize_interactable(record)}
@router.patch("/{interactable_id}")
async def update_interactable(
interactable_id: str,
payload: InteractableUpdate,
_current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
record = await db.get(EarthInteractable, interactable_id)
if record is None or record.is_deleted:
raise HTTPException(status_code=404, detail="Interactable not found")
previous_layer = record.layer
patch = payload.model_dump(exclude_unset=True)
for key, value in patch.items():
setattr(record, key, value)
record.revision += 1
await db.commit()
await db.refresh(record)
invalidate_interactable_cache(previous_layer)
if record.layer != previous_layer:
invalidate_interactable_cache(record.layer)
await publish_interactable_event("updated", record)
return {"item": serialize_interactable(record)}
@router.delete("/{interactable_id}")
async def delete_interactable(
interactable_id: str,
_current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
record = await db.get(EarthInteractable, interactable_id)
if record is None or record.is_deleted:
raise HTTPException(status_code=404, detail="Interactable not found")
record.is_deleted = True
record.deleted_at = datetime.now(UTC)
record.revision += 1
await db.commit()
await db.refresh(record)
invalidate_interactable_cache(record.layer)
await publish_interactable_event("deleted", record)
event = build_interactable_event(action="deleted", record=record, include_item=True)
return {"deleted": True, "event": event}

View File

@@ -11,6 +11,7 @@ from dotenv import dotenv_values
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.logging import get_logger
from app.core.security import get_current_user
from app.core.time import to_iso8601_utc
from app.core.config import settings as app_settings
@@ -66,8 +67,10 @@ from app.services.llm_provider_catalog import (
from app.services.scheduler import sync_datasource_job
from app.services.tv_streams import DEFAULT_TV_SETTINGS, get_tv_settings_payload, normalize_tv_settings
from app.services.persistent_logs import record_audit_log
from app.services.business_logs import emit_business_log, exception_context
router = APIRouter()
logger = get_logger(__name__, service="api")
AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS = 5
AI_CONNECTION_TEST_PROMPT_KEY = "ai.connection_test"
SECRET_REVEAL_ROLES = {"admin", "super_admin"}
@@ -79,6 +82,7 @@ DEFAULT_SETTINGS = {
"auto_refresh": True,
"data_retention_days": 30,
"max_concurrent_tasks": 5,
"demo_mode": False,
},
"notifications": {
"email_enabled": False,
@@ -203,6 +207,7 @@ class SystemSettingsUpdate(BaseModel):
auto_refresh: bool = True
data_retention_days: int = Field(default=30, ge=1, le=3650)
max_concurrent_tasks: int = Field(default=5, ge=1, le=50)
demo_mode: bool = False
class NotificationSettingsUpdate(BaseModel):
@@ -699,6 +704,18 @@ async def _validate_ai_provider_full_connection(ai_payload: dict) -> dict:
retry_attempts=runtime_config["retry_attempts"],
llm_config=runtime_config.get("llm_config") or {},
)
await emit_business_log(
logger,
event="settings.ai_provider.full_connection.start",
message="AI provider full connection validation started",
category="ai",
service="api",
module=__name__,
context={
"provider": runtime_config.get("llm_config", {}).get("provider"),
"model": runtime_config.get("llm_config", {}).get("model"),
},
)
status_result = await client.get_status()
if not status_result.configured:
raise HTTPException(
@@ -715,6 +732,19 @@ async def _validate_ai_provider_full_connection(ai_payload: dict) -> dict:
constraints=["回复尽量简短。"],
)
)
await emit_business_log(
logger,
event="settings.ai_provider.full_connection.success",
message="AI provider full connection validation completed",
category="ai",
service="api",
module=__name__,
context={
"provider": analysis_result.provider,
"model": analysis_result.model,
"configured": status_result.configured,
},
)
return {
"status": status_result.model_dump(),
"provider": analysis_result.provider,
@@ -1612,9 +1642,34 @@ async def connect_ai_provider_integration(
llm_config=quick_llm_config,
)
await emit_business_log(
logger,
event="settings.ai_provider.connect.start",
message="AI provider connection test started",
category="ai",
service="api",
module=__name__,
user_id=current_user.id,
context={
"provider": payload.provider,
"model": payload.model,
"timeout_seconds": min(int(runtime_config["timeout_seconds"] or 60), AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS),
},
)
try:
status_result = await client.get_status()
if not status_result.configured:
await emit_business_log(
logger,
event="settings.ai_provider.connect.failed",
message="AI provider connection test failed because provider is incomplete",
category="ai",
level="warning",
service="api",
module=__name__,
user_id=current_user.id,
context={"provider": payload.provider, "model": payload.model, "configured": False},
)
return {
"success": False,
"connected": False,
@@ -1628,6 +1683,21 @@ async def connect_ai_provider_integration(
AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS,
),
)
await emit_business_log(
logger,
event="settings.ai_provider.connect.success",
message="AI provider connection test completed",
category="ai",
service="api",
module=__name__,
user_id=current_user.id,
context={
"provider": payload.provider,
"model": payload.model,
"configured": True,
"lightweight_status": lightweight_result.get("status"),
},
)
return {
**lightweight_result,
"status": status_result.model_dump(),
@@ -1639,6 +1709,17 @@ async def connect_ai_provider_integration(
"message": str(exc.detail),
}
except Exception as exc:
await emit_business_log(
logger,
event="settings.ai_provider.connect.failed",
message="AI provider connection test failed",
category="ai",
level="error",
service="api",
module=__name__,
user_id=current_user.id,
context=exception_context(exc, {"provider": payload.provider, "model": payload.model}),
)
return {
"success": False,
"connected": False,
@@ -1786,8 +1867,28 @@ async def connect_web_search_integration(
runtime_config = _runtime_config_from_web_search_payload(draft_web_search_payload)
client = WebSearchClient(runtime_config)
await emit_business_log(
logger,
event="settings.web_search.connect.start",
message="WebSearch connection test started",
category="ai_tool",
service="api",
module=__name__,
user_id=current_user.id,
context={"provider": runtime_config.default_provider},
)
try:
results = await client.test_connection()
await emit_business_log(
logger,
event="settings.web_search.connect.success",
message="WebSearch connection test completed",
category="ai_tool",
service="api",
module=__name__,
user_id=current_user.id,
context={"provider": runtime_config.default_provider, "result_count": len(results)},
)
return {
"success": True,
"connected": True,
@@ -1796,18 +1897,51 @@ async def connect_web_search_integration(
"results": [item.model_dump(mode="json") for item in results[:3]],
}
except WebSearchConfigurationError as exc:
await emit_business_log(
logger,
event="settings.web_search.connect.failed",
message="WebSearch connection test failed because configuration is incomplete",
category="ai_tool",
level="warning",
service="api",
module=__name__,
user_id=current_user.id,
context=exception_context(exc, {"provider": runtime_config.default_provider}),
)
return {
"success": False,
"connected": False,
"message": str(exc),
}
except WebSearchError as exc:
await emit_business_log(
logger,
event="settings.web_search.connect.failed",
message="WebSearch connection test failed",
category="ai_tool",
level="error",
service="api",
module=__name__,
user_id=current_user.id,
context=exception_context(exc, {"provider": runtime_config.default_provider}),
)
return {
"success": False,
"connected": False,
"message": str(exc),
}
except Exception as exc:
await emit_business_log(
logger,
event="settings.web_search.connect.failed",
message="WebSearch connection test failed",
category="ai_tool",
level="error",
service="api",
module=__name__,
user_id=current_user.id,
context=exception_context(exc, {"provider": runtime_config.default_provider}),
)
return {
"success": False,
"connected": False,
@@ -1835,17 +1969,60 @@ async def generate_provider_credential_guide(
ai_client: AIProviderClient = Depends(get_ai_provider_client),
):
try:
await emit_business_log(
logger,
event="settings.credential_guide.generate.start",
message="Credential guide generation started",
category="ai",
service="api",
module=__name__,
user_id=current_user.id,
context={"provider": provider},
)
web_search_client = await get_web_search_client(db)
return {
"guide": await generate_credential_guide(
guide = await generate_credential_guide(
db,
provider,
ai_client,
web_search_client,
)
}
await emit_business_log(
logger,
event="settings.credential_guide.generate.success",
message="Credential guide generation completed",
category="ai",
service="api",
module=__name__,
user_id=current_user.id,
context={"provider": provider},
)
return {"guide": guide}
except ValueError as exc:
await emit_business_log(
logger,
event="settings.credential_guide.generate.failed",
message="Credential guide generation failed",
category="ai",
level="warning",
service="api",
module=__name__,
user_id=current_user.id,
context=exception_context(exc, {"provider": provider}),
)
raise HTTPException(status_code=404, detail=str(exc)) from exc
except Exception as exc:
await emit_business_log(
logger,
event="settings.credential_guide.generate.failed",
message="Credential guide generation failed",
category="ai",
level="error",
service="api",
module=__name__,
user_id=current_user.id,
context=exception_context(exc, {"provider": provider}),
)
raise
@router.post("/credential-guides/{provider}/reset")

View File

@@ -1,6 +1,7 @@
from __future__ import annotations
import os
import json
import subprocess
import sys
@@ -44,6 +45,42 @@ from app.services.earth_layer_cache import earth_layer_cache
router = APIRouter()
def _compact_log_context(context: dict | None) -> str:
if not context:
return ""
allowed = {
key: value
for key, value in (context or {}).items()
if key
in {
"status",
"duration_ms",
"provider",
"model",
"result_provider",
"result_model",
"collector_name",
"datasource_id",
"task_id",
"snapshot_id",
"raw_count",
"transformed_count",
"saved_count",
"created",
"updated",
"unchanged",
"deleted",
"result_count",
"status_code",
"error_type",
"error",
}
}
if not allowed:
return ""
return json.dumps(allowed, ensure_ascii=False, sort_keys=True)
class RestartTaskCreate(BaseModel):
action: str
@@ -374,8 +411,11 @@ async def read_database_log_snapshot(
record.occurred_at.isoformat() if record.occurred_at else "",
record_level.upper(),
record.source,
record.category or "",
record.event or "",
f"request_id={record.request_id}" if record.request_id else "",
record.message,
_compact_log_context(record.context),
]
if part
)

View File

@@ -38,7 +38,7 @@ from app.services.compute_center_locations import (
upsert_compute_center_location,
)
from app.services.ai_client import get_ai_provider_client
from app.api.v1.settings import get_web_search_client
from app.api.v1.settings import get_runtime_web_search_config, get_web_search_client
from app.services.collectors.bgp_common import RIPE_RIS_COLLECTOR_COORDS
from app.services.location.llm_fallback import (
collect_llm_location_fallback_candidate,
@@ -105,8 +105,8 @@ LANDING_POINT_CACHE_POLICY = EarthLayerCachePolicy(
SATELLITE_CACHE_POLICY = EarthLayerCachePolicy(
SATELLITE_CACHE_FRESH_SECONDS,
SATELLITE_CACHE_STALE_SECONDS,
max_features=8000,
max_bytes=10 * BYTES_PER_MIB,
max_features=25000,
max_bytes=32 * BYTES_PER_MIB,
)
COMPUTE_CENTER_CACHE_POLICY = EarthLayerCachePolicy(
COMPUTE_CENTER_CACHE_FRESH_SECONDS,
@@ -1563,15 +1563,7 @@ async def _build_cables_geojson(db: AsyncSession) -> dict[str, Any]:
try:
records = await _load_current_collected_data(db, "arcgis_cables")
if not records:
raise HTTPException(
status_code=404,
detail="No cable data found. Please run the arcgis_cables collector first.",
)
return convert_cable_to_geojson(records)
except HTTPException:
raise
except Exception as e:
logger.exception_event(
"Failed to build cables GeoJSON response",
@@ -1625,16 +1617,8 @@ async def _build_landing_points_geojson(db: AsyncSession) -> dict[str, Any]:
relation_records,
cable_records,
)
if not records:
raise HTTPException(
status_code=404,
detail="No landing point data found. Please run the arcgis_landing_points collector first.",
)
return convert_landing_point_to_geojson(records, city_to_cable_ids_map, cable_id_to_name_map)
except HTTPException:
raise
except Exception as e:
logger.exception_event(
"Failed to build landing points GeoJSON response",
@@ -2071,6 +2055,44 @@ class SaveComputeCenterLocationRequest(BaseModel):
model_config = {"populate_by_name": True}
async def _compute_center_location_web_search_capability(db: AsyncSession) -> Dict[str, Any]:
try:
config = await get_runtime_web_search_config(db)
except Exception as exc:
return {
"enabled": False,
"provider": None,
"reason": f"WebSearch 配置读取失败:{exc}",
}
provider_config = config.active_provider_config
has_api_key = bool((provider_config.api_key or "").strip())
if not config.enabled:
return {
"enabled": False,
"provider": config.default_provider,
"reason": "WebSearch 未开启,无法进行事实核查定位。",
}
if not has_api_key:
return {
"enabled": False,
"provider": config.default_provider,
"reason": f"WebSearch Provider {config.default_provider} 未配置 API Key。",
}
return {
"enabled": True,
"provider": config.default_provider,
"reason": "",
}
@router.get("/compute-centers/location-capability")
async def get_compute_center_location_capability(
db: AsyncSession = Depends(get_db),
):
"""Return whether fact-checked compute-center location collection can run."""
return await _compute_center_location_web_search_capability(db)
@router.post("/compute-centers/{source_id}/collect-location")
async def collect_compute_center_location(
source_id: str,
@@ -2089,6 +2111,9 @@ async def collect_compute_center_location(
"""
if not source_id or not source_id.strip():
raise HTTPException(status_code=400, detail="source_id is required")
capability = await _compute_center_location_web_search_capability(db)
if not capability.get("enabled"):
raise HTTPException(status_code=409, detail=capability)
record = await _load_compute_center_record(db, source_id)
name = payload.name or (record.name if record else None)
@@ -2696,6 +2721,8 @@ async def _build_bgp_collectors_geojson(db: AsyncSession) -> dict[str, Any]:
db,
source_filter=("ris_live_bgp", "bgpstream_bgp"),
)
if not any(int(item.get("observation_count") or 0) > 0 for item in coverage):
return {"type": "FeatureCollection", "features": [], "count": 0}
coverage_by_collector = {
item["collector"]: item
for item in coverage

View File

@@ -150,7 +150,7 @@ class DataBroadcaster:
"timestamp": to_iso8601_utc(datetime.now(UTC)),
"payload": data,
},
channel="all",
channel="datasource_tasks",
)
def start(self):

View File

@@ -150,7 +150,7 @@ DEFAULT_LOGIN_USERS = (
{
"username": "linkong",
"email": "linkong@planet.local",
"password": "12345678",
"password": "LK12345678",
"role": "super_admin",
},
)
@@ -203,6 +203,7 @@ async def init_db():
import app.models.vessel_enrichment # noqa: F401
import app.models.datasource_mapping # noqa: F401
import app.models.earth_news # noqa: F401
import app.models.earth_interactable # noqa: F401
logger.warning_event(
"Database pool settings active",
@@ -258,6 +259,406 @@ async def init_db():
"""
)
)
await conn.execute(
text(
"""
CREATE TABLE IF NOT EXISTS earth_data_change_events (
id BIGSERIAL PRIMARY KEY,
table_name VARCHAR(128) NOT NULL,
operation VARCHAR(16) NOT NULL,
source VARCHAR(128),
entity_key VARCHAR(255),
payload JSONB NOT NULL,
occurred_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
consumed_at TIMESTAMPTZ
)
"""
)
)
await conn.execute(
text(
"""
CREATE INDEX IF NOT EXISTS idx_earth_data_change_events_unconsumed
ON earth_data_change_events (consumed_at, id)
WHERE consumed_at IS NULL
"""
)
)
await conn.execute(
text(
"""
CREATE OR REPLACE FUNCTION planet_emit_earth_data_changed_statement(
change_table TEXT,
change_operation TEXT,
change_source TEXT,
source_record_count INTEGER,
source_entity_keys TEXT[]
)
RETURNS VOID AS $$
DECLARE
change_event_id BIGINT;
change_payload JSONB;
BEGIN
change_payload := jsonb_build_object(
'event', 'earth.layer.changed',
'table', change_table,
'operation', change_operation,
'source', change_source,
'entity_key', NULL,
'entity_keys', COALESCE(to_jsonb(source_entity_keys), '[]'::jsonb),
'records_processed', COALESCE(source_record_count, 0),
'occurred_at', NOW()
);
INSERT INTO earth_data_change_events (
table_name,
operation,
source,
entity_key,
payload,
occurred_at
) VALUES (
change_table,
change_operation,
change_source,
NULL,
change_payload,
NOW()
)
RETURNING id INTO change_event_id;
change_payload := change_payload || jsonb_build_object(
'event_id', change_event_id
);
UPDATE earth_data_change_events
SET payload = change_payload
WHERE id = change_event_id;
PERFORM pg_notify(
'planet_earth_data_changes',
change_payload::text
);
END;
$$ LANGUAGE plpgsql;
"""
)
)
await conn.execute(
text(
"""
CREATE OR REPLACE FUNCTION planet_emit_collected_data_changed_statement(
change_operation TEXT,
change_source TEXT,
source_record_count INTEGER,
source_entity_keys TEXT[]
)
RETURNS VOID AS $$
BEGIN
PERFORM planet_emit_earth_data_changed_statement(
'collected_data',
change_operation,
change_source,
source_record_count,
source_entity_keys
);
END;
$$ LANGUAGE plpgsql;
"""
)
)
await conn.execute(
text(
"""
CREATE OR REPLACE FUNCTION planet_notify_earth_table_changed_statement()
RETURNS trigger AS $$
DECLARE
change_source TEXT;
source_record_count INTEGER;
source_entity_keys TEXT[];
BEGIN
IF TG_OP = 'INSERT' THEN
FOR change_source IN
SELECT DISTINCT COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME)
FROM (SELECT to_jsonb(t) AS row_data FROM new_rows AS t) changed_rows
LOOP
SELECT
COUNT(*),
ARRAY(
SELECT DISTINCT COALESCE(
NULLIF(row_data->>'entity_key', ''),
NULLIF(row_data->>'source_id', ''),
NULLIF(row_data->>'incident_key', ''),
NULLIF(row_data->>'id', ''),
NULLIF(row_data->>'mmsi', '')
)
FROM (SELECT to_jsonb(t) AS row_data FROM new_rows AS t) rows_for_keys
WHERE COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) = change_source
LIMIT 20
)
INTO source_record_count, source_entity_keys
FROM (SELECT to_jsonb(t) AS row_data FROM new_rows AS t) rows_for_count
WHERE COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) = change_source;
PERFORM planet_emit_earth_data_changed_statement(
TG_TABLE_NAME,
TG_OP,
change_source,
source_record_count,
source_entity_keys
);
END LOOP;
ELSIF TG_OP = 'DELETE' THEN
FOR change_source IN
SELECT DISTINCT COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME)
FROM (SELECT to_jsonb(t) AS row_data FROM old_rows AS t) changed_rows
LOOP
SELECT
COUNT(*),
ARRAY(
SELECT DISTINCT COALESCE(
NULLIF(row_data->>'entity_key', ''),
NULLIF(row_data->>'source_id', ''),
NULLIF(row_data->>'incident_key', ''),
NULLIF(row_data->>'id', ''),
NULLIF(row_data->>'mmsi', '')
)
FROM (SELECT to_jsonb(t) AS row_data FROM old_rows AS t) rows_for_keys
WHERE COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) = change_source
LIMIT 20
)
INTO source_record_count, source_entity_keys
FROM (SELECT to_jsonb(t) AS row_data FROM old_rows AS t) rows_for_count
WHERE COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) = change_source;
PERFORM planet_emit_earth_data_changed_statement(
TG_TABLE_NAME,
TG_OP,
change_source,
source_record_count,
source_entity_keys
);
END LOOP;
ELSIF TG_OP = 'UPDATE' THEN
FOR change_source IN
SELECT DISTINCT COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME)
FROM (
SELECT to_jsonb(t) AS row_data FROM new_rows AS t
UNION ALL
SELECT to_jsonb(t) AS row_data FROM old_rows AS t
) changed_rows
LOOP
SELECT
COUNT(*),
ARRAY(
SELECT DISTINCT COALESCE(
NULLIF(row_data->>'entity_key', ''),
NULLIF(row_data->>'source_id', ''),
NULLIF(row_data->>'incident_key', ''),
NULLIF(row_data->>'id', ''),
NULLIF(row_data->>'mmsi', '')
)
FROM (
SELECT to_jsonb(t) AS row_data FROM new_rows AS t
UNION ALL
SELECT to_jsonb(t) AS row_data FROM old_rows AS t
) rows_for_keys
WHERE COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) = change_source
LIMIT 20
)
INTO source_record_count, source_entity_keys
FROM (
SELECT to_jsonb(t) AS row_data FROM new_rows AS t
UNION ALL
SELECT to_jsonb(t) AS row_data FROM old_rows AS t
) rows_for_count
WHERE COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) = change_source;
PERFORM planet_emit_earth_data_changed_statement(
TG_TABLE_NAME,
TG_OP,
change_source,
source_record_count,
source_entity_keys
);
END LOOP;
END IF;
RETURN NULL;
END;
$$ LANGUAGE plpgsql;
"""
)
)
await conn.execute(
text(
"""
CREATE OR REPLACE FUNCTION planet_notify_collected_data_changed_statement()
RETURNS trigger AS $$
DECLARE
change_source TEXT;
source_record_count INTEGER;
source_entity_keys TEXT[];
BEGIN
IF TG_OP = 'INSERT' THEN
FOR change_source IN
SELECT DISTINCT source FROM new_rows WHERE source IS NOT NULL
LOOP
SELECT
COUNT(*),
ARRAY(
SELECT DISTINCT COALESCE(entity_key, source_id, id::text)
FROM new_rows
WHERE source = change_source
LIMIT 20
)
INTO source_record_count, source_entity_keys
FROM new_rows
WHERE source = change_source;
PERFORM planet_emit_collected_data_changed_statement(
TG_OP,
change_source,
source_record_count,
source_entity_keys
);
END LOOP;
ELSIF TG_OP = 'DELETE' THEN
FOR change_source IN
SELECT DISTINCT source FROM old_rows WHERE source IS NOT NULL
LOOP
SELECT
COUNT(*),
ARRAY(
SELECT DISTINCT COALESCE(entity_key, source_id, id::text)
FROM old_rows
WHERE source = change_source
LIMIT 20
)
INTO source_record_count, source_entity_keys
FROM old_rows
WHERE source = change_source;
PERFORM planet_emit_collected_data_changed_statement(
TG_OP,
change_source,
source_record_count,
source_entity_keys
);
END LOOP;
ELSIF TG_OP = 'UPDATE' THEN
FOR change_source IN
SELECT DISTINCT source FROM (
SELECT source FROM new_rows
UNION
SELECT source FROM old_rows
) changed_sources
WHERE source IS NOT NULL
LOOP
SELECT
COUNT(*),
ARRAY(
SELECT DISTINCT COALESCE(entity_key, source_id, id::text)
FROM (
SELECT id, source_id, entity_key, source FROM new_rows
UNION ALL
SELECT id, source_id, entity_key, source FROM old_rows
) changed_rows
WHERE source = change_source
LIMIT 20
)
INTO source_record_count, source_entity_keys
FROM (
SELECT id, source_id, entity_key, source FROM new_rows
UNION ALL
SELECT id, source_id, entity_key, source FROM old_rows
) changed_rows
WHERE source = change_source;
PERFORM planet_emit_collected_data_changed_statement(
TG_OP,
change_source,
source_record_count,
source_entity_keys
);
END LOOP;
END IF;
RETURN NULL;
END;
$$ LANGUAGE plpgsql;
"""
)
)
for statement in (
"DROP TRIGGER IF EXISTS tr_planet_collected_data_changed ON collected_data",
"DROP TRIGGER IF EXISTS tr_planet_collected_data_changed_insert ON collected_data",
"DROP TRIGGER IF EXISTS tr_planet_collected_data_changed_update ON collected_data",
"DROP TRIGGER IF EXISTS tr_planet_collected_data_changed_delete ON collected_data",
"DROP FUNCTION IF EXISTS planet_notify_collected_data_changed()",
"""
CREATE TRIGGER tr_planet_collected_data_changed_insert
AFTER INSERT ON collected_data
REFERENCING NEW TABLE AS new_rows
FOR EACH STATEMENT
EXECUTE FUNCTION planet_notify_collected_data_changed_statement()
""",
"""
CREATE TRIGGER tr_planet_collected_data_changed_update
AFTER UPDATE ON collected_data
REFERENCING OLD TABLE AS old_rows NEW TABLE AS new_rows
FOR EACH STATEMENT
EXECUTE FUNCTION planet_notify_collected_data_changed_statement()
""",
"""
CREATE TRIGGER tr_planet_collected_data_changed_delete
AFTER DELETE ON collected_data
REFERENCING OLD TABLE AS old_rows
FOR EACH STATEMENT
EXECUTE FUNCTION planet_notify_collected_data_changed_statement()
""",
):
await conn.execute(text(statement))
for table_name in (
"bgp_observations",
"bgp_anomalies",
"bgp_incidents",
"bgp_collector_locations",
"vessel_static",
"vessel_position",
"ais_raw_observations",
"ais_source_health",
"compute_center_locations",
"earth_interactables",
"earth_news_items",
):
for statement in (
f"DROP TRIGGER IF EXISTS tr_planet_{table_name}_changed_insert ON {table_name}",
f"DROP TRIGGER IF EXISTS tr_planet_{table_name}_changed_update ON {table_name}",
f"DROP TRIGGER IF EXISTS tr_planet_{table_name}_changed_delete ON {table_name}",
f"""
CREATE TRIGGER tr_planet_{table_name}_changed_insert
AFTER INSERT ON {table_name}
REFERENCING NEW TABLE AS new_rows
FOR EACH STATEMENT
EXECUTE FUNCTION planet_notify_earth_table_changed_statement()
""",
f"""
CREATE TRIGGER tr_planet_{table_name}_changed_update
AFTER UPDATE ON {table_name}
REFERENCING OLD TABLE AS old_rows NEW TABLE AS new_rows
FOR EACH STATEMENT
EXECUTE FUNCTION planet_notify_earth_table_changed_statement()
""",
f"""
CREATE TRIGGER tr_planet_{table_name}_changed_delete
AFTER DELETE ON {table_name}
REFERENCING OLD TABLE AS old_rows
FOR EACH STATEMENT
EXECUTE FUNCTION planet_notify_earth_table_changed_statement()
""",
):
await conn.execute(text(statement))
await conn.execute(
text(
"""
@@ -267,7 +668,16 @@ async def init_db():
ADD COLUMN IF NOT EXISTS phase_message VARCHAR(255),
ADD COLUMN IF NOT EXISTS phase_current BIGINT,
ADD COLUMN IF NOT EXISTS phase_total BIGINT,
ADD COLUMN IF NOT EXISTS phase_unit VARCHAR(30)
ADD COLUMN IF NOT EXISTS phase_unit VARCHAR(30),
ADD COLUMN IF NOT EXISTS source VARCHAR(100),
ADD COLUMN IF NOT EXISTS task_type VARCHAR(30) NOT NULL DEFAULT 'collect',
ADD COLUMN IF NOT EXISTS payload JSONB NOT NULL DEFAULT '{}'::jsonb,
ADD COLUMN IF NOT EXISTS rollback_policy VARCHAR(40) NOT NULL DEFAULT 'keep_committed_batches',
ADD COLUMN IF NOT EXISTS dedupe_key VARCHAR(180),
ADD COLUMN IF NOT EXISTS worker_id VARCHAR(120),
ADD COLUMN IF NOT EXISTS locked_at TIMESTAMPTZ,
ADD COLUMN IF NOT EXISTS requested_cancel_at TIMESTAMPTZ,
ADD COLUMN IF NOT EXISTS cancel_reason TEXT
"""
)
)
@@ -283,6 +693,17 @@ async def init_db():
"""
)
)
await conn.execute(
text(
"""
ALTER TABLE earth_interactables
ADD COLUMN IF NOT EXISTS altitude DOUBLE PRECISION,
ADD COLUMN IF NOT EXISTS revision INTEGER NOT NULL DEFAULT 1,
ADD COLUMN IF NOT EXISTS is_deleted BOOLEAN NOT NULL DEFAULT FALSE,
ADD COLUMN IF NOT EXISTS deleted_at TIMESTAMPTZ
"""
)
)
await conn.execute(
text(
"""
@@ -307,6 +728,48 @@ async def init_db():
"""
)
)
await conn.execute(
text(
"""
CREATE INDEX IF NOT EXISTS idx_earth_interactables_layer_deleted
ON earth_interactables (layer, is_deleted)
"""
)
)
await conn.execute(
text(
"""
CREATE INDEX IF NOT EXISTS idx_earth_interactables_updated_at
ON earth_interactables (updated_at)
"""
)
)
await conn.execute(
text(
"""
CREATE INDEX IF NOT EXISTS idx_collection_tasks_source_status
ON collection_tasks (source, status)
"""
)
)
await conn.execute(
text(
"""
CREATE INDEX IF NOT EXISTS idx_collection_tasks_queue
ON collection_tasks (status, created_at, id)
WHERE status = 'queued'
"""
)
)
await conn.execute(
text(
"""
CREATE INDEX IF NOT EXISTS idx_collection_tasks_dedupe
ON collection_tasks (dedupe_key)
WHERE dedupe_key IS NOT NULL
"""
)
)
await conn.execute(
text(
"""

View File

@@ -24,6 +24,11 @@ from app.services.earth_news_worker import (
start_earth_news_target_worker,
stop_earth_news_target_worker,
)
from app.services.earth_db_change_listener import (
start_earth_db_change_listener,
stop_earth_db_change_listener,
)
from app.services.data_jobs import start_data_job_worker, stop_data_job_worker
configure_logging()
@@ -59,9 +64,13 @@ async def lifespan(app: FastAPI):
start_scheduler()
await sync_scheduler_with_datasources()
broadcaster.start()
start_data_job_worker()
start_earth_db_change_listener()
start_earth_news_target_worker()
yield
await stop_earth_news_target_worker()
await stop_earth_db_change_listener()
await stop_data_job_worker()
broadcaster.stop()
stop_scheduler()

View File

@@ -17,6 +17,7 @@ from app.models.system_log import SystemLog, AuditLog
from app.models.vessel import AISConflictRecord, AISRawObservation, AISSourceHealth, VesselPosition, VesselStatic
from app.models.datasource_mapping import DataSourceMappingTemplate
from app.models.earth_news import EarthNewsItem
from app.models.earth_interactable import EarthInteractable
__all__ = [
"User",
@@ -45,4 +46,5 @@ __all__ = [
"AISSourceHealth",
"DataSourceMappingTemplate",
"EarthNewsItem",
"EarthInteractable",
]

View File

@@ -0,0 +1,30 @@
"""Persistent Earth interactable objects."""
from sqlalchemy import Boolean, Column, DateTime, Float, Index, Integer, JSON, String, Text
from sqlalchemy.sql import func
from app.db.session import Base
class EarthInteractable(Base):
__tablename__ = "earth_interactables"
id = Column(String(160), primary_key=True)
layer = Column(String(80), nullable=False, default="interactables", index=True)
kind = Column(String(80), nullable=False, default="default", index=True)
label = Column(String(255), nullable=False, default="")
description = Column(Text, nullable=False, default="")
latitude = Column(Float, nullable=False)
longitude = Column(Float, nullable=False)
altitude = Column(Float, nullable=True)
revision = Column(Integer, nullable=False, default=1)
properties = Column(JSON, nullable=False, default=dict)
is_deleted = Column(Boolean, nullable=False, default=False, index=True)
created_at = Column(DateTime(timezone=True), server_default=func.now(), nullable=False)
updated_at = Column(DateTime(timezone=True), server_default=func.now(), onupdate=func.now())
deleted_at = Column(DateTime(timezone=True), nullable=True, index=True)
__table_args__ = (
Index("idx_earth_interactables_layer_deleted", "layer", "is_deleted"),
Index("idx_earth_interactables_updated_at", "updated_at"),
)

View File

@@ -1,6 +1,6 @@
"""Collection Task model"""
"""Datasource job model."""
from sqlalchemy import BigInteger, Column, DateTime, Integer, String, Text, Float
from sqlalchemy import BigInteger, Column, DateTime, Float, Integer, JSON, String, Text
from sqlalchemy.sql import func
from app.db.session import Base
@@ -11,7 +11,9 @@ class CollectionTask(Base):
id = Column(Integer, primary_key=True, autoincrement=True)
datasource_id = Column(Integer, nullable=False, index=True)
status = Column(String(20), nullable=False) # pending, running, success, failed, cancelled
source = Column(String(100), nullable=True, index=True)
task_type = Column(String(30), nullable=False, default="collect", index=True)
status = Column(String(20), nullable=False) # queued, running, cancelling, success, failed, cancelled
phase = Column(String(30), default="queued")
phase_progress = Column(Float)
phase_message = Column(String(255))
@@ -24,6 +26,13 @@ class CollectionTask(Base):
total_records = Column(Integer, default=0) # Total records to process
progress = Column(Float, default=0.0) # Progress percentage (0-100)
error_message = Column(Text)
payload = Column(JSON, default=dict)
rollback_policy = Column(String(40), nullable=False, default="keep_committed_batches")
dedupe_key = Column(String(180), nullable=True, index=True)
worker_id = Column(String(120), nullable=True, index=True)
locked_at = Column(DateTime(timezone=True), nullable=True, index=True)
requested_cancel_at = Column(DateTime(timezone=True), nullable=True)
cancel_reason = Column(Text)
created_at = Column(DateTime(timezone=True), server_default=func.now())
def __repr__(self):

View File

@@ -2,18 +2,24 @@ from __future__ import annotations
import asyncio
import json
from time import perf_counter
import httpx
from fastapi import Depends, HTTPException, status
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.config import settings
from app.core.logging import get_logger
from app.db.session import get_db
from app.schemas.ai import (
AIProviderStatusResponse,
SituationalAnalysisRequest,
SituationalAnalysisResponse,
)
from app.services.business_logs import emit_business_log, exception_context
logger = get_logger(__name__, service="ai")
class AIProviderClient:
@@ -64,7 +70,19 @@ class AIProviderClient:
return headers
async def get_status(self, request_id: str | None = None) -> AIProviderStatusResponse:
context = self._base_log_context(operation="status")
if not self.service_url:
await emit_business_log(
logger,
event="ai.provider.status.failed",
message="AI provider status skipped because service URL is not configured",
category="ai",
level="warning",
service="ai",
module=__name__,
request_id=request_id,
context={**context, "status": "unconfigured"},
)
return AIProviderStatusResponse(
provider="unconfigured",
enabled=False,
@@ -73,27 +91,133 @@ class AIProviderClient:
base_url=None,
)
data = await self._request("GET", "/v1/provider/status", request_id=request_id)
return AIProviderStatusResponse.model_validate(data)
started_at = perf_counter()
await emit_business_log(
logger,
event="ai.provider.status.start",
message="AI provider status request started",
category="ai",
service="ai",
module=__name__,
request_id=request_id,
context=context,
)
try:
data = await self._request("GET", "/v1/provider/status", request_id=request_id, operation="status")
result = AIProviderStatusResponse.model_validate(data)
await emit_business_log(
logger,
event="ai.provider.status.success",
message="AI provider status request completed",
category="ai",
service="ai",
module=__name__,
request_id=request_id,
context={
**context,
"status": "success",
"duration_ms": self._duration_ms(started_at),
"result_provider": result.provider,
"result_model": result.model,
"configured": result.configured,
"enabled": result.enabled,
},
)
return result
except Exception as exc:
await emit_business_log(
logger,
event="ai.provider.status.failed",
message="AI provider status request failed",
category="ai",
level="error",
service="ai",
module=__name__,
request_id=request_id,
context=exception_context(exc, {**context, "status": "failed", "duration_ms": self._duration_ms(started_at)}),
)
raise
async def analyze(
self,
payload: SituationalAnalysisRequest,
request_id: str | None = None,
) -> SituationalAnalysisResponse:
context = self._base_log_context(
operation="analyze",
preferred_model=payload.preferred_model,
input_summary=self._summarize_analysis_payload(payload),
)
if not self.service_url:
await emit_business_log(
logger,
event="ai.provider.analyze.failed",
message="AI provider analyze skipped because service URL is not configured",
category="ai",
level="warning",
service="ai",
module=__name__,
request_id=request_id,
context={**context, "status": "unconfigured"},
)
raise HTTPException(
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
detail="AI provider service URL is not configured.",
)
data = await self._request(
"POST",
"/v1/analyze",
json=payload.model_dump(),
started_at = perf_counter()
await emit_business_log(
logger,
event="ai.provider.analyze.start",
message="AI provider analyze request started",
category="ai",
service="ai",
module=__name__,
request_id=request_id,
context=context,
)
return SituationalAnalysisResponse.model_validate(data)
try:
data = await self._request(
"POST",
"/v1/analyze",
json=payload.model_dump(),
request_id=request_id,
operation="analyze",
payload_summary=context["input_summary"],
)
result = SituationalAnalysisResponse.model_validate(data)
await emit_business_log(
logger,
event="ai.provider.analyze.success",
message="AI provider analyze request completed",
category="ai",
service="ai",
module=__name__,
request_id=request_id,
context={
**context,
"status": "success",
"duration_ms": self._duration_ms(started_at),
"result_provider": result.provider,
"result_model": result.model,
"content_block_count": len(result.content_blocks or []),
"thinking_block_count": len(result.thinking_blocks or []),
},
)
return result
except Exception as exc:
await emit_business_log(
logger,
event="ai.provider.analyze.failed",
message="AI provider analyze request failed",
category="ai",
level="error",
service="ai",
module=__name__,
request_id=request_id,
context=exception_context(exc, {**context, "status": "failed", "duration_ms": self._duration_ms(started_at)}),
)
raise
async def _request(
self,
@@ -101,9 +225,12 @@ class AIProviderClient:
path: str,
json: dict | None = None,
request_id: str | None = None,
operation: str = "request",
payload_summary: dict | None = None,
) -> dict:
last_error: Exception | None = None
for attempt in range(1, self.retry_attempts + 1):
attempt_started_at = perf_counter()
try:
async with httpx.AsyncClient(timeout=self.timeout) as client:
response = await client.request(
@@ -117,6 +244,15 @@ class AIProviderClient:
except httpx.HTTPStatusError as exc:
last_error = exc
if attempt < self.retry_attempts and exc.response.status_code >= 500:
await self._log_retry(
operation=operation,
request_id=request_id,
attempt=attempt,
status_code=exc.response.status_code,
duration_ms=self._duration_ms(attempt_started_at),
error=exc,
payload_summary=payload_summary,
)
await asyncio.sleep(0.3 * attempt)
continue
detail = exc.response.text or "AI provider service returned an error"
@@ -127,6 +263,14 @@ class AIProviderClient:
except httpx.HTTPError as exc:
last_error = exc
if attempt < self.retry_attempts:
await self._log_retry(
operation=operation,
request_id=request_id,
attempt=attempt,
duration_ms=self._duration_ms(attempt_started_at),
error=exc,
payload_summary=payload_summary,
)
await asyncio.sleep(0.3 * attempt)
continue
raise HTTPException(
@@ -139,6 +283,71 @@ class AIProviderClient:
detail=f"AI provider service request failed: {last_error}",
)
def _base_log_context(self, **extra: object) -> dict:
llm_provider_apis = self.llm_config.get("model_provider_apis")
return {
"provider": self.llm_config.get("provider") or "",
"provider_api": self.llm_config.get("provider_api") or "",
"model": self.llm_config.get("model") or "",
"base_url_configured": bool(self.llm_config.get("base_url")),
"service_url_configured": bool(self.service_url),
"timeout_seconds": self.timeout,
"retry_attempts": self.retry_attempts,
"model_provider_api_count": len(llm_provider_apis or {}) if isinstance(llm_provider_apis, dict) else 0,
**extra,
}
@staticmethod
def _duration_ms(started_at: float) -> int:
return int((perf_counter() - started_at) * 1000)
@staticmethod
def _summarize_analysis_payload(payload: SituationalAnalysisRequest) -> dict:
context = payload.context if isinstance(payload.context, dict) else {}
thinking = payload.thinking if isinstance(payload.thinking, dict) else payload.thinking
return {
"title_length": len(payload.title or ""),
"objective_length": len(payload.objective or ""),
"observation_count": len(payload.observations or []),
"constraint_count": len(payload.constraints or []),
"has_system_prompt": bool(payload.system_prompt),
"thinking_enabled": bool(thinking),
"context_keys": sorted(str(key) for key in context.keys()),
}
async def _log_retry(
self,
*,
operation: str,
request_id: str | None,
attempt: int,
duration_ms: int,
error: BaseException,
status_code: int | None = None,
payload_summary: dict | None = None,
) -> None:
await emit_business_log(
logger,
event=f"ai.provider.{operation}.retry",
message="AI provider request will retry",
category="ai",
level="warning",
service="ai",
module=__name__,
request_id=request_id,
context=exception_context(
error,
{
**self._base_log_context(operation=operation),
"attempt": attempt,
"next_attempt": attempt + 1,
"status_code": status_code,
"duration_ms": duration_ms,
"input_summary": payload_summary,
},
),
)
async def get_ai_provider_client(db: AsyncSession = Depends(get_db)) -> AIProviderClient:
from app.api.v1.settings import get_runtime_ai_provider_config

View File

@@ -1,11 +1,18 @@
from __future__ import annotations
import hashlib
from time import perf_counter
from urllib.parse import urlparse
import httpx
from bs4 import BeautifulSoup
from app.core.logging import get_logger
from app.services.ai_tools.schemas import FetchedEvidence
from app.services.business_logs import emit_business_log, exception_context
logger = get_logger(__name__, service="ai_tool")
class WebFetchError(RuntimeError):
@@ -29,8 +36,33 @@ async def fetch_url_evidence(
timeout_seconds: int = 20,
max_bytes: int = 1_500_000,
) -> FetchedEvidence:
started_at = perf_counter()
if not url:
await emit_business_log(
logger,
event="ai_tool.web_fetch.failed",
message="WebFetch failed because URL is empty",
category="ai_tool",
level="warning",
service="ai_tool",
module=__name__,
context={"reason": "empty_url"},
)
raise WebFetchError("url is required")
request_host = urlparse(url).netloc
await emit_business_log(
logger,
event="ai_tool.web_fetch.start",
message="WebFetch request started",
category="ai_tool",
service="ai_tool",
module=__name__,
context={
"url_host": request_host,
"timeout_seconds": timeout_seconds,
"max_bytes": max_bytes,
},
)
try:
async with httpx.AsyncClient(
timeout=timeout_seconds,
@@ -41,10 +73,45 @@ async def fetch_url_evidence(
response.raise_for_status()
content = response.content[:max_bytes]
except httpx.HTTPError as exc:
await emit_business_log(
logger,
event="ai_tool.web_fetch.failed",
message="WebFetch request failed",
category="ai_tool",
level="error",
service="ai_tool",
module=__name__,
context=exception_context(
exc,
{
"url_host": request_host,
"status": "failed",
"duration_ms": int((perf_counter() - started_at) * 1000),
},
),
)
raise WebFetchError(f"failed to fetch page: {exc}") from exc
title, text = _extract_title_and_text(content.decode(response.encoding or "utf-8", errors="ignore"))
content_hash = hashlib.sha256(text.encode("utf-8")).hexdigest()
await emit_business_log(
logger,
event="ai_tool.web_fetch.success",
message="WebFetch request completed",
category="ai_tool",
service="ai_tool",
module=__name__,
context={
"url_host": request_host,
"final_url_host": urlparse(str(response.url)).netloc,
"status": "success",
"status_code": response.status_code,
"bytes_read": len(content),
"content_hash": content_hash,
"duration_ms": int((perf_counter() - started_at) * 1000),
"extractor": "beautifulsoup_basic",
},
)
return FetchedEvidence(
url=url,
final_url=str(response.url),
@@ -53,4 +120,3 @@ async def fetch_url_evidence(
content_hash=content_hash,
extractor="beautifulsoup_basic",
)

View File

@@ -1,11 +1,18 @@
from __future__ import annotations
from copy import deepcopy
import hashlib
from time import perf_counter
from typing import Any
import httpx
from app.core.logging import get_logger
from app.services.ai_tools.schemas import SearchEvidence, WebSearchConfig, WebSearchProviderConfig
from app.services.business_logs import emit_business_log, exception_context
logger = get_logger(__name__, service="ai_tool")
WEB_SEARCH_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
@@ -120,29 +127,108 @@ class WebSearchClient:
domains: list[str] | None = None,
freshness_days: int | None = None,
) -> list[SearchEvidence]:
started_at = perf_counter()
if not self.config.enabled:
await emit_business_log(
logger,
event="ai_tool.web_search.unavailable",
message="WebSearch skipped because integration is disabled",
category="ai_tool",
level="warning",
service="ai_tool",
module=__name__,
context={"provider": self.config.default_provider, "reason": "disabled"},
)
raise WebSearchConfigurationError("WebSearch is disabled.")
provider_config = self.config.active_provider_config
provider = normalize_web_search_provider(provider_config.provider)
if provider != "searxng" and not provider_config.api_key:
await emit_business_log(
logger,
event="ai_tool.web_search.unavailable",
message="WebSearch skipped because API key is not configured",
category="ai_tool",
level="warning",
service="ai_tool",
module=__name__,
context={"provider": provider, "reason": "missing_api_key"},
)
raise WebSearchConfigurationError(f"{provider} API key is not configured.")
query = " ".join(str(query or "").split())
if not query:
await emit_business_log(
logger,
event="ai_tool.web_search.failed",
message="WebSearch failed because query is empty",
category="ai_tool",
level="warning",
service="ai_tool",
module=__name__,
context={"provider": provider, "reason": "empty_query"},
)
raise WebSearchConfigurationError("search query is required.")
limit = max_results or provider_config.max_results
if provider == "tavily":
return await self._search_tavily(provider_config, query, limit, domains, freshness_days)
if provider == "brave":
return await self._search_brave(provider_config, query, limit, domains)
if provider == "serpapi":
return await self._search_serpapi(provider_config, query, limit)
if provider == "exa":
return await self._search_exa(provider_config, query, limit, domains)
if provider == "firecrawl":
return await self._search_firecrawl(provider_config, query, limit)
if provider == "searxng":
return await self._search_searxng(provider_config, query, limit, domains)
raise WebSearchConfigurationError(f"Unsupported web search provider: {provider}")
context = {
"provider": provider,
"query_hash": hashlib.sha256(query.encode("utf-8")).hexdigest(),
"query_length": len(query),
"max_results": limit,
"domain_count": len(domains or []),
"freshness_days": freshness_days,
}
await emit_business_log(
logger,
event="ai_tool.web_search.start",
message="WebSearch request started",
category="ai_tool",
service="ai_tool",
module=__name__,
context=context,
)
try:
if provider == "tavily":
results = await self._search_tavily(provider_config, query, limit, domains, freshness_days)
elif provider == "brave":
results = await self._search_brave(provider_config, query, limit, domains)
elif provider == "serpapi":
results = await self._search_serpapi(provider_config, query, limit)
elif provider == "exa":
results = await self._search_exa(provider_config, query, limit, domains)
elif provider == "firecrawl":
results = await self._search_firecrawl(provider_config, query, limit)
elif provider == "searxng":
results = await self._search_searxng(provider_config, query, limit, domains)
else:
raise WebSearchConfigurationError(f"Unsupported web search provider: {provider}")
event = "ai_tool.web_search.success" if results else "ai_tool.web_search.empty"
await emit_business_log(
logger,
event=event,
message="WebSearch request completed" if results else "WebSearch returned no results",
category="ai_tool",
level="info" if results else "warning",
service="ai_tool",
module=__name__,
context={
**context,
"status": "success" if results else "empty",
"result_count": len(results),
"duration_ms": int((perf_counter() - started_at) * 1000),
},
)
return results
except Exception as exc:
await emit_business_log(
logger,
event="ai_tool.web_search.failed",
message="WebSearch request failed",
category="ai_tool",
level="error",
service="ai_tool",
module=__name__,
context=exception_context(exc, {**context, "status": "failed", "duration_ms": int((perf_counter() - started_at) * 1000)}),
)
raise
async def test_connection(self) -> list[SearchEvidence]:
return await self.search("Planet WebSearch connectivity test", max_results=1)
@@ -388,4 +474,3 @@ def _float_or_none(value: Any) -> float | None:
return float(value)
except (TypeError, ValueError):
return None

View File

@@ -0,0 +1,117 @@
from __future__ import annotations
import asyncio
from collections.abc import Mapping
from typing import Any
from app.core.logging import PlanetLoggerAdapter, sanitize_log_value
from app.core.request_context import get_request_id
from app.services.persistent_logs import record_system_log
LEVEL_METHODS = {
"debug": "debug_event",
"info": "info_event",
"warning": "warning_event",
"error": "error_event",
}
def normalize_business_level(level: str | None) -> str:
normalized = str(level or "info").strip().lower()
if normalized in {"warn", "warning"}:
return "warning"
if normalized in {"err", "error", "critical", "fatal"}:
return "error"
if normalized == "debug":
return "debug"
return "info"
def build_business_context(
context: Mapping[str, Any] | None = None,
**fields: Any,
) -> dict[str, Any]:
payload = dict(context or {})
for key, value in fields.items():
if value is not None:
payload[key] = value
return sanitize_log_value(payload)
async def emit_business_log(
logger: PlanetLoggerAdapter,
*,
event: str,
message: str,
category: str,
level: str = "info",
source: str = "backend",
service: str | None = None,
module: str | None = None,
request_id: str | None = None,
user_id: int | None = None,
context: Mapping[str, Any] | None = None,
) -> None:
normalized_level = normalize_business_level(level)
safe_context = build_business_context(context)
log_method = getattr(logger, LEVEL_METHODS[normalized_level])
log_method(message, event=event, context=safe_context)
await record_system_log(
source=source,
level=normalized_level,
message=message,
service=service,
module=module,
event=event,
request_id=request_id or get_request_id(),
user_id=user_id,
category=category,
context=safe_context,
)
def emit_business_log_background(
logger: PlanetLoggerAdapter,
*,
event: str,
message: str,
category: str,
level: str = "info",
source: str = "backend",
service: str | None = None,
module: str | None = None,
request_id: str | None = None,
user_id: int | None = None,
context: Mapping[str, Any] | None = None,
) -> None:
normalized_level = normalize_business_level(level)
safe_context = build_business_context(context)
log_method = getattr(logger, LEVEL_METHODS[normalized_level])
log_method(message, event=event, context=safe_context)
try:
loop = asyncio.get_running_loop()
except RuntimeError:
return
loop.create_task(
record_system_log(
source=source,
level=normalized_level,
message=message,
service=service,
module=module,
event=event,
request_id=request_id or get_request_id(),
user_id=user_id,
category=category,
context=safe_context,
)
)
def exception_context(exc: BaseException, context: Mapping[str, Any] | None = None) -> dict[str, Any]:
return build_business_context(
context,
error_type=type(exc).__name__,
error=str(exc),
)

View File

@@ -4,42 +4,22 @@ import asyncio
from abc import ABC, abstractmethod
from typing import Dict, List, Any, Optional
from datetime import UTC, datetime
from time import perf_counter
from urllib.parse import urlparse
import httpx
from sqlalchemy import select, text
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.collected_data_fields import build_dynamic_metadata, get_record_field
from app.core.countries import normalize_country
from app.core.logging import get_logger
from app.core.time import to_iso8601_utc
from app.core.websocket.broadcaster import broadcaster
from app.services.earth_layer_cache import invalidate_earth_layer_cache_for_source
EARTH_UPDATE_LAYER_HINTS: dict[str, list[str]] = {
"ris_live_bgp": ["bgp"],
"bgpstream_bgp": ["bgp"],
"top500_supercomputers": ["computeCenters"],
"epoch_ai_gpu": ["computeCenters"],
"huggingface_models": ["computeCenters"],
"huggingface_datasets": ["computeCenters"],
"huggingface_spaces": ["computeCenters"],
"telegeography_cables": ["cables"],
"telegeography_landing_points": ["cables"],
"telegeography_cable_systems": ["cables"],
"arcgis_cables": ["cables"],
"fao_landing_points": ["cables"],
"arcgis_landing_points": ["cables"],
"arcgis_cable_landing_relations": ["cables"],
"spacetrack_tle": ["satellites"],
"celestrak_tle": ["satellites"],
"barentswatch_vessels": ["vessels"],
"aisstream_vessels": ["vessels"],
"news_live_streams": ["media"],
"media_news_archive": ["news"],
}
from app.services.business_logs import emit_business_log, exception_context
from app.services.earth_layer_adapters import get_earth_update_layers_for_source
def get_earth_update_layers_for_source(source: str) -> list[str]:
return EARTH_UPDATE_LAYER_HINTS.get(source, [])
logger = get_logger(__name__, service="collector")
class BaseCollector(ABC):
@@ -58,6 +38,7 @@ class BaseCollector(ABC):
self._datasource_id = 1
self._resolved_url: Optional[str] = None
self._last_broadcast_progress: Optional[int] = None
self._last_save_summary: dict[str, int] = {}
async def resolve_url(self, db: AsyncSession) -> None:
from app.core.data_sources import get_data_sources_config
@@ -96,29 +77,6 @@ class BaseCollector(ABC):
)
self._last_broadcast_progress = rounded_progress
async def _publish_earth_update(
self,
*,
action: str,
records_processed: int,
task_id: int | None = None,
) -> None:
layers = get_earth_update_layers_for_source(self.name)
if not layers:
return
await broadcaster.broadcast_earth_update(
{
"action": action,
"source": self.name,
"data_type": self.data_type,
"layers": layers,
"datasource_id": getattr(self, "_datasource_id", None),
"task_id": task_id,
"records_processed": records_processed,
"timestamp": to_iso8601_utc(datetime.now(UTC)),
}
)
async def update_progress(self, records_processed: int, *, commit: bool = False, force: bool = False):
"""Update task progress - call this during data processing"""
if self._current_task and self._db_session:
@@ -322,19 +280,39 @@ class BaseCollector(ABC):
from app.models.data_snapshot import DataSnapshot
start_time = datetime.now(UTC)
started_at = perf_counter()
datasource_id = getattr(self, "_datasource_id", 1)
snapshot_id: Optional[int] = None
if not collector_registry.is_active(self.name):
await self._log_collection_event(
"collector.run.skipped_disabled",
"Collector skipped because it is disabled",
level="info",
context={"status": "skipped", "reason": "disabled"},
)
return {"status": "skipped", "reason": "Collector is disabled"}
task = CollectionTask(
datasource_id=datasource_id,
status="running",
phase="queued",
started_at=start_time,
)
db.add(task)
task = self._current_task if isinstance(self._current_task, CollectionTask) else None
if task is None:
task = CollectionTask(
datasource_id=datasource_id,
source=self.name,
task_type="collect",
status="running",
phase="queued",
started_at=start_time,
)
db.add(task)
else:
task.datasource_id = datasource_id
task.source = task.source or self.name
task.task_type = task.task_type or "collect"
task.status = "running"
task.phase = "queued"
task.started_at = task.started_at or start_time
task.completed_at = None
task.error_message = None
await db.commit()
task_id = task.id
@@ -344,23 +322,76 @@ class BaseCollector(ABC):
await self.resolve_url(db)
await self._publish_task_update(force=True)
await self._log_collection_event(
"collector.run.started",
"Collector run started",
context={"status": "running", "task_id": task_id},
)
try:
phase_started_at = perf_counter()
await self.set_phase("fetching", message="正在拉取原始数据")
await self._log_collection_event(
"collector.phase.fetching.start",
"Collector fetch phase started",
context={"task_id": task_id, "snapshot_id": snapshot_id},
)
raw_data = await self.fetch()
task.total_records = len(raw_data)
await db.commit()
await self._publish_task_update(force=True)
await self._log_collection_event(
"collector.phase.fetching.success",
"Collector fetch phase completed",
context={
"task_id": task_id,
"raw_count": len(raw_data),
"duration_ms": self._duration_ms(phase_started_at),
},
)
if self.fail_on_empty and not raw_data:
raise RuntimeError(f"Collector {self.name} returned no data")
phase_started_at = perf_counter()
await self.set_phase("transforming", message="正在转换采集数据")
await self._log_collection_event(
"collector.phase.transforming.start",
"Collector transform phase started",
context={"task_id": task_id, "raw_count": len(raw_data)},
)
data = self.transform(raw_data)
await self._log_collection_event(
"collector.phase.transforming.success",
"Collector transform phase completed",
context={
"task_id": task_id,
"raw_count": len(raw_data),
"transformed_count": len(data),
"duration_ms": self._duration_ms(phase_started_at),
},
)
snapshot_id = await self._create_snapshot(db, task_id, data, start_time)
phase_started_at = perf_counter()
await self.set_phase("saving", message="正在保存采集数据")
await self._log_collection_event(
"collector.phase.saving.start",
"Collector save phase started",
context={"task_id": task_id, "snapshot_id": snapshot_id, "transformed_count": len(data)},
)
records_count = await self._save_data(db, data, task_id=task_id, snapshot_id=snapshot_id)
await self._log_collection_event(
"collector.phase.saving.success",
"Collector save phase completed",
context={
"task_id": task_id,
"snapshot_id": snapshot_id,
"saved_count": records_count,
**self._last_save_summary,
"duration_ms": self._duration_ms(phase_started_at),
},
)
task.status = "success"
task.phase = "completed"
@@ -374,10 +405,19 @@ class BaseCollector(ABC):
task.completed_at = datetime.now(UTC)
await db.commit()
await self._publish_task_update(force=True)
await self._publish_earth_update(
action="collector_completed",
records_processed=records_count,
task_id=task_id,
await self._log_collection_event(
"collector.run.completed",
"Collector run completed",
context={
"status": "success",
"task_id": task_id,
"snapshot_id": snapshot_id,
"raw_count": len(raw_data),
"transformed_count": len(data),
"saved_count": records_count,
**self._last_save_summary,
"duration_ms": self._duration_ms(started_at),
},
)
return {
@@ -402,6 +442,17 @@ class BaseCollector(ABC):
)
await db.commit()
await self._publish_task_update(force=True)
await self._log_collection_event(
"collector.run.cancelled",
"Collector run cancelled",
level="warning",
context={
"status": "cancelled",
"task_id": task_id,
"snapshot_id": snapshot_id,
"duration_ms": self._duration_ms(started_at),
},
)
raise
except Exception as e:
await db.rollback()
@@ -418,6 +469,20 @@ class BaseCollector(ABC):
snapshot.summary = {"error": str(e)}
await db.commit()
await self._publish_task_update(force=True)
await self._log_collection_event(
"collector.run.failed",
"Collector run failed",
level="error",
context=exception_context(
e,
{
"status": "failed",
"task_id": task_id,
"snapshot_id": snapshot_id,
"duration_ms": self._duration_ms(started_at),
},
),
)
return {
"status": "failed",
@@ -438,6 +503,7 @@ class BaseCollector(ABC):
from app.models.data_snapshot import DataSnapshot
if not data:
self._last_save_summary = {"created": 0, "updated": 0, "unchanged": 0, "deleted": 0}
if snapshot_id is not None:
snapshot = await db.get(DataSnapshot, snapshot_id)
if snapshot:
@@ -584,12 +650,51 @@ class BaseCollector(ABC):
"unchanged": unchanged_count,
"deleted": len(deleted_keys),
}
self._last_save_summary = {
"created": created_count,
"updated": updated_count,
"unchanged": unchanged_count,
"deleted": len(deleted_keys),
}
else:
self._last_save_summary = {
"created": created_count,
"updated": updated_count,
"unchanged": unchanged_count,
"deleted": 0,
}
await db.commit()
invalidate_earth_layer_cache_for_source(self.name)
await self.update_progress(len(data), force=True)
return records_added
@staticmethod
def _duration_ms(started_at: float) -> int:
return int((perf_counter() - started_at) * 1000)
async def _log_collection_event(
self,
event: str,
message: str,
*,
level: str = "info",
context: Dict[str, Any] | None = None,
) -> None:
await emit_business_log(
logger,
event=event,
message=message,
category="collector",
level=level,
service="collector",
module=__name__,
context={
"collector_name": self.name,
"datasource_id": getattr(self, "_datasource_id", None),
**(context or {}),
},
)
async def save(self, db: AsyncSession, data: List[Dict[str, Any]]) -> int:
"""Save data to database (legacy method, use _save_data instead)"""
return await self._save_data(db, data)
@@ -602,10 +707,65 @@ class HTTPCollector(BaseCollector):
headers: Dict[str, str] = {}
async def fetch(self) -> List[Dict[str, Any]]:
started_at = perf_counter()
request_host = urlparse(self.base_url).netloc
await emit_business_log(
logger,
event="collector.http.fetch.start",
message="Collector HTTP request started",
category="collector",
service="collector",
module=__name__,
context={
"collector_name": self.name,
"datasource_id": getattr(self, "_datasource_id", None),
"url_host": request_host,
},
)
async with httpx.AsyncClient(timeout=60.0) as client:
response = await client.get(self.base_url, headers=self.headers)
response.raise_for_status()
return self.parse_response(response.json())
try:
response = await client.get(self.base_url, headers=self.headers)
response.raise_for_status()
payload = response.json()
parsed = self.parse_response(payload)
await emit_business_log(
logger,
event="collector.http.fetch.success",
message="Collector HTTP request completed",
category="collector",
service="collector",
module=__name__,
context={
"collector_name": self.name,
"datasource_id": getattr(self, "_datasource_id", None),
"url_host": request_host,
"status_code": response.status_code,
"response_bytes": len(response.content or b""),
"parsed_count": len(parsed),
"duration_ms": BaseCollector._duration_ms(started_at),
},
)
return parsed
except Exception as exc:
await emit_business_log(
logger,
event="collector.http.fetch.failed",
message="Collector HTTP request failed",
category="collector",
level="error",
service="collector",
module=__name__,
context=exception_context(
exc,
{
"collector_name": self.name,
"datasource_id": getattr(self, "_datasource_id", None),
"url_host": request_host,
"duration_ms": BaseCollector._duration_ms(started_at),
},
),
)
raise
@abstractmethod
def parse_response(self, response: Dict[str, Any]) -> List[Dict[str, Any]]:

View File

@@ -1,15 +1,39 @@
"""CelesTrak TLE Collector
"""CelesTrak TLE Collector.
Collects satellite TLE (Two-Line Element) data from CelesTrak.org.
Free, no authentication required.
Collects the full active satellite GP element set from CelesTrak.
"""
import asyncio
import json
from typing import Dict, Any, List
from pathlib import Path
from time import perf_counter
from typing import Any, Dict, List
from urllib.parse import urlencode, urlparse
import httpx
from app.core.logging import get_logger
from app.core.satellite_tle import build_tle_lines_from_elements
from app.services.business_logs import emit_business_log, exception_context
from app.services.collectors.base import BaseCollector
from app.services.collectors.downloads import DownloadHTTPStatusError, ResumableFileDownloader
logger = get_logger(__name__, service="collector")
ACTIVE_GROUP = "active"
FALLBACK_GROUPS = (
"starlink",
"gps-ops",
"galileo",
"glonass",
"beidou",
"leo",
"geo",
"iridium-next",
)
FETCH_RETRY_ATTEMPTS = 3
FETCH_RETRY_BASE_DELAY_SECONDS = 0.8
CELESTRAK_NOT_UPDATED_MARKER = "GP data has not updated since your last successful"
class CelesTrakTLECollector(BaseCollector):
@@ -18,55 +42,359 @@ class CelesTrakTLECollector(BaseCollector):
module = "L3"
frequency_hours = 24
data_type = "satellite_tle"
_downloader = ResumableFileDownloader(
cache_namespace="celestrak",
default_accept="application/json",
)
@property
def base_url(self) -> str:
return self._resolved_url or ""
def _active_url(self) -> str:
return self._group_url(ACTIVE_GROUP)
def _group_url(self, group: str) -> str:
if not self.base_url:
raise RuntimeError("CelesTrak base URL is not configured")
return f"{self.base_url}?{urlencode({'GROUP': group, 'FORMAT': 'json'})}"
async def fetch(self) -> List[Dict[str, Any]]:
satellite_groups = [
"starlink",
"gps-ops",
"galileo",
"glonass",
"beidou",
"leo",
"geo",
"iridium-next",
]
url = self._active_url()
last_error: Exception | None = None
all_satellites = []
async with httpx.AsyncClient(timeout=120.0) as client:
for group in satellite_groups:
async with httpx.AsyncClient(timeout=180.0, follow_redirects=True) as client:
for attempt in range(1, FETCH_RETRY_ATTEMPTS + 1):
started_at = perf_counter()
try:
url = f"{self.base_url}?GROUP={group}&FORMAT=json"
response = await client.get(url)
await emit_business_log(
logger,
event="collector.celestrak.download.start",
message="CelesTrak active satellite download started",
category="collector",
service="collector",
module=__name__,
context={
"collector_name": self.name,
"datasource_id": getattr(self, "_datasource_id", None),
"group": ACTIVE_GROUP,
"attempt": attempt,
"url_host": urlparse(url).netloc,
},
)
body_path = await self._downloader.download_file(
client,
url,
extension=".json",
accept="application/json",
progress_callback=self._report_download_progress,
validate_existing=self._validate_json_file,
)
data = await self._load_downloaded_payload(body_path, url)
await emit_business_log(
logger,
event="collector.celestrak.download.success",
message="CelesTrak active satellite download completed",
category="collector",
service="collector",
module=__name__,
context={
"collector_name": self.name,
"datasource_id": getattr(self, "_datasource_id", None),
"group": ACTIVE_GROUP,
"attempt": attempt,
"record_count": len(data),
"duration_ms": self._duration_ms(started_at),
},
)
return data
except DownloadHTTPStatusError as exc:
if self._is_not_updated_response(exc):
cached_path = self._downloader.get_cached_file(
url,
".json",
validate_existing=self._validate_json_file,
)
if cached_path is not None:
data = await self._load_downloaded_payload(cached_path, url)
await emit_business_log(
logger,
event="collector.celestrak.download.cached_not_updated",
message="CelesTrak active satellite data has not changed; using cached download",
category="collector",
level="warning",
service="collector",
module=__name__,
context={
"collector_name": self.name,
"datasource_id": getattr(self, "_datasource_id", None),
"group": ACTIVE_GROUP,
"attempt": attempt,
"record_count": len(data),
"duration_ms": self._duration_ms(started_at),
},
)
return data
await emit_business_log(
logger,
event="collector.celestrak.download.not_updated_no_cache",
message="CelesTrak active satellite data has not changed; trying fallback groups",
category="collector",
level="warning",
service="collector",
module=__name__,
context=exception_context(
exc,
{
"collector_name": self.name,
"datasource_id": getattr(self, "_datasource_id", None),
"group": ACTIVE_GROUP,
"attempt": attempt,
"duration_ms": self._duration_ms(started_at),
},
),
)
return await self._fetch_fallback_groups(client, active_error=exc)
raise
except Exception as exc:
last_error = exc
is_final_attempt = attempt >= FETCH_RETRY_ATTEMPTS
await emit_business_log(
logger,
event=(
"collector.celestrak.download.failed"
if is_final_attempt
else "collector.celestrak.download.retry"
),
message=(
"CelesTrak active satellite download failed"
if is_final_attempt
else "CelesTrak active satellite download will retry"
),
category="collector",
level="error" if is_final_attempt else "warning",
service="collector",
module=__name__,
context=exception_context(
exc,
{
"collector_name": self.name,
"datasource_id": getattr(self, "_datasource_id", None),
"group": ACTIVE_GROUP,
"attempt": attempt,
"duration_ms": self._duration_ms(started_at),
},
),
)
if not is_final_attempt:
await asyncio.sleep(FETCH_RETRY_BASE_DELAY_SECONDS * attempt)
if response.status_code == 200:
data = response.json()
if isinstance(data, list):
for item in data:
if isinstance(item, dict):
item["_celestrak_group"] = group
all_satellites.extend(data)
print(f"CelesTrak: Fetched {len(data)} satellites from group '{group}'")
except Exception as e:
print(f"CelesTrak: Error fetching group '{group}': {e}")
raise RuntimeError(f"CelesTrak active satellite download failed after retries: {last_error}")
if not all_satellites:
return self._get_sample_data()
async def _fetch_fallback_groups(
self,
client: httpx.AsyncClient,
*,
active_error: DownloadHTTPStatusError,
) -> List[Dict[str, Any]]:
started_at = perf_counter()
records_by_norad: dict[str, Dict[str, Any]] = {}
group_counts: dict[str, int] = {}
print(f"CelesTrak: Total satellites fetched: {len(all_satellites)}")
await emit_business_log(
logger,
event="collector.celestrak.fallback_groups.start",
message="CelesTrak fallback group download started",
category="collector",
level="warning",
service="collector",
module=__name__,
context={
"collector_name": self.name,
"datasource_id": getattr(self, "_datasource_id", None),
"groups": list(FALLBACK_GROUPS),
"reason": "active_not_updated_without_cache",
},
)
# Return raw data - base.run() will call transform()
return all_satellites
try:
for group in FALLBACK_GROUPS:
group_url = self._group_url(group)
try:
body_path = await self._downloader.download_file(
client,
group_url,
extension=".json",
accept="application/json",
validate_existing=self._validate_json_file,
)
except DownloadHTTPStatusError as exc:
if not self._is_not_updated_response(exc):
raise RuntimeError(f"CelesTrak fallback group '{group}' download failed: {exc}") from exc
cached_path = self._downloader.get_cached_file(
group_url,
".json",
validate_existing=self._validate_json_file,
)
if cached_path is None:
raise RuntimeError(
f"CelesTrak fallback group '{group}' has not updated and no local cached copy is available"
) from exc
body_path = cached_path
group_records = await self._load_downloaded_payload(
body_path,
group_url,
query_group=group,
constellation_group=group,
)
group_counts[group] = len(group_records)
for item in group_records:
norad_cat_id = item.get("NORAD_CAT_ID")
if norad_cat_id is None:
continue
records_by_norad.setdefault(str(norad_cat_id), item)
except Exception as exc:
await emit_business_log(
logger,
event="collector.celestrak.fallback_groups.failed",
message="CelesTrak fallback group download failed",
category="collector",
level="error",
service="collector",
module=__name__,
context=exception_context(
exc,
{
"collector_name": self.name,
"datasource_id": getattr(self, "_datasource_id", None),
"groups": list(FALLBACK_GROUPS),
"completed_groups": list(group_counts),
"duration_ms": self._duration_ms(started_at),
},
),
)
raise RuntimeError(
"CelesTrak active data has not updated since this network's last successful download, "
"no active cache is available, and fallback group mode failed. Wait until CelesTrak "
"publishes the next GP update, restore the Planet download cache, or use Space-Track."
) from active_error
records = list(records_by_norad.values())
if not records:
raise RuntimeError("CelesTrak fallback group mode produced no satellite records")
await emit_business_log(
logger,
event="collector.celestrak.fallback_groups.success",
message="CelesTrak fallback group download completed",
category="collector",
level="warning",
service="collector",
module=__name__,
context={
"collector_name": self.name,
"datasource_id": getattr(self, "_datasource_id", None),
"groups": list(FALLBACK_GROUPS),
"group_counts": group_counts,
"record_count": len(records),
"duration_ms": self._duration_ms(started_at),
},
)
return records
async def _load_downloaded_payload(
self,
body_path: Path,
url: str,
*,
query_group: str = ACTIVE_GROUP,
constellation_group: str | None = None,
) -> List[Dict[str, Any]]:
try:
data = self._load_active_payload(body_path)
except RuntimeError as exc:
await self._log_parse_failure(exc)
raise
for item in data:
item["_celestrak_query_group"] = query_group
item["_celestrak_source_url"] = url
if constellation_group:
item["_celestrak_group"] = constellation_group
return data
@staticmethod
def _is_not_updated_response(exc: DownloadHTTPStatusError) -> bool:
return exc.status_code == 403 and CELESTRAK_NOT_UPDATED_MARKER in exc.body
@staticmethod
def _duration_ms(started_at: float) -> int:
return int((perf_counter() - started_at) * 1000)
async def _report_download_progress(self, downloaded: int, total: int | None) -> None:
if total and total > 0:
await self.update_phase_progress(
current=min(downloaded, total),
total=total,
unit="bytes",
message=f"正在下载 CelesTrak active 卫星数据 {downloaded}/{total} bytes",
commit=True,
)
@staticmethod
def _validate_json_file(path: Path) -> bool:
try:
data = json.loads(path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError, UnicodeDecodeError):
return False
return isinstance(data, list)
async def _log_parse_failure(self, exc: Exception) -> None:
await emit_business_log(
logger,
event="collector.celestrak.parse.failed",
message="CelesTrak active satellite JSON parsing failed",
category="collector",
level="error",
service="collector",
module=__name__,
context=exception_context(
exc,
{
"collector_name": self.name,
"datasource_id": getattr(self, "_datasource_id", None),
"group": ACTIVE_GROUP,
},
),
)
def _load_active_payload(self, path: Path) -> List[Dict[str, Any]]:
try:
raw = json.loads(path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError, UnicodeDecodeError) as exc:
raise RuntimeError(f"CelesTrak active payload is not valid JSON: {exc}") from exc
if not isinstance(raw, list):
raise RuntimeError("CelesTrak active payload is not a JSON array")
records: List[Dict[str, Any]] = []
invalid_count = 0
for item in raw:
if isinstance(item, dict) and item.get("NORAD_CAT_ID") is not None:
records.append(item)
else:
invalid_count += 1
if invalid_count:
raise RuntimeError(f"CelesTrak active payload contains {invalid_count} invalid record(s)")
if not records:
raise RuntimeError("CelesTrak active payload contains no satellite records")
return records
def transform(self, raw_data: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
transformed = []
for item in raw_data:
norad_cat_id = item.get("NORAD_CAT_ID")
tle_line1, tle_line2 = build_tle_lines_from_elements(
norad_cat_id=item.get("NORAD_CAT_ID"),
norad_cat_id=norad_cat_id,
epoch=item.get("EPOCH"),
inclination=item.get("INCLINATION"),
raan=item.get("RA_OF_ASC_NODE"),
@@ -75,14 +403,18 @@ class CelesTrakTLECollector(BaseCollector):
mean_anomaly=item.get("MEAN_ANOMALY"),
mean_motion=item.get("MEAN_MOTION"),
)
constellation_group = self._infer_constellation_group(item)
transformed.append(
{
"source_id": str(norad_cat_id),
"name": item.get("OBJECT_NAME", "Unknown"),
"reference_date": item.get("EPOCH", ""),
"metadata": {
"constellation_group": item.get("_celestrak_group"),
"norad_cat_id": item.get("NORAD_CAT_ID"),
"constellation_group": constellation_group,
"celestrak_query_group": item.get("_celestrak_query_group") or ACTIVE_GROUP,
"celestrak_source_url": item.get("_celestrak_source_url"),
"norad_cat_id": norad_cat_id,
"international_designator": item.get("OBJECT_ID"),
"epoch": item.get("EPOCH"),
"mean_motion": item.get("MEAN_MOTION"),
@@ -105,6 +437,19 @@ class CelesTrakTLECollector(BaseCollector):
)
return transformed
@staticmethod
def _infer_constellation_group(item: Dict[str, Any]) -> str | None:
explicit_group = str(item.get("_celestrak_group") or "").strip().lower()
if explicit_group and explicit_group != ACTIVE_GROUP:
return explicit_group
name = str(item.get("OBJECT_NAME") or "").strip().upper()
if name.startswith("STARLINK"):
return "starlink"
if name.startswith("IRIDIUM"):
return "iridium-next"
return None
def _get_sample_data(self) -> List[Dict[str, Any]]:
return [
{

View File

@@ -4,7 +4,7 @@ from __future__ import annotations
import hashlib
import json
import tempfile
import os
import time
from datetime import UTC, datetime
from pathlib import Path
@@ -17,6 +17,31 @@ ProgressCallback = Callable[[int, int | None], Awaitable[None]]
ValidateCallback = Callable[[Path], bool]
class DownloadHTTPStatusError(RuntimeError):
"""HTTP status error that keeps the upstream response body for caller-specific handling."""
def __init__(self, *, url: str, status_code: int, body: str) -> None:
self.url = url
self.status_code = status_code
self.body = body
preview = body.strip().replace("\r", " ").replace("\n", " ")[:240]
suffix = f": {preview}" if preview else ""
super().__init__(f"HTTP {status_code} while downloading {url}{suffix}")
def default_download_cache_root() -> Path:
configured = os.getenv("PLANET_DOWNLOAD_CACHE_DIR")
if configured:
return Path(configured).expanduser()
planet_cache = os.getenv("PLANET_CACHE_DIR")
if planet_cache:
return Path(planet_cache).expanduser() / "downloads"
xdg_cache = os.getenv("XDG_CACHE_HOME")
if xdg_cache:
return Path(xdg_cache).expanduser() / "planet" / "downloads"
return Path.home() / ".cache" / "planet" / "downloads"
class ResumableFileDownloader:
"""Download files with cache validators and byte-range resume support."""
@@ -26,8 +51,9 @@ class ResumableFileDownloader:
cache_namespace: str,
user_agent: str = "Planet-Intelligence-System/1.0 (Python/collector)",
default_accept: str = "*/*",
cache_root: Path | None = None,
) -> None:
self._cache_dir = Path(tempfile.gettempdir()) / "planet-download-cache" / cache_namespace
self._cache_dir = (cache_root or default_download_cache_root()) / cache_namespace
self._user_agent = user_agent
self._default_accept = default_accept
@@ -43,6 +69,25 @@ class ResumableFileDownloader:
meta_path = self._cache_dir / f"{key}.meta.json"
return final_path, part_path, meta_path
def cached_file_path(self, url: str, extension: str) -> Path:
final_path, _, _ = self._cache_paths(url, extension)
return final_path
def get_cached_file(
self,
url: str,
extension: str,
*,
validate_existing: ValidateCallback | None = None,
) -> Path | None:
final_path = self.cached_file_path(url, extension)
if not final_path.exists():
return None
if validate_existing and not validate_existing(final_path):
final_path.unlink(missing_ok=True)
return None
return final_path
@staticmethod
def _load_meta(meta_path: Path) -> dict[str, Any]:
if not meta_path.exists():
@@ -140,7 +185,9 @@ class ResumableFileDownloader:
if progress_callback and expected_size and expected_size > 0:
await progress_callback(expected_size, expected_size)
return final_path
response.raise_for_status()
if response.status_code >= 400:
body = (await response.aread()).decode("utf-8", errors="replace")
raise DownloadHTTPStatusError(url=url, status_code=response.status_code, body=body)
if response.status_code == 206 and resume_from > 0:
mode = "ab"

View File

@@ -163,32 +163,64 @@ class TeleGeographyLandingPointCollector(BaseCollector):
data_type = "landing_point"
async def fetch(self) -> List[Dict[str, Any]]:
"""Fetch landing point data from GitHub mirror"""
url = self._resolved_url or ""
"""Fetch landing point data, falling back when the old mirror disappears."""
config = get_data_sources_config()
sources = [
self._resolved_url or "",
str(config.get_yaml_value("telegeography.landing_point_url") or ""),
str(config.get_yaml_value("arcgis.landing_point_url") or ""),
]
async with httpx.AsyncClient(timeout=60.0) as client:
response = await client.get(url)
response.raise_for_status()
return self.parse_response(response.json())
last_error: Exception | None = None
async with httpx.AsyncClient(timeout=60.0, follow_redirects=True) as client:
for url in dict.fromkeys(source for source in sources if source):
try:
params = (
{"where": "1=1", "outFields": "*", "returnGeometry": "true", "f": "geojson"}
if "FeatureServer" in url or url.endswith("/query")
else None
)
response = await client.get(url, params=params)
response.raise_for_status()
records = self.parse_response(response.json())
if records:
return records
except Exception as exc:
last_error = exc
continue
def parse_response(self, data: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
if last_error:
raise last_error
return self._get_sample_data()
def parse_response(self, data: List[Dict[str, Any]] | Dict[str, Any]) -> List[Dict[str, Any]]:
"""Parse landing point data"""
result = []
items = data.get("features", []) if isinstance(data, dict) else data
for item in data:
for item in items:
props = item.get("properties", {}) if isinstance(item, dict) else {}
geometry = item.get("geometry", {}) if isinstance(item, dict) else {}
source = props or item
coords = geometry.get("coordinates", []) if isinstance(geometry, dict) else []
longitude = coords[0] if len(coords) > 0 else source.get("longitude")
latitude = coords[1] if len(coords) > 1 else source.get("latitude")
source_id = source.get("id") or source.get("OBJECTID") or source.get("city_id") or ""
try:
entry = {
"source_id": f"telegeo_lp_{item.get('id', '')}",
"name": item.get("name", "Unknown"),
"country": item.get("country", "Unknown"),
"city": item.get("city", item.get("name", "")),
"latitude": str(item.get("latitude", "")),
"longitude": str(item.get("longitude", "")),
"source_id": f"telegeo_lp_{source_id}",
"name": source.get("name", source.get("Name", "Unknown")),
"country": source.get("country", "Unknown"),
"city": source.get("city", source.get("Name", source.get("name", ""))),
"latitude": str(latitude or ""),
"longitude": str(longitude or ""),
"value": "",
"unit": "",
"metadata": {
"cable_count": len(item.get("cables", [])),
"url": item.get("url"),
"cable_count": len(source.get("cables", [])),
"url": source.get("url"),
"objectid": source.get("OBJECTID"),
"city_id": source.get("city_id"),
},
"reference_date": datetime.now(UTC).strftime("%Y-%m-%d"),
}

View File

@@ -11,7 +11,7 @@ startup, so it must stay local and deterministic.
For the full design and the reason behind the abstraction (compute centers,
BGP collectors, BGP events, and future entities all share one pipeline),
see ``docs/plans/location-resolver-shared-pipeline-plan.md``.
see ``docs/technical/zh/location-pipeline-development.md``.
The ``ComputeCenterLocation`` dataclass and the public function signatures are
preserved verbatim so existing callers and tests do not need to change.

View File

@@ -0,0 +1,597 @@
"""Kafka-ready datasource job queue backed by PostgreSQL for v1."""
from __future__ import annotations
import asyncio
from datetime import UTC, datetime, timedelta
from uuid import uuid4
from typing import Any
from sqlalchemy import bindparam, select, text
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.cache import cache
from app.core.config import settings
from app.core.logging import get_logger
from app.core.time import to_iso8601_utc
from app.core.websocket.broadcaster import broadcaster
from app.db.session import async_session_factory
from app.models.collected_data import CollectedData
from app.models.data_snapshot import DataSnapshot
from app.models.datasource import DataSource
from app.models.task import CollectionTask
from app.services.collectors.registry import collector_registry
from app.services.datasource_connectivity import (
build_builtin_connectivity_checksum,
get_builtin_effective_candidate,
save_connectivity_success,
)
from app.services.earth_layer_adapters import (
clear_derived_datasource_data,
get_earth_refresh_strategy_for_change,
get_earth_update_layers_for_source,
)
from app.services.earth_layer_cache import invalidate_earth_layer_cache_for_source
from app.services.scheduler import sync_datasource_job
logger = get_logger(__name__)
JOB_TYPE_COLLECT = "collect"
JOB_TYPE_CLEAR_DATA = "clear_data"
JOB_TYPE_CLEAR_CACHE = "clear_cache"
JOB_TYPE_EARTH_REFRESH = "earth_refresh"
JOB_STATUS_QUEUED = "queued"
JOB_STATUS_RUNNING = "running"
JOB_STATUS_CANCELLING = "cancelling"
JOB_STATUS_SUCCESS = "success"
JOB_STATUS_FAILED = "failed"
JOB_STATUS_CANCELLED = "cancelled"
ACTIVE_JOB_STATUSES = (JOB_STATUS_QUEUED, JOB_STATUS_RUNNING, JOB_STATUS_CANCELLING)
TERMINAL_JOB_STATUSES = (JOB_STATUS_SUCCESS, JOB_STATUS_FAILED, JOB_STATUS_CANCELLED)
DATA_WRITE_JOB_TYPES = (JOB_TYPE_COLLECT, JOB_TYPE_CLEAR_DATA, JOB_TYPE_CLEAR_CACHE)
SOURCE_LOCK_JOB_STATUSES = (JOB_STATUS_RUNNING, JOB_STATUS_CANCELLING)
QUEUE_POLL_SECONDS = 0.35
JOB_STALE_LOCK_MINUTES = 90
DEFAULT_WORKER_CONCURRENCY = 2
RUNNING_DATA_JOB_TASKS: dict[int, asyncio.Task[Any]] = {}
def _utcnow() -> datetime:
return datetime.now(UTC)
def _job_worker_id() -> str:
return f"{settings.PROJECT_NAME}:data-job-worker:{uuid4().hex[:8]}"
def is_terminal_job_status(status: str | None) -> bool:
return status in TERMINAL_JOB_STATUSES
async def enqueue_datasource_job(
db: AsyncSession,
datasource: DataSource,
task_type: str,
*,
payload: dict[str, Any] | None = None,
rollback_policy: str = "keep_committed_batches",
dedupe_key: str | None = None,
) -> CollectionTask:
if dedupe_key:
existing = await _get_active_job_by_dedupe_key(db, dedupe_key)
if existing is not None:
return existing
task = CollectionTask(
datasource_id=datasource.id,
source=datasource.source,
task_type=task_type,
status=JOB_STATUS_QUEUED,
phase="queued",
phase_message="任务已进入队列",
payload=payload or {},
rollback_policy=rollback_policy,
dedupe_key=dedupe_key,
)
db.add(task)
await db.commit()
await db.refresh(task)
await _broadcast_task_update(task)
return task
async def enqueue_earth_refresh_job(
db: AsyncSession,
*,
source: str,
payload: dict[str, Any] | None = None,
) -> CollectionTask | None:
layers = list((payload or {}).get("layers") or get_earth_update_layers_for_source(source))
if not layers:
return None
datasource = await _get_or_create_virtual_datasource(db, source)
refresh_payload = {
"source": source,
"layers": layers,
"refresh_strategy": (payload or {}).get("refresh_strategy")
or get_earth_refresh_strategy_for_change((payload or {}).get("table"), source)
or "clear_then_reload",
**(payload or {}),
}
return await enqueue_datasource_job(
db,
datasource,
JOB_TYPE_EARTH_REFRESH,
payload=refresh_payload,
dedupe_key=f"earth_refresh:{source}",
)
async def enqueue_earth_refresh_from_update(payload: dict[str, Any]) -> None:
source = str(payload.get("source") or "").strip()
if not source:
return
async with async_session_factory() as db:
await enqueue_earth_refresh_job(db, source=source, payload=payload)
async def request_cancel_datasource_task(
db: AsyncSession,
task: CollectionTask,
*,
reason: str = "cancelled_by_operator",
) -> CollectionTask:
if is_terminal_job_status(task.status):
return task
running_task = RUNNING_DATA_JOB_TASKS.get(task.id)
if task.status == JOB_STATUS_QUEUED or (
running_task is None
and (
task.status == JOB_STATUS_CANCELLING
or (task.status == JOB_STATUS_RUNNING and task.task_type != JOB_TYPE_COLLECT)
)
):
return await _cancel_task_without_runner(db, task, reason=reason)
task.status = JOB_STATUS_CANCELLING
task.phase = JOB_STATUS_CANCELLING
task.phase_message = "正在停止任务"
task.requested_cancel_at = _utcnow()
task.cancel_reason = reason
await db.commit()
await db.refresh(task)
if running_task is not None and not running_task.done():
running_task.cancel()
await _broadcast_task_update(task)
return task
async def _cancel_task_without_runner(
db: AsyncSession,
task: CollectionTask,
*,
reason: str,
) -> CollectionTask:
if task.task_type == JOB_TYPE_COLLECT:
await db.execute(CollectedData.__table__.delete().where(CollectedData.task_id == task.id))
snapshot_result = await db.execute(select(DataSnapshot).where(DataSnapshot.task_id == task.id))
for snapshot in snapshot_result.scalars().all():
snapshot.status = JOB_STATUS_CANCELLED
snapshot.completed_at = _utcnow()
snapshot.error_message = "Cancelled after operator stop request; no active worker handle remained"
datasource = await db.get(DataSource, task.datasource_id)
if datasource is not None:
datasource.last_status = JOB_STATUS_CANCELLED
task.status = JOB_STATUS_CANCELLED
task.phase = JOB_STATUS_CANCELLED
task.phase_message = "任务已停止"
task.completed_at = _utcnow()
task.requested_cancel_at = task.requested_cancel_at or _utcnow()
task.cancel_reason = reason
await db.commit()
await db.refresh(task)
await _broadcast_task_update(task)
return task
async def get_active_datasource_job(
db: AsyncSession,
datasource_id: int,
*,
task_types: tuple[str, ...] = DATA_WRITE_JOB_TYPES,
) -> CollectionTask | None:
result = await db.execute(
select(CollectionTask)
.where(CollectionTask.datasource_id == datasource_id)
.where(CollectionTask.task_type.in_(task_types))
.where(CollectionTask.status.in_(ACTIVE_JOB_STATUSES))
.order_by(CollectionTask.created_at.desc().nullslast(), CollectionTask.id.desc())
.limit(1)
)
return result.scalar_one_or_none()
async def _get_active_job_by_dedupe_key(db: AsyncSession, dedupe_key: str) -> CollectionTask | None:
result = await db.execute(
select(CollectionTask)
.where(CollectionTask.dedupe_key == dedupe_key)
.where(CollectionTask.status.in_(ACTIVE_JOB_STATUSES))
.order_by(CollectionTask.created_at.desc().nullslast(), CollectionTask.id.desc())
.limit(1)
)
return result.scalar_one_or_none()
async def _get_or_create_virtual_datasource(db: AsyncSession, source: str) -> DataSource:
result = await db.execute(select(DataSource).where(DataSource.source == source))
datasource = result.scalar_one_or_none()
if datasource is not None:
return datasource
datasource = DataSource(
name=f"Earth refresh: {source}",
source=source,
module="SYS",
collector_class="EarthRefreshJob",
is_active=True,
)
db.add(datasource)
await db.commit()
await db.refresh(datasource)
return datasource
async def _broadcast_task_update(task: CollectionTask) -> None:
await broadcaster.broadcast_datasource_task_update(
{
"datasource_id": task.datasource_id,
"collector_name": task.source,
"task_id": task.id,
"task_type": task.task_type,
"status": task.status,
"phase": task.phase,
"phase_progress": task.phase_progress,
"phase_message": task.phase_message,
"phase_current": task.phase_current,
"phase_total": task.phase_total,
"phase_unit": task.phase_unit,
"progress": task.progress,
"records_processed": task.records_processed,
"total_records": task.total_records,
"started_at": to_iso8601_utc(task.started_at),
"completed_at": to_iso8601_utc(task.completed_at),
"requested_cancel_at": to_iso8601_utc(task.requested_cancel_at),
"error_message": task.error_message,
}
)
class DataJobWorker:
def __init__(self, *, concurrency: int = DEFAULT_WORKER_CONCURRENCY) -> None:
self.worker_id = _job_worker_id()
self.concurrency = max(1, concurrency)
self._task: asyncio.Task[None] | None = None
self._stop_event: asyncio.Event | None = None
self._running: set[asyncio.Task[Any]] = set()
def start(self) -> None:
if self._task and not self._task.done():
return
self._stop_event = asyncio.Event()
self._task = asyncio.create_task(self._run(), name="data-job-worker")
async def stop(self) -> None:
if self._stop_event:
self._stop_event.set()
for task in list(self._running):
task.cancel()
if self._task:
await asyncio.gather(self._task, return_exceptions=True)
if self._running:
await asyncio.gather(*self._running, return_exceptions=True)
async def _run(self) -> None:
assert self._stop_event is not None
await self._recover_stale_running_jobs()
while not self._stop_event.is_set():
self._running = {task for task in self._running if not task.done()}
if len(self._running) >= self.concurrency:
await asyncio.sleep(QUEUE_POLL_SECONDS)
continue
task_id = await self._claim_next_job()
if task_id is None:
await asyncio.sleep(QUEUE_POLL_SECONDS)
continue
runner = asyncio.create_task(self._run_claimed_job(task_id), name=f"data-job:{task_id}")
self._running.add(runner)
async def _recover_stale_running_jobs(self) -> None:
cutoff = _utcnow() - timedelta(minutes=JOB_STALE_LOCK_MINUTES)
async with async_session_factory() as db:
result = await db.execute(
select(CollectionTask)
.where(CollectionTask.status.in_((JOB_STATUS_RUNNING, JOB_STATUS_CANCELLING)))
.where(CollectionTask.locked_at.is_not(None))
.where(CollectionTask.locked_at < cutoff)
)
stale_jobs = list(result.scalars().all())
for job in stale_jobs:
job.status = JOB_STATUS_FAILED
job.phase = JOB_STATUS_FAILED
job.completed_at = _utcnow()
job.error_message = "Marked failed after stale data job lock timeout"
if stale_jobs:
await db.commit()
async def _claim_next_job(self) -> int | None:
async with async_session_factory() as db:
row = await db.execute(
text(
"""
SELECT queued.id
FROM collection_tasks AS queued
WHERE queued.status = :queued_status
AND NOT EXISTS (
SELECT 1
FROM collection_tasks AS active
WHERE active.source = queued.source
AND active.id <> queued.id
AND active.status IN :active_statuses
)
ORDER BY queued.created_at ASC NULLS FIRST, queued.id ASC
LIMIT 1
FOR UPDATE SKIP LOCKED
"""
).bindparams(bindparam("active_statuses", expanding=True)),
{
"queued_status": JOB_STATUS_QUEUED,
"active_statuses": SOURCE_LOCK_JOB_STATUSES,
},
)
task_id = row.scalar_one_or_none()
if task_id is None:
return None
task = await db.get(CollectionTask, int(task_id))
if task is None:
return None
task.status = JOB_STATUS_RUNNING
task.phase = "starting"
task.phase_message = "任务开始执行"
task.started_at = task.started_at or _utcnow()
task.worker_id = self.worker_id
task.locked_at = _utcnow()
await db.commit()
await _broadcast_task_update(task)
return int(task_id)
async def _run_claimed_job(self, task_id: int) -> None:
current_task = asyncio.current_task()
if current_task is not None:
RUNNING_DATA_JOB_TASKS[task_id] = current_task
try:
async with async_session_factory() as db:
task = await db.get(CollectionTask, task_id)
if task is None:
return
await self._execute_job(db, task)
except asyncio.CancelledError:
async with async_session_factory() as db:
task = await db.get(CollectionTask, task_id)
if task is not None and not is_terminal_job_status(task.status):
task.status = JOB_STATUS_CANCELLED
task.phase = JOB_STATUS_CANCELLED
task.phase_message = "任务已停止"
task.completed_at = _utcnow()
await db.commit()
await _broadcast_task_update(task)
raise
except Exception as exc:
logger.exception_event(
"Data job failed",
event="data_jobs.job_failed",
context={"task_id": task_id, "error": str(exc)},
)
async with async_session_factory() as db:
task = await db.get(CollectionTask, task_id)
if task is not None:
task.status = JOB_STATUS_FAILED
task.phase = JOB_STATUS_FAILED
task.phase_message = str(exc)
task.error_message = str(exc)
task.completed_at = _utcnow()
await db.commit()
await _broadcast_task_update(task)
finally:
RUNNING_DATA_JOB_TASKS.pop(task_id, None)
async def _execute_job(self, db: AsyncSession, task: CollectionTask) -> None:
if task.task_type == JOB_TYPE_COLLECT:
await _run_collect_job(db, task)
elif task.task_type == JOB_TYPE_CLEAR_DATA:
await _run_clear_data_job(db, task)
elif task.task_type == JOB_TYPE_CLEAR_CACHE:
await _run_clear_cache_job(db, task)
elif task.task_type == JOB_TYPE_EARTH_REFRESH:
await _run_earth_refresh_job(db, task)
else:
raise RuntimeError(f"Unsupported data job type: {task.task_type}")
async def _run_collect_job(db: AsyncSession, task: CollectionTask) -> None:
datasource = await db.get(DataSource, task.datasource_id)
if datasource is None:
raise RuntimeError("Data source not found")
collector = collector_registry.get(datasource.source)
if collector is None:
raise RuntimeError(f"Collector '{datasource.source}' not found")
if not datasource.is_active:
raise RuntimeError("Data source is disabled")
collector._datasource_id = datasource.id
collector._current_task = task
collector._db_session = db
result = await collector.run(db)
datasource.last_run_at = _utcnow()
datasource.last_status = result.get("status")
if datasource.last_status == JOB_STATUS_SUCCESS:
effective_candidate = await get_builtin_effective_candidate(db, datasource.source)
checksum, _credential_context = await build_builtin_connectivity_checksum(
datasource.source,
effective_candidate["endpoint"],
effective_candidate["auth_type"],
effective_candidate["headers"],
effective_candidate["config"],
db,
)
await save_connectivity_success(
db,
datasource.source,
checksum,
{"status_code": None},
connected_by="collection",
)
await db.commit()
await sync_datasource_job(datasource.id)
async def _run_clear_data_job(db: AsyncSession, task: CollectionTask) -> None:
source = str(task.source or (task.payload or {}).get("source") or "").strip()
if not source:
raise RuntimeError("Clear data job has no source")
task.phase = "clearing_data"
task.phase_message = "正在删除数据库数据"
await db.commit()
await _broadcast_task_update(task)
count_result = await db.execute(
select(CollectedData.id).where(CollectedData.source == source)
)
collected_ids = [row[0] for row in count_result.all()]
derived_deleted_counts = await clear_derived_datasource_data(db, source)
if collected_ids:
await db.execute(CollectedData.__table__.delete().where(CollectedData.id.in_(collected_ids)))
deleted_count = len(collected_ids)
derived_deleted_count = sum(derived_deleted_counts.values())
task.records_processed = deleted_count + derived_deleted_count
task.total_records = task.records_processed
task.progress = 100.0
task.phase_progress = 100.0
task.phase_current = task.records_processed
task.phase_total = task.records_processed
task.phase_unit = "records"
task.payload = {
**(task.payload or {}),
"deleted_count": deleted_count,
"derived_deleted_count": derived_deleted_count,
"derived_deleted_counts": derived_deleted_counts,
}
task.status = JOB_STATUS_SUCCESS
task.phase = "completed"
task.phase_message = "数据库数据已清理"
task.completed_at = _utcnow()
await db.commit()
await _broadcast_task_update(task)
async def _run_clear_cache_job(db: AsyncSession, task: CollectionTask) -> None:
source = str(task.source or (task.payload or {}).get("source") or "").strip()
if not source:
raise RuntimeError("Clear cache job has no source")
earth_deleted_count = invalidate_earth_layer_cache_for_source(source)
dashboard_deleted_count = int(cache.delete("dashboard:stats")) + int(cache.delete("dashboard:summary"))
deleted_count = earth_deleted_count + dashboard_deleted_count
task.records_processed = deleted_count
task.total_records = deleted_count
task.progress = 100.0
task.phase_progress = 100.0
task.phase = "completed"
task.phase_message = "缓存已清理"
task.payload = {
**(task.payload or {}),
"earth_layer_deleted_count": earth_deleted_count,
"dashboard_deleted_count": dashboard_deleted_count,
}
task.status = JOB_STATUS_SUCCESS
task.completed_at = _utcnow()
await db.commit()
await _broadcast_task_update(task)
await enqueue_earth_refresh_job(db, source=source, payload={"operation": "CACHE_INVALIDATED"})
async def _run_earth_refresh_job(db: AsyncSession, task: CollectionTask) -> None:
payload = task.payload or {}
source = str(payload.get("source") or task.source or "").strip()
layers = list(payload.get("layers") or get_earth_update_layers_for_source(source))
if not source or not layers:
task.status = JOB_STATUS_SUCCESS
task.phase = "completed"
task.phase_message = "没有需要刷新的 Earth 图层"
task.completed_at = _utcnow()
await db.commit()
await _broadcast_task_update(task)
return
deleted_cache_entries = invalidate_earth_layer_cache_for_source(source)
update_payload = {
"event": "earth.layer.changed",
"action": "database_changed",
"source": source,
"table": payload.get("table"),
"data_type": source,
"layers": layers,
"refresh_strategy": payload.get("refresh_strategy") or "clear_then_reload",
"records_processed": payload.get("records_processed", 0),
"operations": payload.get("operations") or [payload.get("operation") or "CHANGE"],
"operation": payload.get("operation"),
"cache_entries_invalidated": deleted_cache_entries,
"timestamp": to_iso8601_utc(_utcnow()),
}
if payload.get("entity") == "interactable":
update_payload.update(
{
"entity": "interactable",
"action": payload.get("action") or "changed",
"ids": payload.get("ids") or payload.get("entity_keys") or [],
"item": payload.get("item"),
}
)
await broadcaster.broadcast_earth_update(update_payload)
task.records_processed = int(payload.get("records_processed") or 0)
task.progress = 100.0
task.phase_progress = 100.0
task.phase = "completed"
task.phase_message = "Earth 图层刷新通知已发送"
task.status = JOB_STATUS_SUCCESS
task.completed_at = _utcnow()
task.payload = {**payload, "cache_entries_invalidated": deleted_cache_entries}
await db.commit()
await _broadcast_task_update(task)
_worker = DataJobWorker()
def start_data_job_worker() -> None:
_worker.start()
async def stop_data_job_worker() -> None:
await _worker.stop()

View File

@@ -32,28 +32,31 @@ class DocsMetadata:
DOCS_METADATA: tuple[DocsMetadata, ...] = (
DocsMetadata(DOCS_README_FILENAME, DEFAULT_DOCS_SLUG, "public", "Overview", 0, "技术文档", "Technical Docs"),
DocsMetadata("quickstart.md", "quickstart", "public", "Manual", 1, "快速开始", "Quickstart"),
DocsMetadata("manual.md", "manual", "public", "Manual", 2, "Planet 使用手册", "Planet Manual"),
DocsMetadata("manual.md", "manual", "public", "Manual", 1, "智能星球使用手册", "Intelligent Planet Manual"),
DocsMetadata("quickstart.md", "quickstart", "public", "Manual", 2, "快速开始", "Quickstart"),
DocsMetadata("faq.md", "faq", "public", "Manual", 3, "常见问题", "FAQ"),
DocsMetadata("location-pipeline-user.md", "location-pipeline-user", "public", "Manual", 4, "Earth 位置候选采集使用手册", "Earth Location Candidate Collection User Guide"),
DocsMetadata("earth-frontend-context.md", "earth-frontend-context", "docs_developer", "Earth", 10, "Earth 前端结构", "Earth Frontend Context"),
DocsMetadata("earth-layer-style-reference.md", "earth-layer-style-reference", "docs_developer", "Earth", 11, "Earth 图层样式属性索引", "Earth Layer Style Reference"),
DocsMetadata("earth-render-layer-order.md", "earth-render-layer-order", "docs_developer", "Earth", 12, "Earth 渲染图层顺序", "Earth Render Layer Order"),
DocsMetadata("earth-satellite-footprint-policy.md", "earth-satellite-footprint-policy", "docs_developer", "Earth", 13, "Earth 卫星覆盖策略", "Earth Satellite Footprint Policy"),
DocsMetadata("platform-data-flows.md", "platform-data-flows", "docs_developer", "Architecture", 5, "业务架构与数据流转", "Business Architecture and Data Flows"),
DocsMetadata("naming-glossary.md", "naming-glossary", "docs_developer", "Architecture", 6, "命名与术语对照", "Naming Glossary"),
DocsMetadata("earth-frontend-context.md", "earth-frontend-context", "docs_developer", "Earth", 10, "智能星球前端结构", "Intelligent Planet Frontend Context"),
DocsMetadata("earth-layer-style-reference.md", "earth-layer-style-reference", "docs_developer", "Earth", 11, "智能星球图层样式属性索引", "Intelligent Planet Layer Style Reference"),
DocsMetadata("earth-render-layer-order.md", "earth-render-layer-order", "docs_developer", "Earth", 12, "智能星球渲染图层顺序", "Intelligent Planet Render Layer Order"),
DocsMetadata("earth-satellite-footprint-policy.md", "earth-satellite-footprint-policy", "docs_developer", "Earth", 13, "智能星球卫星覆盖策略", "Intelligent Planet Satellite Footprint Policy"),
DocsMetadata("earth-bgp-context.md", "earth-bgp-context", "docs_developer", "Earth", 14, "BGP 态势上下文", "BGP Context"),
DocsMetadata("earth-news-live-streams-collector-format.md", "earth-news-live-streams-collector-format", "docs_developer", "Earth", 15, "新闻直播采集格式", "News Live Streams Collector Format"),
DocsMetadata("earth-interactable-usage.md", "earth-interactable-usage", "docs_developer", "Earth", 16, "Earth 可交互图标接入", "Earth Interactable Usage"),
DocsMetadata("earth-toolbar-overlay-coordination.md", "earth-toolbar-overlay-coordination", "docs_developer", "Earth", 17, "Earth 工具栏与浮层协同", "Earth Toolbar and Overlay Coordination"),
DocsMetadata("earth-interactable-usage.md", "earth-interactable-usage", "docs_developer", "Earth", 16, "智能星球可交互图标接入", "Intelligent Planet Interactable Usage"),
DocsMetadata("earth-toolbar-overlay-coordination.md", "earth-toolbar-overlay-coordination", "docs_developer", "Earth", 17, "智能星球工具栏与浮层协同", "Intelligent Planet Toolbar and Overlay Coordination"),
DocsMetadata("frontend-admin-frontend-context.md", "frontend-admin-frontend-context", "docs_developer", "Frontend", 20, "控制台前端结构", "Admin Frontend Context"),
DocsMetadata("frontend-layout-guidelines.md", "frontend-layout-guidelines", "docs_developer", "Frontend", 21, "前端布局指南", "Frontend Layout Guidelines"),
DocsMetadata("docs-gatekeeper-development.md", "docs-gatekeeper-development", "docs_developer", "Frontend", 22, "Docs Gatekeeper 开发说明", "Docs Gatekeeper Development Guide"),
DocsMetadata("tactile-ui-components.md", "tactile-ui-components", "docs_developer", "Frontend", 24, "Tactile UI 组件库", "Tactile UI Components"),
DocsMetadata("backend-collectors.md", "backend-collectors", "docs_developer", "Backend", 30, "数据采集系统", "Data Collectors"),
DocsMetadata("backend-system-service-control.md", "backend-system-service-control", "docs_admin", "Backend", 31, "系统服务控制", "System Service Control"),
DocsMetadata("datasource-collector-settings-connectivity.md", "datasource-collector-settings-connectivity", "docs_developer", "Backend", 32, "数据源、采集器设置与连接验证", "Datasource Collector Settings and Connectivity"),
DocsMetadata("backend-datasources-api-performance.md", "backend-datasources-api-performance", "docs_developer", "Backend", 33, "数据源 API 性能", "Datasource API Performance"),
DocsMetadata("location-pipeline-development.md", "location-pipeline-development", "docs_developer", "Backend", 34, "通用位置估算管线开发说明", "Shared Location Resolution Pipeline Development Guide"),
DocsMetadata("data-job-earth-sync-architecture.md", "data-job-earth-sync-architecture", "docs_developer", "Backend", 34, "数据作业与 Outbox 技术架构", "Data Jobs and Outbox Architecture"),
DocsMetadata("location-pipeline-development.md", "location-pipeline-development", "docs_developer", "Backend", 35, "通用位置估算管线开发说明", "Shared Location Resolution Pipeline Development Guide"),
DocsMetadata("earth-news-live-streams-collector-format.md", "earth-news-live-streams-collector-format", "docs_developer", "Backend", 36, "新闻直播采集格式", "News Live Streams Collector Format"),
DocsMetadata("docs-gatekeeper-development.md", "docs-gatekeeper-development", "docs_developer", "Backend", 37, "Docs Gatekeeper 开发说明", "Docs Gatekeeper Development Guide"),
DocsMetadata("agents-aiprovider.md", "agents-aiprovider", "docs_developer", "Agents", 40, "AI Provider 指南", "AI Provider Guide"),
DocsMetadata("ops-runbook.md", "ops-runbook", "docs_admin", "Ops", 49, "Planet 运维手册", "Planet Ops Runbook"),
DocsMetadata("ops-runbook.md", "ops-runbook", "docs_admin", "Ops", 49, "智能星球运维手册", "Intelligent Planet Ops Runbook"),
DocsMetadata("ops-docker-compose-buildx-upgrade.md", "ops-docker-compose-buildx-upgrade", "docs_admin", "Ops", 50, "Docker + Compose + Buildx 升级", "Docker + Compose + Buildx Upgrade"),
DocsMetadata("ops-planet-sh-startup.md", "ops-planet-sh-startup", "docs_admin", "Ops", 51, "planet.sh 启动机制", "planet.sh Startup"),
)

View File

@@ -0,0 +1,482 @@
"""PostgreSQL LISTEN/NOTIFY bridge for Earth layer refresh events."""
from __future__ import annotations
import asyncio
import json
from collections import deque
from dataclasses import dataclass, field
from datetime import UTC, datetime
from time import monotonic
from typing import Any, Awaitable, Callable
import asyncpg
from app.core.config import settings
from app.core.logging import get_logger
from app.core.time import to_iso8601_utc
from app.core.websocket.broadcaster import broadcaster
from app.services.earth_layer_adapters import (
get_earth_refresh_strategy_for_change,
get_earth_update_layers_for_change,
get_earth_update_layers_for_source,
)
from app.services.earth_layer_cache import invalidate_earth_layer_cache_for_source
logger = get_logger(__name__)
EARTH_DATA_CHANGES_CHANNEL = "planet_earth_data_changes"
DEFAULT_DEBOUNCE_SECONDS = 0.25
DEFAULT_MAX_WAIT_SECONDS = 1.5
DELETE_FAST_FLUSH_SECONDS = 0.05
LISTEN_KEEPALIVE_SECONDS = 5.0
OUTBOX_POLL_LIMIT = 5000
MAX_ENTITY_KEY_SAMPLES = 20
MAX_SEEN_EVENT_IDS = 20000
BroadcastFn = Callable[[dict[str, Any]], Awaitable[None]]
InvalidateFn = Callable[[str], int]
def normalize_asyncpg_dsn(dsn: str) -> str:
"""Convert SQLAlchemy asyncpg URLs into asyncpg-compatible URLs."""
return dsn.replace("postgresql+asyncpg://", "postgresql://", 1)
def build_earth_update_from_db_payload(payload: dict[str, Any]) -> dict[str, Any] | None:
table = payload.get("table")
source = str(payload.get("source") or "").strip()
table_name = str(table or "").strip()
if not source and not table_name:
return None
layers = get_earth_update_layers_for_change(table_name, source)
if not layers:
return None
refresh_strategy = get_earth_refresh_strategy_for_change(table_name, source) or "clear_then_reload"
source_has_adapter = bool(get_earth_update_layers_for_source(source))
effective_source = source if source_has_adapter else (table_name if table_name else source)
operation = payload.get("operation")
update: dict[str, Any] = {
"event": "earth.layer.changed",
"action": "database_changed",
"source": effective_source,
"original_source": source or None,
"table": table_name or None,
"data_type": effective_source,
"layers": layers,
"refresh_strategy": refresh_strategy,
"operation": operation,
"entity_key": payload.get("entity_key"),
"timestamp": to_iso8601_utc(datetime.now(UTC)),
}
if table_name == "earth_interactables" and refresh_strategy == "delta":
ids = payload.get("entity_keys")
if not isinstance(ids, list):
ids = [payload.get("entity_key")] if payload.get("entity_key") else []
update.update(
{
"entity": "interactable",
"action": "deleted" if operation == "DELETE" else "changed",
"ids": [str(item) for item in ids if item],
"item": None,
}
)
return {
**update,
}
@dataclass
class PendingEarthDbChange:
source: str
layers: list[str]
table: str | None = None
refresh_strategy: str = "clear_then_reload"
entity: str | None = None
action: str = "database_changed"
records_processed: int = 0
operations: set[str] = field(default_factory=set)
entity_keys: list[str] = field(default_factory=list)
first_occurred_at: str | None = None
last_occurred_at: str | None = None
first_seen_monotonic: float = field(default_factory=monotonic)
last_seen_monotonic: float = field(default_factory=monotonic)
def add(self, payload: dict[str, Any]) -> None:
self.last_seen_monotonic = monotonic()
records_processed = payload.get("records_processed", 1)
try:
records_processed = int(records_processed)
except (TypeError, ValueError):
records_processed = 1
self.records_processed += max(records_processed, 1)
operation = payload.get("operation")
if operation:
self.operations.add(str(operation))
entity_keys = payload.get("entity_keys")
if not isinstance(entity_keys, list):
entity_key = payload.get("entity_key")
entity_keys = [entity_key] if entity_key else []
for entity_key in entity_keys:
if entity_key and len(self.entity_keys) < MAX_ENTITY_KEY_SAMPLES:
self.entity_keys.append(str(entity_key))
occurred_at = payload.get("occurred_at")
if occurred_at:
occurred_at = str(occurred_at)
self.first_occurred_at = self.first_occurred_at or occurred_at
self.last_occurred_at = occurred_at
class EarthDbChangeDispatcher:
"""Debounces database notifications and broadcasts Earth refresh hints."""
def __init__(
self,
*,
broadcast_earth_update: BroadcastFn | None = None,
invalidate_cache: InvalidateFn | None = None,
debounce_seconds: float = DEFAULT_DEBOUNCE_SECONDS,
max_wait_seconds: float = DEFAULT_MAX_WAIT_SECONDS,
) -> None:
self._broadcast_earth_update = broadcast_earth_update or broadcaster.broadcast_earth_update
self._invalidate_cache = invalidate_cache or invalidate_earth_layer_cache_for_source
self._debounce_seconds = debounce_seconds
self._max_wait_seconds = max(max_wait_seconds, debounce_seconds)
self._pending: dict[str, PendingEarthDbChange] = {}
self._flush_tasks: dict[str, asyncio.Task[None]] = {}
self._seen_event_ids: set[int] = set()
self._seen_event_order: deque[int] = deque()
def handle_notification(self, payload_text: str) -> bool:
try:
payload = json.loads(payload_text)
except json.JSONDecodeError:
logger.warning_event(
"Ignoring malformed Earth database change notification",
event="earth.db_changes.notification_malformed",
)
return False
if not isinstance(payload, dict):
return False
return self.handle_payload(payload)
def handle_payload(self, payload: dict[str, Any]) -> bool:
event_id = payload.get("event_id")
if event_id is not None:
try:
normalized_event_id = int(event_id)
except (TypeError, ValueError):
normalized_event_id = None
if normalized_event_id is not None:
if normalized_event_id in self._seen_event_ids:
return False
self._remember_event_id(normalized_event_id)
update = build_earth_update_from_db_payload(payload)
if not update:
return False
source = update["source"]
pending = self._pending.get(source)
if pending is None:
pending = PendingEarthDbChange(
source=source,
layers=list(update["layers"]),
table=update.get("table"),
refresh_strategy=str(update.get("refresh_strategy") or "clear_then_reload"),
entity=update.get("entity"),
action=str(update.get("action") or "database_changed"),
)
self._pending[source] = pending
pending.add(payload)
task = self._flush_tasks.pop(source, None)
if task and not task.done():
task.cancel()
self._flush_tasks[source] = asyncio.create_task(
self._flush_later(source, delay_seconds=self._next_flush_delay(pending))
)
return True
def _next_flush_delay(self, pending: PendingEarthDbChange) -> float:
if "DELETE" in pending.operations and pending.refresh_strategy == "clear_then_reload":
return DELETE_FAST_FLUSH_SECONDS
elapsed = max(0.0, monotonic() - pending.first_seen_monotonic)
remaining = self._max_wait_seconds - elapsed
if remaining <= 0:
return 0.0
return min(self._debounce_seconds, remaining)
def _remember_event_id(self, event_id: int) -> None:
self._seen_event_ids.add(event_id)
self._seen_event_order.append(event_id)
while len(self._seen_event_order) > MAX_SEEN_EVENT_IDS:
expired_event_id = self._seen_event_order.popleft()
self._seen_event_ids.discard(expired_event_id)
async def _flush_later(self, source: str, *, delay_seconds: float) -> None:
try:
if delay_seconds > 0:
await asyncio.sleep(delay_seconds)
await self.flush_source(source)
except asyncio.CancelledError:
raise
except Exception as exc:
logger.exception_event(
"Failed to broadcast debounced Earth database change",
event="earth.db_changes.flush_failed",
context={"source": source, "error": str(exc)},
)
finally:
current = self._flush_tasks.get(source)
if current is asyncio.current_task():
self._flush_tasks.pop(source, None)
async def flush_source(self, source: str) -> None:
pending = self._pending.get(source)
if pending is None:
return
flushed_at = datetime.now(UTC)
deleted_cache_entries = self._invalidate_cache(source)
operations = sorted(pending.operations)
payload: dict[str, Any] = {
"event": "earth.layer.changed",
"action": "database_changed",
"source": source,
"table": pending.table,
"data_type": source,
"layers": pending.layers,
"refresh_strategy": pending.refresh_strategy,
"records_processed": pending.records_processed,
"operations": operations,
"operation": operations[-1] if len(operations) == 1 else None,
"entity_keys": pending.entity_keys,
"entity_key_sample_size": len(pending.entity_keys),
"cache_entries_invalidated": deleted_cache_entries,
"first_occurred_at": pending.first_occurred_at,
"last_occurred_at": pending.last_occurred_at,
"debounce_ms": int((monotonic() - pending.first_seen_monotonic) * 1000),
"total_latency_ms": self._total_latency_ms(pending, flushed_at),
"timestamp": to_iso8601_utc(flushed_at),
}
if pending.entity == "interactable":
payload.update(
{
"entity": "interactable",
"action": "deleted" if "DELETE" in pending.operations else "changed",
"ids": pending.entity_keys,
"item": None,
}
)
await self._broadcast_earth_update(payload)
self._pending.pop(source, None)
logger.info_event(
"Broadcasted Earth database change",
event="earth.db_changes.broadcasted",
context={
"source": source,
"layers": pending.layers,
"records_processed": pending.records_processed,
"cache_entries_invalidated": deleted_cache_entries,
"debounce_ms": int((monotonic() - pending.first_seen_monotonic) * 1000),
"total_latency_ms": payload["total_latency_ms"],
},
)
@staticmethod
def _total_latency_ms(pending: PendingEarthDbChange, flushed_at: datetime) -> int | None:
occurred_at = pending.first_occurred_at
if not occurred_at:
return None
try:
normalized = occurred_at.replace("Z", "+00:00")
occurred = datetime.fromisoformat(normalized)
if occurred.tzinfo is None:
occurred = occurred.replace(tzinfo=UTC)
return max(0, int((flushed_at - occurred.astimezone(UTC)).total_seconds() * 1000))
except ValueError:
return None
async def flush_all(self) -> None:
sources = list(self._pending)
for source in sources:
task = self._flush_tasks.pop(source, None)
if task and not task.done():
task.cancel()
await self.flush_source(source)
async def stop(self) -> None:
tasks = [task for task in self._flush_tasks.values() if not task.done()]
for task in tasks:
task.cancel()
if tasks:
await asyncio.gather(*tasks, return_exceptions=True)
self._flush_tasks.clear()
await self.flush_all()
class EarthDbChangeListener:
def __init__(
self,
*,
dsn: str,
dispatcher: EarthDbChangeDispatcher,
channel: str = EARTH_DATA_CHANGES_CHANNEL,
) -> None:
self._dsn = normalize_asyncpg_dsn(dsn)
self._dispatcher = dispatcher
self._channel = channel
self._task: asyncio.Task[None] | None = None
self._stop_event: asyncio.Event | None = None
self._connection: asyncpg.Connection | None = None
self._loop: asyncio.AbstractEventLoop | None = None
def start(self) -> None:
if self._task and not self._task.done():
return
self._loop = asyncio.get_running_loop()
self._stop_event = asyncio.Event()
self._task = asyncio.create_task(self._run())
async def stop(self) -> None:
if self._stop_event:
self._stop_event.set()
if self._connection:
await self._connection.close()
if self._task:
await asyncio.gather(self._task, return_exceptions=True)
await self._dispatcher.stop()
async def _run(self) -> None:
backoff_seconds = 1.0
assert self._stop_event is not None
while not self._stop_event.is_set():
try:
self._connection = await asyncpg.connect(self._dsn)
await self._connection.add_listener(self._channel, self._on_notification)
logger.info_event(
"Earth database change listener connected",
event="earth.db_changes.connected",
context={"channel": self._channel},
)
backoff_seconds = 1.0
while not self._stop_event.is_set():
try:
await asyncio.wait_for(
self._stop_event.wait(),
timeout=LISTEN_KEEPALIVE_SECONDS,
)
except asyncio.TimeoutError:
await self._poll_outbox()
except asyncio.CancelledError:
raise
except Exception as exc:
logger.exception_event(
"Earth database change listener failed",
event="earth.db_changes.listener_failed",
context={"channel": self._channel, "error": str(exc)},
)
try:
await asyncio.wait_for(self._stop_event.wait(), timeout=backoff_seconds)
except asyncio.TimeoutError:
pass
backoff_seconds = min(backoff_seconds * 2, 30.0)
finally:
if self._connection:
try:
await self._connection.remove_listener(self._channel, self._on_notification)
except Exception:
pass
try:
await self._connection.close()
except Exception:
pass
self._connection = None
async def _poll_outbox(self) -> None:
if self._connection is None:
return
rows = await self._connection.fetch(
"""
SELECT id, payload
FROM earth_data_change_events
WHERE consumed_at IS NULL
ORDER BY id
LIMIT $1
""",
OUTBOX_POLL_LIMIT,
)
accepted_count = 0
consumed_ids: list[int] = []
for row in rows:
payload = row["payload"]
if isinstance(payload, str):
try:
payload = json.loads(payload)
except json.JSONDecodeError:
consumed_ids.append(int(row["id"]))
continue
if isinstance(payload, dict):
if self._dispatcher.handle_payload(payload):
accepted_count += 1
consumed_ids.append(int(row["id"]))
else:
consumed_ids.append(int(row["id"]))
if consumed_ids:
await self._dispatcher.flush_all()
if consumed_ids:
await self._connection.execute(
"""
UPDATE earth_data_change_events
SET consumed_at = NOW()
WHERE id = ANY($1::bigint[])
AND consumed_at IS NULL
""",
consumed_ids,
)
if rows:
logger.info_event(
"Polled Earth database change outbox",
event="earth.db_changes.outbox_polled",
context={"events": len(rows), "accepted": accepted_count},
)
def _on_notification(
self,
_connection: asyncpg.Connection,
_pid: int,
_channel: str,
payload: str,
) -> None:
if self._loop and self._loop.is_running():
self._loop.call_soon_threadsafe(self._dispatcher.handle_notification, payload)
return
self._dispatcher.handle_notification(payload)
_dispatcher = EarthDbChangeDispatcher(
broadcast_earth_update=broadcaster.broadcast_earth_update,
invalidate_cache=invalidate_earth_layer_cache_for_source,
)
_listener: EarthDbChangeListener | None = None
def start_earth_db_change_listener() -> None:
global _listener
if _listener is not None:
return
_listener = EarthDbChangeListener(dsn=settings.DATABASE_URL, dispatcher=_dispatcher)
_listener.start()
async def stop_earth_db_change_listener() -> None:
global _listener
if _listener is None:
await _dispatcher.stop()
return
listener = _listener
_listener = None
await listener.stop()

View File

@@ -0,0 +1,113 @@
from __future__ import annotations
from datetime import UTC, datetime
from typing import Any
from uuid import uuid4
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.time import to_iso8601_utc
from app.core.websocket.broadcaster import broadcaster
from app.models.earth_interactable import EarthInteractable
from app.services.earth_layer_cache import EARTH_LAYER_CACHE_PREFIX, earth_layer_cache
INTERACTABLE_ENTITY = "interactable"
INTERACTABLE_LAYER = "interactables"
def normalize_interactable_id(value: str | None = None) -> str:
raw = str(value or "").strip()
return raw or f"interactable-{uuid4().hex}"
def serialize_interactable(record: EarthInteractable) -> dict[str, Any]:
return {
"id": record.id,
"layer": record.layer,
"kind": record.kind,
"label": record.label,
"description": record.description,
"latitude": record.latitude,
"longitude": record.longitude,
"altitude": record.altitude,
"revision": record.revision,
"properties": record.properties or {},
"is_deleted": bool(record.is_deleted),
"created_at": to_iso8601_utc(record.created_at),
"updated_at": to_iso8601_utc(record.updated_at),
"deleted_at": to_iso8601_utc(record.deleted_at),
}
def interactables_to_geojson(items: list[EarthInteractable]) -> dict[str, Any]:
return {
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"id": item.id,
"geometry": {
"type": "Point",
"coordinates": [item.longitude, item.latitude],
},
"properties": serialize_interactable(item),
}
for item in items
if not item.is_deleted
],
}
def invalidate_interactable_cache(layer: str | None = None) -> int:
layer_key = str(layer or "*").strip() or "*"
deleted = earth_layer_cache.delete_pattern(
f"{EARTH_LAYER_CACHE_PREFIX}:interactables:layer:{layer_key}*"
)
if layer_key != "all":
deleted += earth_layer_cache.delete_pattern(
f"{EARTH_LAYER_CACHE_PREFIX}:interactables:layer:all*"
)
return deleted
def build_interactable_event(
*,
action: str,
record: EarthInteractable,
include_item: bool = True,
) -> dict[str, Any]:
item = serialize_interactable(record)
return {
"entity": INTERACTABLE_ENTITY,
"action": action,
"layer": record.layer,
"layers": [INTERACTABLE_LAYER],
"ids": [record.id],
"revision": record.revision,
"changed_at": item["deleted_at"] or item["updated_at"] or to_iso8601_utc(datetime.now(UTC)),
"item": item if include_item else None,
"source": "earth_interactables",
}
async def publish_interactable_event(action: str, record: EarthInteractable, *, include_item: bool = True) -> None:
await broadcaster.broadcast_earth_update(
build_interactable_event(action=action, record=record, include_item=include_item)
)
async def list_interactables(
db: AsyncSession,
*,
layer: str | None = None,
include_deleted: bool = False,
) -> list[EarthInteractable]:
stmt = select(EarthInteractable)
if layer:
stmt = stmt.where(EarthInteractable.layer == layer)
if not include_deleted:
stmt = stmt.where(EarthInteractable.is_deleted.is_(False))
stmt = stmt.order_by(EarthInteractable.updated_at.desc(), EarthInteractable.id.asc())
result = await db.execute(stmt)
return list(result.scalars().all())

View File

@@ -0,0 +1,179 @@
"""Earth layer adapter registry for datasource-backed refresh behavior."""
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Any
from sqlalchemy.ext.asyncio import AsyncSession
@dataclass(frozen=True)
class EarthLayerAdapter:
sources: frozenset[str]
layers: tuple[str, ...]
cache_patterns: tuple[str, ...]
tables: frozenset[str] = field(default_factory=frozenset)
derived_models: tuple[str, ...] = field(default_factory=tuple)
refresh_strategy: str = "clear_then_reload"
EARTH_LAYER_ADAPTERS: tuple[EarthLayerAdapter, ...] = (
EarthLayerAdapter(
sources=frozenset({"barentswatch_vessels", "aisstream_vessels", "vessel_static", "vessel_position", "ais_raw_observations", "ais_source_health"}),
tables=frozenset({"vessel_static", "vessel_position", "ais_raw_observations", "ais_source_health"}),
layers=("vessels",),
cache_patterns=("vessels*", "summary*"),
),
EarthLayerAdapter(
sources=frozenset(
{
"telegeography_cables",
"telegeography_landing",
"telegeography_landing_points",
"telegeography_systems",
"telegeography_cable_systems",
"arcgis_cables",
"arcgis_landing_points",
"arcgis_cable_landing_relation",
"arcgis_cable_landing_relations",
"fao_landing_points",
}
),
tables=frozenset({"collected_data"}),
layers=("cables",),
cache_patterns=("cables*", "landing-points*", "summary*"),
),
EarthLayerAdapter(
sources=frozenset({"celestrak_tle", "spacetrack_tle"}),
tables=frozenset({"collected_data"}),
layers=("satellites",),
cache_patterns=("satellites*", "summary*"),
),
EarthLayerAdapter(
sources=frozenset(
{
"top500",
"top500_supercomputers",
"epoch_ai_gpu",
"huggingface_models",
"huggingface_datasets",
"huggingface_spaces",
"compute_center_locations",
}
),
tables=frozenset({"compute_center_locations"}),
layers=("computeCenters",),
cache_patterns=("compute-centers*", "summary*"),
refresh_strategy="reload",
),
EarthLayerAdapter(
sources=frozenset(
{
"ris_live_bgp",
"bgpstream_bgp",
"iptoasn_prefix_geo",
"opengeofeed_prefix_geo",
"nro_delegated_prefix_geo",
"bgp_observations",
"bgp_anomalies",
"bgp_incidents",
"bgp_collector_locations",
}
),
tables=frozenset({"bgp_observations", "bgp_anomalies", "bgp_incidents", "bgp_collector_locations"}),
layers=("bgp",),
cache_patterns=("bgp*", "summary*"),
derived_models=("bgp_observations", "bgp_anomalies", "bgp_incidents"),
),
EarthLayerAdapter(
sources=frozenset({"news_live_streams"}),
tables=frozenset({"collected_data"}),
layers=("media",),
cache_patterns=("summary*",),
refresh_strategy="reload",
),
EarthLayerAdapter(
sources=frozenset({"media_news_archive", "earth_news_items"}),
tables=frozenset({"earth_news_items"}),
layers=("news",),
cache_patterns=("summary*",),
refresh_strategy="reload",
),
EarthLayerAdapter(
sources=frozenset({"earth_interactables"}),
tables=frozenset({"earth_interactables"}),
layers=("interactables",),
cache_patterns=("interactables*", "summary*"),
refresh_strategy="delta",
),
)
_ADAPTERS_BY_SOURCE = {
source: adapter
for adapter in EARTH_LAYER_ADAPTERS
for source in adapter.sources
}
_ADAPTERS_BY_TABLE = {
table: adapter
for adapter in EARTH_LAYER_ADAPTERS
for table in adapter.tables
}
def get_earth_layer_adapter_for_source(source: str | None) -> EarthLayerAdapter | None:
return _ADAPTERS_BY_SOURCE.get(str(source or "").strip())
def get_earth_layer_adapter_for_change(table: str | None, source: str | None) -> EarthLayerAdapter | None:
table_key = str(table or "").strip()
source_key = str(source or "").strip()
if table_key and table_key != "collected_data":
adapter = _ADAPTERS_BY_TABLE.get(table_key)
if adapter is not None:
return adapter
return get_earth_layer_adapter_for_source(source_key)
def get_earth_update_layers_for_source(source: str | None) -> list[str]:
adapter = get_earth_layer_adapter_for_source(source)
return list(adapter.layers) if adapter else []
def get_earth_update_layers_for_change(table: str | None, source: str | None) -> list[str]:
adapter = get_earth_layer_adapter_for_change(table, source)
return list(adapter.layers) if adapter else []
def get_earth_refresh_strategy_for_change(table: str | None, source: str | None) -> str | None:
adapter = get_earth_layer_adapter_for_change(table, source)
return adapter.refresh_strategy if adapter else None
def get_earth_cache_patterns_for_source(source: str | None) -> list[str]:
adapter = get_earth_layer_adapter_for_source(source)
return list(adapter.cache_patterns) if adapter else []
async def clear_derived_datasource_data(db: AsyncSession, source: str) -> dict[str, int]:
adapter = get_earth_layer_adapter_for_source(source)
if adapter is None or not adapter.derived_models:
return {}
from app.models.bgp_anomaly import BGPAnomaly
from app.models.bgp_incident import BGPIncident
from app.models.bgp_observation import BGPObservation
model_by_key: dict[str, Any] = {
"bgp_observations": BGPObservation,
"bgp_anomalies": BGPAnomaly,
"bgp_incidents": BGPIncident,
}
deleted_counts: dict[str, int] = {}
for key in adapter.derived_models:
model = model_by_key.get(key)
if model is None:
continue
result = await db.execute(model.__table__.delete().where(model.source == source))
deleted_counts[key] = int(result.rowcount or 0)
return deleted_counts

View File

@@ -268,34 +268,10 @@ def apply_payload_budget(payload: dict[str, Any], policy: EarthLayerCachePolicy)
def invalidate_earth_layer_cache_for_source(source: str) -> int:
from app.services.earth_layer_adapters import get_earth_cache_patterns_for_source
source_key = str(source or "").strip()
patterns = {
"barentswatch_vessels": ["vessels*", "summary*"],
"aisstream_vessels": ["vessels*", "summary*"],
"telegeography_cables": ["cables*", "landing-points*", "summary*"],
"telegeography_landing": ["landing-points*", "summary*"],
"telegeography_landing_points": ["landing-points*", "summary*"],
"telegeography_systems": ["cables*", "summary*"],
"telegeography_cable_systems": ["cables*", "summary*"],
"arcgis_cables": ["cables*", "landing-points*", "summary*"],
"arcgis_landing_points": ["landing-points*", "summary*"],
"arcgis_cable_landing_relation": ["landing-points*", "summary*"],
"arcgis_cable_landing_relations": ["landing-points*", "summary*"],
"fao_landing_points": ["landing-points*", "summary*"],
"celestrak_tle": ["satellites*", "summary*"],
"spacetrack_tle": ["satellites*", "summary*"],
"top500": ["compute-centers*", "summary*"],
"top500_supercomputers": ["compute-centers*", "summary*"],
"epoch_ai_gpu": ["compute-centers*", "summary*"],
"huggingface_models": ["compute-centers*", "summary*"],
"huggingface_datasets": ["compute-centers*", "summary*"],
"huggingface_spaces": ["compute-centers*", "summary*"],
"ris_live_bgp": ["bgp*", "summary*"],
"bgpstream_bgp": ["bgp*", "summary*"],
"iptoasn_prefix_geo": ["bgp*", "summary*"],
"opengeofeed_prefix_geo": ["bgp*", "summary*"],
"nro_delegated_prefix_geo": ["bgp*", "summary*"],
}.get(source_key, [])
patterns = get_earth_cache_patterns_for_source(source_key)
deleted = 0
for layer_pattern in patterns:
deleted += earth_layer_cache.delete_pattern(f"{EARTH_LAYER_CACHE_PREFIX}:{layer_pattern}")

View File

@@ -1063,6 +1063,7 @@ def _build_payload(
lon: float | None,
active_region: str,
items: list[ParsedNewsItem],
cruise_items: list[ParsedNewsItem] | None = None,
sources: list[NewsFeedSource],
errors: list[str],
stale: bool,
@@ -1082,6 +1083,10 @@ def _build_payload(
},
"sources": _serialize_sources(sources),
"items": [_serialize_item(item, active_region=active_region) for item in items],
"cruise_items": [
_serialize_item(item, active_region=active_region)
for item in (cruise_items if cruise_items is not None else items)
],
"errors": errors,
"stale": stale,
}
@@ -1295,6 +1300,7 @@ async def get_earth_news_payload(
from app.services.earth_news_store import (
get_earth_news_freshness,
list_earth_news_cruise_items,
list_earth_news_items,
upsert_earth_news_items,
)
@@ -1312,6 +1318,13 @@ async def get_earth_news_payload(
active_region=active_region,
limit=MAX_ITEMS_TOTAL,
)
if hasattr(db, "execute"):
cruise_items = await list_earth_news_cruise_items(
db,
limit=MAX_ITEMS_TOTAL * len(REGION_ANCHORS),
)
else:
cruise_items = items
await _enqueue_unverified_locations(items)
stale = bool(errors and items)
@@ -1320,6 +1333,7 @@ async def get_earth_news_payload(
lon=lon,
active_region=active_region,
items=items,
cruise_items=cruise_items,
sources=sources,
errors=errors,
stale=stale,

View File

@@ -78,6 +78,24 @@ async def list_earth_news_items(
return [record_to_parsed_news_item(record) for record in result.scalars().all()]
async def list_earth_news_cruise_items(
db: AsyncSession,
*,
limit: int,
) -> list[ParsedNewsItem]:
result = await db.execute(
select(EarthNewsItem)
.order_by(
EarthNewsItem.region.asc(),
EarthNewsItem.published_at.desc().nullslast(),
EarthNewsItem.last_seen_at.desc(),
EarthNewsItem.feed_name.asc(),
)
.limit(limit)
)
return [record_to_parsed_news_item(record) for record in result.scalars().all()]
async def get_earth_news_freshness(
db: AsyncSession,
*,

View File

@@ -10,6 +10,7 @@ from fastapi import HTTPException, status
from sqlalchemy import func, select, update
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.logging import get_logger
from app.db.session import async_session_factory
from app.models.playground_message import PlaygroundMessage
from app.models.playground_session import PlaygroundSession
@@ -27,9 +28,11 @@ from app.schemas.ai import (
SituationalAnalysisRequest,
)
from app.services.ai_client import AIProviderClient
from app.services.business_logs import emit_business_log, exception_context
from app.services.playground_session_store import _to_response as session_to_response
from app.services.playground_session_store import upsert_playground_session
logger = get_logger(__name__, service="ai")
STREAM_CHUNK_SIZE = 24
STREAM_INTERVAL_SECONDS = 0.08
THINKING_PREVIEW_SECONDS = 2.6
@@ -624,7 +627,42 @@ async def _run_assistant_message(
thinking={"type": "enabled"},
)
await emit_business_log(
logger,
event="ai.playground.run.start",
message="Playground AI run started",
category="ai",
service="ai",
module=__name__,
request_id=request_id,
user_id=user_id,
context={
"session_id": session_id,
"session_key": session_key,
"user_message_id": user_message_id,
"assistant_message_id": assistant_message_id,
"preset": payload.selected_preset_key,
},
)
analysis = await provider_client.analyze(request_payload, request_id=request_id)
await emit_business_log(
logger,
event="ai.playground.run.success",
message="Playground AI run completed",
category="ai",
service="ai",
module=__name__,
request_id=request_id,
user_id=user_id,
context={
"session_id": session_id,
"session_key": session_key,
"provider": analysis.provider,
"model": analysis.model,
"content_block_count": len(analysis.content_blocks or []),
"thinking_block_count": len(analysis.thinking_blocks or []),
},
)
async with async_session_factory() as db:
assistant_message = await _mark_message_state(
@@ -704,6 +742,23 @@ async def _run_assistant_message(
await db.flush()
await db.commit()
except asyncio.CancelledError:
await emit_business_log(
logger,
event="ai.playground.run.cancelled",
message="Playground AI run cancelled",
category="ai",
level="warning",
service="ai",
module=__name__,
request_id=request_id,
user_id=user_id,
context={
"session_id": session_id,
"session_key": session_key,
"assistant_message_id": assistant_message_id,
"duration_ms": round((perf_counter() - started_at) * 1000),
},
)
async with async_session_factory() as db:
result = await db.execute(select(PlaygroundMessage).where(PlaygroundMessage.id == assistant_message_id))
message = result.scalar_one_or_none()
@@ -715,6 +770,26 @@ async def _run_assistant_message(
await db.commit()
raise
except Exception as exc:
await emit_business_log(
logger,
event="ai.playground.run.failed",
message="Playground AI run failed",
category="ai",
level="error",
service="ai",
module=__name__,
request_id=request_id,
user_id=user_id,
context=exception_context(
exc,
{
"session_id": session_id,
"session_key": session_key,
"assistant_message_id": assistant_message_id,
"duration_ms": round((perf_counter() - started_at) * 1000),
},
),
)
error_message = _format_run_exception(exc)
async with async_session_factory() as db:
result = await db.execute(select(PlaygroundMessage).where(PlaygroundMessage.id == assistant_message_id))

View File

@@ -13,6 +13,7 @@ from app.db.session import async_session_factory
from app.core.time import to_iso8601_utc
from app.models.datasource import DataSource
from app.models.task import CollectionTask
from app.services.business_logs import emit_business_log, emit_business_log_background, exception_context
from app.services.collectors.registry import collector_registry
from app.services.datasource_connectivity import (
build_builtin_connectivity_checksum,
@@ -124,6 +125,15 @@ async def run_collector_task(collector_name: str):
event="collector.run.skipped_disabled",
context={"collector_name": collector_name},
)
await emit_business_log(
logger,
event="collector.run.skipped_disabled",
message="Skipping disabled collector",
category="collector",
service="scheduler",
module=__name__,
context={"collector_name": collector_name, "datasource_id": datasource.id, "status": "skipped"},
)
return
running_result = await db.execute(
@@ -152,6 +162,21 @@ async def run_collector_task(collector_name: str):
event="collector.run.skipped_already_running",
context={"collector_name": collector_name, "task_id": existing_running.id},
)
await emit_business_log(
logger,
event="collector.run.skipped_already_running",
message="Skipping collector trigger because task is already running",
category="collector",
level="warning",
service="scheduler",
module=__name__,
context={
"collector_name": collector_name,
"datasource_id": datasource.id,
"task_id": existing_running.id,
"status": "skipped",
},
)
return
existing_error = (existing_running.error_message or "").strip()
@@ -173,6 +198,21 @@ async def run_collector_task(collector_name: str):
event="collector.run.stale_task_failed",
context={"collector_name": collector_name, "task_id": existing_running.id},
)
await emit_business_log(
logger,
event="collector.run.stale_task_failed",
message="Marked stale running task as failed before rerun",
category="collector",
level="warning",
service="scheduler",
module=__name__,
context={
"collector_name": collector_name,
"datasource_id": datasource.id,
"task_id": existing_running.id,
"status": "failed",
},
)
try:
datasource_id = datasource.id
@@ -183,6 +223,15 @@ async def run_collector_task(collector_name: str):
event="collector.run.started",
context={"collector_name": collector_name, "datasource_id": datasource_id},
)
await emit_business_log(
logger,
event="collector.run.scheduled_started",
message="Scheduler started collector run",
category="collector",
service="scheduler",
module=__name__,
context={"collector_name": collector_name, "datasource_id": datasource_id, "status": "running"},
)
task_result = await collector.run(db)
datasource = await db.get(DataSource, datasource_id)
if datasource is None:
@@ -217,6 +266,20 @@ async def run_collector_task(collector_name: str):
event="collector.run.completed",
context={"collector_name": collector_name, "datasource_id": datasource_id, "result": task_result},
)
await emit_business_log(
logger,
event="collector.run.scheduled_completed",
message="Scheduler completed collector run",
category="collector",
service="scheduler",
module=__name__,
context={
"collector_name": collector_name,
"datasource_id": datasource_id,
"status": task_result.get("status"),
"result": task_result,
},
)
except asyncio.CancelledError:
await db.rollback()
datasource = await db.get(DataSource, datasource_id)
@@ -228,6 +291,16 @@ async def run_collector_task(collector_name: str):
event="collector.run.cancelled",
context={"collector_name": collector_name, "datasource_id": datasource.id},
)
await emit_business_log(
logger,
event="collector.run.cancelled",
message="Collector cancelled by operator",
category="collector",
level="warning",
service="scheduler",
module=__name__,
context={"collector_name": collector_name, "datasource_id": datasource.id, "status": "cancelled"},
)
raise
except Exception as exc:
await db.rollback()
@@ -240,6 +313,19 @@ async def run_collector_task(collector_name: str):
event="collector.run.failed",
context={"collector_name": collector_name, "datasource_id": datasource.id, "error": str(exc)},
)
await emit_business_log(
logger,
event="collector.run.failed",
message="Collector failed",
category="collector",
level="error",
service="scheduler",
module=__name__,
context=exception_context(
exc,
{"collector_name": collector_name, "datasource_id": datasource.id, "status": "failed"},
),
)
async def cleanup_stale_running_tasks(max_age_hours: int = 2) -> int:
@@ -361,6 +447,16 @@ def run_collector_now(collector_name: str) -> bool:
event="collector.trigger.skipped_already_running",
context={"collector_name": collector_name},
)
emit_business_log_background(
logger,
event="collector.trigger.skipped_already_running",
message="Collector is already running in-memory; skipping duplicate trigger",
category="collector",
level="warning",
service="scheduler",
module=__name__,
context={"collector_name": collector_name, "status": "skipped"},
)
return False
try:
@@ -378,6 +474,15 @@ def run_collector_now(collector_name: str) -> bool:
event="collector.trigger.started",
context={"collector_name": collector_name},
)
emit_business_log_background(
logger,
event="collector.trigger.started",
message="Triggered collector",
category="collector",
service="scheduler",
module=__name__,
context={"collector_name": collector_name, "status": "queued"},
)
return True
except Exception as exc:
logger.error_event(
@@ -385,6 +490,16 @@ def run_collector_now(collector_name: str) -> bool:
event="collector.trigger.failed",
context={"collector_name": collector_name, "error": str(exc)},
)
emit_business_log_background(
logger,
event="collector.trigger.failed",
message="Failed to trigger collector",
category="collector",
level="error",
service="scheduler",
module=__name__,
context=exception_context(exc, {"collector_name": collector_name, "status": "failed"}),
)
return False

View File

@@ -18,7 +18,7 @@ async def create_admin():
if existing_user:
print("用户 linkong 已存在,更新密码...")
existing_user.set_password("12345678")
existing_user.set_password("LK12345678")
existing_user.role = "super_admin"
existing_user.email = "linkong@planet.local"
else:
@@ -26,7 +26,7 @@ async def create_admin():
user = User(
username="linkong",
email="linkong@planet.local",
password_hash=get_password_hash("12345678"),
password_hash=get_password_hash("LK12345678"),
role="super_admin",
is_active=True,
)

View File

@@ -19,7 +19,7 @@ DEFAULT_LOGIN_USERS = (
{
"username": "linkong",
"email": "linkong@planet.local",
"password": "12345678",
"password": "LK12345678",
"role": "super_admin",
},
)

View File

@@ -22,7 +22,7 @@ DEFAULT_LOGIN_USERS = (
{
"username": "linkong",
"email": "linkong@planet.local",
"password": "12345678",
"password": "LK12345678",
"role": "super_admin",
},
)

View File

@@ -0,0 +1,128 @@
import pytest
from fastapi import HTTPException
from app.schemas.ai import SituationalAnalysisRequest
from app.services.ai_tools import web_search as web_search_module
from app.services.ai_tools.schemas import SearchEvidence, WebSearchConfig, WebSearchProviderConfig
from app.services.ai_tools.web_search import WebSearchClient
from app.services import ai_client as ai_client_module
from app.services.ai_client import AIProviderClient
@pytest.mark.asyncio
async def test_ai_client_analyze_logs_summary_without_prompt(monkeypatch):
events = []
async def fake_emit_business_log(_logger, **payload):
events.append(payload)
async def fake_request(self, method, path, json=None, request_id=None, operation="request", payload_summary=None):
return {
"provider": "test-provider",
"model": "test-model",
"content": "ok",
"content_blocks": [],
"text_blocks": ["ok"],
"thinking_blocks": [],
"raw_response": {},
}
monkeypatch.setattr(ai_client_module, "emit_business_log", fake_emit_business_log)
monkeypatch.setattr(AIProviderClient, "_request", fake_request)
client = AIProviderClient(
service_url="http://provider.test",
llm_config={"provider": "openai", "provider_api": "openai-completions", "model": "gpt-test", "api_key": "sk-secret"},
)
result = await client.analyze(
SituationalAnalysisRequest(
title="Sensitive title",
objective="Do not store this full prompt",
observations=["secret observation"],
constraints=["secret constraint"],
context={"source": "test", "private": "value"},
),
request_id="req-ai-test",
)
assert result.model == "test-model"
assert [event["event"] for event in events] == [
"ai.provider.analyze.start",
"ai.provider.analyze.success",
]
serialized = str(events)
assert "Do not store this full prompt" not in serialized
assert "secret observation" not in serialized
assert "sk-secret" not in serialized
start_context = events[0]["context"]
assert start_context["model"] == "gpt-test"
assert start_context["input_summary"]["objective_length"] == len("Do not store this full prompt")
assert start_context["input_summary"]["observation_count"] == 1
assert start_context["input_summary"]["context_keys"] == ["private", "source"]
@pytest.mark.asyncio
async def test_ai_client_analyze_logs_failure(monkeypatch):
events = []
async def fake_emit_business_log(_logger, **payload):
events.append(payload)
async def fake_request(self, method, path, json=None, request_id=None, operation="request", payload_summary=None):
raise HTTPException(status_code=502, detail="provider failed")
monkeypatch.setattr(ai_client_module, "emit_business_log", fake_emit_business_log)
monkeypatch.setattr(AIProviderClient, "_request", fake_request)
client = AIProviderClient(service_url="http://provider.test", llm_config={"provider": "openai", "model": "gpt-test"})
with pytest.raises(HTTPException):
await client.analyze(
SituationalAnalysisRequest(title="T", objective="O", observations=["one"]),
request_id="req-ai-fail",
)
assert events[-1]["event"] == "ai.provider.analyze.failed"
assert events[-1]["level"] == "error"
assert events[-1]["context"]["error_type"] == "HTTPException"
@pytest.mark.asyncio
async def test_web_search_logs_query_hash_without_query(monkeypatch):
events = []
async def fake_emit_business_log(_logger, **payload):
events.append(payload)
async def fake_search_tavily(self, config, query, max_results, domains, freshness_days):
return [
SearchEvidence(
title="Example",
url="https://example.test",
snippet="result",
source_provider="tavily",
)
]
monkeypatch.setattr(web_search_module, "emit_business_log", fake_emit_business_log)
monkeypatch.setattr(WebSearchClient, "_search_tavily", fake_search_tavily)
client = WebSearchClient(
WebSearchConfig(
enabled=True,
default_provider="tavily",
providers={"tavily": WebSearchProviderConfig(provider="tavily", api_key="secret-key")},
)
)
results = await client.search("secret query text", max_results=1)
assert len(results) == 1
assert [event["event"] for event in events] == [
"ai_tool.web_search.start",
"ai_tool.web_search.success",
]
serialized = str(events)
assert "secret query text" not in serialized
assert "secret-key" not in serialized
assert events[0]["context"]["query_length"] == len("secret query text")
assert events[1]["context"]["result_count"] == 1

View File

@@ -1,9 +1,13 @@
"""Unit tests for data collectors"""
import json
import pytest
from unittest.mock import AsyncMock, patch
from app.core.datasource_defaults import DEFAULT_DATASOURCES
from app.services.collectors.celestrak import CelesTrakTLECollector
from app.services.collectors.downloads import DownloadHTTPStatusError, ResumableFileDownloader
from app.services.credential_guides import DEFAULT_CREDENTIAL_GUIDES
from app.services.collectors.top500 import TOP500Collector
from app.services.collectors.registry import collector_registry
@@ -149,6 +153,172 @@ class TestHTTPCollector:
assert callable(collector.parse_response)
class TestCelesTrakTLECollector:
def test_transform_uses_norad_as_source_id_and_preserves_starlink_group(self):
collector = CelesTrakTLECollector()
result = collector.transform([
{
"NORAD_CAT_ID": 44720,
"OBJECT_NAME": "STARLINK-1000",
"OBJECT_ID": "2019-029AZ",
"EPOCH": "2026-03-13T00:00:00Z",
"MEAN_MOTION": 15.79234567,
"ECCENTRICITY": 0.0001234,
"INCLINATION": 53.0,
"RA_OF_ASC_NODE": 10.0,
"ARG_OF_PERICENTER": 20.0,
"MEAN_ANOMALY": 30.0,
"_celestrak_query_group": "active",
"_celestrak_source_url": "https://celestrak.example/gp.php?GROUP=active&FORMAT=json",
}
])
assert result[0]["source_id"] == "44720"
assert result[0]["metadata"]["constellation_group"] == "starlink"
assert result[0]["metadata"]["celestrak_query_group"] == "active"
assert result[0]["metadata"]["norad_cat_id"] == 44720
assert result[0]["metadata"]["tle_line1"]
assert result[0]["metadata"]["tle_line2"]
def test_load_active_payload_rejects_invalid_records(self, tmp_path):
collector = CelesTrakTLECollector()
payload_path = tmp_path / "active.json"
payload_path.write_text(json.dumps([{"OBJECT_NAME": "missing norad"}]), encoding="utf-8")
with pytest.raises(RuntimeError, match="invalid record"):
collector._load_active_payload(payload_path)
def test_load_active_payload_accepts_complete_array(self, tmp_path):
collector = CelesTrakTLECollector()
payload_path = tmp_path / "active.json"
payload_path.write_text(
json.dumps([{"NORAD_CAT_ID": 25544, "OBJECT_NAME": "ISS (ZARYA)"}]),
encoding="utf-8",
)
records = collector._load_active_payload(payload_path)
assert records == [{"NORAD_CAT_ID": 25544, "OBJECT_NAME": "ISS (ZARYA)"}]
@pytest.mark.asyncio
async def test_fetch_retries_and_raises_instead_of_returning_partial_data(self, monkeypatch, tmp_path):
collector = CelesTrakTLECollector()
collector._resolved_url = "https://celestrak.example/NORAD/elements/gp.php"
attempts = 0
async def fake_download_file(*args, **kwargs):
nonlocal attempts
attempts += 1
raise RuntimeError("network interrupted")
async def fake_emit_business_log(*args, **kwargs):
return None
monkeypatch.setattr(collector._downloader, "download_file", fake_download_file)
monkeypatch.setattr("app.services.collectors.celestrak.emit_business_log", fake_emit_business_log)
monkeypatch.setattr("app.services.collectors.celestrak.asyncio.sleep", AsyncMock())
with pytest.raises(RuntimeError, match="failed after retries"):
await collector.fetch()
assert attempts == 3
@pytest.mark.asyncio
async def test_fetch_uses_cache_when_celestrak_reports_not_updated(self, monkeypatch, tmp_path):
collector = CelesTrakTLECollector()
collector._resolved_url = "https://celestrak.example/NORAD/elements/gp.php"
collector._downloader = ResumableFileDownloader(cache_namespace="celestrak-test", cache_root=tmp_path)
url = collector._active_url()
cached_path = collector._downloader.cached_file_path(url, ".json")
cached_path.parent.mkdir(parents=True, exist_ok=True)
cached_path.write_text(
json.dumps([{"NORAD_CAT_ID": 25544, "OBJECT_NAME": "ISS (ZARYA)"}]),
encoding="utf-8",
)
async def fake_download_file(*args, **kwargs):
raise DownloadHTTPStatusError(
url=url,
status_code=403,
body="GP data has not updated since your last successful download of GROUP=active.",
)
async def fake_emit_business_log(*args, **kwargs):
return None
monkeypatch.setattr(collector._downloader, "download_file", fake_download_file)
monkeypatch.setattr("app.services.collectors.celestrak.emit_business_log", fake_emit_business_log)
records = await collector.fetch()
assert records[0]["NORAD_CAT_ID"] == 25544
assert records[0]["_celestrak_query_group"] == "active"
@pytest.mark.asyncio
async def test_fetch_not_updated_without_cache_does_not_retry(self, monkeypatch, tmp_path):
collector = CelesTrakTLECollector()
collector._resolved_url = "https://celestrak.example/NORAD/elements/gp.php"
collector._downloader = ResumableFileDownloader(cache_namespace="celestrak-test", cache_root=tmp_path)
attempts = 0
async def fake_download_file(*args, **kwargs):
nonlocal attempts
attempts += 1
raise DownloadHTTPStatusError(
url=collector._active_url(),
status_code=403,
body="GP data has not updated since your last successful download of GROUP=active.",
)
async def fake_emit_business_log(*args, **kwargs):
return None
monkeypatch.setattr(collector._downloader, "download_file", fake_download_file)
monkeypatch.setattr("app.services.collectors.celestrak.FALLBACK_GROUPS", ("starlink",))
monkeypatch.setattr("app.services.collectors.celestrak.emit_business_log", fake_emit_business_log)
monkeypatch.setattr("app.services.collectors.celestrak.asyncio.sleep", AsyncMock())
with pytest.raises(RuntimeError, match="fallback group mode failed"):
await collector.fetch()
assert attempts == 2
@pytest.mark.asyncio
async def test_fetch_falls_back_to_all_groups_when_active_not_updated_without_cache(self, monkeypatch, tmp_path):
collector = CelesTrakTLECollector()
collector._resolved_url = "https://celestrak.example/NORAD/elements/gp.php"
collector._downloader = ResumableFileDownloader(cache_namespace="celestrak-test", cache_root=tmp_path)
payload_by_group = {
"starlink": [{"NORAD_CAT_ID": 100, "OBJECT_NAME": "STARLINK-100"}],
"gps-ops": [{"NORAD_CAT_ID": 200, "OBJECT_NAME": "GPS BIIR-2"}],
}
async def fake_download_file(_client, url, **_kwargs):
if "GROUP=active" in url:
raise DownloadHTTPStatusError(
url=url,
status_code=403,
body="GP data has not updated since your last successful download of GROUP=active.",
)
group = "starlink" if "GROUP=starlink" in url else "gps-ops"
path = tmp_path / f"{group}.json"
path.write_text(json.dumps(payload_by_group[group]), encoding="utf-8")
return path
async def fake_emit_business_log(*args, **kwargs):
return None
monkeypatch.setattr(collector._downloader, "download_file", fake_download_file)
monkeypatch.setattr("app.services.collectors.celestrak.FALLBACK_GROUPS", tuple(payload_by_group))
monkeypatch.setattr("app.services.collectors.celestrak.emit_business_log", fake_emit_business_log)
records = await collector.fetch()
assert [item["NORAD_CAT_ID"] for item in records] == [100, 200]
assert records[0]["_celestrak_query_group"] == "starlink"
assert records[1]["_celestrak_group"] == "gps-ops"
def test_aisstream_collector_is_registered():
collector = collector_registry.get("aisstream_vessels")

View File

@@ -1,9 +1,11 @@
import asyncio
import inspect
from datetime import datetime, timedelta, timezone
import pytest
from app.api.v1 import datasources as datasources_api
from app.models.datasource import DataSource
from app.models.task import CollectionTask
from app.services import data_jobs
from app.services import earth_layer_cache as earth_cache
@@ -100,6 +102,71 @@ def test_serialize_datasource_row_includes_endpoint_when_requested():
assert row["endpoint"] == "https://example.test/arcgis_landing_points"
def test_serialize_datasource_row_separates_collector_running_from_delete_task():
datasource = make_datasource(9, "top500", last_status="success")
task = CollectionTask(
id=44,
datasource_id=9,
source="top500",
task_type="clear_data",
status="running",
phase="clearing_data",
)
row = datasources_api.serialize_datasource_row(
datasource,
running_tasks={9: task},
latest_tasks={9: task},
record_counts={"top500": 500},
endpoint_overrides={},
config=object(),
include_endpoint=False,
)
assert row["is_task_active"] is True
assert row["is_running"] is False
assert row["task_type"] == "clear_data"
assert row["task_status"] == "running"
def test_cancel_queued_delete_task_finishes_immediately(monkeypatch):
task = CollectionTask(
id=45,
datasource_id=9,
source="top500",
task_type="clear_data",
status="queued",
phase="queued",
)
class FakeDb:
async def commit(self):
return None
async def refresh(self, _task):
return None
async def fake_broadcast(_task):
return None
monkeypatch.setattr(data_jobs, "_broadcast_task_update", fake_broadcast)
async def run():
return await data_jobs.request_cancel_datasource_task(FakeDb(), task)
result = asyncio.run(run())
assert result.status == "cancelled"
assert result.phase == "cancelled"
assert result.completed_at is not None
def test_clear_data_job_relies_on_db_outbox_instead_of_extra_earth_refresh_task():
source = inspect.getsource(data_jobs._run_clear_data_job)
assert "enqueue_earth_refresh_job" not in source
def test_invalidate_earth_layer_cache_for_source_covers_datasource_aliases(monkeypatch):
patterns: list[str] = []
@@ -119,34 +186,36 @@ def test_invalidate_earth_layer_cache_for_source_covers_datasource_aliases(monke
]
@pytest.mark.asyncio
async def test_trigger_datasource_batch_skips_disabled_and_frequency_window(monkeypatch):
now = datetime.now(timezone.utc)
disabled = make_datasource(1, "aisstream_vessels", is_active=False)
not_due = make_datasource(2, "telegeography_cables", last_run_at=now, frequency_minutes=120)
due = make_datasource(3, "ris_live_bgp", last_run_at=now - timedelta(hours=2))
triggered_sources: list[str] = []
def test_trigger_datasource_batch_skips_disabled_and_frequency_window(monkeypatch):
async def run():
now = datetime.now(timezone.utc)
disabled = make_datasource(1, "aisstream_vessels", is_active=False)
not_due = make_datasource(2, "telegeography_cables", last_run_at=now, frequency_minutes=120)
due = make_datasource(3, "ris_live_bgp", last_run_at=now - timedelta(hours=2))
queued_sources: list[str] = []
async def fake_running_tasks(_db, _ids):
return {}
async def fake_running_tasks(_db, _ids):
return {}
async def fake_latest_task_ids(_db, _ids):
return {}
async def fake_enqueue(_db, datasource, task_type, **_kwargs):
queued_sources.append(datasource.source)
return CollectionTask(id=100 + datasource.id, datasource_id=datasource.id, source=datasource.source, task_type=task_type, status="queued")
monkeypatch.setattr(datasources_api, "_load_latest_running_tasks", fake_running_tasks)
monkeypatch.setattr(datasources_api, "_load_latest_task_ids", fake_latest_task_ids)
monkeypatch.setattr(
datasources_api,
"run_collector_now",
lambda source: triggered_sources.append(source) or True,
)
monkeypatch.setattr(datasources_api, "_load_latest_running_tasks", fake_running_tasks)
monkeypatch.setattr(
datasources_api,
"enqueue_datasource_job",
fake_enqueue,
)
result = await datasources_api._trigger_datasource_batch(
object(),
[disabled, not_due, due],
force=False,
)
result = await datasources_api._trigger_datasource_batch(
object(),
[disabled, not_due, due],
force=False,
)
assert [item["source"] for item in result["triggered"]] == ["ris_live_bgp"]
assert {item["reason"] for item in result["skipped"]} == {"disabled", "within_frequency_window"}
assert triggered_sources == ["ris_live_bgp"]
assert [item["source"] for item in result["triggered"]] == ["ris_live_bgp"]
assert {item["reason"] for item in result["skipped"]} == {"disabled", "within_frequency_window"}
assert queued_sources == ["ris_live_bgp"]
asyncio.run(run())

View File

@@ -1,11 +1,15 @@
"""Docs Gatekeeper API tests."""
import re
from pathlib import Path
import pytest
from httpx import ASGITransport, AsyncClient
from app.api.v1 import docs as docs_api
from app.main import app
from app.models.user import User
from app.services.docs_gatekeeper import DOCS_METADATA
def make_user(role: str = "viewer", groups: list[str] | None = None) -> User:
@@ -43,22 +47,45 @@ async def test_public_catalog_only_for_anonymous_user():
assert response.status_code == 200
items = response.json()["items"]
assert {item["access"] for item in items} == {"public"}
assert {item["slug"] for item in items if item["lang"] == "zh"} == {
zh_items = [item for item in items if item["lang"] == "zh"]
assert [item["slug"] for item in zh_items] == [
"overview",
"quickstart",
"manual",
"quickstart",
"faq",
"location-pipeline-user",
}
]
@pytest.mark.asyncio
async def test_developer_catalog_includes_architecture_and_frontend_reference_docs():
response = await get_json(
"/api/v1/docs/catalog",
make_user(role="viewer", groups=["docs_developer"]),
)
assert response.status_code == 200
zh_slugs = {item["slug"] for item in response.json()["items"] if item["lang"] == "zh"}
zh_items = [item for item in response.json()["items"] if item["lang"] == "zh"]
assert [item["group"] for item in zh_items[:4]] == ["Overview", "Manual", "Manual", "Manual"]
assert zh_items[4]["group"] == "Architecture"
assert "platform-data-flows" in zh_slugs
assert "naming-glossary" in zh_slugs
assert "tactile-ui-components" in zh_slugs
@pytest.mark.asyncio
async def test_anonymous_can_read_public_doc():
response = await get_json("/api/v1/docs/zh/quickstart")
manual_response = await get_json("/api/v1/docs/zh/manual")
overview_response = await get_json("/api/v1/docs/zh/overview")
assert response.status_code == 200
assert response.json()["access"] == "public"
assert "快速开始" in response.json()["markdown"]
assert manual_response.status_code == 200
assert manual_response.json()["access"] == "public"
assert overview_response.status_code == 200
assert overview_response.json()["access"] == "public"
@pytest.mark.asyncio
@@ -83,10 +110,16 @@ async def test_developer_group_can_read_developer_but_not_admin_doc():
user = make_user(role="viewer", groups=["docs_developer"])
developer_response = await get_json("/api/v1/docs/zh/backend-collectors", user)
tactile_response = await get_json("/api/v1/docs/zh/tactile-ui-components", user)
glossary_response = await get_json("/api/v1/docs/zh/naming-glossary", user)
admin_response = await get_json("/api/v1/docs/zh/backend-system-service-control", user)
assert developer_response.status_code == 200
assert developer_response.json()["access"] == "docs_developer"
assert tactile_response.status_code == 200
assert tactile_response.json()["access"] == "docs_developer"
assert glossary_response.status_code == 200
assert glossary_response.json()["access"] == "docs_developer"
assert admin_response.status_code == 403
@@ -114,3 +147,29 @@ async def test_unknown_language_slug_and_path_traversal_do_not_read_files():
assert bad_lang.status_code == 404
assert bad_slug.status_code == 404
assert traversal.status_code == 404
def test_public_docs_markdown_links_do_not_create_missing_docs_routes():
repo_root = Path(__file__).resolve().parents[2]
technical_root = repo_root / "docs" / "technical"
registered_filenames = {entry.filename for entry in DOCS_METADATA}
problems: list[str] = []
for markdown_path in sorted(technical_root.glob("*/*.md")):
markdown = markdown_path.read_text(encoding="utf-8")
for match in re.finditer(r"\[([^\]]+)]\(([^)]+\.md(?:#[^)]+)?)\)", markdown):
label, href = match.group(1), match.group(2)
if href.startswith(("http://", "https://", "mailto:")):
continue
href_without_hash = href.split("#", 1)[0].replace("\\", "/")
filename = Path(href_without_hash).name
if "/docs/technical/" in href_without_hash:
if filename not in registered_filenames:
problems.append(f"{markdown_path.relative_to(repo_root)} links unregistered public doc {href!r} ({label})")
continue
if href_without_hash.endswith(".md"):
problems.append(f"{markdown_path.relative_to(repo_root)} links non-public markdown {href!r} ({label})")
assert problems == []

View File

@@ -0,0 +1,454 @@
import asyncio
import json
from app.services.earth_db_change_listener import (
EarthDbChangeDispatcher,
EarthDbChangeListener,
build_earth_update_from_db_payload,
normalize_asyncpg_dsn,
)
def test_normalize_asyncpg_dsn_strips_sqlalchemy_driver():
assert (
normalize_asyncpg_dsn("postgresql+asyncpg://postgres:postgres@localhost:5432/planet_db")
== "postgresql://postgres:postgres@localhost:5432/planet_db"
)
def test_build_earth_update_maps_known_sources_to_layers():
satellite_update = build_earth_update_from_db_payload(
{"source": "celestrak_tle", "operation": "DELETE", "entity_key": "25544"}
)
cable_update = build_earth_update_from_db_payload(
{"source": "arcgis_cables", "operation": "UPDATE", "entity_key": "cable-1"}
)
compute_update = build_earth_update_from_db_payload(
{"source": "top500", "operation": "DELETE", "entity_key": None}
)
landing_update = build_earth_update_from_db_payload(
{"source": "telegeography_landing", "operation": "DELETE", "entity_key": None}
)
assert satellite_update is not None
assert satellite_update["layers"] == ["satellites"]
assert cable_update is not None
assert cable_update["layers"] == ["cables"]
assert compute_update is not None
assert compute_update["layers"] == ["computeCenters"]
assert landing_update is not None
assert landing_update["layers"] == ["cables"]
assert landing_update["refresh_strategy"] == "clear_then_reload"
def test_build_earth_update_maps_derived_tables_to_layers():
bgp_update = build_earth_update_from_db_payload(
{"table": "bgp_anomalies", "source": "ris_live_bgp", "operation": "DELETE", "entity_key": "a1"}
)
compute_update = build_earth_update_from_db_payload(
{"table": "compute_center_locations", "source": "top500", "operation": "UPDATE", "entity_key": "top500_1"}
)
vessel_update = build_earth_update_from_db_payload(
{"table": "vessel_position", "operation": "DELETE", "entity_key": "123456789"}
)
assert bgp_update is not None
assert bgp_update["source"] == "ris_live_bgp"
assert bgp_update["table"] == "bgp_anomalies"
assert bgp_update["layers"] == ["bgp"]
assert bgp_update["refresh_strategy"] == "clear_then_reload"
assert compute_update is not None
assert compute_update["layers"] == ["computeCenters"]
assert compute_update["refresh_strategy"] == "reload"
assert vessel_update is not None
assert vessel_update["source"] == "vessel_position"
assert vessel_update["layers"] == ["vessels"]
def test_build_earth_update_maps_interactable_delete_to_delta():
update = build_earth_update_from_db_payload(
{
"table": "earth_interactables",
"operation": "DELETE",
"entity_keys": ["note-1"],
"records_processed": 1,
}
)
assert update is not None
assert update["entity"] == "interactable"
assert update["action"] == "deleted"
assert update["ids"] == ["note-1"]
assert update["layers"] == ["interactables"]
assert update["refresh_strategy"] == "delta"
def test_build_earth_update_ignores_unmapped_sources():
assert build_earth_update_from_db_payload({"source": "not_for_earth"}) is None
def test_dispatcher_debounces_same_source_notifications():
async def run():
broadcasts = []
invalidated = []
async def broadcast(payload):
broadcasts.append(payload)
def invalidate(source):
invalidated.append(source)
return 2
dispatcher = EarthDbChangeDispatcher(
broadcast_earth_update=broadcast,
invalidate_cache=invalidate,
debounce_seconds=10,
)
dispatcher.handle_payload(
{
"table": "collected_data",
"operation": "INSERT",
"source": "arcgis_cables",
"entity_key": "cable-1",
"occurred_at": "2026-05-22T00:00:00Z",
}
)
dispatcher.handle_payload(
{
"table": "collected_data",
"operation": "UPDATE",
"source": "arcgis_cables",
"entity_key": "cable-2",
"occurred_at": "2026-05-22T00:00:01Z",
}
)
await dispatcher.flush_all()
assert invalidated == ["arcgis_cables"]
assert len(broadcasts) == 1
assert broadcasts[0]["action"] == "database_changed"
assert broadcasts[0]["source"] == "arcgis_cables"
assert broadcasts[0]["layers"] == ["cables"]
assert broadcasts[0]["records_processed"] == 2
assert broadcasts[0]["operations"] == ["INSERT", "UPDATE"]
assert broadcasts[0]["entity_keys"] == ["cable-1", "cable-2"]
assert broadcasts[0]["cache_entries_invalidated"] == 2
asyncio.run(run())
def test_dispatcher_handles_delete_notification_as_earth_update():
async def run():
broadcasts = []
async def broadcast(payload):
broadcasts.append(payload)
dispatcher = EarthDbChangeDispatcher(
broadcast_earth_update=broadcast,
invalidate_cache=lambda source: 1,
debounce_seconds=10,
)
accepted = dispatcher.handle_notification(
json.dumps(
{
"table": "collected_data",
"operation": "DELETE",
"source": "celestrak_tle",
"entity_key": "sat:25544",
"occurred_at": "2026-05-22T00:00:00Z",
}
)
)
await dispatcher.flush_all()
assert accepted is True
assert len(broadcasts) == 1
assert broadcasts[0]["action"] == "database_changed"
assert broadcasts[0]["source"] == "celestrak_tle"
assert broadcasts[0]["layers"] == ["satellites"]
assert broadcasts[0]["operation"] == "DELETE"
assert broadcasts[0]["refresh_strategy"] == "clear_then_reload"
asyncio.run(run())
def test_dispatcher_debounces_bgp_derived_table_events_by_source():
async def run():
broadcasts = []
async def broadcast(payload):
broadcasts.append(payload)
dispatcher = EarthDbChangeDispatcher(
broadcast_earth_update=broadcast,
invalidate_cache=lambda source: 0,
debounce_seconds=10,
)
dispatcher.handle_payload(
{
"event_id": 200,
"table": "bgp_observations",
"operation": "DELETE",
"source": "ris_live_bgp",
"records_processed": 2,
}
)
dispatcher.handle_payload(
{
"event_id": 201,
"table": "bgp_anomalies",
"operation": "DELETE",
"source": "ris_live_bgp",
"records_processed": 3,
}
)
await dispatcher.flush_all()
assert len(broadcasts) == 1
assert broadcasts[0]["source"] == "ris_live_bgp"
assert broadcasts[0]["layers"] == ["bgp"]
assert broadcasts[0]["records_processed"] == 5
assert broadcasts[0]["refresh_strategy"] == "clear_then_reload"
asyncio.run(run())
def test_dispatcher_deduplicates_notify_and_outbox_by_event_id():
async def run():
broadcasts = []
async def broadcast(payload):
broadcasts.append(payload)
dispatcher = EarthDbChangeDispatcher(
broadcast_earth_update=broadcast,
invalidate_cache=lambda source: 0,
debounce_seconds=10,
)
payload = {
"event_id": 42,
"table": "collected_data",
"operation": "INSERT",
"source": "celestrak_tle",
"entity_key": "sat:42",
"occurred_at": "2026-05-23T00:00:00Z",
}
assert dispatcher.handle_payload(payload) is True
assert dispatcher.handle_payload(dict(payload)) is False
await dispatcher.flush_all()
assert len(broadcasts) == 1
assert broadcasts[0]["records_processed"] == 1
assert broadcasts[0]["entity_keys"] == ["sat:42"]
asyncio.run(run())
def test_dispatcher_flushes_continuous_events_at_max_wait():
async def run():
broadcasts = []
sleeps = []
async def broadcast(payload):
broadcasts.append(payload)
dispatcher = EarthDbChangeDispatcher(
broadcast_earth_update=broadcast,
invalidate_cache=lambda source: 0,
debounce_seconds=10,
max_wait_seconds=0.01,
)
dispatcher.handle_payload(
{
"event_id": 300,
"table": "collected_data",
"operation": "INSERT",
"source": "arcgis_cables",
"records_processed": 1,
}
)
await asyncio.sleep(0.02)
dispatcher.handle_payload(
{
"event_id": 301,
"table": "collected_data",
"operation": "UPDATE",
"source": "arcgis_cables",
"records_processed": 1,
}
)
await asyncio.sleep(0)
await dispatcher.flush_all()
assert len(broadcasts) == 1
assert broadcasts[0]["records_processed"] == 2
assert broadcasts[0]["debounce_ms"] >= 0
asyncio.run(run())
def test_dispatcher_fast_flushes_delete_clear_then_reload():
async def run():
broadcasts = []
async def broadcast(payload):
broadcasts.append(payload)
dispatcher = EarthDbChangeDispatcher(
broadcast_earth_update=broadcast,
invalidate_cache=lambda source: 0,
debounce_seconds=10,
max_wait_seconds=10,
)
dispatcher.handle_payload(
{
"event_id": 310,
"table": "collected_data",
"operation": "DELETE",
"source": "celestrak_tle",
"records_processed": 10,
}
)
await asyncio.sleep(0.08)
assert len(broadcasts) == 1
assert broadcasts[0]["source"] == "celestrak_tle"
assert broadcasts[0]["refresh_strategy"] == "clear_then_reload"
asyncio.run(run())
def test_outbox_rows_are_consumed_after_successful_flush():
async def run():
class FakeConnection:
def __init__(self):
self.consumed_ids = []
async def fetch(self, _query, _limit):
return [
{
"id": 501,
"payload": {
"event_id": 501,
"table": "collected_data",
"operation": "DELETE",
"source": "celestrak_tle",
"records_processed": 1,
},
}
]
async def execute(self, _query, consumed_ids):
self.consumed_ids.extend(consumed_ids)
broadcasts = []
async def broadcast(payload):
broadcasts.append(payload)
dispatcher = EarthDbChangeDispatcher(
broadcast_earth_update=broadcast,
invalidate_cache=lambda source: 0,
debounce_seconds=10,
)
listener = EarthDbChangeListener(
dsn="postgresql://example/db",
dispatcher=dispatcher,
)
connection = FakeConnection()
listener._connection = connection
await listener._poll_outbox()
assert connection.consumed_ids == [501]
assert len(broadcasts) == 1
asyncio.run(run())
def test_outbox_rows_remain_unconsumed_when_flush_fails():
async def run():
class FakeConnection:
def __init__(self):
self.consumed_ids = []
async def fetch(self, _query, _limit):
return [
{
"id": 601,
"payload": {
"event_id": 601,
"table": "collected_data",
"operation": "DELETE",
"source": "celestrak_tle",
"records_processed": 1,
},
}
]
async def execute(self, _query, consumed_ids):
self.consumed_ids.extend(consumed_ids)
async def broadcast(_payload):
raise RuntimeError("ws down")
dispatcher = EarthDbChangeDispatcher(
broadcast_earth_update=broadcast,
invalidate_cache=lambda source: 0,
debounce_seconds=10,
)
listener = EarthDbChangeListener(
dsn="postgresql://example/db",
dispatcher=dispatcher,
)
connection = FakeConnection()
listener._connection = connection
try:
await listener._poll_outbox()
except RuntimeError:
pass
assert connection.consumed_ids == []
asyncio.run(run())
def test_dispatcher_uses_aggregated_statement_record_count():
async def run():
broadcasts = []
async def broadcast(payload):
broadcasts.append(payload)
dispatcher = EarthDbChangeDispatcher(
broadcast_earth_update=broadcast,
invalidate_cache=lambda source: 0,
debounce_seconds=10,
)
dispatcher.handle_payload(
{
"event_id": 99,
"table": "collected_data",
"operation": "DELETE",
"source": "top500",
"records_processed": 100,
"entity_keys": ["top500:1", "top500:2"],
"occurred_at": "2026-05-23T00:00:00Z",
}
)
await dispatcher.flush_all()
assert len(broadcasts) == 1
assert broadcasts[0]["source"] == "top500"
assert broadcasts[0]["layers"] == ["computeCenters"]
assert broadcasts[0]["records_processed"] == 100
assert broadcasts[0]["entity_keys"] == ["top500:1", "top500:2"]
asyncio.run(run())

View File

@@ -0,0 +1,79 @@
from datetime import UTC, datetime
from app.models.earth_interactable import EarthInteractable
from app.services import earth_interactables
def make_interactable(**overrides):
values = {
"id": "poi-1",
"layer": "places",
"kind": "note",
"label": "Test POI",
"description": "A point on Earth",
"latitude": 30.25,
"longitude": 120.15,
"altitude": None,
"revision": 3,
"properties": {"owner": "test"},
"is_deleted": False,
"created_at": datetime(2026, 5, 22, 1, 0, tzinfo=UTC),
"updated_at": datetime(2026, 5, 22, 1, 5, tzinfo=UTC),
"deleted_at": None,
}
values.update(overrides)
return EarthInteractable(**values)
def test_interactable_event_uses_object_delta_contract():
record = make_interactable()
event = earth_interactables.build_interactable_event(
action="updated",
record=record,
)
assert event["entity"] == "interactable"
assert event["action"] == "updated"
assert event["layer"] == "places"
assert event["layers"] == ["interactables"]
assert event["ids"] == ["poi-1"]
assert event["revision"] == 3
assert event["item"]["latitude"] == 30.25
assert event["item"]["properties"] == {"owner": "test"}
def test_interactable_geojson_omits_deleted_records():
active = make_interactable(id="active")
deleted = make_interactable(id="deleted", is_deleted=True)
payload = earth_interactables.interactables_to_geojson([active, deleted])
assert payload["type"] == "FeatureCollection"
assert [feature["id"] for feature in payload["features"]] == ["active"]
assert payload["features"][0]["geometry"] == {
"type": "Point",
"coordinates": [120.15, 30.25],
}
def test_interactable_cache_invalidation_clears_layer_and_all(monkeypatch):
patterns = []
def fake_delete_pattern(pattern):
patterns.append(pattern)
return 1
monkeypatch.setattr(
earth_interactables.earth_layer_cache,
"delete_pattern",
fake_delete_pattern,
)
deleted = earth_interactables.invalidate_interactable_cache("places")
assert deleted == 2
assert patterns == [
"earth:layer:v1:interactables:layer:places*",
"earth:layer:v1:interactables:layer:all*",
]

View File

@@ -421,6 +421,62 @@ async def test_earth_news_payload_uses_fresh_database_items_without_rss(monkeypa
assert payload["stale"] is False
@pytest.mark.asyncio
async def test_earth_news_payload_keeps_current_items_and_all_cruise_items(monkeypatch):
class FakeDb:
execute = object()
current_item = ParsedNewsItem(
id="db:current",
title="Current region story",
summary="Current summary",
url="https://example.com/current",
source="Stored Source",
feed_name="Stored Feed",
feed_region="americas",
homepage_url="https://example.com",
published_at=datetime(2026, 5, 15, 3, 0, tzinfo=UTC),
)
cruise_item = ParsedNewsItem(
id="db:apac",
title="APAC story",
summary="APAC summary",
url="https://example.com/apac",
source="Stored Source",
feed_name="Stored Feed",
feed_region="asia-pacific",
homepage_url="https://example.com",
published_at=datetime(2026, 5, 15, 4, 0, tzinfo=UTC),
)
async def fake_get_earth_news_freshness(_db, *, active_region):
return 12, datetime.now(UTC)
async def fake_list_earth_news_items(_db, *, active_region, limit):
return [current_item]
async def fake_list_earth_news_cruise_items(_db, *, limit):
return [current_item, cruise_item]
async def fake_enqueue_target_location_job(_payload, **_kwargs):
return True
async def fail_fetch(_sources):
raise AssertionError("fresh database items should not fetch RSS")
monkeypatch.setattr("app.services.earth_news_store.get_earth_news_freshness", fake_get_earth_news_freshness)
monkeypatch.setattr("app.services.earth_news_store.list_earth_news_items", fake_list_earth_news_items)
monkeypatch.setattr("app.services.earth_news_store.list_earth_news_cruise_items", fake_list_earth_news_cruise_items)
monkeypatch.setattr("app.services.earth_news_queue.enqueue_target_location_job", fake_enqueue_target_location_job)
monkeypatch.setattr("app.services.earth_news._fetch_rss_items_for_sources", fail_fetch)
payload = await get_earth_news_payload(lat=35.0, lon=-100.0, db=FakeDb())
assert [item["id"] for item in payload["items"]] == ["db:current"]
assert [item["id"] for item in payload["cruise_items"]] == ["db:current", "db:apac"]
assert payload["cruise_items"][1]["region"] == "asia-pacific"
@pytest.mark.asyncio
async def test_earth_news_payload_initializes_empty_database_from_rss(monkeypatch):
db = object()

View File

@@ -4,8 +4,11 @@ import logging
from io import StringIO
import pytest
from app.core.logging import PlanetContextFilter, PlanetFormatter, get_logger
from app.core.request_context import set_request_id
from app.services import business_logs
def _capture_output(callback):
@@ -76,3 +79,43 @@ def test_structured_logger_redacts_sensitive_text_and_context():
assert "hunter2" not in output
assert "[REDACTED]" in output
assert '"safe": "visible"' in output
def test_business_context_redacts_nested_sensitive_values():
context = business_logs.build_business_context(
{
"provider": "openai",
"api_key": "sk-secret",
"nested": {
"token": "plain-token",
"safe": "visible",
},
}
)
assert context["api_key"] == "[REDACTED]"
assert context["nested"]["token"] == "[REDACTED]"
assert context["nested"]["safe"] == "visible"
@pytest.mark.asyncio
async def test_emit_business_log_persists_sanitized_system_event(monkeypatch):
events = []
async def fake_record_system_log(**payload):
events.append(payload)
monkeypatch.setattr(business_logs, "record_system_log", fake_record_system_log)
await business_logs.emit_business_log(
get_logger("tests.business"),
event="ai.provider.analyze.success",
message="AI request completed",
category="ai",
context={"model": "gpt-test", "api_key": "sk-secret"},
)
assert events[0]["event"] == "ai.provider.analyze.success"
assert events[0]["category"] == "ai"
assert events[0]["context"]["model"] == "gpt-test"
assert events[0]["context"]["api_key"] == "[REDACTED]"

View File

@@ -816,6 +816,27 @@ async def test_visualization_geo_summary_returns_counts(monkeypatch):
app.dependency_overrides.clear()
@pytest.mark.asyncio
async def test_empty_cable_layers_return_empty_feature_collections():
class _ScalarResult:
def scalars(self):
class _Scalars:
def all(self):
return []
return _Scalars()
class _FakeSession:
async def execute(self, _query):
return _ScalarResult()
cables = await visualization_api._build_cables_geojson(_FakeSession())
landing_points = await visualization_api._build_landing_points_geojson(_FakeSession())
assert cables == {"type": "FeatureCollection", "features": []}
assert landing_points == {"type": "FeatureCollection", "features": []}
@pytest.mark.asyncio
async def test_collect_location_endpoint_returns_candidates_for_known_record(monkeypatch):
def _fake_ror(query):

View File

@@ -15,12 +15,16 @@ services:
retries: 10
aiprovider:
image: ${AI_PROVIDER_IMAGE_NAME:-planet-aiprovider:latest}
build:
context: .
dockerfile: aiprovider/Dockerfile
args:
PYTHON_IMAGE: ${PYTHON_IMAGE:-python:3.14-slim}
UV_IMAGE: ${UV_IMAGE:-ghcr.io/astral-sh/uv:latest}
AI_PROVIDER_BUILD_FINGERPRINT: ${AI_PROVIDER_BUILD_FINGERPRINT:-unknown}
secrets:
- planet_uv_config
container_name: planet_aiprovider
ports:
- "8010:8010"
@@ -43,3 +47,7 @@ services:
volumes:
ollama_data:
secrets:
planet_uv_config:
file: ${PLANET_UV_CONFIG_FILE:-/dev/null}

View File

@@ -2,12 +2,16 @@ version: '3.8'
services:
aiprovider:
image: ${AI_PROVIDER_IMAGE_NAME:-planet-aiprovider:latest}
build:
context: .
dockerfile: aiprovider/Dockerfile
args:
PYTHON_IMAGE: ${PYTHON_IMAGE:-python:3.14-slim}
UV_IMAGE: ${UV_IMAGE:-ghcr.io/astral-sh/uv:latest}
AI_PROVIDER_BUILD_FINGERPRINT: ${AI_PROVIDER_BUILD_FINGERPRINT:-unknown}
secrets:
- planet_uv_config
env_file:
- ./aiprovider/.env
- ${PLANET_AI_PROVIDER_RUNTIME_ENV_FILE:-./aiprovider/.env}
@@ -43,3 +47,7 @@ services:
volumes:
postgres_data:
redis_data:
secrets:
planet_uv_config:
file: ${PLANET_UV_CONFIG_FILE:-/dev/null}

View File

@@ -2,12 +2,16 @@ version: '3.8'
services:
aiprovider:
image: ${AI_PROVIDER_IMAGE_NAME:-planet-aiprovider:latest}
build:
context: .
dockerfile: aiprovider/Dockerfile
args:
PYTHON_IMAGE: ${PYTHON_IMAGE:-python:3.14-slim}
UV_IMAGE: ${UV_IMAGE:-ghcr.io/astral-sh/uv:latest}
AI_PROVIDER_BUILD_FINGERPRINT: ${AI_PROVIDER_BUILD_FINGERPRINT:-unknown}
secrets:
- planet_uv_config
env_file:
- ./aiprovider/.env
- ${PLANET_AI_PROVIDER_RUNTIME_ENV_FILE:-./aiprovider/.env}
@@ -53,3 +57,7 @@ services:
volumes:
postgres_data:
redis_data:
secrets:
planet_uv_config:
file: ${PLANET_UV_CONFIG_FILE:-/dev/null}

View File

@@ -8,6 +8,165 @@ This project follows the repository versioning rule:
- `improvement` -> `+0.0.1`bugfix + 小功能混合)
- `bugfix` -> `+0.0.1`
## [0.66.3] — 2026-05-26
Released: 2026-05-26
### Highlights
- 修复控制台 Admin 动态导入时缺失 `../lib/utils` 导致 Vite 返回 500 的问题。
- 补齐前端依赖安装状态,确保 Markdown Mermaid 渲染器可解析 `mermaid` 包。
### Added / Fixed / Improved
- 新增 Admin 本地 `cn``formatNumber` 工具模块恢复布局、UI 组件和 Dashboard 的共享工具引用。
- 放开 `.gitignore``frontend/src/admin/lib` 的源码例外,避免 utility module 再次被全局 `lib/` 规则漏提交。
- 重启前端开发服务并验证 `/admin` 无 Vite overlay 和 console error。
---
## [0.66.2] — 2026-05-26
Released: 2026-05-26
### Highlights
- 统一中文界面和公开文档中的产品命名:`Earth` 显示为“智能星球”,`Admin` 显示为“控制台”,`Docs` 显示为“文档”。
- 修复文档手册、Docs catalog、控制台入口和智能星球设置中残留的中英混排标题。
### Added / Fixed / Improved
- 更新控制台侧边栏、智能星球系统设置、移动端抽屉和登录页品牌文案。
- 同步前后端 Docs metadata、中文技术文档标题、使用手册、快速开始、术语表和运维手册中的产品命名。
- 补充清理智能星球图层缓存、品牌配置、展示缓存等控制台 toast / dialog 文案。
---
## [0.66.1] — 2026-05-26
Released: 2026-05-26
### Highlights
- 修复 CelesTrak active 更新窗口内清库后无法恢复的问题,新增持久原始下载缓存和完整 fallback group mode。
- 避免数据源任务 WebSocket 与轮询同时完成时重复弹出采集失败 toast。
- `planet.sh destroy` 保留上游原始下载缓存,数据库清空后仍可用缓存重灌。
### Added / Fixed / Improved
- CelesTrak 403 `GP data has not updated` 会先复用 active 缓存;无 active 缓存时完整拉取 `starlink/gps-ops/galileo/glonass/beidou/leo/geo/iridium-next`,任一 group 缺失则整体失败,不保存 partial。
- 下载器缓存从 `/tmp` 迁到 `$PLANET_CACHE_DIR/downloads`,并保留 HTTP 错误响应正文以支持上游限频语义判断。
- 补充 CelesTrak cache/fallback 回归测试和中英文运维/卫星策略文档。
---
## [0.66.0] — 2026-05-26
Released: 2026-05-26
### Highlights
- 将 Admin 正式化为唯一控制台入口,移除旧 AntD 后台、Admin Next 路由痕迹和相关依赖。
- 引入 PostgreSQL 数据作业队列、Earth outbox 同步和可交互对象管线,让采集、清理和 Earth 刷新进入可追踪异步链路。
- 强化 AI、AI 工具、设置连接测试和采集任务的结构化业务日志,关键事件可在系统日志中检索。
- CelesTrak 轨道根数采集改为完整 `active` 目录下载、续传和重试,避免部分分组失败时保存不完整卫星数据。
### Added / Fixed / Improved
- Admin 数据源页修复内置源调度启停按钮,使用 `is_active` 判断 `/enable``/disable`,避免采集状态误导调度开关。
- Earth 新增通用 interactables 层和平台数据流文档,支持对象级 delta 同步与后续小图层扩展。
- `planet.sh init/start/destroy` 补齐 uv 镜像回退、Bun unzip 依赖、HTTPS/LAN 跳转协议和 OOBE 清理语义。
- Markdown code block、数据源队列、演示模式、About 版本展示、TV 链接协议和文档索引同步完成。
---
## [0.65.2] — 2026-05-22
Released: 2026-05-22
### Highlights
- Stabilize AI Provider rebuild detection so unchanged source no longer rebuilds just because file timestamps or local cache state changed.
- Keep `uv.lock` on the official registry while still allowing local and Docker builds to use the user's `uv.toml`.
- Prevent local startup and bootstrap commands from rewriting `uv.lock` when a user-level mirror is configured.
### Added / Fixed / Improved
- AI Provider fingerprints now use content hashes, base image inputs, dependency metadata, and an image label instead of file mtimes.
- `planet.sh` and `scripts/bootstrap-dev.sh` now use `uv sync --frozen`, and backend startup uses `uv run --frozen`.
- Docker Compose passes the AI Provider fingerprint into the image build so future starts can inspect the image label directly.
---
## [0.65.1] — 2026-05-22
Released: 2026-05-22
### Highlights
- 修复 `planet.sh` 与 Docker Compose 对 AI Provider 镜像名不一致导致的重复构建问题。
- 让本地 `uv` 与 Docker build 统一使用用户机器上的 `uv.toml`,避免把镜像源写入 `uv.lock`
- 通过 BuildKit secret 向容器构建传入 uv 配置,保留用户源选择且不把配置写入镜像层。
### Added / Fixed / Improved
- `planet.sh` 自动解析 `UV_CONFIG_FILE`、仓库 `uv.toml` 与用户级 `~/.config/uv/uv.toml`,并清理会覆盖配置的 `UV_INDEX_URL` 环境变量。
- AI Provider 和后端 Dockerfile 在 `uv sync` 阶段挂载 uv 配置 secret并复用 BuildKit uv cache。
- Docker Compose 三套配置统一声明 `planet-aiprovider:latest` 镜像名和 `planet_uv_config` build secret。
---
## [0.65.0] — 2026-05-21
Released: 2026-05-21
### Highlights
- 收敛 Admin Next 数据源触发入口:主按钮在未勾选时触发全部,勾选内置源后切换为“触发已选 N”并移除手填 ID 的批量触发弹窗。
- 优化数据源采集队列入口:右上角按钮常驻,空态显示队列图标,有任务时显示纯圆环进度,队列改为浮层避免挤压表格。
- 强化 `planet.sh destroy` 清理语义,销毁时先硬重置运行中的本地 Postgres `public` schema避免残留采集数据让 OOBE 误判 ready。
### Added / Fixed / Improved
- Admin Next 表格新增可选选择列,仅在 `/datasources` 内置源分区启用,支持当前可见行全选并在筛选、切分区或刷新时清空选择。
- 数据源批量触发复用 `/datasources/trigger-batch``source_ids`,成功后写入现有采集队列并清空勾选。
- `destroy` 补充清理 `planet-aiprovider:latest` 镜像以及 Python/Vite 等本地编译缓存,同时保留源码和 `.env`
- Docs Gatekeeper 与 Tactile UI 文档/样式继续补齐覆盖本轮按钮、队列、OOBE 和销毁流程说明。
---
## [0.64.0] — 2026-05-21
Released: 2026-05-21
### Highlights
- 新增 Earth 首次初始化 OOBE由后端真实采集状态决定是否显示避免 localStorage 清空后误弹,并提供桌面毛玻璃引导与移动端 bottom sheet。
- 数据源页新增下载列表式采集队列,把单源、批量和触发全部的任务进度统一展示,并支持失败重试与跳转详情。
- Earth 内容新增“关于”配置接口和后台 tabEarth 设置页 About 卡片改为运行时读取配置并带默认 fallback。
### Added / Fixed / Improved
- 新增 `/api/v1/earth/oobe-status``/api/v1/earth/about` GET/PUT/DELETE并让 Earth 前端加载 `about.js``oobe.js`
- Admin Next 数据源队列优先消费 `datasource_tasks` WebSocket断线时轮询 `/datasources/{id}/task-status`,刷新后只恢复后端仍在运行的真实任务。
- Admin Next 深色主题滑块补齐 Docs 同款 dark token侧栏主题控件在 dark 模式下不再保持浅色底座。
- 用户手册、快速开始、Earth 前端上下文和 Admin 前端上下文同步记录 OOBE、采集队列、About 配置与主题滑块行为。
---
## [0.63.1] — 2026-05-21
Released: 2026-05-21
### Highlights
- 修复新设备初始化后 Admin Next 无法打开的问题:补上被 `.gitignore``lib/` 规则误忽略的 Admin Next utility module。
### Added / Fixed / Improved
-`frontend/src/admin-next/lib/utils.ts` 纳入版本控制,恢复 `DashboardNext``AdminNextLayout` 等页面对 `cn``formatNumber` 的运行时依赖。
- 避免 Vite dev server 在新 clone 环境中因缺失源码模块而对 `/src/admin-next/*` 返回 500并导致动态导入 AdminNextRoutes 失败。
---
## [0.63.0] — 2026-05-21
Released: 2026-05-21
### Highlights
- 新增 `./planet.sh init` 首次初始化入口,将 uv/bun 依赖同步、env 模板补齐、数据库容器启动、建表 seed 和默认用户生成串成一条空项目引导路径。
- 新增 `./planet.sh destroy` 破坏性重置入口,带 CLI 确认保护,可清理 Planet 容器、卷、镜像和本地编译/运行状态,同时保留源码与 `.env` 配置。
- 改进 `planet.sh` 日志体验,只在带状态标签的输出行末尾追加时间戳,并让 `init` 在应用服务已运行时自动跳过重复初始化。
### Added / Fixed / Improved
- `init` 复用现有 Docker/依赖 helper支持重复执行时不覆盖 env、不清空数据并在完成后提示默认本地登录账号。
- `destroy` 停止本地服务后清理 Docker compose 状态、项目镜像、数据卷、`.venv``node_modules`、前端构建产物和 Planet state/cache。
- README 快速启动补充 `init``destroy` 命令说明,明确首次引导和重置路径。
---
## [0.62.0] — 2026-05-21
Released: 2026-05-21

View File

@@ -16,7 +16,24 @@
- 已被正式实现替代、继续放在 `docs/` 根目录会误导后续开发的计划,归档
- 仍然指导未来开发、尚未完成或仍有明确执行价值的文档,继续保留在 `docs/`
当前替代入口:
- 业务和数据产品链路见 [业务架构与数据流转](/home/ray/dev/linkong/planet/docs/technical/zh/platform-data-flows.md)。
- 当前后端作业、outbox 和 Earth 同步实现见 [数据作业与 Outbox 技术架构](/home/ray/dev/linkong/planet/docs/technical/zh/data-job-earth-sync-architecture.md)。
- 用户操作流程见 [Planet 使用手册](/home/ray/dev/linkong/planet/docs/technical/zh/manual.md)。
- 当前代码结构见 [技术文档索引](/home/ray/dev/linkong/planet/docs/technical/zh/README.md)。
补充说明:
- 一部分归档文档来自外部或临时工作流草案,例如 sisyphus 生成的初稿
- 这类文档如果有可用内容,应先吸收到 `docs/plans/``docs/technical/`,再归档保留来源记录
## 近期归档
- [Docs Gatekeeper 鉴权系统计划](/home/ray/dev/linkong/planet/docs/deprecated/docs-gatekeeper-auth-plan.md):已落地,当前实现见技术文档。
- [Location Resolver 共享管线计划](/home/ray/dev/linkong/planet/docs/deprecated/location-resolver-shared-pipeline-plan.md):已落地,当前实现见技术文档。
- [Earth Surface Hover Info Plan](/home/ray/dev/linkong/planet/docs/deprecated/earth-surface-hover-info-plan.md):已实现,保留为历史记录。
- [Admin Next Dual Track Full Migration Plan](/home/ray/dev/linkong/planet/docs/deprecated/admin-next-dual-track-full-migration-plan.md):已被当前 `frontend/src/admin/` 控制台结构替代。
- [Admin Next Soft Glass Goal Driven Plan](/home/ray/dev/linkong/planet/docs/deprecated/admin-next-soft-glass-goal-driven-plan.md):已被当前 `frontend/src/admin/` 控制台结构替代。
- [Admin Next Parity Checklist](/home/ray/dev/linkong/planet/docs/deprecated/admin-next-parity-checklist.md):旧 `/admin-next/*` 验收资料,保留为迁移历史。
- [Admin Next Parity Audit Closeout](/home/ray/dev/linkong/planet/docs/deprecated/admin-next-parity-audit-closeout-plan.md):旧 `/admin-next/*` 审计资料,保留为迁移历史。

View File

@@ -104,7 +104,7 @@ The old AntD page itself treated these Earth content tabs as placeholder-level c
- `models_3d`
- `news_anchor_strategy`
If backend endpoints are later added, these items must be promoted into `docs/plans/admin-next-parity-checklist.md` with concrete API and UI acceptance criteria.
If backend endpoints are later added, these items must be promoted into a new active plan under `docs/plans/` with concrete API and UI acceptance criteria.
## Final Gate
@@ -114,7 +114,7 @@ After the route promotion, the final gate is no longer “switch old routes.”
2. Run the static checks:
- `rg "map: \\(\\) => \\[\\]|暂不支持保存|placeholder" frontend/src/admin-next`
- `rg "ShadowPage|FeatureConsole|GlassPanel|InspectorDrawer" frontend/src/admin-next`
3. Manually verify every official route listed in `docs/plans/admin-next-parity-checklist.md`.
3. Manually verify every official route listed in `docs/deprecated/admin-next-parity-checklist.md`.
4. Confirm `/legacy/admin/*` still opens old AntD pages during the validation window.
5. Delete old AntD pages and remove AntD dependencies only as a separate final cleanup task after explicit confirmation.

View File

@@ -8,7 +8,7 @@ The redesign must cover desktop and mobile. Data display, icon semantics, table
## Criteria For Success
- This plan exists at `docs/plans/admin-next-soft-glass-goal-driven-plan.md`.
- This archived plan exists at `docs/deprecated/admin-next-soft-glass-goal-driven-plan.md`.
- `/admin-next/*` has real pages for every route; route usage of `ShadowPage` is removed.
- Admin Next supports `system`, `light`, and `dark` theme modes using the same persistence and system-theme idea as Docs.
- The visual language reads as soft-glass / light-neumorphic instead of an AntD reskin: translucent panels, fine borders, subtle glow, cool backgrounds, restrained accent colors, crisp icons, and tactile controls.
@@ -214,4 +214,3 @@ Manual viewport checks:
- low height;
- 125% / 150% browser zoom;
- light / dark / system theme modes.

View File

@@ -1,6 +1,6 @@
# Location Resolver Shared Pipeline Plan
**状态**:已实现,当前用户流程见 [Earth 位置候选采集使用手册](/home/ray/dev/linkong/planet/docs/technical/zh/location-pipeline-user.md),开发接口见 [通用位置估算管线开发说明](/home/ray/dev/linkong/planet/docs/technical/zh/location-pipeline-development.md)。
**状态**:已实现,当前用户流程见 [Planet 使用手册](/home/ray/dev/linkong/planet/docs/technical/zh/manual.md) 的 Earth 位置候选采集章节,开发接口见 [通用位置估算管线开发说明](/home/ray/dev/linkong/planet/docs/technical/zh/location-pipeline-development.md)。
## Goal

View File

@@ -37,7 +37,7 @@ If the same action has both a UI and a CLI path (e.g. user creation), describe t
- a `credential_provider` in `backend/app/core/datasource_defaults.py`;
- a default credential guide in `backend/app/services/credential_guides.py`;
- a supported connectivity provider in `backend/app/services/datasource_connectivity.py`;
- settings UI guidance or a credential form in `frontend/src/pages/Settings/Settings.tsx`;
- settings UI guidance or a credential form in `frontend/src/admin/pages/PlainResourcePages.tsx`;
- a regression test that fails if the guide/provider is missing.
## Recommended Checks

View File

@@ -7,6 +7,12 @@
- 分几期做
- 当前差距和下一步是什么
当前实现、业务架构和数据链路不放在这里。它们分别进入:
- [业务架构与数据流转](/home/ray/dev/linkong/planet/docs/technical/zh/platform-data-flows.md)
- [数据作业与 Outbox 技术架构](/home/ray/dev/linkong/planet/docs/technical/zh/data-job-earth-sync-architecture.md)
- [技术文档索引](/home/ray/dev/linkong/planet/docs/technical/zh/README.md)
适合放入这里的内容:
- Earth / BGP / 地形 / 天球实施方案
@@ -16,38 +22,40 @@
当前重点入口:
- [earth-mobile-drawer-ui-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-mobile-drawer-ui-plan.md)
- [earth-compute-center-bgp-style-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-compute-center-bgp-style-plan.md)
- [earth-renderer-architecture-separation-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-renderer-architecture-separation-plan.md)
- [Earth Mobile Drawer UI Plan](/home/ray/dev/linkong/planet/docs/plans/earth-mobile-drawer-ui-plan.md)
- [Earth Compute Center BGP Style Plan](/home/ray/dev/linkong/planet/docs/plans/earth-compute-center-bgp-style-plan.md)
- [Earth Renderer Architecture Separation Plan](/home/ray/dev/linkong/planet/docs/plans/earth-renderer-architecture-separation-plan.md)
- [Earth 高精度国界静态瓦片计划](/home/ray/dev/linkong/planet/docs/plans/earth-high-precision-boundary-tiles-plan.md)
- [earth-predicted-orbit-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-predicted-orbit-plan.md)
- [earth-webgl-instancing-satellites-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-webgl-instancing-satellites-plan.md)
- [earth-real-terrain-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-real-terrain-plan.md)
- [earth-news-source-configuration-and-collector-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-news-source-configuration-and-collector-plan.md)
- [earth-news-cruise-summary-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-news-cruise-summary-plan.md)
- [Earth Predicted Orbit Plan](/home/ray/dev/linkong/planet/docs/plans/earth-predicted-orbit-plan.md)
- [Earth WebGL Instancing Satellites Plan](/home/ray/dev/linkong/planet/docs/plans/earth-webgl-instancing-satellites-plan.md)
- [Earth Real Terrain Plan](/home/ray/dev/linkong/planet/docs/plans/earth-real-terrain-plan.md)
- [Earth News Source Configuration And Collector Plan](/home/ray/dev/linkong/planet/docs/plans/earth-news-source-configuration-and-collector-plan.md)
- [Earth News Cruise Summary Plan](/home/ray/dev/linkong/planet/docs/plans/earth-news-cruise-summary-plan.md)
- [Earth 动作捕捉手势控制计划](/home/ray/dev/linkong/planet/docs/plans/earth-motion-capture-gesture-control-plan.md)
- [Earth 动捕交互语义 V2 计划](/home/ray/dev/linkong/planet/docs/plans/earth-motion-gesture-interaction-v2-plan.md)
- [Earth Presentation 解耦架构计划](/home/ray/dev/linkong/planet/docs/plans/earth-presentation-decoupled-architecture-plan.md)
- [earth-vessel-rendering-performance-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-vessel-rendering-performance-plan.md)
- [Earth Vessel Rendering Performance Plan](/home/ray/dev/linkong/planet/docs/plans/earth-vessel-rendering-performance-plan.md)
- [AIS 多源采集、冲突记录与聚合接口计划](/home/ray/dev/linkong/planet/docs/plans/earth-vessel-ais-aggregation-plan.md)
- [earth-interactable-layer-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-interactable-layer-plan.md)
- [frontend-public-docs-site-plan.md](/home/ray/dev/linkong/planet/docs/plans/frontend-public-docs-site-plan.md)
- [Docs Gatekeeper 鉴权系统计划](/home/ray/dev/linkong/planet/docs/plans/docs-gatekeeper-auth-plan.md)
- [Location Resolver 共享管线计划](/home/ray/dev/linkong/planet/docs/plans/location-resolver-shared-pipeline-plan.md)
- [frontend-ai-playground-development-plan.md](/home/ray/dev/linkong/planet/docs/plans/frontend-ai-playground-development-plan.md)
- [Earth Interactable Layer Plan](/home/ray/dev/linkong/planet/docs/plans/earth-interactable-layer-plan.md)
- [Frontend Public Docs Site Plan](/home/ray/dev/linkong/planet/docs/plans/frontend-public-docs-site-plan.md)
- [Frontend AI Playground Development Plan](/home/ray/dev/linkong/planet/docs/plans/frontend-ai-playground-development-plan.md)
- [AI Provider OpenClaw-Style Routing Plan](/home/ray/dev/linkong/planet/docs/plans/ai-provider-openclaw-style-routing-plan.md)
- [统一集成配置 Schema 系统计划](/home/ray/dev/linkong/planet/docs/plans/integration-config-schema-system-plan.md)
- [Lightweight Agent Orchestrator 与 WebSearch 证据层计划](/home/ray/dev/linkong/planet/docs/plans/agents-light-orchestrator-websearch-plan.md)
- [Admin Next Parity Checklist](/home/ray/dev/linkong/planet/docs/plans/admin-next-parity-checklist.md)
- [Admin Next Parity Audit Closeout](/home/ray/dev/linkong/planet/docs/plans/admin-next-parity-audit-closeout-plan.md)
- [ue5-mvp-fused-plan.md](/home/ray/dev/linkong/planet/docs/plans/ue5-mvp-fused-plan.md)
- [UE5 MVP Fused Plan](/home/ray/dev/linkong/planet/docs/plans/ue5-mvp-fused-plan.md)
历史计划入口:
- 已完成、已替代或只作为历史决策背景保留的文档,统一放在 [Deprecated Docs](/home/ray/dev/linkong/planet/docs/deprecated/README.md)。
不适合放入这里的内容:
- 当前代码结构说明
- 组件现状和实现入口
- 已经落地的技术上下文说明
- 已经成为当前行为的“架构说明”
这些应放入:
- [docs/technical/README.md](/home/ray/dev/linkong/planet/docs/technical/README.md)
- [技术文档索引](/home/ray/dev/linkong/planet/docs/technical/zh/README.md)
- [Technical Docs Index](/home/ray/dev/linkong/planet/docs/technical/en/README.md)

View File

@@ -178,5 +178,5 @@ Runtime 选择规则:
- `backend/app/services/ai_client.py`
- `aiprovider/main.py`
- `aiprovider/provider_service.py`
- `frontend/src/admin-next/pages/PlainResourcePages.tsx`
- `docs/plans/admin-next-parity-audit-closeout-plan.md`
- `frontend/src/admin/pages/PlainResourcePages.tsx`
- `docs/deprecated/admin-next-parity-audit-closeout-plan.md`

View File

@@ -57,7 +57,7 @@
抽自现 manual.md重新组织
1. 首次启动 — `./planet.sh start`、默认账号(`admin/admin123``linkong/12345678`,引用 `b15d097b` 引入的 `DEFAULT_LOGIN_USERS`
1. 首次启动 — `./planet.sh start`、默认账号(`admin/admin123``linkong/LK12345678`,引用 `b15d097b` 引入的 `DEFAULT_LOGIN_USERS`
2. 启停与按模块重启 — `start/stop/restart``-b -f -a -d`
3. 健康检查 — `./planet.sh health`
4. 日志 — `./planet.sh log``-f -b -a`,日志文件路径

View File

@@ -1,46 +1,52 @@
# Technical Docs
This directory holds "current implementation and current structure" documentation, focusing on:
This is the current Intelligent Planet documentation entry point. Docs are organized by reader path: start with business architecture to understand data products, then move into user manuals or implementation references.
- How the code is organized right now
- Where the current entry points are
- How state and components work
- Which implementation boundaries future changes should follow
## Business Architecture
What belongs here:
- [Business Architecture and Data Flows](/home/ray/dev/linkong/planet/docs/technical/en/platform-data-flows.md): purpose, collection flow, fact tables, derived tables, cache, and WebSocket broadcast path for each Earth data product
- [Naming Glossary](/home/ray/dev/linkong/planet/docs/technical/en/naming-glossary.md): English/Chinese terms used across the console, Earth, backend, and docs
- Quickstart and user manual
- Frontend context
- Earth frontend structure
- Earth satellite footprint policy
- Earth render layer order
- Earth layer style property index
- Backend runtime control
- Collector status
- Collector settings and connectivity validation
- Earth Interactable integration
- Collection format conventions
## Manual
## Entry Points
- [Quickstart](/home/ray/dev/linkong/planet/docs/technical/en/quickstart.md): shortest path to getting Intelligent Planet running from scratch
- [Intelligent Planet Manual](/home/ray/dev/linkong/planet/docs/technical/en/manual.md): user workflows for the console, Earth, Docs, and common features
- [FAQ](/home/ray/dev/linkong/planet/docs/technical/en/faq.md): troubleshooting for Windows / WSL, ports, dependencies, motion capture, credentials, and Docs permissions
- [Quickstart](/home/ray/dev/linkong/planet/docs/technical/en/quickstart.md): The shortest path to getting Planet running from scratch
- [Planet Manual](/home/ray/dev/linkong/planet/docs/technical/en/manual.md): Complete usage guide for the console, `planet.sh`, Earth, and Docs
- [FAQ](/home/ray/dev/linkong/planet/docs/technical/en/faq.md): Central troubleshooting entry for Windows / WSL, ports, dependencies, motion capture, credentials, and Docs permissions
- [Earth Location Candidate Collection User Guide](/home/ray/dev/linkong/planet/docs/technical/en/location-pipeline-user.md): Collect and preview coordinate candidates for compute centers and BGP collectors on Earth
- [Collector Settings and Connectivity Validation](/home/ray/dev/linkong/planet/docs/technical/en/datasource-collector-settings-connectivity.md): Data source catalog, collector settings, connectivity validation, and BarentsWatch credentials
- [Shared Location Resolution Pipeline Development Guide](/home/ray/dev/linkong/planet/docs/technical/en/location-pipeline-development.md): Backend location resolver / pipeline interfaces, registries, and extension points
- [Docs Gatekeeper Development Guide](/home/ray/dev/linkong/planet/docs/technical/en/docs-gatekeeper-development.md): Backend Docs catalog, Markdown content loading, and Gatekeeper permission groups
- [Naming Glossary](/home/ray/dev/linkong/planet/docs/technical/en/naming-glossary.md): English/Chinese term mapping for the console, Earth, backend, and docs
- [Earth Interactable Usage](/home/ray/dev/linkong/planet/docs/technical/en/earth-interactable-usage.md): API, lifecycle, and integration examples for Earth surface icon Interactable
- [Earth Toolbar and Overlay Coordination](/home/ray/dev/linkong/planet/docs/technical/en/earth-toolbar-overlay-coordination.md): Closing matrix and integration rules for toolbar buttons, search, settings, news, and layer overlays
- [Tactile UI Components](/home/ray/dev/linkong/planet/docs/technical/en/tactile-ui-components.md): Portable button, switch, tooltip, and scrollbar APIs, theme tokens, and migration rules
## Earth Implementation
What does not belong here:
- [Earth Frontend Context](/home/ray/dev/linkong/planet/docs/technical/en/earth-frontend-context.md): Earth modules, state, WebSocket refresh, and layer lifecycle
- [Earth Layer Style Reference](/home/ray/dev/linkong/planet/docs/technical/en/earth-layer-style-reference.md): layer colors, symbols, materials, and visual parameters
- [Earth Render Layer Order](/home/ray/dev/linkong/planet/docs/technical/en/earth-render-layer-order.md): renderOrder, depth strategy, picking, and collision avoidance
- [Earth Satellite Footprint Policy](/home/ray/dev/linkong/planet/docs/technical/en/earth-satellite-footprint-policy.md): satellite footprint display boundaries and strategy
- [BGP Context](/home/ray/dev/linkong/planet/docs/technical/en/earth-bgp-context.md): BGP rendering, aggregation, and collector implementation in Earth
- [Earth Interactable Usage](/home/ray/dev/linkong/planet/docs/technical/en/earth-interactable-usage.md): `Interactable` API, lifecycle, and integration examples
- [Earth Toolbar and Overlay Coordination](/home/ray/dev/linkong/planet/docs/technical/en/earth-toolbar-overlay-coordination.md): close matrix for toolbar buttons, search, settings, news, and layer overlays
- Incomplete roadmaps
- Future iteration plans
- Large-scale refactor proposals
## Frontend Implementation
Those belong in:
- [Admin Frontend Context](/home/ray/dev/linkong/planet/docs/technical/en/frontend-admin-frontend-context.md): console page structure, state boundaries, and lazy loading
- [Frontend Layout Guidelines](/home/ray/dev/linkong/planet/docs/technical/en/frontend-layout-guidelines.md): layout, tables, panels, and responsive constraints
- [Tactile UI Components](/home/ray/dev/linkong/planet/docs/technical/en/tactile-ui-components.md): button, switch, tooltip, scrollbar APIs, and theme tokens
- [Plans Index](/home/ray/dev/linkong/planet/docs/plans/README.md)
## Backend Implementation
- [Data Collectors](/home/ray/dev/linkong/planet/docs/technical/en/backend-collectors.md): collectors, task types, save layer, and status updates
- [System Service Control](/home/ray/dev/linkong/planet/docs/technical/en/backend-system-service-control.md): backend service control and system operation APIs
- [Datasource Collector Settings and Connectivity](/home/ray/dev/linkong/planet/docs/technical/en/datasource-collector-settings-connectivity.md): datasource catalog, collector settings, connectivity validation, and credentials
- [Datasource API Performance](/home/ray/dev/linkong/planet/docs/technical/en/backend-datasources-api-performance.md): DataSources list API performance and caching
- [Data Jobs and Outbox Architecture](/home/ray/dev/linkong/planet/docs/technical/en/data-job-earth-sync-architecture.md): PostgreSQL job queue, outbox, listener, and Kafka / Spark evolution boundaries
- [Shared Location Resolution Pipeline Development Guide](/home/ray/dev/linkong/planet/docs/technical/en/location-pipeline-development.md): location resolver / pipeline interfaces, registries, and extension points
- [News Live Streams Collector Format](/home/ray/dev/linkong/planet/docs/technical/en/earth-news-live-streams-collector-format.md): news, live stream, and media collection payload conventions
- [Docs Gatekeeper Development Guide](/home/ray/dev/linkong/planet/docs/technical/en/docs-gatekeeper-development.md): backend Docs catalog, Markdown content loading, and Gatekeeper permission groups
## Agents and Operations
- [AI Provider Guide](/home/ray/dev/linkong/planet/docs/technical/en/agents-aiprovider.md): model provider adapters, task prompts, and invocation boundaries
- [Intelligent Planet Ops Runbook](/home/ray/dev/linkong/planet/docs/technical/en/ops-runbook.md): deployment, startup, troubleshooting, and sensitive operations
- [Docker + Compose + Buildx Upgrade](/home/ray/dev/linkong/planet/docs/technical/en/ops-docker-compose-buildx-upgrade.md): Docker toolchain upgrade steps
- [planet.sh Startup](/home/ray/dev/linkong/planet/docs/technical/en/ops-planet-sh-startup.md): startup script, health checks, and performance optimization
## Plans and History
Incomplete roadmaps, large refactor proposals, and future plans live in the repository path `docs/plans/README.md`. Completed, outdated, or replaced designs live in `docs/deprecated/README.md`.

View File

@@ -101,7 +101,7 @@ The AI settings page uses:
These endpoints require an authenticated user. The `secrets` endpoint is only used when the settings page reveals a key or token; hiding the field restores the masked preview.
Admin Next keeps the AI page aligned with the legacy information architecture:
Admin keeps the AI page aligned with the legacy information architecture:
- `Model Providers`
- Manages provider, wire adapter, default model, LLM API key, proxy URL, proxy token, model refresh, set-as-default, and lightweight connectivity testing.
@@ -272,7 +272,7 @@ Each provider has its own key slot. Resolution order is:
`.env` is only a fallback. After the settings page saves successfully, or after the connection test succeeds, PostgreSQL becomes the global default source.
Admin Next must compute key status per provider or tool:
Admin must compute key status per provider or tool:
- If the database has a key for the current provider/tool, show `configured`.
- If the database has no key but the fallback provider, model, or tool matches the current item, show the fallback masked preview.
@@ -283,7 +283,7 @@ Tool keys follow the same rule. WebSearch and OCR must match the current tool an
### Lightweight Connectivity Testing
The Admin Next plug button performs a lightweight connectivity check and does not save configuration. Common API-platform practice is two-tiered:
The Admin plug button performs a lightweight connectivity check and does not save configuration. Common API-platform practice is two-tiered:
- Check a provider catalog or low-cost endpoint to validate base URL, authentication, and model reachability.
- Send full model requests only when the user explicitly runs Playground or a business task.

View File

@@ -73,6 +73,8 @@ async def run(self, db):
**Core file**: `backend/app/services/collectors/base.py`
Manual trigger, data clearing, and cache clearing now enter the PostgreSQL data job queue. `collection_tasks` remains the task ledger. Collectors only own `fetch -> transform -> save`; the `data_jobs.py` worker claims `collect` / `clear_data` / `clear_cache` / `earth_refresh` jobs and writes progress back. Earth layer refresh relationships live in `earth_layer_adapters.py`; do not hand-code cache invalidation or WebSocket broadcasts inside individual collectors or buttons.
## III. Collector List
| Collector | Data type | Content | Frequency |
@@ -93,7 +95,7 @@ Earth boundaries are no longer data collectors. They are Earth static rendering
TOP500 and Epoch AI compute sources do not always provide usable coordinates. The unified Earth compute-center endpoint uses only valid source-provided coordinates or `compute_center_locations` dimension-table coordinates during the main map startup path; records without coordinates are returned as `unresolved` instead of being rendered from a local registry, country centroid, or guessed city. When users manually collect candidates, the backend queries ROR and Nominatim/OpenStreetMap from source fields; accepted candidates are saved into `compute_center_locations` and rendered from that table on the next layer refresh.
Admin Next collection management follows the business hierarchy instead of flattening every endpoint into one table:
Admin collection management follows the business hierarchy instead of flattening every endpoint into one table:
- `Collectors`: endpoint, authentication, headers, base parameters, enabled state, and credential guides.
- `Collection Schedule`: scheduler state and task controls.

View File

@@ -96,4 +96,4 @@ if (res.data.task_id) {
## Related Files
- [datasources.py](/home/ray/dev/linkong/planet/backend/app/api/v1/datasources.py): `_load_datasource_list_context`, `list_datasources`
- [DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/DataSources/DataSources.tsx): `BuiltInDataSource`, `triggerDatasource`
- [DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx): `BuiltInDataSource`, `triggerDatasource`

View File

@@ -0,0 +1,141 @@
# Data Jobs and Outbox Architecture
This document records the technical boundary for Planet v1 data jobs, database outbox, and Earth refresh. For each data product's business purpose and end-to-end flow, see [Business Architecture and Data Flows](/home/ray/dev/linkong/planet/docs/technical/en/platform-data-flows.md).
## Architecture Boundary
- PostgreSQL is the durable v1 job ledger and outbox. Kafka, Celery, and RQ are intentionally not part of v1.
- `collection_tasks` records collection, data clearing, cache clearing, and non-database Earth refresh jobs.
- `earth_data_change_events` records fact-table or derived-table changes and is the reliable source for Earth sync.
- `LISTEN/NOTIFY` is only the low-latency wakeup path; the listener still polls unconsumed outbox rows.
- Redis is mainly cache, auth helper, OTP / rate limit, temporary logs, and WebSocket support. It is not the durable queue.
## Database Change Sync
```mermaid
flowchart LR
Write["Fact or derived table write"] --> Trigger["PostgreSQL trigger"]
Trigger --> Outbox["earth_data_change_events"]
Trigger --> Notify["planet_earth_data_changes"]
Outbox --> Listener["earth_db_change_listener"]
Notify --> Listener
Listener --> Adapter["earth_layer_adapters"]
Adapter --> Cache["cache invalidation"]
Cache --> WS["earth_updates"]
```
1. Collection, clearing, location resolution, or projection jobs write fact or derived tables.
2. Statement triggers write outbox rows for `INSERT` / `UPDATE` / `DELETE`.
3. The listener wakes through notify or finds pending rows through polling.
4. The listener maps `table + source` to Earth layers, refresh strategy, and cache patterns through `earth_layer_adapters.py`.
5. The listener merges short-window same-layer events, invalidates cache, and broadcasts `earth_updates`.
6. The outbox row is marked consumed only after successful broadcast; failed rows stay retryable.
DB changes no longer create default `earth_refresh` jobs, so they are not blocked by long same-source collection or clearing jobs. `earth_refresh` remains for manual cache clearing and non-DB refresh hints.
## Data Job Queue
`collection_tasks` is the unified job ledger. Workers claim `queued` jobs with PostgreSQL `FOR UPDATE SKIP LOCKED`; write jobs for the same `source` run serially, while different sources may run in parallel.
| task_type | Purpose |
| --- | --- |
| `collect` | Run a built-in datasource collector |
| `clear_data` | Delete collected rows and declared derived rows for the source |
| `clear_cache` | Delete Earth / dashboard cache for the source |
| `earth_refresh` | Invalidate Earth layer cache and broadcast a refresh hint for non-DB changes |
API handlers only create jobs and return `task_id`. Execution, progress, cancellation, and terminal state are written back by workers and pushed to the frontend through the `datasource_tasks` channel.
Cancellation means “keep committed batches”: clicking stop marks the job as `cancelling` and cancels the in-memory coroutine. Already committed batches remain; unfinished batches follow the collector or cleanup rollback path.
## Earth Sync Event Model
The unified event model is `earth.layer.changed`:
```json
{
"event": "earth.layer.changed",
"action": "database_changed",
"source": "celestrak_tle",
"table": "collected_data",
"operation": "DELETE",
"layers": ["satellites"],
"refresh_strategy": "clear_then_reload",
"records_processed": 11125,
"occurred_at": "2026-05-25T10:20:30Z"
}
```
| strategy | Purpose |
| --- | --- |
| `clear_then_reload` | Clear local frontend layer objects first, then force a refetch. Prefer this for deletes. |
| `reload` | Keep old objects until fresh data returns. Use it for location, metadata, or non-destructive updates. |
| `delta` | Used only for `earth_interactables`; upsert or remove objects by id. |
APIs must return HTTP 200 with an empty collection for real zero-data states; 5xx is reserved for real endpoint failures. After a delete event, if refetch fails, the frontend should keep the cleared state and show a lightweight error instead of restoring stale objects.
## Layer Adapter Contract
`earth_layer_adapters.py` is the single registry for sources, derived tables, Earth layers, cache patterns, and refresh strategy. New layers should be added through an adapter entry, not through one-off button handlers, collector branches, or frontend special cases.
Each adapter must declare:
- Which source or table feeds which Earth layer.
- Which Earth cache key patterns must be invalidated.
- Which owned derived tables must be removed during `clear_data`.
- The default refresh strategy for that layer.
When a source is cleared, the `clear_data` job first deletes `collected_data.source = <source>`, then deletes adapter-owned derived rows. Direct derived-table edits also trigger the outbox, so background jobs, admin APIs, and SQL repair scripts reach Earth as long as they mutate fact or derived tables.
## Operations and Troubleshooting
Check whether outbox rows are piling up:
```sql
SELECT id, table_name, operation, source, occurred_at
FROM earth_data_change_events
WHERE consumed_at IS NULL
ORDER BY id
LIMIT 20;
```
Check that triggers exist:
```sql
SELECT tgname, tgrelid::regclass
FROM pg_trigger
WHERE tgname LIKE 'tr_planet_%_changed_%'
ORDER BY 2, 1;
```
Useful log events:
- `earth.db_changes.connected`: the listener connected to PostgreSQL and started listening.
- `earth.db_changes.outbox_polled`: polling found unconsumed outbox rows.
- `earth.db_changes.broadcasted`: an Earth refresh broadcast was produced.
- `data_job.started` / `data_job.completed`: job execution state.
If Earth does not update, check in order: fact table changed, outbox was consumed, adapter covers the `source/table`, the listener is online, frontend WebSocket is connected, and the visualization API returns HTTP 200 with either an empty collection or fresh data.
## Kafka-ready Boundaries
Business code avoids depending on a concrete queue implementation by preserving these boundaries:
- `JobQueue`: submit, claim, cancel, and complete data jobs.
- `DataChangeBus`: publish database fact changes.
- `EarthLayerAdapterRegistry`: declare source, layer, cache, and derived-data relationships.
Kafka becomes appropriate when:
- Several independent services must consume the same data-change stream.
- AIS, BGP, or sensor streams become sustained high-throughput inputs.
- Consumer groups, replay, and service decoupling are required.
Spark becomes appropriate when:
- Historical data reaches tens or hundreds of millions of rows and PostgreSQL aggregation becomes expensive.
- Cross-source, long-window, spatiotemporal analysis is needed.
- Raw data lands in Parquet / Iceberg / Delta and the system starts producing offline derived data products.
For second-level continuous stream processing, evaluate Flink first. Spark is a better fit for batch or micro-batch analytics.

View File

@@ -32,8 +32,8 @@ If endpoint, headers, base configuration, or credential fingerprint changes afte
Files:
- [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/pages/PlainResourcePages.tsx)
- [AdminNextRoutes.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/AdminNextRoutes.tsx)
- [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
- [AdminRoutes.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/AdminRoutes.tsx)
Current behavior:
@@ -57,7 +57,7 @@ Current behavior:
File:
- [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/pages/PlainResourcePages.tsx)
- [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
Current behavior:

View File

@@ -2,7 +2,7 @@
Docs Gatekeeper moves `/docs` from "bundle all Markdown into the frontend" to "return catalog and content from the backend according to permissions." Its goal is to keep public manuals, user docs, developer docs, and admin/ops docs in one searchable Docs page while making every protected Markdown body pass through a server-side whitelist and authorization check.
For the user workflow, see the Docs section in [Planet Manual](/home/ray/dev/linkong/planet/docs/technical/en/manual.md).
For the user workflow, see the Docs section in [Intelligent Planet Manual](/home/ray/dev/linkong/planet/docs/technical/en/manual.md).
## Authorization Model
@@ -61,15 +61,15 @@ DocsMetadata(
"public",
"Manual",
2,
"Planet 使用手册",
"Planet Manual",
"智能星球使用手册",
"Intelligent Planet Manual",
)
```
When adding a public technical doc:
When adding a technical doc that should appear in the Docs page:
- Add both Chinese and English Markdown files.
- Add filename, slug, access, group, order, and titles to server `DOCS_METADATA`.
- Add filename, slug, access, group, order, and titles to server `DOCS_METADATA`. The backend catalog endpoint is authoritative; frontend metadata alone does not publish a document into `/docs` navigation.
- Add matching metadata to frontend [docs-content.ts](/home/ray/dev/linkong/planet/frontend/src/pages/Docs/docs-content.ts) so navigation titles and sorting stay aligned.
- Update `docs/technical/zh/README.md` and `docs/technical/en/README.md` when the document should be discoverable from the README.
@@ -83,7 +83,7 @@ The user API:
- Validates group names on update: only `docs_user`, `docs_developer`, and `docs_admin` are accepted.
- Allows only `super_admin` to modify Gatekeeper groups.
Frontend [Users.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Users/Users.tsx) displays group tags and provides a multi-select in the edit form. Non-`super_admin` users see the field disabled, and submission removes `gatekeeper_groups` before sending.
Frontend [Users.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/Users.tsx) displays group tags and provides a multi-select in the edit form. Non-`super_admin` users see the field disabled, and submission removes `gatekeeper_groups` before sending.
## Frontend Docs Loading

View File

@@ -188,7 +188,7 @@ Current reality:
- that is expected, because incidents are aggregated and de-noised
- but incident-first rendering makes the Earth view look too quiet unless there is another always-available activity layer
Implementation detail for the recommended `activity layer` is expanded in the [BGP Region Aggregation Plan](/home/ray/dev/linkong/planet/docs/plans/earth-bgp-region-aggregation-plan.md).
Implementation detail for the recommended `activity layer` is kept in the repository path `docs/plans/earth-bgp-region-aggregation-plan.md`.
So the immediate next milestone is:
@@ -259,7 +259,7 @@ Reference inspiration:
Relevant page:
- `frontend/src/pages/BGP/BGP.tsx`
- `frontend/src/admin/pages/PlainResourcePages.tsx`
Current BGP console page has three levels:
@@ -321,7 +321,7 @@ Backend:
Frontend:
- `frontend/src/pages/BGP/BGP.tsx`
- `frontend/src/admin/pages/PlainResourcePages.tsx`
- `frontend/public/earth/js/bgp.js`
- `frontend/public/earth/js/main.js`
- `frontend/public/earth/js/info-card.js`

View File

@@ -4,7 +4,7 @@ This document describes the current real structure of the Earth display frontend
Related references:
- [Project Rules](/home/ray/dev/linkong/planet/rules.md)
- Repository root `rules.md`
- [Frontend Layout Guidelines](/home/ray/dev/linkong/planet/docs/technical/en/frontend-layout-guidelines.md)
## Current Goal
@@ -56,6 +56,8 @@ Responsibilities:
- Layer module integration
- Earth-level state synchronization
Earth treats `/ws` `earth_updates` as refresh hints only; real data is fetched again from `/api/v1/visualization/...`. Backend database-driven refresh now has the listener clear cache and broadcast directly instead of going through the default `earth_refresh` job queue. After `database_changed`, the frontend applies the per-layer `clear_then_reload`, `reload`, or `delta` strategy. `clear_then_reload` must clear Three.js objects before a no-store refetch, and summary is only a consistency check, not a reason to skip a layer reload when the count is `0`. See [Data Jobs and Outbox Architecture](/home/ray/dev/linkong/planet/docs/technical/en/data-job-earth-sync-architecture.md) for the technical pipeline and [Business Architecture and Data Flows](/home/ray/dev/linkong/planet/docs/technical/en/platform-data-flows.md) for business data flows.
### 3. Earth Control Layer
- [controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js)
@@ -151,9 +153,14 @@ Each module is responsible for its own:
`brand.js` manages Earth HUD brand resources. Static assets provide the default brand; runtime overrides come from `/api/v1/earth/brand`, and uploaded images are served from `/earth-brand-assets/...`. The frontend must treat logo/title images and text fallback separately: if an image fails, show the text title; if text fields are empty, rely on backend defaults so the HUD brand area never renders blank. The console Earth Content page owns saving and resetting brand configuration; the Earth frontend only consumes it.
The Admin Next Earth Content page must preserve runtime semantics:
`about.js` manages the About card inside Earth settings. Frontend defaults remain as a fallback, while runtime content is loaded from `/api/v1/earth/about`. If the request fails or fields are missing, the renderer must fall back per field so the settings page never renders an empty card. Admin exposes an Earth Content `About` tab; saving uses `PUT /api/v1/earth/about`, and restoring defaults uses `DELETE /api/v1/earth/about`.
`oobe.js` manages the first-run Earth initialization guide. OOBE visibility must be driven by `/api/v1/earth/oobe-status` and its `ready` field, not by `localStorage`. `localStorage` may only store a short-lived "skip on this browser" flag; if the backend reports `ready: true`, logout, cleared browser storage, or a different browser must not show OOBE again. Desktop uses a dark starfield scrim and glass startup panel, while mobile uses a bottom sheet and respects `prefers-reduced-motion`.
The Admin Earth Content page must preserve runtime semantics:
- `Brand`: brand preview should use the same dark starfield background, size, spacing, logo/title rendering, and text fallback as the Earth HUD top-left brand block, not a generic form preview.
- `About`: configures the About card in Earth settings, including logo, kicker, title, version, description, and metadata items. Earth runtime reads `/earth/about` and falls back to defaults on failure.
- `Boundary Precision`: build boundary, refresh status, and restore defaults belong inside this section, not in the global page toolbar.
- `TV`: the list distinguishes built-in, collected, and custom sources. Card state represents enabled, disabled, draft, or error. Built-in sources cannot be deleted; collected and custom sources can. A new live source only enters draft state after the plus button is clicked; save persists it into the list, while cancel destroys the draft.
- `Basemap`, `Layer Resources`, `3D Models`, and `News Anchor Strategy`: if backend capability is not available yet, the console should show an explicit pending state instead of mixing those items into TV or brand configuration.

View File

@@ -17,26 +17,21 @@ Related context:
## Current Local Categories
Current CelesTrak satellite groups in [backend/app/services/collectors/celestrak.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/celestrak.py) include:
The CelesTrak collector now prefers the complete active satellite catalog from `GROUP=active&FORMAT=json`. This prevents one failed CelesTrak group request from being saved as a successful but incomplete batch. The collector only proceeds when the downloaded JSON is a parseable array and records include `NORAD_CAT_ID`; network, resume, or parsing failures are retried, and final failure preserves the previous current dataset.
- `starlink`
- `gps-ops`
- `galileo`
- `glonass`
- `beidou`
- `leo`
- `geo`
- `iridium-next`
CelesTrak applies a repeat-download window to large groups such as `active`. To make database resets recoverable, Planet stores raw downloads under `$PLANET_CACHE_DIR/downloads`. When `active` returns the CelesTrak "GP data has not updated" HTTP 403:
Non-Starlink categories:
- If an `active` cache exists, the collector reuses it to repopulate the database.
- If no `active` cache exists, the collector switches to fallback group mode and downloads `starlink`, `gps-ops`, `galileo`, `glonass`, `beidou`, `leo`, `geo`, and `iridium-next`.
- Fallback group mode requires every group to succeed or have a reusable cache; if any group is missing, the whole collection fails and no partial dataset is saved.
- `gps-ops`
- `galileo`
- `glonass`
- `beidou`
- `leo`
- `geo`
- `iridium-next`
The collector still provides `metadata.constellation_group` to the frontend, but the value now comes from executable inference:
- `OBJECT_NAME` starting with `STARLINK` is marked as `starlink`
- `OBJECT_NAME` starting with `IRIDIUM` is marked as `iridium-next`
- Other active satellites are not forced into the old CelesTrak small-group labels, because a broad source group is not an exact constellation
The product policy therefore still discusses GNSS/RNSS, GEO, generic LEO, and Iridium NEXT semantics. However, only fallback group mode stores the old `gps-ops`, `galileo`, `glonass`, `beidou`, `leo`, or `geo` labels; the active primary path does not force every satellite into those labels.
## Research Conclusions
@@ -136,7 +131,8 @@ This implementation only does the minimum executable version and does not change
1. Backend passes constellation group and footprint policy hint to the frontend
- CelesTrak collector stores `GROUP` in `metadata.constellation_group`
- CelesTrak collector stores the source query in `metadata.celestrak_query_group = active`
- `metadata.constellation_group` only stores inferred business constellations such as `starlink` and `iridium-next`
- Visualization API outputs:
- `properties.constellation_group`
- `properties.footprint_policy`

View File

@@ -4,7 +4,7 @@ This document describes the current real structure of the console frontend. The
Related references:
- [Project Rules](/home/ray/dev/linkong/planet/rules.md)
- Repository root `rules.md`
- [Frontend Layout Guidelines](/home/ray/dev/linkong/planet/docs/technical/en/frontend-layout-guidelines.md)
## Current Goal
@@ -22,7 +22,7 @@ Main entry point:
- [App.tsx](/home/ray/dev/linkong/planet/frontend/src/App.tsx)
Admin Next now owns the official admin routes:
Admin now owns the official admin routes:
- `/admin`
- `/users`
@@ -37,23 +37,7 @@ Admin Next now owns the official admin routes:
- `/collection-management`
- `/settings`
These routes render [AdminNextRoutes.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/AdminNextRoutes.tsx). Page metadata and menu entries come from [manifest.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/routes/manifest.tsx). `/admin-next/*` remains only as a compatibility entry and redirects to the official route; it is no longer a parallel primary entry.
The old AntD console remains available under `/legacy/admin/*` for comparison and rollback:
- `/legacy/admin`
- `/legacy/admin/datasources`
- `/legacy/admin/data`
- `/legacy/admin/collection-management`
- `/legacy/admin/earth-content`
- `/legacy/admin/ai`
- `/legacy/admin/logs`
- `/legacy/admin/settings`
- `/legacy/admin/users`
- `/legacy/admin/bgp`
- `/legacy/admin/alerts/*`
Legacy pages, `AppLayout`, `antd`, and `@ant-design/icons` stay in place during the legacy validation window. Do not remove them before Admin Next parity is accepted.
These routes render [AdminRoutes.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/AdminRoutes.tsx). Page metadata and menu entries come from [manifest.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/routes/manifest.tsx). Admin is the only console entry point; there is no parallel console or rollback route.
`/earth` is a standalone display page and is not part of the console shell.
@@ -61,7 +45,7 @@ Legacy pages, `AppLayout`, `antd`, and `@ant-design/icons` stay in place during
The official admin shell is at:
- [AdminNextLayout.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/components/layout/AdminNextLayout.tsx)
- [AdminLayout.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/components/layout/AdminLayout.tsx)
Responsibilities:
@@ -69,38 +53,13 @@ Responsibilities:
- Current account, version, logout, and theme switching
- Top search, breadcrumbs, and page shortcuts
- Single-screen content-area height closure
- Coordination for Admin Next internal scrolling, tables, detail panels, and mobile detail views
- Coordination for Admin internal scrolling, tables, detail panels, and mobile detail views
The old AntD legacy shell remains at:
Future official admin pages should adapt to `AdminLayout` and Admin page patterns; do not reintroduce a parallel admin shell.
- [AppLayout.tsx](/home/ray/dev/linkong/planet/frontend/src/components/AppLayout/AppLayout.tsx)
## Admin Section Loading
Legacy responsibilities:
- Left-side navigation
- Collapse and expand
- Current account / version information
- Content area height closure
- Site-wide unified sidebar scrollbar
Current structure:
```tsx
<Layout className="dashboard-layout">
<Sider className="dashboard-sider">...</Sider>
<Layout>
<Content className="dashboard-content">
<div className="dashboard-content-inner">{children}</div>
</Content>
</Layout>
</Layout>
```
Future official admin pages should adapt to `AdminNextLayout` and Admin Next page patterns rather than adding new capability to the old `AppLayout`. Only `/legacy/admin/*` maintenance should change the old shell.
## Admin Next Section Loading
Multi-tab pages are currently coordinated by [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/pages/PlainResourcePages.tsx), which hosts the current management and information workbench patterns. Section loading follows these rules:
Multi-tab pages are currently coordinated by [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx), which hosts the current management and information workbench patterns. Section loading follows these rules:
- Initial page load requests only the active tab; it does not prefetch every tab endpoint.
- Switching tabs lazily loads that tab. Loaded tabs stay cached in local `states`, so returning to a tab reuses the previous data.
@@ -110,6 +69,29 @@ Multi-tab pages are currently coordinated by [PlainResourcePages.tsx](/home/ray/
This keeps Earth, AI, collection management, and other multi-section pages from flooding backend APIs on cold start while preserving a fast cached tab-switching experience. Full health checks should use backend health endpoints or explicit refresh flows rather than relying on page initialization to touch every business endpoint.
## Datasource Collection Queue
The Admin datasource page routes single-source trigger, table-selected trigger, and trigger-all into a browser-download-list style collection queue:
- Queue state is managed by [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx). It is a current-session visibility layer and does not fake task history in `localStorage`.
- Progress first consumes the `/ws` `datasource_tasks` channel. If the socket is unavailable or stale, the page polls `/api/v1/datasources/{id}/task-status`.
- Trigger responses immediately insert `triggered`, `skipped`, and `failed` items. After a refresh, the queue restores only real backend rows that are still `running`, `pending`, or `queued`.
- The `Built-in Sources` section uses the table selection column for selected-source triggering. With no rows selected, the primary button is `Trigger All`; after selection, the same button becomes `Trigger Selected N`, replacing the old manual-ID batch button.
- The expanded queue no longer lives in the page content flow, so trigger-all cannot squeeze the table and detail panel. The top-right actions area uses the existing `Button` styling; the empty state shows a `ListChecks` icon, and active queues show only a pure circular total-progress indicator. Clicking it opens a floating panel grouped by running, failed, completed, and skipped.
- Queue rows can jump to the datasource detail panel, and failed rows can retry. The detail panel's task summary only reports the selected source's latest task; it does not save configuration.
- While a single-source collection is running, the action button becomes `Stop Collection`; unfinished queue rows expose a cancel action on the right. Cancellation calls `/api/v1/datasources/{source_id}/tasks/{task_id}/cancel`; the backend keeps committed batches and rolls back unfinished work.
- Clearing database data and clearing display cache enter the same queue. The frontend shows task state and does not assume the API completed synchronously.
This queue is a user-perception layer. Backend task status remains the only source of truth for running, completion, failure, and skipped decisions.
## Admin Theme Slider
The Admin sidebar theme switcher still reuses shared [SegmentedControl.tsx](/home/ray/dev/linkong/planet/frontend/src/components/SegmentedControl/SegmentedControl.tsx), while [styles.css](/home/ray/dev/linkong/planet/frontend/src/admin/styles.css) overrides the segment variables by `data-theme`:
- Light mode uses `--d-segment-bg: #eef3f9`, a white slider, and a light external shadow.
- Dark mode uses the same dark base, `#202938` slider, and dark external shadow semantics as Docs.
- Product code only hides the text label and keeps icon + tooltip behavior; it should not recreate the private slider DOM.
## Current Shared Components
### 1. `Scrollbar`
@@ -143,7 +125,7 @@ Purpose:
Current usage:
- Admin Next data sources, collected data, collection management, logs, alerts, and BGP pages
- Admin data sources, collected data, collection management, logs, alerts, and BGP pages
- Old AntD legacy pages continue using shared scrolling behavior through compatibility wrappers
### 3. `TableScrollRegion`
@@ -167,7 +149,7 @@ Files:
Purpose:
- Admin Next global tool buttons and detail-panel toolbars
- Admin global tool buttons and detail-panel toolbars
- Icon-only ordinary actions with tooltips
- Strong-intent actions such as save, create, confirm, delete, and stop
- Compact switches aligned with the Docs theme slider
@@ -225,39 +207,19 @@ Current constraints:
- Internal document links should be converted to `/docs/:slug` through `transformLink`
- Heading anchors are injected through `getHeadingId`, keeping route state outside the renderer
### 7. `ConnectionTestInput`
### 7. Admin UI Primitives
File:
Files:
- [ConnectionTestInput.tsx](/home/ray/dev/linkong/planet/frontend/src/components/ConnectionTestInput/ConnectionTestInput.tsx)
- [button.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/components/ui/button.tsx)
- [dialog.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/components/ui/dialog.tsx)
- [switch.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/components/ui/switch.tsx)
Purpose:
Use:
- Console form fields that combine an endpoint/Base URL value with a connection check
- Connection-test entry points for AI Provider and WebSearch
- Future collector configuration fields should reuse it when the test action belongs inside the input
Current constraints:
- The input suffix shows a single plug/connector icon, not an adjacent text button
- Disabled integrations must grey out both the input and its connection-test action
- The component only combines the input and action; callers still own form state, loading, disabled state, and the request itself
### 8. `TableActions`
File:
- [TableActions.tsx](/home/ray/dev/linkong/planet/frontend/src/components/TableActions/TableActions.tsx)
Purpose:
- Shared action entry for table operation columns
- Shows inline actions when expanded
- Uses a more-actions dropdown when collapsed
Companion export:
- `actionCellProps`: for action-column `onCell`, preventing action buttons from being ellipsized or wrapped
- Global tool buttons, detail actions, confirmation dialogs, and binary settings.
- Aligned with Tactile UI tokens so Admin controls keep consistent size, hover, disabled, and dark-mode behavior.
- Row actions should prefer icon buttons plus tooltip/title; do not reintroduce a separate action-menu component.
## Current State Sources
@@ -280,7 +242,7 @@ Responsibilities:
File:
- [AISettings.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/AISettings/AISettings.tsx)
- [AISettings.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
Responsibilities:
@@ -316,7 +278,7 @@ Constraints:
Example:
- [Dashboard.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Dashboard/Dashboard.tsx)
- [Dashboard.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/Dashboard.tsx)
Priority goals:
@@ -328,10 +290,10 @@ Priority goals:
Examples:
- [DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/DataSources/DataSources.tsx)
- [DataList.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/DataList/DataList.tsx)
- [Users.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Users/Users.tsx)
- [Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Settings/Settings.tsx)
- [DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
- [DataList.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/DataList.tsx)
- [Users.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/Users.tsx)
- [Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
Constraints:
@@ -341,7 +303,7 @@ Constraints:
### Datasource Directory Page
[DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/DataSources/DataSources.tsx) is the datasource directory and collection operation page. It should not grow back into a configuration editor.
[DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx) is the datasource directory and collection operation page. It should not grow back into a configuration editor.
Current page boundary:
@@ -355,7 +317,9 @@ Keep this boundary: do not put custom datasource editing, built-in endpoint over
### Collectors Page
[Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Settings/Settings.tsx) has three route modes: `/settings` for System Settings, `/earth-content` for Earth Content, and `/collection-management` for Collection Management. The `collector_credentials` tab is shown as `Collectors` under `/collection-management`.
[Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx) has three route modes: `/settings` for System Settings, `/earth-content` for Earth Content, and `/collection-management` for Collection Management. The `collector_credentials` tab is shown as `Collectors` under `/collection-management`.
The `System Display` section under `/settings` includes the `Demo Mode` switch. When enabled, Earth OOBE ignores existing current collected data and the local `browse first` temporary skip state, then opens the initialization guide directly. This switch is only for demos and acceptance checks; it does not change datasources, collection queues, or Earth content resources.
Current boundary:
@@ -367,7 +331,7 @@ Current boundary:
### Earth Content Page
`/earth-content` reuses the same single-screen tab container from [Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Settings/Settings.tsx), but its ownership is separate from System Settings:
`/earth-content` reuses the same single-screen tab container from [Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx), but its ownership is separate from System Settings:
- `TV Livestream` owns the Earth media-panel source configuration.
- `Boundary Precision` owns the Earth static boundary asset state: provider, low-precision fallback, high-precision manifest/PMTiles, source JSON, and build action.
@@ -379,8 +343,8 @@ Do not add Earth experience resources or collection-lifecycle tabs back into `/s
Examples:
- [BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx)
- [Playground.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Playground/Playground.tsx)
- [BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
- [Playground.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
Constraints:

View File

@@ -8,7 +8,8 @@ Admin pages in this project default to a "single-screen workspace" layout standa
Current recommended reference implementations:
- [frontend/src/pages/BGP/BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx)
- [frontend/src/admin/pages/PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
- [frontend/src/admin/pages/DataList.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/DataList.tsx)
- [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css)
## Core Principles
@@ -23,15 +24,15 @@ Admin pages default to:
Recommended structure:
```tsx
<AppLayout>
<AdminLayout>
<div className="page-shell">
<div className="page-shell__header">...</div>
<div className="page-shell__body">...</div>
</div>
</AppLayout>
</AdminLayout>
```
Total page height should be bounded within the `AppLayout` content area, not allowed to grow naturally downward without limit.
Total page height should be bounded within the `AdminLayout` content area, not allowed to grow naturally downward without limit.
### 2. Scrolling Should Happen Inside Modules

View File

@@ -2,7 +2,7 @@
`backend/app/services/location/` is the shared abstraction for any "given a record, decide its lat/lon" workflow. Compute centers, BGP collectors, and BGP events now run on this pipeline. Future entities such as satellite ground stations, user-claimed points, and IXP facilities should plug in here instead of creating another geocoding path.
For the user workflow, see [Earth Location Candidate Collection User Guide](/home/ray/dev/linkong/planet/docs/technical/en/location-pipeline-user.md).
For the user workflow, see the Earth coordinate-candidate section in [Intelligent Planet Manual](/home/ray/dev/linkong/planet/docs/technical/en/manual.md).
## Design Goals

View File

@@ -1,8 +1,8 @@
# Planet Manual
# Intelligent Planet Manual
This manual is for Planet end users. Starting from the browser, it covers account registration, login, configuring collectors, configuring AI, using Earth and the console, and reading the docs site. Every action happens in a browser.
This manual is for Intelligent Planet end users. Starting from the browser, it covers account registration, login, configuring collectors, configuring AI, using Intelligent Planet and the console, and reading the docs site. Every action happens in a browser.
If you are responsible for deployment or on-call duty, read the [Planet Ops Runbook](/home/ray/dev/linkong/planet/docs/technical/en/ops-runbook.md) instead — it covers shell commands, log paths, and CLI fallbacks for user creation.
If you are responsible for deployment or on-call duty, read the [Intelligent Planet Ops Runbook](/home/ray/dev/linkong/planet/docs/technical/en/ops-runbook.md) instead — it covers shell commands, log paths, and CLI fallbacks for user creation.
## Entry Overview
@@ -188,19 +188,22 @@ TV livestreams and boundary precision moved to `/earth-content`; collectors and
`/earth-content` is under the console's Operations and Configuration group and owns resources used by the Earth frontend:
- **Brand Assets**: manages the logo, title image, title text, subtitle, and description used by the Earth HUD. Uploaded images are saved as Earth brand assets and read by the Earth page immediately.
- **About**: manages the About card shown in Earth settings, including logo, kicker, title, version, description, and metadata.
- **TV Livestream**: manages sources shown in the Earth media panel.
- **Boundary Precision**: shows the current provider, low-precision fallback, high-precision PMTiles/manifest status, local source JSON, and manual build action.
- **Base Map**, **Layer Resources**, **3D Assets**, and **News Anchor Strategy**: placeholder tabs for future configuration. They do not display fake data.
The Earth page settings gear also includes Boundary Precision. Switching to High Precision starts a local background download/build, like a game update package, when no high-precision asset exists yet. Progress is shown as a percentage, and the result applies automatically after success without a page reload. Switching back to Low Precision only changes the local display preference.
If the backend decides Earth is not initialized yet, the first visit to `/earth` shows a glassy startup guide that points the user to sign in and collect data. This is based on backend data state; once the system has collected data, clearing browser storage does not make the guide reappear.
### Collection Management
`/collection-management` is also under Operations and Configuration and owns the collection lifecycle:
- **Collectors**: endpoint, headers, credentials, timeout, retry, and connection checks.
- **Collection Scheduling**: the existing scheduling configuration.
- **Collection History / Snapshots**: a placeholder for future collection task, snapshot, and collected-data browsing.
- **Collection History / Snapshots**: groups snapshots by datasource and lets the detail panel switch versions through a Time Capsule selector.
### SMTP Email Settings
@@ -230,7 +233,7 @@ To let a regular user read developer or operations docs, add `docs_developer` or
## Data Exploration
- `/datasources`: source directory. The `Collection Tasks` tab is for one-shot, scheduled, and finite collectors; it can be filtered by product domain, layer/module, enabled state, last run status, whether collected records exist, and search text. Selecting rows triggers only those sources; with no selected rows, `Collect current filter` triggers the filtered scope. The `Realtime Streams` tab is for AISStream / WebSocket long connections and shows connection health, stored totals, time-window counters, and Start / Stop / Reconnect actions. Clicking a name opens an info drawer showing endpoint, headers, base config, and built-in flag; endpoint/credentials editing happens at `/collection-management -> Collectors`. The `Collecting N` tag under the overall progress can be clicked to expand the current collection task list
- `/datasources`: source directory. The `Built-in Sources` tab can be filtered by product domain, layer/module, enabled state, last run status, whether collected records exist, and search text. With no rows selected, the primary button shows `Trigger All`; after selecting rows, it becomes `Trigger Selected N` and submits only those sources. The top-right queue button shows a queue icon when empty and a pure circular total-progress indicator while tasks exist; it opens a floating panel grouped by running, completed, failed, and skipped. Failed rows can retry, and completed rows can jump to detail. `Realtime Sources` is for AISStream / WebSocket long connections and shows connection health, stored totals, time-window counters, and Start / Stop / Reconnect actions. Endpoint/credential/header editing happens at `/collection-management -> Collectors`.
- `/data`: collected data table — used to verify "did data arrive", "is the freshness right", "does a source emit valid records"
- `/bgp`: BGP detail page with list + detail + analysis; complements the BGP layer on Earth
- `/alerts/system`, `/alerts/bgp`, `/alerts/situational`: system, BGP, and situational alerts
@@ -277,7 +280,17 @@ Search finds cables, landing points, satellites, compute centers, BGP events, BG
Compute center and BGP observer detail cards support automatic coordinate-candidate collection. Click the object then use "Collect Coordinate Candidates" or "Re-collect Coordinates". The backend assembles candidates from source coordinates, public-org registry APIs, and online geocoders. When regular sources have no candidate, the current default AI Provider runs one LLM factcheck fallback. BGP observers' stored coordinates only fill query context; they are not returned as candidates.
Candidates preview on Earth directly. Saving a compute-center candidate writes to the `compute_center_locations` dimension table and refreshes the layer immediately. The notification badge at the top-left of the compute-center layer shows the unresolved count; clicking it opens the queue, supports single collection, or "Adopt All" to save the top-confidence candidates from top to bottom. Records without candidates stay in the queue rather than being faked to country centroids. See [Earth Location Candidate Collection User Guide](/home/ray/dev/linkong/planet/docs/technical/en/location-pipeline-user.md).
Candidates preview on Earth directly. Saving a compute-center candidate writes to the `compute_center_locations` dimension table and refreshes the layer immediately. The notification badge at the top-left of the compute-center layer shows the unresolved count; clicking it opens the queue, supports single collection, or "Adopt All" to save the top-confidence candidates from top to bottom. Records without candidates stay in the queue rather than being faked to country centroids.
Recommended single-object flow:
1. Open a compute center or BGP observer detail card.
2. Click "Collect Coordinate Candidates".
3. Wait for candidates; entries that depend on WebSearch / AI factcheck show collection state.
4. Preview candidate positions on Earth.
5. Save the candidate when it is credible; closing the card does not lose the current task state.
Adopt All is for batch processing the compute-center unresolved queue. It starts from the top and adopts the highest-confidence candidate. Records without factual support remain in the queue. When WebSearch is disabled, single locate and Adopt All are disabled because location validation depends on factual lookup.
### Settings
@@ -348,7 +361,7 @@ Mobile uses a drawer layout: layer control moves into a drawer; search/settings/
Docs at `http://localhost:3000/docs` are served by the backend with access control, not bundled into the frontend build.
Anonymous visitors see only `public` docs: README, Quickstart, Manual, FAQ, Earth Location Candidate Collection User Guide. Authenticated users with Gatekeeper groups see more:
Anonymous visitors see only `public` docs: README, Quickstart, Manual, and FAQ. Authenticated users with Gatekeeper groups see more:
- `docs_user`: end-user operational docs
- `docs_developer`: Earth, frontend, backend, collectors, AI Provider development docs
@@ -362,5 +375,4 @@ Docs supports: category navigation, Markdown rendering, tables and code blocks,
- [Quickstart](/home/ray/dev/linkong/planet/docs/technical/en/quickstart.md)
- [FAQ](/home/ray/dev/linkong/planet/docs/technical/en/faq.md)
- [Earth Location Candidate Collection User Guide](/home/ray/dev/linkong/planet/docs/technical/en/location-pipeline-user.md)
- [Planet Ops Runbook](/home/ray/dev/linkong/planet/docs/technical/en/ops-runbook.md)
- [Intelligent Planet Ops Runbook](/home/ray/dev/linkong/planet/docs/technical/en/ops-runbook.md)

View File

@@ -1,6 +1,6 @@
# Naming Glossary
This document standardizes terms used across the Planet console, Earth, backend APIs, and documentation. When adding Chinese UI labels, chart labels, or translated documentation, prefer the Chinese display names listed here to avoid unnecessary mixed Chinese/English copy.
This document standardizes terms used across Intelligent Planet, the console, backend APIs, and documentation. When adding Chinese UI labels, chart labels, or translated documentation, prefer the Chinese display names listed here to avoid unnecessary mixed Chinese/English copy.
## Usage Rules
@@ -14,9 +14,9 @@ This document standardizes terms used across the Planet console, Earth, backend
| English / Key | Chinese Display Name | Usage |
| --- | --- | --- |
| Planet | Planet | Product name |
| Admin Next | 控制台 | New admin console |
| Earth | Earth | Visualization product name |
| Planet | 智能星球 | Product name and primary experience entry |
| Admin | 控制台 | Admin console context |
| Earth | 智能星球 | Visualization product name and `/earth` entry |
| datasource | 数据源 | APIs, lists, filters |
| collector | 采集器 | Collection jobs and credential configuration |
| collected data | 采集数据 | Data list and statistics |

View File

@@ -1,6 +1,6 @@
# Planet Ops Runbook
# Intelligent Planet Ops Runbook
This runbook is for deployment, on-call, and maintenance engineers. End-user UI flows live in the [Planet Manual](/home/ray/dev/linkong/planet/docs/technical/en/manual.md); this document only covers shell, Docker, logs, environment variables, and troubleshooting.
This runbook is for deployment, on-call, and maintenance engineers. End-user UI flows live in the [Intelligent Planet Manual](/home/ray/dev/linkong/planet/docs/technical/en/manual.md); this document only covers shell, Docker, logs, environment variables, and troubleshooting.
## First Startup
@@ -21,7 +21,7 @@ First startup seeds two default accounts (see `DEFAULT_LOGIN_USERS` in `backend/
| Username | Password | Role |
| --- | --- | --- |
| `admin` | `admin123` | `super_admin` |
| `linkong` | `12345678` | `super_admin` |
| `linkong` | `LK12345678` | `super_admin` |
Both seed accounts are created with `email_verified = TRUE` and can log into the console immediately. Any other account must either go through the public registration flow described in the Manual, or be created via `./planet.sh createuser`.
@@ -61,6 +61,22 @@ Per-module restart:
Per-module restart is preferred during development to avoid interrupting unrelated services.
## Destructive Reset
```bash
./planet.sh destroy
```
`destroy` returns a local development environment to a near-empty project state. It requires typing `Y` before it runs; source files and existing `.env` files are preserved.
Cleanup order and boundaries:
- If `planet_postgres` is running, the script first clears the `public` schema in `planet_db`. This prevents old `collected_data.is_current = true` rows from making Earth OOBE report `ready=true` if Docker volume removal later fails.
- Docker cleanup targets resources whose Compose project is `planet`, plus the explicit volumes `planet_postgres_data`, `planet_redis_data`, `postgres_data`, and `redis_data`; do not delete unlabeled volumes by a broad `planet_*` pattern, because another local project could own them.
- Local build state removes `.venv`, frontend `node_modules` / `dist`, Planet state, and scattered Python / Vite cache directories. `$PLANET_CACHE_DIR/downloads` is preserved so upstream raw downloads such as CelesTrak can survive database resets and local rebuild cleanup.
After the reset, run `./planet.sh init` again to recreate tables and default seed data. Old collected records are not restored, and Earth OOBE is evaluated from the backend's real collection state on the next visit. When CelesTrak later returns its "GP data has not updated" HTTP 403, the backend first reuses the preserved download cache to repopulate the database; if no cache exists, wait for the next CelesTrak update window or use Space-Track as a fallback.
## Health Check
```bash
@@ -120,6 +136,20 @@ Useful for:
- Demoing Earth from a phone or tablet
- Other LAN machines reaching the same dev instance
On Windows, the repository-root `planet.cmd` can be used as a one-click entrypoint. It requests Administrator privileges, enters the `Ubuntu` WSL distribution at `/home/linkong/planet`, runs `./planet.sh restart --allow-lan`, opens `http://localhost:3000/earth` after a successful restart, and leaves the terminal inside a WSL shell for log inspection. If the local WSL distribution name or checkout path differs, adjust the `wsl.exe -d ... --cd ...` arguments in `planet.cmd` first.
On a new Windows machine, check the WSL generation first:
```powershell
wsl -l -v
```
Planet development should use WSL2. WSL1 has different networking, filesystem, and process behavior, and can surface as Bun package-manager commands returning only `An unknown error occurred (Unexpected)`, unstable port release, or LAN behavior that does not match the script's assumptions. Convert the distribution if it still runs as WSL1:
```powershell
wsl --set-version Ubuntu 2
```
`--allow-lan` directly exposes the frontend, backend, and AI Provider from the development machine: frontend `3000`, backend `8000`, and AI Provider `8010`. Before startup, the script checks all three ports. If WSL/Linux cannot release a port and a Windows-side listener or stale `portproxy` rule owns it, the script requests Administrator PowerShell cleanup. When Planet runs in WSL, Windows can usually reach it through `localhost`; other LAN machines reaching the Windows LAN IP still need Windows Firewall allow rules.
Diagnose in this order:
@@ -184,13 +214,32 @@ The AI Provider image only rebuilds when code, Dockerfile, Compose config, or Py
./planet.sh restart -a
```
Rebuild detection is based on a content fingerprint rather than only file mtimes. `planet.sh` hashes the `aiprovider/` files, `aiprovider/Dockerfile`, `pyproject.toml`, `uv.lock`, `PYTHON_IMAGE`, `UV_IMAGE`, and the dependency fingerprint into `AI_PROVIDER_BUILD_FINGERPRINT`; Docker writes it into the image label `planet.aiprovider.build-fingerprint`. If the existing `planet-aiprovider:latest` image has a matching label, the script skips rebuild and refreshes the local stamp. Older images without the label fall back to the state/cache stamp.
Docker builds use `uv sync --frozen`. To make container builds reuse the host uv mirror configuration, the script resolves the first config file in this order and mounts it into the build as a BuildKit secret at `/root/.config/uv/uv.toml`:
1. The current `UV_CONFIG_FILE`
2. Repository-root `uv.toml`
3. `${XDG_CONFIG_HOME:-~/.config}/uv/uv.toml`
4. `~/.uv/uv.toml`
If none exists, the script creates an empty state file for the secret so Compose does not fail on a missing file. Before Docker build it unsets `UV_DEFAULT_INDEX`, `UV_INDEX_URL`, and `UV_EXTRA_INDEX_URL`, keeping the build tied to the explicit `UV_CONFIG_FILE`. For a temporary Tsinghua mirror, place this in repository-root `uv.toml`:
```toml
[[index]]
name = "tsinghua"
url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple/"
default = true
```
Diagnose slow builds:
| Symptom | Common cause | Fix |
| --- | --- | --- |
| Large `transferring context` | build context includes unrelated frontend / data files | `.dockerignore` ships only required files |
| `uv sync` is slow | first build or cold cache | wait for the first build; later runs reuse BuildKit cache |
| `uv sync --frozen` is slow | first build, cold cache, or missing uv mirror config | wait for the first build; later runs reuse BuildKit cache; configure `uv.toml` when needed |
| Old keys still in effect after edit | container not restarted | `./planet.sh restart -a` |
| Code changed but the image did not rebuild | fingerprint still matches the image label | Confirm the change is under `aiprovider/`, Dockerfile, or Python dependency inputs; delete `planet-aiprovider:latest` and retry if needed |
## SMTP Email (Required for Public Registration)
@@ -217,6 +266,15 @@ One-time codes are stored in Redis under `otp:{purpose}:{email}` with a 600-seco
## Development Command Conventions
Backend setup and script initialization use the lockfile:
```bash
uv python install 3.14
uv sync --frozen --group dev
```
`--frozen` rejects implicit `uv.lock` rewrites, which is the desired behavior on new machines, CI, and Docker builds. Dependency upgrades should explicitly update `pyproject.toml` / `uv.lock` on a development machine and commit the lockfile.
Frontend must use Bun:
```bash

View File

@@ -0,0 +1,160 @@
# Business Architecture and Data Flows
This document is the business entry point for Intelligent Planet data products. It explains why each Earth data type exists, where it is collected from, which fact or derived tables it uses, and how cache invalidation plus WebSocket hints reach Earth. Frontend, backend, and Earth docs should focus on implementation details; start here when you need the cross-system data flow.
## Overview
Intelligent Planet data moves through three stages:
1. **Collect and normalize**: built-in collectors, admin actions, or the location pipeline write PostgreSQL. Generic raw output lands in `collected_data`; layer-ready projections land in derived tables.
2. **Project and broadcast**: database triggers write fact changes to the `earth_data_change_events` outbox and wake the backend listener with `LISTEN/NOTIFY`. The listener maps the change through layer adapters, invalidates cache, and broadcasts `earth_updates`.
3. **Reload and render in Earth**: Earth receives layer-level refresh hints, applies `clear_then_reload`, `reload`, or `delta`, then refetches `/api/v1/visualization/...`.
```mermaid
flowchart LR
Source["External source / admin action"] --> Collector["Collector or data job"]
Collector --> Facts["collected_data"]
Collector --> Derived["Derived tables"]
Facts --> Outbox["earth_data_change_events"]
Derived --> Outbox
Outbox --> Listener["DB change listener"]
Listener --> Cache["Earth cache invalidation"]
Listener --> WS["earth_updates"]
WS --> Earth["Earth layer reload"]
Earth --> API["Visualization APIs"]
```
`LISTEN/NOTIFY` is only the low-latency wakeup path. The reliable source is the outbox. A real zero-data state is valid and APIs should return HTTP 200 with an empty collection; only actual endpoint failures should return 5xx.
## Data Products
```mermaid
flowchart TB
Satellites["Satellite TLE"] --> SatelliteLayer["satellites layer"]
Cables["Cables + landing points"] --> CableLayer["cables layer"]
Compute["TOP500 / AI GPU / HF"] --> ComputeLocations["compute_center_locations"]
ComputeLocations --> ComputeLayer["computeCenters layer"]
BgpRaw["RIS Live / BGPStream / Prefix"] --> BgpDerived["bgp_observations / anomalies / incidents"]
BgpDerived --> BgpLayer["bgp layer"]
VesselRaw["AIS / BarentsWatch"] --> VesselDerived["vessel_static / vessel_position"]
VesselDerived --> VesselLayer["vessels layer"]
Interactables["earth_interactables"] --> InteractableLayer["interactables layer"]
NewsRaw["RSS / Live / News"] --> NewsItems["earth_news_items"]
NewsItems --> NewsLayer["news / media layer"]
```
| Data product | Business use | Fact sources | Derived / dimension tables | Earth layer | Strategy |
| --- | --- | --- | --- | --- | --- |
| Satellites | In-orbit objects, tracks, coverage, and cruise targets | `celestrak_tle`, `spacetrack_tle` | No long-lived projection; APIs convert TLE on demand | `satellites` | `clear_then_reload` |
| Cables and landing points | Submarine connectivity, landing cities, cable detail, and search | `arcgis_cables`, `arcgis_landing_points`, TeleGeography / FAO landing sources | Cable relation and landing aggregation data | `cables` | `clear_then_reload` |
| Compute centers | TOP500, AI GPU, model platform facilities, and location-completion state | `top500`, `epoch_ai_gpu`, HuggingFace sources | `compute_center_locations` | `computeCenters` | `reload` |
| BGP context | Collectors, anomalies, incidents, route events, and regional context | `ris_live_bgp`, `bgpstream_bgp`, prefix geography sources | `bgp_observations`, `bgp_anomalies`, `bgp_incidents`, `bgp_collector_locations` | `bgp` | `clear_then_reload` |
| Vessels | AIS vessels, positions, tracks, legend, and source health | AIS sources, `barentswatch_vessels` | `vessel_static`, `vessel_position`, `ais_raw_observations`, `ais_source_health` | `vessels` | `clear_then_reload` |
| Interactables | Generic surface icons, manual objects, and future small layers | `earth_interactables` | None | `interactables` | `delta` |
| News and media | Earth news, live streams, cruise summaries, and situation content | news sources | `earth_news_items` | `news` / `media` | `reload` |
## Satellites
Satellite data powers online satellite markers, tracks, footprint policy, and cruise lists. CelesTrak or Space-Track collectors fetch TLE and write `collected_data`. The visualization API converts TLE into current positions and trails on request.
- **Collection entry**: CelesTrak TLE, Space-Track TLE.
- **Fact table**: `collected_data.source IN ('celestrak_tle', 'spacetrack_tle')`.
- **API**: satellite visualization API reads TLE and builds the Earth payload.
- **Delete semantics**: deleting the source broadcasts `satellites` with `clear_then_reload`; the frontend clears satellite markers and trails before refetching. With no TLE, the API returns an empty list.
- **Common failure**: summary is zero but Earth still shows satellites. Usually WS did not fire, the adapter did not cover the source, or the frontend did not clear existing Three.js objects.
## Cables and Landing Points
Cables and landing points show global network connectivity, landing cities, cable details, and search targets. Cable lines and landing points are one business layer; deleting either side must refresh `cables`.
- **Collection entry**: ArcGIS cables, ArcGIS landing points, and TeleGeography / FAO landing related sources.
- **Fact table**: cable and landing point sources in `collected_data`.
- **Derived data**: cable relation tables, landing point aggregation, interface cache.
- **API**: cable visualization API returns cables, landing points, and relation data.
- **Delete semantics**: deleting cable or landing source clears owned projections and broadcasts `cables` with `clear_then_reload`. Empty data returns HTTP 200, not 404.
- **Common failure**: cables disappear for a second and return. Usually a collection replacement or cache refresh window reuses old cache; check duplicate broadcasts and cache patterns.
## Compute Centers
Compute centers represent supercomputers, AI GPU sites, model-platform facilities, and unresolved-location state. Raw sources often only provide organization, country, site, or fuzzy location, so Earth rendering depends on `compute_center_locations`.
- **Collection entry**: TOP500, Epoch AI GPU, HuggingFace related sources.
- **Fact table**: compute sources in `collected_data`.
- **Dimension table**: `compute_center_locations` stores adopted coordinate candidates.
- **API**: compute center visualization API merges raw records with location rows.
- **Delete semantics**: deleting TOP500 or similar sources refreshes `computeCenters`; changing the location table also refreshes the layer. The default is `reload`, because location updates do not always need an immediate clear.
- **Common failure**: collection finished but Earth count did not change. Check visualization cache, location rows, and adapter coverage.
## BGP
BGP data shows route collectors, anomalies, incidents, diffusion rings, and situation summaries. Earth does not render raw BGP rows directly; it renders projected observations, anomalies, and incidents. This is the most common path where deleting raw data can leave old objects visible.
```mermaid
sequenceDiagram
participant UI as Admin delete button
participant Job as clear_data job
participant DB as PostgreSQL
participant Outbox as earth_data_change_events
participant Listener as DB change listener
participant Earth as Earth frontend
UI->>Job: Submit delete for ris_live_bgp / bgpstream_bgp
Job->>DB: Delete collected_data raw rows
Job->>DB: Delete BGP-owned derived tables
DB->>Outbox: trigger writes bgp layer changed
DB-->>Listener: LISTEN/NOTIFY wakeup
Listener->>Listener: merge events and clear bgp cache
Listener-->>Earth: broadcast earth_updates clear_then_reload
Earth->>Earth: clear BGP objects
Earth->>DB: refetch projections through visualization API
```
- **Collection entry**: RIPE RIS Live BGP, CAIDA BGPStream Backfill, IPtoASN / OpenGeoFeed / NRO prefix geography.
- **Fact table**: BGP and prefix sources in `collected_data`.
- **Derived tables**: `bgp_observations`, `bgp_anomalies`, `bgp_incidents`, `bgp_collector_locations`.
- **API**: BGP visualization API reads projections and combines stored or candidate locations.
- **Delete semantics**: deleting RIS Live or BGPStream raw sources must also clear BGP-owned projections and broadcast `bgp` with `clear_then_reload`. Direct derived-table deletes also write outbox events.
- **Common failure**: collectors or incidents remain after raw rows are removed. Check derived tables first, not only `collected_data`.
## Vessels
Vessel data shows AIS vessels, navigation state, vessel-type legend, and source health. Earth rendering uses position snapshots and static vessel information; it should not render every raw AIS observation.
- **Collection entry**: AIS sources, BarentsWatch vessels.
- **Fact table**: `collected_data` or AIS raw observation tables.
- **Derived tables**: `vessel_static`, `vessel_position`, `ais_raw_observations`, `ais_source_health`.
- **API**: vessel visualization API returns current vessel markers and detail fields.
- **Delete semantics**: deleting any vessel source lets owned projections broadcast `vessels` with `clear_then_reload`.
- **Common failure**: the count panel changes but vessels remain. Summary and layer data are separate; Earth should clear objects based on layer updates.
## Interactables
`earth_interactables` is the generic surface-icon capability for manual objects, extensions, and future small layers. Unlike most layers, it keeps object-level delta.
- **Fact table**: `earth_interactables`.
- **API**: interactable API and Earth generic icon interface.
- **Strategy**: upsert for create/update and removeItem for delete; no full layer reload.
- **Delete semantics**: delete payloads must include a stable id so the frontend can remove the object.
- **Common failure**: an object cannot be removed. Usually the stable id is missing, the delta handler did not match the object type, or an older duplicate render path still exists.
## News and Media
News and media support the Earth news ticker, live stream panel, news cruise, and situation summaries. They are content refresh paths rather than stable geographic object layers, so they default to `reload`.
- **Collection entry**: RSS, live streams, news sources.
- **Fact table**: news source rows in `collected_data`.
- **Derived table**: `earth_news_items`.
- **API**: news, live stream, and media content APIs.
- **Delete semantics**: deleting news sources or `earth_news_items` broadcasts `news` / `media` reload; empty responses hide the corresponding content.
- **Common failure**: the live panel shows stale content. Usually the media component ignored the layer update or the content API cache was not invalidated.
## Adding a New Layer
Add a new Earth data product in this order:
1. Define the business purpose and Earth layer name.
2. Identify fact sources, fact tables, and derived tables.
3. Register source/table coverage, cache patterns, owned derived cleanup, and default strategy in the backend layer adapter.
4. Ensure the visualization API returns HTTP 200 with an empty collection for zero data.
5. Keep the Earth frontend driven only by layer and strategy, not database table names.
6. Update this document with the data product flow, then document implementation details in the relevant frontend, backend, or Earth technical doc.

View File

@@ -1,8 +1,8 @@
# Quickstart
This quickstart is for Planet end users who just received an access URL and need the shortest path from "open the browser" to "first useful configuration done". Every action happens in the browser.
This quickstart is for Intelligent Planet end users who just received an access URL and need the shortest path from "open the browser" to "first useful configuration done". Every action happens in the browser.
If you are responsible for deployment or operations, read the [Planet Ops Runbook](/home/ray/dev/linkong/planet/docs/technical/en/ops-runbook.md) instead.
If you are responsible for deployment or operations, read the [Intelligent Planet Ops Runbook](/home/ray/dev/linkong/planet/docs/technical/en/ops-runbook.md) instead.
## 1. Open the URL
@@ -34,7 +34,7 @@ After landing on the `/admin` dashboard, here's a recommended walk-through:
1. `/collection-management?tab=collector_credentials`: pick a collector and click the plug icon to test connectivity. Free collectors (e.g. open BGP) usually work right away; credential-bearing ones like `AISStream` or `BarentsWatch` need an API key / client secret first
2. `/ai?tab=providers`: fill an LLM provider (e.g. `minimax` / `openai`), model, base URL, API key, and click the plug at the end of the base URL to test. WebSearch / OCR tools are optional
3. `/datasources` or `/data`: check whether collectors have produced data. Use `/datasources -> Collection Tasks` for finite collectors, and `/datasources -> Realtime Streams` for AISStream / WebSocket health and counters
3. `/datasources` or `/data`: check whether collectors have produced data. Use `/datasources -> Built-in Sources` for finite collectors: with no rows selected, click `Trigger All`; after selecting rows, the primary button becomes `Trigger Selected N`. The top-right queue button shows progress. Use `/datasources -> Realtime Sources` for AISStream / WebSocket health and counters
4. `/alerts/system`: verify system alerts look right
5. `/users` (super_admin only): open accounts for teammates or adjust their groups
@@ -42,6 +42,8 @@ After landing on the `/admin` dashboard, here's a recommended walk-through:
Visit `/earth`. This is a public page — no login required.
If the system has no collected data yet, Earth shows an initialization guide that points you to sign in and trigger collection. This guide is backend-state driven, so clearing browser storage does not make it appear once the system is ready.
Once in, verify:
- The globe renders, and the right-side layer panel can toggle layers
@@ -60,7 +62,7 @@ Open `/forgot-password`, enter your email, receive a code, then enter the code p
## Next Steps
- Full UI walkthrough: [Planet Manual](/home/ray/dev/linkong/planet/docs/technical/en/manual.md)
- Full UI walkthrough: [Intelligent Planet Manual](/home/ray/dev/linkong/planet/docs/technical/en/manual.md)
- Troubleshooting and configuration questions: [FAQ](/home/ray/dev/linkong/planet/docs/technical/en/faq.md)
- Detailed Earth coordinate candidate flow: [Earth Location Candidate Collection User Guide](/home/ray/dev/linkong/planet/docs/technical/en/location-pipeline-user.md)
- Deployment / operations commands: [Planet Ops Runbook](/home/ray/dev/linkong/planet/docs/technical/en/ops-runbook.md)
- Detailed Earth coordinate candidate flow: see the Earth section in [Intelligent Planet Manual](/home/ray/dev/linkong/planet/docs/technical/en/manual.md)
- Deployment / operations commands: [Intelligent Planet Ops Runbook](/home/ray/dev/linkong/planet/docs/technical/en/ops-runbook.md)

View File

@@ -1,6 +1,6 @@
# Tactile UI Components
Tactile UI is Planet's portable React control layer. It was extracted from the Admin Next button, switch, scrollbar, and tooltip work, but the components themselves do not depend on Admin Next, AntD, Radix, Tailwind, or `an-*` classes. The immediate goal is stable in-repo usage; the structure is intentionally close to something that can later be published as an npm package.
Tactile UI is Planet's portable React control layer. It was extracted from the Admin button, switch, scrollbar, and tooltip work, but the components themselves do not depend on Admin, AntD, Radix, Tailwind, or `an-*` classes. The immediate goal is stable in-repo usage; the structure is intentionally close to something that can later be published as an npm package.
## Design Goals
@@ -49,6 +49,22 @@ Core tokens are exposed as `--tui-*` CSS variables. Product themes should overri
Most controls also accept a `tactile` prop for local width, height, radius, background, border, and shadow overrides. Use local overrides for small special cases; use CSS variables for product-wide styling.
## Portals and Dark Theme
`TactileTooltip` and the Admin `Dialog`, `Select`, and toast controls that use Tactile UI may render through a portal attached to `document.body`. Those nodes are not descendants of `.admin-theme-root[data-theme='dark']`, so dark tokens cannot rely only on an ancestor selector inside the Admin root.
The Admin theme provider mirrors the active theme to `body[data-admin-theme]`. Shared styles need to support both selector paths:
```css
[data-theme='dark'] .tui-button,
body[data-admin-theme='dark'] .tui-button {
--tui-surface: #172033;
--tui-text: #e5edf8;
}
```
When adding a portal-based control, first check whether it renders into body. If it does, add a `body[data-admin-theme='dark']` branch in that component's style entry, or reuse the already covered `--tui-*` / `--an-*` tokens. Avoid hard-coding a one-off dark modal style, because the same contrast problem can reappear in dropdowns, tooltips, toasts, and confirmation dialogs.
## `TactileButton`
The button component covers regular buttons, icon buttons, strong-intent buttons, and link-like buttons.
@@ -74,6 +90,8 @@ Common props:
`variant="neutral"` defaults to a white tactile button. Colored buttons should still keep the same height and external shadow instead of relying on page-specific CSS overrides.
Colored button borders must not use the exact fill color. `primary`, `danger`, and future colored variants should use a lighter border from the same hue, such as `color-mix(in srgb, var(--tui-danger) 64%, white)`. The border still reads as part of the button color, but its visual weight is lower than the fill surface, so red or blue buttons do not look one outline larger than neutral buttons. Hover states should brighten rather than darken: mix a little white into the current fill color, and keep the hover border lighter than the hover fill.
## Icon Presets
Preset icons are maintained in `tactileIconPresets`. Feature pages should call icons by semantic name so actions remain consistent across the console.
@@ -177,7 +195,7 @@ Do not force overlay scrollbars onto textareas. Text selection and the resize gr
## `TableScrollRegion`
`TableScrollRegion` is a convenience wrapper for table scroll areas. The default target selector is `.tui-scroll-target`; Admin Next passes its table viewport selector explicitly so package code does not contain product-specific names.
`TableScrollRegion` is a convenience wrapper for table scroll areas. The default target selector is `.tui-scroll-target`; Admin passes its table viewport selector explicitly so package code does not contain product-specific names.
```tsx
<TableScrollRegion targetSelector=".table-viewport">
@@ -192,9 +210,9 @@ Do not force overlay scrollbars onto textareas. Text selection and the resize gr
- Tooltip text is explanatory only; state must still be represented by text, badges, or `aria-*` attributes.
- Disabled and loading states set `aria-disabled`; real `button` elements also receive `disabled`.
## Admin Next Migration Rules
## Admin Migration Rules
Admin Next should use Tactile UI for global tool buttons, detail-panel toolbars, and list footer actions:
Admin should use Tactile UI for global tool buttons, detail-panel toolbars, and list footer actions:
- Unambiguous actions: `TactileButton iconOnly tooltip`
- Save/create/confirm: `TactileButton variant="primary"`, usually with text

View File

@@ -1,44 +1,52 @@
# 技术文档
这里放“当前实现和当前结构”的文档,重点回答:
这里是智能星球当前文档入口。文档按读者和问题类型分层:先看业务架构理解数据产品,再进入使用手册或技术实现文档。
- 现在代码是怎么组织的
- 当前入口在哪
- 状态和组件如何工作
## 业务架构
适合放入这里的内容:
- [业务架构与数据流转](/home/ray/dev/linkong/planet/docs/technical/zh/platform-data-flows.md):每类智能星球数据的用途、采集链路、事实表、派生表、缓存和 WebSocket 广播链路
- [命名与术语对照](/home/ray/dev/linkong/planet/docs/technical/zh/naming-glossary.md):智能星球、控制台、后端和文档常见名词的中英对照
- 快速开始和使用手册
- 前端上下文
- Earth 前端结构
- Earth 卫星覆盖策略
- Earth 渲染图层顺序
- Earth 图层样式属性索引
- 后端运行控制
- 采集器现状
- 采集器设置与连接验证
- 采集格式约定
## 使用手册
## 使用入口
- [快速开始](/home/ray/dev/linkong/planet/docs/technical/zh/quickstart.md):从零启动智能星球的最短路径
- [智能星球使用手册](/home/ray/dev/linkong/planet/docs/technical/zh/manual.md):控制台、智能星球、文档和常用功能的用户操作说明
- [常见问题](/home/ray/dev/linkong/planet/docs/technical/zh/faq.md)Windows / WSL、端口、依赖、动捕、凭证和文档权限排障
- [快速开始](/home/ray/dev/linkong/planet/docs/technical/zh/quickstart.md):从零启动 Planet 的最短路径
- [Planet 使用手册](/home/ray/dev/linkong/planet/docs/technical/zh/manual.md):控制台、`planet.sh`、Earth 和 Docs 的完整使用手册
- [常见问题](/home/ray/dev/linkong/planet/docs/technical/zh/faq.md)Windows / WSL、端口、依赖、动捕、凭证和 Docs 权限的集中排障入口
- [Earth 位置候选采集使用手册](/home/ray/dev/linkong/planet/docs/technical/zh/location-pipeline-user.md):在 Earth 上为算力中心和 BGP 观测站采集、预览坐标候选
- [数据源、采集器设置与连接验证](/home/ray/dev/linkong/planet/docs/technical/zh/datasource-collector-settings-connectivity.md)数据源目录、采集器设置、连接验证、BarentsWatch 凭证链路
- [通用位置估算管线开发说明](/home/ray/dev/linkong/planet/docs/technical/zh/location-pipeline-development.md):后端 location resolver / pipeline 的接口、注册表和扩展方式
- [Docs Gatekeeper 开发说明](/home/ray/dev/linkong/planet/docs/technical/zh/docs-gatekeeper-development.md):后端 Docs 目录、正文读取和 Gatekeeper 权限组实现
- [命名与术语对照](/home/ray/dev/linkong/planet/docs/technical/zh/naming-glossary.md)控制台、Earth、后端和文档常见名词的中英对照
- [Earth 可交互图标接入](/home/ray/dev/linkong/planet/docs/technical/zh/earth-interactable-usage.md)Earth 地表可交互图标 `Interactable` 的接口、生命周期和接入示例
- [Earth 工具栏与浮层协同](/home/ray/dev/linkong/planet/docs/technical/zh/earth-toolbar-overlay-coordination.md):工具栏按钮与搜索 / 设置 / 新闻 / 图层浮层之间的关闭矩阵和接入规则
- [Tactile UI 组件库](/home/ray/dev/linkong/planet/docs/technical/zh/tactile-ui-components.md)可移植按钮、开关、tooltip 和滚动条组件的 API、主题 token 与迁移约定
## 智能星球技术实现
不适合放入这里的内容:
- [智能星球前端结构](/home/ray/dev/linkong/planet/docs/technical/zh/earth-frontend-context.md)智能星球页面模块、状态、WebSocket 刷新和图层生命周期
- [智能星球图层样式属性索引](/home/ray/dev/linkong/planet/docs/technical/zh/earth-layer-style-reference.md):图层颜色、符号、材质和视觉参数
- [智能星球渲染图层顺序](/home/ray/dev/linkong/planet/docs/technical/zh/earth-render-layer-order.md)renderOrder、深度策略、拾取和同坐标避让
- [智能星球卫星覆盖策略](/home/ray/dev/linkong/planet/docs/technical/zh/earth-satellite-footprint-policy.md):卫星 footprint 的显示边界和策略
- [BGP 态势上下文](/home/ray/dev/linkong/planet/docs/technical/zh/earth-bgp-context.md)BGP 在智能星球中的渲染、聚合和观测站实现
- [智能星球可交互图标接入](/home/ray/dev/linkong/planet/docs/technical/zh/earth-interactable-usage.md)`Interactable` 的接口、生命周期和接入示例
- [智能星球工具栏与浮层协同](/home/ray/dev/linkong/planet/docs/technical/zh/earth-toolbar-overlay-coordination.md):工具栏按钮与搜索、设置、新闻、图层浮层的关闭矩阵
- 尚未完成的路线图
- 未来迭代方案
- 大范围重构计划
## 前端技术实现
这些应放入:
- [控制台前端结构](/home/ray/dev/linkong/planet/docs/technical/zh/frontend-admin-frontend-context.md):控制台页面结构、状态边界和懒加载策略
- [前端布局指南](/home/ray/dev/linkong/planet/docs/technical/zh/frontend-layout-guidelines.md):布局、表格、面板和响应式约束
- [Tactile UI 组件库](/home/ray/dev/linkong/planet/docs/technical/zh/tactile-ui-components.md)按钮、开关、tooltip、滚动条组件 API 和主题 token
- [计划文档索引](/home/ray/dev/linkong/planet/docs/plans/README.md)
## 后端技术实现
- [数据采集系统](/home/ray/dev/linkong/planet/docs/technical/zh/backend-collectors.md):采集器、任务类型、保存层和状态回写
- [系统服务控制](/home/ray/dev/linkong/planet/docs/technical/zh/backend-system-service-control.md):后端服务控制和系统操作接口
- [数据源、采集器设置与连接验证](/home/ray/dev/linkong/planet/docs/technical/zh/datasource-collector-settings-connectivity.md):数据源目录、采集器设置、连接验证和凭证链路
- [数据源 API 性能](/home/ray/dev/linkong/planet/docs/technical/zh/backend-datasources-api-performance.md)DataSources 列表接口性能和缓存策略
- [数据作业与 Outbox 技术架构](/home/ray/dev/linkong/planet/docs/technical/zh/data-job-earth-sync-architecture.md)PostgreSQL 作业队列、outbox、listener 和 Kafka / Spark 演进边界
- [通用位置估算管线开发说明](/home/ray/dev/linkong/planet/docs/technical/zh/location-pipeline-development.md)location resolver / pipeline 的接口、注册表和扩展方式
- [新闻直播采集格式](/home/ray/dev/linkong/planet/docs/technical/zh/earth-news-live-streams-collector-format.md):新闻、直播和媒体采集 payload 约定
- [Docs Gatekeeper 开发说明](/home/ray/dev/linkong/planet/docs/technical/zh/docs-gatekeeper-development.md):后端文档目录、正文读取和 Gatekeeper 权限组实现
## 智能体与运维
- [AI Provider 指南](/home/ray/dev/linkong/planet/docs/technical/zh/agents-aiprovider.md):模型供应商适配、任务 prompt 和调用边界
- [智能星球运维手册](/home/ray/dev/linkong/planet/docs/technical/zh/ops-runbook.md):部署、启动、排障和敏感操作
- [Docker + Compose + Buildx 升级](/home/ray/dev/linkong/planet/docs/technical/zh/ops-docker-compose-buildx-upgrade.md)Docker 工具链升级步骤
- [planet.sh 启动机制](/home/ray/dev/linkong/planet/docs/technical/zh/ops-planet-sh-startup.md):启动脚本、健康检查和性能优化
## 计划与历史
尚未完成的路线图、大范围重构方案和未来计划放在仓库路径 `docs/plans/README.md`。已完成、过时或被新实现替代的方案放在 `docs/deprecated/README.md`

View File

@@ -101,7 +101,7 @@ AI 配置页使用的接口:
这些接口都需要用户登录。`secrets` 接口只用于配置页点击显示 key/token 时取回明文,隐藏时前端恢复为脱敏预览。
Admin Next 的 AI 页面按旧版信息架构组织为:
Admin 的 AI 页面按业务信息架构组织为:
- `模型供应商`
- 管理 provider、协议适配、默认模型、LLM API Key、代理地址、代理 token、模型列表刷新、设为默认和轻量连通性测试。
@@ -272,7 +272,7 @@ AI 配置仍保存在 PostgreSQL不写入 JSON 文件。核心结构如下:
`.env` 只是兜底。配置页保存或测试连接成功后PostgreSQL 中的配置会成为全局默认。
Admin Next 的密钥状态必须按 provider / tool 精确判断:
Admin 的密钥状态必须按 provider / tool 精确判断:
- 数据库中当前 provider/tool 有密钥时,显示为“已配置”。
- 数据库没有密钥,但 fallback provider、model 或 tool 与当前项匹配时,可以显示 fallback 的脱敏预览。
@@ -283,7 +283,7 @@ Admin Next 的密钥状态必须按 provider / tool 精确判断:
### 轻量连通性测试
Admin Next 的插头按钮走轻量连通性测试,不承担保存职责。业界常见做法是分两层:
Admin 的插头按钮走轻量连通性测试,不承担保存职责。业界常见做法是分两层:
- 快速检查 provider 目录或低成本 endpoint确认 base URL、鉴权和当前模型是否可达。
- 只有在用户明确运行 Playground 或业务任务时才发完整模型请求。

View File

@@ -73,6 +73,8 @@ async def run(self, db):
**核心文件**: `backend/app/services/collectors/base.py`
手动触发、删除数据、清理缓存现在统一进入 PostgreSQL 数据作业队列,任务账本仍是 `collection_tasks`。采集器只负责 `fetch -> transform -> save`,由 `data_jobs.py` worker 领取 `collect` / `clear_data` / `clear_cache` / `earth_refresh` 任务并回写进度。Earth 图层刷新关系集中在 `earth_layer_adapters.py`,不要再在单个采集器或按钮里手写缓存失效和 WebSocket 广播。
## 三、采集器列表
| 采集器 | 数据类型 | 数据内容 | 采集频率 |
@@ -90,7 +92,7 @@ async def run(self, db):
AIS 船只类采集器和其它 `CollectedData` 采集器的落库路径不同。BarentsWatch、AISStream 和自定义 `vessel_ais` 源都会进入 AIS 原始观测层,随后由聚合服务合并成 Earth 船只图层使用的 GeoJSON 和详情数据。这样做可以保留来源、传输方式、字段冲突和观测时间,避免某个实时源直接覆盖最终展示表。
Earth 国界不再属于采集器体系。它是 Earth 静态渲染资产,由控制台 `运维与配置 -> Earth 内容 -> 国界精度` 维护源配置,并由 `/api/v1/earth/boundaries/*` 构建 `frontend/public/earth/data/boundaries/earth-boundaries-china-pov-v1.pmtiles`。本地没有高精 PMTiles 时,前端会使用仓库内置的低精度 GeoJSON 作为 fallback不会向 `CollectedData` 写入国界记录。
智能星球国界不再属于采集器体系。它是智能星球静态渲染资产,由控制台 `运维与配置 -> 智能星球内容 -> 国界精度` 维护源配置,并由 `/api/v1/earth/boundaries/*` 构建 `frontend/public/earth/data/boundaries/earth-boundaries-china-pov-v1.pmtiles`。本地没有高精 PMTiles 时,前端会使用仓库内置的低精度 GeoJSON 作为 fallback不会向 `CollectedData` 写入国界记录。
TOP500 和 Epoch AI 算力数据的公开源不总是提供可用经纬度。Earth 统一算力中心接口在主地图启动链路中只使用源数据自带坐标或 `compute_center_locations` 维表坐标;缺少坐标的记录会进入 `unresolved`,不会通过本地注册表、国家质心或猜测城市自动渲染。用户手动采集候选时,后端会用源字段调用 ROR 组织注册 API 和 Nominatim/OpenStreetMap 在线搜索;候选经前端保存后写入 `compute_center_locations`,后续地图刷新再从维表渲染。
@@ -230,7 +232,7 @@ if datasource.last_status == "success":
### 采集管理与快照
Admin Next 的采集管理入口按业务层级组织:
Admin 的采集管理入口按业务层级组织:
- `采集器`:配置 endpoint、认证方式、请求头、基础参数、启用状态和凭证教程。
- `采集调度`:查看和调整调度状态,触发、停止或刷新采集任务。

View File

@@ -96,4 +96,4 @@ if (res.data.task_id) {
## 相关文件
- `backend/app/api/v1/datasources.py``_load_datasource_list_context``list_datasources`
- `frontend/src/pages/DataSources/DataSources.tsx``BuiltInDataSource` interface、`triggerDatasource`
- `frontend/src/admin/pages/PlainResourcePages.tsx``BuiltInDataSource` interface、`triggerDatasource`

View File

@@ -0,0 +1,140 @@
# 数据作业与 Outbox 技术架构
本文记录 Planet v1 的任务队列、数据库 outbox 和 Earth 刷新技术边界。每类数据产品的业务用途和端到端链路见 [业务架构与数据流转](/home/ray/dev/linkong/planet/docs/technical/zh/platform-data-flows.md)。
## 架构边界
- PostgreSQL 是 v1 的可靠任务账本和 outbox不引入 Kafka、Celery 或 RQ。
- `collection_tasks` 记录采集、删除、清缓存和非数据库触发的 Earth refresh 任务。
- `earth_data_change_events` 记录事实表或派生表变化,是 Earth 同步的可靠来源。
- `LISTEN/NOTIFY` 只做低延迟唤醒listener 仍会轮询未消费 outbox。
- Redis 主要用于缓存、认证辅助、OTP / rate limit、临时日志和 WebSocket 辅助,不是可靠队列。
## 数据库变化同步
```mermaid
flowchart LR
Write["Fact or derived table write"] --> Trigger["PostgreSQL trigger"]
Trigger --> Outbox["earth_data_change_events"]
Trigger --> Notify["planet_earth_data_changes"]
Outbox --> Listener["earth_db_change_listener"]
Notify --> Listener
Listener --> Adapter["earth_layer_adapters"]
Adapter --> Cache["cache invalidation"]
Cache --> WS["earth_updates"]
```
1. 采集、删除、定位或派生任务写入事实表或派生表。
2. statement trigger 为 `INSERT` / `UPDATE` / `DELETE` 写入 outbox。
3. listener 被 notify 唤醒,或通过轮询发现未消费事件。
4. listener 使用 `earth_layer_adapters.py``table + source` 映射成 Earth layer、刷新策略和 cache pattern。
5. listener 短窗口合并同 layer 事件,失效缓存并广播 `earth_updates`
6. 广播成功后标记 outbox consumed失败时保留待重试。
DB 变化不再默认创建 `earth_refresh` 任务,因此不会被同 source 的长采集或删除任务阻塞。`earth_refresh` 只保留给手动清缓存和非 DB 变化刷新提示。
## 数据作业队列
`collection_tasks` 是统一 job ledger。worker 使用 PostgreSQL `FOR UPDATE SKIP LOCKED` 领取 `queued` 任务;同一 `source` 的写任务串行,不同 source 可并行。
| task_type | 作用 |
| --- | --- |
| `collect` | 执行内置 datasource 采集 |
| `clear_data` | 删除该 source 的采集数据和声明过的派生数据 |
| `clear_cache` | 删除该 source 对应的 Earth / dashboard 缓存 |
| `earth_refresh` | 非 DB 变化场景下失效智能星球图层缓存并广播刷新提示 |
接口只创建任务并返回 `task_id`。任务执行、进度、取消和终态由 worker 写回 `collection_tasks`,并通过 `datasource_tasks` channel 通知前端。
取消语义是“保留已提交批次”:点击停止后,后端把任务标记为 `cancelling` 并取消内存中的执行协程;已经提交的批次保留,未完成批次按采集器或清理任务的回滚逻辑处理。
## Earth 同步事件模型
统一事件模型是 `earth.layer.changed`
```json
{
"event": "earth.layer.changed",
"action": "database_changed",
"source": "celestrak_tle",
"table": "collected_data",
"operation": "DELETE",
"layers": ["satellites"],
"refresh_strategy": "clear_then_reload",
"records_processed": 11125,
"occurred_at": "2026-05-25T10:20:30Z"
}
```
| strategy | 用途 |
| --- | --- |
| `clear_then_reload` | 先清前端本地图层对象,再强制重拉接口。删除数据时优先使用。 |
| `reload` | 保留旧对象直到新数据返回,适合定位、元数据或非破坏性更新。 |
| `delta` | 只用于 `earth_interactables`,按 id upsert 或 remove。 |
接口在真实 0 数据时必须返回 200 和空集合;只有真实接口异常才返回 5xx。前端收到删除事件后如果重拉失败应保持已清空状态并显示轻量错误不恢复旧对象。
## Layer Adapter 约定
`earth_layer_adapters.py` 是 source、派生表、Earth layer、缓存和刷新策略的唯一注册表。新图层只应新增 adapter不应在按钮 handler、采集器或前端分支里手写同步逻辑。
Adapter 必须声明:
- source 或 table 由哪个 Earth layer 消费。
- 需要清理哪些 Earth cache key pattern。
- `clear_data` 删除 source 时是否需要同时删除 owned 派生表。
- 该 layer 的默认刷新策略。
删除 source 时,`clear_data` 作业先删除 `collected_data.source = <source>`,再根据 adapter 删除 owned 派生表。直接修改派生表也会触发 outbox所以后台任务、管理接口和 SQL 修复脚本只要落到事实表或派生表Earth 都能感知变化。
## 运维排障
检查 outbox 是否堆积:
```sql
SELECT id, table_name, operation, source, occurred_at
FROM earth_data_change_events
WHERE consumed_at IS NULL
ORDER BY id
LIMIT 20;
```
检查触发器是否存在:
```sql
SELECT tgname, tgrelid::regclass
FROM pg_trigger
WHERE tgname LIKE 'tr_planet_%_changed_%'
ORDER BY 2, 1;
```
常用日志事件:
- `earth.db_changes.connected`listener 已连接 PostgreSQL 并开始监听。
- `earth.db_changes.outbox_polled`:轮询到了未消费 outbox。
- `earth.db_changes.broadcasted`:已产生 Earth 刷新广播。
- `data_job.started` / `data_job.completed`:任务执行状态。
如果 Earth 没更新按顺序检查事实表是否变化、outbox 是否消费、adapter 是否覆盖对应 `source/table`、listener 是否在线、前端 WebSocket 是否连接、visualization 接口是否返回 200 空集合或新数据。
## Kafka-ready 边界
业务代码不直接依赖具体队列实现,而是通过这些边界组织:
- `JobQueue`:提交、领取、取消、完成数据作业。
- `DataChangeBus`:发布数据库事实变化。
- `EarthLayerAdapterRegistry`:声明 source、layer、缓存和派生数据关系。
需要 Kafka 的信号:
- 多个独立服务需要消费同一批数据变化。
- AIS、BGP 或传感器流达到持续高吞吐。
- 需要 consumer group、事件回放、跨服务解耦。
需要 Spark 的信号:
- 历史数据到千万或亿级PostgreSQL 聚合开始吃力。
- 需要跨源、长时间窗口、空间时间关联分析。
- 原始数据进入 Parquet / Iceberg / Delta 等湖仓,并开始生产离线派生数据产品。
若目标是秒级连续流计算,优先评估 FlinkSpark 更适合批量或微批分析。

View File

@@ -32,8 +32,8 @@
文件:
- [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/pages/PlainResourcePages.tsx)
- [AdminNextRoutes.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/AdminNextRoutes.tsx)
- [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
- [AdminRoutes.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/AdminRoutes.tsx)
当前行为:
@@ -57,11 +57,11 @@
文件:
- [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/pages/PlainResourcePages.tsx)
- [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
当前行为:
- `/collection-management`旧版层级收敛为 `采集器``采集调度``采集历史 / 快照`
- `/collection-management`业务层级收敛为 `采集器``采集调度``采集历史 / 快照`
- `采集器` 是配置页,左侧列表展示采集器配置;右侧表单编辑 endpoint、认证、请求头、采集参数和启用状态。
- 新增采集器和目标 Schema 使用草稿详情页,不再用透明 JSON 弹窗;保存后才固化到列表,取消会销毁草稿。
- 连接按钮只做连通性测试,不保存配置;保存按钮只持久化表单。
@@ -80,7 +80,7 @@
- `目标 Schema` 维护可写入目标结构。
- `run-mapped``stop-mapped``stream-status` 负责运行映射后的自定义采集器。
这些入口在新版中需要表单化,只有高级字段才折叠为 JSON。不要把模板、Schema、运行状态和采集器配置拍平成同一张表。
这些入口在 Admin 中需要表单化,只有高级字段才折叠为 JSON。不要把模板、Schema、运行状态和采集器配置拍平成同一张表。
## 后端接口
@@ -334,7 +334,7 @@ GET /api/v1/vessels/snapshot?bbox=lon_min,lat_min,lon_max,lat_max&zoom=12&limit=
自定义源现在不是独立的新数据孤岛,而是作为内置数据源的补充源写入目标 schema。当前最完整的目标是 `vessel_ais`:自定义 REST 或 WebSocket 源经过确定性 mapping 后写入 AIS raw observations再通过 `vessels` WebSocket channel 推送给 Earth。
Earth 高精度边界不再使用自定义源目标 schema。国界是 Earth 静态资产,由控制台 `运维与配置 -> Earth 内容 -> 国界精度` 保存本机源配置并触发 PMTiles 构建,不写入 `CollectedData`
智能星球高精度边界不再使用自定义源目标 schema。国界是智能星球静态资产,由控制台 `运维与配置 -> 智能星球内容 -> 国界精度` 保存本机源配置并触发 PMTiles 构建,不写入 `CollectedData`
### 配置语义

View File

@@ -2,7 +2,7 @@
Docs Gatekeeper 把 `/docs` 从“前端构建时打包所有 Markdown”改成“后端按权限返回目录和正文”。它的目标是让公开使用手册、用户文档、开发文档和管理/运维文档在同一个 Docs 页面内可检索,但正文读取必须经过服务端白名单和用户权限检查。
用户侧说明见 [Planet 使用手册](/home/ray/dev/linkong/planet/docs/technical/zh/manual.md) 的 Docs 章节。
用户侧说明见 [智能星球使用手册](/home/ray/dev/linkong/planet/docs/technical/zh/manual.md) 的文档章节。
## 鉴权模型
@@ -16,7 +16,7 @@ Docs 使用两层权限:
| 组 | 用途 |
| --- | --- |
| `docs_user` | 用户操作类文档 |
| `docs_developer` | Earth、前端、后端、采集器和 AI Provider 开发文档 |
| `docs_developer` | 智能星球、前端、后端、采集器和 AI Provider 开发文档 |
| `docs_admin` | 服务控制、运维、环境变量和敏感操作文档 |
继承规则:
@@ -24,7 +24,7 @@ Docs 使用两层权限:
- 未登录用户只能读 `public`
- `docs_developer` 隐含 `docs_user`
- `docs_admin` 隐含 `docs_developer``docs_user`
- `admin``super_admin` 默认拥有全部 Docs 权限。
- `admin``super_admin` 默认拥有全部文档权限。
## 后端入口
@@ -61,15 +61,15 @@ DocsMetadata(
"public",
"Manual",
2,
"Planet 使用手册",
"Planet Manual",
"智能星球使用手册",
"Intelligent Planet Manual",
)
```
新增公开文档时,需要同步:
新增可在 Docs 页面展示的技术文档时,需要同步:
- 新增中英文 Markdown 文件。
- 在服务端 `DOCS_METADATA` 添加 filename、slug、access、group、order、标题。
- 在服务端 `DOCS_METADATA` 添加 filename、slug、access、group、order、标题。后端目录接口以这里为准,只改前端 metadata 不会让文档出现在 `/docs` 导航中。
- 在前端 [docs-content.ts](/home/ray/dev/linkong/planet/frontend/src/pages/Docs/docs-content.ts) 添加同名 metadata保持导航标题和排序一致。
- 如果需要从 README 发现,更新 `docs/technical/zh/README.md``docs/technical/en/README.md`
@@ -83,7 +83,7 @@ DocsMetadata(
- 更新用户时校验组名只能是 `docs_user``docs_developer``docs_admin`
- 只有 `super_admin` 能修改 Gatekeeper 权限组。
前端 [Users.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Users/Users.tsx) 展示权限组标签,并在编辑表单中提供多选框。非 `super_admin` 打开的表单会禁用该字段,并在提交前移除 `gatekeeper_groups`
前端 [Users.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/Users.tsx) 展示权限组标签,并在编辑表单中提供多选框。非 `super_admin` 打开的表单会禁用该字段,并在提交前移除 `gatekeeper_groups`
## 前端 Docs 加载

View File

@@ -188,7 +188,7 @@ Earth info-card 策略:
- 这是预期行为,因为 incident 是聚合和去噪后的结果
- 但 incident-first 渲染会让 Earth 显得过于安静,除非有另一层始终可用的 activity layer
推荐 `activity layer` 的实现细节在 [BGP 区域聚合计划](/home/ray/dev/linkong/planet/docs/plans/earth-bgp-region-aggregation-plan.md) 中展开
推荐 `activity layer`未来实现细节保存在仓库路径 `docs/plans/earth-bgp-region-aggregation-plan.md`
因此最近的里程碑是:
@@ -259,7 +259,7 @@ Earth 的 `incident` 层不应该像一大片发光区域,而应该像紧凑
相关页面:
- `frontend/src/pages/BGP/BGP.tsx`
- `frontend/src/admin/pages/PlainResourcePages.tsx`
当前 BGP 控制台页面有三层:
@@ -321,7 +321,7 @@ BGP 专项测试位于:
前端:
- `frontend/src/pages/BGP/BGP.tsx`
- `frontend/src/admin/pages/PlainResourcePages.tsx`
- `frontend/public/earth/js/bgp.js`
- `frontend/public/earth/js/main.js`
- `frontend/public/earth/js/info-card.js`

View File

@@ -1,10 +1,10 @@
# Earth 前端结构
# 智能星球前端结构
本文件描述当前 Earth 大屏前端的真实结构,重点是帮助后续继续改 HUD、图层、媒体面板、真实地形、BGP 可视化时,不再重复踩结构和状态同步上的坑。
本文件描述当前智能星球前端的真实结构,重点是帮助后续继续改 HUD、图层、媒体面板、真实地形、BGP 可视化时,不再重复踩结构和状态同步上的坑。
相关规则建议一起参考:
- [项目规则](/home/ray/dev/linkong/planet/rules.md)
- 仓库根目录 `rules.md`
- [前端布局指南](/home/ray/dev/linkong/planet/docs/technical/zh/frontend-layout-guidelines.md)
## 当前目标
@@ -56,6 +56,8 @@ React 路由入口:
- 各图层集成
- Earth 级别状态同步
Earth 收到 `/ws``earth_updates` 时只把它当作刷新提示,真实数据仍通过 `/api/v1/visualization/...` 接口重新 GET。数据库驱动的刷新由后端 listener 直接清理缓存再广播,不再默认经过 `earth_refresh` 作业队列;前端收到 `database_changed` 后会按 layer 读取 `clear_then_reload``reload``delta` 策略。`clear_then_reload` 必须先清 Three.js 对象再 no-store 重拉summary 只做一致性校验,不能用 `0` 作为跳过图层重拉的理由。技术链路见 [数据作业与 Outbox 技术架构](/home/ray/dev/linkong/planet/docs/technical/zh/data-job-earth-sync-architecture.md),业务数据流见 [业务架构与数据流转](/home/ray/dev/linkong/planet/docs/technical/zh/platform-data-flows.md)。
### 3. 地球控制层
- [controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js)
@@ -136,7 +138,7 @@ Browser Camera provider 的手势识别管线在 [motion-browser-provider.js](/h
- terrain tile 拉取、解码、位移、着色
- 海陆基座与国界底图的整球 overlay
Earth 地表是多层近似同心球,不是单一 mesh。`earth.js` 的基座球、高清材质 overlay、云层/大气,以及 `country-boundaries.js` 的海陆基座都需要明确半径间距。远距视图下 GPU 深度精度会下降,相邻 shell 过近会 z-fighting表现为黑色闪烁块或雪花。当前稳定策略是让海陆基座使用 `landAltitudeOffset = 0.32`,高清材质使用 `textureOverlayAltitudeOffset = 0.48`;后续新增或调整整球地表 overlay 时,必须同步检查 [Earth 渲染图层顺序](/home/ray/dev/linkong/planet/docs/technical/zh/earth-render-layer-order.md),并在 50% 缩放视图验证。
智能星球地表是多层近似同心球,不是单一 mesh。`earth.js` 的基座球、高清材质 overlay、云层/大气,以及 `country-boundaries.js` 的海陆基座都需要明确半径间距。远距视图下 GPU 深度精度会下降,相邻 shell 过近会 z-fighting表现为黑色闪烁块或雪花。当前稳定策略是让海陆基座使用 `landAltitudeOffset = 0.32`,高清材质使用 `textureOverlayAltitudeOffset = 0.48`;后续新增或调整整球地表 overlay 时,必须同步检查 [智能星球渲染图层顺序](/home/ray/dev/linkong/planet/docs/technical/zh/earth-render-layer-order.md),并在 50% 缩放视图验证。
### 7. 图层模块
@@ -160,11 +162,16 @@ Earth 地表是多层近似同心球,不是单一 mesh。`earth.js` 的基座
`tv.js` 管理 `media-panel` 里的直播 / 态势新闻 tab。toolbar 打开或切换 TV/新闻时,会通过 `earth:tv-visibility-change``earth:tv-tab-change` 回写 Earth 设置:面板可见性仍按 desktop/mobile viewport 存在 `views.<scope>.panelVisibility.media-panel`,当前 tab 存在 `shared.mediaPanelActiveTab`,因此刷新页面后能恢复用户上次打开的直播或新闻状态。`closeTransientMobileOverlays()` 这类临时收起会带 `persist:false`,不会覆盖用户偏好。
`brand.js` 管理 Earth HUD 品牌资源。默认品牌来自静态资源,运行时覆盖值来自 `/api/v1/earth/brand`,上传的图片通过 `/earth-brand-assets/...` 读取。前端必须把 logo/title 图片和文本 fallback 分开处理:图片加载失败时显示文本标题,文本字段为空时使用后端默认值,避免 HUD 品牌区空白。控制台的 Earth 内容页负责保存和重置品牌配置,Earth 前端只消费结果。
`brand.js` 管理智能星球 HUD 品牌资源。默认品牌来自静态资源,运行时覆盖值来自 `/api/v1/earth/brand`,上传的图片通过 `/earth-brand-assets/...` 读取。前端必须把 logo/title 图片和文本 fallback 分开处理:图片加载失败时显示文本标题,文本字段为空时使用后端默认值,避免 HUD 品牌区空白。控制台的智能星球内容页负责保存和重置品牌配置,智能星球前端只消费结果。
Admin Next 的 Earth 内容页必须按运行时语义组织这些配置:
`about.js` 管理智能星球设置里的“关于”卡片。默认内容仍保留在前端作为兜底,运行时优先读取 `/api/v1/earth/about`。接口失败或字段缺失时必须回退默认值避免设置页出现空白。控制台的智能星球内容页提供“关于”tab保存走 `PUT /api/v1/earth/about`,恢复默认走 `DELETE /api/v1/earth/about`
`oobe.js` 管理 Earth 首次初始化引导。是否显示 OOBE 必须由 `/api/v1/earth/oobe-status``ready` 字段决定,不能依赖 `localStorage` 判断系统是否初始化。`localStorage` 只允许记录“本浏览器暂时跳过”的短时状态;如果后端已经认为 `ready: true`,退出登录、清空本地缓存或换浏览器都不应再次弹出 OOBE。桌面端使用深色星空遮罩和毛玻璃启动面板移动端改为底部 sheet并尊重 `prefers-reduced-motion`
控制台的智能星球内容页必须按运行时语义组织这些配置:
- `品牌标识`:品牌预览应使用与 Earth HUD 左上角一致的深色星空背景、尺寸、间距、logo/title 渲染和文本 fallback而不是普通表单预览。
- `关于`:配置 Earth 设置里的 About 卡片,包括 logo、眉标、标题、版本、描述和元信息条目Earth 运行时从 `/earth/about` 读取,失败时回退默认内容。
- `国界精度`:构建边界、刷新状态和恢复默认属于这个分区内部动作,不应放在页面全局工具栏。
- `电视直播`:列表同时区分内置源、采集源和自定义源;卡片状态表达启用、停用、新建或错误。内置源不能删除,采集源和自定义源可以删除。新增直播源在点击加号后才进入草稿状态,保存后固化到列表,取消则销毁草稿。
- `底图资源``图层资源``3D 模型``新闻锚点策略`:如果后端能力未接入,控制台应明确显示待接入空态,不应混入 TV 或品牌配置。
@@ -192,9 +199,7 @@ TV 预览需要尽量复用 Earth 运行时的直播卡片结构和状态标签
当前 BGP 巡航只是这套能力的一个调用方不应再把“按队列巡航”和“BGP 事件展示”混写在同一个状态机里。
新闻巡航摘要计划见:
- [Earth 新闻巡航摘要计划](/home/ray/dev/linkong/planet/docs/plans/earth-news-cruise-summary-plan.md)
新闻巡航摘要的未来计划保存在仓库路径 `docs/plans/earth-news-cruise-summary-plan.md`,不作为公开 Docs 页面入口。
## 当前样式分层
@@ -380,7 +385,7 @@ asset 图标大小由 `Interactable` 的 `icon.fitSize` 控制。SVG / 图片文
接口细节、生命周期和接入示例见:
- [Earth 可交互图标接入](/home/ray/dev/linkong/planet/docs/technical/zh/earth-interactable-usage.md)
- [智能星球可交互图标接入](/home/ray/dev/linkong/planet/docs/technical/zh/earth-interactable-usage.md)
### 视角控制反馈
@@ -452,7 +457,7 @@ Earth 设置面板当前由 [controls.js](/home/ray/dev/linkong/planet/frontend/
SGP4 传播输出是惯性系位置,不能直接当成 Earth 的经纬度固定坐标使用。`satellites.js` 会用当前时间的 `gstime` 把 ECI/TEME 位置转换到 ECF再映射到 `latLonToVector3()` 使用的 Three.js 坐标轴。卫星点和短尾迹使用随采样时间变化的地固坐标,表示相对当前地球表面的实际位置;锁定后的预测轨道线使用锁定时刻固定的 `gstime`把未来一圈惯性轨道投到当前地球姿态上显示因此会闭合并且轨道面倾角应与详情卡一致。fallback 预测轨道也必须使用真正的 RAAN + inclination 轨道平面公式,不能把 inclination 当成恒定纬度。
国界精度偏好独立存储在 `country-boundaries.js``planet.earth.boundaries.highPrecisionEnabled`。未开启高精时,即使本机已经有高精 manifest/PMTiles也继续加载低精 `countries-admin0.min.geojson` fallback开启高精但高精产物缺失时Earth 工具栏设置会调用 `/api/v1/earth/boundaries/build` 启动后台构建并轮询进度。构建成功后调用 `reloadCountryBoundaries()` 热切换,不再刷新整个页面。国界 hover 与 tooltip 解耦:只要地表坐标落在国界 polygon 内就保持高亮如果鼠标同时命中卫星、船只、BGP 等 interactabletooltip 显示 interactable 信息,但国界高亮不应闪烁。
国界精度偏好独立存储在 `country-boundaries.js``planet.earth.boundaries.highPrecisionEnabled`。未开启高精时,即使本机已经有高精 manifest/PMTiles也继续加载低精 `countries-admin0.min.geojson` fallback开启高精但高精产物缺失时智能星球工具栏设置会调用 `/api/v1/earth/boundaries/build` 启动后台构建并轮询进度。构建成功后调用 `reloadCountryBoundaries()` 热切换,不再刷新整个页面。国界 hover 与 tooltip 解耦:只要地表坐标落在国界 polygon 内就保持高亮如果鼠标同时命中卫星、船只、BGP 等 interactabletooltip 显示 interactable 信息,但国界高亮不应闪烁。
## 当前地形链路

View File

@@ -1,6 +1,6 @@
# Earth Interactable 使用说明
# 智能星球 Interactable 使用说明
`Interactable` Earth 地表“图标类可交互元素”的通用渲染入口。它把船只图层验证过的模式抽成公共能力:普通态用批量 `THREE.Points`hover / locked 用少量 overlay拾取走屏幕空间命中图标资源统一转进 canvas texture并在公共层处理 glow、状态、尺寸、贴地渲染和同坐标避让。
`Interactable`智能星球地表“图标类可交互元素”的通用渲染入口。它把船只图层验证过的模式抽成公共能力:普通态用批量 `THREE.Points`hover / locked 用少量 overlay拾取走屏幕空间命中图标资源统一转进 canvas texture并在公共层处理 glow、状态、尺寸、贴地渲染和同坐标避让。
当前已接入:

View File

@@ -1,8 +1,8 @@
# Earth 图层样式属性索引
# 智能星球图层样式属性索引
本文记录当前 Earth 前端各图层的材质、颜色、透明度、线宽、半径偏移和
`renderOrder` 等样式属性。层级关系请配合
[Earth 渲染图层顺序](/home/ray/dev/linkong/planet/docs/technical/zh/earth-render-layer-order.md)
[智能星球渲染图层顺序](/home/ray/dev/linkong/planet/docs/technical/zh/earth-render-layer-order.md)
查看。
## 命名约定
@@ -20,12 +20,12 @@
| 正式名称 | 变量名 | 当前值 | 使用位置 / 说明 |
| --- | --- | --- | --- |
| Earth 基座半径 | `CONFIG.earthRadius` | `100` | `earth.js:createEarth()` |
| Earth 基座颜色 | `EARTH_MATERIAL_CONFIG.color` | `0x010609` | `MeshPhongMaterial.color` |
| Earth 基座 emissive | `EARTH_MATERIAL_CONFIG.emissive` | `0x010609` | `MeshPhongMaterial.emissive` |
| Earth 基座 specular | `EARTH_MATERIAL_CONFIG.specular` | `0x1a2d45` | `MeshPhongMaterial.specular` |
| Earth 基座 shininess | `EARTH_MATERIAL_CONFIG.shininess` | `12` | `MeshPhongMaterial.shininess` |
| Earth 基座 opacity | `EARTH_MATERIAL_CONFIG.opacity` | `1` | `MeshPhongMaterial.opacity` |
| 智能星球基座半径 | `CONFIG.earthRadius` | `100` | `earth.js:createEarth()` |
| 智能星球基座颜色 | `EARTH_MATERIAL_CONFIG.color` | `0x010609` | `MeshPhongMaterial.color` |
| 智能星球基座 emissive | `EARTH_MATERIAL_CONFIG.emissive` | `0x010609` | `MeshPhongMaterial.emissive` |
| 智能星球基座 specular | `EARTH_MATERIAL_CONFIG.specular` | `0x1a2d45` | `MeshPhongMaterial.specular` |
| 智能星球基座 shininess | `EARTH_MATERIAL_CONFIG.shininess` | `12` | `MeshPhongMaterial.shininess` |
| 智能星球基座 opacity | `EARTH_MATERIAL_CONFIG.opacity` | `1` | `MeshPhongMaterial.opacity` |
| 高清材质半径偏移 | `EARTH_MATERIAL_CONFIG.textureOverlayAltitudeOffset` | `0.48` | 独立高清材质球半径;必须与海陆基座和地球基座保持足够深度间距,避免远距 z-fighting |
| 高清材质透明度 | `EARTH_MATERIAL_CONFIG.textureOverlayOpacity` | `0.88` | 高清材质 `MeshPhongMaterial.opacity` |
| 高清材质 renderOrder | `EARTH_MATERIAL_CONFIG.textureOverlayRenderOrder` | `0.96` | `_earthTextureOverlay.renderOrder` |

View File

@@ -1,4 +1,4 @@
# Earth 渲染图层顺序
# 智能星球渲染图层顺序
本文记录当前 Earth 渲染器的图层顺序和每层意图。后续调整
`renderOrder`、半径偏移、深度策略或指针交互时,需要同步更新这里。
@@ -14,9 +14,9 @@
| 顺序 | 图层 | 来源 | 渲染 / 半径策略 | 深度 / 交互策略 | 备注 |
| --- | --- | --- | --- | --- | --- |
| -1000 | 天体背景 mesh | `celestial.js` | 背景球 | 不参与地表拾取 | 位于所有 Earth 内容之后。 |
| -1 | Earth 遮挡球 | `earth.js` | 地球内侧不可见球 | 写入深度 | 遮挡地球背面的对象。 |
| 0 | Earth 基座球 | `earth.js` | `CONFIG.earthRadius` | 地表拾取兜底目标 | 深色基座,所有可选地图层关闭时仍可见。 |
| -1000 | 天体背景 mesh | `celestial.js` | 背景球 | 不参与地表拾取 | 位于所有智能星球内容之后。 |
| -1 | 智能星球遮挡球 | `earth.js` | 地球内侧不可见球 | 写入深度 | 遮挡地球背面的对象。 |
| 0 | 智能星球基座球 | `earth.js` | `CONFIG.earthRadius` | 地表拾取兜底目标 | 深色基座,所有可选地图层关闭时仍可见。 |
| 0.2 | 国界暗色 tint | `country-boundaries.js` | `tintAltitudeOffset` | 禁用 raycast | 高清材质关闭时使用。 |
| 0.86 | 海陆基座填充 | `country-boundaries.js` | `landAltitudeOffset = 0.32`; 海洋 `#010609`,陆地 `#080f1b` | 禁用 raycast | 即使国界线关闭,基座地图仍保持可用;半径与基座球拉开以避免远距 z-fighting。 |
| 0.96 | 高清 Earth 材质 | `earth.js` | `textureOverlayAltitudeOffset = 0.48` | 可见时作为地表拾取目标 | 高清材质始终压过海陆基座填充;半径必须高于海陆基座并与基座球保持足够间距。 |
@@ -63,7 +63,7 @@ Earth 的地表不是单一 mesh而是多层近似同心球基座球、海
| 交互 | 当前规则 |
| --- | --- |
| Earth 坐标 hover | 高清材质可见时使用高清材质 overlay 作为地表拾取目标,否则使用 Earth 基座球。 |
| 智能星球坐标 hover | 高清材质可见时使用高清材质 overlay 作为地表拾取目标,否则使用智能星球基座球。 |
| 国界 hover | 先把地表拾取坐标转成经纬度,再用 GeoJSON 点面判断;国界 hover 线本身不接收 raycast。 |
| 国界 hover 视觉 | hover 时压暗普通国界线,并绘制无深度测试的光晕和实线。 |
| 中国 / 台湾 hover | `CHN``TWN` 被归到同一个 hover 高亮组tooltip 仍显示鼠标实际命中的 feature。 |

View File

@@ -1,10 +1,10 @@
# Earth 卫星覆盖策略
# 智能星球卫星覆盖策略
本文件记录 Earth 卫星图层当前关于 `footprint` 的产品边界、资料依据和已落地实现,目标是避免把 Starlink 这套专用地表覆盖模型误用到其它星座上。
相关上下文:
- [Earth 前端结构](/home/ray/dev/linkong/planet/docs/technical/zh/earth-frontend-context.md)
- [智能星球前端结构](/home/ray/dev/linkong/planet/docs/technical/zh/earth-frontend-context.md)
- [数据采集系统](/home/ray/dev/linkong/planet/docs/technical/zh/backend-collectors.md)
- [backend/app/services/collectors/celestrak.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/celestrak.py)
- [frontend/public/earth/js/satellites.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/satellites.js)
@@ -17,26 +17,21 @@
## 本地实际类别
当前 CelesTrak 卫星分组在 [backend/app/services/collectors/celestrak.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/celestrak.py) 中包括:
当前 CelesTrak 采集器优先从 `GROUP=active&FORMAT=json` 拉取完整活跃卫星目录。这样可以避免某个 CelesTrak 分组请求失败时仍把不完整结果保存为成功批次。采集器只在完整 JSON 数组可解析、且记录含 `NORAD_CAT_ID` 时进入转换和保存;网络、续传或解析失败会重试,最终失败时保留上一批 current 数据。
- `starlink`
- `gps-ops`
- `galileo`
- `glonass`
- `beidou`
- `leo`
- `geo`
- `iridium-next`
CelesTrak 对 `active` 这类大 group 有每次 GP 数据更新窗口内的重复下载限制。为避免清空数据库后无法立即恢复Planet 会把原始下载文件保存在 `$PLANET_CACHE_DIR/downloads`。当 `active` 返回“本轮数据未更新”的 403 时:
其中非 Starlink 类别是:
- 如果 `active` 缓存存在,直接用缓存重新写入数据库。
- 如果 `active` 缓存不存在,才切换到 fallback group mode`starlink``gps-ops``galileo``glonass``beidou``leo``geo``iridium-next` 全部下载并合并。
- fallback group mode 要求所有 group 都成功或有缓存可复用;任意 group 缺失都会整体失败,不保存 partial。
- `gps-ops`
- `galileo`
- `glonass`
- `beidou`
- `leo`
- `geo`
- `iridium-next`
采集结果仍会给前端提供 `metadata.constellation_group`,但该字段现在来自可执行推断:
- `OBJECT_NAME``STARLINK` 开头时标记为 `starlink`
- `OBJECT_NAME``IRIDIUM` 开头时标记为 `iridium-next`
- 其它活跃卫星不强行归入旧 CelesTrak 小分组,避免把泛化类别当成精确星座
因此,非 Starlink 类别在产品策略中仍包括 GNSS/RNSS、GEO、generic LEO 和 Iridium NEXT 等语义;但只有 fallback group mode 会保存旧的 `gps-ops``galileo``glonass``beidou``leo``geo` 分组标签active 主路径不会强行给所有卫星补这类标签。
## 资料结论
@@ -136,7 +131,8 @@
1. 后端把星座分组和 footprint 策略提示透给前端
- CelesTrak collector 会把 `GROUP` 记入 `metadata.constellation_group`
- CelesTrak collector 会把原始查询来源记入 `metadata.celestrak_query_group = active`
- `metadata.constellation_group` 只保存可推断的业务星座,例如 `starlink``iridium-next`
- Visualization API 会输出:
- `properties.constellation_group`
- `properties.footprint_policy`

View File

@@ -1,10 +1,10 @@
# Earth 工具栏与浮层协同
# 智能星球工具栏与浮层协同
本文件描述 Earth 大屏右侧工具栏按钮,以及搜索面板、设置弹窗、新闻直播面板、图层面板这几个浮层之间当前的协同规则。改交互、加按钮、调整面板时按这个表对齐,避免出现「点 A 把不该关的 B 也关了」之类的协同冲突。
相关入口:
- [Earth 前端结构](/home/ray/dev/linkong/planet/docs/technical/zh/earth-frontend-context.md)
- [智能星球前端结构](/home/ray/dev/linkong/planet/docs/technical/zh/earth-frontend-context.md)
- [前端布局指南](/home/ray/dev/linkong/planet/docs/technical/zh/frontend-layout-guidelines.md)
## 工具栏按钮目录

Some files were not shown because too many files have changed in this diff Show More