Compare commits
10 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
887fec972e | ||
|
|
5bf5c73ca0 | ||
| e65267fe21 | |||
| ae982e51cd | |||
|
|
65e6a96c0d | ||
|
|
37e92e7572 | ||
|
|
a37d4b6289 | ||
|
|
69789d7505 | ||
|
|
4f124121e7 | ||
|
|
085bdf9a80 |
@@ -189,6 +189,9 @@
|
||||
|
||||
# 查看服务状态
|
||||
./planet.sh health
|
||||
|
||||
# 删除容器、卷、镜像和本地编译状态,执行前需要输入 Y 确认
|
||||
./planet.sh destroy
|
||||
```
|
||||
|
||||
前端命令约定:
|
||||
|
||||
6
TODO.md
6
TODO.md
@@ -7,9 +7,6 @@ This file is the active backlog only. Completed history belongs in `docs/CHANGEL
|
||||
- [ ] Earth AI command entry: merge natural-language and speech-triggered LLM commands into the existing Earth search panel as described in [Agent Runtime, Earth LLM Command, And Speech Entry Plan](/home/ray/dev/linkong/planet/docs/plans/agents-earth-command-runtime-plan.md).
|
||||
- [ ] Earth action executor: implement safe visualization actions for layer toggles, batch highlights, filters, focus, result panels, and clear-highlight behavior.
|
||||
- [ ] Earth entity matching: support stable entity ids and batch matching for Beidou satellites, mainland China compute centers, BGP, news, vessels, and cables.
|
||||
- [x] High-precision country boundary tile framework: implement the static vector tile builder, versioned seed output, frontend bbox tile loader, debounce, in-flight dedupe, and LRU cache described in [Earth High Precision Boundary Tiles Plan](/home/ray/dev/linkong/planet/docs/plans/earth-high-precision-boundary-tiles-plan.md).
|
||||
- [x] Add the `pmtiles-mvt` frontend tile provider contract, MVT decoder dependencies, static PMTiles Nginx handling, collector artifact registration, production readiness check, and user operation docs for Earth boundaries.
|
||||
- [x] Split Earth boundary ingestion into standard source collectors (`earth_admin0_boundaries`, `earth_coastline`, `earth_claim_lines`) plus the downstream `earth_boundary_tiles` PMTiles builder.
|
||||
- [ ] Replace debug GeoJSON boundary tiles with the real `earth-boundaries-china-pov-v1.pmtiles` production artifact after audited admin-0 / coastline / claim-line sources and the PMTiles toolchain are available.
|
||||
- [ ] Import authoritative China POV / coastline / claim-line source packages through the three standard Earth boundary source collectors, then rebuild a versioned PMTiles artifact so highest zoom `8-10` preserves trusted source geometry instead of seed data.
|
||||
- [ ] Earth boundary data: acquire or generate auditable China POV geometry for Zangnan, Aksai Chin, Taiwan/Penghu, Diaoyu Dao and affiliated islands, Chiwei Yu, South China Sea islands, Kosovo, Gaza, and the official dashed maritime claim line before implementing final visual changes.
|
||||
@@ -61,6 +58,9 @@ Archived items stay here so old context is not lost. Completed items remain chec
|
||||
|
||||
### Completed
|
||||
|
||||
- [x] Implemented the high-precision country boundary tile framework from [Earth High Precision Boundary Tiles Plan](/home/ray/dev/linkong/planet/docs/plans/earth-high-precision-boundary-tiles-plan.md): static vector tile builder, versioned seed output, frontend bbox tile loader, debounce, in-flight dedupe, and LRU cache.
|
||||
- [x] Added the `pmtiles-mvt` frontend tile provider contract, MVT decoder dependencies, static PMTiles Nginx handling, collector artifact registration, production readiness check, and user operation docs for Earth boundaries.
|
||||
- [x] Split Earth boundary ingestion into standard source collectors (`earth_admin0_boundaries`, `earth_coastline`, `earth_claim_lines`) plus the downstream `earth_boundary_tiles` PMTiles builder.
|
||||
- [x] Refined BGP observer and anomaly `hover/click` feel.
|
||||
- [x] Added BGP anomaly relationship display with cables / regions.
|
||||
- [x] Added the Earth BGP activity layer so the map still feels alive when incident density is low.
|
||||
|
||||
@@ -2,10 +2,14 @@
|
||||
|
||||
ARG PYTHON_IMAGE=python:3.14-slim
|
||||
ARG UV_IMAGE=ghcr.io/astral-sh/uv:latest
|
||||
ARG AI_PROVIDER_BUILD_FINGERPRINT=unknown
|
||||
|
||||
FROM ${UV_IMAGE} AS uv
|
||||
FROM ${PYTHON_IMAGE}
|
||||
|
||||
ARG AI_PROVIDER_BUILD_FINGERPRINT
|
||||
LABEL planet.aiprovider.build-fingerprint="${AI_PROVIDER_BUILD_FINGERPRINT}"
|
||||
|
||||
COPY --from=uv /uv /uvx /bin/
|
||||
|
||||
WORKDIR /app
|
||||
@@ -15,12 +19,15 @@ ENV PYTHONUNBUFFERED=1
|
||||
ENV UV_COMPILE_BYTECODE=1
|
||||
ENV UV_LINK_MODE=copy
|
||||
|
||||
RUN mkdir -p /root/.config/uv
|
||||
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
curl \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
COPY pyproject.toml uv.lock /app/
|
||||
RUN --mount=type=cache,target=/root/.cache/uv \
|
||||
--mount=type=secret,id=planet_uv_config,target=/root/.config/uv/uv.toml,required=false \
|
||||
uv sync --frozen --no-dev
|
||||
|
||||
COPY aiprovider /app/aiprovider
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
# syntax=docker/dockerfile:1.7
|
||||
|
||||
ARG PYTHON_IMAGE=python:3.14-slim
|
||||
ARG UV_IMAGE=ghcr.io/astral-sh/uv:latest
|
||||
|
||||
@@ -14,12 +16,16 @@ ENV UV_COMPILE_BYTECODE=1
|
||||
ENV UV_LINK_MODE=copy
|
||||
ENV PYTHONPATH=/app/backend
|
||||
|
||||
RUN mkdir -p /root/.config/uv
|
||||
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
curl \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
COPY pyproject.toml uv.lock /app/
|
||||
RUN uv sync --frozen --no-dev
|
||||
RUN --mount=type=cache,target=/root/.cache/uv \
|
||||
--mount=type=secret,id=planet_uv_config,target=/root/.config/uv/uv.toml,required=false \
|
||||
uv sync --frozen --no-dev
|
||||
|
||||
COPY backend /app/backend
|
||||
COPY VERSION /app/VERSION
|
||||
|
||||
@@ -19,6 +19,7 @@ from app.api.v1 import (
|
||||
vessels,
|
||||
bgp,
|
||||
news,
|
||||
interactables,
|
||||
realtime_sources,
|
||||
system_control,
|
||||
tv,
|
||||
@@ -53,4 +54,5 @@ api_router.include_router(vessels.router, prefix="/vessels", tags=["vessels"])
|
||||
api_router.include_router(bgp.router, prefix="/bgp", tags=["bgp"])
|
||||
api_router.include_router(tv.router, prefix="/tv", tags=["tv"])
|
||||
api_router.include_router(news.router, prefix="/news", tags=["news"])
|
||||
api_router.include_router(interactables.router, prefix="/interactables", tags=["interactables"])
|
||||
api_router.include_router(realtime_sources.router, prefix="/realtime-sources", tags=["realtime-sources"])
|
||||
|
||||
@@ -3,6 +3,7 @@ from uuid import uuid4
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request, Response
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.logging import get_logger
|
||||
from app.core.security import get_current_user
|
||||
from app.db.session import get_db
|
||||
from app.models.user import User
|
||||
@@ -47,8 +48,10 @@ from app.services.playground_chat_service import (
|
||||
stop_message,
|
||||
)
|
||||
from app.services.situational_alert_ai_brief import build_situational_alert_brief_request
|
||||
from app.services.business_logs import emit_business_log, exception_context
|
||||
|
||||
router = APIRouter()
|
||||
logger = get_logger(__name__, service="api")
|
||||
|
||||
|
||||
@router.get("/provider/status", response_model=AIProviderStatusResponse)
|
||||
@@ -122,6 +125,16 @@ async def create_playground_message(
|
||||
provider_client: AIProviderClient = Depends(get_ai_provider_client),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="ai.playground.message.create",
|
||||
message="Playground message creation requested",
|
||||
category="ai",
|
||||
service="api",
|
||||
module=__name__,
|
||||
user_id=current_user.id,
|
||||
context={"session_key": payload.session_key, "preset": payload.selected_preset_key},
|
||||
)
|
||||
return await create_turn(
|
||||
db,
|
||||
user_id=current_user.id,
|
||||
@@ -136,6 +149,16 @@ async def stop_playground_message(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="ai.playground.message.stop",
|
||||
message="Playground message stop requested",
|
||||
category="ai",
|
||||
service="api",
|
||||
module=__name__,
|
||||
user_id=current_user.id,
|
||||
context={"session_key": payload.session_key, "message_id": payload.message_id},
|
||||
)
|
||||
return await stop_message(
|
||||
db,
|
||||
user_id=current_user.id,
|
||||
@@ -150,6 +173,16 @@ async def resend_playground_message(
|
||||
provider_client: AIProviderClient = Depends(get_ai_provider_client),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="ai.playground.message.resend",
|
||||
message="Playground message resend requested",
|
||||
category="ai",
|
||||
service="api",
|
||||
module=__name__,
|
||||
user_id=current_user.id,
|
||||
context={"session_key": payload.session_key, "user_message_id": payload.user_message_id},
|
||||
)
|
||||
return await resend_turn(
|
||||
db,
|
||||
user_id=current_user.id,
|
||||
@@ -214,16 +247,70 @@ async def analyze_bgp_brief(
|
||||
anomaly_limit=payload.anomaly_limit,
|
||||
collector_limit=payload.collector_limit,
|
||||
)
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="ai.brief.bgp.facts_collected",
|
||||
message="BGP brief facts collected",
|
||||
category="ai",
|
||||
service="api",
|
||||
module=__name__,
|
||||
request_id=request_id,
|
||||
user_id=current_user.id,
|
||||
context={
|
||||
"incident_limit": payload.incident_limit,
|
||||
"anomaly_limit": payload.anomaly_limit,
|
||||
"collector_limit": payload.collector_limit,
|
||||
"fact_count": len(facts or []),
|
||||
},
|
||||
)
|
||||
brief_request.preferred_model = payload.preferred_model
|
||||
brief_request.thinking = payload.thinking
|
||||
|
||||
analysis = await provider_client.analyze(brief_request, request_id=request_id)
|
||||
return save_bgp_brief_record(
|
||||
analysis,
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="ai.brief.bgp.start",
|
||||
message="BGP brief AI analysis started",
|
||||
category="ai",
|
||||
service="api",
|
||||
module=__name__,
|
||||
request_id=request_id,
|
||||
facts=facts,
|
||||
context=context,
|
||||
user_id=current_user.id,
|
||||
context={"preferred_model": payload.preferred_model},
|
||||
)
|
||||
try:
|
||||
analysis = await provider_client.analyze(brief_request, request_id=request_id)
|
||||
record = save_bgp_brief_record(
|
||||
analysis,
|
||||
request_id=request_id,
|
||||
facts=facts,
|
||||
context=context,
|
||||
)
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="ai.brief.bgp.completed",
|
||||
message="BGP brief AI analysis saved",
|
||||
category="ai",
|
||||
service="api",
|
||||
module=__name__,
|
||||
request_id=request_id,
|
||||
user_id=current_user.id,
|
||||
context={"provider": analysis.provider, "model": analysis.model, "brief_id": record.id},
|
||||
)
|
||||
return record
|
||||
except Exception as exc:
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="ai.brief.bgp.failed",
|
||||
message="BGP brief AI analysis failed",
|
||||
category="ai",
|
||||
level="error",
|
||||
service="api",
|
||||
module=__name__,
|
||||
request_id=request_id,
|
||||
user_id=current_user.id,
|
||||
context=exception_context(exc, {"preferred_model": payload.preferred_model}),
|
||||
)
|
||||
raise
|
||||
|
||||
|
||||
@router.post("/alerts/brief", response_model=AlertBriefResponse)
|
||||
@@ -242,17 +329,65 @@ async def analyze_alert_brief(
|
||||
db,
|
||||
alert_limit=payload.alert_limit,
|
||||
)
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="ai.brief.alerts.facts_collected",
|
||||
message="Alert brief facts collected",
|
||||
category="ai",
|
||||
service="api",
|
||||
module=__name__,
|
||||
request_id=request_id,
|
||||
user_id=current_user.id,
|
||||
context={"alert_limit": payload.alert_limit, "fact_count": len(facts or [])},
|
||||
)
|
||||
brief_request.preferred_model = payload.preferred_model
|
||||
brief_request.thinking = payload.thinking
|
||||
|
||||
analysis = await provider_client.analyze(brief_request, request_id=request_id)
|
||||
return AlertBriefResponse(
|
||||
**analysis.model_dump(),
|
||||
title=brief_request.title,
|
||||
objective=brief_request.objective,
|
||||
facts=facts,
|
||||
context=context,
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="ai.brief.alerts.start",
|
||||
message="Alert brief AI analysis started",
|
||||
category="ai",
|
||||
service="api",
|
||||
module=__name__,
|
||||
request_id=request_id,
|
||||
user_id=current_user.id,
|
||||
context={"preferred_model": payload.preferred_model},
|
||||
)
|
||||
try:
|
||||
analysis = await provider_client.analyze(brief_request, request_id=request_id)
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="ai.brief.alerts.completed",
|
||||
message="Alert brief AI analysis completed",
|
||||
category="ai",
|
||||
service="api",
|
||||
module=__name__,
|
||||
request_id=request_id,
|
||||
user_id=current_user.id,
|
||||
context={"provider": analysis.provider, "model": analysis.model},
|
||||
)
|
||||
return AlertBriefResponse(
|
||||
**analysis.model_dump(),
|
||||
title=brief_request.title,
|
||||
objective=brief_request.objective,
|
||||
facts=facts,
|
||||
context=context,
|
||||
)
|
||||
except Exception as exc:
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="ai.brief.alerts.failed",
|
||||
message="Alert brief AI analysis failed",
|
||||
category="ai",
|
||||
level="error",
|
||||
service="api",
|
||||
module=__name__,
|
||||
request_id=request_id,
|
||||
user_id=current_user.id,
|
||||
context=exception_context(exc, {"preferred_model": payload.preferred_model}),
|
||||
)
|
||||
raise
|
||||
|
||||
|
||||
@router.post("/situational-alerts/brief", response_model=SituationalAlertBriefResponse)
|
||||
@@ -268,14 +403,62 @@ async def analyze_situational_alert_brief(
|
||||
response.headers["X-Request-ID"] = request_id
|
||||
|
||||
brief_request, facts, context = await build_situational_alert_brief_request(db)
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="ai.brief.situational_alerts.facts_collected",
|
||||
message="Situational alert brief facts collected",
|
||||
category="ai",
|
||||
service="api",
|
||||
module=__name__,
|
||||
request_id=request_id,
|
||||
user_id=current_user.id,
|
||||
context={"fact_count": len(facts or [])},
|
||||
)
|
||||
brief_request.preferred_model = payload.preferred_model
|
||||
brief_request.thinking = payload.thinking
|
||||
|
||||
analysis = await provider_client.analyze(brief_request, request_id=request_id)
|
||||
return SituationalAlertBriefResponse(
|
||||
**analysis.model_dump(),
|
||||
title=brief_request.title,
|
||||
objective=brief_request.objective,
|
||||
facts=facts,
|
||||
context=context,
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="ai.brief.situational_alerts.start",
|
||||
message="Situational alert brief AI analysis started",
|
||||
category="ai",
|
||||
service="api",
|
||||
module=__name__,
|
||||
request_id=request_id,
|
||||
user_id=current_user.id,
|
||||
context={"preferred_model": payload.preferred_model},
|
||||
)
|
||||
try:
|
||||
analysis = await provider_client.analyze(brief_request, request_id=request_id)
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="ai.brief.situational_alerts.completed",
|
||||
message="Situational alert brief AI analysis completed",
|
||||
category="ai",
|
||||
service="api",
|
||||
module=__name__,
|
||||
request_id=request_id,
|
||||
user_id=current_user.id,
|
||||
context={"provider": analysis.provider, "model": analysis.model},
|
||||
)
|
||||
return SituationalAlertBriefResponse(
|
||||
**analysis.model_dump(),
|
||||
title=brief_request.title,
|
||||
objective=brief_request.objective,
|
||||
facts=facts,
|
||||
context=context,
|
||||
)
|
||||
except Exception as exc:
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="ai.brief.situational_alerts.failed",
|
||||
message="Situational alert brief AI analysis failed",
|
||||
category="ai",
|
||||
level="error",
|
||||
service="api",
|
||||
module=__name__,
|
||||
request_id=request_id,
|
||||
user_id=current_user.id,
|
||||
context=exception_context(exc, {"preferred_model": payload.preferred_model}),
|
||||
)
|
||||
raise
|
||||
|
||||
@@ -1,4 +1,3 @@
|
||||
import asyncio
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Optional
|
||||
|
||||
@@ -7,7 +6,7 @@ from pydantic import BaseModel, Field
|
||||
from sqlalchemy import func, or_, select, text
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.cache import cache
|
||||
from app.core.logging import get_logger
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.core.security import get_current_user
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
@@ -22,16 +21,26 @@ from app.models.user import User
|
||||
from app.models.vessel import AISRawObservation
|
||||
from app.services.vessel_ais_aggregation import VESSEL_AIS_SCHEMA
|
||||
from app.services.scheduler import (
|
||||
cancel_running_collector_now,
|
||||
get_latest_task_id_for_datasource,
|
||||
run_collector_now,
|
||||
sync_datasource_job,
|
||||
)
|
||||
from app.services.earth_layer_cache import invalidate_earth_layer_cache_for_source
|
||||
from app.services.data_jobs import (
|
||||
JOB_STATUS_CANCELLING,
|
||||
JOB_STATUS_QUEUED,
|
||||
JOB_STATUS_RUNNING,
|
||||
JOB_TYPE_CLEAR_CACHE,
|
||||
JOB_TYPE_CLEAR_DATA,
|
||||
JOB_TYPE_COLLECT,
|
||||
enqueue_datasource_job,
|
||||
get_active_datasource_job,
|
||||
request_cancel_datasource_task,
|
||||
)
|
||||
from app.services.business_logs import emit_business_log
|
||||
|
||||
router = APIRouter()
|
||||
logger = get_logger(__name__, service="api")
|
||||
STALE_RUNNING_TASK_TIMEOUT_MINUTES = 90
|
||||
|
||||
|
||||
PRODUCT_SOURCE_KEYWORDS: tuple[tuple[str, tuple[str, ...]], ...] = (
|
||||
("vessels", ("vessel", "ais")),
|
||||
("cables", ("cable", "landing", "telegeography", "arcgis", "fao")),
|
||||
@@ -115,7 +124,7 @@ async def _load_latest_running_tasks(
|
||||
_task_rank_column(CollectionTask.started_at),
|
||||
)
|
||||
.where(CollectionTask.datasource_id.in_(datasource_ids))
|
||||
.where(CollectionTask.status == "running")
|
||||
.where(CollectionTask.status.in_((JOB_STATUS_QUEUED, JOB_STATUS_RUNNING, JOB_STATUS_CANCELLING)))
|
||||
.subquery()
|
||||
)
|
||||
result = await db.execute(
|
||||
@@ -126,32 +135,6 @@ async def _load_latest_running_tasks(
|
||||
return {task.datasource_id: task for task in result.scalars().all()}
|
||||
|
||||
|
||||
async def _load_latest_task_ids(
|
||||
db: AsyncSession,
|
||||
datasource_ids: list[int],
|
||||
) -> dict[int, int]:
|
||||
if not datasource_ids:
|
||||
return {}
|
||||
|
||||
ranked_tasks = (
|
||||
select(
|
||||
CollectionTask.id.label("task_id"),
|
||||
CollectionTask.datasource_id.label("datasource_id"),
|
||||
func.row_number().over(
|
||||
partition_by=CollectionTask.datasource_id,
|
||||
order_by=CollectionTask.id.desc(),
|
||||
).label("row_num"),
|
||||
)
|
||||
.where(CollectionTask.datasource_id.in_(datasource_ids))
|
||||
.subquery()
|
||||
)
|
||||
result = await db.execute(
|
||||
select(ranked_tasks.c.datasource_id, ranked_tasks.c.task_id)
|
||||
.where(ranked_tasks.c.row_num == 1)
|
||||
)
|
||||
return {datasource_id: task_id for datasource_id, task_id in result.all()}
|
||||
|
||||
|
||||
async def _load_latest_tasks(
|
||||
db: AsyncSession,
|
||||
datasource_ids: list[int],
|
||||
@@ -303,8 +286,11 @@ def serialize_datasource_row(
|
||||
"last_run": to_iso8601_utc(last_run_at),
|
||||
"last_run_at": to_iso8601_utc(last_run_at),
|
||||
"last_status": last_status,
|
||||
"is_running": running_task is not None,
|
||||
"is_running": running_task is not None and running_task.task_type == JOB_TYPE_COLLECT,
|
||||
"is_task_active": running_task is not None,
|
||||
"task_status": running_task.status if running_task else None,
|
||||
"task_id": display_task.id if display_task else None,
|
||||
"task_type": display_task.task_type if display_task else None,
|
||||
"progress": display_task.progress if display_task else None,
|
||||
"phase": display_task.phase if display_task else None,
|
||||
"phase_progress": display_task.phase_progress if display_task else None,
|
||||
@@ -400,9 +386,25 @@ async def _trigger_datasource_batch(
|
||||
datasources: list[DataSource],
|
||||
*,
|
||||
force: bool,
|
||||
actor_id: int | None = None,
|
||||
trigger_kind: str = "batch",
|
||||
) -> dict:
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event=f"collector.trigger.{trigger_kind}.start",
|
||||
message="Datasource batch trigger started",
|
||||
category="collector",
|
||||
service="api",
|
||||
module=__name__,
|
||||
user_id=actor_id,
|
||||
context={
|
||||
"trigger_kind": trigger_kind,
|
||||
"force": force,
|
||||
"requested_count": len(datasources),
|
||||
},
|
||||
)
|
||||
if not datasources:
|
||||
return {
|
||||
result = {
|
||||
"status": "noop",
|
||||
"message": "No matching data sources to trigger",
|
||||
"force": force,
|
||||
@@ -410,8 +412,18 @@ async def _trigger_datasource_batch(
|
||||
"skipped": [],
|
||||
"failed": [],
|
||||
}
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event=f"collector.trigger.{trigger_kind}.completed",
|
||||
message="Datasource batch trigger completed with no matching sources",
|
||||
category="collector",
|
||||
service="api",
|
||||
module=__name__,
|
||||
user_id=actor_id,
|
||||
context={"trigger_kind": trigger_kind, "force": force, "status": "noop", "triggered_count": 0},
|
||||
)
|
||||
return result
|
||||
|
||||
previous_task_ids: dict[int, Optional[int]] = {}
|
||||
triggered_sources: list[dict] = []
|
||||
skipped_sources: list[dict] = []
|
||||
failed_sources: list[dict] = []
|
||||
@@ -446,9 +458,11 @@ async def _trigger_datasource_batch(
|
||||
}
|
||||
)
|
||||
continue
|
||||
cancelled = await cancel_running_collector_now(datasource.source)
|
||||
if not cancelled:
|
||||
await rollback_orphaned_running_task(db, datasource, running_task)
|
||||
await request_cancel_datasource_task(
|
||||
db,
|
||||
running_task,
|
||||
reason="superseded_by_forced_collection",
|
||||
)
|
||||
|
||||
if not force and not is_due_for_collection(datasource, now):
|
||||
skipped_sources.append(
|
||||
@@ -465,57 +479,51 @@ async def _trigger_datasource_batch(
|
||||
)
|
||||
continue
|
||||
|
||||
previous_task_ids[datasource.id] = None
|
||||
success = run_collector_now(datasource.source)
|
||||
if not success:
|
||||
failed_sources.append(
|
||||
{
|
||||
"id": datasource.id,
|
||||
"source": datasource.source,
|
||||
"name": datasource.name,
|
||||
"reason": "trigger_failed",
|
||||
}
|
||||
)
|
||||
continue
|
||||
task = await enqueue_datasource_job(
|
||||
db,
|
||||
datasource,
|
||||
JOB_TYPE_COLLECT,
|
||||
payload={"force": force, "trigger": "batch"},
|
||||
)
|
||||
|
||||
triggered_sources.append(
|
||||
{
|
||||
"id": datasource.id,
|
||||
"source": datasource.source,
|
||||
"name": datasource.name,
|
||||
"task_id": None,
|
||||
"task_id": task.id,
|
||||
}
|
||||
)
|
||||
|
||||
latest_task_ids = await _load_latest_task_ids(
|
||||
db,
|
||||
[datasource.id for datasource in datasources],
|
||||
)
|
||||
for datasource_id in previous_task_ids:
|
||||
previous_task_ids[datasource_id] = latest_task_ids.get(datasource_id)
|
||||
|
||||
for _ in range(20):
|
||||
await asyncio.sleep(0.1)
|
||||
pending = [item for item in triggered_sources if item["task_id"] is None]
|
||||
if not pending:
|
||||
break
|
||||
latest_task_ids = await _load_latest_task_ids(
|
||||
db,
|
||||
[item["id"] for item in pending],
|
||||
)
|
||||
for item in pending:
|
||||
task_id = latest_task_ids.get(item["id"])
|
||||
if task_id is not None and task_id != previous_task_ids.get(item["id"]):
|
||||
item["task_id"] = task_id
|
||||
|
||||
return {
|
||||
"status": "triggered" if triggered_sources else "partial",
|
||||
"message": f"Triggered {len(triggered_sources)} data sources",
|
||||
result = {
|
||||
"status": "queued" if triggered_sources else "partial",
|
||||
"message": f"Queued {len(triggered_sources)} data source jobs",
|
||||
"force": force,
|
||||
"triggered": triggered_sources,
|
||||
"skipped": skipped_sources,
|
||||
"failed": failed_sources,
|
||||
}
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event=f"collector.trigger.{trigger_kind}.completed",
|
||||
message="Datasource batch trigger completed",
|
||||
category="collector",
|
||||
service="api",
|
||||
module=__name__,
|
||||
user_id=actor_id,
|
||||
context={
|
||||
"trigger_kind": trigger_kind,
|
||||
"force": force,
|
||||
"status": result["status"],
|
||||
"requested_count": len(datasources),
|
||||
"triggered_count": len(triggered_sources),
|
||||
"skipped_count": len(skipped_sources),
|
||||
"failed_count": len(failed_sources),
|
||||
"triggered_sources": [item["source"] for item in triggered_sources],
|
||||
"skipped_reasons": [item["reason"] for item in skipped_sources],
|
||||
},
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
async def get_datasource_record(db: AsyncSession, source_id: str) -> Optional[DataSource]:
|
||||
@@ -780,7 +788,13 @@ async def trigger_all_datasources(
|
||||
.order_by(DataSource.module, DataSource.id)
|
||||
)
|
||||
datasources = result.scalars().all()
|
||||
return await _trigger_datasource_batch(db, datasources, force=force)
|
||||
return await _trigger_datasource_batch(
|
||||
db,
|
||||
datasources,
|
||||
force=force,
|
||||
actor_id=current_user.id,
|
||||
trigger_kind="all",
|
||||
)
|
||||
|
||||
|
||||
@router.post("/trigger-batch")
|
||||
@@ -816,7 +830,13 @@ async def trigger_datasource_batch(
|
||||
collected=None if payload.source_ids else payload.collected,
|
||||
credential_status=None if payload.source_ids else payload.credential_status,
|
||||
)
|
||||
return await _trigger_datasource_batch(db, datasources, force=payload.force)
|
||||
return await _trigger_datasource_batch(
|
||||
db,
|
||||
datasources,
|
||||
force=payload.force,
|
||||
actor_id=current_user.id,
|
||||
trigger_kind="batch",
|
||||
)
|
||||
|
||||
|
||||
@router.get("/snapshots")
|
||||
@@ -992,8 +1012,24 @@ async def trigger_datasource(
|
||||
if not datasource.is_active:
|
||||
raise HTTPException(status_code=400, detail="Data source is disabled")
|
||||
|
||||
running_task = await get_running_task(db, datasource.id)
|
||||
running_task = await get_active_datasource_job(db, datasource.id, task_types=(JOB_TYPE_COLLECT,))
|
||||
if running_task is not None and not force:
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="collector.trigger.single.skipped_already_running",
|
||||
message="Datasource trigger skipped because a task is already running",
|
||||
category="collector",
|
||||
level="warning",
|
||||
service="api",
|
||||
module=__name__,
|
||||
user_id=current_user.id,
|
||||
context={
|
||||
"collector_name": datasource.source,
|
||||
"datasource_id": datasource.id,
|
||||
"task_id": running_task.id,
|
||||
"status": "skipped",
|
||||
},
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=409,
|
||||
detail={
|
||||
@@ -1013,31 +1049,42 @@ async def trigger_datasource(
|
||||
)
|
||||
|
||||
if running_task is not None and force:
|
||||
cancelled = await cancel_running_collector_now(datasource.source)
|
||||
if not cancelled:
|
||||
await rollback_orphaned_running_task(db, datasource, running_task)
|
||||
await request_cancel_datasource_task(
|
||||
db,
|
||||
running_task,
|
||||
reason="superseded_by_forced_collection",
|
||||
)
|
||||
|
||||
previous_task_id = await get_latest_task_id_for_datasource(datasource.id)
|
||||
success = run_collector_now(datasource.source)
|
||||
if not success:
|
||||
raise HTTPException(status_code=500, detail=f"Failed to trigger collector '{datasource.source}'")
|
||||
|
||||
task_id = None
|
||||
for _ in range(20):
|
||||
await asyncio.sleep(0.1)
|
||||
task_id = await get_latest_task_id_for_datasource(datasource.id)
|
||||
if task_id is not None and task_id != previous_task_id:
|
||||
break
|
||||
if task_id == previous_task_id:
|
||||
task_id = None
|
||||
task = await enqueue_datasource_job(
|
||||
db,
|
||||
datasource,
|
||||
JOB_TYPE_COLLECT,
|
||||
payload={"force": force, "trigger": "single"},
|
||||
)
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="collector.trigger.single.completed",
|
||||
message="Datasource trigger queued",
|
||||
category="collector",
|
||||
service="api",
|
||||
module=__name__,
|
||||
user_id=current_user.id,
|
||||
context={
|
||||
"collector_name": datasource.source,
|
||||
"datasource_id": datasource.id,
|
||||
"task_id": task.id,
|
||||
"force": force,
|
||||
"status": "queued",
|
||||
},
|
||||
)
|
||||
|
||||
return {
|
||||
"status": "triggered",
|
||||
"status": "queued",
|
||||
"source_id": datasource.id,
|
||||
"task_id": task_id,
|
||||
"task_id": task.id,
|
||||
"collector_name": datasource.source,
|
||||
"force": force,
|
||||
"message": f"Collector '{datasource.source}' has been triggered",
|
||||
"message": f"Collector '{datasource.source}' has been queued",
|
||||
}
|
||||
|
||||
|
||||
@@ -1051,22 +1098,30 @@ async def clear_datasource_data(
|
||||
if not datasource:
|
||||
raise HTTPException(status_code=404, detail="Data source not found")
|
||||
|
||||
result = await db.execute(
|
||||
select(func.count(CollectedData.id)).where(CollectedData.source == datasource.source)
|
||||
active_task = await get_active_datasource_job(db, datasource.id)
|
||||
if active_task is not None:
|
||||
raise HTTPException(
|
||||
status_code=409,
|
||||
detail={
|
||||
"reason": "datasource_job_in_progress",
|
||||
"message": "当前数据源已有任务在执行,请等待完成或先取消任务。",
|
||||
"task_id": active_task.id,
|
||||
"task_type": active_task.task_type,
|
||||
"status": active_task.status,
|
||||
},
|
||||
)
|
||||
task = await enqueue_datasource_job(
|
||||
db,
|
||||
datasource,
|
||||
JOB_TYPE_CLEAR_DATA,
|
||||
payload={"source": datasource.source},
|
||||
)
|
||||
count = result.scalar() or 0
|
||||
|
||||
if count == 0:
|
||||
return {"status": "success", "message": "No data to clear", "deleted_count": 0}
|
||||
|
||||
delete_query = CollectedData.__table__.delete().where(CollectedData.source == datasource.source)
|
||||
await db.execute(delete_query)
|
||||
await db.commit()
|
||||
|
||||
return {
|
||||
"status": "success",
|
||||
"message": f"Cleared {count} records for data source '{datasource.name}'",
|
||||
"deleted_count": count,
|
||||
"status": "queued",
|
||||
"message": f"Queued data clearing for data source '{datasource.name}'",
|
||||
"task_id": task.id,
|
||||
"deleted_count": None,
|
||||
}
|
||||
|
||||
|
||||
@@ -1080,16 +1135,44 @@ async def clear_datasource_cache(
|
||||
if not datasource:
|
||||
raise HTTPException(status_code=404, detail="Data source not found")
|
||||
|
||||
earth_deleted_count = invalidate_earth_layer_cache_for_source(datasource.source)
|
||||
dashboard_deleted_count = int(cache.delete("dashboard:stats")) + int(cache.delete("dashboard:summary"))
|
||||
deleted_count = earth_deleted_count + dashboard_deleted_count
|
||||
task = await enqueue_datasource_job(
|
||||
db,
|
||||
datasource,
|
||||
JOB_TYPE_CLEAR_CACHE,
|
||||
payload={"source": datasource.source},
|
||||
dedupe_key=f"clear_cache:{datasource.source}",
|
||||
)
|
||||
|
||||
return {
|
||||
"status": "success",
|
||||
"message": f"Cleared {deleted_count} cache keys for data source '{datasource.name}'",
|
||||
"deleted_count": deleted_count,
|
||||
"earth_layer_deleted_count": earth_deleted_count,
|
||||
"dashboard_deleted_count": dashboard_deleted_count,
|
||||
"status": "queued",
|
||||
"message": f"Queued cache clearing for data source '{datasource.name}'",
|
||||
"task_id": task.id,
|
||||
"deleted_count": None,
|
||||
}
|
||||
|
||||
|
||||
@router.post("/{source_id}/tasks/{task_id}/cancel")
|
||||
async def cancel_datasource_task(
|
||||
source_id: str,
|
||||
task_id: int,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
datasource = await get_datasource_record(db, source_id)
|
||||
if not datasource:
|
||||
raise HTTPException(status_code=404, detail="Data source not found")
|
||||
|
||||
task = await db.get(CollectionTask, task_id)
|
||||
if not task or task.datasource_id != datasource.id:
|
||||
raise HTTPException(status_code=404, detail="Task not found")
|
||||
|
||||
task = await request_cancel_datasource_task(db, task)
|
||||
return {
|
||||
"status": "cancelled" if task.completed_at else "cancelling",
|
||||
"task_id": task.id,
|
||||
"task_type": task.task_type,
|
||||
"phase": task.phase,
|
||||
"requested_cancel_at": to_iso8601_utc(task.requested_cancel_at),
|
||||
}
|
||||
|
||||
|
||||
@@ -1109,7 +1192,7 @@ async def get_task_status(
|
||||
if not task or task.datasource_id != datasource.id:
|
||||
raise HTTPException(status_code=404, detail="Task not found")
|
||||
else:
|
||||
task = await get_running_task(db, datasource.id)
|
||||
task = await get_active_datasource_job(db, datasource.id)
|
||||
if task is None:
|
||||
result = await db.execute(
|
||||
select(CollectionTask)
|
||||
@@ -1134,8 +1217,12 @@ async def get_task_status(
|
||||
}
|
||||
|
||||
return {
|
||||
"is_running": task.status == "running",
|
||||
"is_running": task.status in {JOB_STATUS_QUEUED, JOB_STATUS_RUNNING, JOB_STATUS_CANCELLING}
|
||||
and task.task_type == JOB_TYPE_COLLECT,
|
||||
"is_task_active": task.status in {JOB_STATUS_QUEUED, JOB_STATUS_RUNNING, JOB_STATUS_CANCELLING},
|
||||
"task_id": task.id,
|
||||
"task_type": task.task_type,
|
||||
"task_status": task.status,
|
||||
"progress": task.progress,
|
||||
"phase": task.phase,
|
||||
"phase_progress": task.phase_progress,
|
||||
@@ -1146,5 +1233,6 @@ async def get_task_status(
|
||||
"records_processed": task.records_processed,
|
||||
"total_records": task.total_records,
|
||||
"status": task.status,
|
||||
"requested_cancel_at": to_iso8601_utc(task.requested_cancel_at),
|
||||
"error_message": task.error_message,
|
||||
}
|
||||
|
||||
@@ -9,13 +9,18 @@ from uuid import uuid4
|
||||
from fastapi import APIRouter, Depends, File, HTTPException, Request, UploadFile, status
|
||||
from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
|
||||
from pydantic import BaseModel, Field
|
||||
from sqlalchemy import delete, select, text
|
||||
from sqlalchemy import delete, func, select, text
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.config import settings as app_settings
|
||||
from app.core.security import decode_token, get_current_user, redis_client
|
||||
from app.db.session import get_db
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.models.datasource import DataSource
|
||||
from app.models.datasource_config import DataSourceConfig
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.models.user import User
|
||||
from app.services.tv_streams import get_tv_settings_payload
|
||||
from app.services.earth_boundaries import (
|
||||
EarthBoundaryBuildError,
|
||||
get_boundary_build_status,
|
||||
@@ -31,9 +36,17 @@ REPO_ROOT = Path(__file__).resolve().parents[4]
|
||||
EARTH_BRAND_ASSET_DIR = REPO_ROOT / "data" / "earth-brand"
|
||||
EARTH_BRAND_ASSET_URL_PREFIX = "/earth-brand-assets"
|
||||
EARTH_BRAND_CATEGORY = "earth_brand"
|
||||
EARTH_ABOUT_CATEGORY = "earth_about"
|
||||
SYSTEM_SETTINGS_CATEGORY = "system"
|
||||
MAX_EARTH_BRAND_ASSET_BYTES = 3 * 1024 * 1024
|
||||
ALLOWED_EARTH_BRAND_EXTENSIONS = {".png", ".jpg", ".jpeg", ".webp", ".svg"}
|
||||
|
||||
|
||||
def _app_version_label() -> str:
|
||||
version = str(app_settings.VERSION or "").strip() or "0.0.0"
|
||||
return version if version.startswith("v") else f"v{version}"
|
||||
|
||||
|
||||
DEFAULT_EARTH_BRAND = {
|
||||
"logo_src": "/earth/assets/brand/earth-logo.png",
|
||||
"title_src": "/earth/assets/brand/title-zh.png",
|
||||
@@ -44,6 +57,20 @@ DEFAULT_EARTH_BRAND = {
|
||||
"title_alt": "智能星球计划",
|
||||
}
|
||||
|
||||
DEFAULT_EARTH_ABOUT = {
|
||||
"logo_src": "/earth/assets/brand/lim-logo.png",
|
||||
"kicker": "About",
|
||||
"title": "智能星球计划",
|
||||
"version": _app_version_label(),
|
||||
"description": "面向临空场景下的智能媒体研究、全球态势感知与多源开放数据巡航,提供可视化观测、事件聚合与交互式探索能力。",
|
||||
"meta": [
|
||||
{"label": "出品方", "value": "浙江大学临空智能媒体研究院"},
|
||||
{"label": "策划人", "value": "方兴东、黄柳青"},
|
||||
{"label": "产品兼开发者", "value": "钱坤、张鸽、齐鹏"},
|
||||
],
|
||||
}
|
||||
EARTH_ABOUT_LEGACY_PLANNER_VALUE = "黄柳青"
|
||||
|
||||
|
||||
class EarthBoundaryConfigPayload(BaseModel):
|
||||
config: dict[str, Any] = Field(default_factory=dict)
|
||||
@@ -59,6 +86,20 @@ class EarthBrandPayload(BaseModel):
|
||||
title_alt: str = Field(default=DEFAULT_EARTH_BRAND["title_alt"], max_length=200)
|
||||
|
||||
|
||||
class EarthAboutMetaItem(BaseModel):
|
||||
label: str = Field(default="", max_length=80)
|
||||
value: str = Field(default="", max_length=240)
|
||||
|
||||
|
||||
class EarthAboutPayload(BaseModel):
|
||||
logo_src: str = Field(default=DEFAULT_EARTH_ABOUT["logo_src"], max_length=1000)
|
||||
kicker: str = Field(default=DEFAULT_EARTH_ABOUT["kicker"], max_length=80)
|
||||
title: str = Field(default=DEFAULT_EARTH_ABOUT["title"], max_length=160)
|
||||
version: str = Field(default=DEFAULT_EARTH_ABOUT["version"], max_length=80)
|
||||
description: str = Field(default=DEFAULT_EARTH_ABOUT["description"], max_length=800)
|
||||
meta: list[EarthAboutMetaItem] = Field(default_factory=list)
|
||||
|
||||
|
||||
def _normalize_earth_brand_payload(payload: dict[str, Any] | None) -> dict[str, str]:
|
||||
merged = DEFAULT_EARTH_BRAND.copy()
|
||||
if payload:
|
||||
@@ -76,6 +117,47 @@ def _normalize_earth_brand_payload(payload: dict[str, Any] | None) -> dict[str,
|
||||
return merged
|
||||
|
||||
|
||||
def _normalize_earth_about_payload(payload: dict[str, Any] | None) -> dict[str, Any]:
|
||||
merged: dict[str, Any] = {
|
||||
key: value
|
||||
for key, value in DEFAULT_EARTH_ABOUT.items()
|
||||
if key != "meta"
|
||||
}
|
||||
raw_meta = DEFAULT_EARTH_ABOUT["meta"]
|
||||
if payload:
|
||||
for key in ("logo_src", "kicker", "title", "description"):
|
||||
value = payload.get(key)
|
||||
if value is not None:
|
||||
merged[key] = str(value).strip()
|
||||
raw_meta = payload.get("meta") if isinstance(payload.get("meta"), list) else raw_meta
|
||||
merged["version"] = _app_version_label()
|
||||
|
||||
for key, default_value in DEFAULT_EARTH_ABOUT.items():
|
||||
if key == "meta":
|
||||
continue
|
||||
if not merged.get(key):
|
||||
merged[key] = default_value
|
||||
|
||||
normalized_meta: list[dict[str, str]] = []
|
||||
for item in raw_meta:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
label = str(item.get("label") or "").strip()
|
||||
value = str(item.get("value") or "").strip()
|
||||
if label == "策划人" and value == EARTH_ABOUT_LEGACY_PLANNER_VALUE:
|
||||
value = "方兴东、黄柳青"
|
||||
if label or value:
|
||||
normalized_meta.append({"label": label, "value": value})
|
||||
if not normalized_meta:
|
||||
normalized_meta = [dict(item) for item in DEFAULT_EARTH_ABOUT["meta"]]
|
||||
merged["meta"] = normalized_meta
|
||||
return merged
|
||||
|
||||
|
||||
def _is_demo_mode_enabled(payload: Any) -> bool:
|
||||
return bool(payload.get("demo_mode")) if isinstance(payload, dict) else False
|
||||
|
||||
|
||||
async def _get_earth_brand_record(db: AsyncSession) -> SystemSetting | None:
|
||||
result = await db.execute(
|
||||
select(SystemSetting).where(SystemSetting.category == EARTH_BRAND_CATEGORY)
|
||||
@@ -91,6 +173,56 @@ async def _get_earth_brand_payload(db: AsyncSession) -> dict[str, Any]:
|
||||
}
|
||||
|
||||
|
||||
async def _get_earth_about_record(db: AsyncSession) -> SystemSetting | None:
|
||||
result = await db.execute(
|
||||
select(SystemSetting).where(SystemSetting.category == EARTH_ABOUT_CATEGORY)
|
||||
)
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
|
||||
async def _get_earth_about_payload(db: AsyncSession) -> dict[str, Any]:
|
||||
record = await _get_earth_about_record(db)
|
||||
return {
|
||||
"about": _normalize_earth_about_payload(record.payload if record else None),
|
||||
"is_default": record is None,
|
||||
}
|
||||
|
||||
|
||||
async def _get_optional_current_user(
|
||||
credentials: HTTPAuthorizationCredentials | None = Depends(optional_bearer),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
) -> User | None:
|
||||
if credentials is None:
|
||||
return None
|
||||
token = credentials.credentials
|
||||
if redis_client.sismember("blacklisted_tokens", token):
|
||||
return None
|
||||
payload = decode_token(token)
|
||||
if payload is None or payload.get("type") != "access":
|
||||
return None
|
||||
user_id = payload.get("sub")
|
||||
if user_id is None:
|
||||
return None
|
||||
result = await db.execute(
|
||||
text(
|
||||
"SELECT id, username, email, password_hash, role, is_active, gatekeeper_groups FROM users WHERE id = :id"
|
||||
),
|
||||
{"id": int(user_id)},
|
||||
)
|
||||
row = result.fetchone()
|
||||
if row is None or not row[5]:
|
||||
return None
|
||||
user = User()
|
||||
user.id = row[0]
|
||||
user.username = row[1]
|
||||
user.email = row[2]
|
||||
user.password_hash = row[3]
|
||||
user.role = row[4]
|
||||
user.is_active = row[5]
|
||||
user.gatekeeper_groups = row[6] or []
|
||||
return user
|
||||
|
||||
|
||||
@router.get("/brand")
|
||||
async def get_earth_brand(db: AsyncSession = Depends(get_db)):
|
||||
return await _get_earth_brand_payload(db)
|
||||
@@ -159,46 +291,108 @@ async def upload_earth_brand_asset(
|
||||
return {"url": asset_url, "filename": safe_name, "content_type": file.content_type}
|
||||
|
||||
|
||||
@router.get("/about")
|
||||
async def get_earth_about(db: AsyncSession = Depends(get_db)):
|
||||
return await _get_earth_about_payload(db)
|
||||
|
||||
|
||||
@router.put("/about")
|
||||
async def update_earth_about(
|
||||
payload: EarthAboutPayload,
|
||||
_current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
normalized = _normalize_earth_about_payload(payload.model_dump())
|
||||
record = await _get_earth_about_record(db)
|
||||
if record is None:
|
||||
record = SystemSetting(category=EARTH_ABOUT_CATEGORY, payload=normalized)
|
||||
db.add(record)
|
||||
else:
|
||||
record.payload = normalized
|
||||
await db.commit()
|
||||
await db.refresh(record)
|
||||
return {"status": "updated", "about": _normalize_earth_about_payload(record.payload), "is_default": False}
|
||||
|
||||
|
||||
@router.delete("/about")
|
||||
async def reset_earth_about(
|
||||
_current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
await db.execute(delete(SystemSetting).where(SystemSetting.category == EARTH_ABOUT_CATEGORY))
|
||||
await db.commit()
|
||||
return {"status": "reset", "about": _normalize_earth_about_payload(None), "is_default": True}
|
||||
|
||||
|
||||
@router.get("/oobe-status")
|
||||
async def get_earth_oobe_status(
|
||||
current_user: User | None = Depends(_get_optional_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
current_count_result = await db.execute(
|
||||
select(func.count(CollectedData.id)).where(CollectedData.is_current.is_(True))
|
||||
)
|
||||
current_record_count = int(current_count_result.scalar() or 0)
|
||||
system_result = await db.execute(
|
||||
select(SystemSetting).where(SystemSetting.category == SYSTEM_SETTINGS_CATEGORY)
|
||||
)
|
||||
system_record = system_result.scalar_one_or_none()
|
||||
demo_mode = _is_demo_mode_enabled(system_record.payload if system_record else None)
|
||||
|
||||
datasource_count_result = await db.execute(select(func.count(DataSource.id)))
|
||||
datasource_count = int(datasource_count_result.scalar() or 0)
|
||||
active_datasource_count_result = await db.execute(
|
||||
select(func.count(DataSource.id)).where(DataSource.is_active.is_(True))
|
||||
)
|
||||
active_datasource_count = int(active_datasource_count_result.scalar() or 0)
|
||||
config_result = await db.execute(select(func.count(DataSourceConfig.id)))
|
||||
custom_config_count = int(config_result.scalar() or 0)
|
||||
|
||||
tv_payload = await get_tv_settings_payload(db)
|
||||
tv_sources = tv_payload.get("sources") if isinstance(tv_payload, dict) else []
|
||||
tv_source_count = len(tv_sources) if isinstance(tv_sources, list) else 0
|
||||
|
||||
boundary_status = get_boundary_status()
|
||||
has_core_layers = bool(boundary_status.get("ready") or boundary_status.get("available") or boundary_status.get("status") in {"ready", "built", "ok"})
|
||||
has_collected_data = current_record_count > 0
|
||||
ready = has_collected_data
|
||||
|
||||
suggestions: list[str] = []
|
||||
if demo_mode:
|
||||
suggestions.append("演示模式已开启")
|
||||
if not current_user:
|
||||
suggestions.append("登录控制台")
|
||||
if not has_collected_data:
|
||||
suggestions.append("触发数据源采集")
|
||||
if not custom_config_count:
|
||||
suggestions.append("确认采集器配置")
|
||||
if not has_core_layers:
|
||||
suggestions.append("构建或启用 Earth 图层")
|
||||
|
||||
return {
|
||||
"ready": ready,
|
||||
"demo_mode": demo_mode,
|
||||
"authenticated": current_user is not None,
|
||||
"needs_login": current_user is None and not ready and not demo_mode,
|
||||
"has_collected_data": has_collected_data,
|
||||
"has_tv_sources": tv_source_count > 0,
|
||||
"has_core_layers": has_core_layers,
|
||||
"current_record_count": current_record_count,
|
||||
"datasource_count": datasource_count,
|
||||
"active_datasource_count": active_datasource_count,
|
||||
"custom_config_count": custom_config_count,
|
||||
"tv_source_count": tv_source_count,
|
||||
"suggestions": suggestions,
|
||||
"login_url": "/login?next=/datasources",
|
||||
"datasources_url": "/datasources",
|
||||
"collection_url": "/collection-management",
|
||||
}
|
||||
|
||||
|
||||
@router.get("/boundaries/status")
|
||||
async def get_earth_boundary_status():
|
||||
return get_boundary_status()
|
||||
|
||||
|
||||
async def _get_optional_current_user(
|
||||
credentials: HTTPAuthorizationCredentials | None = Depends(optional_bearer),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
) -> User | None:
|
||||
if credentials is None:
|
||||
return None
|
||||
token = credentials.credentials
|
||||
if redis_client.sismember("blacklisted_tokens", token):
|
||||
return None
|
||||
payload = decode_token(token)
|
||||
if payload is None or payload.get("type") != "access":
|
||||
return None
|
||||
user_id = payload.get("sub")
|
||||
if user_id is None:
|
||||
return None
|
||||
result = await db.execute(
|
||||
text(
|
||||
"SELECT id, username, email, password_hash, role, is_active, gatekeeper_groups FROM users WHERE id = :id"
|
||||
),
|
||||
{"id": int(user_id)},
|
||||
)
|
||||
row = result.fetchone()
|
||||
if row is None or not row[5]:
|
||||
return None
|
||||
user = User()
|
||||
user.id = row[0]
|
||||
user.username = row[1]
|
||||
user.email = row[2]
|
||||
user.password_hash = row[3]
|
||||
user.role = row[4]
|
||||
user.is_active = row[5]
|
||||
user.gatekeeper_groups = row[6] or []
|
||||
return user
|
||||
|
||||
|
||||
def _is_loopback_request(request: Request) -> bool:
|
||||
host = request.client.host if request.client else ""
|
||||
return host in {"127.0.0.1", "::1", "localhost"} or host.startswith("127.")
|
||||
|
||||
190
backend/app/api/v1/interactables.py
Normal file
190
backend/app/api/v1/interactables.py
Normal file
@@ -0,0 +1,190 @@
|
||||
"""CRUD APIs for persistent Earth interactables."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Response, status
|
||||
from pydantic import BaseModel, Field, field_validator
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.security import get_current_user
|
||||
from app.db.session import get_db
|
||||
from app.models.earth_interactable import EarthInteractable
|
||||
from app.models.user import User
|
||||
from app.services.earth_interactables import (
|
||||
build_interactable_event,
|
||||
interactables_to_geojson,
|
||||
invalidate_interactable_cache,
|
||||
list_interactables,
|
||||
normalize_interactable_id,
|
||||
publish_interactable_event,
|
||||
serialize_interactable,
|
||||
)
|
||||
from app.services.earth_layer_cache import (
|
||||
EarthLayerCachePolicy,
|
||||
earth_layer_cache,
|
||||
get_or_build_layer_payload,
|
||||
)
|
||||
|
||||
router = APIRouter()
|
||||
INTERACTABLE_CACHE_POLICY = EarthLayerCachePolicy(
|
||||
fresh_ttl_seconds=60,
|
||||
stale_ttl_seconds=10 * 60,
|
||||
max_features=5000,
|
||||
)
|
||||
|
||||
|
||||
class InteractableCreate(BaseModel):
|
||||
id: str | None = Field(default=None, max_length=160)
|
||||
layer: str = Field(default="default", min_length=1, max_length=80)
|
||||
kind: str = Field(default="default", min_length=1, max_length=80)
|
||||
label: str = Field(default="", max_length=255)
|
||||
description: str = Field(default="", max_length=4000)
|
||||
latitude: float = Field(ge=-90, le=90)
|
||||
longitude: float = Field(ge=-180, le=180)
|
||||
altitude: float | None = None
|
||||
properties: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
@field_validator("layer", "kind")
|
||||
@classmethod
|
||||
def normalize_key(cls, value: str) -> str:
|
||||
normalized = str(value or "").strip()
|
||||
if not normalized:
|
||||
raise ValueError("must not be empty")
|
||||
return normalized
|
||||
|
||||
|
||||
class InteractableUpdate(BaseModel):
|
||||
layer: str | None = Field(default=None, min_length=1, max_length=80)
|
||||
kind: str | None = Field(default=None, min_length=1, max_length=80)
|
||||
label: str | None = Field(default=None, max_length=255)
|
||||
description: str | None = Field(default=None, max_length=4000)
|
||||
latitude: float | None = Field(default=None, ge=-90, le=90)
|
||||
longitude: float | None = Field(default=None, ge=-180, le=180)
|
||||
altitude: float | None = None
|
||||
properties: dict[str, Any] | None = None
|
||||
|
||||
|
||||
@router.get("")
|
||||
async def get_interactables(
|
||||
response: Response,
|
||||
layer: str | None = Query(default=None),
|
||||
include_deleted: bool = Query(default=False),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
items = await list_interactables(db, layer=layer, include_deleted=include_deleted)
|
||||
response.headers["X-Planet-Interactables-Count"] = str(len(items))
|
||||
return {"items": [serialize_interactable(item) for item in items]}
|
||||
|
||||
|
||||
@router.get("/geojson")
|
||||
async def get_interactables_geojson(
|
||||
response: Response,
|
||||
layer: str | None = Query(default=None),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
async def build_payload() -> dict[str, Any]:
|
||||
items = await list_interactables(db, layer=layer)
|
||||
return interactables_to_geojson(items)
|
||||
|
||||
payload = await get_or_build_layer_payload(
|
||||
key=earth_layer_cache.key("interactables", layer=layer or "all"),
|
||||
policy=INTERACTABLE_CACHE_POLICY,
|
||||
builder=build_payload,
|
||||
response=response,
|
||||
)
|
||||
response.headers["X-Planet-Interactables-Count"] = str(len(payload.get("features") or []))
|
||||
return payload
|
||||
|
||||
|
||||
@router.post("", status_code=status.HTTP_201_CREATED)
|
||||
async def create_interactable(
|
||||
payload: InteractableCreate,
|
||||
_current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
record_id = normalize_interactable_id(payload.id)
|
||||
existing = await db.get(EarthInteractable, record_id)
|
||||
if existing and not existing.is_deleted:
|
||||
raise HTTPException(status_code=409, detail="Interactable already exists")
|
||||
|
||||
if existing is None:
|
||||
record = EarthInteractable(id=record_id)
|
||||
db.add(record)
|
||||
else:
|
||||
record = existing
|
||||
record.is_deleted = False
|
||||
record.deleted_at = None
|
||||
record.revision += 1
|
||||
|
||||
record.layer = payload.layer
|
||||
record.kind = payload.kind
|
||||
record.label = payload.label
|
||||
record.description = payload.description
|
||||
record.latitude = payload.latitude
|
||||
record.longitude = payload.longitude
|
||||
record.altitude = payload.altitude
|
||||
record.properties = payload.properties
|
||||
|
||||
await db.commit()
|
||||
await db.refresh(record)
|
||||
invalidate_interactable_cache(record.layer)
|
||||
await publish_interactable_event("created", record)
|
||||
return {"item": serialize_interactable(record)}
|
||||
|
||||
|
||||
@router.get("/{interactable_id}")
|
||||
async def get_interactable(interactable_id: str, db: AsyncSession = Depends(get_db)):
|
||||
record = await db.get(EarthInteractable, interactable_id)
|
||||
if record is None or record.is_deleted:
|
||||
raise HTTPException(status_code=404, detail="Interactable not found")
|
||||
return {"item": serialize_interactable(record)}
|
||||
|
||||
|
||||
@router.patch("/{interactable_id}")
|
||||
async def update_interactable(
|
||||
interactable_id: str,
|
||||
payload: InteractableUpdate,
|
||||
_current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
record = await db.get(EarthInteractable, interactable_id)
|
||||
if record is None or record.is_deleted:
|
||||
raise HTTPException(status_code=404, detail="Interactable not found")
|
||||
|
||||
previous_layer = record.layer
|
||||
patch = payload.model_dump(exclude_unset=True)
|
||||
for key, value in patch.items():
|
||||
setattr(record, key, value)
|
||||
record.revision += 1
|
||||
|
||||
await db.commit()
|
||||
await db.refresh(record)
|
||||
invalidate_interactable_cache(previous_layer)
|
||||
if record.layer != previous_layer:
|
||||
invalidate_interactable_cache(record.layer)
|
||||
await publish_interactable_event("updated", record)
|
||||
return {"item": serialize_interactable(record)}
|
||||
|
||||
|
||||
@router.delete("/{interactable_id}")
|
||||
async def delete_interactable(
|
||||
interactable_id: str,
|
||||
_current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
record = await db.get(EarthInteractable, interactable_id)
|
||||
if record is None or record.is_deleted:
|
||||
raise HTTPException(status_code=404, detail="Interactable not found")
|
||||
|
||||
record.is_deleted = True
|
||||
record.deleted_at = datetime.now(UTC)
|
||||
record.revision += 1
|
||||
await db.commit()
|
||||
await db.refresh(record)
|
||||
invalidate_interactable_cache(record.layer)
|
||||
await publish_interactable_event("deleted", record)
|
||||
event = build_interactable_event(action="deleted", record=record, include_item=True)
|
||||
return {"deleted": True, "event": event}
|
||||
@@ -11,6 +11,7 @@ from dotenv import dotenv_values
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.logging import get_logger
|
||||
from app.core.security import get_current_user
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.core.config import settings as app_settings
|
||||
@@ -66,8 +67,10 @@ from app.services.llm_provider_catalog import (
|
||||
from app.services.scheduler import sync_datasource_job
|
||||
from app.services.tv_streams import DEFAULT_TV_SETTINGS, get_tv_settings_payload, normalize_tv_settings
|
||||
from app.services.persistent_logs import record_audit_log
|
||||
from app.services.business_logs import emit_business_log, exception_context
|
||||
|
||||
router = APIRouter()
|
||||
logger = get_logger(__name__, service="api")
|
||||
AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS = 5
|
||||
AI_CONNECTION_TEST_PROMPT_KEY = "ai.connection_test"
|
||||
SECRET_REVEAL_ROLES = {"admin", "super_admin"}
|
||||
@@ -79,6 +82,7 @@ DEFAULT_SETTINGS = {
|
||||
"auto_refresh": True,
|
||||
"data_retention_days": 30,
|
||||
"max_concurrent_tasks": 5,
|
||||
"demo_mode": False,
|
||||
},
|
||||
"notifications": {
|
||||
"email_enabled": False,
|
||||
@@ -203,6 +207,7 @@ class SystemSettingsUpdate(BaseModel):
|
||||
auto_refresh: bool = True
|
||||
data_retention_days: int = Field(default=30, ge=1, le=3650)
|
||||
max_concurrent_tasks: int = Field(default=5, ge=1, le=50)
|
||||
demo_mode: bool = False
|
||||
|
||||
|
||||
class NotificationSettingsUpdate(BaseModel):
|
||||
@@ -699,6 +704,18 @@ async def _validate_ai_provider_full_connection(ai_payload: dict) -> dict:
|
||||
retry_attempts=runtime_config["retry_attempts"],
|
||||
llm_config=runtime_config.get("llm_config") or {},
|
||||
)
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="settings.ai_provider.full_connection.start",
|
||||
message="AI provider full connection validation started",
|
||||
category="ai",
|
||||
service="api",
|
||||
module=__name__,
|
||||
context={
|
||||
"provider": runtime_config.get("llm_config", {}).get("provider"),
|
||||
"model": runtime_config.get("llm_config", {}).get("model"),
|
||||
},
|
||||
)
|
||||
status_result = await client.get_status()
|
||||
if not status_result.configured:
|
||||
raise HTTPException(
|
||||
@@ -715,6 +732,19 @@ async def _validate_ai_provider_full_connection(ai_payload: dict) -> dict:
|
||||
constraints=["回复尽量简短。"],
|
||||
)
|
||||
)
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="settings.ai_provider.full_connection.success",
|
||||
message="AI provider full connection validation completed",
|
||||
category="ai",
|
||||
service="api",
|
||||
module=__name__,
|
||||
context={
|
||||
"provider": analysis_result.provider,
|
||||
"model": analysis_result.model,
|
||||
"configured": status_result.configured,
|
||||
},
|
||||
)
|
||||
return {
|
||||
"status": status_result.model_dump(),
|
||||
"provider": analysis_result.provider,
|
||||
@@ -1612,9 +1642,34 @@ async def connect_ai_provider_integration(
|
||||
llm_config=quick_llm_config,
|
||||
)
|
||||
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="settings.ai_provider.connect.start",
|
||||
message="AI provider connection test started",
|
||||
category="ai",
|
||||
service="api",
|
||||
module=__name__,
|
||||
user_id=current_user.id,
|
||||
context={
|
||||
"provider": payload.provider,
|
||||
"model": payload.model,
|
||||
"timeout_seconds": min(int(runtime_config["timeout_seconds"] or 60), AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS),
|
||||
},
|
||||
)
|
||||
try:
|
||||
status_result = await client.get_status()
|
||||
if not status_result.configured:
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="settings.ai_provider.connect.failed",
|
||||
message="AI provider connection test failed because provider is incomplete",
|
||||
category="ai",
|
||||
level="warning",
|
||||
service="api",
|
||||
module=__name__,
|
||||
user_id=current_user.id,
|
||||
context={"provider": payload.provider, "model": payload.model, "configured": False},
|
||||
)
|
||||
return {
|
||||
"success": False,
|
||||
"connected": False,
|
||||
@@ -1628,6 +1683,21 @@ async def connect_ai_provider_integration(
|
||||
AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS,
|
||||
),
|
||||
)
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="settings.ai_provider.connect.success",
|
||||
message="AI provider connection test completed",
|
||||
category="ai",
|
||||
service="api",
|
||||
module=__name__,
|
||||
user_id=current_user.id,
|
||||
context={
|
||||
"provider": payload.provider,
|
||||
"model": payload.model,
|
||||
"configured": True,
|
||||
"lightweight_status": lightweight_result.get("status"),
|
||||
},
|
||||
)
|
||||
return {
|
||||
**lightweight_result,
|
||||
"status": status_result.model_dump(),
|
||||
@@ -1639,6 +1709,17 @@ async def connect_ai_provider_integration(
|
||||
"message": str(exc.detail),
|
||||
}
|
||||
except Exception as exc:
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="settings.ai_provider.connect.failed",
|
||||
message="AI provider connection test failed",
|
||||
category="ai",
|
||||
level="error",
|
||||
service="api",
|
||||
module=__name__,
|
||||
user_id=current_user.id,
|
||||
context=exception_context(exc, {"provider": payload.provider, "model": payload.model}),
|
||||
)
|
||||
return {
|
||||
"success": False,
|
||||
"connected": False,
|
||||
@@ -1786,8 +1867,28 @@ async def connect_web_search_integration(
|
||||
runtime_config = _runtime_config_from_web_search_payload(draft_web_search_payload)
|
||||
client = WebSearchClient(runtime_config)
|
||||
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="settings.web_search.connect.start",
|
||||
message="WebSearch connection test started",
|
||||
category="ai_tool",
|
||||
service="api",
|
||||
module=__name__,
|
||||
user_id=current_user.id,
|
||||
context={"provider": runtime_config.default_provider},
|
||||
)
|
||||
try:
|
||||
results = await client.test_connection()
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="settings.web_search.connect.success",
|
||||
message="WebSearch connection test completed",
|
||||
category="ai_tool",
|
||||
service="api",
|
||||
module=__name__,
|
||||
user_id=current_user.id,
|
||||
context={"provider": runtime_config.default_provider, "result_count": len(results)},
|
||||
)
|
||||
return {
|
||||
"success": True,
|
||||
"connected": True,
|
||||
@@ -1796,18 +1897,51 @@ async def connect_web_search_integration(
|
||||
"results": [item.model_dump(mode="json") for item in results[:3]],
|
||||
}
|
||||
except WebSearchConfigurationError as exc:
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="settings.web_search.connect.failed",
|
||||
message="WebSearch connection test failed because configuration is incomplete",
|
||||
category="ai_tool",
|
||||
level="warning",
|
||||
service="api",
|
||||
module=__name__,
|
||||
user_id=current_user.id,
|
||||
context=exception_context(exc, {"provider": runtime_config.default_provider}),
|
||||
)
|
||||
return {
|
||||
"success": False,
|
||||
"connected": False,
|
||||
"message": str(exc),
|
||||
}
|
||||
except WebSearchError as exc:
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="settings.web_search.connect.failed",
|
||||
message="WebSearch connection test failed",
|
||||
category="ai_tool",
|
||||
level="error",
|
||||
service="api",
|
||||
module=__name__,
|
||||
user_id=current_user.id,
|
||||
context=exception_context(exc, {"provider": runtime_config.default_provider}),
|
||||
)
|
||||
return {
|
||||
"success": False,
|
||||
"connected": False,
|
||||
"message": str(exc),
|
||||
}
|
||||
except Exception as exc:
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="settings.web_search.connect.failed",
|
||||
message="WebSearch connection test failed",
|
||||
category="ai_tool",
|
||||
level="error",
|
||||
service="api",
|
||||
module=__name__,
|
||||
user_id=current_user.id,
|
||||
context=exception_context(exc, {"provider": runtime_config.default_provider}),
|
||||
)
|
||||
return {
|
||||
"success": False,
|
||||
"connected": False,
|
||||
@@ -1835,17 +1969,60 @@ async def generate_provider_credential_guide(
|
||||
ai_client: AIProviderClient = Depends(get_ai_provider_client),
|
||||
):
|
||||
try:
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="settings.credential_guide.generate.start",
|
||||
message="Credential guide generation started",
|
||||
category="ai",
|
||||
service="api",
|
||||
module=__name__,
|
||||
user_id=current_user.id,
|
||||
context={"provider": provider},
|
||||
)
|
||||
web_search_client = await get_web_search_client(db)
|
||||
return {
|
||||
"guide": await generate_credential_guide(
|
||||
guide = await generate_credential_guide(
|
||||
db,
|
||||
provider,
|
||||
ai_client,
|
||||
web_search_client,
|
||||
)
|
||||
}
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="settings.credential_guide.generate.success",
|
||||
message="Credential guide generation completed",
|
||||
category="ai",
|
||||
service="api",
|
||||
module=__name__,
|
||||
user_id=current_user.id,
|
||||
context={"provider": provider},
|
||||
)
|
||||
return {"guide": guide}
|
||||
except ValueError as exc:
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="settings.credential_guide.generate.failed",
|
||||
message="Credential guide generation failed",
|
||||
category="ai",
|
||||
level="warning",
|
||||
service="api",
|
||||
module=__name__,
|
||||
user_id=current_user.id,
|
||||
context=exception_context(exc, {"provider": provider}),
|
||||
)
|
||||
raise HTTPException(status_code=404, detail=str(exc)) from exc
|
||||
except Exception as exc:
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="settings.credential_guide.generate.failed",
|
||||
message="Credential guide generation failed",
|
||||
category="ai",
|
||||
level="error",
|
||||
service="api",
|
||||
module=__name__,
|
||||
user_id=current_user.id,
|
||||
context=exception_context(exc, {"provider": provider}),
|
||||
)
|
||||
raise
|
||||
|
||||
|
||||
@router.post("/credential-guides/{provider}/reset")
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
@@ -44,6 +45,42 @@ from app.services.earth_layer_cache import earth_layer_cache
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
def _compact_log_context(context: dict | None) -> str:
|
||||
if not context:
|
||||
return ""
|
||||
allowed = {
|
||||
key: value
|
||||
for key, value in (context or {}).items()
|
||||
if key
|
||||
in {
|
||||
"status",
|
||||
"duration_ms",
|
||||
"provider",
|
||||
"model",
|
||||
"result_provider",
|
||||
"result_model",
|
||||
"collector_name",
|
||||
"datasource_id",
|
||||
"task_id",
|
||||
"snapshot_id",
|
||||
"raw_count",
|
||||
"transformed_count",
|
||||
"saved_count",
|
||||
"created",
|
||||
"updated",
|
||||
"unchanged",
|
||||
"deleted",
|
||||
"result_count",
|
||||
"status_code",
|
||||
"error_type",
|
||||
"error",
|
||||
}
|
||||
}
|
||||
if not allowed:
|
||||
return ""
|
||||
return json.dumps(allowed, ensure_ascii=False, sort_keys=True)
|
||||
|
||||
|
||||
class RestartTaskCreate(BaseModel):
|
||||
action: str
|
||||
|
||||
@@ -374,8 +411,11 @@ async def read_database_log_snapshot(
|
||||
record.occurred_at.isoformat() if record.occurred_at else "",
|
||||
record_level.upper(),
|
||||
record.source,
|
||||
record.category or "",
|
||||
record.event or "",
|
||||
f"request_id={record.request_id}" if record.request_id else "",
|
||||
record.message,
|
||||
_compact_log_context(record.context),
|
||||
]
|
||||
if part
|
||||
)
|
||||
|
||||
@@ -38,7 +38,7 @@ from app.services.compute_center_locations import (
|
||||
upsert_compute_center_location,
|
||||
)
|
||||
from app.services.ai_client import get_ai_provider_client
|
||||
from app.api.v1.settings import get_web_search_client
|
||||
from app.api.v1.settings import get_runtime_web_search_config, get_web_search_client
|
||||
from app.services.collectors.bgp_common import RIPE_RIS_COLLECTOR_COORDS
|
||||
from app.services.location.llm_fallback import (
|
||||
collect_llm_location_fallback_candidate,
|
||||
@@ -105,8 +105,8 @@ LANDING_POINT_CACHE_POLICY = EarthLayerCachePolicy(
|
||||
SATELLITE_CACHE_POLICY = EarthLayerCachePolicy(
|
||||
SATELLITE_CACHE_FRESH_SECONDS,
|
||||
SATELLITE_CACHE_STALE_SECONDS,
|
||||
max_features=8000,
|
||||
max_bytes=10 * BYTES_PER_MIB,
|
||||
max_features=25000,
|
||||
max_bytes=32 * BYTES_PER_MIB,
|
||||
)
|
||||
COMPUTE_CENTER_CACHE_POLICY = EarthLayerCachePolicy(
|
||||
COMPUTE_CENTER_CACHE_FRESH_SECONDS,
|
||||
@@ -1563,15 +1563,7 @@ async def _build_cables_geojson(db: AsyncSession) -> dict[str, Any]:
|
||||
try:
|
||||
records = await _load_current_collected_data(db, "arcgis_cables")
|
||||
|
||||
if not records:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail="No cable data found. Please run the arcgis_cables collector first.",
|
||||
)
|
||||
|
||||
return convert_cable_to_geojson(records)
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.exception_event(
|
||||
"Failed to build cables GeoJSON response",
|
||||
@@ -1625,16 +1617,8 @@ async def _build_landing_points_geojson(db: AsyncSession) -> dict[str, Any]:
|
||||
relation_records,
|
||||
cable_records,
|
||||
)
|
||||
|
||||
if not records:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail="No landing point data found. Please run the arcgis_landing_points collector first.",
|
||||
)
|
||||
|
||||
|
||||
return convert_landing_point_to_geojson(records, city_to_cable_ids_map, cable_id_to_name_map)
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.exception_event(
|
||||
"Failed to build landing points GeoJSON response",
|
||||
@@ -2071,6 +2055,44 @@ class SaveComputeCenterLocationRequest(BaseModel):
|
||||
model_config = {"populate_by_name": True}
|
||||
|
||||
|
||||
async def _compute_center_location_web_search_capability(db: AsyncSession) -> Dict[str, Any]:
|
||||
try:
|
||||
config = await get_runtime_web_search_config(db)
|
||||
except Exception as exc:
|
||||
return {
|
||||
"enabled": False,
|
||||
"provider": None,
|
||||
"reason": f"WebSearch 配置读取失败:{exc}",
|
||||
}
|
||||
provider_config = config.active_provider_config
|
||||
has_api_key = bool((provider_config.api_key or "").strip())
|
||||
if not config.enabled:
|
||||
return {
|
||||
"enabled": False,
|
||||
"provider": config.default_provider,
|
||||
"reason": "WebSearch 未开启,无法进行事实核查定位。",
|
||||
}
|
||||
if not has_api_key:
|
||||
return {
|
||||
"enabled": False,
|
||||
"provider": config.default_provider,
|
||||
"reason": f"WebSearch Provider {config.default_provider} 未配置 API Key。",
|
||||
}
|
||||
return {
|
||||
"enabled": True,
|
||||
"provider": config.default_provider,
|
||||
"reason": "",
|
||||
}
|
||||
|
||||
|
||||
@router.get("/compute-centers/location-capability")
|
||||
async def get_compute_center_location_capability(
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""Return whether fact-checked compute-center location collection can run."""
|
||||
return await _compute_center_location_web_search_capability(db)
|
||||
|
||||
|
||||
@router.post("/compute-centers/{source_id}/collect-location")
|
||||
async def collect_compute_center_location(
|
||||
source_id: str,
|
||||
@@ -2089,6 +2111,9 @@ async def collect_compute_center_location(
|
||||
"""
|
||||
if not source_id or not source_id.strip():
|
||||
raise HTTPException(status_code=400, detail="source_id is required")
|
||||
capability = await _compute_center_location_web_search_capability(db)
|
||||
if not capability.get("enabled"):
|
||||
raise HTTPException(status_code=409, detail=capability)
|
||||
|
||||
record = await _load_compute_center_record(db, source_id)
|
||||
name = payload.name or (record.name if record else None)
|
||||
@@ -2696,6 +2721,8 @@ async def _build_bgp_collectors_geojson(db: AsyncSession) -> dict[str, Any]:
|
||||
db,
|
||||
source_filter=("ris_live_bgp", "bgpstream_bgp"),
|
||||
)
|
||||
if not any(int(item.get("observation_count") or 0) > 0 for item in coverage):
|
||||
return {"type": "FeatureCollection", "features": [], "count": 0}
|
||||
coverage_by_collector = {
|
||||
item["collector"]: item
|
||||
for item in coverage
|
||||
|
||||
@@ -150,7 +150,7 @@ class DataBroadcaster:
|
||||
"timestamp": to_iso8601_utc(datetime.now(UTC)),
|
||||
"payload": data,
|
||||
},
|
||||
channel="all",
|
||||
channel="datasource_tasks",
|
||||
)
|
||||
|
||||
def start(self):
|
||||
|
||||
@@ -150,7 +150,7 @@ DEFAULT_LOGIN_USERS = (
|
||||
{
|
||||
"username": "linkong",
|
||||
"email": "linkong@planet.local",
|
||||
"password": "12345678",
|
||||
"password": "LK12345678",
|
||||
"role": "super_admin",
|
||||
},
|
||||
)
|
||||
@@ -203,6 +203,7 @@ async def init_db():
|
||||
import app.models.vessel_enrichment # noqa: F401
|
||||
import app.models.datasource_mapping # noqa: F401
|
||||
import app.models.earth_news # noqa: F401
|
||||
import app.models.earth_interactable # noqa: F401
|
||||
|
||||
logger.warning_event(
|
||||
"Database pool settings active",
|
||||
@@ -258,6 +259,406 @@ async def init_db():
|
||||
"""
|
||||
)
|
||||
)
|
||||
await conn.execute(
|
||||
text(
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS earth_data_change_events (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
table_name VARCHAR(128) NOT NULL,
|
||||
operation VARCHAR(16) NOT NULL,
|
||||
source VARCHAR(128),
|
||||
entity_key VARCHAR(255),
|
||||
payload JSONB NOT NULL,
|
||||
occurred_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
consumed_at TIMESTAMPTZ
|
||||
)
|
||||
"""
|
||||
)
|
||||
)
|
||||
await conn.execute(
|
||||
text(
|
||||
"""
|
||||
CREATE INDEX IF NOT EXISTS idx_earth_data_change_events_unconsumed
|
||||
ON earth_data_change_events (consumed_at, id)
|
||||
WHERE consumed_at IS NULL
|
||||
"""
|
||||
)
|
||||
)
|
||||
await conn.execute(
|
||||
text(
|
||||
"""
|
||||
CREATE OR REPLACE FUNCTION planet_emit_earth_data_changed_statement(
|
||||
change_table TEXT,
|
||||
change_operation TEXT,
|
||||
change_source TEXT,
|
||||
source_record_count INTEGER,
|
||||
source_entity_keys TEXT[]
|
||||
)
|
||||
RETURNS VOID AS $$
|
||||
DECLARE
|
||||
change_event_id BIGINT;
|
||||
change_payload JSONB;
|
||||
BEGIN
|
||||
change_payload := jsonb_build_object(
|
||||
'event', 'earth.layer.changed',
|
||||
'table', change_table,
|
||||
'operation', change_operation,
|
||||
'source', change_source,
|
||||
'entity_key', NULL,
|
||||
'entity_keys', COALESCE(to_jsonb(source_entity_keys), '[]'::jsonb),
|
||||
'records_processed', COALESCE(source_record_count, 0),
|
||||
'occurred_at', NOW()
|
||||
);
|
||||
|
||||
INSERT INTO earth_data_change_events (
|
||||
table_name,
|
||||
operation,
|
||||
source,
|
||||
entity_key,
|
||||
payload,
|
||||
occurred_at
|
||||
) VALUES (
|
||||
change_table,
|
||||
change_operation,
|
||||
change_source,
|
||||
NULL,
|
||||
change_payload,
|
||||
NOW()
|
||||
)
|
||||
RETURNING id INTO change_event_id;
|
||||
|
||||
change_payload := change_payload || jsonb_build_object(
|
||||
'event_id', change_event_id
|
||||
);
|
||||
|
||||
UPDATE earth_data_change_events
|
||||
SET payload = change_payload
|
||||
WHERE id = change_event_id;
|
||||
|
||||
PERFORM pg_notify(
|
||||
'planet_earth_data_changes',
|
||||
change_payload::text
|
||||
);
|
||||
END;
|
||||
$$ LANGUAGE plpgsql;
|
||||
"""
|
||||
)
|
||||
)
|
||||
await conn.execute(
|
||||
text(
|
||||
"""
|
||||
CREATE OR REPLACE FUNCTION planet_emit_collected_data_changed_statement(
|
||||
change_operation TEXT,
|
||||
change_source TEXT,
|
||||
source_record_count INTEGER,
|
||||
source_entity_keys TEXT[]
|
||||
)
|
||||
RETURNS VOID AS $$
|
||||
BEGIN
|
||||
PERFORM planet_emit_earth_data_changed_statement(
|
||||
'collected_data',
|
||||
change_operation,
|
||||
change_source,
|
||||
source_record_count,
|
||||
source_entity_keys
|
||||
);
|
||||
END;
|
||||
$$ LANGUAGE plpgsql;
|
||||
"""
|
||||
)
|
||||
)
|
||||
await conn.execute(
|
||||
text(
|
||||
"""
|
||||
CREATE OR REPLACE FUNCTION planet_notify_earth_table_changed_statement()
|
||||
RETURNS trigger AS $$
|
||||
DECLARE
|
||||
change_source TEXT;
|
||||
source_record_count INTEGER;
|
||||
source_entity_keys TEXT[];
|
||||
BEGIN
|
||||
IF TG_OP = 'INSERT' THEN
|
||||
FOR change_source IN
|
||||
SELECT DISTINCT COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME)
|
||||
FROM (SELECT to_jsonb(t) AS row_data FROM new_rows AS t) changed_rows
|
||||
LOOP
|
||||
SELECT
|
||||
COUNT(*),
|
||||
ARRAY(
|
||||
SELECT DISTINCT COALESCE(
|
||||
NULLIF(row_data->>'entity_key', ''),
|
||||
NULLIF(row_data->>'source_id', ''),
|
||||
NULLIF(row_data->>'incident_key', ''),
|
||||
NULLIF(row_data->>'id', ''),
|
||||
NULLIF(row_data->>'mmsi', '')
|
||||
)
|
||||
FROM (SELECT to_jsonb(t) AS row_data FROM new_rows AS t) rows_for_keys
|
||||
WHERE COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) = change_source
|
||||
LIMIT 20
|
||||
)
|
||||
INTO source_record_count, source_entity_keys
|
||||
FROM (SELECT to_jsonb(t) AS row_data FROM new_rows AS t) rows_for_count
|
||||
WHERE COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) = change_source;
|
||||
|
||||
PERFORM planet_emit_earth_data_changed_statement(
|
||||
TG_TABLE_NAME,
|
||||
TG_OP,
|
||||
change_source,
|
||||
source_record_count,
|
||||
source_entity_keys
|
||||
);
|
||||
END LOOP;
|
||||
ELSIF TG_OP = 'DELETE' THEN
|
||||
FOR change_source IN
|
||||
SELECT DISTINCT COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME)
|
||||
FROM (SELECT to_jsonb(t) AS row_data FROM old_rows AS t) changed_rows
|
||||
LOOP
|
||||
SELECT
|
||||
COUNT(*),
|
||||
ARRAY(
|
||||
SELECT DISTINCT COALESCE(
|
||||
NULLIF(row_data->>'entity_key', ''),
|
||||
NULLIF(row_data->>'source_id', ''),
|
||||
NULLIF(row_data->>'incident_key', ''),
|
||||
NULLIF(row_data->>'id', ''),
|
||||
NULLIF(row_data->>'mmsi', '')
|
||||
)
|
||||
FROM (SELECT to_jsonb(t) AS row_data FROM old_rows AS t) rows_for_keys
|
||||
WHERE COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) = change_source
|
||||
LIMIT 20
|
||||
)
|
||||
INTO source_record_count, source_entity_keys
|
||||
FROM (SELECT to_jsonb(t) AS row_data FROM old_rows AS t) rows_for_count
|
||||
WHERE COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) = change_source;
|
||||
|
||||
PERFORM planet_emit_earth_data_changed_statement(
|
||||
TG_TABLE_NAME,
|
||||
TG_OP,
|
||||
change_source,
|
||||
source_record_count,
|
||||
source_entity_keys
|
||||
);
|
||||
END LOOP;
|
||||
ELSIF TG_OP = 'UPDATE' THEN
|
||||
FOR change_source IN
|
||||
SELECT DISTINCT COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME)
|
||||
FROM (
|
||||
SELECT to_jsonb(t) AS row_data FROM new_rows AS t
|
||||
UNION ALL
|
||||
SELECT to_jsonb(t) AS row_data FROM old_rows AS t
|
||||
) changed_rows
|
||||
LOOP
|
||||
SELECT
|
||||
COUNT(*),
|
||||
ARRAY(
|
||||
SELECT DISTINCT COALESCE(
|
||||
NULLIF(row_data->>'entity_key', ''),
|
||||
NULLIF(row_data->>'source_id', ''),
|
||||
NULLIF(row_data->>'incident_key', ''),
|
||||
NULLIF(row_data->>'id', ''),
|
||||
NULLIF(row_data->>'mmsi', '')
|
||||
)
|
||||
FROM (
|
||||
SELECT to_jsonb(t) AS row_data FROM new_rows AS t
|
||||
UNION ALL
|
||||
SELECT to_jsonb(t) AS row_data FROM old_rows AS t
|
||||
) rows_for_keys
|
||||
WHERE COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) = change_source
|
||||
LIMIT 20
|
||||
)
|
||||
INTO source_record_count, source_entity_keys
|
||||
FROM (
|
||||
SELECT to_jsonb(t) AS row_data FROM new_rows AS t
|
||||
UNION ALL
|
||||
SELECT to_jsonb(t) AS row_data FROM old_rows AS t
|
||||
) rows_for_count
|
||||
WHERE COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) = change_source;
|
||||
|
||||
PERFORM planet_emit_earth_data_changed_statement(
|
||||
TG_TABLE_NAME,
|
||||
TG_OP,
|
||||
change_source,
|
||||
source_record_count,
|
||||
source_entity_keys
|
||||
);
|
||||
END LOOP;
|
||||
END IF;
|
||||
|
||||
RETURN NULL;
|
||||
END;
|
||||
$$ LANGUAGE plpgsql;
|
||||
"""
|
||||
)
|
||||
)
|
||||
await conn.execute(
|
||||
text(
|
||||
"""
|
||||
CREATE OR REPLACE FUNCTION planet_notify_collected_data_changed_statement()
|
||||
RETURNS trigger AS $$
|
||||
DECLARE
|
||||
change_source TEXT;
|
||||
source_record_count INTEGER;
|
||||
source_entity_keys TEXT[];
|
||||
BEGIN
|
||||
IF TG_OP = 'INSERT' THEN
|
||||
FOR change_source IN
|
||||
SELECT DISTINCT source FROM new_rows WHERE source IS NOT NULL
|
||||
LOOP
|
||||
SELECT
|
||||
COUNT(*),
|
||||
ARRAY(
|
||||
SELECT DISTINCT COALESCE(entity_key, source_id, id::text)
|
||||
FROM new_rows
|
||||
WHERE source = change_source
|
||||
LIMIT 20
|
||||
)
|
||||
INTO source_record_count, source_entity_keys
|
||||
FROM new_rows
|
||||
WHERE source = change_source;
|
||||
|
||||
PERFORM planet_emit_collected_data_changed_statement(
|
||||
TG_OP,
|
||||
change_source,
|
||||
source_record_count,
|
||||
source_entity_keys
|
||||
);
|
||||
END LOOP;
|
||||
ELSIF TG_OP = 'DELETE' THEN
|
||||
FOR change_source IN
|
||||
SELECT DISTINCT source FROM old_rows WHERE source IS NOT NULL
|
||||
LOOP
|
||||
SELECT
|
||||
COUNT(*),
|
||||
ARRAY(
|
||||
SELECT DISTINCT COALESCE(entity_key, source_id, id::text)
|
||||
FROM old_rows
|
||||
WHERE source = change_source
|
||||
LIMIT 20
|
||||
)
|
||||
INTO source_record_count, source_entity_keys
|
||||
FROM old_rows
|
||||
WHERE source = change_source;
|
||||
|
||||
PERFORM planet_emit_collected_data_changed_statement(
|
||||
TG_OP,
|
||||
change_source,
|
||||
source_record_count,
|
||||
source_entity_keys
|
||||
);
|
||||
END LOOP;
|
||||
ELSIF TG_OP = 'UPDATE' THEN
|
||||
FOR change_source IN
|
||||
SELECT DISTINCT source FROM (
|
||||
SELECT source FROM new_rows
|
||||
UNION
|
||||
SELECT source FROM old_rows
|
||||
) changed_sources
|
||||
WHERE source IS NOT NULL
|
||||
LOOP
|
||||
SELECT
|
||||
COUNT(*),
|
||||
ARRAY(
|
||||
SELECT DISTINCT COALESCE(entity_key, source_id, id::text)
|
||||
FROM (
|
||||
SELECT id, source_id, entity_key, source FROM new_rows
|
||||
UNION ALL
|
||||
SELECT id, source_id, entity_key, source FROM old_rows
|
||||
) changed_rows
|
||||
WHERE source = change_source
|
||||
LIMIT 20
|
||||
)
|
||||
INTO source_record_count, source_entity_keys
|
||||
FROM (
|
||||
SELECT id, source_id, entity_key, source FROM new_rows
|
||||
UNION ALL
|
||||
SELECT id, source_id, entity_key, source FROM old_rows
|
||||
) changed_rows
|
||||
WHERE source = change_source;
|
||||
|
||||
PERFORM planet_emit_collected_data_changed_statement(
|
||||
TG_OP,
|
||||
change_source,
|
||||
source_record_count,
|
||||
source_entity_keys
|
||||
);
|
||||
END LOOP;
|
||||
END IF;
|
||||
|
||||
RETURN NULL;
|
||||
END;
|
||||
$$ LANGUAGE plpgsql;
|
||||
"""
|
||||
)
|
||||
)
|
||||
for statement in (
|
||||
"DROP TRIGGER IF EXISTS tr_planet_collected_data_changed ON collected_data",
|
||||
"DROP TRIGGER IF EXISTS tr_planet_collected_data_changed_insert ON collected_data",
|
||||
"DROP TRIGGER IF EXISTS tr_planet_collected_data_changed_update ON collected_data",
|
||||
"DROP TRIGGER IF EXISTS tr_planet_collected_data_changed_delete ON collected_data",
|
||||
"DROP FUNCTION IF EXISTS planet_notify_collected_data_changed()",
|
||||
"""
|
||||
CREATE TRIGGER tr_planet_collected_data_changed_insert
|
||||
AFTER INSERT ON collected_data
|
||||
REFERENCING NEW TABLE AS new_rows
|
||||
FOR EACH STATEMENT
|
||||
EXECUTE FUNCTION planet_notify_collected_data_changed_statement()
|
||||
""",
|
||||
"""
|
||||
CREATE TRIGGER tr_planet_collected_data_changed_update
|
||||
AFTER UPDATE ON collected_data
|
||||
REFERENCING OLD TABLE AS old_rows NEW TABLE AS new_rows
|
||||
FOR EACH STATEMENT
|
||||
EXECUTE FUNCTION planet_notify_collected_data_changed_statement()
|
||||
""",
|
||||
"""
|
||||
CREATE TRIGGER tr_planet_collected_data_changed_delete
|
||||
AFTER DELETE ON collected_data
|
||||
REFERENCING OLD TABLE AS old_rows
|
||||
FOR EACH STATEMENT
|
||||
EXECUTE FUNCTION planet_notify_collected_data_changed_statement()
|
||||
""",
|
||||
):
|
||||
await conn.execute(text(statement))
|
||||
for table_name in (
|
||||
"bgp_observations",
|
||||
"bgp_anomalies",
|
||||
"bgp_incidents",
|
||||
"bgp_collector_locations",
|
||||
"vessel_static",
|
||||
"vessel_position",
|
||||
"ais_raw_observations",
|
||||
"ais_source_health",
|
||||
"compute_center_locations",
|
||||
"earth_interactables",
|
||||
"earth_news_items",
|
||||
):
|
||||
for statement in (
|
||||
f"DROP TRIGGER IF EXISTS tr_planet_{table_name}_changed_insert ON {table_name}",
|
||||
f"DROP TRIGGER IF EXISTS tr_planet_{table_name}_changed_update ON {table_name}",
|
||||
f"DROP TRIGGER IF EXISTS tr_planet_{table_name}_changed_delete ON {table_name}",
|
||||
f"""
|
||||
CREATE TRIGGER tr_planet_{table_name}_changed_insert
|
||||
AFTER INSERT ON {table_name}
|
||||
REFERENCING NEW TABLE AS new_rows
|
||||
FOR EACH STATEMENT
|
||||
EXECUTE FUNCTION planet_notify_earth_table_changed_statement()
|
||||
""",
|
||||
f"""
|
||||
CREATE TRIGGER tr_planet_{table_name}_changed_update
|
||||
AFTER UPDATE ON {table_name}
|
||||
REFERENCING OLD TABLE AS old_rows NEW TABLE AS new_rows
|
||||
FOR EACH STATEMENT
|
||||
EXECUTE FUNCTION planet_notify_earth_table_changed_statement()
|
||||
""",
|
||||
f"""
|
||||
CREATE TRIGGER tr_planet_{table_name}_changed_delete
|
||||
AFTER DELETE ON {table_name}
|
||||
REFERENCING OLD TABLE AS old_rows
|
||||
FOR EACH STATEMENT
|
||||
EXECUTE FUNCTION planet_notify_earth_table_changed_statement()
|
||||
""",
|
||||
):
|
||||
await conn.execute(text(statement))
|
||||
await conn.execute(
|
||||
text(
|
||||
"""
|
||||
@@ -267,7 +668,16 @@ async def init_db():
|
||||
ADD COLUMN IF NOT EXISTS phase_message VARCHAR(255),
|
||||
ADD COLUMN IF NOT EXISTS phase_current BIGINT,
|
||||
ADD COLUMN IF NOT EXISTS phase_total BIGINT,
|
||||
ADD COLUMN IF NOT EXISTS phase_unit VARCHAR(30)
|
||||
ADD COLUMN IF NOT EXISTS phase_unit VARCHAR(30),
|
||||
ADD COLUMN IF NOT EXISTS source VARCHAR(100),
|
||||
ADD COLUMN IF NOT EXISTS task_type VARCHAR(30) NOT NULL DEFAULT 'collect',
|
||||
ADD COLUMN IF NOT EXISTS payload JSONB NOT NULL DEFAULT '{}'::jsonb,
|
||||
ADD COLUMN IF NOT EXISTS rollback_policy VARCHAR(40) NOT NULL DEFAULT 'keep_committed_batches',
|
||||
ADD COLUMN IF NOT EXISTS dedupe_key VARCHAR(180),
|
||||
ADD COLUMN IF NOT EXISTS worker_id VARCHAR(120),
|
||||
ADD COLUMN IF NOT EXISTS locked_at TIMESTAMPTZ,
|
||||
ADD COLUMN IF NOT EXISTS requested_cancel_at TIMESTAMPTZ,
|
||||
ADD COLUMN IF NOT EXISTS cancel_reason TEXT
|
||||
"""
|
||||
)
|
||||
)
|
||||
@@ -283,6 +693,17 @@ async def init_db():
|
||||
"""
|
||||
)
|
||||
)
|
||||
await conn.execute(
|
||||
text(
|
||||
"""
|
||||
ALTER TABLE earth_interactables
|
||||
ADD COLUMN IF NOT EXISTS altitude DOUBLE PRECISION,
|
||||
ADD COLUMN IF NOT EXISTS revision INTEGER NOT NULL DEFAULT 1,
|
||||
ADD COLUMN IF NOT EXISTS is_deleted BOOLEAN NOT NULL DEFAULT FALSE,
|
||||
ADD COLUMN IF NOT EXISTS deleted_at TIMESTAMPTZ
|
||||
"""
|
||||
)
|
||||
)
|
||||
await conn.execute(
|
||||
text(
|
||||
"""
|
||||
@@ -307,6 +728,48 @@ async def init_db():
|
||||
"""
|
||||
)
|
||||
)
|
||||
await conn.execute(
|
||||
text(
|
||||
"""
|
||||
CREATE INDEX IF NOT EXISTS idx_earth_interactables_layer_deleted
|
||||
ON earth_interactables (layer, is_deleted)
|
||||
"""
|
||||
)
|
||||
)
|
||||
await conn.execute(
|
||||
text(
|
||||
"""
|
||||
CREATE INDEX IF NOT EXISTS idx_earth_interactables_updated_at
|
||||
ON earth_interactables (updated_at)
|
||||
"""
|
||||
)
|
||||
)
|
||||
await conn.execute(
|
||||
text(
|
||||
"""
|
||||
CREATE INDEX IF NOT EXISTS idx_collection_tasks_source_status
|
||||
ON collection_tasks (source, status)
|
||||
"""
|
||||
)
|
||||
)
|
||||
await conn.execute(
|
||||
text(
|
||||
"""
|
||||
CREATE INDEX IF NOT EXISTS idx_collection_tasks_queue
|
||||
ON collection_tasks (status, created_at, id)
|
||||
WHERE status = 'queued'
|
||||
"""
|
||||
)
|
||||
)
|
||||
await conn.execute(
|
||||
text(
|
||||
"""
|
||||
CREATE INDEX IF NOT EXISTS idx_collection_tasks_dedupe
|
||||
ON collection_tasks (dedupe_key)
|
||||
WHERE dedupe_key IS NOT NULL
|
||||
"""
|
||||
)
|
||||
)
|
||||
await conn.execute(
|
||||
text(
|
||||
"""
|
||||
|
||||
@@ -24,6 +24,11 @@ from app.services.earth_news_worker import (
|
||||
start_earth_news_target_worker,
|
||||
stop_earth_news_target_worker,
|
||||
)
|
||||
from app.services.earth_db_change_listener import (
|
||||
start_earth_db_change_listener,
|
||||
stop_earth_db_change_listener,
|
||||
)
|
||||
from app.services.data_jobs import start_data_job_worker, stop_data_job_worker
|
||||
|
||||
|
||||
configure_logging()
|
||||
@@ -59,9 +64,13 @@ async def lifespan(app: FastAPI):
|
||||
start_scheduler()
|
||||
await sync_scheduler_with_datasources()
|
||||
broadcaster.start()
|
||||
start_data_job_worker()
|
||||
start_earth_db_change_listener()
|
||||
start_earth_news_target_worker()
|
||||
yield
|
||||
await stop_earth_news_target_worker()
|
||||
await stop_earth_db_change_listener()
|
||||
await stop_data_job_worker()
|
||||
broadcaster.stop()
|
||||
stop_scheduler()
|
||||
|
||||
|
||||
@@ -17,6 +17,7 @@ from app.models.system_log import SystemLog, AuditLog
|
||||
from app.models.vessel import AISConflictRecord, AISRawObservation, AISSourceHealth, VesselPosition, VesselStatic
|
||||
from app.models.datasource_mapping import DataSourceMappingTemplate
|
||||
from app.models.earth_news import EarthNewsItem
|
||||
from app.models.earth_interactable import EarthInteractable
|
||||
|
||||
__all__ = [
|
||||
"User",
|
||||
@@ -45,4 +46,5 @@ __all__ = [
|
||||
"AISSourceHealth",
|
||||
"DataSourceMappingTemplate",
|
||||
"EarthNewsItem",
|
||||
"EarthInteractable",
|
||||
]
|
||||
|
||||
30
backend/app/models/earth_interactable.py
Normal file
30
backend/app/models/earth_interactable.py
Normal file
@@ -0,0 +1,30 @@
|
||||
"""Persistent Earth interactable objects."""
|
||||
|
||||
from sqlalchemy import Boolean, Column, DateTime, Float, Index, Integer, JSON, String, Text
|
||||
from sqlalchemy.sql import func
|
||||
|
||||
from app.db.session import Base
|
||||
|
||||
|
||||
class EarthInteractable(Base):
|
||||
__tablename__ = "earth_interactables"
|
||||
|
||||
id = Column(String(160), primary_key=True)
|
||||
layer = Column(String(80), nullable=False, default="interactables", index=True)
|
||||
kind = Column(String(80), nullable=False, default="default", index=True)
|
||||
label = Column(String(255), nullable=False, default="")
|
||||
description = Column(Text, nullable=False, default="")
|
||||
latitude = Column(Float, nullable=False)
|
||||
longitude = Column(Float, nullable=False)
|
||||
altitude = Column(Float, nullable=True)
|
||||
revision = Column(Integer, nullable=False, default=1)
|
||||
properties = Column(JSON, nullable=False, default=dict)
|
||||
is_deleted = Column(Boolean, nullable=False, default=False, index=True)
|
||||
created_at = Column(DateTime(timezone=True), server_default=func.now(), nullable=False)
|
||||
updated_at = Column(DateTime(timezone=True), server_default=func.now(), onupdate=func.now())
|
||||
deleted_at = Column(DateTime(timezone=True), nullable=True, index=True)
|
||||
|
||||
__table_args__ = (
|
||||
Index("idx_earth_interactables_layer_deleted", "layer", "is_deleted"),
|
||||
Index("idx_earth_interactables_updated_at", "updated_at"),
|
||||
)
|
||||
@@ -1,6 +1,6 @@
|
||||
"""Collection Task model"""
|
||||
"""Datasource job model."""
|
||||
|
||||
from sqlalchemy import BigInteger, Column, DateTime, Integer, String, Text, Float
|
||||
from sqlalchemy import BigInteger, Column, DateTime, Float, Integer, JSON, String, Text
|
||||
from sqlalchemy.sql import func
|
||||
|
||||
from app.db.session import Base
|
||||
@@ -11,7 +11,9 @@ class CollectionTask(Base):
|
||||
|
||||
id = Column(Integer, primary_key=True, autoincrement=True)
|
||||
datasource_id = Column(Integer, nullable=False, index=True)
|
||||
status = Column(String(20), nullable=False) # pending, running, success, failed, cancelled
|
||||
source = Column(String(100), nullable=True, index=True)
|
||||
task_type = Column(String(30), nullable=False, default="collect", index=True)
|
||||
status = Column(String(20), nullable=False) # queued, running, cancelling, success, failed, cancelled
|
||||
phase = Column(String(30), default="queued")
|
||||
phase_progress = Column(Float)
|
||||
phase_message = Column(String(255))
|
||||
@@ -24,6 +26,13 @@ class CollectionTask(Base):
|
||||
total_records = Column(Integer, default=0) # Total records to process
|
||||
progress = Column(Float, default=0.0) # Progress percentage (0-100)
|
||||
error_message = Column(Text)
|
||||
payload = Column(JSON, default=dict)
|
||||
rollback_policy = Column(String(40), nullable=False, default="keep_committed_batches")
|
||||
dedupe_key = Column(String(180), nullable=True, index=True)
|
||||
worker_id = Column(String(120), nullable=True, index=True)
|
||||
locked_at = Column(DateTime(timezone=True), nullable=True, index=True)
|
||||
requested_cancel_at = Column(DateTime(timezone=True), nullable=True)
|
||||
cancel_reason = Column(Text)
|
||||
created_at = Column(DateTime(timezone=True), server_default=func.now())
|
||||
|
||||
def __repr__(self):
|
||||
|
||||
@@ -2,18 +2,24 @@ from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from time import perf_counter
|
||||
|
||||
import httpx
|
||||
from fastapi import Depends, HTTPException, status
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.config import settings
|
||||
from app.core.logging import get_logger
|
||||
from app.db.session import get_db
|
||||
from app.schemas.ai import (
|
||||
AIProviderStatusResponse,
|
||||
SituationalAnalysisRequest,
|
||||
SituationalAnalysisResponse,
|
||||
)
|
||||
from app.services.business_logs import emit_business_log, exception_context
|
||||
|
||||
|
||||
logger = get_logger(__name__, service="ai")
|
||||
|
||||
|
||||
class AIProviderClient:
|
||||
@@ -64,7 +70,19 @@ class AIProviderClient:
|
||||
return headers
|
||||
|
||||
async def get_status(self, request_id: str | None = None) -> AIProviderStatusResponse:
|
||||
context = self._base_log_context(operation="status")
|
||||
if not self.service_url:
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="ai.provider.status.failed",
|
||||
message="AI provider status skipped because service URL is not configured",
|
||||
category="ai",
|
||||
level="warning",
|
||||
service="ai",
|
||||
module=__name__,
|
||||
request_id=request_id,
|
||||
context={**context, "status": "unconfigured"},
|
||||
)
|
||||
return AIProviderStatusResponse(
|
||||
provider="unconfigured",
|
||||
enabled=False,
|
||||
@@ -73,27 +91,133 @@ class AIProviderClient:
|
||||
base_url=None,
|
||||
)
|
||||
|
||||
data = await self._request("GET", "/v1/provider/status", request_id=request_id)
|
||||
return AIProviderStatusResponse.model_validate(data)
|
||||
started_at = perf_counter()
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="ai.provider.status.start",
|
||||
message="AI provider status request started",
|
||||
category="ai",
|
||||
service="ai",
|
||||
module=__name__,
|
||||
request_id=request_id,
|
||||
context=context,
|
||||
)
|
||||
try:
|
||||
data = await self._request("GET", "/v1/provider/status", request_id=request_id, operation="status")
|
||||
result = AIProviderStatusResponse.model_validate(data)
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="ai.provider.status.success",
|
||||
message="AI provider status request completed",
|
||||
category="ai",
|
||||
service="ai",
|
||||
module=__name__,
|
||||
request_id=request_id,
|
||||
context={
|
||||
**context,
|
||||
"status": "success",
|
||||
"duration_ms": self._duration_ms(started_at),
|
||||
"result_provider": result.provider,
|
||||
"result_model": result.model,
|
||||
"configured": result.configured,
|
||||
"enabled": result.enabled,
|
||||
},
|
||||
)
|
||||
return result
|
||||
except Exception as exc:
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="ai.provider.status.failed",
|
||||
message="AI provider status request failed",
|
||||
category="ai",
|
||||
level="error",
|
||||
service="ai",
|
||||
module=__name__,
|
||||
request_id=request_id,
|
||||
context=exception_context(exc, {**context, "status": "failed", "duration_ms": self._duration_ms(started_at)}),
|
||||
)
|
||||
raise
|
||||
|
||||
async def analyze(
|
||||
self,
|
||||
payload: SituationalAnalysisRequest,
|
||||
request_id: str | None = None,
|
||||
) -> SituationalAnalysisResponse:
|
||||
context = self._base_log_context(
|
||||
operation="analyze",
|
||||
preferred_model=payload.preferred_model,
|
||||
input_summary=self._summarize_analysis_payload(payload),
|
||||
)
|
||||
if not self.service_url:
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="ai.provider.analyze.failed",
|
||||
message="AI provider analyze skipped because service URL is not configured",
|
||||
category="ai",
|
||||
level="warning",
|
||||
service="ai",
|
||||
module=__name__,
|
||||
request_id=request_id,
|
||||
context={**context, "status": "unconfigured"},
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
|
||||
detail="AI provider service URL is not configured.",
|
||||
)
|
||||
|
||||
data = await self._request(
|
||||
"POST",
|
||||
"/v1/analyze",
|
||||
json=payload.model_dump(),
|
||||
started_at = perf_counter()
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="ai.provider.analyze.start",
|
||||
message="AI provider analyze request started",
|
||||
category="ai",
|
||||
service="ai",
|
||||
module=__name__,
|
||||
request_id=request_id,
|
||||
context=context,
|
||||
)
|
||||
return SituationalAnalysisResponse.model_validate(data)
|
||||
try:
|
||||
data = await self._request(
|
||||
"POST",
|
||||
"/v1/analyze",
|
||||
json=payload.model_dump(),
|
||||
request_id=request_id,
|
||||
operation="analyze",
|
||||
payload_summary=context["input_summary"],
|
||||
)
|
||||
result = SituationalAnalysisResponse.model_validate(data)
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="ai.provider.analyze.success",
|
||||
message="AI provider analyze request completed",
|
||||
category="ai",
|
||||
service="ai",
|
||||
module=__name__,
|
||||
request_id=request_id,
|
||||
context={
|
||||
**context,
|
||||
"status": "success",
|
||||
"duration_ms": self._duration_ms(started_at),
|
||||
"result_provider": result.provider,
|
||||
"result_model": result.model,
|
||||
"content_block_count": len(result.content_blocks or []),
|
||||
"thinking_block_count": len(result.thinking_blocks or []),
|
||||
},
|
||||
)
|
||||
return result
|
||||
except Exception as exc:
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="ai.provider.analyze.failed",
|
||||
message="AI provider analyze request failed",
|
||||
category="ai",
|
||||
level="error",
|
||||
service="ai",
|
||||
module=__name__,
|
||||
request_id=request_id,
|
||||
context=exception_context(exc, {**context, "status": "failed", "duration_ms": self._duration_ms(started_at)}),
|
||||
)
|
||||
raise
|
||||
|
||||
async def _request(
|
||||
self,
|
||||
@@ -101,9 +225,12 @@ class AIProviderClient:
|
||||
path: str,
|
||||
json: dict | None = None,
|
||||
request_id: str | None = None,
|
||||
operation: str = "request",
|
||||
payload_summary: dict | None = None,
|
||||
) -> dict:
|
||||
last_error: Exception | None = None
|
||||
for attempt in range(1, self.retry_attempts + 1):
|
||||
attempt_started_at = perf_counter()
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=self.timeout) as client:
|
||||
response = await client.request(
|
||||
@@ -117,6 +244,15 @@ class AIProviderClient:
|
||||
except httpx.HTTPStatusError as exc:
|
||||
last_error = exc
|
||||
if attempt < self.retry_attempts and exc.response.status_code >= 500:
|
||||
await self._log_retry(
|
||||
operation=operation,
|
||||
request_id=request_id,
|
||||
attempt=attempt,
|
||||
status_code=exc.response.status_code,
|
||||
duration_ms=self._duration_ms(attempt_started_at),
|
||||
error=exc,
|
||||
payload_summary=payload_summary,
|
||||
)
|
||||
await asyncio.sleep(0.3 * attempt)
|
||||
continue
|
||||
detail = exc.response.text or "AI provider service returned an error"
|
||||
@@ -127,6 +263,14 @@ class AIProviderClient:
|
||||
except httpx.HTTPError as exc:
|
||||
last_error = exc
|
||||
if attempt < self.retry_attempts:
|
||||
await self._log_retry(
|
||||
operation=operation,
|
||||
request_id=request_id,
|
||||
attempt=attempt,
|
||||
duration_ms=self._duration_ms(attempt_started_at),
|
||||
error=exc,
|
||||
payload_summary=payload_summary,
|
||||
)
|
||||
await asyncio.sleep(0.3 * attempt)
|
||||
continue
|
||||
raise HTTPException(
|
||||
@@ -139,6 +283,71 @@ class AIProviderClient:
|
||||
detail=f"AI provider service request failed: {last_error}",
|
||||
)
|
||||
|
||||
def _base_log_context(self, **extra: object) -> dict:
|
||||
llm_provider_apis = self.llm_config.get("model_provider_apis")
|
||||
return {
|
||||
"provider": self.llm_config.get("provider") or "",
|
||||
"provider_api": self.llm_config.get("provider_api") or "",
|
||||
"model": self.llm_config.get("model") or "",
|
||||
"base_url_configured": bool(self.llm_config.get("base_url")),
|
||||
"service_url_configured": bool(self.service_url),
|
||||
"timeout_seconds": self.timeout,
|
||||
"retry_attempts": self.retry_attempts,
|
||||
"model_provider_api_count": len(llm_provider_apis or {}) if isinstance(llm_provider_apis, dict) else 0,
|
||||
**extra,
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def _duration_ms(started_at: float) -> int:
|
||||
return int((perf_counter() - started_at) * 1000)
|
||||
|
||||
@staticmethod
|
||||
def _summarize_analysis_payload(payload: SituationalAnalysisRequest) -> dict:
|
||||
context = payload.context if isinstance(payload.context, dict) else {}
|
||||
thinking = payload.thinking if isinstance(payload.thinking, dict) else payload.thinking
|
||||
return {
|
||||
"title_length": len(payload.title or ""),
|
||||
"objective_length": len(payload.objective or ""),
|
||||
"observation_count": len(payload.observations or []),
|
||||
"constraint_count": len(payload.constraints or []),
|
||||
"has_system_prompt": bool(payload.system_prompt),
|
||||
"thinking_enabled": bool(thinking),
|
||||
"context_keys": sorted(str(key) for key in context.keys()),
|
||||
}
|
||||
|
||||
async def _log_retry(
|
||||
self,
|
||||
*,
|
||||
operation: str,
|
||||
request_id: str | None,
|
||||
attempt: int,
|
||||
duration_ms: int,
|
||||
error: BaseException,
|
||||
status_code: int | None = None,
|
||||
payload_summary: dict | None = None,
|
||||
) -> None:
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event=f"ai.provider.{operation}.retry",
|
||||
message="AI provider request will retry",
|
||||
category="ai",
|
||||
level="warning",
|
||||
service="ai",
|
||||
module=__name__,
|
||||
request_id=request_id,
|
||||
context=exception_context(
|
||||
error,
|
||||
{
|
||||
**self._base_log_context(operation=operation),
|
||||
"attempt": attempt,
|
||||
"next_attempt": attempt + 1,
|
||||
"status_code": status_code,
|
||||
"duration_ms": duration_ms,
|
||||
"input_summary": payload_summary,
|
||||
},
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
async def get_ai_provider_client(db: AsyncSession = Depends(get_db)) -> AIProviderClient:
|
||||
from app.api.v1.settings import get_runtime_ai_provider_config
|
||||
|
||||
@@ -1,11 +1,18 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
from time import perf_counter
|
||||
from urllib.parse import urlparse
|
||||
|
||||
import httpx
|
||||
from bs4 import BeautifulSoup
|
||||
|
||||
from app.core.logging import get_logger
|
||||
from app.services.ai_tools.schemas import FetchedEvidence
|
||||
from app.services.business_logs import emit_business_log, exception_context
|
||||
|
||||
|
||||
logger = get_logger(__name__, service="ai_tool")
|
||||
|
||||
|
||||
class WebFetchError(RuntimeError):
|
||||
@@ -29,8 +36,33 @@ async def fetch_url_evidence(
|
||||
timeout_seconds: int = 20,
|
||||
max_bytes: int = 1_500_000,
|
||||
) -> FetchedEvidence:
|
||||
started_at = perf_counter()
|
||||
if not url:
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="ai_tool.web_fetch.failed",
|
||||
message="WebFetch failed because URL is empty",
|
||||
category="ai_tool",
|
||||
level="warning",
|
||||
service="ai_tool",
|
||||
module=__name__,
|
||||
context={"reason": "empty_url"},
|
||||
)
|
||||
raise WebFetchError("url is required")
|
||||
request_host = urlparse(url).netloc
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="ai_tool.web_fetch.start",
|
||||
message="WebFetch request started",
|
||||
category="ai_tool",
|
||||
service="ai_tool",
|
||||
module=__name__,
|
||||
context={
|
||||
"url_host": request_host,
|
||||
"timeout_seconds": timeout_seconds,
|
||||
"max_bytes": max_bytes,
|
||||
},
|
||||
)
|
||||
try:
|
||||
async with httpx.AsyncClient(
|
||||
timeout=timeout_seconds,
|
||||
@@ -41,10 +73,45 @@ async def fetch_url_evidence(
|
||||
response.raise_for_status()
|
||||
content = response.content[:max_bytes]
|
||||
except httpx.HTTPError as exc:
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="ai_tool.web_fetch.failed",
|
||||
message="WebFetch request failed",
|
||||
category="ai_tool",
|
||||
level="error",
|
||||
service="ai_tool",
|
||||
module=__name__,
|
||||
context=exception_context(
|
||||
exc,
|
||||
{
|
||||
"url_host": request_host,
|
||||
"status": "failed",
|
||||
"duration_ms": int((perf_counter() - started_at) * 1000),
|
||||
},
|
||||
),
|
||||
)
|
||||
raise WebFetchError(f"failed to fetch page: {exc}") from exc
|
||||
|
||||
title, text = _extract_title_and_text(content.decode(response.encoding or "utf-8", errors="ignore"))
|
||||
content_hash = hashlib.sha256(text.encode("utf-8")).hexdigest()
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="ai_tool.web_fetch.success",
|
||||
message="WebFetch request completed",
|
||||
category="ai_tool",
|
||||
service="ai_tool",
|
||||
module=__name__,
|
||||
context={
|
||||
"url_host": request_host,
|
||||
"final_url_host": urlparse(str(response.url)).netloc,
|
||||
"status": "success",
|
||||
"status_code": response.status_code,
|
||||
"bytes_read": len(content),
|
||||
"content_hash": content_hash,
|
||||
"duration_ms": int((perf_counter() - started_at) * 1000),
|
||||
"extractor": "beautifulsoup_basic",
|
||||
},
|
||||
)
|
||||
return FetchedEvidence(
|
||||
url=url,
|
||||
final_url=str(response.url),
|
||||
@@ -53,4 +120,3 @@ async def fetch_url_evidence(
|
||||
content_hash=content_hash,
|
||||
extractor="beautifulsoup_basic",
|
||||
)
|
||||
|
||||
|
||||
@@ -1,11 +1,18 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from copy import deepcopy
|
||||
import hashlib
|
||||
from time import perf_counter
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
|
||||
from app.core.logging import get_logger
|
||||
from app.services.ai_tools.schemas import SearchEvidence, WebSearchConfig, WebSearchProviderConfig
|
||||
from app.services.business_logs import emit_business_log, exception_context
|
||||
|
||||
|
||||
logger = get_logger(__name__, service="ai_tool")
|
||||
|
||||
|
||||
WEB_SEARCH_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
|
||||
@@ -120,29 +127,108 @@ class WebSearchClient:
|
||||
domains: list[str] | None = None,
|
||||
freshness_days: int | None = None,
|
||||
) -> list[SearchEvidence]:
|
||||
started_at = perf_counter()
|
||||
if not self.config.enabled:
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="ai_tool.web_search.unavailable",
|
||||
message="WebSearch skipped because integration is disabled",
|
||||
category="ai_tool",
|
||||
level="warning",
|
||||
service="ai_tool",
|
||||
module=__name__,
|
||||
context={"provider": self.config.default_provider, "reason": "disabled"},
|
||||
)
|
||||
raise WebSearchConfigurationError("WebSearch is disabled.")
|
||||
provider_config = self.config.active_provider_config
|
||||
provider = normalize_web_search_provider(provider_config.provider)
|
||||
if provider != "searxng" and not provider_config.api_key:
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="ai_tool.web_search.unavailable",
|
||||
message="WebSearch skipped because API key is not configured",
|
||||
category="ai_tool",
|
||||
level="warning",
|
||||
service="ai_tool",
|
||||
module=__name__,
|
||||
context={"provider": provider, "reason": "missing_api_key"},
|
||||
)
|
||||
raise WebSearchConfigurationError(f"{provider} API key is not configured.")
|
||||
query = " ".join(str(query or "").split())
|
||||
if not query:
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="ai_tool.web_search.failed",
|
||||
message="WebSearch failed because query is empty",
|
||||
category="ai_tool",
|
||||
level="warning",
|
||||
service="ai_tool",
|
||||
module=__name__,
|
||||
context={"provider": provider, "reason": "empty_query"},
|
||||
)
|
||||
raise WebSearchConfigurationError("search query is required.")
|
||||
limit = max_results or provider_config.max_results
|
||||
if provider == "tavily":
|
||||
return await self._search_tavily(provider_config, query, limit, domains, freshness_days)
|
||||
if provider == "brave":
|
||||
return await self._search_brave(provider_config, query, limit, domains)
|
||||
if provider == "serpapi":
|
||||
return await self._search_serpapi(provider_config, query, limit)
|
||||
if provider == "exa":
|
||||
return await self._search_exa(provider_config, query, limit, domains)
|
||||
if provider == "firecrawl":
|
||||
return await self._search_firecrawl(provider_config, query, limit)
|
||||
if provider == "searxng":
|
||||
return await self._search_searxng(provider_config, query, limit, domains)
|
||||
raise WebSearchConfigurationError(f"Unsupported web search provider: {provider}")
|
||||
context = {
|
||||
"provider": provider,
|
||||
"query_hash": hashlib.sha256(query.encode("utf-8")).hexdigest(),
|
||||
"query_length": len(query),
|
||||
"max_results": limit,
|
||||
"domain_count": len(domains or []),
|
||||
"freshness_days": freshness_days,
|
||||
}
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="ai_tool.web_search.start",
|
||||
message="WebSearch request started",
|
||||
category="ai_tool",
|
||||
service="ai_tool",
|
||||
module=__name__,
|
||||
context=context,
|
||||
)
|
||||
try:
|
||||
if provider == "tavily":
|
||||
results = await self._search_tavily(provider_config, query, limit, domains, freshness_days)
|
||||
elif provider == "brave":
|
||||
results = await self._search_brave(provider_config, query, limit, domains)
|
||||
elif provider == "serpapi":
|
||||
results = await self._search_serpapi(provider_config, query, limit)
|
||||
elif provider == "exa":
|
||||
results = await self._search_exa(provider_config, query, limit, domains)
|
||||
elif provider == "firecrawl":
|
||||
results = await self._search_firecrawl(provider_config, query, limit)
|
||||
elif provider == "searxng":
|
||||
results = await self._search_searxng(provider_config, query, limit, domains)
|
||||
else:
|
||||
raise WebSearchConfigurationError(f"Unsupported web search provider: {provider}")
|
||||
event = "ai_tool.web_search.success" if results else "ai_tool.web_search.empty"
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event=event,
|
||||
message="WebSearch request completed" if results else "WebSearch returned no results",
|
||||
category="ai_tool",
|
||||
level="info" if results else "warning",
|
||||
service="ai_tool",
|
||||
module=__name__,
|
||||
context={
|
||||
**context,
|
||||
"status": "success" if results else "empty",
|
||||
"result_count": len(results),
|
||||
"duration_ms": int((perf_counter() - started_at) * 1000),
|
||||
},
|
||||
)
|
||||
return results
|
||||
except Exception as exc:
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="ai_tool.web_search.failed",
|
||||
message="WebSearch request failed",
|
||||
category="ai_tool",
|
||||
level="error",
|
||||
service="ai_tool",
|
||||
module=__name__,
|
||||
context=exception_context(exc, {**context, "status": "failed", "duration_ms": int((perf_counter() - started_at) * 1000)}),
|
||||
)
|
||||
raise
|
||||
|
||||
async def test_connection(self) -> list[SearchEvidence]:
|
||||
return await self.search("Planet WebSearch connectivity test", max_results=1)
|
||||
@@ -388,4 +474,3 @@ def _float_or_none(value: Any) -> float | None:
|
||||
return float(value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
117
backend/app/services/business_logs.py
Normal file
117
backend/app/services/business_logs.py
Normal file
@@ -0,0 +1,117 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from collections.abc import Mapping
|
||||
from typing import Any
|
||||
|
||||
from app.core.logging import PlanetLoggerAdapter, sanitize_log_value
|
||||
from app.core.request_context import get_request_id
|
||||
from app.services.persistent_logs import record_system_log
|
||||
|
||||
|
||||
LEVEL_METHODS = {
|
||||
"debug": "debug_event",
|
||||
"info": "info_event",
|
||||
"warning": "warning_event",
|
||||
"error": "error_event",
|
||||
}
|
||||
|
||||
|
||||
def normalize_business_level(level: str | None) -> str:
|
||||
normalized = str(level or "info").strip().lower()
|
||||
if normalized in {"warn", "warning"}:
|
||||
return "warning"
|
||||
if normalized in {"err", "error", "critical", "fatal"}:
|
||||
return "error"
|
||||
if normalized == "debug":
|
||||
return "debug"
|
||||
return "info"
|
||||
|
||||
|
||||
def build_business_context(
|
||||
context: Mapping[str, Any] | None = None,
|
||||
**fields: Any,
|
||||
) -> dict[str, Any]:
|
||||
payload = dict(context or {})
|
||||
for key, value in fields.items():
|
||||
if value is not None:
|
||||
payload[key] = value
|
||||
return sanitize_log_value(payload)
|
||||
|
||||
|
||||
async def emit_business_log(
|
||||
logger: PlanetLoggerAdapter,
|
||||
*,
|
||||
event: str,
|
||||
message: str,
|
||||
category: str,
|
||||
level: str = "info",
|
||||
source: str = "backend",
|
||||
service: str | None = None,
|
||||
module: str | None = None,
|
||||
request_id: str | None = None,
|
||||
user_id: int | None = None,
|
||||
context: Mapping[str, Any] | None = None,
|
||||
) -> None:
|
||||
normalized_level = normalize_business_level(level)
|
||||
safe_context = build_business_context(context)
|
||||
log_method = getattr(logger, LEVEL_METHODS[normalized_level])
|
||||
log_method(message, event=event, context=safe_context)
|
||||
await record_system_log(
|
||||
source=source,
|
||||
level=normalized_level,
|
||||
message=message,
|
||||
service=service,
|
||||
module=module,
|
||||
event=event,
|
||||
request_id=request_id or get_request_id(),
|
||||
user_id=user_id,
|
||||
category=category,
|
||||
context=safe_context,
|
||||
)
|
||||
|
||||
|
||||
def emit_business_log_background(
|
||||
logger: PlanetLoggerAdapter,
|
||||
*,
|
||||
event: str,
|
||||
message: str,
|
||||
category: str,
|
||||
level: str = "info",
|
||||
source: str = "backend",
|
||||
service: str | None = None,
|
||||
module: str | None = None,
|
||||
request_id: str | None = None,
|
||||
user_id: int | None = None,
|
||||
context: Mapping[str, Any] | None = None,
|
||||
) -> None:
|
||||
normalized_level = normalize_business_level(level)
|
||||
safe_context = build_business_context(context)
|
||||
log_method = getattr(logger, LEVEL_METHODS[normalized_level])
|
||||
log_method(message, event=event, context=safe_context)
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
except RuntimeError:
|
||||
return
|
||||
loop.create_task(
|
||||
record_system_log(
|
||||
source=source,
|
||||
level=normalized_level,
|
||||
message=message,
|
||||
service=service,
|
||||
module=module,
|
||||
event=event,
|
||||
request_id=request_id or get_request_id(),
|
||||
user_id=user_id,
|
||||
category=category,
|
||||
context=safe_context,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def exception_context(exc: BaseException, context: Mapping[str, Any] | None = None) -> dict[str, Any]:
|
||||
return build_business_context(
|
||||
context,
|
||||
error_type=type(exc).__name__,
|
||||
error=str(exc),
|
||||
)
|
||||
@@ -4,42 +4,22 @@ import asyncio
|
||||
from abc import ABC, abstractmethod
|
||||
from typing import Dict, List, Any, Optional
|
||||
from datetime import UTC, datetime
|
||||
from time import perf_counter
|
||||
from urllib.parse import urlparse
|
||||
import httpx
|
||||
from sqlalchemy import select, text
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.collected_data_fields import build_dynamic_metadata, get_record_field
|
||||
from app.core.countries import normalize_country
|
||||
from app.core.logging import get_logger
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.core.websocket.broadcaster import broadcaster
|
||||
from app.services.earth_layer_cache import invalidate_earth_layer_cache_for_source
|
||||
|
||||
EARTH_UPDATE_LAYER_HINTS: dict[str, list[str]] = {
|
||||
"ris_live_bgp": ["bgp"],
|
||||
"bgpstream_bgp": ["bgp"],
|
||||
"top500_supercomputers": ["computeCenters"],
|
||||
"epoch_ai_gpu": ["computeCenters"],
|
||||
"huggingface_models": ["computeCenters"],
|
||||
"huggingface_datasets": ["computeCenters"],
|
||||
"huggingface_spaces": ["computeCenters"],
|
||||
"telegeography_cables": ["cables"],
|
||||
"telegeography_landing_points": ["cables"],
|
||||
"telegeography_cable_systems": ["cables"],
|
||||
"arcgis_cables": ["cables"],
|
||||
"fao_landing_points": ["cables"],
|
||||
"arcgis_landing_points": ["cables"],
|
||||
"arcgis_cable_landing_relations": ["cables"],
|
||||
"spacetrack_tle": ["satellites"],
|
||||
"celestrak_tle": ["satellites"],
|
||||
"barentswatch_vessels": ["vessels"],
|
||||
"aisstream_vessels": ["vessels"],
|
||||
"news_live_streams": ["media"],
|
||||
"media_news_archive": ["news"],
|
||||
}
|
||||
from app.services.business_logs import emit_business_log, exception_context
|
||||
from app.services.earth_layer_adapters import get_earth_update_layers_for_source
|
||||
|
||||
|
||||
def get_earth_update_layers_for_source(source: str) -> list[str]:
|
||||
return EARTH_UPDATE_LAYER_HINTS.get(source, [])
|
||||
logger = get_logger(__name__, service="collector")
|
||||
|
||||
|
||||
class BaseCollector(ABC):
|
||||
@@ -58,6 +38,7 @@ class BaseCollector(ABC):
|
||||
self._datasource_id = 1
|
||||
self._resolved_url: Optional[str] = None
|
||||
self._last_broadcast_progress: Optional[int] = None
|
||||
self._last_save_summary: dict[str, int] = {}
|
||||
|
||||
async def resolve_url(self, db: AsyncSession) -> None:
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
@@ -96,29 +77,6 @@ class BaseCollector(ABC):
|
||||
)
|
||||
self._last_broadcast_progress = rounded_progress
|
||||
|
||||
async def _publish_earth_update(
|
||||
self,
|
||||
*,
|
||||
action: str,
|
||||
records_processed: int,
|
||||
task_id: int | None = None,
|
||||
) -> None:
|
||||
layers = get_earth_update_layers_for_source(self.name)
|
||||
if not layers:
|
||||
return
|
||||
await broadcaster.broadcast_earth_update(
|
||||
{
|
||||
"action": action,
|
||||
"source": self.name,
|
||||
"data_type": self.data_type,
|
||||
"layers": layers,
|
||||
"datasource_id": getattr(self, "_datasource_id", None),
|
||||
"task_id": task_id,
|
||||
"records_processed": records_processed,
|
||||
"timestamp": to_iso8601_utc(datetime.now(UTC)),
|
||||
}
|
||||
)
|
||||
|
||||
async def update_progress(self, records_processed: int, *, commit: bool = False, force: bool = False):
|
||||
"""Update task progress - call this during data processing"""
|
||||
if self._current_task and self._db_session:
|
||||
@@ -322,19 +280,39 @@ class BaseCollector(ABC):
|
||||
from app.models.data_snapshot import DataSnapshot
|
||||
|
||||
start_time = datetime.now(UTC)
|
||||
started_at = perf_counter()
|
||||
datasource_id = getattr(self, "_datasource_id", 1)
|
||||
snapshot_id: Optional[int] = None
|
||||
|
||||
if not collector_registry.is_active(self.name):
|
||||
await self._log_collection_event(
|
||||
"collector.run.skipped_disabled",
|
||||
"Collector skipped because it is disabled",
|
||||
level="info",
|
||||
context={"status": "skipped", "reason": "disabled"},
|
||||
)
|
||||
return {"status": "skipped", "reason": "Collector is disabled"}
|
||||
|
||||
task = CollectionTask(
|
||||
datasource_id=datasource_id,
|
||||
status="running",
|
||||
phase="queued",
|
||||
started_at=start_time,
|
||||
)
|
||||
db.add(task)
|
||||
task = self._current_task if isinstance(self._current_task, CollectionTask) else None
|
||||
if task is None:
|
||||
task = CollectionTask(
|
||||
datasource_id=datasource_id,
|
||||
source=self.name,
|
||||
task_type="collect",
|
||||
status="running",
|
||||
phase="queued",
|
||||
started_at=start_time,
|
||||
)
|
||||
db.add(task)
|
||||
else:
|
||||
task.datasource_id = datasource_id
|
||||
task.source = task.source or self.name
|
||||
task.task_type = task.task_type or "collect"
|
||||
task.status = "running"
|
||||
task.phase = "queued"
|
||||
task.started_at = task.started_at or start_time
|
||||
task.completed_at = None
|
||||
task.error_message = None
|
||||
await db.commit()
|
||||
task_id = task.id
|
||||
|
||||
@@ -344,23 +322,76 @@ class BaseCollector(ABC):
|
||||
|
||||
await self.resolve_url(db)
|
||||
await self._publish_task_update(force=True)
|
||||
await self._log_collection_event(
|
||||
"collector.run.started",
|
||||
"Collector run started",
|
||||
context={"status": "running", "task_id": task_id},
|
||||
)
|
||||
|
||||
try:
|
||||
phase_started_at = perf_counter()
|
||||
await self.set_phase("fetching", message="正在拉取原始数据")
|
||||
await self._log_collection_event(
|
||||
"collector.phase.fetching.start",
|
||||
"Collector fetch phase started",
|
||||
context={"task_id": task_id, "snapshot_id": snapshot_id},
|
||||
)
|
||||
raw_data = await self.fetch()
|
||||
task.total_records = len(raw_data)
|
||||
await db.commit()
|
||||
await self._publish_task_update(force=True)
|
||||
await self._log_collection_event(
|
||||
"collector.phase.fetching.success",
|
||||
"Collector fetch phase completed",
|
||||
context={
|
||||
"task_id": task_id,
|
||||
"raw_count": len(raw_data),
|
||||
"duration_ms": self._duration_ms(phase_started_at),
|
||||
},
|
||||
)
|
||||
|
||||
if self.fail_on_empty and not raw_data:
|
||||
raise RuntimeError(f"Collector {self.name} returned no data")
|
||||
|
||||
phase_started_at = perf_counter()
|
||||
await self.set_phase("transforming", message="正在转换采集数据")
|
||||
await self._log_collection_event(
|
||||
"collector.phase.transforming.start",
|
||||
"Collector transform phase started",
|
||||
context={"task_id": task_id, "raw_count": len(raw_data)},
|
||||
)
|
||||
data = self.transform(raw_data)
|
||||
await self._log_collection_event(
|
||||
"collector.phase.transforming.success",
|
||||
"Collector transform phase completed",
|
||||
context={
|
||||
"task_id": task_id,
|
||||
"raw_count": len(raw_data),
|
||||
"transformed_count": len(data),
|
||||
"duration_ms": self._duration_ms(phase_started_at),
|
||||
},
|
||||
)
|
||||
snapshot_id = await self._create_snapshot(db, task_id, data, start_time)
|
||||
|
||||
phase_started_at = perf_counter()
|
||||
await self.set_phase("saving", message="正在保存采集数据")
|
||||
await self._log_collection_event(
|
||||
"collector.phase.saving.start",
|
||||
"Collector save phase started",
|
||||
context={"task_id": task_id, "snapshot_id": snapshot_id, "transformed_count": len(data)},
|
||||
)
|
||||
records_count = await self._save_data(db, data, task_id=task_id, snapshot_id=snapshot_id)
|
||||
await self._log_collection_event(
|
||||
"collector.phase.saving.success",
|
||||
"Collector save phase completed",
|
||||
context={
|
||||
"task_id": task_id,
|
||||
"snapshot_id": snapshot_id,
|
||||
"saved_count": records_count,
|
||||
**self._last_save_summary,
|
||||
"duration_ms": self._duration_ms(phase_started_at),
|
||||
},
|
||||
)
|
||||
|
||||
task.status = "success"
|
||||
task.phase = "completed"
|
||||
@@ -374,10 +405,19 @@ class BaseCollector(ABC):
|
||||
task.completed_at = datetime.now(UTC)
|
||||
await db.commit()
|
||||
await self._publish_task_update(force=True)
|
||||
await self._publish_earth_update(
|
||||
action="collector_completed",
|
||||
records_processed=records_count,
|
||||
task_id=task_id,
|
||||
await self._log_collection_event(
|
||||
"collector.run.completed",
|
||||
"Collector run completed",
|
||||
context={
|
||||
"status": "success",
|
||||
"task_id": task_id,
|
||||
"snapshot_id": snapshot_id,
|
||||
"raw_count": len(raw_data),
|
||||
"transformed_count": len(data),
|
||||
"saved_count": records_count,
|
||||
**self._last_save_summary,
|
||||
"duration_ms": self._duration_ms(started_at),
|
||||
},
|
||||
)
|
||||
|
||||
return {
|
||||
@@ -402,6 +442,17 @@ class BaseCollector(ABC):
|
||||
)
|
||||
await db.commit()
|
||||
await self._publish_task_update(force=True)
|
||||
await self._log_collection_event(
|
||||
"collector.run.cancelled",
|
||||
"Collector run cancelled",
|
||||
level="warning",
|
||||
context={
|
||||
"status": "cancelled",
|
||||
"task_id": task_id,
|
||||
"snapshot_id": snapshot_id,
|
||||
"duration_ms": self._duration_ms(started_at),
|
||||
},
|
||||
)
|
||||
raise
|
||||
except Exception as e:
|
||||
await db.rollback()
|
||||
@@ -418,6 +469,20 @@ class BaseCollector(ABC):
|
||||
snapshot.summary = {"error": str(e)}
|
||||
await db.commit()
|
||||
await self._publish_task_update(force=True)
|
||||
await self._log_collection_event(
|
||||
"collector.run.failed",
|
||||
"Collector run failed",
|
||||
level="error",
|
||||
context=exception_context(
|
||||
e,
|
||||
{
|
||||
"status": "failed",
|
||||
"task_id": task_id,
|
||||
"snapshot_id": snapshot_id,
|
||||
"duration_ms": self._duration_ms(started_at),
|
||||
},
|
||||
),
|
||||
)
|
||||
|
||||
return {
|
||||
"status": "failed",
|
||||
@@ -438,6 +503,7 @@ class BaseCollector(ABC):
|
||||
from app.models.data_snapshot import DataSnapshot
|
||||
|
||||
if not data:
|
||||
self._last_save_summary = {"created": 0, "updated": 0, "unchanged": 0, "deleted": 0}
|
||||
if snapshot_id is not None:
|
||||
snapshot = await db.get(DataSnapshot, snapshot_id)
|
||||
if snapshot:
|
||||
@@ -584,12 +650,51 @@ class BaseCollector(ABC):
|
||||
"unchanged": unchanged_count,
|
||||
"deleted": len(deleted_keys),
|
||||
}
|
||||
self._last_save_summary = {
|
||||
"created": created_count,
|
||||
"updated": updated_count,
|
||||
"unchanged": unchanged_count,
|
||||
"deleted": len(deleted_keys),
|
||||
}
|
||||
else:
|
||||
self._last_save_summary = {
|
||||
"created": created_count,
|
||||
"updated": updated_count,
|
||||
"unchanged": unchanged_count,
|
||||
"deleted": 0,
|
||||
}
|
||||
|
||||
await db.commit()
|
||||
invalidate_earth_layer_cache_for_source(self.name)
|
||||
await self.update_progress(len(data), force=True)
|
||||
return records_added
|
||||
|
||||
@staticmethod
|
||||
def _duration_ms(started_at: float) -> int:
|
||||
return int((perf_counter() - started_at) * 1000)
|
||||
|
||||
async def _log_collection_event(
|
||||
self,
|
||||
event: str,
|
||||
message: str,
|
||||
*,
|
||||
level: str = "info",
|
||||
context: Dict[str, Any] | None = None,
|
||||
) -> None:
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event=event,
|
||||
message=message,
|
||||
category="collector",
|
||||
level=level,
|
||||
service="collector",
|
||||
module=__name__,
|
||||
context={
|
||||
"collector_name": self.name,
|
||||
"datasource_id": getattr(self, "_datasource_id", None),
|
||||
**(context or {}),
|
||||
},
|
||||
)
|
||||
|
||||
async def save(self, db: AsyncSession, data: List[Dict[str, Any]]) -> int:
|
||||
"""Save data to database (legacy method, use _save_data instead)"""
|
||||
return await self._save_data(db, data)
|
||||
@@ -602,10 +707,65 @@ class HTTPCollector(BaseCollector):
|
||||
headers: Dict[str, str] = {}
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
started_at = perf_counter()
|
||||
request_host = urlparse(self.base_url).netloc
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="collector.http.fetch.start",
|
||||
message="Collector HTTP request started",
|
||||
category="collector",
|
||||
service="collector",
|
||||
module=__name__,
|
||||
context={
|
||||
"collector_name": self.name,
|
||||
"datasource_id": getattr(self, "_datasource_id", None),
|
||||
"url_host": request_host,
|
||||
},
|
||||
)
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(self.base_url, headers=self.headers)
|
||||
response.raise_for_status()
|
||||
return self.parse_response(response.json())
|
||||
try:
|
||||
response = await client.get(self.base_url, headers=self.headers)
|
||||
response.raise_for_status()
|
||||
payload = response.json()
|
||||
parsed = self.parse_response(payload)
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="collector.http.fetch.success",
|
||||
message="Collector HTTP request completed",
|
||||
category="collector",
|
||||
service="collector",
|
||||
module=__name__,
|
||||
context={
|
||||
"collector_name": self.name,
|
||||
"datasource_id": getattr(self, "_datasource_id", None),
|
||||
"url_host": request_host,
|
||||
"status_code": response.status_code,
|
||||
"response_bytes": len(response.content or b""),
|
||||
"parsed_count": len(parsed),
|
||||
"duration_ms": BaseCollector._duration_ms(started_at),
|
||||
},
|
||||
)
|
||||
return parsed
|
||||
except Exception as exc:
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="collector.http.fetch.failed",
|
||||
message="Collector HTTP request failed",
|
||||
category="collector",
|
||||
level="error",
|
||||
service="collector",
|
||||
module=__name__,
|
||||
context=exception_context(
|
||||
exc,
|
||||
{
|
||||
"collector_name": self.name,
|
||||
"datasource_id": getattr(self, "_datasource_id", None),
|
||||
"url_host": request_host,
|
||||
"duration_ms": BaseCollector._duration_ms(started_at),
|
||||
},
|
||||
),
|
||||
)
|
||||
raise
|
||||
|
||||
@abstractmethod
|
||||
def parse_response(self, response: Dict[str, Any]) -> List[Dict[str, Any]]:
|
||||
|
||||
@@ -1,15 +1,39 @@
|
||||
"""CelesTrak TLE Collector
|
||||
"""CelesTrak TLE Collector.
|
||||
|
||||
Collects satellite TLE (Two-Line Element) data from CelesTrak.org.
|
||||
Free, no authentication required.
|
||||
Collects the full active satellite GP element set from CelesTrak.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from typing import Dict, Any, List
|
||||
from pathlib import Path
|
||||
from time import perf_counter
|
||||
from typing import Any, Dict, List
|
||||
from urllib.parse import urlencode, urlparse
|
||||
|
||||
import httpx
|
||||
|
||||
from app.core.logging import get_logger
|
||||
from app.core.satellite_tle import build_tle_lines_from_elements
|
||||
from app.services.business_logs import emit_business_log, exception_context
|
||||
from app.services.collectors.base import BaseCollector
|
||||
from app.services.collectors.downloads import DownloadHTTPStatusError, ResumableFileDownloader
|
||||
|
||||
|
||||
logger = get_logger(__name__, service="collector")
|
||||
ACTIVE_GROUP = "active"
|
||||
FALLBACK_GROUPS = (
|
||||
"starlink",
|
||||
"gps-ops",
|
||||
"galileo",
|
||||
"glonass",
|
||||
"beidou",
|
||||
"leo",
|
||||
"geo",
|
||||
"iridium-next",
|
||||
)
|
||||
FETCH_RETRY_ATTEMPTS = 3
|
||||
FETCH_RETRY_BASE_DELAY_SECONDS = 0.8
|
||||
CELESTRAK_NOT_UPDATED_MARKER = "GP data has not updated since your last successful"
|
||||
|
||||
|
||||
class CelesTrakTLECollector(BaseCollector):
|
||||
@@ -18,55 +42,359 @@ class CelesTrakTLECollector(BaseCollector):
|
||||
module = "L3"
|
||||
frequency_hours = 24
|
||||
data_type = "satellite_tle"
|
||||
_downloader = ResumableFileDownloader(
|
||||
cache_namespace="celestrak",
|
||||
default_accept="application/json",
|
||||
)
|
||||
|
||||
@property
|
||||
def base_url(self) -> str:
|
||||
return self._resolved_url or ""
|
||||
|
||||
def _active_url(self) -> str:
|
||||
return self._group_url(ACTIVE_GROUP)
|
||||
|
||||
def _group_url(self, group: str) -> str:
|
||||
if not self.base_url:
|
||||
raise RuntimeError("CelesTrak base URL is not configured")
|
||||
return f"{self.base_url}?{urlencode({'GROUP': group, 'FORMAT': 'json'})}"
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
satellite_groups = [
|
||||
"starlink",
|
||||
"gps-ops",
|
||||
"galileo",
|
||||
"glonass",
|
||||
"beidou",
|
||||
"leo",
|
||||
"geo",
|
||||
"iridium-next",
|
||||
]
|
||||
url = self._active_url()
|
||||
last_error: Exception | None = None
|
||||
|
||||
all_satellites = []
|
||||
|
||||
async with httpx.AsyncClient(timeout=120.0) as client:
|
||||
for group in satellite_groups:
|
||||
async with httpx.AsyncClient(timeout=180.0, follow_redirects=True) as client:
|
||||
for attempt in range(1, FETCH_RETRY_ATTEMPTS + 1):
|
||||
started_at = perf_counter()
|
||||
try:
|
||||
url = f"{self.base_url}?GROUP={group}&FORMAT=json"
|
||||
response = await client.get(url)
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="collector.celestrak.download.start",
|
||||
message="CelesTrak active satellite download started",
|
||||
category="collector",
|
||||
service="collector",
|
||||
module=__name__,
|
||||
context={
|
||||
"collector_name": self.name,
|
||||
"datasource_id": getattr(self, "_datasource_id", None),
|
||||
"group": ACTIVE_GROUP,
|
||||
"attempt": attempt,
|
||||
"url_host": urlparse(url).netloc,
|
||||
},
|
||||
)
|
||||
body_path = await self._downloader.download_file(
|
||||
client,
|
||||
url,
|
||||
extension=".json",
|
||||
accept="application/json",
|
||||
progress_callback=self._report_download_progress,
|
||||
validate_existing=self._validate_json_file,
|
||||
)
|
||||
data = await self._load_downloaded_payload(body_path, url)
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="collector.celestrak.download.success",
|
||||
message="CelesTrak active satellite download completed",
|
||||
category="collector",
|
||||
service="collector",
|
||||
module=__name__,
|
||||
context={
|
||||
"collector_name": self.name,
|
||||
"datasource_id": getattr(self, "_datasource_id", None),
|
||||
"group": ACTIVE_GROUP,
|
||||
"attempt": attempt,
|
||||
"record_count": len(data),
|
||||
"duration_ms": self._duration_ms(started_at),
|
||||
},
|
||||
)
|
||||
return data
|
||||
except DownloadHTTPStatusError as exc:
|
||||
if self._is_not_updated_response(exc):
|
||||
cached_path = self._downloader.get_cached_file(
|
||||
url,
|
||||
".json",
|
||||
validate_existing=self._validate_json_file,
|
||||
)
|
||||
if cached_path is not None:
|
||||
data = await self._load_downloaded_payload(cached_path, url)
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="collector.celestrak.download.cached_not_updated",
|
||||
message="CelesTrak active satellite data has not changed; using cached download",
|
||||
category="collector",
|
||||
level="warning",
|
||||
service="collector",
|
||||
module=__name__,
|
||||
context={
|
||||
"collector_name": self.name,
|
||||
"datasource_id": getattr(self, "_datasource_id", None),
|
||||
"group": ACTIVE_GROUP,
|
||||
"attempt": attempt,
|
||||
"record_count": len(data),
|
||||
"duration_ms": self._duration_ms(started_at),
|
||||
},
|
||||
)
|
||||
return data
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="collector.celestrak.download.not_updated_no_cache",
|
||||
message="CelesTrak active satellite data has not changed; trying fallback groups",
|
||||
category="collector",
|
||||
level="warning",
|
||||
service="collector",
|
||||
module=__name__,
|
||||
context=exception_context(
|
||||
exc,
|
||||
{
|
||||
"collector_name": self.name,
|
||||
"datasource_id": getattr(self, "_datasource_id", None),
|
||||
"group": ACTIVE_GROUP,
|
||||
"attempt": attempt,
|
||||
"duration_ms": self._duration_ms(started_at),
|
||||
},
|
||||
),
|
||||
)
|
||||
return await self._fetch_fallback_groups(client, active_error=exc)
|
||||
raise
|
||||
except Exception as exc:
|
||||
last_error = exc
|
||||
is_final_attempt = attempt >= FETCH_RETRY_ATTEMPTS
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event=(
|
||||
"collector.celestrak.download.failed"
|
||||
if is_final_attempt
|
||||
else "collector.celestrak.download.retry"
|
||||
),
|
||||
message=(
|
||||
"CelesTrak active satellite download failed"
|
||||
if is_final_attempt
|
||||
else "CelesTrak active satellite download will retry"
|
||||
),
|
||||
category="collector",
|
||||
level="error" if is_final_attempt else "warning",
|
||||
service="collector",
|
||||
module=__name__,
|
||||
context=exception_context(
|
||||
exc,
|
||||
{
|
||||
"collector_name": self.name,
|
||||
"datasource_id": getattr(self, "_datasource_id", None),
|
||||
"group": ACTIVE_GROUP,
|
||||
"attempt": attempt,
|
||||
"duration_ms": self._duration_ms(started_at),
|
||||
},
|
||||
),
|
||||
)
|
||||
if not is_final_attempt:
|
||||
await asyncio.sleep(FETCH_RETRY_BASE_DELAY_SECONDS * attempt)
|
||||
|
||||
if response.status_code == 200:
|
||||
data = response.json()
|
||||
if isinstance(data, list):
|
||||
for item in data:
|
||||
if isinstance(item, dict):
|
||||
item["_celestrak_group"] = group
|
||||
all_satellites.extend(data)
|
||||
print(f"CelesTrak: Fetched {len(data)} satellites from group '{group}'")
|
||||
except Exception as e:
|
||||
print(f"CelesTrak: Error fetching group '{group}': {e}")
|
||||
raise RuntimeError(f"CelesTrak active satellite download failed after retries: {last_error}")
|
||||
|
||||
if not all_satellites:
|
||||
return self._get_sample_data()
|
||||
async def _fetch_fallback_groups(
|
||||
self,
|
||||
client: httpx.AsyncClient,
|
||||
*,
|
||||
active_error: DownloadHTTPStatusError,
|
||||
) -> List[Dict[str, Any]]:
|
||||
started_at = perf_counter()
|
||||
records_by_norad: dict[str, Dict[str, Any]] = {}
|
||||
group_counts: dict[str, int] = {}
|
||||
|
||||
print(f"CelesTrak: Total satellites fetched: {len(all_satellites)}")
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="collector.celestrak.fallback_groups.start",
|
||||
message="CelesTrak fallback group download started",
|
||||
category="collector",
|
||||
level="warning",
|
||||
service="collector",
|
||||
module=__name__,
|
||||
context={
|
||||
"collector_name": self.name,
|
||||
"datasource_id": getattr(self, "_datasource_id", None),
|
||||
"groups": list(FALLBACK_GROUPS),
|
||||
"reason": "active_not_updated_without_cache",
|
||||
},
|
||||
)
|
||||
|
||||
# Return raw data - base.run() will call transform()
|
||||
return all_satellites
|
||||
try:
|
||||
for group in FALLBACK_GROUPS:
|
||||
group_url = self._group_url(group)
|
||||
try:
|
||||
body_path = await self._downloader.download_file(
|
||||
client,
|
||||
group_url,
|
||||
extension=".json",
|
||||
accept="application/json",
|
||||
validate_existing=self._validate_json_file,
|
||||
)
|
||||
except DownloadHTTPStatusError as exc:
|
||||
if not self._is_not_updated_response(exc):
|
||||
raise RuntimeError(f"CelesTrak fallback group '{group}' download failed: {exc}") from exc
|
||||
cached_path = self._downloader.get_cached_file(
|
||||
group_url,
|
||||
".json",
|
||||
validate_existing=self._validate_json_file,
|
||||
)
|
||||
if cached_path is None:
|
||||
raise RuntimeError(
|
||||
f"CelesTrak fallback group '{group}' has not updated and no local cached copy is available"
|
||||
) from exc
|
||||
body_path = cached_path
|
||||
|
||||
group_records = await self._load_downloaded_payload(
|
||||
body_path,
|
||||
group_url,
|
||||
query_group=group,
|
||||
constellation_group=group,
|
||||
)
|
||||
group_counts[group] = len(group_records)
|
||||
for item in group_records:
|
||||
norad_cat_id = item.get("NORAD_CAT_ID")
|
||||
if norad_cat_id is None:
|
||||
continue
|
||||
records_by_norad.setdefault(str(norad_cat_id), item)
|
||||
except Exception as exc:
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="collector.celestrak.fallback_groups.failed",
|
||||
message="CelesTrak fallback group download failed",
|
||||
category="collector",
|
||||
level="error",
|
||||
service="collector",
|
||||
module=__name__,
|
||||
context=exception_context(
|
||||
exc,
|
||||
{
|
||||
"collector_name": self.name,
|
||||
"datasource_id": getattr(self, "_datasource_id", None),
|
||||
"groups": list(FALLBACK_GROUPS),
|
||||
"completed_groups": list(group_counts),
|
||||
"duration_ms": self._duration_ms(started_at),
|
||||
},
|
||||
),
|
||||
)
|
||||
raise RuntimeError(
|
||||
"CelesTrak active data has not updated since this network's last successful download, "
|
||||
"no active cache is available, and fallback group mode failed. Wait until CelesTrak "
|
||||
"publishes the next GP update, restore the Planet download cache, or use Space-Track."
|
||||
) from active_error
|
||||
|
||||
records = list(records_by_norad.values())
|
||||
if not records:
|
||||
raise RuntimeError("CelesTrak fallback group mode produced no satellite records")
|
||||
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="collector.celestrak.fallback_groups.success",
|
||||
message="CelesTrak fallback group download completed",
|
||||
category="collector",
|
||||
level="warning",
|
||||
service="collector",
|
||||
module=__name__,
|
||||
context={
|
||||
"collector_name": self.name,
|
||||
"datasource_id": getattr(self, "_datasource_id", None),
|
||||
"groups": list(FALLBACK_GROUPS),
|
||||
"group_counts": group_counts,
|
||||
"record_count": len(records),
|
||||
"duration_ms": self._duration_ms(started_at),
|
||||
},
|
||||
)
|
||||
return records
|
||||
|
||||
async def _load_downloaded_payload(
|
||||
self,
|
||||
body_path: Path,
|
||||
url: str,
|
||||
*,
|
||||
query_group: str = ACTIVE_GROUP,
|
||||
constellation_group: str | None = None,
|
||||
) -> List[Dict[str, Any]]:
|
||||
try:
|
||||
data = self._load_active_payload(body_path)
|
||||
except RuntimeError as exc:
|
||||
await self._log_parse_failure(exc)
|
||||
raise
|
||||
for item in data:
|
||||
item["_celestrak_query_group"] = query_group
|
||||
item["_celestrak_source_url"] = url
|
||||
if constellation_group:
|
||||
item["_celestrak_group"] = constellation_group
|
||||
return data
|
||||
|
||||
@staticmethod
|
||||
def _is_not_updated_response(exc: DownloadHTTPStatusError) -> bool:
|
||||
return exc.status_code == 403 and CELESTRAK_NOT_UPDATED_MARKER in exc.body
|
||||
|
||||
@staticmethod
|
||||
def _duration_ms(started_at: float) -> int:
|
||||
return int((perf_counter() - started_at) * 1000)
|
||||
|
||||
async def _report_download_progress(self, downloaded: int, total: int | None) -> None:
|
||||
if total and total > 0:
|
||||
await self.update_phase_progress(
|
||||
current=min(downloaded, total),
|
||||
total=total,
|
||||
unit="bytes",
|
||||
message=f"正在下载 CelesTrak active 卫星数据 {downloaded}/{total} bytes",
|
||||
commit=True,
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _validate_json_file(path: Path) -> bool:
|
||||
try:
|
||||
data = json.loads(path.read_text(encoding="utf-8"))
|
||||
except (OSError, json.JSONDecodeError, UnicodeDecodeError):
|
||||
return False
|
||||
return isinstance(data, list)
|
||||
|
||||
async def _log_parse_failure(self, exc: Exception) -> None:
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="collector.celestrak.parse.failed",
|
||||
message="CelesTrak active satellite JSON parsing failed",
|
||||
category="collector",
|
||||
level="error",
|
||||
service="collector",
|
||||
module=__name__,
|
||||
context=exception_context(
|
||||
exc,
|
||||
{
|
||||
"collector_name": self.name,
|
||||
"datasource_id": getattr(self, "_datasource_id", None),
|
||||
"group": ACTIVE_GROUP,
|
||||
},
|
||||
),
|
||||
)
|
||||
|
||||
def _load_active_payload(self, path: Path) -> List[Dict[str, Any]]:
|
||||
try:
|
||||
raw = json.loads(path.read_text(encoding="utf-8"))
|
||||
except (OSError, json.JSONDecodeError, UnicodeDecodeError) as exc:
|
||||
raise RuntimeError(f"CelesTrak active payload is not valid JSON: {exc}") from exc
|
||||
if not isinstance(raw, list):
|
||||
raise RuntimeError("CelesTrak active payload is not a JSON array")
|
||||
|
||||
records: List[Dict[str, Any]] = []
|
||||
invalid_count = 0
|
||||
for item in raw:
|
||||
if isinstance(item, dict) and item.get("NORAD_CAT_ID") is not None:
|
||||
records.append(item)
|
||||
else:
|
||||
invalid_count += 1
|
||||
if invalid_count:
|
||||
raise RuntimeError(f"CelesTrak active payload contains {invalid_count} invalid record(s)")
|
||||
if not records:
|
||||
raise RuntimeError("CelesTrak active payload contains no satellite records")
|
||||
return records
|
||||
|
||||
def transform(self, raw_data: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
|
||||
transformed = []
|
||||
for item in raw_data:
|
||||
norad_cat_id = item.get("NORAD_CAT_ID")
|
||||
tle_line1, tle_line2 = build_tle_lines_from_elements(
|
||||
norad_cat_id=item.get("NORAD_CAT_ID"),
|
||||
norad_cat_id=norad_cat_id,
|
||||
epoch=item.get("EPOCH"),
|
||||
inclination=item.get("INCLINATION"),
|
||||
raan=item.get("RA_OF_ASC_NODE"),
|
||||
@@ -75,14 +403,18 @@ class CelesTrakTLECollector(BaseCollector):
|
||||
mean_anomaly=item.get("MEAN_ANOMALY"),
|
||||
mean_motion=item.get("MEAN_MOTION"),
|
||||
)
|
||||
constellation_group = self._infer_constellation_group(item)
|
||||
|
||||
transformed.append(
|
||||
{
|
||||
"source_id": str(norad_cat_id),
|
||||
"name": item.get("OBJECT_NAME", "Unknown"),
|
||||
"reference_date": item.get("EPOCH", ""),
|
||||
"metadata": {
|
||||
"constellation_group": item.get("_celestrak_group"),
|
||||
"norad_cat_id": item.get("NORAD_CAT_ID"),
|
||||
"constellation_group": constellation_group,
|
||||
"celestrak_query_group": item.get("_celestrak_query_group") or ACTIVE_GROUP,
|
||||
"celestrak_source_url": item.get("_celestrak_source_url"),
|
||||
"norad_cat_id": norad_cat_id,
|
||||
"international_designator": item.get("OBJECT_ID"),
|
||||
"epoch": item.get("EPOCH"),
|
||||
"mean_motion": item.get("MEAN_MOTION"),
|
||||
@@ -105,6 +437,19 @@ class CelesTrakTLECollector(BaseCollector):
|
||||
)
|
||||
return transformed
|
||||
|
||||
@staticmethod
|
||||
def _infer_constellation_group(item: Dict[str, Any]) -> str | None:
|
||||
explicit_group = str(item.get("_celestrak_group") or "").strip().lower()
|
||||
if explicit_group and explicit_group != ACTIVE_GROUP:
|
||||
return explicit_group
|
||||
|
||||
name = str(item.get("OBJECT_NAME") or "").strip().upper()
|
||||
if name.startswith("STARLINK"):
|
||||
return "starlink"
|
||||
if name.startswith("IRIDIUM"):
|
||||
return "iridium-next"
|
||||
return None
|
||||
|
||||
def _get_sample_data(self) -> List[Dict[str, Any]]:
|
||||
return [
|
||||
{
|
||||
|
||||
@@ -4,7 +4,7 @@ from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import tempfile
|
||||
import os
|
||||
import time
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
@@ -17,6 +17,31 @@ ProgressCallback = Callable[[int, int | None], Awaitable[None]]
|
||||
ValidateCallback = Callable[[Path], bool]
|
||||
|
||||
|
||||
class DownloadHTTPStatusError(RuntimeError):
|
||||
"""HTTP status error that keeps the upstream response body for caller-specific handling."""
|
||||
|
||||
def __init__(self, *, url: str, status_code: int, body: str) -> None:
|
||||
self.url = url
|
||||
self.status_code = status_code
|
||||
self.body = body
|
||||
preview = body.strip().replace("\r", " ").replace("\n", " ")[:240]
|
||||
suffix = f": {preview}" if preview else ""
|
||||
super().__init__(f"HTTP {status_code} while downloading {url}{suffix}")
|
||||
|
||||
|
||||
def default_download_cache_root() -> Path:
|
||||
configured = os.getenv("PLANET_DOWNLOAD_CACHE_DIR")
|
||||
if configured:
|
||||
return Path(configured).expanduser()
|
||||
planet_cache = os.getenv("PLANET_CACHE_DIR")
|
||||
if planet_cache:
|
||||
return Path(planet_cache).expanduser() / "downloads"
|
||||
xdg_cache = os.getenv("XDG_CACHE_HOME")
|
||||
if xdg_cache:
|
||||
return Path(xdg_cache).expanduser() / "planet" / "downloads"
|
||||
return Path.home() / ".cache" / "planet" / "downloads"
|
||||
|
||||
|
||||
class ResumableFileDownloader:
|
||||
"""Download files with cache validators and byte-range resume support."""
|
||||
|
||||
@@ -26,8 +51,9 @@ class ResumableFileDownloader:
|
||||
cache_namespace: str,
|
||||
user_agent: str = "Planet-Intelligence-System/1.0 (Python/collector)",
|
||||
default_accept: str = "*/*",
|
||||
cache_root: Path | None = None,
|
||||
) -> None:
|
||||
self._cache_dir = Path(tempfile.gettempdir()) / "planet-download-cache" / cache_namespace
|
||||
self._cache_dir = (cache_root or default_download_cache_root()) / cache_namespace
|
||||
self._user_agent = user_agent
|
||||
self._default_accept = default_accept
|
||||
|
||||
@@ -43,6 +69,25 @@ class ResumableFileDownloader:
|
||||
meta_path = self._cache_dir / f"{key}.meta.json"
|
||||
return final_path, part_path, meta_path
|
||||
|
||||
def cached_file_path(self, url: str, extension: str) -> Path:
|
||||
final_path, _, _ = self._cache_paths(url, extension)
|
||||
return final_path
|
||||
|
||||
def get_cached_file(
|
||||
self,
|
||||
url: str,
|
||||
extension: str,
|
||||
*,
|
||||
validate_existing: ValidateCallback | None = None,
|
||||
) -> Path | None:
|
||||
final_path = self.cached_file_path(url, extension)
|
||||
if not final_path.exists():
|
||||
return None
|
||||
if validate_existing and not validate_existing(final_path):
|
||||
final_path.unlink(missing_ok=True)
|
||||
return None
|
||||
return final_path
|
||||
|
||||
@staticmethod
|
||||
def _load_meta(meta_path: Path) -> dict[str, Any]:
|
||||
if not meta_path.exists():
|
||||
@@ -140,7 +185,9 @@ class ResumableFileDownloader:
|
||||
if progress_callback and expected_size and expected_size > 0:
|
||||
await progress_callback(expected_size, expected_size)
|
||||
return final_path
|
||||
response.raise_for_status()
|
||||
if response.status_code >= 400:
|
||||
body = (await response.aread()).decode("utf-8", errors="replace")
|
||||
raise DownloadHTTPStatusError(url=url, status_code=response.status_code, body=body)
|
||||
|
||||
if response.status_code == 206 and resume_from > 0:
|
||||
mode = "ab"
|
||||
|
||||
@@ -163,32 +163,64 @@ class TeleGeographyLandingPointCollector(BaseCollector):
|
||||
data_type = "landing_point"
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
"""Fetch landing point data from GitHub mirror"""
|
||||
url = self._resolved_url or ""
|
||||
"""Fetch landing point data, falling back when the old mirror disappears."""
|
||||
config = get_data_sources_config()
|
||||
sources = [
|
||||
self._resolved_url or "",
|
||||
str(config.get_yaml_value("telegeography.landing_point_url") or ""),
|
||||
str(config.get_yaml_value("arcgis.landing_point_url") or ""),
|
||||
]
|
||||
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(url)
|
||||
response.raise_for_status()
|
||||
return self.parse_response(response.json())
|
||||
last_error: Exception | None = None
|
||||
async with httpx.AsyncClient(timeout=60.0, follow_redirects=True) as client:
|
||||
for url in dict.fromkeys(source for source in sources if source):
|
||||
try:
|
||||
params = (
|
||||
{"where": "1=1", "outFields": "*", "returnGeometry": "true", "f": "geojson"}
|
||||
if "FeatureServer" in url or url.endswith("/query")
|
||||
else None
|
||||
)
|
||||
response = await client.get(url, params=params)
|
||||
response.raise_for_status()
|
||||
records = self.parse_response(response.json())
|
||||
if records:
|
||||
return records
|
||||
except Exception as exc:
|
||||
last_error = exc
|
||||
continue
|
||||
|
||||
def parse_response(self, data: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
|
||||
if last_error:
|
||||
raise last_error
|
||||
return self._get_sample_data()
|
||||
|
||||
def parse_response(self, data: List[Dict[str, Any]] | Dict[str, Any]) -> List[Dict[str, Any]]:
|
||||
"""Parse landing point data"""
|
||||
result = []
|
||||
items = data.get("features", []) if isinstance(data, dict) else data
|
||||
|
||||
for item in data:
|
||||
for item in items:
|
||||
props = item.get("properties", {}) if isinstance(item, dict) else {}
|
||||
geometry = item.get("geometry", {}) if isinstance(item, dict) else {}
|
||||
source = props or item
|
||||
coords = geometry.get("coordinates", []) if isinstance(geometry, dict) else []
|
||||
longitude = coords[0] if len(coords) > 0 else source.get("longitude")
|
||||
latitude = coords[1] if len(coords) > 1 else source.get("latitude")
|
||||
source_id = source.get("id") or source.get("OBJECTID") or source.get("city_id") or ""
|
||||
try:
|
||||
entry = {
|
||||
"source_id": f"telegeo_lp_{item.get('id', '')}",
|
||||
"name": item.get("name", "Unknown"),
|
||||
"country": item.get("country", "Unknown"),
|
||||
"city": item.get("city", item.get("name", "")),
|
||||
"latitude": str(item.get("latitude", "")),
|
||||
"longitude": str(item.get("longitude", "")),
|
||||
"source_id": f"telegeo_lp_{source_id}",
|
||||
"name": source.get("name", source.get("Name", "Unknown")),
|
||||
"country": source.get("country", "Unknown"),
|
||||
"city": source.get("city", source.get("Name", source.get("name", ""))),
|
||||
"latitude": str(latitude or ""),
|
||||
"longitude": str(longitude or ""),
|
||||
"value": "",
|
||||
"unit": "",
|
||||
"metadata": {
|
||||
"cable_count": len(item.get("cables", [])),
|
||||
"url": item.get("url"),
|
||||
"cable_count": len(source.get("cables", [])),
|
||||
"url": source.get("url"),
|
||||
"objectid": source.get("OBJECTID"),
|
||||
"city_id": source.get("city_id"),
|
||||
},
|
||||
"reference_date": datetime.now(UTC).strftime("%Y-%m-%d"),
|
||||
}
|
||||
|
||||
@@ -11,7 +11,7 @@ startup, so it must stay local and deterministic.
|
||||
|
||||
For the full design and the reason behind the abstraction (compute centers,
|
||||
BGP collectors, BGP events, and future entities all share one pipeline),
|
||||
see ``docs/plans/location-resolver-shared-pipeline-plan.md``.
|
||||
see ``docs/technical/zh/location-pipeline-development.md``.
|
||||
|
||||
The ``ComputeCenterLocation`` dataclass and the public function signatures are
|
||||
preserved verbatim so existing callers and tests do not need to change.
|
||||
|
||||
597
backend/app/services/data_jobs.py
Normal file
597
backend/app/services/data_jobs.py
Normal file
@@ -0,0 +1,597 @@
|
||||
"""Kafka-ready datasource job queue backed by PostgreSQL for v1."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from uuid import uuid4
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import bindparam, select, text
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.cache import cache
|
||||
from app.core.config import settings
|
||||
from app.core.logging import get_logger
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.core.websocket.broadcaster import broadcaster
|
||||
from app.db.session import async_session_factory
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.models.data_snapshot import DataSnapshot
|
||||
from app.models.datasource import DataSource
|
||||
from app.models.task import CollectionTask
|
||||
from app.services.collectors.registry import collector_registry
|
||||
from app.services.datasource_connectivity import (
|
||||
build_builtin_connectivity_checksum,
|
||||
get_builtin_effective_candidate,
|
||||
save_connectivity_success,
|
||||
)
|
||||
from app.services.earth_layer_adapters import (
|
||||
clear_derived_datasource_data,
|
||||
get_earth_refresh_strategy_for_change,
|
||||
get_earth_update_layers_for_source,
|
||||
)
|
||||
from app.services.earth_layer_cache import invalidate_earth_layer_cache_for_source
|
||||
from app.services.scheduler import sync_datasource_job
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
JOB_TYPE_COLLECT = "collect"
|
||||
JOB_TYPE_CLEAR_DATA = "clear_data"
|
||||
JOB_TYPE_CLEAR_CACHE = "clear_cache"
|
||||
JOB_TYPE_EARTH_REFRESH = "earth_refresh"
|
||||
|
||||
JOB_STATUS_QUEUED = "queued"
|
||||
JOB_STATUS_RUNNING = "running"
|
||||
JOB_STATUS_CANCELLING = "cancelling"
|
||||
JOB_STATUS_SUCCESS = "success"
|
||||
JOB_STATUS_FAILED = "failed"
|
||||
JOB_STATUS_CANCELLED = "cancelled"
|
||||
|
||||
ACTIVE_JOB_STATUSES = (JOB_STATUS_QUEUED, JOB_STATUS_RUNNING, JOB_STATUS_CANCELLING)
|
||||
TERMINAL_JOB_STATUSES = (JOB_STATUS_SUCCESS, JOB_STATUS_FAILED, JOB_STATUS_CANCELLED)
|
||||
DATA_WRITE_JOB_TYPES = (JOB_TYPE_COLLECT, JOB_TYPE_CLEAR_DATA, JOB_TYPE_CLEAR_CACHE)
|
||||
SOURCE_LOCK_JOB_STATUSES = (JOB_STATUS_RUNNING, JOB_STATUS_CANCELLING)
|
||||
QUEUE_POLL_SECONDS = 0.35
|
||||
JOB_STALE_LOCK_MINUTES = 90
|
||||
DEFAULT_WORKER_CONCURRENCY = 2
|
||||
|
||||
RUNNING_DATA_JOB_TASKS: dict[int, asyncio.Task[Any]] = {}
|
||||
|
||||
|
||||
def _utcnow() -> datetime:
|
||||
return datetime.now(UTC)
|
||||
|
||||
|
||||
def _job_worker_id() -> str:
|
||||
return f"{settings.PROJECT_NAME}:data-job-worker:{uuid4().hex[:8]}"
|
||||
|
||||
|
||||
def is_terminal_job_status(status: str | None) -> bool:
|
||||
return status in TERMINAL_JOB_STATUSES
|
||||
|
||||
|
||||
async def enqueue_datasource_job(
|
||||
db: AsyncSession,
|
||||
datasource: DataSource,
|
||||
task_type: str,
|
||||
*,
|
||||
payload: dict[str, Any] | None = None,
|
||||
rollback_policy: str = "keep_committed_batches",
|
||||
dedupe_key: str | None = None,
|
||||
) -> CollectionTask:
|
||||
if dedupe_key:
|
||||
existing = await _get_active_job_by_dedupe_key(db, dedupe_key)
|
||||
if existing is not None:
|
||||
return existing
|
||||
|
||||
task = CollectionTask(
|
||||
datasource_id=datasource.id,
|
||||
source=datasource.source,
|
||||
task_type=task_type,
|
||||
status=JOB_STATUS_QUEUED,
|
||||
phase="queued",
|
||||
phase_message="任务已进入队列",
|
||||
payload=payload or {},
|
||||
rollback_policy=rollback_policy,
|
||||
dedupe_key=dedupe_key,
|
||||
)
|
||||
db.add(task)
|
||||
await db.commit()
|
||||
await db.refresh(task)
|
||||
await _broadcast_task_update(task)
|
||||
return task
|
||||
|
||||
|
||||
async def enqueue_earth_refresh_job(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
source: str,
|
||||
payload: dict[str, Any] | None = None,
|
||||
) -> CollectionTask | None:
|
||||
layers = list((payload or {}).get("layers") or get_earth_update_layers_for_source(source))
|
||||
if not layers:
|
||||
return None
|
||||
|
||||
datasource = await _get_or_create_virtual_datasource(db, source)
|
||||
refresh_payload = {
|
||||
"source": source,
|
||||
"layers": layers,
|
||||
"refresh_strategy": (payload or {}).get("refresh_strategy")
|
||||
or get_earth_refresh_strategy_for_change((payload or {}).get("table"), source)
|
||||
or "clear_then_reload",
|
||||
**(payload or {}),
|
||||
}
|
||||
return await enqueue_datasource_job(
|
||||
db,
|
||||
datasource,
|
||||
JOB_TYPE_EARTH_REFRESH,
|
||||
payload=refresh_payload,
|
||||
dedupe_key=f"earth_refresh:{source}",
|
||||
)
|
||||
|
||||
|
||||
async def enqueue_earth_refresh_from_update(payload: dict[str, Any]) -> None:
|
||||
source = str(payload.get("source") or "").strip()
|
||||
if not source:
|
||||
return
|
||||
async with async_session_factory() as db:
|
||||
await enqueue_earth_refresh_job(db, source=source, payload=payload)
|
||||
|
||||
|
||||
async def request_cancel_datasource_task(
|
||||
db: AsyncSession,
|
||||
task: CollectionTask,
|
||||
*,
|
||||
reason: str = "cancelled_by_operator",
|
||||
) -> CollectionTask:
|
||||
if is_terminal_job_status(task.status):
|
||||
return task
|
||||
|
||||
running_task = RUNNING_DATA_JOB_TASKS.get(task.id)
|
||||
if task.status == JOB_STATUS_QUEUED or (
|
||||
running_task is None
|
||||
and (
|
||||
task.status == JOB_STATUS_CANCELLING
|
||||
or (task.status == JOB_STATUS_RUNNING and task.task_type != JOB_TYPE_COLLECT)
|
||||
)
|
||||
):
|
||||
return await _cancel_task_without_runner(db, task, reason=reason)
|
||||
|
||||
task.status = JOB_STATUS_CANCELLING
|
||||
task.phase = JOB_STATUS_CANCELLING
|
||||
task.phase_message = "正在停止任务"
|
||||
task.requested_cancel_at = _utcnow()
|
||||
task.cancel_reason = reason
|
||||
await db.commit()
|
||||
await db.refresh(task)
|
||||
|
||||
if running_task is not None and not running_task.done():
|
||||
running_task.cancel()
|
||||
|
||||
await _broadcast_task_update(task)
|
||||
return task
|
||||
|
||||
|
||||
async def _cancel_task_without_runner(
|
||||
db: AsyncSession,
|
||||
task: CollectionTask,
|
||||
*,
|
||||
reason: str,
|
||||
) -> CollectionTask:
|
||||
if task.task_type == JOB_TYPE_COLLECT:
|
||||
await db.execute(CollectedData.__table__.delete().where(CollectedData.task_id == task.id))
|
||||
snapshot_result = await db.execute(select(DataSnapshot).where(DataSnapshot.task_id == task.id))
|
||||
for snapshot in snapshot_result.scalars().all():
|
||||
snapshot.status = JOB_STATUS_CANCELLED
|
||||
snapshot.completed_at = _utcnow()
|
||||
snapshot.error_message = "Cancelled after operator stop request; no active worker handle remained"
|
||||
datasource = await db.get(DataSource, task.datasource_id)
|
||||
if datasource is not None:
|
||||
datasource.last_status = JOB_STATUS_CANCELLED
|
||||
|
||||
task.status = JOB_STATUS_CANCELLED
|
||||
task.phase = JOB_STATUS_CANCELLED
|
||||
task.phase_message = "任务已停止"
|
||||
task.completed_at = _utcnow()
|
||||
task.requested_cancel_at = task.requested_cancel_at or _utcnow()
|
||||
task.cancel_reason = reason
|
||||
await db.commit()
|
||||
await db.refresh(task)
|
||||
await _broadcast_task_update(task)
|
||||
return task
|
||||
|
||||
|
||||
async def get_active_datasource_job(
|
||||
db: AsyncSession,
|
||||
datasource_id: int,
|
||||
*,
|
||||
task_types: tuple[str, ...] = DATA_WRITE_JOB_TYPES,
|
||||
) -> CollectionTask | None:
|
||||
result = await db.execute(
|
||||
select(CollectionTask)
|
||||
.where(CollectionTask.datasource_id == datasource_id)
|
||||
.where(CollectionTask.task_type.in_(task_types))
|
||||
.where(CollectionTask.status.in_(ACTIVE_JOB_STATUSES))
|
||||
.order_by(CollectionTask.created_at.desc().nullslast(), CollectionTask.id.desc())
|
||||
.limit(1)
|
||||
)
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
|
||||
async def _get_active_job_by_dedupe_key(db: AsyncSession, dedupe_key: str) -> CollectionTask | None:
|
||||
result = await db.execute(
|
||||
select(CollectionTask)
|
||||
.where(CollectionTask.dedupe_key == dedupe_key)
|
||||
.where(CollectionTask.status.in_(ACTIVE_JOB_STATUSES))
|
||||
.order_by(CollectionTask.created_at.desc().nullslast(), CollectionTask.id.desc())
|
||||
.limit(1)
|
||||
)
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
|
||||
async def _get_or_create_virtual_datasource(db: AsyncSession, source: str) -> DataSource:
|
||||
result = await db.execute(select(DataSource).where(DataSource.source == source))
|
||||
datasource = result.scalar_one_or_none()
|
||||
if datasource is not None:
|
||||
return datasource
|
||||
|
||||
datasource = DataSource(
|
||||
name=f"Earth refresh: {source}",
|
||||
source=source,
|
||||
module="SYS",
|
||||
collector_class="EarthRefreshJob",
|
||||
is_active=True,
|
||||
)
|
||||
db.add(datasource)
|
||||
await db.commit()
|
||||
await db.refresh(datasource)
|
||||
return datasource
|
||||
|
||||
|
||||
async def _broadcast_task_update(task: CollectionTask) -> None:
|
||||
await broadcaster.broadcast_datasource_task_update(
|
||||
{
|
||||
"datasource_id": task.datasource_id,
|
||||
"collector_name": task.source,
|
||||
"task_id": task.id,
|
||||
"task_type": task.task_type,
|
||||
"status": task.status,
|
||||
"phase": task.phase,
|
||||
"phase_progress": task.phase_progress,
|
||||
"phase_message": task.phase_message,
|
||||
"phase_current": task.phase_current,
|
||||
"phase_total": task.phase_total,
|
||||
"phase_unit": task.phase_unit,
|
||||
"progress": task.progress,
|
||||
"records_processed": task.records_processed,
|
||||
"total_records": task.total_records,
|
||||
"started_at": to_iso8601_utc(task.started_at),
|
||||
"completed_at": to_iso8601_utc(task.completed_at),
|
||||
"requested_cancel_at": to_iso8601_utc(task.requested_cancel_at),
|
||||
"error_message": task.error_message,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
class DataJobWorker:
|
||||
def __init__(self, *, concurrency: int = DEFAULT_WORKER_CONCURRENCY) -> None:
|
||||
self.worker_id = _job_worker_id()
|
||||
self.concurrency = max(1, concurrency)
|
||||
self._task: asyncio.Task[None] | None = None
|
||||
self._stop_event: asyncio.Event | None = None
|
||||
self._running: set[asyncio.Task[Any]] = set()
|
||||
|
||||
def start(self) -> None:
|
||||
if self._task and not self._task.done():
|
||||
return
|
||||
self._stop_event = asyncio.Event()
|
||||
self._task = asyncio.create_task(self._run(), name="data-job-worker")
|
||||
|
||||
async def stop(self) -> None:
|
||||
if self._stop_event:
|
||||
self._stop_event.set()
|
||||
for task in list(self._running):
|
||||
task.cancel()
|
||||
if self._task:
|
||||
await asyncio.gather(self._task, return_exceptions=True)
|
||||
if self._running:
|
||||
await asyncio.gather(*self._running, return_exceptions=True)
|
||||
|
||||
async def _run(self) -> None:
|
||||
assert self._stop_event is not None
|
||||
await self._recover_stale_running_jobs()
|
||||
while not self._stop_event.is_set():
|
||||
self._running = {task for task in self._running if not task.done()}
|
||||
if len(self._running) >= self.concurrency:
|
||||
await asyncio.sleep(QUEUE_POLL_SECONDS)
|
||||
continue
|
||||
|
||||
task_id = await self._claim_next_job()
|
||||
if task_id is None:
|
||||
await asyncio.sleep(QUEUE_POLL_SECONDS)
|
||||
continue
|
||||
|
||||
runner = asyncio.create_task(self._run_claimed_job(task_id), name=f"data-job:{task_id}")
|
||||
self._running.add(runner)
|
||||
|
||||
async def _recover_stale_running_jobs(self) -> None:
|
||||
cutoff = _utcnow() - timedelta(minutes=JOB_STALE_LOCK_MINUTES)
|
||||
async with async_session_factory() as db:
|
||||
result = await db.execute(
|
||||
select(CollectionTask)
|
||||
.where(CollectionTask.status.in_((JOB_STATUS_RUNNING, JOB_STATUS_CANCELLING)))
|
||||
.where(CollectionTask.locked_at.is_not(None))
|
||||
.where(CollectionTask.locked_at < cutoff)
|
||||
)
|
||||
stale_jobs = list(result.scalars().all())
|
||||
for job in stale_jobs:
|
||||
job.status = JOB_STATUS_FAILED
|
||||
job.phase = JOB_STATUS_FAILED
|
||||
job.completed_at = _utcnow()
|
||||
job.error_message = "Marked failed after stale data job lock timeout"
|
||||
if stale_jobs:
|
||||
await db.commit()
|
||||
|
||||
async def _claim_next_job(self) -> int | None:
|
||||
async with async_session_factory() as db:
|
||||
row = await db.execute(
|
||||
text(
|
||||
"""
|
||||
SELECT queued.id
|
||||
FROM collection_tasks AS queued
|
||||
WHERE queued.status = :queued_status
|
||||
AND NOT EXISTS (
|
||||
SELECT 1
|
||||
FROM collection_tasks AS active
|
||||
WHERE active.source = queued.source
|
||||
AND active.id <> queued.id
|
||||
AND active.status IN :active_statuses
|
||||
)
|
||||
ORDER BY queued.created_at ASC NULLS FIRST, queued.id ASC
|
||||
LIMIT 1
|
||||
FOR UPDATE SKIP LOCKED
|
||||
"""
|
||||
).bindparams(bindparam("active_statuses", expanding=True)),
|
||||
{
|
||||
"queued_status": JOB_STATUS_QUEUED,
|
||||
"active_statuses": SOURCE_LOCK_JOB_STATUSES,
|
||||
},
|
||||
)
|
||||
task_id = row.scalar_one_or_none()
|
||||
if task_id is None:
|
||||
return None
|
||||
|
||||
task = await db.get(CollectionTask, int(task_id))
|
||||
if task is None:
|
||||
return None
|
||||
task.status = JOB_STATUS_RUNNING
|
||||
task.phase = "starting"
|
||||
task.phase_message = "任务开始执行"
|
||||
task.started_at = task.started_at or _utcnow()
|
||||
task.worker_id = self.worker_id
|
||||
task.locked_at = _utcnow()
|
||||
await db.commit()
|
||||
await _broadcast_task_update(task)
|
||||
return int(task_id)
|
||||
|
||||
async def _run_claimed_job(self, task_id: int) -> None:
|
||||
current_task = asyncio.current_task()
|
||||
if current_task is not None:
|
||||
RUNNING_DATA_JOB_TASKS[task_id] = current_task
|
||||
try:
|
||||
async with async_session_factory() as db:
|
||||
task = await db.get(CollectionTask, task_id)
|
||||
if task is None:
|
||||
return
|
||||
await self._execute_job(db, task)
|
||||
except asyncio.CancelledError:
|
||||
async with async_session_factory() as db:
|
||||
task = await db.get(CollectionTask, task_id)
|
||||
if task is not None and not is_terminal_job_status(task.status):
|
||||
task.status = JOB_STATUS_CANCELLED
|
||||
task.phase = JOB_STATUS_CANCELLED
|
||||
task.phase_message = "任务已停止"
|
||||
task.completed_at = _utcnow()
|
||||
await db.commit()
|
||||
await _broadcast_task_update(task)
|
||||
raise
|
||||
except Exception as exc:
|
||||
logger.exception_event(
|
||||
"Data job failed",
|
||||
event="data_jobs.job_failed",
|
||||
context={"task_id": task_id, "error": str(exc)},
|
||||
)
|
||||
async with async_session_factory() as db:
|
||||
task = await db.get(CollectionTask, task_id)
|
||||
if task is not None:
|
||||
task.status = JOB_STATUS_FAILED
|
||||
task.phase = JOB_STATUS_FAILED
|
||||
task.phase_message = str(exc)
|
||||
task.error_message = str(exc)
|
||||
task.completed_at = _utcnow()
|
||||
await db.commit()
|
||||
await _broadcast_task_update(task)
|
||||
finally:
|
||||
RUNNING_DATA_JOB_TASKS.pop(task_id, None)
|
||||
|
||||
async def _execute_job(self, db: AsyncSession, task: CollectionTask) -> None:
|
||||
if task.task_type == JOB_TYPE_COLLECT:
|
||||
await _run_collect_job(db, task)
|
||||
elif task.task_type == JOB_TYPE_CLEAR_DATA:
|
||||
await _run_clear_data_job(db, task)
|
||||
elif task.task_type == JOB_TYPE_CLEAR_CACHE:
|
||||
await _run_clear_cache_job(db, task)
|
||||
elif task.task_type == JOB_TYPE_EARTH_REFRESH:
|
||||
await _run_earth_refresh_job(db, task)
|
||||
else:
|
||||
raise RuntimeError(f"Unsupported data job type: {task.task_type}")
|
||||
|
||||
|
||||
async def _run_collect_job(db: AsyncSession, task: CollectionTask) -> None:
|
||||
datasource = await db.get(DataSource, task.datasource_id)
|
||||
if datasource is None:
|
||||
raise RuntimeError("Data source not found")
|
||||
|
||||
collector = collector_registry.get(datasource.source)
|
||||
if collector is None:
|
||||
raise RuntimeError(f"Collector '{datasource.source}' not found")
|
||||
if not datasource.is_active:
|
||||
raise RuntimeError("Data source is disabled")
|
||||
|
||||
collector._datasource_id = datasource.id
|
||||
collector._current_task = task
|
||||
collector._db_session = db
|
||||
result = await collector.run(db)
|
||||
|
||||
datasource.last_run_at = _utcnow()
|
||||
datasource.last_status = result.get("status")
|
||||
if datasource.last_status == JOB_STATUS_SUCCESS:
|
||||
effective_candidate = await get_builtin_effective_candidate(db, datasource.source)
|
||||
checksum, _credential_context = await build_builtin_connectivity_checksum(
|
||||
datasource.source,
|
||||
effective_candidate["endpoint"],
|
||||
effective_candidate["auth_type"],
|
||||
effective_candidate["headers"],
|
||||
effective_candidate["config"],
|
||||
db,
|
||||
)
|
||||
await save_connectivity_success(
|
||||
db,
|
||||
datasource.source,
|
||||
checksum,
|
||||
{"status_code": None},
|
||||
connected_by="collection",
|
||||
)
|
||||
await db.commit()
|
||||
await sync_datasource_job(datasource.id)
|
||||
|
||||
|
||||
async def _run_clear_data_job(db: AsyncSession, task: CollectionTask) -> None:
|
||||
source = str(task.source or (task.payload or {}).get("source") or "").strip()
|
||||
if not source:
|
||||
raise RuntimeError("Clear data job has no source")
|
||||
|
||||
task.phase = "clearing_data"
|
||||
task.phase_message = "正在删除数据库数据"
|
||||
await db.commit()
|
||||
await _broadcast_task_update(task)
|
||||
|
||||
count_result = await db.execute(
|
||||
select(CollectedData.id).where(CollectedData.source == source)
|
||||
)
|
||||
collected_ids = [row[0] for row in count_result.all()]
|
||||
derived_deleted_counts = await clear_derived_datasource_data(db, source)
|
||||
if collected_ids:
|
||||
await db.execute(CollectedData.__table__.delete().where(CollectedData.id.in_(collected_ids)))
|
||||
deleted_count = len(collected_ids)
|
||||
derived_deleted_count = sum(derived_deleted_counts.values())
|
||||
|
||||
task.records_processed = deleted_count + derived_deleted_count
|
||||
task.total_records = task.records_processed
|
||||
task.progress = 100.0
|
||||
task.phase_progress = 100.0
|
||||
task.phase_current = task.records_processed
|
||||
task.phase_total = task.records_processed
|
||||
task.phase_unit = "records"
|
||||
task.payload = {
|
||||
**(task.payload or {}),
|
||||
"deleted_count": deleted_count,
|
||||
"derived_deleted_count": derived_deleted_count,
|
||||
"derived_deleted_counts": derived_deleted_counts,
|
||||
}
|
||||
task.status = JOB_STATUS_SUCCESS
|
||||
task.phase = "completed"
|
||||
task.phase_message = "数据库数据已清理"
|
||||
task.completed_at = _utcnow()
|
||||
await db.commit()
|
||||
await _broadcast_task_update(task)
|
||||
|
||||
|
||||
async def _run_clear_cache_job(db: AsyncSession, task: CollectionTask) -> None:
|
||||
source = str(task.source or (task.payload or {}).get("source") or "").strip()
|
||||
if not source:
|
||||
raise RuntimeError("Clear cache job has no source")
|
||||
|
||||
earth_deleted_count = invalidate_earth_layer_cache_for_source(source)
|
||||
dashboard_deleted_count = int(cache.delete("dashboard:stats")) + int(cache.delete("dashboard:summary"))
|
||||
deleted_count = earth_deleted_count + dashboard_deleted_count
|
||||
|
||||
task.records_processed = deleted_count
|
||||
task.total_records = deleted_count
|
||||
task.progress = 100.0
|
||||
task.phase_progress = 100.0
|
||||
task.phase = "completed"
|
||||
task.phase_message = "缓存已清理"
|
||||
task.payload = {
|
||||
**(task.payload or {}),
|
||||
"earth_layer_deleted_count": earth_deleted_count,
|
||||
"dashboard_deleted_count": dashboard_deleted_count,
|
||||
}
|
||||
task.status = JOB_STATUS_SUCCESS
|
||||
task.completed_at = _utcnow()
|
||||
await db.commit()
|
||||
await _broadcast_task_update(task)
|
||||
await enqueue_earth_refresh_job(db, source=source, payload={"operation": "CACHE_INVALIDATED"})
|
||||
|
||||
|
||||
async def _run_earth_refresh_job(db: AsyncSession, task: CollectionTask) -> None:
|
||||
payload = task.payload or {}
|
||||
source = str(payload.get("source") or task.source or "").strip()
|
||||
layers = list(payload.get("layers") or get_earth_update_layers_for_source(source))
|
||||
if not source or not layers:
|
||||
task.status = JOB_STATUS_SUCCESS
|
||||
task.phase = "completed"
|
||||
task.phase_message = "没有需要刷新的 Earth 图层"
|
||||
task.completed_at = _utcnow()
|
||||
await db.commit()
|
||||
await _broadcast_task_update(task)
|
||||
return
|
||||
|
||||
deleted_cache_entries = invalidate_earth_layer_cache_for_source(source)
|
||||
update_payload = {
|
||||
"event": "earth.layer.changed",
|
||||
"action": "database_changed",
|
||||
"source": source,
|
||||
"table": payload.get("table"),
|
||||
"data_type": source,
|
||||
"layers": layers,
|
||||
"refresh_strategy": payload.get("refresh_strategy") or "clear_then_reload",
|
||||
"records_processed": payload.get("records_processed", 0),
|
||||
"operations": payload.get("operations") or [payload.get("operation") or "CHANGE"],
|
||||
"operation": payload.get("operation"),
|
||||
"cache_entries_invalidated": deleted_cache_entries,
|
||||
"timestamp": to_iso8601_utc(_utcnow()),
|
||||
}
|
||||
if payload.get("entity") == "interactable":
|
||||
update_payload.update(
|
||||
{
|
||||
"entity": "interactable",
|
||||
"action": payload.get("action") or "changed",
|
||||
"ids": payload.get("ids") or payload.get("entity_keys") or [],
|
||||
"item": payload.get("item"),
|
||||
}
|
||||
)
|
||||
await broadcaster.broadcast_earth_update(update_payload)
|
||||
|
||||
task.records_processed = int(payload.get("records_processed") or 0)
|
||||
task.progress = 100.0
|
||||
task.phase_progress = 100.0
|
||||
task.phase = "completed"
|
||||
task.phase_message = "Earth 图层刷新通知已发送"
|
||||
task.status = JOB_STATUS_SUCCESS
|
||||
task.completed_at = _utcnow()
|
||||
task.payload = {**payload, "cache_entries_invalidated": deleted_cache_entries}
|
||||
await db.commit()
|
||||
await _broadcast_task_update(task)
|
||||
|
||||
|
||||
_worker = DataJobWorker()
|
||||
|
||||
|
||||
def start_data_job_worker() -> None:
|
||||
_worker.start()
|
||||
|
||||
|
||||
async def stop_data_job_worker() -> None:
|
||||
await _worker.stop()
|
||||
@@ -32,26 +32,29 @@ class DocsMetadata:
|
||||
|
||||
DOCS_METADATA: tuple[DocsMetadata, ...] = (
|
||||
DocsMetadata(DOCS_README_FILENAME, DEFAULT_DOCS_SLUG, "public", "Overview", 0, "技术文档", "Technical Docs"),
|
||||
DocsMetadata("quickstart.md", "quickstart", "public", "Manual", 1, "快速开始", "Quickstart"),
|
||||
DocsMetadata("manual.md", "manual", "public", "Manual", 2, "Planet 使用手册", "Planet Manual"),
|
||||
DocsMetadata("manual.md", "manual", "public", "Manual", 1, "Planet 使用手册", "Planet Manual"),
|
||||
DocsMetadata("quickstart.md", "quickstart", "public", "Manual", 2, "快速开始", "Quickstart"),
|
||||
DocsMetadata("faq.md", "faq", "public", "Manual", 3, "常见问题", "FAQ"),
|
||||
DocsMetadata("location-pipeline-user.md", "location-pipeline-user", "public", "Manual", 4, "Earth 位置候选采集使用手册", "Earth Location Candidate Collection User Guide"),
|
||||
DocsMetadata("platform-data-flows.md", "platform-data-flows", "docs_developer", "Architecture", 5, "业务架构与数据流转", "Business Architecture and Data Flows"),
|
||||
DocsMetadata("naming-glossary.md", "naming-glossary", "docs_developer", "Architecture", 6, "命名与术语对照", "Naming Glossary"),
|
||||
DocsMetadata("earth-frontend-context.md", "earth-frontend-context", "docs_developer", "Earth", 10, "Earth 前端结构", "Earth Frontend Context"),
|
||||
DocsMetadata("earth-layer-style-reference.md", "earth-layer-style-reference", "docs_developer", "Earth", 11, "Earth 图层样式属性索引", "Earth Layer Style Reference"),
|
||||
DocsMetadata("earth-render-layer-order.md", "earth-render-layer-order", "docs_developer", "Earth", 12, "Earth 渲染图层顺序", "Earth Render Layer Order"),
|
||||
DocsMetadata("earth-satellite-footprint-policy.md", "earth-satellite-footprint-policy", "docs_developer", "Earth", 13, "Earth 卫星覆盖策略", "Earth Satellite Footprint Policy"),
|
||||
DocsMetadata("earth-bgp-context.md", "earth-bgp-context", "docs_developer", "Earth", 14, "BGP 态势上下文", "BGP Context"),
|
||||
DocsMetadata("earth-news-live-streams-collector-format.md", "earth-news-live-streams-collector-format", "docs_developer", "Earth", 15, "新闻直播采集格式", "News Live Streams Collector Format"),
|
||||
DocsMetadata("earth-interactable-usage.md", "earth-interactable-usage", "docs_developer", "Earth", 16, "Earth 可交互图标接入", "Earth Interactable Usage"),
|
||||
DocsMetadata("earth-toolbar-overlay-coordination.md", "earth-toolbar-overlay-coordination", "docs_developer", "Earth", 17, "Earth 工具栏与浮层协同", "Earth Toolbar and Overlay Coordination"),
|
||||
DocsMetadata("frontend-admin-frontend-context.md", "frontend-admin-frontend-context", "docs_developer", "Frontend", 20, "控制台前端结构", "Admin Frontend Context"),
|
||||
DocsMetadata("frontend-layout-guidelines.md", "frontend-layout-guidelines", "docs_developer", "Frontend", 21, "前端布局指南", "Frontend Layout Guidelines"),
|
||||
DocsMetadata("docs-gatekeeper-development.md", "docs-gatekeeper-development", "docs_developer", "Frontend", 22, "Docs Gatekeeper 开发说明", "Docs Gatekeeper Development Guide"),
|
||||
DocsMetadata("tactile-ui-components.md", "tactile-ui-components", "docs_developer", "Frontend", 24, "Tactile UI 组件库", "Tactile UI Components"),
|
||||
DocsMetadata("backend-collectors.md", "backend-collectors", "docs_developer", "Backend", 30, "数据采集系统", "Data Collectors"),
|
||||
DocsMetadata("backend-system-service-control.md", "backend-system-service-control", "docs_admin", "Backend", 31, "系统服务控制", "System Service Control"),
|
||||
DocsMetadata("datasource-collector-settings-connectivity.md", "datasource-collector-settings-connectivity", "docs_developer", "Backend", 32, "数据源、采集器设置与连接验证", "Datasource Collector Settings and Connectivity"),
|
||||
DocsMetadata("backend-datasources-api-performance.md", "backend-datasources-api-performance", "docs_developer", "Backend", 33, "数据源 API 性能", "Datasource API Performance"),
|
||||
DocsMetadata("location-pipeline-development.md", "location-pipeline-development", "docs_developer", "Backend", 34, "通用位置估算管线开发说明", "Shared Location Resolution Pipeline Development Guide"),
|
||||
DocsMetadata("data-job-earth-sync-architecture.md", "data-job-earth-sync-architecture", "docs_developer", "Backend", 34, "数据作业与 Outbox 技术架构", "Data Jobs and Outbox Architecture"),
|
||||
DocsMetadata("location-pipeline-development.md", "location-pipeline-development", "docs_developer", "Backend", 35, "通用位置估算管线开发说明", "Shared Location Resolution Pipeline Development Guide"),
|
||||
DocsMetadata("earth-news-live-streams-collector-format.md", "earth-news-live-streams-collector-format", "docs_developer", "Backend", 36, "新闻直播采集格式", "News Live Streams Collector Format"),
|
||||
DocsMetadata("docs-gatekeeper-development.md", "docs-gatekeeper-development", "docs_developer", "Backend", 37, "Docs Gatekeeper 开发说明", "Docs Gatekeeper Development Guide"),
|
||||
DocsMetadata("agents-aiprovider.md", "agents-aiprovider", "docs_developer", "Agents", 40, "AI Provider 指南", "AI Provider Guide"),
|
||||
DocsMetadata("ops-runbook.md", "ops-runbook", "docs_admin", "Ops", 49, "Planet 运维手册", "Planet Ops Runbook"),
|
||||
DocsMetadata("ops-docker-compose-buildx-upgrade.md", "ops-docker-compose-buildx-upgrade", "docs_admin", "Ops", 50, "Docker + Compose + Buildx 升级", "Docker + Compose + Buildx Upgrade"),
|
||||
|
||||
482
backend/app/services/earth_db_change_listener.py
Normal file
482
backend/app/services/earth_db_change_listener.py
Normal file
@@ -0,0 +1,482 @@
|
||||
"""PostgreSQL LISTEN/NOTIFY bridge for Earth layer refresh events."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from collections import deque
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import UTC, datetime
|
||||
from time import monotonic
|
||||
from typing import Any, Awaitable, Callable
|
||||
|
||||
import asyncpg
|
||||
|
||||
from app.core.config import settings
|
||||
from app.core.logging import get_logger
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.core.websocket.broadcaster import broadcaster
|
||||
from app.services.earth_layer_adapters import (
|
||||
get_earth_refresh_strategy_for_change,
|
||||
get_earth_update_layers_for_change,
|
||||
get_earth_update_layers_for_source,
|
||||
)
|
||||
from app.services.earth_layer_cache import invalidate_earth_layer_cache_for_source
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
EARTH_DATA_CHANGES_CHANNEL = "planet_earth_data_changes"
|
||||
DEFAULT_DEBOUNCE_SECONDS = 0.25
|
||||
DEFAULT_MAX_WAIT_SECONDS = 1.5
|
||||
DELETE_FAST_FLUSH_SECONDS = 0.05
|
||||
LISTEN_KEEPALIVE_SECONDS = 5.0
|
||||
OUTBOX_POLL_LIMIT = 5000
|
||||
MAX_ENTITY_KEY_SAMPLES = 20
|
||||
MAX_SEEN_EVENT_IDS = 20000
|
||||
|
||||
BroadcastFn = Callable[[dict[str, Any]], Awaitable[None]]
|
||||
InvalidateFn = Callable[[str], int]
|
||||
|
||||
|
||||
def normalize_asyncpg_dsn(dsn: str) -> str:
|
||||
"""Convert SQLAlchemy asyncpg URLs into asyncpg-compatible URLs."""
|
||||
return dsn.replace("postgresql+asyncpg://", "postgresql://", 1)
|
||||
|
||||
|
||||
def build_earth_update_from_db_payload(payload: dict[str, Any]) -> dict[str, Any] | None:
|
||||
table = payload.get("table")
|
||||
source = str(payload.get("source") or "").strip()
|
||||
table_name = str(table or "").strip()
|
||||
if not source and not table_name:
|
||||
return None
|
||||
layers = get_earth_update_layers_for_change(table_name, source)
|
||||
if not layers:
|
||||
return None
|
||||
refresh_strategy = get_earth_refresh_strategy_for_change(table_name, source) or "clear_then_reload"
|
||||
source_has_adapter = bool(get_earth_update_layers_for_source(source))
|
||||
effective_source = source if source_has_adapter else (table_name if table_name else source)
|
||||
operation = payload.get("operation")
|
||||
update: dict[str, Any] = {
|
||||
"event": "earth.layer.changed",
|
||||
"action": "database_changed",
|
||||
"source": effective_source,
|
||||
"original_source": source or None,
|
||||
"table": table_name or None,
|
||||
"data_type": effective_source,
|
||||
"layers": layers,
|
||||
"refresh_strategy": refresh_strategy,
|
||||
"operation": operation,
|
||||
"entity_key": payload.get("entity_key"),
|
||||
"timestamp": to_iso8601_utc(datetime.now(UTC)),
|
||||
}
|
||||
if table_name == "earth_interactables" and refresh_strategy == "delta":
|
||||
ids = payload.get("entity_keys")
|
||||
if not isinstance(ids, list):
|
||||
ids = [payload.get("entity_key")] if payload.get("entity_key") else []
|
||||
update.update(
|
||||
{
|
||||
"entity": "interactable",
|
||||
"action": "deleted" if operation == "DELETE" else "changed",
|
||||
"ids": [str(item) for item in ids if item],
|
||||
"item": None,
|
||||
}
|
||||
)
|
||||
return {
|
||||
**update,
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class PendingEarthDbChange:
|
||||
source: str
|
||||
layers: list[str]
|
||||
table: str | None = None
|
||||
refresh_strategy: str = "clear_then_reload"
|
||||
entity: str | None = None
|
||||
action: str = "database_changed"
|
||||
records_processed: int = 0
|
||||
operations: set[str] = field(default_factory=set)
|
||||
entity_keys: list[str] = field(default_factory=list)
|
||||
first_occurred_at: str | None = None
|
||||
last_occurred_at: str | None = None
|
||||
first_seen_monotonic: float = field(default_factory=monotonic)
|
||||
last_seen_monotonic: float = field(default_factory=monotonic)
|
||||
|
||||
def add(self, payload: dict[str, Any]) -> None:
|
||||
self.last_seen_monotonic = monotonic()
|
||||
records_processed = payload.get("records_processed", 1)
|
||||
try:
|
||||
records_processed = int(records_processed)
|
||||
except (TypeError, ValueError):
|
||||
records_processed = 1
|
||||
self.records_processed += max(records_processed, 1)
|
||||
operation = payload.get("operation")
|
||||
if operation:
|
||||
self.operations.add(str(operation))
|
||||
entity_keys = payload.get("entity_keys")
|
||||
if not isinstance(entity_keys, list):
|
||||
entity_key = payload.get("entity_key")
|
||||
entity_keys = [entity_key] if entity_key else []
|
||||
for entity_key in entity_keys:
|
||||
if entity_key and len(self.entity_keys) < MAX_ENTITY_KEY_SAMPLES:
|
||||
self.entity_keys.append(str(entity_key))
|
||||
occurred_at = payload.get("occurred_at")
|
||||
if occurred_at:
|
||||
occurred_at = str(occurred_at)
|
||||
self.first_occurred_at = self.first_occurred_at or occurred_at
|
||||
self.last_occurred_at = occurred_at
|
||||
|
||||
|
||||
class EarthDbChangeDispatcher:
|
||||
"""Debounces database notifications and broadcasts Earth refresh hints."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
broadcast_earth_update: BroadcastFn | None = None,
|
||||
invalidate_cache: InvalidateFn | None = None,
|
||||
debounce_seconds: float = DEFAULT_DEBOUNCE_SECONDS,
|
||||
max_wait_seconds: float = DEFAULT_MAX_WAIT_SECONDS,
|
||||
) -> None:
|
||||
self._broadcast_earth_update = broadcast_earth_update or broadcaster.broadcast_earth_update
|
||||
self._invalidate_cache = invalidate_cache or invalidate_earth_layer_cache_for_source
|
||||
self._debounce_seconds = debounce_seconds
|
||||
self._max_wait_seconds = max(max_wait_seconds, debounce_seconds)
|
||||
self._pending: dict[str, PendingEarthDbChange] = {}
|
||||
self._flush_tasks: dict[str, asyncio.Task[None]] = {}
|
||||
self._seen_event_ids: set[int] = set()
|
||||
self._seen_event_order: deque[int] = deque()
|
||||
|
||||
def handle_notification(self, payload_text: str) -> bool:
|
||||
try:
|
||||
payload = json.loads(payload_text)
|
||||
except json.JSONDecodeError:
|
||||
logger.warning_event(
|
||||
"Ignoring malformed Earth database change notification",
|
||||
event="earth.db_changes.notification_malformed",
|
||||
)
|
||||
return False
|
||||
if not isinstance(payload, dict):
|
||||
return False
|
||||
return self.handle_payload(payload)
|
||||
|
||||
def handle_payload(self, payload: dict[str, Any]) -> bool:
|
||||
event_id = payload.get("event_id")
|
||||
if event_id is not None:
|
||||
try:
|
||||
normalized_event_id = int(event_id)
|
||||
except (TypeError, ValueError):
|
||||
normalized_event_id = None
|
||||
if normalized_event_id is not None:
|
||||
if normalized_event_id in self._seen_event_ids:
|
||||
return False
|
||||
self._remember_event_id(normalized_event_id)
|
||||
|
||||
update = build_earth_update_from_db_payload(payload)
|
||||
if not update:
|
||||
return False
|
||||
|
||||
source = update["source"]
|
||||
pending = self._pending.get(source)
|
||||
if pending is None:
|
||||
pending = PendingEarthDbChange(
|
||||
source=source,
|
||||
layers=list(update["layers"]),
|
||||
table=update.get("table"),
|
||||
refresh_strategy=str(update.get("refresh_strategy") or "clear_then_reload"),
|
||||
entity=update.get("entity"),
|
||||
action=str(update.get("action") or "database_changed"),
|
||||
)
|
||||
self._pending[source] = pending
|
||||
pending.add(payload)
|
||||
|
||||
task = self._flush_tasks.pop(source, None)
|
||||
if task and not task.done():
|
||||
task.cancel()
|
||||
self._flush_tasks[source] = asyncio.create_task(
|
||||
self._flush_later(source, delay_seconds=self._next_flush_delay(pending))
|
||||
)
|
||||
return True
|
||||
|
||||
def _next_flush_delay(self, pending: PendingEarthDbChange) -> float:
|
||||
if "DELETE" in pending.operations and pending.refresh_strategy == "clear_then_reload":
|
||||
return DELETE_FAST_FLUSH_SECONDS
|
||||
elapsed = max(0.0, monotonic() - pending.first_seen_monotonic)
|
||||
remaining = self._max_wait_seconds - elapsed
|
||||
if remaining <= 0:
|
||||
return 0.0
|
||||
return min(self._debounce_seconds, remaining)
|
||||
|
||||
def _remember_event_id(self, event_id: int) -> None:
|
||||
self._seen_event_ids.add(event_id)
|
||||
self._seen_event_order.append(event_id)
|
||||
while len(self._seen_event_order) > MAX_SEEN_EVENT_IDS:
|
||||
expired_event_id = self._seen_event_order.popleft()
|
||||
self._seen_event_ids.discard(expired_event_id)
|
||||
|
||||
async def _flush_later(self, source: str, *, delay_seconds: float) -> None:
|
||||
try:
|
||||
if delay_seconds > 0:
|
||||
await asyncio.sleep(delay_seconds)
|
||||
await self.flush_source(source)
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception as exc:
|
||||
logger.exception_event(
|
||||
"Failed to broadcast debounced Earth database change",
|
||||
event="earth.db_changes.flush_failed",
|
||||
context={"source": source, "error": str(exc)},
|
||||
)
|
||||
finally:
|
||||
current = self._flush_tasks.get(source)
|
||||
if current is asyncio.current_task():
|
||||
self._flush_tasks.pop(source, None)
|
||||
|
||||
async def flush_source(self, source: str) -> None:
|
||||
pending = self._pending.get(source)
|
||||
if pending is None:
|
||||
return
|
||||
|
||||
flushed_at = datetime.now(UTC)
|
||||
deleted_cache_entries = self._invalidate_cache(source)
|
||||
operations = sorted(pending.operations)
|
||||
payload: dict[str, Any] = {
|
||||
"event": "earth.layer.changed",
|
||||
"action": "database_changed",
|
||||
"source": source,
|
||||
"table": pending.table,
|
||||
"data_type": source,
|
||||
"layers": pending.layers,
|
||||
"refresh_strategy": pending.refresh_strategy,
|
||||
"records_processed": pending.records_processed,
|
||||
"operations": operations,
|
||||
"operation": operations[-1] if len(operations) == 1 else None,
|
||||
"entity_keys": pending.entity_keys,
|
||||
"entity_key_sample_size": len(pending.entity_keys),
|
||||
"cache_entries_invalidated": deleted_cache_entries,
|
||||
"first_occurred_at": pending.first_occurred_at,
|
||||
"last_occurred_at": pending.last_occurred_at,
|
||||
"debounce_ms": int((monotonic() - pending.first_seen_monotonic) * 1000),
|
||||
"total_latency_ms": self._total_latency_ms(pending, flushed_at),
|
||||
"timestamp": to_iso8601_utc(flushed_at),
|
||||
}
|
||||
if pending.entity == "interactable":
|
||||
payload.update(
|
||||
{
|
||||
"entity": "interactable",
|
||||
"action": "deleted" if "DELETE" in pending.operations else "changed",
|
||||
"ids": pending.entity_keys,
|
||||
"item": None,
|
||||
}
|
||||
)
|
||||
await self._broadcast_earth_update(payload)
|
||||
self._pending.pop(source, None)
|
||||
logger.info_event(
|
||||
"Broadcasted Earth database change",
|
||||
event="earth.db_changes.broadcasted",
|
||||
context={
|
||||
"source": source,
|
||||
"layers": pending.layers,
|
||||
"records_processed": pending.records_processed,
|
||||
"cache_entries_invalidated": deleted_cache_entries,
|
||||
"debounce_ms": int((monotonic() - pending.first_seen_monotonic) * 1000),
|
||||
"total_latency_ms": payload["total_latency_ms"],
|
||||
},
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _total_latency_ms(pending: PendingEarthDbChange, flushed_at: datetime) -> int | None:
|
||||
occurred_at = pending.first_occurred_at
|
||||
if not occurred_at:
|
||||
return None
|
||||
try:
|
||||
normalized = occurred_at.replace("Z", "+00:00")
|
||||
occurred = datetime.fromisoformat(normalized)
|
||||
if occurred.tzinfo is None:
|
||||
occurred = occurred.replace(tzinfo=UTC)
|
||||
return max(0, int((flushed_at - occurred.astimezone(UTC)).total_seconds() * 1000))
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
async def flush_all(self) -> None:
|
||||
sources = list(self._pending)
|
||||
for source in sources:
|
||||
task = self._flush_tasks.pop(source, None)
|
||||
if task and not task.done():
|
||||
task.cancel()
|
||||
await self.flush_source(source)
|
||||
|
||||
async def stop(self) -> None:
|
||||
tasks = [task for task in self._flush_tasks.values() if not task.done()]
|
||||
for task in tasks:
|
||||
task.cancel()
|
||||
if tasks:
|
||||
await asyncio.gather(*tasks, return_exceptions=True)
|
||||
self._flush_tasks.clear()
|
||||
await self.flush_all()
|
||||
|
||||
|
||||
class EarthDbChangeListener:
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
dsn: str,
|
||||
dispatcher: EarthDbChangeDispatcher,
|
||||
channel: str = EARTH_DATA_CHANGES_CHANNEL,
|
||||
) -> None:
|
||||
self._dsn = normalize_asyncpg_dsn(dsn)
|
||||
self._dispatcher = dispatcher
|
||||
self._channel = channel
|
||||
self._task: asyncio.Task[None] | None = None
|
||||
self._stop_event: asyncio.Event | None = None
|
||||
self._connection: asyncpg.Connection | None = None
|
||||
self._loop: asyncio.AbstractEventLoop | None = None
|
||||
|
||||
def start(self) -> None:
|
||||
if self._task and not self._task.done():
|
||||
return
|
||||
self._loop = asyncio.get_running_loop()
|
||||
self._stop_event = asyncio.Event()
|
||||
self._task = asyncio.create_task(self._run())
|
||||
|
||||
async def stop(self) -> None:
|
||||
if self._stop_event:
|
||||
self._stop_event.set()
|
||||
if self._connection:
|
||||
await self._connection.close()
|
||||
if self._task:
|
||||
await asyncio.gather(self._task, return_exceptions=True)
|
||||
await self._dispatcher.stop()
|
||||
|
||||
async def _run(self) -> None:
|
||||
backoff_seconds = 1.0
|
||||
assert self._stop_event is not None
|
||||
|
||||
while not self._stop_event.is_set():
|
||||
try:
|
||||
self._connection = await asyncpg.connect(self._dsn)
|
||||
await self._connection.add_listener(self._channel, self._on_notification)
|
||||
logger.info_event(
|
||||
"Earth database change listener connected",
|
||||
event="earth.db_changes.connected",
|
||||
context={"channel": self._channel},
|
||||
)
|
||||
backoff_seconds = 1.0
|
||||
while not self._stop_event.is_set():
|
||||
try:
|
||||
await asyncio.wait_for(
|
||||
self._stop_event.wait(),
|
||||
timeout=LISTEN_KEEPALIVE_SECONDS,
|
||||
)
|
||||
except asyncio.TimeoutError:
|
||||
await self._poll_outbox()
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception as exc:
|
||||
logger.exception_event(
|
||||
"Earth database change listener failed",
|
||||
event="earth.db_changes.listener_failed",
|
||||
context={"channel": self._channel, "error": str(exc)},
|
||||
)
|
||||
try:
|
||||
await asyncio.wait_for(self._stop_event.wait(), timeout=backoff_seconds)
|
||||
except asyncio.TimeoutError:
|
||||
pass
|
||||
backoff_seconds = min(backoff_seconds * 2, 30.0)
|
||||
finally:
|
||||
if self._connection:
|
||||
try:
|
||||
await self._connection.remove_listener(self._channel, self._on_notification)
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
await self._connection.close()
|
||||
except Exception:
|
||||
pass
|
||||
self._connection = None
|
||||
|
||||
async def _poll_outbox(self) -> None:
|
||||
if self._connection is None:
|
||||
return
|
||||
|
||||
rows = await self._connection.fetch(
|
||||
"""
|
||||
SELECT id, payload
|
||||
FROM earth_data_change_events
|
||||
WHERE consumed_at IS NULL
|
||||
ORDER BY id
|
||||
LIMIT $1
|
||||
""",
|
||||
OUTBOX_POLL_LIMIT,
|
||||
)
|
||||
accepted_count = 0
|
||||
consumed_ids: list[int] = []
|
||||
for row in rows:
|
||||
payload = row["payload"]
|
||||
if isinstance(payload, str):
|
||||
try:
|
||||
payload = json.loads(payload)
|
||||
except json.JSONDecodeError:
|
||||
consumed_ids.append(int(row["id"]))
|
||||
continue
|
||||
if isinstance(payload, dict):
|
||||
if self._dispatcher.handle_payload(payload):
|
||||
accepted_count += 1
|
||||
consumed_ids.append(int(row["id"]))
|
||||
else:
|
||||
consumed_ids.append(int(row["id"]))
|
||||
if consumed_ids:
|
||||
await self._dispatcher.flush_all()
|
||||
if consumed_ids:
|
||||
await self._connection.execute(
|
||||
"""
|
||||
UPDATE earth_data_change_events
|
||||
SET consumed_at = NOW()
|
||||
WHERE id = ANY($1::bigint[])
|
||||
AND consumed_at IS NULL
|
||||
""",
|
||||
consumed_ids,
|
||||
)
|
||||
if rows:
|
||||
logger.info_event(
|
||||
"Polled Earth database change outbox",
|
||||
event="earth.db_changes.outbox_polled",
|
||||
context={"events": len(rows), "accepted": accepted_count},
|
||||
)
|
||||
|
||||
def _on_notification(
|
||||
self,
|
||||
_connection: asyncpg.Connection,
|
||||
_pid: int,
|
||||
_channel: str,
|
||||
payload: str,
|
||||
) -> None:
|
||||
if self._loop and self._loop.is_running():
|
||||
self._loop.call_soon_threadsafe(self._dispatcher.handle_notification, payload)
|
||||
return
|
||||
self._dispatcher.handle_notification(payload)
|
||||
|
||||
|
||||
_dispatcher = EarthDbChangeDispatcher(
|
||||
broadcast_earth_update=broadcaster.broadcast_earth_update,
|
||||
invalidate_cache=invalidate_earth_layer_cache_for_source,
|
||||
)
|
||||
_listener: EarthDbChangeListener | None = None
|
||||
|
||||
|
||||
def start_earth_db_change_listener() -> None:
|
||||
global _listener
|
||||
if _listener is not None:
|
||||
return
|
||||
_listener = EarthDbChangeListener(dsn=settings.DATABASE_URL, dispatcher=_dispatcher)
|
||||
_listener.start()
|
||||
|
||||
|
||||
async def stop_earth_db_change_listener() -> None:
|
||||
global _listener
|
||||
if _listener is None:
|
||||
await _dispatcher.stop()
|
||||
return
|
||||
listener = _listener
|
||||
_listener = None
|
||||
await listener.stop()
|
||||
113
backend/app/services/earth_interactables.py
Normal file
113
backend/app/services/earth_interactables.py
Normal file
@@ -0,0 +1,113 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
from uuid import uuid4
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.core.websocket.broadcaster import broadcaster
|
||||
from app.models.earth_interactable import EarthInteractable
|
||||
from app.services.earth_layer_cache import EARTH_LAYER_CACHE_PREFIX, earth_layer_cache
|
||||
|
||||
INTERACTABLE_ENTITY = "interactable"
|
||||
INTERACTABLE_LAYER = "interactables"
|
||||
|
||||
|
||||
def normalize_interactable_id(value: str | None = None) -> str:
|
||||
raw = str(value or "").strip()
|
||||
return raw or f"interactable-{uuid4().hex}"
|
||||
|
||||
|
||||
def serialize_interactable(record: EarthInteractable) -> dict[str, Any]:
|
||||
return {
|
||||
"id": record.id,
|
||||
"layer": record.layer,
|
||||
"kind": record.kind,
|
||||
"label": record.label,
|
||||
"description": record.description,
|
||||
"latitude": record.latitude,
|
||||
"longitude": record.longitude,
|
||||
"altitude": record.altitude,
|
||||
"revision": record.revision,
|
||||
"properties": record.properties or {},
|
||||
"is_deleted": bool(record.is_deleted),
|
||||
"created_at": to_iso8601_utc(record.created_at),
|
||||
"updated_at": to_iso8601_utc(record.updated_at),
|
||||
"deleted_at": to_iso8601_utc(record.deleted_at),
|
||||
}
|
||||
|
||||
|
||||
def interactables_to_geojson(items: list[EarthInteractable]) -> dict[str, Any]:
|
||||
return {
|
||||
"type": "FeatureCollection",
|
||||
"features": [
|
||||
{
|
||||
"type": "Feature",
|
||||
"id": item.id,
|
||||
"geometry": {
|
||||
"type": "Point",
|
||||
"coordinates": [item.longitude, item.latitude],
|
||||
},
|
||||
"properties": serialize_interactable(item),
|
||||
}
|
||||
for item in items
|
||||
if not item.is_deleted
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def invalidate_interactable_cache(layer: str | None = None) -> int:
|
||||
layer_key = str(layer or "*").strip() or "*"
|
||||
deleted = earth_layer_cache.delete_pattern(
|
||||
f"{EARTH_LAYER_CACHE_PREFIX}:interactables:layer:{layer_key}*"
|
||||
)
|
||||
if layer_key != "all":
|
||||
deleted += earth_layer_cache.delete_pattern(
|
||||
f"{EARTH_LAYER_CACHE_PREFIX}:interactables:layer:all*"
|
||||
)
|
||||
return deleted
|
||||
|
||||
|
||||
def build_interactable_event(
|
||||
*,
|
||||
action: str,
|
||||
record: EarthInteractable,
|
||||
include_item: bool = True,
|
||||
) -> dict[str, Any]:
|
||||
item = serialize_interactable(record)
|
||||
return {
|
||||
"entity": INTERACTABLE_ENTITY,
|
||||
"action": action,
|
||||
"layer": record.layer,
|
||||
"layers": [INTERACTABLE_LAYER],
|
||||
"ids": [record.id],
|
||||
"revision": record.revision,
|
||||
"changed_at": item["deleted_at"] or item["updated_at"] or to_iso8601_utc(datetime.now(UTC)),
|
||||
"item": item if include_item else None,
|
||||
"source": "earth_interactables",
|
||||
}
|
||||
|
||||
|
||||
async def publish_interactable_event(action: str, record: EarthInteractable, *, include_item: bool = True) -> None:
|
||||
await broadcaster.broadcast_earth_update(
|
||||
build_interactable_event(action=action, record=record, include_item=include_item)
|
||||
)
|
||||
|
||||
|
||||
async def list_interactables(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
layer: str | None = None,
|
||||
include_deleted: bool = False,
|
||||
) -> list[EarthInteractable]:
|
||||
stmt = select(EarthInteractable)
|
||||
if layer:
|
||||
stmt = stmt.where(EarthInteractable.layer == layer)
|
||||
if not include_deleted:
|
||||
stmt = stmt.where(EarthInteractable.is_deleted.is_(False))
|
||||
stmt = stmt.order_by(EarthInteractable.updated_at.desc(), EarthInteractable.id.asc())
|
||||
result = await db.execute(stmt)
|
||||
return list(result.scalars().all())
|
||||
179
backend/app/services/earth_layer_adapters.py
Normal file
179
backend/app/services/earth_layer_adapters.py
Normal file
@@ -0,0 +1,179 @@
|
||||
"""Earth layer adapter registry for datasource-backed refresh behavior."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class EarthLayerAdapter:
|
||||
sources: frozenset[str]
|
||||
layers: tuple[str, ...]
|
||||
cache_patterns: tuple[str, ...]
|
||||
tables: frozenset[str] = field(default_factory=frozenset)
|
||||
derived_models: tuple[str, ...] = field(default_factory=tuple)
|
||||
refresh_strategy: str = "clear_then_reload"
|
||||
|
||||
|
||||
EARTH_LAYER_ADAPTERS: tuple[EarthLayerAdapter, ...] = (
|
||||
EarthLayerAdapter(
|
||||
sources=frozenset({"barentswatch_vessels", "aisstream_vessels", "vessel_static", "vessel_position", "ais_raw_observations", "ais_source_health"}),
|
||||
tables=frozenset({"vessel_static", "vessel_position", "ais_raw_observations", "ais_source_health"}),
|
||||
layers=("vessels",),
|
||||
cache_patterns=("vessels*", "summary*"),
|
||||
),
|
||||
EarthLayerAdapter(
|
||||
sources=frozenset(
|
||||
{
|
||||
"telegeography_cables",
|
||||
"telegeography_landing",
|
||||
"telegeography_landing_points",
|
||||
"telegeography_systems",
|
||||
"telegeography_cable_systems",
|
||||
"arcgis_cables",
|
||||
"arcgis_landing_points",
|
||||
"arcgis_cable_landing_relation",
|
||||
"arcgis_cable_landing_relations",
|
||||
"fao_landing_points",
|
||||
}
|
||||
),
|
||||
tables=frozenset({"collected_data"}),
|
||||
layers=("cables",),
|
||||
cache_patterns=("cables*", "landing-points*", "summary*"),
|
||||
),
|
||||
EarthLayerAdapter(
|
||||
sources=frozenset({"celestrak_tle", "spacetrack_tle"}),
|
||||
tables=frozenset({"collected_data"}),
|
||||
layers=("satellites",),
|
||||
cache_patterns=("satellites*", "summary*"),
|
||||
),
|
||||
EarthLayerAdapter(
|
||||
sources=frozenset(
|
||||
{
|
||||
"top500",
|
||||
"top500_supercomputers",
|
||||
"epoch_ai_gpu",
|
||||
"huggingface_models",
|
||||
"huggingface_datasets",
|
||||
"huggingface_spaces",
|
||||
"compute_center_locations",
|
||||
}
|
||||
),
|
||||
tables=frozenset({"compute_center_locations"}),
|
||||
layers=("computeCenters",),
|
||||
cache_patterns=("compute-centers*", "summary*"),
|
||||
refresh_strategy="reload",
|
||||
),
|
||||
EarthLayerAdapter(
|
||||
sources=frozenset(
|
||||
{
|
||||
"ris_live_bgp",
|
||||
"bgpstream_bgp",
|
||||
"iptoasn_prefix_geo",
|
||||
"opengeofeed_prefix_geo",
|
||||
"nro_delegated_prefix_geo",
|
||||
"bgp_observations",
|
||||
"bgp_anomalies",
|
||||
"bgp_incidents",
|
||||
"bgp_collector_locations",
|
||||
}
|
||||
),
|
||||
tables=frozenset({"bgp_observations", "bgp_anomalies", "bgp_incidents", "bgp_collector_locations"}),
|
||||
layers=("bgp",),
|
||||
cache_patterns=("bgp*", "summary*"),
|
||||
derived_models=("bgp_observations", "bgp_anomalies", "bgp_incidents"),
|
||||
),
|
||||
EarthLayerAdapter(
|
||||
sources=frozenset({"news_live_streams"}),
|
||||
tables=frozenset({"collected_data"}),
|
||||
layers=("media",),
|
||||
cache_patterns=("summary*",),
|
||||
refresh_strategy="reload",
|
||||
),
|
||||
EarthLayerAdapter(
|
||||
sources=frozenset({"media_news_archive", "earth_news_items"}),
|
||||
tables=frozenset({"earth_news_items"}),
|
||||
layers=("news",),
|
||||
cache_patterns=("summary*",),
|
||||
refresh_strategy="reload",
|
||||
),
|
||||
EarthLayerAdapter(
|
||||
sources=frozenset({"earth_interactables"}),
|
||||
tables=frozenset({"earth_interactables"}),
|
||||
layers=("interactables",),
|
||||
cache_patterns=("interactables*", "summary*"),
|
||||
refresh_strategy="delta",
|
||||
),
|
||||
)
|
||||
|
||||
_ADAPTERS_BY_SOURCE = {
|
||||
source: adapter
|
||||
for adapter in EARTH_LAYER_ADAPTERS
|
||||
for source in adapter.sources
|
||||
}
|
||||
_ADAPTERS_BY_TABLE = {
|
||||
table: adapter
|
||||
for adapter in EARTH_LAYER_ADAPTERS
|
||||
for table in adapter.tables
|
||||
}
|
||||
|
||||
|
||||
def get_earth_layer_adapter_for_source(source: str | None) -> EarthLayerAdapter | None:
|
||||
return _ADAPTERS_BY_SOURCE.get(str(source or "").strip())
|
||||
|
||||
|
||||
def get_earth_layer_adapter_for_change(table: str | None, source: str | None) -> EarthLayerAdapter | None:
|
||||
table_key = str(table or "").strip()
|
||||
source_key = str(source or "").strip()
|
||||
if table_key and table_key != "collected_data":
|
||||
adapter = _ADAPTERS_BY_TABLE.get(table_key)
|
||||
if adapter is not None:
|
||||
return adapter
|
||||
return get_earth_layer_adapter_for_source(source_key)
|
||||
|
||||
|
||||
def get_earth_update_layers_for_source(source: str | None) -> list[str]:
|
||||
adapter = get_earth_layer_adapter_for_source(source)
|
||||
return list(adapter.layers) if adapter else []
|
||||
|
||||
|
||||
def get_earth_update_layers_for_change(table: str | None, source: str | None) -> list[str]:
|
||||
adapter = get_earth_layer_adapter_for_change(table, source)
|
||||
return list(adapter.layers) if adapter else []
|
||||
|
||||
|
||||
def get_earth_refresh_strategy_for_change(table: str | None, source: str | None) -> str | None:
|
||||
adapter = get_earth_layer_adapter_for_change(table, source)
|
||||
return adapter.refresh_strategy if adapter else None
|
||||
|
||||
|
||||
def get_earth_cache_patterns_for_source(source: str | None) -> list[str]:
|
||||
adapter = get_earth_layer_adapter_for_source(source)
|
||||
return list(adapter.cache_patterns) if adapter else []
|
||||
|
||||
|
||||
async def clear_derived_datasource_data(db: AsyncSession, source: str) -> dict[str, int]:
|
||||
adapter = get_earth_layer_adapter_for_source(source)
|
||||
if adapter is None or not adapter.derived_models:
|
||||
return {}
|
||||
|
||||
from app.models.bgp_anomaly import BGPAnomaly
|
||||
from app.models.bgp_incident import BGPIncident
|
||||
from app.models.bgp_observation import BGPObservation
|
||||
|
||||
model_by_key: dict[str, Any] = {
|
||||
"bgp_observations": BGPObservation,
|
||||
"bgp_anomalies": BGPAnomaly,
|
||||
"bgp_incidents": BGPIncident,
|
||||
}
|
||||
deleted_counts: dict[str, int] = {}
|
||||
for key in adapter.derived_models:
|
||||
model = model_by_key.get(key)
|
||||
if model is None:
|
||||
continue
|
||||
result = await db.execute(model.__table__.delete().where(model.source == source))
|
||||
deleted_counts[key] = int(result.rowcount or 0)
|
||||
return deleted_counts
|
||||
@@ -268,34 +268,10 @@ def apply_payload_budget(payload: dict[str, Any], policy: EarthLayerCachePolicy)
|
||||
|
||||
|
||||
def invalidate_earth_layer_cache_for_source(source: str) -> int:
|
||||
from app.services.earth_layer_adapters import get_earth_cache_patterns_for_source
|
||||
|
||||
source_key = str(source or "").strip()
|
||||
patterns = {
|
||||
"barentswatch_vessels": ["vessels*", "summary*"],
|
||||
"aisstream_vessels": ["vessels*", "summary*"],
|
||||
"telegeography_cables": ["cables*", "landing-points*", "summary*"],
|
||||
"telegeography_landing": ["landing-points*", "summary*"],
|
||||
"telegeography_landing_points": ["landing-points*", "summary*"],
|
||||
"telegeography_systems": ["cables*", "summary*"],
|
||||
"telegeography_cable_systems": ["cables*", "summary*"],
|
||||
"arcgis_cables": ["cables*", "landing-points*", "summary*"],
|
||||
"arcgis_landing_points": ["landing-points*", "summary*"],
|
||||
"arcgis_cable_landing_relation": ["landing-points*", "summary*"],
|
||||
"arcgis_cable_landing_relations": ["landing-points*", "summary*"],
|
||||
"fao_landing_points": ["landing-points*", "summary*"],
|
||||
"celestrak_tle": ["satellites*", "summary*"],
|
||||
"spacetrack_tle": ["satellites*", "summary*"],
|
||||
"top500": ["compute-centers*", "summary*"],
|
||||
"top500_supercomputers": ["compute-centers*", "summary*"],
|
||||
"epoch_ai_gpu": ["compute-centers*", "summary*"],
|
||||
"huggingface_models": ["compute-centers*", "summary*"],
|
||||
"huggingface_datasets": ["compute-centers*", "summary*"],
|
||||
"huggingface_spaces": ["compute-centers*", "summary*"],
|
||||
"ris_live_bgp": ["bgp*", "summary*"],
|
||||
"bgpstream_bgp": ["bgp*", "summary*"],
|
||||
"iptoasn_prefix_geo": ["bgp*", "summary*"],
|
||||
"opengeofeed_prefix_geo": ["bgp*", "summary*"],
|
||||
"nro_delegated_prefix_geo": ["bgp*", "summary*"],
|
||||
}.get(source_key, [])
|
||||
patterns = get_earth_cache_patterns_for_source(source_key)
|
||||
deleted = 0
|
||||
for layer_pattern in patterns:
|
||||
deleted += earth_layer_cache.delete_pattern(f"{EARTH_LAYER_CACHE_PREFIX}:{layer_pattern}")
|
||||
|
||||
@@ -1063,6 +1063,7 @@ def _build_payload(
|
||||
lon: float | None,
|
||||
active_region: str,
|
||||
items: list[ParsedNewsItem],
|
||||
cruise_items: list[ParsedNewsItem] | None = None,
|
||||
sources: list[NewsFeedSource],
|
||||
errors: list[str],
|
||||
stale: bool,
|
||||
@@ -1082,6 +1083,10 @@ def _build_payload(
|
||||
},
|
||||
"sources": _serialize_sources(sources),
|
||||
"items": [_serialize_item(item, active_region=active_region) for item in items],
|
||||
"cruise_items": [
|
||||
_serialize_item(item, active_region=active_region)
|
||||
for item in (cruise_items if cruise_items is not None else items)
|
||||
],
|
||||
"errors": errors,
|
||||
"stale": stale,
|
||||
}
|
||||
@@ -1295,6 +1300,7 @@ async def get_earth_news_payload(
|
||||
|
||||
from app.services.earth_news_store import (
|
||||
get_earth_news_freshness,
|
||||
list_earth_news_cruise_items,
|
||||
list_earth_news_items,
|
||||
upsert_earth_news_items,
|
||||
)
|
||||
@@ -1312,6 +1318,13 @@ async def get_earth_news_payload(
|
||||
active_region=active_region,
|
||||
limit=MAX_ITEMS_TOTAL,
|
||||
)
|
||||
if hasattr(db, "execute"):
|
||||
cruise_items = await list_earth_news_cruise_items(
|
||||
db,
|
||||
limit=MAX_ITEMS_TOTAL * len(REGION_ANCHORS),
|
||||
)
|
||||
else:
|
||||
cruise_items = items
|
||||
await _enqueue_unverified_locations(items)
|
||||
stale = bool(errors and items)
|
||||
|
||||
@@ -1320,6 +1333,7 @@ async def get_earth_news_payload(
|
||||
lon=lon,
|
||||
active_region=active_region,
|
||||
items=items,
|
||||
cruise_items=cruise_items,
|
||||
sources=sources,
|
||||
errors=errors,
|
||||
stale=stale,
|
||||
|
||||
@@ -78,6 +78,24 @@ async def list_earth_news_items(
|
||||
return [record_to_parsed_news_item(record) for record in result.scalars().all()]
|
||||
|
||||
|
||||
async def list_earth_news_cruise_items(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
limit: int,
|
||||
) -> list[ParsedNewsItem]:
|
||||
result = await db.execute(
|
||||
select(EarthNewsItem)
|
||||
.order_by(
|
||||
EarthNewsItem.region.asc(),
|
||||
EarthNewsItem.published_at.desc().nullslast(),
|
||||
EarthNewsItem.last_seen_at.desc(),
|
||||
EarthNewsItem.feed_name.asc(),
|
||||
)
|
||||
.limit(limit)
|
||||
)
|
||||
return [record_to_parsed_news_item(record) for record in result.scalars().all()]
|
||||
|
||||
|
||||
async def get_earth_news_freshness(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
|
||||
@@ -10,6 +10,7 @@ from fastapi import HTTPException, status
|
||||
from sqlalchemy import func, select, update
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.logging import get_logger
|
||||
from app.db.session import async_session_factory
|
||||
from app.models.playground_message import PlaygroundMessage
|
||||
from app.models.playground_session import PlaygroundSession
|
||||
@@ -27,9 +28,11 @@ from app.schemas.ai import (
|
||||
SituationalAnalysisRequest,
|
||||
)
|
||||
from app.services.ai_client import AIProviderClient
|
||||
from app.services.business_logs import emit_business_log, exception_context
|
||||
from app.services.playground_session_store import _to_response as session_to_response
|
||||
from app.services.playground_session_store import upsert_playground_session
|
||||
|
||||
logger = get_logger(__name__, service="ai")
|
||||
STREAM_CHUNK_SIZE = 24
|
||||
STREAM_INTERVAL_SECONDS = 0.08
|
||||
THINKING_PREVIEW_SECONDS = 2.6
|
||||
@@ -624,7 +627,42 @@ async def _run_assistant_message(
|
||||
thinking={"type": "enabled"},
|
||||
)
|
||||
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="ai.playground.run.start",
|
||||
message="Playground AI run started",
|
||||
category="ai",
|
||||
service="ai",
|
||||
module=__name__,
|
||||
request_id=request_id,
|
||||
user_id=user_id,
|
||||
context={
|
||||
"session_id": session_id,
|
||||
"session_key": session_key,
|
||||
"user_message_id": user_message_id,
|
||||
"assistant_message_id": assistant_message_id,
|
||||
"preset": payload.selected_preset_key,
|
||||
},
|
||||
)
|
||||
analysis = await provider_client.analyze(request_payload, request_id=request_id)
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="ai.playground.run.success",
|
||||
message="Playground AI run completed",
|
||||
category="ai",
|
||||
service="ai",
|
||||
module=__name__,
|
||||
request_id=request_id,
|
||||
user_id=user_id,
|
||||
context={
|
||||
"session_id": session_id,
|
||||
"session_key": session_key,
|
||||
"provider": analysis.provider,
|
||||
"model": analysis.model,
|
||||
"content_block_count": len(analysis.content_blocks or []),
|
||||
"thinking_block_count": len(analysis.thinking_blocks or []),
|
||||
},
|
||||
)
|
||||
|
||||
async with async_session_factory() as db:
|
||||
assistant_message = await _mark_message_state(
|
||||
@@ -704,6 +742,23 @@ async def _run_assistant_message(
|
||||
await db.flush()
|
||||
await db.commit()
|
||||
except asyncio.CancelledError:
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="ai.playground.run.cancelled",
|
||||
message="Playground AI run cancelled",
|
||||
category="ai",
|
||||
level="warning",
|
||||
service="ai",
|
||||
module=__name__,
|
||||
request_id=request_id,
|
||||
user_id=user_id,
|
||||
context={
|
||||
"session_id": session_id,
|
||||
"session_key": session_key,
|
||||
"assistant_message_id": assistant_message_id,
|
||||
"duration_ms": round((perf_counter() - started_at) * 1000),
|
||||
},
|
||||
)
|
||||
async with async_session_factory() as db:
|
||||
result = await db.execute(select(PlaygroundMessage).where(PlaygroundMessage.id == assistant_message_id))
|
||||
message = result.scalar_one_or_none()
|
||||
@@ -715,6 +770,26 @@ async def _run_assistant_message(
|
||||
await db.commit()
|
||||
raise
|
||||
except Exception as exc:
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="ai.playground.run.failed",
|
||||
message="Playground AI run failed",
|
||||
category="ai",
|
||||
level="error",
|
||||
service="ai",
|
||||
module=__name__,
|
||||
request_id=request_id,
|
||||
user_id=user_id,
|
||||
context=exception_context(
|
||||
exc,
|
||||
{
|
||||
"session_id": session_id,
|
||||
"session_key": session_key,
|
||||
"assistant_message_id": assistant_message_id,
|
||||
"duration_ms": round((perf_counter() - started_at) * 1000),
|
||||
},
|
||||
),
|
||||
)
|
||||
error_message = _format_run_exception(exc)
|
||||
async with async_session_factory() as db:
|
||||
result = await db.execute(select(PlaygroundMessage).where(PlaygroundMessage.id == assistant_message_id))
|
||||
|
||||
@@ -13,6 +13,7 @@ from app.db.session import async_session_factory
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.models.datasource import DataSource
|
||||
from app.models.task import CollectionTask
|
||||
from app.services.business_logs import emit_business_log, emit_business_log_background, exception_context
|
||||
from app.services.collectors.registry import collector_registry
|
||||
from app.services.datasource_connectivity import (
|
||||
build_builtin_connectivity_checksum,
|
||||
@@ -124,6 +125,15 @@ async def run_collector_task(collector_name: str):
|
||||
event="collector.run.skipped_disabled",
|
||||
context={"collector_name": collector_name},
|
||||
)
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="collector.run.skipped_disabled",
|
||||
message="Skipping disabled collector",
|
||||
category="collector",
|
||||
service="scheduler",
|
||||
module=__name__,
|
||||
context={"collector_name": collector_name, "datasource_id": datasource.id, "status": "skipped"},
|
||||
)
|
||||
return
|
||||
|
||||
running_result = await db.execute(
|
||||
@@ -152,6 +162,21 @@ async def run_collector_task(collector_name: str):
|
||||
event="collector.run.skipped_already_running",
|
||||
context={"collector_name": collector_name, "task_id": existing_running.id},
|
||||
)
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="collector.run.skipped_already_running",
|
||||
message="Skipping collector trigger because task is already running",
|
||||
category="collector",
|
||||
level="warning",
|
||||
service="scheduler",
|
||||
module=__name__,
|
||||
context={
|
||||
"collector_name": collector_name,
|
||||
"datasource_id": datasource.id,
|
||||
"task_id": existing_running.id,
|
||||
"status": "skipped",
|
||||
},
|
||||
)
|
||||
return
|
||||
|
||||
existing_error = (existing_running.error_message or "").strip()
|
||||
@@ -173,6 +198,21 @@ async def run_collector_task(collector_name: str):
|
||||
event="collector.run.stale_task_failed",
|
||||
context={"collector_name": collector_name, "task_id": existing_running.id},
|
||||
)
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="collector.run.stale_task_failed",
|
||||
message="Marked stale running task as failed before rerun",
|
||||
category="collector",
|
||||
level="warning",
|
||||
service="scheduler",
|
||||
module=__name__,
|
||||
context={
|
||||
"collector_name": collector_name,
|
||||
"datasource_id": datasource.id,
|
||||
"task_id": existing_running.id,
|
||||
"status": "failed",
|
||||
},
|
||||
)
|
||||
|
||||
try:
|
||||
datasource_id = datasource.id
|
||||
@@ -183,6 +223,15 @@ async def run_collector_task(collector_name: str):
|
||||
event="collector.run.started",
|
||||
context={"collector_name": collector_name, "datasource_id": datasource_id},
|
||||
)
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="collector.run.scheduled_started",
|
||||
message="Scheduler started collector run",
|
||||
category="collector",
|
||||
service="scheduler",
|
||||
module=__name__,
|
||||
context={"collector_name": collector_name, "datasource_id": datasource_id, "status": "running"},
|
||||
)
|
||||
task_result = await collector.run(db)
|
||||
datasource = await db.get(DataSource, datasource_id)
|
||||
if datasource is None:
|
||||
@@ -217,6 +266,20 @@ async def run_collector_task(collector_name: str):
|
||||
event="collector.run.completed",
|
||||
context={"collector_name": collector_name, "datasource_id": datasource_id, "result": task_result},
|
||||
)
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="collector.run.scheduled_completed",
|
||||
message="Scheduler completed collector run",
|
||||
category="collector",
|
||||
service="scheduler",
|
||||
module=__name__,
|
||||
context={
|
||||
"collector_name": collector_name,
|
||||
"datasource_id": datasource_id,
|
||||
"status": task_result.get("status"),
|
||||
"result": task_result,
|
||||
},
|
||||
)
|
||||
except asyncio.CancelledError:
|
||||
await db.rollback()
|
||||
datasource = await db.get(DataSource, datasource_id)
|
||||
@@ -228,6 +291,16 @@ async def run_collector_task(collector_name: str):
|
||||
event="collector.run.cancelled",
|
||||
context={"collector_name": collector_name, "datasource_id": datasource.id},
|
||||
)
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="collector.run.cancelled",
|
||||
message="Collector cancelled by operator",
|
||||
category="collector",
|
||||
level="warning",
|
||||
service="scheduler",
|
||||
module=__name__,
|
||||
context={"collector_name": collector_name, "datasource_id": datasource.id, "status": "cancelled"},
|
||||
)
|
||||
raise
|
||||
except Exception as exc:
|
||||
await db.rollback()
|
||||
@@ -240,6 +313,19 @@ async def run_collector_task(collector_name: str):
|
||||
event="collector.run.failed",
|
||||
context={"collector_name": collector_name, "datasource_id": datasource.id, "error": str(exc)},
|
||||
)
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="collector.run.failed",
|
||||
message="Collector failed",
|
||||
category="collector",
|
||||
level="error",
|
||||
service="scheduler",
|
||||
module=__name__,
|
||||
context=exception_context(
|
||||
exc,
|
||||
{"collector_name": collector_name, "datasource_id": datasource.id, "status": "failed"},
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
async def cleanup_stale_running_tasks(max_age_hours: int = 2) -> int:
|
||||
@@ -361,6 +447,16 @@ def run_collector_now(collector_name: str) -> bool:
|
||||
event="collector.trigger.skipped_already_running",
|
||||
context={"collector_name": collector_name},
|
||||
)
|
||||
emit_business_log_background(
|
||||
logger,
|
||||
event="collector.trigger.skipped_already_running",
|
||||
message="Collector is already running in-memory; skipping duplicate trigger",
|
||||
category="collector",
|
||||
level="warning",
|
||||
service="scheduler",
|
||||
module=__name__,
|
||||
context={"collector_name": collector_name, "status": "skipped"},
|
||||
)
|
||||
return False
|
||||
|
||||
try:
|
||||
@@ -378,6 +474,15 @@ def run_collector_now(collector_name: str) -> bool:
|
||||
event="collector.trigger.started",
|
||||
context={"collector_name": collector_name},
|
||||
)
|
||||
emit_business_log_background(
|
||||
logger,
|
||||
event="collector.trigger.started",
|
||||
message="Triggered collector",
|
||||
category="collector",
|
||||
service="scheduler",
|
||||
module=__name__,
|
||||
context={"collector_name": collector_name, "status": "queued"},
|
||||
)
|
||||
return True
|
||||
except Exception as exc:
|
||||
logger.error_event(
|
||||
@@ -385,6 +490,16 @@ def run_collector_now(collector_name: str) -> bool:
|
||||
event="collector.trigger.failed",
|
||||
context={"collector_name": collector_name, "error": str(exc)},
|
||||
)
|
||||
emit_business_log_background(
|
||||
logger,
|
||||
event="collector.trigger.failed",
|
||||
message="Failed to trigger collector",
|
||||
category="collector",
|
||||
level="error",
|
||||
service="scheduler",
|
||||
module=__name__,
|
||||
context=exception_context(exc, {"collector_name": collector_name, "status": "failed"}),
|
||||
)
|
||||
return False
|
||||
|
||||
|
||||
|
||||
@@ -18,7 +18,7 @@ async def create_admin():
|
||||
|
||||
if existing_user:
|
||||
print("用户 linkong 已存在,更新密码...")
|
||||
existing_user.set_password("12345678")
|
||||
existing_user.set_password("LK12345678")
|
||||
existing_user.role = "super_admin"
|
||||
existing_user.email = "linkong@planet.local"
|
||||
else:
|
||||
@@ -26,7 +26,7 @@ async def create_admin():
|
||||
user = User(
|
||||
username="linkong",
|
||||
email="linkong@planet.local",
|
||||
password_hash=get_password_hash("12345678"),
|
||||
password_hash=get_password_hash("LK12345678"),
|
||||
role="super_admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
@@ -19,7 +19,7 @@ DEFAULT_LOGIN_USERS = (
|
||||
{
|
||||
"username": "linkong",
|
||||
"email": "linkong@planet.local",
|
||||
"password": "12345678",
|
||||
"password": "LK12345678",
|
||||
"role": "super_admin",
|
||||
},
|
||||
)
|
||||
|
||||
@@ -22,7 +22,7 @@ DEFAULT_LOGIN_USERS = (
|
||||
{
|
||||
"username": "linkong",
|
||||
"email": "linkong@planet.local",
|
||||
"password": "12345678",
|
||||
"password": "LK12345678",
|
||||
"role": "super_admin",
|
||||
},
|
||||
)
|
||||
|
||||
128
backend/tests/test_ai_observability.py
Normal file
128
backend/tests/test_ai_observability.py
Normal file
@@ -0,0 +1,128 @@
|
||||
import pytest
|
||||
from fastapi import HTTPException
|
||||
|
||||
from app.schemas.ai import SituationalAnalysisRequest
|
||||
from app.services.ai_tools import web_search as web_search_module
|
||||
from app.services.ai_tools.schemas import SearchEvidence, WebSearchConfig, WebSearchProviderConfig
|
||||
from app.services.ai_tools.web_search import WebSearchClient
|
||||
from app.services import ai_client as ai_client_module
|
||||
from app.services.ai_client import AIProviderClient
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ai_client_analyze_logs_summary_without_prompt(monkeypatch):
|
||||
events = []
|
||||
|
||||
async def fake_emit_business_log(_logger, **payload):
|
||||
events.append(payload)
|
||||
|
||||
async def fake_request(self, method, path, json=None, request_id=None, operation="request", payload_summary=None):
|
||||
return {
|
||||
"provider": "test-provider",
|
||||
"model": "test-model",
|
||||
"content": "ok",
|
||||
"content_blocks": [],
|
||||
"text_blocks": ["ok"],
|
||||
"thinking_blocks": [],
|
||||
"raw_response": {},
|
||||
}
|
||||
|
||||
monkeypatch.setattr(ai_client_module, "emit_business_log", fake_emit_business_log)
|
||||
monkeypatch.setattr(AIProviderClient, "_request", fake_request)
|
||||
|
||||
client = AIProviderClient(
|
||||
service_url="http://provider.test",
|
||||
llm_config={"provider": "openai", "provider_api": "openai-completions", "model": "gpt-test", "api_key": "sk-secret"},
|
||||
)
|
||||
result = await client.analyze(
|
||||
SituationalAnalysisRequest(
|
||||
title="Sensitive title",
|
||||
objective="Do not store this full prompt",
|
||||
observations=["secret observation"],
|
||||
constraints=["secret constraint"],
|
||||
context={"source": "test", "private": "value"},
|
||||
),
|
||||
request_id="req-ai-test",
|
||||
)
|
||||
|
||||
assert result.model == "test-model"
|
||||
assert [event["event"] for event in events] == [
|
||||
"ai.provider.analyze.start",
|
||||
"ai.provider.analyze.success",
|
||||
]
|
||||
serialized = str(events)
|
||||
assert "Do not store this full prompt" not in serialized
|
||||
assert "secret observation" not in serialized
|
||||
assert "sk-secret" not in serialized
|
||||
start_context = events[0]["context"]
|
||||
assert start_context["model"] == "gpt-test"
|
||||
assert start_context["input_summary"]["objective_length"] == len("Do not store this full prompt")
|
||||
assert start_context["input_summary"]["observation_count"] == 1
|
||||
assert start_context["input_summary"]["context_keys"] == ["private", "source"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ai_client_analyze_logs_failure(monkeypatch):
|
||||
events = []
|
||||
|
||||
async def fake_emit_business_log(_logger, **payload):
|
||||
events.append(payload)
|
||||
|
||||
async def fake_request(self, method, path, json=None, request_id=None, operation="request", payload_summary=None):
|
||||
raise HTTPException(status_code=502, detail="provider failed")
|
||||
|
||||
monkeypatch.setattr(ai_client_module, "emit_business_log", fake_emit_business_log)
|
||||
monkeypatch.setattr(AIProviderClient, "_request", fake_request)
|
||||
|
||||
client = AIProviderClient(service_url="http://provider.test", llm_config={"provider": "openai", "model": "gpt-test"})
|
||||
|
||||
with pytest.raises(HTTPException):
|
||||
await client.analyze(
|
||||
SituationalAnalysisRequest(title="T", objective="O", observations=["one"]),
|
||||
request_id="req-ai-fail",
|
||||
)
|
||||
|
||||
assert events[-1]["event"] == "ai.provider.analyze.failed"
|
||||
assert events[-1]["level"] == "error"
|
||||
assert events[-1]["context"]["error_type"] == "HTTPException"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_web_search_logs_query_hash_without_query(monkeypatch):
|
||||
events = []
|
||||
|
||||
async def fake_emit_business_log(_logger, **payload):
|
||||
events.append(payload)
|
||||
|
||||
async def fake_search_tavily(self, config, query, max_results, domains, freshness_days):
|
||||
return [
|
||||
SearchEvidence(
|
||||
title="Example",
|
||||
url="https://example.test",
|
||||
snippet="result",
|
||||
source_provider="tavily",
|
||||
)
|
||||
]
|
||||
|
||||
monkeypatch.setattr(web_search_module, "emit_business_log", fake_emit_business_log)
|
||||
monkeypatch.setattr(WebSearchClient, "_search_tavily", fake_search_tavily)
|
||||
|
||||
client = WebSearchClient(
|
||||
WebSearchConfig(
|
||||
enabled=True,
|
||||
default_provider="tavily",
|
||||
providers={"tavily": WebSearchProviderConfig(provider="tavily", api_key="secret-key")},
|
||||
)
|
||||
)
|
||||
results = await client.search("secret query text", max_results=1)
|
||||
|
||||
assert len(results) == 1
|
||||
assert [event["event"] for event in events] == [
|
||||
"ai_tool.web_search.start",
|
||||
"ai_tool.web_search.success",
|
||||
]
|
||||
serialized = str(events)
|
||||
assert "secret query text" not in serialized
|
||||
assert "secret-key" not in serialized
|
||||
assert events[0]["context"]["query_length"] == len("secret query text")
|
||||
assert events[1]["context"]["result_count"] == 1
|
||||
@@ -1,9 +1,13 @@
|
||||
"""Unit tests for data collectors"""
|
||||
|
||||
import json
|
||||
|
||||
import pytest
|
||||
from unittest.mock import AsyncMock, patch
|
||||
|
||||
from app.core.datasource_defaults import DEFAULT_DATASOURCES
|
||||
from app.services.collectors.celestrak import CelesTrakTLECollector
|
||||
from app.services.collectors.downloads import DownloadHTTPStatusError, ResumableFileDownloader
|
||||
from app.services.credential_guides import DEFAULT_CREDENTIAL_GUIDES
|
||||
from app.services.collectors.top500 import TOP500Collector
|
||||
from app.services.collectors.registry import collector_registry
|
||||
@@ -149,6 +153,172 @@ class TestHTTPCollector:
|
||||
assert callable(collector.parse_response)
|
||||
|
||||
|
||||
class TestCelesTrakTLECollector:
|
||||
def test_transform_uses_norad_as_source_id_and_preserves_starlink_group(self):
|
||||
collector = CelesTrakTLECollector()
|
||||
result = collector.transform([
|
||||
{
|
||||
"NORAD_CAT_ID": 44720,
|
||||
"OBJECT_NAME": "STARLINK-1000",
|
||||
"OBJECT_ID": "2019-029AZ",
|
||||
"EPOCH": "2026-03-13T00:00:00Z",
|
||||
"MEAN_MOTION": 15.79234567,
|
||||
"ECCENTRICITY": 0.0001234,
|
||||
"INCLINATION": 53.0,
|
||||
"RA_OF_ASC_NODE": 10.0,
|
||||
"ARG_OF_PERICENTER": 20.0,
|
||||
"MEAN_ANOMALY": 30.0,
|
||||
"_celestrak_query_group": "active",
|
||||
"_celestrak_source_url": "https://celestrak.example/gp.php?GROUP=active&FORMAT=json",
|
||||
}
|
||||
])
|
||||
|
||||
assert result[0]["source_id"] == "44720"
|
||||
assert result[0]["metadata"]["constellation_group"] == "starlink"
|
||||
assert result[0]["metadata"]["celestrak_query_group"] == "active"
|
||||
assert result[0]["metadata"]["norad_cat_id"] == 44720
|
||||
assert result[0]["metadata"]["tle_line1"]
|
||||
assert result[0]["metadata"]["tle_line2"]
|
||||
|
||||
def test_load_active_payload_rejects_invalid_records(self, tmp_path):
|
||||
collector = CelesTrakTLECollector()
|
||||
payload_path = tmp_path / "active.json"
|
||||
payload_path.write_text(json.dumps([{"OBJECT_NAME": "missing norad"}]), encoding="utf-8")
|
||||
|
||||
with pytest.raises(RuntimeError, match="invalid record"):
|
||||
collector._load_active_payload(payload_path)
|
||||
|
||||
def test_load_active_payload_accepts_complete_array(self, tmp_path):
|
||||
collector = CelesTrakTLECollector()
|
||||
payload_path = tmp_path / "active.json"
|
||||
payload_path.write_text(
|
||||
json.dumps([{"NORAD_CAT_ID": 25544, "OBJECT_NAME": "ISS (ZARYA)"}]),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
records = collector._load_active_payload(payload_path)
|
||||
|
||||
assert records == [{"NORAD_CAT_ID": 25544, "OBJECT_NAME": "ISS (ZARYA)"}]
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_fetch_retries_and_raises_instead_of_returning_partial_data(self, monkeypatch, tmp_path):
|
||||
collector = CelesTrakTLECollector()
|
||||
collector._resolved_url = "https://celestrak.example/NORAD/elements/gp.php"
|
||||
attempts = 0
|
||||
|
||||
async def fake_download_file(*args, **kwargs):
|
||||
nonlocal attempts
|
||||
attempts += 1
|
||||
raise RuntimeError("network interrupted")
|
||||
|
||||
async def fake_emit_business_log(*args, **kwargs):
|
||||
return None
|
||||
|
||||
monkeypatch.setattr(collector._downloader, "download_file", fake_download_file)
|
||||
monkeypatch.setattr("app.services.collectors.celestrak.emit_business_log", fake_emit_business_log)
|
||||
monkeypatch.setattr("app.services.collectors.celestrak.asyncio.sleep", AsyncMock())
|
||||
|
||||
with pytest.raises(RuntimeError, match="failed after retries"):
|
||||
await collector.fetch()
|
||||
|
||||
assert attempts == 3
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_fetch_uses_cache_when_celestrak_reports_not_updated(self, monkeypatch, tmp_path):
|
||||
collector = CelesTrakTLECollector()
|
||||
collector._resolved_url = "https://celestrak.example/NORAD/elements/gp.php"
|
||||
collector._downloader = ResumableFileDownloader(cache_namespace="celestrak-test", cache_root=tmp_path)
|
||||
url = collector._active_url()
|
||||
cached_path = collector._downloader.cached_file_path(url, ".json")
|
||||
cached_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
cached_path.write_text(
|
||||
json.dumps([{"NORAD_CAT_ID": 25544, "OBJECT_NAME": "ISS (ZARYA)"}]),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
async def fake_download_file(*args, **kwargs):
|
||||
raise DownloadHTTPStatusError(
|
||||
url=url,
|
||||
status_code=403,
|
||||
body="GP data has not updated since your last successful download of GROUP=active.",
|
||||
)
|
||||
|
||||
async def fake_emit_business_log(*args, **kwargs):
|
||||
return None
|
||||
|
||||
monkeypatch.setattr(collector._downloader, "download_file", fake_download_file)
|
||||
monkeypatch.setattr("app.services.collectors.celestrak.emit_business_log", fake_emit_business_log)
|
||||
|
||||
records = await collector.fetch()
|
||||
|
||||
assert records[0]["NORAD_CAT_ID"] == 25544
|
||||
assert records[0]["_celestrak_query_group"] == "active"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_fetch_not_updated_without_cache_does_not_retry(self, monkeypatch, tmp_path):
|
||||
collector = CelesTrakTLECollector()
|
||||
collector._resolved_url = "https://celestrak.example/NORAD/elements/gp.php"
|
||||
collector._downloader = ResumableFileDownloader(cache_namespace="celestrak-test", cache_root=tmp_path)
|
||||
attempts = 0
|
||||
|
||||
async def fake_download_file(*args, **kwargs):
|
||||
nonlocal attempts
|
||||
attempts += 1
|
||||
raise DownloadHTTPStatusError(
|
||||
url=collector._active_url(),
|
||||
status_code=403,
|
||||
body="GP data has not updated since your last successful download of GROUP=active.",
|
||||
)
|
||||
|
||||
async def fake_emit_business_log(*args, **kwargs):
|
||||
return None
|
||||
|
||||
monkeypatch.setattr(collector._downloader, "download_file", fake_download_file)
|
||||
monkeypatch.setattr("app.services.collectors.celestrak.FALLBACK_GROUPS", ("starlink",))
|
||||
monkeypatch.setattr("app.services.collectors.celestrak.emit_business_log", fake_emit_business_log)
|
||||
monkeypatch.setattr("app.services.collectors.celestrak.asyncio.sleep", AsyncMock())
|
||||
|
||||
with pytest.raises(RuntimeError, match="fallback group mode failed"):
|
||||
await collector.fetch()
|
||||
|
||||
assert attempts == 2
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_fetch_falls_back_to_all_groups_when_active_not_updated_without_cache(self, monkeypatch, tmp_path):
|
||||
collector = CelesTrakTLECollector()
|
||||
collector._resolved_url = "https://celestrak.example/NORAD/elements/gp.php"
|
||||
collector._downloader = ResumableFileDownloader(cache_namespace="celestrak-test", cache_root=tmp_path)
|
||||
payload_by_group = {
|
||||
"starlink": [{"NORAD_CAT_ID": 100, "OBJECT_NAME": "STARLINK-100"}],
|
||||
"gps-ops": [{"NORAD_CAT_ID": 200, "OBJECT_NAME": "GPS BIIR-2"}],
|
||||
}
|
||||
|
||||
async def fake_download_file(_client, url, **_kwargs):
|
||||
if "GROUP=active" in url:
|
||||
raise DownloadHTTPStatusError(
|
||||
url=url,
|
||||
status_code=403,
|
||||
body="GP data has not updated since your last successful download of GROUP=active.",
|
||||
)
|
||||
group = "starlink" if "GROUP=starlink" in url else "gps-ops"
|
||||
path = tmp_path / f"{group}.json"
|
||||
path.write_text(json.dumps(payload_by_group[group]), encoding="utf-8")
|
||||
return path
|
||||
|
||||
async def fake_emit_business_log(*args, **kwargs):
|
||||
return None
|
||||
|
||||
monkeypatch.setattr(collector._downloader, "download_file", fake_download_file)
|
||||
monkeypatch.setattr("app.services.collectors.celestrak.FALLBACK_GROUPS", tuple(payload_by_group))
|
||||
monkeypatch.setattr("app.services.collectors.celestrak.emit_business_log", fake_emit_business_log)
|
||||
|
||||
records = await collector.fetch()
|
||||
|
||||
assert [item["NORAD_CAT_ID"] for item in records] == [100, 200]
|
||||
assert records[0]["_celestrak_query_group"] == "starlink"
|
||||
assert records[1]["_celestrak_group"] == "gps-ops"
|
||||
|
||||
|
||||
def test_aisstream_collector_is_registered():
|
||||
collector = collector_registry.get("aisstream_vessels")
|
||||
|
||||
|
||||
@@ -1,9 +1,11 @@
|
||||
import asyncio
|
||||
import inspect
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import pytest
|
||||
|
||||
from app.api.v1 import datasources as datasources_api
|
||||
from app.models.datasource import DataSource
|
||||
from app.models.task import CollectionTask
|
||||
from app.services import data_jobs
|
||||
from app.services import earth_layer_cache as earth_cache
|
||||
|
||||
|
||||
@@ -100,6 +102,71 @@ def test_serialize_datasource_row_includes_endpoint_when_requested():
|
||||
assert row["endpoint"] == "https://example.test/arcgis_landing_points"
|
||||
|
||||
|
||||
def test_serialize_datasource_row_separates_collector_running_from_delete_task():
|
||||
datasource = make_datasource(9, "top500", last_status="success")
|
||||
task = CollectionTask(
|
||||
id=44,
|
||||
datasource_id=9,
|
||||
source="top500",
|
||||
task_type="clear_data",
|
||||
status="running",
|
||||
phase="clearing_data",
|
||||
)
|
||||
|
||||
row = datasources_api.serialize_datasource_row(
|
||||
datasource,
|
||||
running_tasks={9: task},
|
||||
latest_tasks={9: task},
|
||||
record_counts={"top500": 500},
|
||||
endpoint_overrides={},
|
||||
config=object(),
|
||||
include_endpoint=False,
|
||||
)
|
||||
|
||||
assert row["is_task_active"] is True
|
||||
assert row["is_running"] is False
|
||||
assert row["task_type"] == "clear_data"
|
||||
assert row["task_status"] == "running"
|
||||
|
||||
|
||||
def test_cancel_queued_delete_task_finishes_immediately(monkeypatch):
|
||||
task = CollectionTask(
|
||||
id=45,
|
||||
datasource_id=9,
|
||||
source="top500",
|
||||
task_type="clear_data",
|
||||
status="queued",
|
||||
phase="queued",
|
||||
)
|
||||
|
||||
class FakeDb:
|
||||
async def commit(self):
|
||||
return None
|
||||
|
||||
async def refresh(self, _task):
|
||||
return None
|
||||
|
||||
async def fake_broadcast(_task):
|
||||
return None
|
||||
|
||||
monkeypatch.setattr(data_jobs, "_broadcast_task_update", fake_broadcast)
|
||||
|
||||
async def run():
|
||||
return await data_jobs.request_cancel_datasource_task(FakeDb(), task)
|
||||
|
||||
result = asyncio.run(run())
|
||||
|
||||
assert result.status == "cancelled"
|
||||
assert result.phase == "cancelled"
|
||||
assert result.completed_at is not None
|
||||
|
||||
|
||||
def test_clear_data_job_relies_on_db_outbox_instead_of_extra_earth_refresh_task():
|
||||
source = inspect.getsource(data_jobs._run_clear_data_job)
|
||||
|
||||
assert "enqueue_earth_refresh_job" not in source
|
||||
|
||||
|
||||
def test_invalidate_earth_layer_cache_for_source_covers_datasource_aliases(monkeypatch):
|
||||
patterns: list[str] = []
|
||||
|
||||
@@ -119,34 +186,36 @@ def test_invalidate_earth_layer_cache_for_source_covers_datasource_aliases(monke
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_trigger_datasource_batch_skips_disabled_and_frequency_window(monkeypatch):
|
||||
now = datetime.now(timezone.utc)
|
||||
disabled = make_datasource(1, "aisstream_vessels", is_active=False)
|
||||
not_due = make_datasource(2, "telegeography_cables", last_run_at=now, frequency_minutes=120)
|
||||
due = make_datasource(3, "ris_live_bgp", last_run_at=now - timedelta(hours=2))
|
||||
triggered_sources: list[str] = []
|
||||
def test_trigger_datasource_batch_skips_disabled_and_frequency_window(monkeypatch):
|
||||
async def run():
|
||||
now = datetime.now(timezone.utc)
|
||||
disabled = make_datasource(1, "aisstream_vessels", is_active=False)
|
||||
not_due = make_datasource(2, "telegeography_cables", last_run_at=now, frequency_minutes=120)
|
||||
due = make_datasource(3, "ris_live_bgp", last_run_at=now - timedelta(hours=2))
|
||||
queued_sources: list[str] = []
|
||||
|
||||
async def fake_running_tasks(_db, _ids):
|
||||
return {}
|
||||
async def fake_running_tasks(_db, _ids):
|
||||
return {}
|
||||
|
||||
async def fake_latest_task_ids(_db, _ids):
|
||||
return {}
|
||||
async def fake_enqueue(_db, datasource, task_type, **_kwargs):
|
||||
queued_sources.append(datasource.source)
|
||||
return CollectionTask(id=100 + datasource.id, datasource_id=datasource.id, source=datasource.source, task_type=task_type, status="queued")
|
||||
|
||||
monkeypatch.setattr(datasources_api, "_load_latest_running_tasks", fake_running_tasks)
|
||||
monkeypatch.setattr(datasources_api, "_load_latest_task_ids", fake_latest_task_ids)
|
||||
monkeypatch.setattr(
|
||||
datasources_api,
|
||||
"run_collector_now",
|
||||
lambda source: triggered_sources.append(source) or True,
|
||||
)
|
||||
monkeypatch.setattr(datasources_api, "_load_latest_running_tasks", fake_running_tasks)
|
||||
monkeypatch.setattr(
|
||||
datasources_api,
|
||||
"enqueue_datasource_job",
|
||||
fake_enqueue,
|
||||
)
|
||||
|
||||
result = await datasources_api._trigger_datasource_batch(
|
||||
object(),
|
||||
[disabled, not_due, due],
|
||||
force=False,
|
||||
)
|
||||
result = await datasources_api._trigger_datasource_batch(
|
||||
object(),
|
||||
[disabled, not_due, due],
|
||||
force=False,
|
||||
)
|
||||
|
||||
assert [item["source"] for item in result["triggered"]] == ["ris_live_bgp"]
|
||||
assert {item["reason"] for item in result["skipped"]} == {"disabled", "within_frequency_window"}
|
||||
assert triggered_sources == ["ris_live_bgp"]
|
||||
assert [item["source"] for item in result["triggered"]] == ["ris_live_bgp"]
|
||||
assert {item["reason"] for item in result["skipped"]} == {"disabled", "within_frequency_window"}
|
||||
assert queued_sources == ["ris_live_bgp"]
|
||||
|
||||
asyncio.run(run())
|
||||
|
||||
@@ -1,11 +1,15 @@
|
||||
"""Docs Gatekeeper API tests."""
|
||||
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
|
||||
from app.api.v1 import docs as docs_api
|
||||
from app.main import app
|
||||
from app.models.user import User
|
||||
from app.services.docs_gatekeeper import DOCS_METADATA
|
||||
|
||||
|
||||
def make_user(role: str = "viewer", groups: list[str] | None = None) -> User:
|
||||
@@ -43,22 +47,45 @@ async def test_public_catalog_only_for_anonymous_user():
|
||||
assert response.status_code == 200
|
||||
items = response.json()["items"]
|
||||
assert {item["access"] for item in items} == {"public"}
|
||||
assert {item["slug"] for item in items if item["lang"] == "zh"} == {
|
||||
zh_items = [item for item in items if item["lang"] == "zh"]
|
||||
assert [item["slug"] for item in zh_items] == [
|
||||
"overview",
|
||||
"quickstart",
|
||||
"manual",
|
||||
"quickstart",
|
||||
"faq",
|
||||
"location-pipeline-user",
|
||||
}
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_developer_catalog_includes_architecture_and_frontend_reference_docs():
|
||||
response = await get_json(
|
||||
"/api/v1/docs/catalog",
|
||||
make_user(role="viewer", groups=["docs_developer"]),
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
zh_slugs = {item["slug"] for item in response.json()["items"] if item["lang"] == "zh"}
|
||||
zh_items = [item for item in response.json()["items"] if item["lang"] == "zh"]
|
||||
assert [item["group"] for item in zh_items[:4]] == ["Overview", "Manual", "Manual", "Manual"]
|
||||
assert zh_items[4]["group"] == "Architecture"
|
||||
assert "platform-data-flows" in zh_slugs
|
||||
assert "naming-glossary" in zh_slugs
|
||||
assert "tactile-ui-components" in zh_slugs
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_anonymous_can_read_public_doc():
|
||||
response = await get_json("/api/v1/docs/zh/quickstart")
|
||||
manual_response = await get_json("/api/v1/docs/zh/manual")
|
||||
overview_response = await get_json("/api/v1/docs/zh/overview")
|
||||
|
||||
assert response.status_code == 200
|
||||
assert response.json()["access"] == "public"
|
||||
assert "快速开始" in response.json()["markdown"]
|
||||
assert manual_response.status_code == 200
|
||||
assert manual_response.json()["access"] == "public"
|
||||
assert overview_response.status_code == 200
|
||||
assert overview_response.json()["access"] == "public"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@@ -83,10 +110,16 @@ async def test_developer_group_can_read_developer_but_not_admin_doc():
|
||||
user = make_user(role="viewer", groups=["docs_developer"])
|
||||
|
||||
developer_response = await get_json("/api/v1/docs/zh/backend-collectors", user)
|
||||
tactile_response = await get_json("/api/v1/docs/zh/tactile-ui-components", user)
|
||||
glossary_response = await get_json("/api/v1/docs/zh/naming-glossary", user)
|
||||
admin_response = await get_json("/api/v1/docs/zh/backend-system-service-control", user)
|
||||
|
||||
assert developer_response.status_code == 200
|
||||
assert developer_response.json()["access"] == "docs_developer"
|
||||
assert tactile_response.status_code == 200
|
||||
assert tactile_response.json()["access"] == "docs_developer"
|
||||
assert glossary_response.status_code == 200
|
||||
assert glossary_response.json()["access"] == "docs_developer"
|
||||
assert admin_response.status_code == 403
|
||||
|
||||
|
||||
@@ -114,3 +147,29 @@ async def test_unknown_language_slug_and_path_traversal_do_not_read_files():
|
||||
assert bad_lang.status_code == 404
|
||||
assert bad_slug.status_code == 404
|
||||
assert traversal.status_code == 404
|
||||
|
||||
|
||||
def test_public_docs_markdown_links_do_not_create_missing_docs_routes():
|
||||
repo_root = Path(__file__).resolve().parents[2]
|
||||
technical_root = repo_root / "docs" / "technical"
|
||||
registered_filenames = {entry.filename for entry in DOCS_METADATA}
|
||||
problems: list[str] = []
|
||||
|
||||
for markdown_path in sorted(technical_root.glob("*/*.md")):
|
||||
markdown = markdown_path.read_text(encoding="utf-8")
|
||||
for match in re.finditer(r"\[([^\]]+)]\(([^)]+\.md(?:#[^)]+)?)\)", markdown):
|
||||
label, href = match.group(1), match.group(2)
|
||||
if href.startswith(("http://", "https://", "mailto:")):
|
||||
continue
|
||||
|
||||
href_without_hash = href.split("#", 1)[0].replace("\\", "/")
|
||||
filename = Path(href_without_hash).name
|
||||
if "/docs/technical/" in href_without_hash:
|
||||
if filename not in registered_filenames:
|
||||
problems.append(f"{markdown_path.relative_to(repo_root)} links unregistered public doc {href!r} ({label})")
|
||||
continue
|
||||
|
||||
if href_without_hash.endswith(".md"):
|
||||
problems.append(f"{markdown_path.relative_to(repo_root)} links non-public markdown {href!r} ({label})")
|
||||
|
||||
assert problems == []
|
||||
|
||||
454
backend/tests/test_earth_db_change_listener.py
Normal file
454
backend/tests/test_earth_db_change_listener.py
Normal file
@@ -0,0 +1,454 @@
|
||||
import asyncio
|
||||
import json
|
||||
|
||||
from app.services.earth_db_change_listener import (
|
||||
EarthDbChangeDispatcher,
|
||||
EarthDbChangeListener,
|
||||
build_earth_update_from_db_payload,
|
||||
normalize_asyncpg_dsn,
|
||||
)
|
||||
|
||||
|
||||
def test_normalize_asyncpg_dsn_strips_sqlalchemy_driver():
|
||||
assert (
|
||||
normalize_asyncpg_dsn("postgresql+asyncpg://postgres:postgres@localhost:5432/planet_db")
|
||||
== "postgresql://postgres:postgres@localhost:5432/planet_db"
|
||||
)
|
||||
|
||||
|
||||
def test_build_earth_update_maps_known_sources_to_layers():
|
||||
satellite_update = build_earth_update_from_db_payload(
|
||||
{"source": "celestrak_tle", "operation": "DELETE", "entity_key": "25544"}
|
||||
)
|
||||
cable_update = build_earth_update_from_db_payload(
|
||||
{"source": "arcgis_cables", "operation": "UPDATE", "entity_key": "cable-1"}
|
||||
)
|
||||
compute_update = build_earth_update_from_db_payload(
|
||||
{"source": "top500", "operation": "DELETE", "entity_key": None}
|
||||
)
|
||||
landing_update = build_earth_update_from_db_payload(
|
||||
{"source": "telegeography_landing", "operation": "DELETE", "entity_key": None}
|
||||
)
|
||||
|
||||
assert satellite_update is not None
|
||||
assert satellite_update["layers"] == ["satellites"]
|
||||
assert cable_update is not None
|
||||
assert cable_update["layers"] == ["cables"]
|
||||
assert compute_update is not None
|
||||
assert compute_update["layers"] == ["computeCenters"]
|
||||
assert landing_update is not None
|
||||
assert landing_update["layers"] == ["cables"]
|
||||
assert landing_update["refresh_strategy"] == "clear_then_reload"
|
||||
|
||||
|
||||
def test_build_earth_update_maps_derived_tables_to_layers():
|
||||
bgp_update = build_earth_update_from_db_payload(
|
||||
{"table": "bgp_anomalies", "source": "ris_live_bgp", "operation": "DELETE", "entity_key": "a1"}
|
||||
)
|
||||
compute_update = build_earth_update_from_db_payload(
|
||||
{"table": "compute_center_locations", "source": "top500", "operation": "UPDATE", "entity_key": "top500_1"}
|
||||
)
|
||||
vessel_update = build_earth_update_from_db_payload(
|
||||
{"table": "vessel_position", "operation": "DELETE", "entity_key": "123456789"}
|
||||
)
|
||||
|
||||
assert bgp_update is not None
|
||||
assert bgp_update["source"] == "ris_live_bgp"
|
||||
assert bgp_update["table"] == "bgp_anomalies"
|
||||
assert bgp_update["layers"] == ["bgp"]
|
||||
assert bgp_update["refresh_strategy"] == "clear_then_reload"
|
||||
assert compute_update is not None
|
||||
assert compute_update["layers"] == ["computeCenters"]
|
||||
assert compute_update["refresh_strategy"] == "reload"
|
||||
assert vessel_update is not None
|
||||
assert vessel_update["source"] == "vessel_position"
|
||||
assert vessel_update["layers"] == ["vessels"]
|
||||
|
||||
|
||||
def test_build_earth_update_maps_interactable_delete_to_delta():
|
||||
update = build_earth_update_from_db_payload(
|
||||
{
|
||||
"table": "earth_interactables",
|
||||
"operation": "DELETE",
|
||||
"entity_keys": ["note-1"],
|
||||
"records_processed": 1,
|
||||
}
|
||||
)
|
||||
|
||||
assert update is not None
|
||||
assert update["entity"] == "interactable"
|
||||
assert update["action"] == "deleted"
|
||||
assert update["ids"] == ["note-1"]
|
||||
assert update["layers"] == ["interactables"]
|
||||
assert update["refresh_strategy"] == "delta"
|
||||
|
||||
|
||||
def test_build_earth_update_ignores_unmapped_sources():
|
||||
assert build_earth_update_from_db_payload({"source": "not_for_earth"}) is None
|
||||
|
||||
|
||||
def test_dispatcher_debounces_same_source_notifications():
|
||||
async def run():
|
||||
broadcasts = []
|
||||
invalidated = []
|
||||
|
||||
async def broadcast(payload):
|
||||
broadcasts.append(payload)
|
||||
|
||||
def invalidate(source):
|
||||
invalidated.append(source)
|
||||
return 2
|
||||
|
||||
dispatcher = EarthDbChangeDispatcher(
|
||||
broadcast_earth_update=broadcast,
|
||||
invalidate_cache=invalidate,
|
||||
debounce_seconds=10,
|
||||
)
|
||||
dispatcher.handle_payload(
|
||||
{
|
||||
"table": "collected_data",
|
||||
"operation": "INSERT",
|
||||
"source": "arcgis_cables",
|
||||
"entity_key": "cable-1",
|
||||
"occurred_at": "2026-05-22T00:00:00Z",
|
||||
}
|
||||
)
|
||||
dispatcher.handle_payload(
|
||||
{
|
||||
"table": "collected_data",
|
||||
"operation": "UPDATE",
|
||||
"source": "arcgis_cables",
|
||||
"entity_key": "cable-2",
|
||||
"occurred_at": "2026-05-22T00:00:01Z",
|
||||
}
|
||||
)
|
||||
|
||||
await dispatcher.flush_all()
|
||||
|
||||
assert invalidated == ["arcgis_cables"]
|
||||
assert len(broadcasts) == 1
|
||||
assert broadcasts[0]["action"] == "database_changed"
|
||||
assert broadcasts[0]["source"] == "arcgis_cables"
|
||||
assert broadcasts[0]["layers"] == ["cables"]
|
||||
assert broadcasts[0]["records_processed"] == 2
|
||||
assert broadcasts[0]["operations"] == ["INSERT", "UPDATE"]
|
||||
assert broadcasts[0]["entity_keys"] == ["cable-1", "cable-2"]
|
||||
assert broadcasts[0]["cache_entries_invalidated"] == 2
|
||||
|
||||
asyncio.run(run())
|
||||
|
||||
|
||||
def test_dispatcher_handles_delete_notification_as_earth_update():
|
||||
async def run():
|
||||
broadcasts = []
|
||||
|
||||
async def broadcast(payload):
|
||||
broadcasts.append(payload)
|
||||
|
||||
dispatcher = EarthDbChangeDispatcher(
|
||||
broadcast_earth_update=broadcast,
|
||||
invalidate_cache=lambda source: 1,
|
||||
debounce_seconds=10,
|
||||
)
|
||||
accepted = dispatcher.handle_notification(
|
||||
json.dumps(
|
||||
{
|
||||
"table": "collected_data",
|
||||
"operation": "DELETE",
|
||||
"source": "celestrak_tle",
|
||||
"entity_key": "sat:25544",
|
||||
"occurred_at": "2026-05-22T00:00:00Z",
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
await dispatcher.flush_all()
|
||||
|
||||
assert accepted is True
|
||||
assert len(broadcasts) == 1
|
||||
assert broadcasts[0]["action"] == "database_changed"
|
||||
assert broadcasts[0]["source"] == "celestrak_tle"
|
||||
assert broadcasts[0]["layers"] == ["satellites"]
|
||||
assert broadcasts[0]["operation"] == "DELETE"
|
||||
assert broadcasts[0]["refresh_strategy"] == "clear_then_reload"
|
||||
|
||||
asyncio.run(run())
|
||||
|
||||
|
||||
def test_dispatcher_debounces_bgp_derived_table_events_by_source():
|
||||
async def run():
|
||||
broadcasts = []
|
||||
|
||||
async def broadcast(payload):
|
||||
broadcasts.append(payload)
|
||||
|
||||
dispatcher = EarthDbChangeDispatcher(
|
||||
broadcast_earth_update=broadcast,
|
||||
invalidate_cache=lambda source: 0,
|
||||
debounce_seconds=10,
|
||||
)
|
||||
dispatcher.handle_payload(
|
||||
{
|
||||
"event_id": 200,
|
||||
"table": "bgp_observations",
|
||||
"operation": "DELETE",
|
||||
"source": "ris_live_bgp",
|
||||
"records_processed": 2,
|
||||
}
|
||||
)
|
||||
dispatcher.handle_payload(
|
||||
{
|
||||
"event_id": 201,
|
||||
"table": "bgp_anomalies",
|
||||
"operation": "DELETE",
|
||||
"source": "ris_live_bgp",
|
||||
"records_processed": 3,
|
||||
}
|
||||
)
|
||||
|
||||
await dispatcher.flush_all()
|
||||
|
||||
assert len(broadcasts) == 1
|
||||
assert broadcasts[0]["source"] == "ris_live_bgp"
|
||||
assert broadcasts[0]["layers"] == ["bgp"]
|
||||
assert broadcasts[0]["records_processed"] == 5
|
||||
assert broadcasts[0]["refresh_strategy"] == "clear_then_reload"
|
||||
|
||||
asyncio.run(run())
|
||||
|
||||
|
||||
def test_dispatcher_deduplicates_notify_and_outbox_by_event_id():
|
||||
async def run():
|
||||
broadcasts = []
|
||||
|
||||
async def broadcast(payload):
|
||||
broadcasts.append(payload)
|
||||
|
||||
dispatcher = EarthDbChangeDispatcher(
|
||||
broadcast_earth_update=broadcast,
|
||||
invalidate_cache=lambda source: 0,
|
||||
debounce_seconds=10,
|
||||
)
|
||||
payload = {
|
||||
"event_id": 42,
|
||||
"table": "collected_data",
|
||||
"operation": "INSERT",
|
||||
"source": "celestrak_tle",
|
||||
"entity_key": "sat:42",
|
||||
"occurred_at": "2026-05-23T00:00:00Z",
|
||||
}
|
||||
|
||||
assert dispatcher.handle_payload(payload) is True
|
||||
assert dispatcher.handle_payload(dict(payload)) is False
|
||||
await dispatcher.flush_all()
|
||||
|
||||
assert len(broadcasts) == 1
|
||||
assert broadcasts[0]["records_processed"] == 1
|
||||
assert broadcasts[0]["entity_keys"] == ["sat:42"]
|
||||
|
||||
asyncio.run(run())
|
||||
|
||||
|
||||
def test_dispatcher_flushes_continuous_events_at_max_wait():
|
||||
async def run():
|
||||
broadcasts = []
|
||||
sleeps = []
|
||||
|
||||
async def broadcast(payload):
|
||||
broadcasts.append(payload)
|
||||
|
||||
dispatcher = EarthDbChangeDispatcher(
|
||||
broadcast_earth_update=broadcast,
|
||||
invalidate_cache=lambda source: 0,
|
||||
debounce_seconds=10,
|
||||
max_wait_seconds=0.01,
|
||||
)
|
||||
dispatcher.handle_payload(
|
||||
{
|
||||
"event_id": 300,
|
||||
"table": "collected_data",
|
||||
"operation": "INSERT",
|
||||
"source": "arcgis_cables",
|
||||
"records_processed": 1,
|
||||
}
|
||||
)
|
||||
await asyncio.sleep(0.02)
|
||||
dispatcher.handle_payload(
|
||||
{
|
||||
"event_id": 301,
|
||||
"table": "collected_data",
|
||||
"operation": "UPDATE",
|
||||
"source": "arcgis_cables",
|
||||
"records_processed": 1,
|
||||
}
|
||||
)
|
||||
await asyncio.sleep(0)
|
||||
await dispatcher.flush_all()
|
||||
|
||||
assert len(broadcasts) == 1
|
||||
assert broadcasts[0]["records_processed"] == 2
|
||||
assert broadcasts[0]["debounce_ms"] >= 0
|
||||
|
||||
asyncio.run(run())
|
||||
|
||||
|
||||
def test_dispatcher_fast_flushes_delete_clear_then_reload():
|
||||
async def run():
|
||||
broadcasts = []
|
||||
|
||||
async def broadcast(payload):
|
||||
broadcasts.append(payload)
|
||||
|
||||
dispatcher = EarthDbChangeDispatcher(
|
||||
broadcast_earth_update=broadcast,
|
||||
invalidate_cache=lambda source: 0,
|
||||
debounce_seconds=10,
|
||||
max_wait_seconds=10,
|
||||
)
|
||||
dispatcher.handle_payload(
|
||||
{
|
||||
"event_id": 310,
|
||||
"table": "collected_data",
|
||||
"operation": "DELETE",
|
||||
"source": "celestrak_tle",
|
||||
"records_processed": 10,
|
||||
}
|
||||
)
|
||||
await asyncio.sleep(0.08)
|
||||
|
||||
assert len(broadcasts) == 1
|
||||
assert broadcasts[0]["source"] == "celestrak_tle"
|
||||
assert broadcasts[0]["refresh_strategy"] == "clear_then_reload"
|
||||
|
||||
asyncio.run(run())
|
||||
|
||||
|
||||
def test_outbox_rows_are_consumed_after_successful_flush():
|
||||
async def run():
|
||||
class FakeConnection:
|
||||
def __init__(self):
|
||||
self.consumed_ids = []
|
||||
|
||||
async def fetch(self, _query, _limit):
|
||||
return [
|
||||
{
|
||||
"id": 501,
|
||||
"payload": {
|
||||
"event_id": 501,
|
||||
"table": "collected_data",
|
||||
"operation": "DELETE",
|
||||
"source": "celestrak_tle",
|
||||
"records_processed": 1,
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
async def execute(self, _query, consumed_ids):
|
||||
self.consumed_ids.extend(consumed_ids)
|
||||
|
||||
broadcasts = []
|
||||
|
||||
async def broadcast(payload):
|
||||
broadcasts.append(payload)
|
||||
|
||||
dispatcher = EarthDbChangeDispatcher(
|
||||
broadcast_earth_update=broadcast,
|
||||
invalidate_cache=lambda source: 0,
|
||||
debounce_seconds=10,
|
||||
)
|
||||
listener = EarthDbChangeListener(
|
||||
dsn="postgresql://example/db",
|
||||
dispatcher=dispatcher,
|
||||
)
|
||||
connection = FakeConnection()
|
||||
listener._connection = connection
|
||||
|
||||
await listener._poll_outbox()
|
||||
|
||||
assert connection.consumed_ids == [501]
|
||||
assert len(broadcasts) == 1
|
||||
|
||||
asyncio.run(run())
|
||||
|
||||
|
||||
def test_outbox_rows_remain_unconsumed_when_flush_fails():
|
||||
async def run():
|
||||
class FakeConnection:
|
||||
def __init__(self):
|
||||
self.consumed_ids = []
|
||||
|
||||
async def fetch(self, _query, _limit):
|
||||
return [
|
||||
{
|
||||
"id": 601,
|
||||
"payload": {
|
||||
"event_id": 601,
|
||||
"table": "collected_data",
|
||||
"operation": "DELETE",
|
||||
"source": "celestrak_tle",
|
||||
"records_processed": 1,
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
async def execute(self, _query, consumed_ids):
|
||||
self.consumed_ids.extend(consumed_ids)
|
||||
|
||||
async def broadcast(_payload):
|
||||
raise RuntimeError("ws down")
|
||||
|
||||
dispatcher = EarthDbChangeDispatcher(
|
||||
broadcast_earth_update=broadcast,
|
||||
invalidate_cache=lambda source: 0,
|
||||
debounce_seconds=10,
|
||||
)
|
||||
listener = EarthDbChangeListener(
|
||||
dsn="postgresql://example/db",
|
||||
dispatcher=dispatcher,
|
||||
)
|
||||
connection = FakeConnection()
|
||||
listener._connection = connection
|
||||
|
||||
try:
|
||||
await listener._poll_outbox()
|
||||
except RuntimeError:
|
||||
pass
|
||||
|
||||
assert connection.consumed_ids == []
|
||||
|
||||
asyncio.run(run())
|
||||
|
||||
|
||||
def test_dispatcher_uses_aggregated_statement_record_count():
|
||||
async def run():
|
||||
broadcasts = []
|
||||
|
||||
async def broadcast(payload):
|
||||
broadcasts.append(payload)
|
||||
|
||||
dispatcher = EarthDbChangeDispatcher(
|
||||
broadcast_earth_update=broadcast,
|
||||
invalidate_cache=lambda source: 0,
|
||||
debounce_seconds=10,
|
||||
)
|
||||
dispatcher.handle_payload(
|
||||
{
|
||||
"event_id": 99,
|
||||
"table": "collected_data",
|
||||
"operation": "DELETE",
|
||||
"source": "top500",
|
||||
"records_processed": 100,
|
||||
"entity_keys": ["top500:1", "top500:2"],
|
||||
"occurred_at": "2026-05-23T00:00:00Z",
|
||||
}
|
||||
)
|
||||
|
||||
await dispatcher.flush_all()
|
||||
|
||||
assert len(broadcasts) == 1
|
||||
assert broadcasts[0]["source"] == "top500"
|
||||
assert broadcasts[0]["layers"] == ["computeCenters"]
|
||||
assert broadcasts[0]["records_processed"] == 100
|
||||
assert broadcasts[0]["entity_keys"] == ["top500:1", "top500:2"]
|
||||
|
||||
asyncio.run(run())
|
||||
79
backend/tests/test_earth_interactables.py
Normal file
79
backend/tests/test_earth_interactables.py
Normal file
@@ -0,0 +1,79 @@
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from app.models.earth_interactable import EarthInteractable
|
||||
from app.services import earth_interactables
|
||||
|
||||
|
||||
def make_interactable(**overrides):
|
||||
values = {
|
||||
"id": "poi-1",
|
||||
"layer": "places",
|
||||
"kind": "note",
|
||||
"label": "Test POI",
|
||||
"description": "A point on Earth",
|
||||
"latitude": 30.25,
|
||||
"longitude": 120.15,
|
||||
"altitude": None,
|
||||
"revision": 3,
|
||||
"properties": {"owner": "test"},
|
||||
"is_deleted": False,
|
||||
"created_at": datetime(2026, 5, 22, 1, 0, tzinfo=UTC),
|
||||
"updated_at": datetime(2026, 5, 22, 1, 5, tzinfo=UTC),
|
||||
"deleted_at": None,
|
||||
}
|
||||
values.update(overrides)
|
||||
return EarthInteractable(**values)
|
||||
|
||||
|
||||
def test_interactable_event_uses_object_delta_contract():
|
||||
record = make_interactable()
|
||||
|
||||
event = earth_interactables.build_interactable_event(
|
||||
action="updated",
|
||||
record=record,
|
||||
)
|
||||
|
||||
assert event["entity"] == "interactable"
|
||||
assert event["action"] == "updated"
|
||||
assert event["layer"] == "places"
|
||||
assert event["layers"] == ["interactables"]
|
||||
assert event["ids"] == ["poi-1"]
|
||||
assert event["revision"] == 3
|
||||
assert event["item"]["latitude"] == 30.25
|
||||
assert event["item"]["properties"] == {"owner": "test"}
|
||||
|
||||
|
||||
def test_interactable_geojson_omits_deleted_records():
|
||||
active = make_interactable(id="active")
|
||||
deleted = make_interactable(id="deleted", is_deleted=True)
|
||||
|
||||
payload = earth_interactables.interactables_to_geojson([active, deleted])
|
||||
|
||||
assert payload["type"] == "FeatureCollection"
|
||||
assert [feature["id"] for feature in payload["features"]] == ["active"]
|
||||
assert payload["features"][0]["geometry"] == {
|
||||
"type": "Point",
|
||||
"coordinates": [120.15, 30.25],
|
||||
}
|
||||
|
||||
|
||||
def test_interactable_cache_invalidation_clears_layer_and_all(monkeypatch):
|
||||
patterns = []
|
||||
|
||||
def fake_delete_pattern(pattern):
|
||||
patterns.append(pattern)
|
||||
return 1
|
||||
|
||||
monkeypatch.setattr(
|
||||
earth_interactables.earth_layer_cache,
|
||||
"delete_pattern",
|
||||
fake_delete_pattern,
|
||||
)
|
||||
|
||||
deleted = earth_interactables.invalidate_interactable_cache("places")
|
||||
|
||||
assert deleted == 2
|
||||
assert patterns == [
|
||||
"earth:layer:v1:interactables:layer:places*",
|
||||
"earth:layer:v1:interactables:layer:all*",
|
||||
]
|
||||
@@ -421,6 +421,62 @@ async def test_earth_news_payload_uses_fresh_database_items_without_rss(monkeypa
|
||||
assert payload["stale"] is False
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_earth_news_payload_keeps_current_items_and_all_cruise_items(monkeypatch):
|
||||
class FakeDb:
|
||||
execute = object()
|
||||
|
||||
current_item = ParsedNewsItem(
|
||||
id="db:current",
|
||||
title="Current region story",
|
||||
summary="Current summary",
|
||||
url="https://example.com/current",
|
||||
source="Stored Source",
|
||||
feed_name="Stored Feed",
|
||||
feed_region="americas",
|
||||
homepage_url="https://example.com",
|
||||
published_at=datetime(2026, 5, 15, 3, 0, tzinfo=UTC),
|
||||
)
|
||||
cruise_item = ParsedNewsItem(
|
||||
id="db:apac",
|
||||
title="APAC story",
|
||||
summary="APAC summary",
|
||||
url="https://example.com/apac",
|
||||
source="Stored Source",
|
||||
feed_name="Stored Feed",
|
||||
feed_region="asia-pacific",
|
||||
homepage_url="https://example.com",
|
||||
published_at=datetime(2026, 5, 15, 4, 0, tzinfo=UTC),
|
||||
)
|
||||
|
||||
async def fake_get_earth_news_freshness(_db, *, active_region):
|
||||
return 12, datetime.now(UTC)
|
||||
|
||||
async def fake_list_earth_news_items(_db, *, active_region, limit):
|
||||
return [current_item]
|
||||
|
||||
async def fake_list_earth_news_cruise_items(_db, *, limit):
|
||||
return [current_item, cruise_item]
|
||||
|
||||
async def fake_enqueue_target_location_job(_payload, **_kwargs):
|
||||
return True
|
||||
|
||||
async def fail_fetch(_sources):
|
||||
raise AssertionError("fresh database items should not fetch RSS")
|
||||
|
||||
monkeypatch.setattr("app.services.earth_news_store.get_earth_news_freshness", fake_get_earth_news_freshness)
|
||||
monkeypatch.setattr("app.services.earth_news_store.list_earth_news_items", fake_list_earth_news_items)
|
||||
monkeypatch.setattr("app.services.earth_news_store.list_earth_news_cruise_items", fake_list_earth_news_cruise_items)
|
||||
monkeypatch.setattr("app.services.earth_news_queue.enqueue_target_location_job", fake_enqueue_target_location_job)
|
||||
monkeypatch.setattr("app.services.earth_news._fetch_rss_items_for_sources", fail_fetch)
|
||||
|
||||
payload = await get_earth_news_payload(lat=35.0, lon=-100.0, db=FakeDb())
|
||||
|
||||
assert [item["id"] for item in payload["items"]] == ["db:current"]
|
||||
assert [item["id"] for item in payload["cruise_items"]] == ["db:current", "db:apac"]
|
||||
assert payload["cruise_items"][1]["region"] == "asia-pacific"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_earth_news_payload_initializes_empty_database_from_rss(monkeypatch):
|
||||
db = object()
|
||||
|
||||
@@ -4,8 +4,11 @@ import logging
|
||||
|
||||
from io import StringIO
|
||||
|
||||
import pytest
|
||||
|
||||
from app.core.logging import PlanetContextFilter, PlanetFormatter, get_logger
|
||||
from app.core.request_context import set_request_id
|
||||
from app.services import business_logs
|
||||
|
||||
|
||||
def _capture_output(callback):
|
||||
@@ -76,3 +79,43 @@ def test_structured_logger_redacts_sensitive_text_and_context():
|
||||
assert "hunter2" not in output
|
||||
assert "[REDACTED]" in output
|
||||
assert '"safe": "visible"' in output
|
||||
|
||||
|
||||
def test_business_context_redacts_nested_sensitive_values():
|
||||
context = business_logs.build_business_context(
|
||||
{
|
||||
"provider": "openai",
|
||||
"api_key": "sk-secret",
|
||||
"nested": {
|
||||
"token": "plain-token",
|
||||
"safe": "visible",
|
||||
},
|
||||
}
|
||||
)
|
||||
|
||||
assert context["api_key"] == "[REDACTED]"
|
||||
assert context["nested"]["token"] == "[REDACTED]"
|
||||
assert context["nested"]["safe"] == "visible"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_emit_business_log_persists_sanitized_system_event(monkeypatch):
|
||||
events = []
|
||||
|
||||
async def fake_record_system_log(**payload):
|
||||
events.append(payload)
|
||||
|
||||
monkeypatch.setattr(business_logs, "record_system_log", fake_record_system_log)
|
||||
|
||||
await business_logs.emit_business_log(
|
||||
get_logger("tests.business"),
|
||||
event="ai.provider.analyze.success",
|
||||
message="AI request completed",
|
||||
category="ai",
|
||||
context={"model": "gpt-test", "api_key": "sk-secret"},
|
||||
)
|
||||
|
||||
assert events[0]["event"] == "ai.provider.analyze.success"
|
||||
assert events[0]["category"] == "ai"
|
||||
assert events[0]["context"]["model"] == "gpt-test"
|
||||
assert events[0]["context"]["api_key"] == "[REDACTED]"
|
||||
|
||||
@@ -816,6 +816,27 @@ async def test_visualization_geo_summary_returns_counts(monkeypatch):
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_empty_cable_layers_return_empty_feature_collections():
|
||||
class _ScalarResult:
|
||||
def scalars(self):
|
||||
class _Scalars:
|
||||
def all(self):
|
||||
return []
|
||||
|
||||
return _Scalars()
|
||||
|
||||
class _FakeSession:
|
||||
async def execute(self, _query):
|
||||
return _ScalarResult()
|
||||
|
||||
cables = await visualization_api._build_cables_geojson(_FakeSession())
|
||||
landing_points = await visualization_api._build_landing_points_geojson(_FakeSession())
|
||||
|
||||
assert cables == {"type": "FeatureCollection", "features": []}
|
||||
assert landing_points == {"type": "FeatureCollection", "features": []}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_collect_location_endpoint_returns_candidates_for_known_record(monkeypatch):
|
||||
def _fake_ror(query):
|
||||
|
||||
@@ -15,12 +15,16 @@ services:
|
||||
retries: 10
|
||||
|
||||
aiprovider:
|
||||
image: ${AI_PROVIDER_IMAGE_NAME:-planet-aiprovider:latest}
|
||||
build:
|
||||
context: .
|
||||
dockerfile: aiprovider/Dockerfile
|
||||
args:
|
||||
PYTHON_IMAGE: ${PYTHON_IMAGE:-python:3.14-slim}
|
||||
UV_IMAGE: ${UV_IMAGE:-ghcr.io/astral-sh/uv:latest}
|
||||
AI_PROVIDER_BUILD_FINGERPRINT: ${AI_PROVIDER_BUILD_FINGERPRINT:-unknown}
|
||||
secrets:
|
||||
- planet_uv_config
|
||||
container_name: planet_aiprovider
|
||||
ports:
|
||||
- "8010:8010"
|
||||
@@ -43,3 +47,7 @@ services:
|
||||
|
||||
volumes:
|
||||
ollama_data:
|
||||
|
||||
secrets:
|
||||
planet_uv_config:
|
||||
file: ${PLANET_UV_CONFIG_FILE:-/dev/null}
|
||||
|
||||
@@ -2,12 +2,16 @@ version: '3.8'
|
||||
|
||||
services:
|
||||
aiprovider:
|
||||
image: ${AI_PROVIDER_IMAGE_NAME:-planet-aiprovider:latest}
|
||||
build:
|
||||
context: .
|
||||
dockerfile: aiprovider/Dockerfile
|
||||
args:
|
||||
PYTHON_IMAGE: ${PYTHON_IMAGE:-python:3.14-slim}
|
||||
UV_IMAGE: ${UV_IMAGE:-ghcr.io/astral-sh/uv:latest}
|
||||
AI_PROVIDER_BUILD_FINGERPRINT: ${AI_PROVIDER_BUILD_FINGERPRINT:-unknown}
|
||||
secrets:
|
||||
- planet_uv_config
|
||||
env_file:
|
||||
- ./aiprovider/.env
|
||||
- ${PLANET_AI_PROVIDER_RUNTIME_ENV_FILE:-./aiprovider/.env}
|
||||
@@ -43,3 +47,7 @@ services:
|
||||
volumes:
|
||||
postgres_data:
|
||||
redis_data:
|
||||
|
||||
secrets:
|
||||
planet_uv_config:
|
||||
file: ${PLANET_UV_CONFIG_FILE:-/dev/null}
|
||||
|
||||
@@ -2,12 +2,16 @@ version: '3.8'
|
||||
|
||||
services:
|
||||
aiprovider:
|
||||
image: ${AI_PROVIDER_IMAGE_NAME:-planet-aiprovider:latest}
|
||||
build:
|
||||
context: .
|
||||
dockerfile: aiprovider/Dockerfile
|
||||
args:
|
||||
PYTHON_IMAGE: ${PYTHON_IMAGE:-python:3.14-slim}
|
||||
UV_IMAGE: ${UV_IMAGE:-ghcr.io/astral-sh/uv:latest}
|
||||
AI_PROVIDER_BUILD_FINGERPRINT: ${AI_PROVIDER_BUILD_FINGERPRINT:-unknown}
|
||||
secrets:
|
||||
- planet_uv_config
|
||||
env_file:
|
||||
- ./aiprovider/.env
|
||||
- ${PLANET_AI_PROVIDER_RUNTIME_ENV_FILE:-./aiprovider/.env}
|
||||
@@ -53,3 +57,7 @@ services:
|
||||
volumes:
|
||||
postgres_data:
|
||||
redis_data:
|
||||
|
||||
secrets:
|
||||
planet_uv_config:
|
||||
file: ${PLANET_UV_CONFIG_FILE:-/dev/null}
|
||||
|
||||
@@ -8,6 +8,135 @@ This project follows the repository versioning rule:
|
||||
- `improvement` -> `+0.0.1`(bugfix + 小功能混合)
|
||||
- `bugfix` -> `+0.0.1`
|
||||
|
||||
## [0.66.1] — 2026-05-26
|
||||
|
||||
Released: 2026-05-26
|
||||
|
||||
### Highlights
|
||||
- 修复 CelesTrak active 更新窗口内清库后无法恢复的问题,新增持久原始下载缓存和完整 fallback group mode。
|
||||
- 避免数据源任务 WebSocket 与轮询同时完成时重复弹出采集失败 toast。
|
||||
- `planet.sh destroy` 保留上游原始下载缓存,数据库清空后仍可用缓存重灌。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- CelesTrak 403 `GP data has not updated` 会先复用 active 缓存;无 active 缓存时完整拉取 `starlink/gps-ops/galileo/glonass/beidou/leo/geo/iridium-next`,任一 group 缺失则整体失败,不保存 partial。
|
||||
- 下载器缓存从 `/tmp` 迁到 `$PLANET_CACHE_DIR/downloads`,并保留 HTTP 错误响应正文以支持上游限频语义判断。
|
||||
- 补充 CelesTrak cache/fallback 回归测试和中英文运维/卫星策略文档。
|
||||
|
||||
---
|
||||
|
||||
## [0.66.0] — 2026-05-26
|
||||
|
||||
Released: 2026-05-26
|
||||
|
||||
### Highlights
|
||||
- 将 Admin 正式化为唯一控制台入口,移除旧 AntD 后台、Admin Next 路由痕迹和相关依赖。
|
||||
- 引入 PostgreSQL 数据作业队列、Earth outbox 同步和可交互对象管线,让采集、清理和 Earth 刷新进入可追踪异步链路。
|
||||
- 强化 AI、AI 工具、设置连接测试和采集任务的结构化业务日志,关键事件可在系统日志中检索。
|
||||
- CelesTrak 轨道根数采集改为完整 `active` 目录下载、续传和重试,避免部分分组失败时保存不完整卫星数据。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- Admin 数据源页修复内置源调度启停按钮,使用 `is_active` 判断 `/enable` 与 `/disable`,避免采集状态误导调度开关。
|
||||
- Earth 新增通用 interactables 层和平台数据流文档,支持对象级 delta 同步与后续小图层扩展。
|
||||
- `planet.sh init/start/destroy` 补齐 uv 镜像回退、Bun unzip 依赖、HTTPS/LAN 跳转协议和 OOBE 清理语义。
|
||||
- Markdown code block、数据源队列、演示模式、About 版本展示、TV 链接协议和文档索引同步完成。
|
||||
|
||||
---
|
||||
|
||||
## [0.65.2] — 2026-05-22
|
||||
|
||||
Released: 2026-05-22
|
||||
|
||||
### Highlights
|
||||
- Stabilize AI Provider rebuild detection so unchanged source no longer rebuilds just because file timestamps or local cache state changed.
|
||||
- Keep `uv.lock` on the official registry while still allowing local and Docker builds to use the user's `uv.toml`.
|
||||
- Prevent local startup and bootstrap commands from rewriting `uv.lock` when a user-level mirror is configured.
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- AI Provider fingerprints now use content hashes, base image inputs, dependency metadata, and an image label instead of file mtimes.
|
||||
- `planet.sh` and `scripts/bootstrap-dev.sh` now use `uv sync --frozen`, and backend startup uses `uv run --frozen`.
|
||||
- Docker Compose passes the AI Provider fingerprint into the image build so future starts can inspect the image label directly.
|
||||
|
||||
---
|
||||
|
||||
## [0.65.1] — 2026-05-22
|
||||
|
||||
Released: 2026-05-22
|
||||
|
||||
### Highlights
|
||||
- 修复 `planet.sh` 与 Docker Compose 对 AI Provider 镜像名不一致导致的重复构建问题。
|
||||
- 让本地 `uv` 与 Docker build 统一使用用户机器上的 `uv.toml`,避免把镜像源写入 `uv.lock`。
|
||||
- 通过 BuildKit secret 向容器构建传入 uv 配置,保留用户源选择且不把配置写入镜像层。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- `planet.sh` 自动解析 `UV_CONFIG_FILE`、仓库 `uv.toml` 与用户级 `~/.config/uv/uv.toml`,并清理会覆盖配置的 `UV_INDEX_URL` 环境变量。
|
||||
- AI Provider 和后端 Dockerfile 在 `uv sync` 阶段挂载 uv 配置 secret,并复用 BuildKit uv cache。
|
||||
- Docker Compose 三套配置统一声明 `planet-aiprovider:latest` 镜像名和 `planet_uv_config` build secret。
|
||||
|
||||
---
|
||||
|
||||
## [0.65.0] — 2026-05-21
|
||||
|
||||
Released: 2026-05-21
|
||||
|
||||
### Highlights
|
||||
- 收敛 Admin Next 数据源触发入口:主按钮在未勾选时触发全部,勾选内置源后切换为“触发已选 N”,并移除手填 ID 的批量触发弹窗。
|
||||
- 优化数据源采集队列入口:右上角按钮常驻,空态显示队列图标,有任务时显示纯圆环进度,队列改为浮层避免挤压表格。
|
||||
- 强化 `planet.sh destroy` 清理语义,销毁时先硬重置运行中的本地 Postgres `public` schema,避免残留采集数据让 OOBE 误判 ready。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- Admin Next 表格新增可选选择列,仅在 `/datasources` 内置源分区启用,支持当前可见行全选并在筛选、切分区或刷新时清空选择。
|
||||
- 数据源批量触发复用 `/datasources/trigger-batch` 的 `source_ids`,成功后写入现有采集队列并清空勾选。
|
||||
- `destroy` 补充清理 `planet-aiprovider:latest` 镜像以及 Python/Vite 等本地编译缓存,同时保留源码和 `.env`。
|
||||
- Docs Gatekeeper 与 Tactile UI 文档/样式继续补齐,覆盖本轮按钮、队列、OOBE 和销毁流程说明。
|
||||
|
||||
---
|
||||
|
||||
## [0.64.0] — 2026-05-21
|
||||
|
||||
Released: 2026-05-21
|
||||
|
||||
### Highlights
|
||||
- 新增 Earth 首次初始化 OOBE,由后端真实采集状态决定是否显示,避免 localStorage 清空后误弹,并提供桌面毛玻璃引导与移动端 bottom sheet。
|
||||
- 数据源页新增下载列表式采集队列,把单源、批量和触发全部的任务进度统一展示,并支持失败重试与跳转详情。
|
||||
- Earth 内容新增“关于”配置接口和后台 tab,Earth 设置页 About 卡片改为运行时读取配置并带默认 fallback。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 新增 `/api/v1/earth/oobe-status`、`/api/v1/earth/about` GET/PUT/DELETE,并让 Earth 前端加载 `about.js` 与 `oobe.js`。
|
||||
- Admin Next 数据源队列优先消费 `datasource_tasks` WebSocket,断线时轮询 `/datasources/{id}/task-status`,刷新后只恢复后端仍在运行的真实任务。
|
||||
- Admin Next 深色主题滑块补齐 Docs 同款 dark token,侧栏主题控件在 dark 模式下不再保持浅色底座。
|
||||
- 用户手册、快速开始、Earth 前端上下文和 Admin 前端上下文同步记录 OOBE、采集队列、About 配置与主题滑块行为。
|
||||
|
||||
---
|
||||
|
||||
## [0.63.1] — 2026-05-21
|
||||
|
||||
Released: 2026-05-21
|
||||
|
||||
### Highlights
|
||||
- 修复新设备初始化后 Admin Next 无法打开的问题:补上被 `.gitignore` 的 `lib/` 规则误忽略的 Admin Next utility module。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 将 `frontend/src/admin-next/lib/utils.ts` 纳入版本控制,恢复 `DashboardNext`、`AdminNextLayout` 等页面对 `cn` 和 `formatNumber` 的运行时依赖。
|
||||
- 避免 Vite dev server 在新 clone 环境中因缺失源码模块而对 `/src/admin-next/*` 返回 500,并导致动态导入 AdminNextRoutes 失败。
|
||||
|
||||
---
|
||||
|
||||
## [0.63.0] — 2026-05-21
|
||||
|
||||
Released: 2026-05-21
|
||||
|
||||
### Highlights
|
||||
- 新增 `./planet.sh init` 首次初始化入口,将 uv/bun 依赖同步、env 模板补齐、数据库容器启动、建表 seed 和默认用户生成串成一条空项目引导路径。
|
||||
- 新增 `./planet.sh destroy` 破坏性重置入口,带 CLI 确认保护,可清理 Planet 容器、卷、镜像和本地编译/运行状态,同时保留源码与 `.env` 配置。
|
||||
- 改进 `planet.sh` 日志体验,只在带状态标签的输出行末尾追加时间戳,并让 `init` 在应用服务已运行时自动跳过重复初始化。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- `init` 复用现有 Docker/依赖 helper,支持重复执行时不覆盖 env、不清空数据,并在完成后提示默认本地登录账号。
|
||||
- `destroy` 停止本地服务后清理 Docker compose 状态、项目镜像、数据卷、`.venv`、`node_modules`、前端构建产物和 Planet state/cache。
|
||||
- README 快速启动补充 `init` 和 `destroy` 命令说明,明确首次引导和重置路径。
|
||||
|
||||
---
|
||||
|
||||
## [0.62.0] — 2026-05-21
|
||||
|
||||
Released: 2026-05-21
|
||||
|
||||
@@ -16,7 +16,24 @@
|
||||
- 已被正式实现替代、继续放在 `docs/` 根目录会误导后续开发的计划,归档
|
||||
- 仍然指导未来开发、尚未完成或仍有明确执行价值的文档,继续保留在 `docs/`
|
||||
|
||||
当前替代入口:
|
||||
|
||||
- 业务和数据产品链路见 [业务架构与数据流转](/home/ray/dev/linkong/planet/docs/technical/zh/platform-data-flows.md)。
|
||||
- 当前后端作业、outbox 和 Earth 同步实现见 [数据作业与 Outbox 技术架构](/home/ray/dev/linkong/planet/docs/technical/zh/data-job-earth-sync-architecture.md)。
|
||||
- 用户操作流程见 [Planet 使用手册](/home/ray/dev/linkong/planet/docs/technical/zh/manual.md)。
|
||||
- 当前代码结构见 [技术文档索引](/home/ray/dev/linkong/planet/docs/technical/zh/README.md)。
|
||||
|
||||
补充说明:
|
||||
|
||||
- 一部分归档文档来自外部或临时工作流草案,例如 sisyphus 生成的初稿
|
||||
- 这类文档如果有可用内容,应先吸收到 `docs/plans/` 或 `docs/technical/`,再归档保留来源记录
|
||||
|
||||
## 近期归档
|
||||
|
||||
- [Docs Gatekeeper 鉴权系统计划](/home/ray/dev/linkong/planet/docs/deprecated/docs-gatekeeper-auth-plan.md):已落地,当前实现见技术文档。
|
||||
- [Location Resolver 共享管线计划](/home/ray/dev/linkong/planet/docs/deprecated/location-resolver-shared-pipeline-plan.md):已落地,当前实现见技术文档。
|
||||
- [Earth Surface Hover Info Plan](/home/ray/dev/linkong/planet/docs/deprecated/earth-surface-hover-info-plan.md):已实现,保留为历史记录。
|
||||
- [Admin Next Dual Track Full Migration Plan](/home/ray/dev/linkong/planet/docs/deprecated/admin-next-dual-track-full-migration-plan.md):已被当前 `frontend/src/admin/` 控制台结构替代。
|
||||
- [Admin Next Soft Glass Goal Driven Plan](/home/ray/dev/linkong/planet/docs/deprecated/admin-next-soft-glass-goal-driven-plan.md):已被当前 `frontend/src/admin/` 控制台结构替代。
|
||||
- [Admin Next Parity Checklist](/home/ray/dev/linkong/planet/docs/deprecated/admin-next-parity-checklist.md):旧 `/admin-next/*` 验收资料,保留为迁移历史。
|
||||
- [Admin Next Parity Audit Closeout](/home/ray/dev/linkong/planet/docs/deprecated/admin-next-parity-audit-closeout-plan.md):旧 `/admin-next/*` 审计资料,保留为迁移历史。
|
||||
|
||||
@@ -104,7 +104,7 @@ The old AntD page itself treated these Earth content tabs as placeholder-level c
|
||||
- `models_3d`
|
||||
- `news_anchor_strategy`
|
||||
|
||||
If backend endpoints are later added, these items must be promoted into `docs/plans/admin-next-parity-checklist.md` with concrete API and UI acceptance criteria.
|
||||
If backend endpoints are later added, these items must be promoted into a new active plan under `docs/plans/` with concrete API and UI acceptance criteria.
|
||||
|
||||
## Final Gate
|
||||
|
||||
@@ -114,7 +114,7 @@ After the route promotion, the final gate is no longer “switch old routes.”
|
||||
2. Run the static checks:
|
||||
- `rg "map: \\(\\) => \\[\\]|暂不支持保存|placeholder" frontend/src/admin-next`
|
||||
- `rg "ShadowPage|FeatureConsole|GlassPanel|InspectorDrawer" frontend/src/admin-next`
|
||||
3. Manually verify every official route listed in `docs/plans/admin-next-parity-checklist.md`.
|
||||
3. Manually verify every official route listed in `docs/deprecated/admin-next-parity-checklist.md`.
|
||||
4. Confirm `/legacy/admin/*` still opens old AntD pages during the validation window.
|
||||
5. Delete old AntD pages and remove AntD dependencies only as a separate final cleanup task after explicit confirmation.
|
||||
|
||||
@@ -8,7 +8,7 @@ The redesign must cover desktop and mobile. Data display, icon semantics, table
|
||||
|
||||
## Criteria For Success
|
||||
|
||||
- This plan exists at `docs/plans/admin-next-soft-glass-goal-driven-plan.md`.
|
||||
- This archived plan exists at `docs/deprecated/admin-next-soft-glass-goal-driven-plan.md`.
|
||||
- `/admin-next/*` has real pages for every route; route usage of `ShadowPage` is removed.
|
||||
- Admin Next supports `system`, `light`, and `dark` theme modes using the same persistence and system-theme idea as Docs.
|
||||
- The visual language reads as soft-glass / light-neumorphic instead of an AntD reskin: translucent panels, fine borders, subtle glow, cool backgrounds, restrained accent colors, crisp icons, and tactile controls.
|
||||
@@ -214,4 +214,3 @@ Manual viewport checks:
|
||||
- low height;
|
||||
- 125% / 150% browser zoom;
|
||||
- light / dark / system theme modes.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Location Resolver Shared Pipeline Plan
|
||||
|
||||
**状态**:已实现,当前用户流程见 [Earth 位置候选采集使用手册](/home/ray/dev/linkong/planet/docs/technical/zh/location-pipeline-user.md),开发接口见 [通用位置估算管线开发说明](/home/ray/dev/linkong/planet/docs/technical/zh/location-pipeline-development.md)。
|
||||
**状态**:已实现,当前用户流程见 [Planet 使用手册](/home/ray/dev/linkong/planet/docs/technical/zh/manual.md) 的 Earth 位置候选采集章节,开发接口见 [通用位置估算管线开发说明](/home/ray/dev/linkong/planet/docs/technical/zh/location-pipeline-development.md)。
|
||||
|
||||
## Goal
|
||||
|
||||
@@ -37,7 +37,7 @@ If the same action has both a UI and a CLI path (e.g. user creation), describe t
|
||||
- a `credential_provider` in `backend/app/core/datasource_defaults.py`;
|
||||
- a default credential guide in `backend/app/services/credential_guides.py`;
|
||||
- a supported connectivity provider in `backend/app/services/datasource_connectivity.py`;
|
||||
- settings UI guidance or a credential form in `frontend/src/pages/Settings/Settings.tsx`;
|
||||
- settings UI guidance or a credential form in `frontend/src/admin/pages/PlainResourcePages.tsx`;
|
||||
- a regression test that fails if the guide/provider is missing.
|
||||
|
||||
## Recommended Checks
|
||||
|
||||
@@ -7,6 +7,12 @@
|
||||
- 分几期做
|
||||
- 当前差距和下一步是什么
|
||||
|
||||
当前实现、业务架构和数据链路不放在这里。它们分别进入:
|
||||
|
||||
- [业务架构与数据流转](/home/ray/dev/linkong/planet/docs/technical/zh/platform-data-flows.md)
|
||||
- [数据作业与 Outbox 技术架构](/home/ray/dev/linkong/planet/docs/technical/zh/data-job-earth-sync-architecture.md)
|
||||
- [技术文档索引](/home/ray/dev/linkong/planet/docs/technical/zh/README.md)
|
||||
|
||||
适合放入这里的内容:
|
||||
|
||||
- Earth / BGP / 地形 / 天球实施方案
|
||||
@@ -16,38 +22,40 @@
|
||||
|
||||
当前重点入口:
|
||||
|
||||
- [earth-mobile-drawer-ui-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-mobile-drawer-ui-plan.md)
|
||||
- [earth-compute-center-bgp-style-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-compute-center-bgp-style-plan.md)
|
||||
- [earth-renderer-architecture-separation-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-renderer-architecture-separation-plan.md)
|
||||
- [Earth Mobile Drawer UI Plan](/home/ray/dev/linkong/planet/docs/plans/earth-mobile-drawer-ui-plan.md)
|
||||
- [Earth Compute Center BGP Style Plan](/home/ray/dev/linkong/planet/docs/plans/earth-compute-center-bgp-style-plan.md)
|
||||
- [Earth Renderer Architecture Separation Plan](/home/ray/dev/linkong/planet/docs/plans/earth-renderer-architecture-separation-plan.md)
|
||||
- [Earth 高精度国界静态瓦片计划](/home/ray/dev/linkong/planet/docs/plans/earth-high-precision-boundary-tiles-plan.md)
|
||||
- [earth-predicted-orbit-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-predicted-orbit-plan.md)
|
||||
- [earth-webgl-instancing-satellites-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-webgl-instancing-satellites-plan.md)
|
||||
- [earth-real-terrain-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-real-terrain-plan.md)
|
||||
- [earth-news-source-configuration-and-collector-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-news-source-configuration-and-collector-plan.md)
|
||||
- [earth-news-cruise-summary-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-news-cruise-summary-plan.md)
|
||||
- [Earth Predicted Orbit Plan](/home/ray/dev/linkong/planet/docs/plans/earth-predicted-orbit-plan.md)
|
||||
- [Earth WebGL Instancing Satellites Plan](/home/ray/dev/linkong/planet/docs/plans/earth-webgl-instancing-satellites-plan.md)
|
||||
- [Earth Real Terrain Plan](/home/ray/dev/linkong/planet/docs/plans/earth-real-terrain-plan.md)
|
||||
- [Earth News Source Configuration And Collector Plan](/home/ray/dev/linkong/planet/docs/plans/earth-news-source-configuration-and-collector-plan.md)
|
||||
- [Earth News Cruise Summary Plan](/home/ray/dev/linkong/planet/docs/plans/earth-news-cruise-summary-plan.md)
|
||||
- [Earth 动作捕捉手势控制计划](/home/ray/dev/linkong/planet/docs/plans/earth-motion-capture-gesture-control-plan.md)
|
||||
- [Earth 动捕交互语义 V2 计划](/home/ray/dev/linkong/planet/docs/plans/earth-motion-gesture-interaction-v2-plan.md)
|
||||
- [Earth Presentation 解耦架构计划](/home/ray/dev/linkong/planet/docs/plans/earth-presentation-decoupled-architecture-plan.md)
|
||||
- [earth-vessel-rendering-performance-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-vessel-rendering-performance-plan.md)
|
||||
- [Earth Vessel Rendering Performance Plan](/home/ray/dev/linkong/planet/docs/plans/earth-vessel-rendering-performance-plan.md)
|
||||
- [AIS 多源采集、冲突记录与聚合接口计划](/home/ray/dev/linkong/planet/docs/plans/earth-vessel-ais-aggregation-plan.md)
|
||||
- [earth-interactable-layer-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-interactable-layer-plan.md)
|
||||
- [frontend-public-docs-site-plan.md](/home/ray/dev/linkong/planet/docs/plans/frontend-public-docs-site-plan.md)
|
||||
- [Docs Gatekeeper 鉴权系统计划](/home/ray/dev/linkong/planet/docs/plans/docs-gatekeeper-auth-plan.md)
|
||||
- [Location Resolver 共享管线计划](/home/ray/dev/linkong/planet/docs/plans/location-resolver-shared-pipeline-plan.md)
|
||||
- [frontend-ai-playground-development-plan.md](/home/ray/dev/linkong/planet/docs/plans/frontend-ai-playground-development-plan.md)
|
||||
- [Earth Interactable Layer Plan](/home/ray/dev/linkong/planet/docs/plans/earth-interactable-layer-plan.md)
|
||||
- [Frontend Public Docs Site Plan](/home/ray/dev/linkong/planet/docs/plans/frontend-public-docs-site-plan.md)
|
||||
- [Frontend AI Playground Development Plan](/home/ray/dev/linkong/planet/docs/plans/frontend-ai-playground-development-plan.md)
|
||||
- [AI Provider OpenClaw-Style Routing Plan](/home/ray/dev/linkong/planet/docs/plans/ai-provider-openclaw-style-routing-plan.md)
|
||||
- [统一集成配置 Schema 系统计划](/home/ray/dev/linkong/planet/docs/plans/integration-config-schema-system-plan.md)
|
||||
- [Lightweight Agent Orchestrator 与 WebSearch 证据层计划](/home/ray/dev/linkong/planet/docs/plans/agents-light-orchestrator-websearch-plan.md)
|
||||
- [Admin Next Parity Checklist](/home/ray/dev/linkong/planet/docs/plans/admin-next-parity-checklist.md)
|
||||
- [Admin Next Parity Audit Closeout](/home/ray/dev/linkong/planet/docs/plans/admin-next-parity-audit-closeout-plan.md)
|
||||
- [ue5-mvp-fused-plan.md](/home/ray/dev/linkong/planet/docs/plans/ue5-mvp-fused-plan.md)
|
||||
- [UE5 MVP Fused Plan](/home/ray/dev/linkong/planet/docs/plans/ue5-mvp-fused-plan.md)
|
||||
|
||||
历史计划入口:
|
||||
|
||||
- 已完成、已替代或只作为历史决策背景保留的文档,统一放在 [Deprecated Docs](/home/ray/dev/linkong/planet/docs/deprecated/README.md)。
|
||||
|
||||
不适合放入这里的内容:
|
||||
|
||||
- 当前代码结构说明
|
||||
- 组件现状和实现入口
|
||||
- 已经落地的技术上下文说明
|
||||
- 已经成为当前行为的“架构说明”
|
||||
|
||||
这些应放入:
|
||||
|
||||
- [docs/technical/README.md](/home/ray/dev/linkong/planet/docs/technical/README.md)
|
||||
- [技术文档索引](/home/ray/dev/linkong/planet/docs/technical/zh/README.md)
|
||||
- [Technical Docs Index](/home/ray/dev/linkong/planet/docs/technical/en/README.md)
|
||||
|
||||
@@ -178,5 +178,5 @@ Runtime 选择规则:
|
||||
- `backend/app/services/ai_client.py`
|
||||
- `aiprovider/main.py`
|
||||
- `aiprovider/provider_service.py`
|
||||
- `frontend/src/admin-next/pages/PlainResourcePages.tsx`
|
||||
- `docs/plans/admin-next-parity-audit-closeout-plan.md`
|
||||
- `frontend/src/admin/pages/PlainResourcePages.tsx`
|
||||
- `docs/deprecated/admin-next-parity-audit-closeout-plan.md`
|
||||
|
||||
@@ -57,7 +57,7 @@
|
||||
|
||||
抽自现 manual.md,重新组织:
|
||||
|
||||
1. 首次启动 — `./planet.sh start`、默认账号(`admin/admin123`、`linkong/12345678`,引用 `b15d097b` 引入的 `DEFAULT_LOGIN_USERS`)
|
||||
1. 首次启动 — `./planet.sh start`、默认账号(`admin/admin123`、`linkong/LK12345678`,引用 `b15d097b` 引入的 `DEFAULT_LOGIN_USERS`)
|
||||
2. 启停与按模块重启 — `start/stop/restart` 及 `-b -f -a -d`
|
||||
3. 健康检查 — `./planet.sh health`
|
||||
4. 日志 — `./planet.sh log` 及 `-f -b -a`,日志文件路径
|
||||
|
||||
@@ -1,46 +1,52 @@
|
||||
# Technical Docs
|
||||
|
||||
This directory holds "current implementation and current structure" documentation, focusing on:
|
||||
This is the current Planet documentation entry point. Docs are organized by reader path: start with business architecture to understand data products, then move into user manuals or implementation references.
|
||||
|
||||
- How the code is organized right now
|
||||
- Where the current entry points are
|
||||
- How state and components work
|
||||
- Which implementation boundaries future changes should follow
|
||||
## Business Architecture
|
||||
|
||||
What belongs here:
|
||||
- [Business Architecture and Data Flows](/home/ray/dev/linkong/planet/docs/technical/en/platform-data-flows.md): purpose, collection flow, fact tables, derived tables, cache, and WebSocket broadcast path for each Earth data product
|
||||
- [Naming Glossary](/home/ray/dev/linkong/planet/docs/technical/en/naming-glossary.md): English/Chinese terms used across the console, Earth, backend, and docs
|
||||
|
||||
- Quickstart and user manual
|
||||
- Frontend context
|
||||
- Earth frontend structure
|
||||
- Earth satellite footprint policy
|
||||
- Earth render layer order
|
||||
- Earth layer style property index
|
||||
- Backend runtime control
|
||||
- Collector status
|
||||
- Collector settings and connectivity validation
|
||||
- Earth Interactable integration
|
||||
- Collection format conventions
|
||||
## Manual
|
||||
|
||||
## Entry Points
|
||||
- [Quickstart](/home/ray/dev/linkong/planet/docs/technical/en/quickstart.md): shortest path to getting Planet running from scratch
|
||||
- [Planet Manual](/home/ray/dev/linkong/planet/docs/technical/en/manual.md): user workflows for the console, Earth, Docs, and common features
|
||||
- [FAQ](/home/ray/dev/linkong/planet/docs/technical/en/faq.md): troubleshooting for Windows / WSL, ports, dependencies, motion capture, credentials, and Docs permissions
|
||||
|
||||
- [Quickstart](/home/ray/dev/linkong/planet/docs/technical/en/quickstart.md): The shortest path to getting Planet running from scratch
|
||||
- [Planet Manual](/home/ray/dev/linkong/planet/docs/technical/en/manual.md): Complete usage guide for the console, `planet.sh`, Earth, and Docs
|
||||
- [FAQ](/home/ray/dev/linkong/planet/docs/technical/en/faq.md): Central troubleshooting entry for Windows / WSL, ports, dependencies, motion capture, credentials, and Docs permissions
|
||||
- [Earth Location Candidate Collection User Guide](/home/ray/dev/linkong/planet/docs/technical/en/location-pipeline-user.md): Collect and preview coordinate candidates for compute centers and BGP collectors on Earth
|
||||
- [Collector Settings and Connectivity Validation](/home/ray/dev/linkong/planet/docs/technical/en/datasource-collector-settings-connectivity.md): Data source catalog, collector settings, connectivity validation, and BarentsWatch credentials
|
||||
- [Shared Location Resolution Pipeline Development Guide](/home/ray/dev/linkong/planet/docs/technical/en/location-pipeline-development.md): Backend location resolver / pipeline interfaces, registries, and extension points
|
||||
- [Docs Gatekeeper Development Guide](/home/ray/dev/linkong/planet/docs/technical/en/docs-gatekeeper-development.md): Backend Docs catalog, Markdown content loading, and Gatekeeper permission groups
|
||||
- [Naming Glossary](/home/ray/dev/linkong/planet/docs/technical/en/naming-glossary.md): English/Chinese term mapping for the console, Earth, backend, and docs
|
||||
- [Earth Interactable Usage](/home/ray/dev/linkong/planet/docs/technical/en/earth-interactable-usage.md): API, lifecycle, and integration examples for Earth surface icon Interactable
|
||||
- [Earth Toolbar and Overlay Coordination](/home/ray/dev/linkong/planet/docs/technical/en/earth-toolbar-overlay-coordination.md): Closing matrix and integration rules for toolbar buttons, search, settings, news, and layer overlays
|
||||
- [Tactile UI Components](/home/ray/dev/linkong/planet/docs/technical/en/tactile-ui-components.md): Portable button, switch, tooltip, and scrollbar APIs, theme tokens, and migration rules
|
||||
## Earth Implementation
|
||||
|
||||
What does not belong here:
|
||||
- [Earth Frontend Context](/home/ray/dev/linkong/planet/docs/technical/en/earth-frontend-context.md): Earth modules, state, WebSocket refresh, and layer lifecycle
|
||||
- [Earth Layer Style Reference](/home/ray/dev/linkong/planet/docs/technical/en/earth-layer-style-reference.md): layer colors, symbols, materials, and visual parameters
|
||||
- [Earth Render Layer Order](/home/ray/dev/linkong/planet/docs/technical/en/earth-render-layer-order.md): renderOrder, depth strategy, picking, and collision avoidance
|
||||
- [Earth Satellite Footprint Policy](/home/ray/dev/linkong/planet/docs/technical/en/earth-satellite-footprint-policy.md): satellite footprint display boundaries and strategy
|
||||
- [BGP Context](/home/ray/dev/linkong/planet/docs/technical/en/earth-bgp-context.md): BGP rendering, aggregation, and collector implementation in Earth
|
||||
- [Earth Interactable Usage](/home/ray/dev/linkong/planet/docs/technical/en/earth-interactable-usage.md): `Interactable` API, lifecycle, and integration examples
|
||||
- [Earth Toolbar and Overlay Coordination](/home/ray/dev/linkong/planet/docs/technical/en/earth-toolbar-overlay-coordination.md): close matrix for toolbar buttons, search, settings, news, and layer overlays
|
||||
|
||||
- Incomplete roadmaps
|
||||
- Future iteration plans
|
||||
- Large-scale refactor proposals
|
||||
## Frontend Implementation
|
||||
|
||||
Those belong in:
|
||||
- [Admin Frontend Context](/home/ray/dev/linkong/planet/docs/technical/en/frontend-admin-frontend-context.md): console page structure, state boundaries, and lazy loading
|
||||
- [Frontend Layout Guidelines](/home/ray/dev/linkong/planet/docs/technical/en/frontend-layout-guidelines.md): layout, tables, panels, and responsive constraints
|
||||
- [Tactile UI Components](/home/ray/dev/linkong/planet/docs/technical/en/tactile-ui-components.md): button, switch, tooltip, scrollbar APIs, and theme tokens
|
||||
|
||||
- [Plans Index](/home/ray/dev/linkong/planet/docs/plans/README.md)
|
||||
## Backend Implementation
|
||||
|
||||
- [Data Collectors](/home/ray/dev/linkong/planet/docs/technical/en/backend-collectors.md): collectors, task types, save layer, and status updates
|
||||
- [System Service Control](/home/ray/dev/linkong/planet/docs/technical/en/backend-system-service-control.md): backend service control and system operation APIs
|
||||
- [Datasource Collector Settings and Connectivity](/home/ray/dev/linkong/planet/docs/technical/en/datasource-collector-settings-connectivity.md): datasource catalog, collector settings, connectivity validation, and credentials
|
||||
- [Datasource API Performance](/home/ray/dev/linkong/planet/docs/technical/en/backend-datasources-api-performance.md): DataSources list API performance and caching
|
||||
- [Data Jobs and Outbox Architecture](/home/ray/dev/linkong/planet/docs/technical/en/data-job-earth-sync-architecture.md): PostgreSQL job queue, outbox, listener, and Kafka / Spark evolution boundaries
|
||||
- [Shared Location Resolution Pipeline Development Guide](/home/ray/dev/linkong/planet/docs/technical/en/location-pipeline-development.md): location resolver / pipeline interfaces, registries, and extension points
|
||||
- [News Live Streams Collector Format](/home/ray/dev/linkong/planet/docs/technical/en/earth-news-live-streams-collector-format.md): news, live stream, and media collection payload conventions
|
||||
- [Docs Gatekeeper Development Guide](/home/ray/dev/linkong/planet/docs/technical/en/docs-gatekeeper-development.md): backend Docs catalog, Markdown content loading, and Gatekeeper permission groups
|
||||
|
||||
## Agents and Operations
|
||||
|
||||
- [AI Provider Guide](/home/ray/dev/linkong/planet/docs/technical/en/agents-aiprovider.md): model provider adapters, task prompts, and invocation boundaries
|
||||
- [Planet Ops Runbook](/home/ray/dev/linkong/planet/docs/technical/en/ops-runbook.md): deployment, startup, troubleshooting, and sensitive operations
|
||||
- [Docker + Compose + Buildx Upgrade](/home/ray/dev/linkong/planet/docs/technical/en/ops-docker-compose-buildx-upgrade.md): Docker toolchain upgrade steps
|
||||
- [planet.sh Startup](/home/ray/dev/linkong/planet/docs/technical/en/ops-planet-sh-startup.md): startup script, health checks, and performance optimization
|
||||
|
||||
## Plans and History
|
||||
|
||||
Incomplete roadmaps, large refactor proposals, and future plans live in the repository path `docs/plans/README.md`. Completed, outdated, or replaced designs live in `docs/deprecated/README.md`.
|
||||
|
||||
@@ -101,7 +101,7 @@ The AI settings page uses:
|
||||
|
||||
These endpoints require an authenticated user. The `secrets` endpoint is only used when the settings page reveals a key or token; hiding the field restores the masked preview.
|
||||
|
||||
Admin Next keeps the AI page aligned with the legacy information architecture:
|
||||
Admin keeps the AI page aligned with the legacy information architecture:
|
||||
|
||||
- `Model Providers`
|
||||
- Manages provider, wire adapter, default model, LLM API key, proxy URL, proxy token, model refresh, set-as-default, and lightweight connectivity testing.
|
||||
@@ -272,7 +272,7 @@ Each provider has its own key slot. Resolution order is:
|
||||
|
||||
`.env` is only a fallback. After the settings page saves successfully, or after the connection test succeeds, PostgreSQL becomes the global default source.
|
||||
|
||||
Admin Next must compute key status per provider or tool:
|
||||
Admin must compute key status per provider or tool:
|
||||
|
||||
- If the database has a key for the current provider/tool, show `configured`.
|
||||
- If the database has no key but the fallback provider, model, or tool matches the current item, show the fallback masked preview.
|
||||
@@ -283,7 +283,7 @@ Tool keys follow the same rule. WebSearch and OCR must match the current tool an
|
||||
|
||||
### Lightweight Connectivity Testing
|
||||
|
||||
The Admin Next plug button performs a lightweight connectivity check and does not save configuration. Common API-platform practice is two-tiered:
|
||||
The Admin plug button performs a lightweight connectivity check and does not save configuration. Common API-platform practice is two-tiered:
|
||||
|
||||
- Check a provider catalog or low-cost endpoint to validate base URL, authentication, and model reachability.
|
||||
- Send full model requests only when the user explicitly runs Playground or a business task.
|
||||
|
||||
@@ -73,6 +73,8 @@ async def run(self, db):
|
||||
|
||||
**Core file**: `backend/app/services/collectors/base.py`
|
||||
|
||||
Manual trigger, data clearing, and cache clearing now enter the PostgreSQL data job queue. `collection_tasks` remains the task ledger. Collectors only own `fetch -> transform -> save`; the `data_jobs.py` worker claims `collect` / `clear_data` / `clear_cache` / `earth_refresh` jobs and writes progress back. Earth layer refresh relationships live in `earth_layer_adapters.py`; do not hand-code cache invalidation or WebSocket broadcasts inside individual collectors or buttons.
|
||||
|
||||
## III. Collector List
|
||||
|
||||
| Collector | Data type | Content | Frequency |
|
||||
@@ -93,7 +95,7 @@ Earth boundaries are no longer data collectors. They are Earth static rendering
|
||||
|
||||
TOP500 and Epoch AI compute sources do not always provide usable coordinates. The unified Earth compute-center endpoint uses only valid source-provided coordinates or `compute_center_locations` dimension-table coordinates during the main map startup path; records without coordinates are returned as `unresolved` instead of being rendered from a local registry, country centroid, or guessed city. When users manually collect candidates, the backend queries ROR and Nominatim/OpenStreetMap from source fields; accepted candidates are saved into `compute_center_locations` and rendered from that table on the next layer refresh.
|
||||
|
||||
Admin Next collection management follows the business hierarchy instead of flattening every endpoint into one table:
|
||||
Admin collection management follows the business hierarchy instead of flattening every endpoint into one table:
|
||||
|
||||
- `Collectors`: endpoint, authentication, headers, base parameters, enabled state, and credential guides.
|
||||
- `Collection Schedule`: scheduler state and task controls.
|
||||
|
||||
@@ -96,4 +96,4 @@ if (res.data.task_id) {
|
||||
## Related Files
|
||||
|
||||
- [datasources.py](/home/ray/dev/linkong/planet/backend/app/api/v1/datasources.py): `_load_datasource_list_context`, `list_datasources`
|
||||
- [DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/DataSources/DataSources.tsx): `BuiltInDataSource`, `triggerDatasource`
|
||||
- [DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx): `BuiltInDataSource`, `triggerDatasource`
|
||||
|
||||
141
docs/technical/en/data-job-earth-sync-architecture.md
Normal file
141
docs/technical/en/data-job-earth-sync-architecture.md
Normal file
@@ -0,0 +1,141 @@
|
||||
# Data Jobs and Outbox Architecture
|
||||
|
||||
This document records the technical boundary for Planet v1 data jobs, database outbox, and Earth refresh. For each data product's business purpose and end-to-end flow, see [Business Architecture and Data Flows](/home/ray/dev/linkong/planet/docs/technical/en/platform-data-flows.md).
|
||||
|
||||
## Architecture Boundary
|
||||
|
||||
- PostgreSQL is the durable v1 job ledger and outbox. Kafka, Celery, and RQ are intentionally not part of v1.
|
||||
- `collection_tasks` records collection, data clearing, cache clearing, and non-database Earth refresh jobs.
|
||||
- `earth_data_change_events` records fact-table or derived-table changes and is the reliable source for Earth sync.
|
||||
- `LISTEN/NOTIFY` is only the low-latency wakeup path; the listener still polls unconsumed outbox rows.
|
||||
- Redis is mainly cache, auth helper, OTP / rate limit, temporary logs, and WebSocket support. It is not the durable queue.
|
||||
|
||||
## Database Change Sync
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Write["Fact or derived table write"] --> Trigger["PostgreSQL trigger"]
|
||||
Trigger --> Outbox["earth_data_change_events"]
|
||||
Trigger --> Notify["planet_earth_data_changes"]
|
||||
Outbox --> Listener["earth_db_change_listener"]
|
||||
Notify --> Listener
|
||||
Listener --> Adapter["earth_layer_adapters"]
|
||||
Adapter --> Cache["cache invalidation"]
|
||||
Cache --> WS["earth_updates"]
|
||||
```
|
||||
|
||||
1. Collection, clearing, location resolution, or projection jobs write fact or derived tables.
|
||||
2. Statement triggers write outbox rows for `INSERT` / `UPDATE` / `DELETE`.
|
||||
3. The listener wakes through notify or finds pending rows through polling.
|
||||
4. The listener maps `table + source` to Earth layers, refresh strategy, and cache patterns through `earth_layer_adapters.py`.
|
||||
5. The listener merges short-window same-layer events, invalidates cache, and broadcasts `earth_updates`.
|
||||
6. The outbox row is marked consumed only after successful broadcast; failed rows stay retryable.
|
||||
|
||||
DB changes no longer create default `earth_refresh` jobs, so they are not blocked by long same-source collection or clearing jobs. `earth_refresh` remains for manual cache clearing and non-DB refresh hints.
|
||||
|
||||
## Data Job Queue
|
||||
|
||||
`collection_tasks` is the unified job ledger. Workers claim `queued` jobs with PostgreSQL `FOR UPDATE SKIP LOCKED`; write jobs for the same `source` run serially, while different sources may run in parallel.
|
||||
|
||||
| task_type | Purpose |
|
||||
| --- | --- |
|
||||
| `collect` | Run a built-in datasource collector |
|
||||
| `clear_data` | Delete collected rows and declared derived rows for the source |
|
||||
| `clear_cache` | Delete Earth / dashboard cache for the source |
|
||||
| `earth_refresh` | Invalidate Earth layer cache and broadcast a refresh hint for non-DB changes |
|
||||
|
||||
API handlers only create jobs and return `task_id`. Execution, progress, cancellation, and terminal state are written back by workers and pushed to the frontend through the `datasource_tasks` channel.
|
||||
|
||||
Cancellation means “keep committed batches”: clicking stop marks the job as `cancelling` and cancels the in-memory coroutine. Already committed batches remain; unfinished batches follow the collector or cleanup rollback path.
|
||||
|
||||
## Earth Sync Event Model
|
||||
|
||||
The unified event model is `earth.layer.changed`:
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "earth.layer.changed",
|
||||
"action": "database_changed",
|
||||
"source": "celestrak_tle",
|
||||
"table": "collected_data",
|
||||
"operation": "DELETE",
|
||||
"layers": ["satellites"],
|
||||
"refresh_strategy": "clear_then_reload",
|
||||
"records_processed": 11125,
|
||||
"occurred_at": "2026-05-25T10:20:30Z"
|
||||
}
|
||||
```
|
||||
|
||||
| strategy | Purpose |
|
||||
| --- | --- |
|
||||
| `clear_then_reload` | Clear local frontend layer objects first, then force a refetch. Prefer this for deletes. |
|
||||
| `reload` | Keep old objects until fresh data returns. Use it for location, metadata, or non-destructive updates. |
|
||||
| `delta` | Used only for `earth_interactables`; upsert or remove objects by id. |
|
||||
|
||||
APIs must return HTTP 200 with an empty collection for real zero-data states; 5xx is reserved for real endpoint failures. After a delete event, if refetch fails, the frontend should keep the cleared state and show a lightweight error instead of restoring stale objects.
|
||||
|
||||
## Layer Adapter Contract
|
||||
|
||||
`earth_layer_adapters.py` is the single registry for sources, derived tables, Earth layers, cache patterns, and refresh strategy. New layers should be added through an adapter entry, not through one-off button handlers, collector branches, or frontend special cases.
|
||||
|
||||
Each adapter must declare:
|
||||
|
||||
- Which source or table feeds which Earth layer.
|
||||
- Which Earth cache key patterns must be invalidated.
|
||||
- Which owned derived tables must be removed during `clear_data`.
|
||||
- The default refresh strategy for that layer.
|
||||
|
||||
When a source is cleared, the `clear_data` job first deletes `collected_data.source = <source>`, then deletes adapter-owned derived rows. Direct derived-table edits also trigger the outbox, so background jobs, admin APIs, and SQL repair scripts reach Earth as long as they mutate fact or derived tables.
|
||||
|
||||
## Operations and Troubleshooting
|
||||
|
||||
Check whether outbox rows are piling up:
|
||||
|
||||
```sql
|
||||
SELECT id, table_name, operation, source, occurred_at
|
||||
FROM earth_data_change_events
|
||||
WHERE consumed_at IS NULL
|
||||
ORDER BY id
|
||||
LIMIT 20;
|
||||
```
|
||||
|
||||
Check that triggers exist:
|
||||
|
||||
```sql
|
||||
SELECT tgname, tgrelid::regclass
|
||||
FROM pg_trigger
|
||||
WHERE tgname LIKE 'tr_planet_%_changed_%'
|
||||
ORDER BY 2, 1;
|
||||
```
|
||||
|
||||
Useful log events:
|
||||
|
||||
- `earth.db_changes.connected`: the listener connected to PostgreSQL and started listening.
|
||||
- `earth.db_changes.outbox_polled`: polling found unconsumed outbox rows.
|
||||
- `earth.db_changes.broadcasted`: an Earth refresh broadcast was produced.
|
||||
- `data_job.started` / `data_job.completed`: job execution state.
|
||||
|
||||
If Earth does not update, check in order: fact table changed, outbox was consumed, adapter covers the `source/table`, the listener is online, frontend WebSocket is connected, and the visualization API returns HTTP 200 with either an empty collection or fresh data.
|
||||
|
||||
## Kafka-ready Boundaries
|
||||
|
||||
Business code avoids depending on a concrete queue implementation by preserving these boundaries:
|
||||
|
||||
- `JobQueue`: submit, claim, cancel, and complete data jobs.
|
||||
- `DataChangeBus`: publish database fact changes.
|
||||
- `EarthLayerAdapterRegistry`: declare source, layer, cache, and derived-data relationships.
|
||||
|
||||
Kafka becomes appropriate when:
|
||||
|
||||
- Several independent services must consume the same data-change stream.
|
||||
- AIS, BGP, or sensor streams become sustained high-throughput inputs.
|
||||
- Consumer groups, replay, and service decoupling are required.
|
||||
|
||||
Spark becomes appropriate when:
|
||||
|
||||
- Historical data reaches tens or hundreds of millions of rows and PostgreSQL aggregation becomes expensive.
|
||||
- Cross-source, long-window, spatiotemporal analysis is needed.
|
||||
- Raw data lands in Parquet / Iceberg / Delta and the system starts producing offline derived data products.
|
||||
|
||||
For second-level continuous stream processing, evaluate Flink first. Spark is a better fit for batch or micro-batch analytics.
|
||||
|
||||
@@ -32,8 +32,8 @@ If endpoint, headers, base configuration, or credential fingerprint changes afte
|
||||
|
||||
Files:
|
||||
|
||||
- [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/pages/PlainResourcePages.tsx)
|
||||
- [AdminNextRoutes.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/AdminNextRoutes.tsx)
|
||||
- [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
|
||||
- [AdminRoutes.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/AdminRoutes.tsx)
|
||||
|
||||
Current behavior:
|
||||
|
||||
@@ -57,7 +57,7 @@ Current behavior:
|
||||
|
||||
File:
|
||||
|
||||
- [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/pages/PlainResourcePages.tsx)
|
||||
- [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
|
||||
|
||||
Current behavior:
|
||||
|
||||
|
||||
@@ -66,10 +66,10 @@ DocsMetadata(
|
||||
)
|
||||
```
|
||||
|
||||
When adding a public technical doc:
|
||||
When adding a technical doc that should appear in the Docs page:
|
||||
|
||||
- Add both Chinese and English Markdown files.
|
||||
- Add filename, slug, access, group, order, and titles to server `DOCS_METADATA`.
|
||||
- Add filename, slug, access, group, order, and titles to server `DOCS_METADATA`. The backend catalog endpoint is authoritative; frontend metadata alone does not publish a document into `/docs` navigation.
|
||||
- Add matching metadata to frontend [docs-content.ts](/home/ray/dev/linkong/planet/frontend/src/pages/Docs/docs-content.ts) so navigation titles and sorting stay aligned.
|
||||
- Update `docs/technical/zh/README.md` and `docs/technical/en/README.md` when the document should be discoverable from the README.
|
||||
|
||||
@@ -83,7 +83,7 @@ The user API:
|
||||
- Validates group names on update: only `docs_user`, `docs_developer`, and `docs_admin` are accepted.
|
||||
- Allows only `super_admin` to modify Gatekeeper groups.
|
||||
|
||||
Frontend [Users.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Users/Users.tsx) displays group tags and provides a multi-select in the edit form. Non-`super_admin` users see the field disabled, and submission removes `gatekeeper_groups` before sending.
|
||||
Frontend [Users.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/Users.tsx) displays group tags and provides a multi-select in the edit form. Non-`super_admin` users see the field disabled, and submission removes `gatekeeper_groups` before sending.
|
||||
|
||||
## Frontend Docs Loading
|
||||
|
||||
|
||||
@@ -188,7 +188,7 @@ Current reality:
|
||||
- that is expected, because incidents are aggregated and de-noised
|
||||
- but incident-first rendering makes the Earth view look too quiet unless there is another always-available activity layer
|
||||
|
||||
Implementation detail for the recommended `activity layer` is expanded in the [BGP Region Aggregation Plan](/home/ray/dev/linkong/planet/docs/plans/earth-bgp-region-aggregation-plan.md).
|
||||
Implementation detail for the recommended `activity layer` is kept in the repository path `docs/plans/earth-bgp-region-aggregation-plan.md`.
|
||||
|
||||
So the immediate next milestone is:
|
||||
|
||||
@@ -259,7 +259,7 @@ Reference inspiration:
|
||||
|
||||
Relevant page:
|
||||
|
||||
- `frontend/src/pages/BGP/BGP.tsx`
|
||||
- `frontend/src/admin/pages/PlainResourcePages.tsx`
|
||||
|
||||
Current BGP console page has three levels:
|
||||
|
||||
@@ -321,7 +321,7 @@ Backend:
|
||||
|
||||
Frontend:
|
||||
|
||||
- `frontend/src/pages/BGP/BGP.tsx`
|
||||
- `frontend/src/admin/pages/PlainResourcePages.tsx`
|
||||
- `frontend/public/earth/js/bgp.js`
|
||||
- `frontend/public/earth/js/main.js`
|
||||
- `frontend/public/earth/js/info-card.js`
|
||||
|
||||
@@ -4,7 +4,7 @@ This document describes the current real structure of the Earth display frontend
|
||||
|
||||
Related references:
|
||||
|
||||
- [Project Rules](/home/ray/dev/linkong/planet/rules.md)
|
||||
- Repository root `rules.md`
|
||||
- [Frontend Layout Guidelines](/home/ray/dev/linkong/planet/docs/technical/en/frontend-layout-guidelines.md)
|
||||
|
||||
## Current Goal
|
||||
@@ -56,6 +56,8 @@ Responsibilities:
|
||||
- Layer module integration
|
||||
- Earth-level state synchronization
|
||||
|
||||
Earth treats `/ws` `earth_updates` as refresh hints only; real data is fetched again from `/api/v1/visualization/...`. Backend database-driven refresh now has the listener clear cache and broadcast directly instead of going through the default `earth_refresh` job queue. After `database_changed`, the frontend applies the per-layer `clear_then_reload`, `reload`, or `delta` strategy. `clear_then_reload` must clear Three.js objects before a no-store refetch, and summary is only a consistency check, not a reason to skip a layer reload when the count is `0`. See [Data Jobs and Outbox Architecture](/home/ray/dev/linkong/planet/docs/technical/en/data-job-earth-sync-architecture.md) for the technical pipeline and [Business Architecture and Data Flows](/home/ray/dev/linkong/planet/docs/technical/en/platform-data-flows.md) for business data flows.
|
||||
|
||||
### 3. Earth Control Layer
|
||||
|
||||
- [controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js)
|
||||
@@ -151,9 +153,14 @@ Each module is responsible for its own:
|
||||
|
||||
`brand.js` manages Earth HUD brand resources. Static assets provide the default brand; runtime overrides come from `/api/v1/earth/brand`, and uploaded images are served from `/earth-brand-assets/...`. The frontend must treat logo/title images and text fallback separately: if an image fails, show the text title; if text fields are empty, rely on backend defaults so the HUD brand area never renders blank. The console Earth Content page owns saving and resetting brand configuration; the Earth frontend only consumes it.
|
||||
|
||||
The Admin Next Earth Content page must preserve runtime semantics:
|
||||
`about.js` manages the About card inside Earth settings. Frontend defaults remain as a fallback, while runtime content is loaded from `/api/v1/earth/about`. If the request fails or fields are missing, the renderer must fall back per field so the settings page never renders an empty card. Admin exposes an Earth Content `About` tab; saving uses `PUT /api/v1/earth/about`, and restoring defaults uses `DELETE /api/v1/earth/about`.
|
||||
|
||||
`oobe.js` manages the first-run Earth initialization guide. OOBE visibility must be driven by `/api/v1/earth/oobe-status` and its `ready` field, not by `localStorage`. `localStorage` may only store a short-lived "skip on this browser" flag; if the backend reports `ready: true`, logout, cleared browser storage, or a different browser must not show OOBE again. Desktop uses a dark starfield scrim and glass startup panel, while mobile uses a bottom sheet and respects `prefers-reduced-motion`.
|
||||
|
||||
The Admin Earth Content page must preserve runtime semantics:
|
||||
|
||||
- `Brand`: brand preview should use the same dark starfield background, size, spacing, logo/title rendering, and text fallback as the Earth HUD top-left brand block, not a generic form preview.
|
||||
- `About`: configures the About card in Earth settings, including logo, kicker, title, version, description, and metadata items. Earth runtime reads `/earth/about` and falls back to defaults on failure.
|
||||
- `Boundary Precision`: build boundary, refresh status, and restore defaults belong inside this section, not in the global page toolbar.
|
||||
- `TV`: the list distinguishes built-in, collected, and custom sources. Card state represents enabled, disabled, draft, or error. Built-in sources cannot be deleted; collected and custom sources can. A new live source only enters draft state after the plus button is clicked; save persists it into the list, while cancel destroys the draft.
|
||||
- `Basemap`, `Layer Resources`, `3D Models`, and `News Anchor Strategy`: if backend capability is not available yet, the console should show an explicit pending state instead of mixing those items into TV or brand configuration.
|
||||
|
||||
@@ -17,26 +17,21 @@ Related context:
|
||||
|
||||
## Current Local Categories
|
||||
|
||||
Current CelesTrak satellite groups in [backend/app/services/collectors/celestrak.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/celestrak.py) include:
|
||||
The CelesTrak collector now prefers the complete active satellite catalog from `GROUP=active&FORMAT=json`. This prevents one failed CelesTrak group request from being saved as a successful but incomplete batch. The collector only proceeds when the downloaded JSON is a parseable array and records include `NORAD_CAT_ID`; network, resume, or parsing failures are retried, and final failure preserves the previous current dataset.
|
||||
|
||||
- `starlink`
|
||||
- `gps-ops`
|
||||
- `galileo`
|
||||
- `glonass`
|
||||
- `beidou`
|
||||
- `leo`
|
||||
- `geo`
|
||||
- `iridium-next`
|
||||
CelesTrak applies a repeat-download window to large groups such as `active`. To make database resets recoverable, Planet stores raw downloads under `$PLANET_CACHE_DIR/downloads`. When `active` returns the CelesTrak "GP data has not updated" HTTP 403:
|
||||
|
||||
Non-Starlink categories:
|
||||
- If an `active` cache exists, the collector reuses it to repopulate the database.
|
||||
- If no `active` cache exists, the collector switches to fallback group mode and downloads `starlink`, `gps-ops`, `galileo`, `glonass`, `beidou`, `leo`, `geo`, and `iridium-next`.
|
||||
- Fallback group mode requires every group to succeed or have a reusable cache; if any group is missing, the whole collection fails and no partial dataset is saved.
|
||||
|
||||
- `gps-ops`
|
||||
- `galileo`
|
||||
- `glonass`
|
||||
- `beidou`
|
||||
- `leo`
|
||||
- `geo`
|
||||
- `iridium-next`
|
||||
The collector still provides `metadata.constellation_group` to the frontend, but the value now comes from executable inference:
|
||||
|
||||
- `OBJECT_NAME` starting with `STARLINK` is marked as `starlink`
|
||||
- `OBJECT_NAME` starting with `IRIDIUM` is marked as `iridium-next`
|
||||
- Other active satellites are not forced into the old CelesTrak small-group labels, because a broad source group is not an exact constellation
|
||||
|
||||
The product policy therefore still discusses GNSS/RNSS, GEO, generic LEO, and Iridium NEXT semantics. However, only fallback group mode stores the old `gps-ops`, `galileo`, `glonass`, `beidou`, `leo`, or `geo` labels; the active primary path does not force every satellite into those labels.
|
||||
|
||||
## Research Conclusions
|
||||
|
||||
@@ -136,7 +131,8 @@ This implementation only does the minimum executable version and does not change
|
||||
|
||||
1. Backend passes constellation group and footprint policy hint to the frontend
|
||||
|
||||
- CelesTrak collector stores `GROUP` in `metadata.constellation_group`
|
||||
- CelesTrak collector stores the source query in `metadata.celestrak_query_group = active`
|
||||
- `metadata.constellation_group` only stores inferred business constellations such as `starlink` and `iridium-next`
|
||||
- Visualization API outputs:
|
||||
- `properties.constellation_group`
|
||||
- `properties.footprint_policy`
|
||||
|
||||
@@ -4,7 +4,7 @@ This document describes the current real structure of the console frontend. The
|
||||
|
||||
Related references:
|
||||
|
||||
- [Project Rules](/home/ray/dev/linkong/planet/rules.md)
|
||||
- Repository root `rules.md`
|
||||
- [Frontend Layout Guidelines](/home/ray/dev/linkong/planet/docs/technical/en/frontend-layout-guidelines.md)
|
||||
|
||||
## Current Goal
|
||||
@@ -22,7 +22,7 @@ Main entry point:
|
||||
|
||||
- [App.tsx](/home/ray/dev/linkong/planet/frontend/src/App.tsx)
|
||||
|
||||
Admin Next now owns the official admin routes:
|
||||
Admin now owns the official admin routes:
|
||||
|
||||
- `/admin`
|
||||
- `/users`
|
||||
@@ -37,23 +37,7 @@ Admin Next now owns the official admin routes:
|
||||
- `/collection-management`
|
||||
- `/settings`
|
||||
|
||||
These routes render [AdminNextRoutes.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/AdminNextRoutes.tsx). Page metadata and menu entries come from [manifest.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/routes/manifest.tsx). `/admin-next/*` remains only as a compatibility entry and redirects to the official route; it is no longer a parallel primary entry.
|
||||
|
||||
The old AntD console remains available under `/legacy/admin/*` for comparison and rollback:
|
||||
|
||||
- `/legacy/admin`
|
||||
- `/legacy/admin/datasources`
|
||||
- `/legacy/admin/data`
|
||||
- `/legacy/admin/collection-management`
|
||||
- `/legacy/admin/earth-content`
|
||||
- `/legacy/admin/ai`
|
||||
- `/legacy/admin/logs`
|
||||
- `/legacy/admin/settings`
|
||||
- `/legacy/admin/users`
|
||||
- `/legacy/admin/bgp`
|
||||
- `/legacy/admin/alerts/*`
|
||||
|
||||
Legacy pages, `AppLayout`, `antd`, and `@ant-design/icons` stay in place during the legacy validation window. Do not remove them before Admin Next parity is accepted.
|
||||
These routes render [AdminRoutes.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/AdminRoutes.tsx). Page metadata and menu entries come from [manifest.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/routes/manifest.tsx). Admin is the only console entry point; there is no parallel console or rollback route.
|
||||
|
||||
`/earth` is a standalone display page and is not part of the console shell.
|
||||
|
||||
@@ -61,7 +45,7 @@ Legacy pages, `AppLayout`, `antd`, and `@ant-design/icons` stay in place during
|
||||
|
||||
The official admin shell is at:
|
||||
|
||||
- [AdminNextLayout.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/components/layout/AdminNextLayout.tsx)
|
||||
- [AdminLayout.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/components/layout/AdminLayout.tsx)
|
||||
|
||||
Responsibilities:
|
||||
|
||||
@@ -69,38 +53,13 @@ Responsibilities:
|
||||
- Current account, version, logout, and theme switching
|
||||
- Top search, breadcrumbs, and page shortcuts
|
||||
- Single-screen content-area height closure
|
||||
- Coordination for Admin Next internal scrolling, tables, detail panels, and mobile detail views
|
||||
- Coordination for Admin internal scrolling, tables, detail panels, and mobile detail views
|
||||
|
||||
The old AntD legacy shell remains at:
|
||||
Future official admin pages should adapt to `AdminLayout` and Admin page patterns; do not reintroduce a parallel admin shell.
|
||||
|
||||
- [AppLayout.tsx](/home/ray/dev/linkong/planet/frontend/src/components/AppLayout/AppLayout.tsx)
|
||||
## Admin Section Loading
|
||||
|
||||
Legacy responsibilities:
|
||||
|
||||
- Left-side navigation
|
||||
- Collapse and expand
|
||||
- Current account / version information
|
||||
- Content area height closure
|
||||
- Site-wide unified sidebar scrollbar
|
||||
|
||||
Current structure:
|
||||
|
||||
```tsx
|
||||
<Layout className="dashboard-layout">
|
||||
<Sider className="dashboard-sider">...</Sider>
|
||||
<Layout>
|
||||
<Content className="dashboard-content">
|
||||
<div className="dashboard-content-inner">{children}</div>
|
||||
</Content>
|
||||
</Layout>
|
||||
</Layout>
|
||||
```
|
||||
|
||||
Future official admin pages should adapt to `AdminNextLayout` and Admin Next page patterns rather than adding new capability to the old `AppLayout`. Only `/legacy/admin/*` maintenance should change the old shell.
|
||||
|
||||
## Admin Next Section Loading
|
||||
|
||||
Multi-tab pages are currently coordinated by [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/pages/PlainResourcePages.tsx), which hosts the current management and information workbench patterns. Section loading follows these rules:
|
||||
Multi-tab pages are currently coordinated by [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx), which hosts the current management and information workbench patterns. Section loading follows these rules:
|
||||
|
||||
- Initial page load requests only the active tab; it does not prefetch every tab endpoint.
|
||||
- Switching tabs lazily loads that tab. Loaded tabs stay cached in local `states`, so returning to a tab reuses the previous data.
|
||||
@@ -110,6 +69,29 @@ Multi-tab pages are currently coordinated by [PlainResourcePages.tsx](/home/ray/
|
||||
|
||||
This keeps Earth, AI, collection management, and other multi-section pages from flooding backend APIs on cold start while preserving a fast cached tab-switching experience. Full health checks should use backend health endpoints or explicit refresh flows rather than relying on page initialization to touch every business endpoint.
|
||||
|
||||
## Datasource Collection Queue
|
||||
|
||||
The Admin datasource page routes single-source trigger, table-selected trigger, and trigger-all into a browser-download-list style collection queue:
|
||||
|
||||
- Queue state is managed by [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx). It is a current-session visibility layer and does not fake task history in `localStorage`.
|
||||
- Progress first consumes the `/ws` `datasource_tasks` channel. If the socket is unavailable or stale, the page polls `/api/v1/datasources/{id}/task-status`.
|
||||
- Trigger responses immediately insert `triggered`, `skipped`, and `failed` items. After a refresh, the queue restores only real backend rows that are still `running`, `pending`, or `queued`.
|
||||
- The `Built-in Sources` section uses the table selection column for selected-source triggering. With no rows selected, the primary button is `Trigger All`; after selection, the same button becomes `Trigger Selected N`, replacing the old manual-ID batch button.
|
||||
- The expanded queue no longer lives in the page content flow, so trigger-all cannot squeeze the table and detail panel. The top-right actions area uses the existing `Button` styling; the empty state shows a `ListChecks` icon, and active queues show only a pure circular total-progress indicator. Clicking it opens a floating panel grouped by running, failed, completed, and skipped.
|
||||
- Queue rows can jump to the datasource detail panel, and failed rows can retry. The detail panel's task summary only reports the selected source's latest task; it does not save configuration.
|
||||
- While a single-source collection is running, the action button becomes `Stop Collection`; unfinished queue rows expose a cancel action on the right. Cancellation calls `/api/v1/datasources/{source_id}/tasks/{task_id}/cancel`; the backend keeps committed batches and rolls back unfinished work.
|
||||
- Clearing database data and clearing display cache enter the same queue. The frontend shows task state and does not assume the API completed synchronously.
|
||||
|
||||
This queue is a user-perception layer. Backend task status remains the only source of truth for running, completion, failure, and skipped decisions.
|
||||
|
||||
## Admin Theme Slider
|
||||
|
||||
The Admin sidebar theme switcher still reuses shared [SegmentedControl.tsx](/home/ray/dev/linkong/planet/frontend/src/components/SegmentedControl/SegmentedControl.tsx), while [styles.css](/home/ray/dev/linkong/planet/frontend/src/admin/styles.css) overrides the segment variables by `data-theme`:
|
||||
|
||||
- Light mode uses `--d-segment-bg: #eef3f9`, a white slider, and a light external shadow.
|
||||
- Dark mode uses the same dark base, `#202938` slider, and dark external shadow semantics as Docs.
|
||||
- Product code only hides the text label and keeps icon + tooltip behavior; it should not recreate the private slider DOM.
|
||||
|
||||
## Current Shared Components
|
||||
|
||||
### 1. `Scrollbar`
|
||||
@@ -143,7 +125,7 @@ Purpose:
|
||||
|
||||
Current usage:
|
||||
|
||||
- Admin Next data sources, collected data, collection management, logs, alerts, and BGP pages
|
||||
- Admin data sources, collected data, collection management, logs, alerts, and BGP pages
|
||||
- Old AntD legacy pages continue using shared scrolling behavior through compatibility wrappers
|
||||
|
||||
### 3. `TableScrollRegion`
|
||||
@@ -167,7 +149,7 @@ Files:
|
||||
|
||||
Purpose:
|
||||
|
||||
- Admin Next global tool buttons and detail-panel toolbars
|
||||
- Admin global tool buttons and detail-panel toolbars
|
||||
- Icon-only ordinary actions with tooltips
|
||||
- Strong-intent actions such as save, create, confirm, delete, and stop
|
||||
- Compact switches aligned with the Docs theme slider
|
||||
@@ -225,39 +207,19 @@ Current constraints:
|
||||
- Internal document links should be converted to `/docs/:slug` through `transformLink`
|
||||
- Heading anchors are injected through `getHeadingId`, keeping route state outside the renderer
|
||||
|
||||
### 7. `ConnectionTestInput`
|
||||
### 7. Admin UI Primitives
|
||||
|
||||
File:
|
||||
Files:
|
||||
|
||||
- [ConnectionTestInput.tsx](/home/ray/dev/linkong/planet/frontend/src/components/ConnectionTestInput/ConnectionTestInput.tsx)
|
||||
- [button.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/components/ui/button.tsx)
|
||||
- [dialog.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/components/ui/dialog.tsx)
|
||||
- [switch.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/components/ui/switch.tsx)
|
||||
|
||||
Purpose:
|
||||
Use:
|
||||
|
||||
- Console form fields that combine an endpoint/Base URL value with a connection check
|
||||
- Connection-test entry points for AI Provider and WebSearch
|
||||
- Future collector configuration fields should reuse it when the test action belongs inside the input
|
||||
|
||||
Current constraints:
|
||||
|
||||
- The input suffix shows a single plug/connector icon, not an adjacent text button
|
||||
- Disabled integrations must grey out both the input and its connection-test action
|
||||
- The component only combines the input and action; callers still own form state, loading, disabled state, and the request itself
|
||||
|
||||
### 8. `TableActions`
|
||||
|
||||
File:
|
||||
|
||||
- [TableActions.tsx](/home/ray/dev/linkong/planet/frontend/src/components/TableActions/TableActions.tsx)
|
||||
|
||||
Purpose:
|
||||
|
||||
- Shared action entry for table operation columns
|
||||
- Shows inline actions when expanded
|
||||
- Uses a more-actions dropdown when collapsed
|
||||
|
||||
Companion export:
|
||||
|
||||
- `actionCellProps`: for action-column `onCell`, preventing action buttons from being ellipsized or wrapped
|
||||
- Global tool buttons, detail actions, confirmation dialogs, and binary settings.
|
||||
- Aligned with Tactile UI tokens so Admin controls keep consistent size, hover, disabled, and dark-mode behavior.
|
||||
- Row actions should prefer icon buttons plus tooltip/title; do not reintroduce a separate action-menu component.
|
||||
|
||||
## Current State Sources
|
||||
|
||||
@@ -280,7 +242,7 @@ Responsibilities:
|
||||
|
||||
File:
|
||||
|
||||
- [AISettings.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/AISettings/AISettings.tsx)
|
||||
- [AISettings.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
|
||||
|
||||
Responsibilities:
|
||||
|
||||
@@ -316,7 +278,7 @@ Constraints:
|
||||
|
||||
Example:
|
||||
|
||||
- [Dashboard.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Dashboard/Dashboard.tsx)
|
||||
- [Dashboard.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/Dashboard.tsx)
|
||||
|
||||
Priority goals:
|
||||
|
||||
@@ -328,10 +290,10 @@ Priority goals:
|
||||
|
||||
Examples:
|
||||
|
||||
- [DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/DataSources/DataSources.tsx)
|
||||
- [DataList.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/DataList/DataList.tsx)
|
||||
- [Users.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Users/Users.tsx)
|
||||
- [Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Settings/Settings.tsx)
|
||||
- [DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
|
||||
- [DataList.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/DataList.tsx)
|
||||
- [Users.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/Users.tsx)
|
||||
- [Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
|
||||
|
||||
Constraints:
|
||||
|
||||
@@ -341,7 +303,7 @@ Constraints:
|
||||
|
||||
### Datasource Directory Page
|
||||
|
||||
[DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/DataSources/DataSources.tsx) is the datasource directory and collection operation page. It should not grow back into a configuration editor.
|
||||
[DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx) is the datasource directory and collection operation page. It should not grow back into a configuration editor.
|
||||
|
||||
Current page boundary:
|
||||
|
||||
@@ -355,7 +317,9 @@ Keep this boundary: do not put custom datasource editing, built-in endpoint over
|
||||
|
||||
### Collectors Page
|
||||
|
||||
[Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Settings/Settings.tsx) has three route modes: `/settings` for System Settings, `/earth-content` for Earth Content, and `/collection-management` for Collection Management. The `collector_credentials` tab is shown as `Collectors` under `/collection-management`.
|
||||
[Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx) has three route modes: `/settings` for System Settings, `/earth-content` for Earth Content, and `/collection-management` for Collection Management. The `collector_credentials` tab is shown as `Collectors` under `/collection-management`.
|
||||
|
||||
The `System Display` section under `/settings` includes the `Demo Mode` switch. When enabled, Earth OOBE ignores existing current collected data and the local `browse first` temporary skip state, then opens the initialization guide directly. This switch is only for demos and acceptance checks; it does not change datasources, collection queues, or Earth content resources.
|
||||
|
||||
Current boundary:
|
||||
|
||||
@@ -367,7 +331,7 @@ Current boundary:
|
||||
|
||||
### Earth Content Page
|
||||
|
||||
`/earth-content` reuses the same single-screen tab container from [Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Settings/Settings.tsx), but its ownership is separate from System Settings:
|
||||
`/earth-content` reuses the same single-screen tab container from [Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx), but its ownership is separate from System Settings:
|
||||
|
||||
- `TV Livestream` owns the Earth media-panel source configuration.
|
||||
- `Boundary Precision` owns the Earth static boundary asset state: provider, low-precision fallback, high-precision manifest/PMTiles, source JSON, and build action.
|
||||
@@ -379,8 +343,8 @@ Do not add Earth experience resources or collection-lifecycle tabs back into `/s
|
||||
|
||||
Examples:
|
||||
|
||||
- [BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx)
|
||||
- [Playground.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Playground/Playground.tsx)
|
||||
- [BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
|
||||
- [Playground.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
|
||||
|
||||
Constraints:
|
||||
|
||||
|
||||
@@ -8,7 +8,8 @@ Admin pages in this project default to a "single-screen workspace" layout standa
|
||||
|
||||
Current recommended reference implementations:
|
||||
|
||||
- [frontend/src/pages/BGP/BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx)
|
||||
- [frontend/src/admin/pages/PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
|
||||
- [frontend/src/admin/pages/DataList.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/DataList.tsx)
|
||||
- [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css)
|
||||
|
||||
## Core Principles
|
||||
@@ -23,15 +24,15 @@ Admin pages default to:
|
||||
Recommended structure:
|
||||
|
||||
```tsx
|
||||
<AppLayout>
|
||||
<AdminLayout>
|
||||
<div className="page-shell">
|
||||
<div className="page-shell__header">...</div>
|
||||
<div className="page-shell__body">...</div>
|
||||
</div>
|
||||
</AppLayout>
|
||||
</AdminLayout>
|
||||
```
|
||||
|
||||
Total page height should be bounded within the `AppLayout` content area, not allowed to grow naturally downward without limit.
|
||||
Total page height should be bounded within the `AdminLayout` content area, not allowed to grow naturally downward without limit.
|
||||
|
||||
### 2. Scrolling Should Happen Inside Modules
|
||||
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
`backend/app/services/location/` is the shared abstraction for any "given a record, decide its lat/lon" workflow. Compute centers, BGP collectors, and BGP events now run on this pipeline. Future entities such as satellite ground stations, user-claimed points, and IXP facilities should plug in here instead of creating another geocoding path.
|
||||
|
||||
For the user workflow, see [Earth Location Candidate Collection User Guide](/home/ray/dev/linkong/planet/docs/technical/en/location-pipeline-user.md).
|
||||
For the user workflow, see the Earth coordinate-candidate section in [Planet Manual](/home/ray/dev/linkong/planet/docs/technical/en/manual.md).
|
||||
|
||||
## Design Goals
|
||||
|
||||
|
||||
@@ -188,19 +188,22 @@ TV livestreams and boundary precision moved to `/earth-content`; collectors and
|
||||
`/earth-content` is under the console's Operations and Configuration group and owns resources used by the Earth frontend:
|
||||
|
||||
- **Brand Assets**: manages the logo, title image, title text, subtitle, and description used by the Earth HUD. Uploaded images are saved as Earth brand assets and read by the Earth page immediately.
|
||||
- **About**: manages the About card shown in Earth settings, including logo, kicker, title, version, description, and metadata.
|
||||
- **TV Livestream**: manages sources shown in the Earth media panel.
|
||||
- **Boundary Precision**: shows the current provider, low-precision fallback, high-precision PMTiles/manifest status, local source JSON, and manual build action.
|
||||
- **Base Map**, **Layer Resources**, **3D Assets**, and **News Anchor Strategy**: placeholder tabs for future configuration. They do not display fake data.
|
||||
|
||||
The Earth page settings gear also includes Boundary Precision. Switching to High Precision starts a local background download/build, like a game update package, when no high-precision asset exists yet. Progress is shown as a percentage, and the result applies automatically after success without a page reload. Switching back to Low Precision only changes the local display preference.
|
||||
|
||||
If the backend decides Earth is not initialized yet, the first visit to `/earth` shows a glassy startup guide that points the user to sign in and collect data. This is based on backend data state; once the system has collected data, clearing browser storage does not make the guide reappear.
|
||||
|
||||
### Collection Management
|
||||
|
||||
`/collection-management` is also under Operations and Configuration and owns the collection lifecycle:
|
||||
|
||||
- **Collectors**: endpoint, headers, credentials, timeout, retry, and connection checks.
|
||||
- **Collection Scheduling**: the existing scheduling configuration.
|
||||
- **Collection History / Snapshots**: a placeholder for future collection task, snapshot, and collected-data browsing.
|
||||
- **Collection History / Snapshots**: groups snapshots by datasource and lets the detail panel switch versions through a Time Capsule selector.
|
||||
|
||||
### SMTP Email Settings
|
||||
|
||||
@@ -230,7 +233,7 @@ To let a regular user read developer or operations docs, add `docs_developer` or
|
||||
|
||||
## Data Exploration
|
||||
|
||||
- `/datasources`: source directory. The `Collection Tasks` tab is for one-shot, scheduled, and finite collectors; it can be filtered by product domain, layer/module, enabled state, last run status, whether collected records exist, and search text. Selecting rows triggers only those sources; with no selected rows, `Collect current filter` triggers the filtered scope. The `Realtime Streams` tab is for AISStream / WebSocket long connections and shows connection health, stored totals, time-window counters, and Start / Stop / Reconnect actions. Clicking a name opens an info drawer showing endpoint, headers, base config, and built-in flag; endpoint/credentials editing happens at `/collection-management -> Collectors`. The `Collecting N` tag under the overall progress can be clicked to expand the current collection task list
|
||||
- `/datasources`: source directory. The `Built-in Sources` tab can be filtered by product domain, layer/module, enabled state, last run status, whether collected records exist, and search text. With no rows selected, the primary button shows `Trigger All`; after selecting rows, it becomes `Trigger Selected N` and submits only those sources. The top-right queue button shows a queue icon when empty and a pure circular total-progress indicator while tasks exist; it opens a floating panel grouped by running, completed, failed, and skipped. Failed rows can retry, and completed rows can jump to detail. `Realtime Sources` is for AISStream / WebSocket long connections and shows connection health, stored totals, time-window counters, and Start / Stop / Reconnect actions. Endpoint/credential/header editing happens at `/collection-management -> Collectors`.
|
||||
- `/data`: collected data table — used to verify "did data arrive", "is the freshness right", "does a source emit valid records"
|
||||
- `/bgp`: BGP detail page with list + detail + analysis; complements the BGP layer on Earth
|
||||
- `/alerts/system`, `/alerts/bgp`, `/alerts/situational`: system, BGP, and situational alerts
|
||||
@@ -277,7 +280,17 @@ Search finds cables, landing points, satellites, compute centers, BGP events, BG
|
||||
|
||||
Compute center and BGP observer detail cards support automatic coordinate-candidate collection. Click the object then use "Collect Coordinate Candidates" or "Re-collect Coordinates". The backend assembles candidates from source coordinates, public-org registry APIs, and online geocoders. When regular sources have no candidate, the current default AI Provider runs one LLM factcheck fallback. BGP observers' stored coordinates only fill query context; they are not returned as candidates.
|
||||
|
||||
Candidates preview on Earth directly. Saving a compute-center candidate writes to the `compute_center_locations` dimension table and refreshes the layer immediately. The notification badge at the top-left of the compute-center layer shows the unresolved count; clicking it opens the queue, supports single collection, or "Adopt All" to save the top-confidence candidates from top to bottom. Records without candidates stay in the queue rather than being faked to country centroids. See [Earth Location Candidate Collection User Guide](/home/ray/dev/linkong/planet/docs/technical/en/location-pipeline-user.md).
|
||||
Candidates preview on Earth directly. Saving a compute-center candidate writes to the `compute_center_locations` dimension table and refreshes the layer immediately. The notification badge at the top-left of the compute-center layer shows the unresolved count; clicking it opens the queue, supports single collection, or "Adopt All" to save the top-confidence candidates from top to bottom. Records without candidates stay in the queue rather than being faked to country centroids.
|
||||
|
||||
Recommended single-object flow:
|
||||
|
||||
1. Open a compute center or BGP observer detail card.
|
||||
2. Click "Collect Coordinate Candidates".
|
||||
3. Wait for candidates; entries that depend on WebSearch / AI factcheck show collection state.
|
||||
4. Preview candidate positions on Earth.
|
||||
5. Save the candidate when it is credible; closing the card does not lose the current task state.
|
||||
|
||||
Adopt All is for batch processing the compute-center unresolved queue. It starts from the top and adopts the highest-confidence candidate. Records without factual support remain in the queue. When WebSearch is disabled, single locate and Adopt All are disabled because location validation depends on factual lookup.
|
||||
|
||||
### Settings
|
||||
|
||||
@@ -348,7 +361,7 @@ Mobile uses a drawer layout: layer control moves into a drawer; search/settings/
|
||||
|
||||
Docs at `http://localhost:3000/docs` are served by the backend with access control, not bundled into the frontend build.
|
||||
|
||||
Anonymous visitors see only `public` docs: README, Quickstart, Manual, FAQ, Earth Location Candidate Collection User Guide. Authenticated users with Gatekeeper groups see more:
|
||||
Anonymous visitors see only `public` docs: README, Quickstart, Manual, and FAQ. Authenticated users with Gatekeeper groups see more:
|
||||
|
||||
- `docs_user`: end-user operational docs
|
||||
- `docs_developer`: Earth, frontend, backend, collectors, AI Provider development docs
|
||||
@@ -362,5 +375,4 @@ Docs supports: category navigation, Markdown rendering, tables and code blocks,
|
||||
|
||||
- [Quickstart](/home/ray/dev/linkong/planet/docs/technical/en/quickstart.md)
|
||||
- [FAQ](/home/ray/dev/linkong/planet/docs/technical/en/faq.md)
|
||||
- [Earth Location Candidate Collection User Guide](/home/ray/dev/linkong/planet/docs/technical/en/location-pipeline-user.md)
|
||||
- [Planet Ops Runbook](/home/ray/dev/linkong/planet/docs/technical/en/ops-runbook.md)
|
||||
|
||||
@@ -15,7 +15,7 @@ This document standardizes terms used across the Planet console, Earth, backend
|
||||
| English / Key | Chinese Display Name | Usage |
|
||||
| --- | --- | --- |
|
||||
| Planet | Planet | Product name |
|
||||
| Admin Next | 新控制台 | New admin console |
|
||||
| Admin | 控制台 | Admin console context |
|
||||
| Earth | Earth | Visualization product name |
|
||||
| datasource | 数据源 | APIs, lists, filters |
|
||||
| collector | 采集器 | Collection jobs and credential configuration |
|
||||
|
||||
@@ -21,7 +21,7 @@ First startup seeds two default accounts (see `DEFAULT_LOGIN_USERS` in `backend/
|
||||
| Username | Password | Role |
|
||||
| --- | --- | --- |
|
||||
| `admin` | `admin123` | `super_admin` |
|
||||
| `linkong` | `12345678` | `super_admin` |
|
||||
| `linkong` | `LK12345678` | `super_admin` |
|
||||
|
||||
Both seed accounts are created with `email_verified = TRUE` and can log into the console immediately. Any other account must either go through the public registration flow described in the Manual, or be created via `./planet.sh createuser`.
|
||||
|
||||
@@ -61,6 +61,22 @@ Per-module restart:
|
||||
|
||||
Per-module restart is preferred during development to avoid interrupting unrelated services.
|
||||
|
||||
## Destructive Reset
|
||||
|
||||
```bash
|
||||
./planet.sh destroy
|
||||
```
|
||||
|
||||
`destroy` returns a local development environment to a near-empty project state. It requires typing `Y` before it runs; source files and existing `.env` files are preserved.
|
||||
|
||||
Cleanup order and boundaries:
|
||||
|
||||
- If `planet_postgres` is running, the script first clears the `public` schema in `planet_db`. This prevents old `collected_data.is_current = true` rows from making Earth OOBE report `ready=true` if Docker volume removal later fails.
|
||||
- Docker cleanup targets resources whose Compose project is `planet`, plus the explicit volumes `planet_postgres_data`, `planet_redis_data`, `postgres_data`, and `redis_data`; do not delete unlabeled volumes by a broad `planet_*` pattern, because another local project could own them.
|
||||
- Local build state removes `.venv`, frontend `node_modules` / `dist`, Planet state, and scattered Python / Vite cache directories. `$PLANET_CACHE_DIR/downloads` is preserved so upstream raw downloads such as CelesTrak can survive database resets and local rebuild cleanup.
|
||||
|
||||
After the reset, run `./planet.sh init` again to recreate tables and default seed data. Old collected records are not restored, and Earth OOBE is evaluated from the backend's real collection state on the next visit. When CelesTrak later returns its "GP data has not updated" HTTP 403, the backend first reuses the preserved download cache to repopulate the database; if no cache exists, wait for the next CelesTrak update window or use Space-Track as a fallback.
|
||||
|
||||
## Health Check
|
||||
|
||||
```bash
|
||||
@@ -120,6 +136,20 @@ Useful for:
|
||||
- Demoing Earth from a phone or tablet
|
||||
- Other LAN machines reaching the same dev instance
|
||||
|
||||
On Windows, the repository-root `planet.cmd` can be used as a one-click entrypoint. It requests Administrator privileges, enters the `Ubuntu` WSL distribution at `/home/linkong/planet`, runs `./planet.sh restart --allow-lan`, opens `http://localhost:3000/earth` after a successful restart, and leaves the terminal inside a WSL shell for log inspection. If the local WSL distribution name or checkout path differs, adjust the `wsl.exe -d ... --cd ...` arguments in `planet.cmd` first.
|
||||
|
||||
On a new Windows machine, check the WSL generation first:
|
||||
|
||||
```powershell
|
||||
wsl -l -v
|
||||
```
|
||||
|
||||
Planet development should use WSL2. WSL1 has different networking, filesystem, and process behavior, and can surface as Bun package-manager commands returning only `An unknown error occurred (Unexpected)`, unstable port release, or LAN behavior that does not match the script's assumptions. Convert the distribution if it still runs as WSL1:
|
||||
|
||||
```powershell
|
||||
wsl --set-version Ubuntu 2
|
||||
```
|
||||
|
||||
`--allow-lan` directly exposes the frontend, backend, and AI Provider from the development machine: frontend `3000`, backend `8000`, and AI Provider `8010`. Before startup, the script checks all three ports. If WSL/Linux cannot release a port and a Windows-side listener or stale `portproxy` rule owns it, the script requests Administrator PowerShell cleanup. When Planet runs in WSL, Windows can usually reach it through `localhost`; other LAN machines reaching the Windows LAN IP still need Windows Firewall allow rules.
|
||||
|
||||
Diagnose in this order:
|
||||
@@ -184,13 +214,32 @@ The AI Provider image only rebuilds when code, Dockerfile, Compose config, or Py
|
||||
./planet.sh restart -a
|
||||
```
|
||||
|
||||
Rebuild detection is based on a content fingerprint rather than only file mtimes. `planet.sh` hashes the `aiprovider/` files, `aiprovider/Dockerfile`, `pyproject.toml`, `uv.lock`, `PYTHON_IMAGE`, `UV_IMAGE`, and the dependency fingerprint into `AI_PROVIDER_BUILD_FINGERPRINT`; Docker writes it into the image label `planet.aiprovider.build-fingerprint`. If the existing `planet-aiprovider:latest` image has a matching label, the script skips rebuild and refreshes the local stamp. Older images without the label fall back to the state/cache stamp.
|
||||
|
||||
Docker builds use `uv sync --frozen`. To make container builds reuse the host uv mirror configuration, the script resolves the first config file in this order and mounts it into the build as a BuildKit secret at `/root/.config/uv/uv.toml`:
|
||||
|
||||
1. The current `UV_CONFIG_FILE`
|
||||
2. Repository-root `uv.toml`
|
||||
3. `${XDG_CONFIG_HOME:-~/.config}/uv/uv.toml`
|
||||
4. `~/.uv/uv.toml`
|
||||
|
||||
If none exists, the script creates an empty state file for the secret so Compose does not fail on a missing file. Before Docker build it unsets `UV_DEFAULT_INDEX`, `UV_INDEX_URL`, and `UV_EXTRA_INDEX_URL`, keeping the build tied to the explicit `UV_CONFIG_FILE`. For a temporary Tsinghua mirror, place this in repository-root `uv.toml`:
|
||||
|
||||
```toml
|
||||
[[index]]
|
||||
name = "tsinghua"
|
||||
url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple/"
|
||||
default = true
|
||||
```
|
||||
|
||||
Diagnose slow builds:
|
||||
|
||||
| Symptom | Common cause | Fix |
|
||||
| --- | --- | --- |
|
||||
| Large `transferring context` | build context includes unrelated frontend / data files | `.dockerignore` ships only required files |
|
||||
| `uv sync` is slow | first build or cold cache | wait for the first build; later runs reuse BuildKit cache |
|
||||
| `uv sync --frozen` is slow | first build, cold cache, or missing uv mirror config | wait for the first build; later runs reuse BuildKit cache; configure `uv.toml` when needed |
|
||||
| Old keys still in effect after edit | container not restarted | `./planet.sh restart -a` |
|
||||
| Code changed but the image did not rebuild | fingerprint still matches the image label | Confirm the change is under `aiprovider/`, Dockerfile, or Python dependency inputs; delete `planet-aiprovider:latest` and retry if needed |
|
||||
|
||||
## SMTP Email (Required for Public Registration)
|
||||
|
||||
@@ -217,6 +266,15 @@ One-time codes are stored in Redis under `otp:{purpose}:{email}` with a 600-seco
|
||||
|
||||
## Development Command Conventions
|
||||
|
||||
Backend setup and script initialization use the lockfile:
|
||||
|
||||
```bash
|
||||
uv python install 3.14
|
||||
uv sync --frozen --group dev
|
||||
```
|
||||
|
||||
`--frozen` rejects implicit `uv.lock` rewrites, which is the desired behavior on new machines, CI, and Docker builds. Dependency upgrades should explicitly update `pyproject.toml` / `uv.lock` on a development machine and commit the lockfile.
|
||||
|
||||
Frontend must use Bun:
|
||||
|
||||
```bash
|
||||
|
||||
160
docs/technical/en/platform-data-flows.md
Normal file
160
docs/technical/en/platform-data-flows.md
Normal file
@@ -0,0 +1,160 @@
|
||||
# Business Architecture and Data Flows
|
||||
|
||||
This document is the business entry point for Planet data products. It explains why each Earth data type exists, where it is collected from, which fact or derived tables it uses, and how cache invalidation plus WebSocket hints reach Earth. Frontend, backend, and Earth docs should focus on implementation details; start here when you need the cross-system data flow.
|
||||
|
||||
## Overview
|
||||
|
||||
Planet data moves through three stages:
|
||||
|
||||
1. **Collect and normalize**: built-in collectors, admin actions, or the location pipeline write PostgreSQL. Generic raw output lands in `collected_data`; layer-ready projections land in derived tables.
|
||||
2. **Project and broadcast**: database triggers write fact changes to the `earth_data_change_events` outbox and wake the backend listener with `LISTEN/NOTIFY`. The listener maps the change through layer adapters, invalidates cache, and broadcasts `earth_updates`.
|
||||
3. **Reload and render in Earth**: Earth receives layer-level refresh hints, applies `clear_then_reload`, `reload`, or `delta`, then refetches `/api/v1/visualization/...`.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Source["External source / admin action"] --> Collector["Collector or data job"]
|
||||
Collector --> Facts["collected_data"]
|
||||
Collector --> Derived["Derived tables"]
|
||||
Facts --> Outbox["earth_data_change_events"]
|
||||
Derived --> Outbox
|
||||
Outbox --> Listener["DB change listener"]
|
||||
Listener --> Cache["Earth cache invalidation"]
|
||||
Listener --> WS["earth_updates"]
|
||||
WS --> Earth["Earth layer reload"]
|
||||
Earth --> API["Visualization APIs"]
|
||||
```
|
||||
|
||||
`LISTEN/NOTIFY` is only the low-latency wakeup path. The reliable source is the outbox. A real zero-data state is valid and APIs should return HTTP 200 with an empty collection; only actual endpoint failures should return 5xx.
|
||||
|
||||
## Data Products
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
Satellites["Satellite TLE"] --> SatelliteLayer["satellites layer"]
|
||||
Cables["Cables + landing points"] --> CableLayer["cables layer"]
|
||||
Compute["TOP500 / AI GPU / HF"] --> ComputeLocations["compute_center_locations"]
|
||||
ComputeLocations --> ComputeLayer["computeCenters layer"]
|
||||
BgpRaw["RIS Live / BGPStream / Prefix"] --> BgpDerived["bgp_observations / anomalies / incidents"]
|
||||
BgpDerived --> BgpLayer["bgp layer"]
|
||||
VesselRaw["AIS / BarentsWatch"] --> VesselDerived["vessel_static / vessel_position"]
|
||||
VesselDerived --> VesselLayer["vessels layer"]
|
||||
Interactables["earth_interactables"] --> InteractableLayer["interactables layer"]
|
||||
NewsRaw["RSS / Live / News"] --> NewsItems["earth_news_items"]
|
||||
NewsItems --> NewsLayer["news / media layer"]
|
||||
```
|
||||
|
||||
| Data product | Business use | Fact sources | Derived / dimension tables | Earth layer | Strategy |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| Satellites | In-orbit objects, tracks, coverage, and cruise targets | `celestrak_tle`, `spacetrack_tle` | No long-lived projection; APIs convert TLE on demand | `satellites` | `clear_then_reload` |
|
||||
| Cables and landing points | Submarine connectivity, landing cities, cable detail, and search | `arcgis_cables`, `arcgis_landing_points`, TeleGeography / FAO landing sources | Cable relation and landing aggregation data | `cables` | `clear_then_reload` |
|
||||
| Compute centers | TOP500, AI GPU, model platform facilities, and location-completion state | `top500`, `epoch_ai_gpu`, HuggingFace sources | `compute_center_locations` | `computeCenters` | `reload` |
|
||||
| BGP context | Collectors, anomalies, incidents, route events, and regional context | `ris_live_bgp`, `bgpstream_bgp`, prefix geography sources | `bgp_observations`, `bgp_anomalies`, `bgp_incidents`, `bgp_collector_locations` | `bgp` | `clear_then_reload` |
|
||||
| Vessels | AIS vessels, positions, tracks, legend, and source health | AIS sources, `barentswatch_vessels` | `vessel_static`, `vessel_position`, `ais_raw_observations`, `ais_source_health` | `vessels` | `clear_then_reload` |
|
||||
| Interactables | Generic surface icons, manual objects, and future small layers | `earth_interactables` | None | `interactables` | `delta` |
|
||||
| News and media | Earth news, live streams, cruise summaries, and situation content | news sources | `earth_news_items` | `news` / `media` | `reload` |
|
||||
|
||||
## Satellites
|
||||
|
||||
Satellite data powers online satellite markers, tracks, footprint policy, and cruise lists. CelesTrak or Space-Track collectors fetch TLE and write `collected_data`. The visualization API converts TLE into current positions and trails on request.
|
||||
|
||||
- **Collection entry**: CelesTrak TLE, Space-Track TLE.
|
||||
- **Fact table**: `collected_data.source IN ('celestrak_tle', 'spacetrack_tle')`.
|
||||
- **API**: satellite visualization API reads TLE and builds the Earth payload.
|
||||
- **Delete semantics**: deleting the source broadcasts `satellites` with `clear_then_reload`; the frontend clears satellite markers and trails before refetching. With no TLE, the API returns an empty list.
|
||||
- **Common failure**: summary is zero but Earth still shows satellites. Usually WS did not fire, the adapter did not cover the source, or the frontend did not clear existing Three.js objects.
|
||||
|
||||
## Cables and Landing Points
|
||||
|
||||
Cables and landing points show global network connectivity, landing cities, cable details, and search targets. Cable lines and landing points are one business layer; deleting either side must refresh `cables`.
|
||||
|
||||
- **Collection entry**: ArcGIS cables, ArcGIS landing points, and TeleGeography / FAO landing related sources.
|
||||
- **Fact table**: cable and landing point sources in `collected_data`.
|
||||
- **Derived data**: cable relation tables, landing point aggregation, interface cache.
|
||||
- **API**: cable visualization API returns cables, landing points, and relation data.
|
||||
- **Delete semantics**: deleting cable or landing source clears owned projections and broadcasts `cables` with `clear_then_reload`. Empty data returns HTTP 200, not 404.
|
||||
- **Common failure**: cables disappear for a second and return. Usually a collection replacement or cache refresh window reuses old cache; check duplicate broadcasts and cache patterns.
|
||||
|
||||
## Compute Centers
|
||||
|
||||
Compute centers represent supercomputers, AI GPU sites, model-platform facilities, and unresolved-location state. Raw sources often only provide organization, country, site, or fuzzy location, so Earth rendering depends on `compute_center_locations`.
|
||||
|
||||
- **Collection entry**: TOP500, Epoch AI GPU, HuggingFace related sources.
|
||||
- **Fact table**: compute sources in `collected_data`.
|
||||
- **Dimension table**: `compute_center_locations` stores adopted coordinate candidates.
|
||||
- **API**: compute center visualization API merges raw records with location rows.
|
||||
- **Delete semantics**: deleting TOP500 or similar sources refreshes `computeCenters`; changing the location table also refreshes the layer. The default is `reload`, because location updates do not always need an immediate clear.
|
||||
- **Common failure**: collection finished but Earth count did not change. Check visualization cache, location rows, and adapter coverage.
|
||||
|
||||
## BGP
|
||||
|
||||
BGP data shows route collectors, anomalies, incidents, diffusion rings, and situation summaries. Earth does not render raw BGP rows directly; it renders projected observations, anomalies, and incidents. This is the most common path where deleting raw data can leave old objects visible.
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant UI as Admin delete button
|
||||
participant Job as clear_data job
|
||||
participant DB as PostgreSQL
|
||||
participant Outbox as earth_data_change_events
|
||||
participant Listener as DB change listener
|
||||
participant Earth as Earth frontend
|
||||
|
||||
UI->>Job: Submit delete for ris_live_bgp / bgpstream_bgp
|
||||
Job->>DB: Delete collected_data raw rows
|
||||
Job->>DB: Delete BGP-owned derived tables
|
||||
DB->>Outbox: trigger writes bgp layer changed
|
||||
DB-->>Listener: LISTEN/NOTIFY wakeup
|
||||
Listener->>Listener: merge events and clear bgp cache
|
||||
Listener-->>Earth: broadcast earth_updates clear_then_reload
|
||||
Earth->>Earth: clear BGP objects
|
||||
Earth->>DB: refetch projections through visualization API
|
||||
```
|
||||
|
||||
- **Collection entry**: RIPE RIS Live BGP, CAIDA BGPStream Backfill, IPtoASN / OpenGeoFeed / NRO prefix geography.
|
||||
- **Fact table**: BGP and prefix sources in `collected_data`.
|
||||
- **Derived tables**: `bgp_observations`, `bgp_anomalies`, `bgp_incidents`, `bgp_collector_locations`.
|
||||
- **API**: BGP visualization API reads projections and combines stored or candidate locations.
|
||||
- **Delete semantics**: deleting RIS Live or BGPStream raw sources must also clear BGP-owned projections and broadcast `bgp` with `clear_then_reload`. Direct derived-table deletes also write outbox events.
|
||||
- **Common failure**: collectors or incidents remain after raw rows are removed. Check derived tables first, not only `collected_data`.
|
||||
|
||||
## Vessels
|
||||
|
||||
Vessel data shows AIS vessels, navigation state, vessel-type legend, and source health. Earth rendering uses position snapshots and static vessel information; it should not render every raw AIS observation.
|
||||
|
||||
- **Collection entry**: AIS sources, BarentsWatch vessels.
|
||||
- **Fact table**: `collected_data` or AIS raw observation tables.
|
||||
- **Derived tables**: `vessel_static`, `vessel_position`, `ais_raw_observations`, `ais_source_health`.
|
||||
- **API**: vessel visualization API returns current vessel markers and detail fields.
|
||||
- **Delete semantics**: deleting any vessel source lets owned projections broadcast `vessels` with `clear_then_reload`.
|
||||
- **Common failure**: the count panel changes but vessels remain. Summary and layer data are separate; Earth should clear objects based on layer updates.
|
||||
|
||||
## Interactables
|
||||
|
||||
`earth_interactables` is the generic surface-icon capability for manual objects, extensions, and future small layers. Unlike most layers, it keeps object-level delta.
|
||||
|
||||
- **Fact table**: `earth_interactables`.
|
||||
- **API**: interactable API and Earth generic icon interface.
|
||||
- **Strategy**: upsert for create/update and removeItem for delete; no full layer reload.
|
||||
- **Delete semantics**: delete payloads must include a stable id so the frontend can remove the object.
|
||||
- **Common failure**: an object cannot be removed. Usually the stable id is missing, the delta handler did not match the object type, or an older duplicate render path still exists.
|
||||
|
||||
## News and Media
|
||||
|
||||
News and media support the Earth news ticker, live stream panel, news cruise, and situation summaries. They are content refresh paths rather than stable geographic object layers, so they default to `reload`.
|
||||
|
||||
- **Collection entry**: RSS, live streams, news sources.
|
||||
- **Fact table**: news source rows in `collected_data`.
|
||||
- **Derived table**: `earth_news_items`.
|
||||
- **API**: news, live stream, and media content APIs.
|
||||
- **Delete semantics**: deleting news sources or `earth_news_items` broadcasts `news` / `media` reload; empty responses hide the corresponding content.
|
||||
- **Common failure**: the live panel shows stale content. Usually the media component ignored the layer update or the content API cache was not invalidated.
|
||||
|
||||
## Adding a New Layer
|
||||
|
||||
Add a new Earth data product in this order:
|
||||
|
||||
1. Define the business purpose and Earth layer name.
|
||||
2. Identify fact sources, fact tables, and derived tables.
|
||||
3. Register source/table coverage, cache patterns, owned derived cleanup, and default strategy in the backend layer adapter.
|
||||
4. Ensure the visualization API returns HTTP 200 with an empty collection for zero data.
|
||||
5. Keep the Earth frontend driven only by layer and strategy, not database table names.
|
||||
6. Update this document with the data product flow, then document implementation details in the relevant frontend, backend, or Earth technical doc.
|
||||
@@ -34,7 +34,7 @@ After landing on the `/admin` dashboard, here's a recommended walk-through:
|
||||
|
||||
1. `/collection-management?tab=collector_credentials`: pick a collector and click the plug icon to test connectivity. Free collectors (e.g. open BGP) usually work right away; credential-bearing ones like `AISStream` or `BarentsWatch` need an API key / client secret first
|
||||
2. `/ai?tab=providers`: fill an LLM provider (e.g. `minimax` / `openai`), model, base URL, API key, and click the plug at the end of the base URL to test. WebSearch / OCR tools are optional
|
||||
3. `/datasources` or `/data`: check whether collectors have produced data. Use `/datasources -> Collection Tasks` for finite collectors, and `/datasources -> Realtime Streams` for AISStream / WebSocket health and counters
|
||||
3. `/datasources` or `/data`: check whether collectors have produced data. Use `/datasources -> Built-in Sources` for finite collectors: with no rows selected, click `Trigger All`; after selecting rows, the primary button becomes `Trigger Selected N`. The top-right queue button shows progress. Use `/datasources -> Realtime Sources` for AISStream / WebSocket health and counters
|
||||
4. `/alerts/system`: verify system alerts look right
|
||||
5. `/users` (super_admin only): open accounts for teammates or adjust their groups
|
||||
|
||||
@@ -42,6 +42,8 @@ After landing on the `/admin` dashboard, here's a recommended walk-through:
|
||||
|
||||
Visit `/earth`. This is a public page — no login required.
|
||||
|
||||
If the system has no collected data yet, Earth shows an initialization guide that points you to sign in and trigger collection. This guide is backend-state driven, so clearing browser storage does not make it appear once the system is ready.
|
||||
|
||||
Once in, verify:
|
||||
|
||||
- The globe renders, and the right-side layer panel can toggle layers
|
||||
@@ -62,5 +64,5 @@ Open `/forgot-password`, enter your email, receive a code, then enter the code p
|
||||
|
||||
- Full UI walkthrough: [Planet Manual](/home/ray/dev/linkong/planet/docs/technical/en/manual.md)
|
||||
- Troubleshooting and configuration questions: [FAQ](/home/ray/dev/linkong/planet/docs/technical/en/faq.md)
|
||||
- Detailed Earth coordinate candidate flow: [Earth Location Candidate Collection User Guide](/home/ray/dev/linkong/planet/docs/technical/en/location-pipeline-user.md)
|
||||
- Detailed Earth coordinate candidate flow: see the Earth section in [Planet Manual](/home/ray/dev/linkong/planet/docs/technical/en/manual.md)
|
||||
- Deployment / operations commands: [Planet Ops Runbook](/home/ray/dev/linkong/planet/docs/technical/en/ops-runbook.md)
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Tactile UI Components
|
||||
|
||||
Tactile UI is Planet's portable React control layer. It was extracted from the Admin Next button, switch, scrollbar, and tooltip work, but the components themselves do not depend on Admin Next, AntD, Radix, Tailwind, or `an-*` classes. The immediate goal is stable in-repo usage; the structure is intentionally close to something that can later be published as an npm package.
|
||||
Tactile UI is Planet's portable React control layer. It was extracted from the Admin button, switch, scrollbar, and tooltip work, but the components themselves do not depend on Admin, AntD, Radix, Tailwind, or `an-*` classes. The immediate goal is stable in-repo usage; the structure is intentionally close to something that can later be published as an npm package.
|
||||
|
||||
## Design Goals
|
||||
|
||||
@@ -49,6 +49,22 @@ Core tokens are exposed as `--tui-*` CSS variables. Product themes should overri
|
||||
|
||||
Most controls also accept a `tactile` prop for local width, height, radius, background, border, and shadow overrides. Use local overrides for small special cases; use CSS variables for product-wide styling.
|
||||
|
||||
## Portals and Dark Theme
|
||||
|
||||
`TactileTooltip` and the Admin `Dialog`, `Select`, and toast controls that use Tactile UI may render through a portal attached to `document.body`. Those nodes are not descendants of `.admin-theme-root[data-theme='dark']`, so dark tokens cannot rely only on an ancestor selector inside the Admin root.
|
||||
|
||||
The Admin theme provider mirrors the active theme to `body[data-admin-theme]`. Shared styles need to support both selector paths:
|
||||
|
||||
```css
|
||||
[data-theme='dark'] .tui-button,
|
||||
body[data-admin-theme='dark'] .tui-button {
|
||||
--tui-surface: #172033;
|
||||
--tui-text: #e5edf8;
|
||||
}
|
||||
```
|
||||
|
||||
When adding a portal-based control, first check whether it renders into body. If it does, add a `body[data-admin-theme='dark']` branch in that component's style entry, or reuse the already covered `--tui-*` / `--an-*` tokens. Avoid hard-coding a one-off dark modal style, because the same contrast problem can reappear in dropdowns, tooltips, toasts, and confirmation dialogs.
|
||||
|
||||
## `TactileButton`
|
||||
|
||||
The button component covers regular buttons, icon buttons, strong-intent buttons, and link-like buttons.
|
||||
@@ -74,6 +90,8 @@ Common props:
|
||||
|
||||
`variant="neutral"` defaults to a white tactile button. Colored buttons should still keep the same height and external shadow instead of relying on page-specific CSS overrides.
|
||||
|
||||
Colored button borders must not use the exact fill color. `primary`, `danger`, and future colored variants should use a lighter border from the same hue, such as `color-mix(in srgb, var(--tui-danger) 64%, white)`. The border still reads as part of the button color, but its visual weight is lower than the fill surface, so red or blue buttons do not look one outline larger than neutral buttons. Hover states should brighten rather than darken: mix a little white into the current fill color, and keep the hover border lighter than the hover fill.
|
||||
|
||||
## Icon Presets
|
||||
|
||||
Preset icons are maintained in `tactileIconPresets`. Feature pages should call icons by semantic name so actions remain consistent across the console.
|
||||
@@ -177,7 +195,7 @@ Do not force overlay scrollbars onto textareas. Text selection and the resize gr
|
||||
|
||||
## `TableScrollRegion`
|
||||
|
||||
`TableScrollRegion` is a convenience wrapper for table scroll areas. The default target selector is `.tui-scroll-target`; Admin Next passes its table viewport selector explicitly so package code does not contain product-specific names.
|
||||
`TableScrollRegion` is a convenience wrapper for table scroll areas. The default target selector is `.tui-scroll-target`; Admin passes its table viewport selector explicitly so package code does not contain product-specific names.
|
||||
|
||||
```tsx
|
||||
<TableScrollRegion targetSelector=".table-viewport">
|
||||
@@ -192,9 +210,9 @@ Do not force overlay scrollbars onto textareas. Text selection and the resize gr
|
||||
- Tooltip text is explanatory only; state must still be represented by text, badges, or `aria-*` attributes.
|
||||
- Disabled and loading states set `aria-disabled`; real `button` elements also receive `disabled`.
|
||||
|
||||
## Admin Next Migration Rules
|
||||
## Admin Migration Rules
|
||||
|
||||
Admin Next should use Tactile UI for global tool buttons, detail-panel toolbars, and list footer actions:
|
||||
Admin should use Tactile UI for global tool buttons, detail-panel toolbars, and list footer actions:
|
||||
|
||||
- Unambiguous actions: `TactileButton iconOnly tooltip`
|
||||
- Save/create/confirm: `TactileButton variant="primary"`, usually with text
|
||||
|
||||
@@ -1,44 +1,52 @@
|
||||
# 技术文档
|
||||
|
||||
这里放“当前实现和当前结构”的文档,重点回答:
|
||||
这里是 Planet 当前文档入口。文档按读者和问题类型分层:先看业务架构理解数据产品,再进入使用手册或技术实现文档。
|
||||
|
||||
- 现在代码是怎么组织的
|
||||
- 当前入口在哪
|
||||
- 状态和组件如何工作
|
||||
## 业务架构
|
||||
|
||||
适合放入这里的内容:
|
||||
- [业务架构与数据流转](/home/ray/dev/linkong/planet/docs/technical/zh/platform-data-flows.md):每类 Earth 数据的用途、采集链路、事实表、派生表、缓存和 WebSocket 广播链路
|
||||
- [命名与术语对照](/home/ray/dev/linkong/planet/docs/technical/zh/naming-glossary.md):控制台、Earth、后端和文档常见名词的中英对照
|
||||
|
||||
- 快速开始和使用手册
|
||||
- 前端上下文
|
||||
- Earth 前端结构
|
||||
- Earth 卫星覆盖策略
|
||||
- Earth 渲染图层顺序
|
||||
- Earth 图层样式属性索引
|
||||
- 后端运行控制
|
||||
- 采集器现状
|
||||
- 采集器设置与连接验证
|
||||
- 采集格式约定
|
||||
|
||||
## 使用入口
|
||||
## 使用手册
|
||||
|
||||
- [快速开始](/home/ray/dev/linkong/planet/docs/technical/zh/quickstart.md):从零启动 Planet 的最短路径
|
||||
- [Planet 使用手册](/home/ray/dev/linkong/planet/docs/technical/zh/manual.md):控制台、`planet.sh`、Earth 和 Docs 的完整使用手册
|
||||
- [常见问题](/home/ray/dev/linkong/planet/docs/technical/zh/faq.md):Windows / WSL、端口、依赖、动捕、凭证和 Docs 权限的集中排障入口
|
||||
- [Earth 位置候选采集使用手册](/home/ray/dev/linkong/planet/docs/technical/zh/location-pipeline-user.md):在 Earth 上为算力中心和 BGP 观测站采集、预览坐标候选
|
||||
- [数据源、采集器设置与连接验证](/home/ray/dev/linkong/planet/docs/technical/zh/datasource-collector-settings-connectivity.md):数据源目录、采集器设置、连接验证、BarentsWatch 凭证链路
|
||||
- [通用位置估算管线开发说明](/home/ray/dev/linkong/planet/docs/technical/zh/location-pipeline-development.md):后端 location resolver / pipeline 的接口、注册表和扩展方式
|
||||
- [Planet 使用手册](/home/ray/dev/linkong/planet/docs/technical/zh/manual.md):控制台、Earth、Docs 和常用功能的用户操作说明
|
||||
- [常见问题](/home/ray/dev/linkong/planet/docs/technical/zh/faq.md):Windows / WSL、端口、依赖、动捕、凭证和 Docs 权限排障
|
||||
|
||||
## Earth 技术实现
|
||||
|
||||
- [Earth 前端结构](/home/ray/dev/linkong/planet/docs/technical/zh/earth-frontend-context.md):Earth 页面模块、状态、WebSocket 刷新和图层生命周期
|
||||
- [Earth 图层样式属性索引](/home/ray/dev/linkong/planet/docs/technical/zh/earth-layer-style-reference.md):图层颜色、符号、材质和视觉参数
|
||||
- [Earth 渲染图层顺序](/home/ray/dev/linkong/planet/docs/technical/zh/earth-render-layer-order.md):renderOrder、深度策略、拾取和同坐标避让
|
||||
- [Earth 卫星覆盖策略](/home/ray/dev/linkong/planet/docs/technical/zh/earth-satellite-footprint-policy.md):卫星 footprint 的显示边界和策略
|
||||
- [BGP 态势上下文](/home/ray/dev/linkong/planet/docs/technical/zh/earth-bgp-context.md):BGP 在 Earth 中的渲染、聚合和观测站实现
|
||||
- [Earth 可交互图标接入](/home/ray/dev/linkong/planet/docs/technical/zh/earth-interactable-usage.md):`Interactable` 的接口、生命周期和接入示例
|
||||
- [Earth 工具栏与浮层协同](/home/ray/dev/linkong/planet/docs/technical/zh/earth-toolbar-overlay-coordination.md):工具栏按钮与搜索、设置、新闻、图层浮层的关闭矩阵
|
||||
|
||||
## 前端技术实现
|
||||
|
||||
- [控制台前端结构](/home/ray/dev/linkong/planet/docs/technical/zh/frontend-admin-frontend-context.md):控制台页面结构、状态边界和懒加载策略
|
||||
- [前端布局指南](/home/ray/dev/linkong/planet/docs/technical/zh/frontend-layout-guidelines.md):布局、表格、面板和响应式约束
|
||||
- [Tactile UI 组件库](/home/ray/dev/linkong/planet/docs/technical/zh/tactile-ui-components.md):按钮、开关、tooltip、滚动条组件 API 和主题 token
|
||||
|
||||
## 后端技术实现
|
||||
|
||||
- [数据采集系统](/home/ray/dev/linkong/planet/docs/technical/zh/backend-collectors.md):采集器、任务类型、保存层和状态回写
|
||||
- [系统服务控制](/home/ray/dev/linkong/planet/docs/technical/zh/backend-system-service-control.md):后端服务控制和系统操作接口
|
||||
- [数据源、采集器设置与连接验证](/home/ray/dev/linkong/planet/docs/technical/zh/datasource-collector-settings-connectivity.md):数据源目录、采集器设置、连接验证和凭证链路
|
||||
- [数据源 API 性能](/home/ray/dev/linkong/planet/docs/technical/zh/backend-datasources-api-performance.md):DataSources 列表接口性能和缓存策略
|
||||
- [数据作业与 Outbox 技术架构](/home/ray/dev/linkong/planet/docs/technical/zh/data-job-earth-sync-architecture.md):PostgreSQL 作业队列、outbox、listener 和 Kafka / Spark 演进边界
|
||||
- [通用位置估算管线开发说明](/home/ray/dev/linkong/planet/docs/technical/zh/location-pipeline-development.md):location resolver / pipeline 的接口、注册表和扩展方式
|
||||
- [新闻直播采集格式](/home/ray/dev/linkong/planet/docs/technical/zh/earth-news-live-streams-collector-format.md):新闻、直播和媒体采集 payload 约定
|
||||
- [Docs Gatekeeper 开发说明](/home/ray/dev/linkong/planet/docs/technical/zh/docs-gatekeeper-development.md):后端 Docs 目录、正文读取和 Gatekeeper 权限组实现
|
||||
- [命名与术语对照](/home/ray/dev/linkong/planet/docs/technical/zh/naming-glossary.md):控制台、Earth、后端和文档常见名词的中英对照
|
||||
- [Earth 可交互图标接入](/home/ray/dev/linkong/planet/docs/technical/zh/earth-interactable-usage.md):Earth 地表可交互图标 `Interactable` 的接口、生命周期和接入示例
|
||||
- [Earth 工具栏与浮层协同](/home/ray/dev/linkong/planet/docs/technical/zh/earth-toolbar-overlay-coordination.md):工具栏按钮与搜索 / 设置 / 新闻 / 图层浮层之间的关闭矩阵和接入规则
|
||||
- [Tactile UI 组件库](/home/ray/dev/linkong/planet/docs/technical/zh/tactile-ui-components.md):可移植按钮、开关、tooltip 和滚动条组件的 API、主题 token 与迁移约定
|
||||
|
||||
不适合放入这里的内容:
|
||||
## 智能体与运维
|
||||
|
||||
- 尚未完成的路线图
|
||||
- 未来迭代方案
|
||||
- 大范围重构计划
|
||||
- [AI Provider 指南](/home/ray/dev/linkong/planet/docs/technical/zh/agents-aiprovider.md):模型供应商适配、任务 prompt 和调用边界
|
||||
- [Planet 运维手册](/home/ray/dev/linkong/planet/docs/technical/zh/ops-runbook.md):部署、启动、排障和敏感操作
|
||||
- [Docker + Compose + Buildx 升级](/home/ray/dev/linkong/planet/docs/technical/zh/ops-docker-compose-buildx-upgrade.md):Docker 工具链升级步骤
|
||||
- [planet.sh 启动机制](/home/ray/dev/linkong/planet/docs/technical/zh/ops-planet-sh-startup.md):启动脚本、健康检查和性能优化
|
||||
|
||||
这些应放入:
|
||||
## 计划与历史
|
||||
|
||||
- [计划文档索引](/home/ray/dev/linkong/planet/docs/plans/README.md)
|
||||
尚未完成的路线图、大范围重构方案和未来计划放在仓库路径 `docs/plans/README.md`。已完成、过时或被新实现替代的方案放在 `docs/deprecated/README.md`。
|
||||
|
||||
@@ -101,7 +101,7 @@ AI 配置页使用的接口:
|
||||
|
||||
这些接口都需要用户登录。`secrets` 接口只用于配置页点击显示 key/token 时取回明文,隐藏时前端恢复为脱敏预览。
|
||||
|
||||
Admin Next 的 AI 页面按旧版信息架构组织为:
|
||||
Admin 的 AI 页面按业务信息架构组织为:
|
||||
|
||||
- `模型供应商`
|
||||
- 管理 provider、协议适配、默认模型、LLM API Key、代理地址、代理 token、模型列表刷新、设为默认和轻量连通性测试。
|
||||
@@ -272,7 +272,7 @@ AI 配置仍保存在 PostgreSQL,不写入 JSON 文件。核心结构如下:
|
||||
|
||||
`.env` 只是兜底。配置页保存或测试连接成功后,PostgreSQL 中的配置会成为全局默认。
|
||||
|
||||
Admin Next 的密钥状态必须按 provider / tool 精确判断:
|
||||
Admin 的密钥状态必须按 provider / tool 精确判断:
|
||||
|
||||
- 数据库中当前 provider/tool 有密钥时,显示为“已配置”。
|
||||
- 数据库没有密钥,但 fallback provider、model 或 tool 与当前项匹配时,可以显示 fallback 的脱敏预览。
|
||||
@@ -283,7 +283,7 @@ Admin Next 的密钥状态必须按 provider / tool 精确判断:
|
||||
|
||||
### 轻量连通性测试
|
||||
|
||||
Admin Next 的插头按钮走轻量连通性测试,不承担保存职责。业界常见做法是分两层:
|
||||
Admin 的插头按钮走轻量连通性测试,不承担保存职责。业界常见做法是分两层:
|
||||
|
||||
- 快速检查 provider 目录或低成本 endpoint,确认 base URL、鉴权和当前模型是否可达。
|
||||
- 只有在用户明确运行 Playground 或业务任务时才发完整模型请求。
|
||||
|
||||
@@ -73,6 +73,8 @@ async def run(self, db):
|
||||
|
||||
**核心文件**: `backend/app/services/collectors/base.py`
|
||||
|
||||
手动触发、删除数据、清理缓存现在统一进入 PostgreSQL 数据作业队列,任务账本仍是 `collection_tasks`。采集器只负责 `fetch -> transform -> save`,由 `data_jobs.py` worker 领取 `collect` / `clear_data` / `clear_cache` / `earth_refresh` 任务并回写进度。Earth 图层刷新关系集中在 `earth_layer_adapters.py`,不要再在单个采集器或按钮里手写缓存失效和 WebSocket 广播。
|
||||
|
||||
## 三、采集器列表
|
||||
|
||||
| 采集器 | 数据类型 | 数据内容 | 采集频率 |
|
||||
@@ -230,7 +232,7 @@ if datasource.last_status == "success":
|
||||
|
||||
### 采集管理与快照
|
||||
|
||||
Admin Next 的采集管理入口按业务层级组织:
|
||||
Admin 的采集管理入口按业务层级组织:
|
||||
|
||||
- `采集器`:配置 endpoint、认证方式、请求头、基础参数、启用状态和凭证教程。
|
||||
- `采集调度`:查看和调整调度状态,触发、停止或刷新采集任务。
|
||||
|
||||
@@ -96,4 +96,4 @@ if (res.data.task_id) {
|
||||
## 相关文件
|
||||
|
||||
- `backend/app/api/v1/datasources.py` — `_load_datasource_list_context`、`list_datasources`
|
||||
- `frontend/src/pages/DataSources/DataSources.tsx` — `BuiltInDataSource` interface、`triggerDatasource`
|
||||
- `frontend/src/admin/pages/PlainResourcePages.tsx` — `BuiltInDataSource` interface、`triggerDatasource`
|
||||
|
||||
141
docs/technical/zh/data-job-earth-sync-architecture.md
Normal file
141
docs/technical/zh/data-job-earth-sync-architecture.md
Normal file
@@ -0,0 +1,141 @@
|
||||
# 数据作业与 Outbox 技术架构
|
||||
|
||||
本文记录 Planet v1 的任务队列、数据库 outbox 和 Earth 刷新技术边界。每类数据产品的业务用途和端到端链路见 [业务架构与数据流转](/home/ray/dev/linkong/planet/docs/technical/zh/platform-data-flows.md)。
|
||||
|
||||
## 架构边界
|
||||
|
||||
- PostgreSQL 是 v1 的可靠任务账本和 outbox,不引入 Kafka、Celery 或 RQ。
|
||||
- `collection_tasks` 记录采集、删除、清缓存和非数据库触发的 Earth refresh 任务。
|
||||
- `earth_data_change_events` 记录事实表或派生表变化,是 Earth 同步的可靠来源。
|
||||
- `LISTEN/NOTIFY` 只做低延迟唤醒;listener 仍会轮询未消费 outbox。
|
||||
- Redis 主要用于缓存、认证辅助、OTP / rate limit、临时日志和 WebSocket 辅助,不是可靠队列。
|
||||
|
||||
## 数据库变化同步
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Write["Fact or derived table write"] --> Trigger["PostgreSQL trigger"]
|
||||
Trigger --> Outbox["earth_data_change_events"]
|
||||
Trigger --> Notify["planet_earth_data_changes"]
|
||||
Outbox --> Listener["earth_db_change_listener"]
|
||||
Notify --> Listener
|
||||
Listener --> Adapter["earth_layer_adapters"]
|
||||
Adapter --> Cache["cache invalidation"]
|
||||
Cache --> WS["earth_updates"]
|
||||
```
|
||||
|
||||
1. 采集、删除、定位或派生任务写入事实表或派生表。
|
||||
2. statement trigger 为 `INSERT` / `UPDATE` / `DELETE` 写入 outbox。
|
||||
3. listener 被 notify 唤醒,或通过轮询发现未消费事件。
|
||||
4. listener 使用 `earth_layer_adapters.py` 把 `table + source` 映射成 Earth layer、刷新策略和 cache pattern。
|
||||
5. listener 短窗口合并同 layer 事件,失效缓存并广播 `earth_updates`。
|
||||
6. 广播成功后标记 outbox consumed;失败时保留待重试。
|
||||
|
||||
DB 变化不再默认创建 `earth_refresh` 任务,因此不会被同 source 的长采集或删除任务阻塞。`earth_refresh` 只保留给手动清缓存和非 DB 变化刷新提示。
|
||||
|
||||
## 数据作业队列
|
||||
|
||||
`collection_tasks` 是统一 job ledger。worker 使用 PostgreSQL `FOR UPDATE SKIP LOCKED` 领取 `queued` 任务;同一 `source` 的写任务串行,不同 source 可并行。
|
||||
|
||||
| task_type | 作用 |
|
||||
| --- | --- |
|
||||
| `collect` | 执行内置 datasource 采集 |
|
||||
| `clear_data` | 删除该 source 的采集数据和声明过的派生数据 |
|
||||
| `clear_cache` | 删除该 source 对应的 Earth / dashboard 缓存 |
|
||||
| `earth_refresh` | 非 DB 变化场景下失效 Earth 图层缓存并广播刷新提示 |
|
||||
|
||||
接口只创建任务并返回 `task_id`。任务执行、进度、取消和终态由 worker 写回 `collection_tasks`,并通过 `datasource_tasks` channel 通知前端。
|
||||
|
||||
取消语义是“保留已提交批次”:点击停止后,后端把任务标记为 `cancelling` 并取消内存中的执行协程;已经提交的批次保留,未完成批次按采集器或清理任务的回滚逻辑处理。
|
||||
|
||||
## Earth 同步事件模型
|
||||
|
||||
统一事件模型是 `earth.layer.changed`:
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "earth.layer.changed",
|
||||
"action": "database_changed",
|
||||
"source": "celestrak_tle",
|
||||
"table": "collected_data",
|
||||
"operation": "DELETE",
|
||||
"layers": ["satellites"],
|
||||
"refresh_strategy": "clear_then_reload",
|
||||
"records_processed": 11125,
|
||||
"occurred_at": "2026-05-25T10:20:30Z"
|
||||
}
|
||||
```
|
||||
|
||||
| strategy | 用途 |
|
||||
| --- | --- |
|
||||
| `clear_then_reload` | 先清前端本地图层对象,再强制重拉接口。删除数据时优先使用。 |
|
||||
| `reload` | 保留旧对象直到新数据返回,适合定位、元数据或非破坏性更新。 |
|
||||
| `delta` | 只用于 `earth_interactables`,按 id upsert 或 remove。 |
|
||||
|
||||
接口在真实 0 数据时必须返回 200 和空集合;只有真实接口异常才返回 5xx。前端收到删除事件后,如果重拉失败,应保持已清空状态并显示轻量错误,不恢复旧对象。
|
||||
|
||||
## Layer Adapter 约定
|
||||
|
||||
`earth_layer_adapters.py` 是 source、派生表、Earth layer、缓存和刷新策略的唯一注册表。新图层只应新增 adapter,不应在按钮 handler、采集器或前端分支里手写同步逻辑。
|
||||
|
||||
Adapter 必须声明:
|
||||
|
||||
- source 或 table 由哪个 Earth layer 消费。
|
||||
- 需要清理哪些 Earth cache key pattern。
|
||||
- `clear_data` 删除 source 时是否需要同时删除 owned 派生表。
|
||||
- 该 layer 的默认刷新策略。
|
||||
|
||||
删除 source 时,`clear_data` 作业先删除 `collected_data.source = <source>`,再根据 adapter 删除 owned 派生表。直接修改派生表也会触发 outbox,所以后台任务、管理接口和 SQL 修复脚本只要落到事实表或派生表,Earth 都能感知变化。
|
||||
|
||||
## 运维排障
|
||||
|
||||
检查 outbox 是否堆积:
|
||||
|
||||
```sql
|
||||
SELECT id, table_name, operation, source, occurred_at
|
||||
FROM earth_data_change_events
|
||||
WHERE consumed_at IS NULL
|
||||
ORDER BY id
|
||||
LIMIT 20;
|
||||
```
|
||||
|
||||
检查触发器是否存在:
|
||||
|
||||
```sql
|
||||
SELECT tgname, tgrelid::regclass
|
||||
FROM pg_trigger
|
||||
WHERE tgname LIKE 'tr_planet_%_changed_%'
|
||||
ORDER BY 2, 1;
|
||||
```
|
||||
|
||||
常用日志事件:
|
||||
|
||||
- `earth.db_changes.connected`:listener 已连接 PostgreSQL 并开始监听。
|
||||
- `earth.db_changes.outbox_polled`:轮询到了未消费 outbox。
|
||||
- `earth.db_changes.broadcasted`:已产生 Earth 刷新广播。
|
||||
- `data_job.started` / `data_job.completed`:任务执行状态。
|
||||
|
||||
如果 Earth 没更新,按顺序检查:事实表是否变化、outbox 是否消费、adapter 是否覆盖对应 `source/table`、listener 是否在线、前端 WebSocket 是否连接、visualization 接口是否返回 200 空集合或新数据。
|
||||
|
||||
## Kafka-ready 边界
|
||||
|
||||
业务代码不直接依赖具体队列实现,而是通过这些边界组织:
|
||||
|
||||
- `JobQueue`:提交、领取、取消、完成数据作业。
|
||||
- `DataChangeBus`:发布数据库事实变化。
|
||||
- `EarthLayerAdapterRegistry`:声明 source、layer、缓存和派生数据关系。
|
||||
|
||||
需要 Kafka 的信号:
|
||||
|
||||
- 多个独立服务需要消费同一批数据变化。
|
||||
- AIS、BGP 或传感器流达到持续高吞吐。
|
||||
- 需要 consumer group、事件回放、跨服务解耦。
|
||||
|
||||
需要 Spark 的信号:
|
||||
|
||||
- 历史数据到千万或亿级,PostgreSQL 聚合开始吃力。
|
||||
- 需要跨源、长时间窗口、空间时间关联分析。
|
||||
- 原始数据进入 Parquet / Iceberg / Delta 等湖仓,并开始生产离线派生数据产品。
|
||||
|
||||
若目标是秒级连续流计算,优先评估 Flink;Spark 更适合批量或微批分析。
|
||||
|
||||
@@ -32,8 +32,8 @@
|
||||
|
||||
文件:
|
||||
|
||||
- [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/pages/PlainResourcePages.tsx)
|
||||
- [AdminNextRoutes.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/AdminNextRoutes.tsx)
|
||||
- [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
|
||||
- [AdminRoutes.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/AdminRoutes.tsx)
|
||||
|
||||
当前行为:
|
||||
|
||||
@@ -57,11 +57,11 @@
|
||||
|
||||
文件:
|
||||
|
||||
- [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/pages/PlainResourcePages.tsx)
|
||||
- [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
|
||||
|
||||
当前行为:
|
||||
|
||||
- `/collection-management` 按旧版层级收敛为 `采集器`、`采集调度`、`采集历史 / 快照`。
|
||||
- `/collection-management` 按业务层级收敛为 `采集器`、`采集调度`、`采集历史 / 快照`。
|
||||
- `采集器` 是配置页,左侧列表展示采集器配置;右侧表单编辑 endpoint、认证、请求头、采集参数和启用状态。
|
||||
- 新增采集器和目标 Schema 使用草稿详情页,不再用透明 JSON 弹窗;保存后才固化到列表,取消会销毁草稿。
|
||||
- 连接按钮只做连通性测试,不保存配置;保存按钮只持久化表单。
|
||||
@@ -80,7 +80,7 @@
|
||||
- `目标 Schema` 维护可写入目标结构。
|
||||
- `run-mapped`、`stop-mapped`、`stream-status` 负责运行映射后的自定义采集器。
|
||||
|
||||
这些入口在新版中需要表单化,只有高级字段才折叠为 JSON。不要把模板、Schema、运行状态和采集器配置拍平成同一张表。
|
||||
这些入口在 Admin 中需要表单化,只有高级字段才折叠为 JSON。不要把模板、Schema、运行状态和采集器配置拍平成同一张表。
|
||||
|
||||
## 后端接口
|
||||
|
||||
|
||||
@@ -66,10 +66,10 @@ DocsMetadata(
|
||||
)
|
||||
```
|
||||
|
||||
新增公开文档时,需要同步:
|
||||
新增可在 Docs 页面展示的技术文档时,需要同步:
|
||||
|
||||
- 新增中英文 Markdown 文件。
|
||||
- 在服务端 `DOCS_METADATA` 添加 filename、slug、access、group、order、标题。
|
||||
- 在服务端 `DOCS_METADATA` 添加 filename、slug、access、group、order、标题。后端目录接口以这里为准,只改前端 metadata 不会让文档出现在 `/docs` 导航中。
|
||||
- 在前端 [docs-content.ts](/home/ray/dev/linkong/planet/frontend/src/pages/Docs/docs-content.ts) 添加同名 metadata,保持导航标题和排序一致。
|
||||
- 如果需要从 README 发现,更新 `docs/technical/zh/README.md` 和 `docs/technical/en/README.md`。
|
||||
|
||||
@@ -83,7 +83,7 @@ DocsMetadata(
|
||||
- 更新用户时校验组名只能是 `docs_user`、`docs_developer`、`docs_admin`。
|
||||
- 只有 `super_admin` 能修改 Gatekeeper 权限组。
|
||||
|
||||
前端 [Users.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Users/Users.tsx) 展示权限组标签,并在编辑表单中提供多选框。非 `super_admin` 打开的表单会禁用该字段,并在提交前移除 `gatekeeper_groups`。
|
||||
前端 [Users.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/Users.tsx) 展示权限组标签,并在编辑表单中提供多选框。非 `super_admin` 打开的表单会禁用该字段,并在提交前移除 `gatekeeper_groups`。
|
||||
|
||||
## 前端 Docs 加载
|
||||
|
||||
|
||||
@@ -188,7 +188,7 @@ Earth info-card 策略:
|
||||
- 这是预期行为,因为 incident 是聚合和去噪后的结果
|
||||
- 但 incident-first 渲染会让 Earth 显得过于安静,除非有另一层始终可用的 activity layer
|
||||
|
||||
推荐 `activity layer` 的实现细节在 [BGP 区域聚合计划](/home/ray/dev/linkong/planet/docs/plans/earth-bgp-region-aggregation-plan.md) 中展开。
|
||||
推荐 `activity layer` 的未来实现细节保存在仓库路径 `docs/plans/earth-bgp-region-aggregation-plan.md`。
|
||||
|
||||
因此最近的里程碑是:
|
||||
|
||||
@@ -259,7 +259,7 @@ Earth 的 `incident` 层不应该像一大片发光区域,而应该像紧凑
|
||||
|
||||
相关页面:
|
||||
|
||||
- `frontend/src/pages/BGP/BGP.tsx`
|
||||
- `frontend/src/admin/pages/PlainResourcePages.tsx`
|
||||
|
||||
当前 BGP 控制台页面有三层:
|
||||
|
||||
@@ -321,7 +321,7 @@ BGP 专项测试位于:
|
||||
|
||||
前端:
|
||||
|
||||
- `frontend/src/pages/BGP/BGP.tsx`
|
||||
- `frontend/src/admin/pages/PlainResourcePages.tsx`
|
||||
- `frontend/public/earth/js/bgp.js`
|
||||
- `frontend/public/earth/js/main.js`
|
||||
- `frontend/public/earth/js/info-card.js`
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
|
||||
相关规则建议一起参考:
|
||||
|
||||
- [项目规则](/home/ray/dev/linkong/planet/rules.md)
|
||||
- 仓库根目录 `rules.md`
|
||||
- [前端布局指南](/home/ray/dev/linkong/planet/docs/technical/zh/frontend-layout-guidelines.md)
|
||||
|
||||
## 当前目标
|
||||
@@ -56,6 +56,8 @@ React 路由入口:
|
||||
- 各图层集成
|
||||
- Earth 级别状态同步
|
||||
|
||||
Earth 收到 `/ws` 的 `earth_updates` 时只把它当作刷新提示,真实数据仍通过 `/api/v1/visualization/...` 接口重新 GET。数据库驱动的刷新由后端 listener 直接清理缓存再广播,不再默认经过 `earth_refresh` 作业队列;前端收到 `database_changed` 后会按 layer 读取 `clear_then_reload`、`reload` 或 `delta` 策略。`clear_then_reload` 必须先清 Three.js 对象再 no-store 重拉,summary 只做一致性校验,不能用 `0` 作为跳过图层重拉的理由。技术链路见 [数据作业与 Outbox 技术架构](/home/ray/dev/linkong/planet/docs/technical/zh/data-job-earth-sync-architecture.md),业务数据流见 [业务架构与数据流转](/home/ray/dev/linkong/planet/docs/technical/zh/platform-data-flows.md)。
|
||||
|
||||
### 3. 地球控制层
|
||||
|
||||
- [controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js)
|
||||
@@ -162,9 +164,14 @@ Earth 地表是多层近似同心球,不是单一 mesh。`earth.js` 的基座
|
||||
|
||||
`brand.js` 管理 Earth HUD 品牌资源。默认品牌来自静态资源,运行时覆盖值来自 `/api/v1/earth/brand`,上传的图片通过 `/earth-brand-assets/...` 读取。前端必须把 logo/title 图片和文本 fallback 分开处理:图片加载失败时显示文本标题,文本字段为空时使用后端默认值,避免 HUD 品牌区空白。控制台的 Earth 内容页负责保存和重置品牌配置,Earth 前端只消费结果。
|
||||
|
||||
Admin Next 的 Earth 内容页必须按运行时语义组织这些配置:
|
||||
`about.js` 管理 Earth 设置里的“关于”卡片。默认内容仍保留在前端作为兜底,运行时优先读取 `/api/v1/earth/about`。接口失败或字段缺失时必须回退默认值,避免设置页出现空白。Admin 的 Earth 内容页提供“关于”tab,保存走 `PUT /api/v1/earth/about`,恢复默认走 `DELETE /api/v1/earth/about`。
|
||||
|
||||
`oobe.js` 管理 Earth 首次初始化引导。是否显示 OOBE 必须由 `/api/v1/earth/oobe-status` 的 `ready` 字段决定,不能依赖 `localStorage` 判断系统是否初始化。`localStorage` 只允许记录“本浏览器暂时跳过”的短时状态;如果后端已经认为 `ready: true`,退出登录、清空本地缓存或换浏览器都不应再次弹出 OOBE。桌面端使用深色星空遮罩和毛玻璃启动面板,移动端改为底部 sheet,并尊重 `prefers-reduced-motion`。
|
||||
|
||||
Admin 的 Earth 内容页必须按运行时语义组织这些配置:
|
||||
|
||||
- `品牌标识`:品牌预览应使用与 Earth HUD 左上角一致的深色星空背景、尺寸、间距、logo/title 渲染和文本 fallback,而不是普通表单预览。
|
||||
- `关于`:配置 Earth 设置里的 About 卡片,包括 logo、眉标、标题、版本、描述和元信息条目;Earth 运行时从 `/earth/about` 读取,失败时回退默认内容。
|
||||
- `国界精度`:构建边界、刷新状态和恢复默认属于这个分区内部动作,不应放在页面全局工具栏。
|
||||
- `电视直播`:列表同时区分内置源、采集源和自定义源;卡片状态表达启用、停用、新建或错误。内置源不能删除,采集源和自定义源可以删除。新增直播源在点击加号后才进入草稿状态,保存后固化到列表,取消则销毁草稿。
|
||||
- `底图资源`、`图层资源`、`3D 模型`、`新闻锚点策略`:如果后端能力未接入,控制台应明确显示待接入空态,不应混入 TV 或品牌配置。
|
||||
@@ -192,9 +199,7 @@ TV 预览需要尽量复用 Earth 运行时的直播卡片结构和状态标签
|
||||
|
||||
当前 BGP 巡航只是这套能力的一个调用方,不应再把“按队列巡航”和“BGP 事件展示”混写在同一个状态机里。
|
||||
|
||||
新闻巡航摘要计划见:
|
||||
|
||||
- [Earth 新闻巡航摘要计划](/home/ray/dev/linkong/planet/docs/plans/earth-news-cruise-summary-plan.md)
|
||||
新闻巡航摘要的未来计划保存在仓库路径 `docs/plans/earth-news-cruise-summary-plan.md`,不作为公开 Docs 页面入口。
|
||||
|
||||
## 当前样式分层
|
||||
|
||||
|
||||
@@ -17,26 +17,21 @@
|
||||
|
||||
## 本地实际类别
|
||||
|
||||
当前 CelesTrak 卫星分组在 [backend/app/services/collectors/celestrak.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/celestrak.py) 中包括:
|
||||
当前 CelesTrak 采集器优先从 `GROUP=active&FORMAT=json` 拉取完整活跃卫星目录。这样可以避免某个 CelesTrak 分组请求失败时仍把不完整结果保存为成功批次。采集器只在完整 JSON 数组可解析、且记录含 `NORAD_CAT_ID` 时进入转换和保存;网络、续传或解析失败会重试,最终失败时保留上一批 current 数据。
|
||||
|
||||
- `starlink`
|
||||
- `gps-ops`
|
||||
- `galileo`
|
||||
- `glonass`
|
||||
- `beidou`
|
||||
- `leo`
|
||||
- `geo`
|
||||
- `iridium-next`
|
||||
CelesTrak 对 `active` 这类大 group 有每次 GP 数据更新窗口内的重复下载限制。为避免清空数据库后无法立即恢复,Planet 会把原始下载文件保存在 `$PLANET_CACHE_DIR/downloads`。当 `active` 返回“本轮数据未更新”的 403 时:
|
||||
|
||||
其中非 Starlink 类别是:
|
||||
- 如果 `active` 缓存存在,直接用缓存重新写入数据库。
|
||||
- 如果 `active` 缓存不存在,才切换到 fallback group mode,按 `starlink`、`gps-ops`、`galileo`、`glonass`、`beidou`、`leo`、`geo`、`iridium-next` 全部下载并合并。
|
||||
- fallback group mode 要求所有 group 都成功或有缓存可复用;任意 group 缺失都会整体失败,不保存 partial。
|
||||
|
||||
- `gps-ops`
|
||||
- `galileo`
|
||||
- `glonass`
|
||||
- `beidou`
|
||||
- `leo`
|
||||
- `geo`
|
||||
- `iridium-next`
|
||||
采集结果仍会给前端提供 `metadata.constellation_group`,但该字段现在来自可执行推断:
|
||||
|
||||
- `OBJECT_NAME` 以 `STARLINK` 开头时标记为 `starlink`
|
||||
- `OBJECT_NAME` 以 `IRIDIUM` 开头时标记为 `iridium-next`
|
||||
- 其它活跃卫星不强行归入旧 CelesTrak 小分组,避免把泛化类别当成精确星座
|
||||
|
||||
因此,非 Starlink 类别在产品策略中仍包括 GNSS/RNSS、GEO、generic LEO 和 Iridium NEXT 等语义;但只有 fallback group mode 会保存旧的 `gps-ops`、`galileo`、`glonass`、`beidou`、`leo`、`geo` 分组标签,active 主路径不会强行给所有卫星补这类标签。
|
||||
|
||||
## 资料结论
|
||||
|
||||
@@ -136,7 +131,8 @@
|
||||
|
||||
1. 后端把星座分组和 footprint 策略提示透给前端
|
||||
|
||||
- CelesTrak collector 会把 `GROUP` 记入 `metadata.constellation_group`
|
||||
- CelesTrak collector 会把原始查询来源记入 `metadata.celestrak_query_group = active`
|
||||
- `metadata.constellation_group` 只保存可推断的业务星座,例如 `starlink` 和 `iridium-next`
|
||||
- Visualization API 会输出:
|
||||
- `properties.constellation_group`
|
||||
- `properties.footprint_policy`
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
|
||||
相关规则建议一起参考:
|
||||
|
||||
- [项目规则](/home/ray/dev/linkong/planet/rules.md)
|
||||
- 仓库根目录 `rules.md`
|
||||
- [前端布局指南](/home/ray/dev/linkong/planet/docs/technical/zh/frontend-layout-guidelines.md)
|
||||
|
||||
## 当前目标
|
||||
@@ -22,7 +22,7 @@
|
||||
|
||||
- [App.tsx](/home/ray/dev/linkong/planet/frontend/src/App.tsx)
|
||||
|
||||
当前正式后台路由已经由 Admin Next 接管:
|
||||
当前正式后台路由已经由 Admin 接管:
|
||||
|
||||
- `/admin`
|
||||
- `/users`
|
||||
@@ -37,23 +37,7 @@
|
||||
- `/collection-management`
|
||||
- `/settings`
|
||||
|
||||
这些路径渲染 [AdminNextRoutes.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/AdminNextRoutes.tsx),页面清单和菜单元信息来自 [manifest.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/routes/manifest.tsx)。`/admin-next/*` 仅作为兼容入口存在,会重定向到上面的正式路径,不再作为并行主入口。
|
||||
|
||||
旧 AntD 控制台保留在 `/legacy/admin/*`,用于对照和回退:
|
||||
|
||||
- `/legacy/admin`
|
||||
- `/legacy/admin/datasources`
|
||||
- `/legacy/admin/data`
|
||||
- `/legacy/admin/collection-management`
|
||||
- `/legacy/admin/earth-content`
|
||||
- `/legacy/admin/ai`
|
||||
- `/legacy/admin/logs`
|
||||
- `/legacy/admin/settings`
|
||||
- `/legacy/admin/users`
|
||||
- `/legacy/admin/bgp`
|
||||
- `/legacy/admin/alerts/*`
|
||||
|
||||
旧页面、`AppLayout`、`antd` 和 `@ant-design/icons` 在 legacy 验收期继续保留。不要在新版 parity 验收前删除这些文件或依赖。
|
||||
这些路径渲染 [AdminRoutes.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/AdminRoutes.tsx),页面清单和菜单元信息来自 [manifest.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/routes/manifest.tsx)。Admin 是唯一后台控制台入口,不再维护并行控制台或回退路由。
|
||||
|
||||
`/earth` 是独立展示页,不属于控制台骨架。
|
||||
|
||||
@@ -61,7 +45,7 @@
|
||||
|
||||
正式后台公共壳层在:
|
||||
|
||||
- [AdminNextLayout.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/components/layout/AdminNextLayout.tsx)
|
||||
- [AdminLayout.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/components/layout/AdminLayout.tsx)
|
||||
|
||||
职责:
|
||||
|
||||
@@ -69,38 +53,13 @@
|
||||
- 当前账号、版本、退出登录和主题切换
|
||||
- 顶部搜索、面包屑和页面快捷入口
|
||||
- 内容区单屏高度闭合
|
||||
- Admin Next 内部滚动、表格、详情面板和移动端详情视图协调
|
||||
- Admin 内部滚动、表格、详情面板和移动端详情视图协调
|
||||
|
||||
旧 AntD legacy 壳层仍在:
|
||||
后续正式控制台页面应适配 `AdminLayout` 和 Admin 页面模式;不要重新引入并行后台壳层。
|
||||
|
||||
- [AppLayout.tsx](/home/ray/dev/linkong/planet/frontend/src/components/AppLayout/AppLayout.tsx)
|
||||
## Admin 分区加载策略
|
||||
|
||||
legacy 职责:
|
||||
|
||||
- 左侧导航
|
||||
- 折叠与展开
|
||||
- 当前账号/版本信息
|
||||
- 内容区高度闭合
|
||||
- 全站统一侧边栏滚动条
|
||||
|
||||
当前结构是:
|
||||
|
||||
```tsx
|
||||
<Layout className="dashboard-layout">
|
||||
<Sider className="dashboard-sider">...</Sider>
|
||||
<Layout>
|
||||
<Content className="dashboard-content">
|
||||
<div className="dashboard-content-inner">{children}</div>
|
||||
</Content>
|
||||
</Layout>
|
||||
</Layout>
|
||||
```
|
||||
|
||||
后续正式控制台页面应优先适配 `AdminNextLayout` 和 Admin Next 页面模式,而不是继续往旧 `AppLayout` 增加新能力。只有维护 `/legacy/admin/*` 时才应修改旧壳层。
|
||||
|
||||
## Admin Next 分区加载策略
|
||||
|
||||
多 tab 页面由 [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin-next/pages/PlainResourcePages.tsx) 统一承载当前的管理型和信息型工作台。分区加载规则是:
|
||||
多 tab 页面由 [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx) 统一承载当前的管理型和信息型工作台。分区加载规则是:
|
||||
|
||||
- 初次进入页面只请求当前 active tab,不预先拉取所有 tab 的接口。
|
||||
- 用户切换 tab 时懒加载该 tab;已经加载过的 tab 保留在本地 `states` 缓存中,切回时直接复用。
|
||||
@@ -110,6 +69,29 @@ legacy 职责:
|
||||
|
||||
这样做是为了降低 Earth、AI、采集管理等多分区页面的冷启动压力,同时保留 tab 数量、状态和用户切换后的缓存体验。需要全量健康巡检时应走后端健康接口或显式刷新流程,不要依赖页面初始化时顺手拉所有业务接口。
|
||||
|
||||
## 数据源采集队列
|
||||
|
||||
Admin 的数据源页把单源触发、表格勾选触发和触发全部统一接入浏览器下载列表式采集队列:
|
||||
|
||||
- 队列状态由 [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx) 管理,只保存当前会话中的可见任务,不用 `localStorage` 伪造历史。
|
||||
- 任务进度优先消费 `/ws` 的 `datasource_tasks` channel;如果 WebSocket 未连接或没有及时返回,则轮询 `/api/v1/datasources/{id}/task-status`。
|
||||
- 触发接口返回的 `triggered`、`skipped`、`failed` 会立即进入队列;刷新页面后只根据后端当前仍在 `running/pending/queued` 的数据源恢复队列。
|
||||
- `内置源` 分区通过表格选择列收敛批量触发。没有勾选时主按钮是“触发全部”;勾选后同一个主按钮变成“触发已选 N”,不再提供手填 ID 的独立批量按钮。
|
||||
- 页面内容流不再承载展开队列,避免全量触发后挤压列表和详情面板。右上角 actions 区的队列按钮沿用现有 `Button` 样式;空态使用 `ListChecks` 图标,有任务时只显示纯圆环总进度。点击后打开浮层,按运行中、失败、完成、跳过分组。
|
||||
- 队列项可以跳转到对应数据源详情,失败项可以重试。详情页内的“采集任务”摘要只展示当前数据源最近任务,不承担保存配置职责。
|
||||
- 运行中的单源采集按钮显示为“停止采集”;队列中未完成项右侧提供取消按钮。取消调用 `/api/v1/datasources/{source_id}/tasks/{task_id}/cancel`,后端语义是保留已提交批次并回滚未完成批次。
|
||||
- 删除数据库数据和清理展示缓存也会进入同一队列,前端只展示任务状态,不假定接口同步完成。
|
||||
|
||||
这个队列是用户感知层,不替代后端调度状态。后端仍然是任务是否运行、完成、失败或跳过的唯一事实来源。
|
||||
|
||||
## Admin 主题滑块
|
||||
|
||||
Admin 侧栏底部主题切换继续复用共享 [SegmentedControl.tsx](/home/ray/dev/linkong/planet/frontend/src/components/SegmentedControl/SegmentedControl.tsx),但主题变量在 [styles.css](/home/ray/dev/linkong/planet/frontend/src/admin/styles.css) 内跟随 `data-theme` 覆盖:
|
||||
|
||||
- light 下使用 `--d-segment-bg: #eef3f9`、白色 slider 和轻投影。
|
||||
- dark 下使用与 Docs 一致的深色底座、`#202938` slider 和深色外投影。
|
||||
- 业务侧只隐藏文字 label 并保留 icon + tooltip,不重写 slider DOM。
|
||||
|
||||
## 当前共享组件
|
||||
|
||||
### 1. `Scrollbar`
|
||||
@@ -143,8 +125,7 @@ legacy 职责:
|
||||
|
||||
当前使用场景:
|
||||
|
||||
- Admin Next 数据源、采集数据、采集管理、日志、告警和 BGP 页面
|
||||
- 旧 AntD legacy 页面通过兼容封装继续使用共享滚动能力
|
||||
- Admin 数据源、采集数据、采集管理、日志、告警和 BGP 页面
|
||||
|
||||
### 3. `TableScrollRegion`
|
||||
|
||||
@@ -167,7 +148,7 @@ legacy 职责:
|
||||
|
||||
用途:
|
||||
|
||||
- Admin Next 全局工具按钮和详情页工具按钮
|
||||
- Admin 全局工具按钮和详情页工具按钮
|
||||
- icon-only + tooltip 的普通操作
|
||||
- 保存、创建、确认、删除、停止等强意图操作
|
||||
- 与 Docs 主题滑块一致的紧凑开关
|
||||
@@ -225,39 +206,19 @@ legacy 职责:
|
||||
- 文档内部链接应通过 `transformLink` 转成 `/docs/:slug`
|
||||
- 标题锚点由 `getHeadingId` 注入,避免渲染器自己理解路由状态
|
||||
|
||||
### 7. `ConnectionTestInput`
|
||||
### 7. Admin UI primitives
|
||||
|
||||
文件:
|
||||
|
||||
- [ConnectionTestInput.tsx](/home/ray/dev/linkong/planet/frontend/src/components/ConnectionTestInput/ConnectionTestInput.tsx)
|
||||
- [button.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/components/ui/button.tsx)
|
||||
- [dialog.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/components/ui/dialog.tsx)
|
||||
- [switch.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/components/ui/switch.tsx)
|
||||
|
||||
用途:
|
||||
|
||||
- Endpoint、Base URL 这类“输入值 + 连接验证”的控制台表单项
|
||||
- AI Provider 和 WebSearch 的连接测试入口
|
||||
- 后续采集器配置如果把连接测试放进输入框,也应复用它
|
||||
|
||||
当前约束:
|
||||
|
||||
- 输入框末端只显示一个插头/连接器图标,不再并排放“测试连接”文字按钮
|
||||
- 禁用的集成能力必须同时置灰输入框和连接测试按钮
|
||||
- 组件只负责输入框与测试入口组合,不保存业务状态;调用方仍负责表单值、loading、disabled 和连接请求
|
||||
|
||||
### 8. `TableActions`
|
||||
|
||||
文件:
|
||||
|
||||
- [TableActions.tsx](/home/ray/dev/linkong/planet/frontend/src/components/TableActions/TableActions.tsx)
|
||||
|
||||
用途:
|
||||
|
||||
- 表格操作列的统一操作入口
|
||||
- 展开状态下直接展示按钮
|
||||
- 收起状态下用更多菜单承载操作
|
||||
|
||||
配套导出:
|
||||
|
||||
- `actionCellProps`:用于操作列 `onCell`,防止操作按钮被省略号截断或换行
|
||||
- 全局工具按钮、详情页动作、确认弹窗和二元设置。
|
||||
- 与 Tactile UI token 对齐,保持 Admin 内部控件尺寸、hover、disabled 和 dark mode 一致。
|
||||
- 表格行内动作优先使用 icon button + tooltip/title,不重新引入独立操作菜单组件。
|
||||
|
||||
## 当前状态来源
|
||||
|
||||
@@ -280,7 +241,7 @@ legacy 职责:
|
||||
|
||||
文件:
|
||||
|
||||
- [AISettings.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/AISettings/AISettings.tsx)
|
||||
- [AISettings.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
|
||||
|
||||
职责:
|
||||
|
||||
@@ -316,7 +277,7 @@ legacy 职责:
|
||||
|
||||
例如:
|
||||
|
||||
- [Dashboard.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Dashboard/Dashboard.tsx)
|
||||
- [Dashboard.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/Dashboard.tsx)
|
||||
|
||||
优先目标:
|
||||
|
||||
@@ -328,10 +289,10 @@ legacy 职责:
|
||||
|
||||
例如:
|
||||
|
||||
- [DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/DataSources/DataSources.tsx)
|
||||
- [DataList.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/DataList/DataList.tsx)
|
||||
- [Users.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Users/Users.tsx)
|
||||
- [Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Settings/Settings.tsx)
|
||||
- [DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
|
||||
- [DataList.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/DataList.tsx)
|
||||
- [Users.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/Users.tsx)
|
||||
- [Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
|
||||
|
||||
约束:
|
||||
|
||||
@@ -341,7 +302,7 @@ legacy 职责:
|
||||
|
||||
### 数据源目录页
|
||||
|
||||
[DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/DataSources/DataSources.tsx) 当前不再承担配置编辑职责,而是数据源目录和采集操作页。
|
||||
[DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx) 当前不再承担配置编辑职责,而是数据源目录和采集操作页。
|
||||
|
||||
当前页面边界:
|
||||
|
||||
@@ -365,7 +326,9 @@ legacy 职责:
|
||||
|
||||
### 采集器设置页
|
||||
|
||||
[Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Settings/Settings.tsx) 会按路由进入三种模式:`/settings` 是系统设置,`/earth-content` 是 Earth 内容,`/collection-management` 是采集管理。`collector_credentials` tab 当前在 `/collection-management` 下显示为“采集器”。
|
||||
[Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx) 会按路由进入三种模式:`/settings` 是系统设置,`/earth-content` 是 Earth 内容,`/collection-management` 是采集管理。`collector_credentials` tab 当前在 `/collection-management` 下显示为“采集器”。
|
||||
|
||||
`/settings` 的“系统显示”分区包含 `演示模式` 开关。开启后,Earth 的 OOBE 会忽略“已有当前采集数据”和本地“先浏览”临时跳过状态,直接展示初始化引导;该开关仅用于演示/验收流程,不改变数据源、采集队列或 Earth 内容资源配置。
|
||||
|
||||
当前页面边界:
|
||||
|
||||
@@ -398,7 +361,7 @@ legacy 职责:
|
||||
|
||||
### Earth 内容页
|
||||
|
||||
`/earth-content` 复用 [Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Settings/Settings.tsx) 的单屏 tab 容器,但页面责任与系统设置分离:
|
||||
`/earth-content` 复用 [Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx) 的单屏 tab 容器,但页面责任与系统设置分离:
|
||||
|
||||
- `电视直播` 迁移原直播源配置,继续管理 Earth 媒体面板内容源。
|
||||
- `国界精度` 管理 Earth 静态国界资产:provider 状态、低精 fallback、高精 manifest/PMTiles、源配置 JSON 和构建动作。
|
||||
@@ -410,8 +373,8 @@ legacy 职责:
|
||||
|
||||
例如:
|
||||
|
||||
- [BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx)
|
||||
- [Playground.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Playground/Playground.tsx)
|
||||
- [BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
|
||||
- [Playground.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
|
||||
|
||||
约束:
|
||||
|
||||
|
||||
@@ -8,7 +8,8 @@
|
||||
|
||||
当前推荐参考实现:
|
||||
|
||||
- [frontend/src/pages/BGP/BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx)
|
||||
- [frontend/src/admin/pages/PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
|
||||
- [frontend/src/admin/pages/DataList.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/DataList.tsx)
|
||||
- [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css)
|
||||
|
||||
## 核心原则
|
||||
@@ -23,15 +24,15 @@
|
||||
推荐结构:
|
||||
|
||||
```tsx
|
||||
<AppLayout>
|
||||
<AdminLayout>
|
||||
<div className="page-shell">
|
||||
<div className="page-shell__header">...</div>
|
||||
<div className="page-shell__body">...</div>
|
||||
</div>
|
||||
</AppLayout>
|
||||
</AdminLayout>
|
||||
```
|
||||
|
||||
页面总高度应被限制在 `AppLayout` 内容区内,而不是继续让整个页面自然向下增长。
|
||||
页面总高度应被限制在 `AdminLayout` 内容区内,而不是继续让整个页面自然向下增长。
|
||||
|
||||
### 2. 滚动优先发生在模块内部
|
||||
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
`backend/app/services/location/` 是所有“给定一条记录,决定它的 lat/lon”业务的共享抽象。算力中心、BGP 观测站、BGP 事件目前都跑在这条管线上。未来需要位置估算的实体,例如卫星地面站、用户认领点位、IXP 设施,也应接入这里,而不是各自再写地理解析逻辑。
|
||||
|
||||
用户侧流程见 [Earth 位置候选采集使用手册](/home/ray/dev/linkong/planet/docs/technical/zh/location-pipeline-user.md)。
|
||||
用户侧流程见 [Planet 使用手册](/home/ray/dev/linkong/planet/docs/technical/zh/manual.md) 的 Earth 位置候选采集章节。
|
||||
|
||||
## 设计目标
|
||||
|
||||
|
||||
@@ -191,19 +191,22 @@ Base URL 输入框尾端的插头图标会触发连接测试。测试通过会
|
||||
`/earth-content` 位于控制台“运维与配置”下,面向 Earth 前端体验资源:
|
||||
|
||||
- **品牌资源**:维护 Earth HUD 使用的 logo、标题图、标题文本、副标题和描述;上传的图片会保存为 Earth 品牌资产并立即供 Earth 页面读取。
|
||||
- **关于**:维护 Earth 设置面板里的关于卡片,包括 logo、眉标、标题、版本、描述和元信息。
|
||||
- **电视直播**:维护 Earth 媒体面板里的直播源。
|
||||
- **国界精度**:查看当前国界 provider、低精 fallback、高精 PMTiles/manifest 状态,编辑本机源配置并手动构建。
|
||||
- **地球底图**、**图层资源**、**三维素材**、**新闻锚点策略**:目前是待接入占位页,不展示假数据。
|
||||
|
||||
Earth 页面工具栏齿轮中也有“国界精度”。切到“高精”时,如果本机尚未构建高精资产,会像游戏更新包一样启动后台下载/构建并显示百分比;构建成功后自动应用,无需刷新。切回“低精”只切换本机显示偏好,不重新下载。
|
||||
|
||||
如果后端判断 Earth 尚未初始化,首次进入 `/earth` 会出现毛玻璃引导,提示登录控制台并采集数据。这个判断来自后端真实数据状态;如果系统已经有已采集数据,清空浏览器缓存也不会重新弹出。
|
||||
|
||||
### 采集管理
|
||||
|
||||
`/collection-management` 位于控制台“运维与配置”下,面向采集生命周期:
|
||||
|
||||
- **采集器**:维护 endpoint、请求头、凭证、timeout、retry,并运行连接检查。
|
||||
- **采集调度**:维护原有调度相关设置。
|
||||
- **采集历史 / 快照**:当前是待接入占位页,后续承载 collection task、snapshot、collected data 浏览能力。
|
||||
- **采集历史 / 快照**:按数据源聚合历史快照,详情页可用 Time Capsule 下拉切换不同版本。
|
||||
|
||||
### SMTP 邮件设置
|
||||
|
||||
@@ -233,7 +236,7 @@ Earth 页面工具栏齿轮中也有“国界精度”。切到“高精”时
|
||||
|
||||
## 数据探索
|
||||
|
||||
- `/datasources`:数据源目录。`采集任务` tab 面向一次性/定时采集器,可以按产品域、层级、启用状态、最近执行状态、是否已有采集数据和关键词筛选;勾选多行后可批量采集选中项,未勾选时“一键采集”触发当前筛选范围。`实时流` tab 面向 AISStream / WebSocket 长连接,展示连接健康、累计入库、时间窗统计和启动 / 停止 / 重连操作。点击名称打开信息抽屉查看 endpoint、请求头、基础配置和是否内置;接口、凭证、请求头的编辑统一在 `/collection-management` 的"采集器"。总体进度下方的 `采集中 N` 标签可点击,展开当前采集任务列表
|
||||
- `/datasources`:数据源目录。`内置源` 支持按产品域、层级、启用状态、最近执行状态、是否已有采集数据和关键词筛选;未勾选时主按钮显示“触发全部”,勾选多行后会变成“触发已选 N”,并只提交所选数据源。右上角队列按钮空态显示队列图标,有任务时显示纯圆环总进度;点击后打开队列浮层,按运行中、完成、失败和跳过分组,失败项可重试,完成项可跳到详情。`实时源` 面向 AISStream / WebSocket 长连接,展示连接健康、累计入库、时间窗统计和启动 / 停止 / 重连操作。接口、凭证、请求头的编辑统一在 `/collection-management` 的"采集器"。
|
||||
- `/data`:采集后数据表,适合排查"数据是否已经进入系统"、"更新时间是否符合预期"、"某个数据源是否产出有效记录"
|
||||
- `/bgp`:BGP 专题页面,列表 + 详情 + 研判,与 Earth 的 BGP 图层互补
|
||||
- `/alerts/system`、`/alerts/bgp`、`/alerts/situational`:系统、BGP、态势告警
|
||||
@@ -280,7 +283,17 @@ AIS 船只图例按船型显示颜色:货轮、油轮、客船、渔船、军
|
||||
|
||||
算力中心和 BGP 观测站详情卡支持自动采集坐标候选。点击对象后用"自动采集坐标候选"或"重新自动采集坐标"按钮,后端会从源坐标、开放组织注册 API 和在线地理编码中整理候选;常规来源没有候选时使用当前默认 AI Provider 做 LLM factcheck 兜底。BGP 观测站的已存储位置只用于补齐查询上下文,不会作为候选直接返回。
|
||||
|
||||
候选可以直接在 Earth 预览。算力中心候选点击"保存"后写入 `compute_center_locations` 维表并刷新图层。算力中心图层左上角的通知气泡显示无法渲染的待定位数量;点击查看列表,单条采集候选,或用"一键采用"从上到下保存最高置信候选。没有可用候选的记录会留在列表中,不会被国家中心点或硬编码 hint 伪造位置。详细流程见 [Earth 位置候选采集使用手册](/home/ray/dev/linkong/planet/docs/technical/zh/location-pipeline-user.md)。
|
||||
候选可以直接在 Earth 预览。算力中心候选点击"保存"后写入 `compute_center_locations` 维表并刷新图层。算力中心图层左上角的通知气泡显示无法渲染的待定位数量;点击查看列表,单条采集候选,或用"一键采用"从上到下保存最高置信候选。没有可用候选的记录会留在列表中,不会被国家中心点或硬编码 hint 伪造位置。
|
||||
|
||||
单个对象的推荐流程:
|
||||
|
||||
1. 打开算力中心或 BGP 观测站详情卡。
|
||||
2. 点击"自动采集坐标候选"。
|
||||
3. 等待候选列表返回;有 WebSearch / AI factcheck 依赖的候选会显示采集中状态。
|
||||
4. 在 Earth 上预览候选位置。
|
||||
5. 确认可用候选后点击"保存";不确定时关闭卡片不会丢失当前任务状态。
|
||||
|
||||
一键定位用于批量处理算力中心待定位队列。它会从列表顶部开始采用最高置信候选;仍没有事实依据的记录会保留在队列中。未开启 WebSearch 时,单个定位和一键定位会置灰,因为位置核验依赖事实查询。
|
||||
|
||||
### 设置
|
||||
|
||||
@@ -351,7 +364,7 @@ Earth 预留了动作捕捉控制入口。实时链路两种输入源:
|
||||
|
||||
文档站 `http://localhost:3000/docs` 由后端按权限读取,不再把全部 Markdown 直接打进前端构建产物。
|
||||
|
||||
未登录访客默认只能看到 `public` 文档:首页、快速开始、使用手册、常见问题、Earth 位置候选采集使用手册。登录用户被分配 Gatekeeper 权限组后可以看到更多技术文档:
|
||||
未登录访客默认只能看到 `public` 文档:首页、快速开始、使用手册、常见问题。登录用户被分配 Gatekeeper 权限组后可以看到更多技术文档:
|
||||
|
||||
- `docs_user`:用户操作类文档
|
||||
- `docs_developer`:Earth、前端、后端、采集器和 AI Provider 等开发文档
|
||||
@@ -365,5 +378,4 @@ Docs 支持分类导航、Markdown 渲染、表格和代码块、文档内目录
|
||||
|
||||
- [快速开始](/home/ray/dev/linkong/planet/docs/technical/zh/quickstart.md)
|
||||
- [常见问题](/home/ray/dev/linkong/planet/docs/technical/zh/faq.md)
|
||||
- [Earth 位置候选采集使用手册](/home/ray/dev/linkong/planet/docs/technical/zh/location-pipeline-user.md)
|
||||
- [Planet 运维手册](/home/ray/dev/linkong/planet/docs/technical/zh/ops-runbook.md)
|
||||
|
||||
@@ -15,7 +15,7 @@
|
||||
| English / Key | 中文显示名 | 使用场景 |
|
||||
| --- | --- | --- |
|
||||
| Planet | Planet | 产品名,保留英文 |
|
||||
| Admin Next | 新控制台 | 新版管理端上下文 |
|
||||
| Admin | 控制台 | 管理端上下文 |
|
||||
| Earth | Earth | 地球可视化产品名,保留英文 |
|
||||
| datasource | 数据源 | API、列表、筛选 |
|
||||
| collector | 采集器 | 采集任务、凭证配置 |
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user