release: bump version to 0.74.0

.gitignore

@@ -25,11 +25,14 @@ __pycache__/
 build/
 develop-eggs/
 dist/
-downloads/
+downloads/*
+!downloads/usbipd-win/
+downloads/usbipd-win/*
+!downloads/usbipd-win/usbipd-win-5.3.0.msi
 eggs/
 .eggs/
-lib/
-lib64/
+/lib/
+/lib64/
 parts/
 sdist/
 var/

AGENTS.md

@@ -1,19 +1,124 @@
-# agents.md
+# AGENTS.md
 
 **AI Agent 角色设定。定义 AI 如何行为、沟通和工作。**
 
 ---
 
+## Harness Compatibility
+
+This file is the single authoritative agent guide for the Planet repository.
+The older lowercase `agents.md` entry has been merged here so coding agents and
+harness tools use one source of truth.
+
+### Read First
+
+Read these files before changing code:
+
+1. `rules.md` - mandatory repository rules. Always load `core`, `security`, and
+   `workflow`; load `docs`, `uiux`, `frontend`, `backend`, `earth`, `ai`, or
+   `release` when the task touches those areas.
+2. `AGENTS.md` - this file, including role, communication, workflow, and
+   harness compatibility guidance.
+3. `project_context.md` - static project background. Prefer newer
+   implementation docs when this context disagrees with current code.
+4. `README.md` - current architecture, startup, and toolchain summary.
+5. `docs/HARNESS.md` - harness workflow, conflict policy, and validation tiers.
+6. `CODEMAP.md` - codebase entry points, ownership boundaries, and deeper docs.
+
+For documentation work, also read `docs/documentation-coverage-rules.md`.
+
+### Start Safely
+
+Before editing:
+
+```bash
+git status --short
+scripts/harness/doctor.sh
+```
+
+Use focused context commands before reading large files:
+
+```bash
+rg -n "<symbol-or-term>" <path>
+git diff --stat HEAD
+git diff --name-only HEAD
+git diff --unified=0 HEAD -- <path>
+```
+
+When the user provides screenshots or images, inspect the actual image before
+making visual claims. If the referenced path is missing, search alternate
+attachment/worktree/local locations or ask for the file; never guess the image
+content from text, filenames, or memory.
+
+Preserve user changes already present in the worktree.
+
+### Validation
+
+Fast local harness validation:
+
+```bash
+scripts/harness/quick-check.sh
+```
+
+Full local validation:
+
+```bash
+scripts/harness/validate.sh
+```
+
+`validate.sh` includes the quick check and the frontend Bun build. Docker image
+smoke builds are intentionally opt-in:
+
+```bash
+PLANET_HARNESS_DOCKER_SMOKE=1 scripts/harness/validate.sh
+```
+
+Harness scripts resolve `bun`, `uv`, and optional delivery tools from the
+current non-interactive environment first. If a tool is missing there, they ask
+the user's login interactive shell instead of assuming a specific dotfile.
+
+### High-Risk Areas
+
+- `planet.sh` owns local lifecycle, ports, WSL/LAN behavior, and destructive
+  `destroy` cleanup.
+- Frontend package management is Bun-only. Do not use npm, pnpm, or yarn.
+- Frontend changes must satisfy `scripts/harness/frontend-rules-check.sh`; use
+  rendered smoke evidence for public pages, auth guards, authenticated admin
+  route/section availability, safe navigation/search/tab interactions, mobile
+  layout, and 125% / 150% zoom, not only a build.
+- Admin or Docs layout changes must load `rules.md` `uiux` and preserve the
+  one-screen (`一屏` / `首屏`) height chain: route roots use `height: 100%`,
+  intermediate wrappers keep `min-height: 0`, and only the intended child owns
+  scrolling.
+- `aiprovider` is a protocol/provider adapter; keep business prompts and product
+  workflows in the backend.
+- Earth rendering depends on layer order, depth behavior, picking, and
+  performance-sensitive Three.js code.
+- Secrets belong in environment files or configured settings stores, never in
+  committed files.
+- Backend service code must use structured logging instead of `print()` or
+  debugger calls; `scripts/harness/backend-rules-check.sh` enforces this.
+
+### Conflict Policy
+
+Existing project rules and workflows win. If new harness guidance conflicts with
+`rules.md`, `AGENTS.md`, current docs, scripts, or CI, keep the existing
+behavior and document the compatibility note in `docs/harness-audit.md` or
+`docs/HARNESS.md`.
+
+---
+
 ## Identity
 
 You are **opencode**, an AI coding assistant specialized in enterprise-level systems.
 
 You are working on the **智能星球计划 (Intelligent Planet Plan)** - a situational awareness system for data-centric competition featuring:
 - Python FastAPI backend
-- React Admin dashboard
-- Unreal Engine 5 3D visualization
+- React admin console, public Docs UI, and browser Web Earth shell
+- AI Provider model gateway
 - Multi-source data collection
-- Polarized 3D large display (4K, 120Hz)
+- Future physical display directions such as UE5 / Cesium remain optional
+  roadmap work, not the active local development loop
 
 ---
 
@@ -226,6 +331,6 @@ class TOP500Collector:
 
 1. **Rules are hard constraints** - follow `rules.md` absolutely
 2. **Context provides understanding** - use `project_context.md` for background
-3. **Role defines behavior** - follow `agents.md` for how to work
+3. **Role defines behavior** - follow `AGENTS.md` for how to work
 4. **Quality over speed** - Enterprise systems require precision
 5. **Communicate clearly** - Precision in, precision out

CODEMAP.md

@@ -0,0 +1,109 @@
+# Code Map
+
+This map gives agents and maintainers a quick orientation without replacing the
+deeper architecture docs. Current implementation docs under `docs/technical/`
+are the source of detail for specific subsystems.
+
+## Top-Level Areas
+
+| Path | Role | Notes |
+| --- | --- | --- |
+| `backend/` | FastAPI backend, auth, APIs, data collectors, AI task orchestration, persistence | Tests live in `backend/tests/`; run backend tests from `backend/` with the root uv project. |
+| `frontend/` | React admin console, Docs UI, Web Earth shell, Vite build | Use Bun only. Public Earth assets live under `frontend/public/earth/`. |
+| `aiprovider/` | Model provider/protocol adapter service | Keep it free of product-specific prompts and workflows. |
+| `motion_agent/` | Motion capture protocol service used by `planet.sh` | Often dry-runs when cameras are unavailable, especially in WSL. |
+| `scripts/` | Utility scripts and harness wrappers | Harness commands live in `scripts/harness/`. |
+| `docs/` | Plans, technical docs, changelog, harness docs | Public technical docs are explicitly registered by the frontend Docs catalog. |
+| `deploy/helm/planet/` | Helm chart for staging/deployment smoke paths | CI runs helm lint/template when delivery checks are available. |
+| `.gitea/workflows/` | CI, release image build, staging deploy workflows | This repository uses Gitea workflow files, not `.github/workflows/`. |
+| `planet.sh` | Main local lifecycle script | Owns init/start/restart/stop/health/log/createuser/destroy. |
+
+## Runtime Entry Points
+
+| Runtime | Entry Point | Validation |
+| --- | --- | --- |
+| Local full stack | `./planet.sh start` | `./planet.sh health` |
+| Backend API | `backend/app/main.py` | `cd backend && uv run --frozen --group dev --project .. python -m pytest -q` |
+| Frontend app | `frontend/src/main.tsx` and `frontend/vite.config.mts` | `cd frontend && bun run build` |
+| AI Provider | `aiprovider/main.py` | `curl http://localhost:8010/health` after startup |
+| Motion Agent | `python -m motion_agent` via `planet.sh` | `./planet.sh health` or dry-run startup |
+| Docs UI | `frontend/src/pages/Docs/` | Docs catalog metadata plus frontend build |
+
+## Ownership Boundaries
+
+- Backend owns business state, auth, evidence collection, prompt selection, AI
+  task orchestration, and database persistence.
+- `aiprovider` owns provider identity, request adapter style, model gateway
+  retries, and health/status endpoints only.
+- Frontend owns operator workflows, Docs presentation, Web Earth orchestration,
+  and client-side state that mirrors backend truth.
+- Web Earth rendering changes must preserve documented layer order, altitude
+  offsets, picking behavior, legend semantics, and performance constraints.
+- `planet.sh` owns local environment bootstrap and service lifecycle. Prefer
+  wrapping it from harness scripts instead of duplicating its internals.
+- Harness scripts source `scripts/harness/lib.sh` so agent shells that cannot
+  see `bun` or `uv` in non-interactive `PATH` can still resolve the user's login
+  interactive command path without hardcoding `.zshrc`.
+
+## Validation Commands
+
+```bash
+scripts/harness/doctor.sh
+scripts/harness/security-check.sh
+scripts/harness/backend-rules-check.sh
+scripts/harness/frontend-rules-check.sh
+scripts/harness/docs-consistency-check.sh
+scripts/harness/quick-check.sh
+scripts/harness/validate.sh
+./planet.sh health
+```
+
+CI-equivalent local checks:
+
+```bash
+cd backend
+uv run --frozen --group dev --project .. python -m pytest -s tests/test_api.py tests/test_realtime_sources.py -q
+
+cd frontend
+bun install --frozen-lockfile
+bun run build
+PLANET_FRONTEND_SMOKE_URL=http://127.0.0.1:4173 bun ../scripts/harness/frontend-smoke.mjs
+```
+
+The frontend smoke covers public routes, unauthenticated admin guards,
+login-error handling, the Earth iframe entry, and authenticated `super_admin`
+admin route/section rendering with mocked API data. Authenticated admin checks
+run on desktop, mobile, and 125% / 150% zoom; desktop and mobile passes also
+check for accidental global horizontal overflow. A second smoke layer exercises
+safe desktop/mobile navigation, admin search, section tab switching, dialog
+opening, and non-destructive shortcut links.
+
+Optional delivery smoke, when Docker and Helm are available:
+
+```bash
+PLANET_HARNESS_DOCKER_SMOKE=1 scripts/harness/validate.sh
+```
+
+## Deeper Docs
+
+| Topic | Start Here |
+| --- | --- |
+| Data products and flows | `docs/technical/zh/platform-data-flows.md` and `docs/technical/en/platform-data-flows.md` |
+| Operations and local lifecycle | `docs/technical/zh/ops-runbook.md` and `docs/technical/en/ops-runbook.md` |
+| `planet.sh` startup behavior | `docs/technical/zh/ops-planet-sh-startup.md` and `docs/technical/en/ops-planet-sh-startup.md` |
+| AI Provider | `docs/technical/zh/agents-aiprovider.md` and `docs/technical/en/agents-aiprovider.md` |
+| Admin frontend | `docs/technical/zh/frontend-admin-frontend-context.md` and `docs/technical/en/frontend-admin-frontend-context.md` |
+| Earth frontend | `docs/technical/zh/earth-frontend-context.md` and `docs/technical/en/earth-frontend-context.md` |
+| Earth render order | `docs/technical/zh/earth-render-layer-order.md` and `docs/technical/en/earth-render-layer-order.md` |
+| Documentation rules | `docs/documentation-coverage-rules.md` |
+| Harness workflow | `docs/HARNESS.md` |
+
+## Known Sharp Edges
+
+- `project_context.md` is static background for agents. It now labels future
+  stack directions separately, but current code and technical docs still win
+  when details diverge.
+- README now describes Web Earth, React admin, FastAPI, and `aiprovider` as the
+  active local development shape.
+- Local `destroy` is intentionally destructive for Planet-owned Docker and build
+  state. Never run it as a validation shortcut.

README.md

@@ -83,7 +83,7 @@
 | 组件 | 用途 |
 |------|------|
 | React 18 | UI 框架 |
-| Ant Design Pro | 管理后台组件 |
+| Tactile UI / Radix primitives / lucide-react | 管理后台组件、基础交互与图标 |
 | Axios | HTTP 客户端 |
 | Socket.io-client | WebSocket 客户端 |
 | ECharts | 统计图表 |

TODO.md

@@ -4,20 +4,16 @@ This file is the active backlog only. Completed history belongs in `docs/CHANGEL
 
 ## Earth
 
+- [ ] Motion Agent v2 hardening: tune the implemented MediaPipe gesture recognizer across camera placements, exercise the UE command/control client, run reconnect and dual-camera soak tests, and continue the v3 calibrated 3D roadmap described in [Motion Agent v2 Control Protocol And 3D Calibration Roadmap](/home/ray/dev/linkong/planet/docs/plans/motion-agent-v2-control-protocol-plan.md).
 - [ ] Earth AI command entry: merge natural-language and speech-triggered LLM commands into the existing Earth search panel as described in [Agent Runtime, Earth LLM Command, And Speech Entry Plan](/home/ray/dev/linkong/planet/docs/plans/agents-earth-command-runtime-plan.md).
 - [ ] Earth action executor: implement safe visualization actions for layer toggles, batch highlights, filters, focus, result panels, and clear-highlight behavior.
 - [ ] Earth entity matching: support stable entity ids and batch matching for Beidou satellites, mainland China compute centers, BGP, news, vessels, and cables.
-- [x] High-precision country boundary tile framework: implement the static vector tile builder, versioned seed output, frontend bbox tile loader, debounce, in-flight dedupe, and LRU cache described in [Earth High Precision Boundary Tiles Plan](/home/ray/dev/linkong/planet/docs/plans/earth-high-precision-boundary-tiles-plan.md).
-- [x] Add the `pmtiles-mvt` frontend tile provider contract, MVT decoder dependencies, static PMTiles Nginx handling, collector artifact registration, production readiness check, and user operation docs for Earth boundaries.
-- [x] Split Earth boundary ingestion into standard source collectors (`earth_admin0_boundaries`, `earth_coastline`, `earth_claim_lines`) plus the downstream `earth_boundary_tiles` PMTiles builder.
 - [ ] Replace debug GeoJSON boundary tiles with the real `earth-boundaries-china-pov-v1.pmtiles` production artifact after audited admin-0 / coastline / claim-line sources and the PMTiles toolchain are available.
 - [ ] Import authoritative China POV / coastline / claim-line source packages through the three standard Earth boundary source collectors, then rebuild a versioned PMTiles artifact so highest zoom `8-10` preserves trusted source geometry instead of seed data.
 - [ ] Earth boundary data: acquire or generate auditable China POV geometry for Zangnan, Aksai Chin, Taiwan/Penghu, Diaoyu Dao and affiliated islands, Chiwei Yu, South China Sea islands, Kosovo, Gaza, and the official dashed maritime claim line before implementing final visual changes.
 - [ ] Earth high-resolution basemap tiles: implement the viewport-loaded imagery layer described in [Earth High Resolution Basemap Tiles Plan](/home/ray/dev/linkong/planet/docs/plans/earth-high-resolution-basemap-tiles-plan.md), using high-precision coastline as the alignment reference instead of replacing the globe with one huge texture.
-- [ ] Presentation controller ownership: replace the singleton card fallback in [presentation-controller.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/presentation-controller.js) with a presentation/card token check before BGP/News migrate onto the shared controller, so connectors only attach to their owning card.
 - [ ] BGP frontend maintainability: split [bgp.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp.js) by responsibility into data loading, marker rendering, overlays, and animation once the current interaction behavior is stable.
 - [ ] Optional BGP marker experiment: evaluate HTML markers for BGP incident/collector points if WebGL marker density or fixed screen-size clickability becomes a real blocker.
-- [ ] Earth news cruise: connect Earth news to the generic cruise queue via a news adapter rather than coupling news-specific sequencing into `main.js`.
 
 ## Compute Centers And Location
 
@@ -48,19 +44,15 @@ This file is the active backlog only. Completed history belongs in `docs/CHANGEL
 - [ ] Compatibility schema: cover adapter type, base URL pattern, auth header, thinking/reasoning defaults, stream path, tool-call capability, multimodal capability, and provider-specific request patches.
 - [ ] BGP geography fallback: evaluate `inetnum` / `inet6num` whois as a finer fallback layer after `prefix_geography`, `OpenGeoFeed`, and RIR delegated data.
 
-## Platform
-
-- [ ] Earth preferences scope: keep current device-local Earth preferences in `localStorage`; only design backend user preferences if account-level synchronization becomes a real product requirement.
-- [ ] System logs: finish a usable Planet log viewing flow that covers backend, frontend, AI Provider, and collector/task logs, with filtering and tailing.
-- [ ] Console UI modernization: gradually replace Ant Design with Planet-owned components and a consistent Tabler Icons based icon system.
-- [ ] Earth live sync: design a unified realtime invalidation path for summary/BGP/satellite updates if polling and current WebSocket channels become insufficient.
-
 ## Archive
 
 Archived items stay here so old context is not lost. Completed items remain checked; obsolete, invalid, or superseded items stay unchecked and include the reason.
 
 ### Completed
 
+- [x] Implemented the high-precision country boundary tile framework from [Earth High Precision Boundary Tiles Plan](/home/ray/dev/linkong/planet/docs/plans/earth-high-precision-boundary-tiles-plan.md): static vector tile builder, versioned seed output, frontend bbox tile loader, debounce, in-flight dedupe, and LRU cache.
+- [x] Added the `pmtiles-mvt` frontend tile provider contract, MVT decoder dependencies, static PMTiles Nginx handling, collector artifact registration, production readiness check, and user operation docs for Earth boundaries.
+- [x] Split Earth boundary ingestion into standard source collectors (`earth_admin0_boundaries`, `earth_coastline`, `earth_claim_lines`) plus the downstream `earth_boundary_tiles` PMTiles builder.
 - [x] Refined BGP observer and anomaly `hover/click` feel.
 - [x] Added BGP anomaly relationship display with cables / regions.
 - [x] Added the Earth BGP activity layer so the map still feels alive when incident density is low.
@@ -75,6 +67,11 @@ Archived items stay here so old context is not lost. Completed items remain chec
 - [x] Added OpenGeoFeed as a high-quality prefix geography override source.
 - [x] Made RIR delegated data a prefix geography fallback rather than the primary source.
 - [x] Added route leak and path instability / flap detectors after the activity layer work.
+- [x] Console UI modernization. Admin is now the only console, legacy Ant Design / Admin Next code paths and dependencies have been removed, and current console UI uses Planet-owned components.
+- [x] Earth news cruise adapter. News cruise now uses `news-cruise-adapter.js` and is wired from `main.js` instead of keeping news-specific sequencing directly in the main Earth loop.
+- [x] Presentation controller ownership. `PresentationController` now guards async ownership through active request identity checks, and current callers pass per-request card targets so stale connector/card work cannot overwrite the active presentation.
+- [x] Earth live sync. Database writes now flow through `earth_data_change_events`, `earth_db_change_listener`, layer adapters, cache invalidation, and the `earth_updates` WebSocket channel; the Earth frontend debounces updates and refreshes BGP, cables, compute centers, satellites, vessels, news, and interactables by layer.
+- [x] System logs. Log sources now normalize into `LogEvent`, Admin supports snapshot filtering plus WebSocket tail/follow, task/detail views deep-link into prefiltered logs, and Admin runtime errors report through the `admin-client` log source.
 
 ### Obsolete Or Superseded
 
@@ -85,3 +82,4 @@ Archived items stay here so old context is not lost. Completed items remain chec
 - [ ] Earth surface material overlay for boundary calibration. Superseded by the high-precision boundary tile plan; future work must use source-faithful boundary/coastline data rather than overlay calibration against the coarse base map.
 - [ ] Hardcoded Earth news source extraction as a standalone task. Superseded by the broader Earth news source configuration and collector plans.
 - [ ] Country-level compute-center fallback placement as a standalone task. Superseded by the shared location pipeline and registry/manual-review backlog.
+- [ ] Earth preferences backend sync scope. Superseded by the current product decision to keep Earth preferences device-local in `localStorage` until account-level synchronization becomes a real requirement.

VERSION

@@ -1 +1 @@
-0.64.0
+0.74.0

aiprovider/Dockerfile

@@ -2,10 +2,14 @@
 
 ARG PYTHON_IMAGE=python:3.14-slim
 ARG UV_IMAGE=ghcr.io/astral-sh/uv:latest
+ARG AI_PROVIDER_BUILD_FINGERPRINT=unknown
 
 FROM ${UV_IMAGE} AS uv
 FROM ${PYTHON_IMAGE}
 
+ARG AI_PROVIDER_BUILD_FINGERPRINT
+LABEL planet.aiprovider.build-fingerprint="${AI_PROVIDER_BUILD_FINGERPRINT}"
+
 COPY --from=uv /uv /uvx /bin/
 
 WORKDIR /app
@@ -15,12 +19,15 @@ ENV PYTHONUNBUFFERED=1
 ENV UV_COMPILE_BYTECODE=1
 ENV UV_LINK_MODE=copy
 
+RUN mkdir -p /root/.config/uv
+
 RUN apt-get update && apt-get install -y --no-install-recommends \
     curl \
     && rm -rf /var/lib/apt/lists/*
 
 COPY pyproject.toml uv.lock /app/
 RUN --mount=type=cache,target=/root/.cache/uv \
+    --mount=type=secret,id=planet_uv_config,target=/root/.config/uv/uv.toml,required=false \
     uv sync --frozen --no-dev
 
 COPY aiprovider /app/aiprovider

backend/Dockerfile

@@ -1,3 +1,5 @@
+# syntax=docker/dockerfile:1.7
+
 ARG PYTHON_IMAGE=python:3.14-slim
 ARG UV_IMAGE=ghcr.io/astral-sh/uv:latest
 
@@ -14,12 +16,16 @@ ENV UV_COMPILE_BYTECODE=1
 ENV UV_LINK_MODE=copy
 ENV PYTHONPATH=/app/backend
 
+RUN mkdir -p /root/.config/uv
+
 RUN apt-get update && apt-get install -y --no-install-recommends \
     curl \
     && rm -rf /var/lib/apt/lists/*
 
 COPY pyproject.toml uv.lock /app/
-RUN uv sync --frozen --no-dev
+RUN --mount=type=cache,target=/root/.cache/uv \
+    --mount=type=secret,id=planet_uv_config,target=/root/.config/uv/uv.toml,required=false \
+    uv sync --frozen --no-dev
 
 COPY backend /app/backend
 COPY VERSION /app/VERSION

backend/app/api/main.py

@@ -19,6 +19,7 @@ from app.api.v1 import (
     vessels,
     bgp,
     news,
+    interactables,
     realtime_sources,
     system_control,
     tv,
@@ -53,4 +54,5 @@ api_router.include_router(vessels.router, prefix="/vessels", tags=["vessels"])
 api_router.include_router(bgp.router, prefix="/bgp", tags=["bgp"])
 api_router.include_router(tv.router, prefix="/tv", tags=["tv"])
 api_router.include_router(news.router, prefix="/news", tags=["news"])
+api_router.include_router(interactables.router, prefix="/interactables", tags=["interactables"])
 api_router.include_router(realtime_sources.router, prefix="/realtime-sources", tags=["realtime-sources"])

backend/app/api/v1/ai.py

@@ -3,6 +3,7 @@ from uuid import uuid4
 from fastapi import APIRouter, Depends, HTTPException, Request, Response
 from sqlalchemy.ext.asyncio import AsyncSession
 
+from app.core.logging import get_logger
 from app.core.security import get_current_user
 from app.db.session import get_db
 from app.models.user import User
@@ -47,8 +48,10 @@ from app.services.playground_chat_service import (
     stop_message,
 )
 from app.services.situational_alert_ai_brief import build_situational_alert_brief_request
+from app.services.business_logs import emit_business_log, exception_context
 
 router = APIRouter()
+logger = get_logger(__name__, service="api")
 
 
 @router.get("/provider/status", response_model=AIProviderStatusResponse)
@@ -122,6 +125,16 @@ async def create_playground_message(
     provider_client: AIProviderClient = Depends(get_ai_provider_client),
     db: AsyncSession = Depends(get_db),
 ):
+    await emit_business_log(
+        logger,
+        event="ai.playground.message.create",
+        message="Playground message creation requested",
+        category="ai",
+        service="api",
+        module=__name__,
+        user_id=current_user.id,
+        context={"session_key": payload.session_key, "preset": payload.selected_preset_key},
+    )
     return await create_turn(
         db,
         user_id=current_user.id,
@@ -136,6 +149,16 @@ async def stop_playground_message(
     current_user: User = Depends(get_current_user),
     db: AsyncSession = Depends(get_db),
 ):
+    await emit_business_log(
+        logger,
+        event="ai.playground.message.stop",
+        message="Playground message stop requested",
+        category="ai",
+        service="api",
+        module=__name__,
+        user_id=current_user.id,
+        context={"session_key": payload.session_key, "message_id": payload.message_id},
+    )
     return await stop_message(
         db,
         user_id=current_user.id,
@@ -150,6 +173,16 @@ async def resend_playground_message(
     provider_client: AIProviderClient = Depends(get_ai_provider_client),
     db: AsyncSession = Depends(get_db),
 ):
+    await emit_business_log(
+        logger,
+        event="ai.playground.message.resend",
+        message="Playground message resend requested",
+        category="ai",
+        service="api",
+        module=__name__,
+        user_id=current_user.id,
+        context={"session_key": payload.session_key, "user_message_id": payload.user_message_id},
+    )
     return await resend_turn(
         db,
         user_id=current_user.id,
@@ -214,16 +247,70 @@ async def analyze_bgp_brief(
         anomaly_limit=payload.anomaly_limit,
         collector_limit=payload.collector_limit,
     )
+    await emit_business_log(
+        logger,
+        event="ai.brief.bgp.facts_collected",
+        message="BGP brief facts collected",
+        category="ai",
+        service="api",
+        module=__name__,
+        request_id=request_id,
+        user_id=current_user.id,
+        context={
+            "incident_limit": payload.incident_limit,
+            "anomaly_limit": payload.anomaly_limit,
+            "collector_limit": payload.collector_limit,
+            "fact_count": len(facts or []),
+        },
+    )
     brief_request.preferred_model = payload.preferred_model
     brief_request.thinking = payload.thinking
 
-    analysis = await provider_client.analyze(brief_request, request_id=request_id)
-    return save_bgp_brief_record(
-        analysis,
+    await emit_business_log(
+        logger,
+        event="ai.brief.bgp.start",
+        message="BGP brief AI analysis started",
+        category="ai",
+        service="api",
+        module=__name__,
         request_id=request_id,
-        facts=facts,
-        context=context,
+        user_id=current_user.id,
+        context={"preferred_model": payload.preferred_model},
     )
+    try:
+        analysis = await provider_client.analyze(brief_request, request_id=request_id)
+        record = save_bgp_brief_record(
+            analysis,
+            request_id=request_id,
+            facts=facts,
+            context=context,
+        )
+        await emit_business_log(
+            logger,
+            event="ai.brief.bgp.completed",
+            message="BGP brief AI analysis saved",
+            category="ai",
+            service="api",
+            module=__name__,
+            request_id=request_id,
+            user_id=current_user.id,
+            context={"provider": analysis.provider, "model": analysis.model, "brief_id": record.id},
+        )
+        return record
+    except Exception as exc:
+        await emit_business_log(
+            logger,
+            event="ai.brief.bgp.failed",
+            message="BGP brief AI analysis failed",
+            category="ai",
+            level="error",
+            service="api",
+            module=__name__,
+            request_id=request_id,
+            user_id=current_user.id,
+            context=exception_context(exc, {"preferred_model": payload.preferred_model}),
+        )
+        raise
 
 
 @router.post("/alerts/brief", response_model=AlertBriefResponse)
@@ -242,17 +329,65 @@ async def analyze_alert_brief(
         db,
         alert_limit=payload.alert_limit,
     )
+    await emit_business_log(
+        logger,
+        event="ai.brief.alerts.facts_collected",
+        message="Alert brief facts collected",
+        category="ai",
+        service="api",
+        module=__name__,
+        request_id=request_id,
+        user_id=current_user.id,
+        context={"alert_limit": payload.alert_limit, "fact_count": len(facts or [])},
+    )
     brief_request.preferred_model = payload.preferred_model
     brief_request.thinking = payload.thinking
 
-    analysis = await provider_client.analyze(brief_request, request_id=request_id)
-    return AlertBriefResponse(
-        **analysis.model_dump(),
-        title=brief_request.title,
-        objective=brief_request.objective,
-        facts=facts,
-        context=context,
+    await emit_business_log(
+        logger,
+        event="ai.brief.alerts.start",
+        message="Alert brief AI analysis started",
+        category="ai",
+        service="api",
+        module=__name__,
+        request_id=request_id,
+        user_id=current_user.id,
+        context={"preferred_model": payload.preferred_model},
     )
+    try:
+        analysis = await provider_client.analyze(brief_request, request_id=request_id)
+        await emit_business_log(
+            logger,
+            event="ai.brief.alerts.completed",
+            message="Alert brief AI analysis completed",
+            category="ai",
+            service="api",
+            module=__name__,
+            request_id=request_id,
+            user_id=current_user.id,
+            context={"provider": analysis.provider, "model": analysis.model},
+        )
+        return AlertBriefResponse(
+            **analysis.model_dump(),
+            title=brief_request.title,
+            objective=brief_request.objective,
+            facts=facts,
+            context=context,
+        )
+    except Exception as exc:
+        await emit_business_log(
+            logger,
+            event="ai.brief.alerts.failed",
+            message="Alert brief AI analysis failed",
+            category="ai",
+            level="error",
+            service="api",
+            module=__name__,
+            request_id=request_id,
+            user_id=current_user.id,
+            context=exception_context(exc, {"preferred_model": payload.preferred_model}),
+        )
+        raise
 
 
 @router.post("/situational-alerts/brief", response_model=SituationalAlertBriefResponse)
@@ -268,14 +403,62 @@ async def analyze_situational_alert_brief(
     response.headers["X-Request-ID"] = request_id
 
     brief_request, facts, context = await build_situational_alert_brief_request(db)
+    await emit_business_log(
+        logger,
+        event="ai.brief.situational_alerts.facts_collected",
+        message="Situational alert brief facts collected",
+        category="ai",
+        service="api",
+        module=__name__,
+        request_id=request_id,
+        user_id=current_user.id,
+        context={"fact_count": len(facts or [])},
+    )
     brief_request.preferred_model = payload.preferred_model
     brief_request.thinking = payload.thinking
 
-    analysis = await provider_client.analyze(brief_request, request_id=request_id)
-    return SituationalAlertBriefResponse(
-        **analysis.model_dump(),
-        title=brief_request.title,
-        objective=brief_request.objective,
-        facts=facts,
-        context=context,
+    await emit_business_log(
+        logger,
+        event="ai.brief.situational_alerts.start",
+        message="Situational alert brief AI analysis started",
+        category="ai",
+        service="api",
+        module=__name__,
+        request_id=request_id,
+        user_id=current_user.id,
+        context={"preferred_model": payload.preferred_model},
     )
+    try:
+        analysis = await provider_client.analyze(brief_request, request_id=request_id)
+        await emit_business_log(
+            logger,
+            event="ai.brief.situational_alerts.completed",
+            message="Situational alert brief AI analysis completed",
+            category="ai",
+            service="api",
+            module=__name__,
+            request_id=request_id,
+            user_id=current_user.id,
+            context={"provider": analysis.provider, "model": analysis.model},
+        )
+        return SituationalAlertBriefResponse(
+            **analysis.model_dump(),
+            title=brief_request.title,
+            objective=brief_request.objective,
+            facts=facts,
+            context=context,
+        )
+    except Exception as exc:
+        await emit_business_log(
+            logger,
+            event="ai.brief.situational_alerts.failed",
+            message="Situational alert brief AI analysis failed",
+            category="ai",
+            level="error",
+            service="api",
+            module=__name__,
+            request_id=request_id,
+            user_id=current_user.id,
+            context=exception_context(exc, {"preferred_model": payload.preferred_model}),
+        )
+        raise

backend/app/api/v1/auth.py

@@ -4,6 +4,7 @@ from sqlalchemy.ext.asyncio import AsyncSession
 from sqlalchemy import text
 
 from app.core.config import settings
+from app.core.enums import OtpPurpose, UserRole
 from app.core.logging import get_logger
 from app.core.security import (
     create_access_token,
@@ -170,7 +171,7 @@ async def get_me(current_user: User = Depends(get_current_user)):
     }
 
 
-async def _send_code_or_raise(db: AsyncSession, email: str, code: str, purpose: str) -> None:
+async def _send_code_or_raise(db: AsyncSession, email: str, code: str, purpose: OtpPurpose) -> None:
     try:
         await send_verification_email(db, to=email, code=code, purpose=purpose)
     except EmailNotConfiguredError as exc:
@@ -207,7 +208,7 @@ async def register(payload: UserRegister, db: AsyncSession = Depends(get_db)):
         username=payload.username,
         email=payload.email,
         password_hash=get_password_hash(payload.password),
-        role="viewer",
+        role=UserRole.VIEWER.value,
         is_active=True,
         email_verified=False,
     )
@@ -215,13 +216,13 @@ async def register(payload: UserRegister, db: AsyncSession = Depends(get_db)):
     await db.commit()
 
     try:
-        code = otp.issue_code(payload.email, "register")
+        code = otp.issue_code(payload.email, OtpPurpose.REGISTER)
     except otp.OtpResendRateLimited as exc:
         raise HTTPException(
             status_code=status.HTTP_429_TOO_MANY_REQUESTS,
             detail={"code": exc.code, "retry_after_seconds": exc.retry_after_seconds},
         ) from exc
-    await _send_code_or_raise(db, payload.email, code, "register")
+    await _send_code_or_raise(db, payload.email, code, OtpPurpose.REGISTER)
     return {"status": "pending_verification", "email": payload.email}
 
 
@@ -266,7 +267,7 @@ async def resend_code(payload: ResendCodeRequest, db: AsyncSession = Depends(get
     if user is None:
         # Avoid email enumeration; pretend success.
         return {"status": "ok"}
-    if payload.purpose == "register" and user.email_verified:
+    if payload.purpose is OtpPurpose.REGISTER and user.email_verified:
         raise HTTPException(
             status_code=status.HTTP_409_CONFLICT,
             detail={"code": "ALREADY_VERIFIED"},
@@ -289,12 +290,17 @@ async def forgot_password(payload: ForgotPasswordRequest, db: AsyncSession = Dep
         # Don't leak whether an email is registered.
         return {"status": "ok"}
     try:
-        code = otp.issue_code(payload.email, "reset_password")
+        code = otp.issue_code(payload.email, OtpPurpose.RESET_PASSWORD)
     except otp.OtpResendRateLimited:
         # Silently accept; the user can retry after the cooldown.
         return {"status": "ok"}
     try:
-        await send_verification_email(db, to=payload.email, code=code, purpose="reset_password")
+        await send_verification_email(
+            db,
+            to=payload.email,
+            code=code,
+            purpose=OtpPurpose.RESET_PASSWORD,
+        )
     except EmailNotConfiguredError as exc:
         raise HTTPException(
             status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
@@ -304,7 +310,11 @@ async def forgot_password(payload: ForgotPasswordRequest, db: AsyncSession = Dep
         logger.warning_event(
             "SMTP send failed",
             event="auth.email.send_failed",
-            context={"email": payload.email, "purpose": "reset_password", "error": str(exc)},
+            context={
+                "email": payload.email,
+                "purpose": OtpPurpose.RESET_PASSWORD.value,
+                "error": str(exc),
+            },
         )
     return {"status": "ok"}
 
@@ -318,7 +328,7 @@ async def reset_password(payload: ResetPasswordRequest, db: AsyncSession = Depen
             detail={"code": "OTP_INVALID"},
         )
     try:
-        otp.verify_code(payload.email, "reset_password", payload.code)
+        otp.verify_code(payload.email, OtpPurpose.RESET_PASSWORD, payload.code)
     except otp.OtpExpired as exc:
         raise HTTPException(
             status_code=status.HTTP_410_GONE,

backend/app/api/v1/datasource_config.py

@@ -13,6 +13,7 @@ import httpx
 
 from app.core.target_schema_registry import get_target_schema, list_target_schemas
 from app.core.datasource_defaults import DEFAULT_DATASOURCES
+from app.core.enums import AuthType, MappingValidationStatus, UserRole
 from app.db.session import get_db
 from app.models.user import User
 from app.models.datasource_config import DataSourceConfig
@@ -34,7 +35,6 @@ from app.services.datasource_mapping import (
 )
 from app.services.custom_datasource_runtime import (
     CustomDatasourceRuntimeError,
-    fetch_rest_payload,
     get_custom_stream_status,
     run_mapped_rest_config,
     run_mapped_websocket_config,
@@ -42,8 +42,6 @@ from app.services.custom_datasource_runtime import (
     stop_custom_stream,
     test_websocket_config,
 )
-
-DATASOURCE_MAPPING_PROMPT_KEY = "datasource.mapping"
 from app.services.datasource_connectivity import (
     _resolve_aisstream_api_key,
     _resolve_spacetrack_credentials_with_override,
@@ -57,7 +55,8 @@ from app.services.persistent_logs import record_audit_log
 
 router = APIRouter()
 
-SECRET_REVEAL_ROLES = {"admin", "super_admin"}
+DATASOURCE_MAPPING_PROMPT_KEY = "datasource.mapping"
+SECRET_REVEAL_ROLES = {UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value}
 
 
 def _user_role_value(user: User) -> str:
@@ -124,7 +123,7 @@ class DataSourceConfigCreate(BaseModel):
     description: Optional[str] = None
     source_type: str = Field(..., description="rest, websocket, http, api, database")
     endpoint: str = Field(..., max_length=500)
-    auth_type: str = Field(default="none", description="none, bearer, api_key, basic")
+    auth_type: AuthType = Field(default=AuthType.NONE, description="none, bearer, api_key, basic")
     auth_config: dict = Field(default={})
     headers: dict = Field(default={})
     config: dict = Field(default={"timeout": 30, "retry": 3})
@@ -135,7 +134,7 @@ class DataSourceConfigUpdate(BaseModel):
     description: Optional[str] = None
     source_type: Optional[str] = None
     endpoint: Optional[str] = Field(None, max_length=500)
-    auth_type: Optional[str] = None
+    auth_type: Optional[AuthType] = None
     auth_config: Optional[dict] = None
     headers: Optional[dict] = None
     config: Optional[dict] = None
@@ -210,7 +209,7 @@ class MappingTemplateCreate(BaseModel):
     mapping_json: dict
     sample_payload: Any | None = None
     sample_payload_hash: Optional[str] = None
-    validation_status: str = Field(default="draft", pattern="^(draft|valid|invalid)$")
+    validation_status: MappingValidationStatus = MappingValidationStatus.DRAFT
     is_active: bool = False
 
 
@@ -219,7 +218,7 @@ class MappingTemplateUpdate(BaseModel):
     mapping_json: Optional[dict] = None
     sample_payload: Any | None = None
     sample_payload_hash: Optional[str] = None
-    validation_status: Optional[str] = Field(default=None, pattern="^(draft|valid|invalid)$")
+    validation_status: Optional[MappingValidationStatus] = None
     is_active: Optional[bool] = None
 
 
@@ -918,6 +917,7 @@ async def get_datasource_target_schemas(
 @router.post("/mappings/propose")
 async def propose_datasource_mapping(
     payload: MappingProposeRequest,
+    db: AsyncSession = Depends(get_db),
     current_user: User = Depends(get_current_user),
     ai_client: AIProviderClient = Depends(get_ai_provider_client),
 ):

backend/app/api/v1/datasources.py

@@ -1,4 +1,3 @@
-import asyncio
 from datetime import datetime, timedelta, timezone
 from typing import Optional
 
@@ -7,7 +6,8 @@ from pydantic import BaseModel, Field
 from sqlalchemy import func, or_, select, text
 from sqlalchemy.ext.asyncio import AsyncSession
 
-from app.core.cache import cache
+from app.core.logging import get_logger
+from app.core.enums import JobStatus, SnapshotStatus
 from app.core.time import to_iso8601_utc
 from app.core.security import get_current_user
 from app.core.data_sources import get_data_sources_config
@@ -20,18 +20,27 @@ from app.models.datasource_config import DataSourceConfig
 from app.models.task import CollectionTask
 from app.models.user import User
 from app.models.vessel import AISRawObservation
-from app.services.vessel_ais_aggregation import VESSEL_AIS_SCHEMA
 from app.services.scheduler import (
-    cancel_running_collector_now,
-    get_latest_task_id_for_datasource,
-    run_collector_now,
     sync_datasource_job,
 )
-from app.services.earth_layer_cache import invalidate_earth_layer_cache_for_source
+from app.services.data_jobs import (
+    JOB_STATUS_CANCELLING,
+    JOB_STATUS_QUEUED,
+    JOB_STATUS_RUNNING,
+    JOB_TYPE_CLEAR_CACHE,
+    JOB_TYPE_CLEAR_DATA,
+    JOB_TYPE_COLLECT,
+    enqueue_datasource_job,
+    get_active_datasource_job,
+    request_cancel_datasource_task,
+)
+from app.services.business_logs import emit_business_log
 
 router = APIRouter()
+logger = get_logger(__name__, service="api")
 STALE_RUNNING_TASK_TIMEOUT_MINUTES = 90
 
+
 PRODUCT_SOURCE_KEYWORDS: tuple[tuple[str, tuple[str, ...]], ...] = (
     ("vessels", ("vessel", "ais")),
     ("cables", ("cable", "landing", "telegeography", "arcgis", "fao")),
@@ -115,7 +124,7 @@ async def _load_latest_running_tasks(
             _task_rank_column(CollectionTask.started_at),
         )
         .where(CollectionTask.datasource_id.in_(datasource_ids))
-        .where(CollectionTask.status == "running")
+        .where(CollectionTask.status.in_((JOB_STATUS_QUEUED, JOB_STATUS_RUNNING, JOB_STATUS_CANCELLING)))
         .subquery()
     )
     result = await db.execute(
@@ -126,32 +135,6 @@ async def _load_latest_running_tasks(
     return {task.datasource_id: task for task in result.scalars().all()}
 
 
-async def _load_latest_task_ids(
-    db: AsyncSession,
-    datasource_ids: list[int],
-) -> dict[int, int]:
-    if not datasource_ids:
-        return {}
-
-    ranked_tasks = (
-        select(
-            CollectionTask.id.label("task_id"),
-            CollectionTask.datasource_id.label("datasource_id"),
-            func.row_number().over(
-                partition_by=CollectionTask.datasource_id,
-                order_by=CollectionTask.id.desc(),
-            ).label("row_num"),
-        )
-        .where(CollectionTask.datasource_id.in_(datasource_ids))
-        .subquery()
-    )
-    result = await db.execute(
-        select(ranked_tasks.c.datasource_id, ranked_tasks.c.task_id)
-        .where(ranked_tasks.c.row_num == 1)
-    )
-    return {datasource_id: task_id for datasource_id, task_id in result.all()}
-
-
 async def _load_latest_tasks(
     db: AsyncSession,
     datasource_ids: list[int],
@@ -182,6 +165,8 @@ async def _load_latest_tasks(
 async def _load_collected_record_counts(
     db: AsyncSession,
     sources: list[str],
+    *,
+    exact_vessel_counts: bool = False,
 ) -> dict[str, int]:
     if not sources:
         return {}
@@ -202,14 +187,46 @@ async def _load_collected_record_counts(
         or "ais" in source
     ]
     if vessel_sources:
-        raw_result = await db.execute(
-            select(AISRawObservation.source, func.count(AISRawObservation.id))
-            .where(AISRawObservation.target_schema == VESSEL_AIS_SCHEMA)
-            .where(AISRawObservation.source.in_(vessel_sources))
-            .group_by(AISRawObservation.source)
+        if exact_vessel_counts:
+            exact_result = await db.execute(
+                select(AISRawObservation.source, func.count(AISRawObservation.id))
+                .where(AISRawObservation.source.in_(vessel_sources))
+                .group_by(AISRawObservation.source)
+            )
+            for source, count in exact_result.all():
+                counts[source] = max(counts.get(source, 0), int(count or 0))
+            return counts
+
+        # AIS raw observations can be tens of millions of rows. Use planner
+        # statistics for the datasource list instead of blocking page load on
+        # source-level count(*) scans.
+        stats_result = await db.execute(
+            text(
+                """
+                SELECT
+                    COALESCE(pg_class.reltuples, 0)::bigint AS total_rows,
+                    pg_stats.most_common_vals::text AS source_values,
+                    pg_stats.most_common_freqs::text AS source_freqs
+                FROM pg_class
+                LEFT JOIN pg_stats
+                  ON pg_stats.schemaname = 'public'
+                 AND pg_stats.tablename = 'ais_raw_observations'
+                 AND pg_stats.attname = 'source'
+                WHERE pg_class.relname = 'ais_raw_observations'
+                LIMIT 1
+                """
+            )
         )
-        for source, count in raw_result.all():
-            counts[source] = max(counts.get(source, 0), int(count or 0))
+        stats = stats_result.mappings().first()
+        if stats:
+            total_rows = int(stats["total_rows"] or 0)
+            values = str(stats["source_values"] or "").strip("{}")
+            freqs = str(stats["source_freqs"] or "").strip("{}")
+            source_values = [value.strip('"') for value in values.split(",") if value]
+            source_freqs = [float(value) for value in freqs.split(",") if value]
+            for source, freq in zip(source_values, source_freqs):
+                if source in vessel_sources:
+                    counts[source] = max(counts.get(source, 0), int(round(total_rows * freq)))
 
     return counts
 
@@ -303,8 +320,11 @@ def serialize_datasource_row(
         "last_run": to_iso8601_utc(last_run_at),
         "last_run_at": to_iso8601_utc(last_run_at),
         "last_status": last_status,
-        "is_running": running_task is not None,
+        "is_running": running_task is not None and running_task.task_type == JOB_TYPE_COLLECT,
+        "is_task_active": running_task is not None,
+        "task_status": running_task.status if running_task else None,
         "task_id": display_task.id if display_task else None,
+        "task_type": display_task.task_type if display_task else None,
         "progress": display_task.progress if display_task else None,
         "phase": display_task.phase if display_task else None,
         "phase_progress": display_task.phase_progress if display_task else None,
@@ -400,9 +420,25 @@ async def _trigger_datasource_batch(
     datasources: list[DataSource],
     *,
     force: bool,
+    actor_id: int | None = None,
+    trigger_kind: str = "batch",
 ) -> dict:
+    await emit_business_log(
+        logger,
+        event=f"collector.trigger.{trigger_kind}.start",
+        message="Datasource batch trigger started",
+        category="collector",
+        service="api",
+        module=__name__,
+        user_id=actor_id,
+        context={
+            "trigger_kind": trigger_kind,
+            "force": force,
+            "requested_count": len(datasources),
+        },
+    )
     if not datasources:
-        return {
+        result = {
             "status": "noop",
             "message": "No matching data sources to trigger",
             "force": force,
@@ -410,8 +446,18 @@ async def _trigger_datasource_batch(
             "skipped": [],
             "failed": [],
         }
+        await emit_business_log(
+            logger,
+            event=f"collector.trigger.{trigger_kind}.completed",
+            message="Datasource batch trigger completed with no matching sources",
+            category="collector",
+            service="api",
+            module=__name__,
+            user_id=actor_id,
+            context={"trigger_kind": trigger_kind, "force": force, "status": "noop", "triggered_count": 0},
+        )
+        return result
 
-    previous_task_ids: dict[int, Optional[int]] = {}
     triggered_sources: list[dict] = []
     skipped_sources: list[dict] = []
     failed_sources: list[dict] = []
@@ -446,9 +492,11 @@ async def _trigger_datasource_batch(
                     }
                 )
                 continue
-            cancelled = await cancel_running_collector_now(datasource.source)
-            if not cancelled:
-                await rollback_orphaned_running_task(db, datasource, running_task)
+            await request_cancel_datasource_task(
+                db,
+                running_task,
+                reason="superseded_by_forced_collection",
+            )
 
         if not force and not is_due_for_collection(datasource, now):
             skipped_sources.append(
@@ -465,57 +513,51 @@ async def _trigger_datasource_batch(
             )
             continue
 
-        previous_task_ids[datasource.id] = None
-        success = run_collector_now(datasource.source)
-        if not success:
-            failed_sources.append(
-                {
-                    "id": datasource.id,
-                    "source": datasource.source,
-                    "name": datasource.name,
-                    "reason": "trigger_failed",
-                }
-            )
-            continue
+        task = await enqueue_datasource_job(
+            db,
+            datasource,
+            JOB_TYPE_COLLECT,
+            payload={"force": force, "trigger": "batch"},
+        )
 
         triggered_sources.append(
             {
                 "id": datasource.id,
                 "source": datasource.source,
                 "name": datasource.name,
-                "task_id": None,
+                "task_id": task.id,
             }
         )
 
-    latest_task_ids = await _load_latest_task_ids(
-        db,
-        [datasource.id for datasource in datasources],
-    )
-    for datasource_id in previous_task_ids:
-        previous_task_ids[datasource_id] = latest_task_ids.get(datasource_id)
-
-    for _ in range(20):
-        await asyncio.sleep(0.1)
-        pending = [item for item in triggered_sources if item["task_id"] is None]
-        if not pending:
-            break
-        latest_task_ids = await _load_latest_task_ids(
-            db,
-            [item["id"] for item in pending],
-        )
-        for item in pending:
-            task_id = latest_task_ids.get(item["id"])
-            if task_id is not None and task_id != previous_task_ids.get(item["id"]):
-                item["task_id"] = task_id
-
-    return {
-        "status": "triggered" if triggered_sources else "partial",
-        "message": f"Triggered {len(triggered_sources)} data sources",
+    result = {
+        "status": "queued" if triggered_sources else "partial",
+        "message": f"Queued {len(triggered_sources)} data source jobs",
         "force": force,
         "triggered": triggered_sources,
         "skipped": skipped_sources,
         "failed": failed_sources,
     }
+    await emit_business_log(
+        logger,
+        event=f"collector.trigger.{trigger_kind}.completed",
+        message="Datasource batch trigger completed",
+        category="collector",
+        service="api",
+        module=__name__,
+        user_id=actor_id,
+        context={
+            "trigger_kind": trigger_kind,
+            "force": force,
+            "status": result["status"],
+            "requested_count": len(datasources),
+            "triggered_count": len(triggered_sources),
+            "skipped_count": len(skipped_sources),
+            "failed_count": len(failed_sources),
+            "triggered_sources": [item["source"] for item in triggered_sources],
+            "skipped_reasons": [item["reason"] for item in skipped_sources],
+        },
+    )
+    return result
 
 
 async def get_datasource_record(db: AsyncSession, source_id: str) -> Optional[DataSource]:
@@ -540,7 +582,7 @@ async def get_running_task(db: AsyncSession, datasource_id: int) -> Optional[Col
     result = await db.execute(
         select(CollectionTask)
         .where(CollectionTask.datasource_id == datasource_id)
-        .where(CollectionTask.status == "running")
+        .where(CollectionTask.status == JobStatus.RUNNING.value)
         .order_by(CollectionTask.started_at.desc())
         .limit(1)
     )
@@ -568,8 +610,8 @@ async def get_running_task(db: AsyncSession, datasource_id: int) -> Optional[Col
             f"Marked failed automatically after stale running timeout "
             f"({STALE_RUNNING_TASK_TIMEOUT_MINUTES}m)"
         )
-        task.status = "failed"
-        task.phase = "failed"
+        task.status = JobStatus.FAILED.value
+        task.phase = JobStatus.FAILED.value
         task.completed_at = now
         task.error_message = f"{existing_error}\n{stale_reason}".strip() if existing_error else stale_reason
         await db.commit()
@@ -606,7 +648,7 @@ async def rollback_orphaned_running_task(
     )
 
     if snapshot is not None:
-        snapshot.status = "cancelled"
+        snapshot.status = SnapshotStatus.CANCELLED.value
         snapshot.is_current = False
         snapshot.completed_at = datetime.now(timezone.utc)
         summary = dict(snapshot.summary or {})
@@ -629,13 +671,13 @@ async def rollback_orphaned_running_task(
                 {"snapshot_id": snapshot.parent_snapshot_id},
             )
 
-    running_task.status = "cancelled"
-    running_task.phase = "cancelled"
+    running_task.status = JobStatus.CANCELLED.value
+    running_task.phase = JobStatus.CANCELLED.value
     running_task.completed_at = datetime.now(timezone.utc)
     existing_error = (running_task.error_message or "").strip()
     cancel_reason = "Cancelled after backend restart because the running task handle was lost; incomplete writes rolled back"
     running_task.error_message = f"{existing_error}\n{cancel_reason}".strip() if existing_error else cancel_reason
-    datasource.last_status = "cancelled"
+    datasource.last_status = JobStatus.CANCELLED.value
     datasource.last_run_at = datetime.now(timezone.utc)
     await db.commit()
 
@@ -670,7 +712,7 @@ async def fail_and_rollback_stale_running_task(
     )
 
     if snapshot is not None:
-        snapshot.status = "failed"
+        snapshot.status = SnapshotStatus.FAILED.value
         snapshot.is_current = False
         snapshot.completed_at = datetime.now(timezone.utc)
         summary = dict(snapshot.summary or {})
@@ -698,11 +740,11 @@ async def fail_and_rollback_stale_running_task(
         f"Marked failed automatically after stale running timeout "
         f"({STALE_RUNNING_TASK_TIMEOUT_MINUTES}m); incomplete writes rolled back"
     )
-    running_task.status = "failed"
-    running_task.phase = "failed"
+    running_task.status = JobStatus.FAILED.value
+    running_task.phase = JobStatus.FAILED.value
     running_task.completed_at = datetime.now(timezone.utc)
     running_task.error_message = f"{existing_error}\n{stale_reason}".strip() if existing_error else stale_reason
-    datasource.last_status = "failed"
+    datasource.last_status = JobStatus.FAILED.value
     datasource.last_run_at = datetime.now(timezone.utc)
     await db.commit()
 
@@ -780,7 +822,13 @@ async def trigger_all_datasources(
         .order_by(DataSource.module, DataSource.id)
     )
     datasources = result.scalars().all()
-    return await _trigger_datasource_batch(db, datasources, force=force)
+    return await _trigger_datasource_batch(
+        db,
+        datasources,
+        force=force,
+        actor_id=current_user.id,
+        trigger_kind="all",
+    )
 
 
 @router.post("/trigger-batch")
@@ -816,7 +864,13 @@ async def trigger_datasource_batch(
         collected=None if payload.source_ids else payload.collected,
         credential_status=None if payload.source_ids else payload.credential_status,
     )
-    return await _trigger_datasource_batch(db, datasources, force=payload.force)
+    return await _trigger_datasource_batch(
+        db,
+        datasources,
+        force=payload.force,
+        actor_id=current_user.id,
+        trigger_kind="batch",
+    )
 
 
 @router.get("/snapshots")
@@ -909,7 +963,7 @@ async def get_datasource_row(
         [datasource],
         include_endpoint=include_endpoint,
     )
-    record_counts = await _load_collected_record_counts(db, [datasource.source])
+    record_counts = await _load_collected_record_counts(db, [datasource.source], exact_vessel_counts=True)
     return {
         "data": serialize_datasource_row(
             datasource,
@@ -992,8 +1046,24 @@ async def trigger_datasource(
     if not datasource.is_active:
         raise HTTPException(status_code=400, detail="Data source is disabled")
 
-    running_task = await get_running_task(db, datasource.id)
+    running_task = await get_active_datasource_job(db, datasource.id, task_types=(JOB_TYPE_COLLECT,))
     if running_task is not None and not force:
+        await emit_business_log(
+            logger,
+            event="collector.trigger.single.skipped_already_running",
+            message="Datasource trigger skipped because a task is already running",
+            category="collector",
+            level="warning",
+            service="api",
+            module=__name__,
+            user_id=current_user.id,
+            context={
+                "collector_name": datasource.source,
+                "datasource_id": datasource.id,
+                "task_id": running_task.id,
+                "status": "skipped",
+            },
+        )
         raise HTTPException(
             status_code=409,
             detail={
@@ -1013,31 +1083,42 @@ async def trigger_datasource(
         )
 
     if running_task is not None and force:
-        cancelled = await cancel_running_collector_now(datasource.source)
-        if not cancelled:
-            await rollback_orphaned_running_task(db, datasource, running_task)
+        await request_cancel_datasource_task(
+            db,
+            running_task,
+            reason="superseded_by_forced_collection",
+        )
 
-    previous_task_id = await get_latest_task_id_for_datasource(datasource.id)
-    success = run_collector_now(datasource.source)
-    if not success:
-        raise HTTPException(status_code=500, detail=f"Failed to trigger collector '{datasource.source}'")
-
-    task_id = None
-    for _ in range(20):
-        await asyncio.sleep(0.1)
-        task_id = await get_latest_task_id_for_datasource(datasource.id)
-        if task_id is not None and task_id != previous_task_id:
-            break
-    if task_id == previous_task_id:
-        task_id = None
+    task = await enqueue_datasource_job(
+        db,
+        datasource,
+        JOB_TYPE_COLLECT,
+        payload={"force": force, "trigger": "single"},
+    )
+    await emit_business_log(
+        logger,
+        event="collector.trigger.single.completed",
+        message="Datasource trigger queued",
+        category="collector",
+        service="api",
+        module=__name__,
+        user_id=current_user.id,
+        context={
+            "collector_name": datasource.source,
+            "datasource_id": datasource.id,
+            "task_id": task.id,
+            "force": force,
+            "status": "queued",
+        },
+    )
 
     return {
-        "status": "triggered",
+        "status": "queued",
         "source_id": datasource.id,
-        "task_id": task_id,
+        "task_id": task.id,
         "collector_name": datasource.source,
         "force": force,
-        "message": f"Collector '{datasource.source}' has been triggered",
+        "message": f"Collector '{datasource.source}' has been queued",
     }
 
 
@@ -1051,22 +1132,30 @@ async def clear_datasource_data(
     if not datasource:
         raise HTTPException(status_code=404, detail="Data source not found")
 
-    result = await db.execute(
-        select(func.count(CollectedData.id)).where(CollectedData.source == datasource.source)
+    active_task = await get_active_datasource_job(db, datasource.id)
+    if active_task is not None:
+        raise HTTPException(
+            status_code=409,
+            detail={
+                "reason": "datasource_job_in_progress",
+                "message": "当前数据源已有任务在执行，请等待完成或先取消任务。",
+                "task_id": active_task.id,
+                "task_type": active_task.task_type,
+                "status": active_task.status,
+            },
+        )
+    task = await enqueue_datasource_job(
+        db,
+        datasource,
+        JOB_TYPE_CLEAR_DATA,
+        payload={"source": datasource.source},
     )
-    count = result.scalar() or 0
-
-    if count == 0:
-        return {"status": "success", "message": "No data to clear", "deleted_count": 0}
-
-    delete_query = CollectedData.__table__.delete().where(CollectedData.source == datasource.source)
-    await db.execute(delete_query)
-    await db.commit()
 
     return {
-        "status": "success",
-        "message": f"Cleared {count} records for data source '{datasource.name}'",
-        "deleted_count": count,
+        "status": "queued",
+        "message": f"Queued data clearing for data source '{datasource.name}'",
+        "task_id": task.id,
+        "deleted_count": None,
     }
 
 
@@ -1080,16 +1169,44 @@ async def clear_datasource_cache(
     if not datasource:
         raise HTTPException(status_code=404, detail="Data source not found")
 
-    earth_deleted_count = invalidate_earth_layer_cache_for_source(datasource.source)
-    dashboard_deleted_count = int(cache.delete("dashboard:stats")) + int(cache.delete("dashboard:summary"))
-    deleted_count = earth_deleted_count + dashboard_deleted_count
+    task = await enqueue_datasource_job(
+        db,
+        datasource,
+        JOB_TYPE_CLEAR_CACHE,
+        payload={"source": datasource.source},
+        dedupe_key=f"clear_cache:{datasource.source}",
+    )
 
     return {
-        "status": "success",
-        "message": f"Cleared {deleted_count} cache keys for data source '{datasource.name}'",
-        "deleted_count": deleted_count,
-        "earth_layer_deleted_count": earth_deleted_count,
-        "dashboard_deleted_count": dashboard_deleted_count,
+        "status": "queued",
+        "message": f"Queued cache clearing for data source '{datasource.name}'",
+        "task_id": task.id,
+        "deleted_count": None,
+    }
+
+
+@router.post("/{source_id}/tasks/{task_id}/cancel")
+async def cancel_datasource_task(
+    source_id: str,
+    task_id: int,
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    datasource = await get_datasource_record(db, source_id)
+    if not datasource:
+        raise HTTPException(status_code=404, detail="Data source not found")
+
+    task = await db.get(CollectionTask, task_id)
+    if not task or task.datasource_id != datasource.id:
+        raise HTTPException(status_code=404, detail="Task not found")
+
+    task = await request_cancel_datasource_task(db, task)
+    return {
+        "status": "cancelled" if task.completed_at else "cancelling",
+        "task_id": task.id,
+        "task_type": task.task_type,
+        "phase": task.phase,
+        "requested_cancel_at": to_iso8601_utc(task.requested_cancel_at),
     }
 
 
@@ -1109,7 +1226,7 @@ async def get_task_status(
         if not task or task.datasource_id != datasource.id:
             raise HTTPException(status_code=404, detail="Task not found")
     else:
-        task = await get_running_task(db, datasource.id)
+        task = await get_active_datasource_job(db, datasource.id)
         if task is None:
             result = await db.execute(
                 select(CollectionTask)
@@ -1134,8 +1251,12 @@ async def get_task_status(
         }
 
     return {
-        "is_running": task.status == "running",
+        "is_running": task.status in {JOB_STATUS_QUEUED, JOB_STATUS_RUNNING, JOB_STATUS_CANCELLING}
+        and task.task_type == JOB_TYPE_COLLECT,
+        "is_task_active": task.status in {JOB_STATUS_QUEUED, JOB_STATUS_RUNNING, JOB_STATUS_CANCELLING},
         "task_id": task.id,
+        "task_type": task.task_type,
+        "task_status": task.status,
         "progress": task.progress,
         "phase": task.phase,
         "phase_progress": task.phase_progress,
@@ -1146,5 +1267,6 @@ async def get_task_status(
         "records_processed": task.records_processed,
         "total_records": task.total_records,
         "status": task.status,
+        "requested_cancel_at": to_iso8601_utc(task.requested_cancel_at),
         "error_message": task.error_message,
     }

backend/app/api/v1/earth.py

@@ -6,12 +6,13 @@ from pathlib import Path
 from typing import Any
 from uuid import uuid4
 
-from fastapi import APIRouter, Depends, File, HTTPException, Request, UploadFile, status
+from fastapi import APIRouter, Depends, File, Form, HTTPException, Query, Request, UploadFile, status
 from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
 from pydantic import BaseModel, Field
 from sqlalchemy import delete, func, select, text
 from sqlalchemy.ext.asyncio import AsyncSession
 
+from app.core.config import settings as app_settings
 from app.core.security import decode_token, get_current_user, redis_client
 from app.db.session import get_db
 from app.models.collected_data import CollectedData
@@ -20,6 +21,26 @@ from app.models.datasource_config import DataSourceConfig
 from app.models.system_setting import SystemSetting
 from app.models.user import User
 from app.services.tv_streams import get_tv_settings_payload
+from app.services.earth_news import (
+    get_earth_news_sources_payload,
+    reset_earth_news_sources_payload,
+    save_earth_news_sources_payload,
+    test_news_source_config,
+)
+from app.services.earth_news_manual import (
+    broadcast_manual_news_changed,
+    create_manual_news_group,
+    delete_manual_news_item,
+    get_news_record_or_404,
+    import_manual_news_items,
+    list_news_groups,
+    list_news_records,
+    parse_manual_news_import_upload,
+    rename_manual_news_group,
+    reprocess_manual_news_item,
+    serialize_news_record,
+    upsert_manual_news_item,
+)
 from app.services.earth_boundaries import (
     EarthBoundaryBuildError,
     get_boundary_build_status,
@@ -36,9 +57,16 @@ EARTH_BRAND_ASSET_DIR = REPO_ROOT / "data" / "earth-brand"
 EARTH_BRAND_ASSET_URL_PREFIX = "/earth-brand-assets"
 EARTH_BRAND_CATEGORY = "earth_brand"
 EARTH_ABOUT_CATEGORY = "earth_about"
+SYSTEM_SETTINGS_CATEGORY = "system"
 MAX_EARTH_BRAND_ASSET_BYTES = 3 * 1024 * 1024
 ALLOWED_EARTH_BRAND_EXTENSIONS = {".png", ".jpg", ".jpeg", ".webp", ".svg"}
 
+
+def _app_version_label() -> str:
+    version = str(app_settings.VERSION or "").strip() or "0.0.0"
+    return version if version.startswith("v") else f"v{version}"
+
+
 DEFAULT_EARTH_BRAND = {
     "logo_src": "/earth/assets/brand/earth-logo.png",
     "title_src": "/earth/assets/brand/title-zh.png",
@@ -53,14 +81,15 @@ DEFAULT_EARTH_ABOUT = {
     "logo_src": "/earth/assets/brand/lim-logo.png",
     "kicker": "About",
     "title": "智能星球计划",
-    "version": "v0.64.0",
+    "version": _app_version_label(),
     "description": "面向临空场景下的智能媒体研究、全球态势感知与多源开放数据巡航，提供可视化观测、事件聚合与交互式探索能力。",
     "meta": [
         {"label": "出品方", "value": "浙江大学临空智能媒体研究院"},
-        {"label": "策划人", "value": "黄柳青"},
+        {"label": "策划人", "value": "方兴东、黄柳青"},
         {"label": "产品兼开发者", "value": "钱坤、张鸽、齐鹏"},
     ],
 }
+EARTH_ABOUT_LEGACY_PLANNER_VALUE = "黄柳青"
 
 
 class EarthBoundaryConfigPayload(BaseModel):
@@ -91,6 +120,39 @@ class EarthAboutPayload(BaseModel):
     meta: list[EarthAboutMetaItem] = Field(default_factory=list)
 
 
+class EarthNewsSourcesPayload(BaseModel):
+    cache_version: int | None = None
+    source_tags: list[dict[str, Any]] = Field(default_factory=list)
+    categories: list[dict[str, Any]] = Field(default_factory=list)
+    item_tag_rules: list[dict[str, Any]] = Field(default_factory=list)
+    sources: list[dict[str, Any]] = Field(default_factory=list)
+    health: dict[str, Any] = Field(default_factory=dict)
+
+
+class EarthNewsSourceTestPayload(BaseModel):
+    source: dict[str, Any] = Field(default_factory=dict)
+
+
+class EarthNewsManualItemPayload(BaseModel):
+    title: str = Field(default="", max_length=500)
+    summary: str = Field(default="", max_length=1200)
+    content: str = Field(default="", max_length=12000)
+    url: str = Field(default="", max_length=2000)
+    source: str = Field(default="", max_length=255)
+    region: str = Field(default="global", max_length=80)
+    published_at: str | None = None
+    category: str = Field(default="other", max_length=80)
+    tags: list[str] = Field(default_factory=list)
+    location: dict[str, Any] | None = None
+    homepage_url: str = Field(default="", max_length=2000)
+    content_language: str = Field(default="", max_length=32)
+    group_id: str | None = Field(default=None, max_length=120)
+
+
+class EarthNewsManualGroupPayload(BaseModel):
+    name: str = Field(default="", max_length=120)
+
+
 def _normalize_earth_brand_payload(payload: dict[str, Any] | None) -> dict[str, str]:
     merged = DEFAULT_EARTH_BRAND.copy()
     if payload:
@@ -116,11 +178,12 @@ def _normalize_earth_about_payload(payload: dict[str, Any] | None) -> dict[str,
     }
     raw_meta = DEFAULT_EARTH_ABOUT["meta"]
     if payload:
-        for key in ("logo_src", "kicker", "title", "version", "description"):
+        for key in ("logo_src", "kicker", "title", "description"):
             value = payload.get(key)
             if value is not None:
                 merged[key] = str(value).strip()
         raw_meta = payload.get("meta") if isinstance(payload.get("meta"), list) else raw_meta
+    merged["version"] = _app_version_label()
 
     for key, default_value in DEFAULT_EARTH_ABOUT.items():
         if key == "meta":
@@ -134,6 +197,8 @@ def _normalize_earth_about_payload(payload: dict[str, Any] | None) -> dict[str,
             continue
         label = str(item.get("label") or "").strip()
         value = str(item.get("value") or "").strip()
+        if label == "策划人" and value == EARTH_ABOUT_LEGACY_PLANNER_VALUE:
+            value = "方兴东、黄柳青"
         if label or value:
             normalized_meta.append({"label": label, "value": value})
     if not normalized_meta:
@@ -142,6 +207,10 @@ def _normalize_earth_about_payload(payload: dict[str, Any] | None) -> dict[str,
     return merged
 
 
+def _is_demo_mode_enabled(payload: Any) -> bool:
+    return bool(payload.get("demo_mode")) if isinstance(payload, dict) else False
+
+
 async def _get_earth_brand_record(db: AsyncSession) -> SystemSetting | None:
     result = await db.execute(
         select(SystemSetting).where(SystemSetting.category == EARTH_BRAND_CATEGORY)
@@ -308,6 +377,194 @@ async def reset_earth_about(
     return {"status": "reset", "about": _normalize_earth_about_payload(None), "is_default": True}
 
 
+@router.get("/news-sources")
+async def get_earth_news_sources(db: AsyncSession = Depends(get_db)):
+    return await get_earth_news_sources_payload(db)
+
+
+@router.put("/news-sources")
+async def update_earth_news_sources(
+    payload: EarthNewsSourcesPayload,
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    return await save_earth_news_sources_payload(db, payload.model_dump())
+
+
+@router.delete("/news-sources")
+@router.post("/news-sources/reset")
+async def reset_earth_news_sources(
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    return await reset_earth_news_sources_payload(db)
+
+
+@router.post("/news-sources/test")
+async def test_earth_news_source(
+    payload: EarthNewsSourceTestPayload,
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    return await test_news_source_config(payload.source, db=db)
+
+
+@router.get("/news-groups")
+async def list_earth_news_groups_admin(
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    return await list_news_groups(db)
+
+
+@router.post("/news-groups")
+async def create_earth_news_group_admin(
+    payload: EarthNewsManualGroupPayload,
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    try:
+        group = await create_manual_news_group(db, payload.name)
+    except ValueError as exc:
+        raise HTTPException(status_code=422, detail=str(exc)) from exc
+    await db.commit()
+    return {"status": "ok", "group": group}
+
+
+@router.put("/news-groups/{group_id:path}")
+async def rename_earth_news_group_admin(
+    group_id: str,
+    payload: EarthNewsManualGroupPayload,
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    try:
+        group = await rename_manual_news_group(db, group_id, payload.name)
+    except ValueError as exc:
+        raise HTTPException(status_code=422, detail=str(exc)) from exc
+    await db.commit()
+    await broadcast_manual_news_changed()
+    return {"status": "ok", "group": group}
+
+
+@router.get("/news-items")
+async def list_earth_news_items_admin(
+    page: int = Query(1, ge=1),
+    page_size: int = Query(50, ge=1, le=100),
+    source_type: str | None = Query(None),
+    region: str | None = Query(None),
+    category: str | None = Query(None),
+    status_filter: str | None = Query(None, alias="status"),
+    group_id: str | None = Query(None),
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    return await list_news_records(
+        db,
+        page=page,
+        page_size=page_size,
+        source_type=source_type,
+        region=region,
+        category=category,
+        status_filter=status_filter,
+        group_id=group_id,
+    )
+
+
+@router.post("/news-items")
+async def create_earth_news_item_admin(
+    payload: EarthNewsManualItemPayload,
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    try:
+        result = await upsert_manual_news_item(db, payload.model_dump())
+    except ValueError as exc:
+        raise HTTPException(status_code=422, detail=str(exc)) from exc
+    except PermissionError as exc:
+        raise HTTPException(status_code=403, detail=str(exc)) from exc
+    await db.commit()
+    await broadcast_manual_news_changed()
+    return {"status": "ok", "created": result.created, "queued": result.queued, "item": serialize_news_record(result.item)}
+
+
+@router.post("/news-items/import")
+async def import_earth_news_items_admin(
+    file: UploadFile = File(...),
+    group_id: str | None = Form(default=None),
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    try:
+        payload = await parse_manual_news_import_upload(await file.read())
+        result = await import_manual_news_items(db, payload, group_id=group_id)
+    except ValueError as exc:
+        raise HTTPException(status_code=422, detail=str(exc)) from exc
+    await db.commit()
+    await broadcast_manual_news_changed()
+    return {"status": "ok", **result}
+
+
+@router.put("/news-items/{item_id:path}")
+async def update_earth_news_item_admin(
+    item_id: str,
+    payload: EarthNewsManualItemPayload,
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    existing = await get_news_record_or_404(db, item_id)
+    if existing is None:
+        raise HTTPException(status_code=404, detail="News item not found.")
+    try:
+        result = await upsert_manual_news_item(
+            db,
+            payload.model_dump(),
+            item_id_override=item_id,
+        )
+    except ValueError as exc:
+        raise HTTPException(status_code=422, detail=str(exc)) from exc
+    except PermissionError as exc:
+        raise HTTPException(status_code=403, detail=str(exc)) from exc
+    await db.commit()
+    await broadcast_manual_news_changed()
+    return {"status": "ok", "created": result.created, "queued": result.queued, "item": serialize_news_record(result.item)}
+
+
+@router.delete("/news-items/{item_id:path}")
+async def delete_earth_news_item_admin(
+    item_id: str,
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    try:
+        deleted = await delete_manual_news_item(db, item_id)
+    except PermissionError as exc:
+        raise HTTPException(status_code=403, detail=str(exc)) from exc
+    if not deleted:
+        raise HTTPException(status_code=404, detail="News item not found.")
+    await db.commit()
+    await broadcast_manual_news_changed()
+    return {"status": "deleted", "id": item_id}
+
+
+@router.post("/news-items/{item_id:path}/reprocess")
+async def reprocess_earth_news_item_admin(
+    item_id: str,
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    existing = await get_news_record_or_404(db, item_id)
+    if existing is None:
+        raise HTTPException(status_code=404, detail="News item not found.")
+    try:
+        queued = await reprocess_manual_news_item(db, item_id)
+    except PermissionError as exc:
+        raise HTTPException(status_code=403, detail=str(exc)) from exc
+    await db.commit()
+    await broadcast_manual_news_changed()
+    return {"status": "queued" if queued else "not_queued", "queued": queued, "id": item_id}
+
+
 @router.get("/oobe-status")
 async def get_earth_oobe_status(
     current_user: User | None = Depends(_get_optional_current_user),
@@ -317,6 +574,11 @@ async def get_earth_oobe_status(
         select(func.count(CollectedData.id)).where(CollectedData.is_current.is_(True))
     )
     current_record_count = int(current_count_result.scalar() or 0)
+    system_result = await db.execute(
+        select(SystemSetting).where(SystemSetting.category == SYSTEM_SETTINGS_CATEGORY)
+    )
+    system_record = system_result.scalar_one_or_none()
+    demo_mode = _is_demo_mode_enabled(system_record.payload if system_record else None)
 
     datasource_count_result = await db.execute(select(func.count(DataSource.id)))
     datasource_count = int(datasource_count_result.scalar() or 0)
@@ -337,6 +599,8 @@ async def get_earth_oobe_status(
     ready = has_collected_data
 
     suggestions: list[str] = []
+    if demo_mode:
+        suggestions.append("演示模式已开启")
     if not current_user:
         suggestions.append("登录控制台")
     if not has_collected_data:
@@ -348,8 +612,9 @@ async def get_earth_oobe_status(
 
     return {
         "ready": ready,
+        "demo_mode": demo_mode,
         "authenticated": current_user is not None,
-        "needs_login": current_user is None and not ready,
+        "needs_login": current_user is None and not ready and not demo_mode,
         "has_collected_data": has_collected_data,
         "has_tv_sources": tv_source_count > 0,
         "has_core_layers": has_core_layers,

backend/app/api/v1/interactables.py

@@ -0,0 +1,190 @@
+"""CRUD APIs for persistent Earth interactables."""
+
+from __future__ import annotations
+
+from datetime import UTC, datetime
+from typing import Any
+
+from fastapi import APIRouter, Depends, HTTPException, Query, Response, status
+from pydantic import BaseModel, Field, field_validator
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.core.security import get_current_user
+from app.db.session import get_db
+from app.models.earth_interactable import EarthInteractable
+from app.models.user import User
+from app.services.earth_interactables import (
+    build_interactable_event,
+    interactables_to_geojson,
+    invalidate_interactable_cache,
+    list_interactables,
+    normalize_interactable_id,
+    publish_interactable_event,
+    serialize_interactable,
+)
+from app.services.earth_layer_cache import (
+    EarthLayerCachePolicy,
+    earth_layer_cache,
+    get_or_build_layer_payload,
+)
+
+router = APIRouter()
+INTERACTABLE_CACHE_POLICY = EarthLayerCachePolicy(
+    fresh_ttl_seconds=60,
+    stale_ttl_seconds=10 * 60,
+    max_features=5000,
+)
+
+
+class InteractableCreate(BaseModel):
+    id: str | None = Field(default=None, max_length=160)
+    layer: str = Field(default="default", min_length=1, max_length=80)
+    kind: str = Field(default="default", min_length=1, max_length=80)
+    label: str = Field(default="", max_length=255)
+    description: str = Field(default="", max_length=4000)
+    latitude: float = Field(ge=-90, le=90)
+    longitude: float = Field(ge=-180, le=180)
+    altitude: float | None = None
+    properties: dict[str, Any] = Field(default_factory=dict)
+
+    @field_validator("layer", "kind")
+    @classmethod
+    def normalize_key(cls, value: str) -> str:
+        normalized = str(value or "").strip()
+        if not normalized:
+            raise ValueError("must not be empty")
+        return normalized
+
+
+class InteractableUpdate(BaseModel):
+    layer: str | None = Field(default=None, min_length=1, max_length=80)
+    kind: str | None = Field(default=None, min_length=1, max_length=80)
+    label: str | None = Field(default=None, max_length=255)
+    description: str | None = Field(default=None, max_length=4000)
+    latitude: float | None = Field(default=None, ge=-90, le=90)
+    longitude: float | None = Field(default=None, ge=-180, le=180)
+    altitude: float | None = None
+    properties: dict[str, Any] | None = None
+
+
+@router.get("")
+async def get_interactables(
+    response: Response,
+    layer: str | None = Query(default=None),
+    include_deleted: bool = Query(default=False),
+    db: AsyncSession = Depends(get_db),
+):
+    items = await list_interactables(db, layer=layer, include_deleted=include_deleted)
+    response.headers["X-Planet-Interactables-Count"] = str(len(items))
+    return {"items": [serialize_interactable(item) for item in items]}
+
+
+@router.get("/geojson")
+async def get_interactables_geojson(
+    response: Response,
+    layer: str | None = Query(default=None),
+    db: AsyncSession = Depends(get_db),
+):
+    async def build_payload() -> dict[str, Any]:
+        items = await list_interactables(db, layer=layer)
+        return interactables_to_geojson(items)
+
+    payload = await get_or_build_layer_payload(
+        key=earth_layer_cache.key("interactables", interactable_layer=layer or "all"),
+        policy=INTERACTABLE_CACHE_POLICY,
+        builder=build_payload,
+        response=response,
+    )
+    response.headers["X-Planet-Interactables-Count"] = str(len(payload.get("features") or []))
+    return payload
+
+
+@router.post("", status_code=status.HTTP_201_CREATED)
+async def create_interactable(
+    payload: InteractableCreate,
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    record_id = normalize_interactable_id(payload.id)
+    existing = await db.get(EarthInteractable, record_id)
+    if existing and not existing.is_deleted:
+        raise HTTPException(status_code=409, detail="Interactable already exists")
+
+    if existing is None:
+        record = EarthInteractable(id=record_id)
+        db.add(record)
+    else:
+        record = existing
+        record.is_deleted = False
+        record.deleted_at = None
+        record.revision += 1
+
+    record.layer = payload.layer
+    record.kind = payload.kind
+    record.label = payload.label
+    record.description = payload.description
+    record.latitude = payload.latitude
+    record.longitude = payload.longitude
+    record.altitude = payload.altitude
+    record.properties = payload.properties
+
+    await db.commit()
+    await db.refresh(record)
+    invalidate_interactable_cache(record.layer)
+    await publish_interactable_event("created", record)
+    return {"item": serialize_interactable(record)}
+
+
+@router.get("/{interactable_id}")
+async def get_interactable(interactable_id: str, db: AsyncSession = Depends(get_db)):
+    record = await db.get(EarthInteractable, interactable_id)
+    if record is None or record.is_deleted:
+        raise HTTPException(status_code=404, detail="Interactable not found")
+    return {"item": serialize_interactable(record)}
+
+
+@router.patch("/{interactable_id}")
+async def update_interactable(
+    interactable_id: str,
+    payload: InteractableUpdate,
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    record = await db.get(EarthInteractable, interactable_id)
+    if record is None or record.is_deleted:
+        raise HTTPException(status_code=404, detail="Interactable not found")
+
+    previous_layer = record.layer
+    patch = payload.model_dump(exclude_unset=True)
+    for key, value in patch.items():
+        setattr(record, key, value)
+    record.revision += 1
+
+    await db.commit()
+    await db.refresh(record)
+    invalidate_interactable_cache(previous_layer)
+    if record.layer != previous_layer:
+        invalidate_interactable_cache(record.layer)
+    await publish_interactable_event("updated", record)
+    return {"item": serialize_interactable(record)}
+
+
+@router.delete("/{interactable_id}")
+async def delete_interactable(
+    interactable_id: str,
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    record = await db.get(EarthInteractable, interactable_id)
+    if record is None or record.is_deleted:
+        raise HTTPException(status_code=404, detail="Interactable not found")
+
+    record.is_deleted = True
+    record.deleted_at = datetime.now(UTC)
+    record.revision += 1
+    await db.commit()
+    await db.refresh(record)
+    invalidate_interactable_cache(record.layer)
+    await publish_interactable_event("deleted", record)
+    event = build_interactable_event(action="deleted", record=record, include_item=True)
+    return {"deleted": True, "event": event}

backend/app/api/v1/layers.py

@@ -105,7 +105,7 @@ def _parse_layer_bbox(bbox: str) -> tuple[float, float, float, float]:
 @router.get("/vessels/snapshot")
 async def get_vessel_layer_snapshot(
     bbox: str = Query(..., description="lon_min,lat_min,lon_max,lat_max"),
-    zoom: int = Query(..., ge=1, le=20),
+    zoom: float = Query(..., ge=1, le=20),
     limit: int = Query(DEFAULT_LAYER_LIMIT, ge=1),
     vessel_type: Optional[str] = Query(None, alias="type"),
     since_minutes: int = Query(60, ge=1, le=1440),

backend/app/api/v1/news.py

@@ -1,16 +1,92 @@
-from fastapi import APIRouter, Depends, Query
+from fastapi import APIRouter, Depends, HTTPException, Query
 from sqlalchemy.ext.asyncio import AsyncSession
 
 from app.db.session import get_db
-from app.services.earth_news import get_earth_news_payload
+from app.services.earth_news import (
+    ALLOWED_NEWS_CATEGORY_KEYS,
+    SUPPORTED_NEWS_LOCALES,
+    REGION_ANCHORS,
+    get_earth_news_payload,
+)
 
 router = APIRouter()
 
 
+def _parse_categories(raw: str | None) -> set[str] | None:
+    if raw is None or not raw.strip():
+        return None
+    requested = {item.strip().lower() for item in raw.split(",") if item.strip()}
+    invalid = sorted(requested - set(ALLOWED_NEWS_CATEGORY_KEYS))
+    if invalid:
+        raise HTTPException(
+            status_code=422,
+            detail={
+                "message": "Unsupported news categories.",
+                "invalid_categories": invalid,
+                "allowed_categories": list(ALLOWED_NEWS_CATEGORY_KEYS),
+            },
+        )
+    return requested or None
+
+
+def _parse_source_ids(raw: str | None) -> set[str] | None:
+    if raw is None or not raw.strip():
+        return None
+    return {item.strip() for item in raw.split(",") if item.strip()} or None
+
+
+def _parse_limit(raw: int | None) -> int:
+    if raw is None:
+        return 12
+    if raw < 1:
+        raise HTTPException(status_code=422, detail={"message": "News limit must be greater than 0."})
+    return min(raw, 100)
+
+
+def _parse_locale(raw: str | None) -> str:
+    if raw is None or not raw.strip():
+        return "zh-CN"
+    requested = raw.strip()
+    if requested not in SUPPORTED_NEWS_LOCALES:
+        raise HTTPException(
+            status_code=422,
+            detail={
+                "message": "Unsupported news locale.",
+                "invalid_locale": requested,
+                "allowed_locales": sorted(SUPPORTED_NEWS_LOCALES),
+            },
+        )
+    return requested
+
+
 @router.get("/earth-feed")
 async def get_earth_feed(
     lat: float | None = Query(None, description="Current Earth view center latitude"),
     lon: float | None = Query(None, description="Current Earth view center longitude"),
+    region: str | None = Query(None, description="Explicit Earth news region for UE/client integrations"),
+    categories: str | None = Query(None, description="Comma-separated news category keys"),
+    sources: str | None = Query(None, description="Comma-separated news source ids"),
+    limit: int | None = Query(None, description="Maximum news items to return, capped at 100"),
+    locale: str | None = Query(None, description="Display locale, zh-CN or en-US"),
     db: AsyncSession = Depends(get_db),
 ):
-    return await get_earth_news_payload(lat=lat, lon=lon, db=db)
+    normalized_region = region.strip().lower() if isinstance(region, str) and region.strip() else None
+    if normalized_region is not None and normalized_region not in REGION_ANCHORS:
+        raise HTTPException(
+            status_code=422,
+            detail={
+                "message": "Unsupported news region.",
+                "invalid_region": normalized_region,
+                "allowed_regions": list(REGION_ANCHORS.keys()),
+            },
+        )
+    return await get_earth_news_payload(
+        lat=lat,
+        lon=lon,
+        region=normalized_region,
+        categories=_parse_categories(categories),
+        source_ids=_parse_source_ids(sources),
+        limit=_parse_limit(limit),
+        locale=_parse_locale(locale),
+        db=db,
+    )

backend/app/api/v1/settings.py

@@ -11,6 +11,8 @@ from dotenv import dotenv_values
 from sqlalchemy import select
 from sqlalchemy.ext.asyncio import AsyncSession
 
+from app.core.logging import get_logger
+from app.core.enums import ProviderApi, TVSourceType, UserRole
 from app.core.security import get_current_user
 from app.core.time import to_iso8601_utc
 from app.core.config import settings as app_settings
@@ -66,11 +68,13 @@ from app.services.llm_provider_catalog import (
 from app.services.scheduler import sync_datasource_job
 from app.services.tv_streams import DEFAULT_TV_SETTINGS, get_tv_settings_payload, normalize_tv_settings
 from app.services.persistent_logs import record_audit_log
+from app.services.business_logs import emit_business_log, exception_context
 
 router = APIRouter()
+logger = get_logger(__name__, service="api")
 AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS = 5
 AI_CONNECTION_TEST_PROMPT_KEY = "ai.connection_test"
-SECRET_REVEAL_ROLES = {"admin", "super_admin"}
+SECRET_REVEAL_ROLES = {UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value}
 
 DEFAULT_SETTINGS = {
     "system": {
@@ -79,6 +83,7 @@ DEFAULT_SETTINGS = {
         "auto_refresh": True,
         "data_retention_days": 30,
         "max_concurrent_tasks": 5,
+        "demo_mode": False,
     },
     "notifications": {
         "email_enabled": False,
@@ -203,6 +208,7 @@ class SystemSettingsUpdate(BaseModel):
     auto_refresh: bool = True
     data_retention_days: int = Field(default=30, ge=1, le=3650)
     max_concurrent_tasks: int = Field(default=5, ge=1, le=50)
+    demo_mode: bool = False
 
 
 class NotificationSettingsUpdate(BaseModel):
@@ -231,7 +237,7 @@ class TVStreamSourceUpdate(BaseModel):
     provider: str = Field(default="Unknown", max_length=100)
     region: str = Field(default="Global", max_length=100)
     language: str = Field(default="und", max_length=32)
-    source_type: str = Field(default="iframe", pattern="^(iframe|hls|video|external|youtube)$")
+    source_type: TVSourceType = TVSourceType.IFRAME
     embed_url: str = ""
     stream_url: str = ""
     homepage_url: str = ""
@@ -274,7 +280,7 @@ class AIProviderIntegrationUpdate(BaseModel):
     service_token: Optional[str] = None
     default_provider: Optional[str] = None
     provider: str = Field(default="minimax", max_length=80)
-    provider_api: str = Field(default="anthropic-messages", max_length=80)
+    provider_api: ProviderApi = ProviderApi.ANTHROPIC_MESSAGES
     base_url: str = Field(default="", max_length=500)
     model: str = Field(default="", max_length=200)
     api_key: Optional[str] = None
@@ -418,7 +424,7 @@ def _get_provider_preset(provider: str) -> dict:
     except ValueError:
         return {
             "provider": provider,
-            "provider_api": "openai-completions",
+            "provider_api": ProviderApi.OPENAI_COMPLETIONS.value,
             "base_url": "",
         "model": "",
         "models": [],
@@ -482,12 +488,12 @@ def _provider_defaults(provider: str) -> dict:
     preset = _get_provider_preset(provider)
     return {
         "provider": provider,
-        "provider_api": preset.get("provider_api") or "openai-completions",
+        "provider_api": preset.get("provider_api") or ProviderApi.OPENAI_COMPLETIONS.value,
         "base_url": preset.get("base_url") or "",
         "model": preset.get("model") or "",
         "api_key": "",
         "max_tokens": (
-            1200 if preset.get("provider_api") == "anthropic-messages" else 4096
+            1200 if preset.get("provider_api") == ProviderApi.ANTHROPIC_MESSAGES.value else 4096
         ),
         "anthropic_version": "2023-06-01",
         "model_provider_apis": preset.get("model_provider_apis") or {},
@@ -660,7 +666,7 @@ def _runtime_config_from_ai_payload(ai_payload: dict) -> dict:
         ),
         "llm_config": {
             "provider": default_provider,
-            "provider_api": provider_config.get("provider_api") or "anthropic-messages",
+            "provider_api": provider_config.get("provider_api") or ProviderApi.ANTHROPIC_MESSAGES.value,
             "base_url": provider_config.get("base_url") or "",
             "model": provider_config.get("model") or "",
             "api_key": api_key,
@@ -699,6 +705,18 @@ async def _validate_ai_provider_full_connection(ai_payload: dict) -> dict:
         retry_attempts=runtime_config["retry_attempts"],
         llm_config=runtime_config.get("llm_config") or {},
     )
+    await emit_business_log(
+        logger,
+        event="settings.ai_provider.full_connection.start",
+        message="AI provider full connection validation started",
+        category="ai",
+        service="api",
+        module=__name__,
+        context={
+            "provider": runtime_config.get("llm_config", {}).get("provider"),
+            "model": runtime_config.get("llm_config", {}).get("model"),
+        },
+    )
     status_result = await client.get_status()
     if not status_result.configured:
         raise HTTPException(
@@ -715,6 +733,19 @@ async def _validate_ai_provider_full_connection(ai_payload: dict) -> dict:
             constraints=["回复尽量简短。"],
         )
     )
+    await emit_business_log(
+        logger,
+        event="settings.ai_provider.full_connection.success",
+        message="AI provider full connection validation completed",
+        category="ai",
+        service="api",
+        module=__name__,
+        context={
+            "provider": analysis_result.provider,
+            "model": analysis_result.model,
+            "configured": status_result.configured,
+        },
+    )
     return {
         "status": status_result.model_dump(),
         "provider": analysis_result.provider,
@@ -751,7 +782,10 @@ def _contains_model(model_ids: list[str], model: str) -> bool:
 
 async def _check_ai_provider_lightweight(llm_config: dict, timeout_seconds: int) -> dict:
     provider = _normalize_provider_id(llm_config.get("provider") or "")
-    configured_api = str(llm_config.get("provider_api") or "").strip() or "openai-completions"
+    configured_api = (
+        str(llm_config.get("provider_api") or "").strip()
+        or ProviderApi.OPENAI_COMPLETIONS.value
+    )
     model = str(llm_config.get("model") or "").strip()
     base_url = str(llm_config.get("base_url") or "").strip().rstrip("/")
     api_key = str(llm_config.get("api_key") or "").strip()
@@ -772,7 +806,7 @@ async def _check_ai_provider_lightweight(llm_config: dict, timeout_seconds: int)
             "message": "当前 provider/base_url/model 未完整配置。",
             "mode": "lightweight_config",
         }
-    if provider_api != "ollama-generate" and not api_key:
+    if provider_api != ProviderApi.OLLAMA_GENERATE.value and not api_key:
         return {
             "success": False,
             "connected": False,
@@ -783,13 +817,13 @@ async def _check_ai_provider_lightweight(llm_config: dict, timeout_seconds: int)
     if provider == "opencode-go":
         url = _join_provider_url(base_url, "/models")
         headers = {"Authorization": f"Bearer {api_key}"}
-    elif provider_api == "ollama-generate":
+    elif provider_api == ProviderApi.OLLAMA_GENERATE.value:
         url = _join_provider_url(base_url, "/api/tags")
         headers: dict[str, str] = {}
-    elif provider_api == "openai-completions":
+    elif provider_api == ProviderApi.OPENAI_COMPLETIONS.value:
         url = _join_provider_url(base_url, "/models")
         headers = {"Authorization": f"Bearer {api_key}"}
-    elif provider_api == "anthropic-messages":
+    elif provider_api == ProviderApi.ANTHROPIC_MESSAGES.value:
         url = _join_provider_url(base_url, "/models")
         headers = {
             "x-api-key": api_key,
@@ -1135,7 +1169,7 @@ async def serialize_external_integrations(db: AsyncSession) -> dict:
         api_key, api_key_source = _resolve_provider_api_key(provider_id, provider_config)
         providers_payload[provider_id] = {
             "provider": provider_id,
-            "provider_api": provider_config.get("provider_api") or "openai-completions",
+            "provider_api": provider_config.get("provider_api") or ProviderApi.OPENAI_COMPLETIONS.value,
             "base_url": provider_config.get("base_url") or "",
             "model": provider_config.get("model") or "",
             "api_key": _mask_secret(api_key, api_key_source),
@@ -1185,7 +1219,7 @@ async def serialize_external_integrations(db: AsyncSession) -> dict:
             "service_token": _mask_secret(*_resolve_service_token(normalized_ai)),
             "default_provider": default_provider,
             "provider": default_provider,
-            "provider_api": display_llm_config.get("provider_api") or "anthropic-messages",
+            "provider_api": display_llm_config.get("provider_api") or ProviderApi.ANTHROPIC_MESSAGES.value,
             "base_url": display_llm_config.get("base_url") or "https://api.minimaxi.com/anthropic",
             "model": display_llm_config.get("model") or "MiniMax-M2.7",
             "api_key": display_llm_config.get("api_key") or _mask_secret(None),
@@ -1428,7 +1462,7 @@ async def update_smtp_settings(
     current_user: User = Depends(get_current_user),
     db: AsyncSession = Depends(get_db),
 ):
-    if current_user.role not in ("admin", "super_admin"):
+    if current_user.role not in (UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value):
         raise HTTPException(status_code=403, detail="Only administrators can change SMTP settings")
     current = await get_setting_payload(db, "smtp")
     merged = _build_smtp_payload(current, payload)
@@ -1442,7 +1476,7 @@ async def test_smtp_settings(
     current_user: User = Depends(get_current_user),
     db: AsyncSession = Depends(get_db),
 ):
-    if current_user.role not in ("admin", "super_admin"):
+    if current_user.role not in (UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value):
         raise HTTPException(status_code=403, detail="Only administrators can test SMTP settings")
     from app.services.email import EmailError, send_email
 
@@ -1612,9 +1646,34 @@ async def connect_ai_provider_integration(
         llm_config=quick_llm_config,
     )
 
+    await emit_business_log(
+        logger,
+        event="settings.ai_provider.connect.start",
+        message="AI provider connection test started",
+        category="ai",
+        service="api",
+        module=__name__,
+        user_id=current_user.id,
+        context={
+            "provider": payload.provider,
+            "model": payload.model,
+            "timeout_seconds": min(int(runtime_config["timeout_seconds"] or 60), AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS),
+        },
+    )
     try:
         status_result = await client.get_status()
         if not status_result.configured:
+            await emit_business_log(
+                logger,
+                event="settings.ai_provider.connect.failed",
+                message="AI provider connection test failed because provider is incomplete",
+                category="ai",
+                level="warning",
+                service="api",
+                module=__name__,
+                user_id=current_user.id,
+                context={"provider": payload.provider, "model": payload.model, "configured": False},
+            )
             return {
                 "success": False,
                 "connected": False,
@@ -1628,6 +1687,21 @@ async def connect_ai_provider_integration(
                 AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS,
             ),
         )
+        await emit_business_log(
+            logger,
+            event="settings.ai_provider.connect.success",
+            message="AI provider connection test completed",
+            category="ai",
+            service="api",
+            module=__name__,
+            user_id=current_user.id,
+            context={
+                "provider": payload.provider,
+                "model": payload.model,
+                "configured": True,
+                "lightweight_status": lightweight_result.get("status"),
+            },
+        )
         return {
             **lightweight_result,
             "status": status_result.model_dump(),
@@ -1639,6 +1713,17 @@ async def connect_ai_provider_integration(
             "message": str(exc.detail),
         }
     except Exception as exc:
+        await emit_business_log(
+            logger,
+            event="settings.ai_provider.connect.failed",
+            message="AI provider connection test failed",
+            category="ai",
+            level="error",
+            service="api",
+            module=__name__,
+            user_id=current_user.id,
+            context=exception_context(exc, {"provider": payload.provider, "model": payload.model}),
+        )
         return {
             "success": False,
             "connected": False,
@@ -1786,8 +1871,28 @@ async def connect_web_search_integration(
     runtime_config = _runtime_config_from_web_search_payload(draft_web_search_payload)
     client = WebSearchClient(runtime_config)
 
+    await emit_business_log(
+        logger,
+        event="settings.web_search.connect.start",
+        message="WebSearch connection test started",
+        category="ai_tool",
+        service="api",
+        module=__name__,
+        user_id=current_user.id,
+        context={"provider": runtime_config.default_provider},
+    )
     try:
         results = await client.test_connection()
+        await emit_business_log(
+            logger,
+            event="settings.web_search.connect.success",
+            message="WebSearch connection test completed",
+            category="ai_tool",
+            service="api",
+            module=__name__,
+            user_id=current_user.id,
+            context={"provider": runtime_config.default_provider, "result_count": len(results)},
+        )
         return {
             "success": True,
             "connected": True,
@@ -1796,18 +1901,51 @@ async def connect_web_search_integration(
             "results": [item.model_dump(mode="json") for item in results[:3]],
         }
     except WebSearchConfigurationError as exc:
+        await emit_business_log(
+            logger,
+            event="settings.web_search.connect.failed",
+            message="WebSearch connection test failed because configuration is incomplete",
+            category="ai_tool",
+            level="warning",
+            service="api",
+            module=__name__,
+            user_id=current_user.id,
+            context=exception_context(exc, {"provider": runtime_config.default_provider}),
+        )
         return {
             "success": False,
             "connected": False,
             "message": str(exc),
         }
     except WebSearchError as exc:
+        await emit_business_log(
+            logger,
+            event="settings.web_search.connect.failed",
+            message="WebSearch connection test failed",
+            category="ai_tool",
+            level="error",
+            service="api",
+            module=__name__,
+            user_id=current_user.id,
+            context=exception_context(exc, {"provider": runtime_config.default_provider}),
+        )
         return {
             "success": False,
             "connected": False,
             "message": str(exc),
         }
     except Exception as exc:
+        await emit_business_log(
+            logger,
+            event="settings.web_search.connect.failed",
+            message="WebSearch connection test failed",
+            category="ai_tool",
+            level="error",
+            service="api",
+            module=__name__,
+            user_id=current_user.id,
+            context=exception_context(exc, {"provider": runtime_config.default_provider}),
+        )
         return {
             "success": False,
             "connected": False,
@@ -1835,17 +1973,60 @@ async def generate_provider_credential_guide(
     ai_client: AIProviderClient = Depends(get_ai_provider_client),
 ):
     try:
+        await emit_business_log(
+            logger,
+            event="settings.credential_guide.generate.start",
+            message="Credential guide generation started",
+            category="ai",
+            service="api",
+            module=__name__,
+            user_id=current_user.id,
+            context={"provider": provider},
+        )
         web_search_client = await get_web_search_client(db)
-        return {
-            "guide": await generate_credential_guide(
+        guide = await generate_credential_guide(
                 db,
                 provider,
                 ai_client,
                 web_search_client,
             )
-        }
+        await emit_business_log(
+            logger,
+            event="settings.credential_guide.generate.success",
+            message="Credential guide generation completed",
+            category="ai",
+            service="api",
+            module=__name__,
+            user_id=current_user.id,
+            context={"provider": provider},
+        )
+        return {"guide": guide}
     except ValueError as exc:
+        await emit_business_log(
+            logger,
+            event="settings.credential_guide.generate.failed",
+            message="Credential guide generation failed",
+            category="ai",
+            level="warning",
+            service="api",
+            module=__name__,
+            user_id=current_user.id,
+            context=exception_context(exc, {"provider": provider}),
+        )
         raise HTTPException(status_code=404, detail=str(exc)) from exc
+    except Exception as exc:
+        await emit_business_log(
+            logger,
+            event="settings.credential_guide.generate.failed",
+            message="Credential guide generation failed",
+            category="ai",
+            level="error",
+            service="api",
+            module=__name__,
+            user_id=current_user.id,
+            context=exception_context(exc, {"provider": provider}),
+        )
+        raise
 
 
 @router.post("/credential-guides/{provider}/reset")

backend/app/api/v1/system_control.py

@@ -1,20 +1,19 @@
 from __future__ import annotations
 
 import os
+import secrets
 import subprocess
 import sys
 
 from datetime import datetime
 
-from fastapi import APIRouter, Depends, HTTPException, Query, Request, status
+from fastapi import APIRouter, Depends, Header, HTTPException, Query, Request, status
 from pydantic import BaseModel
-from sqlalchemy import select
 from sqlalchemy.ext.asyncio import AsyncSession
 
-from app.core.config import ROOT_DIR
+from app.core.config import ROOT_DIR, settings
 from app.core.security import get_current_user
 from app.db.session import get_db
-from app.models.system_log import AuditLog, SystemLog
 from app.models.user import User
 from app.services.persistent_logs import record_audit_log, record_system_log
 from app.services.system_control import (
@@ -37,7 +36,11 @@ from app.services.system_logs import (
     append_buffer_log,
     list_log_sources,
     normalize_log_level,
+    read_database_log_snapshot,
     read_log_snapshot,
+    read_observability_group_events,
+    read_observability_groups,
+    read_observability_raw_events,
 )
 from app.services.earth_layer_cache import earth_layer_cache
 
@@ -109,12 +112,104 @@ class EarthClientLogEventCreate(BaseModel):
     url: str | None = None
     module: str | None = None
     detail: str | None = None
+    fingerprint: str | None = None
+    occurrence_count: int = 1
+    metadata: dict[str, object] | None = None
 
 
 class EarthClientLogEventResponse(BaseModel):
     accepted: bool
     source_id: str
     level: str
+    fingerprint: str | None = None
+
+
+class ServiceLogEventCreate(BaseModel):
+    source: str = "ai-provider"
+    service: str = "ai-provider"
+    module: str | None = None
+    category: str | None = None
+    event: str = "service.runtime_log"
+    level: str = "error"
+    message: str
+    fingerprint: str | None = None
+    occurrence_count: int = 1
+    request_id: str | None = None
+    trace_id: str | None = None
+    task_id: str | None = None
+    source_id: int | str | None = None
+    provider: str | None = None
+    context: dict[str, object] | None = None
+
+
+async def ingest_client_log_event(
+    source_id: str,
+    *,
+    service: str,
+    event: str,
+    default_module: str,
+    default_category: str,
+    payload: EarthClientLogEventCreate,
+    request: Request,
+) -> EarthClientLogEventResponse:
+    normalized_level = normalize_log_level(payload.level)
+    append_buffer_log(
+        source_id,
+        level=normalized_level,
+        message=payload.message,
+        context={
+            "category": payload.category or "",
+            "url": payload.url or "",
+            "module": payload.module or "",
+            "detail": payload.detail or "",
+            "fingerprint": payload.fingerprint or "",
+            "occurrence_count": max(1, int(payload.occurrence_count or 1)),
+            "metadata": payload.metadata or {},
+        },
+    )
+    await record_system_log(
+        source=source_id,
+        service=service,
+        module=payload.module or default_module,
+        event=event,
+        level=normalized_level,
+        message=payload.message,
+        category=payload.category or default_category,
+        context={
+            "url": payload.url or "",
+            "detail": payload.detail or "",
+            "module": payload.module or "",
+            "client_ip": request.client.host if request.client else "",
+            "metadata": payload.metadata or {},
+        },
+        fingerprint=payload.fingerprint,
+        occurrence_count=max(1, int(payload.occurrence_count or 1)),
+    )
+    return EarthClientLogEventResponse(accepted=True, source_id=source_id, level=normalized_level, fingerprint=payload.fingerprint)
+
+
+def require_observability_ingest_token(
+    authorization: str | None,
+    ingest_token: str | None,
+) -> None:
+    expected_token = settings.OBSERVABILITY_INGEST_TOKEN.strip()
+    if not expected_token:
+        raise HTTPException(
+            status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
+            detail="Observability service ingestion is not configured",
+        )
+    provided = ""
+    if ingest_token:
+        provided = ingest_token.strip()
+    elif authorization:
+        scheme, _, token = authorization.partition(" ")
+        if scheme.lower() == "bearer":
+            provided = token.strip()
+    if not provided or not secrets.compare_digest(provided, expected_token):
+        raise HTTPException(
+            status_code=status.HTTP_403_FORBIDDEN,
+            detail="Invalid observability ingestion token",
+        )
 
 
 class EarthLayerCacheStatusResponse(BaseModel):
@@ -339,95 +434,116 @@ async def get_system_log_sources(
     }
 
 
-async def read_database_log_snapshot(
-    source_id: str,
-    *,
-    limit: int,
-    level: str,
-    levels: str | None,
-    start_date: str | None,
-    end_date: str | None,
-    search: str | None,
-    db: AsyncSession,
-) -> dict | None:
-    selected_levels = set(normalize_log_level(item) for item in (levels or level).split(",") if item.strip())
-    selected_levels.discard("all")
-    search_query = (search or "").strip().lower()
-    lines: list[str] = []
+@router.get("/logs/observability/groups")
+async def get_observability_log_groups(
+    limit: int = DEFAULT_LOG_LINE_LIMIT,
+    level: str = "all",
+    levels: str | None = Query(None, description="Comma-separated log levels"),
+    start_date: str | None = Query(None, description="Filter logs from this date (YYYY-MM-DD)"),
+    end_date: str | None = Query(None, description="Filter logs until this date (YYYY-MM-DD)"),
+    search: str | None = Query(None, description="Case-insensitive substring search"),
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    ensure_super_admin(current_user)
+    if limit < 1 or limit > MAX_LOG_LINE_LIMIT:
+        raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=f"limit must be between 1 and {MAX_LOG_LINE_LIMIT}")
+    normalized_start_date = validate_log_date(start_date, "start_date")
+    normalized_end_date = validate_log_date(end_date, "end_date")
+    if normalized_start_date and normalized_end_date and normalized_start_date > normalized_end_date:
+        raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="start_date must be earlier than or equal to end_date")
+    return await read_observability_groups(
+        limit=limit,
+        level=level,
+        levels=levels,
+        start_date=normalized_start_date,
+        end_date=normalized_end_date,
+        search=search,
+        db=db,
+    )
 
-    if source_id == "system-db":
-        query = select(SystemLog).order_by(SystemLog.occurred_at.desc().nullslast(), SystemLog.id.desc()).limit(limit * 5)
-        result = await db.execute(query)
-        records = result.scalars().all()
-        for record in records:
-            record_level = normalize_log_level(record.level)
-            if selected_levels and record_level not in selected_levels:
-                continue
-            occurred_at = record.occurred_at.date().isoformat() if record.occurred_at else ""
-            if start_date and occurred_at and occurred_at < start_date:
-                continue
-            if end_date and occurred_at and occurred_at > end_date:
-                continue
-            line = " ".join(
-                part
-                for part in [
-                    record.occurred_at.isoformat() if record.occurred_at else "",
-                    record_level.upper(),
-                    record.source,
-                    record.event or "",
-                    record.message,
-                ]
-                if part
-            )
-            if search_query and search_query not in line.lower():
-                continue
-            lines.append(line)
-    elif source_id == "audit-db":
-        query = select(AuditLog).order_by(AuditLog.occurred_at.desc().nullslast(), AuditLog.id.desc()).limit(limit * 5)
-        result = await db.execute(query)
-        records = result.scalars().all()
-        for record in records:
-            occurred_at = record.occurred_at.date().isoformat() if record.occurred_at else ""
-            if start_date and occurred_at and occurred_at < start_date:
-                continue
-            if end_date and occurred_at and occurred_at > end_date:
-                continue
-            line = " ".join(
-                part
-                for part in [
-                    record.occurred_at.isoformat() if record.occurred_at else "",
-                    "INFO",
-                    record.action,
-                    record.target_type or "",
-                    record.target_id or "",
-                    record.result or "",
-                ]
-                if part
-            )
-            if search_query and search_query not in line.lower():
-                continue
-            lines.append(line)
-    else:
-        return None
 
-    lines = list(reversed(lines[:limit]))
-    return {
-        "source_id": source_id,
-        "name": "系统事件" if source_id == "system-db" else "审计事件",
-        "kind": "database",
-        "location": "table://system_logs" if source_id == "system-db" else "table://audit_logs",
-        "description": "数据库持久化日志",
-        "category": "database" if source_id == "system-db" else "audit",
-        "status": "ok" if lines else "empty",
-        "level": level,
-        "selected_levels": sorted(selected_levels),
-        "search_query": search or "",
-        "available_levels": ["all", "error", "warning", "info", "debug"],
-        "daily_markers": [],
-        "line_limit": limit,
-        "line_count": len(lines),
-        "lines": lines,
-    }
+@router.get("/logs/observability/groups/{fingerprint}/events")
+async def get_observability_group_events(
+    fingerprint: str,
+    limit: int = DEFAULT_LOG_LINE_LIMIT,
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    ensure_super_admin(current_user)
+    if limit < 1 or limit > MAX_LOG_LINE_LIMIT:
+        raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=f"limit must be between 1 and {MAX_LOG_LINE_LIMIT}")
+    payload = await read_observability_group_events(fingerprint, limit=limit, db=db)
+    if payload is None:
+        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Observability group not found")
+    return payload
+
+
+@router.get("/logs/observability/raw")
+async def get_observability_raw_events(
+    limit: int = DEFAULT_LOG_LINE_LIMIT,
+    level: str = "all",
+    levels: str | None = Query(None, description="Comma-separated log levels"),
+    start_date: str | None = Query(None, description="Filter logs from this date (YYYY-MM-DD)"),
+    end_date: str | None = Query(None, description="Filter logs until this date (YYYY-MM-DD)"),
+    search: str | None = Query(None, description="Case-insensitive substring search"),
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    ensure_super_admin(current_user)
+    if limit < 1 or limit > MAX_LOG_LINE_LIMIT:
+        raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=f"limit must be between 1 and {MAX_LOG_LINE_LIMIT}")
+    normalized_start_date = validate_log_date(start_date, "start_date")
+    normalized_end_date = validate_log_date(end_date, "end_date")
+    return await read_observability_raw_events(
+        limit=limit,
+        level=level,
+        levels=levels,
+        start_date=normalized_start_date,
+        end_date=normalized_end_date,
+        search=search,
+        db=db,
+    )
+
+
+@router.post("/logs/service", response_model=EarthClientLogEventResponse)
+async def ingest_service_log(
+    payload: ServiceLogEventCreate,
+    authorization: str | None = Header(default=None),
+    ingest_token: str | None = Header(default=None, alias="X-Planet-Observability-Token"),
+):
+    require_observability_ingest_token(authorization, ingest_token)
+    normalized_level = normalize_log_level(payload.level)
+    source = (payload.source or "ai-provider").strip() or "ai-provider"
+    context = dict(payload.context or {})
+    if payload.request_id:
+        context["request_id"] = payload.request_id
+    if payload.trace_id:
+        context["trace_id"] = payload.trace_id
+    if payload.task_id:
+        context["task_id"] = payload.task_id
+    if payload.source_id is not None:
+        context["source_id"] = payload.source_id
+    if payload.provider:
+        context["provider"] = payload.provider
+    await record_system_log(
+        source=source,
+        service=(payload.service or source).strip() or source,
+        module=payload.module or source,
+        event=(payload.event or "service.runtime_log").strip() or "service.runtime_log",
+        level=normalized_level,
+        message=payload.message,
+        category=payload.category or "service-runtime",
+        context=context,
+        fingerprint=payload.fingerprint,
+        occurrence_count=max(1, int(payload.occurrence_count or 1)),
+    )
+    return EarthClientLogEventResponse(
+        accepted=True,
+        source_id=source,
+        level=normalized_level,
+        fingerprint=payload.fingerprint,
+    )
 
 
 @router.get("/logs/{source_id}", response_model=SystemLogSnapshotResponse)
@@ -493,31 +609,28 @@ async def ingest_earth_client_log(
     payload: EarthClientLogEventCreate,
     request: Request,
 ):
-    normalized_level = normalize_log_level(payload.level)
-    append_buffer_log(
+    return await ingest_client_log_event(
         "earth-client",
-        level=normalized_level,
-        message=payload.message,
-        context={
-            "category": payload.category or "",
-            "url": payload.url or "",
-            "module": payload.module or "",
-            "detail": payload.detail or "",
-        },
-    )
-    await record_system_log(
-        source="earth-client",
         service="earth",
-        module=payload.module or "earth-client",
         event="earth.client.runtime_log",
-        level=normalized_level,
-        message=payload.message,
-        category=payload.category or "client-runtime",
-        context={
-            "url": payload.url or "",
-            "detail": payload.detail or "",
-            "module": payload.module or "",
-            "client_ip": request.client.host if request.client else "",
-        },
+        default_module="earth-client",
+        default_category="client-runtime",
+        payload=payload,
+        request=request,
+    )
+
+
+@router.post("/logs/admin-client", response_model=EarthClientLogEventResponse)
+async def ingest_admin_client_log(
+    payload: EarthClientLogEventCreate,
+    request: Request,
+):
+    return await ingest_client_log_event(
+        "admin-client",
+        service="admin",
+        event="admin.client.runtime_log",
+        default_module="admin-client",
+        default_category="client-runtime",
+        payload=payload,
+        request=request,
     )
-    return {"accepted": True, "source_id": "earth-client", "level": normalized_level}

backend/app/api/v1/tasks.py

@@ -29,7 +29,8 @@ async def list_tasks(
         SELECT ct.id, ct.datasource_id, ds.name as datasource_name, ct.status,
                ct.started_at, ct.completed_at, ct.records_processed, ct.error_message,
                ct.phase, ct.phase_progress, ct.phase_message, ct.phase_current,
-               ct.phase_total, ct.phase_unit, ct.total_records, ct.progress
+               ct.phase_total, ct.phase_unit, ct.total_records, ct.progress,
+               ct.task_type, ct.source, ds.source as datasource_source
         FROM collection_tasks ct
         JOIN data_sources ds ON ct.datasource_id = ds.id
         WHERE 1=1
@@ -39,12 +40,19 @@ async def list_tasks(
 
     if datasource_id:
         query += " AND ct.datasource_id = :datasource_id"
-        count_query += " WHERE ct.datasource_id = :datasource_id"
+        count_query += " AND ct.datasource_id = :datasource_id"
         params["datasource_id"] = datasource_id
     if status:
-        query += " AND ct.status = :status"
-        count_query += " AND ct.status = :status"
-        params["status"] = status
+        statuses = [item.strip() for item in status.split(",") if item.strip()]
+        if len(statuses) > 1:
+            placeholders = ", ".join(f":status_{index}" for index, _item in enumerate(statuses))
+            query += f" AND ct.status IN ({placeholders})"
+            count_query += f" AND ct.status IN ({placeholders})"
+            params.update({f"status_{index}": item for index, item in enumerate(statuses)})
+        else:
+            query += " AND ct.status = :status"
+            count_query += " AND ct.status = :status"
+            params["status"] = statuses[0] if statuses else status
 
     query += f" ORDER BY ct.created_at DESC LIMIT {page_size} OFFSET {offset}"
 
@@ -76,6 +84,9 @@ async def list_tasks(
                 "phase_unit": t[13],
                 "total_records": t[14],
                 "progress": t[15],
+                "task_type": t[16],
+                "source": t[17] or t[18],
+                "datasource_source": t[18],
             }
             for t in tasks
         ],

backend/app/api/v1/tv.py

@@ -1,3 +1,4 @@
+import re
 from urllib.parse import quote, urljoin
 
 import httpx
@@ -10,6 +11,26 @@ from app.services.tv_streams import get_public_tv_payload, is_allowed_tv_proxy_u
 
 router = APIRouter()
 
+_HLS_URI_ATTRIBUTE_RE = re.compile(r'URI="([^"]+)"')
+
+
+def _proxied_tv_url(url: str) -> str:
+    return f"/api/v1/tv/proxy?url={quote(url, safe='')}"
+
+
+def _rewrite_hls_uri_attributes(line: str, *, base_url: str) -> str:
+    def replace(match: re.Match[str]) -> str:
+        uri = match.group(1)
+        absolute_url = urljoin(base_url, uri)
+        return f'URI="{_proxied_tv_url(absolute_url)}"'
+
+    return _HLS_URI_ATTRIBUTE_RE.sub(replace, line)
+
+
+def _should_strip_hls_metadata_line(line: str) -> bool:
+    normalized = line.strip().upper()
+    return normalized.startswith("#EXT-X-MEDIA:") and "TYPE=SUBTITLES" in normalized
+
 
 @router.get("/streams")
 async def list_public_tv_streams(
@@ -56,11 +77,16 @@ async def proxy_tv_stream(
         rewritten_lines: list[str] = []
         for line in manifest_text.splitlines():
             stripped = line.strip()
-            if not stripped or stripped.startswith("#"):
+            if not stripped:
                 rewritten_lines.append(line)
                 continue
+            if stripped.startswith("#"):
+                if _should_strip_hls_metadata_line(line):
+                    continue
+                rewritten_lines.append(_rewrite_hls_uri_attributes(line, base_url=response_url))
+                continue
             absolute_url = urljoin(response_url, stripped)
-            rewritten_lines.append(f"/api/v1/tv/proxy?url={quote(absolute_url, safe='')}")
+            rewritten_lines.append(_proxied_tv_url(absolute_url))
         return Response(
             content="\n".join(rewritten_lines),
             media_type="application/vnd.apple.mpegurl",

backend/app/api/v1/users.py

@@ -5,6 +5,7 @@ from fastapi import APIRouter, Depends, HTTPException, status
 from sqlalchemy.ext.asyncio import AsyncSession
 from sqlalchemy import text
 
+from app.core.enums import UserRole
 from app.core.security import get_current_user, get_password_hash
 from app.db.session import get_db
 from app.models.user import User
@@ -13,6 +14,8 @@ from app.schemas.user import UserCreate, UserUpdate
 router = APIRouter()
 
 VALID_GATEKEEPER_GROUPS = {"docs_user", "docs_developer", "docs_admin"}
+ADMIN_ROLES = [UserRole.SUPER_ADMIN.value, UserRole.ADMIN.value]
+SUPER_ADMIN_ROLES = [UserRole.SUPER_ADMIN.value]
 
 
 def check_permission(current_user: User, required_roles: List[str]) -> bool:
@@ -32,7 +35,7 @@ async def list_users(
     current_user: User = Depends(get_current_user),
     db: AsyncSession = Depends(get_db),
 ):
-    if not check_permission(current_user, ["super_admin", "admin"]):
+    if not check_permission(current_user, ADMIN_ROLES):
         raise HTTPException(
             status_code=status.HTTP_403_FORBIDDEN,
             detail="Insufficient permissions",
@@ -91,7 +94,7 @@ async def get_user(
     current_user: User = Depends(get_current_user),
     db: AsyncSession = Depends(get_db),
 ):
-    if not check_permission(current_user, ["super_admin", "admin"]) and current_user.id != user_id:
+    if not check_permission(current_user, ADMIN_ROLES) and current_user.id != user_id:
         raise HTTPException(
             status_code=status.HTTP_403_FORBIDDEN,
             detail="Insufficient permissions",
@@ -128,7 +131,7 @@ async def create_user(
     current_user: User = Depends(get_current_user),
     db: AsyncSession = Depends(get_db),
 ):
-    if not check_permission(current_user, ["super_admin"]):
+    if not check_permission(current_user, SUPER_ADMIN_ROLES):
         raise HTTPException(
             status_code=status.HTTP_403_FORBIDDEN,
             detail="Only super_admin can create users",
@@ -196,18 +199,18 @@ async def update_user(
     current_user: User = Depends(get_current_user),
     db: AsyncSession = Depends(get_db),
 ):
-    if not check_permission(current_user, ["super_admin", "admin"]) and current_user.id != user_id:
+    if not check_permission(current_user, ADMIN_ROLES) and current_user.id != user_id:
         raise HTTPException(
             status_code=status.HTTP_403_FORBIDDEN,
             detail="Insufficient permissions",
         )
 
-    if not check_permission(current_user, ["super_admin"]) and user_data.role is not None:
+    if not check_permission(current_user, SUPER_ADMIN_ROLES) and user_data.role is not None:
         raise HTTPException(
             status_code=status.HTTP_403_FORBIDDEN,
             detail="Only super_admin can change user role",
         )
-    if not check_permission(current_user, ["super_admin"]) and user_data.gatekeeper_groups is not None:
+    if not check_permission(current_user, SUPER_ADMIN_ROLES) and user_data.gatekeeper_groups is not None:
         raise HTTPException(
             status_code=status.HTTP_403_FORBIDDEN,
             detail="Only super_admin can change Gatekeeper groups",
@@ -260,7 +263,7 @@ async def delete_user(
     current_user: User = Depends(get_current_user),
     db: AsyncSession = Depends(get_db),
 ):
-    if not check_permission(current_user, ["super_admin"]):
+    if not check_permission(current_user, SUPER_ADMIN_ROLES):
         raise HTTPException(
             status_code=status.HTTP_403_FORBIDDEN,
             detail="Only super_admin can delete users",

backend/app/api/v1/vessels.py

@@ -1,4 +1,4 @@
-"""Bounded vessel snapshot APIs for viewport-first consumers."""
+"""Bounded vessel snapshot APIs backed by the latest vessel state table."""
 
 from typing import Optional
 
@@ -14,8 +14,8 @@ router = APIRouter()
 
 @router.get("/snapshot")
 async def get_vessel_snapshot(
-    bbox: Optional[str] = Query(None, description="Viewport bbox as lon_min,lat_min,lon_max,lat_max"),
-    zoom: int = Query(..., ge=1, le=20, description="Current map zoom level"),
+    bbox: Optional[str] = Query(None, description="Snapshot bbox as lon_min,lat_min,lon_max,lat_max"),
+    zoom: float = Query(..., ge=1, le=20, description="Current map zoom level"),
     type: Optional[str] = Query(
         None,
         description="Comma-separated vessel types: cargo,tanker,passenger,fishing,military,other",

backend/app/api/v1/visualization.py

@@ -18,6 +18,7 @@ from sqlalchemy import select, func
 from typing import List, Dict, Any, Optional
 
 from app.core.collected_data_fields import get_record_field
+from app.core.enums import BGPStatus
 from app.core.satellite_tle import build_tle_lines_from_elements
 from app.core.time import to_iso8601_utc
 from app.db.session import get_db
@@ -25,7 +26,7 @@ from app.models.bgp_anomaly import BGPAnomaly
 from app.models.bgp_incident import BGPIncident
 from app.models.bgp_observation import BGPObservation
 from app.models.collected_data import CollectedData
-from app.models.vessel import AISSourceHealth, VesselPosition, VesselStatic
+from app.models.vessel import AISSourceHealth, VesselCurrentState, VesselPosition, VesselStatic
 from app.services.bgp_collectors import build_bgp_collector_coverage
 from app.services.cable_graph import build_graph_from_data, CableGraph, haversine_distance
 from app.services.compute_center_locations import (
@@ -38,7 +39,7 @@ from app.services.compute_center_locations import (
     upsert_compute_center_location,
 )
 from app.services.ai_client import get_ai_provider_client
-from app.api.v1.settings import get_web_search_client
+from app.api.v1.settings import get_runtime_web_search_config, get_web_search_client
 from app.services.collectors.bgp_common import RIPE_RIS_COLLECTOR_COORDS
 from app.services.location.llm_fallback import (
     collect_llm_location_fallback_candidate,
@@ -47,11 +48,10 @@ from app.services.location.llm_fallback import (
 from app.services.persistent_logs import record_system_log
 from app.services.vessel_ais_aggregation import (
     build_field_conflict_candidates,
-    count_unique_raw_vessel_mmsi,
     get_aggregated_vessel,
     get_aggregated_vessel_track,
     get_aggregated_vessels,
-    get_aggregated_vessels_snapshot,
+    get_current_vessels_snapshot,
     get_vessel_conflict_records,
     get_vessel_raw_observations,
     MAX_SNAPSHOT_LIMIT,
@@ -75,7 +75,6 @@ TERRAIN_TILE_BATCH_MAX_ITEMS = 128
 TERRAIN_TILE_BATCH_CONCURRENCY = 16
 _terrain_tile_cache: OrderedDict[tuple[int, int, int], tuple[bytes, str, dict[str, str]]] = OrderedDict()
 VESSEL_NAME_FALLBACK_PATTERN = re.compile(r"^mmsi\s*\d+$", re.IGNORECASE)
-VESSEL_SNAPSHOT_LEGACY_FALLBACK_ENABLED = True
 SECONDS_PER_MINUTE = 60
 BYTES_PER_MIB = 1024 * 1024
 CABLE_CACHE_FRESH_SECONDS = 6 * 60 * SECONDS_PER_MINUTE
@@ -105,8 +104,8 @@ LANDING_POINT_CACHE_POLICY = EarthLayerCachePolicy(
 SATELLITE_CACHE_POLICY = EarthLayerCachePolicy(
     SATELLITE_CACHE_FRESH_SECONDS,
     SATELLITE_CACHE_STALE_SECONDS,
-    max_features=8000,
-    max_bytes=10 * BYTES_PER_MIB,
+    max_features=25000,
+    max_bytes=32 * BYTES_PER_MIB,
 )
 COMPUTE_CENTER_CACHE_POLICY = EarthLayerCachePolicy(
     COMPUTE_CENTER_CACHE_FRESH_SECONDS,
@@ -839,9 +838,14 @@ def convert_aggregated_vessels_to_geojson(vessels: List[dict[str, Any]]) -> Dict
             continue
         source_summary = {}
         for source, summary in (vessel.get("source_summary") or {}).items():
+            latest_observed_at = summary.get("latest_observed_at")
             source_summary[source] = {
                 **summary,
-                "latest_observed_at": to_iso8601_utc(summary.get("latest_observed_at")),
+                "latest_observed_at": (
+                    to_iso8601_utc(latest_observed_at)
+                    if isinstance(latest_observed_at, datetime)
+                    else latest_observed_at
+                ),
             }
         props = {
             "mmsi": vessel["mmsi"],
@@ -1075,7 +1079,7 @@ async def build_vessel_snapshot_response(
     db: AsyncSession,
     *,
     bbox: tuple[float, float, float, float] | None,
-    zoom: int | None,
+    zoom: float | None,
     type_filter: str | None,
     limit: int | None,
     since_minutes: int = 60,
@@ -1563,15 +1567,7 @@ async def _build_cables_geojson(db: AsyncSession) -> dict[str, Any]:
     try:
         records = await _load_current_collected_data(db, "arcgis_cables")
 
-        if not records:
-            raise HTTPException(
-                status_code=404,
-                detail="No cable data found. Please run the arcgis_cables collector first.",
-            )
-
         return convert_cable_to_geojson(records)
-    except HTTPException:
-        raise
     except Exception as e:
         logger.exception_event(
             "Failed to build cables GeoJSON response",
@@ -1625,16 +1621,8 @@ async def _build_landing_points_geojson(db: AsyncSession) -> dict[str, Any]:
             relation_records,
             cable_records,
         )
-        
-        if not records:
-            raise HTTPException(
-                status_code=404,
-                detail="No landing point data found. Please run the arcgis_landing_points collector first.",
-            )
-        
+
         return convert_landing_point_to_geojson(records, city_to_cable_ids_map, cable_id_to_name_map)
-    except HTTPException:
-        raise
     except Exception as e:
         logger.exception_event(
             "Failed to build landing points GeoJSON response",
@@ -2071,6 +2059,44 @@ class SaveComputeCenterLocationRequest(BaseModel):
     model_config = {"populate_by_name": True}
 
 
+async def _compute_center_location_web_search_capability(db: AsyncSession) -> Dict[str, Any]:
+    try:
+        config = await get_runtime_web_search_config(db)
+    except Exception as exc:
+        return {
+            "enabled": False,
+            "provider": None,
+            "reason": f"WebSearch 配置读取失败：{exc}",
+        }
+    provider_config = config.active_provider_config
+    has_api_key = bool((provider_config.api_key or "").strip())
+    if not config.enabled:
+        return {
+            "enabled": False,
+            "provider": config.default_provider,
+            "reason": "WebSearch 未开启，无法进行事实核查定位。",
+        }
+    if not has_api_key:
+        return {
+            "enabled": False,
+            "provider": config.default_provider,
+            "reason": f"WebSearch Provider {config.default_provider} 未配置 API Key。",
+        }
+    return {
+        "enabled": True,
+        "provider": config.default_provider,
+        "reason": "",
+    }
+
+
+@router.get("/compute-centers/location-capability")
+async def get_compute_center_location_capability(
+    db: AsyncSession = Depends(get_db),
+):
+    """Return whether fact-checked compute-center location collection can run."""
+    return await _compute_center_location_web_search_capability(db)
+
+
 @router.post("/compute-centers/{source_id}/collect-location")
 async def collect_compute_center_location(
     source_id: str,
@@ -2089,6 +2115,9 @@ async def collect_compute_center_location(
     """
     if not source_id or not source_id.strip():
         raise HTTPException(status_code=400, detail="source_id is required")
+    capability = await _compute_center_location_web_search_capability(db)
+    if not capability.get("enabled"):
+        raise HTTPException(status_code=409, detail=capability)
 
     record = await _load_compute_center_record(db, source_id)
     name = payload.name or (record.name if record else None)
@@ -2315,58 +2344,35 @@ async def _load_raw_vessel_snapshot_features(
     observed_since: datetime,
 ) -> tuple[list[dict[str, Any]], dict[str, Any]]:
     if bbox is None:
-        aggregated_vessels = await get_aggregated_vessels(
-            db,
-            limit=limit,
-            observed_since=observed_since,
-        )
-    else:
-        aggregated_vessels = await get_aggregated_vessels_snapshot(
-            db,
-            bbox=bbox,
-            limit=limit,
-            observed_since=observed_since,
-        )
-    raw_geojson = convert_aggregated_vessels_to_geojson(aggregated_vessels)
-    raw_features = raw_geojson.get("features", [])
-    features = raw_features
-    legacy_features: list[dict[str, Any]] = []
-    legacy_fallback_used = False
-    if not raw_features and VESSEL_SNAPSHOT_LEGACY_FALLBACK_ENABLED:
-        legacy_features = await _load_legacy_vessel_snapshot_features(
-            db,
-            bbox=bbox,
-            limit=limit,
-        )
-        features, _merge_diagnostics = _merge_vessel_features(raw_features, legacy_features)
-        legacy_fallback_used = bool(legacy_features)
-
+        return [], {
+            "source": "vessel_current_state",
+            "current_state_count": 0,
+            "final_unique_mmsi": 0,
+        }
+    current_vessels = await get_current_vessels_snapshot(
+        db,
+        bbox=bbox,
+        limit=limit,
+        observed_since=observed_since,
+    )
+    features = convert_aggregated_vessels_to_geojson(current_vessels).get("features", [])
+    unique_mmsi = len(
+        {
+            key
+            for key in (_feature_mmsi_key(feature) for feature in features)
+            if key is not None
+        }
+    )
     return features, {
-        "raw_feature_count": len(raw_features),
-        "raw_unique_mmsi": len(
-            {
-                key
-                for key in (_feature_mmsi_key(feature) for feature in raw_features)
-                if key is not None
-            }
-        ),
-        "legacy_feature_count": len(legacy_features),
-        "legacy_backfilled_mmsi": len(
-            {
-                key
-                for key in (_feature_mmsi_key(feature) for feature in legacy_features)
-                if key is not None
-            }
-        ),
-        "legacy_fallback_enabled": VESSEL_SNAPSHOT_LEGACY_FALLBACK_ENABLED,
-        "legacy_fallback_used": legacy_fallback_used,
-        "final_unique_mmsi": len(
-            {
-                key
-                for key in (_feature_mmsi_key(feature) for feature in features)
-                if key is not None
-            }
-        ),
+        "source": "vessel_current_state",
+        "current_state_count": len(features),
+        "final_unique_mmsi": unique_mmsi,
+        "raw_feature_count": 0,
+        "raw_unique_mmsi": 0,
+        "legacy_feature_count": 0,
+        "legacy_backfilled_mmsi": 0,
+        "legacy_fallback_enabled": False,
+        "legacy_fallback_used": False,
     }
 
 @router.get("/vessels/custom-supplements")
@@ -2696,6 +2702,8 @@ async def _build_bgp_collectors_geojson(db: AsyncSession) -> dict[str, Any]:
         db,
         source_filter=("ris_live_bgp", "bgpstream_bgp"),
     )
+    if not any(int(item.get("observation_count") or 0) > 0 for item in coverage):
+        return {"type": "FeatureCollection", "features": [], "count": 0}
     coverage_by_collector = {
         item["collector"]: item
         for item in coverage
@@ -2732,10 +2740,10 @@ async def _build_visualization_geo_summary(db: AsyncSession) -> dict[str, Any]:
     compute_center_count = supercomputer_count + gpu_cluster_count
 
     active_incident_result = await db.execute(
-        select(func.count(BGPIncident.id)).where(BGPIncident.status == "active"),
+        select(func.count(BGPIncident.id)).where(BGPIncident.status == BGPStatus.ACTIVE.value),
     )
     active_anomaly_result = await db.execute(
-        select(func.count(BGPAnomaly.id)).where(BGPAnomaly.status == "active"),
+        select(func.count(BGPAnomaly.id)).where(BGPAnomaly.status == BGPStatus.ACTIVE.value),
     )
     active_incident_count = int(active_incident_result.scalar() or 0)
     active_anomaly_count = int(active_anomaly_result.scalar() or 0)
@@ -2756,21 +2764,14 @@ async def _build_visualization_geo_summary(db: AsyncSession) -> dict[str, Any]:
         )
     else:
         bgp_collector_count = int(bgp_collector_scalar or 0)
-    raw_unique_window_hours = 24
-    raw_unique_mmsi = await count_unique_raw_vessel_mmsi(
-        db,
-        observed_since=datetime.now(UTC) - timedelta(hours=raw_unique_window_hours),
+    vessel_current_window_minutes = 60
+    vessel_current_result = await db.execute(
+        select(func.count(VesselCurrentState.mmsi)).where(
+            VesselCurrentState.observed_at
+            >= datetime.now(UTC) - timedelta(minutes=vessel_current_window_minutes)
+        )
     )
-    legacy_unique_result = await db.execute(
-        select(func.count(func.distinct(VesselPosition.mmsi)))
-    )
-    legacy_unique_mmsi = int(legacy_unique_result.scalar() or 0)
-    legacy_fallback_active = (
-        VESSEL_SNAPSHOT_LEGACY_FALLBACK_ENABLED
-        and raw_unique_mmsi == 0
-        and legacy_unique_mmsi > 0
-    )
-    vessel_count = legacy_unique_mmsi if legacy_fallback_active else raw_unique_mmsi
+    vessel_count = int(vessel_current_result.scalar() or 0)
     aisstream_health = await db.get(AISSourceHealth, "aisstream_vessels")
 
     return {
@@ -2781,11 +2782,10 @@ async def _build_visualization_geo_summary(db: AsyncSession) -> dict[str, Any]:
             "satellite_count": satellite_count,
             "compute_center_count": compute_center_count,
             "vessel_count": vessel_count,
-            "vessel_count_source": "legacy_fallback" if legacy_fallback_active else "raw_recent",
-            "vessel_legacy_fallback_enabled": VESSEL_SNAPSHOT_LEGACY_FALLBACK_ENABLED,
-            "vessel_raw_unique_mmsi": raw_unique_mmsi,
-            "vessel_raw_unique_window_hours": raw_unique_window_hours,
-            "vessel_legacy_unique_mmsi": legacy_unique_mmsi,
+            "vessel_count_source": "vessel_current_state",
+            "vessel_current_window_minutes": vessel_current_window_minutes,
+            "vessel_raw_unique_mmsi": 0,
+            "vessel_legacy_unique_mmsi": 0,
             "aisstream_connection_state": aisstream_health.connection_state if aisstream_health else None,
             "aisstream_last_seen_at": to_iso8601_utc(aisstream_health.last_seen_at) if aisstream_health else None,
             "aisstream_message_rate": aisstream_health.message_rate if aisstream_health else None,

backend/app/api/v1/websocket.py

@@ -6,11 +6,15 @@ from typing import Optional
 
 from fastapi import APIRouter, WebSocket, WebSocketDisconnect, Query
 from jose import jwt, JWTError
+from sqlalchemy import text
 
 from app.core.config import settings
+from app.core.enums import UserRole
 from app.core.logging import get_logger
 from app.core.time import to_iso8601_utc
 from app.core.websocket.manager import manager
+from app.db.session import async_session_factory
+from app.services.log_tail import LOG_TAIL_CHANNEL, log_tail_manager
 
 logger = get_logger(__name__, service="api")
 router = APIRouter()
@@ -37,6 +41,28 @@ async def authenticate_token(token: str) -> Optional[dict]:
         return None
 
 
+async def load_websocket_user_role(user_id: str | None) -> str | None:
+    if not user_id:
+        return None
+    try:
+        async with async_session_factory() as db:
+            result = await db.execute(
+                text("SELECT role, is_active FROM users WHERE id = :id"),
+                {"id": int(user_id)},
+            )
+            row = result.fetchone()
+    except Exception as exc:
+        logger.warning_event(
+            "WebSocket user role lookup failed",
+            event="auth.websocket.role_lookup_failed",
+            context={"user_id": user_id, "error": str(exc)},
+        )
+        return None
+    if row is None or not row[1]:
+        return None
+    return str(row[0] or "")
+
+
 @router.websocket("/ws")
 async def websocket_endpoint(
     websocket: WebSocket,
@@ -59,6 +85,7 @@ async def websocket_endpoint(
 
     is_anonymous = payload is None
     user_id = str(payload.get("sub")) if payload else f"anonymous:{id(websocket)}"
+    user_role = await load_websocket_user_role(user_id) if payload else None
     supported_channels = ["vessels", "earth_news", EARTH_UPDATES_CHANNEL] if is_anonymous else [
         "gpu_clusters",
         "submarine_cables",
@@ -70,6 +97,8 @@ async def websocket_endpoint(
         "earth_news",
         EARTH_UPDATES_CHANNEL,
     ]
+    if user_role == UserRole.SUPER_ADMIN.value:
+        supported_channels = [*supported_channels, LOG_TAIL_CHANNEL]
     await manager.connect(websocket, user_id)
 
     try:
@@ -100,6 +129,7 @@ async def websocket_endpoint(
                     payload_data = data.get("data", {})
                     if not isinstance(payload_data, dict):
                         payload_data = {}
+                    log_tail_config = None
                     channels = payload_data.get("channels", [])
                     if isinstance(channels, str):
                         channels = [channels]
@@ -108,6 +138,26 @@ async def websocket_endpoint(
                     channel = payload_data.get("channel")
                     if channel and channel not in channels:
                         channels = [*channels, channel]
+                    if LOG_TAIL_CHANNEL in channels:
+                        if user_role != UserRole.SUPER_ADMIN.value:
+                            await websocket.send_json(
+                                {
+                                    "type": "subscription_error",
+                                    "data": {"channel": LOG_TAIL_CHANNEL, "detail": "Only super_admin can subscribe logs"},
+                                }
+                            )
+                            channels = [item for item in channels if item != LOG_TAIL_CHANNEL]
+                        else:
+                            try:
+                                log_tail_config = await log_tail_manager.subscribe(websocket, payload_data)
+                            except ValueError as exc:
+                                await websocket.send_json(
+                                    {
+                                        "type": "subscription_error",
+                                        "data": {"channel": LOG_TAIL_CHANNEL, "detail": str(exc)},
+                                    }
+                                )
+                            channels = [item for item in channels if item != LOG_TAIL_CHANNEL]
                     if is_anonymous:
                         channels = [channel for channel in channels if channel in supported_channels]
                     vessel_subscription = None
@@ -131,14 +181,20 @@ async def websocket_endpoint(
                                 "action": "subscribe",
                                 "channels": [
                                     *channels,
+                                    *([LOG_TAIL_CHANNEL] if log_tail_config else []),
                                     *(["vessels"] if vessel_subscription else []),
                                 ],
                                 "vessels": vessel_subscription,
+                                "logs_tail": log_tail_config.__dict__ if log_tail_config else None,
                             },
                         }
                     )
                 elif data.get("type") == "unsubscribe":
                     channels = data.get("data", {}).get("channels", [])
+                    if isinstance(channels, str):
+                        channels = [channels]
+                    if LOG_TAIL_CHANNEL in channels:
+                        await log_tail_manager.unsubscribe(websocket)
                     manager.unsubscribe(websocket, channels)
                     await websocket.send_json(
                         {
@@ -159,4 +215,5 @@ async def websocket_endpoint(
     except WebSocketDisconnect:
         pass
     finally:
+        await log_tail_manager.disconnect(websocket)
         manager.disconnect(websocket, user_id)

backend/app/core/config.py

@@ -41,6 +41,7 @@ class Settings(BaseSettings):
     AI_PROVIDER_SERVICE_TOKEN: str = ""
     AI_PROVIDER_TIMEOUT_SECONDS: int = 60
     AI_PROVIDER_RETRY_ATTEMPTS: int = 2
+    OBSERVABILITY_INGEST_TOKEN: str = ""
 
     @property
     def REDIS_URL(self) -> str:

backend/app/core/enums.py

@@ -0,0 +1,226 @@
+"""Stable backend protocol enums.
+
+Database columns and JSON payloads continue to store the enum string values.
+Configurable identifiers, user-authored values, and open-ended taxonomies do
+not belong in this module.
+"""
+
+from __future__ import annotations
+
+import logging
+from enum import StrEnum
+from typing import TypeVar
+
+logger = logging.getLogger(__name__)
+
+EnumT = TypeVar("EnumT", bound=StrEnum)
+
+
+def parse_enum(enum_type: type[EnumT], value: object, default: EnumT) -> EnumT:
+    """Parse an external value without breaking reads of legacy data."""
+
+    if value is None or str(value).strip() == "":
+        return default
+    if isinstance(value, enum_type):
+        return value
+    try:
+        return enum_type(str(value).strip().lower())
+    except (TypeError, ValueError):
+        logger.warning(
+            "Unknown %s value %r; falling back to %s",
+            enum_type.__name__,
+            value,
+            default.value,
+        )
+        return default
+
+
+class NewsImportanceLevel(StrEnum):
+    LOW = "low"
+    MEDIUM = "medium"
+    HIGH = "high"
+    CRITICAL = "critical"
+
+
+class BreakingLevel(StrEnum):
+    NONE = "none"
+    WATCH = "watch"
+    BREAKING = "breaking"
+    CRITICAL = "critical"
+
+
+class BreakingScope(StrEnum):
+    REGIONAL = "regional"
+    GLOBAL = "global"
+
+
+class BreakingSource(StrEnum):
+    RULES = "rules"
+    AI = "ai"
+    MANUAL = "manual"
+    MULTI_SOURCE = "multi_source"
+
+
+class NewsSourceType(StrEnum):
+    RSS = "rss"
+    ATOM = "atom"
+    AGGREGATED = "aggregated"
+    REFERENCE = "reference"
+    MANUAL = "manual"
+
+
+class NewsEnrichmentStatus(StrEnum):
+    PENDING = "pending"
+    QUEUED = "queued"
+    ATTEMPTED = "attempted"
+    SUCCESS = "success"
+    CONTENT_ONLY = "content_only"
+    LOCATION_ONLY = "location_only"
+    UNAVAILABLE = "unavailable"
+    PROVIDER_ERROR = "provider_error"
+    PARSE_ERROR = "parse_error"
+    NO_RESULT = "no_result"
+
+
+class NewsMarketImpact(StrEnum):
+    NONE = "none"
+    SECTOR = "sector"
+    NATIONAL = "national"
+    GLOBAL = "global"
+
+
+class NewsTaggingSource(StrEnum):
+    RULES = "rules"
+    AI = "ai"
+    MANUAL = "manual"
+
+
+class JobType(StrEnum):
+    COLLECT = "collect"
+    CLEAR_DATA = "clear_data"
+    CLEAR_CACHE = "clear_cache"
+    EARTH_REFRESH = "earth_refresh"
+
+
+class JobStatus(StrEnum):
+    QUEUED = "queued"
+    RUNNING = "running"
+    CANCELLING = "cancelling"
+    SUCCESS = "success"
+    FAILED = "failed"
+    CANCELLED = "cancelled"
+
+
+class RollbackPolicy(StrEnum):
+    KEEP_COMMITTED_BATCHES = "keep_committed_batches"
+
+
+class MappingValidationStatus(StrEnum):
+    DRAFT = "draft"
+    VALID = "valid"
+    INVALID = "invalid"
+
+
+class SnapshotStatus(StrEnum):
+    RUNNING = "running"
+    SUCCESS = "success"
+    FAILED = "failed"
+    CANCELLED = "cancelled"
+
+
+class DatasourceRunStatus(StrEnum):
+    RUNNING = "running"
+    NOT_RUN = "not_run"
+    COLLECTED = "collected"
+    UNCOLLECTED = "uncollected"
+
+
+class ProviderApi(StrEnum):
+    ANTHROPIC_MESSAGES = "anthropic-messages"
+    OPENAI_COMPLETIONS = "openai-completions"
+    OLLAMA_GENERATE = "ollama-generate"
+
+
+class PlaygroundMessageRole(StrEnum):
+    SYSTEM = "system"
+    USER = "user"
+    ASSISTANT = "assistant"
+    TOOL = "tool"
+
+
+class PlaygroundMessageKind(StrEnum):
+    MESSAGE = "message"
+    THINKING = "thinking"
+    ERROR = "error"
+    STATUS = "status"
+
+
+class PlaygroundMessageStatus(StrEnum):
+    PENDING = "pending"
+    THINKING = "thinking"
+    ANSWERING = "answering"
+    DONE = "done"
+    FAILED = "failed"
+    CANCELLED = "cancelled"
+    ERROR = "error"
+    STOPPED = "stopped"
+
+
+class OtpPurpose(StrEnum):
+    REGISTER = "register"
+    VERIFY_EMAIL = "verify_email"
+    RESET_PASSWORD = "reset_password"
+
+
+class UserRole(StrEnum):
+    VIEWER = "viewer"
+    ADMIN = "admin"
+    SUPER_ADMIN = "super_admin"
+
+
+class AlertSeverity(StrEnum):
+    CRITICAL = "critical"
+    WARNING = "warning"
+    INFO = "info"
+
+
+class AlertStatus(StrEnum):
+    ACTIVE = "active"
+    ACKNOWLEDGED = "acknowledged"
+    RESOLVED = "resolved"
+
+
+class BGPStatus(StrEnum):
+    ACTIVE = "active"
+    ACKNOWLEDGED = "acknowledged"
+    RESOLVED = "resolved"
+
+
+class LogLevel(StrEnum):
+    ALL = "all"
+    ERROR = "error"
+    WARNING = "warning"
+    INFO = "info"
+    DEBUG = "debug"
+
+
+class ConnectionState(StrEnum):
+    DISCONNECTED = "disconnected"
+    CONNECTING = "connecting"
+    CONNECTED = "connected"
+    ERROR = "error"
+
+
+class AuthType(StrEnum):
+    NONE = "none"
+    BEARER = "bearer"
+    API_KEY = "api_key"
+    BASIC = "basic"
+
+
+class TVSourceType(StrEnum):
+    IFRAME = "iframe"
+    HLS = "hls"
+    VIDEO = "video"
+    EXTERNAL = "external"
+    YOUTUBE = "youtube"

backend/app/core/websocket/broadcaster.py

@@ -150,7 +150,7 @@ class DataBroadcaster:
                 "timestamp": to_iso8601_utc(datetime.now(UTC)),
                 "payload": data,
             },
-            channel="all",
+            channel="datasource_tasks",
         )
 
     def start(self):

backend/app/db/session.py

@@ -203,6 +203,7 @@ async def init_db():
     import app.models.vessel_enrichment  # noqa: F401
     import app.models.datasource_mapping  # noqa: F401
     import app.models.earth_news  # noqa: F401
+    import app.models.earth_interactable  # noqa: F401
 
     logger.warning_event(
         "Database pool settings active",
@@ -258,6 +259,407 @@ async def init_db():
                 """
             )
         )
+        await conn.execute(
+            text(
+                """
+                CREATE TABLE IF NOT EXISTS earth_data_change_events (
+                    id BIGSERIAL PRIMARY KEY,
+                    table_name VARCHAR(128) NOT NULL,
+                    operation VARCHAR(16) NOT NULL,
+                    source VARCHAR(128),
+                    entity_key VARCHAR(255),
+                    payload JSONB NOT NULL,
+                    occurred_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+                    consumed_at TIMESTAMPTZ
+                )
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                CREATE INDEX IF NOT EXISTS idx_earth_data_change_events_unconsumed
+                ON earth_data_change_events (consumed_at, id)
+                WHERE consumed_at IS NULL
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                CREATE OR REPLACE FUNCTION planet_emit_earth_data_changed_statement(
+                    change_table TEXT,
+                    change_operation TEXT,
+                    change_source TEXT,
+                    source_record_count INTEGER,
+                    source_entity_keys TEXT[]
+                )
+                RETURNS VOID AS $$
+                DECLARE
+                    change_event_id BIGINT;
+                    change_payload JSONB;
+                BEGIN
+                    change_payload := jsonb_build_object(
+                        'event', 'earth.layer.changed',
+                        'table', change_table,
+                        'operation', change_operation,
+                        'source', change_source,
+                        'entity_key', NULL,
+                        'entity_keys', COALESCE(to_jsonb(source_entity_keys), '[]'::jsonb),
+                        'records_processed', COALESCE(source_record_count, 0),
+                        'occurred_at', NOW()
+                    );
+
+                    INSERT INTO earth_data_change_events (
+                        table_name,
+                        operation,
+                        source,
+                        entity_key,
+                        payload,
+                        occurred_at
+                    ) VALUES (
+                        change_table,
+                        change_operation,
+                        change_source,
+                        NULL,
+                        change_payload,
+                        NOW()
+                    )
+                    RETURNING id INTO change_event_id;
+
+                    change_payload := change_payload || jsonb_build_object(
+                        'event_id', change_event_id
+                    );
+
+                    UPDATE earth_data_change_events
+                    SET payload = change_payload
+                    WHERE id = change_event_id;
+
+                    PERFORM pg_notify(
+                        'planet_earth_data_changes',
+                        change_payload::text
+                    );
+                END;
+                $$ LANGUAGE plpgsql;
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                CREATE OR REPLACE FUNCTION planet_emit_collected_data_changed_statement(
+                    change_operation TEXT,
+                    change_source TEXT,
+                    source_record_count INTEGER,
+                    source_entity_keys TEXT[]
+                )
+                RETURNS VOID AS $$
+                BEGIN
+                    PERFORM planet_emit_earth_data_changed_statement(
+                        'collected_data',
+                        change_operation,
+                        change_source,
+                        source_record_count,
+                        source_entity_keys
+                    );
+                END;
+                $$ LANGUAGE plpgsql;
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                CREATE OR REPLACE FUNCTION planet_notify_earth_table_changed_statement()
+                RETURNS trigger AS $$
+                DECLARE
+                    change_source TEXT;
+                    source_record_count INTEGER;
+                    source_entity_keys TEXT[];
+                BEGIN
+                    IF TG_OP = 'INSERT' THEN
+                        FOR change_source IN
+                            SELECT DISTINCT COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME)
+                            FROM (SELECT to_jsonb(t) AS row_data FROM new_rows AS t) changed_rows
+                        LOOP
+                            SELECT
+                                COUNT(*),
+                                ARRAY(
+                                    SELECT DISTINCT COALESCE(
+                                        NULLIF(row_data->>'entity_key', ''),
+                                        NULLIF(row_data->>'source_id', ''),
+                                        NULLIF(row_data->>'incident_key', ''),
+                                        NULLIF(row_data->>'id', ''),
+                                        NULLIF(row_data->>'mmsi', '')
+                                    )
+                                    FROM (SELECT to_jsonb(t) AS row_data FROM new_rows AS t) rows_for_keys
+                                    WHERE COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) = change_source
+                                    LIMIT 20
+                                )
+                            INTO source_record_count, source_entity_keys
+                            FROM (SELECT to_jsonb(t) AS row_data FROM new_rows AS t) rows_for_count
+                            WHERE COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) = change_source;
+
+                            PERFORM planet_emit_earth_data_changed_statement(
+                                TG_TABLE_NAME,
+                                TG_OP,
+                                change_source,
+                                source_record_count,
+                                source_entity_keys
+                            );
+                        END LOOP;
+                    ELSIF TG_OP = 'DELETE' THEN
+                        FOR change_source IN
+                            SELECT DISTINCT COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME)
+                            FROM (SELECT to_jsonb(t) AS row_data FROM old_rows AS t) changed_rows
+                        LOOP
+                            SELECT
+                                COUNT(*),
+                                ARRAY(
+                                    SELECT DISTINCT COALESCE(
+                                        NULLIF(row_data->>'entity_key', ''),
+                                        NULLIF(row_data->>'source_id', ''),
+                                        NULLIF(row_data->>'incident_key', ''),
+                                        NULLIF(row_data->>'id', ''),
+                                        NULLIF(row_data->>'mmsi', '')
+                                    )
+                                    FROM (SELECT to_jsonb(t) AS row_data FROM old_rows AS t) rows_for_keys
+                                    WHERE COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) = change_source
+                                    LIMIT 20
+                                )
+                            INTO source_record_count, source_entity_keys
+                            FROM (SELECT to_jsonb(t) AS row_data FROM old_rows AS t) rows_for_count
+                            WHERE COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) = change_source;
+
+                            PERFORM planet_emit_earth_data_changed_statement(
+                                TG_TABLE_NAME,
+                                TG_OP,
+                                change_source,
+                                source_record_count,
+                                source_entity_keys
+                            );
+                        END LOOP;
+                    ELSIF TG_OP = 'UPDATE' THEN
+                        FOR change_source IN
+                            SELECT DISTINCT COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME)
+                            FROM (
+                                SELECT to_jsonb(t) AS row_data FROM new_rows AS t
+                                UNION ALL
+                                SELECT to_jsonb(t) AS row_data FROM old_rows AS t
+                            ) changed_rows
+                        LOOP
+                            SELECT
+                                COUNT(*),
+                                ARRAY(
+                                    SELECT DISTINCT COALESCE(
+                                        NULLIF(row_data->>'entity_key', ''),
+                                        NULLIF(row_data->>'source_id', ''),
+                                        NULLIF(row_data->>'incident_key', ''),
+                                        NULLIF(row_data->>'id', ''),
+                                        NULLIF(row_data->>'mmsi', '')
+                                    )
+                                    FROM (
+                                        SELECT to_jsonb(t) AS row_data FROM new_rows AS t
+                                        UNION ALL
+                                        SELECT to_jsonb(t) AS row_data FROM old_rows AS t
+                                    ) rows_for_keys
+                                    WHERE COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) = change_source
+                                    LIMIT 20
+                                )
+                            INTO source_record_count, source_entity_keys
+                            FROM (
+                                SELECT to_jsonb(t) AS row_data FROM new_rows AS t
+                                UNION ALL
+                                SELECT to_jsonb(t) AS row_data FROM old_rows AS t
+                            ) rows_for_count
+                            WHERE COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) = change_source;
+
+                            PERFORM planet_emit_earth_data_changed_statement(
+                                TG_TABLE_NAME,
+                                TG_OP,
+                                change_source,
+                                source_record_count,
+                                source_entity_keys
+                            );
+                        END LOOP;
+                    END IF;
+
+                    RETURN NULL;
+                END;
+                $$ LANGUAGE plpgsql;
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                CREATE OR REPLACE FUNCTION planet_notify_collected_data_changed_statement()
+                RETURNS trigger AS $$
+                DECLARE
+                    change_source TEXT;
+                    source_record_count INTEGER;
+                    source_entity_keys TEXT[];
+                BEGIN
+                    IF TG_OP = 'INSERT' THEN
+                        FOR change_source IN
+                            SELECT DISTINCT source FROM new_rows WHERE source IS NOT NULL
+                        LOOP
+                            SELECT
+                                COUNT(*),
+                                ARRAY(
+                                    SELECT DISTINCT COALESCE(entity_key, source_id, id::text)
+                                    FROM new_rows
+                                    WHERE source = change_source
+                                    LIMIT 20
+                                )
+                            INTO source_record_count, source_entity_keys
+                            FROM new_rows
+                            WHERE source = change_source;
+
+                            PERFORM planet_emit_collected_data_changed_statement(
+                                TG_OP,
+                                change_source,
+                                source_record_count,
+                                source_entity_keys
+                            );
+                        END LOOP;
+                    ELSIF TG_OP = 'DELETE' THEN
+                        FOR change_source IN
+                            SELECT DISTINCT source FROM old_rows WHERE source IS NOT NULL
+                        LOOP
+                            SELECT
+                                COUNT(*),
+                                ARRAY(
+                                    SELECT DISTINCT COALESCE(entity_key, source_id, id::text)
+                                    FROM old_rows
+                                    WHERE source = change_source
+                                    LIMIT 20
+                                )
+                            INTO source_record_count, source_entity_keys
+                            FROM old_rows
+                            WHERE source = change_source;
+
+                            PERFORM planet_emit_collected_data_changed_statement(
+                                TG_OP,
+                                change_source,
+                                source_record_count,
+                                source_entity_keys
+                            );
+                        END LOOP;
+                    ELSIF TG_OP = 'UPDATE' THEN
+                        FOR change_source IN
+                            SELECT DISTINCT source FROM (
+                                SELECT source FROM new_rows
+                                UNION
+                                SELECT source FROM old_rows
+                            ) changed_sources
+                            WHERE source IS NOT NULL
+                        LOOP
+                            SELECT
+                                COUNT(*),
+                                ARRAY(
+                                    SELECT DISTINCT COALESCE(entity_key, source_id, id::text)
+                                    FROM (
+                                        SELECT id, source_id, entity_key, source FROM new_rows
+                                        UNION ALL
+                                        SELECT id, source_id, entity_key, source FROM old_rows
+                                    ) changed_rows
+                                    WHERE source = change_source
+                                    LIMIT 20
+                                )
+                            INTO source_record_count, source_entity_keys
+                            FROM (
+                                SELECT id, source_id, entity_key, source FROM new_rows
+                                UNION ALL
+                                SELECT id, source_id, entity_key, source FROM old_rows
+                            ) changed_rows
+                            WHERE source = change_source;
+
+                            PERFORM planet_emit_collected_data_changed_statement(
+                                TG_OP,
+                                change_source,
+                                source_record_count,
+                                source_entity_keys
+                            );
+                        END LOOP;
+                    END IF;
+
+                    RETURN NULL;
+                END;
+                $$ LANGUAGE plpgsql;
+                """
+            )
+        )
+        for statement in (
+            "DROP TRIGGER IF EXISTS tr_planet_collected_data_changed ON collected_data",
+            "DROP TRIGGER IF EXISTS tr_planet_collected_data_changed_insert ON collected_data",
+            "DROP TRIGGER IF EXISTS tr_planet_collected_data_changed_update ON collected_data",
+            "DROP TRIGGER IF EXISTS tr_planet_collected_data_changed_delete ON collected_data",
+            "DROP FUNCTION IF EXISTS planet_notify_collected_data_changed()",
+            """
+            CREATE TRIGGER tr_planet_collected_data_changed_insert
+            AFTER INSERT ON collected_data
+            REFERENCING NEW TABLE AS new_rows
+            FOR EACH STATEMENT
+            EXECUTE FUNCTION planet_notify_collected_data_changed_statement()
+            """,
+            """
+            CREATE TRIGGER tr_planet_collected_data_changed_update
+            AFTER UPDATE ON collected_data
+            REFERENCING OLD TABLE AS old_rows NEW TABLE AS new_rows
+            FOR EACH STATEMENT
+            EXECUTE FUNCTION planet_notify_collected_data_changed_statement()
+            """,
+            """
+            CREATE TRIGGER tr_planet_collected_data_changed_delete
+            AFTER DELETE ON collected_data
+            REFERENCING OLD TABLE AS old_rows
+            FOR EACH STATEMENT
+            EXECUTE FUNCTION planet_notify_collected_data_changed_statement()
+            """,
+        ):
+            await conn.execute(text(statement))
+        for table_name in (
+            "bgp_observations",
+            "bgp_anomalies",
+            "bgp_incidents",
+            "bgp_collector_locations",
+            "vessel_static",
+            "vessel_position",
+            "vessel_current_state",
+            "ais_raw_observations",
+            "ais_source_health",
+            "compute_center_locations",
+            "earth_interactables",
+            "earth_news_items",
+        ):
+            for statement in (
+                f"DROP TRIGGER IF EXISTS tr_planet_{table_name}_changed_insert ON {table_name}",
+                f"DROP TRIGGER IF EXISTS tr_planet_{table_name}_changed_update ON {table_name}",
+                f"DROP TRIGGER IF EXISTS tr_planet_{table_name}_changed_delete ON {table_name}",
+                f"""
+                CREATE TRIGGER tr_planet_{table_name}_changed_insert
+                AFTER INSERT ON {table_name}
+                REFERENCING NEW TABLE AS new_rows
+                FOR EACH STATEMENT
+                EXECUTE FUNCTION planet_notify_earth_table_changed_statement()
+                """,
+                f"""
+                CREATE TRIGGER tr_planet_{table_name}_changed_update
+                AFTER UPDATE ON {table_name}
+                REFERENCING OLD TABLE AS old_rows NEW TABLE AS new_rows
+                FOR EACH STATEMENT
+                EXECUTE FUNCTION planet_notify_earth_table_changed_statement()
+                """,
+                f"""
+                CREATE TRIGGER tr_planet_{table_name}_changed_delete
+                AFTER DELETE ON {table_name}
+                REFERENCING OLD TABLE AS old_rows
+                FOR EACH STATEMENT
+                EXECUTE FUNCTION planet_notify_earth_table_changed_statement()
+                """,
+            ):
+                await conn.execute(text(statement))
         await conn.execute(
             text(
                 """
@@ -267,7 +669,16 @@ async def init_db():
                 ADD COLUMN IF NOT EXISTS phase_message VARCHAR(255),
                 ADD COLUMN IF NOT EXISTS phase_current BIGINT,
                 ADD COLUMN IF NOT EXISTS phase_total BIGINT,
-                ADD COLUMN IF NOT EXISTS phase_unit VARCHAR(30)
+                ADD COLUMN IF NOT EXISTS phase_unit VARCHAR(30),
+                ADD COLUMN IF NOT EXISTS source VARCHAR(100),
+                ADD COLUMN IF NOT EXISTS task_type VARCHAR(30) NOT NULL DEFAULT 'collect',
+                ADD COLUMN IF NOT EXISTS payload JSONB NOT NULL DEFAULT '{}'::jsonb,
+                ADD COLUMN IF NOT EXISTS rollback_policy VARCHAR(40) NOT NULL DEFAULT 'keep_committed_batches',
+                ADD COLUMN IF NOT EXISTS dedupe_key VARCHAR(180),
+                ADD COLUMN IF NOT EXISTS worker_id VARCHAR(120),
+                ADD COLUMN IF NOT EXISTS locked_at TIMESTAMPTZ,
+                ADD COLUMN IF NOT EXISTS requested_cancel_at TIMESTAMPTZ,
+                ADD COLUMN IF NOT EXISTS cancel_reason TEXT
                 """
             )
         )
@@ -283,6 +694,17 @@ async def init_db():
                 """
             )
         )
+        await conn.execute(
+            text(
+                """
+                ALTER TABLE earth_interactables
+                ADD COLUMN IF NOT EXISTS altitude DOUBLE PRECISION,
+                ADD COLUMN IF NOT EXISTS revision INTEGER NOT NULL DEFAULT 1,
+                ADD COLUMN IF NOT EXISTS is_deleted BOOLEAN NOT NULL DEFAULT FALSE,
+                ADD COLUMN IF NOT EXISTS deleted_at TIMESTAMPTZ
+                """
+            )
+        )
         await conn.execute(
             text(
                 """
@@ -307,6 +729,48 @@ async def init_db():
                 """
             )
         )
+        await conn.execute(
+            text(
+                """
+                CREATE INDEX IF NOT EXISTS idx_earth_interactables_layer_deleted
+                ON earth_interactables (layer, is_deleted)
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                CREATE INDEX IF NOT EXISTS idx_earth_interactables_updated_at
+                ON earth_interactables (updated_at)
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                CREATE INDEX IF NOT EXISTS idx_collection_tasks_source_status
+                ON collection_tasks (source, status)
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                CREATE INDEX IF NOT EXISTS idx_collection_tasks_queue
+                ON collection_tasks (status, created_at, id)
+                WHERE status = 'queued'
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                CREATE INDEX IF NOT EXISTS idx_collection_tasks_dedupe
+                ON collection_tasks (dedupe_key)
+                WHERE dedupe_key IS NOT NULL
+                """
+            )
+        )
         await conn.execute(
             text(
                 """
@@ -323,6 +787,22 @@ async def init_db():
                 """
             )
         )
+        await conn.execute(
+            text(
+                """
+                CREATE INDEX IF NOT EXISTS idx_vessel_current_bbox
+                ON vessel_current_state (lon, lat)
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                CREATE INDEX IF NOT EXISTS idx_vessel_current_observed
+                ON vessel_current_state (observed_at DESC)
+                """
+            )
+        )
         await conn.execute(
             text(
                 """

backend/app/main.py

@@ -24,6 +24,11 @@ from app.services.earth_news_worker import (
     start_earth_news_target_worker,
     stop_earth_news_target_worker,
 )
+from app.services.earth_db_change_listener import (
+    start_earth_db_change_listener,
+    stop_earth_db_change_listener,
+)
+from app.services.data_jobs import start_data_job_worker, stop_data_job_worker
 
 
 configure_logging()
@@ -59,9 +64,13 @@ async def lifespan(app: FastAPI):
     start_scheduler()
     await sync_scheduler_with_datasources()
     broadcaster.start()
+    start_data_job_worker()
+    start_earth_db_change_listener()
     start_earth_news_target_worker()
     yield
     await stop_earth_news_target_worker()
+    await stop_earth_db_change_listener()
+    await stop_data_job_worker()
     broadcaster.stop()
     stop_scheduler()
 

backend/app/models/__init__.py

@@ -13,10 +13,11 @@ from app.models.compute_center_location import ComputeCenterLocationRecord
 from app.models.system_setting import SystemSetting
 from app.models.playground_session import PlaygroundSession
 from app.models.playground_message import PlaygroundMessage
-from app.models.system_log import SystemLog, AuditLog
+from app.models.system_log import AuditLog, ObservabilityEvent, ObservabilityEventGroup, SystemLog
 from app.models.vessel import AISConflictRecord, AISRawObservation, AISSourceHealth, VesselPosition, VesselStatic
 from app.models.datasource_mapping import DataSourceMappingTemplate
 from app.models.earth_news import EarthNewsItem
+from app.models.earth_interactable import EarthInteractable
 
 __all__ = [
     "User",
@@ -36,6 +37,8 @@ __all__ = [
     "ComputeCenterLocationRecord",
     "SystemLog",
     "AuditLog",
+    "ObservabilityEvent",
+    "ObservabilityEventGroup",
     "PlaygroundSession",
     "PlaygroundMessage",
     "VesselPosition",
@@ -45,4 +48,5 @@ __all__ = [
     "AISSourceHealth",
     "DataSourceMappingTemplate",
     "EarthNewsItem",
+    "EarthInteractable",
 ]

backend/app/models/alert.py

@@ -1,26 +1,12 @@
 from datetime import datetime
-from enum import Enum
-from typing import Optional
 
-from sqlalchemy import Column, Integer, String, DateTime, Text, ForeignKey, Enum as SQLEnum
-from sqlalchemy.orm import relationship
+from sqlalchemy import Column, Integer, String, DateTime, Text, Enum as SQLEnum
 
+from app.core.enums import AlertSeverity, AlertStatus
 from app.core.time import to_iso8601_utc
 from app.db.session import Base
 
 
-class AlertSeverity(str, Enum):
-    CRITICAL = "critical"
-    WARNING = "warning"
-    INFO = "info"
-
-
-class AlertStatus(str, Enum):
-    ACTIVE = "active"
-    ACKNOWLEDGED = "acknowledged"
-    RESOLVED = "resolved"
-
-
 class Alert(Base):
     __tablename__ = "alerts"
 

backend/app/models/bgp_anomaly.py

@@ -4,6 +4,7 @@ from datetime import datetime
 
 from sqlalchemy import Column, DateTime, Float, ForeignKey, Index, Integer, JSON, String, Text
 
+from app.core.enums import BGPStatus
 from app.core.time import to_iso8601_utc
 from app.db.session import Base
 
@@ -17,7 +18,7 @@ class BGPAnomaly(Base):
     source = Column(String(100), nullable=False, index=True)
     anomaly_type = Column(String(50), nullable=False, index=True)
     severity = Column(String(20), nullable=False, index=True)
-    status = Column(String(20), nullable=False, default="active", index=True)
+    status = Column(String(20), nullable=False, default=BGPStatus.ACTIVE.value, index=True)
     entity_key = Column(String(255), nullable=False, index=True)
     prefix = Column(String(64), nullable=True, index=True)
     origin_asn = Column(Integer, nullable=True, index=True)

backend/app/models/bgp_incident.py

@@ -4,6 +4,7 @@ from datetime import datetime
 
 from sqlalchemy import Column, DateTime, Float, ForeignKey, Index, Integer, JSON, String, Text
 
+from app.core.enums import BGPStatus
 from app.core.time import to_iso8601_utc
 from app.db.session import Base
 
@@ -20,7 +21,7 @@ class BGPIncident(Base):
     title = Column(String(255), nullable=False)
     summary = Column(Text, nullable=False)
     severity = Column(String(20), nullable=False, index=True)
-    status = Column(String(20), nullable=False, default="active", index=True)
+    status = Column(String(20), nullable=False, default=BGPStatus.ACTIVE.value, index=True)
     confidence = Column(Float, nullable=False, default=0.5)
     started_at = Column(DateTime(timezone=True), nullable=False, default=datetime.utcnow, index=True)
     ended_at = Column(DateTime(timezone=True), nullable=True)

backend/app/models/data_snapshot.py

@@ -1,6 +1,7 @@
 from sqlalchemy import Boolean, Column, DateTime, ForeignKey, Integer, JSON, String
 from sqlalchemy.sql import func
 
+from app.core.enums import SnapshotStatus
 from app.db.session import Base
 
 
@@ -16,7 +17,7 @@ class DataSnapshot(Base):
     started_at = Column(DateTime(timezone=True), server_default=func.now())
     completed_at = Column(DateTime(timezone=True), nullable=True)
     record_count = Column(Integer, default=0)
-    status = Column(String(20), nullable=False, default="running")
+    status = Column(String(20), nullable=False, default=SnapshotStatus.RUNNING.value)
     is_current = Column(Boolean, default=True, index=True)
     parent_snapshot_id = Column(Integer, ForeignKey("data_snapshots.id"), nullable=True, index=True)
     summary = Column(JSON, default={})

backend/app/models/datasource_mapping.py

@@ -3,6 +3,7 @@
 from sqlalchemy import Boolean, Column, DateTime, ForeignKey, Integer, JSON, String
 from sqlalchemy.sql import func
 
+from app.core.enums import MappingValidationStatus
 from app.db.session import Base
 
 
@@ -19,7 +20,7 @@ class DataSourceMappingTemplate(Base):
     target_schema = Column(String(80), nullable=False, index=True)
     mapping_json = Column(JSON, nullable=False, default={})
     sample_payload_hash = Column(String(64), nullable=True)
-    validation_status = Column(String(30), nullable=False, default="draft")
+    validation_status = Column(String(30), nullable=False, default=MappingValidationStatus.DRAFT.value)
     version = Column(Integer, nullable=False, default=1)
     is_active = Column(Boolean, nullable=False, default=False, index=True)
     created_at = Column(DateTime(timezone=True), server_default=func.now())

backend/app/models/earth_interactable.py

@@ -0,0 +1,30 @@
+"""Persistent Earth interactable objects."""
+
+from sqlalchemy import Boolean, Column, DateTime, Float, Index, Integer, JSON, String, Text
+from sqlalchemy.sql import func
+
+from app.db.session import Base
+
+
+class EarthInteractable(Base):
+    __tablename__ = "earth_interactables"
+
+    id = Column(String(160), primary_key=True)
+    layer = Column(String(80), nullable=False, default="interactables", index=True)
+    kind = Column(String(80), nullable=False, default="default", index=True)
+    label = Column(String(255), nullable=False, default="")
+    description = Column(Text, nullable=False, default="")
+    latitude = Column(Float, nullable=False)
+    longitude = Column(Float, nullable=False)
+    altitude = Column(Float, nullable=True)
+    revision = Column(Integer, nullable=False, default=1)
+    properties = Column(JSON, nullable=False, default=dict)
+    is_deleted = Column(Boolean, nullable=False, default=False, index=True)
+    created_at = Column(DateTime(timezone=True), server_default=func.now(), nullable=False)
+    updated_at = Column(DateTime(timezone=True), server_default=func.now(), onupdate=func.now())
+    deleted_at = Column(DateTime(timezone=True), nullable=True, index=True)
+
+    __table_args__ = (
+        Index("idx_earth_interactables_layer_deleted", "layer", "is_deleted"),
+        Index("idx_earth_interactables_updated_at", "updated_at"),
+    )

backend/app/models/playground_message.py

@@ -1,6 +1,7 @@
 from sqlalchemy import JSON, Boolean, Column, DateTime, ForeignKey, Integer, String, Text
 from sqlalchemy.sql import func
 
+from app.core.enums import PlaygroundMessageKind, PlaygroundMessageStatus
 from app.db.session import Base
 
 
@@ -13,8 +14,8 @@ class PlaygroundMessage(Base):
     user_id = Column(Integer, ForeignKey("users.id", ondelete="CASCADE"), nullable=False, index=True)
     parent_message_id = Column(Integer, ForeignKey("playground_messages.id", ondelete="SET NULL"), nullable=True)
     role = Column(String(20), nullable=False)
-    kind = Column(String(20), nullable=False, default="message")
-    status = Column(String(20), nullable=False, default="done")
+    kind = Column(String(20), nullable=False, default=PlaygroundMessageKind.MESSAGE.value)
+    status = Column(String(20), nullable=False, default=PlaygroundMessageStatus.DONE.value)
     title = Column(String(255), nullable=True)
     content = Column(Text, nullable=False, default="")
     thinking_content = Column(Text, nullable=False, default="")

backend/app/models/system_log.py

@@ -38,3 +38,46 @@ class AuditLog(Base):
     ip = Column(String(64), nullable=True)
     details = Column(JSON, nullable=False, default=dict)
     created_at = Column(DateTime(timezone=True), server_default=func.now())
+
+
+class ObservabilityEvent(Base):
+    __tablename__ = "observability_events"
+
+    id = Column(Integer, primary_key=True, autoincrement=True)
+    occurred_at = Column(DateTime(timezone=True), server_default=func.now(), index=True)
+    source = Column(String(50), nullable=False, index=True)
+    service = Column(String(50), nullable=True, index=True)
+    module = Column(String(120), nullable=True, index=True)
+    category = Column(String(80), nullable=True, index=True)
+    event = Column(String(160), nullable=True, index=True)
+    level = Column(String(20), nullable=False, index=True)
+    message = Column(Text, nullable=False)
+    fingerprint = Column(String(80), nullable=False, index=True)
+    request_id = Column(String(64), nullable=True, index=True)
+    trace_id = Column(String(64), nullable=True, index=True)
+    task_id = Column(String(120), nullable=True, index=True)
+    source_ref_id = Column(String(120), nullable=True, index=True)
+    provider = Column(String(120), nullable=True, index=True)
+    user_id = Column(Integer, nullable=True, index=True)
+    context = Column(JSON, nullable=False, default=dict)
+    occurrence_count = Column(Integer, nullable=False, default=1)
+    created_at = Column(DateTime(timezone=True), server_default=func.now())
+
+
+class ObservabilityEventGroup(Base):
+    __tablename__ = "observability_event_groups"
+
+    fingerprint = Column(String(80), primary_key=True)
+    source = Column(String(50), nullable=False, index=True)
+    service = Column(String(50), nullable=True, index=True)
+    module = Column(String(120), nullable=True, index=True)
+    category = Column(String(80), nullable=True, index=True)
+    event = Column(String(160), nullable=True, index=True)
+    last_level = Column(String(20), nullable=False, index=True)
+    sample_message = Column(Text, nullable=False)
+    sample_detail = Column(Text, nullable=True)
+    affected_sources = Column(JSON, nullable=False, default=list)
+    count = Column(Integer, nullable=False, default=0)
+    first_seen_at = Column(DateTime(timezone=True), nullable=False, index=True)
+    last_seen_at = Column(DateTime(timezone=True), nullable=False, index=True)
+    updated_at = Column(DateTime(timezone=True), server_default=func.now(), onupdate=func.now())

backend/app/models/task.py

@@ -1,8 +1,9 @@
-"""Collection Task model"""
+"""Datasource job model."""
 
-from sqlalchemy import BigInteger, Column, DateTime, Integer, String, Text, Float
+from sqlalchemy import BigInteger, Column, DateTime, Float, Integer, JSON, String, Text
 from sqlalchemy.sql import func
 
+from app.core.enums import JobStatus, JobType, RollbackPolicy
 from app.db.session import Base
 
 
@@ -11,8 +12,10 @@ class CollectionTask(Base):
 
     id = Column(Integer, primary_key=True, autoincrement=True)
     datasource_id = Column(Integer, nullable=False, index=True)
-    status = Column(String(20), nullable=False)  # pending, running, success, failed, cancelled
-    phase = Column(String(30), default="queued")
+    source = Column(String(100), nullable=True, index=True)
+    task_type = Column(String(30), nullable=False, default=JobType.COLLECT.value, index=True)
+    status = Column(String(20), nullable=False)  # queued, running, cancelling, success, failed, cancelled
+    phase = Column(String(30), default=JobStatus.QUEUED.value)
     phase_progress = Column(Float)
     phase_message = Column(String(255))
     phase_current = Column(BigInteger)
@@ -24,6 +27,13 @@ class CollectionTask(Base):
     total_records = Column(Integer, default=0)  # Total records to process
     progress = Column(Float, default=0.0)  # Progress percentage (0-100)
     error_message = Column(Text)
+    payload = Column(JSON, default=dict)
+    rollback_policy = Column(String(40), nullable=False, default=RollbackPolicy.KEEP_COMMITTED_BATCHES.value)
+    dedupe_key = Column(String(180), nullable=True, index=True)
+    worker_id = Column(String(120), nullable=True, index=True)
+    locked_at = Column(DateTime(timezone=True), nullable=True, index=True)
+    requested_cancel_at = Column(DateTime(timezone=True), nullable=True)
+    cancel_reason = Column(Text)
     created_at = Column(DateTime(timezone=True), server_default=func.now())
 
     def __repr__(self):

backend/app/models/user.py

@@ -1,6 +1,7 @@
 from sqlalchemy import Boolean, Column, DateTime, Integer, JSON, String
 from sqlalchemy.sql import func
 
+from app.core.enums import UserRole
 from app.db.session import Base
 
 
@@ -11,7 +12,7 @@ class User(Base):
     username = Column(String(50), unique=True, index=True, nullable=False)
     email = Column(String(255), unique=True, index=True, nullable=False)
     password_hash = Column(String(255), nullable=False)
-    role = Column(String(20), default="viewer")
+    role = Column(String(20), default=UserRole.VIEWER.value)
     gatekeeper_groups = Column(JSON, default=list)
     is_active = Column(Boolean, default=True)
     email_verified = Column(Boolean, default=False, nullable=False)

backend/app/models/vessel.py

@@ -3,6 +3,7 @@
 from sqlalchemy import BigInteger, Column, DateTime, Float, Index, Integer, JSON, SmallInteger, String
 from sqlalchemy.sql import func
 
+from app.core.enums import ConnectionState
 from app.core.time import to_iso8601_utc
 from app.db.session import Base
 
@@ -75,6 +76,67 @@ class VesselPosition(Base):
         }
 
 
+class VesselCurrentState(Base):
+    """Latest renderable state for one vessel, independent from AIS history."""
+
+    __tablename__ = "vessel_current_state"
+
+    mmsi = Column(BigInteger, primary_key=True)
+    lat = Column(Float, nullable=False)
+    lon = Column(Float, nullable=False)
+    sog = Column(Float, nullable=True)
+    cog = Column(Float, nullable=True)
+    heading = Column(SmallInteger, nullable=True)
+    nav_status = Column(SmallInteger, nullable=True, index=True)
+    name = Column(String(128), nullable=True)
+    callsign = Column(String(16), nullable=True)
+    vessel_type = Column(SmallInteger, nullable=True, index=True)
+    vessel_type_name = Column(String(64), nullable=True, index=True)
+    flag = Column(String(4), nullable=True, index=True)
+    length = Column(Float, nullable=True)
+    width = Column(Float, nullable=True)
+    draught = Column(Float, nullable=True)
+    imo = Column(BigInteger, nullable=True)
+    source = Column(String(100), nullable=False, index=True)
+    observed_at = Column(DateTime(timezone=True), nullable=False)
+    field_sources = Column(JSON, default=dict)
+    selected_reasons = Column(JSON, default=dict)
+    source_summary = Column(JSON, default=dict)
+    quality_flags = Column(JSON, default=list)
+    updated_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now())
+
+    __table_args__ = (
+        Index("idx_vessel_current_bbox", "lon", "lat"),
+        Index("idx_vessel_current_observed", "observed_at"),
+    )
+
+    def to_dict(self) -> dict:
+        return {
+            "mmsi": self.mmsi,
+            "lat": self.lat,
+            "lon": self.lon,
+            "sog": self.sog,
+            "cog": self.cog,
+            "heading": self.heading,
+            "nav_status": self.nav_status,
+            "name": self.name,
+            "callsign": self.callsign,
+            "vessel_type": self.vessel_type,
+            "vessel_type_name": self.vessel_type_name,
+            "flag": self.flag,
+            "length": self.length,
+            "width": self.width,
+            "draught": self.draught,
+            "imo": self.imo,
+            "source": self.source,
+            "received_at": self.observed_at,
+            "field_sources": self.field_sources or {},
+            "selected_reasons": self.selected_reasons or {},
+            "source_summary": self.source_summary or {},
+            "quality_flags": self.quality_flags or [],
+        }
+
+
 class AISRawObservation(Base):
     """Source-level AIS fact before aggregation and conflict resolution."""
 
@@ -165,7 +227,7 @@ class AISSourceHealth(Base):
     __tablename__ = "ais_source_health"
 
     source = Column(String(100), primary_key=True)
-    connection_state = Column(String(32), nullable=False, default="disconnected", index=True)
+    connection_state = Column(String(32), nullable=False, default=ConnectionState.DISCONNECTED.value, index=True)
     last_seen_at = Column(DateTime(timezone=True), nullable=True, index=True)
     last_success_at = Column(DateTime(timezone=True), nullable=True, index=True)
     last_error = Column(String(500), nullable=True)

backend/app/schemas/ai.py

@@ -2,6 +2,7 @@ from typing import Any
 
 from pydantic import BaseModel, Field
 
+from app.core.enums import PlaygroundMessageKind, PlaygroundMessageRole, PlaygroundMessageStatus
 
 class AIContentBlock(BaseModel):
     type: str
@@ -110,9 +111,9 @@ class PlaygroundSessionUpsertRequest(BaseModel):
 
 class PlaygroundMessageRecord(BaseModel):
     id: str
-    role: str
-    kind: str = "message"
-    status: str = "done"
+    role: PlaygroundMessageRole
+    kind: PlaygroundMessageKind = PlaygroundMessageKind.MESSAGE
+    status: PlaygroundMessageStatus = PlaygroundMessageStatus.DONE
     title: str | None = None
     content: str = ""
     thinking_content: str = ""

backend/app/schemas/user.py

@@ -3,6 +3,7 @@ from typing import Optional
 
 from pydantic import BaseModel, EmailStr, Field
 
+from app.core.enums import OtpPurpose, UserRole
 
 class UserBase(BaseModel):
     username: str
@@ -11,13 +12,13 @@ class UserBase(BaseModel):
 
 class UserCreate(UserBase):
     password: str = Field(..., min_length=8)
-    role: str = "viewer"
+    role: UserRole = UserRole.VIEWER
     gatekeeper_groups: list[str] = Field(default_factory=list)
 
 
 class UserUpdate(BaseModel):
     email: Optional[EmailStr] = None
-    role: Optional[str] = None
+    role: Optional[UserRole] = None
     gatekeeper_groups: Optional[list[str]] = None
     is_active: Optional[bool] = None
 
@@ -59,7 +60,7 @@ class VerifyEmailRequest(BaseModel):
 
 class ResendCodeRequest(BaseModel):
     email: EmailStr
-    purpose: str = Field(default="register", pattern="^(register|verify_email|reset_password)$")
+    purpose: OtpPurpose = OtpPurpose.REGISTER
 
 
 class ForgotPasswordRequest(BaseModel):

backend/app/services/ai_client.py

@@ -2,18 +2,24 @@ from __future__ import annotations
 
 import asyncio
 import json
+from time import perf_counter
 
 import httpx
 from fastapi import Depends, HTTPException, status
 from sqlalchemy.ext.asyncio import AsyncSession
 
 from app.core.config import settings
+from app.core.logging import get_logger
 from app.db.session import get_db
 from app.schemas.ai import (
     AIProviderStatusResponse,
     SituationalAnalysisRequest,
     SituationalAnalysisResponse,
 )
+from app.services.business_logs import emit_business_log, exception_context
+
+
+logger = get_logger(__name__, service="ai")
 
 
 class AIProviderClient:
@@ -64,7 +70,19 @@ class AIProviderClient:
         return headers
 
     async def get_status(self, request_id: str | None = None) -> AIProviderStatusResponse:
+        context = self._base_log_context(operation="status")
         if not self.service_url:
+            await emit_business_log(
+                logger,
+                event="ai.provider.status.failed",
+                message="AI provider status skipped because service URL is not configured",
+                category="ai",
+                level="warning",
+                service="ai",
+                module=__name__,
+                request_id=request_id,
+                context={**context, "status": "unconfigured"},
+            )
             return AIProviderStatusResponse(
                 provider="unconfigured",
                 enabled=False,
@@ -73,27 +91,133 @@ class AIProviderClient:
                 base_url=None,
             )
 
-        data = await self._request("GET", "/v1/provider/status", request_id=request_id)
-        return AIProviderStatusResponse.model_validate(data)
+        started_at = perf_counter()
+        await emit_business_log(
+            logger,
+            event="ai.provider.status.start",
+            message="AI provider status request started",
+            category="ai",
+            service="ai",
+            module=__name__,
+            request_id=request_id,
+            context=context,
+        )
+        try:
+            data = await self._request("GET", "/v1/provider/status", request_id=request_id, operation="status")
+            result = AIProviderStatusResponse.model_validate(data)
+            await emit_business_log(
+                logger,
+                event="ai.provider.status.success",
+                message="AI provider status request completed",
+                category="ai",
+                service="ai",
+                module=__name__,
+                request_id=request_id,
+                context={
+                    **context,
+                    "status": "success",
+                    "duration_ms": self._duration_ms(started_at),
+                    "result_provider": result.provider,
+                    "result_model": result.model,
+                    "configured": result.configured,
+                    "enabled": result.enabled,
+                },
+            )
+            return result
+        except Exception as exc:
+            await emit_business_log(
+                logger,
+                event="ai.provider.status.failed",
+                message="AI provider status request failed",
+                category="ai",
+                level="error",
+                service="ai",
+                module=__name__,
+                request_id=request_id,
+                context=exception_context(exc, {**context, "status": "failed", "duration_ms": self._duration_ms(started_at)}),
+            )
+            raise
 
     async def analyze(
         self,
         payload: SituationalAnalysisRequest,
         request_id: str | None = None,
     ) -> SituationalAnalysisResponse:
+        context = self._base_log_context(
+            operation="analyze",
+            preferred_model=payload.preferred_model,
+            input_summary=self._summarize_analysis_payload(payload),
+        )
         if not self.service_url:
+            await emit_business_log(
+                logger,
+                event="ai.provider.analyze.failed",
+                message="AI provider analyze skipped because service URL is not configured",
+                category="ai",
+                level="warning",
+                service="ai",
+                module=__name__,
+                request_id=request_id,
+                context={**context, "status": "unconfigured"},
+            )
             raise HTTPException(
                 status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
                 detail="AI provider service URL is not configured.",
             )
 
-        data = await self._request(
-            "POST",
-            "/v1/analyze",
-            json=payload.model_dump(),
+        started_at = perf_counter()
+        await emit_business_log(
+            logger,
+            event="ai.provider.analyze.start",
+            message="AI provider analyze request started",
+            category="ai",
+            service="ai",
+            module=__name__,
             request_id=request_id,
+            context=context,
         )
-        return SituationalAnalysisResponse.model_validate(data)
+        try:
+            data = await self._request(
+                "POST",
+                "/v1/analyze",
+                json=payload.model_dump(),
+                request_id=request_id,
+                operation="analyze",
+                payload_summary=context["input_summary"],
+            )
+            result = SituationalAnalysisResponse.model_validate(data)
+            await emit_business_log(
+                logger,
+                event="ai.provider.analyze.success",
+                message="AI provider analyze request completed",
+                category="ai",
+                service="ai",
+                module=__name__,
+                request_id=request_id,
+                context={
+                    **context,
+                    "status": "success",
+                    "duration_ms": self._duration_ms(started_at),
+                    "result_provider": result.provider,
+                    "result_model": result.model,
+                    "content_block_count": len(result.content_blocks or []),
+                    "thinking_block_count": len(result.thinking_blocks or []),
+                },
+            )
+            return result
+        except Exception as exc:
+            await emit_business_log(
+                logger,
+                event="ai.provider.analyze.failed",
+                message="AI provider analyze request failed",
+                category="ai",
+                level="error",
+                service="ai",
+                module=__name__,
+                request_id=request_id,
+                context=exception_context(exc, {**context, "status": "failed", "duration_ms": self._duration_ms(started_at)}),
+            )
+            raise
 
     async def _request(
         self,
@@ -101,9 +225,12 @@ class AIProviderClient:
         path: str,
         json: dict | None = None,
         request_id: str | None = None,
+        operation: str = "request",
+        payload_summary: dict | None = None,
     ) -> dict:
         last_error: Exception | None = None
         for attempt in range(1, self.retry_attempts + 1):
+            attempt_started_at = perf_counter()
             try:
                 async with httpx.AsyncClient(timeout=self.timeout) as client:
                     response = await client.request(
@@ -117,6 +244,15 @@ class AIProviderClient:
             except httpx.HTTPStatusError as exc:
                 last_error = exc
                 if attempt < self.retry_attempts and exc.response.status_code >= 500:
+                    await self._log_retry(
+                        operation=operation,
+                        request_id=request_id,
+                        attempt=attempt,
+                        status_code=exc.response.status_code,
+                        duration_ms=self._duration_ms(attempt_started_at),
+                        error=exc,
+                        payload_summary=payload_summary,
+                    )
                     await asyncio.sleep(0.3 * attempt)
                     continue
                 detail = exc.response.text or "AI provider service returned an error"
@@ -127,6 +263,14 @@ class AIProviderClient:
             except httpx.HTTPError as exc:
                 last_error = exc
                 if attempt < self.retry_attempts:
+                    await self._log_retry(
+                        operation=operation,
+                        request_id=request_id,
+                        attempt=attempt,
+                        duration_ms=self._duration_ms(attempt_started_at),
+                        error=exc,
+                        payload_summary=payload_summary,
+                    )
                     await asyncio.sleep(0.3 * attempt)
                     continue
                 raise HTTPException(
@@ -139,6 +283,71 @@ class AIProviderClient:
             detail=f"AI provider service request failed: {last_error}",
         )
 
+    def _base_log_context(self, **extra: object) -> dict:
+        llm_provider_apis = self.llm_config.get("model_provider_apis")
+        return {
+            "provider": self.llm_config.get("provider") or "",
+            "provider_api": self.llm_config.get("provider_api") or "",
+            "model": self.llm_config.get("model") or "",
+            "base_url_configured": bool(self.llm_config.get("base_url")),
+            "service_url_configured": bool(self.service_url),
+            "timeout_seconds": self.timeout,
+            "retry_attempts": self.retry_attempts,
+            "model_provider_api_count": len(llm_provider_apis or {}) if isinstance(llm_provider_apis, dict) else 0,
+            **extra,
+        }
+
+    @staticmethod
+    def _duration_ms(started_at: float) -> int:
+        return int((perf_counter() - started_at) * 1000)
+
+    @staticmethod
+    def _summarize_analysis_payload(payload: SituationalAnalysisRequest) -> dict:
+        context = payload.context if isinstance(payload.context, dict) else {}
+        thinking = payload.thinking if isinstance(payload.thinking, dict) else payload.thinking
+        return {
+            "title_length": len(payload.title or ""),
+            "objective_length": len(payload.objective or ""),
+            "observation_count": len(payload.observations or []),
+            "constraint_count": len(payload.constraints or []),
+            "has_system_prompt": bool(payload.system_prompt),
+            "thinking_enabled": bool(thinking),
+            "context_keys": sorted(str(key) for key in context.keys()),
+        }
+
+    async def _log_retry(
+        self,
+        *,
+        operation: str,
+        request_id: str | None,
+        attempt: int,
+        duration_ms: int,
+        error: BaseException,
+        status_code: int | None = None,
+        payload_summary: dict | None = None,
+    ) -> None:
+        await emit_business_log(
+            logger,
+            event=f"ai.provider.{operation}.retry",
+            message="AI provider request will retry",
+            category="ai",
+            level="warning",
+            service="ai",
+            module=__name__,
+            request_id=request_id,
+            context=exception_context(
+                error,
+                {
+                    **self._base_log_context(operation=operation),
+                    "attempt": attempt,
+                    "next_attempt": attempt + 1,
+                    "status_code": status_code,
+                    "duration_ms": duration_ms,
+                    "input_summary": payload_summary,
+                },
+            ),
+        )
+
 
 async def get_ai_provider_client(db: AsyncSession = Depends(get_db)) -> AIProviderClient:
     from app.api.v1.settings import get_runtime_ai_provider_config

backend/app/services/ai_tools/web_fetch.py

@@ -1,11 +1,18 @@
 from __future__ import annotations
 
 import hashlib
+from time import perf_counter
+from urllib.parse import urlparse
 
 import httpx
 from bs4 import BeautifulSoup
 
+from app.core.logging import get_logger
 from app.services.ai_tools.schemas import FetchedEvidence
+from app.services.business_logs import emit_business_log, exception_context
+
+
+logger = get_logger(__name__, service="ai_tool")
 
 
 class WebFetchError(RuntimeError):
@@ -29,8 +36,33 @@ async def fetch_url_evidence(
     timeout_seconds: int = 20,
     max_bytes: int = 1_500_000,
 ) -> FetchedEvidence:
+    started_at = perf_counter()
     if not url:
+        await emit_business_log(
+            logger,
+            event="ai_tool.web_fetch.failed",
+            message="WebFetch failed because URL is empty",
+            category="ai_tool",
+            level="warning",
+            service="ai_tool",
+            module=__name__,
+            context={"reason": "empty_url"},
+        )
         raise WebFetchError("url is required")
+    request_host = urlparse(url).netloc
+    await emit_business_log(
+        logger,
+        event="ai_tool.web_fetch.start",
+        message="WebFetch request started",
+        category="ai_tool",
+        service="ai_tool",
+        module=__name__,
+        context={
+            "url_host": request_host,
+            "timeout_seconds": timeout_seconds,
+            "max_bytes": max_bytes,
+        },
+    )
     try:
         async with httpx.AsyncClient(
             timeout=timeout_seconds,
@@ -41,10 +73,45 @@ async def fetch_url_evidence(
             response.raise_for_status()
             content = response.content[:max_bytes]
     except httpx.HTTPError as exc:
+        await emit_business_log(
+            logger,
+            event="ai_tool.web_fetch.failed",
+            message="WebFetch request failed",
+            category="ai_tool",
+            level="error",
+            service="ai_tool",
+            module=__name__,
+            context=exception_context(
+                exc,
+                {
+                    "url_host": request_host,
+                    "status": "failed",
+                    "duration_ms": int((perf_counter() - started_at) * 1000),
+                },
+            ),
+        )
         raise WebFetchError(f"failed to fetch page: {exc}") from exc
 
     title, text = _extract_title_and_text(content.decode(response.encoding or "utf-8", errors="ignore"))
     content_hash = hashlib.sha256(text.encode("utf-8")).hexdigest()
+    await emit_business_log(
+        logger,
+        event="ai_tool.web_fetch.success",
+        message="WebFetch request completed",
+        category="ai_tool",
+        service="ai_tool",
+        module=__name__,
+        context={
+            "url_host": request_host,
+            "final_url_host": urlparse(str(response.url)).netloc,
+            "status": "success",
+            "status_code": response.status_code,
+            "bytes_read": len(content),
+            "content_hash": content_hash,
+            "duration_ms": int((perf_counter() - started_at) * 1000),
+            "extractor": "beautifulsoup_basic",
+        },
+    )
     return FetchedEvidence(
         url=url,
         final_url=str(response.url),
@@ -53,4 +120,3 @@ async def fetch_url_evidence(
         content_hash=content_hash,
         extractor="beautifulsoup_basic",
     )
-

backend/app/services/ai_tools/web_search.py

@@ -1,11 +1,18 @@
 from __future__ import annotations
 
 from copy import deepcopy
+import hashlib
+from time import perf_counter
 from typing import Any
 
 import httpx
 
+from app.core.logging import get_logger
 from app.services.ai_tools.schemas import SearchEvidence, WebSearchConfig, WebSearchProviderConfig
+from app.services.business_logs import emit_business_log, exception_context
+
+
+logger = get_logger(__name__, service="ai_tool")
 
 
 WEB_SEARCH_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
@@ -120,29 +127,108 @@ class WebSearchClient:
         domains: list[str] | None = None,
         freshness_days: int | None = None,
     ) -> list[SearchEvidence]:
+        started_at = perf_counter()
         if not self.config.enabled:
+            await emit_business_log(
+                logger,
+                event="ai_tool.web_search.unavailable",
+                message="WebSearch skipped because integration is disabled",
+                category="ai_tool",
+                level="warning",
+                service="ai_tool",
+                module=__name__,
+                context={"provider": self.config.default_provider, "reason": "disabled"},
+            )
             raise WebSearchConfigurationError("WebSearch is disabled.")
         provider_config = self.config.active_provider_config
         provider = normalize_web_search_provider(provider_config.provider)
         if provider != "searxng" and not provider_config.api_key:
+            await emit_business_log(
+                logger,
+                event="ai_tool.web_search.unavailable",
+                message="WebSearch skipped because API key is not configured",
+                category="ai_tool",
+                level="warning",
+                service="ai_tool",
+                module=__name__,
+                context={"provider": provider, "reason": "missing_api_key"},
+            )
             raise WebSearchConfigurationError(f"{provider} API key is not configured.")
         query = " ".join(str(query or "").split())
         if not query:
+            await emit_business_log(
+                logger,
+                event="ai_tool.web_search.failed",
+                message="WebSearch failed because query is empty",
+                category="ai_tool",
+                level="warning",
+                service="ai_tool",
+                module=__name__,
+                context={"provider": provider, "reason": "empty_query"},
+            )
             raise WebSearchConfigurationError("search query is required.")
         limit = max_results or provider_config.max_results
-        if provider == "tavily":
-            return await self._search_tavily(provider_config, query, limit, domains, freshness_days)
-        if provider == "brave":
-            return await self._search_brave(provider_config, query, limit, domains)
-        if provider == "serpapi":
-            return await self._search_serpapi(provider_config, query, limit)
-        if provider == "exa":
-            return await self._search_exa(provider_config, query, limit, domains)
-        if provider == "firecrawl":
-            return await self._search_firecrawl(provider_config, query, limit)
-        if provider == "searxng":
-            return await self._search_searxng(provider_config, query, limit, domains)
-        raise WebSearchConfigurationError(f"Unsupported web search provider: {provider}")
+        context = {
+            "provider": provider,
+            "query_hash": hashlib.sha256(query.encode("utf-8")).hexdigest(),
+            "query_length": len(query),
+            "max_results": limit,
+            "domain_count": len(domains or []),
+            "freshness_days": freshness_days,
+        }
+        await emit_business_log(
+            logger,
+            event="ai_tool.web_search.start",
+            message="WebSearch request started",
+            category="ai_tool",
+            service="ai_tool",
+            module=__name__,
+            context=context,
+        )
+        try:
+            if provider == "tavily":
+                results = await self._search_tavily(provider_config, query, limit, domains, freshness_days)
+            elif provider == "brave":
+                results = await self._search_brave(provider_config, query, limit, domains)
+            elif provider == "serpapi":
+                results = await self._search_serpapi(provider_config, query, limit)
+            elif provider == "exa":
+                results = await self._search_exa(provider_config, query, limit, domains)
+            elif provider == "firecrawl":
+                results = await self._search_firecrawl(provider_config, query, limit)
+            elif provider == "searxng":
+                results = await self._search_searxng(provider_config, query, limit, domains)
+            else:
+                raise WebSearchConfigurationError(f"Unsupported web search provider: {provider}")
+            event = "ai_tool.web_search.success" if results else "ai_tool.web_search.empty"
+            await emit_business_log(
+                logger,
+                event=event,
+                message="WebSearch request completed" if results else "WebSearch returned no results",
+                category="ai_tool",
+                level="info" if results else "warning",
+                service="ai_tool",
+                module=__name__,
+                context={
+                    **context,
+                    "status": "success" if results else "empty",
+                    "result_count": len(results),
+                    "duration_ms": int((perf_counter() - started_at) * 1000),
+                },
+            )
+            return results
+        except Exception as exc:
+            await emit_business_log(
+                logger,
+                event="ai_tool.web_search.failed",
+                message="WebSearch request failed",
+                category="ai_tool",
+                level="error",
+                service="ai_tool",
+                module=__name__,
+                context=exception_context(exc, {**context, "status": "failed", "duration_ms": int((perf_counter() - started_at) * 1000)}),
+            )
+            raise
 
     async def test_connection(self) -> list[SearchEvidence]:
         return await self.search("Planet WebSearch connectivity test", max_results=1)
@@ -388,4 +474,3 @@ def _float_or_none(value: Any) -> float | None:
         return float(value)
     except (TypeError, ValueError):
         return None
-

backend/app/services/bgp_detectors.py

@@ -6,6 +6,7 @@ from collections import Counter, defaultdict
 from datetime import UTC, datetime
 from typing import Any
 
+from app.core.enums import BGPStatus
 from app.models.bgp_anomaly import BGPAnomaly
 
 
@@ -127,7 +128,7 @@ def detect_origin_change_anomalies(
                     source=source,
                     anomaly_type=anomaly_type,
                     severity=severity,
-                    status="active",
+                    status=BGPStatus.ACTIVE.value,
                     entity_key=f"{anomaly_type}:{prefix}:{new_origin}",
                     prefix=prefix,
                     origin_asn=sorted(historic)[0] if historic else None,
@@ -197,7 +198,7 @@ def detect_more_specific_burst_anomalies(
                 source=source,
                 anomaly_type="more_specific_burst",
                 severity="high",
-                status="active",
+                status=BGPStatus.ACTIVE.value,
                 entity_key=f"more_specific_burst:{root_prefix}:{len(unique_prefixes)}:{len(related_collectors)}",
                 prefix=sample.get("prefix"),
                 origin_asn=sample.get("origin_asn"),
@@ -267,7 +268,7 @@ def detect_mass_withdrawal_anomalies(
                 source=source,
                 anomaly_type="mass_withdrawal",
                 severity=severity,
-                status="active",
+                status=BGPStatus.ACTIVE.value,
                 entity_key=f"mass_withdrawal:{prefix}:{origin_asn}:{len(related_collectors)}:{count}",
                 prefix=prefix,
                 origin_asn=origin_asn,
@@ -354,7 +355,7 @@ def detect_route_leak_anomalies(
                 source=source,
                 anomaly_type="route_leak_candidate",
                 severity="high" if max_path_length >= dominant_length + 3 else "medium",
-                status="active",
+                status=BGPStatus.ACTIVE.value,
                 entity_key=f"route_leak_candidate:{prefix}:{max_path_length}:{len(related_collectors)}",
                 prefix=prefix,
                 origin_asn=sample_metadata.get("origin_asn"),
@@ -435,7 +436,7 @@ def detect_path_flap_anomalies(
                 source=source,
                 anomaly_type="path_flap",
                 severity=severity,
-                status="active",
+                status=BGPStatus.ACTIVE.value,
                 entity_key=f"path_flap:{prefix}:{transitions}:{len(distinct_paths)}",
                 prefix=prefix,
                 origin_asn=sample_metadata.get("origin_asn"),

backend/app/services/bgp_incidents.py

@@ -9,6 +9,7 @@ from sqlalchemy import select
 from sqlalchemy.ext.asyncio import AsyncSession
 
 from app.core.collected_data_fields import get_record_field
+from app.core.enums import BGPStatus
 from app.models.bgp_anomaly import BGPAnomaly
 from app.models.bgp_incident import BGPIncident
 from app.models.collected_data import CollectedData
@@ -290,7 +291,7 @@ async def create_bgp_incidents_for_anomalies(
             existing.title = title
             existing.summary = summary
             existing.severity = severity
-            existing.status = "active"
+            existing.status = BGPStatus.ACTIVE.value
             existing.confidence = confidence
             existing.started_at = primary.started_at or existing.started_at or datetime.now(UTC)
             existing.ended_at = None
@@ -313,7 +314,7 @@ async def create_bgp_incidents_for_anomalies(
                 title=title,
                 summary=summary,
                 severity=severity,
-                status="active",
+                status=BGPStatus.ACTIVE.value,
                 confidence=confidence,
                 started_at=primary.started_at or datetime.now(UTC),
                 affected_prefixes=prefixes,

backend/app/services/business_logs.py

@@ -0,0 +1,117 @@
+from __future__ import annotations
+
+import asyncio
+from collections.abc import Mapping
+from typing import Any
+
+from app.core.logging import PlanetLoggerAdapter, sanitize_log_value
+from app.core.request_context import get_request_id
+from app.services.persistent_logs import record_system_log
+
+
+LEVEL_METHODS = {
+    "debug": "debug_event",
+    "info": "info_event",
+    "warning": "warning_event",
+    "error": "error_event",
+}
+
+
+def normalize_business_level(level: str | None) -> str:
+    normalized = str(level or "info").strip().lower()
+    if normalized in {"warn", "warning"}:
+        return "warning"
+    if normalized in {"err", "error", "critical", "fatal"}:
+        return "error"
+    if normalized == "debug":
+        return "debug"
+    return "info"
+
+
+def build_business_context(
+    context: Mapping[str, Any] | None = None,
+    **fields: Any,
+) -> dict[str, Any]:
+    payload = dict(context or {})
+    for key, value in fields.items():
+        if value is not None:
+            payload[key] = value
+    return sanitize_log_value(payload)
+
+
+async def emit_business_log(
+    logger: PlanetLoggerAdapter,
+    *,
+    event: str,
+    message: str,
+    category: str,
+    level: str = "info",
+    source: str = "backend",
+    service: str | None = None,
+    module: str | None = None,
+    request_id: str | None = None,
+    user_id: int | None = None,
+    context: Mapping[str, Any] | None = None,
+) -> None:
+    normalized_level = normalize_business_level(level)
+    safe_context = build_business_context(context)
+    log_method = getattr(logger, LEVEL_METHODS[normalized_level])
+    log_method(message, event=event, context=safe_context)
+    await record_system_log(
+        source=source,
+        level=normalized_level,
+        message=message,
+        service=service,
+        module=module,
+        event=event,
+        request_id=request_id or get_request_id(),
+        user_id=user_id,
+        category=category,
+        context=safe_context,
+    )
+
+
+def emit_business_log_background(
+    logger: PlanetLoggerAdapter,
+    *,
+    event: str,
+    message: str,
+    category: str,
+    level: str = "info",
+    source: str = "backend",
+    service: str | None = None,
+    module: str | None = None,
+    request_id: str | None = None,
+    user_id: int | None = None,
+    context: Mapping[str, Any] | None = None,
+) -> None:
+    normalized_level = normalize_business_level(level)
+    safe_context = build_business_context(context)
+    log_method = getattr(logger, LEVEL_METHODS[normalized_level])
+    log_method(message, event=event, context=safe_context)
+    try:
+        loop = asyncio.get_running_loop()
+    except RuntimeError:
+        return
+    loop.create_task(
+        record_system_log(
+            source=source,
+            level=normalized_level,
+            message=message,
+            service=service,
+            module=module,
+            event=event,
+            request_id=request_id or get_request_id(),
+            user_id=user_id,
+            category=category,
+            context=safe_context,
+        )
+    )
+
+
+def exception_context(exc: BaseException, context: Mapping[str, Any] | None = None) -> dict[str, Any]:
+    return build_business_context(
+        context,
+        error_type=type(exc).__name__,
+        error=str(exc),
+    )

backend/app/services/collectors/aisstream.py

@@ -322,6 +322,21 @@ class AISStreamCollector(BaseCollector):
             last_success_at=now if data else None,
             lag_seconds=max((now - latest_observed_at).total_seconds(), 0),
         )
+        if snapshot_id is not None:
+            from app.models.data_snapshot import DataSnapshot
+
+            snapshot = await db.get(DataSnapshot, snapshot_id)
+            if snapshot:
+                snapshot.record_count = records_added
+                snapshot.status = "success"
+                snapshot.completed_at = now
+                snapshot.summary = {
+                    "created": records_added,
+                    "updated": 0,
+                    "unchanged": 0,
+                    "deleted": 0,
+                    "storage": "ais_raw_observations",
+                }
         await db.commit()
         await self.update_progress(records_added, force=True)
         return records_added

backend/app/services/collectors/base.py

@@ -4,42 +4,22 @@ import asyncio
 from abc import ABC, abstractmethod
 from typing import Dict, List, Any, Optional
 from datetime import UTC, datetime
+from time import perf_counter
+from urllib.parse import urlparse
 import httpx
 from sqlalchemy import select, text
 from sqlalchemy.ext.asyncio import AsyncSession
 
 from app.core.collected_data_fields import build_dynamic_metadata, get_record_field
 from app.core.countries import normalize_country
+from app.core.enums import JobStatus, SnapshotStatus
+from app.core.logging import get_logger
 from app.core.time import to_iso8601_utc
 from app.core.websocket.broadcaster import broadcaster
-from app.services.earth_layer_cache import invalidate_earth_layer_cache_for_source
-
-EARTH_UPDATE_LAYER_HINTS: dict[str, list[str]] = {
-    "ris_live_bgp": ["bgp"],
-    "bgpstream_bgp": ["bgp"],
-    "top500_supercomputers": ["computeCenters"],
-    "epoch_ai_gpu": ["computeCenters"],
-    "huggingface_models": ["computeCenters"],
-    "huggingface_datasets": ["computeCenters"],
-    "huggingface_spaces": ["computeCenters"],
-    "telegeography_cables": ["cables"],
-    "telegeography_landing_points": ["cables"],
-    "telegeography_cable_systems": ["cables"],
-    "arcgis_cables": ["cables"],
-    "fao_landing_points": ["cables"],
-    "arcgis_landing_points": ["cables"],
-    "arcgis_cable_landing_relations": ["cables"],
-    "spacetrack_tle": ["satellites"],
-    "celestrak_tle": ["satellites"],
-    "barentswatch_vessels": ["vessels"],
-    "aisstream_vessels": ["vessels"],
-    "news_live_streams": ["media"],
-    "media_news_archive": ["news"],
-}
+from app.services.business_logs import emit_business_log, exception_context
 
 
-def get_earth_update_layers_for_source(source: str) -> list[str]:
-    return EARTH_UPDATE_LAYER_HINTS.get(source, [])
+logger = get_logger(__name__, service="collector")
 
 
 class BaseCollector(ABC):
@@ -58,6 +38,7 @@ class BaseCollector(ABC):
         self._datasource_id = 1
         self._resolved_url: Optional[str] = None
         self._last_broadcast_progress: Optional[int] = None
+        self._last_save_summary: dict[str, int] = {}
 
     async def resolve_url(self, db: AsyncSession) -> None:
         from app.core.data_sources import get_data_sources_config
@@ -96,29 +77,6 @@ class BaseCollector(ABC):
         )
         self._last_broadcast_progress = rounded_progress
 
-    async def _publish_earth_update(
-        self,
-        *,
-        action: str,
-        records_processed: int,
-        task_id: int | None = None,
-    ) -> None:
-        layers = get_earth_update_layers_for_source(self.name)
-        if not layers:
-            return
-        await broadcaster.broadcast_earth_update(
-            {
-                "action": action,
-                "source": self.name,
-                "data_type": self.data_type,
-                "layers": layers,
-                "datasource_id": getattr(self, "_datasource_id", None),
-                "task_id": task_id,
-                "records_processed": records_processed,
-                "timestamp": to_iso8601_utc(datetime.now(UTC)),
-            }
-        )
-
     async def update_progress(self, records_processed: int, *, commit: bool = False, force: bool = False):
         """Update task progress - call this during data processing"""
         if self._current_task and self._db_session:
@@ -280,7 +238,7 @@ class BaseCollector(ABC):
             snapshot = await db.get(DataSnapshot, snapshot_id)
             if snapshot:
                 parent_snapshot_id = snapshot.parent_snapshot_id
-                snapshot.status = "cancelled"
+                snapshot.status = SnapshotStatus.CANCELLED.value
                 snapshot.is_current = False
                 snapshot.completed_at = datetime.now(UTC)
                 summary = dict(snapshot.summary or {})
@@ -322,19 +280,39 @@ class BaseCollector(ABC):
         from app.models.data_snapshot import DataSnapshot
 
         start_time = datetime.now(UTC)
+        started_at = perf_counter()
         datasource_id = getattr(self, "_datasource_id", 1)
         snapshot_id: Optional[int] = None
 
         if not collector_registry.is_active(self.name):
+            await self._log_collection_event(
+                "collector.run.skipped_disabled",
+                "Collector skipped because it is disabled",
+                level="info",
+                context={"status": "skipped", "reason": "disabled"},
+            )
             return {"status": "skipped", "reason": "Collector is disabled"}
 
-        task = CollectionTask(
-            datasource_id=datasource_id,
-            status="running",
-            phase="queued",
-            started_at=start_time,
-        )
-        db.add(task)
+        task = self._current_task if isinstance(self._current_task, CollectionTask) else None
+        if task is None:
+            task = CollectionTask(
+                datasource_id=datasource_id,
+                source=self.name,
+                task_type="collect",
+                status="running",
+                phase="queued",
+                started_at=start_time,
+            )
+            db.add(task)
+        else:
+            task.datasource_id = datasource_id
+            task.source = task.source or self.name
+            task.task_type = task.task_type or "collect"
+            task.status = JobStatus.RUNNING.value
+            task.phase = "queued"
+            task.started_at = task.started_at or start_time
+            task.completed_at = None
+            task.error_message = None
         await db.commit()
         task_id = task.id
 
@@ -344,25 +322,78 @@ class BaseCollector(ABC):
 
         await self.resolve_url(db)
         await self._publish_task_update(force=True)
+        await self._log_collection_event(
+            "collector.run.started",
+            "Collector run started",
+            context={"status": "running", "task_id": task_id},
+        )
 
         try:
+            phase_started_at = perf_counter()
             await self.set_phase("fetching", message="正在拉取原始数据")
+            await self._log_collection_event(
+                "collector.phase.fetching.start",
+                "Collector fetch phase started",
+                context={"task_id": task_id, "snapshot_id": snapshot_id},
+            )
             raw_data = await self.fetch()
             task.total_records = len(raw_data)
             await db.commit()
             await self._publish_task_update(force=True)
+            await self._log_collection_event(
+                "collector.phase.fetching.success",
+                "Collector fetch phase completed",
+                context={
+                    "task_id": task_id,
+                    "raw_count": len(raw_data),
+                    "duration_ms": self._duration_ms(phase_started_at),
+                },
+            )
 
             if self.fail_on_empty and not raw_data:
                 raise RuntimeError(f"Collector {self.name} returned no data")
 
+            phase_started_at = perf_counter()
             await self.set_phase("transforming", message="正在转换采集数据")
+            await self._log_collection_event(
+                "collector.phase.transforming.start",
+                "Collector transform phase started",
+                context={"task_id": task_id, "raw_count": len(raw_data)},
+            )
             data = self.transform(raw_data)
+            await self._log_collection_event(
+                "collector.phase.transforming.success",
+                "Collector transform phase completed",
+                context={
+                    "task_id": task_id,
+                    "raw_count": len(raw_data),
+                    "transformed_count": len(data),
+                    "duration_ms": self._duration_ms(phase_started_at),
+                },
+            )
             snapshot_id = await self._create_snapshot(db, task_id, data, start_time)
 
+            phase_started_at = perf_counter()
             await self.set_phase("saving", message="正在保存采集数据")
+            await self._log_collection_event(
+                "collector.phase.saving.start",
+                "Collector save phase started",
+                context={"task_id": task_id, "snapshot_id": snapshot_id, "transformed_count": len(data)},
+            )
             records_count = await self._save_data(db, data, task_id=task_id, snapshot_id=snapshot_id)
+            await self._log_collection_event(
+                "collector.phase.saving.success",
+                "Collector save phase completed",
+                context={
+                    "task_id": task_id,
+                    "snapshot_id": snapshot_id,
+                    "saved_count": records_count,
+                    **self._last_save_summary,
+                    "duration_ms": self._duration_ms(phase_started_at),
+                },
+            )
 
-            task.status = "success"
+            task.status = JobStatus.SUCCESS.value
             task.phase = "completed"
             task.phase_progress = 100.0
             task.phase_message = "采集完成"
@@ -374,10 +405,19 @@ class BaseCollector(ABC):
             task.completed_at = datetime.now(UTC)
             await db.commit()
             await self._publish_task_update(force=True)
-            await self._publish_earth_update(
-                action="collector_completed",
-                records_processed=records_count,
-                task_id=task_id,
+            await self._log_collection_event(
+                "collector.run.completed",
+                "Collector run completed",
+                context={
+                    "status": "success",
+                    "task_id": task_id,
+                    "snapshot_id": snapshot_id,
+                    "raw_count": len(raw_data),
+                    "transformed_count": len(data),
+                    "saved_count": records_count,
+                    **self._last_save_summary,
+                    "duration_ms": self._duration_ms(started_at),
+                },
             )
 
             return {
@@ -388,7 +428,7 @@ class BaseCollector(ABC):
             }
         except asyncio.CancelledError:
             await db.rollback()
-            task.status = "cancelled"
+            task.status = JobStatus.CANCELLED.value
             task.phase = "cancelled"
             task.phase_message = "采集已取消"
             task.error_message = "Collection cancelled by operator and rolled back"
@@ -402,10 +442,21 @@ class BaseCollector(ABC):
                 )
             await db.commit()
             await self._publish_task_update(force=True)
+            await self._log_collection_event(
+                "collector.run.cancelled",
+                "Collector run cancelled",
+                level="warning",
+                context={
+                    "status": "cancelled",
+                    "task_id": task_id,
+                    "snapshot_id": snapshot_id,
+                    "duration_ms": self._duration_ms(started_at),
+                },
+            )
             raise
         except Exception as e:
             await db.rollback()
-            task.status = "failed"
+            task.status = JobStatus.FAILED.value
             task.phase = "failed"
             task.phase_message = str(e)
             task.error_message = str(e)
@@ -413,11 +464,25 @@ class BaseCollector(ABC):
             if snapshot_id is not None:
                 snapshot = await db.get(DataSnapshot, snapshot_id)
                 if snapshot:
-                    snapshot.status = "failed"
+                    snapshot.status = SnapshotStatus.FAILED.value
                     snapshot.completed_at = datetime.now(UTC)
                     snapshot.summary = {"error": str(e)}
             await db.commit()
             await self._publish_task_update(force=True)
+            await self._log_collection_event(
+                "collector.run.failed",
+                "Collector run failed",
+                level="error",
+                context=exception_context(
+                    e,
+                    {
+                        "status": "failed",
+                        "task_id": task_id,
+                        "snapshot_id": snapshot_id,
+                        "duration_ms": self._duration_ms(started_at),
+                    },
+                ),
+            )
 
             return {
                 "status": "failed",
@@ -438,12 +503,13 @@ class BaseCollector(ABC):
         from app.models.data_snapshot import DataSnapshot
 
         if not data:
+            self._last_save_summary = {"created": 0, "updated": 0, "unchanged": 0, "deleted": 0}
             if snapshot_id is not None:
                 snapshot = await db.get(DataSnapshot, snapshot_id)
                 if snapshot:
                     snapshot.record_count = 0
                     snapshot.summary = {"created": 0, "updated": 0, "unchanged": 0}
-                    snapshot.status = "success"
+                    snapshot.status = SnapshotStatus.SUCCESS.value
                     snapshot.completed_at = datetime.now(UTC)
                     await db.commit()
             return 0
@@ -576,7 +642,7 @@ class BaseCollector(ABC):
             snapshot = await db.get(DataSnapshot, snapshot_id)
             if snapshot:
                 snapshot.record_count = records_added
-                snapshot.status = "success"
+                snapshot.status = SnapshotStatus.SUCCESS.value
                 snapshot.completed_at = datetime.now(UTC)
                 snapshot.summary = {
                     "created": created_count,
@@ -584,12 +650,51 @@ class BaseCollector(ABC):
                     "unchanged": unchanged_count,
                     "deleted": len(deleted_keys),
                 }
+            self._last_save_summary = {
+                "created": created_count,
+                "updated": updated_count,
+                "unchanged": unchanged_count,
+                "deleted": len(deleted_keys),
+            }
+        else:
+            self._last_save_summary = {
+                "created": created_count,
+                "updated": updated_count,
+                "unchanged": unchanged_count,
+                "deleted": 0,
+            }
 
         await db.commit()
-        invalidate_earth_layer_cache_for_source(self.name)
         await self.update_progress(len(data), force=True)
         return records_added
 
+    @staticmethod
+    def _duration_ms(started_at: float) -> int:
+        return int((perf_counter() - started_at) * 1000)
+
+    async def _log_collection_event(
+        self,
+        event: str,
+        message: str,
+        *,
+        level: str = "info",
+        context: Dict[str, Any] | None = None,
+    ) -> None:
+        await emit_business_log(
+            logger,
+            event=event,
+            message=message,
+            category="collector",
+            level=level,
+            service="collector",
+            module=__name__,
+            context={
+                "collector_name": self.name,
+                "datasource_id": getattr(self, "_datasource_id", None),
+                **(context or {}),
+            },
+        )
+
     async def save(self, db: AsyncSession, data: List[Dict[str, Any]]) -> int:
         """Save data to database (legacy method, use _save_data instead)"""
         return await self._save_data(db, data)
@@ -602,10 +707,65 @@ class HTTPCollector(BaseCollector):
     headers: Dict[str, str] = {}
 
     async def fetch(self) -> List[Dict[str, Any]]:
+        started_at = perf_counter()
+        request_host = urlparse(self.base_url).netloc
+        await emit_business_log(
+            logger,
+            event="collector.http.fetch.start",
+            message="Collector HTTP request started",
+            category="collector",
+            service="collector",
+            module=__name__,
+            context={
+                "collector_name": self.name,
+                "datasource_id": getattr(self, "_datasource_id", None),
+                "url_host": request_host,
+            },
+        )
         async with httpx.AsyncClient(timeout=60.0) as client:
-            response = await client.get(self.base_url, headers=self.headers)
-            response.raise_for_status()
-            return self.parse_response(response.json())
+            try:
+                response = await client.get(self.base_url, headers=self.headers)
+                response.raise_for_status()
+                payload = response.json()
+                parsed = self.parse_response(payload)
+                await emit_business_log(
+                    logger,
+                    event="collector.http.fetch.success",
+                    message="Collector HTTP request completed",
+                    category="collector",
+                    service="collector",
+                    module=__name__,
+                    context={
+                        "collector_name": self.name,
+                        "datasource_id": getattr(self, "_datasource_id", None),
+                        "url_host": request_host,
+                        "status_code": response.status_code,
+                        "response_bytes": len(response.content or b""),
+                        "parsed_count": len(parsed),
+                        "duration_ms": BaseCollector._duration_ms(started_at),
+                    },
+                )
+                return parsed
+            except Exception as exc:
+                await emit_business_log(
+                    logger,
+                    event="collector.http.fetch.failed",
+                    message="Collector HTTP request failed",
+                    category="collector",
+                    level="error",
+                    service="collector",
+                    module=__name__,
+                    context=exception_context(
+                        exc,
+                        {
+                            "collector_name": self.name,
+                            "datasource_id": getattr(self, "_datasource_id", None),
+                            "url_host": request_host,
+                            "duration_ms": BaseCollector._duration_ms(started_at),
+                        },
+                    ),
+                )
+                raise
 
     @abstractmethod
     def parse_response(self, response: Dict[str, Any]) -> List[Dict[str, Any]]:

backend/app/services/collectors/celestrak.py

@@ -1,15 +1,45 @@
-"""CelesTrak TLE Collector
+"""CelesTrak TLE Collector.
 
-Collects satellite TLE (Two-Line Element) data from CelesTrak.org.
-Free, no authentication required.
+Collects the full active satellite GP element set from CelesTrak.
 """
 
+import asyncio
 import json
-from typing import Dict, Any, List
+from pathlib import Path
+from time import perf_counter
+from typing import Any, Dict, List
+from urllib.parse import urlencode, urlparse
+
 import httpx
 
+from app.core.logging import get_logger
 from app.core.satellite_tle import build_tle_lines_from_elements
+from app.services.business_logs import emit_business_log, exception_context
 from app.services.collectors.base import BaseCollector
+from app.services.collectors.downloads import DownloadHTTPStatusError, ResumableFileDownloader
+
+
+logger = get_logger(__name__, service="collector")
+ACTIVE_GROUP = "active"
+FALLBACK_GROUPS = (
+    "starlink",
+    "gps-ops",
+    "galileo",
+    "glo-ops",
+    "beidou",
+    "geo",
+    "iridium-next",
+    "stations",
+    "visual",
+    "weather",
+    "science",
+    "cubesat",
+    "amateur",
+    "last-30-days",
+)
+FETCH_RETRY_ATTEMPTS = 3
+FETCH_RETRY_BASE_DELAY_SECONDS = 0.8
+CELESTRAK_NOT_UPDATED_MARKER = "GP data has not updated since your last successful"
 
 
 class CelesTrakTLECollector(BaseCollector):
@@ -18,55 +48,360 @@ class CelesTrakTLECollector(BaseCollector):
     module = "L3"
     frequency_hours = 24
     data_type = "satellite_tle"
+    _downloader = ResumableFileDownloader(
+        cache_namespace="celestrak",
+        default_accept="application/json",
+    )
 
     @property
     def base_url(self) -> str:
         return self._resolved_url or ""
 
+    def _active_url(self) -> str:
+        return self._group_url(ACTIVE_GROUP)
+
+    def _group_url(self, group: str) -> str:
+        if not self.base_url:
+            raise RuntimeError("CelesTrak base URL is not configured")
+        return f"{self.base_url}?{urlencode({'GROUP': group, 'FORMAT': 'json'})}"
+
     async def fetch(self) -> List[Dict[str, Any]]:
-        satellite_groups = [
-            "starlink",
-            "gps-ops",
-            "galileo",
-            "glonass",
-            "beidou",
-            "leo",
-            "geo",
-            "iridium-next",
-        ]
+        url = self._active_url()
+        last_error: Exception | None = None
 
-        all_satellites = []
-
-        async with httpx.AsyncClient(timeout=120.0) as client:
-            for group in satellite_groups:
+        async with httpx.AsyncClient(timeout=180.0, follow_redirects=True) as client:
+            for attempt in range(1, FETCH_RETRY_ATTEMPTS + 1):
+                started_at = perf_counter()
                 try:
-                    url = f"{self.base_url}?GROUP={group}&FORMAT=json"
-                    response = await client.get(url)
+                    await emit_business_log(
+                        logger,
+                        event="collector.celestrak.download.start",
+                        message="CelesTrak active satellite download started",
+                        category="collector",
+                        service="collector",
+                        module=__name__,
+                        context={
+                            "collector_name": self.name,
+                            "datasource_id": getattr(self, "_datasource_id", None),
+                            "group": ACTIVE_GROUP,
+                            "attempt": attempt,
+                            "url_host": urlparse(url).netloc,
+                        },
+                    )
+                    body_path = await self._downloader.download_file(
+                        client,
+                        url,
+                        extension=".json",
+                        accept="application/json",
+                        progress_callback=self._report_download_progress,
+                        validate_existing=self._validate_json_file,
+                    )
+                    data = await self._load_downloaded_payload(body_path, url)
+                    await emit_business_log(
+                        logger,
+                        event="collector.celestrak.download.success",
+                        message="CelesTrak active satellite download completed",
+                        category="collector",
+                        service="collector",
+                        module=__name__,
+                        context={
+                            "collector_name": self.name,
+                            "datasource_id": getattr(self, "_datasource_id", None),
+                            "group": ACTIVE_GROUP,
+                            "attempt": attempt,
+                            "record_count": len(data),
+                            "duration_ms": self._duration_ms(started_at),
+                        },
+                    )
+                    return data
+                except DownloadHTTPStatusError as exc:
+                    if self._is_not_updated_response(exc):
+                        cached_path = self._downloader.get_cached_file(
+                            url,
+                            ".json",
+                            validate_existing=self._validate_json_file,
+                        )
+                        if cached_path is not None:
+                            data = await self._load_downloaded_payload(cached_path, url)
+                            await emit_business_log(
+                                logger,
+                                event="collector.celestrak.download.cached_not_updated",
+                                message="CelesTrak active satellite data has not changed; using cached download",
+                                category="collector",
+                                level="warning",
+                                service="collector",
+                                module=__name__,
+                                context={
+                                    "collector_name": self.name,
+                                    "datasource_id": getattr(self, "_datasource_id", None),
+                                    "group": ACTIVE_GROUP,
+                                    "attempt": attempt,
+                                    "record_count": len(data),
+                                    "duration_ms": self._duration_ms(started_at),
+                                },
+                            )
+                            return data
+                        await emit_business_log(
+                            logger,
+                            event="collector.celestrak.download.not_updated_no_cache",
+                            message="CelesTrak active satellite data has not changed; trying fallback groups",
+                            category="collector",
+                            level="warning",
+                            service="collector",
+                            module=__name__,
+                            context=exception_context(
+                                exc,
+                                {
+                                    "collector_name": self.name,
+                                    "datasource_id": getattr(self, "_datasource_id", None),
+                                    "group": ACTIVE_GROUP,
+                                    "attempt": attempt,
+                                    "duration_ms": self._duration_ms(started_at),
+                                },
+                            ),
+                        )
+                        return await self._fetch_fallback_groups(client, active_error=exc)
+                    raise
+                except Exception as exc:
+                    last_error = exc
+                    is_final_attempt = attempt >= FETCH_RETRY_ATTEMPTS
+                    await emit_business_log(
+                        logger,
+                        event=(
+                            "collector.celestrak.download.failed"
+                            if is_final_attempt
+                            else "collector.celestrak.download.retry"
+                        ),
+                        message=(
+                            "CelesTrak active satellite download failed"
+                            if is_final_attempt
+                            else "CelesTrak active satellite download will retry"
+                        ),
+                        category="collector",
+                        level="error" if is_final_attempt else "warning",
+                        service="collector",
+                        module=__name__,
+                        context=exception_context(
+                            exc,
+                            {
+                                "collector_name": self.name,
+                                "datasource_id": getattr(self, "_datasource_id", None),
+                                "group": ACTIVE_GROUP,
+                                "attempt": attempt,
+                                "duration_ms": self._duration_ms(started_at),
+                            },
+                        ),
+                    )
+                    if not is_final_attempt:
+                        await asyncio.sleep(FETCH_RETRY_BASE_DELAY_SECONDS * attempt)
 
-                    if response.status_code == 200:
-                        data = response.json()
-                        if isinstance(data, list):
-                            for item in data:
-                                if isinstance(item, dict):
-                                    item["_celestrak_group"] = group
-                            all_satellites.extend(data)
-                            print(f"CelesTrak: Fetched {len(data)} satellites from group '{group}'")
-                except Exception as e:
-                    print(f"CelesTrak: Error fetching group '{group}': {e}")
+        raise RuntimeError(f"CelesTrak active satellite download failed after retries: {last_error}")
 
-        if not all_satellites:
-            return self._get_sample_data()
+    async def _fetch_fallback_groups(
+        self,
+        client: httpx.AsyncClient,
+        *,
+        active_error: DownloadHTTPStatusError,
+    ) -> List[Dict[str, Any]]:
+        started_at = perf_counter()
+        records_by_norad: dict[str, Dict[str, Any]] = {}
+        group_counts: dict[str, int] = {}
 
-        print(f"CelesTrak: Total satellites fetched: {len(all_satellites)}")
+        await emit_business_log(
+            logger,
+            event="collector.celestrak.fallback_groups.start",
+            message="CelesTrak fallback group download started",
+            category="collector",
+            level="warning",
+            service="collector",
+            module=__name__,
+            context={
+                "collector_name": self.name,
+                "datasource_id": getattr(self, "_datasource_id", None),
+                "groups": list(FALLBACK_GROUPS),
+                "reason": "active_not_updated_without_cache",
+            },
+        )
 
-        # Return raw data - base.run() will call transform()
-        return all_satellites
+        try:
+            for group in FALLBACK_GROUPS:
+                group_url = self._group_url(group)
+                cached_path = self._downloader.get_cached_file(
+                    group_url,
+                    ".json",
+                    validate_existing=self._validate_json_file,
+                )
+                if cached_path is not None:
+                    body_path = cached_path
+                else:
+                    try:
+                        body_path = await self._downloader.download_file(
+                            client,
+                            group_url,
+                            extension=".json",
+                            accept="application/json",
+                            validate_existing=self._validate_json_file,
+                        )
+                    except DownloadHTTPStatusError as exc:
+                        if not self._is_not_updated_response(exc):
+                            raise RuntimeError(f"CelesTrak fallback group '{group}' download failed: {exc}") from exc
+                        raise RuntimeError(
+                            f"CelesTrak fallback group '{group}' has not updated and no local cached copy is available"
+                        ) from exc
+
+                group_records = await self._load_downloaded_payload(
+                    body_path,
+                    group_url,
+                    query_group=group,
+                    constellation_group=group,
+                )
+                group_counts[group] = len(group_records)
+                for item in group_records:
+                    norad_cat_id = item.get("NORAD_CAT_ID")
+                    if norad_cat_id is None:
+                        continue
+                    records_by_norad.setdefault(str(norad_cat_id), item)
+        except Exception as exc:
+            await emit_business_log(
+                logger,
+                event="collector.celestrak.fallback_groups.failed",
+                message="CelesTrak fallback group download failed",
+                category="collector",
+                level="error",
+                service="collector",
+                module=__name__,
+                context=exception_context(
+                    exc,
+                    {
+                        "collector_name": self.name,
+                        "datasource_id": getattr(self, "_datasource_id", None),
+                        "groups": list(FALLBACK_GROUPS),
+                        "completed_groups": list(group_counts),
+                        "duration_ms": self._duration_ms(started_at),
+                    },
+                ),
+            )
+            raise RuntimeError(
+                "CelesTrak active data has not updated since this network's last successful download, "
+                "no active cache is available, and fallback group mode failed. Wait until CelesTrak "
+                "publishes the next GP update, restore the Planet download cache, or use Space-Track."
+            ) from active_error
+
+        records = list(records_by_norad.values())
+        if not records:
+            raise RuntimeError("CelesTrak fallback group mode produced no satellite records")
+
+        await emit_business_log(
+            logger,
+            event="collector.celestrak.fallback_groups.success",
+            message="CelesTrak fallback group download completed",
+            category="collector",
+            level="warning",
+            service="collector",
+            module=__name__,
+            context={
+                "collector_name": self.name,
+                "datasource_id": getattr(self, "_datasource_id", None),
+                "groups": list(FALLBACK_GROUPS),
+                "group_counts": group_counts,
+                "record_count": len(records),
+                "duration_ms": self._duration_ms(started_at),
+            },
+        )
+        return records
+
+    async def _load_downloaded_payload(
+        self,
+        body_path: Path,
+        url: str,
+        *,
+        query_group: str = ACTIVE_GROUP,
+        constellation_group: str | None = None,
+    ) -> List[Dict[str, Any]]:
+        try:
+            data = self._load_active_payload(body_path)
+        except RuntimeError as exc:
+            await self._log_parse_failure(exc)
+            raise
+        for item in data:
+            item["_celestrak_query_group"] = query_group
+            item["_celestrak_source_url"] = url
+            if constellation_group:
+                item["_celestrak_group"] = constellation_group
+        return data
+
+    @staticmethod
+    def _is_not_updated_response(exc: DownloadHTTPStatusError) -> bool:
+        return exc.status_code == 403 and CELESTRAK_NOT_UPDATED_MARKER in exc.body
+
+    @staticmethod
+    def _duration_ms(started_at: float) -> int:
+        return int((perf_counter() - started_at) * 1000)
+
+    async def _report_download_progress(self, downloaded: int, total: int | None) -> None:
+        if total and total > 0:
+            await self.update_phase_progress(
+                current=min(downloaded, total),
+                total=total,
+                unit="bytes",
+                message=f"正在下载 CelesTrak active 卫星数据 {downloaded}/{total} bytes",
+                commit=True,
+            )
+
+    @staticmethod
+    def _validate_json_file(path: Path) -> bool:
+        try:
+            data = json.loads(path.read_text(encoding="utf-8"))
+        except (OSError, json.JSONDecodeError, UnicodeDecodeError):
+            return False
+        return isinstance(data, list)
+
+    async def _log_parse_failure(self, exc: Exception) -> None:
+        await emit_business_log(
+            logger,
+            event="collector.celestrak.parse.failed",
+            message="CelesTrak active satellite JSON parsing failed",
+            category="collector",
+            level="error",
+            service="collector",
+            module=__name__,
+            context=exception_context(
+                exc,
+                {
+                    "collector_name": self.name,
+                    "datasource_id": getattr(self, "_datasource_id", None),
+                    "group": ACTIVE_GROUP,
+                },
+            ),
+        )
+
+    def _load_active_payload(self, path: Path) -> List[Dict[str, Any]]:
+        try:
+            raw = json.loads(path.read_text(encoding="utf-8"))
+        except (OSError, json.JSONDecodeError, UnicodeDecodeError) as exc:
+            raise RuntimeError(f"CelesTrak active payload is not valid JSON: {exc}") from exc
+        if not isinstance(raw, list):
+            raise RuntimeError("CelesTrak active payload is not a JSON array")
+
+        records: List[Dict[str, Any]] = []
+        invalid_count = 0
+        for item in raw:
+            if isinstance(item, dict) and item.get("NORAD_CAT_ID") is not None:
+                records.append(item)
+            else:
+                invalid_count += 1
+        if invalid_count:
+            raise RuntimeError(f"CelesTrak active payload contains {invalid_count} invalid record(s)")
+        if not records:
+            raise RuntimeError("CelesTrak active payload contains no satellite records")
+        return records
 
     def transform(self, raw_data: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
         transformed = []
         for item in raw_data:
+            norad_cat_id = item.get("NORAD_CAT_ID")
             tle_line1, tle_line2 = build_tle_lines_from_elements(
-                norad_cat_id=item.get("NORAD_CAT_ID"),
+                norad_cat_id=norad_cat_id,
                 epoch=item.get("EPOCH"),
                 inclination=item.get("INCLINATION"),
                 raan=item.get("RA_OF_ASC_NODE"),
@@ -75,14 +410,18 @@ class CelesTrakTLECollector(BaseCollector):
                 mean_anomaly=item.get("MEAN_ANOMALY"),
                 mean_motion=item.get("MEAN_MOTION"),
             )
+            constellation_group = self._infer_constellation_group(item)
 
             transformed.append(
                 {
+                    "source_id": str(norad_cat_id),
                     "name": item.get("OBJECT_NAME", "Unknown"),
                     "reference_date": item.get("EPOCH", ""),
                     "metadata": {
-                        "constellation_group": item.get("_celestrak_group"),
-                        "norad_cat_id": item.get("NORAD_CAT_ID"),
+                        "constellation_group": constellation_group,
+                        "celestrak_query_group": item.get("_celestrak_query_group") or ACTIVE_GROUP,
+                        "celestrak_source_url": item.get("_celestrak_source_url"),
+                        "norad_cat_id": norad_cat_id,
                         "international_designator": item.get("OBJECT_ID"),
                         "epoch": item.get("EPOCH"),
                         "mean_motion": item.get("MEAN_MOTION"),
@@ -105,6 +444,19 @@ class CelesTrakTLECollector(BaseCollector):
             )
         return transformed
 
+    @staticmethod
+    def _infer_constellation_group(item: Dict[str, Any]) -> str | None:
+        explicit_group = str(item.get("_celestrak_group") or "").strip().lower()
+        if explicit_group and explicit_group != ACTIVE_GROUP:
+            return explicit_group
+
+        name = str(item.get("OBJECT_NAME") or "").strip().upper()
+        if name.startswith("STARLINK"):
+            return "starlink"
+        if name.startswith("IRIDIUM"):
+            return "iridium-next"
+        return None
+
     def _get_sample_data(self) -> List[Dict[str, Any]]:
         return [
             {

backend/app/services/collectors/downloads.py

@@ -4,7 +4,7 @@ from __future__ import annotations
 
 import hashlib
 import json
-import tempfile
+import os
 import time
 from datetime import UTC, datetime
 from pathlib import Path
@@ -17,6 +17,31 @@ ProgressCallback = Callable[[int, int | None], Awaitable[None]]
 ValidateCallback = Callable[[Path], bool]
 
 
+class DownloadHTTPStatusError(RuntimeError):
+    """HTTP status error that keeps the upstream response body for caller-specific handling."""
+
+    def __init__(self, *, url: str, status_code: int, body: str) -> None:
+        self.url = url
+        self.status_code = status_code
+        self.body = body
+        preview = body.strip().replace("\r", " ").replace("\n", " ")[:240]
+        suffix = f": {preview}" if preview else ""
+        super().__init__(f"HTTP {status_code} while downloading {url}{suffix}")
+
+
+def default_download_cache_root() -> Path:
+    configured = os.getenv("PLANET_DOWNLOAD_CACHE_DIR")
+    if configured:
+        return Path(configured).expanduser()
+    planet_cache = os.getenv("PLANET_CACHE_DIR")
+    if planet_cache:
+        return Path(planet_cache).expanduser() / "downloads"
+    xdg_cache = os.getenv("XDG_CACHE_HOME")
+    if xdg_cache:
+        return Path(xdg_cache).expanduser() / "planet" / "downloads"
+    return Path.home() / ".cache" / "planet" / "downloads"
+
+
 class ResumableFileDownloader:
     """Download files with cache validators and byte-range resume support."""
 
@@ -26,8 +51,9 @@ class ResumableFileDownloader:
         cache_namespace: str,
         user_agent: str = "Planet-Intelligence-System/1.0 (Python/collector)",
         default_accept: str = "*/*",
+        cache_root: Path | None = None,
     ) -> None:
-        self._cache_dir = Path(tempfile.gettempdir()) / "planet-download-cache" / cache_namespace
+        self._cache_dir = (cache_root or default_download_cache_root()) / cache_namespace
         self._user_agent = user_agent
         self._default_accept = default_accept
 
@@ -43,6 +69,25 @@ class ResumableFileDownloader:
         meta_path = self._cache_dir / f"{key}.meta.json"
         return final_path, part_path, meta_path
 
+    def cached_file_path(self, url: str, extension: str) -> Path:
+        final_path, _, _ = self._cache_paths(url, extension)
+        return final_path
+
+    def get_cached_file(
+        self,
+        url: str,
+        extension: str,
+        *,
+        validate_existing: ValidateCallback | None = None,
+    ) -> Path | None:
+        final_path = self.cached_file_path(url, extension)
+        if not final_path.exists():
+            return None
+        if validate_existing and not validate_existing(final_path):
+            final_path.unlink(missing_ok=True)
+            return None
+        return final_path
+
     @staticmethod
     def _load_meta(meta_path: Path) -> dict[str, Any]:
         if not meta_path.exists():
@@ -140,7 +185,9 @@ class ResumableFileDownloader:
                 if progress_callback and expected_size and expected_size > 0:
                     await progress_callback(expected_size, expected_size)
                 return final_path
-            response.raise_for_status()
+            if response.status_code >= 400:
+                body = (await response.aread()).decode("utf-8", errors="replace")
+                raise DownloadHTTPStatusError(url=url, status_code=response.status_code, body=body)
 
             if response.status_code == 206 and resume_from > 0:
                 mode = "ab"

backend/app/services/collectors/peeringdb.py

@@ -11,17 +11,20 @@ To get higher limits, set PEERINGDB_API_KEY environment variable.
 """
 
 import asyncio
-import os
-from typing import Dict, Any, List
 from datetime import UTC, datetime
+import os
+from typing import Any, Dict, List
+from urllib.parse import urlencode
 
 import httpx
-from urllib.parse import urlencode
+
+from app.core.logging import get_logger
 from app.services.collectors.base import HTTPCollector
 
 
 # PeeringDB API key - read from environment variable
 PEERINGDB_API_KEY = os.environ.get("PEERINGDB_API_KEY", "")
+logger = get_logger(__name__, service="collector")
 
 
 class PeeringDBIXPCollector(HTTPCollector):
@@ -39,6 +42,7 @@ class PeeringDBIXPCollector(HTTPCollector):
             "User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
             "Accept": "application/json",
         }
+
     @property
     def request_url(self) -> str:
         base = self._resolved_url or self.base_url
@@ -61,7 +65,11 @@ class PeeringDBIXPCollector(HTTPCollector):
                     if response.status_code == 429:
                         # Rate limited - wait and retry with exponential backoff
                         delay = base_delay * (2**attempt)
-                        print(f"PeeringDB rate limited, waiting {delay}s before retry...")
+                        logger.warning_event(
+                            "PeeringDB rate limited; retrying after delay",
+                            event="collector.peeringdb.rate_limited",
+                            context={"delay_seconds": delay, "attempt": attempt + 1},
+                        )
                         await asyncio.sleep(delay)
                         last_error = "Rate limited"
                         continue
@@ -72,13 +80,21 @@ class PeeringDBIXPCollector(HTTPCollector):
             except httpx.HTTPStatusError as e:
                 if e.response.status_code == 429:
                     delay = base_delay * (2**attempt)
-                    print(f"PeeringDB rate limited, waiting {delay}s before retry...")
+                    logger.warning_event(
+                        "PeeringDB rate limited; retrying after delay",
+                        event="collector.peeringdb.rate_limited",
+                        context={"delay_seconds": delay, "attempt": attempt + 1},
+                    )
                     await asyncio.sleep(delay)
                     last_error = "Rate limited"
                     continue
                 raise
 
-        print(f"Warning: PeeringDB collection failed after {max_retries} retries: {last_error}")
+        logger.warning_event(
+            "PeeringDB collection failed after retries",
+            event="collector.peeringdb.retries_exhausted",
+            context={"max_retries": max_retries, "last_error": last_error},
+        )
         return {}
 
     async def fetch(self) -> List[Dict[str, Any]]:
@@ -146,6 +162,7 @@ class PeeringDBNetworkCollector(HTTPCollector):
             "User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
             "Accept": "application/json",
         }
+
     @property
     def request_url(self) -> str:
         base = self._resolved_url or self.base_url
@@ -167,7 +184,11 @@ class PeeringDBNetworkCollector(HTTPCollector):
 
                     if response.status_code == 429:
                         delay = base_delay * (2**attempt)
-                        print(f"PeeringDB rate limited, waiting {delay}s before retry...")
+                        logger.warning_event(
+                            "PeeringDB rate limited; retrying after delay",
+                            event="collector.peeringdb.rate_limited",
+                            context={"delay_seconds": delay, "attempt": attempt + 1},
+                        )
                         await asyncio.sleep(delay)
                         last_error = "Rate limited"
                         continue
@@ -178,13 +199,21 @@ class PeeringDBNetworkCollector(HTTPCollector):
             except httpx.HTTPStatusError as e:
                 if e.response.status_code == 429:
                     delay = base_delay * (2**attempt)
-                    print(f"PeeringDB rate limited, waiting {delay}s before retry...")
+                    logger.warning_event(
+                        "PeeringDB rate limited; retrying after delay",
+                        event="collector.peeringdb.rate_limited",
+                        context={"delay_seconds": delay, "attempt": attempt + 1},
+                    )
                     await asyncio.sleep(delay)
                     last_error = "Rate limited"
                     continue
                 raise
 
-        print(f"Warning: PeeringDB collection failed after {max_retries} retries: {last_error}")
+        logger.warning_event(
+            "PeeringDB collection failed after retries",
+            event="collector.peeringdb.retries_exhausted",
+            context={"max_retries": max_retries, "last_error": last_error},
+        )
         return {}
 
     async def fetch(self) -> List[Dict[str, Any]]:
@@ -254,6 +283,7 @@ class PeeringDBFacilityCollector(HTTPCollector):
             "User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
             "Accept": "application/json",
         }
+
     @property
     def request_url(self) -> str:
         base = self._resolved_url or self.base_url
@@ -275,7 +305,11 @@ class PeeringDBFacilityCollector(HTTPCollector):
 
                     if response.status_code == 429:
                         delay = base_delay * (2**attempt)
-                        print(f"PeeringDB rate limited, waiting {delay}s before retry...")
+                        logger.warning_event(
+                            "PeeringDB rate limited; retrying after delay",
+                            event="collector.peeringdb.rate_limited",
+                            context={"delay_seconds": delay, "attempt": attempt + 1},
+                        )
                         await asyncio.sleep(delay)
                         last_error = "Rate limited"
                         continue
@@ -286,13 +320,21 @@ class PeeringDBFacilityCollector(HTTPCollector):
             except httpx.HTTPStatusError as e:
                 if e.response.status_code == 429:
                     delay = base_delay * (2**attempt)
-                    print(f"PeeringDB rate limited, waiting {delay}s before retry...")
+                    logger.warning_event(
+                        "PeeringDB rate limited; retrying after delay",
+                        event="collector.peeringdb.rate_limited",
+                        context={"delay_seconds": delay, "attempt": attempt + 1},
+                    )
                     await asyncio.sleep(delay)
                     last_error = "Rate limited"
                     continue
                 raise
 
-        print(f"Warning: PeeringDB collection failed after {max_retries} retries: {last_error}")
+        logger.warning_event(
+            "PeeringDB collection failed after retries",
+            event="collector.peeringdb.retries_exhausted",
+            context={"max_retries": max_retries, "last_error": last_error},
+        )
         return {}
 
     async def fetch(self) -> List[Dict[str, Any]]:

backend/app/services/collectors/spacetrack.py

@@ -1,17 +1,21 @@
-"""Space-Track TLE Collector
+"""Space-Track TLE Collector.
 
 Collects satellite TLE (Two-Line Element) data from Space-Track.org.
 API documentation: https://www.space-track.org/documentation
 """
 
-import json
-from typing import Dict, Any, List
-import httpx
+from typing import Any, Dict, List
 from urllib.parse import urlparse
 
-from app.services.collectors.base import BaseCollector
+import httpx
+
 from app.core.data_sources import get_data_sources_config
+from app.core.logging import get_logger
 from app.core.satellite_tle import build_tle_lines_from_elements
+from app.services.collectors.base import BaseCollector
+
+
+logger = get_logger(__name__, service="collector")
 
 
 class SpaceTrackTLECollector(BaseCollector):
@@ -53,10 +57,16 @@ class SpaceTrackTLECollector(BaseCollector):
         password = settings.SPACETRACK_PASSWORD
 
         if not username or not password:
-            print("SPACETRACK: No credentials configured, using sample data")
+            logger.warning_event(
+                "Space-Track credentials are not configured; using sample data",
+                event="collector.spacetrack.credentials_missing",
+            )
             return self._get_sample_data()
 
-        print(f"SPACETRACK: Attempting to fetch TLE data with username: {username}")
+        logger.info_event(
+            "Space-Track TLE fetch started",
+            event="collector.spacetrack.fetch.start",
+        )
 
         try:
             async with httpx.AsyncClient(
@@ -78,11 +88,17 @@ class SpaceTrackTLECollector(BaseCollector):
                         "password": password,
                     },
                 )
-                print(f"SPACETRACK: Login response status: {login_response.status_code}")
-                print(f"SPACETRACK: Login response URL: {login_response.url}")
+                logger.info_event(
+                    "Space-Track login response received",
+                    event="collector.spacetrack.login.response",
+                    context={"status_code": login_response.status_code},
+                )
 
                 if login_response.status_code == 403:
-                    print("SPACETRACK: Trying alternate login method...")
+                    logger.warning_event(
+                        "Space-Track login returned forbidden; trying alternate method",
+                        event="collector.spacetrack.login.forbidden",
+                    )
 
                     async with httpx.AsyncClient(
                         timeout=120.0,
@@ -90,11 +106,6 @@ class SpaceTrackTLECollector(BaseCollector):
                     ) as alt_client:
                         await alt_client.get(f"{self.site_root}/")
 
-                        form_data = {
-                            "username": username,
-                            "password": password,
-                            "query": "class/gp/NORAD_CAT_ID/25544/format/json",
-                        }
                         alt_login = await alt_client.post(
                             self.login_url,
                             data={
@@ -102,77 +113,59 @@ class SpaceTrackTLECollector(BaseCollector):
                                 "password": password,
                             },
                         )
-                        print(f"SPACETRACK: Alt login status: {alt_login.status_code}")
+                        logger.info_event(
+                            "Space-Track alternate login response received",
+                            event="collector.spacetrack.alt_login.response",
+                            context={"status_code": alt_login.status_code},
+                        )
 
                         if alt_login.status_code == 200:
                             tle_response = await alt_client.get(self.probe_url)
                             if tle_response.status_code == 200:
                                 data = tle_response.json()
-                                print(f"SPACETRACK: Received {len(data)} records via alt method")
+                                logger.info_event(
+                                    "Space-Track alternate query completed",
+                                    event="collector.spacetrack.alt_query.completed",
+                                    context={"record_count": len(data)},
+                                )
                                 return data
 
                 if login_response.status_code != 200:
-                    print(f"SPACETRACK: Login failed, using sample data")
+                    logger.warning_event(
+                        "Space-Track login failed; using sample data",
+                        event="collector.spacetrack.login.failed",
+                        context={"status_code": login_response.status_code},
+                    )
                     return self._get_sample_data()
 
                 tle_response = await client.get(self.probe_url)
-                print(f"SPACETRACK: TLE query status: {tle_response.status_code}")
-
-                if tle_response.status_code != 200:
-                    print(f"SPACETRACK: Query failed, using sample data")
-                    return self._get_sample_data()
-
-                data = tle_response.json()
-                print(f"SPACETRACK: Received {len(data)} records")
-                return data
-        except Exception as e:
-            print(f"SPACETRACK: Error - {e}, using sample data")
-            return self._get_sample_data()
-
-        print(f"SPACETRACK: Attempting to fetch TLE data with username: {username}")
-
-        try:
-            async with httpx.AsyncClient(
-                timeout=120.0,
-                follow_redirects=True,
-                headers={
-                    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
-                    "Accept": "application/json, text/html, */*",
-                    "Accept-Language": "en-US,en;q=0.9",
-                },
-            ) as client:
-                # First, visit the main page to get any cookies
-                await client.get(f"{self.site_root}/")
-
-                # Login to get session cookie
-                login_response = await client.post(
-                    self.login_url,
-                    data={
-                        "identity": username,
-                        "password": password,
-                    },
+                logger.info_event(
+                    "Space-Track TLE query response received",
+                    event="collector.spacetrack.query.response",
+                    context={"status_code": tle_response.status_code},
                 )
-                print(f"SPACETRACK: Login response status: {login_response.status_code}")
-                print(f"SPACETRACK: Login response URL: {login_response.url}")
-                print(f"SPACETRACK: Login response body: {login_response.text[:500]}")
-
-                if login_response.status_code != 200:
-                    print(f"SPACETRACK: Login failed, using sample data")
-                    return self._get_sample_data()
-
-                # Query for TLE data (get first 1000 satellites)
-                tle_response = await client.get(self.query_url)
-                print(f"SPACETRACK: TLE query status: {tle_response.status_code}")
 
                 if tle_response.status_code != 200:
-                    print(f"SPACETRACK: Query failed, using sample data")
+                    logger.warning_event(
+                        "Space-Track TLE query failed; using sample data",
+                        event="collector.spacetrack.query.failed",
+                        context={"status_code": tle_response.status_code},
+                    )
                     return self._get_sample_data()
 
                 data = tle_response.json()
-                print(f"SPACETRACK: Received {len(data)} records")
+                logger.info_event(
+                    "Space-Track TLE fetch completed",
+                    event="collector.spacetrack.fetch.completed",
+                    context={"record_count": len(data)},
+                )
                 return data
         except Exception as e:
-            print(f"SPACETRACK: Error - {e}, using sample data")
+            logger.warning_event(
+                "Space-Track TLE fetch failed; using sample data",
+                event="collector.spacetrack.fetch.failed",
+                context={"error": str(e)},
+            )
             return self._get_sample_data()
 
     def transform(self, raw_data: List[Dict[str, Any]]) -> List[Dict[str, Any]]:

backend/app/services/collectors/telegeography.py

@@ -163,32 +163,64 @@ class TeleGeographyLandingPointCollector(BaseCollector):
     data_type = "landing_point"
 
     async def fetch(self) -> List[Dict[str, Any]]:
-        """Fetch landing point data from GitHub mirror"""
-        url = self._resolved_url or ""
+        """Fetch landing point data, falling back when the old mirror disappears."""
+        config = get_data_sources_config()
+        sources = [
+            self._resolved_url or "",
+            str(config.get_yaml_value("telegeography.landing_point_url") or ""),
+            str(config.get_yaml_value("arcgis.landing_point_url") or ""),
+        ]
 
-        async with httpx.AsyncClient(timeout=60.0) as client:
-            response = await client.get(url)
-            response.raise_for_status()
-            return self.parse_response(response.json())
+        last_error: Exception | None = None
+        async with httpx.AsyncClient(timeout=60.0, follow_redirects=True) as client:
+            for url in dict.fromkeys(source for source in sources if source):
+                try:
+                    params = (
+                        {"where": "1=1", "outFields": "*", "returnGeometry": "true", "f": "geojson"}
+                        if "FeatureServer" in url or url.endswith("/query")
+                        else None
+                    )
+                    response = await client.get(url, params=params)
+                    response.raise_for_status()
+                    records = self.parse_response(response.json())
+                    if records:
+                        return records
+                except Exception as exc:
+                    last_error = exc
+                    continue
 
-    def parse_response(self, data: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
+        if last_error:
+            raise last_error
+        return self._get_sample_data()
+
+    def parse_response(self, data: List[Dict[str, Any]] | Dict[str, Any]) -> List[Dict[str, Any]]:
         """Parse landing point data"""
         result = []
+        items = data.get("features", []) if isinstance(data, dict) else data
 
-        for item in data:
+        for item in items:
+            props = item.get("properties", {}) if isinstance(item, dict) else {}
+            geometry = item.get("geometry", {}) if isinstance(item, dict) else {}
+            source = props or item
+            coords = geometry.get("coordinates", []) if isinstance(geometry, dict) else []
+            longitude = coords[0] if len(coords) > 0 else source.get("longitude")
+            latitude = coords[1] if len(coords) > 1 else source.get("latitude")
+            source_id = source.get("id") or source.get("OBJECTID") or source.get("city_id") or ""
             try:
                 entry = {
-                    "source_id": f"telegeo_lp_{item.get('id', '')}",
-                    "name": item.get("name", "Unknown"),
-                    "country": item.get("country", "Unknown"),
-                    "city": item.get("city", item.get("name", "")),
-                    "latitude": str(item.get("latitude", "")),
-                    "longitude": str(item.get("longitude", "")),
+                    "source_id": f"telegeo_lp_{source_id}",
+                    "name": source.get("name", source.get("Name", "Unknown")),
+                    "country": source.get("country", "Unknown"),
+                    "city": source.get("city", source.get("Name", source.get("name", ""))),
+                    "latitude": str(latitude or ""),
+                    "longitude": str(longitude or ""),
                     "value": "",
                     "unit": "",
                     "metadata": {
-                        "cable_count": len(item.get("cables", [])),
-                        "url": item.get("url"),
+                        "cable_count": len(source.get("cables", [])),
+                        "url": source.get("url"),
+                        "objectid": source.get("OBJECTID"),
+                        "city_id": source.get("city_id"),
                     },
                     "reference_date": datetime.now(UTC).strftime("%Y-%m-%d"),
                 }

backend/app/services/collectors/vessel_ais.py

@@ -6,6 +6,7 @@ from typing import Any
 import httpx
 from sqlalchemy.ext.asyncio import AsyncSession
 
+from app.core.enums import SnapshotStatus
 from app.core.time import to_iso8601_utc
 from app.core.websocket.broadcaster import broadcaster
 from app.services.barentswatch import (
@@ -119,6 +120,21 @@ class VesselAISCollector(BaseCollector):
             last_success_at=now if data else None,
             lag_seconds=max((now - latest_observed_at).total_seconds(), 0),
         )
+        if snapshot_id is not None:
+            from app.models.data_snapshot import DataSnapshot
+
+            snapshot = await db.get(DataSnapshot, snapshot_id)
+            if snapshot:
+                snapshot.record_count = records_added
+                snapshot.status = SnapshotStatus.SUCCESS.value
+                snapshot.completed_at = now
+                snapshot.summary = {
+                    "created": records_added,
+                    "updated": 0,
+                    "unchanged": 0,
+                    "deleted": 0,
+                    "storage": "ais_raw_observations",
+                }
         await db.commit()
         await self._broadcast_vessel_snapshot(data)
         await self.update_progress(records_added, force=True)

backend/app/services/compute_center_locations.py

@@ -11,7 +11,7 @@ startup, so it must stay local and deterministic.
 
 For the full design and the reason behind the abstraction (compute centers,
 BGP collectors, BGP events, and future entities all share one pipeline),
-see ``docs/plans/location-resolver-shared-pipeline-plan.md``.
+see ``docs/technical/zh/location-pipeline-development.md``.
 
 The ``ComputeCenterLocation`` dataclass and the public function signatures are
 preserved verbatim so existing callers and tests do not need to change.

backend/app/services/data_jobs.py

@@ -0,0 +1,720 @@
+"""Kafka-ready datasource job queue backed by PostgreSQL for v1."""
+
+from __future__ import annotations
+
+import asyncio
+from datetime import UTC, datetime, timedelta
+from uuid import uuid4
+from typing import Any
+
+from sqlalchemy import bindparam, select, text
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.core.cache import cache
+from app.core.config import settings
+from app.core.enums import JobStatus, JobType, RollbackPolicy
+from app.core.logging import get_logger
+from app.core.time import to_iso8601_utc
+from app.core.websocket.broadcaster import broadcaster
+from app.db.session import async_session_factory
+from app.models.collected_data import CollectedData
+from app.models.data_snapshot import DataSnapshot
+from app.models.datasource import DataSource
+from app.models.task import CollectionTask
+from app.services.collectors.registry import collector_registry
+from app.services.datasource_connectivity import (
+    build_builtin_connectivity_checksum,
+    get_builtin_effective_candidate,
+    save_connectivity_success,
+)
+from app.services.earth_layer_adapters import (
+    clear_derived_datasource_data,
+    get_earth_refresh_strategy_for_change,
+    get_earth_update_layers_for_source,
+)
+from app.services.earth_layer_cache import invalidate_earth_layer_cache_for_source
+from app.services.scheduler import sync_datasource_job
+
+logger = get_logger(__name__)
+
+JOB_TYPE_COLLECT = JobType.COLLECT.value
+JOB_TYPE_CLEAR_DATA = JobType.CLEAR_DATA.value
+JOB_TYPE_CLEAR_CACHE = JobType.CLEAR_CACHE.value
+JOB_TYPE_EARTH_REFRESH = JobType.EARTH_REFRESH.value
+
+JOB_STATUS_QUEUED = JobStatus.QUEUED.value
+JOB_STATUS_RUNNING = JobStatus.RUNNING.value
+JOB_STATUS_CANCELLING = JobStatus.CANCELLING.value
+JOB_STATUS_SUCCESS = JobStatus.SUCCESS.value
+JOB_STATUS_FAILED = JobStatus.FAILED.value
+JOB_STATUS_CANCELLED = JobStatus.CANCELLED.value
+
+ACTIVE_JOB_STATUSES = (JOB_STATUS_QUEUED, JOB_STATUS_RUNNING, JOB_STATUS_CANCELLING)
+TERMINAL_JOB_STATUSES = (JOB_STATUS_SUCCESS, JOB_STATUS_FAILED, JOB_STATUS_CANCELLED)
+DATA_WRITE_JOB_TYPES = (JOB_TYPE_COLLECT, JOB_TYPE_CLEAR_DATA, JOB_TYPE_CLEAR_CACHE)
+SOURCE_LOCK_JOB_STATUSES = (JOB_STATUS_RUNNING, JOB_STATUS_CANCELLING)
+QUEUE_POLL_SECONDS = 0.35
+JOB_STALE_LOCK_MINUTES = 90
+ORPHAN_CANCELLING_GRACE_SECONDS = 30
+JOB_RECOVERY_SWEEP_SECONDS = 15
+DATA_DELETE_BATCH_SIZE = 50_000
+DEFAULT_WORKER_CONCURRENCY = 2
+
+RUNNING_DATA_JOB_TASKS: dict[int, asyncio.Task[Any]] = {}
+
+
+def _utcnow() -> datetime:
+    return datetime.now(UTC)
+
+
+def _job_worker_id() -> str:
+    return f"{settings.PROJECT_NAME}:data-job-worker:{uuid4().hex[:8]}"
+
+
+def is_terminal_job_status(status: str | None) -> bool:
+    return status in TERMINAL_JOB_STATUSES
+
+
+async def enqueue_datasource_job(
+    db: AsyncSession,
+    datasource: DataSource,
+    task_type: str,
+    *,
+    payload: dict[str, Any] | None = None,
+    rollback_policy: str = RollbackPolicy.KEEP_COMMITTED_BATCHES.value,
+    dedupe_key: str | None = None,
+) -> CollectionTask:
+    if dedupe_key:
+        existing = await _get_active_job_by_dedupe_key(db, dedupe_key)
+        if existing is not None:
+            return existing
+
+    task = CollectionTask(
+        datasource_id=datasource.id,
+        source=datasource.source,
+        task_type=task_type,
+        status=JOB_STATUS_QUEUED,
+        phase="queued",
+        phase_message="任务已进入队列",
+        payload=payload or {},
+        rollback_policy=rollback_policy,
+        dedupe_key=dedupe_key,
+    )
+    db.add(task)
+    await db.commit()
+    await db.refresh(task)
+    await _broadcast_task_update(task)
+    return task
+
+
+async def enqueue_earth_refresh_job(
+    db: AsyncSession,
+    *,
+    source: str,
+    payload: dict[str, Any] | None = None,
+) -> CollectionTask | None:
+    layers = list((payload or {}).get("layers") or get_earth_update_layers_for_source(source))
+    if not layers:
+        return None
+
+    datasource = await _get_or_create_virtual_datasource(db, source)
+    refresh_payload = {
+        "source": source,
+        "layers": layers,
+        "refresh_strategy": (payload or {}).get("refresh_strategy")
+        or get_earth_refresh_strategy_for_change((payload or {}).get("table"), source)
+        or "clear_then_reload",
+        **(payload or {}),
+    }
+    return await enqueue_datasource_job(
+        db,
+        datasource,
+        JOB_TYPE_EARTH_REFRESH,
+        payload=refresh_payload,
+        dedupe_key=f"earth_refresh:{source}",
+    )
+
+
+async def enqueue_earth_refresh_from_update(payload: dict[str, Any]) -> None:
+    source = str(payload.get("source") or "").strip()
+    if not source:
+        return
+    async with async_session_factory() as db:
+        await enqueue_earth_refresh_job(db, source=source, payload=payload)
+
+
+async def request_cancel_datasource_task(
+    db: AsyncSession,
+    task: CollectionTask,
+    *,
+    reason: str = "cancelled_by_operator",
+) -> CollectionTask:
+    if is_terminal_job_status(task.status):
+        return task
+
+    running_task = RUNNING_DATA_JOB_TASKS.get(task.id)
+    if task.status == JOB_STATUS_QUEUED or (
+        running_task is None
+        and (
+            task.status == JOB_STATUS_CANCELLING
+            or (task.status == JOB_STATUS_RUNNING and task.task_type != JOB_TYPE_COLLECT)
+        )
+    ):
+        return await _cancel_task_without_runner(db, task, reason=reason)
+
+    task.status = JOB_STATUS_CANCELLING
+    task.phase = JOB_STATUS_CANCELLING
+    task.phase_message = "正在停止任务"
+    task.requested_cancel_at = _utcnow()
+    task.cancel_reason = reason
+    await db.commit()
+    await db.refresh(task)
+
+    if running_task is not None and not running_task.done():
+        running_task.cancel()
+
+    await _broadcast_task_update(task)
+    return task
+
+
+async def _cancel_task_without_runner(
+    db: AsyncSession,
+    task: CollectionTask,
+    *,
+    reason: str,
+) -> CollectionTask:
+    if task.task_type == JOB_TYPE_COLLECT:
+        await db.execute(CollectedData.__table__.delete().where(CollectedData.task_id == task.id))
+        snapshot_result = await db.execute(select(DataSnapshot).where(DataSnapshot.task_id == task.id))
+        for snapshot in snapshot_result.scalars().all():
+            snapshot.status = JOB_STATUS_CANCELLED
+            snapshot.completed_at = _utcnow()
+            snapshot.error_message = "Cancelled after operator stop request; no active worker handle remained"
+        datasource = await db.get(DataSource, task.datasource_id)
+        if datasource is not None:
+            datasource.last_status = JOB_STATUS_CANCELLED
+
+    task.status = JOB_STATUS_CANCELLED
+    task.phase = JOB_STATUS_CANCELLED
+    task.phase_message = "任务已停止"
+    task.completed_at = _utcnow()
+    task.requested_cancel_at = task.requested_cancel_at or _utcnow()
+    task.cancel_reason = reason
+    await db.commit()
+    await db.refresh(task)
+    await _broadcast_task_update(task)
+    return task
+
+
+async def get_active_datasource_job(
+    db: AsyncSession,
+    datasource_id: int,
+    *,
+    task_types: tuple[str, ...] = DATA_WRITE_JOB_TYPES,
+) -> CollectionTask | None:
+    result = await db.execute(
+        select(CollectionTask)
+        .where(CollectionTask.datasource_id == datasource_id)
+        .where(CollectionTask.task_type.in_(task_types))
+        .where(CollectionTask.status.in_(ACTIVE_JOB_STATUSES))
+        .order_by(CollectionTask.created_at.desc().nullslast(), CollectionTask.id.desc())
+        .limit(1)
+    )
+    return result.scalar_one_or_none()
+
+
+async def _get_active_job_by_dedupe_key(db: AsyncSession, dedupe_key: str) -> CollectionTask | None:
+    result = await db.execute(
+        select(CollectionTask)
+        .where(CollectionTask.dedupe_key == dedupe_key)
+        .where(CollectionTask.status.in_(ACTIVE_JOB_STATUSES))
+        .order_by(CollectionTask.created_at.desc().nullslast(), CollectionTask.id.desc())
+        .limit(1)
+    )
+    return result.scalar_one_or_none()
+
+
+async def _get_or_create_virtual_datasource(db: AsyncSession, source: str) -> DataSource:
+    result = await db.execute(select(DataSource).where(DataSource.source == source))
+    datasource = result.scalar_one_or_none()
+    if datasource is not None:
+        return datasource
+
+    datasource = DataSource(
+        name=f"Earth refresh: {source}",
+        source=source,
+        module="SYS",
+        collector_class="EarthRefreshJob",
+        is_active=True,
+    )
+    db.add(datasource)
+    await db.commit()
+    await db.refresh(datasource)
+    return datasource
+
+
+async def _broadcast_task_update(task: CollectionTask) -> None:
+    await broadcaster.broadcast_datasource_task_update(
+        {
+            "datasource_id": task.datasource_id,
+            "collector_name": task.source,
+            "task_id": task.id,
+            "task_type": task.task_type,
+            "status": task.status,
+            "phase": task.phase,
+            "phase_progress": task.phase_progress,
+            "phase_message": task.phase_message,
+            "phase_current": task.phase_current,
+            "phase_total": task.phase_total,
+            "phase_unit": task.phase_unit,
+            "progress": task.progress,
+            "records_processed": task.records_processed,
+            "total_records": task.total_records,
+            "started_at": to_iso8601_utc(task.started_at),
+            "completed_at": to_iso8601_utc(task.completed_at),
+            "requested_cancel_at": to_iso8601_utc(task.requested_cancel_at),
+            "error_message": task.error_message,
+        }
+    )
+
+
+class DataJobWorker:
+    def __init__(self, *, concurrency: int = DEFAULT_WORKER_CONCURRENCY) -> None:
+        self.worker_id = _job_worker_id()
+        self.concurrency = max(1, concurrency)
+        self._task: asyncio.Task[None] | None = None
+        self._stop_event: asyncio.Event | None = None
+        self._running: set[asyncio.Task[Any]] = set()
+        self._last_recovery_sweep_at: datetime | None = None
+
+    def start(self) -> None:
+        if self._task and not self._task.done():
+            return
+        self._stop_event = asyncio.Event()
+        self._task = asyncio.create_task(self._run(), name="data-job-worker")
+
+    async def stop(self) -> None:
+        if self._stop_event:
+            self._stop_event.set()
+        for task in list(self._running):
+            task.cancel()
+        if self._task:
+            await asyncio.gather(self._task, return_exceptions=True)
+        if self._running:
+            await asyncio.gather(*self._running, return_exceptions=True)
+
+    async def _run(self) -> None:
+        assert self._stop_event is not None
+        await self._recover_stale_running_jobs()
+        while not self._stop_event.is_set():
+            self._running = {task for task in self._running if not task.done()}
+            if (
+                self._last_recovery_sweep_at is None
+                or (_utcnow() - self._last_recovery_sweep_at).total_seconds() >= JOB_RECOVERY_SWEEP_SECONDS
+            ):
+                await self._recover_stale_running_jobs()
+            if len(self._running) >= self.concurrency:
+                await asyncio.sleep(QUEUE_POLL_SECONDS)
+                continue
+
+            task_id = await self._claim_next_job()
+            if task_id is None:
+                await asyncio.sleep(QUEUE_POLL_SECONDS)
+                continue
+
+            runner = asyncio.create_task(self._run_claimed_job(task_id), name=f"data-job:{task_id}")
+            self._running.add(runner)
+
+    async def _recover_stale_running_jobs(self) -> None:
+        self._last_recovery_sweep_at = _utcnow()
+        cutoff = _utcnow() - timedelta(minutes=JOB_STALE_LOCK_MINUTES)
+        orphan_cancelling_cutoff = _utcnow() - timedelta(seconds=ORPHAN_CANCELLING_GRACE_SECONDS)
+        async with async_session_factory() as db:
+            result = await db.execute(
+                select(CollectionTask)
+                .where(CollectionTask.status.in_((JOB_STATUS_RUNNING, JOB_STATUS_CANCELLING)))
+                .where(CollectionTask.locked_at.is_not(None))
+                .where(CollectionTask.locked_at < cutoff)
+            )
+            stale_jobs = list(result.scalars().all())
+            for job in stale_jobs:
+                job.status = JOB_STATUS_FAILED
+                job.phase = JOB_STATUS_FAILED
+                job.completed_at = _utcnow()
+                job.error_message = "Marked failed after stale data job lock timeout"
+            if stale_jobs:
+                await db.commit()
+            orphan_result = await db.execute(
+                select(CollectionTask)
+                .where(CollectionTask.status == JOB_STATUS_CANCELLING)
+                .where(CollectionTask.locked_at.is_(None))
+                .where(CollectionTask.requested_cancel_at.is_not(None))
+                .where(CollectionTask.requested_cancel_at < orphan_cancelling_cutoff)
+            )
+            for job in orphan_result.scalars().all():
+                if job.id in RUNNING_DATA_JOB_TASKS:
+                    continue
+                await _cancel_task_without_runner(
+                    db,
+                    job,
+                    reason=job.cancel_reason or "cancelled_after_orphaned_runner",
+                )
+
+    async def _claim_next_job(self) -> int | None:
+        async with async_session_factory() as db:
+            row = await db.execute(
+                text(
+                    """
+                    SELECT queued.id
+                    FROM collection_tasks AS queued
+                    WHERE queued.status = :queued_status
+                      AND NOT EXISTS (
+                        SELECT 1
+                        FROM collection_tasks AS active
+                        WHERE active.source = queued.source
+                          AND active.id <> queued.id
+                          AND active.status IN :active_statuses
+                      )
+                    ORDER BY queued.created_at ASC NULLS FIRST, queued.id ASC
+                    LIMIT 1
+                    FOR UPDATE SKIP LOCKED
+                    """
+                ).bindparams(bindparam("active_statuses", expanding=True)),
+                {
+                    "queued_status": JOB_STATUS_QUEUED,
+                    "active_statuses": SOURCE_LOCK_JOB_STATUSES,
+                },
+            )
+            task_id = row.scalar_one_or_none()
+            if task_id is None:
+                return None
+
+            task = await db.get(CollectionTask, int(task_id))
+            if task is None:
+                return None
+            task.status = JOB_STATUS_RUNNING
+            task.phase = "starting"
+            task.phase_message = "任务开始执行"
+            task.started_at = task.started_at or _utcnow()
+            task.worker_id = self.worker_id
+            task.locked_at = _utcnow()
+            await db.commit()
+            await _broadcast_task_update(task)
+            return int(task_id)
+
+    async def _run_claimed_job(self, task_id: int) -> None:
+        current_task = asyncio.current_task()
+        if current_task is not None:
+            RUNNING_DATA_JOB_TASKS[task_id] = current_task
+        try:
+            async with async_session_factory() as db:
+                task = await db.get(CollectionTask, task_id)
+                if task is None:
+                    return
+                await self._execute_job(db, task)
+        except asyncio.CancelledError:
+            async with async_session_factory() as db:
+                task = await db.get(CollectionTask, task_id)
+                if task is not None and not is_terminal_job_status(task.status):
+                    task.status = JOB_STATUS_CANCELLED
+                    task.phase = JOB_STATUS_CANCELLED
+                    task.phase_message = "任务已停止"
+                    task.completed_at = _utcnow()
+                    await db.commit()
+                    await _broadcast_task_update(task)
+            raise
+        except Exception as exc:
+            logger.exception_event(
+                "Data job failed",
+                event="data_jobs.job_failed",
+                context={"task_id": task_id, "error": str(exc)},
+            )
+            async with async_session_factory() as db:
+                task = await db.get(CollectionTask, task_id)
+                if task is not None:
+                    task.status = JOB_STATUS_FAILED
+                    task.phase = JOB_STATUS_FAILED
+                    task.phase_message = str(exc)
+                    task.error_message = str(exc)
+                    task.completed_at = _utcnow()
+                    await db.commit()
+                    await _broadcast_task_update(task)
+        finally:
+            RUNNING_DATA_JOB_TASKS.pop(task_id, None)
+
+    async def _execute_job(self, db: AsyncSession, task: CollectionTask) -> None:
+        if task.task_type == JOB_TYPE_COLLECT:
+            await _run_collect_job(db, task)
+        elif task.task_type == JOB_TYPE_CLEAR_DATA:
+            await _run_clear_data_job(db, task)
+        elif task.task_type == JOB_TYPE_CLEAR_CACHE:
+            await _run_clear_cache_job(db, task)
+        elif task.task_type == JOB_TYPE_EARTH_REFRESH:
+            await _run_earth_refresh_job(db, task)
+        else:
+            raise RuntimeError(f"Unsupported data job type: {task.task_type}")
+
+
+async def _run_collect_job(db: AsyncSession, task: CollectionTask) -> None:
+    datasource = await db.get(DataSource, task.datasource_id)
+    if datasource is None:
+        raise RuntimeError("Data source not found")
+
+    collector = collector_registry.get(datasource.source)
+    if collector is None:
+        raise RuntimeError(f"Collector '{datasource.source}' not found")
+    if not datasource.is_active:
+        raise RuntimeError("Data source is disabled")
+
+    collector._datasource_id = datasource.id
+    collector._current_task = task
+    collector._db_session = db
+    result = await collector.run(db)
+
+    datasource.last_run_at = _utcnow()
+    datasource.last_status = result.get("status")
+    if datasource.last_status == JOB_STATUS_SUCCESS:
+        effective_candidate = await get_builtin_effective_candidate(db, datasource.source)
+        checksum, _credential_context = await build_builtin_connectivity_checksum(
+            datasource.source,
+            effective_candidate["endpoint"],
+            effective_candidate["auth_type"],
+            effective_candidate["headers"],
+            effective_candidate["config"],
+            db,
+        )
+        await save_connectivity_success(
+            db,
+            datasource.source,
+            checksum,
+            {"status_code": None},
+            connected_by="collection",
+        )
+    await db.commit()
+    await sync_datasource_job(datasource.id)
+
+
+async def _run_clear_data_job(db: AsyncSession, task: CollectionTask) -> None:
+    source = str(task.source or (task.payload or {}).get("source") or "").strip()
+    if not source:
+        raise RuntimeError("Clear data job has no source")
+
+    task.phase = "clearing_data"
+    task.phase_message = "正在删除数据库数据"
+    await db.commit()
+    await _broadcast_task_update(task)
+
+    deleted_count = await _delete_table_rows_by_source(
+        db,
+        task,
+        table_name="collected_data",
+        source_column="source",
+        source=source,
+    )
+    derived_deleted_counts = await _clear_derived_datasource_data_in_batches(
+        db,
+        task,
+        source,
+        progress_offset=deleted_count,
+    )
+    derived_deleted_count = sum(derived_deleted_counts.values())
+    if any(key.startswith("ais_") for key in derived_deleted_counts):
+        await db.execute(text("ANALYZE ais_raw_observations"))
+
+    task.records_processed = deleted_count + derived_deleted_count
+    task.total_records = task.records_processed
+    task.progress = 100.0
+    task.phase_progress = 100.0
+    task.phase_current = task.records_processed
+    task.phase_total = task.records_processed
+    task.phase_unit = "records"
+    task.payload = {
+        **(task.payload or {}),
+        "deleted_count": deleted_count,
+        "derived_deleted_count": derived_deleted_count,
+        "derived_deleted_counts": derived_deleted_counts,
+    }
+    task.status = JOB_STATUS_SUCCESS
+    task.phase = "completed"
+    task.phase_message = "数据库数据已清理"
+    task.completed_at = _utcnow()
+    datasource = await db.get(DataSource, task.datasource_id)
+    if datasource is not None:
+        datasource.last_status = JOB_STATUS_SUCCESS
+        datasource.last_run_at = task.completed_at
+    await db.execute(
+        DataSnapshot.__table__.update()
+        .where(DataSnapshot.source == source)
+        .values(is_current=False)
+    )
+    await db.commit()
+    await _broadcast_task_update(task)
+
+
+async def _delete_table_rows_by_source(
+    db: AsyncSession,
+    task: CollectionTask,
+    *,
+    table_name: str,
+    source_column: str,
+    source: str,
+    progress_offset: int = 0,
+) -> int:
+    deleted = 0
+    while True:
+        result = await db.execute(
+            text(
+                f"""
+                WITH doomed AS (
+                    SELECT ctid
+                    FROM {table_name}
+                    WHERE {source_column} = :source
+                    LIMIT :batch_size
+                ),
+                deleted_rows AS (
+                    DELETE FROM {table_name}
+                    USING doomed
+                    WHERE {table_name}.ctid = doomed.ctid
+                    RETURNING 1
+                )
+                SELECT COUNT(*) FROM deleted_rows
+                """
+            ),
+            {"source": source, "batch_size": DATA_DELETE_BATCH_SIZE},
+        )
+        batch_deleted = max(int(result.scalar_one() or 0), 0)
+        if batch_deleted <= 0:
+            break
+        deleted += batch_deleted
+        task.records_processed = progress_offset + deleted
+        task.phase_current = task.records_processed
+        task.phase_unit = "records"
+        task.phase_message = f"正在删除数据：{task.records_processed} 条"
+        await db.commit()
+        await _broadcast_task_update(task)
+    return deleted
+
+
+async def _clear_derived_datasource_data_in_batches(
+    db: AsyncSession,
+    task: CollectionTask,
+    source: str,
+    progress_offset: int = 0,
+) -> dict[str, int]:
+    deleted_counts: dict[str, int] = {}
+    if source in {"barentswatch_vessels", "aisstream_vessels"}:
+        deleted_counts["ais_conflict_records"] = await _delete_table_rows_by_source(
+            db,
+            task,
+            table_name="ais_conflict_records",
+            source_column="selected_source",
+            source=source,
+            progress_offset=progress_offset + sum(deleted_counts.values()),
+        )
+        deleted_counts["ais_source_health"] = await _delete_table_rows_by_source(
+            db,
+            task,
+            table_name="ais_source_health",
+            source_column="source",
+            source=source,
+            progress_offset=progress_offset + sum(deleted_counts.values()),
+        )
+        deleted_counts["ais_raw_observations"] = await _delete_table_rows_by_source(
+            db,
+            task,
+            table_name="ais_raw_observations",
+            source_column="source",
+            source=source,
+            progress_offset=progress_offset + sum(deleted_counts.values()),
+        )
+        return deleted_counts
+    return await clear_derived_datasource_data(db, source)
+
+
+async def _run_clear_cache_job(db: AsyncSession, task: CollectionTask) -> None:
+    source = str(task.source or (task.payload or {}).get("source") or "").strip()
+    if not source:
+        raise RuntimeError("Clear cache job has no source")
+
+    earth_deleted_count = invalidate_earth_layer_cache_for_source(source)
+    dashboard_deleted_count = int(cache.delete("dashboard:stats")) + int(cache.delete("dashboard:summary"))
+    deleted_count = earth_deleted_count + dashboard_deleted_count
+
+    task.records_processed = deleted_count
+    task.total_records = deleted_count
+    task.progress = 100.0
+    task.phase_progress = 100.0
+    task.phase = "completed"
+    task.phase_message = "缓存已清理"
+    task.payload = {
+        **(task.payload or {}),
+        "earth_layer_deleted_count": earth_deleted_count,
+        "dashboard_deleted_count": dashboard_deleted_count,
+    }
+    task.status = JOB_STATUS_SUCCESS
+    task.completed_at = _utcnow()
+    await db.commit()
+    await _broadcast_task_update(task)
+    await enqueue_earth_refresh_job(db, source=source, payload={"operation": "CACHE_INVALIDATED"})
+
+
+async def _run_earth_refresh_job(db: AsyncSession, task: CollectionTask) -> None:
+    payload = task.payload or {}
+    source = str(payload.get("source") or task.source or "").strip()
+    layers = list(payload.get("layers") or get_earth_update_layers_for_source(source))
+    if not source or not layers:
+        task.status = JOB_STATUS_SUCCESS
+        task.phase = "completed"
+        task.phase_message = "没有需要刷新的 Earth 图层"
+        task.completed_at = _utcnow()
+        await db.commit()
+        await _broadcast_task_update(task)
+        return
+
+    deleted_cache_entries = invalidate_earth_layer_cache_for_source(source)
+    update_payload = {
+        "event": "earth.layer.changed",
+        "action": "database_changed",
+        "source": source,
+        "table": payload.get("table"),
+        "data_type": source,
+        "layers": layers,
+        "refresh_strategy": payload.get("refresh_strategy") or "clear_then_reload",
+        "records_processed": payload.get("records_processed", 0),
+        "operations": payload.get("operations") or [payload.get("operation") or "CHANGE"],
+        "operation": payload.get("operation"),
+        "cache_entries_invalidated": deleted_cache_entries,
+        "timestamp": to_iso8601_utc(_utcnow()),
+    }
+    if payload.get("entity") == "interactable":
+        update_payload.update(
+            {
+                "entity": "interactable",
+                "action": payload.get("action") or "changed",
+                "ids": payload.get("ids") or payload.get("entity_keys") or [],
+                "item": payload.get("item"),
+            }
+        )
+    await broadcaster.broadcast_earth_update(update_payload)
+
+    task.records_processed = int(payload.get("records_processed") or 0)
+    task.progress = 100.0
+    task.phase_progress = 100.0
+    task.phase = "completed"
+    task.phase_message = "Earth 图层刷新通知已发送"
+    task.status = JOB_STATUS_SUCCESS
+    task.completed_at = _utcnow()
+    task.payload = {**payload, "cache_entries_invalidated": deleted_cache_entries}
+    await db.commit()
+    await _broadcast_task_update(task)
+
+
+_worker = DataJobWorker()
+
+
+def start_data_job_worker() -> None:
+    _worker.start()
+
+
+async def stop_data_job_worker() -> None:
+    await _worker.stop()

backend/app/services/datasource_connectivity.py

@@ -13,6 +13,7 @@ from sqlalchemy import func, select
 
 from app.core.data_sources import get_data_sources_config
 from app.core.datasource_defaults import DEFAULT_DATASOURCES
+from app.core.enums import JobStatus
 from app.models.collected_data import CollectedData
 from app.models.datasource import DataSource
 from app.models.datasource_config import DataSourceConfig
@@ -398,7 +399,7 @@ async def has_collected_data(db, source: str) -> bool:
 
     datasource_result = await db.execute(select(DataSource).where(DataSource.source == source))
     datasource = datasource_result.scalar_one_or_none()
-    return bool(datasource and datasource.last_status == "success")
+    return bool(datasource and datasource.last_status == JobStatus.SUCCESS.value)
 
 
 async def get_builtin_connection_status(

backend/app/services/docs_gatekeeper.py

@@ -6,6 +6,7 @@ from dataclasses import dataclass
 from pathlib import Path
 from typing import Literal
 
+from app.core.enums import UserRole
 from app.models.user import User
 
 DocsAccess = Literal["public", "docs_user", "docs_developer", "docs_admin"]
@@ -32,28 +33,34 @@ class DocsMetadata:
 
 DOCS_METADATA: tuple[DocsMetadata, ...] = (
     DocsMetadata(DOCS_README_FILENAME, DEFAULT_DOCS_SLUG, "public", "Overview", 0, "技术文档", "Technical Docs"),
-    DocsMetadata("quickstart.md", "quickstart", "public", "Manual", 1, "快速开始", "Quickstart"),
-    DocsMetadata("manual.md", "manual", "public", "Manual", 2, "Planet 使用手册", "Planet Manual"),
+    DocsMetadata("manual.md", "manual", "public", "Manual", 1, "智能星球使用手册", "Intelligent Planet Manual"),
+    DocsMetadata("quickstart.md", "quickstart", "public", "Manual", 2, "快速开始", "Quickstart"),
     DocsMetadata("faq.md", "faq", "public", "Manual", 3, "常见问题", "FAQ"),
-    DocsMetadata("location-pipeline-user.md", "location-pipeline-user", "public", "Manual", 4, "Earth 位置候选采集使用手册", "Earth Location Candidate Collection User Guide"),
-    DocsMetadata("earth-frontend-context.md", "earth-frontend-context", "docs_developer", "Earth", 10, "Earth 前端结构", "Earth Frontend Context"),
-    DocsMetadata("earth-layer-style-reference.md", "earth-layer-style-reference", "docs_developer", "Earth", 11, "Earth 图层样式属性索引", "Earth Layer Style Reference"),
-    DocsMetadata("earth-render-layer-order.md", "earth-render-layer-order", "docs_developer", "Earth", 12, "Earth 渲染图层顺序", "Earth Render Layer Order"),
-    DocsMetadata("earth-satellite-footprint-policy.md", "earth-satellite-footprint-policy", "docs_developer", "Earth", 13, "Earth 卫星覆盖策略", "Earth Satellite Footprint Policy"),
+    DocsMetadata("platform-data-flows.md", "platform-data-flows", "docs_developer", "Architecture", 5, "业务架构与数据流转", "Business Architecture and Data Flows"),
+    DocsMetadata("naming-glossary.md", "naming-glossary", "docs_developer", "Architecture", 6, "命名与术语对照", "Naming Glossary"),
+    DocsMetadata("earth-frontend-context.md", "earth-frontend-context", "docs_developer", "Earth", 10, "智能星球前端结构", "Intelligent Planet Frontend Context"),
+    DocsMetadata("earth-layer-style-reference.md", "earth-layer-style-reference", "docs_developer", "Earth", 11, "智能星球图层样式属性索引", "Intelligent Planet Layer Style Reference"),
+    DocsMetadata("earth-render-layer-order.md", "earth-render-layer-order", "docs_developer", "Earth", 12, "智能星球渲染图层顺序", "Intelligent Planet Render Layer Order"),
+    DocsMetadata("earth-satellite-footprint-policy.md", "earth-satellite-footprint-policy", "docs_developer", "Earth", 13, "智能星球卫星覆盖策略", "Intelligent Planet Satellite Footprint Policy"),
     DocsMetadata("earth-bgp-context.md", "earth-bgp-context", "docs_developer", "Earth", 14, "BGP 态势上下文", "BGP Context"),
-    DocsMetadata("earth-news-live-streams-collector-format.md", "earth-news-live-streams-collector-format", "docs_developer", "Earth", 15, "新闻直播采集格式", "News Live Streams Collector Format"),
-    DocsMetadata("earth-interactable-usage.md", "earth-interactable-usage", "docs_developer", "Earth", 16, "Earth 可交互图标接入", "Earth Interactable Usage"),
-    DocsMetadata("earth-toolbar-overlay-coordination.md", "earth-toolbar-overlay-coordination", "docs_developer", "Earth", 17, "Earth 工具栏与浮层协同", "Earth Toolbar and Overlay Coordination"),
+    DocsMetadata("earth-interactable-usage.md", "earth-interactable-usage", "docs_developer", "Earth", 16, "智能星球可交互图标接入", "Intelligent Planet Interactable Usage"),
+    DocsMetadata("earth-interactable-clustering.md", "earth-interactable-clustering", "docs_developer", "Earth", 17, "智能星球可交互图标聚类策略", "Intelligent Planet Interactable Clustering"),
+    DocsMetadata("earth-toolbar-overlay-coordination.md", "earth-toolbar-overlay-coordination", "docs_developer", "Earth", 18, "智能星球工具栏与浮层协同", "Intelligent Planet Toolbar and Overlay Coordination"),
+    DocsMetadata("earth-news-sources.md", "earth-news-sources", "docs_developer", "Earth", 19, "智能星球新闻源配置", "Intelligent Planet News Source Configuration"),
     DocsMetadata("frontend-admin-frontend-context.md", "frontend-admin-frontend-context", "docs_developer", "Frontend", 20, "控制台前端结构", "Admin Frontend Context"),
     DocsMetadata("frontend-layout-guidelines.md", "frontend-layout-guidelines", "docs_developer", "Frontend", 21, "前端布局指南", "Frontend Layout Guidelines"),
-    DocsMetadata("docs-gatekeeper-development.md", "docs-gatekeeper-development", "docs_developer", "Frontend", 22, "Docs Gatekeeper 开发说明", "Docs Gatekeeper Development Guide"),
+    DocsMetadata("tactile-ui-components.md", "tactile-ui-components", "docs_developer", "Frontend", 24, "Tactile UI 组件库", "Tactile UI Components"),
     DocsMetadata("backend-collectors.md", "backend-collectors", "docs_developer", "Backend", 30, "数据采集系统", "Data Collectors"),
     DocsMetadata("backend-system-service-control.md", "backend-system-service-control", "docs_admin", "Backend", 31, "系统服务控制", "System Service Control"),
     DocsMetadata("datasource-collector-settings-connectivity.md", "datasource-collector-settings-connectivity", "docs_developer", "Backend", 32, "数据源、采集器设置与连接验证", "Datasource Collector Settings and Connectivity"),
     DocsMetadata("backend-datasources-api-performance.md", "backend-datasources-api-performance", "docs_developer", "Backend", 33, "数据源 API 性能", "Datasource API Performance"),
-    DocsMetadata("location-pipeline-development.md", "location-pipeline-development", "docs_developer", "Backend", 34, "通用位置估算管线开发说明", "Shared Location Resolution Pipeline Development Guide"),
+    DocsMetadata("data-job-earth-sync-architecture.md", "data-job-earth-sync-architecture", "docs_developer", "Backend", 34, "数据作业与 Outbox 技术架构", "Data Jobs and Outbox Architecture"),
+    DocsMetadata("backend-enum-contracts.md", "backend-enum-contracts", "docs_developer", "Backend", 35, "后端枚举与字符串兼容契约", "Backend Enum and String Compatibility Contract"),
+    DocsMetadata("location-pipeline-development.md", "location-pipeline-development", "docs_developer", "Backend", 35, "通用位置估算管线开发说明", "Shared Location Resolution Pipeline Development Guide"),
+    DocsMetadata("earth-news-live-streams-collector-format.md", "earth-news-live-streams-collector-format", "docs_developer", "Backend", 36, "新闻直播采集格式", "News Live Streams Collector Format"),
+    DocsMetadata("docs-gatekeeper-development.md", "docs-gatekeeper-development", "docs_developer", "Backend", 37, "Docs Gatekeeper 开发说明", "Docs Gatekeeper Development Guide"),
     DocsMetadata("agents-aiprovider.md", "agents-aiprovider", "docs_developer", "Agents", 40, "AI Provider 指南", "AI Provider Guide"),
-    DocsMetadata("ops-runbook.md", "ops-runbook", "docs_admin", "Ops", 49, "Planet 运维手册", "Planet Ops Runbook"),
+    DocsMetadata("ops-runbook.md", "ops-runbook", "docs_admin", "Ops", 49, "智能星球运维手册", "Intelligent Planet Ops Runbook"),
     DocsMetadata("ops-docker-compose-buildx-upgrade.md", "ops-docker-compose-buildx-upgrade", "docs_admin", "Ops", 50, "Docker + Compose + Buildx 升级", "Docker + Compose + Buildx Upgrade"),
     DocsMetadata("ops-planet-sh-startup.md", "ops-planet-sh-startup", "docs_admin", "Ops", 51, "planet.sh 启动机制", "planet.sh Startup"),
 )
@@ -66,9 +73,9 @@ def get_user_gatekeeper_groups(user: User | None) -> set[str]:
         return set()
 
     role = user.role.value if hasattr(user.role, "value") else str(user.role or "")
-    if role == "super_admin":
+    if role == UserRole.SUPER_ADMIN.value:
         return {"docs_user", "docs_developer", "docs_admin"}
-    if role == "admin":
+    if role == UserRole.ADMIN.value:
         return {"docs_user", "docs_developer", "docs_admin"}
 
     groups = set()

backend/app/services/earth_db_change_listener.py

@@ -0,0 +1,482 @@
+"""PostgreSQL LISTEN/NOTIFY bridge for Earth layer refresh events."""
+
+from __future__ import annotations
+
+import asyncio
+import json
+from collections import deque
+from dataclasses import dataclass, field
+from datetime import UTC, datetime
+from time import monotonic
+from typing import Any, Awaitable, Callable
+
+import asyncpg
+
+from app.core.config import settings
+from app.core.logging import get_logger
+from app.core.time import to_iso8601_utc
+from app.core.websocket.broadcaster import broadcaster
+from app.services.earth_layer_adapters import (
+    get_earth_refresh_strategy_for_change,
+    get_earth_update_layers_for_change,
+    get_earth_update_layers_for_source,
+)
+from app.services.earth_layer_cache import invalidate_earth_layer_cache_for_source
+
+logger = get_logger(__name__)
+
+EARTH_DATA_CHANGES_CHANNEL = "planet_earth_data_changes"
+DEFAULT_DEBOUNCE_SECONDS = 0.25
+DEFAULT_MAX_WAIT_SECONDS = 1.5
+DELETE_FAST_FLUSH_SECONDS = 0.05
+LISTEN_KEEPALIVE_SECONDS = 5.0
+OUTBOX_POLL_LIMIT = 5000
+MAX_ENTITY_KEY_SAMPLES = 20
+MAX_SEEN_EVENT_IDS = 20000
+
+BroadcastFn = Callable[[dict[str, Any]], Awaitable[None]]
+InvalidateFn = Callable[[str], int]
+
+
+def normalize_asyncpg_dsn(dsn: str) -> str:
+    """Convert SQLAlchemy asyncpg URLs into asyncpg-compatible URLs."""
+    return dsn.replace("postgresql+asyncpg://", "postgresql://", 1)
+
+
+def build_earth_update_from_db_payload(payload: dict[str, Any]) -> dict[str, Any] | None:
+    table = payload.get("table")
+    source = str(payload.get("source") or "").strip()
+    table_name = str(table or "").strip()
+    if not source and not table_name:
+        return None
+    layers = get_earth_update_layers_for_change(table_name, source)
+    if not layers:
+        return None
+    refresh_strategy = get_earth_refresh_strategy_for_change(table_name, source) or "clear_then_reload"
+    source_has_adapter = bool(get_earth_update_layers_for_source(source))
+    effective_source = source if source_has_adapter else (table_name if table_name else source)
+    operation = payload.get("operation")
+    update: dict[str, Any] = {
+        "event": "earth.layer.changed",
+        "action": "database_changed",
+        "source": effective_source,
+        "original_source": source or None,
+        "table": table_name or None,
+        "data_type": effective_source,
+        "layers": layers,
+        "refresh_strategy": refresh_strategy,
+        "operation": operation,
+        "entity_key": payload.get("entity_key"),
+        "timestamp": to_iso8601_utc(datetime.now(UTC)),
+    }
+    if table_name == "earth_interactables" and refresh_strategy == "delta":
+        ids = payload.get("entity_keys")
+        if not isinstance(ids, list):
+            ids = [payload.get("entity_key")] if payload.get("entity_key") else []
+        update.update(
+            {
+                "entity": "interactable",
+                "action": "deleted" if operation == "DELETE" else "changed",
+                "ids": [str(item) for item in ids if item],
+                "item": None,
+            }
+        )
+    return {
+        **update,
+    }
+
+
+@dataclass
+class PendingEarthDbChange:
+    source: str
+    layers: list[str]
+    table: str | None = None
+    refresh_strategy: str = "clear_then_reload"
+    entity: str | None = None
+    action: str = "database_changed"
+    records_processed: int = 0
+    operations: set[str] = field(default_factory=set)
+    entity_keys: list[str] = field(default_factory=list)
+    first_occurred_at: str | None = None
+    last_occurred_at: str | None = None
+    first_seen_monotonic: float = field(default_factory=monotonic)
+    last_seen_monotonic: float = field(default_factory=monotonic)
+
+    def add(self, payload: dict[str, Any]) -> None:
+        self.last_seen_monotonic = monotonic()
+        records_processed = payload.get("records_processed", 1)
+        try:
+            records_processed = int(records_processed)
+        except (TypeError, ValueError):
+            records_processed = 1
+        self.records_processed += max(records_processed, 1)
+        operation = payload.get("operation")
+        if operation:
+            self.operations.add(str(operation))
+        entity_keys = payload.get("entity_keys")
+        if not isinstance(entity_keys, list):
+            entity_key = payload.get("entity_key")
+            entity_keys = [entity_key] if entity_key else []
+        for entity_key in entity_keys:
+            if entity_key and len(self.entity_keys) < MAX_ENTITY_KEY_SAMPLES:
+                self.entity_keys.append(str(entity_key))
+        occurred_at = payload.get("occurred_at")
+        if occurred_at:
+            occurred_at = str(occurred_at)
+            self.first_occurred_at = self.first_occurred_at or occurred_at
+            self.last_occurred_at = occurred_at
+
+
+class EarthDbChangeDispatcher:
+    """Debounces database notifications and broadcasts Earth refresh hints."""
+
+    def __init__(
+        self,
+        *,
+        broadcast_earth_update: BroadcastFn | None = None,
+        invalidate_cache: InvalidateFn | None = None,
+        debounce_seconds: float = DEFAULT_DEBOUNCE_SECONDS,
+        max_wait_seconds: float = DEFAULT_MAX_WAIT_SECONDS,
+    ) -> None:
+        self._broadcast_earth_update = broadcast_earth_update or broadcaster.broadcast_earth_update
+        self._invalidate_cache = invalidate_cache or invalidate_earth_layer_cache_for_source
+        self._debounce_seconds = debounce_seconds
+        self._max_wait_seconds = max(max_wait_seconds, debounce_seconds)
+        self._pending: dict[str, PendingEarthDbChange] = {}
+        self._flush_tasks: dict[str, asyncio.Task[None]] = {}
+        self._seen_event_ids: set[int] = set()
+        self._seen_event_order: deque[int] = deque()
+
+    def handle_notification(self, payload_text: str) -> bool:
+        try:
+            payload = json.loads(payload_text)
+        except json.JSONDecodeError:
+            logger.warning_event(
+                "Ignoring malformed Earth database change notification",
+                event="earth.db_changes.notification_malformed",
+            )
+            return False
+        if not isinstance(payload, dict):
+            return False
+        return self.handle_payload(payload)
+
+    def handle_payload(self, payload: dict[str, Any]) -> bool:
+        event_id = payload.get("event_id")
+        if event_id is not None:
+            try:
+                normalized_event_id = int(event_id)
+            except (TypeError, ValueError):
+                normalized_event_id = None
+            if normalized_event_id is not None:
+                if normalized_event_id in self._seen_event_ids:
+                    return False
+                self._remember_event_id(normalized_event_id)
+
+        update = build_earth_update_from_db_payload(payload)
+        if not update:
+            return False
+
+        source = update["source"]
+        pending = self._pending.get(source)
+        if pending is None:
+            pending = PendingEarthDbChange(
+                source=source,
+                layers=list(update["layers"]),
+                table=update.get("table"),
+                refresh_strategy=str(update.get("refresh_strategy") or "clear_then_reload"),
+                entity=update.get("entity"),
+                action=str(update.get("action") or "database_changed"),
+            )
+            self._pending[source] = pending
+        pending.add(payload)
+
+        task = self._flush_tasks.pop(source, None)
+        if task and not task.done():
+            task.cancel()
+        self._flush_tasks[source] = asyncio.create_task(
+            self._flush_later(source, delay_seconds=self._next_flush_delay(pending))
+        )
+        return True
+
+    def _next_flush_delay(self, pending: PendingEarthDbChange) -> float:
+        if "DELETE" in pending.operations and pending.refresh_strategy == "clear_then_reload":
+            return DELETE_FAST_FLUSH_SECONDS
+        elapsed = max(0.0, monotonic() - pending.first_seen_monotonic)
+        remaining = self._max_wait_seconds - elapsed
+        if remaining <= 0:
+            return 0.0
+        return min(self._debounce_seconds, remaining)
+
+    def _remember_event_id(self, event_id: int) -> None:
+        self._seen_event_ids.add(event_id)
+        self._seen_event_order.append(event_id)
+        while len(self._seen_event_order) > MAX_SEEN_EVENT_IDS:
+            expired_event_id = self._seen_event_order.popleft()
+            self._seen_event_ids.discard(expired_event_id)
+
+    async def _flush_later(self, source: str, *, delay_seconds: float) -> None:
+        try:
+            if delay_seconds > 0:
+                await asyncio.sleep(delay_seconds)
+            await self.flush_source(source)
+        except asyncio.CancelledError:
+            raise
+        except Exception as exc:
+            logger.exception_event(
+                "Failed to broadcast debounced Earth database change",
+                event="earth.db_changes.flush_failed",
+                context={"source": source, "error": str(exc)},
+            )
+        finally:
+            current = self._flush_tasks.get(source)
+            if current is asyncio.current_task():
+                self._flush_tasks.pop(source, None)
+
+    async def flush_source(self, source: str) -> None:
+        pending = self._pending.get(source)
+        if pending is None:
+            return
+
+        flushed_at = datetime.now(UTC)
+        deleted_cache_entries = self._invalidate_cache(source)
+        operations = sorted(pending.operations)
+        payload: dict[str, Any] = {
+            "event": "earth.layer.changed",
+            "action": "database_changed",
+            "source": source,
+            "table": pending.table,
+            "data_type": source,
+            "layers": pending.layers,
+            "refresh_strategy": pending.refresh_strategy,
+            "records_processed": pending.records_processed,
+            "operations": operations,
+            "operation": operations[-1] if len(operations) == 1 else None,
+            "entity_keys": pending.entity_keys,
+            "entity_key_sample_size": len(pending.entity_keys),
+            "cache_entries_invalidated": deleted_cache_entries,
+            "first_occurred_at": pending.first_occurred_at,
+            "last_occurred_at": pending.last_occurred_at,
+            "debounce_ms": int((monotonic() - pending.first_seen_monotonic) * 1000),
+            "total_latency_ms": self._total_latency_ms(pending, flushed_at),
+            "timestamp": to_iso8601_utc(flushed_at),
+        }
+        if pending.entity == "interactable":
+            payload.update(
+                {
+                    "entity": "interactable",
+                    "action": "deleted" if "DELETE" in pending.operations else "changed",
+                    "ids": pending.entity_keys,
+                    "item": None,
+                }
+            )
+        await self._broadcast_earth_update(payload)
+        self._pending.pop(source, None)
+        logger.info_event(
+            "Broadcasted Earth database change",
+            event="earth.db_changes.broadcasted",
+            context={
+                "source": source,
+                "layers": pending.layers,
+                "records_processed": pending.records_processed,
+                "cache_entries_invalidated": deleted_cache_entries,
+                "debounce_ms": int((monotonic() - pending.first_seen_monotonic) * 1000),
+                "total_latency_ms": payload["total_latency_ms"],
+            },
+        )
+
+    @staticmethod
+    def _total_latency_ms(pending: PendingEarthDbChange, flushed_at: datetime) -> int | None:
+        occurred_at = pending.first_occurred_at
+        if not occurred_at:
+            return None
+        try:
+            normalized = occurred_at.replace("Z", "+00:00")
+            occurred = datetime.fromisoformat(normalized)
+            if occurred.tzinfo is None:
+                occurred = occurred.replace(tzinfo=UTC)
+            return max(0, int((flushed_at - occurred.astimezone(UTC)).total_seconds() * 1000))
+        except ValueError:
+            return None
+
+    async def flush_all(self) -> None:
+        sources = list(self._pending)
+        for source in sources:
+            task = self._flush_tasks.pop(source, None)
+            if task and not task.done():
+                task.cancel()
+            await self.flush_source(source)
+
+    async def stop(self) -> None:
+        tasks = [task for task in self._flush_tasks.values() if not task.done()]
+        for task in tasks:
+            task.cancel()
+        if tasks:
+            await asyncio.gather(*tasks, return_exceptions=True)
+        self._flush_tasks.clear()
+        await self.flush_all()
+
+
+class EarthDbChangeListener:
+    def __init__(
+        self,
+        *,
+        dsn: str,
+        dispatcher: EarthDbChangeDispatcher,
+        channel: str = EARTH_DATA_CHANGES_CHANNEL,
+    ) -> None:
+        self._dsn = normalize_asyncpg_dsn(dsn)
+        self._dispatcher = dispatcher
+        self._channel = channel
+        self._task: asyncio.Task[None] | None = None
+        self._stop_event: asyncio.Event | None = None
+        self._connection: asyncpg.Connection | None = None
+        self._loop: asyncio.AbstractEventLoop | None = None
+
+    def start(self) -> None:
+        if self._task and not self._task.done():
+            return
+        self._loop = asyncio.get_running_loop()
+        self._stop_event = asyncio.Event()
+        self._task = asyncio.create_task(self._run())
+
+    async def stop(self) -> None:
+        if self._stop_event:
+            self._stop_event.set()
+        if self._connection:
+            await self._connection.close()
+        if self._task:
+            await asyncio.gather(self._task, return_exceptions=True)
+        await self._dispatcher.stop()
+
+    async def _run(self) -> None:
+        backoff_seconds = 1.0
+        assert self._stop_event is not None
+
+        while not self._stop_event.is_set():
+            try:
+                self._connection = await asyncpg.connect(self._dsn)
+                await self._connection.add_listener(self._channel, self._on_notification)
+                logger.info_event(
+                    "Earth database change listener connected",
+                    event="earth.db_changes.connected",
+                    context={"channel": self._channel},
+                )
+                backoff_seconds = 1.0
+                while not self._stop_event.is_set():
+                    try:
+                        await asyncio.wait_for(
+                            self._stop_event.wait(),
+                            timeout=LISTEN_KEEPALIVE_SECONDS,
+                        )
+                    except asyncio.TimeoutError:
+                        await self._poll_outbox()
+            except asyncio.CancelledError:
+                raise
+            except Exception as exc:
+                logger.exception_event(
+                    "Earth database change listener failed",
+                    event="earth.db_changes.listener_failed",
+                    context={"channel": self._channel, "error": str(exc)},
+                )
+                try:
+                    await asyncio.wait_for(self._stop_event.wait(), timeout=backoff_seconds)
+                except asyncio.TimeoutError:
+                    pass
+                backoff_seconds = min(backoff_seconds * 2, 30.0)
+            finally:
+                if self._connection:
+                    try:
+                        await self._connection.remove_listener(self._channel, self._on_notification)
+                    except Exception:
+                        pass
+                    try:
+                        await self._connection.close()
+                    except Exception:
+                        pass
+                    self._connection = None
+
+    async def _poll_outbox(self) -> None:
+        if self._connection is None:
+            return
+
+        rows = await self._connection.fetch(
+            """
+            SELECT id, payload
+            FROM earth_data_change_events
+            WHERE consumed_at IS NULL
+            ORDER BY id
+            LIMIT $1
+            """,
+            OUTBOX_POLL_LIMIT,
+        )
+        accepted_count = 0
+        consumed_ids: list[int] = []
+        for row in rows:
+            payload = row["payload"]
+            if isinstance(payload, str):
+                try:
+                    payload = json.loads(payload)
+                except json.JSONDecodeError:
+                    consumed_ids.append(int(row["id"]))
+                    continue
+            if isinstance(payload, dict):
+                if self._dispatcher.handle_payload(payload):
+                    accepted_count += 1
+                consumed_ids.append(int(row["id"]))
+            else:
+                consumed_ids.append(int(row["id"]))
+        if consumed_ids:
+            await self._dispatcher.flush_all()
+        if consumed_ids:
+            await self._connection.execute(
+                """
+                UPDATE earth_data_change_events
+                SET consumed_at = NOW()
+                WHERE id = ANY($1::bigint[])
+                  AND consumed_at IS NULL
+                """,
+                consumed_ids,
+            )
+        if rows:
+            logger.info_event(
+                "Polled Earth database change outbox",
+                event="earth.db_changes.outbox_polled",
+                context={"events": len(rows), "accepted": accepted_count},
+            )
+
+    def _on_notification(
+        self,
+        _connection: asyncpg.Connection,
+        _pid: int,
+        _channel: str,
+        payload: str,
+    ) -> None:
+        if self._loop and self._loop.is_running():
+            self._loop.call_soon_threadsafe(self._dispatcher.handle_notification, payload)
+            return
+        self._dispatcher.handle_notification(payload)
+
+
+_dispatcher = EarthDbChangeDispatcher(
+    broadcast_earth_update=broadcaster.broadcast_earth_update,
+    invalidate_cache=invalidate_earth_layer_cache_for_source,
+)
+_listener: EarthDbChangeListener | None = None
+
+
+def start_earth_db_change_listener() -> None:
+    global _listener
+    if _listener is not None:
+        return
+    _listener = EarthDbChangeListener(dsn=settings.DATABASE_URL, dispatcher=_dispatcher)
+    _listener.start()
+
+
+async def stop_earth_db_change_listener() -> None:
+    global _listener
+    if _listener is None:
+        await _dispatcher.stop()
+        return
+    listener = _listener
+    _listener = None
+    await listener.stop()

backend/app/services/earth_interactables.py

@@ -0,0 +1,113 @@
+from __future__ import annotations
+
+from datetime import UTC, datetime
+from typing import Any
+from uuid import uuid4
+
+from sqlalchemy import select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.core.time import to_iso8601_utc
+from app.core.websocket.broadcaster import broadcaster
+from app.models.earth_interactable import EarthInteractable
+from app.services.earth_layer_cache import EARTH_LAYER_CACHE_PREFIX, earth_layer_cache
+
+INTERACTABLE_ENTITY = "interactable"
+INTERACTABLE_LAYER = "interactables"
+
+
+def normalize_interactable_id(value: str | None = None) -> str:
+    raw = str(value or "").strip()
+    return raw or f"interactable-{uuid4().hex}"
+
+
+def serialize_interactable(record: EarthInteractable) -> dict[str, Any]:
+    return {
+        "id": record.id,
+        "layer": record.layer,
+        "kind": record.kind,
+        "label": record.label,
+        "description": record.description,
+        "latitude": record.latitude,
+        "longitude": record.longitude,
+        "altitude": record.altitude,
+        "revision": record.revision,
+        "properties": record.properties or {},
+        "is_deleted": bool(record.is_deleted),
+        "created_at": to_iso8601_utc(record.created_at),
+        "updated_at": to_iso8601_utc(record.updated_at),
+        "deleted_at": to_iso8601_utc(record.deleted_at),
+    }
+
+
+def interactables_to_geojson(items: list[EarthInteractable]) -> dict[str, Any]:
+    return {
+        "type": "FeatureCollection",
+        "features": [
+            {
+                "type": "Feature",
+                "id": item.id,
+                "geometry": {
+                    "type": "Point",
+                    "coordinates": [item.longitude, item.latitude],
+                },
+                "properties": serialize_interactable(item),
+            }
+            for item in items
+            if not item.is_deleted
+        ],
+    }
+
+
+def invalidate_interactable_cache(layer: str | None = None) -> int:
+    layer_key = str(layer or "*").strip() or "*"
+    deleted = earth_layer_cache.delete_pattern(
+        f"{EARTH_LAYER_CACHE_PREFIX}:interactables:interactable_layer:{layer_key}*"
+    )
+    if layer_key != "all":
+        deleted += earth_layer_cache.delete_pattern(
+            f"{EARTH_LAYER_CACHE_PREFIX}:interactables:interactable_layer:all*"
+        )
+    return deleted
+
+
+def build_interactable_event(
+    *,
+    action: str,
+    record: EarthInteractable,
+    include_item: bool = True,
+) -> dict[str, Any]:
+    item = serialize_interactable(record)
+    return {
+        "entity": INTERACTABLE_ENTITY,
+        "action": action,
+        "layer": record.layer,
+        "layers": [INTERACTABLE_LAYER],
+        "ids": [record.id],
+        "revision": record.revision,
+        "changed_at": item["deleted_at"] or item["updated_at"] or to_iso8601_utc(datetime.now(UTC)),
+        "item": item if include_item else None,
+        "source": "earth_interactables",
+    }
+
+
+async def publish_interactable_event(action: str, record: EarthInteractable, *, include_item: bool = True) -> None:
+    await broadcaster.broadcast_earth_update(
+        build_interactable_event(action=action, record=record, include_item=include_item)
+    )
+
+
+async def list_interactables(
+    db: AsyncSession,
+    *,
+    layer: str | None = None,
+    include_deleted: bool = False,
+) -> list[EarthInteractable]:
+    stmt = select(EarthInteractable)
+    if layer:
+        stmt = stmt.where(EarthInteractable.layer == layer)
+    if not include_deleted:
+        stmt = stmt.where(EarthInteractable.is_deleted.is_(False))
+    stmt = stmt.order_by(EarthInteractable.updated_at.desc(), EarthInteractable.id.asc())
+    result = await db.execute(stmt)
+    return list(result.scalars().all())

backend/app/services/earth_layer_adapters.py

@@ -0,0 +1,187 @@
+"""Earth layer adapter registry for datasource-backed refresh behavior."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Any
+
+from sqlalchemy.ext.asyncio import AsyncSession
+
+
+@dataclass(frozen=True)
+class EarthLayerAdapter:
+    sources: frozenset[str]
+    layers: tuple[str, ...]
+    cache_patterns: tuple[str, ...]
+    tables: frozenset[str] = field(default_factory=frozenset)
+    derived_models: tuple[str, ...] = field(default_factory=tuple)
+    refresh_strategy: str = "clear_then_reload"
+
+
+EARTH_LAYER_ADAPTERS: tuple[EarthLayerAdapter, ...] = (
+    EarthLayerAdapter(
+        sources=frozenset({"barentswatch_vessels", "aisstream_vessels", "vessel_static", "vessel_position", "vessel_current_state", "ais_raw_observations", "ais_source_health"}),
+        tables=frozenset({"vessel_static", "vessel_position", "vessel_current_state", "ais_raw_observations", "ais_source_health"}),
+        layers=("vessels",),
+        cache_patterns=("vessels*", "summary*"),
+        derived_models=("ais_raw_observations", "ais_conflict_records", "ais_source_health"),
+    ),
+    EarthLayerAdapter(
+        sources=frozenset(
+            {
+                "telegeography_cables",
+                "telegeography_landing",
+                "telegeography_landing_points",
+                "telegeography_systems",
+                "telegeography_cable_systems",
+                "arcgis_cables",
+                "arcgis_landing_points",
+                "arcgis_cable_landing_relation",
+                "arcgis_cable_landing_relations",
+                "fao_landing_points",
+            }
+        ),
+        tables=frozenset({"collected_data"}),
+        layers=("cables",),
+        cache_patterns=("cables*", "landing-points*", "summary*"),
+    ),
+    EarthLayerAdapter(
+        sources=frozenset({"celestrak_tle", "spacetrack_tle"}),
+        tables=frozenset({"collected_data"}),
+        layers=("satellites",),
+        cache_patterns=("satellites*", "summary*"),
+    ),
+    EarthLayerAdapter(
+        sources=frozenset(
+            {
+                "top500",
+                "top500_supercomputers",
+                "epoch_ai_gpu",
+                "huggingface_models",
+                "huggingface_datasets",
+                "huggingface_spaces",
+                "compute_center_locations",
+            }
+        ),
+        tables=frozenset({"compute_center_locations"}),
+        layers=("computeCenters",),
+        cache_patterns=("compute-centers*", "summary*"),
+        refresh_strategy="reload",
+    ),
+    EarthLayerAdapter(
+        sources=frozenset(
+            {
+                "ris_live_bgp",
+                "bgpstream_bgp",
+                "iptoasn_prefix_geo",
+                "opengeofeed_prefix_geo",
+                "nro_delegated_prefix_geo",
+                "bgp_observations",
+                "bgp_anomalies",
+                "bgp_incidents",
+                "bgp_collector_locations",
+            }
+        ),
+        tables=frozenset({"bgp_observations", "bgp_anomalies", "bgp_incidents", "bgp_collector_locations"}),
+        layers=("bgp",),
+        cache_patterns=("bgp*", "summary*"),
+        derived_models=("bgp_observations", "bgp_anomalies", "bgp_incidents"),
+    ),
+    EarthLayerAdapter(
+        sources=frozenset({"news_live_streams"}),
+        tables=frozenset({"collected_data"}),
+        layers=("media",),
+        cache_patterns=("summary*",),
+        refresh_strategy="reload",
+    ),
+    EarthLayerAdapter(
+        sources=frozenset({"media_news_archive", "earth_news_items"}),
+        tables=frozenset({"earth_news_items"}),
+        layers=("news",),
+        cache_patterns=("summary*",),
+        refresh_strategy="reload",
+    ),
+    EarthLayerAdapter(
+        sources=frozenset({"earth_interactables"}),
+        tables=frozenset({"earth_interactables"}),
+        layers=("interactables",),
+        cache_patterns=("interactables*", "summary*"),
+        refresh_strategy="delta",
+    ),
+)
+
+_ADAPTERS_BY_SOURCE = {
+    source: adapter
+    for adapter in EARTH_LAYER_ADAPTERS
+    for source in adapter.sources
+}
+_ADAPTERS_BY_TABLE = {
+    table: adapter
+    for adapter in EARTH_LAYER_ADAPTERS
+    for table in adapter.tables
+}
+
+
+def get_earth_layer_adapter_for_source(source: str | None) -> EarthLayerAdapter | None:
+    return _ADAPTERS_BY_SOURCE.get(str(source or "").strip())
+
+
+def get_earth_layer_adapter_for_change(table: str | None, source: str | None) -> EarthLayerAdapter | None:
+    table_key = str(table or "").strip()
+    source_key = str(source or "").strip()
+    if table_key and table_key != "collected_data":
+        adapter = _ADAPTERS_BY_TABLE.get(table_key)
+        if adapter is not None:
+            return adapter
+    return get_earth_layer_adapter_for_source(source_key)
+
+
+def get_earth_update_layers_for_source(source: str | None) -> list[str]:
+    adapter = get_earth_layer_adapter_for_source(source)
+    return list(adapter.layers) if adapter else []
+
+
+def get_earth_update_layers_for_change(table: str | None, source: str | None) -> list[str]:
+    adapter = get_earth_layer_adapter_for_change(table, source)
+    return list(adapter.layers) if adapter else []
+
+
+def get_earth_refresh_strategy_for_change(table: str | None, source: str | None) -> str | None:
+    adapter = get_earth_layer_adapter_for_change(table, source)
+    return adapter.refresh_strategy if adapter else None
+
+
+def get_earth_cache_patterns_for_source(source: str | None) -> list[str]:
+    adapter = get_earth_layer_adapter_for_source(source)
+    return list(adapter.cache_patterns) if adapter else []
+
+
+async def clear_derived_datasource_data(db: AsyncSession, source: str) -> dict[str, int]:
+    adapter = get_earth_layer_adapter_for_source(source)
+    if adapter is None or not adapter.derived_models:
+        return {}
+
+    from app.models.bgp_anomaly import BGPAnomaly
+    from app.models.bgp_incident import BGPIncident
+    from app.models.bgp_observation import BGPObservation
+    from app.models.vessel import AISConflictRecord, AISRawObservation, AISSourceHealth
+
+    model_by_key: dict[str, Any] = {
+        "bgp_observations": BGPObservation,
+        "bgp_anomalies": BGPAnomaly,
+        "bgp_incidents": BGPIncident,
+        "ais_raw_observations": AISRawObservation,
+        "ais_conflict_records": AISConflictRecord,
+        "ais_source_health": AISSourceHealth,
+    }
+    deleted_counts: dict[str, int] = {}
+    for key in adapter.derived_models:
+        model = model_by_key.get(key)
+        if model is None:
+            continue
+        if key == "ais_conflict_records":
+            result = await db.execute(model.__table__.delete().where(model.selected_source == source))
+        else:
+            result = await db.execute(model.__table__.delete().where(model.source == source))
+        deleted_counts[key] = int(result.rowcount or 0)
+    return deleted_counts

backend/app/services/earth_layer_cache.py

@@ -268,34 +268,10 @@ def apply_payload_budget(payload: dict[str, Any], policy: EarthLayerCachePolicy)
 
 
 def invalidate_earth_layer_cache_for_source(source: str) -> int:
+    from app.services.earth_layer_adapters import get_earth_cache_patterns_for_source
+
     source_key = str(source or "").strip()
-    patterns = {
-        "barentswatch_vessels": ["vessels*", "summary*"],
-        "aisstream_vessels": ["vessels*", "summary*"],
-        "telegeography_cables": ["cables*", "landing-points*", "summary*"],
-        "telegeography_landing": ["landing-points*", "summary*"],
-        "telegeography_landing_points": ["landing-points*", "summary*"],
-        "telegeography_systems": ["cables*", "summary*"],
-        "telegeography_cable_systems": ["cables*", "summary*"],
-        "arcgis_cables": ["cables*", "landing-points*", "summary*"],
-        "arcgis_landing_points": ["landing-points*", "summary*"],
-        "arcgis_cable_landing_relation": ["landing-points*", "summary*"],
-        "arcgis_cable_landing_relations": ["landing-points*", "summary*"],
-        "fao_landing_points": ["landing-points*", "summary*"],
-        "celestrak_tle": ["satellites*", "summary*"],
-        "spacetrack_tle": ["satellites*", "summary*"],
-        "top500": ["compute-centers*", "summary*"],
-        "top500_supercomputers": ["compute-centers*", "summary*"],
-        "epoch_ai_gpu": ["compute-centers*", "summary*"],
-        "huggingface_models": ["compute-centers*", "summary*"],
-        "huggingface_datasets": ["compute-centers*", "summary*"],
-        "huggingface_spaces": ["compute-centers*", "summary*"],
-        "ris_live_bgp": ["bgp*", "summary*"],
-        "bgpstream_bgp": ["bgp*", "summary*"],
-        "iptoasn_prefix_geo": ["bgp*", "summary*"],
-        "opengeofeed_prefix_geo": ["bgp*", "summary*"],
-        "nro_delegated_prefix_geo": ["bgp*", "summary*"],
-    }.get(source_key, [])
+    patterns = get_earth_cache_patterns_for_source(source_key)
     deleted = 0
     for layer_pattern in patterns:
         deleted += earth_layer_cache.delete_pattern(f"{EARTH_LAYER_CACHE_PREFIX}:{layer_pattern}")

backend/app/services/earth_news_classification.py

@@ -0,0 +1,258 @@
+"""Classification, importance, and breaking-news policy for Earth news."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from datetime import UTC, datetime, timedelta
+import re
+from typing import Any, Protocol
+
+from app.core.enums import (
+    BreakingLevel,
+    BreakingScope,
+    BreakingSource,
+    NewsImportanceLevel,
+    NewsMarketImpact,
+    NewsTaggingSource,
+    parse_enum,
+)
+
+
+class NewsItemLike(Protocol):
+    title: str
+    summary: str
+    source: str
+    feed_name: str
+    published_at: datetime | None
+    feed_default_category: str
+    category: str
+    item_tags: list[str]
+    tagging_source: str
+    tagging_confidence: float
+    importance_score: int
+    importance_level: str
+    importance_reasons: list[str]
+    market_impact: str
+    source_tags: list[str]
+    breaking_level: str
+    breaking_scope: str
+    breaking_reasons: list[str]
+    breaking_source: str
+    breaking_confidence: float
+    breaking_expires_at: datetime | None
+
+
+class NewsSourceLike(Protocol):
+    default_category: str
+    importance_weight: int
+    source_tags: tuple[str, ...]
+
+
+class NewsFeedLike(Protocol):
+    default_category: str
+
+
+@dataclass(frozen=True)
+class BreakingRule:
+    level: BreakingLevel
+    scope: BreakingScope
+    reason: str
+    keywords: tuple[str, ...]
+
+
+IMPORTANCE_THRESHOLDS: tuple[tuple[int, NewsImportanceLevel], ...] = (
+    (80, NewsImportanceLevel.CRITICAL),
+    (60, NewsImportanceLevel.HIGH),
+    (35, NewsImportanceLevel.MEDIUM),
+    (0, NewsImportanceLevel.LOW),
+)
+
+BREAKING_LEVEL_RANK: dict[BreakingLevel, int] = {
+    BreakingLevel.NONE: 0,
+    BreakingLevel.WATCH: 1,
+    BreakingLevel.BREAKING: 2,
+    BreakingLevel.CRITICAL: 3,
+}
+
+BREAKING_TTL: dict[BreakingLevel, timedelta] = {
+    BreakingLevel.WATCH: timedelta(hours=6),
+    BreakingLevel.BREAKING: timedelta(hours=12),
+    BreakingLevel.CRITICAL: timedelta(hours=24),
+}
+
+BREAKING_RULES: tuple[BreakingRule, ...] = (
+    BreakingRule(BreakingLevel.CRITICAL, BreakingScope.GLOBAL, "核事故或核风险", ("nuclear accident", "nuclear emergency", "radiation leak", "核事故", "核泄漏", "辐射泄漏")),
+    BreakingRule(BreakingLevel.CRITICAL, BreakingScope.REGIONAL, "重大军事冲突升级", ("airstrike", "missile strike", "invasion", "martial law", "空袭", "导弹袭击", "入侵", "戒严")),
+    BreakingRule(BreakingLevel.BREAKING, BreakingScope.REGIONAL, "战争或安全事件", ("war escalates", "terror attack", "coup", "hostage", "战争升级", "恐袭", "政变", "人质")),
+    BreakingRule(BreakingLevel.BREAKING, BreakingScope.REGIONAL, "重大灾害应急", ("major earthquake", "tsunami", "volcanic eruption", "state of emergency", "强震", "海啸", "火山喷发", "紧急状态")),
+    BreakingRule(BreakingLevel.BREAKING, BreakingScope.GLOBAL, "金融市场异常", ("market halt", "trading halt", "flash crash", "bank run", "金融熔断", "交易暂停", "银行挤兑")),
+    BreakingRule(BreakingLevel.WATCH, BreakingScope.GLOBAL, "大规模网络安全事件", ("massive cyberattack", "ransomware attack", "data breach", "大规模网络攻击", "勒索软件", "数据泄露")),
+    BreakingRule(BreakingLevel.WATCH, BreakingScope.REGIONAL, "航天或卫星事故", ("rocket explosion", "satellite collision", "space station emergency", "火箭爆炸", "卫星碰撞", "空间站事故")),
+)
+
+
+def contains_keyword(text: str, keyword: str) -> bool:
+    keyword_text = str(keyword or "").strip().lower()
+    if not keyword_text:
+        return False
+    if re.search(r"[\u4e00-\u9fff]", keyword_text):
+        return keyword_text in text
+    return re.search(rf"(?<![a-z0-9]){re.escape(keyword_text)}(?![a-z0-9])", text) is not None
+
+
+def score_category(text: str, title_text: str, category: dict[str, Any]) -> int:
+    score = 0
+    keywords = category.get("keywords") if isinstance(category.get("keywords"), list) else []
+    for keyword in keywords:
+        if contains_keyword(title_text, keyword):
+            score += 3
+        elif contains_keyword(text, keyword):
+            score += 1
+    return score
+
+
+def importance_level(score: int) -> NewsImportanceLevel:
+    normalized_score = max(0, min(100, int(score)))
+    for threshold, level in IMPORTANCE_THRESHOLDS:
+        if normalized_score >= threshold:
+            return level
+    return NewsImportanceLevel.LOW
+
+
+def normalize_breaking_level(value: object) -> BreakingLevel:
+    return parse_enum(BreakingLevel, value, BreakingLevel.NONE)
+
+
+def normalize_breaking_scope(value: object) -> BreakingScope:
+    return parse_enum(BreakingScope, value, BreakingScope.REGIONAL)
+
+
+def breaking_expires_at(level: object, published_at: datetime | None) -> datetime | None:
+    normalized = normalize_breaking_level(level)
+    if normalized is BreakingLevel.NONE:
+        return None
+    base = published_at or datetime.now(UTC)
+    base = base.replace(tzinfo=UTC) if base.tzinfo is None else base.astimezone(UTC)
+    return base + BREAKING_TTL[normalized]
+
+
+def is_breaking_active(item: NewsItemLike, *, now: datetime | None = None) -> bool:
+    if normalize_breaking_level(item.breaking_level) is BreakingLevel.NONE:
+        return False
+    expires_at = item.breaking_expires_at
+    if expires_at is None:
+        return True
+    expires_at = expires_at.replace(tzinfo=UTC) if expires_at.tzinfo is None else expires_at.astimezone(UTC)
+    return expires_at > (now or datetime.now(UTC))
+
+
+def breaking_sort_rank(item: NewsItemLike) -> int:
+    if not is_breaking_active(item):
+        return 0
+    return BREAKING_LEVEL_RANK[normalize_breaking_level(item.breaking_level)]
+
+
+def highest_breaking_level(items: list[NewsItemLike]) -> BreakingLevel:
+    active = [normalize_breaking_level(item.breaking_level) for item in items if is_breaking_active(item)]
+    return max(active, key=BREAKING_LEVEL_RANK.get) if active else BreakingLevel.NONE
+
+
+def apply_breaking_rules(item: NewsItemLike) -> None:
+    combined_text = f"{item.title} {item.summary} {item.source} {item.feed_name}".lower()
+    best_level = BreakingLevel.NONE
+    best_scope = BreakingScope.REGIONAL
+    reasons: list[str] = []
+    confidence = 0.0
+    for rule in BREAKING_RULES:
+        if not any(contains_keyword(combined_text, keyword) for keyword in rule.keywords):
+            continue
+        if BREAKING_LEVEL_RANK[rule.level] > BREAKING_LEVEL_RANK[best_level]:
+            best_level = rule.level
+            best_scope = rule.scope
+        if rule.reason not in reasons:
+            reasons.append(rule.reason)
+        confidence = max(confidence, 0.72 if rule.level is BreakingLevel.CRITICAL else 0.64 if rule.level is BreakingLevel.BREAKING else 0.52)
+
+    item.breaking_level = best_level.value
+    item.breaking_scope = (best_scope if best_level is not BreakingLevel.NONE else BreakingScope.REGIONAL).value
+    item.breaking_reasons = reasons
+    item.breaking_source = BreakingSource.RULES.value
+    item.breaking_confidence = round(confidence, 2)
+    item.breaking_expires_at = breaking_expires_at(best_level, item.published_at)
+
+
+def apply_news_classification(
+    item: NewsItemLike,
+    source: NewsSourceLike,
+    *,
+    feed: NewsFeedLike | None,
+    config: dict[str, Any],
+) -> NewsItemLike:
+    title_text = item.title.lower()
+    combined_text = f"{item.title} {item.summary} {item.source} {item.feed_name}".lower()
+    feed_default_category = (feed.default_category if feed else item.feed_default_category) or source.default_category or "other"
+    best_key = feed_default_category
+    best_score = second_score = 0
+    for category in config["categories"]:
+        if not isinstance(category, dict) or category.get("enabled") is False:
+            continue
+        score = score_category(combined_text, title_text, category)
+        if score > best_score:
+            second_score, best_score = best_score, score
+            best_key = str(category.get("key") or "other")
+        elif score > second_score:
+            second_score = score
+
+    item_tags: list[str] = []
+    for rule in config["item_tag_rules"]:
+        if not isinstance(rule, dict):
+            continue
+        keywords = rule.get("keywords") if isinstance(rule.get("keywords"), list) else []
+        if any(contains_keyword(combined_text, keyword) for keyword in keywords):
+            tag_key = str(rule.get("key") or "").strip()
+            if tag_key and tag_key not in item_tags:
+                item_tags.append(tag_key)
+            if best_score < 3 and rule.get("category"):
+                best_key, best_score = str(rule["category"]), 3
+
+    confidence = round(best_score / (best_score + second_score + 1), 2) if best_score else 0.35
+    if best_score < 3 and feed_default_category:
+        best_key, confidence = feed_default_category, 0.45
+
+    score = max(0, min(100, 18 + source.importance_weight + best_score * 6))
+    reasons: list[str] = []
+    source_tags = set(source.source_tags)
+    if "official_data" in source_tags:
+        score += 20
+        reasons.append("官方数据源")
+    if "press_release" in source_tags:
+        score = max(0, score - 12)
+        reasons.append("企业公告基础权重较低")
+    if any(contains_keyword(combined_text, term) for term in ("网上零售额", "电商物流指数", "gmv", "订单量", "物流指数", "履约", "直播电商", "跨境电商")):
+        score += 25
+        reasons.append("命中电商数据指标")
+    if any(contains_keyword(combined_text, term) for term in ("amazon", "shopify", "walmart", "alibaba", "jd.com", "pinduoduo", "tiktok shop", "shein", "阿里", "京东", "拼多多", "抖音")):
+        score += 15
+        reasons.append("涉及大型平台")
+    if any(term in combined_text for term in ("同比", "环比", "%", "billion", "million", "增长", "下降")):
+        score += 10
+        reasons.append("包含量化指标")
+
+    score = max(0, min(100, score))
+    item.category = best_key or "other"
+    item.item_tags = item_tags
+    item.tagging_source = NewsTaggingSource.RULES.value
+    item.tagging_confidence = confidence
+    item.importance_score = score
+    item.importance_level = importance_level(score).value
+    item.importance_reasons = reasons or ["按来源权重和分类规则计算"]
+    item.market_impact = (
+        NewsMarketImpact.GLOBAL.value
+        if "global" in source_tags
+        else NewsMarketImpact.NATIONAL.value
+        if {"china", "us"} & source_tags
+        else NewsMarketImpact.SECTOR.value
+    )
+    item.source_tags = list(source.source_tags)
+    apply_breaking_rules(item)
+    return item

backend/app/services/earth_news_manual.py

@@ -0,0 +1,693 @@
+from __future__ import annotations
+
+from dataclasses import dataclass
+from datetime import UTC, datetime
+import hashlib
+import html
+import json
+import re
+from typing import Any
+
+from bs4 import BeautifulSoup
+from sqlalchemy import delete, func, select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.core.enums import NewsEnrichmentStatus, NewsSourceType, NewsTaggingSource
+from app.core.websocket.broadcaster import broadcaster
+from app.models.earth_news import EarthNewsItem
+from app.models.system_setting import SystemSetting
+from app.services.earth_news import (
+    ALLOWED_NEWS_CATEGORY_KEYS,
+    DEFAULT_NEWS_LOCALE,
+    REGION_ANCHORS,
+    NewsFeedEndpoint,
+    NewsFeedSource,
+    NewsTargetLocation,
+    ParsedNewsItem,
+    apply_news_classification,
+    build_anchor_location_patch,
+    build_target_location_job_payload,
+    build_target_location_patch,
+    _serialize_item,
+)
+from app.services.earth_news_queue import enqueue_target_location_job
+from app.services.earth_news_store import record_to_parsed_news_item
+
+
+MANUAL_NEWS_SOURCE_ID = "manual"
+MANUAL_NEWS_SOURCE_LABEL = "手动添加"
+MANUAL_NEWS_MAX_IMPORT_ITEMS = 500
+MANUAL_NEWS_MAX_TITLE_LENGTH = 500
+MANUAL_NEWS_MAX_SUMMARY_LENGTH = 1200
+MANUAL_NEWS_MAX_CONTENT_LENGTH = 12000
+EARTH_NEWS_MANUAL_GROUPS_CATEGORY = "earth_news_manual_groups"
+DEFAULT_MANUAL_NEWS_GROUP_ID = "manual-default"
+DEFAULT_MANUAL_NEWS_GROUP_NAME = "新建新闻组"
+
+
+@dataclass(frozen=True)
+class ManualNewsWriteResult:
+    item: EarthNewsItem
+    created: bool
+    queued: bool
+
+
+@dataclass(frozen=True)
+class ManualNewsGroup:
+    id: str
+    name: str
+    sort_order: int = 0
+
+
+def _clean_text(value: object, *, max_length: int) -> str:
+    raw = "" if value is None else str(value)
+    text = BeautifulSoup(html.unescape(raw), "html.parser").get_text(" ", strip=True)
+    text = re.sub(r"\s+", " ", text).strip()
+    if len(text) > max_length:
+        return text[: max_length - 1].rstrip() + "…"
+    return text
+
+
+def _parse_datetime(value: object) -> datetime | None:
+    if value is None or str(value).strip() == "":
+        return None
+    if isinstance(value, datetime):
+        parsed = value
+    else:
+        try:
+            parsed = datetime.fromisoformat(str(value).strip().replace("Z", "+00:00"))
+        except ValueError as exc:
+            raise ValueError("published_at 必须是 ISO8601 时间。") from exc
+    if parsed.tzinfo is None:
+        return parsed.replace(tzinfo=UTC)
+    return parsed.astimezone(UTC)
+
+
+def _detect_language(*parts: str) -> str:
+    text = " ".join(part for part in parts if part)
+    cjk_count = len(re.findall(r"[\u4e00-\u9fff]", text))
+    latin_count = len(re.findall(r"[A-Za-z]", text))
+    return "zh-CN" if cjk_count >= max(4, latin_count // 3) else "en-US"
+
+
+def _manual_item_id(*, title: str, published_at: datetime | None, url: str, source: str) -> str:
+    published = published_at.isoformat() if published_at else ""
+    basis = "\n".join([title.strip().lower(), published, url.strip().lower(), source.strip().lower()])
+    return f"manual:{hashlib.sha1(basis.encode('utf-8')).hexdigest()[:16]}"
+
+
+def _manual_group_id(name: str) -> str:
+    basis = f"{name.strip().lower()}\n{datetime.now(UTC).isoformat()}"
+    return f"manual-group:{hashlib.sha1(basis.encode('utf-8')).hexdigest()[:10]}"
+
+
+def _news_meta(record: EarthNewsItem) -> dict[str, Any]:
+    location_meta = record.location_meta if isinstance(record.location_meta, dict) else {}
+    news_meta = location_meta.get("news_meta")
+    return dict(news_meta) if isinstance(news_meta, dict) else {}
+
+
+def _record_source_type(record: EarthNewsItem) -> str:
+    return str(_news_meta(record).get("feed_type") or _news_meta(record).get("source_type") or "rss")
+
+
+def _record_manual_group_id(record: EarthNewsItem) -> str:
+    return str(_news_meta(record).get("manual_group_id") or DEFAULT_MANUAL_NEWS_GROUP_ID)
+
+
+def _rss_group_id(record: EarthNewsItem) -> str:
+    basis = "\n".join(
+        [
+            _record_source_type(record),
+            str(record.feed_name or ""),
+            str(record.source or ""),
+        ]
+    )
+    return f"rss:{hashlib.sha1(basis.encode('utf-8')).hexdigest()[:12]}"
+
+
+def _default_manual_group() -> dict[str, Any]:
+    return {
+        "id": DEFAULT_MANUAL_NEWS_GROUP_ID,
+        "name": DEFAULT_MANUAL_NEWS_GROUP_NAME,
+        "sort_order": 0,
+    }
+
+
+def _normalize_manual_groups_payload(payload: Any) -> list[dict[str, Any]]:
+    raw_groups = payload.get("groups") if isinstance(payload, dict) else None
+    normalized: list[dict[str, Any]] = []
+    seen: set[str] = set()
+    for index, item in enumerate(raw_groups if isinstance(raw_groups, list) else []):
+        if not isinstance(item, dict):
+            continue
+        group_id = str(item.get("id") or "").strip()
+        name = _clean_text(item.get("name"), max_length=120)
+        if not group_id or not name or group_id in seen:
+            continue
+        normalized.append(
+            {
+                "id": group_id,
+                "name": name,
+                "sort_order": int(item.get("sort_order") or index),
+            }
+        )
+        seen.add(group_id)
+    if DEFAULT_MANUAL_NEWS_GROUP_ID not in seen:
+        normalized.insert(0, _default_manual_group())
+    return sorted(normalized, key=lambda item: (int(item.get("sort_order") or 0), str(item.get("name") or "")))
+
+
+async def _get_manual_groups_record(db: AsyncSession) -> SystemSetting | None:
+    result = await db.execute(
+        select(SystemSetting).where(SystemSetting.category == EARTH_NEWS_MANUAL_GROUPS_CATEGORY)
+    )
+    return result.scalar_one_or_none()
+
+
+async def get_manual_news_groups(db: AsyncSession) -> list[dict[str, Any]]:
+    record = await _get_manual_groups_record(db)
+    return _normalize_manual_groups_payload(record.payload if record else None)
+
+
+async def _save_manual_news_groups(db: AsyncSession, groups: list[dict[str, Any]]) -> list[dict[str, Any]]:
+    normalized = _normalize_manual_groups_payload({"groups": groups})
+    record = await _get_manual_groups_record(db)
+    payload = {"groups": normalized}
+    if record is None:
+        db.add(SystemSetting(category=EARTH_NEWS_MANUAL_GROUPS_CATEGORY, payload=payload))
+    else:
+        record.payload = payload
+    await db.flush()
+    return normalized
+
+
+async def resolve_manual_news_group(db: AsyncSession, group_id: str | None) -> ManualNewsGroup:
+    normalized_id = str(group_id or DEFAULT_MANUAL_NEWS_GROUP_ID).strip() or DEFAULT_MANUAL_NEWS_GROUP_ID
+    groups = await get_manual_news_groups(db)
+    match = next((item for item in groups if item.get("id") == normalized_id), None)
+    if match is None and normalized_id != DEFAULT_MANUAL_NEWS_GROUP_ID:
+        raise ValueError(f"手动新闻组不存在：{normalized_id}")
+    match = match or _default_manual_group()
+    return ManualNewsGroup(
+        id=str(match["id"]),
+        name=str(match["name"]),
+        sort_order=int(match.get("sort_order") or 0),
+    )
+
+
+async def create_manual_news_group(db: AsyncSession, name: str) -> dict[str, Any]:
+    group_name = _clean_text(name, max_length=120)
+    if not group_name:
+        raise ValueError("新闻组名称不能为空。")
+    groups = await get_manual_news_groups(db)
+    group = {"id": _manual_group_id(group_name), "name": group_name, "sort_order": len(groups)}
+    groups.append(group)
+    await _save_manual_news_groups(db, groups)
+    return group
+
+
+async def rename_manual_news_group(db: AsyncSession, group_id: str, name: str) -> dict[str, Any]:
+    group_name = _clean_text(name, max_length=120)
+    if not group_name:
+        raise ValueError("新闻组名称不能为空。")
+    groups = await get_manual_news_groups(db)
+    match = next((item for item in groups if item.get("id") == group_id), None)
+    if match is None:
+        raise ValueError(f"手动新闻组不存在：{group_id}")
+    match["name"] = group_name
+    await _save_manual_news_groups(db, groups)
+
+    result = await db.execute(
+        select(EarthNewsItem).where(
+            EarthNewsItem.location_meta.op("->")("news_meta").op("->>")("feed_type") == NewsSourceType.MANUAL.value
+        )
+    )
+    for record in result.scalars().all():
+        if _record_manual_group_id(record) != group_id:
+            continue
+        location_meta = dict(record.location_meta or {})
+        news_meta = dict(location_meta.get("news_meta") or {})
+        news_meta["manual_group_name"] = group_name
+        location_meta["news_meta"] = news_meta
+        record.location_meta = location_meta
+    await db.flush()
+    return match
+
+
+def _normalize_region(value: object) -> str:
+    region = str(value or "global").strip().lower() or "global"
+    if region not in REGION_ANCHORS:
+        raise ValueError(f"region 不支持：{region}")
+    return region
+
+
+def _normalize_tags(value: object) -> list[str]:
+    if value is None:
+        return []
+    if isinstance(value, str):
+        parts = re.split(r"[,，\n]", value)
+    elif isinstance(value, list):
+        parts = [str(item) for item in value]
+    else:
+        raise ValueError("tags 必须是字符串数组或逗号分隔字符串。")
+    return [item.strip() for item in parts if item.strip()][:20]
+
+
+def _normalize_location(value: object) -> NewsTargetLocation | None:
+    if value in (None, ""):
+        return None
+    if not isinstance(value, dict):
+        raise ValueError("location 必须是对象。")
+    lat = value.get("latitude")
+    lon = value.get("longitude")
+    if lat in (None, "") and lon in (None, ""):
+        return None
+    try:
+        latitude = float(lat)
+        longitude = float(lon)
+    except (TypeError, ValueError) as exc:
+        raise ValueError("location.latitude / longitude 必须是数字。") from exc
+    if not -90 <= latitude <= 90 or not -180 <= longitude <= 180:
+        raise ValueError("location 经纬度超出范围。")
+    label = _clean_text(value.get("label"), max_length=255)
+    if not label:
+        label = f"{latitude:.4f}, {longitude:.4f}"
+    return NewsTargetLocation(
+        latitude=latitude,
+        longitude=longitude,
+        label=label,
+        source="manual_location",
+        confidence=1.0,
+        country=_clean_text(value.get("country"), max_length=100) or None,
+        city=_clean_text(value.get("city"), max_length=100) or None,
+    )
+
+
+def _manual_source(source_name: str, *, region: str) -> NewsFeedSource:
+    return NewsFeedSource(
+        id=MANUAL_NEWS_SOURCE_ID,
+        name=source_name or MANUAL_NEWS_SOURCE_LABEL,
+        region=region,
+        feed_url="",
+        homepage_url="",
+        source_type=NewsSourceType.MANUAL.value,
+        default_category="other",
+        source_tags=("manual",),
+    )
+
+
+def _manual_feed(category: str) -> NewsFeedEndpoint:
+    return NewsFeedEndpoint(
+        id=MANUAL_NEWS_SOURCE_ID,
+        name=MANUAL_NEWS_SOURCE_LABEL,
+        url="",
+        type=NewsSourceType.MANUAL.value,
+        default_category=category or "other",
+        tags=("manual",),
+        priority=1,
+    )
+
+
+def parsed_manual_news_item(
+    payload: dict[str, Any],
+    *,
+    item_id_override: str | None = None,
+) -> tuple[ParsedNewsItem, NewsTargetLocation | None, str]:
+    title = _clean_text(payload.get("title"), max_length=MANUAL_NEWS_MAX_TITLE_LENGTH)
+    if not title:
+        raise ValueError("title 不能为空。")
+    content = _clean_text(payload.get("content"), max_length=MANUAL_NEWS_MAX_CONTENT_LENGTH)
+    summary = _clean_text(payload.get("summary"), max_length=MANUAL_NEWS_MAX_SUMMARY_LENGTH)
+    if not summary:
+        summary = _clean_text(content, max_length=240) if content else title
+    source = _clean_text(payload.get("source"), max_length=255) or MANUAL_NEWS_SOURCE_LABEL
+    region = _normalize_region(payload.get("region"))
+    published_at = _parse_datetime(payload.get("published_at")) or datetime.now(UTC)
+    url = str(payload.get("url") or "").strip()
+    category = str(payload.get("category") or "other").strip().lower() or "other"
+    if category not in ALLOWED_NEWS_CATEGORY_KEYS:
+        raise ValueError(f"category 不支持：{category}")
+    tags = _normalize_tags(payload.get("tags"))
+    target = _normalize_location(payload.get("location"))
+    language = str(payload.get("content_language") or "").strip() or _detect_language(title, summary, content)
+    localizations = {
+        language: {
+            "title": title,
+            "summary": summary,
+        }
+    }
+    item = ParsedNewsItem(
+        id=item_id_override
+        or _manual_item_id(title=title, published_at=published_at, url=url, source=source),
+        title=title,
+        summary=summary,
+        url=url,
+        source=source,
+        feed_name=MANUAL_NEWS_SOURCE_LABEL,
+        feed_region=region,
+        homepage_url=str(payload.get("homepage_url") or ""),
+        published_at=published_at,
+        content_language=language,
+        localizations=localizations,
+        enrichment_status=NewsEnrichmentStatus.PENDING.value,
+        source_tags=["manual"],
+        feed_id=MANUAL_NEWS_SOURCE_ID,
+        feed_type=NewsSourceType.MANUAL.value,
+        feed_default_category=category,
+        category=category,
+        item_tags=tags,
+        tagging_source=NewsTaggingSource.MANUAL.value if payload.get("category") else NewsTaggingSource.RULES.value,
+        tagging_confidence=0.9 if payload.get("category") else 0.0,
+    )
+    source_config = _manual_source(source, region=region)
+    feed = _manual_feed(category)
+    apply_news_classification(item, source_config, feed=feed)
+    if payload.get("category"):
+        item.category = category
+        item.tagging_source = NewsTaggingSource.MANUAL.value
+        item.tagging_confidence = 0.9
+    if tags:
+        item.item_tags = sorted(set([*item.item_tags, *tags]))
+    return item, target, content
+
+
+def _manual_editable(record: EarthNewsItem) -> bool:
+    if record.id.startswith("manual:"):
+        return True
+    news_meta = (record.location_meta or {}).get("news_meta") if isinstance(record.location_meta, dict) else None
+    return isinstance(news_meta, dict) and news_meta.get("feed_type") == NewsSourceType.MANUAL.value
+
+
+async def _broadcast_news_reload() -> None:
+    await broadcaster.broadcast_earth_update(
+        {
+            "action": "database_changed",
+            "source": "earth_news_items",
+            "layers": ["news"],
+            "refresh_strategy": "reload",
+        }
+    )
+
+
+async def upsert_manual_news_item(
+    db: AsyncSession,
+    payload: dict[str, Any],
+    *,
+    item_id_override: str | None = None,
+    group_id: str | None = None,
+) -> ManualNewsWriteResult:
+    item, target, content = parsed_manual_news_item(payload, item_id_override=item_id_override)
+    group = await resolve_manual_news_group(db, group_id or payload.get("group_id"))
+    existing = await db.get(EarthNewsItem, item.id)
+    created = existing is None
+    patch = build_target_location_patch(item, target) if target else build_anchor_location_patch(item)
+    patch_meta = dict(patch.get("location_meta") or {})
+    patch_news_meta = dict(patch_meta.get("news_meta") or {})
+    patch_news_meta["feed_type"] = NewsSourceType.MANUAL.value
+    patch_news_meta["source_type"] = NewsSourceType.MANUAL.value
+    patch_news_meta["manual_group_id"] = group.id
+    patch_news_meta["manual_group_name"] = group.name
+    patch_meta["news_meta"] = patch_news_meta
+    patch["location_meta"] = patch_meta
+    now = datetime.now(UTC)
+    record = existing or EarthNewsItem(
+        id=item.id,
+        title=item.title,
+        summary=item.summary,
+        content_language=item.content_language,
+        localizations=dict(item.localizations or {}),
+        url=item.url,
+        source=item.source,
+        feed_name=item.feed_name,
+        region=item.feed_region,
+        homepage_url=item.homepage_url,
+        published_at=item.published_at,
+        latitude=patch["latitude"],
+        longitude=patch["longitude"],
+        location_label=patch["location_label"],
+        location_source=patch["location_source"],
+        verified=patch["verified"],
+        location_meta=patch["location_meta"],
+        first_seen_at=now,
+        last_seen_at=now,
+        resolved_at=now if patch["verified"] else None,
+        enrichment_status=item.enrichment_status,
+    )
+    if existing is None:
+        db.add(record)
+    else:
+        if not _manual_editable(record):
+            raise PermissionError("RSS 新闻不允许通过手动新闻接口编辑。")
+        record.title = item.title
+        record.summary = item.summary
+        record.content_language = item.content_language
+        record.localizations = dict(item.localizations or {})
+        record.url = item.url
+        record.source = item.source
+        record.feed_name = item.feed_name
+        record.region = item.feed_region
+        record.homepage_url = item.homepage_url
+        record.published_at = item.published_at
+        record.last_seen_at = now
+        if target is None and record.location_source == "manual_location":
+            merged_meta = dict(record.location_meta or {})
+            patch_meta = patch.get("location_meta") if isinstance(patch, dict) else None
+            patch_news_meta = patch_meta.get("news_meta") if isinstance(patch_meta, dict) else None
+            if isinstance(patch_news_meta, dict):
+                merged_meta["news_meta"] = patch_news_meta
+            record.location_meta = merged_meta
+        else:
+            record.location_meta = patch["location_meta"]
+        if target:
+            record.latitude = patch["latitude"]
+            record.longitude = patch["longitude"]
+            record.location_label = patch["location_label"]
+            record.location_source = patch["location_source"]
+            record.verified = patch["verified"]
+            record.resolved_at = now
+        elif record.location_source != "manual_location":
+            record.latitude = patch["latitude"]
+            record.longitude = patch["longitude"]
+            record.location_label = patch["location_label"]
+            record.location_source = patch["location_source"]
+            record.verified = patch["verified"]
+            record.resolved_at = None
+        record.enrichment_status = NewsEnrichmentStatus.PENDING.value
+        record.enrichment_error = None
+        record.enriched_at = None
+    if content:
+        meta = dict(record.location_meta or {})
+        meta["manual_content"] = content
+        record.location_meta = meta
+    await db.flush()
+
+    queued = await enqueue_target_location_job(build_target_location_job_payload(item), force=True)
+    if queued:
+        record.enrichment_status = NewsEnrichmentStatus.QUEUED.value
+    await db.flush()
+    return ManualNewsWriteResult(item=record, created=created, queued=queued)
+
+
+async def import_manual_news_items(
+    db: AsyncSession,
+    payload: list[Any],
+    *,
+    group_id: str | None = None,
+) -> dict[str, Any]:
+    if len(payload) > MANUAL_NEWS_MAX_IMPORT_ITEMS:
+        raise ValueError(f"单次最多导入 {MANUAL_NEWS_MAX_IMPORT_ITEMS} 条。")
+    created = 0
+    updated = 0
+    queued = 0
+    errors: list[dict[str, Any]] = []
+    for index, raw_item in enumerate(payload):
+        if not isinstance(raw_item, dict):
+            errors.append({"index": index, "error": "条目必须是 JSON 对象。"})
+            continue
+        try:
+            result = await upsert_manual_news_item(db, raw_item, group_id=group_id)
+            created += 1 if result.created else 0
+            updated += 0 if result.created else 1
+            queued += 1 if result.queued else 0
+        except Exception as exc:
+            errors.append({"index": index, "error": str(exc)})
+    if errors and created == 0 and updated == 0:
+        raise ValueError("导入失败，未写入任何新闻。")
+    return {"created": created, "updated": updated, "queued": queued, "failed": len(errors), "errors": errors}
+
+
+async def parse_manual_news_import_upload(raw_bytes: bytes) -> list[Any]:
+    try:
+        payload = json.loads(raw_bytes.decode("utf-8-sig"))
+    except UnicodeDecodeError as exc:
+        raise ValueError("JSON 文件必须使用 UTF-8 编码。") from exc
+    except json.JSONDecodeError as exc:
+        raise ValueError(f"JSON 解析失败：第 {exc.lineno} 行第 {exc.colno} 列。") from exc
+    if not isinstance(payload, list):
+        raise ValueError("JSON 顶层必须是数组。")
+    return payload
+
+
+def serialize_news_record(record: EarthNewsItem, *, locale: str = DEFAULT_NEWS_LOCALE) -> dict[str, Any]:
+    item = record_to_parsed_news_item(record)
+    payload = _serialize_item(item, active_region=item.feed_region, locale=locale)
+    news_meta = _news_meta(record)
+    payload["editable"] = _manual_editable(record)
+    payload["source_type"] = payload.get("feed_type")
+    payload["status"] = record.enrichment_status
+    payload["translated"] = bool((record.localizations or {}).get("zh-CN") and (record.localizations or {}).get("en-US"))
+    payload["manual_content"] = (record.location_meta or {}).get("manual_content") if isinstance(record.location_meta, dict) else None
+    payload["manual_group_id"] = news_meta.get("manual_group_id")
+    payload["manual_group_name"] = news_meta.get("manual_group_name")
+    return payload
+
+
+def _record_matches_group(record: EarthNewsItem, group_id: str) -> bool:
+    source_type = _record_source_type(record)
+    if source_type == NewsSourceType.MANUAL.value:
+        return _record_manual_group_id(record) == group_id
+    return _rss_group_id(record) == group_id
+
+
+async def list_news_records(
+    db: AsyncSession,
+    *,
+    page: int,
+    page_size: int,
+    source_type: str | None = None,
+    region: str | None = None,
+    category: str | None = None,
+    status_filter: str | None = None,
+    group_id: str | None = None,
+) -> dict[str, Any]:
+    page = max(page, 1)
+    page_size = min(max(page_size, 1), 100)
+    query = select(EarthNewsItem)
+    count_query = select(func.count(EarthNewsItem.id))
+    filters = []
+    if region and region != "all":
+        filters.append(EarthNewsItem.region == region)
+    if status_filter and status_filter != "all":
+        filters.append(EarthNewsItem.enrichment_status == status_filter)
+    if source_type and source_type != "all":
+        filters.append(EarthNewsItem.location_meta.op("->")("news_meta").op("->>")("feed_type") == source_type)
+    if category and category != "all":
+        filters.append(EarthNewsItem.location_meta.op("->")("news_meta").op("->>")("category") == category)
+    for clause in filters:
+        query = query.where(clause)
+        count_query = count_query.where(clause)
+    ordered_query = query.order_by(EarthNewsItem.published_at.desc().nullslast(), EarthNewsItem.last_seen_at.desc())
+    if group_id:
+        result = await db.execute(ordered_query)
+        all_records = [record for record in result.scalars().all() if _record_matches_group(record, group_id)]
+        total = len(all_records)
+        records = all_records[(page - 1) * page_size : page * page_size]
+    else:
+        total_result = await db.execute(count_query)
+        result = await db.execute(
+            ordered_query.offset((page - 1) * page_size).limit(page_size)
+        )
+        records = list(result.scalars().all())
+        total = int(total_result.scalar() or 0)
+    return {
+        "items": [serialize_news_record(record) for record in records],
+        "page": page,
+        "page_size": page_size,
+        "total": total,
+    }
+
+
+async def list_news_groups(db: AsyncSession, *, locale: str = DEFAULT_NEWS_LOCALE) -> dict[str, Any]:
+    manual_groups = await get_manual_news_groups(db)
+    manual_by_id: dict[str, dict[str, Any]] = {
+        str(group["id"]): {
+            "id": str(group["id"]),
+            "name": str(group["name"]),
+            "group_type": "manual",
+            "source_type": NewsSourceType.MANUAL.value,
+            "editable": True,
+            "sort_order": int(group.get("sort_order") or 0),
+            "count": 0,
+            "items": [],
+        }
+        for group in manual_groups
+    }
+    rss_by_id: dict[str, dict[str, Any]] = {}
+    result = await db.execute(
+        select(EarthNewsItem).order_by(EarthNewsItem.published_at.desc().nullslast(), EarthNewsItem.last_seen_at.desc())
+    )
+    for record in result.scalars().all():
+        serialized = serialize_news_record(record, locale=locale)
+        source_type = _record_source_type(record)
+        if source_type == NewsSourceType.MANUAL.value:
+            group_id = _record_manual_group_id(record)
+            group = manual_by_id.setdefault(
+                group_id,
+                {
+                    "id": group_id,
+                    "name": str(_news_meta(record).get("manual_group_name") or DEFAULT_MANUAL_NEWS_GROUP_NAME),
+                    "group_type": "manual",
+                    "source_type": NewsSourceType.MANUAL.value,
+                    "editable": True,
+                    "sort_order": len(manual_by_id),
+                    "count": 0,
+                    "items": [],
+                },
+            )
+        else:
+            group_id = _rss_group_id(record)
+            group = rss_by_id.setdefault(
+                group_id,
+                {
+                    "id": group_id,
+                    "name": record.feed_name or record.source or "RSS 新闻",
+                    "group_type": "rss",
+                    "source_type": source_type,
+                    "editable": False,
+                    "region": record.region,
+                    "source": record.source,
+                    "feed_name": record.feed_name,
+                    "count": 0,
+                    "items": [],
+                },
+            )
+        group["count"] = int(group.get("count") or 0) + 1
+        group.setdefault("items", []).append(serialized)
+    manual_items = sorted(manual_by_id.values(), key=lambda item: (int(item.get("sort_order") or 0), str(item.get("name") or "")))
+    rss_items = sorted(rss_by_id.values(), key=lambda item: str(item.get("name") or ""))
+    return {"groups": [*manual_items, *rss_items], "manual_groups": manual_items, "rss_groups": rss_items}
+
+
+async def get_news_record_or_404(db: AsyncSession, item_id: str) -> EarthNewsItem | None:
+    return await db.get(EarthNewsItem, item_id)
+
+
+async def delete_manual_news_item(db: AsyncSession, item_id: str) -> bool:
+    record = await db.get(EarthNewsItem, item_id)
+    if record is None:
+        return False
+    if not _manual_editable(record):
+        raise PermissionError("RSS 新闻不允许通过手动新闻接口删除。")
+    await db.execute(delete(EarthNewsItem).where(EarthNewsItem.id == item_id))
+    await db.flush()
+    return True
+
+
+async def reprocess_manual_news_item(db: AsyncSession, item_id: str) -> bool:
+    record = await db.get(EarthNewsItem, item_id)
+    if record is None:
+        return False
+    if not _manual_editable(record):
+        raise PermissionError("RSS 新闻不允许通过手动新闻接口重新处理。")
+    item = record_to_parsed_news_item(record)
+    queued = await enqueue_target_location_job(build_target_location_job_payload(item), force=True)
+    if queued:
+        record.enrichment_status = NewsEnrichmentStatus.QUEUED.value
+        record.enrichment_error = None
+    await db.flush()
+    return queued
+
+
+async def broadcast_manual_news_changed() -> None:
+    await _broadcast_news_reload()

backend/app/services/earth_news_queue.py

@@ -14,10 +14,14 @@ from app.core.logging import get_logger
 logger = get_logger(__name__, service="earth_news")
 
 TARGET_LOCATION_STREAM = "earth_news:target_location:jobs"
+TARGET_LOCATION_PRIORITY_STREAM = "earth_news:target_location:priority"
 TARGET_LOCATION_GROUP = "earth_news_target_location"
 TARGET_LOCATION_DEAD_LETTER_STREAM = "earth_news:target_location:dead"
 TARGET_LOCATION_RESULT_TTL_SECONDS = 60 * 60 * 12
 TARGET_LOCATION_JOB_DEDUP_TTL_SECONDS = 60 * 60 * 6
+TARGET_LOCATION_PRIORITY_JOB_DEDUP_TTL_SECONDS = 60 * 5
+TARGET_LOCATION_PENDING_RECLAIM_IDLE_MS = 2 * 60 * 1000
+TARGET_LOCATION_PRIORITY_READ_BLOCK_MS = 1
 TARGET_LOCATION_MAX_ATTEMPTS = 3
 
 _redis_client: redis.Redis | None = None
@@ -28,6 +32,7 @@ class NewsTargetLocationMessage:
     message_id: str
     item_id: str
     payload: dict[str, Any]
+    stream_name: str = TARGET_LOCATION_STREAM
     attempts: int = 0
 
 
@@ -44,7 +49,7 @@ class NewsTargetLocationQueue(Protocol):
     ) -> list[NewsTargetLocationMessage]:
         ...
 
-    async def ack(self, message_id: str) -> None:
+    async def ack(self, message: NewsTargetLocationMessage) -> None:
         ...
 
     async def retry_or_dead_letter(
@@ -71,6 +76,10 @@ def _queued_key(item_id: str) -> str:
     return f"earth_news:target_location:queued:{item_id}"
 
 
+def _priority_queued_key(item_id: str) -> str:
+    return f"earth_news:target_location:priority_queued:{item_id}"
+
+
 class RedisStreamsNewsTargetLocationQueue:
     def __init__(self, client: redis.Redis | None = None) -> None:
         self.client = client or _get_redis_client()
@@ -79,34 +88,44 @@ class RedisStreamsNewsTargetLocationQueue:
     async def _ensure_group(self) -> None:
         if self._group_ready:
             return
-        try:
-            await self.client.xgroup_create(
-                TARGET_LOCATION_STREAM,
-                TARGET_LOCATION_GROUP,
-                id="0",
-                mkstream=True,
-            )
-        except ResponseError as exc:
-            if "BUSYGROUP" not in str(exc):
-                raise
+        for stream_name in (TARGET_LOCATION_PRIORITY_STREAM, TARGET_LOCATION_STREAM):
+            try:
+                await self.client.xgroup_create(
+                    stream_name,
+                    TARGET_LOCATION_GROUP,
+                    id="0",
+                    mkstream=True,
+                )
+            except ResponseError as exc:
+                if "BUSYGROUP" not in str(exc):
+                    raise
         self._group_ready = True
 
     async def enqueue(self, *, item_id: str, payload: dict[str, Any], force: bool = False) -> bool:
         await self._ensure_group()
         if force:
-            await self.client.delete(_result_key(item_id), _queued_key(item_id))
+            await self.client.delete(_result_key(item_id))
+            queued_key = _priority_queued_key(item_id)
         elif await self.client.exists(_result_key(item_id)):
             return False
+        else:
+            queued_key = _queued_key(item_id)
+        dedup_ttl = (
+            TARGET_LOCATION_PRIORITY_JOB_DEDUP_TTL_SECONDS
+            if force
+            else TARGET_LOCATION_JOB_DEDUP_TTL_SECONDS
+        )
         queued = await self.client.set(
-            _queued_key(item_id),
+            queued_key,
             "1",
             nx=True,
-            ex=TARGET_LOCATION_JOB_DEDUP_TTL_SECONDS,
+            ex=dedup_ttl,
         )
         if not queued:
-            return bool(await self.client.exists(_queued_key(item_id)))
+            return bool(await self.client.exists(queued_key))
+        stream_name = TARGET_LOCATION_PRIORITY_STREAM if force else TARGET_LOCATION_STREAM
         await self.client.xadd(
-            TARGET_LOCATION_STREAM,
+            stream_name,
             {
                 "item_id": item_id,
                 "attempts": "0",
@@ -123,39 +142,104 @@ class RedisStreamsNewsTargetLocationQueue:
         block_ms: int,
     ) -> list[NewsTargetLocationMessage]:
         await self._ensure_group()
-        streams = await self.client.xreadgroup(
+        streams = []
+        priority_claimed = await self._claim_stale_messages(
+            stream_name=TARGET_LOCATION_PRIORITY_STREAM,
+            consumer_name=consumer_name,
+            count=count,
+        )
+        if priority_claimed:
+            return priority_claimed
+
+        priority_messages = await self.client.xreadgroup(
             TARGET_LOCATION_GROUP,
             consumer_name,
-            {TARGET_LOCATION_STREAM: ">"},
+            {TARGET_LOCATION_PRIORITY_STREAM: ">"},
             count=count,
-            block=block_ms,
+            block=TARGET_LOCATION_PRIORITY_READ_BLOCK_MS,
         )
+        if priority_messages:
+            streams = priority_messages
+        else:
+            regular_claimed = await self._claim_stale_messages(
+                stream_name=TARGET_LOCATION_STREAM,
+                consumer_name=consumer_name,
+                count=count,
+            )
+            if regular_claimed:
+                return regular_claimed
+            streams = await self.client.xreadgroup(
+                TARGET_LOCATION_GROUP,
+                consumer_name,
+                {TARGET_LOCATION_STREAM: ">"},
+                count=count,
+                block=block_ms,
+            )
         messages: list[NewsTargetLocationMessage] = []
-        for _stream_name, stream_messages in streams:
+        for stream_name, stream_messages in streams:
             for message_id, fields in stream_messages:
-                raw_payload = fields.get("payload")
-                item_id = fields.get("item_id")
-                if not raw_payload or not item_id:
-                    await self.ack(message_id)
-                    continue
-                try:
-                    payload = json.loads(raw_payload)
-                except json.JSONDecodeError:
-                    await self.ack(message_id)
-                    continue
-                attempts = int(fields.get("attempts") or 0)
-                messages.append(
-                    NewsTargetLocationMessage(
-                        message_id=message_id,
-                        item_id=item_id,
-                        payload=payload,
-                        attempts=attempts,
-                    )
-                )
+                message = await self._message_from_fields(stream_name, message_id, fields)
+                if message is not None:
+                    messages.append(message)
         return messages
 
-    async def ack(self, message_id: str) -> None:
-        await self.client.xack(TARGET_LOCATION_STREAM, TARGET_LOCATION_GROUP, message_id)
+    async def _claim_stale_messages(
+        self,
+        *,
+        stream_name: str,
+        consumer_name: str,
+        count: int,
+    ) -> list[NewsTargetLocationMessage]:
+        try:
+            _next_id, claimed, _deleted = await self.client.xautoclaim(
+                stream_name,
+                TARGET_LOCATION_GROUP,
+                consumer_name,
+                TARGET_LOCATION_PENDING_RECLAIM_IDLE_MS,
+                start_id="0-0",
+                count=count,
+            )
+        except ResponseError:
+            return []
+        messages: list[NewsTargetLocationMessage] = []
+        for message_id, fields in claimed:
+            message = await self._message_from_fields(stream_name, message_id, fields)
+            if message is not None:
+                messages.append(message)
+        return messages
+
+    async def _message_from_fields(
+        self,
+        stream_name: str,
+        message_id: str,
+        fields: dict[str, str],
+    ) -> NewsTargetLocationMessage | None:
+        raw_payload = fields.get("payload")
+        item_id = fields.get("item_id")
+        if not raw_payload or not item_id:
+            await self._discard_message(stream_name, message_id)
+            return None
+        try:
+            payload = json.loads(raw_payload)
+        except json.JSONDecodeError:
+            await self._discard_message(stream_name, message_id)
+            return None
+        attempts = int(fields.get("attempts") or 0)
+        return NewsTargetLocationMessage(
+            message_id=message_id,
+            item_id=item_id,
+            payload=payload,
+            stream_name=stream_name,
+            attempts=attempts,
+        )
+
+    async def ack(self, message: NewsTargetLocationMessage) -> None:
+        await self.client.xack(message.stream_name, TARGET_LOCATION_GROUP, message.message_id)
+        await self.client.xdel(message.stream_name, message.message_id)
+
+    async def _discard_message(self, stream_name: str, message_id: str) -> None:
+        await self.client.xack(stream_name, TARGET_LOCATION_GROUP, message_id)
+        await self.client.xdel(stream_name, message_id)
 
     async def retry_or_dead_letter(
         self,
@@ -163,7 +247,7 @@ class RedisStreamsNewsTargetLocationQueue:
         *,
         error: str,
     ) -> None:
-        await self.ack(message.message_id)
+        await self.ack(message)
         if message.attempts + 1 >= TARGET_LOCATION_MAX_ATTEMPTS:
             await self.client.xadd(
                 TARGET_LOCATION_DEAD_LETTER_STREAM,
@@ -176,7 +260,7 @@ class RedisStreamsNewsTargetLocationQueue:
             )
             return
         await self.client.xadd(
-            TARGET_LOCATION_STREAM,
+            message.stream_name,
             {
                 "item_id": message.item_id,
                 "attempts": str(message.attempts + 1),
@@ -231,4 +315,4 @@ async def save_target_location_patch(item_id: str, patch: dict[str, Any]) -> Non
         TARGET_LOCATION_RESULT_TTL_SECONDS,
         json.dumps(patch, ensure_ascii=False),
     )
-    await client.delete(_queued_key(item_id))
+    await client.delete(_queued_key(item_id), _priority_queued_key(item_id))

backend/app/services/earth_news_store.py

@@ -3,7 +3,7 @@ from __future__ import annotations
 from datetime import UTC, datetime
 from typing import Any
 
-from sqlalchemy import func, select
+from sqlalchemy import func, or_, select
 from sqlalchemy.ext.asyncio import AsyncSession
 
 from app.models.earth_news import EarthNewsItem
@@ -11,7 +11,24 @@ from app.services.earth_news import (
     ParsedNewsItem,
     apply_enrichment_patch_to_item,
     build_anchor_location_patch,
+    _news_meta_patch,
 )
+from app.services.earth_news_classification import (
+    breaking_sort_rank,
+    normalize_breaking_level,
+    normalize_breaking_scope,
+)
+
+CRUISE_REGION_ORDER = (
+    "americas",
+    "europe",
+    "middle-east-africa",
+    "asia-pacific",
+    "global",
+)
+CRUISE_REGION_QUERY_MULTIPLIER = 12
+CRUISE_REGION_QUERY_MIN_LIMIT = 240
+CRUISE_REGION_QUERY_MAX_LIMIT = 1000
 
 
 def _coerce_datetime(value: datetime | None) -> datetime | None:
@@ -22,6 +39,17 @@ def _coerce_datetime(value: datetime | None) -> datetime | None:
     return value.astimezone(UTC)
 
 
+def _coerce_meta_datetime(value: Any) -> datetime | None:
+    if isinstance(value, datetime):
+        return _coerce_datetime(value)
+    if not isinstance(value, str) or not value.strip():
+        return None
+    try:
+        return _coerce_datetime(datetime.fromisoformat(value.replace("Z", "+00:00")))
+    except ValueError:
+        return None
+
+
 def _location_patch_from_record(record: EarthNewsItem) -> dict[str, Any]:
     return {
         "latitude": record.latitude,
@@ -34,6 +62,8 @@ def _location_patch_from_record(record: EarthNewsItem) -> dict[str, Any]:
 
 
 def record_to_parsed_news_item(record: EarthNewsItem) -> ParsedNewsItem:
+    location_meta = dict(record.location_meta or {})
+    news_meta = location_meta.get("news_meta") if isinstance(location_meta.get("news_meta"), dict) else {}
     item = ParsedNewsItem(
         id=record.id,
         title=record.title,
@@ -49,11 +79,86 @@ def record_to_parsed_news_item(record: EarthNewsItem) -> ParsedNewsItem:
         enrichment_status=record.enrichment_status or "pending",
         enrichment_error=record.enrichment_error,
         enriched_at=_coerce_datetime(record.enriched_at),
+        source_tags=list(news_meta.get("source_tags") or []),
+        feed_id=str(news_meta.get("feed_id") or ""),
+        feed_type=str(news_meta.get("feed_type") or "rss"),
+        feed_default_category=str(news_meta.get("feed_default_category") or "other"),
+        category=str(news_meta.get("category") or "other"),
+        item_tags=list(news_meta.get("item_tags") or []),
+        tagging_source=str(news_meta.get("tagging_source") or "rules"),
+        tagging_confidence=float(news_meta.get("tagging_confidence") or 0),
+        importance_score=int(news_meta.get("importance_score") or 0),
+        importance_level=str(news_meta.get("importance_level") or "low"),
+        importance_reasons=list(news_meta.get("importance_reasons") or []),
+        market_impact=str(news_meta.get("market_impact") or "none"),
+        breaking_level=normalize_breaking_level(news_meta.get("breaking_level")).value,
+        breaking_scope=normalize_breaking_scope(news_meta.get("breaking_scope")).value,
+        breaking_reasons=list(news_meta.get("breaking_reasons") or []),
+        breaking_source=str(news_meta.get("breaking_source") or "rules"),
+        breaking_confidence=float(news_meta.get("breaking_confidence") or 0),
+        breaking_expires_at=_coerce_meta_datetime(news_meta.get("breaking_expires_at")),
     )
     return apply_enrichment_patch_to_item(item, _location_patch_from_record(record))
 
 
+def _sort_parsed_news_items(items: list[ParsedNewsItem], *, active_region: str) -> list[ParsedNewsItem]:
+    return sorted(
+        items,
+        key=lambda item: (
+            -breaking_sort_rank(item),
+            False
+            if active_region == "global"
+            or (breaking_sort_rank(item) > 0 and normalize_breaking_scope(item.breaking_scope).value == "global")
+            else item.feed_region != active_region,
+            item.published_at is None,
+            -(item.published_at.timestamp() if item.published_at else 0),
+            item.feed_name,
+        ),
+    )
+
+
+def _diversify_parsed_news_items_by_region(
+    items: list[ParsedNewsItem],
+    *,
+    limit: int,
+) -> list[ParsedNewsItem]:
+    if limit <= 0:
+        return []
+    sorted_items = _sort_parsed_news_items(items, active_region="global")
+    buckets: dict[str, list[ParsedNewsItem]] = {}
+    for item in sorted_items:
+        region = item.feed_region or "global"
+        buckets.setdefault(region, []).append(item)
+
+    ordered_regions = [
+        *[region for region in CRUISE_REGION_ORDER if buckets.get(region)],
+        *sorted(region for region in buckets if region not in CRUISE_REGION_ORDER),
+    ]
+    diversified: list[ParsedNewsItem] = []
+    cursor = 0
+    while len(diversified) < limit:
+        added = False
+        for region in ordered_regions:
+            bucket = buckets.get(region) or []
+            if cursor >= len(bucket):
+                continue
+            diversified.append(bucket[cursor])
+            added = True
+            if len(diversified) >= limit:
+                break
+        if not added:
+            break
+        cursor += 1
+    return diversified
+
+
 def _query_sort_key(active_region: str):
+    if active_region == "global":
+        return (
+            EarthNewsItem.published_at.is_(None),
+            EarthNewsItem.published_at.desc().nullslast(),
+            EarthNewsItem.feed_name.asc(),
+        )
     return (
         EarthNewsItem.region != active_region,
         EarthNewsItem.published_at.is_(None),
@@ -62,20 +167,90 @@ def _query_sort_key(active_region: str):
     )
 
 
+def _category_filter_clause(categories: set[str] | None):
+    if not categories:
+        return None
+    return EarthNewsItem.location_meta.op("->")("news_meta").op("->>")("category").in_(sorted(categories))
+
+
+def _source_filter_clause(source_ids: set[str] | None):
+    if not source_ids:
+        return None
+    return EarthNewsItem.location_meta.op("->")("news_meta").op("->>")("source_id").in_(sorted(source_ids))
+
+
 async def list_earth_news_items(
     db: AsyncSession,
     *,
     active_region: str,
     limit: int,
+    categories: set[str] | None = None,
+    source_ids: set[str] | None = None,
 ) -> list[ParsedNewsItem]:
-    regions = {"global", active_region}
-    result = await db.execute(
+    query_limit = limit if source_ids else min(max(limit * 20, limit), 500)
+    query = (
         select(EarthNewsItem)
-        .where(EarthNewsItem.region.in_(regions))
         .order_by(*_query_sort_key(active_region))
-        .limit(limit)
+        .limit(query_limit)
+    )
+    if active_region != "global":
+        news_meta = EarthNewsItem.location_meta.op("->")("news_meta")
+        query = query.where(
+            or_(
+                EarthNewsItem.region.in_({"global", active_region}),
+                news_meta.op("->>")("breaking_scope") == "global",
+            )
+        )
+    category_clause = _category_filter_clause(categories)
+    if category_clause is not None:
+        query = query.where(category_clause)
+    source_clause = _source_filter_clause(source_ids)
+    if source_clause is not None:
+        query = query.where(source_clause)
+    result = await db.execute(query)
+    records = list(result.scalars().all())
+    items = _sort_parsed_news_items(
+        [record_to_parsed_news_item(record) for record in records],
+        active_region=active_region,
+    )
+    if active_region == "global" and not source_ids:
+        return _diversify_parsed_news_items_by_region(items, limit=limit)
+    return items[:limit]
+
+
+async def list_earth_news_cruise_items(
+    db: AsyncSession,
+    *,
+    limit: int,
+    categories: set[str] | None = None,
+    source_ids: set[str] | None = None,
+) -> list[ParsedNewsItem]:
+    query_limit = min(
+        max(limit * CRUISE_REGION_QUERY_MULTIPLIER, CRUISE_REGION_QUERY_MIN_LIMIT),
+        CRUISE_REGION_QUERY_MAX_LIMIT,
+    )
+    query = (
+        select(EarthNewsItem)
+        .order_by(
+            EarthNewsItem.published_at.is_(None),
+            EarthNewsItem.published_at.desc().nullslast(),
+            EarthNewsItem.last_seen_at.desc(),
+            EarthNewsItem.region.asc(),
+            EarthNewsItem.feed_name.asc(),
+        )
+        .limit(query_limit)
+    )
+    category_clause = _category_filter_clause(categories)
+    if category_clause is not None:
+        query = query.where(category_clause)
+    source_clause = _source_filter_clause(source_ids)
+    if source_clause is not None:
+        query = query.where(source_clause)
+    result = await db.execute(query)
+    return _diversify_parsed_news_items_by_region(
+        [record_to_parsed_news_item(record) for record in result.scalars().all()],
+        limit=limit,
     )
-    return [record_to_parsed_news_item(record) for record in result.scalars().all()]
 
 
 async def get_earth_news_freshness(
@@ -83,13 +258,13 @@ async def get_earth_news_freshness(
     *,
     active_region: str,
 ) -> tuple[int, datetime | None]:
-    regions = {"global", active_region}
-    result = await db.execute(
-        select(
-            func.count(EarthNewsItem.id),
-            func.max(func.coalesce(EarthNewsItem.published_at, EarthNewsItem.last_seen_at)),
-        ).where(EarthNewsItem.region.in_(regions))
+    query = select(
+        func.count(EarthNewsItem.id),
+        func.max(func.coalesce(EarthNewsItem.published_at, EarthNewsItem.last_seen_at)),
     )
+    if active_region != "global":
+        query = query.where(EarthNewsItem.region.in_({"global", active_region}))
+    result = await db.execute(query)
     count, newest = result.one()
     item_count = int(count or 0)
     if item_count == 0:
@@ -97,6 +272,33 @@ async def get_earth_news_freshness(
     return item_count, _coerce_datetime(newest)
 
 
+async def get_earth_news_feed_coverage(
+    db: AsyncSession,
+    *,
+    active_region: str,
+    recent_after: datetime | None = None,
+) -> set[tuple[str, str]]:
+    query = select(
+        EarthNewsItem.id,
+        EarthNewsItem.location_meta.op("->")("news_meta").op("->>")("source_id"),
+        EarthNewsItem.location_meta.op("->")("news_meta").op("->>")("feed_id"),
+    )
+    if active_region != "global":
+        query = query.where(EarthNewsItem.region.in_({"global", active_region}))
+    if recent_after is not None:
+        query = query.where(func.coalesce(EarthNewsItem.published_at, EarthNewsItem.last_seen_at) >= recent_after)
+    result = await db.execute(query)
+    coverage: set[tuple[str, str]] = set()
+    for item_id, source_id, feed_id in result.all():
+        normalized_source_id = str(source_id or "").strip()
+        normalized_feed_id = str(feed_id or "").strip()
+        if not normalized_source_id and isinstance(item_id, str) and ":" in item_id:
+            normalized_source_id = item_id.split(":", 1)[0]
+        if normalized_source_id and normalized_feed_id:
+            coverage.add((normalized_source_id, normalized_feed_id))
+    return coverage
+
+
 async def upsert_earth_news_items(db: AsyncSession, items: list[ParsedNewsItem]) -> int:
     if not items:
         return 0
@@ -147,12 +349,20 @@ async def upsert_earth_news_items(db: AsyncSession, items: list[ParsedNewsItem])
         record.homepage_url = item.homepage_url
         record.published_at = item.published_at
         record.last_seen_at = now
+        location_meta = dict(record.location_meta or {})
+        location_meta["news_meta"] = _news_meta_patch(item)
+        record.location_meta = location_meta
         if item.localizations:
+            merged_localizations = {
+                **dict(record.localizations or {}),
+                **dict(item.localizations or {}),
+            }
             record.content_language = item.content_language
-            record.localizations = dict(item.localizations or {})
-            record.enrichment_status = item.enrichment_status
-            record.enrichment_error = item.enrichment_error
-            record.enriched_at = item.enriched_at
+            record.localizations = merged_localizations
+            if item.enrichment_status != "pending" or item.enrichment_error or item.enriched_at:
+                record.enrichment_status = item.enrichment_status
+                record.enrichment_error = item.enrichment_error
+                record.enriched_at = item.enriched_at
         changed += 1
     await db.flush()
     return changed
@@ -188,13 +398,28 @@ async def update_earth_news_item_enrichment(
     if record is None:
         return False
     if "latitude" in patch:
-        record.latitude = float(patch["latitude"])
-        record.longitude = float(patch["longitude"])
-        record.location_label = str(patch["location_label"])
-        record.location_source = str(patch["location_source"])
-        record.verified = bool(patch["verified"])
-        record.location_meta = dict(patch.get("location_meta") or {})
-        record.resolved_at = datetime.now(UTC) if record.verified else None
+        patch_meta = dict(patch.get("location_meta") or {})
+        if record.location_source == "manual_location":
+            current_meta = dict(record.location_meta or {})
+            patch_news_meta = patch_meta.get("news_meta")
+            if isinstance(patch_news_meta, dict):
+                current_meta["news_meta"] = patch_news_meta
+            current_meta["manual_enrichment"] = {
+                "resolution_stage": patch_meta.get("resolution_stage"),
+                "ai_attempted": patch_meta.get("ai_attempted"),
+                "ai_status": patch_meta.get("ai_status"),
+                "ai_error": patch_meta.get("ai_error"),
+                "debug_note": patch_meta.get("debug_note"),
+            }
+            record.location_meta = current_meta
+        else:
+            record.latitude = float(patch["latitude"])
+            record.longitude = float(patch["longitude"])
+            record.location_label = str(patch["location_label"])
+            record.location_source = str(patch["location_source"])
+            record.verified = bool(patch["verified"])
+            record.location_meta = patch_meta
+            record.resolved_at = datetime.now(UTC) if record.verified else None
     if "content_language" in patch:
         record.content_language = str(patch.get("content_language") or "en")
     if "localizations" in patch:

backend/app/services/earth_news_worker.py

@@ -29,6 +29,9 @@ logger = get_logger(__name__, service="earth_news")
 WORKER_BATCH_SIZE = 4
 WORKER_BLOCK_MS = 5000
 WORKER_BACKOFF_SECONDS = 5.0
+WORKER_JOB_TIMEOUT_MIN_SECONDS = 20.0
+WORKER_JOB_TIMEOUT_MAX_SECONDS = 90.0
+WORKER_JOB_TIMEOUT_GRACE_SECONDS = 10.0
 
 _worker_task: asyncio.Task | None = None
 
@@ -109,12 +112,25 @@ async def _run_target_location_worker() -> None:
         if not messages:
             continue
         provider_client = await _build_provider_client()
-        for message in messages:
+        job_timeout = _get_worker_job_timeout(provider_client)
+
+        async def handle_message(message: NewsTargetLocationMessage) -> None:
             try:
-                await process_target_location_message(message, provider_client=provider_client)
-                await queue.ack(message.message_id)
+                await asyncio.wait_for(
+                    process_target_location_message(message, provider_client=provider_client),
+                    timeout=job_timeout,
+                )
+                await queue.ack(message)
             except asyncio.CancelledError:
                 raise
+            except TimeoutError as exc:
+                logger.warning_event(
+                    "Earth news target location worker job timed out",
+                    event="earth_news.target_location.worker_job_timeout",
+                    context={"item_id": message.item_id, "timeout_seconds": job_timeout},
+                )
+                with suppress(Exception):
+                    await queue.retry_or_dead_letter(message, error=str(exc) or "job timed out")
             except Exception as exc:
                 logger.warning_event(
                     "Earth news target location worker job failed",
@@ -124,6 +140,8 @@ async def _run_target_location_worker() -> None:
                 with suppress(Exception):
                     await queue.retry_or_dead_letter(message, error=str(exc))
 
+        await asyncio.gather(*(handle_message(message) for message in messages))
+
 
 def start_earth_news_target_worker() -> None:
     global _worker_task
@@ -140,3 +158,15 @@ async def stop_earth_news_target_worker() -> None:
     with suppress(asyncio.CancelledError):
         await task
     _worker_task = None
+
+
+def _get_worker_job_timeout(provider_client: AIProviderClient | None) -> float:
+    timeout = float(getattr(provider_client, "timeout", 0) or WORKER_JOB_TIMEOUT_MIN_SECONDS)
+    retry_attempts = float(getattr(provider_client, "retry_attempts", 1) or 1)
+    return min(
+        max(
+            timeout * retry_attempts + WORKER_JOB_TIMEOUT_GRACE_SECONDS,
+            WORKER_JOB_TIMEOUT_MIN_SECONDS,
+        ),
+        WORKER_JOB_TIMEOUT_MAX_SECONDS,
+    )

backend/app/services/email.py

@@ -9,12 +9,12 @@ yet to keep behavior obvious after settings changes).
 from __future__ import annotations
 
 from email.message import EmailMessage
-from typing import Literal, Optional
+from typing import Optional
 
 import aiosmtplib
 from sqlalchemy.ext.asyncio import AsyncSession
 
-OtpPurpose = Literal["register", "verify_email", "reset_password"]
+from app.core.enums import OtpPurpose
 
 
 class EmailError(Exception):
@@ -81,15 +81,15 @@ async def send_email(
 
 
 _SUBJECTS: dict[OtpPurpose, str] = {
-    "register": "Confirm your Planet account",
-    "verify_email": "Verify your Planet email",
-    "reset_password": "Reset your Planet password",
+    OtpPurpose.REGISTER: "Confirm your Planet account",
+    OtpPurpose.VERIFY_EMAIL: "Verify your Planet email",
+    OtpPurpose.RESET_PASSWORD: "Reset your Planet password",
 }
 
 _HEADLINES: dict[OtpPurpose, str] = {
-    "register": "Welcome to Planet — confirm your email to activate your account.",
-    "verify_email": "Confirm your new email address to keep your Planet account active.",
-    "reset_password": "Use this code to set a new password for your Planet account.",
+    OtpPurpose.REGISTER: "Welcome to Planet — confirm your email to activate your account.",
+    OtpPurpose.VERIFY_EMAIL: "Confirm your new email address to keep your Planet account active.",
+    OtpPurpose.RESET_PASSWORD: "Use this code to set a new password for your Planet account.",
 }
 
 

backend/app/services/log_tail.py

@@ -0,0 +1,161 @@
+from __future__ import annotations
+
+import asyncio
+from dataclasses import dataclass, field
+from datetime import UTC, datetime
+from typing import Any
+
+from fastapi import WebSocket
+
+from app.db.session import async_session_factory
+from app.services.system_logs import (
+    DEFAULT_LOG_LINE_LIMIT,
+    LOG_SOURCES,
+    MAX_LOG_LINE_LIMIT,
+    read_database_log_events,
+    read_log_events,
+)
+
+DATABASE_LOG_SOURCE_IDS = {"system-db", "audit-db"}
+LOG_TAIL_CHANNEL = "logs_tail"
+LOG_TAIL_INTERVAL_SECONDS = 1.5
+LOG_TAIL_SCAN_MULTIPLIER = 5
+
+
+@dataclass(frozen=True)
+class LogTailConfig:
+    source_id: str
+    limit: int = DEFAULT_LOG_LINE_LIMIT
+    level: str = "all"
+    levels: str | None = None
+    start_date: str | None = None
+    end_date: str | None = None
+    search: str | None = None
+
+
+@dataclass
+class LogTailSubscription:
+    config: LogTailConfig
+    emitted_cursors: set[str] = field(default_factory=set)
+    task: asyncio.Task | None = None
+
+
+class LogTailManager:
+    def __init__(self) -> None:
+        self._subscriptions: dict[WebSocket, LogTailSubscription] = {}
+
+    def normalize_config(self, payload: dict[str, Any]) -> LogTailConfig:
+        source_id = str(payload.get("source_id") or payload.get("source") or "").strip()
+        if not source_id:
+            raise ValueError("source_id is required")
+        if source_id not in LOG_SOURCES and source_id not in DATABASE_LOG_SOURCE_IDS:
+            raise ValueError("Log source not found")
+        try:
+            limit = int(payload.get("limit") or DEFAULT_LOG_LINE_LIMIT)
+        except (TypeError, ValueError) as exc:
+            raise ValueError("limit must be a number") from exc
+        if limit < 1 or limit > MAX_LOG_LINE_LIMIT:
+            raise ValueError(f"limit must be between 1 and {MAX_LOG_LINE_LIMIT}")
+        return LogTailConfig(
+            source_id=source_id,
+            limit=limit,
+            level=str(payload.get("level") or "all"),
+            levels=str(payload.get("levels")).strip() if payload.get("levels") else None,
+            start_date=str(payload.get("start_date")).strip() if payload.get("start_date") else None,
+            end_date=str(payload.get("end_date")).strip() if payload.get("end_date") else None,
+            search=str(payload.get("search")).strip() if payload.get("search") else None,
+        )
+
+    async def subscribe(self, websocket: WebSocket, payload: dict[str, Any]) -> LogTailConfig:
+        config = self.normalize_config(payload)
+        await self.unsubscribe(websocket)
+        subscription = LogTailSubscription(config=config)
+        subscription.task = asyncio.create_task(self._run_tail(websocket, subscription))
+        self._subscriptions[websocket] = subscription
+        return config
+
+    async def unsubscribe(self, websocket: WebSocket) -> None:
+        subscription = self._subscriptions.pop(websocket, None)
+        if subscription and subscription.task:
+            subscription.task.cancel()
+            try:
+                await subscription.task
+            except asyncio.CancelledError:
+                pass
+
+    async def disconnect(self, websocket: WebSocket) -> None:
+        await self.unsubscribe(websocket)
+
+    async def _run_tail(self, websocket: WebSocket, subscription: LogTailSubscription) -> None:
+        first_frame = True
+        while True:
+            events = await self._read_events(subscription.config)
+            if first_frame:
+                visible_events = events[-subscription.config.limit :]
+                subscription.emitted_cursors.update(event.cursor for event in visible_events)
+                await self._send_frame(websocket, subscription.config, "snapshot", visible_events)
+                first_frame = False
+            else:
+                new_events = [
+                    event
+                    for event in events
+                    if event.cursor not in subscription.emitted_cursors
+                ]
+                if new_events:
+                    visible_events = new_events[-subscription.config.limit :]
+                    subscription.emitted_cursors.update(event.cursor for event in visible_events)
+                    await self._send_frame(websocket, subscription.config, "append", visible_events)
+            await asyncio.sleep(LOG_TAIL_INTERVAL_SECONDS)
+
+    async def _read_events(self, config: LogTailConfig):
+        scan_limit = max(config.limit * LOG_TAIL_SCAN_MULTIPLIER, config.limit)
+        if config.source_id in DATABASE_LOG_SOURCE_IDS:
+            async with async_session_factory() as db:
+                events = await read_database_log_events(
+                    config.source_id,
+                    scan_limit=scan_limit,
+                    level=config.level,
+                    levels=config.levels,
+                    start_date=config.start_date,
+                    end_date=config.end_date,
+                    search=config.search,
+                    db=db,
+                )
+                return events or []
+        events = read_log_events(
+            config.source_id,
+            scan_limit=scan_limit,
+            level=config.level,
+            levels=config.levels,
+            start_date=config.start_date,
+            end_date=config.end_date,
+            search=config.search,
+        )
+        return events or []
+
+    async def _send_frame(self, websocket: WebSocket, config: LogTailConfig, mode: str, events) -> None:
+        await websocket.send_json(
+            {
+                "type": "data_frame",
+                "channel": LOG_TAIL_CHANNEL,
+                "timestamp": datetime.now(UTC).isoformat(),
+                "payload": {
+                    "mode": mode,
+                    "source_id": config.source_id,
+                    "line_count": len(events),
+                    "lines": [event.line for event in events],
+                    "filters": {
+                        "limit": config.limit,
+                        "level": config.level,
+                        "levels": config.levels,
+                        "start_date": config.start_date,
+                        "end_date": config.end_date,
+                        "search": config.search,
+                    },
+                    "status": "ok",
+                },
+            }
+        )
+
+
+log_tail_manager = LogTailManager()

backend/app/services/otp.py

@@ -9,14 +9,12 @@ from __future__ import annotations
 
 import json
 import secrets
-from typing import Literal
 
 import bcrypt
 
+from app.core.enums import OtpPurpose
 from app.core.security import redis_client
 
-OtpPurpose = Literal["register", "verify_email", "reset_password"]
-
 CODE_TTL_SECONDS = 600  # 10 minutes
 RESEND_COOLDOWN_SECONDS = 60
 MAX_ATTEMPTS = 5

backend/app/services/persistent_logs.py

@@ -1,14 +1,185 @@
 from __future__ import annotations
 
+import hashlib
+import re
+
+from datetime import UTC, datetime
 from typing import Any
 
 from app.core.logging import get_logger, sanitize_log_value
 from app.core.request_context import get_request_id
 from app.db.session import async_session_factory
-from app.models.system_log import AuditLog, SystemLog
+from app.models.system_log import AuditLog, ObservabilityEvent, ObservabilityEventGroup, SystemLog
 
 logger = get_logger(__name__)
 
+HLS_TRANSIENT_RE = re.compile(r"(index|chunk|segment)[_-]?\d+(?:_\d+)?\.(?:ts|m4s|vtt)", re.IGNORECASE)
+QUERY_RE = re.compile(r"([?&](?:m|t|token|expires|signature|X-Amz-[^=]+)=[^&\\s]+)", re.IGNORECASE)
+UUID_RE = re.compile(r"\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b", re.IGNORECASE)
+CONNECTION_RE = re.compile(r"\bconn_[A-Za-z0-9:._-]+\b")
+NUMBER_RE = re.compile(r"\b\d{5,}\b")
+
+
+def normalize_observability_text(value: Any) -> str:
+    text = str(sanitize_log_value(value or "")).strip()
+    text = QUERY_RE.sub("", text)
+    text = HLS_TRANSIENT_RE.sub("<hls-fragment>", text)
+    text = UUID_RE.sub("<uuid>", text)
+    text = CONNECTION_RE.sub("<connection>", text)
+    text = NUMBER_RE.sub("<number>", text)
+    return re.sub(r"\s+", " ", text).strip()
+
+
+def build_observability_fingerprint(
+    *,
+    source: str,
+    service: str | None = None,
+    module: str | None = None,
+    category: str | None = None,
+    event: str | None = None,
+    message: str,
+    context: dict[str, Any] | None = None,
+) -> str:
+    context = context or {}
+    stable_context = {
+        key: context.get(key)
+        for key in (
+            "task_type",
+            "source_id",
+            "source",
+            "provider",
+            "status_code",
+            "error_type",
+            "details",
+        )
+        if context.get(key) not in (None, "")
+    }
+    raw = "|".join(
+        [
+            normalize_observability_text(source),
+            normalize_observability_text(service),
+            normalize_observability_text(module),
+            normalize_observability_text(category),
+            normalize_observability_text(event),
+            normalize_observability_text(message),
+            normalize_observability_text(stable_context),
+        ]
+    )
+    return hashlib.sha1(raw.encode("utf-8", errors="replace")).hexdigest()
+
+
+def _context_text(context: dict[str, Any] | None, key: str) -> str | None:
+    value = (context or {}).get(key)
+    if value in (None, ""):
+        return None
+    return str(value)
+
+
+async def record_observability_event(
+    *,
+    source: str,
+    level: str,
+    message: str,
+    service: str | None = None,
+    module: str | None = None,
+    event: str | None = None,
+    request_id: str | None = None,
+    trace_id: str | None = None,
+    user_id: int | None = None,
+    category: str | None = None,
+    context: dict[str, Any] | None = None,
+    fingerprint: str | None = None,
+    occurred_at: datetime | None = None,
+    occurrence_count: int = 1,
+) -> None:
+    normalized_context = sanitize_log_value(context or {})
+    if not isinstance(normalized_context, dict):
+        normalized_context = {"value": normalized_context}
+    safe_message = str(sanitize_log_value(message))
+    normalized_level = str(level or "info").lower()
+    count = max(1, int(occurrence_count or 1))
+    event_time = occurred_at or datetime.now(UTC)
+    event_fingerprint = fingerprint or build_observability_fingerprint(
+        source=source,
+        service=service,
+        module=module,
+        category=category,
+        event=event,
+        message=safe_message,
+        context=normalized_context,
+    )
+    detail = _context_text(normalized_context, "detail") or _context_text(normalized_context, "error")
+    affected_sources = sorted(
+        {
+            item
+            for item in (
+                source,
+                service,
+                module,
+                _context_text(normalized_context, "source_id"),
+                _context_text(normalized_context, "source"),
+            )
+            if item
+        }
+    )
+    try:
+        async with async_session_factory() as session:
+            session.add(
+                ObservabilityEvent(
+                    source=source,
+                    service=service,
+                    module=module,
+                    category=category,
+                    event=event,
+                    level=normalized_level,
+                    message=safe_message,
+                    fingerprint=event_fingerprint,
+                    occurred_at=event_time,
+                    request_id=request_id or get_request_id(),
+                    trace_id=trace_id,
+                    user_id=user_id,
+                    task_id=_context_text(normalized_context, "task_id"),
+                    source_ref_id=_context_text(normalized_context, "source_id") or _context_text(normalized_context, "source"),
+                    provider=_context_text(normalized_context, "provider"),
+                    context=normalized_context,
+                    occurrence_count=count,
+                )
+            )
+            group = await session.get(ObservabilityEventGroup, event_fingerprint)
+            if group is None:
+                session.add(
+                    ObservabilityEventGroup(
+                        fingerprint=event_fingerprint,
+                        source=source,
+                        service=service,
+                        module=module,
+                        category=category,
+                        event=event,
+                        last_level=normalized_level,
+                        sample_message=safe_message,
+                        sample_detail=detail,
+                        affected_sources=affected_sources,
+                        count=count,
+                        first_seen_at=event_time,
+                        last_seen_at=event_time,
+                    )
+                )
+            else:
+                group.count = int(group.count or 0) + count
+                group.last_seen_at = event_time
+                group.last_level = normalized_level
+                group.sample_message = safe_message
+                group.sample_detail = detail
+                merged_sources = sorted(set(group.affected_sources or []) | set(affected_sources))
+                group.affected_sources = merged_sources
+            await session.commit()
+    except Exception:
+        logger.exception_event(
+            "Failed to persist observability event",
+            event="observability_event.persist.failed",
+            context={"event_name": event, "source": source},
+        )
+
 
 async def record_system_log(
     *,
@@ -23,6 +194,8 @@ async def record_system_log(
     user_id: int | None = None,
     category: str | None = None,
     context: dict[str, Any] | None = None,
+    fingerprint: str | None = None,
+    occurrence_count: int = 1,
 ) -> None:
     try:
         async with async_session_factory() as session:
@@ -48,6 +221,21 @@ async def record_system_log(
             event="system_log.persist.failed",
             context={"event_name": event, "source": source},
         )
+    await record_observability_event(
+        source=source,
+        service=service,
+        module=module,
+        event=event,
+        level=level,
+        message=message,
+        request_id=request_id,
+        trace_id=trace_id,
+        user_id=user_id,
+        category=category,
+        context=context,
+        fingerprint=fingerprint,
+        occurrence_count=occurrence_count,
+    )
 
 
 async def record_audit_log(

backend/app/services/playground_chat_service.py

@@ -7,9 +7,15 @@ from time import perf_counter
 from uuid import uuid4
 
 from fastapi import HTTPException, status
-from sqlalchemy import func, select, update
+from sqlalchemy import func, select
 from sqlalchemy.ext.asyncio import AsyncSession
 
+from app.core.enums import (
+    PlaygroundMessageKind,
+    PlaygroundMessageRole,
+    PlaygroundMessageStatus,
+)
+from app.core.logging import get_logger
 from app.db.session import async_session_factory
 from app.models.playground_message import PlaygroundMessage
 from app.models.playground_session import PlaygroundSession
@@ -20,20 +26,28 @@ from app.schemas.ai import (
     PlaygroundMessageRecord,
     PlaygroundMessageResendRequest,
     PlaygroundMessageStopRequest,
-    PlaygroundSessionResponse,
     PlaygroundSessionState,
     PlaygroundSessionUpsertRequest,
     PlaygroundThreadResponse,
     SituationalAnalysisRequest,
 )
 from app.services.ai_client import AIProviderClient
+from app.services.business_logs import emit_business_log, exception_context
 from app.services.playground_session_store import _to_response as session_to_response
 from app.services.playground_session_store import upsert_playground_session
 
+logger = get_logger(__name__, service="ai")
 STREAM_CHUNK_SIZE = 24
 STREAM_INTERVAL_SECONDS = 0.08
 THINKING_PREVIEW_SECONDS = 2.6
 ORPHANED_RUN_MESSAGE = "后台生成任务已中断，请点击上一条用户消息的重试按钮重新生成。"
+ACTIVE_MESSAGE_STATUSES = frozenset(
+    {
+        PlaygroundMessageStatus.PENDING.value,
+        PlaygroundMessageStatus.THINKING.value,
+        PlaygroundMessageStatus.ANSWERING.value,
+    }
+)
 
 
 class _ActiveRun:
@@ -90,7 +104,11 @@ async def _require_visible_message(
     result = await db.execute(select(PlaygroundMessage).where(*conditions))
     message = result.scalar_one_or_none()
     if message is None:
-        detail = "User message not found" if role == "user" else "Playground message not found"
+        detail = (
+            "User message not found"
+            if role == PlaygroundMessageRole.USER.value
+            else "Playground message not found"
+        )
         raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail=detail)
     return message
 
@@ -105,7 +123,7 @@ def _message_to_record(message: PlaygroundMessage, parent_public_id: str | None
         content=message.content or "",
         thinking_content=message.thinking_content or "",
         meta=list(message.meta or []),
-        markdown=message.role != "system",
+        markdown=message.role != PlaygroundMessageRole.SYSTEM.value,
         provider=message.provider,
         model=message.model,
         request_id=message.request_id,
@@ -194,11 +212,11 @@ async def _reconcile_orphaned_active_messages(
 ) -> list[PlaygroundMessage]:
     changed = False
     for item in messages:
-        if item.status not in {"pending", "thinking", "answering"}:
+        if item.status not in ACTIVE_MESSAGE_STATUSES:
             continue
         if item.public_id in _ACTIVE_RUNS:
             continue
-        item.status = "error"
+        item.status = PlaygroundMessageStatus.ERROR.value
         item.content = item.content or ORPHANED_RUN_MESSAGE
         orphan_meta = "错误: 后台任务已中断"
         if orphan_meta not in (item.meta or []):
@@ -327,9 +345,9 @@ async def create_turn(
         public_id=uuid4().hex,
         session_id=session.id,
         user_id=user_id,
-        role="user",
-        kind="message",
-        status="done",
+        role=PlaygroundMessageRole.USER.value,
+        kind=PlaygroundMessageKind.MESSAGE.value,
+        status=PlaygroundMessageStatus.DONE.value,
         title=payload.selected_preset_key,
         content=payload.input,
         meta=[payload.title],
@@ -340,9 +358,9 @@ async def create_turn(
         session_id=session.id,
         user_id=user_id,
         parent_message_id=None,
-        role="assistant",
-        kind="thinking",
-        status="pending",
+        role=PlaygroundMessageRole.ASSISTANT.value,
+        kind=PlaygroundMessageKind.THINKING.value,
+        status=PlaygroundMessageStatus.PENDING.value,
         title="AI 回应",
         content="",
         thinking_content="",
@@ -394,9 +412,9 @@ async def _create_assistant_retry_turn(
         session_id=session.id,
         user_id=user_id,
         parent_message_id=user_message.id,
-        role="assistant",
-        kind="thinking",
-        status="pending",
+        role=PlaygroundMessageRole.ASSISTANT.value,
+        kind=PlaygroundMessageKind.THINKING.value,
+        status=PlaygroundMessageStatus.PENDING.value,
         title="AI 回应",
         content="",
         thinking_content="",
@@ -436,7 +454,7 @@ async def stop_message(
     session = await _require_session(db, user_id=user_id, session_key=payload.session_key)
     message = await _require_visible_message(db, user_id=user_id, public_id=payload.message_id)
 
-    if message.status not in {"pending", "thinking", "answering"}:
+    if message.status not in ACTIVE_MESSAGE_STATUSES:
         return await _build_action_response(db, session=session)
 
     active_run = _ACTIVE_RUNS.get(message.public_id)
@@ -444,7 +462,7 @@ async def stop_message(
         active_run.stop_requested.set()
         active_run.task.cancel()
 
-    message.status = "stopped"
+    message.status = PlaygroundMessageStatus.STOPPED.value
     if "已手动停止生成" not in (message.meta or []):
         message.meta = [*(message.meta or []), "已手动停止生成"]
     await db.flush()
@@ -466,7 +484,7 @@ async def resend_turn(
         db,
         user_id=user_id,
         public_id=payload.user_message_id,
-        role="user",
+        role=PlaygroundMessageRole.USER.value,
     )
 
     later_messages = await db.execute(
@@ -478,7 +496,7 @@ async def resend_turn(
     )
     for item in later_messages.scalars().all():
         item.is_visible = False
-        if item.status in {"pending", "thinking", "answering"}:
+        if item.status in ACTIVE_MESSAGE_STATUSES:
             active_run = _ACTIVE_RUNS.get(item.public_id)
             if active_run is not None:
                 active_run.stop_requested.set()
@@ -517,7 +535,7 @@ async def edit_user_message(
         db,
         user_id=user_id,
         public_id=payload.user_message_id,
-        role="user",
+        role=PlaygroundMessageRole.USER.value,
     )
 
     user_message.content = payload.content.strip()
@@ -563,12 +581,12 @@ def _build_conversation_history(messages: Sequence[PlaygroundMessage], current_u
     for item in messages:
         if item.id >= current_user_message_id:
             break
-        if item.role == "system":
+        if item.role == PlaygroundMessageRole.SYSTEM.value:
             continue
         history.append(
             {
                 "role": item.role,
-                "kind": item.kind or "message",
+                "kind": item.kind or PlaygroundMessageKind.MESSAGE.value,
                 "title": item.title,
                 "content": item.content or "",
             }
@@ -624,13 +642,52 @@ async def _run_assistant_message(
                 thinking={"type": "enabled"},
             )
 
+        await emit_business_log(
+            logger,
+            event="ai.playground.run.start",
+            message="Playground AI run started",
+            category="ai",
+            service="ai",
+            module=__name__,
+            request_id=request_id,
+            user_id=user_id,
+            context={
+                "session_id": session_id,
+                "session_key": session_key,
+                "user_message_id": user_message_id,
+                "assistant_message_id": assistant_message_id,
+                "preset": payload.selected_preset_key,
+            },
+        )
         analysis = await provider_client.analyze(request_payload, request_id=request_id)
+        await emit_business_log(
+            logger,
+            event="ai.playground.run.success",
+            message="Playground AI run completed",
+            category="ai",
+            service="ai",
+            module=__name__,
+            request_id=request_id,
+            user_id=user_id,
+            context={
+                "session_id": session_id,
+                "session_key": session_key,
+                "provider": analysis.provider,
+                "model": analysis.model,
+                "content_block_count": len(analysis.content_blocks or []),
+                "thinking_block_count": len(analysis.thinking_blocks or []),
+            },
+        )
 
         async with async_session_factory() as db:
             assistant_message = await _mark_message_state(
                 db,
                 message_id=assistant_message_id,
-                status="thinking" if analysis.thinking_blocks else "answering",
+                status=(
+                    PlaygroundMessageStatus.THINKING.value
+                    if analysis.thinking_blocks
+                    else PlaygroundMessageStatus.ANSWERING.value
+                ),
                 title=f"{analysis.provider} / {analysis.model}",
                 provider=analysis.provider,
                 model=analysis.model,
@@ -668,7 +725,7 @@ async def _run_assistant_message(
                 await _mark_message_state(
                     db,
                     message_id=assistant_message_id,
-                    status="answering",
+                    status=PlaygroundMessageStatus.ANSWERING.value,
                     content=content[:cursor],
                 )
                 await db.commit()
@@ -679,7 +736,7 @@ async def _run_assistant_message(
             assistant_message = await _mark_message_state(
                 db,
                 message_id=assistant_message_id,
-                status="done",
+                status=PlaygroundMessageStatus.DONE.value,
                 content=content,
                 meta=[
                     f"Request ID: {request_id}",
@@ -704,23 +761,60 @@ async def _run_assistant_message(
                 await db.flush()
             await db.commit()
     except asyncio.CancelledError:
+        await emit_business_log(
+            logger,
+            event="ai.playground.run.cancelled",
+            message="Playground AI run cancelled",
+            category="ai",
+            level="warning",
+            service="ai",
+            module=__name__,
+            request_id=request_id,
+            user_id=user_id,
+            context={
+                "session_id": session_id,
+                "session_key": session_key,
+                "assistant_message_id": assistant_message_id,
+                "duration_ms": round((perf_counter() - started_at) * 1000),
+            },
+        )
         async with async_session_factory() as db:
             result = await db.execute(select(PlaygroundMessage).where(PlaygroundMessage.id == assistant_message_id))
             message = result.scalar_one_or_none()
-            if message is not None and message.status in {"pending", "thinking", "answering"}:
-                message.status = "stopped"
+            if message is not None and message.status in ACTIVE_MESSAGE_STATUSES:
+                message.status = PlaygroundMessageStatus.STOPPED.value
                 if "已手动停止生成" not in (message.meta or []):
                     message.meta = [*(message.meta or []), "已手动停止生成"]
                 await db.flush()
                 await db.commit()
         raise
     except Exception as exc:
+        await emit_business_log(
+            logger,
+            event="ai.playground.run.failed",
+            message="Playground AI run failed",
+            category="ai",
+            level="error",
+            service="ai",
+            module=__name__,
+            request_id=request_id,
+            user_id=user_id,
+            context=exception_context(
+                exc,
+                {
+                    "session_id": session_id,
+                    "session_key": session_key,
+                    "assistant_message_id": assistant_message_id,
+                    "duration_ms": round((perf_counter() - started_at) * 1000),
+                },
+            ),
+        )
         error_message = _format_run_exception(exc)
         async with async_session_factory() as db:
             result = await db.execute(select(PlaygroundMessage).where(PlaygroundMessage.id == assistant_message_id))
             message = result.scalar_one_or_none()
             if message is not None:
-                message.status = "error"
+                message.status = PlaygroundMessageStatus.ERROR.value
                 message.content = message.content or f"分析失败：{error_message}"
                 message.meta = [
                     *(message.meta or []),

backend/app/services/scheduler.py

@@ -8,11 +8,13 @@ from apscheduler.schedulers.asyncio import AsyncIOScheduler
 from apscheduler.triggers.interval import IntervalTrigger
 from sqlalchemy import select
 
+from app.core.enums import JobStatus
 from app.core.logging import get_logger
 from app.db.session import async_session_factory
 from app.core.time import to_iso8601_utc
 from app.models.datasource import DataSource
 from app.models.task import CollectionTask
+from app.services.business_logs import emit_business_log, emit_business_log_background, exception_context
 from app.services.collectors.registry import collector_registry
 from app.services.datasource_connectivity import (
     build_builtin_connectivity_checksum,
@@ -124,13 +126,22 @@ async def run_collector_task(collector_name: str):
                 event="collector.run.skipped_disabled",
                 context={"collector_name": collector_name},
             )
+            await emit_business_log(
+                logger,
+                event="collector.run.skipped_disabled",
+                message="Skipping disabled collector",
+                category="collector",
+                service="scheduler",
+                module=__name__,
+                context={"collector_name": collector_name, "datasource_id": datasource.id, "status": "skipped"},
+            )
             return
 
         running_result = await db.execute(
             select(CollectionTask)
             .where(
                 CollectionTask.datasource_id == datasource.id,
-                CollectionTask.status == "running",
+                CollectionTask.status == JobStatus.RUNNING.value,
             )
             .order_by(CollectionTask.started_at.desc(), CollectionTask.id.desc())
             .limit(1)
@@ -152,6 +163,21 @@ async def run_collector_task(collector_name: str):
                     event="collector.run.skipped_already_running",
                     context={"collector_name": collector_name, "task_id": existing_running.id},
                 )
+                await emit_business_log(
+                    logger,
+                    event="collector.run.skipped_already_running",
+                    message="Skipping collector trigger because task is already running",
+                    category="collector",
+                    level="warning",
+                    service="scheduler",
+                    module=__name__,
+                    context={
+                        "collector_name": collector_name,
+                        "datasource_id": datasource.id,
+                        "task_id": existing_running.id,
+                        "status": "skipped",
+                    },
+                )
                 return
 
             existing_error = (existing_running.error_message or "").strip()
@@ -159,7 +185,7 @@ async def run_collector_task(collector_name: str):
                 f"Marked failed automatically after stale running timeout "
                 f"({RUNNING_TASK_GUARD_TIMEOUT_MINUTES}m) in scheduler guard"
             )
-            existing_running.status = "failed"
+            existing_running.status = JobStatus.FAILED.value
             existing_running.phase = "failed"
             existing_running.completed_at = now
             existing_running.error_message = (
@@ -173,6 +199,21 @@ async def run_collector_task(collector_name: str):
                 event="collector.run.stale_task_failed",
                 context={"collector_name": collector_name, "task_id": existing_running.id},
             )
+            await emit_business_log(
+                logger,
+                event="collector.run.stale_task_failed",
+                message="Marked stale running task as failed before rerun",
+                category="collector",
+                level="warning",
+                service="scheduler",
+                module=__name__,
+                context={
+                    "collector_name": collector_name,
+                    "datasource_id": datasource.id,
+                    "task_id": existing_running.id,
+                    "status": "failed",
+                },
+            )
 
         try:
             datasource_id = datasource.id
@@ -183,6 +224,15 @@ async def run_collector_task(collector_name: str):
                 event="collector.run.started",
                 context={"collector_name": collector_name, "datasource_id": datasource_id},
             )
+            await emit_business_log(
+                logger,
+                event="collector.run.scheduled_started",
+                message="Scheduler started collector run",
+                category="collector",
+                service="scheduler",
+                module=__name__,
+                context={"collector_name": collector_name, "datasource_id": datasource_id, "status": "running"},
+            )
             task_result = await collector.run(db)
             datasource = await db.get(DataSource, datasource_id)
             if datasource is None:
@@ -194,7 +244,7 @@ async def run_collector_task(collector_name: str):
                 return
             datasource.last_run_at = datetime.now(UTC)
             datasource.last_status = task_result.get("status")
-            if datasource.last_status == "success":
+            if datasource.last_status == JobStatus.SUCCESS.value:
                 effective_candidate = await get_builtin_effective_candidate(db, datasource_source)
                 checksum, _credential_context = await build_builtin_connectivity_checksum(
                     datasource_source,
@@ -217,29 +267,66 @@ async def run_collector_task(collector_name: str):
                 event="collector.run.completed",
                 context={"collector_name": collector_name, "datasource_id": datasource_id, "result": task_result},
             )
+            await emit_business_log(
+                logger,
+                event="collector.run.scheduled_completed",
+                message="Scheduler completed collector run",
+                category="collector",
+                service="scheduler",
+                module=__name__,
+                context={
+                    "collector_name": collector_name,
+                    "datasource_id": datasource_id,
+                    "status": task_result.get("status"),
+                    "result": task_result,
+                },
+            )
         except asyncio.CancelledError:
             await db.rollback()
             datasource = await db.get(DataSource, datasource_id)
             datasource.last_run_at = datetime.now(UTC)
-            datasource.last_status = "cancelled"
+            datasource.last_status = JobStatus.CANCELLED.value
             await db.commit()
             logger.warning_event(
                 "Collector cancelled by operator",
                 event="collector.run.cancelled",
                 context={"collector_name": collector_name, "datasource_id": datasource.id},
             )
+            await emit_business_log(
+                logger,
+                event="collector.run.cancelled",
+                message="Collector cancelled by operator",
+                category="collector",
+                level="warning",
+                service="scheduler",
+                module=__name__,
+                context={"collector_name": collector_name, "datasource_id": datasource.id, "status": "cancelled"},
+            )
             raise
         except Exception as exc:
             await db.rollback()
             datasource = await db.get(DataSource, datasource_id)
             datasource.last_run_at = datetime.now(UTC)
-            datasource.last_status = "failed"
+            datasource.last_status = JobStatus.FAILED.value
             await db.commit()
             logger.exception_event(
                 "Collector failed",
                 event="collector.run.failed",
                 context={"collector_name": collector_name, "datasource_id": datasource.id, "error": str(exc)},
             )
+            await emit_business_log(
+                logger,
+                event="collector.run.failed",
+                message="Collector failed",
+                category="collector",
+                level="error",
+                service="scheduler",
+                module=__name__,
+                context=exception_context(
+                    exc,
+                    {"collector_name": collector_name, "datasource_id": datasource.id, "status": "failed"},
+                ),
+            )
 
 
 async def cleanup_stale_running_tasks(max_age_hours: int = 2) -> int:
@@ -249,7 +336,7 @@ async def cleanup_stale_running_tasks(max_age_hours: int = 2) -> int:
     async with async_session_factory() as db:
         result = await db.execute(
             select(CollectionTask).where(
-                CollectionTask.status == "running",
+                CollectionTask.status == JobStatus.RUNNING.value,
                 CollectionTask.started_at.is_not(None),
                 CollectionTask.started_at < cutoff,
             )
@@ -257,7 +344,7 @@ async def cleanup_stale_running_tasks(max_age_hours: int = 2) -> int:
         stale_tasks = result.scalars().all()
 
         for task in stale_tasks:
-            task.status = "failed"
+            task.status = JobStatus.FAILED.value
             task.phase = "failed"
             task.completed_at = datetime.now(UTC)
             existing_error = (task.error_message or "").strip()
@@ -361,6 +448,16 @@ def run_collector_now(collector_name: str) -> bool:
             event="collector.trigger.skipped_already_running",
             context={"collector_name": collector_name},
         )
+        emit_business_log_background(
+            logger,
+            event="collector.trigger.skipped_already_running",
+            message="Collector is already running in-memory; skipping duplicate trigger",
+            category="collector",
+            level="warning",
+            service="scheduler",
+            module=__name__,
+            context={"collector_name": collector_name, "status": "skipped"},
+        )
         return False
 
     try:
@@ -378,6 +475,15 @@ def run_collector_now(collector_name: str) -> bool:
             event="collector.trigger.started",
             context={"collector_name": collector_name},
         )
+        emit_business_log_background(
+            logger,
+            event="collector.trigger.started",
+            message="Triggered collector",
+            category="collector",
+            service="scheduler",
+            module=__name__,
+            context={"collector_name": collector_name, "status": "queued"},
+        )
         return True
     except Exception as exc:
         logger.error_event(
@@ -385,6 +491,16 @@ def run_collector_now(collector_name: str) -> bool:
             event="collector.trigger.failed",
             context={"collector_name": collector_name, "error": str(exc)},
         )
+        emit_business_log_background(
+            logger,
+            event="collector.trigger.failed",
+            message="Failed to trigger collector",
+            category="collector",
+            level="error",
+            service="scheduler",
+            module=__name__,
+            context=exception_context(exc, {"collector_name": collector_name, "status": "failed"}),
+        )
         return False
 
 

backend/app/services/situational_alert_ai_brief.py

@@ -6,6 +6,7 @@ from typing import Any
 from sqlalchemy import func, select
 from sqlalchemy.ext.asyncio import AsyncSession
 
+from app.core.enums import BGPStatus
 from app.models.alert import Alert, AlertSeverity, AlertStatus
 from app.models.bgp_anomaly import BGPAnomaly
 from app.models.bgp_incident import BGPIncident
@@ -49,11 +50,11 @@ async def build_situational_alert_brief_request(
 
     total_incidents_result = await db.execute(select(func.count(BGPIncident.id)))
     active_incidents_result = await db.execute(
-        select(func.count(BGPIncident.id)).where(BGPIncident.status == "active")
+        select(func.count(BGPIncident.id)).where(BGPIncident.status == BGPStatus.ACTIVE.value)
     )
     bgp_severity_result = await db.execute(
         select(BGPIncident.severity, func.count(BGPIncident.id))
-        .where(BGPIncident.status == "active")
+        .where(BGPIncident.status == BGPStatus.ACTIVE.value)
         .group_by(BGPIncident.severity)
     )
     bgp_region_counter: Counter[str] = Counter()
@@ -65,11 +66,11 @@ async def build_situational_alert_brief_request(
 
     total_anomalies_result = await db.execute(select(func.count(BGPAnomaly.id)))
     active_anomalies_result = await db.execute(
-        select(func.count(BGPAnomaly.id)).where(BGPAnomaly.status == "active")
+        select(func.count(BGPAnomaly.id)).where(BGPAnomaly.status == BGPStatus.ACTIVE.value)
     )
     anomaly_type_result = await db.execute(
         select(BGPAnomaly.anomaly_type, func.count(BGPAnomaly.id))
-        .where(BGPAnomaly.status == "active")
+        .where(BGPAnomaly.status == BGPStatus.ACTIVE.value)
         .group_by(BGPAnomaly.anomaly_type)
         .order_by(func.count(BGPAnomaly.id).desc())
         .limit(6)

backend/app/services/system_control.py

@@ -7,6 +7,7 @@ from pathlib import Path
 from typing import Any
 
 from app.core.config import ROOT_DIR
+from app.core.enums import UserRole
 from app.core.security import redis_client
 
 SYSTEM_TASK_TTL_SECONDS = 24 * 60 * 60
@@ -47,7 +48,7 @@ def normalize_user_role(role: Any) -> str:
 
 
 def require_super_admin(user_role: Any) -> bool:
-    return normalize_user_role(user_role) == "super_admin"
+    return normalize_user_role(user_role) == UserRole.SUPER_ADMIN.value
 
 
 def build_task_id(prefix: str = "restart") -> str:

backend/app/services/system_logs.py

@@ -5,6 +5,7 @@ import os
 import re
 import shutil
 import subprocess
+import hashlib
 
 from collections import Counter, deque
 from dataclasses import dataclass
@@ -12,7 +13,11 @@ from datetime import UTC, datetime
 from pathlib import Path
 from typing import Any
 
+from app.core.enums import LogLevel
 from app.core.security import redis_client
+from app.models.system_log import AuditLog, ObservabilityEvent, ObservabilityEventGroup, SystemLog
+from sqlalchemy import select
+from sqlalchemy.ext.asyncio import AsyncSession
 
 DEFAULT_LOG_LINE_LIMIT = 200
 MAX_LOG_LINE_LIMIT = 1000
@@ -20,11 +25,11 @@ BUFFER_LOG_LIMIT = 1000
 BUFFER_LOG_TTL_SECONDS = 7 * 24 * 60 * 60
 LOG_BUFFER_KEY_PREFIX = "planet:system_logs"
 
-LOG_LEVEL_ERROR = "error"
-LOG_LEVEL_WARNING = "warning"
-LOG_LEVEL_INFO = "info"
-LOG_LEVEL_DEBUG = "debug"
-LOG_LEVEL_ALL = "all"
+LOG_LEVEL_ERROR = LogLevel.ERROR.value
+LOG_LEVEL_WARNING = LogLevel.WARNING.value
+LOG_LEVEL_INFO = LogLevel.INFO.value
+LOG_LEVEL_DEBUG = LogLevel.DEBUG.value
+LOG_LEVEL_ALL = LogLevel.ALL.value
 
 SUPPORTED_LOG_LEVELS = {
     LOG_LEVEL_ALL,
@@ -99,6 +104,16 @@ class StructuredLogEntry:
     search_text: str
 
 
+@dataclass(frozen=True)
+class LogEvent:
+    source_id: str
+    cursor: str
+    timestamp: datetime | None
+    level: str | None
+    line: str
+    search_text: str
+
+
 @dataclass
 class DailyLogMarker:
     date_token: str
@@ -106,6 +121,10 @@ class DailyLogMarker:
     dominant_level: str
 
 
+def _normalize_search_query(search: str | None) -> str:
+    return (search or "").strip().lower()
+
+
 def _planet_state_dir() -> Path:
     configured = os.getenv("PLANET_STATE_DIR")
     if configured:
@@ -135,7 +154,7 @@ LOG_SOURCES: dict[str, LogSource] = {
         name="前端开发服务",
         kind="file",
         location=_state_log_path("frontend.log"),
-        description="控制台与 Earth 前端开发服务输出。",
+        description="控制台与智能星球前端开发服务输出。",
         category="service",
         fallback_locations=("/tmp/planet_frontend.log",),
     ),
@@ -150,13 +169,22 @@ LOG_SOURCES: dict[str, LogSource] = {
     ),
     "earth-client": LogSource(
         source_id="earth-client",
-        name="Earth 浏览器端",
+        name="智能星球浏览器端",
         kind="buffer",
         location="redis://planet:system_logs:earth-client",
-        description="Earth 浏览器端上报的运行时错误与关键业务日志。",
+        description="智能星球浏览器端上报的运行时错误与关键业务日志。",
         category="client",
         buffer_key=f"{LOG_BUFFER_KEY_PREFIX}:earth-client",
     ),
+    "admin-client": LogSource(
+        source_id="admin-client",
+        name="控制台浏览器端",
+        kind="buffer",
+        location="redis://planet:system_logs:admin-client",
+        description="控制台浏览器端上报的运行时错误。",
+        category="client",
+        buffer_key=f"{LOG_BUFFER_KEY_PREFIX}:admin-client",
+    ),
 }
 
 
@@ -365,6 +393,59 @@ def build_buffer_entry(payload: dict[str, Any]) -> StructuredLogEntry:
     )
 
 
+def compact_log_context(context: dict | None) -> str:
+    if not context:
+        return ""
+    allowed = {
+        key: value
+        for key, value in (context or {}).items()
+        if key
+        in {
+            "status",
+            "duration_ms",
+            "provider",
+            "model",
+            "result_provider",
+            "result_model",
+            "collector_name",
+            "datasource_id",
+            "task_id",
+            "snapshot_id",
+            "raw_count",
+            "transformed_count",
+            "saved_count",
+            "created",
+            "updated",
+            "unchanged",
+            "deleted",
+            "result_count",
+            "status_code",
+            "error_type",
+            "error",
+            "route",
+            "module",
+        }
+    }
+    if not allowed:
+        return ""
+    return json.dumps(allowed, ensure_ascii=False, sort_keys=True)
+
+
+def context_search_aliases(context: dict | None) -> str:
+    if not context:
+        return ""
+    aliases: list[str] = []
+    for key, value in sorted((context or {}).items()):
+        if value is None or isinstance(value, (dict, list, tuple, set)):
+            continue
+        normalized_key = str(key).strip()
+        normalized_value = str(value).strip()
+        if not normalized_key or not normalized_value:
+            continue
+        aliases.append(f"{normalized_key}={normalized_value}")
+    return " ".join(aliases)
+
+
 def read_file_entries(source: LogSource, scan_limit: int) -> list[StructuredLogEntry]:
     path = resolve_file_log_path(source)
     if not path.exists():
@@ -437,6 +518,400 @@ def read_source_entries(source: LogSource, scan_limit: int) -> list[StructuredLo
     return []
 
 
+def _database_event_from_system_record(record: SystemLog) -> LogEvent:
+    record_level = normalize_log_level(record.level)
+    line = " ".join(
+        part
+        for part in [
+            record.occurred_at.isoformat() if record.occurred_at else "",
+            record_level.upper(),
+            record.source,
+            record.category or "",
+            record.event or "",
+            f"request_id={record.request_id}" if record.request_id else "",
+            record.message,
+            compact_log_context(record.context),
+        ]
+        if part
+    )
+    search_text = " ".join(
+        [
+            line,
+            f"id={record.id}",
+            f"user_id={record.user_id}" if record.user_id else "",
+            context_search_aliases(record.context),
+            json.dumps(record.context or {}, ensure_ascii=False, sort_keys=True),
+        ]
+    ).lower()
+    return LogEvent(
+        source_id="system-db",
+        cursor=f"system-db:{record.id}",
+        timestamp=record.occurred_at,
+        level=None if record_level == LOG_LEVEL_ALL else record_level,
+        line=line,
+        search_text=search_text,
+    )
+
+
+def _database_event_from_audit_record(record: AuditLog) -> LogEvent:
+    line = " ".join(
+        part
+        for part in [
+            record.occurred_at.isoformat() if record.occurred_at else "",
+            "INFO",
+            record.action,
+            record.target_type or "",
+            record.target_id or "",
+            record.result or "",
+            f"request_id={record.request_id}" if record.request_id else "",
+        ]
+        if part
+    )
+    search_text = " ".join(
+        [
+            line,
+            f"id={record.id}",
+            f"actor_id={record.actor_id}" if record.actor_id else "",
+            record.actor_name or "",
+            context_search_aliases(record.details),
+            json.dumps(record.details or {}, ensure_ascii=False, sort_keys=True),
+        ]
+    ).lower()
+    return LogEvent(
+        source_id="audit-db",
+        cursor=f"audit-db:{record.id}",
+        timestamp=record.occurred_at,
+        level=LOG_LEVEL_INFO,
+        line=line,
+        search_text=search_text,
+    )
+
+
+async def read_database_log_events(
+    source_id: str,
+    *,
+    scan_limit: int,
+    level: str = LOG_LEVEL_ALL,
+    levels: str | None = None,
+    start_date: str | None = None,
+    end_date: str | None = None,
+    search: str | None = None,
+    db: AsyncSession,
+) -> list[LogEvent] | None:
+    selected_levels = normalize_log_levels(level, levels)
+    search_query = (search or "").strip()
+    if source_id == "system-db":
+        query = select(SystemLog).order_by(SystemLog.occurred_at.desc().nullslast(), SystemLog.id.desc()).limit(scan_limit)
+        result = await db.execute(query)
+        events = [_database_event_from_system_record(record) for record in result.scalars().all()]
+    elif source_id == "audit-db":
+        query = select(AuditLog).order_by(AuditLog.occurred_at.desc().nullslast(), AuditLog.id.desc()).limit(scan_limit)
+        result = await db.execute(query)
+        events = [_database_event_from_audit_record(record) for record in result.scalars().all()]
+    else:
+        return None
+
+    events = list(reversed(events))
+    return [
+        event
+        for event in events
+        if event_matches_levels(event, selected_levels)
+        and event_matches_search(event, search_query)
+        and event_matches_date_range(event, start_date, end_date)
+    ]
+
+
+async def read_database_log_snapshot(
+    source_id: str,
+    *,
+    limit: int,
+    level: str,
+    levels: str | None,
+    start_date: str | None,
+    end_date: str | None,
+    search: str | None,
+    db: AsyncSession,
+) -> dict[str, Any] | None:
+    events = await read_database_log_events(
+        source_id,
+        scan_limit=limit * 5,
+        level=level,
+        levels=levels,
+        start_date=start_date,
+        end_date=end_date,
+        search=search,
+        db=db,
+    )
+    if events is None:
+        return None
+    visible_events = events[-limit:]
+    selected_levels = normalize_log_levels(level, levels)
+    return {
+        "source_id": source_id,
+        "name": "系统事件" if source_id == "system-db" else "审计事件",
+        "kind": "database",
+        "location": "table://system_logs" if source_id == "system-db" else "table://audit_logs",
+        "description": "数据库持久化日志",
+        "category": "database" if source_id == "system-db" else "audit",
+        "status": "ok" if visible_events else "empty",
+        "level": level,
+        "selected_levels": list(selected_levels),
+        "search_query": search or "",
+        "available_levels": ["all", "error", "warning", "info", "debug"],
+        "daily_markers": build_daily_log_markers_from_events(events),
+        "line_limit": limit,
+        "line_count": len(visible_events),
+        "lines": [event.line for event in visible_events],
+    }
+
+
+def _observability_group_matches(
+    group: ObservabilityEventGroup,
+    *,
+    selected_levels: tuple[str, ...],
+    start_date: str | None,
+    end_date: str | None,
+    search: str | None,
+) -> bool:
+    if selected_levels and group.last_level not in selected_levels:
+        return False
+    if start_date or end_date:
+        if group.last_seen_at is None:
+            return False
+        date_token = group.last_seen_at.astimezone(UTC).date().isoformat()
+        if start_date and date_token < start_date:
+            return False
+        if end_date and date_token > end_date:
+            return False
+    query = _normalize_search_query(search)
+    if not query:
+        return True
+    haystack = " ".join(
+        [
+            group.fingerprint or "",
+            group.source or "",
+            group.service or "",
+            group.module or "",
+            group.category or "",
+            group.event or "",
+            group.last_level or "",
+            group.sample_message or "",
+            group.sample_detail or "",
+            json.dumps(group.affected_sources or [], ensure_ascii=False, sort_keys=True),
+        ]
+    ).lower()
+    return query in haystack
+
+
+def _serialize_observability_group(group: ObservabilityEventGroup) -> dict[str, Any]:
+    return {
+        "fingerprint": group.fingerprint,
+        "source": group.source,
+        "service": group.service,
+        "module": group.module,
+        "category": group.category,
+        "event": group.event,
+        "level": group.last_level,
+        "message": group.sample_message,
+        "detail": group.sample_detail,
+        "affected_sources": group.affected_sources or [],
+        "count": group.count or 0,
+        "first_seen_at": group.first_seen_at.isoformat() if group.first_seen_at else None,
+        "last_seen_at": group.last_seen_at.isoformat() if group.last_seen_at else None,
+    }
+
+
+def _serialize_observability_event(record: ObservabilityEvent) -> dict[str, Any]:
+    return {
+        "id": record.id,
+        "source": record.source,
+        "service": record.service,
+        "module": record.module,
+        "category": record.category,
+        "event": record.event,
+        "level": record.level,
+        "message": record.message,
+        "fingerprint": record.fingerprint,
+        "occurred_at": record.occurred_at.isoformat() if record.occurred_at else None,
+        "request_id": record.request_id,
+        "trace_id": record.trace_id,
+        "task_id": record.task_id,
+        "source_id": record.source_ref_id,
+        "provider": record.provider,
+        "user_id": record.user_id,
+        "context": record.context or {},
+        "occurrence_count": record.occurrence_count or 1,
+    }
+
+
+async def read_observability_groups(
+    *,
+    limit: int,
+    level: str = LOG_LEVEL_ALL,
+    levels: str | None = None,
+    start_date: str | None = None,
+    end_date: str | None = None,
+    search: str | None = None,
+    db: AsyncSession,
+) -> dict[str, Any]:
+    selected_levels = normalize_log_levels(level, levels)
+    scan_limit = max(limit * 5, limit, DEFAULT_LOG_LINE_LIMIT)
+    result = await db.execute(
+        select(ObservabilityEventGroup)
+        .order_by(ObservabilityEventGroup.last_seen_at.desc().nullslast())
+        .limit(scan_limit)
+    )
+    groups = [
+        group
+        for group in result.scalars().all()
+        if _observability_group_matches(
+            group,
+            selected_levels=selected_levels,
+            start_date=start_date,
+            end_date=end_date,
+            search=search,
+        )
+    ][:limit]
+    return {
+        "mode": "grouped",
+        "line_limit": limit,
+        "line_count": len(groups),
+        "groups": [_serialize_observability_group(group) for group in groups],
+        "filters": {
+            "level": level,
+            "levels": list(selected_levels),
+            "start_date": start_date,
+            "end_date": end_date,
+            "search": search or "",
+        },
+    }
+
+
+async def read_observability_group_events(
+    fingerprint: str,
+    *,
+    limit: int,
+    db: AsyncSession,
+) -> dict[str, Any] | None:
+    group = await db.get(ObservabilityEventGroup, fingerprint)
+    if group is None:
+        return None
+    result = await db.execute(
+        select(ObservabilityEvent)
+        .where(ObservabilityEvent.fingerprint == fingerprint)
+        .order_by(ObservabilityEvent.occurred_at.desc().nullslast(), ObservabilityEvent.id.desc())
+        .limit(limit)
+    )
+    events = list(reversed(result.scalars().all()))
+    return {
+        "fingerprint": fingerprint,
+        "group": _serialize_observability_group(group),
+        "line_limit": limit,
+        "line_count": len(events),
+        "events": [_serialize_observability_event(record) for record in events],
+    }
+
+
+async def read_observability_raw_events(
+    *,
+    limit: int,
+    level: str = LOG_LEVEL_ALL,
+    levels: str | None = None,
+    start_date: str | None = None,
+    end_date: str | None = None,
+    search: str | None = None,
+    db: AsyncSession,
+) -> dict[str, Any]:
+    selected_levels = normalize_log_levels(level, levels)
+    query = select(ObservabilityEvent).order_by(ObservabilityEvent.occurred_at.desc().nullslast(), ObservabilityEvent.id.desc())
+    if selected_levels:
+        query = query.where(ObservabilityEvent.level.in_(selected_levels))
+    result = await db.execute(query.limit(max(limit * 5, limit)))
+    records = result.scalars().all()
+    search_query = _normalize_search_query(search)
+    visible: list[ObservabilityEvent] = []
+    for record in records:
+        if start_date or end_date:
+            if record.occurred_at is None:
+                continue
+            date_token = record.occurred_at.astimezone(UTC).date().isoformat()
+            if start_date and date_token < start_date:
+                continue
+            if end_date and date_token > end_date:
+                continue
+        if search_query:
+            haystack = " ".join(
+                [
+                    record.source or "",
+                    record.service or "",
+                    record.module or "",
+                    record.category or "",
+                    record.event or "",
+                    record.message or "",
+                    record.fingerprint or "",
+                    record.request_id or "",
+                    record.trace_id or "",
+                    record.task_id or "",
+                    record.source_ref_id or "",
+                    record.provider or "",
+                    json.dumps(record.context or {}, ensure_ascii=False, sort_keys=True),
+                ]
+            ).lower()
+            if search_query not in haystack:
+                continue
+        visible.append(record)
+        if len(visible) >= limit:
+            break
+    visible = list(reversed(visible))
+    return {
+        "mode": "raw",
+        "line_limit": limit,
+        "line_count": len(visible),
+        "events": [_serialize_observability_event(record) for record in visible],
+        "lines": [
+            " ".join(
+                part
+                for part in [
+                    record.occurred_at.isoformat() if record.occurred_at else "",
+                    record.level.upper(),
+                    record.source,
+                    record.category or "",
+                    record.event or "",
+                    f"fingerprint={record.fingerprint}",
+                    record.message,
+                ]
+                if part
+            )
+            for record in visible
+        ],
+    }
+
+
+def _stable_hash(value: str) -> str:
+    return hashlib.sha1(value.encode("utf-8", errors="replace")).hexdigest()[:16]
+
+
+def build_log_events(source_id: str, entries: list[StructuredLogEntry]) -> list[LogEvent]:
+    events: list[LogEvent] = []
+    seen: dict[str, int] = {}
+    for entry in entries:
+        stable_value = entry.raw_line or entry.display_line
+        digest = _stable_hash(stable_value)
+        occurrence = seen.get(digest, 0) + 1
+        seen[digest] = occurrence
+        events.append(
+            LogEvent(
+                source_id=source_id,
+                cursor=f"{source_id}:{digest}:{occurrence}",
+                timestamp=entry.timestamp,
+                level=entry.level,
+                line=entry.display_line,
+                search_text=entry.search_text,
+            )
+        )
+    return events
+
+
 def matches_levels(entry: StructuredLogEntry, selected_levels: tuple[str, ...]) -> bool:
     if not selected_levels:
         return True
@@ -469,6 +944,34 @@ def matches_search(entry: StructuredLogEntry, search: str | None) -> bool:
     return query in entry.search_text
 
 
+def event_matches_levels(event: LogEvent, selected_levels: tuple[str, ...]) -> bool:
+    if not selected_levels:
+        return True
+    return event.level in selected_levels
+
+
+def event_matches_date_range(event: LogEvent, start_date: str | None, end_date: str | None) -> bool:
+    if not start_date and not end_date:
+        return True
+    if event.timestamp is None:
+        return False
+    date_token = event.timestamp.astimezone(UTC).date().isoformat()
+    if start_date and date_token < start_date:
+        return False
+    if end_date and date_token > end_date:
+        return False
+    return True
+
+
+def event_matches_search(event: LogEvent, search: str | None) -> bool:
+    if search is None:
+        return True
+    query = search.strip().lower()
+    if not query:
+        return True
+    return query in event.search_text
+
+
 def build_daily_log_markers(entries: list[StructuredLogEntry]) -> list[dict[str, Any]]:
     grouped: dict[str, list[StructuredLogEntry]] = {}
     for entry in entries:
@@ -503,6 +1006,69 @@ def build_daily_log_markers(entries: list[StructuredLogEntry]) -> list[dict[str,
     return [marker.__dict__ for marker in markers]
 
 
+def build_daily_log_markers_from_events(events: list[LogEvent]) -> list[dict[str, Any]]:
+    grouped: dict[str, list[LogEvent]] = {}
+    for event in events:
+        if event.timestamp is None:
+            continue
+        date_token = event.timestamp.astimezone(UTC).date().isoformat()
+        grouped.setdefault(date_token, []).append(event)
+
+    markers: list[DailyLogMarker] = []
+    for date_token, group in sorted(grouped.items()):
+        level_counts = Counter(
+            event.level
+            for event in group
+            if event.level in SUPPORTED_LOG_LEVELS and event.level != LOG_LEVEL_ALL
+        )
+        dominant_level = LOG_LEVEL_INFO
+        if level_counts:
+            dominant_level = sorted(
+                level_counts.items(),
+                key=lambda item: (
+                    -item[1],
+                    ("error", "warning", "info", "debug").index(item[0]),
+                ),
+            )[0][0]
+        markers.append(
+            DailyLogMarker(
+                date_token=date_token,
+                total=len(group),
+                dominant_level=dominant_level,
+            )
+        )
+    return [marker.__dict__ for marker in markers]
+
+
+def read_log_events(
+    source_id: str,
+    scan_limit: int,
+    *,
+    level: str = LOG_LEVEL_ALL,
+    levels: str | None = None,
+    start_date: str | None = None,
+    end_date: str | None = None,
+    search: str | None = None,
+) -> list[LogEvent] | None:
+    source = LOG_SOURCES.get(source_id)
+    if source is None:
+        return None
+
+    selected_levels = normalize_log_levels(level, levels)
+    search_query = (search or "").strip()
+    events = build_log_events(source_id, read_source_entries(source, scan_limit))
+    marker_events = [
+        event
+        for event in events
+        if event_matches_levels(event, selected_levels) and event_matches_search(event, search_query)
+    ]
+    return [
+        event
+        for event in marker_events
+        if event_matches_date_range(event, start_date, end_date)
+    ]
+
+
 def read_log_snapshot(
     source_id: str,
     limit: int,
@@ -520,18 +1086,18 @@ def read_log_snapshot(
     selected_levels = normalize_log_levels(level, levels)
     search_query = (search or "").strip()
     scan_limit = max(min(MAX_LOG_LINE_LIMIT * 5, 5000), limit * 5, BUFFER_LOG_LIMIT if source.kind == "buffer" else 1000)
-    all_entries = read_source_entries(source, scan_limit)
-    marker_entries = [
-        entry
-        for entry in all_entries
-        if matches_levels(entry, selected_levels) and matches_search(entry, search_query)
+    all_events = build_log_events(source_id, read_source_entries(source, scan_limit))
+    marker_events = [
+        event
+        for event in all_events
+        if event_matches_levels(event, selected_levels) and event_matches_search(event, search_query)
     ]
-    filtered_entries = [
-        entry
-        for entry in marker_entries
-        if matches_date_range(entry, start_date, end_date)
+    filtered_events = [
+        event
+        for event in marker_events
+        if event_matches_date_range(event, start_date, end_date)
     ]
-    visible_entries = filtered_entries[-limit:]
+    visible_events = filtered_events[-limit:]
 
     compatibility_level = selected_levels[0] if len(selected_levels) == 1 else LOG_LEVEL_ALL
     return {
@@ -552,8 +1118,8 @@ def read_log_snapshot(
             LOG_LEVEL_INFO,
             LOG_LEVEL_DEBUG,
         ],
-        "daily_markers": build_daily_log_markers(marker_entries),
+        "daily_markers": build_daily_log_markers_from_events(marker_events),
         "line_limit": limit,
-        "line_count": len(visible_entries),
-        "lines": [entry.display_line for entry in visible_entries],
+        "line_count": len(visible_events),
+        "lines": [event.line for event in visible_events],
     }

backend/app/services/vessel_ais_aggregation.py

@@ -9,7 +9,12 @@ from sqlalchemy import select
 from sqlalchemy import Float
 from sqlalchemy.ext.asyncio import AsyncSession
 
-from app.models.vessel import AISConflictRecord, AISRawObservation, AISSourceHealth
+from app.models.vessel import (
+    AISConflictRecord,
+    AISRawObservation,
+    AISSourceHealth,
+    VesselCurrentState,
+)
 from app.services.vessel_aggregation_strategy import (
     DEFAULT_STRATEGY,
     load_strategy,
@@ -44,6 +49,17 @@ CONFLICT_FIELDS = (
     "width",
     "draught",
 )
+CURRENT_STATE_STATIC_FIELDS = (
+    "name",
+    "callsign",
+    "vessel_type",
+    "vessel_type_name",
+    "flag",
+    "length",
+    "width",
+    "draught",
+    "imo",
+)
 
 
 def _json_default(value: Any) -> Any:
@@ -489,9 +505,115 @@ async def record_vessel_ais_observation(
         quality_flags=quality_flags or [],
     )
     db.add(observation)
+    await upsert_vessel_current_state(
+        db,
+        source=source,
+        normalized_payload=normalized_json,
+        observed_at=observed_at,
+        quality_flags=quality_flags or [],
+    )
     return observation
 
 
+async def upsert_vessel_current_state(
+    db: AsyncSession,
+    *,
+    source: str,
+    normalized_payload: dict[str, Any],
+    observed_at: datetime,
+    quality_flags: list[str] | None = None,
+) -> VesselCurrentState | None:
+    """Keep one latest renderable row per MMSI while preserving useful static fields."""
+
+    if not _has_valid_position(normalized_payload):
+        return None
+    mmsi = int(normalized_payload["mmsi"])
+    current = await db.get(VesselCurrentState, mmsi)
+    if current is not None and current.observed_at is not None:
+        current_observed_at = _coerce_datetime(current.observed_at)
+        if current_observed_at is not None and observed_at < current_observed_at:
+            return current
+
+    if current is None:
+        current = VesselCurrentState(mmsi=mmsi)
+        db.add(current)
+
+    current.lat = float(normalized_payload["lat"])
+    current.lon = float(normalized_payload["lon"])
+    current.source = source
+    current.observed_at = observed_at
+    current.updated_at = datetime.now(UTC)
+    updated_fields: set[str] = {"lat", "lon"}
+    for field in DYNAMIC_FIELDS:
+        if field in {"lat", "lon"}:
+            continue
+        value = _payload_value(normalized_payload, field)
+        if value is not None:
+            setattr(current, field, value)
+            updated_fields.add(field)
+    field_sources = dict(current.field_sources or {})
+    for field in CURRENT_STATE_STATIC_FIELDS:
+        value = _payload_value(normalized_payload, field)
+        if value is None:
+            continue
+        existing_source = field_sources.get(field)
+        existing_value = getattr(current, field, None)
+        if (
+            existing_value in (None, "")
+            or _strategy_source_rank(source, DEFAULT_STRATEGY)
+            >= _strategy_source_rank(str(existing_source or ""), DEFAULT_STRATEGY)
+        ):
+            setattr(current, field, value)
+            updated_fields.add(field)
+
+    current.vessel_type_name = current.vessel_type_name or normalize_vessel_type_name(
+        current.vessel_type
+    )
+    selected_reasons = dict(current.selected_reasons or {})
+    for field in updated_fields:
+        field_sources[field] = source
+        selected_reasons[field] = (
+            "newest_observation" if field in DYNAMIC_FIELDS else "source_priority"
+        )
+    current.field_sources = field_sources
+    current.selected_reasons = selected_reasons
+    current.source_summary = {
+        **dict(current.source_summary or {}),
+        source: {
+            "latest_observed_at": observed_at.isoformat(),
+        },
+    }
+    current.quality_flags = sorted(set((current.quality_flags or []) + (quality_flags or [])))
+    return current
+
+
+async def get_current_vessels_snapshot(
+    db: AsyncSession,
+    *,
+    bbox: tuple[float, float, float, float],
+    limit: int = 1000,
+    observed_since: datetime,
+) -> list[dict[str, Any]]:
+    """Read the bounded latest-state table used by Earth rendering."""
+
+    safe_limit = min(max(int(limit or 1000), 1), MAX_SNAPSHOT_LIMIT)
+    lon_min, lat_min, lon_max, lat_max = bbox
+    stmt = (
+        select(VesselCurrentState)
+        .where(VesselCurrentState.observed_at >= observed_since)
+        .where(VesselCurrentState.lon >= lon_min)
+        .where(VesselCurrentState.lon <= lon_max)
+        .where(VesselCurrentState.lat >= lat_min)
+        .where(VesselCurrentState.lat <= lat_max)
+        .order_by(VesselCurrentState.observed_at.desc(), VesselCurrentState.mmsi.asc())
+        .limit(safe_limit)
+    )
+    result = await db.execute(stmt)
+    if not hasattr(result, "scalars"):
+        return []
+    return [item.to_dict() for item in result.scalars().all()]
+
+
 async def aggregate_vessel_observations(
     db: AsyncSession,
     observations: Iterable[AISRawObservation],

backend/pytest.ini

@@ -1,4 +1,5 @@
 [pytest]
+pythonpath = ..
 asyncio_mode = auto
 testpaths = tests
 python_files = test_*.py

backend/tests/test_ai_observability.py

@@ -0,0 +1,128 @@
+import pytest
+from fastapi import HTTPException
+
+from app.schemas.ai import SituationalAnalysisRequest
+from app.services.ai_tools import web_search as web_search_module
+from app.services.ai_tools.schemas import SearchEvidence, WebSearchConfig, WebSearchProviderConfig
+from app.services.ai_tools.web_search import WebSearchClient
+from app.services import ai_client as ai_client_module
+from app.services.ai_client import AIProviderClient
+
+
+@pytest.mark.asyncio
+async def test_ai_client_analyze_logs_summary_without_prompt(monkeypatch):
+    events = []
+
+    async def fake_emit_business_log(_logger, **payload):
+        events.append(payload)
+
+    async def fake_request(self, method, path, json=None, request_id=None, operation="request", payload_summary=None):
+        return {
+            "provider": "test-provider",
+            "model": "test-model",
+            "content": "ok",
+            "content_blocks": [],
+            "text_blocks": ["ok"],
+            "thinking_blocks": [],
+            "raw_response": {},
+        }
+
+    monkeypatch.setattr(ai_client_module, "emit_business_log", fake_emit_business_log)
+    monkeypatch.setattr(AIProviderClient, "_request", fake_request)
+
+    client = AIProviderClient(
+        service_url="http://provider.test",
+        llm_config={"provider": "openai", "provider_api": "openai-completions", "model": "gpt-test", "api_key": "sk-secret"},
+    )
+    result = await client.analyze(
+        SituationalAnalysisRequest(
+            title="Sensitive title",
+            objective="Do not store this full prompt",
+            observations=["secret observation"],
+            constraints=["secret constraint"],
+            context={"source": "test", "private": "value"},
+        ),
+        request_id="req-ai-test",
+    )
+
+    assert result.model == "test-model"
+    assert [event["event"] for event in events] == [
+        "ai.provider.analyze.start",
+        "ai.provider.analyze.success",
+    ]
+    serialized = str(events)
+    assert "Do not store this full prompt" not in serialized
+    assert "secret observation" not in serialized
+    assert "sk-secret" not in serialized
+    start_context = events[0]["context"]
+    assert start_context["model"] == "gpt-test"
+    assert start_context["input_summary"]["objective_length"] == len("Do not store this full prompt")
+    assert start_context["input_summary"]["observation_count"] == 1
+    assert start_context["input_summary"]["context_keys"] == ["private", "source"]
+
+
+@pytest.mark.asyncio
+async def test_ai_client_analyze_logs_failure(monkeypatch):
+    events = []
+
+    async def fake_emit_business_log(_logger, **payload):
+        events.append(payload)
+
+    async def fake_request(self, method, path, json=None, request_id=None, operation="request", payload_summary=None):
+        raise HTTPException(status_code=502, detail="provider failed")
+
+    monkeypatch.setattr(ai_client_module, "emit_business_log", fake_emit_business_log)
+    monkeypatch.setattr(AIProviderClient, "_request", fake_request)
+
+    client = AIProviderClient(service_url="http://provider.test", llm_config={"provider": "openai", "model": "gpt-test"})
+
+    with pytest.raises(HTTPException):
+        await client.analyze(
+            SituationalAnalysisRequest(title="T", objective="O", observations=["one"]),
+            request_id="req-ai-fail",
+        )
+
+    assert events[-1]["event"] == "ai.provider.analyze.failed"
+    assert events[-1]["level"] == "error"
+    assert events[-1]["context"]["error_type"] == "HTTPException"
+
+
+@pytest.mark.asyncio
+async def test_web_search_logs_query_hash_without_query(monkeypatch):
+    events = []
+
+    async def fake_emit_business_log(_logger, **payload):
+        events.append(payload)
+
+    async def fake_search_tavily(self, config, query, max_results, domains, freshness_days):
+        return [
+            SearchEvidence(
+                title="Example",
+                url="https://example.test",
+                snippet="result",
+                source_provider="tavily",
+            )
+        ]
+
+    monkeypatch.setattr(web_search_module, "emit_business_log", fake_emit_business_log)
+    monkeypatch.setattr(WebSearchClient, "_search_tavily", fake_search_tavily)
+
+    client = WebSearchClient(
+        WebSearchConfig(
+            enabled=True,
+            default_provider="tavily",
+            providers={"tavily": WebSearchProviderConfig(provider="tavily", api_key="secret-key")},
+        )
+    )
+    results = await client.search("secret query text", max_results=1)
+
+    assert len(results) == 1
+    assert [event["event"] for event in events] == [
+        "ai_tool.web_search.start",
+        "ai_tool.web_search.success",
+    ]
+    serialized = str(events)
+    assert "secret query text" not in serialized
+    assert "secret-key" not in serialized
+    assert events[0]["context"]["query_length"] == len("secret query text")
+    assert events[1]["context"]["result_count"] == 1

backend/tests/test_api.py

@@ -655,6 +655,8 @@ async def test_ingest_earth_client_log_accepts_public_events():
                             "message": "登陆点加载失败: 登陆点接口返回 HTTP 500",
                             "category": "startup-load",
                             "module": "layer-startup",
+                            "fingerprint": "client-test",
+                            "occurrence_count": 3,
                         },
                     )
         assert response.status_code == 200
@@ -668,10 +670,98 @@ async def test_ingest_earth_client_log_accepts_public_events():
         assert persisted_kwargs["event"] == "earth.client.runtime_log"
         assert persisted_kwargs["category"] == "startup-load"
         assert persisted_kwargs["level"] == "error"
+        assert persisted_kwargs["fingerprint"] == "client-test"
+        assert persisted_kwargs["occurrence_count"] == 3
     finally:
         app.dependency_overrides.clear()
 
 
+@pytest.mark.asyncio
+async def test_ingest_admin_client_log_accepts_public_events():
+    transport = ASGITransport(app=app)
+    try:
+        with patch("app.api.v1.system_control.append_buffer_log") as mock_append_buffer_log:
+            with patch("app.api.v1.system_control.record_system_log", new_callable=AsyncMock) as mock_record_system_log:
+                async with AsyncClient(transport=transport, base_url="http://test") as client:
+                    response = await client.post(
+                        "/api/v1/system/logs/admin-client",
+                        json={
+                            "level": "error",
+                            "message": "控制台发生未处理 Promise 错误",
+                            "category": "unhandledrejection",
+                            "module": "admin",
+                            "url": "http://test/logs",
+                            "detail": "stack preview",
+                        },
+                    )
+        assert response.status_code == 200
+        data = response.json()
+        assert data["accepted"] is True
+        assert data["source_id"] == "admin-client"
+        mock_append_buffer_log.assert_called_once()
+        mock_record_system_log.assert_awaited_once()
+        persisted_kwargs = mock_record_system_log.await_args.kwargs
+        assert persisted_kwargs["source"] == "admin-client"
+        assert persisted_kwargs["event"] == "admin.client.runtime_log"
+        assert persisted_kwargs["category"] == "unhandledrejection"
+        assert persisted_kwargs["context"]["url"] == "http://test/logs"
+    finally:
+        app.dependency_overrides.clear()
+
+
+@pytest.mark.asyncio
+async def test_ingest_service_log_requires_configured_token(monkeypatch):
+    monkeypatch.setattr(settings, "OBSERVABILITY_INGEST_TOKEN", "")
+    transport = ASGITransport(app=app)
+    async with AsyncClient(transport=transport, base_url="http://test") as client:
+        response = await client.post(
+            "/api/v1/system/logs/service",
+            json={"message": "AI provider failed"},
+            headers={"X-Planet-Observability-Token": "secret"},
+        )
+
+    assert response.status_code == 503
+
+
+@pytest.mark.asyncio
+async def test_ingest_service_log_accepts_internal_token(monkeypatch):
+    monkeypatch.setattr(settings, "OBSERVABILITY_INGEST_TOKEN", "service-secret")
+    transport = ASGITransport(app=app)
+    with patch("app.api.v1.system_control.record_system_log", new_callable=AsyncMock) as mock_record_system_log:
+        async with AsyncClient(transport=transport, base_url="http://test") as client:
+            response = await client.post(
+                "/api/v1/system/logs/service",
+                json={
+                    "source": "ai-provider",
+                    "service": "ai-provider",
+                    "module": "provider",
+                    "category": "connectivity",
+                    "event": "ai.provider.test.failed",
+                    "level": "error",
+                    "message": "Provider connectivity failed",
+                    "fingerprint": "ai-provider-test",
+                    "occurrence_count": 4,
+                    "provider": "minimax",
+                    "trace_id": "trace-123",
+                    "context": {"status_code": 502},
+                },
+                headers={"Authorization": "Bearer service-secret"},
+            )
+
+    assert response.status_code == 200
+    data = response.json()
+    assert data["accepted"] is True
+    assert data["source_id"] == "ai-provider"
+    mock_record_system_log.assert_awaited_once()
+    persisted_kwargs = mock_record_system_log.await_args.kwargs
+    assert persisted_kwargs["event"] == "ai.provider.test.failed"
+    assert persisted_kwargs["fingerprint"] == "ai-provider-test"
+    assert persisted_kwargs["occurrence_count"] == 4
+    assert persisted_kwargs["context"]["provider"] == "minimax"
+    assert persisted_kwargs["context"]["trace_id"] == "trace-123"
+    assert persisted_kwargs["context"]["status_code"] == 502
+
+
 @pytest.mark.asyncio
 async def test_earth_layer_cache_status_requires_super_admin(auth_headers, monkeypatch):
     def override_get_current_user():

backend/tests/test_collectors.py

@@ -1,9 +1,13 @@
 """Unit tests for data collectors"""
 
+import json
+
 import pytest
 from unittest.mock import AsyncMock, patch
 
 from app.core.datasource_defaults import DEFAULT_DATASOURCES
+from app.services.collectors.celestrak import CelesTrakTLECollector
+from app.services.collectors.downloads import DownloadHTTPStatusError, ResumableFileDownloader
 from app.services.credential_guides import DEFAULT_CREDENTIAL_GUIDES
 from app.services.collectors.top500 import TOP500Collector
 from app.services.collectors.registry import collector_registry
@@ -149,6 +153,172 @@ class TestHTTPCollector:
         assert callable(collector.parse_response)
 
 
+class TestCelesTrakTLECollector:
+    def test_transform_uses_norad_as_source_id_and_preserves_starlink_group(self):
+        collector = CelesTrakTLECollector()
+        result = collector.transform([
+            {
+                "NORAD_CAT_ID": 44720,
+                "OBJECT_NAME": "STARLINK-1000",
+                "OBJECT_ID": "2019-029AZ",
+                "EPOCH": "2026-03-13T00:00:00Z",
+                "MEAN_MOTION": 15.79234567,
+                "ECCENTRICITY": 0.0001234,
+                "INCLINATION": 53.0,
+                "RA_OF_ASC_NODE": 10.0,
+                "ARG_OF_PERICENTER": 20.0,
+                "MEAN_ANOMALY": 30.0,
+                "_celestrak_query_group": "active",
+                "_celestrak_source_url": "https://celestrak.example/gp.php?GROUP=active&FORMAT=json",
+            }
+        ])
+
+        assert result[0]["source_id"] == "44720"
+        assert result[0]["metadata"]["constellation_group"] == "starlink"
+        assert result[0]["metadata"]["celestrak_query_group"] == "active"
+        assert result[0]["metadata"]["norad_cat_id"] == 44720
+        assert result[0]["metadata"]["tle_line1"]
+        assert result[0]["metadata"]["tle_line2"]
+
+    def test_load_active_payload_rejects_invalid_records(self, tmp_path):
+        collector = CelesTrakTLECollector()
+        payload_path = tmp_path / "active.json"
+        payload_path.write_text(json.dumps([{"OBJECT_NAME": "missing norad"}]), encoding="utf-8")
+
+        with pytest.raises(RuntimeError, match="invalid record"):
+            collector._load_active_payload(payload_path)
+
+    def test_load_active_payload_accepts_complete_array(self, tmp_path):
+        collector = CelesTrakTLECollector()
+        payload_path = tmp_path / "active.json"
+        payload_path.write_text(
+            json.dumps([{"NORAD_CAT_ID": 25544, "OBJECT_NAME": "ISS (ZARYA)"}]),
+            encoding="utf-8",
+        )
+
+        records = collector._load_active_payload(payload_path)
+
+        assert records == [{"NORAD_CAT_ID": 25544, "OBJECT_NAME": "ISS (ZARYA)"}]
+
+    @pytest.mark.asyncio
+    async def test_fetch_retries_and_raises_instead_of_returning_partial_data(self, monkeypatch, tmp_path):
+        collector = CelesTrakTLECollector()
+        collector._resolved_url = "https://celestrak.example/NORAD/elements/gp.php"
+        attempts = 0
+
+        async def fake_download_file(*args, **kwargs):
+            nonlocal attempts
+            attempts += 1
+            raise RuntimeError("network interrupted")
+
+        async def fake_emit_business_log(*args, **kwargs):
+            return None
+
+        monkeypatch.setattr(collector._downloader, "download_file", fake_download_file)
+        monkeypatch.setattr("app.services.collectors.celestrak.emit_business_log", fake_emit_business_log)
+        monkeypatch.setattr("app.services.collectors.celestrak.asyncio.sleep", AsyncMock())
+
+        with pytest.raises(RuntimeError, match="failed after retries"):
+            await collector.fetch()
+
+        assert attempts == 3
+
+    @pytest.mark.asyncio
+    async def test_fetch_uses_cache_when_celestrak_reports_not_updated(self, monkeypatch, tmp_path):
+        collector = CelesTrakTLECollector()
+        collector._resolved_url = "https://celestrak.example/NORAD/elements/gp.php"
+        collector._downloader = ResumableFileDownloader(cache_namespace="celestrak-test", cache_root=tmp_path)
+        url = collector._active_url()
+        cached_path = collector._downloader.cached_file_path(url, ".json")
+        cached_path.parent.mkdir(parents=True, exist_ok=True)
+        cached_path.write_text(
+            json.dumps([{"NORAD_CAT_ID": 25544, "OBJECT_NAME": "ISS (ZARYA)"}]),
+            encoding="utf-8",
+        )
+
+        async def fake_download_file(*args, **kwargs):
+            raise DownloadHTTPStatusError(
+                url=url,
+                status_code=403,
+                body="GP data has not updated since your last successful download of GROUP=active.",
+            )
+
+        async def fake_emit_business_log(*args, **kwargs):
+            return None
+
+        monkeypatch.setattr(collector._downloader, "download_file", fake_download_file)
+        monkeypatch.setattr("app.services.collectors.celestrak.emit_business_log", fake_emit_business_log)
+
+        records = await collector.fetch()
+
+        assert records[0]["NORAD_CAT_ID"] == 25544
+        assert records[0]["_celestrak_query_group"] == "active"
+
+    @pytest.mark.asyncio
+    async def test_fetch_not_updated_without_cache_does_not_retry(self, monkeypatch, tmp_path):
+        collector = CelesTrakTLECollector()
+        collector._resolved_url = "https://celestrak.example/NORAD/elements/gp.php"
+        collector._downloader = ResumableFileDownloader(cache_namespace="celestrak-test", cache_root=tmp_path)
+        attempts = 0
+
+        async def fake_download_file(*args, **kwargs):
+            nonlocal attempts
+            attempts += 1
+            raise DownloadHTTPStatusError(
+                url=collector._active_url(),
+                status_code=403,
+                body="GP data has not updated since your last successful download of GROUP=active.",
+            )
+
+        async def fake_emit_business_log(*args, **kwargs):
+            return None
+
+        monkeypatch.setattr(collector._downloader, "download_file", fake_download_file)
+        monkeypatch.setattr("app.services.collectors.celestrak.FALLBACK_GROUPS", ("starlink",))
+        monkeypatch.setattr("app.services.collectors.celestrak.emit_business_log", fake_emit_business_log)
+        monkeypatch.setattr("app.services.collectors.celestrak.asyncio.sleep", AsyncMock())
+
+        with pytest.raises(RuntimeError, match="fallback group mode failed"):
+            await collector.fetch()
+
+        assert attempts == 2
+
+    @pytest.mark.asyncio
+    async def test_fetch_falls_back_to_all_groups_when_active_not_updated_without_cache(self, monkeypatch, tmp_path):
+        collector = CelesTrakTLECollector()
+        collector._resolved_url = "https://celestrak.example/NORAD/elements/gp.php"
+        collector._downloader = ResumableFileDownloader(cache_namespace="celestrak-test", cache_root=tmp_path)
+        payload_by_group = {
+            "starlink": [{"NORAD_CAT_ID": 100, "OBJECT_NAME": "STARLINK-100"}],
+            "gps-ops": [{"NORAD_CAT_ID": 200, "OBJECT_NAME": "GPS BIIR-2"}],
+        }
+
+        async def fake_download_file(_client, url, **_kwargs):
+            if "GROUP=active" in url:
+                raise DownloadHTTPStatusError(
+                    url=url,
+                    status_code=403,
+                    body="GP data has not updated since your last successful download of GROUP=active.",
+                )
+            group = "starlink" if "GROUP=starlink" in url else "gps-ops"
+            path = tmp_path / f"{group}.json"
+            path.write_text(json.dumps(payload_by_group[group]), encoding="utf-8")
+            return path
+
+        async def fake_emit_business_log(*args, **kwargs):
+            return None
+
+        monkeypatch.setattr(collector._downloader, "download_file", fake_download_file)
+        monkeypatch.setattr("app.services.collectors.celestrak.FALLBACK_GROUPS", tuple(payload_by_group))
+        monkeypatch.setattr("app.services.collectors.celestrak.emit_business_log", fake_emit_business_log)
+
+        records = await collector.fetch()
+
+        assert [item["NORAD_CAT_ID"] for item in records] == [100, 200]
+        assert records[0]["_celestrak_query_group"] == "starlink"
+        assert records[1]["_celestrak_group"] == "gps-ops"
+
+
 def test_aisstream_collector_is_registered():
     collector = collector_registry.get("aisstream_vessels")
 

backend/tests/test_datasources_batch.py

@@ -1,9 +1,11 @@
+import asyncio
+import inspect
 from datetime import datetime, timedelta, timezone
 
-import pytest
-
 from app.api.v1 import datasources as datasources_api
 from app.models.datasource import DataSource
+from app.models.task import CollectionTask
+from app.services import data_jobs
 from app.services import earth_layer_cache as earth_cache
 
 
@@ -100,6 +102,71 @@ def test_serialize_datasource_row_includes_endpoint_when_requested():
     assert row["endpoint"] == "https://example.test/arcgis_landing_points"
 
 
+def test_serialize_datasource_row_separates_collector_running_from_delete_task():
+    datasource = make_datasource(9, "top500", last_status="success")
+    task = CollectionTask(
+        id=44,
+        datasource_id=9,
+        source="top500",
+        task_type="clear_data",
+        status="running",
+        phase="clearing_data",
+    )
+
+    row = datasources_api.serialize_datasource_row(
+        datasource,
+        running_tasks={9: task},
+        latest_tasks={9: task},
+        record_counts={"top500": 500},
+        endpoint_overrides={},
+        config=object(),
+        include_endpoint=False,
+    )
+
+    assert row["is_task_active"] is True
+    assert row["is_running"] is False
+    assert row["task_type"] == "clear_data"
+    assert row["task_status"] == "running"
+
+
+def test_cancel_queued_delete_task_finishes_immediately(monkeypatch):
+    task = CollectionTask(
+        id=45,
+        datasource_id=9,
+        source="top500",
+        task_type="clear_data",
+        status="queued",
+        phase="queued",
+    )
+
+    class FakeDb:
+        async def commit(self):
+            return None
+
+        async def refresh(self, _task):
+            return None
+
+    async def fake_broadcast(_task):
+        return None
+
+    monkeypatch.setattr(data_jobs, "_broadcast_task_update", fake_broadcast)
+
+    async def run():
+        return await data_jobs.request_cancel_datasource_task(FakeDb(), task)
+
+    result = asyncio.run(run())
+
+    assert result.status == "cancelled"
+    assert result.phase == "cancelled"
+    assert result.completed_at is not None
+
+
+def test_clear_data_job_relies_on_db_outbox_instead_of_extra_earth_refresh_task():
+    source = inspect.getsource(data_jobs._run_clear_data_job)
+
+    assert "enqueue_earth_refresh_job" not in source
+
+
 def test_invalidate_earth_layer_cache_for_source_covers_datasource_aliases(monkeypatch):
     patterns: list[str] = []
 
@@ -119,34 +186,36 @@ def test_invalidate_earth_layer_cache_for_source_covers_datasource_aliases(monke
     ]
 
 
-@pytest.mark.asyncio
-async def test_trigger_datasource_batch_skips_disabled_and_frequency_window(monkeypatch):
-    now = datetime.now(timezone.utc)
-    disabled = make_datasource(1, "aisstream_vessels", is_active=False)
-    not_due = make_datasource(2, "telegeography_cables", last_run_at=now, frequency_minutes=120)
-    due = make_datasource(3, "ris_live_bgp", last_run_at=now - timedelta(hours=2))
-    triggered_sources: list[str] = []
+def test_trigger_datasource_batch_skips_disabled_and_frequency_window(monkeypatch):
+    async def run():
+        now = datetime.now(timezone.utc)
+        disabled = make_datasource(1, "aisstream_vessels", is_active=False)
+        not_due = make_datasource(2, "telegeography_cables", last_run_at=now, frequency_minutes=120)
+        due = make_datasource(3, "ris_live_bgp", last_run_at=now - timedelta(hours=2))
+        queued_sources: list[str] = []
 
-    async def fake_running_tasks(_db, _ids):
-        return {}
+        async def fake_running_tasks(_db, _ids):
+            return {}
 
-    async def fake_latest_task_ids(_db, _ids):
-        return {}
+        async def fake_enqueue(_db, datasource, task_type, **_kwargs):
+            queued_sources.append(datasource.source)
+            return CollectionTask(id=100 + datasource.id, datasource_id=datasource.id, source=datasource.source, task_type=task_type, status="queued")
 
-    monkeypatch.setattr(datasources_api, "_load_latest_running_tasks", fake_running_tasks)
-    monkeypatch.setattr(datasources_api, "_load_latest_task_ids", fake_latest_task_ids)
-    monkeypatch.setattr(
-        datasources_api,
-        "run_collector_now",
-        lambda source: triggered_sources.append(source) or True,
-    )
+        monkeypatch.setattr(datasources_api, "_load_latest_running_tasks", fake_running_tasks)
+        monkeypatch.setattr(
+            datasources_api,
+            "enqueue_datasource_job",
+            fake_enqueue,
+        )
 
-    result = await datasources_api._trigger_datasource_batch(
-        object(),
-        [disabled, not_due, due],
-        force=False,
-    )
+        result = await datasources_api._trigger_datasource_batch(
+            object(),
+            [disabled, not_due, due],
+            force=False,
+        )
 
-    assert [item["source"] for item in result["triggered"]] == ["ris_live_bgp"]
-    assert {item["reason"] for item in result["skipped"]} == {"disabled", "within_frequency_window"}
-    assert triggered_sources == ["ris_live_bgp"]
+        assert [item["source"] for item in result["triggered"]] == ["ris_live_bgp"]
+        assert {item["reason"] for item in result["skipped"]} == {"disabled", "within_frequency_window"}
+        assert queued_sources == ["ris_live_bgp"]
+
+    asyncio.run(run())

backend/tests/test_docs_gatekeeper.py

@@ -1,11 +1,15 @@
 """Docs Gatekeeper API tests."""
 
+import re
+from pathlib import Path
+
 import pytest
 from httpx import ASGITransport, AsyncClient
 
 from app.api.v1 import docs as docs_api
 from app.main import app
 from app.models.user import User
+from app.services.docs_gatekeeper import DOCS_METADATA
 
 
 def make_user(role: str = "viewer", groups: list[str] | None = None) -> User:
@@ -43,22 +47,45 @@ async def test_public_catalog_only_for_anonymous_user():
     assert response.status_code == 200
     items = response.json()["items"]
     assert {item["access"] for item in items} == {"public"}
-    assert {item["slug"] for item in items if item["lang"] == "zh"} == {
+    zh_items = [item for item in items if item["lang"] == "zh"]
+    assert [item["slug"] for item in zh_items] == [
         "overview",
-        "quickstart",
         "manual",
+        "quickstart",
         "faq",
-        "location-pipeline-user",
-    }
+    ]
+
+
+@pytest.mark.asyncio
+async def test_developer_catalog_includes_architecture_and_frontend_reference_docs():
+    response = await get_json(
+        "/api/v1/docs/catalog",
+        make_user(role="viewer", groups=["docs_developer"]),
+    )
+
+    assert response.status_code == 200
+    zh_slugs = {item["slug"] for item in response.json()["items"] if item["lang"] == "zh"}
+    zh_items = [item for item in response.json()["items"] if item["lang"] == "zh"]
+    assert [item["group"] for item in zh_items[:4]] == ["Overview", "Manual", "Manual", "Manual"]
+    assert zh_items[4]["group"] == "Architecture"
+    assert "platform-data-flows" in zh_slugs
+    assert "naming-glossary" in zh_slugs
+    assert "tactile-ui-components" in zh_slugs
 
 
 @pytest.mark.asyncio
 async def test_anonymous_can_read_public_doc():
     response = await get_json("/api/v1/docs/zh/quickstart")
+    manual_response = await get_json("/api/v1/docs/zh/manual")
+    overview_response = await get_json("/api/v1/docs/zh/overview")
 
     assert response.status_code == 200
     assert response.json()["access"] == "public"
     assert "快速开始" in response.json()["markdown"]
+    assert manual_response.status_code == 200
+    assert manual_response.json()["access"] == "public"
+    assert overview_response.status_code == 200
+    assert overview_response.json()["access"] == "public"
 
 
 @pytest.mark.asyncio
@@ -83,10 +110,16 @@ async def test_developer_group_can_read_developer_but_not_admin_doc():
     user = make_user(role="viewer", groups=["docs_developer"])
 
     developer_response = await get_json("/api/v1/docs/zh/backend-collectors", user)
+    tactile_response = await get_json("/api/v1/docs/zh/tactile-ui-components", user)
+    glossary_response = await get_json("/api/v1/docs/zh/naming-glossary", user)
     admin_response = await get_json("/api/v1/docs/zh/backend-system-service-control", user)
 
     assert developer_response.status_code == 200
     assert developer_response.json()["access"] == "docs_developer"
+    assert tactile_response.status_code == 200
+    assert tactile_response.json()["access"] == "docs_developer"
+    assert glossary_response.status_code == 200
+    assert glossary_response.json()["access"] == "docs_developer"
     assert admin_response.status_code == 403
 
 
@@ -114,3 +147,29 @@ async def test_unknown_language_slug_and_path_traversal_do_not_read_files():
     assert bad_lang.status_code == 404
     assert bad_slug.status_code == 404
     assert traversal.status_code == 404
+
+
+def test_public_docs_markdown_links_do_not_create_missing_docs_routes():
+    repo_root = Path(__file__).resolve().parents[2]
+    technical_root = repo_root / "docs" / "technical"
+    registered_filenames = {entry.filename for entry in DOCS_METADATA}
+    problems: list[str] = []
+
+    for markdown_path in sorted(technical_root.glob("*/*.md")):
+        markdown = markdown_path.read_text(encoding="utf-8")
+        for match in re.finditer(r"\[([^\]]+)]\(([^)]+\.md(?:#[^)]+)?)\)", markdown):
+            label, href = match.group(1), match.group(2)
+            if href.startswith(("http://", "https://", "mailto:")):
+                continue
+
+            href_without_hash = href.split("#", 1)[0].replace("\\", "/")
+            filename = Path(href_without_hash).name
+            if "/docs/technical/" in href_without_hash:
+                if filename not in registered_filenames:
+                    problems.append(f"{markdown_path.relative_to(repo_root)} links unregistered public doc {href!r} ({label})")
+                continue
+
+            if href_without_hash.endswith(".md"):
+                problems.append(f"{markdown_path.relative_to(repo_root)} links non-public markdown {href!r} ({label})")
+
+    assert problems == []

backend/tests/test_earth_db_change_listener.py

@@ -0,0 +1,454 @@
+import asyncio
+import json
+
+from app.services.earth_db_change_listener import (
+    EarthDbChangeDispatcher,
+    EarthDbChangeListener,
+    build_earth_update_from_db_payload,
+    normalize_asyncpg_dsn,
+)
+
+
+def test_normalize_asyncpg_dsn_strips_sqlalchemy_driver():
+    assert (
+        normalize_asyncpg_dsn("postgresql+asyncpg://postgres:postgres@localhost:5432/planet_db")
+        == "postgresql://postgres:postgres@localhost:5432/planet_db"
+    )
+
+
+def test_build_earth_update_maps_known_sources_to_layers():
+    satellite_update = build_earth_update_from_db_payload(
+        {"source": "celestrak_tle", "operation": "DELETE", "entity_key": "25544"}
+    )
+    cable_update = build_earth_update_from_db_payload(
+        {"source": "arcgis_cables", "operation": "UPDATE", "entity_key": "cable-1"}
+    )
+    compute_update = build_earth_update_from_db_payload(
+        {"source": "top500", "operation": "DELETE", "entity_key": None}
+    )
+    landing_update = build_earth_update_from_db_payload(
+        {"source": "telegeography_landing", "operation": "DELETE", "entity_key": None}
+    )
+
+    assert satellite_update is not None
+    assert satellite_update["layers"] == ["satellites"]
+    assert cable_update is not None
+    assert cable_update["layers"] == ["cables"]
+    assert compute_update is not None
+    assert compute_update["layers"] == ["computeCenters"]
+    assert landing_update is not None
+    assert landing_update["layers"] == ["cables"]
+    assert landing_update["refresh_strategy"] == "clear_then_reload"
+
+
+def test_build_earth_update_maps_derived_tables_to_layers():
+    bgp_update = build_earth_update_from_db_payload(
+        {"table": "bgp_anomalies", "source": "ris_live_bgp", "operation": "DELETE", "entity_key": "a1"}
+    )
+    compute_update = build_earth_update_from_db_payload(
+        {"table": "compute_center_locations", "source": "top500", "operation": "UPDATE", "entity_key": "top500_1"}
+    )
+    vessel_update = build_earth_update_from_db_payload(
+        {"table": "vessel_position", "operation": "DELETE", "entity_key": "123456789"}
+    )
+
+    assert bgp_update is not None
+    assert bgp_update["source"] == "ris_live_bgp"
+    assert bgp_update["table"] == "bgp_anomalies"
+    assert bgp_update["layers"] == ["bgp"]
+    assert bgp_update["refresh_strategy"] == "clear_then_reload"
+    assert compute_update is not None
+    assert compute_update["layers"] == ["computeCenters"]
+    assert compute_update["refresh_strategy"] == "reload"
+    assert vessel_update is not None
+    assert vessel_update["source"] == "vessel_position"
+    assert vessel_update["layers"] == ["vessels"]
+
+
+def test_build_earth_update_maps_interactable_delete_to_delta():
+    update = build_earth_update_from_db_payload(
+        {
+            "table": "earth_interactables",
+            "operation": "DELETE",
+            "entity_keys": ["note-1"],
+            "records_processed": 1,
+        }
+    )
+
+    assert update is not None
+    assert update["entity"] == "interactable"
+    assert update["action"] == "deleted"
+    assert update["ids"] == ["note-1"]
+    assert update["layers"] == ["interactables"]
+    assert update["refresh_strategy"] == "delta"
+
+
+def test_build_earth_update_ignores_unmapped_sources():
+    assert build_earth_update_from_db_payload({"source": "not_for_earth"}) is None
+
+
+def test_dispatcher_debounces_same_source_notifications():
+    async def run():
+        broadcasts = []
+        invalidated = []
+
+        async def broadcast(payload):
+            broadcasts.append(payload)
+
+        def invalidate(source):
+            invalidated.append(source)
+            return 2
+
+        dispatcher = EarthDbChangeDispatcher(
+            broadcast_earth_update=broadcast,
+            invalidate_cache=invalidate,
+            debounce_seconds=10,
+        )
+        dispatcher.handle_payload(
+            {
+                "table": "collected_data",
+                "operation": "INSERT",
+                "source": "arcgis_cables",
+                "entity_key": "cable-1",
+                "occurred_at": "2026-05-22T00:00:00Z",
+            }
+        )
+        dispatcher.handle_payload(
+            {
+                "table": "collected_data",
+                "operation": "UPDATE",
+                "source": "arcgis_cables",
+                "entity_key": "cable-2",
+                "occurred_at": "2026-05-22T00:00:01Z",
+            }
+        )
+
+        await dispatcher.flush_all()
+
+        assert invalidated == ["arcgis_cables"]
+        assert len(broadcasts) == 1
+        assert broadcasts[0]["action"] == "database_changed"
+        assert broadcasts[0]["source"] == "arcgis_cables"
+        assert broadcasts[0]["layers"] == ["cables"]
+        assert broadcasts[0]["records_processed"] == 2
+        assert broadcasts[0]["operations"] == ["INSERT", "UPDATE"]
+        assert broadcasts[0]["entity_keys"] == ["cable-1", "cable-2"]
+        assert broadcasts[0]["cache_entries_invalidated"] == 2
+
+    asyncio.run(run())
+
+
+def test_dispatcher_handles_delete_notification_as_earth_update():
+    async def run():
+        broadcasts = []
+
+        async def broadcast(payload):
+            broadcasts.append(payload)
+
+        dispatcher = EarthDbChangeDispatcher(
+            broadcast_earth_update=broadcast,
+            invalidate_cache=lambda source: 1,
+            debounce_seconds=10,
+        )
+        accepted = dispatcher.handle_notification(
+            json.dumps(
+                {
+                    "table": "collected_data",
+                    "operation": "DELETE",
+                    "source": "celestrak_tle",
+                    "entity_key": "sat:25544",
+                    "occurred_at": "2026-05-22T00:00:00Z",
+                }
+            )
+        )
+
+        await dispatcher.flush_all()
+
+        assert accepted is True
+        assert len(broadcasts) == 1
+        assert broadcasts[0]["action"] == "database_changed"
+        assert broadcasts[0]["source"] == "celestrak_tle"
+        assert broadcasts[0]["layers"] == ["satellites"]
+        assert broadcasts[0]["operation"] == "DELETE"
+        assert broadcasts[0]["refresh_strategy"] == "clear_then_reload"
+
+    asyncio.run(run())
+
+
+def test_dispatcher_debounces_bgp_derived_table_events_by_source():
+    async def run():
+        broadcasts = []
+
+        async def broadcast(payload):
+            broadcasts.append(payload)
+
+        dispatcher = EarthDbChangeDispatcher(
+            broadcast_earth_update=broadcast,
+            invalidate_cache=lambda source: 0,
+            debounce_seconds=10,
+        )
+        dispatcher.handle_payload(
+            {
+                "event_id": 200,
+                "table": "bgp_observations",
+                "operation": "DELETE",
+                "source": "ris_live_bgp",
+                "records_processed": 2,
+            }
+        )
+        dispatcher.handle_payload(
+            {
+                "event_id": 201,
+                "table": "bgp_anomalies",
+                "operation": "DELETE",
+                "source": "ris_live_bgp",
+                "records_processed": 3,
+            }
+        )
+
+        await dispatcher.flush_all()
+
+        assert len(broadcasts) == 1
+        assert broadcasts[0]["source"] == "ris_live_bgp"
+        assert broadcasts[0]["layers"] == ["bgp"]
+        assert broadcasts[0]["records_processed"] == 5
+        assert broadcasts[0]["refresh_strategy"] == "clear_then_reload"
+
+    asyncio.run(run())
+
+
+def test_dispatcher_deduplicates_notify_and_outbox_by_event_id():
+    async def run():
+        broadcasts = []
+
+        async def broadcast(payload):
+            broadcasts.append(payload)
+
+        dispatcher = EarthDbChangeDispatcher(
+            broadcast_earth_update=broadcast,
+            invalidate_cache=lambda source: 0,
+            debounce_seconds=10,
+        )
+        payload = {
+            "event_id": 42,
+            "table": "collected_data",
+            "operation": "INSERT",
+            "source": "celestrak_tle",
+            "entity_key": "sat:42",
+            "occurred_at": "2026-05-23T00:00:00Z",
+        }
+
+        assert dispatcher.handle_payload(payload) is True
+        assert dispatcher.handle_payload(dict(payload)) is False
+        await dispatcher.flush_all()
+
+        assert len(broadcasts) == 1
+        assert broadcasts[0]["records_processed"] == 1
+        assert broadcasts[0]["entity_keys"] == ["sat:42"]
+
+    asyncio.run(run())
+
+
+def test_dispatcher_flushes_continuous_events_at_max_wait():
+    async def run():
+        broadcasts = []
+        sleeps = []
+
+        async def broadcast(payload):
+            broadcasts.append(payload)
+
+        dispatcher = EarthDbChangeDispatcher(
+            broadcast_earth_update=broadcast,
+            invalidate_cache=lambda source: 0,
+            debounce_seconds=10,
+            max_wait_seconds=0.01,
+        )
+        dispatcher.handle_payload(
+            {
+                "event_id": 300,
+                "table": "collected_data",
+                "operation": "INSERT",
+                "source": "arcgis_cables",
+                "records_processed": 1,
+            }
+        )
+        await asyncio.sleep(0.02)
+        dispatcher.handle_payload(
+            {
+                "event_id": 301,
+                "table": "collected_data",
+                "operation": "UPDATE",
+                "source": "arcgis_cables",
+                "records_processed": 1,
+            }
+        )
+        await asyncio.sleep(0)
+        await dispatcher.flush_all()
+
+        assert len(broadcasts) == 1
+        assert broadcasts[0]["records_processed"] == 2
+        assert broadcasts[0]["debounce_ms"] >= 0
+
+    asyncio.run(run())
+
+
+def test_dispatcher_fast_flushes_delete_clear_then_reload():
+    async def run():
+        broadcasts = []
+
+        async def broadcast(payload):
+            broadcasts.append(payload)
+
+        dispatcher = EarthDbChangeDispatcher(
+            broadcast_earth_update=broadcast,
+            invalidate_cache=lambda source: 0,
+            debounce_seconds=10,
+            max_wait_seconds=10,
+        )
+        dispatcher.handle_payload(
+            {
+                "event_id": 310,
+                "table": "collected_data",
+                "operation": "DELETE",
+                "source": "celestrak_tle",
+                "records_processed": 10,
+            }
+        )
+        await asyncio.sleep(0.08)
+
+        assert len(broadcasts) == 1
+        assert broadcasts[0]["source"] == "celestrak_tle"
+        assert broadcasts[0]["refresh_strategy"] == "clear_then_reload"
+
+    asyncio.run(run())
+
+
+def test_outbox_rows_are_consumed_after_successful_flush():
+    async def run():
+        class FakeConnection:
+            def __init__(self):
+                self.consumed_ids = []
+
+            async def fetch(self, _query, _limit):
+                return [
+                    {
+                        "id": 501,
+                        "payload": {
+                            "event_id": 501,
+                            "table": "collected_data",
+                            "operation": "DELETE",
+                            "source": "celestrak_tle",
+                            "records_processed": 1,
+                        },
+                    }
+                ]
+
+            async def execute(self, _query, consumed_ids):
+                self.consumed_ids.extend(consumed_ids)
+
+        broadcasts = []
+
+        async def broadcast(payload):
+            broadcasts.append(payload)
+
+        dispatcher = EarthDbChangeDispatcher(
+            broadcast_earth_update=broadcast,
+            invalidate_cache=lambda source: 0,
+            debounce_seconds=10,
+        )
+        listener = EarthDbChangeListener(
+            dsn="postgresql://example/db",
+            dispatcher=dispatcher,
+        )
+        connection = FakeConnection()
+        listener._connection = connection
+
+        await listener._poll_outbox()
+
+        assert connection.consumed_ids == [501]
+        assert len(broadcasts) == 1
+
+    asyncio.run(run())
+
+
+def test_outbox_rows_remain_unconsumed_when_flush_fails():
+    async def run():
+        class FakeConnection:
+            def __init__(self):
+                self.consumed_ids = []
+
+            async def fetch(self, _query, _limit):
+                return [
+                    {
+                        "id": 601,
+                        "payload": {
+                            "event_id": 601,
+                            "table": "collected_data",
+                            "operation": "DELETE",
+                            "source": "celestrak_tle",
+                            "records_processed": 1,
+                        },
+                    }
+                ]
+
+            async def execute(self, _query, consumed_ids):
+                self.consumed_ids.extend(consumed_ids)
+
+        async def broadcast(_payload):
+            raise RuntimeError("ws down")
+
+        dispatcher = EarthDbChangeDispatcher(
+            broadcast_earth_update=broadcast,
+            invalidate_cache=lambda source: 0,
+            debounce_seconds=10,
+        )
+        listener = EarthDbChangeListener(
+            dsn="postgresql://example/db",
+            dispatcher=dispatcher,
+        )
+        connection = FakeConnection()
+        listener._connection = connection
+
+        try:
+            await listener._poll_outbox()
+        except RuntimeError:
+            pass
+
+        assert connection.consumed_ids == []
+
+    asyncio.run(run())
+
+
+def test_dispatcher_uses_aggregated_statement_record_count():
+    async def run():
+        broadcasts = []
+
+        async def broadcast(payload):
+            broadcasts.append(payload)
+
+        dispatcher = EarthDbChangeDispatcher(
+            broadcast_earth_update=broadcast,
+            invalidate_cache=lambda source: 0,
+            debounce_seconds=10,
+        )
+        dispatcher.handle_payload(
+            {
+                "event_id": 99,
+                "table": "collected_data",
+                "operation": "DELETE",
+                "source": "top500",
+                "records_processed": 100,
+                "entity_keys": ["top500:1", "top500:2"],
+                "occurred_at": "2026-05-23T00:00:00Z",
+            }
+        )
+
+        await dispatcher.flush_all()
+
+        assert len(broadcasts) == 1
+        assert broadcasts[0]["source"] == "top500"
+        assert broadcasts[0]["layers"] == ["computeCenters"]
+        assert broadcasts[0]["records_processed"] == 100
+        assert broadcasts[0]["entity_keys"] == ["top500:1", "top500:2"]
+
+    asyncio.run(run())

backend/tests/test_earth_interactables.py

@@ -0,0 +1,79 @@
+from datetime import UTC, datetime
+
+from app.models.earth_interactable import EarthInteractable
+from app.services import earth_interactables
+
+
+def make_interactable(**overrides):
+    values = {
+        "id": "poi-1",
+        "layer": "places",
+        "kind": "note",
+        "label": "Test POI",
+        "description": "A point on Earth",
+        "latitude": 30.25,
+        "longitude": 120.15,
+        "altitude": None,
+        "revision": 3,
+        "properties": {"owner": "test"},
+        "is_deleted": False,
+        "created_at": datetime(2026, 5, 22, 1, 0, tzinfo=UTC),
+        "updated_at": datetime(2026, 5, 22, 1, 5, tzinfo=UTC),
+        "deleted_at": None,
+    }
+    values.update(overrides)
+    return EarthInteractable(**values)
+
+
+def test_interactable_event_uses_object_delta_contract():
+    record = make_interactable()
+
+    event = earth_interactables.build_interactable_event(
+        action="updated",
+        record=record,
+    )
+
+    assert event["entity"] == "interactable"
+    assert event["action"] == "updated"
+    assert event["layer"] == "places"
+    assert event["layers"] == ["interactables"]
+    assert event["ids"] == ["poi-1"]
+    assert event["revision"] == 3
+    assert event["item"]["latitude"] == 30.25
+    assert event["item"]["properties"] == {"owner": "test"}
+
+
+def test_interactable_geojson_omits_deleted_records():
+    active = make_interactable(id="active")
+    deleted = make_interactable(id="deleted", is_deleted=True)
+
+    payload = earth_interactables.interactables_to_geojson([active, deleted])
+
+    assert payload["type"] == "FeatureCollection"
+    assert [feature["id"] for feature in payload["features"]] == ["active"]
+    assert payload["features"][0]["geometry"] == {
+        "type": "Point",
+        "coordinates": [120.15, 30.25],
+    }
+
+
+def test_interactable_cache_invalidation_clears_layer_and_all(monkeypatch):
+    patterns = []
+
+    def fake_delete_pattern(pattern):
+        patterns.append(pattern)
+        return 1
+
+    monkeypatch.setattr(
+        earth_interactables.earth_layer_cache,
+        "delete_pattern",
+        fake_delete_pattern,
+    )
+
+    deleted = earth_interactables.invalidate_interactable_cache("places")
+
+    assert deleted == 2
+    assert patterns == [
+        "earth:layer:v1:interactables:interactable_layer:places*",
+        "earth:layer:v1:interactables:interactable_layer:all*",
+    ]

backend/tests/test_earth_news.py

@@ -1,21 +1,30 @@
-from datetime import UTC, datetime
+from datetime import UTC, datetime, timedelta
 from types import SimpleNamespace
 
 import pytest
 
 from app.services.earth_news import (
+    NewsFeedEndpoint,
     NewsFeedSource,
     NewsTargetLocation,
     ParsedNewsItem,
+    apply_news_classification,
+    default_earth_news_sources_payload,
+    normalize_earth_news_sources_payload,
+    _fetch_source,
+    _diversify_news_items_for_locale,
     _enrich_items_with_target_locations,
     _extract_target_location_from_text,
     _parse_feed_entries,
+    _rank_and_trim_items,
     _serialize_item,
     get_earth_news_payload,
+    test_news_source_config as run_news_source_config_test,
 )
 from app.services.earth_news_queue import NewsTargetLocationMessage
 from app.services.earth_news_worker import process_target_location_message
 from app.services.collectors.media_news_archive import MediaNewsArchiveCollector
+from app.services.earth_news_store import _diversify_parsed_news_items_by_region
 
 
 def test_serialize_item_includes_region_anchor_for_cruise():
@@ -44,6 +53,188 @@ def test_serialize_item_includes_region_anchor_for_cruise():
     assert payload["published_at"] == "2026-04-23T02:30:00Z"
 
 
+def test_serialize_item_includes_breaking_fields():
+    item = ParsedNewsItem(
+        id="breaking:test",
+        title="Major market halt",
+        summary="Trading halt after flash crash",
+        url="https://example.com/breaking",
+        source="Example Source",
+        feed_name="Example Feed",
+        feed_region="global",
+        homepage_url="https://example.com",
+        published_at=datetime(2026, 5, 15, 2, 0, tzinfo=UTC),
+        breaking_level="critical",
+        breaking_scope="global",
+        breaking_reasons=["重大金融市场异常"],
+        breaking_source="rules",
+        breaking_confidence=0.72,
+        breaking_expires_at=datetime(2026, 5, 16, 2, 0, tzinfo=UTC),
+    )
+
+    payload = _serialize_item(item, active_region="europe")
+
+    assert payload["breaking_level"] == "critical"
+    assert payload["breaking_scope"] == "global"
+    assert payload["breaking_reasons"] == ["重大金融市场异常"]
+    assert payload["breaking_source"] == "rules"
+    assert payload["breaking_confidence"] == 0.72
+    assert payload["breaking_expires_at"] == "2026-05-16T02:00:00Z"
+
+
+def test_rank_and_trim_items_prioritizes_active_breaking():
+    older_breaking = ParsedNewsItem(
+        id="global:critical",
+        title="Nuclear accident reported",
+        summary="A nuclear accident has been reported.",
+        url="https://example.com/critical",
+        source="Global Source",
+        feed_name="Global Feed",
+        feed_region="global",
+        homepage_url="https://example.com",
+        published_at=datetime.now(UTC) - timedelta(hours=2),
+        breaking_level="critical",
+        breaking_scope="global",
+        breaking_expires_at=datetime.now(UTC) + timedelta(hours=6),
+    )
+    newer_regular = ParsedNewsItem(
+        id="europe:regular",
+        title="Regular Europe story",
+        summary="A newer regular story.",
+        url="https://example.com/regular",
+        source="Europe Source",
+        feed_name="Europe Feed",
+        feed_region="europe",
+        homepage_url="https://example.com",
+        published_at=datetime.now(UTC),
+    )
+    expired_breaking = ParsedNewsItem(
+        id="europe:expired",
+        title="Expired breaking",
+        summary="Expired breaking story.",
+        url="https://example.com/expired",
+        source="Europe Source",
+        feed_name="Europe Feed",
+        feed_region="europe",
+        homepage_url="https://example.com",
+        published_at=datetime.now(UTC) + timedelta(minutes=1),
+        breaking_level="critical",
+        breaking_scope="regional",
+        breaking_expires_at=datetime.now(UTC) - timedelta(minutes=1),
+    )
+
+    ranked = _rank_and_trim_items(
+        [newer_regular, expired_breaking, older_breaking],
+        active_region="europe",
+        limit=3,
+    )
+
+    assert [item.id for item in ranked] == ["global:critical", "europe:expired", "europe:regular"]
+
+
+def test_diversify_news_items_prefers_display_ready_content_across_sources():
+    published_at = datetime(2026, 6, 11, 3, 0, tzinfo=UTC)
+
+    def make_item(source_id: str, suffix: str, *, zh_ready: bool) -> ParsedNewsItem:
+        return ParsedNewsItem(
+            id=f"{source_id}:{suffix}",
+            title=f"{source_id} title {suffix}",
+            summary=f"{source_id} summary {suffix}",
+            url=f"https://example.com/{source_id}/{suffix}",
+            source=source_id,
+            feed_name=source_id,
+            feed_region="global",
+            homepage_url="https://example.com",
+            published_at=published_at,
+            content_language="en",
+            localizations={
+                "zh-CN": {
+                    "title": f"{source_id} 中文标题 {suffix}",
+                    "summary": f"{source_id} 中文摘要 {suffix}",
+                }
+            } if zh_ready else {},
+        )
+
+    items = [
+        make_item("source-a", "1", zh_ready=False),
+        make_item("source-a", "2", zh_ready=False),
+        make_item("source-a", "3", zh_ready=False),
+        make_item("source-b", "1", zh_ready=True),
+        make_item("source-c", "1", zh_ready=True),
+    ]
+
+    result = _diversify_news_items_for_locale(
+        items,
+        active_region="global",
+        limit=3,
+        locale="zh-CN",
+    )
+
+    assert [item.id.split(":", 1)[0] for item in result] == ["source-b", "source-c", "source-a"]
+
+
+def test_cruise_news_diversity_keeps_regions_from_being_starved():
+    published_at = datetime(2026, 6, 26, 8, 0, tzinfo=UTC)
+
+    def make_item(region: str, index: int) -> ParsedNewsItem:
+        return ParsedNewsItem(
+            id=f"{region}:{index}",
+            title=f"{region} story {index}",
+            summary=f"{region} summary {index}",
+            url=f"https://example.com/{region}/{index}",
+            source=region,
+            feed_name=region,
+            feed_region=region,
+            homepage_url="https://example.com",
+            published_at=published_at - timedelta(minutes=index),
+        )
+
+    items = [
+        *[make_item("asia-pacific", index) for index in range(40)],
+        make_item("europe", 1),
+        make_item("middle-east-africa", 1),
+        make_item("americas", 1),
+        make_item("global", 1),
+    ]
+
+    result = _diversify_parsed_news_items_by_region(items, limit=8)
+    regions = [item.feed_region for item in result]
+
+    assert "europe" in regions
+    assert "middle-east-africa" in regions
+    assert "americas" in regions
+    assert regions.count("asia-pacific") < len(regions)
+
+
+def test_global_news_diversity_uses_same_region_balance():
+    published_at = datetime(2026, 6, 26, 8, 0, tzinfo=UTC)
+
+    def make_item(region: str, index: int) -> ParsedNewsItem:
+        return ParsedNewsItem(
+            id=f"{region}:global:{index}",
+            title=f"{region} story {index}",
+            summary=f"{region} summary {index}",
+            url=f"https://example.com/{region}/global/{index}",
+            source=region,
+            feed_name=region,
+            feed_region=region,
+            homepage_url="https://example.com",
+            published_at=published_at - timedelta(minutes=index),
+        )
+
+    items = [
+        *[make_item("asia-pacific", index) for index in range(24)],
+        *[make_item("europe", index) for index in range(2)],
+        *[make_item("middle-east-africa", index) for index in range(2)],
+        *[make_item("americas", index) for index in range(2)],
+    ]
+
+    result = _diversify_parsed_news_items_by_region(items, limit=6)
+    regions = {item.feed_region for item in result}
+
+    assert {"europe", "middle-east-africa", "americas"}.issubset(regions)
+
+
 def test_serialize_item_falls_back_to_global_anchor():
     item = ParsedNewsItem(
         id="custom:test",
@@ -164,6 +355,486 @@ def test_parse_aggregated_rss_splits_publisher_from_title():
     assert items[0].source == "Reuters"
 
 
+def test_parse_chinese_rss_marks_source_language_and_keeps_zh_localization():
+    source = NewsFeedSource(
+        id="36kr",
+        name="36氪",
+        region="asia-pacific",
+        feed_url="https://36kr.com/feed",
+        homepage_url="https://www.36kr.com/",
+        source_tags=("china", "business_news"),
+        default_category="business",
+    )
+    xml = """
+    <rss>
+      <channel>
+        <item>
+          <title>中国电商平台发布季度增长数据</title>
+          <description>平台表示，跨境电商订单量同比增长。</description>
+          <link>https://36kr.com/p/example</link>
+        </item>
+      </channel>
+    </rss>
+    """
+
+    items = _parse_feed_entries(xml, source)
+    payload_zh = _serialize_item(items[0], active_region="global", locale="zh-CN")
+    payload_en = _serialize_item(items[0], active_region="global", locale="en-US")
+
+    assert items[0].content_language == "zh-CN"
+    assert items[0].localizations["zh-CN"]["title"] == "中国电商平台发布季度增长数据"
+    assert payload_zh["display_title"] == "中国电商平台发布季度增长数据"
+    assert payload_en["display_title"] == ""
+
+    items[0].localizations["en-US"] = {
+        "title": "Chinese e-commerce platform reports quarterly growth",
+        "summary": "The platform said cross-border orders rose year over year.",
+    }
+    payload_en_ready = _serialize_item(items[0], active_region="global", locale="en-US")
+    assert payload_en_ready["display_title"] == "Chinese e-commerce platform reports quarterly growth"
+
+
+def test_default_news_sources_include_business_and_ecommerce_sources():
+    payload = default_earth_news_sources_payload()
+    sources_by_id = {source["id"]: source for source in payload["sources"]}
+    source_ids = {source["id"] for source in payload["sources"]}
+    category_keys = {category["key"] for category in payload["categories"]}
+    tag_keys = {tag["key"] for tag in payload["source_tags"]}
+
+    assert "cnbc-business" in source_ids
+    assert "36kr" in source_ids
+    assert "techcrunch" in source_ids
+    assert "retaildive" in source_ids
+    assert "prnewswire-retail" in source_ids
+    assert "google-news" in source_ids
+    assert "global-scan" not in source_ids
+    assert "google-americas" not in source_ids
+    assert "google-europe" not in source_ids
+    assert "google-mea" not in source_ids
+    assert "google-apac" not in source_ids
+    assert "businesswire-ecommerce" in source_ids
+    assert "us-census-ecommerce" in source_ids
+    assert "mofcom-data" in source_ids
+    assert "stats-china-online-retail" in source_ids
+    assert "ebrun" in source_ids
+    assert sources_by_id["36kr"]["source_type"] == "rss"
+    assert sources_by_id["36kr"]["homepage_url"] == "https://www.36kr.com/"
+    assert sources_by_id["36kr"]["feed_directory_url"] == "https://www.36kr.com/rss-center"
+    kr_feeds = {feed["id"]: feed for feed in sources_by_id["36kr"]["feeds"]}
+    assert set(kr_feeds) == {"feed", "article", "newsflash", "moment"}
+    assert kr_feeds["feed"]["url"] == "https://36kr.com/feed"
+    assert kr_feeds["article"]["url"] == "https://36kr.com/feed-article"
+    assert kr_feeds["newsflash"]["url"] == "https://36kr.com/feed-newsflash"
+    assert kr_feeds["moment"]["url"] == "https://36kr.com/feed-moment"
+    assert all(feed["enabled"] is True for feed in kr_feeds.values())
+    assert all(feed["default_category"] == "business" for feed in kr_feeds.values())
+    assert "https://36kr.com/feed-article" in sources_by_id["36kr"]["feed_urls"]
+    assert "https://36kr.com/feed-newsflash" in sources_by_id["36kr"]["feed_urls"]
+    assert "https://36kr.com/feed-moment" in sources_by_id["36kr"]["feed_urls"]
+    assert sources_by_id["ebrun"]["source_type"] == "rss"
+    assert sources_by_id["ebrun"]["homepage_url"] == "https://www.ebrun.com/"
+    assert sources_by_id["ebrun"]["feed_directory_url"] == "https://www.ebrun.com/rss/"
+    ebrun_feeds = {feed["id"]: feed for feed in sources_by_id["ebrun"]["feeds"]}
+    assert {"b2c", "b2b", "retail", "o2o", "service", "data", "policy"}.issubset(ebrun_feeds)
+    assert all(feed["enabled"] is True for feed in ebrun_feeds.values())
+    assert all(feed["default_category"] == "ecommerce" for feed in ebrun_feeds.values())
+    assert "https://www.ebrun.com/rss/news_b2c.xml" in sources_by_id["ebrun"]["feed_urls"]
+    assert "https://www.ebrun.com/rss/news_retail.xml" in sources_by_id["ebrun"]["feed_urls"]
+    assert sources_by_id["businesswire-ecommerce"]["source_type"] == "reference"
+    assert sources_by_id["businesswire-ecommerce"]["enabled"] is False
+    assert sources_by_id["google-news"]["source_type"] == "aggregated"
+    assert sources_by_id["google-news"]["homepage_url"] == "https://news.google.com/"
+    assert sources_by_id["google-news"]["feed_directory_url"] == "https://news.google.com/rss"
+    google_feeds = {feed["id"]: feed for feed in sources_by_id["google-news"]["feeds"]}
+    assert set(google_feeds) == {"world", "americas", "europe", "middle-east-africa", "asia-pacific"}
+    assert all(feed["type"] == "aggregated" for feed in google_feeds.values())
+    assert all(feed["enabled"] is True for feed in google_feeds.values())
+    assert google_feeds["world"]["region"] == "global"
+    assert google_feeds["europe"]["region"] == "europe"
+    assert sources_by_id["stats-china-online-retail"]["source_type"] == "rss"
+    assert sources_by_id["stats-china-online-retail"]["enabled"] is True
+    assert "https://www.stats.gov.cn/sj/zxfb/rss.xml" in sources_by_id["stats-china-online-retail"]["feed_urls"]
+    assert {"business", "ecommerce", "finance"}.issubset(category_keys)
+    assert {"official_data", "business_news", "ecommerce", "press_release", "finance", "logistics"}.issubset(tag_keys)
+
+
+def test_default_enabled_fetchable_sources_have_explicit_types_and_urls():
+    payload = default_earth_news_sources_payload()
+    for source in payload["sources"]:
+        source_type = source["source_type"]
+        assert source_type in {"rss", "atom", "aggregated", "reference"}
+        if source_type == "reference":
+            assert source["enabled"] is False
+            assert source["feeds"] == []
+            continue
+        if source["enabled"]:
+            assert source["feed_url"]
+            assert source["feed_urls"]
+            assert source["feeds"]
+            assert any(feed["enabled"] for feed in source["feeds"])
+            for feed in source["feeds"]:
+                assert feed["url"] != source["homepage_url"]
+                assert feed["url"] != source.get("feed_directory_url", "")
+
+
+def test_legacy_news_source_urls_migrate_to_feed_children():
+    payload = normalize_earth_news_sources_payload(
+        {
+            "sources": [
+                {
+                    "id": "legacy-source",
+                    "name": "Legacy Source",
+                    "region": "global",
+                    "source_type": "rss",
+                    "feed_urls": ["https://example.com/a.xml", "https://example.com/b.xml"],
+                    "default_category": "business",
+                }
+            ]
+        }
+    )
+
+    source = payload["sources"][0]
+
+    assert source["feed_urls"] == ["https://example.com/a.xml", "https://example.com/b.xml"]
+    assert [feed["url"] for feed in source["feeds"]] == ["https://example.com/a.xml", "https://example.com/b.xml"]
+    assert [feed["id"] for feed in source["feeds"]] == ["feed-1", "feed-2"]
+    assert all(feed["default_category"] == "business" for feed in source["feeds"])
+
+
+def test_builtin_news_source_legacy_directory_url_is_repaired():
+    payload = normalize_earth_news_sources_payload(
+        {
+            "sources": [
+                {
+                    "id": "36kr",
+                    "name": "36氪",
+                    "region": "asia-pacific",
+                    "source_type": "rss",
+                    "homepage_url": "https://www.36kr.com/",
+                    "feed_url": "https://www.36kr.com/rss-center",
+                    "feed_urls": ["https://www.36kr.com/rss-center"],
+                    "feeds": [
+                        {
+                            "id": "feed-1",
+                            "name": "36氪",
+                            "url": "https://www.36kr.com/rss-center",
+                            "type": "rss",
+                            "enabled": True,
+                            "default_category": "business",
+                        }
+                    ],
+                    "default_category": "business",
+                }
+            ]
+        }
+    )
+
+    source = payload["sources"][0]
+    feed_urls = {feed["url"] for feed in source["feeds"]}
+
+    assert source["homepage_url"] == "https://www.36kr.com/"
+    assert source["feed_directory_url"] == "https://www.36kr.com/rss-center"
+    assert "https://www.36kr.com/rss-center" not in feed_urls
+    assert {
+        "https://36kr.com/feed",
+        "https://36kr.com/feed-article",
+        "https://36kr.com/feed-newsflash",
+        "https://36kr.com/feed-moment",
+    }.issubset(feed_urls)
+
+
+def test_builtin_news_source_without_feed_children_gets_explicit_defaults():
+    payload = normalize_earth_news_sources_payload(
+        {
+            "sources": [
+                {
+                    "id": "ebrun",
+                    "name": "亿邦动力",
+                    "region": "asia-pacific",
+                    "source_type": "rss",
+                    "homepage_url": "https://www.ebrun.com/",
+                    "feed_url": "https://www.ebrun.com/rss/news_b2c.xml",
+                    "feed_urls": ["https://www.ebrun.com/rss/news_b2c.xml"],
+                    "default_category": "ecommerce",
+                }
+            ]
+        }
+    )
+
+    source = payload["sources"][0]
+    feed_urls = {feed["url"] for feed in source["feeds"]}
+
+    assert source["feed_directory_url"] == "https://www.ebrun.com/rss/"
+    assert "https://www.ebrun.com/rss/" not in feed_urls
+    assert {
+        "https://www.ebrun.com/rss/news_b2c.xml",
+        "https://www.ebrun.com/rss/news_b2b.xml",
+        "https://www.ebrun.com/rss/news_retail.xml",
+        "https://www.ebrun.com/rss/news_o2o.xml",
+        "https://www.ebrun.com/rss/news_service.xml",
+        "https://www.ebrun.com/rss/news_data.xml",
+        "https://www.ebrun.com/rss/news_policy.xml",
+    }.issubset(feed_urls)
+
+
+def test_builtin_fetchable_source_saved_as_reference_is_repaired():
+    payload = normalize_earth_news_sources_payload(
+        {
+            "sources": [
+                {
+                    "id": "stats-china-online-retail",
+                    "name": "国家统计局数据发布",
+                    "region": "asia-pacific",
+                    "source_type": "reference",
+                    "enabled": False,
+                    "homepage_url": "https://www.stats.gov.cn/sj/zxfb/",
+                    "feed_url": "https://www.stats.gov.cn/sj/zxfb/",
+                    "default_category": "ecommerce",
+                }
+            ]
+        }
+    )
+
+    source = payload["sources"][0]
+
+    assert source["source_type"] == "rss"
+    assert source["enabled"] is True
+    assert source["priority"] == 19
+    assert source["source_tags"] == ["official_data", "ecommerce", "retail", "china"]
+    assert source["default_category"] == "ecommerce"
+    assert source["importance_weight"] == 36
+    assert source["feed_directory_url"] == ""
+    assert source["feeds"] == [
+        {
+            "id": "release",
+            "name": "数据发布",
+            "url": "https://www.stats.gov.cn/sj/zxfb/rss.xml",
+            "type": "rss",
+            "region": "asia-pacific",
+            "enabled": True,
+            "default_category": "ecommerce",
+            "tags": [],
+            "priority": 1,
+        }
+    ]
+
+
+def test_legacy_google_sources_merge_into_google_news_source():
+    payload = normalize_earth_news_sources_payload(
+        {
+            "sources": [
+                {
+                    "id": "global-scan",
+                    "name": "Global Monitor / World",
+                    "region": "global",
+                    "source_type": "aggregated",
+                    "feed_url": "https://news.google.com/rss/search?q=world",
+                    "homepage_url": "https://news.google.com/",
+                },
+                {
+                    "id": "google-europe",
+                    "name": "Global Monitor / Europe",
+                    "region": "europe",
+                    "source_type": "aggregated",
+                    "feed_url": "https://news.google.com/rss/search?q=europe",
+                    "homepage_url": "https://news.google.com/",
+                },
+            ]
+        }
+    )
+
+    sources_by_id = {source["id"]: source for source in payload["sources"]}
+
+    assert "global-scan" not in sources_by_id
+    assert "google-europe" not in sources_by_id
+    assert "google-news" in sources_by_id
+    assert {feed["id"] for feed in sources_by_id["google-news"]["feeds"]} == {
+        "world",
+        "americas",
+        "europe",
+        "middle-east-africa",
+        "asia-pacific",
+    }
+
+
+def test_feed_child_default_category_overrides_source_default():
+    source = NewsFeedSource(
+        id="multi-feed",
+        name="Multi Feed",
+        region="global",
+        feed_url="https://example.com/source.xml",
+        homepage_url="https://example.com",
+        default_category="business",
+    )
+    feed = NewsFeedEndpoint(
+        id="ecommerce-feed",
+        name="Ecommerce Feed",
+        url="https://example.com/ecommerce.xml",
+        default_category="ecommerce",
+    )
+    xml = """
+    <rss>
+      <channel>
+        <item>
+          <title>Quarterly results released</title>
+          <description>Company update.</description>
+          <link>https://example.com/results</link>
+        </item>
+      </channel>
+    </rss>
+    """
+
+    items = _parse_feed_entries(xml, source, feed=feed)
+
+    assert items[0].feed_id == "ecommerce-feed"
+    assert items[0].feed_name == "Ecommerce Feed"
+    assert items[0].feed_default_category == "ecommerce"
+    assert items[0].category == "ecommerce"
+
+
+@pytest.mark.asyncio
+async def test_fetch_source_only_requests_enabled_feed_children(monkeypatch):
+    source = NewsFeedSource(
+        id="multi-feed",
+        name="Multi Feed",
+        region="global",
+        feed_url="https://example.com/source.xml",
+        homepage_url="https://example.com",
+        feeds=(
+            NewsFeedEndpoint(id="enabled", name="Enabled", url="https://example.com/enabled.xml", enabled=True),
+            NewsFeedEndpoint(id="disabled", name="Disabled", url="https://example.com/disabled.xml", enabled=False),
+        ),
+    )
+    calls = []
+
+    async def fake_fetch_single(_client, feed_source, feed, *, config_payload=None):
+        calls.append(feed.id)
+        item = ParsedNewsItem(
+            id=f"{feed_source.id}:{feed.id}:1",
+            title="Fetched story",
+            summary="Fetched summary",
+            url=f"https://example.com/{feed.id}",
+            source="Example",
+            feed_name=feed.name,
+            feed_region="global",
+            homepage_url="https://example.com",
+            published_at=None,
+            feed_id=feed.id,
+        )
+        return feed_source, [item], None, {"source_id": feed_source.id, "feed_id": feed.id, "ok": True, "status": "ok", "item_count": 1, "count": 1}
+
+    monkeypatch.setattr("app.services.earth_news._fetch_single_feed_url", fake_fetch_single)
+
+    source_result, items, error, health = await _fetch_source(object(), source)
+
+    assert source_result.id == "multi-feed"
+    assert calls == ["enabled"]
+    assert error is None
+    assert [item.feed_id for item in items] == ["enabled"]
+    assert health["ok"] is True
+    assert [result["feed_id"] for result in health["feed_results"]] == ["enabled"]
+
+
+@pytest.mark.asyncio
+async def test_fetch_source_filters_google_feed_children_by_active_region(monkeypatch):
+    source = NewsFeedSource(
+        id="google-news",
+        name="Google News",
+        region="global",
+        feed_url="https://news.google.com/rss",
+        homepage_url="https://news.google.com/",
+        source_type="aggregated",
+        feeds=(
+            NewsFeedEndpoint(id="world", name="全球", url="https://example.com/world.xml", type="aggregated", region="global"),
+            NewsFeedEndpoint(id="europe", name="欧洲", url="https://example.com/europe.xml", type="aggregated", region="europe"),
+            NewsFeedEndpoint(id="americas", name="美洲", url="https://example.com/americas.xml", type="aggregated", region="americas"),
+        ),
+    )
+    calls = []
+
+    async def fake_fetch_single(_client, feed_source, feed, *, config_payload=None):
+        calls.append(feed.id)
+        item = ParsedNewsItem(
+            id=f"{feed_source.id}:{feed.id}:1",
+            title=f"{feed.name} headline",
+            summary="Fetched summary",
+            url=f"https://example.com/{feed.id}",
+            source="Example",
+            feed_name=feed.name,
+            feed_region=feed.region,
+            homepage_url="https://example.com",
+            published_at=None,
+            feed_id=feed.id,
+        )
+        return feed_source, [item], None, {"source_id": feed_source.id, "feed_id": feed.id, "ok": True, "status": "ok", "item_count": 1, "count": 1}
+
+    monkeypatch.setattr("app.services.earth_news._fetch_single_feed_url", fake_fetch_single)
+
+    _source_result, items, error, health = await _fetch_source(object(), source, active_region="europe")
+
+    assert error is None
+    assert calls == ["world", "europe"]
+    assert [item.feed_region for item in items] == ["global", "europe"]
+    assert [result["feed_id"] for result in health["feed_results"]] == ["world", "europe"]
+
+
+def test_parse_rdf_rss_items_with_namespaces():
+    source = NewsFeedSource(
+        id="dw-top",
+        name="DW Top Stories",
+        region="europe",
+        feed_url="https://rss.dw.com/rdf/rss-en-top",
+        homepage_url="https://www.dw.com/en/top-stories/s-9097",
+    )
+    xml = """
+    <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+             xmlns="http://purl.org/rss/1.0/">
+      <item rdf:about="https://example.com/dw">
+        <title>German retail sales rise</title>
+        <link>https://example.com/dw</link>
+        <description>Retail summary</description>
+      </item>
+    </rdf:RDF>
+    """
+
+    items = _parse_feed_entries(xml, source)
+
+    assert len(items) == 1
+    assert items[0].title == "German retail sales rise"
+
+
+def test_news_classification_marks_ecommerce_and_importance():
+    source = NewsFeedSource(
+        id="ebrun",
+        name="亿邦动力",
+        region="asia-pacific",
+        feed_url="https://www.ebrun.com/rss/",
+        homepage_url="https://www.ebrun.com/",
+        source_tags=("business_news", "ecommerce", "china"),
+        default_category="ecommerce",
+        importance_weight=14,
+    )
+    item = ParsedNewsItem(
+        id="ebrun:test",
+        title="跨境电商平台 GMV 同比增长，物流履约效率提升",
+        summary="订单量和网上零售额继续增长。",
+        url="https://example.com/ecommerce",
+        source="亿邦动力",
+        feed_name="亿邦动力",
+        feed_region="asia-pacific",
+        homepage_url="https://www.ebrun.com/",
+        published_at=None,
+    )
+
+    apply_news_classification(item, source)
+
+    assert item.category == "ecommerce"
+    assert "cross_border_ecommerce" in item.item_tags
+    assert "logistics_fulfillment" in item.item_tags
+    assert item.importance_level in {"high", "critical"}
+    assert "命中电商数据指标" in item.importance_reasons
+
+
 @pytest.mark.asyncio
 async def test_enrich_items_with_target_locations_uses_ai_and_geocode(monkeypatch):
     item = ParsedNewsItem(
@@ -338,8 +1009,8 @@ async def test_earth_news_payload_returns_anchor_items_and_enqueues_location_job
         published_at=datetime(2026, 5, 15, 3, 0, tzinfo=UTC),
     )
 
-    async def fake_fetch_source(_client, feed_source):
-        return feed_source, [item], None
+    async def fake_fetch_source(_client, feed_source, **_kwargs):
+        return feed_source, [item], None, {"source_id": feed_source.id, "ok": True, "status": "ok", "count": 1}
 
     async def fake_get_cached_target_location_patch(_item_id):
         return None
@@ -402,9 +1073,11 @@ async def test_earth_news_payload_uses_fresh_database_items_without_rss(monkeypa
     async def fake_get_earth_news_freshness(_db, *, active_region):
         return 12, datetime.now(UTC)
 
-    async def fake_list_earth_news_items(_db, *, active_region, limit):
-        assert limit == 12
-        return [item]
+    async def fake_list_earth_news_items(_db, *, active_region, limit, categories=None, source_ids=None):
+        if source_ids is None:
+            assert limit == 12
+            return [item]
+        return []
 
     async def fail_fetch(_sources):
         raise AssertionError("fresh database items should not fetch RSS")
@@ -421,6 +1094,148 @@ async def test_earth_news_payload_uses_fresh_database_items_without_rss(monkeypa
     assert payload["stale"] is False
 
 
+@pytest.mark.asyncio
+async def test_earth_news_payload_keeps_current_items_and_all_cruise_items(monkeypatch):
+    class FakeDb:
+        execute = object()
+
+    current_item = ParsedNewsItem(
+        id="db:current",
+        title="Current region story",
+        summary="Current summary",
+        url="https://example.com/current",
+        source="Stored Source",
+        feed_name="Stored Feed",
+        feed_region="americas",
+        homepage_url="https://example.com",
+        published_at=datetime(2026, 5, 15, 3, 0, tzinfo=UTC),
+    )
+    cruise_item = ParsedNewsItem(
+        id="db:apac",
+        title="APAC story",
+        summary="APAC summary",
+        url="https://example.com/apac",
+        source="Stored Source",
+        feed_name="Stored Feed",
+        feed_region="asia-pacific",
+        homepage_url="https://example.com",
+        published_at=datetime(2026, 5, 15, 4, 0, tzinfo=UTC),
+    )
+
+    async def fake_get_earth_news_freshness(_db, *, active_region):
+        return 12, datetime.now(UTC)
+
+    async def fake_list_earth_news_items(_db, *, active_region, limit, categories=None):
+        return [current_item]
+
+    async def fake_list_earth_news_cruise_items(_db, *, limit, categories=None):
+        return [current_item, cruise_item]
+
+    async def fake_enqueue_target_location_job(_payload, **_kwargs):
+        return True
+
+    async def fail_fetch(_sources):
+        raise AssertionError("fresh database items should not fetch RSS")
+
+    monkeypatch.setattr("app.services.earth_news_store.get_earth_news_freshness", fake_get_earth_news_freshness)
+    monkeypatch.setattr("app.services.earth_news_store.list_earth_news_items", fake_list_earth_news_items)
+    monkeypatch.setattr("app.services.earth_news_store.list_earth_news_cruise_items", fake_list_earth_news_cruise_items)
+    monkeypatch.setattr("app.services.earth_news_queue.enqueue_target_location_job", fake_enqueue_target_location_job)
+    monkeypatch.setattr("app.services.earth_news._fetch_rss_items_for_sources", fail_fetch)
+
+    payload = await get_earth_news_payload(lat=35.0, lon=-100.0, db=FakeDb())
+
+    assert [item["id"] for item in payload["items"]] == ["db:current"]
+    assert [item["id"] for item in payload["cruise_items"]] == ["db:current", "db:apac"]
+    assert payload["cruise_items"][1]["region"] == "asia-pacific"
+
+
+@pytest.mark.asyncio
+async def test_earth_news_payload_passes_region_and_category_filters_to_store(monkeypatch):
+    class FakeDb:
+        execute = object()
+
+    captured = {}
+    item = ParsedNewsItem(
+        id="db:business",
+        title="Business story",
+        summary="Business summary",
+        url="https://example.com/business",
+        source="Stored Source",
+        feed_name="Stored Feed",
+        feed_region="europe",
+        homepage_url="https://example.com",
+        published_at=datetime(2026, 5, 15, 3, 0, tzinfo=UTC),
+        category="business",
+    )
+
+    async def fake_get_earth_news_freshness(_db, *, active_region):
+        captured["freshness_region"] = active_region
+        return 12, datetime.now(UTC)
+
+    async def fake_list_earth_news_items(_db, *, active_region, limit, categories=None, source_ids=None):
+        captured["items_region"] = active_region
+        captured["items_categories"] = categories
+        captured.setdefault("items_source_ids", []).append(source_ids)
+        return [item] if source_ids is None else []
+
+    async def fake_list_earth_news_cruise_items(_db, *, limit, categories=None, source_ids=None):
+        captured["cruise_categories"] = categories
+        captured["cruise_source_ids"] = source_ids
+        return [item]
+
+    async def fake_enqueue_target_location_job(_payload, **_kwargs):
+        return True
+
+    monkeypatch.setattr("app.services.earth_news_store.get_earth_news_freshness", fake_get_earth_news_freshness)
+    monkeypatch.setattr("app.services.earth_news_store.list_earth_news_items", fake_list_earth_news_items)
+    monkeypatch.setattr("app.services.earth_news_store.list_earth_news_cruise_items", fake_list_earth_news_cruise_items)
+    monkeypatch.setattr("app.services.earth_news_queue.enqueue_target_location_job", fake_enqueue_target_location_job)
+    monkeypatch.setattr("app.services.earth_news._fetch_rss_items_for_sources", lambda _sources: (_ for _ in ()).throw(AssertionError("fresh database items should not fetch RSS")))
+
+    payload = await get_earth_news_payload(
+        lat=35.0,
+        lon=-100.0,
+        region="europe",
+        categories={"business", "ecommerce"},
+        db=FakeDb(),
+    )
+
+    assert captured["freshness_region"] == "europe"
+    assert captured["items_region"] == "europe"
+    assert captured["items_categories"] == {"business", "ecommerce"}
+    assert captured["items_source_ids"][0] is None
+    assert any(source_ids for source_ids in captured["items_source_ids"][1:])
+    assert captured["cruise_categories"] == {"business", "ecommerce"}
+    assert captured["cruise_source_ids"] is None
+    assert payload["filters"] == {
+        "region": "europe",
+        "categories": ["business", "ecommerce"],
+        "sources": [],
+        "limit": 12,
+        "locale": "zh-CN",
+        "has_breaking": False,
+        "highest_breaking_level": "none",
+    }
+    assert payload["items"][0]["category"] == "business"
+
+
+@pytest.mark.asyncio
+async def test_news_source_test_treats_type_reference_as_non_fetching():
+    result = await run_news_source_config_test(
+        {
+            "id": "reference-only",
+            "name": "Reference Only",
+            "type": "reference",
+            "feed_url": "https://example.com",
+        }
+    )
+
+    assert result["ok"] is False
+    assert result["health"]["status"] == "reference"
+    assert "不参与 RSS/Atom 抓取" in result["error"]
+
+
 @pytest.mark.asyncio
 async def test_earth_news_payload_initializes_empty_database_from_rss(monkeypatch):
     db = object()
@@ -456,14 +1271,14 @@ async def test_earth_news_payload_initializes_empty_database_from_rss(monkeypatc
     async def fake_get_earth_news_freshness(_db, *, active_region):
         return 0, None
 
-    async def fake_fetch_rss_items_for_sources(_sources):
-        return [item], []
+    async def fake_fetch_rss_items_for_sources(_sources, **_kwargs):
+        return [item], [], {"test-feed": {"source_id": "test-feed", "ok": True, "status": "ok", "count": 1}}
 
     async def fake_upsert_earth_news_items(_db, items):
         upserted.extend(items)
         return len(items)
 
-    async def fake_list_earth_news_items(_db, *, active_region, limit):
+    async def fake_list_earth_news_items(_db, *, active_region, limit, categories=None):
         return [item]
 
     async def fake_enqueue_target_location_job(payload, **_kwargs):
@@ -512,14 +1327,14 @@ async def test_earth_news_payload_supplements_stale_database_items(monkeypatch):
     async def fake_get_earth_news_freshness(_db, *, active_region):
         return 12, datetime(2026, 5, 14, 3, 0, tzinfo=UTC)
 
-    async def fake_fetch_rss_items_for_sources(_sources):
+    async def fake_fetch_rss_items_for_sources(_sources, **_kwargs):
         fetched.append(True)
-        return [old_item], []
+        return [old_item], [], {"stored": {"source_id": "stored", "ok": True, "status": "ok", "count": 1}}
 
     async def fake_upsert_earth_news_items(_db, items):
         return len(items)
 
-    async def fake_list_earth_news_items(_db, *, active_region, limit):
+    async def fake_list_earth_news_items(_db, *, active_region, limit, categories=None):
         return [old_item]
 
     async def fake_enqueue_target_location_job(_payload, **_kwargs):
@@ -574,8 +1389,8 @@ async def test_earth_news_payload_merges_cached_location_patch(monkeypatch):
         },
     }
 
-    async def fake_fetch_source(_client, feed_source):
-        return feed_source, [item], None
+    async def fake_fetch_source(_client, feed_source, **_kwargs):
+        return feed_source, [item], None, {"source_id": feed_source.id, "ok": True, "status": "ok", "count": 1}
 
     async def fake_get_cached_target_location_patch(_item_id):
         return cached_patch
@@ -641,8 +1456,8 @@ async def test_earth_news_payload_requeues_cached_failed_localization(monkeypatc
     }
     enqueued = []
 
-    async def fake_fetch_source(_client, feed_source):
-        return feed_source, [item], None
+    async def fake_fetch_source(_client, feed_source, **_kwargs):
+        return feed_source, [item], None, {"source_id": feed_source.id, "ok": True, "status": "ok", "count": 1}
 
     async def fake_get_cached_target_location_patch(_item_id):
         return cached_patch