Compare commits

...
3 Commits
Author SHA1 Message Date
linkong 3265d22af5 release: bump version to 0.71.1 2026-06-26 17:34:19 +08:00
linkong 899e3bce43 release: bump version to 0.71.0 2026-06-11 16:47:24 +08:00
linkong 8c204717cd release: bump version to 0.70.0 2026-06-04 17:16:23 +08:00
125 changed files with 7415 additions and 1017 deletions

No files matched your search

+4 -1
View File
@@ -25,7 +25,10 @@ __pycache__/
build/
develop-eggs/
dist/
downloads/
downloads/*
!downloads/usbipd-win/
downloads/usbipd-win/*
!downloads/usbipd-win/usbipd-win-5.3.0.msi
eggs/
.eggs/
/lib/
+81
View File
@@ -0,0 +1,81 @@
# Planet Agent Entry Point
This is the compatibility entry point for coding agents. The older root
`agents.md` file remains authoritative for repository-specific agent behavior;
do not delete or replace it.
## Read First
Read these files before changing code:
1. `rules.md` - mandatory repository rules. Always load `core`, `security`, and
`workflow`; load `docs`, `frontend`, `backend`, `earth`, `ai`, or `release`
when the task touches those areas.
2. `agents.md` - existing agent role, communication, and workflow guidance.
3. `project_context.md` - static project background. Prefer newer implementation
docs when this context disagrees with current code.
4. `README.md` - current architecture, startup, and toolchain summary.
5. `docs/HARNESS.md` - harness workflow, conflict policy, and validation tiers.
6. `CODEMAP.md` - codebase entry points, ownership boundaries, and deeper docs.
For documentation work, also read `docs/documentation-coverage-rules.md`.
## Start Safely
Before editing:
```bash
git status --short
scripts/harness/doctor.sh
```
Use focused context commands before reading large files:
```bash
rg -n "<symbol-or-term>" <path>
git diff --stat HEAD
git diff --name-only HEAD
git diff --unified=0 HEAD -- <path>
```
Preserve user changes already present in the worktree.
## Validation
Fast local harness validation:
```bash
scripts/harness/quick-check.sh
```
Full local validation:
```bash
scripts/harness/validate.sh
```
`validate.sh` includes the quick check and the frontend Bun build. Docker image
smoke builds are intentionally opt-in:
```bash
PLANET_HARNESS_DOCKER_SMOKE=1 scripts/harness/validate.sh
```
## High-Risk Areas
- `planet.sh` owns local lifecycle, ports, WSL/LAN behavior, and destructive
`destroy` cleanup.
- Frontend package management is Bun-only. Do not use npm, pnpm, or yarn.
- `aiprovider` is a protocol/provider adapter; keep business prompts and product
workflows in the backend.
- Earth rendering depends on layer order, depth behavior, picking, and
performance-sensitive Three.js code.
- Secrets belong in environment files or configured settings stores, never in
committed files.
## Conflict Policy
Existing project rules and workflows win. If new harness guidance conflicts with
`rules.md`, `agents.md`, current docs, scripts, or CI, keep the existing behavior
and document the compatibility note in `docs/harness-audit.md` or
`docs/HARNESS.md`.
+93
View File
@@ -0,0 +1,93 @@
# Code Map
This map gives agents and maintainers a quick orientation without replacing the
deeper architecture docs. Current implementation docs under `docs/technical/`
are the source of detail for specific subsystems.
## Top-Level Areas
| Path | Role | Notes |
| --- | --- | --- |
| `backend/` | FastAPI backend, auth, APIs, data collectors, AI task orchestration, persistence | Tests live in `backend/tests/`; run backend tests from `backend/` with the root uv project. |
| `frontend/` | React admin console, Docs UI, Web Earth shell, Vite build | Use Bun only. Public Earth assets live under `frontend/public/earth/`. |
| `aiprovider/` | Model provider/protocol adapter service | Keep it free of product-specific prompts and workflows. |
| `motion_agent/` | Motion capture protocol service used by `planet.sh` | Often dry-runs when cameras are unavailable, especially in WSL. |
| `scripts/` | Utility scripts and harness wrappers | Harness commands live in `scripts/harness/`. |
| `docs/` | Plans, technical docs, changelog, harness docs | Public technical docs are explicitly registered by the frontend Docs catalog. |
| `deploy/helm/planet/` | Helm chart for staging/deployment smoke paths | CI runs helm lint/template when delivery checks are available. |
| `.gitea/workflows/` | CI, release image build, staging deploy workflows | This repository uses Gitea workflow files, not `.github/workflows/`. |
| `planet.sh` | Main local lifecycle script | Owns init/start/restart/stop/health/log/createuser/destroy. |
## Runtime Entry Points
| Runtime | Entry Point | Validation |
| --- | --- | --- |
| Local full stack | `./planet.sh start` | `./planet.sh health` |
| Backend API | `backend/app/main.py` | `cd backend && uv run --frozen --group dev --project .. python -m pytest -q` |
| Frontend app | `frontend/src/main.tsx` and `frontend/vite.config.mts` | `cd frontend && bun run build` |
| AI Provider | `aiprovider/main.py` | `curl http://localhost:8010/health` after startup |
| Motion Agent | `python -m motion_agent` via `planet.sh` | `./planet.sh health` or dry-run startup |
| Docs UI | `frontend/src/pages/Docs/` | Docs catalog metadata plus frontend build |
## Ownership Boundaries
- Backend owns business state, auth, evidence collection, prompt selection, AI
task orchestration, and database persistence.
- `aiprovider` owns provider identity, request adapter style, model gateway
retries, and health/status endpoints only.
- Frontend owns operator workflows, Docs presentation, Web Earth orchestration,
and client-side state that mirrors backend truth.
- Web Earth rendering changes must preserve documented layer order, altitude
offsets, picking behavior, legend semantics, and performance constraints.
- `planet.sh` owns local environment bootstrap and service lifecycle. Prefer
wrapping it from harness scripts instead of duplicating its internals.
## Validation Commands
```bash
scripts/harness/doctor.sh
scripts/harness/quick-check.sh
scripts/harness/validate.sh
./planet.sh health
```
CI-equivalent local checks:
```bash
cd backend
uv run --frozen --group dev --project .. python -m pytest -s tests/test_api.py tests/test_realtime_sources.py -q
cd frontend
bun install --frozen-lockfile
bun run build
```
Optional delivery smoke, when Docker and Helm are available:
```bash
PLANET_HARNESS_DOCKER_SMOKE=1 scripts/harness/validate.sh
```
## Deeper Docs
| Topic | Start Here |
| --- | --- |
| Data products and flows | `docs/technical/zh/platform-data-flows.md` and `docs/technical/en/platform-data-flows.md` |
| Operations and local lifecycle | `docs/technical/zh/ops-runbook.md` and `docs/technical/en/ops-runbook.md` |
| `planet.sh` startup behavior | `docs/technical/zh/ops-planet-sh-startup.md` and `docs/technical/en/ops-planet-sh-startup.md` |
| AI Provider | `docs/technical/zh/agents-aiprovider.md` and `docs/technical/en/agents-aiprovider.md` |
| Admin frontend | `docs/technical/zh/frontend-admin-frontend-context.md` and `docs/technical/en/frontend-admin-frontend-context.md` |
| Earth frontend | `docs/technical/zh/earth-frontend-context.md` and `docs/technical/en/earth-frontend-context.md` |
| Earth render order | `docs/technical/zh/earth-render-layer-order.md` and `docs/technical/en/earth-render-layer-order.md` |
| Documentation rules | `docs/documentation-coverage-rules.md` |
| Harness workflow | `docs/HARNESS.md` |
## Known Sharp Edges
- `project_context.md` includes older roadmap-era assumptions such as Celery,
Kafka, TimescaleDB, MinIO, and UE5 being part of the active local stack. Treat
it as background unless current README/docs/code confirm the same behavior.
- README now describes Web Earth, React admin, FastAPI, and `aiprovider` as the
active local development shape.
- Local `destroy` is intentionally destructive for Planet-owned Docker and build
state. Never run it as a validation shortcut.
+1
View File
@@ -4,6 +4,7 @@ This file is the active backlog only. Completed history belongs in `docs/CHANGEL
## Earth
- [ ] Motion Agent v2 hardening: tune the implemented MediaPipe gesture recognizer across camera placements, exercise the UE command/control client, run reconnect and dual-camera soak tests, and continue the v3 calibrated 3D roadmap described in [Motion Agent v2 Control Protocol And 3D Calibration Roadmap](/home/ray/dev/linkong/planet/docs/plans/motion-agent-v2-control-protocol-plan.md).
- [ ] Earth AI command entry: merge natural-language and speech-triggered LLM commands into the existing Earth search panel as described in [Agent Runtime, Earth LLM Command, And Speech Entry Plan](/home/ray/dev/linkong/planet/docs/plans/agents-earth-command-runtime-plan.md).
- [ ] Earth action executor: implement safe visualization actions for layer toggles, batch highlights, filters, focus, result panels, and clear-highlight behavior.
- [ ] Earth entity matching: support stable entity ids and batch matching for Beidou satellites, mainland China compute centers, BGP, news, vessels, and cables.
+1 -1
View File
@@ -1 +1 @@
0.69.0
0.71.1
+17
View File
@@ -4,6 +4,23 @@
---
## Harness Compatibility
Common agent tools should start at `AGENTS.md`. This file remains the existing
behavior guide and must not be replaced by harness docs. For safe repository
orientation, use:
- `rules.md` for mandatory project rules
- `project_context.md` for static background
- `docs/HARNESS.md` for validation tiers and conflict policy
- `CODEMAP.md` for subsystem entry points and ownership boundaries
- `docs/harness-audit.md` for the latest harness compatibility notes
Existing project rules and workflows stay authoritative when they conflict with
new harness guidance.
---
## Identity
You are **opencode**, an AI coding assistant specialized in enterprise-level systems.
+19 -9
View File
@@ -4,6 +4,7 @@ from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy import text
from app.core.config import settings
from app.core.enums import OtpPurpose, UserRole
from app.core.logging import get_logger
from app.core.security import (
create_access_token,
@@ -170,7 +171,7 @@ async def get_me(current_user: User = Depends(get_current_user)):
}
async def _send_code_or_raise(db: AsyncSession, email: str, code: str, purpose: str) -> None:
async def _send_code_or_raise(db: AsyncSession, email: str, code: str, purpose: OtpPurpose) -> None:
try:
await send_verification_email(db, to=email, code=code, purpose=purpose)
except EmailNotConfiguredError as exc:
@@ -207,7 +208,7 @@ async def register(payload: UserRegister, db: AsyncSession = Depends(get_db)):
username=payload.username,
email=payload.email,
password_hash=get_password_hash(payload.password),
role="viewer",
role=UserRole.VIEWER.value,
is_active=True,
email_verified=False,
)
@@ -215,13 +216,13 @@ async def register(payload: UserRegister, db: AsyncSession = Depends(get_db)):
await db.commit()
try:
code = otp.issue_code(payload.email, "register")
code = otp.issue_code(payload.email, OtpPurpose.REGISTER)
except otp.OtpResendRateLimited as exc:
raise HTTPException(
status_code=status.HTTP_429_TOO_MANY_REQUESTS,
detail={"code": exc.code, "retry_after_seconds": exc.retry_after_seconds},
) from exc
await _send_code_or_raise(db, payload.email, code, "register")
await _send_code_or_raise(db, payload.email, code, OtpPurpose.REGISTER)
return {"status": "pending_verification", "email": payload.email}
@@ -266,7 +267,7 @@ async def resend_code(payload: ResendCodeRequest, db: AsyncSession = Depends(get
if user is None:
# Avoid email enumeration; pretend success.
return {"status": "ok"}
if payload.purpose == "register" and user.email_verified:
if payload.purpose is OtpPurpose.REGISTER and user.email_verified:
raise HTTPException(
status_code=status.HTTP_409_CONFLICT,
detail={"code": "ALREADY_VERIFIED"},
@@ -289,12 +290,17 @@ async def forgot_password(payload: ForgotPasswordRequest, db: AsyncSession = Dep
# Don't leak whether an email is registered.
return {"status": "ok"}
try:
code = otp.issue_code(payload.email, "reset_password")
code = otp.issue_code(payload.email, OtpPurpose.RESET_PASSWORD)
except otp.OtpResendRateLimited:
# Silently accept; the user can retry after the cooldown.
return {"status": "ok"}
try:
await send_verification_email(db, to=payload.email, code=code, purpose="reset_password")
await send_verification_email(
db,
to=payload.email,
code=code,
purpose=OtpPurpose.RESET_PASSWORD,
)
except EmailNotConfiguredError as exc:
raise HTTPException(
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
@@ -304,7 +310,11 @@ async def forgot_password(payload: ForgotPasswordRequest, db: AsyncSession = Dep
logger.warning_event(
"SMTP send failed",
event="auth.email.send_failed",
context={"email": payload.email, "purpose": "reset_password", "error": str(exc)},
context={
"email": payload.email,
"purpose": OtpPurpose.RESET_PASSWORD.value,
"error": str(exc),
},
)
return {"status": "ok"}
@@ -318,7 +328,7 @@ async def reset_password(payload: ResetPasswordRequest, db: AsyncSession = Depen
detail={"code": "OTP_INVALID"},
)
try:
otp.verify_code(payload.email, "reset_password", payload.code)
otp.verify_code(payload.email, OtpPurpose.RESET_PASSWORD, payload.code)
except otp.OtpExpired as exc:
raise HTTPException(
status_code=status.HTTP_410_GONE,
+8 -8
View File
@@ -13,6 +13,7 @@ import httpx
from app.core.target_schema_registry import get_target_schema, list_target_schemas
from app.core.datasource_defaults import DEFAULT_DATASOURCES
from app.core.enums import AuthType, MappingValidationStatus, UserRole
from app.db.session import get_db
from app.models.user import User
from app.models.datasource_config import DataSourceConfig
@@ -34,7 +35,6 @@ from app.services.datasource_mapping import (
)
from app.services.custom_datasource_runtime import (
CustomDatasourceRuntimeError,
fetch_rest_payload,
get_custom_stream_status,
run_mapped_rest_config,
run_mapped_websocket_config,
@@ -42,8 +42,6 @@ from app.services.custom_datasource_runtime import (
stop_custom_stream,
test_websocket_config,
)
DATASOURCE_MAPPING_PROMPT_KEY = "datasource.mapping"
from app.services.datasource_connectivity import (
_resolve_aisstream_api_key,
_resolve_spacetrack_credentials_with_override,
@@ -57,7 +55,8 @@ from app.services.persistent_logs import record_audit_log
router = APIRouter()
SECRET_REVEAL_ROLES = {"admin", "super_admin"}
DATASOURCE_MAPPING_PROMPT_KEY = "datasource.mapping"
SECRET_REVEAL_ROLES = {UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value}
def _user_role_value(user: User) -> str:
@@ -124,7 +123,7 @@ class DataSourceConfigCreate(BaseModel):
description: Optional[str] = None
source_type: str = Field(..., description="rest, websocket, http, api, database")
endpoint: str = Field(..., max_length=500)
auth_type: str = Field(default="none", description="none, bearer, api_key, basic")
auth_type: AuthType = Field(default=AuthType.NONE, description="none, bearer, api_key, basic")
auth_config: dict = Field(default={})
headers: dict = Field(default={})
config: dict = Field(default={"timeout": 30, "retry": 3})
@@ -135,7 +134,7 @@ class DataSourceConfigUpdate(BaseModel):
description: Optional[str] = None
source_type: Optional[str] = None
endpoint: Optional[str] = Field(None, max_length=500)
auth_type: Optional[str] = None
auth_type: Optional[AuthType] = None
auth_config: Optional[dict] = None
headers: Optional[dict] = None
config: Optional[dict] = None
@@ -210,7 +209,7 @@ class MappingTemplateCreate(BaseModel):
mapping_json: dict
sample_payload: Any | None = None
sample_payload_hash: Optional[str] = None
validation_status: str = Field(default="draft", pattern="^(draft|valid|invalid)$")
validation_status: MappingValidationStatus = MappingValidationStatus.DRAFT
is_active: bool = False
@@ -219,7 +218,7 @@ class MappingTemplateUpdate(BaseModel):
mapping_json: Optional[dict] = None
sample_payload: Any | None = None
sample_payload_hash: Optional[str] = None
validation_status: Optional[str] = Field(default=None, pattern="^(draft|valid|invalid)$")
validation_status: Optional[MappingValidationStatus] = None
is_active: Optional[bool] = None
@@ -918,6 +917,7 @@ async def get_datasource_target_schemas(
@router.post("/mappings/propose")
async def propose_datasource_mapping(
payload: MappingProposeRequest,
db: AsyncSession = Depends(get_db),
current_user: User = Depends(get_current_user),
ai_client: AIProviderClient = Depends(get_ai_provider_client),
):
+12 -11
View File
@@ -7,6 +7,7 @@ from sqlalchemy import func, or_, select, text
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.logging import get_logger
from app.core.enums import JobStatus, SnapshotStatus
from app.core.time import to_iso8601_utc
from app.core.security import get_current_user
from app.core.data_sources import get_data_sources_config
@@ -581,7 +582,7 @@ async def get_running_task(db: AsyncSession, datasource_id: int) -> Optional[Col
result = await db.execute(
select(CollectionTask)
.where(CollectionTask.datasource_id == datasource_id)
.where(CollectionTask.status == "running")
.where(CollectionTask.status == JobStatus.RUNNING.value)
.order_by(CollectionTask.started_at.desc())
.limit(1)
)
@@ -609,8 +610,8 @@ async def get_running_task(db: AsyncSession, datasource_id: int) -> Optional[Col
f"Marked failed automatically after stale running timeout "
f"({STALE_RUNNING_TASK_TIMEOUT_MINUTES}m)"
)
task.status = "failed"
task.phase = "failed"
task.status = JobStatus.FAILED.value
task.phase = JobStatus.FAILED.value
task.completed_at = now
task.error_message = f"{existing_error}\n{stale_reason}".strip() if existing_error else stale_reason
await db.commit()
@@ -647,7 +648,7 @@ async def rollback_orphaned_running_task(
)
if snapshot is not None:
snapshot.status = "cancelled"
snapshot.status = SnapshotStatus.CANCELLED.value
snapshot.is_current = False
snapshot.completed_at = datetime.now(timezone.utc)
summary = dict(snapshot.summary or {})
@@ -670,13 +671,13 @@ async def rollback_orphaned_running_task(
{"snapshot_id": snapshot.parent_snapshot_id},
)
running_task.status = "cancelled"
running_task.phase = "cancelled"
running_task.status = JobStatus.CANCELLED.value
running_task.phase = JobStatus.CANCELLED.value
running_task.completed_at = datetime.now(timezone.utc)
existing_error = (running_task.error_message or "").strip()
cancel_reason = "Cancelled after backend restart because the running task handle was lost; incomplete writes rolled back"
running_task.error_message = f"{existing_error}\n{cancel_reason}".strip() if existing_error else cancel_reason
datasource.last_status = "cancelled"
datasource.last_status = JobStatus.CANCELLED.value
datasource.last_run_at = datetime.now(timezone.utc)
await db.commit()
@@ -711,7 +712,7 @@ async def fail_and_rollback_stale_running_task(
)
if snapshot is not None:
snapshot.status = "failed"
snapshot.status = SnapshotStatus.FAILED.value
snapshot.is_current = False
snapshot.completed_at = datetime.now(timezone.utc)
summary = dict(snapshot.summary or {})
@@ -739,11 +740,11 @@ async def fail_and_rollback_stale_running_task(
f"Marked failed automatically after stale running timeout "
f"({STALE_RUNNING_TASK_TIMEOUT_MINUTES}m); incomplete writes rolled back"
)
running_task.status = "failed"
running_task.phase = "failed"
running_task.status = JobStatus.FAILED.value
running_task.phase = JobStatus.FAILED.value
running_task.completed_at = datetime.now(timezone.utc)
running_task.error_message = f"{existing_error}\n{stale_reason}".strip() if existing_error else stale_reason
datasource.last_status = "failed"
datasource.last_status = JobStatus.FAILED.value
datasource.last_run_at = datetime.now(timezone.utc)
await db.commit()
+191 -1
View File
@@ -6,7 +6,7 @@ from pathlib import Path
from typing import Any
from uuid import uuid4
from fastapi import APIRouter, Depends, File, HTTPException, Request, UploadFile, status
from fastapi import APIRouter, Depends, File, Form, HTTPException, Query, Request, UploadFile, status
from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
from pydantic import BaseModel, Field
from sqlalchemy import delete, func, select, text
@@ -27,6 +27,20 @@ from app.services.earth_news import (
save_earth_news_sources_payload,
test_news_source_config,
)
from app.services.earth_news_manual import (
broadcast_manual_news_changed,
create_manual_news_group,
delete_manual_news_item,
get_news_record_or_404,
import_manual_news_items,
list_news_groups,
list_news_records,
parse_manual_news_import_upload,
rename_manual_news_group,
reprocess_manual_news_item,
serialize_news_record,
upsert_manual_news_item,
)
from app.services.earth_boundaries import (
EarthBoundaryBuildError,
get_boundary_build_status,
@@ -119,6 +133,26 @@ class EarthNewsSourceTestPayload(BaseModel):
source: dict[str, Any] = Field(default_factory=dict)
class EarthNewsManualItemPayload(BaseModel):
title: str = Field(default="", max_length=500)
summary: str = Field(default="", max_length=1200)
content: str = Field(default="", max_length=12000)
url: str = Field(default="", max_length=2000)
source: str = Field(default="", max_length=255)
region: str = Field(default="global", max_length=80)
published_at: str | None = None
category: str = Field(default="other", max_length=80)
tags: list[str] = Field(default_factory=list)
location: dict[str, Any] | None = None
homepage_url: str = Field(default="", max_length=2000)
content_language: str = Field(default="", max_length=32)
group_id: str | None = Field(default=None, max_length=120)
class EarthNewsManualGroupPayload(BaseModel):
name: str = Field(default="", max_length=120)
def _normalize_earth_brand_payload(payload: dict[str, Any] | None) -> dict[str, str]:
merged = DEFAULT_EARTH_BRAND.copy()
if payload:
@@ -375,6 +409,162 @@ async def test_earth_news_source(
return await test_news_source_config(payload.source, db=db)
@router.get("/news-groups")
async def list_earth_news_groups_admin(
_current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
return await list_news_groups(db)
@router.post("/news-groups")
async def create_earth_news_group_admin(
payload: EarthNewsManualGroupPayload,
_current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
try:
group = await create_manual_news_group(db, payload.name)
except ValueError as exc:
raise HTTPException(status_code=422, detail=str(exc)) from exc
await db.commit()
return {"status": "ok", "group": group}
@router.put("/news-groups/{group_id:path}")
async def rename_earth_news_group_admin(
group_id: str,
payload: EarthNewsManualGroupPayload,
_current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
try:
group = await rename_manual_news_group(db, group_id, payload.name)
except ValueError as exc:
raise HTTPException(status_code=422, detail=str(exc)) from exc
await db.commit()
await broadcast_manual_news_changed()
return {"status": "ok", "group": group}
@router.get("/news-items")
async def list_earth_news_items_admin(
page: int = Query(1, ge=1),
page_size: int = Query(50, ge=1, le=100),
source_type: str | None = Query(None),
region: str | None = Query(None),
category: str | None = Query(None),
status_filter: str | None = Query(None, alias="status"),
group_id: str | None = Query(None),
_current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
return await list_news_records(
db,
page=page,
page_size=page_size,
source_type=source_type,
region=region,
category=category,
status_filter=status_filter,
group_id=group_id,
)
@router.post("/news-items")
async def create_earth_news_item_admin(
payload: EarthNewsManualItemPayload,
_current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
try:
result = await upsert_manual_news_item(db, payload.model_dump())
except ValueError as exc:
raise HTTPException(status_code=422, detail=str(exc)) from exc
except PermissionError as exc:
raise HTTPException(status_code=403, detail=str(exc)) from exc
await db.commit()
await broadcast_manual_news_changed()
return {"status": "ok", "created": result.created, "queued": result.queued, "item": serialize_news_record(result.item)}
@router.post("/news-items/import")
async def import_earth_news_items_admin(
file: UploadFile = File(...),
group_id: str | None = Form(default=None),
_current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
try:
payload = await parse_manual_news_import_upload(await file.read())
result = await import_manual_news_items(db, payload, group_id=group_id)
except ValueError as exc:
raise HTTPException(status_code=422, detail=str(exc)) from exc
await db.commit()
await broadcast_manual_news_changed()
return {"status": "ok", **result}
@router.put("/news-items/{item_id:path}")
async def update_earth_news_item_admin(
item_id: str,
payload: EarthNewsManualItemPayload,
_current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
existing = await get_news_record_or_404(db, item_id)
if existing is None:
raise HTTPException(status_code=404, detail="News item not found.")
try:
result = await upsert_manual_news_item(
db,
payload.model_dump(),
item_id_override=item_id,
)
except ValueError as exc:
raise HTTPException(status_code=422, detail=str(exc)) from exc
except PermissionError as exc:
raise HTTPException(status_code=403, detail=str(exc)) from exc
await db.commit()
await broadcast_manual_news_changed()
return {"status": "ok", "created": result.created, "queued": result.queued, "item": serialize_news_record(result.item)}
@router.delete("/news-items/{item_id:path}")
async def delete_earth_news_item_admin(
item_id: str,
_current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
try:
deleted = await delete_manual_news_item(db, item_id)
except PermissionError as exc:
raise HTTPException(status_code=403, detail=str(exc)) from exc
if not deleted:
raise HTTPException(status_code=404, detail="News item not found.")
await db.commit()
await broadcast_manual_news_changed()
return {"status": "deleted", "id": item_id}
@router.post("/news-items/{item_id:path}/reprocess")
async def reprocess_earth_news_item_admin(
item_id: str,
_current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
existing = await get_news_record_or_404(db, item_id)
if existing is None:
raise HTTPException(status_code=404, detail="News item not found.")
try:
queued = await reprocess_manual_news_item(db, item_id)
except PermissionError as exc:
raise HTTPException(status_code=403, detail=str(exc)) from exc
await db.commit()
await broadcast_manual_news_changed()
return {"status": "queued" if queued else "not_queued", "queued": queued, "id": item_id}
@router.get("/oobe-status")
async def get_earth_oobe_status(
current_user: User | None = Depends(_get_optional_current_user),
+1 -1
View File
@@ -105,7 +105,7 @@ def _parse_layer_bbox(bbox: str) -> tuple[float, float, float, float]:
@router.get("/vessels/snapshot")
async def get_vessel_layer_snapshot(
bbox: str = Query(..., description="lon_min,lat_min,lon_max,lat_max"),
zoom: int = Query(..., ge=1, le=20),
zoom: float = Query(..., ge=1, le=20),
limit: int = Query(DEFAULT_LAYER_LIMIT, ge=1),
vessel_type: Optional[str] = Query(None, alias="type"),
since_minutes: int = Query(60, ge=1, le=1440),
+20 -16
View File
@@ -12,6 +12,7 @@ from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.logging import get_logger
from app.core.enums import ProviderApi, TVSourceType, UserRole
from app.core.security import get_current_user
from app.core.time import to_iso8601_utc
from app.core.config import settings as app_settings
@@ -73,7 +74,7 @@ router = APIRouter()
logger = get_logger(__name__, service="api")
AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS = 5
AI_CONNECTION_TEST_PROMPT_KEY = "ai.connection_test"
SECRET_REVEAL_ROLES = {"admin", "super_admin"}
SECRET_REVEAL_ROLES = {UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value}
DEFAULT_SETTINGS = {
"system": {
@@ -236,7 +237,7 @@ class TVStreamSourceUpdate(BaseModel):
provider: str = Field(default="Unknown", max_length=100)
region: str = Field(default="Global", max_length=100)
language: str = Field(default="und", max_length=32)
source_type: str = Field(default="iframe", pattern="^(iframe|hls|video|external|youtube)$")
source_type: TVSourceType = TVSourceType.IFRAME
embed_url: str = ""
stream_url: str = ""
homepage_url: str = ""
@@ -279,7 +280,7 @@ class AIProviderIntegrationUpdate(BaseModel):
service_token: Optional[str] = None
default_provider: Optional[str] = None
provider: str = Field(default="minimax", max_length=80)
provider_api: str = Field(default="anthropic-messages", max_length=80)
provider_api: ProviderApi = ProviderApi.ANTHROPIC_MESSAGES
base_url: str = Field(default="", max_length=500)
model: str = Field(default="", max_length=200)
api_key: Optional[str] = None
@@ -423,7 +424,7 @@ def _get_provider_preset(provider: str) -> dict:
except ValueError:
return {
"provider": provider,
"provider_api": "openai-completions",
"provider_api": ProviderApi.OPENAI_COMPLETIONS.value,
"base_url": "",
"model": "",
"models": [],
@@ -487,12 +488,12 @@ def _provider_defaults(provider: str) -> dict:
preset = _get_provider_preset(provider)
return {
"provider": provider,
"provider_api": preset.get("provider_api") or "openai-completions",
"provider_api": preset.get("provider_api") or ProviderApi.OPENAI_COMPLETIONS.value,
"base_url": preset.get("base_url") or "",
"model": preset.get("model") or "",
"api_key": "",
"max_tokens": (
1200 if preset.get("provider_api") == "anthropic-messages" else 4096
1200 if preset.get("provider_api") == ProviderApi.ANTHROPIC_MESSAGES.value else 4096
),
"anthropic_version": "2023-06-01",
"model_provider_apis": preset.get("model_provider_apis") or {},
@@ -665,7 +666,7 @@ def _runtime_config_from_ai_payload(ai_payload: dict) -> dict:
),
"llm_config": {
"provider": default_provider,
"provider_api": provider_config.get("provider_api") or "anthropic-messages",
"provider_api": provider_config.get("provider_api") or ProviderApi.ANTHROPIC_MESSAGES.value,
"base_url": provider_config.get("base_url") or "",
"model": provider_config.get("model") or "",
"api_key": api_key,
@@ -781,7 +782,10 @@ def _contains_model(model_ids: list[str], model: str) -> bool:
async def _check_ai_provider_lightweight(llm_config: dict, timeout_seconds: int) -> dict:
provider = _normalize_provider_id(llm_config.get("provider") or "")
configured_api = str(llm_config.get("provider_api") or "").strip() or "openai-completions"
configured_api = (
str(llm_config.get("provider_api") or "").strip()
or ProviderApi.OPENAI_COMPLETIONS.value
)
model = str(llm_config.get("model") or "").strip()
base_url = str(llm_config.get("base_url") or "").strip().rstrip("/")
api_key = str(llm_config.get("api_key") or "").strip()
@@ -802,7 +806,7 @@ async def _check_ai_provider_lightweight(llm_config: dict, timeout_seconds: int)
"message": "当前 provider/base_url/model 未完整配置。",
"mode": "lightweight_config",
}
if provider_api != "ollama-generate" and not api_key:
if provider_api != ProviderApi.OLLAMA_GENERATE.value and not api_key:
return {
"success": False,
"connected": False,
@@ -813,13 +817,13 @@ async def _check_ai_provider_lightweight(llm_config: dict, timeout_seconds: int)
if provider == "opencode-go":
url = _join_provider_url(base_url, "/models")
headers = {"Authorization": f"Bearer {api_key}"}
elif provider_api == "ollama-generate":
elif provider_api == ProviderApi.OLLAMA_GENERATE.value:
url = _join_provider_url(base_url, "/api/tags")
headers: dict[str, str] = {}
elif provider_api == "openai-completions":
elif provider_api == ProviderApi.OPENAI_COMPLETIONS.value:
url = _join_provider_url(base_url, "/models")
headers = {"Authorization": f"Bearer {api_key}"}
elif provider_api == "anthropic-messages":
elif provider_api == ProviderApi.ANTHROPIC_MESSAGES.value:
url = _join_provider_url(base_url, "/models")
headers = {
"x-api-key": api_key,
@@ -1165,7 +1169,7 @@ async def serialize_external_integrations(db: AsyncSession) -> dict:
api_key, api_key_source = _resolve_provider_api_key(provider_id, provider_config)
providers_payload[provider_id] = {
"provider": provider_id,
"provider_api": provider_config.get("provider_api") or "openai-completions",
"provider_api": provider_config.get("provider_api") or ProviderApi.OPENAI_COMPLETIONS.value,
"base_url": provider_config.get("base_url") or "",
"model": provider_config.get("model") or "",
"api_key": _mask_secret(api_key, api_key_source),
@@ -1215,7 +1219,7 @@ async def serialize_external_integrations(db: AsyncSession) -> dict:
"service_token": _mask_secret(*_resolve_service_token(normalized_ai)),
"default_provider": default_provider,
"provider": default_provider,
"provider_api": display_llm_config.get("provider_api") or "anthropic-messages",
"provider_api": display_llm_config.get("provider_api") or ProviderApi.ANTHROPIC_MESSAGES.value,
"base_url": display_llm_config.get("base_url") or "https://api.minimaxi.com/anthropic",
"model": display_llm_config.get("model") or "MiniMax-M2.7",
"api_key": display_llm_config.get("api_key") or _mask_secret(None),
@@ -1458,7 +1462,7 @@ async def update_smtp_settings(
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
if current_user.role not in ("admin", "super_admin"):
if current_user.role not in (UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value):
raise HTTPException(status_code=403, detail="Only administrators can change SMTP settings")
current = await get_setting_payload(db, "smtp")
merged = _build_smtp_payload(current, payload)
@@ -1472,7 +1476,7 @@ async def test_smtp_settings(
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
if current_user.role not in ("admin", "super_admin"):
if current_user.role not in (UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value):
raise HTTPException(status_code=403, detail="Only administrators can test SMTP settings")
from app.services.email import EmailError, send_email
+10 -7
View File
@@ -5,6 +5,7 @@ from fastapi import APIRouter, Depends, HTTPException, status
from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy import text
from app.core.enums import UserRole
from app.core.security import get_current_user, get_password_hash
from app.db.session import get_db
from app.models.user import User
@@ -13,6 +14,8 @@ from app.schemas.user import UserCreate, UserUpdate
router = APIRouter()
VALID_GATEKEEPER_GROUPS = {"docs_user", "docs_developer", "docs_admin"}
ADMIN_ROLES = [UserRole.SUPER_ADMIN.value, UserRole.ADMIN.value]
SUPER_ADMIN_ROLES = [UserRole.SUPER_ADMIN.value]
def check_permission(current_user: User, required_roles: List[str]) -> bool:
@@ -32,7 +35,7 @@ async def list_users(
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
if not check_permission(current_user, ["super_admin", "admin"]):
if not check_permission(current_user, ADMIN_ROLES):
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail="Insufficient permissions",
@@ -91,7 +94,7 @@ async def get_user(
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
if not check_permission(current_user, ["super_admin", "admin"]) and current_user.id != user_id:
if not check_permission(current_user, ADMIN_ROLES) and current_user.id != user_id:
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail="Insufficient permissions",
@@ -128,7 +131,7 @@ async def create_user(
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
if not check_permission(current_user, ["super_admin"]):
if not check_permission(current_user, SUPER_ADMIN_ROLES):
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail="Only super_admin can create users",
@@ -196,18 +199,18 @@ async def update_user(
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
if not check_permission(current_user, ["super_admin", "admin"]) and current_user.id != user_id:
if not check_permission(current_user, ADMIN_ROLES) and current_user.id != user_id:
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail="Insufficient permissions",
)
if not check_permission(current_user, ["super_admin"]) and user_data.role is not None:
if not check_permission(current_user, SUPER_ADMIN_ROLES) and user_data.role is not None:
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail="Only super_admin can change user role",
)
if not check_permission(current_user, ["super_admin"]) and user_data.gatekeeper_groups is not None:
if not check_permission(current_user, SUPER_ADMIN_ROLES) and user_data.gatekeeper_groups is not None:
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail="Only super_admin can change Gatekeeper groups",
@@ -260,7 +263,7 @@ async def delete_user(
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
if not check_permission(current_user, ["super_admin"]):
if not check_permission(current_user, SUPER_ADMIN_ROLES):
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail="Only super_admin can delete users",
+3 -3
View File
@@ -1,4 +1,4 @@
"""Bounded vessel snapshot APIs for viewport-first consumers."""
"""Bounded vessel snapshot APIs backed by the latest vessel state table."""
from typing import Optional
@@ -14,8 +14,8 @@ router = APIRouter()
@router.get("/snapshot")
async def get_vessel_snapshot(
bbox: Optional[str] = Query(None, description="Viewport bbox as lon_min,lat_min,lon_max,lat_max"),
zoom: int = Query(..., ge=1, le=20, description="Current map zoom level"),
bbox: Optional[str] = Query(None, description="Snapshot bbox as lon_min,lat_min,lon_max,lat_max"),
zoom: float = Query(..., ge=1, le=20, description="Current map zoom level"),
type: Optional[str] = Query(
None,
description="Comma-separated vessel types: cargo,tanker,passenger,fishing,military,other",
+51 -78
View File
@@ -18,6 +18,7 @@ from sqlalchemy import select, func
from typing import List, Dict, Any, Optional
from app.core.collected_data_fields import get_record_field
from app.core.enums import BGPStatus
from app.core.satellite_tle import build_tle_lines_from_elements
from app.core.time import to_iso8601_utc
from app.db.session import get_db
@@ -25,7 +26,7 @@ from app.models.bgp_anomaly import BGPAnomaly
from app.models.bgp_incident import BGPIncident
from app.models.bgp_observation import BGPObservation
from app.models.collected_data import CollectedData
from app.models.vessel import AISSourceHealth, VesselPosition, VesselStatic
from app.models.vessel import AISSourceHealth, VesselCurrentState, VesselPosition, VesselStatic
from app.services.bgp_collectors import build_bgp_collector_coverage
from app.services.cable_graph import build_graph_from_data, CableGraph, haversine_distance
from app.services.compute_center_locations import (
@@ -47,11 +48,10 @@ from app.services.location.llm_fallback import (
from app.services.persistent_logs import record_system_log
from app.services.vessel_ais_aggregation import (
build_field_conflict_candidates,
count_unique_raw_vessel_mmsi,
get_aggregated_vessel,
get_aggregated_vessel_track,
get_aggregated_vessels,
get_aggregated_vessels_snapshot,
get_current_vessels_snapshot,
get_vessel_conflict_records,
get_vessel_raw_observations,
MAX_SNAPSHOT_LIMIT,
@@ -75,7 +75,6 @@ TERRAIN_TILE_BATCH_MAX_ITEMS = 128
TERRAIN_TILE_BATCH_CONCURRENCY = 16
_terrain_tile_cache: OrderedDict[tuple[int, int, int], tuple[bytes, str, dict[str, str]]] = OrderedDict()
VESSEL_NAME_FALLBACK_PATTERN = re.compile(r"^mmsi\s*\d+$", re.IGNORECASE)
VESSEL_SNAPSHOT_LEGACY_FALLBACK_ENABLED = True
SECONDS_PER_MINUTE = 60
BYTES_PER_MIB = 1024 * 1024
CABLE_CACHE_FRESH_SECONDS = 6 * 60 * SECONDS_PER_MINUTE
@@ -839,9 +838,14 @@ def convert_aggregated_vessels_to_geojson(vessels: List[dict[str, Any]]) -> Dict
continue
source_summary = {}
for source, summary in (vessel.get("source_summary") or {}).items():
latest_observed_at = summary.get("latest_observed_at")
source_summary[source] = {
**summary,
"latest_observed_at": to_iso8601_utc(summary.get("latest_observed_at")),
"latest_observed_at": (
to_iso8601_utc(latest_observed_at)
if isinstance(latest_observed_at, datetime)
else latest_observed_at
),
}
props = {
"mmsi": vessel["mmsi"],
@@ -1075,7 +1079,7 @@ async def build_vessel_snapshot_response(
db: AsyncSession,
*,
bbox: tuple[float, float, float, float] | None,
zoom: int | None,
zoom: float | None,
type_filter: str | None,
limit: int | None,
since_minutes: int = 60,
@@ -2340,58 +2344,35 @@ async def _load_raw_vessel_snapshot_features(
observed_since: datetime,
) -> tuple[list[dict[str, Any]], dict[str, Any]]:
if bbox is None:
aggregated_vessels = await get_aggregated_vessels(
db,
limit=limit,
observed_since=observed_since,
)
else:
aggregated_vessels = await get_aggregated_vessels_snapshot(
db,
bbox=bbox,
limit=limit,
observed_since=observed_since,
)
raw_geojson = convert_aggregated_vessels_to_geojson(aggregated_vessels)
raw_features = raw_geojson.get("features", [])
features = raw_features
legacy_features: list[dict[str, Any]] = []
legacy_fallback_used = False
if not raw_features and VESSEL_SNAPSHOT_LEGACY_FALLBACK_ENABLED:
legacy_features = await _load_legacy_vessel_snapshot_features(
db,
bbox=bbox,
limit=limit,
)
features, _merge_diagnostics = _merge_vessel_features(raw_features, legacy_features)
legacy_fallback_used = bool(legacy_features)
return [], {
"source": "vessel_current_state",
"current_state_count": 0,
"final_unique_mmsi": 0,
}
current_vessels = await get_current_vessels_snapshot(
db,
bbox=bbox,
limit=limit,
observed_since=observed_since,
)
features = convert_aggregated_vessels_to_geojson(current_vessels).get("features", [])
unique_mmsi = len(
{
key
for key in (_feature_mmsi_key(feature) for feature in features)
if key is not None
}
)
return features, {
"raw_feature_count": len(raw_features),
"raw_unique_mmsi": len(
{
key
for key in (_feature_mmsi_key(feature) for feature in raw_features)
if key is not None
}
),
"legacy_feature_count": len(legacy_features),
"legacy_backfilled_mmsi": len(
{
key
for key in (_feature_mmsi_key(feature) for feature in legacy_features)
if key is not None
}
),
"legacy_fallback_enabled": VESSEL_SNAPSHOT_LEGACY_FALLBACK_ENABLED,
"legacy_fallback_used": legacy_fallback_used,
"final_unique_mmsi": len(
{
key
for key in (_feature_mmsi_key(feature) for feature in features)
if key is not None
}
),
"source": "vessel_current_state",
"current_state_count": len(features),
"final_unique_mmsi": unique_mmsi,
"raw_feature_count": 0,
"raw_unique_mmsi": 0,
"legacy_feature_count": 0,
"legacy_backfilled_mmsi": 0,
"legacy_fallback_enabled": False,
"legacy_fallback_used": False,
}
@router.get("/vessels/custom-supplements")
@@ -2759,10 +2740,10 @@ async def _build_visualization_geo_summary(db: AsyncSession) -> dict[str, Any]:
compute_center_count = supercomputer_count + gpu_cluster_count
active_incident_result = await db.execute(
select(func.count(BGPIncident.id)).where(BGPIncident.status == "active"),
select(func.count(BGPIncident.id)).where(BGPIncident.status == BGPStatus.ACTIVE.value),
)
active_anomaly_result = await db.execute(
select(func.count(BGPAnomaly.id)).where(BGPAnomaly.status == "active"),
select(func.count(BGPAnomaly.id)).where(BGPAnomaly.status == BGPStatus.ACTIVE.value),
)
active_incident_count = int(active_incident_result.scalar() or 0)
active_anomaly_count = int(active_anomaly_result.scalar() or 0)
@@ -2783,21 +2764,14 @@ async def _build_visualization_geo_summary(db: AsyncSession) -> dict[str, Any]:
)
else:
bgp_collector_count = int(bgp_collector_scalar or 0)
raw_unique_window_hours = 24
raw_unique_mmsi = await count_unique_raw_vessel_mmsi(
db,
observed_since=datetime.now(UTC) - timedelta(hours=raw_unique_window_hours),
vessel_current_window_minutes = 60
vessel_current_result = await db.execute(
select(func.count(VesselCurrentState.mmsi)).where(
VesselCurrentState.observed_at
>= datetime.now(UTC) - timedelta(minutes=vessel_current_window_minutes)
)
)
legacy_unique_result = await db.execute(
select(func.count(func.distinct(VesselPosition.mmsi)))
)
legacy_unique_mmsi = int(legacy_unique_result.scalar() or 0)
legacy_fallback_active = (
VESSEL_SNAPSHOT_LEGACY_FALLBACK_ENABLED
and raw_unique_mmsi == 0
and legacy_unique_mmsi > 0
)
vessel_count = legacy_unique_mmsi if legacy_fallback_active else raw_unique_mmsi
vessel_count = int(vessel_current_result.scalar() or 0)
aisstream_health = await db.get(AISSourceHealth, "aisstream_vessels")
return {
@@ -2808,11 +2782,10 @@ async def _build_visualization_geo_summary(db: AsyncSession) -> dict[str, Any]:
"satellite_count": satellite_count,
"compute_center_count": compute_center_count,
"vessel_count": vessel_count,
"vessel_count_source": "legacy_fallback" if legacy_fallback_active else "raw_recent",
"vessel_legacy_fallback_enabled": VESSEL_SNAPSHOT_LEGACY_FALLBACK_ENABLED,
"vessel_raw_unique_mmsi": raw_unique_mmsi,
"vessel_raw_unique_window_hours": raw_unique_window_hours,
"vessel_legacy_unique_mmsi": legacy_unique_mmsi,
"vessel_count_source": "vessel_current_state",
"vessel_current_window_minutes": vessel_current_window_minutes,
"vessel_raw_unique_mmsi": 0,
"vessel_legacy_unique_mmsi": 0,
"aisstream_connection_state": aisstream_health.connection_state if aisstream_health else None,
"aisstream_last_seen_at": to_iso8601_utc(aisstream_health.last_seen_at) if aisstream_health else None,
"aisstream_message_rate": aisstream_health.message_rate if aisstream_health else None,
+3 -2
View File
@@ -9,6 +9,7 @@ from jose import jwt, JWTError
from sqlalchemy import text
from app.core.config import settings
from app.core.enums import UserRole
from app.core.logging import get_logger
from app.core.time import to_iso8601_utc
from app.core.websocket.manager import manager
@@ -96,7 +97,7 @@ async def websocket_endpoint(
"earth_news",
EARTH_UPDATES_CHANNEL,
]
if user_role == "super_admin":
if user_role == UserRole.SUPER_ADMIN.value:
supported_channels = [*supported_channels, LOG_TAIL_CHANNEL]
await manager.connect(websocket, user_id)
@@ -138,7 +139,7 @@ async def websocket_endpoint(
if channel and channel not in channels:
channels = [*channels, channel]
if LOG_TAIL_CHANNEL in channels:
if user_role != "super_admin":
if user_role != UserRole.SUPER_ADMIN.value:
await websocket.send_json(
{
"type": "subscription_error",
+226
View File
@@ -0,0 +1,226 @@
"""Stable backend protocol enums.
Database columns and JSON payloads continue to store the enum string values.
Configurable identifiers, user-authored values, and open-ended taxonomies do
not belong in this module.
"""
from __future__ import annotations
import logging
from enum import StrEnum
from typing import TypeVar
logger = logging.getLogger(__name__)
EnumT = TypeVar("EnumT", bound=StrEnum)
def parse_enum(enum_type: type[EnumT], value: object, default: EnumT) -> EnumT:
"""Parse an external value without breaking reads of legacy data."""
if value is None or str(value).strip() == "":
return default
if isinstance(value, enum_type):
return value
try:
return enum_type(str(value).strip().lower())
except (TypeError, ValueError):
logger.warning(
"Unknown %s value %r; falling back to %s",
enum_type.__name__,
value,
default.value,
)
return default
class NewsImportanceLevel(StrEnum):
LOW = "low"
MEDIUM = "medium"
HIGH = "high"
CRITICAL = "critical"
class BreakingLevel(StrEnum):
NONE = "none"
WATCH = "watch"
BREAKING = "breaking"
CRITICAL = "critical"
class BreakingScope(StrEnum):
REGIONAL = "regional"
GLOBAL = "global"
class BreakingSource(StrEnum):
RULES = "rules"
AI = "ai"
MANUAL = "manual"
MULTI_SOURCE = "multi_source"
class NewsSourceType(StrEnum):
RSS = "rss"
ATOM = "atom"
AGGREGATED = "aggregated"
REFERENCE = "reference"
MANUAL = "manual"
class NewsEnrichmentStatus(StrEnum):
PENDING = "pending"
QUEUED = "queued"
ATTEMPTED = "attempted"
SUCCESS = "success"
CONTENT_ONLY = "content_only"
LOCATION_ONLY = "location_only"
UNAVAILABLE = "unavailable"
PROVIDER_ERROR = "provider_error"
PARSE_ERROR = "parse_error"
NO_RESULT = "no_result"
class NewsMarketImpact(StrEnum):
NONE = "none"
SECTOR = "sector"
NATIONAL = "national"
GLOBAL = "global"
class NewsTaggingSource(StrEnum):
RULES = "rules"
AI = "ai"
MANUAL = "manual"
class JobType(StrEnum):
COLLECT = "collect"
CLEAR_DATA = "clear_data"
CLEAR_CACHE = "clear_cache"
EARTH_REFRESH = "earth_refresh"
class JobStatus(StrEnum):
QUEUED = "queued"
RUNNING = "running"
CANCELLING = "cancelling"
SUCCESS = "success"
FAILED = "failed"
CANCELLED = "cancelled"
class RollbackPolicy(StrEnum):
KEEP_COMMITTED_BATCHES = "keep_committed_batches"
class MappingValidationStatus(StrEnum):
DRAFT = "draft"
VALID = "valid"
INVALID = "invalid"
class SnapshotStatus(StrEnum):
RUNNING = "running"
SUCCESS = "success"
FAILED = "failed"
CANCELLED = "cancelled"
class DatasourceRunStatus(StrEnum):
RUNNING = "running"
NOT_RUN = "not_run"
COLLECTED = "collected"
UNCOLLECTED = "uncollected"
class ProviderApi(StrEnum):
ANTHROPIC_MESSAGES = "anthropic-messages"
OPENAI_COMPLETIONS = "openai-completions"
OLLAMA_GENERATE = "ollama-generate"
class PlaygroundMessageRole(StrEnum):
SYSTEM = "system"
USER = "user"
ASSISTANT = "assistant"
TOOL = "tool"
class PlaygroundMessageKind(StrEnum):
MESSAGE = "message"
THINKING = "thinking"
ERROR = "error"
STATUS = "status"
class PlaygroundMessageStatus(StrEnum):
PENDING = "pending"
THINKING = "thinking"
ANSWERING = "answering"
DONE = "done"
FAILED = "failed"
CANCELLED = "cancelled"
ERROR = "error"
STOPPED = "stopped"
class OtpPurpose(StrEnum):
REGISTER = "register"
VERIFY_EMAIL = "verify_email"
RESET_PASSWORD = "reset_password"
class UserRole(StrEnum):
VIEWER = "viewer"
ADMIN = "admin"
SUPER_ADMIN = "super_admin"
class AlertSeverity(StrEnum):
CRITICAL = "critical"
WARNING = "warning"
INFO = "info"
class AlertStatus(StrEnum):
ACTIVE = "active"
ACKNOWLEDGED = "acknowledged"
RESOLVED = "resolved"
class BGPStatus(StrEnum):
ACTIVE = "active"
ACKNOWLEDGED = "acknowledged"
RESOLVED = "resolved"
class LogLevel(StrEnum):
ALL = "all"
ERROR = "error"
WARNING = "warning"
INFO = "info"
DEBUG = "debug"
class ConnectionState(StrEnum):
DISCONNECTED = "disconnected"
CONNECTING = "connecting"
CONNECTED = "connected"
ERROR = "error"
class AuthType(StrEnum):
NONE = "none"
BEARER = "bearer"
API_KEY = "api_key"
BASIC = "basic"
class TVSourceType(StrEnum):
IFRAME = "iframe"
HLS = "hls"
VIDEO = "video"
EXTERNAL = "external"
YOUTUBE = "youtube"
+17
View File
@@ -626,6 +626,7 @@ async def init_db():
"bgp_collector_locations",
"vessel_static",
"vessel_position",
"vessel_current_state",
"ais_raw_observations",
"ais_source_health",
"compute_center_locations",
@@ -786,6 +787,22 @@ async def init_db():
"""
)
)
await conn.execute(
text(
"""
CREATE INDEX IF NOT EXISTS idx_vessel_current_bbox
ON vessel_current_state (lon, lat)
"""
)
)
await conn.execute(
text(
"""
CREATE INDEX IF NOT EXISTS idx_vessel_current_observed
ON vessel_current_state (observed_at DESC)
"""
)
)
await conn.execute(
text(
"""
+2 -16
View File
@@ -1,26 +1,12 @@
from datetime import datetime
from enum import Enum
from typing import Optional
from sqlalchemy import Column, Integer, String, DateTime, Text, ForeignKey, Enum as SQLEnum
from sqlalchemy.orm import relationship
from sqlalchemy import Column, Integer, String, DateTime, Text, Enum as SQLEnum
from app.core.enums import AlertSeverity, AlertStatus
from app.core.time import to_iso8601_utc
from app.db.session import Base
class AlertSeverity(str, Enum):
CRITICAL = "critical"
WARNING = "warning"
INFO = "info"
class AlertStatus(str, Enum):
ACTIVE = "active"
ACKNOWLEDGED = "acknowledged"
RESOLVED = "resolved"
class Alert(Base):
__tablename__ = "alerts"
+2 -1
View File
@@ -4,6 +4,7 @@ from datetime import datetime
from sqlalchemy import Column, DateTime, Float, ForeignKey, Index, Integer, JSON, String, Text
from app.core.enums import BGPStatus
from app.core.time import to_iso8601_utc
from app.db.session import Base
@@ -17,7 +18,7 @@ class BGPAnomaly(Base):
source = Column(String(100), nullable=False, index=True)
anomaly_type = Column(String(50), nullable=False, index=True)
severity = Column(String(20), nullable=False, index=True)
status = Column(String(20), nullable=False, default="active", index=True)
status = Column(String(20), nullable=False, default=BGPStatus.ACTIVE.value, index=True)
entity_key = Column(String(255), nullable=False, index=True)
prefix = Column(String(64), nullable=True, index=True)
origin_asn = Column(Integer, nullable=True, index=True)
+2 -1
View File
@@ -4,6 +4,7 @@ from datetime import datetime
from sqlalchemy import Column, DateTime, Float, ForeignKey, Index, Integer, JSON, String, Text
from app.core.enums import BGPStatus
from app.core.time import to_iso8601_utc
from app.db.session import Base
@@ -20,7 +21,7 @@ class BGPIncident(Base):
title = Column(String(255), nullable=False)
summary = Column(Text, nullable=False)
severity = Column(String(20), nullable=False, index=True)
status = Column(String(20), nullable=False, default="active", index=True)
status = Column(String(20), nullable=False, default=BGPStatus.ACTIVE.value, index=True)
confidence = Column(Float, nullable=False, default=0.5)
started_at = Column(DateTime(timezone=True), nullable=False, default=datetime.utcnow, index=True)
ended_at = Column(DateTime(timezone=True), nullable=True)
+2 -1
View File
@@ -1,6 +1,7 @@
from sqlalchemy import Boolean, Column, DateTime, ForeignKey, Integer, JSON, String
from sqlalchemy.sql import func
from app.core.enums import SnapshotStatus
from app.db.session import Base
@@ -16,7 +17,7 @@ class DataSnapshot(Base):
started_at = Column(DateTime(timezone=True), server_default=func.now())
completed_at = Column(DateTime(timezone=True), nullable=True)
record_count = Column(Integer, default=0)
status = Column(String(20), nullable=False, default="running")
status = Column(String(20), nullable=False, default=SnapshotStatus.RUNNING.value)
is_current = Column(Boolean, default=True, index=True)
parent_snapshot_id = Column(Integer, ForeignKey("data_snapshots.id"), nullable=True, index=True)
summary = Column(JSON, default={})
+2 -1
View File
@@ -3,6 +3,7 @@
from sqlalchemy import Boolean, Column, DateTime, ForeignKey, Integer, JSON, String
from sqlalchemy.sql import func
from app.core.enums import MappingValidationStatus
from app.db.session import Base
@@ -19,7 +20,7 @@ class DataSourceMappingTemplate(Base):
target_schema = Column(String(80), nullable=False, index=True)
mapping_json = Column(JSON, nullable=False, default={})
sample_payload_hash = Column(String(64), nullable=True)
validation_status = Column(String(30), nullable=False, default="draft")
validation_status = Column(String(30), nullable=False, default=MappingValidationStatus.DRAFT.value)
version = Column(Integer, nullable=False, default=1)
is_active = Column(Boolean, nullable=False, default=False, index=True)
created_at = Column(DateTime(timezone=True), server_default=func.now())
+3 -2
View File
@@ -1,6 +1,7 @@
from sqlalchemy import JSON, Boolean, Column, DateTime, ForeignKey, Integer, String, Text
from sqlalchemy.sql import func
from app.core.enums import PlaygroundMessageKind, PlaygroundMessageStatus
from app.db.session import Base
@@ -13,8 +14,8 @@ class PlaygroundMessage(Base):
user_id = Column(Integer, ForeignKey("users.id", ondelete="CASCADE"), nullable=False, index=True)
parent_message_id = Column(Integer, ForeignKey("playground_messages.id", ondelete="SET NULL"), nullable=True)
role = Column(String(20), nullable=False)
kind = Column(String(20), nullable=False, default="message")
status = Column(String(20), nullable=False, default="done")
kind = Column(String(20), nullable=False, default=PlaygroundMessageKind.MESSAGE.value)
status = Column(String(20), nullable=False, default=PlaygroundMessageStatus.DONE.value)
title = Column(String(255), nullable=True)
content = Column(Text, nullable=False, default="")
thinking_content = Column(Text, nullable=False, default="")
+4 -3
View File
@@ -3,6 +3,7 @@
from sqlalchemy import BigInteger, Column, DateTime, Float, Integer, JSON, String, Text
from sqlalchemy.sql import func
from app.core.enums import JobStatus, JobType, RollbackPolicy
from app.db.session import Base
@@ -12,9 +13,9 @@ class CollectionTask(Base):
id = Column(Integer, primary_key=True, autoincrement=True)
datasource_id = Column(Integer, nullable=False, index=True)
source = Column(String(100), nullable=True, index=True)
task_type = Column(String(30), nullable=False, default="collect", index=True)
task_type = Column(String(30), nullable=False, default=JobType.COLLECT.value, index=True)
status = Column(String(20), nullable=False) # queued, running, cancelling, success, failed, cancelled
phase = Column(String(30), default="queued")
phase = Column(String(30), default=JobStatus.QUEUED.value)
phase_progress = Column(Float)
phase_message = Column(String(255))
phase_current = Column(BigInteger)
@@ -27,7 +28,7 @@ class CollectionTask(Base):
progress = Column(Float, default=0.0) # Progress percentage (0-100)
error_message = Column(Text)
payload = Column(JSON, default=dict)
rollback_policy = Column(String(40), nullable=False, default="keep_committed_batches")
rollback_policy = Column(String(40), nullable=False, default=RollbackPolicy.KEEP_COMMITTED_BATCHES.value)
dedupe_key = Column(String(180), nullable=True, index=True)
worker_id = Column(String(120), nullable=True, index=True)
locked_at = Column(DateTime(timezone=True), nullable=True, index=True)
+2 -1
View File
@@ -1,6 +1,7 @@
from sqlalchemy import Boolean, Column, DateTime, Integer, JSON, String
from sqlalchemy.sql import func
from app.core.enums import UserRole
from app.db.session import Base
@@ -11,7 +12,7 @@ class User(Base):
username = Column(String(50), unique=True, index=True, nullable=False)
email = Column(String(255), unique=True, index=True, nullable=False)
password_hash = Column(String(255), nullable=False)
role = Column(String(20), default="viewer")
role = Column(String(20), default=UserRole.VIEWER.value)
gatekeeper_groups = Column(JSON, default=list)
is_active = Column(Boolean, default=True)
email_verified = Column(Boolean, default=False, nullable=False)
+63 -1
View File
@@ -3,6 +3,7 @@
from sqlalchemy import BigInteger, Column, DateTime, Float, Index, Integer, JSON, SmallInteger, String
from sqlalchemy.sql import func
from app.core.enums import ConnectionState
from app.core.time import to_iso8601_utc
from app.db.session import Base
@@ -75,6 +76,67 @@ class VesselPosition(Base):
}
class VesselCurrentState(Base):
"""Latest renderable state for one vessel, independent from AIS history."""
__tablename__ = "vessel_current_state"
mmsi = Column(BigInteger, primary_key=True)
lat = Column(Float, nullable=False)
lon = Column(Float, nullable=False)
sog = Column(Float, nullable=True)
cog = Column(Float, nullable=True)
heading = Column(SmallInteger, nullable=True)
nav_status = Column(SmallInteger, nullable=True, index=True)
name = Column(String(128), nullable=True)
callsign = Column(String(16), nullable=True)
vessel_type = Column(SmallInteger, nullable=True, index=True)
vessel_type_name = Column(String(64), nullable=True, index=True)
flag = Column(String(4), nullable=True, index=True)
length = Column(Float, nullable=True)
width = Column(Float, nullable=True)
draught = Column(Float, nullable=True)
imo = Column(BigInteger, nullable=True)
source = Column(String(100), nullable=False, index=True)
observed_at = Column(DateTime(timezone=True), nullable=False)
field_sources = Column(JSON, default=dict)
selected_reasons = Column(JSON, default=dict)
source_summary = Column(JSON, default=dict)
quality_flags = Column(JSON, default=list)
updated_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now())
__table_args__ = (
Index("idx_vessel_current_bbox", "lon", "lat"),
Index("idx_vessel_current_observed", "observed_at"),
)
def to_dict(self) -> dict:
return {
"mmsi": self.mmsi,
"lat": self.lat,
"lon": self.lon,
"sog": self.sog,
"cog": self.cog,
"heading": self.heading,
"nav_status": self.nav_status,
"name": self.name,
"callsign": self.callsign,
"vessel_type": self.vessel_type,
"vessel_type_name": self.vessel_type_name,
"flag": self.flag,
"length": self.length,
"width": self.width,
"draught": self.draught,
"imo": self.imo,
"source": self.source,
"received_at": self.observed_at,
"field_sources": self.field_sources or {},
"selected_reasons": self.selected_reasons or {},
"source_summary": self.source_summary or {},
"quality_flags": self.quality_flags or [],
}
class AISRawObservation(Base):
"""Source-level AIS fact before aggregation and conflict resolution."""
@@ -165,7 +227,7 @@ class AISSourceHealth(Base):
__tablename__ = "ais_source_health"
source = Column(String(100), primary_key=True)
connection_state = Column(String(32), nullable=False, default="disconnected", index=True)
connection_state = Column(String(32), nullable=False, default=ConnectionState.DISCONNECTED.value, index=True)
last_seen_at = Column(DateTime(timezone=True), nullable=True, index=True)
last_success_at = Column(DateTime(timezone=True), nullable=True, index=True)
last_error = Column(String(500), nullable=True)
+4 -3
View File
@@ -2,6 +2,7 @@ from typing import Any
from pydantic import BaseModel, Field
from app.core.enums import PlaygroundMessageKind, PlaygroundMessageRole, PlaygroundMessageStatus
class AIContentBlock(BaseModel):
type: str
@@ -110,9 +111,9 @@ class PlaygroundSessionUpsertRequest(BaseModel):
class PlaygroundMessageRecord(BaseModel):
id: str
role: str
kind: str = "message"
status: str = "done"
role: PlaygroundMessageRole
kind: PlaygroundMessageKind = PlaygroundMessageKind.MESSAGE
status: PlaygroundMessageStatus = PlaygroundMessageStatus.DONE
title: str | None = None
content: str = ""
thinking_content: str = ""
+4 -3
View File
@@ -3,6 +3,7 @@ from typing import Optional
from pydantic import BaseModel, EmailStr, Field
from app.core.enums import OtpPurpose, UserRole
class UserBase(BaseModel):
username: str
@@ -11,13 +12,13 @@ class UserBase(BaseModel):
class UserCreate(UserBase):
password: str = Field(..., min_length=8)
role: str = "viewer"
role: UserRole = UserRole.VIEWER
gatekeeper_groups: list[str] = Field(default_factory=list)
class UserUpdate(BaseModel):
email: Optional[EmailStr] = None
role: Optional[str] = None
role: Optional[UserRole] = None
gatekeeper_groups: Optional[list[str]] = None
is_active: Optional[bool] = None
@@ -59,7 +60,7 @@ class VerifyEmailRequest(BaseModel):
class ResendCodeRequest(BaseModel):
email: EmailStr
purpose: str = Field(default="register", pattern="^(register|verify_email|reset_password)$")
purpose: OtpPurpose = OtpPurpose.REGISTER
class ForgotPasswordRequest(BaseModel):
+6 -5
View File
@@ -6,6 +6,7 @@ from collections import Counter, defaultdict
from datetime import UTC, datetime
from typing import Any
from app.core.enums import BGPStatus
from app.models.bgp_anomaly import BGPAnomaly
@@ -127,7 +128,7 @@ def detect_origin_change_anomalies(
source=source,
anomaly_type=anomaly_type,
severity=severity,
status="active",
status=BGPStatus.ACTIVE.value,
entity_key=f"{anomaly_type}:{prefix}:{new_origin}",
prefix=prefix,
origin_asn=sorted(historic)[0] if historic else None,
@@ -197,7 +198,7 @@ def detect_more_specific_burst_anomalies(
source=source,
anomaly_type="more_specific_burst",
severity="high",
status="active",
status=BGPStatus.ACTIVE.value,
entity_key=f"more_specific_burst:{root_prefix}:{len(unique_prefixes)}:{len(related_collectors)}",
prefix=sample.get("prefix"),
origin_asn=sample.get("origin_asn"),
@@ -267,7 +268,7 @@ def detect_mass_withdrawal_anomalies(
source=source,
anomaly_type="mass_withdrawal",
severity=severity,
status="active",
status=BGPStatus.ACTIVE.value,
entity_key=f"mass_withdrawal:{prefix}:{origin_asn}:{len(related_collectors)}:{count}",
prefix=prefix,
origin_asn=origin_asn,
@@ -354,7 +355,7 @@ def detect_route_leak_anomalies(
source=source,
anomaly_type="route_leak_candidate",
severity="high" if max_path_length >= dominant_length + 3 else "medium",
status="active",
status=BGPStatus.ACTIVE.value,
entity_key=f"route_leak_candidate:{prefix}:{max_path_length}:{len(related_collectors)}",
prefix=prefix,
origin_asn=sample_metadata.get("origin_asn"),
@@ -435,7 +436,7 @@ def detect_path_flap_anomalies(
source=source,
anomaly_type="path_flap",
severity=severity,
status="active",
status=BGPStatus.ACTIVE.value,
entity_key=f"path_flap:{prefix}:{transitions}:{len(distinct_paths)}",
prefix=prefix,
origin_asn=sample_metadata.get("origin_asn"),
+3 -2
View File
@@ -9,6 +9,7 @@ from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.collected_data_fields import get_record_field
from app.core.enums import BGPStatus
from app.models.bgp_anomaly import BGPAnomaly
from app.models.bgp_incident import BGPIncident
from app.models.collected_data import CollectedData
@@ -290,7 +291,7 @@ async def create_bgp_incidents_for_anomalies(
existing.title = title
existing.summary = summary
existing.severity = severity
existing.status = "active"
existing.status = BGPStatus.ACTIVE.value
existing.confidence = confidence
existing.started_at = primary.started_at or existing.started_at or datetime.now(UTC)
existing.ended_at = None
@@ -313,7 +314,7 @@ async def create_bgp_incidents_for_anomalies(
title=title,
summary=summary,
severity=severity,
status="active",
status=BGPStatus.ACTIVE.value,
confidence=confidence,
started_at=primary.started_at or datetime.now(UTC),
affected_prefixes=prefixes,
+9 -9
View File
@@ -12,11 +12,11 @@ from sqlalchemy.ext.asyncio import AsyncSession
from app.core.collected_data_fields import build_dynamic_metadata, get_record_field
from app.core.countries import normalize_country
from app.core.enums import JobStatus, SnapshotStatus
from app.core.logging import get_logger
from app.core.time import to_iso8601_utc
from app.core.websocket.broadcaster import broadcaster
from app.services.business_logs import emit_business_log, exception_context
from app.services.earth_layer_adapters import get_earth_update_layers_for_source
logger = get_logger(__name__, service="collector")
@@ -238,7 +238,7 @@ class BaseCollector(ABC):
snapshot = await db.get(DataSnapshot, snapshot_id)
if snapshot:
parent_snapshot_id = snapshot.parent_snapshot_id
snapshot.status = "cancelled"
snapshot.status = SnapshotStatus.CANCELLED.value
snapshot.is_current = False
snapshot.completed_at = datetime.now(UTC)
summary = dict(snapshot.summary or {})
@@ -308,7 +308,7 @@ class BaseCollector(ABC):
task.datasource_id = datasource_id
task.source = task.source or self.name
task.task_type = task.task_type or "collect"
task.status = "running"
task.status = JobStatus.RUNNING.value
task.phase = "queued"
task.started_at = task.started_at or start_time
task.completed_at = None
@@ -393,7 +393,7 @@ class BaseCollector(ABC):
},
)
task.status = "success"
task.status = JobStatus.SUCCESS.value
task.phase = "completed"
task.phase_progress = 100.0
task.phase_message = "采集完成"
@@ -428,7 +428,7 @@ class BaseCollector(ABC):
}
except asyncio.CancelledError:
await db.rollback()
task.status = "cancelled"
task.status = JobStatus.CANCELLED.value
task.phase = "cancelled"
task.phase_message = "采集已取消"
task.error_message = "Collection cancelled by operator and rolled back"
@@ -456,7 +456,7 @@ class BaseCollector(ABC):
raise
except Exception as e:
await db.rollback()
task.status = "failed"
task.status = JobStatus.FAILED.value
task.phase = "failed"
task.phase_message = str(e)
task.error_message = str(e)
@@ -464,7 +464,7 @@ class BaseCollector(ABC):
if snapshot_id is not None:
snapshot = await db.get(DataSnapshot, snapshot_id)
if snapshot:
snapshot.status = "failed"
snapshot.status = SnapshotStatus.FAILED.value
snapshot.completed_at = datetime.now(UTC)
snapshot.summary = {"error": str(e)}
await db.commit()
@@ -509,7 +509,7 @@ class BaseCollector(ABC):
if snapshot:
snapshot.record_count = 0
snapshot.summary = {"created": 0, "updated": 0, "unchanged": 0}
snapshot.status = "success"
snapshot.status = SnapshotStatus.SUCCESS.value
snapshot.completed_at = datetime.now(UTC)
await db.commit()
return 0
@@ -642,7 +642,7 @@ class BaseCollector(ABC):
snapshot = await db.get(DataSnapshot, snapshot_id)
if snapshot:
snapshot.record_count = records_added
snapshot.status = "success"
snapshot.status = SnapshotStatus.SUCCESS.value
snapshot.completed_at = datetime.now(UTC)
snapshot.summary = {
"created": created_count,
@@ -6,6 +6,7 @@ from typing import Any
import httpx
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.enums import SnapshotStatus
from app.core.time import to_iso8601_utc
from app.core.websocket.broadcaster import broadcaster
from app.services.barentswatch import (
@@ -125,7 +126,7 @@ class VesselAISCollector(BaseCollector):
snapshot = await db.get(DataSnapshot, snapshot_id)
if snapshot:
snapshot.record_count = records_added
snapshot.status = "success"
snapshot.status = SnapshotStatus.SUCCESS.value
snapshot.completed_at = now
snapshot.summary = {
"created": records_added,
+12 -11
View File
@@ -12,6 +12,7 @@ from sqlalchemy.ext.asyncio import AsyncSession
from app.core.cache import cache
from app.core.config import settings
from app.core.enums import JobStatus, JobType, RollbackPolicy
from app.core.logging import get_logger
from app.core.time import to_iso8601_utc
from app.core.websocket.broadcaster import broadcaster
@@ -36,17 +37,17 @@ from app.services.scheduler import sync_datasource_job
logger = get_logger(__name__)
JOB_TYPE_COLLECT = "collect"
JOB_TYPE_CLEAR_DATA = "clear_data"
JOB_TYPE_CLEAR_CACHE = "clear_cache"
JOB_TYPE_EARTH_REFRESH = "earth_refresh"
JOB_TYPE_COLLECT = JobType.COLLECT.value
JOB_TYPE_CLEAR_DATA = JobType.CLEAR_DATA.value
JOB_TYPE_CLEAR_CACHE = JobType.CLEAR_CACHE.value
JOB_TYPE_EARTH_REFRESH = JobType.EARTH_REFRESH.value
JOB_STATUS_QUEUED = "queued"
JOB_STATUS_RUNNING = "running"
JOB_STATUS_CANCELLING = "cancelling"
JOB_STATUS_SUCCESS = "success"
JOB_STATUS_FAILED = "failed"
JOB_STATUS_CANCELLED = "cancelled"
JOB_STATUS_QUEUED = JobStatus.QUEUED.value
JOB_STATUS_RUNNING = JobStatus.RUNNING.value
JOB_STATUS_CANCELLING = JobStatus.CANCELLING.value
JOB_STATUS_SUCCESS = JobStatus.SUCCESS.value
JOB_STATUS_FAILED = JobStatus.FAILED.value
JOB_STATUS_CANCELLED = JobStatus.CANCELLED.value
ACTIVE_JOB_STATUSES = (JOB_STATUS_QUEUED, JOB_STATUS_RUNNING, JOB_STATUS_CANCELLING)
TERMINAL_JOB_STATUSES = (JOB_STATUS_SUCCESS, JOB_STATUS_FAILED, JOB_STATUS_CANCELLED)
@@ -80,7 +81,7 @@ async def enqueue_datasource_job(
task_type: str,
*,
payload: dict[str, Any] | None = None,
rollback_policy: str = "keep_committed_batches",
rollback_policy: str = RollbackPolicy.KEEP_COMMITTED_BATCHES.value,
dedupe_key: str | None = None,
) -> CollectionTask:
if dedupe_key:
@@ -13,6 +13,7 @@ from sqlalchemy import func, select
from app.core.data_sources import get_data_sources_config
from app.core.datasource_defaults import DEFAULT_DATASOURCES
from app.core.enums import JobStatus
from app.models.collected_data import CollectedData
from app.models.datasource import DataSource
from app.models.datasource_config import DataSourceConfig
@@ -398,7 +399,7 @@ async def has_collected_data(db, source: str) -> bool:
datasource_result = await db.execute(select(DataSource).where(DataSource.source == source))
datasource = datasource_result.scalar_one_or_none()
return bool(datasource and datasource.last_status == "success")
return bool(datasource and datasource.last_status == JobStatus.SUCCESS.value)
async def get_builtin_connection_status(
+7 -3
View File
@@ -6,6 +6,7 @@ from dataclasses import dataclass
from pathlib import Path
from typing import Literal
from app.core.enums import UserRole
from app.models.user import User
DocsAccess = Literal["public", "docs_user", "docs_developer", "docs_admin"]
@@ -43,7 +44,9 @@ DOCS_METADATA: tuple[DocsMetadata, ...] = (
DocsMetadata("earth-satellite-footprint-policy.md", "earth-satellite-footprint-policy", "docs_developer", "Earth", 13, "智能星球卫星覆盖策略", "Intelligent Planet Satellite Footprint Policy"),
DocsMetadata("earth-bgp-context.md", "earth-bgp-context", "docs_developer", "Earth", 14, "BGP 态势上下文", "BGP Context"),
DocsMetadata("earth-interactable-usage.md", "earth-interactable-usage", "docs_developer", "Earth", 16, "智能星球可交互图标接入", "Intelligent Planet Interactable Usage"),
DocsMetadata("earth-interactable-clustering.md", "earth-interactable-clustering", "docs_developer", "Earth", 17, "智能星球可交互图标聚类策略", "Intelligent Planet Interactable Clustering"),
DocsMetadata("earth-toolbar-overlay-coordination.md", "earth-toolbar-overlay-coordination", "docs_developer", "Earth", 17, "智能星球工具栏与浮层协同", "Intelligent Planet Toolbar and Overlay Coordination"),
DocsMetadata("earth-news-sources.md", "earth-news-sources", "docs_developer", "Earth", 19, "智能星球新闻源配置", "Intelligent Planet News Source Configuration"),
DocsMetadata("frontend-admin-frontend-context.md", "frontend-admin-frontend-context", "docs_developer", "Frontend", 20, "控制台前端结构", "Admin Frontend Context"),
DocsMetadata("frontend-layout-guidelines.md", "frontend-layout-guidelines", "docs_developer", "Frontend", 21, "前端布局指南", "Frontend Layout Guidelines"),
DocsMetadata("tactile-ui-components.md", "tactile-ui-components", "docs_developer", "Frontend", 24, "Tactile UI 组件库", "Tactile UI Components"),
@@ -52,7 +55,8 @@ DOCS_METADATA: tuple[DocsMetadata, ...] = (
DocsMetadata("datasource-collector-settings-connectivity.md", "datasource-collector-settings-connectivity", "docs_developer", "Backend", 32, "数据源、采集器设置与连接验证", "Datasource Collector Settings and Connectivity"),
DocsMetadata("backend-datasources-api-performance.md", "backend-datasources-api-performance", "docs_developer", "Backend", 33, "数据源 API 性能", "Datasource API Performance"),
DocsMetadata("data-job-earth-sync-architecture.md", "data-job-earth-sync-architecture", "docs_developer", "Backend", 34, "数据作业与 Outbox 技术架构", "Data Jobs and Outbox Architecture"),
DocsMetadata("location-pipeline-development.md", "location-pipeline-development", "docs_developer", "Backend", 35, "通用位置估算管线开发说明", "Shared Location Resolution Pipeline Development Guide"),
DocsMetadata("backend-enum-contracts.md", "backend-enum-contracts", "docs_developer", "Backend", 35, "后端枚举与字符串兼容契约", "Backend Enum and String Compatibility Contract"),
DocsMetadata("location-pipeline-development.md", "location-pipeline-development", "docs_developer", "Backend", 36, "通用位置估算管线开发说明", "Shared Location Resolution Pipeline Development Guide"),
DocsMetadata("earth-news-live-streams-collector-format.md", "earth-news-live-streams-collector-format", "docs_developer", "Backend", 36, "新闻直播采集格式", "News Live Streams Collector Format"),
DocsMetadata("docs-gatekeeper-development.md", "docs-gatekeeper-development", "docs_developer", "Backend", 37, "Docs Gatekeeper 开发说明", "Docs Gatekeeper Development Guide"),
DocsMetadata("agents-aiprovider.md", "agents-aiprovider", "docs_developer", "Agents", 40, "AI Provider 指南", "AI Provider Guide"),
@@ -69,9 +73,9 @@ def get_user_gatekeeper_groups(user: User | None) -> set[str]:
return set()
role = user.role.value if hasattr(user.role, "value") else str(user.role or "")
if role == "super_admin":
if role == UserRole.SUPER_ADMIN.value:
return {"docs_user", "docs_developer", "docs_admin"}
if role == "admin":
if role == UserRole.ADMIN.value:
return {"docs_user", "docs_developer", "docs_admin"}
groups = set()
+2 -2
View File
@@ -20,8 +20,8 @@ class EarthLayerAdapter:
EARTH_LAYER_ADAPTERS: tuple[EarthLayerAdapter, ...] = (
EarthLayerAdapter(
sources=frozenset({"barentswatch_vessels", "aisstream_vessels", "vessel_static", "vessel_position", "ais_raw_observations", "ais_source_health"}),
tables=frozenset({"vessel_static", "vessel_position", "ais_raw_observations", "ais_source_health"}),
sources=frozenset({"barentswatch_vessels", "aisstream_vessels", "vessel_static", "vessel_position", "vessel_current_state", "ais_raw_observations", "ais_source_health"}),
tables=frozenset({"vessel_static", "vessel_position", "vessel_current_state", "ais_raw_observations", "ais_source_health"}),
layers=("vessels",),
cache_patterns=("vessels*", "summary*"),
derived_models=("ais_raw_observations", "ais_conflict_records", "ais_source_health"),
+156 -105
View File
@@ -20,10 +20,27 @@ from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.countries import COUNTRY_VARIANTS_MAP, get_country_centroid, normalize_country
from app.core.enums import (
BreakingLevel,
BreakingScope,
BreakingSource,
NewsEnrichmentStatus,
NewsImportanceLevel,
NewsMarketImpact,
NewsSourceType,
NewsTaggingSource,
)
from app.models.system_setting import SystemSetting
from app.ai_tasks.prompts import EffectiveAIPrompt, get_effective_prompt
from app.schemas.ai import SituationalAnalysisRequest
from app.services.ai_client import AIProviderClient
from app.services.earth_news_classification import (
apply_news_classification as _apply_news_classification,
breaking_sort_rank as _breaking_sort_rank,
highest_breaking_level as _highest_breaking_level,
normalize_breaking_level as _normalize_breaking_level_enum,
normalize_breaking_scope as _normalize_breaking_scope_enum,
)
from app.services.location.resolvers.nominatim import build_default_nominatim_geocoder
@@ -69,7 +86,7 @@ class NewsFeedEndpoint:
id: str
name: str
url: str
type: str = "rss"
type: str = NewsSourceType.RSS.value
region: str = ""
enabled: bool = True
default_category: str = "other"
@@ -85,7 +102,7 @@ class NewsFeedSource:
feed_url: str
homepage_url: str
feed_directory_url: str = ""
source_type: str = "rss"
source_type: str = NewsSourceType.RSS.value
feed_urls: tuple[str, ...] = ()
feeds: tuple[NewsFeedEndpoint, ...] = ()
priority: int = 100
@@ -120,7 +137,7 @@ class ParsedNewsItem:
published_at: datetime | None
content_language: str = "en"
localizations: dict[str, dict[str, str]] = field(default_factory=dict)
enrichment_status: str = "pending"
enrichment_status: str = NewsEnrichmentStatus.PENDING.value
enrichment_error: str | None = None
enriched_at: datetime | None = None
target_location: NewsTargetLocation | None = None
@@ -132,16 +149,22 @@ class ParsedNewsItem:
location_patch: dict[str, Any] | None = None
source_tags: list[str] = field(default_factory=list)
feed_id: str = ""
feed_type: str = "rss"
feed_type: str = NewsSourceType.RSS.value
feed_default_category: str = "other"
category: str = "other"
item_tags: list[str] = field(default_factory=list)
tagging_source: str = "rules"
tagging_source: str = NewsTaggingSource.RULES.value
tagging_confidence: float = 0.0
importance_score: int = 0
importance_level: str = "low"
importance_level: str = NewsImportanceLevel.LOW.value
importance_reasons: list[str] = field(default_factory=list)
market_impact: str = "none"
market_impact: str = NewsMarketImpact.NONE.value
breaking_level: str = BreakingLevel.NONE.value
breaking_scope: str = BreakingScope.REGIONAL.value
breaking_reasons: list[str] = field(default_factory=list)
breaking_source: str = BreakingSource.RULES.value
breaking_confidence: float = 0.0
breaking_expires_at: datetime | None = None
@dataclass
@@ -1863,34 +1886,12 @@ def _parse_feed_entries(
return items
def _contains_keyword(text: str, keyword: str) -> bool:
keyword_text = str(keyword or "").strip().lower()
if not keyword_text:
return False
if re.search(r"[\u4e00-\u9fff]", keyword_text):
return keyword_text in text
return re.search(rf"(?<![a-z0-9]){re.escape(keyword_text)}(?![a-z0-9])", text) is not None
def _normalize_breaking_level(value: str | None) -> str:
return _normalize_breaking_level_enum(value).value
def _score_category(text: str, title_text: str, category: dict[str, Any]) -> int:
score = 0
keywords = category.get("keywords") if isinstance(category.get("keywords"), list) else []
for keyword in keywords:
if _contains_keyword(title_text, keyword):
score += 3
elif _contains_keyword(text, keyword):
score += 1
return score
def _importance_level(score: int) -> str:
if score >= 80:
return "critical"
if score >= 60:
return "high"
if score >= 35:
return "medium"
return "low"
def _normalize_breaking_scope(value: str | None) -> str:
return _normalize_breaking_scope_enum(value).value
def apply_news_classification(
@@ -1901,74 +1902,7 @@ def apply_news_classification(
config_payload: dict[str, Any] | None = None,
) -> ParsedNewsItem:
config = normalize_earth_news_sources_payload(config_payload)
title_text = item.title.lower()
combined_text = f"{item.title} {item.summary} {item.source} {item.feed_name}".lower()
feed_default_category = (feed.default_category if feed else item.feed_default_category) or source.default_category or "other"
best_key = feed_default_category
best_score = 0
second_score = 0
for category in config["categories"]:
if not isinstance(category, dict) or category.get("enabled") is False:
continue
score = _score_category(combined_text, title_text, category)
if score > best_score:
second_score = best_score
best_score = score
best_key = str(category.get("key") or "other")
elif score > second_score:
second_score = score
item_tags: list[str] = []
for rule in config["item_tag_rules"]:
if not isinstance(rule, dict):
continue
keywords = rule.get("keywords") if isinstance(rule.get("keywords"), list) else []
if any(_contains_keyword(combined_text, keyword) for keyword in keywords):
tag_key = str(rule.get("key") or "").strip()
if tag_key and tag_key not in item_tags:
item_tags.append(tag_key)
if best_score < 3 and rule.get("category"):
best_key = str(rule["category"])
best_score = 3
confidence = round(best_score / (best_score + second_score + 1), 2) if best_score else 0.35
if best_score < 3 and feed_default_category:
best_key = feed_default_category
confidence = 0.45
importance_score = max(0, min(100, 18 + source.importance_weight + best_score * 6))
reasons: list[str] = []
source_tag_set = set(source.source_tags)
if "official_data" in source_tag_set:
importance_score += 20
reasons.append("官方数据源")
if "press_release" in source_tag_set:
importance_score = max(0, importance_score - 12)
reasons.append("企业公告基础权重较低")
ecommerce_terms = ("网上零售额", "电商物流指数", "gmv", "订单量", "物流指数", "履约", "直播电商", "跨境电商")
if any(_contains_keyword(combined_text, term) for term in ecommerce_terms):
importance_score += 25
reasons.append("命中电商数据指标")
major_platforms = ("amazon", "shopify", "walmart", "alibaba", "jd.com", "pinduoduo", "tiktok shop", "shein", "阿里", "京东", "拼多多", "抖音")
if any(_contains_keyword(combined_text, term) for term in major_platforms):
importance_score += 15
reasons.append("涉及大型平台")
if any(term in combined_text for term in ("同比", "环比", "%", "billion", "million", "增长", "下降")):
importance_score += 10
reasons.append("包含量化指标")
importance_score = max(0, min(100, importance_score))
item.category = best_key or "other"
item.item_tags = item_tags
item.tagging_source = "rules"
item.tagging_confidence = confidence
item.importance_score = importance_score
item.importance_level = _importance_level(importance_score)
item.importance_reasons = reasons or ["按来源权重和分类规则计算"]
item.market_impact = "global" if "global" in source_tag_set else "national" if {"china", "us"} & source_tag_set else "sector"
item.source_tags = list(source.source_tags)
return item
return _apply_news_classification(item, source, feed=feed, config=config)
def _serialize_sources(sources: list[NewsFeedSource]) -> list[dict[str, Any]]:
@@ -2020,6 +1954,10 @@ def _serialize_enriched_at(value: datetime | None) -> str | None:
return value.isoformat().replace("+00:00", "Z") if value else None
def _serialize_breaking_expires_at(value: datetime | None) -> str | None:
return value.isoformat().replace("+00:00", "Z") if value else None
def _content_patch(item: ParsedNewsItem) -> dict[str, Any]:
return {
"content_language": item.content_language,
@@ -2047,6 +1985,12 @@ def _news_meta_patch(item: ParsedNewsItem) -> dict[str, Any]:
"importance_level": item.importance_level,
"importance_reasons": list(item.importance_reasons or []),
"market_impact": item.market_impact,
"breaking_level": _normalize_breaking_level(item.breaking_level),
"breaking_scope": _normalize_breaking_scope(item.breaking_scope),
"breaking_reasons": list(item.breaking_reasons or []),
"breaking_source": item.breaking_source,
"breaking_confidence": item.breaking_confidence,
"breaking_expires_at": _serialize_breaking_expires_at(item.breaking_expires_at),
}
@@ -2142,6 +2086,12 @@ def build_target_location_job_payload(item: ParsedNewsItem) -> dict[str, Any]:
"importance_level": item.importance_level,
"importance_reasons": list(item.importance_reasons or []),
"market_impact": item.market_impact,
"breaking_level": _normalize_breaking_level(item.breaking_level),
"breaking_scope": _normalize_breaking_scope(item.breaking_scope),
"breaking_reasons": list(item.breaking_reasons or []),
"breaking_source": item.breaking_source,
"breaking_confidence": item.breaking_confidence,
"breaking_expires_at": _serialize_breaking_expires_at(item.breaking_expires_at),
}
@@ -2173,6 +2123,12 @@ def parsed_news_item_from_job_payload(payload: dict[str, Any]) -> ParsedNewsItem
importance_level=str(payload.get("importance_level") or "low"),
importance_reasons=list(payload.get("importance_reasons") or []),
market_impact=str(payload.get("market_impact") or "none"),
breaking_level=_normalize_breaking_level(str(payload.get("breaking_level") or "none")),
breaking_scope=_normalize_breaking_scope(str(payload.get("breaking_scope") or "regional")),
breaking_reasons=list(payload.get("breaking_reasons") or []),
breaking_source=str(payload.get("breaking_source") or "rules"),
breaking_confidence=float(payload.get("breaking_confidence") or 0),
breaking_expires_at=_parse_datetime(_coerce_str(payload.get("breaking_expires_at"))),
)
@@ -2218,6 +2174,12 @@ def _serialize_item(item: ParsedNewsItem, *, active_region: str, locale: str = D
"importance_level": item.importance_level,
"importance_reasons": list(item.importance_reasons or []),
"market_impact": item.market_impact,
"breaking_level": _normalize_breaking_level(item.breaking_level),
"breaking_scope": _normalize_breaking_scope(item.breaking_scope),
"breaking_reasons": list(item.breaking_reasons or []),
"breaking_source": item.breaking_source,
"breaking_confidence": item.breaking_confidence,
"breaking_expires_at": _serialize_breaking_expires_at(item.breaking_expires_at),
}
@@ -2239,6 +2201,9 @@ def _build_payload(
) -> dict[str, Any]:
profile = get_region_profile(active_region)
timestamp = generated_at or datetime.now(UTC)
visible_items = list(items)
cruise_visible_items = list(cruise_items if cruise_items is not None else items)
highest_breaking_level = _highest_breaking_level(visible_items + cruise_visible_items)
return {
"generated_at": timestamp.isoformat().replace("+00:00", "Z"),
"focus": {
@@ -2256,11 +2221,13 @@ def _build_payload(
"sources": sorted(source_ids or []),
"limit": limit,
"locale": locale,
"has_breaking": highest_breaking_level != "none",
"highest_breaking_level": highest_breaking_level,
},
"items": [_serialize_item(item, active_region=active_region, locale=locale) for item in items],
"items": [_serialize_item(item, active_region=active_region, locale=locale) for item in visible_items],
"cruise_items": [
_serialize_item(item, active_region=active_region, locale=locale)
for item in (cruise_items if cruise_items is not None else items)
for item in cruise_visible_items
],
"errors": errors,
"stale": stale,
@@ -2282,7 +2249,11 @@ def _rank_and_trim_items(
return sorted(
deduped.values(),
key=lambda item: (
False if active_region == "global" else item.feed_region != active_region,
-_breaking_sort_rank(item),
False
if active_region == "global"
or (_breaking_sort_rank(item) > 0 and _normalize_breaking_scope(item.breaking_scope) == "global")
else item.feed_region != active_region,
item.published_at is None,
-(item.published_at.timestamp() if item.published_at else 0),
item.feed_name,
@@ -2311,6 +2282,53 @@ def _filter_news_items_by_source_ids(
]
def _news_item_source_id(item: ParsedNewsItem) -> str:
return item.id.split(":", 1)[0] if ":" in item.id else ""
def _is_news_item_display_ready(item: ParsedNewsItem, *, locale: str) -> bool:
return bool(
_get_locale_text(item, "title", locale=locale)
and _get_locale_text(item, "summary", locale=locale)
)
def _diversify_news_items_for_locale(
items: list[ParsedNewsItem],
*,
active_region: str,
limit: int,
locale: str,
) -> list[ParsedNewsItem]:
ranked = sorted(
_rank_and_trim_items(items, active_region=active_region, limit=max(len(items), limit)),
key=lambda item: (not _is_news_item_display_ready(item, locale=locale),),
)
buckets: dict[str, list[ParsedNewsItem]] = {}
order: list[str] = []
for item in ranked:
if active_region == "global":
key = item.feed_region or "global"
else:
key = _news_item_source_id(item) or item.source or item.feed_name or item.id
if key not in buckets:
buckets[key] = []
order.append(key)
buckets[key].append(item)
diversified: list[ParsedNewsItem] = []
while len(diversified) < limit and order:
next_order: list[str] = []
for source_id in order:
bucket = buckets.get(source_id) or []
if bucket and len(diversified) < limit:
diversified.append(bucket.pop(0))
if bucket:
next_order.append(source_id)
order = next_order
return diversified
async def _call_store_list_items(list_fn, db: AsyncSession, **kwargs):
try:
return await list_fn(db, **kwargs)
@@ -2736,6 +2754,7 @@ async def get_earth_news_payload(
)
await record_earth_news_sources_health(db, health_by_source)
ranked_fetched_items = _rank_and_trim_items(fetched_items, active_region=active_region)
await _enqueue_unverified_locations(ranked_fetched_items)
await upsert_earth_news_items(db, ranked_fetched_items)
items = await _call_store_list_items(
@@ -2746,6 +2765,37 @@ async def get_earth_news_payload(
categories=categories,
source_ids=source_ids,
)
if not source_ids:
ready_sources = {
_news_item_source_id(item)
for item in items
if _is_news_item_display_ready(item, locale=locale)
}
missing_ready_sources = [
source.id
for source in sources
if source.id and source.id not in ready_sources
]
if missing_ready_sources:
extra_items: list[ParsedNewsItem] = []
for missing_source_id in missing_ready_sources:
extra_items.extend(
await _call_store_list_items(
list_earth_news_items,
db,
active_region=active_region,
limit=3,
categories=categories,
source_ids={missing_source_id},
)
)
if extra_items:
items = _diversify_news_items_for_locale(
[*items, *extra_items],
active_region=active_region,
limit=limit,
locale=locale,
)
if hasattr(db, "execute"):
cruise_items = await _call_store_list_items(
list_earth_news_cruise_items,
@@ -2756,7 +2806,8 @@ async def get_earth_news_payload(
)
else:
cruise_items = items
await _enqueue_unverified_locations(items)
enqueue_candidates = {item.id: item for item in [*items, *cruise_items] if item.id}
await _enqueue_unverified_locations(list(enqueue_candidates.values()))
stale = bool(errors and items)
return _build_payload(
@@ -0,0 +1,258 @@
"""Classification, importance, and breaking-news policy for Earth news."""
from __future__ import annotations
from dataclasses import dataclass
from datetime import UTC, datetime, timedelta
import re
from typing import Any, Protocol
from app.core.enums import (
BreakingLevel,
BreakingScope,
BreakingSource,
NewsImportanceLevel,
NewsMarketImpact,
NewsTaggingSource,
parse_enum,
)
class NewsItemLike(Protocol):
title: str
summary: str
source: str
feed_name: str
published_at: datetime | None
feed_default_category: str
category: str
item_tags: list[str]
tagging_source: str
tagging_confidence: float
importance_score: int
importance_level: str
importance_reasons: list[str]
market_impact: str
source_tags: list[str]
breaking_level: str
breaking_scope: str
breaking_reasons: list[str]
breaking_source: str
breaking_confidence: float
breaking_expires_at: datetime | None
class NewsSourceLike(Protocol):
default_category: str
importance_weight: int
source_tags: tuple[str, ...]
class NewsFeedLike(Protocol):
default_category: str
@dataclass(frozen=True)
class BreakingRule:
level: BreakingLevel
scope: BreakingScope
reason: str
keywords: tuple[str, ...]
IMPORTANCE_THRESHOLDS: tuple[tuple[int, NewsImportanceLevel], ...] = (
(80, NewsImportanceLevel.CRITICAL),
(60, NewsImportanceLevel.HIGH),
(35, NewsImportanceLevel.MEDIUM),
(0, NewsImportanceLevel.LOW),
)
BREAKING_LEVEL_RANK: dict[BreakingLevel, int] = {
BreakingLevel.NONE: 0,
BreakingLevel.WATCH: 1,
BreakingLevel.BREAKING: 2,
BreakingLevel.CRITICAL: 3,
}
BREAKING_TTL: dict[BreakingLevel, timedelta] = {
BreakingLevel.WATCH: timedelta(hours=6),
BreakingLevel.BREAKING: timedelta(hours=12),
BreakingLevel.CRITICAL: timedelta(hours=24),
}
BREAKING_RULES: tuple[BreakingRule, ...] = (
BreakingRule(BreakingLevel.CRITICAL, BreakingScope.GLOBAL, "核事故或核风险", ("nuclear accident", "nuclear emergency", "radiation leak", "核事故", "核泄漏", "辐射泄漏")),
BreakingRule(BreakingLevel.CRITICAL, BreakingScope.REGIONAL, "重大军事冲突升级", ("airstrike", "missile strike", "invasion", "martial law", "空袭", "导弹袭击", "入侵", "戒严")),
BreakingRule(BreakingLevel.BREAKING, BreakingScope.REGIONAL, "战争或安全事件", ("war escalates", "terror attack", "coup", "hostage", "战争升级", "恐袭", "政变", "人质")),
BreakingRule(BreakingLevel.BREAKING, BreakingScope.REGIONAL, "重大灾害应急", ("major earthquake", "tsunami", "volcanic eruption", "state of emergency", "强震", "海啸", "火山喷发", "紧急状态")),
BreakingRule(BreakingLevel.BREAKING, BreakingScope.GLOBAL, "金融市场异常", ("market halt", "trading halt", "flash crash", "bank run", "金融熔断", "交易暂停", "银行挤兑")),
BreakingRule(BreakingLevel.WATCH, BreakingScope.GLOBAL, "大规模网络安全事件", ("massive cyberattack", "ransomware attack", "data breach", "大规模网络攻击", "勒索软件", "数据泄露")),
BreakingRule(BreakingLevel.WATCH, BreakingScope.REGIONAL, "航天或卫星事故", ("rocket explosion", "satellite collision", "space station emergency", "火箭爆炸", "卫星碰撞", "空间站事故")),
)
def contains_keyword(text: str, keyword: str) -> bool:
keyword_text = str(keyword or "").strip().lower()
if not keyword_text:
return False
if re.search(r"[\u4e00-\u9fff]", keyword_text):
return keyword_text in text
return re.search(rf"(?<![a-z0-9]){re.escape(keyword_text)}(?![a-z0-9])", text) is not None
def score_category(text: str, title_text: str, category: dict[str, Any]) -> int:
score = 0
keywords = category.get("keywords") if isinstance(category.get("keywords"), list) else []
for keyword in keywords:
if contains_keyword(title_text, keyword):
score += 3
elif contains_keyword(text, keyword):
score += 1
return score
def importance_level(score: int) -> NewsImportanceLevel:
normalized_score = max(0, min(100, int(score)))
for threshold, level in IMPORTANCE_THRESHOLDS:
if normalized_score >= threshold:
return level
return NewsImportanceLevel.LOW
def normalize_breaking_level(value: object) -> BreakingLevel:
return parse_enum(BreakingLevel, value, BreakingLevel.NONE)
def normalize_breaking_scope(value: object) -> BreakingScope:
return parse_enum(BreakingScope, value, BreakingScope.REGIONAL)
def breaking_expires_at(level: object, published_at: datetime | None) -> datetime | None:
normalized = normalize_breaking_level(level)
if normalized is BreakingLevel.NONE:
return None
base = published_at or datetime.now(UTC)
base = base.replace(tzinfo=UTC) if base.tzinfo is None else base.astimezone(UTC)
return base + BREAKING_TTL[normalized]
def is_breaking_active(item: NewsItemLike, *, now: datetime | None = None) -> bool:
if normalize_breaking_level(item.breaking_level) is BreakingLevel.NONE:
return False
expires_at = item.breaking_expires_at
if expires_at is None:
return True
expires_at = expires_at.replace(tzinfo=UTC) if expires_at.tzinfo is None else expires_at.astimezone(UTC)
return expires_at > (now or datetime.now(UTC))
def breaking_sort_rank(item: NewsItemLike) -> int:
if not is_breaking_active(item):
return 0
return BREAKING_LEVEL_RANK[normalize_breaking_level(item.breaking_level)]
def highest_breaking_level(items: list[NewsItemLike]) -> BreakingLevel:
active = [normalize_breaking_level(item.breaking_level) for item in items if is_breaking_active(item)]
return max(active, key=BREAKING_LEVEL_RANK.get) if active else BreakingLevel.NONE
def apply_breaking_rules(item: NewsItemLike) -> None:
combined_text = f"{item.title} {item.summary} {item.source} {item.feed_name}".lower()
best_level = BreakingLevel.NONE
best_scope = BreakingScope.REGIONAL
reasons: list[str] = []
confidence = 0.0
for rule in BREAKING_RULES:
if not any(contains_keyword(combined_text, keyword) for keyword in rule.keywords):
continue
if BREAKING_LEVEL_RANK[rule.level] > BREAKING_LEVEL_RANK[best_level]:
best_level = rule.level
best_scope = rule.scope
if rule.reason not in reasons:
reasons.append(rule.reason)
confidence = max(confidence, 0.72 if rule.level is BreakingLevel.CRITICAL else 0.64 if rule.level is BreakingLevel.BREAKING else 0.52)
item.breaking_level = best_level.value
item.breaking_scope = (best_scope if best_level is not BreakingLevel.NONE else BreakingScope.REGIONAL).value
item.breaking_reasons = reasons
item.breaking_source = BreakingSource.RULES.value
item.breaking_confidence = round(confidence, 2)
item.breaking_expires_at = breaking_expires_at(best_level, item.published_at)
def apply_news_classification(
item: NewsItemLike,
source: NewsSourceLike,
*,
feed: NewsFeedLike | None,
config: dict[str, Any],
) -> NewsItemLike:
title_text = item.title.lower()
combined_text = f"{item.title} {item.summary} {item.source} {item.feed_name}".lower()
feed_default_category = (feed.default_category if feed else item.feed_default_category) or source.default_category or "other"
best_key = feed_default_category
best_score = second_score = 0
for category in config["categories"]:
if not isinstance(category, dict) or category.get("enabled") is False:
continue
score = score_category(combined_text, title_text, category)
if score > best_score:
second_score, best_score = best_score, score
best_key = str(category.get("key") or "other")
elif score > second_score:
second_score = score
item_tags: list[str] = []
for rule in config["item_tag_rules"]:
if not isinstance(rule, dict):
continue
keywords = rule.get("keywords") if isinstance(rule.get("keywords"), list) else []
if any(contains_keyword(combined_text, keyword) for keyword in keywords):
tag_key = str(rule.get("key") or "").strip()
if tag_key and tag_key not in item_tags:
item_tags.append(tag_key)
if best_score < 3 and rule.get("category"):
best_key, best_score = str(rule["category"]), 3
confidence = round(best_score / (best_score + second_score + 1), 2) if best_score else 0.35
if best_score < 3 and feed_default_category:
best_key, confidence = feed_default_category, 0.45
score = max(0, min(100, 18 + source.importance_weight + best_score * 6))
reasons: list[str] = []
source_tags = set(source.source_tags)
if "official_data" in source_tags:
score += 20
reasons.append("官方数据源")
if "press_release" in source_tags:
score = max(0, score - 12)
reasons.append("企业公告基础权重较低")
if any(contains_keyword(combined_text, term) for term in ("网上零售额", "电商物流指数", "gmv", "订单量", "物流指数", "履约", "直播电商", "跨境电商")):
score += 25
reasons.append("命中电商数据指标")
if any(contains_keyword(combined_text, term) for term in ("amazon", "shopify", "walmart", "alibaba", "jd.com", "pinduoduo", "tiktok shop", "shein", "阿里", "京东", "拼多多", "抖音")):
score += 15
reasons.append("涉及大型平台")
if any(term in combined_text for term in ("同比", "环比", "%", "billion", "million", "增长", "下降")):
score += 10
reasons.append("包含量化指标")
score = max(0, min(100, score))
item.category = best_key or "other"
item.item_tags = item_tags
item.tagging_source = NewsTaggingSource.RULES.value
item.tagging_confidence = confidence
item.importance_score = score
item.importance_level = importance_level(score).value
item.importance_reasons = reasons or ["按来源权重和分类规则计算"]
item.market_impact = (
NewsMarketImpact.GLOBAL.value
if "global" in source_tags
else NewsMarketImpact.NATIONAL.value
if {"china", "us"} & source_tags
else NewsMarketImpact.SECTOR.value
)
item.source_tags = list(source.source_tags)
apply_breaking_rules(item)
return item
+693
View File
@@ -0,0 +1,693 @@
from __future__ import annotations
from dataclasses import dataclass
from datetime import UTC, datetime
import hashlib
import html
import json
import re
from typing import Any
from bs4 import BeautifulSoup
from sqlalchemy import delete, func, select
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.enums import NewsEnrichmentStatus, NewsSourceType, NewsTaggingSource
from app.core.websocket.broadcaster import broadcaster
from app.models.earth_news import EarthNewsItem
from app.models.system_setting import SystemSetting
from app.services.earth_news import (
ALLOWED_NEWS_CATEGORY_KEYS,
DEFAULT_NEWS_LOCALE,
REGION_ANCHORS,
NewsFeedEndpoint,
NewsFeedSource,
NewsTargetLocation,
ParsedNewsItem,
apply_news_classification,
build_anchor_location_patch,
build_target_location_job_payload,
build_target_location_patch,
_serialize_item,
)
from app.services.earth_news_queue import enqueue_target_location_job
from app.services.earth_news_store import record_to_parsed_news_item
MANUAL_NEWS_SOURCE_ID = "manual"
MANUAL_NEWS_SOURCE_LABEL = "手动添加"
MANUAL_NEWS_MAX_IMPORT_ITEMS = 500
MANUAL_NEWS_MAX_TITLE_LENGTH = 500
MANUAL_NEWS_MAX_SUMMARY_LENGTH = 1200
MANUAL_NEWS_MAX_CONTENT_LENGTH = 12000
EARTH_NEWS_MANUAL_GROUPS_CATEGORY = "earth_news_manual_groups"
DEFAULT_MANUAL_NEWS_GROUP_ID = "manual-default"
DEFAULT_MANUAL_NEWS_GROUP_NAME = "新建新闻组"
@dataclass(frozen=True)
class ManualNewsWriteResult:
item: EarthNewsItem
created: bool
queued: bool
@dataclass(frozen=True)
class ManualNewsGroup:
id: str
name: str
sort_order: int = 0
def _clean_text(value: object, *, max_length: int) -> str:
raw = "" if value is None else str(value)
text = BeautifulSoup(html.unescape(raw), "html.parser").get_text(" ", strip=True)
text = re.sub(r"\s+", " ", text).strip()
if len(text) > max_length:
return text[: max_length - 1].rstrip() + "…"
return text
def _parse_datetime(value: object) -> datetime | None:
if value is None or str(value).strip() == "":
return None
if isinstance(value, datetime):
parsed = value
else:
try:
parsed = datetime.fromisoformat(str(value).strip().replace("Z", "+00:00"))
except ValueError as exc:
raise ValueError("published_at 必须是 ISO8601 时间。") from exc
if parsed.tzinfo is None:
return parsed.replace(tzinfo=UTC)
return parsed.astimezone(UTC)
def _detect_language(*parts: str) -> str:
text = " ".join(part for part in parts if part)
cjk_count = len(re.findall(r"[\u4e00-\u9fff]", text))
latin_count = len(re.findall(r"[A-Za-z]", text))
return "zh-CN" if cjk_count >= max(4, latin_count // 3) else "en-US"
def _manual_item_id(*, title: str, published_at: datetime | None, url: str, source: str) -> str:
published = published_at.isoformat() if published_at else ""
basis = "\n".join([title.strip().lower(), published, url.strip().lower(), source.strip().lower()])
return f"manual:{hashlib.sha1(basis.encode('utf-8')).hexdigest()[:16]}"
def _manual_group_id(name: str) -> str:
basis = f"{name.strip().lower()}\n{datetime.now(UTC).isoformat()}"
return f"manual-group:{hashlib.sha1(basis.encode('utf-8')).hexdigest()[:10]}"
def _news_meta(record: EarthNewsItem) -> dict[str, Any]:
location_meta = record.location_meta if isinstance(record.location_meta, dict) else {}
news_meta = location_meta.get("news_meta")
return dict(news_meta) if isinstance(news_meta, dict) else {}
def _record_source_type(record: EarthNewsItem) -> str:
return str(_news_meta(record).get("feed_type") or _news_meta(record).get("source_type") or "rss")
def _record_manual_group_id(record: EarthNewsItem) -> str:
return str(_news_meta(record).get("manual_group_id") or DEFAULT_MANUAL_NEWS_GROUP_ID)
def _rss_group_id(record: EarthNewsItem) -> str:
basis = "\n".join(
[
_record_source_type(record),
str(record.feed_name or ""),
str(record.source or ""),
]
)
return f"rss:{hashlib.sha1(basis.encode('utf-8')).hexdigest()[:12]}"
def _default_manual_group() -> dict[str, Any]:
return {
"id": DEFAULT_MANUAL_NEWS_GROUP_ID,
"name": DEFAULT_MANUAL_NEWS_GROUP_NAME,
"sort_order": 0,
}
def _normalize_manual_groups_payload(payload: Any) -> list[dict[str, Any]]:
raw_groups = payload.get("groups") if isinstance(payload, dict) else None
normalized: list[dict[str, Any]] = []
seen: set[str] = set()
for index, item in enumerate(raw_groups if isinstance(raw_groups, list) else []):
if not isinstance(item, dict):
continue
group_id = str(item.get("id") or "").strip()
name = _clean_text(item.get("name"), max_length=120)
if not group_id or not name or group_id in seen:
continue
normalized.append(
{
"id": group_id,
"name": name,
"sort_order": int(item.get("sort_order") or index),
}
)
seen.add(group_id)
if DEFAULT_MANUAL_NEWS_GROUP_ID not in seen:
normalized.insert(0, _default_manual_group())
return sorted(normalized, key=lambda item: (int(item.get("sort_order") or 0), str(item.get("name") or "")))
async def _get_manual_groups_record(db: AsyncSession) -> SystemSetting | None:
result = await db.execute(
select(SystemSetting).where(SystemSetting.category == EARTH_NEWS_MANUAL_GROUPS_CATEGORY)
)
return result.scalar_one_or_none()
async def get_manual_news_groups(db: AsyncSession) -> list[dict[str, Any]]:
record = await _get_manual_groups_record(db)
return _normalize_manual_groups_payload(record.payload if record else None)
async def _save_manual_news_groups(db: AsyncSession, groups: list[dict[str, Any]]) -> list[dict[str, Any]]:
normalized = _normalize_manual_groups_payload({"groups": groups})
record = await _get_manual_groups_record(db)
payload = {"groups": normalized}
if record is None:
db.add(SystemSetting(category=EARTH_NEWS_MANUAL_GROUPS_CATEGORY, payload=payload))
else:
record.payload = payload
await db.flush()
return normalized
async def resolve_manual_news_group(db: AsyncSession, group_id: str | None) -> ManualNewsGroup:
normalized_id = str(group_id or DEFAULT_MANUAL_NEWS_GROUP_ID).strip() or DEFAULT_MANUAL_NEWS_GROUP_ID
groups = await get_manual_news_groups(db)
match = next((item for item in groups if item.get("id") == normalized_id), None)
if match is None and normalized_id != DEFAULT_MANUAL_NEWS_GROUP_ID:
raise ValueError(f"手动新闻组不存在:{normalized_id}")
match = match or _default_manual_group()
return ManualNewsGroup(
id=str(match["id"]),
name=str(match["name"]),
sort_order=int(match.get("sort_order") or 0),
)
async def create_manual_news_group(db: AsyncSession, name: str) -> dict[str, Any]:
group_name = _clean_text(name, max_length=120)
if not group_name:
raise ValueError("新闻组名称不能为空。")
groups = await get_manual_news_groups(db)
group = {"id": _manual_group_id(group_name), "name": group_name, "sort_order": len(groups)}
groups.append(group)
await _save_manual_news_groups(db, groups)
return group
async def rename_manual_news_group(db: AsyncSession, group_id: str, name: str) -> dict[str, Any]:
group_name = _clean_text(name, max_length=120)
if not group_name:
raise ValueError("新闻组名称不能为空。")
groups = await get_manual_news_groups(db)
match = next((item for item in groups if item.get("id") == group_id), None)
if match is None:
raise ValueError(f"手动新闻组不存在:{group_id}")
match["name"] = group_name
await _save_manual_news_groups(db, groups)
result = await db.execute(
select(EarthNewsItem).where(
EarthNewsItem.location_meta.op("->")("news_meta").op("->>")("feed_type") == NewsSourceType.MANUAL.value
)
)
for record in result.scalars().all():
if _record_manual_group_id(record) != group_id:
continue
location_meta = dict(record.location_meta or {})
news_meta = dict(location_meta.get("news_meta") or {})
news_meta["manual_group_name"] = group_name
location_meta["news_meta"] = news_meta
record.location_meta = location_meta
await db.flush()
return match
def _normalize_region(value: object) -> str:
region = str(value or "global").strip().lower() or "global"
if region not in REGION_ANCHORS:
raise ValueError(f"region 不支持:{region}")
return region
def _normalize_tags(value: object) -> list[str]:
if value is None:
return []
if isinstance(value, str):
parts = re.split(r"[,,\n]", value)
elif isinstance(value, list):
parts = [str(item) for item in value]
else:
raise ValueError("tags 必须是字符串数组或逗号分隔字符串。")
return [item.strip() for item in parts if item.strip()][:20]
def _normalize_location(value: object) -> NewsTargetLocation | None:
if value in (None, ""):
return None
if not isinstance(value, dict):
raise ValueError("location 必须是对象。")
lat = value.get("latitude")
lon = value.get("longitude")
if lat in (None, "") and lon in (None, ""):
return None
try:
latitude = float(lat)
longitude = float(lon)
except (TypeError, ValueError) as exc:
raise ValueError("location.latitude / longitude 必须是数字。") from exc
if not -90 <= latitude <= 90 or not -180 <= longitude <= 180:
raise ValueError("location 经纬度超出范围。")
label = _clean_text(value.get("label"), max_length=255)
if not label:
label = f"{latitude:.4f}, {longitude:.4f}"
return NewsTargetLocation(
latitude=latitude,
longitude=longitude,
label=label,
source="manual_location",
confidence=1.0,
country=_clean_text(value.get("country"), max_length=100) or None,
city=_clean_text(value.get("city"), max_length=100) or None,
)
def _manual_source(source_name: str, *, region: str) -> NewsFeedSource:
return NewsFeedSource(
id=MANUAL_NEWS_SOURCE_ID,
name=source_name or MANUAL_NEWS_SOURCE_LABEL,
region=region,
feed_url="",
homepage_url="",
source_type=NewsSourceType.MANUAL.value,
default_category="other",
source_tags=("manual",),
)
def _manual_feed(category: str) -> NewsFeedEndpoint:
return NewsFeedEndpoint(
id=MANUAL_NEWS_SOURCE_ID,
name=MANUAL_NEWS_SOURCE_LABEL,
url="",
type=NewsSourceType.MANUAL.value,
default_category=category or "other",
tags=("manual",),
priority=1,
)
def parsed_manual_news_item(
payload: dict[str, Any],
*,
item_id_override: str | None = None,
) -> tuple[ParsedNewsItem, NewsTargetLocation | None, str]:
title = _clean_text(payload.get("title"), max_length=MANUAL_NEWS_MAX_TITLE_LENGTH)
if not title:
raise ValueError("title 不能为空。")
content = _clean_text(payload.get("content"), max_length=MANUAL_NEWS_MAX_CONTENT_LENGTH)
summary = _clean_text(payload.get("summary"), max_length=MANUAL_NEWS_MAX_SUMMARY_LENGTH)
if not summary:
summary = _clean_text(content, max_length=240) if content else title
source = _clean_text(payload.get("source"), max_length=255) or MANUAL_NEWS_SOURCE_LABEL
region = _normalize_region(payload.get("region"))
published_at = _parse_datetime(payload.get("published_at")) or datetime.now(UTC)
url = str(payload.get("url") or "").strip()
category = str(payload.get("category") or "other").strip().lower() or "other"
if category not in ALLOWED_NEWS_CATEGORY_KEYS:
raise ValueError(f"category 不支持:{category}")
tags = _normalize_tags(payload.get("tags"))
target = _normalize_location(payload.get("location"))
language = str(payload.get("content_language") or "").strip() or _detect_language(title, summary, content)
localizations = {
language: {
"title": title,
"summary": summary,
}
}
item = ParsedNewsItem(
id=item_id_override
or _manual_item_id(title=title, published_at=published_at, url=url, source=source),
title=title,
summary=summary,
url=url,
source=source,
feed_name=MANUAL_NEWS_SOURCE_LABEL,
feed_region=region,
homepage_url=str(payload.get("homepage_url") or ""),
published_at=published_at,
content_language=language,
localizations=localizations,
enrichment_status=NewsEnrichmentStatus.PENDING.value,
source_tags=["manual"],
feed_id=MANUAL_NEWS_SOURCE_ID,
feed_type=NewsSourceType.MANUAL.value,
feed_default_category=category,
category=category,
item_tags=tags,
tagging_source=NewsTaggingSource.MANUAL.value if payload.get("category") else NewsTaggingSource.RULES.value,
tagging_confidence=0.9 if payload.get("category") else 0.0,
)
source_config = _manual_source(source, region=region)
feed = _manual_feed(category)
apply_news_classification(item, source_config, feed=feed)
if payload.get("category"):
item.category = category
item.tagging_source = NewsTaggingSource.MANUAL.value
item.tagging_confidence = 0.9
if tags:
item.item_tags = sorted(set([*item.item_tags, *tags]))
return item, target, content
def _manual_editable(record: EarthNewsItem) -> bool:
if record.id.startswith("manual:"):
return True
news_meta = (record.location_meta or {}).get("news_meta") if isinstance(record.location_meta, dict) else None
return isinstance(news_meta, dict) and news_meta.get("feed_type") == NewsSourceType.MANUAL.value
async def _broadcast_news_reload() -> None:
await broadcaster.broadcast_earth_update(
{
"action": "database_changed",
"source": "earth_news_items",
"layers": ["news"],
"refresh_strategy": "reload",
}
)
async def upsert_manual_news_item(
db: AsyncSession,
payload: dict[str, Any],
*,
item_id_override: str | None = None,
group_id: str | None = None,
) -> ManualNewsWriteResult:
item, target, content = parsed_manual_news_item(payload, item_id_override=item_id_override)
group = await resolve_manual_news_group(db, group_id or payload.get("group_id"))
existing = await db.get(EarthNewsItem, item.id)
created = existing is None
patch = build_target_location_patch(item, target) if target else build_anchor_location_patch(item)
patch_meta = dict(patch.get("location_meta") or {})
patch_news_meta = dict(patch_meta.get("news_meta") or {})
patch_news_meta["feed_type"] = NewsSourceType.MANUAL.value
patch_news_meta["source_type"] = NewsSourceType.MANUAL.value
patch_news_meta["manual_group_id"] = group.id
patch_news_meta["manual_group_name"] = group.name
patch_meta["news_meta"] = patch_news_meta
patch["location_meta"] = patch_meta
now = datetime.now(UTC)
record = existing or EarthNewsItem(
id=item.id,
title=item.title,
summary=item.summary,
content_language=item.content_language,
localizations=dict(item.localizations or {}),
url=item.url,
source=item.source,
feed_name=item.feed_name,
region=item.feed_region,
homepage_url=item.homepage_url,
published_at=item.published_at,
latitude=patch["latitude"],
longitude=patch["longitude"],
location_label=patch["location_label"],
location_source=patch["location_source"],
verified=patch["verified"],
location_meta=patch["location_meta"],
first_seen_at=now,
last_seen_at=now,
resolved_at=now if patch["verified"] else None,
enrichment_status=item.enrichment_status,
)
if existing is None:
db.add(record)
else:
if not _manual_editable(record):
raise PermissionError("RSS 新闻不允许通过手动新闻接口编辑。")
record.title = item.title
record.summary = item.summary
record.content_language = item.content_language
record.localizations = dict(item.localizations or {})
record.url = item.url
record.source = item.source
record.feed_name = item.feed_name
record.region = item.feed_region
record.homepage_url = item.homepage_url
record.published_at = item.published_at
record.last_seen_at = now
if target is None and record.location_source == "manual_location":
merged_meta = dict(record.location_meta or {})
patch_meta = patch.get("location_meta") if isinstance(patch, dict) else None
patch_news_meta = patch_meta.get("news_meta") if isinstance(patch_meta, dict) else None
if isinstance(patch_news_meta, dict):
merged_meta["news_meta"] = patch_news_meta
record.location_meta = merged_meta
else:
record.location_meta = patch["location_meta"]
if target:
record.latitude = patch["latitude"]
record.longitude = patch["longitude"]
record.location_label = patch["location_label"]
record.location_source = patch["location_source"]
record.verified = patch["verified"]
record.resolved_at = now
elif record.location_source != "manual_location":
record.latitude = patch["latitude"]
record.longitude = patch["longitude"]
record.location_label = patch["location_label"]
record.location_source = patch["location_source"]
record.verified = patch["verified"]
record.resolved_at = None
record.enrichment_status = NewsEnrichmentStatus.PENDING.value
record.enrichment_error = None
record.enriched_at = None
if content:
meta = dict(record.location_meta or {})
meta["manual_content"] = content
record.location_meta = meta
await db.flush()
queued = await enqueue_target_location_job(build_target_location_job_payload(item), force=True)
if queued:
record.enrichment_status = NewsEnrichmentStatus.QUEUED.value
await db.flush()
return ManualNewsWriteResult(item=record, created=created, queued=queued)
async def import_manual_news_items(
db: AsyncSession,
payload: list[Any],
*,
group_id: str | None = None,
) -> dict[str, Any]:
if len(payload) > MANUAL_NEWS_MAX_IMPORT_ITEMS:
raise ValueError(f"单次最多导入 {MANUAL_NEWS_MAX_IMPORT_ITEMS} 条。")
created = 0
updated = 0
queued = 0
errors: list[dict[str, Any]] = []
for index, raw_item in enumerate(payload):
if not isinstance(raw_item, dict):
errors.append({"index": index, "error": "条目必须是 JSON 对象。"})
continue
try:
result = await upsert_manual_news_item(db, raw_item, group_id=group_id)
created += 1 if result.created else 0
updated += 0 if result.created else 1
queued += 1 if result.queued else 0
except Exception as exc:
errors.append({"index": index, "error": str(exc)})
if errors and created == 0 and updated == 0:
raise ValueError("导入失败,未写入任何新闻。")
return {"created": created, "updated": updated, "queued": queued, "failed": len(errors), "errors": errors}
async def parse_manual_news_import_upload(raw_bytes: bytes) -> list[Any]:
try:
payload = json.loads(raw_bytes.decode("utf-8-sig"))
except UnicodeDecodeError as exc:
raise ValueError("JSON 文件必须使用 UTF-8 编码。") from exc
except json.JSONDecodeError as exc:
raise ValueError(f"JSON 解析失败:第 {exc.lineno} 行第 {exc.colno} 列。") from exc
if not isinstance(payload, list):
raise ValueError("JSON 顶层必须是数组。")
return payload
def serialize_news_record(record: EarthNewsItem, *, locale: str = DEFAULT_NEWS_LOCALE) -> dict[str, Any]:
item = record_to_parsed_news_item(record)
payload = _serialize_item(item, active_region=item.feed_region, locale=locale)
news_meta = _news_meta(record)
payload["editable"] = _manual_editable(record)
payload["source_type"] = payload.get("feed_type")
payload["status"] = record.enrichment_status
payload["translated"] = bool((record.localizations or {}).get("zh-CN") and (record.localizations or {}).get("en-US"))
payload["manual_content"] = (record.location_meta or {}).get("manual_content") if isinstance(record.location_meta, dict) else None
payload["manual_group_id"] = news_meta.get("manual_group_id")
payload["manual_group_name"] = news_meta.get("manual_group_name")
return payload
def _record_matches_group(record: EarthNewsItem, group_id: str) -> bool:
source_type = _record_source_type(record)
if source_type == NewsSourceType.MANUAL.value:
return _record_manual_group_id(record) == group_id
return _rss_group_id(record) == group_id
async def list_news_records(
db: AsyncSession,
*,
page: int,
page_size: int,
source_type: str | None = None,
region: str | None = None,
category: str | None = None,
status_filter: str | None = None,
group_id: str | None = None,
) -> dict[str, Any]:
page = max(page, 1)
page_size = min(max(page_size, 1), 100)
query = select(EarthNewsItem)
count_query = select(func.count(EarthNewsItem.id))
filters = []
if region and region != "all":
filters.append(EarthNewsItem.region == region)
if status_filter and status_filter != "all":
filters.append(EarthNewsItem.enrichment_status == status_filter)
if source_type and source_type != "all":
filters.append(EarthNewsItem.location_meta.op("->")("news_meta").op("->>")("feed_type") == source_type)
if category and category != "all":
filters.append(EarthNewsItem.location_meta.op("->")("news_meta").op("->>")("category") == category)
for clause in filters:
query = query.where(clause)
count_query = count_query.where(clause)
ordered_query = query.order_by(EarthNewsItem.published_at.desc().nullslast(), EarthNewsItem.last_seen_at.desc())
if group_id:
result = await db.execute(ordered_query)
all_records = [record for record in result.scalars().all() if _record_matches_group(record, group_id)]
total = len(all_records)
records = all_records[(page - 1) * page_size : page * page_size]
else:
total_result = await db.execute(count_query)
result = await db.execute(
ordered_query.offset((page - 1) * page_size).limit(page_size)
)
records = list(result.scalars().all())
total = int(total_result.scalar() or 0)
return {
"items": [serialize_news_record(record) for record in records],
"page": page,
"page_size": page_size,
"total": total,
}
async def list_news_groups(db: AsyncSession, *, locale: str = DEFAULT_NEWS_LOCALE) -> dict[str, Any]:
manual_groups = await get_manual_news_groups(db)
manual_by_id: dict[str, dict[str, Any]] = {
str(group["id"]): {
"id": str(group["id"]),
"name": str(group["name"]),
"group_type": "manual",
"source_type": NewsSourceType.MANUAL.value,
"editable": True,
"sort_order": int(group.get("sort_order") or 0),
"count": 0,
"items": [],
}
for group in manual_groups
}
rss_by_id: dict[str, dict[str, Any]] = {}
result = await db.execute(
select(EarthNewsItem).order_by(EarthNewsItem.published_at.desc().nullslast(), EarthNewsItem.last_seen_at.desc())
)
for record in result.scalars().all():
serialized = serialize_news_record(record, locale=locale)
source_type = _record_source_type(record)
if source_type == NewsSourceType.MANUAL.value:
group_id = _record_manual_group_id(record)
group = manual_by_id.setdefault(
group_id,
{
"id": group_id,
"name": str(_news_meta(record).get("manual_group_name") or DEFAULT_MANUAL_NEWS_GROUP_NAME),
"group_type": "manual",
"source_type": NewsSourceType.MANUAL.value,
"editable": True,
"sort_order": len(manual_by_id),
"count": 0,
"items": [],
},
)
else:
group_id = _rss_group_id(record)
group = rss_by_id.setdefault(
group_id,
{
"id": group_id,
"name": record.feed_name or record.source or "RSS 新闻",
"group_type": "rss",
"source_type": source_type,
"editable": False,
"region": record.region,
"source": record.source,
"feed_name": record.feed_name,
"count": 0,
"items": [],
},
)
group["count"] = int(group.get("count") or 0) + 1
group.setdefault("items", []).append(serialized)
manual_items = sorted(manual_by_id.values(), key=lambda item: (int(item.get("sort_order") or 0), str(item.get("name") or "")))
rss_items = sorted(rss_by_id.values(), key=lambda item: str(item.get("name") or ""))
return {"groups": [*manual_items, *rss_items], "manual_groups": manual_items, "rss_groups": rss_items}
async def get_news_record_or_404(db: AsyncSession, item_id: str) -> EarthNewsItem | None:
return await db.get(EarthNewsItem, item_id)
async def delete_manual_news_item(db: AsyncSession, item_id: str) -> bool:
record = await db.get(EarthNewsItem, item_id)
if record is None:
return False
if not _manual_editable(record):
raise PermissionError("RSS 新闻不允许通过手动新闻接口删除。")
await db.execute(delete(EarthNewsItem).where(EarthNewsItem.id == item_id))
await db.flush()
return True
async def reprocess_manual_news_item(db: AsyncSession, item_id: str) -> bool:
record = await db.get(EarthNewsItem, item_id)
if record is None:
return False
if not _manual_editable(record):
raise PermissionError("RSS 新闻不允许通过手动新闻接口重新处理。")
item = record_to_parsed_news_item(record)
queued = await enqueue_target_location_job(build_target_location_job_payload(item), force=True)
if queued:
record.enrichment_status = NewsEnrichmentStatus.QUEUED.value
record.enrichment_error = None
await db.flush()
return queued
async def broadcast_manual_news_changed() -> None:
await _broadcast_news_reload()
+128 -44
View File
@@ -14,10 +14,14 @@ from app.core.logging import get_logger
logger = get_logger(__name__, service="earth_news")
TARGET_LOCATION_STREAM = "earth_news:target_location:jobs"
TARGET_LOCATION_PRIORITY_STREAM = "earth_news:target_location:priority"
TARGET_LOCATION_GROUP = "earth_news_target_location"
TARGET_LOCATION_DEAD_LETTER_STREAM = "earth_news:target_location:dead"
TARGET_LOCATION_RESULT_TTL_SECONDS = 60 * 60 * 12
TARGET_LOCATION_JOB_DEDUP_TTL_SECONDS = 60 * 60 * 6
TARGET_LOCATION_PRIORITY_JOB_DEDUP_TTL_SECONDS = 60 * 5
TARGET_LOCATION_PENDING_RECLAIM_IDLE_MS = 2 * 60 * 1000
TARGET_LOCATION_PRIORITY_READ_BLOCK_MS = 1
TARGET_LOCATION_MAX_ATTEMPTS = 3
_redis_client: redis.Redis | None = None
@@ -28,6 +32,7 @@ class NewsTargetLocationMessage:
message_id: str
item_id: str
payload: dict[str, Any]
stream_name: str = TARGET_LOCATION_STREAM
attempts: int = 0
@@ -44,7 +49,7 @@ class NewsTargetLocationQueue(Protocol):
) -> list[NewsTargetLocationMessage]:
...
async def ack(self, message_id: str) -> None:
async def ack(self, message: NewsTargetLocationMessage) -> None:
...
async def retry_or_dead_letter(
@@ -71,6 +76,10 @@ def _queued_key(item_id: str) -> str:
return f"earth_news:target_location:queued:{item_id}"
def _priority_queued_key(item_id: str) -> str:
return f"earth_news:target_location:priority_queued:{item_id}"
class RedisStreamsNewsTargetLocationQueue:
def __init__(self, client: redis.Redis | None = None) -> None:
self.client = client or _get_redis_client()
@@ -79,34 +88,44 @@ class RedisStreamsNewsTargetLocationQueue:
async def _ensure_group(self) -> None:
if self._group_ready:
return
try:
await self.client.xgroup_create(
TARGET_LOCATION_STREAM,
TARGET_LOCATION_GROUP,
id="0",
mkstream=True,
)
except ResponseError as exc:
if "BUSYGROUP" not in str(exc):
raise
for stream_name in (TARGET_LOCATION_PRIORITY_STREAM, TARGET_LOCATION_STREAM):
try:
await self.client.xgroup_create(
stream_name,
TARGET_LOCATION_GROUP,
id="0",
mkstream=True,
)
except ResponseError as exc:
if "BUSYGROUP" not in str(exc):
raise
self._group_ready = True
async def enqueue(self, *, item_id: str, payload: dict[str, Any], force: bool = False) -> bool:
await self._ensure_group()
if force:
await self.client.delete(_result_key(item_id), _queued_key(item_id))
await self.client.delete(_result_key(item_id))
queued_key = _priority_queued_key(item_id)
elif await self.client.exists(_result_key(item_id)):
return False
else:
queued_key = _queued_key(item_id)
dedup_ttl = (
TARGET_LOCATION_PRIORITY_JOB_DEDUP_TTL_SECONDS
if force
else TARGET_LOCATION_JOB_DEDUP_TTL_SECONDS
)
queued = await self.client.set(
_queued_key(item_id),
queued_key,
"1",
nx=True,
ex=TARGET_LOCATION_JOB_DEDUP_TTL_SECONDS,
ex=dedup_ttl,
)
if not queued:
return bool(await self.client.exists(_queued_key(item_id)))
return bool(await self.client.exists(queued_key))
stream_name = TARGET_LOCATION_PRIORITY_STREAM if force else TARGET_LOCATION_STREAM
await self.client.xadd(
TARGET_LOCATION_STREAM,
stream_name,
{
"item_id": item_id,
"attempts": "0",
@@ -123,39 +142,104 @@ class RedisStreamsNewsTargetLocationQueue:
block_ms: int,
) -> list[NewsTargetLocationMessage]:
await self._ensure_group()
streams = await self.client.xreadgroup(
streams = []
priority_claimed = await self._claim_stale_messages(
stream_name=TARGET_LOCATION_PRIORITY_STREAM,
consumer_name=consumer_name,
count=count,
)
if priority_claimed:
return priority_claimed
priority_messages = await self.client.xreadgroup(
TARGET_LOCATION_GROUP,
consumer_name,
{TARGET_LOCATION_STREAM: ">"},
{TARGET_LOCATION_PRIORITY_STREAM: ">"},
count=count,
block=block_ms,
block=TARGET_LOCATION_PRIORITY_READ_BLOCK_MS,
)
if priority_messages:
streams = priority_messages
else:
regular_claimed = await self._claim_stale_messages(
stream_name=TARGET_LOCATION_STREAM,
consumer_name=consumer_name,
count=count,
)
if regular_claimed:
return regular_claimed
streams = await self.client.xreadgroup(
TARGET_LOCATION_GROUP,
consumer_name,
{TARGET_LOCATION_STREAM: ">"},
count=count,
block=block_ms,
)
messages: list[NewsTargetLocationMessage] = []
for _stream_name, stream_messages in streams:
for stream_name, stream_messages in streams:
for message_id, fields in stream_messages:
raw_payload = fields.get("payload")
item_id = fields.get("item_id")
if not raw_payload or not item_id:
await self.ack(message_id)
continue
try:
payload = json.loads(raw_payload)
except json.JSONDecodeError:
await self.ack(message_id)
continue
attempts = int(fields.get("attempts") or 0)
messages.append(
NewsTargetLocationMessage(
message_id=message_id,
item_id=item_id,
payload=payload,
attempts=attempts,
)
)
message = await self._message_from_fields(stream_name, message_id, fields)
if message is not None:
messages.append(message)
return messages
async def ack(self, message_id: str) -> None:
await self.client.xack(TARGET_LOCATION_STREAM, TARGET_LOCATION_GROUP, message_id)
async def _claim_stale_messages(
self,
*,
stream_name: str,
consumer_name: str,
count: int,
) -> list[NewsTargetLocationMessage]:
try:
_next_id, claimed, _deleted = await self.client.xautoclaim(
stream_name,
TARGET_LOCATION_GROUP,
consumer_name,
TARGET_LOCATION_PENDING_RECLAIM_IDLE_MS,
start_id="0-0",
count=count,
)
except ResponseError:
return []
messages: list[NewsTargetLocationMessage] = []
for message_id, fields in claimed:
message = await self._message_from_fields(stream_name, message_id, fields)
if message is not None:
messages.append(message)
return messages
async def _message_from_fields(
self,
stream_name: str,
message_id: str,
fields: dict[str, str],
) -> NewsTargetLocationMessage | None:
raw_payload = fields.get("payload")
item_id = fields.get("item_id")
if not raw_payload or not item_id:
await self._discard_message(stream_name, message_id)
return None
try:
payload = json.loads(raw_payload)
except json.JSONDecodeError:
await self._discard_message(stream_name, message_id)
return None
attempts = int(fields.get("attempts") or 0)
return NewsTargetLocationMessage(
message_id=message_id,
item_id=item_id,
payload=payload,
stream_name=stream_name,
attempts=attempts,
)
async def ack(self, message: NewsTargetLocationMessage) -> None:
await self.client.xack(message.stream_name, TARGET_LOCATION_GROUP, message.message_id)
await self.client.xdel(message.stream_name, message.message_id)
async def _discard_message(self, stream_name: str, message_id: str) -> None:
await self.client.xack(stream_name, TARGET_LOCATION_GROUP, message_id)
await self.client.xdel(stream_name, message_id)
async def retry_or_dead_letter(
self,
@@ -163,7 +247,7 @@ class RedisStreamsNewsTargetLocationQueue:
*,
error: str,
) -> None:
await self.ack(message.message_id)
await self.ack(message)
if message.attempts + 1 >= TARGET_LOCATION_MAX_ATTEMPTS:
await self.client.xadd(
TARGET_LOCATION_DEAD_LETTER_STREAM,
@@ -176,7 +260,7 @@ class RedisStreamsNewsTargetLocationQueue:
)
return
await self.client.xadd(
TARGET_LOCATION_STREAM,
message.stream_name,
{
"item_id": message.item_id,
"attempts": str(message.attempts + 1),
@@ -231,4 +315,4 @@ async def save_target_location_patch(item_id: str, patch: dict[str, Any]) -> Non
TARGET_LOCATION_RESULT_TTL_SECONDS,
json.dumps(patch, ensure_ascii=False),
)
await client.delete(_queued_key(item_id))
await client.delete(_queued_key(item_id), _priority_queued_key(item_id))
+133 -54
View File
@@ -3,7 +3,7 @@ from __future__ import annotations
from datetime import UTC, datetime
from typing import Any
from sqlalchemy import func, select
from sqlalchemy import func, or_, select
from sqlalchemy.ext.asyncio import AsyncSession
from app.models.earth_news import EarthNewsItem
@@ -13,6 +13,22 @@ from app.services.earth_news import (
build_anchor_location_patch,
_news_meta_patch,
)
from app.services.earth_news_classification import (
breaking_sort_rank,
normalize_breaking_level,
normalize_breaking_scope,
)
CRUISE_REGION_ORDER = (
"americas",
"europe",
"middle-east-africa",
"asia-pacific",
"global",
)
CRUISE_REGION_QUERY_MULTIPLIER = 12
CRUISE_REGION_QUERY_MIN_LIMIT = 240
CRUISE_REGION_QUERY_MAX_LIMIT = 1000
def _coerce_datetime(value: datetime | None) -> datetime | None:
@@ -23,6 +39,17 @@ def _coerce_datetime(value: datetime | None) -> datetime | None:
return value.astimezone(UTC)
def _coerce_meta_datetime(value: Any) -> datetime | None:
if isinstance(value, datetime):
return _coerce_datetime(value)
if not isinstance(value, str) or not value.strip():
return None
try:
return _coerce_datetime(datetime.fromisoformat(value.replace("Z", "+00:00")))
except ValueError:
return None
def _location_patch_from_record(record: EarthNewsItem) -> dict[str, Any]:
return {
"latitude": record.latitude,
@@ -64,10 +91,67 @@ def record_to_parsed_news_item(record: EarthNewsItem) -> ParsedNewsItem:
importance_level=str(news_meta.get("importance_level") or "low"),
importance_reasons=list(news_meta.get("importance_reasons") or []),
market_impact=str(news_meta.get("market_impact") or "none"),
breaking_level=normalize_breaking_level(news_meta.get("breaking_level")).value,
breaking_scope=normalize_breaking_scope(news_meta.get("breaking_scope")).value,
breaking_reasons=list(news_meta.get("breaking_reasons") or []),
breaking_source=str(news_meta.get("breaking_source") or "rules"),
breaking_confidence=float(news_meta.get("breaking_confidence") or 0),
breaking_expires_at=_coerce_meta_datetime(news_meta.get("breaking_expires_at")),
)
return apply_enrichment_patch_to_item(item, _location_patch_from_record(record))
def _sort_parsed_news_items(items: list[ParsedNewsItem], *, active_region: str) -> list[ParsedNewsItem]:
return sorted(
items,
key=lambda item: (
-breaking_sort_rank(item),
False
if active_region == "global"
or (breaking_sort_rank(item) > 0 and normalize_breaking_scope(item.breaking_scope).value == "global")
else item.feed_region != active_region,
item.published_at is None,
-(item.published_at.timestamp() if item.published_at else 0),
item.feed_name,
),
)
def _diversify_parsed_news_items_by_region(
items: list[ParsedNewsItem],
*,
limit: int,
) -> list[ParsedNewsItem]:
if limit <= 0:
return []
sorted_items = _sort_parsed_news_items(items, active_region="global")
buckets: dict[str, list[ParsedNewsItem]] = {}
for item in sorted_items:
region = item.feed_region or "global"
buckets.setdefault(region, []).append(item)
ordered_regions = [
*[region for region in CRUISE_REGION_ORDER if buckets.get(region)],
*sorted(region for region in buckets if region not in CRUISE_REGION_ORDER),
]
diversified: list[ParsedNewsItem] = []
cursor = 0
while len(diversified) < limit:
added = False
for region in ordered_regions:
bucket = buckets.get(region) or []
if cursor >= len(bucket):
continue
diversified.append(bucket[cursor])
added = True
if len(diversified) >= limit:
break
if not added:
break
cursor += 1
return diversified
def _query_sort_key(active_region: str):
if active_region == "global":
return (
@@ -95,42 +179,6 @@ def _source_filter_clause(source_ids: set[str] | None):
return EarthNewsItem.location_meta.op("->")("news_meta").op("->>")("source_id").in_(sorted(source_ids))
def _record_source_id(record: EarthNewsItem) -> str:
location_meta = dict(record.location_meta or {})
news_meta = location_meta.get("news_meta") if isinstance(location_meta.get("news_meta"), dict) else {}
source_id = str(news_meta.get("source_id") or "").strip()
if source_id:
return source_id
if isinstance(record.id, str) and ":" in record.id:
return record.id.split(":", 1)[0]
return record.feed_name or record.source or record.id
def _diversify_records_by_source(records: list[EarthNewsItem], *, limit: int) -> list[EarthNewsItem]:
if limit <= 0 or len(records) <= limit:
return records[:limit]
buckets: dict[str, list[EarthNewsItem]] = {}
order: list[str] = []
for record in records:
source_id = _record_source_id(record)
if source_id not in buckets:
buckets[source_id] = []
order.append(source_id)
buckets[source_id].append(record)
diversified: list[EarthNewsItem] = []
while len(diversified) < limit and order:
next_order: list[str] = []
for source_id in order:
bucket = buckets.get(source_id) or []
if bucket and len(diversified) < limit:
diversified.append(bucket.pop(0))
if bucket:
next_order.append(source_id)
order = next_order
return diversified
async def list_earth_news_items(
db: AsyncSession,
*,
@@ -139,14 +187,20 @@ async def list_earth_news_items(
categories: set[str] | None = None,
source_ids: set[str] | None = None,
) -> list[ParsedNewsItem]:
query_limit = limit if source_ids else min(max(limit * 4, limit), 100)
query_limit = limit if source_ids else min(max(limit * 20, limit), 500)
query = (
select(EarthNewsItem)
.order_by(*_query_sort_key(active_region))
.limit(query_limit)
)
if active_region != "global":
query = query.where(EarthNewsItem.region.in_({"global", active_region}))
news_meta = EarthNewsItem.location_meta.op("->")("news_meta")
query = query.where(
or_(
EarthNewsItem.region.in_({"global", active_region}),
news_meta.op("->>")("breaking_scope") == "global",
)
)
category_clause = _category_filter_clause(categories)
if category_clause is not None:
query = query.where(category_clause)
@@ -155,11 +209,13 @@ async def list_earth_news_items(
query = query.where(source_clause)
result = await db.execute(query)
records = list(result.scalars().all())
if not source_ids:
records = _diversify_records_by_source(records, limit=limit)
else:
records = records[:limit]
return [record_to_parsed_news_item(record) for record in records]
items = _sort_parsed_news_items(
[record_to_parsed_news_item(record) for record in records],
active_region=active_region,
)
if active_region == "global" and not source_ids:
return _diversify_parsed_news_items_by_region(items, limit=limit)
return items[:limit]
async def list_earth_news_cruise_items(
@@ -169,15 +225,20 @@ async def list_earth_news_cruise_items(
categories: set[str] | None = None,
source_ids: set[str] | None = None,
) -> list[ParsedNewsItem]:
query_limit = min(
max(limit * CRUISE_REGION_QUERY_MULTIPLIER, CRUISE_REGION_QUERY_MIN_LIMIT),
CRUISE_REGION_QUERY_MAX_LIMIT,
)
query = (
select(EarthNewsItem)
.order_by(
EarthNewsItem.region.asc(),
EarthNewsItem.published_at.is_(None),
EarthNewsItem.published_at.desc().nullslast(),
EarthNewsItem.last_seen_at.desc(),
EarthNewsItem.region.asc(),
EarthNewsItem.feed_name.asc(),
)
.limit(limit)
.limit(query_limit)
)
category_clause = _category_filter_clause(categories)
if category_clause is not None:
@@ -186,7 +247,10 @@ async def list_earth_news_cruise_items(
if source_clause is not None:
query = query.where(source_clause)
result = await db.execute(query)
return [record_to_parsed_news_item(record) for record in result.scalars().all()]
return _diversify_parsed_news_items_by_region(
[record_to_parsed_news_item(record) for record in result.scalars().all()],
limit=limit,
)
async def get_earth_news_freshness(
@@ -334,13 +398,28 @@ async def update_earth_news_item_enrichment(
if record is None:
return False
if "latitude" in patch:
record.latitude = float(patch["latitude"])
record.longitude = float(patch["longitude"])
record.location_label = str(patch["location_label"])
record.location_source = str(patch["location_source"])
record.verified = bool(patch["verified"])
record.location_meta = dict(patch.get("location_meta") or {})
record.resolved_at = datetime.now(UTC) if record.verified else None
patch_meta = dict(patch.get("location_meta") or {})
if record.location_source == "manual_location":
current_meta = dict(record.location_meta or {})
patch_news_meta = patch_meta.get("news_meta")
if isinstance(patch_news_meta, dict):
current_meta["news_meta"] = patch_news_meta
current_meta["manual_enrichment"] = {
"resolution_stage": patch_meta.get("resolution_stage"),
"ai_attempted": patch_meta.get("ai_attempted"),
"ai_status": patch_meta.get("ai_status"),
"ai_error": patch_meta.get("ai_error"),
"debug_note": patch_meta.get("debug_note"),
}
record.location_meta = current_meta
else:
record.latitude = float(patch["latitude"])
record.longitude = float(patch["longitude"])
record.location_label = str(patch["location_label"])
record.location_source = str(patch["location_source"])
record.verified = bool(patch["verified"])
record.location_meta = patch_meta
record.resolved_at = datetime.now(UTC) if record.verified else None
if "content_language" in patch:
record.content_language = str(patch.get("content_language") or "en")
if "localizations" in patch:
+33 -3
View File
@@ -29,6 +29,9 @@ logger = get_logger(__name__, service="earth_news")
WORKER_BATCH_SIZE = 4
WORKER_BLOCK_MS = 5000
WORKER_BACKOFF_SECONDS = 5.0
WORKER_JOB_TIMEOUT_MIN_SECONDS = 20.0
WORKER_JOB_TIMEOUT_MAX_SECONDS = 90.0
WORKER_JOB_TIMEOUT_GRACE_SECONDS = 10.0
_worker_task: asyncio.Task | None = None
@@ -109,12 +112,25 @@ async def _run_target_location_worker() -> None:
if not messages:
continue
provider_client = await _build_provider_client()
for message in messages:
job_timeout = _get_worker_job_timeout(provider_client)
async def handle_message(message: NewsTargetLocationMessage) -> None:
try:
await process_target_location_message(message, provider_client=provider_client)
await queue.ack(message.message_id)
await asyncio.wait_for(
process_target_location_message(message, provider_client=provider_client),
timeout=job_timeout,
)
await queue.ack(message)
except asyncio.CancelledError:
raise
except TimeoutError as exc:
logger.warning_event(
"Earth news target location worker job timed out",
event="earth_news.target_location.worker_job_timeout",
context={"item_id": message.item_id, "timeout_seconds": job_timeout},
)
with suppress(Exception):
await queue.retry_or_dead_letter(message, error=str(exc) or "job timed out")
except Exception as exc:
logger.warning_event(
"Earth news target location worker job failed",
@@ -124,6 +140,8 @@ async def _run_target_location_worker() -> None:
with suppress(Exception):
await queue.retry_or_dead_letter(message, error=str(exc))
await asyncio.gather(*(handle_message(message) for message in messages))
def start_earth_news_target_worker() -> None:
global _worker_task
@@ -140,3 +158,15 @@ async def stop_earth_news_target_worker() -> None:
with suppress(asyncio.CancelledError):
await task
_worker_task = None
def _get_worker_job_timeout(provider_client: AIProviderClient | None) -> float:
timeout = float(getattr(provider_client, "timeout", 0) or WORKER_JOB_TIMEOUT_MIN_SECONDS)
retry_attempts = float(getattr(provider_client, "retry_attempts", 1) or 1)
return min(
max(
timeout * retry_attempts + WORKER_JOB_TIMEOUT_GRACE_SECONDS,
WORKER_JOB_TIMEOUT_MIN_SECONDS,
),
WORKER_JOB_TIMEOUT_MAX_SECONDS,
)
+8 -8
View File
@@ -9,12 +9,12 @@ yet to keep behavior obvious after settings changes).
from __future__ import annotations
from email.message import EmailMessage
from typing import Literal, Optional
from typing import Optional
import aiosmtplib
from sqlalchemy.ext.asyncio import AsyncSession
OtpPurpose = Literal["register", "verify_email", "reset_password"]
from app.core.enums import OtpPurpose
class EmailError(Exception):
@@ -81,15 +81,15 @@ async def send_email(
_SUBJECTS: dict[OtpPurpose, str] = {
"register": "Confirm your Planet account",
"verify_email": "Verify your Planet email",
"reset_password": "Reset your Planet password",
OtpPurpose.REGISTER: "Confirm your Planet account",
OtpPurpose.VERIFY_EMAIL: "Verify your Planet email",
OtpPurpose.RESET_PASSWORD: "Reset your Planet password",
}
_HEADLINES: dict[OtpPurpose, str] = {
"register": "Welcome to Planet — confirm your email to activate your account.",
"verify_email": "Confirm your new email address to keep your Planet account active.",
"reset_password": "Use this code to set a new password for your Planet account.",
OtpPurpose.REGISTER: "Welcome to Planet — confirm your email to activate your account.",
OtpPurpose.VERIFY_EMAIL: "Confirm your new email address to keep your Planet account active.",
OtpPurpose.RESET_PASSWORD: "Use this code to set a new password for your Planet account.",
}
+1 -3
View File
@@ -9,14 +9,12 @@ from __future__ import annotations
import json
import secrets
from typing import Literal
import bcrypt
from app.core.enums import OtpPurpose
from app.core.security import redis_client
OtpPurpose = Literal["register", "verify_email", "reset_password"]
CODE_TTL_SECONDS = 600 # 10 minutes
RESEND_COOLDOWN_SECONDS = 60
MAX_ATTEMPTS = 5
+47 -28
View File
@@ -7,9 +7,14 @@ from time import perf_counter
from uuid import uuid4
from fastapi import HTTPException, status
from sqlalchemy import func, select, update
from sqlalchemy import func, select
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.enums import (
PlaygroundMessageKind,
PlaygroundMessageRole,
PlaygroundMessageStatus,
)
from app.core.logging import get_logger
from app.db.session import async_session_factory
from app.models.playground_message import PlaygroundMessage
@@ -21,7 +26,6 @@ from app.schemas.ai import (
PlaygroundMessageRecord,
PlaygroundMessageResendRequest,
PlaygroundMessageStopRequest,
PlaygroundSessionResponse,
PlaygroundSessionState,
PlaygroundSessionUpsertRequest,
PlaygroundThreadResponse,
@@ -37,6 +41,13 @@ STREAM_CHUNK_SIZE = 24
STREAM_INTERVAL_SECONDS = 0.08
THINKING_PREVIEW_SECONDS = 2.6
ORPHANED_RUN_MESSAGE = "后台生成任务已中断,请点击上一条用户消息的重试按钮重新生成。"
ACTIVE_MESSAGE_STATUSES = frozenset(
{
PlaygroundMessageStatus.PENDING.value,
PlaygroundMessageStatus.THINKING.value,
PlaygroundMessageStatus.ANSWERING.value,
}
)
class _ActiveRun:
@@ -93,7 +104,11 @@ async def _require_visible_message(
result = await db.execute(select(PlaygroundMessage).where(*conditions))
message = result.scalar_one_or_none()
if message is None:
detail = "User message not found" if role == "user" else "Playground message not found"
detail = (
"User message not found"
if role == PlaygroundMessageRole.USER.value
else "Playground message not found"
)
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail=detail)
return message
@@ -108,7 +123,7 @@ def _message_to_record(message: PlaygroundMessage, parent_public_id: str | None
content=message.content or "",
thinking_content=message.thinking_content or "",
meta=list(message.meta or []),
markdown=message.role != "system",
markdown=message.role != PlaygroundMessageRole.SYSTEM.value,
provider=message.provider,
model=message.model,
request_id=message.request_id,
@@ -197,11 +212,11 @@ async def _reconcile_orphaned_active_messages(
) -> list[PlaygroundMessage]:
changed = False
for item in messages:
if item.status not in {"pending", "thinking", "answering"}:
if item.status not in ACTIVE_MESSAGE_STATUSES:
continue
if item.public_id in _ACTIVE_RUNS:
continue
item.status = "error"
item.status = PlaygroundMessageStatus.ERROR.value
item.content = item.content or ORPHANED_RUN_MESSAGE
orphan_meta = "错误: 后台任务已中断"
if orphan_meta not in (item.meta or []):
@@ -330,9 +345,9 @@ async def create_turn(
public_id=uuid4().hex,
session_id=session.id,
user_id=user_id,
role="user",
kind="message",
status="done",
role=PlaygroundMessageRole.USER.value,
kind=PlaygroundMessageKind.MESSAGE.value,
status=PlaygroundMessageStatus.DONE.value,
title=payload.selected_preset_key,
content=payload.input,
meta=[payload.title],
@@ -343,9 +358,9 @@ async def create_turn(
session_id=session.id,
user_id=user_id,
parent_message_id=None,
role="assistant",
kind="thinking",
status="pending",
role=PlaygroundMessageRole.ASSISTANT.value,
kind=PlaygroundMessageKind.THINKING.value,
status=PlaygroundMessageStatus.PENDING.value,
title="AI 回应",
content="",
thinking_content="",
@@ -397,9 +412,9 @@ async def _create_assistant_retry_turn(
session_id=session.id,
user_id=user_id,
parent_message_id=user_message.id,
role="assistant",
kind="thinking",
status="pending",
role=PlaygroundMessageRole.ASSISTANT.value,
kind=PlaygroundMessageKind.THINKING.value,
status=PlaygroundMessageStatus.PENDING.value,
title="AI 回应",
content="",
thinking_content="",
@@ -439,7 +454,7 @@ async def stop_message(
session = await _require_session(db, user_id=user_id, session_key=payload.session_key)
message = await _require_visible_message(db, user_id=user_id, public_id=payload.message_id)
if message.status not in {"pending", "thinking", "answering"}:
if message.status not in ACTIVE_MESSAGE_STATUSES:
return await _build_action_response(db, session=session)
active_run = _ACTIVE_RUNS.get(message.public_id)
@@ -447,7 +462,7 @@ async def stop_message(
active_run.stop_requested.set()
active_run.task.cancel()
message.status = "stopped"
message.status = PlaygroundMessageStatus.STOPPED.value
if "已手动停止生成" not in (message.meta or []):
message.meta = [*(message.meta or []), "已手动停止生成"]
await db.flush()
@@ -469,7 +484,7 @@ async def resend_turn(
db,
user_id=user_id,
public_id=payload.user_message_id,
role="user",
role=PlaygroundMessageRole.USER.value,
)
later_messages = await db.execute(
@@ -481,7 +496,7 @@ async def resend_turn(
)
for item in later_messages.scalars().all():
item.is_visible = False
if item.status in {"pending", "thinking", "answering"}:
if item.status in ACTIVE_MESSAGE_STATUSES:
active_run = _ACTIVE_RUNS.get(item.public_id)
if active_run is not None:
active_run.stop_requested.set()
@@ -520,7 +535,7 @@ async def edit_user_message(
db,
user_id=user_id,
public_id=payload.user_message_id,
role="user",
role=PlaygroundMessageRole.USER.value,
)
user_message.content = payload.content.strip()
@@ -566,12 +581,12 @@ def _build_conversation_history(messages: Sequence[PlaygroundMessage], current_u
for item in messages:
if item.id >= current_user_message_id:
break
if item.role == "system":
if item.role == PlaygroundMessageRole.SYSTEM.value:
continue
history.append(
{
"role": item.role,
"kind": item.kind or "message",
"kind": item.kind or PlaygroundMessageKind.MESSAGE.value,
"title": item.title,
"content": item.content or "",
}
@@ -668,7 +683,11 @@ async def _run_assistant_message(
assistant_message = await _mark_message_state(
db,
message_id=assistant_message_id,
status="thinking" if analysis.thinking_blocks else "answering",
status=(
PlaygroundMessageStatus.THINKING.value
if analysis.thinking_blocks
else PlaygroundMessageStatus.ANSWERING.value
),
title=f"{analysis.provider} / {analysis.model}",
provider=analysis.provider,
model=analysis.model,
@@ -706,7 +725,7 @@ async def _run_assistant_message(
await _mark_message_state(
db,
message_id=assistant_message_id,
status="answering",
status=PlaygroundMessageStatus.ANSWERING.value,
content=content[:cursor],
)
await db.commit()
@@ -717,7 +736,7 @@ async def _run_assistant_message(
assistant_message = await _mark_message_state(
db,
message_id=assistant_message_id,
status="done",
status=PlaygroundMessageStatus.DONE.value,
content=content,
meta=[
f"Request ID: {request_id}",
@@ -762,8 +781,8 @@ async def _run_assistant_message(
async with async_session_factory() as db:
result = await db.execute(select(PlaygroundMessage).where(PlaygroundMessage.id == assistant_message_id))
message = result.scalar_one_or_none()
if message is not None and message.status in {"pending", "thinking", "answering"}:
message.status = "stopped"
if message is not None and message.status in ACTIVE_MESSAGE_STATUSES:
message.status = PlaygroundMessageStatus.STOPPED.value
if "已手动停止生成" not in (message.meta or []):
message.meta = [*(message.meta or []), "已手动停止生成"]
await db.flush()
@@ -795,7 +814,7 @@ async def _run_assistant_message(
result = await db.execute(select(PlaygroundMessage).where(PlaygroundMessage.id == assistant_message_id))
message = result.scalar_one_or_none()
if message is not None:
message.status = "error"
message.status = PlaygroundMessageStatus.ERROR.value
message.content = message.content or f"分析失败:{error_message}"
message.meta = [
*(message.meta or []),
+8 -7
View File
@@ -8,6 +8,7 @@ from apscheduler.schedulers.asyncio import AsyncIOScheduler
from apscheduler.triggers.interval import IntervalTrigger
from sqlalchemy import select
from app.core.enums import JobStatus
from app.core.logging import get_logger
from app.db.session import async_session_factory
from app.core.time import to_iso8601_utc
@@ -140,7 +141,7 @@ async def run_collector_task(collector_name: str):
select(CollectionTask)
.where(
CollectionTask.datasource_id == datasource.id,
CollectionTask.status == "running",
CollectionTask.status == JobStatus.RUNNING.value,
)
.order_by(CollectionTask.started_at.desc(), CollectionTask.id.desc())
.limit(1)
@@ -184,7 +185,7 @@ async def run_collector_task(collector_name: str):
f"Marked failed automatically after stale running timeout "
f"({RUNNING_TASK_GUARD_TIMEOUT_MINUTES}m) in scheduler guard"
)
existing_running.status = "failed"
existing_running.status = JobStatus.FAILED.value
existing_running.phase = "failed"
existing_running.completed_at = now
existing_running.error_message = (
@@ -243,7 +244,7 @@ async def run_collector_task(collector_name: str):
return
datasource.last_run_at = datetime.now(UTC)
datasource.last_status = task_result.get("status")
if datasource.last_status == "success":
if datasource.last_status == JobStatus.SUCCESS.value:
effective_candidate = await get_builtin_effective_candidate(db, datasource_source)
checksum, _credential_context = await build_builtin_connectivity_checksum(
datasource_source,
@@ -284,7 +285,7 @@ async def run_collector_task(collector_name: str):
await db.rollback()
datasource = await db.get(DataSource, datasource_id)
datasource.last_run_at = datetime.now(UTC)
datasource.last_status = "cancelled"
datasource.last_status = JobStatus.CANCELLED.value
await db.commit()
logger.warning_event(
"Collector cancelled by operator",
@@ -306,7 +307,7 @@ async def run_collector_task(collector_name: str):
await db.rollback()
datasource = await db.get(DataSource, datasource_id)
datasource.last_run_at = datetime.now(UTC)
datasource.last_status = "failed"
datasource.last_status = JobStatus.FAILED.value
await db.commit()
logger.exception_event(
"Collector failed",
@@ -335,7 +336,7 @@ async def cleanup_stale_running_tasks(max_age_hours: int = 2) -> int:
async with async_session_factory() as db:
result = await db.execute(
select(CollectionTask).where(
CollectionTask.status == "running",
CollectionTask.status == JobStatus.RUNNING.value,
CollectionTask.started_at.is_not(None),
CollectionTask.started_at < cutoff,
)
@@ -343,7 +344,7 @@ async def cleanup_stale_running_tasks(max_age_hours: int = 2) -> int:
stale_tasks = result.scalars().all()
for task in stale_tasks:
task.status = "failed"
task.status = JobStatus.FAILED.value
task.phase = "failed"
task.completed_at = datetime.now(UTC)
existing_error = (task.error_message or "").strip()
@@ -6,6 +6,7 @@ from typing import Any
from sqlalchemy import func, select
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.enums import BGPStatus
from app.models.alert import Alert, AlertSeverity, AlertStatus
from app.models.bgp_anomaly import BGPAnomaly
from app.models.bgp_incident import BGPIncident
@@ -49,11 +50,11 @@ async def build_situational_alert_brief_request(
total_incidents_result = await db.execute(select(func.count(BGPIncident.id)))
active_incidents_result = await db.execute(
select(func.count(BGPIncident.id)).where(BGPIncident.status == "active")
select(func.count(BGPIncident.id)).where(BGPIncident.status == BGPStatus.ACTIVE.value)
)
bgp_severity_result = await db.execute(
select(BGPIncident.severity, func.count(BGPIncident.id))
.where(BGPIncident.status == "active")
.where(BGPIncident.status == BGPStatus.ACTIVE.value)
.group_by(BGPIncident.severity)
)
bgp_region_counter: Counter[str] = Counter()
@@ -65,11 +66,11 @@ async def build_situational_alert_brief_request(
total_anomalies_result = await db.execute(select(func.count(BGPAnomaly.id)))
active_anomalies_result = await db.execute(
select(func.count(BGPAnomaly.id)).where(BGPAnomaly.status == "active")
select(func.count(BGPAnomaly.id)).where(BGPAnomaly.status == BGPStatus.ACTIVE.value)
)
anomaly_type_result = await db.execute(
select(BGPAnomaly.anomaly_type, func.count(BGPAnomaly.id))
.where(BGPAnomaly.status == "active")
.where(BGPAnomaly.status == BGPStatus.ACTIVE.value)
.group_by(BGPAnomaly.anomaly_type)
.order_by(func.count(BGPAnomaly.id).desc())
.limit(6)
+2 -1
View File
@@ -7,6 +7,7 @@ from pathlib import Path
from typing import Any
from app.core.config import ROOT_DIR
from app.core.enums import UserRole
from app.core.security import redis_client
SYSTEM_TASK_TTL_SECONDS = 24 * 60 * 60
@@ -47,7 +48,7 @@ def normalize_user_role(role: Any) -> str:
def require_super_admin(user_role: Any) -> bool:
return normalize_user_role(user_role) == "super_admin"
return normalize_user_role(user_role) == UserRole.SUPER_ADMIN.value
def build_task_id(prefix: str = "restart") -> str:
+6 -5
View File
@@ -13,6 +13,7 @@ from datetime import UTC, datetime
from pathlib import Path
from typing import Any
from app.core.enums import LogLevel
from app.core.security import redis_client
from app.models.system_log import AuditLog, ObservabilityEvent, ObservabilityEventGroup, SystemLog
from sqlalchemy import select
@@ -24,11 +25,11 @@ BUFFER_LOG_LIMIT = 1000
BUFFER_LOG_TTL_SECONDS = 7 * 24 * 60 * 60
LOG_BUFFER_KEY_PREFIX = "planet:system_logs"
LOG_LEVEL_ERROR = "error"
LOG_LEVEL_WARNING = "warning"
LOG_LEVEL_INFO = "info"
LOG_LEVEL_DEBUG = "debug"
LOG_LEVEL_ALL = "all"
LOG_LEVEL_ERROR = LogLevel.ERROR.value
LOG_LEVEL_WARNING = LogLevel.WARNING.value
LOG_LEVEL_INFO = LogLevel.INFO.value
LOG_LEVEL_DEBUG = LogLevel.DEBUG.value
LOG_LEVEL_ALL = LogLevel.ALL.value
SUPPORTED_LOG_LEVELS = {
LOG_LEVEL_ALL,
+123 -1
View File
@@ -9,7 +9,12 @@ from sqlalchemy import select
from sqlalchemy import Float
from sqlalchemy.ext.asyncio import AsyncSession
from app.models.vessel import AISConflictRecord, AISRawObservation, AISSourceHealth
from app.models.vessel import (
AISConflictRecord,
AISRawObservation,
AISSourceHealth,
VesselCurrentState,
)
from app.services.vessel_aggregation_strategy import (
DEFAULT_STRATEGY,
load_strategy,
@@ -44,6 +49,17 @@ CONFLICT_FIELDS = (
"width",
"draught",
)
CURRENT_STATE_STATIC_FIELDS = (
"name",
"callsign",
"vessel_type",
"vessel_type_name",
"flag",
"length",
"width",
"draught",
"imo",
)
def _json_default(value: Any) -> Any:
@@ -489,9 +505,115 @@ async def record_vessel_ais_observation(
quality_flags=quality_flags or [],
)
db.add(observation)
await upsert_vessel_current_state(
db,
source=source,
normalized_payload=normalized_json,
observed_at=observed_at,
quality_flags=quality_flags or [],
)
return observation
async def upsert_vessel_current_state(
db: AsyncSession,
*,
source: str,
normalized_payload: dict[str, Any],
observed_at: datetime,
quality_flags: list[str] | None = None,
) -> VesselCurrentState | None:
"""Keep one latest renderable row per MMSI while preserving useful static fields."""
if not _has_valid_position(normalized_payload):
return None
mmsi = int(normalized_payload["mmsi"])
current = await db.get(VesselCurrentState, mmsi)
if current is not None and current.observed_at is not None:
current_observed_at = _coerce_datetime(current.observed_at)
if current_observed_at is not None and observed_at < current_observed_at:
return current
if current is None:
current = VesselCurrentState(mmsi=mmsi)
db.add(current)
current.lat = float(normalized_payload["lat"])
current.lon = float(normalized_payload["lon"])
current.source = source
current.observed_at = observed_at
current.updated_at = datetime.now(UTC)
updated_fields: set[str] = {"lat", "lon"}
for field in DYNAMIC_FIELDS:
if field in {"lat", "lon"}:
continue
value = _payload_value(normalized_payload, field)
if value is not None:
setattr(current, field, value)
updated_fields.add(field)
field_sources = dict(current.field_sources or {})
for field in CURRENT_STATE_STATIC_FIELDS:
value = _payload_value(normalized_payload, field)
if value is None:
continue
existing_source = field_sources.get(field)
existing_value = getattr(current, field, None)
if (
existing_value in (None, "")
or _strategy_source_rank(source, DEFAULT_STRATEGY)
>= _strategy_source_rank(str(existing_source or ""), DEFAULT_STRATEGY)
):
setattr(current, field, value)
updated_fields.add(field)
current.vessel_type_name = current.vessel_type_name or normalize_vessel_type_name(
current.vessel_type
)
selected_reasons = dict(current.selected_reasons or {})
for field in updated_fields:
field_sources[field] = source
selected_reasons[field] = (
"newest_observation" if field in DYNAMIC_FIELDS else "source_priority"
)
current.field_sources = field_sources
current.selected_reasons = selected_reasons
current.source_summary = {
**dict(current.source_summary or {}),
source: {
"latest_observed_at": observed_at.isoformat(),
},
}
current.quality_flags = sorted(set((current.quality_flags or []) + (quality_flags or [])))
return current
async def get_current_vessels_snapshot(
db: AsyncSession,
*,
bbox: tuple[float, float, float, float],
limit: int = 1000,
observed_since: datetime,
) -> list[dict[str, Any]]:
"""Read the bounded latest-state table used by Earth rendering."""
safe_limit = min(max(int(limit or 1000), 1), MAX_SNAPSHOT_LIMIT)
lon_min, lat_min, lon_max, lat_max = bbox
stmt = (
select(VesselCurrentState)
.where(VesselCurrentState.observed_at >= observed_since)
.where(VesselCurrentState.lon >= lon_min)
.where(VesselCurrentState.lon <= lon_max)
.where(VesselCurrentState.lat >= lat_min)
.where(VesselCurrentState.lat <= lat_max)
.order_by(VesselCurrentState.observed_at.desc(), VesselCurrentState.mmsi.asc())
.limit(safe_limit)
)
result = await db.execute(stmt)
if not hasattr(result, "scalars"):
return []
return [item.to_dict() for item in result.scalars().all()]
async def aggregate_vessel_observations(
db: AsyncSession,
observations: Iterable[AISRawObservation],
+1
View File
@@ -1,4 +1,5 @@
[pytest]
pythonpath = ..
asyncio_mode = auto
testpaths = tests
python_files = test_*.py
+197 -7
View File
@@ -1,4 +1,4 @@
from datetime import UTC, datetime
from datetime import UTC, datetime, timedelta
from types import SimpleNamespace
import pytest
@@ -12,9 +12,11 @@ from app.services.earth_news import (
default_earth_news_sources_payload,
normalize_earth_news_sources_payload,
_fetch_source,
_diversify_news_items_for_locale,
_enrich_items_with_target_locations,
_extract_target_location_from_text,
_parse_feed_entries,
_rank_and_trim_items,
_serialize_item,
get_earth_news_payload,
test_news_source_config as run_news_source_config_test,
@@ -22,6 +24,7 @@ from app.services.earth_news import (
from app.services.earth_news_queue import NewsTargetLocationMessage
from app.services.earth_news_worker import process_target_location_message
from app.services.collectors.media_news_archive import MediaNewsArchiveCollector
from app.services.earth_news_store import _diversify_parsed_news_items_by_region
def test_serialize_item_includes_region_anchor_for_cruise():
@@ -50,6 +53,188 @@ def test_serialize_item_includes_region_anchor_for_cruise():
assert payload["published_at"] == "2026-04-23T02:30:00Z"
def test_serialize_item_includes_breaking_fields():
item = ParsedNewsItem(
id="breaking:test",
title="Major market halt",
summary="Trading halt after flash crash",
url="https://example.com/breaking",
source="Example Source",
feed_name="Example Feed",
feed_region="global",
homepage_url="https://example.com",
published_at=datetime(2026, 5, 15, 2, 0, tzinfo=UTC),
breaking_level="critical",
breaking_scope="global",
breaking_reasons=["重大金融市场异常"],
breaking_source="rules",
breaking_confidence=0.72,
breaking_expires_at=datetime(2026, 5, 16, 2, 0, tzinfo=UTC),
)
payload = _serialize_item(item, active_region="europe")
assert payload["breaking_level"] == "critical"
assert payload["breaking_scope"] == "global"
assert payload["breaking_reasons"] == ["重大金融市场异常"]
assert payload["breaking_source"] == "rules"
assert payload["breaking_confidence"] == 0.72
assert payload["breaking_expires_at"] == "2026-05-16T02:00:00Z"
def test_rank_and_trim_items_prioritizes_active_breaking():
older_breaking = ParsedNewsItem(
id="global:critical",
title="Nuclear accident reported",
summary="A nuclear accident has been reported.",
url="https://example.com/critical",
source="Global Source",
feed_name="Global Feed",
feed_region="global",
homepage_url="https://example.com",
published_at=datetime.now(UTC) - timedelta(hours=2),
breaking_level="critical",
breaking_scope="global",
breaking_expires_at=datetime.now(UTC) + timedelta(hours=6),
)
newer_regular = ParsedNewsItem(
id="europe:regular",
title="Regular Europe story",
summary="A newer regular story.",
url="https://example.com/regular",
source="Europe Source",
feed_name="Europe Feed",
feed_region="europe",
homepage_url="https://example.com",
published_at=datetime.now(UTC),
)
expired_breaking = ParsedNewsItem(
id="europe:expired",
title="Expired breaking",
summary="Expired breaking story.",
url="https://example.com/expired",
source="Europe Source",
feed_name="Europe Feed",
feed_region="europe",
homepage_url="https://example.com",
published_at=datetime.now(UTC) + timedelta(minutes=1),
breaking_level="critical",
breaking_scope="regional",
breaking_expires_at=datetime.now(UTC) - timedelta(minutes=1),
)
ranked = _rank_and_trim_items(
[newer_regular, expired_breaking, older_breaking],
active_region="europe",
limit=3,
)
assert [item.id for item in ranked] == ["global:critical", "europe:expired", "europe:regular"]
def test_diversify_news_items_prefers_display_ready_content_across_sources():
published_at = datetime(2026, 6, 11, 3, 0, tzinfo=UTC)
def make_item(source_id: str, suffix: str, *, zh_ready: bool) -> ParsedNewsItem:
return ParsedNewsItem(
id=f"{source_id}:{suffix}",
title=f"{source_id} title {suffix}",
summary=f"{source_id} summary {suffix}",
url=f"https://example.com/{source_id}/{suffix}",
source=source_id,
feed_name=source_id,
feed_region="global",
homepage_url="https://example.com",
published_at=published_at,
content_language="en",
localizations={
"zh-CN": {
"title": f"{source_id} 中文标题 {suffix}",
"summary": f"{source_id} 中文摘要 {suffix}",
}
} if zh_ready else {},
)
items = [
make_item("source-a", "1", zh_ready=False),
make_item("source-a", "2", zh_ready=False),
make_item("source-a", "3", zh_ready=False),
make_item("source-b", "1", zh_ready=True),
make_item("source-c", "1", zh_ready=True),
]
result = _diversify_news_items_for_locale(
items,
active_region="global",
limit=3,
locale="zh-CN",
)
assert [item.id.split(":", 1)[0] for item in result] == ["source-b", "source-c", "source-a"]
def test_cruise_news_diversity_keeps_regions_from_being_starved():
published_at = datetime(2026, 6, 26, 8, 0, tzinfo=UTC)
def make_item(region: str, index: int) -> ParsedNewsItem:
return ParsedNewsItem(
id=f"{region}:{index}",
title=f"{region} story {index}",
summary=f"{region} summary {index}",
url=f"https://example.com/{region}/{index}",
source=region,
feed_name=region,
feed_region=region,
homepage_url="https://example.com",
published_at=published_at - timedelta(minutes=index),
)
items = [
*[make_item("asia-pacific", index) for index in range(40)],
make_item("europe", 1),
make_item("middle-east-africa", 1),
make_item("americas", 1),
make_item("global", 1),
]
result = _diversify_parsed_news_items_by_region(items, limit=8)
regions = [item.feed_region for item in result]
assert "europe" in regions
assert "middle-east-africa" in regions
assert "americas" in regions
assert regions.count("asia-pacific") < len(regions)
def test_global_news_diversity_uses_same_region_balance():
published_at = datetime(2026, 6, 26, 8, 0, tzinfo=UTC)
def make_item(region: str, index: int) -> ParsedNewsItem:
return ParsedNewsItem(
id=f"{region}:global:{index}",
title=f"{region} story {index}",
summary=f"{region} summary {index}",
url=f"https://example.com/{region}/global/{index}",
source=region,
feed_name=region,
feed_region=region,
homepage_url="https://example.com",
published_at=published_at - timedelta(minutes=index),
)
items = [
*[make_item("asia-pacific", index) for index in range(24)],
*[make_item("europe", index) for index in range(2)],
*[make_item("middle-east-africa", index) for index in range(2)],
*[make_item("americas", index) for index in range(2)],
]
result = _diversify_parsed_news_items_by_region(items, limit=6)
regions = {item.feed_region for item in result}
assert {"europe", "middle-east-africa", "americas"}.issubset(regions)
def test_serialize_item_falls_back_to_global_anchor():
item = ParsedNewsItem(
id="custom:test",
@@ -881,9 +1066,11 @@ async def test_earth_news_payload_uses_fresh_database_items_without_rss(monkeypa
async def fake_get_earth_news_freshness(_db, *, active_region):
return 12, datetime.now(UTC)
async def fake_list_earth_news_items(_db, *, active_region, limit, categories=None):
assert limit == 12
return [item]
async def fake_list_earth_news_items(_db, *, active_region, limit, categories=None, source_ids=None):
if source_ids is None:
assert limit == 12
return [item]
return []
async def fail_fetch(_sources):
raise AssertionError("fresh database items should not fetch RSS")
@@ -982,8 +1169,8 @@ async def test_earth_news_payload_passes_region_and_category_filters_to_store(mo
async def fake_list_earth_news_items(_db, *, active_region, limit, categories=None, source_ids=None):
captured["items_region"] = active_region
captured["items_categories"] = categories
captured["items_source_ids"] = source_ids
return [item]
captured.setdefault("items_source_ids", []).append(source_ids)
return [item] if source_ids is None else []
async def fake_list_earth_news_cruise_items(_db, *, limit, categories=None, source_ids=None):
captured["cruise_categories"] = categories
@@ -1010,7 +1197,8 @@ async def test_earth_news_payload_passes_region_and_category_filters_to_store(mo
assert captured["freshness_region"] == "europe"
assert captured["items_region"] == "europe"
assert captured["items_categories"] == {"business", "ecommerce"}
assert captured["items_source_ids"] is None
assert captured["items_source_ids"][0] is None
assert any(source_ids for source_ids in captured["items_source_ids"][1:])
assert captured["cruise_categories"] == {"business", "ecommerce"}
assert captured["cruise_source_ids"] is None
assert payload["filters"] == {
@@ -1019,6 +1207,8 @@ async def test_earth_news_payload_passes_region_and_category_filters_to_store(mo
"sources": [],
"limit": 12,
"locale": "zh-CN",
"has_breaking": False,
"highest_breaking_level": "none",
}
assert payload["items"][0]["category"] == "business"
+252
View File
@@ -0,0 +1,252 @@
from datetime import UTC, datetime
import pytest
from app.core.enums import NewsSourceType
from app.models.earth_news import EarthNewsItem
from app.models.system_setting import SystemSetting
from app.services.earth_news import REGION_ANCHORS
from app.services.earth_news_manual import (
DEFAULT_MANUAL_NEWS_GROUP_ID,
create_manual_news_group,
import_manual_news_items,
list_news_groups,
list_news_records,
parse_manual_news_import_upload,
rename_manual_news_group,
upsert_manual_news_item,
)
class _FakeResult:
def __init__(self, rows=None, scalar=None):
self.rows = rows or []
self._scalar = scalar
def scalar_one_or_none(self):
return self._scalar
def scalar(self):
return self._scalar
def scalars(self):
return self
def all(self):
return self.rows
class _FakeNewsSession:
def __init__(self, records=None, setting=None):
self.records = dict(records or {})
self.setting = setting
async def get(self, _model, item_id):
return self.records.get(item_id)
def add(self, item):
if isinstance(item, SystemSetting):
self.setting = item
else:
self.records[item.id] = item
async def execute(self, stmt):
statement = str(stmt)
if "system_settings" in statement:
return _FakeResult(scalar=self.setting)
if "count" in statement.lower():
return _FakeResult(scalar=len(self.records))
return _FakeResult(rows=list(self.records.values()))
async def flush(self):
return None
@pytest.fixture
def fake_news_queue(monkeypatch):
queued = []
async def _enqueue(payload, force=False):
queued.append({"payload": payload, "force": force})
return True
monkeypatch.setattr("app.services.earth_news_manual.enqueue_target_location_job", _enqueue)
return queued
@pytest.mark.asyncio
async def test_manual_news_upsert_uses_region_anchor_and_manual_metadata(fake_news_queue):
db = _FakeNewsSession()
result = await upsert_manual_news_item(
db,
{
"title": "手动添加的新闻",
"summary": "一条用于测试的手动新闻。",
"source": "人工录入",
"region": "europe",
"published_at": "2026-05-15T03:00:00Z",
"tags": ["manual", "test"],
},
)
anchor = REGION_ANCHORS["europe"]
assert result.created is True
assert result.queued is True
assert result.item.id.startswith("manual:")
assert result.item.feed_name == "手动添加"
assert result.item.source == "人工录入"
assert result.item.latitude == anchor.latitude
assert result.item.longitude == anchor.longitude
assert result.item.location_source == "region_anchor"
assert result.item.verified is False
assert result.item.location_meta["news_meta"]["feed_type"] == NewsSourceType.MANUAL.value
assert result.item.location_meta["news_meta"]["source_type"] == NewsSourceType.MANUAL.value
assert result.item.location_meta["news_meta"]["manual_group_id"] == DEFAULT_MANUAL_NEWS_GROUP_ID
assert len(fake_news_queue) == 1
@pytest.mark.asyncio
async def test_manual_news_duplicate_import_upserts_without_duplicate_rows(fake_news_queue):
db = _FakeNewsSession()
payload = {
"title": "Same manual story",
"source": "Manual Desk",
"published_at": "2026-05-15T03:00:00Z",
"region": "global",
}
first = await upsert_manual_news_item(db, payload)
second = await upsert_manual_news_item(db, {**payload, "summary": "Updated summary"})
assert first.created is True
assert second.created is False
assert len(db.records) == 1
assert db.records[first.item.id].summary == "Updated summary"
@pytest.mark.asyncio
async def test_manual_news_edit_without_location_preserves_manual_coordinates(fake_news_queue):
db = _FakeNewsSession()
created = await upsert_manual_news_item(
db,
{
"title": "Taipei-1 data center update",
"summary": "Initial summary.",
"region": "asia-pacific",
"published_at": "2026-05-15T03:00:00Z",
"location": {"label": "Kaohsiung, Taiwan", "latitude": 22.6273, "longitude": 120.3014},
},
)
updated = await upsert_manual_news_item(
db,
{
"title": "Taipei-1 data center update",
"summary": "Edited summary only.",
"region": "asia-pacific",
"published_at": "2026-05-15T03:00:00Z",
},
item_id_override=created.item.id,
)
assert updated.created is False
assert updated.item.latitude == pytest.approx(22.6273)
assert updated.item.longitude == pytest.approx(120.3014)
assert updated.item.location_source == "manual_location"
assert updated.item.verified is True
@pytest.mark.asyncio
async def test_manual_news_api_service_rejects_rss_records(fake_news_queue):
rss_record = EarthNewsItem(
id="bbc-world:example",
title="RSS story",
summary="RSS summary",
source="BBC World",
feed_name="BBC World",
region="global",
latitude=20,
longitude=0,
location_label="全球",
location_source="region_anchor",
verified=False,
location_meta={"news_meta": {"feed_type": "rss"}},
first_seen_at=datetime.now(UTC),
last_seen_at=datetime.now(UTC),
)
db = _FakeNewsSession({rss_record.id: rss_record})
with pytest.raises(PermissionError):
await upsert_manual_news_item(
db,
{"title": "Edited title", "region": "global"},
item_id_override=rss_record.id,
)
@pytest.mark.asyncio
async def test_manual_news_import_reports_per_item_errors(fake_news_queue):
db = _FakeNewsSession()
result = await import_manual_news_items(
db,
[
{"title": "Valid manual news", "region": "global"},
{"summary": "missing title"},
],
)
assert result["created"] == 1
assert result["failed"] == 1
assert result["errors"][0]["index"] == 1
@pytest.mark.asyncio
async def test_manual_news_groups_default_create_and_rename(fake_news_queue):
db = _FakeNewsSession()
initial = await list_news_groups(db)
assert initial["manual_groups"][0]["id"] == DEFAULT_MANUAL_NEWS_GROUP_ID
assert initial["manual_groups"][0]["name"] == "新建新闻组"
group = await create_manual_news_group(db, "专题组")
assert group["name"] == "专题组"
assert db.setting is not None
await upsert_manual_news_item(db, {"title": "Grouped story", "region": "global"}, group_id=group["id"])
renamed = await rename_manual_news_group(db, group["id"], "重命名专题")
record = next(iter(db.records.values()))
assert renamed["name"] == "重命名专题"
assert record.location_meta["news_meta"]["manual_group_id"] == group["id"]
assert record.location_meta["news_meta"]["manual_group_name"] == "重命名专题"
@pytest.mark.asyncio
async def test_manual_news_list_filters_by_group_id(fake_news_queue):
db = _FakeNewsSession()
group = await create_manual_news_group(db, "导入组")
await import_manual_news_items(
db,
[
{"title": "In group", "region": "global"},
{"title": "Also in group", "region": "global"},
],
group_id=group["id"],
)
await upsert_manual_news_item(db, {"title": "Default group", "region": "global"})
grouped = await list_news_records(db, page=1, page_size=20, group_id=group["id"])
default_group = await list_news_records(db, page=1, page_size=20, group_id=DEFAULT_MANUAL_NEWS_GROUP_ID)
assert grouped["total"] == 2
assert {item["manual_group_id"] for item in grouped["items"]} == {group["id"]}
assert default_group["total"] == 1
@pytest.mark.asyncio
async def test_manual_news_import_parser_requires_json_array():
with pytest.raises(ValueError, match="顶层必须是数组"):
await parse_manual_news_import_upload(b'{"title":"not an array"}')
+74
View File
@@ -0,0 +1,74 @@
"""Compatibility contracts for stable backend protocol enums."""
from __future__ import annotations
from datetime import UTC, datetime, timedelta
from types import SimpleNamespace
from app.core.enums import (
BreakingLevel,
BreakingScope,
JobStatus,
NewsImportanceLevel,
NewsSourceType,
PlaygroundMessageKind,
PlaygroundMessageStatus,
UserRole,
parse_enum,
)
from app.services.earth_news_classification import (
BREAKING_LEVEL_RANK,
BREAKING_TTL,
breaking_sort_rank,
importance_level,
)
def test_protocol_enum_values_remain_api_compatible() -> None:
assert [item.value for item in NewsImportanceLevel] == ["low", "medium", "high", "critical"]
assert [item.value for item in BreakingLevel] == ["none", "watch", "breaking", "critical"]
assert [item.value for item in BreakingScope] == ["regional", "global"]
assert [item.value for item in NewsSourceType] == ["rss", "atom", "aggregated", "reference", "manual"]
assert [item.value for item in UserRole] == ["viewer", "admin", "super_admin"]
assert JobStatus.RUNNING.value == "running"
assert PlaygroundMessageKind.THINKING.value == "thinking"
assert PlaygroundMessageStatus.ERROR.value == "error"
assert PlaygroundMessageStatus.STOPPED.value == "stopped"
def test_parse_enum_accepts_legacy_strings_and_safely_falls_back(caplog) -> None:
assert parse_enum(JobStatus, "RUNNING", JobStatus.FAILED) is JobStatus.RUNNING
assert parse_enum(JobStatus, None, JobStatus.QUEUED) is JobStatus.QUEUED
assert parse_enum(JobStatus, "legacy-unknown", JobStatus.FAILED) is JobStatus.FAILED
assert "legacy-unknown" in caplog.text
def test_importance_level_boundaries() -> None:
expected = {
34: NewsImportanceLevel.LOW,
35: NewsImportanceLevel.MEDIUM,
59: NewsImportanceLevel.MEDIUM,
60: NewsImportanceLevel.HIGH,
79: NewsImportanceLevel.HIGH,
80: NewsImportanceLevel.CRITICAL,
}
assert {score: importance_level(score) for score in expected} == expected
def test_breaking_rank_and_ttl_contracts() -> None:
assert BREAKING_LEVEL_RANK[BreakingLevel.CRITICAL] > BREAKING_LEVEL_RANK[BreakingLevel.BREAKING]
assert BREAKING_TTL[BreakingLevel.WATCH] == timedelta(hours=6)
assert BREAKING_TTL[BreakingLevel.BREAKING] == timedelta(hours=12)
assert BREAKING_TTL[BreakingLevel.CRITICAL] == timedelta(hours=24)
now = datetime.now(UTC)
active = SimpleNamespace(
breaking_level=BreakingLevel.BREAKING.value,
breaking_expires_at=now + timedelta(minutes=1),
)
expired = SimpleNamespace(
breaking_level=BreakingLevel.CRITICAL.value,
breaking_expires_at=now - timedelta(minutes=1),
)
assert breaking_sort_rank(active) == BREAKING_LEVEL_RANK[BreakingLevel.BREAKING]
assert breaking_sort_rank(expired) == 0
+117
View File
@@ -40,6 +40,9 @@ def test_gesture_event_serializes_stable_protocol_fields():
assert payload["seq"] == 7
assert payload["source"] == "motion-agent"
assert payload["mode"] == "single"
assert payload["protocol_version"] == "motion.v2"
assert payload["input_mode"] == "single"
assert payload["camera_id"] == "unknown"
assert payload["payload"] == {}
@@ -88,8 +91,13 @@ def test_motion_server_status_includes_dry_run_camera_and_heartbeat():
assert status["camera_count"] == 1
assert status["active_camera_ids"] == ["dry-run:null-camera"]
assert status["recognizer"] == "dry-run"
assert status["protocol_version"] == "motion.v2"
assert status["armed"] is False
assert status["paused"] is False
assert status["devices_open"] is False
assert heartbeat == {
"timestamp_ms": 123,
"protocol_version": "motion.v2",
"source": "motion-agent",
"type": "heartbeat",
}
@@ -109,6 +117,7 @@ def test_skeleton_event_serializes_without_raw_image_fields():
payload = json.loads(event.to_json())
assert payload["type"] == "skeleton"
assert payload["protocol_version"] == "motion.v2"
assert payload["matched_gesture"] == "rotate_left"
assert payload["confidence"] == 0.91
assert payload["camera_id"] == "usb:0"
@@ -120,6 +129,25 @@ def test_skeleton_event_serializes_without_raw_image_fields():
assert "frame" not in payload
def test_v2_gesture_set_accepts_frontend_motion_gestures():
state = GestureStateMachine(confidence_threshold=0.7, cooldown_ms=0)
for gesture in [
"rotate_up",
"rotate_down",
"focus_prev",
"focus_next",
"layer_prev",
"layer_next",
]:
event = state.accept(
GestureObservation(gesture, confidence=0.9, intensity=0.8, timestamp_ms=1000)
)
assert event is not None
assert event.gesture == gesture
def test_dry_run_recognizer_produces_debug_skeleton():
server = MotionAgentServer(MotionAgentConfig(dry_run=True))
@@ -240,3 +268,92 @@ async def test_motion_agent_cli_reports_dependency_error_without_traceback(monke
assert exit_code == 2
assert "Motion agent failed: missing cv stack" in captured.err
assert "Traceback" not in captured.err
@pytest.mark.asyncio
async def test_motion_agent_command_updates_control_state():
server = MotionAgentServer(MotionAgentConfig(dry_run=True))
armed = await server.handle_command(
json.dumps(
{
"type": "command",
"command": "set_armed",
"request_id": "req-armed",
"payload": {"armed": True},
}
)
)
paused = await server.handle_command(
{
"type": "command",
"command": "set_paused",
"request_id": "req-paused",
"payload": {"paused": True},
}
)
assert armed.ok is True
assert armed.request_id == "req-armed"
assert armed.status["armed"] is True
assert paused.ok is True
assert paused.status["paused"] is True
@pytest.mark.asyncio
async def test_motion_agent_open_devices_command_accepts_dual_mode():
server = MotionAgentServer(MotionAgentConfig(dry_run=True))
try:
result = await server.handle_command(
{
"type": "command",
"command": "open_devices",
"request_id": "req-open",
"payload": {"input_mode": "dual_redundant"},
}
)
assert result.ok is True
assert result.status["input_mode"] == "dual_redundant"
assert result.status["active_camera_ids"] == ("dry-run:null-camera",)
assert server._recognition_subprocess is not None
assert server._recognition_subprocess.returncode is None
finally:
await server.stop_recognition_subprocess()
def test_motion_agent_dual_fusion_merges_matching_observations():
server = MotionAgentServer(MotionAgentConfig(dry_run=True))
server.state.mode = "dual_redundant"
selected, fusion = server._fuse_observations(
[
GestureObservation("zoom_in", confidence=0.82, intensity=0.4, camera_id="usb:0"),
GestureObservation("zoom_in", confidence=0.86, intensity=0.8, camera_id="usb:1"),
]
)
assert selected.gesture == "zoom_in"
assert selected.camera_id == "fusion"
assert selected.confidence > 0.86
assert fusion == {
"source_cameras": ["usb:1", "usb:0"],
"window_ms": 120,
"reason": "matched_observations",
}
def test_motion_agent_dual_fusion_suppresses_close_conflict():
server = MotionAgentServer(MotionAgentConfig(dry_run=True, confidence_threshold=0.7))
selected, fusion = server._fuse_observations(
[
GestureObservation("zoom_in", confidence=0.82, intensity=0.5, camera_id="usb:0"),
GestureObservation("zoom_out", confidence=0.78, intensity=0.5, camera_id="usb:1"),
]
)
assert selected.gesture == "zoom_in"
assert selected.confidence == 0
assert fusion["reason"] == "conflict_ignored"
+96 -34
View File
@@ -8,7 +8,7 @@ from app.api.v1 import visualization
from app.api.v1.visualization import convert_vessels_to_geojson
from app.db.session import get_db
from app.main import app
from app.models.vessel import AISRawObservation, VesselPosition, VesselStatic
from app.models.vessel import AISRawObservation, VesselCurrentState, VesselPosition, VesselStatic
from app.services import barentswatch
from app.services.collectors.aisstream import AISStreamCollector
from app.services.collectors.vessel_ais import VesselAISCollector
@@ -17,6 +17,7 @@ from app.services.vessel_ais_aggregation import (
build_field_conflict_candidates,
build_observation_hash,
record_vessel_ais_observation,
upsert_vessel_current_state,
)
@@ -109,6 +110,58 @@ async def test_record_vessel_ais_observation_skips_existing_hash():
assert db.added == []
@pytest.mark.asyncio
async def test_upsert_vessel_current_state_keeps_latest_position_and_static_fields():
current = VesselCurrentState(
mmsi=257123000,
lat=59.91,
lon=10.73,
name="OSLO TRADER",
source="barentswatch_vessels",
observed_at=datetime(2026, 4, 30, 12, 0, tzinfo=timezone.utc),
field_sources={"name": "aisstream_vessels"},
)
class _Session:
async def get(self, _model, _mmsi):
return current
def add(self, _item):
raise AssertionError("existing current state should be updated")
db = _Session()
result = await upsert_vessel_current_state(
db,
source="aisstream_vessels",
normalized_payload={"mmsi": 257123000, "lat": 59.92, "lon": 10.74, "sog": 12.4},
observed_at=datetime(2026, 4, 30, 12, 1, tzinfo=timezone.utc),
)
assert result is current
assert current.lat == pytest.approx(59.92)
assert current.lon == pytest.approx(10.74)
assert current.name == "OSLO TRADER"
assert current.source == "aisstream_vessels"
await upsert_vessel_current_state(
db,
source="barentswatch_vessels",
normalized_payload={"mmsi": 257123000, "lat": 59.93, "lon": 10.75, "name": "LOW PRIORITY"},
observed_at=datetime(2026, 4, 30, 12, 2, tzinfo=timezone.utc),
)
assert current.lat == pytest.approx(59.93)
assert current.name == "OSLO TRADER"
await upsert_vessel_current_state(
db,
source="barentswatch_vessels",
normalized_payload={"mmsi": 257123000, "lat": 1, "lon": 2, "name": "OLD"},
observed_at=datetime(2026, 4, 30, 11, 59, tzinfo=timezone.utc),
)
assert current.lat == pytest.approx(59.93)
assert current.name == "OSLO TRADER"
def test_build_field_conflict_candidates_from_raw_observations():
observations = [
AISRawObservation(
@@ -527,7 +580,7 @@ async def test_vessel_snapshot_filters_type_and_bbox(monkeypatch):
now = datetime(2026, 4, 28, 1, 0, tzinfo=timezone.utc)
monkeypatch.setattr(
visualization,
"get_aggregated_vessels_snapshot",
"get_current_vessels_snapshot",
AsyncMock(
return_value=[
{
@@ -573,6 +626,36 @@ async def test_vessel_snapshot_filters_type_and_bbox(monkeypatch):
app.dependency_overrides.clear()
@pytest.mark.asyncio
async def test_vessel_snapshot_accepts_fractional_zoom(monkeypatch):
monkeypatch.setattr(
visualization,
"get_current_vessels_snapshot",
AsyncMock(return_value=[]),
)
async def override_get_db():
yield object()
app.dependency_overrides[get_db] = override_get_db
transport = ASGITransport(app=app)
try:
async with AsyncClient(transport=transport, base_url="http://test") as client:
response = await client.get(
"/api/v1/vessels/snapshot",
params={
"bbox": "-180,-85.05112878,180,85.05112878",
"zoom": 1.6,
"limit": 3000,
},
)
assert response.status_code == 200
assert response.json()["count"] == 0
finally:
app.dependency_overrides.clear()
@pytest.mark.asyncio
async def test_legacy_vessels_geojson_route_is_not_registered():
transport = ASGITransport(app=app)
@@ -597,7 +680,7 @@ async def test_vessel_snapshot_filters_bbox_and_caps_limit(monkeypatch):
now = datetime(2026, 4, 28, 1, 0, tzinfo=timezone.utc)
captured = {}
async def fake_get_aggregated_vessels_snapshot(db, *, bbox, limit, observed_since):
async def fake_get_current_vessels_snapshot(db, *, bbox, limit, observed_since):
captured["bbox"] = bbox
captured["limit"] = limit
captured["observed_since"] = observed_since
@@ -624,8 +707,8 @@ async def test_vessel_snapshot_filters_bbox_and_caps_limit(monkeypatch):
monkeypatch.setattr(
visualization,
"get_aggregated_vessels_snapshot",
fake_get_aggregated_vessels_snapshot,
"get_current_vessels_snapshot",
fake_get_current_vessels_snapshot,
)
class _Result:
@@ -661,6 +744,8 @@ async def test_vessel_snapshot_filters_bbox_and_caps_limit(monkeypatch):
assert captured["bbox"] == (10.0, 59.0, 11.0, 60.0)
assert captured["limit"] == 5000
assert data["diagnostics"]["bbox_applied"] is True
assert data["diagnostics"]["source"] == "vessel_current_state"
assert data["diagnostics"]["current_state_count"] == 2
assert data["diagnostics"]["legacy_feature_count"] == 0
assert data["diagnostics"]["legacy_backfilled_mmsi"] == 0
finally:
@@ -668,34 +753,12 @@ async def test_vessel_snapshot_filters_bbox_and_caps_limit(monkeypatch):
@pytest.mark.asyncio
async def test_vessel_snapshot_uses_legacy_fallback_when_raw_window_is_empty(monkeypatch):
now = datetime(2026, 4, 28, 1, 0, tzinfo=timezone.utc)
async def test_vessel_snapshot_does_not_fallback_to_history_when_current_state_is_empty(monkeypatch):
monkeypatch.setattr(
visualization,
"get_aggregated_vessels_snapshot",
"get_current_vessels_snapshot",
AsyncMock(return_value=[]),
)
monkeypatch.setattr(
visualization,
"_load_legacy_vessel_snapshot_features",
AsyncMock(
return_value=[
{
"type": "Feature",
"id": 257123000,
"geometry": {"type": "Point", "coordinates": [10.73, 59.91]},
"properties": {
"mmsi": 257123000,
"name": "OSLO TRADER",
"vessel_type": 70,
"vessel_type_name": "Cargo",
"received_at": now.isoformat(),
},
}
]
),
)
result = await visualization.build_vessel_snapshot_response(
object(),
bbox=(10.0, 59.0, 11.0, 60.0),
@@ -705,12 +768,11 @@ async def test_vessel_snapshot_uses_legacy_fallback_when_raw_window_is_empty(mon
since_minutes=60,
)
assert result["count"] == 1
assert result["features"][0]["properties"]["name"] == "OSLO TRADER"
assert result["count"] == 0
assert result["diagnostics"]["raw_feature_count"] == 0
assert result["diagnostics"]["legacy_feature_count"] == 1
assert result["diagnostics"]["legacy_backfilled_mmsi"] == 1
assert result["diagnostics"]["legacy_fallback_used"] is True
assert result["diagnostics"]["legacy_feature_count"] == 0
assert result["diagnostics"]["legacy_backfilled_mmsi"] == 0
assert result["diagnostics"]["legacy_fallback_used"] is False
@pytest.mark.asyncio
+52
View File
@@ -8,6 +8,58 @@ This project follows the repository versioning rule:
- `improvement` -> `+0.0.1`(bugfix + 小功能混合)
- `bugfix` -> `+0.0.1`
## [0.71.1] — 2026-06-26
Released: 2026-06-26
### Highlights
- 修复 Earth 欧洲、美洲、中东与非洲等区域新闻被亚太来源和旧来源过滤饿死的问题,滚动条、面板和巡航重新回到同一批区域 payload。
- 将当前可见新闻和巡航新闻提升到目标位置/翻译优先队列,避免历史普通 Redis backlog 阻塞用户正在看的新闻精修。
- 新增 agent harness 入口、代码地图、验证脚本与双语技术说明,让后续维护能按现有 uv/Bun/Gitea 工作流检查而不替代项目规则。
### Added / Fixed / Improved
- EarthFeed 在全局和巡航队列中按区域轮转候选新闻,保留区域视图的“当前区域 + global”规则,并补充回归测试。
- `news.js` 按区域、类型、来源和数量隔离并发刷新请求,丢弃旧区域响应;跨区域时不再复用旧来源筛选。
- 新闻展示在中文本地化未完成时回退原始标题和摘要,避免出现有内容却显示“新闻汉化中”的卡片。
- 新闻目标位置 worker 新增优先 stream、pending reclaim、任务超时和并发处理;无效消息会确认并删除,减少队列堆积。
- 补充 Earth 新闻源、Earth 前端结构、harness 和版本历史文档,并移除控制台 auth store 的调试日志。
---
## [0.71.0] — 2026-06-11
Released: 2026-06-11
### Highlights
- 将 Motion Agent 升级为可供 Web/UE 共用的双向控制服务,补齐真实 MediaPipe 识别 worker、设备控制、动作白名单和 WSL 摄像头开箱启动链路。
- 新增 Earth 手动新闻内容组、条目、导入与重处理能力,并改进按 locale 和启用来源进行的新闻补充与多样化。
- 对齐动捕模式下的 Earth 点击、详情锁定和卫星轨迹交互,同时完善启动脚本、测试 harness 与运维说明。
### Added / Fixed / Improved
- Motion Agent 支持命令结果、状态、骨架与手势事件,统一单路/双路输入配置,并随 `planet.sh` 默认启动;仓库内提供 usbipd-win fallback 安装包。
- Earth 新闻服务集中处理显示就绪判断和来源多样化,避免存储层与编排层重复筛选;新增手动新闻 API 与回归测试。
- 清理 Motion Agent 重复识别执行路径、前端不稳定随机 key 和过时计划描述,补齐 pytest 路径 harness、双语使用手册、快速开始与数据流文档。
---
## [0.70.0] — 2026-06-04
Released: 2026-06-04
### Highlights
- 新增后端枚举契约治理,将稳定协议状态集中到 `app/core/enums.py`,同时保持数据库和 API 的小写字符串兼容。
- 改进 Earth 新闻分类、重要度与 Breaking 插队链路,并补齐中英文新闻源与枚举契约文档。
- 将 Earth 船只展示改为 `vessel_current_state` 当前状态快照,保留 AIS 原始历史用于轨迹和态势分析。
- 清理错误的船只视口刷新/订阅思路,恢复全球船只显示,并让性能优化集中到批量渲染、关闭动态聚类和减少 hover/rebuild 开销。
### Added / Fixed / Improved
- `earth_news_classification.py` 集中管理新闻类型、重要度和 Breaking 规则,避免抓取编排层重复判断。
- `/api/v1/vessels/snapshot` 支持全球当前状态读取和小数 zoom,诊断信息明确返回 `vessel_current_state` 来源。
- 船只前端使用全球 bbox + `limit=3000`,不再随相机视口重复请求或建立多视口 WebSocket 订阅。
- 中英文技术文档同步更新船只、采集器、数据流、渲染层级、样式参数和历史计划状态。
---
## [0.69.0] — 2026-06-03
Released: 2026-06-03
+167
View File
@@ -0,0 +1,167 @@
# Agent Harness
This harness improves discoverability, repeatability, and agent safety for the
existing Planet project. It does not replace current project rules, scripts, CI,
or release workflows.
## Authority And Conflicts
Existing project rules are authoritative:
1. `rules.md`
2. `agents.md`
3. Current implementation docs under `docs/technical/`
4. Existing scripts, especially `planet.sh`
5. Existing Gitea workflow files under `.gitea/workflows/`
When harness guidance conflicts with any of the above, keep the existing rule,
do not overwrite the existing workflow, and add a compatibility note here or in
`docs/harness-audit.md`.
## Starting Work
Recommended startup flow:
```bash
git status --short
scripts/harness/doctor.sh
```
Then read only the relevant implementation docs:
- Backend/API/data work: `docs/technical/zh/backend-*.md` and matching English
docs when public docs are affected.
- Frontend/admin work: `docs/technical/zh/frontend-admin-frontend-context.md`.
- Earth work: `docs/technical/zh/earth-frontend-context.md`,
`docs/technical/zh/earth-render-layer-order.md`, and style docs when visual
semantics change.
- Operations work: `docs/technical/zh/ops-runbook.md` and
`docs/technical/zh/ops-planet-sh-startup.md`.
- AI Provider work: `docs/technical/zh/agents-aiprovider.md`.
- Documentation work: `docs/documentation-coverage-rules.md`.
Use focused inspection commands before broad reads:
```bash
rg -n "<symbol-or-term>" <path>
git diff --stat HEAD
git diff --name-only HEAD
git diff --unified=0 HEAD -- <path>
```
## Existing Commands
| Purpose | Command |
| --- | --- |
| First setup | `./planet.sh init` |
| Start local stack | `./planet.sh start` |
| Start with LAN access | `./planet.sh start --allow-lan` |
| Restart all services | `./planet.sh restart` |
| Restart one area | `./planet.sh restart -b`, `-f`, `-a`, or `-d` |
| Health check | `./planet.sh health` |
| Logs | `./planet.sh log`, `./planet.sh log -b`, `-f`, `-a`, or `-m` |
| Create local user | `./planet.sh createuser` |
| Destructive local reset | `./planet.sh destroy` |
| Backend smoke tests | `cd backend && uv run --frozen --group dev --project .. python -m pytest -s tests/test_api.py tests/test_realtime_sources.py -q` |
| Frontend build | `cd frontend && bun install --frozen-lockfile && bun run build` |
| Mock AIS WebSocket | `bun run mock:ais-ws` |
## Harness Commands
| Tier | Command | What It Does |
| --- | --- | --- |
| Doctor | `scripts/harness/doctor.sh` | Checks required files, required tools, optional delivery tools, and forbidden frontend lockfiles. |
| Quick | `scripts/harness/quick-check.sh` | Runs doctor, whitespace diff check, shell syntax checks, and CI backend smoke tests. |
| Full | `scripts/harness/validate.sh` | Runs quick check, frontend Bun install/build, optional Helm checks, and opt-in Docker image smoke builds. |
Docker image smoke builds are expensive and are off by default:
```bash
PLANET_HARNESS_DOCKER_SMOKE=1 scripts/harness/validate.sh
```
## Environment Requirements
Required for normal development:
- `zsh` for `planet.sh`
- `uv` for Python dependency and test execution
- `bun` for frontend dependency and build execution
- Python resolved by `uv` from the root `pyproject.toml`
Required for full local stack operation:
- Docker and Docker Compose
- PostgreSQL and Redis containers started by `planet.sh`
Optional for delivery smoke:
- Docker daemon for image builds
- Helm for chart lint/template checks
If a required local tool is missing, do not install system software
automatically. Report the gap and point to `./planet.sh init` or
`scripts/bootstrap-dev.sh` as the existing bootstrap path.
## What Agents Must Not Change Automatically
- Do not replace Bun with npm, pnpm, or yarn.
- Do not migrate CI from `.gitea/workflows/` to `.github/workflows/`.
- Do not rewrite `planet.sh` lifecycle behavior as a parallel script.
- Do not run `./planet.sh destroy` unless explicitly requested.
- Do not commit `.env`, secrets, private keys, logs, or generated build output.
- Do not add external integrations, hooks, or new dependency managers just to
satisfy harness structure.
- Do not publish internal harness docs into the product Docs UI unless a
maintainer explicitly asks for it.
## Hooks And Reminders
No automatic hooks are installed in this phase. Manual reminders:
- Run `scripts/harness/quick-check.sh` before handing off small changes.
- Run `scripts/harness/validate.sh` before larger cross-subsystem changes.
- Add focused tests before modifying backend service behavior or frontend
workflows.
- For docs changes, run the checks listed in
`docs/documentation-coverage-rules.md`.
## Reusable Workflows
### Feature Work
1. Read `rules.md` modules for the touched area.
2. Check `CODEMAP.md` for entry points and ownership boundaries.
3. Inspect existing tests and docs before editing.
4. Make the smallest behavior-preserving or feature-scoped change.
5. Run `scripts/harness/quick-check.sh` or a narrower documented command.
6. Update relevant docs when behavior, workflow, or operations change.
### Bug Fix
1. Reproduce with a focused test or command.
2. Patch the owning module, not a caller-side workaround.
3. Run the focused regression test.
4. Run `scripts/harness/quick-check.sh` when the change is safe to validate
locally.
### Documentation Change
1. Read `docs/documentation-coverage-rules.md`.
2. Route docs by audience: UI users, operations, or second-party developers.
3. Keep Chinese and English public docs consistent when a public doc pair exists.
4. Run the repository-specific docs checks that match the changed files.
### Release Or Delivery Change
Use the existing release skill/workflow and `.gitea/workflows/` files. Harness
validation can smoke-check Helm and Docker locally, but it must not replace the
release process.
## Implementation Notes
- `docs/harness-audit.md` records the discovery pass that led to this harness.
- `AGENTS.md` is a compatibility entry point for tools that expect the uppercase
filename. The existing `agents.md` file remains in place.
- `CODEMAP.md` is intentionally high level; deeper subsystem docs stay in
`docs/technical/{zh,en}/`.
+102
View File
@@ -0,0 +1,102 @@
# Harness Audit
Last audited: 2026-06-26
This audit records the repository state used to add the agent harness. It is a
compatibility note, not a replacement for existing rules or architecture docs.
## Existing Commands
| Area | Existing Command | Notes |
| --- | --- | --- |
| Bootstrap | `./planet.sh init` | Syncs uv/Bun dependencies, creates missing env files, starts data services, seeds defaults. |
| Start | `./planet.sh start` | Starts backend, frontend, AI Provider, PostgreSQL/Redis, and Motion Agent when available. |
| LAN start | `./planet.sh start --allow-lan` | Opens frontend/backend/AI Provider ports and requests Windows firewall/port cleanup when needed. |
| Restart | `./planet.sh restart` | Supports scoped restart flags for backend, frontend, AI Provider, database, and Motion Agent. |
| Health | `./planet.sh health` | Checks containers, backend `/health`, AI Provider `/health`, frontend, and Motion Agent state. |
| Logs | `./planet.sh log` | Supports backend, frontend, AI Provider, and Motion Agent log views. |
| User fallback | `./planet.sh createuser` | Interactive emergency/local account creation. |
| Destructive reset | `./planet.sh destroy` | Requires confirmation and removes Planet-owned Docker/build/runtime state. Not a validation command. |
| Backend CI smoke | `cd backend && uv run --frozen --group dev --project .. python -m pytest -s tests/test_api.py tests/test_realtime_sources.py -q` | Mirrors `.gitea/workflows/ci.yaml`. |
| Frontend build | `cd frontend && bun install --frozen-lockfile && bun run build` | Bun-only workflow. |
| Root helper | `bun run mock:ais-ws` | Runs `scripts/mock-ais-ws-server.ts` from the root package. |
## Existing Agent Instructions
| File | Status | Notes |
| --- | --- | --- |
| `agents.md` | Present | Existing root agent behavior guide. It references `rules.md` and `project_context.md`. |
| `rules.md` | Present | Mandatory modular rules. Always load `core`, `security`, and `workflow`; load topic modules as needed. |
| `project_context.md` | Present | Static context. Some roadmap-era stack details are older than the current README/docs. |
| `.claude/commands/*.md` | Present | Existing command docs for cleanup, docs, goal-driven, and release workflows. |
| `.codex/skills/*.md` | Present | Existing local skills for cleanup, docs, goal-driven, and release. |
| `AGENTS.md` | Added by harness | Compatibility entry point that points to existing rules and harness docs. |
## Existing CI Gates
The repository uses `.gitea/workflows/`, not `.github/workflows/`.
| Workflow | Gate |
| --- | --- |
| `.gitea/workflows/ci.yaml` | Backend smoke tests, frontend Bun build, Docker build smoke, Helm lint/template. |
| `.gitea/workflows/release.yaml` | Builds and pushes frontend, backend, and AI Provider images on main/tag/manual release events. |
| `.gitea/workflows/deploy-staging.yaml` | Deploys Helm release to staging and runs curl smoke tests inside the cluster. |
## Existing Docs And Architecture Maps
| Area | Docs |
| --- | --- |
| Current architecture and startup | `README.md` |
| Technical docs index | `docs/technical/zh/README.md`, `docs/technical/en/README.md` |
| Documentation rules | `docs/documentation-coverage-rules.md` |
| Operations | `docs/technical/zh/ops-runbook.md`, `docs/technical/en/ops-runbook.md` |
| Startup internals | `docs/technical/zh/ops-planet-sh-startup.md`, `docs/technical/en/ops-planet-sh-startup.md` |
| AI Provider | `docs/technical/zh/agents-aiprovider.md`, `docs/technical/en/agents-aiprovider.md` |
| Frontend admin | `docs/technical/zh/frontend-admin-frontend-context.md`, `docs/technical/en/frontend-admin-frontend-context.md` |
| Earth rendering | `docs/technical/zh/earth-frontend-context.md`, `docs/technical/zh/earth-render-layer-order.md`, `docs/technical/zh/earth-layer-style-reference.md` |
| Plans and history | `docs/plans/README.md`, `docs/deprecated/README.md` |
## Release And Deploy Process
- Release workflow is documented in `.codex/skills/release/SKILL.md` and
`.claude/commands/release.md`.
- Version-bearing files include `VERSION`, `frontend/package.json`,
`pyproject.toml`, `uv.lock`, `docs/CHANGELOG.md`, and
`docs/version-history.md`.
- Delivery automation lives in `.gitea/workflows/release.yaml` and
`.gitea/workflows/deploy-staging.yaml`.
- Helm chart entry point is `deploy/helm/planet/Chart.yaml`.
## Missing Or Unclear Areas
- README previously listed `AGENTS.md` in the project tree while only lowercase
`agents.md` existed. The harness adds uppercase `AGENTS.md` as a compatibility
wrapper and preserves `agents.md`.
- `project_context.md` includes older roadmap assumptions such as Celery, Kafka,
TimescaleDB, MinIO, and UE5 as active stack elements. The current README and
technical docs describe Web Earth, React admin, FastAPI, PostgreSQL/Redis, and
`aiprovider` as the active local development shape.
- No safe automatic hook system was already configured. This phase documents
manual reminders instead of adding hooks.
- `.github/workflows/` is absent by design; CI is under `.gitea/workflows/`.
## Conflicts And Preserved Rules
| Conflict Or Tension | Resolution |
| --- | --- |
| Prompt suggested `AGENTS.md`; repository already had `agents.md`. | Added a minimal uppercase compatibility entry and preserved the existing lowercase guide. |
| Harness validation could duplicate CI. | Added wrapper scripts that call existing commands and mirror current CI gates where practical. |
| Full Docker smoke builds are expensive locally. | Kept them opt-in with `PLANET_HARNESS_DOCKER_SMOKE=1`. |
| Internal harness docs could clutter public Docs UI. | Kept `docs/HARNESS.md` and `docs/harness-audit.md` as repository docs, not product Docs entries. |
| Existing frontend toolchain is Bun-only. | Harness scripts and docs use Bun only and flag npm/pnpm/yarn lockfiles as failures. |
## Harness Files Added
| File | Purpose |
| --- | --- |
| `AGENTS.md` | Compatibility agent entry point. |
| `docs/HARNESS.md` | Harness workflow, validation tiers, conflict policy, and manual reminders. |
| `CODEMAP.md` | High-level codebase map and validation references. |
| `scripts/harness/doctor.sh` | Environment and repository-shape check. |
| `scripts/harness/quick-check.sh` | Fast deterministic local validation. |
| `scripts/harness/validate.sh` | Full local validation wrapper with optional delivery smoke. |
+1
View File
@@ -33,6 +33,7 @@
- [Earth News Cruise Summary Plan](/home/ray/dev/linkong/planet/docs/plans/earth-news-cruise-summary-plan.md)
- [Earth 动作捕捉手势控制计划](/home/ray/dev/linkong/planet/docs/plans/earth-motion-capture-gesture-control-plan.md)
- [Earth 动捕交互语义 V2 计划](/home/ray/dev/linkong/planet/docs/plans/earth-motion-gesture-interaction-v2-plan.md)
- [Motion Agent v2 控制协议与 3D 标定路线](/home/ray/dev/linkong/planet/docs/plans/motion-agent-v2-control-protocol-plan.md)
- [Earth Presentation 解耦架构计划](/home/ray/dev/linkong/planet/docs/plans/earth-presentation-decoupled-architecture-plan.md)
- [Earth Vessel Rendering Performance Plan](/home/ray/dev/linkong/planet/docs/plans/earth-vessel-rendering-performance-plan.md)
- [AIS 多源采集、冲突记录与聚合接口计划](/home/ray/dev/linkong/planet/docs/plans/earth-vessel-ais-aggregation-plan.md)
@@ -1,5 +1,7 @@
# Earth Motion Capture Gesture Control Plan
> Update: Motion Agent process/device control, UE/Web shared command protocol, dual-camera redundant fusion, and the next 3D calibration route are now tracked in [Motion Agent v2 Control Protocol And 3D Calibration Roadmap](/home/ray/dev/linkong/planet/docs/plans/motion-agent-v2-control-protocol-plan.md). This document remains useful for the original provider split and gesture-control intent.
## Goal
为 Planet Earth 大屏和未来 3D 展示增加一套解耦的动作捕捉手势控制能力。实时输入分成两条路线:网页端可直接通过浏览器 `getUserMedia` 在本机识别;高级设备可继续使用本机 Motion Capture Edge Agent。两条路线都只输出轻量语义事件,客户端负责把“手势事件”映射到“具体交互函数”。
@@ -2,6 +2,8 @@
**状态**:已实现主体交互,并按实测调整。当前浏览器识别保留右手导航、头部切目标、左手上下切动捕图层、双手张开/收拢缩放;双手上举确认暂时关闭。Motion 目标展示已改为 `CruiseSequencer` + `PresentationController` 的 persistent 展示。
> Update: Agent-side bidirectional commands, UE/Web shared device control, dual-camera redundant fusion, and future calibrated 3D mode are tracked in [Motion Agent v2 Control Protocol And 3D Calibration Roadmap](/home/ray/dev/linkong/planet/docs/plans/motion-agent-v2-control-protocol-plan.md).
## Summary
把动捕从“几个单点手势触发函数”升级为一套更像大屏遥控器的交互层:右手负责地球导航,头部负责候选切换,左手上下切换动捕候选图层,双手负责缩放,调试面板支持“只显示骨骼”和暂停匹配。进入动捕模式后,Earth 自动软选中屏幕中心附近的正面可交互目标;确认动作预留为把目标升级为锁定,并用巡航/引导线式详情打开,不再模拟鼠标点击。
@@ -1,6 +1,6 @@
# AIS 多源采集、冲突记录与聚合接口计划
**状态**:v0-v3 已实现,v3.1-v3.4 为 v4/v5 前置稳定化任务,v4 / v5 已落最小可用子集
**状态**:v0-v3 已实现,v3.1-v3.4 为 v4/v5 前置稳定化任务,v4 / v5 已落最小可用子集;`0.70.0` 起 Earth 展示路径已转向 `vessel_current_state` 当前状态表和全球快照,不再由 raw observation 聚合结果直接驱动首屏显示。
**创建日期**:2026-04-30
**核心原则**:采集器只写原始观测;去重、合并、冲突解释放在聚合接口中完成
@@ -344,11 +344,11 @@ VesselFinder 等服务里的船只图片不属于 AIS 实时数据本身。图
3. 聚合结果返回 `source_summary`,展示每艘船的来源、观测数量、最新观测时间、传输模式和消息类型。
4. 保留 `field_sources` 和 `selected_reasons`,用于解释动态字段来自实时流、静态字段来自可用非空来源。
5. 船名标准化会读取 AISStream `MetaData.ShipName`;船型展示会从 `vessel_type_name` 和 AIS 数字 `vessel_type` 共同归一化,保证 marker 颜色、详情卡、hover 和搜索结果一致。
6. 当前实现已转向 `/api/v1/vessels/snapshot`:必须带 bbox / zoom,默认 `limit=1000`,最大 `limit=5000`,不再支持旧 `/geo/vessels` 全量返回。
6. 当前实现已转向 `/api/v1/vessels/snapshot`:必须带 bbox / zoom,Earth 运行时使用全球 bbox 和 `limit=3000`,后端最大 `limit=5000`,不再支持旧 `/geo/vessels` 全量返回。
### v3.1 — 聚合完整性修复(已被受控 fallback 取代)
### v3.1 — 聚合完整性修复(已被当前状态表取代)
原目标是先保证“所有已采集到的船都能显示”,BarentsWatch 不因为接入 AISStream 而被 raw observation 聚合结果遮蔽。当前实现已经移除旧 `/geo/vessels` 路由,船只入口统一为 `/api/v1/vessels/snapshot`。snapshot 优先读取 `ais_raw_observations` 聚合结果;当当前 raw 窗口为空时,才受控回退到 `vessel_position + vessel_static` 最新点,并通过 `diagnostics.legacy_fallback_used` 标记。
原目标是先保证“所有已采集到的船都能显示”,BarentsWatch 不因为接入 AISStream 而被 raw observation 聚合结果遮蔽。当前实现已经移除旧 `/geo/vessels` 路由,船只入口统一为 `/api/v1/vessels/snapshot`。snapshot 现在读取 `vessel_current_state` 当前状态表;`ais_raw_observations` 继续保留给轨迹、审计和态势分析,不再由展示接口临时扫描聚合。
因此以下旧 `/geo/vessels` 全量 merge 要求作废,保留在文档中只作为历史决策记录:
@@ -9,8 +9,9 @@
- hover / locked 不再放大成世界尺寸 Sprite,而是在原点位叠加同尺寸单点 glow overlay。
- picking 改为屏幕空间命中,拖动和惯性期间跳过 hover picking。
- 普通态关闭 glow,交互态才显示 glow,降低 overdraw 并让默认地图更干净。
- `0.70.0` 起船只继续走全球当前状态快照:`vessel_current_state` + 全球 bbox + `limit=3000`。上一轮“按当前镜头 bbox 刷新/订阅”的方向已撤回,性能优化集中在批量渲染、关闭动态聚类和减少 picking / rebuild 开销。
后续如果需要全球 AIS 或更高船只密度,再评估是否从分桶 `Points` 升级到真正 instanced quad 或视口 bbox / LOD。
后续如果需要更高船只密度,再评估是否从分桶 `Points` 升级到真正 instanced quad 或服务端 LOD;不要把普通旋转/缩放重新接回视口驱动请求。
## 背景
@@ -214,9 +215,9 @@ hover、locked、dimmed 可通过更新少量 instance attribute 实现,不再
当接入全球 AIS 或船只数量显著增加时,再做数据层优化。
### 1. 请求视口范围
### 1. 当前状态快照
前端请求 `/api/v1/vessels/snapshot` 时必须带上当前视口 `bbox`、`zoom` 和受控 `limit`,减少无关船只。旧 `/api/v1/visualization/geo/vessels` 路由已移除。
前端请求 `/api/v1/vessels/snapshot` 时仍必须带 `bbox`、`zoom` 和受控 `limit`,但 Earth 运行时传全球 bbox 和 `limit=3000`,不使用当前镜头视口驱动刷新。旧 `/api/v1/visualization/geo/vessels` 路由已移除。
### 2. 后端排序策略
@@ -224,7 +225,7 @@ hover、locked、dimmed 可通过更新少量 instance attribute 实现,不再
- 数据新鲜度
- 船型优先级
- 当前视口相关性
- 全球当前状态可读性
- 是否正在航行
### 3. 远景聚合
+3 -2
View File
@@ -1,6 +1,7 @@
# 实时船只监控系统 — 实施计划
**状态**:历史计划;实时 AIS 与聚合接口已由 [AIS 多源采集、冲突记录与聚合接口计划](/home/ray/dev/linkong/planet/docs/plans/earth-vessel-ais-aggregation-plan.md) 接管
**更新**:`0.70.0` 起 Earth 船只展示使用 `vessel_current_state` 全球当前状态快照;本文中按当前视口 bbox 驱动显示的 LOD 设想仅作为历史记录。
**创建日期**:2026-04-27
**优先数据源**:BarentsWatch AIS(免费但需要 OAuth client credentials)→ AISStream realtime;AISHub / MarineTraffic 保留为付费备选
@@ -194,9 +195,9 @@ GeoJSON Feature 格式:
|---------|---------|
| > 400 | 默认渲染当前接口返回的全部船只;如性能不足,再引入可配置 LOD 上限 |
| 200–400 | 默认渲染当前接口返回的全部船只;如性能不足,再引入可配置 LOD 上限 |
| < 200 | 渲染当前视口 bbox 内全部船只 |
| < 200 | 历史设想:渲染当前视口 bbox 内全部船只;当前实现仍使用全球当前状态快照 |
前端根据相机位置动态计算 bbox,附加到 API 请求中。
该视口驱动方案已撤回。当前前端使用全球 bbox 请求 `/api/v1/vessels/snapshot`,旋转和缩放不触发船只重拉。
#### 2.4 图层集成
@@ -0,0 +1,175 @@
# Motion Agent v2 Control Protocol And 3D Calibration Roadmap
## Summary
Motion Agent v2 turns the local motion service into a shared WebSocket control plane for the Web Earth page and UE clients. The agent process is still started externally through `planet.sh`, a desktop service, or UE process management. Once the process is running, clients can open cameras, close cameras, switch input mode, arm or pause recognition, and inspect status through the same bidirectional WebSocket protocol.
This phase implements robust single-camera and dual-camera redundant fusion. True calibrated 3D skeleton reconstruction is deliberately reserved for the v3 calibration phase.
## v2 Protocol
The default endpoint remains:
```text
ws://127.0.0.1:8765/ws/gestures
```
The agent emits:
- `gesture`
- `skeleton`
- `status`
- `heartbeat`
- `command_result`
Clients send:
- `open_devices`
- `close_devices`
- `rescan_devices`
- `set_armed`
- `set_paused`
- `set_input_mode`
- `set_camera_config`
- `set_fusion_config`
- `set_debug_options`
- `set_enabled_gestures`
- `get_status`
- `ping`
All v2 messages include additive compatibility fields such as `protocol_version`, `request_id`, `camera_id`, `input_mode`, and optional `fusion`. Older push-only clients can continue to consume `gesture`, `skeleton`, `status`, and `heartbeat`.
Example command:
```json
{
"type": "command",
"command": "open_devices",
"request_id": "req-001",
"payload": {
"input_mode": "dual_redundant",
"camera_indexes": [0, 1]
}
}
```
Example gesture whitelist command:
```json
{
"type": "command",
"command": "set_enabled_gestures",
"request_id": "earth-motion-enabled-gestures",
"payload": {
"gestures": ["rotate_left", "rotate_right", "zoom_in", "zoom_out", "confirm"]
}
}
```
Example result:
```json
{
"type": "command_result",
"protocol_version": "motion.v2",
"request_id": "req-001",
"command": "open_devices",
"ok": true,
"status": {
"armed": false,
"paused": false,
"input_mode": "dual_redundant",
"devices_open": true,
"active_camera_ids": ["usb:0", "usb:1"]
}
}
```
## Device And Control State
Process startup is not a WebSocket feature: the server must exist before a WebSocket client can connect. UE should start the agent as an external process or depend on a system service, then connect to the WebSocket endpoint.
Device wake and control wake are WebSocket features:
- `open_devices` opens USB cameras or URL cameras.
- `close_devices` releases them.
- `set_armed` enables gesture execution.
- `set_paused` pauses recognition without closing the connection.
When `armed=false`, the agent may still emit skeleton and status events, but it does not emit actionable gesture events.
The Earth settings dialog owns a user-facing gesture whitelist. Unchecked gestures are ignored in the Earth client and are also sent to the Motion Agent through `set_enabled_gestures`, so the server does not broadcast disabled actions to UE/Web consumers. The default whitelist enables the full v2 gesture set; disabling gestures is a local display/control preference and does not change the installed recognition model.
## Input Modes
- `single`: one camera.
- `dual_redundant`: two or more cameras observe the same gesture. Matching observations in a short window are fused into a higher-confidence event.
- `single_fallback`: the primary camera is preferred and a secondary input is used as fallback.
- `calibrated_3d`: reserved for v3 and should not be enabled unless a calibration profile exists.
The v2 dual-camera mode is redundant fusion, not 3D reconstruction. It is meant to improve reliability under occlusion and camera noise without requiring calibration.
## v3 3D Calibration Roadmap
The next phase is `Motion Agent v3 3D Calibration`. It upgrades from redundant fusion to calibrated multi-camera skeleton fusion.
Planned capabilities:
- Camera intrinsics: focal length, distortion, resolution.
- Camera extrinsics: relative position, rotation, and baseline distance.
- Calibration workflow: checkerboard, AprilTag, or ArUco board.
- Local calibration profile JSON with query, load, reset, and validation commands.
- `skeleton_3d` event with world-space joints, confidence, and source cameras.
- UE coordinate mapping from Motion Agent coordinates to UE world or widget coordinates.
Reserved v3 input configuration:
```json
{
"input_mode": "calibrated_3d",
"calibration_profile": "desk-dual-camera-v1"
}
```
Reserved v3 event:
```json
{
"type": "skeleton_3d",
"protocol_version": "motion.v3",
"profile": "desk-dual-camera-v1",
"joints": [
{
"name": "right_wrist",
"x": 0.42,
"y": 1.13,
"z": 0.76,
"confidence": 0.91
}
]
}
```
## Test Plan
- Command/result roundtrip for device open, close, rescan, armed, paused, and status.
- Gesture protocol accepts the full Earth v2 gesture set.
- Earth settings can disable individual gestures; disabled gestures are ignored locally and filtered server-side through `set_enabled_gestures`.
- Motion Agent `status` reports the current enabled gesture list for Web/UE diagnostics.
- Single and dual redundant fusion emit compatible gesture payloads.
- Conflicting dual-camera observations below the confidence delta are ignored.
- Web Earth `motion-agent-provider` can send commands over the same socket it uses for events.
- UE mock clients can operate the service without browser-only assumptions.
- Dry-run mode can test commands, status, skeleton, and fusion behavior without camera dependencies.
## Current Limitations
- Production recognition uses a real MediaPipe pose pipeline and heuristic gesture recognizer. It still needs environment-specific threshold tuning, camera framing validation, and long-running reliability checks before it can be treated as calibration-free.
- Dual-camera v2 does not triangulate 3D joint positions.
- `calibrated_3d` is documented as a reserved mode and must not be treated as implemented until v3 lands.
## Implementation Status
- Implemented: bidirectional command/result protocol, device lifecycle controls, dry-run mode, subprocess recognition worker, MediaPipe pose recognition, gesture whitelist, single-camera mode, dual-redundant/fallback scaffolding, Web client integration, and default `planet.sh` lifecycle integration.
- Remaining v2 hardening: tune recognition thresholds across camera placements, exercise UE command/control integration, and run longer soak tests for device reconnect and dual-camera conflicts.
- Planned v3: calibrated multi-camera 3D skeleton fusion and Motion Agent-to-UE coordinate calibration.
+1
View File
@@ -38,6 +38,7 @@ This is the current Intelligent Planet documentation entry point. Docs are organ
- [Datasource Collector Settings and Connectivity](/home/ray/dev/linkong/planet/docs/technical/en/datasource-collector-settings-connectivity.md): datasource catalog, collector settings, connectivity validation, and credentials
- [Datasource API Performance](/home/ray/dev/linkong/planet/docs/technical/en/backend-datasources-api-performance.md): DataSources list API performance and caching
- [Data Jobs and Outbox Architecture](/home/ray/dev/linkong/planet/docs/technical/en/data-job-earth-sync-architecture.md): PostgreSQL job queue, outbox, listener, and Kafka / Spark evolution boundaries
- [Backend Enum and String Compatibility Contract](/home/ray/dev/linkong/planet/docs/technical/en/backend-enum-contracts.md): stable protocol states, historical string compatibility, and Earth news decision boundaries
- [Shared Location Resolution Pipeline Development Guide](/home/ray/dev/linkong/planet/docs/technical/en/location-pipeline-development.md): location resolver / pipeline interfaces, registries, and extension points
- [News Live Streams Collector Format](/home/ray/dev/linkong/planet/docs/technical/en/earth-news-live-streams-collector-format.md): news, live stream, and media collection payload conventions
- [Docs Gatekeeper Development Guide](/home/ray/dev/linkong/planet/docs/technical/en/docs-gatekeeper-development.md): backend Docs catalog, Markdown content loading, and Gatekeeper permission groups
+7 -19
View File
@@ -93,7 +93,9 @@ The CelesTrak TLE collector prefers the complete `active` catalog. If CelesTrak
| BarentsWatch AIS | vessel | AIS vessel positions, speed, heading, MMSI, and related fields | Collector settings |
| AISStream Vessels | vessel_ais | AIS WebSocket realtime stream, written to the raw observation layer and displayed through aggregation | Collector settings |
AIS vessel collectors use a different persistence path from regular `CollectedData` collectors. BarentsWatch, AISStream, and custom `vessel_ais` sources write into the AIS raw observation layer first, then the aggregation service merges those observations into the GeoJSON and detail payloads used by the Earth vessel layer. This preserves source, transport, field conflicts, and observation time instead of letting one realtime source overwrite the final display table.
AIS vessel collectors use a different persistence path from regular `CollectedData` collectors. BarentsWatch, AISStream, and custom `vessel_ais` sources write into the AIS raw observation layer and also upsert `vessel_current_state`: one row per MMSI with the latest position, speed, course, navigation state, vessel type, name, and source metadata. This preserves raw observation history for tracks, audit, and situational analysis while letting the Earth vessel layer read the current-state table instead of scanning historical AIS rows.
`vessel_current_state` only lets newer observations overwrite dynamic position fields; static fields such as name and vessel type are merged by non-empty value and source priority. Earth snapshots return only vessels inside the freshness window, keeping high-frequency AIS history out of the display path.
Earth boundaries are no longer data collectors. They are Earth static rendering assets: the Earth Assets settings panel owns source configuration, and `/api/v1/earth/boundaries/*` builds `frontend/public/earth/data/boundaries/earth-boundaries-china-pov-v1.pmtiles`. When no high-precision PMTiles artifact is available locally, the frontend uses the bundled low-precision GeoJSON fallback and does not write boundary records to `CollectedData`.
@@ -339,32 +341,18 @@ AIS observations do not directly replace final vessel records. They are first sa
- Dynamic fields such as position, speed, and course are selected by freshness and source priority.
- Static fields prefer non-empty values; conflicting candidates are recorded for detail and diagnostics views.
Earth vessel rendering now consumes the bounded snapshot endpoint and realtime delta channel:
Earth vessel rendering now consumes the current-state snapshot endpoint:
```http
GET /api/v1/vessels/snapshot?bbox=lon_min,lat_min,lon_max,lat_max&zoom=12&limit=1000
GET /api/v1/vessels/snapshot?bbox=-180,-85.05112878,180,85.05112878&zoom=12&limit=3000
GET /api/v1/visualization/vessels/{mmsi}
GET /api/v1/visualization/vessels/{mmsi}/track
GET /api/v1/visualization/vessels/{mmsi}/conflicts
```
`/api/v1/vessels/snapshot` requires `bbox` and `zoom`, defaults to `limit=1000`, and caps `limit` at `5000`. It prefers aggregated `ais_raw_observations`; when the current raw window is empty, it can fall back to the latest legacy `vessel_position` / `vessel_static` rows and marks that path with `diagnostics.legacy_fallback_used`. The old `/api/v1/visualization/geo/vessels` route has been removed.
`/api/v1/vessels/snapshot` requires `bbox` and `zoom`, and caps `limit` at `5000`. The Earth frontend uses a global bbox for current state and does not refetch on camera viewport changes. The endpoint reads `vessel_current_state` and reports `diagnostics.source = "vessel_current_state"`. The old `/api/v1/visualization/geo/vessels` route has been removed.
Realtime deltas are sent through the `/ws` `vessels` channel. Clients must subscribe with the current viewport:
```json
{
"type": "subscribe",
"data": {
"channel": "vessels",
"bbox": [120.8, 30.7, 122.1, 31.8],
"zoom": 12,
"limit": 1000
}
}
```
The backend stores lightweight subscription filters per connection and only sends vessel updates that match the subscriber bbox. Collector broadcasts enter a 1-second throttle queue; within each flush window, only the latest update per MMSI is retained.
High-frequency AIS updates must not become per-delta full-layer rebuilds. If Earth uses the `/ws` `vessels` channel, it should send low-frequency reload/dirty hints and let the frontend merge snapshot refreshes. Tracks and conflicts still read historical facts through the single-vessel APIs.
### Layer APIs And Global Stats
@@ -0,0 +1,50 @@
# Backend Enum and String Compatibility Contract
Planet keeps finite, stable protocol states in `backend/app/core/enums.py`. Databases and APIs continue to store and return lowercase strings. Enums provide internal type safety, validation, and deduplication without requiring a migration to database enum types.
## Selection Rules
A value should become an enum only when it is finite and stable, invalid outside the known set, and compared, sorted, or branched on by multiple modules. Typical examples are job states, AI Playground message roles and states, user roles, alert states, log levels, news importance, and Breaking states.
The following must remain configurable strings:
- news categories and tags;
- provider, model, datasource, collector, news source, and Feed identifiers;
- extensible incident and anomaly types; and
- user-authored and free-form text.
## Boundary Conversion
Services should use `StrEnum` internally. Database writes and external responses use `.value`, preserving existing strings such as `JobStatus.RUNNING.value == "running"`.
Use the compatibility parser when reading historical database values, JSON, or external input:
```python
status = parse_enum(JobStatus, raw_status, JobStatus.FAILED)
```
Known historical strings normalize to enum members. Empty values use the explicit default. Unknown values emit a warning and safely fall back instead of breaking historical reads. Pydantic request fields may use enums directly so invalid protocol values return `422`. Ordinary String and JSON database columns must not be converted to SQLAlchemy Enum.
## Earth News Decisions
Earth news decisions live in `backend/app/services/earth_news_classification.py`:
- category answers what the story is and remains configurable;
- importance answers whether it has long-term value; and
- Breaking answers whether it must temporarily jump ahead.
Importance levels are fixed:
| Level | Score |
|---|---:|
| `low` | 0–34 |
| `medium` | 35–59 |
| `high` | 60–79 |
| `critical` | 80–100 |
Breaking rules use the typed `BreakingRule` structure. Levels, scopes, sources, and TTLs are managed by the public classification module. `earth_news.py` remains responsible for fetching, parsing, orchestration, and serialization.
## Regression Prevention
When adding or changing protocol states, check `app/core/enums.py` first and do not redefine duplicate `Literal` aliases, status sets, or normalization helpers in business modules. Add enum value and boundary contracts to `tests/test_enum_contracts.py`, and preserve API strings and database representation.
@@ -318,10 +318,10 @@ Connectivity validation and actual collection are separate actions. A banner suc
The new vessel list entry point is no longer the legacy `/api/v1/visualization/geo/vessels` route. Earth initial state should call:
```http
GET /api/v1/vessels/snapshot?bbox=lon_min,lat_min,lon_max,lat_max&zoom=12&limit=1000
GET /api/v1/vessels/snapshot?bbox=-180,-85.05112878,180,85.05112878&zoom=12&limit=3000
```
That endpoint prefers local aggregated `ais_raw_observations`; when the current raw window is empty, it can fall back to the latest legacy `vessel_position` / `vessel_static` rows and exposes that through `diagnostics.legacy_fallback_used`. Realtime updates use the `/ws` `vessels` channel; subscriptions must include `bbox`, `zoom`, and `limit`. The server filters updates per connection and merges collector broadcasts every second, keeping only the latest position per MMSI.
That endpoint reads `vessel_current_state`, returning the latest point per MMSI inside the freshness window. Raw `ais_raw_observations` remain available for tracks, audit, and situational analysis, but the display endpoint no longer scans and aggregates history on the fly. Earth sends a global bbox rather than the current camera viewport. If the `/ws` `vessels` channel is connected, it should act as a reload/dirty hint for merged refreshes, not as a per-AIS-delta full-layer rebuild path.
## Custom REST / WebSocket Mapping Runtime
+12 -4
View File
@@ -77,6 +77,8 @@ Earth settings are now grouped by `data-settings-tab` and `data-settings-tab-pan
The news category selector in Display reuses the same chip-selector pattern as Cruise Modules. It only filters news categories for the current browser on the Earth frontend. It does not toggle layers, basemap, boundaries, TV, data points, BGP, vessels, satellites, or compute centers; those remain owned by the layer panel, media panel, and admin configuration. `controls.js` persists only `shared.newsCategoryFilters` and broadcasts `earth:news-category-filters-change`; `news.js` sends the selected categories to `/api/v1/news/earth-feed?categories=...&locale=zh-CN`, so Web and UE clients share the same backend category filtering path.
The news panel, ticker, and news cruise must consume `items` / `cruise_items` from the same `/api/v1/news/earth-feed` response instead of keeping separate regional caches. `news.js` builds a refresh request key from region, category, source, and limit; only concurrent requests with the same key reuse the promise, and stale responses from an older region are dropped by token. Source filtering is also region-scoped: when the user moves from Asia Pacific to Europe or another region, source IDs saved for the old region must not be appended to the next fetch. After the new payload arrives, the saved source list is intersected with the available `sources`; if the intersection is empty, the current region falls back to all available sources. This keeps the ticker, panel, and cruise cards aligned after region switches.
Shortcut configuration is a device-local preference owned by `controls.js`: read, capture, enable/disable, and reset all stay in the Earth frontend. It should not be written to backend user settings and should not affect other browsers. New shortcuts must provide a default key, display label, disabled/enabled state, and reset path instead of being hard-coded only in a keydown handler.
### 4. UI and Status Messages
@@ -102,9 +104,9 @@ Responsibilities:
- Map `rotate_left`, `rotate_right`, `rotate_up`, `rotate_down`, `zoom_in`, `zoom_out`, `focus_prev`, `focus_next`, `layer_prev`, `layer_next`, and `confirm` to the action entry points exposed by `main.js`.
- Parse `skeleton` debug events and dispatch `earth:motion-debug-frame`.
Gesture recognition may run locally in the browser or inside the local Agent, but neither path sends realtime camera frames to the SaaS cloud. `main.js` exposes rotation, zoom, target focus, layer switching, and confirm entry points, plus a `window.__planetEarth.motion` debug entry. The adapter starts only when `?motion=1` is present, browser local storage contains `planet-earth-motion-control-enabled=true`, or Earth settings enable Motion Debug Mode.
Gesture recognition may run locally in the browser or inside the local Agent, but neither path sends realtime camera frames to the SaaS cloud. `main.js` exposes rotation, zoom, target focus, layer switching, and confirm entry points, plus a `window.__planetEarth.motion` debug entry. The adapter starts only when `?motion=1` is present, browser local storage contains `planet-earth-motion-control-enabled=true`, or Earth settings enable Motion Debug Mode. `shared.motionEnabledGestures` stores the user-approved gesture whitelist; the browser filters locally, and Motion Agent mode also synchronizes it through `set_enabled_gestures`.
[motion-debug-panel.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/motion-debug-panel.js) owns the debug panel. It listens for `earth:motion-debug-frame` and draws normalized skeleton joints and bones on a canvas. The Browser Camera provider also emits `earth:motion-debug-video-source` with the local `<video>` element so the panel can show a local preview behind the skeleton; `shared.motionDebugSkeletonOnly` switches the panel back to skeleton-only rendering. `Stop Matching Gestures` dispatches `earth:motion-recognition-pause`, which suppresses gesture execution while video and skeleton drawing continue. Unmatched skeletons are red; matched gestures turn green and display the gesture name. Settings are persisted under `shared.motionDebugEnabled`, `shared.motionProvider`, and `shared.motionDebugSkeletonOnly` in `planet.earth.settings.v2`, and both the switch and provider selector reserve `data-gatekeeper-permission="earth.motion_debug"`.
[motion-debug-panel.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/motion-debug-panel.js) owns the debug panel. It listens for `earth:motion-debug-frame` and draws normalized skeleton joints and bones on a canvas. The Browser Camera provider also emits `earth:motion-debug-video-source` with the local `<video>` element so the panel can show a local preview behind the skeleton; `shared.motionDebugSkeletonOnly` switches the panel back to skeleton-only rendering. `Stop Matching Gestures` dispatches `earth:motion-recognition-pause`, which suppresses gesture execution while video and skeleton drawing continue. Unmatched skeletons are red; matched gestures turn green and display the gesture name. Settings are persisted under `shared.motionDebugEnabled`, `shared.motionProvider`, `shared.motionDebugSkeletonOnly`, and `shared.motionEnabledGestures` in `planet.earth.settings.v2`; the switch, provider selector, and gesture whitelist reserve `data-gatekeeper-permission="earth.motion_debug"`.
The Browser Camera provider's gesture pipeline lives in `recognizeGesture()` inside [motion-browser-provider.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/motion-browser-provider.js). Detectors are evaluated in this order, first match wins:
@@ -179,9 +181,15 @@ The `earth:compute-center-location-saved` reconciliation pipeline is deliberatel
### AIS Vessel Layer
The vessel layer now uses `/api/v1/vessels/snapshot` for the initial viewport snapshot and the `/ws` `vessels` channel for realtime deltas. Snapshot requests must include `bbox`, `zoom`, and a bounded `limit`; the backend defaults to `limit=1000` and caps it at `5000`. WebSocket subscriptions must include the same viewport fields so the server can filter updates per connection.
The vessel layer uses `/api/v1/vessels/snapshot` for a global current-state snapshot. Earth sends the global bbox `[-180,-85.05112878,180,85.05112878]`, the current `zoom`, and `limit=3000` when the layer opens. It does not repeatedly refetch by camera viewport, and globe rotation or zoom does not reconnect a vessel WebSocket subscription.
The legacy `/api/v1/visualization/geo/vessels` route has been removed. Frontend code should fetch a snapshot for the current viewport when the layer opens, then subscribe to `vessels` deltas. After map pan or zoom, reload the snapshot and send a fresh vessels subscription. The backend only falls back to legacy `vessel_position` / `vessel_static` rows when the current raw window is empty; frontend code can detect that state through `diagnostics.legacy_fallback_used`.
The backend snapshot endpoint still requires `bbox`, but the Earth runtime treats it as the global current-state entry point. It reads `vessel_current_state`, returning one latest point per MMSI within the freshness window, and no longer scans or aggregates historical `ais_raw_observations` for the display layer. Raw AIS history remains available for tracks, audit, and situational analysis.
Vessel markers are rendered through `createInteractableLayer()` as batched `THREE.Points`, with `cluster` and `avoidance` explicitly disabled. Dense waterways may overlap. Dragging or inertial rotation skips hover picking; normal hover uses screen-space nearest-point picking. Do not reconnect vessels to dynamic screen clustering or per-frame Points rebuilds, because those interaction costs are what make a 3000-marker layer feel heavy.
If the `/ws` `vessels` channel is used by Earth, it should be a low-frequency reload/dirty hint only. Do not create multiple viewport subscriptions, and do not turn every AIS delta into a full layer rebuild.
The legacy `/api/v1/visualization/geo/vessels` route has been removed. Frontend code should keep using `PATHS.vesselsApi` and can verify the current-state path through `diagnostics.source == "vessel_current_state"`.
The new layer API family is `/api/v1/layers/*`, which separates map rendering payloads from aggregate panel statistics. Layer requests must include `bbox`, `zoom`, and a bounded `limit`; responses include `visible_count`, `returned_count`, and `diagnostics`, where `degraded`, `truncated`, and `limit_clamped` are the frontend signals for fallback UI. Right-side aggregate panels should not sum the layer response. They should read `/api/v1/data-products` or `/api/v1/data-products/{product_id}/status`, because those statistics stay global and do not change with the viewport.
@@ -193,13 +193,13 @@ The land/ocean base is an Earth base-map asset and preloads at startup; the "Bor
| Vessel renderOrder | local `VESSEL_RENDER_ORDER` | `4.4` | Normal marker and interactive overlay |
| Vessel track renderOrder | `VESSEL_RENDER_ORDER - 0.1` | `4.3` | Below vessel markers |
| Vessel point pixel size | local `VESSEL_POINT_SIZE` | `34` | Shared size for normal markers and hover / locked overlays |
| Default vessel render cap | `VESSEL_CONFIG.maxRenderedMarkers` | `0` | `0` means the frontend does not clip by default; positive values send `limit` and clip markers |
| Default vessel render cap | `VESSEL_CONFIG.maxRenderedMarkers` | `3000` | Default `limit` sent to `/api/v1/vessels/snapshot` for the global current-state snapshot; the backend cap remains `5000` |
| Vessel texture canvas size | local `VESSEL_ATLAS_CELL_SIZE` | `128` | Canvas point texture |
| Course bucket count | local `VESSEL_COURSE_BINS` | `32` | Moving vessels are bucketed by COG to reduce draw calls while preserving direction |
| Vessel hover picking throttle | local `VESSEL_HOVER_PICK_INTERVAL_MS` | `100` | `main.js` hover picking |
| Vessel screen hit radius | local `VESSEL_POINTER_RADIUS_PX` | `22` | `main.js` screen-space picking |
AIS vessel markers use batched `THREE.Points`, not one `THREE.Sprite` per vessel. Moving vessels stay triangular, anchored or slow vessels stay circular, and hover / locked states add a same-size glow overlay. Vessel type color and info-card type text must come from the same normalized result: `vessels.js` reads both backend `vessel_type_name` and AIS numeric `vessel_type`, derives the color-driving `type`, then exposes `vessel_type_display` for the info card, hover summary, and search results.
AIS vessel markers use batched `THREE.Points`, not one `THREE.Sprite` per vessel. Moving vessels stay triangular, anchored or slow vessels stay circular, and hover / locked states add a same-size glow overlay. Vessels explicitly disable `cluster` and `avoidance`; dense areas may overlap and must not be reattached to dynamic screen clustering. Vessel type color and info-card type text must come from the same normalized result: `vessels.js` reads both backend `vessel_type_name` and AIS numeric `vessel_type`, derives the color-driving `type`, then exposes `vessel_type_display` for the info card, hover summary, and search results.
## Compute Centers
+89
View File
@@ -80,10 +80,61 @@ Each item includes:
Official data, e-commerce metrics, major platforms, and numeric business signals increase importance. Press releases start with a lower baseline and rise only when they match stronger platform, amount, M&A, or regulatory signals.
Importance levels are fixed: `low` 0–34, `medium` 35–59, `high` 60–79, and `critical` 80–100. Category, importance, and Breaking decisions are centralized in `earth_news_classification.py`. Category keys and tags remain configurable, while importance and Breaking protocol states use shared enums. Databases and APIs continue to store compatible lowercase strings.
## Configuration and Cache
`GET /api/v1/earth/news-sources` returns the default or saved configuration. `PUT /api/v1/earth/news-sources` saves it, increments `cache_version`, and clears the process region cache. `POST /api/v1/earth/news-sources/reset` restores defaults. `POST /api/v1/earth/news-sources/test` tests one RSS/Atom/Aggregated source without writing news items.
## Manual News Content
Manual news is a content-management path, not an RSS source configuration. The Admin entry is `Earth Content -> News Content`; the left rail groups items by RSS source and manual news group. After opening a manual group, administrators can add one item or upload a JSON array. Manual items are written to `earth_news_items` with `feed_type/source_type = manual`; the display label is “手动添加” / “Manual”. They do not participate in RSS connectivity tests and are not routed through RSS fetching.
Manual news uses a publish-first, enrich-later flow:
1. Saving immediately writes the item to `earth_news_items`.
2. If no manual coordinates are provided, the selected region anchor is used and `verified=false`.
3. If manual coordinates are provided, the item uses `location_source=manual_location` and `verified=true`; later AI enrichment does not overwrite that location.
4. Create and reprocess actions enqueue the item for cleanup, translation, classification, importance, Breaking, and target-location inference.
5. When enrichment finishes, the same row is updated and Earth receives a news reload / patch so the frontend replaces the item without a manual refresh.
The Admin API is under `/api/v1/earth/news-items`:
- `GET /earth/news-groups`: return RSS virtual source groups and manual news groups.
- `POST /earth/news-groups`: create a manual news group.
- `PUT /earth/news-groups/{group_id}`: rename a manual news group and synchronize metadata for items in that group.
- `GET /earth/news-items`: paginated RSS and manual news list, with filters for source type, region, category, and status.
- `POST /earth/news-items`: create one manual news item.
- `POST /earth/news-items/import`: upload a JSON array; `group_id` selects the current manual news group.
- `PUT /earth/news-items/{id}`: edit a manual news item; RSS items are read-only.
- `DELETE /earth/news-items/{id}`: delete a manual news item and trigger an Earth news reload.
- `POST /earth/news-items/{id}/reprocess`: requeue cleanup, translation, and geolocation.
The first JSON import format supports arrays only:
```json
[
{
"title": "Required title",
"summary": "Optional summary",
"content": "Optional body",
"url": "https://example.com/story",
"source": "Manual",
"region": "global",
"published_at": "2026-05-15T03:00:00Z",
"category": "business",
"tags": ["manual", "analysis"],
"location": {
"label": "Beijing, China",
"latitude": 39.9057,
"longitude": 116.3913
}
}
]
```
Imports deduplicate through a stable `manual:{hash}` ID derived from title, published time, URL, and source. Importing the same manual item again updates the existing row instead of creating a duplicate EarthFeed entry.
## Feed Query and Category Filtering
The Web Earth client and UE client both consume `GET /api/v1/news/earth-feed`. The endpoint supports server-side filtering, so clients do not need to fetch the full list and apply the primary category filter locally.
@@ -105,8 +156,44 @@ Unknown category or locale values return `422` with the allowed values. The resp
The Web Earth category chips only store the current browser preference; changing them triggers a new API request. UE should pass its selected categories through the `categories` query parameter and does not need to perform the primary filtering itself.
When `sources` is omitted, the service layer prefers stories that already have displayable title and summary content for the requested `locale`, supplements candidates from enabled sources, and rotates sources so one source's newest pending items cannot occupy all 12 default slots. An explicit `sources` filter remains precise and does not supplement other sources. The database query layer owns region, category, source, and ordering constraints only; it does not own locale presentation policy.
Source testing only proves that a specific RSS/Atom/XML feed can be parsed. It does not mean those items have already been written to the news table or are visible in the current region/category view. Saving or resetting news sources increments the configuration version and clears cache; if an enabled feed has no recent stored items, the next `earth-feed` request supplements from RSS so newly enabled sources such as 36Kr and Ebrun are not masked by fresh Google News rows.
## Regional Balancing and Async Enrichment
`earth_news_items` is the current-state table for EarthFeed. When no explicit `sources` filter is present, the endpoint sorts by Breaking state, region, and publish time, then applies a regional round-robin so Asia Pacific or any other high-volume source cannot fill the entire global view and cruise queue. The default region order is Americas, Europe, Middle East / Africa, Asia Pacific, then Global; unknown regions participate after the known regions. Regional views still follow the active-region-plus-global rule and do not mix unrelated regions into the regional panel.
Both `items` and `cruise_items` are enqueued for target-location and localization enrichment, but the frontend must not wait for AI before rendering. If the requested display locale is not ready yet, Web Earth falls back to the original `title / summary` so a card does not show a "translation pending" placeholder when readable source content already exists. After translation, classification, Breaking, or target-location enrichment completes, the same `earth_news_items` row is updated and Earth receives a news reload / patch.
Target-location enrichment uses two Redis Streams queues:
- `earth_news:target_location:priority`: priority jobs for currently visible `items` and `cruise_items`, with a short dedupe TTL.
- `earth_news:target_location:jobs`: normal background enrichment jobs, with a longer dedupe TTL.
The worker always drains the priority queue before the regular queue; stale pending messages are reclaimed after the idle threshold, each AI job has a hard timeout, and failures go through retry or dead-letter handling. This prevents a large historical regular backlog from starving the Europe, Americas, or other regional news currently visible to the user.
## Breaking News Insertion
The news system keeps three separate decisions:
- **Category**: what the story is about, such as business, military, or disaster.
- **Importance**: whether the story has long-term value, stored as `importance_score / importance_level`.
- **Breaking**: whether the story must temporarily jump ahead, stored in `location_meta.news_meta.breaking_*`.
Breaking metadata does not add physical columns. It remains in `location_meta.news_meta`:
- `breaking_level`: `none / watch / breaking / critical`.
- `breaking_scope`: `regional / global`.
- `breaking_reasons`: rule or operator reasons.
- `breaking_source`: `rules / ai / manual / multi_source`.
- `breaking_confidence`: 0 to 1.
- `breaking_expires_at`: expiration timestamp.
The backend performs the ordering, so Web and UE clients do not need to reorder items. Active `critical`, `breaking`, and `watch` items appear before normal items in that order. Once expired, the item falls back to normal ordering without being deleted or changing its long-term importance score.
`breaking_scope = global` bypasses the active region and can appear in every regional feed. `regional` breaking follows the normal active-region-plus-global rule. The response `filters` includes `has_breaking` and `highest_breaking_level`, which the Earth client uses to apply restrained panel and card styling.
## Connectivity Monitoring
`POST /api/v1/earth/news-sources/test` tests one source and writes the result to `earth_news_sources.health[source_id]`. Normal RSS/Atom fetches update the same health map.
@@ -140,9 +227,11 @@ Reference links show that they only record a homepage, report page, or future co
```mermaid
flowchart LR
Admin["Admin: Earth Content / News Sources"] --> Source["Source config"]
ManualAdmin["Admin: Earth Content / News Content"] --> ManualAPI["/api/v1/earth/news-items"]
Source --> Feed["Feed children"]
Feed --> ConfigAPI["/api/v1/earth/news-sources"]
ConfigAPI --> Config["SystemSetting: earth_news_sources"]
ManualAPI --> Store
Earth["Earth News Panel"] --> NewsAPI["/api/v1/news/earth-feed"]
NewsAPI --> Resolver["Source Resolver"]
@@ -27,8 +27,10 @@ Note: the layer control panel order and the registration / startup load order ar
| 2.29 | Country border hover glow | `country-boundaries.js` | Hover radius + glow offset | `depthTest: false`, raycast disabled | Additive glow to reinforce border edge and terrain hover visibility. |
| 2.3 | Country border hover line | `country-boundaries.js` | `hoverAltitudeOffset = EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET = 0.48` | `depthTest: false`, raycast disabled | Neon red-orange hover line; aligned with the normal borders and HD texture shell to avoid ghosting or floating; China and Taiwan share the same highlight group. |
| 3 | Satellite footprint fill / Iridium coverage ring | `satellites.js`, `iridium-footprint-adapter.js` | `GROUND_FOOTPRINT_RENDER_ORDER` | depth-tested; Iridium adapter fill / ring use the same renderOrder | Footprint above land / texture / terrain and country borders, below compute centers and satellites. |
| 3-5 | BGP markers and overlays | `bgp.js` | Each marker's own renderOrder | BGP picking path | Preserves existing BGP visual level. |
| 4.5 | Compute centers | `compute-centers.js` | `COMPUTE_CENTER_RENDER_ORDER` | Compute center picking path | Surface facilities, below satellites. |
| 3-4.5 | BGP collectors, event rings, and event markers | `bgp.js`, `interactable.js` | Collector markers use `BGP_COLLECTOR_RENDER_ORDER = 4.4`; event markers use `BGP_EVENT_RENDER_ORDER = 4.5`; overlays remain in BGP-owned groups | Screen-space Interactable picking; BGP collectors/events may use collision avoidance where appropriate | BGP collectors share the surface-facility band with vessels; BGP events share the compute-center band. |
| 4.3 | AIS vessel track lines | `vessels.js` | `VESSEL_RENDER_ORDER - 0.1`; `CONFIG.earthRadius + VESSEL_CONFIG.track.altitudeOffset` | Follows vessel selection, not independently picked | Recent track for the selected vessel, below vessel markers. |
| 4.4 | AIS vessel markers | `vessels.js`, `interactable.js` | `VESSEL_RENDER_ORDER`; business radius `CONFIG.earthRadius + VESSEL_CONFIG.altitudeOffset`; normal markers are bucketed `THREE.Points`, hover / locked states are single-point overlays | `depthTest: true`; `main.js` uses screen-space picking for front-facing markers; `cluster: false` and `avoidance: false` | Moving vessels use triangular point textures, anchored / slow vessels use dots. Dense waterways may overlap and do not participate in dynamic screen clustering, avoiding Points rebuilds during rotation. |
| 4.5 | Compute centers | `compute-centers.js`, `interactable.js` | `COMPUTE_CENTER_RENDER_ORDER` | Screen-space Interactable picking, with surface-icon collision handling | Surface facilities, below satellites. |
| 5 | Satellite background dot | `satellites.js` | Fixed renderOrder; by default TLE/SGP4 altitude is compressed to `CONFIG.earthRadius + 4..25`; with real altitude disabled or propagation failed, uses `fallbackAltitudeOffset = 8` | Screen-space satellite picking | Below satellite dots. |
| 6 | Satellite dots | `satellites.js` | Same compressed / fallback height as satellite backdrop dots | Screen-space satellite picking | Satellite dots above footprints and compute centers. |
| 12+ | Satellite locked ring, halo, predicted orbit | `satellites.js` | `SATELLITE_CONFIG.overlayRenderOrder` and offsets; predicted orbit follows the same real-altitude toggle and fixes the lock-time globe pose to draw a closed inertial orbit; returns to same-sphere mode when real altitude is disabled | Satellite overlay path | Used for selected/locked satellite emphasis. |
+4 -1
View File
@@ -194,6 +194,7 @@ TV livestreams and boundary precision moved to `/earth-content`; collectors and
- **Brand Assets**: manages the logo, title image, title text, subtitle, and description used by the Earth HUD. Uploaded images are saved as Earth brand assets and read by the Earth page immediately.
- **About**: manages the About card shown in Earth settings, including logo, kicker, title, version, description, and metadata.
- **TV Livestream**: manages sources shown in the Earth media panel.
- **News Content**: browses news grouped by RSS source and manual group. RSS items remain read-only; manual groups support create, JSON import, edit, delete, and reprocess.
- **Boundary Precision**: shows the current provider, low-precision fallback, high-precision PMTiles/manifest status, local source JSON, and manual build action.
- **Base Map**, **Layer Resources**, **3D Assets**, and **News Anchor Strategy**: placeholder tabs for future configuration. They do not display fake data.
@@ -298,7 +299,7 @@ Adopt All is for batch processing the compute-center unresolved queue. It starts
### Settings
The settings panel is grouped into Runtime, Display, Panels, Motion, Shortcuts, and System. It covers rotate / cruise / motion mode, cruise modules (BGP/news/compute centers/vessels/cables/satellites), view (satellite display style, hover tooltip, satellite idle breathing, real satellite altitude, track display, compact dots, day-night mode, panel toggles), motion debug mode / input source / skeleton-only, shortcut enablement and remapping, default globe size, terrain opacity, reset.
The settings panel is grouped into Runtime, Display, Panels, Motion, Shortcuts, and System. It covers rotate / cruise / motion mode, cruise modules (BGP/news/compute centers/vessels/cables/satellites), view (satellite display style, hover tooltip, satellite idle breathing, real satellite altitude, track display, compact dots, day-night mode, panel toggles), motion debug mode / input source / skeleton-only / recognized-gesture whitelist, shortcut enablement and remapping, default globe size, terrain opacity, reset.
News categories use the same chip selector as Cruise Modules. They only filter the news panel and news cruise items in the current browser; they do not affect layers, TV, data points, basemap, boundaries, collector jobs, or admin news-source configuration.
@@ -336,6 +337,8 @@ Enable via the settings toggle "Motion Debug Mode", or with URL parameter `?moti
Neither mode uploads camera frames or live gestures; neither reuses the news/RSS aggregation API.
Recognized Gestures can disable rotation, zoom, focus switching, layer switching, or confirmation independently. The browser ignores unchecked actions; when Motion Agent is active, the same whitelist is synchronized through the control protocol.
Gesture semantics:
| Event | Effect |
+44 -13
View File
@@ -244,22 +244,27 @@ The recommended direction is a small platform compatibility layer for port liste
The production frontend shape is `vite build` static output served by nginx/Caddy or an equivalent HTTP server. Do not use `bun run dev` or `vite preview` in production. The project does not maintain a parallel Webpack build chain; if a future enterprise requirement needs closer Webpack-ecosystem compatibility, run an Rsbuild/Rspack spike first. Electron should only be evaluated when the official target becomes an offline desktop application.
## Optional Motion Agent Startup
## Default Motion Agent Startup
`planet.sh` can now manage the local Motion Capture Agent. It is disabled by default so ordinary development machines do not fail startup when cameras, OpenCV, or MediaPipe are unavailable.
`planet.sh` now starts the local Motion Agent by default during `start` and full `restart`. This makes the Earth page, UE clients, and debug clients able to connect to `ws://127.0.0.1:8765/ws/gestures` immediately. If the machine has no usable camera, implicit default startup falls back to dry-run protocol mode and does not block backend/frontend startup. Explicit Motion Agent startup through `--motion-agent`, camera indexes, camera URLs, or WSL USB options still treats live camera failures as real errors.
Start it with:
To skip Motion Agent for this run:
```bash
./planet.sh start --motion-agent
./planet.sh start --non-motion-agent
./planet.sh restart --non-motion-agent
```
Common options:
- `--motion-agent` / `-m`: start or restart the Motion Agent for this command.
- `--non-motion-agent`: do not start Motion Agent for this `start` or full `restart`.
- `--motion-agent` / `-m`: explicitly start or restart the Motion Agent for this command; live camera failures are reported as failures.
- `--motion-agent-port <port>`: override the default WebSocket port `8765`.
- `--motion-agent-mode <mode>`: choose `auto`, `single`, `dual_redundant`, `single_fallback`, or `calibrated_3d`; `dual` is kept as a compatibility alias for redundant dual-camera mode.
- `--motion-agent-camera-indexes <indexes>`: override auto-detected camera indexes, for example `0` or `0,1`. The same can be provided through `MOTION_AGENT_CAMERA_INDEXES=0,1`.
- `--motion-agent-camera-urls <urls>`: use RTSP/HTTP camera streams, useful for WSL, phone cameras, or network cameras. The same can be provided through `MOTION_AGENT_CAMERA_URLS=...`.
- `--motion-agent-wsl-usbipd`: in WSL, try to attach the single detected Windows USB camera through `usbipd-win`.
- `--motion-agent-wsl-usbipd-busid <BUSID>`: in WSL, attach the camera matching a `usbipd list` BUSID; use this when multiple cameras are present.
- `--motion-agent-dry-run`: start only the protocol service without opening cameras or loading CV dependencies; useful for Web client debugging.
Non-dry-run live mode checks `mediapipe` and `opencv-python` before startup. If the current `.venv` is missing them, the script automatically runs:
@@ -268,13 +273,17 @@ Non-dry-run live mode checks `mediapipe` and `opencv-python` before startup. If
uv add mediapipe opencv-python
```
To disable startup-time auto-install:
To disable startup-time Python CV dependency auto-install:
```bash
PLANET_MOTION_AGENT_AUTO_INSTALL=0 ./planet.sh start --motion-agent
PLANET_MOTION_AGENT_AUTO_INSTALL=0 ./planet.sh start
```
Live mode auto-detects `/dev/video*` and passes the first two indexes to the Motion Agent. In WSL, Windows cameras usually do not appear as `/dev/video*` automatically. Check available devices first:
`./planet.sh init` also performs a WSL host dependency preflight for `usbipd-win`. If `usbipd.exe` is missing, the script first tries `winget install -e --id dorssel.usbipd-win`, then reuses the repository-bundled dorssel.usbipd-win MSI fallback. If that cached MSI is missing or an architecture-specific MSI is needed, it downloads one and requests an Administrator PowerShell installation. This is best-effort: failure prints next steps but does not block normal initialization. Use `./planet.sh init --non-motion-agent` to skip this preflight.
Live mode auto-detects `/dev/video*`, then prefers an OpenCV probe to keep only indexes that can open and return frames before passing them to the Motion Agent. In WSL/USB camera setups, one camera can expose multiple `/dev/video*` nodes, and some of them are metadata or non-capture nodes; the script skips those unreadable indexes. In WSL, Windows cameras usually do not appear as `/dev/video*` automatically. Check available devices first:
Live capture defaults to low-latency settings: `640x360` input and roughly `15Hz` recognition events. The worker uses latest-frame reader threads and keeps only the newest frame from each camera, so a slow MediaPipe frame does not make the recognizer drain stale camera backlog. The skeleton debug stream is disabled by default and is only sent at roughly `8Hz` while the Earth motion debug panel is open, so normal gesture control is not slowed down by debug data. Status events report both capture FPS and recognition FPS to separate camera throughput issues from recognition cost.
```bash
ls /dev/video*
@@ -292,20 +301,42 @@ In WSL, the more general path is to connect a phone or network camera through an
./planet.sh start --motion-agent --motion-agent-camera-urls http://192.168.1.20:8080/video
```
If WSL has no `/dev/video*` and no `--motion-agent-camera-urls` is provided, live startup stops and prints guidance instead of silently falling back to dry-run. Choose one of:
To use a Windows USB camera directly from WSL, let the script call `usbipd-win`. This is opt-in because an attached camera is usually temporarily unavailable to Windows apps while WSL owns it.
When there is only one camera:
```bash
./planet.sh start --motion-agent --motion-agent-wsl-usbipd
```
When there are multiple cameras, inspect the BUSID first and pass it explicitly:
```bash
usbipd.exe list
./planet.sh start --motion-agent --motion-agent-wsl-usbipd-busid 3-2
```
If `usbipd attach` says the device is not shared or bound, the script tries to open an Administrator PowerShell to run `usbipd bind`, then retries attach. If UAC is canceled or automatic bind fails, run this manually from an Administrator PowerShell:
```powershell
usbipd bind --busid 3-2
usbipd attach --wsl --busid 3-2
```
If WSL has no `/dev/video*` and no `--motion-agent-camera-urls` is provided, implicit default startup falls back to dry-run. Explicit live startup stops and prints guidance. Choose one of:
```bash
./planet.sh start --motion-agent --motion-agent-camera-urls http://<phone-ip>:8080/video
./planet.sh start --motion-agent --motion-agent-wsl-usbipd
./planet.sh start --motion-agent --motion-agent-dry-run
```
Automatic dry-run fallback only happens when `PLANET_MOTION_AGENT_WSL_ALLOW_DRY_RUN_FALLBACK=1` is explicitly set.
For explicit live startup, automatic dry-run fallback only happens when `PLANET_MOTION_AGENT_WSL_ALLOW_DRY_RUN_FALLBACK=1` is set.
Environment-variable startup is also supported:
`--non-motion-agent` is the command-level opt-out. Environment variables can still tune how the service starts:
```bash
PLANET_START_MOTION_AGENT=1 ./planet.sh start
MOTION_AGENT_DRY_RUN=1 PLANET_START_MOTION_AGENT=1 ./planet.sh start
MOTION_AGENT_DRY_RUN=1 ./planet.sh start
```
Logs:
+9 -9
View File
@@ -39,7 +39,7 @@ flowchart TB
VesselRaw["AIS / BarentsWatch"] --> VesselDerived["vessel_static / vessel_position"]
VesselDerived --> VesselLayer["vessels layer"]
Interactables["earth_interactables"] --> InteractableLayer["interactables layer"]
NewsRaw["RSS / Live / News"] --> NewsItems["earth_news_items"]
NewsRaw["RSS / Manual News / Live"] --> NewsItems["earth_news_items"]
NewsItems --> NewsLayer["news / media layer"]
```
@@ -51,7 +51,7 @@ flowchart TB
| BGP context | Collectors, anomalies, incidents, route events, and regional context | `ris_live_bgp`, `bgpstream_bgp`, prefix geography sources | `bgp_observations`, `bgp_anomalies`, `bgp_incidents`, `bgp_collector_locations` | `bgp` | `clear_then_reload` |
| Vessels | AIS vessels, positions, tracks, legend, and source health | AIS sources, `barentswatch_vessels` | `vessel_static`, `vessel_position`, `ais_raw_observations`, `ais_source_health` | `vessels` | `clear_then_reload` |
| Interactables | Generic surface icons, manual objects, and future small layers | `earth_interactables` | None | `interactables` | `delta` |
| News and media | Earth news, live streams, cruise summaries, and situation content | news sources | `earth_news_items` | `news` / `media` | `reload` |
| News and media | Earth news, live streams, cruise summaries, and situation content | RSS news sources, manual news, live streams | `earth_news_items` | `news` / `media` | `reload` |
## Satellites
@@ -118,12 +118,12 @@ sequenceDiagram
## Vessels
Vessel data shows AIS vessels, navigation state, vessel-type legend, and source health. Earth rendering uses position snapshots and static vessel information; it should not render every raw AIS observation.
Vessel data shows AIS vessels, navigation state, vessel-type legend, and source health. Earth rendering uses the `vessel_current_state` current-state snapshot; it should not render every raw AIS observation.
- **Collection entry**: AIS sources, BarentsWatch vessels.
- **Fact table**: `collected_data` or AIS raw observation tables.
- **Derived tables**: `vessel_static`, `vessel_position`, `ais_raw_observations`, `ais_source_health`.
- **API**: vessel visualization API returns current vessel markers and detail fields.
- **Derived tables**: `vessel_current_state`, `vessel_static`, `vessel_position`, `ais_raw_observations`, `ais_source_health`.
- **API**: `/api/v1/vessels/snapshot` returns current vessel markers and detail fields; the Earth frontend uses a global bbox for the global current-state view.
- **Delete semantics**: deleting any vessel source lets owned projections broadcast `vessels` with `clear_then_reload`.
- **Common failure**: the count panel changes but vessels remain. Summary and layer data are separate; Earth should clear objects based on layer updates.
@@ -141,12 +141,12 @@ Vessel data shows AIS vessels, navigation state, vessel-type legend, and source
News and media support the Earth news ticker, live stream panel, news cruise, and situation summaries. They are content refresh paths rather than stable geographic object layers, so they default to `reload`.
- **Collection entry**: RSS, live streams, news sources.
- **Fact table**: news source rows in `collected_data`.
- **Collection entry**: RSS news sources, manual news from `Earth Content -> News Content`, and live streams.
- **Fact table**: news source rows in `collected_data`; manual news writes directly to `earth_news_items` and marks the content source with `feed_type/source_type=manual`.
- **Derived table**: `earth_news_items`.
- **API**: news, live stream, and media content APIs.
- **API**: `/api/v1/news/earth-feed` reads `earth_news_items`; the Admin API `/api/v1/earth/news-items` supports manual create, JSON import, edit, delete, and reprocess.
- **Delete semantics**: deleting news sources or `earth_news_items` broadcasts `news` / `media` reload; empty responses hide the corresponding content.
- **Common failure**: the live panel shows stale content. Usually the media component ignored the layer update or the content API cache was not invalidated.
- **Common failure**: a newly saved manual item may initially show source text or a region anchor; this is the normal publish-first enrichment window. If it never updates, check the `earth_news_enrichment` queue, AI / Web Search configuration, and `enrichment_status`. If the live panel shows stale content, the media component likely ignored the layer update or the content API cache was not invalidated.
## Adding a New Layer
+1 -1
View File
@@ -50,7 +50,7 @@ Once in, verify:
- Search finds cables, satellites, compute centers, BGP events
- Compute-center and BGP collector detail cards can collect coordinate candidates and preview them on Earth
- Mouse drag, wheel zoom, and the zoom percentage indicator work
- The settings panel can switch rotate / cruise / motion modes; view settings can switch hover tooltip content, and satellite settings can toggle real-altitude layering and track display
- The settings panel can switch rotate / cruise / motion modes; motion settings can select the input source and allowed gestures; view settings can switch hover tooltip content, and satellite settings can toggle real-altitude layering and track display
## 5. Recover a Lost Password
+1
View File
@@ -38,6 +38,7 @@
- [数据源、采集器设置与连接验证](/home/ray/dev/linkong/planet/docs/technical/zh/datasource-collector-settings-connectivity.md):数据源目录、采集器设置、连接验证和凭证链路
- [数据源 API 性能](/home/ray/dev/linkong/planet/docs/technical/zh/backend-datasources-api-performance.md):DataSources 列表接口性能和缓存策略
- [数据作业与 Outbox 技术架构](/home/ray/dev/linkong/planet/docs/technical/zh/data-job-earth-sync-architecture.md):PostgreSQL 作业队列、outbox、listener 和 Kafka / Spark 演进边界
- [后端枚举与字符串兼容契约](/home/ray/dev/linkong/planet/docs/technical/zh/backend-enum-contracts.md):稳定协议状态、历史字符串兼容和 Earth 新闻判定边界
- [通用位置估算管线开发说明](/home/ray/dev/linkong/planet/docs/technical/zh/location-pipeline-development.md):location resolver / pipeline 的接口、注册表和扩展方式
- [新闻直播采集格式](/home/ray/dev/linkong/planet/docs/technical/zh/earth-news-live-streams-collector-format.md):新闻、直播和媒体采集 payload 约定
- [Docs Gatekeeper 开发说明](/home/ray/dev/linkong/planet/docs/technical/zh/docs-gatekeeper-development.md):后端文档目录、正文读取和 Gatekeeper 权限组实现
+7 -19
View File
@@ -94,7 +94,9 @@ CelesTrak TLE 采集优先拉取完整 `active` 目录。如果 CelesTrak 返回
| BarentsWatch AIS | vessel | 船只位置、航速、航向、MMSI 等 AIS 数据 | 依采集器配置 |
| AISStream Vessels | vessel_ais | AIS WebSocket 实时流,写入原始观测层并由聚合接口展示 | 依采集器配置 |
AIS 船只类采集器和其它 `CollectedData` 采集器的落库路径不同。BarentsWatch、AISStream 和自定义 `vessel_ais` 源都会进入 AIS 原始观测层,随后由聚合服务合并成 Earth 船只图层使用的 GeoJSON 和详情数据。这样做可以保留来源、传输方式、字段冲突和观测时间,避免某个实时源直接覆盖最终展示表。
AIS 船只类采集器和其它 `CollectedData` 采集器的落库路径不同。BarentsWatch、AISStream 和自定义 `vessel_ais` 源都会进入 AIS 原始观测层,同时 upsert `vessel_current_state` 当前状态表:每个 MMSI 只保留一行最新位置、航速、航向、状态、船型、名称和来源元数据。这样既保留原始观测历史用于轨迹、审计和态势分析,又让 Earth 船只图层读取当前状态表,不在展示接口里扫描历史 AIS 记录。
`vessel_current_state` 的动态位置只允许被更新观测时间更晚的数据覆盖;名称、船型等静态字段按非空和来源优先策略合并。Earth snapshot 默认只返回有效窗口内的当前船只,避免高频 AIS 历史数据影响地球渲染性能。
智能星球国界不再属于采集器体系。它是智能星球静态渲染资产,由控制台 `运维与配置 -> 智能星球内容 -> 国界精度` 维护源配置,并由 `/api/v1/earth/boundaries/*` 构建 `frontend/public/earth/data/boundaries/earth-boundaries-china-pov-v1.pmtiles`。本地没有高精 PMTiles 时,前端会使用仓库内置的低精度 GeoJSON 作为 fallback,不会向 `CollectedData` 写入国界记录。
@@ -369,32 +371,18 @@ AIS 观测写入后不会直接替换最终船只记录,而是先保存为 raw
- 位置、速度、航向等动态字段会按 freshness 和来源优先级选择。
- 静态字段优先保留非空值;冲突候选会记录到详情接口,便于排查多源差异。
Earth 船只展示现在使用受控快照接口和实时增量通道:
Earth 船只展示现在使用当前状态快照接口:
```http
GET /api/v1/vessels/snapshot?bbox=lon_min,lat_min,lon_max,lat_max&zoom=12&limit=1000
GET /api/v1/vessels/snapshot?bbox=-180,-85.05112878,180,85.05112878&zoom=12&limit=3000
GET /api/v1/visualization/vessels/{mmsi}
GET /api/v1/visualization/vessels/{mmsi}/track
GET /api/v1/visualization/vessels/{mmsi}/conflicts
```
`/api/v1/vessels/snapshot` 必须携带 `bbox` 和 `zoom`,默认 `limit=1000`,最大 `limit=5000`。它优先消费 `ais_raw_observations` 聚合结果;当当前 raw 窗口为空时,会受控回退到 legacy `vessel_position` / `vessel_static` 最新点,并在 `diagnostics.legacy_fallback_used` 中标明。旧 `/api/v1/visualization/geo/vessels` 路由已移除。
`/api/v1/vessels/snapshot` 必须携带 `bbox` 和 `zoom`,后端最大 `limit=5000`。Earth 前端使用全球 bbox 读取当前状态,不随相机视口变化反复请求。接口消费 `vessel_current_state`,并在 `diagnostics.source` 返回 `vessel_current_state`;旧 `/api/v1/visualization/geo/vessels` 路由已移除。
实时增量通过 `/ws` 的 `vessels` channel 推送。客户端订阅时必须带当前视口:
```json
{
"type": "subscribe",
"data": {
"channel": "vessels",
"bbox": [120.8, 30.7, 122.1, 31.8],
"zoom": 12,
"limit": 1000
}
}
```
后端按连接保存轻量订阅条件,只向 bbox 命中的连接发送船只更新。collector 广播会先进入 1 秒节流队列,同一 MMSI 在一个 flush 周期内只保留最新位置,避免高频实时流拖垮 WebSocket。
高频 AIS 更新不要直接推送成每条 delta 的整层重建。`/ws` 的 `vessels` channel 如用于 Earth,应广播低频 reload/dirty 提示,由前端合并刷新 snapshot;轨迹和冲突详情仍按单船接口读取历史事实。
### 图层接口与全量统计分离
@@ -0,0 +1,50 @@
# 后端枚举与字符串兼容契约
Planet 后端使用 `backend/app/core/enums.py` 统一维护有限、稳定、会参与协议判断的状态值。数据库和 API 仍保存、输出小写字符串;枚举用于代码内部的类型安全、校验和去重,不要求数据库迁移为 SQL Enum。
## 使用准则
适合枚举的值必须有限且稳定,非法值应被拒绝或安全回退,并且多个模块会比较、排序或分支处理该值。典型示例包括任务状态、AI Playground 消息角色和状态、用户角色、告警状态、日志级别、新闻重要度和 Breaking 状态。
以下值必须保持可配置字符串:
- 新闻分类与标签。
- Provider、模型、数据源、collector、新闻源和 Feed 标识。
- 可扩展的 incident/anomaly 类型。
- 用户输入和自由文本。
## 边界转换
服务内部优先使用 `StrEnum`。写入数据库或输出外部协议时使用 `.value`,继续得到现有字符串,例如 `JobStatus.RUNNING.value == "running"`。
读取历史数据库、JSON 或外部输入时使用:
```python
status = parse_enum(JobStatus, raw_status, JobStatus.FAILED)
```
合法历史字符串会归一为枚举;空值使用明确默认值;未知值记录 warning 并安全回退,不阻断历史数据读取。Pydantic 请求字段可以直接使用枚举,让非法协议值返回 `422`。普通 String/JSON 数据库列不改成 SQLAlchemy Enum。
## Earth 新闻判定
Earth 新闻判定集中在 `backend/app/services/earth_news_classification.py`:
- 分类回答“新闻是什么”,分类 key 仍是可配置字符串。
- 重要度回答“长期是否值得关注”。
- Breaking 回答“短时间内是否必须插队”。
重要度等级固定为:
| 等级 | 分数 |
|---|---:|
| `low` | 0–34 |
| `medium` | 35–59 |
| `high` | 60–79 |
| `critical` | 80–100 |
Breaking 规则使用带类型的 `BreakingRule`;等级、范围、来源和 TTL 由公开分类模块统一管理。`earth_news.py` 只负责抓取、解析、编排与序列化。
## 防回退检查
新增或修改协议状态时,先检查 `app/core/enums.py`,不要在业务模块重复定义 `Literal`、状态集合或 normalize helper。枚举值与边界行为应补充到 `tests/test_enum_contracts.py`,并保证 API 字符串和数据库表示不变。
@@ -320,10 +320,10 @@ AISStream 使用 WebSocket 实时流,采集器只写入 `ais_raw_observations`
新版本不再使用 legacy `/api/v1/visualization/geo/vessels` 作为船只列表入口。Earth 初始状态应调用:
```http
GET /api/v1/vessels/snapshot?bbox=lon_min,lat_min,lon_max,lat_max&zoom=12&limit=1000
GET /api/v1/vessels/snapshot?bbox=-180,-85.05112878,180,85.05112878&zoom=12&limit=3000
```
该接口优先查询本地 `ais_raw_observations` 聚合结果;当当前 raw 窗口为空时,会受控回退到 legacy `vessel_position` / `vessel_static` 最新点,并通过 `diagnostics.legacy_fallback_used` 暴露。实时更新走 `/ws` 的 `vessels` channel,订阅时必须提供 `bbox`、`zoom` 和 `limit`。服务端按连接过滤 bbox,并对 collector 广播做 1 秒合并,同一 MMSI 只推送最新位置。
该接口查询 `vessel_current_state` 当前状态表,只返回有效窗口内每个 MMSI 的最新点;原始 `ais_raw_observations` 继续保留给轨迹、审计和态势分析,但不再由展示接口临时扫描聚合。Earth 前端统一传全球 bbox,不随当前镜头视口反复请求。实时更新如接入 `/ws` 的 `vessels` channel,应作为 reload/dirty 提示触发合并刷新,不能把每条 AIS delta 直接变成整层重建。
## 自定义 REST / WebSocket 映射运行时
+9 -6
View File
@@ -77,6 +77,8 @@ Earth 设置面板现在按 `data-settings-tab` 和 `data-settings-tab-panel`
`显示` 分类里的新闻类型选择复用巡航模块的 chip 选择器形态,只控制星球端当前浏览器的新闻分类显示。它不会打开或关闭图层、底图、边界、TV、数据点、BGP、船舶、卫星或算力中心;这些仍由图层面板、媒体面板和控制台配置各自负责。`controls.js` 只持久化 `shared.newsCategoryFilters` 并广播 `earth:news-category-filters-change`,`news.js` 会把选中的类型拼到 `/api/v1/news/earth-feed?categories=...&locale=zh-CN`,让 Web 和 UE 走同一套后端类型过滤。
新闻面板、滚动条和新闻巡航必须消费同一次 `/api/v1/news/earth-feed` 响应里的 `items` / `cruise_items`,不能各自缓存区域状态。`news.js` 的刷新请求以区域、类型、来源和数量生成 request key;只有 key 相同的并发请求才复用 promise,旧区域请求返回时会被 token 丢弃。来源过滤也必须按区域作用域处理:当用户从亚太切到欧洲等其它区域时,旧区域保存的来源 ID 不允许继续拼到下一次 fetch 里;拿到新 payload 后再与 `sources` 列表做交集,若没有交集则回退到当前区域所有可用来源。这样滚动条、面板和巡航才会在区域切换后展示同一批新闻。
快捷键配置属于设备本地偏好,由 `controls.js` 负责读取、捕获、启用/禁用和重置。它不应写入后端用户设置,也不应影响其它浏览器。后续新增快捷键时,必须同时提供默认键、显示标签、可禁用状态和重置路径,避免只在 keydown handler 中硬编码。
### 4. UI 与状态消息
@@ -109,9 +111,9 @@ Earth 设置面板现在按 `data-settings-tab` 和 `data-settings-tab-panel`
- 将 `rotate_left`、`rotate_right`、`rotate_up`、`rotate_down`、`zoom_in`、`zoom_out`、`focus_prev`、`focus_next`、`layer_prev`、`layer_next`、`confirm` 映射到 `main.js` 暴露的动作入口。
- 解析 `skeleton` 调试事件并派发 `earth:motion-debug-frame`。
动作捕捉识别可以在浏览器本地执行,也可以在本地 Agent 中执行,但两者都不会把实时视频帧发给 SaaS 云端。`main.js` 暴露旋转、缩放、目标切换、图层切换和确认入口,并通过 `window.__planetEarth.motion` 提供调试入口。默认只有 URL 参数 `?motion=1`、本地存储 `planet-earth-motion-control-enabled=true`,或 Earth 设置中的“动捕调试模式”打开时才启动当前 provider。
动作捕捉识别可以在浏览器本地执行,也可以在本地 Agent 中执行,但两者都不会把实时视频帧发给 SaaS 云端。`main.js` 暴露旋转、缩放、目标切换、图层切换和确认入口,并通过 `window.__planetEarth.motion` 提供调试入口。默认只有 URL 参数 `?motion=1`、本地存储 `planet-earth-motion-control-enabled=true`,或 Earth 设置中的“动捕调试模式”打开时才启动当前 provider。`shared.motionEnabledGestures` 保存用户允许识别的动作;浏览器端先过滤,Motion Agent 模式还会通过 `set_enabled_gestures` 同步给服务端。
动捕调试面板由 [motion-debug-panel.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/motion-debug-panel.js) 负责。它监听 `earth:motion-debug-frame`,用 canvas 绘制归一化骨架点和连线;Browser Camera provider 会额外通过 `earth:motion-debug-video-source` 提供本机 `<video>` 作为调试预览底图,`shared.motionDebugSkeletonOnly` 可切换为只显示骨骼。`停止匹配动作` 通过 `earth:motion-recognition-pause` 暂停 gesture 执行,但继续显示视频和骨架。未匹配动作为红色,匹配后变绿并显示动作名。设置项持久化在 `planet.earth.settings.v2` 的 `shared.motionDebugEnabled`、`shared.motionProvider` 与 `shared.motionDebugSkeletonOnly`,switch 和输入源控件都预留 `data-gatekeeper-permission="earth.motion_debug"`。
动捕调试面板由 [motion-debug-panel.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/motion-debug-panel.js) 负责。它监听 `earth:motion-debug-frame`,用 canvas 绘制归一化骨架点和连线;Browser Camera provider 会额外通过 `earth:motion-debug-video-source` 提供本机 `<video>` 作为调试预览底图,`shared.motionDebugSkeletonOnly` 可切换为只显示骨骼。`停止匹配动作` 通过 `earth:motion-recognition-pause` 暂停 gesture 执行,但继续显示视频和骨架。未匹配动作为红色,匹配后变绿并显示动作名。设置项持久化在 `planet.earth.settings.v2` 的 `shared.motionDebugEnabled`、`shared.motionProvider`、`shared.motionDebugSkeletonOnly` 与 `shared.motionEnabledGestures`,switch、输入源和动作白名单控件都预留 `data-gatekeeper-permission="earth.motion_debug"`。
Browser Camera provider 的手势识别管线在 [motion-browser-provider.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/motion-browser-provider.js) 的 `recognizeGesture()`,按以下顺序匹配,前者命中即返回:
@@ -312,9 +314,8 @@ AIS 船只图层入口:
船只图层当前负责:
- 请求 `/api/v1/vessels/snapshot` 获取当前视口初始快照;请求必须携带 `bbox`、`zoom`,并传入受控 `limit`
- 通过 `/ws` 的 `vessels` channel 订阅后续增量;订阅 payload 同样必须携带当前视口 `bbox`、`zoom`、`limit`
- 将聚合后的 AIS GeoJSON 转为地球局部坐标 marker 数据;后端默认 `limit=1000`,最大 `limit=5000`
- 请求 `/api/v1/vessels/snapshot` 获取全局当前状态快照;Earth 前端统一传全球 bbox、当前 `zoom` 和 `limit=3000`
- 将 `vessel_current_state` 当前状态 GeoJSON 转为地球局部坐标 marker 数据;历史 AIS 原始观测只用于轨迹、审计和态势分析
- 通过 `createInteractableLayer()` 注册 Interactable 图标层
- 用按航向分桶的 `THREE.Points` 批量渲染普通船只 marker
- 按船型映射颜色;`vessels.js` 会用 `vessel_type_name` 和 AIS `vessel_type` 数字共同归一化船型
@@ -330,6 +331,8 @@ AIS 船只图层入口:
- moving 船只按 `VESSEL_COURSE_BINS` 做航向分桶。
- 每个批次是一组 `THREE.PointsMaterial`,位置和颜色写入 `BufferGeometry` attribute。
- 普通态不带 glow;hover / locked 时才在相同点位叠加带 glow 的单点 overlay。
- `cluster: false` 且 `avoidance: false`,密集海域允许重叠,不参与动态屏幕聚类。
- 地球旋转和缩放不会重新请求船只,也不会按当前镜头 bbox 重连 WebSocket。
方向标准以 AIS `course / cog` 为准:从正北开始顺时针。普通态和交互态都通过同一套 canvas 旋转规则生成纹理,避免 hover 后箭头方向和原 marker 不一致。
@@ -337,7 +340,7 @@ AIS 船只图层入口:
AISStream 的 `PositionReport` 常带实时位置和 `MetaData.ShipName`,但船型通常来自低频 `ShipStaticData.Type`。后端会把 `MetaData.ShipName` 补进船名,并将类型码映射为 Cargo / Tanker / Passenger / Fishing / Military;仍缺失的船型需要等待静态 AIS 消息或后续船舶资料 enrichment,不能在前端凭颜色之外的信息臆造细分类。
旧 `/api/v1/visualization/geo/vessels` 路由已移除。前端打开船只图层时应先按当前视口拉一次 `/api/v1/vessels/snapshot`,再用 WebSocket 接收同一视口内的 upsert 增量;地图拖动或缩放后应重新拉取 snapshot 并重发 vessels 订阅。后端只在当前 raw 窗口为空时受控回退到 legacy `vessel_position` / `vessel_static`,前端可通过 `diagnostics.legacy_fallback_used` 识别该状态。
旧 `/api/v1/visualization/geo/vessels` 路由已移除。前端打开船只图层时只应拉取一次全局 `/api/v1/vessels/snapshot`;API 参数里的 bbox 是后端接口约束,Earth 运行时传全球范围,不表示当前镜头视口。`/ws` 的 `vessels` channel 如启用,只作为低频 reload/dirty 提示,不能把每条 AIS delta 直接变成整层重建。后端通过 `diagnostics.source == "vessel_current_state"` 暴露当前状态链路。
新的图层接口族是 `/api/v1/layers/*`,用于把地图渲染数据和聚合面板统计分开。地图层请求必须带 `bbox`、`zoom` 和受控 `limit`,响应会返回 `visible_count`、`returned_count` 和 `diagnostics`,其中 `degraded/truncated/limit_clamped` 用于前端提示降级。右侧聚合统计不要从图层响应累加,应读取 `/api/v1/data-products` 或 `/api/v1/data-products/{product_id}/status`,因为这些统计保持全量/全局口径,不随当前视口变化。
@@ -205,7 +205,7 @@
| 船只 renderOrder | local `VESSEL_RENDER_ORDER` | `4.4` | 普通 marker 和交互 overlay |
| 船只轨迹 renderOrder | `VESSEL_RENDER_ORDER - 0.1` | `4.3` | 低于船只 marker |
| 船只点像素尺寸 | local `VESSEL_POINT_SIZE` | `34` | 普通 marker 与 hover / locked overlay 共享尺寸 |
| 船只默认渲染上限 | `VESSEL_CONFIG.maxRenderedMarkers` | `0` | `0` 表示不在前端默认裁剪;正数才会给接口传 `limit` 并裁剪 marker |
| 船只默认渲染上限 | `VESSEL_CONFIG.maxRenderedMarkers` | `3000` | 前端请求全局当前状态快照时传给 `/api/v1/vessels/snapshot` 的默认 `limit`;后端上限仍为 `5000` |
| 船只纹理画布尺寸 | local `VESSEL_ATLAS_CELL_SIZE` | `128` | canvas 点纹理 |
| 航向分桶数 | local `VESSEL_COURSE_BINS` | `32` | moving 船只按 COG 分桶,降低 draw call 同时保留方向 |
| 船只 hover 拾取节流 | local `VESSEL_HOVER_PICK_INTERVAL_MS` | `100` | `main.js` hover picking |
@@ -216,7 +216,7 @@
| locked 船只透明度 | inline | `1` | locked overlay |
| 船型颜色 | `VESSEL_CONFIG.colors.*` | cargo / tanker / passenger / fishing / military / other | `PointsMaterial.vertexColors` 和 overlay texture |
AIS 船只普通态使用批量 `THREE.Points`,不是逐船 `THREE.Sprite`。航行船只保持三角形,停泊或低速船只保持圆点;普通态不带 glow,hover / locked 时在同一屏幕尺寸上叠加带 glow 的单点 overlay。AIS 航向按 `course / cog` 从正北顺时针解释,普通态和交互态必须使用同一套 canvas 旋转规则。
AIS 船只普通态使用批量 `THREE.Points`,不是逐船 `THREE.Sprite`。航行船只保持三角形,停泊或低速船只保持圆点;普通态不带 glow,hover / locked 时在同一屏幕尺寸上叠加带 glow 的单点 overlay。AIS 航向按 `course / cog` 从正北顺时针解释,普通态和交互态必须使用同一套 canvas 旋转规则。船只显式关闭 `cluster` 和 `avoidance`,密集区域可以重叠,不能重新接入动态屏幕聚类。
船型颜色和详情卡船型文本必须来自同一套归一化结果:`vessels.js` 同时读取后端 `vessel_type_name` 和 AIS 数字 `vessel_type`,先得到颜色用的 `type`,再生成 `vessel_type_display` 给详情卡、hover 和搜索使用。
+89
View File
@@ -80,10 +80,61 @@ Earth 态势新闻使用 `/api/v1/news/earth-feed` 输出给前端。新闻源
官方数据源、电商指标、平台型公司、量化指标会提高重要度;企业公告基础权重较低,只有命中大平台、金额、并购、监管等信号时提升。
重要度等级固定为:`low` 0–34、`medium` 35–59、`high` 60–79、`critical` 80–100。分类、重要度和 Breaking 的计算集中在 `earth_news_classification.py`;分类 key 和标签仍可配置,重要度与 Breaking 协议状态使用统一枚举,数据库和 API 继续保存兼容的小写字符串。
## 配置与缓存
`GET /api/v1/earth/news-sources` 返回默认或已保存配置。`PUT /api/v1/earth/news-sources` 保存配置并递增 `cache_version`,同时清理进程内 region cache。`POST /api/v1/earth/news-sources/reset` 恢复默认源。`POST /api/v1/earth/news-sources/test` 只测试单个 RSS/Atom/Aggregated 源,不写入新闻表。
## 手动新闻内容
手动新闻是内容管理能力,不是 RSS 源配置。Admin 入口是 `智能星球内容 -> 新闻内容`,左栏按 RSS 来源和手动新闻组聚合;进入手动组详情后可以按条添加新闻或上传 JSON 数组批量导入。手动新闻写入 `earth_news_items`,并标记 `feed_type/source_type = manual`;前台展示文案是“手动添加”。它不参与 RSS 连通性测试,也不会进入 RSS 抓取流程。
手动新闻采用“先展示,再精修”的策略:
1. 保存后立即写入 `earth_news_items`。
2. 没有人工坐标时使用所选区域锚点,`verified=false`。
3. 有人工坐标时使用 `location_source=manual_location`,`verified=true`,后续 AI 精修不会覆盖该坐标。
4. 创建或重新处理后进入新闻增强队列,后台补清洗、翻译、分类、重要度、Breaking 和目标位置推断。
5. 精修完成后更新同一条新闻,并通过 Earth news reload / patch 让前端无感替换。
后台接口挂在 `/api/v1/earth/news-items`:
- `GET /earth/news-groups`:返回 RSS 虚拟来源组和手动新闻组。
- `POST /earth/news-groups`:新建手动新闻组。
- `PUT /earth/news-groups/{group_id}`:重命名手动新闻组,并同步组内新闻 meta。
- `GET /earth/news-items`:分页查询 RSS 与手动新闻,支持来源类型、区域、类型和状态过滤。
- `POST /earth/news-items`:新增一条手动新闻。
- `POST /earth/news-items/import`:上传 JSON 数组批量导入,`group_id` 指定当前手动新闻组。
- `PUT /earth/news-items/{id}`:编辑手动新闻;RSS 新闻只读。
- `DELETE /earth/news-items/{id}`:删除手动新闻,并触发 Earth 新闻重载。
- `POST /earth/news-items/{id}/reprocess`:重新进入清洗、翻译和定位队列。
JSON 导入首版只支持数组:
```json
[
{
"title": "必填标题",
"summary": "可选摘要",
"content": "可选正文",
"url": "https://example.com/story",
"source": "手动添加",
"region": "global",
"published_at": "2026-05-15T03:00:00Z",
"category": "business",
"tags": ["manual", "analysis"],
"location": {
"label": "北京市, 中国",
"latitude": 39.9057,
"longitude": 116.3913
}
}
]
```
重复导入使用稳定 ID 去重,ID 由标题、发布时间、URL 和来源生成,格式为 `manual:{hash}`。同一条手动新闻再次导入会更新原记录,不会重复出现在 EarthFeed。
## Feed 查询与类型过滤
星球端和 UE 端统一使用 `GET /api/v1/news/earth-feed` 获取新闻。接口支持服务端过滤,不要求客户端拿全量列表后自行筛选。
@@ -105,8 +156,44 @@ GET /api/v1/news/earth-feed?region=global&categories=business,ecommerce&locale=z
Web 星球端的新闻类型按钮只保存当前浏览器的显示偏好;偏好变化后会重新请求接口。UE 端应直接把类型选择拼到 `categories` 参数里,不需要再做主过滤。
未指定 `sources` 时,服务层会优先选择当前 `locale` 已有可展示标题和摘要的新闻,并从当前启用来源补齐候选后做来源轮转,避免一个来源的最新待处理条目占满默认 12 条。指定 `sources` 时保持精确来源过滤,不做跨来源补齐。数据库查询层只负责区域、类型、来源和排序条件,不包含语言展示策略。
源测试只证明当前 RSS/Atom/XML 能解析到条目,不等于这些条目已经入库展示。展示链路还会检查区域、类型过滤和数据库新鲜度。保存或重置新闻源会递增配置版本并清理缓存;如果当前启用的 Feed 子项在库里没有近期条目,下一次 `earth-feed` 请求会补抓,避免新启用的 36氪、亿邦被旧 Google News 缓存挡住。
## 区域均衡与异步精修
`earth_news_items` 是 EarthFeed 的当前状态表。接口在没有显式 `sources` 过滤时会先按 Breaking、区域、发布时间排序,再做区域轮转,避免亚太或任一高频来源把全球视图和巡航队列全部占满。默认区域顺序是美洲、欧洲、中东与非洲、亚太、全球;未知区域只在已知区域之后参与轮转。区域视图仍遵守“当前区域 + global”的规则,不会把其它区域混进区域面板。
`items` 和 `cruise_items` 都会进入目标位置与本地化精修队列,但前端不能等 AI 完成后再展示。标题或摘要没有目标语言翻译时,Web Earth 先显示原始 `title / summary`,避免卡片出现“新闻汉化中”而实际内容已经可读。后台完成翻译、分类、Breaking 或目标坐标后,会更新同一条 `earth_news_items` 并通过 Earth news reload / patch 刷新前端。
目标位置队列使用 Redis Streams 两级队列:
- `earth_news:target_location:priority`:当前可见 `items` 与 `cruise_items` 的优先精修任务,短 TTL 去重。
- `earth_news:target_location:jobs`:普通后台精修任务,长 TTL 去重。
Worker 总是先处理优先队列,再处理普通队列;pending 消息超过空闲阈值会被 reclaim,单条 AI 任务有超时保护,失败后进入重试或 dead letter。这样即使历史普通队列有大量 backlog,当前打开的欧洲、美洲等区域新闻也不会被长队列饿死。
## Breaking News 插队
新闻体系里有三套互不替代的判断:
- **分类**:新闻是什么,例如商业、军事、灾害。
- **重要度**:长期是否值得关注,写入 `importance_score / importance_level`。
- **Breaking**:短时间内是否必须插队,写入 `location_meta.news_meta.breaking_*`。
Breaking 不新增表字段,继续保存在 `location_meta.news_meta`:
- `breaking_level`:`none / watch / breaking / critical`。
- `breaking_scope`:`regional / global`。
- `breaking_reasons`:触发原因。
- `breaking_source`:`rules / ai / manual / multi_source`。
- `breaking_confidence`:0 到 1。
- `breaking_expires_at`:过期时间。
排序由服务端完成,客户端和 UE 不需要自己重排。未过期的 `critical`、`breaking`、`watch` 会依次排在普通新闻前面;过期后只回到普通排序,不删除新闻,也不改变长期重要度。
`breaking_scope = global` 的新闻会无视当前区域,进入所有区域的 feed;`regional` 只遵守当前区域加 `global` 的普通区域规则。接口响应的 `filters` 会返回 `has_breaking` 和 `highest_breaking_level`,星球端据此给新闻面板和卡片加克制的背景/边框状态。
## 连通性监测
`POST /api/v1/earth/news-sources/test` 会测试单个源并把结果写入 `earth_news_sources.health[source_id]`。实际 RSS/Atom 抓取也会更新同一份健康状态。
@@ -140,9 +227,11 @@ Admin 入口是 `Earth 内容 -> 新闻源`。界面不是整包 JSON 编辑,
```mermaid
flowchart LR
Admin["Admin: Earth 内容 / 新闻源"] --> Source["Source 配置"]
ManualAdmin["Admin: Earth 内容 / 新闻内容"] --> ManualAPI["/api/v1/earth/news-items"]
Source --> Feed["Feed 子项"]
Feed --> ConfigAPI["/api/v1/earth/news-sources"]
ConfigAPI --> Config["SystemSetting: earth_news_sources"]
ManualAPI --> Store
Earth["Earth 新闻面板"] --> NewsAPI["/api/v1/news/earth-feed"]
NewsAPI --> Resolver["Source Resolver"]
@@ -30,7 +30,7 @@
| 3 | 卫星 footprint 填充 / Iridium coverage ring | `satellites.js`, `iridium-footprint-adapter.js` | `GROUND_FOOTPRINT_RENDER_ORDER` | depth-tested;Iridium adapter 的 fill / ring 也使用同一 renderOrder | Footprint 在 land / texture / terrain 和国界线之上,但在算力中心和卫星之下。 |
| 3-4.5 | BGP 观测站、事件扩散圈和事件 marker | `bgp.js`, `interactable.js` | BGP 观测站和事件 marker 均使用 `Interactable` 批量 `THREE.Points`;事件 marker 使用 `BGP_EVENT_RENDER_ORDER = 4.5`;观测站主图标使用 `BGP_COLLECTOR_RENDER_ORDER = 4.4` 和 `BGP_CONFIG.collectorAltitudeOffset = 0.2`;事件 overlay 进入 `bgp-event-overlay-layer`;观测站 halo 和覆盖扇形进入 `bgp-collector-radar-layer` | BGP 事件和观测站都通过 `Interactable` 屏幕空间 picking,并参与同坐标避让 | BGP 观测站主图标与船只同层;BGP 事件与算力中心同层;向外扩散圈、观测站雷达/覆盖动画继续由 BGP 业务逻辑驱动。 |
| 4.3 | AIS 船只轨迹线 | `vessels.js` | `VESSEL_RENDER_ORDER - 0.1`;`CONFIG.earthRadius + VESSEL_CONFIG.track.altitudeOffset` | 跟随船只显隐,不单独参与拾取 | 选中船只后显示最近轨迹,低于船只 marker。 |
| 4.4 | AIS 船只 marker | `vessels.js`, `interactable.js` | `VESSEL_RENDER_ORDER`;业务高度为 `CONFIG.earthRadius + VESSEL_CONFIG.altitudeOffset`;普通 marker 为分桶 `THREE.Points`,hover / locked 为单点 `THREE.Points` overlay | `depthTest: true`;`main.js` 使用屏幕空间 picking,只取正面 marker;参与 Interactable 同坐标避让 | 航行船只用三角点纹理,停泊/低速用圆点;普通态无 glow,交互态叠加同尺寸 glow;低于算力中心 `4.5`。 |
| 4.4 | AIS 船只 marker | `vessels.js`, `interactable.js` | `VESSEL_RENDER_ORDER`;业务高度为 `CONFIG.earthRadius + VESSEL_CONFIG.altitudeOffset`;普通 marker 为分桶 `THREE.Points`,hover / locked 为单点 `THREE.Points` overlay | `depthTest: true`;`main.js` 使用屏幕空间 picking,只取正面 marker;`cluster: false` 且 `avoidance: false` | 航行船只用三角点纹理,停泊/低速用圆点;密集海域允许重叠,不参与动态屏幕聚类,避免旋转时重建 Points;低于算力中心 `4.5`。 |
| 4.5 | 算力中心 | `compute-centers.js`, `interactable.js` | 使用 `COMPUTE_CENTER_RENDER_ORDER` 并由 `Interactable` 绘制 | 通过 `Interactable` 屏幕空间 picking,参与同坐标避让 | 地表设施层,保持在卫星下方。登陆点已下沉到海缆层。 |
| 5 | 卫星背景点 | `satellites.js` | 固定 renderOrder;默认按 TLE/SGP4 真实高度压缩到 `CONFIG.earthRadius + 4..25`,关闭真实高度或传播失败时回退到 `fallbackAltitudeOffset = 8` | 屏幕空间卫星拾取 | 位于卫星点下方。 |
| 6 | 卫星点 | `satellites.js` | 与卫星背景点使用同一压缩高度 / fallback 高度 | 屏幕空间卫星拾取 | 卫星点压过 footprint 和算力中心。 |
+4 -1
View File
@@ -193,6 +193,7 @@ Base URL 输入框尾端的插头图标会触发连接测试。测试通过会
- **品牌资源**:维护智能星球 HUD 使用的 logo、标题图、标题文本、副标题和描述;上传的图片会保存为智能星球品牌资产并立即供智能星球页面读取。
- **关于**:维护智能星球设置面板里的关于卡片,包括 logo、眉标、标题、版本、描述和元信息。
- **电视直播**:维护智能星球媒体面板里的直播源。
- **新闻内容**:按 RSS 来源和手动新闻组查看新闻。RSS 新闻保持只读;手动新闻组可以新增、批量导入 JSON、编辑、删除和重新处理。
- **国界精度**:查看当前国界 provider、低精 fallback、高精 PMTiles/manifest 状态,编辑本机源配置并手动构建。
- **地球底图**、**图层资源**、**三维素材**、**新闻锚点策略**:目前是待接入占位页,不展示假数据。
@@ -297,7 +298,7 @@ AIS 船只图例按船型显示颜色:货轮、油轮、客船、渔船、军
### 设置
设置面板按分类组织:运行、显示、面板、动捕、快捷键、系统。里面包含旋转模式 / 巡航模式 / 动捕模式、巡航模块(BGP/新闻/算力中心/船只/海缆/卫星)、视图设置(卫星显示风格、悬停提示、卫星呼吸闪烁、真实卫星高度、轨迹显示、低缩放圆点、日夜模式、面板显示开关)、动捕调试模式 / 输入源 / 只显示骨骼、快捷键启用与改键、地球默认大小、地形透明度、重置设置。
设置面板按分类组织:运行、显示、面板、动捕、快捷键、系统。里面包含旋转模式 / 巡航模式 / 动捕模式、巡航模块(BGP/新闻/算力中心/船只/海缆/卫星)、视图设置(卫星显示风格、悬停提示、卫星呼吸闪烁、真实卫星高度、轨迹显示、低缩放圆点、日夜模式、面板显示开关)、动捕调试模式 / 输入源 / 只显示骨骼 / 识别动作白名单、快捷键启用与改键、地球默认大小、地形透明度、重置设置。
新闻类型使用与巡航模块一致的标签选择器,只筛选当前浏览器里的新闻面板和新闻巡航条目,不影响图层、TV、数据点、底图、边界、采集任务或后台新闻源配置。
@@ -335,6 +336,8 @@ AIS 船只图例按船型显示颜色:货轮、油轮、客船、渔船、军
两种模式都不会把摄像头帧或实时手势发到云端,也不会复用新闻/RSS 聚合接口。
“识别动作”可以单独关闭旋转、缩放、焦点切换、图层切换或确认手势。浏览器端会忽略未勾选动作;使用 Motion Agent 时,同一白名单也会通过控制协议同步给 Agent。
手势语义:
| 手势事件 | 作用 |
+44 -13
View File
@@ -258,22 +258,27 @@ HTTP 健康检查统一使用 `curl -fsS --max-time`。因此 `/health` 返回 4
前端生产形态是 `vite build` 生成静态资源,再由 nginx/Caddy 等 HTTP 服务器托管。不要在生产中使用 `bun run dev` 或 `vite preview`。当前不维护 Webpack 双构建链;如果未来需要评估更企业化的构建生态,优先做 Rsbuild/Rspack spike。Electron 仅在正式目标变成离线桌面软件时再单独评估。
## Motion Agent 可选启动
## Motion Agent 默认启动
`planet.sh` 现在可以管理本地动作捕捉 Agent,但默认不会启动它,避免普通开发机因为没有摄像头、OpenCV 或 MediaPipe 而影响后端/前端启动。
`planet.sh` 现在默认随 `start` 和全量 `restart` 启动本地 Motion Agent。这样星球端、UE 或调试客户端可以直接连接 `ws://127.0.0.1:8765/ws/gestures`。如果当前机器没有可用摄像头,默认隐式启动会降级为 dry-run 协议服务,不会阻断后端/前端启动;只有显式传入 `--motion-agent`、摄像头 index、摄像头 URL 或 WSL USB 参数时,live 模式缺摄像头才会硬失败。
启动方式:
如果本次不需要 Motion Agent:
```bash
./planet.sh start --motion-agent
./planet.sh start --non-motion-agent
./planet.sh restart --non-motion-agent
```
常用参数:
- `--motion-agent` / `-m`:随本次启动或重启拉起 Motion Agent。
- `--non-motion-agent`:本次启动或全量重启不拉起 Motion Agent。
- `--motion-agent` / `-m`:显式要求本次启动或重启拉起 Motion Agent;此时 live 摄像头失败会作为错误反馈。
- `--motion-agent-port <端口>`:覆盖默认 WebSocket 端口 `8765`。
- `--motion-agent-mode <模式>`:指定输入模式,可选 `auto`、`single`、`dual_redundant`、`single_fallback`、`calibrated_3d`;`dual` 作为兼容别名会进入双路冗余。
- `--motion-agent-camera-indexes <indexes>`:覆盖自动发现的摄像头 index,例如 `0` 或 `0,1`。也可以用环境变量 `MOTION_AGENT_CAMERA_INDEXES=0,1`。
- `--motion-agent-camera-urls <urls>`:使用 RTSP/HTTP 摄像头流,适合 WSL、手机摄像头或网络摄像头。也可以用环境变量 `MOTION_AGENT_CAMERA_URLS=...`。
- `--motion-agent-wsl-usbipd`:在 WSL 中尝试通过 `usbipd-win` 自动把唯一的 Windows USB 摄像头透传到 Linux。
- `--motion-agent-wsl-usbipd-busid <BUSID>`:在 WSL 中指定 `usbipd list` 里的摄像头 BUSID 后透传,适合多摄像头设备。
- `--motion-agent-dry-run`:不打开摄像头、不加载 CV 依赖,只启动协议服务,适合调试 Web 端连接。
非 dry-run 的 live 模式会在启动前检查 `mediapipe` 和 `opencv-python`。如果当前 `.venv` 缺包,脚本会自动执行:
@@ -282,13 +287,17 @@ HTTP 健康检查统一使用 `curl -fsS --max-time`。因此 `/health` 返回 4
uv add mediapipe opencv-python
```
如需禁止启动时自动安装,可设置:
如需禁止启动时自动安装 Python CV 依赖,可设置:
```bash
PLANET_MOTION_AGENT_AUTO_INSTALL=0 ./planet.sh start --motion-agent
PLANET_MOTION_AGENT_AUTO_INSTALL=0 ./planet.sh start
```
live 模式会自动寻找 `/dev/video*`,优先取前两个 index 传给 Motion Agent。在 WSL 中,Windows 摄像头通常不会自动出现在 `/dev/video*`。可先用下面命令看设备:
`./planet.sh init` 会在 WSL 中预检查 `usbipd-win`。如果没有 `usbipd.exe`,脚本会先尝试 `winget install -e --id dorssel.usbipd-win`,失败后复用仓库内置的 dorssel.usbipd-win MSI fallback;如果缓存缺失或需要其他架构版本,再下载 MSI 并请求管理员 PowerShell 安装。该步骤是 best-effort:失败会提示后续处理方式,但不会阻断普通初始化。可通过 `./planet.sh init --non-motion-agent` 跳过该预检。
live 模式会自动寻找 `/dev/video*`,并优先用 OpenCV 实测过滤出真正能打开并读帧的 index,再传给 Motion Agent。在 WSL/USB 摄像头场景中,一个摄像头可能暴露多个 `/dev/video*` 节点,其中部分是 metadata 或非采集节点,脚本会跳过这类不可读 index。在 WSL 中,Windows 摄像头通常不会自动出现在 `/dev/video*`。可先用下面命令看设备:
默认 live 采集使用低延迟参数:`640x360` 输入、约 `15Hz` 识别事件;worker 内部用 latest-frame 读帧线程,只保留每路摄像头的最新帧,避免 MediaPipe 慢帧时继续排队识别旧画面。骨架调试流默认关闭,只在星球端打开动捕调试面板时按约 `8Hz` 推送,避免日常手势控制被调试数据拖慢。状态事件会同时上报采集 FPS 与识别 FPS,方便区分摄像头掉帧和识别耗时。
```bash
ls /dev/video*
@@ -306,20 +315,42 @@ WSL 下更通用的方式是把手机摄像头或网络摄像头以 RTSP/HTTP
./planet.sh start --motion-agent --motion-agent-camera-urls http://192.168.1.20:8080/video
```
如果 WSL 中没有发现 `/dev/video*`,且没有提供 `--motion-agent-camera-urls`,脚本会停止 live 启动并提示处理方式,不会自动降级为 dry-run。可选处理:
如果希望直接使用 Windows USB 摄像头,可以让脚本调用 `usbipd-win` 透传。该能力是显式开启的,因为摄像头附加到 WSL 期间通常会从 Windows 应用中暂时断开。
只有一个摄像头时:
```bash
./planet.sh start --motion-agent --motion-agent-wsl-usbipd
```
多个摄像头时,先查看 BUSID,再指定设备:
```bash
usbipd.exe list
./planet.sh start --motion-agent --motion-agent-wsl-usbipd-busid 3-2
```
如果 `usbipd attach` 提示设备未共享或未绑定,脚本会尝试弹出 Windows 管理员 PowerShell 自动执行 `usbipd bind`,然后重试 attach。若 UAC 被取消或自动 bind 失败,可在 Windows 管理员 PowerShell 中手动执行:
```powershell
usbipd bind --busid 3-2
usbipd attach --wsl --busid 3-2
```
如果 WSL 中没有发现 `/dev/video*`,且没有提供 `--motion-agent-camera-urls`,默认隐式启动会降级为 dry-run。显式 live 启动会停止并提示处理方式。可选处理:
```bash
./planet.sh start --motion-agent --motion-agent-camera-urls http://<手机IP>:8080/video
./planet.sh start --motion-agent --motion-agent-wsl-usbipd
./planet.sh start --motion-agent --motion-agent-dry-run
```
只有显式设置 `PLANET_MOTION_AGENT_WSL_ALLOW_DRY_RUN_FALLBACK=1` 时,WSL 无摄像头才会自动降级。
显式 live 启动时,只有设置 `PLANET_MOTION_AGENT_WSL_ALLOW_DRY_RUN_FALLBACK=1`,WSL 无摄像头才会自动降级。
也可以用环境变量启用:
`--non-motion-agent` 是命令级跳过入口。环境变量仍可调整服务启动方式:
```bash
PLANET_START_MOTION_AGENT=1 ./planet.sh start
MOTION_AGENT_DRY_RUN=1 PLANET_START_MOTION_AGENT=1 ./planet.sh start
MOTION_AGENT_DRY_RUN=1 ./planet.sh start
```
日志入口:
+9 -9
View File
@@ -39,7 +39,7 @@ flowchart TB
VesselRaw["AIS / BarentsWatch"] --> VesselDerived["vessel_static / vessel_position"]
VesselDerived --> VesselLayer["vessels 图层"]
Interactables["earth_interactables"] --> InteractableLayer["interactables 图层"]
NewsRaw["RSS / Live / News"] --> NewsItems["earth_news_items"]
NewsRaw["RSS / 手动新闻 / Live"] --> NewsItems["earth_news_items"]
NewsItems --> NewsLayer["news / media 图层"]
```
@@ -51,7 +51,7 @@ flowchart TB
| BGP 态势 | 展示观测站、异常事件、路由事件和区域态势 | `ris_live_bgp`、`bgpstream_bgp`、prefix geography sources | `bgp_observations`、`bgp_anomalies`、`bgp_incidents`、`bgp_collector_locations` | `bgp` | `clear_then_reload` |
| 船舶 | 展示 AIS 船只、位置、轨迹和源健康 | AIS sources、`barentswatch_vessels` | `vessel_static`、`vessel_position`、`ais_raw_observations`、`ais_source_health` | `vessels` | `clear_then_reload` |
| 可交互对象 | 支撑通用地表图标、人工点位和未来扩展对象 | `earth_interactables` | 无 | `interactables` | `delta` |
| 新闻与媒体 | 支撑 Earth 新闻、直播和巡航摘要 | news sources | `earth_news_items` | `news` / `media` | `reload` |
| 新闻与媒体 | 支撑 Earth 新闻、直播和巡航摘要 | RSS news sources、手动新闻、直播源 | `earth_news_items` | `news` / `media` | `reload` |
## 卫星链路
@@ -118,12 +118,12 @@ sequenceDiagram
## 船舶链路
船舶数据用于展示 AIS 船只、航行状态、船型图例和源健康。Earth 渲染使用位置快照和静态船舶信息,不应依赖原始 AIS 记录逐条渲染。
船舶数据用于展示 AIS 船只、航行状态、船型图例和源健康。Earth 渲染使用 `vessel_current_state` 当前状态快照,不应依赖原始 AIS 记录逐条渲染。
- **采集入口**:AIS sources、BarentsWatch vessels。
- **事实表**:`collected_data` 或 AIS 原始观测表。
- **派生表**:`vessel_static`、`vessel_position`、`ais_raw_observations`、`ais_source_health`。
- **接口**:vessels visualization API 返回当前船只 marker 和必要详情。
- **派生表**:`vessel_current_state`、`vessel_static`、`vessel_position`、`ais_raw_observations`、`ais_source_health`。
- **接口**:`/api/v1/vessels/snapshot` 返回当前船只 marker 和必要详情;Earth 前端使用全球 bbox 请求全局当前状态。
- **删除语义**:删除任一船舶 source 后,owned 派生表变化会广播 `vessels` 的 `clear_then_reload`。
- **常见异常**:数量面板变化但船只仍在,多半是 summary 和图层数据分离,前端应以 layer update 为准清空对象。
@@ -141,12 +141,12 @@ sequenceDiagram
新闻与媒体数据用于 Earth 顶部新闻条、直播面板、新闻巡航和态势摘要。它们的视觉状态比地理对象更偏内容刷新,因此默认使用 `reload`。
- **采集入口**:RSS、直播源、新闻 source。
- **事实表**:新闻 source 的 `collected_data`。
- **采集入口**:RSS 新闻源、`智能星球内容 -> 新闻内容` 的手动新闻、直播源。
- **事实表**:新闻 source 的 `collected_data`;手动新闻直接写入 `earth_news_items`,并以 `feed_type/source_type=manual` 标记内容来源。
- **派生表**:`earth_news_items`。
- **接口**:新闻、直播和媒体 visualization / content API。
- **接口**:`/api/v1/news/earth-feed` 读取 `earth_news_items`;后台管理接口 `/api/v1/earth/news-items` 支持新增、JSON 导入、编辑、删除和重新处理手动新闻。
- **删除语义**:删除新闻 source 或 `earth_news_items` 后广播 `news` / `media` reload;前端重拉后列表为空即隐藏对应内容。
- **常见异常**:直播面板仍显示旧内容,通常是媒体组件本地状态没有响应 layer update,或内容接口缓存未失效。
- **常见异常**:手动新闻保存后只显示原文或大区锚点是正常的“先展示再精修”窗口;若长期不更新,应检查 `earth_news_enrichment` 队列、AI / Web Search 配置和 `enrichment_status`。直播面板仍显示旧内容,通常是媒体组件本地状态没有响应 layer update,或内容接口缓存未失效。
## 扩展新图层
+1 -1
View File
@@ -50,7 +50,7 @@
- 搜索可以查找海缆、卫星、算力中心、BGP 事件
- 算力中心和 BGP 观测站详情卡可以自动采集坐标候选,并能在智能星球上预览
- 鼠标拖动、滚轮缩放、缩放百分比提示工作正常
- 设置面板的旋转 / 巡航 / 动捕模式可以切换;视图设置里可以切换悬停提示,卫星相关设置里可以打开或关闭真实高度分层和轨迹显示
- 设置面板的旋转 / 巡航 / 动捕模式可以切换;动捕设置可以选择输入源和允许识别的动作;视图设置里可以切换悬停提示,卫星相关设置里可以打开或关闭真实高度分层和轨迹显示
## 5. 找回密码
+4 -1
View File
@@ -16,12 +16,15 @@
## Current Version
- `main` 当前主线历史推导到:`0.16.5`
- `dev` 当前开发分支历史推导到:`0.69.0`
- `dev` 当前开发分支历史推导到:`0.71.1`
## Timeline
| Version | Type | Branch | Commit | Summary |
| --- | --- | --- | --- | --- |
| `0.71.1` | bugfix | `dev` | `pending` | 修复 Earth 新闻区域切换、滚动条/面板/巡航一致性和新闻精修队列饿死问题,并补充 agent harness 与双语维护文档 |
| `0.71.0` | feature | `dev` | `pending` | Motion Agent 升级为 Web/UE 共用双向控制与真实识别服务,新增 Earth 手动新闻工作流、来源多样化,并完善启动/测试 harness 与双语文档 |
| `0.70.0` | feature | `dev` | `pending` | 新增后端枚举契约治理、Earth 新闻分类/Breaking 链路和船只当前状态快照,清理错误视口刷新逻辑并同步双语文档 |
| `0.69.0` | feature | `dev` | `pending` | 新增 Earth 新闻源治理、新闻类型服务端过滤、观测日志 fingerprint 聚合和 TV/HLS 播放恢复改进 |
| `0.68.1` | bugfix | `dev` | `pending` | 修复 CelesTrak fallback group/cache 恢复链路,并让数据源任务日志可按 task_id / datasource_id 搜索 |
| `0.68.0` | feature | `dev` | `pending` | 新增数据源任务队列实时指标、AIS 大表分批删除和智能星球可插拔聚类策略,并让新设备启动前同步前端依赖 |
Binary file not shown.
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "planet-frontend",
"version": "0.69.0",
"version": "0.71.1",
"private": true,
"packageManager": "bun@1",
"dependencies": {
+74 -21
View File
@@ -14,12 +14,12 @@
border-radius: 0;
padding: 0 calc(18px * var(--hud-scale));
display: grid;
grid-template-columns: auto minmax(0, 1fr);
grid-template-columns: minmax(calc(112px * var(--hud-scale)), calc(152px * var(--hud-scale))) minmax(0, 1fr);
align-items: center;
gap: calc(12px * var(--hud-scale));
color: var(--hud-text);
background:
linear-gradient(90deg, transparent 0%, rgba(17, 31, 53, 0.82) 11%, rgba(7, 17, 31, 0.78) 89%, transparent 100%),
linear-gradient(90deg, transparent 0%, rgba(17, 31, 53, 0.28) 4%, rgba(17, 31, 53, 0.82) 12%, rgba(7, 17, 31, 0.78) 88%, rgba(7, 17, 31, 0.26) 96%, transparent 100%),
radial-gradient(circle at 50% -80%, rgba(145, 186, 255, 0.18), transparent 58%);
box-shadow:
inset 0 1px 0 rgba(255, 255, 255, 0.08),
@@ -30,6 +30,8 @@
cursor: pointer;
z-index: 24;
overflow: hidden;
-webkit-mask-image: linear-gradient(90deg, rgba(0, 0, 0, 0.84) 0%, #000 3%, #000 97%, rgba(0, 0, 0, 0.84) 100%);
mask-image: linear-gradient(90deg, rgba(0, 0, 0, 0.84) 0%, #000 3%, #000 97%, rgba(0, 0, 0, 0.84) 100%);
transition: opacity 0.18s ease, transform 0.24s ease;
}
@@ -40,41 +42,30 @@
.earth-news-ticker::before,
.earth-news-ticker::after {
content: "";
position: absolute;
top: -1px;
bottom: -1px;
width: calc(96px * var(--hud-scale));
pointer-events: none;
z-index: 2;
}
.earth-news-ticker::before {
left: 0;
background: linear-gradient(90deg, rgba(0, 0, 0, 0), rgba(8, 18, 32, 0.08));
}
.earth-news-ticker::after {
right: 0;
background: linear-gradient(270deg, rgba(0, 0, 0, 0), rgba(8, 18, 32, 0.08));
content: none;
}
.earth-news-ticker__region {
position: relative;
z-index: 3;
min-width: 0;
max-width: 100%;
overflow: hidden;
text-overflow: ellipsis;
color: var(--hud-accent-strong);
font-size: calc(0.66rem * var(--hud-scale));
font-weight: 800;
letter-spacing: 0.12em;
text-transform: uppercase;
white-space: nowrap;
text-align: left;
}
.earth-news-ticker__viewport {
min-width: 0;
overflow: hidden;
-webkit-mask-image: linear-gradient(90deg, transparent 0%, #000 9%, #000 91%, transparent 100%);
mask-image: linear-gradient(90deg, transparent 0%, #000 9%, #000 91%, transparent 100%);
-webkit-mask-image: linear-gradient(90deg, transparent 0%, rgba(0, 0, 0, 0.22) 4%, rgba(0, 0, 0, 0.72) 11%, #000 18%, #000 82%, rgba(0, 0, 0, 0.72) 89%, rgba(0, 0, 0, 0.22) 96%, transparent 100%);
mask-image: linear-gradient(90deg, transparent 0%, rgba(0, 0, 0, 0.22) 4%, rgba(0, 0, 0, 0.72) 11%, #000 18%, #000 82%, rgba(0, 0, 0, 0.72) 89%, rgba(0, 0, 0, 0.22) 96%, transparent 100%);
}
.earth-news-ticker__track {
@@ -490,6 +481,54 @@
0 0 18px rgba(255, 184, 77, 0.12);
}
.earth-news-hud.has-breaking-watch,
.earth-news-ticker.has-breaking-watch,
.earth-mobile-news-board.has-breaking-watch {
box-shadow:
0 0 0 1px rgba(255, 213, 128, 0.08) inset,
0 18px 46px rgba(156, 106, 20, 0.12);
}
.earth-news-hud.has-breaking-breaking,
.earth-news-ticker.has-breaking-breaking,
.earth-mobile-news-board.has-breaking-breaking {
box-shadow:
0 0 0 1px rgba(255, 166, 102, 0.14) inset,
0 18px 48px rgba(185, 89, 32, 0.16);
}
.earth-news-hud.has-breaking-critical,
.earth-news-ticker.has-breaking-critical,
.earth-mobile-news-board.has-breaking-critical {
box-shadow:
0 0 0 1px rgba(255, 118, 118, 0.18) inset,
0 18px 52px rgba(172, 43, 43, 0.18);
}
.news-story-card--breaking-watch {
border-color: rgba(255, 213, 128, 0.28);
background:
linear-gradient(180deg, rgba(255, 248, 220, 0.08), rgba(255, 184, 77, 0.05));
}
.news-story-card--breaking-breaking {
border-color: rgba(255, 166, 102, 0.36);
background:
linear-gradient(180deg, rgba(255, 205, 150, 0.1), rgba(255, 135, 80, 0.07));
}
.news-story-card--breaking-critical {
border-color: rgba(255, 118, 118, 0.42);
background:
linear-gradient(180deg, rgba(255, 155, 155, 0.11), rgba(205, 58, 58, 0.08));
}
.news-story-card--breaking-global {
box-shadow:
0 0 0 1px rgba(255, 255, 255, 0.05) inset,
0 0 18px rgba(255, 168, 96, 0.1);
}
.news-story-meta,
.news-story-tags {
display: flex;
@@ -548,6 +587,12 @@
background: rgba(255, 255, 255, 0.04);
}
.news-story-tag--breaking {
color: rgba(255, 232, 205, 0.96);
border: 1px solid rgba(255, 184, 96, 0.24);
background: rgba(255, 151, 78, 0.11);
}
.news-board-empty {
color: var(--hud-text-muted);
font-size: calc(0.82rem * var(--hud-scale));
@@ -555,6 +600,14 @@
padding: calc(16px * var(--hud-scale)) calc(4px * var(--hud-scale));
}
@media (max-width: 1180px) {
.earth-news-ticker {
--news-ticker-width: min(calc(760px * var(--hud-scale)), 66vw);
grid-template-columns: minmax(calc(86px * var(--hud-scale)), calc(118px * var(--hud-scale))) minmax(0, 1fr);
padding-inline: calc(14px * var(--hud-scale));
}
}
.layout-mode-mobile .earth-news-ticker,
.layout-mode-mobile .earth-news-hud {
display: none !important;
+44
View File
@@ -1173,6 +1173,28 @@
<button type="button" class="earth-mobile-settings-pill" data-motion-provider="motion_agent" aria-pressed="false">Motion Agent</button>
</div>
</div>
<div
class="earth-mobile-settings-card earth-mobile-settings-card--stacked"
data-gatekeeper-permission="earth.motion_debug"
>
<div class="earth-mobile-settings-copy">
<span class="earth-mobile-settings-label">识别动作</span>
<span class="earth-mobile-settings-subtitle">关闭后不会触发对应星球控制</span>
</div>
<div class="earth-mobile-settings-chip-grid" role="group" aria-label="移动端选择动捕识别动作">
<button type="button" class="earth-mobile-settings-chip is-active" data-motion-gesture-toggle="rotate_left" aria-pressed="true">左旋</button>
<button type="button" class="earth-mobile-settings-chip is-active" data-motion-gesture-toggle="rotate_right" aria-pressed="true">右旋</button>
<button type="button" class="earth-mobile-settings-chip is-active" data-motion-gesture-toggle="rotate_up" aria-pressed="true">上旋</button>
<button type="button" class="earth-mobile-settings-chip is-active" data-motion-gesture-toggle="rotate_down" aria-pressed="true">下旋</button>
<button type="button" class="earth-mobile-settings-chip is-active" data-motion-gesture-toggle="zoom_in" aria-pressed="true">放大</button>
<button type="button" class="earth-mobile-settings-chip is-active" data-motion-gesture-toggle="zoom_out" aria-pressed="true">缩小</button>
<button type="button" class="earth-mobile-settings-chip is-active" data-motion-gesture-toggle="focus_prev" aria-pressed="true">上个焦点</button>
<button type="button" class="earth-mobile-settings-chip is-active" data-motion-gesture-toggle="focus_next" aria-pressed="true">下个焦点</button>
<button type="button" class="earth-mobile-settings-chip is-active" data-motion-gesture-toggle="layer_prev" aria-pressed="true">上一图层</button>
<button type="button" class="earth-mobile-settings-chip is-active" data-motion-gesture-toggle="layer_next" aria-pressed="true">下一图层</button>
<button type="button" class="earth-mobile-settings-chip is-active" data-motion-gesture-toggle="confirm" aria-pressed="true">确认</button>
</div>
</div>
</div>
<div class="earth-mobile-settings-group" data-settings-tab-panel="shortcuts" hidden>
<div class="earth-mobile-settings-title">快捷键</div>
@@ -1705,6 +1727,28 @@
<button type="button" class="earth-settings-segmented-btn" data-motion-provider="motion_agent" aria-pressed="false">Motion Agent</button>
</div>
</div>
<div
class="earth-settings-item earth-settings-item--stacked"
data-gatekeeper-permission="earth.motion_debug"
>
<div class="earth-settings-copy">
<span class="earth-settings-item-title">识别动作</span>
<span class="earth-settings-item-subtitle">未勾选的动作不会触发星球端控制;Motion Agent 会同步过滤这些动作</span>
</div>
<div class="earth-settings-chip-grid" role="group" aria-label="选择动捕识别动作">
<button type="button" class="earth-settings-chip is-active" data-motion-gesture-toggle="rotate_left" aria-pressed="true">左旋</button>
<button type="button" class="earth-settings-chip is-active" data-motion-gesture-toggle="rotate_right" aria-pressed="true">右旋</button>
<button type="button" class="earth-settings-chip is-active" data-motion-gesture-toggle="rotate_up" aria-pressed="true">上旋</button>
<button type="button" class="earth-settings-chip is-active" data-motion-gesture-toggle="rotate_down" aria-pressed="true">下旋</button>
<button type="button" class="earth-settings-chip is-active" data-motion-gesture-toggle="zoom_in" aria-pressed="true">放大</button>
<button type="button" class="earth-settings-chip is-active" data-motion-gesture-toggle="zoom_out" aria-pressed="true">缩小</button>
<button type="button" class="earth-settings-chip is-active" data-motion-gesture-toggle="focus_prev" aria-pressed="true">上个焦点</button>
<button type="button" class="earth-settings-chip is-active" data-motion-gesture-toggle="focus_next" aria-pressed="true">下个焦点</button>
<button type="button" class="earth-settings-chip is-active" data-motion-gesture-toggle="layer_prev" aria-pressed="true">上一图层</button>
<button type="button" class="earth-settings-chip is-active" data-motion-gesture-toggle="layer_next" aria-pressed="true">下一图层</button>
<button type="button" class="earth-settings-chip is-active" data-motion-gesture-toggle="confirm" aria-pressed="true">确认</button>
</div>
</div>
</div>
</section>
<section class="earth-settings-section" data-settings-tab-panel="shortcuts" hidden>
+1 -1
View File
@@ -280,7 +280,7 @@ export const PATHS = {
export const VESSEL_CONFIG = {
altitudeOffset: 0.2,
maxRenderedMarkers: 0,
maxRenderedMarkers: 3000,
marker: {
baseScale: 7.5,
baseOpacity: 0.88,
+103 -1
View File
@@ -111,6 +111,7 @@ import {
} from "./layer-button-state.js";
import {
DEFAULT_MOTION_PROVIDER,
MOTION_GESTURES,
normalizeMotionProvider,
} from "./motion-protocol.js";
@@ -125,6 +126,7 @@ let autoRotationSpeed = CONFIG.rotationSpeed;
let motionDebugEnabled = false;
let motionProvider = DEFAULT_MOTION_PROVIDER;
let motionDebugSkeletonOnly = false;
let motionEnabledGestures = [];
let activeCamera = null;
let settingsApplyPromise = Promise.resolve();
let boundaryBuildPollTimer = null;
@@ -159,7 +161,7 @@ const SETTINGS_SHEET_MAX_SCALE_X = 0.22;
const SETTINGS_SHEET_MAX_SCALE_Y = 0.18;
const EARTH_SETTINGS_STORAGE_KEY = "planet.earth.settings.v2";
const LEGACY_EARTH_SETTINGS_STORAGE_KEY = "planet.earth.settings.v1";
const EARTH_SETTINGS_VERSION = 16;
const EARTH_SETTINGS_VERSION = 17;
const GRID_LINES_DEFAULT_VERSION = 3;
const SATELLITE_DISPLAY_DEFAULT_VERSION = 4;
const MEDIA_PANEL_DEFAULT_VERSION = 5;
@@ -173,6 +175,22 @@ const KEYBOARD_SHORTCUTS_DEFAULT_VERSION = 13;
const CRUISE_QUEUE_DEFAULT_VERSION = 14;
const AUTO_ROTATION_SPEED_DEFAULT_VERSION = 15;
const NEWS_CATEGORY_FILTERS_DEFAULT_VERSION = 16;
const MOTION_GESTURES_DEFAULT_VERSION = 17;
const MOTION_GESTURE_DEFINITIONS = [
{ id: "rotate_left", label: "左旋" },
{ id: "rotate_right", label: "右旋" },
{ id: "rotate_up", label: "上旋" },
{ id: "rotate_down", label: "下旋" },
{ id: "zoom_in", label: "放大" },
{ id: "zoom_out", label: "缩小" },
{ id: "focus_prev", label: "上个焦点" },
{ id: "focus_next", label: "下个焦点" },
{ id: "layer_prev", label: "上一图层" },
{ id: "layer_next", label: "下一图层" },
{ id: "confirm", label: "确认" },
];
const DEFAULT_MOTION_ENABLED_GESTURES = MOTION_GESTURE_DEFINITIONS.map((item) => item.id);
motionEnabledGestures = [...DEFAULT_MOTION_ENABLED_GESTURES];
const DEFAULT_NEWS_CATEGORY_FILTERS = {
politics: true,
business: true,
@@ -1427,6 +1445,7 @@ function getCurrentSharedSettingsSnapshot() {
motionDebugEnabled,
motionProvider,
motionDebugSkeletonOnly,
motionEnabledGestures: getMotionEnabledGestures(),
mediaPanelActiveTab: normalizeMediaPanelActiveTab(getActiveTVTab()),
satelliteIdleBreathingEnabled: getSatelliteIdleBreathingEnabled(),
satelliteRealAltitudeEnabled: getSatelliteRealAltitudeEnabled(),
@@ -1489,6 +1508,7 @@ function cloneEarthSettings(settings) {
DEFAULT_MOTION_PROVIDER,
),
motionDebugSkeletonOnly: Boolean(settings.shared.motionDebugSkeletonOnly),
motionEnabledGestures: normalizeMotionEnabledGestures(settings.shared.motionEnabledGestures),
mediaPanelActiveTab: normalizeMediaPanelActiveTab(settings.shared.mediaPanelActiveTab),
satelliteIdleBreathingEnabled:
settings.shared.satelliteIdleBreathingEnabled !== false,
@@ -1621,6 +1641,10 @@ function normalizeEarthSettings(rawSettings, defaults) {
typeof sharedSettings?.motionDebugSkeletonOnly === "boolean"
? sharedSettings.motionDebugSkeletonOnly
: defaults.shared.motionDebugSkeletonOnly;
const nextMotionEnabledGestures =
(rawSettings?.version || 0) >= MOTION_GESTURES_DEFAULT_VERSION
? normalizeMotionEnabledGestures(sharedSettings?.motionEnabledGestures)
: normalizeMotionEnabledGestures(defaults.shared.motionEnabledGestures);
const nextMediaPanelActiveTab =
(rawSettings?.version || 0) >= MEDIA_PANEL_ACTIVE_TAB_DEFAULT_VERSION
? normalizeMediaPanelActiveTab(sharedSettings?.mediaPanelActiveTab)
@@ -1689,6 +1713,7 @@ function normalizeEarthSettings(rawSettings, defaults) {
motionDebugEnabled: nextMotionDebugEnabled,
motionProvider: nextMotionProvider,
motionDebugSkeletonOnly: nextMotionDebugSkeletonOnly,
motionEnabledGestures: nextMotionEnabledGestures,
mediaPanelActiveTab: nextMediaPanelActiveTab,
satelliteIdleBreathingEnabled: nextSatelliteIdleBreathingEnabled,
satelliteRealAltitudeEnabled: nextSatelliteRealAltitudeEnabled,
@@ -1746,6 +1771,17 @@ function syncMotionDebugSkeletonOnlyToggle(nextEnabled = motionDebugSkeletonOnly
});
}
function syncMotionGestureControls() {
const enabledGestures = new Set(getMotionEnabledGestures());
document.querySelectorAll("[data-motion-gesture-toggle]").forEach((button) => {
if (!(button instanceof HTMLButtonElement)) return;
const gesture = button.dataset.motionGestureToggle || "";
const active = enabledGestures.has(gesture);
button.classList.toggle("is-active", active);
button.setAttribute("aria-pressed", active ? "true" : "false");
});
}
function dispatchMotionSettingsChange() {
const effectiveDebugEnabled =
rotationMode === ROTATION_MODE.MOTION && autoRotate && motionDebugEnabled;
@@ -1756,6 +1792,7 @@ function dispatchMotionSettingsChange() {
preferredEnabled: motionDebugEnabled,
provider: motionProvider,
skeletonOnly: motionDebugSkeletonOnly,
enabledGestures: getMotionEnabledGestures(),
},
}),
);
@@ -1866,6 +1903,18 @@ function normalizeCruiseModules(nextModules) {
: [...DEFAULT_CRUISE_MODULES];
}
function normalizeMotionEnabledGestures(nextGestures) {
const sourceGestures = Array.isArray(nextGestures)
? nextGestures
: DEFAULT_MOTION_ENABLED_GESTURES;
const normalizedGestures = Array.from(
new Set(sourceGestures.filter((gesture) => MOTION_GESTURES.has(gesture))),
);
return normalizedGestures.length > 0
? normalizedGestures
: [...DEFAULT_MOTION_ENABLED_GESTURES];
}
function normalizeCruiseQueueMode(mode) {
return ALLOWED_CRUISE_QUEUE_MODES.has(mode) ? mode : DEFAULT_CRUISE_QUEUE_MODE;
}
@@ -2396,6 +2445,10 @@ async function applyEarthSettings(settings, { applyLayers = true } = {}) {
persist: false,
suppressStatus: true,
});
setMotionEnabledGestures(settings.shared.motionEnabledGestures, {
persist: false,
suppressStatus: true,
});
setActiveTVTab(settings.shared.mediaPanelActiveTab);
keyboardShortcuts = normalizeKeyboardShortcuts(settings.shared.keyboardShortcuts);
renderShortcutSettings();
@@ -2428,6 +2481,10 @@ export function getMotionDebugSkeletonOnly() {
return motionDebugSkeletonOnly;
}
export function getMotionEnabledGestures() {
return normalizeMotionEnabledGestures(motionEnabledGestures);
}
export function setMotionDebugEnabled(
nextEnabled,
{ persist = true, suppressStatus = false } = {},
@@ -2517,6 +2574,34 @@ export function setMotionDebugSkeletonOnly(
return motionDebugSkeletonOnly;
}
export function setMotionEnabledGestures(
nextGestures,
{ persist = true, suppressStatus = false } = {},
) {
const normalized = normalizeMotionEnabledGestures(nextGestures);
const previous = getMotionEnabledGestures();
const changed =
normalized.length !== previous.length ||
normalized.some((gesture, index) => previous[index] !== gesture);
motionEnabledGestures = normalized;
syncMotionGestureControls();
ensureMutableEarthSettingsState();
earthSettingsState.shared.motionEnabledGestures = [...motionEnabledGestures];
if (changed) {
dispatchMotionSettingsChange();
}
if (persist) {
persistEarthSettings();
}
if (!suppressStatus && changed) {
showStatusMessage("动捕识别动作已更新", "info");
}
return getMotionEnabledGestures();
}
export async function applyDeferredLayerVisibilitySettings(options = {}) {
const layerVisibility = deferredLayerVisibilitySettings;
deferredLayerVisibilitySettings = null;
@@ -3886,6 +3971,7 @@ function integrateMotionSettingsIntoRuntime() {
markRuntimeModeSection("[data-auto-rotation-speed-slider]", ROTATION_MODE.ROTATE);
markRuntimeModeSection("[data-motion-debug-toggle]", ROTATION_MODE.MOTION);
markRuntimeModeSection("[data-motion-provider]", ROTATION_MODE.MOTION);
markRuntimeModeSection("[data-motion-gesture-toggle]", ROTATION_MODE.MOTION);
syncRuntimeModeSections();
}
@@ -4434,6 +4520,21 @@ function setupSettingsControls() {
});
});
document.querySelectorAll("[data-motion-gesture-toggle]").forEach((motionGestureButton) => {
if (!(motionGestureButton instanceof HTMLButtonElement)) return;
bindListener(motionGestureButton, "click", () => {
const gesture = motionGestureButton.dataset.motionGestureToggle;
if (!gesture) return;
const nextGestures = new Set(getMotionEnabledGestures());
if (nextGestures.has(gesture)) {
nextGestures.delete(gesture);
} else {
nextGestures.add(gesture);
}
setMotionEnabledGestures(Array.from(nextGestures));
});
});
const mobileSettingsReset = document.getElementById("mobile-settings-reset");
bindListener(mobileSettingsReset, "click", () => {
resetEarthSettings();
@@ -4454,6 +4555,7 @@ function setupSettingsControls() {
syncMotionDebugToggle(motionDebugEnabled);
syncMotionProviderControls(motionProvider);
syncMotionDebugSkeletonOnlyToggle(motionDebugSkeletonOnly);
syncMotionGestureControls();
void setupBoundaryPrecisionControls();
}
@@ -165,3 +165,21 @@ export function applyEarthInteractableEvent(earth, payload = {}) {
export function getEarthInteractableMarkers() {
return earthInteractableLayer.getMarkers();
}
export function getEarthInteractablePointerIntersections(options = {}) {
return earthInteractableLayer.getPointerIntersections(options);
}
export function setEarthInteractableMarkerState(marker, state = "normal") {
earthInteractableLayer.setMarkerState(marker, state);
}
export function clearEarthInteractableSelection() {
earthInteractableLayer.getMarkers().forEach((marker) => {
earthInteractableLayer.setMarkerState(marker, "normal");
});
}
export function updateEarthInteractableVisualState(focusType, focusObject, camera) {
earthInteractableLayer.updateVisualState(focusType, focusObject, camera);
}
+16
View File
@@ -1677,6 +1677,22 @@ const CARD_CONFIG = {
{ key: 'length', label: '船长', unit: 'm' },
{ key: 'received_at', label: '更新时间' }
]
},
earth_interactable: {
icon: '📍',
title: '交互点详情',
className: 'earth_interactable',
fields: [
{ key: 'label', label: '名称' },
{ key: 'kind', label: '类型' },
{ key: 'id', label: '标识' },
{ key: 'latitude', label: '纬度' },
{ key: 'longitude', label: '经度' },
{ key: 'description', label: '说明' },
{ key: 'source', label: '来源' },
{ key: 'status', label: '状态' },
{ key: 'updated_at', label: '更新时间' }
]
}
};
Loaded 100 of 125 files, more files were not shown because too many files have changed in this diff. Show more